CLPEX Chat Board

http://www.guardian.co.uk/law/2011/oct/ ... NTCMP=SRCH

This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed.
But more importantly, as far as mathematicians are concerned, the judge also ruled against using similar statistical analysis in the courts in future.

Thanks to David Fairhurst for highlighting this interesting judgment.

It is important to note that the judge apparently qualified his decision.

And so he decided that Bayes' theorem shouldn't again be used unless the underlying statistics are "firm".

This appear to be something less than a complete rejection but a sensible qualification based on the ‘rubbish in, rubbish out’ theory.

I am a member of the group referred to led by Norman Fenton and Amber Marks.

Now, Fenton and his colleague Amber Marks, a barrister and lecturer in evidence at Queen Mary, University of London, have begun assembling a group of statisticians, forensic scientists and lawyers to research a solution to bad statistics. "We want to do what people failed to do in the past, which is really get the legal profession and statisticians and probability guys understanding each other's language," says Fenton.

My motivation for joining the group is a belief that I have not fully understood the possibilities inherent in the application of statistics to expert evidence and the fact that the ‘prosecutor's fallacy’, referred to in the article was one of the reasons Shirley was wrongly charged and put on trial for perjury.

Steve Horn (Outsider) has referred to this fallacy in previous postings.

Fenton believes that the potential for mathematics to improve the justice system is huge. "You could argue that virtually every case with circumstantial evidence is ripe for being improved by Bayesian arguments," he says.
But the real dilemma is finding a way to help people make sense of the calculations. The Royal Statistical Society already offers guidance for forensic scientists, to stop them making mistakes. Lagnado says that flowcharts in the style of family trees also help jurors visualise changing odds more clearly. But neither approach has been entirely successful. And until this complex bit of maths can be simply explained, chances are judges will keep rejecting it.

There is little doubt in my mind that the built in bias against statistical evidence is the result of a general ignorance within the justice system about the issue and a reluctance to be educated. I might say that I share that ignorance but hope that my involvement with Professor Fenton and his team might open my eyes to a further way of maximising the potential for forensic evidence to aid the detection and prevention of crime.

Let me get this staight. A British judge has ruled that Thoman Bayes is unreliable and his theorem and conclusions derived therefrom are not admissible in court.

But no British judge has ruled that the Scottish Criminal Records Office fingerprint examiners with a proven track record of error are unreliable, so their conclusions are still admissible in court.

Those are some interesting judges you've got there.

Can’t argue with you there Tazman.

Same of the very same experts who made two erroneous fingerprint identifications in the Marion Ross murder case and whose colleagues were accused of criminality in a top level enquiry conducted by the police are still preparing and giving evidence in Scottish courts.

This decision was a product of the fertile minds of Scotland’s prosecuting authority, the Crown Office, who despite everything has refused to rescind their authority.

How this stupidity plays out across the world is hard for me to tell but when the Fingerprint Inquiry Report publishes its findings I will certainly be taking immediate steps to challenge the evidence these Scottish experts have given and also any evidence given by Messrs Leadbetter and Swann who were so vociferous in their support of the SCRO.

Such prosecution decisions do however mask an even greater problem in Scotland, and elsewhere across the world, and that is how expert evidence in general is treated by our justice system.

Part of the Scottish problem is our accusatorial system which encourages lawyers to employ experts who will support their argument. This does not appear to be the best way of resolving forensic and other complex issues.

Generally judges are not expert at assessing experts and often allow far too much leeway in accepting the expert conclusions offered without rigorous challenge.

There is absolutely no doubt in my mind that some jurors are incapable of understanding expert evidence and the guidance given to them is inadequate and flawed.

I would like to see expert evidence being prepared for the court and not one side of the other or have it examined by judges who are competent and aware.

The recent case of Amanda Knox who was convicted on badly tainted DNA evidence is just the latest stunning example of what appears to be a world wide problem.

http://blogs.seattleweekly.com/dailywee ... invest.php

Dear Mr. McKie,

What I fail to grasp is why defense attorneys allow these miscreants to testify. In the US, if an examiner made a high profile error, you can bet heavily that any subsequent testimony would be concluded with an aggressive attack by the defense attorney stressing the earlier wrong identifications. Have the examiners in your daughter's case been given a pass on cross examination? Are the defense attorneys uninformed? Is it considered poor manners to impeach a witness? I do not understand the lack of a strong adversarial cross examination. The jurors do not necessarily need to understand complex or sophisticated scientific processes, but why can defense attorneys not use such heavily publicized mistakes to challenge these people when they continue to represent themselves as experts? It should not take another inquiry to settle that issue!

When I challenged my defence Counsel for not calling evidence capable of showing the police had number 2 out his mouth before turning to view my ID Parade his excuse I am afraid shocked me to the core:

I did not want to be seen to be calling the police liars he said

This was also his excuse for not calling evidence that the police intimidated another witness with his criminal injuries claim.

So there is a culture not to attack police evidence in this country which has existed for decades.

The same Counsel went on to defend the Lockerbie Bomber Megrahi.

Says it all really.

There is another problem evident from Shirley McKie's case and David Asbury.

Both were refused legal aid to challenge the experts and only because the BBC paid for this were they able to make such challenges and my god look at the results.

What about all the other cases the BBC did not support or know about?

People are regularly being refused legal aid to make any serious challenges to expert evidence in Scotland.

Our Country is backwards.

Dear Tazman,

I am hoping that the Fingerprint Inquiry Report will address the issue of why these ex-SCRO experts ( and a number of so called independent experts), whose work has been proven to be erroneous, have been allowed to continue preparing evidence for court and delivering that evidence.

A major problem in Scotland, if not the UK, is that expert evidence often goes unchallenged and ‘strong adversarial cross examination’ is the exception rather than the rule.

While there is a dangerous lack of knowledge within the justice system in respect of expert evidence this does not explain the defence failures to challenge these witnesses who continue to give potentially flawed evidence. I believe the cause is part cultural and part inefficiency and lack of preparedness of some defence counsel.

We have a complacent justice system in which even the constituent parts apparently in opposition effectively play the same game. This culture ensures the continuance of the system, sometimes unfortunately at the expense of the innocent.

Where governments and major legal institutions, including the courts, are in denial that any problems exist the job of the challenger is extremely difficult and sometimes impossible.

The focus in the UK is on first aid and not major surgery.

Best wishes,

Iain

I always find that it is amazing how we can get off topic so quickly. By the way, the R v T judgement is freely available to download. Before you start commenting on it, I suggest that you read through the judgement fully. I had to read it twice, as it is lengthy and, if you are not familiar with the arguments surrounding the use of Bayes theorem, will need careful scrutiny. Ian summarised the ruling correctly when he said that the judge was critical of the quality of underlying evidence to support the numbers used in the Bayesian calculation presented in R v T. There are issues with the use of Bayes theorem in court with juries etc. that are not literate in the use of statistical approaches to evidence evaluation. It is interesting that for DNA evidence presentation, these arguments have been well rehearsed and a language for the presentation of results in court articulated (see various articles in Science and Justice over the past fifteen years).

I've read two responses to the R v T decision now.

First this one http://agc-wopac.agc.gov.my/e-docs/Jour ... 020639.pdf

And secondly this one: http://www.scienceandjusticejournal.com ... 1/abstract


I have to say that as someone who already thinks that the use of likelihood ratios in impression evidence is little more than forensic snake oil, these responses come off as 1) a pathetic attempt to paint likelihood ratios as some objective gold standard of presenting evidence and 2) people who are out to protect the funding of their livelihood.

Perhaps they should rename this technique the livelihood ratio because if it gets rejected by the courts, the people writing these responses will have their grant funding and teaching/training profits dry up.

The ethical reality however that lives in stark contrast to these idealized views of the likelihood ratio's use is that statistical calculations in arriving at evaluative weight becomes a crutch and a black box that is abused. This is a scenario that stands on top of the fact that the calculations themselves are suspect given the qualitative nature of what we do and the inability to quantify the full comparison process. Wrap this up with the bow and explicit recognition that at its heart there's still the element of interpretation and subjective probabilities on the part of the examiner, you get a less than stellar reality.

We just had a thread about the abuse of probabilities that required a response from SWGFAST, which obviously shows that people will undoubtedly lean on probabilities and do a bigger disservice to representing the true weight of the Evaluation.

1) The explicit application of Bayes theorem to the evaluation of forensic evidence and it's presentation in court was already banned from UK court since 1996. So nothing really new here
2) The court in R v T was concerned with the lack of data supporting the use of the likelihood ratio for footwear evidence. The court, in a difference way than the NAS report, says that the community should show some data and validate models.
3) Tazman and Iain McKie make very good points, that some individuals tempted to revisit 2000 years of philosophy should consider before using terms such as 'scientific method' as a proxy for ignorance.

1) The explicit application of Bayes theorem to the evaluation of forensic evidence and it's presentation in court was already banned from UK court since 1996. So nothing really new here

Am I understanding this correctly. Using this type of application was banned but it was still being used by the FSS?

I'm a little unclear about the meaning of the word 'explicit' above. I'm wondering if that means that Bayes theorem shouldn't be the sole determinant in arriving at a conclusion. Is it permissible to use it as supporting data in the UK?

Michele,

There is a difference between the Bayes theorem and the likelihood ratio. The likelihood ratio is nothing more than a tool to test and compare hypotheses, in a same way as p-values can be used to test hypotheses. Now it is true that the Bayes theorem uses a likelihood ratio too, but it has other components that were the issue.

Bayes theorem has been explicitly banned from UK courts, LRs are used every day by all British forensic providers, and by most major forensic laboratories in the World when it comes to DNA, trace and other types of forensic evidence.

I don't understand your last 2 questions. They are confusing many different concepts. You may want to refer to "Interpreting Evidence: Evaluating Forensic Science in the Courtroom" by Robertson and Vignaux, Wiley and Sons, 1995.

Cedric:

I think Michele means would it have been ok or is it ok now to express the conclusion in the traditional sense "same source/different source" but then suport that conclusion with a likelihood ratio. Secondly, is it inappropriate to ONLY use the likelihood ratio to be the basis of a conclusion in a footwear examination?

Michele:

I don't know if it is ok now but I think in this case because he couldn't make a same source/different source determination (ie. it was inconclusive) he used the verbal scale (based on the results of the L.R.) discussed in #31 of the opinion to say something more than "I can't tell."

GrayMatter,

The key question that you are asking is "how do we support conclusions from forensic examination when reporting them to our customers?"

It is indeed a very important question. A group of us believe that it is indeed appropriate, and even necessary, to use a LR. Some others believe that it may be more appropriate to use some other statistics. Other also believe that some form of conversion of the statistics in the natural language is better.

The forensic community needs to support and expand on the work done by Saks, Koehler, McQuiston and others on this subject. And as a community we need to have a discussion on what is the most efficient and fair way of reporting our findings.

One thing for sure is that we need to move away from categorical conclusions, support our conclusions by data, and stop relying entirely on 'training and experience'.

PS: I am NOT saying that 'training and experience' is a bad thing. On the contrary, it desperately needed; however, it cannot account for 100% of the support for our conclusions.

GrayMatter,

From the statistics and the chart listed in 31, I can see someone thinking that the conclusion was inconclusive.

On the other hand, from the testimony listed in 24 stating “Overall it is my view somewhat unlikely that the observed correspondence would have been obtained as a result of mere coincidence had the recovered footwear not made the mark in question. … there is at this stage a moderate degree of scientific evidence to support the view that the [Nike trainers recovered from the appellant] had made the footwear mark..”, if I were a juror and without seeing the chart, I’d take this to mean that the expert thought the mark was made by this individual. “Moderate Support” sounds pretty good to me… until you look at the chart and see it’s one of the lower valued conclusions. If I’m reading this correctly, I don’t think the jurors were shown the chart. It goes to show that even if statistics, probabilities or LR’s are used, articulating the meaning behind what you’re saying is extremely important.

I couldn't agree more with Cedric. I think ‘training and experience’ is a weak basis for a conclusion. I don't believe this because of the work of the people he's listed, I believe it because it puts too much weight into elements that can't be looked at and tested. It's like saying 'Trust me, I've been doing this a long time'.