Statistics and Misidentifications - The weeks Detail

[Outsider]

mdavis wrote

Throwing statistics around has, obviously given a generic overview of what a pseudo-mathematician might think when observing a few case or test results for the generic latent print examiner.

Don’t confuse “back of envelope” calculations with pseudo-mathematics. I cannot think of any field of science were rough calculations lead to less understanding than no calculations. By all means criticise the maths and the assumptions.

Speculation of routine numbers of mid-idents stems from the assumption that the fault lies with the inevitability of error with all examiners, when in fact it should be applied to each individually.

This is not quite true. What I am trying to do with my statistical observations about the McKie case is show that, whatever might be the cause of a misidentification, it will happen disproportionately to an accusation where there is no evidence that the suggested wrongdoing happened, compared with a case where the crime is all too obvious. To someone who is “statistically aware” the McKie case looks like an abnormality, to everyone else it just looks like another accusation.

What causes errors and reject products in an interesting topic, one I am very familiar with. In every job it is essential to understand process capability. If the process is not capable of producing what is required then problems are inevitable. In fingerprinting you claim zero errors due to process capability. Whether or not that is true I agree that each examiner, and each department, might have another source of errors in addition to the process capability rate. What is more, this part of the error rate will be unstable, varying from day to day. So, I agree that even if an industry-wide error rate was estimated, there would be no good reason to assume that real probability of error in any one court case would be anything close to this.

The only way to know for sure that the process produces no errors and to know that your department produces no errors is to find a way to test it to breaking point. If a beam has to hold up one ton there is no point in testing at half a ton. Testing at 2 tons would be best. Don’t confuse a test that you might devise to demonstrate something with a test where you really want to understand the processes and mechanisms that you work with. I have seen many of the former in second rate companies where the real purpose of testing and data collecting is for marketing or passing some quality assurance standard. World class quality comes from understanding processes and why results vary.

There is something which makes talk about error rates in fingerprint almost surreal to me. It is the only profession that I can think of that does not have any idea when it has failed. When a doctor kills his patient he at least knows that the patient is dead. When an airline pilot crashes the plane it is all too obvious. Unless some new evidence is unveiled after conviction, nobody has any reason to question whether a fingerprint ID used in court was good. For this reason I don’t think you can use the “if is ain’t broke don’t fix it” argument. I think you should go out of your way to find out if it is broke, using the hardest possible simulations with known sources, harder than anything that might be used in a court case. You can then quote the results of your department’s tests to juries.

[Pat A. Wertheim]

Hi Steve,

You seem to be saying something along the lines of a letter I read about 25 years ago from an old time fingerprint examiner. The first fingerprint computers to look for similar "points" in different prints (AFIS) came into practical use in 1978. I remember reading in the early or mid 1980's (I cannot remember the exact year or even the publication) a letter to the editor of a fingerprint newsletter. The letter cautioned that, in an AFIS environment, a fingerprint examiner would have to be more cautious or use a higher standard for identification than in the old environment. The author said that when a detective comes to an examiner with a suspect, the suspect has already been "prequalified" (my word, not the author's) and there is already an indication he is guilty, but with AFIS there is no such indication or basis on which to conduct a comparison. Therefore, examiners of latent prints to AFIS "hits" needed to be much more careful.

I remember thinking, "Nonsense! We make every single identification 'to the exclusion of any other person in the world' so what does it matter how we come up with a suspect?"

But if ever there was a classic example of what the writer of that letter was talking about, it was Brandon Mayfield.

So, I would be interested in hearing your comments on that old letter. I have searched the old "Identification News" issues without finding it, but my collection is missing a few. But to my old time friends in Texas, I believe that letter may have been in the old TDIAI newsletter sometime in the early to mid 1980's. If anybody has a collection dating back to then and you don't have anything better to do, have a look through and see if you can find it (my old newsletters got pitched during one of several moves since then).

Your thoughts, Steve?

[mdavis]

In fingerprinting you claim zero errors due to process capability. Whether or not that is true I agree that each examiner, and each department, might have another source of errors in addition to the process capability rate.

You assume errors. My point is that a good examiner will avoid errors by avoiding questionable impressions that are not "error prone", and will confirm each non-error by an independent ACE-V verification. Assuming each examiner and each department is aware of the potential for error, it can be effectively eliminated. I assume honesty and integrity, while others assume otherwise.

This is off-topic. I intended to point out that cases involving impressions of marginal sufficiency, such as McKie, are the root of all evil. The simple fact that experts do not agree that the impression is or is not an ident is sufficient that the impression is inadequate. Process capability only counts when the guidelines are being observed, otherwise we lapse into questions of integrity of individuals, which is should be addressed separately and quickly.

[Outsider]

Hi Pat,

In statistical theory the bigger the population of comparisons, the lower the certainty or confidence level of an identification. This is just random matching theory, the more times you throw a handful of dice the more likely that pre-specified pattern will emerge just by chance. If it really is true that individualisation occurs, not just in theory or in the best of conditions but day after day in every fingerprint department, then you are right that it should not matter where the match comes from. Until there is some real evidence of this, I would urge extreme caution with big database searches.

However I can see that there is a built-in safety mechanism in databases. It is extremely unlikely that a random match will hit on someone who has any connection with the crime. This is an additional probability factor and it is perfectly legitimate for a jury to use it to convict, but it is not up to a fingerprint expert to make any inference from anything other than the images. Whether or not a match has identified someone in prison in Alaska or the violent brother-in-law of the victim should not matter to the fingerprint department. A fingerprint expert should only report honestly to the jury about the certainty of an identification. I would be very happy for something lower than 100% certainty to be reported both to the police and the jury and it would be part of the fingerprint expert’s skill to estimate this and convey it in an meaningful way to the jury.

One thing that would worry me would be if any sort of cherry-picking of the hits from an AFIS system takes place on any grounds other than the similarity of the image in the database to the latent print. If I was on a jury and was told that a fingerprint match came from a database search I would assume that the truly best hit was chosen and verified without any knowledge of who had been identified. If it turned out that this person had a motive to commit the crime I would be convinced of guilt. However, if the reality was that every AFIS hit who did not have a connection with the crime was discarded before the detailed human analysis stage, and I was not told this, then I would feel that I had been misled. I don’t know if this happens but it is the logical thing to do if you believe in individualisation.

[Kasey Wertheim]

I remember after an internet search a while ago I found Kasey Werhiem reporting in New Scientist an error rate of about 1 in 3000 comparisons in an informal study (please correct me if I am wrong).

You are correct - an excerpt from the Sept. 17, 2005 article:

"One unpublished study may go some way to answering the critics. It documents the results of exercises in which 92 students with at least one year's training had to match archive and mock "crime scene" prints. Only two out of 5861 of these comparisons were incorrect, an error rate of 0.034 per cent. Kasey Wertheim, a private consultant who co-authored the study, told New Scientist that the results have been submitted for publication."

http://www.newscientist.com/article.ns?id=dn8011

The pilot study was published a few months later in the Journal of Forensic Identification.

-Kasey

[Daktari]

The figure 1 in 3,000 for the probability of error in fingerprint comparisons, especially for students after one years training does seem a bit low. Let’s take a more realistic example. In the Evett-Williams tests the SCRO team got nine out of ten correct. (The Dutch team including Arie Zeelenberg, if I remember correctly, got one out of ten) So, as a working example, we can take the probability of SCRO making an error to be one in ten. That means that an examiner will make an error, on average, once in every ten comparisons or out of ten examiners given the same mark one will make an error. In fingerprint examination there are at least four possible types of error.
Given that in probability theory ‘and’ means ‘times’ and ‘or’ means ‘plus’ if we are asked to believe that they all ten made the same mistake that would be a probability of one in ten thousand million. If the probability of an error is one in twelve then the probability of them all misidentifying the mark rises to just under one in 62,000 million.

The figure of 99 in every 100 errors being ‘trapped’ by verification appears to have been plucked from thin air. Let’s assume that it has some logical reasoning. That would mean that when the Belfast Bureau checked the SCRO officers work for one year before and one year after the disputed identification of mark Y7 they would have found a number of errors that had slipped through the verification process. The reality is that they found the SCRO work to be 100% accurate. Or maybe they too were part of the conspiracy that the McKies allege but have failed to substantiate.

We are asked to believe that several experts made two mistakes in the same case and both these errors were not picked up by verification. Maybe someone with a larger calculator than I have will be able to work out the probability of that happening.

[mdavis]

So-called "tests" focus on less than "routine" comparisons. They wouldn't be potentially meaningful tests otherwise. In daily work, the majority of latent impressions are either insufficient or they are quite easy and obvious. To take test results from truly worthy examples will obviously push errors. If those tests are time-pressured, that adds a factor. If those tests do not allow the examiner to opt-out of making a call, you place the test takers in an artificial dis-comfort zone that may be avoided during routine daily work.

[Daktari]

So Dave Charlton thinks

after all, one universal truth is that press and media will always misquote, miss attribute(sic) and misrepresent.

Maybe that’s why Taggart and McKie (or are they one in the same?) are so reluctant to have their much trumpeted ‘revelations’ published in the Scottish papers.

He goes on to say

Bayle is right to be worried about standards.

Maybe he should start with his own. Leaving aside his association with a number of false misidentification claims his apparent inability to tell the truth is a worry. His involvement in the Lockerbie Trial, his reasons for leaving the Met, are sufficient examples of this. These were not a case of a police officer understandably lying to save her job and pension, they were deliberate attempts to mislead the Justice1 Committee. My understanding is that Bayle may yet be prosecuted for the falsehoods in his evidence in the Leeds drugs trail last July.

and also

I also lay blame at the criminal justice system, the lawyers and barristers as wel (sic) call them in UK.

I agree with the sentiment of that. However there is a danger that presentational skills carry more sway with a jury that the actual substance of the evidence. Lord Johnson made much of this in his summing up in the Shirley Cardwell aka McKie Perjury trial. (Available on the Scottish Courts website.) Had the jury known about her trip to see Peter Swann, her previous conviction for the same thing, the refusal of the Senior Officer to let her see the body despite her asking three times and Procurator Fiscal (a sort of DA) ordering an investigation into the keeping of crime scene visitor logs it may have ended quite differently!


There can be little doubt that the jury was left confused by the amount of contradictory evidence presented. All a lawyer or an advocate, as we call barristers in Scottish Courts, has to do to secure an acquittal is introduce a reasonable degree of doubt.

However all the bickering may soon be at a halt because, as you know, the SNP
Government has promised a Judicial Inquiry into the whole affair and to quote Deputy Leader Nicola Sturgeon, “The truth will be told”.

What about

My request is for all parties to be very very careful about how we discuss issues and debate them.

Like the outrageous Zeelendberg / McKie on-line petition? Bayle was one of only three UK experts who signed up to this. I’ll have to guess that since there were only 171 worldwide signatories the ratio elsewhere was about the same.

Finally

He (Bayle) is right to be concerned to see uniformity of knowledge and understanding across the global forensic domain.

You will be aware that the SCRO experts all received at least five training coupled with regular refresher courses. Contrast that with this statement from a USA practitioner;

“I was in NO WAY qualified to testify to fingerprint identifications when I was first accepted in court as an “expert” in the late 1970’s. I attended a two-week class in Henry classification which also touched on latent print development and latent print identification. Our comparison exercises were all inked to inked. After that, there was no OJT in my department and I had to teach myself. Within two years, I was testifying to identifications. I never had a senior examiner check my work and never had a single proficiency test. And don’t make the mistake of thinking I was the exception.”

Although he will probably be too busy pouring over the Scottish papers looking for the oft-promised ‘revelations’ to read this I’ll spare his blushes anyway.

[H. B. James]

Daktari,

Can you offer nothing constructive? You behave as a spoiled bully who only derives enjoyment from destroying what others have built.

David Charlton is a respected scientist whose work, publications, and opinion is known around the world. With the additional projects he is working on, he has the potential of becoming one of the most respected researchers in the science of fingerprints in a century.

Iain McKie is known and respected throughout Scotland and other countries where he has spoken. His book tells the truth. If it does not, why don't you and the other former SCRO "experts" sue him for slander and liable? The absence of such a law suit is tacit acknowledgement that there is no basis for such a suit, as it is acknowledgment his book is the truth.

Allan Bayle's name is known throughout Europe and North America. He is well respected by the true scientists in our business for his forward thinking adoption of Ridgeology years ahead of the rest of the UK. Yes, he made an erroneous exclusion in reviewing one case in his career, but when question was cast on his conclusion, he reviewed the case, discovered he was wrong, immediately announced that he had made a mistake, and publically apologized. That's more than can be said for you or the other former SCRO "experts."

The jury at Shirley McKie's trial was not "confused." Not one single "guilty" vote, not one single "unproved" vote, but a unanimous 15 votes for "not guilty." That, Daktari, is clear evidence of an absence of confusion.

You quote Pat Wertheim regarding his qualifications in the 1970's. In case you haven't checked the calendar lately, we have traveled thirty years through time since then. I believe his credentials have been well established. To challenge his qualifications today based on a 30 year old curriculum vitae shows the depths to which you will sink in your attempts to detract from your critics.

One thing I noticed in all of this -- the people named above whom you criticize are all known on this discussion board by the real names under which they do business. What is your name, Daktari? Or are you afraid of the consequences of coming out in violation of the "gag order" you agreed to in order to collect your £70,000?

[Daktari]

Just a couple of wee points

You say

The jury at Shirley McKie's trial was not "confused." Not one single "guilty" vote, not one single "unproved" vote, but a unanimous 15 votes for "not guilty." That, Daktari, is clear evidence of an absence of confusion.

My point was that the jury were likely to be confused over the mass of contradictory fingerprint evidence that was delivered. If it was the case that this caused reasonable doubt in their minds, as it probably was designed to do, then they were correct to return a not guilty verdict.

However a not guilty verdict does not mean that the person was innocent!
The under-used Scottish verdict of Not Proven is I think a better verdict since the prosecution has to prove its case, the defence does not.

You say that Allan Bayle apologised after he had a made a mistake. I’m quite sure that if it is ever shown the SCRO people involved in the McKie scandal made a mistake they will be the first to apologise.

Also, I suggest you have a look at Bayle's evidence in the Brazil Trial at Leeds last July.

But if Alex Salmond keeps his promise we may yet find out who is telling lies.

[Steve Everist]

Let's try to keep this thread on topic as much as possible. Although Steve Horn's statistics do reference the McKie case, they only do so relative to a statistical level. I know that most threads tend to morph into something else if they go 10+ posts, but this one has ventured into an area that already has a thread for discussion.

Since there are other threads relating to the non-statistical issues. Feel free to further discuss them there, or start a new thread if you feel it's necessary and the other ones have gotten a bit long in the tooth.

Thanks

[L.J.Steele]

Sorry for the further divergence into the jury issue, but I'm perplexed by the discussion of the UK jury system.

I agree with the sentiment of that. However there is a danger that presentational skills carry more sway with a jury that the actual substance of the evidence.

***

There can be little doubt that the jury was left confused by the amount of contradictory evidence presented. All a lawyer or an advocate, as we call barristers in Scottish Courts, has to do to secure an acquittal is introduce a reasonable degree of doubt.

In the American system, it is the obligation of the prosecution to prove the case beyond a reasonable doubt. The defense need do nothing. When it cross-examines, or offers its own evidence, it is its job to test the prosecution's evidence. And if that evidence cannot stand up to scrutiny, and the jury is left with a reasonable doubt, then an acquittal is the proper result.

I am less familiar with the UK system, but I assume similar principles to apply. How is reasonable doubt defined for jurors in your standard jury instructions?

Much also depends on whether one has respect for the ability of jurors to do their job. Certainly, there have been jury verdicts I've disagreed with, but on the whole, I think the jury system does work and is generally more fair than the alternatives.

The prosecution's job (and the defense's) is to explain technical evidence at a level the jury can understand. This is difficult as, at least in the US, a juror can't raise his hand and ask a question if he's unclear about a concept. Are you suggesting that fingerprint evidence is too complex to be explained to a typical juror? Or doubting the ability of a juror to weigh conflicting testimony?

[Pat A. Wertheim]

Steve has a point -- the SCRO folks have hijacked other threads and will probably hijack more in the future. Be patient with them, Steve. They have one-track minds.

Lisa, the jury's decision in Scotland is drastically different from that in the US. In Scotland, there are 15 jurors. There are three verdicts for each juror to choose from: Guilty, Not Proved, and Not Guilty. If a juror believes the prosecution has proved its case, he/she votes "Guilty." If a juror believes the person is possibly guilty but the prosecution just didn't prove it beyond a reasonable doubt, they can vote "Not Proved." If the juror believes the person is innocent and the prosecution is altogether wrong, they vote "Not Guilty." I'm sure our SCRO friend Daktari will try to correct me on this, but personally, I believe the solicitors and barristers who told me this.

So, each of the 15 jurors casts his/her vote from the three choices and the verdict goes to a simple majority of 8. So even if only 8 jurors vote guilty, the case then goes to sentencing. Now, to be fair, I don't know what happens with a three way split in which no verdict gets the eight votes required. I guess that's what we would call a hung jury in the US and I presume the procurator fiscal's office can then take the case for retrial.

But after Shirley's trial, even the courthouse personnel told Dave Grieve that a unanimous "Not Guilty" verdict showed that the jury absolutely believed in Shirley McKie's innocence. There was no confusion in the jury. To say that I was a showman who confused the jury so that not a single juror voted "Not Proved" is, itself, confusion.

[Daktari]

LJ always a pleasure to answer questions.

How is reasonable doubt defined for jurors in your standard jury instructions?

Basically this is covered in the Judge's summing up as are the range of options a jury can choose from. . Lord Johnson's summing up is particularity interesting in this respect. It's available on the Scottish Courts website or an edited, sorry moderated, version is on you-know-who's website.

Are you suggesting that fingerprint evidence is too complex to be explained to a typical juror?

Well, if pushed, yes it can be. But what I was really driving at was in this particular case the jury was faced with a mass of conflicting evidence. Whilst no one knows how a jury arrives at its decision or what factors influence this, I think it is fair to say that, the Judge's comments will play a big part. One could ask what is a typical juror?

Or doubting the ability of a juror to weigh conflicting testimony?

In this particular case, yes. You may be interested to know that there is some debate taking place over here at the moment about whether complex or technical trials should be heard by jury. Fraud cases for example. In general, it must be difficult, when confronted by two apparently equally qualified and experienced practitioners, to decide who's version to accept. We also have to accept that many jurors simply have little or no interest in what's going on around them and become bored with hours of repetitive testimony

Hope that helps

and Pat

Who said that you were

a showman who confused the jury

?

[L.J.Steele]

Well, if pushed, yes it can be. But what I was really driving at was in this particular case the jury was faced with a mass of conflicting evidence. Whilst no one knows how a jury arrives at its decision or what factors influence this, I think it is fair to say that, the Judge's comments will play a big part. One could ask what is a typical juror?

This is also a question here. I suspect one would have to consider a relatively open-minded person with the equivalent of a high school education and moderate life experience. I assume the counterpart to a jury commissioner here could give one a statistical average of the appropriate jury pool.

In this particular case, yes. You may be interested to know that there is some debate taking place over here at the moment about whether complex or technical trials should be heard by jury. Fraud cases for example. In general, it must be difficult, when confronted by two apparently equally qualified and experienced practitioners, to decide who's version to accept. We also have to accept that many jurors simply have little or no interest in what's going on around them and become bored with hours of repetitive testimony.

The problem there may be the manner in which the testimony is presented. I had the honor of sitting on a jury a few years back. I tried to listen as much as I could during the deliberations, to get a sense of what jurors thought (and to avoid any temptation of letting my legal training interfere with the judge's instructions). I'd say the jurors were attentive to details, including details I'd missed, and were trying to get the right result. Certainly, testimony is one of the least interesting ways to teach someone about a case -- perhaps allowing jurors to ask more questions would help. One could also rely more on documents -- getting transcripts of detailed testimony and/or expert reports to the juries so they can look at the underlying materials.

Hope that helps

and Pat

Who said that you were

a showman who confused the jury

?[/quote]