CLPEX Chat Board

Recently there was some interesting discussion about levels of certainty that latent print examiners are comfortable with.

It seems appropriate now to follow up with a quick poll. Perhaps its time for a change, or maybe our previous standards are still working well.

>95% of what

From what I have read this is where percentages falls down. Is it 95% of a total print, because that would give a jury the impression of a very good match but if its 95% of a partial latent print (say 33%) of a full finger print then that is in reality 31% of a complete match. That's giving the jury a false impression.

Then again if an expert would say it is highly likely this is a match then if I were on a jury I would think that there was room for error.

When I testify to an identification, I am 100% certain. But as I've stated in another post, that is personal belief, not scientific certainty. If the researchers working on probability models come up with something that is scientifically valid, then I will add that to my testimony. I would hope I can put a number greater than 95% on it -- I'm not sure I would be too comfortable with something that low!

What level of certainty will produce the fewest errors and best convey our confidence to a jury?

I voted other because I dont really think that the question is relevant (no offense intended).

It doesn't really matter how certain an examiner claims to be in front of the judge and jury, if the examiner made a mistake they made a mistake. Doesn't matter if they say that they are 100% certain or just very likely, if their opinion is wrong its wrong. They are still getting up there and leading the jurors to believe that it is their opinion that the prints match.

I am an open minded person and am always open to new thoughts and ideas, but I personally believe along the same lines as Pat and I dont think that will change. When I testify to a ID, I am testifying on personal opinion and its going to be that I am 100% positive that these 2 prints were deposited by the same individual.

I just dont really see how a probability approach will help the profession. IMO, I dont think its possible to create one that can apply to all examiners everywhere. I think that our conclusions are opinions and I dont see that changing.

Now if research shows that .01% of all fingerprint ID's are incorrect, that is certainly something that can be used in court, but it wont change my stance that I am 100% positive that these particular prints that I am testifying to is a match. I am not going to get up there and say "Well, I am 99.9% sure that these 2 prints were deposited by the same individual". What is the good in being 99.9% incorrect as opposed to 100% correct?

Andrew Schriever

It doesn't really matter how certain an examiner claims to be in front of the judge and jury, if the examiner made a mistake they made a mistake.

I see your point, and I mostly agree. I would just like to offer that there are more ramifications to 100% certainty than courtroom testimony alone. I believe complete certainty is vitally important in our decision making process during the “E” phase of ACE-V, and culls many unreliable prints from circulation. A hypothetical scenario (hmm have you ever seen this?) would be an examiner who wants to call a badly distorted print a match to the suspect in a big case, but in good conscience is not 100% certain (there is a slight tinge of doubt). To call the print a match could do wonders for his career and no doubt aid the prosecution. But 100% certainty is troublesome, and “highly likely” is much more palatable (or even “double-highly-likely-with-whipped-cream-and-a-cherry-on-top”).

That scenario happens often enough to pose a real problem for our profession, especially if we lower our standards to less than 100% certainty.

As a verifier I have personally put the brakes on identifications because my training told me that if I have any doubt whatsoever that the print “could” match someone else, that I cannot call it a match. I believe anything less than this is diluting our standards because as you said “it really doesn’t matter” if one says it’s a match or if she says it’s “probably” a match, it is still wrong if it is not “actually” a match. Yet therein lies the rub. By saying something is probably a match, without reliable statistics to guide a probable conclusion and peer review, the door is left open for all kinds of shenanigans.

Please look at the Mayfield error and try to imagine you have no prior knowlege about it and decide which of these scenarios is easiest to justify: (1) "It is probably his print". (2) "I believe it is Mayfield's print to a reasonable degree of scientific certainty" (3) "I absolutely believe it is his print and I am willing to stake my career on that statement if I am wrong." Now apply that logic to all the erroneous ID's that have yet to happen, or yet to be re-examined.

I think absolute certainty is the standard that allows for the most scrutiny for examiners, verifiers, administrators, etc.

Good statistics on examiner error rates are a number of years away. Reliable statistics on the probabilities of FRS formations being duplicated in two prints are many more years away. Both will help this profession. But to say right now that it’s OK to be less than 100% certain without good reliable statistics to fill the void is not real science, as some would argue, it is simply beating a drum with no other accompanying instruments.

I think 100% certainty is the best quality assurance device we have currently, and to remove or dilute it at this point in time just opens the door to a lot of “probably” prints masquerading as reliable evidence.

Have you ever seen a print that was too thin to verify because you held yourself to the standard of absolute certainty? I have, and I know that some examiners would dearly love to sidestep that conservatism. I believe that conservatism makes for a more fair criminal justice system for the accused. I certainly support developing statistical models but that does not mean that we are doing anything wrong by saying we must be 100% certain prior to pulling the trigger at this point in time.

Strict Scrutiny wrote:

...(3) "I absolutely believe it is his print and I am willing to stake my career on that statement if I am wrong." Now apply that logic to all the erroneous ID's that have yet to happen, or yet to be re-examined.

I think absolute certainty is the standard that allows for the most scrutiny for examiners, verifiers, administrators, etc.

Good statistics on examiner error rates are a number of years away. Reliable statistics on the probabilities of FRS formations being duplicated in two prints are many more years away. Both will help this profession. But to say right now that it’s OK to be less than 100% certain without good reliable statistics to fill the void is not real science, as some would argue, it is simply beating a drum with no other accompanying instruments.

I think 100% certainty is the best quality assurance device we have currently, and to remove or dilute it at this point in time just opens the door to a lot of “probably” prints masquerading as reliable evidence.

Have you ever seen a print that was too thin to verify because you held yourself to the standard of absolute certainty? I have, and I know that some examiners would dearly love to sidestep that conservatism. I believe that conservatism makes for a more fair criminal justice system for the accused. I certainly support developing statistical models but that does not mean that we are doing anything wrong by saying we must be 100% certain prior to pulling the trigger at this point in time.

Having all this in mind I can conclude then that we are testifying to our opinion, which should be supported by the opinion of at least one other qualified examiner. (if we are applying ACE-V) Hence the reason why I voted "other". How certain am I about my opinion? 100% certain that my opinion is correct if I conduct every comparison as if my career depended on it.

As Lisa Steele mentioned in another post, there is no evidence to show your level of certainty is correlated to how accurate a conclusion is.

When I said my answer to how confident I am is “to a reasonable degree of scientific certainty” I didn’t mean that my conclusion is based on my level of certainty. I personally don’t believe that absolute certainty is or should be our standard for making an identification. I think ‘justifiable’ ‘demonstratable’ ‘reproducible’ and ‘based on sufficient quality and quantity’ are just some of the standards we use. Since sufficiency is subjective, conclusions that have consensus would appear to have more weight behind them. I agree that conservatism is one of the best ways to ensure accuracy.

When I come to a conclusion, I have 100% confidence in the principles and methods I’ve used but I’m constantly skeptical that perhaps there was something I wasn’t seeing or thinking of. I think being 100% confident in my own abilities would be arrogant and may even lead to errors because I wouldn’t be open to objective criticisms.

I don’t have doubts about my conclusion, I’m just constantly skeptical and cautious about my own abilities. I think doubt comes from knowing that you don’t have enough information to back up the claim of an identification. Complete peer review (from people with independent thoughts) may reveal that an examiner doesn’t have sufficient quality or quantity to come to the conclusion, or maybe it would reveal that there’s not enough data to gain support from other experts. Maybe peer review would reveal that there is enough information to come to a conclusion but since the quality and quantity are low, maybe more documentation is needed to support the ID.

You mentioned the Mayfield case…One of the reports about that case stated that one of the problems was overconfidence, although not the primary factor. The lack of rigorous application of principles was the ultimate cause (this is from the OIG report). You proposed 3 scenarios for this case, but lets look at a 4th, “I believe it is Mayfield's print to a reasonable degree of scientific certainty because of the following reasons (then justify how the conclusion was arrived at). The principles, method, and data I used are open for peer review and nobody has found any reason to disagree with this”. This statement couldn’t be made because Spain didn’t accept the reasoning the examiners at the FBI were giving (thus disagreement did exist). Another reason the examiners couldn’t use the 4th scenario was because it’s been determined (I think it was in the Stacey report) that in their office (the FBI), to disagree with a conclusion wasn’t expected (which tells me that the FBI either didn’t use or didn’t understand the value of independent review as a way of assuring accurate conclusions).

I may have gotten way off course here so I’ll try to get back on track, when we talk about quality assurance measures I think we can talk about our own personal measures (so we don’t make an erroneous ID) or QA measures that protect wrong conclusions from being reported out (individual QA measures vs system QA measures). As an individual QA measure, 100% certainty may be valuable. As a system (or method) QA measure, I think one of the best quality assurance measures is independent peer review. It’s actually not just one QA measure, it encompasses several things all in one. I also recognize that “What is independent peer review” is probably under debate also.

Michele, you have touched on so many things that I personally believe is of paramount importance. I think you are attempting to define professional philosophy, and while I do believe there are some who have given this considerable thought and have expressed themselves well, I think the process is constantly ongoing. I hardily support Pat's expression that certainty is a personal belief, and I am more inclined to state my personal conviction in similar terms. But behind that is what I believe absolute certainty to be, and I am not sure that is the same for Pat or for you. I recall having a discussion with Steve McKasson, an extremely insightful person on the various aspects of uniqueness manifestations, on the nuances of questioned document conclusions and asked what was the distinction he made between a positive and probable determination. He replied that his positive was an absolute certainty, as he understood it, that he was right, and probably simply meant he was probably right, not that the suspect probably wrote the unknown.

I don't believe this is just variations in semantics but addresses the heart of an individual's professional philosophy. I am far less concerned about demonstrating a methodology has been tested by others than in the ways I have tested the methodology. I'm not as concerned with certainty being absolute or some degree that might fall within scientific than in my means of satisfying that certainty. You said you have no doubts about the conclusions you provide, yet remain skeptical about your own abilities, which I interpret more as an effort to pursue the null hypothesis as well as the hypothesis during the examination. That, to me, is part of a developing professional philosophy that is admirable. I say that because I think the process of setting our own standards and evaluating how these mesh with others in the profession is a never ending process. I see the same thing in the questions posed by Charlie Parker and the expressions of others who contribute to this exchange.

Many years ago, I was told by a wise person that expert opinion was a privilege granted by adequate demonstration of training and experience, but it should never be thought of as a right. Privilege carries ongoing obligations, and part of that obligation is to provide more than a certain opinion. Bob Stacey may chose to describe the abuse of privilege as overconfidence, but arrogance is more accurate, for a fine line separates the two. Overconfidence or failure to properly execute ACE-V are simplistic explanations of a most complex matter, but the failure to remain skeptical is evident. A trip to Madrid was not to analyze, compare and evaluate in the presence of healthy skepticism, but to demonstrate that absolute certainty was foolishly final.

As Lisa Steele mentioned in another post, there is no evidence to show your level of certainty is correlated to how accurate a conclusion is.

Perhaps you should reread that post. I think Ms. Steele was referring to eyewitness testimony and she was cautioning that judges might confuse that with our expert testimony. Certainly our confidence level does correlate to our degree of accuracy.

When I said my answer to how confident I am is “to a reasonable degree of scientific certainty” I didn’t mean that my conclusion is based on my level of certainty.

Having a degree of certainty about the claims we make as forensic professionals is a primary tenet of performing the job of latent print examiner. Our conclusions (even by definition) are built on how certain we are in our opinion.

I don’t have doubts about my conclusion, I’m just constantly skeptical and cautious about my own abilities.

I think this is compatible with being educated in science, trained in formation of FRS, experienced in the examination of fingerprints, and held to a personal standard of absolute certainty.

I think doubt comes from knowing that you don’t have enough information to back up the claim of an identification.

No disagreement here. If you have doubt, you cannot be certain and you should shuck the claim of a match before it adversely affects someone. If you have thoroughly explored the entire print and are satisfied that it matches the exemplar, without any tinge of doubt, then absolute certainty is an valid expression.

You proposed 3 scenarios for this case, but lets look at a 4th, “I believe it is Mayfield's print to a reasonable degree of scientific certainty because of the following reasons (then justify how the conclusion was arrived at). The principles, method, and data I used are open for peer review and nobody has found any reason to disagree with this”.

This is a bit scary. If Mayfield were to be tried it is likely the prosecution would have argued your response verbatim. Regardless of the protestations of Spanish examiners, or any dissenters in the FBI for that matter. In the game of life dissenters are regularly stifled. The only effect of releasing someone from absolute certainty is a release from liability. The FBI said it was Mayfield’s print with absolute certainty. The consequence for that error was a couple of million dollars and a black eye. They could have been just as overconfident, misleading, and arrogant with the wording “to a reasonable degree of scientific certainty”. Perhaps not as liable under the law however.

As an individual QA measure, 100% certainty may be valuable.

I personally don’t believe that absolute certainty is or should be our standard for making an identification.

I have to say I am baffled at these two statements. We either are transparent, logical, and willing to testify how we actually do our job and perform QA. Or we run people in circles. Which is it?

Dave,

I'm trying to verify a thought I'm having but it may take me a little time. When I find what I'm looking for, I'll get back to you about this.

Scrutiny,

Certainly our confidence level does correlate to our degree of accuracy

I respectively disagree. I remember Ken Moses talking about the Mayfield print and saying he was 100% confident and 100% wrong. Is that the correlation you're referring to?

Having a degree of certainty about the claims we make as forensic professionals is a primary tenant of performing the job of latent print examiner.

If this is a primary tenant, where is this in writing? Surely it's in writing if it's a primary tenant.

Sorry to baffle you with what looks like conflicting statements, let me clarify. I think we all have personal quality assurance measures. These may be scientific in nature or might just be based on preference we have or even maybe semantics, it really doesn't matter since they're personal QA measures. If examiners want to have a personal QA measure of 100% confidence, that's fine with me. It may be valuable in some respect but that isn't a personal QA measure of mine. Personally, I don't know how to distingish between when I'm 98%, 99%, or 100% certain, so to use this number doesn't mean anything to me.

The other statement I made:

I personally don’t believe that absolute certainty is or should be our standard for making an identification.

I wasn't referring to personal QA measures, I was referring to industry QA measures. I don't believe this is or should be an industry standard.

Strict Scrutiny wrote:

As Lisa Steele mentioned in another post, there is no evidence to show your level of certainty is correlated to how accurate a conclusion is.

Perhaps you should reread that post. I think Ms. Steele was referring to eyewitness testimony and she was cautioning that judges might confuse that with our expert testimony. Certainly our confidence level does correlate to our degree of accuracy..

I was talking about two concepts: (1) how a judge or attorney might react to testimony about 100% accuracy in light of challenges to eyewitnesses who say they are "100%" sure of an ID; and (2) that confidence in any field can be manipulated by internal and external factors.

The big one is suggestion. This comes up in the eyewitness ID area when (a) a suggestive procedure artificially inflates the witness' confidence at the time of the ID (badly designed photo array, or subconscious feedback from administrator), or (b) immediate post-ID feedback ("Good, you identified the suspect") artificially inflates the witness' perception of his/her confidence moments before, or (c) the passage of time and the loomng trial date artificially inflate the witness' confidence. If you go to Saul Kassin's web page, you should find some of his papers on confidence and feedback.

This, I fear, can happen to an examiner in the same way it happens to the eyewitness. Assume an examiner has looked at a difficult match and is trying to figure out whether he or she is 100% certain. Co-worker tells them that the news just reported that the suspect confessed. Cognitive psych would suggest that the expert's confidence is going to be artificially inflated by that information. It may cause the examiner to feel more certain at that point and, perversely, to recall being more certain earlier. Studies in the eyewitness ID area show that suggestion not only affects the witness' confidence, but his/her memory for things like lighting, distance, and event duration. (The mind is a weird thing!)

The only effect of releasing someone from absolute certainty is a release from liability.

I think it depends a bit on how the profession views a mis-ID. If one is staking one's career on being correct, regardless of whether one uses the phrase "100% confident" or "to a reasonable scientific certainty", then there doesn't seem to be a practical difference between the phrases.

One of the arguments against phrases like "100% certain" in court is that it is a form of "vouching" testimony. Normally, a witness isn't allowed to say that another witness is telling the truth, or that they believe another witness. In effect, saying I'm 100% certain is vouching for your own credibility.

Perhaps not as liable under the law however.

I'm not a torts lawyer, but I suspect that the difference is meaningless legally, tho it might have some impact on a jury in terms of damages. The key is whether the ID procedure was so flawed as to give rise to legal liability.

Michele Triplett wrote:Scrutiny,

Certainly our confidence level does correlate to our degree of accuracy

I respectively disagree. I remember Ken Moses talking about the Mayfield print and saying he was 100% confident and 100% wrong. Is that the correlation you're referring to?

This is an anecdote and does not represent the correlation I was explaining.

The correlation I was referring to could be measured as follows: 100 IAI certified latent print examiners each report on 100 identifications that they have 100% confidence in. Next these examiners are given latents of lesser quality and these same examiners each report on 100 comparisons that they have less than absolute certainty on.

I predict that the level of accuracy in the first group will be higher than the level of accuracy in the second group. Hence, there will be a correlation between overall level of confidence and accuracy. I believe this is how our field has performed QA over the years. This type of correlation is quite different from the accuracy of eyewitnesses, as opposed to latent print examiners.

Michele Triplett wrote:

Having a degree of certainty about the claims we make as forensic professionals is a primary tenant of performing the job of latent print examiner.

If this is a primary tenant, where is this in writing? Surely it's in writing if it's a primary tenant.

A good place to start on this would be Ashbaugh's book. Probably the chapter on the individualization process (the evaluative phase of ACE-V). You see it is impossible to say a print has been individualized, that no other finger in the world could have made that mark, and yet you are not certain at the same time. Something has got to give. Either you are confident or you are not.

Sorry to baffle you with what looks like conflicting statements, let me clarify.

Apology accepted

You see it is impossible to say a print has been individualized, that no other finger in the world could have made that mark, and yet you are not certain at the same time.

At the ABFDE Daubert conference in Vegas, there was some discussion about ballistics and the counterpart to that phrase -- matched to that firearm and no other in the world. The ATF agent speaking mentioned that he didn't use the phrase because he hadn't looked at every other firearm in the world -- instead he'd testify that he'd never seen a better match in his career thusfar, and did not expect to in the future, even if he looked every day for the rest of his life. In effect, expressing certaintly in an understandible way without using "100%" certainty or "no other in the world".

This type of correlation is quite different from the accuracy of eyewitnesses, as opposed to latent print examiners.

You may still be missing my point, that confidence is something that can be artificially inflated, particularly by extraneous information or the mere passage of time. A QA test is somewhat different from field work, unless you are building such extraneous information or long passages of time into the question.

Examiner A looks at a print on, say 1/1/2001. If he does not record his certainty at that time, his testimony on, say 1/1/2007 that he is 100% confident may not accurately reflect what he felt 6 years earlier.

Or Examiner A looks at a print of lesser quality at noon. He's not absolutely certain, but wants to look at it again after lunch. At lunch, Examienr B mentions that he heard on the news that there's been co-offender confession in the case. Not only might that effect Examiner A's review after lunch, it may affect his memory of his earlier concerns.

That's the point I'm trying to get to --- confidence is a tricky way to think about accuracy and there may be better ways to express that idea in one's court testimony.

Strict Scrutiny wrote: 100 IAI certified latent print examiners each report on 100 identifications that they have 100% confidence in. Next these examiners are given latents of lesser quality and these same examiners each report on 100 comparisons that they have less than absolute certainty on... I predict that the level of accuracy in the first group will be higher than the level of accuracy in the second group. Hence, there will be a correlation between overall level of confidence and accuracy.

This is what Glenn and I showed in our recent JFI article on error rates. There was one error rate for "highest confidence", which was used to represent 100% confidence as you would require for testimony in court, and another error rate for "less than highest confidence" used only for classroom purposes, where you wouldn't testify to it in court (or needed more time, wanted to consult with someone else, etc.). In short, you have already been shown to be correct in your prediction that accuracy is higher when confidence is higher. It also correlates to the quality of the prints, as you stated... it is the lower quality prints that result in lower confidence, and if the student was forced to answer, then also in lower accuracy.

By the way, this article appeared almost exactly a year ago in the JFI, for those who are interested in looking it up.

-Kasey

I’ve been thinking about this……. and I’ve realized something but I’m writing these ideas without thoroughly thinking them through so they’re not only confusing but it’s a great opportunity for peer review (let me know where my logic isn’t working). I think you’ll need a few cups of coffee to understand my thought process this morning.

Prior to today, I’ve been using the terms confident and certain synonymously (100% confident and 100% certain). I was thinking of both of these as meaning ‘certain’ and I equate this with accuracy or maybe the ability to demonstrate how and why you came to your conclusion (well enough to satisfy others). If I go back and reread prior posts, it looks like other people are using both of these words to mean ‘confident’ and equating it to a belief system. I think this small difference in semantics is where a lot the disagreement comes from.

Kasey’s study showed that when examiners were less confident they were also less accurate, but in a real life scenario if the examiners aren’t confident or certain (able to demonstrate the conclusion) then they should be stating their conclusion as inconclusive. Inconclusive equates to not giving a conclusion and therefore the conclusion can’t be wrong, so the accuracy rate shouldn’t change. If a person does give a conclusion which turns out to be erroneous (when their confidence and/or certainty isn’t 100%), then they gave a conclusion without using the method thoroughly. The accuracy rate goes down because of a lack of thoroughness not directly because of their confidence level (since they didn’t do a thorough job, their certainty level goes down, affecting their confidence level). In this situation I’m thinking that in Kasey’s classes if examiners could have used additional equipment (like digital enhancement) they would have felt like they did a more thorough analysis and their confidence levels would have changed.

Just to simplify this, I’m using the term thoroughness (T) meaning to use principles and methods correctly, CE=certainty (being able to demonstrate a conclusion to the satisfaction of others), CO=confidence (a level of personal belief), and A=accuracy.

If T is high then CE is high and CO is high and A is high. If T is low then CE is low and CO is low and it affect A (making it lower).
It appears that CE and CO work together and then you could assume that there is a correlation between CE, CO and A.

But…. let’s looks at the Mayfield case. It’s been established that they didn’t use appropriate principle and methods as thoroughly as they could have (T was low). But the examiners didn’t realize they weren’t being as thorough as they could have been (this may have been a training issue). They were confident (CO was high). Were they certain? If people are using the words CE and CO synonymously then the examiners may say they were certain but if we use confidence to mean a belief system and certain to mean demonstrateable (can demonstrate how and why they came to their conclusion well enough to satisfy others), then they couldn’t have been certain, only confident. I know, they did satisfy a few people but they didn’t satisfy people with independent thoughts.

This example shows that thoroughness (T) can be low, which means CE is low but CO is high. And the end result is that A suffers and goes down. CO and CE aren’t correlated together in this case.

It looks to me that accuracy is dependent on how well you used and understood different principles and methods. Thoroughness and certainty may be correlated to accuracy but confidence isn’t always correlated to accuracy. You can be very confident but not accurate. It would ‘appear’ that confidence is correlated to accuracy but this may be because our industry has such a low low error rate that it’s hard to separate what influences what. Maybe we need to study more erroneous ID’s to really be able to tell. Another important aspect is that we all need to be using terms in the same way just so that we can better communicate our ideas. (That’s sort of funny, talking about how to communicate better right after I wrote something that is probably not at all understandable- but I tried).