CLPEX Chat Board

At a recent local IAI conference Stephen Meagher stated that 90% of bum idents are from cases in which there was only 1 id “made” to the suspect.

My LPE colleagues and I were brainstorming verification procedures and we’ve developed an idea. Though this idea really doesn’t address blind verification as a whole, we think it might help somewhat.

Since the majority of our cases usually contain more that 1 ident to a suspect, verification would remain as it is now (marking the ident, then having the verifier review it). I think we can all agree that blind verification on every case is nearly impossible.

However, in cases where there is only 1 ident to one suspect (and we really don’t have that many), we are considering not making any ident notations and submitting it as is to the verifier. Only after the verifier completes his/her analysis would we notate which print was an ident.

We realize that even this is not true blind verification, because a verifier getting a case without notations would realize that there’s only one ident in the case.

But we are thinking that this may reduce our chances of our agency making a bum, while helping us try to establish some blind verification in our procedures.

What are your suggestions/comments/ideas?

Becky,

Before I get started, I think I view “Blind Verification” differently than is often discussed. Here’s the definition as per Michele’s Fingerprint Dictionary ( http://www.fprints.nwlean.net/ under the "B" link):

Blind Verification
Blind verification is a method of testing a hypothesis. This method is implemented by limiting the information given to practitioners analyzing data, such as previous conclusions. The intent behind blind verification is to decrease the amount of bias involved in an analysis. Blind verification tests the reliability (consistency) of a conclusion but not the validity (accuracy) of the conclusion. This testing method is especially useful when analyzing inherently subjective data.

So as a method of testing a hypothesis, it actually occurs during the Comparison phase, as opposed to being part of the Verification phase. But it is so often referred to as being a part of Verification, most likely due to the word “verification” being a part of the term.

I don’t necessarily see there being a problem having the second examiner looking at a case not knowing the results as some sort of quality assurance method. But I don’t see this as technically being “blind verification” as per the definition, but more verification done blindly.

If you only do this with single-ident cases, it would be pretty obvious what was going on. Now if you were to mix in some single-print, non-ident cases done the same way – so as not to have the expectation of an ID – then it would be “more” blind.

Hi Steve,

Thank you for your response. I agree with your idea about adding a few non-ident cases into the verification stack, so as not to create any expectations.

Thanks also for your definition of verification. I appreciate your feedback!

I think that all you need to do in order to have blind verification is to establish doubt in the verifiers mind that the initial examiner actually identified the prints. Once you establish that seed of doubt in their minds the potential confirmation bias issues go away, because the verifier isn't going to automically assume that you have identified these prints.

Regardless of whether we need to do it or not, I think that people try to make it far more complicated than it really needs to be. Your system sounds like it may work well, but also has the potential to lead the verifier to believe that every set of prints that you give him/her that isn't marked is a single ident case. IMO, in order to establish blind verification you need to give the verifier non ID's on a random basis.

Andrew said, “I think that people try to make it far more complicated than it really needs to be”. If you think it’s being overcomplicated, wait until you read this

The premise is that 90% of erroneous ID’s are from single Identifications to the suspect. From this we are developing way to protect against this happening. This makes sense but before we start developing protection methods maybe we should determine if the premise is even accurate. I’d like to know how this information was arrived at.

I can look at the published list of erroneous ID’s and the premise seems to hold true but I’m also open to looking at other possibilities.

For instance, even if all erroneous ID’s have the ‘single ID’ in common, there may be another common element - that all the ID’s had low quality and quantity. Maybe this is where we need to implement QA measures?

Another thought, is it possible that examiners have a different standard for individualization when it comes to the victim? Perhaps even the verifier has a different level of scrutiny in these ID’s. This could happen because the victim usually isn’t being prosecuted for anything so we don’t need to be as cautious. It might be feasible that a latent print is consistent with the known prints but not sufficient for individualization under normal standards because the consequences aren’t the same. It’s possible that this lower standard is justified, considering that the victims presences has already been established, or it’s possible that erroneous ID’s are made to the victim and just not recognized as errors?

If this is possible then the true error rate for ID’s may be higher than we think it is.

The same hypothesis could be generalized to include suspect ID’s. Suppose we established that a suspect was in vehicle (through an ID to the rearview mirror with plenty of clarity and quantity). Once we’ve established that the suspect was indeed in the vehicle, is it possible that other ID’s are made that may border on sufficiency but nobody questions the ID because it really doesn’t matter as far as guilt is concerned? (I can see a good scientific argument that once presence is established then the standard doesn’t need to be as high, but that’s a subject for another post)

If this is true, then the true error rate may be even higher but these erroneous ID’s aren’t publicized.

Another possibility is that some departments don’t compare all latents to all people – that there are some that stop looking after suspect’s have been ID’d. Thus there could be more potential erroneous ID’s along with the potential for more correct ID’s

I think it’s fair to say that “all errors that have held serious consequences” have come from single ID cases. But are we only concerned with errors that hold serious consequences or should we be concerned with all errors? If we’re concerned with all errors, does this include erroneous exclusions or do we not consider these errors as being serious? Then, because they’re not serious errors, we really don’t need to consider them as errors at all??

Besides the original premise, I’ve also heard another one that’s similar. I’ve heard that the majority of errors are from single ID’s where the AFIS generated the suspect. Is this true? If so, then maybe this is where we need to implement extra quality assurance measures. Looking at this 2nd premise skeptically (I’ve mentioned so many items that this may actually be the 4th or 5th premise), I can see that it may be true in certain circumstances (when the quality and quantity are low). But when the quality and quantity are high, I doubt that erroneous conclusions are a problem. This leads to another premise……that the majority of erroneous ID’s are in single ID cases where AFIS has generated the candidate AND the quality and quantity are low. If we go back to the published lists of erroneous ID’s, it doesn’t seem like the majority of them were produced from AFIS candidates (but many of the brief explanations don’t really indicate how the person became a suspect). AFIS may be a biasing factor but getting a name as a suspect may be just as biasing. This would lead to yet another premise…..that the majority of erroneous ID’s are in single ID cases where the quality and quantity are low (where the suspect is either AFIS generated or not). So this line of reasoning actually leads back to my original thought.

If this is the case then this is when we need to implement QA measures, not in all single ID cases but in cases where the quality and quantity is low. My point is that before we try to implement QA measures, we really need to have a better idea of when erroneous ID’s occur.

One more issue and then I’ll stop (I promise). In many of the erroneous ID cases, there was more than one verifier (showing that in cases where the Q and Q are low, even addition verification doesn’t always help). Would this mean that even if we implement blind verification then we really should be using multiple blind verifiers? Blind verification might take bias out of the equation but does that insure quality results?? I really don't even know if that's been clearly established.

Are people making more out of this than is necessary or do we really need to look at the issues a little closer before trying to find solutions?

Andrew Schriever wrote: Regardless of whether we need to do it or not, I think that people try to make it far more complicated than it really needs to be.

Yes, I think so too. Blind testing is a concept that is not new and does not have to be complicated. Here is my definition of blind verification :

To shield the verifyer from information that may bias his/her decision making during the analysis, comparison, or evaluation stages of the examination process.

I don't see how a conclusion that has been duplicated free of bias is not also more accurate. I'm scratching my head on that one.

I think the FBI hit the nail on the head when they called for BV on single ident cases. Not only do the statistics seem to support that rationale, but single ident cases just logically seem to pose the greatest risk of allowing mistaken evidence against someone. I have not heard of a case that made it to court with two or more bum idents in it.

Just FYI, the FBI verifies all single conclusions in a case, idents, non-idents or inconclusives. That way the verifier has no idea of what they're verifying.

Thanks for the great comments!

I am interested in learning where the 90% figure came from. Does anyone know if this information was released in a published study?

I think the FBI hit the nail on the head when they called for BV on single ident cases. Not only do the statistics seem to support that rationale, but single ident cases just logically seem to pose the greatest risk of allowing mistaken evidence against someone. I have not heard of a case that made it to court with two or more bum idents in it.

I agree very much with this statement. If this statistic is accurate, then I believe that implementing BV procedures can only serve to prevent bad calls.

Michelle,

Great post. You've raised some great points! I am especially intrigued by your comments about single-ident cases and AFIS. There's just so much to consider.

Thanks.

Strict Scrutny said,

I don't see how a conclusion that has been duplicated free of bias is not also more accurate. I'm scratching my head on that one.

What I meant is that two (or three or four) people independently arriving at the same conclusion is great but justifying how they came to that conclusion may be more valuable.

I'm not advocating one QA measure over another, I just think we need to look at the benefits and limitations of all QA measures and use each of them at the appropriate times.

Strict Scrutiny said,

"I have not heard of a case that made it to court with two or more bum idents in it."

I don't see it as having two or more bum idents in one case. The problem occurs when you have multiple "Good Idents" in the case, then there is a latent with low Quality/Quantity that is close....but you've already ID'd the guy several times so its most likely that this one is his also so the examiner is biased into making THAT ident when possibly they shouldn't. The problems I have about the "Single Ident Phenomenon" proposed by the FBI is that I'm not sure where they are getting those numbers. The important Single IDs are pivotal for a case when that is all there is connecting an investigation to a suspect; those of course are the ones you hear about. But stretching that out to 90% of Bum IDs?

If you think it’s being overcomplicated, wait until you read this

Its either going to have to wait until this afternoon or monday. My brain isn't working that well this morning. 8)

"I don't see how a conclusion that has been duplicated free of bias is not also more accurate. I'm scratching my head on that one."

sorry, but I don't know how to make those cute little boxes showing the quote, but this is the only part of the post I am responding to because the rest of you would burn me at the stake for how I feel about the rest of the issue...

Imagine, if you will, it is 1392 (a number I just pulled out of my, uh, ahem). In what is now the United States there is a man who KNOWS that if he walks as far as he can see (what is in actuality the horizon), he will fall off the edge of the earth. He has no doubts about this whatsoever. No one else has told him this. He came to his own independent conclusion based on observable 'facts'. He would bet his life on it he is so sure and therefore he rightfully stays as far away from the edge as possible. (Luckily he finds this oddly easy to do.)

Same time period, but another guy in China and another guy in Africa and another in Europe. All come to the same conclusion. No outside influence. No internet to verify facts and no chat room to see how others feel about it. But they know what they know.

All came to their independent conclusions free of bias.

If you think these theoretical gentlemen are correct, then I am sure I don't need to invite you to join the Flat Earth Society cause you are likely a charter member. For the rest of us, I hope this serves as a reasonable and hopefully humorous example as to why bias does not necessarily have any impact on accuracy.

and if that doesn't work look at it this way: Does that fact that you have verified and had verified say, oh, THOUSANDS of prints in the past where you knew how the suspect was developed (Caught the guy standing over her, bloody knife in hand) make your idents and exclusions any less accurate?

Sandra[/quote]

Imagine, if you will, it is 1392 (a number I just pulled out of my, uh, ahem). In what is now the United States there is a man who KNOWS that if he walks as far as he can see (what is in actuality the horizon), he will fall off the edge of the earth. He has no doubts about this whatsoever. No one else has told him this. He came to his own independent conclusion based on observable 'facts'. He would bet his life on it he is so sure and therefore he rightfully stays as far away from the edge as possible. (Luckily he finds this oddly easy to do.)

Same time period, but another guy in China and another guy in Africa and another in Europe. All come to the same conclusion. No outside influence. No internet to verify facts and no chat room to see how others feel about it. But they know what they know.

We might be comparing apples and oranges here. You see the four gentlemen you spoke of did no hypothesis testing, which is what blind verification (and this thread) is all about—Therefore, they were not practicing our concept of science and are probably irrelevant to this discussion. Secondly I would think that if I could travel to the year 1392 (somewhere in my brain a strange poem is echoing) and interview the gentleman from the Americas I might find that he would be quite strident in his belief that he came to the conclusion by himself, he might swear up and down no one has influenced his belief on the subject and even take umbrage that I would suggest such a thing, but I would guess if I interviewed a few of his friends and relatives I would find that the notion of falling off the edge of the earth actually permeates his peer group and his society, and that group had been exchanging information on the subject for some time with other societies and thus developed a common hypothesis.

Here is an example a little closer to home. Let’s say for the sake of argument that once in about a thousand cases an innocent person gets put behind bars for erroneous eyewitness identification. The courts are somewhat troubled by this so they look into the problem and they find something in the process just doesn’t quite jibe, scientifically speaking. They find that the detective could possibly be indicating to the witness which suspect to pick from the photo lineup. So to remove biasing information, the courts order that all photomontages be done double blind—that is, the tester and the witness both have not been given the correct answer.

My question is this: Do you believe the single blind or the double blind tests will be more accurate if one thousand of the above mentioned photomontage cases are studied with the correct answers? Yes, accuracy is always the goal of blinding.

Strict Scrutiny wrote: Here is an example a little closer to home. Let’s say for the sake of argument that once in about a thousand cases an innocent person gets put behind bars for erroneous eyewitness identification. The courts are somewhat troubled by this so they look into the problem and they find something in the process just doesn’t quite jibe, scientifically speaking. They find that the detective could possibly be indicating to the witness which suspect to pick from the photo lineup. So to remove biasing information, the courts order that all photomontages be done double blind—that is, the tester and the witness both have not been given the correct answer.

The problem I see with this scenario is that eyewitness testimony is not able to be tested or falsified through additional testing. Additional experts can come in after the fact and test the conclusions of a latent print comparison. This can't happen with eyewitness testimony. The conclusions of the witness cannot be tested. The Habers have tried to link the two together, which I'm sure you're aware of. The problem is that eyewitness testimony fails the analogy with print comparison in that it can't be tested - which is an important part of being scientific. The defense can't hire a private-practice "expert eyewitness" to relive the experience and then go through a photolineup. However they can hire a private-practice examiner to go through a latent print case.

My question is this: Do you believe the single blind or the double blind tests will be more accurate if one thousand of the above mentioned photomontage cases are studied with the correct answers? Yes, accuracy is always the goal of blinding.

They may or may not be more accurate, however a bias that the detective could be introducing would be eliminated. In fact, the accuracy level as a result of bias incorporated by a detective that has outside investigative knowledge, could actually be higher than the accuracy level of the unbiased eyewitness. But that would also bring in the idea of getting the "correct" answer as the result of an incorrect process. Also, that would be difficult to incorporate into a blind test for comparison. I think the goal of blind testing is more often to remove the potential for bias. But this is something that happens during the testing of the hypothesis, not after the conclusion has been made.

And what of the situation as Pat Warrick describes it? What about multiple conclusions with one bum ident?