CLPEX Chat Board

Steve Everist wrote:And what of the situation as Pat Warrick describes it? What about multiple conclusions with one bum ident?

I would say that if there are a significant amount of these cases that make it to court, a BV strategy should be developed here also.

Steve Everist wrote: Blind verification tests the reliability (consistency) of a conclusion but not the validity (accuracy) of the conclusion.

I think the cruxes of my posts are from the difficulties I’m having with your definition. By arguing that that the rest of the scientific world “may or may not” be increasing their accuracy by blinding is silly. Those other scientists are no doubt increasing the accuracy of their results. That is why they do it.

That type of denial is almost as fatuous as observing three or four examiners all work in concert misidentifying and mis-verifying a fingerprint, only to later claim confirmation bias was not a contributing factor.

As was mentioned earlier, I too believe there are some strong candidates for membership to the flat earth society on this site.

Strict Scrutiny wrote:

Steve Everist wrote: Blind verification tests the reliability (consistency) of a conclusion but not the validity (accuracy) of the conclusion.

I think the cruxes of my posts are from the difficulties I’m having with your definition. By arguing that that the rest of the scientific world “may or may not” be increasing their accuracy by blinding is silly. Those other scientists are no doubt increasing the accuracy of their results. That is why they do it.

First you'll have to realize that what you are calling "my definition" came from Michele's dictionary. Whether or not you get it is up to you. You may very well get it and be deciding to be argumentative with me or Michele. I've seen it before.

That type of denial is almost as fatuous as observing three or four examiners all work in concert misidentifying and mis-verifying a fingerprint, only to later claim confirmation bias was not a contributing factor.

I think what's fatuous is your idea that the reason that scientists participate in blind testing is purely to improve accuracy and your denial that it's primary reason is to reduce bias. An improvement in accuracy could come as a result, but that's secondary to reducing bias. But like I said, you could just be being argumentative.

As was mentioned earlier, I too believe there are some strong candidates for membership to the flat earth society on this site.

...more "anonymous" personal shots akin of those having been launched in McKie threads. Try and stay on topic.

sandra wiese wrote:I am sure I don't need to invite you to join the Flat Earth Society cause you are likely a charter member. For the rest of us, I hope this serves as a reasonable and hopefully humorous example as to why bias does not necessarily have any impact on accuracy.

The comment originated in this thread. It had nothing to do with the McKie postings. Please feel free to avail yourself of your administrative authority and delete any of my posts that violate your standards of civility.

SWGFAST provides us with three types of conclusions: Individulization, Exclusion and Inconclusive.

opop wrote:Just FYI, the FBI verifies all single conclusions in a case, idents, non-idents or inconclusives. That way the verifier has no idea of what they're verifying.

So if I have a case with 10 latent prints and one has been individulized to a subject, one is inconclusive to a second subject, and eight have been excluded from subject one and two. You have three types of conclusions in one case.

Opop under the FBI SOP would that case go to BV or be exempt since there are three conclusions in one case.

No trap, just trying to understand the 1 id to BV idea.

The problem I see with this scenario is that eyewitness testimony is not able to be tested or falsified through additional testing.

To some extent the conclusion ("he did it") can be tested/falsified by forensic evidence (when available), a valid confession, and/or those rare cases where there's a video of the incident. But you are correct that in most cases, the information available to the print examiner (the latent and exemplar) are available for re-testing. The problem there is that it is very hard to set up a truly blind re-test -- the second person generally will know that some conclusion has been drawn about the latent and exemplar, which leads to the potential for an error based on suggestion or expectation of a match. For the private-practice examiner it is very hard to set up a bias-free test -- the examiner knows he or she has been hired by the defense, may have to go to the agency's lab to view the original latent, and may become exposed to other info in the course of the consultation. In none of the 22ish known mis-IDs did the defense examiner catch the error. That deeply troubles me.

I see why the trial courts see a link. The terminology of suggestion, confirmation bias, and blind testing is more familiar to them thru the eyewitness ID cases, which are far more common than a fingerprint challenge. But it shouldn't be difficult for a prosecutor to point out where that analogy falters and where it has some utility.

If I understand the postings so far.

The FBI is not currently doing “Blind Verification” but are “Verifying Blindly”.

True “Blind Verification” is done during the “Comparison” phase.

True “Blind Verification” is not a form of Peer Review.

In looking at the “Hypothesis Models” that have been proposed in the literature which are:

1. The Wertheim Five Step Version
A. Examination of the unknown (study latent print)
B. Formulation of a Hypothesis (study the know—hypothesis defaults to subject)
C. Experimentation (comparison)
D. Formation of Tentative Conclusion
E. Testing the Conclusion

2. The Triplett—Cooney Version
Hypothesis Testing (default to the subject)
A. Collect Data (Analysis)
B. Testing (Comparison)
C. Conclusion (Evaluation)
D. Peer Review

For true “Blind Verification” to be done it must be done during the Comparison phase (Experimentation under Wertheim version and Testing under Triplett/Cooney version).

I would think that it could not be done at the very beginning of the comparison phase, and at the end of the comparison phase one can move from comparison to the conclusion part very quickly. So true “Blind Verification” must be done after the beginning of comparison and before the end of comparison.

How is this done. Could someone provide a model or written description of how true “Blind Verification” is performed especially when I have not reached a conclusion at the point that Blind Verification must be performed.

If it can only be performed during Comparison, then Ms Steele is correct and that any defense expert is going to be bias walking in the door. Of course we could set money aside to hire defense experts to review all types of cases before the conclusions are reached and the reports are even filed. This way they would not know whether it was Individulized, Excluded, or Inconclusive. The original examiner would not even know yet what the conclusion is.

I would like to see a model of true “Blind Verification” performed before the conclusion is reached or perhaps the definition needs to be changed.

Strict Scrutiny,

I see your point with regard to the definition and I agree that it needs to be changed. What do you think about this idea (I’m writing this as statements but I’m asking if others would agree with these statements)? Accuracy has to do with something being reliable and valid. Usually things are considered reliable if they can be reproduced and usually things are valid if they have logical justification behind them. Then we could say that

Accuracy = reproducibility + justification

Confirmation Bias can be a factor in the verification process, and BV is a way to alleviate this bias (it is part of the reproducibility aspect of the equation). Since accuracy is dependent on reproducibility, then accuracy is also dependent BV (but not the sole factor). How much BV effects accuracy is unknown.

This makes it all look very simple but there’s a lot of factors that effect both reproducibility and justification. As an example, logical justification is dependent on whether or not accepted principles are correct and what the standards are. Whether characteristics are objectively based or subjectively based is a factor. I’m sure we can add a lot of elements but the equation then becomes:

Accuracy = reproducibility (V or BV or Double BV) + justification (objectivity, accepted principles, standards, etc)

My mind wandered quite a bit in an attempt to change the definition but do you think it would be more reasonable to say:

Blind verification tests the reliability (consistency) of a conclusion but not the validity (justification) of the conclusion.

Lisa,

You said that in none of the 22ish known mis-IDs did the defense examiner catch the error.

Is this really true? How many of these cases had defense examiners? I don’t know the answer but I do know that in several of these cases the suspect was released or not charged due to the prints being re-examined or because of defense examiners. Wasn’t the Ricky Jackson erroneous ID found by defense experts and then the erroneous ID was confirmed by the FBI? Even the first erroneous ID recorded (William Stevens) was caught by the defense experts. In these situations I don’t think reworking the case blindly caught the errors. I don’t know the real answer but it’s possible that people who made the errors were not well trained, didn’t have adequate understanding behind what they were doing, or it may have been an ethics issue. Perhaps the examiners who caught the errors had good training, understanding and ethics.

Just a thought.

Since accuracy is dependent on reproducibility, then accuracy is also dependent BV (but not the sole factor). How much BV effects accuracy is unknown.

This makes it all look very simple but there’s a lot of factors that effect both reproducibility and justification.

I agree, I don’t believe the validity, reproducibility, or accuracy of a conclusion can be tied to BV alone, nor can it be tied to a documented comparison alone, but BV can play a role in the validity and accuracy of a conclusion. Peer review (in my opinion) must take a broader and more holistic look at the whole ACE-V process. In my opinion peer review should not be happening at the verification stage, because at minimum the verification should be semi-blind (biasing information should be limited, but not necessarily eliminated). After verification peer review can happen and the reviewer can have access to all the information they desire.

do you think it would be more reasonable to say:

Blind verification tests the reliability (consistency) of a conclusion but not the validity (justification) of the conclusion.

I think that wording is more reasonable but I still do not think it does the trick. This is why: Certain types of comparisons should trigger fully blind verification (like single ident cases), if all the procedures were followed, thoroughly, and bias was removed by BV then that (along with examining the print itself) would bolster a peer reviewer’s opinion that the original ACE-V process produced a valid conclusion. So in that sense BV may not be soley responsible for the validity of a conclusion, but still be tied to it.

I think you’re saying that bias can affect the validity of a conclusion as well and if this is what you’re saying, I’d agree.

I also think that BV doesn’t test the validity in the sense that it looks to see if the logic was correct because I don’t believe that most conclusions come with documented justification behind them (am I wrong?). If (and this is a big IF) documentation did exist from both examiners then the BV process may reproduce the same justification and in a sense is a test also. It would be up to peer reviewer to determine if the conclusions and the justification were the same.

In this case, what BV is testing would change depending on if there was justification behind the conclusions or not. True??

Michele Triplett wrote:I also think that BV doesn’t test the validity in the sense that it looks to see if the logic was correct because I don’t believe that most conclusions come with documented justification behind them (am I wrong?).

You are right that most conclusions don't come with documented justification (I have a feeling you already knew that), and that BV does not directly apply any test to the reasoning of the examiner (it can be [in my opinion] however an indication that sound reasoning was used).

Michele Triplett wrote:If (and this is a big IF) documentation did exist from both examiners then the BV process may reproduce the same justification and in a sense is a test also. It would be up to peer reviewer to determine if the conclusions and the justification were the same.

In this case, what BV is testing would change depending on if there was justification behind the conclusions or not. True??

I think for the process to be a success (if the documentation of the exam were being used) then the examiner and the verifyer would both need to match the print (independent of each other and free of bias), and then when peer review happens, their documentation would not necessarily have to exactly match, just show that they used valid methods.

Its a lot to consider in one shot but those are my current thoughts.

Definition from SWGFAST

4.2.4 Verification
Verification is the confirmation of an examiner’s conclusion by another competent examiner.

Blind verification is the confirmation of an examiner’s conclusion by
another competent examiner who has no expectation or knowledge of
the prior conclusion.

Definition from Michele

Blind Verification
Blind verification is a method of testing a hypothesis. This method is implemented by limiting the information given to practitioners analyzing data, such as previous conclusions. The intent behind blind verification is to decrease the amount of bias involved in an analysis. Blind verification tests the reliability (consistency) of a conclusion but not the validity (accuracy) of the conclusion. This testing method is especially useful when analyzing inherently subjective data.

Wikipedia has no page avaialbe for the definition.

Here is the Wiki treatment of the subject:

http://en.wikipedia.org/wiki/Double-blind

Thanks Scrutiny

I stand corrected it is under Double Blind.

I knew that BV was being performed in drug trials, psychological and social tests. I do not think it was being used in chemistry, biology, mathmatics, geology, botany, engneering, or the other applied or pure sciences.

I had read before that there was discussion about trying to implement it for photo lineups which would be easy to do, but to my knowledge I know of no agencies currently doing it. I do not doubt there are some, I just do not know of any. I do not know of any other forensic discipline that is doing it.

I suppose since the FBI has instituted BV and others are thinking about it the Science of Fingerprints will have to join the ranks of photo lineups as the two forensic disciplines utilizing Blind Verification.

I suppose we need to adopt it since it will be difficult to answer in court that the FBI is conducting BV and how come my agency is not.

It is coming and there is not a damn thing I can do to stop it. Have to take it on the chin and keep plugging.

Thanks for correcting my post Scrutiny. I have printed it out and put it in my science binder.

Charles Parker wrote: I suppose we need to adopt it since it will be difficult to answer in court that the FBI is conducting BV and how come my agency is not.

There's something about this statement that makes me uneasy.