CLPEX Chat Board

Cognitive neuroscientist Itiel Dror finds that analysis of fingerprint data by human examiners can be ruined by unintentional bias. But he offers some relatively simple fixes that can improve the odds of reliable results.

You can read the entire article at:

http://www.miller-mccune.com/legal_affa ... st-up-1312

To Forum Readers:

It was not too long ago when I used to read such articles out of ‘Pulp Journals’ and I would get mad. It seems to me in the past year that I do not get mad anymore I just get amused. I have become desensitized by their writings and statements. Much like the horses my wife used to train--- Desensitized

And that is what this article is to me----just amusing in that those that cry out for more perception have very little themselves!

There was a time when I would spend a couple of hours ticking off each little innuendo and comment that was not a fair representation and try to counter those comments. Who am I fooling by doing that? I have no doubt today that by doing so would not change anyone’s mind on this subject and from their writings, they have not changed mine. No one has come up with the proverbial bullet of ‘When’ and ‘How Often’.

Do not get me wrong forum readers in that I do believe that on some occasions ‘bias’ in its many descriptions as stated in the article do exist. But I am not going to bite on the ‘Blind Verification’ bullet until someone can tell me when bias will occur and how often in case work. I am not going to move to some procedure that will endanger the public safety until someone can show me that by not doing so is more of a danger.

I do feel sad though that in looking at the comments from readers of that article that they do not have what Paul Harvey would say “The Rest of the Story”.

So when you see one of these in a ‘Pulp Journal’ just get amused because getting mad does nothing but add fuel to the fire.

PS---Oh yea the statement that only PhD and Scientist can teach Latent Print Examiners how to do it better. That one is like a 'Screen Door on a Submarine'.

sharon cook wrote:Incompetence, not bias, is the biggest problem in the fingerprint profession at this time. The SCRO people were and are simply incompetent. National standards for testing and training do need to be established in America and around the world.

If I thought addressing competence, as in testing and training, would answer all the issues, I'd be a lot happier, because there's an obvious fix, even if that fix takes a good bit more than just a mandate. I just think it's too easy to cry incompetent. Certainly, it has a lot of truth to it in some cases. If I change incompetence to lack of professionalism, it addresses it a little more broadly and maybe more accurately, since people just tend to be more rigorous when they had to work hard for the privilege, and I think that maybe includes the suspicious doings in Cowans (in what was apparently an ill-formed unit) under the heading of competence/professional. And it's far too glib to write some cases off as evil conspiracy. It's just not that simple.

But I still have to look at cases like Mayfield. I don't think that one can be pinned on training and testing. And it can't be called inattention or taking a case lightly. There could hardly be a bigger fatter deal than working a hit on an international terror case in a domestic database. You know, "bias" has a nasty sound, with more or less subtle associations with hot-button social issues. That make it a sort of hot word itself that immediately puts people on the defensive when the point of the exercise is to learn something about ourselves and what we do.

There are specific potential biases that can be isolated and studied to see if they have any significant effect. That sort are the kind of factors that can be controlled by procedural designs. And there are potential biases that are next to impossible to isolate for study. I would characterize that sort as not systemic factors but rather basic human factors. And therefore they may or may not be active factors, purely on account of differences in personality and self-discipline. If they arise in any one case, you won't even know it. You just design your procedures to catch any actual errors and leave it anyone who doesn't agree with the results to do their own examination and argue it.

And, Charles. I can hardly expect academics to avoid any study related to a field where they are not practitioners, just because the practitioners aren't initiating or joining the studies. They know they can't do a study and expect it to cause change in the field. But when evolving knowledge of how humans do things suggests a line of inquiry, that's not going to make them walk away from the issue. That situation is very much on a lot of minds at this moment. And it really is being joined by both academics and latent print professionals.

Drors research is based on a sample size of 5 and anecdotal evidence. I hardly call it research but it certainly did cause a ruckus.

Ok too harsh. Yes it's research!

Not to take away from the valuable insights Dr. Dror is giving us into the workings of a fingerprint examiner's mind when biased in one way or another, but this article sensationalized an incorrect conclusion. The research cited in the article showed that three out of five experts excluded a suspect when given strong information he did not do the crime. There is a world of difference between an erroneous exclusion and an erroneous identification.

The latents used to trick the experts were ambiguous smudges and partial prints. The majority of crime scene latents examined in the daily course of an expert's work are not those "borderline" latents, but are clearer latents that lead to correct conclusions. The question is whether we should even look at borderline latents. Do we really want to ignore a latent because it is "hard," and risk letting a rapist or murderer go free? Or equally reprehensible, we might be able to exclude a suspect so the police can move on to find the real criminal, but if we ignore a "hard" latent, that alone may lead to an innocent person's conviction on other circumstantial evidence or erroneous eyewitness identification. If we try to work with those borderline latents, we run the risk of making a mistake. Discarding them or ignoring them runs a risk of guilty going free or innocent being convicted, just as making a mistake risks a bad verdict in trial. Do we look at them? "Damned if we do, and damned if we don't."

Other research being done suggests that introducing bias may produce erroneous exclusions, as Dr. Dror's project proved, but that such bias seldom produces erroneous identifications. When put under pressure with heavy bias, fingerprint experts tend to pull in and become more conservative. That is why they make erroneous exclusions. Trying to bias an expert into an erroneous identification is a different proposition entirely. I would not say it never occurs, but I would say that it is not valid to prove erroneous exclusions can be induced the way Dr. Dror did, and then try to imply that erroneous identifications are common. That is worse "science" by far than what this article accuses fingerprint experts of doing.

While Dr. Dror's research suggests those mistakes may be common, he introduced much stronger bias on much weaker latents than we are likely to see in a lifetime on the job. Bias occurs, and Dr. Dror is correct to suggest removing bias from our jobs. But the conclusions, as presented in this article, paint a terribly inaccurate picture of fingerprints. It is still a highly reliable science.

What about cases like Mayfield? Stephen Meahger said in conference presentations that erroneous identifications happen at the FBI about one time in 11,000,000 correct identifications, based on FBI statistics. That's pretty accurate, unless you happen to be that one person who was erroneously identified. But if you were the judge, would you ban fingerprints from the courtroom for an "error rate" of 1 in 11,000,000?

What about cases like Cowens and McKie and Asbury? Having been personally involved in those cases, I would argue there was more than mere bias of an honest examiner at work there. The evidence in those cases leads me to believe you need to look beyond honest mistake, not only on those, but on many of the other "misattributions" cited in the works of Dr. Simon Cole. In many cases, incompetence and dishonesty play a much greater part in the errors than honest mistake.

Dr. Dror's research pointed up a serious issue. To misinterpret his results as this article did for the sake of sensationalism and tabloid journalism is, in itself, an unjust use of "bias."

When put under pressure with heavy bias, fingerprint experts tend to pull in and become more conservative. That is why they make erroneous exclusions.

If an examiner were being more conservative, wouldn’t they have written ‘inconclusive’? It’s my opinion that calling something an exclusion simply because you can’t make an ID isn’t being conservative, it’s just erring in a different direction.

Pat mentions the problems of working with hard latent prints stating, "Damned if we do, and damned if we don't." I don’t see it this way. I think the solution is easily overcome as long as examiners can show the basis for their conclusion... a basis that will hold up to the scrutiny of others. This is something that’s desired in science but wasn’t requested in Dror’s study… and maybe not in the lab that was studied. Maybe part of the problem was the examiner’s lack of understanding about what’s required to arrive at a scientific conclusion. Without following a standard then it’s obvious that examiner’s may arrive at one conclusion one day but arrive at another conclusion another day.

I guess I haven't really thought enough about exclusion. Is there any difference between an erroneous identification and an erroneous exclusion? If we view each as absolute conclusions, we either clearly identify details and find them in agreement with some portion of recorded skin, or we cannot reconcile what we can see of the latent with any portion of the recorded skin. That's a real difference. (I'm assuming here that no one is going to support guessing poorly impressed details in a latent into fitting a record print.) So, identification requires the examiner to characterize each usable detail to the exclusion of all other characterizations.

But to be useful in exclusion, details may be characterized more broadly. It becomes a question of if it's possible that the record skin could have made this latent. That's not a useful question in identification. But in exclusion, I can exclude on account of there being no portion of the record skin that could make any impression like the latent. The detail might, in fact, be lost in a void, but I can see the ridges in and out of the void and know that something happens to a ridge in there. Any prospective portion of record skin under consideration that doesn't have that loss of a ridge isn't the source of that area.

So, what does it take to make an erroneous exclusion. To make an erroneous identification, you have to misinterpret what you see, which includes simply not seeing something. What you thought you saw was an incorrect characterization the skin that made the latent impression. Or, of course, you settled for too little quantity, which I don't think is often the problem. To make an erroneous exclusion, you have to either make a reasonable characterization of what you see and fail to properly examine the record impression, or you have to grossly misinterpret even the general character of the latent. And I suppose you could also make a mistake in deciding that the nature of the latent nailed down the anatomical location of the source skin and improperly looked only there.

Nothing in this suggests to me that either exclusion or identification can legitimately be taken more lightly than the other. I see nothing that would make the examiner's nature as conservative or liberal(?) contribute to error more in one conclusion than the other. You cannot distinguish exclusion and identification on account of their effects on a case or people. If you make that distinction, you're as much as saying your tolerances change with the circumstances of the case.

I don't know if this was a thought in the study, but I actually think I like the idea of studying bias with exclusions. If I attempt to bias with regard to identification, I might be affecting either the interpretations of perceived details or the threshold for identification or both. If I attempt to bias with regard to exclusion, I'm only affecting the interpretations of perceived details. There's no threshold in exclusion. I either find no portion of skin consistent with the nature of the details or I do. A single feature could exclude. I could legitimately study bias from that angle, because I'm purely looking at what can be an effective bias, and I think simpler effects make better studies. Although identification is the brass ring, exclusion is just as powerful a conclusion. If I am studying an analytical method, I cannot make judgments of which conclusions are more sought after, just as the analyst cannot make such judgments without admitting to a powerful influence from outside the lab.

Just exploring the question. But I think we have to take care about criticizing any study as not being structured as "typical" of most examinations. The goal of a lot of ongoing study is to narrow the issues, perhaps discovering which factors don't much affect examiners. What we have to speak up on is when a particular study is improperly applied to make some broad statement about examinations. Medical studies are terribly misused by media and by the public. A study shows some effect. The headlines read, "Halitosis Risk for Bagel Eaters!" Maybe, one day, further studies confirm that bagels cause halitosis. Maybe they show that genetic predisposition to halitosis also causes a liking for bagels. Those who conducted the original study are perfectly correct to point out that their results could mean this or that. But it's only fairly recently that researchers in all fields have learned to expressly caution that one shouldn't assume definitive causation from their results, and people are still going to misuse the results anyway.

Gerald, you wrote:

but I actually think I like the idea of studying bias with exclusions.

We have some data on this issue. We have two studies showing clear differences in bias effects towards exclusions (and inconclusives) v. identifications. The former are significantly easier to influence. The most logical reason is that subconsciously there is risk-cost difference, especially if erroneous id's lead to loss of job, loss or certification, or public "tarring and feathering" on clpex, whereas erroneous exclusions do not have the same pariah like effect. Therefore it makes sense that the decision making processes may be affected subconsciously. Also we identified the same trend in Dror et al's data.

See: Langenburg, Champod, Wertheim, "Testing for Potential Contextual Bias Effects During Verification in Fingeprint Comparisons" J Forensic Science, 2009, 54(3), 571-582.
and Langenburg, "Method Performance Pilot Study..." J Forensic Ident, 2009, 59(2), 219-257.

If you don't have either, email me and I can forward a .pdf copy.

g.

Mackie print- determimed to be erroneous by fingerprint specialists and the fingerprint community. Therefore validated.

Madrid print - determimed to be erroneous by fingerprint specialists and the fingerprint community. Therefore validated.

How do I know a dog is a dog? It looks like a dog and is generally accepted to be a dog. I ask someone else "is that a dog?" validated by asking another member of the public or friend and the biology school training received from (clasification), books, movies, scholarly literature.

How do I know how to identify correctly?

-Scientific study dating back over 100 years, scholarly medical research (embryology, chemistry, biology, physiology) VALIDATED by the medical community.

-Experience at classifying fingerprints and understanding ridge path-flow-detail deposed in different ways and developed by different mediums. Thousands of comparisons validated by my peers.

-Assessed and accepted by the courts and jury for 100 years. Passing the test of general acceptance (as has many different sciences).

What can we do to avoid mistakes?
-regular training.
-peer review.
-certification.
-competency testing.
-accreditation.
-code of ethics.

No amount of statistical analysis is going to make my opinion any less or anymore than what it is. Furthermore, it will be tested by my peers. That is observational validation.

If there happens to be a computer program that provides a likely-hood ratio that my analysis is correct when I have a problematic latent I will go to my peers and let them test my method and conclusions. I will hardly rely on something that cannot see the print.

If I am diagnosed with a disease I am going to ask for a second or third opinion or more from a physician.

I look at actual reproduction of skin surface detail not a computer generated mathematical depiction. It's actual visual evidence.

As for bias - who hasn't got it. That is exactly why our results are verified validated so that we can be assessed as to whether there is some bias and correct it if there is a need to. They too must follow procedure and protocol.

The court system is there so that we can be challenged. That is the adversarial system and I will give my opinion based on scientifically proven methods and procedures. My decision is neither for the prosecution or defence. That is what a code of ethics is for.

In addition it is a good idea to be audited by an external lab.

raybie wrote:The court system is there so that we can be challenged. That is the adversarial system and I will give my opinion based on scientifically proven methods and procedures. My decision is neither for the prosecution or defense. That is what a code of ethics is for.

I agree that is the ultimate cure. I think a lot of the reason that all sorts of bias and error is so hotly discussed is that few identifications (and probably almost no exclusions) get any sort of expert adversarial review. I say "adversarial," because once the report leaves the unit in a prosecution, it becomes purely adversarial. The analyst may be meticulously neutral, but it becomes evidence presented by the State for its purposes. It's not a fantastical proposition to suppose that the Madrid ID might have not been challenged by the Spanish and that Mayfield might have been shown to have been placed in some position to have contributed the latent. One wonders, then, to what extent his counsel would have sought yet another examination.

But that's a high-profile case, and we know that few fingerprint cases get review by another entity. I have seen any number of fingerprint cases in which defense counsel did not seek review of any kind but instead opted to propose more or less unlikely scenarios that presumed the ID was right but that the deposition was innocent. They accepted the State's analysis without question. When you have that sort of situation, along with revealed errors that passed multiple examiners, it just invites criticism. I take the criticism as not a one-dimensional attack on fingerprint identification but on the way the system uses it and its standing. There's little to criticize when evidence gets critical review. (I won't say independent review, because I have enough experience with expert testimony to know that when there's advocacy, there is invariable, virtually by definition, bias.) Both advocates are often depending on the same methods.

Examiners can gain valuable perspective on the ongoing and future research efforts by thinking about how they would respond to two kinds of challenge, one being another expert rendering an opinion contrary to their own and the other scientific and logical challenges. It means thinking about an environment different from what almost all examiners are accustomed to, but I've tracked evolving legal opinion in this and other fields long enough to be convinced that the environment will be changing. How would you respond to another expert, equal or greater in training and experience and personal track record with yourself, who concludes that there is simply not enough to conclude identification? Can you support your conclusion with substantially more than the "he thinks - I think" contest of clinical guessing seen when psychologists differ, which in the nature of legal contests is almost always? Can you show the court the particular accumulated quantity of data in the latent where one or two or three data quanta less would have made it inclusive? Can you effectively show that another examiner is wrong to say that at that same quantity of data one or two or three more data quanta is needed to move it from inclusive to identification? Without quantified research products, you are both arguing from precisely the same position of guessing from knowledge and experience.

Another way of looking at it is to see that in this kind of clinical argument, it's not a matter of whose conclusion reflects the real source of the impression. The argument is rather over whether what can be observed in the latent is sufficient to declare identification. You can be right about the source. The other expert can be right about there being insufficient data to make such a declaration of identity in a weighty matter. How will you show your judgment of sufficiency is correct or more to be relied upon? When you look at it in this way, and consider how either expert would answer the above questions, you see that the number of identifications that might be challenged is far greater than the "close calls."

The notion that "all competent examiners will conclude similarly" is patently not true. It's demonstrably not true. Given the current standard for conclusion, it's absurd that it would ever be thought to be true. Yes, it's almost always true in individual cases. The problem is how you answer a challenge, of the sort not often encountered yet, when in a particular case, it's not true. That is one of the goals of the emerging research, to keep it from becoming a mere clinical opinion contest.

In reply to Sharon Cook
"Probability studies may be the most ridiculous waste of time with regard to the fingerprint profession, ever. There are FAR too many variables to ever turn out meaningful data on the effort human beings put in to fingerprint examination."


I completely disagree with this…… if we are a science as we claim to be, our profession should want to push studies that show meaningful data such as the likelihood that any other person could have made the questioned impression. The latent community makes statements such as “this is made by this source because I have never seen this many ridges alike in two prints….. based on my training and experience”. Using the doctor analogy if you have a disease you would go to multiple doctors to check the diagnosis, but hopefully the doctors that you are going to are running tests (computer aided) in order to come to a diagnosis. I would not want a doctor that simply did a physical examination and then told me I had some hideous disease. It would not set well with me to hear “well I am the expert so lets start some radiation therapy”. If you want to compare us to other sciences then we need to demand more of ourselves. Why should we not push for the use of tools to give more validity to our decisions such as probability models? I would love to have some scientific data that backs what I say. Like a doctor, it does not make me less of an expert to use technology. I say a big thanks to those out there conducting research on probabilities and bias. Further down the road I think that it will be seen as an asset not a waste of time.

Very interesting reading.

What is it with you guys from Texas, is this the OK coral all over again?

I think I would swing toward what Pat is saying if I looked at this in a non bias point of view.

However why on earth would you want to make a doggy ID on a high profile case like Mayfield, I suppose there is the likely hood of a free trip to Spain to think of, that would intise some!

Here in NZ we do have independent fingerprint experts who do regularly check our Idents that head for court, but I did not see a change in the way we did our work, in some ways I think we appreciate this type of validation that is paid for by the defence not the prosecution.

Hi Sharon angels are too big to dance on pin heads, my late wife told me they are larger than life!

Gerald I spent some time with the horoscopes but would you believe it there was no answer to the problem there, should I, in your opinion try tea leaves then? I'm only kidding. I really do think we are not that far apart from what we really at heart believe, I just don't see that introducing statistics will make that much difference.

But Pat isn't eleven million to one the kind of figures used by the DNA guys. Not as impressive as our 10 million million to one for fingerprints though.

Michele I agree with your statement 'If an examiner were being more conservative, wouldn’t they have written ‘inconclusive'.

Tash science does not equate to statistics except to a statistician, and in themselves statistics prove nothing except the ability to argue from two completely different points of view using the same information, what is the difference with that to two f/p experts arguing about whether a point is there or not or a ridge is ending or is forking. If you can't put your case in court strong enough the way things are why on earth do you think that you trying to argue statistics will be any easier?

I suppose there is some truth in all this, it is just that I do wonder if we are trying to use the old one size fits all or you can have any colour so long as it is black, thinking. Everyone is different, like Pat suggests if a detective starts in on me to make a match I will be more conservative because I smell a RAT. I also would take a difficult Ident to some one conservative in their approach rather than some one that would push the envelope.
However I have seen the opposite occur in the past and it did worry me and to compensate I would take on the hat of the conservative examiner.

Good to hear from you Charles I have missed your additions, I hope all is well.