Cognitive Bias

[Iain McKie]

Updates have been provided over the years on the work of Itiel Dror and his team on cognitive bias.

From an initial skeptism among some experts there now appears to be a growing consensus that fingerprint experts, in common with the rest of humanity, can under certain conditions suffer from bias related to emotional and psychological effects.

The check and balances created to reduce their impact are of course critical and I believe that the ongoing work in this direction is important.

I hope that next year’s ‘Fingerprint Inquiry Scotland Report’ will major on these issues and hopefully help in developing a more robust and consistent approach to cognitive dissonance and other similar effects that can potentially bias an expert’s conclusion.

Yesterday BBC Radio 4 broadcast an interesting discussion on this very subject between Dr Dror and others involved in UK fingerprint quality and control that is well worth a listen.

http://www.bbc.co.uk/iplayer/episode/b0 ... 6_11_2010/

[charlton97]

http://www.telegraph.co.uk/news/uknews/ ... judge.html


UK Daily Telegraph
Fingerprint identification evidence questioned by senior judge
A senior judge has raised concerns over fingerprint evidence used in criminal trials, warning that it rests on “assumptions” that have never been scientifically proven.

The judge said that there have been 'numerous' recent cases of innocent people being wrongly singled out by fingerprint evidence Photo: GETTY IMAGES

By Richard Edwards, Crime Correspondent 8:00AM GMT 19 Nov 2010
Lord Justice Leveson, an appeal judge and chairman of the Sentencing Council, called into question the “century old” identification process, which he said was often considered “virtually unassailable” in tying a person to a crime.
The judge said that there have been “numerous” recent cases of innocent people being wrongly singled out by fingerprint evidence.
In a speech to the Forensic Science Society in London, he said the analysis of fingerprints by experts was “fundamentally subjective” and that it was therefore “inherently capable of misidentifications”.
Lord Justice Leveson called for new research to be carried out to ensure fingerprinting is “robust” and reliable.
“There is growing unease among fingerprint examiners and researchers that the century old fingerprint identification process rests on assumptions that have never been tested empirically,” he said.
Speaking about the use of expert evidence in court cases, the senior judge said it was vital to have a “methodology and a hypothesis that are capable of withstanding robust testing”.
“Arguably, as it currently stands, the science of fingerprint identification does not,” he added.
The judge is the most senior member of the judiciary to speak out about the concerns, which have been raised by experts and academics for the past few years.
However it is unlikely to lead to a series of convicted criminals appealing their cases because in most instances where fingerprints are now used in court, the evidence is corroborated by other forensic samples such as DNA testing.
However examples of failings include Brandon Mayfield, who in 2004 was wrongly linked to the Madrid train bombings by FBI fingerprint experts in the United States.
Shirley McKie, a Scottish police officer, was wrongly accused of having been at a murder scene in 1997 after a print supposedly matching hers was found near the body.
Criminal fingerprinting techniques were pioneered at Scotland Yard at the beginning of the 20th century. The first successful conviction using them was of Harry Jackson, a burglar who was jailed in 1902.
No two fingerprints are ever exactly alike in every detail – even two impressions recorded immediately after each other from the same hand.
It requires an expert examiner to determine whether a print taken from a crime scene and one taken from a subject are likely to have originated from the same finger.
Unlike other forensic fields, such as DNA analysis, which give a statistical probability of a match, fingerprint examiners traditionally testify that the evidence constitutes either a 100 per cent certain match or a 100 per cent exclusion.
Lord Justice Leveson said: “The language of certainty that examiners are forced to use hides a great deal of uncertainty, which greatly undermines the examiners’ legitimacy.”
A recent study found that experts do not always make the same judgment on whether a print matches a mark at a crime scene, when presented with the same evidence twice.
Six examiners in several countries were given eight sets of prints to compare on two different occasions, without knowing it was past of a study. They changed their decision in six cases and only two of the experts were consistent with their previous decision.
The research, carried out at Southampton University, found they were more likely to change their decision if given contextual information, such as "the suspect has confessed", that conflicted with their previous judgment.

[Tazman]

Dear Mr. McKie: I hate for cases involving fraud and dishonesty to be lumped in with cases of true erroneous identification. Having followed your daughter's case somewhat, it is my impression there was more to the Scottish Police and fingerprint examiners' position than mere "honest mistake." It seems to me there were other "biases" they might have had against your daughter personally. Do you believe it was a case of true erroneous identification?

Dear Mr. Charlton: How many actual erroneous identifications have you and Mr. Dror been able to induce with bias? I know you have induced erroneous exclusions, and you have induced analysis determinations of suitability on borderline marks through biasing fingerprint examiners. The article published in the JFS by Langenberg indicates that the more you try to bias an expert toward an identification, the more alert the expert becomes and there can even be resistance to making the identification. So I would like to know from you, without reference to your research proving erroneous exclusion or suitability of borderline marks, how many erroneous identifications have you and Mr. Dror been able to induce by biasing a fingerprint expert?

[charlton97]

Tazman,

it is a fact that we have been able to provide data that supports erroneous identifications as well as exclusions. In this regard, both error states exist and can be influenced by bias. The precise weighting of the findings is really a red herring. Even if future studies echo Glenn's findings, that is still very important, since whether the influence is one way, or another, the fact remains that there is an influence. If we assume that justice is not just about catching bad guys, but also about serving exculpatory evidence on the judicial system, then reliable, accurate and consistent reporting of findings must exist. The truth is that such consistency is just not there. If Glenn's finding indicate heightened caution when faced with increased context then that in itself is bad as what we want are examiners being consistent to the evidence observed. If murderers walk free as a result of 'reverse effect bias' then that to me as is serious as bias that induces erroneous identifications. One final point is that the ultimate decision outcome is one thing we can observe, but from a research point of view, what I am interested in is effects of bias and context on decision thresholds. In this regard, decision thresholds are nearly always effected by such influences, the directionality of such changes is a matter for ongoing research, however, it should be noted that changes to decision thresholds does not always impact upon the final outcome decision (ident/non ident), but given that errors occur in very poor quality latents, it suggests that the movement of the decision threshold becomes more important as the comparison becomes more difficult.

[Iain McKie]

Dear Tazman,

Your point is well made and highlights one of the potential unforseen problems of this research.
...
Such dishonest experts could therefore claim such bias as a way of escaping responsibility for their dishonest acts.

I remain confident however that the up and coming inquiry report will see through their deceit.

This said it would suit the purposes of the Crown Office that they be allowed this escape route.

I would also hope that the cognitive researchers in their enthusiasm to see their theories gain credibility were not drawn to seeking to explain such criminal acts away as the unforseen result of unconscious pressures on the experts.

Best wishes,

Iain

[Tazman]

it is a fact that we have been able to provide data that supports erroneous identifications as well as exclusions. In this regard, both error states exist and can be influenced by bias.

Dear Mr. Charlton,

The question I asked was "How many actual erroneous identifications have you and Mr. Dror been able to induce with bias?" You danced around the topic without answering the question. I believe the correct answer you gave between the lines is "Zero, but it could happen." Am I wrong in that interpretation of your answer?

The problem, Mr. Charlton, is that we who work in the trenches have to answer defense attorneys in court who shake your papers at us and allege falsely that you and Mr. Dror have induced erroneous identifications through manipulated bias. You have induced erroneous exclusions and you have shown that examiners will work harder on gruesome, violent crimes than they will on misdemeanors. But unless I am mistaken, you have never yet actually led an expert to make an erroneous identification through your manipulations of contextual and confirmational bias. Am I wrong?

While some of the SCRO experts were probably affected by psychological, cultural and other bias a hard core were criminal and dishonest in their intent.

Dear Mr. McKie,

That is part of the problem with articles such as those by Simon Cole in which he lists "erroneous attributions." He lumps the fabricated evidence and perjured cases in with the "honest mistakes" to prove his point that fingerprint identification is unreliable. What happened to Shirley was a travesty of the worst order, but I think you agree that "honest mistake" or "confirmation bias" may have only been applicable during the first week or less after Shirley's fingerprint was first "identified" at the murder scene. After that, the treatment she suffered was criminal. I hope this latest inquiry (What is it, the third inquiry? The fourth? The fifth?) confirms such to be the case. Then maybe the Simon Coles of this world will delete her case from their lists of erroneous identifications. And maybe the police will reopen the actual murder investigation, but I doubt it.

Thank you both for your responses.
Tazman

[Dr. Dror]

Dear Tazman,

I think your posting is very important and exemplifies one of the problems: That many forensic scientists work within the adversarial legal system. The scientific culture many times relies on 'questioning', being sceptical and critical, self-doubt, etc.; in the adversarial legal system these can be used as a weapon by the opposing side.

I have no control how our studies and findings are used; and I am sorry if the opposing side used them to attack you; you can also use them to defend yourself (for example, our studies show that these issues are mainly with the difficult and 'hard to call' cases, that some examiners are very consistent and not as affected by context, etc.).

Finally, to directly answer your questions, 'How many actual erroneous identifications have you been able to induce with bias?" Well, the answer is quite a few, dozens of times! Because we are sensitive to the issues you raise (the impact on examiners in the adversarial legal system), we do not always publish and make publically available our data; sometimes we wait until we have 'solutions' to the problems we find.

As per studies of bias, for it to affect examiners, they must truly believe it. So any experiment or study that examines bias, but the participating examiners know the context is not real, part of an experiment, then any effect of the bias is significantly diminished, if not eliminated altogether.

I hope this answers some of your questions. I'd be happy to discuss this further with you and give you more information directly; please feel free to e-mail me, at: i.dror@ucl.ac.uk

Thank you,

Itiel

[Iain McKie]

I have decided to remove my last posting because it is in no way related to the topic of ‘cognitive bias’.

As has happened on previous occasions important threads of discussion become bogged down with charge and counter charge related to the SCRO debacle and this must be offensive to some experts.

While I still hold the SCRO experts completely responsible for their actions criminal or otherwise I am content to let matters rest and await the Fingerprint Inquiry Report.

Dr Dror states:

That many forensic scientists work within the adversarial legal system. The scientific culture many times relies on 'questioning', being sceptical and critical, self-doubt, etc.; in the adversarial legal system these can be used as a weapon by the opposing side.

Any expert witness showing doubt or offering alternative solutions is likely to be attacked or used by opposing counsel and this does I fear lead to some experts believing they have to come to definitive decisions in order to ‘win the day’.

I have long favoured a more Inquisitorial system where ‘expert evidence’ is obtained for the benefit of the court and not to favour the defence or prosecution. It seems to me that the ‘best available’ evidence should be used and while it should be open to criticism and witnesses should be questioned on it the adversarial bullying and grandstanding should be no part of assessing such evidence.
http://en.wikipedia.org/wiki/Inquisitorial_system

[Iain McKie]

For me the value in the cognitive bias research being carried out by Itiel Dror et al is in highlighting the range of variables at play affecting the print analysis process and in fact any human decision making process.

I remain confident that the fingerprint profession is taking on board the issues raised by this work and is developing checks and balances to ensure the credibility of the identification process.

I am not sure how remedies like a structured grading of print quality or providing measurable reliability assessment print to print would work but it is clear that different degrees of difficulty print to print and some prints increased vulnerability to misinterpretation must be recognised. Perhaps in this move away from the imperative of 'infallibility' more experts will realise that 100% certainty is not required to give their profession value and credibility.

As observed in the Mayfield and McKie cases no matter how rigorous the process there will always be ways of consciously or unconsciously undermining it. Peer pressure and over reliance on the decisions of senior experts were powerful factors at play in the two cases and you can see how 'honest' experts could be subverted to the 'cause'. If you add to the mix the potential for criminality then we can see how the identification process has taken on a heretofore unrealised complexity.

As a counsellor/therapist this interplay of cognitive, emotional, psychological and cultural elements, often pulling in different ways, fascinates me. Without this understanding I could not do my work effectively.

I believe that this process of understanding the complexities of fingerprint analysis will one day lead to international standards being developed which will finally silence the critics who claim a lack of scientific rigour.