There's a bias monster under every bed

[Boyd Baumgartner]

http://www.sciencedirect.com/science/jo ... 113681/2/1

The latest bias paper from Dror with 12 commentaries by the ususal suspects.

The funniest thing about the paper:

Examiners should work “linear” rather than “circular,” ... Indeed, the FBI recently revised its Standard Operating Procedures (SOPs) to “include some steps to avoid bias: examiners must complete and document analysis of the latent fingerprint before looking at any known fingerprint”.

Followed by:

Acknowledgements

This work was supported in part by a research grant by the National Institute of Standards and Technology (NIST), Federal Bureau of Investigation (FBI), and Department of Defense (DoD/CTTSO/TSWG), #N41756-10-C-3382, awarded to Itiel Dror; and in part by funds provided by the Research Foundation of the City University of New York to Saul Kassin.

Is this research or advertising?


Dror's papers always reminds me of a Mad Magazine spoof I read on Murder She Wrote in 1986. In the parody, when the Detectives learn that everytime Angela Lansbury's character goes somewhere there's a murder, they realize perhaps she should be treated as a suspect. The translation here being that perhaps someone should study the biasability of bias researchers as his experiments always seem to turn up bias of every kind in every aspect of every task.

All snark aside, as I've said before, I think that there is still a lack of mechanism illustrated by Dror, but rather he appeals to a nebulous 'psychology' and he ends up having to cite everyone from Cicero to Amos Tvsersky to the Wason selection task to make his point. At the end of the day, it's nothing he hasn't already said since 2008 (http://www.clpex.com/Articles/TheDetail ... ail367.htm). As far as our industry goes, I think we lack a theory of value and we end up having to settle with notions of confidence that are disconnected from what we're seeing. Just look at the latest swgfast docs.

However, since Dror mentions mitigating bias through the use of a linear comparison process that he should test that. Put the FBI's process to the test. If he's right we should find a statistically signifcant difference in the intra/inter rater reliability metric.

[Tazman]

From Dr. Dror's article:

Abstract
As illustrated by the mistaken, high-profile fingerprint identification of Brandon Mayfield in the Madrid Bomber case, and consistent with a recent critique by the National Academy of Sciences (2009), it is clear that the forensic sciences are subject to contextual bias and fraught with error. . . .

From the very first sentence of the article, Dr. Dror alleges that the practice of fingerprint identification is "fraught with error." Then he goes on to cite the Brandon Mayfield case, as if the one error he can cite in the past fifteen years makes us "fraught with error" in spite of possibly millions of correct identifications across that time span.

I would suggest that if a single cognitive scientist anywhere in the world has made a single error in the past fifteen years, then cognitive science is more "fraught with error" than fingerprint identification because there are far more fingerprint examiners doing far more reliable work and making far fewer errors than Dr. Dror and others of his profession. If one mistake in a science in fifteen years makes an entire field unreliable, how can we believe anything Dr. Dror says unless he alleges that no cognitive scientist anywhere has ever made a single mistake?

[kevin]

I think it is funny that fingerprints is criticized for not being a true science by the psychology field....

[kevin]

as an addendum I do appreciate that Dr. Dror has also included DNA and firearms in his research on bias...

[Tazman]

If Dr. Dror really wants to do valuable research that would qualify as good science by any definition, he should design a protocol for collecting data to calculate the true error rate of fingerprint identification. Specifically, how many correct identifications are made for each error like Mayfield. But the sad truth for him is that if such an error rate were actually known, it would put cognitive scientists who explore bias out of business.

[PCC]

I keep looking for the "like" button on everyone's posts (too much Face Book).

[timbo]

Confucius say "Psychologist who live in glass house should not throw rocks - especially if he only has 1 or 2 rocks...."

[Tazman]

Aw, shucks. He ain't got no rocks, no how.

[timbo]

Well, not a statistically significant subset of rocks, anyway......

[David Fairhurst]

A valid comment Boyd:

"Studies that look for confirmation bias are likely to find it, due to confirmation bias"

Discuss.

[David Fairhurst]

And perhaps we should examine to what extent status quo bias is a real mover for change.

[David Fairhurst]

Looking to the future...

When we get a paper revealing hindsight bias, everyone will shout "but we knew that all along!"

[David Fairhurst]

And finally (for now)...

First attempts to study framing bias were focussed too narrowly to have any scientific validity.

[L.J.Steele]

And what should we use instead of published, peer-reviewed scientific articles?

One can, and I often do, question a study's methodology or its conclusions, but ad hominem attacks on the researcher himself are childish, folks. In our own ways, all of us are supposed to be adult professionals, surely we can treat each other and the folks we disagree with respectfully. (Maybe set a good example for our political leaders about reasoned debate.)

[Tazman]

Dear Ms Steele,

You are correct, of course. By descending into name calling, we have not contributed professionally to the quality of this discussion board. We attack Dr. Dror out of frustration. He is a professional critic who performs and publishes the same research over and over and over. He cites his previous papers and presents his same old hypothesis that our work is "fraught with error." To prove his hpothesis, Dr. Dror dredges up one example of an erroneous identification from a decade ago to make his point.

The truth is that there are thousands of honest, consciencious latent fingerprint examiners around the world who spend their entire careers spanning decades without ever once making an erroneous identification. We all fail to find latent prints that we should identify. We occasionally wright down a wrong finger in an administrative error. Sometimes we even exclude the person who actually made the latent. If you include all of those errors, then perhaps the word "fraught" may arguably apply. But when Dr. Dror uses that word, he implies that our work is fraught with erroneous identifications. And that is just plain WRONG.

In his studies on bias, Dr. Dror has proven that he can induce erroneous exclusions and missed identifications. He has had tremendous difficulty inducing an erroneous identification. In fact, I suspect he had been unable to do so. Why? Because the problem he is trying to sensationalize is so very, very small that if its true rate were known, Dr. Dror would be out of business.

So, allow us to bash this man. He has made a career out of bashing us and in doing so, has cast an undeserved shadow across a field of forensic science that is far more accurate and reliable than his own.