Debating research, especially when you do not agree with it!

[Dr. Dror]

I am well aware that our research is unpleasant reading and has caused quite a lot of anxiety and resistance among some fingerprint examiners. Lately this has expanded from hostility of individuals to attempts to play down and even censor our work from people in key and official positions in the fingerprint community both in the UK and in the US.

Debating, questioning, and criticising research is not only acceptable, but it is a corner stone of any scientific venture. We not only welcome it, but we actively seek it, as it helps us think and learn, and improve our research (and ultimately enhance the scientific underpinning of this domain).

Of course --in contrast to a lot of comments we encounter-- warranted responses and concerns should be:
1. Focused on the merits and weaknesses of the research itself (and not on personal hostility and personal attacks on the authors).
2. Based on the judgement of the research itself (and not based on whether the outcomes suit your wishes, preconceived notions, beliefs, ideology, etc.).
3. Used to understand and promote the fingerprint domain (and not as a tool and opportunity to settle personal scores, an outlet for personal emotions and agendas, etc.).

I am writing this because of an upcoming open debate which I would like to invite ALL of you to participate and contribute to. This is an opportunity to question and to voice criticism about our research. I am writing to encourage everyone to engage in this debate and to take this opportunity to express any concerns that you may have. Please circulate the Call for Criticism (see below) as widely as possible and please do write all the criticism and weakness you think our research has. We are looking forward to engage, interact, and learn from everyone’s comments.

This debate is taking place in the August 2007 issue of Fingerprint Whorld, The journal of the Fingerprint Society, UK. Deadline for sending responses is Friday, the 30th June 2007. The official Call for Criticism is going to be published later this month in the May issue of the journal with our paper "Expert Error: The Mind Trap". I am providing you with the Call for Criticism as early as possible, so as to provide you with as much time and opportunity to prepare your comments. Please send them directly to the journal: fingerprintwhorld@yahoo.co.uk.

Although the actual paper "Expert Error: The Mind Trap" is only coming out later this month (Dave Charlton will place a notice and copy of the paper when it comes out), the paper mainly reviews our findings, which can be accessed at:

http://users.ecs.soton.ac.uk/~id/ACP%20 ... 0ident.pdf

http://users.ecs.soton.ac.uk/~id/FSI%20 ... uences.pdf

http://users.ecs.soton.ac.uk/~id/JFI%20 ... 0error.pdf

And, you are welcome to also criticise any other aspect of our work, presentations, etc., including the TV interview (see: http://users.ecs.soton.ac.uk/~id/bbc.html ) and other work which can be accessed at: http://users.ecs.soton.ac.uk/~id/biometrics.html

Dave Charlton and I are looking forward to hearing all the criticism and weakness that you find in our research. We really hope that all those who do have concerns about our research will take this opportunity to raise them. Thank you.


This is the official Call for Criticism to be published in the May issue of Fingerprint Whorld:

Editor’s note:
It is very encouraging to hear that, following such a stimulating article with many points of discussion, the authors of the preceding paper are keen to open up a debate on the matters discussed. Should any readers want to pick up any issues and contribute to the debate with agreements, disagreements, further areas of discussion and debate then please do. Drop a line to: fingerprintwhorld@yahoo.co.uk.

Deadline for responses is Friday 30th June.

Responses can be anonymous and they can be as long or as short as you wish. Thank you in advance for any time and effort that you put into the debates on this subject, your opinions and contributions can help push forward the way we understand and work with fingerprints.

[Michele]

Itiel,

You state that people have anxiety over your research, they’re resistant to it, it’s been down played, and even censored. Can you give us some examples of how this has occurred? Almost every examiner I know recognizes the potential of biasing information and is continually looking at ways to protect against it. I’ve never heard one person react to your research as you state they have.

You mention censorship. Your research has been published everywhere(including the Weekly Detail). What more do you want? Do you want the same articles published over and over again? Your articles are already recognized as valuable and published. Am I missing something?

What’s the intent behind the up coming debate? I’m sure you’ve already heard plenty of ways to improve your research, what’s the real reason for this event? It appears to be more of an attempt to promote and publicize your work than an attempt for peer review.

I know this sounds a little direct but I’m not good at being tactful.

[mdavis]

On one hand, research such as this can be valuable to remind examiners that bias does exist and can influence decisions. On the other hand, reports given to the news media and defense often fail to criticize the experiment in terms of real world conditions and the application of methods which seem, on the outside, to be deliberately designed to produce a pre-determined result. In other words, the results will be reported without mention of the conditions. I'm sure this has been beaten to death elsewhere, but consider the conditions.

First, subjects were all students. They were all volunteers. They were all young. They were predominately female. There are many variables here that do not reflect working examiners, professionals, non-volunteers, experienced and trained, and of unknown sex ratios.

Second, the example of "ambiguous pair" would not be ambiguous to a professional examiner. Such ambiguity is a value judgement made by the researchers, another variable.

Third, examiners are seldom exposed to lurid accounts of the crime, and certainly not to photographs of traumatic injury in preparation for examining latent prints allegedly related to the crime. Most examiners are "bench workers" who simply compare latent prints, case by case, with seldom any knowledge of their acquisition or circumstances.

Fourth, subliminal suggestions in the form of flashed messages on a computer screen would never be encountered by working examiners.

Fifth, examiners may use a number of devices to enlarge or enhance images, but seldom if ever make decisions on a 17" monitor with 72dpi resolution.

Sixth, there was time pressure to respond "as quickly as possible". Some of the "ambiguous" comparisons might well be accurately called with adequate time.

Seven, the response keys were placed adjacent ("b" and "n") on the keyboard and did not correspond the the first letter of the required response ("same" and "different"). Was this designed to intentionally cause confusion and potential error?

Eight, there was no response corresponding to "insufficient" which might be an escape to untrained observers from forcing a hasty conclusion under duress.

How often are these experiments used as the basis of claims by the defense that fingerprint science is laden with errors when a single error is found. Are the scientific conditions of the experiment challenged by the courts? Experiments such as this can be valuable as a frequent reminder to professional examiners that conservatism is essential, honesty and integrity are assumed and practiced, and questionable idents are always honestly verified and either reported or thrown out as insufficient.

[Pat A. Wertheim]

I have no real criticism of the research being done by Dr. Dror nor any of the other scientists on both sides of the pond who are exploring the psychology, cognition, bias, and other topics that seem in contention today. My only concern is that opposing attorneys on both sides in court will twist whatever is written to suit their own purposes. They seem to care not for the actual, scientific conclusions of the research as stated by the researchers, but will reword it as they see fit.

I would very much appreciate a strongly worded statement or disclaimer at the beginning or end of these research papers to the effect that the research scenarios do not equal real world fingerprint situations experienced daily by experts everywhere, and therefor, while something may be learned from the research, assumptions or interpretations should not be made regarding the application of the research to any specific case at trial in court.

[Michele]

Mike,

I think your comment about reporting results without mentioning the conditions is key to several points!! You talked about the first article, I just wanted to mention some things I remember about the second article (it’s been a long time since I’ve read it).

The second study had 5 experts.
1st, that’s a very small sample group. A small sample group doesn’t always mean the data is wrong but the results and the sample size do need to be carefully looked at.
2nd, I’ve never been satisfied that these were ‘experts’. How was this determined? If I recall it was determined by the number of years in the field, which many people have argued that this doesn’t determine someone’s abilities. How could these people be considered experts if they didn’t recognize the Mayfield latent?? This was the biggest event in the history of fingerprints yet this study found 5 ‘experts’ that didn’t recognize it???
3rd, how was the study administered? This may have been in the article and I may have forgotten the answer but in many offices it would be impossible to give someone the same evidence 5 years later because it would have been written on. If you gave them a photograph of it then this would solve that problem but who determines if the photograph was the same quality as the original? If the quality were just slightly degraded then that may account for the examiners changing their conclusions and giving more conservative results. How did they find 5 agencies that could give examiners the same materials that they looked at years earlier without the examiners knowing it? This would be impossible in many agencies.
4th, Where are the images the examiners looked at? Why weren’t these made available to the reader?? To do a full peer review of this study we need all the information. I can see why everything may not be published in an article but has anyone seen this information (besides the authors)? Did anyone fully peer review this research?
5th, due to examiners skills, knowledge, training, experiences, and historical events, maybe examiners became more conservative with time. Was this looked into? In a scientific research project we should be looking at alternative possibilities for the outcomes that were observed.
6th, What happened to the 3 examiners who gave conflicting conclusions? Were these cases reviewed by the agency or was there some sort of agreement by the agency prior to the study?
7th, Was the consent form in the article? It would be interesting to see what it said. It may have had some information that would have been valuable to the reader.

I could go on and on. Examiners can be biased and I think this was accepted prior to this research. This research did make the degree of bias seem larger than I originally thought it might be but this paper was also only looking at latents with a low amount of information and examiners were only biased towards being more conservative (because that’s how this research was designed). These are really important parts of this research that are easily forgotten by defense attorney’s. I wish I had time to review all the papers that have been written but I don’t have the time to read most things once, let alone twice.

[mdavis]

I guess I'll have to go back and read the 3rd cited link/reference. I read #1. #2 seemed to be a condensation of #1 in an alternate publication.

Pat's suggestions are very valid, but I doubt we'll ever see such disclosure in a report. My overall sense is that this was a psychology experiment that incidentally used fingerprint comparison as one of the variables, rather than a study designed specifically to measure potential bias in latent print examinations. The experiment "stacked the deck" to make sure bias was recorded....rather like slamming a car into a barrier at 60mph to crash-test the 5mph bumpers.

We need carefully designed studies such as these to better understand our craft. What we don't need are sociologists, psychologists, attorneys and judges taking these studies out of context and applying them to real world cases resulting in valid idents being tossed out on inappropriate studies. To the continuing credit of latent print examiners nearly everywhere, when these bad idents are uncovered, the examiners are usually removed from the pool. No one seems to want to give credit to those who are left. In a world in which our courts preach "innocent until proven guilty", we need to ask if they are indeed giving examiners the benefit of the doubt. I think most courts still are. The exceptions are disturbing.

[Pat A. Wertheim]

Regarding this type of research, it is interesting to note the behavior of specific individuals in the research pool. For example, in Dr. Dror's and Mr. Charlton's first research project, of the five examiners subjected to the biased test, one refused to be affected by the negative bias and instead verified the original identification. Why was that examiner stronger than the others?

It might also be noted that in a concurrent thread on this site, allegations have been brought against an expert in Florida with 25 years experience. These allegations involve four or more cases of questionable identifications. If you average potential errors by that one examiner with the error free work of the other examiners in the agency, you come up with an error rate that is neither accurate for the others or for her. The same is true of Dr. Dror's and Mr. Charlton's study. One might say that the study showed an 80% effect of contextual bias, but that is grossly unfair to the examiner who unknowingly stood by his original identification. His rate of effect by bias is 0% in that study, and the rate of bias for at least three of the others in the study was 100%.

"Error rate" of the science, or of bias in verifications, cannot be calculated for large numbers of people, then applied across the board.

[mdavis]

My only concern is that opposing attorneys on both sides in court will twist whatever is written to suit their own purposes. They seem to care not for the actual, scientific conclusions of the research as stated by the researchers, but will reword it as they see fit.

As Oliver Wendell Holmes once said, "This is a court of law, young man, not a court of justice." The courtroom is not purely a test of what is right or true, but which lawyer presents the best argument. Truth helps but does not always win in the face of obfuscation.

[Pat A. Wertheim]

As Oliver Wendell Holmes once said, "This is a court of law, young man, not a court of justice." The courtroom is not purely a test of what is right or true, but which lawyer presents the best argument. Truth helps but does not always win in the face of obfuscation.

Not only "which lawyer presents the best argument," but also which witness can communicate the best. Going to court in possession of the truth is what an ethical examiner does. Presenting that evidence so the judge and jury understand and accept it is what an effective examiner does.

[mdavis]

Good point. I think that, as expert witnesses, we want to assume our testimony is the key to the case. Actually, all we can do is present the facts as we see them as clearly as possible, and leave the court to "do its thing." I seldom hear the results of my testimony, except now, with accreditation requirements, we must solicit feedback from prosecutors on our performance on the witness stand.

[charlton97]

You all raise some valid and logical arguments. I hope that you will submit such arguments to Fingerprint Whorld when our next paper is published at end of month. It is important that these discussions are recorded in a professional peer reviewed journal.

The size of the study is an interesting observation well made. And of course the power of any data is increased through repetition and size of data pool. But experimental methodology can and should be considered important here too. Whether data collection is 'within subject' or 'between subject' will have a great bearing on the power of the data analysis. 'Within subject' is very powerful evidence since the subject is being analysed using stimuli pertinent to the individual over time. This means the results are self calibrated by the individual participants. Making the results more reliable and powerful than would otherwise be the case.

I also think it is worth remembering that while this is an academic study, I, too, am very involved in this research in the same way that John Vanderkolk is with Dr Busey. Glenn Langenburg is also conducting research with Dr Champod in Europe. I think it is fair to say that some within the fingerprint community are proactive in engaging academia and that academia is responsive to engaging the profession. It is also important to note that Simon Cole, Mr Epstein and James starrs are all academics who raise interesting concerns over our profession. We can and should question all research, but we must be careful to listen to the research and comment objectively. I for one actually believe that Simon Cole and others have valuable contributions to make to the way we conduct our business. As a profession we must argue against research, with research of our own, disprove the hypothesis or agree with it. That is how research works. My PhD centres on academic arguments raging in psychology today. I weigh up the opposing discussions, then I formulate a research question I want to answer, then I design an appropriate experiment to answer it.

I will not reply to any other comments using CLPEX myself, but these are just the sort of academic discussions that can and should be had now and in the future.

[Pat A. Wertheim]

Actually, all we can do is present the facts as we see them as clearly as possible, and leave the court to "do its thing."

Well, not exactly. When I got into this business, testimony training consisted of a senior examiner telling me to "just tell the truth." That kind of rubbed me the wrong way. What did you think, I was going to lie? But now that you've told me to "just tell the truth," that would make an honest man out of somebody who would lie under oath in the first place?

But what I have learned is this -- you can tell the truth in a way that nobody gives you any credibility in the first place. And I have seen other police personnel lie and perjure themselves in a way that the court accepted their testimony. So, truly, it is a mistake to think "all we can do is present the facts as we see them as clearly as possible, and leave the court to do its thing."

If you think you would like to be more effective in your court presentation, then let me give you suggestions. The single most important key is simply to become better at public speaking. Here are some ways:

1. Dale Carnegie Course -- not cheap, but very effective, this fourteen week course uses positive feedback to help you improve. Improvement can be very great.

2. Toastmasters Club -- not as expensive as Dale Carnegie, and much more drawn out. Not for the thin skinned, as negative feedback (criticism) is more prominent than positive feedback in my experiences with these clubs. But if you can take constructive criticism, this can be a highly effective way to improve.

3. Courses at the Community College -- public speaking, extemporaneous speaking, debate, even acting classes -- anything that makes you more comfortable and more effective at communicating with and speaking in front of groups of people.

4. Public speaking opportunities -- from "Career Days" at schools to talks to Boy Scouts or Girl Scouts, to talks to civic clubs in your town (they are starving for interesting speakers, and you get a free lunch in return for a twenty minute talk on what you do).

Formal training such as Dale Carnegie, Toastmasters, or college classes will serve you the best, but even a talk to the Brownie Scouts will help you become more comfortable and more effective. And the more you practice, the better you get -- just as with any skill.

Testimony isn't just about "telling the truth," -- it's about telling the truth so those in the jury UNDERSTAND and ACCEPT that truth.

[mdavis]

Well, I took Dale Carnegie right out of my undergraduate years, so I can vouch for the effectiveness of their training, assuming it hasn't changed since the late 60's. And then the patina of gray hair and the confidence (as well as caution) that goes with it does help immensely in front of a jury. And we've all seen green attorneys butcher prosecution and/or defense presentations in the courtroom. Presentation skills and self-confidence do make a difference, unfortunately often when they should or shouldn't. Good point again.

[L.J.Steele]

To the continuing credit of latent print examiners nearly everywhere, when these bad idents are uncovered, the examiners are usually removed from the pool. No one seems to want to give credit to those who are left.

Are they? One of the Boston papers reported that two fingerprint experts reviewed the Cowans materials for the defense and told the defense experts it was a match, not a mis-ID. The names of those defense experts have, to my knowledge, never been disclosed. For all I know, they may still be doing print work. There may have been defense examiners in those Florida cases who missed whatever problems exist. Is there some mechanism to identify them?

Other forensic fields have had folks who've given problem testimony for years, and kept their jobs, or got passed around from agency to agency. Is there something different about print examiners that prevents that from happening? Is there someone at IAI or FBI or some other group to whom questionable situations can be reported and who can provide an objective assessment and discipline?

I'd like to think this is true -- that the bad apples are quickly spotted and removed, but I fear that the existing mechanism are weak and sporadically applied.

[L.J.Steele]

"Error rate" of the science, or of bias in verifications, cannot be calculated for large numbers of people, then applied across the board.

I don't think error rate is as simple as the averages you suggest. I'm not a statistician, but I know there are statistical tools for looking at large amounts of data to come up with the error rates used for things like medical tests. Certainly, there are risks in trying to generalize a study or even a small number of studies, but that doesn't mean the Dror studies don't provide a large flashing yellow "caution" light that the bias problem may be greater than hitherto suspected.