Are we scientists?

[Gerald Clough]

[Qualification as an expert is usually not an issue. Few litigants will try to put up an unqualified expert.

I'd qualify this to most litigants won't try to put up an expert they do not believe is qualified -- and there are some ethics rules about candor to the court that ought to preclude putting up a known fraud. However, one of the problems here is that there are some frauds and charlitans out there (Zain, anyone?) who talk a good game and who apparently no one looked into deeply enough to catch them at an early stage.

Yeah. There's that. It sometimes takes a fair amount of specialized knowledge to evaluate their qualifications. Which is part of why the same experts appear again and again, once they've passed the opposition sniff test a few times.

[Dr. Dror]

I think there are a few very different issues here. Who is a scientist is a different question/issue from who & how the courts/judges view & determine anyone as a scientist. I truly understand that the pragmatics of what the court/judge decide is very important in a practical way, but that is different from who a scientist really is (I can see the court accepting someone as a scientist, who is not; and visa versa). That is especially true since there isn't an archetypal scientist for the court to use as a model for such a decision, and the judges do not have the tools, or knowledge to make such a decision. So, what do they rely on? Things like qualifications, degrees, etc. That does not guarantee anything; as there are people with Ph.D.s who are not scientists; and visa versa, good scientists who have no formal qualifications or degrees. Ideally, a judgement on who is a scientist will be based on their abilities, knowledge, etc.,; but without an easy & clear way to do that, maybe checking their qualification and degrees is pragmatically and practically the best way to do it, but it is far from perfect. These, however, are practical (as important as they are) issues of what & how the court decides such matters; they are pragmatic and not focusing on the issue of what really makes anyone a scientist. They even have a danger that we devote too much thinking and time on of how to 'look like' scientists, to satisfy the court, rather than focus on being scientists.

There is also a difference between being an expert and being a scientist, and the discussion in this thread has shifted, and swings between them.

Scientist, or expert, even if someone is one of these, then are they good, can they be trusted? And how can we determine this?

To really ascertain if someone is a scientist (or an expert), and a good one at that, which can be trusted, I need to be a scientist/expert myself...! (which defeats the entire purpose of needing one to begin with). So, if I am not a scientist/expert myself, I need to trust some formal pieces of papers, be it a degree, a proficiency test, etc. (but we all know the limitations of these; e.g., see point 1, above).

The original posting by Michele was spot on, as it focused on specific issues, content issues, that relate to how we go about our work (rather than who gets the label of 'scientist', or/and what the court/judge decides). I think the field can move forward by focusing on the issues (see my previous posing too) of how to be better at our work.

One of the issues that Michele raised and was discussed in this thread is 'objectivity’, which I think is central and critical for being a scientist (and an expert). And the question is how do we ensure that we are as objective as possible.

A clear characteristic of being a scientist is to continuously question things, trying to be more objective, reflecting on your own work, thinking & exploring ways to improve. And Michele posting/questions are a clear sign of that!

[Ann Horsman]

Skill- an art. Practice, study, questions, testing- gets conclusions. I look at myself as a latent print analyst, I have training, testing and years of experience, I conduct experiments, I research and read technical papers and articles. I speak with other practitioners, etc, etc, etc.

Forensics is a science in my mind. I don't see that it matters that I am a scientist or not, it matters that I have the skill and knowlege to do the job at hand.

I recently had to have my knee replaced. I didn't ask my surgeon if he was a scientist. I wanted to know if he was trained and had the experience to do the job. I did not ask what his error rate was either.

[Pat A. Wertheim]

There is also a difference between being an expert and being a scientist, and the discussion in this thread has shifted, and swings between them.

Dr. Dror, that is an obvious distinction that we have not specifically discussed in this thread. I see your point exactly. If the examiner/witness is truly and expert in the subject matter, what does it matter whether he or she can be deemed a "scientist?"

I recently had to have my knee replaced. I didn't ask my surgeon if he was a scientist. I wanted to know if he was trained and had the experience to do the job. I did not ask what his error rate was either.

This seems to be what Ann is saying, too. You wanted to know if the surgeon was an expert, not whether he was a scientist or was proceeding with a scientific protocol during the surgery. He was an expert at knee replacement surgery and that was the thing that mattered.

Interestingly, when I got into this business in the 1970s, I was warned by old timers in the field to never admit on the witness stand to being an "expert." The idea was that once you had made that admission, you could be questioned from any text on any topic relating to fingerprints and the defense attorney could keep you there for days and if you missed a single answer, you would leave the courtroom in shame. In the mid 1980s, the question was put to me, "If you are NOT an expert in fingerprints, what the hell are you doing giving evidence about it in court?" That question made a lot more sense than the original admonition to never claim I was an expert.

In the mid 1990s, I began to have conversations with some of the senior examiners in Great Britain. Interestingly, they shunned the term "scientist" and many felt it was an insult. Rather, they embraced the label of "expert."

[Dr. Dror]

I recently had to have my knee replaced. I didn't ask my surgeon if he was a scientist. I wanted to know if he was trained and had the experience to do the job. I did not ask what his error rate was either.

Ann, I hope your knee replacement went well and you are all better. I would have asked the surgeon if he was a scientist..., but if he would have said 'yes', then I would not let him operate on me! As many scientist have great understanding, but are often unable to 'do' much with it; they know 'that', but do not know 'how' (to borrow from Ryle, G. (1949) "The Concept of Mind"), or what we cognitive people like to call 'declarative knowledge' vs. 'procedural knowledge' (I have a whole paper on expertise in knowing 'that' vs. expertise in knowing 'how' --if you, or anyone else, are interested, drop me an e-mail, i.dror@ucl.ac.uk ,and I will happily send it to you).

But I wonder why you did not want to know his 'error rate'. Imagine that one surgeon has an error rate of 5% (95% of the patients that s/he perform knee replacements on, totally recover within 3 months), vs. another surgeon who has an error rate of 50% (50% of his patients, end up having their leg amputate). Wouldn't you want to know that? Or even at a general level, would you not want to know that on average X% totally recover, while Y% are much worse off? Then you could make educated, data based, decisions, etc. There has been much debate in the medical domain about having such 'error rates' available (from % of C-sections performed, to individual surgeons' performance). It is a very complex issues. Most important, is that your surgeon, I hope, did a good job and you are better.

Itiel

[Gerald Clough]

There is also a difference between being an expert and being a scientist, and the discussion in this thread has shifted, and swings between them.

Dr. Dror, that is an obvious distinction that we have not specifically discussed in this thread. I see your point exactly. If the examiner/witness is truly and expert in the subject matter, what does it matter whether he or she can be deemed a "scientist?"

Then what is important?

The practical legal position has to be seen from the broad perspective of testimony from special knowledge. Remember that the more recent defining cases were civil, pointing up that there's no fundamental difference in expertise issues in criminal cases. A licensed electrical contractor, a class of expert most unlikely to ever consider invoking "science," is a qualified expert witness. Would an electrical engineer be a better expert witness? It is unlikely that the engineer would be called upon to testify to more than the electrician. The engineer could, of course, testify to the actions of electrons and inductive fields, which the electrician could not, but eyes throughout the courtroom would begin to glaze over. However, in a contested issue with an electrician and an engineer on opposing side offering materially different opinions, the engineer has a distinct edge in credibility. Now, imagine the same situation, except with a physicist substituted for the engineer. The physicist possesses and can testify to exactly the same principles, the same facts about things like how voltage divides across loads, as the electrician and the engineer. But the physicist has never designed or installed an electrical application. I think it's reasonable to perhaps doubt if the physicist would ever be chosen as an appropriate expert, even though he possesses the same material knowledge.

So, the physicist, unquestionably a scientist by any definition and possessed of a doctoral degree, is of questionable value.
The engineer, a product of science education who has at least replicated the classic experiments upon which his knowledge is grounded and possessed of official engineering credentials, is fully versed in the practical applications and in fact is a person allowed to design and certify the most critical applications.
The electrician, a product of apprenticeship and practical training, possessed of a license to practice and not expected to be in any way a scientist, is entrusted with basic and standard design and with physically installing systems.

Are all three experts? Certainly. Are all three qualified to render opinions? Presuming for the sake of this discussion that the issue falls within their expertise, yes. I would judge the engineer the most persuasive expert. Why is that?

I think the factor that is far more influential than the aura of "science" is what might be called professional status. Engineers are recognized as "professionals," along with physicians, attorneys and clinical psychologists who are required to have completed rigorous educational requirements (engineering program, medical school, law school, PhD in psychology), are required to pass rigorous licensing qualifications, usually serious testing, are commonly required to attend some minimum of CE, maintain membership and some participation in respected professional organizations, and work in a professional setting. And I might add make or are imagined to commonly make substantial income and frequently practice as the principal in their own enterprise. The last one is important. That is that they practice and are experienced practitioners. They are not merely theorists or pure researchers.

If we're talking about how an examiner is viewed by the only people who matter, the fact finders, what tends to make them view the examiner as a "professional" and therefore relatively more credible? It's not "science."

[Michele]

Gerald,

I can see where you’d say that it only matters to the fact finder (the judge) but I think it matters to more people as well.

I recently know of a case where the technician’s evidence (not fingerprint related) was omitted because he wasn’t considered an expert. He’d been doing his job for probably over 20 years but that didn’t make him an expert. Even though he was prepared to testify to what he did in the case, he couldn’t explain why he did it and he may not have been using best practices. For the evidence to be admitted, I’d agree that it’s the judge’s decision, but whether or not a person is the expert that they claim to be is important to other people as well. This person’s primary job function was for crime scene investigation. If they aren’t considered an expert then what they do might have very little importance to the DA, making their job function unnecessary.

In our discipline, we’ve represented ourselves as a science. The NAS found that many disciplines make this claim yet don’t qualify as a science. If a discipline doesn’t qualify as a science then it may not be eligible for funding that is designed for sciences. Our discipline in particular was shown to be making claims outside of the realm of science (absolute conclusions and zero error rates).

Initially, a person’s qualifications matter to the judge but the ripple effect can be extreme. Whether or not a person qualifies as how they are representing themselves matters to the judge, to them and their career, to their agency, and to their discipline.

Instead of asking if we are scientists, perhaps a better question is “do we adhere to scientific protocols”? Many would say yes but some of the people who are saying this are still testifying to absolute conclusions. This makes me wonder if people understand what scientific protocols are.

[Michele]

I should have added that I think this is a good time to review ACE-V and if it’s as scientific as we say it is. Years ago David Ashbaugh stated, “Verification is a form of peer review and is part of most sciences. Many organizations erroneously use verification as a method of protecting against errors in place of adequate training. While verification may prevent the occasional error, its purpose is to verify process and objectivity as opposed to only check results. It is also an excellent vehicle for training.”

Many of us say that the verification process is a repeat of the ACE process. This is different than peer review. If we are using it as a repeat of ACE, does it still qualify as a scientific method?

[printlady]

Hmmmm… scientist, expert, analyst, technician, operator, examiner; all of these could apply. So which one do we choose to describe ourselves? Is one better than the other? Can we use the terms interchangeably? There is a difference in how the courts view us versus the public or our critics but should that be the determining factor?

[Pat A. Wertheim]

Verification is a form of peer review and is part of most sciences. Many organizations erroneously use verification as a method of protecting against errors in place of adequate training. While verification may prevent the occasional error, its purpose is to verify process and objectivity as opposed to only check results.

Is Ashbaugh saying that it is less important to catch errors than it is to just repeat the process? If that is the case, then one might conclude that verification worked the way it was supposed to in the Brandon Mayfield case, the Shirley McKie case, the David Asbury case, the Stephan Cowans case, etc.

[Ann Horsman]

I recently had to have my knee replaced. I didn't ask my surgeon if he was a scientist. I wanted to know if he was trained and had the experience to do the job. I did not ask what his error rate was either.

Ann, I hope your knee replacement went well and you are all better. I would have asked the surgeon if he was a scientist..., but if he would have said 'yes', then I would not let him operate on me! As many scientist have great understanding, but are often unable to 'do' much with it; they know 'that', but do not know 'how' (to borrow from Ryle, G. (1949) "The Concept of Mind"), or what we cognitive people like to call 'declarative knowledge' vs. 'procedural knowledge' (I have a whole paper on expertise in knowing 'that' vs. expertise in knowing 'how' --if you, or anyone else, are interested, drop me an e-mail, i.dror@ucl.ac.uk ,and I will happily send it to you).

But I wonder why you did not want to know his 'error rate'. Imagine that one surgeon has an error rate of 5% (95% of the patients that s/he perform knee replacements on, totally recover within 3 months), vs. another surgeon who has an error rate of 50% (50% of his patients, end up having their leg amputate). Wouldn't you want to know that? Or even at a general level, would you not want to know that on average X% totally recover, while Y% are much worse off? Then you could make educated, data based, decisions, etc. There has been much debate in the medical domain about having such 'error rates' available (from % of C-sections performed, to individual surgeons' performance). It is a very complex issues. Most important, is that your surgeon, I hope, did a good job and you are better.

Itiel

Dr. Dror,

I was being facetious, sarcastic, because there is a potential for error in EVERYTHING. OK, that's my humble opinion. But we are issuing opinions too when we analyze a latent print and make a conclusion. Or maybe better written as an expert opinion. We, as LP examiners, are bound by the 'is there enough information' question to make identifications. We are also bound by our skill level and what technology we possess. I certainly feel we are far above technician status. I also note that like a well known surgeon, LPE's also carry reputations which influnce courts. I sat in a court room yesterday to listen to the defense stipulate to my report because the judge read off my qualifications and that I have testified in courts at least 20 times....

I used my surgeon as an example because it is his skill which is most important, followed by the technology to achieve a successful outcome.

We (LPE's) do not need to be scientists. ACE-V is a scientific method, using it does not make one a scientist.

Off topic: Two reasons I did not pester my surgeon on his error rate: One- He's sliced and diced several co-workers prior to me so he has a reputation, the other is, I am not an average patient for him. I am in my forties with destroyed joints (I tend to play hard- go look at my facebook photo) so I will have an error rate in the scheme because I am not 75 years old and playing shuffle board. The surgeon knows I will be back someday because I will wear this thing out. So is that an error? For most people a joint replacement will last a lifetime due to their age, it probably won't be for me. I'd also blow his stats because I was back to work 6 weeks PO and sit here now having already been logging trees on my mountainside property. I am told I will hurt for a year though but I am an optimist and plan to beat that odd too. 8)

[Gerald Clough]

First, I dislike calling it peer review. Sounds like reaching for SCIENCE. Of course, I'm not all that fond of "verification." But, come on. It's all checking for error. If we're still talking science, correct guessing doesn't count. Unless other researchers, using their own or the previous study design, can replicate the results, some one is wrong. If no one can replicate the results, the original inquiry was in error.

This does sound a lot like LPE verification, but it's not. In scientific research, the person experimenting to see if they can replicate the results knows exactly the data interpretations and logical chain of the original work, because they read the paper. (Barring, of course, the unknowing simultaneous duplication of the same study. In that case, if they get different results, they settle it with a knife fight at the next conference.) They may follow the original design to see if the result seems to consistently derive from the elements of the study or if some different result suggests those elements don't account for all factors. Or it may use a new design, either because the current researcher believes there is a better way to study the same phenomenon or because it seems useful to look at it from a different angle.

LPE verification is being handed the same phenomenon and independently mining it for data and engaging in a new study, (which is automatically done with different "instruments," because it's a new observer with different tolerances and experiences), and therefore concluding from what may well be different data given different weight. And there's "blind verification." I wonder if that's well reasoned. I don't note that being a feature in "science." There is blind peer review for publication, but it merely blinds both parties to the identify of the other. Is it more productive to follow the examiner's observed and interpreted data and weights used in concluding or to do it again from scratch? I don't know, but I'm not convinced that blind verification is the chosen method by anything more than guess.

And LPE verification is so different from publication peer review that it's not worth talking about.

An LPE verification is frankly a check to see if, in the opinion of the reviewer, the examiner made a mistake. It's simply taking an additional step out of abundance of caution, and it ought to be able to stand on it own as just that, without any hocus-pocus. Could I make a mistake. Sure, I could. I'm human. That's why I had another examiner do another examination, because this is important enough that I want to catch even a most unlikely mistake.

But read the intent of Ashbaugh's statement. It's cautioning organizations against accepting inadequate training and intending to catch any resulting errors with verification. It's not saying that verification isn't to catch errors, only that you shouldn't have it in place because you expect errors you could have avoided. The only part of the statement I take issue with is imagining that verification checks process and objectivity. It can't do that, because the verifier isn't privy to the first examiner's process and interpretations and therefore can't judge his objectivity. I take this to say that verification should be able to assume the first examiner was adequately trained, that he had the ability to do the examination properly. The verification, then, hopes to imply that he actually did it properly, because the results are the same, and we hope that both examiner and verifier aren't making the same mistakes. It's a mere implication, because replicating the conclusion doesn't raise the conclusion beyond the original confident belief, no matter how many times it's done. (I mean, if you weren't all the way confident to begin with, why did you report it out.) It's the classic medical "second opinion," which quite properly makes the person who must choose more confident if you get the same answer. (Not a perfect analogy, since medical diagnosis is much less reliable than LPE and involves more complicated balances of risk.) There's nothing unscientific about second opinions. It's pretty good science (and anything else) to ask someone if you might have gotten it wrong.

As for ACE-V, poor thing. It's like Siddartha might say. Just tried to set a good example, and now some damn fools are worshiping me.

[Charles Parker]

Pat Stated
[quote]Dr. Dror, that is an obvious distinction that we have not specifically discussed in this thread. I see your point exactly. If the examiner/witness is truly and expert in the subject matter, what does it matter whether he or she can be deemed a "scientist?"[/quote]

Amen Brother---finally someone after five pages hit the nail on the head.

Thanks Pat, Thanks Dr. Dror, Thanks Ann.

Legal Term: Expert Legal Term: Opinion

Become the best you can be---do the best you can do----learn as much as possilbe--keep an open mind----and progress and always take the time to learn something new.

[Michele]

Let’s suppose that scientific methods aren’t used, yet the results of identifications are extremely accurate, maybe an error rate of .000001 (lets make these assumptions just for the sake of argument). Does this mean that our conclusions are so accurate that we don’t need to improve? The few people who have been misidentified would probably hope that we would always look at ways to improve so that what happened to them won’t happen to others.

What measures could we take to prevent errors? I think the best way to determine this would be to look at the past errors and determine what can be improved in each case. The hard part about this is that we might not have enough information about errors to thoroughly analyze them (but we can try).

In the Mayfield case,

1) They rationalized things that didn’t fit even though there was no observable details to support such rationalizations (by claiming the top left was a different impression even though there was no visual clues to make this conclusion). Science values having objective data to support all claims.

2) Verification was used to reproduce the results, it wasn’t used to determine if the examiner was arriving at a valid conclusion. Reproducible results have value in science but it’s not as important as having good (valid) justification behind the conclusion.

3) The examiner and the verifiers didn’t try to disprove the tentative conclusion, they were only trying to confirm it. Science values trying to disprove a conclusion over trying to confirm one.

4) The examiner and the verifiers weren’t impartial – perhaps they were very entrenched in solving a high profile case. Science values impartiality.

5) The justification they gave didn’t diminish doubt in others (such as the Spanish Police). Science values having sufficient evidence to diminish doubt in others.

In the Caldwell case, the ID was made on the original object. Once the latent faded, the justification behind the conclusion couldn’t be explained to others. Again, science values conclusions that can be explained to others (with objective data that others can see).

In the Derris Lewis case, they concluded that the print was left in blood (which changed the probative value of the latent print). The tests to determine this weren't valid tests to arrive at this conclusion. Science insists that the premises used in making conclusions be valid.

Scientific conclusions aren’t absolute or conclusive, they are the most reasonable conclusion. The NAS has recently reiterated this as a scientific criterion.

Several people have asked me, don’t you believe that some people have better abilities than others. Of course I do, but science doesn’t value people with extreme abilities. Science values conclusions that are well justified enough to diminish doubt in others. I think most people are looking for correct conclusions but science isn't looking for the correct conclusion, they are looking for well justified conclusions. Well justified conclusions lead to correct conclusions.

I don’t think conclusions have to be scientific, but I do think that following scientific protocols can improve the quality of conclusions… even if they are already very accurate.

[Pat A. Wertheim]

What measures could we take to prevent errors? I think the best way to determine this would be to look at the past errors and determine what can be improved in each case. The hard part about this is that we might not have enough information about errors to thoroughly analyze them (but we can try).

Good point. A study to determine sources of errors would require a statistically significant number of errors. If the study were to help avoid errors in the future, there would have to be a common cause among errors in the study, maybe not of all of the errors in the study, but of a high enough percentage to draw valid conclusions that point to common causes of errors. Otherwise, every error could be a one off, and nothing that was changed because of a single error could be expected to prevent future errors.

Science values conclusions that are well justified enough to diminish doubt in others. I think most people are looking for correct conclusions but science isn't looking for the correct conclusion, they are looking for well justified conclusions. Well justified conclusions lead to correct conclusions.

If I understand you correctly, Michele, just because, as a fingerprint expert, I make an identification and give it to my lab partner, and she does a separate comparison and verifies the identification, that doesn’t make it good science. I have to consider justifying my conclusion, even as I’m doing the comparison. Arriving at the justification along with the conclusion and using that justification to remove doubt in others helps add to the scientific validity of the conclusion.

But doesn’t that fly in the face of what we have been told all along, that verification should be independent? How can it be an independent verification if I am presenting my justification along with my conclusion for verification?

Or does the idea of justification apply when verification is not used primarily to confirm, but more importantly to question a conclusion, as you suggest here:

Science values trying to disprove a conclusion over trying to confirm one.

So a more scientific scheme of verification would be the situation in which the second examiner questions the conclusion to try and disprove it, while the original examiner actually presents the justification that was formulated during the original comparison?