Are we scientists?

[Michele]

Pat,
I’m not sure how the word ‘independent’ came into the verification stage. Roy Huber first articulated ACE plus verification and related it to the scientific method. He indicated that verification was the ‘… most reliable form of proof”. Later in his article he states, “… constantly asking “why?” And constantly provokes by challenging “so what”.” Repeating a conclusion may be good, but looking at ‘why’ is even better.

Tuthill’s book uses the word ‘independent’. Along with ‘independent’, Tuthill states that verification “will confirm the opinion”. He states, “We stress independent because we must be careful not to program the verifier to see what we want him to see. We must keep in mind what we know about imaginative perception. If we provide our verifier with marked exhibits then he will be programmed to see characteristics at the points we have marked. (Here’s the good part) We simply provide him with the exhibits, tell him we have arrived at an opinion of individualization with respect to those exhibits and ask him to confirm or verify that our opinion is correct.” (There’s no biasing effects there).

It seems as though Tuthill was using the term ‘independent’ to diminish bias but still left biasing effects in.

I would agree with Huber (and how you stated it) that verification should be used to question the conclusion. You can’t do that without knowing how someone arrived at their conclusion.

It’s the same as in a math class, you may have the correct answer but if you didn’t show your work, or if your work is wrong then you still get marked down for it. That’s because this is a basic element of science.

Just because we’ve been told that ACE-V (with verification as replicating the conclusion) is a scientific method, doesn’t mean that it truly is. People used to think the world was flat. Others agreed. Having others arrive at the same conclusion doesn’t make something scientific, having good reasoning behind your conclusions does (which I think most examiners usually have).

Notice that the SWGFAST definition of verification now includes that it is also a review of the initial conclusion.

[Pat A. Wertheim]

Okay, Michele, let me run this by you.

As I understand it, in the Netherlands, an identification with twelve or more points is presented to another examiner for verification in the way we would normally seek verification here. But if the number of points falls below twelve, it is considered a "Questioned Identification" and is subjected to a more stringent review. A group of three other examiners convenes and goes through the comparison point by point, discussing it as they go and each justifying their observations and conclusions. Only if the committee of three reviewers agree unanimously that the identification is correct does it pass verification and get reported as an identification. If any one of the three does not agree, the verification fails and the comparison is reported as inconclusive. Each of the three, in effect, have absolute veto power. That, then, would be much better science if I understand you correctly in applying the concept of justifying your conclusion.

[Michele]

I'd agree, it's better, but might not be perfect if it's not used correctly.

It's not only about the vote, it's about having good justification. If they are all in favor of making an ID but they're all using characteristics that are very subjective, then it weakens it as a scientific conclusion. If only one person has good justification then they should be able to make the others understand why it's good justification and then they'd agree. It's not about convincing others, it's about educating them (for lack of a better word).

[Pat A. Wertheim]

Okay, then. So can I infer from this that in an "easy" ident, the normal comparison/verification scheme by an expert technician could be considered reliable or accurate? But with a comparison/conclusion that is less "easy," a stricter protocol for "verification" would assure reliability through "justification?"

This plays into Lisa Steele's occasional comment that she is not worried about indentifications involving clear latent prints with lots of points. As an appellate defense attorney, she worries about the distorted smudges with a minimal number of points that might be considered "borderline."

I guess I'm asking if we should employ the stricted justification protocol for all cases? Or if the conclusion is self-evident, is a simple verification still "scientific?"

Or am I just getting confused?

[Gerald Clough]

I don’t think conclusions have to be scientific, but I do think that following scientific protocols can improve the quality of conclusions… even if they are already very accurate.

We would have to say what we mean by an improvement. The errors that have the most to say about the examination process itself tend to be about "improvement" in the form of doing more with less. That relates to:

Several people have asked me, don’t you believe that some people have better abilities than others. Of course I do, but science doesn’t value people with extreme abilities.

We have to be cautious about the limits of "ability." It often comes down to a proposed ability to conclude or discriminate from less clear or sparse data on account of experience. Of course, another caution should prompt one to ask if the analyst is working with the most that can be had from the subject of the analysis and is not substituting an imagined ability to work with lesser data they see as enough, when it could, with some effort, be better. An example is the amount of data mined from the Mayfield latent by the OIG. But the more common situation is that it isn't so simple as just doing great enlargements of individual details.

I think the value in protocols is not so much that they insure any particular quality in themselves (although they're protocols exactly because they seemed to maintain quality) but that they fill in a potential unknown, which is how the analysis was done. We don't report with anything like the detail of something like a published study. We don't need to, but when established protocols are followed, anyone looking at the results has more information about how it was done.

I don't know that science doesn't value extreme abilities. A significant body of knowledge has been gained by those abilities to observe and see something that others have not seen. But science does look very closely and critically indeed at what those folks produce. In science, you don't normally get a pass on rigor on account of being highly regarded or very experienced. And in science, most theorists know when they are claiming some special or new insight and anticipate the challenges by being particularly rigorous in their justifications. We don't commonly report that.

You know, if you imagine that latent print examination had evolved to a standard of delivering some sorts of graded likelihoods, rather than nothing short of a statement of belief in the identity of the actual source, we might have a better grasp of this kind of issue. If examiners had always reported from a continuum of likelihood from Extremely Unlikely to Extremely Likely or anywhere between, a great many conclusions would have been argued. The "hard ones," those that required fine judgments and which would confound an inexperienced examiner, would most often be subject to challenge in which those with different positions would have to state and demonstrate exactly what they believed they were seeing and what it meant. Since, when you look at the interesting errors (meaning non-fraud), it's notable that once the error is revealed, just about everyone says they can see that it's an error, and they often wonder how the original examiner could make such a mistake. Very few cases are subjected to more than routine review. It generally doesn't happen unless something forces the intense review.

Now, most of the day-to-day cases where the source is identified are not so hard, and aside from higher level philosophical argument, it's hard to doubt that pretty much any examiner who looked at the case would agree with the identification, even if put on high alert to potential error, or at least agree that the examination was properly done and reasonably concluded. Part of our difficulty in talking about improvement is that we don't know much about errors that weren't revealed, either frank errors of identity or correct identification concluded improperly. So what would improvement be? If it's strictly about eliminating error, we could just independently identify the "hard ones" and not analyze them. That would, of course, throw out some valuable evidence. We could, perhaps, identify the "hard ones" and flag them for very rigorous examination and review. Does the frequency of revealed error justify the cost?

On the other hand, given that so far as we know, erroneous identifications are very rare, it may be that the discipline as it now concludes is at about the limit of its potential reliability, even with the flawed and fragmented qualifications standards, and the appropriate response lies with the justice system to make expert review more accessible. And it may be an economic and practical reality that the justice system is already at its limits in that regard.

[Pat A. Wertheim]

. . . given that so far as we know, erroneous identifications are very rare, it may be that the discipline as it now concludes is at about the limit of its potential reliability, even with the flawed and fragmented qualifications standards, and the appropriate response lies with the justice system to make expert review more accessible.

Gerald,

As I read this, you are suggesting that we may already be doing things well enough that no more rigorous scientific protocols need be written into our policies and procedures at the individual lab level. What is needed is availability of a sufficient number of experts for the defense to review fingerprint evidence. That would provide a level of scientific scruitiny and add reliability.

As I recall, the NAS Report discussed the imbalance of virtually unlimited resources for forensic examinations to police and prosecutors, but very limited access to similar services for defendants. Speaking strictly of criminal cases here, not civil, would providing sufficient experts for defense review in all fields of forensic science introduce both more science and reliability/accuracy to the criminal justice process?

[Gerald Clough]

Pat,

Are we doing things well enough? I would say yes, that we're more than adequately reliable. But as always, we are interested in doing better, both doing better with what we have right now and freely exploring other ways of working. We (and many forensic disciplines) must bear an ethical burden in light of the fact that the reality is that official examiners are so often the first, last, and only word. The minimum responsibility is to do all we can practically do to produce the most valid and reliable products. Out of the whole range of expertise available at law, I think we have one of the best positions to produce results without bias regarding the issue at trial, even though there is always the possibility for bad acts. And we have a pretty good history to build on. While we can be better, I think we're overall at a high enough point on what we might call a reliability scale that getting "better" doesn't mean dramatically better.

Of course, we all know the government resources are not "virtually unlimited." We could easily think of things that would likely work to eliminate some few of the few known errors, but the resources are not so unlimited that we wouldn't be talking about serious balancing. As to the specifics, I don't think it's at all a matter of some kind of shortage of experts. On the defense side, it doesn't take all that much to know when you might want to have your own expert look at something (assuming you're aware enough to even think about it, which is decidedly not our problem) and when you can reasonably judge you don't. Even if you were obsessively careful and wanted every print case looked at, it wouldn't take much to screen them for potential. For typical expert fees, I'm pretty sure a sufficient number of qualified independent experts would be available. But courts are just like every other entity. They have budgets.

In a perfect world, any time a case involves expert analysis of evidence, defense would have access to independent review. Would that inject more science? Not in the way many would think of. But perhaps in admissibility arguments where regular call for argument stimulates scientific inquiry, just as the federal effort is today stimulating research. Perhaps not. Those admissibility arguments seem pretty much formula, aren't really argued very often, and have pretty foregone outcomes. In plain issues of "did the examiner get it right," no; I don't think it would somehow become more "scientific." It certainly hasn't worked that way in the fields where opposing experts always appear. It does, however make you think pretty hard if you know there will likely be a challenge.

Better reliability or accuracy? I'm not sure how I would tell if accuracy was being improved or not. It's opinion evidence. In opinion evidence, for all that each expert believes wholeheartedly in his own belief, there is still a whole range of degrees to which others may or may not form the same belief, or reject that belief, or form a different belief. If you look at any revealed error case and can see where the error was made, you would presumably testify to that as an expert. You, of course, have done that. Had there not been resources to hire opposing expertise in your defense cases, how would they have come out? Would the error have been revealed in any way that mattered? Was the ultimate outcome more accurate with the adversarial expertise? Arguably, it was infinitely more accurate. But we just don't know how many of those cases should have had that advantage and didn't, exactly because we expose so few to adversarial expertise. I still think it's very, very few that would be substantially affected, a tiny number compared to even the relative few wrongly convicted by lay human witnesses. (Eyewitness evidence might win the grand prize for frank wrong convictions.)

I think it's undeniable that few if any forms of evidence requiring expert analysis are ever represented in their most full and accurate meaning and given their most appropriate weight when only one analytic source is heard. Even when the court itself sponsors the expert, there's no critical review. We can't argue against that proposition by assuming it would induce less accurate results, not so long as you render opinion evidence, which is all we have.

But this is pretty much a pipe dream. So is much of the NAS wish list. Would latent units independent of law enforcement change things? Not really. Not in the way we're talking about here. Will more research make things "better"? Sure, but it can't fix plain old error. Better training? Nope. Who are some of the folks who made spectacular errors? How would you have trained them differently? There's a pervasive attitude today that, if something bad happens, there just must be someone to blame, and there just must be a way to make it perfect. A justice system with absolutely correct outcomes. Wars in which no one has to die. Non-violent responses that always neutralize an enemy. Hurling two-ton metal machines down the road with no one getting killed. Oils wells that never leak. Preachers who never diddle. None of them are possible. All you can is do the best you can, work to do better, and blame the media.

[Dennis Degler]

" . . . Paul, thou art beside thyself; much learning doth make thee mad" Acts 26:24

[Pat A. Wertheim]

" . . . Paul, thou art beside thyself; much learning doth make thee mad" Acts 26:24

Okay, Preacher, Gerald made reference to you in the third from last sentence of his latest post.

[Gerald Clough]

"Though this be madness, yet there is method in 't.". - Shakespeare to the assembled peanut gallery, via Polonius to Hamlet.

[David Fairhurst]

Sorry for the jump back in time but I'm returning from a one day absence, exercising my democratic right, to stay up all night watching election coverage, going to bed for 3 hours and waking up still unsure what the result is. With some voters unable to vote because they turned up late in the evening to see 2 hour queues and being locked out at 10pm when polling closed, it looks like were in for our own versions of "Florida Recounts".

Any way, to get to the point,

I would have asked the surgeon if he was a scientist..., but if he would have said 'yes', then I would not let him operate on me! As many scientist have great understanding, but are often unable to 'do' much with it; they know 'that', but do not know 'how' (to borrow from Ryle, G. (1949) "The Concept of Mind"), performance). It is a very complex issues. Most important, is that your surgeon, I hope, did a good job and you are better.

As one of my old University teachers used to recite to us:

A Physician knows everything but can do nothing.
A Surgeon knows nothing but can do everything.
A General Practitioner knows nothing and can do nothing.
A Pathologist knows everything and can do everything; but by the time he gets to the patient it's too bloody late!

We have equivalents of all four of these disicplines in our field too; with all the associated benefits and drawbacks.

[Dr. Dror]

Several people have asked me, don’t you believe that some people have better abilities than others.

Well, yes, there are individual differences. Does a latent print examiner need & rely on certain abilities; well, yes, definitely. Just like many other professions, certain abilities characterize certain professions. An interior designer and an accountant depend on different abilities to perform well at their respective jobs. The same goes to most specialized professions, each having their own unique 'Cognitive Profile' (see, e.g., http://CCI-hq.com/Dror_JAP_visual-spati ... pilots.pdf).

What are the abilities (Cognitive Profile) needed to do latent print examination? And how can we measure them? Are important questions. For example, these can be used for better selection of candidates during recruitment. As we speak, I am working with a team of researchers on these very issues. We have been collecting data for about 6 months, and in another year we hope to have scientific answers to these questions, and provide to the community (free of charge) such tests to help recruit those people who have the specific abilities needed to do well as latent print examiners.

Itiel

[Pat A. Wertheim]

Michele, you asked a lot of questions in your original post in this discussion. Most of them have been skipped over. The one thing that seemed to have become a point of focus was "verification." Scientific verification would not be a rubber stamp, it would not be a second examiner trying to confirm the conclusion of the first examiner, and from what I understand here, it would not be "blind." True scientific verification, if I understand you and Gerald correctly, would involve a second examiner trying to disprove the first examiner's conclusion, while the first examiner has to justify his/her conclusion in order to show the second examiner why the conclusion is correct.

If that is the case, I wonder whether we will ever have such an environment in a police managed forensic laboratory where the line between "Scientist" and "Member of the Investigative Team" often get blurred. (Just last week, I was trying to explain something to an officer that he did not want to hear, and his admonition to me was to just quit talking about it and concentrate on "being a member of the investigative team." There it is, one of NAS's biggest criticisms, and even in an ASCLD lab with strict separation to remove that kind of influence, the pressure is still there.) But I question whether a team of LPE's, even removed from the pressure to perform to the satisfaction of the officers, can seriously question each other to justify and prove their conclusions year after year. Over time, I wonder about the same climate developing that allowed the Mayfield error. I realize we are not talking about this type "justification" to a skeptical verifier in all cases. Some latents are so clear and obviously correct identifications that it would be ludicrous to debate the merits of the conclusion. But even in the few that are "borderline," how serious a debate can occur in a lab where people work together all the time and an argument has the potential of creating a hostile working environment?

It seems to me that attention to one of the other NAS criticisms, the lack of adequate resources for defense review of forensic evidence, could provide a viable answer. And no, Gerald, defense attorneys do NOT always have the resources they need. One defense case I am currently working pro bono is a death penalty case going up for appeal. The appellate attorney asked the court to fund a defense expert to help her review the case. The court refused. And this is not a case with a broad variety of other evidence -- only the fingerprints and nothing else, but there are serious questions in my mind already. The truth is that I work more pro bono cases than paid cases. The police and prosecutors do not have to beg for free expert services in any investigations; the defense frequently does. The system is terribly lopsided.

In your perfect world, Gerald, the police examiner would always get it right, but as you point out, that perfect world does not exist. Thankfully, you are also right that the error rate, although unknown exactly, is exceedingly low. But as Michele asks, might we not be able to make it even lower with better science? So again I ask, wouldn't a competent, ethical defense expert who caused the original law enformcement examiner to have to justify his conclusion introduce better science into the situation?

[Michele]

Pat,

In an earlier post you asked about documentation. Sorry it took me so long to get back to you. I agree, with a simple conclusion, very little (if any) documentation is necessary (unless someone asks for it, then the examiner should be about to create the supporting documentation).

Regarding whether or not verifiers can seriously question the examiner without creating a hostile work environment, I believe our office has created this but it would be nice to hear from some of the examiners in our lab. When a new examiner in our office has their work questioned (and this could be an experienced examiner from another lab), people get their feelings hurt. Then when they are verifying that persons work, they try very hard to find something wrong with it. When they find something wrong or an additional ID, it’s like they’ve won a prize. They’ve proven that they are qualified enough to do the task at hand. In our office, the verifier isn’t an administrative rubber stamp, the verifier has the utmost authority. Their task is to find anything and everything wrong in the case. It doesn’t take long before they relish this responsibility.

Our office has been asked by defense attorneys to review the work of other office too. At first the other office was pissed (to say the least). An internal investigation was filed against us. The other agency, our own prosecutors, and our legal unit were against us doing this. We explained that we work for everyone (the prosecution, the defense, and ultimately the citizens). If something was done wrong in this case, it’s better to find out now. If nothing is wrong in the case then there should be nothing to fear. Once our chain of command agreed with this (yet still leery), they then argued that it would open the door to everyone asking us for reviews. We doubted that would happen and in fact it never did. In the end, the internal investigation was dropped. I should add that even thought the other agency didn’t appreciate us reviewing these cases, the specific examiners had no problem with it. They were glad to know that if the defense was going to have their case reviewed, that it was being done by someone qualified to do it.

An additional benefit came from this. Our examiners can also testify that they don't work for the prosecution, we do work for anyone that asks.

[Boyd Baumgartner]

I would agree, Michele. I think the office has a healthy culture when it comes to verification. I think it stems from these factors:

Education: We've all had exposure to your Scientific Methodology class which teaches Objective Justification, peer review as scrutiny and the heavy influence of Popper's Critical Rationalism (aka Falsifiability) in both the original examination AND the verification. We're aware that the OIG report in Mayfield cited office culture as a contributor. We stay up to date on current events, and hold weekly discussions that critically dissect industry related papers, even ones that show fingerprints in a favorable light. We KNOW how to think critically.

Policy/Accountability: Our SOP's hold the verifier just as responsible for any mistake as the original examiner (assuming its verified). CYA.

Practice and Theory: Practically speaking, the theory of justification must be to the satisfaction of consensus. If I don't think someone else can see what I'm seeing, it creates a feedback loop to make me question the objectivity of my observations. 'Pushing the envelope' is not encouraged. An overabundance of rationalized detail is not considered objective, even if it's justified. We're conservative in our conclusions. Inconclusive: Consistent with finger #x but not enough Quality or Quantity of information to individualize is a conclusion you might see.

QA: We utilize consultation and blind testing.

Documentation/Transparency: We've gotten pretty good at documenting the basis for our conclusions, zoning areas of less reliability and being transparent about what we see.

Humility: Everyone has their domain of talent in the office, but there are no 'Superstars' and egos are in check.

Culture: People accept that this is the way we do things. There is no shame in being scrutinized, it only contributes to the integrity of the conclusion.

Motivation: We aren't perfect, but we're introspective enough to learn from our shortcomings and are always looking to improve.

The bottom line is that ACE-V doesn't automagically make good conclusions. It is only as good as the office and examiners employing it. Failure to question the original examiner can have far greater consequences than offending someone's sensibilities.