Daubert Debate in MD

[Pat A. Wertheim]

It seems that "error rate" is one of the more important areas of the current Daubert debate. Lisa and I discussed it before, during, and after the debate at the Chesapeake Bay Division conference last Friday. Lisa's position, as Simon Cole's, is that error is error. It doesn't matter the source. If you are the defendant convicted on the basis of a mistaken fingerprint identification, it makes little difference whether the mistake resulted from the methodology or the practioner.

I agree that to a wrongly convicted or arrested person, the source of the erroneous identification is irrelevant. Be the error from methodology or practitioner, life in a cell with Bubba is unpleasant.

But from a legal perspective, there is a huge difference. If a methodology is inherently unreliable, then the court wants to know that and may exclude the evidence itself on that ground alone, regardless of the skill of the practitioner. But if the methodology is reliable, the question then becomes whether the practitioner is qualified to present a conclusion in court. You may not be able to say that the error rate of one is exactly X% and the error rate of the other is exactly Y%, but you still must consider them separately.

In a Daubert challenge to any science, the error rate of the methodology is the point at issue. In a Daubert hearing, the SCIENCE is on trial. If the witness who will present the fingerprint at trial is unreliable, then that should be brought to the forefront by the defense attorney on voir dire at the trial itself, not in the Daubert hearing. The two issues are distinctly separate.

There are five areas of expert testimony, as follow:
1. Witness Qualifications
2. Scientific Principles (in fingerprints, permanence and uniqueness)
3. Origin and Chain of Custody of the Evidence
4. Methodology used in the case on trial (ACE-V)
6. The Witness' conclusions

Daubert is concerned with #2 and #4 above. #1 is a completely separate issue.

[Becky]

Who can we contact for a copy of the debate video? LPE's in NC are having a meeting in May and we could benefit from the info.

Thanks!!!

[Charles Parker]

David you are right for a long time they could not figure out how the bumble bee could fly. But the problem was not aerodynamics if I remember right the problem was:

How could that wing size move that body weight on the energy expended.

A Mass to Energy to Wing problem

A group of researchers finally developed the answer. The wing of the bumble bee is attached to the body in a different way than other insects. The muscles and joints rotate and there is like a clutch mechanism that allows the bee to move the wings to fly on very little energy.

So it was not an aerodynamics problem it was technical problem in developing a system that the wings could move efficiently with little energy and sufficiently with enough speed and lift to move the body mass.

Saw it on Discovery a few months back (I think).

[Charles Parker]

David, I found a site. It appears that the flight of the Bumble Bee and not being able to fly is a myth.

Try this site: http://www.sciencenews.org/articles/200 ... thtrek.asp

Another urban legend.

[L.J.Steele]

It seems that "error rate" is one of the more important areas of the current Daubert debate. * * * Lisa's position, as Simon Cole's, is that error is error. It doesn't matter the source.

Not exactly. I can't figure out how one can distinguish between the two. The ACE-V method depends on judgments made by the practitioner. I think one could design tests with a statistically valid sample size to give an indication of the error rate for a typical practitioner doing typical casework. Is that the entire answer? No. But it would give us a rough number.

To be honest, I think this is more molehill than mountain. The fingerprint community doesn't want to accept any number bigger than zero. The critics think zero is implausible and some of the analogies made appealing, but inappropriate. The Court does accept various disciplines with error rates as high as (I think) 5%. I'm fairly confident that if an error rate for fingerprint ID were determined, it would be less than 5%. Admitting to a possibility of error, IMHO, isn't likely to make print IDs inadmissible.

In a Daubert challenge to any science, the error rate of the methodology is the point at issue. In a Daubert hearing, the SCIENCE is on trial.

Again, yes and no. Remember that the resolution of the Plaza II challenge involved a discussion of the practitioner's training and credentials and the agencies testing and policise. Daubert, as applied, seems to be dealing with both the science in general and the application of it in a specific case.

As I said in the debate, I'm willing to grant permanence as a working theory. (Tho are there studies on permanence and Lvl 3?)

I'm willing to grant uniqueness when we are dealing with clear images of relatively large areas of friction ridge skin (say the size of a whole adult fingerprint as found on a typical 10 card). I'm more concerned about uniqueness of small amounts of friction ridge skin such as may be found in a latent. My concerns here are a bundle of issues -- the OIG report on Mayfield's comments on the use of Lvl 3 detail and the need for more study there; clarity of the latent and whether whatever unique features exist can be clearly detected/distinguished, and the question of just how far uniqueness goes -- is every dot or island unique? How small a portion of print would one be willing to identify? (Yes, I know, "it depends".)
[/i]

[Bill]

I am presuming that no one from Chesapeake Bay Division can say for certain when the video will be available. I think it is a bit early to make any assumptions. After all, we are only three days removed from the event.

I am sure the folks who put the enormous effort into the program will work hard to make the DVD available. But, putting on a conference such as this one is very exausting and I would give them some time to rest up and get caught up on the other important things in their lives--like family or case work. I am sure that once it does become available, word will spread quickly.

Patience is a virtue.

Bill

[Pat A. Wertheim]

Hi Lisa, I enjoyed the debate in Maryland and I am enjoying it still.

Here is one reason it matters to me whether the error rate is practitioner or methodology: If the predominant error rate is in the methodology, then it applies across the board to all examiners, myself included. If, on the other hand, the predominant error rate is found in individual practioners, then I do not want to be averaged in with those who make erroneous identifications. I will grant you that even if you lump all error rates into the mix, the grand total is still very, very small. But error rate of practioners cannot be applied across the board as a brush to paint all examiners with. I do not know how to separate the two, either. But I know that it is not right to lump them together, either.

Thank goodness for this: There are very bright researchers working on this problem even as we debate the issue. Before too many more years, we should have a better idea of how to determine and apply error rate. Until then, even lumped together, it does not necessitate a warning that fingerprint identification is an unreliable science.

[RL Tavernaro]

... How small a portion of print would one be willing to identify? (Yes, I know, "it depends".)

I once had a unique confirmation of an identification on one of the smallest areas of friction ridge I have ever identified (approximately 1/8"x1/4"). I testified in the homicide trial, using a charted enlargement demonstrating the precise area on the finger identified. Although the area of friction ridge skin was small, there was a significant amount of both level 2 & 3 clear detail present. This single partial print from the headboard of the victim's bed was the only one identified to the defendant, and was the only evidence tying him to the scene (the body was found buried in the desert months later). Testimony also covered evidence of extensive washing/cleaning of the scene.

At the conclusion of direct examination, the defendent's attorney asked to approach the bench (prior to cross exam). After a brief discussion at bench, the judge excused the jury, announcing that the defendent had changed his plea to guilty.

I was then asked to talk with the defendent, who proceeded to explain that he thought that he was being railroaded by the state regarding a fingerprint identification, since he had been wearing rubber kitchen gloves when he committed the murder, and had been very careful and thorough in cleaning up afterwards, still wearing the gloves. However, on hearing my testimony, and seeing my chart, he recalled that there was a small hole in one finger of the gloves that coincided with the finger and area that I had identified. He could see that the hole must have stretched out just enough to allow a small portion of his finger to be exposed. He congratulated me on my work and that was the end of my day in court!
Regards, RLT

[Patrick Warrick]

...error rate for a typical practitioner doing typical casework.

How the heck are you going to figure that out? Who is going to be the "judge" to determine what a typical practioner is and what a typical case is?

If the idea that friction ridge skin is unique is an accepted theory (which seems to be the stance the judicial community is taking, for now), the ability to individualize between two impressions made by the same area of fiction ridge skin also must be accepted. According to the American Heritage Dictionary of the English Language, to compare is "To examine in order to note the similarities or differences of." Everyone knows what comparison means, the issue is the ability of the individual to do it correctly.

For example; two impressions, one has the friction ridges as black, the other white. One examiner might compare them and erroneously determine that they are from different sources. A second examiner compares them, but has the abilty to correctly take into account the color differences and determines that they are both from the same source. Again, everyone knows what comparison means. It is the ability of the individual to determine whether they are the same, different or inconclusive because they do not have enough information or the right kind of information to tell the difference.

[L.J.Steele]

How the heck are you going to figure that out? Who is going to be the "judge" to determine what a typical practioner is and what a typical case is?

Likely, the fingerprint examiner community, in the persons of the editorial board of JFI or JFS, which would most likely be the ones publishing the results of those studies. Likely you folks again, in the persons of the NIJ grant committes who would be funding the kinds of studies that would be needed to answer this question.

If the idea that friction ridge skin is unique is an accepted theory (which seems to be the stance the judicial community is taking, for now), the ability to individualize between two impressions made by the same area of fiction ridge skin also must be accepted.

This is where I see the big logic leap.

Yes, likely if you could magnifty a clear image enough, you would find uniquness in even a "dot". But typical casework probably doesn't provide latents or exemplars of that detail. Just because something is theoretically unique doesn't mean that one has the tools to accurately measure it 100% of the time.

In general, no one is arguing that accurate matching does happen. The problem is whether there is a rate of error inherent in any procedure that relies on human beings to perceive details and make judgment calls about what any differences between source and exemplar mean.

If the predominant error rate is in the methodology, then it applies across the board to all examiners, myself included. If, on the other hand, the predominant error rate is found in individual practioners, then I do not want to be averaged in with those who make erroneous identifications.

I'm not sure how one can get away from it. If we take the false positive/false negative rate for a medical test, that's weighing the careful lab techs with the careless ones, the careful manufacturers of the tests with the slipshod ones. The number is not an aspersion cast on all examiners, but a recognition that even the most careful person could, rarely, make a mistake.

[Patrick Warrick]

Yes, likely if you could magnifty a clear image enough, you would find uniquness in even a "dot". But typical casework probably doesn't provide latents or exemplars of that detail. Just because something is theoretically unique doesn't mean that one has the tools to accurately measure it 100% of the time.

In general, no one is arguing that accurate matching does happen. The problem is whether there is a rate of error inherent in any procedure that relies on human beings to perceive details and make judgment calls about what any differences between source and exemplar mean.

That is my point. You admit yourself that the impressions are unique, its the tools available for the person to determine their uniqueness. It is the ability of the individual and the error rate of that practioner that is in question. That practioner should recognize if the exemplars or latents do not have enough detail. The practioner should recognize if the sample size is not enough to individualize. The practioner should recognize if there is too much distortion between the samples.

[Pat A. Wertheim]

Well stated, Patrick, and that is my point exactly. In the example David Fairhurst gave, the error rate is in the methodology, independent of the practitioner's skill in performing the test. Therefore, in that situation, the only way to discover anything about error is to tell the jury the error rate of the scientific methodology being presented into evidence. But error rate of an examiner is discoverable under voir dire and is different from one examiner to the next. To calculate that and paint all examiners with the same black brush is NOT science. And to use that one-size fits all error rate to predict the chances a good examiner will make a mistake in a specific case at trial is an even worse application of error rate in science.

[L.J.Steele]

You admit yourself that the impressions are unique

Not exactly. What I said was that I'll grant you uniqueness of a large area of friction ridge skin until proven otherwise. If you magnify an tiny area of skin (a single dot, FREX) enough, I'm wiling to assume you could find uniqueness there -- that doesn't mean that at the typical level of clarity found in latents and exemplars that if that uniqueness exists, that it is discernable in small portions of skin. Someone upthread mentioned a match using an area about 1/4 inch by 1/8 inch. I'm still troubled by the potential that if we took, say a latent of an area 1/8th inch square, that it could seem identical to more than one exemplar at the resolutions you typically work with.

I don't think we can resolve that one, tho, without some actual studies. For me, it is a caution sign if I had a case where the match involved a very small amount of friction ridge skin.

the error rate is in the methodology, independent of the practitioner's skill in performing the test

But, Pat, the individual IS the testing mechanism. That's the problem.

And we are talking about the error rate for ACE-V, so we are talking about errors that get past the examiner AND the verifier. I think that's a slightly different question that errors by an individual examiner.

Let's assume that we were talking about a complex chemical test that involved a long series of steps that had to be done with a great deal of precision in a very specific order and that you only know if it is right when you perform the final step. We have a statistically significant number of trained chemists perform the test and find that they get it right 95% of the time. The problem isn't chemistry as a science or the equipment used. The problem is the method may be too complicated for even a trained person to do right every single time. It is human error, but it is inherent in such a complex system. Would it be wrong to talk about the test having an error rate of 5%?

Let's assume we were able to create an ideal test -- a statistically valid number of latents and exemplars given to, let's say, all the currently practicing IAI certified examiners. The test is designed to reflect typical work and the administrators are asked to slip the problems into the examiner's daily routine. The test questions are also verified only by an IAI certified examiner. We've hypothetically eliminated most of the complaints about using proficiency and certification tests, or testing trainees, or having the test-takers know they are being tested. We count only false positives or false negatives -- an inconclusive result doesn't count as an error.

Assume that the test, scored anonymously, came back with an error rate (false positives and false negatives) of say, 1 in 5,000. Would that be a valid measure of the error rate for ACE-V as applied by IAI certified examiners?

[/quote]

[David Fairhurst]

Charles,

Thanks so much for the link. I found it most interesting and somewhat more relevant to our own field than it might first appear.

I brought up the bumble-bee as an example of how two different methodologies may be employed to achieve the same goal.

My understanding of the bumble-bee paradox was that it doesn't use aerodynamics at all. I thought it relied on the viscosity of the air in much the same way a penguin does when it "flies" underwater. Whether this is true or not I don't know, but I think my point is still valid. There's more than one way to skin a cat/fly/test for pregnancy/identify fingerprints/etc.

The article you found, however, is equally useful in illustrating how simple mathematical models cannot accurately predict real-life scenarios. to quote the article: "The real issue is that there's a crucial difference between a "thing" and a mathematical model of the "thing." "
It might be useful in future when approaching probabilistic models for fingerprints to refer back to this myth.

David

[Patrick Warrick]

Lisa,

The problem is that comparing one unique object to another unique object is not the same thing as doing a bunch of steps in a chemical test and if you do something wrong in the end it doesn't match the correct result.

You are comparing two things, two objects, two impressions to try and determine if they come from a common source. Don't we all as human beings compare things...anything...basically the same way. We look at (gather data from) one object and see if there is corresponding detail (data) in the other object and vice versa. We also make note of differences between the two objects; sometimes the differences matter, sometimes they can be explained and they don't matter.

Your ideal test for IAI certified examiners....I read that and I see a competency test for examiners. You are testing whether the examiner is making an error, its not testing the act of comparison.