Daubert Debate in MD

[L.J.Steele]

Just a reminder that Pat W and I will be going head to head on the Daubert issues at the end of March for Chesapeake Bay Division of IAI and we'll likely be talking about Langill as well as the other Daubert cases.

http://www.cbdiai.org/

[Pat A. Wertheim]

And a Good Time will be had by all !!!

Of course, Lisa and I will debate each other on the standard Daubert issues, but let's get some input here, too. What do you, the Readers, think I should stress? What questions should I direct to Lisa? What are some new science or Daubert related questions you have experienced in court? Where do you see the attacks coming from in the next year? What weaknesses in the way we do things should we address?

Lisa and I have been planning this for months, but I would like some direction from the community on where you think this would be the most help to you.

Looking forward to your input, here or at my email foridents@aol.com

Hope to see you there!
Pat Wertheim

[L.J.Steele]

I would like some direction from the community on where you think this would be the most help to you.

Same here -- let me know what questions you'd like me to talk about.

And if there's stuff you'd like me to address (or ask Pat W. about) that you don't want to post to the list, you can email me at:

steelelaw @ earthlink DOT net

(That might slow down the spambots...)

I'll also be talking about amicus briefs and Patterson, so if there's questions you have on that topic, let me know!

[amcdonald]

Is there anyway we can see the transcript or something after this debate for those who cannot attend the CBDIAI conference? I think it would be valuable information for all examiners.

[Cindy Rennie]

Yes, please! I would also love to get a transcript or a report of that debate.

[Red.Sox.Fan]

The debate will be video taped so it will be available for those who cannnot attend the CBD-IAI conference.

[sharon cook]

I would like to see a thorough airing of the error rate issue, in language that us non-legal scholars can understand.

[L.J.Steele]

I would like to see a thorough airing of the error rate issue, in language that us non-legal scholars can understand.

I'll see what we can do, tho it isn't as much one of my pet peeves so I may not focus as much on it.

Off the cuff...
Daubert asks about error rates, so it is something the courts would like counsel to talk about in their argument, thus we ask y'all about it.

We know of about 20 mis-IDs, some of which made it to the Courts. Simon Cole's article, More than Zero, lists those cases. I believe that there are more than 20 mis-IDs in the US and UK in the past 80ish years, but I don't know how many more and it is somewhat hard to figure that out.

Simon notes in a more recent article that he's become aware of a few more mis-IDs, but hasn't been able to get more information on them.

http://www.facesofwrongfulconviction.or ... final1.pdf

And we get into problems about testing -- can you use proficiency tests to show error rates? If not, how would one do it?

The usual answer when lawyers ask y'all about error rate is that the error rate for ACE-V (the method) is zero. Those 20ish cases (and presumably the others too) are all practitioner error. (Like saying that our airplanes are perfect, any crash is a result of pilot error.)

To which some folks say, that can't be right. The method depends on the judgment of the practitioner. You can't seperate the two. The plane doesn't fly itself, it depends on the pilot's skills.

And we then start arguing about analogies -- someone trots out the math example -- an error balancing your check book doesn't show that math is wrong. And the critics say that fingeprint IDs are more like trying to balance your checkbook using some receipts that have been in your pocket for a couple of weeks, then thru the wash, and you have to make a decision about what the figures are. The procedure of balancing your checkbook depends on an operation (math), but also subjective judgments about what the receipts say.

And it is hard to find other examples in the sciences of a procedure that depends on human judgment that makes the claim that you can have a practitioner error rate seperate from a method error rate.

IMHO, this may be somewhat irrelevant to a judge. The judge needs to know that the method isn't foolproof. Mistakes happen. We don't know how often they happen. I think even the critics would agree that they are likely rare. But we don't yet have a good system for telling if this is one of those cases where there's a high risk of a mistake. And that's were the problem lies.

Does that help?

[sharon cook]

Airplanes are subject to structural fatigue, whereas the SCIENCE of fingerprints is not. The science has a zero error rate. The mistakes ARE all practitioner error, and we don't know how to measure the frequency of that with any kind of accuracy.

[RL Tavernaro]

Lisa,
Thank you for your post responding to Sharon's question. I found it to be very helpful; about as concise and insightful, as well as fairly representing both sides of the matter, as I have seen.
Regards, RLT

[Michele]

I know this sounds like a naïve question but what is the justification behind saying that the error rate is zero? Is it the same to say that ‘the error rate of our methodology is zero’ and ‘the error rate of the science is zero’?

I’ve heard people say that no 2 fingerprints are alike therefore the inability to recognize the uniqueness must be that of the practitioner.

The problem I have with this line of reasoning is there seems to be a leap in logic. The above statement isn’t even stating what the methodology is so how can this determine that the error rate of the methodology is zero? I would agree that through studying embryonic development scientists have determined that no two fingerprints can be the same but this seems to be stating that the theory of differential growth is flawless, it doesn’t say anything about the entire science, the methodology, or the practitioner’s abilities.

I’ve never done an error rate study, so maybe I’m just ignorant about what’s needed, but I’m guessing that in order to look at the error rate of the methodology we need to look at 4 things. First, we need to determine what we are considering an error to be (clerical errors, false positives, false negatives, all of the above?). Second, how are errors determined considering that we usually don’t have the ground truth? I’m guessing that some standard needs to be decided. Third, we also need to describe the methodology we are using to arrive at our conclusions (guessing, flipping a coin, counting points, using ridgeology, using the constellation method-some people call this the pin-cushion method, using ace-v, or relying on AFIS technology with a lights-out approach). I’m guessing that each of these methods would produce a different error rate but at this time it would be hard to determine the error rate of each method without asking practitioners what method they are using. And fourth, each method might have a different practitioner error rate associated with it. We might not only want to ask which method a practitioner is using but also have the practitioner ‘show their work’ so we can see if the error was an error in the method or a practitioner error.

All the error rate studies I can think of have produced statistical numbers but since the above questions weren’t asked we can’t determine if the error rate was from the method or the practitioner. Then again, does it really matter where the errors are from? Even though errors are rare (as Lisa stated), I think it's important to find out when and why errors occur so we know when they're more likely to happen. If we have this information then we can try to prevent them from happening again.

Wow, I’ve strayed pretty far from my original question so I’ll restate it: What is the justification behind stating that the error rate of the science (or methodology) is zero?

[David Fairhurst]

We can say that if friction ridge skin is unique that there will be no errors caused by a methodology that uses this as a premise.
However we cannot say this for the other two categories of error; caused by apparatus failure and operator error.
Our problem is that the apparatus and the operator are inseperable.

To use the ususal analogy of the pregnancy test.
Premise - pregant women will have increased levels of protein in their urine.
Methodology - measure the level of protein in women's urine. If the level is above the predetermined threshold then return a positive result.
False positive error rate = the proportion of non-pregnant women who have protien levels above the threshold
False negative error rate = the proportion of pregant women with protein levels below the threshold
Apparatus failure rate = the proportion of "pee-sticks" that don't work properly
Operator error rate = the proportion of times a woman doesn't pee on a stick correctly

Premise - friction ridge patterns and the details in small areas of friction ridges are unique and never repeated (Ashbaugh)
Methodology - demonstrate the pattern and details in small areas of two friction ridge impressions are the same. If the quantity and quality of the details is above the threshold reoprt an identification.
False positive error rate = the proportion of pairs of impressions from different sources that exceed the threshold
False negative error rate = the proportion of pair of impressions from the same source that do not exceed the threshold
Apparatus failure = the proportion of examiners who cannot examine prints accurately
Operator error rate = the proportion of times an examiner does not examine prints accurately

However we can separate the methodology from the apparatus and operator.

A woman uses brand A pregnancy test kit and gets a positive result.
If the kit was not defective and she did it correctly then repeated tests with brand A will always return a positive result, even if performed by another person. Only by re-testing with brand B (or by waiting nine months to see what happens) can we discover if this was a false positive error.

An examiner uses ACE to compare two friction ridge impressions and reports an identification.
If he is a competent examiner and his methodology is sound and applied correctly then repeated examinations with the same methodology, even by other examiners, will always result in identification. Only by employing another methodology (which we currently don't have) or by proving by other means that the prints come from different sources can we discover this error.

what does this mean?

1. That if an error is discovered, or provable, by re-examination by another examiner using the same methodology then it must have been due to incompetence (apparatus failure) or negligence (operator error).

2. That errors of methodology are undetectable in everyday casework because that requires another methodology or additional information on the source of the prints.

What we need then, is a controlled study, using impressions from known sources, that reports how many pairs of impressions can fool "all of the people, all of the time" for this is the real test of our methodology. Coming up with 20+ errors from live cases that have been discovered through re-examination and cannot be proved by other means, only shows that examiners occasionally make mistakes.

[Charles Parker]

Sharon, here is my take on error rate.

Daubert is concentrated on the methodology. All five Daubert criteria are aimed at the processes behind the methodology and that includes error rate. That is why when asked about error rate you need to separate the error rate of the methodology from the human error.

The critics of fingerprints would like to have the 20 plus talked about as that helps their position. However the critics have yet to say or show which of the 20 plus are attributed to errors of the methodology or human errors. They also like to point out the proficiency errors, but again how many are human errors and how many are attributed to the methodology. That is why we use examples to juries and courts to explain the difference between human error and methodology error.

When a plane goes down there is four possibilities of the reason why:
A. Human error (pilot, ground control, mechanics, inspection)
B. Weather
C. Third party intervention to destroy the aircraft
D. The principles of aerodynamics failed

I have seen the cause of plane crashes from A, B and C but not from D.

Planes do not fly themselves and fingerprints do not ID themselves. Both rely on human judgment.

When you balance your checkbook either from a clean just printed statement or from receipts that have been washed and faded and there is an error, the error is from one of the following reasons:
A. Human error
B. The principles of mathematics have failed.

I have never seen an occurrence where the principles of mathematics have failed.

If working from faded receipts and the number interpreted as a 1 when it is really a 7 that is a human interpretative error and not a problem with mathematics.

When a fingerprint error occurs there are two possibilities for the error.
A. Human interpretative error
B. The principles of random growth and uniqueness have failed.

Every fingerprint error I have seen or read about has been from human interpretative error and not because an area of friction ridge skin was duplicated.

In science error rates are generally attributed to machines and instrumentation. If you get 100 scientists in a room and ask them if they know of any scientific endeavor that is free of error, they will state “NO”. If you ask them if there is a difference between instrumentation error and human error they will say “YES”. If you ask them what is the human error rate of their discipline they will not be able to answer. Human error rates are not tracked in any scientific endeavor. It is how you ask the questions that matter.

If you ask a DNA analyst what is the error rate of their instrumentation they can tell you. If you ask a DNA analyst what is the human error rate of their discipline is they cannot provide you with an answer as they do not track human error rates. If you ask a DNA analyst what is the error rate of the principle of genetic inheritance, they will say it has never been documented.

How the question is asked is very important as to the response that is provided.

IMHO I think judges are quite aware that human beings make mistakes and that no method is foolproof. Daubert is clearly about the methodology and the critics do not like that. They believe the human element should be added to the methodology. However the courts deal with the human element in a different way by allowing defense experts to testify and cross examination of the training, skills and experience of the expert.

Fingerprint identification is a human endeavor and does not rely on instrumentation or machines and the critics do not like that at all. Because fingerprint identification is subjective that is our Achilles Heel. You just have to make sure as humanly possibly that your subjective opinion was arrived at as objectively as possible.

[David Fairhurst]

Charles,

How does your model account for the bumble-bee? A creature that, according to the principles of aerodynamics, should not be able to fly; but it does.

D &:^)

[L.J.Steele]

Is there any discipline that tries to seperate method and practitioner error other than fingerprints in a Daubert, or other context?

I'm just not persuaded that fingerprints are similar to mathematics. The nature of an identification inherently includes working with data of varying quality. Two prints (latent or exemplar) are never 100% in agreement. And the threshold for identification depends on whether one can explain the differences.

That's as if in math you had to decide each time what the figure is before you perform an operation on it -- under those circumstances, I don't think mathematics would claim to be error free.

You can completely remove the practitioner from many mathematical operations -- that's what adding machines, calculators, and computers do. If the device is correctly made, the result is error free. When you try to remove the practitioner from fingerprint comparisons with various scanners and databases you get errors. (I suppose one could argue the devices are not yet correctly made, I suppose.)

I'm also not persauded by the arguments about the one examiner with a 50% error rate on a test, and his buddy with a 0% error rate who combine to a 75% error rate. Of course that doesn't make sense -- but the problem there is a basic statistical error of trying to draw conclusions from one test, and a sample size of two. If we were able, somehow, to give a series of fair tests that resemble typical casework to every examiner in the U.S., one might be able to determine a very rough approximation of error rate.

Does that mean that every practitioner has the same rate? No. No more that it means that every lab tech processing David F's pregnancy tests has the same likelihood of error. It does mean that we can have an idea in the abstract of the likelihood of error, and can then look at that particular examiner's work to see if this case is an error.

MIchele T. is right that one problem is defining error. Daubert doesn't define it. One could look at a variety of options. Since verification is part of ACE-V, I'd assume ACE-V's error rate would only include stuff that got past the verifier -- however it was caught thereafter.

Charles P. -- I talk about "20ish" or "about twenty" examples, in part because I agree that we don't know enough about how each of those mistakes happened and there's some dispute about a few of them like McKie. Some may be true misidentification, some may be fabrications, we don't know. But recall that it is the prosecutor's job to prove the admissibility of the evidence -- it isn't the critic's job to show how the errors happened -- it is the proponent of the evidence's job to show why those cases don't matter or aren't significant to this case. The prosecutors and y'all are in a much better position to get the data and interpret it than the defense bar and academics. FREX--we still haven't seen the Cowans enlargments that went to the jury -- the prosecutors and presumably Boston PD have them and won't turn them over. All I can work with in discussing the case is the newspaper accounts and I know those are likely incomplete.

Remember that to some juries and some judges, fingerprints are infallible -- they've never heard of errors and don't realize that it is possible. So the critics talk about errors to say, yes, it could happen -- to get past that initial reluctance to allow for the chance of error.

I'm willing to grant that the incidence of error is likely low, probably very low. But it would be useful to have an idea how low -- one in a thousand? one in ten thousand? One in a million? One in ten million? (Any of these numbers would likely get past Daubert.)

Did it happen here -- that's a different question and one directed to this specific print, this specific examiner. And that's when we talk about that specific examiner's training and practices.
[/i]