Funny: "Using the NSA [sic] Report to "Junk" their "Science"

[Kasey Wertheim]

http://centrallaw.com/JunkScienceInFede ... /frame.htm

Using the NSA Report to “Junk” their “Science”
Frank Godbold Federal Defender Office, Tampa

“The law’s greatest dilemma in its heavy reliance on forensic evidence concerns the question whether – and to what extent – there is science in any given ‘forensic science’ discipline.”
-- NSA Report at 3-2

Two very important questions should be answered before admitting and relying on forensic evidence in criminal trials: (1) The extent to which a forensic discipline is based on a reliable scientific methodology that can accurately measure findings; and, (2)The extent to which forensic experts rely on human interpretation that could be tainted by error, bias, lack of sound procedures, and strong performance demands.
-- NSA Report at 3-2

Findings Include:•Little scientific research confirms the reliability of most forensic disciplines. •Forensic disciplines lack quantifiable measures of uncertainty in their conclusions. •Little research on human error and observer bias. •Lack of crime lab autonomy. •Lack of certification requirements, uniform accreditation of labs, performance standards, consistent terminology in reporting results, and oversight.

Why do we care about experts?
What is “Science?”

“Science” Defined
Knowledge or a system of knowledge covering general truths or the operation of general laws as obtained through and tested through the scientific method.
-- Meriam Webster

The “Scientific Method”
• A scientific method consists of the collection of data through observation and experimentation and the formulation and testing of hypothesis.

• Scientific researchers propose hypothesis . . . and design experimental studies to test these hypotheses
-- Wikipedia

What is “Junk Science?”
“Junk Science” is a field that has not tested its claims.

What does this have to do with the law?
The Daubert Factors •Testing •Error Rates •Standards •Publication and Peer Review •General Acceptance by the Relevant Scientific Community

Three Fields of “Junk Science”
•Fingerprints
•Handwriting
•Toolmarks/Firearms

“Junk Science” Is Often Based on the Same Untested Premise: Uniqueness
Everything in the world is unique. Every fingerprint, everyone’s

handwriting, every tool, every gun.

Junk Scientists Make the Same Claim:Because everything in the world is unique, we can make an identification to the exclusion of every other object in the world -- every finger with respect to fingerprints, every person with respect to handwriting, and every tool with respect to tool marks.

Our Goal:Expose each of these fields for the junk science that it actually is. • Pre-trial Motion to Preclude • Convince the jury that they should have a reasonable doubt regarding these fields.

How are We Going to Do It?
Learn the fields that we’re fighting.

1) David Faigman, et al., Modern Scientific Evidence: The Law and Science of Expert Testimony (West 2008) 2) Robert Epstein, Fingerprints Meet Daubert: The Myth of Fingerprint Science is Revealed, 75 Southern California L. Rev 605 (2002) 3) Adina Schwartz, A Systemic Challenge to the Reliability and Admissibility of Firearms and Toolmark Identification, 6 Columbia Sci. & Tech. L. Rev 2 4) Michael Risinger, et al., Exorcism of Ignorance as a Proxy for Rational Knowledge: The Lessons of Handwriting Identification “Expertise”, 137 U. Pa. L. Rev 731 (1989)

How Do We Fight It? Hire Experts!
•Counter experts
–Fingerprints – Ralph Haber - humanfactorsconsultants.com

–Handwriting – Mark Denbeaux - denbeama@shu.edu; Michael Saks -

michael.saks@asu.edu
–Toolmark’s – Adina Schwartz - aschwartz@jjay.cuny.edu

How Do We Fight It?
Get Discovery!
•Get the demonstrative evidence for you and your experts •Get the manuals of the crime lab •Get the government expert’s bench notes

-- New Hampshire v. Langill, 05-S-1129 (4/2/07 Sup. Ct. N.H.)

Challenging Fingerprint “Junk Science”

No Probabilities
Different people can have a number of matching ridge characteristics and there has been no probability testing to tell us the probability of that occurring.

Latent Print Training
The harsh reality is that latent print training as a structured, organized course of study is scarce. Traditionally, fingerprint training has centered around a type of apprenticeship, tutelage, or on-the-job training, in its best form, and essentially a type of self study, in its worst. Many training programs are the “look and learn” variety, and aside from some basic classroom instruction in pattern interpretation and classification methods, are often impromptu sessions dictated more by the schedule and duties of the trainer than the needs of the student. Such apprenticeship is most often expressed in terms of duration, not in specific goals and objectives, and often end with a subjective assessment that the trainer is ready. Such apprenticeships often end with a subjective assessment that the trainer is ready!
-- David L. Grieve, The Identification Process: The Quest For Quality, 40 J. of Forensic Identification 109, 110-111 (1990)

Examiners Do Not Know How Much They Have to See to Declare a Match No Agreed-Upon Standard:
1. FBI: No standard
2. Local U.S. crime labs: 8-12
3. France and Italy: 16
4. Brazil and Argentina: 30

Even Fingerprint Experts Recognize the Subjectivity of Their Opinions“[The] opinion as to whether there is sufficient uniqueness of detail present in the friction ridge print to eliminate everyone else in the world as a possible donor… is very subjective.”

-- David Ashbaugh, The Premises of Friction Ridge Identification, 44 J. Forensic Identification 499 (1994).

“Standards” is a Daubert FactorWithout standards we have nothing to hold the expert to.

How Are We Going to Challenge These Fields?
•Learn the fields •Hire experts •Request discovery •Hold the fields to the Daubert factors –Standards

Challenging Handwriting “Junk Science”

Handwriting: Two Principles
1.No two people write exactly the same way (inter-writer differences)
2.No person writes exactly the same way twice (natural variation)
3. Can forensic document examiners distinguish between inter-writer differences and natural variation?

Different People Can Have Similar Handwriting
The writing of different people can have significant similarity and we do not know what the probability is of this occurring.

No Standards
Handwriting “experts” have no standards for comparing handwriting and there is no standard for declaring an identification.

Challenging Toolmarks “Junk Science”

What is a “Toolmark?”Toolmarks simply marks left by a tool on a surface. Firearms identification is a subspecies of toolmark identification dealing with the toolmarks that bullets, and cartridge cases acquire by being fired. The gun in effect is the tool, the ammunition is the surface upon which the tool has left its mark.

Toolmarks Change Over TimeThe marks that a tool will make will change as the tool is used as a result of wear, and/or damage and corrosion. What this means, for example, is that there will be significant dissimilarity between bullets fired from the same gun. -- Alfred Biasotti & John Murdock, Criteria for Identification, 16(4) #$$@’n Firearms & Tool Mark Examiners 16,17 (Only 21-38% of the striae on pairs of bullets fired from the same revolver matched).

Different Toolmarks’ Similarity
Different tools can leave marks that have significant similarity and we don’t know what the probability is of that occurring

No Toolmark Standards
No standards for comparing toolmark impressions and no agreed upon identification standard for declaring a match

Testing
Have these fields been tested? NO

Error Rates
No testing – no error rates

Publication & Peer Review
•Purpose of publication and peer

review
•Internal non-blind verification does not insure reliability

General Acceptance
•Relevant scientific community
•Must look beyond the practitioners of the field itself
•State of Maryland v. Bryan Rose, K06-0545 (Cir. Balt. Co. 2008) (“general acceptance of latent print identification by its practitioners does not constitute general acceptance by the ‘scientific community’ . . .”); United States v. Saelee, 162 F.Supp.2d 1097 (“Finally, the evidence does indicate that there is general acceptance of the theories and techniques involved in the field of handwriting analysis among the closed universe of forensic document examiners. This proves nothing.”)
•Government has not and will not be able to produce anyone beyond law enforcement technicians

Expose the Lack of Qualifications of the Government “Expert”

Expose the Logical Fallacy of the “Uniqueness” Premise

“[T]he crux of the matter is not the individuality of the friction skin ridges but the ability of the examiner to recognize sufficient information for the disclosure of identity from a small distorted latent fingerprint fragment that may reveal only limited information in terms of quantity or quality”

-- Christophe Champod & Ian W. Evett, A Probabilistic Approach to

Fingerprint Evidence, 51(2) J. Forensic Identification 101, 115 (2001).

One Last Thing to Do
Expose the government’s expert for the “salesman” that he is: “A fingerprint expert is a salesperson selling the identification to the jury.”

This categorical requirement of absolute certainty has no particular scientific principle but has evolved from a principle shaped more from allegiance to dogma than a foundation in science . . .
-- David Grieve, Possession of Truth, 46 J. of Forensic Identification 521, 527-28 (1996) (Ex.2).g Whatever this may be, it is not science!

National Research Council on FirearmsConclusions drawn in firearms identification should not be made to simply the presence of a firm statistical basis where none has been demonstrated. Specifically, …

examiners tend to cast their assessments in bold absolutes, commonly asserting that a match can be made ‘to the exclusion of all other firearms in the world.’ Such comments cloak an inherently subjective assessment of a match with an extreme probability statement that has no firm grounding and unrealistically implies an error rate of zero.

How Do We Challenge “Junk Science?”
•Learn the field –Primary sources –Literature of the field
•Hire experts –Technicians –Counter experts
•Get Discovery –Bench notes –Lab manuals –Proficiency test file
•Hold the field to the Daubert factors –Standards –Testing –Error rates –Publication and peer review –General acceptance
•Expose the lack of training and qualifications of the government’s experts
•Expose the logical fallacy of the uniqueness premise
•Expose the government’s expert for the used car salesman that he is
•Tailor our attack to the specific opinion that’s being offered

Can it Work?
It already has
Handwriting - United States v. Hines, 55 F.Supp.2d 62 (D. Mass. 19990; United States v. Santillan, 1999 WL 1201765 (N.D. Cal. 1999); United States v. Rutherford, 104 F.Supp.2d 1190 (D. Neb. 2000); United States v. Brown, No. CR-184ABC (C.D. Cal. Dec. 1, 1999); United States v. Fuji, 152 F.Supp.2d 989 (N.D. Ill. 2000); United States v. Saelee, 162 F.Supp.2d 1097 (D. Alaska 2001)
Fingerprints - State of Maryland v. Bryan Rose, K06-0545 (Balt. Co. 2008); New Hampshire v. Langill, 05-5-1129 (Apr. 2, 2007 Sup. Ct. N.H.)
Toolmarks - Ramirez v. State, 810 So.2d 836 (Fla. 2001); United States v. Green, 405 F.Supp.2d 104 (D. Mass. 2005)

The End!

[Pat A. Wertheim]

"Funny" -- not really. It's frightening that defense attorneys would believe this stuff. The author plucks quotes from Dave Grieve and David Asbaugh out of context to "prove" that fingerprint identifications are not reliable. It's frightening to think that a defense attorney who doesn't know any better might read this article and use it to attack a fingerprint examiner who is not prepared to answer the objections presented in the article.

[Gerald Clough]

Exactly why examiners need to be prepared (and to prepare prosecutors) to take the reliability case into its proper context, its value in forensic decision making, not in argument over scientific truth. It's equally sad that naive attorneys will see things like this and believe they're now well oriented and will wonder why the courts don't seem to get it the way they had it fed to them. And it's sad that we have to prepare examiners to deal with this when preparing to answer the legitimate questions is so much more productive and beneficial to the examiner's solidifying his presentation. At least it's a rare attorney who knows enough to mount a coherent attack on cross-examination. Too busy trying to throw out the baby to think about how to muddy the bath water.

[Carl Speckels]

So for anyone that might know, in what context did Dave Grieve make those statements? It would be nice to know should a defense attorney make reference to them.

[sharon cook]

.

[David L. Grieve]

Carl, I thought about responding but I think anyone who reads the entire material can do so better than I can. The 1990 quote is from what was actually a presentation to the IAI conference in 1989. The topic was the history of training in the US up to that time. The other material was in response to the 1995 CTS latent print test, not fingerprints finest hour.

Since those two examples were used, I suggest anyone concerned should read them in their entirety. While at it, read all of Ashbaugh. One matter might be worthy of note. In 1999, in the Daubert hearing in US v, Mitchell, the defense cited those two quotes out of context. I was on the government's witness list which means the defense could have called me to the stand to explain what I meant. They did not, which means it isn't that difficult to dimiss those out of context examples.

[Gerald Clough]

...Why, I think formation seances are a perfectly reasonable...oh...

Never mind.

But are the operators of cutting edge recycling plants advised by junk scientist?

And what of the theoretical foundations of traditional Chinese shipbuilding?

[Neville]

I think courts should reject all eye witness evidence as it has been proved to be unreliable also.

It seems to me that DNA is the only reliable means of ID, provided there is no cross contamination (now that is a biggie even before the police arrive), no identical twins (a complete unknown), no blood transfusions ( a good escape root any time) and no bone marrow replacement (has happened!). The fact that by the DNA scientists own admission that the same DNA reoccurs in other people, yet we are supposed to think it is the best means of personal identification!

We could then maybe only allow the defendants own admission convict them. Wait, no, signed confessions mean nothing also, so why not just forget the whole thing and have anarchy instead.

Have a happy new year.

[sharon cook]

[Neville]

Hi Sharon
It is fabulous how these stats seem so similar no matter which office they come from, is it because they are true or is there no scientific proof therefore they are unbelievable. I remember being told that as the data base on the AFIS got bigger then the Hit rate would diminish, do you think that may happen with DNA, when they finally get a data base even 1/2 the size of the Fingerprint data base will it effect their Hit rate. I'm sure if you only did 12 comparisons a month, less than 3 a week; you would have a 50% hit rate too.

This line in the above made me smile “A fingerprint expert is a salesperson selling the identification to the jury.”
If I was to repeat that one to my family they would die laughing, I couldn't sell bullets in a gun shop. Being a sales person was never part of my job description.

I guess at the end of the day we must look at our opponents and ask what's in it for them?

I will get paid if I identify the defendant or not, if I do I then have to answer these idiot questions so where is my incentive?

I for one would be happy if I never needed to go to court again.

Now all you fingerprint officers out there go get another 10 IDs this week. In return you can have less pay and no days off and as a bonus you can go to court and sit outside and fret over what nasty questions solicitors may ask you inside. Does it work for you, no of cause not. If they were to say go get 10 more Ids and you can have a bonus say $100 for every extra Ident and extra days off too and there will be no sitting outside court ever again, well now that would work and I could understand all the fuss, I could understand why the knives are out. So I then come back to that question What's In It For Them?

[Dr. Dror]

As per DNA and cognitive issues, I think it is interesting to point out that DNA is also susceptible to bias. Present the same identical DNA profiles within different extraneous context, and they reach different and conflicting conclusions! I am currently writing up these data for publication.

Itiel

[L.J.Steele]

I think courts should reject all eye witness evidence as it has been proved to be unreliable also.

* * *
We could then maybe only allow the defendants own admission convict them. Wait, no, signed confessions mean nothing also, so why not just forget the whole thing and have anarchy instead.

Courts are starting to get more skeptical about eyewitness ID. I'm very encouraged by this recent Utah opinion on eyewitness ID experts

http://www.utcourts.gov/opinions/supopi ... 121809.pdf

The language about what an expert can testify to may be useful in cases involving defense experts in forensics cases who want to talk about methods but not about a conclusion of identification or exclusion. Adopting the _1999_ DoJ guide on Eyewitness ID, adding in blind procedures and recording where feasible, would likely solve many of these issues.

Much of the disputes about confessions could be solved by following the UK's lead -- since _1984_, all custodial interrogations in the UK are recorded from start to finish. That would solve many of the disputes about what was said, in what tone, and whether there was any coercision or pressure.

[sharon cook]

[Dr. Dror]

Dear Sharon,

1. I do not love 'extraneous context', it is only one issue that I study.
2. Extraneous context is an issue that almost all science have addressed and dealt with.
3. I am happy to hear that when you examine evidence you "know nothing about a case", and hope that this is true for everyone.
4. No one (definitely not me) has suggested that you (or anyone) sits around and intentionally "make our results fit what the detective wants"; rather, that exposure to certain information may affect what you perceive, how you interpret and judge information, and a whole set of established cognitive processes and influences.
5. As per your comment, "be sure and let us know whether you used actual DNA analysts, and not some random people off the street or "students" like you did in your famous fingerprint bias "study"":
First, please note that only in one study I used students, and all the other bias studies are with experience latent fingerprint examiners.
Second, also, please note that in the single study that I did use students, it is very clearly stated in the paper (The "Participants" section reads: "The participants in this study were 27 university student volunteers, with a mean age of 23 (nine were males and 18 were females)." page 802).
Third, yes, there were actual DNA analyst, as will be described in the 'Participants" section in the new paper.

Thanks for your comments, and happy New Year.


Itiel

[Neville]

Hi Lisa
Yes, we in NZ had a very high profile case recently where the complainant was certain that she knew that one of the defendants was one of her rapists, but in court she had not notice that he would have been using a walking stick and have a pronounced limp at the time of the offence, had he been there.

I think that many believe that if scientific study shows x is the situation in the lab then that applies in real life, this is obviously not the case, it only proves that under certain conditions (usually very controlled) it is the case. Look at all the wonderful methods of developing fingerprints used in labs over the years; rarely do they get beyond a years use in the field. Look at LCN profiling how many cases has it been successfully used in a prosecution? It's a tool that can be useful but it will not replace all other forensic evidence, yet to see what some people say one could conclude that the demise of fingerprints and other disciplines is just around the corner.

I am not suggesting that scientific study is a waste of time we have benefited from Goultons studies and many others over the years including math, i.e. Laser, Ninhydrin and a number of good blood reagents, I am suggesting though that we need to be a bit careful that we do not see it as some kind of unquestionable all knowing godlike thing. Scientific studies can be proven wrong. Years of considered observations can be as conclusive as a scientific theory. I hope that Dr Dror's studies go toward building up our discipline and not tearing it down. I suppose that depends on those in the field and how they apply themselves with this knowledge. Certainly the knowledge will not change only how it is used.