FBI exaggerates

[Tazman]

Uh oh.

Is it any wonder forensic science is under attack?
http://www.washingtonpost.com/local/cri ... ml?hpid=z1

Washington Post:

FBI overstated forensic hair matches in nearly all trials before 2000
By Spencer S. Hsu April 18 at 5:44 PM

The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000.

Of 28 examiners with the FBI Laboratory’s microscopic hair comparison unit, 26 overstated forensic matches in ways that favored prosecutors in more than 95 percent of the 268 trials reviewed so far, according to the National Association of Criminal Defense Lawyers (NACDL) and the Innocence Project, which are assisting the government with the country’s largest post-conviction review of questioned forensic evidence.

The cases include those of 32 defendants sentenced to death. Of those, 14 have been executed or died in prison, the groups said under an agreement with the government to release results after the review of the first 200 convictions.

more . . . . .

[Boyd Baumgartner]

These are the interesting topics...

I'm no hair expert (insert bald jokes here), but from what I've read the Hair Analysis crowd very much frames their work in a similar light as fingerprint examination.


From the documents linked below:

Hair comparisons are a combination of a pattern-recognition process and a step-by-step analysis of a questioned hair and a known sample

Evaluation of shared and distinguishing characteristics is essentially the process used in forensic hair examinations

The microscopic characteristics allow for hair to be categorized into smaller groups

Following their analyses, hair examiners may conclude the following:

The questioned hair exhibits the same microscopic characteristics as the hairs in the known hair sample and, accordingly, is consistent with originating from the source of the known hairs.


The questioned hair is microscopically dissimilar to the hairs found in the known hair sample and, accordingly, cannot be associated to the source of the known hairs.


Similarities and slight differences were observed between the questioned hair and hairs in the known hair sample. Accordingly, no conclusion could be reached as to whether the questioned hair originated from the same source as the known hairs.

Someone please correct me if I'm wrong, but basically you Analyze a hair, visually compare two hairs (One unknown, and one Exemplar) based on progressively smaller levels of detail (classificatory -> specific features) and arrive at an evaluation that includes association, no association, inconclusive. (ACE)


(Sources for statements)

Forensic Hair Comparison: Background Information for Interpretation

Hair Evidence

To their credit, they qualify the strength of their association but their determinations depend on linear analysis, probabilistic assumptions and experience of the examiners. They've admitted this is wrong, but it almost seems as this is where we are headed (linear +probabilistic + Examiner).

The questions I have are:
Is an overstated conclusion an error?
Is an individualization an overstatement?
Would a scale of conclusions (Identification, Strong Association, Weak Association, No Association) (insert your own terms)) create a higher risk of overstating since there are categorically more statements?
Currently, does an 'inconclusive' give the impression of leaning towards individualization and thus represent an overstatement? (we've seen topics where AFIS inconclusives are reported out as leads) Link to CLPEX thread
Why would we be moving towards a model that is being acknowledged as less than best practice?

Feel free to answer any/all of the questions

[Tazman]

Another old LPE and I were talking about this over coffee yesterday. He pointed out to me that the press has not been friendly to us in latent prints since the Mayfield case. They keep throwing that up as if that one case makes millions of other fingerprint identifications suspect. They thrive on creating controversy where little basis exists for serious disagreement. Is that what they are doing here?

The current beliefs tell us that "100% positive identification to the exclusion of anybody else on earth" is not scientifically valid. Yet that is how we testified fo fingerprint identifications back during the time frame these FBI hair analysts are said to have misrepresented their evidence. If what the FBI hair examiners said back then is viewed by the NACDL and the press today as an exaggeration, how is that different from what the NACDL and the press have been saying about latent prints since Mayfield?

My friend said he trusts the FBI examiners to be ethical and honest. He says he finds it much easier to believe that reporters would exaggerate the findings of a report in which the NACDL was a primary mover, than to believe that there was wholesale intentional misrepresentation of evidence by the folks at the FBI Crime Lab.

A reasoned consideration leads me to conclude that without the original report, I have little basis to understand anything that happened. The smart money would bet on the ethics of the FBI over ethics of the press any day.

[Boyd Baumgartner]

Sure, the fact that it's 'man bites dog' journalism speaks to the frequency of such occurrences, etc. and I don't think it's an ethical failing of examiners outright. It does speak to a potential systematic problem though as opposed to the usual one off controversies that are limited to a specific person/comparison.

That aside though, the basis for conclusions has been at the heart of forensic identification controversy from the get go. Arguably, that focus was at the center of the recommendations of the NAS and functionally there was a willingness to look at the cases by the FBI in the first place, so it can't just be dismissed as sensational journalism.

Link to 2013 article

The question, which pertains to any examiner has to do with stating the significance of the findings (see quote). And while this was more pertaining to court, an evaluation and the concept of sufficiency is a statement of significance. Conclusions aren't based solely on: Confidence (Mayfield), Similarity (Mayfield), Correspondence (Lana Canen), Rarity, or Justifications(Shirley McKie). I've said it before and I'll say it again, but I'd say that there's a lack of value theory that exists in our discipline that overlaps with what's going on here. The data you use has to be recognizable/recognized as being as functional and valuable by others.

From the article:

“There is no reason to believe the FBI Laboratory employed ‘flawed’ forensic techniques,” FBI Special Agent Ann Todd, a spokeswoman for the bureau, said in an email. “The purpose of the review is to determine if FBI Laboratory examiner testimony and reports properly reflect the bounds of the underlying science.”

Todd called microscopic hair analysis “a valid forensic technique and one that is still conducted at the lab” in conjunction with DNA testing. But in some cases, defense attorneys say, lab analysts have overstated the significance of their findings.

In 1981, for instance, an 18-year-old Washington resident named Kirk Odom was convicted of rape and sodomy. At his trial, an FBI analyst testified that Odom’s hair samples and samples taken from the crime scene “were indistinguishable” and that this was very rare. Odom was convicted and served about 22 years in prison. In 2011, he was exonerated by DNA testing that wasn’t available at the time of his original trial.

Odom’s case, along with the cases of two other Washington men who’d also been exonerated by DNA testing after convictions based, in part, on testimony about hair analysis, prompted the broader FBI review.

“The number of cases that involve potential testimony errors is not known, at this time,” Todd said.

[L.J.Steele]

Fred Whitehurst has some comments on the FBI's culture and why this continues to go on:

https://www.themarshallproject.org/2015 ... 150427-167

Bullet Lead Comparison
Mayfield and zero error rate
Hair comparison

I don't deal with the FBI in my cases and don't have a personal opinion, but there's more here than the press picking on the labs.

[Tazman]

I do not disagree with Mr. Whitehurst's suggestion of a National Defense Forensic Institute. As I understand him, that would be a national crime laboratory with the mission of serving the defense attorneys who need a review of prosecutions' forensic evidence. There are a couple of major problems that come to mind.

1) How would such a forensics lab be funded? Federal dollars? Would it then examine only Federal cases? Or would it operate as a quasi-private lab and charge a fee for services? If so, who would fund the fees? Most courts do not fund the reasonable defense requests for a forensic review of prosecution's evidence now. What would make such a new National Defense Forensic Institute viable?

2) Would a defense attorney have to seriously justify why she thinks a forensic review is necessary? Or would the problem enter the equation that courts complain about now, i.e., defense attorneys would just use the new lab for delays, or as fishing expeditions with no expectation of actually finding flaw with the police evidence?

I suspect other serious questions would have to be answered. But those two pop immediately to mind. Funding, and frivolous use of valuable resources.

[L.J.Steele]

There's a good response to Whitehurst's idea here:

We need to get the bad science out of the courtroom, period. Defense experts tend to start at a disadvantage — jurors often view them as hired guns, while seeing prosecution experts as public servants. Jurors also like certainty. They find it convincing. Experts willing to eschew scientific probabilities for unscientific statements of certainty will often sound more convincing. This isn’t about biasing jurors toward defendants. It’s about making sure they don’t hear evidence that isn’t grounded in scientific research, no matter how persuasive it may sound.

http://www.washingtonpost.com/news/the- ... c-analyst/

I'm with this -- bad science does not belong in a courtroom (or anywhere else).

More funding for attorney training (both sides) so folks can recognize bad science when they see it; more funds for defense experts so the playing field is more level; and a culture of science in the courts so judges pay attention to challenges, keep out the stuff that shouldn't be there, and think about the issues rather than relying heavily on precedent.

There are logistical issues. I expect the defense would be as prone as the prosecution to over-testing. There might also be confidentiality/attorney-client work product issues if the same lab is testing evidence involving co-offenders for different defense counsel.

[SConner]

Assuming that we ever reach a point where there is such a National Defense Forensic Institute, would you expect/require their personnel to be subject to the same rules of disclosure as the prosecution? A common example, state crime lab LPE says that the latents match the defendant and the NDFI LPE also says that the latents match the defendant. Clearly the state crime lab employee's report will be known to the defense, however, will the NDFI employee's report be known to the prosecution, admitted as evidence, can the NDFI examiner be compelled to testify?

[Tazman]

. . . . . will the NDFI employee's report be known to the prosecution, admitted as evidence, can the NDFI examiner be compelled to testify?

I should think not!!!

The American Criminal Justice System is predicated on the principle that a person is innocent until proven guilty. Proving guilt beyond a reasonable doubt is the prosecution's burden. It is never the defense's job to prove the defendant innocent, much less is it the defense's job to help the State prove the defendant guilty.

The reason for a competent defense expert is not to help the State prove its case. It is to prevent the State from convicting an innocent person through mistake, or worse, through fabrication of evidence.

[SConner]

Well now, that idea would be in direct conflict with the Code of Ethics of the IAI now wouldn't it? I'm also assuming that you would want this NDFI to be an accredited lab. I don't believe that there is an accrediting body that would knowingly allow employees of a lab that they accredit to withhold information if requested to provide it. If the prosecution doesn't ask, that's one thing, but if something like an NDFI gets up and running, I would expect their results to be disclosed just the same as mine when mine are requested. There's an old saying, don't ask questions that you don't know the answers to or don't want the answers to.

[calvinmollett]

The reason for a competent defense expert is not to help the State prove its case. It is to prevent the State from convicting an innocent person through mistake, or worse, through fabrication of evidence.

Or sometimes - "It is to prevent the State from convicting a guilty person through lies, or worse, through fabrication of evidence."

[Alan C]

Well now, that idea would be in direct conflict with the Code of Ethics of the IAI now wouldn't it? I'm also assuming that you would want this NDFI to be an accredited lab. I don't believe that there is an accrediting body that would knowingly allow employees of a lab that they accredit to withhold information if requested to provide it. If the prosecution doesn't ask, that's one thing, but if something like an NDFI gets up and running, I would expect their results to be disclosed just the same as mine when mine are requested. There's an old saying, don't ask questions that you don't know the answers to or don't want the answers to.

I'm not a lawyer and don't even play one on TV, but it's my understanding that the disclosure requirement is pretty much a one-way street. The state is required to disclose evidence to the defense but not the other way around due to the constitutionally guaranteed rights of the accused. The state doesn't have a corresponding right because the burden of proof is on the state. That would suggest that if there were an NDFI it couldn't be forced to disclose its results.

[SConner]

I understand that, however, science is generally supposed to be an open and collaborative atmosphere. Quite frankly, how would anybody know that an NDFI like facility was practicing good science if they never had to release any of their results?

[Alan C]

I understand that, however, science is generally supposed to be an open and collaborative atmosphere. Quite frankly, how would anybody know that an NDFI like facility was practicing good science if they never had to release any of their results?

Hmmmm . . . here's where the needs of the adversarial legal system don't quite mesh with the collegial ideals of science. Scrutiny of data (with identities of cases and defendants carefully excised) by an accrediting body? Proficiency testing of the examiners? Just a couple thoughts.

[SConner]

Exactly, you can't have your cake and eat it too. If you want "good" science available to both sides than my humble opinion is that the NDFI would have to play by the same rules as the labs that primarily serve the interest of the prosecution side.

Instead of lamenting the lack of science available to the defense, maybe more focus should be placed on the significance of the science being presented by the prosecution. I've processed many a case where based on the "facts" as presented there wasn't much that latent prints or DNA would be of use to the trier of fact. For the most part, DNA and latent prints are not going to help when a defendant is claiming self defense, when the jury is being asked to weigh premeditation or not, when the jury is being asked to weigh insanity, or when the jury is being asked to weigh what influence a deceased brother may have had over the surviving brother. Those last two examples are, of course, in reference to the Aurora theatre shooting and the Boston Marathon bombing. There was a lot of forensic evidence presented in both of those trials, but at the end of the day, what did that all matter when the defendants are not claiming that they weren't involved in the crime? However, the more "evidence" that a prosecutor is able to put forth, the more witnesses they're able to call, I'm of the opinion that the jury is going to favor whatever argument their making. That's another issue. How much of this "bad forensic science" is really prosecutors overstating our results in closing arguments?