This is making it's way around. I just testified in a case not too long ago where I discussed the disagreement about standards at the Fed level between the OSAC and the DOJ, so it seems rather timely. The FRS response to the DOJ ULTRs was interesting to say the least. I hadn't seen it before.
What do you think?
Forensics, Justice, and the Case for Science-Based Decision Making
Simon A. Cole, UCS Science Network, UCS | November 14, 2018, 10:01 am EST
This post is a part of a series on Science For Justice
Forensic science—and the language forensic scientists use to talk about their findings–has real-world impacts, sometimes life-or-death impacts, for real people. If the criminal justice system is going to really serve the cause of justice, it needs to be informed by the best available science. Unfortunately, the United States Department of Justice (DOJ) is ignoring scientific best practices, reversing progress toward improving forensic science in the U.S.
At the end of July 2018, the DOJ announced the release of eight new Uniform Language for Testimony and Reporting documents (ULTRs) at the annual meeting of the International Association for Identification. An ULTR is a document meant to ensure that all forensic practitioners from the same discipline in DOJ forensic science laboratories use the same language in reporting the results of their analyses to police, lawyers, judges, and juries. While an ULTR is only binding on DOJ laboratories, state and local laboratories often follow DOJ’s lead.
The Deputy Attorney General said at the meeting that these documents “meet the highest scientific and ethical standards.” But do they?
All nine of the ULTRs use what is sometimes described as a “categorical” reporting framework. This framework sorts all reports into a small number of categories. For example, the categorical framework for firearms evidence is:
1.Source identification (i.e., identified)
2.Source exclusion (i.e., excluded)
3.Inconclusive
Categorical reporting has long been widely criticized because the artificial boundaries between the categories render the system prone to perverse cliff effects. A better way would be what might be called “continuous” reporting, in which the weight of the evidence is reported as it is, rather than by reference to its place in a relatively crude three-category framework.
Another criticism of categorical reporting is that it implies certainty, as for example in the firearm example above in which the analyst would tell the jury “that two toolmarks originated from the same source.” Science doesn’t deal in certainties, and these ULTRs violate basic probabilistic reasoning. They are neither logical, nor scientific. That very point was made in the public comments on the draft ULTRs by several commentators and in a recent report on latent print analysis by the American Academy for the Advancement of Science (AAAS).
A discouraging omen
In April 2017, Attorney General Jeff Sessions shut down the National Commission on Forensic Science, a roughly 30-member advisory panel of scientists, forensic and non-forensic, and legal and law enforcement professionals. The Commission had been launched in 2013 after a 2009 report by the National Research Council, the official science advisor to the US Congress, found “serious deficiencies in the nation’s forensic science system” and called “for major reforms.” With the closing of the Commission, the DOJ turned its forensic reform effort over to the Forensic Science Working Group, the current publisher of the ULTRs.
Given that the ULTRs are the first official documents produced by the Forensic Science Working Group as part of its “plans to advance forensic science,” these documents are a discouraging sign for a future in which forensic reform is driven by the DOJ. Since the ULTRs were supposed to “serve as a model for demonstrating” the DOJ’s “commitment to strengthening forensic science, now and in the future,” their flaws don’t portend well.
Not making sense
After stating that the forensic experts should report that they know the source of a forensic trace, the ULTRs go on to make a number of statements that sound more uncertain. It might seem like the ULTRs are trying to tone down their claims of certainty, but the result is that the ULTRs try to support reports of certainty with statements of uncertainty. That doesn’t make any sense.
It also seems like the ULTRs are suggesting that small probabilities can be rounded down to zero for the “consumer” of the evidence. But it is unclear why that would be a scientific, or a just, thing to do.
It is helpful that the ULTRs contain lists of statements that should not be said, such as “zero error rate” and “100% certain.” These statements were made for years, including by DOJ forensic analysts, and they have now been largely discredited. However, a lot of the “banned” statements are what I call “false concessions.” It appears that the DOJ is conceding something important, but in fact they are conceding little or nothing because analysts are still permitted to make statements that are logically equivalent to the banned statements.
Scientists, not just forensic scientists, can weigh in to protect the role of evidence
In recent years, some progress has been made toward recognizing the inherently probabilistic nature of all scientific evidence and seeking ways of communicating those probabilities to lay audiences. The ULTRs signal that the DOJ is not yet ready to join that effort. This is unfortunate, given the DOJ’s power and influence.
Scientists don’t need to know anything about forensic science to understand that categorical statements of certainty are not plausible. Any scientist can help by letting the DOJ know that their statements are not scientifically credible and that the opinions of individual scientists and scientific institutions should be taken seriously by the nation’s most important purveyor of justice.
Overstating the certainty of forensic evidence has been implicated in many miscarriages of justice. And it is scientifically wrong. The people who are the ultimate consumers of forensic evidence deserve better.
Simon A. Cole is a Professor at University of California, Irvine’s Department of Criminology, Law and Society.
Science Network Voices gives Equation readers access to the depth of expertise and broad perspective on current issues that our Science Network members bring to UCS. The views expressed in Science Network posts are those of the author alone.
