Legitimacy of the PCAST report

[Boyd Baumgartner]

I ran into this post over at David Kaye's blog on Forensic Science.

The summary is that in Washington State, the Court of Appeals found (in a ballistic identification case) that:

[T]he PCAST report acknowledged its own dubious value to courts, stating, “Judges’ decisions about the admissibility of scientific evidence rest solely on legal standards; they are exclusively the province of the courts and PCAST does not opine on them.” -- Chief Judge Marlin J. Appelwick, Washington Court of Appeals, Mar. 11, 2019

Obviously, Kaye's position pushes back that the legal standards (e.g. Frye/Daubert) are and should be informed by scientific standards (validity anyone?) and that PCAST is precisely the type of entity to do that work.

He also quotes the PCAST Report which says:

Neither experience, nor judgment, nor good professional practices (such as certification programs and accreditation programs, standardized protocols, proficiency testing, and codes of ethics) can substitute for actual evidence of foundational validity and reliability. (P. 6)

This somewhat spills over into the Illinois thread as we can see because the crux of the argument there is diametrically opposed. In that thread, the article that prompted the discussion basically says of the Latent Print Unit that: 'The conclusions aren't valid because the experience, judgement and professional practices are lacking'. Whereas in the Kaye article, the PCAST report minimizes experience, judgment and good professional practices as having significant bearing on foundational validity and reliability.

This must be true for both foundational validity and foundational validity as applied which ironically is given...wait for it...without a means by which said validity could be demonstrable. (read: what's good for fingerprints is good for those critiquing fingerprints)

And this is the second time this month PCAST is being dismissed because in one of the letters to the editor in the latest JFI regarding the error rate estimates of the Miami Dade Study (used in the PCAST report re: latent prints), the author states:

I agree with PCAST that the error rates can be valuable tools to assess foundational validity of forensic tests; however, I believe PCAST to be perhaps a bit misguided (and maybe even biased in their views) when offering their overly authoritative requirements on communicating error rates.

Question? Jokingly.. in the same spirit as those who seek scientific validity, do we also need to measure the biasability of the PCAST members and establish that critique as scientifically valid?

So....the big idea here is this: the validity behind various policy questions are appeals to the authorities issuing them.

From a mere, 'Examiners are the ultimate consumer/evangelists of forensic science and policy' what do you think the spin is via each side and how do you correct it on the stand?

[Pat A. Wertheim]

The PCAST report on forensic science is presented as if it were a definitive document setting forth unanimous agreement on standards in science and why forensics, as currently practiced, does not meet those standards. I offer the following insight not as a suggestion for any other LPE to use against a challenge that Latent Print Identification is not scientific, but as my personal observations and brief summary of how I handle that line of questioning.

In a laboratory where I worked previously, our Director hosted a day-long lab tour for Dr. S. James “Jim” Gates, a member of the President’s Council of Advisor on Science and Technology (PCAST) at the time. I was honored when I was approached by one of the administrative assistants and told that Dr. Gates had told the Director he specifically wanted to meet with me while he was there. My balloon was deflated somewhat when I learned Dr. Gates had singled me out simply because I was the oldest looking guy with the most gray hair he had seen in the hallway on his initial walkthrough of the lab.

A half hour was set aside for our meeting. Jim, as he asked me to call him, explained that he had learned that he could discover more about a lab from talking to the old timers than he could by listening to what the managers wanted him to hear. Jim asked for my thoughts and perspective on the changes I have seen during my career, on the practice of forensic science today, and where I thought we were headed or what we should be aiming for in the future.

As a career latent print examiner, I focused my comments on the evolving situation in latents from before the first Daubert challenge to fingerprints (the Byron Mitchell case), the Stacy report (Brandon Mayfield case), the NAS report, and the PCAST report. I discussed Karl Popper’s falsifiability test as the definition of science when I got into the business. I also brought up Carl Hempel’s definition of science (the use of validated natural laws to provide explanations and make predictions) and Karl Kuhn’s historical assessment that “normal science” is the use of consensus dogma among practitioners to solve small, everyday problems. I pointed out that none of these accepted authorities on science in the last half of the Twentieth Century demanded numbers to define science. Then I decried the recent demand that without numbers such as probabilities, likelihood ratios, error rates, etc., a field did not qualify as science. I explained that I thought this was a misapplication of Bayes Theorem, first developed in the mid-18th Century, to assess probabilities

Our laboratory Director was a strict numbers guy. He and I had lunch together once a month and debated the issue relentlessly, neither of us convincing the other, but never getting angry at each other’s refusal to budge. He knew my arguments and was quietly flinching in the corner as Jim Gates and I talked about the drift in the definition of science over the last half century. While Jim never defended the absolute requirement of numbers in the definition of science, neither did he necessarily agree with me that no numerical validity could be established in latent prints. He mostly asked questions and listened. What he did comment, which I thought was significant, was that the co-chairman who was dominant in the writing of the PCAST report on forensic science was himself a very strict numbers guy who brokered no compromise on the issue. I inferred that there was not unanimous, unambiguous agreement among PCAST members on all of the conclusions in the final report.

When I testify in court and question comes up of whether fingerprint identification is science, I discuss the philosophies of Popper, Hempel, and Kuhn. I then point out that the current emphasis on numbers, at least in fingerprints, is a throwback to the application of Bayes Theorem from the mid 1700s to derive probabilities in fingerprints.

I ask the court’s permission to draw a diagrams to demonstrate a concept of science. I have never been refused permission. I draw a Venn Diagram starting with a large circle, which I label “Universe of Science.” Inside that large circle, I draw three medium circles, which I label “Exact,” “Descriptive,” and “Applied.” These medium circles may overlap a little with each other. I explain that “exact sciences” are those that use formulas and equations and numbers to solve different types of equations. I draw smaller circles in “Exact Sciences” to represent physics and math. I then explain “Descriptive” sciences as those that rely on descriptions, not numbers. I label a smaller circle inside of “Descriptive Sciences” as “Ornithology” and point out that you define a mockingbird by describing its plumage and song. You do not use numbers to differentiate a mockingbird from a cardinal or a chicken. Then I draw a smaller circle inside the medium circle marked for “Applied Science” and label it “Forensics.” I add that perhaps medical practice is also an applied science. Inside the “Forensics” circle, I draw even smaller circles for “Firearms” and another for “Fingerprints.” I may point out that some disciplines in forensics may fall in the overlap between “Exact” and “Applied” sciences, such as DNA. But some sciences involving physical comparisons more accurately exist in the middle of “Applied” or may even fall in the overlap between “Applied” and “Descriptive.”

I then add a comment that, as in most things, biases can affect opinions. So if you ask a scientist established solidly in the middle of the “Exact” circle, that person may deny that anything in the “Descriptive” or “Applied” circles are even science at all. I express the belief that those attacking fingerprints as pseudoscience come from the persuasion that without numbers as in the “Exact” circle, nothing is science. I assert that is false and express my opinion that the definition of science is a complex idea without a narrow definition. Popper, Hempel, and Kuhn have valid ideas, as well as Bayes, but to accept only one and deny all the others is itself "junk science."