Quantifying the Weight of Evidence

[Boyd Baumgartner]

The video from the NIST conference regarding quantifying the weight of evidence is up.

Day 1

youtu.be/NjDfCg8tIU8


Day 2

youtu.be/36ruxZy93io


I haven't finished watching the whole thing, but some interesting observations are arising. At this point there are 4 camps on how things are done and they vary more in form than type which is rather comical to see people talking past each other just to arrive at the same conclusion.

They are (in no particular order)
Camp 1: The old school, where conclusions are based on experience, heuristics (knowing that fingerprint comparison works, not why) and intuition (Evett goes on to discuss the PCAST notion of black box as intuition which is WAAAAAY off base)

Camp 2: The Evett/Champod school which says there is not one single L/R but L/Rs that are married to the people using them. (Read: Subjective, conditional probabilities). The problem being, its priors are fed by experience even though they don't know how it works, the fact that you plug it into Bayesian formula makes it 'work'. (I mean if it doesn't work in the observation of an examiner, plugging it in to a L/R doesn't make it suddenly work. Conversely, if it's working in the observation of an examiner then why use a L/R at all?) Champod then goes on to articulate in the Panel Discussion that the Weight of Evidence is merely a belief with an associated strength that he calls an opinion. (I fail to see how this is any different than the old school other than the fact that the veneer of authority has shifted from the person to the math with the math really just being a regurgitation of the person's opinion with the unfortunate side effect of overstating the conclusion because.....math)

Camp 3: The Joseph Kadane/USACIL school which is still L/R based but is frequentist based. (Evett acknowledges this in the Q&A when Kadane asks his question after the first presentation on day 1). This flies in the face of what Evett/Champod are saying because it's a one size fits all approach. The limitations of both camps 2 and 3 is that they fail to consider more than 1 hypothesis for the defense meaning they are in essence p-hacking for the most favorable hypothesis to the prosecution. (I believe this is what Bill Thompson is in essence saying in the Q&A)

Camp 4: The 'Don't lump me in with those guys camp', which says fingerprints don't compare themselves and ridges are continuous entities not merely discreet galton points. Therefore, it takes people to do this task and any frequency model will fail to take into account statistically relevant data and will therefore be necessarily underperforming. People have brains which contain the right object recognition abilities to perform the task of identification and expertise contributes to the accuracy of this ability (Busey, Vanderkolk et al), but how it works is not fully understood because brain science is still in its infancy, however it can be demonstrated. (hence the brain as a black box reference). Lastly, it should be of significance that any error that has occurred has been demonstrated to be one using the same method as the one that produced the error. Does anyone know of an instance where a L/R has deemed something to be not an ID where examiners said it was? I believe I heard of one such instance at USACIL, but if anyone has some more info it would be interesting to hear.

If anyone has thoughts or got something other than what I did out of it, let's hear it.

[John Vanderkolk]

I identify with the 'Don't lump me in with those guys camp.' I look forward to August 7, 11:00 a.m. at the IAI seminar. JohnV

[Bill Schade]

I'm curious as to which camp you (or your agency) are in.

100 words or less please

[Boyd Baumgartner]

I'm in camp 4.5.1:

I'm just an examiner just trying to look at prints, not save the discipline from itself.

P.S. get off my lawn

[Shane Turnidge]

Clint Eastwood said it best... "Man's gotta know his limitations..."

[NRivera]

The USACIL model is not "frequentist" based. At least my novice understanding of the term frequentist refers purely to the frequency of individual features. I would envision a frequency-based model to involve numbers a lot like DNA. The USACIL model is based on distortion studies and feature similarities in a reference dataset of about 2000 known prints. 1) It requires a human examiner to arrive at an identification decision first; 2) It provides a similarity score based on whether or not the marked features in the known and questioned impressions are within an acceptable "threshold" in terms of spatial and directional relationship; 3) It provides two distinct probabilities of the marked features being observed in same-source vs. different source prints and; 4) the number being reported is the ratio of those two probabilities. The model is meant to provide additional quantitative data to support the underlying "association". (I don't like this word, but it's what they're using, maybe for another discussion.) If the quantitative data is insufficient to support an "association" it stands to reason that the evaluation be amended accordingly or that the lack of quantitative data support be disclosed.