Quantifying the Weight of Evidence
Posted: Tue Jul 25, 2017 8:39 am
The video from the NIST conference regarding quantifying the weight of evidence is up.
Day 1
youtu.be/NjDfCg8tIU8
Day 2
youtu.be/36ruxZy93io
I haven't finished watching the whole thing, but some interesting observations are arising. At this point there are 4 camps on how things are done and they vary more in form than type which is rather comical to see people talking past each other just to arrive at the same conclusion.
They are (in no particular order)
Camp 1: The old school, where conclusions are based on experience, heuristics (knowing that fingerprint comparison works, not why) and intuition (Evett goes on to discuss the PCAST notion of black box as intuition which is WAAAAAY off base)
Camp 2: The Evett/Champod school which says there is not one single L/R but L/Rs that are married to the people using them. (Read: Subjective, conditional probabilities). The problem being, its priors are fed by experience even though they don't know how it works, the fact that you plug it into Bayesian formula makes it 'work'. (I mean if it doesn't work in the observation of an examiner, plugging it in to a L/R doesn't make it suddenly work. Conversely, if it's working in the observation of an examiner then why use a L/R at all?) Champod then goes on to articulate in the Panel Discussion that the Weight of Evidence is merely a belief with an associated strength that he calls an opinion. (I fail to see how this is any different than the old school other than the fact that the veneer of authority has shifted from the person to the math with the math really just being a regurgitation of the person's opinion with the unfortunate side effect of overstating the conclusion because.....math)
Camp 3: The Joseph Kadane/USACIL school which is still L/R based but is frequentist based. (Evett acknowledges this in the Q&A when Kadane asks his question after the first presentation on day 1). This flies in the face of what Evett/Champod are saying because it's a one size fits all approach. The limitations of both camps 2 and 3 is that they fail to consider more than 1 hypothesis for the defense meaning they are in essence p-hacking for the most favorable hypothesis to the prosecution. (I believe this is what Bill Thompson is in essence saying in the Q&A)
Camp 4: The 'Don't lump me in with those guys camp', which says fingerprints don't compare themselves and ridges are continuous entities not merely discreet galton points. Therefore, it takes people to do this task and any frequency model will fail to take into account statistically relevant data and will therefore be necessarily underperforming. People have brains which contain the right object recognition abilities to perform the task of identification and expertise contributes to the accuracy of this ability (Busey, Vanderkolk et al), but how it works is not fully understood because brain science is still in its infancy, however it can be demonstrated. (hence the brain as a black box reference). Lastly, it should be of significance that any error that has occurred has been demonstrated to be one using the same method as the one that produced the error. Does anyone know of an instance where a L/R has deemed something to be not an ID where examiners said it was? I believe I heard of one such instance at USACIL, but if anyone has some more info it would be interesting to hear.
If anyone has thoughts or got something other than what I did out of it, let's hear it.
Day 1
youtu.be/NjDfCg8tIU8
Day 2
youtu.be/36ruxZy93io
I haven't finished watching the whole thing, but some interesting observations are arising. At this point there are 4 camps on how things are done and they vary more in form than type which is rather comical to see people talking past each other just to arrive at the same conclusion.
They are (in no particular order)
Camp 1: The old school, where conclusions are based on experience, heuristics (knowing that fingerprint comparison works, not why) and intuition (Evett goes on to discuss the PCAST notion of black box as intuition which is WAAAAAY off base)
Camp 2: The Evett/Champod school which says there is not one single L/R but L/Rs that are married to the people using them. (Read: Subjective, conditional probabilities). The problem being, its priors are fed by experience even though they don't know how it works, the fact that you plug it into Bayesian formula makes it 'work'. (I mean if it doesn't work in the observation of an examiner, plugging it in to a L/R doesn't make it suddenly work. Conversely, if it's working in the observation of an examiner then why use a L/R at all?) Champod then goes on to articulate in the Panel Discussion that the Weight of Evidence is merely a belief with an associated strength that he calls an opinion. (I fail to see how this is any different than the old school other than the fact that the veneer of authority has shifted from the person to the math with the math really just being a regurgitation of the person's opinion with the unfortunate side effect of overstating the conclusion because.....math)
Camp 3: The Joseph Kadane/USACIL school which is still L/R based but is frequentist based. (Evett acknowledges this in the Q&A when Kadane asks his question after the first presentation on day 1). This flies in the face of what Evett/Champod are saying because it's a one size fits all approach. The limitations of both camps 2 and 3 is that they fail to consider more than 1 hypothesis for the defense meaning they are in essence p-hacking for the most favorable hypothesis to the prosecution. (I believe this is what Bill Thompson is in essence saying in the Q&A)
Camp 4: The 'Don't lump me in with those guys camp', which says fingerprints don't compare themselves and ridges are continuous entities not merely discreet galton points. Therefore, it takes people to do this task and any frequency model will fail to take into account statistically relevant data and will therefore be necessarily underperforming. People have brains which contain the right object recognition abilities to perform the task of identification and expertise contributes to the accuracy of this ability (Busey, Vanderkolk et al), but how it works is not fully understood because brain science is still in its infancy, however it can be demonstrated. (hence the brain as a black box reference). Lastly, it should be of significance that any error that has occurred has been demonstrated to be one using the same method as the one that produced the error. Does anyone know of an instance where a L/R has deemed something to be not an ID where examiners said it was? I believe I heard of one such instance at USACIL, but if anyone has some more info it would be interesting to hear.
If anyone has thoughts or got something other than what I did out of it, let's hear it.
