Article: False Conviction, How fingerprint and firearm experts use misleading math to appear infallible.

[Michele]

https://slate.com/news-and-politics/202 ... isons.html

When I read this article I disagreed completely. My question was, what test(s) consider 'inconclusive' as a correct answer, or doesn't count an inconclusive conclusion in the grading? (Spoiler alert: I suggest people read the article before reading my thoughts below).

After I went back to the original paper (attached), I see that the paper isn't talking about competency/proficiency tests, they are talking about tests which are taken for error rate studies.

I now agree completely, using error rate studies (or studies that show that experts are better than laymen) to support accuracy seem misleading to me as well. I would say the probability of error is dependent on the complexity of the comparison, the probability of error is not determined by error rate studies.

[4n6Dave]

When you calculate the positive or negative predictive value you would ignore any inconclusive results.

I would agree that in most of the forced decision tests an inconclusive should not be included as an error but it should not be included as a correct answer either. That is why the positive predictive value can be a more useful statistic to the court - When an examiner says ID how often are they correct? This calculation ignores inconclusives altogether.

The Slate and the Dror article misrepresent the statistics presented. There are reasons in a study design that would lead to more inconclusive results. Sample selection, time constraints, exemplars, lack of normal tools, lack of consults... I would challenge the authors to design a better study to explore their ideas rather than drawing conclusions out of whole cloth.

[Boyd Baumgartner]

If there's a guy who can spot people using math problematically, it's the guy who tried to argue that a sample size of 6 was valid for an experiment.

But I digress. The paper cited is really just a reformulation of a paper that he already wrote which was widely critiqued.

This paper just comes across as kitchen sink problematizing. Decisions are only normative relative to a standard, if you fail to articulate a standard there is no 'ought', only an 'is' and to deduce what one ought to do from the data that is observed is actually a fallacy.

[Texas Pat]

As usual, Boyd, you bring brilliant insights into the analysis of a claim, but this time you outdid yourself for brevity. Best read in a long time!