CLPEX Chat Board

https://slate.com/news-and-politics/202 ... isons.html

When I read this article I disagreed completely. My question was, what test(s) consider 'inconclusive' as a correct answer, or doesn't count an inconclusive conclusion in the grading? (Spoiler alert: I suggest people read the article before reading my thoughts below).

After I went back to the original paper (attached), I see that the paper isn't talking about competency/proficiency tests, they are talking about tests which are taken for error rate studies.

I now agree completely, using error rate studies (or studies that show that experts are better than laymen) to support accuracy seem misleading to me as well. I would say the probability of error is dependent on the complexity of the comparison, the probability of error is not determined by error rate studies.

When you calculate the positive or negative predictive value you would ignore any inconclusive results.

I would agree that in most of the forced decision tests an inconclusive should not be included as an error but it should not be included as a correct answer either. That is why the positive predictive value can be a more useful statistic to the court - When an examiner says ID how often are they correct? This calculation ignores inconclusives altogether.

The Slate and the Dror article misrepresent the statistics presented. There are reasons in a study design that would lead to more inconclusive results. Sample selection, time constraints, exemplars, lack of normal tools, lack of consults... I would challenge the authors to design a better study to explore their ideas rather than drawing conclusions out of whole cloth.

If there's a guy who can spot people using math problematically, it's the guy who tried to argue that a sample size of 6 was valid for an experiment.

But I digress. The paper cited is really just a reformulation of a paper that he already wrote which was widely critiqued.

This paper just comes across as kitchen sink problematizing. Decisions are only normative relative to a standard, if you fail to articulate a standard there is no 'ought', only an 'is' and to deduce what one ought to do from the data that is observed is actually a fallacy.

As usual, Boyd, you bring brilliant insights into the analysis of a claim, but this time you outdid yourself for brevity. Best read in a long time!