Proficiency Tests and the 5-Point OSAC Scale

[Steve Everist]

I've been in several conversations regarding PT's lately. One of many discussion topics is the lack of Inconclusive in the PT's. For a test-maker, this would be a pretty complex conclusion to include, for a variety of reasons. This may have had some discussion previously on the board. Due to this complexity, we're pretty much given a 2-point scale for PT's.

However, with the OSAC pushing forward with the notion of a 5-point scale, how would this be addressed by RS&A, CTS, etc... when creating their tests? Whatever complexity exists to simply add inconclusive to the current tests, expanding on that seems like it would be a nightmare.

I'm not looking for discussion on the 5-point scale, specifically, but the application of conclusions other than ID/EXC in proficiency tests. If we can't have a simple inconclusive from a 3-point scale, how can we incorporate three additional conclusions?

And if that can't be done, especially if the 5-point is standardized or even adopted by some agencies, is it really a test of proficiency?

[josher89]

One could look at inconclusive as a lack of a conclusion; I can't say yes, I can't say no. I would imagine that if the pre-distribution test resulted in some disagreement of a latent (some saying it couldn't be ID'd or excluded), the PT provider would eliminate that one from the actual distribution test. They would want to know the expected results (when knowing ground truth) would result in an ID or EXC.

As far as how the expanded set would work, I think it could be applied to PTs but it would have to be up to the FSP to determine a "pass/fail" for an inconclusive result (or at least where the PT provider indicated an ID and a test taker indicated SSS). The FSP should determine what data they would accept for the conclusion. If this were the case, then a PT provider could continue to crank out their tests as is but allow the FSP to address an inconclusive result. Basically put this on the FSP and not the provider.

[Boyd Baumgartner]

One could look at inconclusive as a lack of a conclusion; I can't say yes, I can't say no.

Even this is problematic in some sense though because it would require standards for ID/Exc and the Inconclusive standard would then be the inability to rise to this standard.

Correct me if I'm wrong, but I don't think the current proficiency tests even define ID/EXC other than to say something to the effect that 'these prints have been determined to be sufficient for identification when that is the case'. It goes back to the age old conundrum of defining sufficiency.

[Mike French]

I support the concept of a 5 point scale, but as this post aptly points out it is difficult to standardize in the current state-of-the-practice, without more objective measurements. I'm also concerned with how this will be applied to tenprint work. What happens to criminal history reporting when a tenprint comparison falls short of the identification threshold, yet is considered highly likely?

[Alan C]

I support the concept of a 5 point scale, but as this post aptly points out it is difficult to standardize in the current state-of-the-practice, without more objective measurements. I'm also concerned with how this will be applied to tenprint work. What happens to criminal history reporting when a tenprint comparison falls short of the identification threshold, yet is considered highly likely?

As a tenprint examiner, I'll venture a guess.

1) I can't see criminal history reporting changing much. If we're processing full sets we virtually always have sufficient information for ID or exclusion, and if not, agencies can just do what they do now--reject the prints and/or request better ones.

2) If we're doing a comparison for court and testifying on the results, I imagine we'd follow the prevailing methodology, so if we go to a 5 point scale, we'd follow that.

[Mike French]

Thanks Alan. So if I were designing the perfect AFIS, the system should offer the tenprint examiner ID, Exclude, and Inconclusive decisions, and an inconclusive decision would trigger a reject workflow. Or would you recommend a different approach?

[Alan C]

Since there's already a reject function, I'm not sure it's necessary to have an Inconclusive that triggers a reject. Of course some agencies might want that, but where I work I think it would be considered redundant.