Forensic Assurance PT Provider

[josher89]

Does anyone that monitors this chat utilize Forensic Assurance for any PTs but especially latent print comparisons and latent print examinations?

[Snyder22]

We've used them for processing, but not comparisons...not sure if that's what you're looking for!

[josher89]

I have an interesting (VERY INTERESTING) thing to discuss about their winter 2022 comparison test. I'm still going to an internal process since I took the test but I will post more when I can.

[4n6Dave]

We currently use FA for comparison PT tests and have used them for processing ones in the past.

[josher89]

Follow-up on this test:

They included a latent print that was from the right interdigital area of a palm. One of the included exemplars did not have the interdigital recorded so I concluded that it was an exclusion to the others and inconclusive to that particular exemplar. Their manufacturer's report, while not calling it an error, said the correct conclusion was an exclusion.

I argued with my QA that that is actually an incorrect conclusion as you need to see something in order to exclude it. While the ground truth of the latent print was from and right interdigital and it was not made by the person who's right interdigital was missing (meaning FA knew that it was an exclusion but no one else taking the exam should have known that), it should have been scored as an inconclusive and not an exclusion (or not included on the PT at all).

To that same end, the folks that reported an exclusion are actually incorrect. My QAS was able to get FA to look at the test again and they wound up throwing out that result. I don't remember the exact number but about 70% of the respondents also said inconclusive but it wasn't the 75% or so needed to toss out the print. I had questions about the pre-distribution of the test and why this wasn't caught during that time, and I don't remember what the exact answer was, but they did assure us that this wasn't likely to happen again.

[anwilson]

Thanks for the follow-up on this. I was really interested in hearing more information. I agree with you that in that situation the "correct" answer if the test is to mimic casework would be Inconclusive/Incomplete to that exemplar depending on your agencies wording. Those who put exclusion are using the PT as a test and not something related to casework so I wouldn't necessarily say they were wrong. They answered based on the parameters of the test (either ID or EXC). This is a limitation of the proficiency tests (and certification test). There aren't clear criteria on appropriate conclusions that mimic how we would treat certain comparisons in casework. I imagine there are criteria used by manufacturers when selecting latents and knowns; however, I haven't been able to find what that criterion is.

Here's the wording we use in our SOPs to account for the fact that we have criteria for exclusions and incomplete that aren't reflected on a proficiency test:

i. The test manufacturer establishes the accuracy of the conclusions through ground truth; however, the correctness of the conclusions may be established through the SOP upon appeal. All conclusions recorded as part of the proficiency test shall conform to the Decisions and Conclusions SOP and are expected to follow the “Latent Print Guide to Exclusions” where applicable.
ii. Answers are generally based on the published ground truth by the test manufacturer. Conclusions that conflict with the test manufacturers shall be counted against the score unless reversed upon appeal.

If an examiner noted incomplete for that exemplar on a PT in our unit, they could argue that incomplete would be the appropriate conclusion per our SOP; therefore, it wouldn't be treated as an error if the appeal was successful (in your situation it would be a successful appeal). They would pass the test and it would be noted that their conclusion complies with our SOP.

[josher89]

They answered based on the parameters of the test (either ID or EXC)

FA does allow for an inconclusive decision so I was actually really hoping they included this latent as a way to create a PT that have Inconclusive as the correct option so we can 'normalize' its use as a conclusion a bit more. I don't agree with the cert/re-cert test allowing for inconclusive but making you take it over again - you may as well say exclusion when it comes to just the test part.

We do have policies in place to account for things outside of what is the expected result on a PT and say that a successful completion of any CAR associated with a PT error can still be considered a successfully completed proficiency. I think an erroneous ID would not fall within that policy though - I don't want to test it!!!

Anyway, I was very pleased to know that FA owned this example, did a root cause after I brought it up, and chose to re-issue the manufacturer's report. I would have been okay within my agency's policies, but I wanted to make sure this was cleared up since we have to send our results to ANAB. I didn't want them thinking we have someone who failed a proficiency by result only and not by policy.

The OSAC document on Proficiency Testing does a good job of making sure a PT can be used to test two things; examiner performance and lab policy performance. It gives tolerances for dealing with examiners getting results outside of the expected results which is also a huge bonus for labs that are seeking accreditation. I like how your policy is worded but I wonder if it should be up to the PT provider to reverse a response (think they have their heels dug in and don't want to change it). Ultimately, the FSP gets to decide if a person is competent and proficient at their job; using PTs is a way to demonstrate that but also shouldn't be the only thing used. I think AR and TRs are probably the best way to demonstrate it (outside of knowing ground truth) routinely.