CLPEX Chat Board

If you have thoughts to add about the praise or problems with this practice, feel free to add those too.

I changed the guest permissions on this forum so that you don't have to have a registered account to vote in the polls.

Proficiency tests are to be treated as casework. As such, PT tests should be verified. In reality, it would be better if PT tests could look just like casework but that would be extremely improbable for most agencies.

Are errors (including both misidentifications and missed exclusions) then treated according to your SOP?
Also, what would you do if it was revealed you submitted an erroneous conclusion based on the proficiency test answers being published?

If the miss wasn't reported but found in review, it is handled during tech review /verification. Accreditation is pretty clear (not perfectly clear) that successful PT tests are either getting all of the expected results or conducting a successful CAR to determine why a result may be unexpected. That is how it's written in our UTM.

A proficiency test is about the process, not just about getting the right answers. I am aware of some labs that will have everyone take the test, they go over the results, and submit one answer sheet to say the whole lab is proficient. I don't know that I agree with this approach as accreditation does require PT tests for individuals on a specific schedule but that's up to the lab to determine and their accrediting body.

If there was an erroneous conclusion reported, a CAR would be conducted to determine the cause of the error. If a successful CAP was implemented and verified to be effective, then the PT test would be considered successful.

What is the purpose of a proficiency test?

If the purpose is to test the individual examiner, then it should not be verified. But if the department's policy mandates that the proficiency test be handled like normal casework, then it should not be used without verification to test the individual examiner unless proficiency tests are exempted from the verification requirement.

If the purpose is to test for systemic problems in the laboratory, then the results should be verified, just as normal casework is verified.

My agency does not verify results sent to CTS, even though we purport to handle the proficiency like normal casework. But only one examiner's results are sent to CTS unverified. The other examiners verify each others' work as normal and their results are not reported to CTS. Their tests are deemed to be internal proficiencies. If the examiner chosen for CTS for the year makes a mistake, he/she gets a CAR. Be very, very careful when it's your turn in the barrel.

If you are submitting comparison conclusions to any reporting body and are not getting those results verified, you are not conducting ACE-V. Therefore, you shouldn't be able to say that you are any better or any worse if you get all of the correct results (you 'guessed' right) or incorrect results (you 'guessed' poorly) on PT tests. There is a flaw in thinking that we are only testing the examiner on a PT. If that is truly the case, then an internal PT is all that is needed. There should be no reporting results to any outside body without a verification. You wouldn't do that with casework so why do it with a PT? If you know you need to be extra careful when it's your time inthe barrel, I think the ol' pucker factor goes up magnitudes and creates additional pressure on that examiner.

If a PT test is sent out without verification, you are setting up an examiner to fail. Pure and simple. What happens to an examiner if a PT test was submitted (and not verified) and they got an incorrect answer? Just a CAR? Are they punished, suspended, or fired? If a mistake is made when ground truth is known, this is the best time to make it! It provides a learning opportunity to determine why the error occurred and additional training if needed.

We need to get past vilifying examiners for making mistakes. We don't want to make them. That's why we should be verifying everything.

We always thought that our proficiency tests were designed to check the entire lab system from property and evidence checkout through comparisons, verification, report writing, and review.

However, a couple of years ago an analyst had an erroneous exclusion on a proficiency test. The verifier handed the comparison packet back to original examiner, and since it wasn't a regular case, didn't point out the error, but just said that the conclusions should be reviewed. The original examiner found the mistake, changed the answer, and documented everything according to procedures.

When ASCLD/LAB came through and looked at the report and notes for the proficiency tests, they said that PT's should be treated slightly differently from actual casework. It's fine to have it go through all of the normal steps of the lab's process, but if there is an erroneous decision, then the examiner failed the test and needs to take another one.

In our lab that examiner took a second PT, passed it, and everything was fine.

I am not aware of anything in 17025 that sets pass/fail criteria for a PT. I can only surmise that this may actually be covered in a DIFFERENT ISO standard (ISO/IEC 17043:2010), of which I don't have a copy.

5.9.1 The laboratory shall have quality control procedures for monitoring the validity of tests and calibrations
undertaken. The resulting data shall be recorded in such a way that trends are detectable and, where
practicable, statistical techniques shall be applied to the reviewing of the results...

5.9.2 Quality control data shall be analysed and, where they are found to be outside pre-defined criteria,
planned action shall be taken to correct the problem and to prevent incorrect results from being reported.

emphasis added by me

Our "pre-defined criteria", consists of a memo outlining what the expected results of a PT are before it is given to analysts. For example, one could state that the expected result would be "in agreement with an external provider's reported ground truth" in the case of an external PT, or where the actual ground truth for the PT is recorded for internally-created PT's. This is also where an agency may choose to give some leeway based on their procedures for inconclusive results, regardless of the ground truth.

We do verify PT results (internal and external) because in that regard we do treat them just like casework. The only difference for us, is that we don't create new evidence photos since we are going to retain the originals and we don't do AFIS searches on unidentified latents. The catch here is how your procedures are written and how good of a conflict resolution procedure do you have. The intent of the standard is clearly and explicitly stated: "to prevent incorrect results from being reported." We have got to get past this idea that we can't make any mistakes at all. It's just not reasonable. Yes, we should have robust procedures in place to prevent errors from occurring. Yes, we should have controls in place that detect and weed out those practitioners who don't have the skills or abilities. The consequences should be proportional to the actions (or inactions).

What I'm truly wondering though, is if this thread was prompted by a certain PT whose ground truth was recently released and sported an unusually high number of false positives from a select few practicing examiners from 2 unknown agencies. (Sorry, just realized I typed that out loud )

The answer is decidedly 'yes' to your assumption about the PT results that were just released. I plan on starting a new thread.

That might turn into a candidate for the notorious threads forum. The ethical implications alone...