Latest CTS proficiency test results

[Tazman]

Bill,

I'm flabbergasted! I can't get it to come up, either. Must be a server down somewhere. Surely an organization that is dedicated to science wouldn't remove a page of data, would they?

[David Fairhurst]

Number crunching after re-reading the summary page

The way I read that page is that when the 199 students(?) are taken out, of the 376 remaining, 7 made erroneous IDs (the eighth just forgot to write item numbers, I guess). That equates to 0.15% erroneous ID rate. A further 23 made misses for a 0.51% miss rate. Note that that is individual examiner error rates only (and only if my math is correct), not error rates for the ACE-V process.

Good reading timbo, but your maths doesn't quite add up.

If you take the 199 from the total 607 participants that leaves 408.
Of those 408, 376 had perfect scores and one just missed the responses for the NI's (376 + 1 = 377)
23 missed at least one ID but made no erroneous IDs (377 + 23 = 400)
8 made one or more erroneous IDs (400 + 8 = 408)
(4 of these 8 also missed at least one)

So that gives 92.4% of candidates perfect, 6.62% of candidates missing at least one ID, and 1.96% making erroneous IDs.

I have found the one who didn't write the item numbers, XGRH4R-515. I'm ignoring him now.

A comb of the data reveals 65 candidates with misses but no wrong IDs. (obviously some of these are in the 199 discounted candidates) 33 candidates only missed one. If we assume that the 23 with only misses and the 4 who made wrong IDs and misses, only missed one each, that gives us a miss rate of at least 0.74% (27 misses out of 407x9=3663 possible IDs)
Similarly, if you assue ony one wrong ID for each of the 7, our minimum erroneous ID rate is 0.14% (7 out of a total 407x12=4884 latent examinations)

Sorry, no whisky for anyone testifying to these numbers.

[Bill Schade]

While the statistics are interesting. My original question remains

Who the heck took this latest test. Are agencies at fault for administering this proficiency test to non practitioners. It seems to me that agencies should be alarmed if these are the practitioners they are using on casework

[Steve Everist]

It appears that they removed the results. Did anyone happen to save a copy of the PDF?

[Ernie Hamm]

Yes

[Shane Turnidge]

This type of data is underwhelming to me. Anyone who has participated in Glenn Lanngenburg's fingerprint statistics class would know that this type of data is pretty standard.
The big question for me is will the IAI accreditiation body address the problem and how? If being a CLPE is to hold any weight something needs to change.
Shane Turnidge

[Bill Schade]

Shane

How does this conversation have anything to do with the IAI, certification, or accreditation?

[josher89]

CTS--I just called them and they "were just made aware" of the issue of the summary report not being accessible online anymore. So, we'll see how long it takes to get it back up and maybe Ernie can compare his copy with the 'fixed' copy (in case there are changes/errors in the original reporting).

[timbo]

The link is back up now with the comment on the summary page:

(Summary comments updated 5-21-13 to include information regarding test objective)

Some of that information is "this test was designed to allow participants to assess their proficiency...."

[bficken]

At least they did us the favor of putting this statement in the disclaimer:

"...the results compiled in the Summary Report are not intended to be an overview of the quality of work performed in the profession and cannot be interpreted as such. The Summary Comments are included for the benefit of participants to assist with maintaining or enhancing the quality of their results. These comments are not intended to reflect the general state of the art within the profession."

[Shane Turnidge]

I'm sorry Bill, I probably should have included a personal experience I had when I took Glenn Langenburg's Fingerprint Statistics class. We were shown results from all the participants within the class. I asked the question how many participants in the class were certified examiners. I believe I was the only one that was not. The results in the class seemed very similar to the CTS data published +/- a point or two.

If my experience and the CTS data prove to be systemic then I think it would in the interest of the IAI to take a closer look at their accreditation process, particularly given that some states/jurisdictions require IAI certification. What we shouldn't accept is the false positive rates identified by the tests.

Shane Turnidge

[Steve Everist]

If my experience and the CTS data prove to be systemic then I think it would in the interest of the IAI to take a closer look at their accreditation process, particularly given that some states/jurisdictions require IAI certification. What we shouldn't accept is the false positive rates identified by the tests.

Shane Turnidge

I'm confused by this. Neither the IAI or nor CTS accredit agencies. And there is no connection between CTS tests and the IAI's certification process.
So I'm not sure how CTS and the IAI are connected.

When you say there are states/jurisdictions the require IAI certification, are you talking about the states (via laws, etc...) themselves or the agencies within the states and their employment requirements? Or are you talking about accreditation and that some states require accreditation (which does not require certification) in their forensic labs?

I'm just unable to connect all the dots in what you're saying.

[timbo]

Perhaps this can help:
- The individual examiner is certified (by the IAI in the USA)
- The agency/lab/unit is accredited to ISO 17025 or 17020 (by ASCLD/LAB, FQS, etc. in the USA)
- Accreditation and/or internal organization requirements mandate proficiency testing for individual examiners.
- CTS is just one company of a number that provide external proficiency tests for agencies to use (CTS is the most widely used worldwide to my knowledge)

Certification / accreditation / proficiency are all very different things.

[Alan C]

My favorite comments from test takers:

"Please inform us in time and give us lunch."

"My eyes are giving me the problem."