The Curious Case of Curated Conclusions

[Boyd Baumgartner]

The Spring 2018 Proficiency test results were just released on the Ron Smith and Associates site (attached) with some rather interesting results. There were some 'inconsistencies'. RSA uses that word for answers that don't meet ground truth because they allow for each agency to define standards for conclusions and some of those standards may not line up with the ground truth. This doesn't necessarily mean there's an error, although legitimate errors will also be tagged with this verbiage. So for instance, if your agency uses 'inconclusive' to mean 'too much similarity to exclude, not enough to ID' then that's an 'inconsistency' because all the latents have been determined to be sourced or excluded even though inconclusive may be a valid conclusion for your agency. 'Inconsistency' will also include a straight up bum ID and erroneous exclusion as well.

That being said, the summary report then breaks out 'inconsistencies' in various dimensions of the data in various charts and graphs. So looking at the beginning of the report, we can see right away that inconsistencies seem to be lopsided towards a very specific demographic of people/agencies in the discipline; non-certified, sworn personnel from non-accredited labs (at least based on this test anyway).

The report actually outlines all the responses by all the people who submitted the test, anonymously using their TEST ID. The inconsistencies are highlighted in yellow. As we saw from the previous thread regarding proficiency tests, not all agencies submit them in the same fashion. Some only choose one person from the agency, and some submit all/more than one person per agency. The thing that actually got me interested in this data was the 'Participants Additional Comments' where the TEST ID is associated with a comment as well.

Brilliant test of latent Print Comparison skills, you have to look beyond what you see, take into consideration pressure, slippage and thickness of ridges. Your ridge counting and ridge tracing plays a vital role. As a manager it was great to lead by example. Thanking you.

I found the statement 'you have to look beyond what you see' a rather interesting turn of phrase, so I scrolled up to the data for that same TEST ID and what did I find, but 5 bum IDs. But that wasn't all, there were actually 3 other people who made the exact same bum IDs including the finger to which the latent was falsely attributed.

BumIDs.jpg

That wasn't the only pattern of similar answers that was present, as another group of 3 had the same thing happen. Errors on the same latents with the same incorrect results.

BumIDs_2.jpg

The common sense and most charitable reading of the data is that these two agencies verified their proficiency tests, but rather than catch error, the errors were propagated. This becomes concerning in light of the pie charts and graphs when plotted against hours of training and years of experience, especially seeing that 'training and experience' is becoming more of a problematic concept in the articulation of our process and conclusions because it doesn't ensure or even speak to accuracy.

training_experience.jpg

Obviously, I don't know exactly what happened, like whether or not the manager associated with that comment influenced their staff. Did people compare notes prior to submission, or was this just extremely ineffective verification? What do you think accounts for this data? Also, considering there is a huge difference between best effort and bad results vs sloppiness or lack of effort, is there an ethical obligation for proficiency test providers to report this level of error?

[NRivera]

Another observation: How many examiners out there "expect" there to be 10 ID's and 2 exclusions on a given proficiency test?

I've been taking PT's annually since around 2005. I've seen a few of the older ones in training (1995 anyone?). My very personal perception, is that CTS created this artificial expectation. I've been careful not to let that influence my test responses, but I'd be lying if I said that it didn't prompt a cause for concern and another look at the images to double-check when my results didn't meet that criteria. I took this particular test as an internal PT and it threw me for a loop that there were that many exclusions because I have never seen that in a PT before. How much did that have to do with these errors? Maybe we'll never know for sure but it's worth considering.

I'm also curious to hear the opinions as to the ethical implications here. In an ideal world, these errors should prompt a pretty robust corrective action response from the agencies themselves. Accredited labs have a mechanism for addressing these things, but according to the report these are not accredited labs. I foresee some uncomfortable lines of questioning on the stand for some.

[Boyd Baumgartner]

Since you mention it, Dr. Boraccho mentioned an aspect of it, and the Human Factors Guidance Document draft was just released and the White Box Study mentions it, I think it's worth discussing.

How many examiners out there "expect" there to be 10 ID's and 2 exclusions on a given proficiency test?

My agency does not verify results sent to CTS, even though we purport to handle the proficiency like normal casework. But only one examiner's results are sent to CTS unverified. The other examiners verify each others' work as normal and their results are not reported to CTS. Their tests are deemed to be internal proficiencies. If the examiner chosen for CTS for the year makes a mistake, he/she gets a CAR. Be very, very careful when it's your turn in the barrel.

Because participants knew that they were being tested, some may have reacted to the test (“Hawthorne effect”) by trying harder than usual to reach conclusions, or by being more or less cautious than during casework.

There's definitely something to be said about the use of internal proficiency tests for this very reason. I completely relate to the CTS 10 and 2 phenomenon and would go further with it to recognize that the distribution of the suspects who were ID'd had a pattern as well. Steve and I would often talk about how the test taking strategy is contrary to that of casework. In casework, if you ID someone, you immediately search all the latents against that person because obviously, they're now associated with the case. In proficiency testing, there was usually a distribution of the IDs to 3 of the subjects with something like a 3,3,4 split. This meant that once you had ID'd three latents, you could effectively optimize your search strategy by putting that person to the side, which is the opposite of how I described case work.

The big idea being that people behave in tests differently than they behave in casework when they know they're being tested. Was that a factor here? Potentially, however I can't say that I can understand how expecting a certain outcome makes minutiae appear in the same spatial location, sequence and orientation of non matching fingerprints.

[timbo]

If I've deduced these results and manufacturer's observations correctly, here are some (unconfirmed) details. I'm not sure how accurate this is, but the way that I read the results and the observations, this is the only way that it could be in my opinion - unless I've misunderstood an observation.

Two agencies were involved in performing 'inconsistent' results - which ranged from an inconclusive finding to an erroneous exclusion to an erroneous ID.

The first agency, which is accredited, had two responses with inconsistencies: one response had 3 inconclusive findings, and another response had one inconclusive. Neither respondent is certified. This accounts for the 4 inconsistent findings by accredited agencies.
Total: 4 inconsistent findings by the first agency.

The second agency, which is not accredited, had:
- Four examiners with 5 erroneous IDs each (all the same erroneous findings as each other)
- Seven examiners with 1 erroneous ID and 1 erroneous exclusion each (all had the same erroneous findings as each other)
* One of these seven examiners is certified, accounting for the 2 inconsistent findings for certified examiners
- One examiner with 1 inconclusive finding
Total: 35 inconsistent findings by the second agency.


One of the examiners (from either agency) who had 1 inconclusive finding was civilian. All the rest were sworn.

Anybody else see any other possible scenario that I'm not seeing??

[Bill Schade]

I don't think this is about:

accredited labs vs non-accredited labs,
sworn vs civilian,
time in the business(training and experience)
certified vs non-certified examiners

I think it is about examiners believing they can reliably make exclusions because they were told they should be able to without any real rules about when that decision is apropriate. Saying that ground truth is known on a test and there are only two choices is not testing application of rules or methods.

The hardest comparisons I have ever done were the ones where I didn't "see" a match.

Lets take this a little further and ask the LP certification board to publish this type of analysis on re-certification test results. I think the diccscipline would be shocked to know the number of inconsistencies among certified examiners. Changing the rules on erroneous exclusions at last years IAI conference is just a small step and does not solve the issue of examiners "missing" a match, either in casework or on a test.

Is there any examiner reading this today who can say "I have never missed one" Is that really the best measure of ones ability?

[Snyder22]

[quote="Bill Schade" post_id=18014 time=1527005786 user_id=47] Changing the rules on erroneous exclusions at last years IAI conference is just a small step and does not solve the issue of examiners "missing" a match, either in casework or on a test.[/quote]

Bill - I wasn't able to be at the IAI last year, could you elaborate on that statement/what rules were changed?

Just want to make sure I'm caught up on everything, sorry if this is rehashing old news for anyone else.

[Steve Everist]

When looking at the results, and not breaking down the many graphs, my initial thought was that the inconsistencies (non-Inc) fell into two groups:

Those that made the same five mis-ID's
Those that made the same mis-ID and same mis-Exc

I separated out, in my mind, those that made the Inc inconsistencies as being separate outliers as one had three (should have been EXC) and one had one (should have been EXC) and one had one (should have been ID).

But regarding the numbers of the first two groups, the graph that seems the most confusing is Appendix 11. It says that 35 inconsistencies were not reviewed (verified) by another LPE prior to submission. But the number of inconsistencies of the mis-ID and mis-Exc variety total 34. How is it that these (or at least 30 of them) were not reviewed by someone else, yet returned two groups of identical misses? What's the likelihood associated with that happening?:

Appendix_11.JPG

[Steve Everist]

This addresses some of my thoughts, taken from the Manufacturer's Additional Observations:

[Steve Everist]

If I've deduced these results and manufacturer's observations correctly, here are some (unconfirmed) details. I'm not sure how accurate this is, but the way that I read the results and the observations, this is the only way that it could be in my opinion - unless I've misunderstood an observation.

Two agencies were involved in performing 'inconsistent' results - which ranged from an inconclusive finding to an erroneous exclusion to an erroneous ID.

The first agency, which is accredited, had two responses with inconsistencies: one response had 3 inconclusive findings, and another response had one inconclusive. Neither respondent is certified. This accounts for the 4 inconsistent findings by accredited agencies.
Total: 4 inconsistent findings by the first agency.

The second agency, which is not accredited, had:
- Four examiners with 5 erroneous IDs each (all the same erroneous findings as each other)
- Seven examiners with 1 erroneous ID and 1 erroneous exclusion each (all had the same erroneous findings as each other)
* One of these seven examiners is certified, accounting for the 2 inconsistent findings for certified examiners
- One examiner with 1 inconclusive finding
Total: 35 inconsistent findings by the second agency.


One of the examiners (from either agency) who had 1 inconclusive finding was civilian. All the rest were sworn.

Anybody else see any other possible scenario that I'm not seeing??

Although I just put the 2nd manufacturer's observation above, I completely glossed over the first one stating that there were only two agencies submitting the inconsistencies. The way they break down seems confusing, especially considering the grouping and lack of self-reported review before submission.

[Bill Schade]

"Bill - I wasn't able to be at the IAI last year, could you elaborate on that statement/what rules were changed?"



Although an erroneous exclusion results in loss of certification (same as an erroneous identification), you can retake the comparison test over in six months instead of waiting a year and taking the full certification test.

[Boyd Baumgartner]

Lets take this a little further and ask the LP certification board to publish this type of analysis on re-certification test results. I think the diccscipline would be shocked to know the number of inconsistencies among certified examiners.

Unless you know a judge who owes you a favor, I don't see them being very forthcoming with that info.

https://www.theiai.org/certifications/l ... nt/faq.php

8) Are my test results made public?

The only time the test results can be made public is in the case of an official court subpoena. In this case, any subpoena for results would first be discussed with the IAI Legal Counsel.

[Bill Schade]

You could always take a poll here on CLPEX

"As a certified LPE, have you ever had a missed Identification reported as an exclusion?"

[Pat A. Wertheim]

"As a certified LPE, have you ever had a missed Identification reported as an exclusion?"

Of course! Any examiner in this business for even a short length of time has made erroneous exclusions. In the days before exclusions were verified (which started in about 1993 or 1994 at Arizona DPS when I was there), there was no safety net for exclusions. In one case, I reported out an exclusion which, six months later, I identified using a different set of exemplars. I went back and checked my first exemplars, and sure enough, the ident was there but I had missed it. In court, I had to explain to the jury that I had missed it the first time through. They understood and the identification from the second set of exemplars was accepted. I never tried to hide the fact or rationalize the erroneous exclusion. I missed it, plain and simple.

In those days, there was no stigma to an erroneous exclusion, but a single erroneous identification was a career terminator. Now, it seems, an erroneous exclusion carries the same penalty as an erroneous identification, but neither is particularly detrimental to a career.

Maybe the old days were too hard on an erroneous identification and too easy on an erroneous exclusion, but I think the pendulum has swung past the middle point and today we are too easy on erroneous identifications and too hard on erroneous exclusions.