Possible areas to improve (needs assessment)

[Michele]

Good Morning,

I rarely post on the CLPEX chat board, instead I usually just send out emails, but today I feel that a topic may be beneficial to both groups.

This article came out this morning and it makes me think.

https://www.wesh.com/article/fdle-analy ... s/27024163

There’s not enough information in the video or the article to judge this situation but can we look at the limited information and speculate on if this could apply to any of us? Perhaps we can ask ourselves some questions which could help us modify our procedures in an effort to improve BEFORE it happens to us (or to our agencies).

Questions:
What does ‘inconclusive’ mean? Is it important that all forensic analysts use the same verbiage? Does inconclusive mean that better exemplars are needed (some agencies call this inconclusive while others label this as ‘incomplete’)? Does inconclusive mean that there are similarities but not enough to ID?

Do some agencies only verify ID’s? Is it valuable or necessary to verify everything before a report goes out and/or before testimony? I’m not suggesting full verification, just proposing that perhaps verification should be performed based on the complexity of the comparison instead of based on the conclusion.

Can being too conservative be a problem? How do we know if someone is too conservative without a standard/criteria?

How do we know if someone is ‘wrong’? Is it a vote (“we’d all ID it, therefore you must be wrong”)? Or perhaps there should be a criteria for what is an ID. Then accuracy could be judged on whether or not a conclusion was inside or outside of the criteria.

Do proficiency tests (PTs) test for the ability to ‘find’ consistency? Do they test for the ability to get the ground truth? Do they test for being too conservative or too liberal in conclusions? Do PTs test to see if a practitioner would use appropriate information (perhaps incipients) or ignore pertinent information? How do we test for what ‘should’ be used before determining what ‘should’ be used? Perhaps PTs could be improved BUT, maybe standards need to be determined before PTs can be improved? Which makes me wonder, are we using PTs because they really judge proficiency or do they just make us look good?

Lots of things to think about.
Michele

[Boyd Baumgartner]

What does ‘inconclusive’ mean? Is it important that all forensic analysts use the same verbiage? Does inconclusive mean that better exemplars are needed (some agencies call this inconclusive while others label this as ‘incomplete’)? Does inconclusive mean that there are similarities but not enough to ID?

Agreed on the point that the video/article doesn't give enough info. I looked for online SOPs from FDLE, but didn't find any. The AFTE has their standards for conclusions listed online.

An inconclusive result is noted as the outcome of a comparison in which there is:

• some agreement of individual characteristics and all discernable class characteristics, but insufficient for identification,
• agreement of all discernable class characteristics without agreement or disagreement of individual characteristics due to an absence, insufficiency, or lack of reproducibility,
• agreement of all discernable class characteristics and disagreement of individual characteristics, but insufficient for an elimination.

If you look up the SWGGUN criteria for identification from the OSAC Firearms and Toolmark Subcommittee page, you'll see that an identification is still listed as:

2.2.3 Currently the interpretation of individualization/identification is subjective in nature, founded on scientific principles and based on the examiner’s training and experience.

Based on these two documents which I'll give you are incomplete with regards to the case at hand, it would seem likely that the examiner in question was out of consensus for whatever cases were being scrutinized. I think, to your question about error being defined as being outside a vote or consensus, then at least the standard should be greater than 'training and experience', it should be on the data in the items being examined. Otherwise you end up with what amounts to mob rule.

It's not outside of some of the recent literature in our discipline however. The paper “Cannot Decide” : The Fine Line Between Appropriate Inconclusive Determinations Versus Unjustifiably Deciding Not To Decide, while essentially just reiterating the lyrics from Free Will by Rush, 'If you choose not to decide, you still have made a choice', attempted to frame the 'inconclusive' in a probabilistic way that wasn't well received by some. So there seems to be some contention around the topic, to the point of it potentially being framed in a way that is damaging to examiners.

[Pat A. Wertheim]

In my laboratory, we verify all decisions and conclusions. A decision of "suitable" or "not suitable" is verified and tech reviewed if a report is issued in the absence of a suspect comparison or an AFIS search. Likewise, all idents, exclusions, and inconclusives are verified. In the event of a disagreement between examiners over a suitability decision or a conclusion verification, the two examiners involved consult to see if they can resolve the question. (By our policy definition, a "conflict" arises when an erroneous conclusion is reported out, not if a disagreement can be resolved during the examination process. But that's a matter for a different discussion.) If two examiners cannot agree during a consultation, the matter is taken to the supervisor (also a CLPE) for a decision. If the conclusion is inconclusive, then the formal report reflects either that there is insufficient detail in the latent or that major case prints are required to complete the examination.

In the SWGFAST Q-Q graph, there are no hard lines, only fuzzy boundaries. This reflects the fact that subjective judgement is an unavoidable fact when reaching conclusions. Denying that reality is folly. If there were no subjectivity in the process, then it would not be an "opinion" and a blind AFIS technician could scan lifts into AFIS and submit the printouts to court as proof.

As for proficiency testing, a big problem is that a certain percentage of examiners subconsciously pay less attention to them because they are not real cases and are getting in the way of their backlog. Another percentage of examiners take them far more seriously than regular casework. Call it gun-shy. I suspect it is a minority of examiners who actually do a proficiency test with the same level of care, no more, no less, than regular casework. I think most examiners BELIEVE they do proficiency tests the same as real casework, but I doubt that is true. Until someone designs a foolproof method of blind proficiency testing that is cost effective, I doubt that problem will change.

HOWEVER, as already pointed out, too much is unknown about the case in the article/video to understand much of the detail regarding what is going on in Florida. I would suspect that inconclusives were not verified in the fifteen cases in question. If strict and competent verification had been performed, then I would not expect them to have all of those cases pulled now for reexamination. Maybe somebody from Florida could chime in and add some detail.

[Boyd Baumgartner]

As for proficiency testing, a big problem is that a certain percentage of examiners subconsciously pay less attention to them because they are not real cases and are getting in the way of their backlog. Another percentage of examiners take them far more seriously than regular casework. Call it gun-shy. I suspect it is a minority of examiners who actually do a proficiency test with the same level of care, no more, no less, than regular casework. I think most examiners BELIEVE they do proficiency tests the same as real casework, but I doubt that is true. Until someone designs a foolproof method of blind proficiency testing that is cost effective, I doubt that problem will change.

I've always laughed at this footnote in the PCAST report regarding proficiency tests.

133 Christopher Czyryca, the president of Collaborative Testing Services, Inc., the leading proficiency testing firm in the U.S., has publicly stated that “Easy tests are favored by the community.” August 2015 meeting of the National Commission on Forensic Science, a presentation at the Accreditation and Proficiency Testing Subcommittee. www.justice.gov/ncfs/file/761061/download.

[g.]

As a matter of course, firearms examiners only have ID's identified. I am sure there might be exceptions or instances, but SWGGUN does not go as strongly as SWGFAST where Exclusions and Inconclusive decisions are recommended (not required) to be verified.

So.... to Michele's point, it is unlikely they would have gone through a normal process. My guess is, after her retirement, someone needed to re-visit or re-do one of her cases for a trial, found a disagreement, which led to a quality review which led to disagreements of "sufficiency" for conclusions.

Like Pat said, "it's subjective". People are going to have differing sufficiency thresholds, assignments of weights, etc.

Good reason to at a minimum have some percentage of conclusions 100% verified (either before report is release) or as an audit later.

g.