Quality Assurance in Small Labs

[Jeremy Johnson]

Events over the last few years appear to have some Lab Managers thinking about Quality Assurance issues anew. It is my understanding that after the Mayfield debarcle the FBI instituted some changes in the work flow for individualizations. This included true blind verification of single print idividualizations because these were assessed as being the most likely to cause problems.

My Lab Director recently approached the latent print unit with a view to exploring what Quality Assurance measures we could put in place. We already are an accredited lab and as such we do proficiency testing etc. Neither examiner is currently IAI certified (but we both intend to be.)

The proposed arrangement, which is not likely to go ahead, would have seen the lab place an administrative Quality Assurance measure in the form of a numerical standard (I use the term "standard" cautiously I just can't think of a better one right now) over the top of the examiners activities. By this I mean as examiners our practice of comparing prints and calling identifications using ACE-V would not change. However, in cases where there were not present a certain number of level two details, these comparisons would be subject to some other process (third check, blind verification, alternative reporting procedure). In the proposal presented to us it was an alternate reporting procedure.

It was acknowledeged by the lab management that a count of level two detail is not a scientific basis for comparing prints, nor was it intended to be, it was simply a simple way of measuring quality. And even in this respect we all know it is not fool proof. The concept was not dissimilar to the FBI thought process....identify where the most likleyhood of error is (poor quality prints) and do something to mitigate the possibility of an error in those cases. As a small lab with only two examiners third checks and true blind verification are problematic to apply to all individualizations with the level of resources we have.

As you can imagine the floating of a policy based on this thought of boosting Quality Assurance, using a numerical count of second level detail as the measure, met with some fairly passionate negative responses. Issues such as defending the numerical value chosen in court, (why X instead of Y) and the fact that it may invite additional challenges where the examiners want to testify on an individualization which falls below the administrative "standard" were raised. As was the issue of trust in the examiners by the laboratory management.

What I am interested to know is if any other examiners have been faced with a push to introduce additional quality control measures (over and above accreditation and certification), what those measures looked like, especially if you are from a small lab and what the outcome was. I am somewhat torn, as an examiner who is philosophically a ridgeologist I want to keep it simple, I make my calls on prints and I testify to my opinion based on all I know of the fingerprint science. On the other hand I am a systems kind of guy and I can understand why a Lab Manager might want to put in place some robust quality assurance measures and have systems that ensure a quality product irrespective of who the examiners are. If those measures cant apply across the board then the system could at least recognise where errors are most likely and address those situations. FBI style.

I am also concerned that these types of discussions are ones that the Fingerprint Community should be involved in. I think there is a risk that if we are not at least involved in the discussion the day might come when forces outside the community seek to impose a solution to our perceived problems despite us.

[g.]

Jeremy,

I thought this was a very thought provoking question and that you presented this quite well. I wish to respond:

My personal opinion (and no reflection on any agency or entity by whom I am employed) is that I am not opposed to this type of QA mechanism.

I do believe there is a significant difference between a complex case and an easy case. I believe that b/c there is a distinction we might need different "rules" for the cases. A complex case may deserve full charting of the identification in the case notes, a complex analysis narative, blind testing of the result, etc. For a straightforward, clear as day, 100 minutiae comparison that would be a waste of time and resources.

So however an agency determines what is a complex case and what rules apply, is currently at the discretion of the scientists. I am ok with that too. If your agency decides to go down this path, as long as some discretion can be allowed, and the sole basis for opinion is not on the minutiae, then I wouldn't have a problem with it (as a quality assurance decision for deciding how to process/document/verify etc.). It's a slippery slope, but if done correctly, could be beneficial.

Now having said all that, I can most assuredly think of a case where I had 20+ minutiae in agreement and was on the fence. So the real question is:

IS a minutiae count an adequate identifier of complex cases. The academic answer may be 'no' (it's quantity + quality + specificity + other case details, bias, etc.). However, maybe operationally, in the real world, most easy cases have A LOT OF MINUTIAE....

So I would suggest before implementing this QA procedure, test it. Throw it back at your QA folks and get a good research project about this. Monitor the the number of minutiae in every latent claimed coming in for awhile (have people count, using standardized rules, measure mean and sd) and then also monitor what happens. Give a pool of comparisons with known answers and see if below say 10 minutiae, most of the time there was disagreement, yet if we had over 20, there was always 100% consensus...in other words, research it, and develop a QA procedure based on empirical evidence IN YOUR LABORATORY (with your examiners). Your results might be quite different than another laboratory.

And then report it/publish it AND please update us with the effects. Anytime QA procedures are introduced, they SHOULD be institituted to have an effect. What is the desired effect? More consistency? More accuracy? Keeping it conservative? etc.

That's my two cents. I am sure you will get some responses on the other end of the spectrum but that's just what science is about.

(PS Locard suggested a similar qa procedure 100 years ago in the tripartite rule. 13+ minutiae, easy, id can be positively establish w/o question, 8-12, can make an id, but need to have 1-2 verifiers, based on q&q, etc., below 8, can't establish id, but can make probabilistic statements....) So if you have to pick an arbitrary one w/o the research, pick Locard's and blame it on him!

If nothing else, try it, monitor it and measure the effect (if any). You could try a "trial phase" before deciding to institute. Nothing wrong with that before it became a standard policy.

g.

[Patrick Warrick]

Jeremy,

I would have to include myself as one of the individuals giving a "passionate negative response". I completely understand the desire and need for robust Quality Assurance Measures, but putting numerical thresholds based on Level 2 Detail would be far more problematic than helpful. Especially, if I understood this correctly, you would report the results out differently if the comparison fell below certain thresholds.

It is difficult to get most examiners, even those in the same office (mine included), to agree on Level 2 and Level 3 Detail. I know that there are thresholds present in other sciences, such as DNA, but the units that are used to reach the threshold are clear cut and unchanging. This is not the case in the comparative sciences. I do agree with Glenn in that I don't know how you would decide where to put that threshold. Plus, I seem to remember a study that was done (and of course I can't remember who did it and when) that concluded if given a numerical threshold and the examiner reaches one point under it, he is more likely to "stretch" a point to surpass the threshold.

As much of a pain as it is, my opinion is more supportive of the other QA Measures you mentioned such as blind verifications; third party verifications; a percentage of 100% peers/verifications.....


Glenn,

20+ minutae and on the fence?

[RL Tavernaro]

I do agree with Glenn in that I don't know how you would decide where to put that threshold. Plus, I seem to remember a study that was done (and of course I can't remember who did it and when) that concluded if given a numerical threshold and the examiner reaches one point under it, he is more likely to "stretch" a point to surpass the threshold.

I believe that study would be:
I. W. Evett & R. L. Williams, A Review of the Sixteen Points Fingerprint Standard in England and Wales, 46 J. FORENSIC IDENTIFICATION 49, 51 (1996).

Regards, RLT