OIG Progess Review

[ER]

We recommended that the FBI Laboratory revise the SOPs for Examining Friction Ridge Impressions to clarify that the “inconclusive” conclusion is available where an examiner, during the evaluation phase, is unable to achieve adequate certainty either as to the quantity and quality of detail in agreement or the sufficiency of the explanations for differences.

OIG FBI Mayfield Progess Review

http://www.justice.gov/oig/special/s1105.pdf

I still insist that the federal policy preventing the use of an inconclusive decision (unless you need better exemplars) and 're-analyzing' a print and deeming it to be of no value for ID is dishonest and misleading. The examiner has clearly come to an inconclusive decision but is forced to label the result as no comparison taking place. This policy logically leads to erroneous exclusions and missed identifications.

This Review recommends that the FBI change their policy to allow for the use of inconclusive when the examiner has not observed sufficient detail for an ID or an Exclusion. Has the policy been changed? The Review lists the FBI's reasons for not changing their policy, but never clearly says if there is a new policy or what the new policy is. Does anyone know if the FBI has changed its definition of inconclusive?

[Boyd Baumgartner]

Somebody better email SWGFAST and have them add 'adequate certainty' to the glossary.

Attorney: How certain are you that this impression was made by the defendant?
Examiner: I'm 100% absolutely positive that I am adequately certain.
Attorney: What is the likelihood of adequate certainty
Examiner: *inaudible mumble*
Attorney: Did I hear a 'niner' in there? What? Are we talking on walkie talkies.

Let me guess, we borrowed this term from some other discipline...

ACEV.jpg

[ER]

'Adequate certainty' is certainly a more accurate term than 'absolute certainty'. Absolute certainty overstates the value of an identification decision. Adequate certainty, while awkward, at least represents what actually happened. It describes a 'sufficiency threshold'.

However, these terms are not the point. The OIG recommends that the FBI revise their definition of inconclusive to improve their protocols. That's the point. Federal examiners are making erroneous exclusions and missing identifications because of the way their protocols are written. Change the verbiage to whatever you want. Call it adequacity if you want. On this point, the OIG recommendation should be implemented, and soon.

[Tazman]

'Adequate certainty' . . . . . . . 'sufficiency threshold'

Instead of taking a "proficiency test" we would take an annual "adequacy test." Instead of being "proficient," we could testify we are "sufficient" at our job.

Instead of a fingerprint being "unique," we could merely claim that it is "adequate."

If we can't meet the higher standard, let's lower the standard. Sweet!

[Michael W. Wieners]

To those of you recently posting on this topic, I would respectfully suggest a more thorough reading of the entire document. There are many things in a document that can be taken out of context and misconstrued without the benefit of oral/written explanations of what is meant in the text. While I respect your right to disagree and comment, I am disappointed by accusations of being misleading/dishonest and by the sarcasm used in your posts.

The FBI has gone to great lengths to improve our internal issues in a transparent manner. The FBI and OIG have used many outside experts to assist us in this effort. Many of the solutions we employed are the result of these collaborations and many of the experts that provided assistance are regular posters on this website. I apologize for being sensitive and I beg your understanding. I am very proud of where we are now and how we got there. Please look for future presentations on this document at various IAI conferences and bring your questions. We would be happy to answer.

[Boyd Baumgartner]

You can't really expect to just release something like this and expect people to ask questions and make comments only to your liking and at your convenience can you? Perhaps my copy of "How to Win Friends and Influence People" needs a dusting off or I read too much MAD magazine as a kid, but illustrating absurdity by being absurd is as old as time and serves a real point.

Maybe it's just me, but I think you can see a shift in the consumption of information by the community as a whole. You don't get a free pass just because you come from an agency with an acronymn. Who is saying things doesn't carry the same weight as what is being said these days, which is good. I'd attribute it to the social aspect of science in general at work and the amount of available information to those interested in consuming and evaluating it. Information is not monopolized by people selling or giving away 'training' anymore, Our agency discusses papers and concepts all the time.

With that social commentary out of the way, I won't take the time to refute every concept in the document, but I'll take on one so you can see an alternate perspective.

From Page 16 of the report, the section titled "The OIG's Conclusions Regarding the Causes of the Mayfield Identification" this point is listed:

Bias or “circular reasoning” caused by the original examiner’s use of features he observed in Mayfield’s known fingerprint to change his original analysis of the Madrid latent fingerprint.

The remedy to this cause is to implement a linear form of ACE-V listed on page 25 under the section "The FBI Laboratory's Implementation of the OIG's Recommendations" which states:

We also note that, even without objective standards for sufficiency, the FBI Laboratory has adopted other measures intended to reduce the risk that an examiner’s “gut” reaction might lead to an incorrect conclusion, including linear application of the ACE-V methodology; a disciplined, ridge-by-ridge approach to the analysis phase; separate documentation of the analysis and comparison phases; and blind verification in certain cases, including cases involving single identifications (like Mayfield) with the highest risk of error.

So, now that we can eliminate the 'out of context' excuse we can evaluate the problem and the remedy. To sum up the problem, circular reasoning was cited as a contributing factor. The remedy is to eliminate iterative,recursive reasoning which is a standard virtue of the scientific method. Circular and Recursive are two different things. So we have people from the FBI publishing papers putting ACE-V smack dab in the middle of the Scientific Method http://www.ncjrs.gov/app/publications/A ... ?id=251801 and then we have the Organization saying we use it linearly which takes it out of the realm of a Scientific Methodology by definition. It's like looking at the sky, seeing it's clear and determining it's going to be a nice day. Later, at lunch you see dark clouds and have to still believe it's going to be a nice day because you can't ever go back and update your beliefs in light of new information.


I have no clue why an examiner's gut reaction is even listed as having any sort of bearing. I could have saved you 7 years and millions of dollars to tell you that your Evaluations should be 100% verified, demonstrable, based on objective data, and justified to the consensus of the lab.

[ER]

In reading the OIG's report, I was very impressed by the diligence and completeness of the review. In speaking with FBI examiners, I have found them to be among the most well-trained and most talented in our discipline. I have asked the FBI examiners that I've met many questions, most related to the FBI policy on inconclusive decisions. Those examiners seem uncomfortable speaking about the topic. Reading between the lines, it seems that they are defending a position that they don't believe in. Specifically, that an inconclusive decision should only be used when better exemplars are needed, and the corollary, that when an identification or an exclusion cannot be made, the value decision is changed and no comparison is reported.

Forcing examiners to pick between Identification, Exclusion, and No Value for Identification leads to errors. Hopefully, research in our discipline by the FBI and others can show this.

I hope that the OIG's recommendation of changing this policy is implemented at the FBI. While this is the OIG's recommendation, after reading their review, I am unclear whether this change has been implemented. It had not been changed as of my last conversation with FBI examiners and with other members of SWGFAST.

While I understand the arguments that those at the FBI makes supporting their protocols, I firmly disagree with them on this point. Once the examiner begins the comparison phase, a comparison has taken place. Re-analysis is perfectly acceptable, but once a comparison has begun, a conclusion should be reported (even if the conclusion is inconclusive). I maintain that it is dishonest to conduct a comparison, and then not report that one has taken place.

I apologize for any sarcasm in my posts, it was aimed not at the FBI or the OIG but at the other posters that seem to be missing the point. Boyd now brings up a good point. Limiting ACE to a linear process is bad policy. Not only does this contradict accepted practice in the discipline, but it contradicts the FBI's own policy of re-analyzing a print and deeming it of no value.

Again, I applaud the efforts to improve your agency and the discipline as a whole. I was most impressed with the recently published Black Box study. However, there are still glaring issues that need to be addressed.

[Michael W. Wieners]

ER- I understand and appreciate the comments on linear ACE-V. I am not going to get into the nuts and bolts here but I will say that every step of ACE-V is documented in the case notes, including all material, notes and conversations that led to original decisions and changed decisions. Nothing is witheld or hidden from the contibutors, attorneys or courts.

Also, all of the OIGs recommendations were addressed to their satisfaction. That's why all but one are now closed and the remaining one is resolved and will be closed once the Capitol Case Reviews are concluded.

Mr. Baumgartner - I also understand and appreciate your concerns; however, they are diminished by your use of sentences like "You can't really expect to just release something like this and expect people to ask questions and make comments only to your liking and at your convenience can you? Perhaps my copy of "How to Win Friends and Influence People" needs a dusting off or I read too much MAD magazine as a kid, but illustrating absurdity by being absurd is as old as time and serves a real point," and especially "Maybe it's just me, but I think you can see a shift in the consumption of information by the community as a whole. You don't get a free pass just because you come from an agency with an acronymn."

My words in the previous post, "While I respect your right to disagree and comment, I am disappointed by accusations of being misleading/dishonest and by the sarcasm used in your posts," should indicate that I do not expect a "free pass" from anyone. I do respect your right to ask questions and make comments. In fact, I welcome respectful, educated dialogue on any issue. We are always looking for ways to improve; however, the comment regarding a free pass and agencies with acronyms indicate to me that you have a bit of an inferiority complex. I know this last comment will incense you and I apologize but it needed to be said. We at the FBI Lab consider ourselves partners with all state, local, federal, municipal and tribal laboratories. No one here belives we are better than anyone else, merely collaborators in making the discipline stronger. The policies we have implemented apply to how we do our work. They are not meant to be a guide for how everyone should do their work. That is SWGFAST''s charter. We are keenly aware that the differences in our procedures may be perceived to have a negative impact to the rest of the community but we must and will take the necessary steps we feel will help us do our job as best we can.

[Pat]

See SWGFAST document "STANDARDS FOR EXAMINING FRICTION RIDGE IMPRESSIONS AND RESULTING CONCLUSIONS" at: http://www.swgfast.org/documents/conclu ... FT_1.1.pdf

Of particular interest to this discussion, go to Section 4.1.4.2, which reads:

4.1.4.2 There are commonly two approaches to the determination of suitability as
determined by agency policy:
• Approach #1: Only impressions of value for individualization are compared.
Value for individualization indicates an impression that is deemed to be
identifiable. When adopting this approach, impressions lacking value for
individualization are not further compared.
• Approach #2: Impressions of value for individualization or exclusion are
compared.
• Conclusions in the evaluation phase following both approaches are:
individualization, exclusion, or inconclusive.

The differences between "Approach 1" and "Approach 2" may seem minor, especially to people who have only worked under one approach and are not keenly aware of the ramifications of the other. The truth is that the differences are not minor, as evidenced by the discussion in this thread. It has been my observation that folks who have familiarity with only one approach will deny even the validity of the other approach. Until one understands both approaches and philosophically accepts the alternate validity of each, no discussion of the use or meaning of "inconclusive" can go very far.

I particularly like Boyd's analysis and conclusion regarding the OIG's insistance that ACE-V be applied as a purely linear process. Ashbaugh taught it as "circular," (Did he mean "recursive?", and would you mind explaining that one again, please, Boyd?) and I have taught it the same. Even if you testify that you apply the process in strictly linear fashion, your mind is constantly reanalyzing and adjusting based on new information. In that sense, the process can never be purely linear.

I guess my big problem with Approach 1 is that it takes away a large amount of examiner discretion. That may make it easier for people who don't want to explain why they did what they did under an aggressive cross examination, but I don't think it makes fingerprint examination better science.

[Boyd Baumgartner]

Mr. Wieners, I think you want the Forensic Psychology chatboard, that's two websites down on the left.....

There are now two people who've questioned the linear use of ACE-V put forth in the report.

I'll quote myself from the book I wrote and teach from, because you can't have an inferiority complex and not be a narcissist, right?

While this speaks specifically to Qualitative methodologies, it illustrates the recursive point better than the tired old hypothesis testing reference.

Qualitative methodology, like quantitative methodology seeks to reduce the phenomenon being investigated and attempts to code it, sort it into categories, and search for emerging trends and patterns. This is done through an iterative, progressive, recursive, holistic process of Noticing, Collecting and Thinking about data.

The process is iterative and progressive due to the fact that it is cyclical, but moves toward an eventual goal of understanding. For instance, while thinking about data, you may notice something new and collect it, which will cycle back into re-thinking about how the new data incorporates into the 'big picture‘. The process is recursive due to the fact that the researcher may go back to a previous part. For instance, while collecting data, you may notice something new, which causes you to collect the new data. The process is holistic due to the fact that each step contains the whole process.
For instance, when a researcher first notices something, they are already collecting it and thinking about it.

The processes of Noticing, Collecting and Thinking can be thought about in terms of what they are doing. In the Noticing phase, the researcher is merely analyzing a phenomenon and breaking up, separating, or disassembling the phenomenon into pieces, parts, elements, or units. In the Collection phase the researcher sorts the elements by coding them. Lastly, in the Thinking phase, information is reconstructed in a meaningful way by searching for patterns, sequences, etc. [26] The patterns that emerge from such thinking are then evaluated by determining whether or not they are substantively significant. This can be influenced by anticipation of what is consensually validated by other researchers. Ultimately, this is a form of inter-subjective peer review.

If you're still confused, maybe this will sort it out: http://www.youtube.com/watch?v=VEgu7jdc_fs#t=25

[ER]

Michael,

I understand that the entire process if very thoroughly documented, and that every step is documented in the case notes. However, my understanding is that latents that are compared but re-analyzed are not reported as being compared.

Also, I saw in the review that the OIG had closed this issue. Please correct me if I'm wrong, but it sounds like the OIG recommended changing the definition of inconclusive, the FBI chose not to make changes on this point, and the OIG closed the issue. While the OIG is satisfied, it still recommends that the FBI change this policy.

The results of the recently published Black Box study surprised me in a couple of ways. The biggest being the high rate of erroneous exclusions. Since a large percentage of the participants in this study work for the FBI, I am very curious as to how the FBI's policy on inconclusive decisions might increase the rate of erroneous exclusions. Especially when compared to other similar studies. Other studies on comparisons have contained more difficult comparisons, but showed a lower rate of erroneous exclusions. They also had fewer participants from the FBI. The examiners at the FBI are among the best in the world. However, the approach in the FBI's protocols may lead to more erroneous exclusions.

For instance: If an examiner begins analysis on a latent print with 15 2nd level points, virtually no 3rd level, no core, and no delta with significant distortion, they would probably claim that latent as being of value for identification. After comparing the latent to the suspect, the examiner may not find any corresponding features in the knowns. Under the FBI's approach, the examiner must come to a conclusion of Exclusion. There are no other options. It's too good to re-analyze and call 'no value'. There's nothing that matches. The distortion prevented the examiner from seeing the matching features, and the verifier missed it too. Erroneous Exclusion. If the examiner had the option of coming to a conclusion of inconclusive based on the extensive distortion in the latent, this would prevent the error. An Exclusion decision should only be reached when there is sufficient detail in disagreement to come to that conclusion. An Exclusion decision should not be reached just because it's the only option.

But I'm sure this argument has been made before in more rarefied halls than this message board. My main reason for beginning this thread was to confirm that the OIG's recommendations had not been followed on this point. I appreciate the response and discussion.