Latent Print Examination and Human Factors

Discuss, Discover, Learn, and Share. Feel free to share information.

Moderators: orrb, saw22

g.
Posts: 247
Joined: Wed Jul 06, 2005 1:27 pm
Location: St. Paul, MN

Re: Latent Print Examination and Human Factors

Post by g. »

do you have any comments on the content that is in the report?
I have to agree and would like to hear more about what's in the report. But the exchange between you guys was interesting--so much so, that I have to ask a few questions to clarify.

Boyd,

I have to admit these terms are new to me and you have correctly pointed out I, and many in this field, are lacking in this background (e.g. semiosis, valuation, Iconic representation, knowledge of aesthetic judgements).

So is your view the following: that discussing these terms and scientific philosophy will be more helpful to the courts and laypeople in understanding the value, strengths/limitations of fingerprint evidence? Right now an awful lot of attention is being spent on teaching people how to articulate and compute error rates, sufficiency, and measure (although very imperfectly) the rarity/strength/weight of features (currently limited to patterns and minutiae) with statistical models. After reading your posts, I conclude that you think the profession is moving in the wrong direction.

Do you think we should not be discussing error rates and statistics AT ALL? And we should just focus on a better way to articulate what we do using terms like: semiosis, valuation, Iconic representation, knowledge of aesthetic judgements, neuroaesthetics, gestalt organization in the visual cortex, 'infinite semiosis', temporal aspect-relativity (synchronic/diachronic), modalities, Indexical signs, valuation of symmetry is modal to the valuation of information, etc. I confess, I don't know many of these terms and have not thought about how one would express them to a jury.

I guess I'm trying to get a vision of what this testimony would look like. Since currently I testify using the traditional "here's what ACE-V is" and here is the research that shows it can be applied accurately, reliably, and there is a valid basis to the decisions we report. I've never had a judge turn to me and say "well thank-you Mr. Langenburg for your answer on the rate of false positives in these studies, but could you please tell me about "how semiosis and neuroaesthetics" plays a role in fingerprint examinations.

How do you see examiners communicating and testifying using these terms? AND are they compatible with the current "trajectory"/trend of moving towards errors rates/statistics, OR they should REPLACE this trajectory, and we are headed in the wrong, and potentially worse, direction? Do you think these terms and explanations will meet the Daubert burden?

Let me ask one last question. How would you tackle a Frye/Daubert hearing?
-What studies would you use (any of the error rate studies? FBI black box? Cedric's JRSS paper, Dror papers?).
-OR Would you JUST stick to the premise of showing biological uniqueness (and use only those studies: Babler, Hale, Cummins, etc.)
-OR Not use any studies at all, but rather focus on articulating the process, under the framework of semiosis and aesthetic judgments.
-OR some combo?
-I'm guess I'm asking: what's your strategy for preparing for a Daubert hearing. Obviously you know mine, but I had not considered a different way.

Thanks in advance. Feel free to start a different thread if this is taking us too far off topic.

g.
ER
Posts: 351
Joined: Tue Dec 18, 2007 3:23 pm
Location: USA

Re: Latent Print Examination and Human Factors

Post by ER »

Here are some of my thoughts on Chapter 10: Summary of Recommendations. Overall, I agreed with most of their recommendations. Some of the recommendations I would agree to with certain caveats.
Because empirical evidence and statistical reasoning do not support a source attribution to the exclusion of all other individuals in the world, latent print examiners should not report or testify, directly or by implication, to a source attribution to the exclusion of all others in the world. (Recommendation 3.7)
Agree
The friction ridge examiner community and other experts should determine under what circumstances a qualified rather than an absolute conclusion is warranted. (Recommendation 3.8 )
I would only add ...or under what circumstances the current, absolute, categorical conclusions are the best way to report results.
A report and contemporaneous supporting notes or materials should document the examination to make the interpretive process as transparent as possible. Although the degree of detail may vary depending on the perceived complexity of the comparison, documentation should, at a minimum, be sufficient to permit another examiner to assess the accuracy and validity of the initial examiner’s assessment of the evidence. (Recommendation 3.1)
I agree, but would add that most comparisons are simple enough that they require little or no documentation of the interpretive process or that the documentation can be completed at a later date after the report is issued since the compared prints are never 'used up'. In most cases, the latent itself should be sufficient documentation to allow another qualified examiner to assess the validity of the conclusion. Obviously, tougher comparisons would require more.
Modifications to the results of any stage of latent print analysis (e.g., feature selection, utility assessment, discrepancy interpretation) after seeing a known exemplar should be viewed with caution. Such modifications should be specifically documented as having occurred after comparison has begun. (Recommendation 3.2)
I generally agree, but would suggest a simple system (like GYRO) to document these features.
Procedures should be implemented to protect examiners from exposure to extraneous (domain-irrelevant) information in a case. (Recommendation 3.3)
I would agree so long as I get to decide what's relevant.
Each agency or forensic service provider should define “suitable” or “sufficient” in its standard operations procedures. These guidelines should be as explicit as possible about what is expected for sufficiency determinations at different stages of the latent print examination process. (Recommendation 3.4)
While I agree that agencies should define suitability/sufficiency (value for ID, value for exclusion), good luck coming up with explicit guidelines that don't occasionally leave out a print that should be compared or include a print that shouldn't be. How can you really be sure that a print is identifiable until you identify it? The best you can really hope for is comparing all prints that might be identifiable, and then reporting inconclusive on the prints that weren't quite good enough. But you can't know for sure that it is or is not identifiable until you complete Evaluation.
When comparing latent prints to exemplars generated through AFIS searches, examiners must recognize the possibility and dangers of incidental similarity. Adjustments such as a higher decision threshold, stricter tolerances for differences in appearance, and explicit feature weighting need to be considered. Modified quality assurance practices for this scenario also should be considered. (Recommendation 3.6)
Generally agree.
The report of the examination should ensure that the findings and their limitations are intelligible to non-experts. (Recommendation 5.1)
I agree that the report and findings should be intelligible to non-experts and that interviews, depositions, and testimony should provide an understandable explanation of limitations, but disagree about listing limitations in every report. (Especially since there is no definitive list of limitations to include.)
Second, to maximize the likelihood that the evidence will be used properly, fully, and fairly at trial, the report of the results of an examination should:
a. Identify the latent print examiner(s);
b. Describe the items submitted to the examiner(s);
c. List the procedures used by the examiner to develop, visualize, or enhance the friction ridge impressions;
d. List all comparisons conducted;
e. State all conclusions with the method used to reach them;
f. Note any important limitations to the conclusions;
g. Indicate whether a verification was made and whether there was any conflict of opinion among examiners prior to the reported conclusions;
h. Note (or refer to external documentation of) any information about the case that the examiner(s) received;
i. Note the existence of additional documentation; and
j. Define important technical terms, either explicitly or by reference to an authoritative, readily available source. (Recommendation 5.2)
A, B, and D: absolutely.
C, G, H, and I: can stay in the bench notes and be given to counsel as soon as the case comes to court.
J and, in most cases, F: can stay in the protocols.
E: all conclusions should be in the report, the method to reach the conclusions will always be ACE-V and the explanation of that method can stay in protocols.
Forensic service providers should adopt codes of ethics that require testifying in a nonpartisan manner; answering questions from both the prosecution and the defense directly, accurately, and fully; and providing appropriate scientific information before, during, and after trial. (Recommendation 6.2)
Agree
The trial preparation process should address the presentation of technical information in lay terms, the organization of the direct examination, possible cross-examination, and the possible use of visual aids. (Recommendation 6.1)
Agree
A testifying expert should be familiar with the literature related to error rates. A testifying expert should be prepared to describe the steps taken in the examination process to reduce the risk of observational and judgmental error. The expert should not state that errors are inherently impossible or that a method inherently has a zero error rate. (Recommendation 6.3)
Agree
An agency that employs latent print examiners should establish requirements and guidelines for reporting, documentation, and testimony. The agency should review a sample of every examiner’s case reports, documentation, and testimony and affidavits at least annually to verify compliance with these requirements and guidelines. (Recommendation 6.4)
Agree
Management should foster a culture in which it is understood that some human error is inevitable and that openness about errors leads to improvements in practice. (Recommendation 9.1)
Management should employ a system to identify and track errors and their causes. (Recommendation 9.2)
Management should establish policies and procedures for case review and conflict resolution, corrective action, and preventive measures. (Recommendation 9.4)
All forensic service providers should be accredited by a recognized accrediting body. (Recommendation 9.3)
The latent print community should develop and implement a comprehensive testing program that includes competency testing, certification testing, and proficiency testing. (Recommendation 9.5)
Agree to all five, even though some smaller agencies will have a difficult time with accredidation.
Forensic service providers should require personnel to become certified through an accredited program. Certifying bodies should review current certification tests with the aid of professionals in test design and psychometrics. (Recommendation 8.7 in part)
Agree, but would clearly state that latent print examiners need to be involved in the test design as well.
Forensic service providers should institute medical surveillance for examiners with respect to activities specific to friction ridge examination. Corrective measures should be taken when appropriate and recorded. The program should include a baseline visual examination followed by annual vision testing to detect vision deficiencies that may affect interpretation of the evidence. (Recommendation 7.2 in part)
I agree with the concept, but think that further research into the medical problems that would affect examiners needs to be completed before implementing a program like this.
Forensic service providers should require personnel to participate in continuing education. (Recommendation 8.7 in part)
The latent print examiner community should expand the training of examiners in elementary probability theory to enable examiners to properly utilize the output of probabilistic models. (Recommendation 4.3)
The federal government should consider funding research to determine what educational and cognitive abilities should be prerequisites for training a latent print examiner. Forensic service providers should have systematic training programs that include these abilities as well as education in the scientific method, reading and understanding relevant scientific literature, communication skills, and methods for logically developing conclusions. (Recommendation 8.1)
Agencies should develop formal mentoring programs that define the responsibilities of the mentor, trainee, and supervisor. (Recommendation 8.3)
Certifying bodies should ensure that examiners have been trained to competency. Laboratory accrediting bodies should evaluate the effectiveness of an agency’s training program. (Recommendation 8.8 )
Because statistical information plays a fundamental role in weighting latent print feature evidence, training should include the best available empirical information and should educate examiners about probabilistic reasoning in using that information. (Recommendation 3.5)
The federal government should establish an expert group to develop a latent print educational textbook, practical exercises, and assessment tests (print and online) in consultation with professional organizations, senior latent print examiners, researchers with content knowledge, experienced trainers, and professionals experienced in curriculum development. (Recommendation 8.4)
The federal government should support a clearinghouse of materials from existing training programs as well as publications on latent print identification. (Recommendation 8.6)
Training materials should include topics beyond the technical aspects of friction ridge analysis, such as:
a. Documentation of work and case notes and written and oral communication;
b. Professional ethics;
c. Human factors issues such as fatigue, bias, cognitive influences, perceptual influences, and error;
d. Research methods; and
e. Legal and ethical aspects of expert testimony. (Recommendation 8.5)
The federal government should develop a forensic latent print examiner scholarship program to:
a. Fund the post-hire training program and, upon successful completion, provide tuition reimbursement, loan forgiveness, or other financial assistance to degreed latent print examiner trainees working in state, local, or private (not-for-profit) forensic laboratories; and
b. Reimburse the employer for a percentage of the salaries during the training period. (Recommendation 8.2)
Generally agree.
A structured approach that involves users throughout the entire design and implementation process should be followed when designing technology systems and the physical work environment. In addition, forensic service providers should perform usability testing to optimize user performance and user satisfaction before deployment. (Recommendation 7.1)
Agree, especially to the 'user satisfaction' part.
The federal government should support a research program that aims to:
a. Develop measures and metrics relevant to the analysis of latent prints;
b. Use such metrics to assess the reproducibility, reliability, and validity of various interpretive stages of latent print analysis; and
c. Identify key factors related to variations in performance of latent print examiners during the interpretation process. (Recommendation 3.9)
I'll almost never disagree to more research, but would add to this point that the measures and metrics that are developed should be tested and be proven to provide a benefit to the analysis before being implemented. Developing metrics does not guarantee that they will be useful or beneficial. They may just be slow. Or they could improve the entire process.
To facilitate the validation of probabilistic models and other statistical research, the federal government should create large, anonymous databases of exemplars and latent prints. (Recommendation 4.2)
Sounds great. Build a giant sandbox, so long as I get to play with the database too.
The federal government should support research to determine the most appropriate tests of visual function for friction ridge examiners. (Recommendation 7.2 in part)
Agree. 'Most appropriate' being the key words here. Some people are just good and comparing prints. Hopefully, we can find a test to predict this.
The federal government should support research programs to improve automated fingerprint identification systems. Such programs could address the following issues:
a. Expanding the algorithms used to match prints to account for the fact that the diagnostic value of minutiae depends on the region in which they are located;
b. Making fingerprint and palm print databases interoperable among local, state, and federal automated identification systems; and
c. Increasing compatibility between automated identification systems and other latent print software tools, including digital enhancement programs, probability calculation programs, and automated quality assessment programs. (Recommendation 4.1)
Agree, especially with B & C. I would add a caveat to A: only if this improves the matchers.
The federal government should continue funding research into automation of the initial quality assessment step in latent print analysis. (Recommendation 4.4)
Disagree. The automation of this step is unlikely at best. Also, this recommendation conflicts with the previous recommendation to research likely, unlikely, or probable conclusions. More of the smaller prints that are currently not compared would have to be included to use this increased range of conclusions. This recommendation suggests that this automation would only let the prints of value for identification go through the comparison process. This is just not a process that can be reliably automated. It's just not.

This section also suggests that comparing borderline prints may result in erroneous exclusions. I strongly disagree here. That is only the case in agencies that forbid their examiners from reaching inconclusive results. Training in exclusions is the solution to erroneous exclusions, not raising the bar on what you deem "comparable".
Translating these ideas into practice will make the analysis and presentation of a crucial source of information even more effective and reliable.
While this final sentence of the summary sounds good, there is no way to know whether each and every one of the above recommendations WILL MAKE the process more effective and reliable.

If you made it this far, thanks for reading.
Tazman
Posts: 244
Joined: Fri Apr 23, 2010 7:25 am

Re: Latent Print Examination and Human Factors

Post by Tazman »

Good analysis, ER. Down to the last series of paragraph quotes and responses about the Federal Government should fund this, and that, and oh yes, this, too. A real cash cow, the Federal government. Currently, we are borrowing 40 cents out of every dollar spent at the Federal level. The Democrats won't cut a penny of expenses and the Republicans wont increase a penny of taxes. The are not going to fund new programs. Do the state and local agencies have the money to fund any of this? Not likely -- most of them are furloughing people and leaving vacancies unfilled and frozen.

If implementation of all these grand ideas costs money, we would be better off having a Forensic Science Bake Sale than counting on your agency or mine or the Federal government to fund anything.
"Man was born free, but he is everywhere in chains." -- Jean-Jacques Rousseau
g.
Posts: 247
Joined: Wed Jul 06, 2005 1:27 pm
Location: St. Paul, MN

Re: Latent Print Examination and Human Factors

Post by g. »

Eric,

Thanks for taking the time to share your thoughts on the Human Factors report. As always I am surprised there is not more discussion on issues like this. Especially since I have already seen recommendations and quotes taken out of context and thrown at examiners on the stand...and it's only been out a few weeks.

It looks like you agree with far more than you disagree with in the report. That is comforting.

I won't go blow by blow but 3 things stood out:

1. Your comments on the "report writing" recommendations.

This was (in addition to addressing bias issues) one of the most contentious debates among the group. THere was clear division between the practitioners and the academics/law professors/legal minds here (surprise right?). In the end I think we could live with the recommendation as a SHOULD (not a MUST) and more detail in the body of the recommendation. The practitioner voice tried to convey that we are writing our reports for the client (the submitting officers). The counter point is "no, the entire CJ system is your client". We argued (I include myself here) that it's in the notes, just discover them. In the end "SHOULD" allowed most of us to live with it as a compromise.

2. Quality algorithms

You don't think it can be automated. Meh. We should talk about this over beers some time. I can fill you in on some emerging research which shows that one could make some probabilistic assessments up front about value based on a few different factors which may allow one to categorize the latent into different categories of value (value for exclusion only, value for ID, value for comparison, value for AFIS, etc.). Don't think about the problem as a black and white, all or nothing value decision approach, and you might be able to envision a broad (but obviously still imperfect) system for assigning value/complexity, up front.

3. "Marginal value prints may lead to erroneous IDs/exclusions"

Yeah, I think we would have to have a lengthy discussion on this one. I think it might be poorly phrased a few times in the report, but I think it is trying to convey the message "use caution when dealing with borderline impressions" (Capt. Obvious to the rescue). I think though it is important to capture that even in these large categorical decisions (ID, Exclusion, Inconclusive), the impressions vary in their strength, potential for error, etc. Some of that is lost when they are placed in those big bins since they are "one size fits all" (i.e. an easy ID is treated often in court, for errors, etc. the same as a difficult ID and vice versa). Both are presented as ultra strong and accurate decisions.

But your point is noted about "proper training" (I might add to it "... and effective quality control and decision strategies") instead of "raising the decision threshold".

Thanks for taking the time to comment on it. i found your comments insightful and overall, think we conveyed most of what we wanted to effectively. Much appreciated.

My view is that the NAS report makes excellent, but broad, reform recommendations for all of forensic science (and does a poor job of drilling down into the specific disciplines, and frankly was not necessary, helpful, or were they particularly informed enough to make those decisions for a number of disciplines in such a short time). Conversely, the Human Factors group WAS dedicated to studying one discipline thoroughly for 2 years and does make very specific recommendations which many can be taken on board now. I think given the task and the goals of the Human Factors group, I think it's an impressive report with a number of valuable recommendations.

g.
ER
Posts: 351
Joined: Tue Dec 18, 2007 3:23 pm
Location: USA

Re: Latent Print Examination and Human Factors

Post by ER »

On the quality algorithms, I can see it working for most prints. But I can't see it working for the prints that are near the line. And 'near the line' is the only place that this would be useful. Since we don't generally struggle with giving a value statement to prints that are clearly of value or clearly not of value, then we don't need an automated assessment. Things may be getting better, but the quality assessment models that I've seen or heard of shade in good/bad areas of the print. This generally discounts areas that are bad. But there's sometimes good information in the bad area. If there are four ridges that flow into a bad area and three that flow out, then I know that there's a ridge event in there hole. While it doesn't have the weight or a clear bifurcation, it does have some weight, and the quality algorithms seem to discount it.

On the marginal prints, I think I should rephrase what I mean. Comparing borderline prints does not necessarily lead to a higher error rate. Only one of the six bad ID's in the FBI's study could be considered a 'borderline' print. Most examiners excluded those prints with only a few going inconclusive. Lowering the bar on what you compare won't increase your error rate as long as inconclusive is an acceptable conclusion. However, I do think that comparing prints that are only of value for exclusion will lead to more erroneous exclusions. We definitely need training and consistency on what constitutes sufficiency for exclusion.

I was very disappointed that the section on errors (Chapter 2) made no mention about an error rate that is dependent on the quality and quantity of the latent. I would think that a chapter on errors would need to focus on this point. The difficulty/complexity of the comparison directly affects the possibility of an error. This chapter also suggests that there is always ONE correct answer to every comparison. I would argue that for some comparisons there may be more than one valid answer. Sometimes, either inconclusive or no value is valid. Sometimes, either inconclusive or exclusion is valid.
g.
Posts: 247
Joined: Wed Jul 06, 2005 1:27 pm
Location: St. Paul, MN

Re: Latent Print Examination and Human Factors

Post by g. »

Again, I agree with you.

And yes Chapter 2 on Error Rate calcs takes the oversimplified approach to calculating errors. I agree that academics that write on this topic keep missing the point of inconclusive decisions. I like that the FBI Black Box, my White Box Study (Informed Judgments paper), and the new SWGFAST standard for defining error rates recognize this issue and all discuss it. I think it is important to alert some commentators that one can still calculate these values and include inconclusive. When there is conflict over how to handle the inconclusive decisions, my preference is to use "false discovery rates/predictive values" (i.e. posterior probabilities) since these calcs are close to false positive/neg. rates and are calculated without the inclusion/exclusion of the inconclusive decision.

g.
Post Reply