CLPEX Chat Board

Goodmorning from down under where the fires are out and the floods receding,

In the latest Weekly Detail Boyd Baumgartner has submitted a very well structured and scientific look into the issue of false negatives. He has examined the process of hypothesis testing under the ACEV methodology and proposed a preferred alternative of ternary predicate logic which as an expression I will have to go away and learn a bit about. There are two assumptions that Boyd introduces as part of where we have come from in fingerprint science. The first is that we have access to the collective experience/knowledge of all experts in the determination that no two fingerprints have been found in two people which are identical. While I agree and support this position the converse is the case with some critics who view our examinations not as observations made under scientific experimental conditions and therefore not credible for use as empirical data. It is not just the examinations which are viewed as non-scientific but we as examiners also. This is a challenge for the whole fingerprint community since the weight of our argument is that fingerprints are unique since duplication has not been discovered. Boyd's argument is that hypothesis testing for fingerprint examinations is probably not the best approach as the results do not allow for the situation of an 'inconclusive result' but only individualisation or exclusion. This is the second assumption as it supposes we all use the same hypothesis to start an examination. While not stated by Boyd I am assuming that the hypothesis is something like " does the latent fingerprint and the exemplar form a match", with results being yes or no. In my understanding of hypothesis testing and its application to fingerprint examination the approach I take is to generalise the conditions of the hypothesis allowing for a development phase and a testing phase. I use a simple hypothesis of " If friction ridge detail sufficient for comparison exists in these two impressions under examination then a conclusion may or may not be possible". Interestingly in these examinations I include the analysis of all three levels of detail with my starting point generally being at a classified anatomical area of the core or delta. I was wondering with those who use hypothesis testing if they agree with Boyd's position that a yes/no condition is how examinations are conducted?

Thanks for the thoughtful reply Les, now I know that at least two people have read something I wrote (me included).

I wrote this particular piece after having researched the role of epistemology in fingerprint identification. Epistemology simply referring to the philosophy of the theory of knowledge (what is knowledge, how do we acquire it, how do we know what we know, etc) Fingerprint identification being couched in scientific method terminology always seems to be a dog chasing its tail. No two people or disciplines seem to have the same understanding of what that method is, how it's performed, or the best practices involved therein. As a result, the articulated 'science of fingerprints' has some large holes that skeptics love to exploit.

That is not to say I don't see scientific methodology in ACE-V, I do and have written as such on this very board. Obviously, science is concerned with knowledge acquisition as is epistemology and science is a branch of the philosophy of epistemology, however there is more nuance to what we do than just 'big science'. I am of the opinion that ternary predicate logic is an alternative way to see the evaluation of information we analyze and compare in the print. An alternate view of Analysis being axiomatization and an alternate view of Comparison being existential unique quantification.

With regards to the skeptics, critics, etc, the understanding and integration of epistemological philosophy in fingerprint identification is one of the failings they see and have articulated in writing. The NAS report refers to Roger Koppl's paper on sequential unmasking, and he's repeatedly called for 'independant epistemic checks' on fingerprint work. Here's a copy of a presentation he gave http://www7.nationalacademies.org/stl/Koppl.pdf. A complete presentation with audio and discussion is available on brightalk.com (title: "How to Improve Forensic Science") Bright talk is free, but you have to register with the site to watch the presentation.

In the ongoing process of trying to articulate what we do in terms that satisfy "science," I wrote an article in 2000 that was published in Fingerprint Whorld. The text of the article is available at http://www.scafo.org/library/160501.html

Basically, I propose the use of a five step formula to get us to an indentification or an exclusion. The first step is study of the unknown, or to use the ACE terminology, the "A" part of that methodology -- thorough analysis of the latent or unknown print. The second step is the formation of a hypothesis and counterhypothesis (I did not use the classic "null hypothesis" because the counterhypothesis is slightly different in application here). For simplicity's sake, I proposed that arbitrarily, the hypothesis is "identification" and the counterhypothesis is "exclusion." In the third step, the examiner proceeds to the comparison itself, or "experimentation" to support either the hypothesis or the counterhypothesis. The fourth step is when the examiner's mind clicks and the belief that the hypothesis is correct, or that the counterhypothesis is correct. The fifth step is for the examiner to then predict future points to "prove" the hypothesis and, at the same time, disprove the counterhypothesis by finding more features in either the known or the unknown and predicting that they will be present in the other, then finding them in the other. Likewise, it would prove the counterhypothesis by finding features in one print and predicting that they will not be present in the other, then finding that indeed they do not exist in the other. It is the relationship of reliable predictability, or the conclusion that you can reliably predict all of the features in one print by first finding them in the other, that proves the hypothesis and establishes the identification while, at the same time, disproving the counterhypothesis (and vice versa -- no reliability at all in predicting features in one print by finding them in another proves the counterhypothesis).

The result of "inconclusive" comes when you cannot reach a conclusion in step five above. We've all had that borderline print when we can even get to step four and we think, "If I could just find one more point, I would feel comfortable calling this an ident," but we just can't get that one more point or feature. That is, in effect, an inability to firmly establish that you can reliably predict what you will find in one print based on what you find in the other. In that case, there is no relationship of "reliable predictability," nor is there proof that you cannot reliably predict features in one based on an examination of the other, and neither the hypothesis nor the counterhypothesis is proven. Thus, the examination is inconclusive.

Some of my friends like this proposed formula and some do not. Click on the link above for the whole paper. What do you think Boyd? Les? Does this answer your concerns?

Hello Pat,

It is often the case that we complicate things even when our intentions are correct. Rather than attempt to meet the expectations of scientists by re-profiling fingerprint science into a different frame we should stop and smell the roses, and consider our strengths. I think your approach is the correct one for how hypothesis is used in examination process. The evidence that is in the fingerprint directs the success or failure of the examination. Following your suggestion would be a refinement on what has been the tried and true method of arriving at the truth. By doing so we are not shaping our science into a strange mixture of variables and probabilities and arbitrary values. If your method is accepted then what is left to fix are the type of technologies for recording how the examinations are conducted. Once we capture that data then other scientists can work out if the error rate of fingerprint science is calculable. The hypothesis being that all scientists make errors.

Thank you, Les. Your comments are most concise and meaningful. I am not trying to suggest a "new" methodology, merely trying to articulate the correct process that I think goes on in the head of every examiner when doing a comparison. You are right, there are more important things to worry about, like error rate, documentation, etc. What we do is correct and it is good science. It is the ability to put things into words sometimes that makes the job more difficult than the examination itself actually is.

Hi Pat,
I've read your article before, and I like it. I particularly like the notion that sufficiency is reached when your observations become reliably predictable. I still believe that hypothesis testing is at the 50,000 ft level of articulating what we do. I don't think that people understand the logical implications of hypothesis testing.

I see a huge benefit in going deeper to articulate the soundness of what we do as a logically deductive process. There exists a whole body of literature out there that addresses what is proper logic, logical fallacies, and the like. As academia is becoming more and more of a stakeholder in the review of forensic comparative analysis, I've seen the arguments shift to include these topics. Examples include Simon Cole claiming meta-expertise, the Institute of Forensic Science Administration publishing articles on Bayesian Bias in Forensic Science. Even David Kaye's article "Probability and Individuality in Forensic Science Evidence" published in the Brooklyn Law Review directly deals with the logical fallacies put forth by Saks and Koehler in their condemnation of forensic disciplines.

To illustrate the point, I put together this little exercise that tests the abilities of people to test hypothesis using a logic puzzle. Go here http://www.fs2d2.com/learncreate a login, and take the test. The results are tracked, so if you want anonymity don't put your real name.

Enjoy,

Boyd

I get confused whenever I read posts here regarding hypothesis testing. And, it's because I think of hypothesis testing as coming at a different part of the identification process. It seems to me that we have three theories in fingerprint identification:

1 - Uniqueness - each print is different, no two prints are the same
2 - Permanence - prints don't change over time
3 - Identification - there is a necessary and sufficient criteria for identification

These are true theories only if they have been tested. Once they have been tested, then we accept them as theories and then apply those theories to specific prints. We don't retest these hypotheses when we apply them, we simply base any specific comparison on the underlying theories. What we determine is if, 1) there is sufficient data to match a mark to a specific section of skin, and if so, then we apply our methodology of comparison to the mark and a set of prints. But, the methodology is not hypothesis testing itself - it is comparison.

I think the fingerprint community has done itself a disservice in trying to "make scientific" a process in the wrong manner, and has ended up confusing things rather than clarifying things. The process of identification is the application of a methodology, but we are not testing an hypothesis with each individual comparison.

Also, I believe that epistemology is a branch of philosophy, and philosophy is a branch of natural science; rather than science being a branch of epistemology as stated earlier in this thread.

George

Mr. Reis, I agree with you.

I thought this would be an interesting thread, but it kind of died out. Any other thoughts on Hypothesis Testing?

I dislike the notion of "hypothesis testing" in a description of an examination. It seems to be an attempt to embed science. But science doesn't just concoct every imaginable theory that may reflect physical reality and then go test them all. In the scientific realm, a theory is formulated to explain specific observations. It asserts the underlying reality behind those observations. We do not, for instance, create an infinity of theories of gravity without some very specific observations. Those observations suggest a theory. We test that. We may then develop a more sophisticated theory based on more sophisticated observations, observations we were lead to make as we speculated upon all the variables that might need to be accounted for in a more general theory of how objects move in out environment.

Historically, there have been many "theories" that were not properly theories, because they did not seek to explain observations of the physical world. Physiology was full of them, in things like the seat of thought and the nature of blood and circulation. They weren't theories. They were notions, often concocted metaphysically. And they weren't tested. Eventually, someone made coherent observations, without the metaphysics and rejecting traditional dogma, formulated a theory and tested it.

When we receive two impressions, we are not making a theory. That would be like holding a ball and saying we have three theories: It will fall, it will rise, or it will not move, when we drop it. We can measure it and drop it in known air and observe and speak to its mass. It might well fall, drop, or float. Those are the three possible outcomes. We'll observe that, but what it means to us is just an application of knowledge. If we know its mass ahead of time, we won't even bother to drop it. We know what it will do, just as we know that if we make two impressions of one finger they will match.

The three "hypotheses" are merely the three possible outcomes of our observational examination to answer a specific question. We have a perfectly acceptable theory that friction ridge skin characteristics can be recorded and observed and that if a sufficient quantity of detail is recorded, we can discriminate between individuals. No one who matters doubts that. We are arguably missing validation of just when we can conclude from our observations and validation of just how a conclusion can be stated, but we are working beyond the theory, just applying it. I would be amazed to hear a metallurgist state that prior to a spectrometric analysis of a sample to determine if it was lead, he formulated theories that it was lead, wasn't lead, or was an alloy. Calling these hypotheses just makes scientists shake their heads.

Hi all,

Its starting to get interesting. Scientists with a voice are stating that our science has not validated the conclusions we reach and report in particular individualisation. They continue to question the lack of standard in 'sufficiency' leaving their criticism open ended while highlighting our results are highly subjective (black box), not measureable (no statisitcal data) and therefore not scientific. We know that conclusions of individualisation are verified and our performance standard is that of competent examiners so the system is there to correct error ( I've chosen those words carefully). The challenge for us is to be able to put our scientific house in order so it meets most of the expectations of traditional science. So far I can see we have defined our principles, technique, method and conclusions. There is a standard for individualisation and it is expressed in negative terms of what is not required (no predetermined number). This thread deals not with the standard or the principles but what is the starting point of each examination. We have no prior knowledge of the content of the latent or inked impressions this forms an objective frame, we may have knowledge of their relationship (verification) but our professional opinion still needs to be convinced that the relationship is justified by both the evidence and the standard using the technique. We are also testing the principles by undertaking a comparison which could falsify the established parameter of no duplication. In science terms we are conducting an experiment. I'm happy if the term hypothesis is replaced with proposition. The result of the experiment/examination will either prove or disprove a relationship between the two impressions. The proposition forms the purpose of the plan as to what is likely to be achieved subject to the variables in the impressions. Without a planned approach to an examination are we being scientific? In my examinations I dont write down the principles being tested or the hypothesis/proposition being planned, it forms the mental preparation ahead of the activity. Regards from oz. Les

I guess that's more or less the thing here. An examination begins with an internal set of propositions. Since we're considering only three possible outcomes, that's pretty much inevitable. It's really not very important whether we are thinking of an examination as an "experiment" or if we think of it as a task of gathering data and following it to see which outcome it leads to. I really don't think many examiners conduct an examination with the kind of internal state that they would identify with their internal state if they were conducting a frank experiment, concocting a reasonable theory and testing it. I think that for most it becomes more like trying to spot our own car in a huge parking lot. We're guided in part by general characteristics that we don't consciously articulate and probably aren't aware of and by specific details.

The real point is following the data and only the data. That's what the process has in common with properly crafted experiment. I think there are interesting differences in the general processes of concocting theories and the process of testing them. Theories arise from all sorts of subtle combinations of things. Observations, speculations, prior knowledge, the trend of thought in the field, personal traits, and more. But when it's time to conduct the experiment to test the theory against the real world, it's strictly a matter of data. Or we try to make it strictly the data. But many fields depend on human observation as the filter between the thing observed and the recorded observation. Nothing changed in animal behavior, but the Skinner folks observed one thing, and later behaviorists observed something else.

I suppose that if viewing an examination as an experiment helps enforce adherence to the flow of gathering data before interpreting its meaning, it is a good thing. So long as it isn't taken to imply anything else. I guess my dislike of the references to "experiment" and "hypothesis testing" is really more a feeling that it's slightly embarrassing, that being something that should be take for granted. That's probably naive.

Hi again,

While this thread has been ticking along I've been off reading about sampling for statistical purposes and validation and models of probability. Sleeping very well. One thing that stands out is that our pragmatic experience is a true benchmark supporting the basis of uniqueness. We test the principle every time a comparison is conducted. If we call the whole approach our fingerprint examination model (tenprints and latents) then we have formalised what is needed for validation studies. This means that any critic who is inclined to put the model to a rigorous test would expect to be able to prove/disprove the principle. What has not been tested is any probability model for statistical analysis of fingerprints. Several have good design considerations used under ideal clinical conditions but the truth lies in how to address the range of case variables for each pattern, each transfer, each development (if needed) and ultimately each examination. The human element of the examiner will always be a source for challenge and that is true of any profession. We cant be complacent about our model while the issue of 'sufficiency' remains subjective and until we can prove individualisation using more objective elements/terms and be able to articulate those convincingly. Cheers from oz. Les