Show me the OSAC Scales: A technical discussion and practical example

[Boyd Baumgartner]

You don't have to be logged in to take the poll

I'm spinning this off of the Process Map thread, just for clarity. It also goes along with the other thread of siloed forensic research because there's a myriad of other disciplines that have already wrestled with this.

We'll go from this:

To those that are argue against the OSAC 5 Conclusion scale because...
- There are no instructions on HOW to use it;
- There has been no validation, testing, or studies using it;
- The conclusions don't protect against errors; or
- Any of the other myriad reasons:

…

Preventing progress on these very basic first steps because subsequent steps haven't been completed is insanity. It's asking the baby to run before she crawls.

I respectfully disagree. The discipline tends to act first, think second. Just like the whole likelihood ratio....'we have to do something' mentality, followed by mea culpas and journal publications that directly contradict the assertions that were being made at the time.

What we have here is a time tested debate between Dichotomous Scales and Likert Scales. Dichotomous simply meaning yes/no, so for us it'd be ID/EXC. Likert scales are just psychometric scales named after the guy who invented them in 1932 and are usually a 5 scale system that attempt to measure attitudes around a certain phenomenon. For us, that is the 5 conclusion scale that is being offered up by the OSAC. Those are: Source ID; Support for Same Source; Inconclusive; Support for Different Source; Source Exclusion. (I Would argue the adoption of Inconclusive to the ID/EXC framework puts us in a likert scenario)

Each has it's disadvantages. Dichotomous approaches can force a choice, bias to the extremes and distort the fact that there are true neutral responses. Likert scales create more variability which in turn diminishes reliability of the scale and falsely imply precision. Even numbered Likert scales (one without a neutral option) can also force a choice (Take Busey's/Vanderkolk's workshop at an IAI some time, you'll see.)

You could argue that the critique that we are overstating our conclusions is just the phenomenon of forcing choices which bias to the extreme.

This debate centers around what we've been talking about, validity and reliability. Here's a great discussion in plain english that shows that concern around validity/reliability of likert scales is real, long standing (since the 1930s),exists in domains outside of fingerprints, and is directly applicable.

And this brings us into the instruction aspect of what ER wrote above because there cannot be instructions on how to use it. Likert scales are, by definition, non-parameterized. Meaning it is ranked, not numerical (read: subject to statistical methods) and they are not derived from an equal-interval scale. The difference between Inconclusive and what counts as Support for Identification and Support for Exclusion are not anywhere near the same distance if we consider the scale as a measurement. This is what gives rise to the variability. Just keep reading and you'll see.

So, for me, the thing that really counts is performance testing. That's what should be done up front. Whitebox, blackbox, greybox...you name it. Adopting these prior to showing them to have value will be problematic. So, let's do a micro performance test.

All of that being said, I bring you the prints. Please rate on the poll above which OSAC conclusion you would use.

Prints.jpg

[ER]

Like I mentioned in my post, you're comparing the new proposed conclusions against the idealized, but non-existent, current conclusions. The current conclusions are not a dichotomous ID/EX choice because a) most agencies have some version of ID/INC/EX or even ID/SSS/INC/EX; and b) because the current conclusions are an unstandardized jumble of terms and definitions that can't be discussed as a single entity and thus can't be linked to existing research.

You're describing the newly proposed conclusions as if they've never existed before and as a wholly unfamiliar and alien concept. There is ONE new conclusion (which isn't even all that new for some agencies). The other four are common terms with new definitions of LONG ESTABLISHED conclusions.

This new standard brings all of the varied conclusions currently spread across the country each into a single term and a single definition. This is a STANDARD document. Do what you're already doing, just use these terms and definitions. Don't do anything new or different. Just use these terms and definitions when you do what you already do. If you don't use "inconclusive with similarities" or "could not exclude", fine. Keep not doing that. SSS won't apply to you. Don't start using it. But if you want to start providing that level of information. This is the STANDARD way to do it.

You're right about the research. The Black Box research didn't explore how SSS or SDS options would affect accuracy or other performance metrics. We should get moving on that research. However (and this a pretty big however), SSS is already being used and has been for a long time. We need to standardize the terminology AND conduct the research, but since the conclusions are already in use, it makes no sense to delay the standard terminology while we wait for the research. Hell, the researchers never included it in the first place because it wasn't in the "standard".

the thing that really counts is performance testing... Adopting these prior to showing them to have value will be problematic

That's the problem. They've already been adopted. But in an unstandardized way. Which is exactly why this STANDARD is the way to proceed. We can't even properly test these Likert Scales because no one's even using the same words, much less definitions. You're arguing that we can't open Pandora's Box until proper tests have been conducted. I'd be right there with you and totally agree, except the box is open and smashed and scattered to the winds. The only way forward is to set up a standard list of terms of definitions, test examiner performance against the standard, and make corrections as necessary.

Again, this is a STANDARD document. It is not a training class. It is not a research paper. It is not an instruction manual. It will never be any of those things. The goal of this STANDARD is to STANDARDIZE the terms and definitions for conclusions already in use. It does that. It does it well. It should be passed.

[g.]

Boyd,

Thank-you for using a real-world example. Makes this much easier.

1) As presented it's a straightforward EXCLUSION (extremely strong support for different sources). There are more than 10 or so clear L2D differences above and below a central area of about 12 ridge endings in correspondence. The differences far outweigh the low specificity corresponding minutiae.


2) I'll take it to the next step, if you crop out the upper half and now just show the bottom half:

It is INCONCLUSIVE. There are 12 or so corresponding L2D, but there are a few L2D differences. Net effect is: they cancel each other out. INC. Neither proposition is supported over the other, in the end.

3) Finally if you crop out the bottom, AND crop the top and just leave the 12 or so corresponding ridge endings, then I would report SUPPORT FOR SAME SOURCE (it's moderate to be more specific*). You can't exclude b/c you have corresponding features, no clear L2D differences, and you can't ID either because the ridge endings in this area are very non-specific and not that discriminating. I can further support that conclusion using a stats model to demonstrate that these 12 features are around Log LR = 5, which is (moderate) SUPPORT FOR SAME SOURCE.

So depending on how you wish to present these images, you can apply the conclusions differently because the conclusions should represent the perceived weight of evidence (the LR).

g.

*Within the broad category of SUPPORT FOR SAME SOURCE, I prefer to further subdivide to LIMITED<MODERATE<STRONG (as suggested in the OSAC document and as suggested by IAN EVETT, GRAHAM JACKSON, R. COOK, and other 25 years ago.)

[Boyd Baumgartner]

g. and ER. Thanks for your input. Discussion around these topics is helpful and the fact that we tend to disagree in general, makes it interesting because it's the reconciliation of our ideas or at least discussion around the edge cases that makes for fruitful conversation... at least for me.

ER, I disagree with your framing of my position, but that's somewhat irrelevant other than to note that what you put forth isn't my assertion at all.

Where I'm coming from ultimately in this thread is 'what counts as support'? I'm keyed into what I said earlier about the more categories giving the illusion of more precision (I think LR's based on Examiner input perpetuates this problem) while still masking the weight of the data behind categorical conclusions.


I chose this print ultimately because it's easy, but it demonstrates the point but with an abundance of data. There's support for same source in this print and support for different source at the same time (I'm using these terms not in their categorical capacity but just merely as 'ridges with corresponding sequence and spatial arrangement' and 'ridges that don't have correspondence between the two prints. (No, I'm not saying this is anything but an exclusion). What I am saying is that as this problem presents itself with either less data, less clear data, or less anatomical data (palm? finger? foot?), the scales are going to shift in ways that are going to present the same problem; variability (read: non standardization) in the application of the scale. (g. pointed it out by dissecting the print into different areas, but we're basically saying the same thing.). This is why I agree with ER about the lack of standardization but disagree that some more precise or standardized language is going to solve the problem. Language is still qualitative. Just go look at the other thread. We're all experts in the field and we're misinterpreting each other and starting out with 'how I'm using it is..' And baring anyone trolling my unscientific poll up above, we can see at least hints of what I am saying as being something worth considering. We're playing a semantic shell game without addressing the language game at the center.

g. made the point in the other thread that these factors can be dissected from the conclusions because they speak to the weight of the evidence, but I disagree. The weight of the evidence is derived from the weight of the data used in arriving at the conclusion. There's a reason Ashbaugh's book is titled 'Quantitative-Qualitative Friction Ridge Analysis'. If we were to move to the 'gold standard' of DNA style science, we'd be knee deep in population statistics as they relate to the prevalence of minutiae position, direction, relative distance. Giving us a way more quantitative assessment and then statistics behind how to weigh the quantity. What we have instead is still a mixed method with, I would argue, less transparency and more appeal to objectivity. It feels like a sleight of hand that never addresses the overstating problem. There's still a fact/value split between the facts in the print (read: data) and the value judgment (read: conclusion) we apply. If we were to lean on stats, I would like it to be in a more functional way more like this tool from Christophe or the Quality Metric.
.
.

Correspondence.png

and

Difference.png

[Boyd Baumgartner]

So, for me, the thing that really counts is performance testing. That's what should be done up front. Whitebox, blackbox, greybox...you name it. Adopting these prior to showing them to have value will be problematic. So, let's do a micro performance test.

It would seem that I wasn't alone in my thinking about this. How timely...

You may have heard about discussions at OSAC, the IAI, or other organizations about the possible switch to different scales for reporting conclusions from friction ridge comparisons. Before the scientific community can make a recommendation, however, we need to document how new scales might be used by examiners.

In 2019, the National Institutes of Justice funded a grant to Indiana University to study how expanded conclusion scales might be used by examiners. I'm contacting you because you previously indicated a willingness to be considered for participation in future studies of this kind.

We are now recruiting participants for a study that will begin in early January 2020. If you would like to participate, simply reply to this email with your willingness to participate and I will enroll you and send you login information, so you can begin. You are also welcome to send this to other latent print examiners who are qualified to testify in the US.
If you need more information, the approval letter, or the consent form to obtain approval from your agency, please ask and I can send it to you.

If your agency allows it, or you plan to do it off duty, the grant provides for an honorarium of a $30 Amazon gift card for those who complete the study. It will probably take about 300 minutes to complete the 60 comparisons in the study. I anticipate the results will directly contribute to the debates happening at OSAC, ASB, and other policy-making organizations. Your contribution will have a real effect on future policy decisions.


Sincerely,

Thomas Busey busey@indiana.edu
Professor of Cognitive Science
Indiana University, Bloomington, IN

[Boyd Baumgartner]

I'm sure this will be a follow up to Tom's previously published work that discussed the trade off of expanding the scale.


From the site:

Conclusions
We found that examiners did not get objectively worse with the expanded scale, and that the new ‘support for..’ categories did provide useful investigative leads for many of the formerly inconclusive conclusions. However, examiners did seem to redefine what they mean by ‘Identification’, reserving it for only the strongest comparisons.

In addition, there were a fair number of what might be thought of as ‘erroneous support for common source’ conclusions. These are situations where an examiner might be comfortable only suggesting an investigative lead, but the detective treats it as a definitive conclusion. Care must be taken to educate the consumer that a stronger conclusion was available, but not used in this case.

It will be interesting to see what findings come out of the newest study.

[Dr. Borracho]

Sometime during 1985, the first year that AFIS was operational in the US (it was called AIDS back then -- Automated ID System, but that acronym was abandoned a few years later when the disease with the same acronym came onto the scene), I read a letter to the editor in a state division newsletter. The writer expressed alarm that computers could find similarities and point us toward identifications because, as the writer proclaimed, we would have to use a higher standard for identification on a hit from a large data base than we would when a detective developed a suspect and gave us a name.

A higher standard for identification??? I thought the writer was out of his mind. Remember, there was no numerical point standard in most places and we made an identification "to the exclusion of any other person who was alive, ever lived, or ever would live." So if that was true when the detective gave us a suspect's name to compare, why would it NOT be true if the computer gave us the name?

But after AFIS had been in operation a few years and more systems were coming on line, almost every agency using one had a close non-match charted on the wall as a freak occurrence. But, of course, such occurrences are not freak.

To go back to the letter cited in the first paragraph above. I realize now that the letter's author was correct. So what about Boyd's chart? If I had a latent with only the dozen or so "matching" points and it was a suspect named by the detective, would the likelihood of it being a correct identification be higher than if it were a candidate from IAFIS? Would it be "Strong Support" from a named suspect, but only "Support" from an AFIS hit?

How would you express that, g.? ER? Boyd?

[Boyd Baumgartner]

The short answer to your question is one that I've answered before: Fingerprints have a moral decision making component especially when the comparison is complex.

The letter writer in your post makes some assumptions. First, that the prints lifted were relevant to demonstrating that a crime occurred and that person had a part in it. Secondly, that the names submitted for comparison are relevant. Sometimes, we just get a list of 10 people known to be active in the area and they want several cases compared against those fingerprints. Are those people more likely to hit in my case than a candidate from AFIS just because a detective submitted their names?

Remember, there was no numerical point standard in most places and we made an identification "to the exclusion of any other person who was alive, ever lived, or ever would live."

Let's just lay this to rest. That isn't even a thing. Considering most things you find fingerprints on are manufactured, it's irrelevant to consider someone born in the distant past or the future could have been included in the population of people who were even in the same vicinity as the object at any given time, since its manufacture date and fingerprint deposition date precludes them from even being considered the source. That's just a surrogate way of stating your confidence in a print.

If I had a latent with only the dozen or so "matching" points and it was a suspect named by the detective, would the likelihood of it being a correct identification be higher than if it were a candidate from IAFIS? Would it be "Strong Support" from a named suspect, but only "Support" from an AFIS hit?

It's an interesting question. Let's make it easy and get out of the ether of hypotheticals and take it into the practical.

This is that print out of Israel that presented it self in a similar fashion to what you're describing.
.
.

AFIS-Inconclusive.jpg

Here's my simplest explanation. You should have a higher standard for what's run through AFIS, not necessarily for an Identification as a result of using AFIS. No matter if it's a detective submitting a name or an AFIS run, the comparison boils down to significance, inference, and confidence of the data in the print. As I said before, the less data, the more ambiguous the data and the less you know about where it came from (finger, palms, sole), the less significance can be assigned to the data, the less inferences can be made about the data and the less confidence you will have in the data.

I think this is relatively uncontroversial and is preloaded in some sense because agencies don't run every print, so there's a standard for running through AFIS, whether that's an unwritten office culture or it is in your SOP. I would venture to say that in the most vague of terms, people are running prints they think they can ID, and as the data in the print becomes more ambiguous or limited, you start running into cases where your initial assumption of 'identifiable' might not work out. Also, when you don't ID a print from an AFIS run, it's not really the same thing as excluding. You don't include all the people from the candidate list in your final report and associate an exclusion to them. It boils down to what do you call support.

I wouldn't have run that print in the first place and it's significance really only comes from the fact that the DNA hit on the same person, the significance did not come from the data in the print. However, given that you're asking the question, I'd say that it is Inconclusive - Lacking support. 6 around a delta and three of them have some interpretation.

[ER]

ER, I disagree with your framing of my position

That was unintentional, and I'm sorry for that.

I'm keyed into what I said earlier about the more categories giving the illusion of more precision (I think LR's based on Examiner input perpetuates this problem) while still masking the weight of the data behind categorical conclusions.

I do understand your points here. However, for me, the debate over more categories is secondary to the discussion of the proposed standards. The proposed standards to not require more categories. It simply provides a common definition for the categories that agencies already use. Separate from the question of whether or not there should be 3 or 5 or more are more relevant questions.

For the agencies that choose to use 3 conclusions, are the definitions of those 3 in the proposed standard acceptable?
For the agencies that choose to use 5 (or 4) conclusions, are those definitions acceptable?

This is why I agree with ER about the lack of standardization but disagree that some more precise or standardized language is going to solve the problem.

I don't understand. You don't think that standardized language would solve the problem of a lack of standardization? Or even be a step in that direction? It seems fairly tautological. And in looking at that other thread, a big part of the problem is that there are different terms and definitions for the same conclusion. A standard would begin to solve that problem.

The weight of the evidence is derived from the weight of the data used in arriving at the conclusion... If we were to move to the 'gold standard' of DNA style science, we'd be knee deep in population statistics as they relate to the prevalence of minutiae position, direction, relative distance.

It is a false conclusion that "weight of evidence" must be expressed with population statistics. Weight of evidence may be expressed subjectively. I agree that the use of objective measurements may be useful additions to our conclusions in the near future. While you may want weight of evidence expressed objectively and weight of evidence may be best measured and reported objectively, it is still possible to convey weight of evidence subjectively.

But again, I'd like to return to the proposed standard. Only considering the conclusions that you use, what needs to be different? Since some agencies are already reporting limited identifications, what standard SSS language would you propose that they use?

Let's just lay this to rest. That isn't even a thing. Considering most things you find fingerprints on are manufactured, it's irrelevant to consider someone born in the distant past or the future could have been included in the population of people who were even in the same vicinity as the object at any given time, since its manufacture date and fingerprint deposition date precludes them from even being considered the source. That's just a surrogate way of stating your confidence in a print.

I want to understand clearly. Are you saying that there was never support for "Identification = Exclusion of All Others"? Or are you saying that there was never support for "Identification = Exclusion of All Others Ever"? Do you see value in one but not the other?

[ER]

Dr.,

A higher standard for identification???

I've got a story on this topic.

A few years ago I was training a large group of latent print examiners. They were all working through the training samples that I brought, and I saw one very experienced examiner just staring at her markup of the latent and known. I asked her if she had any questions about the samples that might help her. She said, "I see a lot that corresponds but some that doesn't. If I just knew this was from an AFIS search, I'd say 'Inconclusive' and be done, and if it wasn't from an AFIS search, I'd say 'ID' and be done." I replied, "Well, do you have any question for me?" She asked if the known was an AFIS candidate. I said yes. She wrote down Inconclusive.

I think that it's clearly down to risk. What are the chances that the detective just happened to find some random guy with 10 points corresponding to the latent in this case? One in a million? (Insert Dumb and Dumber joke here) Now, what are the odds that there was one finger in a database of 1 billion with 10 points corresponding? Well, I don't know, but it's better than one in a million.

Detectives will bring you matches and non-matches. It is exceptionally rare that a detective will bring you a close-non-match. However, AFIS will only ever show you matches or close-non-matches. If we treat IDs from AFIS like we always treated non-AFIS IDs, then there is an increased risk for error.

I would agree with Boyd that the "AFIS-Inconclusive" print shouldn't have been run in the first place. However, that doesn't solve the problem. There will eventually be a latent searched in AFIS that brings back the closest non-match you've ever seen. Now what? The plan to not search borderline latents is fine, but now I'm staring at a lot of similarities. So what do I do? Understand that AFIS brings you close-non-matches, have a slightly higher threshold for ID because the risk of error is higher, and proceed accordingly.

And since this is all still a subjective conclusion, there's no hard and fast number of extra points that you need. As always, it depends on the print. The important part isn't writing a separate standard into the protocols for AFIS searches. The important thing is to be aware of close-non-matches, practice with close-non-matches, and to consider the different risk of error given the different scenarios.

And my conclusion on this sample?
For the Latent: Unsuitable for identification; Suitable for exclusion
For the Comparison: Inconclusive due to insufficient detail in the latent print (I don't think there are sufficient discriminating features to go to SSS)

[Dr. Borracho]

Thanks, ER.

My mind is in a bit of a fog when it comes to Bayes Theorem, so please help me clarify whether that concept would come into play regarding the difference between a suspect named by the Detective, and a high-ranking candidate on a list from NGI. How would we quantify the probabilities? Or can we? What if the #1 candidate in NGI lived only a couple of blocks from the liquor store that was robbed and he had a record of committing armed robberies? What if the #1 candidate lived 2,000 miles away and had only a single arrest for forging a check in his home state? Would those factors affect either the prior or posterior probabilities?

Or, do those questions lead to some form of bias that we are supposed to avoid at all costs?

I get the general concept of Bayes Theorem and the probability of rain when I walk out of the house in the morning and see a thunderhead versus when the sky is clear. But can that concept be fine tuned to come into play in a named suspect versus AFIS hit situation?

It's all so confusing. Doing fingerprints was much easier in the old days when it was either him or it wasn't and you gambled your whole career on every ident you made, so you had to be absolutely sure before you made a "positive identification to the exclusion of all others."

[Boyd Baumgartner]

This is why I agree with ER about the lack of standardization but disagree that some more precise or standardized language is going to solve the problem.

I don't understand. You don't think that standardized language would solve the problem of a lack of standardization? Or even be a step in that direction? It seems fairly tautological. And in looking at that other thread, a big part of the problem is that there are different terms and definitions for the same conclusion. A standard would begin to solve that problem.

That's exactly what I believe, it's also the conclusion to various competing schools of scientific explanation.

So is scientific objectivity desirable? Is it attainable? That, as we have seen, depends crucially on how the term is understood.

If objectivity is dependent on how the term is understood, objectivity is paradoxically subjective....or at best intersubjective. What it's not is tautological.


The problem I have with the standards can be best summed up in the quote by George Box

Since all models are wrong the scientist cannot obtain a "correct" one by excessive elaboration. On the contrary following William of Occam he should seek an economical description of natural phenomena. Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity

Identification has always meant 'I believe print X came from person Y' and no amount of doublespeak in an articulation document will change that. Especially in light of what juries believe. An identification should be evidence based, not linguistically determined, so show me the print.

I'm of the mindset that it's better to be critical and it's actually intellectually healthy to have various schools of thought in the industry. As long as you know what you believe and why you believe it.

I suppose we'll just have to agree to disagree.

[Dr. Borracho]

Identification has always meant 'I believe print X came from person Y'

Exactly! The very definition of "opinion," as in "opinion testimony," as in a trained, experienced person who has been tested and found competent to perform an examination and present an opinion based on that examination.

We have drifted away from that whole concept because of pressure, primarily from outside of our field (as I see it), to hammer the square peg of comparison sciences into the round hole of instrumental analysis.

As I implied in an earlier post, if it is only about numbers and probabilities, why not just hire blind technicians who can lay a print card on a scanner, search AFIS, then run the top ten candidates (or whatever number you want) through a software program to calculate the LRs, and finally, report and testify to the LRs? No opinion necessary, just "real science."

[Boyd Baumgartner]

As I implied in an earlier post, if it is only about numbers and probabilities, why not just hire blind technicians

You've unknowingly stumbled onto the classical thought experiment known as the Knowledge Argument

[Dr. Borracho]

Thank you, Boyd. As with Occam's Razor, sometimes simple thought experiments help cut through the confusion of extraneous details to reduce a philosophical question to its simplest elements. Neither Occam's Razor nor an appropriate thought experiment will always indicate the best path through a difficult problem, but in most cases they can help clear some of the fog.