Statistical Cage Match

[Boyd Baumgartner]

If you haven't been paying attention, there's the nerd equivalent of a cage match brewing. It has to do with forensic statistics and I would say that it's p<.05 (significant.....ok, I'll stop with the jokes). In one corner we have the FRStat model (Attached Paper) and in the other corner we have what I'll call a Convergence model (Second Attached paper) just because it accounts for the deficits of two previous models to converge on a formal Bayes factor.


This might not be a big deal if it wasn't for this pesky little statement made in the Convergence model paper:

During this project, we demonstrated that most techniques proposed in the forensic literature that attempt to quantify the weight of forensic evidence using similarity measures (known as “score-based likelihood ratios”) are logically incoherent and violate the laws of probability.

Cue needle scratching across record.


That, my friends is what you consider a white glove being slapped accross your face in the world of journal publications. Let's rewind, shall we?


Unless you're new to the field or have been living under a rock for the past decade, you'll undoubtedly have noticed that post-NAS report, forensic statistics have been framed as somewhat of a cure-all for what ails the discipline....specifically giving the proper weight to forensic evidence. So, if the major critique of Examiners is that categorical testimony (Individualization, exclusion to all others, 100% certain, etc) overstates the significance of the weight of the evidence, well then, forensic statistics is just what the Doctor ordered....seemingly.

The problem in a nutshell is that forensic statistics (as they relate to pattern disciplines anyway) are sort of a 'you are what you eat' type of phenomenon, and no one really knew how to weight the inputs therefore the outputs were all over the place. Over the past few years we've seen various conferences, organizations and committees spring up to discuss and debate the various proposals around handling the task of developing a formal, valid statistical solution.

If you had the CSAFE Training (slide 106/112) you may remember the discussion of the 'score-based likelihood ratio' which is what is under assault here in the Convergence paper. Take a look at that quote above one more time... 'violating the laws of probability' is a pretty damning statement. Especially when they've been accepted in peer reviewed journals and have been taught by CSAFE itself. I would venture to say that adhering to the laws of probability is probably a pretty low bar to expect when having something published in a statistical journal, wouldn't you?

FRStat, according to it's own description is precisely a similarity based (read 'score-based likelihood ratio') model.

From the paper:

In light of this gap, this paper presents a method, developed as a stand-alone software application, FRStat, which measures the similarity between two configurations of friction ridge skin features and calculates a similarity metric.

Given the assertion that FRStat is not valid, and it's being used in casework. It seems like only a matter of time before a defense attorney picks up on this and challenges its use.

Here's where it gets somewhat interesting though, because like FRStat there has to be a practical application or what's the point? The Convergence model paper puts a practical application out there as a suggestion:

One of these models is particularly suited to be used in conjunction with database search algorithms and can easily be deployed on system such as AFIS or NIBIN.

Then at the end of the paper it says:

A prototype expert system is currently being researched by a large biometric company.

This is interesting because the marrying of an AFIS with a statistical calculation add on could start correlating scores with hit decisions with or without Examiner encoding [think ULW LFIS vs LFFS].

If you marry these developments up with some of the deep learning research also being done on fingerprint recognition algorithms, maybe Skynet will be here before you know it!

[Dr. Borracho]

Homework has been assigned.
Reading must be done.
Reading here, Boss.
Thanks!

[Boyd Baumgartner]

If you want a less technical summary prior to all the technical aspects of the Convergence paper, go to the 2018 Impression, Pattern and Trace Evidence Symposium Conference brochure.

Page 33 of the pdf has a description of a presentation that laid out the premise of the paper.

[ER]

So there's a lot going on here, and I want to clarify some things. I've heard comments from a few people that lead me to believe that there is a widespread misunderstanding about FRStat and then, by extension, the overall discussion of stat models.

First off, a traditional "score-based" model to obtain an LR for a fingerprint comparison is based on getting a score of how well the latent print matches the known print DIVIDED BY a score of how well the latent print matches the closest non-matching print in a large database. This is a way-simplified version of things, but it's accurate enough for an overall discussion.
If the latent and the known overlay very well, then the numerator will be higher, and it's a "better" match. If the latent and the known overlay poorly because of distortion (or the prints not actually matching), then the numerator will be lower and lead to a lower LR.
If the latent matches really well with another non-matching print in a large database, then that means that the latent isn't very discriminating. The denominator will then be higher, which will result in a lower LR.

There seems to be a belief that FRStat operates in this way. It does not. FRStat does not evaluate latent prints against a large database of non-matching prints. It does not generate the denominator. It only takes the numerator score and then compares it to the similarity scores of a static set of known matches and non-matches. Essentially, this provides a measurement of how well the latent and known overlay on top of each other. While this is part of what we do as LPE's, we also evaluate how discriminating the features are in total. Again, FRStat provides no evaluation to how discriminating the minutiae are in the specific latent that is being measured.


Now, in Cedric's paper linked above, he outlines his new model (Boyd calls it the convergence model) and says that it's superior to previous "score-based" models. I'm still trying to parse out and understand the basics of how his new model works, and I'm far from being able to understand why or why not it's superior to the score-based models. However, Cedric isn't comparing his model to FRStat. He's comparing it to actual score-based models that include both the numerator and denominator. I'm not going to speak to how his new model compares to score-based models or whether his assertions that score-based models are "are logically incoherent and violate the laws of probability". However, he's not directly referencing FRStat. FRStat is only half of a score-based model and therefore not even good enough to be compared against his new method. (My words, not his.)


To summarize:
- FRStat does not produce an LR
- FRStat does not compare each new latent print against a large database
- FRStat measures overlay similarity between two prints but not within an entire database
- We've got a lot to learn about the new "Convergence" model

[Boyd Baumgartner]

..the overall performance of FRStat is worse than the subjective comparison and decision-making method that it is meant to supplement.

This is why I reject the early adopter approach to 'innovations' in the discipline.

Wasn't FRStat validated by CSAFE?

[Bill Schade]

Well someone has to be brave and put these theories to the test in the real world and then weather the storm of criticism and scrutiny.

That is how science works (I thought), propose an idea and then improve it as others try to falsify it.

In the world of technology its called the "bleeding edge" for a reason

[Shane Turnidge]

this sort of reminds me of "Rocket Man" - Elton John
"... all this science i don't understand, it's just my job five days a week"

I appreciate the scientific challenge with the two statistical models but how relevant will it be when places like the UK have all but banned anything resembling the Bayes theorem from being introduced as evidence in a Court?

As the late, great, Albert Popwell said, "I gotsta know".

Shane Turnidge

[ER]

Well someone has to be brave and put these theories to the test in the real world and then weather the storm of criticism and scrutiny.

Absolutely, Bill. Totally agree. I very much appreciate the work that Henry and Co. did with FRStat. Overall, I think they also did a fine job in describing with FRStat is and isn't. But others still described it as an LR, and that misconception still floats around.

I appreciate the scientific challenge with the two statistical models but how relevant will it be when places like the UK have all but banned anything resembling the Bayes theorem from being introduced as evidence in a Court?

When a true LR is fully available to our field (and I think it's very, very close), it will provide a tremendous amount of assistance to our subjective conclusions. The relevance will start there. If a court bans Bayes theorem, well, not much to do about that, but it will still be literally our job to understand it and be able to explain it. I don't think that explaining the basics of Bayes and an LR is beyond the capabilities of a LPE. In the end, the part that needs explaining to the jury is a bit of logic and a bit of division. This isn't Rocket (Man) science. I anticipate that the UK court will be somewhat unique in its ban. The rest of us will quickly adapt to the new technology, just like LPE's had to adapt to explain AFIS 30 years ago.

[Shane Turnidge]

ER, you'd be amazed how many people still cannot articulate AFIS in a Courtroom.

FWIW The US legal system is very different to the British model. In our system the trier of fact is appointed not elected. That distinction has a very significant effect on legal decisions rendered by the Court.

Shane Turnidge

[ER]

Completely agree about not being able to articulate AFIS. It's very disappointing, especially since there are VERY few people working in this field that pre-date AFIS. AFIS vendors should review their training programs and make sure to offer this training to their customers/users.


As for the triers of fact in the US... well, it's complicated.

Many judges are appointed in the US as well. Elections for judges vary widely. Where I'm at, judges appear on the ballot in a list (without party listed). The voter then selects which judges should be removed. Blank = stay a judge. Check = remove. Most voters leave the whole section blank.
Still, I agree with your overall point. The way that judges are selected has an effect on the entire legal process.