Examiners' Attitudes towards Probabilistic Models

[Boyd Baumgartner]

Simon Cole discusses results from a survey sent out a while back. This came out in the FIGS email too if you subscribe to that. It's a good 'warts and all' discussion. Definitely worth the watch to gauge where you are in the discussion.



youtu.be/UBfm-jJiErk

[Alan C]

I'm bothered by a number of the reasons given for objecting to probabilistic reporting. It might be used by defense attorney to sow doubt, for instance. Well, that's their job. Anyone who is concerned about justice obviously wants to see people who commit serious crimes face appropriate consequences. But a profession that's based in science should follow the science, and if probabilistic reporting is more scientifically valid than categorical, absolute conclusions, and that leads to fewer convictions, so be it. We need to remember that we work for the truth and not for the prosecutor.

[Mike French]

I don't see a like button, so consider this a "like" for Alan's last comment. Not much to add other than binary decision making is so ingrained in the profession, that even those who support probabilistic methods tend to mistake categorical decision making with probabilistic, which was also mentioned in the presentation video.

[4n6Dave]

First I would like to say that I agree in principle that the discipline should be moving toward more objective measurement as that technology becomes available.

But I think that the authors of the paper are missing the point when it comes to the adoption of new technology. The acceptance of any new technology generally follows a curve where very few people use it in the beginning and then you get the early adopters then it would build to general use.

We are still in the innovator stage of probabilistic reporting. Given that a model has not yet been developed that is meant for use for any area of friction ridge skin we aren't even at a place where it would be possible for probabilistic/statistical reporting to replace all of Latent print conclusions.
In the US the only lab that I am aware of that uses FrStat is the Army Crime Lab. The paper reads like it is chastising current attitudes vs trying to meet examiners where they are and bridging the gap between the current practice and this new technology. The could have instead focused their efforts on getting a second then third Forensic Lab to adopt probabilistic reporting like FrStat.

The title "Mt. Everest—we are going to lose many: a survey of fingerprint examiners’ attitudes towards probabilistic reporting" was from a comment made by a participant and in the paper they admit they don't know what that means. The use of a comment that they don't know the connotation of on a paper about perceptions I think speaks to the fact that they may have missed the point.
Like Mt. Everest is probabilistic testimony in latent prints a lofty goal that can only be achieved by a slim few in our field?

[Boyd Baumgartner]

Where I would tend to interject here is that math is not a 1:1 overlap with science. It's more technology. AFIS matching algorithms are basically complex triangulation formulas. The chart provided here is the one that's used in the adoption of technology like Bitcoin for instance.

There's a reason the gap exists where it does, because the adoption is really ushered in by the pragmatics. Visionaries that cannot demonstrate pragmatic utility may really just be hallucinating. I consider myself a pragmatic on the issue and it's why I start threads like Show Me the Print or the OSAC 5 Conclusion Scale Research or Statistical Cage Match

Living in an area of the country where tech rules, I can tell you that the way technology advances is through A/B testing, where two competing processes/algorithms etc are placed side by side and their performance tested. At the end of the day, you don't insult your way into getting new ideas implemented, you demonstrate their utility. That hasn't been done yet.

[4n6Dave]

The point I was getting at was that efforts would be better spent getting early adopter "visionary" labs to sign on to probabilistic methodology/technology rather than lambasting the discipline for its views.

Until we start to see probabilistic methodology/technology in action we wont see the type of adoption that the authors are hoping for.

[Mike French]

I would add that AFIS follow statistical models, and that each vendor has developed their own model through decades of R&D, and none of these operational systems are related to LR. However, since they are proprietary systems, only the research scientists tucked away deep within those companies have access to the data necessary to map their scores to probabilities, or even know the specifics of the data used in their matching algorithms, but we know it's more than minutiae.

[Boyd Baumgartner]

I would add that AFIS follow statistical models, and that each vendor has developed their own model through decades of R&D, and none of these operational systems are related to LR.

^^^ This ^^^

I'm guessing if there was a demand, it would be noticed and met first and foremost by an AFIS vendor. All they'd have to do is sell it as a value added product in their offerings.

[Mike French]

I'm guessing if there was a demand, it would be noticed and met first and foremost by an AFIS vendor. All they'd have to do is sell it as a value added product in their offerings.

10 years ago it would have been unthinkable for an AFIS vendor to follow this type of business model, because their systems were, and still are, a black box (highly guarded trade secret), and also because customer attitudes would have opposed it.

I know Simon has started to ask the question why haven’t AFIS vendors led the way in developing a tool for probabilistic reporting. One big problem I see is changing the culture to be more transparent (changing the black box to a white box), since any tool for probability analysis should go through rigorous and independent validation.

In my opinion this would require the vendor to expose more secrets about how the specific features are used for calculation of the probability. Could they? Yes. Would they...? An alternative could be to integrate a third party algorithm into the AFIS workstation and bypass the AFIS matcher altogether. This would be a shortcut that would allow any vendor to protect their trade secrets. This is also why transparency and independent validation are vitally important.

[4n6Dave]

AFIS systems are image recognition systems and do not currently provide anything like a random match probability or LR.

Statistical models based on AFIS scores have been shown to have significant flaws in determining the "strength of evidence." Perhaps someone can refresh my recollection but I believe that Cedric Neumann who once proposed a model based on AFIS scores has since rescinded that support.

Since the bulk of expense and design of AFIS systems are for the ten-print record keeping portion I dont see that being a viable solution. We cant just throw some money at an AFIS vendor and have them turn on the statistical model setting.

[Boyd Baumgartner]

Here's the paper I think 4n6Dave is referencing:


Defence against the modern arts: the curse of statistics—Part II: ‘Score-based likelihood ratios’

[Mike French]

I've composed some thoughts on the AFIS part of this post in my Blog and on LinkedIn

[Shane Turnidge]

I may be just "blue-skying" but I wonder if there is a way to have real-time casework with results run parallel with another group that focusses on statistical/probabilistic support?
Then you would have an apples to apples comparison with hard data.

Perhaps this could happen in the background of an AFIS
Shane Turnidge

[Mike French]

@Shane Turnidge I don't think that’s blue skying at all, at least it shouldn't be. I helped devise such a study a few years ago, only to see a collapse in the supporting logistics at time, so we were not able to solicit funding. But the test design can be fairly simple. It's just getting commitment, which sadly is more complicated than the actual study.