My Twitter length rant on the SWGFAST Draft for Comment

[Boyd Baumgartner]

I will never get up on the stand and say I 'mated' two prints.

[Boyd Baumgartner]

http://www.swgfast.org/documents/error/ ... FT_1.0.doc.

2 Terminology
2.1 Definition of source attribution terms
2.1.4 Mated
Two impressions from the same source, as determined by ground truth knowledge or consensus determination.

This is becoming comical in the sense that no one sees that SWGFAST is constantly chasing its tail with a linguistic shell game, blind to the fact that they semantically hamstringing those who hold allegiance to their guidelines(I'd be happy to elaborate if anyone cares to have the discussion). While I obviously cannot impugne people's motives for doing such things, it appears that these moves are a misguided attempt to legitimize the discipline by putting a new coat of paint on it, but doing little to address the underlying rust.

[sharon cook]

"Mated"??????? WTF??????? OMG!!!!!!

[kevin]

What is 'ground truth' supposed to mean here? What does it have to do with fingerprint examination? I've only heard that term in use for military targets and map grids.....not the most intuitive appication of the word here in my opinion. Are we at war?

"Mated"??????? WTF??????? OMG!!!!!!

Get behind that bifurcation Sharon - you'll have perfect defilade from the incoming bs!

[g.]

Interesting that THIS is what upset people in that document.

This document includes a lot of new terminology for fingerprint examiners. Fingerprint examiners are not trained traditionally to discuss error rates in a meaningful way. This document attempts to translate and give guidance to concepts such as error rates, discovery rates, etc; terms found when discussing error rates in a context that are consistent with other established fields, such as signal detection theory, diagnostic testing, discriminability tests, or binary hypothesis testing.

I would invite those of you surprised to see the term "mated pairs" to google +mated +pairs +images +fingerprints. YOu will notice the top dozen or so hits will reference NIST, the national institute of standards and technology in the U.S. These terms, while unfamiliar to many traditional fingerprint examiners, are not unfamiliar to those who routinely measure error rates, or in biometrics, where FAR/FRR (false acceptance/rejection rates) are commonly measured for biometric systems. NIST, THE STANDARDS-making-agency in the U.S., uses these terms constantly in reference to testing fingerprint systems.

This SWGFAST document is does ask a lot of traditionally trained fingerprint examiners, namely to come in line with generally accepted practices for measuring error rates of diagnostic tests (e.g. ACE-V), but it is in line with what the NAS has tasked this discipline and SWGFAST. It definitely asks the community of practitioners that either intend to attempt to measure error rates at their laboratory or report them in court in some manner, to know what the standard terminology is in other sciences that use error rates. It means that this profession yet again has more pressure placed on it to grow and meet the challenges placed on it by bodies such as the NAS, the courts, and even members of the fingerprint profession.

As always, the community's input is considered by SWGFAST, and if you don't like these terms, offer others that are accepted generally in the wider sciences.

PS-to Kevin's question, "ground truth" refers to the administers/developers of the test know as a matter of fact whether an individual is the donor of a latent print or not. i.e. we know the "correct answer" for source. A CTS proficiency test has the ground truth for all trials. One knows if the latent print originated from the donor or not. There are a couple of published fingerprint studies with this term already in use and several more in publication where error rates are measured under conditions where the ground truth was available. And again, these are standard terms (common outside of this discipline).

g.

[g.]

PS

I will never get up on the stand and say I 'mated' two prints.

And no one is asking you to. As your next post shows, mated refers to two images from the same source as DETERMINED BY GROUND TRUTH.

This does not apply to case work since ground truth does not exist in case work. In case work, you would still "Identify" or "Exclude", these are the standard conclusions you would provide.

But in proficiency testing, or other experiments, you might tell the reader of the paper that out of 100 trials of paired latent prints/exemplar prints, 50 were mated (from the same source) and 50 were non-mated (from different sources). The number of mated/non-mated pairs presented to the test-taker is critical for establishing the denominator in the error rates.

g.

[Tazman]

By gosh, I was born in the 20th Century, trained in the 20th Century, and worked for a decade or more in the 20th Century. The way we did things in the 1990s before Daubert and NIJ was just fine. You have no right to drag me kicking and screaming into the 21st Century. Just who do you think you are, anyway?

P.S. -- Thanks, g.

[Boyd Baumgartner]

Glenn, if NIST jumped off a bridge would you? Sorry, I couldn't resist.

In all seriousness however, your post brings up more philsophical issues that aren't being addressed, which is really the source of any displeasure I may have. I'll address them and try to keep it short and to the point, as long posts just tend to kill any discussion.

1) Over-reliance on Statistics

Statistical fit does not equate to scientific meaningfulness. The models that I've seen, read and heard about fail for the following reasons:

They're uninformative. They're non frequentist, retrodictive, exclusionary of data considered in an actual examination and are just as subjective as the subjective probabilities of individuals that are the result of cognitive processes that occur in an examination.

An example may work in this scenario.

questionmark.jpg

What will statistical models say about the question mark shape in the middle? Absolutely nothing. There will be two points which may or may not take into consideration directionality. However the anecdotal reality that this print made it on to a gallery of 'remarkable' configurations in fingerprint impressions as determined by subject matter experts tells me something that statistics won't.

What about the Mayfield print. Put the Mayfield print into any statistical model using the events shown in the OIG report and tell me what it says.



2) Inarticulate Leadership / Agendas

The history and the Philosophy of science is not new. Every time a critic puts out a paper, we're jumping through hoops in a reactionary way. No one can actually articulate that no matter what school of thought wants to attack what we do, there are rational examples that pertain to that school of thought. I'm not going to go into nerddom too much here, but if it's a Kantian paradigmatic model, or an assymetric theory of evidence they want, it's really not too hard to show. In addition to the lack of articulation, people dogmatically accept new studies without question. We don't testify to probabilites, now we do, now we don't, it's Glenn's model, no it's Cedric's, no it's a new one even though their assumptions aren't clear or entirely accurate. Just throw up some box plots and some Neyman Pearson terminology and all the Examiners will line up and sing a chorus of praises to our new hero. Allow me to put forth this prediction, soon (within the next few years) this will devolve into some examiner jumping ship to academia in order to have their agenda funded, all the while learning at the common Examiner's expense.


3) Semantic Shell Game

While I made mention to this earlier, it's a little more than just making fun of the word 'mated'. Consensus and ground truth references mean squat if people don't even have an idea what they mean or how they are relevant. For instance, (venture into nerddom again) what is the difference between Mob Psychology and consensus justification? (Kuhn's stance).

Or better yet, does ground truth trump consensus? It sure didn't in CTS test 10-516

http://www.ctsforensics.com/assets/news/3016_Web.pdf

Page 3 Although there were twelve latent prints to be examined in this test, the results for item 5D were such that CTS did not consider that a consensus result had been obtained and therefore no inconsistencies were assigned to the reporded results for this item. There were 236 participants (71%) who identified this print with the expected "1RT" however ninety five participants (29%) did not identify it ("NI" or blank.)

Here they are saying that ground truth doesn't matter, but neither does consensus. Statistics are just as susceptible to bias as qualitative processes. The old increasing your N until your unintended observation becomes 'statistically insignificant' is something never mentioned. Or as the old saying goes, 'the solution to pollution is dilution'.

[g.]

Boyd,

You seem to have a few frustrations here. I'm not sure what the source of your displeasure is, but sorry you feel that way. Instead of blogging into the void, why not publish a paper or perform some tests of your own? If you can write a better paper on error rates and give better definitions than SWGFAST has (using those that are traditionally used in other domains), than I would certainly welcome that and it can subjected to peer review and scrutiny. You could change the entire landscape and views on error rates in this domain and others.

Point 1

Glenn, if NIST jumped off a bridge would you?

At least it would be a standardized bridge...

They're uninformative. They're non frequentist, retrodictive, exclusionary of data considered in an actual examination and are just as subjective as the subjective probabilities of individuals that are the result of cognitive processes that occur in an examination.

An example may work in this scenario.

You've already made up your mind then about the models. I won't try and sway you in a forum like this. However your example is very telling. Why do you think statistics (which are just representations of DATA) would NOT be helpful or informative. So if you went to court on that print and tried to explain the significance of the ? core, what would you say? "Based on my training and experience, I know this core is blah blah blah?" (I'm asking a serious question, how would you defend the significance of the ? core. You are using it as an example here, so I assume you are putting significance on it.).

Yet, I could get an intern and in 2 days probably manually go through 1000 cards (10,000 fingers) maybe 10,000 cards in a week or two....looking for any examples of "?" in the core. And I could walk into court and say "Based on a survey of randomly selected whorl patterned fingers from this database, the occurrence of such a feature was observed less than 1/100,000 times. These data comport with my personal observations, based on the hundreds of thousands of comparisons I have performed (i.e. training and experience) in my career."

Which answer is better in court? Data/Statistics enhancing training and experience...or training and experience alone. I'll take data any day over "I'm an expert". Data tends to shut down most arguments.

Statistics alone are NOT the answer. They enhance and support what we do. Examiners who say that statistics is just the new fad shoved down our throat as a solution to solve our problems have 1) missed the message, 2) not taken stats classes from Christophe, Cedric or me (we certainly aren't saying otherwise), or 3) promoted their own agenda out of fear of something they at this stage they do not feel prepared enough to talk about in court. Doesn't mean the rest of us can't evolve...

Point 2

Not sure what you are getting at...or implying. I couldn't tell if that was directed at me.

Point 3

I feel your frustration for using terms that are new to examiners. That's the point of the document: to inform the community and give them definitions they can use. If you go to the source documents for those terms, they can be overly complex with mathematical formulae. SWGFAST attempted to translate those terms for analysts so they would have a better grasp of these concepts in our own language. If you don't think SWGFAST was successful in doing so, offer better language. It will definitely be considered.

Back to you then. If you don't intend to discuss error rates in court in this manner, how will you answer questions about measuring decision error rates for ACE-V in court? What sort of answer do YOU give in court? Is it consistent with the other examiners at King Co.? I'm always willing to listen to new answers to error rates, if they are sound and defensible, (with data).

g.

PS-just for the record, I was NOT in the committee/one of the author's of that SWGFAST document. So this is not personal for me. When I read the draft, I was really pleased that SWGFAST was leaping into the next century and finally embracing terms and concepts that are in line with STANDARD (I can't even begin to express HOW standard and common they are elsewhere) testing procedures. Finally a true move away from semantics and what I saw as the real "shell" game: "zero error rate", "no error rate", "one can't be calculated", "impossible to calculate for the method", etc...

[Boyd Baumgartner]

I wouldn't necessarily call my examples frustrations as much as I would call them observed philosophical deficiencies with the direction of current trends.
I wasn’t necessarily looking for a refutation of points as once again, they are largely philosophical.

You mistake my observation of the deficiencies of statistics for ‘my mind being made up’. It’s as if there’s a little ‘Chicken or the Egg’ being played here. I’m not interested in being convinced, I’m interested in the author of such a study point out or at least recognize their model’s limits and assumptions. My observations on statistics are actually the result of historical merit, pragmatic reading of the results, and hold so far as to be predictive. Run the Mayfield print through PiaNOS and see what comes out. Beyond that however, Retrodiction runs counter to the basic Popperian notion of falsifiability. Not that I am a strict Popperian, as it has its own limitations, but confirmation is cheap. Even if I were just playing epistemic anarchist, the notions still hold, numbers ignore qualitative elements of what we do and numbers at some level require qualitative interpretation of what qualifies as phenomenon X being observed. I’ve seen some ‘close non matches’ that I don’t agree with where I think the person was trying too hard to make it a close non match. If that’s what the database that underlies your statistics is based on, it’s not helpful.

The history of scientific endeavors is chock full of people putting numbers behind things to make them sound rational and appear causal. Look at the history of Eugenic Science looking at skull size by race and making causal inferences that aren’t justified. Let’s look at a more recent example, Vioxx. All the numbers were there, it’s just that the fit was adjusted to throw out all the data that was contrary to their belief
http://en.wikipedia.org/wiki/Rofecoxib#VIGOR_study

Just because data comes in numeric form doesn’t make it any more causally relevant. I’m not eschewing the use of data all together, it can be useful, but only in as much as it is useful. (how profound). If your model doesn’t make use of contextual information specific to the case at hand, how useful is it? That was the point of the question mark in the fingerprint. Not all fingerprints are created equal, so why use a statistically rigid model that treats them as such?

For anyone interested, here’s the first chapter to a book called ‘The Cult of Statistical Significance’, it’s not written for stats nerds, it’s written for the everyman. It has been influential in my developing my stance. http://faculty.roosevelt.edu/Ziliak/doc ... 20etc).pdf

While I’m not a die hard ‘training and experience’ guy, tell me they don’t matter the next time before you get on an airplane. As Kuhn points out so rightly, expectation and perception matter. Look at the card and tell me they don’t.



The philosophical difference between us is that you seem to want to go one direction and I would prefer a different direction. This discipline wants to abandon the Qualitative aspects of the profession so adequately pointed out by Ashbaugh’s seminal book “Qualitative Quantitative Friction Ridge Analysis”. I say rather than hang our hats on overstated statistical estimations that will undoubtedly come back to bite us the way our overstated certainty estimations [absolute conclusions] did, we pursue the philosophy of science (no surprise there given my frequent references) and look for how the answer was/is found in other disciplines. I’m more interested in the sociology of knowledge; how do groups ‘know’ something, as that seems to be more pertinent especially given the notion of consensus.

[kevin]

PS-to Kevin's question, "ground truth" refers to the administers/developers of the test know as a matter of fact whether an individual is the donor of a latent print or not. i.e. we know the "correct answer" for source. A CTS proficiency test has the ground truth for all trials. One knows if the latent print originated from the donor or not. There are a couple of published fingerprint studies with this term already in use and several more in publication where error rates are measured under conditions where the ground truth was available. And again, these are standard terms (common outside of this discipline).

The fact that it is used alot more outside the scientific community and has a different connotation is what bothers me about it. Sure we can define it in a SWG document but it isn't going to change how the term is perceived - including the vernacular. It simply isn't the proper context for using the word I think. What is wrong with saying donor when we are referring to a donor?

But I think the terminology document has good stuff in there - having everyone on the same page with how they define terms is a good idea as well as including a framework for if/when we can include any statistical data. It is a living document and we'll see plenty of other changes I'm sure. But I think keeping terminology specific to the LP community and doing a minimal of borrowing from other disciplines would behoove us as a whole. But if you say ground truth is used in other research I'll buy it Glen - cheers.

We could replace some of the other definitions with jargon from Top Gun? 'Erroneous individualization' (also known as Type I error, You've lost that loving feeeeeling) - I try to keep it light in here you know

[g.]

Thanks for keeping it light Kevin!

Boyd, I have found in the past, any more than 3 responses from 1 person in a thread and people just tune out and get sick of it. So I will just respond this last time and let other people chime in. This is something that does affect us all, and can shape testimony in these admissibility hearings. It's not trivial.

We can differ, as you say philosophically here. Your views are similar to those I've heard Les Bush, John Vanderkolk and John Black strongly support. I do not deny the power of ridgeology and the QQ approach. Saying that I am moving away from (not that you directly did) shows a fundamental lack of understanding my view. I want nothing more than to use the QQ approach, while enhancing it with measurements (statistics/data) where we can. What's wrong with coming up with stats for the L1D and L2D and saying the models got us "this far = X". Tack on some L3D that can't be measured (although technically, it can, for example in the Jain model and other emerging models). We don't need a stat for EVERYTHING, but why not use it when it's available.

What bothers me the most, and I don't believe it's deliberate at all, and I have always found you to be a very intelligent, well-read, insightful scientist, but I have concerns about the information you are giving about stats/models. Some of your statements I think need clarification. For example:

I’m interested in the author of such a study point out or at least recognize their model’s limits and assumptions

I'm not aware of any of the authors of modern papers/models who ARE NOT discussing the limitations of their models. Whether in print, in workshops, or conferences, I hear quite a bit of focus on the limitations of the models. In fact, that's about ALL I seem to hear, rather than the merits sometimes.

Run the Mayfield print through PiaNOS

1. We have. Anyone who has taken our 40 stats workshop knows that this is one of exercises. Each student runs the Mayfield print through THE MODEL. They do it in two parts, the lower half (the 7 points searched in IAFIS) and then add the upper left and top right and see how it affects the numbers. The model suggests that it is impossible for "normal twisting and movement distortion to explain" the discrepancies. Which is true in the Mayfield case, they used "triple tap" to explain the differences. The model says "it's not him", but like in real life, the human examiners (the real weak link in the chain) would over ride and invoke the magical "triple tap".
2. PiAnoS is not THE MODEL. PiAnoS is simply a documentation system. It's like saying, run the print through the Photoshop model to see what you get for a statistic. The fact that PiAnoS is confused with the actual statistical model is a concern...they are different.

numbers ignore qualitative elements

I agree. So measure what we can with current technology and tack on the qualitative stuff as we already do. Why make the WHOLE process qualitative, when we can measure and quantify a portion of it?

If your model doesn’t make use of contextual information specific to the case at hand.

But it DOES make use of the specific minutiae arrangements for THAT latent print. Are we talking about the same model? True, It doesn't account for ALL of them, or every type (i.e. it can't handle dots or your question mark, but I showed how that could be handled separately.)

Not all fingerprints are created equal, so why use a statistically rigid model that treats them as such

It doesn't. Again, have you read the same papers that I have? Are we talking about models post 2004? I'm confused when I see these statements...

I think the person was trying too hard to make it a close non match. If that’s what the database that underlies your statistics is based on, it’s not helpful.

Again. What model is using a close non-match database? Where are you getting these statements from? Maybe it's just a slip, but this is not accurately characterizing what the models do...at least those I am familiar with...

This discipline wants to abandon the Qualitative aspects of the profession so adequately pointed out by Ashbaugh’s seminal book “Qualitative Quantitative Friction Ridge Analysis”.

I love Ashbaugh. I love Locard. I love what they have done for the profession. But if they were "adequate" enough, and that's all we needed, why then did the courts, the NAS, and scores of other academics, not just read Ashbaugh, close the book and say "Done and Done". Grade A. Stamp of Approval. There it is. Done.

Seems like something might have been missing?


(For the record, thanks this was a good debate and I do appreciate your views. We hear them all the time. But I also know some die-hard ridgeologists who are coming around and seeing value in BOTH approaches, which is what we actually are promoting). Have a good day. I'm turning it over to someone else...contrary to popular belief, I do actually have to do casework...

g.

[Boyd Baumgartner]

Likewise, I appreciate the discussion. Apologies for the sloppy terminology on the stats/model side I have to actually work in between these posts. I know that PiAnoS stands for Picture Annotation System and the demo runs on an Ubuntu LiveCD at that, I've downloaded it. Sub PiAnoS for algorithm in the above dialogue.

With regards to rigidity of the stats model, I don't mean rigid in the sense that it doesn't take into consideration variation, I mean rigid in that there's not a 'one size fits all' sense.

Blame all the other inconsistencies on Anne's ability to articulate what's going on with your research.

[atorres]

Well thank you SO much Boyd for dragging me into this rant...

As far as a reply goes... from the very beginning I told you that I am not the one to explain it or do it justice, since I am JUST learning it all myself. When in the learning stage, one starts to understand it in their head BEFORE they can articulate it to others. With that being said, my notebook from Glenn and Cedric's class is right there in my cubicle for anyone to read AND there are lots of other articles and information AND people, that would provide you with a better understanding of all this! So... don't go blaming me for all the other inconsistencies when it's best to check and couble check your sources and information before posting.

As always... thanks for keeping it interesting Boyd!

Anne

[Boyd Baumgartner]

So... don't go blaming me for all the other inconsistencies when it's best to check and couble check your sources and information before posting.

I'm sure your insistance to 'couble check' is a parody of itself......