IAI resolution 2010-18 (The Detail #462, 26/07/10)

[David Fairhurst]

Okay, I missed the business meeting last Friday, but I'd like to get this one out for discussion.

7. The use of mathematically based models to assess the associative value of the evidence may provide a scientifically sound basis for supporting the examiner’s opinion. Examiners shall only use mathematically based models that have been accepted as valid by the IAI in partnership with the relevant scientific community and in which they have been trained to competency.

The IAI seems to be setting itself up as an agent for the approval of such models for use. On what basis can an organisation such as the IAI do this?

[charlton97]

David raises an interesting point. I can see a situation potentially where I as an overseas active member would be asked to present evidence probabilistically using a standard or software that was not sanctioned by the IAI. Where would that leave me? I could be expelled because my home country insists I use a model not sanctioned by the IAI? I would also ask the question as to what the IAI means by the 'relevent scientific community'. I know only too well in my own research area that, like all good science, there is debate and discussion and not all scientists agree on the most basic of principles. I wonder what would happen if the IAI placed all it's eggs in one probabilistic basket, only for a rival developer to decide that other and perhaps better models exist? What may happen next? Will the IAI be in a position to enforce standards on AFR accuracy? AFR vendors with agreed qualities and accuracy levels? I think the SWGs could easily be set tasks around setting standards for practitioners that may not be enforcable? Thoughts?

[Gerald Clough]

The IAI seems to be setting itself up as an agent for the approval of such models for use. On what basis can an organisation such as the IAI do this?

Of course, any organization can pretend to whatever authority they choose. But when an organization gets very specific about what can and cannot be opined and requires members to adhere to those rules, they are gambling that they have sufficient authority. So, what's the nature of a resolution? If it's merely an expression of belief on the part of some leadership body of the organization, it's pretty safe and not binding on members. But the way this one is worded, at some point some mathematical model or models will be considered for "validation." Whatever form "validation" takes, you either declare as a matter of organizational belief that a model is valid, or you do not. If the validity of the model is accepted by most who won't simply refuse to accept any such thing, it's rather a trivial thing for an organization to sign onto it as a "me too." If it's that well developed, it's unlikely many experts would put forth some alternate model.

What I see as premature ans short-sighted is that we really don't have a very good idea of just what kind of models may develop with any credibility. It may well be that alternative models come to be routinely argued. Is the IAI going to "validate" one and not "validate" others while various experts are presenting alternative models, none of which are patently absurd? The presumption that for each purpose, there will one and only one valid model is pretending to more reliable knowledge than can be justified right now. I mean, here is the IAI pretending to pronounce on mathematical models at a time when they still can't say more than that such models may support an opinion. Genuine experts with mathematical tools developed appropriately won't and shouldn't much care whether the IAI has "validated" the model. To me, at first glance, it has a bit of a smell of trying to bring such modeling into the body of dogma, and not in a good way. There's still a good bit of "ole time religion" about the IAI.

And I suppose this implies that the IAI will "validating" any given scientific community as "relevant" before it accepts its input. Yeah, that's really gonna work well.

[David Fairhurst]

I just had word from Bob Garrett. He has cleared a couple of things up for me. in particular:

The intent of clause #7 was not necessarily that the IAI would approve the model itself, but would guide its members to those models that have been appropriately validated through an accepted peer review process. That is why we used the phrase "in partnership with the relevant scientific community." We didn't want to just say that the model must be validated and leave it to the developer to decide what "validation" meant.

I see the idea of the IAI guiding its members in the right direction as a good one and Bob has also raised my awareness of how the IAI is regarded in the US at least (i.e. nothing like The Fingerprint Society is in the UK).

Rather than attacking this resolution for its lack of commitment to the idea of probabilistic models I, at least, see it as a step in the right direction, away from 1980-5.
Still there may be an issue of my membership being at risk if my agency adopts a model that has not yet been validated. But then I would be more worried about my overall professional reputation in that case.

[Pat A. Wertheim]

Here's my caveat: I was not at the IAI, I did not attend the business meeting, and I do not know all of the conversation and debate surrounding this resolution.

That said, I liked the Resolution VII that has been the IAI policy for so many years because it said that while "possible, probable, and likely" identifications are outside the accepted practice of our discipline, if one were backed into a corner on the witness stand and forced to render such an answer, one could be excused from an ethical violation by adding words to the effect that the print might have been made by someone else. So, hypothetically, you might be forced by a prosecutor to admit that if you had just one more point, it might tip the scales and lead you to conclude that it was a correct identification. In the absense of that "one more point," you have rendered an "inconclusive" opinion, but deep down you feel it is possibly the defendant. Being forced to answer that way would be okay, as long as you said, "It's possibly him, but it could be someone else."

As I read the current resolution that has replaced the old Resolution VII, it offers the fingerprint expert on the witness stand no such "out." What if you are participating in a validation study and you come up with a likelihood ratio, but the model has not been finally and formally validated? No matter how strong the prosecutor pressures you, if you testify to the model, you are guilty of conduct unbecoming a member. This resolution should have a clause allowing an expert to testify with explanation, not simply prohibiting such testimony.

[Aaron Davies]

Friction ridge skin impressions can display varying levels of commonality (pattern type, ridge flow) in appearance with other impressions which do not derive from the same source.

Friction ridge skin impressions can share class characteristics (pattern type, ridge flow) and any associations based on these criteria require, ethically and professionally, that the examiner clearly state any limitations of their conclusions.

The use of mathematically based models to assess the associative value of the evidence may provide a scientifically sound basis for supporting the examiner’s opinion. Examiners shall only use mathematically based models that have been accepted as valid by the IAI in partnership with the relevant scientific community and in which they have been trained to competency.

Mathematically based models may not be used as the sole determinant when concluding that friction ridge impressions share a common source. The use of mathematically based models does not relieve the examiner of responsibility for their expert opinion.

I don't think this limits us to using mathematical models. I think it allows us to use them once they are validated to help explain our conclusions.

This resolution allows us to give true testimony. I think that the old resolution made it difficult to say you could not include or exclude and many examiners simply said the results were negative or changed the latent to no value for comparison/identification. I think you can now comfortably say that the comparison is inconclusive and here is why. Then you can show what's matching and what is not and explain that although dissimilarities are allowed, this particular print is out of tolerance.

The key words, to me, are "ethically and professionally". Some people could push the envelope but those same people might have pushed the envelope to identification before.

I also believe that this is a very small percentage of cases where this is an issue. Typically it will be and identification, an exclusion or you need better prints. Very rarely should this come up. I would also still stay away from the "possible, probably and likely" answers.

Aaron

[David Johnson]

7. The use of mathematically based models to assess the associative value of the evidence may provide a scientifically sound basis for supporting the examiner’s opinion. Examiners shall only use mathematically based models that have been accepted as valid by the IAI in partnership with the relevant scientific community and in which they have been trained to competency.

The original way it was written had the scientific community listed first and the IAI as the partner. It was suggested that since this was an IAI resolution that the IAI should be in the driver's seat. It does seem a little presumptuous on the face of things but if you think about it, it kind of makes sense.

Since this is an IAI resolution that is affecting its membership the IAI should have a strong hand in which models we as a group chooses to accept. It does also say that the model would be accepted by the scientific community as well.

4. It is the responsibility of forensic experts to offer a clear and unambiguous presentation of their conclusions.


5. Friction ridge skin impressions can display varying levels of commonality (pattern type, ridge flow) in appearance with other impressions which do not derive from the same source.


6. Friction ridge skin impressions can share class characteristics (pattern type, ridge flow) and any associations based on these criteria require, ethically and professionally, that the examiner clearly state any limitations of their conclusions.

Pat, these provisions replace that part of resolution VII. Only now you are not limited to when you are asked about it. You can clear state your results and conclusions even if it is an inconclusive without having to be asked and backed into a corner.

[g.]

I was at the conference, the business meeting, and can also offer some background on this
resolution. It really came as a result of the Standardization 2 Committee's work over the
last 2 years. Unanimously, the Committee found that this profession would benefit by exploring
the use of probability models (in casework) and they could be useful tools. Now, to what degree
we should use them, which ones, how to validate and implement and train, which cases/who could
use them, etc....well that's where the 12 round fights were...

SO the Std 2 Comm. Report that was produced clearly spells out those findings and recommendations to the IAI
Board. (Unfortunately, due to some NIJ rules, the report is not available yet, but it was read
IN WHOLE!, by Ron Smith at the meeting. NIJ will hopefully make it available soon after it has
undergone a few more edits (although the authors of the FRS sourcebook are still waiting to see the
chapters we wrote 3-5 years ago actually appear in print one of these days...). This resoultion
(and the findings of the Std 2 Report) were the best compromise between groups of people that disagreed
strongly on a controversial topic. But yet we all agreed it was important to go down that road.

This resolution, which I DO fully support, removes the obstacles for going down that road. It opens the
door and now puts the ball back in the courts of the probability folks to say "Demonstrate that this thing
works and produces valid and reliable results". Now that the roadblocks are removed you will begin to
see more work in this direction, like some NIST benchmark and performance testing, validation of the models under ISO,
and more training to bring the community up to speed and become comfortable with new language and a new view on this evidence.

Keep in mind too, that the IAI is an international organization. While there is no doubt that the U.S. enjoys
the bulk of the membership and predominantly influences the direction of the IAI, it is still to represent
International views. In Europe, probabilities in fingerprints have begun to be reported (France, Switzerland,
Belgium, to name some)...and possible soon, some others. This just isn't a "NAS-thing", it is a world-wide
shift in paradigms and views.

In the meantime, Pat, nothing has really changed, your court answer will still be "at the moment the profession
in the U.S. does not report statistics or probabilities, like DNA, but still provides an opinion of source
attribution (ID, Exc, Inc). Groups of researchers are currently developing, testing, and refining stats models
that will quite soon provide these numbers, but at the moment, we do not have a generally accepted model or approach
yet."

Aaron, I suspect you're right. Based on some prelim numbers here at our lab in field testing the FSS model, we
found that only a fraction of our cases 1-5% or less may involve comparisons where the model would provide useful
additional information. And even then, one could administratively decide to use only in certain types of cases
(i.e. cases with a court date, crimes against people, etc.). These would be administrative decisions to be
decided with a DA's office, lab mgt, QA, etc.

I fully support this resolution. I think it is an appropriate, progressive direction to take, yet still
provides the caution and forethought to integrate (for some) very new and radical concepts.

Besides when people started really using ridgeology, Level 3 Detail, creases, and the holistic approaches we see commonly in
a Q&Q approach, did EVERYONE start doing this overnight? Did utter chaos occur in the courts? Or was it a slow
transition, that took almost decade, with a few big leaps along the way?

Just recognize, now that the door is open, there is still a looooooooooooooooong hall to go down before
we arrive. At least now, the door is open.

g.

[Neville]

And I still think it's a load of bollocks!

Just a crutch to make you feel like 'I am now a scientist and therefore I am right'.

If it is 50 to one chance of it being someone else's or a zillion to one, it still amounts to the same thing, the next fingerprint form I turn over may be the one the proves that fingerprint identification is a crock! and any defence lawyer worth anything at all will work that out in a flash. It makes no difference in my mind if you are suggesting it is a zillion zillion to one the next set of fingerprints you turn over maybe that one, what you are suggesting is that the whole identification of fingerprints is based on that fact that there must be two people with the same fingerprint.

Sorry I for one do not see that as a really good idea. In the last one hundred plus years of identification work this has never been found to be the case and I will be the first to say I was wrong let out all the prisoners who have been convicted on fingerprint evidence, close down all fingerprint offices its a crock. But my guess is that in 1000 years from now fingerprint identification will be just as valid.

Aaron this statement of yours - 'This resolution allows us to give true testimony. I think that the old resolution made it difficult to say you could not include or exclude and many examiners simply said the results were negative or changed the latent to no value for comparison/identification. I think you can now comfortably say that the comparison is inconclusive and here is why. Then you can show what's matching and what is not and explain that although dissimilarities are allowed, this particular print is out of tolerance.' - left me feeling very sad that that is where fingerprint identification has slid down to in your part of the world.

A very sad state of affairs.

[atorres]

Neville wrote: "what you are suggesting is that the whole identification of fingerprints is based on that fact that there must be two people with the same fingerprint."

I haven't seen or heard of anyone (who are in support of probabilities) who have suggested that there are two people with the same fingerprint. That is if we are talking about the entire known print. The only thing that I have ever heard suggested, is that there is a chance that someone else may have the same or similar arrangement of detail found in the small (distorted) latent print as in the known print. I could go on, but the short of it, is that even though you may still think it is a load of bollocks, I felt that I needed to at least clarify that what you are saying has been suggested, in fact has not been (or at least I have not heard it) and the probability option is looked at in a different light. Where it is not looked at as a crutch, but a useful tool.

Anne

[kevin]

I assume there is going to be some inevitable conflict with an examiner testifying to ident/non-ident/and inconclusive and have it not jive with a population or statistical model. Even if this happens in a small percentage of cases - I wonder what is the IAI's position is going to be on that? Was anything along those lines discussed? i.e. conflict resolution for lack of a better term??

I don't think it is going to be a problem in most idents or exclusions or even inconclusives but I imagine it has the potential to happen (?)

[g.]

Kevin,

An excellent question. It was NOT discussed at the business meeting when the resolution was passed by an OVERWHELMING (95+%) majority. That question did come up in Std 2 Committee meetings. Those types of questions are exactly the types of questions that come up routinely in the probability workshops and courses that Cedric Neumann and I teach (and others are teaching too: Christophe Champod, Paul Chamberlain, etc). At the IAI, Steve Meagher asked many questions similar to that one and others during his presentation: "I am Probably an Expert". Since we don't have a model validated and accepted by the IAI yet, there's no point in having a definitive answer to that question, but suffice to say, that when a model is available, those issues will be dealt. At the moment, Cedric and I actually give cases in the class/workshop that usually give a differing LR than the expected opinion of the analyst. This allows the students in the class to see how to handle exactly that type of situation. Typically if one remembers the limitations of the model, what it does and doesn't do, those are fairly simple questions. Have you ever taken an ID to court that had a low AFIS score? Or excluded an AFIS top candidate even though the score was well over your threshold. Same thing. The numbers never should replace the expert judgment of the analyst. And of course, given the limitations of the model, they won't always support (i.e. be in the same direction as the decison of the analyst). Having worked with 100's of students and this issue, I can say from experience, analysts will have little trouble dealing with this issue.

As for Neville's post. Well, Neville is certainly entitled to his opinions on a professional board such as this. Atorres hit the nail on the head. He's arguing into a mirror apparently, since the issues he's raising have nothing to do with the price of kiwis. Argue away Neville.

[[[[Psst!!! (if you read Neville's post and thought "right on! Good point". We should get you in a probability workshop stat. SInce those issues have nothing to do with the probability models and where this profession is headed in the future).]]]]

And finally, Aaron I thought you made perfect sense. You want to know how to accurately convey the weight of an "inconclusive" decision where there is agreement between latent and exemplar, but insufficient to ID (and better exemplars would not help). While many agencies would simply "no value" this impression. Many would not and would report "inconclusive". How do we convey "almost an ID decision" to the jury and yes, a statistical model would obviously convey that information, easily and with proper training, it could be articulated to any juror. Hopefully next year at the IAI, if we get our moot court session for probabilities going, we will demonstrate that.

g.

[Langdon Alger]

I too fail to worship at the alter of probability models. We spent all this time working away from the image of being point counters just to return to a mathematical model that counts points? I'm not buying it, especially if Glenn is selling it. Remember, you can't spell ego without a g.

I've taken the stats class and I have a hard time accepting any model that is based upon 'close non matches'. The notion is merely a red herring distracting from real research from the likes of Anil Jain among others


http://citeseerx.ist.psu.edu/viewdoc/do ... 1&type=pdf

[g.]

Thanks Langdon. I will put you on my fan club mailing list.

As for the salient part of your post, I'm glad you put a link to Anil Jain's work. Jain's group HAS been doing some fantastic work indeed over the last decade at MSU. They even presented at the IAI on using scars as well to automatically detect potential persons who may have deliberately mutilated their fingerprints to hide identification.

However, every bit of work Jain has done, including the article you posted, has been for the biometric community. The use of models and latent prints hasn't really been in the focus of his work. While his models are quite promising for 10 print and US visit machines, I am not so sure they will do well with latents (and he hasn't done anything yet to show they will work with latents). But I agree, he's quite clever and should be enticed into working on the latent problem. But I suspect the money is much better in the biometric community.

If you are attending next week's NIJ pattern evidence symposium, there is a presentation from a colleague from Lausanne named Alexandre Anthionoz. His and Christophe's research will show you why Jain's model will probably not be helpful for latent work. L3D is simply not reproducible enough in case work and since it is so hard to find matching L3D without cherry picking and seeing it in the latent and known consistently, it makes it very difficult to build a latent model on it. It is not impossible, just not practical per se.

And as for your comments about the model...sure the FSS model is limited to L1D and L2D. You seem to support a statistical approach that includes L1D, L2D, and L3D. Great, me too...if it's feasible and practical. I don't care what model is used. I don't. I don't work for FSS. The stats courses we teach only teach the concept of probabilities and stats. Promote the theory, let them pick the best model available. If Jain's model will work for latents, then they should submit for benchmark testing and if/when NIST puts a call out for testing. If it holds up for latents, great. In the meantime, the FSS model has been most useful at least to demonstrate the concept, I have direct access to the designer (Cedric), and it works reasonably well enough to demonstrate a concept, plus they are well on their way through several validation/testing/study steps. Those studies put them far ahead of the curve.

g.

(nice Simpson's reference...very obscure...but if you want to debate probabilities, no need to hide behind a pseudonym...we're all reasonable professionals here).

[Graham F]

It has been mentioned in this thread that the future models may well be a "crutch" for examiners. I am of the opinion that this may hold some water, especially examiners report out a high percentage of inconclusives on their cases (INCONCLUSIVE COURT TESTIMONY thread +70% [47%] inconclusives reported).

There will always be people who "sit on the fence" rather than make a decision and a stat model will aid that process by providing two crutches so examiners do not fall of that lofty fence.