Statistics, Ridgeology and ACE-V Course

[jesmeade]

Hello everyone,

Glenn Langenburg and Dr. Cedric Neumann will be teaching their Statistics, Ridgelogoy and ACE-V course in Kissimmee, Florida at the Osceola County Sheriff's Office the week of December 12-16, 2011. If anyone is interested in attending this course please click or copy the the link below to read the details of the course and fill out the enrollment form. I hope many of you will be able to attend and enjoy some decent weather for December.

http://www.twonsforensics.com/Two_Ns_Fo ... ining.html

So, for you guys that went to the IAI conference in Milwaukee, what were your thoughts on the moot court presented by Glenn?

Jessica

[Boyd Baumgartner]

This will probably come as no surprise to anyone who has read any of my other postings on the topic, but I am not sold in the least bit.

I'll just paste an email conversation I had with someone about the topic:

Reminds of me of the quote "Statistics are like bikinis, what they reveal is suggestive but what they conceal is vital."

After thinking about the moot court, I have to say that statistics are no substitute for judgment. I found them to be irresponsible at best, deceitful at worst. Statistics are useful for measuring risk, not uncertainty. It’s the difference between Vegas and Wall Street. In addition to that realization, the use of statistics was ambitious as opposed to conservative. Why do we want to push the boundaries of decision making when civil liberties are at stake? Will Glenn and Cedric be responsible for the mistakes when they happen (not if, but when)? There are ethical implications that are not being discussed. The whole appeal to modernity by Glenn came off as a little melodramatic.

I think the key to dismantling their claims is twofold. First in the fact that they mentioned they validated the model through AFIS. The implication here is that they basically created a matcher algorithm, and we know how the many variables can affect what AFIS spits out. As a matter of fact, it’s single ID AFIS cases that generate the most errors. We do not accept candidates in AFIS as matches based upon probabilistic accuracy of the matcher as indicated by the score or rank. Instead we evaluate an individualizations based upon the merit of the information in the impression. It’s context dependant. Meagher alluded to that somewhat with his drawings of the loop and whorl with similar features marked.

Secondly, the wild variability in the number as the minutiae are plotted is a huge weakness that can be exploited in some manner. It would be easy to see how this could be abused by manipulating the plotted points until the number is the largest it can be. All in all it seems like a step backwards rather than a step forward. It’s like a return to point counting…

Bottom line, it's way too soon to even begin selling this as anything approaching functional for case work.

[cchampod]

Dear Boyd,

May I suggest the following book that may help demystifying statistics, probabilities, uncertainty and decision making:

D.V. Lindley, Understanding Uncertainty. Hoboken: John Wiley & Sons, Inc., 2006.

It is great small book that helps to live with uncertainty without a bikini.

Kind regards

Christophe

[Cedric]

Dear Boyd,

Firstly, for those who did not attend the moot court at the IAI2011 in Milwaukee, we need to explain that we reported (during the moot court) latent print evidence associated using statistical support. The court had a Judge (played by a local DA), a DA (played by a public defender from MN) and a defense counsel (played by Il Dottore Steve Meagher ). The court had also a jury of 11 individuals literally taken from the street (so not forensic scientists, or lawyers, or statisticians). The aim of the experiment was to see if a popular jury could be told about fingerprint statistics and handle the information that they receive in a fair, rational and reasonable way.

The results were impressive:
- on the latent print that was 'identified', all but one jurors had no doubt that the print had been left by the defendant.
- on the latent print that was reported using probabilities, all jurors completely understood that the association with the defendant was strong but that other pieces of evidence were necessary to confirm/infirm the association, in order to reach a conclusion as to whether or not the defendant was truly the source of the print.

Bottom line, the recurrent statement that latent print examiners are some sort of guardians of the temple and that they have the duty/right/power to decide what evidence get presented in court, to protect the court against itself (i.e. from making an error), is not only unacceptable, but is plain wrong:
- it is unacceptable given the foundations of the legal system that separate the roles and duties of the various actors. The only individuals in a courtroom who have the right/duty to make decisions are the judge and the jurors; NOT the witness (whether expert or not);
- it is wrong (and very arrogant) to look down on jurors and assume from the get go that they can't possibly understand and handle rationally information, if properly explained. If one really believe that, one should run for Attorney General of the US and make sure that popular jury are simply abandoned.

I do agree (and that's something that we have stated every single time we lectured or gave workshop) that we, as a community, need to work on how to best express complex concepts in the courtroom, that we need to properly define SOPs, and that training needs to be developed and provided. There is a long road ahead of us.

Secondly, it seems that you are right on one point: we did not give the jury the possibility to hear your way of testifying, in order to study how they would react to your more 'traditional' and non 'melodramatic' way of reporting AND defending latent print evidence in court.

It seems to me that you are volunteering to be on the grill at the next moot court in 2012? (don't worry, we won't have you testify in bikini)

Best,

Cedric

[g.]

Thanks Boyd for your comments and initiating some discussion.

To add to the background for those that couldn't attend: We surveyed the "jurors" after each witness/testimony and then displayed the results at the end for the audience to see. We also allowed the audience to ask questions of the jurors and vice versa at the end. Lastly, we also surveyed the AUDIENCE too using a survey form, and these were completed PRIOR to the audence seeing the juror results.

I have had a chance to review the audience surveys. They are fascinating. About 40% of attendees agreed in part, mostly with, or completely with everything Boyd said. i.e. they think it's anywhere from on the one end, dangerous, unethical, or immoral ranging to simply skeptical that probabilities can be reported (as we did) in our single case example for an "inconclusive"/no value impression that bore 7 corresponding L2D to defendant on the trigger of the gun in a mock case (there was other evidence in the case for the jury to consider as well, as Cedric noted). Again the audience surveys were taken before seeing the jury results, although I doubt they changed view points after seeing jury results, other than maybe some surprise at juror unaninimity.

On the other hand, from the audience surveys, almost 60% of audience members disagreed with Boyd's position. This half felt anywhere from "there's some value here, but i am skeptical and uncertain about this approach and I need more information and training on the topic before I can decide, ranging all the way to "it's about time!" "please! let's get this validated and start using it tomorrow".

So clearly with a near 50/50 split on view points on such a controversial topic, it will require as Boyd said, a LOT of discussion on how to prepare for the use of probabilities, how to appropriately present them, and how to train the analysts that will use these tools how to present in court. It will no doubt be a challenge since still at least half the examiners are rather strongly opposed to using them.

But regardless, to use your phrase Boyd, it is no longer an issue of "if they are to be introduced", but "WHEN they are to be introduced" in the courtroom, I hope the members of the profession have taken the time to adequately learn about this topic to make an informed decision on why they will choose to use them or choose NOT to use them. [Just as we currently have large numbers of examiners who by agency policy or unspoken policy CHOOSE not to use creases, pores, or simultaneous impressions in their decision making. To me it is the same thing, some examiners have had sufficient training, and are comfortable making a "crease only id" or simultaneous id where none stand alone. Not all examiners are willing to use all approaches and tools available to them, but this does not invalidate ridgeology, or even diminsh the validity of those that DO choose to use them].

g.

PS-We (i.e. Cedric, Steve Meagher, Christine Funk, and I) plan to publish these results and maybe some opinion/editorial commentary as soon as possible in JFI or something so people will have the opportunity to see what was done and how we approached it in this "pilot study"/moot court exercise.

PPSS-publicly, I wanted to say "great job to Steve M, Christine F, and Cedric N for their hard work and commitment to this training exercise and exploratory venture. And I cannot even fathom getting up in front of 300 people who know this discipline as well or better than you, to testify to something as complex and controversial as statistics, in A FOREIGN language, as Cedric did (if you don't know Cedric, French is his native tongue). So kudos to you, man. Whether people agreed with the approach or not, I heard many people remark that they nonetheless enjoyed the exercise and found it ranging from highly informative to deeply disturbing.

[Bill Schade]

I just noticed this thread out here in IAI topics. I think this discussion should be moved to the main message board so everyone can participate.

I was present at this presentation and I'm trying to remain open minded.

I agree with Glenn, everyone involved should be commended. Presenting to a group of your peers in that environment was no small feat.

I disagree with Cedric however that "The results were impressive" I would say that the results were "interesting" but I'm not sure they can be related to the real world and a real jury.
This was an entertainment environment, not a real courtroom where jurors are making important decisions on someones fate. It would also be a surprise to me if a lay person stood up in that presentation and said "I don't get it" Much easier to go along, nod your head and tell everyone you understand the "doctor". How many times was that said during the presentation? Perhaps it was ment to lighten the mood, but the pessimist in me says you were trying to bolster what I thought was weak testimony.

I have been under the impression that the "push" to use statistics was from the defense and is intended to weaken fingerprint testimony from the days of saying "100% absolute conclusions to the exclusion of all others". I thought it was an effort to be more realistic in presenting our identifications.

Now it seems we are going to use probabilities to present evidence that would never have made it to the jury before! Do we really want to be telling jurors, "well I can't positively identify the print as the subject, but along with other information developed, it is probably his print. This is progress? Do we really think that the defense will sit still for this and not protest.


I have to wonder if this change of direction was not an attempt to make your concept more palatable to the audience of practitioners.


I'm not sold yet, but I will keep an open mind

[g.]

Hi Bill,

I should probably clarify some behind the scenes stuff so that there is no misunderstanding.

The choice to use the model on the inconclusive was my decision. There are easily about 10 different scenarios one could envision using the model under. For example, one could run ALL decisions/marks through a model, one could run only IDs through the model (as you had envisioned), one could report stats only (like DNA), one could get numbers but use a qualified verbal scale. In fact in the literature many different ways are discussed for using a model. I chose one that I thought was a realistic scenario for use and one that myself and other examiners have been frustrated by in our careers: the inability to adequately express an inconclusive.

I recognize, especially for the older generation of examiners, this approach/use of a model may be hard to stomach. Anyone raised in the environment where inconclusive decisions aren't used except to mean "I need better knowns" (i.e. it's him, it's not, it's no value, or I need better knowns) will have a particularly difficult time using the model in this fashion.

But this was one of many ways to use the model, and it was a choice I made given the time limits, Steve's scenario (he picked the case details) and to show one way it could be used. We could just have easily picked a scenario where a Daubert hearing failed, rejecting the fingerprint evidence unless the examiner came with stats to support the identification.

The point is, as a community, we need to discuss how best to use such a tool. And if the community feels this is inappropriate to open the inconclusive box in this manner, so be it.

But 2 points,: 1)from the audience surveys some, like you, were VERY uncomfortable with the use for that scenario and equallly as many in fact loved the idea. Second point, there were 3 defense attys present. Two in the survey said "this is great", one shared your concern.

It is clear ALL the stakeholders need to sit down and discuss how to implement such an approach once it has been validated.

Thanks Bill for your insights.

g.

[Michele]

Shouldn't any model be validated and find out how accurate it is under different situations before we decide when it should be used?

[Boyd Baumgartner]

So, if I'm reading this thread correctly, it all boils down to this:

(click once on the image to see it full size)

DinosaurExaminer.jpg

My issues here have always been primarily twofold: Assumptions and Philosophy.

Obviously, it's not like I'm saying I disagree with you because Cedric forgot to carry the one in his calculations of the denominator therefore the stats are all wrong. I'm attempting to understand the assumptions here. For instance, at the IAI and here on the thread, you keep bringing up the phrase "like DNA". This brings up a good issue, namely population statistics. At the heart of what DNA does lies Hardy-Weinberg and Linkage Equilibrium, which is the principle that allelle frequencies can be predicted. What analog do we have for minutiae? Does your model assume minutaie frequencies and locations are predictable and tied to populations? There's also the meat and potatoes issue of the fact that value is not a sole derivative of frequency, but that's another thread in it's own right.

DNA also brings up the issue of subgroups(subpopulations), how does your model deal with this? Was it developed using some database of white people from europe who have the genetic variability of Rebel Clones in Episode II of Star Wars, or does it apply equally to African and Asian subgroups? What about a mixed race society that is on the rise in America?

There's also the issue of the 'close non-match', can you explain how your research makes use of this phenomenon and if so, how exactly do you define it. I think this topic itself would raise a few eyebrows in the community. This topic flows nicely into the idea of how your model incorporates the likelihood of misinterpretation of features (error) and how it incorporates incompatible features between examiners.

Why I believe this week's detail dealt with just such an issue: http://www.psychologicalscience.org/ind ... t-csi.html

Ultimately, all of this rolls up into the philosophical problem of Verissimilitude, the notion of truthlikeness which is at the heart of the matter. Another issue raised by Karl Popper himself, whom the profession loves to dote over. The classic example here is two clocks, one stuck at 6:00 and the other off by four hours. The question is, at any given time, which clock is closer to the true time? The problem is that they're both wrong, one is just less wrong. How much less wrong is the likelihood ratio than ACE-V?

How does Karl Popper, broken clocks and 50 cent words relate to this topic you ask? Because it turns out we've been knee deep in statistics and truth-likeness since 1892.

Stats.jpg

I've got a shiny nickel for anyone who can dig up the literature citing the authors of these studies that say "My model will be replaced by a more truthlike model in X years!" Christophe, why aren't people testifying to your 1995-1996 model? Is it not good enough now, or was it not good enough then?

The same holds true with the likelihood ratio in my estimation.

[ER]

Michele brings up a good point. How we can use this tool when it hasn't been proven or completely tested yet?

On the other hand, no one is using these statistical models yet. They are still being tested. Why are we arguing over the minutiae of something that is still being developed? (no pun intended; well maybe a little intended)

Boyd: While I share many of your reservations, I am also interested in seeing what they can come up with. All of your questions (and a whole lot more) will have to be answered before the statistical tool can be used in our discipline. But they are trying to answer these questions. Exploring the limits and the possibilities of statistical models of fingerprint identifications has potential to be extremely useful. I'm hopeful that it eventually passes every rigorous test that you can think of. Think of it. If a statistical model could actually answer each of your questions, wouldn't that be a powerful tool? I have my doubts of whether that will ever happen, but I'm hopeful that Cristophe, Glenn, Cedric, or someone can make it work. And yes, DNA is very different. But look at what was actually said. Glenn didn't say that fingerprint stats were like DNA stats. He said that we could choose to only present stats (without giving a definitive answer) like DNA chooses to present only stats (without giving a definitive answer).
In other words, stop saying, "You can't do that." Let them prove it.

Cedric: No matter how powerful or accurate your model eventually becomes, it will still have to be presented and accepted by this community if it is to ever achieve widespread use. If the statistical model is presented in an arrogant and academic way (ie. "Il Dottore"), you will have a tough time convincing many LPE's that this is the way to go. The model has great potential, but still has a LONG way to go. Admit that, make the model better, and show us how robust it can be. Answer the questions, address the concerns, don't dismiss them. Don't look down on those that question you. Most examiners will accept statistics as a part of fingerprint comparisons when they are shown that it works well. No one will believe you just because of your title. It is wrong (and very arrogant) to look down on the latent print community and assume that they will accept this major change without question. The questions that have been raised have some validity and deserve answers. I hope that your detractors are eventually proven wrong, but I also hope that your detractors will eventually be won over to your side. Mocking them does not help you.
In other words, stop saying, "Trust us. We can do it." And prove it.

Glenn: With regards to which approach we should use to present statistics in court. I foresee that the two most likely scenarios for fingerprint statistics in court are (1) "I didn't ID, and I didn't exclude. My result is inconclusive. However, there are some things that line up that the jury should know about." and (2) "I ID'd this print to Joe Schmoe, and I have these statistics to back me up."

[Boyd Baumgartner]

Good points ER. But to clarify I'm not saying don't do research, I'm saying don't do the application yet. There's too many holes.

For instance, the first thing I do if I'm a smart defense attorney is Melendez Diaz everybody who touches PiANoS as part of the case and force them to testify about stats considering they contributed to the likelihood ratio. Depending on how many people have plotted minutiae and their experience level/knowledge about stats in general and this method specifically it's not hard to imagine a jury losing confidence in the experts and returning an acquittal.

[Cedric]

Michele, Isn't it what our IAI committee is supposed to do ;o) ? As often discussed, I completely agree that the models need to be validated. However, nothing should prevent the community to discuss other related issues while we are doing it, in order for all of the pieces of the puzzle to be ready when the time will come.

ER, The 'Il dottore' aimed at Steve was a joke with respect to the exchange we had at the moot court, where he was adamant to call me 'Mr'. This has nothing to do with 'academic' arrogance and if you have seen Steve and me interacting, you will have noticed that we tease each other a lot but that we have the fondest respect for each other. That said, I would expect to be addressed by my title in Court, in the same way as a latent print examiner would expect to be respected for his/her experience and achievements. I believe that the countless hours of my involvement in the community for many years (IAI, SWGFAST, workshops, meetings, lectures, etc.) show that I am clearly contributing to the discussion and that I am definitely not assuming that it's a done and dusted deal. Yes, there is a LONG way to go and I (or Glenn or Christophe) have never claimed otherwise. Maybe the fact that I agree with 99% of the comments made by the community, and that we are working hard to address them, is lost because of my accent ;o)

Your assessment of the two possibilities for using statistics in court are exactly what we are considering at the moment, and your (2) is how we have already presented it in court.

A question that I have always asked myself for a long time is why does it seem that the requirements in terms of validation, robustness, acceptance, presentation, etc., that we are discussing should be significantly more stringent for "fingerprint statistical models" than for any other technology (or technique) used in forensic science?

For example, when AFIS systems were introduced 40 years ago, they were far from being robust and reliable. Nonetheless, they were useful and got implemented. Even today, they are not 100% accurate. This technology is here to support latent print examiners, who have perfectly the power to decide that the individuals in the candidate lists are not the correct ones. In addition, different AFIS models will have different scores, will present different candidate lists for the same latent print, or are replaced by better and more accurate ones every decade (Boyd's point). In our view, this is the same for statistical models (or any other technology).

Another example is the current examination process (some call it ACE-V), which has never been properly validated (to the level required for the stat models), which is also not very robust (see the error rates and variability study reported by Austin Hicklin and the FBI). Nonetheless, examining latent prints is very useful and it has a significant contribution to the criminal justice system, and therefore it is used even if it is not perfect, and even if validation could be better.

Finally, an example with respect to DNA: there are various models to interpret and report DNA statistics, even within the US. It is completely possible that my blood would be reported in different US courts with different statistics, depending on which loci has been considered, which sub-population has been used, which statistical model has been used, and which SOPs are being followed. I am NOT saying that this is a good thing, since it could be very confusing. Nonetheless, as imperfect as it is, nobody can claim that DNA evidence should not be used, or reported.

What I am trying to say is not that we should lower our standards, or that we should not validate models,that we should not be careful about what we are doing, or that the community should not be leading this project (in fact, nothing would please me more than the community taking a very active lead on the subject). But as a community, we should be careful to not shoot ourselves in the foot by setting up the bar so high that some clever critic would use that against us and ask that this quality standards should be applied retroactively to all other aspects of the examination of latent prints.

[Bill Schade]

Your assessment of the two possibilities for using statistics in court are exactly what we are considering at the moment, and your (2) is how we have already presented it in court.



.

Are you saying that you have used this in a "real" trial already? Did the court require statistical backing of the conclusion or was it just a test flight of the concept?

How did it go?

[ER]

Sounds good. Looks like everyone's on the same page.

The researchers are going to keep researching. The critics are going to keep criticizing. And eventually a statistical model for fingerprint comparisons will evolve that will be useful to some degree. The argument over how useful it will be can be shelved until more work has been done and more tests have been performed.

[Cedric]

Bill,

I have only been involved in part of it. Glenn knows more about the case and what happened to it. I believe the Judge has still to give his opinion.

It was a Frye-Mack hearing in MN. It was interesting. The initial hearing, in which I was involved (I think 2 more took place), took place over 2 days in May. Glenn testified during the first half day, I did during the second half day, the examiner in the case testified during the 3rd half day and Dr Simon Cole testified during the last half day. Dr Sandy Zabell testified on behalf of the defense during a subsequent hearing.

Glenn and I mostly presented data and results of research projects showing that (among other points):
1) features of latent prints (general pattern and L2) are extremely discriminative and provide very strong evidence
2) there is no scientific basis for a "minimum number of points" and therefore, the current approach stands
3) most ID decisions are reached with a quantity of features waaaayyyyyyy above what we ever processed with the model (i.e. the statistics reported by the model on the test data were very high, so in the real world, it would have been so high that we wouldn't bother to compute it).
4) the error rates measured by various researchers (Glenn in two papers, Hicklin and FBI, etc.) is low

Our point was not that we had breath taking and revolutionary results, but it was to mostly to present the data that support what the latent print community has known for decades but without being able to demonstrate it. Point 1 and 2 were supported by the data from the fingerprint statistical research that will be published in January 2012 in the Journal of the Royal Statistical Society and that was used in Milwaukee. Point 2 was supported by the field study that Glenn and I did at the BCA and published in Forensic Science International. Finally, point 3 was supported by Glenn's PhD data (and Hicklin's data).

Simon Cole was himself: he explained that it was not scientific to claim "individualization to the exclusion of all others" and that based on the NAS report, it appears that the examination of latent prints is not "generally accepted by the relevant scientific community" (the community being the NAS community).

Sandy Zabell didn't say much, apart from that more research like the ones we presented is needed.

Now for the examiner in the case... well, that's another story. And unfortunately an unexpected and fairly concerning one. Bottom line, let say that I experienced first hand why SWGFAST, the IAI and other leaders in the community are fighting for better training, for continuing education, for competency and proficiency testing, for more stringent standards, etc.

In essence, we used the results of the research in the way ER described in his (2) and that part went well. For the rest, the Judge has still to give his opinion.

Best

Cedric