CLPEX Chat Board

Skflynn posted in the NAS thread but I am responding in a separate thread because I think his post will be lost over there and I think it has the merit for a good discussion.
(Two main topics - The foundation for our conclusions / reliability of the method v the practitioner).

Topic 1:
The idea that AFIS has never found two prints to be the same may support that prints are unique but it doesn’t support how much is needed to establish that 2 prints came from the same source. It’s because of this that we can’t use AFIS to be the solid foundation for latent print conclusions.

Topic 2:
Many people state that properly trained examiners applying ACE-V can produce reliable results and erroneous results are a reflection on the examiner and not a reflection of the method. This statement has many issues.

Issue 1 - What is a properly trained examiner?
Let me play the devils advocate on this one. So many things I was trained in are now being thrown out the window. Does that mean I’m not properly trained? Many people have had no training in ACE-V, scientific ideas, or current appropriate philosophies. Where is this kind of training available? Most training classes only give a certificate of attendance but don’t have an actual test you must pass. How do we know if someone is properly trained?

Issue 2 - How should ACE-V be applied?
There are many explanations of ACE-V and some of them contradict each other. Which one is recommended and by whom is it recommended (the IAI, SWGFAST, your own agency)? Which application is the one that produces the most reliable results?

Issue 3 - How do we know an error is due to the examiner or the method without taking a good hard look at the method the examiner has used? Do we ask examiners to describe the method they use or do we just assume we’re all using ACE-V in the same way?
Errors haven’t been studied enough to know the cause. The common approach is just to blame the examiner and move on. This approach doesn’t tell us why the error happened and without knowing why it happened we can’t prevent it from happening again. Is it possible that errors could be due to improper training and inadequate procedures? If so, then shouldn’t an agency be blamed instead of the examiner?


Many examiners have had great training and only see other examiners who are comparable to themselves. But this doesn’t mean that everyone is in that situation. There are just as many other examiners who are at the other end of the spectrum. They are simply hired to do a job and they have to do it with little to no training, little to no input from their agencies, and probably no information about what’s going on in the community outside their office.

Yesterday I spoke with a CLPE who wasn’t aware of the IAI’s memo that was released. Isn’t it a shame when you’re handed something by the defense in court that you’re unaware of? It surely doesn’t show that you’re an expert when the defense knows more about your profession than you do.

I can’t tell you how many agencies aren’t allowing someone from their office to go to the April 3 and 4th conference that will talk about the NAS report (at the Arizona University Law School) because management doesn’t think it’s pertinent.

What I’m saying is that you’re fortunate if you have had good training and work in an environment that is supportive but I don’t believe that the majority of us feel like we’d be included in that group. It is probably this group that is looking at the NAS report as a positive step forward.

The "R" word.

The word "reliable" comes up very often, and I think it frustrates and diverts attention from realities. It's not correct, reasonable, or equitable in criminal law to hold that any one conclusion of a particular analysis is unreservedly reliable in the sense that it cannot be challenged. What you can ask is that a process and conclusion can be explained with reference to what is generally accepted scientific knowledge and the experience of the analyst and the discipline as a whole. There is simply no process, no method, no experience, and no training that can justify as inherently "reliable" a conclusion that comes out of that combination. The "reliability" that is of value is that something produces evidence worthy of consideration by a trier of fact. It is reliable in the sense that a trier may appropriately rely on it but may not. Some things do not meet that standard. They are not admitted as evidence.

To be reliable in that sense, the elements of the knowledge, observation, analysis, and conclusion must each be expressed appropriately according to their certainty. Some of our knowledge is generally accepted scientific knowledge. Some is empirical knowledge (AFIS is contributing to that). Some is articulated analysis (ACE-V), with reasons for differential decisions (experience). What matters is that it can be presented and justified - and that means it can be argued. Remember one of the most important words in Daubert: Falsifiable. Nothing is acceptable unless it can be demonstrated to be false, meaning it can be argued to be false by someone relying on the same elements that went toward the conclusion being disputed. "Reliability" cannot be applied to our analysis to mean infallibly true and is therefore not really an appropriate terms in the venue of criminal evidence of our sort. You cannot declare some evidence to be "reliable" without hijacking the responsibility of the trier of fact. The jury decides just what is reliable, what they take to be sufficiently accurate to be part of their weighty decision.

Proper training is that which provides the analyst with the current scientific knowledge, processes that militate against improper internal and external biases, and the basis for forming a reasonably credible conclusion. Discussion of ACE-V is not a matter of one version being right and another being wrong. What matters is that the order of operations, no matter how formed, is understood in terms of value and hazards. The thrust of efforts urged in the NAS Report is not to determine reliability. It is to determine and to continually reevaluate HOW latent print evidence has value and what statements are appropriate products of analysis.

Do not wish for true "reliability." Reliable, to a very high certainty, of latent print conclusions can indeed be established as a generally accepted matter. But if that's the test you want, you will have to be satisfied with losing a lot of the value of latent print examination.

I think you’re taking something that can be discussed easily and making it more difficult by focusing on semantics. Would you prefer if I talked about whether or not you can ‘trust’ the method? If it can be trusted, then to what degree? Surely, some applications of the method produce better results than other applications, or is the method infallible?

Remember one of the most important words in Daubert: Falsifiable. Nothing is acceptable unless it can be demonstrated to be false, meaning it can be argued to be false by someone relying on the same elements that went toward the conclusion being disputed.

I don’t remember the word Falsifiable being in Daubert but regardless, I’ve never heard it articulated in this manner. I'm going to refrain from commenting further on the use of this word unless it ends up in a thread of its own.

Hi Michelle,

Falsification is a principle first expounded by the 20th century philosopher of science, Karl Popper. As Popper pointed out, although no natural theory can be proven true, if it is a good theory, it can be falsified. That is, it can be proven to be false.

For instance, consider our own theory that all friction skin is unique. Though we can never prove that any two people have had the same qualitative and quantitative friction ridge arrangements, all it would take to disprove or falsify this theory is one instance where two people were found to have the same qualitative/quantitative friction skin arrangements. At least as a theoretical statement the theory of uniquenss is by this standard, a good scientific theory.

The quote from justice Blackmun is this,

Ordinarily, a key question to be answered in determining whether a theory or technique is scientific knowledge that will assist the trier of fact will be whether it can be (and has been) tested. “Scientific methodology today is based on generating hypotheses and testing them to see if they can be falsified; indeed, this methodology is what distinguishes science from other fields of human inquiry.”

A more direct quote of Popper by Blackmun was,

K. Popper, Conjectures and Refutations: The Growth of Scientific Knowledge 37 (5th ed. 1989) (“[T]he criterion of the scientific status of a theory is its falsifiability, or refutability, or testability”).

Roy Marzioli, Manager
Central Identification Services
Contra Costa County Sheriff's Office
Martinez, Ca.

Don't forget that according to Popper in "Conjectures and Refutations" that falsification needs to be scrutinized and distinguished as a failure of the hypothesis versus a failure in the presentation of the evidence.

It seems that the foundation for reliability/falsification/justifications/etc. boils down to documentation.

Topic 1: AFIS is used for latent prints and there are many many searches done where small pieces of latents are searched and in general examiners are not finding duplication. AFIS can be and is used to support the theories behind latent print identification.

If I search a latent and I find numerous close -but- non matching areas of friction ridge skin that is going to help to adjust any kind of internal standard for identification. Through AFIS we are able to find close non-matches and that does help examiner determine what is not enough information needed in order to identify someone.

Topic 2: You have a point with this one and I think that it is more of a semantic question. The real question should be are "trained" examiner reliable and accurate. This is (or should be) tested thoughout training and with proficiency tests and throught the verification stage of ACE-V. The question the court should be concerned with is how accurate is latent print identification and is this examiner qualified to be doing it.
Even with highly publizized errors the actual error rate for latent prints is very low. You can make the arguments about ground truth and all that but in the practical world how else can you even hope to judge error rate.

To seperate the method from the examiner is all based on how you define each. If the method is so vauge that it places all onus on the examiner then it can never really be wrong. The truth is that the method and the examiner can not be seperated in most cases and is there really any value in trying.

Saying that the error rate for the method is zero really does not tell me anything and I dont think it adds any value to an argument. The error rate is very low for examiner and method combined and while that might be nice to know it still would never be able to answer the question: What is the chance that you made an error in this case?

The good thing about latent prints is that the evidence is not consumed in analysis and should always be available for review.
What people like to forget is that most errors were uncovered by other latent print examiners.

-And to all you Poper fans while he may have some ideas on how science should be preformed he is not the be and end all of scientific thought or design.

Even with highly publizized errors the actual error rate for latent prints is very low.

How does anyone know this, what is it based on, and what does "low" mean?

I used to ask people to raise their hands in my workshops if they ever caught misidentifications in their agency, and there were always a few hands that would be raised. There simply isn't a way to quantify how many misidentifications there are, that are not caught by someone. I don't think anyone can give a reliable estimate as to how large the error rate is.

The good thing about latent prints is that the evidence is not consumed in analysis and should always be available for review.

Good point.

It's beyond me why public defenders offices don't have some staff trained in latent print examination to check all print cases. I know budgets for PD offices are always tight, but it seems to me that sending a few paralegals out for training so they can, at least, review all cases would make sense.

What people like to forget is that most errors were uncovered by other latent print examiners.

I don't know who likes to forget this, or who would claim this isn't true. Who else would uncover the errors?

George

I second George's suggestion - The Public Defender's office should employ a latent print examiner to review cases with latent print work. Since a significant portion of my examination yields no suitable latents, I don't think the caseload would be that high in an average county. Maybe a large city could have a couple. You still would not get a great idea of the error rate (because you could probably only account for bad ID's, not missed ID's), but it would be a definite improvement over the current system of "rather than review the evidence, we'll just attack whatever we can make seem like a weak point."

The Public Defender's office should employ a latent print examiner to review cases with latent print work.

Remember, tho., that latent prints are only one part of the forensics that come up in a criminal case -- what you are suggesting is that the PD's office build a parrallel lab with investigators trained to at least a minimal standard in fingerprints, DNA, firearms ID, bitemarks, toolmarks, crime scene reconstruction, drug analysis, autopsies, and probably another half-dozen areas I've missed. That's a lot of investment of time and money.

And that only works to the extend that a case goes through a public defender. It doesn't do much for the middle-class person who has just enough money not to qualify for the public defender, but not enough to pay for an OJ Simpson level re-investigation. It also doesn't work in places that rely mostly on private attorneys working as special public defenders without much in the way of institutional staff support or resources.

I'm not saying that developing some in-house expertise and a nose for when there's something odd about a case isn't a bad idea, but that particularly with the hits that public defender (and all criminal justice) budgets are taking right now, it would be hard to institute.

George's & Amy's suggestion is a good one, but as Lisa points out the expense of funding a lab may not be justified due to its limited use. But I think the NAS has already made a recommendation that has merit and would satisfy George, Amy, and Lisa -- a separate "Department of Forensic Science Services" outside the umbrella of law enforcement. All parties would have equal access. The police, the prosecutor, or the defense could submit evidence or have it reviewed.

Of course, there are problems to work out. First, the crime lab's initial work is part of the investigation of a crime, not part of the prosecution at that point. Shouldn't an investigative function be part of the police department? Second, if one examiner in this independent DFSS makes an identification, would it be practical for a defense attorney to resubmit the same evidence for a second examination?

Lisa, can you suggest how this DFSS might be made to work as both a police support service and a defense review resource?

"I don't know who likes to forget this, or who would claim this isn't true. Who else would uncover the errors?"

Mainly the critics. You cant say that ACE-V method doesnt work when it is the method used to uncover the errors.

Even with highly publizized errors the actual error rate for latent prints is very low.

How does anyone know this, what is it based on, and what does "low" mean?

Published studies on error and CTS data do reflect a rather low error rate. In the most recent issue of the JFI Glen Langenburg has a great article where he examines this issue in his own lab. And while this might not apply across the board it does support the idea that for the most part we know what we are doing and we do it well.

The following email showed up in my mailbox this morning:

SPECIAL REPORT - THE NATIONAL ACADEMY OF SCIENCES REPORT
March 4, 2009 by Crime Lab Report

http://www.crimelabreport.com/library/m ... pecial.htm

We appreciate our readers. Thanks for your interest and support.

Crime Lab Report Editors
editors@crimelabreport.com

Click on the link. The report is not too long but is worth a read.

My favorite quote from this report is the following paragraph:

"Ballistics and fingerprint studies performed by law professors and economists will be the first clue that something has gone terribly wrong."

Regarding an independent lab for prosecutors and defenders to use. I've thought that this would be good also - but I wonder if it's one of those things that sounds great but doesn't turn out like anyone expected.

I don't have the answer, and I know that public defenders can't afford to hire analysts to review all forensic evidence. But, I do think that they can, at least, afford to train some current personnel in fingerprint comparison, video analysis, footwear and tool mark comparison, and perhaps a few other disciplines so that can, at least, do some review of cases involving this evidence.

Regarding the low error rates. I look forward to reading Glen's article. I hope he addresses the fact that in the US there are 18,000+ independent, autonomous agencies. Some of them may have a patrol officer doing fingerprint work with no one doing verification. Many cases may get verified and still be misidentifications. Of those, many may never make it to court - misidentified victim prints, for example; or misidentified prints to the suspect, who coincidentally is guilty and pleads, etc. I am NOT stating that the error rate isn't low - I'm just stating that I don't think anyone knows what it is.

George

I received the JFI the other evening and started reading the article by 'g' and it consumed about 1.5 hours of my time (Not being a statistician I had to read it slowly).

I found the article very interesting and brought up several good points and issues.

In the previous post by G. Reis on this thread, Mr. Reis talks about the other LPE in the country. I think 'g' takes on part of that issue in his concept of "Precision and Accuracy". In my POV I find that his concept of "Precision and Accuracy" in the latent print community to be very good.

I would like to see what others might think of the concept of "Precison and Accuracy" as offered by 'g' or their own concept of those two terms?

Pat A. Wertheim wrote:George's & Amy's suggestion is a good one, but as Lisa points out the expense of funding a lab may not be justified due to its limited use. But I think the NAS has already made a recommendation that has merit and would satisfy George, Amy, and Lisa -- a separate "Department of Forensic Science Services" outside the umbrella of law enforcement. All parties would have equal access. The police, the prosecutor, or the defense could submit evidence or have it reviewed.

Of course, there are problems to work out. First, the crime lab's initial work is part of the investigation of a crime, not part of the prosecution at that point. Shouldn't an investigative function be part of the police department? Second, if one examiner in this independent DFSS makes an identification, would it be practical for a defense attorney to resubmit the same evidence for a second examination?

Lisa, can you suggest how this DFSS might be made to work as both a police support service and a defense review resource?

To go back to the defense-expertise problem again, just for a sec, I've also got some concerns about confirmation bias. There were some news reports that two defense experts signed off on Cowans prior to his trial -- tho they've never been named. And the court-appointed expert missed the Mayfield mis-ID. Let's face it -- the defense reviewer is going to likely see a lot of routine IDs with no obvious problems and may be lulled into confirmation bias, even if they approach the case with a skeptical intent.

For the joint lab, the biggest problem I see is confidentiality. The prosecution has an ethical obligation to disclose any exculpatory evidence to the defense. But a combination of the protection against self-incrimination and attorney-client confidentiality mean that defense experts are bound by the hiring attorney's confidentiality obligations -- depending on the specifics of the case, they may not be able to admit that they were even consulted, never mind what their results were.

Would that cause tensions in a lab if Examiner A not only can't tell any of his/her colleages what case he or she is reviewing, but not what result was obtained, especially if the result is inculpatory. I also wonder about conflicts of interest and how that applies to experts. If I take a case, I have a duty of loyalty to that client -- there are strict rules that may disqualify me if I ever represented a co-defendant or key prosecution witness. I haven't looked into if or how that affects experts -- if Examiner A has privileged information about a case from working for the defense, and is later involved in an investigation in which that information would be important either to inculpate the defendant or exculpate another suspect, will there be restrictions on how he/she can use the information.

I haven't looked into this area -- have these problems already come up? Are there already procedures in place for conflicts checking or review?