NAS Report & ACE-V

[David Johnson]

It was surprising that the National Academy of SCIENCE would publish a review of forensic science with so little science in it. There have been studies on error rates, and there are scientific studies on the development of expertise in other fields. Many cases where experts have to rely on (heaven forbid) training and experiance in order to come to conclusions.

One of the citations for the ACE method came from the Habers and while I welcome their criticisms I do not think that they should be considered an athoratative source for the ACE-V methodology.

Limited poorly done studies on bias weighed heavily when most other expert reasearch has shown that some of this Outside information can lead to more accurate restults and studies done by Glen have shown that bias makes examiners more cautious.

I welcome any research that would help move the science of fingerprinting along but I think the first task that is needed is to find out what the problems are. A comprehensive case review of latent labs across the country would go a long way in an effort to understand probably the most important question. What is our error rate?

If it is not that bad say less that .2% as some testing sugests then we need to ask ourselves is this a training/proficency issue with these limited cases or is really a problem with the science?

[izatso]

Hi David,

It was surprising that the National Academy of SCIENCE would publish a review of forensic science with so little science in it. There have been studies on error rates, and there are scientific studies on the development of expertise in other fields. Many cases where experts have to rely on (heaven forbid) training and experiance in order to come to conclusions.

I think the criticisms of the report are valid. The lack of science that you allude to above is our own lack of scientific studies, not the National Academy of Science's. We have known for quite some time that this research needed to be done, but we haven't done it. Actually some research has been and is going on currently, but we have much more to do. The NAS is not going to do it for us. We are going to have to do it ourselves or we are going have to partner with others who may have expertice, say in statistical analysis, that we may not have.

This report is only telling us what we have been hearing ourselves from out detractors. I think we have been lulled into a false sense of security based upon the fact that we have been able to withstand quite a number of Daubert challenges into thinking we didn't need to do anything. Even though we have prevailed in these challenges, there has been an underlying theme which taken hold in our own community which is that we need to do the research to support our own science. I'd say it time to get busy.

[Outsider]

The NAS report does not seem to say anything about the important subject of how forensic evidence should be evaluated and used. While I can see that this is no business of the people who provide the evidence in a specific case, I think everyone would want to make sure that evidence is used appropriately.

We talk about the criminal justice “system”. In a system every part is interdependent on every other part. The police commission forensic work and direct it to locations they think will yield useful results. The police then steer the course of inquiries as a result of forensic reports. Law officials decide if there is enough evidence, and it is in the public interest, to initiate a prosecution. These are processes of selection and they will alter the likelihood that a forensic report used in a particular case is erroneous. They will also affect the error rate of a type of evidence when it is used in court.

An erroneous forensic result is likely to suggest suspicious behaviour and be disputed, making it stand out from other evidence for special attention. Modern probability theory, and long-established legal principles, suggest that evidence carries more risk when there is little other evidence against a suspect. Some time ago I posted an imaginary exercise on this forum which I hope shows how this can work (link below).

I am not suggesting that police officers and lawyers should have to endure a course in statistics but some training in the relevant logic could provide an awareness of specific dangers (I don’t know if anything like this is currently taught). I think the difference between the general or average likelihood of error for a forensic technique, and the likelihood that a specific case contains an error, can be much greater than some people imagine.

I suppose the question boils down to this. How much of a case is it safe to build on a single item of disputed evidence? Is it OK for the police and prosecutors to proceed with a confident belief that an identified person is guilty if there is nothing else to link that person to a crime? (As appears to have happened in the McKie and Mayfield cases).

Fingerprinting Innocentville:
http://www.stevehornsc.pwp.blueyonder.c ... tville.htm

[L.J.Steele]

An erroneous forensic result is likely to suggest suspicious behaviour and be disputed, making it stand out from other evidence for special attention. Modern probability theory, and long-established legal principles, suggest that evidence carries more risk when there is little other evidence against a suspect. Some time ago I posted an imaginary exercise on this forum which I hope shows how this can work.

It is somewhat worse that just suggesting suspicious behaviour which the investigator or prosecutor would weigh as _a_ factor. The folks dealing with false confession issues mention that once someone has come to a conclusion that the suspect is lying/concealing something/guilty, they will read ambiguous information in the worst light AND in their interactions with the suspect, the suspect will tend to react to their body language by behaving in a more stereotypically guilty manner.

Check out Saul Kassin's papers on this in the interrogation context:
http://www.williams.edu/Psychology/Facu ... y_2003.pdf
http://www.williams.edu/Psychology/Facu ... n_2002.pdf

This effect is also why I get concerned when anyone working on the forensics gets any potentially baising information about the investigation before reaching AND documenting their results.

I am not suggesting that police officers and lawyers should have to endure a course in statistics but some training in the relevant logic could provide an awareness of specific dangers (I don’t know if anything like this is currently taught).

George Mason Law school requires a class in economics for JDs which may include statistics, but they may be unique.

I suppose the question boils down to this. How much of a case is it safe to build on a single item of disputed evidence? Is it OK for the police and prosecutors to proceed with a confident belief that an identified person is guilty if there is nothing else to link that person to a crime? (As appears to have happened in the McKie and Mayfield cases).

IMHO, no. I'd like to see at least two independent pieces of reliable evidence for a prosecution. OTOH, you folks all know my biases in this area. And to answer the inevitable question, were I a crime victim, I'd feel very uncomfortable knowing the suspect was being prosecuted based on a single disputed bit of evidence. The decision is, of course, the prosecutor's, but it would make me very uncomfortable that they haven't yet caught the true culprit.

[Gerald Clough]

IMHO, no. I'd like to see at least two independent pieces of reliable evidence for a prosecution. OTOH, you folks all know my biases in this area. And to answer the inevitable question, were I a crime victim, I'd feel very uncomfortable knowing the suspect was being prosecuted based on a single disputed bit of evidence. The decision is, of course, the prosecutor's, but it would make me very uncomfortable that they haven't yet caught the true culprit.

It kind of depends on what the single piece of evidence is. Charles may remember my burglary case he helped with. House burglary. Door forced but not left standing open or otherwise inviting others to enter. Owner's wristwatch that had been in a bowl on the dresser where he left it when he wasn't wearing it. Found dropped on the floor in the middle of the room. Beautiful fat thumb print across the back of the watch case. Hits on a laborer type who had once been arrested as a drunk riding with a DWI drunk. Not the slightest connection to the house, except that on the day of the burglary, there had been some framing projects going on down the county road. Single piece of evidence - and plenty to convict with the highest confidence. (And also enough to cut a rare warrant and indictment identifying the individual by fingerprint and photograph, since we had no idea if the name given at booking for the drunk charge was correct and did not want some poor schnook arrested just because this idiot used his name.)

It's not often just the lack of a second or third piece. It's more often the single piece or all the evidence being represented as being more meaningful than it is. "Consistent with" is, I think, one of the greater evils. Means nothing. But it is sometimes touted as very meaningful by one side or the other, when the actual testimony amounted to little more than it looked like it might be the same material as the proposed source, and they both probably originated on Planet Earth. And the idea of bolstering the meaning of iffy evidence by another and better piece of evidence is patently bogus. One or two of the federal circuits got the notion that expert testimony on the general nature of eyewitness identification should only be required if there was no corroborating evidence. That's absolutely the wrong view of evidence weighting. Each piece is what it is and means only as much as it means. Surrounding it by more discriminating or more reliable evidence doesn't make it more reliable or meaningful.

[L.J.Steele]

It kind of depends on what the single piece of evidence is.

As I said, it would make me uncomfortable as the victim or a juror to go on just one piece of evidence even if it is a very good print in an inculpatory spot which makes innocent access pretty unlikely. I'm not saying it couldn't be proof beyond a reasonable doubt, but it would make me nervous that the right bad guy had been caught.

Out of curiosity, did the guy plea, or did it go to trial and, if so, what was his defense?

Certainly agreed about the significance of evidence being misinterpreted by attorneys, jurors, judges, appellate courts, etc. I think part of the problem is that same confirmation bias issue -- each side is interpreting the ambiguity of "consistent with" in a way that fits their beliefs about the case.

One or two of the federal circuits got the notion that expert testimony on the general nature of eyewitness identification should only be required if there was no corroborating evidence. That's absolutely the wrong view of evidence weighting. Each piece is what it is and means only as much as it means. Surrounding it by more discriminating or more reliable evidence doesn't make it more reliable or meaningful.

<sigh> I'm in the middle of fighting with CT about expert ID testimony at present. One of the sub-issues here is harmless error. FREX: Let's take your case above and assume that there was also a witness who saw the culprit coming out of the house and in a badly mishandled ID procedure ID'd the defendant. Defense offers ID expert to show that procedure was badly mishanded and impeach witness' ID. Judge denies expert, says "that's all common sense, counsel." Appellate court avoids ruling on expert admissibility by focusing on whether defendant was harmed by lack of expert and says that the print evidence is so strong that the lack of the expert could not have affected the jury verdict. In the harmless error analysis, strength of the other evidence is important -- and I think that's what those circuits are looking at. IMHO, the limited number of such experts and ever smaller defense budgets will tend to keep experts out of inappropriate cases.

[Michele]

Lisa,

Would you still feel uncomfortable going with one piece of evidence if the evidence was a DNA match (since it's considered to be more objective and more scientific)?

[L.J.Steele]

Would you still feel uncomfortable going with one piece of evidence if the evidence was a DNA match (since it's considered to be more objective and more scientific)?

Yup. It is one single piece of evidence and that inherently makes me nervous about reliability for a conviction. Even DNA can be contaminated, fabricated/fradulently placed, mis-interpreted, etc. -- and that may mean the actual culprit is still out there.

[Gerald Clough]

It kind of depends on what the single piece of evidence is.

As I said, it would make me uncomfortable as the victim or a juror to go on just one piece of evidence even if it is a very good print in an inculpatory spot which makes innocent access pretty unlikely. I'm not saying it couldn't be proof beyond a reasonable doubt, but it would make me nervous that the right bad guy had been caught.

Out of curiosity, did the guy plea, or did it go to trial and, if so, what was his defense?

Oh. He pleaded. I didn't mention that the first time he was stopped, they had what was pretty clearly a burglary tool kit, and the second time, it was actually for another burglary. I admit it was kind of a special case. There was just no other even halfway rational explanation for his print on the watch that had been on the dresser that morning, and it was as good a print as you can ever find. New patrol deputy, two weeks on the job, found and lifted it.

One or two of the federal circuits got the notion that expert testimony on the general nature of eyewitness identification should only be required if there was no corroborating evidence. That's absolutely the wrong view of evidence weighting. Each piece is what it is and means only as much as it means. Surrounding it by more discriminating or more reliable evidence doesn't make it more reliable or meaningful.

<sigh> I'm in the middle of fighting with CT about expert ID testimony at present. One of the sub-issues here is harmless error. FREX: Let's take your case above and assume that there was also a witness who saw the culprit coming out of the house and in a badly mishandled ID procedure ID'd the defendant. Defense offers ID expert to show that procedure was badly mishanded and impeach witness' ID. Judge denies expert, says "that's all common sense, counsel." Appellate court avoids ruling on expert admissibility by focusing on whether defendant was harmed by lack of expert and says that the print evidence is so strong that the lack of the expert could not have affected the jury verdict. In the harmless error analysis, strength of the other evidence is important -- and I think that's what those circuits are looking at. IMHO, the limited number of such experts and ever smaller defense budgets will tend to keep experts out of inappropriate cases.

One or two circuits actually have taken the position that the trial judge should flat have allowed the general nature testimony. I dislike the harm analysis approach to something like this, because I think it tends to encourage the trial courts to also do a kind of harm analysis and freely allow piling on of evidence that should always be presented with the assistance of experts. Having other strong evidence doesn't make the iffy stuff more amenable to "common sense" than it would have been with no other evidence. The appellate court has to do harm analysis to decide what to do with the appeal, but that's a very different issue from whether or not the trial judge needs correction for error, no matter how harmless.

In my case, had it gone to trial with an eyewitness ID issue, and the judge disallowed a general nature expert on the grounds that it was common sense, he'd be wrong. If if meant that the jury could use "common sense" to evaluate the eyewitness evidence in the light of the fingerprint, he'd be even more wrong.