Statistics and Misidentifications - The weeks Detail

[Michele]

I really think this weeks Detail is nothing more than mathematical manipulation.

Here are two statements, the 1st one is my interpretation of what’s being said and the second is a direct quote.

1) Whether or not someone touched something and lied about it can be reached without looking at the quality of the fingerprint work.

2) “The method of choosing a crime scene has a large effect on the certainty of an identification”.

Whether you have a background in statistics or not, I think anyone knowledgeable about fingerprint comparisons can see how ridiculous these statements are.

Suppose 100 latent prints were found in a home. Logically, the probability of any one of them belonging to the homeowner is high (since their presence has already been established). Knowing the probability may be interesting but it’s not useful information in arriving at a conclusion. Using this information to come to a conclusion or even supporting a conclusion is a biasing factor. The probability of an erroneous ID is irrelevant to whether or not an erroneous ID actually happened. As an example, the probability of me getting in a car accident today is probably low. But that’s irrelevant if I actually do get in a car accident.

I believe you can make numbers show whatever you want them to (remember the saying Lies, Damn Lies, and Statistics?). Whenever you’re looking at any type of research or how conclusions were arrived at you need to also look at the authors expertise and if the author has an agenda. In this case, it’s obvious this author does have some sort of agenda. Regardless of which side you are on with regard to the McKie case (even if this data supports your conclusion), I don’t believe it’s ethically right to use false statistics to support your conclusion.

[Outsider]

Whenever you’re looking at any type of research or how conclusions were arrived at you need to also look at the authors expertise and if the author has an agenda.

Let's start with this one. I have an engineering degree from the 1970s and I am now a computer programmer. I study the areas of statistics that I need to get a job done. This matter just needs a bit of statistical thinking (taking an overview rather than looking from the point of view of one case) and some school level arithmetic - multiplication, division, ratios. I DO claim to have a talent for getting quickly to the core logic of a situation.

I have no agenda other than the fact that I believe I have a useful contribution to make. I don't know anybody involved in the case. I saw snippets of the Scottish Parliament inquiry on local TV and I realised that the McKie allegation was a case of the Texas Sharpshooter fallacy. It is not the classic example that is given in the text books but the core logic is the same.

"Here are two statements, the 1st one is my interpretation of what’s being said and the second is a direct quote.

1) Whether or not someone touched something and lied about it can be reached without looking at the quality of the fingerprint work.

2) “The method of choosing a crime scene has a large effect on the certainty of an identification”.

Quote no 1 is definitely NOT what I am saying. Quote no 2 IS what I am saying. I don't know whether McKie is lying or not. I am talking about certainty. You are in the business of calling someone a liar, to do this you need something very near to certainty. If you read my conclusions carefully (I wrote them carefully) you will see that I am saying that there is no CERTAINTY that McKie is lying and you can come to this conclusion without looking at the quality of the fingerprint work (although looking at the quality of work is a good idea).

the probability of me getting in a car accident today is probably low. But that’s irrelevant if I actually do get in a car accident.

Fair point but in fingerprinting you never know when you have a car accident (an erroneous ID). Probability is all you have got.

What is important is the mechanism which brings a particular ID out of the millions of IDs declared annually into a court room as the basis of a prosecution case. Before this mechanism starts, we can assume that every ID has a very very small probability of error. The quality of the fingerprint work and some other factors will decide exactly how low this is. All the good IDs of the homeowner will never be the basis of a prosecution case. It is like having a barrel of apples with one or two bad ones. If you take out hundreds of good apples the proportion of bad to good left in the barrel is higher than it was.

If IDs were chosen at random to base prosecution cases on then you would get the fabulously low initial error rate in the court room. But you don't get that because there are two mechanisms which make the police choose a particular ID for prosecution. One mechanism is that the ID is good and the person identified cannot give an innocent explanation for being at the location so he lies and says that he was never there. The second mechanism is that it is a misidentification and the misidentified person becomes the accused. This is a classic non-random self selecting sample, the ID selects itself (if any latent in a case is misidentified it will be the one the police choose), so you have lost your initial very low error rate. To have enough confidence or certainty to call the accused a liar, you must know that that first mechanism is overwhelmingly more likely than the second. Drawing a cordon round the crime scene is a simple and effective way of doing this. It bumps up the chances of first mechanism working because we know the criminal entered the crime scene and there is a good chance that he left a latent, and it bumps down the second mechanism by limiting the total number of latents in the case, thus managing the risk from misidentification. The Shirley McKie perjury case did not have the benefit a cordon round the crime scene (you cannot count the cordon for Marion Ross' murder). The self-selecting nature of a misidentification has been working over every fingerprint case, week in week out. So in the McKie case we are left with a shaky "balance of probability" argument which is not enough to call her a liar (I suppose some people DO enter crime scenes without permission). Actually in statistical thinking it is wrong to even talk about "the" Shirley McKie case, a random misidentification could result in any police officer – or anyone else – being accused of entering any crime scene.

If the fingerprinting world had understood these selecting mechanisms in 1997 then I am sure that the Shirley McKie case would have been nipped in the bud and quietly handled within the industry.

Now I know that there is more to the accusations of incompetence in the SCRO but nothing in the Asbury case would change anything I have said here. Let's take one thing at a time.

For my take on why the McKie case is an example of the Texas Sharpshooter fallacy, go here and click on the link near the bottom.

http://www.stevehornsc.pwp.blueyonder.co.uk/tfp.htm

[Michele]

When I was speaking of looking at the agenda of the researcher, I was speaking of any researcher, not just you. It is well known in science that researchers have reasons to publish research (job requirements, notoriety, financial gain, etc) and many times these reasons lead to unsound research, whether intentional or not. The desire to ignore data that doesn’t fit or to make our research fit our preconceived ideas is so strong sometimes we don’t even realize we’re doing it. This was not meant to be an attack on you, just something we all need to be aware of. I can guarantee you that I’ve said the same thing to friends of mine doing research, not meant as an accusation, just as a rational scientific form of peer review. Let me ask you this, did you try to prove that your conclusions were false?

I also didn’t mean to misquote you, what I was reading was:

“With no other prior evidence than the fingerprint there is no certainty that Shirley McKie is lying and she should not have been prosecuted for perjury. This conclusion can be reached without looking at the quality of the fingerprint work.”

I don’t want to talk about the Mckie case, but as a general statement I believe this statement to be false. In most cases you can reach a sound conclusion by looking at the quality of the fingerprint work. By comparing the fingerprints you can scientifically determine if the person touched an item or is lying. There’s no need to use probabilities when there’s a better determining factor, fingerprint comparisons. In an estimated 99% of erroneous ID’s, this has been how a conclusion was ultimately arrived at. You mentioned that with an erroneous ID, probability is all you have got. Like I said above, in most cases you also have the latent to re-examine.

I agree with your example about apples but in your apple scenario, each time you pull out an apple the proportion changes making each event dependent on how many apples have been taken out. Erroneous ID’s are independent events and you can’t always use the same statistical theorems for both dependent and independent events.

“To have enough confidence or certainty to call the accused a liar, you must know that that first mechanism is overwhelmingly more likely than the second.”

This would be true if identifications were based on confidence levels, but as I’ve mentioned before, I don’t believe they are (or should be) based on confidence levels. Identifications should be made based on reproducible and justifiable characteristics between the prints. The problem with the Mckie case is that the characteristics between the images don’t seem to be reproducible to most examiners.

One thing just dawned on me (which would support your view), while the majority of people I work with believe fingerprint conclusions must be based on reproducible and justifiable characteristics, other countries believe that it’s strictly ‘opinion evidence’. Maybe that’s the real basis for our disagreement. I see a huge difference between someone’s opinion and a justifiable determination, but that topic should be dealt with in a different thread. If fingerprint evidence was solely based on someone’s opinion, then I agree with you, statistics and probabilities would be very beneficial. But in all the erroneous ID’s I’ve seen (I’m not including the McKie case) whether or not it was an erroneous ID could be establish by looking at the prints.

[Outsider]

Let me start by giving you a picture of what is in my head and why I am spending so much time on this. Imagine that a pilot was flying his plane upside down and it crashed. You would expect the fact that it was being flown upside down to be discussed at some point during the crash inquiry. You can argue that the plane was designed to be flown that way and pilots should be able to handle the plane upside down and maybe it had no bearing on the crash. But you would at least expect it to be discussed. Flying upside down is not usual, maybe the assumptions of safety that apply to normal flying do not apply upside down. I see an allegation of lying without a prior crime as a major departure from normal fingerprint practise and I think it should be discussed.

I am not suggesting that the probability model is better than looking at the quality of fingerprint work for determining if a misID has occurred. I can't think of any other way other to know if a misID has occurred other than looking at the prints.

All I am asking you to do is to think what a world with a misID rate of, say, one per million IDs looks like. How these misIDs turn up in court cases. In the normal crime-led situation (case 3 of the Town Fingerprint Project) a misID will occasionally be used in to prosecute someone. Unfortunately there is no way to prevent this but for every miscarriage of justice we will get a thousand criminals put behind bars - and we need this. In an ID-led situation, (case 2) I reckon we only get 4 or 5 liars for every person falsely accused. If nobody is suggesting that a crime has occurred (independently of the ID) then why bother making the allegation of lying at such a great risk? The thing that makes case 2 so risky is what statisticians call the 'population' (the total number of IDs that carry the risk of misID). In a crime-led case this is limited, in an ID-led case it is not.

The work of fingerprint officers in the McKie affair was checked back a year and every ID was found to be OK. I say that this does not tell us very much if all the cases had only a few latents and a few strong suspects. The chances of a fingerprint of a suspect looking very much like a latent which he did not deposit is very low in these circumstances, so the skills of the fingerprint examiners are not being tested. I would rather use probability modelling to predict which type of case is most likely to lead to an error, and check these first. For example:
· any case where an allegation was made not related to the crime under investigation
· big database searches
· cases where the police have no clear ideas what they are looking for before the fingerprint work starts ("go to all the suspects houses, develop fingerprints and we will come up with ideas after we get the identifications back" - this has no prior hypothesis ).

[Daktari]

I see an allegation of lying without a prior crime as a major departure from normal fingerprint practise and I think it should be discussed

.
Get real Steve; Shirley had been in trouble before over leaving her prints where they should not be. The so-called 'dead baby case' for example.
Read what MacKay says about that!
That's why she was banned from entering 43 Irvine Road.
Incidentally, she stated on BBC’s Frontline Scotland, that she had never been in trouble before. Maybe she was acting on her lawyers’ instructions then too.

[Outsider]

Get real Steve; Shirley had been in trouble before over leaving her prints where they should not be.

Daktari, this is a fingerprint forum not a forum for pointing the finger. Stick to fingerprint issues.

[Daktari]

Steve did you not say

and I think it should be discussed

?

[Outsider]

Steve did you not say Quote:
and I think it should be discussed

No Daktari, what I said was

I see an allegation of lying without a prior crime as a major departure from normal fingerprint practise and I think it should be discussed.

I certainly don't think we should be discussing Shirley McKie's character. I was about to say that it has no relevance to the certainty of a fingerprint ID but have a look at my estimate of the likelihood of McKie lying:

http://www.stevehornsc.pwp.blueyonder.c ... ieprob.htm

What Daktari is trying to do is to bump up the first number in the balance of probabilities argument. Fingerprinting should have nothing to do with balance of probabilities arguments.

Remember that we cannot define "the given area and time period" in my estimate as Marion Ross' house in February '97 because that is painting the target after firing the shot.

[Pat A. Wertheim]

Boys, boys, boys. Can't reasonable scientists carry on a discussion without the thread being diverted to another senseless argument over the McKies?

For the record, I checked on the dead baby case. It seems Shirley McKie was involved in the investigation of a dead baby in a garbage bag. She wore latex gloves, as required by her department policy and good hygiene. But this was before her agency was aware of the possibility of leaving fingerprints through latex gloves. She was called onto the carpet for handling the bag without gloves. Because she was so insistant that she was wearing gloves, the agency experimented and found that fingerprints can be left through gloves. In the US, there was published literature about the possibility of leaving prints through single gloves years ahead of that incident, but it took Shirley McKie and the dead baby case to make the Scots aware of it. So they changed their policy to require double gloving, but never acknowledged that perhaps Shirley was telling the truth about wearing gloves in the first place.

The real truth here seems to be that Shirley McKie is a strong woman. She was the youngest female ever promoted to Detective Constable in her agency. But besides being strong, she was NOT submissive. That, my friends, was her unforgivable sin. In a male dominated culture, she was a strong and competent woman who stood up to the males. For that, they can NEVER forgive her. Therefore, she had to be stopped. Even though a judge and jury have acquitted her, even though the government threw in the towel and awarded her the entire amount of damages she originally sought, the men she embarrassed will never give up. Perhaps because she won on both counts, they will never give up. They can beat her no other way, so they have turned to character assassination.

Daktari, won't you PLEASE confine your rants to the threads specifically focused on the McKie case and allow reasonable scientists to carry on other discussions without your interference?

I am calling on you to do so.

[Daktari]

Who’s ranting Pat?
You managed to use 348 words, seven of which are either capitalised or underlined, the on screen version of ‘shouting’ in your last posting.
I used an average of 35.5 words over two posting and never ‘shouted’ once.
You asked

Can't reasonable scientists carry on a discussion without the thread being diverted to another senseless argument over the McKies?

Yet a glance at Michele’s first post will show mention of the McKie case
Have another glance at this week’s Detail’s’ home page’ and you will see reference to the Marion Ross murder case.
Cheers

[Outsider]

I have found a new way of making my point in a discourse, rather than using an imaginary exercise. I was hoping to make it a soundbite, bit it ended up a bit of a soundmeal. An online version is here:

http://www.stevehornsc.pwp.blueyonder.c ... ndbite.htm
(the Town Fingerprint Project is still available).

Here is the text:

Fingerprint departments identify very large numbers of people from fingerprints every year – hundreds of thousands I would imagine. The vast majority of these are people who have an innocent explanation for being at the location - the people who live or work in the crime scene for example. The police will just ignore these and pick out for prosecution the people who deny having deposited the print because they cannot give an innocent explanation for being there.

Let's assume that the error rate is something like one per million identifications (the number does not matter). These misidentifications are not going to look innocent - the gardener has been in the bedroom, the neighbour has handled the money tin. The person misidentified will not even try to give an explanation, they will simply deny having deposited the print. So unlike the good identifications where only a tiny proportion cannot be explained innocently, nearly all misidentifications cannot be explained innocently.

How then can we be certain or almost certain that someone accused of a crime is guilty? The answer is simple, do what the police and fingerprint departments normally do, go to a crime scene and put a cordon round it. We know that the criminal has visited the location so there is a fair chance that he has deposited a fingerprint. Most importantly there will always be a limited number of fingerprints connected with any one crime. Lets say that 100 fingerprints were found in the crime scene. This means that there is only a 1 in 10,000 chance (1 million divided by one hundred) that our rogue misidentification has turned up here. But one day the misidentification will be included in an investigation and the misidentified person may be accused of the crime. Every miscarriage of justice is a tragedy but for every one we get thousands of criminals correctly prosecuted, and society needs these people behind bars.

But it is not always possible for the misidentified person to be accused of the crime under investigation. It might be a police officer in the investigation team or someone with a cast iron alibi. What happens now? It is obvious, the police will search around for something else to accuse them of. This time it will not be a 1 in 10,000 chance that they are innocent because we have not been working inside a cordon put up for the new "crime". Any identification from the whole of fingerprinting activity over a number of years could lead us here.

It is also possible for a good identification to alert the police to something that they were previously unaware of. For example it is feasible that a police officer entered the crime scene without permission and without being seen and left a fingerprint. But we are in a very different situation from the normal crime-led situation. We can only speculate about the likely balance of probabilities between the two explanations for the disputed fingerprint. This could be close to 50:50 and will never be "beyond reasonable doubt" in either direction. We don't have the certainty that comes from knowing that the perpetrator of a real crime has visited the location plus almost certainly ruling out the possibility of misidentification by the crime limiting the total number of identifications.

The statistical theory behind this line of thinking is known as the Texas Sharpshooter Fallacy. This is a common misunderstanding of probability illustrated by a gunslinger taking a quick shot at the side of a barn. He then paints target rings round the bullet hole and claims to be a sharpshooter. In normal fingerprint work the police drawing a cordon round the crime scene is painting the target and they do this before firing the shot (the fingerprint analysis). If we make an accusation of lying which is not related to a pre-existing crime when we get a disputed fingerprint identification we are painting the target after firing the shot.

Conclusion
If we have a disputed fingerprint identification that cannot be linked to the crime under investigation we must not accuse the identified person of anything on the fingerprint match alone. Proof of wrongdoing must not include the fingerprint evidence.

[Outsider]

I have moved my latest paper to this location:

http://www.stevehornsc.pwp.blueyonder.co.uk/short.htm

I have changed the conclusion to this:

My conclusion
Unless the error rate of fingerprinting can be proved to be zero, if a person identified to a fingerprint disputes having deposited the print and is not accused of the crime under investigation then:

1. The identification has a significantly lower confidence level than an identification of a person who IS accused of the crime under investigation.

2. The identified person must not be accused of anything on the fingerprint match alone.

3. Proof of wrongdoing must not include the fingerprint evidence.

This is a fairly narrow point but if I am right it is important. I have probably gone as far as I can go to distill out the logical core of my argument. The next stage must be to get an acedemic opinion on these three points. If anybody knows of someone with a qualification in statistics or logic or something similar who would be willing to give an opinion then I think that this would be very helpful.

[Michele]

Steve,

You stated, "If we have a disputed fingerprint identification that cannot be linked to the crime under investigation we must not accuse the identified person of anything on the fingerprint match alone.”

This seems to be your opinion and I have met others that would agree with you. But the majority of people (or at least people who work in this discipline) have another opinion and believe that it is possible to use a fingerprint match to establish some truths.

Personally I believe that it’s not up to me to decide whether or not the court decide to accuse anyone of anything. They know the totality of the evidence and the relevance of the fingerprint identification, which I do not. Even though they ‘think’ the totality of the evidence is sufficient to charge someone with a crime, it’s not up to them to convict someone based on this evidence, it’s up to a judge or jury. And it’s up to the defense to show why, in a particular situation, the totality of the evidence isn’t sufficient for an accusation or a conviction.

Whether or not we truly should or shouldn’t use this kind of information is dependent on many variables and it’s hard to discuss these in a hypothetical situation. Each situation is different and should be looked at individually. A fingerprint identification is a tool, sometimes used as an investigation tool and sometimes used as a conviction tool. How this information is used isn’t up to the fingerprint examiner.

I’ve worked on cases where the fingerprint was disputed and there was no other link to this person to the crime. In one situation the print had so much information in it and after a thorough investigation the detectives couldn’t come up with any other explanation except that this person was in the home. The jury agreed.

In another situation the fingerprint was also disputed and there was no other link to this person to the crime but in this case after a thorough investigation the police found that there was another possibility. It was not only possible but highly probably that the print was left on the building material prior to the house being built. This person wasn’t accused.

A thorough investigation is essential in establishing what information is correct, the relevance of the information, and in making sure the information is being used correctly.

The amount of weight a disputed fingerprint identification has is dependent on how much objective information is in the print as well as other related information. I don’t think we can or should generalize about using or not using the identification without including this information. This isn’t information that just ‘may’ be relevant, the information in the print is highly significant to establishing whether an ID is erroneous or not. In every case where the print has been established to be erroneous, this is how it’s been established and the accuracy rate of using this information has been extremely high.

Hypothetically, if a case came along where this information couldn’t be clearly extracted solely on the information in the print, what would that tell us? This could tell us that there’s not a sufficient amount of information in the print to establish a conclusive conclusion. It could also mean that the information used to make the identification wasn’t presented well enough for others to see. In either of these cases a conclusive decision on identification may not be able to be established, but then this too doesn’t mean the print wasn’t left by the person ID’d. A good example of this is the Caldwell erroneous ID. Were the examiners in this case wrong? Probably not but this is one of the situations lumped into what we call erroneous ID’s.

After a lot of rambling, here’s my conclusion, I believe that statistics (and science) try to extract relevant information to aid in arriving at a conclusion. With Fingerprint ID’s, I believe the relevant information is in the characteristics in the latent print, not in the probability of an erroneous ID or whether the person admits the print is theirs.

I'm really not expecting anyone to agree with me, I'm just stating my view.
Sincerely,

[Strict Scrutiny]

Outsider,

Thank you for your contribution. I don't believe it is manipulative, ridiculous, or unethical. I enjoyed it as an introduction to the Texas Sharpshooter fallacy, a concept I was unaware of, but it made me laugh a couple of times as I have thought about some of the keystone cops I've seen in the past that were guilty of similar flaws in logic. I love the academic debate!

Take this for what it is worth since in I have minimal training in statistics: I don't think you can apply your arguments to fingerprint science in general until a more accurate probabilistic model is developed. You see we tell the world that a match is black and white and that all individualized fingerprints have the same probability of being misidentified. We say that probability is zero. Yes there have been statistical models developed for fingerprints (about 10 major studies) but they are insufficient to explain the complexities of fingerprint identification—so we as a field don’t use them, and we don’t reckon we ever will

We say that all of the following latent prints have the same likelihood of being misidentified (which is a fallacy also): 1) Pristine 2) Extremely Good Quality 3) Fair Quality 4) Poor Quality 5) Extremely Poor Quality (so much so that most examiners wont call it a match but three did and now it is in court as a prime piece of evidence).

Latent prints that are of very low quality are more likely to be misidentified, and latent prints of very high quality would never fit your argument because they can be proven accurate with great ease. But until we develop a metric that accurately measures fingerprint quality folks like you will need to lump all fingerprints together—and lumping all fingerprints together does not work for any probability models--so you're stuck.

There is a vastly smaller amount of information in a poor quality latent than a high quality latent. If more academic vigor were required of our field, an image quality metric would have been put in place years ago. But the courts have been satisfied for years about not fixing something that aint broken--and so are we.

So after all my rambling, I would say that your argument works only for the cases where image quality is quite poor (Like scenario #5 above--Like the McKie case).

Take this for what it is worth; I enjoyed reading your posts. Thanks.

[Outsider]

Strict Scrutiny and Michele Triplett

With Fingerprint ID’s, I believe the relevant information is in the characteristics in the latent print

I generally agree. But a fingerprint expert and department who makes a mistake is going to be just as sure that they are have not made a mistake as anybody else. I say that in a world where the error rate is not zero some sort of statistical thinking or risk analysis is a useful addition to your toolkit, perhaps at supervisory or quality assurance level.

You see we tell the world that a match is black and white and that all individualized fingerprints have the same probability of being misidentified. We say that probability is zero.

That may be so but you must agree that some fingerprint departments in the world will not fully individualize and have an error rate of greater than zero. What the Town Fingerprint Project does is to say, what happens if the error rate is not zero? Normal crime-led work has a safety margin, ID-led accusations do not.

I don't think you can apply your arguments to fingerprint science in general until a more accurate probabilistic model is developed

I doubt if there will ever be a precise probability model for fingerprinting, too many variables. But it can be useful to be aware of relative risks and that is very simple, it is just multiplication.

Attempts to try to assess the risk of an individual identification to error, taking into account the circumstances of what led to the ID - suspects, eliminations, database searching etc may be doomed to failure. But trying to assess the risk of the ID used in a court case being erroneous, taking into account the "population" that it was chosen from is a lot simpler and might be more important.

I think that probability predicts that a bad fingerprint department will have more disputed IDs that cannot be related to the crime under investigation than a good fingerprint department, and we may be able to detect that (a good department who make no errors may still have some).

Here is a question for everybody reading this thread. Do you know of any case (other than McKie) where an ID was used to accuse someone of something that was not the crime under investigation at the time of the ID? Please respond if you know of a case. If there are no responses then you must agree that the Shirley McKie case is, procedurally, highly unusual and perhaps unique in the history of fingerprinting. I think we will all agree that the McKie case is unique in fingerprinting infamy. Now if probability really does say that accusing someone of something which is not the crime under investigation is uniquely dangerous, then surely you must agree that this interesting.

All SCRO IDs were checked for a period of a year before the McKie case and no misidentifications were found. With a knowledge of relative risk, I would say it would be better to look back 20 years and find out if there are any other accusations that were not related to the crime under investigation. Check these.

It is a fair point that in fingerprinting you always have the chance to re-look at the images and start again from scratch. That does set it apart from other probability studies. But if the images are going to be looked at by the same people, or even the same department, then there is the danger that the same errors will be repeated