Confirmation Bias (redux)
Posted: Tue Aug 14, 2007 9:26 am
I'm going to move this response over from the verification thread as it is likely to generate a long discussion of its own.
The Patterson amicus briefs are available on Kasey's site. The two amicus briefs filed in support of the defendant discuss confirmation bias, although both were written before Dr. Dror's research was published and do not include it.
If the attorney is claiming a deliberate failure on the part of an examiner to challenge what he or she knew was a questionable match, that's not a confirmation bias issue. It may be a problem with the lab or agency's culture or procedures, or the relationship between verifier and examiner, but it shouldn't be lumped into the confirmation bias challenge.
If the attorney is claiming a deliberate misrepresentation of a result by an examiner, that's also not confirmation bias.
Confirmation bais challenges deal with subconscious influences. By their very nature, they are not something you, the examiner, would be aware of. There's a fair amount of research in the eyewitness ID field that deals with how the mind can play tricks on one's perceptions and memory, to make one perceive what one expects to perceive, and recall what subsequent information causes one to expect to have perceived at the time. Dror's research suggests that biases can't override clear information -- they won't mislead an examiner into calling a clear non-match as a match, or vice-versa. They can affect how one perceives ambiguous or unclear information -- perhaps turning an inconclusive into a match, or vice-versa.
One way to deal with this is blind-testing. Lawyers have already been having this fight with police and the courts over eyewitness identification procedures. The researchers have been saying for over a decade that it is important to have the police officer who administers an array or line-up be blind to who is the suspect, so that he or she doesn't inadvertently cue the witness by body-language (showing extra attentiveness when the witness looks at a particular image or person) or questions ("are you sure", when the witness picks a foil) or tone. Many police responded, just as you folks are, with indignation, thinking that the researchers were accusing them of deliberately leading the witnesses. Others found that when they applied blind tests, their witnesses were more likely to pick foils or not make a pick, and rejected the tests. (In the recent Chicago study, when non-blind, simultaneous tests were used, not a single witness picked a "foil" in two counties.) Others are testing the recommended methods and finding them workable. It is going to be a while before the courts decide what they will require for admissibility.
As I understand it, the researchers and lawyers suggesting blind testing are not trying to impune the examiners' integrity or training. They are saying that the human mind does not always work in a transparent way, and that there are risks that biasing information given to an examiner or verifier prior to making a decision could affect that decision.
Police have known about memory contamination by post-event information for a while, as MDavis noted in response to SWiese's example, it is important to know if the witnesses were interviewed seperately and did not converse with each other after the accident and prior to the interview. Police interview methods generally stress separating the witnesses as soon as possible to prevent inadvertent contamination of their memories by post-event information. Again, the effect is relatively subtle. It is unlikely that any conversation is going to turn a witness' recall of a "cherry red over white 1958 Corvette convertible" into a late-model beige Toyota sedan, but it might change a witness' perception of a zero on a license plate into a letter "o", or a beige Toyota into a dirty white Nissan. And that's the level of concern here -- information that might change a perception of an inexplicable dissimilarity into an explicable distortion, or vice-versa.
I think there may be a fundamental misunderstanding about the confirmation bias issue. Perhaps my collegues at the bar are not explaining their challenge clearly enough or are confusing what they are reading in other sources.sandra wiese wrote:The problem with so many accusations of potential bias is that these arguments all seem to exclude the reality of independent thought. While I realize there are situations in which one examiner might be cowed into agreeing with a more forceful or experienced examiner, I believe the reality is that these situations are far less common than posts to this site would have me believe. Is there a large percentage of cowardice in this field of work???
The Patterson amicus briefs are available on Kasey's site. The two amicus briefs filed in support of the defendant discuss confirmation bias, although both were written before Dr. Dror's research was published and do not include it.
If the attorney is claiming a deliberate failure on the part of an examiner to challenge what he or she knew was a questionable match, that's not a confirmation bias issue. It may be a problem with the lab or agency's culture or procedures, or the relationship between verifier and examiner, but it shouldn't be lumped into the confirmation bias challenge.
If the attorney is claiming a deliberate misrepresentation of a result by an examiner, that's also not confirmation bias.
Confirmation bais challenges deal with subconscious influences. By their very nature, they are not something you, the examiner, would be aware of. There's a fair amount of research in the eyewitness ID field that deals with how the mind can play tricks on one's perceptions and memory, to make one perceive what one expects to perceive, and recall what subsequent information causes one to expect to have perceived at the time. Dror's research suggests that biases can't override clear information -- they won't mislead an examiner into calling a clear non-match as a match, or vice-versa. They can affect how one perceives ambiguous or unclear information -- perhaps turning an inconclusive into a match, or vice-versa.
One way to deal with this is blind-testing. Lawyers have already been having this fight with police and the courts over eyewitness identification procedures. The researchers have been saying for over a decade that it is important to have the police officer who administers an array or line-up be blind to who is the suspect, so that he or she doesn't inadvertently cue the witness by body-language (showing extra attentiveness when the witness looks at a particular image or person) or questions ("are you sure", when the witness picks a foil) or tone. Many police responded, just as you folks are, with indignation, thinking that the researchers were accusing them of deliberately leading the witnesses. Others found that when they applied blind tests, their witnesses were more likely to pick foils or not make a pick, and rejected the tests. (In the recent Chicago study, when non-blind, simultaneous tests were used, not a single witness picked a "foil" in two counties.) Others are testing the recommended methods and finding them workable. It is going to be a while before the courts decide what they will require for admissibility.
As I understand it, the researchers and lawyers suggesting blind testing are not trying to impune the examiners' integrity or training. They are saying that the human mind does not always work in a transparent way, and that there are risks that biasing information given to an examiner or verifier prior to making a decision could affect that decision.
Police have known about memory contamination by post-event information for a while, as MDavis noted in response to SWiese's example, it is important to know if the witnesses were interviewed seperately and did not converse with each other after the accident and prior to the interview. Police interview methods generally stress separating the witnesses as soon as possible to prevent inadvertent contamination of their memories by post-event information. Again, the effect is relatively subtle. It is unlikely that any conversation is going to turn a witness' recall of a "cherry red over white 1958 Corvette convertible" into a late-model beige Toyota sedan, but it might change a witness' perception of a zero on a license plate into a letter "o", or a beige Toyota into a dirty white Nissan. And that's the level of concern here -- information that might change a perception of an inexplicable dissimilarity into an explicable distortion, or vice-versa.