CLPEX Chat Board

All-

We are conducting an experiment that looks at the tradeoffs that can occur with different decision thresholds.

Josh mentioned this on another thread, and so I thought I would create a separate thread to help recruit subjects as well as allow for a discussion of the experiment.

There is a short, 6 minute video in the beginning, and then there is a visualization that helps us understand the tradeoffs in any forensic discipline. I've discussed this in workshops at IAI and it generated lively discussion.

If you have a few minutes, follow the link below. The goal is not to say whether your values are correct, but to generate discussion of the factors that could affect decision thresholds. You are welcome to look and then not submit your data, but please keep in mind that without input from examiners we can't help improve the science.

Once you have done the experiment (either submitting your data or not), I invite you to discuss it below. If you haven't done it yet, please at least look at it before you read the comments so you can form your own opinion first.

Tom Busey
Indiana University, Bloomington, IN

Participants needed for an online experiment

We are recruiting participants for a study that addresses the decisions made by fingerprint examiners. You do not need to be a fingerprint examiner to participate. The results will allow us to better understand whether the decisions made by examiners correspond to the values of society. We would appreciate it if you would distribute the information below to anyone you think might be interested in participating.

https://goo.gl/JLGyVy

(please copy and paste this into a new window)

Sincerely,

Tom Busey and Willa Mannering, Indiana University

That wasn't what I was expecting. I'll be interested to see how many people feel there is an acceptable amount of erroneous conclusions that affect innocent people if it means catching more of those who are guilty.

I am new to this chat board as a member. Just curious if anyone else is having trouble with clicking on the consent button to go to the video? Not sure if it's my computer, issue with the site, or perhaps the study has closed. Thanks.

afisguy97 wrote:I am new to this chat board as a member. Just curious if anyone else is having trouble with clicking on the consent button to go to the video? Not sure if it's my computer, issue with the site, or perhaps the study has closed. Thanks.

I think a direct click opens this up in the frame, so try to copying the link into a new browser window. Google Chrome is probably the best browser, but Firefox works ok for this.

-Tom

I just participated. Forgive me if I am not comprehending what you feel is obvious, but I fail to understand why there has to be trade offs at all? Our profession is not a matter of cumulative decisions that society may or may not be happy about, it is more like the roulette wheel where ever decision stands on its' own merits: ID, exclusion, inconclusive. These three decisions are independent of each other and absolutely independent of the feelings of society about the conclusions. As such, what is the actual ultimate purpose here?

Thanks for participating and asking your question. The question of why there have to be tradeoffs is really at the core of this research.

Let's think about it from a different domain. Suppose you are a TSA baggage screener looking for banned items in carryon bags. You are going to see some signal (maybe a knife-like object) and you decide to pull the bag for manual inspection. Now, sometimes it will be an actual knife, and sometimes it might be a rubber knife for halloween that is painted with metallic paint so it shows up on xray as a knife. So in this task, you accumulate some amount of evidence (how knife-like does this object look) and you compare that to some internal threshold before you make a decision to pull the bag. Sometimes it will be an actual knife, and sometimes it will be a fake knife. Of course, every time you pull a bag, you slow things down. But of course not pulling a bag could lead to another 9/11. So you have the same kinds of tradeoffs. It is probably safe to be very conservative, which would create a situation where every other bag is pulled and the line really slows down, making the general public mad and threatening the viability of the very air travel that you are trying to preserve (because people will get frustrated with long lines and just not fly). If you become less conservative, however, you make the line go faster but you risk missing a knife that could be used to bring down a plane.

Of course, we could develop better scanners that could separate real from fake knives, and we could improve the training of the experts so that even with the current equipment the better trained experts can distinguish real from fake knives. All of this would help. But no system will be perfect as long as there is noise in the system, or there people like the 'red team' who know how to disguise objects to fool the scanners. And if the system is not perfect there will always be two kinds of errors- bags that get erroneously pulled, and knives that slip through the system. TSA scanners just have to decide which kind of error is worse, or how to adjust their decision threshold to reflect the tradeoffs that can occur.

(note that TSA examiners have control over their decision threshold- if an airport receives an alert about possible terrorist activity, they can adjust their decision threshold and pull any bag that has an object that is even remotely knife-like).

With fingerprints, you have a similar situation, except that you can also use the 'inconclusive' category, which has values that are very ill-defined. What is the utility of an 'inconclusive' response to the justice system? Does it add something? Is it over-used?

Now, there is a way out of this situation for fingerprint examiners that avoids these tradeoffs. Suppose that we were to move the top cloud of points far to the right? That would make it easy to avoid errors, AND exonerate innocent people AND put guilty people in jail. However, the only way to do that is to do what I proposed for TSA screeners- develop a better xray system that can distinguish between metal and metal-painted rubber. But I don't think this is immediately going to happen in fingerprints, because with fingerprints, humans and computers are working with the same information: pixels. In TSA the xray system can use different wavelengths of xrays, or we could develop a 3d scanning x-ray system, or do may other things to make real knives look more knife-like, and fake knives look less knife-like, because we have access to the bag itself. Fingerprints are just residue on a surface, and I don't see computers becoming much more efficient than humans anytime soon.

So where does that leave us? It leaves us with the small but significant improvements that can come from training and experience, which is why you and other folks on this board should be commended for working to improve their abilities. You will still have the tradeoffs, but you will be able to subtly minimize the overall problem because your point clouds will be further apart.

-Tom