John's Quote about Confidence and Probabilities

[g.]

I figured we should start a new thread on this one...sorry Charles, we 'jacked your thread!

This probability of judgment differs from the probability investigated in studying statistical method." .........

"No one, surely, makes a probable judgment when he can make a certain judgment; yet how can the probable be known to approach the certain when the certain is unknown?"

I am having some heartburn with this. I am not sure this is accurate. About a year ago, a good friend and colleague said to me "Well, I think all decision making comes down to probabilities...". I reserved judgment, but at the time disagreed. Now a year later, and several probability classes later, I see truth in his statement.

Let me apply to latent prints:

Once we make an ID, it's an ID, right? Once we are in John's "White" area of sufficiency, it's an ID, it's a unique match, and that's that. It's all the same confidence.

What about this. Let's say you have a clear full latent fingerprint (100+ minutiae) that you have ID'd. That's the same confidence and strength as a small, fragmentary latent print with 10 minutiae, medium clarity, a tiny bit of L3D, but overall we are talking about a 10 point ID with medium to low quality, that is also identified. Nothing special about either. Garden variety IDs. And you believe that both are solid, good, upstanding IDs.

If you ID'd both of these, then in theory, according to this profession, your confidence should be identical for both, right? BECAUSE they are ID's...same confidence...it's an ID.

So here's the rub: If I hold a gun to your head and say: Pick one of these ID's. If you got the ID wrong, I blow your brains out. I guarantee, 100% of us will pick the perfect, clear full latent print.

But if the confidence was the exact same for both, it shouldn't matter. Should be roughly 50-50. Half should pick the full print, half would pick the low quantity print.

Now try the same scenario, but instead of a full print (100+) vs. a low quantity (10) ID, your choice is between a full print (100+) vs. another full-print (100+) ID. In this case, I suspect, it will be roughly 50-50. i.e. it wouldn't matter.

In other words, even in the areas of certainty (the white), there actually still shades of grey. It's all shades of grey and shades of grey are associated with a probabilistic approach. Certainty is never attained, just approximated closer and closer, until the leap of faith that is taken, is infinitely small.

The leap of faith (if it can be measured) is tied to the confidence and directly to the probability of error.

g.

[Charles Parker]

I think I would be safe to say that confidence levels are a human perception.

In your example you have placed another variable in the equation of confidence level. That variable being the human idea of fear or self preservation. You bet, you got me I am going to pick the 100 point one.

Now take another human concept that is different and see what will happen. IE

1. You have made both ID's the 10 point and the 100 point.
2. You are told that one of them MAY not be a match. (Choice and Doubt)
3. You are then asked to pick one. If you pick the 10 point you will get 1 million dollars. If you pick the 100 point one you get 1 dollar. If either one is the no match you get nothing.

The human element of greed sits in. I have looked at them and I know they are a match and my confidence is up so let us roll the dice. To heck with the $1 one I want my million.

Do you think most people would pick the million over the dollar?

Trying to get the human concept, ideas or beliefs out of a human undertaking is going to be real difficult.

I do like your concept and I have to think on it awhile. I am a slow thinker and it takes me a long time to think things through sometimes.

[Charles Parker]

g. PS

I think your friend is right. All decisions come down to probabilities. The level of probability goes up with the knowledge and experience about the subject. Confidence Level

The more you know about that subject, the better you are equipped to make the right decision.

[g.]

I think I would be safe to say that confidence levels are a human perception.

Charles, thanks for the feedback. I think it is safe for us to say that decisions of identification currently are human confidence levels (based on the information present, our knowledge, skills, training and ability).

But confidence CAN be measured and reported statistically (c.f. confidence intervals). But in order to do that, variance (i.e. standard deviation) must be measured. In other words, confidence does not have to be based on human perception (known as subject confidence) but can be reported using objective measurements of data. Since we aren't currently using those tools, reporting anything other than human perception of confidence for latent print IDs seems impossible to me...but I am certainly open to further suggestions.


FYI:
An example of a confidence level is if I am reporting the average height of a certain population (say 70"). The fact is, I don't know what the TRUE answer is unless I measure every single person in the population. So all I can do is estimate what it is from a sample. At the 95% confidence level, I would report the TRUE average is between 68-72". And at the 99% confidence level I would report the TRUE average is between 65-75". Thus if I want to be wrong 1/20 times I can use the 95% confidence, if I want to be wrong only 1/100 times, then I use the 99%.

The truth and math behind this is a little more complicated than I have presented it, but seeing that this is not a stats forum, I suspect it would just bore people (more so than this thread probably already has )

g.

[Charles Parker]

g----I would not have any suggestions for you that would be very good. However I am interested in your perceived or possible data-set for the objective measurements. Unless you are going for publication on it.

Not knowing a lot about statistics is your work in Inferential Statistics or Descriptive Statistics. Curious so I can do some reading?

As you are probably aware talking probabilities to a group that has for years dealt with absolutes is tough. I could give a homespun analogy, but they are probably bored with those as well.

Seriously I would like to see your ideas on the data-set for the objective measurements. On the thread or e-mail me.

[John Vanderkolk]

Since my last quote on this topic was under "Latent Print Community" Posted: Wed Nov 29, 2006 3:01 am, and I want to emphasize the depth of Lonergan's writings, the first quote was from pp324-325 , Collected Works of Bernard Lonergan, Volume 3, "Insight: A Study of Human Understanding", ISBN 0-8020-3455-1.

the following partial quotes are from pp574-575, I know I am omitting much, but:

Accordingly, the probability of a judgment, like the certainty of a judgment, is a property of its content. If that content coincides with what is grasped as virtually unconditioned, then it is a certainty. .....

However, there is a third sense of probability, that is reached by contrasting infallibility with a certitude that admits degrees.......

Thus, one may call upon the judgments of others to support one's own. Detachment and disinterestedness are independent of circumstances, but bias, unless it is general, tends to vary with circumstances. Hence, certitude may be strengthened by the agreement of others, and this strengthening will vary with the numbers of those that agree, the diversity of their circumstances, the consequent virtual elimination of individual and group bias, and the absence of any ground for suspecting general bias. ....

There are, then, degrees of certitude, and their ground lies behind the proximate criterion of the virtually unconditioned in the more obscure region of the remote criterion. Only if this obscure region were to become completely clarified, either in fact, or more radically as a matter of principle, would certitude reach the absolute of infallibility.

I believe I can be certain without reaching the absolute of infallibility. If I can remove the 'irritation of doubt' and 'become fixated in belief' (Charles Sanders Peirce, late 1800's http://www.peirce.org/writings/p107.html and http://www.peirce.org/writings/p119.html and a general discussion of reasoning, specifically abductive reasoning, scientific method, see http://plato.stanford.edu/entries/peirce/

I would put certainty, as humanly possible and not infallibly, in the white, with probable judgments in the gray and black. As I would approach the region of 100% clarity and all the available quantity of features as details in both of the images, I could approach infallibility, with stronger support for certainty, however, reality will never let either image be 100% Quality and with all the Quantity of the source. Therefore, I can be certain, with varying degrees of support, and I can be probable in the black and gray, if I could figure out how to calculate a probable in the gray or black. Either way, I can always choose to be probable in all decisions I make in the comparative sciences, or I can be certain, yet believing in the fallibility of humans. I can only be certain, if I have removed the irritation of doubt, become fixated in the belief, know of no further relevant questions that need to be asked and answered, make sure I am not avoiding the relevant questions because of bias, and seeking support by calling on the support of others.

If judgments are only probable, why are not all of our forensic comparative determinations probable?

Because no two images will be complete quality and quantity of the same source, judgments must be involved, therefore, if only probable judgments and probable conclusions are to be used, they should always be used in every examination. I am not yet advocating that, as I believe we can be certain, while acknowledging different levels or degrees of support for that certainty, and there is always varying support in each examination. There comes a point, when the examiner knows the relationship of the unknown to the known, certainly speaking, when no further inquiry would change the judgment, support is sufficient for certainty without the need for 100% support, the judgment of the examiner or the judgments of the supporting community. No human is infallible, but we can be certain. I can demonstrate the support for my certainty, and it is easier to demonstrate that support with 99.999% clear prints with 99.999% of the volume of the three dimensionsal skin recorded as images.

I do welcome statistical studies of the relationship of details to judgments to confidence to probabilities. Keep collaborating, thanks, John V.

[Bohemian Rhapsody]

This is a fun thread. I like both analogies posed by Glenn and Charles. They are both appropriate. Glenn’s scenario is ultra conservative, and is similar to how our profession characterizes our decisions. Typically this field has maintained that we make “life and death” decisions and any one of these decisions must withstand the acid test of absolute certainty (these decisions can be the death of someone’s liberty or your own career). Yet, his scenario pokes a large logical hole in the rhetoric we have always subscribed to. I remember having difficulty with that logic in court one time. Logically, a blurred image with less detail will provide one less cause for certainty that something that is pristine, but to say so in open court would label you a heretic.

Charles’s scenario is a bit closer to the decision-making that is actually in circulation. It is the HBO expose of our thought process. Unfortunately we all are asked to be riverboat gamblers at times (check out the Mayfield thread). If you think that I am advocating that probabilities will solve our dilemmas, I can say with “absolute certainty” that they won’t. I think a probabilistic approach that Glenn is talking about will help provide a better framework for our decisions, but it can be abused just like any other argument in court.

BR

[John Vanderkolk]

Question: Are there differences in the phrases?

I think the prints were probably made by the same source.
and
I think the prints were made by the same source.

Then,

I know the prints were probably made by the same source.
and
I know the prints were made by the same source.

So the discussion first focusses on thinking versus knowing. If all answers are to be probable, should we only think they are probable or could we know they are probable? Knowing is stronger than thinking. Thinking is still pondering the significance. Knowing is done pondering, for if we were still pondering, asking and answering, we would not be ready to know yet. We would still be thinking or reflecting.

Knowing is the effort to seek the truth. If we know the answer to an examination, how does it become less than truth? But this is not the ultimate, absolute, infallible, unconditioned truth.

Also, I emphasize, there is a difference in absolute truth and universal truth. There is a difference in unconditioned truth and virtually unconditioned truth. Universal truth, or virtually unconditioned truth, or knowledge, is the best known information within the community of knowers. Absolute truth or unconditioned truth of reality is unattainable. How do we know we have reached the ultimate completeness of knowledge of any phenomenon in nature?

How certain of the universal truth can we be, realizing we can not be certain of absolute truth because we have not gotten to the final limit of absolute truth. If I can not trust the universal truth of my belief of others with whom I collaborate and knowledge that I have gained within that community, how can I know or believe anything? Why can we not be certain of universal truth or virtually unconditioned truth as the result of our examination? If we can not be certain, how can we know anything?

In the probabilty discussions, can we ever know truth, or do we think we are approaching truth with greater probabilites? Or, do we know we are approaching truth? In the knowledge, or experience, understanding, judgments of self and our communities, once we know we are beyond the irritation of doubt (not absolutely no doubt), we have no further relevant questions that would change our judgments, we are certain (not infallibly), we are fixated in our belief, we know truth within the universe of our community (virtually with no other conditions available to our knowledge)(not absolutely). I know the prints were made by the same person.

[Michele]

John,

You asked, "Are there differences in the phrases?" To me it looks like the level one detail is the same but the level 2 detail (or perhaps it’s the level 3 detail) is different

On a more serious note....if we state to a jury that we know the prints were made by the same person, this may be true according to our definition of what ‘to know’ is but we may be swaying them into believing that it’s absolute knowledge. Even though we know what we mean, if the jury believes we mean something else, aren’t we overstating our results or misleading them somewhat? I believe my job is to educate the jury about the evidence and not convince them of my opinion.

Absolute knowledge is hard to come by, many would say that it only exists in exact sciences (like algebra, geometry, etc. - I wouldn’t say that mathematics is an exact science because it included disciplines like statistics). We can’t arrive at absolute knowledge but we can reasonable indicate it through carefully controlled tests, rigor, and scrutiny. Some may even say we arrive at our conclusions through a systematic process of logical thinking. The knowledge we arrive at should hold the test of time regardless of our state of mind, the person conducting the tests, the prestige of the person conducting the tests, or the convincing powers of this person. Our conclusions are justified by the data and by the knowledge we have about fingerprints (unique and persistent). As you indicated, this is universal truth.

We all have to testify using words we feel comfortable with and testify to what we truly believe. I testify that after careful analysis, the prints were found to be that of John Doe. If I’m asked about how accurate this conclusion is or how confident I am…..I say “to a reasonable degree of scientific certainty”. I’ve had to go further and explain what I meant by this and the jury seemed to understand perfectly.

[John Vanderkolk]

Michelle wrote:

“to a reasonable degree of scientific certainty”.

I struggle with explaining this phrase because:
1) If there is a reasonable degree, what is an unreasonable degree of scientific certainty?
2) When does an unreasonable degree become a reasonable degree?
3) How many degrees of certainty are there, scientific or otherwise?
4) What is certainty? since it is part of the sentence.
5) Is scientific certainty a greater or lesser degree of certainty than just plain certainty?
6) Is scientific certainty an attempt to say the certainty appears more certain than certain? Or is scientific certainty an attempt to say certainty is less certain than certainty?

Then to keep other discussions going:

If we all do testify using humanly developed words that describe the human process of knowing, being certain, or being probable, and these processes are still being explained imperfectly, and we all have to testify using words we feel comfortable with and testify to what we truly believe, is this not certainty, with or without scientific degrees, if we truly believe what we are knowing? If we choose to use words of probability, can we be certain of the probability or are we only probable of the probability?

If we can never be certain, how can we be certain that probabilities are the only form of actual knowing? If we can be certain that probabilities are knowing, then certainty is acknowledged as a level of knowing. Being certain of probability acknowledges certainty. Yes, absolute knowing of probable certainty or certain probability is hard to come by. Knowing the words we are comfortable with is the first step of testifying. I am doing my best to expand my knowing of probable and certain.

I do enjoy these discussions.

[Norberto Rivera]

As a "certain" rookie in this group I will throw this question at all of you: Considering Michelle's statement that mathematics is considered an exact science, has there ever been any type of statistical study as to the uniqueness and permanence of fingerprints using a mathematical methodology?
I have been tossing around the idea of using a coordinate type system to map and compare minutiae locations in a data set of known fingerprints. This would give each individual point a specific, quantifiable location within each pattern. Mathematics and statistical analysis can then tell us how many times a particular point repeats itself, if at all, at a specific location. Granted, it's considerably more complicated than what I've just explained, but it's the Reader's Digest version. Obviously the type of minutiae and the initial orientation of the patterns within the grid have to be considered and made constant for all patterns studied. I'm far from a statistician and my math could "probably" use some refining, but I think this type of study helps take the "probability" closer to "certainty" in our line of work.

I apologize if I'm hijacking the thread in another direction, but it seemed the appropriate place to post.

I will eagerly await your retorts

[Bohemian Rhapsody]

“to a reasonable degree of scientific certainty”.

I struggle with explaining this phrase because:
1) If there is a reasonable degree, what is an unreasonable degree of scientific certainty?
2) When does an unreasonable degree become a reasonable degree?
3) How many degrees of certainty are there, scientific or otherwise?
4) What is certainty? since it is part of the sentence.
5) Is scientific certainty a greater or lesser degree of certainty than just plain certainty?
6) Is scientific certainty an attempt to say the certainty appears more certain than certain? Or is scientific certainty an attempt to say certainty is less certain than certainty?

Likewise, does this mean we do not hold ourselves to be 100% sure we have a match? And if we still do that can we not simply say that in court?

I think if we release ourselves from absolute certainty then we could slip into being "less certain" in our decision making. "Hey I never said I was absolutely certain, just scientifically certain" is what could arise later. I think instead of qualifying "certain" (in essence "certain" does mean positive) I think we should instead attempt to quantify our work.

To simply change our wording with no real difference in our research just kind of seems like a marketing ploy.

[Michele]

Well, I’d say that to a reasonable degree of scientific certainty, I did not say that mathematics was an exact science, especially statistics. What is scientific certainty? My conclusion can be reproduced by others and the method I used can be peer reviewed to determine if the conclusion was arrived at reasonably through valid means. My first testing method is to reread what I wrote. Is this enough testing? Maybe to most people but not to a reasonable person. More testing…..I’ll look to see if the edit icon indicates that my post was edited. Nope, it appears that this post is in its original condition. Do I need more testing? In my opinion I don’t but I’m human and therefore I have my biases. I’ll leave this open for others to peer review. Maybe someone else will suggest another testing method. We all have our biases but the more people who peer review my method and my conclusion…the more we diminish each individual bias. With repeated testing by others we either arrive at disproving my conclusion or we arrive at consensus. If we have this and no evidence to the contrary then my conclusion is established and it’s scientifically supported.

Am I absolutely certain? How do I quantify this? How do you quantify 99% certain or 100% certain? If we can’t quantify it, maybe we shouldn’t use it. A person who is thinking scientifically is constantly skeptical and therefore never 100% certain.

[Charles Parker]

When dealing with probabilities from a philosophical POV (not scientific, statistical or legal) there are three types: 1) Logical 2) Physical 3) Technical.

Is it logical possible for me to travel from here to the moon in 5 seconds? Yes I can image it with worm holes and hyper drives, but it is not physically or technically possible (speed of light and that kind of thing).

Is it physically possible for me to travel from here to the moon in 1 day? Yes that is within the current limits of science (speed of light), but it is not physically possible because we do not have the technology yet for that type of flight.

Is it technically possible for me to travel from here to the moon in 15 days? Yes that is currently technically possible.

To know the answer about possibilities you not only need to consider what type but also the other three areas listed above. If your desire is to move the fingerprint discipline into the scientific world then you have to consider possibilities from that arena. If your desire is to move the fingerprint discipline into the statistical arena then the concept of possibilities are different than from a mathematical POV. If your desire is to use the term possibility in the legal arena then the concept is different from the mathematical or statistical POV.

Determine what ground you are on and then observe the ground the other person is on. Once you do that the path between the two should be easier (only works if the other person does the same).

Is it possible that this message is a figment of my imagination and I did not write it and I am just in a dream? As Curly would say Sertaintily

Truth is subjective----just pick one! Shark

[Bohemian Rhapsody]

To simply change our wording with no real difference in our research just kind of seems like a marketing ploy.

Forgive me. I was speaking in terms of our entire field. I was trying to picture the majority of examiners using "scientific certainty". Not a bad term, but I don't quite think it does the trick for our profession. I don't think a juror would get the difference between scientifically certain or absolutely certain. But those could be weasel words for an attorney. Scientific certainty is only a slight shift from absolute certainty and I don't think it does anything more than open the door for charlatans to come racing in. I think a real difference would be to quantify our degrees of certainty. This would be complex and require collaboration with other disciplines but could be done if we wanted to.