CLPEX Chat Board

"The committee was not charged with issuing a verdict on whether toolmarks are unique -- that is, whether each gun leaves marks that could not be made by any other firearm. Nor was the committee charged to recommend whether toolmark evidence should be admissible in court. However, it became clear to the committee early on that the assumption that toolmarks are unique to each gun has not yet been fully scientifically demonstrated. Much more research would be needed to determine whether toolmarks are truly unique, or even to estimate the probability that they are unique.

Consequently, the committee advises against one statement frequently made by firearms examiners -- that toolmarks link bullets or casings to a particular gun "to the exclusion of all other firearms." Given that uniqueness has not been demonstrated, such statements do not have a firm statistical basis. These claims, which imply an error rate of zero, also don't reflect the element of subjectivity involved in declaring a match, a determination that's always made by a firearms examiner."

http://www8.nationalacademies.org/onpin ... D=03052007

The final NAS report isn't out yet, but it may pose some problems for the firearms ID folks and add some logs to the fire on fingerprint examiners' claims of zero error rate.

No doubt you're right in drawing the connection among the comparative disciplines on the matter or uniqueness. It's been my growing (and likely not very popular) opinion that it's improper to claim absolute uniqueness without a solid proof. The reason it bothers me is that it tends to focus argument to that claim and sets the uniqueness issue up as the imagined key to cracking the disciplines. I believe the issue grows out of the legal history of the forensics.

In the early days of firearms examination, there were some uniqueness claims made in court that don't stand up to what's known today. And even some current claims have to be qualified, such as rifling in some cases in which the possibility of sequentially manufactured barrels is admitted. (I'm not a firearms examiner, although I could probably play one on television, since I know which end the bullet comes out.) It's also my belief that the power of the comparative disciplines to provide proof evidence is, overall, not fatally diminished by a lack of universally accepted proof of uniqueness. In all of the valid comparative disciplines, the whole body of knowledge and experience, joined with a thorough and competent examination, builds a powerful argument for a proper conclusion of belief beyond a reasonable doubt, whether it be a class conclusion or individualization, and we can demonstrate why we so believe and can argue the conclusion with any other person qualified to render their own opinions.

I offer this to consider repositioning perspective on the issue of the validity of latent print examination and to think about where I happen to think we are headed.

I think it's essential to make it clear what kind of uniqueness we mean. We may propose that, for some sufficient quantify of skin or at some fineness of detail, that portion of skin is unique. Hard to prove, but an overwhelmingly powerful presumption. I could just as well point out that, at some fineness of detail, a individual portion of skin is never the same from one day to the next, each day's example being itself unique. Of course, we hold that the uniqueness of skin we mean is that of being examined for particular characteristics that are unchanging, barring mishap. But in firearms, tools, and prints, for the practical application, we are really talking about characteristics represented by impressions, that the impressions made by what we belief, based upon a huge body of experience and knowledge and to a high degree of certainty, to be a specific source.

My point is that we have the tools to demonstrate conclusions to a degree of certainty far beyond many other examples of powerful evidences in criminal trials. Our fields are not invalidated by lack of proofs of uniqueness. The dogma may be at hazard, but not the value to criminalistics. Criminal finders of fact are powerfully persuaded by skillful presentations of a discipline's knowledge base and the carefully constructed presentation of the specific examination and the opinion of the competent expert that the expert believes the conclusion to a high degree of certainty. We have one of the most powerful evidentiary potentials in the realm of criminal investigation. Far more than most applications of engineering. Far more, to be sure, than psychology. I, for one, do not fear any demand for proof of absolute uniqueness, any more than I fear the lack of entirely satisfactory proof of error rate. I seriously doubt either is attainable, for all that folks will make money and academic paper points for trying.

To proceed without a presumption of absolute uniqueness certainly has practical implications. It means presenting an impression examination in careful detail, which means, of course, equally detailed recording of the examination. Objection that this creates a problem with caseload does not impress me, any more than regrets that the report, "Six points, and that's him." is no longer sufficient impress me. They are real concerns, but they are not proper considerations in the matter of what it takes to make valid use of the evidence.

I could offer some more heresy and get a few more faggots tossed on the fire, but another time.

This is not as bad as one may think at first glance. It is the same old story with a different cover on the book. The study by the National Academy of Sciences (NAS) sounds impressive: Ballistic Imaging: Assessing the Feasibility, Accuracy, and Technical Capability of a National Ballistics Database. This study was tasked to study the effects of IBIS (Integrated Ballistics Identification System) and determine if a national database containing images of ballistic markings from all new and imported guns sold in the U.S. should be created.

You may be surprised (insert sarcasm here) to find out a little about the NAS Committee. The panel assembled was comprised of mostly individuals from the world of academia. Here is a breakdown of the committee assigned to the study:

Five Professors of Statistics
Four employees from Google and Intel
Two Professors of Public Policy
Two Professors of Materials Science
Two Professors of Engineering
Two Professors of Computer Science
One Researcher in Materials Science
One Computer Scientist
One Law Enforcement Consultant
One Research Statistician
and three NAS Research Council Staff (Committee on Law and Justice, Committee of National Statistics, and National Materials Advisory Board)

None of those listed have a background in Firearm Identification. Before the NAS Committee began their research, Board members from the Association of Firearms and Toolmark Examiners (AFTE) met with the directors of this study. It was strongly suggested that they include an trained examiner. The NAS illustrated great resistance.

The committee visited several forensic labs and met with some practitioners and it was evident that many scientists on this panel apparently had no clue regarding the discipline of firearms identification. Reportedly, one NAS Committee member, during a presentation asked, "What is a land?" (rifling inside of a gun barrel). While she was an emeritus professor in the field of metallurgy, she didn't know the difference between a rifle and shotgun.

The NAS report stated : "When the committee was being formed, it was decided not to include an active firearms examiner. Instead, the committee had the counsel of Lawden Yates, a former firearms and toolmark examiner and laboratory director, who also served as general counsel to the Alabama Department of Forensic Sciences and as assistant district attorney for Blount and Sain Clair Counties, Alabama.”
This was done at the urging of AFTE and it has been noted that Mr. Yates was added as a non-voting Consultant. Additionally, AFTE formed a ad-hoc committee to assist the NAS Committee with any issues, technical or otherwise, that may be needed. The NAS Committee was informed of this AFTE committee and given contact information for each of its members. This AFTE committee was not called upon for much consulting at all.

The NAS Committee intimately made a good decision in my opinion that: "A national database containing images of ballistic markings from all new and imported guns sold in the U.S. should not be created at this time". Then they went and overstepped their boundaries and offered opinions on topics that they did not have the proper training, experience or even knowledge.

The NAS Committee did go on to say that: "The committee was not charged with issuing a verdict on whether toolmarks are unique -- that is, whether each gun leaves marks that could not be made by any other firearm. Nor was the committee charged to recommend whether toolmark evidence should be admissible in court. However, it became clear to the committee early on that the assumption that toolmarks are unique to each gun has not yet been fully scientifically demonstrated. Much more research would be needed to determine whether toolmarks are truly unique, or even to estimate the probability that they are unique.” (Although, making the statement that more research is needed, there is no evidence that the NAS Committee had reviewed the 40 years of research published in the AFTE Journal.)

Essentially, the NAS Committee looked at some of the failures of the IBIS system and then tried to use these failures to question the scientific validity of Firearm Identification. That would be comparable to taking any shortcomings with AFIS and deriving that the science of Fingerprint Identification is suspect.

The NAS report at one point stated that:
"In this regard, the analogy of fingerprints may be useful: to date, there exists no definitive proof that no two people can have identical fingerprints. Instead, the credence of fingerprint evidence rests mainly on the assertion that - across all the years in which fingerprints have been manually compared - no two people sharing the same individual prints has yet been found."

Because of the strong statistical background of the NAS Committee, they felt that there was not a “firm statistical basis” to state that bullets or cartridge cases can be associated to a firearm "to the exclusion of all other firearms." I disagree. I am sure they feel the same was about fingerprints. Heck, they should have ruled on all comparative science disciplines since they had an equal understating of each.

Simply being a scientist doesn’t quality you to be an expert in other disciplines outside your area of expertise. We have all learned that the hard way in the judicial system. I thought it was foolish for the NAS Committee not to want experts in the field of forensic firearms identification to be involved. Maybe in their next study, the NAS can get together ten brain surgeons, ten computer scientists, ten statisticians, ten psychologists, ten veterinarians and ten proctologists to study plankton (and refuse any input from marine biologists, botanists, invertebrate zoologists, or oceanographers).

8)

I take a different view. Rather than seeing it as using problems in database implementation and utility to downplay examination value, it seems to me that they use a number of current issues in firearms examination, including the uniqueness issue, to demonstrate specific problems in using a database such as they considered.

They may be right in their overall assessment of the practicality of the database. I'm not in that field, but I recognize that there are significant issues that bear on the database question. And they make some very good practical points in Section 9 on Feasibility.

One would wish that they had agreed to include examiners in the group, if for no other reason than that they are passing on the utility of a database, a question in which examiners have a stake. And it might have avoided the improper reference to "subjectivity" that I see as a very common error, even among forensic examiners.

They may show a reaction that takes them outside their brief when they boldface (in the full report) their statement on uniqueness. I think what may be creeping in is a reaction to perceived or real claims of absolute uniqueness in a field where they had an opportunity to say something. I think it's rightly questioned, and I might even give them a pass on the boldface, if the reason was they recognized a hot button and wanted to make their position clear.

It certainly should make latent print examiners grateful that they work with sources that present far fewer problems in terms of well-defined characteristics and vastly simpler permanence issues.

Steve----Impressive----very impressive.

Good material and interesting.

I agree with Steve (and you too, Charles). It is unfair to say the least to let a bunch of academic brains make statements (or opinions) about a science they have no background or expertise in. Just because a name is followed by a Ph.D does not make you an expert on every thing under the sun. I've been trying to explain that to some people I work with! Publishing a study in a nationally accepted journal doesn't make it the gospel; rather, the peer review that follows will hopefully help clear up any confusion that this may cause in the judicial system.

Excellent discussion.
Question: Does anyone know why panels such as this insist on comprising themselves of folks who lack the expertise in the science that will be the focus of study?
In Forensic Documents, we see this all the time as our critics base their attacks from their perspective of the discipline (which is based from their reading some of the literature in our field). If their perspective only affected court testimony, it would be a limited issue. However, their perspectives (their bias at times is quite noticeable) are filtering into published articles........which are then quoted as a sound reference by other academics. I have come to refer to the referencing of a critic's published allegation (one that has not been researched by them and lacks foundation) as an "inked reality". The danger in their practice of referencing only their own published works is that it provides credibility to a criticism that has not been researched, but appears to be legitimate because it has been referenced in 20+ publications. It also has the potential to corrupt an academic who is sincere and legitimately seeks answers to legitimate criticisms. I can see where a sincere academic reads the "inked reality", and trusting the statements published by earlier critics have been researched, peer reveiwed, and accepted as reliable data, would then incorporate them as referenced resources for his/her perspective.

I don't know if panels such as the Ballistics, or publications by the academics criticizing a discipline where they lack the basic foundational knowledge required of a "trainee" is occurring because the academics believe it eliminates bias from the expert, a perceived notion that the discipline expert will strong arm the direction of a study, or (as I suspect) the academic's dim view of the expert as one who lacks the education and intelligence to participate in such a panel or research study. Eliminating us from academic research does not eliminate bias and it seems to perpetuate a skewed perspective that is not assisting the courts or the progression of the discipline.

So, if anyone knows the answer to my question, I would love to know the answer.
Thanks. Again, great discussion.
Jan

One more point, albeit may not be a popular one:

I feel that having input from academics who are sincere in their quest to conduct above-board research provides many benefits to a discipline. Referring to an academic as "sincere" does not mean he/she publishes only good things about a discipline. We have to take the good with the bad.....which is how we assess and evaluate our skills and our methodologies. However, the academic and discipline experts have to work side by side to get this type of research to publication. The academic and the expert are knowledgable in their respective fields. Harm comes about when one assumes he/she "knows" the other field and begins to publish or testify from a personal perspective, not one based on valid research. (I felt the need to write this as I don't want anyone to think I am "anti-academia". I'm not against academia in general. I am against academic bias cloaking itself behind an inked reality)

Thanks. Again, great discussion.
Jan

I was kind of hoping that the recent rush for nearly every college to start a forensic sciences program might foster some closer collaboration between the applied sciences and the hard sciences. One might also hope that the forensics programs will grow their own higher level folks with academic credibility and applications knowledge.

That would contribute a lot in both directions. It would provide folks who understand how to approach the problem of studying the realistic aspects of the forensics, and it would provide some authoritative voices who wouldn't so quickly trigger the "damn PhD's don't understand what we do" reaction that so often gets in the way of rational considerations of their conclusions and recommendations, rather than focusing on their lack of applications experience. Frankly, I see that here. The discussion has been almost exclusively about who they were, not what they said. If we're arguing that the work was flawed because of the make-up of the panel, it's really not much of an argument unless someone demonstrates that it led them into error. The presumption that they will be wrong because they don't practice the application is a poor one and throws out the good with the bad.

Good morning George:
I agree with your statement that ......."If we're arguing that the work was flawed because of the make-up of the panel, it's really not much of an argument unless someone demonstrates that it led them into error. The presumption that they will be wrong because they don't practice the application is a poor one and throws out the good with the bad."......

I don't know if the panel's work is flawed or not. Ballistics is outside of my expertise. I would hope that the panel's work has value and their ideas/suggestions are meaningful. It would not surprise me, though, their "findings" include comments about portions of a discipline that are painfully obvious (a real "duh"), suggest certain actions that are not prudent (an expert in the field could have told them why), and, make suggestions that are beneficial, but there is funding lacking to implement the action.
I speak from experience of what has occurred in QD and, to a certain extent from what I have read regarding LPE critics and articles. It is a dangerous and slippery slope to have panels or any research project comprised only of parties whose knowledge of an expertise is what they have read, or from supervising scientists in the particular discipline. I do feel having at least one expert from the involved discipline, whether it is a panel, working group, research, etc, is wise as that individual brings something of value to the table, i.e., an in-depth understanding of the discipline. Academics can read all of the articles and books in the world, but it will not gain them the 'real world" application of the discipline. It would also add some validity to the work. Quite honestly, I've read a great deal regarding neurology, even watched a few Discovery programs on brain surgery.....but, you would not want me to be your surgeon.

I agree it is not a good practice to disregard the work of academics. Even when members of an expertise are excluded, reading the published works of the academic profession is worth the time. The work may contain baseless allegations and obvious errors (again, an expert would have caught them) regarding methodology applications, but in amongst this "forest" of information, there are, at times, legitimate criticisms that the community should look at and determine if the criticism is one that is applicable and should be addressed, or not.

My question as to why the practice of excluding experts from the discipline is accepted among academia. I was hoping Lisa or, an academic who visits this wonderful discussion site could offer some enlightenment.
From my perch, this is how the experts and academics learn to understand each other, to respect each other's profession and perspective.

Good discussion. Thanks. Jan

Well, I been university faculty (albeit among the smallest of fishes in the pond), and I know there are strata of recognition that tend to blind one to those below (and confidentially take digs at those above). It's not, of course, unique to academics, military and police organizations being good examples. Where PhD's fit into the layering makes them naturally accustomed to students accepting pretty much anything they say, defending their intellectual territories, and graduate students trying to please them. The order of respect can turn on some rather fine distinctions - the professor of computer science who was considered second rate because his specialty was methodologies of teaching CS - or the professor of equal rank who had a mere masters degree and a real-world career behind him. Not that there weren't some very high level folks indeed who interacted completely and without prejudice with everyone.

I really don't look for the applications experts and the academics in potentially related fields to become widely familiar with each other. For one thing, there's not that much free time. Their meeting ground is often the courtroom, so a lot of the interaction can be negative. Those immersed in pure science are offended by some arguments in forensics that appear to lack scientific rigor. Application experts most often encounter the academics as benefiting by attacking (and misunderstanding) their practices. And both sides have their extremists.

This started, or at least became, a discussion of the makeup of a study group of academics. (Although I'll wager they see themselves in this role as doing a practical application of knowledge beyond the academic.) If they had occasion to consider SWGFAST by looking at the membership list, they might well comment that the membership is academician-free and therefore might be suspected of being unfamiliar with rigorous scientific inquiry. Now, I don't know the backgrounds of everyone in SWGFAST, any more than I am familiar with the body of work or active interests of everyone on the advisory panel. If one looks at just the titles, each group looks like pretty much like different tribes.

You know, I'm not at all sure it's desirable to mix the two in every sort of committee-like effort. Sometimes, a group is there to get things done according to specific goals, not to work out broad and profound issues. Wrangling is a part, probably a necessary part, of working out core principles, and scientific inquiry is not really intended to do anything but identify the next question to be answered. Maybe the realistic process is that there will be misconceptions, and the whole clunky machine grinds forward by dealing with those misunderstandings. Lots of things seem to work that way.