Did SWGFAST just create a new Evaluation without realizing?

[Boyd Baumgartner]

POSITION STATEMENT ON THE ROLE OF AFIS RANKS AND SCORES AND THE ACE-V PROCESS
Standard AFIS ranks and scores cannot currently be considered a mathematical model for assessing the likelihood or probability that a subject deposited a particular friction ridge impression. The purpose of this document is to clarify that AFIS ranks and scores have no role in formulating and stating conclusions based on ACE-V.
Ranks and scores are mechanisms of an AFIS system that provide information about a particular search relative to prospective candidates whose impressions are contained in the AFIS database. They provide possible matching candidates as determined by automated search mechanisms to an examiner.

AFIS does not replace the human expert role in Analysis, Comparison, Evaluation, or Verification (ACE-V). It is inappropriate to use AFIS scores in lieu of Evaluation or Verification decisions. If testifying in court to having followed the ACE-V methodology in response to a “lights-out” AFIS identification, the examiner must have performed a traditional ACE examination separate from the system determination. Additionally a second examiner must have conducted a Verification. This would also apply to searching latent prints in an AFIS database. A latent print examiner cannot use an AFIS system result as the verification step of ACE-V.

According to SWGFAST's most recent currently available document regarding Standards for Examining Friction Ridge Impressions and Resulting Conclusions and this position statement, it appears SWGFAST has created a new conclusion, but doesn't define it anywhere.

http://www.swgfast.org/documents/method ... chived.pdf(There are two more recent ones but one is not linked correctly on the site and one is pending approval)

Let me 'splain (insert gratuitous Ricky Ricardo accent)...

babaloo.jpg

According to the position statement, 'lights out' conclusions are not recognized as being part of the ACE-V process. One of the recognized Evaluations of ACE-V, per the documents of SWGFAST is Individualization. Therefore, an Individualization by definition cannot be reached without having been arrived at through ACE-V. 'Lights Out' systems per the position paper, do not follow ACE-V and therefore cannot by SWGFAST standards, claim to have reached an Individualization.

So, the implication here is that there is a new undefined Evaluation. An Evaluation that occurs when the probability of two rolled impressions have reached an algorithmic standard of sufficiency to be called a match, even though AFIS by SWGFAST's own position cannot be used in that manner. So, in SWGFASTian logic, this means that 'lights out' cannot be used, but if it is (which it can't be), you have to disregard it and use ACE-V, but the only time you'll run into this is when you testify to an individualization.

So the second round of implications are that criminal history records maintained using a 'lights out' system are invalid because they have not been individualized and that any law enforcement agency that has a latent unit and a ten print unit where 'lights out' evaluations occur but are never testified to can not claim to follow SWGFAST, because they are out of compliance with recognized Evaluations.

It's rather like a snake eating it's own tail (hat tip: August Kekulé)



Who knows, maybe the yet to be published document will address this issue and I can add this maniacal rant to my manifesto...

[Boyd Baumgartner]

I should also probably ask what, if any implications this has for likelihood or probability models that attempt to validate themselves using AFIS algorithmetically mapped feature sets. After all, ranking and scoring are a function of feature sets derived through an apparent probabilistically unreliable method...

[Pat]

Let me 'splain (insert gratuitous Ricky Ricardo accent)...

I would hate to claim better expertise in Ricardo-speak, but shouldn't that have been "lemme 'splain?"

To the reason for the SWGFAST position statement, it was reported that some agencies were using AFIS as a verifier, or a high AFIS score as indicating that no verification of an ident was needed. Apparently, in some agencies, an examiner would make an ident to a named suspect, following which the latent would be run through AFIS. If it "hit" on the suspect already identified, that was taken as "verification."

Some examiners were also apparently using a large difference in score value between the number 1 and number 2 candidates as a sort of probability statement.

SWGFAST felt that a high AFIS score cannot be construed as a high probability because no validation study has been done to show that is a reliable use of AFIS scores. Likewise, the use of AFIS as the verifier of an identification is a perversion of the verification phase of the comparison process.

Until a validated probability model comes into existance, no statement should be made about the probability of an ident from numbers (AFIS scores) that have not been validated for that purpose. And no mechanical indication of an identification should be substituted for the verification conclusion of a second human expert.

[Boyd Baumgartner]

Vitavetavegitastic, Pat. Apparently I'm a Babaloser (I promise, no more)....

Thanks for the clarification and context, and I have two points of follow up to what you've said.

1) The agencies doing what you described were already not following SWGFAST. SWGFAST definition of Verification per their latest terminology doc states:

Verification
The independent application of the ACE process as utilized by a subsequent examiner to either support
or refute the conclusions of the original examiner; this may be conducted as blind verification. Verification
may be followed by some level of review as specified by agency policy.

No AFIS system could ever be mistaken for a 'subsequent examiner' and if some agency did make that leap of logic, how exactly would an AFIS system ever refute the original examiner's individualization? What if the card on file in the database was junk?

I'd say what you have in that case is the agency committing the equivalent of comparison dry labbing (ACE-Holing?! Perhaps I should create an Urban Fingerprint Dictionary. Take that Steve and Michele!) In any regard, again Ethics enters the picture on the part of the Agency(ies) who were doing that and as equally as interesting, why would SWGFAST feel the need to clarify a position that was in response to a dilemma that was already outside of their standards?

2) This statement still gives weight to what I was saying before:

AFIS does not replace the human expert role in Analysis, Comparison, Evaluation, or Verification (ACE-V).

10 Print lights out systems, in fact do exactly that (replaces the human expert). SWGFAST is no longer latent only, so they have to account for the reality that agencies with those systems exist. I still stand by the implications I spoke of in the first post.

What say you?

[Kasey Wertheim]

One small point of clarification - SWGFAST isn't proposing that lights-out systems aren't useful or are operating outside of standards. Only that they aren't following ACE-V. SWGFAST is clarifying that no portion of ACE-V is automated. So lights-out results aren't ACE-V and an examiner testifying to a lights-out only result shouldn't testify that ACE-V was followed. Likewise, if an examiner is going to testify to a human examination, they should have it verified by another examiner to testify to ACE-V. Hope this helps 'splain things.

[Boyd Baumgartner]

Thanks Little Ricky!