I FAILED THE IAI RE-CERTIFICATION TEST

[Carl Speckels]

Woah, woah, woah. Let's all take a step back, and slow the roll a bit.

Certification is a worthwhile endeavor and should continue to be encouraged. Just because you don't believe that the IAI Latent Print Certification program isn't perfect or could be improved doesn't mean that everyone should turn in their card, cease requiring it for employment, cease encouraging or rewarding people for certification, or any other knee-jerk or reactionary response. The IAI's latent print certification program is not fundamentally broken. It has been improved over time from having no re-certification test, to having a super-easy re-certification test, to having the current test. I do believe that each step has been an improvement, but that further improvement is necessary, AND that further improvement will occur. Why bail on the program when improvement is virtually certain? We know the people on the board. They're dedicated members of the IAI and dedicated to the latent print field. Of course we can have the necessary conversations to find ways to improve.

Administrators can still encourage certification without making it a condition of employment, rank, pay, promotion, etc. Actually, having no IAI certification condition is very common (FBI, USACIL, several other federal, state, and municipal labs, including yours I believe, do this). I've queried dozens of individuals from dozens of agencies about that very thing and the overwhelming majority do not require certification for anything. Some used to and then changed their policy (LVMPD for example). The problem with making IAI certification a requirement is that agencies are putting their trust in a test that they have zero control over, AND have no way of reviewing if the questions are 1) relevant to the type of work the examiner is doing, e.g. many of the written questions on the cert test are irrelevant to the work that the test-taker is actually performing, and 2) are consistent with the agency's policies/SOP's. Take your exclusion policy at AZDPS, some of the test impressions that have an expected "correct" answer of Exclusion may not meet with your own agency's policy/SOP for Exclusion...but you'll never know that because unlike CTS and FA, your agency will never have the opportunity to see or review the test materials. So, encourage and promote IAI certification, but avoid the administrative headaches by not making it a condition.

The other problem, right now, is that the RE-cert test has apparent associated flaws, at least it would appear so given the 9% failure rate. Administrators have to ask themselves if it makes sense to place conditions on a test (the initial cert test) that appears to be demonstrably valid (meaning that those that are properly trained tend to pass) only to have 9% of their successful certified examiners (those that you would expect to pass, given that they passed it before) fail the "same" test 5 years later. The administrator will invariably be confronted with this problem...one of their examiners just failed the RE-cert test, now, do they get fired, striped of pay, demoted? Why would an administration invite that kind of liability? Why would an administrator put such an emphasis on this one uncontrollable metric over all of the other measurable metrics that their employee has demonstrated like, accreditation compliance, MQ's to include college degrees, competency/proficiency testing, experience and performance records, etc. etc... Even if that same examiner has satisfied every other measure of demonstrating that they can properly perform their duties, if IAI cert conditioned, that administrator may be forced to terminate, demote, reduce pay, or remediate.

No one is saying that the LPCB has maliciously created a test with the intent of revoking certifications but that's what they have right now; a test that results in the revocation of 80 certified examiners per 5 year cycle. And, the members' dedication to the IAI or the field have nothing to do with the amount of reliance or conditions that administrators should put on the IAI test. It appears that the IAI LPCB is in no hurry to correct/modify/change their test/RE-cert test. Having seen an appeal response and the responses to my own emails, they are content with their "high pass rate". And yes, those on the LPCB (seven members) are part of the certified community, and yet, 80 of their certified peers, per 5 year cycle, will have shown them that their test includes samples with inordinately high error rates. Why is the opinion of the 7 greater than that of the 80? The LPCB was selected from the same pool of certified examiners. So, why?

You suggested some good, potential improvements to the current test and hopefully the LPCB will hear you and the others on this chat board. But until then, I don't agree that administrators should slow their roll. In fact, I think they should sprint to their P&P manuals and change any condition relating to promotion, rank, or pay to avoid the impending headaches because certifications are being revoked at a relatively rapid rate. Administrators can avoid the tough decisions that will come with IAI certification conditions by never making certification a condition in the first place. Encouragement is good.

[BuckOFive]

I agree with ER, we don't want to throw the baby out with the bathwater. I think having certification is a nice thing to have for your agency, the jury and yourself. When I am looking at potential employees I want to see that they are professionals and display that by belonging to a professional organization. Certification is a piece of your CV. It has to be one of the components that make you the good examiner you are. We know that just because you don't have certification, doesn't mean you aren't a fine examiner and vice versa. I think the LPB needs to take notice of this issue and start to correct the problems.

[NRivera]

From the LPCB Chair in the current ID News:

It is worth noting that in a testing environment, it is important to have a definitive conclusion in order to be able to properly measure the ability of the test taker against known ground truths. Each latent print in our recertification test has been deemed to be sufficient to render a conclusion.

Emphasis added by me.

There is that pesky little word again: "sufficient". It seems that up to 9% of certified examiners disagree with the LPCB as to how much is "sufficient". I understand that this issue is on the agenda for the board meeting during this year's conference. I can't be there but I hope that room is packed and a common sense solution can be reached.

[josher89]

From Stephanie Howard in the recent IDNews (page 8, July/August/Sept issue):

Therefore, we made a determined effort to ensure the new recertification test was not only more challenging, but more representative of actual casework. This is why the newest incarnation contains an increase in the number of latent prints, the addition of possible exclusions, and the requirement to render a conclusion for all of the comparisons on the test.

I guess what this means is that in actual casework, we cannot render an inconclusive decision; that all decisions must be ID or EXC. Huh, that's really interesting.

While I don't use it a lot, I certainly use inconclusive in casework. I don't like to use it but I know there are times when, for me, it's the only reasonable decision. SWGFAST (and I'm sure OSAC FRS) gave us varying degrees of inconclusive (lack of comparable areas, lack of sufficiency for ID and lack of sufficiency for exclusion). For those groups of really smart people to recognize the value of an inconclusive decision, and for the LPCB to fail to acknowledge that breadth of knowledge is lost on me. We added a fourth category; unable to locate. It's for those latents that you search and search and search and no matter how "on" you are, you just can't find it and exclusion isn't a reasonable or demonstrable decision. Again, not a favorite decision, but sometimes you just can't find it!

OK, so they give us an out by signing something that says we are opting out of making a conclusion that we aren't comfortable with. Thanks! Oh, but the caveat is we still fail the test. Another, huh, that's really interesting.

They printed the certification stats from 2005 on in this issue; will they provide the re-certification stats for the same years?

When people ask me about certification, I will still tell them it's not a bad thing to strive for but I will make sure to say that it doesn't make you a better examiner than if you don't have it. And, it shouldn't be a requirement to have but maybe a perk for your job if you have it.

[ER]

It seems that up to 9% of certified examiners disagree with the LPCB as to how much is "sufficient".

Actually, no. Latent print comparisons involve two separate skill sets. The Search and Sufficiency. This situation suggests that those 9% "failed" in the search. This doesn't necessarily mean that they disagree as to how much is sufficient.

[NRivera]

It seems that up to 9% of certified examiners disagree with the LPCB as to how much is "sufficient".

Actually, no. Latent print comparisons involve two separate skill sets. The Search and Sufficiency. This situation suggests that those 9% "failed" in the search. This doesn't necessarily mean that they disagree as to how much is sufficient.

The resounding complaint is that examiners are being forced into an ID or Exclusion when they feel an inconclusive would be more appropriate. That goes to sufficiency. Granted, not all the failures are due to that circumstance, hence why I said "up to 9%". I wasn't trying to dig into the cause for all the failures there.

I do like how you are thinking in terms of the skill sets required to do the job. We can likely debate at length over what those are, whether or not they should be tested in a certification/re-certification test and how that could best be accomplished. That may even give the board some ideas going forward.

[Bill Schade]

Now we have gotten to the core issue (in my opinion) Search ability vs what is sufficient and why.

Historically the Certification test has been a measure of search ability. (can you find the tough ones) That remains the case and the cert board seems to feel this is fine going forward.

However, as AFIS technology does a better job of helping us "find" the match (or identification or inference or whatever the term of the month is) I strongly believe that examiners will become less proficient in search and find ability. Has anyone been to the store lately and said, "wait, I've got the penny" to the cashier, after the register tells them how much change to give back? The blank look and the finger counting that results (from both young and old cashiers) tells me we have lost the ability to properly calculate change in a transaction. I strongly believe that this will happen to examiners and I think that is what happened to me. My regret on my failure of the recertification test is that I didn't enter the knowns into our AFIS and then search the ten latents. Thats how we do casework, it's how I should have done the test.

Perhaps the time has come for the certification test to evolve into measuring examiners overall ability, not just can you find it. I suggested that earlier in this thread and no one took up the discussion. Maybe you agree with me and didn't realize it.

[ER]

The resounding complaint is that examiners are being forced into an ID or Exclusion when they feel an inconclusive would be more appropriate.

Right, but my point is that if the 9% of examiners were to be shown exactly where to look, then (I'm willing to bet) that almost all of them would reach an identification decision in these comparisons. Therefore, it's more of a searching issue than a sufficiency issue.

I suggested that earlier in this thread and no one took up the discussion.

I did see it. I just agreed and thought you had made the point sufficiently.

One extra point though. If the test evolves to test both "Can you find it?" AND "Can you call it?", or even just starts including the latter, then the test becomes much harder as the test is made up mostly of "close to the line comparisons". Now, I've only taken the old (super-hard) cert test in 2009 (before they were ground truth examples, and the old (easy) re-cert test, but from what people are describing, it seems that the new cert and re-cert tests involve searching, but are fairly straight-forward when it comes to sufficiency measurement during Evaluation. While I would personally like to see a more realistic test that has both "easy-to-find-hard-to-call" and "hard-to-find-easy-to-call" comparisons, I don't think that this would improve the pass rate.

[Carl Speckels]

The resounding complaint is that examiners are being forced into an ID or Exclusion when they feel an inconclusive would be more appropriate.

Right, but my point is that if the 9% of examiners were to be shown exactly where to look, then (I'm willing to bet) that almost all of them would reach an identification decision in these comparisons. Therefore, it's more of a searching issue than a sufficiency issue.

ER, Why would you be willing to bet that when you just wrote that you haven't taken the test? The one that I know I missed, the one with the associated 7.6% erroneous exclusion rate, I believe would have tested many examiners sufficiency threshold, even to compare, when considering the associated distortion. Many people over-extend their conclusion thresholds in testing environments (we see it all the time on CTS and FA proficiency tests), conclusions that they wouldn't have made in actual casework. Why would you assume the IAI RE-certification test is any different? Some agencies have point thresholds for ID and sufficiency thresholds for Exclusions and I would be curious if this particular impression would have satisfied all of them. I'd be willing to bet that it wouldn't have. Obviously, I can't get into the specifics of the latent itself but maybe your bets would be better placed on things you've actually seen and know about.

[ER]

Why would you be willing to bet that when you just wrote that you haven't taken the test?

I was basing that on your description of the test. I must have misunderstood. From the previous descriptions from you and others of the test, I assumed that you had excluded because you had not found any areas of similarity. You were uncomfortable with the exclusion decision, wanted to say inconclusive, but were forced into exclusion by the format of the test.

Was this the case? Or did you find some similarity, consider it insufficient for an ID, wanted to say inconclusive, but were forced into exclusion?

I assumed it to be the first case (which would mean a comparison with a difficult searching component).
If it's the second case (which would mean a difficult sufficiency determination component), then I stand corrected.
If it's both (difficult to find AND difficult to determine sufficiency), then I still believe that most examiners had their issue with the searching component because a) it comes first; and b) it's a skill that examiners don't have to practice as often (as Bill previously stated).

Many people over-extend their conclusion thresholds in testing environments... Why would you assume the IAI RE-certification test is any different?

That's exactly my point. If examiners HAD found the correspondence during the search, then they probably would have called it even if the sufficiency was low because of this exact phenomenon. This led me to assume that most of the misses in that example were due to a failure to locate the similarity in the first place, i.e. The Search.
Again, anyone who's taken then test, please correct me if I'm wrong here.

maybe your bets would be better placed on things you've actually seen and know about

C'mon. I've seen a lot of shitty latents and trained a lot of people on how to avoid erroneous exclusions. With respect, I know what I'm talking about.

[NRivera]

I was basing that on your description of the test. I must have misunderstood. From the previous descriptions from you and others of the test, I assumed that you had excluded because you had not found any areas of similarity. You were uncomfortable with the exclusion decision, wanted to say inconclusive, but were forced into exclusion by the format of the test.

Was this the case? Or did you find some similarity, consider it insufficient for an ID, wanted to say inconclusive, but were forced into exclusion?

I assumed it to be the first case (which would mean a comparison with a difficult searching component).
If it's the second case (which would mean a difficult sufficiency determination component), then I stand corrected.
If it's both (difficult to find AND difficult to determine sufficiency), then I still believe that most examiners had their issue with the searching component because a) it comes first; and b) it's a skill that examiners don't have to practice as often (as Bill previously stated).

ER I get where you're going with this, but I don't think we have enough information to make that assumption. It is definitely possible, you have to be able to find it in order to make the call, but you're ignoring the possibility that they analysis may be flawed as well. If, for example, you assess the latent as a left hypothenar when it's actually a right inter-digital and you never look there, you'll never find it. The same could happen if your L2D analysis is inaccurate. That's just a wild over-simplification but it could happen. This is the reason I would be in favor of greater documentation being provided by those taking the test. I just don't think there is enough information to pin it on a lack of searching skill.

I'm looking at it from a conflict resolution perspective. As was mentioned elsewhere, often times when a verifier disagrees with the original analyst it becomes a matter of "Did you see this?" and the original analyst responds with "Duh! There it is." I don't see that as a a conflict. When we decide we need to stand our ground because we feel the data is there to back up our decision is when it become a conflict. In either case we first need to see what each person is looking at and using as a basis for their decision. (Show me the data anyone?) The decision should emanate from the supporting data, if the decision is incorrect then it points to a flaw in the data being relied upon or its interpretation. This, IMO, is where the real expertise lies. Anyone can play matching games, but the real expertise of the examiner lies in the ability to discern what reliable data is present in the questioned print to begin with and then looking for that agreement or lack thereof in the known.

[Carl Speckels]

Why would you be willing to bet that when you just wrote that you haven't taken the test?

I was basing that on your description of the test. I must have misunderstood. From the previous descriptions from you and others of the test, I assumed that you had excluded because you had not found any areas of similarity. You were uncomfortable with the exclusion decision, wanted to say inconclusive, but were forced into exclusion by the format of the test.

Was this the case? Or did you find some similarity, consider it insufficient for an ID, wanted to say inconclusive, but were forced into exclusion?

maybe your bets would be better placed on things you've actually seen and know about

C'mon. I've seen a lot of shitty latents and trained a lot of people on how to avoid erroneous exclusions. With respect, I know what I'm talking about.

This is where I, and any others that have taken the test, are at a disadvantage, because we cannot reveal specifics about the test, the latent, how it was searched, etc. I’m not questioning your experience ER, and I didn’t say that you don’t know your business. I’m sure that you’ve seen tons of latents at all difficulty levels…but you haven’t seen “this” latent, the latent bearing the 7.6% associated error rate. Along with my RE-cert test, and as NRivera suggested in an earlier post, and albeit atypical for the RE-cert test, I submitted a comprehensive overview of my analysis for this particular latent, expressing its limitations and my desired conclusion. I don’t know how other people would have concluded on this particular latent had they been shown its location. But, even knowing its location, I do think that it could have challenged the conclusion thresholds of any examiner and especially those that treat these types of tests as they would in casework or those that operate under a quantifiable L2 threshold for ID. In my opinion, that’s a problem. And it’s a problem that is self-evident by the number of those that didn’t find it or didn’t call it and consequently failed resulting in cert revocation. It’s easy to take the “wouldn’t happen to me” position if it hasn’t happened to you. Maybe you’re one of the greatest examiners in the world and it won’t. But that’s not the measure that certification and certification maintenance should be about. 80+ certified members, almost 10% of the overall certified membership, all previously in good standing with the IAI, will lose their certifications over this 5-year cycle. Something is wrong with that picture. And if you’re unlucky enough to get the more challenging of the two RE-cert tests (I believe there are two), the one bearing the 7.6% associated error rate, then your odds for failing just went up. That’s also a problem.

[ER]

Please understand. I'm not taking the "it wouldn't happen to me" stance. Not at all. My concern arises from the fact that erroneous exclusions WILL happen to me and every other examiner.

This portion of the thread started from NRivera's comment that up to 9% of examiners disagree with the sufficiency of this latent print. But if examiners fail to find any similarities in a same-source comparison, then their error came from the Search. I know he qualified this number with an "up to". The point that I'm trying to make is that the sufficiency question in same source comparisons and whether different examiners might disagree on sufficiency in this example is not discernible by this 9% number.

If an examiner never found the similarities, then they couldn't have any measure about the sufficiency of those similarities. Even if the latent is on the low end of what examiners would deem suitable and/or sufficient, the search comes first, and if the search fails to locate similarities, then the sufficiency question isn't the issue. I don't understand why this is a controversial stance.

In my personal experience in erroneous exclusions, almost all examiners making this error, when shown the proper orientation, location, and a corresponding target group will find sufficient similarities to identify. Assuming that this comparison is similar to most comparisons that produce erroneous exclusions (and 9% fits as a pretty average example of that), then I believe that it is safe to assume that the issue was in the search and that most examiners would identify when shown where to look. I don't mean this as a criticism of the examiners that missed this comparison. Like I've been saying for years now, it happens all the time to everyone. We need to adapt policy and practice to that reality.

I think we're agreed that a single erroneous exclusion shouldn't result in a test failure. If we can agree on that point, then I think that the test can be adapted and evolved to deal with the reality and inevitability of erroneous exclusions.

[CarrieB]

If an examiner never found the similarities, then they couldn't have any measure about the sufficiency of those similarities. Even if the latent is on the low end of what examiners would deem suitable and/or sufficient, the search comes first, and if the search fails to locate similarities, then the sufficiency question isn't the issue. I don't understand why this is a controversial stance.

There is no way to know if it was a sufficiency or a search problem. Assuming all of the regions were searched, probably multiple times by all of those who ended up excluding this latent, perhaps the quantity of similarities that were seen in the “correct” location were not all that different than the quantity of similar features seen in incorrect locations. This sounds impossible in a test situation, I understand, unless we are discussing latents that are very borderline comparative value and lack sufficiency and specificity. These are latents that have no place in a testing environment with the conclusion options that are currently employed. If these latents are seen as acceptable, then inconclusive has to be an option. Otherwise, the latents need clear, discreet answers.