Skill assessment: is your critical-fail list still critical? — The Skill Lab #9

How many critical-fail items are on the skill sheets our programs use today, and how many were there ten years ago? In many programs the list is longer now, and it did not grow because anyone decided it should. It grew one (seemingly) reasonable addition at a time. There was a near miss in training, a finding from an accreditation visit, an incident report that circulated through the department. Each time, a group of experienced people concluded that a particular step should never be missed again. A critical item decides whether someone is cleared for practice, and a list that decides badly costs us in safety, in fairness to candidates, and in the trust assessors place in their own forms.

Critical items exist for a good reason. Most of a skill sheet is “compensatory”: strength in one area can offset weakness in another. But some errors should not be offset by anything. Good patient care elsewhere in a scenario cannot make up for never checking for a pulse. Skill on the fireground cannot excuse a facepiece that was never sealed. A short list of such items is one of the most valuable things a skill sheet can carry.

But there is a cost, and it shows up in three places.

First, there is what happens to the score. On many forms, a critical fail drives the result to zero. Consider a candidate who performed excellently apart from one missed step, and another who struggled from start to finish. In the data they now look identical, even though they need very different help. That’s a problem because these are two very different candidates.

Second, there is the arithmetic. Even well-written indicators can produce the occasional borderline judgement call. Suppose each of ten critical items carries a 3% chance that such a borderline call is made against (rather than for) the candidate. The chance that at least one of them is called is then roughly 26% (the math may not be immediately obvious - but it is correct). That means about one candidate in four could now fail on probability alone - despite being clearly qualified on 9 critical indicators, and “possibly” qualified on the 10th. (These are generous assumptions. A vague item such as “demonstrates unsafe practice” carries far more than 3%.)

Third, assessors notice. When the penalty feels out of proportion to a borderline error, sophisticated, experienced assessors will often decline to mark it. They do so quietly and with good intentions. Over time, the list stops reliably catching the very things it was built for.

What can we do about it?

The first step is an audit of the list. For each item we need to ask two questions: Would this error, on its own, cause or seriously risk harm, or invalidate the outcome? Is it written as an observable event that two assessors would agree took place? An item that fails either test still matters. It simply belongs somewhere else, such as a heavier weighting or a minimum required score within its domain.

The second step is to rethink how the score is built. Positive observations build a candidate’s potential score. Critical items place caps on it. A single critical fail should cap the result just below the threshold required to be cleared for practice, and each additional one should lower the score still further. The cap always overrides potential, so no strength elsewhere can lift a candidate over the line. But this approach to scoring, while ensuring that a candidate who is not ready is not put into practice, can still tell you the difference between one candidate who has one shortcoming and another candidate who has many shortcomings.

A short critical list that assessors enforce without hesitation is worth far more than a long one they have learned to work around. Next time, we will look at where that threshold should be set in the first place, and on what basis.

Until next time, thanks for reading and keep well!