Skill assessment: where should the pass line go? — The Skill Lab #10

In “Skill Assessment: Is Your Critical-Fail List Still Critical? (#9)”, we looked at how a single critical fail should cap a candidate’s score just below the line required to be cleared for practice. That leaves the question I promised to take up this time: where should that line be set in the first place, and on what basis?

The question is an important one. The cut score (the grade that defines the border between a pass and a fail) decides who is cleared to perform a skill on the job and who goes back for more training. If we set it too low, we clear people who can’t yet do the work. If we set it too high, we hold back capable people, spend more time training than we need to (which has many negative consequences we can dig into in a later post), and tempt assessors to quietly “round up” a candidate they subjectively feel is ready.

So why do so many programs land on a round percentage? I think it is mostly inheritance. Nearly all of us grew up with academic grading, where 50% passes and 80% sounds demanding, so it feels natural to carry that over. The trouble is that a percentage on its own says nothing about competence. Scoring 80% on a skill sheet of ten easy indicators and 80% on a sheet of ten demanding ones are very different achievements. Most domain experts are excellent at deciding what belongs on a skill sheet. Setting the passing line is a different kind of problem, drawn from assessment science.

What should a cut score actually mean?

The best (and clearly obvious) answer the assessment field has come up with is this: the cut score should be the score of a candidate who is just competent. I don’t mean an excellent candidate or an average one. I mean the person who is only barely good enough to be trusted on their own. We call this person the “borderline” or “minimally competent” candidate. If we can describe that person clearly, we can work out what score they would earn, and that is where the line belongs. This is more nuanced than it sounds - but there are well accepted methods to help us make a reasonable approximation that can be used as our passing line.

The first is the Angoff method (named for William Angoff, who described it in 1971). In this method, a small panel of experienced practitioners pictures the borderline candidate together. Then, for each indicator on the sheet, each panelist estimates, as a percentage, how likely it is that this candidate would perform it correctly. Average those estimates across the panel and the indicators, and you have a cut score with a stated basis.

The second is the borderline group method, often used for clinical skills exams in medical education. Assessors complete the skill sheet as usual, but they also record one overall, subjective judgement of each candidate: clear pass, borderline or clear fail. After a testing cycle, we look at the scores of the candidates the assessors called “borderline”, and the middle of that group (the median score) becomes the cut score. This method has a real advantage: although subjective, it is anchored in how experienced assessors actually judged real people performing the skill.

One cut line rarely fits every skill, either. A skill where an error has serious consequences deserves a more demanding cut line than one where an error is easily caught and corrected. Note that this is separate from critical items, which cap the score below the line on their own. The cut score governs everything else: the accumulation of smaller strengths and weaknesses that together show whether someone is ready.

None of this requires new staff or new budget - just a little extra work done ideally each time a new skill sheet is created or a substantive change is made to an existing skill sheet. The great thing about doing this is that it gives you more confidence in your skill sheets and lets you defend a result, whether to a candidate who failed or to an auditor asking how you know your people are ready.

Next time, we’ll look at a closely related design decision: when a simple yes/no indicator is the right tool, and when a rating scale earns its place.

Until next time, thanks for reading.