In “Skill Assessment: Where Should the Pass Line Go? (#10)”, we looked at how to decide where the line between a pass and a fail belongs. I closed by promising to take up a closely related design decision: when a simple yes/no indicator is the right tool, and when a rating scale is preferred.
The two are linked because a cut score is only as good as the marks it is built from. Every indicator on a skill sheet asks the assessor to record something, usually in one of two ways. Either the assessor records whether something happened (yes or no), or they place the performance somewhere on a scale (1 to 5, say). Whichever we choose, the score and the pass decision inherit its weaknesses. If two assessors would record the same performance differently, the candidate’s result depends partly on who happened to be holding the clipboard.
What does a rating scale actually record?
Imagine an indicator that reads “Communicates clearly with the patient”, scored from 1 to 5. One assessor watches a paramedic student introduce themselves, explain what they are about to do and check that the patient understood, and gives a 4. A second assessor watches the same performance and gives a 3, because in their mind a 4 is reserved for someone who also settled an anxious family member. Neither assessor is wrong. They simply carry different pictures of what a 3 and a 4 look like, and the scale gives them no way to reconcile those pictures. Rubrics, of course, can help, but they are often ignored or at least taken as guidance rather than adhered to strictly.
This is a well-studied problem. In a large study of workplace performance ratings (Scullen, Mount and Goff, 2000), the raters’ own tendencies accounted for 62% of the variation in the ratings, while the actual performance of the person being rated accounted for only 21%. Those were ratings of managers rather than of practical skill, but the lesson carries over: a rating can tell us as much about the person giving it as about the person receiving it.
Having said that, rating scales have a real place. Some qualities really do have subjective aspects - leadership potential is a good example. For those, a scale (perhaps alongside some checkboxes) is the right tool, but it needs a lot of data from many assessors, and we need to be clear about what those data tell us. Averaged across a body of assessors, the scale tells us whether those assessors generally believe the candidate has leadership potential. It does not tell us whether the candidate actually has it. That can still be very useful to know, as long as we read it for what it is.
Why lean toward yes/no?
For practical skill, I think the yes/no indicator should be our default, because we are much more likely to agree on whether something happened than to fully digest a rubric and then agree on where a performance sits on a scale. In the past, checkboxes may have been avoided because calculating a score over a large number of granular checkbox ratings could have been error-prone, especially for complex tasks. However, with new skill assessment technologies, the calculation problem is taken out of your hands and done reliably by technology.
Having said that, the research on checkboxes versus ratings is more balanced than you might expect. A systematic review of simulation-based assessment in the health professions (Ilgen and colleagues, 2015) found that global rating scales held up better across different tasks and may capture more of the nuance of expertise. But agreement between assessors was about the same for both formats (if anything slightly higher for checklists). So a well-written yes/no indicator is likely to at least slightly improve agreement, and also tells us something a scale never can: exactly what the candidate did and did not do. That makes the result easy to explain to a candidate, easy to defend to an auditor and easy to compare across assessors and across time. Checkboxes also more directly force an organization to discuss and agree upon what constitutes valid performance, and can be much more informative in understanding which parts of our training are working and which parts are not.
The catch is the phrase “well-written”. A yes/no indicator only earns that agreement if it describes something an assessor can see happen. “Performs an adequate patient assessment” with a yes/no box is really a two-point rating scale in disguise, with all the disagreement of one. “Communicates clearly” is not observable as written, but “Introduces self by name”, “Explains the procedure before starting” and “Confirms the patient understood” are. Very often a scaled item is several observable events that simply haven’t been separated yet.
What does this mean for us when we are writing our assessment? The few judgments that are truly subjective should keep their scale, with ratings gathered from as many assessors as possible and read as those assessors’ considered opinion. But for nearly everything else, checkboxes provide utility that no subjective rating can: exactly what was done, exactly what was missed, and a guarantee of a consistent calculation to derive a score. The reward is data that mean the same thing no matter who recorded them.
Until next time, thanks for reading.
Murray Goldberg was a tenured faculty member in Computer Science at the University of British Columbia, where his research into learning technologies led him to create WebCT in 1995 - the first widely-used learning management system in higher education, eventually serving 14 million students in 80 countries. He has spent three decades working to advance the science and practice of learning and assessment, and is the founder and CEO of SkillGrader.