We asked assessors whether they trust their own results. Here is what they said.

If you assess skills for a living you have probably considered the following question: how much confidence do you have in what your completed assessment forms are actually telling you? If it has not occurred to you, perhaps it should.

I am not asking about confidence in your assessors or about confidence in your rubrics. I specifically mean confidence in the results – the scores, the comments, the accumulated stack of paper that is intended to represent the competency of your workforce. Could you rely on that stack to tell you whether your training is improving? To identify a skill gap before it becomes an incident? To defend a promotion decision?

A few years ago, we had the opportunity to put that question directly to a group of sophisticated, experienced assessors at a highly professional naval training organization. These are people responsible for evaluating skills as demanding as ship handling and inshore navigation, using conventional paper-based assessments. Their answers were revealing (and honestly quite eye-opening for me), and I suspect they will sound familiar no matter what industry you work in.

What the assessors told us

Fewer than 50% of assessors believed their assessments could reliably identify trends or gaps across many assessments. Fewer than 30% felt they could say whether their training was more or less effective than in the past – let alone why. Fewer than 40% believed they could reliably rank trainees, past and present, for a given role. And only 40% agreed that their assessment practices were impartial and data-driven, with a similar number believing their trainees saw them that way.

These were not careless assessors or a weak program. Quite the opposite – this was a serious, safety-conscious organization with experienced evaluators and well-developed procedures. And still, the majority of the people doing the assessments did not trust the results to answer the most basic questions assessment exists to answer.

We go to so much trouble, for good reason, to train people in the skills they need to perform efficiently and safely. But if the assessors themselves feel they cannot rely on the results of their assessments, why are we collecting them?

Why paper fails – even when everyone does their best

The problem is not the people. The problem is the medium and the process it limits us to. Paper-based observational assessment has three fundamental flaws, and no amount of diligence fixes them.

First, paper assessments are subjective – even in the presence of good rubrics. A rubric on a clipboard cannot enforce consistency. Two assessors watching the same performance will interpret “meets standard” differently, weight criteria differently, and be influenced (entirely unconsciously) by different things. The score that lands on the page reflects the performance and the assessor, blended together in unknown proportions. That is exactly what the survey numbers above are telling us: the assessors knew their scores carried this noise, and calibrated their confidence accordingly.

Second, the data paper captures are coarse and ad-hoc. A checkbox and a hastily scribbled comment is a very thin record of a rich performance. Which specific indicators were missed? In what conditions? Compared to whom? The information existed for a moment, in the assessor's head, and most of it evaporated before the pen hit the page.

And third – even if the first two problems didn't exist – paper offers no practical way to turn results into insight. The completed forms go into a filing cabinet, and that is where the story ends. Aggregating hundreds of paper assessments to spot a trend is a heroic manual effort that almost no organization will ever perform. So the questions that matter most: Is our training working? Where are our gaps? Who is genuinely ready? They simply go unanswered.

Why this matters more than it seems

Reliable assessment data is not one nice-to-have among many. It is the foundation of everything that flows downstream.

Reducing operational risk depends on knowing where the competency gaps are. Improving training depends on measuring whether it works. Controlling costs without sacrificing quality depends on knowing what you can safely streamline. Fair promotion and certification decisions depend on defensible records of demonstrated ability. Every one of these rests on the same foundation – trustworthy skill data. If the foundation is unreliable, everything built on it is guesswork.

Simply put: an organization that cannot trust its assessment results is flying blind on the very questions that matter most.

So – what do we do about it?

The encouraging news is that this is a solvable problem, and solving it does not require heroics. It requires structure. When assessments are captured digitally against well-defined indicators, subjectivity shrinks because every assessor is guided through the same criteria the same way. The data becomes granular rather than ad-hoc, because each indicator is recorded, not summarized from memory. And because the results are structured data rather than ink, the analysis that was impossible on paper – trends, gaps, comparisons, training effectiveness – becomes available at the press of a button.

The assessors we surveyed were not wrong to doubt their paper results. They were being honest. The question for the rest of us is whether we are willing to be equally honest about our own – and then to do something about it.

Until next time, thanks for reading and keep well.