Would you even know? The question most training programs can't answer — and how to fix it

If your training weren't working as well as it could, would you even know?

Think about that for a moment – especially if you jumped to a “yes”, because it's harder than it sounds. Would you know which skills your people are consistently weak on? Which parts of your program are quietly failing? Which of your assessors are too easy, and which are too hard? Or would the problem simply sit there, invisible, while people you've certified as competent walk out onto the job not quite ready for it?

For most organizations, in nearly every industry, the honest answer is that they wouldn't know. And here's the useful part: the reason you can't answer those questions usually has nothing to do with effort or care. It's that the data which holds the answers is being thrown away the moment it's created. Fix how you capture it, and the questions become answerable. That's the whole argument. Everything below is just why it's true, what it's costing you, and what to do about it.

The "write-only" system

Here is how skill assessment works almost everywhere: someone observes a trainee performing a task, makes some judgments, marks a score on a sheet of paper, signs it, and files it. Pass or fail. The form goes in a binder, and everyone moves on.

This is a “write-only” system. Data goes in. Almost nothing comes back out.

Think about what that paper form can actually tell you. It records one verdict, about one person, at one moment, and then it gets filed. That is tragic. It can't tell you which skills your people consistently miss. It can't tell you whether one assessor scores systematically harder than another. It can't tell you whether this year's cohort is drifting below last year's, or whether the new module you rolled out actually changed anything. The information needed to answer every one of those questions passes through our hands, and then we let it go.

So when I ask “would you even know?”, that's the answer. You wouldn't. Not because you didn't look, but because the evidence was discarded the moment it was written down.

And it's worth being clear about what that blindness actually costs. A weak skill you can't see is a weak skill that walks out onto the job. In a hospital, a fire ground, a manufacturing line or in a cockpit that gap isn't an abstraction. It is the distance between “we assumed they were ready” and finding out, the hard way, that they weren't. Even where the stakes are less dramatic, the cost is real and compounding: people certified as competent who aren't, a program that's been drifting for three years that nobody flagged, the same training dollars spent again and again on a fix that never actually worked — all because no one could tell that it didn't. You can't correct a problem you can't see; you can't manage it if you don't measure it. And right now, most organizations can't see this one.

Why I take this personally

In the mid-1990s as a computer science faculty member at the University of British Columbia, I built the first commercially successful learning management systems. It was called WebCT and it grew to ultimately serve 14 million students across 80 countries. Millions of small acts of learning flowed through our hands every day. And sadly it never occurred to me what that data one day might be able to do for our students and education in general.

It remains one of my great professional regrets. We had, sitting right in front of us, the richest record of human learning behavior ever assembled at that time, and I never built the technology to help our users mine that data to learn deep insights about their students, their curriculum and their programs. I can't get it back. But you still can.

What changes when the data compounds

Capture assessment data in a structured, digital form (not a scanned clipboard, but real data, indicator by indicator) and it changes in character. It begins to compound.

One assessment tells you little. So does the second. But by the thousandth, patterns surface that no single observation could ever reveal. Which indicators get missed most often? That's a curriculum problem you can finally see. Which assessors run hot or cold? That's an inter-rater reliability problem, invisible on paper. Which cohorts are improving? Which interventions actually worked? None of these are questions about one person. They're questions about your system, and only accumulated, connected data can answer them.

And here's the part that matters most: the value compounds forward. The data you capture today is the only data you'll ever have about today. Anyone can start collecting tomorrow. Nobody can go back and collect yesterday.

We don't even need to know what questions we will ask of our data in the future. But there will be questions we want answers to, and our collected data will be able to answer them. The one shift that matters now is to stop treating assessment as simply a way to determine whether a student is competent. Its value is so much more than that. Incredibly much more. Most importantly it is the front end of a measurement engine that tells us about our organization and allows us to optimize for ALL future trainees; structured, digital, connected across people, skills, assessors, and time.

The day will come when you genuinely need to know whether your training is working — or worse, the day will come when it wasn't, and you'll need to understand why. When that day arrives, you'll want the evidence in hand. Not filed away in a binder where no question can ever reach it.