Flying blind: the hidden cost of not measuring skills

Somewhere in your organization right now, a skill gap exists that you do not know about. And that gap is a serious risk.

It may be in a nurse who completed her certification training two years ago and has drifted in one or two critical procedures since then. It may be in a firefighter who passed his practical assessment — but just barely, and the assessor who signed his form was behind on paperwork that day. It may be in a technician, a driver, a surgical resident, a new hire on the floor. The gap is there. The training records say everything is fine. And you have no way of knowing the difference — because the records were never designed to tell you.

This is not a small problem. And here is the part that I find most striking: it is almost entirely preventable. Not with more training hours, not with a complete overhaul of your program, and not with an enormous investment in infrastructure. The organizations that have solved this problem did so by changing one fundamental thing — they started measuring what their people could actually do, not just whether they had attended a course.

We know this works

We have seen it work, in some very visible places.

Commercial aviation has reduced its fatal accident rate by more than 99% — from roughly 40 accidents per million departures in 1959 to around 0.1 today. How? Not because pilots got smarter. One of the most important enablers of aviation's safety revolution was the widespread use of flight data recorders, which allowed investigators and regulators to systematically analyze what was actually happening during flights and identify risks that would otherwise have remained invisible. Patterns emerged that no one had seen before, because no one had been looking systematically. Near-misses became learning opportunities rather than events that quietly disappeared. The data got better, and so did safety. Of course, embedding recorders across an entire industry required regulatory mandates and enormous institutional investment. But the insight is available to everyone.

In surgery, complications dropped by 36% and deaths fell by 47% — the result of a single intervention. In 2008, Dr. Atul Gawande and a WHO research team introduced a structured surgical safety checklist — a single page, a few minutes to complete, used to verify that critical steps were actually being performed in the operating room. The insight was almost uncomfortable in its simplicity: surgeons already knew the steps. The problem was that no one was systematically checking whether the steps were being done. And notably, the checklist itself cost essentially nothing — the transformation came entirely from the discipline of structured observation.

In 2002, baseball's Oakland Athletics had a payroll of $40 million — 28th out of 30 teams, roughly a third of the New York Yankees' budget that year. Yet they won 103 games, set an American League record with a 20-game winning streak, and tied for the most wins in baseball. The difference was rigorous measurement — analysing what actually predicted winning rather than relying on the gut feel and tradition that had always driven player selection. Better measurement, it turned out, was worth far more than a bigger budget.

In field after field, the same pattern. The organizations that measured what was actually happening — rather than what they assumed or hoped was happening — were the ones that improved, caught problems early, and avoided preventable failures.

What are the risks, specifically?

There are three worth naming clearly. The first is the safety risk — which we have already touched on. A skill gap you do not know about does not surface until something goes wrong. Paper-based checklists completed by a busy assessor tell you the form was filled out. They cannot tell you whether the performance was sound or dangerous. There is no data to act on, and no early warning before the incident that makes the gap impossible to ignore.

The second is the audit and compliance risk. Regulators are increasingly asking not just whether training happened, but whether competency was actually verified — and whether you can prove it. Paper records that cannot be analyzed, that show inconsistent scoring across assessors, or that go missing when you need them most, are not a defense. They are, in fact, an exhibit.

And finally, there is the financial risk — perhaps the most underappreciated of the three. Incidents, liability exposure, remediation costs, turnover driven by undetected underperformance — these are expensive. Far more expensive, in almost every case, than the cost of measuring properly in the first place.

This is now accessible to everyone

The good news — and there genuinely is good news here — is that this is now accessible to everyone.

For a long time, the kind of structured, data-driven skill assessment that transformed aviation and medicine felt like the exclusive province of large institutions with large budgets and large regulatory mandates. And it is true that building the infrastructure those industries built required exactly that. But that is no longer the case. Technology has changed the equation entirely. Today, any organization — a community college allied health program, a regional fire department, a mid-sized manufacturer, a hospital network — can implement structured observational assessment without a major IT project, without an army of consultants, and without months of preparation. The tools exist. They are accessible. And organizations are typically up and running in days, not quarters.

What those tools provide is exactly what the flight recorder provided for aviation and the surgical checklist provided for medicine — a consistent, rubric-based record of what people can actually do, generating data that tells you who is performing to standard and who is not, before something goes wrong. Data that holds up to an auditor. Data that reveals gaps your training program did not know it had. Data that, over time, becomes the foundation for genuine continuous improvement.

The flight recorder did not make aviation safer by itself. What it did was make the invisible visible. It turned guesswork into knowledge. That capability is no longer reserved for industries with the resources to mandate it from the top down. It is available to you, now, at whatever scale makes sense for your organization.

The question is simply whether you choose to use it.