The Man Who Measured Everything Except the Thing That Mattered

My colleague Darren was explaining, with the measured patience of a man who has explained this same thing perhaps forty times, why we needed to add a seventh column to a spreadsheet that already had six columns nobody was reading. The spreadsheet tracked customer satisfaction. Or rather, it tracked the things we had decided to measure about customer satisfaction, which is not the same thing, in the same way that a photograph of a meal is not the same thing as lunch. Darren's argument was that the seventh column, which would capture "sentiment trajectory" (a phrase I am not making up), would finally give leadership "the full picture." He said "full picture" the way someone describes a jigsaw puzzle they're assembling: with confidence that the missing piece is the one preventing the image from making sense, rather than considering the possibility that they might be assembling the wrong puzzle.

I sat there nodding, which is what I do in these meetings, and I thought about something said in a lecture I'd watched the previous evening, which was that most businesses operate in only two modes: cost reduction and regulatory paranoia. Don't fall foul of the regulator. Don't have a reputational disaster. Spend as little money as you can. And that marketing and innovation, the only two functions Peter Drucker believed actually create customers, sit in almost direct opposition to both of those instincts.[1] Darren's spreadsheet was not a tool for understanding customers. It was a tool for defending ourselves to people who might ask whether we understood customers. These are different activities, and the distance between them is where most of the interesting damage happens.

Here is something I have been turning over for weeks now, and I want to be careful about how I say it because I think the precision matters. There is a pattern in organizations (mine included, I should say, before anyone accuses me of throwing stones from inside a glass house[2]) where the act of measuring a thing gradually, almost imperceptibly, begins to replace the act of understanding the thing. The measurement becomes the conversation. The dashboard becomes the strategy. And at some point, the question shifts from "what is actually happening with our customers?" to "what do the numbers say is happening with our customers?" which, again, sounds the same but isn't.

The British economist Charles Goodhart figured this out in 1975, working on monetary policy of all things, and his observation has since been compressed into a sentence so clean it almost feels like a proverb: when a measure becomes a target, it ceases to be a good measure.[3] Which, okay, sounds obvious when you read it on a poster. It is considerably less obvious when you are inside the machine it describes. Because inside the machine, the measure doesn't announce itself as a target. It arrives as accountability. It arrives as rigor. It arrives dressed in the language of data-driven decision-making, and who's going to argue against data-driven decision-making? You might as well argue against oxygen.

And yet. A Soviet nail factory given a production target measured in number of nails produces millions of tiny, useless nails. A nail factory given a target measured in weight produces a few enormous, equally useless nails.[4] A hospital system targeted on reducing average length of stay begins discharging patients too early, producing readmission rates that more than offset whatever efficiency was gained. A call center measured on calls-per-hour starts hanging up on customers. The metric improves. The thing the metric was supposed to represent gets worse. And the people who designed the metric, who are typically not the people who have to live inside it, walk away claiming the credit for a number moving in the right direction.

This is what I've heard called the doorman fallacy, and I think it might be the single most useful mental model for understanding why so many well-intentioned optimization efforts end up making things worse. You go to a hotel. You notice they have a doorman. You define the doorman's job as opening the door. You replace the doorman with an automatic door. You claim the cost savings. You leave. And five years later the hotel's rate has collapsed, there are vagrants sleeping in the lobby, the most loyal guests have quietly decamped to a competitor, and nobody connects any of this to the decision to remove the doorman. Because the doorman was never just opening the door. He was hailing taxis. He was recognizing regulars. He was providing a subtle signal of status and safety that made people willing to pay more for a room. But none of that showed up on the spreadsheet, so none of it was defended when the spreadsheet came for him.[5]

I keep thinking about a story I heard about Royal Mail.[6] They poured enormous resources into improving operational efficiency: on-time delivery, reliability, consistency. All the measurable stuff. And none of it, not one bit, correlated with how much people liked the brand. What actually determined whether you had a positive perception of Royal Mail was whether you liked your mail carrier. If your postal worker was the kind of person who noticed you were on vacation and left your package in the back porch, you thought Royal Mail was a world-class organization. If your postal worker was surly, you didn't. The entire operational apparatus, the logistics, the sorting facilities, the fleet management, all of it was less predictive of customer affection than one person's willingness to be kind.

This is, I think, what Robert McNamara got catastrophically wrong during Vietnam, and what Daniel Yankelovich later described in four devastating steps.[7] First you measure whatever can be easily measured. Fine. Second you disregard what can't be easily measured. Artificial and misleading. Third you presume what can't be measured isn't important. Blindness. Fourth you say what can't be measured doesn't exist. And this, Yankelovich said flatly, is suicide. McNamara measured body counts. The body counts said America was winning. America was not winning. General Edward Lansdale told McNamara he needed to add an "x-factor" to his list of metrics: the feelings of the Vietnamese people. McNamara wrote it down, then erased it, and told Lansdale he couldn't measure feelings, so they must not matter.

I have a version of this story that is less dramatic and involves no geopolitics, only a customer care operation I used to manage. We had a CSAT score. The CSAT score was good. The CSAT score was, in fact, consistently above benchmark, which meant that in every quarterly review I could point to a number and say: see, customers are satisfied. What I could not point to, because I had no column for it, was the number of customers who had quietly stopped calling us altogether. Not because their problems were solved. Because they had given up. The satisfied customers were the ones who stayed in the system long enough to be surveyed. The dissatisfied ones had already left. They existed in the silence between the data points, and the silence didn't have a column.[8]

There is a joke (or maybe a parable, or maybe just a sad observation) about a drunk man searching for his keys under a streetlight. A cop walks by and asks if he lost them there. "No," says the drunk, "I lost them in the park. But the light is better here." This is called the streetlight effect, and it describes, with uncomfortable accuracy, the way most organizations approach measurement. We measure what's under the light. We optimize what's under the light. We celebrate improvements under the light. And the thing we actually need to understand is sitting in a dark park, unexamined, growing more consequential by the day.

There are two completely different philosophies of what a business is for. One school says a business is an efficiency mechanism. The purpose is to give people what they already know they want at the lowest possible cost. This school loves spreadsheets. It fits beautifully into mathematical models. It is aggregatable and calculable and, because it is aggregatable and calculable, it has won. The other school, the Austrian school, says a business is a discovery mechanism. Its purpose is to explore, to find new sources of value, to operate in the territory between what people say they want and what they would actually love if someone showed it to them. Ludwig von Mises argued there is no useful distinction between the value created by the cook and the value created by the person who sweeps the floor. The cook produces the food. The floor-sweeper produces the environment in which you can appreciate the food. Both are value creation. But one of them shows up on the balance sheet and the other one doesn't, and that asymmetry is the seed from which most organizational dysfunction grows.

I notice this in my own work, constantly. The temptation to measure what I can measure, to report what I can report, to optimize the visible machinery while leaving the invisible machinery (the trust, the relationships, the institutional knowledge, the willingness of one team to help another at 4:47pm on a Friday) entirely unmeasured and therefore, by the logic of the fallacy, increasingly invisible. I had a period a few months ago where I was building dashboards with genuine enthusiasm, real creative energy poured into visualizing ticket volumes and response times and resolution rates, and at some point my wife said something that stopped me cold. She said: "You seem to know everything about how fast people are answering, and nothing about whether the answers are any good." She was not trying to be devastating. She was eating a sandwich. But she was right, in the way that people who aren't inside the system are often right, because they can see the shape of it from outside.

The Chesterton's fence principle says: before you tear down a fence, first understand why someone built it.[9] This is the inverse of the doorman fallacy, or maybe its complement. The doorman fallacy is about removing value you didn't measure. Chesterton's fence is about removing structure you didn't understand. Together they describe a kind of organizational amnesia where we keep dismantling things and then being surprised when the absence of the thing we dismantled turns out to have been load-bearing.

Here is where it gets personal, and maybe a little uncomfortable. I have spent most of my career in customer support, which is an industry that has been on the wrong end of the doorman fallacy for decades. Support is, in the spreadsheet view, a cost center. It exists to resolve problems that, ideally, wouldn't exist in the first place. And so the default organizational posture toward support is: how do we make this cheaper? How do we deflect more contacts? How do we automate more interactions? How do we drive customers to the lowest-cost channel?[10] And each of these questions, taken individually, has a defensible answer. Taken collectively, they amount to a systematic dismantling of the human relationship between a company and the people who pay it money. They are doorman fallacy at industrial scale.

The self-checkout problem is the perfect microcosm.[11] As an option, self-checkout is wonderful. I'm in a hurry, I have five items, the line is long. Great. As a mandate, imposed because someone in finance noticed it's cheaper to have customers do the scanning, it becomes something else entirely. It becomes the systematic removal of the human element from a transaction, and the human element, it turns out, was doing work that nobody measured. The cashier who recognized you, who asked about your day, who noticed you were buying ingredients for what was obviously a dinner party and said something pleasant about it. None of that has a column. All of it has value.[12]

And now comes AI, which is the doorman fallacy's final boss, or possibly its apotheosis. Because AI's first-wave sales pitch is cost reduction. It has to be. Insane amounts of money have been spent building these systems, and the bill is coming due, and the easiest way to sell a technology to a business in cost-reduction mode is to frame it as a way to reduce headcount. Fewer support agents. Fewer content writers. Fewer middle managers. Fewer doormen. The savings are calculable, presentable, and immediately legible on a quarterly earnings call. The losses, the quality degradation, the erosion of customer trust, the disappearance of institutional knowledge, the slow hemorrhaging of the intangible stuff that made the organization function, all of that will show up later, and when it does, the people who sold the efficiency will have moved on to their next engagement.

But (and this is the part I keep coming back to, the part that gives me something closer to hope than pessimism) there is a second phase. And possibly a third: that the first phase of any new technology is "the same, but cheaper." Replace the steam engine with an electric motor. The gains are trivial. Only when you rethink the entire production process around the new capability (small motors! individual machines! they can turn off!) do you get the actual transformation. The factories of the late 1800s didn't just swap power sources. They reinvented the factory. And the businesses that will thrive in the AI era won't be the ones that used AI to fire the doorman. They'll be the ones that figured out how to hire ten more doormen because AI made everything else so much cheaper that the human part, the part that actually determines whether a customer loves you or tolerates you, could finally get the investment it deserved.

I am back at my desk now, looking at Darren's spreadsheet. The seventh column has been added. Sentiment trajectory. It has numbers in it. The numbers are green, which means they are good, because green means good in every dashboard ever built, as if the color itself constitutes an argument. I scroll to the right, past all seven columns, to the place where the spreadsheet ends and the blank cells begin. There is nothing there, of course. There is never anything there. But I have started to think of that blank space, the space past the last column, as the most important real estate in the document. That's where the feelings of the Vietnamese people go. That's where the postal worker's kindness goes. That's where the doorman's ability to recognize a regular and remember their name goes. All the things we can't measure, can't quantify, can't defend in a quarterly review. All the things that, if Yankelovich was right, we will eventually pretend don't exist.[13]

Unless someone speaks up for them. Which is, I think, the actual job.[14]

[1]Rory Sutherland describes this beautifully in Alchemy, where he argues that a business is not just an efficiency mechanism but a discovery mechanism. The Austrian school of economics, particularly Ludwig von Mises, held that value is fundamentally subjective, and therefore marketing (which creates perceived value) is every bit as real an operation as manufacturing. The opposing school, which unfortunately won, treats humans as rational calculators with stable preferences who just need the cheapest possible delivery of what they already know they want. ↩︎

[2]I should note here that I am aware of the irony of working in my role and writing an essay about the dangers of over-systematization. I contain multitudes. Or at least contradictions. ↩︎

[3]Charles Goodhart was a monetary policy advisor at the Bank of England when he formulated this in 1975. Anthropologist Marilyn Strathern later restated it more crisply. The original context was about inflation targeting, which makes it sound dry, but the principle applies to every human system that has ever tried to optimize for a number. ↩︎

[4]There is an apocryphal story, possibly Soviet, possibly just too good to verify, about a nail factory that was given a production target measured in number of nails. The factory promptly began producing millions of tiny, useless nails. When the target was changed to weight of nails, they produced a few enormous, equally useless nails. The story may be fake. The principle is not. ↩︎

[5]The entire concept of "gain share" agreements deserves its own essay, or maybe its own criminal investigation. The basic structure is: a consultancy comes in, identifies cost savings, takes a percentage of those savings as their fee, and then leaves before anyone can measure whether the savings actually produced net value or just net subtraction disguised as efficiency. It is a business model built on the assumption that nobody will check back in two years. ↩︎

[6]The posty anecdote is Rory Sutherland's, and I've been unable to stop thinking about it. Royal Mail spent enormous sums improving on-time delivery. It had zero measurable effect on brand perception. What actually determined whether you liked Royal Mail was whether you liked your postal carrier. Literally. The entire multimillion-pound operational improvement was irrelevant compared to whether the person who brought your mail was friendly. ↩︎

[7]Yankelovich laid out the four steps of the fallacy in a 1972 essay called "Corporate Priorities": First, measure whatever can be easily measured. This is fine. Second, disregard what can't easily be measured or give it an arbitrary quantitative value. This is artificial and misleading. Third, presume that what can't be measured easily isn't important. This is blindness. Fourth, say that what can't be easily measured doesn't exist. This is suicide. ↩︎

[8]I once watched a colleague spend forty-five minutes building a dashboard to track a metric that, when I asked what decision it would inform, produced a silence so complete you could hear the HVAC system cycling. ↩︎

[9]G.K. Chesterton, writing in The Thing (1929), argued that before you remove a fence, you should first understand why someone built it. The principle applies with uncomfortable precision to corporate reorganizations. ↩︎

[10]Phone conversion rates at online travel agencies run around 30%. Website visitor conversion rates hover at about 0.5%. And yet the entire strategic direction of most online businesses is to hide the phone number. Because the phone costs more per interaction. Because the spreadsheet says so. Because nobody is held accountable for the bookings that never happen because the customer gave up trying to find a human. ↩︎

[11]The self-checkout thing is genuinely instructive. As an option, it works beautifully. I'm in a hurry, I have five items, the queue is long, let me scan myself. As an obligation, imposed because finance noticed it's cheaper, it becomes a different thing entirely. Shoplifting surged. Some supermarkets were, as one observer put it, selling more onions than they were buying, because customers were scanning avocados as onions. It was the miracle of the feeding of the five thousand (with produce fraud). ↩︎

[12]Estate agents, apparently, go to extraordinary lengths to make sure the buyer and seller of a house don't meet in person until documents are signed. Because if one doesn't personally like the other, the deal collapses. A friend's mother was once offered the asking price and responded: "It wasn't those awful people who came round yesterday, was it? I'm not selling." Asking price. Refused. Because of vibes. ↩︎

[13]Which sounds simple but isn't, because it requires the person doing the thinking to be present in rooms where decisions are made, and those rooms are typically controlled by the people who think in spreadsheets. Getting invited to the room is, itself, a political act disguised as a scheduling request. ↩︎

[14]Concorde is the perfect example. All engineers, no marketers. Flying east to west: you leave London at 9am and land in New York before breakfast. Brilliant. The return leg: instead of going to JFK at 9pm and sleeping on the plane, you have to stay an extra night in New York, get up early, spend an entire working day in the air, and land in London at 6pm exhausted. They optimized for speed. They forgot to ask what speed feels like to a human being trying to get home. ↩︎