Data-Driven Is the Easy Part. An Honest Number Isn’t.

Everyone wants to be data-driven. It is one of those phrases that has stopped meaning anything, because no one ever stands up to argue for being data-ignorant. So let me argue for something narrower and more useful: most of what passes for a data-driven decision is decoration, and the analysis was never the hard part.

I run operations for businesses that move physical things: machines, vehicles, freight. A while ago I went looking for a simple answer. Which of our ads actually produced a paying job? I expected a messy afternoon of joining tables. Instead I found that the question had no answer at all. Out of several hundred completed jobs, almost none recorded where the customer had come from. The field existed on the form. Dispatchers, under pressure, skipped it every time. The most important number in the whole operation was the one nobody had ever collected.

That is the real shape of the problem. Not “we lack analytics.” We lacked a fact. And you cannot analyse your way out of a fact that was never written down.

A number earns its keep only when a decision is waiting on it

Here is the test I now apply before anyone builds a dashboard. Name the decision. Name the specific action that changes depending on what the number says. If the answer is “it would be good to know,” the number is decoration. Keep it off the wall.

A dashboard full of figures that no one acts on is worse than no dashboard, because it manufactures the feeling of being informed. People point at a rising line in a meeting and call it evidence. Evidence for what decision? Half the time the line is going up because the denominator shrank, or because someone changed how a field was tagged three weeks ago. The chart is real. The confidence it produces is not.

The discipline is boring and it is the whole game: decide what you will do differently at each possible value of the number, before you look. If nothing changes, stop measuring it and go measure something that a decision is genuinely waiting on.

The bias you should fear is upstream, in what got recorded

Most warnings about data bias point at algorithms. The more common and more mundane failure sits one step upstream, in collection. Your data is a record of what the people doing the work had time and reason to write down, filtered through what the form let them enter and what got them yelled at for leaving blank. It is never a neutral photograph of reality.

The lead-source field I mentioned was not missing at random. It was missing precisely on the busiest days, when the phones did not stop, which are the days that matter most. So even after I fixed the form, the early data lied in a specific direction: it under-counted exactly the conditions I most needed to understand. Knowing the shape of what is missing is worth more than another decimal place on what is present.

What this means for development, without the fairy tale

The original version of this essay, which I wrote years ago and no longer believe, said data science would help solve poverty, food insecurity, and healthcare gaps in Sudan. I want to be honest about why that framing is hollow.

In a stable, well-recorded economy, the data mostly exists and the argument is about interpreting it. In much of the world, including where my own family is from, the binding constraint is upstream of interpretation. The birth was never registered. The clinic keeps paper, and the paper burned, or the clinic closed, or the war moved the people the records described. The most decision-relevant facts are the ones no functioning institution was standing to collect.

So the useful contribution is almost never a cleverer model. It is the unglamorous work of building a trustworthy record: a census that people believe enough to answer honestly, a supply chain where each handoff is logged by someone with a reason to log it, a ledger a stranger can audit. That work is closer to plumbing and trust-building than to data science, and it is what actually moves a number you can then act on.

I am not against measurement. I have spent a large part of my working life building the systems that produce it. I am against the version of “data-driven” that skips the hard, physical, human problem of getting an honest number, and jumps straight to the dashboard, because the dashboard is the part that looks like progress in a slide.

Name the decision. Fix the collection. Learn the shape of what is missing. Then, and only then, is it worth arguing about the analysis.