Chapter 5 — The Engine He Lacked

Chapter 5 — The Engine He Lacked

Everything Sherlock Holmes was faking, one Presbyterian minister worked out for real, and then — being a modest man with no gift for self-promotion — he died without publishing it, and a friend found the paper in a drawer.

The minister was Thomas Bayes. The paper, read to the Royal Society in 1763, two years after his death, contained the single most important idea in the history of inference: a piece of arithmetic that does, rigorously and honestly, the exact thing Holmes only mimed. It tells you how much a clue should change your mind. Not "aha, the tan line means Afghanistan" — but precisely how much a tan line should shift your belief, given how common tan lines are, how common soldiers are, and how often the two go together. It is the engine Holmes never had, because Doyle handed him certainty for free and a man who is given certainty has no need of arithmetic.

Let me build the engine in front of you. No mathematics beyond arithmetic you already own, I promise; the ideas are the hard part, and the ideas are worth it.

The three numbers

Every honest inference has three quantities in it, and the whole art is keeping them separate.

The first is the prior — how likely your hypothesis was before you saw the clue. This is the base rate from Chapter 2, the number Holmes never computes, the boring statistic about how common the thing is in the general run of the world. Before you know anything about the man in front of you, how likely is any given Londoner to be an invalided Afghan veteran? Low. That lowness is the prior, and it is not optional; it is the floor every inference stands on.

The second is the likelihood — if your hypothesis were true, how expected is the clue? If the man really is a returned soldier, how likely is the tan-and-fair-wrists pattern? Fairly likely. But — and this is the question the amateur always forgets — you must also ask the shadow version: if he is not a soldier, how likely is that same pattern anyway? Because tan lines are also produced by cricket, by gardening, by holidays in Nice. A clue is only worth something to the degree it is more expected under your theory than under its rivals. A clue that every explanation predicts equally is not a clue at all. It is decoration.

The third number is the one you actually want: the posterior — how likely your hypothesis is after you've properly weighed the clue against the prior. Bayes' theorem is simply the machine that turns the first two into the third, honestly, without letting the vividness of the clue stampede you into forgetting how rare the thing was to begin with.

Here is the whole of it in one worked case, and if you follow this you have understood more about inference than Sherlock Holmes ever demonstrated.

The scandal of the positive test

Forget detectives for a moment; the cleanest version of this lives in a doctor's office, and it will transfer back to Baker Street with a vengeance.

A disease afflicts one person in a thousand. There is a test for it. The test is good: if you have the disease, it almost always catches you, and if you don't, it raises a false alarm only five percent of the time. You take the test. It comes back positive. What is the chance you have the disease?

Most people — and, alarmingly, most doctors — answer something like ninety percent. In 1978 researchers put essentially this question to a room of Harvard Medical School students and staff, and the most common answer was ninety-five percent. The correct answer is about two percent.

Watch it work, in plain counting. Take a hundred thousand people. One in a thousand has the disease: that's a hundred sick people, and the good test finds essentially all of them — a hundred positives. But the other ninety-nine thousand nine hundred are healthy, and the test falsely flags five percent of them — that's roughly five thousand false alarms. So the test has produced about five thousand one hundred positive results, and only a hundred of those are truly sick. A hundred out of five thousand one hundred. Your positive test — the alarming, vivid, specific result — leaves you with about a two percent chance of being ill. The base rate you wanted to ignore, that boring "one in a thousand," reached up and swallowed the clue almost whole.

This is the number Holmes never runs. When he sees the tan line and announces "soldier," he is reading a positive test and calling it a diagnosis — ignoring the vast healthy population of tanned civilians who set off the same alarm. In Doyle's rigged world it doesn't matter, because Doyle has quietly set the base rate of "the tanned man really is the soldier" to one hundred percent. In your world it matters more than anything else, and Bayes is the discipline that forces you to look at it. The theorem is, at heart, a machine for making you respect the thing you least want to respect: how rare your exciting conclusion was before you got excited.

The fallacy that hangs people

Now let us take this engine into the one arena where getting it wrong doesn't just embarrass a detective — it destroys real human beings under the seal of the state. The courtroom. Because the single most dangerous inferential error a human can make has a name, and it is named after the very profession that commits it: the prosecutor's fallacy.

The fallacy is a confusion of two questions that sound identical and are worlds apart. How likely is this evidence, if the defendant is innocent? is one question. How likely is the defendant to be innocent, given this evidence? is a completely different one. Swap them — and juries, lawyers, and expert witnesses swap them constantly — and you can march a wholly innocent person to prison on a number that felt like certainty.

Consider the case the California Supreme Court had to clean up in 1968, People v. Collins. A couple had been robbed; witnesses recalled a Black man with a beard and moustache, a white woman with a blonde ponytail, and a yellow car. The prosecution called a maths instructor who assigned a probability to each trait — one in ten for the yellow car, one in four for the moustache, one in three for the interracial couple, and so on — multiplied them all together, and produced a thunderclap of a figure: the odds of another such couple existing were one in twelve million. The jury convicted. The Supreme Court reversed, in a judgment that ought to be framed in every statistics classroom, on two grounds. First, the trait-probabilities were invented — plucked from the air, not measured. And second, even taking the number at face value, the prosecutor had answered the wrong question: in a large city, the chance that at least one other matching couple existed was better than forty percent. The "one in twelve million" was real and irrelevant. It described the rarity of the traits, not the guilt of the defendants, and the gap between those two things is where the injustice lived.

If that were an isolated antique, we could smile at it. It is not. In 1999 an English solicitor named Sally Clark was convicted of murdering her two infant sons, who had died in what she said were successive cot deaths. A distinguished paediatrician, Sir Roy Meadow, told the jury that the chance of two natural cot deaths in one family was one in seventy-three million — a number he produced by taking the odds of one such death and simply squaring them. The figure was catastrophically wrong for a reason Chapter 4 already taught us: squaring assumes the two deaths were independent, when in truth whatever genetic or environmental factor causes one cot death makes a second in the same family far more likely, not coincidental. The Royal Statistical Society was so alarmed that it issued a public statement in 2001 declaring the reasoning "statistically invalid." Sally Clark's conviction was quashed in 2003. She had spent more than three years in prison, branded a murderer of her own children, on the strength of a squared number. She never recovered, and died a few years after her release.

And in the Netherlands, a nurse named Lucia de Berk was convicted in 2003 of a string of murders because so many deaths had occurred on her shifts — a coincidence a court was told had odds of one in three hundred and forty-two million. When statisticians finally re-examined the calculation, the true figure was something closer to one in fifty; the "impossibility" was an artefact of bad multiplication, the very error Meadow made, dressed in different clothes. She was fully exonerated in 2010, after six years in prison.

Three cases. One fallacy. Each one a person crushed by a large, confident, precise number that answered the wrong question — the exact failure mode of a mind that has fallen in love with a vivid statistic and forgotten to ask what it actually means. This is Holmes's disease metastasised into the justice system: the seduction of the specific, the worship of the improbable figure, the fatal confidence. Bayes is the antidote, and these tragedies are, in a sense, what a society looks like when it does its inference the Holmesian way — dazzled by the number, blind to the base rate — with real prisons at the end of it.

The maxim that the law struck down

Which brings us, at last, to the sentence every Sherlock Holmes fan can recite, and which I have been sharpening a knife for since Chapter 1:

"When you have eliminated the impossible, whatever remains, however improbable, must be the truth."

It is a magnificent sentence and it is, as a method, a quiet menace. It has two fatal soft spots. The first is the word impossible — for you can almost never enumerate every possible explanation, let alone prove all but one genuinely impossible; the true answer may be a hypothesis that never even occurred to you, sitting outside the little lineup of suspects your imagination happened to assemble. Philosophers call this the problem of the "bad lot": your best explanation is only the best of the ones you thought of, and nothing guarantees the truth was among them. The second soft spot is that word improbable, waved away as though improbability were a mere inconvenience — when improbability is precisely the thing Bayes exists to take seriously. "However improbable" is doing the work of a shrug, and shrugs convict the innocent.

Do not take my word that the maxim fails in the real world. Take the House of Lords'. In a 1985 shipping case with the wonderful name of Rhesa Shipping Co. v. Edmunds — everyone calls it "The Popi M," after the sunken ship — a judge had reasoned in the purest Holmesian style: the likeliest-sounding remaining explanation for why the vessel sank, improbable though it was, must therefore be accepted as what happened. The Law Lords overturned him, and Lord Brandon delivered what amounts to a judicial refutation of Sherlock Holmes. A court, he ruled, is not obliged to crown the least-improbable surviving theory as the truth. It has a third option that Holmes's maxim forbids: it may simply find that the case has not been proven — that the party who had to demonstrate what happened has failed to, and that "we don't know" is a lawful, honourable verdict. The eliminate-the-impossible maxim denies the mind its most important escape hatch: the right to remain in doubt. English law reached into the canon and cut that maxim out.

And here is the exquisite irony that closes the chapter, the thing Doyle put in the text with his own hand and never seems to have noticed. Holmes had the right idea available to him — he simply preferred the wrong one because it sounded better. In The Hound of the Baskervilles, in an unguarded moment, Holmes describes his method quite differently: "We balance probabilities and choose the most likely. It is the scientific use of the imagination." That — balancing probabilities, choosing the most likely, holding the rest in reserve — is Bayes in a sentence. That is the honest engine. It was sitting right there in the great detective's own mouth, the road not taken, drowned out by the louder, prouder, deadlier music of "eliminate the impossible." Doyle wrote the cure and the disease into the same character and gave the disease the better lines.

The resurrection has its engine now. But an engine needs something to read — real traces, from the real world, that don't come pre-labelled by an author. So we descend from the mathematics into the mud, into the actual science of physical evidence, which was built, brick by unglamorous brick, in the decades after Holmes — by people who loved him, and then did the work he skipped.