The best almost always get worse.

Any measurement that involves a roll of the dice — a test, a sales quarter, a season, a blood-pressure reading — can be split into two parts: a stable component (skill, true level, underlying rate) and a fluctuating one (luck, noise, a good or bad day). When you select the very top performers on one occasion, you are not selecting the most skilful people. You are selecting people who are skilful and got lucky, because both had to line up to land at the extreme.

Skill carries over to the next measurement. Luck does not. So when you measure the top group again, the skill remains but the luck re-rolls to something more ordinary, and the group's average falls back toward the middle. The bottom group, selected partly for bad luck, rises for the same reason. This drift has a name — regression to the mean — and the crucial, counter-intuitive fact is that it requires no cause whatsoever. It is what randomness looks like when you select on it.

The phenomenon was named by Francis Galton in the 1880s, studying the heights of parents and their grown children. He found, to his initial puzzlement, that exceptionally tall parents had children who were tall but less tall on average — closer to the population mean — and exceptionally short parents had children who were short but less so. He first called it “regression toward mediocrity in hereditary stature,” and the word regression has meant this statistical pull-to-the-middle ever since.

Galton's deeper realisation is the one people still miss: regression isn't a force pulling things to the centre, and it isn't about heredity. It is a mathematical consequence of measuring two things that are correlated but not perfectly. Whenever the correlation between first and second measurement is less than one — which is to say, whenever luck is involved at all — the extremes on one must, on average, be less extreme on the other.

The danger isn't the drift itself; it's that we reach for a story to explain it. Because we intervene precisely when things are at their worst — when symptoms peak, when accidents spike, when a team is bottom of the league — the subsequent regression toward normal arrives right on cue, looking exactly like the effect of whatever we did.

The same illusion is everywhere once you look. The Sports Illustrated cover “jinx”: athletes make the cover after a career-best run, then regress, and a curse gets the blame. The rookie who wins an award and suffers a “sophomore slump.” Most memorably, the flight instructors in Daniel Kahneman's Thinking, Fast and Slow, who noticed that trainees praised after an excellent manoeuvre tended to do worse next time, while those screamed at after a bad one improved — and concluded, with apparent evidence, that criticism works and praise backfires. They had discovered nothing about feedback. They had rediscovered regression to the mean, and drawn a cruel lesson from it.

The fix is the one that fixes most of these illusions: a comparison group. Regression to the mean afflicts a treated group and an untreated group equally, so if you enrol both — patients at their worst, half treated and half not — both will improve as their symptoms regress. The real effect of the treatment is only the extra improvement in the treated arm, over and above the regression that the controls reveal.

This is a large part of why randomised controlled trials exist. It is also why before-and-after studies, single-group pilots, and “we tried it and things got better” testimonials are so treacherous for anything that was started during a crisis: the crisis was the extreme, and recovery toward normal was coming regardless. Randomisation matters too, because it stops you from quietly loading the treatment group with the most extreme cases, who have the most room to regress.

The triumph of mediocrity. In 1933 the economist Horace Secrist published a 468-page study, The Triumph of Mediocrity in Business, marshalling reams of data to show that exceptional firms became more average over time and concluding that competition grinds everyone toward mediocrity. The statistician Harold Hotelling pointed out, witheringly, that Secrist had written an enormous book documenting nothing but regression to the mean — the same pull-to-the-middle would appear in any data with noise, including data running time backwards. It remains the cautionary classic: an entire theory erected on a statistical artefact.

The general shape. Reach for regression whenever three things are true: a group was selected for being extreme, on a measure that contains some luck, and is then measured again. Test-retest gains for the lowest scorers, the apparent benefit of an intervention begun at a low point, the “hot hand” that cools, the analyst who was right once and is now wrong — all carry its fingerprints.

What it is not. Regression is not destiny and not a force: individuals don't get dragged to the average, and populations don't collapse to a point — the spread stays the same, because for every lucky high-scorer drifting down there is an unlucky one drifting up to replace them. It is purely a statement about averages of selected extremes. And it is symmetric in time: the high scorers also had, on average, more ordinary scores before their peak. A drift that runs both directions in time cannot be an effect of anything that happened at the peak.

If the cases were picked for being remarkable and there's any luck in the measure, the answer is usually yes — and the honest test is a control group, not a better story. Regression to the mean is the silent partner behind a surprising number of the other illusions in this compendium; once you can see it, you stop paying the wrong cause.