01 · You didn't cheat. That's the point.
No single dishonest step
Notice what you didn't do. You didn't alter a number, delete an inconvenient person, or run the same test twice. Every choice on the panel — which outcome to feature, whether to focus on the subgroup where the drug "obviously" helps, whether to trim implausible responders, whether to adjust for an imbalance in age — is a choice a careful, honest researcher makes every day, and can defend in a methods section.
The trouble is only visible from above: those defensible choices multiply. Five outcomes times five subgroups times two outlier rules times two adjustments times two test directions is two hundred analyses, and you reported the one that worked. The statisticians Andrew Gelman and Eric Loken called this the garden of forking paths — you needn't consciously try all two hundred for the logic to bite. If you'd have stopped hunting the moment any result turned up, the effective number of shots is still two hundred.
A p-value answers "how surprising is this one result?" It cannot see the other 199 analyses you'd have accepted. Only you know how many shots were really fired — and the published paper never shows the wall.
02 · The honest fixes
Paint the target first
Every cure amounts to fixing the target before firing. Pre-registration commits you to one outcome, one subgroup and one test before you see the data, so there is only ever one shot. Correcting for multiplicity — dividing the threshold Bonferroni-style, or controlling the false-discovery rate — lowers the bar to match the number of analyses, which is why the reveal's second line sits so far down. A held-out sample lets the hunt generate a hypothesis on one half and test it honestly on the other. And the cheapest discipline of all: treat any subgroup-specific, outlier-trimmed, one-tailed surprise as a question for the next study, never a conclusion from this one.
The specification curve in the reveal is itself a fix. Plotting all the analyses, rather than one, is the modern multiverse approach: if a finding is real, it tends to survive most reasonable choices; if it vanishes the moment you nudge a defensible dial, it was probably a painted target.
03 · Read more
The salmon, the star signs, and the rest
This sandbox is the playable companion to the compendium's entry on the Texas sharpshooter fallacy — home to the dead salmon that "passed" an fMRI test, the heart-attack trial that found aspirin useless for patients born under Gemini, and the finance backtests that manufacture profitable strategies from noise. If torturing this dataset was satisfying, the entry is where the same trick gets caught in the wild.