P-Hacking

P-hacking is the practice of searching a dataset until something passes a significance test, then reporting that result as though it had been the hypothesis all along. No individual step need be dishonest; the dishonesty is in what goes unreported.

It is the only entry in this library that is a behaviour rather than a property of data.

Example of P-Hacking

A researcher tests whether a supplement affects twenty health markers. One comes back significant at p < 0.05 and the paper reports that one.

But p < 0.05 means roughly a one-in-twenty chance of a false positive per test. Running twenty tests makes about one spurious hit the expected outcome. The finding is exactly what you would see if the supplement did nothing at all.

Note

The usual remedies are pre-registering the hypothesis before collecting data, and correcting the threshold for the number of comparisons made.

In more detail

P-hacking rarely looks like misconduct from the inside, which is what makes it durable. The decisions that produce it are individually defensible: excluding an outlier, trying a different model, adding a covariate, stopping collection once the result stabilises. Each is reasonable in isolation. Taken together, and taken only in the direction that helps, they amount to a search.

The statistical point is narrow and unforgiving. A p-value describes the probability of seeing a result this extreme if nothing is going on — but only for a single, pre-specified test. Once the analysis is chosen after seeing the data, the number no longer means what it says, and no amount of care in computing it restores the meaning.

For a reader, the useful questions are how many comparisons were available and how many were reported. A paper testing one hypothesis specified in advance is making a far stronger claim than a paper reporting the one significant result from a panel of twenty, even when the two print the same p-value.