Regression to the Mean

Regression to the mean is the tendency of an extreme measurement to be followed by one closer to average, purely because the extreme result was partly luck and luck does not repeat. It is a property of measurement, not a force acting on anything.

The fallacy is not the regression itself. It is attributing the return to average to whatever happened in between.

Example of Regression to the Mean

A school tests every pupil, then gives extra tuition to the worst-scoring 10%. Retested, that group improves markedly. The tuition looks effective.

But the lowest scorers on any one test include pupils who were ill, unlucky with the questions, or having a bad morning. On a second test those pupils score closer to their true level regardless of tuition. Some improvement was guaranteed before the first lesson was taught.

Note

The same effect runs in the other direction: praise the best performers and they will usually do worse next time. This produces the durable and entirely false belief that criticism works and praise does not.

In more detail

Regression to the mean is the single most common source of false positives in evaluating interventions, and it is invisible unless you are looking for it. Any programme that selects people because they scored badly will show improvement. Any programme that selects because they scored well will show decline. Neither tells you anything about the programme.

It is particularly destructive in medicine, because people seek treatment when symptoms are at their worst — which is exactly the point from which improvement is most likely regardless of what is done. This is a large part of why uncontrolled trials of almost anything look encouraging, and why the control group is not an administrative formality but the whole of the evidence.

The defence is structural rather than analytical. If a group was chosen for being extreme, compare it against an equally extreme group that received nothing. Where that is impossible, treat the improvement as the expected baseline rather than as a result, and ask how much of it exceeds what regression alone would have produced.