Sampling Bias

Sampling bias is a flaw in how data was collected rather than in how it was analysed: some members of the population were systematically more likely to end up in the sample than others. Every subsequent calculation can be faultless and the answer will still be wrong.

Unlike a small sample, a biased one does not improve by getting bigger. More data collected the same way is more of the same error.

Example of Sampling Bias

The 1936 Literary Digest poll predicted a landslide for Alf Landon over Franklin Roosevelt, from 2.4 million responses — an enormous sample for the period.

The names came from telephone directories and car registrations, and the responses from people who chose to post a reply. In 1936 that meant the comparatively wealthy, and the comparatively motivated. Roosevelt won 46 of 48 states.

Note

Note that this is a methodological bias, not a cognitive one — it lives in the sampling frame, not in anyone's head. It shares a name with the cognitive biases and little else.

In more detail

The Literary Digest failure is the standard illustration because of its scale: it is the clearest demonstration that sample size cannot compensate for sample bias. A biased sample of 2.4 million was beaten decisively by a far smaller random one.

Modern versions are less obvious but no less common. Online polls sample people who saw the poll and chose to answer. App telemetry samples users who did not uninstall. Customer surveys sample those with strong enough feelings to respond, which is why satisfaction data tends to be bimodal. In each case the gap between the sampled population and the intended one is invisible in the data itself.

The practical test is to describe precisely who could not have ended up in the sample, and then ask whether their absence would change the answer. Where it would, the figure is measuring the sampling frame rather than the population — which is also the exact relationship survivorship bias bears to this entry: survivorship is sampling bias with the filter applied by outcome.