Module 10 of 15

Regression to the Mean

Why extreme results tend to be followed by less extreme ones.

AMathematical Foundation

When a measurement combines a stable component and random noise, units selected for an extreme first measurement will, on average, be less extreme on a second measurement. If true score T has variance σ²_T and noise has variance σ²_e, the reliability is ρ = σ²_T / (σ²_T + σ²_e). For a group selected on its first score, the expected second score is μ + ρ(x̄₁ − μ): it moves a fraction (1 − ρ) of the way back to the population mean. With perfect reliability there is no regression; with pure noise the group returns fully to the mean.

E[X_2\mid X_1=x]=\mu+\rho\,(x-\mu),\qquad \rho=\frac{\sigma_T^2}{\sigma_T^2+\sigma_e^2}

In words: The expected second score given a first score x is the mean plus reliability times the distance of x from the mean.

$\mu$: population mean
$\rho$: reliability: share of score variance due to stable differences
$\sigma_T^2, \sigma_e^2$: variance of true scores and of noise

Assumptions

The two measurements have the same distribution and independent noise.
The formula is exact when true scores and noise are normally distributed (more generally, when the expected second score is a straight-line function of the first); otherwise it is the best straight-line prediction.
Nothing systematic changed between measurements.

BIntuitive Explanation

A footballer who has an exceptional first season was probably both good and lucky. Next season, the luck is redrawn and performance tends to fall back, prompting talk of a “second-season slump”. Nothing has to go wrong for this to happen. Likewise, schools placed on an improvement plan after a terrible year will tend to improve the next year, plan or no plan.

CConnection to Debate

Policies targeted at extreme cases (worst-performing schools, the most-retracted journals, neighbourhoods with a spike in crime, a club in a relegation season) will show “improvement” partly by regression alone. A debater claiming that an intervention worked must show that a comparable group selected the same way, but not treated, improved less. Conversely, a speaker can use regression to undermine an opponent's dramatic before-and-after statistic.

DReal-World Application

Striking initial findings and replication
Competing interpretation
Studies published because they found unusually large or significant effects will tend to find smaller ones on replication, even if nothing was done wrong: selecting on a noisy estimate selects for favourable noise. The Open Science Collaboration reproduced only one-third to one-half of 100 published psychology findings, and Registered Reports, which remove selection on results, report positive results far less often (44% against 96%). Regression to the mean combined with selective publication is one explanation, alongside differences between original and replication methods and false original findings.
Sources
- Estimating the reproducibility of psychological scienceOpen Science Collaboration (Science 349(6251), aac4716), 2015-08-28(abstract only)
- An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered ReportsAnne M. Scheel, Mitchell R. M. J. Schijen, Daniël Lakens (Advances in Methods and Practices in Psychological Science 4(2)), 2021-04-16(abstract only)

These examples describe real policies and research. The numbers in the interactive below and in the practice questions are hypothetical teaching data unless a question says otherwise.

EInteractive Visualization

Select the highest or lowest scorers on a first test and see their second-test scores move towards the average with no intervention.

Illustrative example — hypothetical data.400 simulated students take two equally difficult tests. Each score is a stable ability plus random noise. Nobody receives any treatment between tests.

Select students who scored

Share selected10%

Measurement noise (SD)8

Ability SD is fixed at 8. More noise means a less reliable test.

Selected group, test 1

51.3

Same group, test 2

61.0

Change with no intervention

+9.6

Test reliability

0.50

Predicted test 2: 60.7

The lowest scorers on test 1 were partly unlucky: selecting on an extreme score also selects extreme noise. On test 2 the noise is fresh, so the group moves up by 9.6 points with no help at all.

If a tutoring scheme, sanction or policy is applied only to extreme performers, some of the later change is regression, not effect. A comparison group selected the same way is needed.

FDebate Practice

GLimitations

Regression to the mean applies to groups selected on noisy measurements. It does not imply that everything evens out, that extreme units are not really different, or that interventions never work. Its size depends on reliability, which can be estimated.

AMathematical Foundation

E[X_2\mid X_1=x]=\mu+\rho\,(x-\mu),\qquad \rho=\frac{\sigma_T^2}{\sigma_T^2+\sigma_e^2}

In words: The expected second score given a first score x is the mean plus reliability times the distance of x from the mean.

$\mu$: population mean
$\rho$: reliability: share of score variance due to stable differences
$\sigma_T^2, \sigma_e^2$: variance of true scores and of noise

Assumptions

The two measurements have the same distribution and independent noise.
The formula is exact when true scores and noise are normally distributed (more generally, when the expected second score is a straight-line function of the first); otherwise it is the best straight-line prediction.
Nothing systematic changed between measurements.

BIntuitive Explanation

CConnection to Debate

DReal-World Application

Striking initial findings and replication
Competing interpretation
Studies published because they found unusually large or significant effects will tend to find smaller ones on replication, even if nothing was done wrong: selecting on a noisy estimate selects for favourable noise. The Open Science Collaboration reproduced only one-third to one-half of 100 published psychology findings, and Registered Reports, which remove selection on results, report positive results far less often (44% against 96%). Regression to the mean combined with selective publication is one explanation, alongside differences between original and replication methods and false original findings.
Sources
- Estimating the reproducibility of psychological scienceOpen Science Collaboration (Science 349(6251), aac4716), 2015-08-28(abstract only)
- An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered ReportsAnne M. Scheel, Mitchell R. M. J. Schijen, Daniël Lakens (Advances in Methods and Practices in Psychological Science 4(2)), 2021-04-16(abstract only)

These examples describe real policies and research. The numbers in the interactive below and in the practice questions are hypothetical teaching data unless a question says otherwise.

EInteractive Visualization

Select the highest or lowest scorers on a first test and see their second-test scores move towards the average with no intervention.

Illustrative example — hypothetical data.400 simulated students take two equally difficult tests. Each score is a stable ability plus random noise. Nobody receives any treatment between tests.

Select students who scored

Share selected10%

Measurement noise (SD)8

Ability SD is fixed at 8. More noise means a less reliable test.

Selected group, test 1

51.3

Same group, test 2

61.0

Change with no intervention

+9.6

Test reliability

0.50

Predicted test 2: 60.7

The lowest scorers on test 1 were partly unlucky: selecting on an extreme score also selects extreme noise. On test 2 the noise is fresh, so the group moves up by 9.6 points with no help at all.

If a tutoring scheme, sanction or policy is applied only to extreme performers, some of the later change is regression, not effect. A comparison group selected the same way is needed.

FDebate Practice