Module 03 of 15

Selection Bias

When the way a sample is chosen distorts what it appears to show.

AMathematical Foundation

Selection bias arises when the chance that a unit enters the sample is related to the quantity being measured. Write S = 1 if a unit is selected and π(y) = P(S = 1 | Y = y) for the chance that a unit with value y is selected. The units you observe then have mean E[Y | S = 1], a selection-weighted average of the population: values that are more likely to be selected count for more. The gap between the two is exactly Cov(π(Y), Y) / E[π(Y)]. So the selected units have the population mean whenever the chance of selection is uncorrelated with Y (in particular, when it does not depend on Y at all), and they are biased upward when units with higher values are more likely to be selected, downward when lower values are. Unequal selection chances do not by themselves shift the mean; they shift it through their covariance with Y. How exactly this carries over to a sample mean depends on how the sample is drawn. If a fixed number of units is drawn, or each draw picks unit i with probability proportional to a weight, the expected sample mean equals the weighted average Σπᵢyᵢ / Σπᵢ exactly. If instead each unit is included independently, the sample size is itself random and the sample mean is a ratio of two random quantities; the expectation of a ratio is not in general the ratio of expectations, so Σπᵢyᵢ / Σπᵢ is then only an approximation that improves as the expected sample grows. Counterexample: a population of two people with values 0 and 10, where the first is always included and the second with probability 0.5. Half the time the sample is {0} (mean 0) and half the time {0, 10} (mean 5), so the expected sample mean is 2.5, while Σπᵢyᵢ / Σπᵢ = 5 / 1.5 ≈ 3.33 and the population mean is 5. Common forms of selection: self-selection (volunteers differ from non-volunteers), survivorship (only those who persisted are observed), visibility (successful cases are reported more) and attrition (people drop out for reasons related to the outcome).

E[Y\mid S=1]=\frac{E[\pi(Y)\,Y]}{E[\pi(Y)]},\qquad E[Y\mid S=1]-E[Y]=\frac{\operatorname{Cov}\big(\pi(Y),\,Y\big)}{E[\pi(Y)]}

In words: The mean among selected units equals the expected value of selection probability times Y, divided by the overall selection probability. Its gap from the population mean equals the covariance between the selection probability and Y, divided by the overall selection probability, so the gap is zero exactly when the two are uncorrelated.

$S$: 1 if a unit is selected, 0 otherwise
$\pi(y)$: P(S = 1 | Y = y): chance that a unit with value y is selected
$E[Y\mid S=1]$: average value among the units that end up selected
$E[Y]$: the population mean

E[\bar y_{\text{sample}}]=\frac{\sum_i \pi_i\,y_i}{\sum_i \pi_i}\ \text{(fixed-size or weighted draws)};\qquad E[\bar y_{\text{sample}}]\approx\frac{\sum_i \pi_i\,y_i}{\sum_i \pi_i}\ \text{(independent selection, large samples)}

In words: For a finite population: when the sample size is fixed, or each draw picks units with probability proportional to a weight, the expected sample mean equals the inclusion-weighted mean exactly. When each unit is included independently, so that the sample size is random, this is only an approximation that holds for large samples.

$\pi_i$: probability that unit i is included (or, for weighted draws, its share of the total weight)
$y_i$: the value for unit i
$\bar y_{\text{sample}}$: the ordinary average of the sampled values

\hat{\bar Y}_{\text{HT}}=\frac{1}{N}\sum_{i\in\text{sample}}\frac{y_i}{\pi_i}

In words: The Horvitz–Thompson estimator divides each sampled value by its inclusion probability. It is an unbiased estimate of the population mean when every inclusion probability is positive and known.

$N$: population size
$\hat{\bar Y}_{\text{HT}}$: weighted estimate of the population mean

Assumptions

The population of interest must be defined before you can say a sample is biased relative to it.
The first formula describes a large population in which each unit is selected with a probability that depends on its value; it is an exact consequence of conditional probability and needs E[π(Y)] > 0 (someone can be selected).
The exact finite-population result needs a fixed sample size or weighted draws; with independent selection the sample size is random and the result is approximate (and the mean is undefined if nobody is selected).
Weighting can correct selection only if every unit had a positive, known chance of selection; units that could never be selected cannot be recovered.
Increasing sample size reduces random error but does not reduce selection bias.

BIntuitive Explanation

If you estimate the typical income of content creators by studying the creators who appear in news stories, you study the winners. Thousands of creators earning little are invisible because nobody writes about them. The problem is not that the sample is small; it is that it was chosen by the outcome itself.

CConnection to Debate

Debaters regularly cite success stories as if they were typical: the esports star, the child influencer family that earned millions, the virtual school graduate admitted to a top university. Ask how the case was selected. Survey evidence raises the same question: who answered? A survey of remote workers recruited online, or of fans of a sports podcast, tells you about those respondents, not about all graduates or all fans. A precise rebuttal names the selection mechanism and the direction of the bias it creates.

DReal-World Application

Esports and creator careers
Competing interpretation
Public rankings of esports prize money list the most successful players: each of the ten highest earners has more than $5 million in recorded tournament winnings (Esports Earnings). A list like this is selected on the outcome. It says nothing about the far larger number of aspiring players who never appear on it, so career-risk arguments in the esports and child-influencer motions depend on a base of all aspirants that these sources do not observe.
Sources
- Top Players by Overall EarningsEsports Earnings (esportsearnings.com)
Self-selected remote work
Competing interpretation
Workers who choose remote arrangements differ from those who do not (caring responsibilities, commute, seniority, job type), so comparing volunteers with non-volunteers mixes selection with any effect of remote work. Bloom et al. (2024) avoided this by randomly assigning eligible employees to two work-from-home days a week: quitting fell by about a third, with no detectable effect on performance reviews or promotions over two years. The experiment excluded interns and probationary new hires, so it does not directly answer the question for new graduates.
Sources
- Hybrid working from home improves retention without damaging performanceNicholas Bloom, Ruobing Han, James Liang (Nature 630, 920–925), 2024-06-12

These examples describe real policies and research. The numbers in the interactive below and in the practice questions are hypothetical teaching data unless a question says otherwise.

EInteractive Visualization

Apply random, volunteer, visibility and survivorship sampling rules to a simulated population and compare the estimate with the truth.

Illustrative example — hypothetical data.A simulated population of 5,000 content creators with right-skewed monthly income. Not real income data.

How is the sample chosen?

How strong is the selection?0.80

The top 2% of earners are always seen; everyone else is seen with probability 1 − strength. At strength 1 only the top 2% are seen. At strength 0 this rule is a simple random sample.

Sample size200

True population mean

$183

True median creator

$89

Sample estimate of the mean

$320

Error of the estimate

+75%

What this rule estimates (selection-weighted mean)

$317

This rule over-represents some creators, so the sample suggests a typical income of $320 when the true mean is $183 and the median creator earns $89. A bigger sample does not fix this: each draw picks a creator with probability proportional to their selection weight, so the sample mean estimates the selection-weighted mean of $317, a systematic bias of $134. A larger sample only gives a more precise estimate of that wrong quantity.

FDebate Practice

GLimitations

Selection bias can sometimes be corrected by weighting, if you know how inclusion depended on observed characteristics, but not if it depended on unobserved ones. Not every non-random sample is uselessly biased: what matters is whether inclusion is related to the outcome. Saying “the sample is biased” is only a rebuttal when you can explain the direction and plausible size of the bias.