Module 09 of 15

Sample Size and Variability

Why small samples swing widely and large samples settle down.

AMathematical Foundation

A sample statistic varies from sample to sample. For the sample mean of n independent observations from a population with standard deviation σ, the standard deviation of the sampling distribution (the standard error) is σ/√n. Precision therefore improves with the square root of sample size: quadrupling n halves the standard error. The central limit theorem adds that, for large enough n, the sampling distribution of the mean is approximately normal even when the population is skewed.

\mathrm{SE}(\bar x)=\frac{\sigma}{\sqrt n}

In words: The standard error of the mean is sigma divided by the square root of n.

$\sigma$: population standard deviation
$n$: sample size

Assumptions

Observations are independent; clustered data (students in the same class) carry less information than their raw count suggests.
Sample size reduces random error only; it does nothing about bias.

BIntuitive Explanation

Flip a fair coin 10 times and getting 7 heads is unremarkable. Flip it 1,000 times and 700 heads would be astonishing. Small groups produce extreme rates by chance: the schools, hospitals or towns with the very highest and very lowest rates of almost anything are disproportionately small ones.

CConnection to Debate

Be cautious with striking results from small samples: a pilot programme with 30 participants, a single season's statistics for a young player, a case study of one city. Ask for the sample size and the interval. Equally, do not dismiss a well-designed small study merely for being small: if the effect is large and the design sound, small samples can be informative. The question is how much the result could vary by chance.

DReal-World Application

Small pilot programmes
Competing interpretation
Evaluations of new education models vary greatly in size. The AIR evaluation of online Algebra I covered 68 schools and 1,885 students; many pilots start with a handful of classrooms. With small cohorts, results vary widely by chance, so dramatic early gains should be read with their uncertainty and expected to shrink as samples grow (see Regression to the Mean). A larger sample reduces random error but not bias: a large sample of self-selected volunteers is still self-selected.
Sources
- Access to Algebra I: The Effects of Online Mathematics for Grade 8 Students (NCEE 2012-4021, final report)Jessica B. Heppen et al., American Institutes for Research and Education Development Center, for the Institute of Education Sciences (REL Northeast and Islands), 2011-12

These examples describe real policies and research. The numbers in the interactive below and in the practice questions are hypothetical teaching data unless a question says otherwise.

EInteractive Visualization

Change the sample size and population shape, and watch the distribution of 1,000 sample means narrow.

Illustrative example — hypothetical data.Simulated population with mean 50. Each bar counts how many of 1,000 samples produced a sample mean in that range.

Sample size (n)10

Population shape

\mathrm{SE}(\bar x)=\frac{\sigma}{\sqrt n}

In words: The standard error of the sample mean equals the population standard deviation divided by the square root of n.

Spread of sample means (simulated SD)

15.30

Theory: σ/√n

15.81

Samples within ±5 of the truth

28%

To halve the spread

n = 40

Four times the sample size

With n = 10, sample means typically miss the true value of 50 by about 15.3. Small samples produce dramatic-looking results by chance alone, which is why a striking finding from a small study deserves caution until it is replicated.

FDebate Practice

GLimitations

The σ/√n formula assumes independent observations. With clustering, repeated measures or dependence over time, effective sample size is smaller. Large samples can make trivially small effects statistically significant, so size of effect still needs judging separately.