Module 08 of 15

Confidence Intervals

What a range estimate does and does not tell you about uncertainty.

AMathematical Foundation

A confidence interval is a range computed from sample data by a procedure that, across repeated samples, contains the true parameter a stated proportion of the time (the confidence level). For a mean with known population standard deviation σ, the z-interval is x̄ ± z*·σ/√n, where z* is the critical value (1.96 for 95%). In the frequentist interpretation the parameter is fixed and the interval is random: a 95% level describes the long-run success rate of the method, not the probability that one particular interval contains the parameter.

\bar x \pm z^{*}\,\frac{\sigma}{\sqrt n}

In words: The sample mean plus or minus the critical value times sigma divided by the square root of n.

$\bar x$: sample mean
$z^{*}$: critical value: 1.645 (90%), 1.960 (95%), 2.576 (99%)
$\sigma$: population standard deviation (estimated by s, with a t critical value, when unknown)
$n$: sample size

Assumptions

Observations are independent random draws from the population of interest.
The sampling distribution of the mean is approximately normal (true for normal data, or for large n by the central limit theorem).
The interval only reflects random sampling error, not bias from measurement or selection.

BIntuitive Explanation

Think of the interval-building method as a ring-toss that lands on the target 95% of the time. After one throw, the ring has either landed on the target or not; you just do not know which. Making the ring bigger (a higher confidence level) catches the target more often but tells you less. Taking more observations shrinks the ring without losing accuracy.

CConnection to Debate

When evidence reports an effect “of 3 points (95% CI −1 to 7)”, the interval includes zero and negative effects: the study is compatible with no effect or a small harm. A wide interval means the evidence is weak, not that the effect is large. Conversely, a narrow interval near zero is good evidence that the effect is small. Debaters should read the interval, not just the point estimate.

DReal-World Application

Replication and effect sizes
Competing interpretation
The Open Science Collaboration replicated 100 psychology experiments, and roughly one-third to one-half reproduced, depending on the criterion. Confidence intervals help read such results: a replication estimate can differ from the original for sampling reasons alone, so the question is whether the difference is larger than both intervals make plausible. A 95% interval describes how often the procedure captures the true value in repeated sampling under its assumptions; it does not guarantee that a new study's estimate will fall inside the original interval, and it offers no protection against an original estimate inflated by selective publication.
Sources
- Estimating the reproducibility of psychological scienceOpen Science Collaboration (Science 349(6251), aac4716), 2015-08-28(abstract only)

These examples describe real policies and research. The numbers in the interactive below and in the practice questions are hypothetical teaching data unless a question says otherwise.

EInteractive Visualization

Draw repeated samples, build an interval from each, and count how often the intervals contain the true mean.

Illustrative example — hypothetical data.Simulated samples from a population with known mean 50 and standard deviation 10.

Sample size (n)25

Confidence level

Intervals containing 50 (of 50)

94%

Coverage over 2,000 samples

95.0%

Margin of error (±)

3.92

Interval width

7.84

\bar x \pm z^{*}\,\frac{\sigma}{\sqrt n},\quad z^{*}=1.960

In words: The sample mean plus or minus z star times sigma over root n, where z star is 1.960.

A 95% confidence level describes the method: across many samples, about 95% of intervals built this way contain the true mean. Any single interval either contains it or does not.

It is not correct to say “there is a 95% probability the true mean lies in this particular interval” in the frequentist framework. Larger n narrows the interval; a higher level widens it.

FDebate Practice

GLimitations

Confidence intervals capture sampling error only. A precisely estimated effect from a biased design is precisely wrong. Intervals from many studies of different populations should not be pooled naively, and an interval that excludes zero does not show that an effect is large or important.

AMathematical Foundation

\bar x \pm z^{*}\,\frac{\sigma}{\sqrt n}

In words: The sample mean plus or minus the critical value times sigma divided by the square root of n.

$\bar x$: sample mean
$z^{*}$: critical value: 1.645 (90%), 1.960 (95%), 2.576 (99%)
$\sigma$: population standard deviation (estimated by s, with a t critical value, when unknown)
$n$: sample size

Assumptions

Observations are independent random draws from the population of interest.
The sampling distribution of the mean is approximately normal (true for normal data, or for large n by the central limit theorem).
The interval only reflects random sampling error, not bias from measurement or selection.

BIntuitive Explanation

CConnection to Debate

DReal-World Application

Replication and effect sizes
Competing interpretation
The Open Science Collaboration replicated 100 psychology experiments, and roughly one-third to one-half reproduced, depending on the criterion. Confidence intervals help read such results: a replication estimate can differ from the original for sampling reasons alone, so the question is whether the difference is larger than both intervals make plausible. A 95% interval describes how often the procedure captures the true value in repeated sampling under its assumptions; it does not guarantee that a new study's estimate will fall inside the original interval, and it offers no protection against an original estimate inflated by selective publication.
Sources
- Estimating the reproducibility of psychological scienceOpen Science Collaboration (Science 349(6251), aac4716), 2015-08-28(abstract only)

These examples describe real policies and research. The numbers in the interactive below and in the practice questions are hypothetical teaching data unless a question says otherwise.

EInteractive Visualization

Draw repeated samples, build an interval from each, and count how often the intervals contain the true mean.

Illustrative example — hypothetical data.Simulated samples from a population with known mean 50 and standard deviation 10.

Sample size (n)25

Confidence level

Intervals containing 50 (of 50)

94%

Coverage over 2,000 samples

95.0%

Margin of error (±)

3.92

Interval width

7.84

\bar x \pm z^{*}\,\frac{\sigma}{\sqrt n},\quad z^{*}=1.960

In words: The sample mean plus or minus z star times sigma over root n, where z star is 1.960.

A 95% confidence level describes the method: across many samples, about 95% of intervals built this way contain the true mean. Any single interval either contains it or does not.

It is not correct to say “there is a 95% probability the true mean lies in this particular interval” in the frequentist framework. Larger n narrows the interval; a higher level widens it.

FDebate Practice