Module 15 of 15

Sensitivity Analysis

Testing whether a conclusion survives changes to its assumptions.

AMathematical Foundation

Sensitivity analysis asks how much a model's output changes when its inputs change. In one-at-a-time analysis, each input is moved across a plausible range while the others stay at base values; the resulting swing in the output shows which assumptions drive the conclusion. A break-even value is the input level at which the conclusion flips (for example, net benefit = 0). A conclusion is robust if it holds across plausible ranges of all inputs, and fragile if a modest change in one uncertain input reverses it.

\text{Net benefit}=N\cdot p\cdot b-C,\qquad p_{\text{break-even}}=\frac{C}{N\,b}

In words: Net benefit equals people reached times the probability the policy works times benefit per person, minus cost. The break-even probability is cost divided by people times benefit.

$N$: people reached
$p$: probability the policy works as intended
$b$: benefit per person if it works
$C$: total cost

Assumptions

One-at-a-time analysis ignores interactions between inputs; inputs that move together need joint (scenario) analysis.
The plausible range for each input must be justified, ideally by evidence.

BIntuitive Explanation

Before trusting a bridge, engineers ask what happens if the load is heavier or the steel weaker than expected. Before trusting an argument, ask the same: if the effect is half as large as claimed, does the conclusion still hold? If the answer is no, the debate should be about that number.

CConnection to Debate

Sensitivity analysis is how to find the crux of a debate. If the Proposition's case only works when compliance is above 80%, and the evidence suggests 50%, the Opposition should attack compliance, not everything else. Proposition speakers can strengthen their case by showing it survives pessimistic assumptions. This is also how to concede gracefully: grant uncertain numbers and show the conclusion still holds.

DReal-World Application

Housing supply effects under different assumptions
Competing interpretation
How much a reform such as California's SB 79 (applying from 1 July 2026) will lower rents depends on assumptions no one can yet observe: how many permitted units are built, how quickly and where. Estimates from earlier reforms differ widely, from an estimated 34,614 additional permits in 2016–2021 after Auckland's upzoning to no measurable increase in construction within five years of Chicago's targeted upzonings. A useful analysis shows how its conclusion changes across this range rather than reporting a single number.
Sources
- SB 79 Transit-Oriented DevelopmentCalifornia Department of Housing and Community Development (HCD)
- The impact of upzoning on housing construction in AucklandRyan Greenaway-McGrevy, Peter C. B. Phillips (Journal of Urban Economics, 2023), 2023
- Upzoning Chicago: Impacts of a Zoning Reform on Property Values and Housing ConstructionYonah Freemark (Urban Affairs Review 56(3)), 2019-01-29(abstract only)

These examples describe real policies and research. The numbers in the interactive below and in the practice questions are hypothetical teaching data unless a question says otherwise.

EInteractive Visualization

Adjust the assumptions of a simple net-benefit model, see a tornado chart of which inputs matter most, and find break-even values.

Illustrative example — hypothetical data.A made-up policy model in abstract benefit units. It teaches the method, not the merits of any real policy.

\text{net benefit} = N \times p \times b - C

In words: net benefit equals people reached times probability the policy works times benefit per person, minus total cost.

People reached (N)50,000

Probability policy works (p)40%

Benefit per person (b)30

Total cost (C)400,000

Uncertainty range tested±50%

Each input is moved this far up and down while the others stay fixed.

Net benefit at the base values

200,000

The conclusion flips if the probability the policy works falls below 26.7%.

Within ±50%, the sign of the result depends on: probability policy works, people reached, benefit per person.

Tornado chart: which assumption matters most?

Bars show the range of net benefit as each input moves across the tested range.
Red bars cross zero: that assumption alone can reverse the conclusion. Dashed line = base result.

FDebate Practice

GLimitations

Sensitivity analysis explores a model; it cannot reveal a factor the model omits. Ranges can be chosen to produce a desired result, so they should be justified. And a robust cost-benefit result still leaves open whether costs and benefits are distributed fairly, or whether rights constrain the choice.