Confidence intervals

A survey estimate is a point surrounded by a range. A change smaller than that range is not a change.

The range is the finding

Every disability prevalence figure from a continuous survey comes with a confidence interval: the band within which the true population value almost certainly falls, typically at the 95% level. A finding that 22% of the working-age population is disabled means, in practice, that the true figure is likely somewhere between, say, 21.3% and 22.7%. The point estimate is the headline; the interval is the finding.

Sample size drives the width. Large, well-funded continuous surveys such as the Annual Population Survey produce relatively narrow intervals because they draw on tens of thousands of responses. Smaller subgroup analyses — disabled women in a particular region, say, or a specific impairment type — draw on far fewer cases, and their intervals widen accordingly. A figure reported to one decimal place can sit inside a band several percentage points wide. The precision of the presentation and the precision of the estimate are not the same thing.

Year-on-year comparisons are where intervals do their most important work. If the employment gap narrows by 0.4 percentage points between one annual release and the next, the question is not whether 0.4 is positive or negative but whether it exceeds the margin of uncertainty around both estimates. When analysts test for statistical significance and report that a change is not significant, they are saying the observed movement is smaller than what random sampling variation alone could produce. Reporting that figure as a real shift is a misreading of the data.

Confidence intervals also interact directly with comparability over time. When a survey question is reworded or a methodology revised, the producing body typically runs old and new approaches in parallel for a period and publishes both series. Even then, the intervals on the overlap estimates need to be read alongside the series break, not instead of it — the overlap tells you whether the two methods produce statistically distinguishable results.

Statistical bulletins from the Office for National Statistics and from the Department for Work and Pensions publish confidence intervals in their accompanying tables, sometimes as standard errors rather than the derived interval bounds. A reader who works only from the headline figure in a release, and not from the table, is working without the interval. The caveat sections of most bulletins note this explicitly, though they do not always repeat it near every chart. The intervals are where the caveats live.

Threshold is an independent publication about disability statistics and governance. It does not provide advice, guidance or support on benefits, eligibility, legal matters or individual circumstances.