Comparability over time

A time series is only as stable as the question that generates it.

When improvement breaks a series

Statistical agencies face a persistent tension: the question that best captures disability today may not be the question that was asked ten years ago. When ONS, the Department for Work and Pensions, or any other producer revises the wording of a survey instrument — adding a condition, changing the threshold from "limiting" to "substantially limiting", or splitting a single item into two — the count moves. The problem is that nobody can tell, after the fact, how much of that movement reflects a real change in the population and how much reflects the new question.

This is not a hypothetical concern. The Family Resources Survey has carried the disability question through several specification changes since the 1990s. Each time the wording shifted, producers issued a break-in-series notice and warned users not to read the figures across the discontinuity as a trend. The warning is technically correct and practically inconvenient for anyone trying to establish whether the disability employment gap, for example, has genuinely narrowed over a decade.

The discontinuity problem compounds when two separate bodies change their instruments at different times. If the Labour Force Survey revises its question in one year and the Annual Population Survey follows with a different revision two years later, the resulting figures are out of step not only with their own pasts but with each other. Reconciling them requires knowing exactly when each change was made and what the change was — information that lives in methodology notes that most users of headline bulletins never read.

Parallel running — fielding both the old and new question simultaneously for one or two survey waves — is the standard remedy. It generates an overlap estimate that lets producers calculate an approximate adjustment factor. But parallel running costs money and survey space, and it is not always done. When it is skipped, the break is permanent: the old series ends and the new one begins, with no bridge between them.

The wording decides the number in any single snapshot; comparability over time asks a harder question — whether the instrument that generated last year's figure is sufficiently close to this year's to support subtraction. Where the answer is no, the honest response is to treat them as two separate series that happen to share a name, and to say so explicitly when citing either.