The Anatomy and Evolution of Survey Error

Bruce Meyer, Nikolas Mittag, Derek Wu, Anthony Tatarka and Patrick Langetieg

No 35680, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Surveys provide much of what we know in social science and are often the only source of information on private behaviors, attitudes and expectations. Strands of the literature document four broad error sources: survey coverage, unit non-response, item non-response, and measurement error. Yet the absence of comparable error measures across sources and over time has left the relative severity of error sources difficult to assess, limiting efforts by survey producers to improve accuracy and users to interpret survey statistics. We unify the fragmented literature by analyzing and decomposing survey error in an empirical Total Survey Error (TSE) framework, quantifying for the first time every error component on a common scale of bias and absolute error. We apply our approach to comprehensively measure survey error in eighteen income and program receipt variables from the CPS ASEC – one of the oldest and most rigorously tested surveys, and the official source of income and poverty statistics in the U.S. – using linked administrative records over more than two decades. For nearly half of the variables, bias exceeds 40 percent and eliminating all individual-level error would require changing more than 80 percent of the true total. Contrary to the conventional emphasis on non-response, we find that the dominant source of bias is measurement error among respondents who fail to report receipt (false negatives). The largest potential gains therefore come from improving the accuracy of respondent reporting rather than raising response rates. This underreporting also propagates into imputed values for non-respondents, which are drawn from (misreported) responses and are particularly noisy, with positive and negative errors that tend to offset in net terms. Moreover, net bias can be an incomplete guide to data quality, as we find that absolute error has tended to rise over time even when net bias is stable. Overall, our approach provides a new method for measuring and decomposing survey error, clarifying which sources of bias require user corrections and where survey design improvements could be most consequential.

JEL-codes: C81 C83 D31 I38 (search for similar items in EconPapers)
Date: 2026-08
Note: AG CH LS PE
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.nber.org/papers/w35680.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:35680

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w35680
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().