Robustness? Range Tests for Equality and Equivalence Across Specifications

David Jaeger

No 21802, CEPR Discussion Papers from Centre for Economic Policy Research

Abstract: Applied economists routinely compare estimates across specifications, observe that they are "similar," and conclude that their results are "robust." This common procedure makes an implicit inferential claim about the range of estimates, but usually does not account for their joint sampling distribution when the estimates are generated from the same data. I formalize informal practice with two bootstrap statistics that account for the dependence among estimates. The minimum equivalence bound is the smallest tolerance within which the estimates can be judged equivalent. The range-based equality p-value tests whether the estimates are statistically distinguishable. Together they distinguish failure to detect differences from affirmative evidence of agreement. Simulations across a range of designs show approximately correct size and demonstrate that common visual rules often provide false comfort. Applications to five prominent papers using common identification strategies validate some conventional robustness claims while revealing cases in which apparent agreement reflects imprecision rather than stability. In a pre-registered survey of CEPR and NBER affiliates, expert judgments align with the framework in obvious cases but diverge in intermediate cases, where the verdict depends on the joint sampling distribution rather than on the visible spread of point estimates alone. I suggest that the minimum equivalence bound the equality p-value be reported whenever multiple specifications are presented as evidence of robustness.

Date: 2026-07
References: Add references at CitEc
Citations:

Downloads: (external link)
https://cepr.org/publications/DP21802 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cpr:ceprdp:21802

Ordering information: This working paper can be ordered from
https://cepr.org/publications/DP21802

Access Statistics for this paper

More papers in CEPR Discussion Papers from Centre for Economic Policy Research 33 Great Sutton Street, London EC1V 0DX, UK.
Bibliographic data for series maintained by CEPR ().