Striking the Right Balance: Why Standard Balance Tests Over-Reject the Null, and How to Fix It

Jason Kerwin, Nada Rostom (nada.rostom@uantwerpen.be) and Olivier Sterck
Additional contact information
Nada Rostom: University of Antwerp

No 17217, IZA Discussion Papers from Institute of Labor Economics (IZA)

Abstract: Economists often use balance tests to demonstrate that the treatment and control groups are comparable prior to an intervention. We show that typical implementations of balance tests have poor statistical properties. Pairwise t-tests leave it unclear how many rejections indicate overall imbalance. Omnibus tests of joint orthogonality, in which the treatment is regressed on all the baseline covariates, address this ambiguity but substantially over-reject the null hypothesis using the sampling-based p-values that are typical in the literature. This problem is exacerbated when the number of covariates is high compared to the number of observations. We examine the performance of alternative tests, and show that omnibus F-tests of joint orthogonality with randomization inference p-values have the correct size and reasonable power. We apply these tests to data from two prominent recent articles, where standard F-tests indicate imbalance, and show that the study arms are actually balanced when appropriate tests are used.

Keywords: balance tests; power; size; randomization inference (search for similar items in EconPapers)
JEL-codes: C1 C9 O12 (search for similar items in EconPapers)
Pages: 58 pages
Date: 2024-08
New Economics Papers: this item is included in nep-ecm and nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://docs.iza.org/dp17217.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp17217

Ordering information: This working paper can be ordered from
IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany
library@iza.org

Access Statistics for this paper

More papers in IZA Discussion Papers from Institute of Labor Economics (IZA) IZA, P.O. Box 7240, D-53072 Bonn, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Holger Hinte (hinte@iza.org).