The Balance Permutation Test: A Machine Learning Replacement for Balance Tables

Jack T. Rametta and Sam Fuller

No xcwt9, OSF Preprints from Center for Open Science

Abstract: Balance tests are standard for experiments in numerous fields, with many journals across disciplines recommending or requiring them for publication. This standard persists despite significant evidence of balance tests' inadequacies and the development of better tools for detecting failures of random assignment and covariate imbalance. To date there is still no consensus on how randomization and balance should be checked, and also how these failures and imbalances should be addressed, or if they should be addressed at all. In this article we provide clear guidelines and implement a new statistical test, the "balance permutation test," designed to detect arbitrarily complex randomization failures. Our approach leverages a combination of permutation inference and the predictive power of machine learning to accomplish this task. Additionally, we advocate reporting both simple unadjusted and "doubly robust" treatment effect estimates in all experimental contexts, but particularly in situations where failures are detected. To justify our recommendations and the use of our method, we report the results of two sets of applications. First, we show how the balance permutation test is able to detect complex imbalance in real, simulated, and even fabricated data. Second, using an extensive set of Monte Carlo simulations, we demonstrate the overwhelming advantages of doubly robust treatment effect estimation over existing methods. Finally, we introduce an efficient, easy-to-use R package, MLbalance, that implements the balance permutation test approach. Our hope is that this method helps resolve the longstanding debate over how to detect and adjust for assignment failures in experiments.

Date: 2024-01-12
New Economics Papers: this item is included in nep-big and nep-exp
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/65a1c555f2240f097b32e835/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:xcwt9

DOI: 10.31219/osf.io/xcwt9

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().