Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies

Arun Advani, Toru Kitagawa () and Tymon Słoczyński
Additional contact information
Toru Kitagawa: Institute for Fiscal Studies and University College London

No CWP56/18, CeMMAP working papers from Centre for Microdata Methods and Practice, Institute for Fiscal Studies

Abstract: Currently there is little practical advice on which treatment effect estimator to use when trying to adjust for observable differences. A recent suggestion is to compare the performance of estimators in simulations that somehow mimic the empirical context. Two ways to run such ‘empirical Monte Carlo studies’ (EMCS) have been proposed. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real-world setting where estimator performance is known. We find that in our setting both EMCS approaches are worse than random at selecting estimators which minimise absolute bias. They are better when selecting estimators that minimise mean squared error. However, using a simple bootstrap is at least as good and often better. For now researchers would be best advised to use a range of estimators and compare estimates for robustness.

Keywords: empirical Monte Carlo studies; program evaluation; selection on observables; treatment effects (search for similar items in EconPapers)
Date: 2018-09-27
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (5)

Downloads: (external link)
https://www.ifs.org.uk/uploads/cemmap/wps/CWP561818.pdf (application/pdf)
Our link check indicates that this URL is bad, the error code is: 404 Not Found (https://www.ifs.org.uk/uploads/cemmap/wps/CWP561818.pdf [302 Found]--> https://ifs.org.uk/uploads/cemmap/wps/CWP561818.pdf)

Related works:
Working Paper: Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies (2018)
Working Paper: Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies (2018)
Working Paper: Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies (2013)
Working Paper: Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies (2013)
Working Paper: Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies (2013)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ifs:cemmap:56/18

Ordering information: This working paper can be ordered from
The Institute for Fiscal Studies 7 Ridgmount Street LONDON WC1E 7AE

Access Statistics for this paper

More papers in CeMMAP working papers from Centre for Microdata Methods and Practice, Institute for Fiscal Studies The Institute for Fiscal Studies 7 Ridgmount Street LONDON WC1E 7AE. Contact information at EDIRC.
Bibliographic data for series maintained by Emma Hyman ().