EconPapers    
Economics at your fingertips  
 

Addressing validity and generalizability concerns in field experiments

Gerhard Riener, Sebastian Schneider and Valentin Wagner

No 345, DICE Discussion Papers from University of Düsseldorf, Düsseldorf Institute for Competition Economics (DICE)

Abstract: In this paper, we systematically analyze the empirical importance of standard conditions for the validity and generalizability of field experiments: the internal and external overlap and unconfoundedness conditions. We experimentally varied the degree of overlap in disjoint sub-samples from a recruitment experiment with more than 3,000 public schools, mimicking small scale field experiments. This was achieved by using different techniques for treatment assignment. We applied standard methods, such as pure randomization, and the novel minMSE treatment assignment method. This new technique should achieve improved overlap by balancing covariate dependencies and variances instead of focusing on individual mean values. We assess the relevance of the overlap condition by linking the estimation precision in the disjoint sub-samples to measures of overlap and balance in general. Unconfoundedness is addressed by using a rich set of administrative data on institution and municipality characteristics to study potential self-selection. We find no evidence for the violation of unconfoundedness and establish that improved overlap, and balancedness, as achieved by the minMSE method, reduce the bias of the treatment effect estimation by more than 35% compared to pure randomization, illustrating the importance of, and suggesting a solution to, addressing overlap also in (field) experiments.

Keywords: external validity; field experiments; generalizability; treatment effect; overlap; balance; precision; treatment assignment; unconfoundedness; self-selection bias; site-selection bias (search for similar items in EconPapers)
JEL-codes: C9 C90 C93 D04 (search for similar items in EconPapers)
Date: 2020
New Economics Papers: this item is included in nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations: Track citations by RSS feed

Downloads: (external link)
https://www.econstor.eu/bitstream/10419/222345/1/1724895958.pdf (application/pdf)

Related works:
Working Paper: Addressing Validity and Generalizability Concerns in Field Experiments (2020) Downloads
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zbw:dicedp:345

Access Statistics for this paper

More papers in DICE Discussion Papers from University of Düsseldorf, Düsseldorf Institute for Competition Economics (DICE) Contact information at EDIRC.
Bibliographic data for series maintained by ZBW - Leibniz Information Centre for Economics ().

 
Page updated 2020-10-01
Handle: RePEc:zbw:dicedp:345