Cherry picking with synthetic controls
Cristine Pinto () and
Vitor Possebom ()
No 420, Textos para discussão from FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil)
The synthetic control (SC) method has been recently proposed as an alternative method to estimate treatment e ects in comparative case studies. Abadie et al.  and Abadie et al.  argue that one of the advantages of the SC method is that it imposes a data-driven process to select the comparison units, providing more transparency and less discretionary power to the researcher. However, an important limitation of the SC method is that it does not provide clear guidance on the choice of predictor variables used to estimate the SC weights. We show that such lack of speci c guidances provides signi cant opportunities for the researcher to search for speci cations with statistically signi cant results, undermining one of the main advantages of the method. Considering six alternative speci cations commonly used in SC applications, we calculate in Monte Carlo simulations the probability of nding a statistically signi cant result at 5% in at least one speci cation. We nd that this probability can be as high as 13% (23% for a 10% signi cance test) when there are 12 pre-intervention periods and decay slowly with the number of pre-intervention periods. With 230 pre-intervention periods, this probability is still around 10% (18% for a 10% signi cance test). We show that the speci cation that uses the average pre-treatment outcome values to estimate the weights performed particularly bad in our simulations. However, the speci cation-searching problem remains relevant even when we do not consider this speci cation. We also show that this speci cation-searching problem is relevant in simulations with real datasets looking at placebo interventions in the Current Population Survey (CPS). In order to mitigate this problem, we propose a criterion to select among SC di erent speci cations based on the prediction error of each speci cations in placebo estimations
New Economics Papers: this item is included in nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (13) Track citations by RSS feed
Downloads: (external link)
http://bibliotecadigital.fgv.br/dspace/bitstream/1 ... into%20e%20Vitor.pdf (application/pdf)
Journal Article: Cherry Picking with Synthetic Controls (2020)
Working Paper: Cherry Picking with Synthetic Controls (2017)
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
Persistent link: https://EconPapers.repec.org/RePEc:fgv:eesptd:420
Access Statistics for this paper
More papers in Textos para discussão from FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil) Contact information at EDIRC.
Bibliographic data for series maintained by Núcleo de Computação da FGV EPGE ().