Inference with difference-in-differences with a small number of groups: a review, simulation study and empirical application using SHARE data

Slawa Rokicki, Jessica Cohen, Günther Fink, Joshua Salomon and Mary Beth Landrum

No 18-01, CHaRMS Working Papers from Centre for HeAlth Research at the Management School (CHaRMS)

Abstract: Difference-in-differences (DID) estimation has become increasingly popular as an approach to evaluate the effect of a group-level policy on individual-level outcomes. Several statistical methodologies have been proposed to correct for the within-group correlation of model errors resulting from the clustering of data. Little is known about how well these corrections perform with the often small number of groups observed in health research using longitudinal data. First, we review the most commonly used modelling solutions in DID estimation for panel data, including generalized estimating equations (GEE), permutation tests, clustered standard errors (CSE), wild cluster bootstrapping, and aggregation. Second, we compare the empirical coverage rates and power of these methods using a Monte Carlo simulation study in scenarios in which we vary the degree of error correlation, the group size balance, and the proportion of treated groups. Third, we provide an empirical example using the Survey of Health, Ageing and Retirement in Europe (SHARE). When the number of groups is small, CSE are systematically biased downwards in scenarios when data are unbalanced or when there is a low proportion of treated groups. This can result in over-rejection of the null even when data are composed of up to 50 groups. Aggregation, permutation tests, bias-adjusted GEE and wild cluster bootstrap produce coverage rates close to the nominal rate for almost all scenarios, though GEE may suffer from low power. In DID estimation with a small number of groups, analysis using aggregation, permutation tests, wild cluster bootstrap, or bias-adjusted GEE is recommended.

Keywords: Difference-in-differences; Clustered standard errors; Inference; Monte Carlo simulation; GEE (search for similar items in EconPapers)
JEL-codes: C18 C52 I10 (search for similar items in EconPapers)
Pages: 34 pages
Date: 2018-01
New Economics Papers: this item is included in nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (8)

Downloads: (external link)
ftp://ftp.qub.ac.uk/pub/users/repec/qub/charms/MS_WPS_CHARMS_18_01.pdf (application/pdf)
Our link check indicates that this URL is bad, the error code is: 500 Failed to connect to FTP server ftp.qub.ac.uk: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:qub:charms:1801

Access Statistics for this paper

More papers in CHaRMS Working Papers from Centre for HeAlth Research at the Management School (CHaRMS) Contact information at EDIRC.
Bibliographic data for series maintained by Mark McGovern ( this e-mail address is bad, please contact ).