Do Replications Make a Difference?

Tom Coupé and W. Reed ()

Working Papers in Economics from University of Canterbury, Department of Economics and Finance

Abstract: This study examines the effect of negative replications on the citation rates of replicated studies. It makes three contributions. First, we explain why previous research has not adequately addressed this subject. Second, we develop a matched difference-in-difference (DID) procedure that does not assume parallel trends (PT). Previous research has shown that studies that fail to replicate have different trends prior to replication than studies that successfully replicate. Given this difference, imposing the assumption of PT biases estimation of citation effects. Our DID procedure avoids this bias. Lastly, we study a set of 204 replicated studies and investigate whether there is a citation penalty associated with negative replications. Replicated studies are matched with non-replicated studies, with the matched controls being used to “predict” the counterfactual citation performance of replicated studies. Our preferred estimates indicate that studies that fail to replicate receive more citations than studies that have positive or mixed replications. Our less preferred estimates, based on looser matching criteria, find evidence of a citation penalty for negative replications, but the estimated effects are small and statistically insignificant. We conclude that replications have not been correcting the scientific record in the manner that proponents might have hoped.

Keywords: Replications; Citations; Matching; Meta-science; Self-correcting science (search for similar items in EconPapers)
JEL-codes: A11 A14 B41 C18 (search for similar items in EconPapers)
Pages: 42 pages
Date: 2023-08-01
New Economics Papers: this item is included in nep-dem, nep-hpe and nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://repec.canterbury.ac.nz/cbt/econwp/2310.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cbt:econwp:23/10

Access Statistics for this paper

More papers in Working Papers in Economics from University of Canterbury, Department of Economics and Finance Private Bag 4800, Christchurch, New Zealand. Contact information at EDIRC.
Bibliographic data for series maintained by Albert Yee ().