Open Science in Impact Evaluation: What Impact Evaluators can Learn from the Replication Crisis in Social Psychology

Ben McNamee, Patrick S. Forscher and Michael Lenczner
Additional contact information
Patrick S. Forscher: University of Wisconsin-Madison

No cwftm_v1, MetaArXiv from Center for Open Science

Abstract: Since 2011, the academic field of social psychology has been undergoing a crisis of confidence in its results. This so-called "replication crisis" has inspired a widespread examination of the practices that can lead to less than credible results, the incentives that inspire such practices, and the practices that could protect or enhance research credibility. We believe that the credibility of the field of impact evaluation may be similarly threatened by incentives to use poor research practices. We assess the history of both social psychology and impact evaluation and conclude that, although the source of the incentives for impact evaluators is different in kind than those for social psychologists, impact evaluators may be similarly incentivized to make their results look better than they really are. We review three parallels in the history of the two fields, including the use of research as sales, the pressurized, competitive environments in which the work takes place, and the use of research methods as rhetorical devices. Impact evaluators may be able to learn from the credibility-enhancing solutions that worked for social psychology, though the field can consider new tools, such as standards, investment mechanisms, administrative data labs, and innovations inevidence-sharing and synthesis. Rather than sleepwalking through a crisis we cannot see, impact evaluators should consider that their field may be in crisis, a crisis that may require widespread and collective efforts –a credibility renaissance – to discover and deploy methods to protect its legitimacy.

Date: 2025-02-21
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/67dfba69bb0557890f053b5a/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:cwftm_v1

DOI: 10.31219/osf.io/cwftm_v1

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().