Selective publication of findings: Why does it matter, and what should we do about it?
Maximilian Kasy ()
No xwngs, MetaArXiv from Center for Open Science
This essay argues that different justifiable objectives for scientific institutions lead to contradictory recommendations. We need to be explicit about our objectives in order to discuss the tradeoffs between them. Replicability and the validity of conventional statistical inference constitute one such objective, and they indeed require that publication decisions do not depend on findings. This is what motivates much of current reform efforts. Validity of inference is presumably not the only objective, however -- it could easily be achieved by estimates derived from a random number generator. Relevance of findings might be another objective. If our goal is to inform decision makers or to maximize social learning, there is a strong rationale to selectively publish surprising findings. A third objective could be the plausibility of published findings. If there is some uncertainty about the quality of studies and we want to avoid publishing incorrect results, we might want to selectively publish unsurprising findings. How can we resolve the tension between these contradictory recommendations? I will outline one possibility below, proposing a functionally differentiated publication system, with different outlets focusing on different objectives. Measures that are promoted by current reformers, such as pre-analysis plans and registered reports, would have to play a crucial role in such a system. Following these policy proposals, I will take a step back and argue that these debates raise some fundamental questions for statistical theory. In order to coherently discuss these issues, statistical theory needs a model of the work of empirical research that goes beyond the single-agent model of statistical decision theory. We should understand statistics (quantitative empirical research) as a social process of communication and collective learning that involves many different actors with differences in knowledge and expertise, different objectives, and constraints on their attention and time, who engage in strategic behavior.
New Economics Papers: this item is included in nep-hme
References: Add references at CitEc
Citations: Track citations by RSS feed
Downloads: (external link)
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:xwngs
Access Statistics for this paper
More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().