Researchers’ data analysis choices: an excess of false positives?

James A. Ohlson ()
Additional contact information
James A. Ohlson: Hong Kong Polytechnic University

Review of Accounting Studies, 2022, vol. 27, issue 2, No 8, 649-667

Abstract: Abstract This paper examines commonly applied methods of data analysis. Predicated on these methods, the main issue pertains to the plausibility of the studies’ end products, that is, their conclusions. I argue that the methods chosen often lead to unwarranted conclusions: the data analyses chosen tend to produce looked-for null rejections even though the null may be much more plausible on prior grounds. Two aspects of data analyses applied cause obvious problems. First, researchers tend to dismiss “preliminary” findings when the findings contradict the expected outcome of the research question (the “screen-picking” issue). Second, researchers rarely acknowledge that small p-values should be expected when the number of observations runs into the tens of thousands (the “large N” issue). This obviously enhances the chance for a null rejection even if the null hypothesis holds for all practical purposes. The discussion elaborates on these two aspects to explain why researchers generally avoid trying to mitigate false positives via supplementary data analyses. In particular, for no apparent good reasons, most research studiously avoids the use of hold-out samples. An additional topic in this paper concerns the dysfunctional consequences of the standard (“A-journal”) publication process, which tends to buttress the use of research methods prone to false or unwarranted null-rejections.

Keywords: Data analysis; False positives; Publication process (search for similar items in EconPapers)
JEL-codes: C18 (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (4)

Downloads: (external link)
http://link.springer.com/10.1007/s11142-021-09620-w Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:reaccs:v:27:y:2022:i:2:d:10.1007_s11142-021-09620-w

Ordering information: This journal article can be ordered from
http://www.springer.com/accounting/journal/11142

DOI: 10.1007/s11142-021-09620-w

Access Statistics for this article

Review of Accounting Studies is currently edited by Paul Fischer

More articles in Review of Accounting Studies from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().