EconPapers    
Economics at your fingertips  
 

Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias

Christopher Snyder and Ran Zhuo

No 8vdrh, MetaArXiv from Center for Open Science

Abstract: The increasing demand for rigor in empirical economics has led to the growing use of auxiliary tests (balance, specification, over-identification, placebo, etc.) supporting the credibility of a paper’s main results. We dub these “sniff tests” because standards for passing are subjective and rejection is bad news for the author. Sniff tests offer a new window into publication bias since authors prefer them to be insignificant, the reverse of standard statistical tests. Collecting a sample of nearly 30,000 sniff tests across 60 economics journals, we provide the first estimate of their aggregate probability-value (p-value) distribution. For the subsample of balance tests in randomized controlled trials (for which the distribution of p-values is known to be uniform absent publication bias, allowing reduced-form methods to be employed) estimates suggest that 45% of failed tests remain in the “file drawer” rather than being published. For the remaining sample with an unknown distribution of p-values, structural estimates suggest an even larger file-drawer problem, as high as 91%. Fewer significant sniff tests show up in top-tier journals, smaller tables, and more recent articles. We find no evidence of author manipulation other than a tendency to overly attribute significant sniff tests to bad luck.

New Economics Papers: this item is included in nep-hme, nep-hpe and nep-sog
Date: 2018-11-30
References: View references in EconPapers View complete reference list from CitEc
Citations: Track citations by RSS feed

Downloads: (external link)
https://osf.io/download/5c015f44f5d46c001a23ce09/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:8vdrh

DOI: 10.31219/osf.io/8vdrh

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().

 
Page updated 2020-01-17
Handle: RePEc:osf:metaar:8vdrh