Confidence intervals in health and medical journals show an implausible excess of statistically significant results

Adrian Barnett and Jonathan Wren
Additional contact information
Adrian Barnett: Queensland University of Technology

No z8jd6, OSF Preprints from Center for Open Science

Abstract: Results that are statistically significant are more likely to be reported by authors and more likely to be accepted by journals. These common biases warp the published evidence and undermine the ability of research to improve health by giving an incomplete body of evidence. We aimed to show the effect of the bias towards statistical significance on the evidence-base using published confidence intervals. We examined over 968,000 ratios and their confidence intervals in the abstracts of health and medical journals from Medline between 1976 and January 2019. We plotted the distributions of lower and upper confidence interval limits to visually show the strong bias for statistically significant results. There was a striking change in the distributions around 1, which is the statistically significant threshold for ratios. There was an excess of lower intervals just above 1, corresponding to a statistically significant increase in risk. There was a similar excess of upper intervals just below 1, corresponding to a statistically significant decrease in risk. These biases have not improved in recent years. The huge excesses of confidence intervals that are just above and below the statistically significant threshold are not statistically plausible. Large changes in research practice are needed to provide more results that better reflect the truth.

Date: 2019-03-28
New Economics Papers: this item is included in nep-hea
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/5c9c44f9e0a7de001795898a/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:z8jd6

DOI: 10.31219/osf.io/z8jd6

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().