Practical alternatives to the “significant/non-significant” statistical dichotomy

Alessandro Rovetta
Additional contact information
Alessandro Rovetta: Mensana srls

No nkd34, OSF Preprints from Center for Open Science

Abstract: Science is a discipline that selects, collects, and summarizes the best available information on a given phenomenon in order to model it. It is closer to a bet with the highest probability of winning than to the search for truth. The dichotomous categorization of significance (e.g., acceptance vs. rejection of objective hypotheses) is unsuitable for the scientific nature of a statistical investigation and inevitably leads to dangerous exaggerations of the state of evidence (especially on single studies). Indeed, no study can prove that a result is (not) significant (not even at the clinical level) since uncertainty is and always will be part of scientific action. At most, it can be stated that a certain result, in terms of both statistical surprise and effect size, together with other evidence (e.g., biological, psychological, etc.), provides a certain degree of information “against” or “in favor of” substantial consequences in the clinical field. Therefore, based on the above considerations and an analysis of the (function of) costs and benefits, it can be decided whether a certain therapy meets the threshold of scientific evidence required for its approval. In this regard, this manuscript proposes and discusses alternative concepts to statistical dichotomy such as ranges of significance (or, better, ranges of compatibility with hypotheses) or the recommended evaluation of results in terms of statistical surprise (by comparison with obtaining S consecutive heads when flipping an unbiased coin). Furthermore, it emphasizes the absolute necessity to investigate all relevant objective hypotheses (not just the null one) as well as all model assumptions (the violation of which can make significance/compatibility/surprise uninformative for the scientific goal). Finally, it proposes a compact framework for a complete presentation of results.

Date: 2023-05-13
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/645f47017b916a2977299b26/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:nkd34

DOI: 10.31219/osf.io/nkd34

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().