Improving reproducibility probability estimation and preserving RP-testing

Lucio De Capitani () and Daniele De Martini ()
Additional contact information
Lucio De Capitani: University of Milano-Bicocca
Daniele De Martini: University of Milano-Bicocca

Statistical Methods & Applications, 2021, vol. 30, issue 1, No 2, 49-77

Abstract: Abstract Experimental results are often interpreted through statistical tests, where the alternative hypothesis represents the theory to be evinced; if the experimental results lead to the rejection of the null hypothesis, the theory is supported by empirical evidence. In these cases, the reproducibility of this empirical evidence can be measured by the Reproducibility Probability (RP) of the test, which coincides with the probability of rejecting the null hypothesis. The terminology “Reproducibility” Probability stems from the fact that it is usually computed when an experiment provides a significant result to evaluate the probability that a further identical and independent experiment confirms the statistical significance. In recent literature, some RP estimators have been proposed. They are useful for two reasons: they allow us to evaluate the reliability of the obtained statistical significance and some estimates can be used as a test statistic, owing to the so-called “RP-testing” decision rule (reject the null hypothesis if and only if the RP estimate is greater than 1/2). Unfortunately, the usually adopted RP estimators are affected by a high mean squared error. In this paper, a new class of RP estimators is introduced and examined to improve their estimation precision. Specifically, the performances of the new RP estimators have been compared with those of the existing estimators and a 30% greater reduction in the mean squared error (on average) was observed. Moreover, the new estimator with the best performance allowed the use of the RP-testing decision rule. Hence, this work achieves the double goal of improving Reproducibility Probability estimation and preserving RP-testing.

Keywords: Conservative RP estimation; Bayesian RP estimation; Average conservative RP estimation; RP-testing (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10260-020-00513-x Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:stmapp:v:30:y:2021:i:1:d:10.1007_s10260-020-00513-x

Ordering information: This journal article can be ordered from
http://www.springer. ... cs/journal/10260/PS2

DOI: 10.1007/s10260-020-00513-x

Access Statistics for this article

Statistical Methods & Applications is currently edited by Tommaso Proietti

More articles in Statistical Methods & Applications from Springer, Società Italiana di Statistica
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().