Using simulations to explore sampling distributions: an antidote to hasty and extravagant inferences

Guillaume A Rousselet

No f5q7r, OSF Preprints from Center for Open Science

Abstract: Most statistical inferences in psychology are based on frequentist statistics, which rely on sampling distributions: the long-run outcomes of multiple experiments, given a certain model. Yet, sampling distributions are poorly understood and rarely explicitly considered when making inferences. In this article, I demonstrate how to use simulations to illustrate sampling distributions to answer simple practical questions: for instance, if we could run thousands of experiments, what would the outcome look like? What do these simulations tell us about the results from a single experiment? Such simulations can be run a priori, given expected results, or a posteriori, using existing datasets. Both approaches can help make explicit the data generating process and the sources of variability; they also reveal the large variability in our experimental estimation and lead to the sobering realisation that, in most situations, we should not make a big deal out of results from a single experiment. Simulations can also help demonstrate how the selection of effect sizes conditional on some arbitrary cut-off (p≤0.05) leads to a literature crammed with false positives, a powerful illustration of the damage done in part by researchers’ over-confidence in their statistical tools. The article focuses on graphical descriptions and covers examples using correlation analyses, percent correct data and response latency data.

Date: 2019-12-05
New Economics Papers: this item is included in nep-exp and nep-rmg
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/5de98421eb3ee6000b89b52a/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:f5q7r

DOI: 10.31219/osf.io/f5q7r

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().