When high reliability does not signal reliable detection of experimental effects

Selena Wang and Paul De Boeck

No gz8pw, OSF Preprints from Center for Open Science

Abstract: The replication crisis has led to a renewed discussion about the impacts of measurement quality on the precision of psychology research. High measurement quality is associated with low measurement error, yet the relationship between reliability and measurement error is not yet well understood. In this study, we attempt to understand the role of reliability in the replication research through its relationship with power. We outline a latent variable framework to understand this nuanced relationship through their equations. We find that, under the influence of the latent variable variance, reliability and power can be negatively associated. We proceed to demonstrate the impact of this relationship between reliability and power in replication research by imitating different scenarios of large-scale replications. We find negative correlations between reliability and power when there are sizable differences in the latent variable variance and negligible differences in the other parameters across studies. Finally, we apply our conclusions to the data from replications of the ego depletion effect (Hagger et al., 2016) and the replications of the grammatical aspect effect (Eerland et al., 2016). The applications show that a negative relationship between reliability and power is a realistic possibility, which eliminates doubts one might have about the applicability of our results to the applied work.

Date: 2020-03-01
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/5e5be8c4b070fc01fbe5856e/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:gz8pw

DOI: 10.31219/osf.io/gz8pw

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().