Leakage explains the apparent superiority of Bayesian random effect models – a preregistered comment on Claessens, Kyritsis and Atkinson (2023)

Sascha Wolfer and Alexander Koplenig
Additional contact information
Sascha Wolfer: Leibniz Institute for the German Language

No ex267, OSF Preprints from Center for Open Science

Abstract: In a previous study, Claessens, Kyritsis, and Atkinson (CKA) demonstrated the importance of controlling for geographic proximity and cultural similarity in cross-national analyses. Based on a simulation study, CKA showed that methods commonly used to control for spatial and cultural non-independence are insufficient in reducing false positives while maintaining the ability to detect true effects. CKA strongly advocate the use of Bayesian random effect models in such situations, arguing that among the studied model types, they are the only ones that reduced false positives while maintaining high statistical power. However, in this comment, we argue that the apparent superiority of such models is overstated by CKA due to a form of methodological circularity called 'leakage' in statistics and machine learning, because the same proximity matrix is used both to generate the simulated data and as an input to only the Bayesian models for comparison. When this leakage is controlled for, we show that Bayesian models do not outperform most other methods.

Date: 2024-07-30
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/66a7593e65e93aa1f1c9abbb/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:ex267

DOI: 10.31219/osf.io/ex267

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().