Bayesian Specification Curve Analysis

Christoph Semken and David Rossell

No cahyq, OSF Preprints from Center for Open Science

Abstract: Science suffers from a reproducibility crisis. Specification Curve Analysis (SCA) helps address this crisis by preventing the selective reporting of results and arbitrary data analysis choices. SCA plots the variability (or heterogeneity) of treatment effects against all ‘reasonable specifications’ (ways to conduct analysis). However, SCA has also been used for formal statistical inference on a type of global average (median) treatment effect (ATE), leading a study by Orben & Przybylski to conclude that ‘the association of [adolescent mental] well-being with regularly eating potatoes was nearly as negative as the association with technology use.’ In contrast, we find relevant associations between certain technologies and well-being, and sharp discrepancies between parent and teenager assessments. These heterogeneous effects are masked by taking medians. In layman’s terms, an ATE may appear practically irrelevant due to averaging over apples and oranges. In addition, the SCA median can have large bias and variance, due to over-weighting statistically implausible control variable specifications. With the Bayesian Specification Curve Analysis (BSCA) we extend SCA to estimate both individual and, if desired, average treatment effects, with controls weighted via Bayesian Model Averaging. The strategy allows to test individual effects, a missing feature in SCA, while improving statistical properties and protecting against false positives. We provide R code that implements BSCA and reproduces our analyses.

Date: 2020-10-26
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/5fa57ec323568c0210d653f8/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:cahyq

DOI: 10.31219/osf.io/cahyq

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().