Causal inference with textual data: A quasi-experimental design assessing the association between author metadata and acceptance among ICLR submissions from 2017 to 2022

Chen Chang (), Zhang Jiayao (), Ye Ting (), Roth Dan () and Zhang Bo ()
Additional contact information
Chen Chang: Department of Biostatistics, University of North Carolina at Chapel Hill, NC 27514, United States
Zhang Jiayao: Department of Computer and Information Science and Department of Statistics and Data Science, University of Pennsylvania, PA 19104, United States
Ye Ting: Department of Biostatistics, University of Washington, WA 98195, United States
Roth Dan: Department of Computer and Information Science, University of Pennsylvania, PA 19104, United States
Zhang Bo: Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center, WA 98109, United States

Journal of Causal Inference, 2024, vol. 12, issue 1, 20

Abstract: Many recent studies have probed status bias in the peer-review process of academic journals and conferences. In this article, we investigated the association between author metadata and area chairs’ final decisions (Accept/Reject) using our compiled database of 5,313 borderline submissions to the International Conference on Learning Representations from 2017 to 2022 under a matched observational study framework. We carefully defined elements in a cause-and-effect analysis, including the treatment and its timing, pre-treatment variables, potential outcomes (POs) and causal null hypothesis of interest, all in the context of study units being textual data and under Neyman and Rubin’s PO framework. We found some weak evidence that author metadata was associated with articles’ final decisions. We also found that, under an additional stability assumption, borderline articles from high-ranking institutions (top-30% or top-20%) were less favored by area chairs compared to their matched counterparts. The results were consistent in two different matched designs (odds ratio = 0.82 [95% confidence interval (CI): 0.67 to 1.00] in a first design and 0.83 [95% CI: 0.64 to 1.07] in a strengthened design) and most pronounced in the subgroup of articles with low ratings. We discussed how to interpret these results in the context of multiple interactions between a study unit and different agents (reviewers and area chairs) in the peer-review system.

Keywords: matched observational study; natural language processing; peer-review; quasi-experimental design; status bias (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1515/jci-2023-0052 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bpj:causin:v:12:y:2024:i:1:p:20:n:1001

DOI: 10.1515/jci-2023-0052

Access Statistics for this article

Journal of Causal Inference is currently edited by Elias Bareinboim, Jin Tian and Iván Díaz

More articles in Journal of Causal Inference from De Gruyter
Bibliographic data for series maintained by Peter Golla ().