Aligning agent-based testing (ABT) with the experimental research paradigm: a literature review and best practices

Patrick Schwabl (), Mario Haim and Julian Unkel
Additional contact information
Patrick Schwabl: LMU Munich
Mario Haim: LMU Munich
Julian Unkel: LMU Munich

Journal of Computational Social Science, 2024, vol. 7, issue 2, No 18, 1625-1644

Abstract: Abstract The study of algorithmically curated media environments through emulated browsing has become a key method of computational social science. Here, we review underlying concepts and typical implementations of these so-called agent-based testing (ABT) studies, particularly those that experimentally vary inputs on search engines, social media, or aggregation platforms, to study subsequent outputs. This review aims to identify, evaluate, and discuss implementations and reporting standards of ABT studies within the experimental research paradigm. First, we discuss general assumptions of the experimental research paradigm and to what extent ABT experiments align with them, finding a considerable overlap with minor deviations. Second, we systematically reviewed 66 ABT studies from the social and information sciences, finding little reference to the experimental research paradigm vis-à-vis large variations in implementation and reporting practices. Third, our findings then inform five best-practice guidelines for future ABT experiments where we suggest to explicitly and much more strictly align those studies with the experimental research paradigm. We argue that ABT experiments would benefit from such an adaptation of already established experimental practices, thereby improving reproducibility and replicability.

Keywords: Agent-based testing; Algorithmic content curation; Standardization; Data collection; Computational social science (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s42001-024-00283-6 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:jcsosc:v:7:y:2024:i:2:d:10.1007_s42001-024-00283-6

Ordering information: This journal article can be ordered from
http://www.springer. ... iences/journal/42001

DOI: 10.1007/s42001-024-00283-6

Access Statistics for this article

Journal of Computational Social Science is currently edited by Takashi Kamihigashi

More articles in Journal of Computational Social Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().