Does big data serve policy? Not without context. An experiment with in silico social science

Chris Graziul, Alexander Belikov, Ishanu Chattopadyay, Ziwen Chen, Hongbo Fang, Anuraag Girdhar, Xiaoshuang Jia, P. M. Krafft, Max Kleiman-Weiner, Candice Lewis, Chen Liang, John Muchovej, Alejandro Vientós, Meg Young and James Evans ()
Additional contact information
Chris Graziul: University of Chicago
Alexander Belikov: University of Chicago
Ishanu Chattopadyay: University of Chicago
Ziwen Chen: University of Chicago
Hongbo Fang: Carnegie Mellon University
Anuraag Girdhar: University of Chicago
Xiaoshuang Jia: Sun Yat-sen University
P. M. Krafft: University of Oxford
Max Kleiman-Weiner: MIT
Candice Lewis: University of Chicago
Chen Liang: University of Chicago
John Muchovej: MIT
Alejandro Vientós: MIT
Meg Young: Cornell University
James Evans: University of Chicago

Computational and Mathematical Organization Theory, 2023, vol. 29, issue 1, No 7, 188-219

Abstract: Abstract The DARPA Ground Truth project sought to evaluate social science by constructing four varied simulated social worlds with hidden causality and unleashed teams of scientists to collect data, discover their causal structure, predict their future, and prescribe policies to create desired outcomes. This large-scale, long-term experiment of in silico social science, about which the ground truth of simulated worlds was known, but not by us, reveals the limits of contemporary quantitative social science methodology. First, problem solving without a shared ontology—in which many world characteristics remain existentially uncertain—poses strong limits to quantitative analysis even when scientists share a common task, and suggests how they could become insurmountable without it. Second, data labels biased the associations our analysts made and assumptions they employed, often away from the simulated causal processes those labels signified, suggesting limits on the degree to which analytic concepts developed in one domain may port to others. Third, the current standard for computational social science publication is a demonstration of novel causes, but this limits the relevance of models to solve problems and propose policies that benefit from the simpler and less surprising answers associated with most important causes, or the combination of all causes. Fourth, most singular quantitative methods applied on their own did not help to solve most analytical challenges, and we explored a range of established and emerging methods, including probabilistic programming, deep neural networks, systems of predictive probabilistic finite state machines, and more to achieve plausible solutions. However, despite these limitations common to the current practice of computational social science, we find on the positive side that even imperfect knowledge can be sufficient to identify robust prediction if a more pluralistic approach is applied. Applying competing approaches by distinct subteams, including at one point the vast TopCoder.com global community of problem solvers, enabled discovery of many aspects of the relevant structure underlying worlds that singular methods could not. Together, these lessons suggest how different a policy-oriented computational social science would be than the computational social science we have inherited. Computational social science that serves policy would need to endure more failure, sustain more diversity, maintain more uncertainty, and allow for more complexity than current institutions support.

Keywords: Computational social science; Simulated societies; Policy; Quantitative social science; Machine learning; Deep learning; Simulation (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10588-022-09362-3 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:comaot:v:29:y:2023:i:1:d:10.1007_s10588-022-09362-3

Ordering information: This journal article can be ordered from
http://www.springer.com/journal/10588

DOI: 10.1007/s10588-022-09362-3

Access Statistics for this article

Computational and Mathematical Organization Theory is currently edited by Terrill Frantz and Kathleen Carley

More articles in Computational and Mathematical Organization Theory from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().