A many-designs study of 516 algorithms challenges the predictability of wisdom of the crowd forecast aggregation

Christian König-Kersting (), Yana Litovsky (), Robert Böhm (), Igor Grossmann (), Jürgen Huber, Michael Kirchler and WoCCAP Consortium

Working Papers from Faculty of Economics and Statistics, Universität Innsbruck

Abstract: Accurate forecasts are central to decision-making in many domains. Although aggregating independent judgments can improve forecast accuracy, a phenomenon known as “Wisdom of the Crowd,” methods for effectively combining these forecasts remain underexplored. In this preregistered many-designs study, 129 research teams independently submitted a total of 516 algorithms to aggregate forecasts over six months, across four domains: economics, politics, climate, and sports. By testing hundreds of independently developed aggregation algorithms under identical conditions, we established an empirical benchmark for forecast aggregation. Drawing on forecasts by 1,182 people, we evaluated algorithm accuracy and variability. The share of algorithms significantly outperforming the mean and median varied by domain, with the strongest gains over simple benchmarks in politics and climate. Algorithm performance was also more persistent over time in politics and climate than in economics and sports. When asked to predict algorithm performance, the participating researchers were overconfident about the success of their own submissions and of others. We also examined potential predictors of algorithm accuracy and variability. Although researchers expected various features to predict accuracy, no algorithmic or researcher feature consistently explained why algorithms varied in performance. The main exception was the use by algorithms of previously observed realized outcomes, typically to compute past forecast error or train a model, which predicted greater accuracy in politics and climate. The null findings contrast with prior evidence that certain weighting criteria can improve forecast aggregation, and illustrate the value of preregistered many-designs studies which report the full distribution of submitted approaches, revealing when previously supported approaches fail to generalize.

Keywords: Meta-science; Forecasting; Wisdom of the Crowd (search for similar items in EconPapers)
Pages: 47
Date: 2026-05
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www2.uibk.ac.at/downloads/c9821000/wpaper/2026-05.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inn:wpaper:2026-05

Access Statistics for this paper

More papers in Working Papers from Faculty of Economics and Statistics, Universität Innsbruck Contact information at EDIRC.
Bibliographic data for series maintained by Judith Courian ( this e-mail address is bad, please contact ).