Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model

Ville A. Satopää, Marat Salikhov, Philip E. Tetlock and Barbara Mellers

International Journal of Forecasting, 2023, vol. 39, issue 1, 470-485

Abstract: Aggregating predictions from multiple judges often yields more accurate predictions than relying on a single judge, which is known as the wisdom-of-the-crowd effect. However, a wide range of aggregation methods are available, which range from one-size-fits-all techniques, such as simple averaging, prediction markets, and Bayesian aggregators, to customized (supervised) techniques that require past performance data, such as weighted averaging. In this study, we applied a wide range of aggregation methods to subjective probability estimates from geopolitical forecasting tournaments. We used the bias–information–noise (BIN) model to disentangle three mechanisms that allow aggregators to improve the accuracy of predictions: reducing bias and noise, and extracting valid information across forecasters. Simple averaging operates almost entirely by reducing noise, whereas more complex techniques such as prediction markets and Bayesian aggregators exploit all three pathways to allow better signal extraction as well as greater noise and bias reduction. Finally, we explored the utility of a BIN approach for the modular construction of aggregators.

Keywords: Judgmental forecasting; Partial information; Prediction markets; Wisdom of crowds; Bayesian Statistics Shapley Value (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0169207021002168
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:intfor:v:39:y:2023:i:1:p:470-485

DOI: 10.1016/j.ijforecast.2021.12.010

Access Statistics for this article

International Journal of Forecasting is currently edited by R. J. Hyndman

More articles in International Journal of Forecasting from Elsevier
Bibliographic data for series maintained by Catherine Liu ().