Extracting the collective wisdom in probabilistic judgments

Cem Peker ()
Additional contact information
Cem Peker: Erasmus University Rotterdam

Theory and Decision, 2023, vol. 94, issue 3, No 5, 467-501

Abstract: Abstract How should we combine disagreeing expert judgments on the likelihood of an event? A common solution is simple averaging, which allows independent individual errors to cancel out. However, judgments can be correlated due to an overlap in their information, resulting in a miscalibration in the simple average. Optimal weights for weighted averaging are typically unknown and require past data to estimate reliably. This paper proposes an algorithm to aggregate probabilistic judgments under shared information. Experts are asked to report a prediction and a meta-prediction. The latter is an estimate of the average of other individuals’ predictions. In a Bayesian setup, I show that if average prediction is a consistent estimator, the percentage of predictions and meta-predictions that exceed the average prediction should be the same. An “overshoot surprise” occurs when the two measures differ. The Surprising Overshoot algorithm uses the information revealed in an overshoot surprise to correct for miscalibration in the average prediction. Experimental evidence suggests that the algorithm performs well in moderate to large samples and in aggregation problems where individuals disagree in their predictions.

Keywords: Wisdom of crowds; Judgment aggregation; Forecasting; Shared information (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11238-022-09899-4 Abstract (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:kap:theord:v:94:y:2023:i:3:d:10.1007_s11238-022-09899-4

Ordering information: This journal article can be ordered from
http://www.springer. ... ry/journal/11238/PS2

DOI: 10.1007/s11238-022-09899-4

Access Statistics for this article

Theory and Decision is currently edited by Mohammed Abdellaoui

More articles in Theory and Decision from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().