Hidden Experts in the Crowd: Using Meta-Predictions to Leverage Expertise in Single-Question Prediction Problems

Tom Wilkening (), Marcellin Martinie () and Piers D. L. Howe ()
Additional contact information
Tom Wilkening: Department of Economics, The University of Melbourne, Parkville, Victoria 3010, Australia
Marcellin Martinie: Melbourne School of Psychological Sciences, The University of Melbourne, Parkville, Victoria 3010, Australia
Piers D. L. Howe: Melbourne School of Psychological Sciences, The University of Melbourne, Parkville, Victoria 3010, Australia

Management Science, 2022, vol. 68, issue 1, 487-508

Abstract: Modern forecasting algorithms use the wisdom of crowds to produce forecasts better than those of the best identifiable expert. However, these algorithms may be inaccurate when crowds are systematically biased or when expertise varies substantially across forecasters. Recent work has shown that meta-predictions—a forecast of the average forecasts of others—can be used to correct for biases even when no external information, such as forecasters’ past performance, is available. We explore whether meta-predictions can also be used to improve forecasts by identifying and leveraging the expertise of forecasters. We develop a confidence-based version of the Surprisingly Popular algorithm proposed by Prelec, Seung, and McCoy. As with the original algorithm, our new algorithm is robust to bias. However, unlike the original algorithm, our version is predicted to always weight forecasters with more informative private signals more than forecasters with less informative ones. In a series of experiments, we find that the modified algorithm does a better job in weighting informed forecasters than the original algorithm and show that individuals who are correct more often on similar decision problems contribute more to the final decision than other forecasters. Empirically, the modified algorithm outperforms the original algorithm for a set of 500 decision problems.

Keywords: expertise; meta-knowledge; wisdom of crowds; forecasting; aggregation (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.2020.3919 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:68:y:2022:i:1:p:487-508

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().