Boosting the Wisdom of Crowds Within a Single Judgment Problem: Weighted Averaging Based on Peer Predictions

Asa B. Palley () and Ville A. Satopää ()
Additional contact information
Asa B. Palley: Kelley School of Business, Indiana University, Bloomington, Indiana 47405
Ville A. Satopää: Technology and Operations Management Department, INSEAD, 77305 Fontainebleau, France

Management Science, 2023, vol. 69, issue 9, 5128-5146

Abstract: A combination of point estimates from multiple judges often provides a more accurate aggregate estimate than a point estimate from a single judge, a phenomenon called “the wisdom of crowds.” However, if the judges use shared information when forming their estimates, the simple average will end up overemphasizing this common component at the expense of the judges’ private information. A decision maker could in theory obtain a more accurate estimate by appropriately combining all information behind the judges’ opinions. Although this information underlies the judges’ individual estimates, it is typically unobservable and thus cannot be directly aggregated by a decision maker. In this article, we propose a weighting of judges’ individual estimates that appropriately combines their collective information within a single estimation problem. Judges are asked to provide both a point estimate of the quantity of interest and a prediction of the average estimate that will be given by all other judges. Predictions of others are then used as part of a criterion to determine weights that are applied to each judge’s estimate to form an aggregate estimate. Our weighting procedure is robust to noise in the judges’ responses and can be expressed in closed form. We use both simulation and data from a collection of experimental studies to illustrate that the weighting procedure outperforms existing methods. An R package called metaggR implements our method and is available on the Comprehensive R Archive Network.

Keywords: forecasting; estimation; judgment aggregation; wisdom of crowds; shared information (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.2022.4648 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:69:y:2023:i:9:p:5128-5146

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().