Aggregating Large Sets of Probabilistic Forecasts by Weighted Coherent Adjustment

Guanchun Wang (), Sanjeev R. Kulkarni (), H. Vincent Poor () and Daniel N. Osherson ()
Additional contact information
Guanchun Wang: Department of Electrical Engineering, Princeton University, Princeton, New Jersey 08544
Sanjeev R. Kulkarni: Department of Electrical Engineering, Princeton University, Princeton, New Jersey 08544
H. Vincent Poor: Department of Electrical Engineering, Princeton University, Princeton, New Jersey 08544
Daniel N. Osherson: Department of Psychology, Princeton University, Princeton, New Jersey 08544

Decision Analysis, 2011, vol. 8, issue 2, 128-144

Abstract: Probability forecasts in complex environments can benefit from combining the estimates of large groups of forecasters (“judges”). But aggregating multiple opinions raises several challenges. First, human judges are notoriously incoherent when their forecasts involve logically complex events. Second, individual judges may have specialized knowledge, so different judges may produce forecasts for different events. Third, the credibility of individual judges might vary, and one would like to pay greater attention to more trustworthy forecasts. These considerations limit the value of simple aggregation methods like unweighted linear averaging. In this paper, a new algorithm is proposed for combining probabilistic assessments from a large pool of judges, with the goal of efficiently implementing the coherent approximation principle (CAP) while weighing judges by their credibility. Two measures of a judge's likely credibility are introduced and used in the algorithm to determine the judge's weight in aggregation. As a test of efficiency, the algorithm was applied to a data set of nearly half a million probability estimates of events related to the 2008 U.S. presidential election ((sim)16,000 judges). Compared with unweighted scalable CAP algorithms, the proposed weighting schemes significantly improved the stochastic accuracy with a comparable run time, demonstrating the efficiency and effectiveness of the weighting methods for aggregating large numbers and varieties of forecasts.

Keywords: judgment aggregation; combining forecasts; weighting; incoherence penalty; consensus deviation (search for similar items in EconPapers)
Date: 2011
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (15)

Downloads: (external link)
http://dx.doi.org/10.1287/deca.1110.0206 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ordeca:v:8:y:2011:i:2:p:128-144

Access Statistics for this article

More articles in Decision Analysis from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().