Probabilistic Coherence Weighting for Optimizing Expert Forecasts

Christopher W. Karvetski (), Kenneth C. Olson (), David R. Mandel () and Charles R. Twardy ()
Additional contact information
Christopher W. Karvetski: Department of Applied Information Technology, George Mason University, Fairfax, Virginia 22030
Kenneth C. Olson: Department of Applied Information Technology, George Mason University, Fairfax, Virginia 22030
David R. Mandel: Socio-Cognitive Systems Section, DRDC Toronto; and Department of Psychology, York University, Toronto, Ontario M3J 1P3, Canada
Charles R. Twardy: Command, Control, Communications, Computing, and Intelligence Center, George Mason University, Fairfax, Virginia 22030

Decision Analysis, 2013, vol. 10, issue 4, 305-326

Abstract: Methods for eliciting and aggregating expert judgment are necessary when decision-relevant data are scarce. Such methods have been used for aggregating the judgments of a large, heterogeneous group of forecasters, as well as the multiple judgments produced from an individual forecaster. This paper addresses how multiple related individual forecasts can be used to improve aggregation of probabilities for a binary event across a set of forecasters. We extend previous efforts that use probabilistic incoherence of an individual forecaster's subjective probability judgments to weight and aggregate the judgments of multiple forecasters for the goal of increasing the accuracy of forecasts. With data from two studies, we describe an approach for eliciting extra probability judgments to (i) adjust the judgments of each individual forecaster, and (ii) assign weights to the judgments to aggregate over the entire set of forecasters. We show improvement of up to 30% over the established benchmark of a simple equal-weighted averaging of forecasts. We also describe how this method can be used to remedy the “fifty--fifty blip” that occurs when forecasters use the probability value of 0.5 to represent epistemic uncertainty.

Keywords: probabilistic coherence; forecast aggregation; crowdsourcing; linear opinion pool; fifty--fifty blip; practice (search for similar items in EconPapers)
Date: 2013
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (12)

Downloads: (external link)
http://dx.doi.org/10.1287/deca.2013.0279 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ordeca:v:10:y:2013:i:4:p:305-326

Access Statistics for this article

More articles in Decision Analysis from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().