Shrinking the Variance in Experts’ “Classical” Weights Used in Expert Judgment Aggregation

Gayan Dharmarathne (), Gabriela F. Nane, Andrew Robinson and Anca M. Hanea
Additional contact information
Gayan Dharmarathne: Department of Statistics, University of Colombo, Colombo 00300, Sri Lanka
Gabriela F. Nane: Delft Institute of Applied Mathematics, Delft University of Technology, 2628 CD Delft, The Netherlands
Andrew Robinson: Centre of Excellence for Biosecurity Risk Analysis, School of BioSciences, The University of Melbourne, Melbourne, VIC 3010, Australia
Anca M. Hanea: Centre of Excellence for Biosecurity Risk Analysis, School of BioSciences, The University of Melbourne, Melbourne, VIC 3010, Australia

Forecasting, 2023, vol. 5, issue 3, 1-14

Abstract: Mathematical aggregation of probabilistic expert judgments often involves weighted linear combinations of experts’ elicited probability distributions of uncertain quantities. Experts’ weights are commonly derived from calibration experiments based on the experts’ performance scores, where performance is evaluated in terms of the calibration and the informativeness of the elicited distributions. This is referred to as Cooke’s method, or the classical model (CM), for aggregating probabilistic expert judgments. The performance scores are derived from experiments, so they are uncertain and, therefore, can be represented by random variables. As a consequence, the experts’ weights are also random variables. We focus on addressing the underlying uncertainty when calculating experts’ weights to be used in a mathematical aggregation of expert elicited distributions. This paper investigates the potential of applying an empirical Bayes development of the James–Stein shrinkage estimation technique on the CM’s weights to derive shrinkage weights with reduced mean squared errors. We analyze 51 professional CM expert elicitation studies. We investigate the differences between the classical and the (new) shrinkage CM weights and the benefits of using the new weights. In theory, the outcome of a probabilistic model using the shrinkage weights should be better than that obtained when using the classical weights because shrinkage estimation techniques reduce the mean squared errors of estimators in general. In particular, the empirical Bayes shrinkage method used here reduces the assigned weights for those experts with larger variances in the corresponding sampling distributions of weights in the experiment. We measure improvement of the aggregated judgments in a cross-validation setting using two studies that can afford such an approach. Contrary to expectations, the results are inconclusive. However, in practice, we can use the proposed shrinkage weights to increase the reliability of derived weights when only small-sized experiments are available. We demonstrate the latter on 49 post-2006 professional CM expert elicitation studies.

Keywords: shrinkage estimation; James–Stein; performance weights; classical model; structured expert judgment (search for similar items in EconPapers)
JEL-codes: A1 B4 C0 C1 C2 C3 C4 C5 C8 M0 Q2 Q3 Q4 (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2571-9394/5/3/29/pdf (application/pdf)
https://www.mdpi.com/2571-9394/5/3/29/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jforec:v:5:y:2023:i:3:p:29-535:d:1223595

Access Statistics for this article

Forecasting is currently edited by Ms. Joss Chen

More articles in Forecasting from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().