Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments

Abigail R Colson and Roger Cooke

Review of Environmental Economics and Policy, 2018, vol. 12, issue 1, 113-132

Abstract: The inclusion of expert judgments along with other forms of data in science, engineering, and decision making is inevitable. Expert elicitation refers to formal procedures for obtaining and combining expert judgments. Expert elicitation is required when existing data and models cannot provide needed information. This makes validating expert judgments a challenge because they are used when other data do not exist and thus measuring their accuracy is difficult. This article examines the classical model of structured expert judgment, which is an elicitation method that includes validation of the experts’ assessments against empirical data. In the classical model, experts assess both the unknown target questions and a set of calibration questions, which are items from the experts’ field that have observed true values. The classical model scores experts on their performance in assessing the calibration questions and then produces performance-weighted combinations of the experts. From 2006 through March 2015, the classical model has been used in thirty-three unique applications. Less than one-third of the individual experts in these studies were statistically accurate, highlighting the need for validation. Overall, the performance-based combination of experts produced in the classical model is more statistically accurate and more informative than an equal weighting of experts.

Date: 2018
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (17)

Downloads: (external link)
http://hdl.handle.net/10.1093/reep/rex022 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:renvpo:v:12:y:2018:i:1:p:113-132.

Ordering information: This journal article can be ordered from
https://academic.oup.com/journals

Access Statistics for this article

Review of Environmental Economics and Policy is currently edited by Robert Stavins

More articles in Review of Environmental Economics and Policy from Association of Environmental and Resource Economists Oxford University Press, Great Clarendon Street, Oxford OX2 6DP, UK. Contact information at EDIRC.
Bibliographic data for series maintained by Oxford University Press ().