Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation

Christopher W. Karvetski, David R. Mandel and Daniel Irwin

Risk Analysis, 2020, vol. 40, issue 5, 1040-1057

Abstract: As in other areas of expert judgment, intelligence analysis often requires judging the probability that hypotheses are true. Intelligence organizations promote the use of structured methods such as “Analysis of Competing Hypotheses” (ACH) to improve judgment accuracy and analytic rigor, but these methods have received little empirical testing. In this experiment, we pitted ACH against a factorized Bayes's theorem (FBT) method, and we examined the value of recalibration (coherentization) and aggregation methods for improving the accuracy of probability judgment. Analytic techniques such as ACH and FBT were ineffective in improving accuracy and handling correlated evidence, and ACH in fact decreased the coherence of probability judgments. In contrast, statistical postanalytic methods (i.e., coherentization and aggregation) yielded large accuracy gains. A wide range of methods for instantiating these techniques were tested. The interactions among the factors considered suggest that prescriptive theorists and interventionists should examine the value of ensembles of judgment‐support methods.

Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (2)

Downloads: (external link)
https://doi.org/10.1111/risa.13443

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:riskan:v:40:y:2020:i:5:p:1040-1057

Access Statistics for this article

More articles in Risk Analysis from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().