Performance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd

Robert N. Collins, David R. Mandel and David V. Budescu
Additional contact information
Robert N. Collins: Intelligence, Influence, and Collaboration Section, Defence Research & Development Canada
David R. Mandel: Intelligence, Influence, and Collaboration Section, Defence Research & Development Canada
David V. Budescu: Fordham University

Chapter Chapter 7 in Judgment in Predictive Analytics, 2023, pp 185-214 from Springer

Abstract: Abstract The benefits of judgment aggregation are intuitive and well-documented. By combining the input of several judges, practitioners may enhance information sharing and signal strength while cancelling out biases and noise. The resulting judgment is more accurate than the average accuracy of the individual judgments—a phenomenon known as the wisdom of crowds. Although an unweighted arithmetic average is often sufficient to improve judgment accuracy, sophisticated performance-weighting methods have been developed to further improve accuracy. By weighting the judges according to: (1) past performance on similar tasks, (2) performance on closely related tasks, and/or (3) the internal consistency (or coherence) of judgments, practitioners can exploit individual differences in probabilistic judgment skill to ferret out bone fide experts within the crowd. Each method has proven useful, with associated benefits and potential drawbacks. In this chapter, we review the evidence for-and-against performance weighting strategies, discussing the circumstances in which they are appropriate and beneficial to apply. We describe how to implement these methods, with a focus on mathematical functions and formulas that translate performance metrics into aggregation weights.

Keywords: Judgment; Weighted aggregation; Accuracy; Correspondence (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:isochp:978-3-031-30085-1_7

Ordering information: This item can be ordered from
http://www.springer.com/9783031300851

DOI: 10.1007/978-3-031-30085-1_7

Access Statistics for this chapter

More chapters in International Series in Operations Research & Management Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().