Talent Spotting in Crowd Prediction

Pavel Atanasov () and Mark Himmelstein
Additional contact information
Pavel Atanasov: Pytho LLC
Mark Himmelstein: Fordham University

Chapter Chapter 6 in Judgment in Predictive Analytics, 2023, pp 135-184 from Springer

Abstract: Abstract Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: (1) accuracy-related measures, such as proper scores, model-based estimates of accuracy and excess volatility scores; (2) intersubjective measures, including proxy, surrogate and similarity scores; (3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; (4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and (5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term, existential risk prediction and corporate forecasting tournaments.

Keywords: Forecasting; Prediction; Crowdsourcing; Skill assessment (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:isochp:978-3-031-30085-1_6

Ordering information: This item can be ordered from
http://www.springer.com/9783031300851

DOI: 10.1007/978-3-031-30085-1_6

Access Statistics for this chapter

More chapters in International Series in Operations Research & Management Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().