A neglected dimension of good forecasting judgment: The questions we choose also matter

Edgar C. Merkle, Mark Steyvers, Barbara Mellers and Philip E. Tetlock

International Journal of Forecasting, 2017, vol. 33, issue 4, 817-832

Abstract: Forecasters are typically evaluated via proper scoring rules such as the Brier score. These scoring rules use only the reported forecasts for assessment, neglecting related variables such as the specific questions that a person chose to forecast. In this paper, we study whether information related to question selection influences our estimates of forecaster ability. In other words, do good and bad forecasters tend to select questions in different ways? If so, can we capitalize on these selections when estimating forecaster ability? We address these questions by extending a recently-developed psychometric model of forecasts to include question selection data. We compare the extended psychometric model to a simpler model, studying its unidimensionality assumption and highlighting the unique information that it can provide. We find that the model can make use of the fact that good forecasters tend to select more questions than bad forecasters, and we conclude that question selection data can be beneficial above and beyond reported forecasts. As a side benefit, the resulting model can potentially provide unique incentives for forecaster participation.

Keywords: Forecast evaluation; Brier score; Question selection; Forecast nonresponse (search for similar items in EconPapers)
Date: 2017
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0169207017300377
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:intfor:v:33:y:2017:i:4:p:817-832

DOI: 10.1016/j.ijforecast.2017.04.002

Access Statistics for this article

International Journal of Forecasting is currently edited by R. J. Hyndman

More articles in International Journal of Forecasting from Elsevier
Bibliographic data for series maintained by Catherine Liu ().