Elicitability of Instance and Object Ranking

Tino Werner ()
Additional contact information
Tino Werner: Institute for Mathematics, Carl von Ossietzky University Oldenburg, 26111 Oldenburg, Germany

Decision Analysis, 2022, vol. 19, issue 2, 123-140

Abstract: Assessing the quality of a forecasting model crucially depends on a proper scoring rule or suitable loss function. As for point forecasts, the existence of a strictly consistent loss function that allows for a fair comparison of competing forecast models has to be guaranteed, which means that the corresponding statistical functional has to be elicitable. We consider instance and object ranking problems that intend to correctly predict the ordering of instances in a data set. A ranking prediction is naturally identified with a point forecast in the respective symmetric group, that is, the forecaster predicts one single permutation of the row indices. We show that, in the presence of ties, this strategy does not allow for strictly consistent scoring functions because of multiple true permutations. Those multiple optima cannot be entirely covered by a single point forecast, which causes all corresponding optima to be minimizers of standard scoring functions that operate on symmetric groups, so these scoring functions are not strictly consistent. As a remedy, we consider accurately accounting for ties. This is done by treating each configuration of clear orderings and ties as an additional category, which induces extended decision spaces with a clearly defined single optimum. Because these decision spaces are still finite, each type of instance ranking problem that we consider in this work and corresponding ranking functional, mapping into a symmetric group, can be identified with a certain classification problem and corresponding classification functional, mapping into one of our extended decision spaces, which is elicitable.

Keywords: elicitability; ranking functionals; classification functionals; scoring functions; information retrieval (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://dx.doi.org/10.1287/deca.2021.0446 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ordeca:v:19:y:2022:i:2:p:123-140

Access Statistics for this article

More articles in Decision Analysis from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().