Honest calibration assessment for binary outcome predictions

Timo Dimitriadis, Lutz Dümbgen, Alexander Henzi, Marius Puke and Johanna Ziegel

Biometrika, 2023, vol. 110, issue 3, 663-680

Abstract: SummaryProbability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability , it should materialize with approximately that frequency, which means that the so-called calibration curveshould equal the identity, i.e.,for allin the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid subject to only the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well-specified model. We show that our bands have a finite-sample coverage guarantee, are narrower than those of existing approaches, and adapt to the local smoothness of the calibration curveand the local variance of the binary observations. In an application to modelling predictions of an infant having low birth weight, the bounds give informative insights into model calibration.

Keywords: Binary regression; Calibration validation; Confidence band; Goodness-of-fit; Isotonic regression; Universally valid inference (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1093/biomet/asac068 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:biomet:v:110:y:2023:i:3:p:663-680.

Ordering information: This journal article can be ordered from
https://academic.oup.com/journals

Access Statistics for this article

Biometrika is currently edited by Paul Fearnhead

More articles in Biometrika from Biometrika Trust Oxford University Press, Great Clarendon Street, Oxford OX2 6DP, UK.
Bibliographic data for series maintained by Oxford University Press ().