From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

Agathe Fernandes Machado (), Arthur Charpentier, Emmanuel Flachaire () and Ewen Gallic
Additional contact information
Agathe Fernandes Machado: Département de Mathématiques, Université du Québec à Montréal, Montréal, Québec, Canada
Arthur Charpentier: Département de Mathématiques, Université du Québec à Montréal, Montréal, Québec, Canada
Emmanuel Flachaire: Aix Marseille Univ, CNRS, AMSE, Marseille, France, https://www.amse-aixmarseille.fr/en/members/flachaire

No 2404, AMSE Working Papers from Aix-Marseille School of Economics, France

Abstract: The assessment of binary classifier performance traditionally centers on discriminative ability using metrics, such as accuracy. However, these metrics often disregard the model’s inherent uncertainty, especially when dealing with sensitive decision-making domains, such as finance or healthcare. Given that model-predicted scores are commonly seen as event probabilities, calibration is crucial for accurate interpretation. In our study, we analyze the sensitivity of various calibration measures to score distortions and introduce a refined metric, the Local Calibration Score. Comparing recalibration methods, we advocate for local regressions, emphasizing their dual role as effective recalibration tools and facilitators of smoother visualizations. We apply these findings in a real-world scenario using Random Forest classifier and regressor to predict credit default while simultaneously measuring calibration during performance optimization.

Keywords: Calibration; Binary classification; Local regression (search for similar items in EconPapers)
Pages: 32 pages
Date: 2024-02
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.amse-aixmarseille.fr/sites/default/fil ... /wp_2024_-_nr_04.pdf (application/pdf)

Related works:
Working Paper: From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:aim:wpaimx:2404

Access Statistics for this paper

More papers in AMSE Working Papers from Aix-Marseille School of Economics, France AMU-AMSE - 5-9 Boulevard Maurice Bourdet, CS 50498 - 13205 Marseille Cedex 1. Contact information at EDIRC.
Bibliographic data for series maintained by Gregory Cornu ().