From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

Agathe Fernandes Machado (), Arthur Charpentier, Emmanuel Flachaire, Ewen Gallic and François Hu
Additional contact information
Agathe Fernandes Machado: UQAM - Université du Québec à Montréal = University of Québec in Montréal
Arthur Charpentier: UQAM - Université du Québec à Montréal = University of Québec in Montréal
Emmanuel Flachaire: AMSE - Aix-Marseille Sciences Economiques - EHESS - École des hautes études en sciences sociales - AMU - Aix Marseille Université - ECM - École Centrale de Marseille - CNRS - Centre National de la Recherche Scientifique
François Hu: UdeM - Université de Montréal

Working Papers from HAL

Abstract: The assessment of binary classifier performance traditionally centers on discriminative ability using metrics, such as accuracy. However, these metrics often disregard the model's inherent uncertainty, especially when dealing with sensitive decision-making domains, such as finance or healthcare. Given that model-predicted scores are commonly seen as event probabilities, calibration is crucial for accurate interpretation. In our study, we analyze the sensitivity of various calibration measures to score distortions and introduce a refined metric, the Local Calibration Score. Comparing recalibration methods, we advocate for local regressions, emphasizing their dual role as effective recalibration tools and facilitators of smoother visualizations. We apply these findings in a real-world scenario using Random Forest classifier and regressor to predict credit default while simultaneously measuring calibration during performance optimization.

Keywords: Calibration; Binary classification; Local regression (search for similar items in EconPapers)
Date: 2024-02-12
Note: View the original document on HAL open archive server: https://hal.science/hal-04464994
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hal.science/hal-04464994/document (application/pdf)

Related works:
Working Paper: From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:wpaper:hal-04464994

Access Statistics for this paper

More papers in Working Papers from HAL
Bibliographic data for series maintained by CCSD ().