Recalibration Methods for Improved Clinical Utility of Risk Scores

Anu Mishra, Robyn L. McClelland, Lurdes Y. T. Inoue and Kathleen F. Kerr
Additional contact information
Anu Mishra: School of Public Health, Imperial College London, London, UK
Robyn L. McClelland: Department of Biostatistics, University of Washington, Seattle, WA, USA
Lurdes Y. T. Inoue: Department of Biostatistics, University of Washington, Seattle, WA, USA
Kathleen F. Kerr: Department of Biostatistics, University of Washington, Seattle, WA, USA

Medical Decision Making, 2022, vol. 42, issue 4, 500-512

Abstract: Background An established risk model may demonstrate miscalibration, meaning predicted risks do not accurately capture event rates. In some instances, investigators can identify and address the cause of miscalibration. In other circumstances, it may be appropriate to recalibrate the risk model. Existing recalibration methods do not account for settings in which the risk score will be used for risk-based clinical decision making. Methods We propose 2 new methods for risk model recalibration when the intended purpose of the risk model is to prescribe an intervention to high-risk individuals. Our measure of risk model clinical utility is standardized net benefit. The first method is a weighted strategy that prioritizes good calibration at or around the critical risk threshold. The second method uses constrained optimization to produce a recalibrated risk model with maximum possible net benefit, thereby prioritizing good calibration around the critical risk threshold. We also propose a graphical tool for assessing the potential for recalibration to improve the net benefit of a risk model. We illustrate these methods by recalibrating the American College of Cardiology (ACC)–American Heart Association (AHA) atherosclerotic cardiovascular disease (ASCVD) risk score within the Multi-Ethnic Study of Atherosclerosis (MESA) cohort. Results New methods are implemented in the R package ClinicalUtilityRecal . Recalibrating the ACC-AHA-ASCVD risk score for a MESA subcohort results in higher estimated net benefit using the proposed methods compared with existing methods, with improved calibration in the most clinically impactful regions of risk. Conclusion The proposed methods target good calibration for critical risks and can improve the net benefit of a risk model. We recommend constrained optimization when the risk model net benefit is paramount. The weighted approach can be considered when good calibration over an interval of risks is important.

Keywords: calibration; recalibration; clinical utility; net benefit; risk prediction (search for similar items in EconPapers)
Date: 2022
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0272989X211044697 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:medema:v:42:y:2022:i:4:p:500-512

DOI: 10.1177/0272989X211044697

Access Statistics for this article

More articles in Medical Decision Making
Bibliographic data for series maintained by SAGE Publications ().