Forecasting Out-of-Time Credit Scoring Model Risk

Valter Jr., Rafael Schiozer, Alan Genaro and Toni Santos

No 645, Working Papers Series from Central Bank of Brazil, Research Department

Abstract: This paper addresses the challenge of forecasting the best-performing credit scoring model in outof-time settings, focusing on the decision between segmented (bank-specific) and full data (financial system-wide) models. Building upon the Credit Scoring Model Risk (CSMR) metric, defined as one minus the correlation between observed defaults and predicted scores, we highlight the instability of in-sample performance measures when applied to evolving loan portfolios and changing macroeconomic conditions. We propose three complementary approaches to predict out-of-time model performance: (i) an analytical method based on Copas shrinkage concept utilizing estimated covariances and prediction variances; (ii) a Monte Carlo simulation leveraging average model predictions to simulate default events; and (iii) a Bayesian estimation framework for covariances grounded in conditional expectations of predictions given default. Empirical analysis using a large Brazilian loan dataset reveals that segmented models outperform full data models in in-sample contexts but not consistently out-of-time. Among the approaches, the Monte Carlo simulation achieved the highest accuracy (70.8%) in forecasting the superior out-of-time model, followed by the Bayesian method (66.7%) and the analytical shrinkage approach (54.2%). The study underscores the importance of considering population shifts via the Population Stability Index (PSI) to detect model decalibration and overfitting. The proposed methodologies offer practitioners and regulators practical tools for informed model selection, enhancing predictive reliability over time amid portfolio and economic dynamics.

Date: 2026-04
New Economics Papers: this item is included in nep-rmg
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.bcb.gov.br/content/publicacoes/WorkingPaperSeries/WP645v2.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bcb:wpaper:645

Access Statistics for this paper

More papers in Working Papers Series from Central Bank of Brazil, Research Department
Bibliographic data for series maintained by Rodrigo Barbone Gonzalez ().