The finer points of model comparison in machine learning: forecasting based on russian banks’ data

Denis Shibitov () and Mariam Mamedli
Additional contact information
Denis Shibitov: Bank of Russia, Russian Federation

No wps43, Bank of Russia Working Paper Series from Bank of Russia

Abstract: We evaluate the forecasting ability of machine learning models to predict bank license withdrawal and the violation of statutory capital and liquidity requirements (capital adequacy ratio N1.0, common equity Tier 1 adequacy ratio N1.1, Tier 1 capital adequacy ratio N1.2, N2 instant and N3 current liquidity). On the basis of 35 series from the accounting reports of Russian banks, we form two data sets of 69 and 721 variables and use them to build random forest and gradient boosting models along with neural networks and a stacking model for different forecasting horizons (1, 2, 3, 6, 9 months). Based on the data from February 2014 to October 2018 we show that these models with fine-tuned architectures can successfully compete with logistic regression usually applied for this task. Stacking and random forest generally have the best forecasting performance comparing to the other models. We evaluate models with commonly used performance metrics (ROC-AUC and F1) and show that, depending on the task, F1-score could be better at defining the model’s performance. Comparison of the results depending on the metrics applied and types of cross-validation used illustrate the importance of choosing the appropriate metric for performance evaluation and the cross-validation procedure, which accounts for the characteristics of the data set and the task under consideration. The developed approach shows the advantages of non-linear methods for bank regulation tasks and provides the guidelines for the application of machine learning algorithms to these tasks.

Keywords: machine learning; random forest; neural networks; gradient boosting; forecasting; bank supervision (search for similar items in EconPapers)
JEL-codes: C45 C5 C52 C53 (search for similar items in EconPapers)
Pages: 50 pages
Date: 2019-08
New Economics Papers: this item is included in nep-ban, nep-big, nep-cis, nep-cmp, nep-for and nep-pay
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://cbr.ru/Content/Document/File/87572/wp43_e.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bkr:wpaper:wps43

Access Statistics for this paper

More papers in Bank of Russia Working Paper Series from Bank of Russia Contact information at EDIRC.
Bibliographic data for series maintained by BoR Research ( this e-mail address is bad, please contact ).