The effect of imperfect data on default prediction validation tests

Heather Russell and Douglas Dwyer and Qing Kang Tang

Journal of Risk Model Validation

Abstract: ABSTRACT Analysts often find themselves working with less than perfect development and/or validation samples, and data issues typically affect the interpretation of default prediction validation tests. Discriminatory power and calibration of default probabilities are two key aspects of validating default probability models. This paper considers how data issues affect three important power tests: the accuracy ratio, the Kolmogorov-Smirnov test and the conditional information entropy ratio. The effect of data issues upon the Hosmer-Lemeshow test, a default probability calibration test, is also considered. A simulation approach is employed that allows the impact of data issues on model performance, when the exact nature of the data issue is known, to be assessed. We obtain several results from the tests of discriminatory power. For example, we find that random missing defaults have little impact on model power, while false defaults have a large impact on power. As with other common level calibration test statistics, the Hosmer-Lemeshow test statistic simply indicates to what degree the level calibration passes or fails. We find that the presence of any data issue tends to cause this test to fail, and, thus, we introduce additional statistics to describe how realized default probabilities differ from those expected. In particular, we introduce statistics to compare overall default probability level with the realized default rate, and to compare the sensitivity of the default rate to changes in the predicted default probability.

References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.risk.net/journal-of-risk-model-validat ... ion-validation-tests (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:rsk:journ5:2164412

Access Statistics for this article

More articles in Journal of Risk Model Validation from Journal of Risk Model Validation
Bibliographic data for series maintained by Thomas Paine ().