On policymakers' loss function and the evaluation of early warning systems

Peter Sarlin ()

No 1509, Working Paper Series from European Central Bank

Abstract: This paper introduces a new loss function and Usefulness measure for evaluating early warning systems (EWSs) that incorporate policymakers' preferences between issuing false alarms and missing crises, as well as individual observations. The novelty derives from three enhancements: i) accounting for unconditional probabilities of the classes, ii) computing the proportion of available Usefulness that the model captures, and iii) weighting observations by their importance for the policymaker. The proposed measures are model free such that they can be used to assess signals issued by any type of EWS, such as logit and probit analysis and the signaling approach, and flexible for any type of crisis EWSs, such as banking, debt and currency crises. Applications to two renowned EWSs, and comparisons to two commonly used evaluation measures, illustrate three key implications of the new measures: i) further highlights the importance of an objective criterion for choosing a final specification and threshold value, and for models to be useful ii) the need to be more concerned about the rare class and iii) the importance of correctly classifying observations of the most relevant entities. Beyond financial stability surveillance, this paper also opens the door for cost-sensitive evaluations of predictive models in other tasks. JEL Classification: E44, E58, F01, F37, G01

Keywords: early warning systems; misclassification costs (search for similar items in EconPapers)
Date: 2013-02
New Economics Papers: this item is included in nep-cba
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (70)

Downloads: (external link)
https://www.ecb.europa.eu//pub/pdf/scpwps/ecbwp1509.pdf (application/pdf)

Related works:
Journal Article: On policymakers’ loss functions and the evaluation of early warning systems (2013)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ecb:ecbwps:20131509

Access Statistics for this paper

More papers in Working Paper Series from European Central Bank 60640 Frankfurt am Main, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Official Publications ().