The evaluation of binary classification tasks in economical prediction

Evaluace binárních klasifikačních úloh v ekonomické predikci

Martin Pokorný
Additional contact information
Martin Pokorný: Ústav informatiky, Mendelova univerzita v Brně, Zemědělská 1, 613 00, Brno, Česká republika

Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis, 2010, vol. 58, issue 6, 369-378

Abstract: In the area of economical classification tasks, the accuracy maximization is often used to evaluate classifier performance. Accuracy maximization (or error rate minimization) suffers from the assumption of equal positive and negative error costs. Furthermore, accuracy is not able to express true classifier performance under skewed class distribution. Due to these limitations, the use of accuracy on real tasks is questionable. In a real binary classification task, the difference between the costs of positive and negative error is usually critical. To overcome this issue, the Receiver Operating Characteristic (ROC) method in relation to decision-analytic principles can be used. One essential advantage of this method is the possibility of classifier performance visualization by means of a ROC graph. This paper presents concrete examples of binary classification, where the inadequacy of accuracy as the evaluation metric is shown, and on the same examples the ROC method is applied. From the set of possible classification models, the probabilistic classifier with continuous output is under consideration. Mainly two questions are solved. Firstly, the selection of the best classifier from a set of possible classifiers. For example, accuracy metric rates two classifiers almost equivalently (87.7 % and 89.3 %), whereas decision analysis (via costs minimization) or ROC analysis reveal different performance according to target conditions of unequal error costs of positives and negatives. Secondly, the setting of an optimal decision threshold at classifier's output. For example, accuracy maximization finds the optimal threshold at classifier's output in value of 0.597, but the optimal threshold respecting higher costs of negatives is discovered by costs minimization or ROC analysis in a value substantially lower (0.477).

Keywords: binary classification; bankruptcy prediction; classifier performance evaluation; accuracy maximization; receiver operating characteristic (ROC) (search for similar items in EconPapers)
Date: 2010
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://acta.mendelu.cz/doi/10.11118/actaun201058060369.html (text/html)
http://acta.mendelu.cz/doi/10.11118/actaun201058060369.pdf (application/pdf)
free of charge

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:mup:actaun:actaun_2010058060369

DOI: 10.11118/actaun201058060369

Access Statistics for this article

Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis is currently edited by Markéta Havlásková

More articles in Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis from Mendel University Press
Bibliographic data for series maintained by Ivo Andrle ().