A Comparative Study of Traditional, Ensemble and Neural Network-Based Natural Language Processing Algorithms

Achraf Chikhi, Seyed Sahand Mohammadi Ziabari () and Jan-Willem van Essen
Additional contact information
Achraf Chikhi: Faculty of Science, Mathematics and Computer Science, University of Amsterdam, 1098 XH Amsterdam, The Netherlands
Seyed Sahand Mohammadi Ziabari: Faculty of Science, Mathematics and Computer Science, University of Amsterdam, 1098 XH Amsterdam, The Netherlands
Jan-Willem van Essen: Department of IT Advisory, Baker Tilly, 1114 AA Amsterdam, The Netherlands

JRFM, 2023, vol. 16, issue 7, 1-20

Abstract: Accurate data analysis is an important part of data-driven financial audits. Given the increased data availability and various systems from which audit files are generated, RCSFI provides a way for standardization on behalf of analysis. This research attempted to automate this hierarchical text classification task in order to save financial auditors time and avoid errors. Several studies have shown that ensemble-based models and neural-network-based natural language processing (NLP) techniques achieved encouraging results for classification problems in various domains. However, there has been limited empirical research comparing the performance of both of the aforementioned techniques in a hierarchical multi-class classification setting. Moreover, neural-network- based NLP techniques have commonly been applied to English datasets and not to Dutch financial datasets. Additionally, this research took the implementation of hierarchical approaches into account for the traditional and ensemble-based models and found that the performance did not increase when implementing the included hierarchical approaches. DistilBERT achieved the highest scores on level 1-2-3-4 and outperformed the traditional and ensemble-based models. The model obtained a F1 of 94.50% for level 1-2-3-4. DistilBERT also outperformed BERTje at level 1-2-3-4 despite BERTje being specifically pre-trained on Dutch datasets.

Keywords: audit; BERT; BERTje; DistilBERT; classification; financial; hierarchical; LightGBM; RCSFI; XGBoost (search for similar items in EconPapers)
JEL-codes: C E F2 F3 G (search for similar items in EconPapers)
Date: 2023
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1911-8074/16/7/327/pdf (application/pdf)
https://www.mdpi.com/1911-8074/16/7/327/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jjrfmx:v:16:y:2023:i:7:p:327-:d:1191353

Access Statistics for this article

JRFM is currently edited by Ms. Chelthy Cheng

More articles in JRFM from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().