On Comparing Cross-Validated Forecasting Models with a Novel Fuzzy-TOPSIS Metric: A COVID-19 Case Study

Dalton Garcia Borges de Souza, Erivelton Antonio dos Santos, Francisco Tarcísio Alves Júnior and Mariá Cristina Vasconcelos Nascimento
Additional contact information
Dalton Garcia Borges de Souza: Institute of Science and Technology, Fluminense Federal University, Rio das Ostras 28890-000, Brazil
Erivelton Antonio dos Santos: Institute of Industrial Engineering and Management, Federal University of Itajubá, Itajubá 37500-903, Brazil
Francisco Tarcísio Alves Júnior: Collegiate of Industrial Engineering, University of Amapa State, Macapá 68900-070, Brazil
Mariá Cristina Vasconcelos Nascimento: Division of Computer Science, Aeronautics Institute of Technology, São José dos Campos 12228-900, Brazil

Sustainability, 2021, vol. 13, issue 24, 1-25

Abstract: Time series cross-validation is a technique to select forecasting models. Despite the sophistication of cross-validation over single test/training splits, traditional and independent metrics, such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), are commonly used to assess the model’s accuracy. However, what if decision-makers have different models fitting expectations to each moment of a time series? What if the precision of the forecasted values is also important? This is the case of predicting COVID-19 in Amapá, a Brazilian state in the Amazon rainforest. Due to the lack of hospital capacities, a model that promptly and precisely responds to notable ups and downs in the number of cases may be more desired than average models that only have good performances in more frequent and calm circumstances. In line with this, this paper proposes a hybridization of the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and fuzzy sets to create a similarity metric, the closeness coefficient (CC), that enables relative comparisons of forecasting models under heterogeneous fitting expectations and also considers volatility in the predictions. We present a case study using three parametric and three machine learning models commonly used to forecast COVID-19 numbers. The results indicate that the introduced fuzzy similarity metric is a more informative performance assessment metric, especially when using time series cross-validation.

Keywords: fuzzy sets; TOPSIS; multicriteria decision making; decision support systems; forecasting (search for similar items in EconPapers)
JEL-codes: O13 Q Q0 Q2 Q3 Q5 Q56 (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2071-1050/13/24/13599/pdf (application/pdf)
https://www.mdpi.com/2071-1050/13/24/13599/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jsusta:v:13:y:2021:i:24:p:13599-:d:698410

Access Statistics for this article

Sustainability is currently edited by Ms. Alexandra Wu

More articles in Sustainability from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().