fev-bench: A Realistic Benchmark for Time Series Forecasting

Oleksandr Shchur, Abdul Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider and Yuyang Wang
Additional contact information
Oleksandr Shchur: AWS
Abdul Ansari: AWS
Caner Turkmen: AWS
Lorenzo Stella: AWS
Nick Erickson: AWS
Michael Bohlke-Schneider: AWS
Yuyang Wang: AWS

No 1101, Boston College Working Papers in Economics from Boston College Department of Economics

Abstract: Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly given the recent rise of pre-trained models. Existing benchmarks often have narrow domain coverage or overlook important real-world settings, such as tasks with covariates. Additionally, their aggregation procedures often lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks also fail to provide infrastructure for consistent evaluation or are too rigid to integrate into existing pipelines. To address these gaps, we propose fev-bench, a benchmark comprising 100 forecasting tasks across seven domains, including 46 tasks with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for benchmarking forecasting models that emphasizes reproducibility and seamless integration with existing workflows. Using fev, fev-bench employs principled aggregation methods with bootstrapped confidence intervals to report model performance along two complementary dimensions: win rates and skill scores. We report results on fev-bench for various pre-trained, statistical and baseline models, and identify promising directions for future research.

Keywords: forecasting; benchmarks; time series (search for similar items in EconPapers)
JEL-codes: C78 E32 F34 (search for similar items in EconPapers)
Date: 2025-11-01
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://fmwww.bc.edu/EC-P/wp1101.pdf main text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:boc:bocoec:1101

Access Statistics for this paper

More papers in Boston College Working Papers in Economics from Boston College Department of Economics Boston College, 140 Commonwealth Avenue, Chestnut Hill MA 02467 USA. Contact information at EDIRC.
Bibliographic data for series maintained by Christopher F Baum ().