CQEformer: A Causal and Query-Enhanced Transformer Variant for Time-Series Forecasting

Yuze Tao and Lu Li ()
Additional contact information
Yuze Tao: School of Mathematics, Physics and Statistics, Shanghai University of Engineering Science, Shanghai 201620, China
Lu Li: School of Mathematics, Physics and Statistics, Shanghai University of Engineering Science, Shanghai 201620, China

Mathematics, 2025, vol. 13, issue 23, 1-18

Abstract: Structural breaks and volatility clustering are fundamental challenges in time series analysis. We propose CQEformer, an encoder-only Transformer variant for time-series modeling that addresses these challenges via two complementary innovations. First, the causal residual embedding (CRE) ensures temporal causality and improves local adaptivity to abrupt structural changes. Second, the query-enhanced multi-head self-attention (QEAttention) incorporates multi-order moment statistics and entropy to guide attention dynamically toward high-volatility regions while preserving global dependence structures. For parameter optimization, we derive analytical gradients for all components and update them using the Adam stochastic optimization algorithm. Empirical evaluations on financial time series datasets and the public Traffic dataset show that CQEformer consistently outperforms established baselines, including LSTM, GRU, TCN, and the standard Transformer. Time-window sensitivity analyses demonstrate the robustness of the framework, while ablation studies further confirm that the proposed modules are complementary and contribute to improved forecasting performance across different volatility regimes.

Keywords: CQEformer; causal residual embedding; Query-Enhanced Attention; structural breaks; volatility clustering; time-series forecasting (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/23/3750/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/23/3750/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:23:p:3750-:d:1800637

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().