Smartformer: An intelligent transformer compression framework for time-series modeling

Xiaojian Wang, Yinan Wang, Jin Yang and Ying Chen

IISE Transactions, 2025, vol. 57, issue 9, 1027-1040

Abstract: Transformer, as one of the cutting-edge deep neural networks (DNNs), has achieved outstanding performance in time-series data analysis. However, this model usually requires large numbers of parameters to fit. Over-parameterization not only brings storage challenges in a resource-limited setting, but also inevitably results in the model over-fitting. Even though literature works introduced several ways to reduce the parameter size of Transformers, none of them addressed this over-parameterized issue by concurrently achieving the following three objectives: preserving the model architecture, maintaining the model performance, and reducing the model complexity (number of parameters). In this study, we propose an intelligent model compression framework, Smartformer, by incorporating reinforcement learning and CP-decomposition techniques to satisfy the aforementioned three objectives. In the experiment, we apply Smartformer and five baseline methods to two existing time-series Transformer models for model compression. The results demonstrate that our proposed Smartformer is the only method that consistently generates the compressed model on various scenarios by satisfying the three objectives. In particular, the Smartformer can mitigate the overfitting issue and thus improve the accuracy of the existing time-series models in all scenarios.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/24725854.2024.2376645 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:uiiexx:v:57:y:2025:i:9:p:1027-1040

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/uiie20

DOI: 10.1080/24725854.2024.2376645

Access Statistics for this article

IISE Transactions is currently edited by Jianjun Shi

More articles in IISE Transactions from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().