MMGPT4LF: Leveraging an optimized pre-trained GPT-2 model with multi-modal cross-attention for load forecasting

Mingyang Gao, Suyang Zhou, Wei Gu, Zhi Wu, Haiquan Liu, Aihua Zhou and Xinliang Wang

Applied Energy, 2025, vol. 392, issue C, No S0306261925006956

Abstract: Accurate load forecasting is crucial for maintaining power system balance. Traditionally, forecasting relies on time series data such as historical loads and corresponding meteorological information. However, non-time-series data like news reports and holiday schedules can also significantly influence outcomes. Existing research primarily focuses on time series data and lacks effective handling of multi-modal inputs. Recent advances in Large Language Models (LLMs) demonstrate inherent advantages in capturing long-term dependencies and complex textual patterns, indicating their potential for load forecasting. Nevertheless, the application of LLMs in this field remains limited. Thus, to fill this gap, we propose MMGPT4LF, a model that combines the pre-trained GPT-2 model with multi-modal data inputs for load forecasting. Specifically, the model designs an additional time series input head to more effectively capture temporal dependencies, particularly the periodicity and long-term trends present in power load data. Furthermore, the model incorporates a Multi-Modal Cross-Attention (MMCA) mechanism, enabling efficient alignment and fusion of high-dimensional feature representations from both time series and textual inputs. Through this framework, MMGPT4LF not only enhances the effectiveness of multi-modal data fusion but also accurately handles the interactions between different modalities, thereby significantly improving load forecasting accuracy and the model’s generalization ability. Extensive experiments on two open-source load forecasting datasets, compared with nine advanced time series forecasting models, validate the effectiveness and accuracy of MMGPT4LF in load forecasting tasks.

Keywords: Load forecasting; Pre-trained large language model; Multi-modal; Deep learning (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0306261925006956
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:appene:v:392:y:2025:i:c:s0306261925006956

Ordering information: This journal article can be ordered from
http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/bibliographic
http://www.elsevier. ... 405891/bibliographic

DOI: 10.1016/j.apenergy.2025.125965

Access Statistics for this article

Applied Energy is currently edited by J. Yan

More articles in Applied Energy from Elsevier
Bibliographic data for series maintained by Catherine Liu ().