Coupling time-scale reinforcement learning methods for building operational optimization with waste heat

Zhe Chen, Tian Xing, Yu Wang, Yunlin Zhuang, Meng Zheng, Qianchuan Zhao and Qing-Shan Jia

Applied Energy, 2025, vol. 391, issue C, No S0306261925005811

Abstract: This paper focuses on the joint optimization of fan coil units (FCUs) and heat pumps in HVAC systems for multi-zone building environments. The core problem involves balancing fast local control of FCUs with slower global control of heat pumps to ensure energy efficiency and indoor comfort. To tackle this, we propose a coupling time-scale reinforcement learning (RL) algorithm, specifically a Deep Q-Network (DQN)-based approach that employs a multi-task learning network to manage both FCU and heat pump control efficiently through the utilization of shared state information. Moreover, the agents are trained and make decisions collaboratively across different timescales. In addition, we develop a high-fidelity building simulation that incorporates detailed thermal models, dynamic loading, and waste heat modules to evaluate the performance of the system in real-world conditions. The experimental results show that the proposed coupling time-scale DQN algorithm improves the accuracy of temperature control by 35.4 % and 26.21 % compared to traditional DQN and the occupant-centric control (OCC) algorithms. Additionally, it reduces regional power fluctuations by 25.18 % and 56.74 % relative to these traditional algorithms. Simultaneously, the proposed algorithm achieves the lowest heat pump energy consumption (2964 W), outperforming traditional DQN (2977 W) and OCC (3051 W) respectively, while maintaining superior temperature control accuracy. These quantitative improvements collectively demonstrate the proposed algorithm’s capability to synergistically balance thermal comfort, power fluctuation, and energy consumption.

Keywords: Reinforcement learning; Time-scale; Occupant-centric control; HVAC optimization; Waste heat utilization (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0306261925005811
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:appene:v:391:y:2025:i:c:s0306261925005811

Ordering information: This journal article can be ordered from
http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/bibliographic
http://www.elsevier. ... 405891/bibliographic

DOI: 10.1016/j.apenergy.2025.125851

Access Statistics for this article

Applied Energy is currently edited by J. Yan

More articles in Applied Energy from Elsevier
Bibliographic data for series maintained by Catherine Liu ().