A Cooperative Multi-Agent Q-Learning Control Framework for Real-Time Energy Management in Energy Communities

Andrea Tortorelli, Giulia Sabina and Barbara Marchetti ()
Additional contact information
Andrea Tortorelli: Dipartimento di Scienze Teoriche e Applicate DiSTA, Faculty of Engineering, eCampus University, Via Isimbardi 10, 22060 Novedrate, Italy
Giulia Sabina: Dipartimento di Scienze Teoriche e Applicate DiSTA, Faculty of Engineering, eCampus University, Via Isimbardi 10, 22060 Novedrate, Italy
Barbara Marchetti: Dipartimento di Scienze Teoriche e Applicate DiSTA, Faculty of Engineering, eCampus University, Via Isimbardi 10, 22060 Novedrate, Italy

Energies, 2024, vol. 17, issue 20, 1-27

Abstract: Residential and commercial buildings are responsible for 35% of the EU energy-related greenhouse gas (GHG) emissions. Reducing their emissions is crucial for meeting the challenging EU objective of the agenda for becoming a net-zero continent by 2050. The diffusion and integration of distributed renewable energy sources (RESs) and energy storage systems (ESSs), as well as the creation of energy communities (ECs), have proven to be crucial aspects in reducing GHG emissions. In this context, this article proposes a multi-agent AI-based control framework to solve the EC’s energy management problem in the presence of distributed RESs and ESSs as well as considering a shared ESS. The objectives of the proposed control framework are to satisfy the EC members’ load demand to maximize self-consumption and to manage ESSs charging and discharging processes, to enforce cooperative behavior among the EC members by adopting fair and personalized strategies and to maximize EC members’ profits. The proposed control procedure is based on three sequential stages, each solved by a dedicated local RL agent exploiting the Q-Learning algorithm. To reduce the computational complexity of the proposed approach, specifically defined state aggregation criteria were defined to map the RL agents’ continuous state spaces into discrete state spaces of limited dimensions. During the training phase, the EC members’ profiles and the ESSs’ and RESs’ characteristics were randomly changed to allow the RL agents to learn the correct policy to follow in any given scenario. Simulations proved the effectiveness of the proposed approach for different costumers’ load demand profiles and different EC configurations. Indeed, the trained RL agents proved to be able to satisfy the EC members’ load demands to maximize self-consumption, to correctly use the distributed and shared ESSs, to charge them according to respective personalized criteria and to sell the energy surplus, prioritizing sales to the EC. The proposed control framework also proved to be a useful tool for understanding EC performance in different configurations and, thus, for properly dimensioning the EC elements.

Keywords: energy community; energy management; multi-agent Q-learning; multi-objective optimization (search for similar items in EconPapers)
JEL-codes: Q Q0 Q4 Q40 Q41 Q42 Q43 Q47 Q48 Q49 (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1996-1073/17/20/5199/pdf (application/pdf)
https://www.mdpi.com/1996-1073/17/20/5199/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jeners:v:17:y:2024:i:20:p:5199-:d:1501816

Access Statistics for this article

Energies is currently edited by Ms. Agatha Cao

More articles in Energies from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().