Exploring the Preference for Discrete over Continuous Reinforcement Learning in Energy Storage Arbitrage

Jaeik Jeong (), Tai-Yeon Ku and Wan-Ki Park ()
Additional contact information
Jaeik Jeong: Energy ICT Research Section, Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea
Tai-Yeon Ku: Energy ICT Research Section, Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea
Wan-Ki Park: Energy ICT Research Section, Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea

Energies, 2024, vol. 17, issue 23, 1-17

Abstract: In recent research addressing energy arbitrage with energy storage systems (ESS s), discrete reinforcement learning (RL) has often been employed, while the underlying reasons for this preference have not been explicitly clarified. This paper aims to elucidate why discrete RL tends to be more suitable than continuous RL for energy arbitrage problems. When using continuous RL, the charging and discharging actions determined by the agent often exceed the physical limits of the ESS, necessitating clipping to the boundary values. This introduces a critical issue where the learned actions become stuck at the state of charge (SoC) boundaries, hindering effective learning. Although recent advancements in constrained RL offer potential solutions, their application often results in overly conservative policies, preventing the full utilization of ESS capabilities. In contrast, discrete RL, while lacking in granular control, successfully avoids these two key challenges, as demonstrated by simulation results showing superior performance. Additionally, it was found that, due to its characteristics, discrete RL more easily drives the ESS towards fully charged or fully discharged states, thereby increasing the utilization of the storage system. Our findings provide a solid justification for the prevalent use of discrete RL in recent studies involving energy arbitrage with ESSs, offering new insights into the strategic selection of RL methods in this domain. Looking ahead, improving performance will require further advancements in continuous RL methods. This study provides valuable direction for future research in continuous RL, highlighting the challenges and potential strategies to overcome them to fully exploit ESS capabilities.

Keywords: energy arbitrage; energy storage system; discrete reinforcement learning; continuous reinforcement learning; constrained reinforcement learning (search for similar items in EconPapers)
JEL-codes: Q Q0 Q4 Q40 Q41 Q42 Q43 Q47 Q48 Q49 (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1996-1073/17/23/5876/pdf (application/pdf)
https://www.mdpi.com/1996-1073/17/23/5876/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jeners:v:17:y:2024:i:23:p:5876-:d:1527487

Access Statistics for this article

Energies is currently edited by Ms. Agatha Cao

More articles in Energies from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().