Reinforcement Learning with Value Function Decomposition for Hierarchical Multi-Agent Consensus Control

Xiaoxia Zhu ()
Additional contact information
Xiaoxia Zhu: School of Intelligent Manufacturing, Shanghai Zhongqiao Vocational and Technical University, Shanghai 201514, China

Mathematics, 2024, vol. 12, issue 19, 1-18

Abstract: A hierarchical consensus control algorithm based on value function decomposition is proposed for hierarchical multi-agent systems. To implement the consensus control algorithm, the reward function of the multi-agent systems can be decomposed, and two value functions can be obtained by analyzing the communication content and the corresponding control objective of each layer in the hierarchical multi-agent systems. Therefore, for each agent in the systems, a dual-critic network and a single-actor network structure are applied to realize the objective of each layer. In addition, the target network is introduced to prevent overfitting in the critic network and improve the stability of the online learning process. During the updating of network parameters, a soft updating mechanism and experience replay buffer are introduced to slow down the update rate of the network and improve the utilization rate of training data. The convergence and stability of the consensus control algorithm with the soft updating mechanism are analyzed theoretically. Finally, the correctness of the theoretical analysis and the effectiveness of the algorithm were verified by two experiments.

Keywords: reinforcement learning; value function decomposition; multi-agent; consensus (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/12/19/3062/pdf (application/pdf)
https://www.mdpi.com/2227-7390/12/19/3062/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:12:y:2024:i:19:p:3062-:d:1489240

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().