A General Framework for Learning Mean-Field Games

Xin Guo (), Anran Hu (), Renyuan Xu () and Junzi Zhang ()
Additional contact information
Xin Guo: Industrial Engineering and Operations Research Department, University of California–Berkeley, Berkeley, California 94720; Amazon, Seattle, Washington 98109
Anran Hu: Industrial Engineering and Operations Research Department, University of California–Berkeley, Berkeley, California 94720
Renyuan Xu: Daniel J. Epstein Department of Industrial Systems & Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, California 90089; Mathematical Institute, University of Oxford, Oxford OX2 6GG, United Kingdom
Junzi Zhang: Amazon, Seattle, Washington 98109; Institute for Computational & Mathematical Engineering, Stanford University, California 94305

Mathematics of Operations Research, 2023, vol. 48, issue 2, 656-686

Abstract: This paper presents a general mean-field game (GMFG) framework for simultaneous learning and decision making in stochastic games with a large population. It first establishes the existence of a unique Nash equilibrium to this GMFG, and it demonstrates that naively combining reinforcement learning with the fixed-point approach in classical mean-field games yields unstable algorithms. It then proposes value-based and policy-based reinforcement learning algorithms (GMF-V and GMF-P, respectively) with smoothed policies, with analysis of their convergence properties and computational complexities. Experiments on an equilibrium product pricing problem demonstrate that two specific instantiations of GMF-V with Q-learning and GMF-P with trust region policy optimization—GMF-V-Q and GMF-P-TRPO, respectively—are both efficient and robust in the GMFG setting. Moreover, their performance is superior in convergence speed, accuracy, and stability when compared with existing algorithms for multiagent reinforcement learning in the N -player setting.

Keywords: Primary: 49N80; 91A07; 93E35; 68T05; 91A26; mean-field game; reinforcement learning; multiagent reinforcement learning; stochastic games (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/moor.2022.1274 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormoor:v:48:y:2023:i:2:p:656-686

Access Statistics for this article

More articles in Mathematics of Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().