A Gradient-based reinforcement learning model of market equilibration

He, Zhongzhi (Lawrence)

Journal of Economic Dynamics and Control, 2023, vol. 152, issue C

Abstract: This paper formulates a game-theoretic reinforcement learning model based on the stochastic gradient method whereby players start from their initial circumstances with dispersed information, using the expected gradient to update choice propensities, and converge to the predicted equilibrium of belief-based models. Gradient-based reinforcement learning (G-RL) entails a model-free simulation method to estimate the gradient of expected payoff with respect to choice propensities in repeated games. As the gradient points to the steepest direction towards discovering steady-state equilibrium, G-RL provides a theoretical justification for a probability-weighed time-varying updating rule that optimally balances the trade-off between reinforcing past successful strategies (‘exploitation’) and exploring other strategies (‘exploration’) in choosing actions. The effectiveness and stability of G-RL are demonstrated in a simulated call market, where both the actual effect and the foregone effect are simultaneously updated during market equilibration. In contrast, the failure of payoff-based reinforcement learning (P-RL) is due to its constant-sensitivity updating rule, which causes an imbalance between exploitation and exploration in complex environments.

Keywords: Reinforcement learning; Machine learning; Stochastic gradient method; Model free simulation; Call market; Market equilibration; Exploitation and exploration (search for similar items in EconPapers)
JEL-codes: C73 D81 (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0165188923000763
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:dyncon:v:152:y:2023:i:c:s0165188923000763

DOI: 10.1016/j.jedc.2023.104670

Access Statistics for this article

Journal of Economic Dynamics and Control is currently edited by J. Bullard, C. Chiarella, H. Dawid, C. H. Hommes, P. Klein and C. Otrok

More articles in Journal of Economic Dynamics and Control from Elsevier
Bibliographic data for series maintained by Catherine Liu ().