Spatial-Temporal Traffic Flow Control on Motorways Using Distributed Multi-Agent Reinforcement Learning

Krešimir Kušić, Edouard Ivanjko, Filip Vrbanić, Martin Gregurić and Ivana Dusparic
Additional contact information
Krešimir Kušić: Faculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, Croatia
Edouard Ivanjko: Faculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, Croatia
Filip Vrbanić: Faculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, Croatia
Martin Gregurić: Faculty of Transport and Traffic Sciences, University of Zagreb, Vukelićeva Street 4, HR-10 000 Zagreb, Croatia
Ivana Dusparic: School of Computer Science and Statistics, Trinity College Dublin, Dublin 2, Ireland

Mathematics, 2021, vol. 9, issue 23, 1-28

Abstract: The prevailing variable speed limit (VSL) systems as an effective strategy for traffic control on motorways have the disadvantage that they only work with static VSL zones. Under changing traffic conditions, VSL systems with static VSL zones may perform suboptimally. Therefore, the adaptive design of VSL zones is required in traffic scenarios where congestion characteristics vary widely over space and time. To address this problem, we propose a novel distributed spatial-temporal multi-agent VSL (DWL-ST-VSL) approach capable of dynamically adjusting the length and position of VSL zones to complement the adjustment of speed limits in current VSL control systems. To model DWL-ST-VSL, distributed W-learning (DWL), a reinforcement learning (RL)-based algorithm for collaborative agent-based self-optimization toward multiple policies, is used. Each agent uses RL to learn local policies, thereby maximizing travel speed and eliminating congestion. In addition to local policies, through the concept of remote policies, agents learn how their actions affect their immediate neighbours and which policy or action is preferred in a given situation. To assess the impact of deploying additional agents in the control loop and the different cooperation levels on the control process, DWL-ST-VSL is evaluated in a four-agent configuration (DWL4-ST-VSL). This evaluation is done via SUMO microscopic simulations using collaborative agents controlling four segments upstream of the congestion in traffic scenarios with medium and high traffic loads. DWL also allows for heterogeneity in agents’ policies; cooperating agents in DWL4-ST-VSL implement two speed limit sets with different granularity. DWL4-ST-VSL outperforms all baselines (W-learning-based VSL and simple proportional speed control), which use static VSL zones. Finally, our experiments yield insights into the new concept of VSL control. This may trigger further research on using advanced learning-based technology to design a new generation of adaptive traffic control systems to meet the requirements of operating in a nonstationary environment and at the leading edge of emerging connected and autonomous vehicles in general.

Keywords: intelligent transport systems; traffic control; spatial-temporal variable speed limit; multi-agent systems; reinforcement learning; distributed W-learning; urban motorways (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://www.mdpi.com/2227-7390/9/23/3081/pdf (application/pdf)
https://www.mdpi.com/2227-7390/9/23/3081/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:9:y:2021:i:23:p:3081-:d:691444

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().