Maximum entropy heterogeneous-agent reinforcement learning
Files
(Published version)
Date
2024
Authors
Liu, J.
Zhong, Y.
Hu, S.
Fu, H.
Fu, Q.
Chang, X.
Yang, Y.
Editors
Advisors
Journal Title
Journal ISSN
Volume Title
Type:
Conference paper
Citation
12th International Conference on Learning Representations, ICLR 2024, 2024, pp.1-42
Statement of Responsibility
Conference Name
12th International Conference on Learning Representations, ICLR 2024 (7 May 2024 - 11 May 2024 : Hybrid, Vienna)
Abstract
Multi-agent reinforcement learning (MARL) has been shown effective for cooperative games in recent years. However, existing state-of-the-art methods face challenges related to sample complexity, training instability, and the risk of converging to a suboptimal Nash Equilibrium. In this paper, we propose a unified framework for learning stochastic policies to resolve these issues. We embed cooperative MARL problems into probabilistic graphical models, from which we derive the maximum entropy (MaxEnt) objective for MARL. Based on the MaxEnt framework, we propose Heterogeneous-Agent Soft Actor-Critic (HASAC) algorithm. Theoretically, we prove the monotonic improvement and convergence to quantal response equilibrium (QRE) properties of HASAC.
Furthermore, we generalize a unified template for MaxEnt algorithmic design named Maximum Entropy Heterogeneous-Agent Mirror Learning (MEHAML), which provides any induced method with the same guarantees as HASAC. We evaluate HASAC on six benchmarks: Bi-DexHands, Multi-Agent MuJoCo, StarCraft Multi-Agent Challenge, Google Research Football, Multi-Agent Particle Environment, and Light Aircraft Game. Results show that HASAC consistently outperforms strong baselines, exhibiting better sample efficiency, robustness, and sufficient exploration.
School/Discipline
Dissertation Note
Provenance
Description
Data source: Supplementary material, https://openreview.net/forum?id=tmqOhBC4a5
Access Status
Rights
© the author(s).