Policy Gradient Methods for Multi-Agent Reinforcement Learning: A Comparative Study

Abstract

Multi-Agent Reinforcement Learning (MARL) has proven to be a compelling tool for addressing decision-making tasks with multiple agents who are engaged in complex interactions and collaborations. This study evaluates a variety of policy gradient techniques that are currently used in MARL, specifying three up-to-date methods: Counterfactual Multi-Agent Policy Gradient (COMA), Meta-Learning Policy Gradient (Meta-PG), and Status Quo Policy Gradient (SQPG). The performance of each method is determined by its convergence speed across training, success in varying environments, and the stability of the variance. The experimental results indicate that Meta-PG is the fastest algorithm, achieving the highest performance metrics in both shared and teamwork-based tasks, being optimal in such cases. COMA, however, exhibits great stability and effectiveness in adversarial settings; and employs the novel idea of counterfactual credit assignment for better learning. Though SQPG offers an overall balanced performance in every environment, it suffers from generally low variance and long convergence due to its equilibrium-seeking characteristic. These research results exhibit the trade-offs in terms of learning speed, stability, and adaptability in MARL. From the study, Meta-PG is noted to be effective for fast learning, COMA for adversarial interactions, while SQPG is a general method that needs more refinement. MARL's real-world applicability should be enhanced by focusing on hybrid models that combine the advantages of these approaches, improved variance reduction techniques, and iterative testing on a larger population of agents.