Hierarchical multi-agent reinforcement learning |
| |
Authors: | Mohammad Ghavamzadeh Sridhar Mahadevan Rajbala Makar |
| |
Affiliation: | (1) Department of Computing Science, University of Alberta, AB T6G 2E8, Canada;(2) Department of Computer Science, University of Massachusetts Amherst, MA 01003, USA;(3) Agilent Technologies, Santa Rosa, CA 95403, USA |
| |
Abstract: | In this paper, we investigate the use of hierarchical reinforcement learning (HRL) to speed up the acquisition of cooperative
multi-agent tasks. We introduce a hierarchical multi-agent reinforcement learning (RL) framework, and propose a hierarchical
multi-agent RL algorithm called Cooperative HRL. In this framework, agents are cooperative and homogeneous (use the same task decomposition). Learning is decentralized,
with each agent learning three interrelated skills: how to perform each individual subtask, the order in which to carry them
out, and how to coordinate with other agents. We define cooperative subtasks to be those subtasks in which coordination among agents significantly improves the performance of the overall task. Those
levels of the hierarchy which include cooperative subtasks are called cooperation levels. A fundamental property of the proposed approach is that it allows agents to learn coordination faster by sharing information
at the level of cooperative subtasks, rather than attempting to learn coordination at the level of primitive actions. We study the empirical performance of the
Cooperative HRL algorithm using two testbeds: a simulated two-robot trash collection task, and a larger four-agent automated guided vehicle
(AGV) scheduling problem. We compare the performance and speed of Cooperative HRL with other learning algorithms, as well as several well-known industrial AGV heuristics. We also address the issue of rational
communication behavior among autonomous agents in this paper. The goal is for agents to learn both action and communication
policies that together optimize the task given a communication cost. We extend the multi-agent HRL framework to include communication
decisions and propose a cooperative multi-agent HRL algorithm called COM-Cooperative HRL. In this algorithm, we add a communication level to the hierarchical decomposition of the problem below each cooperation level. Before an agent makes a decision at a cooperative subtask, it decides if it is worthwhile to perform a communication action. A communication action has a certain cost and provides
the agent with the actions selected by the other agents at a cooperation level. We demonstrate the efficiency of the COM-Cooperative HRL algorithm as well as the relation between the communication cost and the learned communication policy using a multi-agent
taxi problem. |
| |
Keywords: | Hierarchical reinforcement learning Cooperative multi-agent systems Coordination Communication |
本文献已被 SpringerLink 等数据库收录! |
|