Multiagent Model-based Credit Assignment for Continuous Control
Extracted source text · arXiv 2112.13937
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% LaTeX Template for AAMAS-2022 (based on sample-sigconf.tex)
%%% Prepared by Ana L. C. Bazzan and Lucas N. Alegre, with the contribution of the AAMAS-2022 Program Chairs. Thanks to Natasha Alechina. (version 2022-07-08)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Start your document with the \documentclass command.
%%% Use the first variant below for the final paper.
%%% Use the second variant below for submission.
\pdfoutput=1
\documentclass[sigconf]{aamas}
%\documentclass[sigconf,anonymous]{aamas}
%%% Load required packages here (note that many are included already).
%\usepackage{balance} % for balancing columns on the final page
\usepackage{todonotes}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% AAMAS-2022 copyright block (do not change!)
%\setcopyright{ifaamas}
%\acmConference[AAMAS '22]{Proc.\@ of the 21st International Conference
%on Autonomous Agents and Multiagent Systems (AAMAS 2022)}{May 9--13, 2022}
%{Auckland, New Zealand}{P.~Faliszewski, V.~Mascardi, C.~Pelachaud,
%M.E.~Taylor (eds.)}
%\copyrightyear{2022}
%\acmYear{2022}
%% do not change the following lines
\usepackage{flushend}
\setcopyright{ifaamas} % do not change this line!
\acmDOI{doi} % do not change this line!
\acmISBN{} % do not change this line!
\acmConference[AAMAS'22]{Proc.\@ of the 21st International Conference
on Autonomous Agents and Multiagent Systems (AAMAS 2022),
P.~Faliszewski, V.~Mascardi, C.~Pelachaud,
M.E.~Taylor (eds.)}{May 2022}{Auckland, New Zealand}
% do not change this line!
\acmYear{2022} % do not change this line!
\copyrightyear{2022} % do not change this line!
%\acmDOI{doi}
\acmPrice{}
\acmISBN{}
\newcommand{\subsectioninline}[1]{\vspace{0.2cm} \noindent \textbf{#1:}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Use this command to specify your EasyChair submission number.
%%% In anonymous mode, it will be printed on the first page.
%\acmSubmissionID{694}
%%% Use this command to specify the title of your paper.
\title{Multiagent Model-based Credit Assignment for \\Continuous Control}
%%% Provide names, affiliations, and email addresses for all authors.
% \author{Dongge Han}
% \affiliation{
% \institution{University of Oxford}
% \city{Oxford}
% \country{United Kingdom}
%}
% \email{dongge.han@cs.ox.ac.uk}
% \author{Chris Xiaoxuan Lu}
% \affiliation{
% \institution{University of Edinburgh}
% \city{Edinburgh}
% \country{United Kingdom}
%}
% \email{xiaoxuan.lu@ed.ac.uk}
% \author{Tomasz Michalak}
% \affiliation{
% \institution{University of Warsaw}
% \city{Warsaw}
% \country{Poland}}
% \email{tpm@mimuw.edu.pl}
% \author{Michael Wooldridge}
% \affiliation{
% \institution{University of Oxford}
% \city{Oxford}
% \country{United Kingdom}}
% \email{mjw@cs.ox.ac.uk}
\author{
Dongge Han$^{1}$,
Chris Xiaoxuan Lu$^{2}$,
Tomasz Michalak$^{3}$,
Michael Wooldridge$^{1}$
}
\affiliation{
\institution{$^{1}$ University of Oxford, Oxford, United Kingdom}
\institution{$^{2}$ University at Edinburgh, Edinburgh, United Kingdom}
\institution{$^{3}$ University at Warsaw, Warsaw, Poland}
\country{}
}
%%% Use this environment to specify a short abstract for your paper.
\begin{abstract}
\input{sections/abstract}
\end{abstract}
%%% The code below was generated by the tool at http://dl.acm.org/ccs.cfm.
%%% Please replace this example with code appropriate for your own paper.
%%% Use this command to specify a few keywords describing your work.
%%% Keywords should be separated by commas.
\keywords{Multiagent Systems, Reinforcement Learning, Cooperative Game Theory, Locomotion}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Include any author-defined commands here.
\newcommand{\BibTeX}{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em\TeX}
\input{macros}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
%%% The following commands remove the headers in your paper. For final
%%% papers, these will be inserted during the pagination process.
\pagestyle{fancy}
\fancyhead{}
%%% The next command prints the information defined in the preamble.
\maketitle
%\balance
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\input{sections/intro}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Background}
\input{sections/background}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{Method}
\input{sections/method}
\section{Experiments}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{sections/experiments}
\section{Related Work}
\input{sections/related}
\section{Conclusions}
\input{sections/conclusions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% The next two lines define, first, the bibliography style to be
%%% applied, and, second, the bibliography file to be used.
\bibliographystyle{plain}
%\balance % do not change this line
\bibliography{sample}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Deep reinforcement learning (RL) has recently shown great promise in robotic continuous control tasks. Nevertheless, prior research in this vein center around the centralized learning setting that largely relies on the communication availability among all the components of a robot. However, agents in the real world often operate in a decentralised fashion without communication due to latency requirements, limited power budgets and safety concerns. By formulating robotic components as a system of decentralised agents, this work presents a decentralised multiagent reinforcement learning framework for continuous control. To this end, we first develop a cooperative multiagent PPO framework that allows for centralized optimisation during training and decentralised operation during execution. However, the system only receives a global reward signal which is not attributed towards each agent. To address this challenge, we further propose a generic game-theoretic credit assignment framework which computes agent-specific reward signals. Last but not least, we also incorporate a model-based RL module into our credit assignment framework, which leads to significant improvement in sample efficiency. We demonstrate the effectiveness of our framework on experimental results on Mujoco locomotion control tasks. For a demo video please visit: \url{https://youtu.be/gFyVPm4svEY}.
\usepackage{xspace}
\usepackage{tabularx}
\usepackage{booktabs}
%\usepackage[labelfont=bf,format=plain,justification=raggedright,singlelinecheck=false]{caption}
% RL notations
\newcommand{\mdp}{\mathcal{M}}
\newcommand{\abstractmdp}{\bar{\mathcal{M}}}
\newcommand{\states}{\ensuremath{\mathcal{S}}}
\newcommand{\actions}{\ensuremath{\mathcal{A}}}
\newcommand{\dynamics}{\ensuremath{P}}
\newcommand{\data}{\ensuremath{\mathcal{D}}}
\newcommand{\reward}{\ensuremath{r}}
\newcommand{\policy}{\mathbf{\pi}}
% Model-based RL notations
\newcommand{\tmodel}{\ensuremath{f_s}}
\newcommand{\rmodel}{\ensuremath{f_r}}
\newcommand{\mparam}{\ensuremath{\phi}}
\newcommand{\aparam}{\ensuremath{\theta}}
\newcommand{\vparam}{\ensuremath{\omega}}
% Game theory notations
\newcommand{\coalition}{\ensuremath{\mathcal{C}}}
\newcommand{\amc}{\ensuremath{\mathcal{AMC}}}
\newcommand{\mc}{\ensuremath{\mathcal{MC}}}
\newcommand{\cv}{\ensuremath{v}}
\newcommand{\val}{\ensuremath{\psi}} % generic value
\newcommand{\vshapley}{\ensuremath{\psi_{\textnormal{Shapley}}}}
\newcommand{\vbanzhaf}{\ensuremath{\psi_{\textnormal{Banzhaf}}}}
\newcommand{\vloo}{\ensuremath{\psi_{\textnormal{LOO}}}}
\newcommand{\weight}[2]{\ensuremath{w_{#1}}}
\newcommand{\dongge}[1]{{\color{red} Dongge: #1}}
\newcommand{\chris}[1]{{\color{orange} Chris: #1}}
%\newcommand{\sysname}{\texttt{AutoTune}\xspace}
% for experiments
\newcommand{\mbshapley}{\texttt{MB-Shapley}\xspace}
\newcommand{\mbbanzhaf}{\texttt{MB-Banzhaf}\xspace}
\newcommand{\mbloo}{\texttt{MB-Loo}\xspace}
\newcommand{\qshapley}{\texttt{Q-Shapley}\xspace}
\newcommand{\mappo}{\texttt{MAPPO}\xspace}
\newcommand{\mbsize}[1]{MB-c={#1}}
% Packages
\usepackage{bbm}
\usepackage[export]{adjustbox}%http://ctan.org/pkg/adjustbox
\usepackage{tikz}
\usetikzlibrary{fit}
\usetikzlibrary{arrows.meta}
\usepackage{amsmath,amsthm,mathtools,nccmath}
\usepackage{graphicx}
\usepackage{subfig}
\usepackage{pdfpages}
\usepackage{subfiles}
\usepackage{float}
\usepackage{natbib}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{wrapfig}
\usepackage{aligned-overset}
\usepackage{thm-restate}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{pifont}
% trend
Reinforcement learning (RL) has recently shown many remarkable successes, e.g., in playing Atari and Go at a superhuman level~\cite{mnih2015human,silver2017mastering}. Recently, there have been wide research interest and significant advances in robotics learning using RL techniques~\cite{levine2016end,finn2016guided}. The topics of RL and classical control theory are closely related: both aim to find an optimal policy that optimizes an objective function, given a system represented by states and transition dynamics. Therefore, RL algorithms have the potential of enabling robots to learn %and improve through trial and error
in complex real-world tasks such as locomotion, manipulation and navigation.
Unlike classical RL tasks, which have discrete action spaces and underlying state spaces (e.g. Atari and Go), problems in robotics often have high-dimensional continuous states and actions, and are often limited by real-world sample budgets~\cite{kober2013reinforcement}. To this end, prior research in robotic learning have developed RL algorithms capable of performing continuous control~\cite{lillicrap2015continuous, gu2017deep, schulman2017proximal, haarnoja2018soft}, and sample-efficient learning methods, e.g.,~\cite{andrychowicz2017hindsight,janner2019trust, feinberg2018model}.
% Prior centralised work. However (assumption gap), communications etc. -> decentralised
%While prior RL techniques can learn control policies using high-dimensional, continuous states and action spaces,
Despite the success demonstrated by RL methods in continuous control, when applying RL algorithms to robotic applications, it is essential not to overlook real-life physical limits such as sensing noise and communication delays~\cite{kober2013reinforcement}. Specifically, prior RL approaches typically model a robot as a single, fully-centralised agent which observes the whole state consisting of observations from each body/joint and learns an optimal policy that outputs a combined action for all controllers. Though a centralized controller can compensate for the state of the whole system, when deployed in complex, unseen real-life settings, local observations may be noisy, and communications of the local observations between body components can be delayed due to physical limitations. Moreover, robots with a centralised controller do not scale up to large number of controllers and can easily become incapacitated if any sub-component is compromised. In such scenarios, a decentralised control system is desirable, where the robot controllers learn and execute their own policies.
% Multiagent is a good way for decentralised control.
Fortunately, many real-life robots consist of multiple connected links which can be modelled as a multiagent system~\cite{wooldridge2009introduction}. %Multiagent Systems research~\cite{wooldridge2009introduction} studies the interactions of decentralised, intelligent agents, e.g., coordination, cooperation, negotiation.
To enable coordination among agents with local observations and decentralised controllers, a standard approach is to perform centralized optimisation during training and decentralised operation during execution (CTDE)~\cite{foerster2018counterfactual, lowe2017multi}, i.e., the system can benefit from the state of the whole system during training, while acting in a decentralised manner. Using this framework, multiagent systems with RL agents have demonstrated coordination skills in complex coordination tasks such as Starcraft~\cite{samvelyan2019starcraft, rashid2018qmix, yu2021surprising}.
% However, technical challenge: 1. reward attribution, 2. model-free evaluation
Inspired by the above framework, we apply multiagent RL to robotics which exhibits high-dimensional, continuous state and action spaces. Specifically, we extend Proximal Policy Optimisation (PPO)~\cite{schulman2017proximal}---an algorithm widely used in both discrete and continuous control tasks---to a fully cooperative multiagent framework with CTDE.
However, as the robot interacts with the environment as a whole, \emph{only a global reward signal for the whole system is available while a reward function which signifies the contribution of each individual agent is absent.}
To address this problem, one can adopt solution concepts from cooperative game theory~\cite{chalkiadakis2011computational} that perform credit assignments for each agent, using its (marginal) contributions towards all possible coalitions of other agents. %, i.e., the difference between the value (characteristic value~\cite{chalkiadakis2011computational}) of a coalition before and after the agent joins.
Several prior multiagent RL methods have used one such well-known solution concept, the Shapley value~\cite{chalkiadakis2011computational}), for credit assignment in applications with discrete action spaces (such as traffic junction~\cite{wang2020shapley}, Starcraft~\cite{li2021shapley} and cooperative sensing~\cite{xu2020learning}). However, \emph{no model of this kind has been proposed for continuous action spaces so far}.
Our first contribution in this work is the formulation of a generic game-theoretic credit assignment framework for multiagent continuous control using semivalues~\cite{dubey1981value}---a wide family of solution concepts that encompass many common credit assignment methods such as the Shapley value, the Banzhaf value, etc. This formulation provides a useful tool for studying the underlying relationship and differences between the commonly used credit assignment methods in multiagent RL.
% Model-based intro
The formulation of game-theoretic credit assignment allows us to fairly evaluate the contribution of each agent.
However, to compute the credit assignments, \emph{how can we obtain the value of different coalitions of agents, when only the value of the grand coalition is available (i.e., the global reward of the whole system)?}
As we will discuss in Section~\ref{sub:mb}, if we simply apply the method to estimate values of coalitions that was proposed for models with discrete action spaces~\cite{li2021shapley} to continues ones, we would suffer from highly inaccurate results. This stems from a common problem in the multiagent robotics control domain, caused by the high dimensional continuous state and action spaces. To address this, we further propose a model-based framework for accurately estimating the values of coalitions of agents. This model-based framework greatly improves credit assignment in continuous control tasks and significantly boosts the rewards and sample-efficiency.
Finally, we empirically evaluate our proposed methods on MuJoCo robotic continuous control tasks. We demonstrate that our model-based game-theoretic credit assignment leads to significant improvements in rewards and sample-efficiency compared to a shared advantage function and model-free credit assignment methods.
%An obvious approach is to perform extra simulations, where for each simulation the grand coalition is replaced by a different coalitions of agents. This, however, would require exponentially more simulations which is expensive for mutliagent RL domains in general. Another approach which does not require extra simulations is to infer the value of coalitions from the present simulations~\cite{li2021shapley}. However, this approach suffer from inaccurate estimations in the multiagent robotics control domain, due to the high dimensional continuous state and action spaces. To accurately estimate the value of different coalitions, we draw inspiration from model-based RL, and propose to incorporate an additional model of the environment. In this way, we obtain a better value estimation through imaginary simulations of different coalitions from the model. Meanwhile, optimisation of the model is straightforward (via supervised learning), and does not require extra environment interactions: 1. the model is only used during (centralised) training time, hence only one centralised model is needed and can compensate for the state of the whole system, 2. like common model-based RL methods, we simultaneously learn the model and agent policies from scratch, i.e., iterate between fitting the model and updating the agent policies using the data collected from environment interactions, 3. we only use the model for generating imaginary simulations of short-horizons, which avoids model prediction errors due to long horizons. We will show in the experiments that the model-based credit assignment multiagent RL framework demonstrate significant improvement in performance and sample-efficiency as compared with using global rewards and their model-free counterparts.
%However, computing the credit assignment can be difficult, as it still requires evaluating the values of different coalitions of agents. Typically these values are not available (except for the grand coalition which can use the global reward for the whole system), and obtaining the values would require exponentially more simulations for different combinations of agents. Prior methods~\cite{li2021shapley} which choose to estimate the value of coalitions without extra simulations suffer from inaccurate estimations in the multiagent robotics control domain due to the high dimensional continuous state and action spaces. To this end, we propose to incorporate a model-based RL module for estimating the values of different coalitions.
% In summary, our contributions are as follows
%In summary, we make the following contributions:
%\begin{enumerate}
% \item{We model robotic continuous control problem as a fully-cooperative multiagent system and develop a multiagent PPO framework for continuous control using centralised training, decentralised execution.}
% \item{We propose a game-theoretic credit assignment framework for multiagent RL using semivalues, which provides a generic framework for evaluating contributions of each agent towards the global robotic control task, and encompass a wide class of common solution concepts.}
% \item{We propose a model-based framework for accurately estimating the values of coalitions of agents, which greatly improves credit assignment in continuous control tasks, and significantly improves in performance and sample-efficiency.}
% \item{We empirically evaluate our proposed methods on MuJoCo robotic locomotion tasks.}
%\end{enumerate}
\noindent In this section, we introduce our notation and main definitions (see Table~\ref{table:notations} for a summary of notation).
\subsection{Reinforcement Learning:}\label{subseq_background:rl}
A (single-agent) RL task is commonly defined by a reward function~\cite{sutton2018reinforcement}, and the goal of an agent is to learn an optimal policy which maximizes the cumulative future rewards. This can be modelled by the Markov decision process (MDP): $\mdp = \langle \states, \actions, \dynamics, \reward, \gamma \rangle$. At each time step $t$, an agent in state $s_t \in \states$, chooses an action $a_t \in \actions$, receives a reward $r_t = \reward(s_t,a_t)$, and transits to the next state $s_{t+1}\in \states$ according to an unknown transition dynamics $\dynamics:\states\times\actions\rightarrow\states$. The agent aims to maximise the future return $G_t = \sum_{k=0}^\infty{\gamma^k} r_{t+k+1}$ where $\gamma\in [0, 1]$ is a discount factor. We denote an agent's experience tuples as $\data_t = \langle s_t, a_t, s_{t+1}, r_t \rangle$. The policy of an agent is a mapping from states to probabilities of selecting a possible action, denoted as $\policy:\states\times\actions\rightarrow [0,1]$.
% add definition of policies, Values, Q-values, Advantage, reward-to-go. Bellman Equations
\subsubsection{Value Functions}\label{subsec:QV}
The \emph{(state) value function}, denoted $V^\pi(s)$, is the expected return of an agent which starts in $s$ and follows policy $\pi$ thereafter:
\begin{equation}
V^\pi(s) \coloneqq \mathbbm{E}_\pi[G_t|s_t=s] = \mathbbm{E}_\pi[\sum_{k=0}^\infty \gamma^k r_{t+k+1}| s_t=s], \forall s\in \states.
\end{equation}
The \emph{(action) value function} $Q^\pi(s, a)$ defines the expected return when starting from $s$, taking action $a$, and following $\pi$ thereafter:
\begin{align}
Q^\pi(s,a) &\coloneqq \mathbbm{E}_\pi[G_t|s_t=s, a_t=a]\nonumber\\ &\phantom{:}= \mathbbm{E}_\pi[\sum_{k=0}^\infty \gamma^k r_{t+k+1}| s_t=s, a_t=a], \forall s\in \states.
\end{align}
\begin{table}
\caption{Notation}\label{table:notations}
\begin{tabularx}{\linewidth}{p{0.1\textwidth}X}
\toprule
$i$, $t$, $k$ & Indices for agents, timesteps, iterations\\
$\pi^i$ & Policy of agent $i$ (Sec.~\ref{subseq_background:rl},~\ref{sub:mappo})\\
$G_t$ & Discounted return from $t$ onwards (Sec.~\ref{subseq_background:rl})\\
$V^\pi(s), Q^\pi(s, \mathbf{a})$ & State and action-value functions (Sec.~\ref{subsec:QV})\\
$A^\pi(s, \mathbf{a})$ & Advantage function (Sec.~\ref{subsec:PPO},~\ref{sub:mappo})\\
$\vparam$, $\aparam$, $\mparam_s$, $\mparam_r$ & parameters of $V, Q, \tmodel, \rmodel$ (Sec.~\ref{sub:mappo},~\ref{sub:mb})\\
$\cv^\coalition (s,\mathbf{a})$ & (Characteristic) value of coalition (Sec.~\ref{subsec:mc})\\
$\Tilde{\mathbf{a}}_t$ & Actions $\mathbf{a}_t$ masked by coalitions (Sec.~\ref{subsec:mc})\\
$\mc^i(\coalition, s, \mathbf{a})$ & Marginal contribution of agent $i$ to coalition $\coalition$ \\
$\val^i(v)$, $p_c$ & semivalues and its probability indices (Sec.~\ref{subsec:semi})\\
$\tmodel(s,\mathbf{a}),\rmodel(s,\mathbf{a})$ & dynamics and reward models (Sec.~\ref{subsec:model})\\
$\hat{s}_{t+1}, \hat{r}_t$ & States/Rewards predicted by model (Sec.~\ref{subsec:model})\\
\bottomrule
\end{tabularx}
\end{table}
\subsubsection{Proximal Policy Optimisation (PPO)}\label{subsec:PPO}
PPO~\cite{schulman2017proximal} is an actor-critic policy gradient RL algorithm which is widely used in both discrete and continuous control tasks, due to its simplicity in implementation, sample complexity, and ease of hyperparameter tuning. Specifically, an \emph{actor} corresponds to the policy $\pi(a|s)$, while a \emph{critic} corresponds to the value function $V^\pi(s)$. The actor update is guided by an \emph{advantage function} $A(s, a)$ estimated using the critic.
The advantage function $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)$ measures
how much better (worse) taking action $a$ in state $s$ is compared with the policy’s default behavior.
Intuitively, if the advantage $A(s, a)$ is positive, the actor parameters move towards the direction where this action becomes more likely. To improve stability, PPO avoids parameter updates that change the policy too much by clipping the ratio between old and new policies.
In practise, $A^\pi(s,a)$ can be estimated using methods such as general advantage estimation (GAE)~\cite{schulman2015high}, which makes use of value function $V^\pi(s)$ (the details of GAE can be found in~\cite{schulman2015high}).
\subsubsection{Centralised Training, Decentralised Execution}\label{subsec:CTDE} A standard practise in training a decentralised multiagent RL system is
Centralised Training, Decentralised Execution (CTDE)~\cite{foerster2018counterfactual}.
For example, in a multiagent system with agents $N=\{1,\ldots, n\}$, a \emph{centralised critic} (typically, action value functions $Q(\mathbf{s}_t, a_1, \ldots, a_n)$)
and decentralised actors $(\pi_1, \ldots, \pi_n)$ are learned.
The centralised critic takes as input the observations and joint actions of all agents, while the decentralised actors take the agents' local observations as input and outputs an action per agent.
Compared with having decentralised critics (one per agent) which only conditions on the local observations and actions, the advantage of CTDE is that the centralised critic prevents an agent from experiencing un-stationary environment as a result of treating other learning agents as part of the environment. Moreover, the critic is only used during training; hence, the agents learn coordinated behaviours which can be executed in a decentralised manner during execution.
%\textbf{Counterfactual Credit Assignment}~\cite{foerster2018counterfactual} In a cooperative multiagent system, the group of agents aim to optimise a shared goal, and receive a single shared reward when interacting with the environment. Counterfactual credit assignment methods allow the system to divide the reward and assign to each agent a credit according to its contribution. For example, an agent $i$'s contribution can be evaluated according to its \emph{counterfactual value} to the system, i.e., the difference between the future return of the system with and without the agent.
\noindent
\subsection{Cooperative Game Theory}\label{subsec:cgt}
A \emph{cooperative game} is given by a pair $G = (N, \cv)$, where $N = \{1,\ldots,n\}$ is the set of agents (players), and $\cv\colon 2^N\rightarrow{\mathbb{R}}$ is the \emph{characteristic function}, which assigns a real value $\cv(\coalition)$ to every coalition $\coalition$ reflecting its performance.
We assume $\cv(\emptyset) = 0$.
Subsets of agents $\coalition \subseteq N$ are called \emph{coalitions}.
%We will call $\cv(\coalition)$ the value of a coalition for simplicity.
To measure the contribution of an individual agent to a cooperative task, % (e.g. how much a front foot of a cheetah robot contributed to the locomotion task),
we use solution concepts~\cite{chalkiadakis2011computational} from cooperative game theory that evaluate the contribution of each agent $i \in N$ to the game. We will denote it $\val^i\in \mathbb{R}$. Specifically, we focus on a wide class of solution concepts called semivalues~\cite{dubey1981value}. For clarity, we will introduce the general definition of semivalues in Section~\ref{sub:gt}. Before this, let us introduce the concept of \emph{marginal contribution} and two well-known semivalues: the Shapley and Banzhaf values. Intuitively, the marginal contribution of $i$ to coalition $\coalition$ is the difference that this agent makes towards $\coalition$ before and after joining it, i.e., $\mc^i(\coalition) \coloneqq \cv(\coalition\cup\{i\}) - \cv(\coalition)$.
%A common class of solution concepts (e.g., Shapley value) defines the contribution of a player $i$ as a weighted sum of $i$'s marginal contributions towards coalitions of other players,
%i.e.,
%\begin{equation}
%\val^i(N, \cv) = \sum_{\coalition\subseteq N\setminus\{i\}} \weight{\coalition}{N} \mc^i(\coalition)
%\end{equation}
%\noindent
The Shapley value~\cite{Shapley+2016+307+318} is the most common solution concept. It is defined as follows: %assigning a uniform weight $\frac{1}{N}$ towards the average value of marginal contributions of different sized coalitions:
\begin{equation*}
\vshapley^i(N, \cv) = %\sum_{S\subseteq N \setminus \{i\}} \frac{1}{N} \tbinom{N-1}{|\coalition|}^{-1} \mc^i(\coalition)
\sum_{S\subseteq N \setminus \{i\}}\frac{|\coalition|!(|N|-|\coalition| -1)!}{|N|!} \mc^i(\coalition).
\end{equation*}
The Banzhaf value~\cite{banzhaf1964weighted} is another common solution concept: %It is defined by the average marginal contribution of a player towards all coalitions of other players:
\begin{equation*}
\vbanzhaf^i(N, \cv) = \frac{1}{2^{|N|-1}}\sum_{\coalition\subseteq N \setminus \{i\}} \mc^i(\coalition).
\end{equation*}
Intuitively, the Banzhaf value of $i$ is simply the average marginal contribution of this agent across all possible coalitions. The Shapley value is the weighted average, where the weight of each marginal contribution depends on the size of the coalition and the total number of agents.
%\item{{\it Leave-one-out (LOO)}
%assigns to each player the marginal contribution towards the coalition of all other players:
% \begin{equation*}
% \vloo^i = \mc^i(N\setminus\{i\}).
% \end{equation*}}
\subsection{Kinematics Tree of Robots}\label{subsec:mujoco}
A robot capable of performing continuous control can be defined via a kinematics tree~\cite{spong2006robot, wang2018nervenet} of nested bodies, specifying the mechanical structure and physical properties of the robot. Within such a kinematics tree, there are three major types of entities: \texttt{Bodies, Joints and Actuators}. The \texttt{Bodies} represent the rigid bodies of the robot, e.g., foot. Physical properties such as the relative position and geometry are specified for each body. The \texttt{Joints} are moveable components which connect parent and child bodies, and creates motion degrees of freedom between them. Typical joints include ball joints (3 rotational degree of freedom), hinge joints (1 rotational degree of freedom) and slide joints (1 translational degree of freedom). For example, in a cheetah robot (Figure~\ref{fig:mujoco}), the torso (root \texttt{body}) can perform translational and rotational movements through slide and hinge \texttt{joints}. The back foot (\texttt{body}) is attached to its parent back shin (\texttt{body}) through a hinge \texttt{joint}. Attached to the joints are \texttt{Actuators} which are the components that actuate the joints. Many real life robots consist of multiple connected links, %including torsos and legs, while each leg is composed of thigh, shin and foot.
we now introduce how to model the robot as a multiagent system. % using their kinematics trees.
%In this section, we present our Multiagent RL framework based on Proximal Policy Optimisation (PPO)~\cite{schulman2017proximal} and the Centralised training decentralised execution paradigm (CTDE)~\cite{foerster2018counterfactual}. In particular, we will first discuss the extension of the standard PPO algorithm to a cooperative multiagent setting in Section~\ref{sub:mappo}. Then in Section~\ref{sub:gt}, we will present a generic game-theoretic framework for credit assignment of agents, which explains the relation and differences among commonly adopted solution concepts. In Section~\ref{sub:mb}, we propose a model-based learning framework which provides improved estimation of the counterfactual values, hence enables the computation of game-theoretic credit assignment solutions.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{img/mujoco.pdf}
\caption{Multiagent MuJoCo Cheetah}
\label{fig:mujoco}
\end{figure}
\iffalse
\begin{figure*}[t]
\centering
\subfloat[Multiagent MuJoCo Cheetah]{\includegraphics[width=0.4\linewidth]{img/mujoco.pdf}\label{fig:mujoco}}%
\hfill
\subfloat[Multiagent PPO]{\includegraphics[width=0.5\linewidth]{img/mppo.pdf}\label{fig:mappo}}%
\caption{Multiagent Locomotion}
\label{fig:mppo}
\end{figure*}
\fi
\section{Multiagent Continuous Control}\label{sub:maloco}
\subsection{Problem Formulation}\label{subsec:robot_multiagent}
\noindent \subsubsection{Robot Joints as a Multiagent System} Figure~\ref{fig:mujoco} shows an example cheetah robot built using MuJoCo~\cite{brockman2016openai} (more details in Section \ref{exp:mujoco}).
We model the robot as a multiagent system of $N$ agents, where each agent represents: 1. a rigid body component (e.g., the back foot) of the robot, 2. the joint which attaches the body to its parent (e.g., the ankle joint which attaches the back foot to the back shin), and 3. any actuators attached to the joint. In this way, the agent receives input of the local sensor observations to its joint (e.g., positions, angles, translational and angular velocities, external forces and torques), and outputs a control action for the actuator.
\noindent \subsubsection{Multi-agent based Robot Locomotion.} Typically in the robotic control problems, the actuators have a continuous action space which are normalised to $[-1,1]$ for the convenience of learning. In this work, we focus on the robot locomotion task, which is a representative continuous robot control problem in this family.
The objectives of robot locomotion tasks are to train the robot to learn control policies in order to transport from place to place through walking, hopping, swimming, etc. Here we model the locomotion tasks as a fully cooperative Multiagent decision making problem, described by a Markov game~\cite{littman1994markov} denoted by a tuple $\langle N, \states, \actions, \dynamics, \reward, \gamma \rangle$, where $N$ is the set of all agents $N=\{1,\ldots, n\}$, and $\states$ are global states of the environment. Each agent $i$ has a continuous action space $\actions_i$ and the combined action space of the robot is $\actions = \actions_1 \times \ldots \times \actions_n$. At each step $t$, each agent $i$ makes a local observation $s^i_t = o^i(s_t)$, and chooses an action simultaneously according to its own policy $a^i_t\sim\pi^i(s^i_t)$. Then, a combined action composed of the actions of all agents $\mathbf{a}_t = (a^1_t, \ldots, a^n_t)$ is performed on the environment (the combined policy of all agents is denoted as $\pi = (\pi^1,\ldots, \pi^n)$). Upon taking the action, all agents receive a shared reward $r_t = \reward(s_t, \mathbf{a}_t)$ which evaluates how well the robot performed (e.g., the distance travelled along a specified direction, the control energy consumption, the stability of the robot, etc), and the environment transitions to the next state according to the transition dynamics $\dynamics:\states\times\actions\rightarrow\states$. At each step, the agents aim to maximise the cumulative future discounted return $G_t = \sum_k{\gamma^k}r_{t+k+1}$ where $\gamma\in [0, 1]$ is a discount factor.
\subsection{Multiagent PPO with Shared Advantage}\label{sub:mappo}
Having formulated the multiagent decision problem, we next introduce an algorithm for optimising the agents' policies.
To optimise the continuous control policies, we extend the standard PPO algorithm with actor-critic framework (c.f. Section~\ref{subsec:PPO}) to cooperative multiagent PPO. In particular, we adopt the centralised training, decentralised execution (CTDE) paradigm~\cite{foerster2018counterfactual}, c.f. Section~\ref{subsec:CTDE}. As the agents are fully cooperative and the whole robot receives a global reward signal, we let the agents share a centralised critic which estimates the value function of the global state $V_\vparam: \states\rightarrow \mathbb{R}$. The centralised critic is parametrised by $\vparam$, and in our case, $\vparam$ is a neural network. The critic is only used during training as a guide for optimising the actors and will be dismissed in the execution phase---during the execution only the actors are used to choose the actions. The actors are decentralised, i.e., for the $i-$th agent, its actor $\pi_{\aparam^i}: \states^i \rightarrow \actions^i$ parametrised by neural network $\aparam^i$, chooses its own action given the local observations. Most locomotion tasks have continuous action space by nature; hence we model each agent's policy outputs by a Gaussian distribution. At each step, action is sampled from the Gaussian distribution. During training, each actor learns to update the mean of the Gaussian distribution, and the standard deviation can be either static or learned.
We optimise the centralised critic and decentralised actors following the PPO algorithm:
At each epoch $k$, the agents collect a set of trajectories $\data_k$ by running their current policy $\pi_k = (\pi^1(\aparam^1_k), \ldots \pi^n(\aparam^n_k))$ in the environment.
Then, the critic which estimates the (state) value function $V(s)$ is updated by regression. The loss is defined as the mean squared error between the predicted value of the state $s_t$ the empirical return $\hat{G}_t$ from $t$ onwards:
\begin{equation*}
\vparam \leftarrow \arg\min_\vparam \frac{1}{|\data_k|T}\sum_{\tau\in{\data_k}}\sum_{t=0}^T\bigl(V_\vparam(s_t) - \hat{G}_t\bigr)^2.
\end{equation*}
The actor of each agent is updated by optimising the PPO clipped surrogate objective:
\begin{align*}
\aparam^i_{k+1} = \arg\max_{\aparam^i} \frac{1}{|\data_k|T} \sum_{\tau\in\data_k}\sum_{t=0}^T \min\Big(\frac{\pi_{\aparam^i}(a^i_t|s^i_t)}{\pi_{\aparam^i_k}(a^i_t|s^i_t)} A(s_t, \mathbf{a_t}),\\ \textnormal{clip}\bigl(\frac{\pi_{\aparam^i}(a^i_t|s^i_t)}{\pi_{\aparam^i_k}(a^i_t|s^i_t)}, 1-\epsilon, 1+\epsilon\bigr)A(s_t, \mathbf{a_t}) \Big),
\end{align*}
\noindent where $A(s_t, \mathbf{a}_t)$ is the shared advantage function computed using the critic through generalised advantage estimation (c.f. Sec.~\ref{subsec:PPO}). Intuitively, if the advantage $A(s, a)$ is positive, the actor parameters update in the direction where this action becomes more likely. PPO avoids parameter updates that change the policy too much by clipping the ratio between old and new policies~\cite{schulman2017proximal}.
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{img/mb_drawing.pdf}
\caption{Model-based Multiagent Credit Assignment. Our framework consists of three modules: (yellow) the centralised critic module which consists of a dynamics/reward model $f=(f_s, f_r)$ and a centralised state-value function $V^\pi(s)$; (red) credit assignment module which queries the critic for coalition values and computes the counterfactual solution concepts assigned to each agent; (green) actors module with $N$ agents, which are updated using the credits from the credit assignment module.}
\label{fig:mb}
\end{figure*}
\input{sections/gametheory}
\subsection{Model-based Advantage Estimation}\label{sub:mb}
% Motivation for having a model
The semivalues (e.g., the Shapley and Banzhaf values) as agent-specific pseudo advantages, allow us to evaluate the contribution of each agent and its chosen action.
\emph{However, how can we obtain the coalitions' values, when only the value of the grand coalition is available (i.e., the global reward of the whole system)?} %Inferring the coalition values using a single Q-value critic suffer from inaccurate estimations in the multiagent robotics control domain, due to the high dimensional continuous state and action spaces.
An obvious approach is to perform extra simulations, where for each simulation the grand coalition is replaced by a different coalitions of agents. This, however, would require exponentially more simulations which is expensive for mutliagent RL domains in general. Another approach which does not require extra simulations is to infer the value of coalitions from the present simulations~\cite{li2021shapley}. However, this approach suffers from inaccurate estimations in the multiagent robotics control domain, due to the high dimensional continuous state and action spaces.
To accurately estimate the value of different coalitions, we draw inspiration from model-based RL and propose incorporating an additional model of the environment dynamics and rewards. In this way, we obtain a better coalition value estimation through model-based simulations of different coalitions. Meanwhile, optimisation of the model is straightforward (via supervised learning) and does not require extra environment interactions.%:
%We will show in the experiments that the model-based credit assignment multiagent RL framework demonstrate significant improvement in performance and sample-efficiency as compared with using global rewards and their model-free counterparts.
%To train the multiagent PPO using the agent-specific advantages, accurate estimations of the coalition values is crucial. In this section, we propose a model-based learning framework which significantly improves the estimation of coalition values and sample efficiency.
%Recall that to compute the semivalue of each agent, we need to estimate the values of all (or sampled) coalitions, which are defined as the state-action value $Q^\pi(s, \Tilde{a}^1, \ldots, \Tilde{a}^n)$, i.e., the expected discounted future cumulative rewards when the agents are at state $s$ and taking joint action $\Tilde{\mathbf{a}} = (\Tilde{a}^1, \ldots, \Tilde{a}^n)$, where $\Tilde{a}^i$ means the actions of agents outside the coalition are masked to the default value.
%However, using a single critic neural network which estimates the $Q$-value often generates inaccurate estimations, as the critic is optimised using the experiences where the actions are not masked to default ones.
%To produce better estimations for the coalition values, we use the model-based RL framework. Through model-based RL, we are able to estimate the state-action value function $Q^\policy(s, \mathbf{\Tilde{a}})$ using the state-value function $V^\policy(s)$ and a dynamics/reward model of the environment. The details are explained in the following.
\input{sections/algorithm}
\subsubsection{Dynamics and Reward Model}\label{subsec:model}
% formulation of a model
We use a \emph{model} parametrised by $\mparam = (\mparam_s, \mparam_r)$, which consists of a transition model $\tmodel$ which maps the state-action tuple to the the difference between the next and current state (alternatively, $\tmodel$ can map the state-action tuple to the next state directly), and a reward model $\rmodel$ which maps the state-action tuple to the reward~\cite{nagabandi2018neural}:
\begin{align}
\forall s_t\in \states, a_t\in \actions, \quad
\hat{s}_{t+1} &= \tmodel(s_t, a_t) + s_{t},\\
\hat{r}_{t} &= \rmodel(s_t, a_t).
\end{align}
The model can be optimised through supervised learning, by regression using mean-squared error between the predicted next states/rewards and actual ones.
\begin{align}
&\min_{\mparam_s} \frac{1}{|\data|}\sum_{\langle s_t, s_{t+1}, \mathbf{a}_t, r_t\rangle\in \data}\|s_{t+1} - (s_{t} + f_s(s_t, \mathbf{a}_t))\|^2\\
&\min_{\mparam_r} \frac{1}{|\data|}\sum_{\langle s_t, s_{t+1}, \mathbf{a}_t, r_t\rangle\in \data}\|(r_t - f_r(s_t, \mathbf{a}_t)\|^2.
\end{align}
The model is only used in (centralised) training, hence only one centralised model is needed and can compensate for the state of the whole system. During each iteration of training, we alternate between model fitting and agent updates using the data collected from environment interactions. Moreover, we only use the model for generating imaginary simulations of short-horizons, which mitigates the error-prone model predictions drifted in long horizons.
\subsubsection{Estimating the coalition values using the model}
Inspired by model-based value expansion~\cite{feinberg2018model}, we use the model to perform estimation of the one-step transition dynamics and rewards. In this way, we obtain a better value estimation through imaginary simulations of different coalitions using the dynamics/reward model. Through the Bellman equation, we can estimate the action-value function through a state-value function $V^\policy$ in the following:
\begin{align}
Q^\policy(s_t, \Tilde{\mathbf{a}}_t)
&= \mathbbm{E} [r_t + \gamma V^\policy(s_{t+1}) |\phantom{s} s_t, a_t ] \nonumber\\
&= \mathbbm{E} [\hat{r}_t + \gamma V^\policy(\hat{s}_{t+1}) |\phantom{s} s_t, a_t ]\nonumber\\
&\approx f_r(s_t, \Tilde{\mathbf{a}}_t) + \gamma V^\policy(s_t + f_s(s_t, \Tilde{\mathbf{a}}_t)).
\end{align}
%To summarise the procedure, a
A schematic of the framework is shown in Figure~\ref{fig:mb}. At each training step, to produce the credit value for each agent $i$, the credit assignment module first samples coalitions $\coalition_m\subseteq N\setminus\{i\}$ according to the semivalue, and computes the average over counterfactual value of $i$ towards the sampled coalitions $\mc_i(\coalition_m) = \cv(\coalition_m\cup\{i\}) - \cv(\coalition_m)$. To obtain the value of the coalitions, say $\cv^{\coalition_m}(s_t, \mathbf{\Tilde{a}}_t)=Q^\pi(s_t, \Tilde{\mathbf{a}}_t)$, the credit assignment module queries the critic module (yellow in Figure~\ref{fig:mb}): the model (blue) first produces the next state $\hat{s}_{t+1} = f_s(s_t, \Tilde{\mathbf{a}}_t)$ and reward $\hat{r}_t = f_r(s_t, \Tilde{\textbf{a}}_t)$, then the centralised state-value critic (grey) produces $V^\pi(\hat{s}_{t+1})$. Together with the reward $r_t$ produced by the model, the critic module outputs the value of the coalition $\coalition_m$ as $Q^\pi(s_t, \Tilde{\textbf{a}}_t) = \hat{r}_t + \gamma V^\pi(\hat{s}_{t+1})$. Finally, having obtained the credit value assigned to each agent, the actors module (green) can update the decentralised actors through multiagent PPO.
% usage of the model
The full algorithm is shown in Algorithm~\ref{algo:mb}.
\subsection{Multiagent PPO with Game-theoretic Agent-specific Advantage}\label{sub:gt}
%As observed by~\cite{yu2021surprising} in discrete domains (e.g., Starcraft) and in our experiments (continuous domains), t
The above Multiagent PPO with shared advantage function often yields decent performance for cooperative multiagent tasks. However, the shared advantage function $A(s_t, \mathbf{a}_t)$ only evaluates the quality of the combined action, which fails to assess each agent's individual contribution. As we will discuss in Sec.~\ref{exp:central_vs_multi}, failing to address per-agent contribution can result in low sample efficiency oftentimes, for example, an under-actuated agent can equally share and update its policy using the global advantage. To deal with this issues, we leverage the fair credit assignment methods from cooperative game theory, and present a generic game theoretic framework for agent-specific advantage computation in Multiagent PPO.
%\dongge{removed contents:}
%Specifically, at each training step, we evaluate the credit (contribution) of an agent's chosen action towards the whole system. This credit measures how much better (or worse) an agent's chosen action contributed towards other agents, compared with its default action. Then the computed credit will be used as the agent-specific advantage to guide their actor optimisations.
\subsubsection{The Characteristic Function}\label{subsec:mc}
%When defining the characteristic value of a coalition, we adopt the convention where actions of agents outside the coalition are masked to a default value~\cite{li2021shapley}. That is, given the agents are at states $\mathbf{s}=(s_1, \ldots, s_n)$ and taking action $\mathbf{a}=(a_1, \ldots, a_n)$, the value of a coalition $\coalition$ is given by:
To compute the value of an agent assigned by the game-theoretic solution concepts, we first need to define the characteristic function. At each timestep $t$, given the environment state $s_t$ and agents' joint action $\mathbf{a}_t = (a^1_t,\ldots, a^n_t)$, let the value of a coalition $\coalition$ be defined as:
\begin{align}
\cv^\coalition( s_t, \mathbf{a}_t) &= Q^\pi(s_t, \Tilde{a}^1_t, \ldots, \Tilde{a}^n_t), \textnormal{ where}\label{eq:coalition}\\
\Tilde{a}^i_t &=
\begin{cases}
a^i_t \quad \quad \quad \textnormal{if } i\in\coalition\\
a_\textnormal{default} \quad \textnormal{if } i\in N\setminus\coalition,
\end{cases}\nonumber
\end{align}
\noindent where $\Tilde{a}^i_t$ denotes the action of agent $i$ is replaced by a default one if $i$ is outside the coalition, a widely adopted practice~\cite{li2021shapley,wolpert2002optimal}. To follow the game theoretic conventions where empty coalitions have zero value, we can normalise the characteristic value function by subtracting a baseline value $Q^\pi(s_t, \mathbf{a}_\textnormal{default})$. However, the marginal contributions will stay invariant so we will use the above definition for simplicity. Given this characteristic function, agent $i$'s marginal contributions towards coalition $\coalition$:
%Having defined the value of a coalition, we will now define the counterfactual value of agents. In particular, the counterfactual value of an agent $i$ shows the difference it makes towards a coalition of agents $\coalition$, i.e., the difference between the value of the coalition before and after $i$ joined. Given the agents are in state $\mathbf{s}$ and taken action $\mathbf{a}$, we define the counterfactual value of a agent $i$ towards coalition $\coalition$ as
\begin{align}
\mc^i(\coalition, s_t, \mathbf{a}_t) &= \cv^{\coalition \cup \{i\}}(s_t, \mathbf{a}_t) - \cv^{\coalition}(s_t, \mathbf{a}_t),
\end{align}
\noindent represents the difference made by $i$'s chosen action towards coalition $\coalition$, compared with the default action.
%\input{sections/sampling_algorithm}
\subsubsection{Agent-Specific Advantage}\label{subsec:semi}
%\dongge{Common counterfactual methods (e.g. Shapley value~\cite{li2021shapley}, Counterfactual value~\cite{foerster2018counterfactual}) are used for credit assignment in Multiagent RL.
%{However, these methods are often considered distinct and their relationship being overlooked.}
We now introduce a generic game-theoretic credit assignment framework for multiagent continuous control using semivalues~\cite{dubey1981value} -- a wide family of solution concepts that encompass many common credit assignment methods such as the Shapley value, Banzhaf value, etc.
%We consider a class of solution concepts where the weight $\weight{S}{N}$ for agent $i$'s counterfactual value towards coalition $\coalition$ only depends on the size of $\coalition$ but not on the agents inside. That is, $|\coalition_1|=|\coalition_2| \implies \weight{\coalition_1}{N}= \weight{\coalition_2}{N}$.
%Hence these solution concepts can be defined by
%$\varphi_i(N, v) = \sum_{\coalition\subseteq N\setminus\{i\}} \weight{c}{N} \mc_i(\coalition)$, where $c=|\coalition|$.
%By grouping together equal-sized coalitions, the solution concepts can be expressed as a weighted sum of agent $i$'s \emph{average counterfactual value towards size-$c$ coalitions $\amc_i(c)$}. For most common solution concepts, the weights form a probability distribution and sums up to 1: $\sum_{c=0}^{N-1}\alpha_{c} = 1$, and the class of solution concepts are called Semivalues~\cite{dubey1981value}.
\begin{equation*}
\val^i(\cv) = \sum_{\coalition\subseteq N\setminus\{i\}} \weight{c}{N} \mc^i(\coalition) \textnormal{ where } c=|\coalition|, \sum_{c=0}^{|N|-1}\weight{c}{N}\tbinom{|N|-1}{c} = 1.
\end{equation*}
To better understand them, we can rewrite the semivalues as:
\begin{equation}
\val^i(\cv) = \sum_{c=0}^{|N|-1} p_c \frac{\sum_{\coalition\subseteq N\setminus\{i\}, |\coalition| =c} \mc^i(\coalition)}{\tbinom{|N|-1}{c}}, \textnormal{where} \sum_{c=0}^{|N|-1}p_c = 1.
\label{eq:semivalue}
\end{equation}
Note that the denominator $\tbinom{|N|-1}{c}$ is the number of size-$c$ coalitions excluding player $i$. Intuitively, the semivalue is a weighted sum of an agent's average marginal contribution towards different sized coalitions,
where $p_c$ form a probability distribution. In particular,
the Shapley value places uniform weight on all coalition sizes: $p_c = \frac{1}{|N|}$;
%The counterfactual value only considers size $N-1$ coalition, i.e., $p_c =\mathbbm{1}_{c = |N|-1}$;
and the Banzhaf value has a bell-shaped distribution, i.e., $p_c = \frac{1}{2^{|N|-1}}\tbinom{|N|-1}{c}$. Any other probability distribution $p_c$ also specifies a semivalue.
In our model, \emph{at each timestep, the semivalue $\val^i$ of an agent computes the contributions of the agent's chosen action towards the global future return of the robot, and hence will be used as the agent-specific pseudo advantage for multiagent PPO updates.}
We call the semivalues ``pseudo'' advantage because they are not defined strictly as $A(s, a) = Q(s, a) - V(s)$. Nevertheless, the semivalues provide an effective proxy of advantage as they can evaluate the individual contribution of an agent's action towards the global value.
Since RL training procedures typically require millions of timesteps, enumerating all possible coalitions and computing their characteristic value at each time step is inefficient.
A simple procedure for estimating the semivalue is through Monte-Carlo sampling and output the average over the agent's marginal contribution towards all sampled coalitions. For each sample drawn, we can first sample a coalition size $c_m$ according to the semivalue distribution $p_c$, then uniformly sample a coalition $\coalition_m$ of the size $c_m$.
%As a special case, the Banzhaf value can be easily estimated by uniformly sampling coalitions of arbitrary coalition size.
%\begin{equation*}
% \hat{\val}_i = \frac{1}{M} \sum_{m=1}^M \mc_i(\coalition_m), \textnormal{where } \coalition_m\sim \mathit{Uniform}(2^{N\setminus\{i\}})
%\end{equation*}
\begin{algorithm}[t]
\caption{Model-based Multiagent Credit Assignment}
\label{algo:mb}
\begin{algorithmic}[1]
\State{\textbf{Input:} $N=\{1,\ldots,n\}$: the set of all agents.}
\State{\textbf{Initialise: } model $f = (f_s, f_r)$, centralised critic $V_{\vparam}$, decentralised actors $\pi^i_{\aparam}$ for each agent $i\in N$}
\For{iterations $k=0,1,2,\ldots $}
\State{Collect sets of trajectories $\data_k=\{\tau\}$ by running policy $\pi_k = (\pi^1_{\aparam^1_k}, \ldots \pi^n_{\aparam^n_k})$ in the environment. Compute returns $\hat{G}_t$}
\State{$\#$ Fit the dynamics/reward model (in minibatches):}
\State{\phantom{$\#$} \resizebox{.9\hsize}{!}{$f_s \leftarrow \arg\min_{f_s} \frac{1}{|\data_k| T}\sum_{\tau \in \data_k}\sum_{t=0}^{T}\|s_{t+1} - (s_{t} + f_s(s_t, \mathbf{a}_t))\|^2$}}
\State{\phantom{$\#$} \resizebox{.8\hsize}{!}{$f_r \leftarrow \arg\min_{f_r} \frac{1}{|\data_k|T}\sum_{\tau \in \data_k}\sum_{t=0}^{T}\|(r_t - f_r(s_t, \mathbf{a}_t)\|^2$}}
% now the RL updates
\State{$\#$ Compute per-agent advantage $\val_i$ for all timesteps $t$:}
\State{\phantom{$\#$} Sample coalitions $\coalition_m$}
\State{\phantom{$\#$} Compute coalition values by model $f$ and critic $V_{\vparam_k}$: }
\State{\phantom{$\#$}\phantom{----}$v(C_m, s_t, \mathbf{{a}}_t)) = f_r(s_t, \mathbf{\Tilde{a}}_t) + \gamma V^\pi_{\vparam}(f_s(s_t, \mathbf{\Tilde{a}}_t) + s_t)$}
\State{\phantom{$\#$} Compute marginal contribution $\mc^i(\coalition_m, s_t, \mathbf{a}_t)$}
\State{\phantom{$\#$} Compute semivalues $\val^i$ using the marginal contributions. }
\State{$\#$ For each agent $i$, update the policy by maximising the PPO-Clip objective:}
\State{\phantom{$\#$} $\aparam^i_{k+1} = \arg\max_{\aparam^i} \frac{1}{|\data_k|T} \sum_{\tau\in\data_k}\sum_{t=0}^T \min\Bigl(\frac{\pi_{\aparam^i}(a^i_t|s^i_t)}{\pi_{\aparam^i_k}(a^i_t|s^i_t)}\val^i(s_t, \mathbf{a_t}),$} \State{\phantom{ space}$\textnormal{clip}\bigl(\frac{\pi_{\aparam^i}(a^i_t|s^i_t)}{\pi_{\aparam^i_k}(a^i_t|s^i_t)}, 1-\epsilon, 1+\epsilon\bigr)\val^i(s_t, \mathbf{a_t}) \Bigr)$,}
\State{$\#$ Fit the centralised critic by regression:}
\State{\phantom{$\#$} $\vparam_{k+1} = \arg\min_\vparam \frac{1}{|\data_k|T}\sum_{\tau\in{\data_k}}\sum_{t=0}^T\bigl(V_\vparam(s_t) - \hat{G}_t\bigr)^2$}
\EndFor
\end{algorithmic}
\end{algorithm}
\begin{figure*}[t]
\centering
\begin{minipage}{1\linewidth}\centering
\includegraphics[height=0.04\linewidth]{img/legend.pdf}
\end{minipage}
\vfill
\vspace{-3mm}
\subfloat[Cheetah (Rewards)]{\includegraphics[width=0.47\linewidth]{img/Cheetah.pdf}\label{fig:cheetah}}%
\hfill
\subfloat[Ant (Rewards)]{\includegraphics[width=0.47\linewidth]{img/Ant.pdf}\label{fig:ant}}%
\vfill
\vspace{-3mm}
\caption{Average Rewards Multiagent Cheetah and Ant}
\label{fig:all_results}
\end{figure*}
\begin{figure}
\centering
\begin{minipage}{1\linewidth}\centering
\includegraphics[height=0.05\linewidth]{img/legend.pdf}
\vspace{-1cm}
\end{minipage}
\vfill
\subfloat[Cheetah (Average Actions)]{\includegraphics[width=0.5\linewidth]{img/Cheetah_actions.pdf}\label{fig:cheetah_actions}}%
\hfill
\subfloat[Ant (Average Actions)]{\includegraphics[width=0.5\linewidth]{img/Ant_actions.pdf}\label{fig:ant_actions}}%
\caption{Average Actions Multiagent Cheetah and Ant}
\vspace{-5mm}
\end{figure}
In this section, we present empirical results of our model-based multiagent RL framework on continuous tasks.
\subsection{Experiment Setups}
\subsubsection{Experiment Settings}\label{exp:mujoco}
For the experiments we use OpenAI Gym MuJoCo~\cite{brockman2016openai, todorov2012mujoco} locomotion tasks, which are the standard benchmarks for continuous control with RL~\cite{wang2018nervenet, nagabandi2018neural}. %To adapt the tasks for decentralised control, an environment wrapper is implemented to extract the local observations for each agent, following the convention used by~\cite{wang2018nervenet}.
Two exemplar robot locomotion tasks are selected as our case studies: (1) the Cheetah movement in 2D space and (2) the Ant movement in 3D space. The cheetah model in MuJoCo has 7 joints, including one root joint and 6 joints each paired with one actuator. Following the definition in Section~\ref{subsec:robot_multiagent}, we consider each joint as an agent with an actuator that applies a joint torque. For all agents, the action space is normalised to $[-1, 1]$. The root agent observes the position, angle, velocity and angular velocity and for other agents, their observation includes the relative angle with respect to the body that they are attached to, and the angular velocity. The global observation utilised by the critic and the dynamics model (c.f. Figure~\ref{fig:mb}) is given by a concatenation of local observations. The objective of the agent is to move forward in the $x$-direction. On the other hand, the Ant model can move in a 3D space and the agents are similarly defined as in the Cheetah. Notably, the agents of an Ant model in MuJoCo can additionally observe external forces such as friction. The objective of the Ant locomotion task is also to move forward in the $x$-direction.
\subsubsection{Models and Training Details}
Our multiagent system consists of a model $f = (f_s, f_r)$ of the environment, the centralised critic, and the decentralised actors.
We use PyTorch~\cite{paszke2019pytorch} for implementing and optimising the neural network models. For the dynamics model $f_s$, we use 4 fully-connected layers with ReLU activation function~\cite{nwankpa2018activation} and a linear output layer with the dimension of $128$. For the reward model $f_r$, we use 3 fully-connected layers with ReLU activation function and a linear output layer with the dimension of $128$.
The model is optimised in minibatches of size $64$ using Adam optimiser~\cite{kingma2014adam} with the learning rate of $10^{-3}$. For the centralised critic, we use $3$ fully-connected layers of the dimension of $32$ with Tanh~\cite{nwankpa2018activation} activation and a linear output layer which produces the value of the input state. For the decentralised actors, each actor uses 3 fully-connected layers of the dimension of $32$ with Tanh activation, and an output layer with Tanh activation. The default actions $a_\textnormal{default}$ are given by zero vectors. Both the critic and the actor are optimised using the Adam optimiser. The learning rate for the critic is $10^{-3}$ and is $3\times 10^{-4}$ for the actor. All graphs plot mean and standard deviation across 5 seeds.
\subsection{Overall Results}
Figure~\ref{fig:all_results} shows the overall performance of our algorithms and baseline algorithms on the MuJoCo locomotion tasks with cheetah and ant robots. The $y$-axis shows the episode rewards and $x$-axis shows the train steps.
In particular, we follow Eq.~\ref{eq:semivalue} to implement the following variants of our semivalue based credit assignment algorithms. \texttt{MB} is for Model-based coalition value estimation:
\begin{itemize}
\item{\mbshapley: the Shapley value ($p_c=\frac{1}{|N|}$);} %for credit assignment and model-based coalition value estimation;}
\item{\mbbanzhaf: the Banzhaf value ($p_c=\frac{1}{2^{|N|-1}}\tbinom{|N|-1}{c}$);}% for credit assignment and model-based coalition value estimation;}
\item{\mbloo: the Leave-one-out value ($p_c=\mathbbm{1}_{c=|N|-1}$), where we compute the marginal contribution of an agent towards the grand coalition (made of all other agents);}% for credit assignment and model-based coalition value estimation;}
\end{itemize}
For \emph{baselines}, we compare with the following state-of-the-art algorithms in multiagent RL that use the Shapley value-based credit assignment and the global advantage respectively:
\begin{itemize}
\item{\qshapley~\cite{li2021shapley}: we adapt the discrete domain algorithm~\cite{li2021shapley} which used Shapley value for credit assignment, to our multiagent PPO which can be used for continuous control. The coalition values are computed using a centralised critic of Q-values in a model-free manner.}
\item{\mappo (Multiagent PPO)~\cite{yu2021surprising}: In \mappo, the centralised critic computes a \emph{shared advantage function (GAE)} for all agents based on the global reward signal. And the agents are trained using centralised training, decentralised execution.}
\end{itemize}
\subsubsection{Shared advantage vs. agent-specific advantage. }\label{exp:central_vs_multi}
We are now in a position to present the results and findings.
In both the cheetah (Figure~\ref{fig:cheetah}) and ant (Figure~\ref{fig:ant}) cases, we observe that our credit assignment methods that use model-based estimation (i.e., \mbshapley, \mbbanzhaf, \mbbanzhaf) significantly outperform multiagent PPO (\mappo) in terms of both the \emph{average rewards} and \emph{sample efficiency}. For example, in the Ant case, our model-based algorithm with Banzhaf credit assignment (\mbbanzhaf) quickly reaches the reward of 1000 within 50 train steps, while \mappo needs around 150 train steps to reach the same level. \mbbanzhaf also converges to a higher average reward ($\sim1400$) than \mappo ($\sim1100$).
In all our results, performing credit assignment and evaluating the per-agent contribution (c.f. Sec.~\ref{sub:gt}) is consistently better than agents using a shared advantage function. A demo video is provided in the supplementary.
\subsubsection{Model-based vs. Model-free credit assignment.}
We next compare our \mbshapley with the baseline method \qshapley~\cite{li2021shapley} in order to understand the role of model-based credit assignment. In our \mbshapley, the coalition values are estimated using the dynamics/reward model and the state-value critic $V^\pi$. While in \qshapley, the coalition values are estimated in a model-free way with action-value critic $Q^\pi$. As shown in
Figure~\ref{fig:cheetah} and~\ref{fig:ant}, for both locomotion tasks, \mbshapley outperforms \qshapley.
This shows that when only the reward of the grand coalition (i.e., the whole robot) is provided by the simulations, it is difficult to infer the values $\cv^\coalition$ of different coalitions using a model-free Q-value critic. In contrast, a better estimation of values of different coalitions is obtained with the predictions of our model-based module, which enables more effective agent-specific credit assignment and sample-efficient learning.
\subsubsection{Robustness of Different Semivalue Variants.}
Our model-based coalition value estimation supports the broad class of semivalue credit assignments. Still, from Figure~\ref{fig:all_results}, we observe that the agents trained using different selected semivalues (\mbshapley, \mbbanzhaf, \mbloo) all deliver decent performance. In the case of cheetah (Figure~\ref{fig:cheetah}), both \mbshapley and \mbbanzhaf yield the highest rewards, and in the case of ant (Figure~\ref{fig:ant}), \mbbanzhaf yields the highest rewards. Overall, \mbbanzhaf has the lowest variance among the three variants.
\subsubsection{Average Actions}
Figures~\ref{fig:cheetah_actions} and~\ref{fig:ant_actions} show the mean absolute value of actions, averaged across the agents (joints actuators). This quantity refers to the controller output (normalised between $[0,1]$), and higher action values mean higher energy consumption in a practical context. In cheetah, \mbloo has the highest average action, with \mbshapley and \mbbanzhaf having medium values. In the ant case, \mbshapley, \mbbanzhaf, and \mbloo all have low control output. We observe that the ant robot trained using \mbshapley, \mbbanzhaf learn to move forward using a subset of joints. In contrast, other joints are mainly learned for steering, implying that using these two semivalues can also encourage diverse roles of different agents in a locomotion task.
\subsection{Component-wise Analysis}
\begin{figure}[t]
\centering
\subfloat[Cheetah]{\includegraphics[width=0.5\linewidth]{img/Cheetah_size_variations.pdf}\label{fig:cheetah_size}}%
\hfill
\subfloat[Ant]{\includegraphics[width=0.5\linewidth]{img/Ant_size_variations.pdf}\label{fig:ant_size}}%
\caption{Results for Varying Coalition Sizes}
\end{figure}
\subsubsection{Variation in Coalition Sizes}\label{subseq:coalition}
In this section, we discuss the model-based semivalues from the perspective of coalition sizes. Figures~\ref{fig:cheetah_size} and ~\ref{fig:ant_size} show the performance of agents learned using two semivalues \mbshapley ($p_c =\frac{1}{N}$), \mbbanzhaf ($p_c =\frac{1}{2^N}\tbinom{N-1}{c}$), and some semivalues defined by $p_c =\mathbbm{1}_{c=x}$, e.g., a semivalue with $p_c =\mathbbm{1}_{c=3}$ is an agent's average marginal contributions towards size-$3$ coalitions. In the case of cheetah (Figure~\ref{fig:cheetah_size}),
the semivalues that use small coalition sizes (e.g., $\mathbbm{1}_{c=0}$) yield unsatisfactory performance, while those using mid-sized coalitions (e.g., $\mathbbm{1}_{c=0}$) yield a high reward. In the case of the ant (Figure~\ref{fig:ant_size}), the semivalues focused on
coalitions of heterogeneous sizes all yield high rewards. This suggests that the agents' actions in the cheetah case are more dependent; hence, they require agents to closely coordinate their actions. While for the ant, the agents have more distinct roles and their actions are more independent. The advantage of Shapley value and Banzhaf value is that they take into account all coalitions sizes and therefore are robust across different robots. Shapley value uniformly weighs all coalition sizes while Banzhaf value puts higher weights on mid-sized coalitions.
\begin{figure}[t]
\centering
\subfloat[Cheetah]{\includegraphics[width=0.5\linewidth]{img/Cheetah_rewards_sample_sizes.pdf}\label{fig:cheetah_sample_size}}%
\hfill
\subfloat[Ant]{\includegraphics[width=0.5\linewidth]{img/Ant_rewards_sample_sizes.pdf}\label{fig:ant_sample_size}}%
\caption{Results for Varying Sample Sizes}
\end{figure}
\subsubsection{Variation in Sample Sizes} Lastly, we examine how the number of coalitions sampled per agent per step (when estimating the semivalues) affects the robot's performance. We compare \mbshapley for the number of samples of $s=1,3,$ and $5$. We do the same for \mbbanzhaf.
Figures~\ref{fig:cheetah_sample_size} and~\ref{fig:ant_sample_size} show that for both robots and both semivalue variants, reducing the number of sampled coalitions per node does not drastically decrease the performance in terms of reward and sample efficiency.
Even using a small number of sampled coalitions, such as one per agent in each step, can still yield sufficiently good performance. We conjecture that this robustness property arises because, despite the small number of sampled coalitions per step, an RL algorithm typically runs on the scale of million timesteps. Hence, the number of sampled coalitions throughout the entire learning phase often will be sufficiently large. Importantly, because RL requires many credit assignment computations where each involves neural network forward propagation, this property helps us significantly reduce computation cost without compromising the learning performance.
% Locomotion + continuous control
% multiagent related works
% COMA, MADDPG, SQDDPG, HUAWEI, MAPPO
\noindent \textbf{Multiagent RL.} The simplest form of multiagent RL is Independent Q-learning (IQL)~\cite{tan1993multi}, where a group of agents each learns on their own and treats other agents as part of the environment. While IQL delivers decent performance, the agents frequently face the challenge that the environment appears unstationary, as other agents simultaneously learn and update their policies. To address this issue, centralised training, decentralised execution framework was first proposed separately by~\cite{foerster2018counterfactual} and ~\cite{lowe2017multi}. This training paradigm allows the agents to access the global information during training and has been a standard approach to recent multiagent RL algorithms since then. \cite{foerster2018counterfactual} proposed an actor-critic algorithm that addresses multiagent credit assignment explicitly using the counterfactual value of each agent and is applied to domains with discrete action space such as Starcraft. \cite{lowe2017multi} proposed a multiagent policy gradient algorithm where each agent receives a separate reward and uses their critic. This method can apply to competitive settings but does not address the multiagent credit assignment problem. Another line of work using centralised training, decentralised execution, and Q-learning are the implicit credit assignment methods such as~\cite{rashid2018qmix}~\cite{rashid2020weighted}~\cite{wang2020qplex}. More recently, \cite{yu2021surprising} showed the effectiveness of PPO in multiagent problems for discrete action space domains. Their model is optimised using a centralised reward and demonstrated to outperform baseline methods such as \cite{lowe2017multi}\cite{rashid2018qmix}.
\noindent \textbf{Game-theoretic Credit Assignment for Multiagent RL.} Several works have considered the Shapley value for credit assignment in multiagent RL with discrete action spaces~\cite{li2021shapley}\cite{wang2020shapley}\cite{wang2021shaq}\cite{xu2020learning}. Most notably among these works, \cite{li2021shapley} has shown state-of-the-art performance on Starcraft. In our experiments, we adapt their credit assignment framework from discrete domains to the continuous multiagent PPO. Unlike most prior credit assignment methods, which only consider one of the common values (e.g., Shapley value), we introduce semivalues to credit assignment in multiagent RL and define a more generic game-theoretic framework that encompasses a family of common solution concepts and analyzes the relation between them. Moreover,
we use the new framework to address the problem of multiagent continuous control, and show that our model-based RL module can better estimate the coalition values for credit assignment and improve the sample efficiency accordingly.
\vspace{1ex}\noindent\textbf{Model-based RL.}
Model-based RL approaches typically alternate between fitting a predictive model of the environment dynamics/rewards and updating the control policies. The model can be used in various ways, such as execution-time planning~\cite{chua2018deep, nagabandi2018neural}, generating imaginary experiences for training the control policy~\cite{sutton1991dyna, janner2019trust}), etc. Our work is inspired by~\cite{feinberg2018model}, which addresses the problem of error in long-horizon model dynamics prediction. \cite{feinberg2018model} presents a hybrid algorithm that uses the model to simulate the short-term horizon and Q-learning to estimate the long-term value beyond the simulation horizon.
Unlike the previous model-based RL works, our work is the first to introduce model-based RL for enabling game-theoretic credit assignment in multiagent continuous control.
In this paper, we studied multiagent robotic continuous control with model-based game-theoretic credit assignments. We first modelled robot joints as a fully-cooperative multiagent system, and optimised the system using a multiagent version of PPO. We then proposed a generic game-theoretic credit assignment framework using semivalues for evaluating agent-specific advantage functions. Furthermore, we proposed a model-based framework which significantly improved estimation of coalition values, which empowers game-theoretic credit assignments for multiagent continuous control. Finally, we empirically demonstrate that our model-based credit assignments leads to sample-efficient and robust multiagent learning on MuJoCo robot locomotion tasks.
In the future, we would like to study other continuous control algorithms and other robot variants, including real robots. Furthermore, it would interesting to study the model-based RL methods with other configurations such as longer prediction horizons.
\begin{thebibliography}{10}
\bibitem{andrychowicz2017hindsight}
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong,
Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba.
\newblock Hindsight experience replay.
\newblock {\em arXiv preprint arXiv:1707.01495}, 2017.
\bibitem{banzhaf1964weighted}
John~F Banzhaf~III.
\newblock Weighted voting doesn't work: A mathematical analysis.
\newblock {\em Rutgers L. Rev.}, 19:317, 1964.
\bibitem{brockman2016openai}
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman,
Jie Tang, and Wojciech Zaremba.
\newblock Openai gym.
\newblock {\em arXiv preprint arXiv:1606.01540}, 2016.
\bibitem{chalkiadakis2011computational}
Georgios Chalkiadakis, Edith Elkind, and Michael Wooldridge.
\newblock {\em Computational Aspects of Cooperative Game Theory}.
\newblock Morgan \& Claypool Publishers, 2011.
\bibitem{chua2018deep}
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.
\newblock Deep reinforcement learning in a handful of trials using
probabilistic dynamics models.
\newblock {\em arXiv preprint arXiv:1805.12114}, 2018.
\bibitem{dubey1981value}
Pradeep Dubey, Abraham Neyman, and Robert~James Weber.
\newblock Value theory without efficiency.
\newblock {\em Mathematics of Operations Research}, 6(1):122--128, 1981.
\bibitem{feinberg2018model}
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael~I Jordan, Joseph~E Gonzalez,
and Sergey Levine.
\newblock Model-based value estimation for efficient model-free reinforcement
learning.
\newblock {\em arXiv preprint arXiv:1803.00101}, 2018.
\bibitem{finn2016guided}
Chelsea Finn, Sergey Levine, and Pieter Abbeel.
\newblock Guided cost learning: Deep inverse optimal control via policy
optimization.
\newblock In {\em International conference on machine learning}, pages 49--58.
PMLR, 2016.
\bibitem{foerster2018counterfactual}
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and
Shimon Whiteson.
\newblock Counterfactual multi-agent policy gradients.
\newblock In {\em Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~32, 2018.
\bibitem{gu2017deep}
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine.
\newblock Deep reinforcement learning for robotic manipulation with
asynchronous off-policy updates.
\newblock In {\em 2017 IEEE international conference on robotics and automation
(ICRA)}, pages 3389--3396. IEEE, 2017.
\bibitem{haarnoja2018soft}
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
\newblock Soft actor-critic: Off-policy maximum entropy deep reinforcement
learning with a stochastic actor.
\newblock In {\em International conference on machine learning}, pages
1861--1870. PMLR, 2018.
\bibitem{janner2019trust}
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine.
\newblock When to trust your model: Model-based policy optimization.
\newblock {\em arXiv preprint arXiv:1906.08253}, 2019.
\bibitem{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}, 2014.
\bibitem{kober2013reinforcement}
Jens Kober, J~Andrew Bagnell, and Jan Peters.
\newblock Reinforcement learning in robotics: A survey.
\newblock {\em The International Journal of Robotics Research},
32(11):1238--1274, 2013.
\bibitem{levine2016end}
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.
\newblock End-to-end training of deep visuomotor policies.
\newblock {\em The Journal of Machine Learning Research}, 17(1):1334--1373,
2016.
\bibitem{li2021shapley}
Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun
Xiao.
\newblock Shapley counterfactual credits for multi-agent reinforcement
learning.
\newblock {\em arXiv preprint arXiv:2106.00285}, 2021.
\bibitem{lillicrap2015continuous}
Timothy~P Lillicrap, Jonathan~J Hunt, Alexander Pritzel, Nicolas Heess, Tom
Erez, Yuval Tassa, David Silver, and Daan Wierstra.
\newblock Continuous control with deep reinforcement learning.
\newblock {\em arXiv preprint arXiv:1509.02971}, 2015.
\bibitem{littman1994markov}
Michael~L Littman.
\newblock Markov games as a framework for multi-agent reinforcement learning.
\newblock In {\em Machine learning proceedings 1994}, pages 157--163. Elsevier,
1994.
\bibitem{lowe2017multi}
Ryan Lowe, Yi~Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch.
\newblock Multi-agent actor-critic for mixed cooperative-competitive
environments.
\newblock {\em arXiv preprint arXiv:1706.02275}, 2017.
\bibitem{mnih2015human}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei~A Rusu, Joel Veness,
Marc~G Bellemare, Alex Graves, Martin Riedmiller, Andreas~K Fidjeland, Georg
Ostrovski, et~al.
\newblock Human-level control through deep reinforcement learning.
\newblock {\em nature}, 518(7540):529--533, 2015.
\bibitem{nagabandi2018neural}
Anusha Nagabandi, Gregory Kahn, Ronald~S Fearing, and Sergey Levine.
\newblock Neural network dynamics for model-based deep reinforcement learning
with model-free fine-tuning.
\newblock In {\em 2018 IEEE International Conference on Robotics and Automation
(ICRA)}, pages 7559--7566. IEEE, 2018.
\bibitem{nwankpa2018activation}
Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall.
\newblock Activation functions: Comparison of trends in practice and research
for deep learning.
\newblock {\em arXiv preprint arXiv:1811.03378}, 2018.
\bibitem{paszke2019pytorch}
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et~al.
\newblock Pytorch: An imperative style, high-performance deep learning library.
\newblock {\em Advances in neural information processing systems},
32:8026--8037, 2019.
\bibitem{rashid2020weighted}
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson.
\newblock Weighted qmix: Expanding monotonic value function factorisation for
deep multi-agent reinforcement learning.
\newblock {\em arXiv preprint arXiv:2006.10800}, 2020.
\bibitem{rashid2018qmix}
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob
Foerster, and Shimon Whiteson.
\newblock Qmix: Monotonic value function factorisation for deep multi-agent
reinforcement learning.
\newblock In {\em International Conference on Machine Learning}, pages
4295--4304. PMLR, 2018.
\bibitem{samvelyan2019starcraft}
Mikayel Samvelyan, Tabish Rashid, Christian~Schroeder De~Witt, Gregory
Farquhar, Nantas Nardelli, Tim~GJ Rudner, Chia-Man Hung, Philip~HS Torr,
Jakob Foerster, and Shimon Whiteson.
\newblock The starcraft multi-agent challenge.
\newblock {\em arXiv preprint arXiv:1902.04043}, 2019.
\bibitem{schulman2015high}
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter
Abbeel.
\newblock High-dimensional continuous control using generalized advantage
estimation.
\newblock {\em arXiv preprint arXiv:1506.02438}, 2015.
\bibitem{schulman2017proximal}
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
\newblock Proximal policy optimization algorithms.
\newblock {\em arXiv preprint arXiv:1707.06347}, 2017.
\bibitem{Shapley+2016+307+318}
L.~S. Shapley.
\newblock {\em 17. A Value for n-Person Games}.
\newblock Princeton University Press, 2016.
\bibitem{silver2017mastering}
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja
Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton,
et~al.
\newblock Mastering the game of go without human knowledge.
\newblock {\em nature}, 550(7676):354--359, 2017.
\bibitem{spong2006robot}
Mark~W Spong, Seth Hutchinson, Mathukumalli Vidyasagar, et~al.
\newblock {\em Robot modeling and control}, volume~3.
\newblock wiley New York, 2006.
\bibitem{sutton1991dyna}
Richard~S Sutton.
\newblock Dyna, an integrated architecture for learning, planning, and
reacting.
\newblock {\em ACM Sigart Bulletin}, 2(4):160--163, 1991.
\bibitem{sutton2018reinforcement}
Richard~S Sutton and Andrew~G Barto.
\newblock {\em Reinforcement learning: An introduction}.
\newblock MIT press, 2018.
\bibitem{tan1993multi}
Ming Tan.
\newblock Multi-agent reinforcement learning: Independent vs. cooperative
agents.
\newblock In {\em Proceedings of the tenth international conference on machine
learning}, pages 330--337, 1993.
\bibitem{todorov2012mujoco}
Emanuel Todorov, Tom Erez, and Yuval Tassa.
\newblock Mujoco: A physics engine for model-based control.
\newblock In {\em 2012 IEEE/RSJ International Conference on Intelligent Robots
and Systems}, pages 5026--5033. IEEE, 2012.
\bibitem{wang2020qplex}
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang.
\newblock Qplex: Duplex dueling multi-agent q-learning.
\newblock {\em arXiv preprint arXiv:2008.01062}, 2020.
\bibitem{wang2021shaq}
Jianhong Wang, Jinxin Wang, Yuan Zhang, Yunjie Gu, and Tae-Kyun Kim.
\newblock Shaq: Incorporating shapley value theory into q-learning for
multi-agent reinforcement learning.
\newblock {\em arXiv preprint arXiv:2105.15013}, 2021.
\bibitem{wang2020shapley}
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu.
\newblock Shapley q-value: a local reward approach to solve global reward
games.
\newblock In {\em Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~34, pages 7285--7292, 2020.
\bibitem{wang2018nervenet}
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler.
\newblock Nervenet: Learning structured policy with graph neural networks.
\newblock In {\em International Conference on Learning Representations}, 2018.
\bibitem{wolpert2002optimal}
David~H Wolpert and Kagan Tumer.
\newblock Optimal payoff functions for members of collectives.
\newblock In {\em Modeling complexity in economic and social systems}, pages
355--369. World Scientific, 2002.
\bibitem{wooldridge2009introduction}
Michael Wooldridge.
\newblock {\em An introduction to multiagent systems}.
\newblock John wiley \& sons, 2009.
\bibitem{xu2020learning}
Jing Xu, Fangwei Zhong, and Yizhou Wang.
\newblock Learning multi-agent coordination for enhancing target coverage in
directional sensor networks.
\newblock {\em arXiv preprint arXiv:2010.13110}, 2020.
\bibitem{yu2021surprising}
Chao Yu, Akash Velu, Eugene Vinitsky, Yu~Wang, Alexandre Bayen, and Yi~Wu.
\newblock The surprising effectiveness of mappo in cooperative, multi-agent
games.
\newblock {\em arXiv preprint arXiv:2103.01955}, 2021.
\end{thebibliography}