Soft-Bellman Equilibrium in Affine Markov Games: Forward Solutions and Inverse Learning
Extracted source text · arXiv 2304.00163
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
% 1 2 3 4 5 6 7 8
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4 paper
\IEEEoverridecommandlockouts % This command is only needed if
% you want to use the \thanks command
\overrideIEEEmargins % Needed to meet printer requirements.
\usepackage[natbib=true,backend=biber,style=numeric-comp,sorting=none,maxbibnames=99]{biblatex}
\let\proof\relax
\let\endproof\relax
\usepackage{graphicx,algorithm,algpseudocode, amsthm,amsmath,amsfonts,verbatim,mathtools,enumerate,bm,hyperref,dsfont,xcolor, caption, subcaption, rotating, tikz, pgfplots}
\usetikzlibrary{shapes, positioning}
\usepackage[export]{adjustbox}
\definecolor{preyred}{rgb}{0.8392156862745098, 0.15294117647058825, 0.1568627450980392}
\definecolor{predatorblue}{rgb}{0.12156862745098039, 0.4666666666666667, 0.7058823529411765}
\setcounter{MaxMatrixCols}{20}
\usepackage[capitalize]{cleveref}
\pdfminorversion=4
\allowdisplaybreaks
\input defs.tex
\bibliography{reference}
\renewcommand*{\bibfont}{\small}
\setlength{\belowcaptionskip}{-0.5cm}
\DeclareCaptionFormat{myformat}{\small\itshape #1#2#3}
\captionsetup[figure]{format=myformat}
\title{\LARGE \bf Soft-Bellman Equilibrium in Affine Markov Games:\\ Forward Solutions and Inverse Learning
}
\author{Shenghui~Chen, Yue~Yu, David~Fridovich-Keil, and Ufuk~Topcu% <-this % stops a space
\thanks{S. Chen, Y. Yu, D. Fridovich-Keil, and U. Topcu are with the Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, TX, 78712, USA (emails: shenghui.chen@utexas.edu,\, yueyu@utexas.edu,\, dfk@utexas.edu,\, utopcu@utexas.edu).}%
}
\begin{document}
\maketitle
\thispagestyle{empty}
\pagestyle{empty}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
Markov games model interactions among multiple players in a stochastic, dynamic environment. Each player in a Markov game maximizes its expected total discounted reward, which depends upon the policies of the other players. We formulate a class of Markov games, termed \emph{affine Markov games}, where an affine reward function couples the players' actions. We introduce a novel solution concept, the \emph{soft-Bellman equilibrium}, where each player is boundedly rational and chooses a soft-Bellman policy rather than a purely rational policy as in the well-known Nash equilibrium concept. We provide conditions for the existence and uniqueness of the soft-Bellman equilibrium and propose a nonlinear least-squares algorithm to compute such an equilibrium in the \emph{forward problem}. We then solve the \emph{inverse game problem} of inferring the players’ reward parameters from observed state-action trajectories via a projected-gradient algorithm. Experiments in a predator-prey OpenAI Gym environment show that the reward parameters inferred by the proposed algorithm outperform those inferred by a baseline algorithm: they reduce the Kullback-Leibler divergence between the equilibrium policies and observed policies by at least two orders of magnitude.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{1_introduction/introduction}
\input{2_related_work/related_work}
\input{3_model/mdpgames}
\input{3_model/forward}
\input{3_model/inverse}
\input{4_experiments/experiments}
\input{5_conclusion/conclusion}
%\addtolength{\textheight}{-12cm} % This command serves to balance the column lengths
% on the last page of the document manually. It shortens
% the textheight of the last page by a suitable amount.
% This command does not take effect until the next page
% so it should come on the page before the last. Make
% sure that you do not shorten the textheight too much.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section*{APPENDIX}
\section*{ACKNOWLEDGMENT} The authors would like to thank Negar Mehr and Xiao Xiang for their constructive feedback.
% \bibliographystyle{IEEEtran}
% \bibliography{IEEEabrv,reference}
\printbibliography
\end{document}
\newcommand*\circled[4]{\tikz[baseline=(char.base)]{
\node[shape=circle, fill=#2, draw=#3, text=#4, inner sep=2pt] (char) {#1};}}
\newcommand*\octagoned[3]{\tikz[baseline=(char.base)]{
\node[regular polygon, regular polygon sides=8, fill=#2, text=#3, inner sep=2pt] (char) {#1};}}
% \node[regular polygon, regular polygon sides=8, draw, minimum size=3cm] at (0,0) {};
\newcommand{\ones}{\mathbf 1}
\newcommand{\reals}{{\mbox{\bf R}}}
\newcommand{\integers}{{\mbox{\bf Z}}}
\newcommand{\symm}{{\mbox{\bf S}}} % symmetric matrices
\newcommand{\nullspace}{{\mathcal N}}
\newcommand{\range}{{\mathcal R}}
\newcommand{\Rank}{\mathop{\bf Rank}}
\newcommand{\tr}{\mathop{\rm tr}}
\newcommand{\diag}{\mathop{\rm diag}}
\newcommand{\blkdiag}{\mathop{\rm blkdiag}}
\newcommand{\proj}{\mathop{\rm Proj}}
\newcommand{\card}{\mathop{\bf card}}
\newcommand{\rank}{\mathop{\bf rank}}
\newcommand{\rem}{\mathop{\rm rem}}
\newcommand{\quot}{\mathop{\rm quo}}
\newcommand{\vect}{\mathop{\rm vec}}
\newcommand{\conv}{\mathop{\bf conv}}
\newcommand{\prox}{\mathbf{prox}}
\newcommand{\Expect}{\mathop{\bf E{}}}
\newcommand{\Prob}{\mathop{\bf Prob}}
\newcommand{\Co}{{\mathop {\bf Co}}} % convex hull
\newcommand{\dist}{\mathop{\bf dist{}}}
\newcommand{\argmin}{\mathop{\rm argmin}}
\newcommand{\argmax}{\mathop{\rm argmax}}
\newcommand{\epi}{\mathop{\bf epi}} % epigraph
\newcommand{\Vol}{\mathop{\bf vol}}
\newcommand{\dom}{\mathop{\bf dom}} % domain
\newcommand{\intr}{\mathop{\bf int}}
\newcommand{\sign}{\mathop{\bf sign}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\mnorm}[1]{{\left\vert\kern-0.25ex\left\vert\kern-0.25ex\left\vert #1
\right\vert\kern-0.25ex\right\vert\kern-0.25ex\right\vert}}
\theoremstyle{definition}
\newtheorem{definition}{Definition}
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{corollary}{Corollary}
\newtheorem{remark}{Remark}
\newtheorem{proposition}{Proposition}
\newtheorem{assumption}{Assumption}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}
\newcommand{\cf}{{\it cf.}}
\newcommand{\eg}{{\it e.g.}}
\newcommand{\ie}{{\it i.e.}}
\newcommand{\etc}{{\it etc.}}
\newcommand{\ba}[2][]{\todo[color=orange!40,size=\footnotesize,#1]{[BA] #2}}
\newcommand{\david}[1]{\textsf{\textcolor{orange}{[David: #1]}}}
\newcommand{\yueyu}[1]{\textsf{\textcolor{violet}{[Yue: #1]}}}
\newcommand{\revised}[1]{{\textcolor{blue}{[#1]}}}
\newcommand{\vivian}[1]{\textsf{\textcolor{magenta}{[Vivian: #1]}}}
\usepackage{soul}
\section{Introduction}
\label{sec: introduction}
Markov games model the interaction of multiple decision makers in stochastic and dynamic environments \cite{shapley1953stochastic}. In a Markov game, each player's transition and reward depend on the policies of the other players, and each player aims to find an optimal policy that maximizes its expected discounted total reward.
% ----
% The Nash equilibrium is a fundamental concept for analyzing strategic interactions among multiple decision-makers \cite{nash1996non}. In a Markov game, it refers to a collection of policies where no player can benefit by unilaterally changing its policy \cite{shapley1953stochastic}. The Nash equilibrium concept assumes that the players are perfectly rational, seeking to maximize their rewards without perceptual errors or cognitive biases.
% However, not all players are perfectly rational. Humans, for example, have limited cognitive capacity and are subject to biases and heuristics that can affect their decision-making, bounding their rationality. As a result, the outcomes of games played by humans may not always align with the predictions from the Nash equilibrium concept.
% Recent efforts attempt to address this limitation of the Nash equilibrium by accounting for players' bounded rationality in games with specific structures, including matrix games \cite{waugh2013computational}, fully cooperative games \cite{barrett2017making, le2017coordinated, vsovsic2016inverse}, and two-player games \cite{lin2017multiagent, lin2019multi}. Another recent work tackles the same limitation in dynamic games with continuous state and action spaces \cite{mehr2023maximum}. However, to our best knowledge, no work has addressed this limitation in \emph{general-sum, multi-player} Markov games with \emph{discrete} state and action spaces yet.
The concept of Nash equilibrium, which refers to a set of policies where no player can benefit by unilaterally changing their policy \cite{shapley1953stochastic}, overlooks the reality that players are often boundedly rational. For example, humans have limited cognitive capacity and are subject to biases and heuristics that can affect their decision-making. As a result, the outcomes of games played by humans may not always align with the predictions from the Nash equilibrium concept. Recent efforts attempt to address this limitation by accounting for players' bounded rationality in games with specific structures, including matrix games \cite{waugh2013computational}, fully cooperative games \cite{barrett2017making, le2017coordinated, vsovsic2016inverse}, and two-player games \cite{lin2017multiagent, lin2019multi}. Another recent work tackles the same limitation in dynamic games with continuous state and action spaces \cite{mehr2023maximum}. However, to our best knowledge, no work has addressed this limitation in \emph{general-sum, multi-player} Markov games with \emph{discrete} state and action spaces yet.
% ----
We propose the \emph{soft-Bellman equilibrium} as a new solution concept to capture the dynamics of boundedly rational players in \emph{affine Markov games}.
Affine Markov games are a class of Markov games where each player has independent dynamics and an affine reward function couples the players’ actions.
In a soft-Bellman equilibrium, each player chooses a policy that maximizes the expected reward with causal entropy regularization while satisfying independent transition dynamics.
We provide conditions for the existence and uniqueness of the soft-Bellman equilibrium.
% ----
% forward; inverse
We study the \emph{forward problem} of computing a soft-Bellman equilibrium in a given affine Markov game.
We propose a least-squares-based algorithm to solve this problem by minimizing the residuals of the soft-Bellman equilibrium conditions.
% We propose a least-squares-based algorithm to solve a causal-entropy regularized equilibrium problem by minimizing the residuals of equilibrium conditions.
We then turn to the \emph{inverse game problem} of inferring the players’ reward parameters that best explain observed interactions. We propose an iterative algorithm that leverages the solutions to the forward problem. In each iteration, the algorithm computes the soft-Bellman equilibrium given the current reward parameters and then updates those parameters with a projected-gradient method based on the implicit function theorem \cite{dontchev2014implicit}.
The proposed inverse game algorithm outperforms a baseline algorithm that ignores the coupling between players.
Experiments in a predator-prey OpenAI Gym environment \cite{magym} show that the reward parameters inferred by the proposed algorithm reduce the Kullback-Leibler divergence between the equilibrium policies and observed policies by at least two orders of magnitude than the baseline algorithm.
% Using a synthetic dataset in a predator-prey OpenAI Gym environment \cite{magym}, we compare the proposed inverse game algorithm with a baseline algorithm that ignores the coupling between players.
% Results show that the proposed algorithm terminates with a Kullback-Leibler divergence between the equilibrium policies and observed policies at least two orders of magnitude lower than that of the baseline algorithm.
\section{Related Work}
\label{sec: related}
In single-agent settings, literature in inverse reinforcement learning studies the problem of inferring reward parameters from human experts' trajectories. The principle of maximum entropy is a popular approach in this direction \cite{ziebart2008maximum}. Subsequent studies further extend this principle to accommodate stochastic transitions using causal entropy \cite{ziebart2013principle}. For example, recent work extends the maximum causal entropy framework in inverse reinforcement learning to an infinite time horizon setting and proposes the concept of stationary soft-Bellman policy \cite{zhou2017infinite}. This policy concept inspires the formulation of the soft-Bellman equilibrium to account for the players' bounded rationality, a feature lacking in the Nash equilibrium concept.
In multi-agent settings, most existing works that try to address the limitation of Nash equilibrium assume specific game structures, including matrix games \cite{waugh2013computational}, fully cooperative games \cite{barrett2017making, le2017coordinated, vsovsic2016inverse}, two-player zero-sum games \cite{lin2017multiagent}, and two-player general-sum games \cite{lin2019multi}.
This paper generalizes the existing works to \emph{multi-player, general-sum} Markov games.
First formulated in normal-form and extensive-form games, the quantal response equilibrium is a solution concept to model the bounded rationality of human players \cite{mckelvey1995quantal, mckelvey1998quantal}.
Inspired by this solution concept, recent work proposes the entropic cost equilibrium to extend the quantal response equilibrium to games with \emph{continuous} states and actions \cite{mehr2023maximum}.
The current work, on the other hand, proposes the soft-Bellman equilibrium to support stochastic transitions in affine Markov games with \emph{discrete} state and action spaces.
Although both the entropic cost equilibrium and the soft-Bellman equilibrium extend the quantal response equilibrium, the soft-Bellman equilibrium is different in choosing the state-action frequency matrix, instead of the policy, as the variable to optimize for each player. This subtle difference changes the expected reward from a nonconvex function to a convex one, laying the groundwork for establishing conditions that ensure the existence and uniqueness of solutions.
% Some other works that attempt to solve the inverse game problem have different assumptions than ours. Using machine learning, \citet{song2018multi} extend the generative adversarial imitation learning to multi-agent settings based on the notion of Nash equilibrium. Also based on adversarial training, \citet{yu2019multi} propose a multi-agent IRL method for high-dimensional state and action spaces with \emph{unknown} dynamics.
% \section{Affine Markov games and soft-Bellman equilibrium}
% We introduce a class of Markov games, termed \emph{affine Markov games}, along with the notion of soft-Bellman equilibrium.
\section{Models}
We present our main theoretical models: a special class of Markov games, along with a novel equilibrium concept that accounts for bounded rationality.
\subsection{Affine Markov Games}
We consider a Markov game \cite{shapley1953stochastic} where each player solves an MDP with independent dynamics and an affine reward function that couples the players' actions. We let \(p\in\mathbb{N}\) denote the number of players. Player \(i\in[p]\) solves an MDP specified by a tuple which includes a set of states, a set of actions, a transition kernel, an initial state distribution, a reward matrix, and a discount factor. We let \(n^i\in\mathbb{N}\) and \(m^i\in\mathbb{N}\) denote the number of states and actions for player \(i\), respectively. We let \(S_t^i\in[n^i]\) and \(A_t^i\in[m^i]\) denote the state and action of player \(i\) at time \(t\in\mathbb{N}\). Each action triggers a stochastic transition between the current state to the next state. We let \(T^i\in\mathbb{R}^{n^i\times m^i\times n^i}\) denote the \emph{transition kernel} of player \(i\) such that
\begin{equation}\label{eqn: transition kernel}
T_{saj}^i\coloneqq \mathds{P}(S_{t+1}^i=j|S_t^i=s, A_t^i=a)
\end{equation}
for all \(t\in\mathbb{N}\), \(s, j\in[n^i]\) and \(a\in[m^i]\). We let \(q^i\in\mathbb{R}^{n^i}\) denote the \emph{initial state distribution} of player \(i\) such that
\begin{equation}\label{eqn: init dist}
q_s^i\coloneqq \mathds{P}(S_0^i=s)
\end{equation}
for all \(s\in[n^i]\). We let \(R^i\in\mathbb{R}^{n^i\times m^i}\) denote the reward matrix, where
\(R_{sa}^i\) denotes the reward of player \(i\) for choosing choosing action \(a\) in state \(s\). Finally, we let \(\gamma\in[0, 1)\) denote a reward discount factor. For each player \(i\in[p]\), a {stationary policy} maps each state to a probability distribution over actions. We denote such a policy as a matrix \(\Pi^i\in\mathbb{R}^{n^i\times m^i}\) where
\begin{equation}\label{eqn: policy}
\Pi_{sa}^i\coloneqq \mathds{P}(A_t^i=a| S_t^i=s)
\end{equation}
for all \(t\in\mathbb{N}, s\in[n^i], a\in[m^i]\). An optimal stationary policy in an MDP minimizes the following expected total discounted state-action reward
\begin{equation}\label{eqn: reward sum}
\sum_{t=0}^\infty \sum_{s=1}^n\sum_{a=1}^m\gamma^t \mathds{P}(S_t^i=s, A_t^i=a) R_{sa}^i.
\end{equation}
We let \(Y^i\in\mathbb{R}^{n^i\times m^i}\) denote the state-action frequency matrix of player \(i\in[p]\) such that
\begin{equation}\label{eqn: state-action frequency}
Y_{sa}^i\coloneqq\sum_{t=0}^\infty \gamma^t \mathds{P}(S_t^i=s, A_t^i=a).
\end{equation}
for all \(s\in[n^i]\) and \(a\in[m^i]\).
We now introduce the definition of a \(p\)-player affine Markov game.
\begin{definition}
A \(p\)-player affine Markov game is a collection of MDPs \(\{\mathcal{M}^i=\{[n^i], [m^i], q^i, T^i, R^i, \gamma\}\}_{i=1}^p\) such that there exists \(b^i\in\mathbb{R}^{m^in^i}\) and \(C^{ij}\in\mathbb{R}^{m^in^i\times m^jn^j}\) for each \(i, j\in[p]\) such that
\begin{equation}\label{eqn: game reward}
\vect(R^i)=b^i+\sum_{j=1}^p C^{ij} \vect(Y^j)
\end{equation}
for all \(i\in[p]\), where \(Y^i\in\mathbb{R}^{m^i\times n^i}\) satisfies \eqref{eqn: state-action frequency}.
\end{definition}
The affine reward structure in \eqref{eqn: game reward} couples different players' decisions together: the reward for a player is not a fixed number, but depends on the other players' state-action frequencies. Similar coupling appears in matrix games where each player has a finite number of candidate options \cite{rosen1965existence,yu2022inverse}. This function has two parameters: $b$ pertains to the individual player, and $C$ considers the coupling between the players.
\subsection{Soft-Bellman Equilibrium}
We now introduce the notion of \emph{soft-Bellman equilibrium}.
% It extends the quantal response equilibrium in static games to dynamic games.
It extends the notion of quantal response equilibrium in games with deterministic dynamics to Markov games with stochastic dynamics \cite{mckelvey1995quantal, mckelvey1998quantal}.
Unlike Nash equilibrium, it states that all players choose a soft-Bellman policy---rather than the optimal policy that satisfies the Bellman equations---given other players' actions.
\begin{definition}\label{def: soft-Bellman equilibrium}
Let \(\{\mathcal{M}^i=\{[n^i], [m^i], q^i, T^i, R^i, \gamma\}\}_{i=1}^p\) be an affine Markov game. Let \(\Pi^i\in\mathbb{R}^{n^i\times m^i}\)
be a stationary policy matrix of player \(i\in[p]\). If there exists \(v^i\in\mathbb{R}^{n^i}\) and \(Q^i\in\mathbb{R}^{n^i\times m^i}\) such that
\begin{subequations}
\begin{align}
\Pi_{sa}^i&=\frac{\exp( Q_{sa}^i)}{\sum_{j=1}^{m^i}\exp(Q_{sj}^i)},\label{eqn: soft policy} \\
Q_{sa}^i&=R_{sa}^i+\gamma \sum_{j=1}^{n^i} T_{saj}^iv_j^i,\label{eqn: soft Q} \\
v_s^i &= \log\left(\sum_{a=1}^{m^i} \exp(Q_{sa}^i)\right),\label{eqn: soft V}
\end{align}
\end{subequations}
for all \(s\in[n^i]\) and \(a\in[m^i]\), then \(\{\Pi^i\}_{i=1}^p\), is a \emph{soft-Bellman equilibrium} for \(\{\mathcal{M}^i\}_{i=1}^p\).
\end{definition}
Previous studies have proposed similar notions of equilibrium, e.g., Markov quantal response equilibrium in \cite{eibelshauser2019markov}. However, unlike \Cref{eqn: soft V} in \cref{def: soft-Bellman equilibrium}, their formulations are inconsistent with the characterization of soft-Bellman policies. There is a close connection between the soft-Bellman equilibrium and the following optimization over state-action frequency matrix:
\begin{equation}\label{opt: best response}
\begin{array}{ll}
\underset{Y\in\mathbb{R}^{n^i\times m^i}}{\mbox{maximize}} & \ell^i(Y)+h(Y)\\
\mbox{subject to} & \sum\limits_{a=1}^{m^i}Y_{sa}=q_s^i + \gamma \sum\limits_{j=1}^{n^i}\sum\limits_{a=1}^{m^i} T_{jas}^i Y_{ja},\enskip s\in[n^i],
\end{array}
\end{equation}
where
\begin{subequations}
\begin{align}
&\begin{aligned}
\ell^i(Y) &\coloneqq \vect(Y)^\top b^i+\frac{1}{2}\vect(Y)^\top C^{ii} \vect(Y)\\
&+\sum_{\substack{j=1, j\neq i}}^p \vect(Y^j)^\top C^{ij}\vect(Y),
\end{aligned} \label{eqn: potential}\\
&h(Y)\coloneqq\sum_{s=1}^n\sum_{a=1}^mY_{sa}\left(\log\left(\sum_{j=1}^m Y_{sj}\right)-\log(Y_{sa})\right).\label{eqn: entropy}
\end{align}
\end{subequations}
The following theorem shows that, if each player chooses its policy by solving optimization~\eqref{opt: best response}, then the resulting policies form a soft-Bellman equilibrium.
\begin{theorem}\label{lem: soft-Bellman equilibrium}
Let \(\{\mathcal{M}^i=\{[n^i], [m^i], q^i, T^i, R^i, \gamma\}\}_{i=1}^p\) be an affine Markov game. Suppose that \(C^{ii}\preceq 0\) and \(Y^i\in\mathbb{R}_{>0}^{n^i\times m^i}\) is an optimal solution of optimization~\eqref{opt: best response} for all \(i\in[p]\). Let \(\Pi^i\in\mathbb{R}^{n^i\times m^i}\) be such that
\begin{equation}\label{eqn: Y2Pi}
\Pi_{sa}^i=
\frac{Y_{sa}^i}{\sum_{j=1}^{m^i} Y_{sj}^i}
\end{equation}
for all \(i\in[p]\), \(s\in[n^i]\), and \(a\in[m^i]\). Then \(\{\Pi^i\}_{i=1}^p\) is a soft-Bellman equilibrium for \(\{\mathcal{M}^i\}_{i=1}^p\).
\end{theorem}
\begin{proof}
First, \(h(Y)\) is a concave function of matrix \(Y\) \cite{bloem2014infinite,zhou2017infinite}, and \(\ell^i(Y)\) is also a concave function of \(Y\) since \(C^{ii}\preceq 0\). Next, by applying the chain rule to \eqref{eqn: potential} and \eqref{eqn: entropy} we can show the following:
\begin{equation*}
\textstyle \partial_{Y_{sa}}\ell^i(Y)= R_{sa}^i,\enskip \partial_{Y_{sa}} h(Y) = \log\left(\sum_{j=1}^m Y_{sj}\right)-\log (Y_{sa}),
\end{equation*}
where \(R^i\in\mathbb{R}^{n^i\times m^i}\) satisfies \eqref{eqn: game reward}. Since \(Y\in\mathbb{R}^{n^i\times m^i}_{>0}\) is an optimal solution for optimization~\eqref{opt: best response}, there exists \(v^i\in\mathbb{R}^{n^i}\) such that the following Karush-Kuhn-Tucker conditions hold:\begin{subequations}
\begin{align}
& \sum\limits_{a=1}^{m^i}Y_{sa}^i= q_s^i + \gamma \sum\limits_{j=1}^{n^i}\sum\limits_{a=1}^{m^i} T_{jas}^i Y_{ja}^i,\\
& \log (Y_{sa}^i)-\log\left(\sum_{j=1}^{m^i} Y_{sj}^i\right)=Q_{sa}^i-v_s^i,\label{eqn: kkt dual}
\end{align}
\end{subequations}
for all \(s\in[n^i]\) and \(a\in[m^i]\), where \(Q_{sa}^i\) is given by \eqref{eqn: soft Q}. Let \(\Pi_{sa}^i\) be given by \eqref{eqn: Y2Pi}, then \eqref{eqn: kkt dual} implies that
\begin{subequations}
\begin{align}
\Pi_{sa}^i&=\exp(Q_{sa}^i-v_s^i),\label{eqn: pi a}\\
1=\sum_{j=1}^{m^i} \Pi_{sj}^i&=\sum_{j=1}^{m^i}\exp(Q_{sj}^i-v_s^i), \label{eqn: pi b}
\end{align}
\end{subequations}
for all \(s\in[n^i]\) and \(a\in[m^i]\). By combining \eqref{eqn: pi a} with \eqref{eqn: pi b} one can obtain the condition in \eqref{eqn: soft policy}. Finally, multiplying both sides of \eqref{eqn: pi b} by \(\exp(v_s^i)\) gives \eqref{eqn: soft V}, which completes the proof.
\end{proof}
\section{Forward Solution via Nonlinear Least-squares}
% We now discuss how to compute a soft-Bellman equilibrium by solving a nonlinear least-squares problem, together with the existence and uniqueness of the solution.
We now establish the existence and uniqueness of a soft-Bellman equilibrium, followed by a discussion on how to compute it by solving a nonlinear least-squares problem.
To this end, we introduce the following notation:
\begin{equation}
\begin{aligned}
l& \coloneqq \sum_{i=1}^p m^in^i, \enskip b \coloneqq \begin{bmatrix}
(b^1)^\top & (b^2)^\top & \cdots & (b^p)^\top
\end{bmatrix}^\top, \\
r &\coloneqq \sum_{i=1}^p n^i,\enskip q \coloneqq \begin{bmatrix}
(q^1)^\top & (q^2)^\top & \cdots & (q^p)^\top
\end{bmatrix}^\top.
\end{aligned}
\end{equation}
Let matrices \(D^i, E^i\in\mathbb{R}^{n^i\times m^in^i}\) be such that
\begin{equation}\label{eqn: incidence game}
D^i= I_{n^i}\otimes (\mathbf{1}_{m^i}^\top), \enskip E^i_{kj} = T_{\quot(j, m)+1,\rem(j, m), k}^i,
\end{equation}
for all \(k\in[n^i]\) and \(j\in[m^in^i]\). Furthermore, let
\begin{equation}
\begin{aligned}
H & \coloneqq \blkdiag(D^1-\gamma E^1, D^2-\gamma E^2, \ldots, D^p-\gamma E^p),\\
K & \coloneqq \blkdiag( (D^1)^\top D^1, (D^2)^\top D^2, \ldots, (D^p)^\top D^p),\\
C & \coloneqq \begin{bmatrix}
C^{11} & C^{12} & \cdots & C^{1p}\\
C^{21} & C^{22} & \cdots & C^{2p}\\
\vdots & \vdots & \ddots & \vdots\\
C^{p1} & C^{p2} & \cdots & C^{pp}\\
\end{bmatrix}.
\end{aligned}
\end{equation}
With these notations, we are ready to establish the following results on the existence and uniqueness of the soft-Bellman equilibrium.
\begin{theorem}\label{thm: existence & uniqueness}
Let \(\{\mathcal{M}^i=\{[n^i], [m^i], q^i, T^i, R^i, \gamma\}\}_{i=1}^p\) be an affine Markov game where \(C^{ii}\preceq 0\) for all \(i\in[p]\). Then \(Y^i\in\mathbb{R}^{n^i\times m^i}_{>0}\) is an optimal solution of optimization~\eqref{opt: best response} for all \(i\in[p]\) if and only if there exists \(v\in\mathbb{R}^r\) such that
\begin{equation}\label{eqn: nl eqn}
\begin{aligned}
\log(y) &= \log(K y) +b+Cy-H^\top v,\\
Hy&=q,
\end{aligned}
\end{equation}
where
\begin{equation}
y = \begin{bmatrix}
\vect(Y^1)^\top & \vect(Y^2)^\top & \cdots & \vect(Y^p)^\top
\end{bmatrix}^\top.
\end{equation}
Furthermore, there exists \(y\in\mathbb{R}^{l}\) such that \eqref{eqn: nl eqn} holds for some \(v\in\mathbb{R}^{r}\). If \(C+C^\top\preceq 0\), then such a \(y\) is unique.
\end{theorem}
\begin{proof}
First of all, the conditions \eqref{eqn: nl eqn} are the union of the KKT conditions for optimization~\eqref{opt: best response} for all \(i\in[p]\). Due to the assumption that \(C^{ii}\preceq 0\), \(C+C^\top \preceq 0\), and the strict concavity of logarithm function, one can verify that \(Y^i\in\mathbb{R}^{m^i\times n^i}\) is an optimal solution of optimization~\eqref{opt: best response} for all \(i\in[p]\) if and only if \(\{Y^i\}_{i=1}^p\) is a Nash equilibrium of a \(p\)-player diagonally strictly concave game, which exists and is unique \cite{rosen1965existence}.
\end{proof}
As a result of \Cref{thm: existence & uniqueness}, one can compute a soft-Bellman equilibrium by solving the following nonlinear least-squares problem:
\begin{equation}\label{opt: nl ls}
\begin{array}{ll}
\underset{y, v}{\mbox{minimize}} & \norm{\log(Ky)+b+Cy-H^\top v-\log(y)}^2\\&+\norm{Hy-q}^2
\end{array}
\end{equation}
Notice that the optimal value of the above optimization is zero, since there exists at least one solution for the nonlinear equations in \eqref{eqn: nl eqn}.
\section{Inverse Learning via Implicit Differentiation}
Given the parameters of an affine Markov game, one can compute a soft-Bellman equilibrium of this game by solving the nonlinear least-squares problem in \eqref{opt: nl ls}. The question remains, however, of how to infer these parameters such that they best explains observed decisions, a problem also known as the \emph{inverse game}. Next, we answer this question by developing a projected-gradient method for parameter calibration.
The inverse game problem is a parameter optimization problem defined as follows. We start with a set of empirically observed equilibrium state-action frequencies,
\begin{equation}
\hat{Y}^1, \hat{Y}^2, \ldots, \hat{Y}^p,
\end{equation}
where \(\hat{Y}^i_{sa}\in\mathbb{R}^{m^i\times n^i}\) denotes the empirical probability for player \(i\) to choose action \(a\) in state \(s\). Let
\begin{equation}\label{eqn: observation}
\hat{y}\coloneqq \begin{bmatrix}
\vect(\hat{Y}^1)^\top & \vect(\hat{Y}^2)^\top & \cdots & \vect(\hat{Y}^p)^\top
\end{bmatrix}^\top.
\end{equation}
To find the best parameters that explain the observed state-action frequency matrices in \eqref{eqn: observation}, one can solve the following optimization problem
\begin{equation}\label{opt: bilevel}
\begin{array}{ll}
\underset{y, v, b, C}{\mbox{minimize}} & \norm{y-\hat{y}}^2\\
\mbox{subject to} & \log(y) = \log(K y) +b+Cy-H^\top v,\\
& Hy=q, \enskip b\in\mathbb{B}, \enskip C\in\mathbb{D},
\end{array}
\end{equation}
where \(\mathbb{B}\subset\mathbb{R}^l\) and \(\mathbb{D}\subset\mathbb{R}^{l\times l}\) are closed convex constraint sets for vector \(b\) and matrix \(C\), respectively.
Solving problem \eqref{opt: bilevel} is numerically challenging because this optimization contains both nonlinear equation constraints and possible positive semi-definite cone constraints in set \(\mathbb{D}\). As a remedy, we propose an approximate projected-gradient method that combines nonlinear equation solving with efficient projections. To this end, we let
\begin{equation}
J=\begin{bmatrix}
K\diag(Ky)^{-1} -\diag(y)^{-1}+C & -H^\top\\
H & 0_{r\times r}.
\end{bmatrix}
\end{equation}
By using the chain rule and the implicit function theorem \cite{dontchev2014implicit} one can show that, if \eqref{eqn: nl eqn} holds and matrix \(J\) is nonsingular, then
\begin{subequations}
\begin{align}
\partial_b \norm{y-\hat{y}}^2&=-2\begin{bmatrix}
(y-\hat{y})^\top & 0_{r}
\end{bmatrix}J^{-1} \begin{bmatrix}
I_l\\
0_{r\times l}
\end{bmatrix},\\
\partial_{C_j} \norm{y-\hat{y}}^2&=-2y_j\begin{bmatrix}
(y-\hat{y})^\top & 0_{r}
\end{bmatrix}J^{-1} \begin{bmatrix}
I_l\\
0_{r\times l}
\end{bmatrix},
\end{align}
\end{subequations}
for all \(j\in[l]\), where \(C_j\in\mathbb{R}^l\) is the \(j\)-th column of matrix \(C\). Hence one can compute the approximate gradient for vector \(b\) and matrix \(C\) that locally decreases the value of the objective function in \eqref{opt: bilevel} as follows:
\begin{subequations}\label{eqn: approx grad}
\begin{align}
\tilde{\nabla}_b \norm{y-\hat{y}}^2 & \coloneqq -2\begin{bmatrix}
I_l & 0_{l\times r}
\end{bmatrix}(J^\dagger)^\top \begin{bmatrix}
y-\hat{y}\\
0_{r}
\end{bmatrix},\label{eqn: b grad}\\
\tilde{\nabla}_C \norm{y-\hat{y}}^2 & \coloneqq -2\begin{bmatrix}
I_l & 0_{l\times r}
\end{bmatrix}(J^\dagger)^\top \begin{bmatrix}
y-\hat{y}\\
0_{r}
\end{bmatrix}y^\top.\label{eqn: C grad}
\end{align}
\end{subequations}
Notice that we approximate \(J^{-1}\) with the Moore-Penrose pseudoinverse \(J^\dagger\) in \eqref{eqn: approx grad}. Such an approximation is exact if \(J\) is nonsingular, and still well-defined even if \(J\) is singular or \(J^{-1}\) is numerically unstable to compute.
Based on the formulas in \eqref{eqn: approx grad}, we propose an approximate projected-gradient method, summarized in Algorithm~\ref{alg: proj grad}, to solve optimization~\eqref{opt: bilevel}, where we let
\begin{subequations}
\begin{align}
\text{Proj}_{\mathbb{B}}(b) & \coloneqq \underset{z\in\mathbb{B}}{\argmin} \norm{z-b},\\
\text{Proj}_{\mathbb{D}}(C) & \coloneqq \underset{X\in\mathbb{D}}{\argmin} \norm{X-C}_F,
\end{align}
\end{subequations}
for all \(b\in\mathbb{R}^l\) and \(C\in\mathbb{R}^{l\times l}\).
Each iteration of this method first solves the nonlinear least-squares problem in \eqref{opt: nl ls}, then performs a projected-gradient step on \(b\) and \(C\).
\begin{algorithm}[!ht]
\caption{Approximated projected-gradient method.}
\begin{algorithmic}[1]
\Require Step size \(\alpha\in\mathbb{R}_{>0}\), number of iterations \(k_{\max}\in\mathbb{N}\), random initial parameters $b_{init}\in\mathbb{R}^{l}, C_{init}\in\mathbb{R}^{l\times l}$, tolerance \(\epsilon\in\mathbb{R}\).
\State Initialize \(k=1\), \(b=b_{init}\), \(C=C_{init}\).
\While{\(k<k_{\max}\)}
\State Solve optimization \eqref{opt: nl ls} for \(y\).
\If{change in $\norm{y-\hat{y}}^2 < \epsilon$}
\State terminate.
\EndIf
\State \(b\gets \text{Proj}_{\mathbb{B}}(b-\alpha \tilde{\nabla}_b\norm{y-\hat{y}}^2)\) \Comment{cf. \eqref{eqn: b grad}} \label{lst:line:b_update}
\State \(C\gets \text{Proj}_{\mathbb{D}}(C-\alpha \tilde{\nabla}_C\norm{y-\hat{y}}^2)\) \Comment{cf. \eqref{eqn: C grad}} \label{lst:line:C_update}
\State \(k\gets k+1\)
\EndWhile
\Ensure Vector \(b\) and matrix \(C\).
\end{algorithmic}
\label{alg: proj grad}
\end{algorithm}
\section{Experiments}
\label{sec: experiments}
We evaluate the performance of the proposed algorithm against a baseline algorithm that neglects the fact that players' reward functions depend upon each other's actions in a predator-prey OpenAI Gym environment \cite{magym}.
% We compare our algorithm with a baseline algorithm that applies IRL for each player individually. Results show that by considering coupling between players, the reward parameters inferred by the proposed algorithm outperform those inferred by the baseline by reducing the Kullback-Leibler divergence between the equilibrium policies and observed policies by at least two orders of magnitude.
We solve the forward problem by specifying the nonlinear least-squares problem \eqref{opt: nl ls} in Julia \cite{bezanson2017julia} using the JuMP \cite{dunning2017jump} interface and the COIN-OR IPOPT \cite{wachter2006implementation} optimizer. The source code is publicly available at \url{https://github.com/vivianchen98/Inverse_MDPGame}.
\subsection{Baseline}
The baseline algorithm is a decoupled version of \Cref{alg: proj grad}, that is, it solves optimization \eqref{opt: best response} with the coupling parameter $C^{ij}=0$ for all players $i,j\in [p]$.
Dropping this parameter frees the baseline algorithm to solve an optimization for each player independently, similar to many existing multi-agent inverse reinforcement learning algorithms.
% That is, for $s\in[n]$, each player solves
% \begin{equation}
% \begin{array}{ll}
% \underset{Y\in\mathbb{R}^{n\times m}}{\mbox{maximize}} & \vect(Y)^\top b +h(Y)\\
% \mbox{subject to} & \sum\limits_{a=1}^{m}Y_{sa}=p_s + \gamma \sum\limits_{j=1}^{n}\sum\limits_{a=1}^{m} T_{jas} Y_{ja}
% \end{array}
% \end{equation}
% \begin{figure}[h]
% \centering
% \begin{subfigure}[b]{0.49\linewidth}
% \includegraphics[width=\linewidth]{figures/predator_prey_env.pdf}
% \caption{Predator-prey environment.}
% \label{fig:figure1}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.49\linewidth}
% \includegraphics[width=\linewidth]{figures/test score.pdf}
% \caption{Test return of MARL models.}
% \label{fig:figure2}
% \end{subfigure}
% \caption{Environment and Data Collection}
% \label{fig:both}
% \end{figure}
\subsection{Algorithm Parameters}
For the projected-gradient method in \Cref{alg: proj grad}, we use a backtracking line search technique to fine-tune the step size in line~\ref{lst:line:b_update} and \ref{lst:line:C_update} based upon the Armijo (sufficient decrease) condition \cite{armijo1966minimization}. Each iteration starts with an initial step size $\alpha=1$, and the algorithm reduces the step size by half until it meets the sufficient decrease condition. Both algorithms terminate when the change in $\norm{y-\hat{y}}^2$ is below a given tolerance $\epsilon=0.005$. The maximum number of iterations $k_{\max}$ is $100$, and the discount factor $\gamma$ is $0.99$. We sample the values of the vector $b_{init}$ and the matrix $C_{init}$ from a random number generator given a seed. We run both algorithms from seed $1$ to $10$.
\subsection{Predator-Prey Environment}
\label{subsec: predator_prey_env}
\begin{figure}
\centering
\begin{tikzpicture}[scale=0.6]
% Draw the 5x5 grid
\draw[step=1cm,color=black] (0,0) grid (5,5);
% Fill light blue cells
\fill[predatorblue!20] (1.1,1.1) rectangle (1.9,1.9);
\fill[predatorblue!20] (2.1,1.1) rectangle (2.9,1.9);
\fill[predatorblue!20] (2.1,0.1) rectangle (2.9,0.9);
\fill[predatorblue!20] (2.1,2.1) rectangle (2.9,2.9);
\fill[blue!20] (3.1,1.1) rectangle (3.9,1.9);
\fill[predatorblue!20] (4.1,0.1) rectangle (4.9,0.9);
\fill[predatorblue!20] (4.1,1.1) rectangle (4.9,1.9);
\fill[predatorblue!20] (4.1,2.1) rectangle (4.9,2.9);
% Draw the blue octagon nodes in two cells
\node[fill=predatorblue, regular polygon, regular polygon sides=8, inner sep=2pt] at (2.5,1.5) {\scriptsize\textcolor{white}{1}};
\node[fill=predatorblue, regular polygon, regular polygon sides=8, inner sep=2pt] at (4.5,1.5) {\scriptsize\textcolor{white}{2}};
% Draw the red octagon node in one cell
\node[fill=preyred, regular polygon, regular polygon sides=8, inner sep=2pt] at (3.5,0.5) {\scriptsize\textcolor{white}{1}};
% Add the legend
\matrix [right] at (current bounding box.east) {
\node [fill=predatorblue, regular polygon, regular polygon sides=8,label=right:{\small Predator}] {}; \\
\node [fill=preyred, regular polygon, regular polygon sides=8,label=right:{\small Prey}] {}; \\
};
\end{tikzpicture}
\caption{Two predators (blue) and one prey (red) moving in a 5x5 GridWorld. The light blue cells represent the catching region of the predators, and the light purple cell represents the overlapping of both predators' catching regions. This episode terminates when the prey is inside a light purple cell.}
\label{fig:predator_prey_env}
\end{figure}
We consider a predator-prey environment from a collection of multi-agent environments based on OpenAI Gym \cite{magym}. As shown in \cref{fig:predator_prey_env}, two predators attempt to capture one randomly moving prey in a $5\times 5$ GridWorld. Each predator has observations of all players and the coordinates of the prey relative to itself and selects one of five actions: \texttt{left}, \texttt{right}, \texttt{up}, \texttt{down}, or \texttt{stop}. The prey is caught when it is within the catching region (light blue cells in \cref{fig:predator_prey_env}) of at least one predator. An episode terminates when the prey is caught by more than one predator (inside a light purple cell in \cref{fig:predator_prey_env}), resulting in a positive reward. For every new episode, the environment initializes the prey into random locations and the prey never moves voluntarily into the predators' neighborhood.
In this environment, only the two predators are controllable, but we collect the trajectories of all three players, including the prey, to solve the inverse game problem.
\subsection{Observed Dataset Collection}
We collect all players' trajectories as the observed interactions. Each trajectory is a sequence of states and actions until termination for the current episode.
We train a policy using a multi-agent reinforcement learning algorithm \cite{sunehag2017value} and sample trajectories from this policy. The players in this policy exhibit uncertainties in their decision-making process that are difficult to articulate explicitly, much like humans. As a result, the data from these models can serve as a proxy for human datasets.
% \david{this sentence needs revision}
We process the collected trajectories from all three players by first pruning those shorter than the $50$th percentile of trajectory lengths and then capping the remaining trajectories to the same length. After processing, we attain $100$ useful trajectories of length $6$. We compute the collection of state-action frequencies for all three players $\hat{y}$ and approximate the initial state distributions and the transition probabilities for all players using the observed data.
\begin{figure}
\centering
\begin{tikzpicture}
\begin{semilogyaxis}[
xlabel={{\small Iteration}},
ylabel={{\small $\norm{y-\hat{y}}^2$}},
xmin=1, xmax=50,
ymin=0, ymax=5000,
width=0.65\linewidth,
height =4.5cm,
xtick={1,10,20,30,40,50},
xtick pos=left,
ytick={1,10,100,1000},
ytick style={draw=none},
legend style={at={(1.65,0.6)},anchor=north east, legend cell align={left}},
ymajorgrids = true,
% xmajorgrids = true
]
% Custom legend symbol
\addlegendimage{orange,mark=o, thick}
\addlegendentry{{\small \bf Proposed}}
\addlegendimage{teal,mark=o, thick}
\addlegendentry{{\small Baseline}}
% game terminations
\addplot[only marks, mark=o, orange, thick]coordinates {(34,0.532066938000832) (32,0.4992389546) (30,0.5306287569) (28,0.5563287063) (33,0.5180440768) (29,0.5098950252) (38,0.4764974208) (32,0.5275647948) (29,0.5169438494) (25,0.6065868245)};
% decoupled terminations
\addplot[only marks, mark=o, teal, thick]coordinates {(7,398.3584228383) (79,1025.82921208205) (100,1026.258229) (49, 57.06564416) (55,1025.238772) (18, 67.55653781) (17,126.8949034) (83,1025.054204) (59,1024.884266) (33,129.5089324)};
\addplot [mark=none, orange, thick] table [x=iter, y=seed1, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed2, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed3, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed4, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed5, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed6, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed7, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed8, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed9, col sep=comma]{psi_game.dat};
\addplot [mark=none, orange, thick] table [x=iter, y=seed10, col sep=comma]{psi_game.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed1, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed2, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed3, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed4, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed5, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed6, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed7, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed8, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed9, col sep=comma]{psi_decoupled.dat};
\addplot [mark=none, dashed, teal, thick] table [x=iter, y=seed10, col sep=comma]{psi_decoupled.dat};
\end{semilogyaxis}
\end{tikzpicture}
\caption{Algorithms for the inverse game problem with termination marked in circles (the lower the better).}
\label{fig:psi_convergence}
\end{figure}
% pi gridworld 3x2
\begin{figure}[h]
\begin{tabular}{p{0.1cm}cc}
& {\small \bf \centering Proposed} & {\small Baseline}\\
\rotatebox[origin=c]{90}{{\small Predator} \tiny\octagoned{1}{predatorblue}{white}} & \includegraphics[width=0.44\linewidth,valign=m]{figures/kl/kl_game1.png} & \includegraphics[width=.44\linewidth,valign=m]{figures/kl/kl_decoupled1.png}\\
\rotatebox[origin=c]{90}{{\small Predator} \tiny\octagoned{2}{predatorblue}{white}} & \includegraphics[width=.44\linewidth,valign=m]{figures/kl/kl_game2.png} & \includegraphics[width=.44\linewidth,valign=m]{figures/kl/kl_decoupled2.png}\\
\rotatebox[origin=c]{90}{{\small Prey} \tiny\octagoned{1}{preyred}{white}} & \includegraphics[width=.44\linewidth,valign=m]{figures/kl/kl_game3.png} & \includegraphics[width=.44\linewidth,valign=m]{figures/kl/kl_decoupled3.png}\\
\end{tabular}
\caption{Heatmaps showing Kullback–Leibler divergence $D_{\text{KL}}(\Pi^i_s \parallel \hat{\Pi}^i_s)$ between the equilibrium policy $\Pi^i_s$ and the observed policy $\hat{\Pi}^i_s$ at each state $s$ in the GridWorld for all three players. All values rounded to two decimal places, the smaller (lighter color) the better.}
\label{fig:kl_gridworld}
\end{figure}
\subsection{Numerical Results}
We demonstrate \cref{alg: proj grad} and the baseline algorithm on the predator-prey environment introduced in \cref{subsec: predator_prey_env}. \cref{fig:psi_convergence} shows $\norm{y-\hat{y}}^2$, the squared norm of the difference between the computed state-action frequency $y$ and the observed state-action frequency matrices $\hat{y}$, with respect to the number of iterations.
% Throughout we let initial step size be $\alpha=1$, maximum number of iterations be $k_{\max}=100$, tolerance be $\delta = 0.005$, and the discount factor be $\gamma=0.99$. We run both the proposed algorithm and the baseline for ten different initializations from seeds $1$ to $10$.
Results show the proposed algorithm ends in $31.0\pm 3.6$ iterations, while the baseline algorithm takes $50.0\pm 31.3$ iterations to terminate. As shown in \cref{fig:psi_convergence}, the final iterate of the proposed algorithm has $\norm{y-\hat{y}}^2$ below $1$, while the baseline algorithm on average terminates with a value above $590.7$. This comparison highlights the importance of accounting for the coupling between the players.
Given a state-action frequency matrices $y^i$ for player $i$, we compute the corresponding policies $\Pi^i_{sa}$ by \eqref{eqn: Y2Pi}, and denote the equilibrium policy at each state $s$ as a probability distribution
$$\Pi^i_s = \begin{bmatrix}\Pi^i_{s,\texttt{left}} & \Pi^i_{s,\texttt{right}} & \Pi^i_{s,\texttt{up}} & \Pi^i_{s,\texttt{down}} & \Pi^i_{s,\texttt{stop}}\end{bmatrix}.$$
We report the Kullback–Leibler divergence $D_{\text{KL}}(\Pi^i_s \parallel \hat{\Pi}^i_s)$ between the equilibrium policy $\Pi^i_s$, computed using the proposed and the baseline algorithms, and the observed policy $\hat{\Pi}^i_s$ at each state $s$ for all three players. \cref{fig:kl_gridworld} shows that \cref{alg: proj grad} arrives at an equilibrium policy closer to the observed policy than the baseline algorithm does.
% % kl 2x3 across two cols (more space)
% \begin{figure*}[h]
% \centering
% \begin{tabular}{cccc}
% & Predator \scriptsize\octagoned{1}{predatorblue}{white} & Predator \scriptsize\octagoned{2}{predatorblue}{white} & Prey \scriptsize\octagoned{1}{preyred}{white}\\
% {\bf Ours} & \includegraphics[width=0.25\textwidth,valign=m]{figures/kl/kl_game1.png} & \includegraphics[width=0.25\textwidth,valign=m]{figures/kl/kl_game2.png} & \includegraphics[width=0.25\textwidth,valign=m]{figures/kl/kl_game3.png}\\
% Baseline & \includegraphics[width=0.25\textwidth,valign=m]{figures/kl/kl_decoupled1.png} & \includegraphics[width=0.25\textwidth,valign=m]{figures/kl/kl_decoupled2.png} & \includegraphics[width=0.25\textwidth,valign=m]{figures/kl/kl_decoupled3.png}\\
% & $D_{\text{KL}}(\Pi^{1}_s \parallel \hat{\Pi}^1_s)$ & $D_{\text{KL}}(\Pi^2_s \parallel \hat{\Pi}^2_s)$ & $D_{\text{KL}}(\Pi^3_s \parallel \hat{\Pi}^3_s)$
% \end{tabular}
% \caption{Heatmaps showing Kullback–Leibler divergence between the learned policy $\Pi_s$ from each algorithm and the observed policy $\hat{\Pi}_s$ at each state $s$ in the GridWorld. All values rounded to two decimal places, the smaller (lighter color) the better.}
% \label{fig:kl_gridworld}
% \end{figure*}
% kl 3x2 (less space)
% \begin{figure}[h]
% \centering
% \begin{tabular}{ccc}
% \rotatebox{90}{Predator 1} & \includegraphics[width=.45\linewidth,valign=m]{figures/kl/kl_game1.png} & \includegraphics[width=.45\linewidth,valign=m]{figures/kl/kl_decoupled1.png}\\
% \rotatebox{90}{Predator 2} & \includegraphics[width=.45\linewidth,valign=m]{figures/kl/kl_game2.png} & \includegraphics[width=.45\linewidth,valign=m]{figures/kl/kl_decoupled2.png}\\
% \rotatebox{90}{Prey} & \includegraphics[width=.45\linewidth,valign=m]{figures/kl/kl_game3.png} & \includegraphics[width=.45\linewidth,valign=m]{figures/kl/kl_decoupled3.png}\\
% & Ours & Baseline\\
% \end{tabular}
% \caption{}
% \label{fig:kl_gridworld}
% \end{figure}
%% value maps
% \begin{figure}[h]
% \centering
% \begin{subfigure}[b]{0.32\linewidth}
% \includegraphics[width=\linewidth]{figures/values/vdn_v1.pdf}
% \caption{Predator 1.}
% \label{fig:v1}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.32\linewidth}
% \includegraphics[width=\linewidth]{figures/values/vdn_v2.pdf}
% \caption{Predator 2.}
% \label{fig:v2}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.32\linewidth}
% \includegraphics[width=\linewidth]{figures/values/vdn_v3.pdf}
% \caption{Prey.}
% \label{fig:v3}
% \end{subfigure}
% \caption{Values of the players.}
% \label{fig:values}
% \end{figure}
%% pi gridworld 3x3
% \begin{figure*}[t]
% \centering
% \begin{tabular}{cccc}
% \rotatebox{90}{Predator 1} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/groundtruth_pi_1.png} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/vdn_pi_1.png} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/vdn_decoupled_pi_1.png}\\
% \rotatebox{90}{Predator 2} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/groundtruth_pi_2.png} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/vdn_pi_2.png} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/vdn_decoupled_pi_2.png}\\
% \rotatebox{90}{Prey} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/groundtruth_pi_3.png} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/vdn_pi_3.png} & \includegraphics[width=.3\linewidth,valign=m]{figures/gridworlds/vdn_decoupled_pi_3.png}\\
% & Ground-truth & Ours & MA-IRL\\
% \end{tabular}
% \caption{Policy in gridworlds.}
% \label{fig:policy_gridworld}
% \end{figure*}
\section{Conclusion \& Future Work}
\label{sec: conclusion}
We proposed soft-Bellman equilibrium as a novel solution concept in affine Markov games, a class of Markov games where an affine reward function couples the players' actions, to capture interactions of boundedly rational players in stochastic, dynamic environments. We provided conditions for the existence and uniqueness of the soft-Bellman equilibrium. We solved the forward problem of computing such an equilibrium for a given affine Markov game and proposed an algorithm to tackle the inverse game problem of inferring players' reward parameters from observed interactions.
Future work should validate the effectiveness of the proposed algorithms using human datasets instead of synthetic datasets. For example, the INTERACTION dataset contains human driving trajectories in interactive traffic scenes \cite{zhan2019interaction}, and can serve as a more representative dataset for the inverse game problem.
% $ biblatex auxiliary file $
% $ biblatex bbl format version 3.2 $
% Do not modify the above lines!
%
% This is an auxiliary file used by the 'biblatex' package.
% This file may safely be deleted. It will be recreated by
% biber as required.
%
\begingroup
\makeatletter
\@ifundefined{ver@biblatex.sty}
{\@latex@error
{Missing 'biblatex' package}
{The bibliography requires the 'biblatex' package.}
\aftergroup\endinput}
{}
\endgroup
\refsection{0}
\datalist[entry]{none/global//global/global}
\entry{shapley1953stochastic}{article}{}
\name{author}{1}{}{%
{{hash=aa2b96ffd25f090ad898c0022f7efa12}{%
family={Shapley},
familyi={S\bibinitperiod},
given={Lloyd\bibnamedelima S},
giveni={L\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
}
\strng{namehash}{aa2b96ffd25f090ad898c0022f7efa12}
\strng{fullhash}{aa2b96ffd25f090ad898c0022f7efa12}
\strng{bibnamehash}{aa2b96ffd25f090ad898c0022f7efa12}
\strng{authorbibnamehash}{aa2b96ffd25f090ad898c0022f7efa12}
\strng{authornamehash}{aa2b96ffd25f090ad898c0022f7efa12}
\strng{authorfullhash}{aa2b96ffd25f090ad898c0022f7efa12}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Proceedings of the National Academy of Sciences}
\field{title}{Stochastic games}
\field{year}{1953}
\endentry
\entry{waugh2013computational}{article}{}
\name{author}{3}{}{%
{{hash=aeb84ca821a159cd3bd67ab094769e36}{%
family={Waugh},
familyi={W\bibinitperiod},
given={Kevin},
giveni={K\bibinitperiod}}}%
{{hash=8e68984dfc0a273545306feb9b211436}{%
family={Ziebart},
familyi={Z\bibinitperiod},
given={Brian\bibnamedelima D},
giveni={B\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=49832523c57a7bf40445f071b2ad2c0a}{%
family={Bagnell},
familyi={B\bibinitperiod},
given={J\bibnamedelima Andrew},
giveni={J\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
}
\strng{namehash}{4476cae964e43a054867c480bd3a06d6}
\strng{fullhash}{4476cae964e43a054867c480bd3a06d6}
\strng{bibnamehash}{4476cae964e43a054867c480bd3a06d6}
\strng{authorbibnamehash}{4476cae964e43a054867c480bd3a06d6}
\strng{authornamehash}{4476cae964e43a054867c480bd3a06d6}
\strng{authorfullhash}{4476cae964e43a054867c480bd3a06d6}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{International Conference on Machine Learning (ICML)}
\field{title}{Computational rationalization: The inverse equilibrium problem}
\field{year}{2013}
\endentry
\entry{barrett2017making}{article}{}
\name{author}{4}{}{%
{{hash=aa0ad09025989909025e9b45533546f9}{%
family={Barrett},
familyi={B\bibinitperiod},
given={Samuel},
giveni={S\bibinitperiod}}}%
{{hash=e9eeab762d0f4344fe106bba36b8c646}{%
family={Rosenfeld},
familyi={R\bibinitperiod},
given={Avi},
giveni={A\bibinitperiod}}}%
{{hash=de674f6370185bbdaef21079761fcd7f}{%
family={Kraus},
familyi={K\bibinitperiod},
given={Sarit},
giveni={S\bibinitperiod}}}%
{{hash=3f42a499da2cddc0bae71b76c81f7c46}{%
family={Stone},
familyi={S\bibinitperiod},
given={Peter},
giveni={P\bibinitperiod}}}%
}
\strng{namehash}{90a110870ce6a59b6f2e42cfd263971b}
\strng{fullhash}{f8b3e8a1cc4ad0325af9f1e0e0d727d1}
\strng{bibnamehash}{f8b3e8a1cc4ad0325af9f1e0e0d727d1}
\strng{authorbibnamehash}{f8b3e8a1cc4ad0325af9f1e0e0d727d1}
\strng{authornamehash}{90a110870ce6a59b6f2e42cfd263971b}
\strng{authorfullhash}{f8b3e8a1cc4ad0325af9f1e0e0d727d1}
\field{sortinit}{4}
\field{sortinithash}{9381316451d1b9788675a07e972a12a7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Artificial Intelligence}
\field{title}{Making friends on the fly: Cooperating with new teammates}
\field{year}{2017}
\endentry
\entry{le2017coordinated}{inproceedings}{}
\name{author}{4}{}{%
{{hash=29a0a0474001e6d80f600571cd161ac1}{%
family={Le},
familyi={L\bibinitperiod},
given={Hoang\bibnamedelima M},
giveni={H\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=d433272a8da287e7d9b17c3f80798a3f}{%
family={Yue},
familyi={Y\bibinitperiod},
given={Yisong},
giveni={Y\bibinitperiod}}}%
{{hash=8f942ac1911200f60a35f4a6a69cf793}{%
family={Carr},
familyi={C\bibinitperiod},
given={Peter},
giveni={P\bibinitperiod}}}%
{{hash=7ec74a05c0565b3f616f47382c901164}{%
family={Lucey},
familyi={L\bibinitperiod},
given={Patrick},
giveni={P\bibinitperiod}}}%
}
\strng{namehash}{5def486c338a2a5d1f88fc5619f4a869}
\strng{fullhash}{9bcda44d83b2f11ae09f26fa8caff950}
\strng{bibnamehash}{9bcda44d83b2f11ae09f26fa8caff950}
\strng{authorbibnamehash}{9bcda44d83b2f11ae09f26fa8caff950}
\strng{authornamehash}{5def486c338a2a5d1f88fc5619f4a869}
\strng{authorfullhash}{9bcda44d83b2f11ae09f26fa8caff950}
\field{sortinit}{4}
\field{sortinithash}{9381316451d1b9788675a07e972a12a7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{International Conference on Machine Learning (ICML)}
\field{title}{Coordinated multi-agent imitation learning}
\field{year}{2017}
\endentry
\entry{vsovsic2016inverse}{inproceedings}{}
\name{author}{4}{}{%
{{hash=a41b551ba01941f0498ae8a8ab62e712}{%
family={Šošić},
familyi={Š\bibinitperiod},
given={Adrian},
giveni={A\bibinitperiod}}}%
{{hash=35b149edc995c4cdc34ad8725bb2af9d}{%
family={KhudaBukhsh},
familyi={K\bibinitperiod},
given={Wasiur\bibnamedelima R},
giveni={W\bibinitperiod\bibinitdelim R\bibinitperiod}}}%
{{hash=2cb73baeffbbd402ac6c599a3b7f078e}{%
family={Zoubir},
familyi={Z\bibinitperiod},
given={Abdelhak\bibnamedelima M},
giveni={A\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=836aadfafb99a418e0308c121a0ac626}{%
family={Koeppl},
familyi={K\bibinitperiod},
given={Heinz},
giveni={H\bibinitperiod}}}%
}
\strng{namehash}{9ee623ad2f7fa6e54da398a28a497a07}
\strng{fullhash}{db67b071065a1a655fd9c908742f8a97}
\strng{bibnamehash}{db67b071065a1a655fd9c908742f8a97}
\strng{authorbibnamehash}{db67b071065a1a655fd9c908742f8a97}
\strng{authornamehash}{9ee623ad2f7fa6e54da398a28a497a07}
\strng{authorfullhash}{db67b071065a1a655fd9c908742f8a97}
\field{sortinit}{4}
\field{sortinithash}{9381316451d1b9788675a07e972a12a7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{International Conference on Autonomous Agents and Multiagent Systems (AAMAS)}
\field{title}{Inverse reinforcement learning in swarm systems}
\field{year}{2016}
\endentry
\entry{lin2017multiagent}{article}{}
\name{author}{3}{}{%
{{hash=f21a54409c1b2a96b74067f7f78d82a4}{%
family={Lin},
familyi={L\bibinitperiod},
given={Xiaomin},
giveni={X\bibinitperiod}}}%
{{hash=240db8d2d8fd318d47cdfb5f8fd81f56}{%
family={Beling},
familyi={B\bibinitperiod},
given={Peter\bibnamedelima A},
giveni={P\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=e54658f1b6e9d9a94b4bcdf79d1bb71e}{%
family={Cogill},
familyi={C\bibinitperiod},
given={Randy},
giveni={R\bibinitperiod}}}%
}
\strng{namehash}{4a47260d8745838f5db8cfa2825a3753}
\strng{fullhash}{4a47260d8745838f5db8cfa2825a3753}
\strng{bibnamehash}{4a47260d8745838f5db8cfa2825a3753}
\strng{authorbibnamehash}{4a47260d8745838f5db8cfa2825a3753}
\strng{authornamehash}{4a47260d8745838f5db8cfa2825a3753}
\strng{authorfullhash}{4a47260d8745838f5db8cfa2825a3753}
\field{sortinit}{5}
\field{sortinithash}{20e9b4b0b173788c5dace24730f47d8c}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Trans. on Games}
\field{title}{Multiagent inverse reinforcement learning for two-person zero-sum games}
\field{year}{2017}
\endentry
\entry{lin2019multi}{article}{}
\name{author}{3}{}{%
{{hash=f21a54409c1b2a96b74067f7f78d82a4}{%
family={Lin},
familyi={L\bibinitperiod},
given={Xiaomin},
giveni={X\bibinitperiod}}}%
{{hash=27296db7f8096772bf0507d16211be4d}{%
family={Adams},
familyi={A\bibinitperiod},
given={Stephen\bibnamedelima C},
giveni={S\bibinitperiod\bibinitdelim C\bibinitperiod}}}%
{{hash=240db8d2d8fd318d47cdfb5f8fd81f56}{%
family={Beling},
familyi={B\bibinitperiod},
given={Peter\bibnamedelima A},
giveni={P\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
}
\strng{namehash}{a1b2b49705236d3725111e149cf6dfed}
\strng{fullhash}{a1b2b49705236d3725111e149cf6dfed}
\strng{bibnamehash}{a1b2b49705236d3725111e149cf6dfed}
\strng{authorbibnamehash}{a1b2b49705236d3725111e149cf6dfed}
\strng{authornamehash}{a1b2b49705236d3725111e149cf6dfed}
\strng{authorfullhash}{a1b2b49705236d3725111e149cf6dfed}
\field{sortinit}{5}
\field{sortinithash}{20e9b4b0b173788c5dace24730f47d8c}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Journal of Artificial Intelligence Research}
\field{title}{Multi-agent inverse reinforcement learning for certain general-sum stochastic games}
\field{year}{2019}
\endentry
\entry{mehr2023maximum}{article}{}
\name{author}{4}{}{%
{{hash=8a1e16de8c190e9174090b57c3a625fd}{%
family={Mehr},
familyi={M\bibinitperiod},
given={Negar},
giveni={N\bibinitperiod}}}%
{{hash=331bf7dced49d10c4c6ca7710edd5a3e}{%
family={Wang},
familyi={W\bibinitperiod},
given={Mingyu},
giveni={M\bibinitperiod}}}%
{{hash=a0d4779457cbacdab06dba4eb8a789de}{%
family={Bhatt},
familyi={B\bibinitperiod},
given={Maulik},
giveni={M\bibinitperiod}}}%
{{hash=c54985d67e300fac8041d549c20bb2f0}{%
family={Schwager},
familyi={S\bibinitperiod},
given={Mac},
giveni={M\bibinitperiod}}}%
}
\strng{namehash}{9e302fd8f6273bed58165501d20006de}
\strng{fullhash}{e8461578c8e1261b8399ff114795ffbf}
\strng{bibnamehash}{e8461578c8e1261b8399ff114795ffbf}
\strng{authorbibnamehash}{e8461578c8e1261b8399ff114795ffbf}
\strng{authornamehash}{9e302fd8f6273bed58165501d20006de}
\strng{authorfullhash}{e8461578c8e1261b8399ff114795ffbf}
\field{sortinit}{6}
\field{sortinithash}{b33bc299efb3c36abec520a4c896a66d}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Trans. on Robotics}
\field{title}{Maximum-entropy multi-agent dynamic games: Forward and inverse solutions}
\field{year}{2023}
\endentry
\entry{dontchev2014implicit}{book}{}
\name{author}{2}{}{%
{{hash=91816db2c832267127fb52d51fe253ac}{%
family={Dontchev},
familyi={D\bibinitperiod},
given={Asen\bibnamedelima L},
giveni={A\bibinitperiod\bibinitdelim L\bibinitperiod}}}%
{{hash=18de24308c36b3348b1477a2f7aa442f}{%
family={Rockafellar},
familyi={R\bibinitperiod},
given={R\bibnamedelima Tyrrell},
giveni={R\bibinitperiod\bibinitdelim T\bibinitperiod}}}%
}
\list{publisher}{1}{%
{Springer}%
}
\strng{namehash}{0cbd4005abfc3812f7e581a33e0686e3}
\strng{fullhash}{0cbd4005abfc3812f7e581a33e0686e3}
\strng{bibnamehash}{0cbd4005abfc3812f7e581a33e0686e3}
\strng{authorbibnamehash}{0cbd4005abfc3812f7e581a33e0686e3}
\strng{authornamehash}{0cbd4005abfc3812f7e581a33e0686e3}
\strng{authorfullhash}{0cbd4005abfc3812f7e581a33e0686e3}
\field{sortinit}{7}
\field{sortinithash}{108d0be1b1bee9773a1173443802c0a3}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{title}{Implicit functions and solution mappings: A view from variational analysis}
\field{year}{2014}
\endentry
\entry{magym}{misc}{}
\name{author}{1}{}{%
{{hash=989db01bb8f72d74658f84139b2a3a55}{%
family={Koul},
familyi={K\bibinitperiod},
given={Anurag},
giveni={A\bibinitperiod}}}%
}
\strng{namehash}{989db01bb8f72d74658f84139b2a3a55}
\strng{fullhash}{989db01bb8f72d74658f84139b2a3a55}
\strng{bibnamehash}{989db01bb8f72d74658f84139b2a3a55}
\strng{authorbibnamehash}{989db01bb8f72d74658f84139b2a3a55}
\strng{authornamehash}{989db01bb8f72d74658f84139b2a3a55}
\strng{authorfullhash}{989db01bb8f72d74658f84139b2a3a55}
\field{sortinit}{8}
\field{sortinithash}{a231b008ebf0ecbe0b4d96dcc159445f}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{howpublished}{\url{https://github.com/koulanurag/ma-gym}}
\field{journaltitle}{GitHub repository}
\field{title}{ma-gym: Collection of multi-agent environments based on OpenAI gym.}
\field{year}{2019}
\endentry
\entry{ziebart2008maximum}{inproceedings}{}
\true{moreauthor}
\true{morelabelname}
\name{author}{4}{}{%
{{hash=8e68984dfc0a273545306feb9b211436}{%
family={Ziebart},
familyi={Z\bibinitperiod},
given={Brian\bibnamedelima D},
giveni={B\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=ed7b150d84a40a9f9067abd89a23a133}{%
family={Maas},
familyi={M\bibinitperiod},
given={Andrew\bibnamedelima L},
giveni={A\bibinitperiod\bibinitdelim L\bibinitperiod}}}%
{{hash=49832523c57a7bf40445f071b2ad2c0a}{%
family={Bagnell},
familyi={B\bibinitperiod},
given={J\bibnamedelima Andrew},
giveni={J\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=383a958e7dc63d5a57bbeec2a4e3ac93}{%
family={Dey},
familyi={D\bibinitperiod},
given={Anind\bibnamedelima K},
giveni={A\bibinitperiod\bibinitdelim K\bibinitperiod}}}%
}
\strng{namehash}{6744e752339e5653b85d150b016f3ef3}
\strng{fullhash}{ce80c51f0b8dd285e3c25c6d9bb17b7d}
\strng{bibnamehash}{ce80c51f0b8dd285e3c25c6d9bb17b7d}
\strng{authorbibnamehash}{ce80c51f0b8dd285e3c25c6d9bb17b7d}
\strng{authornamehash}{6744e752339e5653b85d150b016f3ef3}
\strng{authorfullhash}{ce80c51f0b8dd285e3c25c6d9bb17b7d}
\field{sortinit}{9}
\field{sortinithash}{0a5ebc79d83c96b6579069544c73c7d4}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{Association for the Advancement of Artificial Intelligence (AAAI)}
\field{title}{Maximum entropy inverse reinforcement learning}
\field{year}{2008}
\endentry
\entry{ziebart2013principle}{article}{}
\name{author}{3}{}{%
{{hash=8e68984dfc0a273545306feb9b211436}{%
family={Ziebart},
familyi={Z\bibinitperiod},
given={Brian\bibnamedelima D},
giveni={B\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=49832523c57a7bf40445f071b2ad2c0a}{%
family={Bagnell},
familyi={B\bibinitperiod},
given={J\bibnamedelima Andrew},
giveni={J\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=383a958e7dc63d5a57bbeec2a4e3ac93}{%
family={Dey},
familyi={D\bibinitperiod},
given={Anind\bibnamedelima K},
giveni={A\bibinitperiod\bibinitdelim K\bibinitperiod}}}%
}
\strng{namehash}{33de8443c3958a3a81dc145be8e876b7}
\strng{fullhash}{33de8443c3958a3a81dc145be8e876b7}
\strng{bibnamehash}{33de8443c3958a3a81dc145be8e876b7}
\strng{authorbibnamehash}{33de8443c3958a3a81dc145be8e876b7}
\strng{authornamehash}{33de8443c3958a3a81dc145be8e876b7}
\strng{authorfullhash}{33de8443c3958a3a81dc145be8e876b7}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Trans. on Information Theory}
\field{title}{The principle of maximum causal entropy for estimating interacting processes}
\field{year}{2013}
\endentry
\entry{zhou2017infinite}{article}{}
\name{author}{3}{}{%
{{hash=92bcf64f6a434d65d69d4718e65a964b}{%
family={Zhou},
familyi={Z\bibinitperiod},
given={Zhengyuan},
giveni={Z\bibinitperiod}}}%
{{hash=86035c23cd736e603f0d46b2fe8f8c4e}{%
family={Bloem},
familyi={B\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=24d79d32df4f0bf49ed468479df9b85d}{%
family={Bambos},
familyi={B\bibinitperiod},
given={Nicholas},
giveni={N\bibinitperiod}}}%
}
\strng{namehash}{3037575ab08fa51cd9a442de9bb983e3}
\strng{fullhash}{3037575ab08fa51cd9a442de9bb983e3}
\strng{bibnamehash}{3037575ab08fa51cd9a442de9bb983e3}
\strng{authorbibnamehash}{3037575ab08fa51cd9a442de9bb983e3}
\strng{authornamehash}{3037575ab08fa51cd9a442de9bb983e3}
\strng{authorfullhash}{3037575ab08fa51cd9a442de9bb983e3}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Trans. Autom. Control}
\field{title}{Infinite time horizon maximum causal entropy inverse reinforcement learning}
\field{year}{2017}
\endentry
\entry{mckelvey1995quantal}{article}{}
\name{author}{2}{}{%
{{hash=0e9527c01edc1545dbf5751af3036020}{%
family={McKelvey},
familyi={M\bibinitperiod},
given={Richard\bibnamedelima D},
giveni={R\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=50a037359efff932a346c01880337647}{%
family={Palfrey},
familyi={P\bibinitperiod},
given={Thomas\bibnamedelima R},
giveni={T\bibinitperiod\bibinitdelim R\bibinitperiod}}}%
}
\strng{namehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{fullhash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{bibnamehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{authorbibnamehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{authornamehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{authorfullhash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\field{extraname}{1}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Games and Economic Behavior}
\field{title}{Quantal response equilibria for normal form games}
\field{year}{1995}
\endentry
\entry{mckelvey1998quantal}{article}{}
\name{author}{2}{}{%
{{hash=0e9527c01edc1545dbf5751af3036020}{%
family={McKelvey},
familyi={M\bibinitperiod},
given={Richard\bibnamedelima D},
giveni={R\bibinitperiod\bibinitdelim D\bibinitperiod}}}%
{{hash=50a037359efff932a346c01880337647}{%
family={Palfrey},
familyi={P\bibinitperiod},
given={Thomas\bibnamedelima R},
giveni={T\bibinitperiod\bibinitdelim R\bibinitperiod}}}%
}
\strng{namehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{fullhash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{bibnamehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{authorbibnamehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{authornamehash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\strng{authorfullhash}{5eb58b6913fb87724eb363b7b7d1b9b6}
\field{extraname}{2}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Experimental Economics}
\field{title}{Quantal response equilibria for extensive form games}
\field{year}{1998}
\endentry
\entry{rosen1965existence}{article}{}
\name{author}{1}{}{%
{{hash=ab41c0d9ee3e094ecc03667be30183b5}{%
family={Rosen},
familyi={R\bibinitperiod},
given={J\bibnamedelima Ben},
giveni={J\bibinitperiod\bibinitdelim B\bibinitperiod}}}%
}
\strng{namehash}{ab41c0d9ee3e094ecc03667be30183b5}
\strng{fullhash}{ab41c0d9ee3e094ecc03667be30183b5}
\strng{bibnamehash}{ab41c0d9ee3e094ecc03667be30183b5}
\strng{authorbibnamehash}{ab41c0d9ee3e094ecc03667be30183b5}
\strng{authornamehash}{ab41c0d9ee3e094ecc03667be30183b5}
\strng{authorfullhash}{ab41c0d9ee3e094ecc03667be30183b5}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Econometrica: J. Econ. Soc.}
\field{title}{Existence and uniqueness of equilibrium points for concave n-person games}
\field{year}{1965}
\endentry
\entry{yu2022inverse}{article}{}
\name{author}{4}{}{%
{{hash=512fbf6290ad46e1e3408d0e31b6f54f}{%
family={Yu},
familyi={Y\bibinitperiod},
given={Yue},
giveni={Y\bibinitperiod}}}%
{{hash=5f176711f375af54488a8c32c75c708d}{%
family={Salfity},
familyi={S\bibinitperiod},
given={Jonathan},
giveni={J\bibinitperiod}}}%
{{hash=56ebe8db4552eeddec6e242a9f07620a}{%
family={Fridovich-Keil},
familyi={F\bibinithyphendelim K\bibinitperiod},
given={David},
giveni={D\bibinitperiod}}}%
{{hash=e90ce17540283bba8aa35dabd5db39b1}{%
family={Topcu},
familyi={T\bibinitperiod},
given={Ufuk},
giveni={U\bibinitperiod}}}%
}
\strng{namehash}{57916960d88f63855cd3664a9732175b}
\strng{fullhash}{54c90660caa6a81047663e4e34d74d4e}
\strng{bibnamehash}{54c90660caa6a81047663e4e34d74d4e}
\strng{authorbibnamehash}{54c90660caa6a81047663e4e34d74d4e}
\strng{authornamehash}{57916960d88f63855cd3664a9732175b}
\strng{authorfullhash}{54c90660caa6a81047663e4e34d74d4e}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Control Syst. Lett.}
\field{title}{Inverse matrix games with unique quantal response equilibrium}
\field{year}{2022}
\endentry
\entry{eibelshauser2019markov}{article}{}
\name{author}{2}{}{%
{{hash=df418209009a8be51afcf00384d2c8f2}{%
family={Eibelshäuser},
familyi={E\bibinitperiod},
given={Steffen},
giveni={S\bibinitperiod}}}%
{{hash=a5b354650583fdfe18c8e4ec53af855e}{%
family={Poensgen},
familyi={P\bibinitperiod},
given={David},
giveni={D\bibinitperiod}}}%
}
\strng{namehash}{4bd4484da43481bdc4e0e2ddc82d5ab4}
\strng{fullhash}{4bd4484da43481bdc4e0e2ddc82d5ab4}
\strng{bibnamehash}{4bd4484da43481bdc4e0e2ddc82d5ab4}
\strng{authorbibnamehash}{4bd4484da43481bdc4e0e2ddc82d5ab4}
\strng{authornamehash}{4bd4484da43481bdc4e0e2ddc82d5ab4}
\strng{authorfullhash}{4bd4484da43481bdc4e0e2ddc82d5ab4}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Available at SSRN 3314404}
\field{title}{Markov quantal response equilibrium and a homotopy method for computing and selecting markov perfect equilibria of dynamic stochastic games}
\field{year}{2019}
\endentry
\entry{bloem2014infinite}{inproceedings}{}
\name{author}{2}{}{%
{{hash=86035c23cd736e603f0d46b2fe8f8c4e}{%
family={Bloem},
familyi={B\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=24d79d32df4f0bf49ed468479df9b85d}{%
family={Bambos},
familyi={B\bibinitperiod},
given={Nicholas},
giveni={N\bibinitperiod}}}%
}
\strng{namehash}{e8299e5ee54c3b6146b4de5f7218062e}
\strng{fullhash}{e8299e5ee54c3b6146b4de5f7218062e}
\strng{bibnamehash}{e8299e5ee54c3b6146b4de5f7218062e}
\strng{authorbibnamehash}{e8299e5ee54c3b6146b4de5f7218062e}
\strng{authornamehash}{e8299e5ee54c3b6146b4de5f7218062e}
\strng{authorfullhash}{e8299e5ee54c3b6146b4de5f7218062e}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{Proc. IEEE Conference on Decision Control (CDC)}
\field{title}{Infinite time horizon maximum causal entropy inverse reinforcement learning}
\field{year}{2014}
\endentry
\entry{bezanson2017julia}{article}{}
\name{author}{4}{}{%
{{hash=17e68031e06f57a7e83caeb4e0aed827}{%
family={Bezanson},
familyi={B\bibinitperiod},
given={Jeff},
giveni={J\bibinitperiod}}}%
{{hash=01a6ca61fcd12a3a071ec49304df57f8}{%
family={Edelman},
familyi={E\bibinitperiod},
given={Alan},
giveni={A\bibinitperiod}}}%
{{hash=bca7c93f55e669f71c4ff95e68ac9538}{%
family={Karpinski},
familyi={K\bibinitperiod},
given={Stefan},
giveni={S\bibinitperiod}}}%
{{hash=de802ff42d9c902868c57a332dbac5e0}{%
family={Shah},
familyi={S\bibinitperiod},
given={Viral\bibnamedelima B},
giveni={V\bibinitperiod\bibinitdelim B\bibinitperiod}}}%
}
\strng{namehash}{07e3452af3652a626dc1d02355fda942}
\strng{fullhash}{651af5e2dc744eabe31cd448eb05d640}
\strng{bibnamehash}{651af5e2dc744eabe31cd448eb05d640}
\strng{authorbibnamehash}{651af5e2dc744eabe31cd448eb05d640}
\strng{authornamehash}{07e3452af3652a626dc1d02355fda942}
\strng{authorfullhash}{651af5e2dc744eabe31cd448eb05d640}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{SIAM Review}
\field{title}{Julia: A fresh approach to numerical computing}
\field{year}{2017}
\endentry
\entry{dunning2017jump}{article}{}
\name{author}{3}{}{%
{{hash=6a09fd9d4b66412cec2802755aa5942e}{%
family={Dunning},
familyi={D\bibinitperiod},
given={Iain},
giveni={I\bibinitperiod}}}%
{{hash=3d4093b368f12c6a917914c63f0d75d1}{%
family={Huchette},
familyi={H\bibinitperiod},
given={Joey},
giveni={J\bibinitperiod}}}%
{{hash=ae562f612365bfa1287b8268e8b021df}{%
family={Lubin},
familyi={L\bibinitperiod},
given={Miles},
giveni={M\bibinitperiod}}}%
}
\strng{namehash}{298dd979b252c07511d86580ba18757c}
\strng{fullhash}{298dd979b252c07511d86580ba18757c}
\strng{bibnamehash}{298dd979b252c07511d86580ba18757c}
\strng{authorbibnamehash}{298dd979b252c07511d86580ba18757c}
\strng{authornamehash}{298dd979b252c07511d86580ba18757c}
\strng{authorfullhash}{298dd979b252c07511d86580ba18757c}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{SIAM Review}
\field{title}{JuMP: A modeling language for mathematical optimization}
\field{year}{2017}
\endentry
\entry{wachter2006implementation}{article}{}
\name{author}{2}{}{%
{{hash=dc724f071bd0fcc1e9eabf5f96209481}{%
family={Wächter},
familyi={W\bibinitperiod},
given={Andreas},
giveni={A\bibinitperiod}}}%
{{hash=b4cf951e494c45f4de462df04ec3b5df}{%
family={Biegler},
familyi={B\bibinitperiod},
given={Lorenz\bibnamedelima T},
giveni={L\bibinitperiod\bibinitdelim T\bibinitperiod}}}%
}
\strng{namehash}{842d46b1a24346d113a0e8274fa09dd0}
\strng{fullhash}{842d46b1a24346d113a0e8274fa09dd0}
\strng{bibnamehash}{842d46b1a24346d113a0e8274fa09dd0}
\strng{authorbibnamehash}{842d46b1a24346d113a0e8274fa09dd0}
\strng{authornamehash}{842d46b1a24346d113a0e8274fa09dd0}
\strng{authorfullhash}{842d46b1a24346d113a0e8274fa09dd0}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Mathematical Programming}
\field{title}{On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming}
\field{year}{2006}
\endentry
\entry{armijo1966minimization}{article}{}
\name{author}{1}{}{%
{{hash=9b247b523635a52cd440b2d7f421b150}{%
family={Armijo},
familyi={A\bibinitperiod},
given={Larry},
giveni={L\bibinitperiod}}}%
}
\strng{namehash}{9b247b523635a52cd440b2d7f421b150}
\strng{fullhash}{9b247b523635a52cd440b2d7f421b150}
\strng{bibnamehash}{9b247b523635a52cd440b2d7f421b150}
\strng{authorbibnamehash}{9b247b523635a52cd440b2d7f421b150}
\strng{authornamehash}{9b247b523635a52cd440b2d7f421b150}
\strng{authorfullhash}{9b247b523635a52cd440b2d7f421b150}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Pacific Journal of Mathematics}
\field{title}{Minimization of functions having Lipschitz continuous first partial derivatives}
\field{year}{1966}
\endentry
\entry{sunehag2017value}{inproceedings}{}
\true{moreauthor}
\true{morelabelname}
\name{author}{10}{}{%
{{hash=daf6cbed703aa44e31d950c3c2720056}{%
family={Sunehag},
familyi={S\bibinitperiod},
given={Peter},
giveni={P\bibinitperiod}}}%
{{hash=56f0d34650d75bc4863946b239695751}{%
family={Lever},
familyi={L\bibinitperiod},
given={Guy},
giveni={G\bibinitperiod}}}%
{{hash=670a5857297893dc153211fd8a8ec921}{%
family={Gruslys},
familyi={G\bibinitperiod},
given={Audrunas},
giveni={A\bibinitperiod}}}%
{{hash=2e540193e28e5b3aa63fef2e515c637b}{%
family={Czarnecki},
familyi={C\bibinitperiod},
given={Wojciech\bibnamedelima Marian},
giveni={W\bibinitperiod\bibinitdelim M\bibinitperiod}}}%
{{hash=997657b08786dc45aaef854249d8af31}{%
family={Zambaldi},
familyi={Z\bibinitperiod},
given={Vinicius},
giveni={V\bibinitperiod}}}%
{{hash=7dc1446dea7ff50b2b02fb83780cc9c6}{%
family={Jaderberg},
familyi={J\bibinitperiod},
given={Max},
giveni={M\bibinitperiod}}}%
{{hash=3075d0e02c0833f6e5fe1addb880898f}{%
family={Lanctot},
familyi={L\bibinitperiod},
given={Marc},
giveni={M\bibinitperiod}}}%
{{hash=6292996963ada78bbb98cbb8e5cc1651}{%
family={Sonnerat},
familyi={S\bibinitperiod},
given={Nicolas},
giveni={N\bibinitperiod}}}%
{{hash=698896c85bec4f93adca09e2a21cdf78}{%
family={Leibo},
familyi={L\bibinitperiod},
given={Joel\bibnamedelima Z},
giveni={J\bibinitperiod\bibinitdelim Z\bibinitperiod}}}%
{{hash=abe873d21d37af0af9b6711c1efd9caa}{%
family={Tuyls},
familyi={T\bibinitperiod},
given={Karl},
giveni={K\bibinitperiod}}}%
}
\strng{namehash}{3aadb03dc863012ae319303e8682f1a2}
\strng{fullhash}{6837e20cbb9f81b5b5b4c5ebef8b8204}
\strng{bibnamehash}{6837e20cbb9f81b5b5b4c5ebef8b8204}
\strng{authorbibnamehash}{6837e20cbb9f81b5b5b4c5ebef8b8204}
\strng{authornamehash}{3aadb03dc863012ae319303e8682f1a2}
\strng{authorfullhash}{6837e20cbb9f81b5b5b4c5ebef8b8204}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{International Conference on Autonomous Agents and Multiagent Systems (AAMAS)}
\field{title}{Value-decomposition networks for cooperative multi-agent learning}
\field{year}{2018}
\endentry
\entry{zhan2019interaction}{article}{}
\true{moreauthor}
\true{morelabelname}
\name{author}{10}{}{%
{{hash=bd50c6be2b05731bffd7ee47e638f4e2}{%
family={Zhan},
familyi={Z\bibinitperiod},
given={Wei},
giveni={W\bibinitperiod}}}%
{{hash=46fad125fda69044259286abd9b862f1}{%
family={Sun},
familyi={S\bibinitperiod},
given={Liting},
giveni={L\bibinitperiod}}}%
{{hash=ae66067c3dea21e00e9c5b777faa1a4e}{%
family={Wang},
familyi={W\bibinitperiod},
given={Di},
giveni={D\bibinitperiod}}}%
{{hash=cd2bd313a408f727aad36f311ebe45f3}{%
family={Shi},
familyi={S\bibinitperiod},
given={Haojie},
giveni={H\bibinitperiod}}}%
{{hash=3bf06ad812e5207d957d4682f0e671ee}{%
family={Clausse},
familyi={C\bibinitperiod},
given={Aubrey},
giveni={A\bibinitperiod}}}%
{{hash=adce5c007fa2d9736aa678be03846387}{%
family={Naumann},
familyi={N\bibinitperiod},
given={Maximilian},
giveni={M\bibinitperiod}}}%
{{hash=e24d58fb6f2422a4d6642cf82bd2bb1c}{%
family={Kummerle},
familyi={K\bibinitperiod},
given={Julius},
giveni={J\bibinitperiod}}}%
{{hash=d5cafd4f82634a9c374915fddb2fa80d}{%
family={Konigshof},
familyi={K\bibinitperiod},
given={Hendrik},
giveni={H\bibinitperiod}}}%
{{hash=ca1de2d211c8e4a0f976b3b90a625cb8}{%
family={Stiller},
familyi={S\bibinitperiod},
given={Christoph},
giveni={C\bibinitperiod}}}%
{{hash=e1b899aa831b0ed225393f7ead37e614}{%
family={La\bibnamedelima Fortelle},
familyi={L\bibinitperiod\bibinitdelim F\bibinitperiod},
given={Arnaud},
giveni={A\bibinitperiod},
prefix={de},
prefixi={d\bibinitperiod}}}%
}
\strng{namehash}{a5fc35210ab51e876c6796237c46d871}
\strng{fullhash}{3b52e88b0bfaeedb6d42dd889cf796af}
\strng{bibnamehash}{3b52e88b0bfaeedb6d42dd889cf796af}
\strng{authorbibnamehash}{3b52e88b0bfaeedb6d42dd889cf796af}
\strng{authornamehash}{a5fc35210ab51e876c6796237c46d871}
\strng{authorfullhash}{3b52e88b0bfaeedb6d42dd889cf796af}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{arXiv preprint arXiv:1910.03088}
\field{title}{Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps}
\field{year}{2019}
\endentry
\enddatalist
\endrefsection
\endinput