Satisficing Paths and Independent Multi-Agent Reinforcement Learning in Stochastic Games
Extracted source text ยท arXiv 2110.04638
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2020
% ready for submission
% \usepackage{neurips_2020}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
%\usepackage[preprint]{neurips_2020}
% to compile a camera-ready version, add the [final] option, e.g.:
%\usepackage[final]{neurips_2020}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2020}
\usepackage[nonatbib,preprint]{neurips_2020}
\usepackage[normalem]{ulem}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{amsmath}
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
%\usepackage{algorithm}
\usepackage{sgame}
\usepackage{upgreek}
\usepackage{bm}
\usepackage{amsmath,amssymb,amsfonts}
\renewcommand{\P}{{\rm{Pr}}}
\newcommand{\FF}{\mathcal{F}}
\newcommand{\BB}{\mathcal{B}}
\newcommand{\NN}{\mathcal{N}}
\newcommand{\PP}{\mathcal{P}}
\newcommand{\nn}{\mathbb{N}}
\newcommand{\zz}{\mathbb{Z}}
\newcommand{\xx}{\mathbb{X}}
\newcommand{\uu}{\mathbb{U}}
\newcommand{\ww}{\mathbb{W}}
\newcommand{\rr}{\mathbb{R}}
\newcommand{\ii}{\mathbb{I}}
%\renewcommand{\zz}{\mathbb{Z}}
\newcommand{\zzz}{\mathbb{Z}_{\geq 0 }}
\newcommand{\bpi}{\bm{\pi}}
\newcommand{\bPi}{\bm{\Pi}}
\newcommand{\U}{\textbf{U}}
\newcommand{\bu}{\textbf{u}}
\newcommand{\ba}{\textbf{a}}
\newcommand{\bgamma}{\bm{\gamma}}
\newcommand{\bDelta}{\bm{\Delta}}
\newcommand{\eq}{\bm{\Delta}_{\rm eq}}
\newcommand{\BR}{{\rm BR}}
\newcommand{\All}{{\rm All}}
\newcommand{\Each}{{\rm Each}}
\newcommand{\Qcorrect}{\textrm{Q-Correct}}
\newcommand{\uniform}{\text{Unif}}
\newcommand{\phat}{\hat{p}_{\min}}
%\usepackage{ulem} %N.B: this changes the \emph{} and \em commands to underlines instead of italics
\usepackage[dvipsnames]{xcolor}
\newcommand{\by}[1]{{\color{red}#1}}
\newcommand{\green}[1]{{\color{ForestGreen}\bf#1}}
\newcommand{\sy}[1]{{\color{magenta}#1}}
%\usepackage{pdfsync}
\newcommand{\ga}[1]{{\color{blue}#1}}
\usepackage{graphicx}
%\usepackage[ruled,vlined]{algorithm2e} % looks a lot like the TAC submission's formatting
%\usepackage[linesnumbered,boxruled,noend]{algorithm2e} %see section 7, esp subsection 7.3, from the algorithm2e ctan page for details. i like boxruled, but ruled, boxed are other options.... option longend prints stuff like endfor, option shortend (default) prints only end, and option noend doesn't print an end. i like this one, personally.
\usepackage[linesnumbered, ruled,noend]{algorithm2e}
\newtheorem{lemma}{Lemma}
\newtheorem{proof}{Proof}
\newtheorem{definition}{Definition}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{condition}{Condition}
\newtheorem{assumption}{Assumption}
\newtheorem{remark}{Remark}
%\usepackage{graphicx} %i'm not sure if this will be the right way to put in figures...
\title{An Independent Learning Algorithm for a Class of Symmetric Stochastic Games}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Bora Yongacoglu%\thanks{Use footnote for providing further information
% about author (webpage, alternative address)---\emph{not} for acknowledging
%funding agencies.} \\
\\
Department of Mathematics and Statistics\\
Queen's University\\
Kingston, ON K7L 3N6 \\
\texttt{1bmy@queensu.ca} \\
% examples of more authors
\And
G\"urdal Arslan \\
Department of Electrical Engineering \\
University of Hawaii at Manoa \\
Honolulu, HI 96822 \\
\texttt{gurdal@hawaii.edu} \\
\And
Serdar Y\"uksel \\
Department of Mathematics and Statistics\\
Queen's University\\
Kingston, ON K7L 3N6 \\
\texttt{yuksel@queensu.ca} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}% <- trailing '%' for backward compatibility of .sty file
In multi-agent reinforcement learning, independent learners are those that do not access the action selections of other learning agents in the system. This paper investigates the feasibility of using independent learners to find approximate equilibrium policies in non-episodic, discounted stochastic games. We define a property, here called the $\epsilon$-revision paths property, and prove that a class of games exhibiting symmetry among the players has this property for any $\epsilon \geq 0$. Building on this result, we present an independent learning algorithm that comes with high probability guarantees of approximate equilibrium in this class of games. This guarantee is made assuming symmetry alone, without additional assumptions such as a zero sum, team, or potential game structure.%
%Edit this later.
\end{abstract}
\section{Introduction } \label{sec:intro}
%\section{Introduction \green{(Uncomment \LaTeX for roadmap)}} \label{sec:intro}
%
%\green{
%\footnotesize
%Section \ref{sec:intro}, on p. \pageref{sec:intro}:
%\begin{itemize}
% \item Set-up
% \item \S \ref{ss:literature} describes related literature.
%\end{itemize}
%
%
%\noindent Section \ref{sec:model}, on p. \pageref{sec:model}:
%\begin{itemize}
% \item \green{\S \ref{ss:model} is the model, definitions of best-responding and equilibrium. }
% \item \green{\S \ref{ss:symmetric-games} defines symmetric games. }
% \item \green{\S \ref{ss:learning-in-MDPs} and \S \ref{ss:learning-in-games} cover Q- and J-learning in MDPs and Games, respectively. }
%\end{itemize}
%
%\noindent Section \ref{sec:epsilon-paths}, on p. \ref{sec:epsilon-paths}:
%\begin{itemize}
% \item \green{Def $\epsilon$-revision paths; proof that symmetric games have $\epsilon$-paths property $\forall \epsilon \geq 0$. }
%\end{itemize}
%
%\noindent Section \ref{sec:approx-results}, on p. \pageref{sec:approx-results}:
%\begin{itemize}
% \item \green{\S \ref{ss:quantized-policies} describes quantizing the policy set; }
% \item \green{\S \ref{ss:perturbed policies} describes perturbed policies and gives the lemmas from AY2017}
%\end{itemize}
%
%\noindent Section \ref{sec:paradigm}, on p. \pageref{sec:paradigm}:
%\begin{itemize}
% \item Presents algorithmic ideas, compares and contrasts to our previous work,
% \item \by{This part needs your attention and revising, I think.}
%\end{itemize}
%
%\noindent Section \ref{sec:main}, on p. \pageref{sec:main}:
%\begin{itemize}
% \item Presents the algorithm and main result.
%\end{itemize}
%
%\noindent Sections 1, 7, and 8 are not included:
%\begin{itemize}
% \item Section \ref{sec:discussion} discusses the significance of the results of this paper for mean field teams;
% \item Section \ref{sec:simulations} will be a simulation study;
% \item Section \ref{sec:conclusion} concludes
%\end{itemize}
%
%\noindent Appendices present proofs.
%}
%\newpage
%\begin{itemize}
% \item Brief description of existing literature
% \item Why it matters for ML
% \item Why it's hard (can't exhaustively search the policy space, even when quantizing, because I can't see what you're doing.)
% \item Description of our result
% \item Roadmap for paper
%\end{itemize}
%
%\vspace{11pt}
%\textbf{I think the paragraph below has been superseded later on down.}
%
%The remainder of this paper is organized as follows. The model and necessary background on multi-agent learning in stochastic games are given in Section \ref{sec:model}. In Section \ref{sec:paradigm}, we present our algorithmic approach, which decouples the tasks of learning and adaptation. This section also compares our paradigm with existing works. Section \ref{sec:main} presents the main algorithm and results. Some simulations are included in Section \ref{sec:simulations}, and the final section concludes. Proofs and supporting technical material are included in the appendices.
%
%\vspace{11pt}
%\by{\textit{I will insert a notation paragraph later.}}
Reinforcement learning (RL) algorithms use experience and feedback information to improve one's performance in a control task. In recent years, the field of RL has advanced tremendously both in terms of fundamental theoretical contributions and successful applications. With improvements in computer hardware, these advances have lead to the deployment of RL algorithms in large scale engineering systems in which many learning agents act, observe, and learn in a shared environment. Multi-agent reinforcement learning (MARL) is the study of emergent behaviour in such complex, strategic environments, and is one of the important frontiers in modern artificial intelligence and automatic control research.
The literature on MARL is relatively small when compared to that of single-agent RL, and this owes largely to the inherent challenges of multi-agent settings: decentralized information and non-stationarity. The first such challenge is that some relevant information will be unavailable to some of the players. This may occur due to strategic considerations, as competing agents may wish to hide their actions or knowledge from their rivals, or it may occur simply because of obstacles in communicating, observing, or storing large quantities of information in decentralized systems.
The second challenge inherent to MARL comes from the non-stationarity of the environment from the point of view of any individual agent \cite{hernandez2017survey}. As one agent learns how to improve its performance, it will alter its behaviour. This can have a destabilizing effect on the learning processes of the remaining agents, who may change their policies in response to outdated strategies. Notably, this issue arises when one tries to apply single-agent RL algorithms---which typically rely on state-action value estimates or gradient estimates that are made using historical data---in multi-agent settings. A number of studies have reported non-convergent play when single-agent algorithms using local information are employed (without modification) in multi-agent settings, e.g. \cite{tan1993multi}, \cite{Claus1998}.
Designing decentralized learning algorithms with desirable convergence properties is a task of great practical importance that lies at the intersection of the two challenges above. %While appropriate notions of decentralization and desirable convergence behaviour may depend on the application, \by{this task has attracted considerable research attention in recent years \cite{whatever we want}.}
The notion of decentralization considered in this paper involves agents that observe a global state variable but do not observe the actions of other agents. Learning algorithms suitable for this information structure are called \textit{independent learners} in the machine learning literature \cite{Claus1998, zhang2021multi, matignon2012survey, matignon2009coordination, daskalakis2021independent, wei2016lenient}; they have also been called \textit{payoff-based} and \textit{radically uncoupled} in the control and game theory literatures, respectively, \cite{ Marden2009payoff, marden2012revisiting, foster2006regret}.
In this paper, our goal is to provide a MARL algorithm that is suitable for independent learners in a complex system, requires little coordination among agents, and comes with provable guarantees for its long-run performance. For our theoretical framework, we consider many player symmetric stochastic games with a discounted cost criterion. In this setting, we propose an independent learning algorithm and prove that, under mild conditions, this algorithm leads to approximate equilibrium policies in self-play.
The algorithm presented in this paper builds on the exploration phase technique of \cite{AY2017} for policy evaluation, but differs considerably in how players update their policies and explore their policy spaces. Here, players discretize their policy space with a uniform quantizer and use an ``$\epsilon$-satisficing'' rule to explore this quantized set, occasionally using random search when unsatisfied. Our proof of convergence exploits a structural property enjoyed by all symmetric games (defined formally in \S\ref{ss:symmetric-games}), and this allows us to avoid assuming additional structure in the game, such as the existence of a potential function or zero-sum costs.
\vspace{5.5pt}
\noindent \textbf{Contributions:}
\begin{itemize}
\item[(i)] For any stochastic game and $\epsilon \geq 0$, we define \textit{$\epsilon$-revision paths} and a related \textit{$\epsilon$-revision paths property};
\item[(ii)] In Theorem \ref{thm:symmetric-paths}, we prove that all symmetric games %\footnote{\by{It might be more accurate to name the class of games in Def \ref{def:symmetric game} something like ``doubly symmetric games''.}}
(as defined in Definition \ref{def:symmetric game}) have the $\epsilon$-revision paths property, for all $\epsilon \geq 0$;
\item[(iii)] In Algorithm~\ref{algo:main}, we provide an algorithm for playing symmetric stochastic games and, in Theorem \ref{thm:main}, we prove that self-play drives the baseline policy to $\epsilon$-equilibrium.
\end{itemize}
\subsection{Related Work} \label{ss:literature}
The study of learning in games, beginning with Brown's fictitious play algorithm \cite{Brown1951iterative} and Robinson's analysis thereof \cite{Robinson1951iterative}, is nearly as old as game theory itself. There is a large literature on fictitious play and its variants, with most works in this line considering a different information structure than the decentralized one studied here. The bulk of work on fictitious play focuses on settings with perfect monitoring of the actions of other players, but some recent works consider various decentralized information structures, e.g. \cite{ swenson2018distributed, eksin2017distributed}.
A number of empirical works in the 1990s studied the behaviour resulting from independent RL agents coexisting in various shared environments, e.g. \cite{tan1993multi, sen94, Claus1998}. Contemporaneously, \cite{Littman1994} popularized using stochastic games as the framework for MARL. Several \textit{joint action learners} (learners that require access to the past actions of all other agents) were then proposed for playing stochastic games and proven to converge to equilibrium under various assumptions. A representative sampling of this stream of algorithms includes the Minimax Q-learning algorithm of \cite{Littman1994}, the Nash Q-learning algorithm of \cite{Hu2003}, and the Friend-or-Foe Q-learning algorithm of \cite{Littman2001ffq}.
Early work on independent learners includes the following: \cite{Claus1998}, which popularized the terminology of joint action learners and independent learners and stated conjectures; \cite{lauer2000}, which presented an independent learner for teams with deterministic state transitions and cost realizations and proved its convergence to optimality in that (restricted) setting; and \cite{Bowling}, which proposed the WoLF-Policy Hill Climbing algorithm for general-sum stochastic games and conducted simulation studies.
Due in part to the challenges posed by non-stationarity and decentralized information, most contributions to the literature on independent learners focused either on the stateless case of repeated games and produced formal guarantees, such as the works of \cite{foster2006regret, germano2007global, Chasparis2013aspiration, Marden2009payoff, marden2012revisiting, marden2014achieving}, or otherwise studied the stateful setting and presented only empirical results, such as the works \cite{matignon2007hysteretic, matignon2009coordination, wei2016lenient}.
More recently, there has been a number of papers that study independent learners for games with non-trivial state dynamics while still presenting rigorous guarantees. These include \cite{daskalakis2021independent} and \cite{sayin2021decentralized}, which study two-player, zero-sum games, and \cite{YAY-TAC} and \cite{AY2017}, which study teams, potential games, and their generalizations. In each of those works, the algorithm studied takes advantage of structural properties of the games considered.
In this paper, we study symmetric stochastic games with $N \geq 2$ players. This class of games is distinct from the classes listed above: an $N$-player symmetric stochastic game need not be a zero-sum game, admit a potential function, or be weakly acyclic under best-replies. As such, existing algorithms such as those studied in \cite{daskalakis2021independent, sayin2021decentralized, YAY-TAC} and \cite{AY2017} do not come with convergence guarantees when applied to symmetric stochastic games. Nevertheless, games in this class exhibit structural properties that do not require further assumptions on the cost function. This paper shows how to exploit this structure.
\subsection*{Organization}
The remainder of the paper is organized as follows: Section \ref{sec:model} describes the stochastic games model and covers background results on Q-learning and value learning; Section \ref{sec:epsilon-paths} defines $\epsilon$-revision paths and proves that symmetric games have the $\epsilon$-revision paths property for all $\epsilon \geq 0$. A number of approximation results, concerning quantization of the policy sets and perturbations of policies, are presented in Section \ref{sec:approx-results}. The architecture and paradigm of the main algorithm is presented in Section \ref{sec:paradigm}. Our algorithm and main theorem is presented in Section \ref{sec:main}. A simulation study is summarized in Section~\ref{sec:simulation}. Some discussion about the limitations and consequences of this work is provided in Section~\ref{sec:discussion}. The final section concludes. All proofs omitted from the body of the text can be found in the technical appendices.
\subsection*{Notation} \label{ss:notation}
$\mathbb{R}$ denotes the real numbers, $\zz_{\geq 0}$ and $\mathbb{N}$ denote the nonnegative and positive integers, respectively. $\P(\cdot)$ and $E(\cdot)$ denote the probability and the expectation, respectively. For a finite set $S$, $\mathcal{P}(S)$ denotes the set of probability distributions over $S$. For finite sets $S, S'$, we let $\mathcal{P} ( S' | S)$ denote the set of stochastic kernels on $S'$ given $S$. An element $\mathcal{T} \in \mathcal{P} ( S' | S)$ is a collection of probabilities distributions on $S'$, with one distribution for each $s \in S$, and we write $\mathcal{T} ( \cdot | s )$ for $s \in S$ to make this distributional dependence on $s$ explicit. We write $Y \sim f$ to denote that the random variable $Y$ has distribution $f$. If the distribution of $Y$ is a mixture of other distributions, say with mixture components $f_i$ and weights $p_i$ for $1 \leq i \leq n$, we write $Y \sim \sum_{i = 1}^n p_i f_i$. The Dirac distribution concentrated at $x \in \mathbb{R}$ is denoted $\mathbb{I}_x$. For a finite set $S$, $\uniform(S)$ denotes the uniform distribution over $S$ and $2^S$ denotes the set of subsets of $S$. %$(x)^+:=\max\{x,0\}$, for $x\in\mathbb{R}$.
\section{Model} \label{sec:model}
\subsection{Stochastic games with discounted costs} \label{ss:model}
A finite, discounted stochastic game $G$ is described by the list
\begin{equation} \label{def:stochasticGame}
G = (\NN, \xx, \{ \mathbb{U}^i \}_{i \in \NN }, \{ c^i \}_{i \in \NN }, \{ \beta^i \}_{i \in \NN}, P, \nu_0 ).
\end{equation}
The components of $G$ are the following: $\NN$ is a finite set of $N \in \nn$ players/agents. $\xx$ is a finite set of states. For agent $i \in \NN$, $\uu^i$ is a finite set of actions, and we write $\U := \times_{i \in \NN} \uu^i$. An element $\bu \in \U$ is called a \emph{joint action}. For agent $i$, $c^i : \xx \times \U \to \rr$ is a stage cost function, and $\beta^i \in [0,1)$ is a discount factor. A random initial state $x_0 \in \xx$ is given by $x_0 \sim \nu_0$. $P$ is a Markov transition kernel, which describes state transition probabilities through equation (\ref{eq:transKer}), below.
At time $t \in \zzz$, the state variable is denoted by $x_t \in \xx$, and the action selected by agent $i \in \NN$ is denoted by $u^i_t \in \uu^i$, while the joint action is denoted by $\bu_t = (u^i_t)_{i \in \NN} \in \U$. For all $t \in \zzz$, the state process $\{x_t\}_{t \geq 0}$ evolves in a Markovian fashion according to \eqref{eq:transKer}:
\begin{align} \label{eq:transKer}
&\P ( x_{t+1} = x' \vert x_k = s_k , \bu_k = \ba_k, 0 \leq k \leq t ) \nonumber \\
= &P ( x' \vert s_t, \ba_t ), \quad \forall x' , s_0, \dots, s_t \in \xx, \; \ba_0, \dots, \ba_t \in \U.
\end{align}
A policy for agent $i \in \NN$ is a rule for selecting a sequence of actions $(u^i_t)_{t \geq 0}$ based on information that is locally available at the time of each decision. The action $u^i_t \in \uu^i$ is chosen according to a (possibly random) function of agent $i$'s observations up to time $t$. In this paper, we focus on \textit{independent learners}, which are agents that do not use/cannot access the complete joint action $\bu_t$ for any time $t$. Instead, at time $t$ an independent learner may use only the history of observations of states ($x_0, \dots, x_t$), its own local actions ($u^i_0, \dots, u^i_{t-1}$), and its numerical cost realizations.\footnote{Agent $i$ does not know the function $c^i$ but observes the scalars $c^i ( x_0, u^i_0, \bu^{-i}_0), \dots, c^i ( x_{t-1}, u^i_{t-1} , \bu^{-i}_{t-1})$.} Independent learners are contrasted with \textit{joint action learners}, which are learners that have access also to the actions of other agents \cite{Claus1998}.
The set of stationary policies for player $i$ is identified with the set $\Delta^i := \PP (\uu^i | \xx) $ of probability distributions on $\uu^i$ given $\xx$. When agent $i$ uses policy $\gamma^i \in \Delta^i$, it selects its action $u^i_t \sim \gamma^i ( \cdot | x_t )$. For ease of notation, we denote the collected policies of all agents $j \not= i$ by using $-i$ in the agent index and we use boldface characters to refer to joint objects, e.g. $\bgamma^{-i} = (\gamma^j)_{j \not= i}$. We let $\bDelta^{-i}:=\times_{j\not=i} \Delta^j$ and $\bDelta:=\times_j \Delta^j$. Using these conventions, we can re-write a joint policy $(\gamma^1,\dots, \gamma^N)$ as $(\gamma^{i},\bgamma^{-i})$, a joint action can be re-written as $\bu = (u^i , \bu^{-i})$, and so on.
Given a joint policy $\bgamma$, we use $\P^{\bgamma}$ to denote the resulting probability measure on trajectories $\{ (x_t, u_t) \}_{t \geq 0}$ and we use $E^{\bgamma}$ to denote the associated expectation. The objective of agent $i \in \NN$ is to find a policy that minimizes the expectation of its series of discounted costs, given by
\begin{equation} \label{eq:discountedCost}
J^i_{x} (\gamma^i, \bgamma^{-i}) = E^{\bgamma} \left[ \sum_{t \geq 0} (\beta^i)^t c^i\left(x_t,u^i_t, \bu^{-i}_t \right) \middle| x_0 = x \right]
\end{equation}
for all $x \in \xx$. Note that agent $i$ controls only its own policy, $\gamma^i$, but its cost is affected by the actions of the remaining agents. Since agents have possibly different cost functions, we adopt the notion of equilibrium to represent those policies that are \textit{person-by-person optimal} and stationary.
\begin{definition}
Let $i \in \NN$, $\epsilon \geq 0$. For $\bgamma^{-i} \in \bDelta^{-i}$, a policy $\gamma^{*i} \in \Delta^i$ is called an $\epsilon$-\emph{best-response} to $\bgamma^{-i}$ if
\begin{equation} \label{def:BR}
J_x^i( \gamma^{*i}, \bgamma^{-i} ) \leq \inf_{\pi^i\in \Delta^i} J_x^i(\pi^{i},\bgamma^{-i}) + \epsilon, \quad \forall x \in \xx.
\end{equation}
\end{definition}
\begin{definition}
A joint policy $\bgamma^* = (\gamma^{*i})_{i \in \NN} \in \bDelta$ constitutes a (Markov perfect) $\epsilon$-equilibrium if $\gamma^{*i}$ is an $\epsilon$-best-response to $\bgamma^{*-i}$ for each agent $i$.
%\begin{equation} \label{def:equilibrium}
%J_x^i(\pi^{*i},\pi^{*-i})=\min_{\pi^i\in\Delta^i}J_x^i(\pi^{i},\pi^{*-i}), \quad \forall x \in \xx, i \in \N.
%\end{equation}
\end{definition}
For the special case of $\epsilon = 0$, a $0$-best-response is simply called a best-response and a $0$-equilibrium is called an equilibrium. Let $\bDelta^{\epsilon}_{\rm eq}$ denote the set of $\epsilon$-equilibrium policies, for $\epsilon \geq 0$. For any stochastic game, we have $\Delta^{0}_{\rm eq} \not= \varnothing$ (see \cite{fink1964equilibrium}), and since a 0-best-response is, \textit{a fortiori}, an $\epsilon$-best-response, this implies that $\Delta^{\epsilon}_{\rm eq} \not= \varnothing$.
%\subsection*{Temporary Subsection}
%
%I think I will change the order of presentation...
%\begin{enumerate}
% \item Description of algorithmic ideas: (maybe with a block/flow diagram?)
% \begin{itemize}
% \item two timescales to decouple the learning and adapting
% \item An adaptation mechanism (e.g. inertial best-responding, e.g. policy gradient, e.g. random search) for updating policies
% \end{itemize}
%\end{enumerate}
\subsection{Symmetric Games} \label{ss:symmetric-games}
In some applications, the strategic environment being modelled exhibits symmetry in the agents. %(\by{mean field games?}).
To model such settings, we define a class of symmetric games with the following properties: (1) each agent has access to the same set of actions; (2) the state dynamics depend only on the profile of actions taken by all players, without special dependence on the identities of the agents. That is, permuting the agents' actions in a joint action leaves the conditional probabilities for the next state unchanged; (3) such a permutation results in a corresponding permutation of costs incurred. We formalize and clarify these points in the definition below. First, we introduce additional notation: if $\uu^i = \uu^j$ for all $i,j \in \NN$, given a permutation $\sigma : \NN \to \NN$ and joint action $\ba = (a^i)_{i \in \NN}$, we define $\sigma( \ba) \in \U$ to be the joint action in which $i$'s component is given by $ a^{\sigma(i)}$. That is, player $i$'s action in $\sigma(\ba)$ is given by player $\sigma(i)$'s action in $\ba$.
\begin{definition} \label{def:symmetric game}
A discounted stochastic game $G$ is called \emph{symmetric} if the following holds:
\begin{itemize}
\item $\uu^i = \uu^j $ and $\beta^i = \beta^j$ for any $i, j \in \NN$;
\item For any $i \in \NN$, permutation $\sigma$ on $\NN$, and $\ba \in \U$, we have
\[
c^i ( x, \sigma( \ba) ) = c^{\sigma(i)} ( x, \ba ) , \quad \text{and} \quad P \left( \cdot | x, \ba \right) = P \left( \cdot | x , \sigma(\ba) \right).
\]
\end{itemize}
\end{definition}
Observe the following useful fact about symmetric games:
\begin{lemma} \label{lemma:symmetric-equality}
Let $G$ be a symmetric game and let $\bgamma$ be a joint policy. For $i, j \in \NN$, if $\gamma^i = \gamma^j$, then
\[
i \text{ is $\epsilon$-best-responding to } \bgamma^{-i} \iff j \text{ is $\epsilon$-best-responding to } \bgamma^{-j}
\]
\end{lemma}
\textbf{Proof: } By symmetry, $\bgamma^{-i} = \bgamma^{-j}$ and the proof is immediate. \hfill $\square$
\subsection{Learning in MDPs} \label{ss:learning-in-MDPs}
In online independent learning, pertinent information for policy updating is not available to the players. Player $i$ does not know the policy used by players $-i$, the value of its current policy against those of the other players, or whether its current policy is an $\epsilon$-best-response. We now recall some background on Q-learning and summarize how it can be used to address these uncertainties.
% In the following sections, we will present algorithms for playing a stochastic game. In these algorithms, a given player $i$ will play according to a policy for some period of time, then will revise its policy to improve its performance. The specific mechanism by which player $i$ updates its policy will center on whether $i$ is $\epsilon$-best-responding or not. As the base skeleton for our algorithm, we will use a simple policy update rule of the form ``stay if satisfied; change if unsatisfied." In the online, independent learning applications of interest, player $i$ cannot directly access $J^i_x ( \gamma^i, \bgamma^{-i})$ for several reasons: first, $\bgamma^{-i}$ is unknown; second the cost function $c^i$ and state dynamics are unknown, so computing the expectation is not possible. As a result, $i$ cannot determine whether it is $\epsilon$-best-responding at any given time. We now present Q-learning, which we use as a workaround to this obstacle.
Markov decision processes (MDPs) can be viewed as a stochastic game with one player, i.e. $\vert \NN \vert = 1$. In standard Q-learning \cite{Watkins89}, a single agent interacts with its MDP environment using some policy and maintains a vector of Q-factors, the $t^{th}$ iterate denoted $Q_t \in \rr^{\xx \times \uu}$. Upon selecting action $u_t$ at state $x_t$ and observing the subsequent state $x_{t+1} \sim P ( \cdot | x_t, u_t) $ and cost $c ( x_t, u_t)$, the Q-learning agent updates its Q-factors as follows:
\begin{align} \label{eq:q-factors}
Q_{t+1} ( x_t, u_t ) = Q_t ( x_t, u_t ) + \theta_t (x_t, u_t ) \left( c (x_t, u_t ) + \beta \min_{a \in \uu} Q_t ( x_{t+1} , a ) - Q_t ( x_t, u_t ) \right)
\end{align}
where $\theta_t (x_t, u_t) \in [0,1]$ is a random step-size parameter and $Q_{t+1} ( s, a) = Q_t (s,a)$ for all $(s,a) \not= (x_t, u_t)$.\footnote{We are interested in the online variant, where access to the state, action, and cost feedback arrive piece-by-piece as the agent interacts with its environment. This is in contrast to some studies that update multiple entries of the vector of Q-factors at each time.}
Under mild conditions, $Q_t \to Q^*$ almost surely as $t \to \infty$, where $Q^* \in \rr^{\xx \times \uu}$ is called the optimal Q-factors\footnote{$Q^*$ is also called the state-action value function and the action value function.} \cite{WatkinsDayan92}, \cite{tsitsiklis1994asynchronous}. The value $Q^*(x,u)$ represents the expected discounted cost-to-go from the initial state $x$, assuming that the agent initially chooses action $u$ and follows an optimal policy thereafter. The vector $Q^*$ can be used to construct an optimal policy $\pi^*$ by selecting
\[
\pi^* ( x) \in \left\{ u \in \uu : Q^* ( x, u ) = \min_{a \in \uu} Q^* (x, a) \right\}.
\]
\subsection{Learning in Stochastic Games} \label{ss:learning-in-games}
In the single-agent literature, the MDP is fixed and the $Q^*$ notation is used, but in principle one could introduce notation to specify the relevant MDP. Returning to the game setting, if all agents except $i$ follow a stationary policy $\bgamma^{-i} \in \bDelta^{-i}$, agent $i$ faces an environment that is equivalent to an MDP that depends on $\bgamma^{-i}$. We denote agent $i$'s $t^{th}$ Q-factor iterate by $Q^i_t$ and $i$'s optimal Q-factors when playing against $\bgamma^{-i}$ by $Q^{*i}_{\bgamma^{-i}} \in \rr^{\xx \times \uu^i}$. With this notation, $Q^{*i}_{\bgamma^{-i}}(x,u^i)$ represents agent $i$'s expected discounted cost-to-go from the initial state $x$ assuming that agent $i$ initially chooses $u^i$ and uses an optimal policy thereafter while the other agents use $\bgamma^{-i}$, a fixed stationary policy. We note that an optimal policy for $i$ is guaranteed to exist since $i$ faces a finite, discounted MDP, and that any optimal policy for $i$ in this MDP is a $0$-best-response to $\bgamma^{-i}$ in the underlying game $G$. More generally, we have the following fact, which will be leveraged in the design of our main algorithm in Section \ref{sec:main}. For any $i \in \NN$, $\bgamma^{-i} \in \bDelta^{-i}$,
\[
\pi^i \text{ is an $\epsilon$-best-response to } \bgamma^{-i} \iff J^i_x ( \pi^i , \bgamma^{-i} ) \leq \min_{u^i \in \uu^i } Q^{*i}_{\bgamma^{-i}} ( x, u^i ) + \epsilon , \quad \forall x \in \xx.
\]
%This fact is useful for estimating when an independent learning agent is $\epsilon$-best-responding to a stationary opponent. Our main algorithm, presented in Section \ref{sec:main}, leverages this fact. \green{Re-wording nec.}
\section{$\epsilon$-revision processes in games} \label{sec:epsilon-paths}
%\subsection{A common feature of many decision-making paradigms}
The idea of ``satisficing'' refers to becoming satisfied and halting search when a sufficiently good input has been found in an optimization problem \cite{simon1956rational}. Satisficing has a long history in both single-agent decision theory (e.g. \cite{radner1975satisficing}, \cite{cassidy1972solution}) and also multi-agent game theory (e.g. \cite{charnes1963deterministic}). Recently, there has been some interest in studying learning dynamics in games where agents only change their policy when they are not $\epsilon$-best-responding. For example, see \cite[Section 5]{candogan2013near}. Other works in the same vein include the aspiration learning algorithms of \cite{Chasparis2013aspiration} and \cite{YAY-TAC}.
It is natural to ask the following: what assumptions must be made on a game in order to guarantee that some type of satisficing dynamics can lead to $\eq^{\epsilon}$? With this question in mind, we give our next definition.
\begin{definition}
A (possibly finite) path of joint policies $( \bgamma_k )_{k \geq 0}$ is called an \emph{$\epsilon$-revision path} if, for every $k \geq 0$, $\gamma^i_k \in \BR^i_{\epsilon} ( \bgamma^{-i}_k )$ implies $\gamma^i_{k+1} = \gamma^i_k$.
\end{definition}
\begin{definition}
For $\epsilon \geq 0$, a game $G$ is said to have the $\epsilon$-revision paths property if for every $\bgamma_0 \in \bDelta$, there exists an $\epsilon$-revision path of finite length, say $\bgamma_0, \bgamma_1, \dots, \bgamma_L$, such that $\bgamma_L \in \eq^{\epsilon}$.
\end{definition}
We note that the definitions above are not attached to any particular dynamical system on $\bDelta$. They do not require that a player must switch a best-response when not already $\epsilon$-best-responding. As such, one may (loosely) interpret the $\epsilon$-revision paths property as a necessary condition for convergence to $\eq^{\epsilon}$ when players employ an arbitrary $\epsilon$-satisficing rule for updating their policy. (We say loosely here because you may fail to satisfy the $\epsilon$-revision paths property but still admit \textit{infinite} sequences converging to $\eq^{\epsilon}$.)
It is easy to see that teams and exact potential games both have the $\epsilon$-revision paths property for any $\epsilon \geq 0$. In this paper, we choose to not focus on these classes of games, as there are a number of existing independent learning algorithms that take advantage of some structural properties of games in those classes that do not hold for symmetric games in general. For a sampling of some such algorithms, we cite \cite{YAY-TAC} and \cite{AY2017} for weakly acyclic games and teams; for two-player, zero-sum stochastic games we cite \cite{daskalakis2021independent} and \cite{sayin2021decentralized}; and for potential games we cite \cite{zhang2021gradient}.
\begin{theorem} \label{thm:symmetric-paths}
If $G$ is a symmetric game, then $G$ has the $\epsilon$-paths property for all $\epsilon \geq 0$.
\end{theorem}
\textbf{Proof:} We prove the claim by explicitly constructing a valid $\epsilon$-revision path into $\eq^{\epsilon}$. Intuitively, beginning from an arbitrary policy, unsatisfied players (i.e. players not $\epsilon$-best-responding) can change policies to match the policy of other players. We create a cohort of players using the same policy and progressively grow the cohort until either we stop because we have found an $\epsilon$-equilibrium or because no player is satisfied, which allows us to move in one step to an arbitrary $\epsilon$-equilibrium.
Let $\bpi_0 \in \bDelta$ be an initial policy. Let $S_0 := \left\{ i \in \NN : \pi^i_0 \notin \BR^i_{\epsilon} ( \bpi^{-i}_0 ) \} \right\}$ be the set of players not $\epsilon$-best-responding at $\bpi_0$. If $S_0 = \varnothing$ then $\bpi_0 \in \eq^{\epsilon}$ and $(\bpi_0)$ is a valid $\epsilon$-revision path from $\bpi_0$ into $\eq^{\epsilon}$. If $S_0 = \NN$, then $( \bpi_0, \bpi^*)$ with any $\bpi^* \in \eq^0 \subset \eq^{\epsilon}$ is a valid $\epsilon$-revision path.
Suppose now that $1 \leq | S_0 | \leq N-1$. Select a distinguished player $j(1) \notin S_0$, and construct a successor policy $\bpi_1$ as follows:
\[
\pi^i_1 := \begin{cases}
\pi^i_0 , &\text{if } i \notin S_0, \\
\pi^{j(1)}_0, &\text{if } i \in S_0 .
\end{cases}
\]
Now, define $S_1 := \left\{ i \in \NN : \pi^i_1 = \pi^{j(1)}_0 \right\}$ to be the set of all players whose policy matches $j(1)$'s policy under $\bpi_1$, and note that $S_0 \cup \{ j(1) \} \subseteq S_1$. We have thus constructed a valid $\epsilon$-revision path $(\bpi_0 ,\bpi_1)$ and a sequence of sets $(S_0, S_1)$ such that the following three properties hold for $k = 1$:
\begin{itemize}
\item[(I)] All players in $S_k$ use the same policy, i.e. $\pi^i_k = \pi^j_k$ for all $i, j \in S_k$;
\item[(II)] $|S_k | \geq |S_{k-1} | + 1$.
\item[(III)] If player $j \notin S_k$, then $\pi^j_k \not= \pi^i_k$ for any $i \in S_k$.
\end{itemize}
% Having established a base case, we now proceed by induction on $k$.
For $n \geq 1$, suppose $(\bpi_0, \bpi_1, \dots, \bpi_n)$ is a valid $\epsilon$-revision path, and suppose $S_0, S_1, \dots, S_n$ is a sequence of subsets of $\NN$ such that for any $k \in \{1, \dots, n\}$ we have (I) $\pi^i_k = \pi^j_k$ for any $i, j \in S_k$; (II) $| S_{k} | \geq |S_{k-1} | + 1$; %(Note that this implies that $|S_n| \geq |S_0| + n \geq 1 + n$, as we have assumed that $1 \leq |S_0 | \leq N-1$.)
(III) if $i \in S_k, j \notin S_k$, then $\pi^i_k \not= \pi^j_k$.
If $\bpi_n \in \eq^{\epsilon}$, then $(\bpi_0 ,\dots, \bpi_n )$ is an $\epsilon$-revision path from $\bpi_0$ into $\eq^{\epsilon}$. On the other hand, if $\bpi_n \notin \eq^{\epsilon}$, we proceed in cases.
\underline{Case 1:} ($|S_n | = N$ and $\bpi_n \notin \eq^{\epsilon}$) %By Lemma \ref{lemma:symmetric-equality}, either all agents are $\epsilon$-best-responding at $\bpi_n$---and so $\bpi_n \in \eq^{\epsilon}$, which is not true in Case 1---or no agent is $\epsilon$-best-responding at $\bpi_n$. Focusing on the latter case, for any $\bpi^* \in \eq^{\epsilon}$, we have that $(\bpi_0, \dots, \bpi_n, \bpi^* )$ is a valid $\epsilon$-revision path from $\bpi_0$ into $\eq^{\epsilon}$.
By (I), $\pi^i_n = \pi^j_n$ for ever $i, j \in S_n = \NN$. By Lemma~\ref{lemma:symmetric-equality}, no agent is $\epsilon$-best-responding at $\bpi_n$. (Otherwise some player is $\epsilon$-best-responding, and therefore all are and so $\bpi_n \in \eq^{\epsilon}$, which we have ruled out.) Then, for any $\bpi^* \in \eq^{\epsilon}$, we have that $(\bpi_0, \dots, \bpi_n, \bpi^* )$ is a valid $\epsilon$-revision path from $\bpi_0$ into $\eq^{\epsilon}$.
\underline{Case 2:} ($|S_n | \leq N -1$ and $\bpi_n \notin \eq^{\epsilon}$) Again by Lemma \ref{lemma:symmetric-equality}, either (2a) all agents in $S_n$ are $\epsilon$-best-responding at $\bpi_n$ or (2b) none are. We treat Case 2a first. Since $\bpi_n$ is not an $\epsilon$-equilibrium and each player in $S_n$ is $\epsilon$-best-responding, there must exist a player $j(n+1) \in \NN \setminus S_n$ such that $j(n+1)$ is not $\epsilon$-best-responding at $\bpi_n$, and we construct a policy $\bpi_{n+1}$ as
\[
\pi^i_{n+1} = \begin{cases}
\pi^i_n , &\text{if } i \not= j(n+1) \\
\pi^{s}_n &\text{if } i = j(n+1) ,
\end{cases}
\]
where $s \in S_n$ is any player in $S_n$. Thus, we have that (I) for all $i, j \in S_{n+1} := S_n \cup \{ j(n+1) \}$, $\pi^i_{n+1} = \pi^j_{n+1}$; (II) $| S_{n+1}| \geq |S_n | + 1$; and (III) if $j \notin S_{n+1}$, then $\pi^j_{n+1} \not= \pi^i_{n+1}$ for any $i \in S_{n+1}$. Note also that $( \bpi_0, \dots, \bpi_n, \bpi_{n+1})$ is a valid $\epsilon$-revision path.
\vspace{5.5pt}
In Case 2b, players in $S_n$ are not $\epsilon$-best-responding at $\bpi_n$. We select $j(n+1) \in \NN \setminus S_n$ and construct $\bpi_{n+1}$ as
\[
\pi^i_{n+1} := \begin{cases}
\pi^{j(n+1)}_n, &\text{if } i \in S_n , \\
\pi^i_n , &\text{if } i \notin S_n.
\end{cases}
\]
We define $S_{n+1} := \left\{ i \in \NN : \pi^i_{n+1} = \pi^{ j ( n+1)}_{n} \right\}$. Once again, properties (I)--(III) hold and $( \bpi_0, \dots, \bpi_n, \bpi_{n+1})$ is a valid $\epsilon$-revision path.
% \vspace{5.5pt}
Note that this process---of producing $S_{n+1}$ and $\bpi_{n+1}$ out of $S_n$ and $\bpi_n$---can be repeated only finitely many times before stopping. This is because $1 + n \leq |S_0| + n \leq |S_n| \leq N$, and so we are constrained to $n \leq N-1$. In every case, we use this process to produce an $\epsilon$-revision path of length at most $ N+1$ from $\bpi_0$ to $\eq^{\epsilon}$. \hfill $\square$
With this important structural property of symmetric games now established, we will proceed by developing algorithms for playing such games.
%\newpage
\section{Approximation results} \label{sec:approx-results}
\subsection{Quantized policies} \label{ss:quantized-policies}
For ease of analysis and for algorithm design, we will restrict player $i$'s policy selection from the uncountable set $\Delta^i$ to a finite subset $\Pi^i \subset \Delta^i$. The set $\Pi^i$ is obtained via uniform quantization of $\Delta^i$, and restriction to $\Pi^i$ is justified using the following result.
\begin{lemma} \label{lemma:continuous-cost} For every $i \in \NN, x \in \xx$, the cost functional $J^i_x : \bDelta \to \rr$ is continuous.
\end{lemma}
The proof of Lemma \ref{lemma:continuous-cost} can be found on page \pageref{pf:lemma-cost-cty}.
Since $\bDelta$ is compact, Lemma \ref{lemma:continuous-cost} implies that each cost functional $J^i_x$ is also uniformly continuous. From this we have that for any $\epsilon > 0$, there exists $\xi ( \epsilon ) > 0$ such that if two joint policies $\bgamma, \bpi$ are $\xi$-close,\footnote{That is, the distance $d ( \bgamma, \bpi) := \max \left\{ \left| \gamma^i ( a^i | s ) - \pi^i (a^i | s ) \right| : i \in \NN, a^i \in \uu^i, s \in \xx \right\} < \xi$} then $| J^i_x ( \bgamma ) - J^i_x ( \bpi) | < \epsilon$, for any $i \in \NN, x \in \xx$.
A quantization $\bPi$ of $\bDelta$ into bins of radius less than $\xi ( \epsilon)$ has the desirable property that player $i$ always has an $\epsilon$-best-response in $\Pi^i$ to any policy $\bgamma^{-i} \in \bDelta^{-i}$ for the remaining players. Moreover, as there is at least one equilibrium in $\bDelta$, say $\bgamma^*$, we are guaranteed at least one $\epsilon$-equilibrium in $\bPi$.%---that where each agent $i$ plays a policy $\delta$-close to $\gamma^{*i}$.
\begin{corollary}
For any $\epsilon > 0$, there exists $\xi = \xi ( \epsilon, G ) > 0$ such that if $\bPi$ is a quantization of $\bDelta$ into bins with radius no greater than $\xi$, we have $\bPi \cap \eq^{\epsilon} \not= \varnothing$.
\end{corollary}
%\by{It might be advantageous to directly introduce a metric on each player's simplex, i.e. metrics $d^i$ on $\Delta^i$, and then combine them using the maximum over $i \in \NN$ to get a metric $d$ on $\bDelta$. Then I could avoid any non-constructive proofs and directly tell them that if we have $d ( \bpi , \bpi^* ) < \epsilon/N$ for some $\bpi \in \eq^0$, then $\bpi \in \eq^{\epsilon}$. I have removed the footnote above/below accordingly. }
In this paper, we avoid the question of how one should choose the quantization $\bPi$. Bounds on the radii of the quantization bins could be produced in terms of the transition probability function $P$ and the quantity $\max \{ \left| c^i ( x, \bu ) \right| : x \in \xx, \bu \in \U \}$, but we leave that for future research. For the rest of this paper, we instead make the following assumption.
\begin{assumption} \label{ass:quantized-set}
Let $\epsilon > 0$ be fixed throughout the rest of the paper. Assume the game $G$ is symmetric, and the sets of quantized policies $\{ \Pi^i \}_{i \in \NN}$ satisfy
\begin{itemize}
\item $\Pi^i = \Pi^j$ for all $i, j \in \NN$;
\item $\bPi \cap \eq^{\epsilon} \not= \varnothing$, where $\bPi = \times_{i \in \NN} \Pi^i$.
\item For any $\bgamma^{-i} \in \bDelta^{-i}$, the set $\bPi^i$ contains an $\epsilon$-best-response to $\bgamma^{-i}$.
\end{itemize}
\end{assumption}
\subsection{Perturbed policies} \label{ss:perturbed policies}
We now introduce perturbed policies, which play an important role in the design of our algorithm in the subsequent sections. %In other literatures, such policies are sometimes called ``softened policies.''
\begin{definition}
Let $j \in \NN$, $\rho^j \in (0,1)$, and $\gamma^j \in \Delta^j$. We define a policy $\widehat{\gamma}^j \in \bDelta^j$ as
\[
\widehat{\gamma}^j \left( \cdot | x \right) = (1-\rho^j) \gamma^j ( \cdot | x ) + \rho^j \uniform(\uu^j), \quad \forall x \in \xx,
\]
and we refer to $\widehat{\gamma}^j$ as the $\rho^j$-perturbation of $\gamma^j$.
\end{definition}
The dependence on $\rho^j$ above is implicit but important. The quantity $\rho^j$ can be interpreted as the frequency with which player $j$ experiments with uniform random action selection, while following a baseline policy $\pi^j$ with frequency $1-\rho^j$. If $\bm{\rho} \in (0,1)^\NN$, an analogous construction $\widehat{\bpi}$ will be called the $\bm{\rho}$-perturbation of the joint policy $\bpi$.
%Although we defer a detailed explanation of our algorithm until subsequent sections, we now discuss two principal reasons for introducing perturbed policies. Recall that, for $i \in \NN$, when $-i$ follows the stationary policy $\bgamma^{-i} \in \bDelta$, player $i$ faces an environment equivalent to an MDP, with optimal Q-factors $Q^{*i}_{\bgamma^{-i}}$. Player $i$ can then use Q-learning to obtain $Q^{*i}_{\bgamma^{-i}}$ and find a best-response to $\bgamma^{-i}$.
%One of the mild conditions required for guaranteeing the convergence of Q-factors is that all state-actions are visited infinitely often with probability 1. Some policies for player $i$, such as those that place zero probability on a particular action in a given state, will fail to visit all state-actions infinitely often. Thus, the first reason for using perturbed policies in the sequel is to avoid under-exploring the state-action space when Q-learning. This will be discussed in greater detail in the coming sections.
%The second reason for using perturbed policies is related to the idea of teaching while learning: if player $i$ and $j$ are both running Q-learning algorithms, then by following a perturbed policy with small perturbation parameter $\rho^i$, player $i$ can remain stationary while learning and allow for player $j$ to simultaneously learn its environment. \by{Edits necessary.}
We now state two results about the approximation when players jointly switch from a particular policy in $\bPi$ to its perturbation, where $\bPi$ is the restricted set of joint policies of Assumption \ref{ass:quantized-set}.
\begin{lemma} \label{lemma:AY-lemma3}
For any, $\psi > 0$, there exists $\rho(\psi ) > 0 $ such that, if $\rho^j \leq \rho(\psi )$ for all $ j \in \NN$, we have
\[
\left\| Q^{*i}_{\bpi^{-i}} - Q^{*i}_{\widehat{\bpi}^{-i}} \right\|_{\infty} \leq \psi , \quad \forall \, i \in \NN \text{ and } \bpi \in \bPi,
\]
where $\widehat{\bpi}$ is the $\bm{\rho}$-perturbed policy associated to $\bpi$.
\end{lemma}
\textbf{Proof:} See \cite[Lemma 3]{AY2017}. \hfill $\square$
\begin{lemma} \label{lemma:analog3}
For any $\psi > 0$, there exists $\tilde{\rho} ( \psi) > 0$ such that if $\rho^j \leq \tilde{\rho}(\psi)$ for every $j \in \NN$, then we have
\[
\left| J^i_x ( \bpi) - J^i_x ( \widehat{\bpi}) \right| \leq \psi, \quad \forall \, i \in \NN, x \in \xx \text{ and } \bpi \in \bPi,
\]
where $\widehat{\bpi}$ is the $\bm{\rho}$-perturbed policy associated to $\bpi$.
\end{lemma}
\textbf{Proof:} This follows from Lemma \ref{lemma:continuous-cost}. \hfill $\square$
%\newpage
\section{Decoupling learning and adapting} \label{sec:paradigm}
We now outline our algorithmic approach to finding $\epsilon$-equilibrium in a symmetric stochastic game. The approach taken here builds on a technique presented in \cite{AY2017}, which decouples learning and adaptation. This decoupled design is used to mitigate the challenges relating to learning in a non-stationary environment. %which is among the core difficulties in MARL \cite{hernandez2017survey, zhang2019survey}.
During a learning phase, each agent follows a fixed perturbed policy and estimates whether it is $\epsilon$-best-responding to the (unobserved) joint policy of the remaining agents. At the end of a learning phase, the agents synchronously update their policies, which will then be followed for the subsequent learning phase. At its core, this approach consists of four parts:
\begin{enumerate}
\item Time is partitioned into intervals called ``exploration phases," the $k^{th}$ lasting $T_k \in \nn$ stage games, beginning with the stage game at time $t_k := \sum_{ l = 0}^{k-1} T_l$ and ending after the stage game at $t_k + T_k - 1$;
\item \textit{Within} an exploration phase, agent $i \in \NN$ follows a fixed policy and obtains feedback data on state-action-cost trajectories.
\item \textit{Within} an exploration phase, agent $i$ processes feedback data for policy evaluation, estimation of best-response sets, and estimation of state-action values.
\item \textit{Between} the $k^{th}$ and $(k+1)^{th}$ exploration phases, agent $i$ uses the learned information to update the baseline policy from $\pi^i_k$ to $\pi^i_{k+1} \in \Pi^i$. %The mechanism for updating policies is a design choice, and several approaches are justified (e.g. policy gradient \cite{daskalakis}; inertial best-responding \cite{ay2017}, random search \cite{chasparis}, etc.).
We focus here on $\epsilon$-satisficing update rules; that is, we focus on algorithms that prescribe no updating when player $i$ is already $\epsilon$-best-responding to the remaining players. \label{adapt-item}
\end{enumerate}
Although the exploration phase technique is employed in our main algorithm here, this paper differs from \cite{AY2017} and \cite{YAY-TAC} in multiple substantial ways. The most obvious difference is the class of games considered, with the present paper focusing on symmetric games and the earlier papers studying stochastic teams and their generalizations. The latter classes always admit an equilibrium in the set of stationary and deterministic policies, and so search can be restricted to this small finite set. Moreover, in the case of weakly ayclic games, inertial best-response dynamics can be used to drive play to such a deterministic equilibrium policy. Neither of these properties holds for general symmetric games. For example, consider the game of Rock-Paper-Scissors, presented in Figure \ref{fig:rps}. In this stateless game, the unique 0-equilibrium is to select each action with uniform probability, and one can show that for sufficiently small $\epsilon$, all $\epsilon$-equilibria will be in a small neighbourhood of the unique 0-equilibrium \cite{awasthi2010nash}. Inertial best-response dynamics---which restricts agents to selecting deterministic policies---results in cyclical behaviour in this game, and cannot be used to find an $\epsilon$-equilibrium.
\begin{figure}[h]
\begin{center}
\begin{game}{3}{3}[Rock-Paper-Scissors game]
&Rock &Paper &Scissors \\
Rock &0,0 &$-1$,1 &1,$-1$\\
Paper &1,$-1$ &0,0 &$-1$,1 \\
Scissors &$-1$,1 &1,$-1$ &0,0
\end{game}
\caption[]{Player I (II) picks a row (column), and its reward (to be maximized) is the 1st (2nd) entry of the chosen cell. }
\label{fig:rps}
\end{center}
\end{figure}
As the preceding example shows, the adaptation mechanisms used in \cite{AY2017} and \cite{YAY-TAC} may be ill-suited for use in some symmetric games. In order to include randomized policies in the search space, the main algorithm of this paper finely quantizes the policy spaces $\Delta^i$ and allows for policy updates that move through the interior of this set rather than traversing boundary points. Moreover, to avoid making specific structural assumptions on the cost function, we include random search of the quantized set when a player is not $\epsilon$-best-responding in order to allow for a greater variety of revision paths to be explored during play. The subsequent subsections elaborate on this process.
\subsection{Policy revision with oracle}
We now specify a particular policy update rule to be used in the sequel, and we study the behaviour resulting from this rule under the unrealistic assumption that each player has access to an oracle for obtaining the information required for its policy update. This section focuses on the purely adaptive part of our algorithm, with the challenges of learning postponed to the next section. Our main algorithm is later analyzed as a noise perturbed version of this oracle process.
%This section presents a particular design choice for the adaptation process outlined in item \ref{adapt-item} above. We consider a policy revision mechanism in which a player chooses not to change its policy when it is $\epsilon$-best responding to the opponent policy, and when not $\epsilon$-best responding changes with a mixture of random experimentation within $\Pi^i$ and (for example) stepping toward a best-response.
We propose an update rule that builds on the principle of satisficing, described above in Section~\ref{sec:epsilon-paths}. In particular, our rule instructs an agent to not change its policy when it is already $\epsilon$-best-responding. When not already $\epsilon$-best-responding, agent $i$ is instructed to update its policy as follows: with small probability, $e^i \in (0,1)$, select a policy in $\Pi^i$ uniformly at random; with complement probability, $1-e^i$, switch to another policy in $\Pi^i$ that is determined by the Q-factors for the current environment. The mechanism actually used is up to the algorithm designer and can incorporate knowledge of the game if desired, provided this update only uses the state-action values and the current policy of the agent when deciding the next policy. For concreteness, we now give one potential subroutine for stepping in the direction of a best-response, called UpdateRule$^i$. This subroutine is taken from \cite{Bowling} (c.f. Table 5), and we note that it is not the only alternative; more effective subroutines may exist, depending on the setting.\footnote{For example, if the game was known to be a team, one could replace this routine with inertial best-responding.}
\begin{algorithm}[h] \label{algo:update-rule}
\SetAlgoLined
\DontPrintSemicolon
\SetKw{input}{Input}
\SetKw{output}{Output}
\SetKw{Receive}{Receive}
\SetKw{parameters}{Set Parameters}
\SetKw{initialize}{Initialize}
%\SetKwFor{For}{for}{}{endfor}
\SetKwBlock{For}{for}{end}
\input Joint policy $\bpi = (\pi^i_{\rm old}, \bpi^{-i}) \in \bPi$ %; Q-factors $Q^{*i}_{\bpi^{-i}}$ \;
\BlankLine
\parameters $\eta^i > 0$ \;
%\Indp
%$e^i \in (0,1)$: experimentation probability when not $\epsilon$-best-responding \;
%UpdateRule$^i \in \PP ( \Pi^i | \bPi )$: (described above) \;
%%$\cdots$ \;
%\Indm
\BlankLine
%
%
%\For($k \geq 0$ ($k^{th}$ policy update{)}){ %i had to put { and } around ) because otherwise it misunderstands it as ending the for-loops condition.
%
% \BlankLine
% %Put the if suite from below.
%
% %\If(\tcp*[f]{Policy update}){ $\pi^i_k \in \BR^i_k$ }{ $\pi^i_{k+1} \leftarrow \pi^i_k$ }
% \If { $ \pi^i_k$ is an $\epsilon$-best-response to $\bpi^{-i}_k $ } { $\pi^i_{k+1} = \pi^i_k$ }
% \Else(){
% $\pi^i_{k+1} \leftarrow \begin{cases}
% \gamma^i \sim {\rm UpdateRule}^i ( \cdot | \bpi_k ), &{\rm w.p. } \quad 1 - e^i \\
% \pi^i \sim \uniform ( \Pi^i ), &{\rm w.p.} \quad e^i
% \end{cases}$
% \BlankLine
% }
%
%
%
% Go to $k+1$ \;
%}
\For (each $(x,u^i{)} \in \xx \times \uu^i$){
\BlankLine
$\eta^i_{x, u^i} := \min \left\{ \pi^i_{\rm old} ( u^i | x ) , \eta^i / \left( | \uu^i | - 1 \right) \right\}$\;
}
\vspace{5.5pt}
\Receive $Q^{*i}_{\bpi^{-i}}$ (from oracle) \;
\vspace{5.5pt}
\For (each $(x,u^i{)} \in \xx \times \uu^i$){
$H^i_{x, u^i} := \begin{cases} - \eta^i_{x,u^i} , &\text{if } u^i \not= {\rm argmax}_{a^i} Q^{*i}_{\bpi^{-i}} ( x, a^i ) \\
\sum_{ a^i \not= u^i } \eta^i_{x, a^i}, &\text{else.} \end{cases} $ \;
\vspace{2pt}
$\pi^i_{\rm mid} ( u^i | x ) := \pi^i_{\rm old} ( u^i | x ) + H^i_{x, u^i}$ \;
}
\vspace{5.5pt}
\output $\pi^i_{\rm new} = {\rm proj}_{\Pi^i} ( \pi^i_{\rm mid} )$, $\quad$ (the policy in $\Pi^i$ closest to $\pi^i_{\rm mid}$)
%\caption{Policy update rule for agent $i$ (with oracle access to Q-factors)} \label{algo:update-rule}
\caption{UpdateRule$^i (\cdot | \bpi)$ (with oracle access to Q-factors)} \label{algo:update-rule}
\end{algorithm}
%---------
% Old version of update rule below:
%---------
%Let $N^i ( \pi^i) \subset \Pi^i$ denote the neighbours\footnote{i.e. the reconstruction levels/centers of adjacent quantizer bins.} of $\pi^i \in \Pi^i$, and let $\BR^i ( \bpi^{-i})$ denote $i$'s set of (deterministic) best-responses to the policy $\bpi^{-i}$. We define UpdateRule$^i$ to be a stochastic kernel on $\Pi^i$ given $\Pi$ as follows: given $\bpi = (\pi^i, \bpi^{-i})$, first select a single best-response $\pi^{*i}$ uniformly at random from $\BR^i ( \bpi^{-i})$. Then, select a neighbour $\tilde{\pi}^i \in N^i ( \pi^i)$ that minimizes $\| \tilde{\pi}^i - \pi^{*i} \|_{\infty}$, breaking ties uniformly at random. The probability mass function UpdateRule$^i ( \gamma^i | \bpi )$ encodes the probability of $\gamma^i$ being selected by this procedure.
%We emphasize here that UpdateRule$^i$ requires access only to two pieces of information: (i) whether or not $\pi^i_k$ is an $\epsilon$-best-response to $\bpi^{-i}_k$, and (ii) the set $\BR^i ( \bpi^{-i}_k)$. In Section \ref{sec:main}, we introduce a related update rule, called IndependentUpdateRule$^i$, which is designed to approximate UpdateRule$^i$ by answering questions (i)-(ii) using estimates obtained from learning. \by{edit this part, too.}
In UpdateRule$^i$, we have assumed oracle access to Q-factors for the environment determined by the game and the joint policy $\bpi^{-i}$. Later, in Algorithm~\ref{algo:independent-update-rule}, we will introduce a subroutine called IndependentUpdateRule$^i$ that is effectively the same as UpdateRule$^i$, except that it uses learned Q-factors instead of the correct values. (We note that if the argmax in Line 7 is not a singleton, then any tie-breaking procedure may be used to select among the maximizers.)
\begin{algorithm}[h] \label{algo:oracle}
\SetAlgoLined
\DontPrintSemicolon
\SetKw{Receive}{Receive}
\SetKw{parameters}{Set Parameters}
\SetKw{initialize}{Initialize}
%\SetKwFor{For}{for}{}{endfor}
\SetKwBlock{For}{for}{end}
\parameters \;
\Indp
$e^i \in (0,1)$: experimentation probability when not $\epsilon$-best-responding \;
%UpdateRule$^i \in \PP ( \Pi^i | \bPi )$: (described above) \;
$\eta^i > 0$ for UpdateRule$^i$ (Algorithm~\ref{algo:update-rule}) \;
%$\cdots$ \;
\Indm
\BlankLine
\initialize \;
\Indp
$\pi_0^i \in \Pi^i$: initial policy \;
\Indm
\BlankLine
%\Receive $x_0$ \\
\For($k \geq 0$ ($k^{th}$ policy update{)}){ %i had to put { and } around ) because otherwise it misunderstands it as ending the for-loops condition.
\BlankLine
%Put the if suite from below.
%\If(\tcp*[f]{Policy update}){ $\pi^i_k \in \BR^i_k$ }{ $\pi^i_{k+1} \leftarrow \pi^i_k$ }
\If { $ \pi^i_k$ is an $\epsilon$-best-response to $\bpi^{-i}_k $ } { $\pi^i_{k+1} = \pi^i_k$ }
\Else(){
$\pi^i_{k+1} \leftarrow \begin{cases}
\gamma^i \sim {\rm UpdateRule}^i ( \cdot | \bpi_k ), &{\rm w.p. } \quad 1 - e^i \\
\pi^i \sim \uniform ( \Pi^i ), &{\rm w.p.} \quad e^i
\end{cases}$
\BlankLine
}
Go to $k+1$ \;
}
\caption{ Policy Revision Process for agent $i \in \NN$ (with oracle access)} \label{algo:oracle}
\end{algorithm}
We assume that players update their policies in an independent manner, conditional on the oracle information. The process resulting from all players' following Algorithm \ref{algo:oracle} is a time homogenous Markov chain on $\bPi$, and it is immediate that any $\epsilon$-equilibrium in $\bPi$ is an absorbing state.
\begin{lemma} \label{lemma:oracle}
Let $G$ be a symmetric game. If all players update their policies according to Algorithm \ref{algo:oracle} and Assumption \ref{ass:quantized-set} holds, then
\[
\lim_{k \to \infty} \Pr ( \bpi_k \in \eq^{\epsilon} ) = 1.
\]
\end{lemma}
\textbf{Proof:} As noted above, any $\epsilon$-equilibrium of $G$ in the set $\bPi$ is an absorbing set for this Markov chain. Furthermore, the argument used in the proof of Theorem \ref{thm:symmetric-paths} can be used here to show that there exist $\epsilon$-revision paths within the restricted set $\bPi$ from any $\bpi_0 \in \bPi$ to $\bPi \cap \eq^{\epsilon}$.
For each $\bgamma \in \bPi$, let $L_{\bgamma}$ denote the shortest path of positive probability from $\bgamma$ to an $\epsilon$-equilibrium in $\bPi$, and let $p_{\bgamma} > 0 $ be the probability of following this path when starting at $\bgamma$. Define $L = \max \{ L_{\bgamma} : \bgamma \in \bPi \}$ and $p := \min \{ p_{\bgamma} : \bgamma \in \bPi \} > 0$, where the inequality holds because $\bPi$ is a finite set. Then,
\[
\P \left( \bigcap_{j = 1}^J \left\{ \bpi_{k+jL} \text{ is not $\epsilon$-equilibrium} \right\} \middle| \bpi_k \right) \leq (1-p)^J \to 0, \text{ as } J \to \infty.
\]
\hfill $\square $
Although Algorithm \ref{algo:oracle} cannot be used by online independent learners, it does offer some insights for designing the policy adaptation mechanism. In the next section, we combine the adaptation mechanism of Algorithm \ref{algo:oracle} with learning to replace the oracle and achieve guarantees on finding $\epsilon$-equilibrium even with independent learners in an online setting.
\section{Synchronized two timescale learning algorithm } \label{sec:main}
In this section, we present our main algorithm, Algorithm~\ref{algo:main}, which can be interpreted as a noisy, two-timescale implementation of Algorithm~\ref{algo:oracle}. Before giving the main algorithm, we present IndependentUpdateRule$^i$, which is a variant of UpdateRule$^i$ that can be implemented by independent learners who have estimated their Q-factors and do not know the joint policy of the remaining agents. IndependentUpdateRule$^i$ is presented in Algorithm~\ref{algo:independent-update-rule}.
\begin{algorithm}[h] \label{algo:independent-update-rule}
\SetAlgoLined
\DontPrintSemicolon
\SetKw{input}{Input}
\SetKw{output}{Output}
\SetKw{Receive}{Receive}
\SetKw{parameters}{Set Parameters}
\SetKw{initialize}{Initialize}
%\SetKwFor{For}{for}{}{endfor}
\SetKwBlock{For}{for}{end}
\input Individual policy $\pi^i_{\rm old} \in \Pi^i$; estimated Q-factors $Q^i \in \rr^{\xx \times \uu^i}$ \;
\BlankLine
\parameters $\eta^i > 0$ \;
%\Indp
%$e^i \in (0,1)$: experimentation probability when not $\epsilon$-best-responding \;
%UpdateRule$^i \in \PP ( \Pi^i | \bPi )$: (described above) \;
%%$\cdots$ \;
%\Indm
\BlankLine
%
%
%\For($k \geq 0$ ($k^{th}$ policy update{)}){ %i had to put { and } around ) because otherwise it misunderstands it as ending the for-loops condition.
%
% \BlankLine
% %Put the if suite from below.
%
% %\If(\tcp*[f]{Policy update}){ $\pi^i_k \in \BR^i_k$ }{ $\pi^i_{k+1} \leftarrow \pi^i_k$ }
% \If { $ \pi^i_k$ is an $\epsilon$-best-response to $\bpi^{-i}_k $ } { $\pi^i_{k+1} = \pi^i_k$ }
% \Else(){
% $\pi^i_{k+1} \leftarrow \begin{cases}
% \gamma^i \sim {\rm UpdateRule}^i ( \cdot | \bpi_k ), &{\rm w.p. } \quad 1 - e^i \\
% \pi^i \sim \uniform ( \Pi^i ), &{\rm w.p.} \quad e^i
% \end{cases}$
% \BlankLine
% }
%
%
%
% Go to $k+1$ \;
%}
\For (each $(x,u^i{)} \in \xx \times \uu^i$){
\BlankLine
$\eta^i_{x, u^i} := \min \left\{ \pi^i_{\rm old} ( u^i | x ) , \eta^i / \left( | \uu^i | - 1 \right) \right\}$\;
}
%\vspace{5.5pt}
\For (each $(x,u^i{)} \in \xx \times \uu^i$){
$H^i_{x, u^i} := \begin{cases} - \eta^i_{x,u^i} , &\text{if } u^i \not= {\rm argmax}_{a^i} Q^i ( x, a^i ) \\
\sum_{ a^i \not= u^i } \eta^i_{x, a^i}, &\text{else.} \end{cases} $ \;
\vspace{2pt}
$\pi^i_{\rm mid} ( u^i | x ) := \pi^i_{\rm old} ( u^i | x ) + H^i_{x, u^i}$ \;
}
\vspace{5.5pt}
\output $\pi^i_{\rm new} = {\rm proj}_{\Pi^i} ( \pi^i_{\rm mid} )$, $\quad$ (the policy in $\Pi^i$ closest to $\pi^i_{\rm mid}$)
%\caption{Policy update rule for agent $i$ (with oracle access to Q-factors)} \label{algo:update-rule}
\caption{IndependentUpdateRule$^i (\cdot | \pi^i_{\rm old} , Q^i )$ (with estimated Q-factors)} \label{algo:independent-update-rule}
\end{algorithm}
\begin{algorithm}[h]
\SetAlgoLined
\DontPrintSemicolon
\SetKw{Receive}{Receive}
\SetKw{parameters}{Set Parameters}
\SetKw{initialize}{Initialize}
%\SetKwFor{For}{for}{}{endfor}
\SetKwBlock{For}{for}{end}
\parameters \;
\Indp
$\mathbb{Q}^i \subset \mathbb{R}^{ \mathbb{X}\times\mathbb{U}^i }$ and $\mathbb{J}^i \subset \rr^{\xx}$: compact sets \;
%\hspace*{5mm} $\{T_k\}_{k\geq0}$: sequence of integers in $[1,\infty)$ \\
$\{ T_k \}_{k \geq 0}$: a sequence in $\nn$ of learning phase lengths \;
\vspace{1pt}
\Indp set $t_0 = 0$ and $t_{k+1} = t_k + T_k$ for all $k \geq 0.$ \;
\vspace{1pt}
\Indm $\rho^i\in(0,1)$: action experimentation probability \;
$e^i \in (0,1)$: random policy updating probability \;
$\eta^i \in (0,1)$: parameter for IndependentUpdateRule$^i$ (Algorithm~\ref{algo:independent-update-rule}) \;
%$\lambda^i\in(0,1)$: probability of inertia when updating baseline policy \;
$\delta^i\in(0,\infty)$: tolerance level for sub-optimality \;
$\{\alpha_{n}^{i}\}_{n\geq0}$: step sizes such that $\alpha_{n}^{i}\in[0,1]$, $\sum_n\alpha_n^{i} = \infty$, $\sum_{n} \big(\alpha_n^{i}\big)^2 < \infty$ \;
\Indm
\BlankLine
\initialize baseline policy $\pi_0^i \in \Pi^i$, $Q_0^i\in\mathbb{Q}^i$, $J^i_0 \in \mathbb{J}^i$ (all arbitrary) \\
\Receive Initial state $x_0 \in \xx$ \\
\For($k \geq 0$ ($k^{th}$ exploration phase{)}){ %i had to put { and } around ) because otherwise it misunderstands it as ending the for-loops condition.
\For( $t = t_k, t_k +1, \dots, t_{k+1} - 1$ \tcp*[f]{Policy evaluation loop} ){
Select $u^i_t = \begin{cases}
\pi^i_k ( x_t ), \quad &{\rm w.p. } \quad 1 - \rho^i \\
u^i \sim \uniform ( \uu^i ), \quad &{\rm w.p. } \quad \rho^i
\end{cases}$ \label{perturbed-policy} \;
\Receive cost $c^i( x_t, u^i_t, \bu^{-i}_t)$, state $x_{t+1}$ \;
Set $n_t^i =$ the number of visits to $(x_t,u_t^i)$ in the $k^{th}$ learning phase up to $t$ \;
Set $m_t^i =$ the number of visits to $x_t$ in the $k^{th}$ learning phase up to $t$\;
\vspace{3pt}
$Q_{t+1}^i(x_t,u_t^i) = (1-\alpha_{n_t^i}^{i})Q_t^i(x_t,u_t^i) + \alpha_{n_t^i}^{i} \big[ c^i(x_t,u_t^i, \bu_t^{-i}) + \beta^i \min_{v^i} Q_t^i(x_{t+1},v^i) \big]$ \;
% $Q_{t+1}^i(x,u^i) = Q_t^i(x,u^i)$, for all $(x,u^i)\not=(x_t,u_t^i)$\;
\vspace{3pt}
$J^i_{t+1} ( x_t) = (1 - \alpha^i_{m^i_t}) J^i_t ( x_t ) + \alpha^i_{m^i_t} \left[ c^i ( x_t, u^i_t, \bu^{-i}_t) + \beta^i J^i_t ( x_{t+1} ) \right] $ \;
}
\vspace{6pt}
\If($i$ is satisfied){$\forall x \in \xx, J^i_{t_{k+1}} ( x ) \leq \min_{a^i} Q^i_{t_{k+1}} ( x, a^i ) + \epsilon + \delta^i$ }{
}
\Else($i$ not satisfied){
}
% ${\rm BR}^i_k := \{ \pi^i \in \Pi^i : Q^i_{t^i_{k+1}} ( x, \pi^i ( x) ) \leq \min_{v^i} Q^i_{t^i_{k+1}} (x, v^i) + \delta^i, \forall x \in \xx \}$ \;
%Put the if suite from below.
\If(\tcp*[f]{Baseline policy update}){ $i$ is satisfied}{ $\pi^i_{k+1} \leftarrow \pi^i_k$ }
\Else(){
$\pi^i_{k+1} \leftarrow \begin{cases}
\hat{\pi}^i \sim \text{IndependentUpdateRule}^i ( \cdot | \pi^i_k, Q^i_{t_{k+1}} ) , &{\rm w.p. } \quad 1 - e^i \\
\pi^i \sim \uniform ( \Pi^i ), &{\rm w.p.} \quad e^i
\end{cases}$
\BlankLine
}
Reset $Q_{t_{k+1}}^i$ to any $Q^i\in\mathbb{Q}^i$ (e.g., project $Q_{t^i_{k+1}}^i$ on $\mathbb{Q}^i $) \;
Reset $J^i_{t_{k+1}}$ to any $J^i \in \mathbb{J}^i$ \;
}
\caption{for agent $i$} \label{algo:main}
\end{algorithm}
%
%In Algorithm \ref{algo:oracle}, oracle access allows agents to receive instantaneous feedback about the performance of their current policy against the current policies of the remaining players, as well as accurate best-response information. Since their information is up-to-date, agents can revise their policies at every increment of time. In contrast, in an online learning application with independent learners, an agent does not get instantaneous, accurate information about its environment. Due to state transitions and chronologically delayed costs, the immediate cost of a particular state-action does not carry the full signal about the value of that state-action.
%
%Agents following Algorithm \ref{algo:main} update their policies infrequently to allow for the learning phase to yield more informative estimates for policy evaluation. \by{Insert: discussion of interruption/disturbance of learning when others change, therefore we opt for a synchronized execution.}
%
%Within a learning phase, each agent $i$ follows a behaviour policy that is a slight perturbation of their target policy, $\pi^i_k$. This can be seen in Line \ref{perturbed-policy} of Algorithm \ref{algo:main}, where $u^i_t$ is chosen according to a mixture of $\pi^i_k$ and uniform action selection. This mixture is controlled by the parameter $\rho^i \in (0,1)$, and the purpose of mixing is to ensure that all state-actions can be sampled for Q-learning. This is crucial for learning best-responses: $\pi^i_k$ may place zero probability on certain state-actions, so following $\pi^i_k$ itself may result in incorrect assessment of whether the agent is best-responding.
%
%The parameter $\rho^i$ must be positive but should not be too large, as the stochastic approximation involving the iterates $J^i_n$ will converge to something other than $J^i_x$, and the departure from $J^i_x$ depends on $\rho^i$. As a secondary reason for keeping $\rho^i$ small, the larger $\rho^i$ is chosen to be, the less agent $i$ behaves like $\pi^i_k$. This means that agents $-i$ will be learning about (and later responding to) an altogether different environment.
%
%{\bf \by{Need for analogs to Lemmas 1-4 in \cite{AY2017}. Esp. to assert existence of $\bar{\delta}, \bar{\rho}$, etc. }}
\subsection{Main Results}
Our main algorithm is given below, as Algorithm~\ref{algo:main}. To state our main results, we recall that the game $G$ and constant $\epsilon > 0$ are fixed and that the restricted set of policies $\bPi$ is chosen to satisfy Assumption \ref{ass:quantized-set}.
\begin{assumption} \label{ass:rho-delta}
For all $i \in \NN$, $\rho^i \in (0, \bar{\rho})$ and $\delta^i \in (0, \bar{\delta})$, where $\bar{\rho}$ and $\bar{\delta}$ are constants defined in the appendices, on page \pageref{def:rho-delta}.
\end{assumption}
The constant $\bar{\delta}$ depends only on the game, $\epsilon$, and $\bPi$, while $\bar{\rho}$ depends on the game, $\bar{\delta}$, and the collection $\{ \delta^i \}_{i \in \NN}$. We also make the following standard assumption on the state transition kernel $P$, which is necessary to ensure that no proper subset of states is absorbing.
\begin{assumption} \label{ass:states-body}
For any $x, x' \in \xx$, there exists $H \in \nn$, states $x = s_0, \dots, s_H, s_{H+1} = x'$, and joint actions $\ba_0, \dots, \bu_H$ such that
\[
\prod_{j = 0}^H P ( s_{j+1} | s_j, \ba_j ) > 0.
\]
\end{assumption}
% old/future version, with parts b and c included:
%\begin{theorem} \label{thm:main}
% Let $G$ be a \by{doubly} symmetric game, $\epsilon> 0$, and $\bPi \subset \bDelta$ be the quantized set of joint policies satisfying Assumption \ref{ass:quantized-set}. Suppose all players follow Algorithm \ref{algo:main}, and suppose Assumptions \ref{ass:quantized-set}, \ref{ass:rho-delta}, and \ref{ass:states-body} hold. Then,
% \begin{itemize}
% \item[a.] For any $\psi > 0$, there exists $\tilde{T} = \tilde{T}(\psi, \epsilon, \bPi, \{ \delta^i, \rho^i \}_{i \in \NN} ) \in \nn$ such that if $T_k \geq \tilde{T} $ for all $k$, then
% \[
% \P ( \bpi_k \in \eq^{\epsilon} ) \geq 1 - \psi , \text{ for all sufficiently large $k$.}
% \]
%% \item[b.] If $T_k$ increases to $\infty$, then
%% \[
%% \lim_{k \to \infty} \P ( \pi_k \in \eq^{\epsilon} ) = 1.
%% \]
%% \item[c.] There exists finite integers $\bar{T}_k$ such that if $T_k \geq \bar{T}_k$ for all $k$, then
%% \[
%% \lim_{k \to \infty} \P \left( \bigcup_{\bpi^* \in \eq^{\epsilon} \cap \bPi } \left\{ \lim_{k \to \infty} \pi_k = \bpi^* \right\} \right) = 1.
%% \]
% \end{itemize}
%\end{theorem}
% temporary version o main result, for arXiv 1st version:
\begin{theorem} \label{thm:main}
Let $G$ be a symmetric game, $\epsilon> 0$, and $\bPi \subset \bDelta$ be the quantized set of joint policies satisfying Assumption \ref{ass:quantized-set}. Suppose all players follow Algorithm \ref{algo:main}, and suppose Assumptions \ref{ass:quantized-set}, \ref{ass:rho-delta}, and \ref{ass:states-body} hold. Then, for any $\psi > 0$, there exists $\tilde{T} = \tilde{T}(\psi, \epsilon, \bPi, \{ \delta^i, \rho^i \}_{i \in \NN} ) \in \nn$ such that if $T_k \geq \tilde{T} $ for all $k$, then
\[
\P ( \bpi_k \in \eq^{\epsilon} ) \geq 1 - \psi , \text{ for all sufficiently large $k$.}
\]
% \item[b.] If $T_k$ increases to $\infty$, then
% \[
% \lim_{k \to \infty} \P ( \pi_k \in \eq^{\epsilon} ) = 1.
% \]
% \item[c.] There exists finite integers $\bar{T}_k$ such that if $T_k \geq \bar{T}_k$ for all $k$, then
% \[
% \lim_{k \to \infty} \P \left( \bigcup_{\bpi^* \in \eq^{\epsilon} \cap \bPi } \left\{ \lim_{k \to \infty} \pi_k = \bpi^* \right\} \right) = 1.
% \]
\end{theorem}
The proof of Theorem~\ref{thm:main} is given in Appendix~\ref{appendix:C}.
%
%\section{Simulations} \label{sec:simulations}
%
%In this section, we present a simulation study of Algorithm \ref{algo:main} applied to the familiar game of \textit{Rock-Paper-Scissors} (RPS), whose stage game is recalled in Figure~\ref{fig:rps2}.
%
%\begin{figure}[h]
%\begin{center}
%\begin{game}{3}{3}[Rock-Paper-Scissors game]
% &Rock &Paper &Scissors \\
%Rock &0,0 &$-1$,1 &1,$-1$\\
%Paper &1,$-1$ &0,0 &$-1$,1 \\
%Scissors &$-1$,1 &1,$-1$ &0,0
%\end{game}
%\caption[]{Player $A$ ($B$) picks a row (column), and its reward (to be maximized) is the 1st (2nd) entry of the chosen cell. }
%\label{fig:rps2}
%\end{center}
%\end{figure}
%
%In our first simulation, we set $\beta = 0, \epsilon = 0.2$ and ran 100 independent trials of Algorithm~\ref{algo:main}, with each trial consisting of 1500 exploration phases of length 10,000, for a total of 15 million stage games per trial. The quantized set of policies $\Pi^i$ for each player $i \in \{ A, B \}$ was chosen to be the set of probability vectors with with each entry having a single-digit after the decimal point, i.e.
%\[
%\bPi^i = \left\{ (p_a, p_b, p_c) \in \rr^3 : p_a + p_b + p_c = 1 , \text{ and } p_a, p_b, p_c \in \{ 0, 0.1, \cdots, 0.9, 1 \} \right\}.
%\]
%
%When not randomly updating its policy, the agent employed a modification of the policy update in WoLF-PHC, whereby the probability of the optimal action (with respect to the learned Q-factors) was increased and the probability of the remaining actions was decreased.\footnote{Two modifications were made: first, the step-size used in \cite{Bowling} could take only two values, while we took the step-size to be proportional to the difference $J^i(x) - \min_{a} Q^i (x,a)$; second, after making the update, the player projected the resulting policy back onto the quantized set.} The results, which are in line with our theoretical results, are summarized in Figure~\ref{fig:graph}.
%
%\begin{figure}[h]
%\includegraphics[width = 0.95\textwidth]{July_22_matlab_graph}
%\centering
%\caption[]{Frequency of $\{ \pi_t \in \bDelta^{\epsilon}_{\rm eq} \cap \bPi \}$, averaged over 100 trials.}
%\label{fig:graph}
%\end{figure}
%\subsection*{Other remarks}
%\by{
%In comparison with $N$-player general-sum symmetric games, independent learning in two-player zero-sum games (such as RPS) has been studied extensively. Our algorithm does not take advantage of the special zero-sum structure. Algorithms such as the two-timescale policy gradient method of \cite{daskalakis2021independent} or the WoLF-PHC algorithm of \cite{Bowling} could have been applied. Such methods would likely find an $\epsilon$-equilibrium in a fewer number of stage games.
%
%Despite the small size and simplicity of the game considered, this example demonstrates the major drawback of our algorithmic approach: our algorithm explores the set of joint policies $\bPi$ slowly, and finding $\epsilon$-equilibria is therefore a slow process. This is especially true when parameters are not finely tuned to optimize performance, as was the case in our experiments above.
%
%A conservative implementation might take $\bPi$ to be a very large set and thus take the EPs to be exorbitantly long to obtain more accurate Q-factors from learning. While this might reduce the number of EPs needed to find an $\epsilon$-equilibrium, the number of stage games can become quite high.
%
%A final point, to be edited later: relying on random exploration to find $\epsilon$-equilibrium is far easier when $\epsilon$ is large and the size of $\bPi$ is relatively small. Using our digits of precision approach to quantizing $\Pi^i$ for each $i$ results in an exponential blow-up of $\bPi$; as such, purely random dynamics can take an exceedingly long time to find an $\epsilon$-equilibrium in this set. Combining our $\epsilon$-satisficing + random search ideas with policy gradient approaches may offer better performance in cases where the quantized sets are very large.
%
%}
%\section{Discussion} \label{sec:discussion}
%
%\green{To Insert:}
%\begin{itemize}
% \item Admission that using zero-sum and/or weakly acyclic structure of game should result in much faster convergence; our approach is slow and doesn't scale well in general.
%\end{itemize}
\section{Simulations} \label{sec:simulation}
In this section, we present a simulation study of Algorithm \ref{algo:main} applied to the familiar game of Rock-Paper-Scissors (RPS), whose stage game was given in \ref{fig:rps} and is recalled in Figure~\ref{fig:rps2}.
\begin{figure}[h]
\begin{center}
\begin{game}{3}{3}[Rock-Paper-Scissors game]
&Rock &Paper &Scissors \\
Rock &0,0 &$-1$,1 &1,$-1$\\
Paper &1,$-1$ &0,0 &$-1$,1 \\
Scissors &$-1$,1 &1,$-1$ &0,0
\end{game}
\caption[]{Player $A$ ($B$) picks a row (column), and its reward (to be maximized) is the 1st (2nd) entry of the chosen cell. }
\label{fig:rps2}
\end{center}
\end{figure}
We set $\beta = 0, \epsilon = 0.2$ and ran 100 independent trials of Algorithm~\ref{algo:main}, with each trial consisting of 1500 exploration phases of length 10,000, for a total of 15 million stage games per trial. The quantized set of policies $\Pi^i$ for each player $i \in \{ A, B \}$ was chosen to be the set of probability vectors with with each entry having a single-digit after the decimal point, i.e.
\[
\bPi^i = \left\{ (p_a, p_b, p_c) \in \rr^3 : p_a + p_b + p_c = 1 , \text{ and } p_a, p_b, p_c \in \{ 0, 0.1, \cdots, 0.9, 1 \} \right\}.
\]
When not randomly updating its policy, the agent employed a modified version of the independent update rule of Algorithm~\ref{algo:independent-update-rule}. %, where the quantity $H^i_{x, u^i}$ is scaled by the estimate of how badly the current policy is performing in that state, i.e. it is scaled proportional to $J^i(x) - \min_{a} Q^i (x,a)$.
The results, which are in line with our theoretical results, are summarized in Figure~\ref{fig:graph}.
\begin{figure}[h]
\includegraphics[width = 0.95\textwidth]{July_22_matlab_graph}
\centering
\caption[]{Frequency of $\{ \pi_t \in \bDelta^{\epsilon}_{\rm eq} \cap \bPi \}$, averaged over 100 trials.}
\label{fig:graph}
\end{figure}
\section{Discussion} \label{sec:discussion}
The algorithm presented here is---to our knowledge---the first independent learner that can provably drive play to approximate equilibrium in $N$ player symmetric games without assuming further structure, such as a zero-sum structure or admission of a potential function. One disadvantage of this generality is that, without relying on game-specific structure, the algorithm can be quite slow to converge to $\epsilon$-equilibrium when $\epsilon$ is small. This is most likely to happen when the choice of update function is not well-suited to the game at hand, in which case play is driven to $\epsilon$-equilibrium through random search alone. This reliance on random search is problematic when employing uniform quantizers, because the cardinality of the finite set $\bPi$ will explode as the radius of a bin is taken to 0. %\footnote{For illustration, the number of bins required to cover a 1-by-1 square with radius $r$ tiles (in the max norm) grows on the order of $\left( \frac 1 r \right)^2$.} %\by{Quick and dirty approximation: if $r$ is the radius of the bin and $d = | \uu^i|$, then I think the number of bins grows on the order of $ | \xx | \cdot ( \frac 1 r )^d $.}
In that case, the mixing time of the underlying Markov chain will also be extremely large, and the algorithm will not be reliable in the short-run. It would be desirable to give an upper bound on the requisite bin radius $\xi ( \epsilon, G)$ in terms of the cost function and transition kernel of the game. This way, one could potentially determine quickly whether this algorithm can be used as is or whether it requires modification.
A second shortcoming of the approach presented here (and elsewhere, e.g. \cite{AY2017, YAY-TAC, Marden2009payoff,foster2006regret,germano2007global}) is that it requires agents to synchronously change policies at the end of exploration phases. In some settings this is justifiable, but in other settings it can be seen as a form of coordination between players, which might be questionable given the decentralized nature of the problem. Empirically, it appears as though perfect synchrony is not needed for driving play to equilibrium, but a rigorous proof has yet to be given. The difficulty here comes in large part from the complication of conditional probabilities: one agent's learning carries information about the state-action trajectory and so is correlated with the learning of another agent, so arguments that rely on conditioning and an appeal to the oracle process do not go through cleanly. %\by{Another alternative is every-step policy updating, i.e. putting $T^i_k = 1$ and not resetting Q-factors. This underlies several recent papers, e.g. \cite{daskalakis2021independent}, but hasn't been explored with random jumps, to my knowledge.}
%
% See above for another future direction that I'm not putting in the first arXiv version.
Another direction for future work is to switch focus away from the tabular setting, where each state-action is treated individually, and to consider action value learners that employ function approximation instead. With minor modifications, the main algorithmic ideas can be extended to be applicable to games with large state and action sets.
Finally, we note that the special structure required for the algorithm design---$\epsilon$-revision paths---is likely a structural property shared by other interesting classes of games. In a different class of games, if one can prove that a particular quantization $\bPi$ of $\bDelta$ admits $\epsilon$-revision paths from any initial policy $\bpi_0 \in \bPi$ to an approximate equilibrium policy in $\bPi$, then the same algorithm can be used for that class. However, guaranteeing that such paths exist in the absence of symmetry may be difficult.
\section{Conclusions} \label{sec:conclusion}
In this paper, we considered independent learning algorithms for a class of stochastic games that is characterized by symmetry across players, both in costs and in state dynamics. To develop an algorithm that drives play to approximate equilibrium, we studied the properties of symmetric games in order to find exploitable structure. To this end, we defined $\epsilon$-revision paths and the $\epsilon$-revision paths property, and we showed that our class of symmetric games satisfies the $\epsilon$-revision paths property for any $\epsilon \geq 0$.
After establishing the exploitable structure of the $\epsilon$-revision paths property, we presented a new algorithm for playing symmetric stochastic games, and we prove that this algorithm drives play to approximate equilibrium in the long run. The algorithm can be thought of as a two timescale procedure, wherein learning and adaptation occur at different timescales: players hold their policies constant during the fast timescale and use their observations to perform policy evaluation, and then players change their policies on the slower timescale to leverage this learned information. Our main algorithm, Algorithm~\ref{algo:main}, uses the same `exploration phase' technique that was presented in \cite{AY2017}, but here we employ a very different adaptation mechanism for changing policies and furthermore we explore an altogether different policy set. Our main algorithm combines ``$\epsilon$-satisficing" with random search to explore a quantized subset of the agent's policies.
\newpage
\appendix
\section{Proofs of the lemmas}
\subsection*{Proof of Lemma \ref{lemma:continuous-cost}} \label{pf:lemma-cost-cty}
Fix $\epsilon > 0$ and $\bgamma \in \bDelta$. For some $T \in \nn$, we note that the probability of a given trajectory $\{ (s_k, \ba_k) \}_{k = 0}^T$ is the following:
\begin{equation} \label{cty-of-prob}
\P^{\bgamma} \left( \{ (x_k =s_k, \bu_k = \ba_k) \}_{k = 0}^T \middle| x_0 = x \right) = \prod_{k =0}^T \bgamma ( \ba_k | s_k ) \cdot \prod_{k = 0}^{T-1} P ( s_{k+1} | s_k, \ba_k),
\end{equation}
where $\bgamma ( \ba_k | s_k) = \prod_{j \in \NN} \gamma^j ( a^j_k | s_k)$ for each $0 \leq k \leq T$. \vspace{5.5pt}%As it is a product, this expression is clearly continuous in the probabilities $\bgamma( \cdot | \cdot)$.
\noindent Let $\tilde{\bgamma} \in \bDelta$ be any policy. Taking $T$ large enough that
\[
\frac{\beta^{T+1}}{1- \beta }\max \left\{ | c^j ( x, \bu ) | : j \in \NN, x \in \xx, \bu \in \U \right\}< \frac \epsilon 4
\]
holds, we have that
\begin{align*}
\left| J^i_x ( \bgamma) - J^i_x ( \tilde{\bgamma} ) \right| \leq \left| E^{\bgamma} \left( \sum_{k = 0}^T (\beta^i)^k c^i ( x_k, \bu_k ) | x_0 = x \right) - E^{\tilde{\bgamma}} \left( \sum_{k = 0}^T (\beta^i)^k c^i ( x_k, \bu_k ) | x_0 = x \right) \right| + \epsilon / 2,
\end{align*}
\noindent Define a function $F^i_T : \left( \xx \times \U \right)^{T+1} \to \rr$ by $F^i_t ( s_0, \ba_0, \dots, s_T, \ba_T) = \sum_{k = 0}^T (\beta^i)^k c^i ( s_k, \ba_k )$. We then have
\[
E^{\bgamma} \left( \sum_{k = 0}^T (\beta^i)^k c^i ( x_k, \bu_k ) \middle| x_0 = x \right) = \sum_{\bm{\omega} \in \left( \xx \times \uu \right)^{T+1}} F^i_T ( \bm{\omega} ) \P^{\bgamma} \left( \{ (x_k , \bu_k \}_{k =0}^T = \bm{\omega} \middle| x_k = 0 \right).
\]
\noindent For ease of notation, we write $\P^{\bgamma} ( \bm{\omega} | x_0 = x ) := \P^{\bgamma} \left( \{ (x_k , \bu_k \}_{k =0}^T = \bm{\omega} \middle| x_k = 0 \right)$. Then,
\begin{flalign} \label{eq:cty-bound}
& \left| E^{\bgamma} \left( \sum_{k = 0}^T (\beta^i)^k c^i ( x_k, \bu_k ) | x_0 = x \right) - E^{\tilde{\bgamma}} \left( \sum_{k = 0}^T (\beta^i)^k c^i ( x_k, \bu_k ) | x_0 = x \right) \right| \nonumber \\
& \leq \sum_{\bm{\omega} \in \left( \xx \times \uu \right)^{T+1}} \left| F^i_T ( \bm{\omega} ) \right| \cdot \left| \P^{\bgamma} \left( \bm{\omega} \middle| x_ 0 = x \right) - \P^{\tilde{\bgamma}} \left( \bm{\omega} \middle| x_0 = x \right) \right| .
% & \leq \sum_{\bm{\omega} \in \left( \xx \times \uu \right)^{T+1}} \left| F^i_T ( \bm{\omega} ) \right| \cdot \left| \P^{\bgamma} \left( \{ (x_k , \bu_k \}_{k =0}^T = \bm{\omega} \middle| x_ 0 = x \right) - \P^{\tilde{\bgamma}} \left( \{ (x_k , \bu_k \}_{k =0}^T = \bm{\omega} \middle| x_0 = x \right) \right| .
\end{flalign}
From the continuity of the right-hand side of \eqref{cty-of-prob}, the result follows: taking $\left| \gamma^j ( a^j | s ) - \tilde{\gamma}^j ( a^j | s ) \right|$ to be sufficiently small for all $(s, a^j) \in \xx \times \uu^j$ for every $i \in \NN$, we can ensure that
\[
% \left| \P^{\bgamma} \left( \{ (x_k , \bu_k \}_{k =0}^T = \bm{\omega} \middle| x_ 0 = x \right) - \P^{\tilde{\bgamma}} \left( \{ (x_k , \bu_k \}_{k =0}^T = \bm{\omega} \middle| x_0 = x \right) \right| \leq \frac{ \epsilon }{ 2 \left| \left( \xx \times \U \right) \right|^{T+1} \max_{\bm{\omega}} | F^i_T ( \bm{\omega}) | },
\left| \P^{\bgamma} \left( \bm{\omega} \middle| x_ 0 = x \right) - \P^{\tilde{\bgamma}} \left( \bm{\omega} \middle| x_0 = x \right) \right| \leq \frac{ \epsilon }{ 2 \left| \left( \xx \times \U \right)^{T+1} \right|\max_{\bm{\omega'}} | F^i_T ( \bm{\omega'}) | },
\]
which bounds the righthand side of \eqref{eq:cty-bound} by $\epsilon/2$, completing the proof. \hfill $\square$
\vspace{11pt}
Note that the preceding proof did not require symmetry, and indeed Lemma \ref{lemma:continuous-cost} holds for any stochastic game \eqref{def:stochasticGame}.
\newpage
\section{Supporting Lemmata and results from the literature} \label{appendix:B}
\subsection*{Two results for single-agent MDPs}
Consider a single-agent MDP, using the notation from \eqref{def:stochasticGame} but omitting the player indices. Suppose the agent uses a behaviour policy $\gamma \in \Delta$ to interact with its environment and collect observations in the form of a trajectory $\{ (x_t, u_t, c(x_t, u_t) \}_{t \geq 0}$. Let $Q_0 \in \rr^{\xx \times \uu}$ be a given initial condition and suppose the agent updates its Q-factors according to the following rule:
\begin{align} \label{eq:q-factors-appendix}
Q_{t+1} ( x,u) = Q_t ( x, u) + \theta_t ( x,u ) \left( c (x_t, u_t ) + \beta \min_{a \in \uu} Q_t ( x_{t+1} , a ) - Q_t ( x, u ) \right),
\end{align}
for all $(x,u) \in \xx \times \uu, \; t \geq 0$, where $\theta_t (x,u) \in [0,1]$ is a random step-size parameter.%\footnote{We are interested in the online variant, where access to the state, action, and cost feedback arrive piece-by-piece as the agent interacts with its environment. This is in contrast to some studies that update multiple entries of the vector of Q-factors at each time.}
In this paper, for simplicity, we assume a particular form for the step-size parameters $\theta_t ( x,u)$ that ensure they are determined by the sample path up to time $t$: that is, $\theta_t ( x,u)$ is measurable with respect to the $\sigma$-algebra generated by the variables $\{ (x_k, u_k \}_{k = 0}^t \}$.
\begin{assumption} \label{ass:step-sizes}
Let $\{ \alpha_t \}_{t \geq 0}$ be a sequence in $[0,1]$ such that
\[
\sum_{t \geq 0} \alpha_t = \infty \qquad \sum_{t \geq 0} \alpha_t^2 < \infty.
\]
For any $t \geq 0$, $(x,u) \in \xx \times \uu$, define the random variables
\[
n_t ( x, u) = \left| \{ k : (x_k, u_k) = (x,u), 0 \leq k \leq t \}\right|.
\]
Assume that the step-size variables $\theta_t$ in \eqref{eq:q-factors-appendix} are given by
\[
\theta_t ( x,u) = \begin{cases}
\alpha_{n_t (x,u)} &\text{ if } (x_t, u_t) = (x,u) , \\
0 &\text{ otherwise}. \\
\end{cases}
\]
\end{assumption}
%For simplicity, we assume throughout this paper that Q-factors are updated in the following manner, which uses state-action visitation counts in determining the step-sizes. Let $\{ \alpha_t \}_{t \geq 0}$ be a sequence of numbers in $[0,1]$ such that $\sum_{t \geq 0} \alpha_t = \infty$ and $\sum_{t \geq 0} \alpha_t^2 < \infty$. For each $t \in \zz_{\geq 0}$ and $(x,u) \in \xx \times \uu$, define $n_t ( x,u)$ to be the number of visits to $(x,u)$ by time $t$: $n_t ( x, u) = \left| \{ k : (x_k, u_k) = (x,u), 0 \leq k \leq t \}\right|.$ We note that this quantity is a random variable, as it depends on the state-action trajectory $\{ (x_t, u_t )\}_{t \geq 0}$.
We now state a useful result on the uniformity of convergence of $Q_t \to Q^*$ in its initial condition $Q_0$.
\begin{lemma} \label{lemma:AY-lemma1}
Let $\mathbb{Q} \subset \rr^{\xx \times \uu}$ be a compact set and let $\gamma \in \Delta$ be the policy used by the agent. Let $(Q_t)_{t \geq 0}$ be the sequence obtained via Q-learning and suppose Assumption \ref{ass:step-sizes} holds. If every $(x,u) \in \xx \times \uu$ is visited infinitely often $\P^{\gamma}$-almost surely, then for any $\eta > 0$, there exists $T = T(\eta, \mathbb{Q}) \in \nn$ such that
\[
\P^{\gamma} \left( \sup_{t \geq T} \left\| Q_t - Q^* \right\|_{\infty} \leq \eta \right) \geq 1 - \eta, \quad \forall \, Q_0 \in \mathbb{Q}.
\]
\end{lemma}
\noindent For a proof of Lemma \ref{lemma:AY-lemma1}, see \cite[Lemma 1]{AY2017}.
An analogous result holds when estimating $J_x (\gamma)$ using a stochastic approximation algorithm similar to Q-learning. Suppose the agent follows a policy $\gamma$ and observes a trajectory of states, actions, and costs $\{ (x_t, u_t, c(x_t, u_t) \}_{t \geq 0}$, and updates its value function estimates according to the rule
\begin{equation} \label{eq:J-learning}
J_{t+1} ( x ) = \begin{cases}
(1 - \alpha_{m_t (x) }) J_t ( x_t) + \alpha_{m_t ( x_t) } \left( c(x_t,u_t) + \beta J_t ( x_{t+1} ) \right) &\text{ if } x_t = x\\
J_t ( x ), &\text{ if } x_t \not= x ,
\end{cases}
\end{equation}
where $J_0 \in \rr^{\xx}$ is given and $m_t ( x) = \left| \{ k : x_k = x, 0 \leq k \leq t \} \right|$, i.e. $m_t ( x) $ is the number of visits to $x \in \xx$ by time $t$.
\begin{lemma} \label{lemma:analog1}
Let $\gamma \in \Delta$ be a stationary policy and let $\mathbb{J} \subset \rr^{\xx}$ be a compact set. Suppose that any state $x \in \xx$ is visited infinitely often $\P^{\gamma}$-almost surely. Let $(J_t)_{t \geq 0}$ be the sequence obtained via \eqref{eq:J-learning}, and suppose Assumption \ref{ass:step-sizes} holds. For any $\eta > 0$, there exists $\tilde{T} = \tilde{T} ( \eta, \mathbb{J}) \in \nn$ such that
\[
\P^{\gamma} \left( \sup_{t \geq T } \| J_t - J^* \|_{\infty} \leq \eta \right) \geq 1 - \eta, \text{ for any } J_0 \in \mathbb{J},
\]
where $J^* ( x ) = J_x ( \gamma)$ for each $x \in \xx$.
\end{lemma}
The proof of Lemma~\ref{lemma:analog1} parallels that of \cite[Lemma 1]{AY2017}, and is omitted.
%
%\subsection*{Proof of Lemma \ref{lemma:analog1}---\green{Some polishing required}}
%
%\green{This proof parallels that of \cite[Lemma 1]{AY2017}, but I'm including it for our internal review. Could we take it out? There's a lot of details skipped in the 2017 version that are valid but kind of slow to grasp when proving it for the first time.}
%
%Using $\gamma \in \Delta$, define a linear operator $T : \rr^{\xx} \to \rr^{\xx}$ as
%\[
%(TJ) (x) = \sum_{u \in \uu} \gamma( u | x ) \left[ c(x,u) + \beta \sum_{y \in \xx} J(y) P(y | x,u) \right].
%\]
%
%$T$ is a contraction with respect to $\| \cdot \|_{\infty}$ with contraction coefficient $\beta$, and thus $T$ admits a unique fixed point, and it is readily seen that $J^*$, with $J^*(x) = J_x ( \gamma)$, is this unique fixed point. Our assumptions on state visitation and the summation properties of $\{ \alpha_t \}_{ t \geq 0}$ allow us to invoke Theorem 3 of \cite{tsitsiklis1994asynchronous} and conclude that $J \to J^*$ $\P^{\gamma}$-almost surely, independent of the initial condition.
%
%The following argument holds pathwise for any sample path $\{ (x_t , u_t) \}_{t \geq 0}$ along which both of the following hold: (i) $J_t \to J^*$, and (ii) every $(x,u) \in \xx \times \uu$ is visited infinitely often. (By our assumptions and the previous remark on $J_t \to J^*$, we note that the set of such sample paths has mass 1 under $\P^{\gamma}$.) As such, we suppress the notation of dependence on the sample path.
%
%Let $\{ J'_t\}_{t \geq 0}$ and $\{ J''_t \}_{t \geq 0}$ be the trajectories corresponding to initial conditions $J'_0$ and $J''_0 \in \mathbb{J}$, respectively.
%
%Observe that for any $t \geq 0$ we have $\left| J'_{t+1} (x) - J''_{t+1} ( x) \right| = \left| J'_{t+1} (x) - J''_{t+1} ( x) \right|$ for any $x \not= x_t$, and (omitting cancelling terms) we have
%\begin{align} \label{ineq:lemma1}
%\left| J'_{t+1} (x_t) - J''_{t+1} (x_t) \right| &= \left| (1 - \alpha_{m_t ( x_t) } ) \left( J'_t (x_t) - J''_t ( x_t ) \right) + \alpha_{m_t (x_t)} \beta \left( J'_t (x_{t+1} ) - J''_{t+1} (x_{t+1}) \right) \right| \nonumber \\
%&\leq ( 1 - \alpha_{m_t(x_t)} ) \left| J'_t (x_t) - J''_t ( x_t) \right| + \alpha_{m_t(x_t)} \beta \left| J'_t ( x_{t+1}) - J''_t ( x_{t+1} ) \right| \nonumber \\
%&\leq (1- \alpha_{m_t(x_t)} ) \left| J'_t ( x_t) - J''_t ( x_t) \right| + \alpha_{m_t(x_t)} \beta \left\|J'_t - J''_t \right\|_{\infty}
%\end{align}
%
%Since $\mathbb{J}$ is compact, the above implies that $M_t = \sup_{J'_0, J''_0 \in \mathbb{J}} \left\| J'_t - J''_t \right\|_{\infty}$ is non-increasing (\green{pf in \TeX comments}) and therefore convergent. We now show that the limit is 0, $\P^{\gamma}$-almost surely.
%%Proof that $M_t$ is non-increasing: suppose that $M_{t+1} > M_t$ for some $t$. Then, since $\mathbb{J}$ is compact, any difference $\| J'_{t+1} - J''_{t+1} \|_{\infty}$ is finite for finite $t$. Hence, we can approximate the supremum defining $M_{t+1}$. In particular, let $\xi = (M_{t+1} - M_t)/4$. There exists $J'_0$ and $J''_0$ such that $\| J'_{t+1} - J''_t \|_{\infty} \geq M_{t+1} - \xi$. But we already established that $\| J'_t - J''_t \|_{\infty} \geq \| J'_{t+1} - J''_{t+1} \|_{\infty}$, so $M_t \geq M_{t+1} - \xi > M_t$, a contradiction. So indeed $( M_t )_t$ is non-increasing.
%
%Suppose $M_t \to \bar{M} > 0$. Since $\beta < 1$ (and so $(1 + 1/\beta) > 2$), there exists $\bar{t} \in \nn$ such that $M_t \leq \bar{M} (1 + 1/\beta )/2$ for every $t \geq \bar{t}$. Using \eqref{ineq:lemma1}, we recursively upper bound $\| J'_t - J''_t \|_{\infty}$ for $t \geq \bar{t}$ as follows: for any $x \in \xx$, $K \geq 0$, we have
%
%\begin{align*}
%| J'_{\bar{t} + K} ( x ) - J''_{\bar{t} + K } ( x) | &\leq M_{\bar{t} } \prod_{k \in I(x,K)} (1 - \alpha_k ) + \left( 1 - \prod_{k \in I(x, K)} (1 - \alpha_k) \right) \beta M_{\bar{t}} \\
%&\leq M_{\bar{t} }\prod_{k \in I(x,K) } (1 - \alpha_k ) + \left( 1 - \prod_{k \in I(x,K) } (1 - \alpha_k) \right) \beta \bar{M} \left( \frac{ 1 + 1/ \beta}{2} \right) \\
%&< M_{\bar{t} } \prod_{k \in I(x,K) } (1 - \alpha_k ) + \left( 1 - \prod_{k \in I(x,K) } (1 - \alpha_k) \right) \bar{M} ,
%\end{align*}
%
%where $I(x,K) = \{ m_{\bar{t}} ( x) ,m_{\bar{t}} ( x) + 1, \dots, m_{\bar{t} + K-1} ( x ) \}$. That is, the products over $I(x,K)$ involve visitation counts to $x$ in the interval $[\bar{t}, \bar{t} + K-1]$. The first inequality holds recursively by \eqref{ineq:lemma1}, and can be verified by induction; %see June 10th in my personal notebook for proof.
%the second inequality holds by assumption that $M_t \leq M_{\bar{t}} \leq \bar{M} ( 1 + 1/\beta)/2$ for each $t \geq \bar{t}$; and the final inequality holds since $\beta ( 1 + 1/\beta)/2 < 1$.
%
%By assumption that all states are visited $\P^{\gamma}$ infinitely often, we have that for any $x \in \xx$,
%\[
%\lim_{K \to \infty} \prod_{k \in I(x,K)} (1 -\alpha_k ) = 0, \quad \P^{\gamma}\text{-almost surely} .
%\]
%
%Note that since $\bar{M} \leq \| J'_{\bar{t} + K} - J''_{\bar{t} + K} \|_{\infty}$, the above then implies that $\bar{M} < \bar{M}$, leading to a contradiction on a set of measure 1 under $\P^{\gamma}$.
%
%To complete the proof, fix any $J'_0 \in \mathbb{J}$. Since $J'_t \to J^*$ $\P^{\gamma}$-almost surely, there exists $T_1 (\eta) \in \nn$ such that
%\[
%\P^{\gamma} \left( \sup_{t \geq T_1 } \| J'_t - J^* \|_{\infty} \leq \frac \eta 2 \right) \geq 1 - \frac \eta 2.
%\]
%
%Similarly, since $M_t \to 0$ w.p. 1, there exists $T_2 ( \eta) \in \nn$ such that
%\[
%\P^{\gamma} \left( \sup_{t \geq T_2} \{ M_t \} < \frac \eta 2 \right) \geq 1 - \frac \eta 2.
%\]
%
%\noindent Define $T(\eta) = \max \{ T_1, T_2 \}$. Then, for any $J_0'' \in \mathbb{J}$, we have
%\begin{align*}
%\P^{\gamma} \left( \sup_{t \geq T } \| J''_t - J^* \|_{\infty} \leq \eta \right) &\geq \P^{\gamma} \left( \sup_{t \geq T } \| J''_t - J'_t \|_{\infty} + \sup_{t \geq T} \|J'_t - J^* \|_{\infty} \leq \eta \right) \\
%&\geq \P^{\gamma} \left( \sup_{t \geq T} \{ M_t \} + \sup_{t \geq T} \| J'_ - J^* \|_{\infty} \leq \eta \right) \\
%&\geq 1 - \eta,
%\end{align*}
%where the final inequality holds by a union bound.
%\hfill $\square$
\subsection*{Results for learning in stochastic games}
As discussed in \S \ref{ss:learning-in-games}, if players $-i$ follow a stationary policy $\bgamma^{-i} \in \bDelta^{-i}$, then player $i$ faces an environment equivalent to an MDP, and we denoted $i$'s optimal Q-factors for this environment by $Q^{*i}_{\bgamma^{-i}}$. We now review several existing results pertaining to the regularity of Q-learning in stochastic games.
%
%
%
\begin{assumption} \label{ass:step-sizes-for-game}
For each agent $i \in \NN$, $\{ \alpha^i_t \}_{t \geq 0}$ is a sequence of step-size parameters in $[0,1]$ satisfying $\sum_{t \geq 0} \alpha^i_t = \infty$, and $ \sum_{t \geq 0} (\alpha^i_t) ^2 < \infty.$ Assume that each agent $i$ updates its Q-factors according to the rule
\[
Q^i_{t+1} ( x, u^i) = \left(1 - \theta^i_t ( x,u^i )\right) Q^i_t ( x, u^i) + \theta^i_t ( x, u^i ) \left( c^i(x,u^i_t, \bu^{-i}_t) + \beta^i \min_{a^i \in \uu^i} Q^i_t ( x_{t+1}, a^i) \right),
\]
with
\[
\theta^i_t ( x, u^i ) = \begin{cases} \alpha_{ n^i_t (x,u^i) } , &\text{ if } (x_t, u^i_t ) = (x,u^i) , \\
0 , &\text{ otherwise, }
\end{cases}
\]
where for each $(x,u^i) \in \xx \times \uu^i$, $n^i_t ( x, u^i ) := \left| \{ k : (x_k, u^i_k) = (x,u^i), \text{ for } 0 \leq k \leq t \} \right|$ counts the number of visits to $(x,u^i)$ up to time $t$.
\end{assumption}
%%% Excluding this part, which used to be body:
%
%The next lemma is concerned with the mapping $\bgamma^{-i} \mapsto Q^{*i}_{\bgamma^{-i}}$. One might expect that if $-i$'s policy was perturbed by a small amount, then $i$'s environment would also change by a small amount, which would then be reflected by a small change in $i$'s optimal Q-factors.
\vspace{5.5pt}
To state our next lemma, we recall Assumption~\ref{ass:states-body}:
\textbf{Assumption~\ref{ass:states-body}}\textit{
For any $x, x' \in \xx$, there exists $H \in \nn$, states $x = s_0, \dots, s_H, s_{H+1} = x'$, and joint actions $\ba_0, \dots, \bu_H$ such that
\[
\prod_{j = 0}^H P ( s_{j+1} | s_j, \ba_j ) > 0.
\]
}
% %% As Lemma 6...
%\begin{assumption} \label{ass:states}
% For any $x, x' \in \xx$, there exists $H \in \nn$, states $x = s_0, \dots, s_H, s_{H+1} = x'$, and joint actions $\ba_0, \dots, \bu_H$ such that
% \[
% \prod_{j = 0}^H P ( s_{j+1} | s_j, \ba_j ) > 0
% \]
%\end{assumption}
%\green{
%Assumption \ref{ass:states} is really about $P$, not $\P$: we're assuming that there exists $x_0 = x, x_1, \dots, x_H, x_{H+1} = x'$ and joint actions $\ba_0, \dots, \ba_H$ such that
%\[
%\prod_{j = 0}^H P ( x_{j+1} | x_j, \ba_j ) > 0
%\]
%}
Assumption~\ref{ass:states-body} is a standard condition on the transition probabilities that requires that any state can be reached from any other. We note that under Assumption~\ref{ass:states-body}, if players follow any positively perturbed policy (i.e. perturbed with $\rho^i > 0$ for each agent $i$) then any $(s, \ba) \in \xx \times \uu$ will be visited infinitely often with probability one.
With the preceding assumption, we now consider what happens to agent $i$'s Q-factors when $i$ follows some perturbed policy $\widehat{\pi}^i$ for some $\pi^i \in \Pi^i$, its quantized set of policies, while the remaining players follow a policy $\widehat{\bpi}^{-i} \in \bPi^{-i}$.
\begin{lemma} \label{lemma:AY-lemma2}
Let $\bpi \in \bPi$ be a given stationary joint policy and let $\widehat{\bpi}$ be the associated $\bm{\rho}$-perturbed policy, with $\rho^i > 0$ for each $i$. For each $i \in \NN$, let $\mathbb{Q}^i \subset \rr^{\xx \times \uu^i}$ be a compact set. Suppose Assumptions \ref{ass:states-body} and \ref{ass:step-sizes-for-game} hold. For any $\eta > 0$, there exists $T = T(\eta) \in \nn$ such that if $t \geq T$ and $Q^i_0 \in \mathbb{Q}^i$ for each $i \in \NN$, then
\[
\P^{\widehat{\bpi}} \left( \left\| Q^i_{ t } - Q^{*i}_{\widehat{\bpi}^{-i}} \right\|_{\infty} \leq \eta , \forall i \in \NN \right) \geq 1 - \eta .
\]
\end{lemma}
\textbf{Proof:} By Assumption \ref{ass:states-body}, all states are visited infinitely often $\P^{\widehat{\bpi}}$-almost surely. We then invoke Lemma \ref{lemma:AY-lemma1} on each of the finitely many players in $\NN$ and the proof follows. (C.f. Lemma 2 in \cite[Appendix B]{AY2017}.) \hfill $\square$
% I think this was included in Section 4...
%
%\begin{lemma} \label{lemma:AY-lemma3}
% For any $\eta > 0$, there exists $\rho(\eta ) > 0 $ such that, if $\rho^i \leq \rho(\eta)$ for each $i \in \NN$, we have
% \[
% \left\| Q^{*i}_{\bpi^{-i}} - Q^{*i}_{\widehat{\bpi}^{-i}} \right\|_{\infty} \leq \eta, \forall i \in \NN \text{ and } \bpi \in \bPi,
% \]
% where $\widehat{\bpi}$ is the $\bm{\rho}$-perturbed policy associated to $\bpi$.
%\end{lemma}
%
%
%The proof of Lemma \ref{lemma:AY-lemma3} parallels\footnote{\by{I never worked out the details of the parallel proof, but we spoke about it one time and we all said it seemed fine. Thoughts?}} the proof of Lemma 3 in \cite{AY2017}, and is therefore omitted.
We now get a result analogous to Lemma \ref{lemma:AY-lemma2} but pertaining to estimates of $J^i_x$.
\begin{assumption} \label{ass:step-sizes-for-game2}
For each agent $i \in \NN$, $\{ \alpha^i_t \}_{t \geq 0}$ is a sequence of step-size parameters in $[0,1]$ satisfying $\sum_{t \geq 0} \alpha^i_t = \infty$, and $ \sum_{t \geq 0} (\alpha^i_t) ^2 < \infty.$ Assume that each agent $i$ updates its so-called J-factors according to the rule
\[
J^i_{t+1} ( x ) = \left(1 - \hat{\theta}^i_t ( x )\right) J^i_t ( x) + \hat{\theta}^i_t ( x ) \left( c^i(x,u^i_t, \bu^{-i}_t) + \beta^i J^i_t ( x_{t+1} ) \right),
\]
with
\[
\hat{\theta}^i_t ( x ) = \begin{cases} \alpha_{ m^i_t (x ) } , &\text{ if } x_t = x , \\
0 , &\text{ otherwise, }
\end{cases}
\]
where for each $x \in \xx $, $m^i_t ( x ) := \left| \{ k : x_k = x, \text{ for } 0 \leq k \leq t \} \right|$ counts the number of visits to $x$ up to time $t$.
\end{assumption}
\begin{lemma} \label{lemma:analog2}
Let $\bpi \in \bPi$ be a given stationary joint policy and let $\widehat{\bpi}$ be the associated $\bm{\rho}$-perturbed policy, with $\rho^i > 0$ for each $i$. For each $i \in \NN$, let $\mathbb{J}^i \subset \rr^{\xx}$ be a compact set. Suppose Assumptions \ref{ass:states-body} and \ref{ass:step-sizes-for-game2} hold. For any $\psi > 0$, there exists $\tilde{T} = \tilde{T}(\psi) \in \nn$ such that if $t \geq \tilde{T}$, then
\[
\P^{\widehat{\bpi}} \left( \left| J^i_t - J^i_x ( \widehat{\bpi}) \right| \leq \psi , \forall i \in \NN , x \in \xx \right) \geq 1 - \psi ,
\]
for any $J^i_0 \in \mathbb{J}^i$.
\end{lemma}
\textbf{Proof:} Since $\rho^i > 0$ for each $i \in \NN$, Assumption~\ref{ass:states-body} guarantees that each $x \in \xx$ is visited $\P^{\widehat{\bpi}}$-almost surely. The result then follows by invoking Lemma \ref{lemma:analog1} for each player $i \in \NN$ and taking the maximum time over all (finitely many) players. Invoking Lemma \ref{lemma:analog1} is possible as $i$ faces an MDP when $-i$ follows baseline policy $\bpi^{-i}$ (that is to say, $-i$ actually follows policy $\widehat{\bpi}^{-i}$). \hfill $\square$
% i think the following commented part was included in Section 4, also.
%
%\begin{lemma} \label{lemma:analog3}
% For any $\eta > 0$, there exists $\tilde{\rho} ( \eta) > 0$ such that if $\rho^i \leq \tilde{\rho}(\eta)$ for every $i \in \NN$, then we have
% \[
% \left| J^i_x ( \bpi) - J^i_x ( \widehat{\bpi}) \right| \leq \eta, \forall i \in \NN, x \in \xx \text{ and } \bpi \in \bPi,
% \]
% where $\widehat{\bpi}$ is the $\bm{\rho}$-perturbed policy associated to $\bpi$.
%\end{lemma}
%
%\begin{proof}
% By Lemma \ref{lemma:continuous-cost} and the compactness of $\bDelta$, for each $i \in \NN, x \in \xx$, $J^i_x$ is uniformly continuous in the joint policy. The result follows.
%\end{proof}
\subsection*{Parameter Restrictions}
With the same (fixed) $\epsilon > 0$ used in Assumption \ref{ass:quantized-set}, define $\bar{\delta} = \bar{\delta} ( \epsilon , \bPi)$ to be the minimum \textit{positive} distance between $\epsilon$ and the amount by which some player's policy is suboptimal. That is,
$\bar{\delta} := \min \left( S \setminus \{ 0 \} \right)$, where
\[
S := \left\{ \left| \epsilon - \left( J^i_x ( \pi^i, \bpi^{-i} ) - \min_{a^i \in \uu^i} Q^{*i}_{ \bpi^{-i}} (x, a^i) \right) \right| : i \in \NN, \pi^i \in \Pi^i, \bpi^{-i} \in \bPi^{-i}, x \in \xx \right\}.
\]
The set $S$ is finite, since $\NN, \xx,$ and $\bPi$ are finite. We note that $\bar{\delta}$ depends on both $\epsilon > 0$ and the finite quantization $\bPi$.
$\bar{\delta}$ will serve as an upper bound for additive terms that each agent will use when determining whether they are $\epsilon$-best-responding. i.e. $\delta^i \in (0, \bar{\delta})$. We fix $\{ \delta^i \}_{i \in \NN}$ and introduce an upper bound $\bar{\rho}$ such that $\rho^i \leq \bar{\rho}$ for all $i \in \NN$ implies
% old:
%\begin{align} \label{ineq:AY-(9)}
%\left\| Q^{*i}_{\bpi^{-i}} - Q^{*i}_{\widehat{\bpi}^{-i}} \right\| &< \frac 1 2 \min_{j \in \NN} \{ \delta^j, \bar{\delta} - \delta^j \}, \quad \forall i \in \NN, \bpi^{-i} \in \bPi^{-i}, \quad \text{and} \\
%\left\| J^i_x ( \bpi^{-i} ) - J^i_x ( \widehat{\bpi}^{-i} ) \right\| &< \frac 1 2 \min_{j \in \NN} \{ \delta^j, \bar{\delta} - \delta^j \}, \quad \forall i \in \NN, \bpi^{-i} \in \bPi^{-i}.
%\end{align}
% new:
\begin{align} \label{ineq:AY-(9)}
\left\| Q^{*i}_{\bpi^{-i}} - Q^{*i}_{\widehat{\bpi}^{-i}} \right\| &< \frac 1 2 \min_{j \in \NN} \{ \delta^j, \bar{\delta} - \delta^j \}, \quad \forall i \in \NN, \bpi^{-i} \in \bPi^{-i}, \quad \text{and}
\end{align}
\begin{align} \label{ineq:AY-(9)-prime}
\left\| J^i_x ( \bpi^{-i} ) - J^i_x ( \widehat{\bpi}^{-i} ) \right\| &< \frac 1 2 \min_{j \in \NN} \{ \delta^j, \bar{\delta} - \delta^j \}, \quad \forall i \in \NN, \bpi^{-i} \in \bPi^{-i}.
\end{align}
Such $\bar{\rho}$ exists by Lemmas \ref{lemma:AY-lemma3} and \ref{lemma:analog3}. \label{def:rho-delta}% \by{Should these be $\frac 1 4$ not $\frac 1 2$?}
\begin{lemma} \label{lemma:AY-lemma4}
Let $\bpi \in \bPi$ be a joint policy, let $\widehat{\bpi}$ be its $\bm{\rho}$-perturbed version, and let $\P^{\widehat{\bpi}}$ denote the associated probability measure. Suppose $\delta^i \in (0, \bar{\delta})$ and $\rho^i \in (0, \bar{\rho})$ for each $i \in \NN$. For any $\psi > 0$, there exists $T = T(\psi)$ such that, if $t \geq T$, we have
\[
\P^{\widehat{\bpi}} \left( \left\| Q^i_{t} - Q^{*i}_{\bpi^{-i}} \right\|_{\infty} < \frac 1 2 \min_{j \in \NN} \{ \delta^j, \bar{\delta} - \delta^j \} , \forall i \in \NN \right) \geq 1 - \psi,
\]
for any $Q^i_0 \in \mathbb{Q}^i$.
\end{lemma}
\begin{proof}
This follows from Lemma \ref{lemma:AY-lemma2} and \eqref{ineq:AY-(9)}. See also Lemma 4 in \cite[Appendix B]{AY2017}.
\end{proof}
\begin{lemma} \label{lemma:analog4}
Let $\bpi \in \bPi$ be a joint policy, let $\widehat{\bpi}$ be its $\bm{\rho}$-perturbed version, and let $\P^{\widehat{\bpi}}$ denote the associated probability measure. Suppose $\delta^i \in (0, \bar{\delta})$ and $\rho^i \in (0, \bar{\rho})$ for each $i \in \NN$. For any $\psi > 0$, there exists $\tilde{T} = \tilde{T} (\psi)$ such that, if $t \geq \tilde{T}$, we have
\[
\P^{\widehat{\bpi}} \left( \left| J^i_x ( \bpi ) - J^i_t ( x ) \right| < \frac 1 2 \min_{j \in \NN} \{ \delta^j, \bar{\delta} - \delta^j \} , \forall i \in \NN , x \in \xx \right) \geq 1 - \psi,
\]
for any $J^i_0 \in \mathbb{J}^i$.
\end{lemma}
\begin{proof}
This follows from Lemma \ref{lemma:analog2} and \eqref{ineq:AY-(9)-prime}.
\end{proof}
Strictly speaking, the quantity $\bar{\rho}$ used in both of Lemmas \ref{lemma:AY-lemma4} and \ref{lemma:analog4} depends on the choice of $\{ \delta^i \}_{i \in \NN}$. We suppress this dependence for ease of notation.
\section{Proofs of main results} \label{appendix:C}
\begin{lemma} \label{lemma:recursion}
Let $0 < p < u < 1$ and $y_0 \in [0,1]$. For each $k \geq 0$, define $y_{k+1} = u y_k + p (1-y_k)$. Then,
\[
\lim_{k \to \infty} y_k = \frac{p}{ 1 - u + p }
\]
\end{lemma}
\textbf{Proof:} We find an explicit form for $y_k = (u-p)^k y_0 + p \sum_{j = 0}^{k-1} (u-p)^j$, and the result follows by taking limits.
\hfill$\square$
\subsection*{Proof of Theorem \ref{thm:main}}
%For ease of notation, we use $J^i_{\star} ( \bpi^{-i} )$ to denote the vector in $\rr^{\xx}$ with component $x$ given by $J^i_x ( \bpi^{-i} )$.
%
%\vspace{5.5pt}
%
%First, note that by construction of $\bar{\delta} ( \epsilon, \bPi)$ and Assumption \ref{ass:rho-delta}, we have that conditional on the events
%\begin{align*}
%\left\| J^i_{t_{k+1}} - J^i_{\star} ( \bpi^{-i}_k ) \right\|_{\infty} &< \frac 1 2 \min_j \{ \delta^j , \bar{\delta} - \delta^j \} , \quad \text{and}\\
%\left\| Q^i_{t_{k+1}} - Q^{*i}_{\bpi^{-i}_k} \right\|_{\infty} &< \frac 1 2 \min_j \{ \delta^j , \bar{\delta} - \delta^j \} ,
%\end{align*}
%%
%player $i \in \NN$ correctly determines whether it is $\epsilon$-best-responding at $\bpi_k$ (i.e. $\pi^i_k \in \BR^i_{\epsilon} ( \bpi^{-i}_k )$) by checking whether
%\[
%J^i_{t_{k+1}} ( x) < \min_{a^i \in \uu^i } Q^i_{t_{k+1}} (x, a^i) + \epsilon + \delta^i , \quad \forall x \in \xx.
%\]
%
%We define the event $\Qcorrect^i_k$ to be the intersection of the preceding events for all players, i.e.
%\[
%\Qcorrect^i_k := \bigcap_{i \in \NN} \min \left\{ \left\| J^i_{t_{k+1}} - J^i_{\star} ( \bpi^{-i}_k ) \right\|_{\infty} , \left\| Q^i_{t_{k+1}} - Q^{*i}_{\bpi^{-i}_k} \right\|_{\infty} \right\} < \frac 1 2 \min_{j \in \NN} \{ \delta^j, \bar{\delta} - \delta^j \}.
%\]
%
%For ease of notation, we define $\Qcorrect_k := \cap_{i \in \NN} \Qcorrect^i_k$ and also
%\[
%\Qcorrect_{k: k+l} = \Qcorrect_k \cap \Qcorrect_{k+1} \cap \cdots \cap \Qcorrect_{k+l}, \text{ for } k, l \in \nn.
%\]
%
%Next, we define $B_k := \{ \bpi_k \in \bPi \cap \eq^{\epsilon} \} $. Recall from the proof of Lemma~\ref{lemma:oracle} that we defined $L \in \nn$ and $p > 0$ to be two objects associated to the oracle process in Algorithm~\ref{algo:oracle}. In particular, we have that---in the oracle process---the probability of transitioning from any policy to an $\epsilon$-equilibrium in $\bPi$ in $L$ steps occurs with probability no less than $p$.
%
%\begin{align*}
%\P ( B_{k+L} ) &= \P ( B_{k+L} | B_k ) \P ( B_k ) + \P ( B_{k+L} | B_k^c ) ( 1 - \P (B_k)) \\
% &\geq \P ( B_{k+L} | \Qcorrect_{k:k+L} \cap B_k ) \P ( \Qcorrect_{k : k+L} | B_k ) \P (B_k) \\
% &\quad + \P ( B_{k+L} | \Qcorrect_{k:k+L} \cap B^c_k ) \P ( \Qcorrect_{k : k+L} | B^c_k ) ( 1 - \P (B_k) )
%\end{align*}
%
%When a player correctly determines whether it is $\epsilon$-best-responding at an $\epsilon$-equilibrium baseline policy, it does not change its policy. Thus, $\P ( B_{k+L} | \Qcorrect_{k:k+L} \cap B_k ) = 1$. Furthermore, supposing each player correctly determines whether it is $\epsilon$-best-responding for each exploration phase in the interval $[ k, k + L ]$, then the baseline policy process follows an $\epsilon$-revision path into $\bPi\cap \eq^{\epsilon}$ in $L$ steps with (conditional) probability at least $p$. That is, $\P ( B_{k+L} | \Qcorrect_{k:k+L} \cap B^c_k ) \geq p > 0$. Thus, we have
%
%\begin{align*}
%\P ( B_{k+L} ) \geq \P ( \Qcorrect_{k : k+L} | B_k ) \P (B_k) + p \P ( \Qcorrect_{k : k+L} | B^c_k ) ( 1 - \P (B_k) )
%\end{align*}
%
%By the lemmata of Appendix~\ref{appendix:B}, the conditional probabilities $\P( \Qcorrect_{k:k+L} | B_k)$ and $\P ( \Qcorrect_{k:k+L} | B_k^c )$ can be made arbitrarily high by taking the exploration phase lengths $T_k, \cdots, T_{k+L}$ to be suitably large. Following this observation, the result then follows easily, using a recursive argument similar to that of \cite[Theorem 1.1]{AY2017}. \hfill $\square$
%
%%But by our previous lemmata, the conditional events above can be made to have arbitrarily high probability. Hence, the true stochastic process can be made to look like a slight perturbation of the process studied in Lemma \ref{lemma:oracle}. We then use a proof technique that parallels the one I wrote for \cite{YAY-TAC}'s first theorem.
%
%%Parts b. and c. can be proved by paralleling the steps from \cite{AY2017}.
We begin by defining a number of events of interest. For any $k \geq 0$, we let $E^J_k$ be the event that J-factor learning in the $k^{th}$ exploration phase is accurate to our desired precision, i.e.
\[
E^J_k := \left\{ \left| J^i_{t_{k+1}} ( x ) - J^i_x ( \bpi_k) \right| < \frac 1 2 \min_{j \in \NN} \left\{ \bar{\delta} , \bar{\delta} - \delta^j \right\}, \forall x \in \xx, i \in \NN \right\}.
\]
The event $E^Q_k$ is defined analogously for Q-factors:
\[
E^Q_k := \left\{ \left\| Q^i_{t_{k+1}} - Q^{*i}_{\bpi^{-i}_k } \right\|_{\infty} < \frac 1 2 \min_{j \in \NN} \left\{ \bar{\delta} , \bar{\delta} - \delta^j \right\}, \forall i \in \NN \right\}.
\]
Then we define $E_k = E^J_k \cap E^Q_k$, and for $l \geq 1$ we define $E_{k:k+l} := E_k \cap E_{k+1} \cap \cdots E_{k+l}$. Conditional on $E_k$, it holds that $\pi^i_k \in \BR^i_{\epsilon} ( \bpi^{-i}_k)$ if and only if $J^i_{t_{k+1}} (x ) < \min_{a^i} Q^i_{t_{k+1}} ( x, a^i ) + \epsilon + \delta^i$ for every state $x \in \xx$. This is true by the final remark of \S\ref{ss:learning-in-games} and the construction of $\bar{\delta}$. Thus, once $E_k$ is assumed, players correctly determine whether or not they are $\epsilon$-best-responding to the other players' policy.
\vspace{5.5pt}
Recall the object $L$ from the proof of Lemma~\ref{lemma:oracle}: for any $\bpi_0 \in \bPi \setminus \eq^{\epsilon}$, there exists an $\epsilon$-revision path into $\bPi \cap \eq^{\epsilon}$, and therefore there exists a shortest such path, whose length is denoted $L_{\bpi_0}$. We then take $L := \max \left\{ L_{\bpi_0} : \bpi_0 \in \bPi \right\}$.
In the proof of Lemma~\ref{lemma:oracle}, we also introduced a quantity $p$, to lower bound the probability of transitioning from any initial policy into $\bPi \cap \eq^{\epsilon}$ in $L$ steps. Here, when the Q-factors are noise perturbed, this lower bound may no longer hold, as the dynamics pushing play to equilibrium may place lower probability on each necessary transition. Nevertheless, another lower bound exists for the case where Q-factors are estimated:
\[
\P \left( \bpi_{k+L} \in \bPi \cap \eq^{\epsilon} \middle| E_k, E_{k+1}, \cdots, E_{k+L}, \bpi_k \notin \bPi \cap \eq^{\epsilon} \right) \geq \prod_{i \in \NN} \left( \frac{e^i}{ | \Pi^i | } \right)^L =: p_{\min} > 0.
\]
The preceding bound is explained as follows. Given any $\bpi_0 \in \bPi \setminus \eq^{\epsilon}$, take a shortest $\epsilon$-revision path into $\bPi \cap \eq^{\epsilon}$. Such a path has length at most $L$, and at every step along this path (say the $(k+l)^{th}$ for $0 \leq l \leq L-1$) each player either stays put when $\epsilon$-best-responding (which has conditional probability 1 when the event $E_{k+l}$ occurs) or the player might switch (or stay put) when not $\epsilon$-best-responding. The conditional probability of selecting the correct successor policy to follow this path, conditional on $E_{k+l}$ occurring, is then $e^i/ | \Pi^i |$ for player $i$.
For ease of notation, let $B_k := \left\{ \bpi_k \in \bPi \cap \eq^{\epsilon} \right\}$, for $k \geq 0$. By the previous remarks, we have the following inequality, for any $k \geq 0$:
\begin{align*}
\P ( B_{k+L} ) &= \P (B_{k+L} | B_k ) \P (B_k) + \P (B_{k+L} | B_k^c ) \P (B_k^c ) \\
&\geq \P (B_{k+L} | B_k \cap E_{k:k+L} ) \P (E_{k:k+L} | B_k ) \P (B_k) \\
&\quad + \P (B_{k+L} | B_k^c \cap E_{k:k+L} ) \P (E_{k:k+L} | B_k^c ) \P (B_k^c) \\
&= 1 \cdot \P (E_{k:k+L} | B_k ) \P (B_k) + p_{\min} \P (E_{k:k+L} | B_k^c ) (1 - \P ( B_k))
\end{align*}
We claim that $\P ( E_{k:k+L} | B_k)$ and $\P ( E_{k:k+L} | B_k^c )$ can be made arbitrarily high by taking the exploration phase lengths $T_k, \dots, T_{k+L}$ to be suitably large. Indeed, the Q-factors and J-factors for the $k^{th}$ exploration phase and onward are conditionally independent of the past up to time $t_k$, given the initial conditions, $Q^i_{t_k}, J^i_{t_k}, x_{t_k}$. Since Lemma~\ref{lemma:AY-lemma1} and Lemma~\ref{lemma:analog1} hold uniformly in their initial conditions (see \cite[Appendix A]{AY2017} for more details), this claim holds by a union bound argument, making $\P ( E_{k+l} | B_k )$ and $\P ( E_{k+l} | B_k^c)$ suitably large for each $0 \leq l \leq L$. (The final remark is enabled by Lemmas~\ref{lemma:AY-lemma4} and \ref{lemma:analog4}.)
We now consider what happens when $\P( E_{k:k+L} | B_k ) \geq u$ and $\P ( E_{k:k+L} | B_k^c ) \geq u$ for some $u \in (0,1)$. For fixed $k \in \{0, 1, \dots, L - 1 \}$, define the following quantities:
\begin{align*}
h^{(k)}_m &:= \P ( B_{k + mL} ), \quad \text{for } m \geq 0 \\
y^{(k)}_0 &:= \P ( B_k ) , \\
y^{(k)}_{m+1} &:= u y^{(k)}_m + u p_{\min} (1 - y^{(k)}_m ), \quad \text{for } m \geq 0
\end{align*}
We have that $h^{(k)}_{m+1} \geq u h^{(k)}_m + u p_{\min} (1 - h^{(k)}_m)$ by the law of total probability, conditioning on whether $B_{k+mL}$ occurred. It is easy to verify, then, that $h^{(k)}_m \geq y^{(k)}_m$ for all $m \geq 0$. (Since $h^{(k)}_0 = y^{(k)}_0$ and $h^{(k)}_{m+1} \geq (u - u p_{\min}) h^{(k)}_m + u p_{\min} \geq (u - u p_{\min}) y^{(k)}_m + u p_{\min} = y^{(k)}_{m+1}$.)
But then, by Lemma~\ref{lemma:recursion} with $u = u$ and $p = u p_{\min}$, we have that $\lim_{m \to \infty} y^{(k)}_m = \frac{ u p_{\min}}{1 - u + u p_{\min}}$.
Thus, we select $u^*$ sufficiently large that
\[
\frac{ u^* p_{\min}}{1 - u^* + u^* p_{\min}} > 1 - \frac \psi 2.
\]
Finally, by taking exploration phase lengths $\{ T_k \}_{k \geq 0}$ large enough that $\P ( E_{k:k+L} | B_k ) \geq u^*$ and $\P ( E_{k: k+L} | B_k^c ) \geq u^*$ hold for all $k \geq 0$, we get our result.
\hfill $\square$
\newpage
\vskip 0.2in
\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\begin{ack}
%Use unnumbered first level headings for the acknowledgments. All acknowledgments
%go at the end of the paper before the list of references. Moreover, you are required to declare
%funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
%More information about this disclosure can be found at: \url{https://neurips.cc/Conferences/2020/PaperInformation/FundingDisclosure}.
%
%
%Do {\bf not} include this section in the anonymized submission, only in the final paper. You can use the \texttt{ack} environment provided in the style file to autmoatically hide this section in the anonymized submission.
%\end{ack}
%\bibliographystyle{plain}
\bibliographystyle{ieeetr}
\bibliography{BoraPhD}
\end{document}
\begin{thebibliography}{10}
\bibitem{hernandez2017survey}
P.~Hernandez-Leal, M.~Kaisers, T.~Baarslag, and E.~M. de~Cote, ``A survey of
learning in multiagent environments: Dealing with non-stationarity,'' {\em
arXiv preprint arXiv:1707.09183}, 2017.
\bibitem{tan1993multi}
M.~Tan, ``Multi-agent reinforcement learning: Independent vs. cooperative
agents,'' in {\em Proceedings of the Tenth International Conference on
Machine Learning}, pp.~330--337, 1993.
\bibitem{Claus1998}
C.~Claus and C.~Boutilier, ``The dynamics of reinforcement learning in
cooperative multiagent systems,'' in {\em Proceedings of the Tenth Innovative
Applications of Artificial Intelligence Conference, Madison, Wisconsin},
pp.~746--752, 1998.
\bibitem{zhang2021multi}
K.~Zhang, Z.~Yang, and T.~Ba{\c{s}}ar, ``Multi-agent reinforcement learning: A
selective overview of theories and algorithms,'' {\em Handbook of
Reinforcement Learning and Control}, pp.~321--384, 2021.
\bibitem{matignon2012survey}
L.~Matignon, G.~J. Laurent, and N.~Le~Fort-Piat, ``Independent reinforcement
learners in cooperative markov games: a survey regarding coordination
problems.,'' {\em Knowledge Engineering Review}, vol.~27, no.~1, pp.~1--31,
2012.
\bibitem{matignon2009coordination}
L.~Matignon, G.~J. Laurent, and N.~Le~Fort-Piat, ``Coordination of independent
learners in cooperative {M}arkov games.,'' {\em HAL preprint hal-00370889},
2009.
\bibitem{daskalakis2021independent}
C.~Daskalakis, D.~J. Foster, and N.~Golowich, ``Independent policy gradient
methods for competitive reinforcement learning,'' {\em arXiv preprint
arXiv:2101.04233}, 2021.
\bibitem{wei2016lenient}
E.~Wei and S.~Luke, ``Lenient learning in independent-learner stochastic
cooperative games,'' {\em The Journal of Machine Learning Research}, vol.~17,
no.~1, pp.~2914--2955, 2016.
\bibitem{Marden2009payoff}
J.~R. Marden, H.~P. Young, G.~Arslan, and J.~S. Shamma, ``Payoff-based dynamics
for multiplayer weakly acyclic games,'' {\em SIAM Journal on Control and
Optimization}, vol.~48, no.~1, pp.~373--396, 2009.
\bibitem{marden2012revisiting}
J.~R. Marden and J.~S. Shamma, ``Revisiting log-linear learning: Asynchrony,
completeness and payoff-based implementation,'' {\em Games and Economic
Behavior}, vol.~75, no.~2, pp.~788--808, 2012.
\bibitem{foster2006regret}
D.~P. Foster and H.~P. Young, ``Regret testing: Learning to play {N}ash
equilibrium without knowing you have an opponent,'' {\em Theoretical
Economics}, vol.~1, pp.~341--367, 2006.
\bibitem{AY2017}
G.~Arslan and S.~Y\"{u}ksel, ``Decentralized {Q}-learning for stochastic teams
and games,'' {\em IEEE Transactions on Automatic Control}, vol.~62, no.~4,
pp.~1545--1558, 2017.
\bibitem{Brown1951iterative}
G.~Brown, ``Iterative solutions of games by fictitious play,'' in {\em Activity
Analysis of Production and Allocation} (T.~Koopmans, ed.), pp.~374--376, New
York: Wiley, 1951.
\bibitem{Robinson1951iterative}
J.~Robinson, ``An iterative method of solving a game,'' {\em Annals of
Mathematics}, vol.~{\bf 54}, pp.~296--301, 1951.
\bibitem{swenson2018distributed}
B.~Swenson, C.~Eksin, S.~Kar, and A.~Ribeiro, ``Distributed inertial
best-response dynamics,'' {\em IEEE Transactions on Automatic Control},
vol.~63, no.~12, pp.~4294--4300, 2018.
\bibitem{eksin2017distributed}
C.~Eksin and A.~Ribeiro, ``Distributed fictitious play for multiagent systems
in uncertain environments,'' {\em IEEE Transactions on Automatic Control},
vol.~63, no.~4, pp.~1177--1184, 2017.
\bibitem{sen94}
S.~Sen, M.~Sekaran, and J.~Hale, ``Learning to coordinate without sharing
information,'' in {\em Proceedings of the 12th National Conference on
Artificial Intelligence}, pp.~426--431, 1994.
\bibitem{Littman1994}
M.~L. Littman, ``Markov games as a framework for multi-agent reinforcement
learning,'' in {\em Machine Learning Proceedings 1994}, pp.~157--163,
Elsevier, 1994.
\bibitem{Hu2003}
J.~Hu and M.~P. Wellman, ``Nash q-learning for general-sum stochastic games,''
{\em Journal of Machine Learning Research}, vol.~4, no.~Nov, pp.~1039--1069,
2003.
\bibitem{Littman2001ffq}
M.~L. Littman, ``Friend-or-foe {Q}-learning in general-sum games,'' in {\em
ICML}, vol.~1, pp.~322--328, 2001.
\bibitem{lauer2000}
M.~Lauer and M.~Riedmiller, ``An algorithm for distributed reinforcement
learning in cooperative multi-agent systems,'' in {\em In Proceedings of the
Seventeenth International Conference on Machine Learning}, Citeseer, 2000.
\bibitem{Bowling}
M.~Bowling and M.~Veloso, ``Multiagent learning using a variable learning
rate,'' {\em Artificial Intelligence}, vol.~136, pp.~215--250, 2002.
\bibitem{germano2007global}
F.~Germano and G.~Lugosi, ``Global {N}ash convergence of {F}oster and {Y}oung's
regret testing,'' {\em Games and Economic Behavior}, vol.~60, no.~1,
pp.~135--154, 2007.
\bibitem{Chasparis2013aspiration}
G.~C. Chasparis, A.~Arapostathis, and J.~S. Shamma, ``Aspiration learning in
coordination games,'' {\em {SIAM} J. Control and Optimization}, vol.~51,
no.~1, pp.~465--490, 2013.
\bibitem{marden2014achieving}
J.~R. Marden, H.~P. Young, and L.~Y. Pao, ``Achieving {P}areto optimality
through distributed learning,'' {\em SIAM Journal on Control and
Optimization}, vol.~52, no.~5, pp.~2753--2770, 2014.
\bibitem{matignon2007hysteretic}
L.~Matignon, G.~J. Laurent, and N.~Le~Fort-Piat, ``Hysteretic {Q}-learning: an
algorithm for decentralized reinforcement learning in cooperative multi-agent
teams,'' in {\em 2007 IEEE/RSJ International Conference on Intelligent Robots
and Systems}, pp.~64--69, IEEE, 2007.
\bibitem{sayin2021decentralized}
M.~O. Sayin, K.~Zhang, D.~S. Leslie, T.~Basar, and A.~Ozdaglar, ``Decentralized
{Q}-learning in zero-sum {M}arkov games,'' {\em arXiv preprint
arXiv:2106.02748}, 2021.
\bibitem{YAY-TAC}
B.~Yongacoglu, G.~Arslan, and S.~Y{\"u}ksel, ``Decentralized learning for
optimality in stochastic dynamic teams and games with local control and
global state information,'' {\em IEEE Transactions on Automatic Control}, to
appear.
\bibitem{fink1964equilibrium}
A.~M. Fink, ``Equilibrium in a stochastic $n$-person game,'' {\em Journal of
Science of the Hiroshima University, Series AI (Mathematics)}, vol.~28,
no.~1, pp.~89--93, 1964.
\bibitem{Watkins89}
C.~Watkins, {\em Learning from Delayed Rewards}.
\newblock PhD thesis, Cambridge University, 1989.
\bibitem{WatkinsDayan92}
C.~Watkins and P.~Dayan, ``Q-{L}earning,'' {\em Machine Learning}, vol.~{{\bf{
8}}}, no.~3, pp.~279--292, 1992.
\bibitem{tsitsiklis1994asynchronous}
J.~N. Tsitsiklis, ``Asynchronous stochastic approximation and {Q}-learning,''
{\em Machine Learning}, vol.~16, no.~3, pp.~185--202, 1994.
\bibitem{simon1956rational}
H.~A. Simon, ``Rational choice and the structure of the environment.,'' {\em
Psychological review}, vol.~63, no.~2, p.~129, 1956.
\bibitem{radner1975satisficing}
R.~Radner, ``Satisficing,'' in {\em Optimization Techniques IFIP Technical
Conference}, pp.~252--263, Springer, 1975.
\bibitem{cassidy1972solution}
R.~Cassidy, C.~Field, and M.~Kirby, ``Solution of a satisficing model for
random payoff games,'' {\em Management Science}, vol.~19, no.~3,
pp.~266--271, 1972.
\bibitem{charnes1963deterministic}
A.~Charnes and W.~W. Cooper, ``Deterministic equivalents for optimizing and
satisficing under chance constraints,'' {\em Operations Research}, vol.~11,
no.~1, pp.~18--39, 1963.
\bibitem{candogan2013near}
O.~Candogan, A.~Ozdaglar, and P.~A. Parrilo, ``Near-potential games: Geometry
and dynamics,'' {\em ACM Transactions on Economics and Computation (TEAC)},
vol.~1, no.~2, pp.~1--32, 2013.
\bibitem{zhang2021gradient}
R.~Zhang, Z.~Ren, and N.~Li, ``Gradient play in multi-agent {M}arkov stochastic
games: Stationary points and convergence,'' {\em arXiv preprint
arXiv:2106.00198}, 2021.
\bibitem{awasthi2010nash}
P.~Awasthi, M.-F. Balcan, A.~Blum, O.~Sheffet, and S.~Vempala, ``On
{N}ash-equilibria of approximation-stable games,'' in {\em International
Symposium on Algorithmic Game Theory}, pp.~78--89, Springer, 2010.
\end{thebibliography}