A Reduction from Reinforcement Learning to No-Regret Online Learning
Extracted source text ยท arXiv 1911.05873
\documentclass[twoside]{article}
\usepackage[accepted]{aistats2020}
% If your paper is accepted, change the options for the package
% aistats2020 as follows:
%
% \usepackage[accepted]{aistats2020}
%
% This option will print headings for the title of your paper and
% headings for the authors names, plus a copyright note at the end of
% the first column of the first page.
% If you set papersize explicitly, activate the following three lines:
%\special{papersize = 8.5in, 11in}
%\setlength{\pdfpageheight}{11in}
%\setlength{\pdfpagewidth}{8.5in}
% If you use natbib package, activate the following three lines:
\usepackage[numbers,sort&compress]{natbib}
%\usepackage[round]{natbib}
\renewcommand{\bibname}{References}
\renewcommand{\bibsection}{\subsubsection*{\bibname}}
% If you use BibTeX in apalike style, activate the following line:
%\bibliographystyle{apalike}
\bibliographystyle{unsrtnat}
%
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage[hidelinks]{hyperref} % hyperlinks % hidelinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
%\usepackage{subcaption}
\usepackage{graphicx}
\usepackage{subfig}
\graphicspath{{figures/} }
\usepackage{xspace}
\input{shortcuts}
\newtheorem*{theorem*}{Theorem}
\newtheorem*{lemma*}{Lemma}
\newtheorem*{proposition*}{Proposition}
\def\mub{{\bm{\mu}}}
\def\pib{{\bm{\pi}}}
\def\Pib{{\bm{\Pi}}}
\def\etab{{\bm{\eta}}}
\def\Phib{{\bm{\Phi}}}
\def\Psib{{\bm{\Psi}}}
\def\phib{{\bm{\phi}}}
\def\psib{{\bm{\psi}}}
\def\thetab{{\bm{\theta}}}
\def\deltab{{\bm{\delta}}}
\def\alphab{{\bm{\alpha}}}
\def\betab{{\bm{\beta}}}
\newcommand{\cheng}[1]{{\color{RubineRed}\textbf{Ching-An:} #1}}
\newcommand{\boots}[1]{{\color{red}\textbf{BB:} #1}}
\newcommand{\tachet}[1]{{\color{Blue}\textbf{Remi:} #1}}
\newcommand{\paper}[1]{{\color{PineGreen} \textbf{#1}}\newline}
\def\piccolo{\textsc{PicCoLO}\xspace}
\def\piccolofull{\textbf{P}red\textbf{ic}tor-\textbf{Co}rrector On\textbf{l}ine \textbf{O}ptimization\xspace}
\def\Sol{\textrm{Sol}}
\def\VI{\textrm{VI}}
\def\DVI{\textrm{DVI}}
\def\EP{\textrm{EP}}
\def\DEP{\textrm{DEP}}
\def\FP{\textrm{FP}}
\def\gap{\mathrm{gap}}
\newcommand{\since}[1]{\text{($\because$ #1)}}
\newcommand{\aand}{\qquad \text{and} \qquad}
%\renewcommand{\red}[1]{\leavevmode{\color{red}{#1}}}
\renewcommand{\red}[1]{\leavevmode{\color{black}{#1}}}
\renewcommand{\blue}[1]{\leavevmode{\color{black}{#1}}}
%\title{A Reduction from Reinforcement Learning to No-Regret Online Learning}
\begin{document}
% If your paper is accepted and the title of your paper is very long,
% the style will print as headings an error message. Use the following
% command to supply a shorter title of your paper so that it can be
% used as headings.
%
%\runningtitle{I use this title instead because the last one was very long}
% If your paper is accepted and the number of authors is large, the
% style will print as headings an error message. Use the following
% command to supply a shorter version of the authors names so that
% they can be used as headings (for example, use only the surnames)
%
%\runningauthor{Surname 1, Surname 2, Surname 3, ...., Surname n}
\twocolumn[
\runningtitle{A Reduction from Reinforcement Learning to Online Learning}
\aistatstitle{
A Reduction from Reinforcement Learning to \\No-Regret Online Learning
}
%\aistatsauthor{Anonymous Authors}
%\aistatsaddress{Anonymous Institutions} ]
\aistatsauthor{ Ching-An Cheng \And Remi Tachet des Combes \And Byron Boots \And Geoff Gordon}
\aistatsaddress{ Georgia Tech \And Microsoft Research Montreal \And UW \And Microsoft Research Montreal } ]
\begin{abstract}
%Reinforcement learning (RL) is fundamental to sequential decision making.
%
%In this work,
We present a reduction from reinforcement learning (RL) to no-regret online learning based on the saddle-point formulation of RL, by which \emph{any} online algorithm with sublinear regret can generate policies with provable performance guarantees.
%
This new perspective decouples the RL problem into two parts: regret minimization and function approximation.
%
The first part admits a standard online-learning analysis, and the second part can be quantified independently of the learning algorithm.
%
Therefore, the proposed reduction can be used as a tool to systematically design new RL algorithms.
%
We demonstrate this idea by devising a simple RL algorithm based on mirror descent and the generative-model oracle.
For any $\gamma$-discounted tabular RL problem, with probability at least $1-\delta$, it learns an $\epsilon$-optimal policy using at most
$\tilde{O}\left(\frac{|\SS||\AA|\log(\frac{1}{\delta})}{(1-\gamma)^4\epsilon^2}\right)$ samples.
Furthermore, this algorithm admits a direct extension to linearly parameterized function approximators for large-scale applications, \red{with computation and sample complexities independent of $|\SS|$,$|\AA|$, though at the cost of potential approximation bias.}
%still depend on $|\SS||\AA|$.
%
\end{abstract}
\section{INTRODUCTION}
%%Importance and difficulties of reinforcement learning: complex, noisy, sample inefficient, difficulty to provide guarantees.
Reinforcement learning (RL) is a fundamental problem for sequential decision making in unknown environments.
One of its core difficulties, however, is the need for algorithms to infer long-term consequences based on limited, noisy, short-term feedback.
As a result, designing RL algorithms that are both scalable and provably sample efficient has been challenging.
%reduction from rl to online learning by combining recent advances from two fields: online learning, equilibrium
In this work, we revisit the classic linear-program (LP) formulation of RL~\citep{manne1959linear,denardo1968multichain} in an attempt to tackle this long-standing question.
%This LP problem has been extensively studied, including its primal, dual, and linearly approximated forms~\citep{de2003linear,de2004constraint,desai2009smoothed,desai2012approximate,wang2007dual,petrik2010feature,pazis2011non,sutter2017data,abbasi2014linear}. Despite nice theoretical properties, when state and action spaces are large, a direct application of classic LP algorithms, like the simplex method\footnote{The simplex method with a certain block pivoting strategy turns out to be the same as policy iteration~\cite{ye2011simplex}.}, is usually computationally impractical.
%
%As a workaround,
We focus on the associated saddle-point problem of the LP (given by Lagrange duality), which has recently gained traction due to its potential for computationally efficient algorithms with theoretical guarantees~\citep{wang2016online,chen2016stochastic,wang2017randomized,lee2018stochastic,wang2017primal,lin2017revisiting,dai2017boosting,chen2018scalable,lakshminarayanan2018linearly}.
%
But in contrast to these previous works based on stochastic approximation,
here we consider a reformulation through the lens of online learning, i.e. regret minimization. Since the pioneering work of~\citet{gordon1999regret,zinkevich2003online}, online learning has evolved into a ubiquitous tool for systematic design and analysis of iterative algorithms.
%
Therefore, if we can identify a reduction from RL to online learning, we can potentially leverage it to build efficient RL algorithms.
We will show this idea is indeed feasible. We present a reduction by which \emph{any} no-regret online algorithm, after observing $N$ samples, can find a policy $\hat{\pi}_N$ in a policy class $\Pi$ satisfying
$V^{\hat{\pi}_N}(p) \geq V^{\pi^*}(p) - o(1) - \epsilon_{\Pi}$, where $V^\pi(p)$ is the accumulated reward of policy $\pi$ with respect to some unknown initial state distribution $p$, $\pi^*$ is the optimal policy, and $\epsilon_{\Pi}\geq0$ is a measure of the expressivity of $\Pi$ (see \cref{sec:function approximation} for definition).
% motivated by imitation learning (IL)~\citep{ross2011reduction,lee2018dynamic,cheng2018convergence,cheng2019accelerating}.
%online learning, equilibrium
Our reduction is built on a refinement of online learning, called Continuous Online Learning (COL), which was proposed to model problems where loss gradients across rounds change continuously with the learner's decisions~\citep{cheng2019online}. COL has a strong connection to equilibrium problems (EPs)~\citep{blum1994optimization,bianchi1996generalized}, and any monotone EP (including our saddle-point problem of interest) can be framed as no-regret learning in a properly constructed COL problem~\cite{cheng2019online}.
%
Using this idea, our reduction follows naturally by first converting an RL problem to an EP and then the EP to a COL problem.
Framing RL as COL reveals new insights into the relationship between approximate solutions to the saddle-point problem and approximately optimal policies.
%
Importantly, this new perspective shows that the RL problem can be separated into two parts: regret minimization and function approximation.
%
The first part admits standard treatments from the online learning literature, and the second part can be quantified \emph{independently} of the learning process.
For example, one can accelerate learning by adopting optimistic online algorithms~\citep{rakhlin2012online,cheng2018predictor} that account for the predictability in COL, without worrying about function approximators.
%
Because of these problem-agnostic features, the proposed reduction can be used to systematically design efficient RL algorithms with performance guarantees.
As a demonstration, we design an RL algorithm based on arguably the simplest online learning algorithm: mirror descent.
%
Assuming a generative model\footnote{In practice, it can be approximated by running a behavior policy with sufficient exploration~\citep{kearns1999finite}.}, %\footnote{One can sample from the unknown initial distribution and the transition at any state-action pair desired.},
we prove that, for \emph{any} tabular Markov decision process (MDP), with probability at least $1-\delta$, this algorithm learns an $\epsilon$-optimal policy for the $\gamma$-discounted accumulated reward, using at most
$\tilde{O}\left(\frac{|\SS||\AA|\log(\frac{1}{\delta})}{(1-\gamma)^4\epsilon^2}\right)$ samples, where $|\SS|$,$|\AA|$ are the sizes of state and action spaces, and $\gamma$ is the discount rate.
%
Furthermore, thanks to the separation property above, our algorithm admits a natural extension with linearly parameterized function approximators, \red{whose sample and per-round computation complexities are \emph{linear} in the number of parameters and independent of $|\SS|$,$|\AA|$, though at the cost of policy performance bias due to approximation error.}
%, where each update can be computed efficiently independent of the problem size and therefore can work in large-scale or even continuous problems.\footnote{The sample complexity bound doesn't change in general, so the overall runtime still depends linearly on $|\SS||\AA|$.}
This sample complexity improves the current best provable rate of the saddle-point RL setup~\citep{wang2016online,chen2016stochastic,wang2017randomized,lee2018stochastic} by a large factor of $\frac{|\SS|^2}{(1-\gamma)^2}$, \emph{without} making any assumption on the MDP.\footnote{\citep{wang2017randomized} has the same sample complexity but requires the MDP to be ergodic under any policy.}
%
\red{This improvement is attributed to our new online-learning-style analysis that uses a cleverly selected comparator in the regret definition.}
%
While it is possible to devise a minor modification of the previous stochastic mirror descent algorithms, e.g.~\citep{wang2017randomized}, achieving the same rate with our new analysis, we remark that our algorithm is considerably simpler and removes a projection required in previous work~\citep{wang2016online,chen2016stochastic,wang2017randomized,lee2018stochastic}.
%
Finally, we do note that the same sample complexity can also be achieved, e.g., by
model-based RL and (phased) Q-learning~\citep{kearns1999finite,kakade2003sample}. However, these methods either have super-linear runtime, with no obvious route for improvement, or could become unstable when using function approximators without further assumption.
\section{SETUP \& PRELIMINARIES}
%\subsection{Setup}
%
Let $\SS$ and $\AA$ be state and action spaces, which can be discrete or continuous.
We consider $\gamma$-discounted infinite-horizon problems for $\gamma \in [0,1)$. Our goal is to find a policy $\pi(a|s)$ that maximizes the discounted average return $V^\pi(p) \coloneqq \E_{s\sim p}[V^\pi(s)]$,
where $p$ is the initial state distribution,
\begin{align} \label{eq:value function}
\textstyle
\hspace{-2mm}
V^{\pi}(s) \coloneqq (1-\gamma)\E_{\xi \sim \rho_\pi(s)} \left[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t) \right]
\hspace{-1mm}
\end{align}
is the value function of $\pi$ at state $s$, $r:\SS\times\AA\to[0,1]$ is the reward function,
and $\rho_\pi(s)$ is the distribution of trajectory $\xi = s_0, a_0, s_1, \dots$ generated by running $\pi$ from $s_0=s$ in an MDP.
We assume that the initial distribution $p(s_0)$, the transition $\PP(s'|s,a)$, and the reward function $r(s,a)$ in the MDP are unknown but can be queried through a \emph{generative model}, i.e. we can sample $s_0$ from $p$, $s'$ from $\PP$, and $r(s,a)$ for any $s\in\SS$ and $a\in\AA$. %, but the knowledge of probability values is unavailable.
%Without loss of generality, we assume the reward function is known, as the learning difficulty here mostly comes from the unknown transition. %; a straightforward analysis applies to the general case.
%
We remark that the definition of $V^{\pi}$ in \eqref{eq:value function} contains a $(1-\gamma)$ factor. %, which converges to the average reward as $\gamma \to 1$.
We adopt this setup to make writing more compact. We denote the optimal policy as $\pi^*$ and its value function as $V^*$ for short.
%Let us first recap the duality between different RL formulations.
%Our reduction is based on the linear program formulation of the RL problem
\subsection{Duality in RL} \label{sec:duality and stationarity}
Our reduction is based on the linear-program (LP) formulation of RL. We provide a short recap here (please see \cref{app:different RL setups} and \citep{puterman2014markov} for details).
To show how $\max_\pi V^\pi(p)$ can be framed as a LP, let us define
the
{average} %\footnote{As $\gamma\to1$, $d^\pi$ is known as the limiting average state distribution, which is the same as the stationary distribution of the MDP under $\pi$, if one exists.}
state distribution under $\pi$,
$
d^\pi(s) \coloneqq (1-\gamma) \sum_{t=0}^{\infty} \gamma^t d_t^\pi(s)
$, where $d_t^\pi$ is the state distribution at time $t$ visited by running $\pi$ from $p$ (e.g. $d_0^\pi = p$). By construction, $d^\pi$ satisfies the stationarity property,
\begin{align} \label{eq:stationarity (distribution)}
d^\pi(s') = (1-\gamma)p(s') + \gamma \E_{s \sim d^\pi} \E_{a\sim \pi |s } [\PP(s'|s,a)].
\end{align}
With $d^\pi$, we can write $V^\pi(p) =\E_{s \sim d^\pi}\E_{a \sim \pi |s} \left[ r(s,a) \right]
$ and our objective $\max_\pi V^\pi(p)$ equivalently as:
\begin{align} \label{eq:LP (flow)}
\begin{split}
&\textstyle \max_{\mub \in \R^{|\SS||\AA|} : \mub \geq \zerob} \quad \rb^\t \mub \\
&\text{s.t.}\quad %& \mub \geq \zerob\\
\textstyle(1-\gamma)\pb + \gamma \Pb^\t \mub = \Eb^\t \mub
\end{split}
\end{align}
where $\rb \in \R^{|\SS||\AA|}$, $\pb \in \R^{|\SS|}$, and $\Pb \in \R^{|\SS||\AA|\times|\SS|}$ are vector forms of $r$, $p$, and $\PP$, respectively, and $\Eb = \Ib \otimes \oneb \in \R^{|\SS||\AA|\times|\SS|}$ (we use $|\cdot|$ to denote the cardinality of a set, $\otimes$ the Kronecker product, $\Ib\in\R^{|\SS|\times|\SS|}$ is the identity, and $\oneb \in \R^{|\AA|}$ the vector of ones). %\footnote{This definition is chosen such that actions of the same state are grouped along the $|\SS||\AA|$ indices.}
In \eqref{eq:LP (flow)}, $\SS$ and $\AA$ %are implicitly assumed to have
\blue{may seem to have finite cardinalities}, but the same formulation extends to countable or even continuous spaces (under proper regularity assumptions; see~\citep{hernandez2012discrete}).
We adopt this abuse of notation (emphasized by bold-faced symbols) for compactness.
The variable $\mub$ of the LP in \eqref{eq:LP (flow)} resembles a joint distribution $d^\pi(s) \pi(a|s)$. To see this, notice that the constraint in~\eqref{eq:LP (flow)} is reminiscent of~\eqref{eq:stationarity (distribution)}, and implies $\norm{\mub}_1 =1 $, i.e. $\mub$ is a probability distribution. Then one can show $\mu(s,a) = d^\pi(s) \pi(a|s)$ when the constraint is satisfied, which implies that \eqref{eq:LP (flow)} is the same as $\max_\pi V^\pi(p)$ and its solution $\mub^*$ corresponds to $\mu^*(s,a) = d^{\pi^*}(s) \pi^*(a|s)$ of the optimal policy $\pi^*$.
As \eqref{eq:LP (flow)} is a LP, it suggests looking at its dual, which turns out to be the classic LP formulation of RL\footnote{Our setup in~\eqref{eq:LP (value)} differs from the classic one in the $(1-\gamma)$ factor in the constraint due to the average setup.},
\begin{align} \label{eq:LP (value)}
\begin{split}
&\textstyle \min_{\vb\in\Rbb^{|\SS|}}\quad \pb^\t \vb \\
&\text{s.t.} \quad (1-\gamma) \rb + \gamma \Pb \vb \leq \Eb \vb.
\end{split}
\end{align}
It can be verified that for all $\pb >0$, the solution to~\eqref{eq:LP (value)} satisfies the Bellman equation~\citep{bellman1954theory}
%$V^*(s) = \max_{a\in\AA} (1-\gamma)r(s,a) + \gamma \E_{s'\sim\PP|s,a} \left[ V^*(s') \right] $
and therefore is the optimal value function $\vb^*$ (the vector form of $V^*$). We note that, for any policy $\pi$, $V^\pi$ by definition satisfies a stationarity property
\begin{align} \label{eq:statinoarity (value)}
\hspace{-2mm}
V^\pi(s) = \E_{a \sim \pi | s} \left[ (1-\gamma)r(s,a) + \gamma \E_{s'\sim \PP |s,a} \left[ V^\pi(s') \right] \right]
\hspace{-1mm}
\end{align}
which can be viewed as a dual equivalent of~\eqref{eq:stationarity (distribution)} for $d^\pi$. Because, for any $s\in\SS$ and $a\in\AA$, $r(s,a)$ is in $[0,1]$, \eqref{eq:statinoarity (value)} implies $V^\pi(s)$ lies in $[0,1]$ too.
\subsection{Toward RL: the Saddle-Point Setup} \label{sec:saddle-point setup}
The LP formulations above require knowing the probabilities $p$ and $\PP$ and are computationally inefficient. %, i.e. they are more planning than learning.
When only generative models are available (as in our setup), one can alternatively exploit the duality relationship between the two LPs in \eqref{eq:LP (flow)} and~\eqref{eq:LP (value)}, and frame RL as a saddle-point problem~\citep{wang2016online}.
Let us define
\begin{align} \label{eq:advantage function}
\textstyle
\ab_{\vb} \coloneqq \rb + \frac{1}{1-\gamma}(\gamma \Pb - \Eb )\vb
\end{align}
as the \emph{advantage function} with respect to $\vb$ (where $\vb$ is not necessarily a value function). Then the Lagrangian connecting the two LPs can be written as
\begin{align} \label{eq:Lagrangian (normalized)}
\LL(\vb, \mub)
\coloneqq \pb^\t \vb + \mub^\t \ab_\vb,
\end{align}
which leads to the saddle-point formulation,
\begin{align} \label{eq:saddle-point problem of RL}
\min_{\vb \in \VV} \max_{\mub \in \MM} \LL(\vb, \mub),
\end{align}
where the constraints are
\begin{align}
\VV &= \{\vb \in \R^{|\SS|}: \vb \geq 0, \norm{\vb}_\infty \leq 1\} \label{eq:set of value} \\
\MM &= \{\mub\in \R^{|\SS||\AA|}: \mub \geq 0, \norm{\mub}_1 = 1 \}. \label{eq:set of flow}
\end{align}
The solution to \eqref{eq:saddle-point problem of RL} is exactly $(\vb^*, \mub^*)$, but notice that extra constraints on the norm of $\mub$ and $\vb$ are introduced in $\VV, \MM$, compared with \eqref{eq:LP (flow)} and~\eqref{eq:LP (value)}.
This is a common practice, which uses known bound on the solutions of \eqref{eq:LP (flow)} and~\eqref{eq:LP (value)} (discussed above) to make the search spaces $\VV$ and $\MM$ in \eqref{eq:saddle-point problem of RL} compact and as small as possible so that optimization converges faster.
Having compact variable sets allows using first-order stochastic methods, such as stochastic mirror descent and mirror-prox~\citep{nemirovski2009robust,juditsky2011solving}, to efficiently solve the problem. These methods only require using the generative model to compute unbiased estimates of the gradients $\nabla_\vb \LL = \bb_\mub$ and $\nabla_\mub \LL = \ab_\vb$, where we define
\begin{align} \label{eq:consistency function}
\textstyle \bb_\mub \coloneqq \pb + \frac{1}{1-\gamma}(\gamma\Pb- \Eb)^\t\mub
\end{align}
as the \emph{balance function} with respect to $\mub$. $\bb_\mub$ measures whether $\mub$ violates the stationarity constraint in~\eqref{eq:LP (flow)} and can be viewed as the dual of $\ab_\vb$.
%
When the state or action space is too large, one can resort to function approximators to represent $\vb$ and $\mub$, which are often realized by linear basis functions for the sake of analysis~\citep{chen2018scalable}.
\subsection{COL and EPs}
\label{sec:COL and EPs}
Finally, we review the COL setup in~\citep{cheng2019online}, which we will use to design the reduction from the saddle-point problem in \eqref{eq:saddle-point problem of RL} to online learning in the next section.
%We present a reduction from RL to no-regret online learning.
Recall that an online learning problem describes the iterative interactions between a learner and an opponent. In round $n$, the learner chooses a decision $x_n$ from a decision set $\XX$, the opponent chooses a per-round loss function $l_n:\XX \to \R$ based on the learner's decisions, and then information about $l_n$ (e.g. its gradient $\nabla l_n(x_n)$) is revealed to the learner. %In this paper, we will focus on the case where $l_n$ is a convex function in the decision variable.
The performance of the learner is usually measured in terms of regret with respect to some $x'\in\XX$,
\begin{align*}
\textstyle
\regret_N(x')\coloneqq \sum_{n=1}^{N} l_n(x_n) - \sum_{n=1}^N l_n(x').
\end{align*}
% this needs to be on page 3
When $l_n$ is convex and $\XX$ is compact and convex, many no-regret (i.e. $\regret_N(x') = o(N)$) algorithms are available, such as mirror descent and follow-the-regularized-leader~\citep{cesa2006prediction,shalev2012online,hazan2016introduction}.
% COL motivated by trial-and-error and imitation learning~\citep{ross2011reduction,lee2018dynamic,cheng2018convergence,cheng2019accelerating}
%Our reduction is based on a recent work on continuous online learning
COL is a subclass of online learning problems where the loss sequence changes continuously with respect to the played decisions of the learner~\citep{cheng2019online}.
%This setup was proposed to better describe online learning faced in practice, where the losses and feedback are not completely adversarial.
%
In COL, the opponent is equipped with a bifunction $f:(x,x') \mapsto f_{x}(x')$, where any fixed $x'\in\XX$, %$f_x(\cdot)$ is convex, and
$\nabla f_x (x')$ is continuous in $x \in \XX$. %\footnote{We define $\nabla f_{x}(x')$ as the derivative with respect to $x'$.}
The opponent selects per-round losses based on $f$, but the learner does not know $f$: in round $n$, if the learner chooses $x_n$, the opponent sets
\begin{align} \label{eq:regular per-round loss_main}
l_n(x) = f_{x_n} (x),
\end{align}
and returns, e.g., a stochastic estimate of $\nabla l_n(x_n)$ (the regret is still measured in terms of the noise-free $l_n$).
%In $f_{x}(x')$, $x$ and $x'$ are called the \emph{query} and \emph{decision} arguments, respectively.
%In general, $f$ can be defined online as queried, but the same $f_{x}(\cdot)$ must be returned when the learner plays the same $x$.
%An important property of COL problems is that, due to continuity, the loss sequences are \emph{predictable} from the past information. We will leverage this fact to design accelerated algorithms later.
In \citep{cheng2019online}, a natural connection is shown between COL and equilibrium problems (EPs). As EPs include the saddle-point problem of interest, we can use this idea to turn \eqref{eq:saddle-point problem of RL} into a COL problem.
Recall an EP is defined as follows: Let $\XX$ be compact and $F: (x, x') \mapsto F(x,x')$ be a bifunction s.t. $\forall x,x'\in\XX$, $F(\cdot, x')$ is continuous, $F(x, \cdot)$ is convex, and $F(x,x)\geq0$.\footnote{We restrict ourselves to this convex and continuous case as it is sufficient for our problem setup.} The problem $\EP(\XX,F)$ aims to find $x^\* \in \XX$ s.t.
\begin{align} \label{eq:EP}
F(x^\*,x) \geq 0, \qquad \forall x \in \XX.
\end{align}
%Its dual problem $\DEP(\XX,F)$ finds $x_{\*\*} \in \XX$ such that
%\begin{align} \label{eq:DEP}
%F(x, x_{\*\*}) \leq 0, \qquad \forall x \in \XX
%\end{align}
By its definition, a natural residual function to quantify the quality of an approximation solution $x$ to EP is
$
r_{ep}(x) \coloneqq - \min_{x'\in\XX}F(x,x')
$
which describes the degree to which~\eqref{eq:EP} is violated at $x$.
%A residual for the DEP can be defined similarly
%\begin{align} \label{eq:DEP residual}
%r_{dep}(x) \coloneqq \max_{x'\in\XX} F(x',x)
%\end{align}
%Therefore, EPs are sometimes called {maxInf} problems~\citep{jofre2014variational}. %, because $ x^\* \in \argmin_{x\in\XX} r_{ep}(x) = \argmax_{x\in\XX} \min_{x'\in\XX}F(x,x') $.
%In a special case where $F(\cdot, x)$ is concave, the EP reduces to a saddle-point problem.
We say a bifunction $F$ is \emph{monotone} if, $\forall x,x'\in\XX$, $F(x,x') + F(x',x) \leq 0$, and \emph{skew-symmetric} if the equality holds.
EPs with monotone bifunctions represent general convex problems, including convex optimization problems, saddle-point problems, variational inequalities, etc. For instance, a convex-concave problem $\min_{y\in\YY} \max_{z\in\ZZ} \phi(y, z)$ can be cast as $\EP(\XX,F)$ with $\XX = \YY\times\ZZ$ and the skew-symmetric bifunction \citep{jofre2014variational}
\begin{align} \label{eq:EP bifunction_main}
F(x,x') \coloneqq \phi(y', z) - \phi(y, z'),
\end{align}
where $x = (y,z)$ and $x' = (y',z')$. In this case, $r_{ep}(x) = \max_{z'\in\ZZ} \phi(y,z') - \min_{y'\in\YY}\phi(y',z)$ is the duality gap.
\citet{cheng2019online} show that a learner achieves sublinear dynamic regret in COL if and only if the same algorithm can solve $\EP(\XX,F)$ with $F(x,x') = f_x(x') - f_x(x)$. Concretely, they show that, given a monotone $\EP(\XX,F)$ with $F(x,x)=0$ (which is satisfied by \eqref{eq:EP bifunction_main}), one can construct a COL problem by setting $f_{x'}(x) \coloneqq F(x',x)$, i.e. $l_n(x) = F(x_n, x)$, such that any no-regret algorithm can generate an approximate solution to the EP. %They further show the following:
\begin{proposition}\label{pr:regret and residue} %(also satisfied by \eqref{eq:EP bifunction_main})
{\normalfont\citep{cheng2019online}}
If $F$ is \emph{skew-symmetric} and $l_n(x) = F(x_n, x)$, then $r_{ep}(\hat{x}_N) \leq \frac{1}{N}\regret_N$, where $\regret_N = \max_{x\in\XX}\regret_N(x)$, and $\hat{x}_N = \frac{1}{N} \sum_{n=1}^N x_n$; the same guarantee holds also for the best decision in $\{x_n\}_{n=1}^N$.
\end{proposition}
%
%
%
%\paragraph{LP in terms of stationary state-action distribution}
%Define the Lagrangian function
%\begin{align} \label{eq:Lagrangian}
%\LL(\vb, \fb) \coloneqq \pb^\t \vb + \fb^\t ((1-\gamma)\rb + \gamma \Pb \vb - \Eb \vb)
%\end{align}
%where $\fb\geq\zerob \in \R^{|\SS||\AA|}$ is the Lagrangian multiplier. %for the inequality constraint.
%By Lagrangian duality, the dual problem of~\eqref{eq:LP (value)} is given as
%$
%\max_{\fb\geq \zerob} \left(\min_{\vb} \LL(\vb, \fb) \right)$.
%Or after substituting the optimality condition of $\vb$
%%$0 = \nabla_{\vb}\LL(\vb, \fb) = \pb + \gamma \Pb^\t \fb - \Eb^\t \fb$
%and define $\mub \coloneqq (1-\gamma) \fb$,
%we can write the dual problem as another LP problem
%%\begin{align*}% \label{eq:LP (flow)}
%%\begin{split}
%%\max_{\fb} \quad & (1-\gamma)\rb^\t \fb \\
%%s.t. \quad & \fb \geq \zerob\\
%%& \pb + \gamma \Pb^\t \fb = \Eb^\t \fb
%%\end{split}
%%\end{align*}
\section{AN ONLINE LEARNING VIEW} \label{sec:revist}
We present an alternate online-learning perspective on the saddle-point formulation in \eqref{eq:saddle-point problem of RL}.
This analysis paves a way for of our reduction in the next section. By reduction, we mean realizing the two steps below: %
\begin{enumerate}
\item
Define a sequence of online losses such that any algorithm with sublinear regret can produce an approximate solution to the saddle-point problem.
\item
Convert the approximate solution in the first step to an approximately optimal policy in RL.
\end{enumerate}
Methods to achieve these two steps individually are not new.
The reduction from convex-concave problems to no-regret online learning is well known~\citep{abernethy2011blackwell}. Likewise, the relationship between the approximate solution of \eqref{eq:saddle-point problem of RL} and policy performance is also available; this is how the saddle-point formulation~\citep{wang2017randomized} works in the first place. So couldn't we just use these existing approaches? % to design the reduction?
%
We argue that purely combining these two techniques fails to fully capture important structure that resides in RL.
While this will be made precise in the later analyses, we highlight the main insights here.
Instead of treating \eqref{eq:saddle-point problem of RL} as an \emph{adversarial} two-player online learning problem~\citep{abernethy2011blackwell}, we adopt the recent reduction to COL~\cite{cheng2019online} reviewed in \cref{sec:COL and EPs}.
%
The main difference is that the COL approach takes a single-player setup and retains the Lipschitz continuity in the source saddle-point problem.
%
This single-player perspective is in some sense cleaner and, as we will show in \cref{sec:function approximation}, provides a simple setup to analyze effects of function approximators.
%
Additionally, due to continuity, the losses in COL are predictable and therefore make designing fast algorithms possible.
With the help of the COL reformulation, we study the relationship between the approximate solution to~\eqref{eq:saddle-point problem of RL} %(returned by some no-regret algorithm)
and the performance of the associated policy in RL.
%Freed from algebraic complications,
We are able to establish a tight bound between the residual and the performance gap, resulting in a large improvement of $\frac{|\SS|^2}{(1-\gamma)^2}$ in sample complexity compared with the best bounds in the literature of the saddle-point setup, \emph{without} adding extra constraints on $\XX$ and assumptions on the MDP. Overall, this means that \emph{stronger} sample complexity guarantees can be attained by \emph{simpler} algorithms, as we demonstrate in \cref{sec:algorithm demos}.
%we surprisingly found that the variable set $\VV = \{\vb \in \R^{|\SS|}: \vb \geq 0, \norm{\vb}_\infty \leq 1\}$ in \eqref{eq:set of value}, used throughout the literature, actually hinders the policy performance, though it was proposed based on insights about the property of $\vb^*$ (cf. \cref{sec:saddle-point setup}
%Freed from algebraic complications, we surprisingly found that the variable set $\VV = \{\vb \in \R^{|\SS|}: \vb \geq 0, \norm{\vb}_\infty \leq 1\}$ in \eqref{eq:set of value}, used throughout the literature, actually hinders the policy performance, though it was proposed based on insights about the property of $\vb^*$ (cf. \cref{sec:saddle-point setup}).
%
%Precisely, using $\VV$ in~\eqref{eq:set of value} introduces an undesirable factor, as large as $\frac{|\SS|^2}{(1-\gamma)^2}$, to the sample complexity, which we show is not due to analysis techniques but fundamentally due to the choice of $\VV$ in \eqref{eq:saddle-point problem of RL}. %, proposed by \citep{wang2016online} and adopted in all follow-up papers based on the saddle-point idea.
%Luckily, with the insights learned from the analyses, we are able to provide a simple fix in the next section, which is done by just making $\VV$ \emph{twice} larger \red{Isn't it $3^{\SS}$ times larger technically?}.
%
The missing proofs of this section are in \cref{app:missing proofs}.
\subsection{The COL Formulation of RL}
First, let us exercise the above COL idea with the saddle-point formulation of RL in \eqref{eq:saddle-point problem of RL}.
To construct the EP, we can let $\XX = \{x=(\vb, \mub): \vb\in\VV, \mub\in\MM \}$, which is compact.
According to~\eqref{eq:EP bifunction_main}, the bifunction $F$ of the associated $\EP(\XX,F)$ is naturally given as
\begin{align} \label{eq:bifunction (RL)}
F(x, x')
&\coloneqq \LL(\vb', \mub) - \LL(\vb, \mub') \nonumber \\
&= \pb^\t \vb' + \mub^\t \ab_{\vb'} - \pb^\t \vb - \mub'^\t \ab_\vb
\end{align}
which is skew-symmetric, and $x^* \coloneqq (\vv^*, \mub^*)$ is a solution to $\EP(\XX,F)$.
This identification gives us a COL problem with the loss in the $n$th round defined as
\begin{align} \label{eq:per-round loss of RL}
l_n(x) \coloneqq \pb^\t \vb + \mub_n^\t \ab_{\vb} - \pb^\t \vb_n - \mub^\t \ab_{\vb_n}
\end{align}
where $x_n = (\vb_n, \mub_n)$.
We see $l_n$ is a linear loss.
Moreover, because of the continuity in $\LL$, it is {predictable}, i.e. $l_n$ can be (partially) inferred from past feedback as the MDP involved in each round is the same.
\subsection{Policy Performance and Residual}
By \cref{pr:regret and residue}, any no-regret algorithm, when applied to \eqref{eq:per-round loss of RL}, provides guarantees in terms of the residual function $r_{ep}(x)$ of the EP.
%For example, if we adopt mirror descent with an unbiased estimate of $\nabla l_n(x_n)$, we have convergence in $O(N^{-1/2})$. % and one can easily verify the resultant algorithm is the same as applying stochastic mirror descent to \eqref{eq:saddle-point problem of RL}. %, recovering the algorithms in~\cite{wang2016online,wang2017randomized}.
But this is not the end of the story.
We also need to relate the learner decision $x \in \XX$ to a policy $\pi$ in RL and then convert bounds on $r_{ep}(x)$ back to the policy performance $V^\pi(p)$.
%
Here we follow the common rule in the literature and associate each $x = (\vb, \mub) \in \XX$ with a policy $\pi_\mub$ defined as
\begin{align}\label{eq:policy from flow}
\pi_\mub(a|s) \propto \mu(s,a).
\end{align}
%As shorthand, we will write $\pi_n \coloneqq \pi_{\mub_n}$ for the online decisions.
\blue{In the following, we relate the residual $r_{ep}(x)$ to the performance gap $V^*(p) - V^{\pi_{\mub}}(p)$
%More precisely, we will discuss upper bounds on the performance gap $V^*(p) - V^{\pi_{\mub}}(p)$ in terms of
through a relative performance measure defined as
\begin{align} \label{eq:relative residual}
r_{ep}(x;x') \coloneqq F(x,x) - F(x,x') = - F(x,x')
\end{align}
for $x,x'\in\XX$, where the last equality follows from the skew-symmetry of $F$ in \eqref{eq:bifunction (RL)}.
Intuitively, we can view $r_{ep}(x;x')$ as comparing the performance of $x$ with respect to the comparator $x'$ under an optimization problem proposed by $x$, e.g. we have $l_n(x_n) - l_n(x') = r_{ep}(x_n;x')$. And by the definition in \eqref{eq:relative residual}, it holds that $r_{ep}(x;x') \leq \max_{x'\in\XX}- F(x,x') = r_{ep}(x)$.
}
%For convenience, we define a relative performance measure for the skew-symmetric bifunction $F$ in \eqref{eq:bifunction (RL)}:
%. By \eqref{eq:relative residual},
%, for some properly chosen $x'\in\XX$.
%
%We now analyze the relationship between $r_{ep}(x;x')$ and $V^*(p) - V^{\pi_{\mub}}(p)$ for $x = (\vb, \mub) \in \XX$.
We are looking for inequalities in the form $V^*(p) - V^{\pi_\mub}(p) \leq \kappa(r_{ep}(x;x'))$ that hold for \emph{all} $x \in \XX$ with some strictly increasing function $\kappa$ and some $x'\in\XX$, so we can get \emph{non-asymptotic} performance guarantees once we combine the two steps described at the beginning of this section.
%
\blue{For example, by directly applying results of \citep{cheng2019online} to the COL in \eqref{eq:per-round loss of RL}, we get
$V^*(p) - V^{\hat{\pi}_N}(p) \leq \kappa(\frac{\regret_N}{N})$, where $\hat{\pi}_N$ is the policy associated with the average/best decision in $\{x_n\}^N_{1 = n}$.}
%, these results can translate the convergence of residual $r_{ep}(x)$ (determined by the regret in COL) to the convergence of the policy performance gap.
\subsubsection{The Classic Result} \label{sec:the classic result}
Existing approaches (e.g.~\citep{chen2016stochastic,wang2017randomized,lee2018stochastic}) to the saddle-point point formulation in \eqref{eq:saddle-point problem of RL} rely on the relative residual $r_{ep}(x;x^*)$ with respect to the optimal solution to the problem $x^*$, which we restate in our notation.
\begin{restatable}{proposition}{RoughResidueBound}
\label{pr:rough residual bound}
For any $x=(\vb,\mub)\in\XX$, if $\Eb^\t \mub \geq (1-\gamma) \pb$, $r_{ep}(x;x^*) \geq (1-\gamma) \min_s p(s) \norm{\vb^* - \vb^{\pi_\mub}}_\infty$.
\end{restatable}
Therefore, although the original saddle-point problem in \eqref{eq:saddle-point problem of RL} is framed using $\VV$ and $\MM$, in practice, an extra constraint, such as $\Eb^\t\mub \geq (1-\gamma)\pb$, is added into $\MM$, i.e. these algorithms consider instead
\begin{align}\label{eq:restricuted set of flow}
\MM' &= \{\mub\in \R^{|\SS||\AA|}: \mub\in\MM, \Eb^\t\mub \geq (1-\gamma)\pb \},
\end{align}
so that the marginal of the estimate $\mub$ can have the sufficient coverage required in \cref{pr:rough residual bound}. This condition is needed to establish non-asymptotic guarantees on the performance of the policy generated by $\mub$~\citep{wang2016online,wang2017randomized,lee2018stochastic}, but it can sometimes be impractical to realize, e.g., when $\pb$ is unknown. Without it, extra assumptions (like ergodicity~\citep{wang2017randomized}) on the MDP are needed.
%\cref{pr:rough residual bound} is the rationale behind the choice of \eqref{eq:restricuted set of flow}.
%
%The sufficient condition is chosen to contain the optimal solution (recall $\Eb^\t\mub^*\geq (1-\gamma)\pb$ by \eqref{eq:stationarity (distribution)}) and provide sufficient coverage of the state space.
%
However, \cref{pr:rough residual bound} is undesirable for a number of reasons. First, the bound is quite conservative, as it concerns the uniform error $\norm{\vb^* - \vb^{\pi_\mub} }_\infty$ whereas the objective in RL is about the gap $V^*(p) - V^{\pi_{\mub}}(p) = \pb^\t (\vb^*-\vb^{\pi_\mub})$ with respect to the initial distribution $p$ (i.e. a weighted error).
%
Second, the constant term $(1-\gamma) \min_s p(s)$ can be quite small (e.g. when $p$ is uniform, it is $\frac{1-\gamma}{|\SS|}$) which can significantly amplify the error in the residual.
%Suppose a fixed amount of samples from $p$ and $\PP$ are collected in each round of online learning to compute unbiased gradients.
%When the regret is in $O(N^{1/2})$, the residual converges in $O(N^{-1/2})$ and this factor can become a multiplier as large as $\frac{|\SS|^2}{(1-\gamma)^2}$ on the sample complexity.
Because a no-regret algorithm typically decreases the residual in $O(N^{-1/2})$ after seeing $N$ samples, the factor of $\frac{1-\gamma}{|\SS|}$ earlier would turn into a multiplier of $\frac{|\SS|^2}{(1-\gamma)^2}$ in sample complexity.
%
This makes existing saddle-point approaches sample inefficient in comparison with other RL methods like Q-learning~\citep{kakade2003sample}.
%
Lastly, enforcing $\Eb^\t \mub \geq (1-\gamma) \pb$ requires knowing $\pb$ (which is unavailable in our setup) and adds extra projection steps during optimization. When $\pb$ is unknown, while it is possible to modify this constraint to use a uniform distribution, this might worsen the constant factor and could introduce bias.
One may conjecture that the bound in \cref{pr:rough residual bound} could perhaps be tightened by better analyses. However, we prove this is impossible in general.
\begin{restatable}{proposition}{ResidueLowerBound} \label{pr:residual lower bound}
There is a class of MDPs such that, for some $x\in\XX$, \cref{pr:rough residual bound} is an equality.
\end{restatable}
We note that \cref{pr:residual lower bound} does not hold for \emph{all} MDPs.
Indeed, if one makes stronger assumptions on the MDP, such as that the Markov chain induced by \emph{every} policy is ergodic~\citep{wang2017randomized}, then it is possible to show, for all $x\in\XX$, $r_{ep}(x;x^*) = c \norm{\vb^* - \vb^{\pi_\mub}}_\infty$ for some constant $c$ independent of $\gamma$ and $|\SS|$, when one constrains $\Eb^\t\mub \geq (1-\gamma + \gamma \sqrt{c}) \pb$. Nonetheless, this construct still requires adding an undesirable constraint to $\XX$.
\subsubsection{Curse of Covariate Shift}
Why does this happen? We can view this issue as a form of \emph{covariate shift}, i.e. a mismatch between distributions. To better understand it, we notice a simple equality, which has often been used implicitly, e.g. in the technical proofs of \cite{wang2017randomized}. %; but here we provide an alternate and simplified proof.
\begin{restatable}{lemma}{ResidueEquality}\label{lm:residual}
For any $x = (\vb, \mub)$, if $x'\in\XX$ satisfies~\eqref{eq:stationarity (distribution)} and~\eqref{eq:statinoarity (value)} (i.e. $\vb'$ and $\mub'$ are the value function and state-action distribution of policy $\pi_{\mub'}$), $r_{ep}(x;x') = - \mub^\t \ab_{\vb'} $.
\end{restatable}
\cref{lm:residual} implies $r_{ep}(x;x^*) = - \mub^\t \ab_{\vb^*}$, which is non-negative. This term is similar to an equality called the performance difference lemma~\cite{ng1999policy,kakade2002approximately}.
\begin{restatable}{lemma}{PerformanceDifferenceLemma} \label{lm:performance difference lemma}
Let $\vb^\pi$ and $\mub^\pi$ denote the value and state-action distribution of some policy $\pi$. Then for \emph{any} function $\vb'$, it holds that
$
\pb^\t (\vb^\pi - \vb')
= (\mub^\pi)^\t \ab_{\vb'}
$. In particular, it implies $V^\pi(p) - V^{\pi'}(p) = (\mub^\pi)^\t \ab_{\vb^{\pi'}}$.
\end{restatable}
From \cref{lm:residual,lm:performance difference lemma}, we see that the difference between the residual $r_{ep}(x;x^*) = - \mub^\t \ab_{\vb^*}$ and the performance gap $V^{\pi_\mub}(p) - V^{\pi^*}(p) = (\mub^{\pi_\mub})^\t \ab_{\vb^*}$ is due to the mismatch between $\mub$ and $\mub^{\pi_\mub}$, or more specifically, the mismatch between the two marginals $\db = \Eb^\t \mub$ and $\db^{\pi_\mub} = \Eb^\t\mub^{\pi_\mub}$.
Indeed, when $\db = \db^{\pi_\mub}$, the residual is equal to the performance gap. However, in general, we do not have control over that difference for the sequence of variables $\{x_n = (\vb_n,\mub_n) \in \XX \}$ an algorithm generates.
The sufficient condition in \cref{pr:rough residual bound} attempts to mitigate the difference, using the fact $\db^{\pi_\mub} = (1-\gamma) \pb + \gamma \Pb_{\pi_\mub}^\t \db^{\pi_\mub}$ from \eqref{eq:stationarity (distribution)}, where $\Pb_{\pi_\mub}$ is the transition matrix under $\pi_\mub$. But the missing half $\gamma \Pb_{\pi_\mub}^\t \db^{\pi_\mub}$ (due to the long-term effects in the MDP) introduces the unavoidable, weak constant $ (1-\gamma) \min_s p(s)$, if we want to have an uniform bound on $\norm{\vb^* - \vb^{\pi_\mub}}_\infty$.
The counterexample in \cref{pr:residual lower bound} was designed to maximize the effect of covariate shift, so that $\mub$ fails to captures state-action pairs with high advantage. To break the curse, we must properly weight the gap between $\vb^*$ and $\vb^{\pi_\mub}$ instead of relying on the uniform bound on $\norm{\vb^* - \vb^{\pi_\mub}}_\infty$ as before.
%, making \emph{all} methods derived from the current saddle-point formulation or its direct reduction to online learning in \cref{sec:COL and EPs} inefficient in terms of policy performance.
\section{THE REDUCTION} \label{sec:the reduction}
The analyses above reveal both good and bad properties of the saddle-point setup in \eqref{eq:saddle-point problem of RL}.
On the one hand, we showed that approximate solutions to the saddle-point problem in \eqref{eq:saddle-point problem of RL} can be obtained by running any no-regret algorithm in the single-player COL problem defined in \eqref{eq:per-round loss of RL}; many efficient algorithms are available from the online learning literature. On the other hand, we also discovered a root difficulty in converting an approximate solution of \eqref{eq:saddle-point problem of RL} to an approximately optimal policy in RL (\cref{pr:rough residual bound}), even after imposing strong conditions like \eqref{eq:restricuted set of flow}.
%
At this point, one may wonder if the formulation based on \eqref{eq:saddle-point problem of RL} is fundamentally sample inefficient compared with other approaches to RL, but this is actually not true.
Our main contribution shows that learning a policy through running a no-regret algorithm in the COL problem in \eqref{eq:per-round loss of RL} is, in fact, as sample efficient in policy performance as other RL techniques, even without the common constraint in \eqref{eq:restricuted set of flow} or extra assumptions on the MDP like ergodicity imposed in the literature.
\begin{theorem}\label{th:reduction of RL}
Let $X_N = \{x_n\in\XX\}_{n=1}^N$ be \emph{any} sequence. Let $\hat{\pi}_N$ be the policy given by $\hat{x}_N$ via \eqref{eq:policy from flow}, which is either the average or the best decision in $X_N$. Define $y_N^* \coloneqq (\vb^{\hat{\pi}_N}, \mub^*)$. Then
$
\textstyle
V^{\hat{\pi}_N}(p) \geq V^*(p) - \frac{\regret_N(y_N^*)}{N}
$.
\end{theorem}
\cref{th:reduction of RL} shows that if $X_N$ has sublinear regret, then both the average policy and the best policy in $X_N$ converge to the optimal policy in performance with a rate $O(\regret_N(y_N^*)/N)$. Compared with existing results obtained through \cref{pr:rough residual bound}, the above result removes the factor $(1-\gamma) \min_s p(s)$ and impose \emph{no} assumption on $X_N$ or the MDP.
%
Indeed \cref{th:reduction of RL} holds for \emph{any} sequence. For example, when $X_N$ is generated by stochastic feedback of $l_n$, \cref{th:reduction of RL} continues to hold, as the regret is defined in terms of $l_n$, not of the sampled loss.
Stochasticity only affects the regret rate.
%Because online learning is non-anticipating, it is straightforward to derive bounds on regret in expectation or in high probability.
In other words, we have shown that when $\mub$ and $\vb$ can be directly parameterized, an approximately optimal policy for the RL problem can be obtained by running any no-regret online learning algorithm, and that the policy quality is simply dictated by the regret rate.
To illustrate, in \cref{sec:algorithm demos} we will prove that simply running mirror descent in this COL produces an RL algorithm that is as sample efficient as other common RL techniques.
%
One can further foresee that algorithms leveraging the continuity in COL---e.g. mirror-prox~\citep{juditsky2011solving} or PicCoLO~\citep{cheng2018predictor}---and variance reduction can lead to more sample efficient RL algorithms.
%are more sample efficient than those that ignore this predictable structure (e.g. mirror descent).
Below we will also demonstrate how to use the fact that COL is \emph{single-player} (see \cref{sec:COL and EPs}) to cleanly incorporate the effects of using function approximators to model $\mub$ and $\vb$. We will present a corollary of \cref{th:reduction of RL}, which separates the problem of \emph{learning} $\mub$ and $\vb$, and that of \emph{approximating} $\MM$ and $\VV$ with function approximators.
The first part is controlled by the rate of regret in online learning, and the second part depends on only the chosen class of function approximators, independently of the learning process.
%
As these properties are agnostic to problem setups and algorithms, our reduction leads to a framework for systematic synthesis of new RL algorithms with performance guarantees.
%For example, we obtain simpler versions of the algorithms proposed in \cite{wang2016online,wang2017randomized,chen2018scalable} but with improved performance guarantee. We will also show how to constructively design novel RL algorithms with better sample complexity, through translating recent advances in the online learning literature to solve the proposed COL problem.
The missing proofs of this section are in \cref{app:missing proofs of reduction}.
\subsection{Proof of \cref{th:reduction of RL}} \label{sec:proof of reduction}
The main insight of our reduction is to adopt, in defining $r_{ep}(x;x')$, a comparator $x'\in\XX$ based on the output of the algorithm (represented by $x$), instead of the fixed comparator $x^*$ (the optimal pair of value function and state-action distribution) that has been used conventionally, e.g. in \cref{pr:rough residual bound}.
While this idea seems unnatural from the standard saddle-point or EP perspective, it is possible, because the regret in online learning is measured against the worst-case choice in $\XX$, which is allowed to be selected in \emph{hindsight}. % (as reflected in $r_{ep}(x) = \max_{x'\in\XX} r_{ep}(x;x')$).
%
Specifically, we propose to select the following comparator to directly bound $ V^*(p) - V^{\hat{\pi}_N}(p)$ instead of the conservative measure $\norm{V^* - V^{\hat{\pi}_N}}_\infty$ used before.
\begin{restatable}{proposition}{CleverResidueBound}
\label{pr:clever residual bound}
For $x = (\vb, \mub)\in\XX$, define $y_x^* \coloneqq (\vb^{\pi_\mub}, \mub^*) \in \XX$. It holds $r_{ep}(x;y_x^*) = V^*(p) - V^{\pi_\mub}(p)$.
\end{restatable}
To finish the proof, let $\hat{x}_N $ be either $\frac{1}{N} \sum_{n=1}^N x_n$ or \red{$\argmin_{x\in X_N} r_{ep}(x;y_x^*)$,} and let $\hat{\pi}_N$ denote the policy given by~\eqref{eq:policy from flow}.
First, $V^*(p) - V^{\hat{\pi}_N}(p) = r_{ep}(\hat{x}_N; y_N^*)$ by \cref{pr:clever residual bound}. \red{Next we follow the proof idea of \cref{pr:regret and residue} in~\citep{cheng2019online}: because $F$ is skew-symmetric and $F(y_N^*, \cdot)$ is convex, we have by \eqref{eq:relative residual}}
\begin{align*}
&V^*(p) - V^{\hat{\pi}_N}(p) = r_{ep}(\hat{x}_N; y_N^*) = -F(\hat{x}_N,y_N^*)\\
&\textstyle = F(y_N^*, \hat{x}_N)\leq \frac{1}{N} \sum_{n=1}^N F(y_N^*, x_n) \\
&= \textstyle \frac{1}{N} \sum_{n=1}^N - F(x_n, y_N^*) = \frac{1}{N} \regret_N(y_N^*).
\end{align*}
%
%Because $F$ in \eqref{eq:bifunction (RL)} is skew-symmetric and $F(x,x)=0$, by Theorem 2 in \citep{cheng2019online}, we have $r_{ep}(\hat{x}_N) \leq \frac{1}{N}\regret_N$ (cf.~\cref{sec:COL and EPs}), where $\regret_N = \max_{x\in\XX}\regret_N(x)$.
%Finally, we use \cref{pr:clever residual bound} to relate this bound of COL to the policy performance gap.
%%
%As $y_N^* \in \XX$, it follows by the definition in \eqref{eq:relative residual} that $r_{ep}(\hat{x}_N) = \max_{x'\in\XX} r_{ep}(\hat{x}_N; x') \geq r_{ep}(\hat{x}_N; y_N^*)$. Therefore, we have proved
%$
%\frac{1}{N}\regret_N \geq r_{ep}(\hat{x}_N; y_N^*) = V^*(p) - V^{\hat{\pi}_N}(p)
%$ as desired.
\subsection{Function Approximators} \label{sec:function approximation}
When the state and action spaces are large or continuous, directly optimizing $\vb$ and $\mub$ can be impractical. Instead we can consider optimizing over a subset of feasible choices parameterized by function approximators
\begin{align} \label{eq:function approximators}
\XX_\Theta = \{ \xb_\theta = (\phib_{\theta}, \psib_\theta): \psib_\theta \in \MM, \theta \in \Theta \},
\end{align}
where $\phib_\theta$ and $\psib_\theta$ are functions parameterized by $\theta \in \Theta$, and $\Theta$ is a parameter set.
%
Because COL is a single-player setup, we can extend the previous idea and \cref{th:reduction of RL} to provide performance bounds in this case by a simple rearrangement (see \cref{app:missing proofs of reduction}), which is a common trick used in the online imitation learning literature~\citep{ross2011reduction,cheng2018convergence,cheng2018accelerating}.
Notice that, in \eqref{eq:function approximators}, we require only $\psib_\theta \in \MM$, but not $\phib_{\theta} \in \VV$, because for the performance bound in our reduction to hold, we only need the constraint $\MM$ (see \cref{lm:advantage bound} in proof of \cref{pr:clever residual bound}).%
\begin{restatable}{corollary}{ReductionForFunctionApporximator}\label{cr:reduction for funcapp}
Let $X_N = \{x_n\in\XX_\theta \}_{n=1}^N$ be any sequence. Let $\hat{\pi}_N$ be the policy given either by the average or the best decision in $X_N$. It holds that
\begin{align*}
\textstyle
V^{\hat{\pi}_N}(p) \geq V^*(p) - \frac{\normalfont{\regret}_N(\Theta)}{N} - \epsilon_{\Theta,N}
\end{align*}
where
$\epsilon_{\Theta,N} = \min_{x_\theta\in\XX_{\theta}} r_{ep}(\hat{x}_N;y_N^*) - r_{ep}(\hat{x}_N; x_\theta)$ measures the expressiveness of $X_\theta$, and
$\normalfont{\regret}_N(\Theta) \coloneqq \sum_{n=1}^{N} l_n(x_n) - \min_{x\in\XX_\Theta} \sum_{n=1}^N l_n(x)$.
\end{restatable}
We can quantify $\epsilon_{\Theta,N}$ with the basic H\"older's inequality.%
\begin{restatable}{proposition}{SizeOfEpsilon} \label{pr:size of epsilon}
Let $\hat{x}_N = (\hat{\vb}_N, \hat{\mub}_N)$.
Under the setup in \cref{cr:reduction for funcapp}, \emph{regardless} of the parameterization, it is true that $\epsilon_{\Theta,N}$ is no larger than
\begin{small}
\begin{align*}
&\hspace{-1mm}
\min_{(\vb_\theta,\mub_\theta) \in \XX_\Theta}
\hspace{-1mm}
\frac{\norm{\mub_\theta - \mub^*}_1}{1-\gamma} +
\min_{\wb:\wb\geq 1} \norm{\bb_{\hat{\mub}_N}}_{1,\wb} \norm{\vb_\theta - \vb^{\hat{\pi}_N}}_{\infty,1/\wb}
\\
&\leq
\min_{(\vb_\theta,\mub_\theta) \in \XX_\Theta }
\frac{1}{1-\gamma} \left( \norm{\mub_\theta - \mub^*}_1
+ 2 \norm{\vb_\theta - \vb^{\hat{\pi}_N}}_\infty \right).
\end{align*}
\end{small}%
where the norms are defined as $\norm{\xb}_{1,\wb} = \sum_i w_i |x_i|$ and $\norm{\xb}_{\infty,1/\wb} = \max_i w_i^{-1} |x_i|$.
\end{restatable}
\cref{pr:size of epsilon} says $\epsilon_{\Theta,N}$ depends on how well $\XX_\Theta$ captures the value function of the output policy $\vb^{\hat{\pi}_N}$ and the optimal state-action distribution $\mub^*$.
We remark that this result is independent of how $\vb^{\hat{\pi}_N}$ is generated.
%
Furthermore, \cref{pr:size of epsilon} makes \emph{no} assumption whatsoever on the structure of function approximators. It even allows sharing parameters $\theta$ between $\vb =\phib_{\theta}$ and $ \mub = \psib_\theta$, e.g., they can be a bi-headed neural network, which is common for learning shared feature representations. More precisely, the structure of the function approximator would only affect whether $l_n((\phib_\theta,\psib_\theta))$ remains a convex function in $\theta$, which determines the difficulty of designing algorithms with sublinear regret.
%\red{We discuss some potential parameterizations of $\phib_\theta$ and $\psib_\theta$ in \cref{app:parameterization}.}
In other words, the proposed COL formulation provides a reduction which dictates the policy performance with two separate factors: 1) the rate of regret $\regret_N(\Theta)$ which is controlled by the choice of online learning algorithm; 2) the approximation error $\epsilon_{\Theta,N}$ which is determined by the choice of function approximators. These two factors can almost be treated independently, except that the choice of function approximators would determine the properties of $l_n((\phib_\theta,\psib_\theta))$ as a function of $\theta$, and the choice of $\Theta$ needs to ensure \eqref{eq:function approximators} is admissible.
\section{SAMPLE COMPLEXITY OF MIRROR DESCENT}
\label{sec:algorithm demos}
\begin{algorithm}[t]
{\small
\caption{Mirror descent for RL}\label{alg:md for RL}
\begin{algorithmic} [1]
\renewcommand{\algorithmicensure}{\textbf{Input:}}
\renewcommand{\algorithmicrequire}{\textbf{Output:}}
\ENSURE $\epsilon$ optimality of the $\gamma$-average return\\
\hspace{5mm} $\delta$ maximal failure probability\\
\hspace{5mm} generative model of an MDP
\REQUIRE $\hat{\pi}_N = \pi^{\hat{\mub}_N}$
\STATE $x_1 = (\vb_1, \mub_1)$ where $\mub_1$ is uniform and $\vb_1 \in\VV$
\STATE Set $N = \tilde{\Omega}(\frac{|\SS||\AA|\log(\frac{1}{\delta})}{(1-\gamma)^2\epsilon^2})$ and $\eta = (1-\gamma)(|\SS||\AA|N)^{-1/2}$
\STATE Set the Bregman divergence as \eqref{eq:Bregman divergence choice}
\FOR {$n = 1\dots N-1$}
\STATE Sample $g_n$ according to \eqref{eq:stochastic gradient estimate}
\STATE Update to $x_{n+1}$ according to \eqref{eq:mirror descent}
\\
\ENDFOR
\STATE Set $ (\hat{\vb}_N, \hat{\mub}_N) = \hat{x}_N = \frac{1}{N}\sum_{n=1}^{N} x_n$
\end{algorithmic}
}
\end{algorithm}
We demonstrate the power of our reduction by applying perhaps the simplest online learning algorithm, mirror descent, to the proposed COL problem in \eqref{eq:per-round loss of RL} with stochastic feedback (\cref{alg:md for RL}).
%
For transparency, we discuss the tabular setup. We will show a natural extension to basis functions at the end.
%linearly parameterized function approximators at the end.
Recall that mirror descent is a first-order algorithm, whose update rule can be written as
\begin{align} \label{eq:mirror descent}
\textstyle
x_{n+1} = \argmin_{x\in\XX} \lr{g_n}{x} + \frac{1}{\eta} B_{R}(x||x_n)
\end{align}
where $\eta >0$ is the step size, $g_n$ is the feedback direction, and $B_R(x||x') = R(x) - R(x') - \lr{\nabla R(x')}{x - x'}$ is the Bregman divergence generated by a strictly convex function $R$.
Based on the geometry of $\XX = \VV \times \MM$, we consider a natural Bregman divergence of the form
\begin{align} \label{eq:Bregman divergence choice}
\textstyle
B_R(x'||x) = \frac{1}{2|\SS|} \norm{\vb' - \vb}_2^2 + KL(\mub'||\mub)
\end{align}
This choice mitigates the effects of dimension (e.g. if we set $x_1 = (\vb_1, \mub_1)$ with $\mub_1$ being the uniform distribution, it holds
$
B_R(x'||x_1) = \tilde{O}(1)
$ for any $x' \in\XX$).
To define the feedback direction $g_n$, we slightly modify the per-round loss $l_n$ in \eqref{eq:per-round loss of RL} and consider a new loss
\begin{align} \label{eq:modified per-round loss}
\textstyle
h_n(x) \coloneqq \bb_{\mub_n}^\t \vb + \mub^\t (\frac{1}{1-\gamma} \oneb - \ab_{\vb_n})
\end{align}
that shifts $l_n$ by a constant, where $\oneb$ is the vector of ones.
One can verify that $l_n(x)-l_n(x') = h_n(x) - h_n(x')$, for all $x,x' \in \XX$ \blue{when $\mub, \mub'$ in $x$ and $x'$ satisfy $\norm{\mub}_1 = \norm{\mub'}_1$ (which holds for \cref{alg:md for RL}).} Therefore, using $h_n$ does not change regret. The reason for using $h_n$ instead of $l_n$ is to make $\nabla_\mub h_n((\vb,\mub))$ (and its unbiased approximation) a positive vector, % (because $\frac{2}{1-\gamma} \geq \norm{\ab_{\vb}}_\infty$ for any $\vb \in \VV$),
so the regret bound can have a better dimension dependency. This is a common trick used in online learning (e.g. EXP3 \cite{journals/ftopt/Hazan16}) for optimizing variables living in a simplex ($\mub$ here).
We set the first-order feedback $g_n$ as an unbiased \emph{sampled} estimate of $\nabla h_n(x_n)$. In round $n$, this is realized by two independent calls of the generative model:
%\begin{small}
\begin{align} \label{eq:stochastic gradient estimate}
\hspace{-2mm} g_n
%= \begin{bmatrix}
%\gb_{n,v}\\
%\gb_{n,\mu}
%\end{bmatrix}
%
= \begin{bmatrix}
\tilde{\pb}_n + \frac{1}{1-\gamma}(\gamma\tilde{\Pb}_n -\Eb_n)^\t \tilde{\mub}_n \\
|\SS||\AA| (\frac{1}{1-\gamma
} \hat{\oneb}_n - \hat{\rb}_n - \frac{1}{1-\gamma}(\gamma\hat{\Pb}_n-\hat{\Eb}_n)\vb_n)
\end{bmatrix}
\end{align}
%\end{small}%
Let $g_n = [\gb_{n,v}; \gb_{n,\mu}]$.
For $\gb_{n,v}$, we sample $\pb$, sample $\mub_n$ to get a state-action pair, and query the transition $\Pb$ at the state-action pair sampled from $\mub_n$. ($\tilde{\pb}_n$, $\tilde{\Pb}_n$, and $\tilde{\mub}_n$ denote the single-sample estimate of these probabilities.)
%
For $\gb_{n,\mu}$, we first sample \emph{uniformly} a state-action pair (which explains the factor $|\SS||\AA|$), and then query the reward $\rb$ and the transition $\Pb$. ($\hat{\oneb}_n$, $\hat{\rb}_n$, $\hat{\Pb}_n$, and $\hat{\Eb}_n$ denote the single-sample estimates.) To emphasize, we use $\tilde{\cdot}$ and $\hat{\cdot}$ to distinguish the empirical quantities obtained by these two independent queries. By construction, we have $\gb_{n,\mu} \geq 0$.
It is clear that this direction $g_n$ is unbiased, i.e. $\E[g_n] = \nabla h_n(x_n)$. Moreover, it is extremely sparse and can be computed using $O(1)$ sample, computational, and memory complexities.
Below we show this algorithm, despite being extremely simple, has strong theoretical guarantees. In other words, we obtain simpler versions of the algorithms proposed in \citep{wang2016online,wang2017randomized,chen2018scalable} but with improved performance.
\begin{restatable}{theorem}{SampleComplexityOfMirrorDescent} \label{th:sample complexity of mirror descent}
With probability $1-\delta$, \cref{alg:md for RL} learns an $\epsilon$-optimal policy with
$
\tilde{O}\left(\frac{|\SS||\AA|\log(\frac{1}{\delta})}{(1-\gamma)^2\epsilon^2}\right)
$ samples.
\end{restatable}%
Note that the above statement makes no assumption on the MDP (except the tabular setup for simplifying analysis).
Also, because the definition of value function in \eqref{eq:value function} is scaled by a factor $(1-\gamma)$, the above result translates into a sample complexity in $\tilde{O}\left(\frac{|\SS||\AA|\log(\frac{1}{\delta})}{(1-\gamma)^4\epsilon^2}\right)$ for the conventional discounted accumulated rewards.
\subsection{Proof Sketch of \cref{th:sample complexity of mirror descent}} \label{sec:proof sketch of sample complexity of mirror descent}
The proof is based on the basic property of mirror descent and martingale concentration. We provide a sketch here; please refer to \cref{app:proof of sample complexity of mirror descent} for details.
Let $y_N^* = (\vb^{\hat{\pi}_N}, \mub^*)$.
We bound the regret in \cref{th:reduction of RL} by the following rearrangement, where the first equality below is because $h_n$ is a constant shift from $l_n$. %, and the inequality uses the fact $h_n(\cdot)$ is linear.
\begin{small}
\begin{align*}
&\regret_N (y_N^*)
= \sum_{n=1}^{N} h_n(x_n) - \sum_{n=1}^N h_n(y_N^* ) \nonumber \\
&\leq
\left(\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n \right)
+ \left( \max_{x\in\XX} \sum_{n=1}^N g_n^\t (x_n -x) \right)
\\ &\quad +
\left( \sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^* \right)
\end{align*}
\end{small}%
We recognize the first term is a martingale, because $x_n$ does not depend on $g_n$. Therefore, we can appeal to a Bernstein-type martingale concentration and prove it is in $\tilde{O}( \frac{\sqrt{N|\SS||\AA|\log(\frac{1}{\delta})}}{1-\gamma} )$. For the second term, by treating $g_n^\t x$ as the per-round loss, we can use standard regret analysis of mirror descent and show a bound in $\tilde{O}(\frac{\sqrt{N|\SS||\AA|}}{1-\gamma})$.
For the third term, because $\vb^{\hat{\pi}_N}$ in $y_N^* = (\vb^{\hat{\pi}_N}, \mub^*)$ depends on $\{g_n\}_{n=1}^N$, it is \emph{not} a martingale. Nonetheless, we are able to handle it through a union bound and show it is again no more than $\tilde{O}( \frac{\sqrt{N|\SS||\AA|\log(\frac{1}{\delta})}}{1-\gamma})$. Despite the union bound, it does not increase the rate because we only need to handle $\vb^{\hat{\pi}_N}$, not $\mub^*$ which induces a martingale.
%
To finish the proof, we substitute this high-probability regret bound into \cref{th:reduction of RL} to obtain the desired claim.
%
\subsection{Extension to Function Approximators}
The above algorithm assumes the tabular setup for illustration purposes.
\red{In \cref{app:sample complexity using function approximators}, we describe a direct extension of \cref{alg:md for RL} that uses linearly parameterized function approximators of the form $x_\theta = (\Phib\thetab_v, \Psib\thetab_\mu)$, where columns of bases $\Phib,\Psib$ belong to $\VV$ and $\MM$, respectively, and $(\thetab_v,\thetab_\mu) \in \Theta$. % are the parameters to learn.
Overall the algorithm stays the same, except the gradient is computed by chain-rule, which can be done in $O(dim(\Theta))$ time and space. While this seems worse, the computational complexity per update actually improves to $O(dim(\Theta))$ from the slow $O(|\SS||\AA|)$ (required before for the projection in \eqref{eq:mirror descent}), as now we only optimize in $\Theta$. %So the algorithm runs faster per iteration.
Moreover, we prove that its sample complexity is also better, though at the cost of bias $\epsilon_{\Theta,N}$ in \cref{cr:reduction for funcapp}. Therefore, the algorithm becomes applicable to large-scale or continuous problems.
\begin{restatable}{theorem}{SampleComplexityOfMirrorDescentWithBasis} \label{th:sample complexity of mirror descent with basis}
Under a proper choice of $\Theta$ and $B_R$, with probability $1-\delta$, \cref{alg:md for RL} learns an $(\epsilon+\epsilon_{\Theta,N})$-optimal policy with
$
\tilde{O}\left(\frac{dim(\Theta)\log(\frac{1}{\delta})}{(1-\gamma)^2\epsilon^2}\right)
$ samples.
\end{restatable}%
The proof is in \cref{app:sample complexity using function approximators}, and mainly follows \cref{sec:proof sketch of sample complexity of mirror descent}. First, we choose some $\Theta$ to satisfy \eqref{eq:function approximators} so we can use \cref{cr:reduction for funcapp} to reduce the problem into regret minimization.
%
To make the sample complexity independent of $|\SS|$,$|\AA|$, the key is to uniformly sample over the columns of $\Psib$ (instead of over all states and actions like \eqref{eq:stochastic gradient estimate}) when computing unbiased estimates of $\nabla_{\thetab_\mu} h_n((\thetab_v,\thetab_\mu))$. The intuition is that we should only focus on the places our basis functions care about (of size $dim(\Theta)$), instead of wasting efforts to visit all possible combinations (of size $|\SS||\AA|$). %In turn, this would make the average regret and martingales concentrate toward zero faster.
}
\section{CONCLUSION}
We propose a reduction from RL to no-regret online learning % with continuous loss variations (COL).
%
that %shows new insights to the saddle-point setup of RL, and
provides a systematic way to design new RL algorithms with performance guarantees. % through regret minimization.
%
Compared with existing approaches, our framework makes no assumption on the MDP and naturally works with function approximators.
%
To illustrate, we design a simple RL algorithm based on mirror descent; it achieves similar sample complexity as other RL techniques, but uses minimal assumptions on the MDP and is scalable to large or continuous problems.
%
This encouraging result evidences the strength of the online learning perspective.
%
As a future work, we believe even faster learning in RL is possible by leveraging control variate for variance reduction and by applying more advanced online techniques~\citep{rakhlin2012online,cheng2018predictor} that exploit the continuity in COL to predict the future gradients.
%
%\red{To validate this, we corroborate our new theoretical insights with simulations (which can be found in \cref{app:exps}); the empirical results agree with the trend suggested by the theory, even in the case of learning with neural networks.}
\subsubsection*{Acknowledgements}
This research is partially supported by NVIDIA Graduate Fellowship.
\bibliography{ref}
%\subsubsection*{References}
%References follow the acknowledgements. Use an unnumbered third level
%heading for the references section. Any choice of citation style is
%acceptable as long as you are consistent. Please use the same font
%size for references as for the body of the paper---remember that
%references do not count against your page length total.
%====================================================================
\clearpage
\onecolumn
\input{appendix}
\end{document}
% last update: 2019-03-05
% created by Ching-An Cheng
% PACKAGES
% math
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{bm}
\usepackage{bbm}
\usepackage{mathtools}
\usepackage{enumitem}
\usepackage{thmtools,thm-restate}
% algorithms
\usepackage{algorithm}
\usepackage{algorithmic}
% ref
\usepackage{natbib}
% misc
\usepackage[capitalise]{cleveref}
\usepackage{color}
\usepackage[dvipsnames]{xcolor}
\usepackage{graphicx}
\usepackage{comment}
%\usepackage[latin1]{inputenc} % for German
% EDITS
\newcommand{\todo}[1]{{\leavevmode\color{red}TODO: #1}}
\newcommand{\rev}[1]{{\leavevmode\color{blue}#1}}
\newcommand{\note}[1]{{\leavevmode\color{Green}Note: #1}}
% SHORTCUTS
%% theorem setting
%\renewcommand\qedsymbol{$\blacksquare$}
\theoremstyle{plain}
\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}
\newtheorem{proposition}{Proposition}
\newtheorem{corollary}{Corollary}
\theoremstyle{definition}
\newtheorem{definition}{Definition}
\newtheorem{example}{Example}
%\theoremstyle{remark}
\newtheorem{remark}{Remark}
\newtheorem{assumption}{Assumption}
\newenvironment{proofsketch}{\textit{Proof sketch:}}{\qed \par}
% colors
\newcommand{\red}[1]{\textcolor{red}{#1}}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\newcommand{\green}[1]{\textcolor{green}{#1}}
\newcommand{\pink}[1]{\textcolor{magenta}{#1}}
\newcommand{\cyan}[1]{\textcolor{cyan}{#1}}
%% fonts
\def\AA{\mathcal{A}}\def\BB{\mathcal{B}}\def\CC{\mathcal{C}}
\def\DD{\mathcal{D}}\def\EE{\mathcal{E}}\def\FF{\mathcal{F}}
\def\GG{\mathcal{G}}\def\HH{\mathcal{H}}\def\II{\mathcal{I}}
\def\JJ{\mathcal{J}}\def\KK{\mathcal{K}}\def\LL{\mathcal{L}}
\def\MM{\mathcal{M}}\def\NN{\mathcal{N}}\def\OO{\mathcal{O}}
\def\PP{\mathcal{P}}\def\QQ{\mathcal{Q}}\def\RR{\mathcal{R}}
\def\SS{\mathcal{S}}\def\TT{\mathcal{T}}\def\UU{\mathcal{U}}
\def\VV{\mathcal{V}}\def\WW{\mathcal{W}}\def\XX{\mathcal{X}}
\def\YY{\mathcal{Y}}\def\ZZ{\mathcal{Z}}
\def\Ab{\mathbf{A}}\def\Bb{\mathbf{B}}\def\Cb{\mathbf{C}}
\def\Db{\mathbf{D}}\def\Eb{\mathbf{E}}\def\Fb{\mathbf{F}}
\def\Gb{\mathbf{G}}\def\Hb{\mathbf{H}}\def\Ib{\mathbf{I}}
\def\Jb{\mathbf{J}}\def\Kb{\mathbf{K}}\def\Lb{\mathbf{L}}
\def\Mb{\mathbf{M}}\def\Nb{\mathbf{N}}\def\Ob{\mathbf{O}}
\def\Pb{\mathbf{P}}\def\Qb{\mathbf{Q}}\def\Rb{\mathbf{R}}
\def\Sb{\mathbf{S}}\def\Tb{\mathbf{T}}\def\Ub{\mathbf{U}}
\def\Vb{\mathbf{V}}\def\Wb{\mathbf{W}}\def\Xb{\mathbf{X}}
\def\Yb{\mathbf{Y}}\def\Zb{\mathbf{Z}}
%
\def\ab{\mathbf{a}}\def\bb{\mathbf{b}}\def\cb{\mathbf{c}}
\def\db{\mathbf{d}}\def\eb{\mathbf{e}}\def\fb{\mathbf{f}}
\def\gb{\mathbf{g}}\def\hb{\mathbf{h}}\def\ib{\mathbf{i}}
\def\jb{\mathbf{j}}\def\kb{\mathbf{k}}\def\lb{\mathbf{l}}
\def\mb{\mathbf{m}}\def\nb{\mathbf{n}}\def\ob{\mathbf{o}}
\def\pb{\mathbf{p}}\def\qb{\mathbf{q}}\def\rb{\mathbf{r}}
\def\sbb{\mathbf{s}}\def\tb{\mathbf{t}}\def\ub{\mathbf{u}}
\def\vb{\mathbf{v}}\def\wb{\mathbf{w}}\def\xb{\mathbf{x}}
\def\yb{\mathbf{y}}\def\zb{\mathbf{z}}
%
\def\Abb{\mathbb{A}}\def\Bbb{\mathbb{B}}\def\Cbb{\mathbb{C}}
\def\Dbb{\mathbb{D}}\def\Ebb{\mathbb{E}}\def\Fbb{\mathbb{F}}
\def\Gbb{\mathbb{G}}\def\Hbb{\mathbb{H}}\def\Ibb{\mathbb{I}}
\def\Jbb{\mathbb{J}}\def\Kbb{\mathbb{K}}\def\Lbb{\mathbb{L}}
\def\Mbb{\mathbb{M}}\def\Nbb{\mathbb{N}}\def\Obb{\mathbb{O}}
\def\Pbb{\mathbb{P}}\def\Qbb{\mathbb{Q}}\def\Rbb{\mathbb{R}}
\def\Sbb{\mathbb{S}}\def\Tbb{\mathbb{T}}\def\Ubb{\mathbb{U}}
\def\Vbb{\mathbb{V}}\def\Wbb{\mathbb{W}}\def\Xbb{\mathbb{X}}
\def\Ybb{\mathbb{Y}}\def\Zbb{\mathbb{Z}}
\def\Af{\mathfrak{A}}\def\Bf{\mathfrak{B}}\def\Cf{\mathfrak{C}}
\def\Df{\mathfrak{D}}\def\Ef{\mathfrak{E}}\def\Ff{\mathfrak{F}}
\def\Gf{\mathfrak{G}}\def\Hf{\mathfrak{H}}\def\If{\mathfrak{I}}
\def\Jf{\mathfrak{J}}\def\Kf{\mathfrak{K}}\def\Lf{\mathfrak{L}}
\def\Mf{\mathfrak{M}}\def\Nf{\mathfrak{N}}\def\Of{\mathfrak{O}}
\def\Pf{\mathfrak{P}}\def\Qf{\mathfrak{Q}}\def\Rf{\mathfrak{R}}
\def\Sf{\mathfrak{S}}\def\Tf{\mathfrak{T}}\def\Uf{\mathfrak{U}}
\def\Vf{\mathfrak{V}}\def\Wf{\mathfrak{W}}\def\Xf{\mathfrak{X}}
\def\Yf{\mathfrak{Y}}\def\Zf{\mathfrak{Z}}
\def\Av{\boldsymbol{A}}\def\Bv{\boldsymbol{B}}\def\Cv{\boldsymbol{C}}
\def\Dv{\boldsymbol{D}}\def\Ev{\boldsymbol{E}}\def\Fv{\boldsymbol{F}}
\def\Gv{\boldsymbol{G}}\def\Hv{\boldsymbol{H}}\def\Iv{\boldsymbol{I}}
\def\Jv{\boldsymbol{J}}\def\Kv{\boldsymbol{K}}\def\Lv{\boldsymbol{L}}
\def\Mv{\boldsymbol{M}}\def\Nv{\boldsymbol{N}}\def\Ov{\boldsymbol{O}}
\def\Pv{\boldsymbol{P}}\def\Qv{\boldsymbol{Q}}\def\Rv{\boldsymbol{R}}
\def\Sv{\boldsymbol{S}}\def\Tv{\boldsymbol{T}}\def\Uv{\boldsymbol{U}}
\def\Vv{\boldsymbol{V}}\def\Wv{\boldsymbol{W}}\def\Xv{\boldsymbol{X}}
\def\Yv{\boldsymbol{Y}}\def\Zv{\boldsymbol{Z}}
\def\av{\boldsymbol{a}}\def\bv{\boldsymbol{b}}\def\cv{\boldsymbol{c}}
\def\dv{\boldsymbol{d}}\def\ev{\boldsymbol{e}}\def\fv{\boldsymbol{f}}
\def\gv{\boldsymbol{g}}\def\hv{\boldsymbol{h}}\def\iv{\boldsymbol{i}}
\def\jv{\boldsymbol{j}}\def\kv{\boldsymbol{k}}\def\lv{\boldsymbol{l}}
\def\mv{\boldsymbol{m}}\def\nv{\boldsymbol{n}}\def\ov{\boldsymbol{o}}
\def\pv{\boldsymbol{p}}\def\qv{\boldsymbol{q}}\def\rv{\boldsymbol{r}}
\def\sv{\boldsymbol{s}}\def\tv{\boldsymbol{t}}\def\uv{\boldsymbol{u}}
\def\vv{\boldsymbol{v}}\def\wv{\boldsymbol{w}}\def\xv{\boldsymbol{x}}
\def\yv{\boldsymbol{y}}\def\zv{\boldsymbol{z}}
\def\Att{\mathtt{A}}\def\Btt{\mathtt{B}}\def\Ctt{\mathtt{C}}
\def\Dtt{\mathtt{D}}\def\Ett{\mathtt{E}}\def\Ftt{\mathtt{F}}
\def\Gtt{\mathtt{G}}\def\Htt{\mathtt{H}}\def\Itt{\mathtt{I}}
\def\Jtt{\mathtt{J}}\def\Ktt{\mathtt{K}}\def\Ltt{\mathtt{L}}
\def\Mtt{\mathtt{M}}\def\Ntt{\mathtt{N}}\def\Ott{\mathtt{O}}
\def\Ptt{\mathtt{P}}\def\Qtt{\mathtt{Q}}\def\Rtt{\mathtt{R}}
\def\Stt{\mathtt{S}}\def\Ttt{\mathtt{T}}\def\Utt{\mathtt{U}}
\def\Vtt{\mathtt{V}}\def\Wtt{\mathtt{W}}\def\Xtt{\mathtt{X}}
\def\Ytt{\mathtt{Y}}\def\Ztt{\mathtt{Z}}
\def\att{\mathtt{a}}\def\btt{\mathtt{b}}\def\ctt{\mathtt{c}}
\def\dtt{\mathtt{d}}\def\ett{\mathtt{e}}\def\ftt{\mathtt{f}}
\def\gtt{\mathtt{g}}\def\htt{\mathtt{h}}\def\itt{\mathtt{i}}
\def\jtt{\mathtt{j}}\def\ktt{\mathtt{k}}\def\ltt{\mathtt{l}}
\def\mtt{\mathtt{m}}\def\ntt{\mathtt{n}}\def\ott{\mathtt{o}}
\def\ptt{\mathtt{p}}\def\qtt{\mathtt{q}}\def\rtt{\mathtt{r}}
\def\stt{\mathtt{s}}\def\ttt{\mathtt{t}}\def\utt{\mathtt{u}}
\def\vtt{\mathtt{v}}\def\wtt{\mathtt{w}}\def\xtt{\mathtt{x}}
\def\ytt{\mathtt{y}}\def\ztt{\mathtt{z}}
\def\Asf{\mathsf{A}}\def\Bsf{\mathsf{B}}\def\Csf{\mathsf{C}}
\def\Dsf{\mathsf{D}}\def\Esf{\mathsf{E}}\def\Fsf{\mathsf{F}}
\def\Gsf{\mathsf{G}}\def\Hsf{\mathsf{H}}\def\Isf{\mathsf{I}}
\def\Jsf{\mathsf{J}}\def\Ksf{\mathsf{K}}\def\Lsf{\mathsf{L}}
\def\Msf{\mathsf{M}}\def\Nsf{\mathsf{N}}\def\Osf{\mathsf{O}}
\def\Psf{\mathsf{P}}\def\Qsf{\mathsf{Q}}\def\Rsf{\mathsf{R}}
\def\Ssf{\mathsf{S}}\def\Tsf{\mathsf{T}}\def\Usf{\mathsf{U}}
\def\Vsf{\mathsf{V}}\def\Wsf{\mathsf{W}}\def\Xsf{\mathsf{X}}
\def\Ysf{\mathsf{Y}}\def\Zsf{\mathsf{Z}}
\def\asf{\mathsf{a}}\def\bsf{\mathsf{b}}\def\csf{\mathsf{c}}
\def\dsf{\mathsf{d}}\def\esf{\mathsf{e}}\def\fsf{\mathsf{f}}
\def\gsf{\mathsf{g}}\def\hsf{\mathsf{h}}\def\isf{\mathsf{i}}
\def\jsf{\mathsf{j}}\def\ksf{\mathsf{k}}\def\lsf{\mathsf{l}}
\def\msf{\mathsf{m}}\def\nsf{\mathsf{n}}\def\osf{\mathsf{o}}
\def\psf{\mathsf{p}}\def\qsf{\mathsf{q}}\def\rsf{\mathsf{r}}
\def\ssf{\mathsf{s}}\def\tsf{\mathsf{t}}\def\usf{\mathsf{u}}
\def\vsf{\mathsf{v}}\def\wsf{\mathsf{w}}\def\xsf{\mathsf{x}}
\def\ysf{\mathsf{y}}\def\zsf{\mathsf{z}}
% math
\def\R{\Rbb}\def\Q{\Qbb}\def\Z{\Zbb}\def\N{\Nbb}\def\C{\Cbb}
\def\one{{\mathbbm1}}
\def\oneb{{\mathbf1}}
\def\zero{{\mathbbm0}}
\def\zerob{{\mathbf0}}
\def\GP{\GG\PP}
\def\K{\mathcal{K}}
\def\der{\mathrm{d}}
\def\zinf{[0, \infty) }
\def\afsystem{\dot{x} =f(x) + g(x)u}
\def\const{\mathrm{const.}}
\def\diag{\mathrm{diag}}
\def\bdiag{\mathrm{blkdiag}}
\def\mvec{\mathrm{vec}}
\def\t{\top}
\def\*{\star}
\newcommand{\norm}[1]{ \| #1 \| }
\newcommand{\abs}[1]{ \left| #1 \right| }
\newcommand{\tr}[1]{ \mathrm{tr}\left( #1\right)}
\newcommand{\lr}[2]{ \left\langle #1, #2 \right\rangle}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
% online learning
\def\regret{\textrm{Regret}}
% statistics
\newcommand{\KL}[2]{D_{KL}(#1, #2)} %\newcommand{\KL}[2]{KL[#1 || #2 ]}
\newcommand{\TV}[2]{D_{TV}(#1, #2)}
\newcommand{\HE}[2]{D_{H}(#1, #2)}
\newcommand{\WA}[2]{D_{W}(#1, #2)}
\newcommand{\FD}[2]{D_{f}(#1, #2)}
\newcommand{\E}{\Ebb}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}
%
%% Below is the code for optionally collecting all the proofs at the end
%% source: https://tex.stackexchange.com/questions/33229/how-to-place-all-proofs-automatically-in-appendix
%\usepackage{etex,etoolbox}
%%\usepackage{amsthm,amssymb}
%%\usepackage{thmtools}
%\usepackage{environ}
%
%\makeatletter
%\providecommand{\@fourthoffour}[4]{#4}
%% We define an addition for the theorem-like environments; when
%% \newtheorem{thm}{Theorem} is declared, the macro \thm expands
%% to {...}{...}{...}{Theorem} and with \@fourthoffour we access
%% to it; then we make available \@currentlabel (the theorem number)
%% also outside the environment.
%\newcommand\fixstatement[2][\proofname\space of]{%
% \ifcsname thmt@original@#2\endcsname
% % the theorem has been declared with \declaretheorem
% \AtEndEnvironment{#2}{%
% \xdef\pat@label{\expandafter\expandafter\expandafter
% \@fourthoffour\csname thmt@original@#2\endcsname\space\@currentlabel}%
% \xdef\pat@proofof{\@nameuse{pat@proofof@#2}}%
% }%
% \else
% % the theorem has been declared with \newtheorem
% \AtEndEnvironment{#2}{%
% \xdef\pat@label{\expandafter\expandafter\expandafter
% \@fourthoffour\csname #1\endcsname\space\@currentlabel}%
% \xdef\pat@proofof{\@nameuse{pat@proofof@#2}}%
% }%
% \fi
% \@namedef{pat@proofof@#2}{#1}%
%}
%
%% We allocate a block of 1000 token registers; in this way \prooftoks
%% is 1000 and we can access the following registers of the block by
%% \prooftoks+n (0<n<1000); we'll use a dedicated counter for it
%% that is stepped at every proof
%\globtoksblk\prooftoks{1000}
%\newcounter{proofcount}
%
%% We gather the contents of the proof as argument to \proofatend
%% and then we store
%% "\begin{proof}[Proof of <theoremname> <theoremnumber>]#1\end{proof}"
%% in the next token register of the allocated block
%\NewEnviron{proofatend}{%
% \edef\next{%
% \noexpand\begin{proof}[\pat@proofof\space\pat@label]%
% \unexpanded\expandafter{\BODY}}%
% \global\toks\numexpr\prooftoks+\value{proofcount}\relax=\expandafter{\next\end{proof}}
% \stepcounter{proofcount}}
%
%% \printproofs simply loops over the used token registers of the
%% block, freeing their contents
%\def\printproofs{%
% \count@=\z@
% \loop
% \the\toks\numexpr\prooftoks+\count@\relax
% \ifnum\count@<\value{proofcount}%
% \advance\count@\@ne
% \repeat}
%\makeatother
%
%%% Here starts the example, with two theorem declarations
%%\declaretheorem[style=plain,name=Theorem,qed=$\square$,numberwithin=section]{thm}
%%%\declaretheorem[style=plain,name=Lemma,qed=$\square$,numberlike=thm]{lem}
%%%\newtheorem{thm}{Theorem}
%%\newtheorem{lem}[thm]{Lemma}
%%\fixstatement{thm}
%%\fixstatement[Demonstration of]{lem}
%
%\fixstatement{lemma}
%\fixstatement{theorem}
%\fixstatement{proposition}
%\fixstatement{corollary}
\appendix
{\centering{\large\textbf{Appendix}}\par}
\section{Review of RL Setups} \label{app:different RL setups}
We provide an extended review of different formulations of RL for interested readers.
First, let us recall the problem setup.
Let $\SS$ and $\AA$ be state and action spaces, and let $\pi(a|s)$ denote a policy. %conditional distribution of action $a\in\AA$ given state $s\in\SS$.
For $\gamma \in [0,1)$, we are interested in solving a $\gamma$-discounted infinite-horizon RL problem:
\begin{align} \label{eq:canonical RL}
\textstyle
\max_\pi V^\pi(p) , \qquad\text{s.t.}\quad V^\pi(p) \coloneqq (1-\gamma) \E_{s_0 \sim p}\E_{\xi \sim \rho_\pi(s_0)} \left[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t) \right]
\end{align}
where $V^\pi(p)$ is the discounted average return, $r:\SS\times\AA\to[0,1]$ is the reward function,
$\rho_\pi(s_0)$ denotes the distribution of trajectory $\xi = s_0, a_0, s_1, \dots$ generated by running $\pi$ from state $s_0$ in a Markov decision process (MDP),
and $p$ is a fixed but unknown initial state distribution.
\subsection{Coordinate-wise Formulations} \label{sec:coordinate-wise formulation}
\paragraph{RL in terms of stationary state distribution}
Let $d_t^\pi(s)$ denote the state distribution at time $t$ given by running $\pi$ starting from $p$. % (note $d_{\pi,0} = p$).
We define its $\gamma$-weighted mixture as
\begin{align} \label{generalized stationary state distribution}
\textstyle
d^\pi(s) \coloneqq (1-\gamma) \sum_{t=0}^{\infty} \gamma^t d_t^\pi(s)
\end{align}
We can view $d^\pi$ in~\eqref{generalized stationary state distribution} as a {form of stationary state distribution of $\pi$}, because it is a valid probability distribution of state and satisfies the stationarity property below,
\begin{align} \tag{\ref{eq:stationarity (distribution)}}
d^\pi(s') = (1-\gamma)p(s') + \gamma \E_{s \sim d^\pi} \E_{a\sim \pi |s } [\PP(s'|s,a)]
\end{align}
where $\PP(s'|s,a)$ is the transition probability of the MDP.
The definition in~\eqref{generalized stationary state distribution} generalizes the concept of stationary distribution of MDP; %and converges to one (if it exists) as $\gamma \to 1$. Precisely,
as $\gamma\to1$, $d^\pi$ is known as the limiting average state distribution, which is the same as the stationary distribution of the MDP under $\pi$, if one exists.
%
Moreover, with the property in~\eqref{eq:stationarity (distribution)}, $d^\pi$ summarizes the Markov structure of RL, and allows us to
write \eqref{eq:canonical RL} simply as
\begin{align} \label{eq:RL in stationary distribution}
\max_{\pi} V^\pi(p), \qquad \text{s.t.}\quad V^\pi(p) =\E_{s \sim d^\pi}\E_{a \sim \pi |s} \left[ r(s,a) \right]
\end{align}
after commuting the order of expectation and summation. That is, an RL problem aims to maximize the expected reward under the stationary state-action distribution generated by the policy $\pi$.
\paragraph{RL in terms of value function}
We can also write~\eqref{eq:canonical RL} in terms of value function.
Recall %\footnote{The $(1-\gamma)$ factor is due to the normalized definition in~\eqref{eq:canonical RL}.}
\begin{align} \tag{\ref{eq:value function}}
\textstyle
V^\pi(s) \coloneqq (1-\gamma) \E_{\xi \sim \rho_\pi(s_0)|s_0=s }\left[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t) \right]
\end{align}
is the value function of $\pi$.
By definition, $V^\pi$ (like $d^\pi$) satisfies a stationarity property
\begin{align} \tag{\ref{eq:statinoarity (value)}}
V^\pi(s) = \E_{a \sim \pi | s} \left[ (1-\gamma)r(s,a) + \gamma \E_{s'\sim\PP|s,a} \left[ V^\pi(s') \right] \right]
\end{align}
which can be viewed as a dual equivalent of~\eqref{eq:stationarity (distribution)}. Because $r$ is in $[0,1]$, \eqref{eq:statinoarity (value)} implies $V^\pi$ lies in $[0,1]$.
The value function $V^*$ (a shorthand of $V_{\pi^*}$) of the optimal policy $\pi^*$ of the RL problem satisfies the so-called Bellman equation~\citep{bellman1954theory}:
$
V^*(s) = \max_{a\in\AA} (1-\gamma)r(s,a) + \gamma \E_{s'\sim\PP|s,a} \left[ V^*(s') \right]
$,
where the optimal policy $\pi^*$ can be recovered as the $\argmax$.
Equivalently, by the definition of $\max$, the Bellman equation amounts to finding the smallest $V$ such that
$
V(s) \geq (1-\gamma)r(s,a) + \gamma \E_{s'\sim\PP|s,a} \left[ V(s') \right]
$, $\forall s\in \SS, a \in \AA$.
In other words, the RL problem in~\eqref{eq:canonical RL} can be written as
\begin{align} \label{eq:value-based RL}
\min_{V} \E_{s \sim p} [V(s)] \qquad \text{s.t.}\quad V(s) \geq (1-\gamma)r(s,a) + \gamma \E_{s'\sim\PP|s,a} \left[ V(s') \right], \qquad \forall s\in \SS, a \in \AA
\end{align}
\subsection{Linear Programming Formulations} \label{sec:LP for RL}
%We showed two alternate expressions of RL using basic algebras.
We now connect the above two alternate expressions through the classical LP setup of RL~\cite{manne1959linear,denardo1968multichain}. %,schweitzer1985generalized}.
\paragraph{LP in terms of value function}
The classic LP formulation\footnote{Our setup in~\eqref{eq:LP (value)} differs from the classic one in the $(1-\gamma)$ factor in the constraint to normalize the problem.} is simply a restatement of~\eqref{eq:value-based RL}:
\begin{align} \tag{\ref{eq:LP (value)}}
\min_{\vb}\quad \pb^\t \vb \qquad
\text{s.t.} \quad (1-\gamma) \rb + \gamma \Pb \vb \leq \Eb \vb
\end{align}
where $\pb \in \R^{|\SS|}$, $\vb \in \R^{|\SS|}$, and $\rb \in \R^{|\SS||\AA|}$ are the vector forms of $p$, $V$, $r$, respectively, $\Pb \in \R^{|\SS||\AA|\times|\SS|}$ is the transition probability\footnote{We arrange the coordinates in a way such that along the $|\SS||\AA|$ indices are contiguous in actions.}, and $\Eb = \Ib \otimes \oneb \in \R^{|\SS||\AA|\times|\SS|}$ (we use $|\cdot|$ to denote the cardinality of a set, $\otimes$ the Kronecker product, $\Ib\in\R^{|\SS|\times|\SS|}$ is the identity, and $\oneb \in \R^{|\AA|}$ a vector of ones).
It is easy to verify that for all $\pb >0$, the solution to~\eqref{eq:LP (value)} is the same and equal to $\vb^*$ (the vector form of $V^*$). %; in general it at least matches~\eqref{eq:canonical RL}.
%For the optimal value function $\vb^*$ (the vector form of $V^*$), it satisfies
%$
%(1-\gamma) \Pib^* (\rb + \gamma \Pb \vb) = \vb
%$ where $\Pib^* \in \R^{|\SS|\times |\SS||\Abb|}$ denotes the policy matrix defined by the greedy policy of $\vb^*$.
\paragraph{LP in terms of stationary state-action distribution}
Define the Lagrangian function
\begin{align} \label{eq:Lagrangian}
\LL(\vb, \fb) \coloneqq \pb^\t \vb + \fb^\t ((1-\gamma)\rb + \gamma \Pb \vb - \Eb \vb)
\end{align}
where $\fb\geq\zerob \in \R^{|\SS||\AA|}$ is the Lagrangian multiplier. %for the inequality constraint.
By Lagrangian duality, the dual problem of~\eqref{eq:LP (value)} is given as
$
\max_{\fb\geq \zerob} \min_{\vb} \LL(\vb, \fb) $.
Or after substituting the optimality condition of $\vb$
%$0 = \nabla_{\vb}\LL(\vb, \fb) = \pb + \gamma \Pb^\t \fb - \Eb^\t \fb$
and define $\mub \coloneqq (1-\gamma) \fb$,
we can write the dual problem as another LP problem
%\begin{align*}% \label{eq:LP (flow)}
%\begin{split}
%\max_{\fb} \quad & (1-\gamma)\rb^\t \fb \\
%s.t. \quad & \fb \geq \zerob\\
%& \pb + \gamma \Pb^\t \fb = \Eb^\t \fb
%\end{split}
%\end{align*}
\begin{align} \tag{\ref{eq:LP (flow)}}
\max_{\mub \geq \zerob} \quad \rb^\t \mub \qquad
\text{s.t.}\quad %& \mub \geq \zerob\\
(1-\gamma)\pb + \gamma \Pb^\t \mub = \Eb^\t \mub
\end{align}
Note that this problem like~\eqref{eq:LP (value)} is normalized: we have $\norm{\mub}_1 = 1$ because $\norm{\pb}_1 = 1$, and
\begin{align*}
\norm{\mub}_1 = \oneb^\t \Eb^\t \mub = (1-\gamma) \oneb^\t \pb + \gamma \oneb^\t \Pb^\t \mub = (1-\gamma) \norm{\pb}_1 + \gamma \norm{\mub}_1
\end{align*}
where we use the facts that $\mub \geq \zerob $ and $\Pb$ is a stochastic transition matrix.
This means that $\mub$ is a valid state-action distribution, from which we see that the equality constraint in~\eqref{eq:LP (flow)} is simply a vector form~\eqref{eq:stationarity (distribution)}. Therefore, \eqref{eq:LP (flow)} is the same as \eqref{eq:RL in stationary distribution} if we define the policy $\pi$ as the conditional distribution based on $\mub$.
% $\pib = \mub \oslash \Eb \Eb^\t \mub$.
\section{Missing Proofs of \cref{sec:revist}} \label{app:missing proofs}
\subsection{Proof of \cref{lm:residual}}
\ResidueEquality*
\begin{proof}
First note that $F(x,x) = 0$. Then as $x'$ satisfies stationarity, we can use \cref{lm:performance difference lemma} below and write
\begin{align*}
r_{ep}(x;x') &= F(x,x) - F(x, x') \\
&= - F(x, x')\\
&= - (\pb^\t \vb' - \pb^\t \vb) - \mub^\t \ab_{\vb'} + \mub'^\t \ab_\vb &\since{Definition of $F$ in \eqref{eq:bifunction (RL)}}\\
&= - \mub' \ab_\vb - \mub^\t \ab_{\vb'} + \mub'^\t \ab_\vb &\since{\cref{lm:performance difference lemma}} \\
&= - \mub^\t \ab_{\vb'}
\end{align*}
\end{proof}
\subsection{Proof of~\cref{lm:performance difference lemma}}
\PerformanceDifferenceLemma*
\begin{proof}
This is the well-known performance difference lemma.
The proof is based on the stationary properties in \eqref{eq:stationarity (distribution)} and \eqref{eq:statinoarity (value)}, which can be stated in vector form as
\begin{align*}
(\mub^\pi)^\t \Eb \vb^\pi = (\mub^\pi)^\t ((1-\gamma)\rb + \gamma \Pb \vb^\pi) \qquad \text{and} \qquad
(1-\gamma)\pb + \gamma \Pb^\t \mub^\pi = \Eb^\t \mub^\pi
\end{align*}
The proof is a simple application of these two properties.
\begin{align*}
\pb^\t (\vb^\pi- \vb') &= \frac{1}{1-\gamma}(\Eb^\t \mub^\pi- \gamma \Pb^\t \mub^\pi)^\t(\vb^\pi- \vb') \\
&= \frac{1}{1-\gamma } (\mub^\pi)^\t( (\Eb - \gamma \Pb) \vb^\pi- (\Eb - \gamma \Pb) \vb')\\
&= \frac{1}{1-\gamma } (\mub^\pi)^\t ( (1-\gamma) \rb - (\Eb - \gamma \Pb) \vb') = (\mub^\pi)^\t \ab_{\vb'}
\end{align*}
where we use the stationarity property of $\mub^\pi$ in the first equality and that $\vb^\pi$ in the third equality.
\end{proof}
\subsection{Proof of \cref{pr:rough residual bound}}
\RoughResidueBound*
\begin{proof}
This proof mainly follows the steps in \cite{wang2017randomized} but written in our notation.
First \cref{lm:residual} shows $r_{ep}(x;x^*) = -\mub^\t \ab_{\vb^*}$. We then lower bound $-\mub^\t \ab_{\vb^*}$ by reversing the proof of the performance difference lemma (\cref{lm:performance difference lemma}).
\begin{align*}
\mub^\t \ab_{\vb^*} &= \frac{1}{1-\gamma } \mub^\t( (1-\gamma) \rb - (\Eb - \gamma \Pb) \vb^*) &\since{Definition of $\ab_{\vb^*}$}\\
&= \frac{1}{1-\gamma} \mub^\t( (\Eb - \gamma \Pb) \vb^{\pi_\mub}- (\Eb - \gamma \Pb) \vb^*) &\since{Stationarity of $\vb^{\pi_\mub}$}
\\
%&=\frac{1}{1-\gamma}(\Eb^\t \mub- \gamma \Pb^\t \mub)^\t(\vb^{\pi_\mub}- \vb^*) \\
&=\frac{1}{1-\gamma} \mub^\t (\Eb- \gamma \Pb)(\vb^{\pi_\mub}- \vb^*) \\
&=\frac{1}{1-\gamma}\db^\t(\Ib- \gamma \Pb_{\pi_\mub})(\vb^{\pi_\mub}- \vb^*)
\end{align*}
where we define $\db \coloneqq \Eb^\t \mub$ and $\Pb_{\pi_\mub}$ as the state-transition of running policy $\pi_\mub$.
%\begin{align*}
%&\pb(\vb^{\pi_\mub}- \vb^*) + \frac{1}{1-\gamma}(\mub - \mub_{\pi_\mub})^\t(\Eb- \gamma \Pb)(\vb^{\pi_\mub}- \vb^*) \\
%&\leq J(\pi_\mub) - J(\pi^*) + \frac{1}{1-\gamma}(\db - \db_{\pi_\mub})^\t(\Ib- \gamma \Pb_{\pi_\mub})(\vb^{\pi_\mub}- \vb^*) \\
%%
%&\leq J(\pi_\mub) - J(\pi^*) + \norm{(\db - \db_{\pi_\mub})_{-} }_1 \norm{\vb^{\pi_\mub}- \vb^*}_{\infty}
%\end{align*}
We wish to further upper bound this quantity.
To proceed, we appeal to the Bellman equation of the optimal value function $\vb^*$ and the stationarity of $\vb^{\pi_\mub}$:
\begin{align*}
\vb^* \geq (1-\gamma)\rb_{\pi_\mub} + \gamma \Pb_{\pi_\mub} \vb^*
%
\qquad \text{and}\qquad
%
\vb^{\pi_\mub} = (1-\gamma)\rb_{\pi_\mub} + \gamma \Pb_{\pi_\mub} \vb^{\pi_\mub},
\end{align*}
which together imply that $(\Ib- \gamma \Pb_{\pi_\mub})(\vb^{\pi_\mub}- \vb^*) \leq 0$.
We will also use the stationarity of $\db^{\pi_\mub}$ (the average state distribution of $\pi_\mub$):
$
\db^{\pi_\mub} = (1-\gamma) \pb + \gamma \Pb_{\pib_\mub}^\t \db^{\pi_\mub}
$.
Since $\db\geq (1-\gamma) \pb$ in the assumption, we can then write
\begin{align*}
\mub^\t \ab_{\vb^*}
&= \frac{1}{1-\gamma}\db^\t(\Ib- \gamma \Pb_{\pi_\mub})(\vb^{\pi_\mub}- \vb^*) \\
&\leq \pb^\t(\Ib- \gamma \Pb_{\pi_\mub})(\vb^{\pi_\mub}- \vb^*) \\
&\leq - \min_s p(s) \norm{(\Ib- \gamma \Pb_{\pi_\mub})(\vb^{\pi_\mub}- \vb^*)}_\infty \\
&\leq - \min_s p(s) (1-\gamma) \norm{\vb^{\pi_\mub}- \vb^*}_\infty.
\end{align*}
Finally, flipping the sign of the inequality concludes the proof.
\end{proof}
\subsection{Proof of \cref{pr:residual lower bound}}
\ResidueLowerBound*
\begin{proof}
We show this equality holds for a class of MDPs.
For simplicity, let us first consider an MDP with three states $1$, $2$, $3$ and for each state there are three actions ($left$, $right$, $stay$). They correspond to intuitive, deterministic transition dynamics
\begin{align*}
\PP(\max\{s-1,1\}|s,left) =1, \quad \PP(\min\{s+1,3\}|s,right) =1, \quad \PP(s|s,stay)=1.
\end{align*}
We set the reward as
$r(s,right)=1$ for $s=1,2,3$ and zero otherwise.
It is easy to see that the optimal policy is $\pi^*(right|s)=1$, which has value function $\vb^* = [1, 1, 1]^\t$.
Now consider $x= (\vb,\mub)\in\XX$. To define $\mub$, let $\mu(s,a) = d(s)\pi_\mub(a|s)$. We set
\begin{align*}
\pi_\mub(right|1) = 1, \quad \pi_\mub(stay|2) = 1, \quad \quad \pi_\mub(right|3) = 1
\end{align*}
That is, $\pi_\mub$ is equal to $\pi^*$ except when $s=2$.
One can verify the value function of this policy is $\vb^{\pi_\mub} = [(1-\gamma), 0, 1]^\t$.
As far as $d$ is concerned ($\db = \Eb^\t\mub$), suppose the initial distribution is uniform, i.e. $\pb = [1/3,1/3,1/3]^\t$, we choose $d$ as $\db = (1-\gamma)\pb + \gamma [1, 0 , 0]^\t$, which satisfies the assumption in \cref{pr:rough residual bound}. Therefore, we have $\mub \in \MM'$ and we will let $\vb$ be some arbitrary point in $\VV$.
Now we show for this choice $x= (\vb,\mub)\in \VV \times \MM'$, the equality in \cref{pr:rough residual bound} holds. By \cref{lm:residual}, we know $r_{ep}(x;x') = - \mub^\t \ab_{\vb^*}$. Recall the advantage is defined as
$
\ab_{\vb^*} = \rb + \frac{1}{1-\gamma}(\gamma \Pb - \Eb) \vb^*
$.
Let $A_{V^*}(s,a)$ denote the functional form of $\ab_{\vb^*}$ and
define the expected advantage:
\begin{align*}
A_{V^*}(s,\pi_\mub) \coloneqq \E_{a\sim\pi_\mub}[A_{V^*}(s,a)].
\end{align*}
We can verify it has the following values:
\begin{align*}
A_{V^*}(1,\pi_\mub) = 0, \quad A_{V^*}(2,\pi_\mub) = -1, \quad
A_{V^*}(3,\pi_\mub) = 0.
\end{align*}
Thus, the above construction yields
\begin{align*}
r_{ep}(x;x^*) = - \mub^\t \ab_{\vb^*} = \frac{(1-\gamma)}{3} = (1-\gamma) \min_s p(s) \norm{\vb^* - \vb^{\pi_\mub}}_\infty
\end{align*}
One can easily generalize this $3$-state MDP to an $|\SS|$-state MDP where states are partitioned into three groups.
\end{proof}
\section{Missing Proofs of \cref{sec:the reduction}} \label{app:missing proofs of reduction}
\subsection{Proof of \cref{pr:clever residual bound}}
\CleverResidueBound*
\begin{proof}%[Proof of \cref{pr:clever residual bound}]
First we generalize \cref{lm:residual}.
\begin{restatable}{lemma}{RelaxedResidueEquality}\label{lm:relaxed residual}
Let $x = (\vb, \mub)$ be arbitrary. Consider $\tilde{x}' = (\vb'+\ub', \mub')$, where $\vb'$ and $\mub'$ are the value function and state-action distribution of policy $\pi_{\mub'}$, and $\ub'$ is arbitrary. It holds that
$
r_{ep}(x;\tilde{x}') = - \mub^\t \ab_{\vb'} - \bb_{\mub}^\t \ub'
$.
\end{restatable}
%Let $\deltab = \vb^{\pi_\mub} - \vb^*$.
To proceed, we write $y_x^* = (\vb^* + (\vb^{\pi_\mub} - \vb^*), \mub^*)$ and use \cref{lm:relaxed residual}, which gives
$
r_{ep}(x;y_x^*) = - \mub^\t \ab_{\vb^*} - \bb_{\mub}^\t (\vb^{\pi_\mub} - \vb^*)
$.
To relate this equality to the policy performance gap, we also need the following equality.
\begin{restatable}{lemma}{AdvantageBound}\label{lm:advantage bound}
For $\mub\in\MM$, it holds that $- \mub^\t \ab_{\vb^*}= V^{*}(p) - V^{\pi_\mub}(p) + \bb_\mub^\t (\vb^{\pi_\mub} - \vb^*)$.
\end{restatable}
Together they imply the desired equality
$r_{ep}(x;y_x^*) = V^{*}(p) - V^{\pi_\mub}(p)$.
\end{proof}
\subsubsection{Proof of \cref{lm:relaxed residual}}
\RelaxedResidueEquality*
\begin{proof}
Let $x' = (\vb',\mub')$.
As shorthand, define $\fb'\coloneqq \vb'+\ub'$, and $\Lb \coloneqq \frac{1}{1-\gamma}(\gamma \Pb - \Eb)$ (i.e. we can write $\ab_\fb = \rb + \Lb \fb$). Because $r_{ep}(x;x') = - F(x, x') = - (\pb^\t \vb' + \mub^\t \ab_{\vb'} - \pb^\t \vb - \mub'^\t \ab_\vb)$, we can write
\begin{align*}
r_{ep}(x; \tilde{x}')
&= - \pb^\t \fb' - \mub^\t \ab_{\fb'} + \pb^\t \vb + \mub'^\t \ab_\vb\\
&= \left(- \pb^\t \vb' - \mub^\t \ab_{\vb'} + \pb^\t \vb + \mub'^\t \ab_\vb \right) - \pb^\t \ub' - \mub^\t \Lb \ub' \\
&= r_{ep}(x;x') - \pb^\t \ub' - \mub^\t \Lb \ub' \\
&= r_{ep}(x;x') - \bb_\mub^\t \ub'
\end{align*}
Finally, by \cref{lm:residual}, we have also $r_{ep}(x;x')= - \mub^\t \ab_{\vb'}$ and therefore the final equality.
\end{proof}
%\subsection{Proof of \cref{lm:relaxed residual}}
%\RelaxedResidueEquality*
%\begin{proof}
% Let $x = (\vb, \mub) \in \XX$ and let $x' = (\vb',\mub') \in \XX$ be a pair for some policy $\pi'$.
% %
% Let $\ub$ and $\ub_{\bar{x}}'$ be arbitrary and define $\bar{x} = (\vb+\ub, \mub)$ and $\bar{x}' = (\vb'+\ub_{\bar{x}}', \mub')$. As shorthand, let $\fb = \vb+\ub$, $\fb'=\vb'+\ub_{\bar{x}}'$, and $\Lb = \frac{1}{1-\gamma}(\gamma \Pb - \Eb)$ (i.e. we can write $\ab_\fb = \rb + \Lb \fb$). Because $r_{ep}(\bar{x};\bar{x}') \coloneqq - F(\bar{x}, \bar{x}')$ and $F(x, x') = \pb^\t \vb' + \mub^\t \ab_{\vb'} - \pb^\t \vb - \mub'^\t \ab_\vb$
% from \eqref{eq:bifunction (RL)}, we can write
% \begin{align*}
% r_{ep}(\bar{x};\bar{x}')
% &= - \pb^\t \fb' - \mub^\t \ab_{\fb'} + \pb^\t \fb + \mub'^\t \ab_\fb\\
% %&= - (\pb^\t \vb' - \pb^\t \vb) - (\pb^\t \ub_{\bar{x}}' - \pb^\t \ub) \\
% %&\quad - \mub^\t \ab_{\vb'} + \mub'^\t \ab_\vb - \mub^\t \Lb \ub_{\bar{x}}' + \mub'\Lb \ub \\
% &= r_{ep}(x;x') - \pb^\t \ub_{\bar{x}}' - \mub^\t \Lb \ub_{\bar{x}}' + \pb^\t \ub + \mub'^\t\Lb \ub.
% \end{align*}
% On one hand, by \cref{lm:residual}, we have $r_{ep}(x;x')= - \mub^\t \ab_{\vb'}$.
% %\begin{align*}
% %&- (\pb^\t \vb' - \pb^\t \vb) - \mub^\t \ab_{\vb'} + \mub'^\t \ab_\vb
% %&= - \mub' \ab_\vb - \mub^\t \ab_{\vb'} + \mub'^\t \ab_\vb &\since{\cref{lm:performance difference lemma}} \\
% %&= - \mub^\t \ab_{\vb'}
% %\end{align*}
% On the other hand, we can derive
% \begin{align*}
% r_{ep}(x; \tilde{x}')
% &= - \pb^\t \fb' - \mub^\t \ab_{\fb'} + \pb^\t \vb + \mub'^\t \ab_\vb\\
% &= r_{ep}(x;x') - \pb^\t \ub' - \mub^\t \Lb \ub' \\
% &= r_{ep}(x;x') - \bb_\mub^\t \ub'
% \end{align*}
% Finally, by \cref{lm:residual}, we have also $r_{ep}(x;x')= - \mub^\t \ab_{\vb'}$ and therefore the final equality.
%\end{proof}
\subsubsection{Proof of \cref{lm:advantage bound}}
\AdvantageBound*
\begin{proof}
Following the setup in \cref{lm:relaxed residual}, we prove the statement by the rearrangement below:
\begin{align*}
- \mub^\t \ab_{\vb'}
&= - (\mub^{\pi_\mub})^\t \ab_{\vb'} + (\mub^{\pi_\mub})^\t \ab_{\vb'} - \mub^\t \ab_{\vb'}\\
&= V^{\pi'}(p) - V^{\pi_\mub}(p) + (\mub^{\pi_\mub} - \mub)^\t \ab_{\vb'}\\ %&\since{\cref{lm:performance difference lemma}}\\
&= \left(V^{\pi'}(p) - V^{\pi_\mub}(p) \right) + (\mub^{\pi_\mub} - \mub)^\t \rb + (\mub^{\pi_\mub} - \mub)^\t \Lb {\vb'}
\end{align*}
where the second equality is due to the performance difference lemma, i.e. \cref{lm:performance difference lemma}, and the last equality uses the definition $\ab_{\vb'} = \rb + \Lb \vb'$.
%
For the second term above, let $\rb_{\pi_{\mub}}$ and $\Pb_{\pi_\mub}$ denote the expected reward and transition under $\pi_{\mub}$. Because $\mub \in \MM$, we can rewrite it as
\begin{align*}
(\mub^{\pi_\mub} - \mub)^\t \rb
&= (\Eb^\t\mub^{\pi_\mub} - \Eb^\t\mub) \rb_{\pi_{\mub}}\\
&= ((1-\gamma) \pb + \gamma \Pb^\t\mub^{\pi_\mub} - \Eb^\t\mub) \rb_{\pi_{\mub}} \\
&= (1-\gamma) \bb_\mub^\t \rb_{\pi_{\mub}} +
\gamma (\mub^{\pi_\mub} -\mub)^\t \Pb \rb_{\pi_{\mub}} \\
&= (1-\gamma) \bb_\mub^\t \left( \rb_{\pi_{\mub}} + \gamma \Pb_{\pi_\mub} \rb_{\pi_{\mub}} + \gamma^2 \Pb_{\pi_\mub}^2 \rb_{\pi_{\mub}} + \dots \right)\\
&= %1-\gamma) \bb_\mub^\t \frac{1}{1-\gamma} \vb^{\pi_\mub}
\bb_\mub^\t \vb^{\pi_\mub}
\end{align*}
where the second equality uses the stationarity of $\mub^{\pib_\mub}$ given by \eqref{eq:stationarity (distribution)}.
For the third term, it can be written
\begin{align*}
(\mub^{\pi_\mub} - \mub)^\t \Lb {\vb'} = (- \pb - \Lb^\t \mub)^\t {\vb'} %&\since{stationarity, i.e. $\bb_{\mub^{\pi_\mub}} = 0$}\\
= - \bb_\mub^\t\vb'
\end{align*}
where the first equality uses stationarity, i.e. $\bb_{\mub^{\pi_\mub}}= \pb + \Lb^\t \mub^{\pi_\mub}= 0$.
Finally combining the three steps, we have
\begin{align*}
- \mub^\t \ab_{\vb'}
&= V^{\pi'}(p) - V^{\pi_\mub}(p) + \bb_\mub(\vb^{\pi_\mub} - \vb')
\end{align*}
\end{proof}
\subsection{Proof of \cref{cr:reduction for funcapp}}
\ReductionForFunctionApporximator*
\begin{proof}
This can be proved by a simple rearrangement
\begin{align*}
&V^*(p) - V^{\hat{\pi}_N}(p)
= r_{ep}(\hat{x}_N;y_N^*)= \epsilon_{\Theta,N} + \max_{x_\theta\in\XX_\theta} r_{ep}(\hat{x}_N; x_\theta)\leq \epsilon_{\Theta,N} + \frac{\regret_N(\Theta)}{N}
\end{align*}
where the first equality is \cref{pr:clever residual bound} and the last inequality is due to the skew-symmetry of $F$, similar to the proof of \cref{th:reduction of RL}.
\end{proof}
\subsection{Proof of \cref{pr:size of epsilon}}
\SizeOfEpsilon*
\begin{proof}
For shorthand, let us set $x = (\vb, \mub) = \hat{x}_N$ and write also $\pi_\mub = \hat{\pi}_N$ as the associated policy.
Let $y_{x}^* = (\vb^{\pi_\mub}, \mub^*)$ % and let $\deltab =\vb^{\pi_\mub} - \vb^*$.
and similarly let $x_\theta = (\vb_\theta, \mub_\theta) \in \XX_\Theta$.
%\begin{align} \label{eq:bifunction (RL)}
%F(x, x')
%&\coloneqq \LL(\vb', \mub) - \LL(\vb, \mub') \nonumber \\
%&= \pb^\t \vb' + \mub^\t \ab_{\vb'} - \pb^\t \vb - \mub'^\t \ab_\vb
%\end{align}
With $r_{ep}(x;x') = -F(x,x')$ and \eqref{eq:bifunction (RL)}, we can write
\begin{align*}
r_{ep}(x;y_{x}^*) - r_{ep}(x; x_\theta)
&= \left(
- \pb^\t \vb^{\pi_\mub} - \mub^\t \ab_{\vb^{\pi_\mub}} + \pb^\t \vb + {\mub^*}^\t \ab_\vb \right)
- \left(
- \pb^\t \vb_\theta - \mub^\t \ab_{\vb_\theta} + \pb^\t \vb + \mub_\theta^\t \ab_\vb \right)\\
%
&= \pb^\t (\vb_\theta - \vb^{\pi_\mub}) + (\mub^* - \mub_\theta)^\t \ab_{\vb} + \mub^\t (\ab_{\vb_\theta} - \ab_{\vb^{\pi_\mub}} )\\
%
&= \bb_{\mub}^\t (\vb_\theta - \vb^{\pi_\mub}) + (\mub^* - \mub_\theta)^\t \ab_{\vb}
%\\
%&\leq \norm{\bb_\mub}_1 \norm{\vb_\theta - \vb^{\pi_\mub}}_\infty + \norm{\ab_{\vb}}_\infty \norm{\mub^* - \mub_\theta}_1
\end{align*}
%Let $\Delta$ denotes the set of distributions.
Next we quantize the size of $\ab_\vb$ and $\bb_\mub$.
\begin{lemma} \label{lm:size of a and b}
For $(\vb, \mub) \in \XX$,
$\norm{\ab_\vb}_\infty \leq \frac{1}{1-\gamma}$ and $\norm{\bb_\mub}_1 \leq \frac{2}{1-\gamma}$.
\end{lemma}
\begin{proof}[Proof of \cref{lm:size of a and b}]
Let $\Delta$ denote the set of distributions
\begin{align*}
\norm{\ab_\vb}_\infty &= \frac{1}{1-\gamma} \norm{(1-\gamma)\rb+\gamma\Pb\vb - \Eb\vb}_\infty \leq \frac{1}{1-\gamma
} \max_{a,b\in[0,1]} |a-b|\leq \frac{1}{1-\gamma}
\\
\norm{\bb_\mub}_1 &= \frac{1}{1-\gamma} \norm{(1-\gamma)\pb + \gamma \Pb^\t \mub - \Eb^\t\mub}_1 \leq \frac{1}{1-\gamma} \max_{\qb,\qb'\in\Delta}\norm{\qb - \qb'}_1 \leq \frac{2}{1-\gamma}
\end{align*}
\end{proof}
Therefore, we have preliminary upper bounds:
\begin{align*}
(\mub^* - \mub_\theta)^\t \ab_{\vb} &\leq \norm{\ab_{\vb}}_\infty \norm{\mub^* - \mub_\theta}_1 \leq \frac{1}{1-\gamma} \norm{\mub^* - \mub_\theta}_1
\\
\bb_{\mub}^\t (\vb_\theta - \vb^{\pi_\mub}) &\leq \norm{\bb_\mub}_1 \norm{\vb_\theta - \vb^{\pi_\mub}}_\infty \leq \frac{2}{1-\gamma} \norm{\vb_\theta - \vb^{\pi_\mub}}_\infty
\end{align*}
However, the second inequality above can be very conservative, especially when $\bb_\mub \approx 0$ which can be likely when it is close to the end of policy optimization. To this end, we introduce a free vector $\wb \geq 1$. Define norms $\norm{\vb}_{\infty, 1/\wb} = \max_i \frac{|v_i|}{w_i} $ and $\norm{\deltab}_{1,\wb} = \sum_{i} w_i |\delta_i|$. Then we can instead have an upper bound
\begin{align*}
\bb_{\mub}^\t (\vb_\theta - \vb^{\pi_\mub}) \leq \min_{\wb:\wb\geq 1} \norm{\bb_\mub}_{1,\wb} \norm{\vb_\theta - \vb^{\pi_\mub}}_{\infty,1/\wb}
\end{align*}
Notice that when $\wb = \oneb$ the above inequality reduces to $\bb_{\mub}^\t (\vb_\theta - \vb^{\pi_\mub}) \leq \norm{\bb_\mub}_1 \norm{\vb_\theta - \vb^{\pi_\mub}}_\infty$, which as we showed has an upper bound $\frac{2}{1-\gamma} \norm{\vb_\theta - \vb^{\pi_\mub}}_\infty$.
Combining the above upper bounds, we have an upper bound on $\epsilon_{\Theta,N}$:
\begin{align*}
\epsilon_{\Theta,N} = r_{ep}(x;y_{x}^*) - r_{ep}(x; x_\theta)
&\leq \frac{1}{1-\gamma} \norm{\mub_\theta - \mub^*}_1 +
\min_{\wb:\wb\geq 1} \norm{\bb_\mub}_{1,\wb} \norm{\vb_\theta - \vb^{\pi_\mub}}_{\infty,1/\wb}\\
&\leq
\frac{1}{1-\gamma} \left( \norm{\mub_\theta - \mub^*}_1
+ 2 \norm{\vb_\theta - \vb^{\pi_\mub}}_\infty \right).
\end{align*}
Since it holds for any $\theta \in \Theta$, we can minimize the right-hand side over all possible choices.
\end{proof}
%Then we prove a technical lemma, which will be used to upper bound $\norm{\ab_\vb}_\infty$ and $\norm{\bb_\mub}_1$.
%\begin{lemma}\label{lm:norm of G}
% For $\gamma\in[0,1)$, let $\Pb$ be a stochastic transition matrix, $\Eb = \Ib \otimes \oneb $ and $\Gb = \gamma \Pb - \Eb$. It holds that
% \begin{align*}
% \max_{\vb:\norm{\vb}_\infty \leq 1, \vb \geq 0} \norm{\Gb \vb}_\infty\leq 1 \qquad \text{and} \qquad
% \max_{\mub:\norm{\mub}_1 \leq 1, \mub\geq 0} \norm{\Gb^\t \mub}_1 \leq 1
% \end{align*}
%\end{lemma}
%\begin{proof}[Proof of \cref{lm:norm of G}]
% As $\Pb$ is a stochastic transition matrix, we have $\norm{\Pb \vb}_\infty \leq 1$, $\Pb \vb \geq 0$, $\norm{\Eb \vb }_\infty \leq 1 $ and $ \Eb \vb \geq 0$. Let $\alphab = \gamma\Pb\vb$ and $\betab = \Eb\vb$. It follows
% \begin{align*}
% \norm{\Gb \vb}_\infty = \norm{\alphab-\betab}_\infty = \max_k |\alpha_k - \beta_k| = \max_{a\in[0,\gamma], b\in[0,1]} |a-b|\leq 1
% \end{align*}
%
% Similarly, as $\Pb$ is a stochastic transition matrix, we have $\norm{\Pb^\t \mub}_1 \leq 1$, $\Pb^\t \mub \geq 0$, $\norm{\Eb^\t \mub}_1 \leq 1$ and $\Eb^\t \mub \geq 0$. With abuse of notation, let $\alphab = \gamma\Pb^\t\mub$ and $\betab = \Eb^\t\mub$. It follows (because $\alphab,\betab\geq0$)
% \begin{align*}
% \norm{\Gb^\t \mub}_1
% = \sum_k |\alpha_k - \beta_k|
% %
%% = \max \left\{ \sum_k \alpha_k - \beta_k, \sum_k \beta_k - \alpha_k \right\}
% %
%% = \max \{\norm{\alphab}_1-\norm{\betab}_1, \norm{\betab}_1 -\norm{\alphab}_1 \}
% \leq 2.
% \end{align*}
%\end{proof}
%This lemma implies that, for any $\vb\in\VV$ and $\mub\in\MM$, it holds $\norm{\ab_\vb}_\infty, \norm{\bb_\mub}_1 \leq 1 + \frac{1}{1-\gamma} = \frac{2-\gamma}{1-\gamma}$.
%%
%Thus, we have an upper bound on $\epsilon_\Theta$:
%\begin{align*}
%\epsilon_ = r_{ep}(x;y_{x}^*) - r_{ep}(x; x_\theta) \leq \frac{2-\gamma}{1-\gamma} \left( \norm{\mub_\theta - \mub^*}_1 + \norm{\vb_\theta - \vb^{\pi_\mub}}_\infty \right).
%\end{align*}
\section{Proof of Sample Complexity of Mirror Descent} \label{app:proof of sample complexity of mirror descent}
\SampleComplexityOfMirrorDescent*
The proof is a combination of the basic property of mirror descent (\cref{lm:mirror descent}) and the martingale concentration.
Define $K = |\SS||\AA|$ and $\kappa = \frac{1}{1-\gamma}$ as shorthands. We first slightly modify the per-round loss used to compute the gradient.
Recall $
l_n(x) \coloneqq \pb^\t \vb + \mub_n^\t \ab_{\vb} - \pb^\t \vb_n - \mub^\t \ab_{\vb_n}
$ and let us consider instead a loss function
\begin{align*}
h_n(x) \coloneqq \bb_{\mub_n}^\t \vb + \mub^\t (\kappa \oneb - \ab_{\vb_n})
\end{align*}
which shifts $l_n$ by a constant in each round. (Note for all the decisions $(\vb_n, \mub_n)$ produced by \cref{alg:md for RL} $\mub_n$ always satisfies $\norm{\mub_n}_1 = 1$).
One can verify that $l_n(x)-l_n(x') = h_n(x) - h_n(x')$, for all $x,x'\in \XX$, when $\mub, \mub'$ in $x$ and $x'$ satisfy $\norm{\mub}_1 = \norm{\mub'}_1$ (which holds for \cref{alg:md for RL}). As the definition of regret is relative, we may work with $h_n$ in online learning and use it to define the feedback.
The reason for using $h_n$ instead of $l_n$ is to make $\nabla_\mub h_n((\vb,\mub))$ (and its unbiased approximation) a positive vector (because $\kappa \geq \norm{\ab_{\vb}}_\infty$ for any $\vb \in \VV$), so that the regret bound can have a better dependency on the dimension for learning $\mub$ that lives in the simplex $\MM$. This is a common trick used in the online learning, e.g. in EXP3.
To run mirror descent, we set the first-order feedback $g_n$ received by the learner as an unbiased estimate of $\nabla h_n(x_n)$. For round $n$, we construct $g_n$ based on \emph{two} calls of the generative model:
\begin{align*}
g_n
= \begin{bmatrix}
\gb_{n,v}\\
\gb_{n,\mu}
\end{bmatrix}
%
= \begin{bmatrix}
\tilde{\pb}_n + \frac{1}{1-\gamma}(\gamma\tilde{\Pb}_n -\Eb_n)^\t \tilde{\mub}_n \\
K (\kappa \hat{\oneb}_n - \hat{\rb}_n - \frac{1}{1-\gamma}(\gamma\hat{\Pb}_n-\hat{\Eb}_n)\vb_n)
\end{bmatrix}
\end{align*}
For $\gb_{n,v}$, we sample $\pb$, then sample $\mub_n$ to get a state-action pair, and finally query the transition dynamics $\Pb$ at the state-action pair sampled from $\mub_n$. ($\tilde{\pb}_n$, $\tilde{\Pb}_n$, and $\tilde{\mub}_n$ denote the single-sample empirical approximation of these probabilities.)
%
For $\gb_{n,\mu}$, we first sample \emph{uniformly} a state-action pair (which explains the factor $K$), and then query the reward $\rb$ and the transition dynamics $\Pb$. ($\hat{\oneb}_n$, $\hat{\rb}_n$, $\hat{\Pb}_n$, and $\hat{\Eb}_n$ denote the single-sample empirical estimates.) To emphasize, we use $\tilde{}$ and $\hat{}$ to distinguish the empirical quantities obtained by these two independent queries. By construction, we have $\gb_{n,\mu} \geq 0$.
It is clear that this direction $g_n$ is unbiased, i.e. $\E[g_n] = \nabla h_n(x_n)$. Moreover, it is extremely sparse and can be computed using $O(1)$ sample, computational, and memory complexities.
Let $y_N^* = (\vb^{\hat{\pi}_N}, \mub^*)$.
We bound the regret by the following rearrangement. %, using the fact that $h_n$ in \eqref{eq:per-round loss of RL} is a linear function.
\begin{align} \label{eq:regret rearrangement}
\regret_N (y_N^*)
&= \sum_{n=1}^{N} l_n(x_n) - \sum_{n=1}^N l_n(y_N^*) \nonumber \\
&= \sum_{n=1}^{N} h_n(x_n) - \sum_{n=1}^N h_n(y_N^* ) \nonumber \\
&= \sum_{n=1}^{N} \nabla h_n(x_n)^\t (x_n-y_N^*) \nonumber \\
%
&= \left(\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n\right) + \left(\sum_{n=1}^N g_n^\t (x_n - y_N^*) \right) + \left(\sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^* \right) \nonumber\\
&\leq
\left(\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n\right)
+ \left( \max_{x\in\XX} \sum_{n=1}^N g_n^\t (x_n -x) \right)+ \left(\sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^* \right),
\end{align}
where the third equality comes from $h_n$ being linear. We recognize the first term is a martingale $M_N = \sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n$, because $x_n$ does not depend on $g_n$. Therefore, we can appeal to standard martingale concentration property. For the second term, it can be upper bounded by standard regret analysis of mirror descent, by treating $g_n^\t x$ as the per-round loss. For the third term, because $y_N^* = (\vb^{\hat{\pi}_N}, \mub^*)$ depends on $\{g_n\}_{n=1}^N$, it is not a martingale. Nonetheless, we will be able to handle it through a union bound.
%
%We remark that using two independent calls per iteration to define $g_n$ is critical for deducing a union bound with a sensible dependency on the dimension.
Below, we give details for bounding these three terms.
\subsection{The First Term: Martingale Concentration} \label{app:martingale concentration}
For the first term, $\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n$, we use a martingale concentration property.
Specifically, we adopt a Bernstein-type inequality~\citep[Theorem 3.15]{mcdiarmid1998concentration}:
\begin{lemma} \label{lm:martingale (Bernstein)}
{\normalfont\citep[Theorem 3.15]{mcdiarmid1998concentration} }
Let $M_0, \dots, M_N$ be a martingale and let $F_0\subseteq F_1 \subseteq \dots \subseteq F_n$ be the filtration such that $M_n = \E_{|F_{n}}[M_{n+1}]$. Suppose there are $b, \sigma < \infty$ such that for all $n$, given $F_{n-1}$, $M_n - M_{n-1} \leq b$, and $\Vbb_{|F_{n-1}}[M_n - M_{n-1}] \leq \sigma^2$ almost surely. Then for any $\epsilon\geq0$,
\begin{align*}
P(M_N - M_0 \geq \epsilon) \leq \exp\left(\frac{- \epsilon^2}{2 N \sigma^2 (1+\frac{b\epsilon}{3N\sigma^2})} \right).
\end{align*}
\end{lemma}
\cref{lm:martingale (Bernstein)} implies, with probability at least $1-\delta$,
\begin{align*}
M_N - M_0 \leq \sqrt{2 N \sigma^2 (1+ o(1)) \log\left(\frac{1}{\delta}\right)},
\end{align*}
where $o(1)$ means convergence to $0$ as $N\to\infty$.
To apply \cref{lm:martingale (Bernstein)}, we need to provide bounds on the properties of the martingale difference:
\begin{align*}
M_n - M_{n-1}
&= (\nabla h_n(x_n)-g_n)^\t x_n \\
&= (\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub_n + (\bb_{\mub_n}-\gb_{n,v})^\t \vb_n.
\end{align*}
For the first term $(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub_n$, we use the lemma below:
\begin{lemma} \label{lm:martingale difference for mu}
Let $\mub \in \MM$ be arbitrary, chosen independently from the randomness of $\gb_{n,\mu}$ when $F_{n-1}$ is given. Then it holds
$ |(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub| \leq \frac{2(1+K)}{1-\gamma}$
and $\Vbb_{|F_{n-1}}[(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub] \leq \frac{4K}{(1-\gamma)^2}$.
\end{lemma}
\begin{proof}
By triangular inequality,
\begin{align*}
|(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub| \leq |(\kappa \oneb - \ab_{\vb_n} )^\t \mub |+ | \gb_{n,\mu}^\t \mub|.
\end{align*}
For the deterministic part, using \cref{lm:size of a and b} and H\"older's inequality,
\begin{align*}
|(\kappa \oneb - \ab_{\vb_n})^\t \mub| \leq \kappa + \norm{\ab_{\vb_n}}_\infty\norm{\mub}_1 \leq \frac{2}{1-\gamma}.
\end{align*}
For the stochastic part, let $i_n$ be index of the sampled state-action pair and $j_n$ be the index of the transited state sampled at the pair given by $i_n$. With abuse of notation, we will use $i_n$ to index both $\SS\times\AA$ and $\SS$.
With this notation, we may derive
\begin{align*}
|\gb_{n,\mu}^\t \mub |
&= |K \mub^\t (\kappa \hat{\oneb}_n - \hat{\rb}_n - \frac{1}{1-\gamma}(\gamma\hat{\Pb}_n-\hat{\Eb}_n)\vb_n)|\\
%
&=K \mu_{i_n}|\kappa - r_{i_n} - \frac{\gamma v_{n,j_n} - v_{n,i_n}}{1-\gamma} |\\
&\leq \frac{2 K \mu_{i_n}}{1-\gamma} \leq \frac{2 K}{1-\gamma}
\end{align*}
where we use the facts that $r_{i_n}, v_{n,j_n}, v_{n,i_n} \in [0,1]$ and $\mu_{i_n} \leq 1$.
For $\Vbb_{|F_{n-1}}[(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub_n]$, we can write it as
\begin{align*}
\Vbb_{|F_{n-1}}[(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub]
&= \Vbb_{|F_{n-1}}[ \gb_{n,\mu}^\t \mub] \\
&\leq \E_{|F_{n-1}}[ |\gb_{n,\mu}^\t \mub_n|^2] \\
&= \sum_{i_n} \frac{1}{K}\E_{j_n|i_n} \left[ K^2 \mu_{i_n}^2 \left(\kappa - r_{i_n} - \frac{\gamma v_{n,j_n} - v_{n,i_n}}{1-\gamma}\right)^2 \right]\\
&\leq \frac{4 K }{(1-\gamma)^2} \sum_{i_n} \mu_{i_n}^2 \\
&\leq \frac{4 K }{(1-\gamma)^2} \left( \sum_{i_n} \mu_{i_n}\right)^2 \leq \frac{4 K }{(1-\gamma)^2}
\end{align*}
where in the second inequality we use the fact that $|\kappa - r_{i_n} - \frac{\gamma v_{n,j_n} - v_{n,i_n}}{1-\gamma} | \leq \frac{2}{1-\gamma}$.
\end{proof}
For the second term $(\bb_{\mub_n}-\gb_{n,v})^\t \vb_n$, we use the following lemma.
\begin{lemma} \label{lm:martingale difference for v}
Let $\vb \in \VV$ be arbitrary, chosen independently from the randomness of $\gb_{n,v}$ when $F_{n-1}$ is given.. Then it holds that
$ |(\bb_{\mub_n}-\gb_{n,v})^\t \vb| \leq \frac{4}{1-\gamma}$
and $\Vbb_{|F_{n-1}}[(\bb_{\mub_n}-\gb_{n,v})^\t \vb] \leq \frac{4}{(1-\gamma)^2}$.
\end{lemma}
\begin{proof}
We appeal to \cref{lm:size of a and b}, which shows $\norm{\bb_{\mub_n}}_1,\norm{\gb_{n,v}}_1 \leq \frac{2}{1-\gamma}$, and derive
\begin{align*}
|(\bb_{\mub_n}-\gb_{n,v})^\t \vb| \leq (\norm{\bb_{\mub_n}}_1 + \norm{\gb_{n,v}}_1 )\norm{\vb}_\infty \leq \frac{4}{1-\gamma}.
\end{align*}
Similarly, for the variance, we can write
\begin{align*}
\Vbb_{|F_{n-1}}[(\bb_{\mub_n}-\gb_{n,v})^\t \vb]
&= \Vbb_{|F_{n-1}}[\gb_{n,v}^\t \vb]\leq \Ebb_{|F_{n-1}}[(\gb_{n,v}^\t \vb)^2] \leq \frac{4}{(1-\gamma)^2}. \qedhere
\end{align*}
\end{proof}
Thus, with helps from the two lemmas above, we are able to show
\begin{align*}
M_n - M_{n-1} \leq |(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub_n| + |(\bb_{\mub_n}-\gb_{n,v})^\t \vb_n |\leq \frac{4+2(1+K)}{1-\gamma}
\end{align*}
as well as (because $\gb_{n,\mu}$ and $\gb_{n,b}$ are computed using independent samples)
\begin{align*}
\Vbb_{|F_{n-1}}[M_n - M_{n-1}]
&\leq \E_{|F_{n-1}}[|(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub_n|^2] + \E_{|F_{n-1}}[|(\bb_{\mub_n}-\gb_{n,v})^\t \vb_n|^2]
%
\leq \frac{4(1+K)}{(1-\gamma)^2}
\end{align*}
Now, since $M_0 = 0$, by martingale concentration in \cref{lm:martingale (Bernstein)}, we have
\begin{align*}
P\left( \sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n > \epsilon \right) \leq \exp\left(\frac{- \epsilon^2}{2 N \sigma^2 (1+\frac{b\epsilon}{3N\sigma^2})} \right)
\end{align*}
with $b = \frac{6+2K}{1-\gamma}$ and $\sigma^2 = \frac{4(1+K)}{(1-\gamma)^2}$. This implies that, with probability at least $1-\delta$, it holds
\begin{align*}
\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n \leq \sqrt{ N\frac{8(1+K)}{(1-\gamma)^2} (1+o(1))\log\left(\frac{1}{\delta}\right)}
= \tilde{O}\left( \frac{\sqrt{NK\log(\frac{1}{\delta})}}{1-\gamma} \right)
% = \epsilon
%\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n \leq \sqrt{ N\frac{8(1+K)}{(1-\gamma)^2} + \frac{2}{3}\frac{6+K}{1-\gamma} \epsilon } \sqrt{\log\left(\frac{1}{\delta}\right)} = \epsilon
\end{align*}
%We note that $o(1)$ term is
%$
%\frac{b\epsilon}{3N\sigma^2} = O\left( (1-\gamma) \frac{\epsilon}{N} \right)
%$, which from the definition of $\epsilon$ above is $o(K)$
\subsection{Static Regret of Mirror Descent} \label{app:static regret of mirror descent}
Next we move onto deriving the regret bound of mirror descent with respect to the online loss sequence:
\begin{align*}
\max_{x\in\XX} \sum_{n=1}^N g_n^\t (x_n -x)
\end{align*}
This part is quite standard; nonetheless, we provide complete derivations below.
We first recall a basic property of mirror descent
\begin{lemma} \label{lm:mirror descent}
Let $\XX$ be a convex set. Suppose $R$ is 1-strongly convex with respect to some norm $\norm{\cdot}$. Let $g$ be an arbitrary vector and define, for $x\in\XX$,
\begin{align*}
y = \argmin_{x' \in \XX} \lr{g}{x'} + B_{R}(x'||x)
%= P_{g,\eta}(x)
\end{align*}
Then for all $z \in \XX$,
\begin{align} \label{eq:mirror descent new decision}
\lr{g}{y - z} &\leq B_R(z||x) - B_R(z||y) - B_R(y||x)
\end{align}
% which implies
% \begin{align} \label{eq:mirror descent old decision}
% \lr{g}{x - z} \leq B_R(z||x) - B_R(z||y) + \frac{1}{2} \norm{g}_*^2
% \end{align}
\end{lemma}
\begin{proof}
Recall the definition $B_R(x'||x) = R(x') - R(x) - \lr{\nabla R(x)}{x'-x}$.
The optimality of the proximal map can be written as
\begin{align*}
\lr{ g + \nabla R(y) - \nabla R(x) }{y - z} \leq 0, \qquad \forall z \in \XX.
\end{align*}
By rearranging the terms, we can rewrite the above inequality in terms of Bregman divergences as follows and derive the first inequality~\eqref{eq:mirror descent new decision}:
\begin{align*}
\lr{ g }{y - z} &\leq \lr{ \nabla R(x) - \nabla R(y) }{y - z} \\
&= B_R(z||x) - B_R(z||y) + \lr{ \nabla R(x) - \nabla R(y) }{y} - \lr{\nabla R(x)}{x} + \lr{\nabla R(y)}{y} + R(x) - R(y) \\
%&= B_R(z||x) - B_R(z||y) + \lr{ \nabla R(x) }{y} - \lr{\nabla R(x)}{x} + R(x) - R(y) \\
&= B_R(z||x) - B_R(z||y) + \lr{ \nabla R(x) }{y - x} + R(x) - R(y) \\
&= B_R(z||x) - B_R(z||y) - B_R(y||x),
\end{align*}
which concludes the lemma.
% The second inequality is a consequence of~\eqref{eq:mirror descent new decision}. First, we rewrite~\eqref{eq:mirror descent new decision} as
% \begin{align*}
% \lr{ g }{x - z}
% &= B_R(z||x) - B_R(z||y) - B_R(y||x) + \lr{ g }{x - y}
% \end{align*}
% Then we use the fact that $B_R$ is 1-strongly convex with respect to $\norm{\cdot}$, which implies
% \begin{align*}
% - B_R(y||x) + \lr{ g }{x - y} \leq - \frac{1}{2}\norm{x-y}^2 + \lr{ g }{x - y} \leq \frac{1}{2} \norm{g}_*^2
% \end{align*}
% Combining the two inequalities yields~\eqref{eq:mirror descent old decision}.
\end{proof}
Let $x'\in\XX$ be arbitrary.
Applying this lemma to the $n$th iteration of mirror descent in \eqref{eq:mirror descent}, we get
\begin{align*} %\label{eq:mirror descent new decision}
\lr{g_n}{x_{n+1} - x'} &\leq \frac{1}{\eta} \left( B_R(x'||x_n) - B_R(x'||x_{n+1}) - B_R(x_{n+1}||x_n) \right)
\end{align*}
By a telescoping sum, we then have
\begin{align*}
\sum_{n=1}^{N} \lr{g_n}{x_{n} - x'} &\leq \frac{1}{\eta} B_R(x'||x_1) + \sum_{n=1}^{N} \lr{g_n}{x_{n+1} - x_n} - \frac{1}{\eta} B_R(x_{n+1}||x_n).
\end{align*}
We bound the right-hand side as follows.
Recall that based on the geometry of $\XX = \VV \times \MM$, we considered a natural Bregman divergence of the form:
\begin{align*}
B_R(x'||x) = \frac{1}{2|\SS|} \norm{\vb' - \vb}_2^2 + KL(\mub'||\mub)
\end{align*}
Let $x_1 = (\vb_1, \mub_1)$ where $\mub_1$ is uniform.
By this choice, we have:
\begin{align*}
\frac{1}{\eta} B_R(x'||x_1) \leq \frac{1}{\eta} \max_{x\in\XX}B_R(x||x_1) \leq \frac{1}{\eta}\left( \frac{1}{2} + \log(K) \right).
\end{align*}
We now decompose each item in the above sum as:
\begin{align*}
\lr{g_n}{x_{n+1} - x_n} - \frac{1}{\eta} B_R(x_{n+1}||x_n)
=& \left( \gb_{n,v}^\t (\vb_{n+1} - \vb_n) - \frac{1}{2\eta |\SS|} \norm{\vb_n - \vb_{n+1}}_2^2 \right) \\
&+ \left( \gb_{n,\mu}^\t(\mub_{n+1} - \mub_n) - \frac{1}{\eta}KL(\mub_{n+1}||\mub_n) \right)
\end{align*}
and we upper bound them using the two lemmas below (recall $\gb_{n,\mu} \geq 0 $ due to the added $\kappa \oneb$ term).
\begin{restatable}{lemma}{GradientDescentBound} \label{lm:gradient descent bound}
For any vector $x, y, g$ and scalar $\eta >0$, it holds
$\lr{g}{x-y} - \frac{1}{2\eta}\norm{x-y}_2^2 \leq \frac{\eta \norm{g}_2^2}{2}$.
\end{restatable}
\begin{proof}
By Cauchy-Swartz inequality,
$
\lr{g}{x-y} - \frac{1}{2\eta}\norm{x-y}_2^2 \leq \norm{g}_2\norm{x - y}_2 - \frac{1}{2\eta}\norm{x-y}_2^2 \leq \frac{ \eta \norm{g}_2^2}{2}
$.
\end{proof}
\begin{restatable}{lemma}{EXPBound}
\label{lm:exp3 bound}
Suppose $B_R(x||y) = KL(x||y)$ and $x,y$ are probability distributions, and $g \geq 0$ element-wise. Then, for $\eta > 0$,
\begin{align*}
-\frac{1}{\eta} B_R(y||x) + \lr{ g }{x - y}
\leq \frac{\eta}{2} \sum_i x_i( g_i)^2 = \frac{\eta}{2} \norm{g}_{x}^2.
\end{align*}
\end{restatable}
\begin{proof}
Let $\Delta$ denotes the unit simplex.
\begin{align*}
- B_R(y||x) + \lr{ \eta g }{x - y}
&\leq \lr{ \eta g }{x } + \max_{y' \in \Delta} \lr{ - \eta g }{y} - B_R(y'||x)\\
&= \lr{ \eta g }{x } + \log\left( \sum_i x_i \exp(-\eta g_i) \right) &\since{convex conjugate of KL divergence}\\
&\leq \lr{ \eta g }{x } + \log\left( \sum_i x_i \left( 1 - \eta g_i + \frac{1}{2} (\eta g_i)^2 \right) \right) &\since{$e^{x} \leq 1 + x + \frac{1}{2}x^2$ for $x\leq 0$}\\
&= \lr{ \eta g }{x } + \log\left( 1+ \sum_i x_i \left( -\eta g_i + \frac{1}{2} (\eta g_i)^2 \right) \right) \\
&\leq \lr{ \eta g }{x } + \sum_i x_i \left( -\eta g_i + \frac{1}{2} (\eta g_i)^2 \right) &\since{$\log(x) \leq x -1$} \\
&= \frac{1}{2} \sum_i x_i(\eta g_i)^2 = \frac{\eta^2}{2} \norm{g}_{x}^2.
\end{align*}
Finally, dividing both sides by $\eta$, we get the desired result.
\end{proof}
Thus, we have the upper bound
$
\lr{g_n}{x_{n+1} - x_n} -\frac{1}{\eta} B_R(x_{n+1}||x_n)
= \frac{\eta|\SS| \norm{\gb_{n,v}}_2^2}{2} + \frac{\eta \norm{\gb_{n,\mu}}_{\mub_n}^2}{2}
$. Together with the upper bound on $\frac{1}{\eta} B_R(x'||x_1)$, it implies that
\begin{align} \label{eq:initial telesum of md}
\sum_{n=1}^{N} \lr{g_n}{x_{n} - x'} &\leq \frac{1}{\eta} B_R(x'||x_1) + \sum_{n=1}^{N} \lr{g_n}{x_{n+1} - x_n} - \frac{1}{\eta} B_R(x_{n+1}||x_n) \nonumber \\
&\leq \frac{1}{\eta}\left( \frac{1}{2} + \log(K) \right) + \frac{\eta}{2} \sum_{n=1}^{N} |\SS|\norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2.
\end{align}
We can expect, with high probability, $\sum_{n=1}^{N} |\SS|\norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2$ concentrates toward its expectation, i.e.
\begin{align*}
\sum_{n=1}^{N} |\SS| \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2 \leq \sum_{n=1}^{N} \E[|\SS| \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2] + o(N).
\end{align*}
Below we quantify this relationship using martingale concentration. First we bound the expectation.
\begin{lemma} \label{lm:size of sampled grad}
$
\E[\norm{\gb_{n,v}}_2^2] \leq \frac{4}{(1-\gamma)^2} $ and $\E[\norm{\gb_{n,\mu}}_{\mub_n}^2] \leq \frac{4K}{(1-\gamma)^2}$.
\end{lemma}
\begin{proof}
For the first statement, using the fact that $\norm{\cdot}_2 \leq \norm{\cdot}_1$ and \cref{lm:size of a and b}, we can write
\begin{align*}
\E[\norm{\gb_{n,v}}_2^2]
\leq \E[\norm{\gb_{n,v}}_1^2]
= \E[\norm{\tilde{\pb}_n + \frac{1}{1-\gamma}(\gamma\tilde{\Pb}_n -\Eb_n)^\t \tilde{\mub}_n }_1^2] \leq \frac{4}{(1-\gamma)^2}.
\end{align*}
For the second statement, let $i_n$ be the index of the sampled state-action pair and $j_n$ be the index of the transited-to state sampled at the pair given by $i_n$. With abuse of notation, we will use $i_n$ to index both $\SS\times\AA$ and $\SS$.
\begin{align*}
\E [ \norm{\gb_{n,\mu}}_{\mub_n}^2]
&= \E\left[ \sum_{i_n} \frac{1}{K}\E_{j_n|i_n} \left[ K^2 \mu_{i_n} \left(\kappa - r_{i_n} - \frac{\gamma v_{n,j_n} - v_{n,i_n}}{1-\gamma}\right)^2 \right] \right]\\
&\leq \frac{4 K }{(1-\gamma)^2} \E\left[ \sum_{i_n} \mu_{i_n} \right] \leq \frac{4 K }{(1-\gamma)^2}. \qedhere
\end{align*}
\end{proof}
To bound the tail, we resort to the H\"offding-Azuma inequality of martingale~\citep[Theorem 3.14]{mcdiarmid1998concentration}.
\begin{lemma}[Azuma-Hoeffding] \label{lm:martgingale (Hoeffding)}
Let $M_0, \dots, M_N$ be a martingale and let $F_0\subseteq F_1 \subseteq \dots \subseteq F_n$ be the filtration such that $M_n = \E_{|F_{n}}[M_{n+1}]$. Suppose there exists $b < \infty$ such that for all $n$, given $F_{n-1}$, $|M_n - M_{n-1}|\leq b$. Then for any $\epsilon\geq0$,
\begin{align*}
P(M_N - M_0 \geq \epsilon) \leq \exp\left(\frac{-2\epsilon^2}{N b^2 } \right)
\end{align*}
\end{lemma}
To apply \cref{lm:martgingale (Hoeffding)}, we consider the martingale
\begin{align*}
M_N = \sum_{n=1}^{N}|\SS| \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2 - \left(\sum_{n=1}^{N}\E[|\SS| \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2] \right)
\end{align*}
To bound the change of the size of martingale difference $|M_n - M_{n-1}|$, we follow similar steps to \cref{lm:size of sampled grad}.
\begin{lemma}
$\norm{\gb_{n,v}}_2^2 \leq \frac{4}{(1-\gamma)^2}$ and $\norm{\gb_{n,\mu}}_{\mub_n}^2 \leq \frac{4K^2}{(1-\gamma)^2}$.
\end{lemma}
Note $\norm{\gb_{n,\mu}}_\mub^2$ is $K$-factor larger than $\E[\norm{\gb_{n,\mu}}_\mub^2]$) and $K\geq1$.
Therefore, we have
\begin{align*}
|M_n - M_{n-1}| \leq|\SS| \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2 +|\SS| \E[\norm{\gb_{n,v}}_2^2] +\E[ \norm{\gb_{n,\mu}}_{\mub_n}^2] \leq \frac{8(|\SS|+K^2)}{(1-\gamma)^2}
\end{align*}
Combining these results, we have, with probability as least $1-\delta$,
\begin{align*}
\sum_{n=1}^{N} |\SS|\norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2
%
&\leq \sum_{n=1}^{N} \E[|\SS|\norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2] + \frac{4\sqrt{2}(|\SS|+K^2)}{(1-\gamma)^2} \sqrt{N \log\left(\frac{1}{\delta}\right)}\\
%
&\leq \frac{4(K+|\SS|)}{(1-\gamma)^2} N + \frac{4\sqrt{2}(|\SS|+K^2)}{(1-\gamma)^2} \sqrt{N \log\left(\frac{1}{\delta}\right)}
\end{align*}
Now we suppose we set $\eta = \frac{1-\gamma}{\sqrt{KN}}$. From \eqref{eq:initial telesum of md}, we then have
\begin{align*}
\sum_{n=1}^{N} \lr{g_n}{x_{n} - x'}
&\leq \frac{1}{\eta}\left( \frac{1}{2} + \log(K) \right) + \frac{\eta}{2} \sum_{n=1}^{N}|\SS| \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\mub_n}^2 \\
&\leq \frac{\sqrt{KN}}{1-\gamma} \left( \frac{1}{2} + \log(K) \right)
+
\frac{1-\gamma}{\sqrt{KN}} \left( \frac{2(K+|\SS|)}{(1-\gamma)^2} N + \frac{2\sqrt{2}(|\SS|+K^2)}{(1-\gamma)^2} \sqrt{N \log\left(\frac{1}{\delta}\right)}\right)\\
%
&\leq \tilde{O}\left(\frac{\sqrt{KN}}{1-\gamma} + \frac{\sqrt{K^3 \log\frac{1}{\delta}}}{1-\gamma} \right).
\end{align*}
%Note this is actually quite obviously because one can show $KL(y||x)$ divergence (by Hessian) is 1-strongly convex in $\norm{\cdot}_{1/z}$, for some $z = \tau x + (1-\tau) y$ with $\tau\in[0,1]$, i.e.,
%\begin{align*}
%KL(y||x) \geq \frac{1}{2}\norm{x-y}_{1/z}^2 = \frac{1}{2} \sum_i \frac{(x_i-y_i)^2}{z_i}
%\end{align*}
%\begin{proof}
%Let $f_b(a) = a \log\frac{a}{b}$. Then $\nabla f_b(a) = \log\frac{a}{b}+1$ and $\nabla^2 f_b(a) = \frac{1}{a}$. Therefore, by Taylor's theorem, for some $z = \tau x + (1-\tau) y$ with $\tau\in[0,1]$,
%\begin{align*}
%KL(y||x) &= KL(x||x) + \nabla KL(x||x)^\t (y-x) + \frac{1}{2}(y-x)^\t \nabla^2 KL(z||x) (y-x)\\
%&= \frac{1}{2}(y-x)^\t \nabla^2 KL(z||x) (y-x)\\
%&= \frac{1}{2}\norm{x-y}_{1/z}^2
%\end{align*}
%\end{proof}
\subsection{Union Bound} \label{app:union bound}
Lastly, we provide an upper bound on the last component:
\begin{align*}
\sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^*.
\end{align*}
Because $y_N^*$ depends on $g_n$, this term does not obey martingale concentration like the first component $\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n$ which we analyzed in \cref{app:martingale concentration} To resolve this issue, we utilize the concept of covering number and derive a union bound.
Recall for a compact set $\ZZ$ in a norm space, the covering number $\NN(\ZZ,\epsilon)$ with $\epsilon>0$ is the minimal number of $\epsilon$-balls that covers $\ZZ$. That is, there is a set $ \{ z_i \in \ZZ \}_{i=1}^{\NN(\ZZ,\epsilon)}$ such that $\max_{z\in\ZZ} \min_{z'\in B(\ZZ,\epsilon)} \norm{z-z'} \leq \epsilon$. Usually the covering number $\NN(\ZZ,\epsilon)$ is polynomial in $\epsilon$ and perhaps exponential in the ambient dimension of $\ZZ$.
The idea of covering number can be used to provide a union bound of concentration over compact sets, which we summarize as a lemma below.
\begin{lemma} \label{lm:union bound of Lipshitz functions}
Let $f,g$ be two random $L$-Lipschitz functions. Suppose for some $a>0$ and some fixed $z\in\ZZ$ selected independently of $f,g$, it holds
\begin{align*}
P\left( \left| f(z) - g(z) \right| > \epsilon \right) \leq \exp\left( -a\epsilon^2\right)
\end{align*}
Then it holds that
\begin{align*}
P\left( \sup_{z\in\ZZ} \left| f(z) - g(z) \right| > \epsilon \right) \leq \NN\left(\ZZ,\frac{\epsilon}{4L}\right) \exp \left( \frac{-a\epsilon^2}{4}\right)
\end{align*}
\end{lemma}
\begin{proof}
Let $\CC$ denote a set of covers of size $\NN(\ZZ,\frac{\epsilon}{4L})$
Then, for any $z \in \ZZ$ which could depend on $f,g$,
\begin{align*}
\left| f(z) - g(z) \right|
&\leq \min_{z' \in \CC} \left| f(z) - f(z') \right| + \left| f(z') - g(z') \right| + \left| g(z') - g(z) \right|\\
%
&\leq \min_{z' \in \CC} 2L\norm{z-z'}+ \left| f(z') - g(z')] \right| \\
%
&\leq \frac{\epsilon}{2} + \max_{z'\in \CC}\left| f(z') - g(z') \right|
\end{align*}
Thus,
$
\sup_{z\in\ZZ} \left| f(z) - g(z) \right| > \epsilon \implies \max_{z'\in \CC}\left| f(z') - g(z') \right| > \frac{\epsilon}{2}
$. Therefore, we have the union bound.
\begin{align*}
P\left( \sup_{z\in\ZZ} \left| f(z) - \E[f(z)] \right| > \epsilon \right) &\leq \NN \left(\ZZ,\frac{\epsilon}{4L}\right) \exp \left( \frac{-a\epsilon^2}{4}\right). \qedhere
\end{align*}
\end{proof}
We now use \cref{lm:union bound of Lipshitz functions} to bound the component $\sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^*$.
We recall by definition, for $x = (\vb, \mub)$,
\begin{align*}
(\nabla h_n(x_n) - g_n)^\t x = (\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \mub + (\bb_{\mub_n}-\gb_{n,v})^\t \vb
\end{align*}
Because $y_N^* = (\vb^{\hat{\pi}_N}, \mub^*)$, we can write the sum of interest as
\begin{align*}
\sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^*
&= \sum_{n=1}^N (\gb_{n,\mu} - \kappa \oneb + \ab_{\vb_n})^\t \mub^* + \sum_{n=1}^N (\gb_{n,v}- \bb_{\mub_n})^\t \vb^{\hat{\pi}_N}
\end{align*}
For the first term, because $\mub^*$ is set beforehand by the MDP definition and does not depend on the randomness during learning, it is a martingale and we can apply the steps in \cref{app:martingale concentration} to show,
\begin{align*}
\sum_{n=1}^N (\gb_{n,\mu} - \kappa \oneb + \ab_{\vb_n})^\t \mub^* =\tilde{O}\left( \frac{\sqrt{NK\log(\frac{1}{\delta})}}{1-\gamma} \right)
\end{align*}
For the second term, because $\vb^{\hat{\pi}_N}$ depends on the randomness in the learning process, we need to use a union bound. Following the steps in \cref{app:martingale concentration}, we see that for some fixed $\vb \in \VV$, it holds
\begin{align*}
P\left( \left| \sum_{n=1}^N (\gb_{n,v}- \bb_{\mub_n})^\t \vb \right| > \epsilon \right) \leq \exp\left( - \frac{(1 - \gamma)^2}{N}\epsilon^2 \right)
\end{align*}
% \begin{align*}
% \sum_{n=1}^N (\gb_{n,v}- \bb_{\mub_n})^\t \vb = \tilde{O}\left( \frac{\sqrt{N\log(\frac{1}{\delta})}}{1-\gamma} \right)
% \end{align*}
where some constants were ignored for the sake of conciseness. Note also that it does not have the $\sqrt{K}$ factor because of
\cref{lm:martingale difference for v}. To apply \cref{lm:union bound of Lipshitz functions}, we need to know the order of covering number of $\VV$. Since $\VV$ is an $|\SS|$-dimensional unit cube in the positive orthant, it is straightforward to show
$
\NN\left(\VV,\epsilon\right) \leq \max(1, (1/\epsilon)^{|\SS|})
$ (by simply discretizing evenly in each dimension). Moreover, the functions $\sum_{n=1}^N \gb_{n,v}^\t \vb$ and $\sum_{n=1}^N \bb_{\mub_n}^\t \vb$ are $\frac{N}{1-\gamma}$-Lipschitz in $\norm{\cdot}_\infty$.
Applying \cref{lm:union bound of Lipshitz functions} then gives us:
\begin{align*}
P\left( \sup_{v\in\VV} \left| \sum_{n=1}^N (\gb_{n,v}- \bb_{\mub_n})^\t \vb \right| > \epsilon \right) \leq \NN \left(\VV,\frac{\epsilon (1 - \gamma)}{4N}\right) \exp\left( - \frac{(1 - \gamma)^2}{4N} \epsilon^2\right).
\end{align*}
% This would imply that
% \begin{align*}
% \sup_{v\in\VV} \sum_{n=1}^N (\gb_{n,v}- \bb_{\mub_n})^\t \vb = \epsilon = \tilde{O}\left( \frac{\sqrt{N\log(\frac{\NN\left(\VV,\frac{\epsilon(1-\gamma)}{N}\right)}{\delta})}}{1-\gamma} \right)= \tilde{O}\left( \frac{\sqrt{N|\SS|\log(\frac{1}{\delta})}}{1-\gamma} \right).
% \end{align*}
For a given $\delta$, we thus want to find the smallest $\epsilon$ such that:
\begin{align*}
\delta &\geq \NN \left(\VV,\frac{\epsilon (1 - \gamma)}{4N}\right) \exp\left( - \frac{(1 - \gamma)^2}{4N} \epsilon^2\right).
\end{align*}
That is:
\begin{align*}
\log(\frac{1}{\delta}) &\leq \frac{(1 - \gamma)^2}{4N} \epsilon^2 + |\SS| \min(0, \log(\frac{\epsilon (1 - \gamma)}{4N})).
\end{align*}
Picking $\epsilon = O\left(\log(N) \frac{\sqrt{N\log(\frac{1}{\delta})}}{1-\gamma} \right) = \tilde{O}\left( \frac{\sqrt{N\log(\frac{1}{\delta})}}{1-\gamma} \right) $ guarantees that the inequality is verified asymptotically.
Combining these two steps, we have shown overall, with probability at least $1-\delta$,
\begin{align*}
\sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^* =\tilde{O}\left( \frac{\sqrt{NK\log(\frac{1}{\delta})}}{1-\gamma} \right).
\end{align*}
% \red{
% The functions $\gb_{n,v}^\t \vb$ and $\bb_{\mub_n}^\t \vb$ are $\frac{1}{1-\gamma}$-Lipschitz in $\norm{\cdot}_\infty$, we then take a sum. A posteriori, we need $L = \frac{\sqrt{N}}{1 - \gamma}$ for a direct application of Lemma 15. Feels like it should be true w.h.p. Overall, there's a non-trivial inversion between the martingale concentration inequality (that provides the $\sqrt{N}$ bound) and Lemma 15 (that handles the sup).
% With that L, we want for a given $\delta$ to find the smallest $\epsilon$ such that:
% \begin{align*}
% \log(\frac{1}{\delta}) \leq \frac{(1 - \gamma)^2}{N} \epsilon^2 + |\SS| \min(0, \log(\frac{\epsilon (1 - \gamma)}{\sqrt{N}}))
% \end{align*}
% The value above satisfies the inequality asymptotically.
% We have the following:
% \begin{align*}
% P\left( \left| \sum_{n=1}^N (\gb_{n,v}- \bb_{\mub_n})^\t \vb \right| > \epsilon \right) \leq \exp\left( - \frac{(1 - \gamma)^2}{N} \epsilon^2\right)
% \end{align*}
% where the function is $L = \frac{N}{1 - \gamma}$-Lipschitz. Applying \cref{lm:union bound of Lipshitz functions} then gives us:
% \begin{align*}
% P\left( \sup_{v\in\VV} \left| \sum_{n=1}^N (\gb_{n,v}- \bb_{\mub_n})^\t \vb \right| > \epsilon \right) \leq \NN \left(\VV,\frac{\epsilon (1 - \gamma)}{4N}\right) \exp\left( - \frac{(1 - \gamma)^2}{4N} \epsilon^2\right)
% \end{align*}
% For a given $\delta$, we want to find the smallest $\epsilon$ such that:
% \begin{align*}
% \delta &\geq \NN \left(\VV,\frac{\epsilon (1 - \gamma)}{4N}\right) \exp\left( - \frac{(1 - \gamma)^2}{4N} \epsilon^2\right)
% \end{align*}
% That is:
% \begin{align*}
% \log(\frac{1}{\delta}) &\leq \frac{(1 - \gamma)^2}{4N} \epsilon^2 + |\SS| \min(0, \log(\frac{\epsilon (1 - \gamma)}{4N}))
% \end{align*}
% With $\epsilon = \sqrt{N}$, the second term seems to "win" asymptotically.
% }
% \cheng{Why is there a $0$ in $\min$? I can see your concern. So when $\epsilon= \sqrt{N}$ the second term goes to negative and therefore the inequality will break. But I think the inequality asymptotically holds when $\epsilon$ is a little bit larger than $\sqrt{N}$, For example, if $\epsilon = \sqrt{N} \log N$, the first term grows in $\log(N)^2$ and the second term is no smaller than $-\log N$. So it should be okay when $N$ is large.}
\subsection{Summary}
In the previous subsections, we have provided high probability upper bounds for each term in the decomposition
\begin{align*}
\regret_N (y_N^*)\leq
\left(\sum_{n=1}^{N} (\nabla h_n(x_n)-g_n)^\t x_n\right)
+ \left( \max_{x\in\XX} \sum_{n=1}^N g_n^\t (x_n -x) \right)+ \left(\sum_{n=1}^N (g_n - \nabla h_n(x_n))^\t y_N^* \right)
\end{align*}
implying with probability at least $1-\delta$,
\begin{align*}
\regret_N (y_N^*) \leq \tilde{O}\left( \frac{\sqrt{NK\log(\frac{1}{\delta})}}{1-\gamma} \right) + \tilde{O}\left(\frac{\sqrt{KN}}{1-\gamma} + \frac{\sqrt{K^3 \log\frac{1}{\delta}}}{1-\gamma} \right)
= \tilde{O}\left( \frac{\sqrt{N|\SS||\AA|\log(\frac{1}{\delta})}}{1-\gamma} \right)
\end{align*}
By \cref{th:reduction of RL}, this would imply with probability at least $1-\delta$,
\begin{align*}
V^{\hat{\pi}_N}(p) \geq V^*(p) - \frac{\regret_N(y_N^*)}{N} \geq V^*(p) - \tilde{O}\left( \frac{\sqrt{|\SS||\AA|\log(\frac{1}{\delta})}}{(1-\gamma)\sqrt{N}} \right)
\end{align*}
In other words, the sample complexity of mirror descent to obtain an $\epsilon$ approximately optimal policy (i.e. $ V^*(p) - V^{\hat{\pi}_N}(p) \leq \epsilon$) is at most
$
\tilde{O}\left(\frac{|\SS||\AA|\log(\frac{1}{\delta})}{(1-\gamma)^2 \epsilon^2} \right)
$.
\section{Sample Complexity of Mirror Descent with Basis Functions} \label{app:sample complexity using function approximators}
%
Here we provide further discussions on the sample complexity of running \cref{alg:md for RL} with linearly parameterized function approximators and the proof of \cref{th:sample complexity of mirror descent with basis}.
\SampleComplexityOfMirrorDescentWithBasis*
%
%It is easy to anticipate that using function approximators would make the update of each iteration faster. Here we will also show that the sample complexity now favorably scales in
%$
%\tilde{O}\left(\frac{dim(\Theta)\log(\frac{1}{\delta})}{(1-\gamma)^2 \epsilon^2} \right)
%$ independent of the problem size. The price we pay for this improvement is the function approximation error $\epsilon_{\Theta,N}$ in \cref{cr:reduction for funcapp}, which we recall is defined as below.
%\ReductionForFunctionApporximator*
\subsection{Setup}
We suppose that the decision variable is parameterized in the form $x_\theta = (\Phib\thetab_v, \Psib\thetab_\mu)$, where $\Phib,\Psib$ are given nonlinear basis functions and $(\thetab_v,\thetab_\mu) \in \Theta$ are the parameters to learn.
For modeling the value function, we suppose each column in $\Phib$ is a vector (i.e. function) such that its $\norm{\cdot}_\infty$ is less than one. For modeling the state-action distribution, we suppose each column in $\Psib$ is a state-action distribution from which we can draw samples.
This choice implies that every column of $\Phib$ belongs to $\VV$, and every column of $\Psib$ belongs to $\MM$.
Considering the geometry of $\Phib$ and $\Psib$, we consider a compact and convex parameter set
\begin{align*}
\Theta = \{ \theta = (\thetab_v, \thetab_\mu): \norm{\thetab_v}_2 \leq \frac{C_v}{\sqrt{dim(\thetab_v)}}, \thetab_\mu \geq 0, \norm{\thetab_\mu}_1 = 1 \}
\end{align*}
where $C_v < \infty$.
The constant $C_v$ acts as a regularization in learning: if it is too small, the bias (captured as $\epsilon_{\Theta,N}$ in \cref{cr:reduction for funcapp} restated below) becomes larger; if it is too large, the learning becomes slower.
This choice of $\Theta$ makes sure, for $\theta = (\thetab_v, \thetab_\mu) \in \Theta$, $\Psib\thetab_\mu \in \MM$ and $\norm{\Phib\thetab_v}_\infty \leq \norm{\thetab_v}_1 \leq C_v$.
Therefore, by the above construction, we can verify that the requirement in \cref{cr:reduction for funcapp} is satisfied, i.e. for $\theta = (\thetab_v, \thetab_\mu) \in \Theta$, we have
$(\Phib\thetab_v, \Psib \thetab_\mu) \in \XX_\Theta
$.
\ReductionForFunctionApporximator*
\subsection{Online Loss and Sampled Gradient}
Let $\theta = (\thetab_v,\thetab_\mu) \in \Theta$.
In view of the parameterization above, we can identify the online loss in \eqref{eq:modified per-round loss} in the parameter space as
\begin{align} \label{eq:modified per-round loss (parameter space)}
\textstyle
h_n(\theta) \coloneqq \bb_{\mub_n}^\t \Phib \thetab_v + \thetab_\mu^\t \Psib^\t (\frac{1}{1-\gamma} \oneb - \ab_{\vb_n})
\end{align}
where we have the natural identification $x_n = (\vb_n, \mub_n) = (\Phib\thetab_{v,n}, \Psib\thetab_{\mu,n})$ and $\theta_n = (\thetab_{v,n},\thetab_{\mu,n}) \in \Theta$ is the decision made by the online learner in the $n$th round. Note that because this extension of \cref{alg:md for RL} makes sure $\norm{\thetab_{\mu,n}}_1 = 1$ for every iteration, we can still use $h_n$.
For writing convenience, we will continue to overload $h_n$ as a function of parameter $\theta$ in the following analyses.
%Let $d_v = dim(\thetab_v)$ and $d_\mu = dim(\thetab_\mu)$.
Mirror descent requires gradient estimates of $\nabla h_n(\theta_n)$. Here we construct an unbiased stochastic estimate of $\nabla h_n(\theta_n)$ as
\begin{align} \label{eq:stochastic gradient estimate (parameter space)}
\hspace{-2mm} g_n
= \begin{bmatrix}
\gb_{n,v}\\
\gb_{n,\mu}
\end{bmatrix}
= \begin{bmatrix}
\Phib^\t (\tilde{\pb}_n + \frac{1}{1-\gamma}(\gamma\tilde{\Pb}_n -\Eb_n)^\t \tilde{\mub}_n) \\
dim(\thetab_\mu)\hat{\Psib}_n^\t (\frac{1}{1-\gamma
} \hat{\oneb}_n - \hat{\rb}_n - \frac{1}{1-\gamma}(\gamma\hat{\Pb}_n-\hat{\Eb}_n)\vb_n)
\end{bmatrix}
\end{align}
using two calls of the generative model (again we overload the symbol $g_n$ for the analyses in this section):
\begin{itemize}
\item The upper part $\gb_{n,v}$ is constructed similarly as before in \eqref{eq:stochastic gradient estimate}: First we sample the initial state from the initial distribution,
the state-action pair using the learned state-action distribution, and then the transited-to state at the sampled state-action pair. We evaluate $\Phib$'s values at those samples to construct $\gb_{n,v}$. Thus, $\gb_{n,v}$ would generally be a dense vector of size $dim(\thetab_v)$ (unless the columns of $\Phib$ are sparse to begin with).
\item The lower part $\gb_{n,\mu}$ is constructed slightly differently. Recall for the tabular version in \eqref{eq:stochastic gradient estimate}, we uniformly sample over the state and action spaces. Here instead we first sample uniformly a column (i.e. a basis function) in $\Psib$ and then sample a state-action pair according to the sampled column, which is a distribution by design. Therefore, the multiplier due to uniform sampling in the second row of \eqref{eq:stochastic gradient estimate (parameter space)} is now $dim(\thetab_\mu)$ rather than $|\SS||\AA|$ in \eqref{eq:stochastic gradient estimate}.
The matrix $\hat{\Psib}_n$ is extremely sparse, where only the single sampled entry (the column and the state-action pair) is one and the others are zero.
In fact, one can think of the tabular version as simply using basis functions $\Psib = \Ib$, i.e. each column is a delta distribution. Under this identification, the expression in \eqref{eq:stochastic gradient estimate (parameter space)} matches the one in \eqref{eq:stochastic gradient estimate}.
\end{itemize}
It is straightforward to verify that $\E[g_n] = \nabla h_n(\theta_n)$ for $g_n$ in \eqref{eq:stochastic gradient estimate (parameter space)}.
\subsection{Proof of \cref{th:sample complexity of mirror descent with basis}}
We follow the same steps of the analysis of the tabular version. We will highlight the differences/improvement due to using function approximations.
First, we use \cref{cr:reduction for funcapp} in place of \cref{th:reduction of RL}. To properly handle the randomness, we revisit its derivation to slightly tighten the statement, which was simplified for the sake of cleaner exposition.
Define
\begin{align*}
y_{N,\theta}^* = (\vb_{N,\theta}^*, \mub_\theta^*) \coloneqq \argmax_{x_\theta\in\XX_\theta} r_{ep}(\hat{x}_N; x_\theta).
\end{align*}
For writing convenience, let us also denote $\theta_N^* = (\thetab_{v,N}^*,\thetab_{\mu}^*) \in \Theta$ as the corresponding parameter of $y_{N,\theta}^*$.
We remark that $\mub_\theta^*$ (i.e. $\thetab_{\mu}^*$), which tries to approximate $\mub^*$, is fixed before the learning process, whereas $\vb_{N,\theta}^*$ (i.e. $\thetab_{v,N}^*$) could depend on the stochasticity in the learning.
%Therefore, we will need to use the union bound to handle $\vb_{N,\theta}^*$ like we did for \cref{th:sample complexity of mirror descent}.
Using this new notation and the steps in the proof of \cref{cr:reduction for funcapp}, we can write
\begin{align*}
&V^*(p) - V^{\hat{\pi}_N}(p)
= r_{ep}(\hat{x}_N;y_N^*)\\
&= \epsilon_{\Theta,N} + r_{ep}(\hat{x}_N;y_{N,\theta}^*) \leq \epsilon_{\Theta,N} + \frac{\regret_N(y_{N,\theta}^*)}{N}
\end{align*}
where the first equality is \cref{pr:clever residual bound}, the last inequality follows the proof of \cref{th:reduction of RL}, and we recall the definition
$
\epsilon_{\Theta,N} = r_{ep}(\hat{x}_N;y_N^*) - r_{ep}(\hat{x}_N;y_{N,\theta}^*)
$.
The rest of the proof is very similar to that of \cref{th:reduction of RL}, because linear parameterization does not change the convexity of the loss sequence. Let $y_N^* = (\vb^{\hat{\pi}_N}, \mub^*)$. We bound the regret by the following rearrangement. %, using the fact that $h_n$ in \eqref{eq:per-round loss of RL} is a linear function.
\begin{align} \label{eq:regret rearrangement (parameter space)}
\regret_N (y_{N,\theta}^*)
&= \sum_{n=1}^{N} l_n(x_n) - \sum_{n=1}^N l_n(y_{N,\theta}^*) \nonumber \\
&= \sum_{n=1}^{N} h_n(\theta_n) - \sum_{n=1}^N h_n(\theta_N^* ) \nonumber \\
&= \sum_{n=1}^{N} \nabla h_n(\theta_n)^\t (\theta_n-\theta_N^*) \nonumber \\
%
&= \left(\sum_{n=1}^{N} (\nabla h_n(\theta_n)-g_n)^\t \theta_n\right) + \left(\sum_{n=1}^N g_n^\t (\theta_n - \theta_N^*) \right) + \left(\sum_{n=1}^N (g_n - \nabla h_n(\theta_n))^\t \theta_N^* \right) \nonumber\\
&\leq
\left(\sum_{n=1}^{N} (\nabla h_n(\theta_n)-g_n)^\t \theta_n\right)
+ \left( \max_{\theta\in\Theta} \sum_{n=1}^N g_n^\t (\theta_n -\theta) \right)+ \left(\sum_{n=1}^N (g_n - \nabla h_n(\theta_n))^\t \theta_N^* \right)
\end{align}
where the second equality is due to the identifcation in \eqref{eq:modified per-round loss (parameter space)}.
We will solve this online learning problem with mirror descent
\begin{align} \label{eq:mirror descent (parameter space)}
\theta_{n+1} = \argmin_{\theta\in\Theta} \lr{g_n}{\theta} + \frac{1}{\eta} B_{R}(\theta||\theta_n)
\end{align}
with step size $\eta>0$ and a Bregman divergence that is a straightforward extension of \eqref{eq:Bregman divergence choice}
\begin{align} \label{eq:Bregman divergence choice (paramter space)}
\textstyle
B_R(\theta'||\theta) = \frac{1}{2} \frac{dim(\theta_v)}{C_v^2} \norm{\thetab_v' - \thetab_v}_2^2 + KL(\thetab_\mu'||\thetab_\mu)
\end{align}
where the constant $\frac{dim(\theta_v)}{C_v^2}$ is chosen to make the size of Bregman divergence dimension-free (at least up to $\log$ factors).
Below we analyze the size of the three terms in \eqref{eq:regret rearrangement (parameter space)} like what we did for \cref{th:sample complexity of mirror descent}.
\subsection{The First Term: Martingale Concentration}
The first term is a martingale. We will use this part to highlight the different properties due to using basis functions.
The proof follows the steps in \cref{app:martingale concentration}, but now the martingale difference of interest is instead
\begin{align*}
M_n - M_{n-1}
&= (\nabla h_n(\theta_n)-g_n)^\t \theta_n \\
&= ( \Psib^\t(\kappa \oneb - \ab_{\vb_n}) - \gb_{n,\mu})^\t \thetab_{\mu,n}
+ (\Phib^\t \bb_{\mub_n} -\gb_{n,v})^\t \thetab_{v,n}
\end{align*}
They now have nicer properties due to the way $\gb_{n,\mu}$ is sampled.
For the first term $(\Psib^\t(\kappa \oneb - \ab_{\vb_n}) - \gb_{n,\mu})^\t \thetab_{\mu,n}$, we use the lemma below, where we recall the filtration $F_n$ is naturally defined as $\{\theta_1, \dots, \theta_n\}$.
\begin{lemma} \label{lm:martingale difference for mu (parameter space)}
Let $\theta = (\thetab_v, \thetab_\mu) \in \Theta$ be arbitrary that is chosen independent of the randomness of $\gb_{n,\mu}$ when $F_{n-1}$ is given. Then it holds
$ |(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \thetab| \leq \frac{2(1+dim(\thetab_\mub))}{1-\gamma}$
and $\Vbb_{|F_{n-1}}[(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \thetab_n] \leq \frac{4dim(\thetab_\mub)}{(1-\gamma)^2}$.
\end{lemma}
\begin{proof}
By triangular inequality,
\begin{align*}
|(\Psib^\t(\kappa \oneb - \ab_{\vb_n}) - \gb_{n,\mu})^\t \thetab_\mu| \leq |(\kappa \oneb - \ab_{\vb_n})^\t \Psib \thetab_\mu |+ | \gb_{n,\mu}^\t \thetab_\mu|
\end{align*}
For the deterministic part, using \cref{lm:size of a and b} and H\"older's inequality,
\begin{align*}
|(\kappa \oneb - \ab_{\vb_n})^\t \Psib \thetab_\mu|
\leq \kappa + \norm{\ab_{\vb_n}}_\infty\norm{ \Psib \thetab_\mu}_1 \leq \frac{2}{1-\gamma}
\end{align*}
For the stochastic part, let $k_n$ denote the sampled column index,
$i_n$ be index of the sampled state-action pair using the column of $k_n$, and $j_n$ be the index of the transited state sampled at the pair given by $i_n$. With abuse of notation, we will use $i_n$ to index both $\SS\times\AA$ and $\SS$.
Let $\mub = \Psib \thetab_\mu$. With this notation, we may derive
\begin{align*}
|\gb_{n,\mu}^\t \thetab_\mu |
&= |dim(\thetab_\mu) \thetab_\mub^\t \hat{\Psib}_n^\t (\kappa \hat{\oneb}_n - \hat{\rb}_n - \frac{1}{1-\gamma}(\gamma\hat{\Pb}_n-\hat{\Eb}_n)\vb_n)|\\
%
&=dim(\thetab_\mu) \theta_{\mu, k_n}|\kappa - r_{i_n} - \frac{\gamma v_{n,j_n} - v_{n,i_n}}{1-\gamma} |\\
&\leq \frac{2 dim(\thetab_\mu) \theta_{\mu, k_n}}{1-\gamma} \leq \frac{2 dim(\thetab_\mu)}{1-\gamma}
\end{align*}
where we use the facts $r_{i_n}, v_{n,j_n}, v_{n,i_n} \in [0,1]$ and $\theta_{\mu, k_n} \leq 1$.
Let $\psib_{\mu}^{(k)} $ denote the $k$th column of $\Psib$.
For $\Vbb_{|F_{n-1}}[(\kappa \oneb - \ab_{\vb_n} - \gb_{n,\mu})^\t \thetab_n]$, we can write it as
\begin{align*}
\Vbb_{|F_{n-1}}[(\Psib^\t(\kappa \oneb - \ab_{\vb_n}) - \gb_{n,\mu})^\t \thetab_\mu]
&= \Vbb_{|F_{n-1}}[ \gb_{n,\mu}^\t \thetab_n] \\
&\leq \E_{|F_{n-1}}[ |\gb_{n,\mu}^\t \thetab_n|^2] \\
&= \sum_{k_n} \frac{1}{dim(\thetab_\mu)} \sum_{i_n} \psi_{\mu,i_n}^{(k_n)} \E_{j_n|i_n} \left[ dim(\thetab_\mu)^2 \theta_{\mu, k_n}^2 \left(\kappa - r_{i_n} - \frac{\gamma v_{n,j_n} - v_{n,i_n}}{1-\gamma}\right)^2 \right]\\
&\leq \frac{4 dim(\thetab_\mu) }{(1-\gamma)^2} \sum_{k_n} \theta_{\mu,k_n}^2 \sum_{i_n} \psi_{\mu,i_n}^{(k_n)} \\
&\leq \frac{4 dim(\thetab_\mu) }{(1-\gamma)^2} \left( \sum_{k_n} \theta_{\mu,k_n}\right)^2 \leq \frac{4 dim(\thetab_\mu) }{(1-\gamma)^2}
\end{align*}
where in the second inequality we use the fact that $|\kappa - r_{i_n} - \frac{\gamma v_{n,j_n} - v_{n,i_n}}{1-\gamma} | \leq \frac{2}{1-\gamma}$.
\end{proof}
For the second term $ (\Phib^\t \bb_{\mub_n} -\gb_{n,v})^\t \thetab_{v,n}$, we use this lemma.
\begin{lemma} \label{lm:martingale difference for v (paramter space)}
Let $\thetab \in \VV$ be arbitrary that is chosen independent of the randomness of $\gb_{n,v}$ when $F_{n-1}$ is given. Then it holds
$ |(\Phib^\t \bb_{\mub_n} -\gb_{n,v})^\t \thetab| \leq \frac{4C_v}{1-\gamma}$
and $\Vbb_{|F_{n-1}}[(\Phib^\t \bb_{\mub_n} -\gb_{n,v})^\t \thetab] \leq \frac{4C_v^2}{(1-\gamma)^2}$.
\end{lemma}
\begin{proof}
We appeal to \cref{lm:size of a and b}, which shows $\norm{\bb_{\mub_n}}_1 \leq \frac{2}{1-\gamma}$ and
\begin{align*}
\norm{\tilde{\pb}_n + \frac{1}{1-\gamma}(\gamma\tilde{\Pb}_n -\Eb_n)^\t \tilde{\mub}_n}_1 \leq \frac{2}{1-\gamma}
\end{align*}
Therefore, overall we can derive
\begin{align*}
|(\Phib^\t \bb_{\mub_n} -\gb_{n,v})^\t \thetab| &\leq \left( \norm{\bb_{\mub_n}}_1\ + \norm{\tilde{\pb}_n + \frac{1}{1-\gamma}(\gamma\tilde{\Pb}_n -\Eb_n)^\t \tilde{\mub}_n}_1 \right) \norm{\Phib\thetab_v}_\infty \leq \frac{4 C_v}{1-\gamma}
\end{align*}
where we use again each column in $\Phib$ has $\norm{\cdot}_\infty$ less than one, and $\norm{\cdot}_\infty \leq \norm{\cdot}_2$.
Similarly, for the variance, we can write
\begin{align*}
\Vbb_{|F_{n-1}}[(\Phib^\t \bb_{\mub_n} -\gb_{n,v})^\t \thetab]
&= \Vbb_{|F_{n-1}}[\gb_{n,v}^\t \thetab]\leq \Ebb_{|F_{n-1}}[(\gb_{n,v}^\t \thetab)^2] \leq \frac{4C_v^2}{(1-\gamma)^2} \qedhere
\end{align*}
\end{proof}
From the above two lemmas, we see the main difference from the what we had in \cref{app:martingale concentration} for the tabular case is that, the martingale difference now scales in $O\left(\frac{C_v + dim(\thetab_\mub)}{1-\gamma}\right)$ instead of $O\left(\frac{|\SS||\AA|}{1-\gamma}\right)$, and its variance scales in $O\left(\frac{C_v^2 + dim(\thetab_\mub)}{(1-\gamma)^2}\right)$ instead of $O\left(\frac{|\SS||\AA|}{(1-\gamma)^2}\right)$. We note the constant $C_v$ is universal, independent of the problem size.
Following the similar steps in \cref{app:martingale concentration}, these new results imply that
\begin{align*}
P\left( \sum_{n=1}^{N} (\nabla h_n(\theta_n)-g_n)^\t \theta_n > \epsilon \right) \leq \exp\left(\frac{- \epsilon^2}{2 N \sigma^2 (1+\frac{b\epsilon}{3N\sigma^2})} \right)
\end{align*}
with $b = O\left(\frac{C_v + dim(\thetab_\mub)}{1-\gamma}\right)$ and $O\left(\frac{C_v^2 + dim(\thetab_\mub)}{(1-\gamma)^2}\right)$. This implies that, with probability at least $1-\delta$, it hold
\begin{align*}
\sum_{n=1}^{N} (\nabla h_n(\theta_n)-g_n)^\t \theta_n
= \tilde{O}\left( \frac{\sqrt{N( C_v^2+dim(\thetab_\mub))\log(\frac{1}{\delta})}}{1-\gamma} \right)
\end{align*}
\subsection{Static Regret of Mirror Descent}
Again the steps here are very similar to those in \cref{app:static regret of mirror descent}.
We concern bounding the static regret.
\begin{align*}
\max_{\theta\in\Theta} \sum_{n=1}^N g_n^\t (\theta_n -\theta)
\end{align*}
From \cref{app:static regret of mirror descent}, we recall this can be achieved by the mirror descent's optimality condition. The below inequality is true, for any $\theta'\in\Theta$:
\begin{align*}
\sum_{n=1}^{N} \lr{g_n}{\theta_{n} - \theta'} &\leq \frac{1}{\eta} B_R(\theta'||\theta_1) + \sum_{n=1}^{N} \lr{g_n}{\theta_{n+1} - \theta_n} - \frac{1}{\eta} B_R(\theta_{n+1}||\theta_n)
\end{align*}
Based on our choice of Bregman divergence given in \eqref{eq:Bregman divergence choice (paramter space)}, i.e.
\begin{align} \tag{\ref{eq:Bregman divergence choice (paramter space)}}
\textstyle
B_R(\theta'||\theta) = \frac{1}{2} \frac{dim(\theta_v)}{C_v^2} \norm{\thetab_v' - \thetab_v}_2^2 + KL(\thetab_\mu'||\thetab_\mu),
\end{align}
we have
$
\frac{1}{\eta} B_R(\theta'||\theta_1)\leq \frac{\tilde{O}(1)}{\eta}
$.
For each $\lr{g_n}{\theta_{n+1} - \theta_n} - \frac{1}{\eta} B_R(\theta_{n+1}||\theta_n)$, we will use again the two basic lemmas we proved in \cref{app:static regret of mirror descent}.
\GradientDescentBound*
\EXPBound*
Thus, we have the upper bound
\begin{align*}
\lr{g_n}{\theta_{n+1} - \theta_n} - \frac{1}{\eta} B_R(\theta_{n+1}||\theta_n)
= \frac{C_v^2}{dim(\theta_v)} \frac{\eta \norm{\gb_{n,v}}_2^2}{2} + \frac{\eta \norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2}{2}
\end{align*}
Together with the upper bound on $\frac{1}{\eta} B_R(x'||x_1)$, it implies that
\begin{align} \label{eq:initial telesum of md}
\sum_{n=1}^{N} \lr{g_n}{x_{n} - x'} &\leq \frac{1}{\eta} B_R(x'||x_1) + \sum_{n=1}^{N} \lr{g_n}{x_{n+1} - x_n} - \frac{1}{\eta} B_R(x_{n+1}||x_n) \nonumber \\
&\leq \frac{\tilde{O}(1)}{\eta} + \frac{\eta}{2} \sum_{n=1}^{N} \frac{C_v^2}{dim(\theta_v)} \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2
\end{align}
We can expect, with high probability, $\sum_{n=1}^{N} \frac{C_v^2}{dim(\theta_v)} \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2$ concentrates toward its expectation, i.e.
\begin{align*}
\sum_{n=1}^{N} \frac{C_v^2}{dim(\theta_v)} \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2 \leq \sum_{n=1}^{N} \E\left[\frac{C_v^2}{dim(\theta_v)}\norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2\right] + o(N)
\end{align*}
To bound the right-hand side, we will use the upper bounds below, which largely follow the proof of \cref{lm:martingale difference for mu (parameter space)} and \cref{lm:martingale difference for v (paramter space)}.
\begin{lemma} \label{lm:size of sampled grad (parameter space)}
$
\E[\norm{\gb_{n,v}}_2^2] \leq \frac{4dim(\thetab_v)}{(1-\gamma)^2} $ and $\E[\norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2] \leq \frac{4 dim(\thetab_\mu)}{(1-\gamma)^2}$.
\end{lemma}
\begin{lemma}
$\norm{\gb_{n,v}}_2^2 \leq \frac{4 dim(\thetab_v)}{(1-\gamma)^2}$ and $\norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2 \leq \frac{4 dim(\thetab_\mu)^2}{(1-\gamma)^2}$.
\end{lemma}
By Azuma-Hoeffding's inequality in \cref{lm:martgingale (Hoeffding)},
\begin{align*}
\sum_{n=1}^{N} \frac{C_v^2}{dim(\theta_v)} \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\thetab_{\mub,n}}^2
%
&\leq \sum_{n=1}^{N} \E\left[ \frac{C_v^2}{dim(\theta_v)} \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\thetab_{\mub,n}}^2 \right] +
O\left(\frac{C_v^2+dim(\thetab_\mu)^2}{(1-\gamma)^2} \sqrt{N \log\left(\frac{1}{\delta}\right)} \right) \\
%
&\leq O\left( \frac{C_v^2+dim(\thetab_\mu)}{(1-\gamma)^2} N \right)+ O\left(\frac{C_v^2+dim(\thetab_\mu)^2}{(1-\gamma)^2} \sqrt{N \log\left(\frac{1}{\delta}\right)} \right)
\end{align*}
Now we suppose we set $\eta = O\left( \frac{1-\gamma}{\sqrt{N(C_v^2+dim(\thetab_\mu))}} \right)$. We have
\begin{align*}
\sum_{n=1}^{N} \lr{g_n}{\theta_{n} - \theta'}
&\leq
\frac{\tilde{O}(1)}{\eta} + \frac{\eta}{2} \sum_{n=1}^{N} \frac{C_v^2}{dim(\theta_v)} \norm{\gb_{n,v}}_2^2 + \norm{\gb_{n,\mu}}_{\thetab_{\mu,n}}^2
\leq \tilde{O}\left(\frac{\sqrt{(C_v^2+dim(\thetab_\mu))N}}{1-\gamma} \right)
\end{align*}
\subsubsection{Union Bound}
Lastly we use an union bound to handle the term
\begin{align*}
\sum_{n=1}^N (g_n - \nabla h_n(\theta_n))^\t \theta_N^*
\end{align*}
We follow the steps in \cref{app:union bound}: we will use again the fact that $\theta_N^* = (\thetab_{v,N}^*,\thetab_{\mu}^*) \in \Theta$, so we can handle the part with $\thetab_{\mu}^*$ using the standard martingale concentration, and the part with $\thetab_{v,N}^*$ using the union bound.
Using the previous analyses, we see can first show that the martingale due to the part $\thetab_{\mu}^*$ concentrates in
$
\tilde{O}\left( \frac{\sqrt{Ndim(\thetab_\mub) \log(\frac{1}{\delta})}}{1-\gamma} \right)
$.
Likewise, using the union bound, we can show
the martingale due to the part $\thetab_{v,N}^*$ concentrates in
$
\tilde{O}\left( \frac{\sqrt{N C_v^2\log(\frac{\NN }{\delta})}}{1-\gamma} \right)
$
where $\NN$ some proper the covering number of the set $\left\{\thetab_v : \norm{\thetab_v}_2 \leq \frac{C_v}{\sqrt{dim(\thetab_v)}} \right\}$.
Because $\log\NN = O(\dim(\thetab_v))$ for an Euclidean ball. We can combine the two bounds and show together
\begin{align*}
\sum_{n=1}^N (g_n - \nabla h_n(\theta_n))^\t \theta_N^* = \tilde{O}\left( \frac{\sqrt{N (C_v^2\dim(\thetab_v) +dim(\thetab_\mub)) \log(\frac{1}{\delta})}}{1-\gamma} \right)
\end{align*}
\subsubsection{Summary}
Combining the results of the three parts above, we have, with probability $1-\delta$,
\begin{align*}
&\regret_N (y_{N,\theta}^*)\\
&\leq
\left(\sum_{n=1}^{N} (\nabla h_n(\theta_n)-g_n)^\t \theta_n\right)
+ \left( \max_{\theta\in\Theta} \sum_{n=1}^N g_n^\t (\theta_n -\theta) \right)+ \left(\sum_{n=1}^N (g_n - \nabla h_n(\theta_n))^\t \theta_N^* \right)\\
%
&= \tilde{O}\left( \frac{\sqrt{N(dim(\thetab_\mub) + C_v^2)\log(\frac{1}{\delta})}}{1-\gamma} \right) + \tilde{O}\left(\frac{\sqrt{(C_v^2+dim(\thetab_\mu))N}}{1-\gamma} \right) + \tilde{O}\left( \frac{\sqrt{N (C_v^2\dim(\thetab_v) +dim(\thetab_\mub)) \log(\frac{1}{\delta})}}{1-\gamma} \right)\\
&= \tilde{O}\left( \frac{\sqrt{N dim(\Theta) \log(\frac{1}{\delta})}}{1-\gamma} \right)
\end{align*}
where the last step is due to $C_v$ is a universal constant. Or equivalently, the above bounds means a sample complexity in $\tilde{O}\left( \frac{ dim(\Theta) \log(\frac{1}{\delta})}{(1-\gamma)^2 \epsilon^2} \right)$. Finally, we recall the policy performance has a bias $\epsilon_{\Theta,N}$ in \cref{cr:reduction for funcapp} due to using function approximators. Considering this effect, we have the final statement.
\clearpage
%
%\section{Practical Sampling Strategy and Parameterization} \label{app:parameterization}
%
%The parameterization of $\vb$ and $\mub$ affects whether the resultant online learning problem is still an online convex problem. Therefore, for theoretical purposes, one might consider linear parameterization to retain convexity.
%
%In addition to convexity, when choosing parameterizations one must also consider the available exploration setup, as computing the stochastic estimate of online gradient $g_n$ requires sampling according to the state-action distribution $\mub_n$ in round $n$.
%For the enumeration and the sampling oracles, basically this concerns if sampling from $\mub_n$ can be done efficiently under the parameterization.
%
%
%But for the rollout oracle, the situation becomes more complicated.
%As $\mu_n$ might not satisfy stationarity constraint, sampling from $\mub_n$ can be difficult, \emph{because we do not have direct access to the transition dynamics but can only indirectly generate trajectory samples through rolling out a policy.}
%
%Considering this practical constraint, we propose to parametrize $\mub_n$ as follows:
%Given a set $\BB_n$ of policies, we use a convex parameterization of $\mu_n$:
%\begin{align} \label{eq:convex mixture parameterization}
%\mu_n(s,a) = \sum_{\pi,\pi'\in\BB_n} \eta_{n;\pi,\pi'} d_{\pi'}(s) \pi(a|s)
%\end{align}
%where $\eta_{n;\pi,\pi'} \geq 0$ is the mixing parameters that satisfies $\sum_{\pi,\pi'\in\BB_n} \eta_{n;\pi,\pi'} = 1$ and will be updated by \piccolo. The set $\BB_n$ can be expanded during learning, e.g., by adding a new policy $\pi$ into $\BB_n$ and initializing it with a zero mixing weight.
%
%
%Noticing in~\eqref{eq:convex mixture parameterization} $\eta_{n;\pi,\pi'}$ includes also the cross terms as opposed to the diagonal version
%\begin{align*}
%\mu_n(s,a) = \sum_{\pi\in\BB_n} \eta_{n;\pi} d_{\pi}(s) \pi(a|s) = \sum_{\pi\in\BB_n} \eta_{n;\pi} \mu_\pi(s,a)
%\end{align*}
%By~\eqref{eq:LP (flow)}, optimizing over the diagonal version can be simplified as finding the best policy inside $\BB_n$ (since the constraint of \eqref{eq:LP (flow)} is always satisfied). By contrast the full version expands the size of the search space.
%
%
%Sampling from~\eqref{eq:convex mixture parameterization} can be realized by
%\begin{enumerate}
% \item Sampling $\pi,\pi'$ according to $\eta_{n;\pi,\pi'}$
% \item Rollout $\pi'$ to generate a trajectory (i.e. $\mu_{\pi'}(s,a)$)
% \item Approximate $\mu_n$ by \emph{one-step} importance sampling
% $
% \mu_n(s,a) \approx \mu_{\pi'}(s,a) \frac{\pi(a|s)}{\pi'(a|s)}
% $.
%\end{enumerate}
%The above scheme can be extended by changing how $\pi,\pi'$ are sampled. A nice feature of the above scheme is that as the importance sampling is only \emph{one-step} and not over the entire trajectory, the sampling noise will be constant (as opposed to growing with $\gamma$).
%
%
%In practice, convexity may not matter, since we will be using a deep neural network. We can then consider different specializations/variations of the above scheme in~\eqref{eq:convex mixture parameterization}. For example, we can explicitly separate the role of behavior policy and target policy with the parameterization
%\begin{align*}
%\mu_n(s,a) = d_{\pi_{\eta_n}}(s) \pi_{\theta_n}(a|s)
%\end{align*}
%where $\pi_{\theta_n}$ parameterized by $\theta_n$ is the learner's target policy, and $\pi_{\eta_n}$ parameterized by $\eta_n$ is a behavior policy. Both policies can be e.g. parameterized by a deep neural network if necessary.
%Such choice makes normalizing $\mu_n$ effortless.
%
%
%We can more explicitly relate this parameterization with the above convex one in~\eqref{eq:convex mixture parameterization} if we consider
%the behavior policy defined as a stochastic mixture of policies in $\BB_n \backslash \{\pi_{\theta_n} \}$, i.e.,
%\begin{align*}
%d_{\pi_{\eta_n}} = \sum_{\pi' \in \BB_n \backslash \{\pi_{\theta_n} \}} \eta_{n,\pi'} d_{\pi'}
%\end{align*}
%i.e., in~\eqref{eq:convex mixture parameterization} we have $\eta_{n;\pi,\pi'} = \eta_{n,\pi}$ for $\pi' \neq \pi = \pi_{\theta_n} $ and $\eta_{n;\pi,\pi'} = 0 $ otherwise.
%
%From the above examples, we can view the online learning reduction as an \emph{off-policy} policy gradient algorithm, which results from allowing $\mu$ to not satisfy the stationary constraint. The usual policy gradient on the other hand corresponds to the case where $\pi_{\eta_n} = \pi_{\theta_n}$.
%
%Finally we include a note on how to take the gradient with respect to $\pi_{\eta_n}$ when $\pi_{\eta_n}$ is just a parameterized conditional distribution (as opposed to stochastic mixing like above). Because $d_{\pi_{\eta_n}}$ is generated indirectly from $\pi_{\eta_n}$ we need to also back propagate through the transition dynamics.
%Suppose we want to take the derivative of $\E_{s \sim d_{\pi}} [f(s)]=\db_{\pi}^\t \fb$ with respect to $\pi$.
%
%\begin{lemma} \label{lm:derivative of d pi}
% Suppose $\pi$ is parameterized by $\theta$. We have
% \begin{align*}
% \nabla_\theta \E_{s \sim d_{\pi}} [f(s)]
% = \E_{s \sim d_{\pi}} \left( \nabla_\theta \E_{a \sim \pi|s} \right) \E_{s_1, s_2, \dots \sim \rho_\pi| s_0 = s }\left[ \sum_{t=1}^{\infty} \gamma^t f(s_{t}) \right]
% \end{align*}
%\end{lemma}
%\begin{proof}
% We notice that $d_\pi$ satisfies the stationarity property
% \begin{align*}
% \db_{\pi} = (1-\gamma) \pb + \gamma\Pb_{\pi} \db_{\pi}
% \end{align*}
% or equivalently we can write it as for any $\fb$,
% \begin{align*}
% \db_{\pi}^\t \fb = (1-\gamma) \pb^\t \fb + \gamma \db_{\pi}^\t \Pb_{\pi}^\t \fb
% \end{align*}
% This implies that
% \begin{align*}
% \nabla_\theta \db_{\pi}^\t \fb
% &= \gamma \db_{\pi}^\t (\nabla_\theta \Pb_{\pi}^\t) \fb + \gamma (\nabla_\theta \db_{\pi}^\t) \Pb_{\pi}^\t \fb \\
% &= \gamma \db_{\pi}^\t (\nabla_\theta \Pb_{\pi}^\t) \fb + \gamma \left( \gamma \db_{\pi}^\t (\nabla_\theta \Pb_{\pi}^\t) \Pb_{\pi}^\t \fb + \gamma (\nabla_\theta \db_{\pi}^\t) \Pb_{\pi}^\t \Pb_{\pi}^\t \fb \right) \\
% &= \gamma \db_{\pi}^\t (\nabla_\theta \Pb_{\pi}^\t) \sum_{t=0}^{\infty} \gamma^t (\Pb_{\pi}^{\t})^t \fb\\
% &= \E_{s \sim d_{\pi}} \left( \nabla_\theta \E_{a \sim \pi|s} \right) \E_{s_1, s_2, \dots \sim \rho_\pi| s_0 = s }\left[ \sum_{t=1}^{\infty} \gamma^t f(s_{t}) \right] \qedhere
% \end{align*}
%\end{proof}
%
%We can use \cref{lm:derivative of d pi} to derive the policy gradient as well.
%\begin{corollary}
% Suppose $\pi$ is parameterized by $\theta$.
% Let $V^\pi(p) = \E_{s \sim d_{\pi}} \E_{a \sim \pi |s} [r(s,a)] $. It holds
% \begin{align*}
% \nabla_\theta V^\pi(p) = \E_{s \sim d_{\pi}} \left( \nabla \E_{a \sim \pi |s} \right) \left( \E_{s_1, a_1, s_2, \dots \rho_\pi |s_0 =s, a_0 = a } \left[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t) \right] \right)
% %= \frac{\E_{s \sim d_{\pi}} \left( \nabla \E_{a \sim \pi |s} \right) [Q_\pi(s,a)]}{1-\gamma}
% \end{align*}
%\end{corollary}
%\begin{proof}
% We can first write
% \begin{align*}
% \nabla_\theta V^\pi(p) = \left( \nabla_\theta \E_{s \sim d_{\pi}} \right) \E_{a \sim \pi |s} [r(s,a)] + \E_{s \sim d_{\pi}} \left( \nabla_\theta \E_{a \sim \pi |s} \right) [r(s,a)]
% \end{align*}
% Let $f(s) = \E_{a \sim \pi |s} [r(s,a)]$ and then we can use \cref{lm:derivative of d pi} for the first term. This leads to
% \begin{align*}
% \nabla_\theta V^\pi(p)
% &= \E_{s \sim d_{\pi}} \left( \nabla_\theta \E_{a \sim \pi|s} \right) \left[ \sum_{t=1}^{\infty} \gamma^t \E_{s_1, s_2, \dots \rho_\pi } \E_{a_t \sim \pi |s_t}[r(s_t,a_t)] |s_0 = s \right] + \E_{s \sim d_{\pi}} \left( \nabla_\theta \E_{a \sim \pi |s} \right) [r(s,a)] \\
% &= \E_{s \sim d_{\pi}} \left( \nabla \E_{a \sim \pi |s} \right) \E_{s_1, a_1, s_2, \dots \rho_\pi |s_0 =s, a_0 = a } \left[\sum_{t=0}^{\infty} \gamma^t r(s_t,a_t) \right] \qedhere
% %&= \E_{s \sim d_{\pi}} \left( \nabla \E_{a \sim \pi |s} \right) [Q_\pi(s,a)]
% \end{align*}
%\end{proof}
%
%From the above relationship, we see that the separation into behavior policy $\pi_{\eta_n}$ (which generates the state distribution $d_{\pi_{\eta_n}}$) offloads the difficult, noisy gradient away from the target policy $\pi_{\theta_n}$. The mismatch between the behavior policy $d_{\pi_{\eta_n}}$ and the target policy $d_{\pi_{\theta_n}}$ is then handled through the value function estimate $\vb_n$.
%
%
%
%
%%Alternatively, if we choose to parameterize $d_n$ indirectly through a behavior policy $\pi_{\eta_n}$ i.e. $d_n = d_{\pi_{\eta_n}}$. We would need to back propagate through the transition dynamics, i.e. it would require estimating the value of $\pi_{\eta_n}$ as if $r(s,a) + \frac{1}{1-\gamma} \E_{s'\sim\PP|s,a}[V_n(s')] - V_n(s)$ is the instant reward.
\begin{thebibliography}{37}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Manne et~al.(1959)]{manne1959linear}
Alan~S Manne et~al.
\newblock Linear programming and sequential decision models.
\newblock Technical report, Cowles Foundation for Research in Economics, Yale
University, 1959.
\bibitem[Denardo and Fox(1968)]{denardo1968multichain}
Eric~V Denardo and Bennett~L Fox.
\newblock Multichain {M}arkov renewal programs.
\newblock \emph{SIAM Journal on Applied Mathematics}, 16\penalty0 (3):\penalty0
468--487, 1968.
\bibitem[Wang and Chen(2016)]{wang2016online}
Mengdi Wang and Yichen Chen.
\newblock An online primal-dual method for discounted {M}arkov decision
processes.
\newblock In \emph{Conference on Decision and Control}, pages 4516--4521. IEEE,
2016.
\bibitem[Chen and Wang(2016)]{chen2016stochastic}
Yichen Chen and Mengdi Wang.
\newblock Stochastic primal-dual methods and sample complexity of reinforcement
learning.
\newblock \emph{arXiv preprint arXiv:1612.02516}, 2016.
\bibitem[Wang(2017{\natexlab{a}})]{wang2017randomized}
Mengdi Wang.
\newblock Randomized linear programming solves the discounted {M}arkov decision
problem in nearly-linear running time.
\newblock \emph{ArXiv e-prints}, 2017{\natexlab{a}}.
\bibitem[Lee and He(2018)]{lee2018stochastic}
Donghwan Lee and Niao He.
\newblock Stochastic primal-dual {Q}-learning.
\newblock \emph{arXiv preprint arXiv:1810.08298}, 2018.
\bibitem[Wang(2017{\natexlab{b}})]{wang2017primal}
Mengdi Wang.
\newblock Primal-dual {$\pi$} learning: Sample complexity and sublinear run
time for ergodic {M}arkov decision problems.
\newblock \emph{arXiv preprint arXiv:1710.06100}, 2017{\natexlab{b}}.
\bibitem[Lin et~al.(2017)Lin, Nadarajah, and Soheili]{lin2017revisiting}
Qihang Lin, Selvaprabu Nadarajah, and Negar Soheili.
\newblock Revisiting approximate linear programming using a saddle point based
reformulation and root finding solution approach.
\newblock Technical report, working paper, U. of Il. at Chicago and U. of Iowa,
2017.
\bibitem[Dai et~al.(2018)Dai, Shaw, He, Li, and Song]{dai2017boosting}
Bo~Dai, Albert Shaw, Niao He, Lihong Li, and Le~Song.
\newblock Boosting the actor with dual critic.
\newblock In \emph{International Conference on Learning Representation}, 2018.
\bibitem[Chen et~al.(2018)Chen, Li, and Wang]{chen2018scalable}
Yichen Chen, Lihong Li, and Mengdi Wang.
\newblock Scalable bilinear {$\pi$} learning using state and action features.
\newblock \emph{arXiv preprint arXiv:1804.10328}, 2018.
\bibitem[Lakshminarayanan et~al.(2018)Lakshminarayanan, Bhatnagar, and
Szepesv{\'a}ri]{lakshminarayanan2018linearly}
Chandrashekar Lakshminarayanan, Shalabh Bhatnagar, and Csaba Szepesv{\'a}ri.
\newblock A linearly relaxed approximate linear program for {M}arkov decision
processes.
\newblock \emph{IEEE Transactions on Automatic Control}, 63\penalty0
(4):\penalty0 1185--1191, 2018.
\bibitem[Gordon(1999)]{gordon1999regret}
Geoffrey~J Gordon.
\newblock Regret bounds for prediction problems.
\newblock In \emph{Conference on Learning Theory}, volume~99, pages 29--40,
1999.
\bibitem[Zinkevich(2003)]{zinkevich2003online}
Martin Zinkevich.
\newblock Online convex programming and generalized infinitesimal gradient
ascent.
\newblock In \emph{International Conference on Machine Learning}, pages
928--936, 2003.
\bibitem[Cheng et~al.(2019{\natexlab{a}})Cheng, Lee, Goldberg, and
Boots]{cheng2019online}
Ching-An Cheng, Jonathan Lee, Ken Goldberg, and Byron Boots.
\newblock Online learning with continuous variations: Dynamic regret and
reductions.
\newblock \emph{arXiv preprint arXiv:1902.07286}, 2019{\natexlab{a}}.
\bibitem[Blum(1994)]{blum1994optimization}
Eugen Blum.
\newblock From optimization and variational inequalities to equilibrium
problems.
\newblock \emph{Math. student}, 63:\penalty0 123--145, 1994.
\bibitem[Bianchi and Schaible(1996)]{bianchi1996generalized}
M~Bianchi and S~Schaible.
\newblock Generalized monotone bifunctions and equilibrium problems.
\newblock \emph{Journal of Optimization Theory and Applications}, 90\penalty0
(1):\penalty0 31--43, 1996.
\bibitem[Rakhlin and Sridharan(2012)]{rakhlin2012online}
Alexander Rakhlin and Karthik Sridharan.
\newblock Online learning with predictable sequences.
\newblock \emph{arXiv preprint arXiv:1208.3728}, 2012.
\bibitem[Cheng et~al.(2019{\natexlab{b}})Cheng, Yan, Ratliff, and
Boots]{cheng2018predictor}
Ching-An Cheng, Xinyan Yan, Nathan Ratliff, and Byron Boots.
\newblock Predictor-corrector policy optimization.
\newblock In \emph{International Conference on Machine Learning},
2019{\natexlab{b}}.
\bibitem[Kearns and Singh(1999)]{kearns1999finite}
Michael~J Kearns and Satinder~P Singh.
\newblock Finite-sample convergence rates for q-learning and indirect
algorithms.
\newblock In \emph{Advances in neural information processing systems}, pages
996--1002, 1999.
\bibitem[Kakade et~al.(2003)]{kakade2003sample}
Sham~Machandranath Kakade et~al.
\newblock \emph{On the sample complexity of reinforcement learning}.
\newblock PhD thesis, University of London London, England, 2003.
\bibitem[Puterman(2014)]{puterman2014markov}
Martin~L Puterman.
\newblock \emph{Markov decision processes: discrete stochastic dynamic
programming}.
\newblock John Wiley \& Sons, 2014.
\bibitem[Hern{\'a}ndez-Lerma and Lasserre(2012)]{hernandez2012discrete}
On{\'e}simo Hern{\'a}ndez-Lerma and Jean~B Lasserre.
\newblock \emph{Discrete-time Markov control processes: basic optimality
criteria}, volume~30.
\newblock Springer Science \& Business Media, 2012.
\bibitem[Bellman(1954)]{bellman1954theory}
Richard Bellman.
\newblock The theory of dynamic programming.
\newblock \emph{Bulletin of the American Mathematical Society}, 60\penalty0
(6):\penalty0 503--515, 1954.
\bibitem[Nemirovski et~al.(2009)Nemirovski, Juditsky, Lan, and
Shapiro]{nemirovski2009robust}
Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro.
\newblock Robust stochastic approximation approach to stochastic programming.
\newblock \emph{SIAM Journal on optimization}, 19\penalty0 (4):\penalty0
1574--1609, 2009.
\bibitem[Juditsky et~al.(2011)Juditsky, Nemirovski, and
Tauvel]{juditsky2011solving}
Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel.
\newblock Solving variational inequalities with stochastic mirror-prox
algorithm.
\newblock \emph{Stochastic Systems}, 1\penalty0 (1):\penalty0 17--58, 2011.
\bibitem[Cesa-Bianchi and Lugosi(2006)]{cesa2006prediction}
Nicolo Cesa-Bianchi and Gabor Lugosi.
\newblock \emph{Prediction, learning, and games}.
\newblock Cambridge university press, 2006.
\bibitem[Shalev-Shwartz et~al.(2012)]{shalev2012online}
Shai Shalev-Shwartz et~al.
\newblock Online learning and online convex optimization.
\newblock \emph{Foundations and Trends{\textregistered} in Machine Learning},
4\penalty0 (2):\penalty0 107--194, 2012.
\bibitem[Hazan et~al.(2016)]{hazan2016introduction}
Elad Hazan et~al.
\newblock Introduction to online convex optimization.
\newblock \emph{Foundations and Trends{\textregistered} in Optimization},
2\penalty0 (3-4):\penalty0 157--325, 2016.
\bibitem[Jofr{\'e} and Wets(2014)]{jofre2014variational}
Alejandro Jofr{\'e} and Roger J-B Wets.
\newblock Variational convergence of bifunctions: motivating applications.
\newblock \emph{SIAM Journal on Optimization}, 24\penalty0 (4):\penalty0
1952--1979, 2014.
\bibitem[Abernethy et~al.(2011)Abernethy, Bartlett, and
Hazan]{abernethy2011blackwell}
Jacob Abernethy, Peter~L Bartlett, and Elad Hazan.
\newblock Blackwell approachability and no-regret learning are equivalent.
\newblock In \emph{Annual Conference on Learning Theory}, pages 27--46, 2011.
\bibitem[Ng et~al.(1999)Ng, Harada, and Russell]{ng1999policy}
Andrew~Y Ng, Daishi Harada, and Stuart Russell.
\newblock Policy invariance under reward transformations: Theory and
application to reward shaping.
\newblock In \emph{International Conference on Machine Learning}, volume~99,
pages 278--287, 1999.
\bibitem[Kakade and Langford(2002)]{kakade2002approximately}
Sham Kakade and John Langford.
\newblock Approximately optimal approximate reinforcement learning.
\newblock In \emph{International Conference on Machine Learning}, volume~2,
pages 267--274, 2002.
\bibitem[Ross et~al.(2011)Ross, Gordon, and Bagnell]{ross2011reduction}
St{\'e}phane Ross, Geoffrey Gordon, and Drew Bagnell.
\newblock A reduction of imitation learning and structured prediction to
no-regret online learning.
\newblock In \emph{International Conference on Artificial Intelligence and
Statistics}, pages 627--635, 2011.
\bibitem[Cheng and Boots(2018)]{cheng2018convergence}
Ching-An Cheng and Byron Boots.
\newblock Convergence of value aggregation for imitation learning.
\newblock \emph{International Conference on Artificial Intelligence and
Statistics}, 2018.
\bibitem[Cheng et~al.(2019{\natexlab{c}})Cheng, Yan, Theodorou, and
Boots]{cheng2018accelerating}
Ching-An Cheng, Xinyan Yan, Evangelos~A Theodorou, and Byron Boots.
\newblock Accelerating imitation learning with predictive models.
\newblock In \emph{International Conference onArtificial Intelligence and
Statistics}, 2019{\natexlab{c}}.
\bibitem[Hazan(2016)]{journals/ftopt/Hazan16}
Elad Hazan.
\newblock Introduction to online convex optimization.
\newblock \emph{Foundations and Trends in Optimization}, 2\penalty0
(3-4):\penalty0 157--325, 2016.
\newblock URL
\url{http://dblp.uni-trier.de/db/journals/ftopt/ftopt2.html#Hazan16}.
\bibitem[McDiarmid(1998)]{mcdiarmid1998concentration}
Colin McDiarmid.
\newblock Concentration.
\newblock In \emph{Probabilistic methods for algorithmic discrete mathematics},
pages 195--248. Springer, 1998.
\end{thebibliography}