Inverse Q-Learning Done Right: Offline Imitation Learning in $Q^\pi$-Realizable MDPs
Extracted source text · arXiv 2505.19946
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
\usepackage[preprint]{settings/neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{ninecolors}
\usepackage[colorlinks=true, linkcolor=blue3, citecolor=blue3, urlcolor=blue3]{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{wrapfig}
\input{settings/lmacros.tex}
\input{settings/amacros.tex}
\input{settings/gmacros.tex}
\usepackage{booktabs}
\usepackage{pifont}
\newcommand{\blackcross}{\ding{55}}
\usepackage{multirow}
\usepackage{array}
\newcommand{\PreserveBackslash}[1]{\let\temp=\\#1\let\\=\temp}
%\newcolumntype{C}[1]{>{\PreserveBackslash\centering}m{#1}}
\newcolumntype{C}[1]{>{\centering\arraybackslash}m{#1}}
\newcolumntype{R}[1]{>{\PreserveBackslash\raggedleft}p{#1}}
\newcolumntype{L}[1]{>{\PreserveBackslash\raggedright}p{#1}}
% \usepackage{todonotes}
% \definecolor{PalePurp}{rgb}{0.66,0.57,0.66}
% \newcommand{\todoG}[1]{\todo[color=PalePurp!30]{\textbf{Grg:} #1}}
% \newcommand{\todoGI}[1]{\todo[inline,color=PalePurp!30]{#1}}
% \newcommand{\grg}[1]{\textcolor{red}{[\textbf{Grg:} #1]}}
% \newcommand{\redd}[1]{\textcolor{red}{#1}}
\usepackage{pgfplots}
\pgfplotsset{compat=1.17}
\usepackage{enumitem}
% for table of contents appendix
\usepackage{tocloft}
\addtocontents{toc}{\protect\setcounter{tocdepth}{0}}
% \title{
% Offline Imitation Learning in $Q^\pi$-Realizable MDPs without Expert Realizability
% }
\title{
Inverse Q-Learning Done Right:
\\
Offline Imitation Learning in $Q^\pi$-Realizable MDPs
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Antoine Moulin \\
Universitat Pompeu Fabra \\
\texttt{antoine.moulin@upf.edu} \\
\And
Gergely Neu \\
Universitat Pompeu Fabra \\
\texttt{gergely.neu@gmail.com} \\
\And
Luca Viano \\
EPFL\\
\texttt{luca.viano@epfl.ch} \\
}
\begin{document}
\maketitle
\begin{abstract}
We study the problem of offline imitation learning in Markov decision processes (MDPs), where the goal is to learn a well-performing policy given a dataset of state-action pairs generated by an expert policy. Complementing a recent line of work on this topic that assumes the expert belongs to a tractable class of known policies, we approach this problem from a new angle and leverage a different type of structural assumption about the environment. Specifically, for the class of linear $Q^\pi$-realizable MDPs, we introduce a new algorithm called saddle-point offline imitation learning (\SPOIL), which is guaranteed to match the performance of any expert up to an additive error $\varepsilon$ with access to $\mathcal{O}(\varepsilon^{-2})$ samples. Moreover, we extend this result to possibly non-linear $Q^\pi$-realizable MDPs at the cost of a worse sample complexity of order $\mathcal{O}(\varepsilon^{-4})$. Finally, our analysis suggests a new loss function for training critic networks from expert data in deep imitation learning. Empirical evaluations on standard benchmarks demonstrate that the neural net implementation of \SPOIL is superior to behavior cloning and competitive with state-of-the-art algorithms.
\end{abstract}
\input{sections/introduction.tex}
\input{sections/preliminaries.tex}
\input{sections/algorithm.tex}
\input{sections/analysis.tex}
\input{sections/experiments.tex}
\input{sections/conclusion.tex}
\begin{ack}
Luca Viano is funded through a PhD fellowship of the Swiss Data Science Center, a joint venture between EPFL and ETH Zurich. Gergely Neu and Antoine Moulin are funded through a European Research Council (ERC) project, under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 950180).
\end{ack}
\bibliography{ref}
\bibliographystyle{plainnat}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\renewcommand{\contentsname}{Contents of Appendix}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}
{
% \hypersetup{hidelinks}
\setlength{\cftbeforesecskip}{.8em}
\setlength{\cftbeforesubsecskip}{.5em}
\tableofcontents
}
\newpage
\input{sections/additional_related_works.tex}
\newpage
\input{sections/omitted_proofs.tex}
\newpage
\input{sections/technical_tools.tex}
\newpage
\input{sections/misspecified_bc.tex}
\newpage
\input{sections/experimental_details.tex}
\newpage
\input{sections/future.tex}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
\usepackage{amsmath,amssymb,amsthm}
\usepackage{thmtools}
\usepackage{thm-restate}
%\usepackage[intlimits]{amsmath}
\usepackage{mathtools}
\usepackage{dsfont}
\usepackage{stmaryrd}
\usepackage{comment}
\usepackage{setspace}
%\usepackage{thmtools,thm-restate}
%\usepackage{cite}
% if you use cleveref..
\usepackage{tcolorbox}
%\usepackage{hyperref}
% \usepackage[unicode,psdextra]{hyperref}
\newcommand\pdfmath[1]{\texorpdfstring{$#1$}{#1}}
\usepackage[capitalize,noabbrev]{cleveref}
% brackets
\newcommand{\bc}[1]{\left\{{#1}\right\}} % curly brackets
\newcommand{\br}[1]{\left({#1}\right)} % parentheses
\newcommand{\bs}[1]{\left[{#1}\right]} % square brackets
\newcommand{\abs}[1]{\left| {#1} \right|} % absolute value
\newcommand{\ceil}[1]{\left\lceil #1 \right\rceil} % ceil function
\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor} % floor function
\newcommand{\bsd}[1]{\left\llbracket{#1}\right\rrbracket} % double square brackets
\newcommand{\ip}[2]{\left\langle{#1},{#2}\right\rangle} % angle brackets (inner product)
\newcommand{\inner}[1]{\langle#1\rangle} % inner product
\newcommand{\norm}[1]{\left\| {#1} \right\|} % norm
% custom
\newcommand{\RMAX}{R_{\max}}
\newcommand{\WMAX}{W_{\max}}
% probabilities
\renewcommand{\P}[1]{\mathbb{P}\bs{{#1}}}
\newcommand{\Pp}[2]{\underset{#1}{\mathbb{P}}\bs{{#2}}}
\newcommand{\Ppp}[3]{\mathbb{P}_{#1}^{#2}\bs{{#3}}}
% expectations
\newcommand{\E}[1]{\mathbb{E}\bs{{#1}}}
\newcommand{\Ee}[2]{\underset{#1}{\mathbb{E}}\bs{{#2}}}
\newcommand{\Eee}[2]{\mathbb{E}_{#1}\bs{{#2}}}
% argmin and argmax
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\newcounter{protocol}
\makeatletter
\newenvironment{protocol}[1][htb]{%
\let\c@algorithm\c@protocol
\renewcommand{\ALG@name}{Protocol}% Update algorithm name
\begin{algorithm}[#1]%
}{\end{algorithm}
}
\usepackage{amssymb}
% \usepackage{amsfonts}
\usepackage{amsthm}
\usepackage{graphics}
\usepackage{graphicx}
\usepackage{setspace}
\usepackage{subcaption}
%\usepackage{selectp}
%\outputonly{1-18}
\newtheorem{theorem}{\protect\theoremname}
\newtheorem{corollary}{\protect\corrolaryname}
% \newtheorem{defn}{\protect\definitionname}
\newtheorem{definition}{\protect\definitionname}
\newtheorem{proposition}{\protect\propositionname}
\newtheorem{example}{\protect\examplename}
\newtheorem{remark}{Remark}
\newtheorem{conditions}{\protect\conditionsname}
\newtheorem{assumption}{\protect\assumptionname}
\newtheorem*{assumption*}{\protect\assumptionname}
\newenvironment{proofof}[1]{\begin{proof}[{#1}]}{\end{proof}}
\providecommand{\definitionname}{Definition}
\providecommand{\examplename}{Example}
\providecommand{\lemmaname}{Lemma}
\providecommand{\corrolaryname}{Corollary}
\providecommand{\propositionname}{Proposition}
\providecommand{\conditionsname}{Conditions}
\providecommand{\theoremname}{Theorem}
\providecommand{\assumptionname}{Assumption}
\def \M{\mathcal{M}}
\def \S{\mathcal{S}}
\def \A{\mathcal{A}}
\def \R{\mathbb{R}}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2022}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2022}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2022}
%commands from AK
%====================================================================================================================================================================================
%new
%state space S
\newcommand{\qv}{\mbf{q}}
\newcommand{\dv}{\mbf{d}}
\newcommand{\dve}{\mbf{d}^E}
\newcommand{\trans}{\intercal}
\newcommand{\qval}{\mbf{Q}}
\newcommand{\FEV}[1]{\mbs{\rho}_{\phim}(#1)}
\newcommand{\EFEV}[1]{\mbs{\rho}_{\phim}(\widehat{#1})}
%==============================================================================
%old
\newcommand{\mcf}{\mathcal}
\newcommand{\mbf}{\mathbf}
\newcommand{\mbb}{\mathbb}
\DeclareMathOperator{\Exp}{\mathds{E}} % expectation
\DeclareMathOperator{\Prob}{\mathds{P}} %probability
\newcommand{\ones}{\mathds{1}}
\renewcommand{\Re}{\mathds{R}} %real number
\newcommand{\ar}{\mathds{R}} %real number
\newcommand{\sspace}{\mcf{S}} % state space
\newcommand{\aspace}{\mcf{A}} % action space
\newcommand{\wspace}{\mcf{W}} % cost weights space
\newcommand{\histspace}{\mcf{H}} % history space
\newtcolorbox{boxF}{
boxrule = 1.5pt,
colframe = white, % making the base for dash line
}
\newcommand{\abss}[1]{|#1|} %absolute value
\newcommand{\normm}[1]{\lVert#1\rVert} %norm
\newcommand{\normbig}[1]{\big\lVert#1\big\rVert}
\newcommand{\half}{\tfrac{1}{2}}
\newcommand{\innerprod}[2]{\left\langle{#1},{#2}\right\rangle}
\newcommand{\expert}{{\pi_{\scriptscriptstyle\textup{E}}}}
\newcommand{\barexpert}{{\bar{\pi}_{\scriptscriptstyle\textup{E}}}}
\newcommand{\experth}{{\pi_{\scriptscriptstyle\textup{E}, h}}}
\newcommand{\PiE}{{\Pi^{\scriptscriptstyle\textup{E}}}}
\newcommand{\PiElin}{{\Pi^{\scriptscriptstyle\textup{E}}_{\mathrm{lin}}}}
\newcommand{\PiENN}{{\Pi^{\scriptscriptstyle\textup{E}}_{\mathrm{NN}}}}
\newcommand{\XEi}{X_{\scriptscriptstyle\textup{E}}^i}
\newcommand{\AEi}{A_{\scriptscriptstyle\textup{E}}^i}
\newcommand{\tauE}{\tau_{\scriptscriptstyle\textup{E}}}
\newcommand{\apprentice}{\pi_{\textup{A}}}
\newcommand{\mbs}{\boldsymbol}
\newcommand{\cost}{\mbf{c}}
\newcommand{\true}{\mbf{c_{\textup{true}}}}
\newcommand{\weight}{\mbf{w}}
\newcommand{\uv}{\mbf{u}}
\newcommand{\mv}{\mbs{\mu}}
\newcommand{\op}{\mbf{T}_{\gamma}}
\newcommand{\initial}{\nu_0}
\newcommand{\val}{\mbf{V}}
\newcommand{\thv}{\mbs{\theta}}
\newcommand{\lv}{\mbs{\lambda}}
\newcommand{\lag}{\mcf{L}}
\newcommand{\rlag}{{\mcf{L}_r}}
\newcommand{\mlag}{\bar{\mcf{L}}_r}
\newcommand{\elag}{\widehat{\mcf{L}}_r}
\newcommand{\phim}{\mbs{\Phi}}
\newcommand{\psim}{\mbs{\Psi}}
\newcommand{\cma}{\mbf{C}}
\newcommand{\psiv}{\mbs{\psi}}
\newcommand{\phiv}{\mbs{\phi}}
\newcommand{\rhov}{\mbs{\rho}}
\newcommand{\fev}{\mbs{\rho}_{\cma}(\widehat{\expert})}
\newcommand{\efev}{\mbs{\rho}_{\cma}(\widehat{\widehat{\expert}})}
\newcommand{\fevphi}{\mbs{\rho}_{\phim}(\widehat{\expert})}
\newcommand{\efevphi}{\mbs{\rho}_{\phim}(\widehat{\widehat{\expert}})}
\newcommand{\mth}{\mv_{\thv}}
\newcommand{\ul}{\uv_{\lv}}
\newcommand{\cw}{\cost_{\weight}}
\newcommand{\cbg}{C_{\beta,\gamma}}
\newcommand{\mexp}{\mv_{\widehat{\expert}}}
\newcommand{\xv}{\mbf{x}}
\newcommand{\yv}{\mbf{y}}
\newcommand{\zv}{\mbf{z}}
\newcommand{\pth}{\pi_{\thv}}
\newcommand{\pem}{\pi_{\mv}}
\newcommand{\tha}{\thv_{\textup{A}}}
\newcommand{\la}{\lv_{\textup{A}}}
\newcommand{\wa}{\weight_{\textup{A}}}
\newcommand{\tht}{\tilde{\thv}}
\newcommand{\ut}{\tilde{\uv}}
\newcommand{\wt}{\tilde{\weight}}
\newcommand{\mt}{\tilde{\mv}}
\newcommand{\lt}{\tilde{\lv}}
\newcommand{\ma}{\mv_{\textup{A}}}
\newcommand{\ua}{\uv_{\textup{A}}}
\newcommand{\mmat}{\mbf{M}}
\newcommand{\bmat}{\mbf{B}}
\newcommand{\pmat}{\mbf{P}}
\newcommand{\eith}{\mbf{e}_{i_{\thv}}}
\newcommand{\cith}{\cost_{i_{\thv}}}
\newcommand{\ith}{i_{\thv}}
\newcommand{\mpa}{\mv_{\apprentice}}
\newcommand{\weirdv}{\val_{\cith}^{\pth}}
\newcommand{\weirdrho}{\rho_{\cith}(\pth)}
\newcommand{\thN}{\widehat{\thv}_N}
\newcommand{\lN}{\widehat{\lv}_N}
\newcommand{\wN}{\widehat{\weight}_N}
\newcommand{\pN}{\widehat{\pi}_N}
\newcommand{\zN}{\widehat{\zv}_N}
\newcommand{\fn}{\mathcal{F}_{n-1}}
\newcommand{\hi}{\hat{\iota}}
\newcommand{\hx}{\hat{x}}
\newcommand{\ha}{\hat{a}}
\newcommand{\hy}{\hat{y}}
\newcommand{\ti}{\tilde{\iota}}
\newcommand{\tx}{\tilde{x}}
\newcommand{\ta}{\tilde{a}}
\newcommand{\ty}{\tilde{y}}
\newcommand{\txp}{\tilde{x}'}
\newcommand{\bi}{\bar{\iota}}
\newcommand{\bx}{\bar{x}}
\newcommand{\ba}{\bar{a}}
\newcommand{\gv}{\mbf{g}}
\newcommand{\gth}{\mbf{g}_{\thv}}
\newcommand{\gl}{\mbf{g}_{\lv}}
\newcommand{\gw}{\mbf{g}_{\weight}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\sadav}{\epsilon_{\textup{sad}}(\zN)}
\newcommand{\mpm}{\mv_{\pem}}
\newcommand{\sad}{\epsilon_{\textup{sad}}(\boldsymbol{\theta},\boldsymbol{\lambda},\weight)}
\newcommand{\lvz}{\lv_0}
\newcommand{\ulz}{\uv_{\lvz}}
\newcommand{\wtrue}{\weight_{\textup{true}}}
%========================================================
% Comment boxes
% \usepackage{xcolor}
\newcommand{\colorNoteBox}[3]{
\begin{center}
\vspace{-2ex}\small
\fcolorbox[rgb]{#1}{#2}{\parbox[t]{\linewidth}{\setlength{\parskip}{1.5ex}%
#3}}
\end{center}}
\newcommand{\LV}[1]{\colorNoteBox{1,0,0}{0.9,0.9,0.9}{LV:~#1}}
\newcommand{\IK}[1]{\colorNoteBox{0,0,1}{0.9,0.9,0.9}{IK:~#1}}
\newcommand{\AK}[1]{\colorNoteBox{0,1,0}{0.9,0.9,0.9}{AK:~#1}}
\newcommand{\AM}[1]{{\color{blue}[AM: #1]}}
% \usepackage[utf8]{inputenc} % allow utf-8 input
% \usepackage[T1]{fontenc} % use 8-bit T1 fonts
% \usepackage{url} % simple URL typesetting
% \usepackage{booktabs} % professional-quality tables
% \usepackage{amsfonts} % blackboard math symbols
% \usepackage{nicefrac} % compact symbols for 1/2, etc.
% \usepackage{microtype} % microtypography
\usepackage{pifont}
\newcommand{\greentick}{\textcolor{green}{\ding{51}}}
\newcommand{\redcross}{\textcolor{red}{\ding{55}}}
%\newtheorem{Lem}[theorem]{Lemma}
\newtheorem{Lem}{Lemma}
\newcommand{\antoine}[1]{%
\ifmmode
\text{\textcolor{blue}{[Antoine: #1]}}
\else
\textcolor{blue}{[Antoine: #1]}
\fi
}
\newcommand{\tmpassumption}[1]{%
\ifmmode
\text{\textcolor{orange}{[Assumption: #1]}}
\else
\textcolor{orange}{[Assumption: #1]}
\fi
}
% common abbreviations
\usepackage{xspace}
\makeatletter
\DeclareRobustCommand\onedot{\futurelet\@let@token\@onedot}
\def\@onedot{\ifx\@let@token.\else.\null\fi\xspace}
\def\cf{\textit{c.f}\onedot}
\def\eg{\textit{e.g}\onedot}
\def\etal{\textit{et al}\onedot}
\def\etc{\textit{etc}\onedot}
\def\ie{\textit{i.e}\onedot}
\def\wrt{i.i.d\onedot}
\def\vs{\textit{vs}\onedot}
\def\wrt{w.r.t\onedot}
\makeatother
% surrounding symbols
%\newcommand{\spr}[1]{\left( #1 \right)}
\DeclarePairedDelimiter{\spr}{(}{)}
%\newcommand{\sbr}[1]{\left[ #1 \right]}
\DeclarePairedDelimiter{\sbr}{[}{]}
\DeclarePairedDelimiter{\sdbr}{[\![}{]\!]}
\DeclarePairedDelimiter{\scbr}{\{}{\}}
%\newcommand{\scbr}[1]{\left\{ #1 \right\}}
\newcommand{\inp}[1]{\left\langle #1 \right\rangle}
% basic algebra symbols
\newcommand{\sumiK}{\sum_{i=1}^K}
\newcommand{\sumjK}{\sum_{j=1}^K}
\newcommand{\sumkK}{\sum_{k=1}^K}
\newcommand{\sumtT}{\sum_{t=1}^T}
\newcommand{\sumst}{\sum_{s=1}^t}
\newcommand{\sumtinfty}{\sum_{t=0}^\infty}
\newcommand{\sumsinfty}{\sum_{s=0}^\infty}
% \newcommand{\abs}[1]{\left| #1 \right|}
\newcommand*\diff{\mathop{}\!\mathrm{d}}
\usepackage{mathtools} % for \mathclap
%\newcommand{\given}{\mathrel{}\middle|\mathrel{}}
%\newcommand{\given}{\;\middle|\;}
\newcommand{\given}{\nonscript\mathpunct{}\middle|\nonscript\mathpunct{}}
\newcommand{\DKL}[2]{\mathcal{D}_{\mathsf{KL}}\spr*{#1,#2}}
\newcommand{\Dhel}[2]{\mathcal{D}_{\mathsf{H}}\spr*{#1,#2}}
\newcommand{\Dhels}[2]{\mathcal{D}_{\mathsf{H}}^2\spr*{#1,#2}}
% \newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{{\transpose}}{^\mathsf{\scriptscriptstyle T}}
% \newcommand{{\transpose}}{^{\intercal}}
\newcommand{\upplus}{\mathsf{\scriptscriptstyle +}} % superscript plus
\newcommand{\wh}[1]{\widehat{#1}}
\usepackage{mathtools} % for '\mathrlap' command (necessary)
\newcommand{\custombar}[3]{%
\mathrlap{\hspace{#2}\overline{\scalebox{#1}[1]{\phantom{\ensuremath{#3}}}}}\ensuremath{#3}
} % random example: \custombar{0.8}{.5pt}{y}
% blackboard bold letters
\newcommand{\bbC}{\mathbb{C}}
\newcommand{\bbE}{\mathbb{E}}
\newcommand{\bbF}{\mathbb{F}}
\newcommand{\bbI}{\mathbb{I}}
\newcommand{\bbN}{\mathbb{N}}
\newcommand{\bbP}{\mathbb{P}}
\newcommand{\bbQ}{\mathbb{Q}}
\newcommand{\bbR}{\mathbb{R}}
\newcommand{\bbRnnK}{\mathbb{R}_{\geq 0}^K}
\newcommand{\bbRpK}{\mathbb{R}_{>0}^K}
\newcommand{\bbV}{\mathbb{V}}
\newcommand{\bbZ}{\mathbb{Z}}
% bold letters
\newcommand{\bfone}{\mathbf{1}}
\newcommand{\bfe}{\mathbf{e}}
\newcommand{\bfu}{\mathbf{u}}
\newcommand{\bfv}{\mathbf{v}}
\newcommand{\bfw}{\mathbf{w}}
\newcommand{\bfx}{\mathbf{x}}
\newcommand{\bfy}{\mathbf{y}}
\newcommand{\bfyt}{\mathbf{y}_t}
% caligraphic letters
\newcommand{\cA}{\mathcal{A}}
\newcommand{\cB}{\mathcal{B}}
\newcommand{\cC}{\mathcal{C}}
\newcommand{\cD}{\mathcal{D}}
\newcommand{\cE}{\mathcal{E}}
\newcommand{\cF}{\mathcal{F}}
\newcommand{\cG}{\mathcal{G}}
\newcommand{\cH}{\mathcal{H}}
\newcommand{\cI}{\mathcal{I}}
\newcommand{\cJ}{\mathcal{J}}
\newcommand{\cK}{\mathcal{K}}
\newcommand{\cL}{\mathcal{L}}
\newcommand{\cM}{\mathcal{M}}
\newcommand{\cN}{\mathcal{N}}
\newcommand{\cO}{\mathcal{O}}
\newcommand{\cP}{\mathcal{P}}
\newcommand{\cQ}{\mathcal{Q}}
\newcommand{\cR}{\mathcal{R}}
\newcommand{\cS}{\mathcal{S}}
\newcommand{\cT}{\mathcal{T}}
\newcommand{\cU}{\mathcal{U}}
\newcommand{\cV}{\mathcal{V}}
\newcommand{\cX}{\mathcal{X}}
\newcommand{\cY}{\mathcal{Y}}
\newcommand{\cZ}{\mathcal{Z}}
% fraktur letters
\newcommand{\fkB}{\mathfrak{B}}
\newcommand{\fkR}{\mathfrak{R}}
% math operators
% \DeclareMathOperator*{\argmax}{arg\,max}
% \DeclareMathOperator*{\argmin}{arg\,min}
% \DeclareMathOperator{\opE}{E}
% \DeclareMathOperator{\opP}{P}
% \DeclareMathOperator{\ophP}{\widehat{P}}
% \DeclareMathOperator{\ophPt}{\widehat{P}_t}
% \DeclareMathOperator{\ophPk}{\widehat{P}_k}
% \DeclareMathOperator{\opPplus}{P_t^\mathsf{\scriptscriptstyle +}}
% \DeclareMathOperator{\opPplusk}{P_k^\mathsf{\scriptscriptstyle +}}
% \DeclareMathOperator{\ophPplus}{\widehat{P}_t^\mathsf{\scriptscriptstyle +}}
% \DeclareMathOperator{\ophPplusk}{\widehat{P}_k^\mathsf{\scriptscriptstyle +}}
\newcommand{\opE}{E}
\newcommand{\opP}{P}
\newcommand{\ophP}{\widehat{P}}
\newcommand{\ophPt}{\widehat{P}_t}
\newcommand{\ophPk}{\widehat{P}_k}
\newcommand{\opPplus}{P_t^\mathsf{\scriptscriptstyle +}}
\newcommand{\opPplusk}{P_k^\mathsf{\scriptscriptstyle +}}
\newcommand{\ophPplus}{\widehat{P}_t^\mathsf{\scriptscriptstyle +}}
\newcommand{\ophPplusk}{\widehat{P}_k^\mathsf{\scriptscriptstyle +}}
\DeclareMathOperator{\co}{co}
\DeclareMathOperator{\optrace}{trace}
\DeclareMathOperator{\Diag}{Diag} % diagonal matrices
\DeclareMathOperator{\diam}{diam} % diameter
\DeclareMathOperator{\dom}{dom} % domain
\DeclareMathOperator{\interior}{int}
\DeclareMathOperator{\Lip}{Lip} % lipschitz
\DeclareMathOperator{\scvx}{sc} % strong convexity
% \DeclareMathOperator{\regret}{\texttt{regret}}
\DeclareMathOperator{\regretplus}{\mathfrak{R}_T^\mathsf{\scriptscriptstyle +}}
\DeclareMathOperator{\regretKplus}{\mathfrak{R}_K^\mathsf{\scriptscriptstyle +}}
% \DeclareMathOperator{\sign}{sign}
% \DeclareMathOperator{\tr}{tr} % trace
\DeclareMathOperator{\CB}{CB}
\DeclareMathOperator{\softmax}{softmax}
% others
\newcommand{\simplex}{\Delta}
\newcommand{\piunif}{\pi_{\texttt{unif}}}
\newcommand{\QMAX}{Q_{\max}}
%\newcommand{\VMAX}{V_{\max}}
\newcommand{\VMAX}{Q_{\max}}
\newcommand{\LMAX}{L_{\max}}
% \newcommand{\WMAX}{W_{\max}}
\newcommand{\reset}{reset\xspace}
\DeclareMathOperator{\LSE}{LSE}
\newcommand{\piout}{\pi^{\mathrm{out}}}
\newcommand{\regretpi}{\mathfrak{R}^\pi_K}
\newcommand{\regretQ}{\mathfrak{R}^Q_K}
\newcommand{\hatgk}{\widehat{g}_k}
%\newcommand{\ThetaMAX}{\Theta_{\max}}
\newcommand{\ThetaMAX}{B_\theta}
\newcommand{\phiMAX}{B_\phi}
\newcommand{\Pilin}{\Pi_{\mathrm{lin}}}
\newcommand{\Qlin}{\cQ_{\mathrm{lin}}}
% algorithms
%\newcommand{\PDOIL}{\texttt{PDOIL}\xspace}
\newcommand{\SPOIL}{\texttt{SPOIL}\xspace}
\newcommand{\BC}{\texttt{BC}\xspace}
\newcommand{\IQLearn}{\texttt{IQ-Learn}\xspace}
\newcommand{\IQL}{\texttt{IQL}\xspace}
\newcommand{\PIL}{\texttt{P$^2$IL}\xspace}
\newcommand{\SoftDQN}{\texttt{Soft DQN}\xspace}
\newcommand{\cartpole}{\texttt{CartPole-v1}\xspace}
\newcommand{\acrobot}{\texttt{Acrobot-v1}\xspace}
\newcommand{\lunarlander}{\texttt{LunarLander-v2}\xspace}
\newcommand{\grg}[1]{\textcolor{red}{[\textbf{Grg:} #1]}}
\newcommand{\redd}[1]{\textcolor{red}{#1}}
\newcommand{\Loss}{\mathcal{L}}
\newcommand{\hLoss}{\wh{\mathcal{L}}}
\newcommand{\bp}{\overline{p}}
\newcommand{\N}{\mathcal{N}}
% \newcommand{\I}{\mathcal{I}}
\newcommand{\real}{\mathbb{R}}
\newcommand{\Dw}{\mathcal{D}}
%\newcommand{\Sw}{\mathcal{S}}
\newcommand{\Rw}{\mathcal{R}}
\newcommand{\HH}{\mathcal{H}}
\newcommand{\DD}{\mathcal{D}}
\newcommand{\OO}{\mathcal{O}}
\newcommand{\tOO}{\wt{\OO}}
\newcommand{\trace}[1]{\mbox{tr}\left(#1\right)}
\newcommand{\II}[1]{\mathbb{I}_{\left\{#1\right\}}}
\newcommand{\PP}[1]{\mathbb{P}\left[#1\right]}
\newcommand{\PPZ}[1]{\mathbb{P}_Z\left[#1\right]}
\newcommand{\EEZ}[1]{\mathbb{E}_Z\left[#1\right]}
\newcommand{\EE}[1]{\mathbb{E}\left[#1\right]}
\newcommand{\EEb}[1]{\mathbb{E}\bigl[#1\bigr]}
\newcommand{\EEtb}[1]{\mathbb{E}_t\bigl[#1\bigr]}
\newcommand{\EXP}{\mathbb{E}}
\newcommand{\EEs}[2]{\mathbb{E}_{#2}\left[#1\right]}
\newcommand{\EEu}[1]{\mathbb{E}_{\u}\left[#1\right]}
\newcommand{\EEst}[1]{\mathbb{E}_{*}\left[#1\right]}
\newcommand{\EEo}[1]{\mathbb{E}_{0}\left[#1\right]}
\newcommand{\PPpi}[1]{\mathbb{P}^\pi\left[#1\right]}
\newcommand{\EEpi}[1]{\mathbb{E}^\pi\left[#1\right]}
\newcommand{\EEtp}[1]{\mathbb{E}_t^+\left[#1\right]}
\newcommand{\EEtt}[1]{\mathbb{E}_{t-1}\left[#1\right]}
\newcommand{\PPi}[1]{\mathbb{P}_i\left[#1\right]}
\newcommand{\EEi}[1]{\mathbb{E}_i\left[#1\right]}
\newcommand{\PPc}[2]{\mathbb{P}\left[#1\left|#2\right.\right]}
\newcommand{\PPs}[2]{\mathbb{P}_{#2}\left[#1\right]}
\newcommand{\PPcs}[3]{\mathbb{P}_{#3}\left[#1\left|#2\right.\right]}
\newcommand{\PPct}[2]{\mathbb{P}_t\left[#1\left|#2\right.\right]}
\newcommand{\PPcc}[2]{\mathbb{P}\left[\left.#1\right|#2\right]}
\newcommand{\PPcct}[2]{\mathbb{P}_t\left[\left.#1\right|#2\right]}
\newcommand{\PPcci}[2]{\mathbb{P}_i\left[\left.#1\right|#2\right]}
\newcommand{\EEc}[2]{\mathbb{E}\left[#1\middle|#2\right]}
\newcommand{\EEcc}[2]{\mathbb{E}\left[\left.#1\right|#2\right]}
\newcommand{\EEcs}[3]{\mathbb{E}_{#3}\left[\left.#1\right|#2\right]}
\newcommand{\EEcct}[2]{\mathbb{E}_t\left[\left.#1\right|#2\right]}
\newcommand{\EEcctt}[2]{\mathbb{E}_{t-1}\left[\left.#1\right|#2\right]}
\newcommand{\EEcci}[2]{\mathbb{E}_i\left[\left.#1\right|#2\right]}
\renewcommand{\th}{\ensuremath{^{\mathrm{th}}}}
\def\argmin{\mathop{\mbox{ arg\,min}}}
\def\argmax{\mathop{\mbox{ arg\,max}}}
\newcommand{\ra}{\rightarrow}
\newcommand{\siprod}[2]{\langle#1,#2\rangle}
\newcommand{\iprod}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\biprod}[2]{\bigl\langle#1,#2\bigr\rangle}
\newcommand{\Biprod}[2]{\Bigl\langle#1,#2\Bigr\rangle}
\newcommand{\onenorm}[1]{\norm{#1}_1}
\newcommand{\twonorm}[1]{\norm{#1}_2}
\newcommand{\infnorm}[1]{\norm{#1}_\infty}
\newcommand{\opnorm}[1]{\norm{#1}_{\text{op}}}
\newcommand{\Hnorm}[1]{\norm{#1}_{\HH}}
\newcommand{\ev}[1]{\left\{#1\right\}}
\newcommand{\pa}[1]{\left(#1\right)}
\newcommand{\bpa}[1]{\bigl(#1\bigr)}
\newcommand{\Bpa}[1]{\Bigl(#1\Bigr)}
\newcommand{\BPA}[1]{\Biggl(#1\Biggr)}
\newcommand{\sign}{\mbox{sign}}
\newcommand{\loss}{\ell}
\newcommand{\hloss}{\wh{\loss}}
\newcommand{\bloss}{\overline{\loss}}
\newcommand{\lambdamin}{\lambda_{\min}}
\newcommand{\lambdamax}{\lambda_{\max}}
\newcommand{\hr}{\wh{r}}
\newcommand{\hR}{\wh{R}}
%\newcommand{\tX}{\wt{X}}
\newcommand{\tX}{X_0}
\newcommand{\htheta}{\wh{\theta}}
\newcommand{\hTheta}{\wh{\Theta}}
\newcommand{\ttheta}{\wt{\theta}}
\newcommand{\hf}{\wh{f}}
\newcommand{\hs}{\wh{s}}
\newcommand{\bA}{\overline{A}}
\newcommand{\bW}{\overline{\W}}
\newcommand{\deff}{d_{\text{eff}}}
\newcommand{\Sp}{\Sigma^+}
\newcommand{\Spt}{\Sigma^{+2}}
\newcommand{\regret}{\mathrm{regret}}
%\newcommand{\qed}{\hfill\BlackBox\\[2mm]}
\newcommand{\hL}{\wh{L}}
\newcommand{\expexpexp}{\textsc{Exp3}\xspace}
\newcommand{\linexp}{\textsc{LinExp3}\xspace}
\newcommand{\linexpreal}{\textsc{RealLinExp3}\xspace}
\newcommand{\linexprobust}{\textsc{RobustLinExp3}\xspace}
\newcommand{\linucb}{\textsc{LinUCB}\xspace}
\renewcommand{\phi}{\varphi}
\section{Introduction}
In imitation learning (IL), a learner observes a finite dataset of state-action pairs generated by an expert policy interacting with an environment modeled as a Markov Decision Process (MDP; \citet{Puterman:1994}). The learner's objective is to find a policy that performs nearly as well as the expert policy with respect to an unknown ground-truth reward function. This work focuses on \emph{offline imitation learning}, where the learner cannot collect new state-action sequences from the MDP used for generating the expert's data. In this context, we propose new algorithms that operate under a previously under-explored set of structural assumptions on the learning environment.
Recent years have seen a quite significant surge of interest in the problem of imitation learning, not unlikely due to its relevance to next-token prediction in generative language models \citep{rajaraman2020toward,foster2024behavior,rohatgi2025computational}. A common feature of these recent works is that they all make the assumption that the expert data has been generated by a fixed policy that belongs to a known, finite class of policies and they return policies within the same class. Such an assumption is often referred to as \emph{expert realizability} and can be formally stated as follows.
%
\begin{assumption*}[Expert realizability]
The learner has access to a function class $\PiE$ that contains the unknown expert policy $\expert$, that is, such that $\expert \in \PiE$.
\end{assumption*}
%
Several clean and elegant results were proved under this assumption, in particular showing the existence of conceptually simple algorithms achieving tight upper bounds on the sample complexity of finding good solutions, and lower bounds demonstrating the near-optimality of these algorithms under said assumptions. These bounds typically depend on a measure of complexity of the policy class (as measured by, say, its covering number). However, further scrutiny reveals that these assumptions may not always be verified or even reasonable: in many cases of significant practical interest, there is no reason to believe that the expert policy may be easily modeled within a simple and tractable policy class. For instance, in the popular use case of learning from human feedback, it is arguably quite unlikely that data would be generated in a consistent, systematically predictable way that can be modeled as a simple policy mapping states to actions. Indeed, human behavior can be nonstationary, irrational, or even be influenced by unobserved confounders not captured by the state representation. We address these limitations by exploring an alternative framework for imitation learning, which reasons about the structure of the \emph{value functions} of the policies used by the \emph{learning algorithm} itself, as opposed to making assumptions about the structure of the policy followed by the expert.
Furthermore, the sample complexity guarantess in \citet{rajaraman2020toward,foster2024behavior,rohatgi2025computational} scale with $\log \abs{\PiE}$ (assuming $\PiE$ is finite), meaning large policy classes, potentially necessary to realize the expert, lead to deteriorated guarantees. Additionally, the consequences of misspecification, \ie $\expert \notin \PiE$, are often severe. For instance, \citet{rohatgi2025computational} demonstrated that if the policy class $\PiE$ is misspecified, then it is computationally intractable to learn $\argmin_{\pi \in \PiE} \Dhels{\bbP^\pi}{\bbP^\expert}$, the best in-class policy under the Hellinger distance, in an offline manner. However, this theoretical intractability under misspecification seems at odds with practical scenarios, such as training large language models via next-token prediction (a form of offline IL), which perform well despite the expert policy (derived from human-written text) likely not belonging to any reasonable policy class $\PiE$.
To address this apparent discrepancy, we initiate the study of offline imitation learning by leveraging structural assumptions about the MDP rather than relying on expert realizability. For example, in language tasks, structural assumptions might involve deterministic, tree-shaped MDPs. In robotics, one might assume that next states are determined by compact feature representations of current state-action pairs. More generally, we consider MDPs where the action-value function of any policy can be written as a linear combination of features known to the learner. Such MDPs are referred to as linear $Q^\pi$-realizable MDPs, a class that has been central to recent works in reinforcement learning theory \citep{weisz2023online,mhammedi2024sample,tkachuk2024trajectory}. Our primary contribution is to show that, for this class of MDPs, it is possible to develop algorithms that guarantee to output a policy performing arbitrarily close to the expert policy \emph{without imposing expert realizability}.
The algorithm is based on a simple primal-dual formulation of the problem of imitation learning, which characterizes the solution as the saddle-point of a convex-concave objective function. The primal variables correspond to policies in the MDP and the dual variables to Q-functions, which motivates a very simple saddle-point optimization algorithm for imitation learning: in a sequence of rounds, the primal player (the \emph{actor}) picks a policy and the dual player (the \emph{critic}) picks a Q-function, respectively trying to minimize and maximize the objective. We accordingly call the method \SPOIL, standing for Saddle-Point Offline Imitation Learning. In the case of linear function approximation, both update steps of \SPOIL can be performed very efficiently (in time linear in the feature dimension). For general function approximation, the Q-function updates can be performed by solving a simple linear optimization problem, which is straightforward to solve in practical scenarios. When instantiated with neural networks, empirical experiments show its performance is competitive with (and in some cases superior to, \eg, behavior cloning) state-of-the-art offline imitation learning algorithms. Interestingly, our algorithm shares a good degree of similarity with the state-of-the-art method of \citet{Garg:2021} called \IQLearn, which is also derived from a primal-dual perspective. We discuss these similarities in depth and argue that \SPOIL provides a superior solution to the \IQLearn objective (at least inasmuch as it is more amenable to theoretical analysis).
To the best of our knowledge, this is the first result showing that leveraging structural assumptions of the underlying MDP can guarantee matching the expert performance as the number of expert transitions goes to infinity without imposing any form of expert realizability assumption. For clarity, we compare our contribution with existing results in Table~\ref{tab:related}. We denoted $\cN_\epsilon \spr*{\cQ}$ the covering number of the function class $\cQ$ (see Theorem~\ref{thm:mainnonlinear}), and $\tauE$ the number of trajectories needed to make the difference in total expected return between the expert and the output policy smaller than $\varepsilon$.
\renewcommand{\arraystretch}{1.8}
\newcommand{\verticaloffset}{0.8ex}
\newcommand{\spacingfirstcol}{-1.75ex}
\begin{table}[t]
\centering
\vspace{.4em}
\caption{\label{tab:related} Comparison with related algorithms. %The column labelled Expert Traj. $(\tauE)$ denotes the required number of expert trajectories to find a policy $\pi$ such that $(1-\gamma)^{-1}\bbE{\rho^\expert - \rho^\pi}\leq \varepsilon $.
We denoted the class of deterministic linear experts as $\Pi^{\scriptscriptstyle \textup{E}}_{\mathrm{det, lin}} = \scbr*{\pi: \exists \theta \in \fkB \spr*{\ThetaMAX}, \pi \spr*{\cdot} = \argmax_{a \in \cA} \innerprod{\theta}{\varphi \spr*{\cdot, a}}}$, and an arbitrary policy class as $\PiE$. We also define $W = \max_{\pi \in \PiE, \spr*{x, a} \in \cX \times \cA, h \in \sbr*{H}} \frac{\pi_{{\scriptscriptstyle \textup{E}}, h} \spr*{a \given x}}{\pi_h \spr*{a \given x}}$ and $\epsilon_{\mathrm{miss}} = \min_{\pi \in \PiE} \Dhels{\bbP^\pi}{\bbP^\expert}$.}
\vspace{.4em}
%\setlength{\tabcolsep}{4pt} % Reduce horizontal padding
%\renewcommand{\arraystretch}{1.5} % Increase vertical spacing
\resizebox{\textwidth}{!}{%
\begin{tabular}{C{3.7cm} C{3.8cm} C{3.9cm} C{2.3cm} C{1.9cm} C{3.2cm}}
\toprule
\textbf{Algorithm} & \textbf{Structural assumptions} & \textbf{Avoids expert realizability} & \textbf{Infinite horizon} & \textbf{Expert class} & \textbf{Expert Traj. $(\tauE)$} \\ \midrule
\texttt{BC} with log loss & \multirow{2}{*}[\verticaloffset]{None} & \multirow{2}{*}[\verticaloffset]{\blackcross} & \multirow{2}{*}[\verticaloffset]{\blackcross} & \multirow{2}{*}[\verticaloffset]{$\PiE$} & \multirow{2}{*}[\verticaloffset]{\large $\cO \spr*{\frac{H^2 \log \abs{\PiE}}{\varepsilon^2}}$} \\[\spacingfirstcol]
\citep{foster2024behavior} & & & & & \\ \midrule
\texttt{BC} with 0\,-1 loss & \multirow{2}{*}[\verticaloffset]{None} & \multirow{2}{*}[\verticaloffset]{\blackcross} & \multirow{2}{*}[\verticaloffset]{\blackcross} & \multirow{2}{*}[\verticaloffset]{$\Pi^{\scriptscriptstyle \textup{E}}_{\mathrm{det, lin}}$} & \multirow{2}{*}[\verticaloffset]{\large $\widetilde{\cO} \spr*{\frac{H^2 d}{\varepsilon}}$} \\[\spacingfirstcol]
\citep{rajaraman2021value} & & & & & \\ \midrule
\texttt{BoostedLogLossBC} & \multirow{2}{*}[\verticaloffset]{None} & \checkmark\xspace with a misspecification & \multirow{2}{*}[\verticaloffset]{\blackcross} & \multirow{2}{*}[\verticaloffset]{$\PiE$} & \multirow{2}{*}[\verticaloffset]{\large $\cO \spr*{\frac{H^2 \log \abs{\PiE}}{\varepsilon^2}}$} \\[\spacingfirstcol]
\citep{rohatgi2025computational} & & error of $\widetilde{\cO} \spr*{H \log \spr*{W} \epsilon_{\mathrm{miss}}}$ & & & \\ \midrule
\texttt{Projection} & Linear reward & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{$-$} & \multirow{2}{*}[\verticaloffset]{\large $\widetilde{\cO} \spr*{\frac{d}{\spr*{1 - \gamma}^2 \varepsilon^2}}$} \\[\spacingfirstcol]
\citep{Abbeel:2004} & Known transitions & & & & \\ \midrule
\texttt{MWAL} & Linear reward & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{$-$} & \multirow{2}{*}[\verticaloffset]{\large $\widetilde{\cO} \spr*{\frac{\log \spr*{d}}{\spr*{1 - \gamma}^2 \varepsilon^2}}$} \\[\spacingfirstcol]
\citep{Syed:2007} & Known transitions & & & & \\ \midrule
\textbf{\SPOIL} & \multirow{2}{*}[\verticaloffset]{Linear $Q^\pi$-realizability} & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{$-$} & \multirow{2}{*}[\verticaloffset]{\large $\widetilde{\cO} \spr*{\frac{d}{\spr*{1 - \gamma}^4 \varepsilon^2}}$} \\[\spacingfirstcol]
\textbf{(Theorem~\ref{thm:main_formal})} & & & & & \\ \midrule
\textbf{\SPOIL} & \multirow{2}{*}[\verticaloffset]{$Q^\pi$-realizability} & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{\checkmark} & \multirow{2}{*}[\verticaloffset]{$-$} & \multirow{2}{*}[\verticaloffset]{\large $\widetilde{\cO} \spr*{\frac{\log \cN_\varepsilon \spr*{\cQ}}{\spr*{1 - \gamma}^8 \varepsilon^4}}$} \\[\spacingfirstcol]
\textbf{(Theorem~\ref{thm:mainnonlinear})} & & & & & \\ \bottomrule
\end{tabular}}
\end{table}
\renewcommand{\arraystretch}{1.0}
\vspace{-5pt}
\paragraph{Notation.} We use $\Delta \spr*{\cZ}$ to denote the simplex over the countable set $\cZ$. Given two probability distributions $p, q \in \Delta \spr*{\cZ}$, we denote the Kullback-Leibler divergence as $\DKL{p}{q} = \sum_{z \in \cZ} p \spr*{z} \log \frac{p \spr*{z}}{q \spr*{z}}$. We denote $\inp{\cdot, \cdot}$ the inner product between two finite-dimensional vectors, and $\norm{\cdot}$ the Euclidean norm. We denote $\cU \spr*{\sbr*{K}}$ the uniform distribution over the set $\sbr*{K} = \ev{1, \dots, K}$. The Euclidean ball of radius $R > 0$ centered at the origin is denoted as $\fkB \spr*{R}$.
\section{Preliminaries}
We begin by introducing the problem of offline imitation learning in discounted MDPs together with the assumptions we will consider throughout the paper.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Markov decision processes.} We formalize the learning problem in a discounted MDP $\cM = \spr*{\cX, \cA, r, P, \gamma, \initial}$, where $\cX$ is the state space which we assume finite but too large to be enumerated, $\cA$ is a finite action space with $A$ actions, $r: \cX \times \cA \rightarrow \sbr*{0, 1}$ is the unknown reward function, $P: \cX \times \cA \rightarrow \Delta \spr*{\cX}$ is the unknown transition kernel, $\gamma \in [0, 1)$ is the discount factor, and $\initial \in \Delta \spr*{\cX}$ is the initial state distribution. For any state-action-state triplet $\spr*{x, a, x'}$, $P \spr*{x' \given x, a}$ denotes the probability of landing in state $x'$ after taking action $a$ in state $x$. A \emph{stationary policy} (or simply \emph{policy}) $\pi: \cX \rightarrow \Delta \spr*{\cA}$ is a mapping from states to distributions over actions. The interaction of a policy $\pi$ with the environment $\cM$ unfolds as follows: an initial state $X_0 \sim \nu_0$ is drawn, and for each subsequent time step $h \geq 0$, an action $A_h \sim \pi \spr*{\cdot \given X_h}$ is taken, a reward $r \spr*{X_h, A_h}$ is received, and the agent transitions to a new state $X_{h+1} \sim P \spr*{\cdot \given X_h, A_h}$. We denote $\bbP^\pi$ the resulting probability distribution over trajectories, and $\bbE^\pi$ the corresponding expectation operator. For any state $x \in \cX$, we define the state value function of the policy $\pi$ as $V^\pi \spr*{x} = \bbE^\pi \sbr*{\sum^{\infty}_{h=0} \gamma^h r \spr*{X_h, A_h} \given X_0 = x}$. Analogously, we define the state-action value function as $Q^\pi \spr*{x, a} = \bbE^\pi \sbr*{\sum^{\infty}_{h=0}\gamma^h r \spr*{X_h, A_h} \given X_0 = x, A_0 = a}$. The value functions are tied together via the \emph{Bellman equations}
%
\begin{equation*}
V^\pi \spr*{x} = \sum_{a \in \cA} \pi \spr*{a \given x} Q^\pi \spr*{x, a}\,, \quad \text{and} \quad Q^\pi \spr*{x, a} = r \spr*{x, a} + \gamma \sum_{x' \in \cX} P \spr*{x' \given x, a} V^\pi \spr*{x'}\,.
\end{equation*}
%
Additionally, we will sometimes use the notation $Q \spr*{x, \pi}$ to denote $\sum_a \pi \spr*{a \given x} Q \spr*{x, a}$ for any policy $\pi$ and any function $Q\colon \cX \times \cA \to \bbR$. Note that this notation allows us to write $V^\pi \spr*{x} = Q^\pi \spr*{x, \pi}$. Any policy $\pi$ induces an \emph{occupancy measure} $\mu^\pi \in \Delta \spr*{\cX \times \cA}$ over state-action pairs, defined as the discounted total expected times that each state-action pair is visited by policy $\pi$. The same quantity defined for states is called the state-occupancy measure and is denoted as $\nu^\pi \in \Delta \spr*{\cX}$. For any state-action pair $\spr*{x, a} \in \cX \times \cA$, they are respectively defined as
%
\begin{equation*}
\nu^\pi \spr*{x} = \spr*{1 - \gamma} \sum^\infty_{h=0} \gamma^h \bbP^\pi \sbr*{X_h = x}\,, \quad \text{and} \quad
\mu^\pi \spr*{x, a} = \spr*{1 - \gamma} \sum^\infty_{h=0} \gamma^h \bbP^\pi \sbr*{X_h = x, A_h = a}\,,
\end{equation*}
%
and they are related to each other by the \emph{flow conditions} (sometimes called ``Bellman flow conditions'')
%
\begin{equation} \label{eq:flow}
\nu^\pi \spr*{x} = \gamma \sum_{x', a'} P \spr*{x \given x',a'} \mu^\pi \spr*{x', a'} + \spr*{1 - \gamma} \nu_0 \spr*{x}\,.
\end{equation}
%
Notably, these definitions and the flow conditions remain valid for general history-dependent policies $\pi$ that may take the entire history of state-action pairs $\spr*{X_1, A_1, \dots, X_h}$ into account when selecting each action $A_h$. Finally, we let $\rho^\pi = (1-\gamma)\bbE^\pi \sbr*{\sum_{h=0}^\infty \gamma^h r \spr*{X_h, A_h}}$ stand for the normalized expected return of a (potentially nonstationary) policy $\pi$. The following useful result, commonly called the \emph{performance-difference lemma} (\citealt{kakade2002approximately}, see also Eq.~7.14 in \citealt{howard1960dynamic}), gives a useful expression for the performance gap between two policies.
%
\begin{restatable}{Lem}{pdl} \label{lem:performance-difference}
Let $\pi$ be a stationary policy and $\pi'$ be any policy. Then,
%
\begin{equation*}
\rho^{\pi'} - \rho^\pi = \bbE_{\spr*{X, A} \sim \mu^{\pi'}} \sbr*{Q^\pi \spr*{X, A} - V^\pi \spr*{X}}\,.
\end{equation*}
\end{restatable}
%
Note that this lemma is generally stated for stationary policies, but we will find it useful later to use it with general history-dependent policies. We provide the straightforward proof in Appendix~\ref{app:proof_pdl}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Imitation Learning.} We consider the problem of offline imitation learning. Given a dataset $\cD^\expert = \scbr*{\XEi, \AEi}^{\tauE}_{i=1}$ of state-action pairs sampled from an expert policy's occupancy measure $\mu^\expert$, our objective is to design an algorithm, $\mathrm{Alg}$, that produces a policy $\piout$ satisfying
%
\begin{equation} \label{eq:goal}
\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} \leq \varepsilon\,.
\end{equation}
%
The algorithm is not allowed any further interaction with the expert policy or the MDP $\cM$ and only has to work with the record of state-action pairs contained in the data set. As stated in the introduction, we aim to achieve this \emph{without imposing expert realizability}. Instead, we consider the following structural assumption on the environment.
%
\begin{assumption}[Linear $Q^\pi$-realizability] \label{ass:linQ}
The action value function of any policy $\pi$ can be written as a linear combination of known features. That is, there exists a known mapping $\varphi: \cX \times \cA \rightarrow \bbR^d$ such that for any policy $\pi$, there exists a vector $\theta^\pi \in \bbR^d$ such that for any state-action pair $\spr*{x, a}$, $Q^\pi \spr*{x, a} = \inp{\varphi \spr*{x, a}, \theta^\pi}$. Moreover, we assume $\norm{\theta^\pi} \leq \ThetaMAX$ for all $\pi$, and $\sup_{x, a} \norm{\varphi \spr*{x, a}} \leq \phiMAX$.
\end{assumption}
%
We will also consider the general function approximation setting, where the action value function of any policy $\pi$ can be represented by some function class $\cQ \subset \bbR^{\cX \times \cA}$.
%
\begin{assumption}[$Q^\pi$-realizability] \label{asp:Qpi-realizable}
An MDP is said $Q^\pi$-realizable if there exists a function class $\cQ \subset \bbR^{\cX \times \cA}$ such that for any policy $\pi$, it holds that $Q^\pi \in \cQ$, and for any $Q \in \cQ$, $\norm{Q}_\infty \leq \frac{1}{1 - \gamma}$.
\end{assumption}
%
For this assumption to make sense, we typically require the function class $\cQ$ to have bounded capacity. We quantify this via covering numbers, defined as follows.
%
\begin{definition}[Covering number]
Let $\spr*{M, d}$ be a metric space, $K$ be a subset of $M$, and $\epsilon > 0$. A set $\cC_\epsilon \spr*{K, d}$ is an $\epsilon$-covering of $K$ if for any $x \in K$, there exists $y \in \cC_\epsilon \spr*{K, d}$ such that $d \spr*{x, y} \leq \epsilon$. The covering number of $K$, $\cN_\epsilon \spr*{K, d}$, is the minimum cardinality of any such covering of $K$.
\end{definition}
\section{Primal-dual offline imitation learning}
In order to introduce our main algorithmic idea, we define the following objective function:
%
\begin{equation*}
\Loss \spr*{\pi; Q} = \bbE_{\spr*{X, A} \sim \mu^{\expert}} \sbr*{Q \spr*{X, A} - Q \spr*{X, \pi}}\,,
\end{equation*}
%
where we denoted $Q \spr*{X, \pi} = \bbE_{A' \sim \pi \spr*{\cdot \given X}} \sbr*{Q \spr*{X, A'}}$. Our main observation is that the main objective function we consider can be rewritten in terms of this function as follows:
%
\begin{equation*}
\rho^{\expert} - \rho^\pi = \Loss \spr*{\pi; Q^\pi} \le \textstyle\sup_{Q \in \cQ} \Loss \spr*{\pi; Q}.
\end{equation*}
%
This suggests a good policy $\piout$ may be found by solving the \emph{saddle-point} optimization problem $\min_\pi \sup_{Q \in \cQ} \Loss \spr*{\pi; Q}$. Indeed, if one is able to produce a policy $\piout$ satisfying $\sup_{Q \in \cQ} \Loss \spr*{\piout; Q} \le \varepsilon$, then the above inequality implies that the suboptimality of $\piout$ as compared to $\expert$ will also be at most $\varepsilon$.
Inspired by this observation, we set out to design an incremental \emph{primal-dual} optimization algorithm to approximate the saddle point of the function $\Loss$. In each iteration $k = 1, 2, \dots, K$, the algorithm performs two updates: a primal update that corresponds to policy updates aiming to minimize $\Loss$, and a dual update that computes action-value function estimates and aims to maximize $\Loss$. Following a common terminology in reinforcement learning, we will sometimes refer to the primal updates as \emph{actor} updates and the dual updates as \emph{critic} updates.
In order to turn these insights into a practical algorithm, we define the following empirical estimate of the objective function $\Loss$:
%
\begin{equation*}
\hLoss \spr*{\pi; Q} = \frac{1}{\tauE} \sum^{\tauE}_{i=1} \spr*{Q \spr*{\XEi, \AEi} - Q \spr*{\XEi, \pi}}\,.
\end{equation*}
%
For a fixed $Q$ and $\pi$, this is clearly an unbiased estimator of $\Loss$. In line with the derivations above, we choose our critic and actor updates respectively as
%
\begin{equation*}
Q_k \in \argmax_{Q \in \cQ} \hLoss \spr*{\pi_k; Q}, \quad \text{and} \quad\;
\pi_{k+1} \spr*{a \given x} = \frac{\pi_k \spr*{a \given x} e^{\eta Q_k \spr*{x, a}}}{\sum_{a' \in \cA} \pi_k \spr*{a' \given x} e^{\eta Q_k \spr*{x, a'}}},
\end{equation*}
%
where $\eta > 0$ is a \emph{learning-rate} (or \emph{stepsize}) parameter that modulates the strength of the policy updates. After performing $K$ updates, the algorithm chooses a random index $I$ uniformly on the integers in $\sdbr*{1, K}$, and returns $\piout = \pi_I$. We refer to this algorithm as Saddle-Point Offline Imitation Learning (\SPOIL). This algorithm design is justified by the following simple error decomposition that lies at the heart of our main results.
%
\begin{proposition} \label{prop:decomp}
Let $\Delta \spr*{\pi} = \bbE \sbr*{\sup_{Q \in \cQ} \abs{\Loss \spr*{\pi; Q} - \hLoss \spr*{\pi; Q}}}$. The output of \SPOIL satisfies
%
\begin{equation*}
\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} \le \frac1K \sumkK \bbE \sbr*{\Loss \spr*{\pi_k; Q_k}} + \frac 2K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}}\,.
\end{equation*}
\end{proposition}
%
\begin{proof}
The proof simply follows by noticing
%
\begin{align*}
&\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} = \frac1K \sumkK \bbE \sbr*{\Loss \spr*{\pi_k; Q^{\pi_k}}} \le \frac1K \sumkK \bbE \sbr*{\hLoss \spr*{\pi_k; Q^{\pi_k}}} + \frac1K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}} \\
&\qquad\le \frac1K \sumkK \bbE \sbr*{\hLoss \spr*{\pi_k; Q_k}} + \frac1K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}} \le \frac1K \sumkK \bbE \sbr*{\Loss \spr*{\pi_k; Q_k}} + \frac2K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}}\,,
\end{align*}
%
where we have used the definitions of $\Delta$ and $Q_k$ in the first and second lines, respectively.
\end{proof}
%
The first term in this decomposition corresponds to the \emph{regret} of the policy player $\pi$ against the comparator strategy $\expert$ and can be controlled with probability $1$ via standard tools of online learning (as found in the excellent books of \citealt{Cesa-Bianchi:2006} and \citealt{orabona2023modern}). The second term measures the estimation error of the objective function $\Loss$ uniformly over the space of action-value functions $\cQ$ and along the policies played by the algorithm, and can be controlled via standard concentration arguments. Altogether, the proposition suggests that \SPOIL will return a good policy if these estimation errors can be bounded reasonably---a fact we will formally show in the next section.
Before stating our performance guarantees for the concrete settings we consider in this paper, we pause to point out a peculiar connection between the algorithm described above and the inverse Q-learning (\IQLearn) algorithm of \citet{Garg:2021}. While motivated using completely different arguments, the saddle-point objective function optimized by \IQLearn is nearly identical to our function $\Loss$: after removing entropy-regularization and setting their reward regularizer $\psi$ to zero, one can verify using the flow constraint (Eq.~\ref{eq:flow}) that their function $\cJ$ is \emph{identical} to our $\cL$. Ultimately, \citet{Garg:2021} draw different conclusions from this saddle-point formulation, and propose to solve it by computing $\pi_Q = \argmin_\pi \cJ \spr*{Q}$ and optimize the \emph{dual function} $g \spr*{Q} = \min_\pi \cL \spr*{\pi; Q}$. This function, however, can be highly nonsmooth and difficult to optimize, which is why \IQLearn needs to heavily rely on regularization both in $\pi$ and $Q$. In contrast, our algorithm can be seen as trying to optimize the \emph{primal function} $f \spr*{\pi} = \max_Q \cL \spr*{\pi; Q}$ in terms of the policy $\pi$, which can be done in a stable way by incremental policy updates. Additionally, as Proposition~\ref{prop:decomp} clearly reveals, optimizing the primal objective allows us to directly reason about the performance of the output policy. In contrast, we do not see a clear way to do this for the dual objective optimized by \IQLearn.
Furthermore, we also note that \SPOIL shares similarities with the algorithm \texttt{AdVIL} proposed by \citet{swamy2021moments}. Specifically, both \SPOIL and \texttt{AdVIL} consider the same objective $\Loss$ but the two methods differ in their proposed algorithmic solutions and analytical approaches. Notably, \citet{swamy2021moments} employed simultaneous gradient descent-ascent updates that made little use of the specific problem structure, whereas we consider an asymmetric scheme where the policy player uses mirror descent and the $Q$-player plays the best response. Therefore, our approach is more akin to minimizing the function $\pi \mapsto \max_{Q \in \cQ} \Loss \spr*{\pi, Q}$ rather than using a primal-dual scheme. This is an important difference since Proposition~\ref{prop:decomp} makes evident that the best response update of the $Q$ player is crucial for our analysis.
In what follows, we instantiate \SPOIL in two settings of particular interest, depending on the Q-function class being used. We first provide a set of results for linear function approximation (where the algorithm is very easy to implement and analyze) and for general function classes (where implementation and analysis are both less straightforward).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\SPOIL for linear function approximation}
We first provide a set of guarantees under the assumption that the function class is linear in some known features that realize the action-value functions of all policies $\pi$ as linear combinations (see Assumption~\ref{ass:linQ}). In this setting, the actor and critic updates both simplify. For the actor, notice that the policy update can be rewritten as $\pi_k \spr*{a \given x} \propto e^{\eta \sum_{i=1}^{k-1} Q_i \spr*{x, a}}$, which only requires storing $\sum_{i=1}^{k-1} Q_i$ in memory. For linear function approximation, this means that it suffices to maintain a single $d$-dimensional vector $\bar{\theta}_k = \sum_{i=1}^k \theta_i$ in memory and update it incrementally after each critic update. As for the critic update itself, notice that the objective function $\Loss$
and its empirical counterpart $\hLoss$ can be rewritten in terms of the gap between the feature-expectation vectors
%
\begin{equation*}
g_k = \bbE_{\spr*{X, A} \sim \mu^\expert} \sbr*{\varphi \spr*{X, A} - \varphi \spr*{X, \pi_k}}, \quad\mbox{and}\quad\;
\hatgk = \frac{1}{\tauE} \sum_{i=1}^{\tauE} \spr*{\varphi \spr*{\XEi, \AEi} - \varphi \spr*{\XEi,\pi_k}}\,.
\end{equation*}
%
When considering linear functions $Q_\theta\colon \spr*{x, a} \mapsto \inp{\varphi \spr*{x, a}, \theta}$, the objective can be written as
%
\begin{equation*}
\Loss \spr*{\pi_k; Q_\theta} = \inp{\theta, g_k}, \quad\mbox{and}\quad\;
\hLoss \spr*{\pi_k; Q_\theta} = \inp{\theta, \hatgk}\,,
\end{equation*}
%
and the critic update can be simply written as $\theta_k = \argmax_{\theta \in \fkB \spr*{\ThetaMAX}} \inp{\theta, \hatgk}$, which is trivial to compute. All in all, both actor and critic updates can be performed efficiently while only working in a $d$-dimensional Euclidean space. The following theorem provides our main result for \SPOIL.
\begin{restatable}{theorem}{linQresult} \label{thm:main_formal}
Let Assumption~\ref{ass:linQ} hold. Run Algorithm~\ref{alg:main} for $K = \frac{2 \log A}{\spr*{1 - \gamma}^2\varepsilon^2}$ iterations, with a learning rate $\eta = \spr*{1 - \gamma} \sqrt{2 \log A / K} $, and $\tauE = \cO \spr*{\frac{d}{\spr*{1 - \gamma}^2 \varepsilon^2} \log \spr*{\frac{\ThetaMAX \phiMAX A}{\spr*{1 - \gamma} \varepsilon}}}$ samples collected by any expert policy $\expert$. Then, the output satisfies $\bbE \bigl[\rho^\expert - \rho^{\pi^{\mathrm{out}}}\bigr] \leq 5 \varepsilon$.
\end{restatable}
%
The proof is in Appendix~\ref{app:proof_main_formal}. It is important to highlight that no assumptions are made concerning the expert policy. In particular, we do not require knowledge of a class $\PiE$ realizing the expert policy and as a consequence the bound on $\tauE$ does not scale at all with a complexity measure of $\PiE$. This is in stark contrast with the theoretical guarantees for behavioural cloning (\eg, \citealp[Chapter 15]{agarwal2019reinforcement}, and \citealp{foster2024behavior}) which show bounds on the expert samples scaling with $\log \abs{\PiE}$ (or the log covering number for continuous classes). It follows that no matter how complex the expert policy is, \SPOIL suffers only the complexity of the environment (\ie, the feature dimensionality $d$). Before moving to the next section, we emphasize that for consistency with the literature, Table~\ref{tab:related} reports the number of expert trajectories required to guarantee that the difference between unnormalized returns, $\spr*{1 - \gamma}^{-1}\bbE \bigl[\rho^\expert - \rho^{\piout}\bigr]$, is bounded by $\cO \spr*{\varepsilon}$.
%
\begin{figure}[!t]
\begin{minipage}[t]{0.49\textwidth}
\begin{algorithm}[H]
\caption{\SPOIL with linear FA} \label{alg:main}
\textbf{Input: } Number of expert trajectories $\tauE$, learning rate $\eta$, number of iterations $K$. \\
\textbf{Initialize: } $\theta_0 = 0$, uniform policy $\pi_0$. \\
\textbf{For $k = 1, 2, \dots, K$:}
\begin{enumerate}[leftmargin=13pt, parsep=3pt, itemsep=1pt]
\item $\pi_k \spr*{a \given x} \propto \pi_{k-1} \spr*{a \given x} e^{\eta \inp{\varphi\spr*{x, a}, \theta_{k-1}} }$.
\item $\hatgk = \tauE^{-1} \sum^{\tauE}_{i=1} \spr*{\varphi \spr*{\XEi, \AEi} - \varphi \spr*{\XEi, \pi_k}}$.
\item $\theta_k = \displaystyle\argmax_{\theta: \norm{\theta} \leq \ThetaMAX} \inp{\theta, \hatgk} = \frac{\ThetaMAX}{\norm{\hatgk}} \hatgk$.
\end{enumerate}
\textbf{Output:} $\piout = \pi_I$, where $I \sim \cU \spr*{\sbr*{K}}$.
\end{algorithm}
\end{minipage}\hfill
\begin{minipage}[t]{0.49\textwidth}
\begin{algorithm}[H]
\caption{\SPOIL with general FA}\label{alg:spoil-general-fa}
\textbf{Input: } Number of expert trajectories $\tauE$, learning rate $\eta$, number of iterations $K$. \\
\textbf{Initialize: } $Q_0 = 0$, uniform policy $\pi_0$. \\
\textbf{For $k = 1, 2, \dots, K$:}
\begin{enumerate}[leftmargin=13pt, parsep=3pt, itemsep=1pt]
\item $\pi_k \spr*{a \given x} \propto \pi_{k-1} \spr*{a \given x} e^{\eta Q_{k-1} \spr*{x, a}}$.
\item $Q_k \in \displaystyle\argmax_{Q \in \cQ} \wh{\cL} \spr*{\pi_k, Q}$.
\end{enumerate}
\textbf{Output:} $\piout = \pi_I$, where $I \sim \cU \spr*{\sbr*{K}}$.
\end{algorithm}
\end{minipage}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{\SPOIL for general function approximation}
For more complex $Q^\pi$-realizable MDPs, we analyze the version of \SPOIL given in Algorithm~\ref{alg:spoil-general-fa}. Notice that the updates can no longer use the linear structure of the value functions, and thus the critic update cannot be computed in closed form. Nevertheless, the algorithm remains well-defined, and satisfies the following performance guarantee.
%
\begin{restatable}{theorem}{mainnonlinear} \label{thm:mainnonlinear}
Let Assumption~\ref{asp:Qpi-realizable} hold. Run Algorithm~\ref{alg:spoil-general-fa} for $K = \frac{2 \log A}{\spr*{1 - \gamma}^2 \varepsilon^2}$ iterations, with a learning rate $\eta = \spr*{1 - \gamma} \sqrt{2 \log A / K} $ and $\tauE = \cO \Bigl(\frac{\log A}{\spr*{1 - \gamma}^4 \varepsilon^4} \log \Bigl(\frac{\cN_{\epsilon'} \spr*{\cQ, \norm{\cdot}_\infty}}{\varepsilon \spr*{1 - \gamma}}\Bigr)\Bigr)$ samples collected by any expert $\expert$, where $\epsilon' = \spr*{\sqrt{128} K^{3/2} A \log A}^{-1}$. Then, the output satisfies $\bbE \bigl[\rho^\expert - \rho^{\piout}\bigr] = \cO \spr*{\varepsilon}$.
\end{restatable}
There are two important remarks for the nonlinear extension. First, the maximization of $\hLoss \spr*{\pi_k; Q}$ with respect to $Q$ is no longer available in closed form and it might not even be a concave optimization problem depending on the choice of the function class $\cQ$. Therefore, computational efficiency cannot be ensured. Nevertheless, the form of the objective function remains very simple in terms of $Q$, and is arguably easier to optimize than other popular objective functions that are routinely optimized within deep RL with good empirical success (\eg, the objective functions appearing in \citep{Mnih:2015}) and deep IL \citep{Garg:2021}. Secondly, the expert sample complexity bound degrades from $\cO \spr*{\varepsilon^{-2}}$ achieved in the linear case to $\cO \spr*{\varepsilon^{-4}}$ in the nonlinear case due to the higher complexity of the policies produced by the algorithm (which results in a larger covering number of the policy class as highlighted in the proof sketch included in the next section).
\section{Analysis}
In this section we outline the proof of our two main results. Both proofs are based on two key steps which are self evident from Proposition~\ref{prop:decomp}. The first one consists of a regret analysis to show that $\sumkK \Loss \spr*{\pi_k; Q_k}$ is bounded sublinearly in $K$. At a high level, the proof makes use of a classic technique of decomposing the ``global'' regret into the average of ``local'' regrets in each MDP state, first proposed by \citet{Even-dar:2008,Even-dar:2009} and used in numerous other works (\eg, \citep{Abbasi-Yadkori:2019b,Geist:2019,lan2023policy,moulin2023optimistic}). In proving this result, a little care is needed in handling the potentially nonstationary nature of the expert policy. We circumvent the issue by using the performance difference lemma and controlling the regret at each state against the stationary comparator which induces the same state-action occupancy measure of the expert. Formally, we have the following bound, which we prove in Appendix~\ref{app:proof_mirror}.
%
\begin{restatable}{Lem}{mirrorlemma} \label{lemma:mirror}
For any $k$ and any state-action pair $\spr*{x, a}$, consider the sequence of policies starting with $\pi_1$ as the uniform policy and updated as $\pi_{k+1} \spr*{a \given x} \propto \pi_k \spr*{a \given x} e^{\eta Q_k \spr*{x, a}}$ for some function $Q_k\colon \cX \times \cA \rightarrow \bbR$ such that $\norm{Q_k}_\infty \leq \frac{1}{1 - \gamma}$. Then,
$
\sumkK \Loss \spr*{\pi_k; Q_k} \leq \frac{\log A}{\eta} + \frac{\eta K}{2 \spr*{1 - \gamma}^2}
$.
\end{restatable}
%
This lemma applies to both the linear and nonlinear settings. The next and final step of the analysis is to establish concentration of the empirical objective and bound $\Delta \spr*{\pi_k}$ for each $k$. The main challenge in this step is the correlation between the iterates $\scbr*{\pi_k}^K_{k=1}$ and the expert dataset. This can be handled via a uniform bound over the policy class to which all the algorithm iterates belong to. Importantly, this class is much smaller than the class of all policies, and allows us to make massive sample-complexity savings as compared to methods that need to control estimation errors associated with arbitrary policies. We provide the technical details separately for the linear and nonlinear cases.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Linear function approximation}
In order to bound the estimation errors $\Delta \spr*{\pi_k}$, we apply a covering argument over the class of linear softmax policies. We have the following result.
%
\begin{restatable}{Lem}{concentration} \label{lemma:vec_concentration}
Let $\scbr*{\pi_k}_{k \in \sbr*{K}}$ be the sequence of policies generated by Algorithm~\ref{alg:main} and let $\Delta \spr*{\pi_k}$ be defined as in Proposition~\ref{prop:decomp}. Then, with probability at least $1 - \delta$, it holds that for all $k \in \sbr*{K}$
%
\begin{equation*}
\Delta \spr*{\pi_k} \leq \frac1K + 4 \sqrt{\frac{d}{\spr*{1 - \gamma}^2 \tauE} \log \spr*{\frac{2 + 32 K^2 \eta \ThetaMAX \phiMAX A}{\spr*{1 - \gamma} \delta}}}\,,
\end{equation*}
\end{restatable}
%
We defer the proof to Appendix~\ref{app:proof_vec_concentration}. We can use the above result to sketch the proof of Theorem~\ref{thm:main_formal}.
%
\begin{proof}[\textit{Proof sketch of Theorem~\ref{thm:main_formal}}]
Using Lemma~\ref{lemma:mirror} with $\eta = \spr*{1 - \gamma} \sqrt{\frac{2 \log A}{K}}$ and dividing by $K$, we obtain that
%
$
\frac1K \sumkK \Loss \spr*{\pi_k; Q_k} \leq \sqrt{\frac{2 \log A}{\spr*{1 - \gamma}^2 K}}
$.
Therefore, setting $K = \frac{2 \log A}{\spr*{1 - \gamma}^2 \varepsilon^2}$ guarantees $\frac1K \sumkK \Loss \spr*{\pi_k; Q_k} \leq \varepsilon$. Then, using the high-probability bound in Lemma~\ref{lemma:vec_concentration} and the fact that $K^{-1}\sumkK \Delta \spr*{\pi_k}$ is a random random variable bounded by $\spr*{1 - \gamma}^{-1}$ almost surely, we obtain the following expectation bound which holds for all $\delta > 0$
%
\begin{align*}
\frac1K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}} &\leq \frac1K + C \sqrt{\frac{d}{\spr*{1 - \gamma}^2 \tauE} \log \spr*{\frac{\ThetaMAX \phiMAX A}{\spr*{1 - \gamma}^3 \varepsilon^2 \delta}}} + \frac{\delta}{1 - \gamma}\,,
\end{align*}
%
for some $C \in \bbR$. Noticing that the choice of parameters ensures $\frac1K \leq \frac{\varepsilon}{2}$ and setting $\delta = \frac{\varepsilon \spr*{1 - \gamma}}{2}$ and
%
$
\tauE \geq \frac{ C^2 d }{\spr*{1 - \gamma}^2\varepsilon^2} \log \spr*{\frac{\ThetaMAX \phiMAX A }{\spr*{1 - \gamma}^3 \varepsilon^2 \delta}}
$, this bound implies that $\frac2K \sumkK \bbE \sbr*{\Delta_k} \leq 4 \varepsilon$. Invoking Proposition~\ref{prop:decomp}, we conclude that $\bbE \bigl[\rho^{\expert} - \rho^{\piout}\bigr] \leq 5 \epsilon$. The full proof is in Appendix~\ref{app:proof_main_formal}.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{General function approximation}
The proof for the nonlinear setup follows the same conceptual steps but requires a more general concentration result for the objective function. Namely, the following lemma is the general counterpart of Lemma~\ref{lemma:vec_concentration}. The feature dimension $d$ appearing in the linear case is replaced by the complexity (as measured by the covering number) of the policy and value function classes containing the iterates.
%
\begin{Lem} \label{lemma:concentration-nonlinear}
Let $\Pi_\cQ$ denote the policy class containing the iterates $\scbr*{\pi_k}^K_{k=1}$ produced by Algorithm~\ref{alg:spoil-general-fa}, then with probability at least $1 - \delta$, for all $k \in \sbr*{K}$ it holds that
%
\begin{equation*}
\Delta \spr*{\pi_k} = \sup_{Q \in \cQ} \abs{\hLoss \spr*{\pi_k; Q} - \Loss \spr*{\pi_k; Q}} \leq \frac{1}{2 K} + \sqrt{\frac{2 \spr*{K + 1} \log \br{2 \cN_{\frac{1 - \gamma}{8 K^2 \eta A}} \spr*{\cQ, \norm{\cdot}_\infty} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}\,.
\end{equation*}
\end{Lem}
%
The proof is in Appendix~\ref{app:proof_concentration_nonlinear}. Note that in the nonlinear case, the complexity of the policy class increases linearly with the number of iterations $K$ (see Lemma~\ref{lemma:covering_non_linear}). On the contrary, in the linear case, the policies generated by Algorithm~\ref{alg:main} are parameterized by $d$ parameters and only the magnitude of these parameters increases with $K$. With this lemma, we present the proof sketch of Theorem~\ref{thm:mainnonlinear}.
%
\begin{proof}[\textit{Proof sketch of Theorem~\ref{thm:mainnonlinear}}]
Applying the decomposition in Proposition~\ref{prop:decomp}, the regret bound in Lemma~\ref{lemma:mirror}, the concentration in Lemma~\ref{lemma:concentration-nonlinear}, we obtain $\bbE \bigl[\rho^\expert - \rho^{\piout}\bigr] = \tilde{\cO} \Bigl(\frac{1}{\sqrt{K}} + \sqrt{\frac{K}{\tauE}}\Bigr)$. Setting $K = \tilde{\cO} \spr*{\varepsilon^{-2}}$, and $\tauE = \tilde{\cO} \spr*{\varepsilon^{-4}}$, we get $\bbE \bigl[\rho^\expert - \rho^{\piout}\bigr] = \varepsilon$. The full proof is in Appendix~\ref{app:proof_main_nonlinear}.
\end{proof}
\section{Numerical experiments}
We conduct experiments to verify that we can efficiently imitate complex experts in linear $Q^\pi$ environments, and can achieve massive improvements over behavioral cloning with large policy classes.\footnote{Code is available at: \url{https://github.com/antoine-moulin/spoil}.}
%
\begin{wrapfigure}{r}{0.4\textwidth}
\centering
\begin{tabular}{c}
\subfloat{%
\includegraphics[width=.95\linewidth]{experiments/Easy_expert100.pdf}
} \\
\subfloat{%
\includegraphics[width=.95\linewidth]{experiments/Complex_expert100.pdf}
}
\end{tabular}
\caption{Experiments with simple and complex experts. Curves are averaged across $10$ seeds. \label{exp:first}}
\end{wrapfigure}
%
To investigate this, we consider a randomly generated large linear MDP (a special case of linear $Q^\pi$-realizable MDP) with $\abs{\cX} = 500$ and $A = 1000$ but with a small feature dimension $d = 7$. We instantiate two experts. The first expert is trained to be the soft optimal policy in this environment. This policy is parametrized by only $d$ parameters and can be realized by the following policy class $\PiElin = \scbr*{\pi \spr*{a \given x} = \frac{\exp \spr*{\inp{\varphi \spr*{x, a}, \theta}}}{\sum_{b \in \cA} \exp \spr*{\inp{\varphi \spr*{x, b}, \theta}}}, \theta \in \bbR^d, \norm{\theta} \leq \ThetaMAX}$. In addition, we consider the second expert, which belongs to the class of three-layer neural networks denoted by $\PiENN$. This expert was trained to minimize the KL divergence with respect to the linear expert.
%
As evident from Figure~\ref{exp:first}, our algorithm \SPOIL performs well for both experts. This is in perfect agreement with the theory which provides a sample complexity bound that is independent of the expert policy class. On the other hand, behavioural cloning (\BC) struggles with the complexity of the neural network expert policy class, and performs much worse. This is despite the fact that the dataset perfectly satisfies the realizability condition required by \BC. This clearly demonstrates that complex behavior policies may indeed be problematic for \BC to deal with, and we expect that such issues may also cause real performance drops in practical applications as well. Notice that in this experiment, \SPOIL outperforms \BC because the environment complexity is much lower than the policy class complexity. For fairness, we point out that the opposite situation is not unusual in RL and IL. In that case, it is reasonable to expect \BC to be superior to \SPOIL.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Continuous states experiments}
We run the general function approximation version of our algorithm in continuous-states environments from the \texttt{gym} library \citep{towers2024gymnasium}. In particular, we consider the environments \cartpole, \acrobot and \lunarlander where the expert is trained via \SoftDQN. We use the expert data provided in the code base of \cite{Garg:2021}. The learner aims at imitating the expert performance given as input a variable number of expert trajectories. In order to make the task more challenging the trajectories are subsampled each $20$ steps in \cartpole, \acrobot and each $5$ in \lunarlander.\footnote{This is common practice in IL experiments (see, \eg, \citealp{Garg:2021}).} We compare the performance of the best policy found by each of these algorithms as a function of the number of expert trajectories given as input.
%
\begin{figure*}[h]
\centering
\begin{tabular}{ccc}
\subfloat{%
\includegraphics[width=0.3\linewidth]{experiments/CartPole-v1_offline.pdf}
} &
\subfloat{%
\includegraphics[width=0.3\linewidth]{experiments/Acrobot-v1_offline.pdf}
} &
\subfloat{%
\includegraphics[width=0.3\linewidth]{experiments/LunarLander-v2_offline.pdf}
}
\end{tabular}
\caption{Experiments in continuous states domains. Curves are averaged across $10$ seeds. \label{fig:deep_pdoil}}
\end{figure*}
%
In practice the maximization $\argmax_{Q \in \cQ} \hLoss \spr*{\pi_k, Q}$ is approximated by performing a gradient ascent step. On the other hand, the actor update is approximated via \SoftDQN \citep{haarnoja2017reinforcement}. In Figure~\ref{fig:deep_pdoil}, we can see that \SPOIL performs comparably to the state-of-the-art algorithm \IQLearn \citep{Garg:2021} and improves upon \BC \citep{Pomerleau:1991,foster2024behavior} and \PIL \citep{viano2022proximal}.
\section{Conclusions}
In this work, we proposed analyses that leverages structural assumptions on the MDP without requiring trajectory access. This is made possible thanks to a novel regret decomposition that shifts the focus from updating a reward sequence based on expert data to updating a sequence of state-action value functions. To the best of our knowledge, these are the first rigorous theoretical guarantees for IL methods that learn state-action value functions from expert data, a technique popularized in practice by \citet{Garg:2021}. Among the many potential ways to improve extend and improve our work, we highlight a handful of directions in Appendix~\ref{app:future}.
All previous theory work has focused either on imitation learning with additional trajectory access to the environment, both in tabular MDPs \citep{Shani:2021,xu2023provably} and with additional structural assumptions \citep{Liu:2022,viano2022proximal,viano2024imitation,moulin2025optimistically}, or learning based on offline data only but under structural assumptions about the policy class used by the expert \citep{rajaraman2021value,swamy2022minimax,foster2024behavior,rohatgi2025computational}. The first of these assumptions is clearly more restrictive than what we have considered in this work, and we have pointed out potential issues with the second set of methods when the policy class is exceedingly complex. This is not to say though that we consider our approach strictly superior to policy-based IL methods: as is often the case in RL, there is no single approach that dominates all others in all problems, and sometimes policy-based methods are more suitable for the job than value-based ones. Thus, even if our approach is not the ultimate answer to all questions in imitation learning, our results show that it is one potential alternative to consider in situations where other methods fail.
\section{Additional related works}
Classical analyses by \citet{Ross:2010,Ross:2011} on behavioural cloning (\BC) established an error propagation framework relating the suboptimality of the learned policy to the worst-case generalization error incurred in predicting the expert policy. They proved that this suboptimality gap is upper-bounded by the generalization error up to a multiplicative factor $H^2$ (where $H$ is the horizon), a factor that is unavoidable when using the 0-1 loss for supervised learning. However, these results do not quantify the \emph{expert sample complexity}, or the number of samples required to make the generalization error small.
A recent line of work has begun to investigate the expert sample complexity assuming knowledge of a policy class $\PiE$ that realizes (or nearly realizes) the expert policy. For instance, \citet{rajaraman2021value} assume that the expert is deterministic and belongs to the class of deterministic linear policies $\Pi_{\mathrm{det, lin}}$ (defined in the caption of Table~\ref{tab:related}). They prove a bound on the required number of expert samples of order $\widetilde{\cO} \spr*{\spr*{H^2 d}/{\varepsilon}}$, where $d$ is the feature dimension in the definition of $\Pi_{\mathrm{det, lin}}$. Their technique is a reduction to the problem of multiclass classification in supervised learning, but their result is not informative for settings with general stochastic expert policies.
Further contributions to understanding the sample complexity of IL under policy class assumptions were made by \citet{foster2024behavior}. Specifically, assuming the expert is included within a known class, $\expert \in \PiE$, they showed that one can learn an $\varepsilon$-optimal policy (as defined in Equation~\eqref{eq:goal}) after observing $\cO \spr*{\spr*{H^2 \log \abs{\PiE}}/{\varepsilon}}$ samples for a deterministic expert or $\cO \spr*{\spr*{H^2 \log \abs{\PiE}}/{\varepsilon^2}}$ samples for a stochastic one (we report the dense reward case for brevity, though their bounds improve for sparse rewards). Addressing scenarios where the expert policy might only be almost well-specified, \citet{rohatgi2025computational} demonstrate that there exists a computationally efficient algorithm that outputs an $\varepsilon$-optimal policy up to an additional approximation error of $H \log \spr*{W} \min_{\pi \in \PiE} \Dhels{\bbP^\pi}{\bbP^\expert}$. In this context, $\bbP^\pi$ is the trajectory distribution induced by $\pi$, $W$ is a density ratio defined as
%
\begin{equation*}
W = \max_{\pi \in \PiE} \max_{\spr*{x, a} \in \cX \times \cA} \max_{h \in \sbr*{H}} \frac{\experth \spr*{a \given x}}{\pi_h \spr*{a \given x}}\,.
\end{equation*}
%
It is worth noting that these guarantees become vacuous when the policy class $\PiE$ is such that at least one policy in $\PiE$ fails to provide sufficient coverage for the expert's actions (leading to $W = +\infty$ as $\pi_h \spr*{a \given x}$ gets close to zero for relevant state-action pairs and timestep where $\experth \spr*{a \given x} > 0$), or if the minimum Hellinger distance $\min_{\pi \in \PiE} \Dhels{\bbP^\pi}{\bbP^\expert}$ is large. Alternatively, \citet{foster2024behavior} proved a misspecification result where the additional error is $\min_{\pi \in \PiE} \chi^2 \spr*{\bbP^\pi, \bbP^\expert}$. This misspecification error is measured by the $\chi^2$ divergence, with a leading coefficient constant in $H$ and $W$. However, the $\chi^2$ divergence is an upper bound on the Hellinger distance that is often way too loose to be practical. In a similar vein, \cite{espinosa2025efficient} proved a benefit in terms of error propagation for a local search algorithm over behavioural cloning in misspecified settings, under the assumption that the learned policy is allowed to reset to states visited in the expert dataset.
Our work aligns with the recent renewed interest in proving refined expert sample complexity guarantees for offline imitation learning but distinguishes itself by swapping out the expert realizability assumption with a structural assumption on the environment. Early explorations for similar settings can be found in classical works by \citet{Abbeel:2004} and \citet{Syed:2007}. These studies proposed offline learning algorithms for MDPs with reward functions linear in a collection of features known to the learner, under the assumption that transition dynamics of the environment is also known. Versions of their approaches that do not assume such knowledge typically incur a worse sample complexity and often apply only in the tabular setting. Our work generalizes these classical approaches by removing the need for known transitions and for rewards to be linear in the features, as well as going beyond tabular MDPs. Notably, the linear $Q^\pi$-realizability assumption can hold even if the reward function and the transition dynamics are nonlinear. We summarize our comparison with these and other related works in Table~\ref{tab:related}.
Our work focuses on \emph{learning a $Q$-value from expert data} and, in this regard, is closely related to the practical work of \citet{Garg:2021}. The novel regret decomposition employed in our analysis of \SPOIL demonstrates, we believe for the first time, that provable guarantees are achievable by directly learning an action-value function from expert data. This contrasts with the majority of theoretical and practical imitation learning approaches, which typically first use the expert data to learn a reward function and subsequently use this learned reward function to infer an action-value function.
As we mentioned, \SPOIL is very related to \texttt{AdVIL}. However, a key difference lies in the analysis: \citet{swamy2021moments} conduct an error propagation analysis for \texttt{AdVIL}. From this, they conclude that \texttt{AdVIL} is equivalent to \BC in the sense that if the loss for either method is at most $\varepsilon$ in every state, then the suboptimality of the extracted policy in an episodic setting with horizon $H$ is of order $H^2 \varepsilon$ for both. However, this type of result does not investigate the assumptions or the number of samples needed to ensure these losses are indeed less than $\varepsilon$. Our work addresses this open question, establishing a clear distinction between the sample complexities of \SPOIL and \BC. Specifically, \SPOIL and \BC (and their respective analyses) rely on largely orthogonal sets of assumptions, making the two approaches complementary to each other: we expect \SPOIL to be more suitable for imitation tasks with complex experts but simpler environments, while \BC may be the preferred choice when this situation is reversed. Our sample complexity analysis for \SPOIL critically relies on the $Q$-player using a best response strategy, and it is unlikely that equivalent results could be achieved using a standard gradient ascent step for the $Q$-player instead.
Very recently, \citet{simchowitz2025pitfalls} analyzed the error propagation properties of offline imitation learning algorithms in continuous action MDPs, showing that an exponential dependence on the horizon of the problem is unavoidable if no structure is imposed on the environment. On the other hand, the same authors point out that if the state-action value functions were Lipschitz in the action space, then efficient learning would be possible. Conceptually, we believe that the \SPOIL algorithm could also be applied in the continuous action case. Such an extension would suggest that another scenario enabling effective imitation learning in continuous action spaces arises when the learner has access to a suitably expressive class of state-action value functions.
Following a similar line of research that studies imitation learning from a control-theoretic perspective, \citet{block2023provable} studied guarantees for generative behavioural cloning, assuming access to a stabilizing policy dubbed a \emph{synthesis oracle}. These policies can be computed exactly if the dynamics are known, an assumption which is not imposed in our work. However, when provided with such an oracle, \citet{block2023provable} derive bounds on a stricter metric for imitation. Specifically, they bound the probability that expert and learner trajectories diverge at some time step, as opposed to the difference in cumulative return that we analyze in our work.
\section{Omitted proofs}
\label{app:omitted}
In this appendix, we provide the omitted proofs of the main results.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proof of Lemma~\ref{lem:performance-difference} (performance difference lemma)} \label{app:proof_pdl}
We start presenting the performance difference lemma proven in a more general form which allows one policy to be nonstationary.
\pdl*
\begin{proof}
Consider the Bellman equations for the stationary policy $\pi$. For any state-action pair $\spr*{x, a}$, we have
%
\begin{equation*}
Q^\pi \spr*{x, a} = r \spr*{x, a} + \gamma \sum_{x'\in \cX} P \spr*{x' \given x, a} V^\pi \spr*{x'}\,.
\end{equation*}
%
Averaging both sides with the distribution $\mu^{\pi'}$ and reordering the terms, we obtain
%
\begin{align*}
\sum_{x, a} \mu^{\pi'} \spr*{x, a} r \spr*{x, a} &= \sum_{x, a} \mu^{\pi'} \spr*{x, a} \spr*{Q^\pi \spr*{x, a} - \gamma \sum_{x'\in \cX} P \spr*{x' \given x, a} V^\pi \spr*{x'}} \\
&= \spr*{1 - \gamma} \sum_x \nu_0 \spr*{x} V^\pi \spr*{x} + \sum_{x,a} \mu^{\pi'} \spr*{x, a} \bpa{Q^\pi \spr*{x, a} - V^\pi \spr*{x}}\,,
\end{align*}
%
where we used the flow condition of the occupancy measure $\mu^{\pi'}$ in the last step (see Equation~\ref{eq:flow}). The claim then follows by noticing that $\rho^\pi = \spr*{1 - \gamma} \sum_x \nu_0 \spr*{x} V^\pi \spr*{x}$ and $\rho^{\pi'} = \sum_{x, a} \mu^{\pi'} \spr*{x, a} r \spr*{x, a}$.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proof of Lemma~\ref{lemma:mirror} (regret of the policy player)} \label{app:proof_mirror}
Next, we apply Lemma~\ref{lemma:mirror_orabona} to the special case of the exponential weights update, where the divergence is chosen to be the KL divergence, and use it to derive a bound on the regret of the policy player.
\mirrorlemma*
\begin{proof}
Let us recall that
%
\begin{equation*}
\Loss \spr*{\pi_k, Q_k} = \bbE_{\spr*{X, A} \sim \mu^{\expert}} \sbr*{Q_k \spr*{X, A} - Q_k \spr*{X, \pi_k}}\,,
\end{equation*}
%
where $\expert$ is a potentially nonstationary policy. To continue, let us consider the stationary policy $\barexpert\colon \cX \rightarrow \Delta \spr*{\cA}$ that induces the same state-action occupancy measure of the expert, \ie, such that $\mu^\barexpert = \mu^\expert$. This equality can be guaranteed by choosing, for any $\spr*{x, a} \in \cX \times \cA$, $\barexpert \spr*{a \given x} = \frac{\mu^\expert \spr*{x, a}}{\nu^\expert \spr*{x}}$ if $\nu^\expert \spr*{x} \neq 0$ and $\pi_0 \spr*{a}$ otherwise, where $\pi_0 \in \Delta \spr*{\cA}$ is an arbitrary distribution. Then, we continue as follows
%
\begin{align*}
\Loss \spr*{\pi_k, Q_k} &= \bbE_{\spr*{X, A} \sim \mu^\expert} \sbr*{Q_k \spr*{X, A} - Q_k \spr*{X, \pi_k}} \\
&= \bbE_{\spr*{X, A} \sim \mu^\barexpert} \sbr*{Q_k \spr*{X, A} - Q_k \spr*{X, \pi_k}} \\
&= \sum_{x \in \cX} \nu^\barexpert \spr*{x} \sum_{a \in \cA} Q_k \spr*{x, a} \spr*{\barexpert \spr*{a \given x} - \pi_k \spr*{a \given x}}\,.
\end{align*}
%
Summing over $k \in \sbr*{K}$, we obtain
%
\begin{equation*}
\sumkK \Loss \spr*{\pi_k, Q_k} = \sum_{x \in \cX} \nu^{\barexpert} \spr*{x} \sumkK\sum_{a \in \cA} Q_k \spr*{x, a} \spr*{\barexpert \spr*{a \given x} - \pi_k \spr*{a \given x}}\,.
\end{equation*}
%
It remains to prove the following bound.
%
\begin{align*}
\sumkK \sum_{a \in \cA} Q_k \spr*{x, a} \spr*{\barexpert \spr*{a \given x} - \pi_k \spr*{a \given x}} \leq \frac{\log A}{\eta} + \frac{\eta K}{2 \spr*{1 - \gamma}^2}\,.
\end{align*}
%
The result is proven as a particular case of Lemma~\ref{lemma:mirror_orabona}. Specifically, we have that when $V$ is the $A$-dimensional simplex and the Bregman divergence is the KL divergence, it holds that
%
\begin{equation*}
x_{k+1} = \argmin_{v \in V} \scbr*{\inp{\ell_k, v} + \frac1\eta D \spr*{v, x_k}} = \frac{x_k \odot \exp \spr{- \eta \ell_k}}{\inp{\bfone, x_k \odot \exp \spr*{- \eta \ell_k}}}\,,
\end{equation*}
%
where $\odot$ is the elementwise product. We apply Lemma~\ref{lemma:mirror_orabona} for each state $x \in \cX$, replacing $x_k = \pi_k \spr*{\cdot \given x}$ and $\ell_k = - Q_k \spr*{x, \cdot}$. We obtain that for the update $\pi_{k+1} \spr*{a \given x} \propto \pi_k \spr*{a \given x} e^{\eta Q_k \spr{x, a}}$, the guarantee in Lemma~\ref{lemma:mirror_orabona} holds. Moreover, in this setting we have $\lambda = 1$, and $\ell_{\mathrm{max}} = \frac{1}{1 - \gamma}$. Given that for any state-action pair $\spr*{x, a}$, the initial policy is $\pi_1 \spr*{a \given x} = \frac{1}{A}$, we have that $D \spr*{\pi \spr*{\cdot \given x}, \pi_1 \spr*{\cdot \given x}} \leq \log A$. Thus, we have the following bound
%
\begin{equation*}
\sum_{a \in \cA} Q_k \spr*{x, a} \spr*{\barexpert \spr*{a \given x} - \pi_k \spr*{a \given x}} \leq \frac{\log A}{\eta} + \frac{\eta K}{2 \spr*{1 - \gamma}^2}\,,
\end{equation*}
%
and the conclusion follows from $\nu^\barexpert$ being a probability distribution.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{General concentration argument}
To prove the main results of this paper, we prove a general concentration inequality that we will use for the iterates produced by both Algorithm~\ref{alg:main} and Algorithm~\ref{alg:spoil-general-fa}. Specifically, when analyzing Algorithm~\ref{alg:main},
we consider the policy class $\Pilin$ defined as follows
%
\begin{equation} \label{eq:Pilin}
\Pilin = \scbr*{
\pi \in \Delta \spr*{\cA}^\cX: \exists \spr*{\theta_k}_{k \in \sbr{K}} \subset \fkB \spr*{\ThetaMAX}, \pi \spr*{a \given x} = \frac{\exp \spr*{\eta \sumkK \inp{\varphi \spr*{x, a}, \theta_k}}}{\sum_{b \in \cA} \exp \spr*{\eta \sumkK \innerprod{\varphi \spr*{x, b}}{\theta_k}}}
}\,,
\end{equation}
%
while in the nonlinear case (Algorithm~\ref{alg:spoil-general-fa}), we will consider the policy class
%
\begin{equation} \label{eq:PiQ}
\Pi_\cQ = \scbr*{
\pi \in \Delta \spr*{\cA}^\cX: \exists \spr*{Q_k}_{k \in \sbr{K}} \subset \cQ, \pi \spr*{a \given x} = \frac{\exp \spr*{\eta \sumkK Q_k \spr*{x, a}}}{\sum_{b \in \cA} \exp \spr*{\eta \sumkK Q_k \spr*{x, b}}}}\,.
\end{equation}
%
The result is the following.
%
\begin{restatable}{Lem}{unifconc} \label{lemma:unif_conc}
Let us consider a value function class $\cQ \subset \bbR^{\cX \times \cA}$ and the sequences of estimated objective functions $\bc{\hLoss \spr*{\pi_k, Q}}^K_{k=1}$ for a policy sequence $\bc{\pi_k}^K_{k=1}$ belonging to a policy class $\Pi$. For any $k \in \sbr*{K}$, recall that for any policy $\pi$ and function $Q$, the objective function is defined as
%
\begin{equation*}
\Loss \spr*{\pi; Q} = \bbE_{\spr*{X, A} \sim \mu^\expert} \sbr*{Q \spr*{X, A} - Q \spr*{X, \pi}}\,.
\end{equation*}
%
Then, with probability larger than $1 - \delta$, it holds that for all $k \in \sbr*{K}$ simultaneously that
%
\begin{equation*}
\Delta \spr*{\pi_k} = \sup_{Q \in \cQ} \abs{\hLoss \spr*{\pi_k, Q} - \Loss \spr*{\pi_k, Q}} \leq \inf_{\epsilon: \epsilon > 0} \scbr*{\frac{4 \epsilon}{1 - \gamma} + \sqrt{\frac{2 \log \spr*{2 \cN_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty,1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}}\,,
\end{equation*}
%
where, for any $\spr*{Q, \pi} \in \cQ \times \Pi$, we defined the norm $\norm{\spr*{Q, \pi}}_{\infty, 1} = \norm{Q}_\infty +\max_{x \in \cX} \norm{\pi \spr*{\cdot \given x}}_1 $.
\end{restatable}
\begin{proof}
Let us recall that for any $Q \in \cQ$ and any $k \in \sbr*{K}$, we have
%
\begin{equation*}
\hLoss \spr*{\pi_k, Q} = \frac{1}{\tauE} \sum^{\tauE}_{i=1} \spr*{Q \spr*{\XEi, \AEi} - \sum_{a \in \cA} \pi_k \spr*{a \given \XEi} Q \spr*{\XEi, a}}\,,
\end{equation*}
%
and notice that $\hLoss \spr*{\pi_k, Q}$ is not an unbiased estimator of $\Loss \spr*{\pi_k, Q}$ since the policy $\pi_k$ depends on the expert data. Therefore, we aim at establishing a uniform concentration bound over the policy class $\Pi$. To this end, let us consider a fixed pair $\spr*{Q, \pi} \in \cC_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty, 1}}$, and notice that $\hLoss \spr*{\pi, Q}$ is a sum of random variables of the form
%
\begin{equation*}
W_i = \frac{1}{\tauE} \spr*{Q \spr*{\XEi, \AEi} - \sum_{a \in \cA} \pi \spr*{a \given \XEi} Q \spr*{\XEi, a}}\,,
\end{equation*}
%
where $i \in \sbr*{\tauE}$. Each $W_i$ an unbiased estimator of $\Loss \spr*{\pi, Q}$ since $\pi$ is fixed (\ie, $\pi$ is not a random quantity depending on the expert data) and $\spr*{\XEi, \AEi} \sim \mu^\expert$ for all $i \in \sbr*{\tauE}$. Thus, for any $i \in \sbr*{\tauE}$, $\bbE \sbr*{W_i} = \Loss \spr*{\pi, Q}$. Moreover, notice that for all $i \in \sbr*{\tauE}$, $- \frac{1}{\tauE \spr*{1 - \gamma}} \leq W_i \leq \frac{1}{\tauE \spr*{1 - \gamma}}$. Therefore, by an application of Hoeffding's inequality (see Lemma~\ref{lemma:hoeffding}), we have that for all $t> 0$,
%
\begin{equation*}
\bbP \sbr*{\abs{\hLoss \spr*{\pi, Q} - \Loss \spr*{\pi, Q}} \geq t} \leq 2 \exp \spr*{- \frac{2 t^2 \tauE \spr*{1 - \gamma}^2}{4}}\,.
\end{equation*}
%
That is, choosing $t = \frac{2 \log \spr*{2 / \delta}}{\spr*{1 - \gamma}^2 \tauE}$ guarantees that with probability at least $1 - \delta$,
%
\begin{equation*}
\abs{\hLoss \spr*{\pi, Q} - \Loss \spr*{\pi, Q}} \leq \sqrt{\frac{2 \log \spr*{2 / \delta}}{\spr*{1 - \gamma}^2\tauE}}\,.
\end{equation*}
%
Applying a union bound, we further have that with probability at least $1 - \delta$, for all $\spr*{Q, \pi} \in \cC_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty, 1}}$ it holds that
%
\begin{equation*}
\abs{\hLoss \spr*{\pi, Q} - \Loss \spr*{\pi, Q}} \leq \sqrt{\frac{2 \log \spr*{2 \cN_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty, 1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}\,.
\end{equation*}
%
Recall that $\cC_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty, 1}}$ is assumed to be an $\epsilon$-covering set of the space $\cQ \times \Pi$ with respect to the norm
$\spr*{Q, \pi} \mapsto \norm{\spr*{Q, \pi}}_{\infty, 1} = \norm{Q}_\infty +\max_{x \in \cX} \norm{\pi \spr*{\cdot \given x}}_1$. For any pair $\spr*{Q, \pi_k} \in \cQ \times \Pi$, let $\spr*{Q_\epsilon, \pi_{k, \epsilon}} \in \cC_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty, 1}}$ denote the element of the covering such that $\norm{\spr*{Q, \pi_k} - \spr*{Q_\epsilon, \pi_{k, \epsilon}}}_{\infty, 1} \leq \epsilon$. Then, we have that
%
\begin{align*}
\abs{\hLoss \spr*{\pi_k, Q} - \hLoss \spr*{\pi_{k, \epsilon}, Q_\epsilon}} &\leq \abs{\frac{1}{\tauE} \sum^{\tauE}_{i=1} \spr*{Q \spr*{\XEi, \AEi} - Q_\epsilon \spr*{\XEi, \AEi}}} \\
&\phantom{\leq} + \abs{\frac{1}{\tauE} \sum^{\tauE}_{i=1} \sum_{a \in \cA} \spr*{\pi_{k, \epsilon} \spr*{a \given \XEi} Q_\epsilon \spr*{\XEi, a} - \pi_k \spr*{a \given \XEi} Q \spr*{\XEi, a}}} \\
&\hspace*{-2.6pt}\leq \norm{Q - Q_\epsilon}_\infty + \abs{\frac{1}{\tauE} \sum^{\tauE}_{i=1} \sum_{a \in \cA} \spr*{\pi_{k, \epsilon} \spr*{a \given \XEi} - \pi_k \spr*{a \given \XEi}} Q_\epsilon \spr*{\XEi, a}} \\
&\phantom{\leq} + \abs{\frac{1}{\tauE} \sum^{\tauE}_{i=1} \sum_{a \in \cA} \pi_k \spr*{a \given \XEi} \spr*{Q \spr*{\XEi, a} - Q_\epsilon \spr*{\XEi, a}}}\,.
\end{align*}
%
Noting that for any $Q \in \cQ$, $\norm{Q}_\infty \leq \frac{1}{1 - \gamma}$, and that for any state $x$, $\pi_k \spr*{\cdot \given x} \in \Delta \spr*{\cA}$, using Hölder's inequality, we further have
%
\begin{align*}
\abs{\hLoss \spr*{\pi_k, Q} - \hLoss \spr*{\pi_{k, \epsilon}, Q_\epsilon}} &\leq \norm{Q - Q_\epsilon}_\infty + \frac{\max_{x \in \cX} \norm{\pi_{k, \epsilon} \spr*{\cdot \given x} - \pi_k \spr*{\cdot \given x}}_1}{1 - \gamma} + \norm{Q - Q_\epsilon}_\infty \\
&\leq \frac{2 \epsilon}{1 - \gamma}\,,
\end{align*}
%
where we used the definition of $\spr*{\pi_{k, \epsilon}, Q_\epsilon}$ and $\gamma \in \spr*{0, 1}$ in the last inequality. Similarly, for the true objective we have that
%
\begin{align*}
\abs{\Loss \spr*{\pi_k, Q} - \Loss \spr*{\pi_{k, \epsilon}, Q_\epsilon}} &\leq \abs{\bbE_{\spr*{X, A} \sim \mu^\expert} \sbr*{Q \spr*{X, A} - Q_\epsilon \spr*{X, A}}} \\
&\phantom{\leq}+ \abs{\bbE_{X \sim \nu^\expert} \sbr*{Q \spr*{X, \pi_k} - Q_\epsilon \spr*{X, \pi_{k, \epsilon}}}} \\
&\leq \norm{Q - Q_\epsilon}_\infty + \abs{\bbE_{X \sim \nu^\expert} \sbr*{Q \spr*{X, \pi_k} - Q \spr*{X, \pi_{k, \epsilon}}}} \\
&\phantom{\leq}+ \abs{\bbE_{X \sim \nu^\expert} \sbr*{Q \spr*{X, \pi_{k, \epsilon}} - Q_\epsilon \spr*{X, \pi_{k, \epsilon}}}} \\
&\leq \norm{Q - Q_\epsilon}_\infty + \frac{\max_{x \in \cX} \norm{\pi_{k, \epsilon} \spr*{\cdot \given x} - \pi_k \spr*{\cdot \given x}}_1}{1 - \gamma} + \norm{Q - Q_\epsilon}_\infty \\
&\leq \frac{2 \epsilon}{1 - \gamma}\,.
\end{align*}
%
Therefore, with probability at least $1 - \delta$, it holds that for any $k \in \sbr*{K}$ and any $Q \in \cQ$,
%
\begin{align*}
\abs{\hLoss \spr*{\pi_k, Q} - \Loss \spr*{\pi_k, Q}} &\leq \abs{\hLoss \spr*{\pi_k, Q} - \hLoss \spr*{\pi_{k, \epsilon}, Q_\epsilon}} + \abs{\hLoss \spr*{\pi_{k, \epsilon}, Q_\epsilon} - \Loss \spr*{\pi_{k, \epsilon}, Q_\epsilon}} \\
&\phantom{\leq}+ \abs{\Loss \spr*{\pi_k, Q} - \Loss \spr*{\pi_{k, \epsilon}, Q_\epsilon}} \\
&\leq \frac{4 \epsilon}{1 - \gamma} + \sqrt{\frac{2 \log \spr*{2 \cN_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty, 1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}\,.
\end{align*}
%
Moreover, since the above bound holds for all $Q \in \cQ$, it holds for the supremum over this class. With probability at least $1 - \delta$, we have for any $k \in \sbr*{K}$ that
%
\begin{align*}
\sup_{Q \in \cQ} \abs{\hLoss \spr*{\pi_k, Q} - \Loss \spr*{\pi_k, Q}} \leq
\frac{4 \epsilon}{1 - \gamma} + \sqrt{\frac{2 \log \spr*{2 \cN_\epsilon \spr*{\cQ \times \Pi, \norm{\cdot}_{\infty, 1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}\,.
\end{align*}
%
The proof is concluded by noting that the above proof holds for any covering size $\epsilon > 0$.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proof of Lemma~\ref{lemma:vec_concentration} (concentration linear case)} \label{app:proof_vec_concentration}
We now instantiate Lemma~\ref{lemma:unif_conc} in the linear $Q^\pi$-realizable setting. For this purpose, we compute a bound on the covering number of the class $\Pilin$, defined in Equation~\eqref{eq:Pilin}.
%
\begin{Lem}[Covering number of $\Pilin$] \label{lemma:covering}
It holds that the covering number of the policy class $\Pilin$ can be bounded as
%
\begin{equation*}
\cN_\epsilon \spr*{\Pilin, \norm{\cdot}_1} \leq \spr*{1 + \frac{2 K \eta \ThetaMAX \phiMAX A}{\epsilon}}^d\,,
\end{equation*}
%
where, with a slight abuse of notation, $\norm{\cdot}_1$ denotes the norm defined for any $\pi \in \Pilin$ as $\norm{\pi}_1 = \sup_{x \in \cX} \norm{\pi \spr*{\cdot \given x}}_1$. Moreover, let
%
\begin{equation*}
\cQ_{\mathrm{lin}} = \scbr*{Q\colon \cX \times \cA \rightarrow \bbR: \exists \theta \in \fkB \spr*{\ThetaMAX}, \forall \spr*{x, a} \in \cX \times \cA,\; Q \spr*{x, a} = \inp{\theta, \varphi \spr*{x, a}}}
\end{equation*}
%
be the class of linear action-value functions. Then, it holds that
%
\begin{equation*}
\cN_\epsilon \spr*{\cQ_{\mathrm{lin}} \times \Pilin, \norm{\cdot}_{\infty, 1}} \leq \spr*{1 + \frac{4 K \eta \ThetaMAX \phiMAX A}{\epsilon}}^{2 d}\,.
\end{equation*}
\end{Lem}
\begin{proof}
Let us consider two policies $\pi$ and $\pi'$ in the class $\Pilin$. There exist $\theta_1, \dots, \theta_K \in \fkB \spr*{\ThetaMAX}$ and $\theta_1', \dots, \theta_K' \in \fkB \spr*{\ThetaMAX}$ such that for any state-action pair $\spr*{x, a} \in \cX \times \cA$, $\pi$ and $\pi'$ can be written as
%
\begin{equation*}
\pi \spr*{a \given x} = \frac{\exp \spr*{\eta \innerprod{\varphi \spr*{x, a}}{\sumkK\theta_k}}}{\sum_{b \in \cA} \exp \spr*{\eta \innerprod{\varphi \spr*{x, b}}{\sumkK\theta_k}}}\,,
\end{equation*}
%
and
%
\begin{equation*}
\pi' \spr*{a \given x} = \frac{\exp \spr*{\eta \innerprod{\varphi \spr*{x, a}}{\sumkK\theta'_k}}}{\sum_{b \in \cA} \exp \spr*{\eta \innerprod{\varphi \spr*{x, b}}{\sumkK\theta'_k}}}\,.
\end{equation*}
%
In particular, let us fix a state $x \in \cX$, and denote $\bar{\theta}_K = \sumkK \theta_k$, $\bar{\theta}_K' = \sumkK \theta_k'$. First, by Cauchy-Schwartz's inequality, we have
%
\begin{equation*}
\norm{\pi \spr*{\cdot \given x} - \pi' \spr*{\cdot \given x}}_1 \leq \sqrt{A} \norm{\pi \spr*{\cdot \given x} - \pi' \spr*{\cdot \given x}}\,.
\end{equation*}
%
By $1$-Lipshitzness of the softmax function (Lemma~\ref{lem:softmax-lipschitz}), it holds that
%
\begin{align*}
\norm{\pi \spr*{\cdot \given x} - \pi' \spr*{\cdot \given x}}_1 &\leq \eta \sqrt{A} \norm{\inp{\varphi \spr*{x, \cdot}, \bar{\theta}_K - \bar{\theta}_K'}} \\
&= \eta \sqrt{A \sum_{a \in \cA} \spr*{\inp{\varphi \spr*{x, a}, \bar{\theta}_K - \bar{\theta}_K'}}^2} \\
&\leq \eta \sqrt{A \sum_{a \in \cA} \norm{\varphi \spr*{x, a}}^2 \norm{\bar{\theta}_K - \bar{\theta}_K'}^2} & \text{(Cauchy-Schwartz)} \\
&\leq \eta \phiMAX A \norm{\bar{\theta}_K - \bar{\theta}_K'} & \text{(Assumption~\ref{ass:linQ})} \\
&\leq \eta \phiMAX A \sumkK \norm{\theta_k - \theta_k'} & \text{(Triangle inequality)} \\
&\leq K \eta \phiMAX A \max_{k \in \sbr*{K}} \norm{\theta_k - \theta_k'}\,.
\end{align*}
%
Therefore, the $\epsilon$-covering number for $\Pilin$ with respect to the norm $\norm{\cdot}_1$, $\cN_\epsilon \spr*{\Pilin, \norm{\cdot}_1}$, is upper-bounded by the $\frac{\epsilon}{K \eta \phiMAX A}$-covering number of the Euclidean ball $\fkB \spr*{\ThetaMAX}$ with respect to the norm $\norm{\cdot}$, and
%
\begin{align*}
\cN_\epsilon \spr*{\Pilin, \norm{\cdot}_1} &\leq \cN_{\frac{\epsilon}{K \eta \phiMAX A}} \spr*{\fkB \spr*{\ThetaMAX}, \norm{\cdot}} \\
&\leq \spr*{1 + \frac{2 K \eta \ThetaMAX \phiMAX A}{\epsilon}}^d\,,
\end{align*}
%
where we used Lemma~\ref{lem:covering-ball} in the last inequality. For the second part of the lemma, let us consider $Q, Q' \in \Qlin$. By definition of $\Qlin$, there exists $\theta, \theta' \in \fkB \spr*{\ThetaMAX}$ such that for any state-action pair $\spr*{x, a}$, $Q \spr*{x, a} = \inp{\varphi \spr*{x, a}, \theta}$ and $Q' \spr*{x, a} = \inp{\varphi \spr*{x, a}, \theta'}$. Then,
%
\begin{equation*}
\max_{x, a \in \cX \times \cA} \abs{Q \spr*{x, a} - Q' \spr*{x, a}} = \max_{x, a \in \cX \times \cA} \abs{\inp{\varphi \spr*{x, a}, \theta - \theta'}} \leq \phiMAX \norm{\theta - \theta'}\,.
\end{equation*}
%
Therefore, the $\epsilon$-covering number of $\Qlin$, $\cN_\epsilon \spr*{\Qlin, \norm{\cdot}_\infty}$, is upper-bounded by the $\epsilon / \phiMAX$-covering number of the
$d$-dimensional ball with radius $\ThetaMAX$, $\cN_{\epsilon / \phiMAX} \spr*{\fkB \spr*{\ThetaMAX}, \norm{\cdot}}$. We have
%
\begin{equation*}
\cN_\epsilon \spr*{\Qlin, \norm{\cdot}_\infty} \leq \cN_{\epsilon / \phiMAX} \spr*{\fkB \spr*{\ThetaMAX}, \norm{\cdot}} \leq \spr*{1 + \frac{2 \ThetaMAX \phiMAX}{\epsilon}}^d\,.
\end{equation*}
%
Finally, the proof is concluded by noting that
%
\begin{equation*}
\cN_\epsilon \spr*{\Qlin \times \Pilin, \norm{\cdot}_{\infty, 1}} \leq \cN_{\epsilon / 2} \spr*{\Pilin, \norm{\cdot}_1} \cN_{\epsilon / 2} \spr*{\Qlin, \norm{\cdot}_\infty}\,.
\end{equation*}
\end{proof}
Finally, the following result proves the concentration of the estimators used in Algorithm~\ref{alg:main}.
\concentration*
\begin{proof}
By Lemma~\ref{lemma:unif_conc}, it holds that for all $k\in[K]$
%
\begin{align*}
\Delta \spr*{\pi_k} &\leq \inf_{\epsilon: \epsilon > 0} \scbr*{\frac{4 \epsilon}{1 - \gamma} + 2 \sqrt{\frac{2 \log \spr*{2 \cN_\epsilon \spr*{\Qlin \times \Pilin, \norm{\cdot}_{\infty, 1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}} \\
&\leq \frac{1}{K} + 2 \sqrt{\frac{2 \log \spr*{2 \cN_{\spr*{1 - \gamma} / 4 K} \spr*{\Qlin \times \Pilin, \norm{\cdot}_{\infty, 1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}} \\
&\leq \frac{1}{K} + 2 \sqrt{\frac{2}{\spr*{1 - \gamma}^2 \tauE} \log \spr*{\frac2\delta \spr*{1 + \frac{16 K^2 \eta \ThetaMAX \phiMAX A}{1 - \gamma}}^{2 d}}} \\
&\leq \frac{1}{K} + 4 \sqrt{\frac{d}{\spr*{1 - \gamma}^2 \tauE} \log \spr*{\frac{2 + 32 K^2 \eta \ThetaMAX \phiMAX A}{\spr*{1 - \gamma} \delta}}}\,,
\end{align*}
%
where the third inequality follows from Lemma~\ref{lemma:covering}.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proof of Theorem~\ref{thm:main_formal} (sample complexity guarantee for linear \texorpdfstring{$Q^\pi$}{Q-pi}-realizable MDPs)} \label{app:proof_main_formal}
\linQresult*
\begin{proof}
By Proposition~\ref{prop:decomp}, we have
%
\begin{equation*}
\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} \leq \frac1K \sumkK \bbE \sbr*{\Loss \spr*{\pi_k; Q_k}} + \frac2K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}}\,.
\end{equation*}
%
Using Lemma~\ref{lemma:mirror} with a learning rate of $\eta = \spr*{1 - \gamma} \sqrt{\frac{2 \log A}{K}}$ and dividing by $K$, we obtain that
%
\begin{equation*}
\frac1K \sumkK \Loss \spr*{\pi_k; Q_k} \leq \sqrt{\frac{2 \log A}{\spr*{1 - \gamma}^2 K}}\,.
\end{equation*}
%
Therefore, setting $K = \frac{2 \log A}{\spr*{1 - \gamma}^2 \varepsilon^2}$ guarantees $\frac1K \sumkK \Loss \spr*{\pi_k; Q_k} \leq \varepsilon$. Then, using the high-probability bound in Lemma~\ref{lemma:vec_concentration} and the fact that $\frac1K \sumkK \Delta \spr*{\pi_k}$ is a random random variable bounded by $2 \spr*{1 - \gamma}^{-1}$ almost surely, we obtain the following expectation bound which holds for all $\delta > 0$,
%
\begin{align*}
\frac1K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}} &\leq \frac1K + C \sqrt{\frac{d}{\spr*{1 - \gamma}^2 \tauE} \log \spr*{\frac{ \ThetaMAX \phiMAX A}{\spr*{1 - \gamma} \delta \varepsilon}}} + \frac{2 \delta}{1 - \gamma}\,,
\end{align*}
%
for some $C \in \bbR$. Note that the choice of parameters ensures $\frac1K \leq \frac{\varepsilon}{2}$. Setting $\delta = \frac{\varepsilon \spr*{1 - \gamma}}{4}$ and
%
\begin{equation*}
\tauE \geq \frac{C^2 d}{\spr*{1 - \gamma}^2 \varepsilon^2} \log \spr*{\frac{\ThetaMAX \phiMAX A}{\spr*{1 - \gamma} \delta \varepsilon}}\,
\end{equation*}
%
this bound implies that $\frac2K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}} \leq 4 \varepsilon$. Thus, we conclude that $\mathbb{E}\bigl[\rho^{\expert} - \rho^{\piout}\bigr] \leq 5 \varepsilon$.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proof of Lemma~\ref{lemma:concentration-nonlinear} (concentration nonlinear case)} \label{app:proof_concentration_nonlinear}
Before presenting the proof of Theorem~\ref{thm:mainnonlinear}, we provide a bound on the covering number of the class $\cQ \times \Pi_\cQ$, where $\Pi_\cQ$ is defined in Equation~\eqref{eq:PiQ}. It turns out that the covering number of this class is exponential in $K$. In the linear case, the exponential dependence in $K$ was avoided because the state-action value class is closed under addition.
%
\begin{Lem}[Covering number of $\Pi_\cQ$] \label{lemma:covering_non_linear}
It holds that the covering number of the policy class $\Pi_{\mathcal{Q}}$ can be bounded as
%
\begin{equation*}
\cN_\epsilon \spr*{\Pi_\cQ, \norm{\cdot}_1} \leq \cN_{\frac{\epsilon}{K \eta A}} \spr*{\cQ, \norm{\cdot}_\infty}^K\,,
\end{equation*}
%
where, with a slight abuse of notation, $\norm{\cdot}_1$ denotes the norm defined for any $\pi \in \Pi_\cQ$ as $\norm{\pi}_1 = \sup_{x \in \cX} \norm{\pi \spr*{\cdot \given x}}_1$. Moreover,
\begin{equation*}
\cN_\epsilon \spr*{\cQ \times \Pi_\cQ, \norm{\cdot}_{\infty, 1}} \leq \cN_{\frac{\epsilon}{K \eta A}} \spr*{\cQ, \norm{\cdot}_\infty}^{K+1}\,.
\end{equation*}
\end{Lem}
\begin{proof}
Let us consider two policies $\pi$ and $\pi'$ in the class $\Pi_\cQ$. There exist $Q_1, \dots, Q_K \in \cQ$ and $Q_1', \dots, Q_K' \in \cQ$ such that for any state-action pair $\spr*{x, a} \in \cX \times \cA$, $\pi$ and $\pi'$ can be written as
%
\begin{equation*}
\pi \spr*{a \given x} = \frac{\exp \spr*{\eta \sumkK Q_k \spr*{x, a}}}{\sum_{b \in \cA} \exp \spr*{\eta \sumkK Q_k \spr*{x, b}}}\,,
\end{equation*}
%
and
%
\begin{equation*}
\pi' \spr*{a \given x} = \frac{\exp \spr*{\eta \sumkK Q_k' \spr*{x, a}}}{\sum_{b \in \cA} \exp \spr*{\eta \sumkK Q_k' \spr*{x, b}}}\,.
\end{equation*}
%
Let $x \in \cX$. Using $\norm{\cdot}_1 \leq \sqrt{A} \norm{\cdot}$ in $\bbR^A$ and by $1$-Lipshitzness of the softmax function (Lemma~\ref{lem:softmax-lipschitz}), it holds that
%
\begin{align*}
\norm{\pi \spr*{\cdot \given x} - \pi' \spr*{\cdot \given x}}_1 &\leq \sqrt{A} \norm{\pi \spr*{\cdot \given x} - \pi' \spr*{\cdot \given x}} \\
&\leq \eta \sqrt{A} \norm{\sumkK \spr*{Q_k \spr*{x, \cdot} - Q_k' \spr*{x, \cdot}}} \\
&\leq \eta \sqrt{A} \sumkK \norm{Q_k \spr*{x, \cdot} - Q_k' \spr*{x, \cdot}} & \text{(Triangle inequality)} \\
&\leq \eta A \sumkK \sup_{a \in \cA} \abs{Q_k \spr*{x, a} - Q_k' \spr*{x, a}} & (\norm{\cdot} \leq \sqrt{A} \norm{\cdot}_\infty) \\
&\leq \eta A \sup_{x \in \cX} \scbr*{\sumkK \sup_{a \in \cA} \abs{Q_k \spr*{x, a} - Q_k' \spr*{x, a}}} \\
&\leq \eta A \sumkK \norm{Q_k - Q_k'}_\infty & \text{(Triangle inequality)}\,.
\end{align*}
%
In particular, this implies
%
\begin{align*}
\max_{x \in \cX} \norm{\pi \spr*{\cdot \given x} - \pi' \spr*{\cdot \given x}}_1 \leq \eta A \sumkK \norm{Q_k' - Q_k }_\infty\,.
\end{align*}
%
Thus, the $\epsilon$-covering number for $\Pi_\cQ$, $\cN_\epsilon \spr*{\Pi_\cQ, \norm{\cdot}_1}$, is upper-bounded by the $\frac{\epsilon}{K \eta A}$-covering number of the class $\cQ$ to the power $K$, \ie, $\cN_{\frac{\epsilon}{K \eta A}} \spr*{\cQ, \norm{\cdot}_{\infty}}^K$. Thus,
%
\begin{equation*}
\cN_\epsilon \spr*{\Pi_\cQ, \norm{\cdot}_1} \leq \cN_{\frac{\epsilon}{K \eta A}} \spr*{\cQ, \norm{\cdot}_\infty}^K\,.
\end{equation*}
%
The proof is concluded by noting that the covering number increases with the precision (when $\epsilon$ decreases), and therefore, we can write
%
\begin{align*}
\cN_\epsilon \spr*{\cQ \times \Pi_\cQ, \norm{\cdot}_{\infty, 1}} &\leq \cN_{\epsilon / 2} \spr*{\cQ, \norm{\cdot}_\infty} \cN_{\epsilon / 2} \spr*{\Pi_\cQ, \norm{\cdot}_1} \\
&\leq \cN_{\epsilon / 2} \spr*{\cQ, \norm{\cdot}_\infty} \cN_{\frac{\epsilon}{K \eta A}} \spr*{\cQ, \norm{\cdot}_\infty}^K \\
&\leq \cN_{\frac{\epsilon}{K \eta A}} \spr*{\cQ, \norm{\cdot}_\infty}^{K+1}\,.
\end{align*}
\end{proof}
Finally, the following result proves the concentration of the estimators used in Algorithm~\ref{alg:spoil-general-fa}.
%
\begin{restatable}{Lem}{concnonlinear} \label{lemma:technonlinear}
Let $\scbr*{\pi_k}_{k \in \sbr*{K}}$ be the sequence of policies generated by Algorithm~\ref{alg:spoil-general-fa}. Then, with probability at least $1 - \delta$, for any $k \in \sbr*{K}$, it holds that
%
\begin{equation*}
\sup_{Q \in \cQ} \abs{\hLoss \spr*{\pi_k, Q} - \Loss \spr*{\pi_k, Q}} \leq \frac{1}{2 K} + \sqrt{\frac{2 \spr*{K + 1} \log \spr*{2 \cN_{\spr*{1 - \gamma} / 8 K} \spr*{\cQ, \norm{\cdot}_\infty} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}\,.
\end{equation*}
\end{restatable}
\begin{proof}
Note that by construction, the policy sequence $\scbr*{\pi_k}_{k \in \sbr*{K}}$ generated by Algorithm~\ref{alg:spoil-general-fa} belongs to the policy class $\Pi_\cQ$. Therefore, invoking Lemma~\ref{lemma:unif_conc}, we have that with probability at least $1 - \delta$, for any $k \in \sbr*{K}$, it holds that
%
\begin{equation*}
\Delta \spr*{\pi_k} \leq \inf_{\epsilon: \epsilon > 0} \scbr*{\frac{4 \epsilon}{1 - \gamma} + \sqrt{\frac{2 \log \spr*{2 \cN_\epsilon \spr*{\cQ \times \Pi_\cQ, \norm{\cdot}_{\infty, 1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}}\,.
\end{equation*}
%
Therefore, choosing $\epsilon = \frac{1 - \gamma}{8 K}$, we get
%
\begin{align*}
\Delta \spr*{\pi_k} &\leq \frac{1}{2 K} + \sqrt{\frac{2 \log \spr*{2 \cN_{\spr*{1 - \gamma} / 8 K} \spr*{\cQ \times \Pi_\cQ, \norm{\cdot}_{\infty, 1}} / \delta}}{\spr*{1 - \gamma}^2 \tauE}} \\
\\ & \leq \frac{1}{2 K} + \sqrt{\frac{2 \spr*{K + 1} \log \spr*{2 \cN_{\frac{1 - \gamma}{8 K^2 \eta A}} \spr*{\cQ, \norm{\cdot}_\infty} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}\,.
\end{align*}
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proof of Theorem~\ref{thm:mainnonlinear} (sample complexity guarantee for \texorpdfstring{$Q^\pi$}{Q-pi}-realizable MDPs)} \label{app:proof_main_nonlinear}
We are now ready for the proof of Theorem~\ref{thm:mainnonlinear}, which we restate for convenience.
\mainnonlinear*
\begin{proof}
Recall that by Proposition~\ref{prop:decomp}, we have
%
\begin{equation*}
\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} \le \frac1K \sumkK \bbE \sbr*{\Loss \spr*{\pi_k; Q_k}} + \frac2K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}}\,.
\end{equation*}
%
Then, by Lemma~\ref{lemma:mirror}, it holds that
%
\begin{equation*}
\frac1K \sumkK \bbE \sbr*{\Loss \spr*{\pi_k; Q_k}} \leq \frac{\log \spr*{A}}{\eta K} + \frac{\eta}{\spr*{1 - \gamma}^2}\,.
\end{equation*}
%
Moreover, by Lemma~\ref{lemma:technonlinear}, with probability at least $1 - \delta$, it holds that
%
\begin{align*}
2 \sumkK \Delta \spr*{\pi_k} \leq 1 + 2 K \sqrt{\frac{2 \spr*{K + 1} \log \spr*{2 \cN_{\frac{1 - \gamma}{8 K^2 \eta A}} \spr*{\cQ, \norm{\cdot}_\infty} / \delta}}{\spr*{1 - \gamma}^2 \tauE}}\,.
\end{align*}
%
Since $\frac1K \sumkK \Delta \spr*{\pi_k}$ is bounded almost surely by $4 \spr*{1 - \gamma}^{-1}$, we have that for any $\delta > 0$
%
\begin{align*}
\frac2K \sumkK \bbE \sbr*{\Delta \spr*{\pi_k}} \leq \frac1K + 2 \sqrt{\frac{2 \spr*{K + 1} \log \spr*{2 \cN_{\frac{1 - \gamma}{8 K^2 \eta A}} \spr*{\cQ, \norm{\cdot}_\infty} / \delta}}{\spr*{1 - \gamma}^2 \tauE}} + \frac{4 \delta}{1 - \gamma}\,.
\end{align*}
%
Setting $\eta = \spr*{1 - \gamma} \sqrt{2 \log \spr*{A} / K}$, we get
%
\begin{equation*}
\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} \leq \sqrt{\frac{2 \log \spr*{A}}{\spr*{1 - \gamma}^2 K}} + \frac1K + 2 \sqrt{\frac{2 \spr*{K + 1} \log \spr*{2 \cN_{\epsilon'} \spr*{\cQ, \norm{\cdot}_\infty} / \delta}}{\spr*{1 - \gamma}^2 \tauE}} + \frac{4 \delta}{1 - \gamma}\,,
\end{equation*}
%
where we denoted $\epsilon' = \frac{1}{\sqrt{2} 8 K^{3 / 2} \sqrt{\log \spr*{A}} A}$. Setting $\delta = \frac{\spr*{1 - \gamma} \varepsilon}{4}$ and $K = \frac{2 \log A}{\spr*{1 - \gamma}^2 \varepsilon^2}$, we further have
%
\begin{equation*}
\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} \leq \varepsilon + \frac{\varepsilon}{2} + 4 \sqrt{\frac{\log \spr*{A}}{\spr*{1 - \gamma}^4 \varepsilon^2 \tauE} \log \spr*{\frac{2 \cN_{\epsilon'} \spr*{\cQ, \norm{\cdot}_\infty}}{\delta}}}\,.
\end{equation*}
%
Finally, setting
%
\begin{equation*}
\tauE \geq \frac{16 \log \spr*{A}}{\spr*{1 - \gamma}^4 \varepsilon^4} \log \spr*{\frac{2 \cN_{\epsilon'} \spr*{\cQ, \norm{\cdot}_\infty}}{\delta}}\,,
\end{equation*}
%
we guarantee that
%
\begin{equation*}
\bbE \sbr*{\rho^{\expert} - \rho^{\piout}} = \cO \spr*{\varepsilon}\,.
\end{equation*}
\end{proof}
\section{Technical tools}
\label{app:techtools}
\begin{Lem}[Hoeffding's inequality] \label{lemma:hoeffding}
Let $X_1, \ldots, X_n$ be independent random variables such that $\abs{X_i} \leq M$ for all $i$. Then, for any $\epsilon > 0$,
%
\begin{equation*}
\bbP \spr*{\abs{\frac1n \sum^n_{i=1} \pa{X_i - \bbE \spr*{X_i}}} > \epsilon} \leq 2 e^{-\frac{2 n
\epsilon^2}{M^2}}\,.
\end{equation*}
\end{Lem}
%
\begin{Lem}[Simplified version of \citealp{orabona2023modern}, Theorem~6.10] \label{lemma:mirror_orabona}
Let us consider a non-empty closed convex set $V$, an arbitrary sequence of adaptively chosen loss vectors $\spr*{\ell_k}^K_{k=1}$ such that $\norm{\ell_k}_{\infty} \leq \ell_{\max}$, and let $D: V \times \mathrm{int} \spr*{V} \rightarrow \bbR$ be a Bregman divergence induced by a $\lambda$-strongly convex function in the $\ell_1$-norm. Then, for all $u \in V$, the sequence $\spr*{x_k}^K_{k=1}$ generated for any $k$ as
%
\begin{equation*}
x_{k + 1} = \argmin_{v \in V} \scbr*{\inp{\ell_k, v} + \frac1\eta D \spr*{v, x_k}}
\end{equation*}
%
for an arbitrary initial $x_1$ satisfies
%
\begin{equation*}
\sum^K_{k=1} \inp{\ell_k, x_k - u} \leq \frac{D \spr*{u, x_1}}{\eta} + \frac{\eta K \ell^2_{\max}}{2 \lambda}\,.
\end{equation*}
\end{Lem}
%
\begin{Lem}[\citealp{gao2017properties}, Proposition~4] \label{lem:softmax-lipschitz}
For any $\eta > 0$, let the softmax function be defined for any $z \in \bbR^n$ as
%
\begin{equation*}
\softmax \spr*{z} = \spr*{\frac{e^{\eta z_i}}{\sum^n_{j=1} e^{\eta z_j}}}_{i \in \sbr*{n}}\,.
\end{equation*}
%
Then, the softmax function is $\eta$-Lipschitz with respect to $\norm{\cdot}_2$. That is, for any $z, z' \in \bbR^n$, we have
%
\begin{equation*}
\norm{\softmax \spr*{z} - \softmax \spr*{z'}}_2 \leq \eta \norm{z - z'}_2\,.
\end{equation*}
\end{Lem}
\begin{Lem}[Covering number of a Euclidean ball] \label{lem:covering-ball}
The covering number of the Euclidean ball of radius $R$ in $\bbR^d$, $\fkB \spr*{R}$, is
\begin{equation*}
\cN_\epsilon \spr*{\fkB \spr*{R}, \norm{\cdot}} \leq \spr*{1 + \frac{2 R}{\epsilon}}^d\,.
\end{equation*}
\end{Lem}
\section{On the guarantees of misspecified \BC in linear \texorpdfstring{$Q^\pi$}{Q-pi}-realizable MDPs}
It is natural to question whether existing bounds for behavioral cloning (\BC) in misspecified settings
\citep[e.g.,][]{rohatgi2025computational,foster2024behavior} offer satisfactory sample complexity guarantees for
imitating an arbitrarily complex expert within a linear $Q^\pi$-realizable MDP.
This section presents a negative result, demonstrating that the approximation error incurred by \BC, when restricted to a linear softmax policy class (denoted $\Pi_{\text{lin}}$), can be large even in a simple linear $Q^\pi$-realizable MDP.
Consider a single-state MDP defined as follows. Let $A \in \bbN^*$ be the number of actions, with the action space $\cA = \sdbr*{1, \dots, A}$. For each action $a \in \cA$, there is a scalar feature $\phi \spr*{a} = - \frac{A}{2} + a \in \bbR$. To ensure the MDP is linear $Q^\pi$-realizable, the true reward function is $r_{\text{true}} \spr*{a} = \zeta \phi\spr*{a}$ for some parameter $\zeta \in \bbR$ unknown to the learner. We define a \emph{softmax quadratic} expert policy $\expert$ as
%
\begin{equation*}
\expert \spr*{a} = \frac{\exp \spr*{\phi \spr*{a}^2}}{\sum_{b \in \cA} \exp \spr*{\phi \spr*{b}^2}}\,.
\end{equation*}
%
This expert policy assigns the highest probability to extremal actions (\ie, $a = 1$ and $a = A$). In contrast, linear softmax policies $\pi \in \Pi_{\text{lin}}$ (which are commonly used for \BC in feature-based settings) are inherently designed to produce monotonic probability distributions over the action space when features are ordered (\ie, for actions $a, a' \in \cA$ with $a' > a$, either $\pi\spr*{a} \leq \pi\spr*{a'}$ or $\pi\spr*{a} \geq \pi\spr*{a'}$). Consequently, for $A > 2$, no policy in $\Pi_{\text{lin}}$ can achieve a small Hellinger distance to this softmax quadratic expert. We illustrate this in Figure~\ref{fig:quadratic-vs-linear-softmax}, where we compare the softmax quadratic expert with two linear softmax policies. Due to the monotonicity constraint, the linear softmax policies are unable to approximate the expert policy everywhere.
It remains an open question whether behavioral cloning analyses can be refined to better leverage the underlying MDP structure in such misspecified scenarios. Specifically, for the constructed example, it would be advantageous if the misspecification error in existing bounds were characterized in terms of feature expectations (\eg, $\sum_{a \in \cA} \pi \spr*{a} \phi \spr*{a}$) rather than state-action distributions.
\begin{figure}
\centering
\caption{Comparison of linear and quadratic softmax policies with $A = 5$ actions and features $\phi(a) = a - 3$.}
\label{fig:quadratic-vs-linear-softmax}
\begin{tikzpicture}
\begin{axis}[
% title={Comparison of linear and quadratic softmax policies with $A = 5$ actions and features $\phi(a) = a - 3$.},
xlabel={Action ($a$)},
ylabel={Probability ($\pi \spr*{a}$)},
ymin=0,
ymax=0.7,
xtick={1,2,3,4,5},
xticklabels={$a_1$, $a_2$, $a_3$, $a_4$, $a_5$},
legend pos=north east,
legend style={font=\small},
legend columns=1,
enlarge x limits=0.12,
width=11cm,
height=7cm
]
% Quadratic softmax expert policy (A=5, phi(a) = a - 3, expert prop to exp(phi(a)^2))
% Probabilities: 0.4721, 0.0235, 0.0086, 0.0235, 0.4721
\addplot+[black, mark=*, mark options={fill=black}] coordinates {
(1, 0.4721)
(2, 0.0235)
(3, 0.0086)
(4, 0.0235)
(5, 0.4721)
};
\addlegendentry{$\expert \spr*{a} \propto e^{\spr*{a - 3}^2}$}
% Linear softmax policy (A=5, theta = 1, phi(a) = a - 3, prop to exp(1 * phi(a)))
% Probabilities: 0.0117, 0.0317, 0.0861, 0.2341, 0.6364
\addplot+[black, mark=square*, mark options={fill=black}, dashed] coordinates {
(1, 0.0117)
(2, 0.0317)
(3, 0.0861)
(4, 0.2341)
(5, 0.6364)
};
\addlegendentry{$\pi_{\mathrm{lin}, 1} \spr*{a} \propto e^{1 \cdot \spr*{a - 3}}$}
% Linear softmax policy (A=5, theta = 1, phi(a) = a - 3, prop to exp(1 * phi(a)))
% Probabilities: 0.6364, 0.2341, 0.0861, 0.0317, 0.0117
\addplot+[black, mark=triangle*, mark options={fill=black}, dotted] coordinates {
(1, 0.6364)
(2, 0.2341)
(3, 0.0861)
(4, 0.0317)
(5, 0.0117)
};
\addlegendentry{$\pi_{\mathrm{lin}, -1} \spr*{a} \propto e^{-1 \cdot \spr*{a - 3}}$}
\end{axis}
\end{tikzpicture}
\end{figure}
\section{Experimental details}
\label{sec:experimental-details}
For the first experiment shown in Figure~\ref{exp:first}, one may wonder if the underperformance of behavioural cloning might be due to underoptimizing the empirical log-likehood. We have ruled out this possibility by going into great lengths to optimize the likelihood, and in fact the log-likelihood has approached its minimum value of zero very closely in our experiment (meaning that the probability assigned to the actions seen in the expert dataset is almost $1$). For this optimization task, we have used Adam with default parameter settings. For the experiments in Figure~\ref{fig:deep_pdoil}, algorithms are implemented using a shared neural network architecture consisting of 3 layers with 64 neurons per layer. This architecture matches the one used for experiments in the same environments by \citet{Garg:2021}. For behavioral cloning, we employ a separate three-layer multilayer perceptron with 128 neurons per layer. Implementations of \IQLearn and \PIL utilize their original hyperparameter configurations as reported in their respective publications. All networks are optimized using the Adam optimizer \citep{kingma2014adam} with a learning rate of $5 \times 10^{-3}$ and default momentum parameters ($\beta_1=0.9, \beta_2=0.999$). The implementations are built using PyTorch \citep{paszke2019pytorch}.
For algorithms with a primal-dual structure (\ie, \IQLearn, \PIL, and \SPOIL), the policy update is performed using a \texttt{Soft\,DQN}-style update \citep[cf.][]{haarnoja2017reinforcement} with a fixed temperature parameter. These three algorithms thus only differ in terms of their Q-value updates, and thus this experiment serves to assess the effectiveness of the novel critic loss introduced in this work.
\section{Future directions}
\label{app:future}
\paragraph{Possible improvements.} The most interesting immediate question that one can ask about our result is if the $\cO \spr*{\varepsilon^{-4}}$ scaling featured in our general bound is improvable under the conditions we assume. We believe that substantially different algorithmic and analytic ideas would be necessary to answer this question, but we also think that our primal-dual framework provides a good starting point towards making such improvements. Furthermore, we would be curious to investigate appropriate notions of misspecification that our algorithm can deal with. It can be easily shown that requiring $Q^\pi$-realizability only up to a worst-case additive error of order $\varepsilon_{\text{approx}}$ would incur the same additional term in the error bounds, but we believe that this assumption is too strong to warrant interest and we did not include an explicit statement. A much more interesting question is if this approximation guarantee would only be required to hold locally in the state-action pairs visited by the expert, or only for specific policies (most ideally only the expert policy). Given the numerous negative results in RL theory about such weaker function approximators, we are not optimistic that these latter improvements are possible, but nevertheless (and once again), we feel that our analytic framework can provide suitable tools for analyzing such questions.
\paragraph{Learning from features only.} In the case of linear function approximation, the current approach critically relies on observing the expert state-action pairs to compute the vectors $\scbr*{\hatgk}^K_{k=1}$. It would be interesting to check if an alternative algorithm can achieve the same guarantees by only observing the feature vectors instead. In other words, the design of an algorithm taking as input a dataset $\scbr*{\varphi \spr*{\XEi, \AEi}}^{\tauE}_{i=1}$ is an interesting open problem.
\paragraph{New efficient algorithms that learn state-action value functions from expert data.} Despite having proven successful in practice \citep{Garg:2021}, the idea of learning a state-action value function from expert data without passing through a learned reward function has not been used to develop theoretically grounded algorithms. Our work is the first example of an algorithm enjoying theoretical guarantees applying this principle. We expect this principle to find other applications in imitation learning theory, for example on the open problem of learning to imitate an expert from state-only trajectory given trajectory access to a linear-$Q^\pi$ realizable MDP.
\begin{thebibliography}{42}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abbasi-Yadkori et~al.(2019)Abbasi-Yadkori, Bartlett, Bhatia, Lazic, Szepesvari, and Weisz]{Abbasi-Yadkori:2019b}
Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gell{\'e}rt Weisz.
\newblock Politex: Regret bounds for policy iteration using expert prediction.
\newblock In \emph{International Conference on Machine Learning (ICML)}, 2019.
\bibitem[Abbeel and Ng(2004)]{Abbeel:2004}
P.~Abbeel and A.~Y. Ng.
\newblock Apprenticeship learning via inverse reinforcement learning.
\newblock In \emph{International Conference on Machine Learning (ICML)}, 2004.
\bibitem[Agarwal et~al.(2019)Agarwal, Jiang, Kakade, and Sun]{agarwal2019reinforcement}
Alekh Agarwal, Nan Jiang, Sham~M Kakade, and Wen Sun.
\newblock Reinforcement learning: Theory and algorithms.
\newblock \emph{CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep}, 32, 2019.
\bibitem[Block et~al.(2023)Block, Jadbabaie, Pfrommer, Simchowitz, and Tedrake]{block2023provable}
Adam Block, Ali Jadbabaie, Daniel Pfrommer, Max Simchowitz, and Russ Tedrake.
\newblock Provable guarantees for generative behavior cloning: Bridging low-level stability and high-level behavior.
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 48534--48547, 2023.
\bibitem[Cesa-Bianchi and Lugosi(2006)]{Cesa-Bianchi:2006}
Nicolo Cesa-Bianchi and G{\'a}bor Lugosi.
\newblock \emph{Prediction, learning, and games}.
\newblock Cambridge university press, 2006.
\bibitem[Espinosa-Dice et~al.(2025)Espinosa-Dice, Choudhury, Sun, and Swamy]{espinosa2025efficient}
Nicolas Espinosa-Dice, Sanjiban Choudhury, Wen Sun, and Gokul Swamy.
\newblock Efficient imitation under misspecification.
\newblock \emph{arXiv preprint arXiv:2503.13162}, 2025.
\bibitem[Even-Dar et~al.()Even-Dar, Kakade, and Mansour]{Even-dar:2008}
E.~Even-Dar, S.~M. Kakade, and Y.~Mansour.
\newblock Experts in a {M}arkov decision process.
\newblock In \emph{Neural Information Processing Systems}, pages 401--408.
\bibitem[Even-Dar et~al.(2009)Even-Dar, Kakade, and Mansour]{Even-dar:2009}
Eyal Even-Dar, Sham.~M. Kakade, and Yishay Mansour.
\newblock Online {M}arkov decision processes.
\newblock \emph{Mathematics of Operations Research}, 34\penalty0 (3):\penalty0 726--736, 2009.
\bibitem[Foster et~al.(2024)Foster, Block, and Misra]{foster2024behavior}
Dylan~J Foster, Adam Block, and Dipendra Misra.
\newblock Is behavior cloning all you need? understanding horizon in imitation learning.
\newblock \emph{arXiv preprint arXiv:2407.15007}, 2024.
\bibitem[Gao and Pavel(2017)]{gao2017properties}
Bolin Gao and Lacra Pavel.
\newblock On the properties of the softmax function with application in game theory and reinforcement learning.
\newblock \emph{arXiv preprint arXiv:1704.00805}, 2017.
\bibitem[Garg et~al.(2021)Garg, Chakraborty, Cundy, Song, and Ermon]{Garg:2021}
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon.
\newblock {IQ}-learn: Inverse soft-{Q} learning for imitation.
\newblock In \emph{Advances in Neural Information Processing Systems (NeuRIPS)}, 2021.
\bibitem[Geist et~al.(2019)Geist, Scherrer, and Pietquin]{Geist:2019}
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin.
\newblock {A Theory of Regularized {Markov} Decision Processes}.
\newblock In \emph{International Conference on Machine Learning (ICML)}, 2019.
\bibitem[Haarnoja et~al.(2017)Haarnoja, Tang, Abbeel, and Levine]{haarnoja2017reinforcement}
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.
\newblock Reinforcement learning with deep energy-based policies.
\newblock In \emph{Proceedings of the 34th International Conference on Machine Learning}, pages 1352--1361. PMLR, 2017.
\bibitem[Howard(1960)]{howard1960dynamic}
Ronald~A Howard.
\newblock Dynamic programming and {Markov} processes.
\newblock 1960.
\bibitem[Kakade and Langford(2002)]{kakade2002approximately}
Sham Kakade and John Langford.
\newblock Approximately optimal approximate reinforcement learning.
\newblock In \emph{Proceedings of the nineteenth international conference on machine learning}, pages 267--274, 2002.
\bibitem[Kingma and Ba(2014)]{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{arXiv preprint arXiv:1412.6980}, 2014.
\bibitem[Lan(2023)]{lan2023policy}
Guanghui Lan.
\newblock Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes.
\newblock \emph{Mathematical programming}, 198\penalty0 (1):\penalty0 1059--1106, 2023.
\bibitem[Liu et~al.(2022)Liu, Zhang, Fu, Yang, and Wang]{Liu:2022}
Zhihan Liu, Yufeng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang.
\newblock Learning from demonstration: Provably efficient adversarial policy imitation with linear function approximation.
\newblock In \emph{International Conference on Machine Learning (ICML)}, 2022.
\bibitem[Mhammedi(2024)]{mhammedi2024sample}
Zakaria Mhammedi.
\newblock Sample and oracle efficient reinforcement learning for mdps with linearly-realizable value functions.
\newblock \emph{arXiv preprint arXiv:2409.04840}, 2024.
\bibitem[Mnih et~al.(2015)Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie, Sadik, Antonoglou, King, Kumaran, Wierstra, Legg, and Hassabis]{Mnih:2015}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei~A. Rusu, Joel Veness, Marc~G. Bellemare, Alex Graves, Martin Riedmiller, Andreas~K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.
\newblock Human-level control through deep reinforcement learning.
\newblock \emph{Nature}, 518\penalty0 (7540):\penalty0 529--533, 2015.
\bibitem[Moulin and Neu(2023)]{moulin2023optimistic}
Antoine Moulin and Gergely Neu.
\newblock Optimistic planning by regularized dynamic programming.
\newblock In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, \emph{Proceedings of the 40th International Conference on Machine Learning}, volume 202 of \emph{Proceedings of Machine Learning Research}, pages 25337--25357. PMLR, 23--29 Jul 2023.
\newblock URL \url{https://proceedings.mlr.press/v202/moulin23a.html}.
\bibitem[Moulin et~al.(2025)Moulin, Neu, and Viano]{moulin2025optimistically}
Antoine Moulin, Gergely Neu, and Luca Viano.
\newblock Optimistically optimistic exploration for provably efficient infinite-horizon reinforcement and imitation learning.
\newblock \emph{arXiv preprint arXiv:2502.13900}, 2025.
\bibitem[Orabona(2023)]{orabona2023modern}
Francesco Orabona.
\newblock A modern introduction to online learning, 2023.
\bibitem[Paszke(2019)]{paszke2019pytorch}
A~Paszke.
\newblock Pytorch: An imperative style, high-performance deep learning library.
\newblock \emph{arXiv preprint arXiv:1912.01703}, 2019.
\bibitem[{Pomerleau}(1991)]{Pomerleau:1991}
D.~A. {Pomerleau}.
\newblock Efficient training of artificial neural networks for autonomous navigation.
\newblock \emph{Neural Computation}, 3\penalty0 (1):\penalty0 88--97, 1991.
\bibitem[Puterman(1994)]{Puterman:1994}
M.~L. Puterman.
\newblock \emph{{M}arkov {D}ecision {P}rocesses: Discrete Stochastic Dynamic Programming}.
\newblock John Wiley \& Sons, Inc., USA, 1st edition, 1994.
\bibitem[Rajaraman et~al.(2020)Rajaraman, Yang, Jiao, and Ramchandran]{rajaraman2020toward}
Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran.
\newblock Toward the fundamental limits of imitation learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 33:\penalty0 2914--2924, 2020.
\bibitem[Rajaraman et~al.(2021)Rajaraman, Han, Yang, Liu, Jiao, and Ramchandran]{rajaraman2021value}
Nived Rajaraman, Yanjun Han, Lin Yang, Jingbo Liu, Jiantao Jiao, and Kannan Ramchandran.
\newblock On the value of interaction and function approximation in imitation learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 1325--1336, 2021.
\bibitem[Rohatgi et~al.(2025)Rohatgi, Block, Huang, Krishnamurthy, and Foster]{rohatgi2025computational}
Dhruv Rohatgi, Adam Block, Audrey Huang, Akshay Krishnamurthy, and Dylan~J Foster.
\newblock Computational-statistical tradeoffs at the next-token prediction barrier: Autoregressive and imitation learning under misspecification.
\newblock \emph{arXiv preprint arXiv:2502.12465}, 2025.
\bibitem[Ross et~al.(2011)Ross, Gordon, and Bagnell]{Ross:2011}
S.~Ross, G.~Gordon, and D.~Bagnell.
\newblock A reduction of imitation learning and structured prediction to no-regret online learning.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics (AISTATS)}, 2011.
\bibitem[Ross and Bagnell(2010)]{Ross:2010}
St{\'e}phane Ross and Drew Bagnell.
\newblock Efficient reductions for imitation learning.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics (AISTATS)}, 2010.
\bibitem[Shani et~al.(2021)Shani, Zahavy, and Mannor]{Shani:2021}
Lior Shani, Tom Zahavy, and Shie Mannor.
\newblock Online apprenticeship learning.
\newblock \emph{arXiv:2102.06924}, 2021.
\bibitem[Simchowitz et~al.(2025)Simchowitz, Pfrommer, and Jadbabaie]{simchowitz2025pitfalls}
Max Simchowitz, Daniel Pfrommer, and Ali Jadbabaie.
\newblock The pitfalls of imitation learning when actions are continuous.
\newblock \emph{arXiv preprint arXiv:2503.09722}, 2025.
\bibitem[Swamy et~al.(2021)Swamy, Choudhury, Bagnell, and Wu]{swamy2021moments}
Gokul Swamy, Sanjiban Choudhury, J~Andrew Bagnell, and Steven Wu.
\newblock Of moments and matching: A game-theoretic framework for closing the imitation gap.
\newblock In \emph{International Conference on Machine Learning}, pages 10022--10032. PMLR, 2021.
\bibitem[Swamy et~al.(2022)Swamy, Rajaraman, Peng, Choudhury, Bagnell, Wu, Jiao, and Ramchandran]{swamy2022minimax}
Gokul Swamy, Nived Rajaraman, Matt Peng, Sanjiban Choudhury, J~Bagnell, Steven~Z Wu, Jiantao Jiao, and Kannan Ramchandran.
\newblock Minimax optimal online imitation learning via replay estimation.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 7077--7088, 2022.
\bibitem[Syed and Schapire(2007)]{Syed:2007}
U.~Syed and R.~E. Schapire.
\newblock A game-theoretic approach to apprenticeship learning.
\newblock In \emph{Advances in Neural Information Processing Systems (NeurIPS)}, 2007.
\bibitem[Tkachuk et~al.(2024)Tkachuk, Weisz, and Szepesvari]{tkachuk2024trajectory}
Volodymyr Tkachuk, Gell{\'e}rt Weisz, and Csaba Szepesvari.
\newblock Trajectory data suffices for statistically efficient learning in offline {RL} with linear \$q{\textasciicircum}{\textbackslash}pi\$-realizability and concentrability.
\newblock In \emph{The Thirty-eighth Annual Conference on Neural Information Processing Systems}, 2024.
\newblock URL \url{https://openreview.net/forum?id=TusuJSbRxm}.
\bibitem[Towers et~al.(2024)Towers, Kwiatkowski, Terry, Balis, De~Cola, Deleu, Goulao, Kallinteris, Krimmel, KG, et~al.]{towers2024gymnasium}
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John~U Balis, Gianluca De~Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et~al.
\newblock Gymnasium: A standard interface for reinforcement learning environments.
\newblock \emph{arXiv preprint arXiv:2407.17032}, 2024.
\bibitem[Viano et~al.(2022)Viano, Kamoutsi, Neu, Krawczuk, and Cevher]{viano2022proximal}
Luca Viano, Angeliki Kamoutsi, Gergely Neu, Igor Krawczuk, and Volkan Cevher.
\newblock Proximal point imitation learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 24309--24326, 2022.
\bibitem[Viano et~al.(2024)Viano, Skoulakis, and Cevher]{viano2024imitation}
Luca Viano, Stratis Skoulakis, and Volkan Cevher.
\newblock Imitation learning in discounted linear {MDP}s without exploration assumptions.
\newblock In \emph{Forty-first International Conference on Machine Learning}, 2024.
\newblock URL \url{https://openreview.net/forum?id=DChQpB4AJy}.
\bibitem[Weisz et~al.(2023)Weisz, Gy{\"o}rgy, and Szepesv{\'a}ri]{weisz2023online}
Gell{\'e}rt Weisz, Andr{\'a}s Gy{\"o}rgy, and Csaba Szepesv{\'a}ri.
\newblock Online rl in linearly $q^\pi$-realizable mdps is as easy as in linear mdps if you learn what to ignore.
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 59172--59205, 2023.
\bibitem[Xu et~al.(2023)Xu, Li, Yu, and Luo]{xu2023provably}
Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo.
\newblock Provably efficient adversarial imitation learning with unknown transitions.
\newblock \emph{arXiv preprint arXiv:2306.06563}, 2023.
\end{thebibliography}