Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
Extracted source text · arXiv 2412.10616
\section{Experiment Details}\label{appendix:experiments}
We consider a linear contextual bandit problem, where we set $\mathcal{X} = \mathbb{R}^2$ and $|\mathcal{Y}| = 50$. For each $(x, y)$ pair, the ground truth reward is given by $r^\star(x, y) = \langle \phi(x, y), \theta^\star \rangle$, where $\theta^\star \in \mathbb{R}^{10}$ is randomly sampled from $\mathcal{U}([0, 1])$, and the feature vector $\phi(x, y)$ is the output of the hidden layer of a fixed two-layer MLP, with the input given by the concatenation of $x$ and the one-hot encoding of $y$. The activation function is set to $\tanh$. The context vector $x$ is drawn from standard normal distribution.
We focus on log-linear policy class $\pi_\theta(\cdot|x) = \rm{softmax}(\langle \theta, \phi(x, \cdot) \rangle)$, and set $\pi_{\text{ref}} = \pi_{\theta_{\text{ref}}}$ with $\theta_{\text{ref}}(x, y)$ sampled i.i.d. from $\mathcal{U}([0, 1])$. We use mini-batch samples of size 5 in every iteration. We approximately solve the optimization problems by performing 20 AdamW optimization steps with learning rate 0.01 and weight decay rate 0.01 in every iteration for the online setting and 1000 steps for the offline setting.
\section{Lower Bound Proofs}
\label{app:lbproofs}
In this section, we will analyze the proofs of \cref{thm:lbon} and \cref{thm:lboff}. Our proofs rely on a key observation that the RLHF problem, as described in \cref{sec:prelim} can essentially be seen as a BTL-based dueling bandit (DB) \add{cite DB} setting \citep{rc,bengs2022stochastic}, both for the online and offline problem. For completeness, we first describe the preference model below:
\textbf{BTL-based Pairwise Preference (Dueling) Model:}
Consider a decision space $\cD$. Each action/item $\a \in \cD$ is associated to a linear score/reward $s(\a) \in \R$. The probability of action $\a$ preferred over action $\b$ is given by:
\[
P(\a \succ \b) = \big(\sigma\big(s(\a) - s(\b)\big)\big),
\]
where $\sigma(\cdot)$ being the sigmoid transformation given by $\sigma(x) = \frac{1}{1+e^{-x}}$ for any $x \in \R$. Note the above preference model exactly boils down to the BTL model-based preferences described in \cref{eq:btl} in \cref{sec:prelim}. We will denote this preference model as BTL-DB.
This essentially establishes the connection between the RLHF feedback model and BTL-based preference model BTL-DB (note that the trajectories $(\tau)$ lie in the action space $\cD$ for the DB problem). Further note that the reward/ score function $s(\cdot)$ essentially corresponds to the $J(\cdot)$ function of \cref{eq:KLobj} (for $\beta =0$). We next establish a connection between two well-studied feedback models in learning theory: \emph{linear score BTL-DB} feedback and linear bandits (\linb) feedback. We first describe the two feedback models below:
\paragraph{Linear score based BTL-DB (\lindb) Feedback model.}
In addition to the modeling assumptions described above for BTL-DB, we further assume $\cD \subset \R^d$ and the underlying score/reward function $(s)$ is linear, i.e. $s(\a) = \nangle{\a, \w}$, where $\w \in \R^d$ is an unknown direction in $\R^d$. We will also denote by $\a^* := \arg\max_{\a \in \cD}s(\a)$ the action with the highest score.
The feedback model outputs a $1$-bit binary feedback $o \sim \text{Ber}(P(\a \succ \b))$ upon receiving any pair of actions $(\a,\b) \in \cD\times \cD$ -- we will use the abbreviation \lindb\ for this feedback model.
\paragraph{Linear Bandits (\linb) Feedback Model.}
Similar to the \lindb\ setting above, we again assume a decision space $\cD\subset \R^d$ with each arm $\a \in \cD$ being associated to an underlying linear score/reward function $s(\a) = \nangle{\a, \w}$, $\w \in \R^d$ is unknown. As oppose to the preference feedback observed in the \lindb\ case, in this feedback model one gets to observe a noisy feedback of the true reward of any queried arm $\a \in \cD$ given by $r(\a) = s(\a) + \eta$, where $\eta$ is usually a $0$ mean noise. For example, $\eta \sim \text{Gaussian}(0,1)$ etc. We will use the abbreviation \linb\ for this feedback model.
\subsection{Simulating \lindb\ feedback with \linb\ Feedback.}
\label{sec:red}
Based on the reduction idea of \cite[Lemma 8]{saha2021optimal}, we know that one can always simulate $1$ unit of \lindb\ pairwise preference feedback from $2$ units of \linb\ reward/score feedback. Formally the claim is given by:
\begin{lemma}
\label{lem:red}
Given any two actions $\a$ and $\b \in \cD$ if $r(\a) = s(\a) + \eta_1$ and $r(\b) = s(\b) + \eta_2$ are two (noisy) \linb\ feedback, s.t. $\eta_1, \eta_2 \overset{\text{iid}}{\sim} \text{Gumbel(0,1)}$ are iid draws from Gumbel$(0,1)$ distribution, then the binary outcome $o = \mathbf 1(r(\a) > r(\b)) \sim P(\a \succ \b) $ follows \lindb \ feedback.
\end{lemma}
Interested readers are encouraged to go over the proof of Lemma 9 of \cite{saha2021optimal} to see the proof of \cref{lem:red} above.
%Sample complexity can be at least twice
Given the above result, we are now ready to prove the lower bound results of \cref{thm:lbon} and \cref{thm:lboff} as discussed in the following two subsections. We would first define the online and offline versions of the best-arm identification problem (BAI) with \lindb\ feedback.
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.43\textwidth]{lb_red.png}
\caption{Simulating \lindb\ feedback with \linb\ feedback}
\label{fig:red}
\end{center}
\end{figure}
\subsection{Proof of \cref{thm:lbon}}
\label{app:lbon}
The \emph{online version of BAI problem with \lindb\ feedback} is an active sequential decision-making process where at each round $t$ the learner (i.e. the algorithm) is supposed to play a pair of actions $(\a_t,\b_t) \in \cD \times \cD$, upon which it gets to see a binary preference feedback $o_t \sim P(\a_t \succ \b_t)$ is the preference feedback of the pair $(\a_t,\b_t)$ generated according to the \lindb\ feedback model. Given a horizon of $T$ rounds, the objective of the learner is to output an arm $\a_T \in \cD$ such that $\E[r(\a_T)] = s(\a_t)$ is maximized or the difference $s(\a^*) - s(\a_T)$ is minimized. Let us denote this problem as \emph{Online-BAI-}\lindb\ problem.
Similarly, the \emph{online version of BAI problem with \linb\ feedback} is an active sequential decision-making process where at each round $t$ the learner (i.e. the algorithm) is supposed to play an action $\a_t \in \cD$, upon which it gets to see a real-valued reward/score feedback $r_t$ generated according to the \linb\ feedback model. Given a horizon of $T$ rounds, the objective of the learner is to output an arm $\a_T \in \cD$ such that $\E[r(\a_T)] = s(\a_t)$ is maximized or the difference $s(\a^*) - s(\a_T)$ is minimized. Let us denote this problem as \emph{Online-BAI-}\linb\ problem.
\textbf{Key Idea: Reducing \emph{Online-BAI-}\linb\ to \emph{Online-BAI-}\lindb.} Owing to \cref{lem:red}, it is easy to see that one could reduce the \emph{Online-BAI-}\linb\ problem to \emph{Online-BAI-}\lindb\ problem, i.e. given any algorithm $\cA^{\lindb}$ for the latter problem, one can use it to solve the former. We described the idea in \cref{alg:red} below to construct $\cA^{\linb}$---an \emph{Online-BAI-}\linb\ algorithm using $\cA^{\lindb}$. %Interested readers are encouraged go through the reduction ideas in \cite[Algorithm 2]{saha2021optimal} for more details.
\vspace{-5pt}
\begin{center}
\begin{algorithm}[H]
\caption{Simulating $\cA^{\linb}$ using $\cA^{\lindb}$ } %(using $\cA^{cdb}$)
\label{alg:red}
\begin{algorithmic}[1]
\STATE \textbf{Input:} Time horizon $T$.
\FOR {$t = 1, 2, \ldots \lceil\frac{T}{2}\rceil$}
\STATE Receive: $(\a_t,\b_t) \leftarrow$ duel played by $\cA^{\lindb}$ at time $t$.
\STATE Play $\a_t$ at round $(2t-1)$. Receive $r(\a_t)$.
\STATE Play $\b_t$ at round $2t$. Receive $r(\b_t)$.
\STATE Feedback: $o_t = \mathbf 1(r(\a_t) > r(\b_t))$ to $\cA^{\lindb}$.
\ENDFOR
\end{algorithmic}
\end{algorithm}
\end{center}
\vspace{-5pt}
But since a single round of \emph{Online-BAI-}\lindb\ corresponds to two rounds of \emph{Online-BAI-}\linb\, a valid
BAI lower bound for the latter would immediately imply a BAI lower bound for the former problem. The result of \cref{thm:lbon} now follows from the known existing lower bound for the \emph{Online-BAI-}\linb\ problem from \citep[Theorem 2]{wagenmaker2022reward}.
\subsection{Proof of \cref{thm:lboff}}
\label{app:lboff}
The \emph{offline version of BAI problem with \lindb\ feedback} is a batch decision-making process where the learner is provided with an offline dataset $\cD_{\text{off}} = \{(\a_i,\b_i,o_i)\}_{i = 1}^n$, where for each triplet $(\a_i,\b_i,o_i)$, $(\a_i,\b_i) \in \cD\times \cD$ denotes a pair of action and $o_i \sim P(\a_i \succ \b_i)$ is the preference feedback of the pair $(\a_i,\b_i)$ generated according to the \lindb\ feedback model. The objective of the learner is to output an arm $\a_n \in \cD$ using the $n$ samples of $\cD_{\text{off}}$ such that $\E[r(\a_n)] = s(\a_t)$ is maximized or the difference $s(\a^*) - s(\a_n)$ is minimized. Let us denote this problem as \emph{Offline-BAI-}\lindb\ problem.
In the same spirit, the \emph{offline version of BAI problem with \lindb\ feedback} is a batch decision-making process where the learner is provided with an offline dataset $\cD_{\text{off}} = \{(\a_i,r_i)\}_{i = 1}^n$, where $\a_i \in \cD$ is an arbitrary action in $\cD$ and $r_i$ is a noisy sample of the score/reward of arm $i$ drawn according to the \linb\ feedback model. The objective of the learner is to output an arm $\a_n \in \cD$ using the $n$ samples of $\cD_{\text{off}}$ such that $\E[r(\a_n)] = s(\a_t)$ is maximized or the difference $s(\a^*) - s(\a_n)$ is minimized. Let us denote this problem as \emph{Offline-BAI-}\linb\ problem.
\textbf{Reducing \emph{Offline-BAI-}\linb\ to \emph{Offline-BAI-}\lindb.} Owing to \cref{lem:red} and following a similar approach described in \cref{alg:red} above it is easy to see that one can obtain an algorithm for the \emph{Offline-BAI-}\linb\ problem using an algorithm for the \emph{Offline-BAI-}\lindb\ algorithm. %Interested readers are encouraged go through the reduction ideas in \cite[Algorithm 2]{saha2021optimal} for more details.
% \red{Ideally we should have written the reduction algorithm for the offline case as well.}
Following the same argument used for proving \cref{thm:lbon} in \cref{app:lbon} above, the result of \cref{thm:lboff} now follows from the known existing lower bound for the \emph{Offline-BAI-}\linb\ problem from \citep[Theorem 2]{li2022pessimism}.
\section{Upper Bound Proofs}\label{app:proofs}
The proof structure for Theorem~\ref{thm:main_general} is similar to Theorem 3.1 \citep{xie2024exploratory}. However due to the hybrid nature of our algorithm, we are able to derive concentration lemmas with a faster convergence rate. We highlight the main differences in our analysis, which lead to the reduced sample complexity.
\subsection{Concentration Lemmas}
Let $\mathbb{P}_{\rm off}$ denote the nominal distribution of trajectory pairs in the offline dataset $\mathcal{D}_{\rm off}$. Define $\bm{\mu}^{(t)}_{\rm hyb} = \frac{1}{t - 1 + \gamma} (\sum_{i=1}^{t-1} \pi^{(i)} \otimes \tilde{\pi}^{(i)} + \gamma \mathbb{P}_{\rm off})$.
Define
\begin{align*}
f_{\pi}(\tau, \taut) = \beta \log \frac{\pi(\tau)}{\piref(\tau)} - \beta \log \frac{\pi(\taut)}{\piref(\taut)}.
\end{align*}
\begin{lemma}[Concentration for Algorithm \ref{alg:hybrid_rlhf}]\label{lem:concentration}
Suppose that Assumptions \ref{ass:realizability},\ref{ass:vmax} hold. Then
Algorithm \ref{alg:hybrid_rlhf} guarantees that with probability at least
$1-\delta$, for all steps $t\in [T]$,
\begin{align*}
&\alpha\cdot\E_{s_1\sim\rho,\tau\sim\Tilde{\pi}\ind{t-1}}{[\log(\pi\ind{t}(\tau))-\log(\pistarb(\tau))]} + \kappa \cdot\left[\E_{s_1\sim\rho{},(\tau,\taut)\sim{}\bm{\mu}\ind{t}_{\rm hyb}\mid{}s_1}{[f_{\pi\ind{t}}(\tau,\taut)
- f_{\pistarb}(\tau,\taut)]^2}\right] \notag \\&\leq \frac{2\log(2|\Pi|T\delta^{-1})}{\gamma + t-1} + \frac{\alpha}{\beta}\Vmax\sqrt{\frac{2^4\log(2|\Pi|T\delta^{-1})}{\gamma + t-1}}, \end{align*}
for $\kappa = (8(\Rmax+\Vmax)e^{2\Rmax})^{-2}$.
\end{lemma}
\begin{proof}
Fix $t \in \{2, \ldots, T+1\}$.
We wish to decompose the objective in HPO, Eq~\eqref{eq:HPO} into 2 terms as defined below:
Define the following quantities:
\begin{align}\label{eq:HPO_LB}
\pi^{(t)} = \argmin_{\pi \in \Pi} \bigg[\widehat{L}^{(t)}(\pi) + \widehat{B}^{(t)}(\pi)\bigg].
\end{align}
\begin{align*}
\widehat{L}^{(t)}(\pi) = \sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^{(t)}_{\rm hyb}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})})\bigg].
\end{align*}
\begin{align*}
\widehat{B}^{(t)}(\pi) = \alpha \sum_{\tau \in \mathcal{D}^{(t)}_{\rm opt}} \log \pi(\tau).
\end{align*}
Now we provide concentration lemmas for these individual terms before combining them together.
\begin{lemma}\label{lem:L} For any fixed $t \geq 1$, for all $\pi \in \Pi$ with probability at least $1 - \delta$, we have:
\begin{align*}
\kappa \cdot\left[\E_{s_1\sim\rho{},(\tau,\taut)\sim{}\bm{\mu}\ind{t}_{\rm hyb}\mid{}s_1}{[f_{\pi\ind{t}}(\tau,\taut)
- f_{\pistarb}(\tau,\taut)]^2}\right] \leq \widehat{L}^{(t)}(\pi) - \widehat{L}^{(t)}(\pistarb) + \log(2|\Pi|T\delta^{-1})
\end{align*}
\end{lemma}
\begin{proof}
Use the definition of $\bm{\mu}^{(t)}_{\rm hyb}$ to see the dataset $\mathcal{D}_{\rm hyb}^{(t)}$ is a sequence adapted to the filtration $\mathcal{F}^{(\gamma + t)} = \sigma(\mathcal{D}_{\rm off}, \ldots, \mathcal{D}_{\rm off}, (\tau^{(1)}, \tilde{\tau}^{1}), \ldots, (\tau^{(t-1)}, \tilde{\tau}^{(t-1)}))$. Then apply the Martinagle Chernoff Theorem to get a high probability bound. Combine it with Lemma C.8 \citep{xie2024exploratory} to get the above result.
\end{proof}
\begin{lemma}\label{lem:B} For any fixed $t \geq 1$, for all $\pi \in \Pi$ with probability at least $1 - \delta$, we have:
\begin{align*}
\alpha\cdot (t + \gamma -1) \cdot \E_{s_1\sim\rho,\tau\sim\Tilde{\pi}\ind{t-1}}{[\log(\pi\ind{t}(\tau))-\log(\pistarb(\tau))]} \leq \widehat{B}^{(t)}(\pi) - \widehat{B}^{(t)}(\pistarb) + \frac{\alpha}{\beta}\Vmax\sqrt{2^4\log(2|\Pi|\delta^{-1})}
\end{align*}
\end{lemma}
\begin{proof}
Apply Azuma Hoeffding with its sample average over $t + \gamma$ terms.
\end{proof}
Combining Lemma~\ref{lem:L} and Lemma~\ref{lem:B}, taking an union bound over all time steps $t \in [T]$, and utilizing the definition of $\pi^{(t)}$ in Eq.~\eqref{eq:HPO_LB} to note $\widehat{B}^{(t)}(\pi) + \widehat{L}^{(t)}(\pi) \leq \widehat{B}^{(t)}(\pistarb) + \widehat{L}^{(t)}(\pistarb)$, get we get the stated result.
% Follows from proof of lemma C.5 in \citep{xie2024exploratory} with the following differences:
% \begin{enumerate}
% \item Apply Azuma Hoeffding for the first term with its sample average over $t + \gamma$ terms.
% % \item Use the definition of $\bm{\mu}^{(t)}_{\rm hyb}$ to see the dataset $\mathcal{D}_{\rm hyb}^{(t)}$ is a sequence adapted to the filtration $\mathcal{F}^{(\gamma + t)} = \sigma(\mathcal{D}_{\rm off}, \ldots, \mathcal{D}_{\rm off}, (\tau^{(1)}, \tilde{\tau}^{1}), \ldots, (\tau^{(t-1)}, \tilde{\tau}^{(t-1)}))$.
% % \item Then apply the Martinagle Chernoff Theorem for the second term.
% \end{enumerate}
\end{proof}
\subsection{Proof of Theorem~\ref{thm:main_general}}
We use the standard steps for regret decomposition, and similar proof idea in \citep{xie2024exploratory}. The main difference in our analysis is the definition of the term $\mathcal{I}^{(t)}$, which allows us to use the faster convergence concentration lemma in Lemma~\ref{lem:concentration}, and also use our definition of the SEC in the hybrid RLHF case, $\coefffull$, which is smaller than the SEC in the online case.
\maingeneral*
\begin{proof}
Using the regret decomposition techniques for KL-Regularized MDPs from \citep{xie2024exploratory}, we derive:
\begin{align*}
&J_{\beta}(\pi_{\beta}^\ast) - J_{\beta}(\pi^{(t)}) \\
&\leq \frac{6 \Vmax}{T} + \frac{1}{T} \sum_{t=2}^T \mathbb{E}_{\tau \sim \tilde{\pi}^{(t-1)}} \bigg[ \beta \log(\pi\ind{t}(\tau)) - \beta \log(\pistarb(\tau)) \bigg] \\
&+ \frac{1}{T} \sum_{t=2}^T \mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^{(t)}, \taut \sim \tilde{\pi}^{(t)}} \bigg[ \beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau)} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut) \bigg]
\end{align*}
Recalling the definition of $\bm{\mu}^{(t)}_{\rm hyb} = \frac{1}{t - 1 + \gamma} (\sum_{i=1}^{t-1} \pi^{(i)} \otimes \tilde{\pi}^{(i)} + \gamma \mathbb{P}_{\rm off})$, we define:
\begin{align*}
\mathcal{I}^{(t)} = \frac{\left(\mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^{(t)}, \taut \sim \tilde{\pi}^{(t)}} \bigg[ \beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau)} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut) \bigg]\right)^2}{\Vmax^2 \vee (t - 1 + \gamma) \cdot \mathbb{E}_{s_1 \sim \rho, \tau, \taut \sim \bm{\mu}^{(t)}_{\rm hyb} | s_1} \bigg[ \left(\beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau)} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut)\right)^2\bigg]}
\end{align*}
Using AM-GM inequality we can bound:
\begin{align*}
& \mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^{(t)}, \taut \sim \tilde{\pi}^{(t)}} \bigg[ \beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau)} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut) \bigg]\\
&\leq \frac{\mathcal{I}^{(t)}}{2 \eta} + \frac{\eta}{2}\cdot \left( \Vmax^2 \vee (t - 1 + \gamma) \cdot \mathbb{E}_{s_1 \sim \rho, \tau, \taut \sim \bm{\mu}^{(t)}_{\rm hyb}} \bigg[ \left(\beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau) | s_1} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut)\right)^2\bigg]\right)\\
&\leq \frac{\mathcal{I}^{(t)}}{2 \eta} + \frac{\eta}{2}\cdot \left( \Vmax^2 + (t - 1 + \gamma) \cdot \mathbb{E}_{s_1 \sim \rho, \tau, \taut \sim \bm{\mu}^{(t)}_{\rm hyb} | s_1} \bigg[ \left(\beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau)} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut)\right)^2\bigg]\right)\\
\end{align*}
Recall the definition of $\coefffull$, and note that $\sum_{t=1}^T \mathcal{I}^{(t)} \leq \coefffull$.
Now we can write:
\begin{align*}
&J_{\beta}(\pi_{\beta}^\ast) - J_{\beta}(\pi^{(t)})\\
&\leq \frac{6 \Vmax}{T} + \frac{\coefffull}{2\eta T} + \frac{\eta}{2} \Vmax^2 + \frac{1}{T} \sum_{t=2}^T \bigg(\mathbb{E}_{\tau \sim \tilde{\pi}^{(t-1)}} \bigg[ \beta \log(\pi\ind{t}(\tau)) - \beta \log(\pistarb(\tau)) \bigg] \\
&+ \frac{\eta}{2} \cdot (t - 1 + \gamma) \cdot \mathbb{E}_{s_1 \sim \rho, \tau, \taut \sim \bm{\mu}^{(t)}_{\rm hyb} | s_1} \bigg[ \left(\beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau)} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut)\right)^2\bigg]\bigg)
\end{align*}
For a fixed $t \in \{2, \ldots, T + 1\}$, we consider the term:
\begin{align*}
&\mathbb{E}_{\tau \sim \tilde{\pi}^{(t-1)}} \bigg[ \beta \log(\pi\ind{t}(\tau)) - \beta \log(\pistarb(\tau)) \bigg]\\
&\qquad\qquad + \frac{\eta}{2} \cdot (t - 1 + \gamma) \cdot \mathbb{E}_{s_1 \sim \rho, \tau, \taut \sim \bm{\mu}^{(t)}_{\rm hyb} | s_1} \bigg[ \left(\beta \log \frac{\pi^{(t)}(\tau)}{\pi_{\rm ref}(\tau)} -r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\pi_{\rm ref}(\taut)} + r(\taut)\right)^2\bigg]
\end{align*}
Setting $\eta = \frac{\beta \kappa}{\alpha (T + \gamma)}$, and invoking Lemma~\ref{lem:concentration}, we get:
\begin{align*}
&J_{\beta}(\pi_{\beta}^\ast) - J_{\beta}(\pi^{(t)})\\
&\approxleq \frac{\Vmax}{T} + \frac{\alpha (T + \gamma )\coefffull}{\beta \kappa T} + \frac{\beta \kappa}{\alpha(T + \gamma)} \Vmax^2 +
\\& \sum_{t=2}^T \frac{1}{T} \left( \frac{\beta}{\alpha}\frac{\log(|\Pi|T\delta^{-1})}{\gamma + t-1} + \Vmax\sqrt{\frac{\log(|\Pi|T\delta^{-1})}{\gamma + t-1}} \right) \\
&\approxleq \frac{\Vmax}{T} + \frac{\alpha (T + \gamma )\coefffull}{\beta \kappa T} + \frac{\beta \kappa}{\alpha(T + \gamma)} \Vmax^2 + \\
& \frac{\beta}{\alpha}\frac{\log(|\Pi|T\delta^{-1}) \log(T)}{T} + \Vmax\sqrt{\frac{\log(|\Pi|T\delta^{-1})}{T}}
\end{align*}
Choosing
\begin{align*}
\alpha \propto \frac{\beta}{(\Vmax + \Rmax)e^{2 \Rmax}}\cdot\sqrt{\frac{\log(|\Pi|T\delta^{-1})\log(T)}{(T + \gamma)\cdot \coefffull}}
\end{align*}
and noting $\kappa < \Vmax^{-2}$, we get:
\begin{align*}
&J_{\beta}(\pi_{\beta}^\ast) - J_{\beta}(\pi^{(t)})\\
&\leq \kappa^{-1}\sqrt{\frac{(1 + \gamma / T)\cdot\coefffull\cdot\log(|\Pi|T\delta^{-1})\log(T)}{T}} + \Vmax\sqrt{\frac{\log(|\Pi|T\delta^{-1})}{T}}\\
&\leq \mathcal{O}((\Vmax+\Rmax)e^{2\Rmax})\cdot \sqrt{\frac{(1 + \gamma / T)\cdot\coefffull\cdot\log(|\Pi|T\delta^{-1})\log(T)}{T}}.
\end{align*}
\end{proof}
% \subsection{Linear MDP}
% % \textbf{Hybrid:}
% \linearupper*
% \begin{proof}
% Since $\|\phi\|_2 \leq 1$ by definition of Linear MDPs, we have $\textrm{Trace}(\Lambda_{\rm off}) \leq 4$. Using the elliptical potential theorem \citep{lattimore2020bandit}, we derive:
% \begin{align*}
% \coefffull &\leq 2 \left( \log( |\gamma \Lambda_{\rm off} + (\Vmax^2 + 4T)\mathbf{I}|) - \log(\rm{det}(\gamma\Lambda_{\rm off} + \Vmax^2\mathbf{I}))\right)\\
% &= 2 \sum_{i \in [d]}\left(\log\left(\gamma \lambda_{\rm off}^{(i)} + \Vmax^2 + 4T\right) - \log\left(\gamma \lambda_{\rm off}^{(i)} + \Vmax^2\right)\right)\\
% &\leq 2 \sum_{i \in [d]}\left(\log\left(1 + \frac{4T}{\max(\Vmax^2, \gamma \lambda_{\rm off}^{(i)})}\right)\right)\\
% \end{align*}
% where $\lambda_{\rm off}^{(i)}$ is the $i^{\rm th}$ eigenvalue of $\Lambda_{\rm off}$.
% \end{proof}
% Choosing $\gamma = \mathcal{O}(T)$, only terms where $\lambda_{\rm off}^{(i)} \leq \Omega(\frac{1}{T})$ remain in the summand, corresponding to the directions where the offline dataset has poor exploration. Let $d_{\rm hyb}$ denote the number of indices in $[d]$ such that $\lambda_{\rm off}^{(i)} \leq \Omega(\frac{1}{T})$. Therefore we get $\coefffull \leq \mathcal{O}\left(d_{\rm hyb}\log\left(1 + \frac{4T}{\Vmax^2}\right)\right)$.
% Hence, the SEC scales as the effective number of dimensions yet to be explored sufficiently.
\documentclass{article}
\usepackage{graphicx,fullpage} % Required for inserting images
% \usepackage{wrapfig}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
% \usepackage{xcolor} % colors
\usepackage{amsmath}
\usepackage{multirow}
\usepackage{amsthm, amssymb, adjustbox}
\usepackage{bm}
\usepackage{graphicx}
\usepackage{xparse}
\usepackage{anyfontsize}
\usepackage{thmtools}
\usepackage{thm-restate}
% \usepackage{natbib}
\hypersetup{
colorlinks=true,
linkcolor=Blue,
filecolor=magenta,
urlcolor=Cerulean,
citecolor=blue,
pdftitle={Overleaf Example},
pdfpagemode=FullScreen,
}
\usepackage{wrapfig}
\usepackage{cleveref}
\usepackage[dvipsnames]{xcolor}
\usepackage{subcaption}
%\usepackage[numbers]{natbib}
%\usepackage{subfigure}
% \usepackage{subcaption}
% \usepackage[ruled, vlined]{algorithm2e}
\usepackage{algorithm}
\usepackage{algorithmic}
% \theoremstyle{definition}
\usepackage{xcolor,colortbl}
\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\xmark}{\ding{55}}%
\newcommand*{\Scale}[2][4]{\scalebox{#1}{$#2$}}%
\input{preamble}
\title{Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration}
\author{
Avinandan Bose\\ University of Washington\\
avibose@cs.washington.edu \and Zhihan Xiong\\ University of Washington\\
zhihanx@cs.washington.edu \and Aadirupa Saha \\ Apple \\ aadirupa@apple.com \\ \and Simon Shaolei Du \\ University of Washington \\ ssdu@cs.washington.edu \and Maryam Fazel\\University of Washington\\mfazel@uw.edu
}
\date{}
\begin{document}
\maketitle
\input{main_paper/abstract.tex}
\input{main_paper/introduction}
\input{main_paper/related_work}
\input{main_paper/preliminaries}
\input{main_paper/hpo}
\input{main_paper/linear}
\input{main_paper/experiments}
\input{main_paper/conclusion}
\section*{Acknowledgements}
MF was supported in part by awards NSF TRIPODS II 2023166, CCF 2007036, CCF 2212261, CCF 2312775. SSD acknowledges the support of NSF IIS 2110170, NSF DMS 2134106, NSF CCF 2212261, NSF IIS 2143493, NSF IIS 2229881, Alfred P. Sloan Research Fellowship, and Amazon.
% \newpage
% \newpage
\medskip
\bibliography{References}
% \input{checklist}
\newpage
\appendix
\onecolumn
\input{appendix/upper_bound}
\input{appendix/lower_bound}
% \input{appendix/experiments}
\end{document}
\section*{Checklist}
\begin{enumerate}
\item For all models and algorithms presented, check if you include:
\begin{enumerate}
\item A clear description of the mathematical setting, assumptions, algorithm, and/or model. [Yes]
\item An analysis of the properties and complexity (time, space, sample size) of any algorithm. [Yes]
\item (Optional) Anonymized source code, with specification of all dependencies, including external libraries. [Not Applicable]
\end{enumerate}
\item For any theoretical claim, check if you include:
\begin{enumerate}
\item Statements of the full set of assumptions of all theoretical results. [Yes]
\item Complete proofs of all theoretical results. [Yes]
\item Clear explanations of any assumptions. [Yes]
\end{enumerate}
\item For all figures and tables that present empirical results, check if you include:
\begin{enumerate}
\item The code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL). [Not Applicable]
\item All the training details (e.g., data splits, hyperparameters, how they were chosen). [Not Applicable]
\item A clear definition of the specific measure or statistics and error bars (e.g., with respect to the random seed after running experiments multiple times). [Not Applicable]
\item A description of the computing infrastructure used. (e.g., type of GPUs, internal cluster, or cloud provider). [Not Applicable]
\end{enumerate}
\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include:
\begin{enumerate}
\item Citations of the creator If your work uses existing assets. [Not Applicable]
\item The license information of the assets, if applicable. [Not Applicable]
\item New assets either in the supplemental material or as a URL, if applicable. [Not Applicable]
\item Information about consent from data providers/curators. [Not Applicable]
\item Discussion of sensible content if applicable, e.g., personally identifiable information or offensive content. [Not Applicable]
\end{enumerate}
\item If you used crowdsourcing or conducted research with human subjects, check if you include:
\begin{enumerate}
\item The full text of instructions given to participants and screenshots. [Not Applicable]
\item Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. [Not Applicable]
\item The estimated hourly wage paid to participants and the total amount spent on participant compensation. [Not Applicable]
\end{enumerate}
\end{enumerate}
\documentclass{article}
\usepackage{graphicx,fullpage} % Required for inserting images
% \usepackage{wrapfig}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
% \usepackage{xcolor} % colors
\usepackage{amsmath}
\usepackage{multirow}
\usepackage{amsthm, amssymb}
\usepackage{bm}
\usepackage{graphicx}
\usepackage{xparse}
% \usepackage{natbib}
\hypersetup{
colorlinks=true,
linkcolor=Blue,
filecolor=magenta,
urlcolor=Cerulean,
citecolor=blue,
pdftitle={Overleaf Example},
pdfpagemode=FullScreen,
}
\usepackage{wrapfig}
\usepackage[dvipsnames]{xcolor}
\usepackage{subcaption}
%\usepackage[numbers]{natbib}
%\usepackage{subfigure}
% \usepackage{subcaption}
% \usepackage[ruled, vlined]{algorithm2e}
\usepackage{algorithm}
\usepackage{algorithmic}
% \theoremstyle{definition}
\newtheorem{property}{Property}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{remark}{Remark}
\newtheorem{lemma}{Lemma}
\newtheorem{assumption}{Assumption}
\newtheorem{proposition}{Proposition}
\newtheorem{example}{Example}[section]
\newtheorem{definition}{Definition}[section]
\usepackage{enumitem}
\usepackage{thmtools,thm-restate}
\newcommand{\mf}[1]{{\color{magenta} MF: #1}}
\newcommand{\avi}[1]{{\color{blue} Avi: #1}}
\newcommand{\zhihan}[1]{{\color{red} Zhihan: #1}}
\newcommand{\pisamp}{\wt{\pi}}
\newcommand{\samp}{\mb{\pi}_{\mathsf{samp}}}
% \newcommand{\bmu}{\mb{\mu}}
\newcommand{\bmu}{\mathbf{\mu}}
\newcommand{\coeff}{\rm SEC_{\rm HybRLHF}}
\newcommand{\coefffull}{\coeff(\Pi,T,\beta; \gamma, \mathcal{D}_{\rm off})}
\newcommand{\cDopt}{\cD_{\mathsf{opt}}}
\newcommand{\Lhat}{\wh{L}}%
\newcommand{\bhat}{\wh{b}}%
\newcommand{\Bhat}{\wh{B}}%
% \newcommand{\tautil}{\wt{\tau}}%
\newcommand{\stil}{\wt{s}}
\newcommand{\atil}{\wt{a}}
\newcommand{\Jb}{J_{\beta}}
\newcommand{\Ccov}{C_{\mathsf{cov}}}
\newcommand{\Cconc}{C_{\mathsf{conc}}}
\newcommand{\mainalg}{Exploratory Preference Optimization\xspace}
\newcommand{\alglong}{\mainalg}
\newcommand{\algshort}{\texttt{XPO}\xspace} %
\newcommand{\dpo}{\texttt{DPO}\xspace}
\newcommand{\taup}{\tau_{+}}
\newcommand{\taum}{\tau_{-}}
\newcommand{\pistarb}{\pistar_{\beta}}
\newcommand{\Qstarb}{Q^{\star}_{\beta}}
\newcommand{\Vstarb}{V^{\star}_{\beta}}
\newcommand{\ap}{a_{+}}%
\newcommand{\am}{a_{-}}%
\newcommand{\Shat}{\wh{S}}
\newcommand{\Astar}{A^{\pistar}}
\newcommand{\sigmab}{\wb{\sigma}}
\newcommand{\sigmabs}{\wb{\sigma}_{\pistar}}
\newcommand{\Hstar}{H^{\star}}%
\newcommand{\pitil}{\wt{\pi}}%
\newcommand{\Jhat}{\wh{J}}
\newcommand{\gbar}{\wb{g}}
\newcommand{\mutil}{\wt{\mu}}
\newcommand{\sigmastar}{\sigma_{\pistar}}
\newcommand{\Lmax}{L_{\texttt{max}}}
\newcommand{\Lmaxhat}{\wh{L}_{\texttt{max}}}
\newcommand{\vepsapx}{\veps_{\texttt{apx}}}
\newcommand{\Drho}[2]{\rho\prn*{#1\dmid{}#2}}
\newcommand{\murec}{\mu}
\newcommand{\astart}[1][t]{a^{\star,\sss{#1}}}
\newcommand{\astari}[1][i]{a^{\star,\sss{#1}}}
\newcommand{\vepsapxh}{\veps_{\texttt{apx},h}}
\newcommand{\Qstar}{Q^{\star}}
\newcommand{\Vstar}{V^{\star}}
\newcommand{\Lbc}{L_{\textsf{bc}}}
\newcommand{\Lhatbc}{\wh{L}_{\textsf{bc}}}
\newcommand{\traj}{o}
\newcommand{\aggrevate}{\texttt{Aggrevate}\xspace}
\newcommand{\loglossbc}{\texttt{LogLossBC}\xspace}
\newcommand{\loglossdagger}{\texttt{LogLossDagger}\xspace}
\newcommand{\Unif}{\mathsf{Unif}}
\newcommand{\phistar}{\phi^{\star}}
\newcommand{\gstar}{g^{\star}}
\newcommand{\pibar}{\wb{\pi}}
\newcommand{\Cpush}{C_{\mathsf{push}}}
\newcommand{\sfrak}{\mathfrak{s}}
\newcommand{\afrak}{\mathfrak{a}}
\newcommand{\bfrak}{\mathfrak{b}}
\newcommand{\cfrak}{\mathfrak{c}}
\newcommand{\xfrak}{\mathfrak{x}}
\newcommand{\yfrak}{\mathfrak{y}}
\newcommand{\zfrak}{\mathfrak{z}}
\newcommand{\bxi}{\mb{\xi}}
\newcommand{\bs}{\mathbf{s}}
\newcommand{\ba}{\mathbf{a}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\br}{\mathbf{r}}
\newcommand{\vepsstat}{\veps_{\mathrm{stat}}}
\newcommand{\ppr}{p_{\mathrm{pr}}}
\newcommand{\ppo}{p_{\mathrm{po}}}
\newcommand{\gammastar}{\gamma^{\star}}
\newcommand{\compc}[1][\veps]{\mathsf{dec}_{#1}}
\newcommand{\var}{\mathrm{Var}}
\newcommand{\Var}{\var}
\newcommand{\varplus}{\mathrm{Var}_{\sss{+}}}
\newcommand{\Varplus}{\varplus}
\newcommand{\Varm}[1][M]{\Var\sups{#1}}
\newcommand{\Varmbar}[1][\Mbar]{\Var\sups{#1}}
\newcommand{\pmbar}{p_{\sMbar}}
\newcommand{\pmdist}{p\subs{M}}
\newcommand{\PM}{\bbP\sups{M}}
\newcommand{\PMbar}{\bbP\sups{\Mbar}}
\newcommand{\Dbi}[2]{D_{\mathrm{bi}}\prn*{#1\dmid#2}}
\newcommand{\Dbishort}{D_{\mathrm{bi}}}
\newcommand{\cQm}{\cQ_{\cM}}
\newcommand{\cQmh}[1][h]{\cQ_{\cM;#1}}
\newcommand{\cTm}[1][M]{\cT\sups{#1}}
\newcommand{\cTmstar}[1][\Mstar]{\cT\sups{#1}}
\newcommand{\Confone}{R_1}
\newcommand{\Conftwo}{R_2}
\newcommand{\compbi}{\compgen[\mathrm{bi}]}
\newcommand{\compbidual}{\compgendual[\mathrm{bi}]}
\newcommand{\bloss}{\cL}
\newcommand{\cGmbar}{\cG\sups{\Mbar}}
\newcommand{\gmmbar}[1][M]{g\sups{#1;\Mbar}}
\newcommand{\Lbr}{L_{\mathrm{br}}}
\newcommand{\Lbrfull}{L_{\mathrm{br}}(\cM)}
\newcommand{\Lbrfulls}{L^{2}_{\mathrm{br}}(\cM)}
\newcommand{\mum}[1][M]{\mu\sups{#1}}
\newcommand{\wm}[1][M]{w\sups{#1}}
\newcommand{\phim}[1][M]{\phi\sups{#1}}
\newcommand{\AlgEsth}[1][h]{\mathrm{\mathbf{Alg}}_{\mathsf{Est};#1}}
\newcommand{\EstHelh}[1][h]{\mathrm{\mathbf{Est}}_{\mathsf{H};#1}}
\newcommand{\thetam}[1][M]{\theta(#1)}
\newcommand{\thetambar}[1][\Mbar]{\theta(#1)}
\newcommand{\Copt}{C_{\mathrm{opt}}}
\newcommand{\Calpha}{C_{\alpha}}
\newcommand{\dimbi}{d_{\mathrm{bi}}}
\newcommand{\Lbi}{L_{\mathrm{bi}}}
\newcommand{\dimbifull}{d_{\mathrm{bi}}(\cM)}
\newcommand{\dimbifulls}[1][2]{d^{#1}_{\mathrm{bi}}(\cM)}
\newcommand{\Lbifull}{L_{\mathrm{bi}}(\cM)}
\newcommand{\Lbifulls}{L^{2}_{\mathrm{bi}}(\cM)}
\newcommand{\piest}{\pi^{\mathrm{est}}}
\newcommand{\piestm}[1][M]{\pi^{\mathrm{est}}_{\sss{#1}}}
\newcommand{\lest}{\ell^{\mathrm{est}}}
\newcommand{\lestm}[1][M]{\ell^{\mathrm{est}}_{\sss{#1}}}
\newcommand{\Piest}{\Pi^{\mathrm{est}}}
\newcommand{\qbar}{\bar{q}}
\newcommand{\qopt}{q^{\mathrm{opt}}}
\newcommand{\pialpha}{\pi^{\alpha}}
\newcommand{\pialpham}[1][M]{\pi^{\alpha}_{\sss{#1}}}
\newcommand{\piref}{\pi_{\rm ref}}
% \newcommand{\pisamp}{\pi_{\rm samp}}
\newcommand{\taut}{\Tilde{\tau}}
\newcommand{\con}{x}
\newcommand{\Cspace}{\cX}
\newcommand{\cMx}[1][x]{\cM|_{#1}}
\newcommand{\cMhatx}[1][x]{\cMhat|_{#1}}
\newcommand{\cFmx}[1][x]{\cFm|_{#1}}
\newcommand{\cpol}{\mb{\pi}}
\newcommand{\cpolstar}{\cpol^{\star}}
\newcommand{\cpolm}[1][M]{\cpol\subs{#1}}
\newcommand{\RegCDM}{\Reg_{\mathsf{DM}}}
\newcommand{\EstCD}{\mathrm{\mathbf{Est}}_{\mathsf{D}}}
\newcommand{\relu}{\mathsf{relu}}
\newcommand{\sdis}{\mb{\theta}}
\newcommand{\rdis}{\mb{\psi}}
\newcommand{\Deltanot}{\Delta_0}
\newcommand{\Delm}{\Delta\subs{M}}
\newcommand{\Star}{\mathfrak{s}}
\newcommand{\StarCheck}{\check{\Star}}
\newcommand{\StarDim}{\Star}
\newcommand{\Stardim}{\Star}
\newcommand{\El}{\mathfrak{e}}
\newcommand{\ElCheck}{\check{\El}}
\newcommand{\ElDim}{\El}
\newcommand{\Eldim}{\El}
\newcommand{\Rmax}{R_{\rm max}}
\newcommand{\Vmax}{V_{\rm max}}
\newcommand{\errm}[1][M]{\mathrm{err}\sups{#1}}
\newcommand{\hardfamily}{$(\alpha,\beta,\delta)$-family\xspace}
\newcommand{\hardu}{u}
\newcommand{\hardv}{v}
\newcommand{\igwshort}{IGW\xspace}
\newcommand{\igwtext}{inverse gap weighting\xspace}
\newcommand{\pcigw}{\textsf{PC-IGW}\xspace}
\newcommand{\pcigwb}{\textsf{PC-IGW.Bilinear}\xspace}
\newcommand{\pcigwtext}{policy cover inverse gap weighting\xspace}
\newcommand{\igwspanner}{\textsf{IGW-Spanner}\xspace}
\newcommand{\igwargmax}{\textsf{IGW-ArgMax}\xspace}
\newcommand{\oracle}{\mathrm{\mathbf{Alg}}_{\mathsf{Plan}}\xspace}
\newcommand{\MinimaxReg}{\mathfrak{M}(\cM,T)}
\newcommand{\MinimaxRegBayes}{\underbar{\mathfrak{M}}(\cM,T)}
\renewcommand{\emptyset}{\varnothing}
\newcommand{\NullObs}{\crl{\emptyset}}
\newcommand{\filt}{\mathscr{F}}
\newcommand{\ffilt}{\mathscr{F}}
\newcommand{\gfilt}{\mathscr{G}}
\newcommand{\hfilt}{\mathscr{H}}
\newcommand{\hist}{\mathcal{H}}
\newcommand{\Asig}{\mathscr{P}}
\newcommand{\Rsig}{\mathscr{R}}
\newcommand{\Osig}{\mathscr{O}}
\newcommand{\borel}{\mathfrak{B}}
\newcommand{\Hspace}{\Omega}
\newcommand{\Hsig}{\filt}
\newcommand{\dom}{\nu}
\newcommand{\dens}{m}
\newcommand{\densm}[1][M]{m^{\sss{#1}}}
\newcommand{\abscont}{V(\cM)}
\newcommand{\abscontp}{V(\cM')}
\newcommand{\Vmbar}{V^{\sMbar}}
\newcommand{\vepsup}{\bar{\veps}}
\newcommand{\vepslow}{\ubar{\veps}}
\newcommand{\vepsg}{\veps_{\gamma}}
\newcommand{\vepsupg}{\bar{\veps}_{\gamma}}
\newcommand{\vepslowg}{\ubar{\veps}_{\gamma}}
\newcommand{\gameval}[1]{\cV_{\gamma}\sups{#1}}
\newcommand{\Framework}{Decision Making with Structured Observations\xspace}
\newcommand{\FrameworkShort}{DMSO\xspace}
\newcommand{\learner}{learner\xspace}
\newcommand{\Cloc}{c_{\ell}}
\newcommand{\Cglobal}{C(\cM)}
\newcommand{\act}{\pi}
\newcommand{\Act}{\Pi}
\newcommand{\ActSpace}{\Pi}
\newcommand{\ActSize}{\abs{\Pi}}
\newcommand{\ActSizeB}{\abs{\cA}}
\newcommand{\ActSizeRL}{\abs{\cA}}
\newcommand{\ActDist}{\Delta(\Act)}
\newcommand{\action}{decision\xspace}
\newcommand{\actions}{decisions\xspace}
\newcommand{\actstar}{\pistar}
\newcommand{\actb}{a}
\newcommand{\StateSize}{\abs{\cS}}
\newcommand{\obs}{o}
\newcommand{\Obs}{\mathcal{\cO}}
\newcommand{\ObsSpace}{\mathcal{\cO}}
\newcommand{\Ospace}{\mathcal{\cO}}
\newcommand{\RewardSpace}{\cR}
\newcommand{\RSpace}{\RewardSpace}
\newcommand{\Rspace}{\RewardSpace}
\newcommand{\Model}{\cM}
\newcommand{\cMhat}{\wh{\cM}}
\newcommand{\compbasic}{\mathsf{dec}}
\newcommand{\comp}[1][\gamma]{\mathsf{dec}_{#1}}
\newcommand{\comprand}[1][\gamma]{\mathsf{dec}^{\mathrm{rnd}}_{#1}}
\newcommand{\compshort}[1][\gamma]{\mathsf{d}_{#1}}
\newcommand{\compt}[1][\gamma]{\wt{\mathsf{dec}}_{#1}}
\newcommand{\compbar}[1][\gamma]{\overline{\mathsf{dec}}_{#1}}
\newcommand{\compb}[1][\gamma]{\underline{\mathsf{dec}}_{#1}}
\newcommand{\compdual}[1][\gamma]{\compb[#1]}
\newcommand{\compbayes}[1][\gamma]{\compb[#1]}
\newcommand{\compKL}{\compgen[\mathrm{KL}]}
\newcommand{\compKLdual}{\compgendual[\mathrm{KL}]}
\newcommand{\compSq}{\compgen[\mathrm{Sq}]}
\newcommand{\compSqdual}{\compgendual[\mathrm{Sq}]}
\newcommand{\compopt}{\comp^{\mathrm{opt}}}
\newcommand{\compoptdual}{\compb^{\mathrm{opt}}}
\newcommand{\compH}{\compgen[\mathrm{H}]}
\newcommand{\compHdual}{\compgendual[\mathrm{H}]}
\newcommand{\complocshort}{\comp^{\mathrm{loc}}}
\newcommand{\comploc}[2][\gamma]{\mathsf{dec}_{#1,#2}}
\newcommand{\compthmone}{\comploc{\vepslowg}(\cM)}
\newcommand{\compthmtwo}{\comploc{\vepsupg}(\conv(\cM))}
\newcommand{\comptext}{decision-estimation coefficient\xspace}
\newcommand{\Comptext}{Decision-estimation coefficient\xspace}
\newcommand{\CompText}{Decision-Estimation Coefficient\xspace}
\newcommand{\CompAbbrev}{DEC\xspace}
\newcommand{\CompShort}{\CompAbbrev}
\newcommand{\compgen}[1][D]{\comp^{#1}}
\newcommand{\compgendual}[1][D]{\compb^{#1}}
\newcommand{\compd}[1][D]{\comp^{#1}}
\newcommand{\est}{\mathsf{est}}
\newcommand{\M}[1]{^{{\scriptscriptstyle M}}} %
\newcommand{\sM}{\sss{M}}
\newcommand{\sMbar}{\sss{\Mbar}}
\renewcommand{\ss}[1]{^{{\scriptscriptstyle#1}}}
\newcommand{\sups}[1]{^{{\scriptscriptstyle#1}}}
\newcommand{\subs}[1]{_{{\scriptscriptstyle#1}}}
\renewcommand{\sp}[1]{^{{\scriptscriptstyle#1}}}
\renewcommand{\sb}[1]{_{{\scriptscriptstyle#1}}}
\newcommand{\sss}[1]{{\scriptscriptstyle#1}}
\newcommand{\Ens}[2]{\En^{\sss{#1},#2}}
\newcommand{\Em}[1][M]{\En^{\sss{#1}}}
\newcommand{\Embar}{\En\sups{\Mbar}}
\newcommand{\Enm}[1][M]{\En^{\sss{#1}}}
\newcommand{\Enmbar}[1][\Mbar]{\En^{\sss{#1}}}
\newcommand{\Prs}[1]{\bbP\sups{#1}}
\newcommand{\bbPs}[1]{\bbP\sups{#1}}
\newcommand{\bbPm}[1][M]{\bbP\sups{#1}}
\newcommand{\bbPmbar}[1][\Mbar]{\bbP\sups{#1}}
\newcommand{\fmhat}{f\sups{\Mhat}}
\newcommand{\fm}[1][M]{f\sups{#1}}
\newcommand{\pim}[1][M]{\pi_{\sss{#1}}}
\newcommand{\pimh}[1][h]{\pi_{\sss{M,#1}}}
\newcommand{\gm}{g\sups{M}}
\newcommand{\Mprime}{M'}
\newcommand{\pimprime}{\pi_{\sss{M'}}}
\newcommand{\cFm}{\cF_{\cM}}
\newcommand{\Pim}{\Pi_{\cM}}
\newcommand{\cMf}{\cM_{\cF}}
\newcommand{\fmbar}{f\sups{\Mbar}}
\newcommand{\pimbar}{\pi\subs{\Mbar}}
\newcommand{\fmstar}{f\sups{\Mstar}}
\newcommand{\pimstar}{\pi\subs{\Mstar}}
\newcommand{\fstar}{f^{\star}}
\newcommand{\pistar}{\pi^{\star}}
\newcommand{\pihat}{\wh{\pi}}
\newcommand{\cMloc}[1][\veps]{\cM_{#1}}
\newcommand{\Mbar}{\wb{M}}
\newcommand{\fmi}{f\sups{M_i}}
\newcommand{\pimi}{\pi\subs{M_i}}
\newcommand{\Jm}{J\sups{M}}
\newcommand{\Rm}[1][M]{R\sups{#1}}
\newcommand{\Pm}[1][M]{P\sups{#1}}
\newcommand{\Pmstar}[1][\Mstar]{P\sups{#1}}
\newcommand{\Pmbar}{P\sups{\Mbar}}
\newcommand{\Rmbar}{R\sups{\Mbar}}
\newcommand{\Prm}[2]{\bbP^{\sss{#1},#2}}
\newcommand{\Jmstar}{J\sups{\Mstar}}
\newcommand{\PiNS}{\Pi_{\mathrm{NS}}} %
\newcommand{\PiRNS}{\Pi_{\mathrm{RNS}}} %
\newcommand{\PiNM}{\Pi_{\mathrm{NM}}} %
\newcommand{\PiGen}{\Pi_{\mathrm{RNS}}} %
\newcommand{\Qmistar}[1][i]{Q^{#1,\star}}
\newcommand{\Vmistar}[1][i]{V^{#1,\star}}
\newcommand{\Qmstar}[1][M]{Q^{\sss{#1},\star}}
\newcommand{\Vmstar}[1][M]{V^{\sss{#1},\star}}
\newcommand{\Qmpi}[1][\pi]{Q^{\sss{M},#1}}
\newcommand{\Vmpi}[1][\pi]{V^{\sss{M},#1}}
\newcommand{\Qmbarpi}[1][\pi]{Q^{\sss{M},#1}}
\newcommand{\Vmbarpi}[1][\pi]{V^{\sss{M},#1}}
\newcommand{\Qm}[2]{Q^{\sss{#1},#2}}
\newcommand{\Vm}[2]{V^{\sss{#1},#2}}
\newcommand{\dmpi}[1][\pi]{d^{\sss{M},#1}}
\newcommand{\dmbarpi}[1][\pi]{d^{\sss{\Mbar},#1}}
\newcommand{\dm}[2]{d^{\sss{#1},#2}}
\newcommand{\dbar}{\bar{d}}
\newcommand{\dtil}{\wt{d}}
\newcommand{\Reg}{\mathrm{\mathbf{Reg}}}
\newcommand{\RegBar}{\widebar{\mathrm{\mathbf{Reg}}}}
\newcommand{\Est}{\mathrm{\mathbf{Est}}}
\newcommand{\EstBar}{\widebar{\mathrm{\mathbf{Est}}}}
\newcommand{\EstHel}{\mathrm{\mathbf{Est}}_{\mathsf{H}}}
\newcommand{\EstOnHel}{\mathrm{\mathbf{Est}}^{\mathsf{on}}_{\mathsf{H}}}
\newcommand{\EstBarHel}{\widebar{\mathrm{\mathbf{Est}}}_{\mathsf{H}}}
\newcommand{\EstProbHel}{\mathrm{\mathbf{Est}}_{\mathsf{H}}(T,\delta)}
\newcommand{\EstH}{\EstHel}
\newcommand{\EstD}{\mathrm{\mathbf{Est}}_{\mathsf{D}}}
\newcommand{\EstSq}{\mathrm{\mathbf{Est}}_{\mathsf{Sq}}}
\newcommand{\EstKL}{\mathrm{\mathbf{Est}}_{\mathsf{KL}}}
\newcommand{\RegDM}{\Reg_{\mathsf{DM}}}
\newcommand{\RegBarDM}{\RegBar_{\mathsf{DM}}}
\newcommand{\RegB}{\RegBandit}
\newcommand{\RegMAB}{\RegBandit}
\newcommand{\RegBarMAB}{\RegBar_{\mathsf{B}}}
\newcommand{\RegBandit}{\mathrm{\mathbf{Reg}}_{\mathsf{B}}}
\newcommand{\RegRL}{\Reg_{\mathsf{RL}}}
\newcommand{\RegBarRL}{\RegBar_{\mathsf{RL}}}
\newcommand{\RegOL}{\mathrm{\mathbf{Reg}}_{\mathsf{OL}}(T,\delta)}
\newcommand{\AlgOL}{\mathrm{\mathbf{Alg}}_{\mathsf{OL}}}
\newcommand{\RegOLShort}{\Reg_{\mathsf{OL}}}
\newcommand{\RegBR}{\Reg_{\mathsf{BR}}}
\newcommand{\RegSB}{\Reg_{\mathsf{SB}}}
\newcommand{\RegSquare}{\Reg_{\mathsf{Sq}}}
\newcommand{\RegCB}{\mathrm{\mathbf{Reg}}_{\mathsf{CB}}}
\newcommand{\RegBarCB}{\widebar{\mathrm{\mathbf{Reg}}}_{\mathsf{CB}}}
\newcommand{\AlgEst}{\mathrm{\mathbf{Alg}}_{\mathsf{Est}}}
\newcommand{\logloss}{\ls_{\mathrm{\log}}}
\newcommand{\RegLog}{\Reg_{\mathsf{KL}}}
\newcommand{\RegBarLog}{\widebar{\Reg}_{\mathsf{KL}}}
\newcommand{\RegLogT}{\Reg_{\mathsf{KL}}(T)}
\newcommand{\MComp}{\mathsf{est}(\cM,T)}
\newcommand{\Mcomp}{\mathsf{est}(\cM,T)}
\newcommand{\GComp}{\mathsf{est}(\cG,T)}
\newcommand{\Gcomp}{\mathsf{est}(\cG,T)}
\newcommand{\QComp}{\mathsf{est}(\cQm,T)}
\newcommand{\Qcomp}{\mathsf{est}(\cQm,T)}
\newcommand{\PComp}{\mathsf{est}(\cP,T)}
\newcommand{\PComph}[1][h]{\mathsf{est}(\cP_{#1},T)}
\newcommand{\ActComp}{\mathsf{est}(\Pim,T)}
\newcommand{\AComp}{\mathsf{est}(\Pim,T)}
\newcommand{\Acomp}{\mathsf{est}(\Pim,T)}
\newcommand{\MCov}[1][\veps]{\cN(\cM,#1)}
\newcommand{\ActCov}[1][\veps]{\cN(\Pim,#1)}
\newcommand{\mathrmb}[1]{\mathrm{\mathbf{#1}}}
\newcommand{\Mhat}{\wh{M}}
\newcommand{\Mstar}{M^{\star}}
\newcommand{\bbPl}{\bbP_{\lambda}}
\newcommand{\bbQl}{\bbQ_{\lambda}}
\newcommand{\cAhat}{\wh{\cA}}
\newcommand{\ghat}{\wh{g}}
\newcommand{\thetahat}{\wh{\theta}}
\newcommand{\thetastar}{\theta^{\star}}
\newcommand{\tens}{\otimes}
\newcommand{\Rbar}{\wb{R}}
\newcommand{\Pbar}{\widebar{P}}
\newcommand{\starucb}{star hull UCB\xspace}
\newcommand{\starucblong}{star hull upper confidence bound\xspace}
\newcommand{\optionone}{\textsc{Option I}\xspace}
\newcommand{\optiontwo}{\textsc{Option II}\xspace}
\newcommand{\Dis}{\mathrm{Dis}}
\newcommand{\xdist}{\cD}
\newcommand{\rdist}{\bbP_{r}}
\newcommand{\algcomment}[1]{\textcolor{blue!70!black}{\footnotesize{\texttt{\textbf{//\hspace{2pt}#1}}}}}
\newcommand{\algcommentlight}[1]{\textcolor{blue!70!black}{\transparent{0.5}{\scriptsize{\texttt{\textbf{//\hspace{2pt}#1}}}}}}
\newcommand{\algcommentbig}[1]{\textcolor{blue!70!black}{\footnotesize{\texttt{\textbf{/*
#1~*/}}}}}
\newcommand{\algcolor}[1]{\textcolor{blue!70!black}{#1}}
\newcommand{\algspace}{\hspace{\algorithmicindent}}
\newcommand{\Holder}{H{\"o}lder\xspace}
\newcommand{\Otilde}{\wt{O}}
\newcommand{\midsem}{\,;}
\newcommand{\trn}{\top}
\newcommand{\pinv}{\dagger}
\newcommand{\psdleq}{\preceq}
\newcommand{\psdgeq}{\succeq}
\newcommand{\psdlt}{\prec}
\newcommand{\psdgt}{\succ}
\newcommand{\approxleq}{\lesssim}
\newcommand{\approxgeq}{\gtrsim}
\newcommand{\opnorm}{\mathsf{op}}
\newcommand{\eigmin}{\lambda_{\mathrm{min}}}
\newcommand{\eigmax}{\lambda_{\mathrm{max}}}
% \renewcommand{\tC}{\tilde{\cC}}
\newcommand{\zero}{\mathbf{0}}
\newcommand{\Id}{I}
\newcommand{\bq}{\mathbf{q}}
\newcommand{\xpi}{x^{\pi}}
\newcommand{\phipi}{\phi^{\pi}}
\newcommand{\lw}{\bar{w}}
\newcommand{\bu}{\mb{u}}
\newcommand{\bv}{\mb{v}}
\newcommand{\bz}{\mb{z}}
\newcommand{\bw}{\mb{w}}
\newcommand{\xt}{x_t}
\newcommand{\xto}{x_{t+1}}
\newcommand{\expone}{(1)}
\newcommand{\exptwo}{(2)}
\newcommand{\xl}[1][T]{x_{1:#1}}
\newcommand{\xr}[1][t]{x_{#1:T}}
\newcommand{\ul}[1][T]{u_{1:#1}}
\newcommand{\ur}[1][t]{u_{#1:T}}
\newcommand{\wl}[1][T]{w_{1:#1}}
\renewcommand{\wr}[1][t]{w_{#1:T}}
\newcommand{\fhat}{\wh{f}}
\newcommand{\ahat}{\wh{a}}
\newcommand{\fbar}{\bar{f}}
\newcommand{\ind}[1]{^{{\scriptscriptstyle(#1)}}}
\newcommand{\xh}{x_h}
\newcommand{\rh}{r_h}
\renewcommand{\rh}{r_h}
\newcommand{\xhp}{x_{h+1}}
\newcommand{\bigoh}{O}
\newcommand{\bigoht}{\wt{O}}
\newcommand{\bigom}{\Omega}
\newcommand{\bigomt}{\wt{\Omega}}
\newcommand{\bigthetat}{\wt{\Theta}}
\newcommand{\indic}{\mathbb{I}}
\newcommand{\overleq}[1]{\overset{#1}{\leq{}}}
\newcommand{\overgeq}[1]{\overset{#1}{\geq{}}}
\newcommand{\overlt}[1]{\overset{#1}{<}}
\newcommand{\overgt}[1]{\overset{#1}{>}}
\newcommand{\overeq}[1]{\overset{#1}{=}}
\renewcommand{\Pr}{\bbP}
\newcommand{\ftilde}{\tilde{f}}
\newcommand{\fcheck}{\check{f}}
\newcommand{\gtilde}{\tilde{g}}
\newcommand{\empcov}{\wh{\cF}}
\newcommand{\Sbad}{S_{\mathrm{bad}}}
\newcommand{\aprime}{a'}
\newcommand{\undertext}[2]{\underbrace{#1}_{\text{#2}}}
\newcommand{\fup}{\bar{f}}
\newcommand{\flow}{\underbar{f}}
\newcommand{\aone}{a\ind{1}}
\newcommand{\atwo}{a\ind{2}}
\newcommand{\cb}{\mathsf{A}}
\newcommand{\poly}{\mathrm{poly}}
\newcommand{\polylog}{\mathrm{polylog}}
\newcommand{\Dsq}[2]{D_{\mathsf{Sq}}\prn*{#1,#2}}
\newcommand{\Dsqshort}{D_{\mathsf{Sq}}}
\newcommand{\kl}[2]{D_{\mathsf{KL}}\prn*{#1\,\|\,#2}}
\newcommand{\Dkl}[2]{D_{\mathsf{KL}}\prn*{#1\,\|\,#2}}
\newcommand{\Dklshort}{D_{\mathsf{KL}}}
\newcommand{\Dhel}[2]{D_{\mathsf{H}}\prn*{#1,#2}}
\newcommand{\Dgen}[2]{D\prn*{#1\dmid{}#2}}
\newcommand{\Dgent}[2]{D\ind{t}\prn*{#1\dmid{}#2}}
\newcommand{\Dhels}[2]{D^{2}_{\mathsf{H}}\prn*{#1,#2}}
\newcommand{\Ddel}[2]{D_{\Delta}\prn*{#1,#2}}
\newcommand{\Dhelshort}{D^{2}_{\mathsf{H}}}
\newcommand{\Dchis}[2]{D_{\chi^2}\prn*{#1\dmid{}#2}}
\newcommand{\DchisX}[3]{D_{\chi^2}\prn[#1]{#2\dmid{}#3}}
\newcommand{\Dtris}[2]{D^{2}_{\Delta}\prn*{#1,#2}}
\newcommand{\Dtri}[2]{D_{\Delta}\prn*{#1,#2}}
\newcommand{\Dtv}[2]{D_{\mathsf{TV}}\prn*{#1,#2}}
\newcommand{\Dtvs}[2]{D^2_{\mathsf{TV}}\prn*{#1,#2}}
\newcommand{\dkl}[2]{d_{\mathsf{KL}}\prn*{#1\,\|\,#2}}
\newcommand{\Dphi}[2]{D_{\phi}(#1\;\|\;#2)}
\newcommand{\dphi}[2]{d_{\phi}(#1\;\|\;#2)}
\newcommand{\klr}[2]{D_{\mathsf{re}}(#1\;\|\;#2)}
\newcommand{\tv}[2]{D_{\mathsf{tv}}(#1\;\|\;#2)}
\newcommand{\DhelsX}[3]{D^{2}_{\mathsf{H}}\prn[#1]{#2,#3}}
\newcommand{\Ber}{\mathrm{Ber}}
\newcommand{\Rad}{\mathrm{Rad}}
\newcommand{\dmid}{\;\|\;}
\newcommand{\conv}{\mathrm{co}}
\newcommand{\diam}{\mathrm{diam}}
\newcommand{\bls}{\mb{\ls}}
\newcommand{\brak}{{\tiny[]}}
\newcommand{\Deltagap}{\Delta_{\mathrm{policy}}}
\newcommand{\Picsc}{\Pi^{\mathrm{csc}}}
\newcommand{\vcdim}{\mathrm{vcdim}}
\newcommand{\adacb}{\textsf{AdaCB}\xspace}
\newcommand{\squarecb}{\textsf{SquareCB}\xspace}
\newcommand{\falcon}{\textsf{FALCON}\xspace}
\newcommand{\Vf}{V}
\newcommand{\Qf}{Q}
\newcommand{\Vpi}{\Vf^{\pi}}
\newcommand{\Qpi}{\Qf^{\pi}}
\newcommand{\valueiteration}{\textsf{ValueIteration}\xspace}
\newcommand{\labelme}[1]{\tag{\theequation}\label{#1}}
\newcommand{\Ssafe}{\cS_{\mathrm{safe}}}
\newcommand{\Esafe}{\cE_{\mathrm{safe}}}
\newcommand{\Tsafe}{T_{\mathrm{safe}}}
\newcommand{\starhull}{\mathrm{star}}
\newcommand{\pescape}{P_{\mathrm{escape}}}
\newcommand{\vepsucb}{\veps_{\mathrm{ucb}}}
\newcommand{\vepsstati}{\veps_{\mathrm{stat,I}}}
\newcommand{\vepsstatii}{\veps_{\mathrm{stat,II}}}
\newcommand{\nstat}{n_{\mathrm{stat}}}
\newcommand{\Phat}{\wh{P}}
\newcommand{\Ffilt}{\mathfrak{F}}
\newcommand{\Eescape}{\cE_{\textrm{escape}}}
\newcommand{\vepshat}{\wh{\veps}}
\newcommand{\vepsbar}{\widebar{\veps}}
\newcommand{\igw}{\textsf{IGW}}
\newcommand{\ssep}{;\,}
\newcommand{\sh}[1][h]{s_{#1}}
\newcommand{\qh}{q_h}
\newcommand{\Qh}{Q_h}
\newcommand{\im}{\ind{m}}
\newcommand{\sep}{\ssep}
\newcommand{\Creg}{C_{\mathrm{reg}}}
\newcommand{\creg}{c_{\mathrm{reg}}}
\newcommand{\Cucb}{C_{\mathrm{ucb}}}
\newcommand{\gammam}{\gamma\im}
\newcommand{\Eopt}{\cE_{\mathrm{opt}}}
\newcommand{\emi}{\psi}
\newcommand{\dirsum}{\oplus}
\newcommand{\piunif}{\pi_{\mathrm{unif}}}
\newcommand{\cPhat}{\wh{\cP}}
\newcommand{\supp}{\mathrm{supp}}
\newcommand{\cPbar}{\wb{\cP}}
\newcommand{\Pstar}{P^{\star}}
\newcommand{\Plbar}{\underbar{P}}
\newcommand{\Ptil}{\wt{\cP}}
\newcommand{\astar}{a^{\star}}
\newcommand{\cFhat}{\wh{\cF}}
\newcommand{\cFbar}{\widebar{\cF}}
\newcommand{\Vtil}{\wt{\Vf}}
\newcommand{\Qtil}{\wt{\Qf}}
\newcommand{\ftil}{\tilde{f}}
\newcommand{\mathand}{\quad\text{and}\quad}
\newcommand{\mathwhere}{\quad\text{where}\quad}
\def\multiset#1#2{\ensuremath{\left(\kern-.3em\left(\genfrac{}{}{0pt}{}{#1}{#2}\right)\kern-.3em\right)}}
\newcommand{\grad}{\nabla}
\newcommand{\diag}{\mathrm{diag}}
\newcommand{\op}{\mathrm{op}}
\newcommand{\iid}{i.i.d.\xspace}
% \renewcommand{\ls}{\ell}
\newcommand{\glcomp}{c}
\newcommand{\InfB}{\mathcal{I}_{\mathrm{B}}}
\newcommand{\InfF}{\mathcal{I}_{\mathrm{F}}}
\renewcommand{\emptyset}{\varnothing}
\newcommand{\what}{\wh{w}}
\newcommand{\pbar}{\bar{p}}
\newcommand{\phat}{\wh{p}}
\newcommand{\Meps}{M_{\veps}}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\amin}{\mathrm{argmin}}
\DeclareMathOperator*{\amax}{\mathrm{argmax}}
% \DeclareMathOperator*{\dom}{dom}
\DeclareMathOperator*{\E}{\mathbb{E}}
\newcommand{\mb}{\mathbb}
\newcommand{\mc}{\mathcal}
\usepackage[round, sort&compress]{natbib}
\bibliographystyle{plainnat}
\title{Hybrid RLHF}
\author{Avinandan Bose}
\date{March 2024}
\begin{document}
\maketitle
\begin{abstract}
Reinforcement Learning from Human Feedback (RLHF) is currently the most popular tool for language model alignment. Large Language Models typically have access to large offline datasets to train the models. However, the need to align language models for specific tasks means that these offline datasets are often insufficient to cover the all the requirements of the specific reward model. Using a purely online algorithm to collect preference data can be expensive when the offline dataset with incomplete coverage can be leveraged. Inspired by the recent theoretical advances in Hybrid RL, in this paper, we initiate the study of Hybrid RLHF. We show that hybrid RLHF requires less stringent coverage conditions than pure offline RLHF, and still improves the sample complexity of the online exploration.
\end{abstract}
% \section{Hybrid RLHF}
% \begin{enumerate}
% \item We have a reference language model $\pi_{\textrm{ref}} : \mathcal{X} \mapsto \mathcal{Y}$ obtained from supervised finetuning (also known as $\pi_{\textrm{sft}}$).
% \item All prompts (contexts) are i.i.d. samples from some distribution $\rho$ on $\mathcal{X}$. For a given reward $r$ and policy $\pi$, the objective is defined as the regularized value function:
% \begin{align*}
% J(r, \pi) = \mathbb{E}_{x \sim \rho, y \sim \pi(\cdot|x)}[r(x, y)] - \beta \mathbb{E}_{x \sim \rho}[D_{\rm KL}(\pi(\cdot | x) || \pi_{\rm ref}(\cdot | x))]
% \end{align*}
% \item An offline dataset is available : $\mathcal{D}_{\textrm{off}} = \{x^i, y_{+}^i, y_{-}^i\}_{i=1}^{N_{\textrm{off}}}$, where $x^i \sim \rho$ are i.i.d. samples and responses $y^i \sim \pi_b(\cdot|x^i)$ from a base behavior policy.
% \item We query $N_{\textrm{on}}$ prompts $x^t \sim \rho$. We get responses $y^t_1 \sim \pi_t^1(\cdot|x^t)$, $y^t_2 \sim \pi_t^2(\cdot|x^t)$ and $y^t_3\sim\pi_{\mathrm{ref}}(\cdot| x^t)$, where $\pi_t^1$ and $\pi_t^2$ are policies that we update sequentially for $t = 1, \ldots, N_{\textrm{on}}$. Potentially, $\pi^1_t$ is updated through optimistic exploration by techniques similar to \citet{xie2024exploratory} and $\pi^2_t$ is computed by imitating the behavior policy $\pi_{b}$ in the offline dataset, using techniques similar to \citet{li2024reward}.
% At the end, we will have two datasets, $\mathcal{D}_{\mathrm{explore}}=\{x^t, y^t_1, y^t_3\}_{t=1}^{N_{\mathrm{on}}}$ and $\mathcal{D}_{\mathrm{imitate}}=\{x^t, y^t_2, y^t_3\}_{t=1}^{N_{\mathrm{on}}}$.
% \item Using the idea from \cite{li2024reward} we use the datasets $\mathcal{D} = \mathcal{D}_{\textrm{off}} \cup \mathcal{D}_{\textrm{explore}} \cup \mathcal{D}_{\textrm{imitate}}$ to learn a final policy based on pessimistic offline RLHF.
% % (potentially only a part of it, unused in the previous sequential updates to maintain independence)
% \end{enumerate}
\section{Literature Review and Next Steps}
In algorithm~\ref{alg:hybrid_rlhf}, we highlight the schematic algorithm for hybrid RLHF.
Prior Work:
\begin{enumerate}
\item Analysis in offline RLHF (\cite{xiong2024iterative, cen2024value}) typically needs reward function to be linear, ie. $r(x, \tau) = \langle \theta, \phi(x, \tau)\rangle$ with mild assumptions on $\theta$ and known feature map $\phi$. Find a criterion which captures the coverage of all the data seen so far (all offline + online data collected until this step).
\item Section 4 in \cite{xiong2024iterative} considers the batch hybrid RLHF setting. Although their algorithm uses the offline dataset, both their theorems are essentially pure online or pure offline.
\item As explored in \cite{xie2024exploratory, xiong2024iterative} online RLHF can utilitise a non-symmetric algorithmic structure : have 2 policies $\pi_t^1$ and $\pi_t^2$. \cite{xiong2024iterative} use nice terminology to distinguish the roles of these 2 policies. Main agent $\pi_t^1$ : This policy's goal is to maximize the objective, and will be the output of the algorithm. Enchancer $\pi_t^2$ : This policy's goal is to explore trajectory spaces unseen before. Updates of $\pi_t^1, \pi_t^2$ should trade off between (i) maximizing likelihood of all the preference data collected so far, (ii) incentivising exploration.
Below is a summary of how lines 7,8 in Algorithm~\ref{alg:hybrid_rlhf} differ for prior work.
XPO (\cite{xie2024exploratory}) study only the online setting ($\mathcal{D}_{\rm off} = \emptyset$) and keep $\pi_t^2 = \pi_{\rm ref}$ fixed throughout. $\pi_t^1$ is updated via adding an exploration term $\sum_{i=1}^t \log \pi(\tau_i^2)$ to the DPO loss, ie.
\begin{align*}
\pi_{t+1}^1 = \argmin_{\pi \in \Pi} \bigg[\alpha \sum_{i=1}^t \log \pi(\tau_i^2) -\sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^t_{\rm on}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})}) \bigg] \bigg].
\end{align*}
In \cite{xiong2024iterative}, $\pi_t^1$ is updated by optimizing the DPO loss:
\begin{align*}
\pi_{t+1}^1 = \argmin_{\pi \in \Pi} \bigg[ -\sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^t_{\rm on} \cup \mathcal{D}_{\rm off}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})}) \bigg] \bigg],
\end{align*}
while $\pi_t^2$ is updated via maximizing the relative uncertainty to $\pi_t^1$. Under a linear reward model with feature map $\phi(x, \tau)$:
\begin{align*}
\pi_t^2 = \amax_{\pi \in \Pi_t}\|\mathbb{E}_{x \sim \rho}[\mathbb{E}_{\tau \sim \pi(\cdot | x)}\phi(x, \tau) - \mathbb{E}_{\tau \sim \pi_t^1(\cdot | x)}\phi(x, \tau)]\|_{\Sigma_t^{-1}},
\end{align*}
where $\Pi_t = \{\pi \in \Pi ~|~ \mathbb{E}_{x \sim \rho} D_{\rm KL}\left( \pi(\cdot | x) || \pi_t^1(\cdot | x) \right) \leq \|\mathbb{E}_{x \sim \rho} [\mathbb{E}_{\tau \sim \pi(\cdot | x)}\phi(x, \tau) - \mathbb{E}_{\tau \sim \pi_t^1(\cdot | x)}\phi(x, \tau)]\|_{\Sigma_t^{-1}}\}$ and $\Sigma_t = \sum_{i=1}^t(\phi(x_i, \tau_i^{+}) - \phi(x_i, \tau_i^{-}))(\phi(x_i, \tau_i^{+}) - \phi(x_i, \tau_i^{-}))^\top + \lambda \mathbb{I}$. The idea is to identify a policy ($\pi_{t}^2$) to maximize the information gain with respect to the main agent $\pi_t^1$, while maintaining a moderate KL divergence to it.
\end{enumerate}
\begin{algorithm}
\caption{Hybrid RLHF}
\begin{algorithmic}[1]
\STATE \textbf{Input:} Offline dataset $\mathcal{D}_{\rm off}$
\STATE \textbf{Initialize:} Main agent $\pi_t^1$, Enhancer $\pi_t^2$ appropriately, $\mathcal{D}^0_{\rm on} = \emptyset$
\FOR{$t = 1, \ldots, T$}
\STATE \textbf{Generate response pair:} $x_t \sim \rho$; $\tau_t^1 \sim \pi_t^1(\cdot|x_t), \tau_t^2 \sim \pi_t^2(\cdot|x_t)$.
\STATE \textbf{Label with Preference:} Label $(\tau_t^1, \tau_t^2)$ as $(\tau_t^{+}, \tau_t^{-})$ based on preference feedback.
\STATE \textbf{Update Online Dataset:} $\mathcal{D}^t_{\rm on} \gets \mathcal{D}^{t-1}_{\rm on} \cup (\tau_t^{+}, \tau_t^{-})$
\STATE \textbf{Update Main Agent:} Update $\pi_t^1$ to maximize likelihood of preferences seen so far + regularization terms
\STATE \textbf{Update Enhancer:} Update $\pi_t^2$ so future trajectories satisfy (i) diverse from the trajectories already seen in $\mathcal{D}_{\rm off} \cup \mathcal{D}^t_{\rm on}$, (ii) diverse from trajectories to be generated from $\pi_t^1$.
\ENDFOR
\STATE \textbf{Return} $\pi_T^1$.
\end{algorithmic}
\label{alg:hybrid_rlhf}
% \vskip -0.3in
\end{algorithm}
\subsection{Open Problems}
\begin{enumerate}
\item Online exploration to cover whatever hasn't been seen in the offline dataset hasn't been theoretically studied yet.
\end{enumerate}
\subsection{Goals}
\begin{enumerate}
\item Give a result on the suboptimality of $\pi_T^1$ in terms of $T$ and some coverage condition of $\mathcal{D}_{\rm off}$. Both online and offline RLHF works have established suboptimality rates of $\Tilde{\mathcal{O}}(\sqrt{1/T})$. We hope to show, in a similar vein as in results in hybrid RL in model based RL a constant factor improvement whose value can be very significant depending on the coverage conditions of the initial offline dataset.
\item Identify conditions when hybrid learning is better than pure online ($\mathcal{D}_{\rm off} = \emptyset$) or pure offline ($T=0$).
\item Identify when few shot online fine tuning can be very useful etc.
\end{enumerate}
\subsection{Next Steps}
\begin{enumerate}
\item In XPO (\cite{xie2024exploratory})adding $\sum_{i=1}^t \log \pi(\tau_i^2)$ to the loss term promotes exploration by moving $\pi_t^1$ towards a policy that generates more diverse trajectories than that seen already. Same term is used in VPO (\cite{cen2024value}), but instead of a finite sample average, they take expectation wrt $x \sim \rho$ and $y \sim \pi_{\rm cal}(\cdot|x)$. In fact choosing $\pi_{\rm cal} = \pi_{\rm ref}$ gives the lowest regret bound in their paper. This is almost equivalent to XPO, the difference being in XPO there is a sample average. \textbf{Proposal:} Can we add the trajectories from the offline dataset (or just the negative labelled ones) to this objective and promote coverage with a warm start?
\zhihan{I think we actually have to add trajectories from the offline dataset to this objective because Algorithm \ref{alg:hybrid_rlhf} only outputs $\pi^1$. That is, it will make no sense to use hybrid RLHF if the training of $\pi^1$ itself does not include anything from $\mc{D}_{\mathrm{off}}$. Otherwise according to Algorithm \ref{alg:hybrid_rlhf}, $\pi^1$ will only learn the part that is poorly covered by $\mc{D}_{\mathrm{off}}$. However, although this is intuitively necessary, it may cause some technical difficulty because it requires using the same $\mc{D}_{\mathrm{off}}$ (instead of splitting the offline dataset) in the update of both $\pi^1$ and $\pi^2$ because we hope $\pi^2$ to avoid the region that is well covered by $\mc{D}_{\mathrm{off}}$ and learned by $\pi^1$. This will make their update strongly dependent.}\avi{We can start by keeping $\pi_t^2$ fixed as $\pi_{\rm ref}$ as in XPO, or $\pi_t^1 = \pi_t^2$ as in VPO to avoid this complication in the analysis. Also I'm proposing we do the exploration penalty in both the offline + online (trajectories from $\pi_t^2$.}
\item Use the DPO objective on $\mathcal{D}_{\rm off} \cup \mathcal{D}^t_{\rm on}$, instead of just on $\mathcal{D}^t_{\rm on}$. This intuitively sounds better and in the linear reward model case seems like adding a $\Sigma_{\rm off} = \sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}_{\rm off}}(\phi(x_i, \tau^{+}) - \phi(x_i, \tau^{-}))(\phi(x_i, \tau^{+}) - \phi(x_i, \tau^{-}))^\top$ to some of the analysis.
\item Let's try to get a suboptimality bound using either VPO / XPO analysis styles using the said modifications. We will most likley have a term like $\Sigma_{\rm off}$ show up in the bound.
\zhihan{I think we can first simplify XPO's analysis into bandit formulation, which may make applying their techniques easier. In particular, as XPO is completely in bandit formulation from a pure algorithmic perspective, I guess analyzing it through an MDP formulation is probably an overkill, making things unnecessarily complicated.} \avi{XPO's analyis avoids the MDP formulation though. Atleast VPO's does, XPO is a bit more generally applicable.}
\item Assume $\mathcal{D}_{\rm off}$ was collected using behavior policy $\pi_b$, and based on an appropriate coverage condition, write the bound in terms of $n_{\rm off}$ and $T$ and try to study the extreme cases.
\item \textbf{Proposal:} \begin{align*}
\pi_{t+1}^1 &= \argmin_{\pi \in \Pi} \bigg[\alpha_1 \sum_{i=1}^t \log \pi(\tau_i^2) + \alpha_2 \sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}_{\rm off}}(\log \pi(\tau^{+}) + \log \pi(\tau^{-}))) \\& -\sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^t_{\rm on} \cup \mathcal{D}_{\rm off}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})}) \bigg] \bigg].
\end{align*}
We can either fix $\pi_t^2 = \pi_{\rm ref}$ and follow analysis of XPO, or set $\pi_t^1 = \pi_t^2$ and follow VPO's analysis to see the modifications. Infact, the proof structures of VPO and XPO are very similar and the general version of XPO sets $\pi_{t+1}^2$ as some sampling strategy over $\{\pi_1^1, \ldots, \pi_t^1\}$.
\end{enumerate}
\section{Benefits of Warm Start}
The aim of this section is to understand how the coverage provided from the offline preference dataset influences the sample complexity of the online learning algorithm. To begin with, we use the similar setup as in XPO \citep{xie2024exploratory} and see how their results are modified with addition of the offline dataset to the online learning algorithm. Restating the assumptions made in their paper and their key concentration results. note that VPO \citep{cen2024value} also use the same concentration inequalities however they study it in a linear MDP setting. Let's discuss next steps on what could be further modified in the algorithm and the kind of the results we are pushing for. \avi{A lot of text is verbatim from XPO, will modify later.}
\subsection{Assumptions}
We restate some standard statistical assumptions. The first assumption states that the policy class $\Pi$ is powerful enough to represent the optimal KL-regularized policy.
\begin{assumption}[Policy realizability]
\label{ass:realizability}
The policy class $\Pi$ satisfies $\pistarb\in\Pi$.
\end{assumption}
Policy realizability is a minimal assumption for sample-efficient reinforcement learning
\citep{agarwal2019reinforcement,lattimore2020bandit,foster2023foundations};
through \ref{eq:reward_v}, it is equivalent to a form of reward/value realizability.
For language modeling, $\Pi$ will typically correspond to a class of language
models with fixed architecture but variable weights. Next, we make a regularity assumption on the
policies in $\Pi$ \citep{rosset2024direct}.
\begin{assumption}[Bounded density ratios]\citep{xie2024exploratory}
\label{ass:vmax}
For all $\pi\in\Pi$ and trajectories
$\tau=(s_1,a_1),\ldots,(s_H,a_H)$,
\begin{align}
\left|\log\left|\frac{\pi(\tau)}{\piref(\tau)}\right|\right| \leq\frac{\Vmax}{\beta}.
\end{align}
\end{assumption}
% Note that $\Vmax$ is measurable and controllable in practice; our
% guarantees scale polynomially with this parameter. For log-linear policies where
% $\pi(a\mid{}s)\propto\exp\prn*{\nicefrac{f(s,a)}{\beta}}$, we
% expect $\Vmax\approxleq{}\Rmax$.
\begin{algorithm}
\caption{Hybrid RLHF}
\begin{algorithmic}[1]
\STATE \textbf{Input:} Offline dataset $\mathcal{D}_{\rm off}$ of size $N_{\rm off}$, sampling strategy $\pi_{\rm samp}$, hyper-parameters, $\alpha$, $\gamma \in \mathbb{N}$.
\STATE \textbf{Initialize:} $\pi_t, \Tilde{\pi}_t \leftarrow \piref$ appropriately, $\mathcal{D}^0_{\rm on} = \emptyset$
\FOR{$t = 1, \ldots, T$}
\STATE \textbf{Generate response pair:} $x_t \sim \rho$; $\tau_t \sim \pi_t(\cdot|x_t), \taut_t \sim \piref(\cdot|x_t)$.
\STATE \textbf{Label with Preference:} Label $(\tau_t, \taut_t)$ as $(\tau_t^{+}, \tau_t^{-})$ based on preference feedback.
\STATE \textbf{Update Online Dataset:} $\mathcal{D}^t_{\rm on} \gets \mathcal{D}^{t-1}_{\rm on} \cup (\tau_t^{+}, \tau_t^{-})$
\STATE \textbf{Update Offline dataset minibatch:} Sample $\gamma$ pairs from $\mathcal{D}_{\rm off}$ uniformly randomly with replacement $\mathcal{D}_{\rm off}^{(t)}$
\STATE \textbf{Update Hybrid dataset:} $\mathcal{D}_{\rm hyb}^{(t)} \gets \mathcal{D}_{\rm on}^{(t)} \cup \mathcal{D}_{\rm off}^{(t)}$
\STATE \textbf{Update Optimism Dataset: } Compute $\mathcal{D}^{(t)}_{\rm opt}$ of $t + \gamma$ samples from $\Tilde{\pi}_t$.
\STATE \textbf{Update Main Agent:} Update $\pi_t$ to maximize likelihood of preferences seen so far + regularization terms
\begin{align*}
&\pi_{t+1} = \argmin_{\pi \in \Pi} \bigg[\alpha \sum_{\tau \in \mathcal{D}^t_{\rm opt}} \log \pi(\tau)
-\sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^{(t)}_{\rm hyb}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})})\bigg]\bigg],
\end{align*}
% \STATE \textbf{Update Enhancer:} Update $\pi_t^2$ so future trajectories satisfy (i) diverse from the trajectories already seen in $\mathcal{D}_{\rm off} \cup \mathcal{D}^t_{\rm on}$, (ii) diverse from trajectories to be generated from $\pi_t^1$.
% \STATE \textbf{Update sampling policy:} $\Tilde{\pi}_{t+1} \gets \pi_{\rm samp}(\pi_1, \ldots, \pi_{t+1})$
\ENDFOR
\STATE \textbf{Return} $\widehat{\pi} = \amax_{\pi \in \{\pi_1, \ldots, \pi_T\}} J_{\beta}(\pi)$.
\end{algorithmic}
\label{alg:hybrid_rlhf_warm_start}
% \vskip -0.3in
\end{algorithm}
\subsection{Main Results}
% Our most general sample complexity guarantee for \ref{alg:hybrid_rlhf_warm_start} is stated in terms of the following
Let's first look at the preference-based analogue of the \emph{Sequential Extrapolation Coefficient} (SEC) from \citet{xie2023role} (also known as an eluder coefficient or decoupling coefficient
\citep{zhong2022gec,ye2024theoretical}) and how it changes in the hybrid RLHF case. Recall that for a trajectory
$\tau=(s_1,a_1),\ldots,(s_H,a_H)$, we define
\begin{equation}
\pi(\tau)=\prod_{h=1}^{H}\pi(a_h\mid{}s_h),\quad\textrm{and}\quad{}r(\tau)=\sum_{h=1}^{H}r(s_h,a_h).
\end{equation}
For a pair of policies $\pi$ and $\Tilde{\pi}$, we define $\pi\otimes\Tilde{\pi}$ as
the joint policy that, given $s_1$, samples $\tau\sim{}\pi\mid{}s_1$
and $\taut \sim{} \Tilde{\pi} \mid{}s_1$. We write
$(\tau,\taut)\sim{}\pi\otimes\Tilde{\pi} \mid{}s_1$ as shorthand for this process.
Define $g^{(\pi)}(\tau, \taut) = \left[\beta \log \frac{\pi^{(t)}(\tau)}{\piref(\tau)} - r(\tau) - \beta \log \frac{\pi^{(t)}(\taut)}{\piref(\taut)} + r(\taut) \right]$, $\Tilde{\pi}\ind{t}= \samp(\pi\ind{1},\ldots,\pi\ind{t})$, and $\bmu\ind{t} = \frac{1}{t-1}\sum_{i=1}^{t-1} \pi\ind{i}\otimes \Tilde{\pi}\ind{i}$, with the convention that $\mu\ind{1}$ is arbitrary.
We first define the online sequential exploration coefficient for online RLHF \citep{xie2024exploratory}.
\begin{definition}
\begin{align}
{\rm SEC}_{\rm RLHF}(\Pi, T, \beta, \pi_{\rm samp}) = \sup_{\pi^{(1)}, \ldots, \pi^{(T)} \in \Pi} \sum_{t=1}^T \frac{\left(\mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^{(t)} | s_1, \taut \sim \tilde{\pi}^{(t-1)}| s_1}[g^{(\pi^{(t)})}(\tau, \taut)]\right)^2}{\Vmax^2 \vee \left[(t - 1) \cdot \mathbb{E}_{s_1 \sim \rho, (\tau, \taut) \sim \mu^{(t)} | s_1} [\left(g^{(\pi^{(t)})}(\tau, \taut)\right)^2 ] \right]}.
\end{align}
\end{definition}
Next, we introduce a quantity to capture the coverage of the offline dataset in terms of any arbitrary policy $\pi \in \Pi$:
\begin{align*}
C^{(\pi)}_{\rm off} = \frac{1}{N_{\rm off}}\sum_{(\tau_{+}, \tau_{-}) \in \mathcal{D}_{\rm off} }\left(g^{(\pi)}(\tau_{+}, \tau_{-})\right)^2.
\end{align*}
Below we define the SEC coefficient in the Hybrid RLHF case:
\begin{definition}
\begin{align}
\sup_{\pi^{(1)}, \ldots, \pi^{(T)} \in \Pi} \sum_{t=1}^T \frac{\left(\mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^{(t)} | s_1, \taut \sim \tilde{\pi}^{(t-1)}| s_1}[g^{(\pi^{(t)})}(\tau, \taut)]\right)^2}{\Vmax^2 \vee \left[(t - 1) \cdot \mathbb{E}_{s_1 \sim \rho, (\tau, \taut) \sim \mu^{(t)} | s_1} [\left(g^{(\pi^{(t)})}(\tau, \taut)\right)^2 + \gamma \cdot C^{(\pi^{(t)})}_{\rm off} ] \right]}.
\end{align}
\end{definition}
\begin{theorem}\label{thm:main_general}
Suppose Assumptions \ref{ass:realizability},\ref{ass:vmax} hold.
For any $\beta>0$ and $T\in [N]$, if we set $\alpha=c\cdot\frac{\beta}{(\Vmax + \Rmax)e^{2 \Rmax})}\cdot\sqrt{\frac{\log(|\Pi|T\delta^{-1})\log(T)}{(T + \gamma)\cdot \coefffull}}$
for an absolute constant $c>0$,
then \ref{alg:hybrid_rlhf_warm_start} ensures that
with probability at least $1-\delta$,
\begin{align*}
J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi})
\approxleq
(\Vmax+\Rmax)e^{2\Rmax}\cdot\sqrt{\frac{(1 + \gamma / T)\coefffull\log(|\Pi|T\delta^{-1})\log(T)}{T}}.
\end{align*}
\end{theorem}
Note that $\gamma=0$, gives the pure online case which is studied in XPO \citep{xie2024exploratory}.
Below we take the case of linear MDPs to understand the suboptimality gaps of hybrid, online and offline RLHF.
\begin{example}(Linear MDP)
In a Linear MDP (\citep{jin2020provably}), the transition probability as well as the reward are linear functions of a known feature map $\phi(s,a) \in \mathbb{R}^d$ where $\|\phi\|_2 \leq 1$ and
\begin{align*}
P(s' | s, a) = \phi(s, a)^\top \mu(s') \quad r(s, a) = \phi(s, a)^\top \nu,
\end{align*}
where $\mu(s')$ is an unknown feature map with $\|\sum_{s'} \mu(s')\|_2 \leq \sqrt{d}$, and $\nu \in \mathbb{R}^d$ is an unknown parameter with $\|\nu\|_2 \leq 1$.
% where $d_{\rm hyb} = $
\end{example}
\begin{proof}
Define $\Lambda_{\rm off} = \frac{1}{N_{\rm off}}\sum_{(\tau, \taut) \in \mathcal{D}_{\rm off}}(\phi(\tau) - \phi(\taut))(\phi(\tau) - \phi(\taut))^\top$. Since $\|\phi\|_2 \leq 1$ by definition of Linear MDPs, we have $\textrm{Trace}(\Lambda_{\rm off}) \leq 4$. Using the elliptical potential theorem, we derive:
% \begin{align*}
% \coefffull &\leq 2d \left( \log(d \cdot |\gamma \Lambda_{\rm off} + \Vmax^2 + T|^{1/d}) - \log(d\cdot\rm{det}(\gamma\Lambda_{\rm off} + \Vmax^2\mathbf{I})^{1/d})\right)\\
% &\leq 2d \left( \log(\gamma\rm{Trace}(\Lambda_{\rm off}) + \Vmax^2 + T) - \log(d\cdot\rm{det}(\gamma\Lambda_{\rm off} + \Vmax^2\mathbf{I})^{1/d})\right)\\
% &\leq 2d \left(\log\left(1 + \frac{4\gamma + T}{\Vmax^2}\right) - \log\left(d\cdot |\frac{\gamma}{\Vmax^2} \Lambda_{\rm off} + \mathbf{I}|^{1/d}\right)\right) \\
% &\leq 2d \left(\log\left(1 + \frac{4\gamma + T}{\Vmax^2}\right) - \frac{1}{d}\sum_{i \in [d]}\log\left(\frac{\gamma}{\Vmax^2}d \lambda_{\rm off}^{(i)} + d\right)\right) \\
% &\leq 2d \left(\log\left(1 + \frac{4\gamma + T}{\Vmax^2}\right) - \frac{1}{d}\sum_{i \in [d]}\log\left(1 + \frac{\gamma}{\Vmax^2}d \lambda_{\rm off}^{(i)}\right)\right)\\
% &= 2d \left(\frac{1}{d}\sum_{i \in [d]} \left(\log\left(1 + \frac{4\gamma + T}{\Vmax^2}\right) - \log\left(1 + \frac{\gamma}{\Vmax^2}d \lambda_{\rm off}^{(i)}\right)\right)\right),\\
% \end{align*}
\begin{align*}
\coefffull &\leq 2 \left( \log( |\gamma \Lambda_{\rm off} + (\Vmax^2 + 4T)\mathbf{I}|) - \log(\rm{det}(\gamma\Lambda_{\rm off} + \Vmax^2\mathbf{I}))\right)\\
&= 2 \sum_{i \in [d]}\left(\log\left(\gamma \lambda_{\rm off}^{(i)} + \Vmax^2 + 4T\right) - \log\left(\gamma \lambda_{\rm off}^{(i)} + \Vmax^2\right)\right)\\
&\leq 2 \sum_{i \in [d]}\left(\log\left(1 + \frac{4T}{\max(\Vmax^2, \gamma \lambda_{\rm off}^{(i)})}\right)\right)\\
\end{align*}
where $\lambda_{\rm off}^{(i)}$ is the $i^{\rm th}$ eigenvalue of $\Lambda_{\rm off}$.
\textbf{Purely Online:} Setting $\gamma=0$, gives us the purely online case, and $\rm{SEC}_{\rm RLHF}(\Pi, T, \beta) \leq 2d \cdot \log\left(1 + \frac{4T}{\Vmax^2}\right)$.
\begin{align*}
J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi})
\approxleq \tilde{\mathcal{O}}\left(\Rmax e^{2\Rmax} \sqrt{\frac{d^2}{T}}\right),
\end{align*}
\textbf{Hybrid:} Choosing $\gamma = \mathcal{O}(T)$, only terms where $\lambda_{\rm off}^{(i)} \leq \Omega(\frac{1}{T})$ remain in the summand, corresponding to the directions where the offline dataset has poor exploration. Let $d_{\rm hyb}$ denote the number of indices in $[d]$ such that $\lambda_{\rm off}^{(i)} \leq \Omega(\frac{1}{T})$. Therefore we get $\coefffull \leq \mathcal{O}\left(d_{\rm hyb}\log\left(1 + \frac{4T}{\Vmax^2}\right)\right)$.
Hence, the SEC scales as the effective number of dimensions yet to be explored sufficiently.
\begin{align*}
J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi})
\approxleq \tilde{\mathcal{O}}\left(\Rmax e^{2\Rmax} \sqrt{\frac{d\cdot d_{\rm hyb}}{T}}\right),
\end{align*}
\end{proof}
\subsection{Concentration Lemmas}
Let $\mathbb{P}_{\rm off}$ denote the nominal distribution of trajectory pairs in the offline dataset $\mathcal{D}_{\rm off}$. Define $\bm{\mu}^{(t)}_{\rm hyb} = \frac{1}{t - 1 + \gamma} (\sum_{i=1}^{t-1} \pi^{(i)} \otimes \tilde{\pi}^{(i)} + \gamma \mathbb{P}_{\rm off})$.
\begin{lemma}[Concentration for Algorithm \ref{alg:hybrid_rlhf_warm_start}]
Suppose that Assumptions \ref{ass:realizability},\ref{ass:vmax} hold. Then
Algorithm \ref{alg:hybrid_rlhf_warm_start} guarantees that with probability at least
$1-\delta$, for all steps $t\in [T]$,
\begin{align*}
&\alpha\cdot\E_{s_1\sim\rho,\tau\sim\Tilde{\pi}\ind{t-1}}{[\log(\pi\ind{t}(\tau))-\log(\pistarb(\tau))]} + \kappa \cdot\left[\E_{s_1\sim\rho{},(\tau,\taut)\sim{}\bm{\mu}\ind{t}_{\rm hyb}\mid{}s_1}{[f_{\pi\ind{t}}(\tau,\taut)
- f_{\pistarb}(\tau,\taut)]^2}\right] \notag \\&\leq \frac{2\log(2|\Pi|T\delta^{-1})}{\gamma + t-1} + \frac{\alpha}{\beta}\Vmax\sqrt{\frac{2^4\log(2|\Pi|T\delta^{-1})}{\gamma + t-1}}, \end{align*}
for $\kappa = (8(\Rmax+\Vmax)e^{2\Rmax})^{-2}$.
\end{lemma}
\begin{proof}
% Follows from proof of lemma C.5 in \citep{xie2024exploratory} with the following differences:
\begin{enumerate}
\item Apply Azuma Hoeffding for the first term with its sample average over $t + \gamma$ terms.
\item Use the definition of $\bm{\mu}^{(t)}_{\rm hyb}$ to see the dataset $\mathcal{D}_{\rm hyb}^{(t)}$ is a sequence adapted to the filtration $\mathcal{F}^{(\gamma + t)} = \sigma(\mathcal{D}_{\rm off}, \ldots, \mathcal{D}_{\rm off}, (\tau^{(1)}, \tilde{\tau}^{1}), \ldots, (\tau^{(t-1)}, \tilde{\tau}^{(t-1)}))$.
\item Then apply the Martinagle Chernoff Theorem for the second term.
\end{enumerate}
\end{proof}
\subsection{Examples}
We consider a linear MDP $\mathcal{M}$ where we denote $\pi^\ast_\beta = \amax_{\pi \in \Pi} J_{\beta}(\pi)$.
Let $\Lambda_{\rm off}$ be the offline dataset of rank $d - 1$, and satisfying:
\begin{align*}
\Lambda_{\rm off} \mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^\ast_\beta}[\phi(\tau)] = 0.
\end{align*}
A pessimistic offline algorithm run on $\Lambda_{\rm off}$ is highly like to yield a policy
in the space $\Pi_{\rm off}$ which satisfies:
\begin{align*}
\langle \mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^\ast_\beta}[\phi(\tau)], \mathbb{E}_{s_1 \sim \rho, \tau \sim \pi} [\phi(\tau)] \rangle = 0 \; \forall \pi \in \Pi_{\rm off}.
\end{align*}.
We make the following assumption on $\mathcal{M}$:
\begin{align*}
J_{\beta}(\pi^\ast_\beta) - \amax_{\pi \in \Pi_{\rm off}} J_{\beta}(\pi) \geq \Delta,
\end{align*}
where we choose $\Delta = \Rmax / 2$ say.
\section{TODOs}
\avi{
\begin{enumerate}
\item Utilize the online minimax lower bound for contextual bandits to be $\mathcal{O}(d^2/\epsilon^2)$. This shows that hybrid rlhf can go below this lower bound.
\item Show instance where single policy concentrability fails, thus pessimisitc offline policy doesn't have suboptimality guarantees, yet hyrbid case sees significant improves sample complexity over pure online. I'm guessing we have to show an offline case, where the policy from offline data has a poor (large) lower bound?
% \item Derive the bound for tabular case as well
\item Run experiments
\end{enumerate}
}
% Let the sample complexities to obtain an $\epsilon-$optimal solution for bandit feedback be $\mathcal{O}(g(\epsilon))$, and that for duelling feedback be $\mathcal{O}(h(\epsilon))$. Let the costs associated with each sample be $c_b$ and $c_f$ for the two types of feedback respectively. We wish to devise an algorithm to combine these feedback types to derive a cost sensitive sample complexity.
%Maryam: Another potential optimization idea that might be relevant: making progress on minimizing a cost with only one-bit information (about the location of the optimum relative to the current point) at every iteration. This comes up for example in cutting-plane methods (at any iterate $x_k$, determine the `better' side of a hyperplane through $x_k$ at every iteration). Feels a bit similar to `preference' feedback/oracle though not sure it's helpful.
% \subsection{Choosing the right definition of SEC}
% % \begin{definition}
% % \begin{align}
% % \sup_{\pi^{(1)}, \ldots, \pi^{(T)} \in \Pi} \sum_{t=1}^T \frac{\left(\mathbb{E}_{s_1 \sim \rho, \tau \sim \pi^{(t)} | s_1, \taut \sim \tilde{\pi}^{(t-1)}| s_1}[g^{(\pi^{(t)})}(\tau, \taut)]\right)^2}{\Vmax^2 \vee (t-1) \cdot \left[ \frac{t - 1}{N_{\rm off} + t - 1} \cdot \mathbb{E}_{s_1 \sim \rho, (\tau, \taut) \sim \mu^{(t)} | s_1} [\left(g^{(\pi^{(t)})}(\tau, \taut)\right)^2 + \frac{N_{\rm off}}{N_{\rm off} + t - 1} \cdot C^{(\pi^{(t)})}_{\rm off} ] \right]}.
% % \end{align}
% % \end{definition}
% % The SEC value in the second definition is larger than the SEC value in the first definition. However, the regret term in the first case has an additional factor $\sqrt{1 + N_{\rm off} / T}$.
% Can we analyze at least one of these two definitions in the linear MDP settings and see which is easier to have a more intuitive bound? (see Appendix C.8, proof of example C.2 \citep{xie2024exploratory}). We will most likely need a bound on $\inf_{\pi \in \Pi} C_{\rm off}^{(\pi)}$ or something similar to impose coverage condition on the offline dataset and see how the bound changes. Note the following in the case of linear MDP:
% \begin{align*}
% C_{\rm off}^{(\pi)} = \|\theta_f - w_f\|_{\Lambda_{\rm off}},
% \end{align*}
% where $\Lambda_{\rm off} = \frac{1}{N_{\rm off}}\sum_{(\tau, \taut) \in \mathcal{D}_{\rm off}}(\phi(\tau) - \phi(\taut))(\phi(\tau) - \phi(\taut))^\top$. Can we expect a bound in terms of $\tilde{\mathcal{O}}(d - {\rm rank}(\Lambda_{\rm off}))$?
% \begin{align*}
% \rm{SEC} \leq 2\log \frac{\rm{det}(N_{\rm off} \Lambda_{\rm off} + \Sigma_T)}{\rm{det}(N_{\rm off}\Lambda_{\rm off})}
% \end{align*}
% Note that $\|\phi\| \leq 1$, therefore $\Sigma_T \preceq T \mathbf{I}$. Let $\{\lambda_1, \ldots, \lambda_d \}$ be the the eigenvalues of $\Lambda_{\rm off}$, then $\rm{det}(N_{\rm off} \Lambda_{\rm off} + \Sigma_T) \leq \prod_{i=1}^d (N_{\rm off}\lambda_i + T)$ . Thus we can write:
% \begin{align*}
% \rm{SEC} \leq 2\log \frac{\rm{det}(N_{\rm off} \Lambda_{\rm off} + \Sigma_T)}{\rm{det}(N_{\rm off}\Lambda_{\rm off})} \leq \sum_{i \in [d]} \log(1 + \frac{T}{N_{\rm off} \lambda_i}).
% \end{align*}
% Assume $\lambda_i \geq \frac{1}{N_{\rm off}} \; \forall i \in [d]$. (i) Directions in which offline dataset has poor coverage, the log terms can be upper bounded by $\log(1 + T)$, (ii) directions in whic coverage is very good : $N_{\rm off} \lambda_i >> T$, these terms are negligible. Thus we get $\mathcal{O}((d - {\rm rank}(\Lambda_{\rm off})) \log(T))$.
% For a linear MDP, the SEC coefficient is derived as follows:
% \begin{align*}
% SEC \leq 2\log{\frac{\rm{det}(\gamma \Lambda_{\rm off} + (\Vmax^2 + T) \mathbf{I})}{\rm{det}(\gamma \Lambda_{\rm off} + \Vmax^2 \mathbf{I})}} \leq 2d \log(\gamma\rm{Trace}(\Lambda_{\rm off}) + \Vmax^2 + T) - 2d \log(d\cdot\rm{det}(\gamma\Lambda_{\rm off} + \Vmax^2\mathbf{I})^{1/d})
% \end{align*}
% \begin{corollary}
% \end{corollary}
% \textbf{Remark: }
% \begin{align*}
% \lim_{N_{\rm off} \rightarrow \infty}
% \end{align*}
\bibliography{References}
% \bibliographystyle{plain}
\end{document}
\section{Old Stuff}
\section{Mixture of Feedback types}
We consider a practical example where a language model (LM) $\pi_\theta$ guides auto text completion when replying to text messages or emails. Suppose LM generates response pair $y_1, y_2 \sim \pi_\theta(\cdot|x)$ given context $x$. A user may choose one of these recommendations (say $y_1$) thereby providing preference feedback, or may choose to reject all of the recommendations and type the response themselves (say $y \sim \pi^\ast(\cdot|x)$) thereby providing expert feedback, where $\pi^\ast$ denotes the optimal unknown LM. We would like to learn the parameterized LM $\pi_{\widehat{\theta}}$ such that generalization loss defined below is minimized:
\begin{align*}
\mathcal{L}(\theta) = \mathbb{E}_{x \sim \mathcal{D}}\mathbb{D}_{\rm KL}(\pi_\theta(\cdot | x) || \pi^\ast(\cdot | x))
\end{align*}
% where $\pi_{\theta^\ast}$ is the optimal LM.
% \begin{align*}
% \mathbb{E}_{x \sim \mathcal{D}}[\mathbb{E}_{y \sim \pi_{\theta^\ast}(\cdot|x)}r^\ast(x, y) - \mathbb{E}_{y \sim \pi_{\widehat{\theta}}(\cdot|x)}r^\ast(x, y)].
% \end{align*}
\noindent\textbf{Data generation under preference feedback:}
\begin{align*}
\mathbb{P}[y_1 \succ y_2 | x] = \sigma\left(\beta\left(\frac{\log\pi^\ast(y_1|x)}{\log\pi_{\rm ref}(y_1|x)} - \frac{\log\pi^\ast(y_2|x)}{\log\pi_{\rm ref}(y_2|x)}\right)\right)
\end{align*}
\textbf{Data generation under expert feedback:}
\begin{align*}
\mathbb{P}[y | x] = \pi^\ast(y|x)
\end{align*}
% \begin{align*}
% \mathcal{L}(\theta) = \mathbb{E}_{x \sim \mathcal{D}}[\mathbb{E}_{y \sim \pi_{\theta}(\cdot|x)}r^\ast(x, y) - \beta \mathbb{D}_{\rm KL}(\pi_{\theta}(\cdot|x) || \pi_{\rm ref}(\cdot|x))].
% \end{align*}
% \begin{align*}
% \mathcal{L}(\theta) = \beta \mathbb{E}_{x \sim \mathcal{D}}[\mathbb{E}_{y \sim \pi_{\theta}(\cdot|x)}r^\ast(x, y) - \beta \mathbb{D}_{\rm KL}(\pi_{\theta}(\cdot|x) || \pi_{\rm ref}(\cdot|x))].
% \end{align*}
\textbf{Cost per type of feedback:} Suppose the cost for every preference feedback is $c_p$ and the cost per expert feedback is $c_e$.
\textbf{Budget:} If we draw $n_p$ preference pairs and $n_e$ expert responses, the total cost incurred is $n_p c_p + n_e c_e$. Let the total budget be $C_T$.
% However one typically has a finite budget to compute $\pi_{\widehat{\theta}}$.
\textbf{Objective:} Propose a practical online learning scheme for the LM (essentially evolution of $\theta$), choosing when to get preference vs expert feedback, such that the output generalization loss on the output $\widehat{\theta}$ is minimized given a fixed learning budget.
Read Aarti's paper on combining dueling feedback and bandit feedback.
\section{Ideas from 4/15}
\subsection{General Framework}
We wish to learn a parametric function (eg. probability density function, reward function) $\pi_{\theta}$ that returns a score (eg. probability density, reward) for a response $y \in \mathcal{Y}$ conditioned on input context $x \in \mathcal{X}$, where $x$ comes from some distribution $\mathcal{D}$ with support $\mathcal{X}$. Let $\theta^\ast$ denote the optimal parameter. The suboptimality gap for a given $\theta$ is given by some:
\begin{align*}
\textrm{SubOpt}(\theta) = \mathbb{E}_{x \sim \mathcal{D}_{\mathcal{X}}} [l(\pi_{\theta^\ast}(\cdot|x), \pi_\theta(\cdot|x))].
\end{align*}
A few examples of $l$ include (i) KL-Divergence: $\mathbb{D}_{\rm KL}(\pi_\theta(\cdot | x) || \pi_{\theta^\ast}(\cdot | x))$ when $\pi_\theta(\cdot|x)$ is a probability density function, (ii) Difference between rewards of true and predicted optimal arms: denote $y' = \amax_{y \in \mathcal{Y}} \pi_{\theta}(\cdot|x), y^\ast = \amax_{y \in \mathcal{Y}} \pi_{\theta^\ast}(\cdot|x)$, then $ l(\pi_{\theta^\ast}(\cdot|x), \pi_\theta(\cdot|x)) = \pi_{\theta^\ast}(y^\ast) - \pi_{\theta^\ast}(y')$.
\subsection{Feedback Types}
We outline the methods to collect data, that enables us to learn $\pi_\theta$.
\noindent\textbf{Data generation under preference feedback:} We assume access to a function $\pi_{\rm ref}$ that allows us to draw responses $y \sim \pi_{\rm ref}(\cdot|x)$ given an input context $x$. Examples of $\pi_{\rm ref}$ include a non-perfect robot simulator or a SFT (supervised fine tuned) language model (LM). Upon drawing 2 i.i.d. samples $y_1, y_2$ from $\pi_{\rm ref}(\cdot|x)$, we ask a human (in LMs, GPT-3.5 turbo is considered human when reporting preferences) to report which of the responses is better (more likely to be drawn from $\pi_{\theta^\ast}(\cdot|x)$. A concrete example for the LLM preference alignment loss with a BTL preference model would look like:
\begin{align*}
\mathbb{P}[y_1 \succ y_2 | x] = \sigma\left(\beta\left(\frac{\log\pi^\ast(y_1|x)}{\log\pi_{\rm ref}(y_1|x)} - \frac{\log\pi^\ast(y_2|x)}{\log\pi_{\rm ref}(y_2|x)}\right)\right).
\end{align*}
The cost of such a preference query to a human is quite cheap, and let us denote it by $c_p$.
\textbf{Data generation under expert feedback:} On the other hand a more direct way would be to ask a human/expert (GPT4 is considered gold standard for LM human demonstration) to demonstrate a response on an input context:
\begin{align*}
\mathbb{P}[y | x] = \pi_{\theta^\ast}(y|x).
\end{align*}
While such a response is very informative, it can be very costly and let us denote this cost by $c_e$.
Such cost disparities are very prevalent in LMs, example querying GPT3.5 Turbo with preference feedback is much cheaper than asking GPT 4 to produce an expert demonstration.
\subsection{Cost Sensitive Analysis}
Suppose we have an algorithm which guarantees with probability at least $1 - \delta$, on us estimating $\widehat{\theta}$ that achievies $\textrm{SubOpt}(\widehat{\theta}) \leq \epsilon$, if we collect $N_p \geq p(\epsilon, \delta)$ preference queries, and likewise an algorithm with same guarantees if we collect $N_e \geq e(\epsilon, \delta)$ expert queries. Note that the respective costs to achieve these high probability error bounds are $C_p \geq c_p p(\epsilon, \delta)$ and $C_e \geq c_e e(\epsilon, \delta)$ respectively.
Our goal to devise an algorithm which can minimize the overall cost complexity if we can use both types of queries as feedback, i.e. we want to have the same guarantee with $N'_p, N'_e$ samples from each feedback type respectively such that:
\begin{align*}
c_p N'_p + c_e N'_e \leq \min (c_p p(\epsilon, \delta), c_e e(\epsilon, \delta)).
\end{align*}
\begin{algorithm}
\caption{Combine Feedback}
\begin{algorithmic}[1]
\STATE \textbf{Input:} Costs $c_p, c_e$, Budget $C$.
\STATE \textbf{Initialize:} $\widehat{\theta}$ appropriately and set $N'_p = 0, N'_e = 0$.
\FOR{$t = 1, \ldots$}
\STATE{\textbf{Preference Feedback}}
\STATE{Decide $n_{p,t}$.}
\STATE{Sample $n_t$ contexts from $\mathcal{D}$ to construct $\mathcal{D}_t = \{x_1, \ldots, x_{n_{p,t}}\}$.}
\STATE{Generate $y_1, y_2$ for all $x \in \mathcal{D}_t$ using $\pi_{\rm ref}$.}
\STATE{Observe labels from $z_i \sim \mathbb{P}[y_1 \succ y_2 | x_i]$ for all $i \in [n_{p,t}]$.}
\STATE{Update $\widehat{\theta}$ based on the labels.}
\STATE{$N'_p \longleftarrow N'_p + n_t$.}
\STATE{\textbf{Expert Feedback:}}
\STATE{Decide $n_{e,t}$.}
\STATE{Sample $n_t$ contexts from $\mathcal{D}$ to construct $\mathcal{D}_t = \{x_1, \ldots, x_{n_{e,t}}\}$.}
\STATE{Generate $y$ for all $x \in \mathcal{D}_t$ via bandit access to $\pi_{\rm \theta^\ast}$.}
\STATE{Update $\widehat{\theta}$ based on the data.}
\STATE{$N'_e \longleftarrow N'_e + n_{e,t}$.}
\STATE{\textbf{Check Budget:} If $c_p N'_p + c_e N'_e \geq C$ then break}
% \IF{c_p N'_p + c_e N'_e \geq C}\STATE{break}\ENDIF
\ENDFOR
\STATE \textbf{Return} $\widehat{\theta}$.
\end{algorithmic}
\label{alg:combinefeedback}
% \vskip -0.3in
\end{algorithm}
\begin{abstract}
% \avi{TODOs: The lower bound proofs are valid only for $\beta=0$, needs to be generalized for any arbitrary $\beta$.}
% \red{AS: Have more text explanation why hybrid is useful:
% (i) Pure offline might not have enough good samples (esp. from the optimal policy, which degrades the data quality),
% (ii) Pure online is very costly if we have to learn all the arms/ dimensions through active online exploration. We should also corroborate these claims in the expts later to show how our hybrid approach beats both online and offline strategies.}
Reinforcement Learning from Human Feedback (RLHF) is currently the leading approach for aligning large language models with human preferences. Typically, these models rely on extensive offline preference datasets for training. However, offline algorithms impose strict concentrability requirements, which are often difficult to satisfy.
% making it challenging to guarantee the learned model's near-optimality.
On the other hand,
% online algorithms can be employed to adaptively gather preference data and learn a model efficiently.
while online algorithms can avoid the concentrability issue, pure online exploration could be expensive due to the active preference query cost and real-time implementation overhead. %they fail to leverage the wealth of available offline data.
%Motivated by recent advances in Hybrid Reinforcement Learning, we propose a novel approach: Hybrid Preference Optimization. This method combines online exploration with existing offline preferences, overcoming the limitations of offline RLHF by relaxing the stringent coverage conditions, as well as significantly improving the sample efficiency of online exploration.
In this paper, we propose a novel approach: Hybrid Preference Optimization (HPO) which combines online exploration with existing offline preferences by relaxing the stringent concentrability conditions for offline exploration, as well as significantly improving the sample efficiency for its online counterpart.
%
We give the first provably optimal theoretical bound for Hybrid RLHF with preference feedback, providing sample complexity bounds for policy optimization with matching lower bounds. Our results yield improved sample efficiency of hybrid RLHF over pure offline and online exploration.
\end{abstract}
\section{Conclusion and Future Work}
\label{sec:conclusion}
In this paper, we introduced Hybrid Preference Optimization (HPO), a novel approach that effectively integrates the strengths of both online and offline Reinforcement Learning from Human Feedback (RLHF) methods. By combining online exploration with existing offline preference data, HPO addresses the limitations inherent in each approach when used in isolation. Specifically, it relaxes the stringent concentrability conditions required by offline methods and enhances the sample efficiency of online exploration. Our theoretical analysis provides the first provably optimal bounds for hybrid RLHF with preference feedback, demonstrating that HPO achieves better sample complexity than either pure offline or pure online methods.
The empirical implications of our approach suggest a promising avenue for developing more efficient and practical methods for aligning large language models with human preferences. By leveraging the wealth of existing offline data while simultaneously adapting to new information through online exploration, HPO reduces the need for extensive and costly real-time human feedback, thereby making the process more scalable and less resource-intensive.
\section{Experiments}\label{sec:experiments}
\begin{figure*}[t]
\centering
\begin{subfigure}{.5\textwidth}
\centering
\includegraphics[width=\linewidth]{figs/aistats2.png}
\caption{Cumulative Regret}
\label{fig:sub1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
\centering
\includegraphics[width=\linewidth]{figs/hpo_subopt.png}
\caption{Suboptimality Gap}
\label{fig:sub2}
\end{subfigure}
\caption{(Left) We plot the cumulative regret as the number of online samples grows. (Right) We plot the suboptimality gaps for all online, offline and hybrid algorithms as a function of total number of samples. Since HPO has access to offline samples, to ensure fairness while comparinng, for the hybrid case we report performance with number of offline + online samples seen. HPO used an offline dataset of size $N_{\rm off} = 500$, hence in both the plots the value corresponding to 750, is after 250 additional online samples seen, the value corresponding to 1000, is after 500 additional online samples collected by HPO and so on. }
\label{fig:test}
\end{figure*}
We evaluate the benefits of HPO on the linear contextual bandit setting considered in \citep{cen2024value}. We restate the salient features of their setup here.
We consider a linear contextual bandit problem, where we set the prompt space as $\mathcal{X} = \mathbb{R}^2$ and the response space as $|\mathcal{Y}| = 500$ one-hot vectors. For each $(x, y)$ pair, the ground truth reward is given by $r^\star(x, y) = \langle \phi(x, y), \theta^\star \rangle$, where $\theta^\star \in \mathbb{R}^{100}$ is randomly sampled from $\mathcal{U}([0, 1])$, and the feature vector $\phi(x, y)$ is the output of the hidden layer of a fixed two-layer MLP, with the input given by the concatenation of $x$ and the one-hot encoding of $y$. The activation function is set to $\tanh$. The context vector $x$ is drawn from standard normal distribution.
We focus on log-linear policy class $\pi_\theta(\cdot|x) = \rm{softmax}(\langle \theta, \phi(x, \cdot) \rangle)$, and set $\pi_{\text{ref}} = \pi_{\theta_{\text{ref}}}$ with $\theta_{\text{ref}}(x, y)$ sampled i.i.d. from $\mathcal{U}([0, 1])$.
\textbf{Offline Data Collection Policy:} For the offline data collection process we collected response pairs from the reference policy $\pi_{\text{ref}}$.
\textbf{Optimization Procedure:} We use mini-batch samples of size 5 in every iteration. We approximately solve the optimization problems by performing 20 AdamW optimization steps with learning rate 0.01 and weight decay rate 0.01 in every iteration for the online setting and 1000 steps for the offline setting.
\textbf{Choice of Hyperparameters:} For Online VPO, we tried hyperparameters $\alpha = \{1.0, 10.0\}$ and for offline VPO we tried hperparameters $\alpha = \{\frac{0.1}{\sqrt{N_{\rm off}}} , \frac{1.0}{\sqrt{N_{\rm off}}}, \frac{10.0}{\sqrt{N_{\rm off}}}\}$ and report the results for the best performing hyperparameter here. For HPO we try the same set of $\alpha$'s as Online VPO and set $\gamma = N_{\rm off} = 500$.
We ask the following question:
\textit{Starting from an offline dataset where offline algorithms like DPO \citep{rafailov2024direct}, Offline-VPO \citep{cen2024value} fail to obtain a near optimal policy, can we still utilize information from the dataset to significantly reduce the number of online exploration samples needed to obtain a near optimal policy via HPO as compared to baselines such as Online DPO \citep{guo2024direct}, Online VPO \citep{cen2024value} (XPO \citep{xie2024exploratory})?}
We answer this question affirmatively. In particular, we note the following takeaways from our experiment:
\begin{enumerate}
\item In Figure~\ref{fig:test} (left), the cumulative regret of HPO grows much slower compared to pure online learning baseline algorithms. The value plotted for any $T$ with $T > N_{\rm off}$ is $\sum_{i = 500}^T \left(J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\pi_{i})\right)$. This ensures the online algorithms have seen online $T$ samples, when we are talking about the cumulative regret at $T$, whereas the hybrid algorithm has seen $N_{\rm off}$ offline samples and $T - N_{\rm off}$ offline samples. This ensures fairness while comparing the online and hybrid approaches.
\item In Figure~\ref{fig:test} (right), we plot the suboptimality gaps as a function of total number of samples (online, offline or a mix of both). For the hybrid setting, we use an offline dataset of size $N_{\rm off} = 500$. Note that the suboptimality gaps for offline algorithms run on this size of a dataset is quite poor compared to other (purely online) baselines. However, using the same dataset, the suboptimality gaps for the policy produced by HPO is much better than all baselines.
\end{enumerate}
% Select a $N_{\rm off}$ at which the suboptimality gap is large for offline algorithms like MLE, VPO. Now show suboptimality gaps for pure online and hybrid algorithms in terms of number of offline samples in a bar plot. Highlight that the hybrid algorithm produces near optimal policies with much fewer online samples.
% Plot regret curves of MLE, VPO and HPO as number of online samples, showing the regret can be significantly reduced for hybrid approach.
% Comment on the effect of too much bias from suboptimal offline data (larger $\gamma$) can be bad.
% One plot on total number of samples compared to VPO.
% \begin{figure*}
% \centering
% \begin{subfigure}{.5\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figs/hpo_comparison.png}
% \caption{Cumulative Regret}
% \label{fig:sub1}
% \end{subfigure}%
% \begin{subfigure}{.5\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figs/hpo_subopt.png}
% \caption{Suboptimality Gap}
% \label{fig:sub2}
% \end{subfigure}
% \caption{For Online VPO, we tried hyperparameters $\alpha = \{0.01, 0.1, 1.0, 10.0\}$, and for offline VPO we tried hperparameters $\alpha = \{\frac{0.1}{\sqrt{N_{\rm off}}} , \frac{1.0}{\sqrt{N_{\rm off}}}, \frac{10.0}{\sqrt{N_{\rm off}}}\}$ and report the results for the best performing hyperparameter here. For HPO we try the same set of $\alpha$'s as Online VPO and $\gamma = \{50, 100, 250\}$. }
% \label{fig:test}
% \end{figure*}
% \vspace{-10pt}
\section{Hybrid Preference Optimization}
\label{sec:hpo}
% \vspace{-7pt}
We now present our algorithm Hybrid Preference Optimization (HPO), which integrates the available offline dataset with online exploration. This hybrid approach leverages the strengths of both offline data and online feedback, aiming to enhance the learning process and achieve a near optimal policy using fewer samples than pure online and pure offline methods.
% \vspace{-5pt}
\subsection{Algorithm Description}
% \vspace{-7pt}
We present Hybrid Preference Optimization (HPO) in Algorithm~\ref{alg:hybrid_rlhf}. Given a policy class $\Pi$, an offline dataset $\mathcal{D}_{\rm off}$, the algorithm proceeds through a series of steps over a predefined number of iterations $T$. The algorithm begins with 2 policies $\pi^{(1)}, \tilde{\pi}^{(1)}$ and iteratively updates these policies as described next.
At each step $t \in [T]$ of the online exploration, a context is sampled as $s_1^{(t)} \sim \rho$. The algorithm generates 2 responses: one sampled from the current policy $\tau_t \sim \pi^{(t)}(\cdot| s_1^{(t)})$, and one sampled from the model $\tilde{\tau}_t \sim \tilde{\pi}^{(t)}(\cdot| s_1^{(t)})$. This response pair is then labelled as $(\tau_{+}^{(t)}, \tau_{-}^{(t)})$ based on human preference feedback and added to the online buffer $\mathcal{D}_{\rm on}^{(t)} = \mathcal{D}_{\rm on}^{(t)} \cup (\tau_{+}^{(t)}, \tau_{-}^{(t)})$.
% \vspace{-5pt}
\begin{algorithm}
\caption{Hybrid Preference Optimization}
\begin{algorithmic}[1]
\STATE \textbf{Input:} Offline dataset $\mathcal{D}_{\rm off}$ of size $N_{\rm off}$, sampling strategy $\pi_{\rm samp}$, hyper-parameters, $\alpha \in \mathbb{R}_{+}$, $\gamma \in \mathbb{N}$.
\STATE \textbf{Initialize:} $\pi_t, \Tilde{\pi}_t \leftarrow \piref$, $\mathcal{D}^0_{\rm on} = \emptyset$.
\FOR{$t = 1, \ldots, T$}
\STATE \textbf{Generate response pair:} $s_1^{(t)} \sim \rho$; $\tau^{(t)} \sim \pi^{(t)}(\cdot|s_1^{(t)}), \taut^{(t)} \sim \Tilde{\pi}_t(\cdot|s_1^{(t)})$.
\STATE \textbf{Label with Preference:} Label $(\tau_t, \taut_t)$ as $(\tau_t^{+}, \tau_t^{-})$ based on preference feedback.
\STATE \textbf{Update Online Dataset:} $\mathcal{D}^{(t)}_{\rm on} \gets \mathcal{D}^{(t-1)}_{\rm on} \cup (\tau_{+}^{(t)}, \tau_{-}^{(t)})$.
\STATE \textbf{Update Offline dataset minibatch:} Sample $\gamma$ pairs from $\mathcal{D}_{\rm off}$ uniformly randomly with replacement $\mathcal{D}_{\rm off}^{(t)}$.
\STATE \textbf{Update Hybrid dataset:} $\mathcal{D}_{\rm hyb}^{(t)} \gets \mathcal{D}_{\rm on}^{(t)} \cup \mathcal{D}_{\rm off}^{(t)}$.
\STATE \textbf{Update Optimism Dataset: } Compute $\mathcal{D}^{(t)}_{\rm opt}$ of $t + \gamma$ samples from $\Tilde{\pi}_t$.
\STATE \textbf{Update Policy:} Update $\pi_t$ to maximize likelihood of preferences seen so far + regularization terms
\fontsize{9.5}{9.5}
\begin{align}\label{eq:HPO}
&\pi^{(t+1)} = \argmin_{\pi \in \Pi} \bigg[\alpha \sum_{\tau \in \mathcal{D}^{(t)}_{\rm opt}} \log \pi(\tau) - \sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^{(t)}_{\rm hyb}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})})\bigg]\bigg].
\end{align}\normalsize
\STATE \textbf{Update sampling policy:} \\$\Tilde{\pi}_{t+1} \gets \pi_{\rm samp}(\pi^{(1)}, \ldots, \pi^{(t+1)})$.
\ENDFOR
\STATE \textbf{Return} $\widehat{\pi} = \amax_{\pi \in \{\pi^{(1)}, \ldots, \pi^{(T)}\}} J_{\beta}(\pi)$.
\end{algorithmic}
\label{alg:hybrid_rlhf}
% \vskip -0.3in
\end{algorithm}
% \vspace{-5pt}
A key parameter in HPO is $\gamma \in \mathbb{N}$, which determines the number of labeled pairs drawn from the offline dataset $\mathcal{D}_{\rm off}^{(t)}$. The hybrid preference dataset at time $t$ is the union of the online buffer and $\gamma$ pairs sampled from the offline dataset $\mathcal{D}_{\rm hyb}^{(t)} = \mathcal{D}_{\rm on}^{(t)} \cup \mathcal{D}_{\rm off}^{(t)}$. An optimism dataset $\mathcal{D}_{\rm opt}^{(t)}$ is built by sampling $t + \gamma$ samples from $\tilde{\pi}^{(t)}$.
% We discuss the roles of $\mathcal{D}_{\rm opt}^{(t)}$ and $\mathcal{D}_{\rm hyb}^{(t)}$ shortly.
Now, we update the policy via an optimistic variant of DPO, and similar to XPO as in Eq.~\eqref{eq:HPO}. Here $\alpha$ is an optimism parameter, and the first term in Eq.~\eqref{eq:HPO} tries to encourage the policy to explore more diverse responses compared to those generated by $\tilde{\pi}^{(t)}$ and captured by the samples in $\mathcal{D}_{\rm opt}^{(t)}$. The second term in Eq.~\eqref{eq:HPO} is the DPO objective on the hybrid preference dataset $\mathcal{D}_{\rm hyb}^{(t)}$ which aligns the policy to the human preferences.
If $\gamma = 0$, the algorithm is exactly the same as the XPO in \citep{xie2024exploratory} and it discards all the available offline data during training. We will later show in Theorem~\ref{thm:main_general} how the choice of $\gamma$ shows up in our sample complexity bounds. Intuitively the value of $\gamma$, biases the policy to the behavior of the offline dataset. A too large $\gamma$ will put a large emphasis on the offline dataset and may hinder exploration, while a too small $\gamma$
will fail to effectively utilize the available information in the offline dataset. Thus, $\gamma$ needs to be chosen in accordance to the online exploration budget $T$.
Finally, we can see that HPO is very friendly to application as it is rooted from the vanilla DPO and the extra regularization term and sampling procedure can be easily implemented.
% \vspace{-6pt}
\subsection{Assumptions}
% \vspace{-7pt}
To establish sample complexity guarantees for HPO, we adopt standard statistical assumptions. The first of these assumptions requires that the policy class $\Pi$ be sufficiently expressive to represent the optimal KL-regularized policy.
\begin{assumption}[Policy realizability]
\label{ass:realizability}
The policy class $\Pi$ contains the optimal policy, i.e., $\pistarb\in\Pi$.
\end{assumption}
Policy realizability is a standard assumption for sample-efficient reinforcement learning which can be found in prior works such
\citep{agarwal2019reinforcement,lattimore2020bandit,foster2023foundations}. It is equivalent to a form of reward/value realizability as discussed in \citep{xie2024exploratory}.
In our context, $\Pi$ will typically correspond to a class of language models with fixed architecture and variable weights.
Next, we make a regularity assumption on the
policies in $\Pi$ \citep{rosset2024direct, xie2024exploratory}.
\begin{assumption}[Bounded density ratios]\citep{xie2024exploratory}
\label{ass:vmax}
For all $\pi\in\Pi$ and trajectories
$\tau=(s_1,a_1),\ldots,(s_H,a_H)$, it holds
\begin{align}
\left|\log\left(\frac{\pi(\tau)}{\piref(\tau)}\right)\right| \leq\frac{\Vmax}{\beta}.
\end{align}
\end{assumption}
As discussed in \citep{xie2024exploratory}, $\Vmax$ is measurable and controllable in practice. Specifically for log-linear policies where
$\pi(a\mid{}s)\propto\exp({\nicefrac{f(s,a)}{\beta}})$ with some linear function $f(s, a)$, it can be shown that $\Vmax\approxleq{}\Rmax$ (recall that $\Rmax$ is the range of reward).
\subsection{Model Complexity Measures}
To measure the algorithm's convergence rate towards an optimal policy, we introduce an exploration criterion that limits how frequently the algorithm encounters novel samples where it cannot effectively distinguish between the impact of $\pi$ and $\piref$ on the objective.
First, we define a preference-based analogue of the \emph{Sequential Extrapolation Coefficient} (SEC)\footnote{This is also known as an Eluder coefficient or decoupling coefficient \citep{zhong2022gec,ye2024theoretical}.} from \citep{xie2023role} that corresponds to the hybrid RLHF case. Particularly, define
\fontsize{9}{9}
\begin{align*}
g^{(\pi)}(\tau, \taut) = \left[\beta \log \frac{\pi(\tau)}{\piref(\tau)} - r(\tau) - \beta \log \frac{\pi(\taut)}{\piref(\taut)} + r(\taut) \right]
\end{align*}
\normalsize
as a measure of the difference in the objective from Eq.~\eqref{eq:KLobj} for an arbitrary policy $\pi \in \Pi$ and two arbitrary trajectories $\tau$ and $\taut$.
For a pair of policies $\pi$ and $\Tilde{\pi}$, we define $\pi\otimes\Tilde{\pi}$ as
the joint policy that, given $s_1$, samples $\tau\sim{}\pi\mid{}s_1$
and $\taut \sim{} \Tilde{\pi} \mid{}s_1$. We write
$(\tau,\taut)\sim{}\pi\otimes\Tilde{\pi} \mid{}s_1$ as a shorthand for this process. We further let $\Tilde{\pi}\ind{t}= \samp(\pi\ind{1},\ldots,\pi\ind{t})$, and $\bmu\ind{t} = \frac{1}{t-1}\sum_{i=1}^{t-1} \pi\ind{i}\otimes \Tilde{\pi}\ind{i}$, with the convention that $\mu\ind{1}$ is arbitrary and $\samp$ is some sampling strategy such as $\mathsf{unif}(\cdot)$.
We first define the online sequential exploration coefficient for online RLHF \citep{xie2024exploratory}.
\begin{definition} \citep{xie2024exploratory} Given a policy class $\Pi$, reference policy $\piref$, sampling strategy $\pi_{\rm samp}$, entropy regularization parameter $\beta > 0$, online exploration budget $T \in \mathbb{N}$, we define the sequential exploration coefficient (SEC) as:
\fontsize{8.5}{8.5}
% \begin{align}
% &{\rm SEC}_{\rm RLHF}(\Pi, T, \beta, \pi_{\rm samp}) = \sup_{\pi^{(1)}, \ldots, \pi^{(T)} \in \Pi} \nonumber\\& \sum_{t=1}^T \frac{\left(\mathbb{E}_{\substack{s_1 \sim \rho, \\ \tau \sim \pi^{(t)} | s_1, \\ \taut \sim \tilde{\pi}^{(t-1)}| s_1}}[g^{(\pi^{(t)})}(\tau, \taut)]\right)^2}{\Vmax^2 \vee \left[(t - 1) \cdot \mathbb{E}_{\substack{s_1 \sim \rho, \\ (\tau, \taut) \sim \mu^{(t)} | s_1}} [\left(g^{(\pi^{(t)})}(\tau, \taut)\right)^2 ] \right]}.
% \end{align}
\begin{align}
&{\rm SEC}_{\rm RLHF}(\Pi, T, \beta, \pi_{\rm samp}) = \sup_{\pi^{(1)}, \ldots, \pi^{(T)} \in \Pi} \sum_{t=1}^T \frac{\left(\mathbb{E}_{\substack{s_1 \sim \rho, \\ \tau \sim \pi^{(t)} | s_1, \\ \taut \sim \tilde{\pi}^{(t-1)}| s_1}}[g^{(\pi^{(t)})}(\tau, \taut)]\right)^2}{\Vmax^2 \vee \left[(t - 1) \cdot \mathbb{E}_{\substack{s_1 \sim \rho, \\ (\tau, \taut) \sim \mu^{(t)} | s_1}} [\left(g^{(\pi^{(t)})}(\tau, \taut)\right)^2 ] \right]}.
\end{align}
\normalsize
\end{definition}
Next, we introduce another quantity to capture the coverage of the offline dataset in terms of any arbitrary policy $\pi \in \Pi$:
\begin{align}
C^{(\pi)}_{\rm off} = \frac{1}{N_{\rm off}}\sum_{(\tau_{+}, \tau_{-}) \in \mathcal{D}_{\rm off} }\left(g^{(\pi)}(\tau_{+}, \tau_{-})\right)^2.
\end{align}
Below we define the SEC coefficient in the Hybrid RLHF case:
\begin{definition} Given a policy class $\Pi$, reference policy $\piref$, offline preference dataset $\mathcal{D}_{\rm off}$, offline sampling parameter $\gamma \in \mathbb{N}$, sampling strategy $\pi_{\rm samp}$, entropy regularization parameter $\beta > 0$, online exploration budget $T \in \mathbb{N}$, we define the sequential exploration coefficient (SEC) as:
% \fontsize{8}{8}
% \begin{align}
% &\coefffull=\sup_{\pi^{(1)}, \ldots, \pi^{(T)}\in \Pi} \nonumber\\& \qquad \sum_{t=1}^T \frac{\left(\mathbb{E}_{\substack{s_1 \sim \rho,\\ \tau \sim \pi^{(t)} | s_1,\\ \taut \sim \tilde{\pi}^{(t-1)}| s_1}}[g^{(\pi^{(t)})}(\tau, \taut)]\right)^2}{\Vmax^2 \vee \left[(t - 1) \cdot \mathbb{E}_{\substack{s_1 \sim \rho,\\ (\tau, \taut) \sim \mu^{(t)} | s_1}}[(\widetilde{g}_{\rm off}^{(\pi^{(t)}}))^2 ] \right]},
% \end{align}
\begin{align}
&\coefffull=\sup_{\pi^{(1)}, \ldots, \pi^{(T)}\in \Pi} \qquad \sum_{t=1}^T \frac{\left(\mathbb{E}_{\substack{s_1 \sim \rho,\\ \tau \sim \pi^{(t)} | s_1,\\ \taut \sim \tilde{\pi}^{(t-1)}| s_1}}[g^{(\pi^{(t)})}(\tau, \taut)]\right)^2}{\Vmax^2 \vee \left[(t - 1) \cdot \mathbb{E}_{\substack{s_1 \sim \rho,\\ (\tau, \taut) \sim \mu^{(t)} | s_1}}[(\widetilde{g}_{\rm off}^{(\pi^{(t)}}))^2 ] \right]},
\end{align}
% \normalsize
% where $(\widetilde{g}_{\rm off}^{(\pi)})^2 = \left(g^{(\pi^{(t)})}(\tau, \taut)\right)^2 + \gamma \cdot C^{(\pi^{(t)})}_{\rm off}$.
where $(\widetilde{g}_{\rm off}^{(\pi)})^2 = \left(g^{(\pi)}(\tau, \taut)\right)^2 + \gamma \cdot C^{(\pi)}_{\rm off}$.
\end{definition}
\begin{remark}
Note that because of the additional quantity $C_{\rm off}^{(\pi)}\geq 0$, for all $\gamma \in \mathbb{N}$ and any non-empty $\mathcal{D}_{\rm off}$, we have
% \begin{align*}
% &\coefffull\\
% &\quad < {\rm SEC}_{\rm RLHF}(\Pi, T, \beta, \pi_{\rm samp}).
% \end{align*}
\begin{align*}
&\coefffull < {\rm SEC}_{\rm RLHF}(\Pi, T, \beta, \pi_{\rm samp}).
\end{align*}
In particular, $C_{\rm off}^{(\pi)}$ is a measure of coverability for an arbitrary $\pi$, contributed by the offline dataset.
\end{remark}
\subsection{Main Results}
We state the main sample complexity result of HPO below.
% \begin{theorem}\label{thm:main_general}
\begin{restatable}{theorem}{maingeneral}\label{thm:main_general}
Suppose Assumption \ref{ass:realizability} and \ref{ass:vmax} hold.
For any $\beta>0$ and $T\in [N]$, if we set $\alpha=c\cdot\frac{\beta}{(\Vmax + \Rmax)e^{2 \Rmax})}\cdot\sqrt{\frac{\log(|\Pi|T\delta^{-1})\log(T)}{(T + \gamma)\cdot \coefffull}}$
for some absolute constant $c>0$,
then Algorithm \ref{alg:hybrid_rlhf} ensures that
with probability at least $1-\delta$, we have
% \begin{align*}
% & J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi})
% \approxleq
% (\Vmax+\Rmax)e^{2\Rmax}\cdot \\
% &\quad\sqrt{\frac{(1 + \gamma / T)\cdot\mathsf{SEC}\cdot\log(|\Pi|T\delta^{-1})\log(T)}{T}},
% \end{align*}
\begin{align*}
& J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi})
\approxleq
(\Vmax+\Rmax)e^{2\Rmax}\cdot \quad\sqrt{\frac{(1 + \gamma / T)\cdot\mathsf{SEC}\cdot\log(|\Pi|T\delta^{-1})\log(T)}{T}},
\end{align*}
where $\mathsf{SEC} = \coefffull$.
\end{restatable}
% \end{theorem}
Note that $\gamma=0$ represents the pure online setting, which is studied in XPO \citep{xie2024exploratory}. Here, we can notice that the sample complexity is reduced because the SEC for hybrid RLHF becomes smaller than its pure online counterpart. That is, the size of the space that needs to be searched through online exploration becomes smaller because of the available offline dataset.
% We will elaborate this benefit in next section with a more concise example in linear MDPs.
% Main points:
% \begin{enumerate}
% \item The $C_{\rm off}^{(\pi)}$ term is a measure of coverability for an arbitrary $\pi$.
% \item SEC for hybrid smaller -- less space to search because of available offline dataset
% \end{enumerate}
\section{Introduction}
\label{sec:introduction}
% \avi{Avi TODO}
% \avi{motivate the problem well by highlighting the limitations of pure online and pure offline rlhf, highlighting concentrability. Hybrid gives the best of both worlds}
Reinforcement Learning from Human Feedback (RLHF) stands out as the primary method for aligning large language models with human preferences \citep{christiano2017deep, bai2022training, ouyang2022training}. Instead of starting from scratch with unsupervised training on extensive datasets, RLHF aligns pre-trained models using labeled human preferences on pairs of responses, offering a statistically lightweight approach to making language models more human-like. While labeling response pairs is easier than generating new responses, the volume of these pairs is critical for effective alignment. A large dataset is needed to ensure broad coverage of linguistic nuances, reduce the impact of noisy human feedback, and provide enough statistical power for the model to generalize well. Although labeling individual pairs is simpler, scaling this process can still become resource-intensive, making the volume of response pairs a key factor in successful model alignment. In the light of this, recently a theoretical question of interest has arisen: \textit{How can algorithms be designed to be sample-efficient during this alignment phase?}
Two main approaches have emerged in addressing this question: online RLHF and offline RLHF. Online methods \citep{xie2024exploratory, cen2024value, zhang2024self} have interactive access to human feedback or leverage a more powerful language model to explore diverse and novel responses beyond what the pre-trained model can provide. Online exploration, though theoretically sample-efficient, is costly because it requires frequent human feedback, real-time updates to large language models, and continuous deployment to generate novel responses. These frequent model updates and the need for real-time interaction with human annotators make the process resource-intensive and time-consuming, presenting significant practical challenges in terms of both scalability and cost.
In contrast, offline RLHF relies on large, readily available sets of labeled response pairs to align the language model. While these datasets are easier to collect and use, achieving provable guarantees on the model's optimality requires stringent conditions—specifically, that the dataset is close to data generated by an optimally-aligned model. %closely represents an optimal language model
%\textcolor{magenta}{maybe say: is close to data generated by an optimally-aligned model}.
Offline datasets, however, are static and often lack the diversity and novel responses that online exploration provides. This limitation, coupled with the assumption that the dataset contains enough near-optimal examples, restricts the model's ability to generalize effectively, potentially leading to suboptimal alignment.
This prompts the question: \textit{Can we combine offline preference data while relaxing these stringent conditions to significantly reduce the number of samples required in online exploration, thereby addressing its practical costs effectively?}
In this paper, we answer this question affirmatively.
\begin{table*}[ht]
\centering
\begin{tabular}{l|c|c|c|c|c}
& Category & Upper Bound & Lower Bound & Scalable & Model-free \\
% & & & () & \\
\midrule
\citet{xie2021policy} & Traditional RL & \checkmark\cellcolor{green!25} & \checkmark\cellcolor{green!25} & \xmark\cellcolor{red!25} & \xmark\cellcolor{red!25} \\
\hline
\citet{tan2024hybrid} & Traditional RL & \checkmark\cellcolor{green!25} & \checkmark\cellcolor{green!25} & \xmark\cellcolor{red!25} & \xmark\cellcolor{red!25} \\
\hline
\citet{xiong2024iterative} & RLHF & \checkmark\cellcolor{green!25} & \xmark\cellcolor{red!25} & \xmark\cellcolor{red!25} & \checkmark\cellcolor{green!25} \\
\hline
\citet{gao2024rebel} & RLHF & \checkmark\cellcolor{green!25} & \xmark\cellcolor{red!25} & \checkmark\cellcolor{green!25} & \checkmark\cellcolor{green!25} \\
\hline
\citet{chang2024dataset} & RLHF & \checkmark\cellcolor{green!25} & \xmark\cellcolor{red!25} & \checkmark\cellcolor{green!25} & \checkmark\cellcolor{green!25} \\
\hline
Our work & RLHF & \checkmark\cellcolor{green!25} &\checkmark\cellcolor{green!25} &\checkmark\cellcolor{green!25} & \checkmark\cellcolor{green!25} \\
\end{tabular}
\caption{A comparison with lines of work closest to ours.}
\label{tab:comparison_table}
\vskip -0.2in
\end{table*}
% \avi{Include an illustration to highlight what our algorithm does}
\subsection{Contributions}
% \avi{Main sample complexity guarantee, lower bounds.}
% \avi{Avi TODO}
Here, we list our technical contributions:
\begin{itemize}
\item We introduce the first Hybrid RLHF algorithm that is both provably efficient and practical for implementation. In contrast, previous discussions of hybrid training in \citet{xiong2024iterative}, \citet{gao2024rebel}, and \citet{chang2024dataset} either lack practical applicability or do not provide a theoretical analysis of the advantages of hybrid training. See Table~\ref{tab:comparison_table} for a detailed comparison.
\item Our algorithm demonstrates superior sample complexity compared to both online and offline RLHF approaches (Theorem \ref{thm:main_general}). Specifically, in the case of linear MDPs, our method surpasses the limitations imposed by the lower bounds of purely online and offline training (Theorems \ref{thm:lboff} and \ref{thm:lbon}).
% \avi{Characterize a little bit on offline coverage and how to choose $\gamma$ in the simple linear case.}
% \item Matching lower bound for linear model (\cref{thm:lbon},\cref{thm:lboff} in \cref{sec:lb}).
\end{itemize}
\subsection{Paper Organization}
The remaining of the paper is organized as the following. We review the related work in Section \ref{sec:related_work} and briefly introduce the background of MDPs and RLHF in Section \ref{sec:prelim}. We then present our algorithm Hybrid Preference Optimization (HPO) and its theoretical guarantee in Section \ref{sec:hpo}. Section \ref{sec:lb} examines the special case of linear MDPs and compare our results with its lower bounds in both online and offline training. We present our empirical findings in Section \ref{sec:experiments} following by a conclusion in Section \ref{sec:conclusion}.
\section{Breaking Lower Bounds through Hybrid Learning} \label{sec:lb}
% \zhihan{Is this section title too long and too detailed? Maybe something like ``Breaking Lower Bounds through Hybrid Learning''?}
In order to paint a clearer picture of why Hybrid RLHF is more sample-efficient than pure online and pure offline RLHF, we consider the special case of linear MDPs and study the suboptimality gaps of hybrid, online and offline RLHF. Our main goal is to show that the upper bounds for hybrid RLHF are smaller than the minimax lower bounds for pure online and pure offline RLHF.
We begin by formally defining the linear MDP setting.
\begin{definition}(Linear MDP \citep{jin2020provably})
In a Linear MDP, the transition probability as well as the reward are linear functions of a known feature map $\phi(s,a) \in \mathbb{R}^d$, where $\|\phi\|_2 \leq 1$ and
\begin{align*}
P(s' | s, a) = \phi(s, a)^\top \mu(s') \quad r(s, a) = \phi(s, a)^\top \nu.
\end{align*}
Here, $\mu(s')$ is an unknown feature map with $\|\sum_{s'} \mu(s')\|_2 \leq \sqrt{d}$, and $\nu \in \mathbb{R}^d$ is an unknown parameter with $\|\nu\|_2 \leq 1$.
% where $d_{\rm hyb} = $
\end{definition}
% \subsection{Single Policy Concentrability limits offline RLHF}
\subsection{Lower Bounds for Pure Offline and Online RLHF}
Define $\Lambda_{\rm off} = \frac{1}{N_{\rm off}}\sum_{(\tau, \taut) \in \mathcal{D}_{\rm off}}(\phi(\tau) - \phi(\taut))(\phi(\tau) - \phi(\taut))^\top$ as the empirical feature covariance matrix of the offline preference dataset. Let $\nu^ \ast = \frac{\mathbb{E}_{s \sim \rho, \tau \sim \pi^\ast_\beta(\cdot| s)}[\phi(s, \tau)]}{\|\mathbb{E}_{s \sim \rho, \tau \sim \pi^\ast_\beta(\cdot| s)}[\phi(s, \tau)]\|_2}$ denote a unit vector corresponding to the feature projection along the optimal policy $\pi^\ast_\beta$. We define the following quantity, for a given instance, i.e. a linear MDP $\mathcal{M} = \{\phi, \mu, \nu\}$ and offline dataset $\mathcal{D}_{\rm off}$:
\begin{align*}
\rm{C}^{\ast}(\mathcal{M}, \mathcal{D}_{\rm off}) = \|\Lambda_{\rm off}^{-\frac{1}{2}} \nu^\ast\|_{2}.
\end{align*}
This measures the coverage of the offline dataset with respect to the optimal policy $\pi^\ast_\beta$. A Larger value indicates poorer coverage. Since for any arbitrary instance this quantity may be unbounded above, we consider all families of linear MDPs such that the concentrability is at most $\Delta$, i.e.
\begin{align*}
\rm{CB}(\Delta) = \{\mathcal{M}, \mathcal{D}_{\rm off} | \rm{C}^{\ast}(\mathcal{M}, \mathcal{D}_{\rm off}) \leq \Delta\}.
\end{align*} Then by extending results from \cite{li2022pessimism}, there exists an instance in $\rm{CB}(\Delta)$ such that the suboptimality for any algorithm using $\ell_2$ confidence sets is lower bounded by :
\begin{align*}
J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi}) \geq \Omega\left(\sqrt{\frac{d}{N_{\rm off}}} \Delta \right).
\end{align*}
% We define the single policy concentrability coefficient for a given instance, i.e. a linear MDP $\mathcal{M} = \{\phi, \mu, \nu\}$ and offline dataset $\mathcal{D}_{\rm off}$ w.r.t. the $\ell_q$ norm as:
% \begin{align*}
% \rm{C}^{\ast}_q(\mathcal{M}, \mathcal{D}_{\rm off}) = \|\Lambda_{\rm off}^{-\frac{1}{2}}\mathbb{E}_{s \sim \rho, \tau \sim \pi^\ast_\beta(\cdot| s)}[\phi(s, \tau)]\|_{q}.
% \end{align*}
% This measures how well the offline dataset covers samples along the optimal policy.
% Consider all families of linear MDPs such that the concentrability is at most $\lambda$, i.e.
% \begin{align*}
% \rm{CB}_{q}(\lambda) = \{\mathcal{M}, \mathcal{D}_{\rm off} | \rm{C}^{\ast}_q(\mathcal{M}, \mathcal{D}_{\rm off}) \leq \lambda\}.
% \end{align*}Then by \citep{li2022pessimism}, there exists an instance in $\rm{CB}_{q}(\lambda)$ such that the suboptimality for any algorithm using $\ell_q$ confidence sets is lower bounded by :
% \begin{align*}
% J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi}) \geq \Omega(\sqrt{\frac{d^{2-2/q}}{N_{\rm off}}} \lambda).
% \end{align*}
% For the class of instances where $\lambda = \mathcal{O}(d^{1/q})$, the suboptimailty is lower bounded by $\Omega(\sqrt{\frac{d^2}{N_{\rm off}}})$.
Thus, by choosing $\Delta = \mathcal{O}(\sqrt{d})$ even in the restricted class $\rm{CB}(\mathcal{O}(\sqrt{d}))$, the suboptimality lower bound is $\Omega\left(\sqrt{\frac{d^2}{N_{\rm off}}}\right)$. We formally state this as a minimax lower bound for the pure offline case:
\begin{theorem}[Sample Complexity Lower Bound for Offline RLHF]
\label{thm:lboff}
For any algorithm $\cA$, there exists an instance of offline RLHF problem and choice of $\beta$, such that if $\hat \pi_n^{\cA}$ is the policy returned by $\cA$ after any $n \geq d^2$ pairwise preference samples, then $J_\beta(\pi^*_\beta) - J_\beta(\hat \pi_n^{\cA}) \geq \Omega\left(\sqrt{\frac{d^2}{n}}\right)$.
\end{theorem}
% \subsection{Lower Bounds for Online RLHF}
We now state a minimax lower bound for the pure online case:
% \avi{Aadirupa TODO}.
\begin{theorem}[Sample Complexity Lower Bound for Online RLHF]
\label{thm:lbon}
For any algorithm $\cA$, there exists an instance of online RLHF problem and choice of $\beta$, such that if $\hat \pi_T^{\cA}$ is the policy returned by $\cA$ after $T$ rounds, then $J_\beta(\pi^*_\beta) - J_\beta(\hat \pi_T^{\cA}) \geq \Omega\left(\sqrt{\frac{d^2}{T}}\right)$.
\end{theorem}
% \subsection{Lower Bounds for Offline RLHF}
% \avi{Zhihan can we add how concentrability factors in the offline rlhf case, however our hybrid bound only depends on coverability.}
The proof of \cref{thm:lboff} and \cref{thm:lbon} respectively follows the existing lower bounds for online \citep[Theorem 2]{wagenmaker2022reward} and offline \citep[Theorem 2]{li2022pessimism} linear contextual bandits along with the reduction to idea argued in \cite[Lemma 8]{saha2021optimal}. We have added the detailed proofs in \cref{app:lbproofs}.
\subsection{Upper Bounds for Hybrid RLHF}
% Define $\Lambda_{\rm off} = \frac{1}{N_{\rm off}}\sum_{(\tau, \taut) \in \mathcal{D}_{\rm off}}(\phi(\tau) - \phi(\taut))(\phi(\tau) - \phi(\taut))^\top$ as the empirical feature covariance matrix of the offline preference dataset and
Define $\Tilde{\Lambda}_{\rm off} = \Lambda_{\rm off} + \frac{\Vmax^2}{\gamma} \mathbf{I}$ as the empirical coverage.
Let $\lambda^{(1)}_{\rm off}, \dots, \lambda^{(d)}_{\rm off}$ be the eigenvalues of $\Tilde{\Lambda}_{\rm off}$. Let $\{v_1, v_2, \ldots, v_d\}$ denote the orthonormal eigenvectors corresponding to $\Tilde{\Lambda}_{\rm off}$. We adopt the convention that the eigenvalues corresponding to these eigenvectors are in increasing order. For a fixed threshold $\lambda$ we can define:
\begin{align*}
d_{\rm hyb} = \max_{i \in [d]} \left\{\|\Tilde{\Lambda}_{\rm off}^{-\frac{1}{2}} v_i\|_{2} \approxleq \Omega\left(\frac{1}{\sqrt{T}}\right)\right\}.
% \text{ \zhihan{This seems to be equivalent to $\max_{i\in[d]}\sqrt{\lambda^{(i)}_{\rm off}}\leq\Omega(1/T)$.}}
\end{align*}
$d_{\rm hyb}$ measures the coverage of the offline dataset. A larger value of $d_{\rm hyb}$ indicates poor coverage. Note that the definition of $d_{\rm hyb}$ is also equivalent to $\max_{i\in[d]}\lambda^{(i)}_{\rm off}\leq\Omega(1/T)$.
We first state an upper bound on the sample complexity of HPO.
% \begin{theorem}
% \label{thm:linear_upper}
\begin{restatable}{theorem}{linearupper}\label{thm:linear_upper}
By choosing $\gamma = \mathcal{O}(T)$, we then have
\begin{align*}
J_{\beta}(\pi^\ast_\beta) - J_{\beta}(\widehat{\pi})
\approxleq \tilde{\mathcal{O}}\left(\Rmax e^{2\Rmax} \sqrt{\frac{d\cdot d_{\rm hyb}}{T}}\right).
\end{align*}
% \end{theorem}
\end{restatable}
\begin{proof}
We highlight the key steps of this proof.
% and the detailed proof is deferred to Appendix~\ref{app:proofs}.
Using the elliptical potential lemma \cite{lattimore2020bandit}, we can derive
\begin{align*}
&\coefffull \\
&\leq 2 \log( |\gamma \Lambda_{\rm off} + (\Vmax^2 + 4T)\mathbf{I}|) - 2\log(|\gamma\Lambda_{\rm off} + \Vmax^2\mathbf{I}|)\\
&=2 \log( |\Tilde{\Lambda}_{\rm off} + 4T\mathbf{I} / \gamma|) - 2\log(|\Tilde{\Lambda}_{\rm off}|)\\
&= 2 \sum_{i \in [d]}\log\left(\lambda_{\rm off}^{(i)} + 4T/\gamma\right) -2\sum_{i \in [d]} \log\left(\lambda_{\rm off}^{(i)}\right)\\
&\leq 2 \sum_{i \in [d]}\left(\log\left(1 + \frac{4T}{ \gamma \lambda_{\rm off}^{(i)}}\right)\right).
\end{align*}
Choosing $\gamma = \mathcal{O}(T)$, then only terms with $\lambda_{\rm off}^{(i)} \leq \Omega(\frac{1}{T})$ will remain in the summand, corresponding to the directions where the offline dataset has poor exploration. By defintion, $d_{\rm hyb}$ denote the number of indices in $[d]$ such that $\lambda_{\rm off}^{(i)} \leq \Omega(\frac{1}{T})$. Noting that $\gamma \lambda_{\rm off}^{(i)} \geq \Vmax^2 \quad \forall i \in [d]$, we get $\coefffull \leq \mathcal{O}\left(d_{\rm hyb}\log\left(1 + \frac{4T}{\Vmax^2}\right)\right)$.
Hence, the SEC scales as the effective number of dimensions yet to be explored sufficiently. Plugging this into Theorem~\ref{thm:main_general} gives the desired result.
\end{proof}
Note that $d_{\rm hyb}$ is always upper bounded by $d$. Thus for any arbitrary instance any arbitrary offline dataset, this upper bound is still $\tilde{\mathcal{O}}\left(\sqrt{\frac{d^2}{T}}\right)$.
% \subsection{Discussion}
% Here are several points worth noting from this analysis:
Furthermore, from the above analysis, it is clear that compared with pure offline and pure online RLHF, HPO has the following two advantages:
\begin{enumerate}
\item The upper bound of HPO in Theorem \ref{thm:linear_upper} beats the lower bounds of both pure online and offline RLHF in Theorem \ref{thm:lboff} and \ref{thm:lbon}, as long as $d_{\rm hyb}$ is non-trivially smaller than $d$. That is, our proposed hybrid RLHF is provably better than both pure online and offline RLHF.
\item To achieve this upper bound, HPO does not require any assumptions in all-policy or single-policy concentrability coefficient, which is usually imposed in traditional offline RL \citep{zhan2022offline}. To satisfy these kinds of assumptions, the behavior policy used to collect the offline dataset needs to cover all possible actions of the optimal policy, which can sometimes be stringent. However, HPO can effectively utilize the offline dataset even if the behavior policy covers no action of the optimal policy as $d_{\rm hyb}$ shrinks even through coverage in the offline datasets collected via by suboptimal policies.
\end{enumerate}
\section{Preliminaries}
\label{sec:prelim}
Our study of RLHF is under the general reinforcement learning setup similar to \citep{xie2024exploratory} which is a strict generalization of the token-level MDP proposed in \citep{rafailov2024r}.
\subsection{Reinforcement Learning from Human Feedback}
\label{sec:rlhf}
We consider an episodic finite horizon markov decision process characterized by $\mathcal{M} = (H, \mathcal{S}, \mathcal{A}, P, r, \rho)$ where $H$ is the time horizon, $\mathcal{S}$ denotes the state space, $\mathcal{A}$ denotes the action space, $P : \mathcal{S} \times \mathcal{A} \mapsto \Delta(\mathcal{S})$ denotes the transition function, $r : \mathcal{S} \times \mathcal{A} \mapsto \mathbb{R}$ denotes the reward function, and $\rho \in \Delta(\mathcal{S})$ denotes the initial state distribution. A randomized policy $\pi : \mathcal{S} \mapsto \Delta(\mathcal{A})$ generates a trajectory $\tau = \{(s_1, a_1), \ldots, (s_H, a_H)\}$ and rewards $r_1, \ldots, r_H$ through the following procedure. In particular, the initial state is sampled as $s_1 \sim \rho$, and for all subsequent time steps $h \in [H]$, actions are sampled as $a_h \sim \pi(s_h)$, reward is received as $r_h = r(s_h, a_h)$, and $s_{h+1} \sim P(s_h, a_h)$. We use shorthand notations $\mathbb{E}_{\pi}[\cdot]$ and $\mathbb{P}_{\pi}[\cdot]$ to denote the expectation and probability of quantities of interest under this process induced by the policy $\pi$. We assume the total reward is non-negative and bounded, i.e. $\sum_{h \in [H]} r_h \in [0, \Rmax]$. For compact notation at the trajectory level, we denote $r(\tau) = \sum_{h \in [H]} r(s_h, a_h)$ and $\pi(\tau) = \prod_{h \in [H]} \pi(a_h | s_h)$. The expected reward under policy $\pi$ is defined as $J(\pi) = \mathbb{E}_{\tau \sim \pi}[r(\tau)]$.
% \zhihan{TODO: add DCMDP}
\paragraph{Preference Model}
In RLHF, the reward signal from a trajectory $(r_1, \dots, r_H)$ is generally unobservable. Instead, the feedback from the environment is provided through a preference oracle.
In particular, given a pair of trajectories $(\tau, \tilde{\tau})$, both having initial state $s_1$, we follow the standard assumptions as in \citep{christiano2017deep, rafailov2024direct, rafailov2024r}, that the probability of $\tau$ being preferred over $\tilde{\tau}$ follows the Bradley-Terry Model \citep{bradley1952rank}, which is defined as
\begin{align}
\label{eq:btl}
\mathbb{P}(\tau \succ \tilde{\tau} | s_1) = \frac{\exp{(r(\tau))}}{\exp{(r(\tau))} + \exp{(r(\tilde{\tau}))}}.
\end{align}
Typically, we have access to a pre-trained policy $\piref$, and the goal of RLHF is to find the policy that can maximize $J(\pi)$ while staying not too far away from $\piref$. That is, with regularization parameter $\beta>0$, we aim to maximize the following objective:
\begin{align}\label{eq:KLobj}
J_{\beta}(\pi) &= J(\pi) - \beta \mathbb{E}_{\pi}\left[\sum_{h \in [H]} D_{KL}(\pi(\cdot| s_h) || \piref(\cdot| s_h)) \right] \nonumber\\
&= \mathbb{E}_{\pi}\left[r(\tau) - \beta \log \frac{\pi(\tau)}{\piref(\tau)}\right].
\end{align}
Here, we denote the optimal policy of some policy class $\Pi$ as $\pi^\star_{\beta}=\argmax_{\pi\in\Pi}J_{\beta}(\pi)$.
\paragraph{Token-level MDPs.}
Given our primary focus on language models, we are particularly interested in the \textit{token-level MDP} framework, as formulated and studied by \citet{rafailov2024r}. In this context, a language model typically generates a sequence of responses from an initial prompt. The initial prompt is treated as the initial state, \( s_1 \sim \rho \), and each generated token, \( a_h \in \mathcal{A} \), is considered an action, where \( \mathcal{A} \) represents the vocabulary. Under this formulation, the state at step \( h \) is represented as \( s_h = (s_1, a_1, \dots, a_{h-1}) \), a concatenation of the initial prompt and all tokens generated before step \( h \). The state at last step, \( s_H \), can then be interpreted as the model's complete response to the initial prompt, including the prompt itself.
% Given our major focus on language models, we will be particularly interested in \textit{token-level MDP}, which was formulated and studied in \citet{rafailov2024r}. Specifically, a language model will usually generate a sequence of response from an initial prompt. During this process, we treat the initial prompt as the initial state $s_1\sim\rho$ and each generated token $a_h\in\mathcal{A}$ as the action, where $\mathcal{A}$ is the vocabulary. By following this formulation, the state at step $h$ can be represented as $s_h=(s_1, a_1, \dots, a_{h-1})$ a concatenation of initial prompt and tokens generated before step $h$ and $s_H$ can be interpreted as the response of this language model to the initial prompt $s_1$ plus this prompt itself.
Clearly, from this formulation, the transition is deterministic from $s_h$ to $s_{h+1}$ given action $a_h$ since it is simply a concatenation. This is summarized as \textit{Deterministic Contextual MDPs} (DCMDPs) in \citet{xie2024exploratory}, which proposes that the optimal policy $\pi^\star_{\beta}$ of the objective \eqref{eq:KLobj} in a DCMDP satisfies
\begin{equation}
\label{eq:DCMDP}
\beta\log\frac{\pi^\star_{\beta}(\tau)}{\piref(\tau)}=r(\tau) - V^\star_{\beta}(s_1),\quad \forall\tau,
\end{equation}
where $V^\star_{\beta}=\max_{\pi\in\Pi}J_\beta(\pi)$. More details of DCMDP can be found in \citet{xie2024exploratory}.
To optimize objective \eqref{eq:KLobj}, two primary approaches are available: offline RLHF and online RLHF. The choice between these methods depends on the specific requirements of the application and we will discuss both approaches in details in the following sections.
% \paragraph{Direct Preference Optimization}
\subsection{Offline, Online and Hybrid RLHF}
\label{sec:rlhfpl}
\paragraph{Offline RLHF.}
Offline RLHF methods are restricted to the preferences captured by the reference model $\piref$ and the offline preference dataset $\mathcal{D}_{\rm off}$. Typically the offline preference dataset is a set of labelled pairs $\mathcal{D}_{\rm off} = \{(\tau_{+}^{(i)}, \tau_{-}^{(i)})\}_{i \in [N_{\rm off}]}$. To learn a policy from this offline dataset, one of the most popular approaches is Direct Preference Optimization (DPO) introduced in \citet{rafailov2024direct}, which also serves as a starting point of our work.
DPO is motivated by a closed-form solution for the policy that optimizes the KL-regularized objective in Eq.~\eqref{eq:KLobj}, and condenses the two-step process above into a single policy optimization
objective, removing the need for reward function estimation. Concretely, DPO solves the following problem:
\fontsize{8.5}{8.5}
\begin{align}\label{eq:dpo}
\widehat{\pi} = \argmin_{\pi \in \Pi} \sum_{(\tau_{+}, \tau_{-}) \in \mathcal{D}_{\rm off}} -\log \left[\sigma\left(\beta z(\tau_+, \tau_-, \piref, \pi) \right)\right],
\end{align}
\normalsize
where $z(\tau_+, \tau_-, \piref, \pi) = \log \frac{\pi(\tau_{+})}{\piref(\tau_{+})} - \log \frac{\pi(\tau_{-})}{\piref(\tau_{-})}$, $\sigma(x)=\frac{\exp(x)}{1+\exp(x)}$ is the sigmoid function and $\Pi$ is some user-specified policy class such as parameterized neural networks.
% \subsection{Online RLHF}
\paragraph{Online RLHF}
Although achieving great empirical success, offline DPO is inherently limited by the support of reference policy $\piref$ and offline dataset $\mathcal{D}_{\mathrm{off}}$. Therefore, people turn more attention into online DPO, which was first theoretically studied in \citet{xiong2024iterative}. Specifically, online RLHF collects preference feedback from pairs of responses generated by the language model sequentially updated during training.
Recently, \citet{xie2024exploratory} analyzed the necessity of deliberate exploration in online DPO and proposed XPO as a combination of DPO and optimistic exploration. In particular, the original DPO loss is regularized by $\alpha\sum_{i=1}^{t}\log\pi(\widetilde{\tau}^{(i)})$ in XPO to encourage the policy to be more diverse, where $\alpha>0$ is the regularization strength and $\widetilde{\tau}^{(i)}$ is the trajectory generated by $\piref$. It was analyzed under the framework of DCMDP and achieved $\widetilde{O}(1/\sqrt{T})$ optimality gap under appropriate assumptions, where $T$ represents the number of queried preference labels. Concurrently, similar theoretical guarantee was also obtained by \citet{cen2024value}.
% However, theoretically proven works \citep{xie2024exploratory}. \citep{cen2024value} discard the offline preference dataset $\mathcal{D}_{\rm off}$ in the process of learning. Typically, at each round $t \in [T]$, upon sampling an initial state $s_1^{(t)} \sim \rho$, a pair of responses $\tau_t \sim \pi^{(t)}(\cdot| s_1^{(t)})$ and $\tilde{\tau}_t \sim \tilde{\pi}^{(t)}(\cdot| s_1^{(t)})$ are sampled which get labeled as $(\tau_{+}^{(t)}, \tau_{-}^{(t)})$. This is now added to the online buffer as $\mathcal{D}_{\rm on}^{(t)} = \mathcal{D}_{\rm on}^{(t-1)} \cup (\tau_{+}^{(t)}, \tau_{-}^{(t)})$, and subsequently the policies are updated to $\pi^{(t+1)}, \tilde{\pi}^{(t+1)}$.
% Purely online RLHF breaks free from the lack of coverage of behaviors by $\piref$ and $\mathcal{D}_{\rm off}$. Formally, online RLHF collects preference feedback from pairs of responses generated by the language model sequentially updated during training.
% However, theoretically proven works \citep{xie2024exploratory}. \citep{cen2024value} discard the offline preference dataset $\mathcal{D}_{\rm off}$ in the process of learning. Online exploration proceeds in $T$ rounds, where $T$ represents the number of queries to label pairs of responses. Typically, at each round $t \in [T]$, upon sampling an initial state $s_1^{(t)} \sim \rho$, a pair of responses $\tau_t \sim \pi^{(t)}(\cdot| s_1^{(t)})$ and $\tilde{\tau}_t \sim \tilde{\pi}^{(t)}(\cdot| s_1^{(t)})$ are sampled which get labeled as $(\tau_{+}^{(t)}, \tau_{-}^{(t)})$. This is now added to the online buffer as $\mathcal{D}_{\rm on}^{(t)} = \mathcal{D}_{\rm on}^{(t-1)} \cup (\tau_{+}^{(t)}, \tau_{-}^{(t)})$, and subsequently the policies are updated to $\pi^{(t+1)}, \tilde{\pi}^{(t+1)}$.
\paragraph{Hybrid RLHF}
Although the efficiency of online RLHF is justified both theoretically and experimentally, in application, real-time query of labels can be expensive while a large amount of offline preference dataset can often be easily and cheaply obtained. Therefore, it is natural to consider a hybrid training scheme which can potentially combine the advantage of both offline and online RLHF.
From a theoretical perspective, the sample complexity of pure online RLHF depends solely on the coverage provided by $\piref$, meaning that the larger the coverage provided by $\piref$, the smaller the space to search for online RLHF, thereby requiring fewer samples.
However, pure online RLHF neglects the coverage provided by the offline preference dataset which can significantly reduce the search space. Therefore, we propose to use online feedback to explore trajectories that are not supported in $\piref$ while still taking advantage of the available offline dataset to speed up exploration. Informally, even starting with an offline dataset on which any offline algorithm may give arbitrarily poor policy, our proposed algorithm HPO will be able to use it to significantly reduce the sample complexity, as compared to XPO.
% \textbf{Informal Result.} Starting with an offline preference dataset on which any offline algorithm gives arbitrarily poor policy, we can use it to significantly reduce order of magnitude of sample complexity for the online exploration in HPO, as compared to XPO.
\section{Related Work}
\label{sec:related_work}
% \avi{Zhihan TODO}
% \zhihan{Still a draft}
\paragraph{DPO-related RLHF.}
The Direct Preference Optimization (DPO) algorithm, first introduced by \citet{rafailov2024direct}, has gained considerable attention due to its effectiveness and simplicity. Since its introduction, various DPO variants have been proposed, each with unique enhancements aimed at improving performance. Several of these methods focus on modifying the loss function, such as SLiC \citep{zhao2023slic}, RSO \citep{liu2023statistical}, P3O \citep{wu2024pairwise}, PCO \citep{xu2023some}, and SimPO \citep{meng2024simpo}. Other approaches take a broader view by incorporating general preference modeling, integrating social choice theory, and introducing algorithms like Nash-MD \citep{munos2023nash}, DNO \citep{rosset2024direct}, KTO \citep{ethayarajh2024kto}, and IPO \citep{azar2024general}.
DPO's popularity has also captured the interest of the theoretical research community. \citet{xiong2024iterative}. explored the benefits of online iterative training with an offline warm-up phase, while \citet{ye2024theoretical}. extended this analysis from a reward-based framework to a general preference oracle setting. Both studies, however, limit their analysis to pure bandit environments. \citet{rafailov2024r} further reinterpreted DPO through the lens of implicit $Q^\star$ estimation within KL-constrained deterministic token-level Markov Decision Processes (MDPs). \citet{gao2024rebel} introduced a study on relative reward modeling, while recent work by \citet{cen2024value} and \citet{xie2024exploratory} analyzed the sample efficiency of combining DPO with optimistic exploration. This research has also inspired the development of the online exploration component in our Hybrid Preference Optimization (HPO) algorithm.
Despite these advancements, none of these has deep dive into a hybrid training setting. \citet{cen2024value} and \citet{ye2024theoretical} have discussed offline and online learning separately, but neither has explored the potential benefits of combining the two. \citet{xiong2024iterative} addressed hybrid training but relied on impractical assumptions, such as coverage over the optimal policy $\pi^\star$, and failed to provide a quantitative characterization of hybrid RLHF compared to purely online or offline approaches. Similarly, while \citet{gao2024rebel} mentioned hybrid training in their work, they did not delve deeply into the topic, leaving an open area for further investigation. Finally, although the same algorithm name \textit{Hybrid Preference Optimization} is also used in \citet{song2024understanding}, that version is essentially an offline DPO regularized by unlabeled online data, which is very different from our approach of enhancing online exploration through additional offline data.
% - Since the emergence of DPO (Rafailov et al.), it draws a lot of attention because of its effectiveness and simplicity. Many DPO variants are then proposed. Some of them focus on modifying the loss of DPO to improve its performance such as SLiC (Zhao et al.), RSO (Liu et al.), P3O (Wu et al.), PCO (Xu et al.) and SimPO (Meng et al.); some of them consider more general preference and are combined with social choice theory such as Nash-MD (Munos et al.), DNO (Rosset et al.) KTO (Ethayarajh et al.), IPO (Azar et al.).
% - The popularity of DPO also draws attention from theoretical community. Xiong et al. studies the benefit of online iterative training with offline warm-up. Ye et al. extends Xiong et al. from reward-based setting to general preference oracle setting. However, both of them only give analysis in pure bandit setting. Rafailov et al. interprets DPO as implicit $Q^\star$ estimation in KL-constrained deterministic token-level MDPs. Gao et al. studies a relative reward modeling. Recently, both Cen et al. and Xie et al. study the sample efficiency of the combination of DPO and optimistic exploration, which also motivates the online exploration part of our HPO algorithm.
% - Among these, Cen et al. and Ye et al. discuss both offline and online learning separately, but does not study a combination of them. Xiong et al. discusses hybrid training, but requires coverage assumption over $\pi^\star$, proposes impractical algorithms and misses quantitative characterization of the advantage of hybrid RLHF over pure online and pure offline RLHF. Gao et al. mentions hybrid training but does not study it in details.
\paragraph{Other Theoretical Work in RLHF}
Other theoretical work on RLHF primarily centers around learning reward models and analyzing sample complexities for specific classes of Markov Decision Processes (MDPs). Among them, \citet{zhu2023principled} and \citet{zhu2024iterative} have focused on learning various reward models, while others, including \citet{du2024exploration}, \citet{zhan2023provable}, and \citet{wu2023making}, have addressed on sample complexities under tabular or linear function approximations. Some studies, such as those by \citet{chen2022human}, \citet{wang2023rlhf}, and \citet{chang2024dataset}, also explore the use of general function approximation methods. However, these algorithms are often designed around function classes with certain complexity measures, making them less applicable in practical settings. Besides, \citet{nika2024reward} provides a detailed theoretical comparison between DPO-related RLHF and reward-learning-related RLHF. Among these studies, \citet{chang2024dataset} provides a more detailed discussion on hybrid training in RLHF. Despite this, their work shares similar limitations to \citet{xiong2024iterative}, particularly in terms of too strong assumptions on single-concentrability coefficients.
% and the absence of practical algorithms that can be easily applied to real-world problems.
% - Other theoretical work of RLHF mostly focuses on learning reward models (Zhu et al., Wang et al.) or sample complexities tailored with respect to particular class of MDPs under either tabular or linear function approximation (Du et al. Zhan et al. Wu and Sun.). Some work (Chen et al. Wang et al. and Chang et al.) also considers general function approximation. However, these algorithms are usually tailored to function class with certain complexity measure and cannot lead to practical algorithms. Among these, Chang et al. has specific discussion regarding the hybrid training, but falls similar shortcomings as Xiong et al.
\paragraph{Theory of Hybrid RL.}
\citet{kalashnikov2018scalable} empirically demonstrate that in robot manipulation, a small amount of online fine-tunning can largely improve the performance of offline training. Following this, much of the theoretical work on hybrid RL has focused on quantitatively analyzing the advantages of combining online exploration with an initial offline dataset. Key studies by \citet{xie2021policy}, \citet{song2022hybrid}, and \citet{amortila2024harnessing} have explored these benefits, often under the assumption of certain concentrability conditions related to the behavior policy. However, recent research by \citet{tan2024hybrid} removes this requirement, offering an approach that does not impose explicit constraints on the quality of the behavior policy. Building on this line of research, our work extends the benefits of hybrid RL to the RLHF framework. Importantly, we adopt the same philosophy as \citet{tan2024hybrid} by not imposing additional requirements on the offline dataset, allowing for more flexible and practical applications in RLHF scenarios.
% - Kalashnikov et al. first empirically shows that hybrid RL can work better than pure online RL and pure offline RL in practice. Current theoretical work of hybrid RL mostly focuses on quantitatively analyzing the benefits of learning with initial offline dataset (Xie et al., Song et al., Amortila et al.). While most of them requires some kinds of concentrability condition of the behavior policy, this is removed in recent work of Tan et al, which puts no explicit requirements on the quality of the behavior policy. Motivated by this line of work, we brought the benefits of hybrid RL to RLHF, without imposing any additional requirements to the offline dataset.
% \paragraph{Sample Complexity Lower Bound.}
% \zhihan{May need some rewriting}
% The near-optimal sample complexity lower bound for online linear contextual bandits was first established by \citet{li2019nearly} This foundational work was later extended to linear Markov Decision Processes (MDPs) by \citet{zhou2021nearly} In parallel, \citet{saha2021optimal} derived the lower bound for preference-based linear contextual bandits. Building on these contributions, our work complements this research by analyzing the lower bound for preference-based linear MDPs, further advancing the understanding of sample complexity in preference-based settings.
% The near optimal sample complexity lower bound for online linear contextual bandit was established in \citet{li2019nearly}. This was then extended to the linear MDPs in \citet{zhou2021nearly}. The lower bound for preference-based linear contextual bandit was also established in \citet{saha2021optimal} and we complement this line of work by analyzing the lower bound preference-based linear MDPs.
% \paragraph{Empirical Hybrid Exploration in RLHF.}
% In algorithm~\ref{alg:hybrid_rlhf}, we highlight the schematic algorithm for hybrid RLHF.
% Prior Work:
% \begin{enumerate}
% \item Analysis in offline RLHF (\cite{xiong2024iterative, cen2024value}) typically needs reward function to be linear, ie. $r(x, \tau) = \langle \theta, \phi(x, \tau)\rangle$ with mild assumptions on $\theta$ and known feature map $\phi$. Find a criterion which captures the coverage of all the data seen so far (all offline + online data collected until this step).
% \item Section 4 in \cite{xiong2024iterative} considers the batch hybrid RLHF setting. Although their algorithm uses the offline dataset, both their theorems are essentially pure online or pure offline.
% \item As explored in \cite{xie2024exploratory, xiong2024iterative} online RLHF can utilitise a non-symmetric algorithmic structure : have 2 policies $\pi_t^1$ and $\pi_t^2$. \cite{xiong2024iterative} use nice terminology to distinguish the roles of these 2 policies. Main agent $\pi_t^1$ : This policy's goal is to maximize the objective, and will be the output of the algorithm. Enchancer $\pi_t^2$ : This policy's goal is to explore trajectory spaces unseen before. Updates of $\pi_t^1, \pi_t^2$ should trade off between (i) maximizing likelihood of all the preference data collected so far, (ii) incentivising exploration.
% Below is a summary of how lines 7,8 in Algorithm~\ref{alg:hybrid_rlhf} differ for prior work.
% XPO (\cite{xie2024exploratory}) study only the online setting ($\mathcal{D}_{\rm off} = \emptyset$) and keep $\pi_t^2 = \pi_{\rm ref}$ fixed throughout. $\pi_t^1$ is updated via adding an exploration term $\sum_{i=1}^t \log \pi(\tau_i^2)$ to the DPO loss, ie.
% \begin{align*}
% \pi_{t+1}^1 = \argmin_{\pi \in \Pi} \bigg[\alpha \sum_{i=1}^t \log \pi(\tau_i^2) -\sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^t_{\rm on}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})}) \bigg] \bigg].
% \end{align*}
% In \cite{xiong2024iterative}, $\pi_t^1$ is updated by optimizing the DPO loss:
% \begin{align*}
% \pi_{t+1}^1 = \argmin_{\pi \in \Pi} \bigg[ -\sum_{(\tau^{+}, \tau^{-}) \in \mathcal{D}^t_{\rm on} \cup \mathcal{D}_{\rm off}} \log \bigg[ \sigma(\beta \log \frac{\pi(\tau^{+})}{\pi_{\rm ref}(\tau^{+})} - \beta \log \frac{\pi(\tau^{-})}{\pi_{\rm ref}(\tau^{-})}) \bigg] \bigg],
% \end{align*}
% while $\pi_t^2$ is updated via maximizing the relative uncertainty to $\pi_t^1$. Under a linear reward model with feature map $\phi(x, \tau)$:
% \begin{align*}
% \pi_t^2 = \amax_{\pi \in \Pi_t}\|\mathbb{E}_{x \sim \rho}[\mathbb{E}_{\tau \sim \pi(\cdot | x)}\phi(x, \tau) - \mathbb{E}_{\tau \sim \pi_t^1(\cdot | x)}\phi(x, \tau)]\|_{\Sigma_t^{-1}},
% \end{align*}
% where
% $$\Pi_t = \{\pi \in \Pi ~|~ \mathbb{E}_{x \sim \rho} D_{\rm KL}(\pi(\cdot | x) || \pi_t^1(\cdot | x)) \leq \|\mathbb{E}_{x \sim \rho} [\mathbb{E}_{\tau \sim \pi(\cdot | x)}\phi(x, \tau) - \mathbb{E}_{\tau \sim \pi_t^1(\cdot | x)}\phi(x, \tau)]\|_{\Sigma_t^{-1}}\},$$
% and $\Sigma_t = \sum_{i=1}^t(\phi(x_i, \tau_i^{+}) - \phi(x_i, \tau_i^{-}))(\phi(x_i, \tau_i^{+}) - \phi(x_i, \tau_i^{-}))^\top + \lambda \mathbb{I}$. The idea is to identify a policy ($\pi_{t}^2$) to maximize information gain wrt the main agent $\pi_t^1$, while maintaining a moderate KL divergence to it.
% \end{enumerate}
% \begin{algorithm}
% \caption{Hybrid RLHF}
% \begin{algorithmic}[1]
% \STATE \textbf{Input:} Offline dataset $\mathcal{D}_{\rm off}$
% \STATE \textbf{Initialize:} Main agent $\pi_t^1$, Enhancer $\pi_t^2$ appropriately, $\mathcal{D}^0_{\rm on} = \emptyset$
% \FOR{$t = 1, \ldots, T$}
% \STATE \textbf{Generate response pair:} $x_t \sim \rho$; $\tau_t^1 \sim \pi_t^1(\cdot|x_t), \tau_t^2 \sim \pi_t^2(\cdot|x_t)$.
% \STATE \textbf{Label with Preference:} Label $(\tau_t^1, \tau_t^2)$ as $(\tau_t^{+}, \tau_t^{-})$ based on preference feedback.
% \STATE \textbf{Update Online Dataset:} $\mathcal{D}^t_{\rm on} \gets \mathcal{D}^{t-1}_{\rm on} \cup (\tau_t^{+}, \tau_t^{-})$
% \STATE \textbf{Update Main Agent:} Update $\pi_t^1$ to maximize likelihood of preferences seen so far + regularization terms
% \STATE \textbf{Update Enhancer:} Update $\pi_t^2$ so future trajectories satisfy (i) diverse from the trajectories already seen in $\mathcal{D}_{\rm off} \cup \mathcal{D}^t_{\rm on}$, (ii) diverse from trajectories to be generated from $\pi_t^1$.
% \ENDFOR
% \STATE \textbf{Return} $\pi_T^1$.
% \end{algorithmic}
% \label{alg:hybrid_rlhf}
% % \vskip -0.3in
% \end{algorithm}
\newtheorem{property}{Property}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{assumption}{Assumption}
\newtheorem{proposition}{Proposition}
\newtheorem{example}{Example}[section]
\newtheorem{definition}{Definition}[section]
\usepackage{enumitem}
\usepackage{thmtools,thm-restate}
\theoremstyle{remark}
\newtheorem*{remark}{Remark}
\newcommand{\pisamp}{\wt{\pi}}
\newcommand{\samp}{\mb{\pi}_{\mathsf{samp}}}
% \newcommand{\bmu}{\mb{\mu}}
\newcommand{\bmu}{\mathbf{\mu}}
\newcommand{\coeff}{\mathrm{SEC}_{\mathrm{HybRLHF}}}
\newcommand{\coefffull}{\coeff(\Pi,T,\beta, \pi_{\rm samp}; \gamma, \mathcal{D}_{\rm off})}
\newcommand{\cDopt}{\cD_{\mathsf{opt}}}
\newcommand{\Lhat}{\wh{L}}%
\newcommand{\bhat}{\wh{b}}%
\newcommand{\Bhat}{\wh{B}}%
% \newcommand{\tautil}{\wt{\tau}}%
\newcommand{\stil}{\wt{s}}
\newcommand{\atil}{\wt{a}}
\newcommand{\cA}{{\mathcal A}}
\newcommand{\Jb}{J_{\beta}}
\newcommand{\Ccov}{C_{\mathsf{cov}}}
\newcommand{\Cconc}{C_{\mathsf{conc}}}
\newcommand{\mainalg}{Exploratory Preference Optimization\xspace}
\newcommand{\alglong}{\mainalg}
\newcommand{\algshort}{\texttt{XPO}\xspace} %
\newcommand{\dpo}{\texttt{DPO}\xspace}
\newcommand{\taup}{\tau_{+}}
\newcommand{\taum}{\tau_{-}}
\newcommand{\pistarb}{\pistar_{\beta}}
\newcommand{\Qstarb}{Q^{\star}_{\beta}}
\newcommand{\Vstarb}{V^{\star}_{\beta}}
\newcommand{\ap}{a_{+}}%
\newcommand{\am}{a_{-}}%
\newcommand{\Shat}{\wh{S}}
\newcommand{\Astar}{A^{\pistar}}
\newcommand{\sigmab}{\wb{\sigma}}
\newcommand{\sigmabs}{\wb{\sigma}_{\pistar}}
\newcommand{\Hstar}{H^{\star}}%
\newcommand{\pitil}{\wt{\pi}}%
\newcommand{\Jhat}{\wh{J}}
\newcommand{\gbar}{\wb{g}}
\newcommand{\mutil}{\wt{\mu}}
\newcommand{\sigmastar}{\sigma_{\pistar}}
\newcommand{\Lmax}{L_{\texttt{max}}}
\newcommand{\Lmaxhat}{\wh{L}_{\texttt{max}}}
\newcommand{\vepsapx}{\veps_{\texttt{apx}}}
\newcommand{\Drho}[2]{\rho\prn*{#1\dmid{}#2}}
\newcommand{\murec}{\mu}
\newcommand{\astart}[1][t]{a^{\star,\sss{#1}}}
\newcommand{\astari}[1][i]{a^{\star,\sss{#1}}}
\newcommand{\vepsapxh}{\veps_{\texttt{apx},h}}
\newcommand{\Qstar}{Q^{\star}}
\newcommand{\Vstar}{V^{\star}}
\newcommand{\Lbc}{L_{\textsf{bc}}}
\newcommand{\Lhatbc}{\wh{L}_{\textsf{bc}}}
\newcommand{\traj}{o}
\newcommand{\aggrevate}{\texttt{Aggrevate}\xspace}
\newcommand{\loglossbc}{\texttt{LogLossBC}\xspace}
\newcommand{\loglossdagger}{\texttt{LogLossDagger}\xspace}
\newcommand{\Unif}{\mathsf{Unif}}
\newcommand{\phistar}{\phi^{\star}}
\newcommand{\gstar}{g^{\star}}
\newcommand{\pibar}{\wb{\pi}}
\newcommand{\Cpush}{C_{\mathsf{push}}}
\newcommand{\sfrak}{\mathfrak{s}}
\newcommand{\afrak}{\mathfrak{a}}
\newcommand{\bfrak}{\mathfrak{b}}
\newcommand{\cfrak}{\mathfrak{c}}
\newcommand{\xfrak}{\mathfrak{x}}
\newcommand{\yfrak}{\mathfrak{y}}
\newcommand{\zfrak}{\mathfrak{z}}
\newcommand{\bxi}{\mb{\xi}}
\newcommand{\bs}{\mathbf{s}}
\newcommand{\ba}{\mathbf{a}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\br}{\mathbf{r}}
\newcommand{\vepsstat}{\veps_{\mathrm{stat}}}
\newcommand{\ppr}{p_{\mathrm{pr}}}
\newcommand{\ppo}{p_{\mathrm{po}}}
\newcommand{\gammastar}{\gamma^{\star}}
\newcommand{\compc}[1][\veps]{\mathsf{dec}_{#1}}
\newcommand{\var}{\mathrm{Var}}
\newcommand{\Var}{\var}
\newcommand{\varplus}{\mathrm{Var}_{\sss{+}}}
\newcommand{\Varplus}{\varplus}
\newcommand{\Varm}[1][M]{\Var\sups{#1}}
\newcommand{\Varmbar}[1][\Mbar]{\Var\sups{#1}}
\newcommand{\pmbar}{p_{\sMbar}}
\newcommand{\pmdist}{p\subs{M}}
\newcommand{\PM}{\bbP\sups{M}}
\newcommand{\PMbar}{\bbP\sups{\Mbar}}
\newcommand{\Dbi}[2]{D_{\mathrm{bi}}\prn*{#1\dmid#2}}
\newcommand{\Dbishort}{D_{\mathrm{bi}}}
\newcommand{\cQm}{\cQ_{\cM}}
\newcommand{\cQmh}[1][h]{\cQ_{\cM;#1}}
\newcommand{\cTm}[1][M]{\cT\sups{#1}}
\newcommand{\cTmstar}[1][\Mstar]{\cT\sups{#1}}
\newcommand{\Confone}{R_1}
\newcommand{\Conftwo}{R_2}
\newcommand{\compbi}{\compgen[\mathrm{bi}]}
\newcommand{\compbidual}{\compgendual[\mathrm{bi}]}
\newcommand{\bloss}{\cL}
\newcommand{\cGmbar}{\cG\sups{\Mbar}}
\newcommand{\gmmbar}[1][M]{g\sups{#1;\Mbar}}
\newcommand{\Lbr}{L_{\mathrm{br}}}
\newcommand{\Lbrfull}{L_{\mathrm{br}}(\cM)}
\newcommand{\Lbrfulls}{L^{2}_{\mathrm{br}}(\cM)}
\newcommand{\mum}[1][M]{\mu\sups{#1}}
\newcommand{\wm}[1][M]{w\sups{#1}}
\newcommand{\phim}[1][M]{\phi\sups{#1}}
\newcommand{\AlgEsth}[1][h]{\mathrm{\mathbf{Alg}}_{\mathsf{Est};#1}}
\newcommand{\EstHelh}[1][h]{\mathrm{\mathbf{Est}}_{\mathsf{H};#1}}
\newcommand{\thetam}[1][M]{\theta(#1)}
\newcommand{\thetambar}[1][\Mbar]{\theta(#1)}
\newcommand{\Copt}{C_{\mathrm{opt}}}
\newcommand{\Calpha}{C_{\alpha}}
\newcommand{\dimbi}{d_{\mathrm{bi}}}
\newcommand{\Lbi}{L_{\mathrm{bi}}}
\newcommand{\dimbifull}{d_{\mathrm{bi}}(\cM)}
\newcommand{\dimbifulls}[1][2]{d^{#1}_{\mathrm{bi}}(\cM)}
\newcommand{\Lbifull}{L_{\mathrm{bi}}(\cM)}
\newcommand{\Lbifulls}{L^{2}_{\mathrm{bi}}(\cM)}
\newcommand{\piest}{\pi^{\mathrm{est}}}
\newcommand{\piestm}[1][M]{\pi^{\mathrm{est}}_{\sss{#1}}}
\newcommand{\lest}{\ell^{\mathrm{est}}}
\newcommand{\lestm}[1][M]{\ell^{\mathrm{est}}_{\sss{#1}}}
\newcommand{\Piest}{\Pi^{\mathrm{est}}}
\newcommand{\qbar}{\bar{q}}
\newcommand{\qopt}{q^{\mathrm{opt}}}
\newcommand{\pialpha}{\pi^{\alpha}}
\newcommand{\pialpham}[1][M]{\pi^{\alpha}_{\sss{#1}}}
\newcommand{\piref}{\pi_{\rm ref}}
% \newcommand{\pisamp}{\pi_{\rm samp}}
\newcommand{\taut}{\Tilde{\tau}}
\newcommand{\con}{x}
\newcommand{\Cspace}{\cX}
\newcommand{\cMx}[1][x]{\cM|_{#1}}
\newcommand{\cMhatx}[1][x]{\cMhat|_{#1}}
\newcommand{\cFmx}[1][x]{\cFm|_{#1}}
\newcommand{\cpol}{\mb{\pi}}
\newcommand{\cpolstar}{\cpol^{\star}}
\newcommand{\cpolm}[1][M]{\cpol\subs{#1}}
\newcommand{\RegCDM}{\Reg_{\mathsf{DM}}}
\newcommand{\EstCD}{\mathrm{\mathbf{Est}}_{\mathsf{D}}}
\newcommand{\relu}{\mathsf{relu}}
\newcommand{\sdis}{\mb{\theta}}
\newcommand{\rdis}{\mb{\psi}}
\newcommand{\Deltanot}{\Delta_0}
\newcommand{\Delm}{\Delta\subs{M}}
\newcommand{\Star}{\mathfrak{s}}
\newcommand{\StarCheck}{\check{\Star}}
\newcommand{\StarDim}{\Star}
\newcommand{\Stardim}{\Star}
\newcommand{\El}{\mathfrak{e}}
\newcommand{\ElCheck}{\check{\El}}
\newcommand{\ElDim}{\El}
\newcommand{\Eldim}{\El}
\newcommand{\Rmax}{R_{\rm max}}
\newcommand{\Vmax}{V_{\rm max}}
\newcommand{\errm}[1][M]{\mathrm{err}\sups{#1}}
\newcommand{\hardfamily}{$(\alpha,\beta,\delta)$-family\xspace}
\newcommand{\hardu}{u}
\newcommand{\hardv}{v}
\newcommand{\igwshort}{IGW\xspace}
\newcommand{\igwtext}{inverse gap weighting\xspace}
\newcommand{\pcigw}{\textsf{PC-IGW}\xspace}
\newcommand{\pcigwb}{\textsf{PC-IGW.Bilinear}\xspace}
\newcommand{\pcigwtext}{policy cover inverse gap weighting\xspace}
\newcommand{\igwspanner}{\textsf{IGW-Spanner}\xspace}
\newcommand{\igwargmax}{\textsf{IGW-ArgMax}\xspace}
\newcommand{\oracle}{\mathrm{\mathbf{Alg}}_{\mathsf{Plan}}\xspace}
\newcommand{\MinimaxReg}{\mathfrak{M}(\cM,T)}
\newcommand{\MinimaxRegBayes}{\underbar{\mathfrak{M}}(\cM,T)}
\renewcommand{\emptyset}{\varnothing}
\newcommand{\NullObs}{\crl{\emptyset}}
\newcommand{\filt}{\mathscr{F}}
\newcommand{\ffilt}{\mathscr{F}}
\newcommand{\gfilt}{\mathscr{G}}
\newcommand{\hfilt}{\mathscr{H}}
\newcommand{\hist}{\mathcal{H}}
\newcommand{\Asig}{\mathscr{P}}
\newcommand{\Rsig}{\mathscr{R}}
\newcommand{\Osig}{\mathscr{O}}
\newcommand{\borel}{\mathfrak{B}}
\newcommand{\Hspace}{\Omega}
\newcommand{\Hsig}{\filt}
\newcommand{\dom}{\nu}
\newcommand{\dens}{m}
\newcommand{\densm}[1][M]{m^{\sss{#1}}}
\newcommand{\abscont}{V(\cM)}
\newcommand{\abscontp}{V(\cM')}
\newcommand{\Vmbar}{V^{\sMbar}}
\newcommand{\vepsup}{\bar{\veps}}
\newcommand{\vepslow}{\ubar{\veps}}
\newcommand{\vepsg}{\veps_{\gamma}}
\newcommand{\vepsupg}{\bar{\veps}_{\gamma}}
\newcommand{\vepslowg}{\ubar{\veps}_{\gamma}}
\newcommand{\gameval}[1]{\cV_{\gamma}\sups{#1}}
\newcommand{\Framework}{Decision Making with Structured Observations\xspace}
\newcommand{\FrameworkShort}{DMSO\xspace}
\newcommand{\learner}{learner\xspace}
\newcommand{\Cloc}{c_{\ell}}
\newcommand{\Cglobal}{C(\cM)}
\newcommand{\act}{\pi}
\newcommand{\Act}{\Pi}
\newcommand{\ActSpace}{\Pi}
\newcommand{\ActSize}{\abs{\Pi}}
\newcommand{\ActSizeB}{\abs{\cA}}
\newcommand{\ActSizeRL}{\abs{\cA}}
\newcommand{\ActDist}{\Delta(\Act)}
\newcommand{\action}{decision\xspace}
\newcommand{\actions}{decisions\xspace}
\newcommand{\actstar}{\pistar}
\newcommand{\actb}{a}
\newcommand{\StateSize}{\abs{\cS}}
\newcommand{\obs}{o}
\newcommand{\Obs}{\mathcal{\cO}}
\newcommand{\ObsSpace}{\mathcal{\cO}}
\newcommand{\Ospace}{\mathcal{\cO}}
\newcommand{\RewardSpace}{\cR}
\newcommand{\RSpace}{\RewardSpace}
\newcommand{\Rspace}{\RewardSpace}
\newcommand{\Model}{\cM}
\newcommand{\cMhat}{\wh{\cM}}
\newcommand{\compbasic}{\mathsf{dec}}
\newcommand{\comp}[1][\gamma]{\mathsf{dec}_{#1}}
\newcommand{\comprand}[1][\gamma]{\mathsf{dec}^{\mathrm{rnd}}_{#1}}
\newcommand{\compshort}[1][\gamma]{\mathsf{d}_{#1}}
\newcommand{\compt}[1][\gamma]{\wt{\mathsf{dec}}_{#1}}
\newcommand{\compbar}[1][\gamma]{\overline{\mathsf{dec}}_{#1}}
\newcommand{\compb}[1][\gamma]{\underline{\mathsf{dec}}_{#1}}
\newcommand{\compdual}[1][\gamma]{\compb[#1]}
\newcommand{\compbayes}[1][\gamma]{\compb[#1]}
\newcommand{\compKL}{\compgen[\mathrm{KL}]}
\newcommand{\compKLdual}{\compgendual[\mathrm{KL}]}
\newcommand{\compSq}{\compgen[\mathrm{Sq}]}
\newcommand{\compSqdual}{\compgendual[\mathrm{Sq}]}
\newcommand{\compopt}{\comp^{\mathrm{opt}}}
\newcommand{\compoptdual}{\compb^{\mathrm{opt}}}
\newcommand{\compH}{\compgen[\mathrm{H}]}
\newcommand{\compHdual}{\compgendual[\mathrm{H}]}
\newcommand{\complocshort}{\comp^{\mathrm{loc}}}
\newcommand{\comploc}[2][\gamma]{\mathsf{dec}_{#1,#2}}
\newcommand{\compthmone}{\comploc{\vepslowg}(\cM)}
\newcommand{\compthmtwo}{\comploc{\vepsupg}(\conv(\cM))}
\newcommand{\comptext}{decision-estimation coefficient\xspace}
\newcommand{\Comptext}{Decision-estimation coefficient\xspace}
\newcommand{\CompText}{Decision-Estimation Coefficient\xspace}
\newcommand{\CompAbbrev}{DEC\xspace}
\newcommand{\CompShort}{\CompAbbrev}
\newcommand{\compgen}[1][D]{\comp^{#1}}
\newcommand{\compgendual}[1][D]{\compb^{#1}}
\newcommand{\compd}[1][D]{\comp^{#1}}
\newcommand{\est}{\mathsf{est}}
\newcommand{\M}[1]{^{{\scriptscriptstyle M}}} %
\newcommand{\sM}{\sss{M}}
\newcommand{\sMbar}{\sss{\Mbar}}
\renewcommand{\ss}[1]{^{{\scriptscriptstyle#1}}}
\newcommand{\sups}[1]{^{{\scriptscriptstyle#1}}}
\newcommand{\subs}[1]{_{{\scriptscriptstyle#1}}}
\renewcommand{\sp}[1]{^{{\scriptscriptstyle#1}}}
\renewcommand{\sb}[1]{_{{\scriptscriptstyle#1}}}
\newcommand{\sss}[1]{{\scriptscriptstyle#1}}
\newcommand{\Ens}[2]{\En^{\sss{#1},#2}}
\newcommand{\Em}[1][M]{\En^{\sss{#1}}}
\newcommand{\Embar}{\En\sups{\Mbar}}
\newcommand{\Enm}[1][M]{\En^{\sss{#1}}}
\newcommand{\Enmbar}[1][\Mbar]{\En^{\sss{#1}}}
\newcommand{\Prs}[1]{\bbP\sups{#1}}
\newcommand{\bbPs}[1]{\bbP\sups{#1}}
\newcommand{\bbPm}[1][M]{\bbP\sups{#1}}
\newcommand{\bbPmbar}[1][\Mbar]{\bbP\sups{#1}}
\newcommand{\fmhat}{f\sups{\Mhat}}
\newcommand{\fm}[1][M]{f\sups{#1}}
\newcommand{\pim}[1][M]{\pi_{\sss{#1}}}
\newcommand{\pimh}[1][h]{\pi_{\sss{M,#1}}}
\newcommand{\gm}{g\sups{M}}
\newcommand{\Mprime}{M'}
\newcommand{\pimprime}{\pi_{\sss{M'}}}
\newcommand{\cFm}{\cF_{\cM}}
\newcommand{\Pim}{\Pi_{\cM}}
\newcommand{\cMf}{\cM_{\cF}}
\newcommand{\fmbar}{f\sups{\Mbar}}
\newcommand{\pimbar}{\pi\subs{\Mbar}}
\newcommand{\fmstar}{f\sups{\Mstar}}
\newcommand{\pimstar}{\pi\subs{\Mstar}}
\newcommand{\fstar}{f^{\star}}
\newcommand{\pistar}{\pi^{\star}}
\newcommand{\pihat}{\wh{\pi}}
\newcommand{\cMloc}[1][\veps]{\cM_{#1}}
\newcommand{\Mbar}{\wb{M}}
\newcommand{\fmi}{f\sups{M_i}}
\newcommand{\pimi}{\pi\subs{M_i}}
\newcommand{\Jm}{J\sups{M}}
\newcommand{\Rm}[1][M]{R\sups{#1}}
\newcommand{\Pm}[1][M]{P\sups{#1}}
\newcommand{\Pmstar}[1][\Mstar]{P\sups{#1}}
\newcommand{\Pmbar}{P\sups{\Mbar}}
\newcommand{\Rmbar}{R\sups{\Mbar}}
\newcommand{\Prm}[2]{\bbP^{\sss{#1},#2}}
\newcommand{\Jmstar}{J\sups{\Mstar}}
\newcommand{\PiNS}{\Pi_{\mathrm{NS}}} %
\newcommand{\PiRNS}{\Pi_{\mathrm{RNS}}} %
\newcommand{\PiNM}{\Pi_{\mathrm{NM}}} %
\newcommand{\PiGen}{\Pi_{\mathrm{RNS}}} %
\newcommand{\Qmistar}[1][i]{Q^{#1,\star}}
\newcommand{\Vmistar}[1][i]{V^{#1,\star}}
\newcommand{\Qmstar}[1][M]{Q^{\sss{#1},\star}}
\newcommand{\Vmstar}[1][M]{V^{\sss{#1},\star}}
\newcommand{\Qmpi}[1][\pi]{Q^{\sss{M},#1}}
\newcommand{\Vmpi}[1][\pi]{V^{\sss{M},#1}}
\newcommand{\Qmbarpi}[1][\pi]{Q^{\sss{M},#1}}
\newcommand{\Vmbarpi}[1][\pi]{V^{\sss{M},#1}}
\newcommand{\Qm}[2]{Q^{\sss{#1},#2}}
\newcommand{\Vm}[2]{V^{\sss{#1},#2}}
\newcommand{\dmpi}[1][\pi]{d^{\sss{M},#1}}
\newcommand{\dmbarpi}[1][\pi]{d^{\sss{\Mbar},#1}}
\newcommand{\dm}[2]{d^{\sss{#1},#2}}
\newcommand{\dbar}{\bar{d}}
\newcommand{\dtil}{\wt{d}}
\newcommand{\Reg}{\mathrm{\mathbf{Reg}}}
\newcommand{\RegBar}{\widebar{\mathrm{\mathbf{Reg}}}}
\newcommand{\Est}{\mathrm{\mathbf{Est}}}
\newcommand{\EstBar}{\widebar{\mathrm{\mathbf{Est}}}}
\newcommand{\EstHel}{\mathrm{\mathbf{Est}}_{\mathsf{H}}}
\newcommand{\EstOnHel}{\mathrm{\mathbf{Est}}^{\mathsf{on}}_{\mathsf{H}}}
\newcommand{\EstBarHel}{\widebar{\mathrm{\mathbf{Est}}}_{\mathsf{H}}}
\newcommand{\EstProbHel}{\mathrm{\mathbf{Est}}_{\mathsf{H}}(T,\delta)}
\newcommand{\EstH}{\EstHel}
\newcommand{\EstD}{\mathrm{\mathbf{Est}}_{\mathsf{D}}}
\newcommand{\EstSq}{\mathrm{\mathbf{Est}}_{\mathsf{Sq}}}
\newcommand{\EstKL}{\mathrm{\mathbf{Est}}_{\mathsf{KL}}}
\newcommand{\RegDM}{\Reg_{\mathsf{DM}}}
\newcommand{\RegBarDM}{\RegBar_{\mathsf{DM}}}
\newcommand{\RegB}{\RegBandit}
\newcommand{\RegMAB}{\RegBandit}
\newcommand{\RegBarMAB}{\RegBar_{\mathsf{B}}}
\newcommand{\RegBandit}{\mathrm{\mathbf{Reg}}_{\mathsf{B}}}
\newcommand{\RegRL}{\Reg_{\mathsf{RL}}}
\newcommand{\RegBarRL}{\RegBar_{\mathsf{RL}}}
\newcommand{\RegOL}{\mathrm{\mathbf{Reg}}_{\mathsf{OL}}(T,\delta)}
\newcommand{\AlgOL}{\mathrm{\mathbf{Alg}}_{\mathsf{OL}}}
\newcommand{\RegOLShort}{\Reg_{\mathsf{OL}}}
\newcommand{\RegBR}{\Reg_{\mathsf{BR}}}
\newcommand{\RegSB}{\Reg_{\mathsf{SB}}}
\newcommand{\RegSquare}{\Reg_{\mathsf{Sq}}}
\newcommand{\RegCB}{\mathrm{\mathbf{Reg}}_{\mathsf{CB}}}
\newcommand{\RegBarCB}{\widebar{\mathrm{\mathbf{Reg}}}_{\mathsf{CB}}}
\newcommand{\AlgEst}{\mathrm{\mathbf{Alg}}_{\mathsf{Est}}}
\newcommand{\logloss}{\ls_{\mathrm{\log}}}
\newcommand{\RegLog}{\Reg_{\mathsf{KL}}}
\newcommand{\RegBarLog}{\widebar{\Reg}_{\mathsf{KL}}}
\newcommand{\RegLogT}{\Reg_{\mathsf{KL}}(T)}
\newcommand{\MComp}{\mathsf{est}(\cM,T)}
\newcommand{\Mcomp}{\mathsf{est}(\cM,T)}
\newcommand{\GComp}{\mathsf{est}(\cG,T)}
\newcommand{\Gcomp}{\mathsf{est}(\cG,T)}
\newcommand{\QComp}{\mathsf{est}(\cQm,T)}
\newcommand{\Qcomp}{\mathsf{est}(\cQm,T)}
\newcommand{\PComp}{\mathsf{est}(\cP,T)}
\newcommand{\PComph}[1][h]{\mathsf{est}(\cP_{#1},T)}
\newcommand{\ActComp}{\mathsf{est}(\Pim,T)}
\newcommand{\AComp}{\mathsf{est}(\Pim,T)}
\newcommand{\Acomp}{\mathsf{est}(\Pim,T)}
\newcommand{\MCov}[1][\veps]{\cN(\cM,#1)}
\newcommand{\ActCov}[1][\veps]{\cN(\Pim,#1)}
\newcommand{\mathrmb}[1]{\mathrm{\mathbf{#1}}}
\newcommand{\Mhat}{\wh{M}}
\newcommand{\Mstar}{M^{\star}}
\newcommand{\bbPl}{\bbP_{\lambda}}
\newcommand{\bbQl}{\bbQ_{\lambda}}
\newcommand{\cAhat}{\wh{\cA}}
\newcommand{\ghat}{\wh{g}}
\newcommand{\thetahat}{\wh{\theta}}
\newcommand{\thetastar}{\theta^{\star}}
\newcommand{\tens}{\otimes}
\newcommand{\Rbar}{\wb{R}}
\newcommand{\Pbar}{\widebar{P}}
\newcommand{\starucb}{star hull UCB\xspace}
\newcommand{\starucblong}{star hull upper confidence bound\xspace}
\newcommand{\optionone}{\textsc{Option I}\xspace}
\newcommand{\optiontwo}{\textsc{Option II}\xspace}
\newcommand{\Dis}{\mathrm{Dis}}
\newcommand{\xdist}{\cD}
\newcommand{\rdist}{\bbP_{r}}
\newcommand{\algcomment}[1]{\textcolor{blue!70!black}{\footnotesize{\texttt{\textbf{//\hspace{2pt}#1}}}}}
\newcommand{\algcommentlight}[1]{\textcolor{blue!70!black}{\transparent{0.5}{\scriptsize{\texttt{\textbf{//\hspace{2pt}#1}}}}}}
\newcommand{\algcommentbig}[1]{\textcolor{blue!70!black}{\footnotesize{\texttt{\textbf{/*
#1~*/}}}}}
\newcommand{\algcolor}[1]{\textcolor{blue!70!black}{#1}}
\newcommand{\algspace}{\hspace{\algorithmicindent}}
\newcommand{\Holder}{H{\"o}lder\xspace}
\newcommand{\Otilde}{\wt{O}}
\newcommand{\midsem}{\,;}
\newcommand{\trn}{\top}
\newcommand{\pinv}{\dagger}
\newcommand{\psdleq}{\preceq}
\newcommand{\psdgeq}{\succeq}
\newcommand{\psdlt}{\prec}
\newcommand{\psdgt}{\succ}
\newcommand{\approxleq}{\lesssim}
\newcommand{\approxgeq}{\gtrsim}
\newcommand{\opnorm}{\mathsf{op}}
\newcommand{\eigmin}{\lambda_{\mathrm{min}}}
\newcommand{\eigmax}{\lambda_{\mathrm{max}}}
% \renewcommand{\tC}{\tilde{\cC}}
\newcommand{\zero}{\mathbf{0}}
\newcommand{\Id}{I}
\newcommand{\bq}{\mathbf{q}}
\newcommand{\xpi}{x^{\pi}}
\newcommand{\phipi}{\phi^{\pi}}
\newcommand{\lw}{\bar{w}}
\newcommand{\bu}{\mb{u}}
\newcommand{\bv}{\mb{v}}
\newcommand{\bz}{\mb{z}}
\newcommand{\bw}{\mb{w}}
\newcommand{\xt}{x_t}
\newcommand{\xto}{x_{t+1}}
\newcommand{\expone}{(1)}
\newcommand{\exptwo}{(2)}
\newcommand{\xl}[1][T]{x_{1:#1}}
\newcommand{\xr}[1][t]{x_{#1:T}}
\newcommand{\ul}[1][T]{u_{1:#1}}
\newcommand{\ur}[1][t]{u_{#1:T}}
\newcommand{\wl}[1][T]{w_{1:#1}}
\renewcommand{\wr}[1][t]{w_{#1:T}}
\newcommand{\fhat}{\wh{f}}
\newcommand{\ahat}{\wh{a}}
\newcommand{\fbar}{\bar{f}}
\newcommand{\ind}[1]{^{{\scriptscriptstyle(#1)}}}
\newcommand{\xh}{x_h}
\newcommand{\rh}{r_h}
\renewcommand{\rh}{r_h}
\newcommand{\xhp}{x_{h+1}}
\newcommand{\bigoh}{O}
\newcommand{\bigoht}{\wt{O}}
\newcommand{\bigom}{\Omega}
\newcommand{\bigomt}{\wt{\Omega}}
\newcommand{\bigthetat}{\wt{\Theta}}
\newcommand{\indic}{\mathbb{I}}
\newcommand{\overleq}[1]{\overset{#1}{\leq{}}}
\newcommand{\overgeq}[1]{\overset{#1}{\geq{}}}
\newcommand{\overlt}[1]{\overset{#1}{<}}
\newcommand{\overgt}[1]{\overset{#1}{>}}
\newcommand{\overeq}[1]{\overset{#1}{=}}
\renewcommand{\Pr}{\bbP}
\newcommand{\ftilde}{\tilde{f}}
\newcommand{\fcheck}{\check{f}}
\newcommand{\gtilde}{\tilde{g}}
\newcommand{\empcov}{\wh{\cF}}
\newcommand{\Sbad}{S_{\mathrm{bad}}}
\newcommand{\aprime}{a'}
\newcommand{\undertext}[2]{\underbrace{#1}_{\text{#2}}}
\newcommand{\fup}{\bar{f}}
\newcommand{\flow}{\underbar{f}}
\newcommand{\aone}{a\ind{1}}
\newcommand{\atwo}{a\ind{2}}
\newcommand{\cb}{\mathsf{A}}
\newcommand{\poly}{\mathrm{poly}}
\newcommand{\polylog}{\mathrm{polylog}}
\newcommand{\Dsq}[2]{D_{\mathsf{Sq}}\prn*{#1,#2}}
\newcommand{\Dsqshort}{D_{\mathsf{Sq}}}
\newcommand{\kl}[2]{D_{\mathsf{KL}}\prn*{#1\,\|\,#2}}
\newcommand{\Dkl}[2]{D_{\mathsf{KL}}\prn*{#1\,\|\,#2}}
\newcommand{\Dklshort}{D_{\mathsf{KL}}}
\newcommand{\Dhel}[2]{D_{\mathsf{H}}\prn*{#1,#2}}
\newcommand{\Dgen}[2]{D\prn*{#1\dmid{}#2}}
\newcommand{\Dgent}[2]{D\ind{t}\prn*{#1\dmid{}#2}}
\newcommand{\Dhels}[2]{D^{2}_{\mathsf{H}}\prn*{#1,#2}}
\newcommand{\Ddel}[2]{D_{\Delta}\prn*{#1,#2}}
\newcommand{\Dhelshort}{D^{2}_{\mathsf{H}}}
\newcommand{\Dchis}[2]{D_{\chi^2}\prn*{#1\dmid{}#2}}
\newcommand{\DchisX}[3]{D_{\chi^2}\prn[#1]{#2\dmid{}#3}}
\newcommand{\Dtris}[2]{D^{2}_{\Delta}\prn*{#1,#2}}
\newcommand{\Dtri}[2]{D_{\Delta}\prn*{#1,#2}}
\newcommand{\Dtv}[2]{D_{\mathsf{TV}}\prn*{#1,#2}}
\newcommand{\Dtvs}[2]{D^2_{\mathsf{TV}}\prn*{#1,#2}}
\newcommand{\dkl}[2]{d_{\mathsf{KL}}\prn*{#1\,\|\,#2}}
\newcommand{\Dphi}[2]{D_{\phi}(#1\;\|\;#2)}
\newcommand{\dphi}[2]{d_{\phi}(#1\;\|\;#2)}
\newcommand{\klr}[2]{D_{\mathsf{re}}(#1\;\|\;#2)}
\newcommand{\tv}[2]{D_{\mathsf{tv}}(#1\;\|\;#2)}
\newcommand{\DhelsX}[3]{D^{2}_{\mathsf{H}}\prn[#1]{#2,#3}}
\newcommand{\Ber}{\mathrm{Ber}}
\newcommand{\Rad}{\mathrm{Rad}}
\newcommand{\dmid}{\;\|\;}
\newcommand{\conv}{\mathrm{co}}
\newcommand{\diam}{\mathrm{diam}}
\newcommand{\bls}{\mb{\ls}}
\newcommand{\brak}{{\tiny[]}}
\newcommand{\Deltagap}{\Delta_{\mathrm{policy}}}
\newcommand{\Picsc}{\Pi^{\mathrm{csc}}}
\newcommand{\vcdim}{\mathrm{vcdim}}
\newcommand{\adacb}{\textsf{AdaCB}\xspace}
\newcommand{\squarecb}{\textsf{SquareCB}\xspace}
\newcommand{\falcon}{\textsf{FALCON}\xspace}
\newcommand{\Vf}{V}
\newcommand{\Qf}{Q}
\newcommand{\Vpi}{\Vf^{\pi}}
\newcommand{\Qpi}{\Qf^{\pi}}
\newcommand{\valueiteration}{\textsf{ValueIteration}\xspace}
\newcommand{\labelme}[1]{\tag{\theequation}\label{#1}}
\newcommand{\Ssafe}{\cS_{\mathrm{safe}}}
\newcommand{\Esafe}{\cE_{\mathrm{safe}}}
\newcommand{\Tsafe}{T_{\mathrm{safe}}}
\newcommand{\starhull}{\mathrm{star}}
\newcommand{\pescape}{P_{\mathrm{escape}}}
\newcommand{\vepsucb}{\veps_{\mathrm{ucb}}}
\newcommand{\vepsstati}{\veps_{\mathrm{stat,I}}}
\newcommand{\vepsstatii}{\veps_{\mathrm{stat,II}}}
\newcommand{\nstat}{n_{\mathrm{stat}}}
\newcommand{\Phat}{\wh{P}}
\newcommand{\Ffilt}{\mathfrak{F}}
\newcommand{\Eescape}{\cE_{\textrm{escape}}}
\newcommand{\vepshat}{\wh{\veps}}
\newcommand{\vepsbar}{\widebar{\veps}}
\newcommand{\igw}{\textsf{IGW}}
\newcommand{\ssep}{;\,}
\newcommand{\sh}[1][h]{s_{#1}}
\newcommand{\qh}{q_h}
\newcommand{\Qh}{Q_h}
\newcommand{\im}{\ind{m}}
\newcommand{\sep}{\ssep}
\newcommand{\Creg}{C_{\mathrm{reg}}}
\newcommand{\creg}{c_{\mathrm{reg}}}
\newcommand{\Cucb}{C_{\mathrm{ucb}}}
\newcommand{\gammam}{\gamma\im}
\newcommand{\Eopt}{\cE_{\mathrm{opt}}}
\newcommand{\emi}{\psi}
\newcommand{\dirsum}{\oplus}
\newcommand{\piunif}{\pi_{\mathrm{unif}}}
\newcommand{\cPhat}{\wh{\cP}}
\newcommand{\supp}{\mathrm{supp}}
\newcommand{\cPbar}{\wb{\cP}}
\newcommand{\Pstar}{P^{\star}}
\newcommand{\Plbar}{\underbar{P}}
\newcommand{\Ptil}{\wt{\cP}}
\newcommand{\astar}{a^{\star}}
\newcommand{\cFhat}{\wh{\cF}}
\newcommand{\cFbar}{\widebar{\cF}}
\newcommand{\Vtil}{\wt{\Vf}}
\newcommand{\Qtil}{\wt{\Qf}}
\newcommand{\ftil}{\tilde{f}}
\newcommand{\mathand}{\quad\text{and}\quad}
\newcommand{\mathwhere}{\quad\text{where}\quad}
\def\multiset#1#2{\ensuremath{\left(\kern-.3em\left(\genfrac{}{}{0pt}{}{#1}{#2}\right)\kern-.3em\right)}}
\newcommand{\grad}{\nabla}
\newcommand{\diag}{\mathrm{diag}}
\newcommand{\op}{\mathrm{op}}
\newcommand{\iid}{i.i.d.\xspace}
% \renewcommand{\ls}{\ell}
\newcommand{\glcomp}{c}
\newcommand{\InfB}{\mathcal{I}_{\mathrm{B}}}
\newcommand{\InfF}{\mathcal{I}_{\mathrm{F}}}
\renewcommand{\emptyset}{\varnothing}
\newcommand{\what}{\wh{w}}
\newcommand{\pbar}{\bar{p}}
\newcommand{\phat}{\wh{p}}
\newcommand{\Meps}{M_{\veps}}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\amin}{\mathrm{argmin}}
\DeclareMathOperator*{\amax}{\mathrm{argmax}}
% \DeclareMathOperator*{\dom}{dom}
\DeclareMathOperator*{\E}{\mathbb{E}}
\newcommand{\mb}{\mathbb}
\newcommand{\mc}{\mathcal}
\newcommand{\red}[1]{\textcolor{red}{#1}}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
% \DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
%\newcommand{\add}[1]{\textcolor{cyan}{#1}}
\newcommand{\add}[1]{\textcolor{cyan}{}}
\newcommand{\lindb}{\texttt{linDB}}
\newcommand{\linb}{\texttt{linB}}
\newcommand{\bign}[1]{\big(#1\big)}
\newcommand{\biggn}[1]{\bigg(#1\bigg)}
\newcommand{\Bign}[1]{\Big(#1\Big)}
\newcommand{\Biggn}[1]{\Bigg(#1\Bigg)}
\newcommand{\bigsn}[1]{\big[#1\big]}
\newcommand{\biggsn}[1]{\bigg[#1\bigg]}
\newcommand{\Bigsn}[1]{\Big[#1\Big]}
\newcommand{\Biggsn}[1]{\Bigg[#1\Bigg]}
\newcommand{\nangle}[1]{\langle #1\rangle}
\renewcommand{\l}{{\boldsymbol l}}
\renewcommand{\a}{{\mathbf a}}
\renewcommand{\b}{{\mathbf b}}
\newcommand{\e}{{\mathbf e}}
\newcommand{\f}{{\mathbf f}}
\newcommand{\g}{{\mathbf g}}
\newcommand{\h}{{\mathbf h}}
\newcommand{\p}{{\mathbf p}}
\newcommand{\q}{{\mathbf q}}
\newcommand{\qq}{{\mathbf{qq}}}
\renewcommand{\r}{{\mathbf r}}
\newcommand{\s}{{\mathbf s}}
\renewcommand{\t}{{\mathbf t}}
\renewcommand{\u}{{\mathbf u}}
\newcommand{\tu}{\tilde{\u}}
\renewcommand{\v}{{\mathbf v}}
\newcommand{\w}{{\mathbf w}}
\newcommand{\x}{{\mathbf x}}
\newcommand{\y}{{\mathbf y}}
\newcommand{\z}{{\mathbf z}}
\newcommand{\R}{{\mathbb R}}
\newcommand{\cD}{{\mathcal D}}
%\newcommand{\cA}{{\mathcal A}}
\newcommand{\mf}[1]{{\color{magenta} MF: #1}}
\newcommand{\avi}[1]{{\color{blue} Avi: #1}}
\newcommand{\zhihan}[1]{{\color{red} Zhihan: #1}}
\usepackage[round, sort&compress]{natbib}
\bibliographystyle{plainnat}
\begin{thebibliography}{49}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Agarwal et~al.(2019)Agarwal, Jiang, and Kakade]{agarwal2019reinforcement}
Alekh Agarwal, Nan Jiang, and Sham~M Kakade.
\newblock Reinforcement learning: Theory and algorithms.
\newblock \emph{Preprint}, 2019.
\bibitem[Amortila et~al.(2024)Amortila, Foster, Jiang, Sekhari, and Xie]{amortila2024harnessing}
Philip Amortila, Dylan~J Foster, Nan Jiang, Ayush Sekhari, and Tengyang Xie.
\newblock Harnessing density ratios for online reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2401.09681}, 2024.
\bibitem[Azar et~al.(2024)Azar, Guo, Piot, Munos, Rowland, Valko, and Calandriello]{azar2024general}
Mohammad~Gheshlaghi Azar, Zhaohan~Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello.
\newblock A general theoretical paradigm to understand learning from human preferences.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, pages 4447--4455. PMLR, 2024.
\bibitem[Bai et~al.(2022)Bai, Jones, Ndousse, Askell, Chen, DasSarma, Drain, Fort, Ganguli, Henighan, et~al.]{bai2022training}
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et~al.
\newblock Training a helpful and harmless assistant with reinforcement learning from human feedback.
\newblock \emph{arXiv preprint arXiv:2204.05862}, 2022.
\bibitem[Bengs et~al.(2022)Bengs, Saha, and H{\"u}llermeier]{bengs2022stochastic}
Viktor Bengs, Aadirupa Saha, and Eyke H{\"u}llermeier.
\newblock Stochastic contextual dueling bandits under linear stochastic transitivity models.
\newblock In \emph{International Conference on Machine Learning}, pages 1764--1786. PMLR, 2022.
\bibitem[Bradley and Terry(1952)]{bradley1952rank}
Ralph~Allan Bradley and Milton~E Terry.
\newblock Rank analysis of incomplete block designs: I. the method of paired comparisons.
\newblock \emph{Biometrika}, 39\penalty0 (3/4):\penalty0 324--345, 1952.
\bibitem[Cen et~al.(2024)Cen, Mei, Goshvadi, Dai, Yang, Yang, Schuurmans, Chi, and Dai]{cen2024value}
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo~Dai.
\newblock Value-incentivized preference optimization: A unified approach to online and offline rlhf.
\newblock \emph{arXiv preprint arXiv:2405.19320}, 2024.
\bibitem[Chang et~al.(2024)Chang, Shan, Oertell, Brantley, Misra, Lee, and Sun]{chang2024dataset}
Jonathan~D Chang, Wenhao Shan, Owen Oertell, Kiant{\'e} Brantley, Dipendra Misra, Jason~D Lee, and Wen Sun.
\newblock Dataset reset policy optimization for rlhf.
\newblock \emph{arXiv preprint arXiv:2404.08495}, 2024.
\bibitem[Chen et~al.(2022)Chen, Zhong, Yang, Wang, and Wang]{chen2022human}
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang.
\newblock Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation.
\newblock In \emph{International Conference on Machine Learning}, pages 3773--3793. PMLR, 2022.
\bibitem[Christiano et~al.(2017)Christiano, Leike, Brown, Martic, Legg, and Amodei]{christiano2017deep}
Paul~F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.
\newblock Deep reinforcement learning from human preferences.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Du et~al.(2024)Du, Winnicki, Dalal, Mannor, and Srikant]{du2024exploration}
Yihan Du, Anna Winnicki, Gal Dalal, Shie Mannor, and R~Srikant.
\newblock Exploration-driven policy optimization in {RLHF}: Theoretical insights on efficient data utilization.
\newblock \emph{arXiv preprint arXiv:2402.10342}, 2024.
\bibitem[Ethayarajh et~al.(2024)Ethayarajh, Xu, Muennighoff, Jurafsky, and Kiela]{ethayarajh2024kto}
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.
\newblock {KTO}: Model alignment as prospect theoretic optimization.
\newblock \emph{arXiv preprint arXiv:2402.01306}, 2024.
\bibitem[Foster and Rakhlin(2023)]{foster2023foundations}
Dylan~J Foster and Alexander Rakhlin.
\newblock Foundations of reinforcement learning and interactive decision making.
\newblock \emph{arXiv preprint arXiv:2312.16730}, 2023.
\bibitem[Gao et~al.(2024)Gao, Chang, Zhan, Oertell, Swamy, Brantley, Joachims, Bagnell, Lee, and Sun]{gao2024rebel}
Zhaolin Gao, Jonathan~D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kiant{\'e} Brantley, Thorsten Joachims, J~Andrew Bagnell, Jason~D Lee, and Wen Sun.
\newblock {REBEL}: Reinforcement learning via regressing relative rewards.
\newblock \emph{arXiv preprint arXiv:2404.16767}, 2024.
\bibitem[Guo et~al.(2024)Guo, Zhang, Liu, Liu, Khalman, Llinares, Rame, Mesnard, Zhao, Piot, et~al.]{guo2024direct}
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et~al.
\newblock Direct language model alignment from online {AI} feedback.
\newblock \emph{arXiv preprint arXiv:2402.04792}, 2024.
\bibitem[Jin et~al.(2020)Jin, Yang, Wang, and Jordan]{jin2020provably}
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael~I Jordan.
\newblock Provably efficient reinforcement learning with linear function approximation.
\newblock In \emph{Conference on Learning Theory}, pages 2137--2143, 2020.
\bibitem[Kalashnikov et~al.(2018)Kalashnikov, Irpan, Pastor, Ibarz, Herzog, Jang, Quillen, Holly, Kalakrishnan, Vanhoucke, et~al.]{kalashnikov2018scalable}
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et~al.
\newblock Scalable deep reinforcement learning for vision-based robotic manipulation.
\newblock In \emph{Conference on robot learning}, pages 651--673. PMLR, 2018.
\bibitem[Lattimore and Szepesv{\'a}ri(2020)]{lattimore2020bandit}
Tor Lattimore and Csaba Szepesv{\'a}ri.
\newblock \emph{Bandit algorithms}.
\newblock Cambridge University Press, 2020.
\bibitem[Li et~al.(2022)Li, Ma, and Srebro]{li2022pessimism}
Gene Li, Cong Ma, and Nati Srebro.
\newblock Pessimism for offline linear contextual bandits using $\ell_p $ confidence sets.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 20974--20987, 2022.
\bibitem[Liu et~al.(2023)Liu, Zhao, Joshi, Khalman, Saleh, Liu, and Liu]{liu2023statistical}
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter~J Liu, and Jialu Liu.
\newblock Statistical rejection sampling improves preference optimization.
\newblock \emph{arXiv preprint arXiv:2309.06657}, 2023.
\bibitem[Meng et~al.(2024)Meng, Xia, and Chen]{meng2024simpo}
Yu~Meng, Mengzhou Xia, and Danqi Chen.
\newblock Simpo: Simple preference optimization with a reference-free reward.
\newblock \emph{arXiv preprint arXiv:2405.14734}, 2024.
\bibitem[Munos et~al.(2023)Munos, Valko, Calandriello, Azar, Rowland, Guo, Tang, Geist, Mesnard, Michi, et~al.]{munos2023nash}
R{\'e}mi Munos, Michal Valko, Daniele Calandriello, Mohammad~Gheshlaghi Azar, Mark Rowland, Zhaohan~Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et~al.
\newblock Nash learning from human feedback.
\newblock \emph{arXiv preprint arXiv:2312.00886}, 2023.
\bibitem[Negahban et~al.(2012)Negahban, Oh, and Shah]{rc}
Sahand Negahban, Sewoong Oh, and Devavrat Shah.
\newblock Iterative ranking from pair-wise comparisons.
\newblock \emph{Advances in neural information processing systems}, 25, 2012.
\bibitem[Nika et~al.(2024)Nika, Mandal, Kamalaruban, Tzannetos, Radanovi{\'c}, and Singla]{nika2024reward}
Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovi{\'c}, and Adish Singla.
\newblock Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences.
\newblock \emph{arXiv preprint arXiv:2403.01857}, 2024.
\bibitem[Ouyang et~al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, et~al.]{ouyang2022training}
Long Ouyang, Jeffrey Wu, Xu~Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et~al.
\newblock Training language models to follow instructions with human feedback.
\newblock \emph{Advances in neural information processing systems}, 35:\penalty0 27730--27744, 2022.
\bibitem[Rafailov et~al.(2023)Rafailov, Sharma, Mitchell, Manning, Ermon, and Finn]{rafailov2024direct}
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher~D Manning, Stefano Ermon, and Chelsea Finn.
\newblock Direct preference optimization: Your language model is secretly a reward model.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2023.
\bibitem[Rafailov et~al.(2024)Rafailov, Hejna, Park, and Finn]{rafailov2024r}
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.
\newblock {From $r$ to $Q^{\star}$}: Your language model is secretly a {Q}-function.
\newblock \emph{arXiv preprint arXiv:2404.12358}, 2024.
\bibitem[Rosset et~al.(2024)Rosset, Cheng, Mitra, Santacroce, Awadallah, and Xie]{rosset2024direct}
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie.
\newblock {Direct Nash Optimization}: Teaching language models to self-improve with general preferences.
\newblock \emph{arXiv preprint arXiv:2404.03715}, 2024.
\bibitem[Saha(2021)]{saha2021optimal}
Aadirupa Saha.
\newblock Optimal algorithms for stochastic contextual preference bandits.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 30050--30062, 2021.
\bibitem[Song et~al.(2022)Song, Zhou, Sekhari, Bagnell, Krishnamurthy, and Sun]{song2022hybrid}
Yuda Song, Yifei Zhou, Ayush Sekhari, J~Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun.
\newblock Hybrid rl: Using both offline and online data can make rl efficient.
\newblock \emph{arXiv preprint arXiv:2210.06718}, 2022.
\bibitem[Song et~al.(2024)Song, Swamy, Singh, Bagnell, and Sun]{song2024understanding}
Yuda Song, Gokul Swamy, Aarti Singh, J~Andrew Bagnell, and Wen Sun.
\newblock Understanding preference fine-tuning through the lens of coverage.
\newblock \emph{arXiv preprint arXiv:2406.01462}, 2024.
\bibitem[Tan et~al.(2024)Tan, Fan, and Wei]{tan2024hybrid}
Kevin Tan, Wei Fan, and Yuting Wei.
\newblock Hybrid reinforcement learning breaks sample size barriers in linear mdps.
\newblock \emph{arXiv preprint arXiv:2408.04526}, 2024.
\bibitem[Wagenmaker et~al.(2022)Wagenmaker, Chen, Simchowitz, Du, and Jamieson]{wagenmaker2022reward}
Andrew~J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson.
\newblock Reward-free rl is no harder than reward-aware rl in linear markov decision processes.
\newblock In \emph{International Conference on Machine Learning}, pages 22430--22456. PMLR, 2022.
\bibitem[Wang et~al.(2023)Wang, Liu, and Jin]{wang2023rlhf}
Yuanhao Wang, Qinghua Liu, and Chi Jin.
\newblock Is {RLHF} more difficult than standard {RL}?
\newblock \emph{arXiv preprint arXiv:2306.14111}, 2023.
\bibitem[Wu and Sun(2023)]{wu2023making}
Runzhe Wu and Wen Sun.
\newblock Making {RL} with preference-based feedback efficient via randomization.
\newblock \emph{arXiv preprint arXiv:2310.14554}, 2023.
\bibitem[Wu et~al.(2024)Wu, Zhu, Zhang, Wen, Ramchandran, and Jiao]{wu2024pairwise}
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao.
\newblock Pairwise proximal policy optimization: Large language models alignment via comparative rl.
\newblock 2024.
\bibitem[Xie et~al.(2021)Xie, Jiang, Wang, Xiong, and Bai]{xie2021policy}
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu~Bai.
\newblock Policy finetuning: Bridging sample-efficient offline and online reinforcement learning.
\newblock \emph{Advances in neural information processing systems}, 34:\penalty0 27395--27407, 2021.
\bibitem[Xie et~al.(2023)Xie, Foster, Bai, Jiang, and Kakade]{xie2023role}
Tengyang Xie, Dylan~J Foster, Yu~Bai, Nan Jiang, and Sham~M Kakade.
\newblock The role of coverage in online reinforcement learning.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\bibitem[Xie et~al.(2024)Xie, Foster, Krishnamurthy, Rosset, Awadallah, and Rakhlin]{xie2024exploratory}
Tengyang Xie, Dylan~J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin.
\newblock Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf.
\newblock \emph{arXiv preprint arXiv:2405.21046}, 2024.
\bibitem[Xiong et~al.(2024)Xiong, Dong, Ye, Wang, Zhong, Ji, Jiang, and Zhang]{xiong2024iterative}
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang.
\newblock Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.
\newblock In \emph{Forty-first International Conference on Machine Learning}, 2024.
\bibitem[Xu et~al.(2023)Xu, Lee, Sukhbaatar, and Weston]{xu2023some}
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston.
\newblock Some things are more cringe than others: Preference optimization with the pairwise cringe loss.
\newblock \emph{arXiv preprint arXiv:2312.16682}, 2023.
\bibitem[Ye et~al.(2024)Ye, Xiong, Zhang, Jiang, and Zhang]{ye2024theoretical}
Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang.
\newblock A theoretical analysis of {Nash} learning from human feedback under general {KL}-regularized preference.
\newblock \emph{arXiv preprint arXiv:2402.07314}, 2024.
\bibitem[Zhan et~al.(2022)Zhan, Huang, Huang, Jiang, and Lee]{zhan2022offline}
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee.
\newblock Offline reinforcement learning with realizability and single-policy concentrability.
\newblock In \emph{Conference on Learning Theory}, pages 2730--2775. PMLR, 2022.
\bibitem[Zhan et~al.(2023)Zhan, Uehara, Kallus, Lee, and Sun]{zhan2023provable}
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason~D Lee, and Wen Sun.
\newblock Provable offline preference-based reinforcement learning.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2023.
\bibitem[Zhang et~al.(2024)Zhang, Yu, Sharma, Yang, Wang, Hassan, and Wang]{zhang2024self}
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang.
\newblock Self-exploring language models: Active preference elicitation for online alignment, 2024.
\bibitem[Zhao et~al.(2023)Zhao, Joshi, Liu, Khalman, Saleh, and Liu]{zhao2023slic}
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter~J Liu.
\newblock Slic-hf: Sequence likelihood calibration with human feedback.
\newblock \emph{arXiv preprint arXiv:2305.10425}, 2023.
\bibitem[Zhong et~al.(2022)Zhong, Xiong, Zheng, Wang, Wang, Yang, and Zhang]{zhong2022gec}
Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang.
\newblock {GEC}: A unified framework for interactive decision making in {MDP}, {POMDP}, and beyond.
\newblock \emph{arXiv preprint arXiv:2211.01962}, 2022.
\bibitem[Zhu et~al.(2023)Zhu, Jordan, and Jiao]{zhu2023principled}
Banghua Zhu, Michael Jordan, and Jiantao Jiao.
\newblock Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.
\newblock In \emph{International Conference on Machine Learning}, pages 43037--43067. PMLR, 2023.
\bibitem[Zhu et~al.(2024)Zhu, Jordan, and Jiao]{zhu2024iterative}
Banghua Zhu, Michael~I Jordan, and Jiantao Jiao.
\newblock Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf.
\newblock \emph{arXiv preprint arXiv:2401.16335}, 2024.
\end{thebibliography}