Information-Theoretic Policy Pre-Training with Empowerment
Extracted source text · arXiv 2510.05996
\documentclass{article}
\makeatletter
\def\blindfootnote{\xdef\@thefnmark{}\@footnotetext}
\makeatother
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{booktabs} % for professional tables
\usepackage[numbers]{natbib}
\usepackage{svg}
\usepackage{wrapfig}
% hyperref makes hyperlinks in the resulting PDF.
\usepackage{hyperref}
% ready for submission
\usepackage[preprint]{neurips_2025}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{xfrac}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
\usepackage[textsize=tiny]{todonotes}
%%%%%%%%%%%%%%%%%%%%%%%%%%% Packages include myself
% instead of subfigure
\usepackage{subcaption}
\usepackage{enumitem}
% Better tables
\usepackage{tabularx}
% Snowflake icon
\usepackage{pifont}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newtheorem*{remark}{Remark}
\title{Information-Theoretic Policy Pre-Training with Empowerment}
\author{
Moritz Schneider\,\textsuperscript{\normalfont 1 2 3 *}
\enskip
Robert Krug\,\textsuperscript{\normalfont 1 2}
\enskip
Narunas Vaskevicius\,\textsuperscript{\normalfont 1 2}
\\
\textbf{Michael Volpp}\,\textsuperscript{\normalfont 1 2}
\enskip
\textbf{Luigi Palmieri}\,\textsuperscript{\normalfont 1 2}
\enskip
\textbf{Joschka Boedecker}\,\textsuperscript{\normalfont 3 4}\\
\textsuperscript{\normalfont 1} Bosch Center for Artificial Intelligence \enskip
\textsuperscript{\normalfont 2} Bosch Corporate Research \\
\textsuperscript{\normalfont 3} University of Freiburg \enskip
\textsuperscript{\normalfont 4} BrainLinks-BrainTools
}
\begin{document}
\maketitle
\begin{abstract}
\input{sections/00_abstract}
\blindfootnote{
\parbox{0.7\textwidth}{
\textsuperscript{*}Correspondence to: \texttt{moritz.schneider@de.bosch.com}
}
}
\end{abstract}
\input{sections/01_introduction}
\input{sections/02_related_work}
\input{sections/03_empowerment_init}
\input{sections/04_experiments}
\input{sections/05_discussion}
\input{sections/06_acknowledgments}
\newpage
\clearpage
\bibliography{references}
\bibliographystyle{unsrtnat}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\clearpage
\appendix
%\onecolumn
%\raggedbottom
\input{sections/99_appendix}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
Empowerment, an information-theoretic measure of an agent's potential influence on its environment, has emerged as a powerful intrinsic motivation and exploration framework for reinforcement learning (RL).
Besides for unsupervised RL and skill learning algorithms, the specific use of empowerment as a pre-training signal has received limited attention in the literature.
We show that empowerment can be used as a pre-training signal for data-efficient downstream task adaptation.
For this we extend the traditional notion of empowerment by introducing discounted empowerment, which balances the agent's control over the environment across short- and long-term horizons.
Leveraging this formulation, we propose a novel pre-training paradigm that initializes policies to maximize discounted empowerment, enabling agents to acquire a robust understanding of environmental dynamics.
We analyze empowerment-based pre-training for various existing RL algorithms and empirically demonstrate its potential as a general-purpose initialization strategy: empowerment-maximizing policies with long horizons are data-efficient and effective, leading to improved adaptability in downstream tasks.
Our findings pave the way for future research to scale this framework to high-dimensional and complex tasks, further advancing the field of RL.
\section{Introduction} \label{sec:introduction}
Pre-training has emerged as a critical strategy in reinforcement learning (RL) to improve sample efficiency, stability, and generalization, particularly in environments where reward signals are sparse or delayed \cite{levine_offline_2020,xie_pretraining_2022}. By enabling agents to acquire useful priors about the structure of the environment or about transferable behavioral patterns, pre-training can significantly increase data efficiency during downstream task adaptation \cite{xie_pretraining_2022,springenberg_offline_2024}. Empowerment \cite{klyubin_all_2005,klyubin_empowerment_2005,salge_empowermentintroduction_2014}, an information-theoretic measure of an agent's potential influence over future state distributions, offers a principled approach to unsupervised pre-training \cite{mohamed_variational_2015,karl_unsupervised_2019,leibfried_unified_2019,zhao_efficient_2020}. Unlike tasks with externally given rewards, empowerment does not rely on external rewards but instead encourages the agent to explore and occupy states from which it retains a high degree of control in the environment. This intrinsic motivation enables the development of broadly applicable behaviors that can be rapidly adapted to a variety of downstream objectives.
Unsupervised pre-training has become a cornerstone in the development of foundation models in both computer vision \cite{radford_learning_2021,girdhar_imagebind_2023,oquab_dinov2_2024} and natural language processing \cite{brown_language_2020,gemini_team_gemini_2024,gemini_team_gemini_2024-1}, where large-scale models are trained on vast amounts of unlabeled data to learn general-purpose representations. Despite these advances, a comparable paradigm has not yet been established within RL. One promising domain for RL pre-training is video data \cite{yang_position_2024,mccarthy_towards_2024}, which naturally embodies sequential dynamics and agent-environment interactions. Utilizing such data would be additionally useful due to the difficulty of collecting real-world data for RL agents as video data is broadly available. However, the absence of explicit reward signals in such data precludes the direct application of traditional RL techniques. This limitation necessitates the use of unsupervised RL approaches, where intrinsic objectives—such as curiosity~\cite{pathak_curiosity-driven_2017}, information gain~\cite{sekar_planning_2020,rhinehart_information_2021} or empowerment~\cite{mohamed_variational_2015,karl_unsupervised_2019}—serve as substitutes for task-specific rewards. These unsupervised signals enable agents to learn useful representations and behaviors that can generalize across tasks, potentially paving the way for foundation models in RL analogous to those in vision and language. We show that empowerment can be effectively employed as a pre-training signal, enabling agents to learn useful representations and behaviors that can be rapidly adapted to various downstream tasks.
A key distinction between our work and prior research on empowerment lies in the role of empowerment-based objectives. Previous studies have predominantly employed empowerment and mutual information maximization as intrinsic reward signals aimed at enhancing exploration during RL \cite{houthooft_vime_2016,leibfried_unified_2019,becker-ehmck_exploration_2021,zhang_metacure_2021} or complete unsupervised control \cite{mohamed_variational_2015,karl_unsupervised_2019,zhao_efficient_2020}, in contrast to targeting improved initialization of policies before actual downstream task learning. Its application as a standalone pre-training signal remains largely underexplored. In other cases, empowerment has been leveraged to facilitate the unsupervised acquisition of diverse skills, which are typically pre-trained and subsequently utilized in downstream tasks \cite{sharma_dynamics-aware_2020,laskin_contrastive_nodate,eysenbach_diversity_2019}. However, \citet{eysenbach_information_2022} demonstrated that, contrary to conventional assumptions, maximizing mutual information may not necessarily result in the development of distinct skills, but rather serves to generate a beneficial initialization for skill acquisition. We argue that such an initialization would be beneficial for a general policy itself which is not dependent on a skill identifier.
In short, we address these gaps by the following key contributions:
\begin{itemize}
\item We propose a generic framework for unsupervised pre-training in RL based on empowerment, emphasizing its utility for initializing policies in downstream tasks.
\item We introduce discounted empowerment as a flexible objective that balances short- and long-term control, enabling a simple pre-training strategy without the need to tune a specific horizon for multi-step empowerment.
\item We demonstrate through experiments that empowerment-based pre-training with our discounted empowerment reward accelerates the adaption process and improves learning efficiency on a variety of downstream RL tasks.
\end{itemize}
\section{Related Work} \label{sec:related_work}
\textbf{Unsupervised Control.}\label{para:empowerment}
Recent research in intrinsically motivated RL has focused on leveraging information-theoretic principles to drive agent behavior without extrinsic rewards. \citet{klyubin_all_2005,klyubin_empowerment_2005} introduced empowerment as a universal, agent-centric measure of control, quantifying an agent's potential to influence its future states via channel capacity between actions and sensor states. This concept was further operationalized in subsequent works \cite{klyubin_keep_2008,salge_approximation_2013,salge_changing_2014,salge_empowerment_2017}, where empowerment served as a general-purpose utility function for sensorimotor systems, promoting behaviors that maximize an agent's action-perception possibilities. Building on this foundation, \citet{karl_unsupervised_2019} and \citet{mohamed_variational_2015} proposed variational approaches towards empowerment maximization. Both methods enable real-time control via a tractable, unsupervised variational approximation of empowerment. \citet{houthooft_vime_2016}, \citet{leibfried_unified_2019} and \citet{navneet_madhu_kumar_empowerment-driven_2018} introduced empowerment-based exploration strategies, which leverage empowerment to guide exploration in RL tasks additionally to the extrinsically given reward signal. While empowerment can clearly excel as an exploration signal, such a use case does not lead to an intrinsically competent agent as the ultimate goal is still to maximize the extrinsic reward.
\citet{liu_behavior_2021} introduced APT, an entropy maximization method leveraging contrastive representation learning for efficient pre-training and adaptation in visual RL tasks.
\citet{mendonca_discovering_2021} presents LEXA, a model-based approach that trains separate explorer and achiever policies on imagined trajectories using a learned world model, exploring diverse states and enabling goal-conditioned behavior via learned latent distances.
\citet{yarats_reinforcement_2021} introduced Proto-RL, which utilizes prototypical representations and particle-based entropy maximization for effective exploration and representation learning in image-based RL.
In another direction, \citet{myers_learning_2024} proposed assistance policies that, instead of maximizing the empowerment of the agent itself, learn to preserve human autonomy through human empowerment maximization. Even though, all those works focus on unsupervised and empowerment-based control, none of these actually use empowerment-based policies to initialize a policy for downstream tasks. Instead, they focus on completely independent unsupervised task-solving or exploration.
Similarly, \textbf{Unsupervised Skill Discovery}\label{para:mi_skill} algorithms focus on learning a repertoire of diverse behaviors without explicit external rewards by including learned skill identifiers. Methods like VALOR \cite{achiam_variational_2018}, VIC \cite{gregor_variational_2017}, DIAYN \cite{eysenbach_diversity_2019} and DADS \cite{sharma_dynamics-aware_2020} leverage variational lower bounds on mutual information to learn distinguishable skills.
Contrastive Intrinsic Control (CIC) \citep{laskin_contrastive_nodate} employs contrastive learning between state transitions and skills to learn behavior embeddings, using their entropy as an intrinsic reward.
\citet{liu_aps_2021} (who builds upon APT) and \citet{hansen_fast_2020} incorporate successor features while \citet{park_controllability-aware_2023} moves away from mutual information and instead optimizes a certain Wasserstein distance to learn skills.
In a similar approach, \citet{zheng_can_2025} connects successor features, contrastive learning and mutual information maximization.
\citet{eysenbach_information_2022} examined the relationship between skill learning and the geometry of state marginal distributions. The paper reveals that skill learning with mutual information maximization leads to favorable initializations of skill policies and the learned skills may not cover all possible behaviors.
Choreographer \cite{mazzaglia_choreographer_2022}, decouples exploration from skill discovery and uses methods like VQ-VAE\cite{van_den_oord_neural_2017} to learn a codebook of skills based on world model states.
In contrast to those skill discovery approaches, our work does not focus on learning a set of individual skill policies, but rather on learning a single policy that is capable of quickly adapting to any downstream task. Our goal is to learn a good initialization for a single policy that can be fine-tuned fast on any downstream task.
\textbf{Meta-Reinforcement Learning}\label{para:metarl} is a subfield of RL that focuses on training agents to learn new tasks quickly by leveraging prior experience. The goal is to enable agents to adapt to new environments or tasks with minimal data and time. \citet{finn_model-agnostic_2017} proposed Model-Agnostic Meta-Learning (MAML), which trains a model on a variety of tasks such that it can quickly adapt to new tasks with only a few gradient updates. It does so without introducing additional learnable parameters and purely with gradient descent.
\citet{wang_learning_2016} and \citet{duan_rl2_2016} are model-agnostic as both methods rely on recurrent neural networks to learn a policy that can adapt to new tasks quickly.
\citet{zintgraf_varibad_2020} introduces variBAD, a method to approximate Bayes-optimal behavior, which uses meta-learning to utilize knowledge obtained in related tasks and perform approximate inference in unknown environments.
The only work known to us which combines empowerment with meta learning is \citet{zhang_metacure_2021} who combine meta RL with empowerment-based exploration.
All of these methods can be interpreted as initializing a policy for efficient adaptation to unseen tasks. In other words, all learn an initialization of the policy that is closer to the optimal policy for the new task than a random initialization. Our work is related in the sense that we utilize empowerment to learn similar initializations. However, in contrast to meta-RL, an empowerment-based initialization is not based on extrinsically-defined pre-training tasks and it does not require a distribution of tasks for pre-training. Instead, it yields a generic initialization that can be used for any downstream task in a given MDP.
\section{Policy Pre-Training with Empowerment} \label{sec:optimal_init}
In this section, we introduce our approach for empowerment-based pre-training of RL policies. The goal is to initialize the policy in a way that enables quick adaptation to every downstream task at hand.
\subsection{Reinforcement Learning} \label{sec:rl_intro}
Reinforcement learning (RL) is a framework for decision-making in which an agent learns optimal behavior by interacting with an environment and receiving feedback in the form of rewards. The environment is typically modeled as a Markov Decision Process (MDP) defined by the tuple~$(\mathcal{S}, \mathcal{A}, \mathcal{P}, \rho_0, R, \gamma)$, with the set of states~$\mathcal{S}$ and the set of actions~$\mathcal{A}$. $\mathcal{P}(\mathbf{s'} \mid \mathbf{s}, \mathbf{a})$ denotes the transition probability density, which defines the probability of transitioning to state~$\mathbf{s}' \in \mathcal{S}$ when choosing action~$\mathbf{a} \in \mathcal{A}$ in state~$\mathbf{s} \in \mathcal{S}$ where $\mathcal{P}: \mathcal{S} \times \mathcal{A} \rightarrow \Delta (\mathcal{S})$. Here, $\Delta (\mathcal{S})$ is the space of probability distributions over $\mathcal{S}$. $R(\mathbf{s})$ is a scalar function which specifies the immediate reward received when reaching state~$\mathbf{s}$. The goal of an agent is to find a policy~$\pi(\mathbf{a} \mid \mathbf{s})$, defining a conditional probability distribution over actions given a state, which maximizes the expected future cumulative discounted reward for a given state $\mathbf{s}$:
\begin{equation}
J(\pi) = \mathbb{E} \left [ \sum_{t=0}^\infty \gamma^t R(\mathbf{s}_{t+1}) \right ].
\end{equation}
Here, $\mathbf{s}_t \in \mathcal{S}$ denotes the state at time~$t$ and the expectation is w.r.t.~the state distribution induced by the policy $\pi$ and the transition probability density $\mathcal P$, starting from an initial state distribution $\rho_0(\mathbf{s}_0)$. The discount factor~$\gamma \in [0, 1)$ weights future rewards relative to immediate ones.
\subsection{Empowerment} \label{sec:empowerment}
We start by defining the information-theoretic concept of channel capacity. Intuitively, channel capacity quantifies the maximum number of bits of information that can be reliably transmitted over a communication channel \citep{cover_elements_2005}. Mathematically, a channel with input $X$ and output $Y$ is characterized by the conditional probability density $p(y \mid x)$, and we define its channel capacity~$\mathcal{C}$ as the maximum mutual information between output and input,
\begin{equation}
\mathcal{C} \doteq \max_{p(x)} \mathcal{I}(X; Y) = \max_{p(x)} \left ( \mathcal{H}(X) - \mathcal{H}(X \mid Y) \right ),
\end{equation}
where the maximum is taken over all possible input distributions $p(x)$ and $\mathcal{H}(X)$ denotes the entropy of $X$.
The empowerment formalism applies this idea to sensorimotor systems. In the context of our RL setup, the channel is represented by the transition dynamics~$\mathcal{P}(\mathbf{s}' \mid \mathbf{s}, \mathbf{a})$, considering the action~$\mathbf{a}$ as the input and the next state~$\mathbf{s'}$ as the output. The contextual empowerment~$\mathcal{E}$ for a state~$\mathbf{s}$ is then defined as the channel capacity between next states and actions given the current state~$\mathbf{s}$ as context \cite{klyubin_keep_2008,salge_empowermentintroduction_2014}:
\begin{equation}
\label{eq:empowerment}
\mathcal{E}(\mathbf{s}) \doteq \max_{p(\mathbf{a})} \mathcal{I}(\mathbf{s}'; \mathbf{a} \mid \mathbf{s}) \doteq \max_{p(\mathbf{a})} \mathbb{E}_{p(\mathbf{a}) \mathcal{P}(\mathbf{s}' | \mathbf{s}, \mathbf{a})} \left[\log{\frac{\mathcal{P}(\mathbf{s}'\mid \mathbf{s}, \mathbf{a})}{\mathcal{P}(\mathbf{s}' \mid \mathbf{s})}}\right].
\end{equation}
Since empowerment is computed over all possible policies~$p(\mathbf{a})$, it becomes a property of the state~$\mathbf{s}$ alone, independent of any specific agent behavior or policy. Consequently, empowerment characterizes the environment's intrinsic structure, capturing the degree of influence an agent can potentially exert from a given state, irrespective of its current policy.
Empowerment can be interpreted as a proxy for an agent's preparedness \cite{salge_empowermentintroduction_2014}, reflecting its ability to influence future states in the environment. From this perspective, a highly empowered agent is one that possesses a broad repertoire of future options, enabling it to navigate toward states that are potentially advantageous or preferable, even in the absence of explicit external rewards. This capability implies a form of competence, whereby the agent acquires an implicit understanding of the environmental structure. By prioritizing states that maximize the agent's future influence, the agent demonstrates a form of intrinsic preparedness.
We assume that an agent has maximal competence if it has figured out a policy that achieves or maximizes the empowerment available in its environment. The agent's empowerment can be increased by either (1) going to more empowered states or by (2) changing the agent or environment \cite{salge_empowermentintroduction_2014}. In the standard RL setup, it is not possible to change the sensors and available actions of the agent, nor it is possible to change the environment beyond the state-space of the MDP. Thus, the only way to increase empowerment is by going to more empowered states. Thus, it would be natural to pre-train agents to maximize empowerment in the environment (which is depicted on the left part of Figure~\ref{fig:method}).
\paragraph{$n$-step Empowerment.} The concept of $n$-step empowerment~$\mathcal{E}_n$ generalizes the traditional notion of empowerment by extending the temporal scope from immediate state-action transitions to sequences of transitions over a fixed horizon of $n$ steps. Specifically, rather than measuring the mutual information between a single action~$\mathbf{a}_t$ and the resulting next state~$\mathbf{s}_{t+1}$, $n$-step empowerment evaluates the mutual information between a sequence of actions~$\mathbf{a}_{t:t+n} \in \mathcal{A}^n$ and the resulting state~$\mathbf{s}_{t+n+1}$, conditioned on the initial state~$\mathbf{s}_t$:
\begin{equation}
\mathcal{E}_{n} (\mathbf{s}_t) \doteq \max_{p(\mathbf{a}_t, \dots, \mathbf{a}_{t+n})} \mathcal{I}(\mathbf{s}_{t+n+1}; \mathbf{a}_{t:t+n}| \mathbf{s}_t).
\end{equation}
This formulation enables the agent to assess its potential influence over the environment across a longer time horizon, thereby capturing more temporally extended notions of controllability. By incorporating multi-step dynamics, $n$-step empowerment provides a richer, more comprehensive measure of how much control an agent can exert over its future, beyond the scope of immediate transitions.
\subsection{Discounted Empowerment} \label{sec:discounted_empowerment}
A fundamental limitation of $n$-step empowerment with large horizon values arises from the agent's ability to access an increasingly uniform set of reachable states. As $n$ grows, the empowerment definition may include the possibility to reach virtually any state from any starting point, resulting in a uniformly flat empowerment landscape \cite{salge_empowermentintroduction_2014}. In such cases, empowerment degenerates into a trivial signal thereby losing its ability to differentiate states based on their available future options. Such uniform empowerment landscape for a deterministic gridworld can be seen in the fourth grid of Figure~\ref{fig:gridworld}.
To address this issue, we introduce \textit{discounted empowerment}: Instead of training the agent on a specific $n$-step empowerment horizon, we evaluate agents on discounted versions of multiple $n$-step horizons. Similar to $n$-step empowerment, it is a measure of the amount of control an agent has over its environment, taking into account different horizons with the discount factor~$\lambda \in (0, 1]$ and total length~$H$. It is defined as the discounted sum of channel capacities between the agent's actions and the future states of the environment, with the discount factor~$\lambda$ applied to empowerment values of larger horizons. The discounted empowerment can be expressed as:
\begin{equation}
\mathcal{E}_{\lambda} (\mathbf{s}_i) \doteq \sum_{k=0}^{H} \lambda^{k} \max_{p(\mathbf{a}_i, \dots, \mathbf{a}_{i+k})} \mathcal{I}(\mathbf{s}_{i+k+1}; \mathbf{a}_{i:i+k}| \mathbf{s}_i).
\end{equation}
Here, $H$ denotes the episode length. This formulation allows us to capture the short- and long-term effects of actions on the state distribution, which we assume are both crucial for effective pre-training. Consequently, the concept of discounted empowerment obviates the necessity of determining an appropriate empowerment horizon tailored to the specific characteristics of the environment.
It is important to note that while our discount factor $\lambda$ resembles the RL discount factor $\gamma$, the two serve distinct purposes and are not equivalent.
\subsection{Pre-Training with Empowerment}
\input{figures/method/figure}
To pre-train an agent, we first compute the empowerment values for each state within the environment, quantifying the agent's potential influence over future states without depending on a specific policy. These empowerment values are then employed as intrinsic rewards $R(\mathbf{s}_t) = \mathcal{E}_i(\mathbf{s}_t)$, enabling the agent to be pre-trained using a RL algorithm of choice with the objective of maximizing empowerment.
Here, $\mathcal{E}_i$ can be one-step empowerment~$\mathcal{E}$, $n$-step empowerment~$\mathcal{E}_n$ or discounted empowerment~$\mathcal{E}_\lambda$.
The resulting pre-trained policy~$\pi_\mathcal{E}$, optimized for empowerment maximization, is subsequently deployed directly to a downstream task without requiring additional modifications to the policy. Notably, the MDP of the downstream task differs solely in its extrinsic reward function, which is constructed using expert human knowledge to align with the specific objectives of the task. This scheme is illustrated in Figure~\ref{fig:method}.
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{\linewidth}
\includegraphics[width=\linewidth]{figures/method/method}
\end{subfigure}
%\vspace*{10pt}
\caption{\label{fig:method}
\textbf{Empowerment-based Pre-Training.} The untrained agent is pre-trained to optimize empowerment $\mathcal{E}$ in an environment-centric manner, which allows it to learn a policy that can be fine-tuned for specific tasks. The initialization achieved by this pre-training is expected to be closer to the optimal policy than a random initialization, as it has already learned to achieve options that are helpful for downstream tasks. The empowerment value is solely based on the environment characteristics without further human input whereas the extrinsic fine-tuning reward is based on expert human knowledge of the task.}
\vspace{-2ex}
%\vspace*{-10pt}
\end{figure*}
\section{Experiments} \label{sec:experiments}
\input{figures/gridworld/figure}
We perform experiments in a discrete gridworld nevigation environment with $5$ discrete actions for moving forward, backward, left, right and waiting. The agent aims to reach a certain goal state.
This environment provides a controlled and interpretable setting where empowerment can be computed efficiently.
From Equation~\ref{eq:empowerment} it is easy to see, that in the deterministic case, empowerment of a state $\mathbf{s}_t$ is determined by the logarithm of the number of states reachable within a specified horizon, offering a straightforward and tractable measure of an agent's influence over its environment.
For the stochastic case, we employ the Blahut-Arimoto algorithm \cite{blahut_computation_1972, arimoto_algorithm_1972}, a well-established method for calculating channel capacity in probabilistic systems. Stochasticity is introduced by allowing the agent to move to the left or to the right relative to its current direction with a predefined probability, thereby incorporating an element of uncertainty into the agent's actions.
In our experiments, we compare five distinct approaches: on-policy REINFORCE~\cite{williams_simple_1992} agents with and without a baseline, on-policy Actor-Critic (AC) agents incorporating a critic trained with temporal-difference (TD) learning~\cite{sutton_reinforcement_2018}, on-policy Proximal Policy Optimization (PPO)~\cite{schulman_proximal_2017} agents incorporating generalized advantage estimation~\cite{schulman_high-dimensional_2016}, and off-policy Deep Q-Network (DQN)~\cite{mnih_human-level_2015} agents. PPO and DQN agents are trained using raw RGB image representations of the environment, whereas REINFORCE and AC agents operate on a low-dimensional state representation based on a one-hot encoding of the agent's location. For the REINFORCE and AC agents, each pre-trained model is fine-tuned individually on every possible goal state within the environment, resulting in a total of $835$ fine-tuning runs for each algorithm. In contrast, the RGB observations used by PPO and DQN agents include both the agent's state and the goal position, enabling fine-tuning in an episodic manner with goals that are randomly sampled from the set of all valid states. This allows us to evaluate the benefit of empowerment within two distinct tasks: (1) episodic fine-tuning with randomly sampled goals on RGB images (PPO and DQN), and (2) individual fine-tuning on every possible goal state within the environment on the encoded agent position (REINFORCE and AC). For more details on our experimental setup we refer to Appendix~\ref{sec:appx_implementation_details}.
\subsection{Capacity-Achieving vs. Capacity-Maximizing Policies}
The distinction between capacity-achieving and capacity-maximizing policies lies in their objectives and outcomes.
A capacity-achieving policy~$\pi^*(\mathbf{a} | \mathbf{s})$ (often referred to as source policy \cite{mohamed_variational_2015,karl_unsupervised_2019}) is one that is needed to calculate the empowerment value~$\mathcal{E}(\mathbf{s})$ for a given state~$\mathbf{s}$ (i.e. the policy $\pi$ which achieves empowerment in Equation~\ref{eq:empowerment}).
It is the outcome of the Blahut-Arimoto algorithm \cite{blahut_computation_1972,arimoto_algorithm_1972} and the optimal one from previously reported results on optimal initializations for unsupervised skill learning methods \cite{eysenbach_information_2022}, cf.~Appendix \ref{sec:appx_info_geom_empowerment}.
Such a policy visits each state equally often under the transition dynamics~$\mathcal{P}(\mathbf{s}' | \mathbf{s}, \mathbf{a})$.
It does not seek states with high empowerment values but rather focuses on optimizing the mutual information~$\mathcal{I}(\mathbf{s}'; \mathbf{a} | \mathbf{s})$ at that specific state.
On the other hand, a capacity-maximizing policy refers to a policy that actively seeks to maximize empowerment over time by navigating the environment to reach states with higher empowerment values (i.e. directly maximizing the given empowerment values). Previous work advocate for capacity-maximizing policies as the optimal ones for empowerment-based, unsupervised control \cite{klyubin_all_2005,klyubin_empowerment_2005,klyubin_keep_2008,salge_empowermentintroduction_2014}.
To assess which pre-training approach is more effective for fine-tuning on downstream task, we empirically compare the sample efficiency during fine-tuning of agents pre-trained with capacity-maximizing policies to those pre-trained with capacity-achieving policies on the gridworld navigation task. All agents are trained using the REINFORCE algorithm \cite{williams_simple_1992}. The left panel of Figure~\ref{fig:exp_horizon_and_capacity} shows that the capacity-maximizing policies are more data-efficient than the capacity-achieving policies. Nevertheless, both methods yield pre-trained agents that outperform the baseline when fine-tuned to downstream tasks.
In the following, we will focus on capacity-maximizing policies.
\subsection{Influence of Empowerment Horizon}
\input{figures/exp_horizon_and_capacity/figure}
Next, we evaluate our proposed formalism of discounted empowerment by investigating the impact of the empowerment horizon on the downstream performance of REINFORCE agents.
Specifically, the agents are pre-trained using varying empowerment horizons and subsequently fine-tuned across all possible goal states within the environment.
The aggregated results, as illustrated in the right panel of Figure~\ref{fig:exp_horizon_and_capacity}, again demonstrate that agents pre-trained with empowerment consistently achieve superior data-efficiency during fine-tuning. These findings suggest that extended (effective) horizons $n > 1$ enable more effective exploration.
Moreover, we observe that the benefits of pre-training diminish as the empowerment horizon shortens. Conversely, if the horizon is too long, performance also degrades. This suggests the existence of an optimal $n$-step empowerment horizon that is environment-dependent.
Our proposed formulation instead uses a discounted empowerment objective, which balances short- and long-term empowerment through a discount factor~$\lambda$. Notably, we found that performance is sensitive to the choice of $n$, but largely insensitive to $\lambda$. This allows us to fix $\lambda = 0.95$ across all experiments, significantly reducing the burden of hyperparameter tuning. Thus, discounted empowerment is the most effective choice for pre-training.
\subsection{Deterministic Gridworld}
We now focus on discounted empowerment as our pre-training reward and compare the performance of several RL algorithms within the deterministic gridworld environment. The results of the comparison of these methods are presented in Figure~\ref{fig:exp_gridworld}.
\input{figures/exp_gridworld_alternative/figure}
\textbf{REINFORCE (Policy only) \cite{williams_simple_1992}.}
The empirical results demonstrate that agents pre-trained with empowerment-based objectives consistently outperform from-scratch trained REINFORCE agents in terms of data efficiency. This highlights the utility of empowerment as a pre-training signal.
\textbf{REINFORCE with Baseline \cite{williams_simple_1992}.}
Additionally, we train REINFORCE agents with a baseline which can reduce the variance of the policy gradient estimates. Previous work on fine-tuning report results that naive policy initialization with critics often result in suboptimal performance due to catastrophic forgetting \cite{uchendu_jump-start_2023}.
Our results show that while introducing a baseline accelerates convergence, empowerment-based pre-training still provides additional benefits. In particular, the data efficiency improvements from empowerment remain consistent, matching or exceeding those observed with REINFORCE agents without a baseline.
\textbf{On-Policy Actor-Critic \cite{sutton_reinforcement_2018}.}
While empowerment-based pre-training continues to perform favorably, the performance gap compared to actor-critic agents trained from scratch becomes smaller. This is expected, as our actor-critic method incorporates TD learning for updating both policy and value functions, which introduces additional variance reduction beyond what a baseline alone provides. As a result, the relative advantage of empowerment diminishes somewhat, though it remains a beneficial addition.
\textbf{PPO (On-Policy Actor-Critic) \cite{schulman_proximal_2017}.}
The results obtained with PPO agents indicate that policies fine-tuned using this advanced actor-critic framework exhibit distinct behavioral patterns compared to those fine-tuned with REINFORCE. Notably, we observe that empowerment-based pre-training does not lead to a clear performance improvement in PPO. However, it also does not degrade performance, indicating that empowerment remains a valid and safe pre-training objective.
Furthermore, we observe comparable fine-tuning performance when only the policy is initialized with pre-trained weights and the critic is re-initialized with random weights, suggesting that PPO's performance is robust to the critic's initialization.
\textbf{DQN (Value-based Off-Policy RL) \cite{mnih_human-level_2015}.}
The experiments involving DQN agents use a value-based, off-policy reinforcement learning approach, where the policy is implicitly derived from the state-action value function. Our results show that empowerment-based pre-training not only accelerates learning but also leads to improved final performance. This indicates that, beyond enhancing data efficiency, empowerment can guide the agent toward better solutions in the DQN setting.
From these results, we assume that empowerment can have an effect similar to a variance reduction mechanism by stabilizing the learning process. While methods like PPO and DQN already include strong built-in variance reduction techniques—such as TD learning~\cite{sutton_reinforcement_2018} and generalized advantage estimation~\cite{schulman_high-dimensional_2016}—empowerment consistently supports or enhances performance. Rather than being redundant, it provides a promising, complementary mechanism to further address the high-variance challenges in reinforcement learning. We suppose that this effect is especially valuable in complex environments where standard methods may struggle early in training, highlighting empowerment as a robust and general-purpose enhancement across algorithms.
\input{figures/exp_stochastic/figure}
\subsection{Stochastic Gridworld}
We now assess the effectiveness of empowerment-based pre-training in a stochastic gridworld environment. In comparison to the deterministic experiments presented above, the stochastic setup provides a more challenging testbed for evaluating the robustness of pre-trained policies and aims to determine whether empowerment-based pre-training enhances the agent's ability to adapt and perform effectively under stochastic dynamics.
Specifically, we compare the fine-tuning performance of REINFORCE agents pre-trained with an empowerment horizon of $5$ steps against agents trained from scratch without any pre-training. The results, presented in Figure~\ref{fig:exp_stochastic}, indicate that agents pre-trained with a $5$-steps empowerment horizon exhibit superior performance in terms of data efficiency compared to those trained from scratch.
Our results demonstrate the effectiveness of empowerment in deterministic as well as in stochastic environments, underscoring its potential as a pre-training signal for enhancing the performance of RL agents.
\begin{figure*}[t]
\centering
\hfill
\begin{subfigure}[t]{0.16\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/empowerment_1_step.png}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.16\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/empowerment_3_steps.png}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.16\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/empowerment_5_steps.png}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.16\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/empowerment_32_steps.png}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.16\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/empowerment_discounted.png}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.16\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/goal.png}
\end{subfigure}
\hfill
%\vspace*{10pt}
\caption{\label{fig:gridworld} Empowerment-values of our deterministic gridworld environment. The empowerment values are calculated for $1$-, $3$-, $5$ and $32$-steps and for the discounted case (from left to right). The $32$-steps empowerment grid shows a mostly uniform empowerment landscape due to the issue that most other states can be reached from any state in a horizon of $32$ steps. The last image shows the reward map for an exemplary goal state of the environment.}
\vspace{-2ex}
%\vspace*{-10pt}
\end{figure*}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{\linewidth}
\includegraphics[width=\linewidth]{figures/exp_horizon_and_capacity/plot.pdf}
\end{subfigure}
%\vspace*{10pt}
\caption{\label{fig:exp_horizon_and_capacity} REINFORCE training curves for fine-tuning on individual goal states in a deterministic grid-world environment. \textbf{Left panel}: Both capacity-achieving and capacity-maximizing policies outperform the baseline in terms of data efficiency, with the capacity-maximizing agent performing best. \textbf{Right panel}: Comparison of the performance of agents pre-trained with $n$-step empowerment with different empowerment horizons and using our proposed discounted empowerment reward. Discounted empowerment performs favorably, without the need to tune the horizon length.}
\vspace{-2ex}
%\vspace*{-10pt}
\end{figure*}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{\linewidth}
\includegraphics[width=\linewidth]{figures/exp_gridworld_alternative/plot.pdf}
\end{subfigure}
%\vspace*{10pt}
\caption{\label{fig:exp_gridworld} The gridworld results demonstrate that empowerment-based agents consistently outperform agents trained from scratch, with the most significant improvement observed in the case of REINFORCE and Actor-Critic. In contrast, the observed differences for PPO and DQN are comparatively minor. We presume that while empowerment contributes to variance reduction during learning, its impact diminishes in algorithms such as PPO and DQN, which already incorporate effective variance reduction mechanisms.}
\vspace{-2ex}
%\vspace*{-10pt}
\end{figure*}
\begin{wrapfigure}{r}{0.382\linewidth}
\includegraphics[width=\linewidth]{figures/exp_stochastic/plot.pdf}
\caption{\label{fig:exp_stochastic} REINFORCE training curves in a stochastic gridworld, demonstrating the effectiveness of empowerment-based pretraining also for complex stochastic environments.}
\end{wrapfigure}
\section{Discussion} \label{sec:discussion}
This paper explores the concept of empowerment as a foundational principle for pre-training RL agents. By extending the traditional notion of empowerment to include our discounted formulation, we provide a more nuanced understanding of how agents can exert control over their environments across varying temporal horizons without any external knowledge.
Our experiments demonstrate the efficacy of empowerment-based pre-training in both deterministic and stochastic settings, highlighting its potential to improve data efficiency and adaptability in downstream tasks. In particular, our discounted formulation of multi-step empowerment proves to be an effective pre-training reward, offering strong performance while mitigating the need for hyperparameter tuning.
Additionally, our analysis of empowerment-based pre-training across diverse RL algorithms—including REINFORCE, AC, PPO, and DQN—demonstrates that empowerment consistently performs on par with or better than standard training, often accelerating convergence and, in some cases, leading to improved final performance. These benefits are consistent with the interpretation of empowerment as a form of variance reduction, offering a robust and complementary strategy across both policy- and value-based methods. Another key contribution of this work is the distinction between capacity-achieving and capacity-maximizing policies.
While capacity-maximizing policies exhibit greater data efficiency, both approaches outperform baseline methods, validating the general utility of empowerment as a pre-training signal.
In conclusion, this study establishes empowerment as a robust and versatile pre-training objective, capable of accelerating learning and improving performance across a range of RL algorithms. Our results show that empowerment consistently performs on par with or better than standard training, and its variance-reducing properties make it a compelling addition to existing learning pipelines. While gains with advanced actor-critic methods are sometimes limited, we believe this reflects the strength of these baselines rather than a limitation of empowerment itself. In more complex, high-dimensional environments—where variance is harder to control and exploration more challenging—we expect the benefits of empowerment-based pre-training to become even more pronounced. Realizing this potential will require tackling the computational challenges associated with estimating empowerment efficiently at scale, which we leave for future work. Overall, our findings position empowerment as a promising direction for further research, particularly in the broader context of unsupervised pre-training for large-scale RL agents.
\textbf{Broader impact.} As a general pre-training method for RL, this approach is not inherently tied to any specific application; however, we acknowledge that fine-tuned instances of such pre-trained policies may still pose risks and produce harmful outcomes depending on their deployment context.
In value-based methods, the value learned during pre-training is not the same as the value function for the downstream task. However, this doesn’t seem to affect the performance, can you comment on that, why should this work for value-network initialization?
%\section*{Acknowledgements}
\begin{ack}
\label{sec:acknowledgements}
Joschka Boedecker is part of BrainLinks-BrainTools which is funded by the Federal Ministry of Economics, Science and Arts of Baden-Württemberg within the sustainability program for projects of the excellence initiative II.
\end{ack}
\clearpage
\section{Implementation Details}\label{sec:appx_implementation_details}
\subsection{Stochastic Gridworld}
Stochasticity is introduced by allowing the environment, with a probability of $20$\%, to move the agent randomly either to the left or right relative to its intended direction (i.e. with a probability of $10$\% to the left and $10$\% to the right).
\input{figures/gridworld/figure_stochastic}
\subsection{Experiments}\label{sec:appx_experiments}
Each episode in our gridworld environment consists of $32$ steps. Rewards are normalized to the range $[0,1]$, with the goal state assigned a reward of $1$. In empowerment-maximization environments, the goal state corresponds to the state with maximal empowerment. We calculate discounted empowerment with a discount factor of $\lambda=0.95$ and a time horizon of $H=32$ steps. To ensure exploration during later fine-tuning, we penalize low entropy of the policy during pre-training and fine-tuning. All our networks are trained using the Adam optimizer \cite{kingma_adam_2017} with default parameters.
We evaluate each agent every $25,000$ environment steps for $5$ (REINFORCE and AC) or $25$ (PPO and DQN) episodes. The graphs in our plots show the mean return and the standard deviation of the mean over the evaluation results of all runs of the experiment.
Our implementation is based on JAX \cite{bradbury_jax_2018}. PPO and DQN agents are trained using NVIDIA V100 GPUs, with each training run requiring approximately two hours using a single GPU. We utilize the Stable-Baselines (SBX) framework \cite{raffin_stable-baselines3_2021} for these two algorithms. All other algorithms are executed on CPUs, with each run completing in approximately $45$ minutes (for pre-training and fine-tuning). Across all experiments, we allocate $80$ GB of memory; however, lower memory usage may suffice depending on the specific algorithm.
\subsubsection{Training of the Capacity-Achieving Policy}
The capacity-achieving policy is the output of the Blahut-Arimoto algorithm \cite{blahut_computation_1972,arimoto_algorithm_1972}. The algorithm provides only action distributions for each state independently. We pre-train our neural network policies by approximating these distributions through behavior cloning from sampled actions, enabling effective subsequent fine-tuning.
\clearpage
\subsection{Hyperparameters}
\input{tables/hyperparameters_reinforce}
\input{tables/hyperparameters_ac}
\input{tables/hyperparameters_ppo}
\input{tables/hyperparameters_dqn}
\clearpage
\section{Information Geometry of Empowerment-based RL}
\label{sec:appx_info_geom_empowerment}
\input{figures/information_geometry/figure}
%\todo[inline]{Here we have to answer why it is preferable to lie in the center of the information ball in the state-marginal simplex}
We now want to give an intuition why it is beneficial from an information geometric point of view to maximize mutual information. We build on prior work to provide an information geometric interpretation of what mutual information maximization does \cite{eysenbach_information_2022}. But in contrast we transfer the interpretation from latent skills to ordinary actions.
The average state distribution can be written as
$$\rho(\mathbf{s}' | \mathbf{s}) = \sum_\mathbf{a} \rho(\mathbf{s}', \mathbf{a} | \mathbf{s}) = \sum_\mathbf{a} \mathcal{P}(\mathbf{s}' | \mathbf{s}, \mathbf{a}) \pi(\mathbf{a} | \mathbf{s}).$$
\begin{remark}
\textup{(Theorem 13.1.1 from \citet[page 430]{cover_elements_2005})}
Maximizing mutual information is equivalent to minimizing the divergence between the average state distribution $\rho(\mathbf{s'} \mid \mathbf{s})$ and the furthest achievable state distributions \cite{eysenbach_information_2022,cover_elements_2005}:
$$\mathcal{E}(s)=\max_{\pi(\mathbf{a} | \mathbf{s})} I(\mathbf{s}';\mathbf{a} | \mathbf{s}) = \min_{\rho(\mathbf{s}' | \mathbf{s})} \max_a D_{KL} (\rho(\mathbf{s}'| \mathbf{s}, \mathbf{a}) | \rho(\mathbf{s}' | \mathbf{s}))$$
\end{remark}
In other words: By maximizing mutual information, we minimize the maximum distance of the average state distribution $\rho(\mathbf{s}' | \mathbf{s})$ to all other achievable state distributions (under the information ball given by $\mathcal{P}(\mathbf{s}'|\mathbf{s}, \mathbf{a})$) \cite[page 430, Theorem 13.1.1]{cover_elements_2005}.
Thus, the optimal prior of $\rho(\mathbf{s}' | \mathbf{s})$ that achieves the minimum is the output distribution $\rho^*(\mathbf{s}' | \mathbf{s})$ induced by the capacity-achieving input distribution $\pi^*(\mathbf{a} | \mathbf{s})$:
$$\rho^*(\mathbf{s}' | \mathbf{s}) = \sum_\mathbf{a} \mathcal{P}(\mathbf{s}' | \mathbf{s}, \mathbf{a}) \pi^*(\mathbf{a} | \mathbf{s}) = \mathbb{E}_{\pi^*(\mathbf{a} | \mathbf{s})} [\mathcal{P}(\mathbf{s}' | \mathbf{s}, \mathbf{a})]$$
Thus, the empowerment-achieving policy has a state marginal distribution in the center of the information ball bringing the average state distribution close to other policies.
\citet{eysenbach_information_2022} argues that, in context of skill learning, such an average state distribution would be a beneficial initialization for downstream task learning. Whether such an initialization is suitable for unconditioned policy pre-training—i.e., policies not conditioned on skills—remains an open question, as a fundamental distinction exists between actions, which are explicitly grounded in the MDP, and skills, which are abstract latent variables.
\clearpage
\begin{figure}[!h]
\centering
\begin{subfigure}[t]{0.4\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/empowerment_5_steps_stochastic_easy.png}
\end{subfigure}
%
\hspace*{10pt}
\begin{subfigure}[t]{0.4\linewidth}
\includegraphics[width=\linewidth]{figures/gridworld/empowerment_5_steps_stochastic_hard.png}
\end{subfigure}
\label{fig:gridworld_stochastic}
\caption{$5$-steps empowerment-values of the stochastic gridworld environment. In the left environment the agent randomly moves to the left or right with a probability of $10$\% and in the right environment the agent randomly moves to the left or right with a probability of $20$\%. Otherwise the agent moves in the intended direction of the chosen action.}
\end{figure}
\begin{table}[h]
\centering
\caption{\label{tab:hyperparameters} Overview of the hyperparameters for REINFORCE.}
\begin{tabularx}{0.6\textwidth}{@{}ll@{}}
\toprule
& \textbf{REINFORCE} \\
\midrule
\textbf{General} & \\
Random seeds & $5$ \\
Number of environments & $16$ \\
Pretraining steps & $16 \times 1,000,000$ \\
Finetuning steps & $16 \times 1,000,000$ \\
Batch size & $32$ \\
\midrule
\textbf{Actor} & \\
Hidden layers & $2$ \\
Hidden layer dim. & $256$ \\
Learning Rate & $1 \times 10^{-4}$ \\
Entropy coefficient & $1 \times 10^{-1}$ \\
Discount factor $\gamma$ & $0.99$ \\
\midrule
\textbf{Baseline (if used)} & \\
Hidden layers & $2$ \\
Hidden layer dim. & $256$ \\
Learning rate & $3 \times 10^{-4}$ \\
\bottomrule
\end{tabularx}
\end{table}
\begin{table}[h]
\centering
\caption{\label{tab:hyperparameters} Overview of the hyperparameters for AC.}
\begin{tabularx}{0.6\textwidth}{@{}ll@{}}
\toprule
& \textbf{AC} \\
\midrule
\textbf{General} & \\
Random seeds & $5$ \\
Number of environments & $16$ \\
Pretraining steps & $16 \times 1,000,000$ \\
Finetuning steps & $16 \times 1,000,000$ \\
Batch size & $32$ \\
\midrule
\textbf{Actor} & \\
Hidden layers & $2$ \\
Hidden layer dim. & $256$ \\
Learning Rate & $1 \times 10^{-4}$ \\
Entropy coefficient & $1 \times 10^{-1}$ \\
Discount factor $\gamma$ & $0.99$ \\
\midrule
\textbf{Critic} & \\
Hidden layers & $2$ \\
Hidden layer dim. & $256$ \\
Learning rate & $1 \times 10^{-4}$ \\
\bottomrule
\end{tabularx}
\end{table}
\begin{table}[h]
\centering
\caption{\label{tab:hyperparameters_ppo} Overview of the hyperparameters for PPO.}
\begin{tabularx}{0.6\textwidth}{@{}ll@{}}
\toprule
& \textbf{PPO} \\
\midrule
\textbf{General} & \\
Random seeds & $6$ \\
Number of environments & $4$ \\
Pretraining steps & $1,000,000$ \\
Finetuning steps & $1,250,000$ \\
Batch size & $64$ \\
GAE-$\lambda$ & $0.95$ \\
Advantage normalization & \texttt{True} \\
Number of epochs per update & $10$ \\
Steps between updates & $1024$ \\
Max. gradient norm clipping & $0.5$ \\
\midrule
\textbf{Actor} & \\
Clip range & $0.2$ \\
CNN architecture & Same as in \citet{mnih_human-level_2015} \\
Learning rate & $1 \times 10^{-4}$ \\
Entropy coefficient & $1 \times 10^{-2}$ \\
Discount factor $\gamma$ & $0.99$ \\
\midrule
\textbf{Critic} & \\
Critic loss weight coeff. & $0.5$ \\
CNN architecture & Same as in \citet{mnih_human-level_2015} \\
Learning rate & $1 \times 10^{-4}$ \\
\bottomrule
\end{tabularx}
\end{table}
\begin{table}[h]
\centering
\caption{\label{tab:hyperparameters_dqn} Overview of the hyperparameters for DQN.}
\begin{tabularx}{0.6\textwidth}{@{}ll@{}}
\toprule
& \textbf{DQN} \\
\midrule
\textbf{General} & \\
Random seeds & $6$ \\
Number of environments & $1$ \\
Replay buffer size & $1,000,000$ \\
Pretraining steps & $1,000,000$ \\
Finetuning steps & $1,250,000$ \\
Batch size & $32$ \\
Number of epochs per update & $10$ \\
Steps between updates & $4$ \\
Seed steps & $100$ \\
Initial exploration rate & $1.0$ \\
Final exploration rate & $0.01$ \\
Decay exploration rate until & $100,000$ steps \\
\midrule
\textbf{Critic} & \\
Discount factor $\gamma$ & $0.99$ \\
Target critic $\tau$ & $0.95$ \\
Target update interval & Every $1,000$ steps \\
CNN architecture & Same as in \citet{mnih_human-level_2015} \\
Learning rate & $1 \times 10^{-4}$ \\
\bottomrule
\end{tabularx}
\end{table}
\begin{figure*}[h]
\centering
\begin{subfigure}[t]{0.45\linewidth}
\includegraphics[width=\linewidth]{figures/information_geometry/simplex.pdf}
\end{subfigure}
%\hfill
\begin{subfigure}[t]{0.45\linewidth}
%\reflectbox{\includegraphics[width=0.9\linewidth]{figures/information_geometry/skills.pdf}}
\includegraphics[width=0.95\linewidth]{figures/information_geometry/skills.pdf}
\end{subfigure}
%\vspace*{10pt}
\caption{\label{fig:info_geom} The left panel illustrates the state-marginal simplex of an environment comprising three discrete states, while the right panel depicts the corresponding state-marginal polytope. The green point represents the initialization of an agent that has been pre-trained using empowerment objectives. All other points located within the information ball correspond to feasible state-marginals, encompassing all possible initializations as well as all achievable task-specific policies. The dark red points indicate policies positioned along the boundary of the information ball, representing those that are maximally distant from the empowerment-pretrained agent. Notably, because the empowerment-pretrained agent is situated near the center of the information ball, it is expected to exhibit a high degree of adaptability, enabling it to transfer and adapt to a wide range of downstream tasks more rapidly on average. Adapted from \cite{eysenbach_information_2022}.}
%\vspace{-2ex}
%\vspace*{-10pt}
\end{figure*}
\begin{thebibliography}{61}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Levine et~al.(2020)Levine, Kumar, Tucker, and Fu]{levine_offline_2020}
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.
\newblock Offline {Reinforcement} {Learning}: {Tutorial}, {Review}, and
{Perspectives} on {Open} {Problems}, November 2020.
\newblock URL \url{http://arxiv.org/abs/2005.01643}.
\newblock arXiv:2005.01643 [cs, stat].
\bibitem[Xie et~al.(2022)Xie, Lin, Li, Li, and Ye]{xie_pretraining_2022}
Zhihui Xie, Zichuan Lin, Junyou Li, Shuai Li, and Deheng Ye.
\newblock Pretraining in {Deep} {Reinforcement} {Learning}: {A} {Survey},
November 2022.
\newblock URL \url{http://arxiv.org/abs/2211.03959}.
\newblock arXiv:2211.03959 [cs].
\bibitem[Springenberg et~al.(2024)Springenberg, Abdolmaleki, Zhang, Groth,
Bloesch, Lampe, Brakel, Bechtle, Kapturowski, Hafner, Heess, and
Riedmiller]{springenberg_offline_2024}
Jost~Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth,
Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Maria~Elisabeth
Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Heess, and Martin
Riedmiller.
\newblock Offline {Actor}-{Critic} {Reinforcement} {Learning} {Scales} to
{Large} {Models}.
\newblock In \emph{Forty-first {International} {Conference} on {Machine}
{Learning}}, 2024.
\newblock URL \url{https://openreview.net/forum?id=tl2qmO5kpD}.
\bibitem[Klyubin et~al.(2005{\natexlab{a}})Klyubin, Polani, and
Nehaniv]{klyubin_all_2005}
Alexander~S. Klyubin, Daniel Polani, and Chrystopher~L. Nehaniv.
\newblock All {Else} {Being} {Equal} {Be} {Empowered}.
\newblock In Mathieu~S. Capcarrère, Alex~A. Freitas, Peter~J. Bentley,
Colin~G. Johnson, and Jon Timmis, editors, \emph{Advances in {Artificial}
{Life}}, pages 744--753, Berlin, Heidelberg, 2005{\natexlab{a}}. Springer.
\newblock ISBN 978-3-540-31816-3.
\newblock \doi{10.1007/11553090_75}.
\bibitem[Klyubin et~al.(2005{\natexlab{b}})Klyubin, Polani, and
Nehaniv]{klyubin_empowerment_2005}
A.S. Klyubin, D.~Polani, and C.L. Nehaniv.
\newblock Empowerment: {A} {Universal} {Agent}-{Centric} {Measure} of
{Control}.
\newblock In \emph{2005 {IEEE} {Congress} on {Evolutionary} {Computation}},
volume~1, pages 128--135, Edinburgh, Scotland, UK, 2005{\natexlab{b}}. IEEE.
\newblock ISBN 978-0-7803-9363-9.
\newblock \doi{10.1109/CEC.2005.1554676}.
\newblock URL \url{http://ieeexplore.ieee.org/document/1554676/}.
\bibitem[Salge et~al.(2014{\natexlab{a}})Salge, Glackin, and
Polani]{salge_empowermentintroduction_2014}
Christoph Salge, Cornelius Glackin, and Daniel Polani.
\newblock Empowerment–{An} {Introduction}.
\newblock In Mikhail Prokopenko, editor, \emph{Guided {Self}-{Organization}:
{Inception}}, pages 67--114. Springer, Berlin, Heidelberg,
2014{\natexlab{a}}.
\newblock ISBN 978-3-642-53734-9.
\newblock \doi{10.1007/978-3-642-53734-9_4}.
\newblock URL \url{https://doi.org/10.1007/978-3-642-53734-9_4}.
\bibitem[Mohamed and Jimenez~Rezende(2015)]{mohamed_variational_2015}
Shakir Mohamed and Danilo Jimenez~Rezende.
\newblock Variational information maximisation for intrinsically motivated
reinforcement learning.
\newblock \emph{Advances in neural information processing systems}, 28, 2015.
\bibitem[Karl et~al.(2019)Karl, Becker-Ehmck, Soelch, Benbouzid, van~der Smagt,
and Bayer]{karl_unsupervised_2019}
Maximilian Karl, Philip Becker-Ehmck, Maximilian Soelch, Djalel Benbouzid,
Patrick van~der Smagt, and Justin Bayer.
\newblock Unsupervised real-time control through variational empowerment.
\newblock In \emph{The {International} {Symposium} of {Robotics} {Research}},
pages 158--173. Springer, 2019.
\bibitem[Leibfried et~al.(2019)Leibfried, Pascual-Díaz, and
Grau-Moya]{leibfried_unified_2019}
Felix Leibfried, Sergio Pascual-Díaz, and Jordi Grau-Moya.
\newblock A {Unified} {Bellman} {Optimality} {Principle} {Combining} {Reward}
{Maximization} and {Empowerment}.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}},
volume~32. Curran Associates, Inc., 2019.
\newblock URL
\url{https://proceedings.neurips.cc/paper/2019/hash/13384ffc9d8bdb21c53c6f72d46f7866-Abstract.html}.
\bibitem[Zhao et~al.(2020)Zhao, Lu, Abbeel, and Tiomkin]{zhao_efficient_2020}
Ruihan Zhao, Kevin Lu, Pieter Abbeel, and Stas Tiomkin.
\newblock Efficient {Empowerment} {Estimation} for {Unsupervised}
{Stabilization}.
\newblock October 2020.
\newblock URL \url{https://openreview.net/forum?id=u2YNJPcQlwq}.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal,
Sastry, Askell, Mishkin, Clark, and {others}]{radford_learning_2021}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
and {others}.
\newblock Learning transferable visual models from natural language
supervision.
\newblock In \emph{International conference on machine learning}, pages
8748--8763. PMLR, 2021.
\bibitem[Girdhar et~al.(2023)Girdhar, El-Nouby, Liu, Singh, Alwala, Joulin, and
Misra]{girdhar_imagebind_2023}
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan~Vasudev
Alwala, Armand Joulin, and Ishan Misra.
\newblock Imagebind: {One} embedding space to bind them all.
\newblock In \emph{Proceedings of the {IEEE}/{CVF} {Conference} on {Computer}
{Vision} and {Pattern} {Recognition}}, pages 15180--15190, 2023.
\bibitem[Oquab et~al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov,
Fernandez, HAZIZA, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li,
Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and
Bojanowski]{oquab_dinov2_2024}
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy~V. Vo, Marc Szafraniec,
Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin
El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao
Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel
Synnaeve, Hu~Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin,
and Piotr Bojanowski.
\newblock {DINOv2}: {Learning} {Robust} {Visual} {Features} without
{Supervision}.
\newblock \emph{Transactions on Machine Learning Research}, 2024.
\newblock ISSN 2835-8856.
\newblock URL \url{https://openreview.net/forum?id=a68SUt6zFt}.
\bibitem[Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal,
Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan,
Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess,
Clark, Berner, McCandlish, Radford, Sutskever, and
Amodei]{brown_language_2020}
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared~D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris
Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,
and Dario Amodei.
\newblock Language {Models} are {Few}-{Shot} {Learners}.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}},
volume~33, pages 1877--1901. Curran Associates, Inc., 2020.
\newblock URL
\url{https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html}.
\bibitem[Team et~al.(2024{\natexlab{a}})Team, Anil, Borgeaud, Alayrac, Yu,
Soricut, Schalkwyk, Dai, Hauth, Millican, Silver, Johnson, Antonoglou,
Schrittwieser, Glaese, Chen, Pitler, Lillicrap, Lazaridou, Firat, Molloy,
Isard, Barham, Hennigan, Lee, Viola, Reynolds, Xu, Doherty, Collins, Meyer,
Rutherford, Moreira, Ayoub, Goel, Krawczyk, Du, Chi, Cheng, Ni, Shah, Kane,
Chan, Faruqui, Severyn, Lin, Li, Cheng, Ittycheriah, Mahdieh, Chen, Sun,
Tran, Bagri, Lakshminarayanan, Liu, Orban, Güra, Zhou, Song, Boffy,
Ganapathy, Zheng, Choe, Weisz, Zhu, Lu, Gopal, Kahn, Kula, Pitman, Shah,
Taropa, Merey, Baeuml, Chen, Shafey, Zhang, Sercinoglu, Tucker, Piqueras,
Krikun, Barr, Savinov, Danihelka, Roelofs, White, Andreassen, von Glehn,
Yagati, Kazemi, Gonzalez, Khalman, Sygnowski, Frechette, Smith, Culp,
Proleev, Luan, Chen, Lottes, Schucher, Lebron, Rrustemi, Clay, Crone,
Kocisky, Zhao, Perz, Yu, Howard, Bloniarz, Rae, Lu, Sifre, Maggioni, Alcober,
Garrette, Barnes, Thakoor, Austin, Barth-Maron, Wong, Joshi, Chaabouni,
Fatiha, Ahuja, Tomar, Senter, Chadwick, Kornakov, Attaluri, Iturrate, Liu,
Li, Cogan, Chen, Jia, Gu, Zhang, Grimstad, Hartman, Garcia, Pillai, Devlin,
Laskin, Casas, Valter, Tao, Blanco, Badia, Reitter, Chen, Brennan, Rivera,
Brin, Iqbal, Surita, Labanowski, Rao, Winkler, Parisotto, Gu, Olszewska,
Addanki, Miech, Louis, Teplyashin, Brown, Catt, Balaguer, Xiang, Wang,
Ashwood, Briukhov, Webson, Ganapathy, Sanghavi, Kannan, Chang, Stjerngren,
Djolonga, Sun, Bapna, Aitchison, Pejman, Michalewski, Yu, Wang, Love, Ahn,
Bloxwich, Han, Humphreys, Sellam, Bradbury, Godbole, Samangooei, Damoc,
Kaskasoli, Arnold, Vasudevan, Agrawal, Riesa, Lepikhin, Tanburn, Srinivasan,
Lim, Hodkinson, Shyam, Ferret, Hand, Garg, Paine, Li, Li, Giang, Neitz,
Abbas, York, Reid, Cole, Chowdhery, Das, Rogozińska, Nikolaev, Sprechmann,
Nado, Zilka, Prost, He, Monteiro, Mishra, Welty, Newlan, Jia, Allamanis, Hu,
de~Liedekerke, Gilmer, Saroufim, Rijhwani, Hou, Shrivastava, Baddepudi,
Goldin, Ozturel, Cassirer, Xu, Sohn, Sachan, Amplayo, Swanson, Petrova,
Narayan, Guez, Brahma, Landon, Patel, Zhao, Villela, Wang, Jia, Rahtz,
Giménez, Yeung, Keeling, Georgiev, Mincu, Wu, Haykal, Saputro, Vodrahalli,
Qin, Cankara, Sharma, Fernando, Hawkins, Neyshabur, Kim, Hutter, Agrawal,
Castro-Ros, Driessche, Wang, Yang, Chang, Komarek, McIlroy, Lučić, Zhang,
Farhan, Sharman, Natsev, Michel, Bansal, Qiao, Cao, Shakeri, Butterfield,
Chung, Rubenstein, Agrawal, Mensch, Soparkar, Lenc, Chung, Pope, Maggiore,
Kay, Jhakra, Wang, Maynez, Phuong, Tobin, Tacchetti, Trebacz, Robinson,
Katariya, Riedel, Bailey, Xiao, Ghelani, Aroyo, Slone, Houlsby, Xiong, Yang,
Gribovskaya, Adler, Wirth, Lee, Li, Kagohara, Pavagadhi, Bridgers, Bortsova,
Ghemawat, Ahmed, Liu, Powell, Bolina, Iinuma, Zablotskaia, Besley, Chung,
Dozat, Comanescu, Si, Greer, Su, Polacek, Kaufman, Tokumine, Hu, Buchatskaya,
Miao, Elhawaty, Siddhant, Tomasev, Xing, Greer, Miller, Ashraf, Roy, Zhang,
Ma, Filos, Besta, Blevins, Klimenko, Yeh, Changpinyo, Mu, Chang, Pajarskas,
Muir, Cohen, Lan, Haridasan, Marathe, Hansen, Douglas, Samuel, Wang, Austin,
Lan, Jiang, Chiu, Lorenzo, Sjösund, Cevey, Gleicher, Avrahami, Boral,
Srinivasan, Selo, May, Aisopos, Hussenot, Soares, Baumli, Chang, Recasens,
Caine, Pritzel, Pavetic, Pardo, Gergely, Frye, Ramasesh, Horgan, Badola,
Kassner, Roy, Dyer, Campos, Tomala, Tang, Badawy, White, Mustafa, Lang,
Jindal, Vikram, Gong, Caelles, Hemsley, Thornton, Feng, Stokowiec, Zheng,
Thacker, Ünlü, Zhang, Saleh, Svensson, Bileschi, Patil, Anand, Ring,
Tsihlas, Vezer, Selvi, Shevlane, Rodriguez, Kwiatkowski, Daruki, Rong, Dafoe,
FitzGerald, Gu-Lemberg, Khan, Hendricks, Pellat, Feinberg, Cobon-Kerr,
Sainath, Rauh, Hashemi, Ives, Hasson, Noland, Cao, Byrd, Hou, Wang, Sottiaux,
Paganini, Lespiau, Moufarek, Hassan, Shivakumar, van Amersfoort, Mandhane,
Joshi, Goyal, Tung, Brock, Sheahan, Misra, Li, Rakićević, Dehghani, Liu,
Mittal, Oh, Noury, Sezener, Huot, Lamm, De~Cao, Chen, Mudgal, Stella, Brooks,
Vasudevan, Liu, Chain, Melinkeri, Cohen, Wang, Seymore, Zubkov, Goel, Yue,
Krishnakumaran, Albert, Hurley, Sano, Mohananey, Joughin, Filonov, Kępa,
Eldawy, Lim, Rishi, Badiezadegan, Bos, Chang, Jain, Padmanabhan, Puttagunta,
Krishna, Baker, Kalb, Bedapudi, Kurzrok, Lei, Yu, Litvin, Zhou, Wu, Sobell,
Siciliano, Papir, Neale, Bragagnolo, Toor, Chen, Anklin, Wang, Feng, Gholami,
Ling, Liu, Walter, Moghaddam, Kishore, Adamek, Mercado, Mallinson, Wandekar,
Cagle, Ofek, Garrido, Lombriser, Mukha, Sun, Mohammad, Matak, Qian, Peswani,
Janus, Yuan, Schelin, David, Garg, He, Duzhyi, Älgmyr, Lottaz, Li, Yadav,
Xu, Chinien, Shivanna, Chuklin, Li, Spadine, Wolfe, Mohamed, Das, Dai, He,
von Dincklage, Upadhyay, Maurya, Chi, Krause, Salama, Rabinovitch, M, Selvan,
Dektiarev, Ghiasi, Guven, Gupta, Liu, Sharma, Shtacher, Paul, Akerlund,
Aubet, Huang, Zhu, Zhu, Teixeira, Fritze, Bertolini, Marinescu, Bölle,
Paulus, Gupta, Latkar, Chang, Sanders, Wilson, Wu, Tan, Thiet, Doshi, Lall,
Mishra, Chen, Luong, Benjamin, Lee, Andrejczuk, Rabiej, Ranjan, Styrc, Yin,
Simon, Harriott, Bansal, Robsky, Bacon, Greene, Mirylenka, Zhou, Sarvana,
Goyal, Andermatt, Siegler, Horn, Israel, Pongetti, Chen, Selvatici, Silva,
Wang, Tolins, Guu, Yogev, Cai, Agostini, Shah, Nguyen, Donnaile, Pereira,
Friso, Stambler, Kurzrok, Kuang, Romanikhin, Geller, Yan, Jang, Lee, Fica,
Malmi, Tan, Banica, Balle, Pham, Huang, Avram, Shi, Singh, Hidey, Ahuja,
Saxena, Dooley, Potharaju, O'Neill, Gokulchandran, Foley, Zhao, Dusenberry,
Liu, Mehta, Kotikalapudi, Safranek-Shrader, Goodman, Kessinger, Globen,
Kolhar, Gorgolewski, Ibrahim, Song, Eichenbaum, Brovelli, Potluri, Lahoti,
Baetu, Ghorbani, Chen, Crawford, Pal, Sridhar, Gurita, Mujika, Petrovski,
Cedoz, Li, Chen, Santo, Goyal, Punjabi, Kappaganthu, Kwak, LV, Velury,
Choudhury, Hall, Shah, Figueira, Thomas, Lu, Zhou, Kumar, Jurdi, Chikkerur,
Ma, Yu, Kwak, Ähdel, Rajayogam, Choma, Liu, Barua, Ji, Park, Hellendoorn,
Bailey, Bilal, Zhou, Khatir, Sutton, Rzadkowski, Macintosh, Shagin, Medina,
Liang, Zhou, Shah, Bi, Dankovics, Banga, Lehmann, Bredesen, Lin, Hoffmann,
Lai, Chung, Yang, Balani, Bražinskas, Sozanschi, Hayes, Alcalde, Makarov,
Chen, Stella, Snijders, Mandl, Kärrman, Nowak, Wu, Dyck, Vaidyanathan, R,
Mallet, Rudominer, Johnston, Mittal, Udathu, Christensen, Verma, Irving,
Santucci, Elsayed, Davoodi, Georgiev, Tenney, Hua, Cideron, Leurent,
Alnahlawi, Georgescu, Wei, Zheng, Scandinaro, Jiang, Snoek, Sundararajan,
Wang, Ontiveros, Karo, Cole, Rajashekhar, Tumeh, Ben-David, Jain, Uesato,
Datta, Bunyan, Wu, Zhang, Stanczyk, Zhang, Steiner, Naskar, Azzam, Johnson,
Paszke, Chiu, Elias, Mohiuddin, Muhammad, Miao, Lee, Vieillard, Park, Zhang,
Stanway, Garmon, Karmarkar, Dong, Lee, Kumar, Zhou, Evens, Isaac, Irving,
Loper, Fink, Arkatkar, Chen, Shafran, Petrychenko, Chen, Jia, Levskaya, Zhu,
Grabowski, Mao, Magni, Yao, Snaider, Casagrande, Palmer, Suganthan, Castaño,
Giannoumis, Kim, Rybiński, Sreevatsa, Prendki, Soergel, Goedeckemeyer,
Gierke, Jafari, Gaba, Wiesner, Wright, Wei, Vashisht, Kulizhskaya, Hoover,
Le, Li, Iwuanyanwu, Liu, Ramirez, Khorlin, Cui, LIN, Wu, Aguilar, Pallo,
Chakladar, Perng, Abellan, Zhang, Dasgupta, Kushman, Penchev, Repina, Wu,
van~der Weide, Ponnapalli, Kaplan, Simsa, Li, Dousse, Yang, Piper, Ie,
Pasumarthi, Lintz, Vijayakumar, Andor, Valenzuela, Lui, Paduraru, Peng, Lee,
Zhang, Greene, Nguyen, Kurylowicz, Hardin, Dixon, Janzer, Choo, Feng, Zhang,
Singhal, Du, McKinnon, Antropova, Bolukbasi, Keller, Reid, Finchelstein,
Raad, Crocker, Hawkins, Dadashi, Gaffney, Franko, Bulanova, Leblond, Chung,
Askham, Cobo, Xu, Fischer, Xu, Sorokin, Alberti, Lin, Evans, Dimitriev,
Forbes, Banarse, Tung, Omernick, Bishop, Sterneck, Jain, Xia, Amid, Piccinno,
Wang, Banzal, Mankowitz, Polozov, Krakovna, Brown, Bateni, Duan, Firoiu,
Thotakuri, Natan, Geist, Girgin, Li, Ye, Roval, Tojo, Kwong, Lee-Thorp, Yew,
Sinopalnikov, Ramos, Mellor, Sharma, Wu, Miller, Sonnerat, Vnukov, Greig,
Beattie, Caveness, Bai, Eisenschlos, Korchemniy, Tsai, Jasarevic, Kong, Dao,
Zheng, Liu, Yang, Zhu, Teh, Sanmiya, Gladchenko, Trdin, Toyama, Rosen,
Tavakkol, Xue, Elkind, Woodman, Carpenter, Papamakarios, Kemp, Kafle,
Grunina, Sinha, Talbert, Wu, Owusu-Afriyie, Du, Thornton, Pont-Tuset,
Narayana, Li, Fatehi, Wieting, Ajmeri, Uria, Ko, Knight, Héliou, Niu, Gu,
Pang, Li, Levine, Stolovich, Santamaria-Fernandez, Goenka, Yustalim, Strudel,
Elqursh, Deck, Lee, Li, Levin, Hoffmann, Holtmann-Rice, Bachem, Arora, Koh,
Yeganeh, Põder, Tariq, Sun, Ionita, Seyedhosseini, Tafti, Liu, Gulati, Liu,
Ye, Chrzaszcz, Wang, Sethi, Li, Brown, Singh, Fan, Parisi, Stanton,
Koverkathu, Choquette-Choo, Li, Lu, Ittycheriah, Shroff, Varadarajan,
Bahargam, Willoughby, Gaddy, Desjardins, Cornero, Robenek, Mittal, Albrecht,
Shenoy, Moiseev, Jacobsson, Ghaffarkhah, Rivière, Walton, Crepy, Parrish,
Zhou, Farabet, Radebaugh, Srinivasan, van~der Salm, Fidjeland, Scellato,
Latorre-Chimoto, Klimczak-Plucińska, Bridson, de~Cesare, Hudson,
Mendolicchio, Walker, Morris, Mauger, Guseynov, Reid, Odoom, Loher, Cotruta,
Yenugula, Grewe, Petrushkina, Duerig, Sanchez, Yadlowsky, Shen, Globerson,
Webb, Dua, Li, Bhupatiraju, Hurt, Qureshi, Agarwal, Shani, Eyal, Khare,
Belle, Wang, Tekur, Kale, Wei, Sang, Saeta, Liechty, Sun, Zhao, Lee, Nayak,
Fritz, Vuyyuru, Aslanides, Vyas, Wicke, Ma, Eltyshev, Martin, Cate, Manyika,
Amiri, Kim, Xiong, Kang, Luisier, Tripuraneni, Madras, Guo, Waters, Wang,
Ainslie, Baldridge, Zhang, Pruthi, Bauer, Yang, Mansour, Gelman, Xu,
Polovets, Liu, Cai, Chen, Sheng, Xue, Ozair, Angermueller, Li, Sinha, Wang,
Wiesinger, Koukoumidis, Tian, Iyer, Gurumurthy, Goldenson, Shah, Blake, Yu,
Urbanowicz, Palomaki, Fernando, Durden, Mehta, Momchev, Rahimtoroghi,
Georgaki, Raul, Ruder, Redshaw, Lee, Zhou, Jalan, Li, Hechtman, Schuh, Nasr,
Milan, Mikulik, Franco, Green, Nguyen, Kelley, Mahendru, Hu, Howland, Vargas,
Hui, Bansal, Rao, Ghiya, Wang, Ye, Sarr, Preston, Elish, Li, Kaku, Gupta,
Pasupat, Juan, Someswar, M., Chen, Amini, Fabrikant, Chu, Dong, Muthal,
Buthpitiya, Jauhari, Hua, Khandelwal, Hitron, Ren, Rinaldi, Drath, Dabush,
Jiang, Godhia, Sachs, Chen, Fan, Taitelbaum, Noga, Dai, Wang, Liang, Hamer,
Ferng, Elkind, Atias, Lee, Listík, Carlen, van~de Kerkhof, Pikus, Zaher,
Müller, Zykova, Stefanec, Gatsko, Hirnschall, Sethi, Xu, Ahuja, Tsai,
Stefanoiu, Feng, Dhandhania, Katyal, Gupta, Parulekar, Pitta, Zhao, Bhatia,
Bhavnani, Alhadlaq, Li, Danenberg, Tu, Pine, Filippova, Ghosh, Limonchik,
Urala, Lanka, Clive, Sun, Li, Wu, Hongtongsak, Li, Thakkar, Omarov,
Majmundar, Alverson, Kucharski, Patel, Jain, Zabelin, Pelagatti, Kohli,
Kumar, Kim, Sankar, Shah, Ramachandruni, Zeng, Bariach, Weidinger, Vu,
Andreev, He, Hui, Kashem, Subramanya, Hsiao, Hassabis, Kavukcuoglu, Sadovsky,
Le, Strohman, Wu, Petrov, Dean, and Vinyals]{gemini_team_gemini_2024}
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu,
Radu Soricut, Johan Schalkwyk, Andrew~M. Dai, Anja Hauth, Katie Millican,
David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser,
Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki
Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul~R. Barham, Tom
Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan
Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem
Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed~Chi, Heng-Tze Cheng, Eric Ni,
Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn,
Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia
Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah
Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy,
Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu,
Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin
Shah, Emanuel Taropa, Majd~Al Merey, Martin Baeuml, Zhifeng Chen, Laurent~El
Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras,
Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs,
Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran
Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette,
Charlotte Smith, Laura Culp, Lev Proleev, Yi~Luan, Xi~Chen, James Lottes,
Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone,
Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam
Bloniarz, Jack~W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred
Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel
Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun
Ahuja, Gaurav~Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov,
Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy
Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale~Jakse Hartman,
Xavier Garcia, Thanumalayan~Sankaranarayana Pillai, Jacob Devlin, Michael
Laskin, Diego de~Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco,
Adrià~Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara
Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi
Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi
Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot
Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov,
Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang,
Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison,
Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love,
Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam,
James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli,
Sébastien M.~R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa,
Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah
Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom~Le
Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah
York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das,
Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas
Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris
Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara~Huiyi Hu, Raoul
de~Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou,
Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin
Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald~Kim Amplayo, Craig
Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma,
Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao
Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev,
Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James
Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam
Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros,
George van~den Driessche, Tao Wang, Fan Yang, Shuo-yiin Chang, Paul Komarek,
Ross McIlroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman,
Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak
Shakeri, Christina Butterfield, Justin Chung, Paul~Kishan Rubenstein, Shivani
Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan
Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez,
Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson,
Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani,
Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena
Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara,
Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali
Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Iinuma, Polina
Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu,
Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël~Lopez Kaufman,
Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed
Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen
Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos
Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi
Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline~Le Lan,
Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar
Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu,
Jaime~Alonso Lorenzo, Lars~Lowe Sjösund, Sébastien Cevey, Zach Gleicher,
Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May,
Konstantinos Aisopos, Léonard Hussenot, Livio~Baldini Soares, Kate Baumli,
Michael~B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip
Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan,
Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor~Campos
Campos, Alex Tomala, Yunhao Tang, Dalia~El Badawy, Elspeth White, Basil
Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi
Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec,
Ce~Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh,
James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring,
Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez,
Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald,
Keren Gu-Lemberg, Mina Khan, Lisa~Anne Hendricks, Marie Pellat, Vladimir
Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed~Hadi Hashemi,
Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le~Hou, Qingze
Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre
Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol
Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah
Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani,
Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot,
Matthew Lamm, Nicola De~Cao, Charlie Chen, Sidharth Mudgal, Romina Stella,
Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri,
Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer
Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad
Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kępa, Yomna Eldawy, Jiawern
Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain,
Sri Gayatri~Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie
Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu,
Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan
Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin,
Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules
Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan
Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido,
Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul~Rahman Mohammad, Josip
Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana
David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Älgmyr, Timothée Lottaz,
Qi~Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr
Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata
Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha
Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam~G. Rabinovitch, Pavan
Kumar~Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven,
Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan~Heimlich Shtacher, Shachi Paul,
Oscar Akerlund, François-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu,
Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana-Eleonora
Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max
Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam~Nguyen Thiet,
Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth
Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej, Vipul Ranjan,
Krzysztof Styrc, Pengcheng Yin, Jon Simon, Malcolm~Rose Harriott, Mudit
Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen
Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben
Horn, Assaf Israel, Francesco Pongetti, Chih-Wei~"Louis" Chen, Marco
Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev,
Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah~Ó
Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok,
Chenkai Kuang, Yan Romanikhin, Mark Geller, Z.~J. Yan, Kane Jang, Cheng-Chun
Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan
Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey,
Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya~Pranavi Potharaju, Eileen
O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan
Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew
Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski,
Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri,
Preethi Lahoti, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini
Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis
Cedoz, Chenmei Li, Shiyuan Chen, Niccolò~Dal Santo, Siddharth Goyal, Jitesh
Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury,
Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas,
Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma,
Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu,
Aditya Barua, Colin Ji, Ji~Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan
Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojciech Rzadkowski,
Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou,
Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann,
Marissa Bredesen, Zifan Lin, John~Eric Hoffmann, Jonathan Lai, Raynald Chung,
Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes,
Héctor~Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella,
Liselotte Snijders, Michael Mandl, Ante Kärrman, Paweł Nowak, Xinyi Wu,
Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch
Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen,
Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz
Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard
Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan
Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang,
Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal
Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu
Wu, John Zhang, Piotr Stanczyk, Ye~Zhang, David Steiner, Subhajit Naskar,
Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume~Sanchez
Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino
Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit
Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens,
William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar,
Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm
Levskaya, Zhenkai Zhu, Peter Grabowski, Yu~Mao, Alberto Magni, Kaisheng Yao,
Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso
Castaño, Irene Giannoumis, Wooyeol Kim, Mikołaj Rybiński, Ashwin
Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi
Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana~Gage Wright, Yawen
Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu~Li, Chimezie
Iwuanyanwu, Lu~Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN,
Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng,
Elena~Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo
Penchev, Alena Repina, Xihui Wu, Tom van~der Weide, Priya Ponnapalli,
Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff
Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel
Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine
Lee, Shuyuan Zhang, Somer Greene, Duc~Dung Nguyen, Paula Kurylowicz, Cassidy
Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang,
Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi,
Orgad Keller, David Reid, Daniel Finchelstein, Maria~Abi Raad, Remi Crocker,
Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova,
Rémi Leblond, Shirley Chung, Harry Askham, Luis~C. Cobo, Kelvin Xu, Felix
Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin
Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark
Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid,
Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel~J. Mankowitz, Alex
Polozov, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan,
Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser~tan Girgin,
Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp,
Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek
Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig,
Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex
Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng,
Frederick Liu, Fan Yang, Rui Zhu, Tian~Huey Teh, Jason Sanmiya, Evgeny
Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting
Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert
Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu,
Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna
Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria,
Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang,
Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamaria-Fernandez, Sonam
Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee,
Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem,
Sho Arora, Christy Koh, Soheil~Hassas Yeganeh, Siim Põder, Mukarram Tariq,
Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu,
Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi,
Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton,
Vinod Koverkathu, Christopher~A. Choquette-Choo, Yunjie Li, T.~J. Lu, Abe
Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob
Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek,
Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik
Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément
Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh,
Praveen Srinivasan, Claudia van~der Salm, Andreas Fidjeland, Salvatore
Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucińska, David Bridson,
Dario de~Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex
Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia
Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia
Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir
Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt,
Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare,
Shreyas~Rammohan Belle, Lei Wang, Chetan Tekur, Mihir~Sanjay Kale, Jinliang
Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi~Sun, Yao Zhao, Stephan
Lee, Pandu Nayak, Doug Fritz, Manish~Reddy Vuyyuru, John Aslanides, Nidhi
Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate,
James Manyika, Keyvan Amiri, Yelin Kim, Xi~Xiong, Kai Kang, Florian Luisier,
Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang,
Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng
Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji~Liu, Honglong
Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof
Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger,
Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark
Goldenson, Parashar Shah, M.~K. Blake, Hongkun Yu, Anthony Urbanowicz,
Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola
Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder,
Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake
Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana
Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua
Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya,
Emma Wang, Ke~Ye, Jean~Michel Sarr, Melanie~Moranski Preston, Madeleine
Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan
Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi
Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi
Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail
Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng
Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny
Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít
Listík, Mathias Carlen, Jan van~de Kerkhof, Marcin Pikus, Krunoslav Zaher,
Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph
Hirnschall, Ashwin Sethi, Xingyu~Federico Xu, Chetan Ahuja, Beth Tsai, Anca
Stefanoiu, Bo~Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva
Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar
Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova,
Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya~Krishna Lanka, Derik
Clive, Yi~Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind
Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael
Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan
Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi
Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu~Vu, Alek
Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie
Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor
Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals.
\newblock Gemini: {A} {Family} of {Highly} {Capable} {Multimodal} {Models},
June 2024{\natexlab{a}}.
\newblock URL \url{http://arxiv.org/abs/2312.11805}.
\newblock arXiv:2312.11805 [cs].
\bibitem[Team et~al.(2024{\natexlab{b}})Team, Georgiev, Lei, Burnell, Bai,
Gulati, Tanzer, Vincent, Pan, Wang, Mariooryad, Ding, Geng, Alcober, Frostig,
Omernick, Walker, Paduraru, Sorokin, Tacchetti, Gaffney, Daruki, Sercinoglu,
Gleicher, Love, Voigtlaender, Jain, Surita, Mohamed, Blevins, Ahn, Zhu,
Kawintiranon, Firat, Gu, Zhang, Rahtz, Faruqui, Clay, Gilmer, Co-Reyes,
Penchev, Zhu, Morioka, Hui, Haridasan, Campos, Mahdieh, Guo, Hassan, Kilgour,
Vezer, Cheng, de~Liedekerke, Goyal, Barham, Strouse, Noury, Adler,
Sundararajan, Vikram, Lepikhin, Paganini, Garcia, Yang, Valter, Trebacz,
Vodrahalli, Asawaroengchai, Ring, Kalb, Soares, Brahma, Steiner, Yu, Mentzer,
He, Gonzalez, Xu, Kaufman, Shafey, Oh, Hennigan, Driessche, Odoom, Lucic,
Roelofs, Lall, Marathe, Chan, Ontanon, He, Teplyashin, Lai, Crone, Damoc, Ho,
Riedel, Lenc, Yeh, Chowdhery, Xu, Kazemi, Amid, Petrushkina, Swersky,
Khodaei, Chen, Larkin, Pinto, Yan, Badia, Patil, Hansen, Orr, Arnold,
Grimstad, Dai, Douglas, Sinha, Yadav, Chen, Gribovskaya, Austin, Zhao, Patel,
Komarek, Austin, Borgeaud, Friso, Goyal, Caine, Cao, Chung, Lamm,
Barth-Maron, Kagohara, Olszewska, Chen, Shivakumar, Agarwal, Godhia, Rajwar,
Snaider, Dotiwalla, Liu, Barua, Ungureanu, Zhang, Batsaikhan, Wirth, Qin,
Danihelka, Doshi, Chadwick, Chen, Jain, Le, Kar, Gurumurthy, Li, Sang, Liu,
Lamprou, Munoz, Lintz, Mehta, Howard, Reynolds, Aroyo, Wang, Blanco,
Cassirer, Griffith, Das, Lee, Sygnowski, Fisher, Besley, Powell, Ahmed,
Paulus, Reitter, Borsos, Joshi, Pope, Hand, Selo, Jain, Sethi, Goel, Makino,
May, Yang, Schalkwyk, Butterfield, Hauth, Goldin, Hawkins, Senter, Brin,
Woodman, Ritter, Noland, Giang, Bolina, Lee, Blyth, Mackinnon, Reid, Sarvana,
Silver, Chen, Wang, Maggiore, Chang, Attaluri, Thornton, Chiu, Bunyan,
Levine, Chung, Eltyshev, Si, Lillicrap, Brady, Aggarwal, Wu, Xu, McIlroy,
Badola, Sandhu, Moreira, Stokowiec, Hemsley, Li, Tudor, Shyam, Rahimtoroghi,
Haykal, Sprechmann, Zhou, Mincu, Li, Addanki, Krishna, Wu, Frechette, Eyal,
Dafoe, Lacey, Whang, Avrahami, Zhang, Taropa, Lin, Toyama, Rutherford, Sano,
Choe, Tomala, Safranek-Shrader, Kassner, Pajarskas, Harvey, Sechrist,
Fortunato, Lyu, Elsayed, Kuang, Lottes, Chu, Jia, Chen, Humphreys, Baumli,
Tao, Samuel, Santos, Andreassen, Rakićević, Grewe, Kumar, Winkler, Caton,
Brock, Dalmia, Sheahan, Barr, Miao, Natsev, Devlin, Behbahani, Prost, Sun,
Myaskovsky, Pillai, Hurt, Lazaridou, Xiong, Zheng, Pardo, Li, Horgan,
Stanton, Ambar, Xia, Lince, Wang, Mustafa, Webson, Lee, Anil, Wicke, Dozat,
Sinha, Piqueras, Dabir, Upadhyay, Boral, Hendricks, Fry, Djolonga, Su,
Walker, Labanowski, Huang, Misra, Chen, Skerry-Ryan, Singh, Rijhwani, Yu,
Castro-Ros, Changpinyo, Datta, Bagri, Hrafnkelsson, Maggioni, Zheng, Sulsky,
Hou, Paine, Yang, Riesa, Rogozinska, Marcus, Badawy, Zhang, Wang, Miller,
Greer, Sjos, Nova, Zen, Chaabouni, Rosca, Jiang, Chen, Liu, Sainath, Krikun,
Polozov, Lespiau, Newlan, Cankara, Kwak, Xu, Chen, Coenen, Meyer, Tsihlas,
Ma, Gottweis, Xing, Gu, Miao, Frank, Cankara, Ganapathy, Dasgupta,
Hughes-Fitt, Chen, Reid, Rong, Fan, van Amersfoort, Zhuang, Cohen, Gu,
Mohananey, Ilic, Tobin, Wieting, Bortsova, Thacker, Wang, Caveness, Chiu,
Sezener, Kaskasoli, Baker, Millican, Elhawaty, Aisopos, Lebsack, Byrd, Dai,
Jia, Wiethoff, Davoodi, Weston, Yagati, Ahuja, Gao, Pundak, Zhang, Azzam,
Sim, Caelles, Keeling, Sharma, Swing, Li, Liu, Bostock, Bansal, Nado, Anand,
Lipschultz, Karmarkar, Proleev, Ittycheriah, Yeganeh, Polovets, Faust, Sun,
Rrustemi, Li, Shivanna, Liu, Welty, Lebron, Baddepudi, Krause, Parisotto,
Soricut, Xu, Bloxwich, Johnson, Neyshabur, Mao-Jones, Wang, Ramasesh, Abbas,
Guez, Segal, Nguyen, Svensson, Hou, York, Milan, Bridgers, Gworek,
Tagliasacchi, Lee-Thorp, Chang, Guseynov, Hartman, Kwong, Zhao, Kashem, Cole,
Miech, Tanburn, Phuong, Pavetic, Cevey, Comanescu, Ives, Yang, Du, Li, Zhang,
Iinuma, Hu, Roy, Bijwadia, Zhu, Martins, Saputro, Gergely, Zheng, Jia,
Antonoglou, Sadovsky, Gu, Bi, Andreev, Samangooei, Khan, Kocisky, Filos,
Kumar, Bishop, Yu, Hodkinson, Mittal, Shah, Moufarek, Cheng, Bloniarz, Lee,
Pejman, Michel, Spencer, Feinberg, Xiong, Savinov, Smith, Shakeri, Tran,
Chesus, Bohnet, Tucker, von Glehn, Muir, Mao, Kazawa, Slone, Soparkar,
Shrivastava, Cobon-Kerr, Sharman, Pavagadhi, Araya, Misiunas, Ghelani,
Laskin, Barker, Li, Briukhov, Houlsby, Glaese, Lakshminarayanan, Schucher,
Tang, Collins, Lim, Feng, Recasens, Lai, Magni, De~Cao, Siddhant, Ashwood,
Orbay, Dehghani, Brennan, He, Xu, Gao, Saroufim, Molloy, Wu, Arnold, Chang,
Schrittwieser, Buchatskaya, Radpour, Polacek, Giordano, Bapna, Tokumine,
Hellendoorn, Sottiaux, Cogan, Severyn, Saleh, Thakoor, Shefey, Qiao, Gaba,
Chang, Swanson, Zhang, Lee, Rubenstein, Song, Kwiatkowski, Koop, Kannan, Kao,
Schuh, Stjerngren, Ghiasi, Gibson, Vilnis, Yuan, Ferreira, Kamath, Klimenko,
Franko, Xiao, Bhattacharya, Patel, Wang, Morris, Strudel, Sharma, Choy,
Hashemi, Landon, Finkelstein, Jhakra, Frye, Barnes, Mauger, Daun, Baatarsukh,
Tung, Farhan, Michalewski, Viola, Quitry, Lan, Hudson, Wang, Fischer, Zheng,
White, Dragan, Alayrac, Ni, Pritzel, Iwanicki, Isard, Bulanova, Zilka, Dyer,
Sachan, Srinivasan, Muckenhirn, Cai, Mandhane, Tariq, Rae, Wang, Ayoub,
FitzGerald, Zhao, Han, Alberti, Garrette, Krishnakumar, Gimenez, Levskaya,
Sohn, Matak, Iturrate, Chang, Xiang, Cao, Ranka, Brown, Hutter, Mirrokni,
Chen, Yao, Egyed, Galilee, Liechty, Kallakuri, Palmer, Ghemawat, Liu, Tao,
Thornton, Green, Jasarevic, Lin, Cotruta, Tan, Fiedel, Yu, Chi, Neitz,
Heitkaemper, Sinha, Zhou, Sun, Kaed, Hulse, Mishra, Georgaki, Kudugunta,
Farabet, Shafran, Vlasic, Tsitsulin, Ananthanarayanan, Carin, Su, Sun, V,
Carvajal, Broder, Comsa, Repina, Wong, Chen, Hawkins, Filonov, Loher,
Hirnschall, Wang, Ye, Burns, Cate, Wright, Piccinini, Zhang, Lin, Gog,
Kulizhskaya, Sreevatsa, Song, Cobo, Iyer, Tekur, Garrido, Xiao, Kemp, Zheng,
Li, Agarwal, Ngani, Goshvadi, Santamaria-Fernandez, Fica, Chen, Gorgolewski,
Sun, Garg, Ye, Eslami, Hua, Simon, Joshi, Kim, Tenney, Potluri, Thiet, Yuan,
Luisier, Chronopoulou, Scellato, Srinivasan, Chen, Koverkathu, Dalibard, Xu,
Saeta, Anderson, Sellam, Fernando, Huot, Jung, Varadarajan, Quinn, Raul, Le,
Habalov, Clark, Jalan, Bullard, Singhal, Luong, Wang, Rajayogam, Eisenschlos,
Jia, Finchelstein, Yakubovich, Balle, Fink, Agarwal, Li, Dvijotham, Pal,
Kang, Konzelmann, Beattie, Dousse, Wu, Crocker, Elkind, Jonnalagadda, Lee,
Holtmann-Rice, Kallarackal, Liu, Vnukov, Vats, Invernizzi, Jafari, Zhou,
Taylor, Prendki, Wu, Eccles, Liu, Kopparapu, Beaufays, Angermueller, Marzoca,
Sarcar, Dib, Stanway, Perbet, Trdin, Sterneck, Khorlin, Li, Wu, Goenka,
Madras, Goldshtein, Gierke, Zhou, Liu, Liang, White, Li, Singh, Bahargam,
Epstein, Basu, Lao, Ozturel, Crous, Zhai, Lu, Tung, Gaur, Walton, Dixon,
Zhang, Globerson, Uy, Bolt, Wiles, Nasr, Shumailov, Selvi, Piccinno, Aguilar,
McCarthy, Khalman, Shukla, Galic, Carpenter, Villela, Zhang, Richardson,
Martens, Bosnjak, Belle, Seibert, Alnahlawi, McWilliams, Singh, Louis, Ding,
Popovici, Simicich, Knight, Mehta, Gupta, Shi, Fatehi, Mitrovic, Grills,
Pagadora, Munkhdalai, Petrova, Eisenbud, Zhang, Yates, Mittal, Tripuraneni,
Assael, Brovelli, Jain, Velimirovic, Akbulut, Mu, Macherey, Kumar, Xu,
Qureshi, Comanici, Wiesner, Gong, Ruddock, Bauer, Felt, GP, Arnab, Zelle,
Rothfuss, Rosgen, Shenoy, Seybold, Li, Mudigonda, Erdogan, Xia, Simsa, Michi,
Yao, Yew, Kan, Caswell, Radebaugh, Elisseeff, Valenzuela, McKinney, Paterson,
Cui, Latorre-Chimoto, Kim, Zeng, Durden, Ponnapalli, Sosea, Choquette-Choo,
Manyika, Robenek, Vashisht, Pereira, Lam, Velic, Owusu-Afriyie, Lee,
Bolukbasi, Parrish, Lu, Park, Venkatraman, Talbert, Rosique, Cheng,
Sozanschi, Paszke, Kumar, Austin, Li, Salama, Perz, Kim, Dukkipati,
Baryshnikov, Kaplanis, Sheng, Chervonyi, Unlu, Casas, Askham,
Tunyasuvunakool, Gimeno, Poder, Kwak, Miecnikowski, Mirrokni, Dimitriev,
Parisi, Liu, Tsai, Shevlane, Kouridi, Garmon, Goedeckemeyer, Brown,
Vijayakumar, Elqursh, Jazayeri, Huang, Carthy, Hoover, Kim, Kumar, Chen,
Biles, Bingham, Rosen, Wang, Tan, Engel, Pongetti, de~Cesare, Hwang, Yu,
Pullman, Narayanan, Levin, Gopal, Li, Aharoni, Trinh, Lo, Casagrande, Vij,
Matthey, Ramadhana, Matthews, Carey, Johnson, Goranova, Shah, Ashraf,
Dasgupta, Larsen, Wang, Vuyyuru, Jiang, Ijazi, Osawa, Smith, Boppana, Bilal,
Koizumi, Xu, Altun, Shabat, Bariach, Korchemniy, Choo, Ronneberger,
Iwuanyanwu, Zhao, Soergel, Hsieh, Cai, Iqbal, Sundermeyer, Chen, Bursztein,
Malaviya, Biadsy, Shroff, Dhillon, Latkar, Dyer, Forbes, Nicosia, Nikolaev,
Greene, Georgiev, Wang, Martin, Sedghi, Zhang, Banzal, Fritz, Rao, Wang,
Zhang, Patraucean, Du, Mordatch, Jurin, Liu, Dubey, Mohan, Nowakowski, Ion,
Wei, Tojo, Raad, Hudson, Keshava, Agrawal, Ramirez, Wu, Nguyen, Liu, Sewak,
Petrini, Choi, Philips, Wang, Bica, Garg, Wilkiewicz, Agrawal, Li, Guo, Xue,
Shaik, Leach, Khan, Wiesinger, Jerome, Chakladar, Wang, Ornduff, Abu,
Ghaffarkhah, Wainwright, Cortes, Liu, Maynez, Terzis, Samangouei, Mansour,
Kępa, Aubet, Algymr, Banica, Weisz, Orban, Senges, Andrejczuk, Geller,
Santo, Anklin, Merey, Baeuml, Strohman, Bai, Petrov, Wu, Hassabis,
Kavukcuoglu, Dean, and Vinyals]{gemini_team_gemini_2024-1}
Gemini Team, Petko Georgiev, Ving~Ian Lei, Ryan Burnell, Libin Bai, Anmol
Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh
Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark
Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti,
Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love,
Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins,
Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing
Zhang, Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, J.~D.
Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna
Haridasan, Victor Campos, Mahdis Mahdieh, Mandy Guo, Samer Hassan, Kevin
Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de~Liedekerke, Siddharth Goyal,
Paul Barham, D.~J. Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan,
Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang,
Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chulayuth Asawaroengchai, Roman
Ring, Norbert Kalb, Livio~Baldini Soares, Siddhartha Brahma, David Steiner,
Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael~Lopez
Kaufman, Laurent~El Shafey, Junhyuk Oh, Tom Hennigan, George van~den
Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe,
Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil
Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh,
Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia
Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario
Pinto, Geng Yan, Adria~Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave
Orr, Sebastien M.~R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas,
Rishika Sinha, Vikas Yadav, Xi~Chen, Elena Gribovskaya, Jacob Austin, Jeffrey
Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda
Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm,
Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik
Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider,
Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang,
Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi,
Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu
Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz,
Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan
Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan
Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali
Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan
Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel,
Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butterfield,
Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver
Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim
Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander
Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory
Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii
Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi
Wu, Yuanzhong Xu, Ross McIlroy, Kartikeya Badola, Paramjit Sandhu, Erica
Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam,
Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu,
Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan
Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye~Zhang, Emanuel
Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, HyunJeong
Choe, Alex Tomala, Chalence Safranek-Shrader, Nora Kassner, Mantas Pajarskas,
Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin
Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen,
Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero
Nogueira~dos Santos, Anders Andreassen, Nemanja Rakićević, Dominik Grewe,
Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia,
Hannah Sheahan, Iain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal
Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky,
Thanumalayan~Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi~Xiong,
Ce~Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei
Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee,
Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras,
Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa~Anne Hendricks, Corey Fry,
Josip Djolonga, Yi~Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant
Misra, Jeremy Chen, R.~J. Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu,
Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar~Mar
Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou,
Tom~Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus,
Dalia~El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars~Lowe
Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang,
Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov,
Jean-Baptiste Lespiau, Josh Newlan, Zeyncep Cankara, Soo Kwak, Yunhan Xu,
Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj
Gottweis, Jinwei Xing, Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara,
Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid,
Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen,
Shixiang~Shane Gu, Anhad Mohananey, Anastasija Ilic, Taylor Tobin, John
Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin
Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed
Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia,
Matthew Wiethoff, Elnaz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja,
Isabel Gao, Golan Pundak, Susan Zhang, Michael Azzam, Khe~Chai Sim, Sergi
Caelles, James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu,
Carrie~Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh
Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil~Hassas
Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li,
Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh
Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn
Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang,
Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Duc~Dung Nguyen,
James Svensson, Le~Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor
Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov,
Ale~Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth
Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien
Cevey, Ramona Comanescu, Richard Ives, Sherry Yang, Cosmo Du, Bo~Li, Zizhao
Zhang, Mariko Iinuma, Clara~Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu,
Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia,
Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina
Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton
Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre
Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, Paul Michel,
Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savinov, Charlotte
Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker,
Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar
Soparkar, Disha Shrivastava, James Cobon-Kerr, Michael Sharman, Jay
Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin,
David Barker, Qiujia Li, Anton Briukhov, Neil Houlsby, Mia Glaese, Balaji
Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim,
Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De~Cao,
Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan,
Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb
Arnold, Solomon Chang, Julian Schrittwieser, Elena Buchatskaya, Soroush
Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent
Hellendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad
Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo-yiin
Chang, Craig Swanson, Biao Zhang, Benjamin Lee, Paul~Kishan Rubenstein, Gan
Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel
Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye~Yuan, Felipe~Tiengo
Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro
Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek
Sharma, Peter Choy, Sayed~Hadi Hashemi, Jessica Landon, Mara Finkelstein,
Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen
Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Felix
de~Chaumont Quitry, Charline~Le Lan, Tom Hudson, Qingze Wang, Felix Fischer,
Ivy Zheng, Elspeth White, Anca Dragan, Jean-baptiste Alayrac, Eric Ni,
Alexander Pritzel, Adam Iwanicki, Michael Isard, Anna Bulanova, Lukas Zilka,
Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn,
Honglong Cai, Amol Mandhane, Mukarram Tariq, Jack~W. Rae, Gary Wang, Kareem
Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan
Garrette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn,
Josip Matak, Inaki Iturrate, Michael~B. Chang, Jackie Xiang, Yuan Cao,
Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen,
Kaisheng Yao, Zoltan Egyed, Francois Galilee, Tyler Liechty, Praveen
Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe
Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan,
Noah Fiedel, Hongkun Yu, Ed~Chi, Alexander Neitz, Jens Heitkaemper, Anu
Sinha, Denny Zhou, Yi~Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria
Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic,
Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun,
Shashank V, Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina,
William Wong, Warren~Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher,
Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate,
Diana~Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, Ionel Gog,
Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis~C. Cobo, Anand Iyer,
Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu~Steven
Zheng, Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca
Santamaria-Fernandez, Wojciech Fica, Xinyun Chen, Chris Gorgolewski, Sean
Sun, Roopal Garg, Xinyu Ye, S.~M.~Ali Eslami, Nan Hua, Jon Simon, Pratik
Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam~Nguyen Thiet, Quan Yuan,
Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen
Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu,
Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot,
Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan
Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang
Luong, Boyu Wang, Sujeevan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel
Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal,
Jing Li, Dj~Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer
Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind,
Siddhartha~Reddy Jonnalagadda, Jong Lee, Dan Holtmann-Rice, Krystal
Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen
Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles,
Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller,
Andreea Marzoca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Nejc
Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka,
David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie
Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein,
Sujoy Basu, Li~Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung,
Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant
Uy, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi,
Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal
Shukla, Vlado Galic, John Carpenter, Kevin Villela, Haibin Zhang, Harry
Richardson, James Martens, Matko Bosnjak, Shreyas~Rammohan Belle, Jeff
Seibert, Mahmoud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen
Ding, Dan Popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh
Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph
Pagadora, Tsendsuren Munkhdalai, Dessie Petrova, Danielle Eisenbud, Zhishuai
Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yannis Assael,
Thomas Brovelli, Prateek Jain, Mihajlo Velimirovic, Canfer Akbulut, Jiaqi Mu,
Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici,
Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt,
Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish
Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei
Xia, Jiri Simsa, Andrea Michi, Yi~Yao, Christopher Yew, Steven Kan, Isaac
Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney,
Kim Paterson, Albert Cui, Eri Latorre-Chimoto, Solomon Kim, William Zeng, Ken
Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher~A. Choquette-Choo, James
Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko
Velic, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish,
Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique,
Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin,
Lu~Li, Khalid Salama, Bartek Perz, Wooyeol Kim, Nandita Dukkipati, Anthony
Baryshnikov, Christos Kaplanis, XiangHai Sheng, Yuri Chervonyi, Caglar Unlu,
Diego de~Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim
Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron
Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon,
Adrian Goedeckemeyer, Adam~R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh
Jazayeri, Jin Huang, Sara~Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei
Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan,
David Engel, Francesco Pongetti, Dario de~Cesare, Dongseong Hwang, Lily Yu,
Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li,
Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic
Matthey, Bramandia Ramadhana, Austin Matthews, C.~J. Carey, Matthew Johnson,
Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus
Larsen, Yicheng Wang, Manish~Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki
Osawa, Celine Smith, Ramya~Sree Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu,
Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf
Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh,
Irene Cai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein,
Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi
Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer
Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang,
Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica
Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi
Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria~Abi Raad,
Drew~A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu,
Hoang Nguyen, Ji~Liu, Madhavi Sewak, Bryce Petrini, DongHyun Choi, Ivan
Philips, Ziyue Wang, Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka
Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach,
Sadh~MNM Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar,
Alek~Wenjiao Wang, Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus
Wainwright, Mario Cortes, Frederick Liu, Joshua Maynez, Andreas Terzis, Pouya
Samangouei, Riham Mansour, Tomasz Kępa, François-Xavier Aubet, Anton
Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa
Andrejczuk, Mark Geller, Niccolo~Dal Santo, Valentin Anklin, Majd~Al Merey,
Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis
Hassabis, Koray Kavukcuoglu, Jeff Dean, and Oriol Vinyals.
\newblock Gemini 1.5: {Unlocking} multimodal understanding across millions of
tokens of context, December 2024{\natexlab{b}}.
\newblock URL \url{http://arxiv.org/abs/2403.05530}.
\newblock arXiv:2403.05530 [cs].
\bibitem[Yang et~al.(2024)Yang, Walker, Parker-Holder, Du, Bruce, Barreto,
Abbeel, and Schuurmans]{yang_position_2024}
Sherry Yang, Jacob~C Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre
Barreto, Pieter Abbeel, and Dale Schuurmans.
\newblock Position: {Video} as the {New} {Language} for {Real}-{World}
{Decision} {Making}.
\newblock In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian
Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,
\emph{Proceedings of the 41st {International} {Conference} on {Machine}
{Learning}}, volume 235 of \emph{Proceedings of {Machine} {Learning}
{Research}}, pages 56465--56484. PMLR, July 2024.
\newblock URL \url{https://proceedings.mlr.press/v235/yang24z.html}.
\bibitem[McCarthy et~al.(2024)McCarthy, Tan, Schmidt, Acero, Herr, Du,
Thuruthel, and Li]{mccarthy_towards_2024}
Robert McCarthy, Daniel C.~H. Tan, Dominik Schmidt, Fernando Acero, Nathan
Herr, Yilun Du, Thomas~G. Thuruthel, and Zhibin Li.
\newblock Towards {Generalist} {Robot} {Learning} from {Internet} {Video}: {A}
{Survey}, June 2024.
\newblock URL \url{http://arxiv.org/abs/2404.19664}.
\newblock arXiv:2404.19664 [cs].
\bibitem[Pathak et~al.(2017)Pathak, Agrawal, Efros, and
Darrell]{pathak_curiosity-driven_2017}
Deepak Pathak, Pulkit Agrawal, Alexei~A. Efros, and Trevor Darrell.
\newblock Curiosity-driven {Exploration} by {Self}-supervised {Prediction}.
\newblock In Doina Precup and Yee~Whye Teh, editors, \emph{Proceedings of the
34th {International} {Conference} on {Machine} {Learning}}, volume~70 of
\emph{Proceedings of {Machine} {Learning} {Research}}, pages 2778--2787.
PMLR, August 2017.
\newblock URL \url{https://proceedings.mlr.press/v70/pathak17a.html}.
\bibitem[Sekar et~al.(2020)Sekar, Rybkin, Daniilidis, Abbeel, Hafner, and
Pathak]{sekar_planning_2020}
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner,
and Deepak Pathak.
\newblock Planning to explore via self-supervised world models.
\newblock In \emph{International {Conference} on {Machine} {Learning}}, pages
8583--8592. PMLR, 2020.
\bibitem[Rhinehart et~al.(2021)Rhinehart, Wang, Berseth, Co-Reyes, Hafner,
Finn, and Levine]{rhinehart_information_2021}
Nicholas Rhinehart, Jenny Wang, Glen Berseth, John Co-Reyes, Danijar Hafner,
Chelsea Finn, and Sergey Levine.
\newblock Information is {Power}: {Intrinsic} {Control} via {Information}
{Capture}.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}},
volume~34, pages 10745--10758. Curran Associates, Inc., 2021.
\newblock URL
\url{https://proceedings.neurips.cc/paper/2021/hash/59112692262234e3fad47fa8eabf03a4-Abstract.html}.
\bibitem[Houthooft et~al.(2016)Houthooft, Chen, Chen, Duan, Schulman, De~Turck,
and Abbeel]{houthooft_vime_2016}
Rein Houthooft, Xi~Chen, Xi~Chen, Yan Duan, John Schulman, Filip De~Turck, and
Pieter Abbeel.
\newblock {VIME}: {Variational} {Information} {Maximizing} {Exploration}.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}},
volume~29. Curran Associates, Inc., 2016.
\newblock URL
\url{https://papers.nips.cc/paper_files/paper/2016/hash/abd815286ba1007abfbb8415b83ae2cf-Abstract.html}.
\bibitem[Becker-Ehmck et~al.(2021)Becker-Ehmck, Karl, Peters, and
Smagt]{becker-ehmck_exploration_2021}
Philip Becker-Ehmck, Maximilian Karl, Jan Peters, and Patrick van~der Smagt.
\newblock Exploration via {Empowerment} {Gain}: {Combining} {Novelty},
{Surprise} and {Learning} {Progress}.
\newblock July 2021.
\newblock URL \url{https://openreview.net/forum?id=pmN5h1McUWV}.
\bibitem[Zhang et~al.(2021)Zhang, Wang, Hu, Chen, Chen, Fan, and
Zhang]{zhang_metacure_2021}
Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and
Chongjie Zhang.
\newblock {MetaCURE}: {Meta} {Reinforcement} {Learning} with
{Empowerment}-{Driven} {Exploration}.
\newblock In \emph{Proceedings of the 38th {International} {Conference} on
{Machine} {Learning}}, pages 12600--12610. PMLR, July 2021.
\newblock URL \url{https://proceedings.mlr.press/v139/zhang21w.html}.
\newblock ISSN: 2640-3498.
\bibitem[Sharma et~al.(2020)Sharma, Gu, Levine, Kumar, and
Hausman]{sharma_dynamics-aware_2020}
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman.
\newblock Dynamics-{Aware} {Unsupervised} {Discovery} of {Skills}.
\newblock In \emph{International {Conference} on {Learning} {Representations}},
2020.
\bibitem[Laskin et~al.()Laskin, Liu, Peng, Yarats, Rajeswaran, and
Abbeel]{laskin_contrastive_nodate}
Michael Laskin, Hao Liu, Xue~Bin Peng, Denis Yarats, Aravind Rajeswaran, and
Pieter Abbeel.
\newblock Contrastive {Intrinsic} {Control} for {Unsupervised} {Reinforcement}
{Learning}.
\bibitem[Eysenbach et~al.(2019)Eysenbach, Gupta, Ibarz, and
Levine]{eysenbach_diversity_2019}
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.
\newblock Diversity is {All} {You} {Need}: {Learning} {Skills} without a
{Reward} {Function}.
\newblock In \emph{International {Conference} on {Learning} {Representations}},
2019.
\newblock URL \url{https://openreview.net/forum?id=SJx63jRqFm}.
\bibitem[Eysenbach et~al.(2022)Eysenbach, Salakhutdinov, and
Levine]{eysenbach_information_2022}
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine.
\newblock The {Information} {Geometry} of {Unsupervised} {Reinforcement}
{Learning}.
\newblock In \emph{International {Conference} on {Learning} {Representations}},
2022.
\newblock URL \url{https://openreview.net/forum?id=3wU2UX0voE}.
\bibitem[Klyubin et~al.(2008)Klyubin, Polani, and Nehaniv]{klyubin_keep_2008}
Alexander~S. Klyubin, Daniel Polani, and Chrystopher~L. Nehaniv.
\newblock Keep {Your} {Options} {Open}: {An} {Information}-{Based} {Driving}
{Principle} for {Sensorimotor} {Systems}.
\newblock \emph{PLoS ONE}, 3\penalty0 (12):\penalty0 e4018, December 2008.
\newblock ISSN 1932-6203.
\newblock \doi{10.1371/journal.pone.0004018}.
\newblock URL \url{https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2607028/}.
\bibitem[Salge et~al.(2013)Salge, Glackin, and
Polani]{salge_approximation_2013}
Christoph Salge, Cornelius Glackin, and Daniel Polani.
\newblock Approximation of empowerment in the continuous domain.
\newblock \emph{Advances in Complex Systems}, 16\penalty0 (02n03):\penalty0
1250079, May 2013.
\newblock ISSN 0219-5259.
\newblock \doi{10.1142/S0219525912500798}.
\newblock URL
\url{https://www.worldscientific.com/doi/10.1142/S0219525912500798}.
\newblock Publisher: World Scientific Publishing Co.
\bibitem[Salge et~al.(2014{\natexlab{b}})Salge, Glackin, and
Polani]{salge_changing_2014}
Christoph Salge, Cornelius Glackin, and Daniel Polani.
\newblock Changing the {Environment} {Based} on {Empowerment} as {Intrinsic}
{Motivation}.
\newblock \emph{Entropy}, 16\penalty0 (5):\penalty0 2789--2819, May
2014{\natexlab{b}}.
\newblock ISSN 1099-4300.
\newblock \doi{10.3390/e16052789}.
\newblock URL \url{https://www.mdpi.com/1099-4300/16/5/2789}.
\newblock Number: 5 Publisher: Multidisciplinary Digital Publishing Institute.
\bibitem[Salge and Polani(2017)]{salge_empowerment_2017}
Christoph Salge and Daniel Polani.
\newblock Empowerment {As} {Replacement} for the {Three} {Laws} of {Robotics}.
\newblock \emph{Frontiers in Robotics and AI}, 4, 2017.
\newblock ISSN 2296-9144.
\newblock URL
\url{https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2017.00025}.
\bibitem[{Navneet Madhu
Kumar}(2018)]{navneet_madhu_kumar_empowerment-driven_2018}
{Navneet Madhu Kumar}.
\newblock Empowerment-driven {Exploration} using {Mutual} {Information}
{Estimation}.
\newblock In \emph{Bayesian {Deep} {Learning} {Workshop}}, December 2018.
\bibitem[Liu and Abbeel(2021{\natexlab{a}})]{liu_behavior_2021}
Hao Liu and Pieter Abbeel.
\newblock Behavior {From} the {Void}: {Unsupervised} {Active} {Pre}-{Training}.
\newblock In \emph{Advances in {Neural} {Information} {Processing} {Systems}},
volume~34, pages 18459--18473. Curran Associates, Inc., 2021{\natexlab{a}}.
\newblock URL
\url{https://proceedings.neurips.cc/paper/2021/hash/99bf3d153d4bf67d640051a1af322505-Abstract.html}.
\bibitem[Mendonca et~al.(2021)Mendonca, Rybkin, Daniilidis, Hafner, and
Pathak]{mendonca_discovering_2021}
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak
Pathak.
\newblock Discovering and {Achieving} {Goals} via {World} {Models}.
\newblock In A.~Beygelzimer, Y.~Dauphin, P.~Liang, and J.~Wortman Vaughan,
editors, \emph{Advances in {Neural} {Information} {Processing} {Systems}},
2021.
\newblock URL \url{https://openreview.net/forum?id=6vWuYzkp8d}.
\bibitem[Yarats et~al.(2021)Yarats, Fergus, Lazaric, and
Pinto]{yarats_reinforcement_2021}
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.
\newblock Reinforcement {Learning} with {Prototypical} {Representations}.
\newblock In \emph{Proceedings of the 38th {International} {Conference} on
{Machine} {Learning}}, pages 11920--11931. PMLR, July 2021.
\newblock URL \url{https://proceedings.mlr.press/v139/yarats21a.html}.
\newblock ISSN: 2640-3498.
\bibitem[Myers et~al.(2024)Myers, Ellis, Levine, Eysenbach, and
Dragan]{myers_learning_2024}
Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, and Anca Dragan.
\newblock Learning to {Assist} {Humans} without {Inferring} {Rewards}.
\newblock In \emph{The {Thirty}-eighth {Annual} {Conference} on {Neural}
{Information} {Processing} {Systems}}, 2024.
\newblock URL \url{https://openreview.net/forum?id=WCnJmb7cv1}.
\bibitem[Achiam et~al.(2018)Achiam, Edwards, Amodei, and
Abbeel]{achiam_variational_2018}
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel.
\newblock Variational {Option} {Discovery} {Algorithms}, July 2018.
\newblock URL \url{http://arxiv.org/abs/1807.10299}.
\newblock arXiv:1807.10299 [cs].
\bibitem[Gregor et~al.(2017)Gregor, Rezende, and
Wierstra]{gregor_variational_2017}
Karol Gregor, Danilo~Jimenez Rezende, and Daan Wierstra.
\newblock Variational {Intrinsic} {Control}.
\newblock February 2017.
\newblock URL \url{https://openreview.net/forum?id=Skc-Fo4Yg}.
\bibitem[Liu and Abbeel(2021{\natexlab{b}})]{liu_aps_2021}
Hao Liu and Pieter Abbeel.
\newblock {APS}: {Active} pretraining with successor features.
\newblock In \emph{International {Conference} on {Machine} {Learning}}, pages
6736--6747. PMLR, 2021{\natexlab{b}}.
\bibitem[Hansen et~al.(2020)Hansen, Dabney, Barreto, Warde-Farley, Wiele, and
Mnih]{hansen_fast_2020}
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van~de
Wiele, and Volodymyr Mnih.
\newblock Fast {Task} {Inference} with {Variational} {Intrinsic} {Successor}
{Features}.
\newblock In \emph{International {Conference} on {Learning} {Representations}},
2020.
\newblock URL \url{https://openreview.net/forum?id=BJeAHkrYDS}.
\bibitem[Park et~al.(2023)Park, Lee, Lee, and
Abbeel]{park_controllability-aware_2023}
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel.
\newblock Controllability-{Aware} {Unsupervised} {Skill} {Discovery}.
\newblock In \emph{Proceedings of the 40th {International} {Conference} on
{Machine} {Learning}}, pages 27225--27245. PMLR, July 2023.
\newblock URL \url{https://proceedings.mlr.press/v202/park23h.html}.
\newblock ISSN: 2640-3498.
\bibitem[Zheng et~al.(2025)Zheng, Tuyls, Peng, and Eysenbach]{zheng_can_2025}
Chongyi Zheng, Jens Tuyls, Joanne Peng, and Benjamin Eysenbach.
\newblock Can a {MISL} {Fly}? {Analysis} and {Ingredients} for {Mutual}
{Information} {Skill} {Learning}.
\newblock In \emph{The {Thirteenth} {International} {Conference} on {Learning}
{Representations}}, 2025.
\newblock URL \url{https://openreview.net/forum?id=xoIeVdFO7U}.
\bibitem[Mazzaglia et~al.(2022)Mazzaglia, Verbelen, Dhoedt, Lacoste, and
Rajeswar]{mazzaglia_choreographer_2022}
Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai
Rajeswar.
\newblock Choreographer: {Learning} and {Adapting} {Skills} in {Imagination}.
\newblock September 2022.
\newblock URL \url{https://openreview.net/forum?id=PhkWyijGi5b}.
\bibitem[Van Den~Oord et~al.(2017)Van Den~Oord, Vinyals, and
{others}]{van_den_oord_neural_2017}
Aaron Van Den~Oord, Oriol Vinyals, and {others}.
\newblock Neural discrete representation learning.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Finn et~al.(2017)Finn, Abbeel, and Levine]{finn_model-agnostic_2017}
Chelsea Finn, Pieter Abbeel, and Sergey Levine.
\newblock Model-{Agnostic} {Meta}-{Learning} for {Fast} {Adaptation} of {Deep}
{Networks}.
\newblock In \emph{Proceedings of the 34th {International} {Conference} on
{Machine} {Learning}}, pages 1126--1135. PMLR, July 2017.
\newblock URL \url{https://proceedings.mlr.press/v70/finn17a.html}.
\newblock ISSN: 2640-3498.
\bibitem[Wang et~al.(2016)Wang, Kurth-Nelson, Tirumala, Soyer, Leibo, Munos,
Blundell, Kumaran, and Botvinick]{wang_learning_2016}
Jane~X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel~Z. Leibo,
Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick.
\newblock Learning to reinforcement learn, November 2016.
\newblock URL \url{https://arxiv.org/abs/1611.05763v3}.
\bibitem[Duan et~al.(2016)Duan, Schulman, Chen, Bartlett, Sutskever, and
Abbeel]{duan_rl2_2016}
Yan Duan, John Schulman, Xi~Chen, Peter~L. Bartlett, Ilya Sutskever, and Pieter
Abbeel.
\newblock {RL}\${\textasciicircum}2\$: {Fast} {Reinforcement} {Learning} via
{Slow} {Reinforcement} {Learning}, November 2016.
\newblock URL \url{https://arxiv.org/abs/1611.02779v2}.
\bibitem[Zintgraf et~al.(2020)Zintgraf, Shiarlis, Igl, Schulze, Gal, Hofmann,
and Whiteson]{zintgraf_varibad_2020}
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin
Gal, Katja Hofmann, and Shimon Whiteson.
\newblock {VariBAD}: {A} {Very} {Good} {Method} for {Bayes}-{Adaptive} {Deep}
{RL} via {Meta}-{Learning}.
\newblock In \emph{International {Conference} on {Learning} {Representations}},
2020.
\newblock URL \url{https://openreview.net/forum?id=Hkl9JlBYvr}.
\bibitem[Cover and Thomas(2005)]{cover_elements_2005}
Thomas~M. Cover and Joy~A. Thomas.
\newblock \emph{Elements of {Information} {Theory}}.
\newblock Wiley, 1 edition, September 2005.
\newblock ISBN 978-0-471-24195-9 978-0-471-74882-3.
\newblock \doi{10.1002/047174882X}.
\newblock URL
\url{https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X}.
\bibitem[Blahut(1972)]{blahut_computation_1972}
R.~Blahut.
\newblock Computation of channel capacity and rate-distortion functions.
\newblock \emph{IEEE Transactions on Information Theory}, 18\penalty0
(4):\penalty0 460--473, July 1972.
\newblock ISSN 0018-9448.
\newblock \doi{10.1109/TIT.1972.1054855}.
\newblock URL \url{http://ieeexplore.ieee.org/document/1054855/}.
\bibitem[Arimoto(1972)]{arimoto_algorithm_1972}
S.~Arimoto.
\newblock An algorithm for computing the capacity of arbitrary discrete
memoryless channels.
\newblock \emph{IEEE Transactions on Information Theory}, 18\penalty0
(1):\penalty0 14--20, January 1972.
\newblock ISSN 0018-9448.
\newblock \doi{10.1109/TIT.1972.1054753}.
\newblock URL \url{http://ieeexplore.ieee.org/document/1054753/}.
\bibitem[Williams(1992)]{williams_simple_1992}
Ronald~J. Williams.
\newblock Simple statistical gradient-following algorithms for connectionist
reinforcement learning.
\newblock \emph{Machine Learning}, 8\penalty0 (3):\penalty0 229--256, May 1992.
\newblock ISSN 1573-0565.
\newblock \doi{10.1007/BF00992696}.
\newblock URL \url{https://doi.org/10.1007/BF00992696}.
\bibitem[Sutton and Barto(2018)]{sutton_reinforcement_2018}
Richard~S. Sutton and Andrew~G. Barto.
\newblock \emph{Reinforcement learning: an introduction}.
\newblock Adaptive computation and machine learning series. The MIT Press,
Cambridge, Massachusetts, second edition edition, 2018.
\newblock ISBN 978-0-262-03924-6.
\bibitem[Schulman et~al.(2017)Schulman, Wolski, Dhariwal, Radford, and
Klimov]{schulman_proximal_2017}
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
\newblock Proximal {Policy} {Optimization} {Algorithms}, August 2017.
\newblock URL \url{http://arxiv.org/abs/1707.06347}.
\newblock arXiv:1707.06347 [cs].
\bibitem[Schulman et~al.(2016)Schulman, Moritz, Levine, Jordan, and
Abbeel]{schulman_high-dimensional_2016}
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter
Abbeel.
\newblock High-{Dimensional} {Continuous} {Control} {Using} {Generalized}
{Advantage} {Estimation}.
\newblock In \emph{International {Conference} on {Learning} {Representations}},
2016.
\bibitem[Mnih et~al.(2015)Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare,
Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie, Sadik,
Antonoglou, King, Kumaran, Wierstra, Legg, and
Hassabis]{mnih_human-level_2015}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei~A. Rusu, Joel Veness,
Marc~G. Bellemare, Alex Graves, Martin Riedmiller, Andreas~K. Fidjeland,
Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis
Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and
Demis Hassabis.
\newblock Human-level control through deep reinforcement learning.
\newblock \emph{Nature}, 518\penalty0 (7540):\penalty0 529--533, February 2015.
\newblock ISSN 1476-4687.
\newblock \doi{10.1038/nature14236}.
\newblock URL \url{https://www.nature.com/articles/nature14236}.
\newblock Number: 7540 Publisher: Nature Publishing Group.
\bibitem[Uchendu et~al.(2023)Uchendu, Xiao, Lu, Zhu, Yan, Simon, Bennice, Fu,
Ma, Jiao, Levine, and Hausman]{uchendu_jump-start_2023}
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine
Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and
Karol Hausman.
\newblock Jump-{Start} {Reinforcement} {Learning}, July 2023.
\newblock URL \url{http://arxiv.org/abs/2204.02372}.
\newblock arXiv:2204.02372 [cs].
\bibitem[Kingma and Ba(2017)]{kingma_adam_2017}
Diederik~P. Kingma and Jimmy Ba.
\newblock Adam: {A} {Method} for {Stochastic} {Optimization}, January 2017.
\newblock URL \url{http://arxiv.org/abs/1412.6980}.
\newblock arXiv:1412.6980 [cs].
\bibitem[Bradbury et~al.(2018)Bradbury, Frostig, Hawkins, Johnson, Leary,
Maclaurin, Necula, Paszke, VanderPlas, Wanderman-Milne, and
Zhang]{bradbury_jax_2018}
James Bradbury, Roy Frostig, Peter Hawkins, Matthew~James Johnson, Chris Leary,
Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye
Wanderman-Milne, and Qiao Zhang.
\newblock {JAX}: composable transformations of {Python}+{NumPy} programs, 2018.
\newblock URL \url{http://github.com/jax-ml/jax}.
\bibitem[Raffin et~al.(2021)Raffin, Hill, Gleave, Kanervisto, Ernestus, and
Dormann]{raffin_stable-baselines3_2021}
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian
Ernestus, and Noah Dormann.
\newblock Stable-{Baselines3}: {Reliable} {Reinforcement} {Learning}
{Implementations}.
\newblock \emph{Journal of Machine Learning Research}, 22\penalty0
(268):\penalty0 1--8, 2021.
\newblock URL \url{http://jmlr.org/papers/v22/20-1364.html}.
\end{thebibliography}