Learning to Look: Seeking Information for Decision Making via Policy Factorization
Extracted source text · arXiv 2410.18964
\documentclass{article}
\usepackage[preprint]{corl_2024} %
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{subcaption}
\usepackage{lipsum}
\usepackage{enumitem}
\usepackage{twemojis}
\newcommand{\shivin}[1]{\textcolor{red}{shivin: #1}}
\newcommand{\roberto}[1]{\textcolor{blue}{[rmm: #1]}}
\newcommand{\commentp}[1]{\textcolor{blue}{[PS: #1]}}
\newcommand{\changes}[1]{\textcolor{blue}{#1}}
\usepackage{todonotes}
\usepackage{appendix}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{makecell}
\usepackage{wrapfig}
\usepackage{hyperref}
\newcommand{\methodname}{DISaM}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\title{Learning to Look \includegraphics[height=1.1em]{figures/eyes_image2.png}:
Seeking Information for Decision Making via Policy Factorization}
\author{
Shivin Dass$^{1}$, Jiaheng Hu$^{1}$, Ben Abbatematteo$^{1}$\\ \textbf{Peter Stone$^{1,2}$, Roberto Martín-Martín$^{1}$}\\
${}^{1}$The University of Texas at Austin, ${}^{2}$Sony AI\\
\texttt{\{sdass, jiahengh, abba, pstone, robertomm\}@utexas.edu}
}
\begin{document}
\maketitle
\renewcommand*{\thefootnote}{\fnsymbol{footnote}}
\begin{abstract}
Many robot tasks require active or interactive exploration behavior in order to be performed successfully. Such tasks are ubiquitous in embodied domains, where agents must actively search for the information necessary for each stage of a task, e.g., moving the head of the robot to find information relevant to manipulation, or in multi-robot domains, where one scout robot may search for the information that another robot needs to make informed decisions. We identify these tasks with a new type of problem, factorized Contextual Markov Decision Processes, and propose \methodname{}, a dual-policy solution composed of an information-seeking policy that explores the environment to find the relevant contextual information and an information-receiving policy that exploits the context to achieve the manipulation goal. This factorization allows us to train both policies separately, using the information-receiving one to provide reward to train the information-seeking policy. At test time, the dual agent balances exploration and exploitation based on the uncertainty the manipulation policy has on what the next best action is. We demonstrate the capabilities of our dual policy solution in five manipulation tasks that require information-seeking behaviors, both in simulation and in the real-world, where \methodname{} significantly outperforms existing methods. More information at \href{https://robin-lab.cs.utexas.edu/learning2look/}{robin-lab.cs.utexas.edu/learning2look/}.
\end{abstract}
\keywords{Active Vision, Manipulation, Interactive Perception}
\input{sections/01_introduction}
\input{sections/02_rw}
\input{sections/03_prob_v2}
\input{sections/04_method}
\input{sections/05_experiments}
\input{sections/06_limitations}
\input{sections/07_conclusion}
\clearpage
\acknowledgments{
We thank Arpit Bahety and Ruchira Ray for their feedback on the manuscript.
This work took place at the Robot Interactive Intelligence Lab (RobIn) at UT Austin. RobIn is supported in part by DARPA TIAMAT program (HR0011-24-9-0428).
A part of this work also took place at the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (FAIN-2019844, NRT-2125858), ONR (N00014-18-2243), ARO (W911NF-23-2-0004), Lockheed Martin, and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
}
\bibliography{main} %
\input{sections/A_Appendix}
\end{document}
\section{Introduction}
\label{s:intro}
Intelligent decisions can only be made based on the right information. When operating in the environment, an intelligent agent actively seeks the information that enables it to select the right actions and proceeds with the task only when it is confident enough. For example, when following a video recipe, a chef would look at the TV to obtain information about the next ingredient to grasp, and later look at a timer to decide when to turn off the stove. In contrast, current learning robots assume that the information needed for manipulation is readily available in their sensor signals (e.g., from a stationary camera looking at a tabletop manipulation setting) or rely on a given low-dimensional state representation predefined by a human (e.g., object pose) that also has to provide the means for the robot to perceive it. In this work, our goal is to endow robots with the capabilities to learn to perform information-seeking actions to find the information that enables manipulation, using as supervision the quality of the informed actions and switching between active perception and manipulation only based on the uncertainty about what manipulation action should come next.
Performing actions to reveal information has been previously explored in the subfields of active and interactive perception.
In active perception~\cite{bajcsy1988active, aloimonos1988active, bajcsy2018revisiting}, an agent changes the parameters of its sensors (e.g., camera pose~\cite{pito1999solution, scott2003view, delmerico2018comparison} or parameters~\cite{tenenbaum1971accommodation, garvey1976perceptual, pahlavan1992head}) to infer information such as object pose, shape, or material. Interactive perception~\cite{bohg2017interactive} solutions go one step further and enable agents to change the state of the environment to create information-rich signals to perceive kinematics~\cite{katz2008manipulating, martin2022coupled}, material~\cite{sinapov2011interactive}, or other properties~\cite{metta2003early, kroemer2010combining, gadre2021act, jiang2024roboexp}. However, these solutions fully factorize the problem into perception and subsequent task-oriented manipulation, creating the need for a human to specify what information to gather. Belief space planning approaches~\cite{kaelbling1998planning, platt2010belief, kaelbling2013integrated} jointly reason about how actions of an agent reduce uncertainty, and how this, in turn, leads to better actuation. However, correctly predicting the information that future actions may reveal requires accurate forward and observation models that are generally not available, leading to plans that grow quickly in complexity with the number of actions and the intricacy of the state and sensor spaces. Instead of planning, other solutions to Partially Observable Markov Decision Processes (POMDPs)~\cite{monahan1982state} rely on reinforcement learning to train a unified agent with memory that first finds information and then acts based on it. However, finding such long sequences of actions is prohibitively challenging, especially with sparse task reward, thus limiting their applicability to simple settings~\cite{mirowski2016learning, mnih2016asynchronous, anderson2018vision}.
\begin{figure}[t]
\centering
\includegraphics[clip,width=1.0\textwidth]{figures/teaser_2.pdf}
\caption{\textbf{\methodname{} for tasks with information-seeking behavior. }To make the right decision in a task (e.g., what beverage to pick or in what dining set to place it), a robot may need to seek task-relevant information (the time of day to decide the beverage or the person at the table to choose where to place it). We formalize such information-seeking tasks as factorized contextual MDPs and solve them with a dual policy collaborative approach where an information-seeking policy ($\pi^\mathit{IS}$) takes active perception actions to search for the right contextual information, and an information-receiving policy ($\pi^\mathit{IR}$) consumes this retrieved context to select the right manipulation actions.}
\label{fig:teaser}
\vspace{-1em}
\end{figure}
Our main insight is to formulate these manipulation tasks with information-seeking behavior as Contextual Markov Decision Processes (CMDP)~\cite{hallak2015contextual} and solve them with two separate collaborative policies, one controlling an information-seeking (IS) agent and the other an information-receiving agent (IR). We present \methodname{}, a \textbf{D}ual \textbf{I}nformation-\textbf{S}eeking \textbf{a}nd \textbf{M}anipulation policy solution for CMDPs, where an IS agent acts to determine the context, while the IR agent acts based on it to accomplish an overall manipulation task.
The context represents the information that is unknown to the agent a priori and has to be discovered by the IS to enable IR to take the right subsequent actions. For instance, the context could be a task label represented either as a one-hot vector or a language instruction.
In \methodname{}, both agents can share the same embodiment, as in our cooking example above, or have separate embodiments, for instance when one agent is a scout and the other is an operator awaiting information, but in both cases, our policy factorization matches a natural factorization of the agents based on different action and/or observation spaces.
Two main benefits result from factorizing into collaborative IS and IR policies to solve the CMDP. First, if we assume that the correct context is given, we may render the IR-learning problem fully observable and solve it with a simpler solution like imitation learning (IL).
Second, the factorization transforms the original long-horizon problem into two shorter-horizon ones, providing an opportunity to exploit additional supervision at the interface; in \methodname{}, we use the IR policy to provide intrinsic reward for the IS policy, making the IS-learning problem a simple POMDP.
These design decisions, on top of making the training process more efficient and tractable, make \methodname{} compatible with a variety of existing recent IL models in robotics~\cite{brohan2022rt,brohan2023rt,padalkar2023open,team2024octo}.
Finally, at test time, \methodname{} can simply balance between the information-seeking and the manipulation behaviors (IS and IR policies) based on the uncertainty the IR policy has about the next action to take, using an ensemble architecture.
Our contributions include 1) a general formalism for tasks with information-seeking behavior as factorized CMDPs (fCMDPs), and 2) our proposed dual-policy solution for fCMDPs, \methodname{}. We carry out multiple experiments in simulation and on real robots, where we demonstrate the capabilities of \methodname{} to enable robots (single or multi-robot systems) to seek and use the right information for manipulation. Additionally we balance between exploring for context and exploiting it, even with multiple stages that require several seeking/acting behavior switching phases. Our proposed approach substantially outperforms the baselines in 3 simulation and 2 real world tasks.
\section{Related Work}
\textbf{POMDPs and CMDPs.}
POMDPs provide a principled framework for decision making under partial observability. They have been extensively studied in the robotics literature, with applications including localization, navigation, manipulation, and human-robot interaction~\cite{kaelbling1998planning, kurniawati2022partially, lauri2022partially}. The intractability of exactly solving POMDPs is well-established~\cite{smallwood1973optimal, littman1994memoryless}, necessitating approximation algorithms~\cite{jaakkola1994reinforcement, pineau2006anytime,ng2013pegasus}. Scaling POMDP planning to real-world settings with image observations remains a challenge~\cite{kurniawati2022partially, lauri2022partially, zheng2021multi, nguyen2023language}, and the assumption of known dynamics and observation models remains prohibitively restrictive in open world settings. Belief-space task-and-motion planning methods similarly require substantial engineering of predicates, skills, and perception routines~\cite{kaelbling2011hierarchical, garrett2020online}. Applying model-free RL with some notion of memory (e.g. recurrent neural networks) has achieved some success in partially-observable problems~\cite{mirowski2016learning, mnih2016asynchronous, anderson2018vision}, though these methods generally struggle in long-horizon, hard exploration, sparse reward settings like finding and exploiting task-oriented information in manipulation.
Contextual MDPs (CMDPs)~\cite{hallak2015contextual} or Latent MDPs~\cite{kwon2021rl} have been proposed as a special class of POMDPs in which the reward and transition dynamics are determined by a (typically unobserved) \textit{context}, effectively parameterizing a family of MDPs. This model captures the case in which latent variables change slowly over time or are constant within an episode, e.g., serving a user a web application~\cite{hallak2015contextual}. Theory about these problems has been developed in multi-task~\cite{taylor2009transfer, brunskill2013sample, sodhani2021multi}, model-based~\cite{sun2019model}, continual~\cite{sodhani2022block}, and inverse RL settings~\cite{belogolovsky2021inverse}, with applications ranging from medical decision making~\cite{steimle2021multi} to human-robot interaction~\cite{chen2023asking}. We build on this framework to develop a CMDP model of active perception in which one agent seeks information to determine the latent context and a second acts to maximize the context dependant reward. MPC dual-control methods~\cite{heirung2012towards, heirung2013mpc} follow a similar motivation, simultaneously optimizing for system identification and a control objective, but require knowledge of the dynamics of the system and designation of the uncertain parameters.
\textbf{Active and Interactive Perception.}
Active perception~\cite{bajcsy1988active, aloimonos1988active, bajcsy2018revisiting} refers to the ability of an agent to intelligently seek out informative observations. Example applications include next-best-view planning for 3D reconstruction~\cite{pito1999solution, scott2003view, delmerico2018comparison} and active perception for grasping~\cite{breyer2022closed, morrison2019multi, jauhri2023active}.
When the agent is also equipped with information-gathering manipulation capabilities, the problem is referred to as \textit{interactive perception}~\cite{bohg2017interactive}, which has been studied in the context of object segmentation~\cite{metta2003early,martin2016integrated,gadre2021act}, articulation model estimation~\cite{katz2008manipulating, sturm2010vision,martin2022coupled}, material property estimation~\cite{sinapov2011interactive}, grasping~\cite{kroemer2010combining, hsiao2009reactive, koval2016pre}, scene understanding~\cite{jiang2024roboexp}, and beyond. While these approaches demonstrate the necessity and complexity of information-seeking behaviors, they typically define a priori the information that must be recovered from the scene. In contrast, our goal is to enable agents to discover what needs to be inferred through reinforcement.
Active perception has also been considered in the reinforcement learning setting~\cite{whitehead1990active, whitehead1991learning, tang2024deep, liu2021decoupling, xielearning}. Similar to our factorization, \citet{liu2021decoupling} decouple exploration and exploitation in a meta-RL setting but assume that a single exploration phase is sufficient for gathering all task relevant information.
Some works learn to crop observations to select salient visual inputs~\cite{james2022q, james2022coarse, tai2023scone}.
Recent work has developed RL methods capable of jointly learning manipulation behaviors with active vision to achieve clear view of the manipulator~\cite{cheng2018reinforcement, zaky2020active, jangir2022look, lv2023sam, shang2024active, goransson2022deep, grimes2022learning}. For example, \citet{shang2024active} propose jointly training manipulation and sensory policies by developing an intrinsic reward based on action prediction, encouraging the camera to view the manipulator. Similarly, \citet{goransson2022deep} train active perception policies to predict sensor observations of the manipulator.
These works generally fail to enable sophisticated information gathering behaviors and instead focus on viewpoint selection to facilitate short-horizon manipulation tasks. In contrast, we seek to enable agents to search for task-relevant information, decoupling information seeking from manipulation.
\section{Problem Formulation: factorized Contextual Markov Decision Processes}
\label{s:pf}
We formulate the problem of solving a manipulation task with information-seeking behavior as a \textit{Contextual Markov Decision Process (CMDP)}, a special form of MDP where the dynamics and the reward depend on a hidden contextual variable~\cite{hallak2015contextual,sun2019model,belogolovsky2021inverse}.
A CMDP is represented by the tuple $\mathcal{M}_C=(C, S, A, O, \mathcal{M}(c))$, where $c \in C$ is the set of contexts, $s \in S$ is the set of states, $a \in A$ is the set of actions
, $o\in O$ is the set of observations
, and $\mathcal{M}(c)$ is a function that maps the context to an MDP
$\mathcal{M}(c)=(S, A, O, T_c, R_c, \rho_c)$, where $T_c: S \times A \times S \rightarrow [0, 1]$ is the transition function, $R_c: S \times A \rightarrow \mathbb{R}$ is an overall task-reward function, and $\rho_c$ is the initial state distribution, all dependent on the context.
The goal is to find the policy, $\pi_c: S\rightarrow A$, that maximizes the expected total discounted sum of rewards, $\mathbb{E}_c \left[ \sum_{t=0}^{\infty} \gamma^t R_c(s_t, a_t) \mid s_0 \right]$.
Naively training a single policy to solve the CMDP is challenging and usually requires a complex iterative procedure~\cite{hallak2015contextual,rezaei2023hypernetworks}.
Instead, we make the important observation that in many practical CMDP problems, including robotic manipulation, the action space of the original agent can be naturally factored into two parts, $A=A_{\mathit{IS}}\cup A_{\mathit{IR}}$, where one part of the action space $A_{\mathit{IS}}$ is only useful for context seeking (e.g. a moving camera) while the other part $A_{\mathit{IR}}$ is sufficient to complete the task given the right context (e.g. a robot manipulator).
Given this assumption, we propose to factorize the CMDP into a factorized CMDP (fCMDP) with two sub-problems that can be addressed with two collaborative policies trained separately: an \textit{information-seeking (IS)} policy, $\pi^\mathit{IS}: O_{\mathit{IS}}\rightarrow A_{\mathit{IS}}$, that searches and provides context information $c_\mathit{IS} \in C$ to a second policy, an \textit{information-receiving (IR)} policy, $\pi^\mathit{IR}: O_{\mathit{IR}} \times C \rightarrow A_{\mathit{IR}}$, that consumes the context and takes reward-seeking actions based on it.
Specifically, for each IS observation $O_{\mathit{IS}}$, the information is generated by a given or learned function, $f_c: O_{\mathit{IS}} \rightarrow C$.
As different information may be required across the span of a task, these policies would operate iteratively, where $\pi^\mathit{IS}$ continuously searches for information that allows $\pi^\mathit{IR}$ to make the right decision. We provide further details about the formulation in Appendix~\ref{app:problem}.
Our proposed factorization of the CMDP problem provides new opportunities that our proposed method, \methodname{}, exploits. In the following we explain how \methodname{} trains a solution for the proposed fCMDP, focusing especially on the less explored problem of how to train collaborative information-seeking policies.
\section{\methodname{}, Dual Information-Seeking and Manipulation Policies for fCMDP}
\label{s:method}
In \methodname{}, we model a partially-observed manipulation problem as an fCMDP, and devise a solution to train a policy for each of the factorized subproblems: an information-seeking policy, $\pi^\mathit{IS}_{\theta}$ and an information-receiving policy, $\pi^\mathit{IR}$.
Additionally, we do not assume that $f_c$, the function mapping observations from the IS policy into context for the IR policy, is given; instead, we learn this mapping from IS observations into context, $E_{\phi}: O_\mathit{IS}\rightarrow C$.
\methodname{} utilizes a novel optimization procedure that first trains the IR agent on the original task with the reward of the fCMDP assuming access to the true context, and then trains the information-seeking agent to infer the context for the IR agent. To that end, it minimizes a loss based on the difference between the IR policy conditioned on the IS-provided context vs. ground truth context
\footnote{In general, actions by the IR policy based on the ground truth context may not lead to the highest task return (e.g., if the IR policy is far from optimal); in those cases, it would be better to directly optimize IS based on task reward. However, we assume that IR policy is a good approximation of the optimal policy once trained.}.
In the following, we first explain how the IR agent is instantiated~(Sec.~\ref{method:acting}); then, we discuss our novel iterative optimization procedure that utilizes intrinsic rewards generated by the IR agent to train the information-seeking agent~(Sec.~\ref{method:looking}); and finally, we provide details on how the uncertainty of the IR policy is used in \methodname{} to determine, in deployment, when the collaborative dual agent needs more information to complete the current task, querying the information-seeking agent, or when it is ready to pursue the task objective via the IR agent~(Sec.~\ref{method:full}).
\subsection{Information-Receiving Agent}
\label{method:acting}
The IR policy takes the current observation and controls the IR agent to interact with the environment, conditioned on the context.
We assume that only the actions of the IR agent can achieve the original manipulation goal.
In \methodname{}, we exploit the factorization in the fCMPD and assume we can train the IR policy with access to the ground truth context: $\pi^\mathit{IR}: O_\mathit{IR} \times C \rightarrow A_\mathit{IR}$. In this case, the IR agent is essentially operating in a standard MDP and can be optimized relatively easily using either imitation learning or reinforcement learning.
In this work we opt for imitation learning and collect a dataset of expert human demonstrations, $D=\{\tau_1, \tau_2...\tau_n\}$, that lead to the manipulation goal, labeled with ground truth context, $c$.
While in general CMDPs, the context determines the reward, transitions, and initial state distribution, in our experiments we focus on context-dependent rewards. Thus, the context provides information about the goal of the task, including relevant subgoals.
In principle, this context can be represented in any format as long as it can express multiple goals; we primarily consider task IDs, but also explore using natural language instructions as the context.
Given the dataset of expert demonstrations with ground truth context, we train $\pi^\mathit{IR}$ with behavior cloning (Fig.~\ref{fig:sysdiag}, Phase~1).
The observation space for the IR agent in our experiments, $o_\mathit{IR} \in O_\mathit{IR}$, consists of images of the robot's workspace acquired by a stationary camera, as well as robot proprioception.
Other forms of training the IR policy (e.g., with reinforcement learning) may be possible; we leave the exploration of these alternatives to future work.
Our simple IL solution to train the information-receiving policy aligns with multiple recent works~\cite{brohan2022rt,brohan2023rt,padalkar2023open,team2024octo} that provide trained goal-conditioned policies conditioned on task labels (IDs or language).
\begin{figure}[t]
\centering
\includegraphics[,clip,width=1.\textwidth]{figures/training_loop_3.pdf}
\caption{\textbf{Two learning stages of \methodname{}.} In Phase~1, we learn the information-receiving policy $\pi_\mathit{IR}$ that takes in ground-truth context information and controls the movement of the robot. In Phase~2, we learn an information-seeking policy $\pi_\mathit{IS}$ as well as an image encoder $E_\phi$ such that the context can be correctly reconstructed from the camera observation.
Once all parts are trained, together they create a system that takes in image observations and controls both the robot and the camera.
}
\label{fig:sysdiag}
\end{figure}
\subsection{Information-Seeking Agent}
\label{method:looking}
Once an IR policy, $\pi^\mathit{IR}$, is trained, \methodname{} uses it to train an information-seeking agent that includes the IS policy, $\pi^\mathit{IS}_{\theta}$, and an encoder mapping observations of the IS agent into context for the IR policy, $E_{\phi}$. The factorization in the fCMDP assumes that the actions from the IS agent alone are not able to directly achieve the original manipulation goal, rather they are sufficient only for determining the context required by the IR agent to solve the task.
The observations for the information-seeking agent in our experiments, $o_\mathit{IS} \in O_\mathit{IS}$, consist of images from the camera controlled by IS policy, the camera's pose, and, in the case of an interactive IS agent, proprioceptive signals from a robot arm.
The observation encoder, $E_{\phi}: O_\mathit{IS} \rightarrow C$, then converts the observations from the IS agent into contexts to be passed to the IR agent.
In cases where the IS agent may need to collect different context information depending on the IR agent's state, we augment the observations of the IS agent with observations from the IR agent, $O'_\mathit{IS} = O_\mathit{IS}\cup O_\mathit{IR}$.
A possible objective for the information-seeking agent is to maximize the original reward of the fCMDP, i.e., the original reward of the manipulation task that the IR policy optimizes.
However, the task reward may be sparse, posing significant challenges to the training process of the IS agent.
To address this, \methodname{} leverages a key insight: we can approximately optimize for the task reward by assuming that the IR policy makes the right decisions if provided with the right context.
By comparing the actions taken by the IR policy based on the IS-provided context, $a_\mathit{IR} \sim \pi^\mathit{IR}(o, c_\mathit{IS})$, with the IR policy actions based on the ground truth context (assumed to be provided during training), $a_\mathit{GT} \sim \pi^\mathit{IR}(o, c_\mathit{GT})$, we can train the IS policy to seek and encode the context information that minimizes the action differences at each time step.
Hence, the objective for IS agent is:
\begin{equation}
\label{eq:optim}
\argmin_{\theta, \phi} \mathbb{E}_{ \pi_{\theta}, \pi^\mathit{IR}} \ \mathcal{L}[\pi^\mathit{IR}(o_\mathit{IR}, E_{\phi}(o_\mathit{IS})), \pi^\mathit{IR}(o_\mathit{IR}, c_{GT})].
\end{equation}
\methodname{} uses the distance $\mathcal{L}$ between the two action distributions (e.g., cross entropy)
to define reward for the IS policy~$\pi^{IS}_{\theta}$, given as
$R_\mathit{IS}= \alpha-\beta\cdot\mathcal{L}(\pi^\mathit{IR}(o, c_\mathit{IS}), \pi^\mathit{IR}(o, c_\mathit{GT})))$, with hyperparameters $\alpha$ and $\beta$. This objective is denser than sparse task rewards and can be optimized efficiently with reinforcement learning~(Fig.~\ref{fig:sysdiag}, Step 2.2).
On the other hand, the ground truth context $c_\mathit{GT}$ also enables training the encoder, $E_{\phi}$, via supervised learning~(Fig.~\ref{fig:sysdiag}, Step 2.1).
We optimize these two objectives
iteratively, alternating between training $E_{\phi}$ while keeping $\pi^\mathit{IS}_{\theta}$ fixed, and vice versa, as visualized in Fig.~\ref{fig:sysdiag} and summarized in Algorithm~\ref{alg:dual_optimization} in Appendix~\ref{app:training}.
\subsection{Deciding between Information-Seeking and Information-Receiving Actions at Test Time}
\label{method:full}
\begin{wrapfigure}{r}{0.35\textwidth}
\centering
\vspace{-1em}
\includegraphics[,clip,width=0.35\textwidth]{figures/deployment.pdf}
\caption{\textbf{Deployment of \methodname{}.}
When uncertainty over IR's next action is low, \methodname{} follows the IR actions; when the uncertainty is high, \methodname{} follows IS policy.
}
\label{fig:deployment_fig}
\vspace{-1em}
\end{wrapfigure}
While the sections above describe the training objectives of the IS agent and the IR agent, we still need to specify a way to decide when to query the information-seeking agent and when to hand control back to the IR agent during deployment.
As shown in Fig.~\ref{fig:deployment_fig}, \methodname{} uses the uncertainty of the IR agent to determine when to act and when to seek new information. Specifically, \methodname{} maintains an ensemble of encoders $E_{\phi}(o^\mathit{IS}_{t})$. At each decision-making step, \methodname{} samples $n$ contexts $\{c_t^{i}\}_{i=1}^n$, from these encoders and conditions the IR agent on each of them to generate $n$ action distributions, $\{\pi^\mathit{IR}(o, c^{i}_\mathit{IS})\}_{i=1}^n$
. \methodname{} then computes the average KL-divergence between each pair of action distributions as a measure of uncertainty~\cite{menda2019ensembledagger}. If the value of uncertainty is higher than some threshold, $\delta$, \methodname{} assumes that the IR agent does not have sufficient context information to generate reliable actions, and it queries the information-seeking agent to seek further context information. When the uncertainty drops below $\delta$, it indicates that the IR policy is confident about the next manipulation action to take, and \methodname{} executes the IR policy until the uncertainty rises again. A pseudocode of \methodname{} full test-time system operation is provided in Appendix~\ref{app:testtime} with a discussion of the hyperparameter $\delta$.
\section{Experimental Evaluation}
\label{sec:experiment}
We evaluate our method on five distinct tasks in both simulation and the real world. The \texttt{Cooking}~(Fig.~\ref{fig:cooking}) task involves preparing and serving a side dish, with decisions based on the diner’s preference, the clock, and the serving area. In \texttt{Walls}~(Fig.~\ref{fig:walls}), a pick-and-place task, the block to be picked is identified by a hidden matching block, while the placement region is determined by another block on the right. The \texttt{Assembly}~(Fig.~\ref{fig:assembly}) task requires the IR agent to assemble nuts on pegs according to instructions found in a drawer.
In \texttt{Button}~(Fig.~\ref{fig:button}), the IS agent turns on a TV by pressing a red button, and the IR agent serves the next ingredient based on what appears on the screen.
Finally, in \texttt{Teatime}~(Fig.~\ref{fig:teatime}), the IS agent checks the time, selects the appropriate drink, and serves it to the person seated at the table. Further task details are provided in Appendix~\ref{app:task_details}.
We first train the IR agent on data collected in these environments using a behavior cloning objective. In the sim environments, the IR agent's action space consists of skills, and the IS agent's action space consists of camera pan/tilt actions as well as navigation actions in \texttt{Walls} and manipulation skills in \texttt{Assembly}.
In real world, we use transformer-based visuomotor policies~\cite{liu2024libero} as the IR policies and collect demonstrations using TeleMoMa~\cite{dass2024telemoma}.
Using the trained IR policies, we train the IS policy as described in Sec.~\ref{method:looking}. To improve the real world training time, we opt for a mixed sim and real training setup described in Appendix~\ref{app:domains}. The trained IS policy is then deployed in the environment to seek the correct information and provide it to the IR policy. The control between IR and IS is exchanged based on the test time protocol described in Sec.~\ref{method:full}.
We compare our method with the following baselines~(more details in Appendix~\ref{app:baselines}),
\begin{itemize}[leftmargin=*]
\item \texttt{\methodname{} (reward)}: An ablation of our method which uses task rewards to train the IS agent instead of our proposed intrinsic reward. From initial experimentation we found that sparse rewards are too difficult to learn from, so we provide hand designed rewards at the end of certain key stages.
\item \texttt{Full RL}: A reinforcement learning baseline that jointly optimizes the IR and IS agents using the task reward with PPO~\cite{schulman2017proximal}. We use stage rewards as described above to train this policy as well.
\item \texttt{Random Cam}: A test time algorithm where the IS agent performs random walks~(see Appendix~\ref{app:baselines}) and uses a pretrained encoder $E_{\phi}$ along with IR policy's uncertainty~(similar to our test time protocol) to determine when to hand control back to the IR policy.
\item \texttt{Sampled Context}: Providing IR with randomly sampled context vector from the set of contexts.
\end{itemize}
\input{figures/task_fig}
In our experimental evaluation, we aim to answer the following questions.
\textit{1) Can our proposed framework effectively learn to seek, infer, and exploit contextual information?}
In Fig.~\ref{fig:results} we summarize the results of our experiments and observe that \methodname{} significantly outperforms the baselines in all simulation tasks. Specifically, in longer horizon tasks that involve multiple stages --- \texttt{Cooking}, \texttt{Walls} and \texttt{Teatime} --- the baselines struggle whereas \methodname{} is able achieve high success rates. We provide the stage-wise evaluation results in Appendix~\ref{app:additional_exps}.
Qualitatively, we observe that \texttt{Full RL} uses the IS policy infrequently, leading to inadequate context recovery and suboptimal performance. \texttt{Random Cam} demonstrates reasonable performance but struggles with large action spaces and requires more time to obtain relevant observations. In contrast, \methodname{}, by separating information-seeking and manipulation, enables efficient context recovery and improved task completion.
We further discuss \methodname{}'s failure modes and robustness to noisy observations in Appendix~\ref{app:failure} and \ref{app:robustness} respectively.
\textit{2) How important is it to optimize the behavior cloning loss in \methodname{}'s objective instead of sparse task reward?}
We ablate \methodname{} by reinforcing the IS policy with the true stage rewards instead of our proposed intrinsic reward, shown by \texttt{\methodname{} (task reward)} in Fig.~\ref{fig:results}.
While learning directly from the task reward is challenging due to reward's sparsity, resulting in low success rates, we can see that \methodname{}'s performance is significantly boosted by the use of intrinsic reward.
\begin{figure}
\begin{minipage}[c]{0.685\textwidth}
\includegraphics[width=\textwidth]{figures/main_overall_tall.pdf}
\end{minipage}
\hfill
\begin{minipage}[c]{0.305\textwidth}
\caption{\textbf{Evaluation results.} The evaluations are performed across three seeds with 50 rollouts each in sim environment and 1 seed with 10 rollouts in the real world tasks.
Across all 5 tasks, \methodname{} significantly outperforms the baselines.
}
\label{fig:results}
\end{minipage}
\vspace{-2em}
\end{figure}
\textit{3) Can \methodname{} handle different representations of context to define the task?}
We compare \methodname{} with a variant, \texttt{\methodname{} (language)}, which uses CLIP embeddings of sentences as context vectors, rather than a onehot encoding. This is challenging since learning a mapping from observations to language embeddings is difficult for the observation encoder. We compare between the two in the \texttt{Cooking} and \texttt{Walls} task as shown in Fig.~\ref{fig:results}. In \texttt{Cooking}, surprisingly we observe only a small performance drop. \texttt{Walls}, however, is more challenging due to the IS agent needing to move the camera in order to see around the walls, and a modest drop in performance is observed. This agent still outperforms the baselines, suggesting that using language as context is possible in \methodname{}. We provide further details about the architecture and language instructions in Appendix~\ref{app:language}.
\def\taskFigHeight{0.7in}
\def\taskFigWidth{0.2\textwidth}
\begin{figure}[!t]
\captionsetup[subfigure]{aboveskip=1pt,belowskip=1pt, justification=centering}
\centering
\begin{subfigure}[t]{\taskFigWidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/tasks/kitchen.jpg}
\caption{\texttt{Cooking}}
\label{fig:cooking}
\end{subfigure}
\hfill
\begin{subfigure}[t]{\taskFigWidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/tasks/walls.jpg}
\caption{\texttt{Walls}}
\label{fig:walls}
\end{subfigure}
\hfill
\begin{subfigure}[t]{\taskFigWidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/tasks/assembly.jpg}
\caption{\texttt{Assembly}}
\label{fig:assembly}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.175\textwidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/tasks/button2.png}
\caption{\texttt{Button}}
\label{fig:button}
\end{subfigure}
\hfill
\begin{subfigure}[t]{\taskFigWidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/tasks/teatime.jpg}
\caption{\texttt{Teatime}}
\label{fig:teatime}
\end{subfigure}
\caption{\textbf{Tasks in our evaluation of \methodname{}}. We evaluated \methodname{} on 3 simulation tasks --- \texttt{Cooking}, \texttt{Walls}, \texttt{Assembly} --- and two real-world tasks with the Tiago robot --- \texttt{Button}, and \texttt{Teatime}. These tasks each require different information-gathering strategies, and demonstrate the sophisticated active and interactive information gathering capabilities of \methodname{}.}
\label{fig:tasks}
\end{figure}
\section{Limitations}
While our proposed framework provides a foundation for addressing complex information-seeking tasks, there are several parts of \methodname{} that we could improve in the future. First, \methodname{} does not employ observation history or memory in the IS agent, which could enable more complex information-gathering behaviors, especially the ones incorporating long-horizon temporal dependencies or perceiving not only static images but movements and behaviors of other agents. Second, while our fCMDP formalism does not require it, for optimal performance \methodname{} requires that there is minimal overlap between the action spaces of information-seeking and information-receiving agents. Otherwise one agent could counteract the other. This assumption prevents more subtle interactions between information seeking and acting, for example in mobile manipulators where motion of the base moves the onboard cameras and the end-effectors at the same time, and is a critical avenue for future work.
Third, even though \methodname{} uses intrinsic reward based on the action discrepancy loss to enhance the density of rewards, learning long horizon information-seeking behavior is still challenging, for instance \methodname{} requires long training times for long horizon tasks such as Walls (sim). We expect that using some form of exploration bonus can significantly improve the sample efficiency during training.
We plan to address the above challenges of \methodname{} in future work to handle more sophisticated problems in robotics that involve complex manipulation skills and information-seeking behaviors.
\section{Conclusion}
\label{s:conc}
We presented \methodname{}, a novel dual-policy solution for manipulation tasks with information-seeking behaviors. \methodname{} provides a solution with two policies for the two subproblems in factorized contextual MDP: 1) finding the context and 2) manipulating based on a given context. We exploit the factorization by training both policies consecutively, using simple IL for the manipulation policy that then use it to provide rewards to train the information-seeking policy. We demonstrated empirically the potential of \methodname{}'s dual solution to solve complex multi-stage problems alternating between information-seeking and manipulating phases based on the uncertainty of the next action, both in simulation and the real-world. We believe our proposed fCMDP formalism can be applied to other problems with partial observability requiring active perception (with other sensor modalities, with more agents); we plan to explore that next.
\clearpage
\newpage
\begin{appendices}
\renewcommand{\thesection}{\Alph{section}}
\section{Appendix}
\subsection{Factorized Contextual Markov Decision Process}
\label{app:problem}
As introduced in Sec.~\ref{s:pf}, we model our problem as a contextual Markov Decision Process (CMDP) and propose a novel factorization of it into two subproblems that can be addressed with two collaborative policies trained separately: an \textit{information-seeking (IS)} policy, $\pi_\mathit{IS}$, that searches and provides context to a second policy, an \textit{information-receiving (IR)} policy, $\pi_\mathit{IR}$, that consumes the context and takes reward-seeking (manipulation) actions based on it.
In the factorized CMDP (fCMDP), we assume that the trajectories generated by the optimal information-receiving policy, $[a^*(t=0),\ldots,a^*(T)]$, will achieve maximum return in the CMDP, $R_c$, due to the values in some of the dimensions of the action vector, independently of the values in others, and that these actions can be inferred from a subspace of the observation space.
Similarly, we assume that the trajectories generated by an optimal information-seeking policy will reveal the true context because of the values in some of the dimensions of the action vector, independently of the values in others, and that this true context can be inferred from a subset of the observation space.
We factorize the action and observation spaces of each agent in the fCMDP based on what action dimensions are necessary to control to achieve the task reward vs. to reveal information, such that:
$\pi^\mathit{IS}: O_{\mathit{IS}}\rightarrow A_{\mathit{IS}}$ and $\pi^\mathit{IR}: O_{\mathit{IR}} \times C \rightarrow A_{\mathit{IR}}$, with $O=O_{\mathit{IS}}\cup O_{\mathit{IR}}$.
Therefore, only IS actions can lead to IS observations with information to infer the right context, and IR actions can change the state toward the overall task goal.
The context can be inferred by the IS agent by mapping its observation(s) into a context with a given or learned function, $f_c: O_{\mathit{IS}} \rightarrow C$. However, for this function to map to the right context (i.e. the one that leads the IR policy to accumulate the highest return), the IS policy needs to take the right actions that reveal an information-rich observation.
There is no constraint in the overlap between action and observation spaces of both policies but our method performs best in cases where there is little or no overlap.
This factorized CMDP (fCMDP) matches a natural factorization into agents with different action and observation spaces, e.g., when the IS agent controls the head of a humanoid and the IR agent controls the arm and manipulates the environment, or when the IS agent is a navigating agent scouting the environment for an IR agent that waits for the contextual information to act in front of a table.
\subsection{IS Policy Training Procedure}
\label{app:training}
Alg.~\ref{alg:dual_optimization} includes the pseudo-code for training \methodname{}'s Information Seeking (IS) agent. Specifically, \methodname{} trains the IS agent by iterating between the IS policy optimization loop and the encoder optimization loop. The policy optimization uses on-policy data whereas the encoder optimization utilizes data sampled from a replay buffer aggregated during IS policy training. The IR agent takes action if and only if it can reconstruct the true IR actions based on the information $c_\mathit{IS}$ provided by the IS agent.
\input{figures/algo_dual_optim}
\subsection{\methodname{} Deployment Procedure}
\label{app:testtime}
Alg.~\ref{alg:deployment} includes the pseudo-code of the \methodname{}'s deployment solution.
During deployment, \methodname{} relies only on environmental observations to make decisions (no oracle or ground truth); the uncertainty of the IR policy is compared against a hyperparameter, $\delta$, to determine when to query the IS agent or when to execute IR's actions~(see Fig.~\ref{fig:deployment_fig}).
To compute the uncertainty, \methodname{} samples $n$ contexts $\{c_t^{i}\}_{i=1}^n$ from the ensemble of trained encoders (see Sec.~\ref{method:full}), $E_\phi$, and conditions the IR agent on each of them to generate $n$ action distributions, $\{\pi^\mathit{IR}(o, c^{i}_\mathit{IS})\}_{i=1}^n$. \methodname{} then computes the average KL-divergence between each pair of action distributions as a measure of uncertainty.
Due to the difference in the scale of the KL-divergence, the threshold $\delta$ needs to be adapted to each action space.
However, we found our method relatively robust to this parameter: we simply use $\delta = 0.5$ for all skill-based tasks (discrete action space), and $\delta = 1e5$ for visuomotor tasks (continuous action space), working well across different tasks.
\input{figures/algo_deployment}
\subsection{Using Language as Context}
\label{app:language}
In several of our tasks in simulation, the context is specified using a language instruction that specifies the goal for the task. Each task stage is specified with a different instruction; the set of possible language instructions for an entire task results from the Cartesian product of possible instruction for each stage.
Thus, the language instructions used in the \texttt{Cooking~(sim)} task include \{\textit{``Lift up the bread''}, \textit{``Grasp the meat''}\}$\times$\{\textit{``Cook for a short amount of time''}, \textit{``Cook until it is well-done''}\}$\times$\{\textit{``Place the pot on the red region''}, \textit{``Put the pot on the green area''}\}.
The instructions used in the \texttt{Walls~(sim)} task include \{\textit{``Pick up the blue cube''}, \textit{``Lift the wooden cube''}\}$\times$\{\textit{``Place the cube on the red region''}, \textit{``Put the cube on the green area''}\}.
When the environment is initialized we select the sentences that correspond to the correct instructions, concatenate them to form a single sentence, and process them with CLIP~\cite{radford2021clip} to generate a language feature that acts as contextual information about the goal of the task.
Since language embeddings are continuous vectors, we model them as Gaussian mixture models using Mixture Density Networks~(MDNs)~\cite{bishop1994mixture} as the prediction head for $E_\phi$ and use the negative log-likelihood loss to train $E_\phi$. This is different from how we model and train the encoder when contexts are one-hot vectors. In that case, while using the same backbone network of $E_\phi$ as language contexts, we replace the MDN head with a categorical distribution and train $E_\phi$ using the cross-entropy loss.
\subsection{Stage Wise Success Rates and Analysis}
\label{app:additional_exps}
In our experiments, three tasks consist of multiple stages leading to several phases of rising uncertainty, exploration with IS, and decreasing uncertainty, which are dynamically handled by our deployment procedure (Appendix~\ref{app:testtime}). Specifically, a stage may arise when the entire context cannot be reconstructed using a single information-seeking observation, $o_{IS}$, triggering a phase in the dual policy execution where the IS policy searches and provides the IR agent with the partial context necessary to complete the current stage before moving to the next one. For example, in the \texttt{Cooking~(sim)} task, the side dish to prepare, the cooking time, and the serving location are determined by the IS policy looking over to different locations: the dinner, the clock, and the serving region respectively; each of these parts is denoted a ``stage'' of the task.
In \texttt{Walls~(sim)}, the stages are picking and placing, with the block to pick determined by an identical block placed behind the wall on the left and the placing region determined by the color of the block on the right. Finally, in the \texttt{Teatime~(real)} task, the agent must look at the time to determine the appropriate drink, then the person to determine the serving location. Note that knowledge of these stages is not required by \methodname{}, which automatically infers the necessary information to collect at each stage through the intrinsic reward of minimizing the IR policy loss.
We evaluated \methodname{}, ablations, and baselines on each stage in isolation in addition to the overall task performance reported in the main text. Fig.~\ref{fig:multistage} shows the success rates for each method for each stage of the multi-stage task as well as the overall success rate. In the simulation tasks, we performed 50 policy rollouts per seed and calculated the success rate of each stage as (\# of stage successes)/(\# times stage reached) to avoid accumulating in the results the effect of the previous stages. In the real environment, 10 rollouts were performed by resetting the environment to the initial state for each stage and evaluating the agent. The methods that receive task reward information (\texttt{DISaM (task reward)} and \texttt{Full RL}) use a reward provided at the end of each stage rather than the sparse task reward.
\texttt{DISaM (language)} and \texttt{DISaM (task reward)} are generally competitive in the individual stages for \texttt{Cooking (sim)}, but the stage-wise failure rates compound resulting in lower overall success. On the more challenging \texttt{Walls (sim)} and \texttt{Teatime (real)}, the variants are less competitive in each stage. Across all tasks, the baselines \texttt{Full RL}, \texttt{Random Cam}, and \texttt{Sampled Context} generally perform poorly in all stages, with the exception of \texttt{Random Cam} succeeding in some of the simpler stages (e.g., \texttt{Cooking (sim)} stage A, \texttt{Walls (sim)} stage B). \texttt{Full RL} achieves consistent success only in the first stage of \texttt{Cooking (sim)} and \texttt{Walls (sim)}, perhaps due to experiencing these early stages more frequently during training.
\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/stages_plot.pdf}
\caption{\textbf{Stage-wise success rates.} Success rates for each method are plotted for each stage of the multi-stage tasks: \texttt{Cooking (sim)}, \texttt{Walls (sim)}, and \texttt{Teatime (real)}. }
\label{fig:multistage}
\end{figure}
\subsection{Real World Training leveraging Synced Simulation}
\label{app:domains}
Performing reinforcement learning training procedures in the real world is challenging. Among the known challenges~\cite{zhu2020ingredients}, two are significantly problematic: safety of the agent and the environment, and the need for constant resets (usually by a human).
In our real-world training procedure we propose solutions for these two challenges, leading to an autonomous and safe training process:
we developed a mixed sim-real training setup for training the IS policy and observation encoder using \methodname{} in real-world tasks.
Firstly, we create a sim version of the real-world task and collect data in both environments to train a corresponding IR policy. For IS policy training (see Fig.~\ref{fig:sysdiag} and Alg.~\ref{alg:iterative_optim}), we train the IS policy and encoder in the real world and generate the real-world context vectors but we exploit the simulator to interpret the context vector with the IR policy and provide rewards to the real IS policy. This eliminates the need to roll out the real IR policy, minimizing the resets and human supervision required during training. Once the IS policy is trained, we can then deploy it with the real IR policy following the system depicted in Fig.~\ref{fig:deployment_fig} and described in Alg.~\ref{alg:deployment}.
When the domains of the training and testing IR policy are different, we cannot use the IR observations to inform the IS policy decisions. One limitation arising from this is that we cannot do tasks requiring multiple stages: the IS policy cannot utilize the IR observations to learn which information is being queried. Hence, in the \texttt{Teatime~(real)} task, we separately train and evaluate the IS and IR policy in the two separate stages of the task, but consider them as a single task with two stages in our evaluations as they execute consecutively.
Another issue that results from training the IS policy in the real world is the constant need to update the information in the environment that represents context, e.g., the time on the clock or the person sitting next to the table. Since in this work, we focus on the visual cues to represent the context, we choose to portray this information on screens, which allows us to programmatically update the context altering easily the environment whenever the visual cue needs to be updated (as described in Appendix~\ref{app:task_details}).
While in the simulation we use separate cameras for the IR and IS agents, in the real-world tasks both agents share the same camera, mounted on the head of a PAL Tiago++~\cite{pages2016tiago}. We enable this by choosing a resting head position for the IR agent and resetting back to it whenever we switch to the IR policy deployment phase from the IS policy deployment phase. This is analogous to installing a second camera stationary pointing at the manipulation area. Additionally, during the IS deployment phase, we store the latest observation from the IR deployment phase and use it to calculate the uncertainty of the IR policy.
\subsection{Baselines}
\label{app:baselines}
Below we provide more information and implementation details of the baselines,
\texttt{\methodname{} (reward)}: An ablation of our method which uses task rewards to train the IS agent instead of our proposed intrinsic reward. This baseline is used to test if a high-performance IS policy can be learned just by using task rewards. Instead of using sparse task rewards, which are difficult to learn from, we use hand-designed rewards at the end of key stages. While this stage-wise reward requires some domain knowledge to design, we found it to be important for the learning of the IS policy. Specifically, after completing a stage the reward obtained is $+10$ in simulation environments and $+5$ in real-world settings. For all other timesteps, the reward is $-0.1$. Across all our experiments we use the same hyperparameters and network architectures as we used for \methodname{}.
\texttt{Full RL}: A reinforcement learning baseline that jointly optimizes the IR and IS agents. The policy architecture is a shallow convolutional neural network to process image inputs followed by an MLP, that takes observations of both IR and IS agents as inputs and outputs an action over the combined action space of the two agents. The policy is trained using PPO~\cite{schulman2017proximal} with the same stage-wise rewards as described above in the \texttt{\methodname{} (reward)} section.
\texttt{Random Cam}: A baseline that replaces the trained IS agent in \methodname{} with an agent that performs random movements but utilizes the trained encoder $E_{\phi}$ to encode the context and hands control back to the IR policy when it's uncertainty is lower than the prefixed threshold $\delta$~(similar to our test time protocol). Instead of sampling actions uniformly and randomly at every step, we perform weighted sampling by putting more weight on the last action performed, enabling the policy to explore larger distances in the state space. Specifically, with probability $0.5+\frac{1}{n\_actions}$, the action of the policy stays the same as in the previous timestep and otherwise uniformly randomly sampled from among the remaining actions.
\texttt{Sampled Context}: We sample the context vector from a prior probability distribution over contexts and use it to produce IR actions using an IR policy trained with behavior cloning. In all our settings, the prior distribution consists of a uniform probability distribution over all possible contexts.
\subsection{Comparing Visuomotor and Skill-based IR policies}
The training procedure between visuomotor and skill-based IR policies is similar except for some minor differences that we note here. Primarily the distance metric $\mathcal{L}$ in Equation~\ref{eq:optim} varies between the the two IR policies. When the IR policy is skill-based, $\mathcal{L}$ is the cross-entropy loss between the action distribution induced by the true and the predicted contexts. When the IR policy is visuomotor, we measure the negative log-likelihood of sampled predicted actions under the distribution of true actions.
To demonstrate that the performance of \methodname{} is not affected by the type of IR policy used, we compare \methodname{}'s performance trained with a skill-based ~\texttt{\methodname{}~(skill)} and a visuomotor~\texttt{\methodname{}~(motor)} IR policy on the \texttt{Walls (sim)} task. \texttt{\methodname{}~(skill)} achieves an overall success rate of $47.67\pm0.58$ and \texttt{\methodname{}~(motor)} achieves an overall success of $47.33\pm0.58$, measured over 50 rollouts across 3 seeds. As we can see, the overall performance of the system stays consistent across both forms of the IR policy.
\subsection{Failure Modes of \methodname{}}
\label{app:failure}
In the following we discuss the failure modes of \methodname{}.
\begin{itemize}[leftmargin=*]
\item \textbf{Failure to observe the correct visual cue~(IS policy failure)}: A failure mode of the IS policy in which the policy fails to take correct sequence of actions to observe the required visual cue and hence may not be able to predict the context correctly. During evaluation, since we terminate the execution of the policy after a fixed maximum number of environment steps, this failure mode typically arises in tasks where the sequence of correct actions to be taken to observe the visual cue is large such as in the simulated Walls task.
\item \textbf{Failure to interpret the observation~(Image encoder failure)}: Once the IS policy is looking at the correct visual cue, the observation encoder must encode it so that it could be properly handled by the IR policy. If the encoder has not been trained sufficiently or has made some spurious correlations with some other parts of the environment then this failure could arise. This failure mode is rare since the observation encoder requires significantly less training samples than the IS policy to achieve adequate performance.
\item \textbf{Failure of the IR policy}: Even when the IS policy and the observation encoder have correctly identified the visual cue, the IR policy may fail during execution. This failure mode is typically observed in the real world setting where data collection is expensive and we could not collect sufficient data for the IR policy to always be successful when given the correct context.
\end{itemize}
\begin{figure}[!t]
\captionsetup[subfigure]{aboveskip=1pt,belowskip=1pt, justification=centering}
\begin{subfigure}[t]{0.33\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figures/sankey_cooking.png}
\caption{Cooking Failures}
\end{subfigure}
\centering
\begin{subfigure}[t]{0.33\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figures/sankey_walls.png}
\caption{Walls Failures}
\end{subfigure}
\begin{subfigure}[t]{0.3\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figures/sankey_teatime.png}
\caption{Teatime Failures}
\end{subfigure}
\caption{Distribution of failures for the multi-stage tasks (a)~Cooking (b)~Walls and (c)~Teatime.}
\label{fig:sankey}
\vspace{-1em}
\end{figure}
We visualize the distribution of failures of \methodname{} on the three multi-stage tasks \texttt{Cooking}, \texttt{Walls} and \texttt{Teatime} task in Fig.\ref{fig:sankey}. We note that IR policy failure is a common failure in the real world due to high data collection costs leading to limited availability of training data.
\subsection{Robustness to Noisy Observations}
\label{app:robustness}
\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/noise_plot.png}
\caption{The distribution of success and failure of \methodname{} with varying levels of observation noise. For low sensor noise, \methodname{} achieves high performance but as the sensor noise increases the failure to map images to correct contexts starts to dominate.}
\label{fig:noise}
\end{figure}
In simulation, once the policies have been fully trained (e.g., the IS policy), the full \methodname{} dual system performs with very high performance, close to 100\% success. As a proxy for the real world, we performed an additional experiment by adding varying levels of Gaussian noise to the observations of both the IS and the IR policy, representing sensor noise. We present results for the the 3 simulation tasks with varying levels of noise in Fig.\ref{fig:noise} where noise scale represents the standard deviation of the zero centered Gaussian noise.
We observe that \methodname{} remains robust with low levels of noise and that after sufficient increase, the most significant errors are caused by failure to encode the images into the correct context since the noisy images are out-of-distribution for the encoder and make the image encoders overly confident about specific contexts, thus misleading the IR policy to take incorrect actions.
\subsection{Tasks}
\label{app:task_details}
In Fig.\ref{fig:detailed_task} we visualize rollouts in all tasks. In the following we describe the observation and action spaces of the IR and IS agents in each task~(summarized in Table~\ref{tab:task_info}).
\input{figures/task_info}
\texttt{Walls~(sim)}: Shown in Fig.\ref{fig:detailed_task}a, the IR agent is required to pick a block and place it on one of the serving regions. The color of the block to pick is determined by an identical block placed behind a wall on the table on the left and the color of the block on the table on the right corresponds to the correct serving region. Here the IR agent is the robot hand that uses the images from the eye-in-hand camera along with it's proprioceptives as observations and outputs a skill ID to choose between pick, place and go-near actions. The IS policy observes the scene from a floating camera with discrete actions that allow the camera to pan left/right, tilt up/down, move forward/backward/left/right by some fixed amount.
\texttt{Assembly~(sim)}: Shown in Fig.\ref{fig:detailed_task}b, the IR agent is required to assemble the screw in the correct colored peg. The color of the correct peg is determined by a similarly colored block placed inside one of the drawers on the table on the right. Here the IR agent is the robot hand that uses the images from the eye-in-hand camera along with it's proprioceptives as observations and outputs a skill ID to chooses between pick-place and go-near actions. The IS policy observes the scene from a floating camera and also can control the second robot arm near the drawers to interact with them. The IS policy has a discrete action space that allows it to pan and tilt the camera, along with controlling the second robot arm to open the drawers and pick/place the blocks.
\input{figures/detailed_task_figure}
\texttt{Cooking~(sim)}: Shown in Fig.\ref{fig:detailed_task}e, the IR agent is required to cook a meal.
The proper meal to prepare is determined by the dish being prepared, the cook time depends on the timer and the serving region is determined by the check mark.
Here the IR agent is the robot hand that uses the images from the eye-in-hand camera along with it's proprioceptives as observations and outputs a skill ID to choose between pick-place and go-near actions. The IS policy observes the scene from a floating camera with discrete actions that allow the camera to pan left/right, and tilt up/down.
\texttt{Teatime~(real)}: Shown in Fig.\ref{fig:detailed_task}c, the IR agent needs to determine the time of day to decide on the beverage to serve and look at the person at the table to choose where to place the beverage.
Here the IR agent is a Tiago++ robot that uses the images from it's head camera along with it's proprioceptives as observations and outputs a continuous delta action in the Cartesian space to control it's hand. The IS policy observes the scene from Tiago's head camera as well~(see Appendix~\ref{app:domains}) with discrete actions that allow the camera to pan left/right, and tilt up/down.
\texttt{Button~(real)}: Shown in Fig.\ref{fig:detailed_task}d, the IR agent is required to place the correct fruit in the bowl depending on the recipe. The IS policy is required to first turn the monitor on by pressing a button and then look at the screen to determine what fruit to use.
Here the IR agent is a Tiago++ robot's right hand that uses the images from it's head camera along with it's proprioceptives as observations and outputs a continuous delta action in the Cartesian space to control the hand. The IS policy observes the scene from Tiago's head camera as well~(see Appendix~\ref{app:domains}) and in addition to controlling the head movement, also controls Tiago's left arm. The IS policy has a discrete action space that allows it to pan and tilt the camera, along with controlling the second robot arm to press at various positions on the table.
\end{appendices}
\begin{algorithm}[h!]
\caption{Iterative Optimization for the Information Seeking Agent}
\label{alg:dual_optimization}
\begin{algorithmic}[1]
\State Initialize
Information-seeking policy $\pi_{\theta}^\mathit{IS}$,
observation encoder $E_{\psi}$,
Encoder Replay Buffer $\mathcal{B_E}$,
Rollout Buffer $\mathcal{B}$,
switch threshold $\mathcal{T}$
\For{i in $1, 2,...,K$}
\State Empty $\mathcal{B}$
\State Recieve initial observations $o_\mathit{IS}, o_\mathit{IR}$
\For{i in $1, 2...,K_{\pi}$}
\State $o_\mathit{IS}, o_\mathit{IR} \leftarrow $ CollectRollout$(o_\mathit{IS}, o_\mathit{IR})$
\EndFor
\State Optimize $\pi_{\theta}^\mathit{IS}$ on $\mathcal{B}$ with PPO objective
\For{i in $1, 2...,K_{enc}$}
\State $o_\mathit{IS}, c_{GT}$ $\sim \mathcal{B_E}$ \Comment{Sample a batch from replay buffer}
\State UpdateEncoder($o_\mathit{IS}, c_{GT}$)
\EndFor
\EndFor
\\
\Procedure{CollectRollout}{$o_\mathit{IS}, o_\mathit{IR}$}
\State Take actions $a_\mathit{IS}\sim \pi_{\theta}^\mathit{IS}(o_\mathit{IS})$ and obtain $o_\mathit{IS}', c_{GT}$ from the environment
\State $c_\mathit{IS}$ $\leftarrow E_\psi (o_\mathit{IS}')$
\State $r = $max$(1 -$ LossFunc$(o_\mathit{IR}, c_\mathit{IS}, c_{GT}), -1)$
\State add ($o_\mathit{IS}, a_\mathit{IS}, r, o_\mathit{IS}')$ to $\mathcal{B}$
\State add ($o_\mathit{IS}', c_{GT})$ to $\mathcal{B_E}$
\While{LossFunc($o_\mathit{IR}, c_\mathit{IS}, c_{GT}) < \mathcal{T}$}
\State Take actions $a_\mathit{IR} \sim $$\pi^\mathit{IR} (o_\mathit{IR}, c_{GT})$ and obtain $o_\mathit{IR}$ from the environment
\EndWhile
\State return $o_\mathit{IS}'$, $o_\mathit{IR}$
\EndProcedure
\\
\Procedure{UpdateEncoder}{$o_\mathit{IS}, c_{GT}$}
\State $c_\mathit{IS}$ $\leftarrow E_\psi (o_\mathit{IS})$
\State $\psi = $argmin$_\psi$ EncoderLoss$(c_\mathit{IS}, c_{GT})$ \Comment{Optimize $E_\psi$ using gradient descent}
\EndProcedure
\\
\Procedure{LossFunc}{$o_\mathit{IR}, c_\mathit{IS}, c_{GT}$}
\State return Distance$[\pi^\mathit{IR}(o_\mathit{IR}, c_\mathit{IS}), \pi^\mathit{IR}(o_\mathit{IR}, c_{GT})]$
\EndProcedure
\end{algorithmic}
\label{alg:iterative_optim}
\end{algorithm}
\begin{algorithm}
\caption{\methodname{} Deployment}
\begin{algorithmic}[1]
\Procedure{DeployAgent}{$\pi_{\theta}^\mathit{IS}$, $\pi^\mathit{IR}$, $E_{\psi}$}
\State Recieve initial observations $o_\mathit{IS}, o_\mathit{IR}$
\Repeat
\State $\{c_t^{i}\}_{i=1}^n$ $\leftarrow E_\psi (o_\mathit{IS})$
\State $A_{\text{dist}}$ $ \leftarrow \{\pi^\mathit{IR}(o_\mathit{IR}, c^{i}_\mathit{IS})\}_{i=1}^n$
\State Uncertainty $u$ $\leftarrow$ PairwiseKL($A_{\text{dist}}$)
\If{$u < \delta $}
\State Take action $a_\mathit{IR} \sim A_{\text{dist}}$
\State Receive observations $o_\mathit{IS}, o_\mathit{IR}$
\Else
\State Take action $a_\mathit{IS} \sim \pi_{\theta}^\mathit{IS}(o_\mathit{IS})$
\State Receive observations $o_\mathit{IS}, o_\mathit{IR}$
\EndIf
\Until{episode done}
\EndProcedure
\end{algorithmic}
\label{alg:deployment}
\end{algorithm}
\begin{table}[t]
\vspace{0.2cm}
\centering
\resizebox{1 \textwidth}{!}{
\begin{tabular}{cccccc}
\toprule
& \textbf{Cooking} & \textbf{Walls} & \textbf{Assembly} & \textbf{Button} & \textbf{Teatime}\\
\midrule
\multirow{2}{*}{IR Actions} & Manipulation & Manipulation & Manipulation & 6D Cartesian & 6D Cartesian + \\
& Skills & Skills & Skills & Deltas & 3D Nav. Deltas \\
\midrule
\multirow{2}{*}{IS Actions} & \multirow{2}{*}{Cam Pan/Tilt} & Cam Pan/Tilt & Cam Pan/Tilt + & Cam Pan/Tilt + & \multirow{2}{*}{Cam Pan/Tilt} \\
& & and XY Nav. & Manip. Skills & Manip. Skills & \\
\midrule
\# Stages & 3 & 2 & 1 & 1 & 2 \\
\bottomrule
\end{tabular}
}
\vspace{1em}
\caption{\textbf{Task Summary.} The action space of the IR and IS agents as well the number of stages for each task in our experiments.}
\label{tab:task_info}
\end{table}
\def\taskFigHeight{1.5in}
\begin{figure}[ht!]
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}[t]{0.70\textwidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/detailed_tasks/detailed_walls.pdf}
\caption{\methodname{} rollouts on \texttt{Walls}.}
\end{subfigure}
\hspace{-3em}
\begin{subfigure}[t]{0.29\textwidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/detailed_tasks/detailed_assembly.pdf}
\caption{\methodname{} rollouts on \texttt{Assembly}.}
\end{subfigure}
\vspace{1em}
\begin{subfigure}[t]{0.70\textwidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/detailed_tasks/detailed_teatime.pdf}
\caption{\methodname{} rollouts on \texttt{Teatime}.}
\end{subfigure}
\hspace{-3em}
\begin{subfigure}[t]{0.29\textwidth}
\centering
\includegraphics[height=\taskFigHeight]{figures/detailed_tasks/detailed_button.pdf}
\caption{\methodname{} rollouts on \texttt{Button}.}
\end{subfigure}
\vspace{1em}
\begin{subfigure}[t]{\textwidth}
\centering
\includegraphics[height=2in]{figures/detailed_tasks/detailed_cooking.pdf}
\caption{\methodname{} rollouts on \texttt{Cooking}.}
\end{subfigure}
\caption{Here we demonstrate the rollouts of \methodname{} on the 5 tasks that we study. We show the uncertainty during the rollouts and 1)~Highlight IR observations that have high uncertainty, and 2)~Sequence of IS observations before finding information that reduce uncertainty.}
\label{fig:detailed_task}
\end{figure}
\begin{thebibliography}{82}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bajcsy(1988)]{bajcsy1988active}
R.~Bajcsy.
\newblock Active perception.
\newblock \emph{Proceedings of the IEEE}, 76\penalty0 (8):\penalty0 966--1005, 1988.
\bibitem[Aloimonos et~al.(1988)Aloimonos, Weiss, and Bandyopadhyay]{aloimonos1988active}
J.~Aloimonos, I.~Weiss, and A.~Bandyopadhyay.
\newblock Active vision.
\newblock \emph{International journal of computer vision}, 1:\penalty0 333--356, 1988.
\bibitem[Bajcsy et~al.(2018)Bajcsy, Aloimonos, and Tsotsos]{bajcsy2018revisiting}
R.~Bajcsy, Y.~Aloimonos, and J.~K. Tsotsos.
\newblock Revisiting active perception.
\newblock \emph{Autonomous Robots}, 42:\penalty0 177--196, 2018.
\bibitem[Pito(1999)]{pito1999solution}
R.~Pito.
\newblock A solution to the next best view problem for automated surface acquisition.
\newblock \emph{IEEE Transactions on pattern analysis and machine intelligence}, 21\penalty0 (10):\penalty0 1016--1030, 1999.
\bibitem[Scott et~al.(2003)Scott, Roth, and Rivest]{scott2003view}
W.~R. Scott, G.~Roth, and J.-F. Rivest.
\newblock View planning for automated three-dimensional object reconstruction and inspection.
\newblock \emph{ACM Computing Surveys (CSUR)}, 35\penalty0 (1):\penalty0 64--96, 2003.
\bibitem[Delmerico et~al.(2018)Delmerico, Isler, Sabzevari, and Scaramuzza]{delmerico2018comparison}
J.~Delmerico, S.~Isler, R.~Sabzevari, and D.~Scaramuzza.
\newblock A comparison of volumetric information gain metrics for active 3d object reconstruction.
\newblock \emph{Autonomous Robots}, 42\penalty0 (2):\penalty0 197--208, 2018.
\bibitem[Tenenbaum(1971)]{tenenbaum1971accommodation}
J.~M. Tenenbaum.
\newblock \emph{Accommodation in computer vision}.
\newblock Stanford University, 1971.
\bibitem[Garvey(1976)]{garvey1976perceptual}
T.~D. Garvey.
\newblock Perceptual strategies for purposive vision.
\newblock 1976.
\bibitem[Pahlavan and Eklundh(1992)]{pahlavan1992head}
K.~Pahlavan and J.-O. Eklundh.
\newblock A head-eye system—analysis and design.
\newblock \emph{CVGIP: Image Understanding}, 56\penalty0 (1):\penalty0 41--56, 1992.
\bibitem[Bohg et~al.(2017)Bohg, Hausman, Sankaran, Brock, Kragic, Schaal, and Sukhatme]{bohg2017interactive}
J.~Bohg, K.~Hausman, B.~Sankaran, O.~Brock, D.~Kragic, S.~Schaal, and G.~S. Sukhatme.
\newblock Interactive perception: Leveraging action in perception and perception in action.
\newblock \emph{IEEE Transactions on Robotics}, 33\penalty0 (6):\penalty0 1273--1291, 2017.
\bibitem[Katz and Brock(2008)]{katz2008manipulating}
D.~Katz and O.~Brock.
\newblock Manipulating articulated objects with interactive perception.
\newblock In \emph{2008 IEEE International Conference on Robotics and Automation}, pages 272--277. IEEE, 2008.
\bibitem[Mart{\'\i}n-Mart{\'\i}n and Brock(2022)]{martin2022coupled}
R.~Mart{\'\i}n-Mart{\'\i}n and O.~Brock.
\newblock Coupled recursive estimation for online interactive perception of articulated objects.
\newblock \emph{The International Journal of Robotics Research}, 41\penalty0 (8):\penalty0 741--777, 2022.
\bibitem[Sinapov et~al.(2011)Sinapov, Bergquist, Schenck, Ohiri, Griffith, and Stoytchev]{sinapov2011interactive}
J.~Sinapov, T.~Bergquist, C.~Schenck, U.~Ohiri, S.~Griffith, and A.~Stoytchev.
\newblock Interactive object recognition using proprioceptive and auditory feedback.
\newblock \emph{The International Journal of Robotics Research}, 30\penalty0 (10):\penalty0 1250--1262, 2011.
\bibitem[Metta and Fitzpatrick(2003)]{metta2003early}
G.~Metta and P.~Fitzpatrick.
\newblock Early integration of vision and manipulation.
\newblock \emph{Adaptive behavior}, 11\penalty0 (2):\penalty0 109--128, 2003.
\bibitem[Kroemer et~al.(2010)Kroemer, Detry, Piater, and Peters]{kroemer2010combining}
O.~B. Kroemer, R.~Detry, J.~Piater, and J.~Peters.
\newblock Combining active learning and reactive control for robot grasping.
\newblock \emph{Robotics and Autonomous systems}, 58\penalty0 (9):\penalty0 1105--1116, 2010.
\bibitem[Gadre et~al.(2021)Gadre, Ehsani, and Song]{gadre2021act}
S.~Y. Gadre, K.~Ehsani, and S.~Song.
\newblock Act the part: Learning interaction strategies for articulated object part discovery.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 15752--15761, 2021.
\bibitem[Jiang et~al.(2024)Jiang, Huang, Wu, Li, Garg, Nayyeri, Wang, and Li]{jiang2024roboexp}
H.~Jiang, B.~Huang, R.~Wu, Z.~Li, S.~Garg, H.~Nayyeri, S.~Wang, and Y.~Li.
\newblock Roboexp: Action-conditioned scene graph via interactive exploration for robotic manipulation.
\newblock \emph{arXiv preprint arXiv:2402.15487}, 2024.
\bibitem[Kaelbling et~al.(1998)Kaelbling, Littman, and Cassandra]{kaelbling1998planning}
L.~P. Kaelbling, M.~L. Littman, and A.~R. Cassandra.
\newblock Planning and acting in partially observable stochastic domains.
\newblock \emph{Artificial intelligence}, 101\penalty0 (1-2):\penalty0 99--134, 1998.
\bibitem[Platt~Jr et~al.(2010)Platt~Jr, Tedrake, Kaelbling, and Lozano-Perez]{platt2010belief}
R.~Platt~Jr, R.~Tedrake, L.~P. Kaelbling, and T.~Lozano-Perez.
\newblock Belief space planning assuming maximum likelihood observations.
\newblock In \emph{Robotics: Science and Systems}, volume~2, 2010.
\bibitem[Kaelbling and Lozano-P{\'e}rez(2013)]{kaelbling2013integrated}
L.~P. Kaelbling and T.~Lozano-P{\'e}rez.
\newblock Integrated task and motion planning in belief space.
\newblock \emph{The International Journal of Robotics Research}, 32\penalty0 (9-10):\penalty0 1194--1227, 2013.
\bibitem[Monahan(1982)]{monahan1982state}
G.~E. Monahan.
\newblock State of the art—a survey of partially observable markov decision processes: theory, models, and algorithms.
\newblock \emph{Management science}, 28\penalty0 (1):\penalty0 1--16, 1982.
\bibitem[Mirowski et~al.(2016)Mirowski, Pascanu, Viola, Soyer, Ballard, Banino, Denil, Goroshin, Sifre, Kavukcuoglu, et~al.]{mirowski2016learning}
P.~Mirowski, R.~Pascanu, F.~Viola, H.~Soyer, A.~J. Ballard, A.~Banino, M.~Denil, R.~Goroshin, L.~Sifre, K.~Kavukcuoglu, et~al.
\newblock Learning to navigate in complex environments.
\newblock \emph{arXiv preprint arXiv:1611.03673}, 2016.
\bibitem[Mnih et~al.(2016)Mnih, Badia, Mirza, Graves, Lillicrap, Harley, Silver, and Kavukcuoglu]{mnih2016asynchronous}
V.~Mnih, A.~P. Badia, M.~Mirza, A.~Graves, T.~Lillicrap, T.~Harley, D.~Silver, and K.~Kavukcuoglu.
\newblock Asynchronous methods for deep reinforcement learning.
\newblock In \emph{International conference on machine learning}, pages 1928--1937. PMLR, 2016.
\bibitem[Anderson et~al.(2018)Anderson, Wu, Teney, Bruce, Johnson, S{\"u}nderhauf, Reid, Gould, and Van Den~Hengel]{anderson2018vision}
P.~Anderson, Q.~Wu, D.~Teney, J.~Bruce, M.~Johnson, N.~S{\"u}nderhauf, I.~Reid, S.~Gould, and A.~Van Den~Hengel.
\newblock Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 3674--3683, 2018.
\bibitem[Hallak et~al.(2015)Hallak, Di~Castro, and Mannor]{hallak2015contextual}
A.~Hallak, D.~Di~Castro, and S.~Mannor.
\newblock Contextual markov decision processes.
\newblock \emph{arXiv preprint arXiv:1502.02259}, 2015.
\bibitem[Brohan et~al.(2022)Brohan, Brown, Carbajal, Chebotar, Dabis, Finn, Gopalakrishnan, Hausman, Herzog, Hsu, et~al.]{brohan2022rt}
A.~Brohan, N.~Brown, J.~Carbajal, Y.~Chebotar, J.~Dabis, C.~Finn, K.~Gopalakrishnan, K.~Hausman, A.~Herzog, J.~Hsu, et~al.
\newblock Rt-1: Robotics transformer for real-world control at scale.
\newblock \emph{arXiv preprint arXiv:2212.06817}, 2022.
\bibitem[Brohan et~al.(2023)Brohan, Brown, Carbajal, Chebotar, Chen, Choromanski, Ding, Driess, Dubey, Finn, et~al.]{brohan2023rt}
A.~Brohan, N.~Brown, J.~Carbajal, Y.~Chebotar, X.~Chen, K.~Choromanski, T.~Ding, D.~Driess, A.~Dubey, C.~Finn, et~al.
\newblock Rt-2: Vision-language-action models transfer web knowledge to robotic control.
\newblock \emph{arXiv preprint arXiv:2307.15818}, 2023.
\bibitem[Padalkar et~al.(2023)Padalkar, Pooley, Jain, Bewley, Herzog, Irpan, Khazatsky, Rai, Singh, Brohan, et~al.]{padalkar2023open}
A.~Padalkar, A.~Pooley, A.~Jain, A.~Bewley, A.~Herzog, A.~Irpan, A.~Khazatsky, A.~Rai, A.~Singh, A.~Brohan, et~al.
\newblock Open x-embodiment: Robotic learning datasets and rt-x models.
\newblock \emph{arXiv preprint arXiv:2310.08864}, 2023.
\bibitem[Team et~al.(2024)Team, Ghosh, Walke, Pertsch, Black, Mees, Dasari, Hejna, Kreiman, Xu, et~al.]{team2024octo}
O.~M. Team, D.~Ghosh, H.~Walke, K.~Pertsch, K.~Black, O.~Mees, S.~Dasari, J.~Hejna, T.~Kreiman, C.~Xu, et~al.
\newblock Octo: An open-source generalist robot policy.
\newblock \emph{arXiv preprint arXiv:2405.12213}, 2024.
\bibitem[Kurniawati(2022)]{kurniawati2022partially}
H.~Kurniawati.
\newblock Partially observable markov decision processes and robotics.
\newblock \emph{Annual Review of Control, Robotics, and Autonomous Systems}, 5:\penalty0 253--277, 2022.
\bibitem[Lauri et~al.(2022)Lauri, Hsu, and Pajarinen]{lauri2022partially}
M.~Lauri, D.~Hsu, and J.~Pajarinen.
\newblock Partially observable markov decision processes in robotics: A survey.
\newblock \emph{IEEE Transactions on Robotics}, 39\penalty0 (1):\penalty0 21--40, 2022.
\bibitem[Smallwood and Sondik(1973)]{smallwood1973optimal}
R.~D. Smallwood and E.~J. Sondik.
\newblock The optimal control of partially observable markov processes over a finite horizon.
\newblock \emph{Operations research}, 21\penalty0 (5):\penalty0 1071--1088, 1973.
\bibitem[Littman(1994)]{littman1994memoryless}
M.~L. Littman.
\newblock Memoryless policies: Theoretical limitations and practical results.
\newblock 1994.
\bibitem[Jaakkola et~al.(1994)Jaakkola, Singh, and Jordan]{jaakkola1994reinforcement}
T.~Jaakkola, S.~Singh, and M.~Jordan.
\newblock Reinforcement learning algorithm for partially observable markov decision problems.
\newblock \emph{Advances in neural information processing systems}, 7, 1994.
\bibitem[Pineau et~al.(2006)Pineau, Gordon, and Thrun]{pineau2006anytime}
J.~Pineau, G.~Gordon, and S.~Thrun.
\newblock Anytime point-based approximations for large pomdps.
\newblock \emph{Journal of Artificial Intelligence Research}, 27:\penalty0 335--380, 2006.
\bibitem[Ng and Jordan(2013)]{ng2013pegasus}
A.~Y. Ng and M.~I. Jordan.
\newblock Pegasus: A policy search method for large mdps and pomdps.
\newblock \emph{arXiv preprint arXiv:1301.3878}, 2013.
\bibitem[Zheng et~al.(2021)Zheng, Sung, Konidaris, and Tellex]{zheng2021multi}
K.~Zheng, Y.~Sung, G.~Konidaris, and S.~Tellex.
\newblock Multi-resolution pomdp planning for multi-object search in 3d.
\newblock In \emph{2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages 2022--2029. IEEE, 2021.
\bibitem[Nguyen et~al.(2023)Nguyen, Hrosinkov, Rosen, and Tellex]{nguyen2023language}
T.~Nguyen, V.~Hrosinkov, E.~Rosen, and S.~Tellex.
\newblock Language-conditioned observation models for visual object search.
\newblock In \emph{2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages 10894--10901. IEEE, 2023.
\bibitem[Kaelbling and Lozano-P{\'e}rez(2011)]{kaelbling2011hierarchical}
L.~P. Kaelbling and T.~Lozano-P{\'e}rez.
\newblock Hierarchical task and motion planning in the now.
\newblock In \emph{2011 IEEE International Conference on Robotics and Automation}, pages 1470--1477. IEEE, 2011.
\bibitem[Garrett et~al.(2020)Garrett, Paxton, Lozano-P{\'e}rez, Kaelbling, and Fox]{garrett2020online}
C.~R. Garrett, C.~Paxton, T.~Lozano-P{\'e}rez, L.~P. Kaelbling, and D.~Fox.
\newblock Online replanning in belief space for partially observable task and motion problems.
\newblock In \emph{2020 IEEE International Conference on Robotics and Automation (ICRA)}, pages 5678--5684. IEEE, 2020.
\bibitem[Kwon et~al.(2021)Kwon, Efroni, Caramanis, and Mannor]{kwon2021rl}
J.~Kwon, Y.~Efroni, C.~Caramanis, and S.~Mannor.
\newblock Rl for latent mdps: Regret guarantees and a lower bound.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 24523--24534, 2021.
\bibitem[Taylor and Stone(2009)]{taylor2009transfer}
M.~E. Taylor and P.~Stone.
\newblock Transfer learning for reinforcement learning domains: A survey.
\newblock \emph{Journal of Machine Learning Research}, 10\penalty0 (7), 2009.
\bibitem[Brunskill and Li(2013)]{brunskill2013sample}
E.~Brunskill and L.~Li.
\newblock Sample complexity of multi-task reinforcement learning.
\newblock \emph{arXiv preprint arXiv:1309.6821}, 2013.
\bibitem[Sodhani et~al.(2021)Sodhani, Zhang, and Pineau]{sodhani2021multi}
S.~Sodhani, A.~Zhang, and J.~Pineau.
\newblock Multi-task reinforcement learning with context-based representations.
\newblock In \emph{International Conference on Machine Learning}, pages 9767--9779. PMLR, 2021.
\bibitem[Sun et~al.(2019)Sun, Jiang, Krishnamurthy, Agarwal, and Langford]{sun2019model}
W.~Sun, N.~Jiang, A.~Krishnamurthy, A.~Agarwal, and J.~Langford.
\newblock Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches.
\newblock In \emph{Conference on learning theory}, pages 2898--2933. PMLR, 2019.
\bibitem[Sodhani et~al.(2022)Sodhani, Meier, Pineau, and Zhang]{sodhani2022block}
S.~Sodhani, F.~Meier, J.~Pineau, and A.~Zhang.
\newblock Block contextual mdps for continual learning.
\newblock In \emph{Learning for Dynamics and Control Conference}, pages 608--623. PMLR, 2022.
\bibitem[Belogolovsky et~al.(2021)Belogolovsky, Korsunsky, Mannor, Tessler, and Zahavy]{belogolovsky2021inverse}
S.~Belogolovsky, P.~Korsunsky, S.~Mannor, C.~Tessler, and T.~Zahavy.
\newblock Inverse reinforcement learning in contextual mdps.
\newblock \emph{Machine Learning}, 110\penalty0 (9):\penalty0 2295--2334, 2021.
\bibitem[Steimle et~al.(2021)Steimle, Kaufman, and Denton]{steimle2021multi}
L.~N. Steimle, D.~L. Kaufman, and B.~T. Denton.
\newblock Multi-model markov decision processes.
\newblock \emph{IISE Transactions}, 53\penalty0 (10):\penalty0 1124--1139, 2021.
\bibitem[Chen et~al.(2023)Chen, Zhang, Zhang, Zhao, and Chen]{chen2023asking}
X.~Chen, S.~Zhang, P.~Zhang, L.~Zhao, and J.~Chen.
\newblock Asking before action: Gather information in embodied decision making with language models.
\newblock \emph{arXiv preprint arXiv:2305.15695}, 2023.
\bibitem[Heirung et~al.(2012)Heirung, Ydstie, and Foss]{heirung2012towards}
T.~A.~N. Heirung, B.~E. Ydstie, and B.~Foss.
\newblock Towards dual mpc.
\newblock \emph{IFAC Proceedings Volumes}, 45\penalty0 (17):\penalty0 502--507, 2012.
\bibitem[Heirung et~al.(2013)Heirung, Ydstie, and Foss]{heirung2013mpc}
T.~A.~N. Heirung, B.~E. Ydstie, and B.~Foss.
\newblock An mpc approach to dual control.
\newblock \emph{IFAC Proceedings Volumes}, 46\penalty0 (32):\penalty0 69--74, 2013.
\bibitem[Breyer et~al.(2022)Breyer, Ott, Siegwart, and Chung]{breyer2022closed}
M.~Breyer, L.~Ott, R.~Siegwart, and J.~J. Chung.
\newblock Closed-loop next-best-view planning for target-driven grasping.
\newblock In \emph{2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages 1411--1416. IEEE, 2022.
\bibitem[Morrison et~al.(2019)Morrison, Corke, and Leitner]{morrison2019multi}
D.~Morrison, P.~Corke, and J.~Leitner.
\newblock Multi-view picking: Next-best-view reaching for improved grasping in clutter.
\newblock In \emph{2019 International Conference on Robotics and Automation (ICRA)}, pages 8762--8768. IEEE, 2019.
\bibitem[Jauhri et~al.(2023)Jauhri, Lueth, and Chalvatzaki]{jauhri2023active}
S.~Jauhri, S.~Lueth, and G.~Chalvatzaki.
\newblock Active-perceptive motion generation for mobile manipulation.
\newblock \emph{arXiv preprint arXiv:2310.00433}, 2023.
\bibitem[Mart{\'\i}n-Mart{\'\i}n et~al.(2016)Mart{\'\i}n-Mart{\'\i}n, H{\"o}fer, and Brock]{martin2016integrated}
R.~Mart{\'\i}n-Mart{\'\i}n, S.~H{\"o}fer, and O.~Brock.
\newblock An integrated approach to visual perception of articulated objects.
\newblock In \emph{2016 IEEE international conference on robotics and automation (ICRA)}, pages 5091--5097. IEEE, 2016.
\bibitem[Sturm et~al.(2010)Sturm, Konolige, Stachniss, and Burgard]{sturm2010vision}
J.~Sturm, K.~Konolige, C.~Stachniss, and W.~Burgard.
\newblock Vision-based detection for learning articulation models of cabinet doors and drawers in household environments.
\newblock In \emph{2010 IEEE International Conference on Robotics and Automation}, pages 362--368. IEEE, 2010.
\bibitem[Hsiao et~al.(2009)Hsiao, Nangeroni, Huber, Saxena, and Ng]{hsiao2009reactive}
K.~Hsiao, P.~Nangeroni, M.~Huber, A.~Saxena, and A.~Y. Ng.
\newblock Reactive grasping using optical proximity sensors.
\newblock In \emph{2009 IEEE International Conference on Robotics and Automation}, pages 2098--2105. IEEE, 2009.
\bibitem[Koval et~al.(2016)Koval, Pollard, and Srinivasa]{koval2016pre}
M.~C. Koval, N.~S. Pollard, and S.~S. Srinivasa.
\newblock Pre-and post-contact policy decomposition for planar contact manipulation under uncertainty.
\newblock \emph{The International Journal of Robotics Research}, 35\penalty0 (1-3):\penalty0 244--264, 2016.
\bibitem[Whitehead and Ballard(1990)]{whitehead1990active}
S.~D. Whitehead and D.~H. Ballard.
\newblock Active perception and reinforcement learning.
\newblock In \emph{Machine Learning Proceedings 1990}, pages 179--188. Elsevier, 1990.
\bibitem[Whitehead and Ballard(1991)]{whitehead1991learning}
S.~D. Whitehead and D.~H. Ballard.
\newblock Learning to perceive and act by trial and error.
\newblock \emph{Machine Learning}, 7:\penalty0 45--83, 1991.
\bibitem[Tang et~al.(2024)Tang, Abbatematteo, Hu, Chandra, Mart{\'\i}n-Mart{\'\i}n, and Stone]{tang2024deep}
C.~Tang, B.~Abbatematteo, J.~Hu, R.~Chandra, R.~Mart{\'\i}n-Mart{\'\i}n, and P.~Stone.
\newblock Deep reinforcement learning for robotics: A survey of real-world successes.
\newblock \emph{arXiv preprint arXiv:2408.03539}, 2024.
\bibitem[Liu et~al.(2021)Liu, Raghunathan, Liang, and Finn]{liu2021decoupling}
E.~Z. Liu, A.~Raghunathan, P.~Liang, and C.~Finn.
\newblock Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices.
\newblock In \emph{International conference on machine learning}, pages 6925--6935. PMLR, 2021.
\bibitem[Xie et~al.()Xie, Bhamidipaty, Liu, Hong, Levine, and Finn]{xielearning}
A.~Xie, L.~M. Bhamidipaty, E.~Z. Liu, J.~Hong, S.~Levine, and C.~Finn.
\newblock Learning to explore in pomdps with informational rewards.
\newblock In \emph{Forty-first International Conference on Machine Learning}.
\bibitem[James and Davison(2022)]{james2022q}
S.~James and A.~J. Davison.
\newblock Q-attention: Enabling efficient learning for vision-based robotic manipulation.
\newblock \emph{IEEE Robotics and Automation Letters}, 7\penalty0 (2):\penalty0 1612--1619, 2022.
\bibitem[James et~al.(2022)James, Wada, Laidlow, and Davison]{james2022coarse}
S.~James, K.~Wada, T.~Laidlow, and A.~J. Davison.
\newblock Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 13739--13748, 2022.
\bibitem[Tai et~al.(2023)Tai, Chiu, Chao, and Chen]{tai2023scone}
Y.-L. Tai, Y.~C. Chiu, Y.-W. Chao, and Y.-T. Chen.
\newblock Scone: A food scooping robot learning framework with active perception.
\newblock In \emph{Conference on Robot Learning}, pages 849--865. PMLR, 2023.
\bibitem[Cheng et~al.(2018)Cheng, Agarwal, and Fragkiadaki]{cheng2018reinforcement}
R.~Cheng, A.~Agarwal, and K.~Fragkiadaki.
\newblock Reinforcement learning of active vision for manipulating objects under occlusions.
\newblock In \emph{Conference on Robot Learning}, pages 422--431. PMLR, 2018.
\bibitem[Zaky et~al.(2020)Zaky, Paruthi, Tripp, and Bergstra]{zaky2020active}
Y.~Zaky, G.~Paruthi, B.~Tripp, and J.~Bergstra.
\newblock Active perception and representation for robotic manipulation.
\newblock \emph{arXiv preprint arXiv:2003.06734}, 2020.
\bibitem[Jangir et~al.(2022)Jangir, Hansen, Ghosal, Jain, and Wang]{jangir2022look}
R.~Jangir, N.~Hansen, S.~Ghosal, M.~Jain, and X.~Wang.
\newblock Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation.
\newblock \emph{IEEE Robotics and Automation Letters}, 7\penalty0 (2):\penalty0 3046--3053, 2022.
\bibitem[Lv et~al.(2023)Lv, Feng, Zhang, Zhao, Shao, and Lu]{lv2023sam}
J.~Lv, Y.~Feng, C.~Zhang, S.~Zhao, L.~Shao, and C.~Lu.
\newblock Sam-rl: Sensing-aware model-based reinforcement learning via differentiable physics-based simulation and rendering.
\newblock In \emph{Robotics science and systems}, 2023.
\bibitem[Shang and Ryoo(2024)]{shang2024active}
J.~Shang and M.~S. Ryoo.
\newblock Active vision reinforcement learning under limited visual observability.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[G{\"o}ransson(2022)]{goransson2022deep}
R.~G{\"o}ransson.
\newblock \emph{Deep Reinforcement Learning with Active Vision on Atari Environments}.
\newblock PhD thesis, Master’s Thesis. Lund University, 2022.
\bibitem[Grimes et~al.(2022)Grimes, Modayil, Mirowski, Rao, and Hadsell]{grimes2022learning}
M.~K. Grimes, J.~V. Modayil, P.~W. Mirowski, D.~Rao, and R.~Hadsell.
\newblock Learning to look by self-prediction.
\newblock \emph{Transactions on Machine Learning Research}, 2022.
\bibitem[Rezaei-Shoshtari et~al.(2023)Rezaei-Shoshtari, Morissette, Hogan, Dudek, and Meger]{rezaei2023hypernetworks}
S.~Rezaei-Shoshtari, C.~Morissette, F.~R. Hogan, G.~Dudek, and D.~Meger.
\newblock Hypernetworks for zero-shot transfer in reinforcement learning.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~37, pages 9579--9587, 2023.
\bibitem[Menda et~al.(2019)Menda, Driggs-Campbell, and Kochenderfer]{menda2019ensembledagger}
K.~Menda, K.~Driggs-Campbell, and M.~J. Kochenderfer.
\newblock Ensembledagger: A bayesian approach to safe imitation learning.
\newblock In \emph{2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages 5041--5048. IEEE, 2019.
\bibitem[Liu et~al.(2024)Liu, Zhu, Gao, Feng, Liu, Zhu, and Stone]{liu2024libero}
B.~Liu, Y.~Zhu, C.~Gao, Y.~Feng, Q.~Liu, Y.~Zhu, and P.~Stone.
\newblock Libero: Benchmarking knowledge transfer for lifelong robot learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Dass et~al.(2024)Dass, Ai, Jiang, Singh, Hu, Zhang, Stone, Abbatematteo, and Martin-Martin]{dass2024telemoma}
S.~Dass, W.~Ai, Y.~Jiang, S.~Singh, J.~Hu, R.~Zhang, P.~Stone, B.~Abbatematteo, and R.~Martin-Martin.
\newblock Telemoma: A modular and versatile teleoperation system for mobile manipulation.
\newblock \emph{arXiv preprint arXiv:2403.07869}, 2024.
\bibitem[Schulman et~al.(2017)Schulman, Wolski, Dhariwal, Radford, and Klimov]{schulman2017proximal}
J.~Schulman, F.~Wolski, P.~Dhariwal, A.~Radford, and O.~Klimov.
\newblock Proximal policy optimization algorithms.
\newblock \emph{arXiv preprint arXiv:1707.06347}, 2017.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever]{radford2021clip}
A.~Radford, J.~W. Kim, C.~Hallacy, A.~Ramesh, G.~Goh, S.~Agarwal, G.~Sastry, A.~Askell, P.~Mishkin, J.~Clark, G.~Krueger, and I.~Sutskever.
\newblock Learning transferable visual models from natural language supervision.
\newblock \emph{CoRR}, abs/2103.00020, 2021.
\newblock URL \url{https://arxiv.org/abs/2103.00020}.
\bibitem[Bishop(1994)]{bishop1994mixture}
C.~M. Bishop.
\newblock Mixture density networks.
\newblock 1994.
\bibitem[Zhu et~al.(2020)Zhu, Yu, Gupta, Shah, Hartikainen, Singh, Kumar, and Levine]{zhu2020ingredients}
H.~Zhu, J.~Yu, A.~Gupta, D.~Shah, K.~Hartikainen, A.~Singh, V.~Kumar, and S.~Levine.
\newblock The ingredients of real-world robotic reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2004.12570}, 2020.
\bibitem[Pages et~al.(2016)Pages, Marchionni, and Ferro]{pages2016tiago}
J.~Pages, L.~Marchionni, and F.~Ferro.
\newblock Tiago: the modular robot that adapts to different research needs.
\newblock In \emph{International workshop on robot modularity, IROS}, volume 290, 2016.
\end{thebibliography}