Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Extracted source text ยท arXiv 2409.05865
\documentclass[11pt, letterpaper]{nyu}
\usepackage[all]{hypcap}
\bibliographystyle{yearnat}
\usepackage{hyperref}[citecolor=lightblue]
\hypersetup{
colorlinks = true,
}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} %
\usepackage{float}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{caption}
\usepackage{mathrsfs}
\usepackage{nicefrac}
\usepackage{dsfont}
\usepackage{enumitem}
\usepackage{float}
\usepackage[authoryear, sort&compress, round]{natbib}
\usepackage{xspace}
\usepackage[capitalize,noabbrev]{cleveref}
\usepackage{subcaption}
\usepackage{wrapfig}
\usepackage{lipsum}
\usepackage{listings}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{bbm}
\usepackage{algpseudocode}
\usepackage{setspace}
\usepackage{color}
\usepackage[skins,theorems]{tcolorbox}
\usepackage{xcolor}
\usepackage{tikz}
\setlength\parindent{0pt}
\definecolor{deepblue}{rgb}{0,0,0.5}
\definecolor{deepred}{rgb}{0.6,0,0}
\definecolor{deepgreen}{rgb}{0,0.5,0}
\newcommand\pythonstyle{\lstset{
basicstyle=\ttfamily\footnotesize,
language=Python,
morekeywords={self, clip, exp, mse_loss, uniform_sample, concatenate, logsumexp}, %
keywordstyle=\color{deepblue},
emph={MyClass,__init__}, %
emphstyle=\color{deepred}, %
stringstyle=\color{deepgreen},
frame=single, %
showstringspaces=false
}}
\lstnewenvironment{python}[1][]
{
\pythonstyle
\lstset{#1}
}
{}
\newcommand\pythonexternal[2][]{{
\pythonstyle
\lstinputlisting[#1]{#2}}}
\newcommand\pythoninline[1]{{\pythonstyle\lstinline!#1!}}
\makeatletter
\def\mathcolor#1#{\@mathcolor{#1}}
\def\@mathcolor#1#2#3{%
\protect\leavevmode
\begingroup
\color#1{#2}#3%
\endgroup
}
\makeatother
\usepackage[textsize=tiny]{todonotes}
\pdfstringdefDisableCommands{%
\let\Cref\crtCref
\let\cref\crtcref
}
\newcommand{\creftitle}[1]{\crtcref{#1}}
\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\usepackage{crossreftools}
\usepackage{dsfont}
\usepackage{nicefrac}
\usepackage{inconsolata}
\usepackage{algorithm}
\usepackage{amssymb}
\usepackage{csquotes}
\newcommand{\xxnote}[3]{}
\renewcommand{\xxnote}[3]{}
\newcommand{\LP}[1]{\textcolor{red}{LP: #1}}
\newcommand{\CP}[1]{\textcolor{orange}{CP: #1}}
\newcommand{\MS}[1]{\textcolor{blue}{MS: #1}}
\newcommand{\SL}[1]{\textcolor{green}{SL: #1}}
\newcommand{\HE}[1]{\textcolor{cyan}{HE: #1}}
\newcommand{\website}{\href{https://robotutilitymodels.com}{robotutilitymodels.com}}
\newtheorem{theorem}{Theorem}
\newcommand{\longname}{Robot Utility Models}
\newcommand{\longnamesingular}{Robot Utility Model}
\newcommand{\shortname}{RUMs}
\newcommand{\shortnamesingular}{RUM}
\newcommand{\finalSuccessRate}{90\%}
\newcommand{\finalSuccessRateWithoutRetry}{76.8\%}
\title{Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments}
\websitedef{\website}
\reportnumber{}
\author[1]{Haritheja Etukuru$^{*}$}
\author[1]{Norihito Naka}
\author[1]{Zijin Hu}
\author[1]{Seungjae Lee}
\author[2]{Julian Mehu}
\author[2]{Aaron Edsinger}
\author[2]{Chris Paxton}
\author[3]{Soumith Chintala}
\author[1]{Lerrel Pinto}
\author[1,2]{Nur Muhammad ``Mahi'' Shafiullah$^{*}$}
\affil[1]{New York University}
\affil[2]{Hello Robot Inc.}
\affil[3]{Meta Inc.}
\correspondingauthor{ \href{mailto:mahi@cs.nyu.edu}{mahi@cs.nyu.edu}. $(^{*})$~denotes equal contribution.}
\begin{abstract}
\vspace{-1em}
Robot models, particularly those trained with large amounts of data, have recently shown a plethora of real-world manipulation and navigation capabilities. Several independent efforts have shown that given sufficient training data in an environment, robot policies can generalize to demonstrated variations in that environment. However, needing to finetune robot models to every new environment stands in stark contrast to models in language or vision that can be deployed zero-shot for open-world problems.
In this work, we present \textit{\longname{} (\shortname{})}, a framework for training and deploying zero-shot robot policies that can directly generalize to new environments without any finetuning. To create \shortname{} efficiently, we develop new tools to quickly collect data for mobile manipulation tasks, integrate such data into a policy with multi-modal imitation learning, and deploy policies on-device on Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier for retrying. We train five such utility models for opening cabinet doors, opening drawers, picking up napkins, picking up paper bags, and reorienting fallen objects. Our system, on average, achieves \textbf{\finalSuccessRate{} success rate in unseen, novel environments interacting with unseen objects}. Moreover, the utility models can also succeed in different robot and camera set-ups with no further data, training, or fine-tuning.
Primary among our lessons are the importance of training data over training algorithm and policy class, guidance about data scaling, necessity for diverse yet high-quality demonstrations, and a recipe for robot introspection and retrying to improve performance on individual environments.
\end{abstract}
\begin{document}
\maketitle
\input{sections/1_intro}
\input{sections/2_methods}
\input{sections/3_experiments}
\input{sections/4_related_work}
\input{sections/5_discussion}
\section*{Acknowledgements}
We thank Shenglong Wang and the NYU HPC team for helping us with compute, Blaine Matulevic and Binit Shah for supporting our hardware needs, and Siddhant Haldar and Jeff Cui for providing feedback on the paper. NYU authors are supported by grants from Honda, Hyundai, NSF award 2339096 and ONR awards N00014-21-1-2758 and N00014-22-1-2773. MS is supported by the Apple Fellowship. LP is supported by the Packard Fellowship. SL is supported by the Daishin Songchon Foundation. Hello Robot authors are supported by NIH NIA R43AG072982.
\newpage
\bibliography{references}
\newpage
\appendix
\input{sections/6_appendix}
\end{document}
\begin{figure}[h]
\vskip -0.5em
\centering
\includegraphics[width=1.0\linewidth]{figures/intro_home_opt.pdf}
\caption{\longname{} are trained on a diverse set of environments and objects, and then can be deployed in novel environments with novel objects without any further data or training.}
\label{fig:intro-figure}
\vskip -1em
\end{figure}
\section{Introduction}
We have seen rapid progress in training manipulation skills recently~\citep{zhao2023aloha,fu2024mobile,brohan2023rt2,haldar2024baku,fu2024humanplus,lin2024learning,kim2024openvla}, largely brought about by fitting deep networks on data collected by teleoperating robots~\citep{mandlekar2018roboturk,iyer2024open,arunachalam2023dexterous,cheng2024opentelevisionteleoperationimmersiveactive,khazatsky2024droid}. The mechanism for deploying such skills in new environments mimics the pretrain-then-finetune strategy first developed by the vision community circa 2014~\citep{girshick2014rich}. There, models were first pretrained on ImageNet and then finetuned on task-specific data such as detection, segmentation, and pose estimation~\citep{girshick2014rich,GkioxariHGM14}.
In the context of robotics, this strategy involves pretraining on large robot datasets~\citep{padalkar2023open,khazatsky2024droid,shafiullah2023bringing,walke2023bridgedata} to produce a robot foundation model, which is then fine-tuned on data collected in new environments or tasks~\citep{shafiullah2023bringing,team2024octo,kim2024openvla}. This need to fine-tune the foundation model for each and every new environment is limiting as it requires humans to collect data in the very environment where the robot is expected to perform. So while vision and language models have moved on to zero-shot deployments, i.e. without any environment-specific finetuning data, such a capability eludes most robot manipulators. This is not to say that there have not been attempts to create zero-shot manipulation models -- several foundational work in grasping and pick-and-place~\citep{fang2023anygrasp,sundermeyer2021contact,Mahler2017DexNet2D} have tackled this problem albeit with a task-specific solution.
So what makes creating a general policy for an arbitrary task that can work zero-shot hard? First is the concern about sufficient data -- the necessary amount of data to train such a general model could be large. Since collecting robot data is hard, creating a large dataset is also hard and often expensive since humans are usually tasked to collect robot demonstrations. Second, when a large dataset is collected in the open-world it would necessarily have large diversity and multiple modes in demonstrator behavior. Fitting a robot models on this diverse data is a challenge. Third, unlike vision and language, where the native form of data, i.e. images and text are largely standard, robotics is far from having a standard camera and hardware setup along with physical challenges of running models in realtime on onboard compute. Creating zero-shot models that can run with even minor changes to hardware setup between training and deployment requires careful attention to details.
Finally, any model deployed zero shot on a novel environment naturally has a higher failure rate than a model that has been fine-tuned on that environment. Thus, to deploy a model zero-shot, it is important to have a mechanism for error detection and recovery.
In this work, we introduce~\longname{} (\shortname{}), a new framework for training focused and functional \textit{utility} models to complete helpful tasks that can be deployed zero-shot \textbf{without further training or fine-tuning} in novel environments. This is done by taking a systems-first approach. To scale up our datasets without compromising on data quality, we develop a new tool, building on prior work in untethered data collection~\citep{shafiullah2023bringing,chi2024universal}. We train policies on these diverse dataset with state-of-the-art multi-modal behavior learning algorithms~\citep{lee2024behavior,chi2023diffusion} and show how they can absorb and scale with large-scale demonstration data. Finally, we deploy the policy in multiple different environments out of the box, with self-critique via mLLMs~\citep{guo2023doremi} and retrying, showing how the policy can be robustly executed on cheap, general-purpose hardware. A selection of our trained models are available on the Hello Robot Stretch without much modifications. Beyond the default Stretch deployment, we also enable deployment on other robot arms, cameras, and lighting conditions, showing the generalizability of our approach.
Creating and deploying~\shortname{} led us to several interesting lessons. First, we find that the quantity and quality of data is crucial for training a utility model, with the choice of model architecture being less critical. Second, we see that the diversity of the data collected is crucial for the model to generalize to new environments, and more important than the raw quantity of data. Third, we find that the model can be made more capable in single environments by performing self-critique on the model performance with an independent model and retrying when appropriate.
To validate \shortname{}, we run a total of 2,950 robot rollouts in real-world environments including homes in New York City (NY), Jersey City (NJ), and Pittsburgh (PA). These experiments reveal the following:
\begin{itemize}[leftmargin=*]
\item We show that it is possible to create general \longname{} with a moderate amount of data in the order of 1,000 demonstrations (Section~\ref{sec:methods}). These \shortname{} achieve a 90\% average success rate on zero-shot deployment in 25 novel environments (Section~\ref{sec:unseen-environments}).
\item The success of \shortname{} relies primarily on two key techniques. First, the use of multi-modal policies (Section~\ref{sec:model-training}) provides a zero-shot success rate of 74.4\% (Section~\ref{sec:policy-architecture}). Second, the mLLM based self-critique and retrying system (Section~\ref{sec:retrying}) further improves the success rate by 15.6\% (Section~\ref{sec:introspection-retrying}).
\item While the overall framework for \shortname{} is straightforward, the devil is in the details, where we find gains from unexpected sources, e.g. data diversity vs. data quantity (Section~\ref{sec:data-diversity} and~\ref{sec:expert-vs-play}).
\end{itemize}
To encourage the development of \shortname{} for a wider variety of tasks, our code, data, models, hardware designs, as well as our experiment and deployment videos are open sourced and can be found on our website: \website.
\section{Robot Utility Models}
\label{sec:methods}
We take a full-stack approach to create \longname{}. At its core, our system follows the imitation learning framework.
However, to effectively scale imitation learning to the point where our trained policies are deployable zero-shot, we create new tools and techniques to improve data collection, model training, inference, and deployment.
\subsection{Data collection tool}
\label{sec:data-collection-tool}
One of the primary requirements of our system is to be able to scale up diverse yet accurate demonstration data for cheap. To this end, we continue on the evolutionary path of hand-held, portable data collection tools~\citep{song2020grasping,young2020visual,pari2021surprising,shafiullah2023bringing,chi2024universal} that let us quickly collect precise demonstrations. Following our previous work~\citep{shafiullah2023bringing}, we call this tool \textit{Stick-v2}, which is a hand-held data collection tool built out of an iPhone Pro and a bill of materials that adds up to \$25. We combine inspirations from the quick deployability of Stick-v1, and the compact, handheld form factor of UMI gripper. For a detailed build instruction and the bill of materials, we refer the reader to the supplementary materials (Appendix~\ref{sec:app:bom}).
Our design decisions are predicated on a few factors: portability, convenience, and set-up speed. We experimentally found these factors to be important to quickly scale up robot datasets and training \shortname{}. As we show with experiments in Section~\ref{sec:scaling-datasets}, one of the most crucial aspect of data collection for \shortname{} is data diversity, i.e. collecting data from a large number of diverse environmets. Thus, it is crucial to have a portable tool that is easy to mass-print, carry, and deploy in a new environment. Secondly, it is important for the collected data to be accurate across many environments with many variations. Finally, it is important to minimize the ``per-environment set-up time'', whether that time is spent setting up the data collection system, calibrating the camera, or the tool's SLAM system.
\begin{figure}[t!]
\centering
\includegraphics[width=\textwidth]{figures/stickv2.pdf}
\caption{Stick-v2, our data collection tool (left: real photo, right: render), is built out of an iPhone Pro and a bill of materials that adds up to \$25. The tool is portable, robust, and makes it easy to start collecting data in a new environment in seconds.}
\label{fig:stick-v2}
\end{figure}
For the above reason, we design our data collection tool, Stick-v2, around the ARKit API from the widely available and used iPhone Pro (Figure~\ref{fig:stick-v2}). Given its technical capabilities, the only digital component in our Stick-v2 is this iPhone, which makes our tool particularly robust to shipping and handling. The iPhone, and therefore Stick-v2, can collect RGB video and depth data at up to 60 Hz and high precision 6D pose and position information from the ARKit API at up to 100Hz. To capture the gripper opening information, we trained an RGB-based model that predicts the gripper aperture from images. Furthermore, this data is automatically synchronized and timestamped by the iPhone without the need for any calibration. This allows us to collect data from a wide variety of environments with no set-up time. This is in contrast to other data collection tools based on visual SLAM systems which has limited precision and are non-robust around ``textureless'' scenes such as close to flat walls, ceilings, or corners~\citep{chi2024universal,young2020visual}. Finally, not needing camera calibration makes our system deployable out-of-the-box in any environment, especially in the real world where the environment is not controlled. This enables us to, for example, collect data from retail home goods stores with minimal interruptions to enrich our datasets, which would be hindered if we had to calibrate the camera and odometry system for each new environment.
\subsection{Collected datasets}
\label{sec:collected-datasets}
We collect data for each of our five tasks, which are as defined below:
\begin{itemize}[leftmargin=*]
\item \textbf{Door opening:} Open doors with a long handle, on e.g. cabinets and microwaves. Due to hardware limitations, our robot cannot open doors with round knobs, so we exclude them from our dataset.
\item \textbf{Drawer opening:} Open a drawer with a handle. We exclude drawers with knobs from our dataset for similar reasons as above.
\item \textbf{Reorientation:} Pick up a cylindrical object (e.g. bottle) lying on a flat surface and place it upright on the same surface.
\item \textbf{Tissue pickup:} Pick up a soft, flexible tissue paper from any tissue paper box.
\item \textbf{Bag pickup:} Pick up a kraft paper bag or similar other bags from a flat surface.
\end{itemize}
\begin{figure}[t!]
\centering
\includegraphics[width=\textwidth]{figures/data_figure_opt.pdf}
\caption{A small sample of environment and objects from our collected dataset. We collect data for each of our five tasks on a diverse set of environments and objects using Stick-v2.}
\label{fig:dataset-fig}
\end{figure}
For each of our five~\shortname{}, we focused on gathering approximately 1,000 demonstrations
on approximately~40 environments, with about 25 demonstrations per environment on average. The only exceptions are door opening with~1,200 and drawer opening with~525 demonstrations. A small collection of such environments are shown in Figure~\ref{fig:dataset-fig}. For the door opening task, we seeded this dataset with the Homes of New York dataset~\citep{shafiullah2023bringing} as well as demonstrations collected during the Dobb$\cdot$E experiments. For the other tasks, our dataset consists of new demonstrations collected using the Stick-v2 tool on a novel set of environments and objects. For demonstrations collected from the previous dataset by inexperienced data collectors, we do a manual quality check and exclude any environment that has a high number of low-quality demonstrations, such as failed demonstrations.
Note that, to keep our experiments unbiased, we hold out test environments and objects and never collect any data on them. To gain quick insight on different task data we use for training, we created an interactive data diversity visualization tool: \href{https://robotutilitymodels.com/data_diversity/}{robotutilitymodels.com/data\_diversity/}.
\subsection{Model training}
\label{sec:model-training}
Given that our data is collected by a large set of demonstration collectors, conceptually it is important for the model to handle any resultant multi-modality in the dataset. In this work, we train a large set of policy classes on our datasets for each task. Among the policy classes, the best performing ones are VQ-BeT~\citep{lee2024behavior} and Diffusion Policy (DP)~\citep{chi2023diffusion}. We also train ACT~\citep{zhao2023aloha} and MLP-BC policies on a limited set of tasks. Each policy class shares some features, such as a ResNet34-based vision encoder initialized to the HPR encoder from~\cite{shafiullah2023bringing}, and a transformer-based policy trunk. We also train each model for the same 500 epochs. Beyond that, we sweep to find the best hyperparameters for learning rate, history length, and chunk size, and use the recommended hyperparameters from the original papers for each model.
Our final VQ-BeT models are trained on data subsampled at 3.75Hz, and uses 6 most recent frames of history to predict the next action. All of our models predict the action in relative 6D space for the robot end-effector, and absolute value in the range $[0, 1]$ for the gripper opening.
We discuss the impact of choosing different training algorithms in Section~\ref{sec:policy-architecture}.
Training all of our models took between 24 and 48 hours on 2 Nvidia A100 GPUs on our cluster, with proportional speed-ups by using more GPUs or using more recent GPUs like H100s.
\subsection{Retrying with GPT-4o feedback}
\label{sec:retrying}
\begin{figure}[ht!]
\centering
\includegraphics[width=\textwidth]{figures/mllm_figure_3.pdf}
\caption{Automated retrying with feedback from multimodal LLM critic. We use a multimodal LLM (\texttt{gpt-4o-2024-05-13} in our experiments) to verify the success of a task given a summary of robot observations. If the mLLM detects a failure, we automatically reset the robot and retry the task with a new initial robot state until success or timeout.}
\label{fig:mllm-feedback}
\end{figure}
While a pre-trained model can solve the task in a new environment, to achieve the best possible performance, it is helpful to have additional runtime support for the model. For our deployment, we use an multimodal LLM (\texttt{gpt-4o-2024-05-13}) as an introspection module for our policies for a success detection and retrying mechanism. We define a single verification prompt for each task, and ask the mLLM to verify the success of the task given a summary of robot observations. As for the run summary, we give the mLLM every other frame from the robot camera, which is either from the head or the wrist camera depending on the task. If the mLLM detects a failure (Figure~\ref{fig:mllm-feedback}), ~\shortnamesingular{} automatically resets the robot to a home position and retries the task with a new initial robot state.
\subsection{Deployment Details}
\label{sec:deployment}
\begin{figure}[t!]
\centering
\includegraphics[width=\textwidth]{figures/deploy_figure.pdf}
\caption{Picture of the some robot setups where our~\longname{} can be deployed. We show the Hello Robot: Stretch, and the xArm 7 robot with iPhone Pros on the wrist. Beyond these, we also deploy on Stretch robots with default D405 wrist cameras.
}
\label{fig:robot-setup}
\end{figure}
Our primary hardware for \longname{} deployment is the Hello Robot: Stretch robots with an iPhone on the wrist, but we support deploying our models on any robot arm with relative 6D pose and position control (Figure~\ref{fig:robot-setup}). We design and release an associated robot end-effector that can be mounted on standard robot arms, such as the xArm or Franka Panda. Similarly, while we primarily use the iPhone Pro as the deployment camera, we also show deployment on other wrist cameras, such as the Intel Realsense D405, which is the default wrist camera on Hello Stretch Edition 3 onwards. Overall, our deployment hardware system really relies on three things: our end-effector with a flexible two-fingered gripper and gripper tips, a wrist camera with a sufficient field of view, and an arm with six degrees of freedom to mount our wrist. We release default integration code for Hello Stretch 3 and an xArm wrist mount that we created, which should serve as illustrative examples for other arms.
\section{Capabilities of Robot Utility Models}
\label{sec:experiments}
To understand the capabilities of \shortname{}, we evaluate each of our models on a diverse set of environments. At the same time, we try to examine our recipe for training utility models and answer a set of questions about the trained models by running a set of ablation experiments. The primary questions that we try to answer are the following:
\begin{itemize}[leftmargin=*]
\item How well do \longname{} solve a task in an unseen environment while operating on unseen objects?
\item What is the relative importance of different components of~\longname{}, such as training data, training algorithm, and self-verification?
\begin{itemize}
\item What scale of data is needed to train capable \shortname{}?
\item What properties of data are most important for training \shortname{}?
\item How does mLLM-based self-critique affect~\shortname{}, and where does it succeed or fail?
\end{itemize}
\item How well can we deploy \shortname{} on new robot embodiments?
\end{itemize}
\paragraph{Evaluation details:} For each task, we set up 25 novel environments -- five for each task -- with objects and props not seen in the training dataset. To create these evaluation environments, we take the robot to previously unseen kitchens, purchase new furniture online (door and drawer opening), and source new objects manually verified to not be in the training set (reorientation, bag and tissue pick up).
We show sample pictures of each of the environments and objects on our Appendix~\ref{sec:app:eval_envs}.
We evaluate each system and policy for 10 trials in each of these environments, starting from the same grid of starting positions facing the task space used by~\cite{shafiullah2023bringing} as we show in Appendix Figure~\ref{fig:run-schedule}.
For the retrying-based experiments, while \shortname{} take 1.31 tries in average to succeed (Section~\ref{sec:introspection-retrying}), we set a 10-try timeout to avoid getting stuck in infinite retry loops.
\subsection{Zero-shot evaluation of \shortname{} on unseen environments}
\label{sec:unseen-environments}
The most important test of capability for a \longnamesingular{} is whether such a model is capable of solving the target task in a new environment operating on new objects. We test for this capability by running our \shortname{} on our set of 25 eval environments and objects not seen during training.
\begin{figure}[ht!]
\centering
\includegraphics[width=\linewidth]{figures/final_success_rate.pdf}
\caption{Success rate of~\longname{} on average over five novel scenes in five different tasks. The X's on the figure denote success rates from individual environments.}
\label{fig:final-success-rate}
\end{figure}
On Figure~\ref{fig:final-success-rate}, we see that on unseen and novel environments, \shortname{} perform well, achieving a \finalSuccessRate{} success rate overall, and ranging between 84\% to 94\% on individual tasks. We discuss some of the failure cases we observe in the Appendix Section~\ref{app:sec:failure-modes}. Additionally, we show the performance of \shortname{} on each test environment on Table~\ref{table:success-statistics-with-retry}, showing that across all of our evaluation experiments, \shortname{} achieves some success in every environment. This success implies that our policies have a general idea of solving the target task; then such policies are further boosted with post-training methods (Section~\ref{sec:introspection-retrying}).
On all of our following experiments, we try to understand these two factors separately: the raw performance of the underlying~\shortnamesingular{} policies, and the effect of introspection and retrying on the performance of~\shortname{}.
\subsection{Effect of policy architecture and training method on \shortname{}}
\label{sec:policy-architecture}
Once we have verified that \shortname{} can actually solve tasks in novel environments, we investigate the relative importance of different components within the training recipe. In particular, we compare the raw performance of different policy architectures on our dataset without the introspection component. We train a set of policy classes on our datasets for each task, including VQ-BeT~\citep{lee2024behavior}, Diffusion Policy (DP)~\citep{chi2023diffusion}, and as baselines, ACT~\citep{zhao2023aloha} and MLP-BC on two of the tasks. We show the relative comparison of the base success rates of different policy architectures, without retrying, in Figure~\ref{fig:algorithm-vs-data} and~\ref{fig:more-algorithm-vs-data}.
\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{figures/raw_success_rate_with_error.pdf}
\caption{Relative comparison of the success rate (with standard error) of different policy architectures on our dataset on all five tasks without automated error correction. We see that the performance of VQ-BeT and Diffusion Policy is generally close, with VQ-BeT narrowly outperforming Diffusion Policy.
}
\label{fig:algorithm-vs-data}
\end{figure}
\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{figures/comparison_of_more_algorithms.pdf}
\caption{Relative comparison of different policy architectures on our dataset on two tasks without automated error correction. We see that while the performance of VQ-BeT and Diffusion Policy is generally neck-to-neck, while the performance of other algorithms is not far behind.
Our experiment implies that the training data is significantly more important than training algorithm.}
\label{fig:more-algorithm-vs-data}
\end{figure}
As we see in Figure~\ref{fig:algorithm-vs-data}, VQ-BeT and DP are the top two algorithms in terms of performance, with comparable performance in most tasks and overlapping error bars. Moreover, we see from Figure~\ref{fig:more-algorithm-vs-data} that while ACT and MLP-BC are not exactly on par, they are not far behind either. This observation implies that with training data of sufficient quality, the choice of algorithm may not be a make-or-break decision, and more energy should be spent on collecting diverse and accurate data. While we have similar performances on the test environment, we use VQ-BeT over DP for our final models due the higher performance and a lower latency on the robot CPU itself during deployment.
\subsection{Effect of scaling datasets on \shortname{}}
\label{sec:scaling-datasets}
\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{figures/scaling_laws.pdf}
\caption{Understanding the performance change of \shortname{} as the dataset scales up on three of our tasks, with standard error on error bars. We see better performance from Diffusion Policy (DP) on smaller datasets, but as we scale up, VQ-BeT outperforms DP in 900--1,200 demonstrations limit.}
\label{fig:scaling-laws}
\end{figure}
As our experiments show the importance of training data in creating~\shortname{}, we investigate the properties of the dataset that a successful~\shortname{} relies on. In particular, we dig into the scale of dataset at which reliable generalization emerges, and how \shortname' performance vary with dataset size. We train our policies on a random subset of environments from the task-specific datasets, and evaluate them on our evaluation environments.
In Figure~\ref{fig:scaling-laws}, we show the performance of VQ-BeT and Diffusion Policy without retrying trained on such data subsets on our evaluation environments as we scale up the dataset. We see that while Diffusion Policy performs better on smaller datasets, it saturates on larger datasets where VQ-BeT outperforms it. This observation implies that while a smaller dataset may be sufficient for training a capable \shortname{}, a larger dataset is crucial for achieving the best performance. Even on our largest datasets, we see that the performance of VQ-BeT continues to improve as the dataset scales up, implying that more data may improve~\shortname{} even further.
\subsection{Importance of data diversity in training \shortname{}}
\label{sec:data-diversity}
\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{figures/important_data_qualities.pdf}
\caption{
Understanding the importance of different qualities of data in training \shortname{}. On the left, we see that diverse datasets are more valuable than more uniform datasets, with strong effects on the reorientation task with many unseen environments and object.
On the right, we see that usually expert data is more valuable than non-expert or play data while learning behavior on a same sized dataset. Moreover, we see that co-training with expert data and play data may sometimes reduce the policy performance, contrary to common knowledge.
}
\label{fig:expert-vs-play}
\end{figure}
Beyond the scale of the dataset, we also investigate how the diversity of the training data impacts the performance of~\shortname{} in Figure~\ref{fig:expert-vs-play} (left). We create two alternate datasets of equal size for the door opening and the object reorientation tasks. The first datasets are composed of a large number of diverse environments with roughly 25 demonstrations in each environment. The second dataset is composed of fewer, between 5 and 6, distinct environments with roughly 200 demonstrations on each environment. We see that on the door opening task, where the scene diversity is narrower, both diverse and uniform environment trained policies performed well. However, in the reorientation task, with many different unseen environments and objects, only diverse-environment trained \shortnamesingular{} policy performs well -- the policy trained on more uniform environments experiences a 50\% performance drop. This result implies that to train an effective~\shortnamesingular{}, collecting a diverse dataset is important.
\subsection{Impact of using expert demonstrations on training policies}
\label{sec:expert-vs-play}
While scaling up the dataset size and diversity is important for training~\shortname{}, an important question to consider is the quality of the training dataset. Namely, while it may be easy to collect a large number of demonstrations by a large number of demonstrators, the quality of the demonstrations may vary. In this section, we investigate the value of using expert demonstrations in training~\shortname{}.
In Figure~\ref{fig:expert-vs-play} (right) we compare the performance of~\shortname{} trained on roughly 500 demonstrations, where the data is either sampled from expert or non-expert demonstration collectors. Here, ``expertise'' is defined as experience deploying Dobb$\cdot$E policies on the robot. We see that in general, expert data is more valuable than non-expert data, with expert data outperforming non-expert data in all tasks. Moreover, we see that co-training with expert and non-expert data can sometimes, but not always, improve the performance of the policy. This observation implies depending on the task, data quality can have different levels of suboptimality, and in extreme cases may even hurt performance in co-training, which goes against a common practice in some earlier works~\citep{zhao2023aloha,khazatsky2024droid}.
\subsection{Effects of introspection and retrying with self-critique in \shortname{}}
\label{sec:introspection-retrying}
\begin{figure}[h!]
\centering
\includegraphics[width=\linewidth]{figures/gpt_data.pdf}
\caption{
Understanding the details of introspection and retrying in \shortname{}. On the left, we see that retrying improves the performance of \shortname{} significantly, with an average 15.6\% improvement. In the middle, we see that with retrying, most tasks get solved quite fast, on average with 1.31 tries. On the right, we see that while the mLLM is able to help, it can also have false positives (4.8\% average over five tasks) which may let some errors slip past.
}
\label{fig:gpt-retrying}
\end{figure}
In~\shortname{}, we are using a multimodal large language model (mLLM) as a self-critique method to identify failures. However, a pretrained mLLM in practice is just another layer of fail-safe for our robot deployment, and not a guarantee of success in itself. Thus, in this section we try to understand how it helps, and how such introspection method can fail.
In Figure~\ref{fig:gpt-retrying} (left), we can see the improvement rate of using self-critique over simply using the~\shortnamesingular{} policies without any retrying mechanism. On average over our 5 tasks, we see a 15.6\% improvement over simply using~\shortnamesingular{} policies. While retrying is crucial to a higher success rate, a system that is stuck retrying for a long time is much less useful. Thankfully, on average, when~\shortname{} succeeds, it does so within 1.31 tries on average, as we see from Figure~\ref{fig:gpt-retrying} (middle). Finally, we analyze the primary failure mode of mLLMs, which is predicting false positives: classifying a trajectory as a success when it's actually a failure. On average, 4.8\% of our trajectories exhibit such behavior, constituting of half of the total errors, as seen on Figure~\ref{fig:gpt-retrying} (right).
\subsection{Transferring \shortname{} to different embodiments }
\label{sec:transferring-embodiments}
\begin{wrapfigure}{r}{0.45\textwidth}
\vskip -0.5cm
\centering
\includegraphics[width=0.44\textwidth]{figures/different_embodiment.pdf}
\caption{Performance of \shortname{} without corrections on different embodiments as shown in Figure~\ref{fig:robot-setup}: \shortname{} can transfer to different embodiments with minimal loss in performance. }
\label{fig:embodiment-transfer}
\vskip -0.5cm
\end{wrapfigure}
Finally, we investigate the ability of~\shortname{} to be transferred to different embodiments and cameras. We test the performance of two~\shortname{} on the other robot setup shown in Figure~\ref{fig:robot-setup}: UFactory xArm 7, which is different from the Hello Robot Stretch setup we run other experiments on. We see that~\shortname{} can be transferred to different embodiments and cameras with minimal loss in performance: roughly 10\% drop in performance in both cases without corrective mLLM feedback, as shown in Figure~\ref{fig:embodiment-transfer}. We expect combining~\shortname{} with the mLLM self-critique would result in similar increase in performance in other embodiments as well; in fact, with an external third person camera, we expect to see a higher portion of the errors being caught and corrected. This experiment implies that~\shortname{} can be easily deployed on different robots and cameras with minimal effort, making it a versatile tool for a wide range of robotic applications.
\section{Related works}
\label{sec:related_work}
\paragraph{Large Scale Data Collection:} The data acquisition pipeline represents one of the most critical element of a data-driven robot learning framework. Previous works has employed a diverse array of data acquisition techniques, combining many open-sourced datasets across diverse simulation or real-world data including diverse robot embodiment from many institutions across the globe ~\citep{reed2022a, padalkar2023open, brohan2023rt2, khazatsky2024droid}.
The most common approaches to robot demonstration collection involves pairing the robot or end-effector with remote controller devices or kinematically isomorphic equipment. The devices utilized have a range of complexity and forms: they encompass full robotic exoskeletons ~\citep{zhao2023wearable, ishiguro2020bilateral, fang2023airexo}, as well as simpler data collection tools ~\citep{zhao2023aloha, wu2023gello,fu2024mobile}, and also methods that don't require physically moving a robot ~\citep{shafiullah2023bringing, song2020grasping, pari2021surprising, young2020visual, chi2024universal}. Additionally, various control methods have also been employed, including the use of video game controllers ~\citep{liu2024libero, sian2004whole}, Virtual Reality (VR) devices ~\citep{iyer2024open, cui2022play, cheng2024opentelevisionteleoperationimmersiveactive, yang2024acecrossplatformvisualexoskeletonslowcost,park2024using,arunachalam2022dexterous,arunachalam2023holodex,fu2024humanplus}, and mobile phones~\citep{mandlekar2018roboturk}.
While the most intuitive method is to physically move a real robot, it is both difficult to do and hard to scale to a diverse set of environments. The hardware controller approach can be inefficient because it requires the demonstrator to mentally map robot behavior to controller inputs. The opposite, using a device without moving the robot is efficient in that the demonstrator's movements can be mapped directly to the robot, but it is challenging to apply force feedback. Studies that provides perspective on the relative merits of these two direction are~\citep{shafiullah2023bringing, chi2024universal}, which combines the versatility of simple controller with the intuitiveness of moving a physical end-effector. In this work, we employ a device that inherits and improves the device proposed from ~\citep{shafiullah2023bringing, chi2024universal} for our data collection pipeline.
\paragraph{Pretrained Robot Models:}
Pre-trained foundation models have demonstrated a wide range of generalization performance across various domains, with the capability to learn from internet-scale pre-training data ~\citep{devlin2018bert, radford2021clip, dubey2024llama, kirillov2023segment}. However, in comparison to these vision and language pre-trained models, learning a foundation model for robotics has been considered a relatively challenging area, due to the limited quantity of available datasets ~\citep{kappler2015leveraging, levine2016end, depierre2018jacquard, zhu2023fanuc}, the significant discrepancy across the domains~\citep{dasari2019robonet, kalashnikov2021mt, padalkar2023open}, and the inherently challenging nature of the action datasets in terms of tokenization~\citep{lee2024behavior, brohan2023rt2, zhengprise}.
To address these issues, recent research is increasingly adopting techniques that introduce modular and hierarchical systems, incorporate pre-trained language and visual models~\citep{li2023vision, nair2022r3m, karamcheti2023language, shafiullah2022clip, liu2024ok, gupta2024opening}, and collect large scale data with efficient data collection schemes~\citep{khazatsky2024droid, brohan2023rt2, walke2023bridgedata, ebert2021bridge, fang2023rh20t}. Consequently, they have enabled the pre-trained foundation robot models to exhibit enhanced generalization performance, thereby showcasing that the robotic agents are capable of operating in more than one robot embodiment and operating environment~\citep{team2024octo, reed2022a, kim2024openvla, doshi2024scaling}. In contrast with the aforementioned approaches, which follow a method of training on internet-scale data and fine-tuning on task-specific data, our approach does not expect that the model will have access to a dataset in the environments where the robot is expected to operate. Rather, this project demonstrates the capacity of generalizable performance without a necessity to fine-tune the model for each novel robot embodiment and environment.
\paragraph{Large Models Feedback and Improvement:} Due to their capacity to comprehend intricate semantics and relations, Natural language and Large language models (LLM), have recently been applied to robotic agents powered by imitation learning ~\citep{fried2018speaker-follower, kim2024openvla, shridhar2022cliport, jang2022bc} and reinforcement learning~\citep{du2023guiding, goyal2021pixl2r}.
Among the wide capabilities afforded by language models, those commonly employed in the context of decision-making include providing feedback in the resolution of uncertain information ~\citep{ren2023robots, mullen2024towards, huang2022inner,liu2023reflect,guo2023doremi,park2023clara,gao2024physically}, suggesting affordance of what is possible in the environments by combining with Value functions~\citep{ahn2022can}, and imagination of outcomes \citep{zhang2024rail} or planning and decompose complex tasks into mid-level plans~\citep{song2023llm, huang2022language, zeng2022socratic, sharma2021skill}. Language models could also be used to improve the overall performance of autonomous agent systems by improving reward signal ~\citep{nair2022learning, goyal2021pixl2r, ma2023eureka}, leveraging their long-horizon reasoning ~\citep{dalal2024plan, zhou2023generalizable, blukis2022persistent}, or designing environments ~\citep{ma2024dreureka}. In this project, we employ the mLLM to provide feedback in the form of a reset signal in open-ended environments, a manner analogous to that of the studies above.
\section{Limitations and Conclusion}
While in this work we create~\longname{} that can perform particular tasks zero-shot in novel environments, there are certain limitations that future versions can improve upon. The primary limitation that we see are of hardware: for example, two-fingered grippers like our Stick-v2 are unable to open doors with round doorknobs. Similarly, while flexible fingertips can be more lenient for the policy, it makes it hard to manipulate heavy objects. We encourage more research on better gripper and fingertip design to address these issues. Secondly, we assume navigation to be a separate component, and in this work assume that the robot is in the task space facing the task objective. Combining with modular navigation work such as~\citep{liu2024ok} should address this issue. Finally, for mLLM introspection and retrying, we assume that the errors made by our model (a) leaves the task-space somewhat in-distribution, and (b) allows for an easy reset of the robot to the initial state. Increasing training data with failure recovery behavior in our dataset should let our robots recover more naturally from such failure cases.
\section{Appendix}
\subsection{Detailed Results from Experiments with Self-critique and Retrying}
\label{sec:app:auto-retrying}
\begin{table}[ht]
\centering
\captionsetup{justification=centering}
\begin{tabular}{llr}
\toprule
\textbf{Task} & \textbf{Environment/Object} & \textbf{Success $\cdot / 10$} \\
\midrule
\textbf{Door Opening} & Kitchen Trash Door & 7 \\
& Kitchen Cabinet Door & 10 \\
& Brown Cabinet Door & 10 \\
& Metal Cabinet Door & 10 \\
& White File Cabinet Door & 10 \\
\midrule
\textbf{Drawer Opening} & Kitchen Drawer & 10 \\
& Cloth Drawer & 9 \\
& White File Cabinet Drawer & 10 \\
& Small File Cabinet Drawer & 10 \\
& Dresser Drawer & 8 \\
\midrule
\textbf{Bag Pick Up} & Hollister Bag & 9 \\
& American Eagle Bag & 10 \\
& Qdoba Bag & 8 \\
& Journey's Bag & 9 \\
& Yellow Bag & 6 \\
\midrule
\textbf{Tissue Pick Up} & White Tall Box & 10 \\
& White Short Box & 10 \\
& Black Square Box & 9 \\
& Red Square Box & 10 \\
& Kleenex Box & 7 \\
\midrule
\textbf{Object Reorientation} & Pink Bottle & 9 \\
& White Board Cleaner & 8 \\
& Spices Container & 8 \\
& Coke Can & 8 \\
& Compressed Air & 10 \\
\bottomrule
\end{tabular}
\caption{Detailed success statistics of~\shortname{} on our evaluation environments.}
\label{table:success-statistics-with-retry}
\end{table}
\newpage
\subsection{Evaluation Environments}
\label{sec:app:eval_envs}
\begin{figure}[ht!]
\centering
\includegraphics[width=0.9\linewidth]{figures/eval_envs_1.pdf}
\caption{Picture of evaluation environments for the tasks Reorientation, Drawer opening, and Door opening.}
\label{fig:eval_envs_1}
\end{figure}
\begin{figure}[ht!]
\centering
\includegraphics[width=0.9\linewidth]{figures/eval_envs_2.pdf}
\caption{Pictures of the evaluation environments for the task Tissue pick up and Bag pick up.}
\label{fig:eval_envs_2}
\end{figure}
\newpage
\subsection{Multimodal Large Language Model Prompts for Success Verification}
\label{sec:app:mllm-prompt}
Here, we present the prompt that we use to verify~\shortname{} success with mLLMs.
\definecolor{violetbg}{RGB}{176, 88, 228}
\definecolor{violetframe}{RGB}{112, 41, 156}
\definecolor{white}{RGB}{255, 255, 255}
\tcbset{
myboxstyle/.style={
colback=violetbg!20,
colframe=violetframe,
width=0.8\textwidth,
arc=5mm,
boxrule=0.5mm,
left=5mm,
right=5mm,
top=2mm,
bottom=2mm,
fonttitle=\bfseries,
coltitle=white
}
}
\begin{center}
\begin{tcolorbox}[title=Door Opening, myboxstyle]
As the timesteps progress, does the robotic arm open the door AND is the robot arm grasping the handle in the LAST timestep?\\
\textbf{Please respond with only 'Yes' or 'No'.}
\end{tcolorbox}
\vspace{5mm}
\begin{tcolorbox}[title=Drawer Opening, myboxstyle]
As the timesteps progress, does the robotic arm grasp the drawer handle and open it AND is the drawer open in the last timestep?\\
\textbf{Please respond with only 'Yes' or 'No'.}
\end{tcolorbox}
\vspace{5mm}
\begin{tcolorbox}[title=Reorientation, myboxstyle]
As the timesteps progress, does the robotic arm/gripper reorient the object upright AND is the object upright in the LAST frame?\\
\textbf{Please respond with only 'Yes' or 'No'.}
\end{tcolorbox}
\vspace{5mm}
\begin{tcolorbox}[title=Tissue Pick-Up, myboxstyle]
As the timesteps progress, does the robotic arm/gripper grasp the tissue AND is the gripper grasping the tissue in the LAST timestep?\\
\textbf{Please respond with only 'Yes' or 'No'.}
\end{tcolorbox}
\vspace{5mm}
\begin{tcolorbox}[title=Bag Pick-Up, myboxstyle]
As the timesteps progress, does the robotic arm/gripper grasp the bag AND is the gripper grasping the bag in the LAST timestep?\\
\textbf{Please respond with only 'Yes' or 'No'.}
\end{tcolorbox}
\end{center}
\subsection{Evaluation Schedule}
In Figure~\ref{fig:run-schedule}, we show the starting position of the robot for our 10-run evaluations to understand the positional generalization capabilities of \longname{}.
\begin{figure}[h!]
\centering
\includegraphics[width=0.75\linewidth]{figures/run_schedule.pdf}
\caption{10-run evaluation schedule used to evaluate \longname{}, with robot starting positions denoted by the pale blue dots in the image. We assume that the robot is at the task space facing the object, but it can be at different offsets with respect to the target object. On our object centric tasks (reorientation, bag and tissue pickup) we also randomize the position of the object itself.}
\label{fig:run-schedule}
\end{figure}
\subsection{Bill of Materials}
\label{sec:app:bom}
Here, we present the bill of materials for our hardware components, assuming that the interested researcher or user owns an iPhone Pro already. The total cost comes out to be slightly below $\$25$ for the entire setup.
\begin{table}[ht]
\centering
\captionsetup{justification=centering}
\begin{tabular}{lrrr}
\toprule
\textbf{Item} & \textbf{Price} & \textbf{Unit Price} & \textbf{Qty} \\
\midrule
Reacher Grabber Tool & 26.99 & 13.50 & 1 \\
Brass Tapered Heat-Set Inserts & 21.82 & 0.22 & 3 \\
Thread-Forming Screws & 7.75 & 0.31 & 3 \\
Button Head Screw - M4 x 0.70 - 8mm & 12.91 & 0.13 & 1 \\
Button Head Screw - M4 x 0.70 - 5mm & 8.64 & 0.09 & 2 \\
Button Head Screw - M4 x 0.70 - 35mm & 16.77 & 0.34 & 2 \\
Nylon-Insert Locknut & 5.57 & 0.06 & 2 \\
Dowel Pin & 16.09 & 0.32 & 3 \\
Nylon Unthreaded Spacer & 18.41 & 0.18 & 2 \\
Kevlar Cord & 20.99 & 20.99 & 1/100 \\
Heat Shrink Tubing & 10.79 & 10.79 & 1/30 \\
Black 3D Printer Filament & 25.99 & 25.99 & 3/20 \\
\midrule
\textbf{Total} & & {\textbf{21.99}} \\
\bottomrule
\end{tabular}
\caption{Stick-v2 Main Body}
\end{table}
\begin{table}[ht]
\centering
\captionsetup{justification=centering}
\begin{tabular}{lrrr}
\toprule
\textbf{Item} & \textbf{Price} & \textbf{Unit Price} & \textbf{Qty} \\
\midrule
Socket Head Screw - M3 x 0.5mm - 8mm & 12.52 & 0.13 & 2 \\
Steel Hex Nut - M3 x 0.5mm & 2.62 & 0.03 & 2 \\
M3 Steel Washer & 2.19 & 0.02 & 2 \\
Red 3D Printer Filament & 25.99 & 25.99 & 3/1000 \\
Oomoo 25 Silicone Rubber & 33.99 & 33.99 & 1/200 \\
\midrule
\textbf{Total} & & {\textbf{0.61}} \\
\bottomrule
\end{tabular}
\caption{Gripper Tips}
\end{table}
\begin{table}[ht]
\centering
\captionsetup{justification=centering}
\begin{tabular}{lrrr}
\toprule
\textbf{Item} & \textbf{Price} & \textbf{Unit Price} & \textbf{Qty} \\
\midrule
Socket Head Screw - M5 x 0.8mm - 20mm & 17.10 & 0.17 & 1 \\
Socket Head Screw - M5 x 0.8mm - 50mm & 4.26 & 0.85 & 1 \\
Steel Hex Nut - M5 x 0.8mm & 5.24 & 0.05 & 2 \\
Button Head Screw - M4 x 0.70 - 8mm & 12.91 & 0.13 & 1 \\
Black 3D Printer Filament & 25.99 & 25.99 & 3/20 \\
\midrule
\textbf{Total} & & {\textbf{2.03}} \\
\bottomrule
\end{tabular}
\caption{Phone Holder}
\end{table}
\newpage
\subsection{Deploying on Stretch's Default D405 Camera}
\label{app:sec:d405}
Deploying our~\longname{} on the standard Hello Robot Stretch SE3 requires normalizing the image coming out of the default Intel Realsense D405 wrist camera. We created an affine transformation that maps the D405 image to the same pixel coordinates as the iPhone camera.
\begin{figure}[h!]
\centering
\includegraphics[width=\linewidth]{figures/d405.pdf}
\caption{We can see the corresponding D405 camera image alongside the iPhone Pro image. While in the long range, the images look similar, in the short range iPhone images are out of focus because of the different focal lengths of the cameras.}
\label{fig:d405-transform}
\end{figure}
As we can see from Figure~\ref{fig:d405-transform}, applying the affine transform to the D405 camera maps it to pretty similar viewpoint as the wrist mounted iPhone. While we can run~\shortname{} directly with this camera transform, we see a performance drop which we hypothesize happens because of the especially apparent difference in close-range. This difference is caused by the different focal lengths of the two cameras, and may be solved in the future with image augmentations.
\newpage
\subsection{Failure Modes}
\label{app:sec:failure-modes}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/failure_examples.pdf}
\caption{Examples of some failures in real world rollouts. Since~\shortname{} retries on failure with mLLM feedback, the failure modes tend to be peculiar, some examples of which are shown here.}
\label{fig:failure-examples}
\end{figure}
As we mention in the main paper, with mLLM guided retries, our failures tend to be more peculiar than simply ``robot failed to complete task''. In this section, we try to shine some light on what kind of failures we experience in our system.
\begin{itemize}[leftmargin=*]
\item \textbf{Reorientation:} Primary failure modes for this task are when retry becomes impossible because of environmental issues, such as the target bottle rolling away on the table, being dropped off the surface (an example of which is shown on the Figure~\ref{fig:failure-examples}), pushing it too far into the table (to a position too far for our robot arm), or being rotated sideways by the gripper before grasping. In out-of-distribution surfaces, it can be hard to estimate how large the surface is visually and thus placing the object after reorientation may miss the surface or the robot may run into the surface.
\item \textbf{Drawer opening:} Beyond the most direct failure mode of missing the drawer handle, we experienced some failure modes where the model does not know when to stop pulling on cloth drawers and thus pulls out the entire drawer. Without force feedback, it can be hard to tell visually when the drawer starts sagging. Force feedback on the fingertips would help the robot correct for it.
\item \textbf{Door opening:} Here, the primary failure mode we experience are on unusual doors, such as the trash cabinet door with a hole in it. There, GPT sometimes classifies the door as ``open'' even when it is closed. In some rare cases, when door handles are close together, the robot may grasp around both handles and fail to reset as it gets stuck when retracting.
\item \textbf{Tissue pick up:} The tissue box itself being light and easy to move means that sometimes the box moves with the tissue as its being picked up. As a result, the box may get picked up with the tissue, or get pushed off from its table by the robot (Figure~\ref{fig:failure-examples}.)
\item \textbf{Bag pick up:} The case of bag picking up is interesting because it has one of the highest success rates from the raw~\shortnamesingular{} policy but also sees the smallest improvement (4\%) from GPT feedback. This failure from mLLM feedback happens usually because from the robot wrist or head camera, it can be hard to tell whether the bag has been picked up. As a result, GPT tends to have a high number of false positives for this task. Having a better third-person view of the workspace should help address this issue.
\end{itemize}
\begin{thebibliography}{83}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Sian et~al.(2004)Sian, Yokoi, Kajita, Kanehiro, and Tanie]{sian2004whole}
Neo~Ee Sian, Kazuhito Yokoi, Shuuji Kajita, Fumio Kanehiro, and Kazuo Tanie.
\newblock Whole body teleoperation of a humanoid robot development of a simple master device using joysticks.
\newblock \emph{Journal of the Robotics Society of Japan}, 22\penalty0 (4):\penalty0 519--527, 2004.
\bibitem[Girshick et~al.(2014)Girshick, Donahue, Darrell, and Malik]{girshick2014rich}
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.
\newblock Rich feature hierarchies for accurate object detection and semantic segmentation.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 580--587, 2014.
\bibitem[Gkioxari et~al.(2014)Gkioxari, Hariharan, Girshick, and Malik]{GkioxariHGM14}
Georgia Gkioxari, Bharath Hariharan, Ross~B. Girshick, and Jitendra Malik.
\newblock R-cnns for pose estimation and action detection.
\newblock \emph{CoRR}, abs/1406.5212, 2014.
\newblock URL \url{http://arxiv.org/abs/1406.5212}.
\bibitem[Kappler et~al.(2015)Kappler, Bohg, and Schaal]{kappler2015leveraging}
Daniel Kappler, Jeannette Bohg, and Stefan Schaal.
\newblock Leveraging big data for grasp planning.
\newblock In \emph{ICRA}, 2015.
\bibitem[Levine et~al.(2016)Levine, Finn, Darrell, and Abbeel]{levine2016end}
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.
\newblock End-to-end training of deep visuomotor policies.
\newblock \emph{JMLR}, 17\penalty0 (1):\penalty0 1334--1373, 2016.
\bibitem[Mahler et~al.(2017)Mahler, Liang, Niyaz, Laskey, Doan, Liu, Ojea, and Goldberg]{Mahler2017DexNet2D}
Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan~Aparicio Ojea, and Ken Goldberg.
\newblock Dex-{N}et 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics.
\newblock In \emph{Robotics: Science and Systems (RSS)}, 2017.
\bibitem[Depierre et~al.(2018)Depierre, Dellandr{\'e}a, and Chen]{depierre2018jacquard}
Amaury Depierre, Emmanuel Dellandr{\'e}a, and Liming Chen.
\newblock Jacquard: A large scale dataset for robotic grasp detection.
\newblock In \emph{2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages 3511--3516. IEEE, 2018.
\bibitem[Devlin et~al.(2018)Devlin, Chang, Lee, and Toutanova]{devlin2018bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
\newblock Bert: Pre-training of deep bidirectional transformers for language understanding.
\newblock \emph{arXiv preprint arXiv:1810.04805}, pages 4171--4186, 2018.
\newblock \doi{10.18653/v1/N19-1423}.
\bibitem[Fried et~al.(2018)Fried, Hu, Cirik, Rohrbach, Andreas, Morency, Berg{-}Kirkpatrick, Saenko, Klein, and Darrell]{fried2018speaker-follower}
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis{-}Philippe Morency, Taylor Berg{-}Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell.
\newblock Speaker-follower models for vision-and-language navigation.
\newblock In \emph{Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr{\'{e}}al, Canada}, pages 3318--3329, 2018.
\bibitem[Mandlekar et~al.(2018)Mandlekar, Zhu, Garg, Booher, Spero, Tung, Gao, Emmons, Gupta, Orbay, et~al.]{mandlekar2018roboturk}
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et~al.
\newblock Roboturk: A crowdsourcing platform for robotic skill learning through imitation.
\newblock In \emph{Conference on Robot Learning}, pages 879--893. PMLR, 2018.
\bibitem[Dasari et~al.(2019)Dasari, Ebert, Tian, Nair, Bucher, Schmeckpeper, Singh, Levine, and Finn]{dasari2019robonet}
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn.
\newblock Robo{N}et: Large-scale multi-robot learning.
\newblock In \emph{Conference on Robot Learning ({CoRL})}, volume 100, pages 885--897. PMLR, 2019.
\bibitem[Ishiguro et~al.(2020)Ishiguro, Makabe, Nagamatsu, Kojio, Kojima, Sugai, Kakiuchi, Okada, and Inaba]{ishiguro2020bilateral}
Yasuhiro Ishiguro, Tasuku Makabe, Yuya Nagamatsu, Yuta Kojio, Kunio Kojima, Fumihito Sugai, Yohei Kakiuchi, Kei Okada, and Masayuki Inaba.
\newblock Bilateral humanoid teleoperation system using whole-body exoskeleton cockpit tablis.
\newblock \emph{IEEE Robotics and Automation Letters}, 5\penalty0 (4):\penalty0 6419--6426, 2020.
\bibitem[Song et~al.(2020)Song, Zeng, Lee, and Funkhouser]{song2020grasping}
Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser.
\newblock Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations.
\newblock \emph{IEEE Robotics and Automation Letters}, 5\penalty0 (3):\penalty0 4978--4985, 2020.
\bibitem[Young et~al.(2020)Young, Gandhi, Tulsiani, Gupta, Abbeel, and Pinto]{young2020visual}
Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto.
\newblock Visual imitation made easy.
\newblock \emph{arXiv e-prints}, pages arXiv--2008, 2020.
\bibitem[Goyal et~al.(2021)Goyal, Niekum, and Mooney]{goyal2021pixl2r}
Prasoon Goyal, Scott Niekum, and Raymond Mooney.
\newblock Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards.
\newblock In \emph{Conference on Robot Learning}, pages 485--497. PMLR, 2021.
\bibitem[Jang et~al.(2021)Jang, Irpan, Khansari, Kappler, Ebert, Lynch, Levine, and Finn]{jang2022bc}
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn.
\newblock {BC}-{Z}: Zero-shot task generalization with robotic imitation learning.
\newblock In \emph{Conference on Robot Learning (CoRL)}, pages 991--1002, 2021.
\bibitem[Kalashnikov et~al.(2021)Kalashnikov, Varley, Chebotar, Swanson, Jonschkowski, Finn, Levine, and Hausman]{kalashnikov2021mt}
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman.
\newblock {MT}-{O}pt: Continuous multi-task robotic reinforcement learning at scale.
\newblock \emph{arXiv preprint arXiv:2104.08212}, 2021.
\bibitem[Pari et~al.(2021)Pari, Shafiullah, Arunachalam, and Pinto]{pari2021surprising}
Jyothish Pari, Nur~Muhammad Shafiullah, Sridhar~Pandian Arunachalam, and Lerrel Pinto.
\newblock The surprising effectiveness of representation learning for visual imitation, 2021.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever]{radford2021clip}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
\newblock Learning transferable visual models from natural language supervision.
\newblock In \emph{International Conference on Machine Learning (ICML)}, volume 139, pages 8748--8763, 2021.
\bibitem[Sharma et~al.(2021)Sharma, Torralba, and Andreas]{sharma2021skill}
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas.
\newblock Skill induction and planning with latent language.
\newblock \emph{arXiv preprint arXiv:2110.01517}, 2021.
\bibitem[Sundermeyer et~al.(2021)Sundermeyer, Mousavian, Triebel, and Fox]{sundermeyer2021contact}
Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox.
\newblock Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes.
\newblock In \emph{2021 IEEE International Conference on Robotics and Automation (ICRA)}, pages 13438--13444. IEEE, 2021.
\bibitem[Ahn et~al.(2022)Ahn, Brohan, Brown, Chebotar, Cortes, David, Finn, Fu, Gopalakrishnan, Hausman, et~al.]{ahn2022can}
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et~al.
\newblock Do as i can, not as i say: Grounding language in robotic affordances.
\newblock \emph{arXiv preprint arXiv:2204.01691}, 2022.
\bibitem[Arunachalam et~al.(2022)Arunachalam, Silwal, Evans, and Pinto]{arunachalam2022dexterous}
Sridhar~Pandian Arunachalam, Sneha Silwal, Ben Evans, and Lerrel Pinto.
\newblock Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation.
\newblock \emph{arXiv preprint arXiv:2203.13251}, 2022.
\bibitem[Blukis et~al.(2022)Blukis, Paxton, Fox, Garg, and Artzi]{blukis2022persistent}
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi.
\newblock A persistent spatial semantic representation for high-level natural language instruction execution.
\newblock In \emph{Conference on Robot Learning}, pages 706--717. PMLR, 2022.
\bibitem[Cui et~al.(2022)Cui, Wang, Shafiullah, and Pinto]{cui2022play}
Zichen~Jeff Cui, Yibin Wang, Nur Muhammad~Mahi Shafiullah, and Lerrel Pinto.
\newblock From play to policy: Conditional behavior generation from uncurated robot data.
\newblock \emph{arXiv preprint arXiv:2210.10047}, 2022.
\bibitem[Ebert et~al.(2022)Ebert, Yang, Schmeckpeper, Bucher, Georgakis, Daniilidis, Finn, and Levine]{ebert2021bridge}
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine.
\newblock Bridge data: Boosting generalization of robotic skills with cross-domain datasets.
\newblock In \emph{Robotics: Science and Systems ({RSS}) XVIII}, 2022.
\bibitem[Huang et~al.(2022{\natexlab{a}})Huang, Abbeel, Pathak, and Mordatch]{huang2022language}
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.
\newblock Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.
\newblock In \emph{International conference on machine learning}, pages 9118--9147. PMLR, 2022{\natexlab{a}}.
\bibitem[Huang et~al.(2022{\natexlab{b}})Huang, Xia, Xiao, Chan, Liang, Florence, Zeng, Tompson, Mordatch, Chebotar, et~al.]{huang2022inner}
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et~al.
\newblock Inner monologue: Embodied reasoning through planning with language models.
\newblock \emph{arXiv preprint arXiv:2207.05608}, 2022{\natexlab{b}}.
\bibitem[Nair et~al.(2022{\natexlab{a}})Nair, Mitchell, Chen, Savarese, Finn, et~al.]{nair2022learning}
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et~al.
\newblock Learning language-conditioned robot behavior from offline data and crowd-sourced annotation.
\newblock In \emph{Conference on Robot Learning}, pages 1303--1315. PMLR, 2022{\natexlab{a}}.
\bibitem[Nair et~al.(2022{\natexlab{b}})Nair, Rajeswaran, Kumar, Finn, and Gupta]{nair2022r3m}
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.
\newblock R3m: A universal visual representation for robot manipulation.
\newblock In \emph{CoRL}, 2022{\natexlab{b}}.
\bibitem[Reed et~al.(2022)Reed, Zolna, Parisotto, Colmenarejo, Novikov, Barth-maron, Gim{\'e}nez, Sulsky, Kay, Springenberg, Eccles, Bruce, Razavi, Edwards, Heess, Chen, Hadsell, Vinyals, Bordbar, and de~Freitas]{reed2022a}
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio~G{\'o}mez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Gim{\'e}nez, Yury Sulsky, Jackie Kay, Jost~Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de~Freitas.
\newblock A generalist agent.
\newblock \emph{Transactions on Machine Learning Research}, 2022.
\newblock ISSN 2835-8856.
\bibitem[Shafiullah et~al.(2022)Shafiullah, Paxton, Pinto, Chintala, and Szlam]{shafiullah2022clip}
Nur Muhammad~Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam.
\newblock Clip-fields: Weakly supervised semantic fields for robotic memory.
\newblock \emph{arXiv preprint arXiv:2210.05663}, 2022.
\bibitem[Shridhar et~al.(2022)Shridhar, Manuelli, and Fox]{shridhar2022cliport}
Mohit Shridhar, Lucas Manuelli, and Dieter Fox.
\newblock Cliport: What and where pathways for robotic manipulation.
\newblock In \emph{Conference on Robot Learning}, pages 894--906. PMLR, 2022.
\bibitem[Zeng et~al.(2022)Zeng, Attarian, Ichter, Choromanski, Wong, Welker, Tombari, Purohit, Ryoo, Sindhwani, et~al.]{zeng2022socratic}
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et~al.
\newblock Socratic models: Composing zero-shot multimodal reasoning with language.
\newblock \emph{arXiv preprint arXiv:2204.00598}, 2022.
\bibitem[Arunachalam et~al.(2023{\natexlab{a}})Arunachalam, G{\"u}zey, Chintala, and Pinto]{arunachalam2023holodex}
Sridhar~Pandian Arunachalam, Irmak G{\"u}zey, Soumith Chintala, and Lerrel Pinto.
\newblock Holo-dex: Teaching dexterity with immersive mixed reality.
\newblock In \emph{2023 IEEE International Conference on Robotics and Automation (ICRA)}, pages 5962--5969. IEEE, 2023{\natexlab{a}}.
\bibitem[Arunachalam et~al.(2023{\natexlab{b}})Arunachalam, Silwal, Evans, and Pinto]{arunachalam2023dexterous}
Sridhar~Pandian Arunachalam, Sneha Silwal, Ben Evans, and Lerrel Pinto.
\newblock Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation.
\newblock In \emph{2023 ieee international conference on robotics and automation (icra)}, pages 5954--5961. IEEE, 2023{\natexlab{b}}.
\bibitem[Brohan et~al.(2023)Brohan, Brown, Carbajal, Chebotar, Chen, Choromanski, Ding, Driess, Dubey, Finn, et~al.]{brohan2023rt2}
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi~Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et~al.
\newblock Rt-2: Vision-language-action models transfer web knowledge to robotic control.
\newblock \emph{arXiv preprint arXiv:2307.15818}, 2023.
\bibitem[Chi et~al.(2023)Chi, Feng, Du, Xu, Cousineau, Burchfiel, and Song]{chi2023diffusion}
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song.
\newblock Diffusion policy: Visuomotor policy learning via action diffusion.
\newblock \emph{arXiv preprint arXiv:2303.04137}, 2023.
\bibitem[Du et~al.(2023)Du, Watkins, Wang, Colas, Darrell, Abbeel, Gupta, and Andreas]{du2023guiding}
Yuqing Du, Olivia Watkins, Zihan Wang, C{\'e}dric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas.
\newblock Guiding pretraining in reinforcement learning with large language models.
\newblock In \emph{International Conference on Machine Learning}, pages 8657--8677. PMLR, 2023.
\bibitem[Fang et~al.(2023{\natexlab{a}})Fang, Fang, Tang, Liu, Wang, Zhu, and Lu]{fang2023rh20t}
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Junbo Wang, Haoyi Zhu, and Cewu Lu.
\newblock {RH20T}: A robotic dataset for learning diverse skills in one-shot.
\newblock In \emph{RSS 2023 Workshop on Learning for Task and Motion Planning}, 2023{\natexlab{a}}.
\bibitem[Fang et~al.(2023{\natexlab{b}})Fang, Wang, Fang, Gou, Liu, Yan, Liu, Xie, and Lu]{fang2023anygrasp}
Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu.
\newblock Anygrasp: Robust and efficient grasp perception in spatial and temporal domains.
\newblock \emph{IEEE Transactions on Robotics}, 2023{\natexlab{b}}.
\bibitem[Fang et~al.(2023{\natexlab{c}})Fang, Fang, Wang, Ren, Chen, Zhang, Wang, and Lu]{fang2023airexo}
Hongjie Fang, Hao-Shu Fang, Yiming Wang, Jieji Ren, Jingjing Chen, Ruo Zhang, Weiming Wang, and Cewu Lu.
\newblock Low-cost exoskeletons for learning whole-arm manipulation in the wild.
\newblock \emph{arXiv preprint arXiv:2309.14975}, 2023{\natexlab{c}}.
\bibitem[Guo et~al.(2023)Guo, Wang, Zha, Jiang, and Chen]{guo2023doremi}
Yanjiang Guo, Yen-Jen Wang, Lihan Zha, Zheyuan Jiang, and Jianyu Chen.
\newblock Doremi: Grounding language model by detecting and recovering from plan-execution misalignment.
\newblock \emph{arXiv preprint arXiv:2307.00329}, 2023.
\bibitem[Karamcheti et~al.(2023)Karamcheti, Nair, Chen, Kollar, Finn, Sadigh, and Liang]{karamcheti2023language}
Siddharth Karamcheti, Suraj Nair, Annie~S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang.
\newblock Language-driven representation learning for robotics.
\newblock \emph{Robotics: Science and Systems (RSS)}, 2023.
\bibitem[Kirillov et~al.(2023)Kirillov, Mintun, Ravi, Mao, Rolland, Gustafson, Xiao, Whitehead, Berg, Lo, et~al.]{kirillov2023segment}
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander~C Berg, Wan-Yen Lo, et~al.
\newblock Segment anything.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 4015--4026, 2023.
\bibitem[Li et~al.(2023)Li, Liu, Zhang, Yu, Xu, Wu, Cheang, Jing, Zhang, Liu, et~al.]{li2023vision}
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya~Jing, Weinan Zhang, Huaping Liu, et~al.
\newblock Vision-language foundation models as effective robot imitators.
\newblock \emph{arXiv preprint arXiv:2311.01378}, 2023.
\bibitem[Liu et~al.(2023)Liu, Bahety, and Song]{liu2023reflect}
Zeyi Liu, Arpit Bahety, and Shuran Song.
\newblock Reflect: Summarizing robot experiences for failure explanation and correction.
\newblock \emph{arXiv preprint arXiv:2306.15724}, 2023.
\bibitem[Ma et~al.(2023)Ma, Liang, Wang, Huang, Bastani, Jayaraman, Zhu, Fan, and Anandkumar]{ma2023eureka}
Yecheng~Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
\newblock Eureka: Human-level reward design via coding large language models.
\newblock \emph{arXiv preprint arXiv:2310.12931}, 2023.
\bibitem[Padalkar et~al.(2023)Padalkar, Pooley, Jain, Bewley, Herzog, Irpan, Khazatsky, Rai, Singh, Brohan, et~al.]{padalkar2023open}
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et~al.
\newblock Open x-embodiment: Robotic learning datasets and rt-x models.
\newblock \emph{arXiv preprint arXiv:2310.08864}, 2023.
\bibitem[Park et~al.(2023)Park, Lim, Lee, Park, Chang, Yu, and Choi]{park2023clara}
Jeongeun Park, Seungwon Lim, Joonhyung Lee, Sangbeom Park, Minsuk Chang, Youngjae Yu, and Sungjoon Choi.
\newblock Clara: classifying and disambiguating user commands for reliable interactive robotic agents.
\newblock \emph{IEEE Robotics and Automation Letters}, 2023.
\bibitem[Ren et~al.(2023)Ren, Dixit, Bodrova, Singh, Tu, Brown, Xu, Takayama, Xia, Varley, et~al.]{ren2023robots}
Allen~Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et~al.
\newblock Robots that ask for help: Uncertainty alignment for large language model planners.
\newblock \emph{arXiv preprint arXiv:2307.01928}, 2023.
\bibitem[Shafiullah et~al.(2023)Shafiullah, Rai, Etukuru, Liu, Misra, Chintala, and Pinto]{shafiullah2023bringing}
Nur Muhammad~Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto.
\newblock On bringing robots home.
\newblock \emph{arXiv preprint arXiv:2311.16098}, 2023.
\bibitem[Song et~al.(2023)Song, Wu, Washington, Sadler, Chao, and Su]{song2023llm}
Chan~Hee Song, Jiaman Wu, Clayton Washington, Brian~M Sadler, Wei-Lun Chao, and Yu~Su.
\newblock Llm-planner: Few-shot grounded planning for embodied agents with large language models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 2998--3009, 2023.
\bibitem[Walke et~al.(2023)Walke, Black, Lee, Kim, Du, Zheng, Zhao, Hansen-Estruch, Vuong, He, Myers, Fang, Finn, and Levine]{walke2023bridgedata}
Homer Walke, Kevin Black, Abraham Lee, Moo~Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine.
\newblock Bridgedata v2: A dataset for robot learning at scale, 2023.
\bibitem[Wu et~al.(2023)Wu, Shentu, Yi, Lin, and Abbeel]{wu2023gello}
Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel.
\newblock Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators.
\newblock \emph{arXiv preprint arXiv:2309.13037}, 2023.
\bibitem[Zhao et~al.(2023{\natexlab{a}})Zhao, Yang, Yang, and Yu]{zhao2023wearable}
Liang Zhao, Tie Yang, Yang Yang, and Peng Yu.
\newblock A wearable upper limb exoskeleton for intuitive teleoperation of anthropomorphic manipulators.
\newblock \emph{Machines}, 11\penalty0 (4):\penalty0 441, 2023{\natexlab{a}}.
\bibitem[Zhao et~al.(2023{\natexlab{b}})Zhao, Kumar, Levine, and Finn]{zhao2023aloha}
Tony~Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.
\newblock Learning fine-grained bimanual manipulation with low-cost hardware.
\newblock \emph{arXiv preprint arXiv:2304.13705}, 2023{\natexlab{b}}.
\bibitem[Zhou et~al.(2023)Zhou, Ding, Peng, Tomizuka, Shao, and Gan]{zhou2023generalizable}
Haoyu Zhou, Mingyu Ding, Weikun Peng, Masayoshi Tomizuka, Lin Shao, and Chuang Gan.
\newblock Generalizable long-horizon manipulations with large language models.
\newblock \emph{arXiv preprint arXiv:2310.02264}, 2023.
\bibitem[Zhu et~al.(2023)Zhu, Tian, Xu, Huo, Zhan, Tomizuka, and Ding]{zhu2023fanuc}
Xinghao Zhu, Ran Tian, Chenfeng Xu, Mingxiao Huo, Wei Zhan, Masayoshi Tomizuka, and Mingyu Ding.
\newblock Fanuc manipulation: A dataset for learning-based manipulation with fanuc mate 200i{D} robot.
\newblock \url{https://sites.google.com/berkeley.edu/fanuc-manipulation}, 2023.
\bibitem[Cheng et~al.(2024)Cheng, Li, Yang, Yang, and Wang]{cheng2024opentelevisionteleoperationimmersiveactive}
Xuxin Cheng, Jialong Li, Shiqi Yang, Ge~Yang, and Xiaolong Wang.
\newblock Open-television: Teleoperation with immersive active visual feedback, 2024.
\newblock URL \url{https://arxiv.org/abs/2407.01512}.
\bibitem[Chi et~al.(2024)Chi, Xu, Pan, Cousineau, Burchfiel, Feng, Tedrake, and Song]{chi2024universal}
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song.
\newblock Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots.
\newblock \emph{arXiv preprint arXiv:2402.10329}, 2024.
\bibitem[Dalal et~al.(2024)Dalal, Chiruvolu, Chaplot, and Salakhutdinov]{dalal2024plan}
Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov.
\newblock Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks.
\newblock \emph{arXiv preprint arXiv:2405.01534}, 2024.
\bibitem[Doshi et~al.(2024)Doshi, Walke, Mees, Dasari, and Levine]{doshi2024scaling}
Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine.
\newblock Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation.
\newblock \emph{arXiv preprint arXiv:2408.11812}, 2024.
\bibitem[Dubey et~al.(2024)Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Yang, Fan, et~al.]{dubey2024llama}
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et~al.
\newblock The llama 3 herd of models.
\newblock \emph{arXiv preprint arXiv:2407.21783}, 2024.
\bibitem[Fu et~al.(2024{\natexlab{a}})Fu, Zhao, Wu, Wetzstein, and Finn]{fu2024humanplus}
Zipeng Fu, Qingqing Zhao, Qi~Wu, Gordon Wetzstein, and Chelsea Finn.
\newblock Humanplus: Humanoid shadowing and imitation from humans.
\newblock \emph{arXiv preprint arXiv:2406.10454}, 2024{\natexlab{a}}.
\bibitem[Fu et~al.(2024{\natexlab{b}})Fu, Zhao, and Finn]{fu2024mobile}
Zipeng Fu, Tony~Z Zhao, and Chelsea Finn.
\newblock Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.
\newblock \emph{arXiv preprint arXiv:2401.02117}, 2024{\natexlab{b}}.
\bibitem[Gao et~al.(2024)Gao, Sarkar, Xia, Xiao, Wu, Ichter, Majumdar, and Sadigh]{gao2024physically}
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh.
\newblock Physically grounded vision-language models for robotic manipulation.
\newblock In \emph{2024 IEEE International Conference on Robotics and Automation (ICRA)}, pages 12462--12469. IEEE, 2024.
\bibitem[Gupta et~al.(2024)Gupta, Zhang, Sathua, and Gupta]{gupta2024opening}
Arjun Gupta, Michelle Zhang, Rishik Sathua, and Saurabh Gupta.
\newblock Opening cabinets and drawers in the real world using a commodity mobile manipulator.
\newblock \emph{arXiv preprint arXiv:2402.17767}, 2024.
\bibitem[Haldar et~al.(2024)Haldar, Peng, and Pinto]{haldar2024baku}
Siddhant Haldar, Zhuoran Peng, and Lerrel Pinto.
\newblock Baku: An efficient transformer for multi-task policy learning.
\newblock \emph{arXiv preprint arXiv:2406.07539}, 2024.
\bibitem[Iyer et~al.(2024)Iyer, Peng, Dai, Guzey, Haldar, Chintala, and Pinto]{iyer2024open}
Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey, Siddhant Haldar, Soumith Chintala, and Lerrel Pinto.
\newblock Open teach: A versatile teleoperation system for robotic manipulation.
\newblock \emph{arXiv preprint arXiv:2403.07870}, 2024.
\bibitem[Khazatsky et~al.(2024)Khazatsky, Pertsch, Nair, Balakrishna, Dasari, Karamcheti, Nasiriany, Srirama, Chen, Ellis, et~al.]{khazatsky2024droid}
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan~Kumar Srirama, Lawrence~Yunliang Chen, Kirsty Ellis, et~al.
\newblock Droid: A large-scale in-the-wild robot manipulation dataset.
\newblock \emph{arXiv preprint arXiv:2403.12945}, 2024.
\bibitem[Kim et~al.(2024)Kim, Pertsch, Karamcheti, Xiao, Balakrishna, Nair, Rafailov, Foster, Lam, Sanketi, et~al.]{kim2024openvla}
Moo~Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et~al.
\newblock Openvla: An open-source vision-language-action model.
\newblock \emph{arXiv preprint arXiv:2406.09246}, 2024.
\bibitem[Lee et~al.(2024)Lee, Wang, Etukuru, Kim, Shafiullah, and Pinto]{lee2024behavior}
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H~Jin Kim, Nur Muhammad~Mahi Shafiullah, and Lerrel Pinto.
\newblock Behavior generation with latent actions.
\newblock \emph{arXiv preprint arXiv:2403.03181}, 2024.
\bibitem[Lin et~al.(2024)Lin, Zhang, Li, Qi, Yi, Levine, and Malik]{lin2024learning}
Toru Lin, Yu~Zhang, Qiyang Li, Haozhi Qi, Brent Yi, Sergey Levine, and Jitendra Malik.
\newblock Learning visuotactile skills with two multifingered hands.
\newblock \emph{arXiv preprint arXiv:2404.16823}, 2024.
\bibitem[Liu et~al.(2024{\natexlab{a}})Liu, Zhu, Gao, Feng, Liu, Zhu, and Stone]{liu2024libero}
Bo~Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone.
\newblock Libero: Benchmarking knowledge transfer for lifelong robot learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024{\natexlab{a}}.
\bibitem[Liu et~al.(2024{\natexlab{b}})Liu, Orru, Paxton, Shafiullah, and Pinto]{liu2024ok}
Peiqi Liu, Yaswanth Orru, Chris Paxton, Nur Muhammad~Mahi Shafiullah, and Lerrel Pinto.
\newblock Ok-robot: What really matters in integrating open-knowledge models for robotics.
\newblock \emph{arXiv preprint arXiv:2401.12202}, 2024{\natexlab{b}}.
\bibitem[Ma et~al.(2024)Ma, Liang, Wang, Wang, Zhu, Fan, Bastani, and Jayaraman]{ma2024dreureka}
Yecheng~Jason Ma, William Liang, Hung-Ju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman.
\newblock Dreureka: Language model guided sim-to-real transfer.
\newblock \emph{arXiv preprint arXiv:2406.01967}, 2024.
\bibitem[Mullen~Jr and Manocha(2024)]{mullen2024towards}
James~F Mullen~Jr and Dinesh Manocha.
\newblock Towards robots that know when they need help: Affordance-based uncertainty for large language model planners.
\newblock \emph{arXiv preprint arXiv:2403.13198}, 2024.
\bibitem[Park and Agrawal(2024)]{park2024using}
Younghyo Park and Pulkit Agrawal.
\newblock Using apple vision pro to train and control robots, 2024.
\bibitem[Team et~al.(2024)Team, Ghosh, Walke, Pertsch, Black, Mees, Dasari, Hejna, Kreiman, Xu, et~al.]{team2024octo}
Octo~Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et~al.
\newblock Octo: An open-source generalist robot policy.
\newblock \emph{arXiv preprint arXiv:2405.12213}, 2024.
\bibitem[Yang et~al.(2024)Yang, Liu, Qin, Ding, Li, Cheng, Yang, Yi, and Wang]{yang2024acecrossplatformvisualexoskeletonslowcost}
Shiqi Yang, Minghuan Liu, Yuzhe Qin, Runyu Ding, Jialong Li, Xuxin Cheng, Ruihan Yang, Sha Yi, and Xiaolong Wang.
\newblock Ace: A cross-platform visual-exoskeletons system for low-cost dexterous teleoperation, 2024.
\newblock URL \url{https://arxiv.org/abs/2408.11805}.
\bibitem[Zhang et~al.(2024)Zhang, Meng, Qi, and Chirikjian]{zhang2024rail}
Ceng Zhang, Xin Meng, Dongchen Qi, and Gregory~S Chirikjian.
\newblock Rail: Robot affordance imagination with large language models.
\newblock \emph{arXiv preprint arXiv:2403.19369}, 2024.
\bibitem[Zheng et~al.(2024)Zheng, Cheng, Daum{\'e}~III, Huang, and Kolobov]{zhengprise}
Ruijie Zheng, Ching-An Cheng, Hal Daum{\'e}~III, Furong Huang, and Andrey Kolobov.
\newblock Prise: Llm-style sequence compression for learning temporal action abstractions in control.
\newblock In \emph{Forty-first International Conference on Machine Learning}, 2024.
\end{thebibliography}