SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes
Extracted source text ยท arXiv 2306.15620
\documentclass{article}
%=========================packages=====================================================================
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{bm}
\usepackage{adjustbox}
\usepackage{multirow}
\usepackage{wrapfig}
%\usepackage{xcolor} % COMMENT for final draft
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{mathtools}
\usepackage{hyperref}
\usepackage{booktabs} % professional-quality tables
\usepackage{subcaption}
\usepackage{microtype} % microtypography
\usepackage{pifont}
\usepackage{nicefrac}
\captionsetup[sub]{labelformat=simple}
\renewcommand{\thesubfigure}{\arabic{figure}.\alph{subfigure}}
%\usepackage{corl_2023} % Use this for the initial submission.
%\usepackage[final]{corl_2023} % Uncomment for the camera-ready ``final'' version.
\usepackage[preprint]{corl_2023} % Uncomment for pre-prints (e.g., arxiv); This is like ``final'', but will remove the CORL footnote.
%=============end packages=============
%===========custom commands============
% BLUE -- Yu
% RED -- Ninad
\newcommand{\yu}[1]{\textcolor{blue}{[Yu: #1]}}
\newcommand{\nina}[1]{\textcolor{red}{[Ninad: #1]}}
\newcommand{\todo}[1]{\textcolor{red}{[TODO: #1]}}
\newcommand{\jishnu}[1]{\textcolor{magenta}{\hspace{0.1cm}\textbf{[Jishnu: #1]}}}
\newcommand{\sai}[1]{\textcolor{orange}{[Sai: #1]}}
%======end custom commands=========
%========title and authors=========
\title{\input{includes/title}}
% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.
% NOTE: authors will be visible only in the camera-ready and preprint versions (i.e., when using the option 'final' or 'preprint').
% For the initial submission the authors will be anonymized.
\author{\input{includes/authors}}
%=============end title and authors===========================
%============================================
\begin{document}
\maketitle
%=====================================================
\begin{abstract}
\input{includes/abstract}
\end{abstract}
% Two or three meaningful keywords should be added here
\vspace{-3mm}
\keywords{Robot Manipulation, Benchmarking, Real World Scene Replication}
%=========================Introduction===================================================================
\vspace{-3mm}
\section{Introduction}
\vspace{-1mm}
\label{sec:introduction}
Benchmarking has been crucial for successfully boosting the rate of research in machine learning-related communities. With a common benchmark for experiments, different algorithms can be compared in a fair and standardized manner, and a relative improvement between them can be established.
% to show improvement on solving problems supported by the benchmark.
Perhaps the most well-known benchmark in AI research is the ImageNet dataset~\cite{deng2009imagenet} and its associated challenge~\cite{russakovsky2015imagenet}. It has been used to train and test image classification methods and can be attributed to triggering the network architecture revolution in deep learning. The KITTI dataset~\cite{geiger2013vision} has been widely used to study problems related to autonomous driving such as car detection and tracking, lane segmentation, and odometry estimation. In NLP, a wide variety of benchmarks were introduced with different levels of difficulty, such as GLUE~\cite{wang2018glue}, SuperGLUE~\cite{wang2019superglue} and MMLU~\cite{hendrycks2020measuring}. In robotics, researchers have proposed several benchmarks for different tasks, such as the NIST assembly boards~\cite{kimble2020benchmarking} for assembly and the GRASPA~\cite{bottarel2020graspa} for grasping. However, most of them have not seen widespread adoption and usage, thereby limiting the rate of research progress as it becomes harder to compare methods and investigate their true applicability on a problem.
% the success of benchmarking in robotics is not as great as in other AI subfields. The community has not established a widely used benchmark analog to ImageNet in computer vision and deep learning.
The main difficulty in benchmarking in robotics is that robots need to perform tasks in the real world involving a complex pipeline instead of running experiments on fixed datasets.
Fixed datasets have the advantage that every algorithm can be evaluated on the same data for a fair comparison. Real-world tasks are hampered by this requirement of a common test dataset, and it is difficult to make sure that the test scenarios are exactly the same across different research labs. Despite this, the robotics community has made progress along several interesting directions for benchmarking, especially robot manipulation. First, for assembly tasks, the NIST assembly boards~\cite{kimble2020benchmarking} can be used as a reproducible test bed. Second, the YCB Object and Model Set~\cite{calli2015ycb} provides a set of objects with their 3D mesh models that are accessible to all researchers. Although these objects are limited to dozens of rigid objects, using these objects for manipulation provides the same accessible test objects for experiments. Third, several benchmarking protocols for real-world manipulation are introduced using YCB objects. These protocols show how to arrange objects in the real world to create reproducible scenes. For example, \cite{bekiroglu2019benchmarking} introduces a grasp planning protocol that uses drawing on the robot workspace for the placement of objects. GRASPA~\cite{bottarel2020graspa} uses printable boards with AR tags for
reproducible placement of YCB objects. However, \cite{bekiroglu2019benchmarking} cannot guarantee accurate reproducible object placement, while \cite{bottarel2020graspa} has to use AR tags which create unnatural scenes for testing.
In this work, we introduce a new benchmark, SceneReplica, for real-world robot manipulation to overcome the limitations of existing benchmarks. First, we use 16 YCB objects which can be easily purchased either online (online purchase links provided) or in person. Second, we created a dataset of 20 scenes with 5 YCB objects in each scene as a test bed for pick-and-place. We introduce a sequential algorithm for scene generation and selection in simulation to create these 20 scenes. The algorithm considers the reachability space of a robot and the diversity of poses of objects in building the real-world test bed. Finally, we provide an easy-to-use tool to guide users in reproducing these 20 scenes in the real world without using any AR tag or external equipment. To the best of our knowledge, SceneReplica is the only benchmark for real-world reproducible scenes without using AR markers or external cutouts to guide object placement.
% rather than randomly placing objects or dropping everything on table
Another contribution of our work is the benchmarking of two main paradigms for 6D robotic grasping using SceneReplica: model-based grasping and model-free grasping. Model-based grasping leverages 3D models of objects for perception and planning. It consists of a pipeline of using 6D object pose estimation for perception~\cite{xiang2017posecnn,tremblay2018deep,deng2021poserbpf}, 3D model-based grasp planning~\cite{miller2004graspit,diankov2008openrave} and motion planning and motion control~\cite{sucan2012open} to reach the planned grasps. Model-free grasping does not assume the availability of 3D object models. It consists of a pipeline of unseen object instance segmentation~\cite{xie2020best,xiang2021learning,lu2022mean}, object point cloud-based grasp planning~\cite{mousavian20196,sundermeyer2021contact} and motion planning and motion control~\cite{sucan2012open} to reach the planned grasps. Different object perception, grasping planning, motion planning, and control methods can be evaluated within the two 6D robotic grasping paradigms. This high-level view of robot manipulation benchmarking is a departure from existing benchmarks which are specific to some tasks or do not give details about the complete pipeline. We also perform experiments and provide our analyses on a selected set of representative methods. To the best of our knowledge, it is the first effort in which model-based and model-free grasping pipelines are systematically evaluated and fairly compared to each other using the same real-world testing scenes.
By creating SceneReplica, we provide the robotics community with an easy-to-setup, reproducible real-world benchmark for robot manipulation tasks. We plan to continuously maintain the benchmark and evaluate additional and upcoming perception, planning, and control methods on it. It is our hope that these results, which combine effects from the entire pipeline, can serve as a reference for robot manipulation research.
%=========================End Instroduction==============================================================
%========================================================================================================
\begin{table}
\centering
\begin{adjustbox}{max width=0.9\textwidth}
\begin{tabular}{@{}lccccc@{}}
% \hrule
Benchmark & Type & Task & Objects & AR Tag-Free & Scene
Reproducibility \\ \midrule
Meta-World~\cite{yu2020meta} & Simulation & 50 tasks & Synthetic & \textcolor{ForestGreen}{\ding{51}} & \textcolor{ForestGreen}{\ding{51}} \\
RLBench~\cite{james2020rlbench} & Simulation & 100 Tasks & Synthetic & \textcolor{ForestGreen}{\ding{51}} & \textcolor{ForestGreen}{\ding{51}}\\
robosuite~\cite{zhu2020robosuite} & Simulation & 9 Tasks & Synthetic & \textcolor{ForestGreen}{\ding{51}} & \textcolor{ForestGreen}{\ding{51}} \\
\hline
Grasp Planning Protocol~\cite{bekiroglu2019benchmarking} & Real & Grasp Planning & YCB (single) & \textcolor{ForestGreen}{\ding{51}} & \textcolor{red}{\ding{55}} \\
NIST Assembly~\cite{kimble2020benchmarking} & Real & Assembly & Task Boards & \textcolor{ForestGreen}{\ding{51}} & \textcolor{ForestGreen}{\ding{51}} \\
FurnitureBench~\cite{heo2023furniturebench} & Real & Assembly & 3D Printing & \textcolor{red}{\ding{55}} & \textcolor{ForestGreen}{\ding{51}} \\
GRASPA~\cite{bottarel2020graspa} & Real & Grasping & YCB (clutter) & \textcolor{red}{\ding{55}} & \textcolor{ForestGreen}{\ding{51}} \\
OCRTOC~\cite{liu2021ocrtoc} & Real & Rearrangement & YCB + Others & \textcolor{ForestGreen}{\ding{51}} & \textcolor{red}{\ding{55}} \\
\multirow{2}{*}{RB2~\cite{dasari2022rb2}} & \multirow{2}{*}{Real} & Pouring, Scooping& \multirow{2}{*}{Others} & \multirow{2}{*}{\textcolor{ForestGreen}{\ding{51}}} & \multirow{2}{*}{\textcolor{red}{\ding{55}}} \\
& & Zipping, Insertion \\
Box and Blocks Test~\cite{morgan2019benchmarking} & Real & Pick-and-Place & Blocks & \textcolor{ForestGreen}{\ding{51}} & \textcolor{red}{\ding{55}} \\
\textbf{SceneReplica (Ours)} & Real & Pick-and-Place & YCB (clutter) & \textcolor{ForestGreen}{\ding{51}} & \textcolor{ForestGreen}{\ding{51}} \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Comparison between different robotic manipulation benchmarks.}
\label{tab:benchmark}
\vspace{-8mm}
\end{table}
%=========================RELATED WORK =================================================================
\vspace{-4mm}
\section{Related Work}
\vspace{-2mm}
\label{sec:related-work}
% Existing robotic benchmarks can be classified into simulation-based ones and real-world ones.
\textbf{Simulation-based Robotic Benchmarks.} Since reproducibility in the real world is challenging, the setup of testing environments in physics simulators provides a workaround for benchmarking in robotics. For example, the Meta-World~\cite{yu2020meta} benchmark introduces 50 robotic manipulation tasks, which can be used to study meta-reinforcement learning and multitask learning. The robosuite~\cite{zhu2020robosuite} based on the MuJoCo physics engine~\cite{todorov2012mujoco} supports 9 manipulation tasks for different robots. The RLBench benchmark~\cite{james2020rlbench} increases the number of tasks to 100. The advantage of using simulation is that reproducibility is guaranteed for evaluation, and a large number of tasks can be used. However, due to the sim-to-real gap, the performance in simulation cannot simply be transferred to the real world. Although we can work on improving the physics simulators, nuances in the real world such as lighting, friction, external perturbation, etc., are extremely difficult to model in simulation. We need to have real-world benchmarks for robotics.
\textbf{Real-world Robotic Benchmarks.} We summarize a list of representative efforts to benchmark robot manipulation in the real world in Table~\ref{tab:benchmark}. According to the evaluation tasks, they can be classified into assembly~\cite{kimble2020benchmarking,heo2023furniturebench}, grasping~\cite{bekiroglu2019benchmarking,morgan2019benchmarking,bottarel2020graspa}, rearrangement~\cite{liu2021ocrtoc}, and others (pouring, scooping zipping, insertion in~\cite{dasari2022rb2}). In terms of scene reproducibility in evaluation, NIST Assembly~\cite{kimble2020benchmarking}, FurnitureBench~\cite{heo2023furniturebench} and GRASPA~\cite{bottarel2020graspa} are the only three benchmarks whose test scenes are reproducible. NIST Assembly uses a fixed set of assembly task boards, while FurnitureBench and GRASPA use AR markers to guide the scene creation process. For grapsing and picking-and-place, SceneReplica is the only benchmark that can create reproducible scenes without using AR markers in the real world.
%=========================End RELATED WORK =============================================================
%=======================================================================================================
%=========================METHOD========================================================================
\vspace{-2mm}
\section{The SceneReplica Benchmark}
\vspace{-2mm}
\label{sec:method}
Our goal is to introduce reproducible scenes of objects for robot manipulation, where researchers can set up the same set of scenes in different environments for evaluation. Currently, we focus on tabletop cluttered scenes for pick-and-place. The configurations of objects in these scenes are generated in the Gazebo simulation environment to obtain the ground truth data and in order to make it easy to satisfy certain properties of the scenes. Generating in simulation enables access to complete state information, allowing for a seamless sim2real transfer for reproducible real-world experiments.
\subsection{Creating Scenes in Simulation}
\begin{figure}[h]
\centering
\vspace{-3mm} \includegraphics[width=0.9\columnwidth]{figures/ycb.pdf}
\vspace{-3mm}
\caption{16 YCB objects are used in SceneReplica.}
\label{fig:ycb}
\vspace{-2mm}
\end{figure}
\textbf{Object Set.} 16 YCB objects~\cite{calli2015ycb} are used in our benchmark to create scenes. Their 3D mesh models are shown in Fig.~\ref{fig:ycb}. We select these objects because i) they can be easily purchased online or in grocery stores; ii) they are a subset of the YCB Video dataset~\cite{xiang2017posecnn} which has been widely used to study 6D object pose estimation. Therefore, we can leverage existing 6D object pose estimation methods for experiments, for example,~\cite{xiang2017posecnn,deng2021poserbpf}. % since many of the robotics papers utilize this set of objects.
\textbf{Stable Poses of Objects.} To place objects on a table in Gazebo, one option is to drop them from a certain height above the table and then let the physics simulator handle the computation for the final position of the objects. However, we found that this process requires careful parameter tuning of the mass and inertia of objects, and waiting for the objects to settle down is slow in practice. Therefore, we opt to compute stable poses of objects which are object orientations when they rest statically on a planar surface. Such stable poses are also safe to use in real world scenes as the chance of an object falling over and disturbing the scene is low. We used the \url{compute\_stable\_poses()} function in trimesh~\cite{trimesh} to obtain such poses, and Fig.~\ref{fig:ycb} shows examples of stable poses of some YCB objects.
\textbf{Determining the Reachable Space of a Robot.} When placing these objects on a table for manipulation, we need to place them within the reach of the robot. We employ a simple algorithm as depicted in Fig.~\ref{fig:reach} to compute the reachable regions of a tabletop. In Step I, the robot model is loaded in Gazebo and maneuvered to the stow position. Next, a table of dimension $1m$ x $1m$ x $0.745m$ is placed at an offset of $(0.8,0,0)$ with respect to the robot model, where the $x$-axis is the forward direction of the robot. In Step II, the surface of the table is partitioned into grids $16\times 16$ and a block of size $3cm \times 3cm$ is placed in the center of each cell of the grid. In Step III, we determine whether there is a feasible motion plan for the stand-off pose $S_{di}$ for each cube \textit{i}. Blocks that do not have a feasible plan will be removed from the scene. Steps II - III are run for 20 iterations. Finally, the remaining blocks after the iterative process indicate discretized reachable locations for the robot. These locations will be used to place YCB objects to create scenes.
% \begin{enumerate}
% \item describe how the reachable space has been determined. Cubes spawned over entire table -- check motion plan for each location -- remove unreachable points --- run multiple iterations of this -- get the reachable points -- determine the limits
% \item insert gazebo pictures of cubes 1.a all cubes spawned 1.b some cubes removed in the middle of iterations 1.c final space
% \item Reachable space $\mathbb{\textbf{R}}$ of the robotic manipulator here is defined as the set of possible locations where an object can have a valid motion plan from the manipulator. In order to determine $\mathbb{\textbf{R}}$, we placed the robot at a selected location, $[0,0,0]$ here. A table of dimension $1m$ x $1m$ x $0.775m$ is placed at $[0.8,0,0]$ with respect to robot. To determine possible locations of valid motion plan, a grid of $16$ x $16$ has been generated and a cube of dimension $3x3$ has been spawned at each of the grid location
% \end{enumerate}
% \begin{figure*}[t!]
% \centering
% \begin{subfigure}[t]
% \includegraphics[width=4.2cm, height=3.37cm]{figures/reachable_space/initial_plan.png}
% \caption{before planning}
% \end{subfigure}
% \begin{subfigure}[t]
% \includegraphics[width=4.2cm, height=3.37cm]{figures/reachable_space/intermediate_plan.png}
% \caption{while planning}
% \end{subfigure}
% \begin{subfigure}[t]
% \includegraphics[width=4.2cm, height=3.37cm]{figures/reachable_space/final_plan.png}
% \caption{Post planning}
% \end{subfigure}
% \caption{Caption}
% \label{fig:my_label}
% \end{figure*}
% \begin{figure*}[t!]
% \centering
% \begin{subfigure}{.3\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/reachable_space/initial_plan.png}
% \caption{Before planning}
% \label{fig:sub1}
% \end{subfigure}\hfill
% \begin{subfigure}{.3\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/reachable_space/intermediate_plan.png}
% \caption{While planning}
% \label{fig:sub2}
% \end{subfigure}\hfill
% \begin{subfigure}{.3\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/reachable_space/final_plan.png}
% \caption{Post planning}
% \label{fig:sub3}
% \end{subfigure}
% \caption{For a tabletop scene, we use motion planning to see if a Fetch robot can pick up these blue blocks on the table to determine the reachability of the robot. After planning, remaining blocks on the table indicate reachable points of the tabletop.}
% \label{fig:reach}
% \vspace{-4mm}
% \end{figure*}
\begin{figure*}
\centering
\includegraphics[width=0.9\columnwidth]{figures/reachibility.pdf}
\caption{For a tabletop scene, we use motion planning to see if a Fetch robot can pick up these blue blocks on the table to determine the reachability of the robot. After planning, the remaining blocks on the table indicate reachable points of the tabletop.}
\label{fig:reach}
\vspace{-6mm}
\end{figure*}
\textbf{Scene Generation.} Using the stable poses of the 16 YCB objects, we can generate random scenes in Gazebo by placing them in the reachable locations of the Fetch robot. We limit the maximum number of objects in a scene to be five and sequentially add objects to the scene one by one. Initially, an object is placed randomly on the table. When adding the next object, we randomly sample a nearby reachable location with an existing object that is collision-free between these objects. Stable poses of objects are also randomly sampled during placement. This process repeats until five objects are added to the scene. By sampling nearby reachable locations, we tend to generate cluttered scenes where objects occlude each other. In total, we have generated 164 scenes as candidates for scene selection as described below.
% \begin{enumerate}
% \item describe first how the stable poses are selected from set of possible poses.
% \item describe the search process. during this describe how the scenes are generated with valid motion plan for all objects
% \end{enumerate}
% \begin{figure}[!h]
% \centering
% \begin{subfigure}{\textwidth}
% \centering
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000010.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000025.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000027.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000033.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000036.png}
% \vspace{1pt}
% \end{subfigure}
% \begin{subfigure}{\textwidth}
% \centering
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000038.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000039.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000048.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000056.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000065.png}
% \vspace{1pt}
% \end{subfigure}
% \begin{subfigure}{\textwidth}
% \centering
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000068.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000077.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000083.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000084.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000104.png}
% \vspace{1pt}
% \end{subfigure}
% \begin{subfigure}{\textwidth}
% \centering
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000122.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000130.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000141.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000148.png}
% \includegraphics[width=2.7cm]{figures/selected-scenes/color-000161.png}
% \vspace{1pt}
% \end{subfigure}
% \caption{20 scenes in our SceneReplica benchmark with 5 YCB objects in each scene.}
% \label{fig:scenes}
% \end{figure}
\begin{figure*}[h]
\centering
\includegraphics[width=1.0\columnwidth]{figures/scenes.pdf}
\caption{20 scenes in our SceneReplica benchmark with 5 YCB objects in each scene.}
\label{fig:scenes}
\vspace{-4mm}
\end{figure*}
\textbf{Scene Selection.} After generating a set of scenes, our remaining task is to select a subset of it as the final scenes for evaluation. We target at selecting 20 scenes and wish to have the following properties of the selected scenes. i) The number of appearances of each object should be balanced in the selected scenes. We define a range $[C_\text{min}, C_\text{max}]$ such that the counter of each object should be in this range. ii) The selected scenes should cover the stable poses of each object as much as possible. iii) All the objects in a selected scene should be reachable from the robot. To check the reachability of objects in a scene, we use GraspIt! to generate 100 grasps of each YCB object, and check if there is a motion plan of the Fetch robot to reach one of the planned grasps.
With these three properties, our algorithm for scene selection works as follows. We generate a large number of scene sets. Each scene set contains 20 scenes, and it satisfies properties i) and iii). In addition, each set of scenes is given a score on how many stable poses of these 16 YCB objects are covered by the 20 scenes in it. This acts as a measure for the diversity of a set. Finally, we select the scene set with the highest pose coverage score. Fig.~\ref{fig:scenes} shows the final 20 scenes selected for SceneReplica.
% Execution of our experiments involved an important decision of selecting $\mathbf{N_{S}}$ scenes out of the total generated $\mathbf{N_{T}}$ scenes.
% This selection plays a key role in ensuring the reliability and validity of our tests.
%However, an issue arises if these $ \mathbf{N_{S}}$ scenes are selected at random. Random selection, although seem impartial, can lead to the issue of bias towards a particular set of objects in the scenes. The bias here means that some objects could be over-represented in comparison to other objects in the selected scenes. This would skew the results and undermine the generalizability of algorithms.
% Hence it is essential that the scenes are at least uniformly selected with respect to objects
%For this project we decided to select $\mathbf{N_{S}}=20$.
% write here that you would describe eliminating this in next subsection
% In addition to ensuring balanced representation of different objects in the selected scenes, we also account for the variety of object poses. Including a variety of poses for each object in the selected scenes ensures that the algorithms are tested under a wide range of manipulation scenarios.
% \todo{modify this to discuss variety in poses. and then discuss how bias, variety are addressed individual. combine then in final algorithm.}
% To reduce bias of some objects representation in the selected scenes, we adopted a strategy that minimizes this bias and lead to scene selection with more uniform representation of all the objects of interest. In lieu of this we introduce two parameters namely \textit{minimum occurance} $\mathbf{C_{min}}$, \textit{maximum occurance} $\mathbf{C_{max}}$. $\mathbf{C_{min}}$ ensures that each object is represented a certain number of times in the selected scenes, giving minimum level of exposure to it in the experiments. $\mathbf{C_{max}}$ places a limit on the number of times an object can appear in the selected scenes. This cap prevents any object or a set of objects from dominating the scene selection. Through this we create a balanced and representative set of scenes for experiments.
% \todo{creates algo for bias elimination, diversity selection along with varibales definition}
% This algo is just v1. need to update this after final divserity selection
% \begin{algorithm}
% \caption{Scene Selection Algorithm}
% \begin{algorithmic}[1]
% \State Let $C_{min}$, $C_{max}$ be the minimum and maximum occurrence count
% \State Let $O_T$ be the set of objects of interest
% \State Let $C_T$ be the occurrence count of all objects
% \State Let $S$ be the set of all scenes
% \State Let $NS$ be the desired number of scenes
% \For{each scene in $S$}
% \State Let $O_S$ be the set of objects in the scene
% \If{count of each object in $O_S$ in $C_T < C_{max}$}
% \State Select the scene
% \State Update $C_T$ with the objects in the scene
% \If{number of selected scenes $ = NS$}
% \State Break
% \EndIf
% \For{each object in $C_T$}
% \If{count of object $ < C_{min}$}
% \State Scene selection failed, go back to line 6
% \EndIf
% \EndFor
% \EndIf
% \EndFor
% \end{algorithmic}
% \end{algorithm}
% \begin{enumerate}
% \item Present the idea behind choice of scenes. min, max
% \item make an algorithm describing the selection process. explain the parameters and show a picture of distribution
% \end{enumerate}
% Although bias has been reduced through the strategy discussed in \ref{subsec: bias_elim}, it is possible that the selected scenes may not contain diverse poses for some or all objects. To ensure that the selected scenes contain diverse poses for objects, we generated $m$ datasets each containing $N_{S}$ scenes selected after running the scenes through bias reduction. To select the dataset with most diverse poses of the objects, diversity s
% \begin{enumerate}
% \item explain how diversity is defined
% \item explain how it is calculated from selected scenes .. entropy
% \end{enumerate}
\begin{figure*}
\centering
\includegraphics[width=0.95\columnwidth]{figures/replication.pdf}
\caption{The process of replicating a scene in the real world. The image of the reference scene is overlaid on the image of the real camera to guide how to place objects into the real-world scene.}
\label{fig:replication}
\vspace{-5mm}
\end{figure*}
\vspace{-2mm}
\subsection{Reproducing Scenes in the Real World}
\vspace{-2mm}
\begin{wrapfigure}{r}{0.25\textwidth}
\centering
\vspace{-3mm}
\includegraphics[width=0.25\textwidth]{figures/frames.pdf}
\caption{Illustration of the coordinate frames.}
\vspace{-3mm}
\label{fig:frames}
\end{wrapfigure}
To use the 20 selected scenes for experiments, we need to replicate these scenes in the real world. Previous benchmarks rely on AR markers to create reproducible scenes~\cite{bottarel2020graspa,heo2023furniturebench}. In contrast, we show that directly using the scene images in Fig.~\ref{fig:scenes} as reference images, we can place objects in the real world as in the simulator. The process of replicating a scene in the real world is illustrated in Fig.~\ref{fig:replication}. First, the Fetch camera pose in the real world is set to be the same as in the simulation. Therefore, the camera extrinsics, that is, the SE(3) transformation from the robot base frame to the RGB camera frame, is always the same. This can be done by controlling the Fetch head pose. Second, the table height in the real world should be the same as in the simulation, i.e., $0.745m$. We assume that researchers can use a height-adjustable table for experiments. Third, with the same camera extrinsics and table height, we can overlay the half-transparent version of an RGB image of a reference scene to the real RGB image from the Fetch camera, which visually tells a user how to place these YCB objects. Finally, the user can place objects by interactively checking the overlap between the reference image and the real-world image for accurate alignment (Fig.~\ref{fig:replication}). This process makes sure that the poses of objects in the camera frame are replicated according to the reference image. Since camera extrinsics are the same in simulation and in the real world, object poses in the robot base frame are replicated for manipulation. Fig.~\ref{fig:frames} illustrates these coordinate frames.
% \begin{enumerate}
% \item describe the requirements
% \item talk about how the reference scene images are published
% \item describe how we overlay and explain the simplicity
% \end{enumerate}
%=========================End METHOD====================================================================
%=======================================================================================================
%=========================METRICS======================================================================
% \begin{table}[b]
% \centering
% \begin{adjustbox}{width=\linewidth}
% \begin{tabular}{|p{4cm}|p{10cm}|p{2cm}|}
% \hline
% \textbf{Definition} & \textbf{Meaning} & \textbf{Range} \\
% \hline
% Reachability score & Accounts for whether the object is located in a region characterized by a good reachability of the robot. \textcolor{blue}{We are generating the plans such that each object is in its stable pose and is reachable} & [0, 1] \\
% \hline
% Camera-calibration score & Accounts for whether the object is located in a region characterized by a good calibration of the vision system. \textcolor{blue}{not required for this project Now} & [0, 1] \\
% \hline
% Graspability score & Accounts for whether the object can be physically grasped and lifted by the robot, considering its shape and weight.\textcolor{blue}{we can use this to justify the choice of selection of ycb objects. but how is this useful in evaluating the algorithms?} & \{0, 1\} \\
% \hline
% Grasp quality score & Accounts for how contacts are distributed on the object by simulating grasp closure in simulation and computing the grasp wrench space. \textcolor{blue}{we can explain the method we used to determine the choice of grasp. But this can be a choice for evaluating grasp algorithms. is this required now ?}& [0, 1] \\
% \hline
% Binary success score & Accounts for whether the robot actually managed to grasp the object in real tests. \textcolor{red}{Here they are just testing by lifting the object to 15cm height and holding it for 5 seconds. Instead we can alter this metric by doing some operations like roll, pitch} & [0, 1] \\
% \hline
% Grasp stability score & Evaluates the stability of the grasp during the execution of a fixed trajectory. \textcolor{blue}{they are actually doing what we thought in previous row. Gives a sequence of 5 rotations and checks it}& [0, 1] \\
% \hline
% Obstacle avoidance score (Only in cluttered mode) & Accounts for how many objects the robot has hit while executing the grasp. \textcolor{blue}{We are generating the plans such that it avoid the obstacles. all of our plans are motion feasible} & [0, 1] \\
% \hline
% Final per object score & Combines all the scores in order to evaluate the grasping pipeline performance taking into account any limitation of the grasping pipeline \textcolor{blue}{final score is a combination of Grasp quality score, Obstacle avoidance score, Grasp stability score, averaged over all the objects present in the scene}. & [0, 2] (in isolation) or [0, 3] (in clutter) \\
% \hline
% \end{tabular}
% \end{adjustbox}
% \caption{Metrics details. \todo{Needs a good caption}}
% \label{tab: metrics-details}
% \end{table}
%=========================End METRICS===================================================================
%=======================================================================================================
%=========================DATASET=======================================================================
\vspace{-2mm}
\section{Model-based and Model-free 6D Grasping}
\vspace{-2mm}
\label{sec:dataset}
The manipulation task we benchmark with SceneReplica is tabletop pick-and-place. A robot needs to pick up objects and place them on a fixed location on the table. The task requires 6D grasping, where the 3D orientation and 3D translation of a robot gripper is controlled, in contrast to top-down grasping. In the literature, 6D grasping approaches can be classified into model-based ones (Fig.~\ref{fig:pipeline-modelbased}) and model-free ones (Fig.~\ref{fig:pipeline-modelfree}), depending on the availability of 3D object models.
%. Model-based methods leverage 3D models of objects for perception and planning, while model-free approaches do not assume the availability of object 3D models. Both types of approaches consist of a pipeline of object perception, grasp planning, motion planning and control.
% They mainly differ on object perception and grasp planning. Because once a grasp is planed, motion planning algorithms such as OMPL~\cite{sucan2012open} can be used to plan a trajectory to reach the grasp.
\vspace{-2mm}
\subsection{Model-based 6D Grasping}
\vspace{-2mm}
\begin{figure*}[h]
\centering
\includegraphics[width=0.95\columnwidth]{figures/pipeline/pipeline_modelbased.pdf}
\vspace{-2mm}
\caption{Model-based grasping pipeline used for benchmarking}
\label{fig:pipeline-modelbased}
\vspace{-3mm}
\end{figure*}
% Fig.~\ref{fig:pipeline-modelbased} illustrates the pipeline of model-based 6D grasping.
\textbf{Model-based Object Perception.} When 3D models of objects are available, we can tackle the \emph{6D object pose estimation} problem for object perception. The goal is to estimate the 3D orientation and 3D translation of the object with respect to the camera frame. 6D object pose estimation is an active research area in both robotics and computer vision~\cite{workshop}. When considering the problem as a pure perception problem, we can evaluate the accuracy of pose estimation on existing datasets. However, it is unclear how the accuracy of the pose estimation affects grasping. In SceneReplica, we combine 6D object pose estimation methods with planning and control for 6D grasping. Consequently, we can analyze the effect of object pose estimation accuracy on the grasping success rate.
\textbf{Model-based Grasp Planning.} Model-based grasp planning methods such as GraspIt!~\cite{miller2004graspit} and OpenRAVE~\cite{diankov2008openrave} use 3D models of objects in physics simulators to generate grasps. We use GraspIt! in our experiments. We initially generated a large number of grasps from GraspIt! for each YCB object and later sampled the initial set to generate a fixed amount of grasps using the farthest point sampling algorithm which ensures diversity of gripper position with respect to the object.
\textbf{Model-based Motion Planning.} Once the 6D pose of a target object is estimated, we can transform the planned grasps on the 3D object model to the robot base frame. Then we simply loop over each grasp and use a motion planning algorithm such as algorithms in the open motion planning library (OMPL)~\cite{sucan2012open} to see if there is a trajectory to reach the grasp. Once a trajectory is returned, we break the loop and execute the trajectory to grasp the target. Since poses of all the objects in the scene can be estimated, we can use the 3D models of objects as obstacles in the planning scene.
\vspace{-2mm}
\subsection{Model-free 6D Grasping}
\vspace{-2mm}
\begin{figure*}[h]
\centering
\includegraphics[width=0.95\columnwidth]{figures/pipeline/pipeline_modelfree.pdf}
\vspace{-1mm}
\caption{Model-free grasping pipeline used for benchmarking}
\label{fig:pipeline-modelfree}
\vspace{-3mm}
\end{figure*}
% Fig.~\ref{fig:pipeline-modelfree} illustrates the pipeline of model-free 6D grasping.
\textbf{Model-free Object Perception.} When is no 3D model of objects available, we can tackle the \emph{unseen object instance segmentation (UOIS)} problem~\cite{xie2020best,xiang2021learning,lu2022mean} to segment arbitrary objects. Once the objects are segmented, model-free grasp planning methods can be used to generate grasps of the segmented objects. UOIS methods are usually trained using synthetic images of a large number of objects. They learn the concepts of objects and are able to segment novel objects in the real world.
\textbf{Model-free Grasp Planning.} Grasp planning methods such as 6D GraspNet~\cite{mousavian20196,sundermeyer2021contact} take point clouds of objects or whole scenes as input and output grasps of objects in the scenes. These methods are based on trained neural networks for grasp synthesis. Similarly to unseen object instance segmentation in perception, they are trained with large numbers of objects with their corresponding grasps. Therefore, they can be applied for grasp planning of novel objects using partial point clouds.
\textbf{Model-free Motion Planning.} Once a target object is segmented and grasps of the target object are synthesized, similar to model-based motion planning, we can utilize a motion planning algorithm to generate a trajectory to reach one of the planned grasps. The main difference between model-based motion planning is that we do not have 3D models of objects to set up the planning scene for obstacle avoidance. Therefore, we simply compute an oriented 3D bounding box for each segmented object for obstacle avoidance during motion planning.
%=========================End DATASET==================================================================
%======================================================================================================
%=========================EXPERIMENTS===================================================================
\vspace{-3mm}
\section{Experiments}
\vspace{-3mm}
\label{sec:experiments}
For pick-and-place tasks, the most important evaluation metric is the success rate of transporting objects from one location to another. The success rate depends on all the components of a framework. For example, no matter how good the perception is, if grasp planning of an object has error, the robot would not be able to pick up the object. Therefore, we evaluated the success rate of pick-and-place of the 100 objects in SceneReplica, that is, 20 scenes $\times$ 5 objects per scene. In addition, we provide perception evaluation and failure analysis of the results.
\textbf{Perception Metrics.} We consider two perception problems in SceneReplica: 6D object pose estimation for model-based grasping and unseen object instance segmentation for model-free grasping. We follow the standard evaluation metrics for these two problems. We use the average distance for symmetric objects (ADD-S) metric as in \cite{xiang2017posecnn} for pose estimation. For unseen object instance segmentation, we follow~\cite{xie2020best,xiang2021learning} to compute precision, recall, F-measure for region overlap and boundary overlap, as well as the percentage of segmented objects with Overlap F-measure $\geq 75 \%$.
% Given the ground truth rotation $\mathbf{R}$ and translation $\mathbf{T}$ and the estimated rotation $\mathbf{\tilde{R}}$ and translation $\mathbf{\tilde{T}}$, ADD-S is computed as $\textsc{ADD-S} = \frac{1}{m}\sum_{\mathbf{x}_1 \in \mathcal{M}} \min_{\mathbf{x}_2 \in \mathcal{M}} \| (\mathbf{R} \mathbf{x}_1 + \mathbf{T}) - (\mathbf{\tilde{R}} \mathbf{x}_2 + \mathbf{\tilde{T}}) \|$, where $\mathcal{M}$ denotes the set of 3D model points, i.e., 3D vertices of the mesh model and $m$ is the number of points. For unseen object instance segmentation, we follow~\cite{xie2020best,xiang2021learning} to compute precision, recall and F-measure as $P=\frac{\sum_i\left|c_i \cap g\left(c_i\right)\right|}{\sum_i\left|c_i\right|}, R=\frac{\sum_i\left|c_i \cap g\left(c_i\right)\right|}{\sum_j\left|g_j\right|}, F=\frac{2 P R}{P+R}$, where $c_i$ denotes the segmentation of predicted object $i, g\left(c_i\right)$ denotes the segmentation of the matched ground truth object of $c_i$, and $g_j$ is the segmentation for ground truth object $j$. The intersection of a predicted mask and a GT mask can be computed based on region area overlap or region boundary overlap. Therefore, we have Overlap $\mathrm{P} / \mathrm{R} / \mathrm{F}$ and Boundary $\mathrm{P} / \mathrm{R} / \mathrm{F}$. In addition. Overlap F-measure $75 \%$ is the percentage of segmented objects with Overlap F-measure $\geq 75 \%$.
\begin{table*}
\vspace{-2mm}
\centering
\resizebox{0.8\textwidth}{!}{
\begin{tabular}{c|c|c}
\hline
Type & Phase & Description \\ \hline
Perception failure & Pre-Grasping & Target object is not recognized, for example, no pose estimation or no segmentation mask \\
Perception failure & Pre-Grasping & Grasp not found, perception error of obstacle $>$ threshold \\
Perception failure & During-Grasping & Grasp planned, failed to grasp and lift, perception error of target $>$ threshold \\
Perception failure & During-Grasping & Grasp planned, hit obstacle, perception error of obstacle $>$ threshold \\ \hline
Planning failure & Pre-Grasping & No perception failure, but no plan found to grasp target \\
Planning failure & During-Grasping & Grasp planned, no perception failure, failed to grasp and lift or hit obstacle \\ \hline
Execution failure & Post-Grasping & Failed to place object after grasping and lifting \\
\hline
\end{tabular}
}
\caption{Different types of failures in our pick-and-place experimental analysis.}
\vspace{-1mm}
\label{tab:failures}
\end{table*}
\textbf{Pick-and-place Failure Analysis.} When a pick-and-place failure occurred, the failure can be classified into several different categories: (1) perception failure (P$_{E}$F), (2) planning failure (P$_{L}$F) and (3) execution failure (EF). Table~\ref{tab:failures} describes these failures in detail.
\begin{table*}
\vspace{-2mm}
\resizebox{\textwidth}{!}{
\begin{tabular}{c|c|c|c|c|ccc}
\hline
Method \# & Perception & Grasp Planning & Motion Planning & Control & Ordering & Pick-and-Place Success & Grasping Success \\ \hline
\multicolumn{7}{c}{Model-based Grasping} \\ \hline
1 & PoseRBPF~\cite{deng2021poserbpf} & GraspIt!~\cite{miller2004graspit} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Near-to-far & 58 / 100 & 64 / 100\\
1 & PoseRBPF~\cite{deng2021poserbpf} & GraspIt!~\cite{miller2004graspit} + Top-down & OMPL~\cite{sucan2012open} & MoveIt & Fixed & 59 / 100 & 59 / 100\\
2 & PoseCNN~\cite{xiang2017posecnn} & GraspIt!~\cite{miller2004graspit} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Near-to-far & 48 / 100 & 50 / 100\\
2 & PoseCNN~\cite{xiang2017posecnn} & GraspIt!~\cite{miller2004graspit} + Top-down & OMPL~\cite{sucan2012open} & MoveIt & Fixed & 38 / 100 & 44 / 100\\ \hline
\multicolumn{7}{c}{Model-free Grasping} \\ \hline
3 & UCN~\cite{xiang2021learning} & GraspNet~\cite{mousavian20196} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Near-to-far & 43 / 100 & 46 / 100\\
3 & UCN~\cite{xiang2021learning} & GraspNet~\cite{mousavian20196} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Fixed & 39 / 100 & 42 / 100 \\
4 & UCN~\cite{xiang2021learning} & Contact-graspnet~\cite{sundermeyer2021contact} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Near-to-far & 60 / 100 & 63 / 100\\
4 & UCN~\cite{xiang2021learning} & Contact-graspnet~\cite{sundermeyer2021contact} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Fixed & 60 / 100 & 64 / 100 \\
5 & MSMFormer~\cite{lu2022mean} & GraspNet~\cite{mousavian20196} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Near-to-far & 38 / 100 & 41 / 100 \\
5 & MSMFormer~\cite{lu2022mean} & GraspNet~\cite{mousavian20196} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Fixed & 36 / 100 & 41 / 100 \\
6 & MSMFormer~\cite{lu2022mean} & Contact-graspnet~\cite{sundermeyer2021contact} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Near-to-far & 57 / 100 & 65 / 100\\
6 & MSMFormer~\cite{lu2022mean} & Contact-graspnet~\cite{sundermeyer2021contact} + Top-down & OMPL~\cite{sucan2012open} &
MoveIt & Fixed & 61 / 100 & 70 / 100\\
\hline
\end{tabular}
}
\caption{Six frameworks of model-based and model-free grasping evaluated on SceneReplica.}
\label{tab:grasp_methods}
\vspace{-6mm}
\end{table*}
\begin{wraptable}{R}{0.55\textwidth}
\centering
\vspace{-3mm}
\scalebox{0.5}{
\begin{tabular}{|c|c||c|ccccccc|}
\hline
\multirow{2}{*}{Pose Method} & \multirow{2}{*}{ADD-S} &
\multirow{2}{*}{Segmentation Method} & \multicolumn{3}{c|}{Overlap} & \multicolumn{3}{c|}{Boundary} & \\ & & & P & R & \multicolumn{1}{c|}{F} & P & R & \multicolumn{1}{c|}{F} & \%75 \\ \hline
PoseCNN~\cite{xiang2017posecnn} & 66.01
& UCN+ \cite{xiang2021learning} & 70.0 & 81.2 & \multicolumn{1}{c|}{71.9} & 48.3 &57.2 & \multicolumn{1}{c|}{50.3} & 77.6 \\
PoseRBPF~\cite{deng2021poserbpf} & 68.73 & MSMFormer+ \cite{lu2022mean} &76.1 & 83.3 & \multicolumn{1}{c|}{76.3} & 56.0 & 61.4 & \multicolumn{1}{c|}{57.0} & 80.2 \\ \hline
\end{tabular}
}
\caption{Perception evaluation of pose estimation and unseen object instance segmentation in the real world.}
\label{table:perception}
\vspace{-3mm}
\end{wraptable}
\vspace{-2mm}
\subsection{Benchmarking Results}
\vspace{-2mm}
Table~\ref{tab:grasp_methods} shows the six different grasping frameworks that we have benchmarked.
\textbf{Grasping Orders.} For grasping in cluttered scenes, the order of grasping objects matters. We evaluate two orders in SceneReplica: the near-to-far order based on object perception and the fixed order we defined for each scene. Intuitively, the near-to-far order is simpler, since the robot can clear up closer objects first, which are occluders of other objects.
\textbf{Model-based Grasping Frameworks.} For model-based grasping, we have evaluated two 6D object pose estimation methods: PoseCNN~\cite{xiang2017posecnn} and PoseRBPF~\cite{deng2021poserbpf}. GraspIt!~\cite{miller2004graspit} is used for grasp planning in model-based grasping. The OMPL library~\cite{sucan2012open} in MoveIt is used to plan the trajectory of the robot arm and control to follow the planned trajectory. In cases when there is no motion plan to reach all the grasps from GraspIt!, we simply apply top-down grasping to grasp the target object.
From Table~\ref{tab:grasp_methods} we can see that the PoseRBPF-based framework achieves a higher success rate in both grasping and pick-and-place compared to the PoseCNN-based framework. PoseRBPF is a pose-tracking-based estimation method that can leverage temporal information to stabilize pose estimation compared to PoseCNN. The accuracy of pose estimation in terms of ADD-S is presented in Table~\ref{table:perception}, which is calculated using 200 images in the real world, that is, 20 scenes $\times$ 5 grasps per scene $\times$ 2 orders. A better pose estimation model improves the success rate.
\textbf{Model-free Grasping Frameworks.} We have evaluated two unseen object segmentation methods: UCN~\cite{xiang2021learning} and MSMFormer~\cite{lu2022mean}, two grasp planning methods: 6D GraspNet~\cite{mousavian20196} and Contact-GraspNet~\cite{sundermeyer2021contact}. The OMPL library~\cite{sucan2012open} in MoveIt used for motion planning and control. Similarly, top-down grasping is used if there is no plan from the GrapsNets. From the results in Table~\ref{tab:grasp_methods}, we can see that the Contact-GraspNet-based methods achieve a higher success rate than the 6D GraspNet-based methods. Contact-GraspNet grasps usually have higher quality by modeling the contact between the gripper with the target. When comparing the two segmentation methods, MSMFormer achieves higher segmentation accuracy, as shown in Table~\ref{table:perception}. However, UCN-based methods have a higher pick-and-place success rate using 6D GraspNet. We noticed that synthesized grasps using UCN segmentation is better for 6D GraspNet. This result highlights the importance of evaluating the whole pipeline in manipulation. It is also interesting to note that the model-free method \#6 achieves a higher grasping success rate than the model-based methods \#1 and \#2.
\begin{figure*}[h]
\centering
\includegraphics[width=0.95\columnwidth]{figures/failure_examples.pdf}
\caption{Example failures in our experiment pipelines. Green boxes indicate the targets for grasping.}
\label{fig:failure-example}
\vspace{-3mm}
\end{figure*}
\textbf{Failure Analysis.} Fig.~\ref{fig:failure-example} shows some examples of different failure types. Table~\ref{tab:grasp_objects} presents a detailed success and failure analysis for each object and each method. We can see that perception errors and planning errors are dominant and that perception errors are relatively more frequent. Small and thin objects, such as scissors and the marker, are more difficult to grasp.
\begin{table*}
\resizebox{\textwidth}{!}
{
\begin{tabular}{l|c|cccc|cccc|cccc|cccc|cccc|cccc}
\hline
\multirow{2}{*}{Object} & \multirow{2}{*}{Count} & \multicolumn{4}{c|}{Method 1} & \multicolumn{4}{c|}{Method 2} & \multicolumn{4}{c|}{Method 3} & \multicolumn{4}{c|}{Method 4} & \multicolumn{4}{c|}{Method 5} & \multicolumn{4}{c}{Method 6} \\ \cline{3-26}
& & S & P$_{E}$F & P$_{L}$F & EF & S & P$_{E}$F & P$_{L}$F & EF & S & P$_{E}$F & P$_{L}$F & EF & S & P$_{E}$F & P$_{L}$F & EF & S & P$_{E}$F & P$_{L}$F & EF & S & P$_{E}$F & P$_{L}$F & EF \\ \hline
\multicolumn{26}{c}{Near-to-far Ordering} \\ \hline
003 cracker box & 6 & 5 & - & 1 & -
& 1 & 3 & 1 & 1
& 2 & 2 & 2 & -
& 3 & 1 & 2 & -
& 2 & 3 & 1 & -
& 4 & 1 & - & 1\\
004 sugar box & 5 & 5 & - & - & -
& 1 & 4 & - & -
& 2 & 2 & 1 & -
& 5 & - & - & -
& 3 & 1 & 1 & -
& 5 & - & - & -\\
005 tomato soup can & 7 & 6 & 1 & - & -
& 6 & 1 & - & -
& 3 & - & 4 & -
& 6 & - & 1 & -
& 2 & - & 5 & -
& 2 & 2 & 3 & -\\
006 mustard bottle & 7 & 6 & 1 & - & -
& 3 & 2 & 2 & -
& 2 & 1 & 4 & -
& 5 & 1 & 1 & -
& 1 & - & 5 & 1
& 6 & - & 1 & -\\
007 tuna fish can & 6 & 1 & 1 & 4 & -
& 3 & 2 & 1 & -
& 6 & - & - & -
& 5 & 1 & - & -
& 5 & - & 1 & -
& 5 & 1 & - & -\\
008 pudding box & 5 & 5 & - & - & -
& 4 & 1 & - & -
& 3 & - & 2 & -
& 4 & - & 1 & -
& 4 & - & 1 & -
& 4 & 1 & - & -\\
009 gelatin box & 7 & 3 & 4 & - & -
& 2 & 4 & 1 & -
& 3 & 2 & 1 & 1
& 7 & - & - & -
& 4 & - & 3 & -
& 6 & - & 1 & -\\
010 potted meat can & 7 & 6 & 1 & - & -
& 3 & 2 & 2 & -
& 4 & 1 & 2 & -
& 3 & 2 & 1 & 1
& 1 & - & 6 & -
& 5 & 2 & - & -\\
011 banana & 7 & 4 & - & 2 & 1
& 3 & 3 & 1 & -
& 1 & - & 6 & -
& 2 & - & 5 & -
& 5 & - & 2 & -
& 6 & - & - & 1\\
021 bleach cleanser & 5 & 3 & - & 1 & 1
& 1 & 1 & 2 & 1
& 2 & 1 & 1 & 1
& 2 & - & 2 & 1
& - & 1 & 3 & 1
& - & 1 & 2 & 2\\
024 bowl & 7 & 2 & 4 & 1 & -
& 5 & 2 & - & -
& 5 & - & 2 & -
& 7 & - & - & -
& 5 & - & 1 & 1
& 6 & - & - & 1\\
025 mug & 5 & 2 & 1 & - & 2
& 3 & 2 & - & -
& 3 & 1 & 1 & -
& 1 & 1 & 3 & -
& 2 & - & 3 & -
& 3 & - & 2 & 1\\
037 scissors & 7 & 1 & 2 & 4 & -
& 1 & 3 & 3 & -
& 1 & - & 5 & 1
& 3 & 2 & 2 & -
& - & 2 & 5 & -
& - & 2 & 2 & 3\\
035 power drill & 7 & 2 & 1 & 3 & 1
& 3 & 2 & 2 & -
& - & 7 & - & -
& 2 & 4 & - & 1
& 1 & 3 & 3 & -
& 3 & 3 & - & 1\\
040 large marker & 6 & 1 & 4 & 1 & -
& 4 & 1 & 1 & -
& 3 & 1 & 2 & -
& 1 & 2 & 3 & -
& 3 & - & 3 & -
& 2 & 2 & 2 & 1\\
052 extra large clamp & 6 & 6 & - & - & -
& 5 & 1 & - & -
& 3 & - & 3 & -
& 4 & 1 & 1 & -
& - & 1 & 5 & -
& 2 & 1 & 2 & 1\\ \hline
ALL & 100 & 58 & 20 & 17 & 5 &
48 & 34 & 16 & 2 & 43 & 18 & 36 & 3 & 60 & 15 & 22 & 3 & 38 & 11 & 48 & 3 & 57 & 16 & 15 & 12 \\
\hline
\multicolumn{26}{c}{Randomly Fixed Ordering} \\ \hline
003 cracker box & 6 & 5 & 1 & - & -
& 2 & 4 & - & -
& 3 & 2 & 1 & -
& 4 & 1 & 1 & -
& 3 & 2 & 1 & -
& 4 & 1 & 1 & -\\
004 sugar box & 5 & 4 & 1 & - & -
& - & 4 & 1 & -
& 2 & - & 3 & -
& 3 & - & 2 & -
& 3 & 1 & 1 & -
& 5 & - & - & -\\
005 tomato soup can & 7 & 7 & - & - & -
& 5 & 2 & - & -
& 1 & 1 & 5 & -
& 6 & 1 & - & -
& 1 & - & 5 & 1
& 7 & - & - & -\\
006 mustard bottle & 7 & 7 & - & - & -
& 2 & 4 & 1 & -
& 2 & 2 & 2 & 1
& 1 & 1 & 3 & 1
& - & - & 6 & 1
& 3 & - & 1 & 3\\
007 tuna fish can & 6 & 2 & 4 & - & -
& 1 & 5 & - & -
& 5 & 1 & - & -
& 4 & 1 & 1 & -
& 2 & 1 & 3 & -
& 4 & 1 & - & 1\\
008 pudding box & 5 & 4 & - & 1 & -
& 3 & 1 & 1 & -
& 3 & - & 2 & -
& 4 & 1 & - & -
& 4 & - & 1 & -
& 4 & - & - & 1\\
009 gelatin box & 7 & 1 & 6 & - & -
& 3 & 4 & - & -
& 4 & - & 2 & 1
& 7 & - & - & -
& 4 & 1 & 2 & -
& 6 & - & 1 & -\\
010 potted meat can & 7 & 7 & - & - & -
& 4 & 1 & 2 & -
& 2 & 2 & 2 & 1
& 6 & 1 & - & -
& 4 & 1 & 2 & -
& 4 & - & 2 & 2\\
011 banana & 7 & 2 & 1 & 4 & -
& 2 & 4 & 1 & -
& 4 & 1 & 2 & -
& 1 & - & 6 & -
& 3 & 1 & 2 & 1
& 5 & 1 & 1 & -\\
021 bleach cleanser & 5 & 3 & 2 & - & -
& 1 & 3 & 1 & -
& 2 & - & 2 & 1
& 1 & 1 & 2 & 1
& 1 & 1 & 3 & -
& 1 & - & 2 & 2\\
024 bowl & 7 & 4 & 3 & - & -
& 2 & 3 & 2 & -
& 4 & 1 & 2 & -
& 7 & - & - & -
& 4 & - & 2 & 1
& 7 & - & - & -\\
025 mug & 5 & 5 & - & - & -
& 3 & 2 & - & -
& 4 & - & 1 & -
& 4 & - & - & 1
& 2 & 1 & 2 & -
& 2 & - & 3 & -\\
037 scissors & 7 & - & 1 & 6 & -
& 1 & 3 & 2 & 1
& 1 & 2 & 4 & -
& 2 & 2 & 3 & -
& 2 & 2 & 3 & -
& 2 & 4 & 1 & -\\
035 power drill & 7 & 2 & 1 & 4 & -
& 2 & 2 & 3 & -
& - & 6 & - & 1
& 3 & 4 & - & -
& - & 3 & 3 & 1
& 4 & 1 & 1 & 1\\
040 large marker & 6 & - & 5 & 1 & -
& 2 & 4 & - & -
& - & 1 & 5 & -
& 3 & 2 & 1 & -
& 2 & 1 & 3 & -
& 2 & 4 & - & -\\
052 extra large clamp & 6 & 6 & - & - & -
& 5 & 1 & - & -
& 2 & - & 4 & -
& 3 & 2 & 1 & -
& 1 & - & 5 & -
& 2 & 2 & 2 & - \\
\hline
ALL & 100 & 59 & 25 & 16 & - & 38 & 47 & 14 & 1 & 39 & 19 & 37 & 5 & 60 & 17 & 20 & 3 & 36 & 15 & 44 & 5 & 61 & 14 & 15 & 10 \\
\hline
\end{tabular}
}
\caption{Statistics of our grasping experiments for each YCB object. S: \#pick-and-place success, P$_{E}$F: \#perception failure, P$_L$F: \#planning failure, EF: \#execution failure}
\label{tab:grasp_objects}
\vspace{-4mm}
\end{table*}
%=========================End EXPERIMENTS==============================================================
%=======================================================================================================
%=========================CONCLUSION====================================================================
\vspace{-2mm}
\section{Discussion and Future Work}
\vspace{-2mm}
\textbf{Limitations.} Our SceneReplica benchmark is limited by the objects that are used for testing, i.e., 16 YCB objects. Ideally, using more diverse objects can better test robot manipulation systems. Second, our benchmark currently only supports the Fetch mobile manipulator. Since a head camera is integrated into the robot, we do not need to perform hand-eye calibration. Extending the benchmark to robots without cameras would require setting up a third-person view camera and conducting a hand-eye calibration of the camera. Once the camera is setup, we can re-render the scene reference images using the new camera pose, and these images can be used for scene recreation.
Our experiments on SceneReplica hint at several directions to improve robot pick-and-place. First, object perception and grasp planning accuracy can be further improved. Second, one important aspect that is not considered in the model-based and model-free grasping methods is force feedback and force closure. Slippage failures can be reduced by introducing force feedback. It can also improve the grasping of fragile objects.
For future work, we plan to continuously evaluate more methods on SceneReplica and release these results to the community. One line of work that is missing from our current evaluation is reinforcement learning-based approaches for 6D grasping. We plan to evaluate some end-to-end grasping approaches on SceneReplica. We hope the community can use SceneReplica to benchmark robot manipulation methods. It can also be extended to other manipulation tasks such as object rearrangement in the future.
\label{sec:conclusion}
%=========================End CONCLUSION================================================================
%=======================================================================================================
% The maximum paper length is 8 pages excluding references and acknowledgements, and 10 pages including references and acknowledgements
\clearpage
% The acknowledgments are automatically included only in the final and preprint versions of the paper.
% \acknowledgments{If a paper is accepted, the final camera-ready version will (and probably should) include acknowledgments. All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support.}
\acknowledgments{
\vspace{-2mm}
This work was supported in part by the DARPA Perceptually-enabled Task Guidance (PTG) Program under contract number HR00112220005.}
%===============================================================================
% no \bibliographystyle is required, since the corl style is automatically used.
\bibliography{main} % .bib
\newpage
\input{includes/supp}
\end{document}
%RepeatScenes: Benchmarking Robot Manipulation by Creating Repeatable Scenes in the Real World
\textsc{SceneReplica}: Benchmarking Real-World Robot Manipulation by Creating Reproducible Scenes
Ninad Khargonkar* \hspace{3px} Sai Haneesh Allu* \hspace{3px} Yangxiao Lu \hspace{3px} Jishnu Jaykumar P \\ \textbf{Balakrishnan Prabhakaran} \hspace{3px} \textbf{Yu Xiang}\\
The University of Texas at Dallas \\
\tt\small \{firstname.lastname\}@utdallas.edu \hspace{3px} *equal contribution \\
% We aim to develop a simple but effective benchmark for robot manipulation by focusing on the task of object grasping. We take a holisitc view of the overall task and attempt to develop a real world benchmark that can reliably test different components in manipulation like perception and planning. Separating out the components in the pipeline also allows for testing perception algorithms in a robotics setting thereby making the benchmarking useful for other robot learning tasks. Furthermore, by design, the benchmark is easily reproducible and uses the YCB object set which is common in the robotics community.
We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible to researchers and practitioners. We also provide our experimental results and analyzes for model-based and model-free 6D robotic grasping on the benchmark, where representative algorithms are evaluated for object perception, grasping planning, and motion planning. We believe that our benchmark will be a valuable tool for advancing the field of robot manipulation. By providing a standardized evaluation framework, researchers can more easily compare different techniques and algorithms, leading to faster progress in developing robot manipulation methods. \footnote{Project page is available at \url{https://irvlutd.github.io/SceneReplica}}
% Our benchmark takes a holistic approach to evaluating the entire manipulation pipeline, including perception, planning and control. By separating out these components, we can test and evaluate each individually, allowing for more targeted improvements in robot manipulation.
\appendix
{\Large \textbf{Appendix}}
\section{Scene Generation}
One of the criteria for our scenes is that each object should be reachable with respect to the robot arm. In order to accomplish this we discretize the table's region into a dense grid and check for each grid location's reachability. Next, we only spawn the objects at the selected grid locations in a randomly sampled stable pose along with a random rotation along global Z axis. The random rotation along Z axis ensures that even if the same stable pose is selected for an object, the scenes for it look different. The object spawning is made collision free by checking for collisions using their 2D bounding boxes in X-Y plane. Once we have a candidate scene with 5 collision-free spawned objects, we test motion planning to each of them using the offline grasp dataset. If any of the objects does not have a feasible motion plan, we reject the scene and move on with a new scene generation.
This pipeline thus ensures both diverse and feasible scenes. Finally, the set of 20 scenes is selected from hundreds of candidate scenes using criteria on an object's count in the set and a scoring on pose-diversity of the set. While picking a scene set from the list of generated scene, we ensure that the count distribution of objects in the set is roughly uniform. Once we have a valid set of 20 scenes, we then compute the pose-based diversity. For a given set of 20 scenes, we first compute the count of unique poses for each object. This also gives us a count-based probability distribution using which we compute the entropy and use that as the scoring function. This ensures that scene-sets with the same repeated poses for objects are penalized and sets with diversity are encouraged.
\section{Grasp and Motion Planning}
We used Graspit to compute the 6D-grasps for each object in the benchmark. The grasps were computed in an offline manner due to the time constraints imposed by Graspit. Attempting to run Graspit for each iteration in a scene would take too long time and hence, instead we decided to compute the grasps offline. We repeatedly run the grasp sampling algorithm until we obtain a desired number of grasps around the object. Finally, on the initially generated grasp set, we apply farthest point sampling on grasp translation component to ensure that the grasps are spread out across an object's surface. Fig.~\ref{fig:graspit-grasps} illustrates the generated grasps for all 16 YCB objects.
\begin{figure}[!h]
\centering
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=12cm]{figures/graspit_grasps/graspit_grasps_row1.pdf}
\vspace{1pt}
\end{subfigure}
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=12cm]{figures/graspit_grasps/graspit_grasps_row2.pdf}
\vspace{1pt}
\end{subfigure}
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=12cm]{figures/graspit_grasps/graspit_grasps_row3.pdf}
\vspace{1pt}
\end{subfigure}
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=12cm]{figures/graspit_grasps/graspit_grasps_row4.pdf}
\vspace{1pt}
\end{subfigure}
\caption{Offline grasps generated using Graspit! for model-based grasping experiments. Pre-grasp poses shown for better visualization of underlying object.}
\label{fig:graspit-grasps}
\end{figure}
In all our experiment pipelines we try top-down grasping in case motion planning fails for a given grasp. Both model-based top-down grasping and model-free top-down grasping work in a similar fashion by considering a top-down view of object's point cloud. We compute a top-down grasp for an object by obtaining the two principal components for the point cloud's X-Y dimensions. We use the smaller component for obtaining the gripper width and it also gives the orientation to align the gripper with respect to the object. The only difference between model-based and model-free top-down grasping is that with a model-free method, we are restricted to the partial point cloud of the object. For motion planning obstacles, in model-based grasping we simply use the estimated pose with the object's known 3D mesh. In model-free methods since we don't have models, one strategy is to simply use the bounding box of the partial point cloud. However, this did not work well in practice for cluttered scenes due to large obstacle boxes for motion planning, which then inevitably fails to find a motion plan to grasp. Hence, we resort to a similar strategy as used before for top-down grasp alignment and compute an oriented bounding box using the PCA of object points in X-Y plane.
\vspace{-2mm}
\begin{figure*}[h]
\centering
\includegraphics[width=0.8\columnwidth]{figures/topdown.pdf}
\vspace{-1mm}
\caption{Top down grasp from point cloud}
\label{fig:topdown}
\vspace{-3mm}
\end{figure*}
\begin{thebibliography}{31}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Deng et~al.(2009)Deng, Dong, Socher, Li, Li, and
Fei-Fei]{deng2009imagenet}
J.~Deng, W.~Dong, R.~Socher, L.-J. Li, K.~Li, and L.~Fei-Fei.
\newblock Imagenet: A large-scale hierarchical image database.
\newblock In \emph{2009 IEEE conference on computer vision and pattern
recognition}, pages 248--255. Ieee, 2009.
\bibitem[Russakovsky et~al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma,
Huang, Karpathy, Khosla, Bernstein, et~al.]{russakovsky2015imagenet}
O.~Russakovsky, J.~Deng, H.~Su, J.~Krause, S.~Satheesh, S.~Ma, Z.~Huang,
A.~Karpathy, A.~Khosla, M.~Bernstein, et~al.
\newblock Imagenet large scale visual recognition challenge.
\newblock \emph{International journal of computer vision}, 115:\penalty0
211--252, 2015.
\bibitem[Geiger et~al.(2013)Geiger, Lenz, Stiller, and
Urtasun]{geiger2013vision}
A.~Geiger, P.~Lenz, C.~Stiller, and R.~Urtasun.
\newblock Vision meets robotics: The kitti dataset.
\newblock \emph{The International Journal of Robotics Research}, 32\penalty0
(11):\penalty0 1231--1237, 2013.
\bibitem[Wang et~al.(2018)Wang, Singh, Michael, Hill, Levy, and
Bowman]{wang2018glue}
A.~Wang, A.~Singh, J.~Michael, F.~Hill, O.~Levy, and S.~R. Bowman.
\newblock Glue: A multi-task benchmark and analysis platform for natural
language understanding.
\newblock \emph{arXiv preprint arXiv:1804.07461}, 2018.
\bibitem[Wang et~al.(2019)Wang, Pruksachatkun, Nangia, Singh, Michael, Hill,
Levy, and Bowman]{wang2019superglue}
A.~Wang, Y.~Pruksachatkun, N.~Nangia, A.~Singh, J.~Michael, F.~Hill, O.~Levy,
and S.~Bowman.
\newblock Superglue: A stickier benchmark for general-purpose language
understanding systems.
\newblock \emph{Advances in neural information processing systems}, 32, 2019.
\bibitem[Hendrycks et~al.(2020)Hendrycks, Burns, Basart, Zou, Mazeika, Song,
and Steinhardt]{hendrycks2020measuring}
D.~Hendrycks, C.~Burns, S.~Basart, A.~Zou, M.~Mazeika, D.~Song, and
J.~Steinhardt.
\newblock Measuring massive multitask language understanding.
\newblock \emph{arXiv preprint arXiv:2009.03300}, 2020.
\bibitem[Kimble et~al.(2020)Kimble, Van~Wyk, Falco, Messina, Sun, Shibata,
Uemura, and Yokokohji]{kimble2020benchmarking}
K.~Kimble, K.~Van~Wyk, J.~Falco, E.~Messina, Y.~Sun, M.~Shibata, W.~Uemura, and
Y.~Yokokohji.
\newblock Benchmarking protocols for evaluating small parts robotic assembly
systems.
\newblock \emph{IEEE robotics and automation letters}, 5\penalty0 (2):\penalty0
883--889, 2020.
\bibitem[Bottarel et~al.(2020)Bottarel, Vezzani, Pattacini, and
Natale]{bottarel2020graspa}
F.~Bottarel, G.~Vezzani, U.~Pattacini, and L.~Natale.
\newblock Graspa 1.0: Graspa is a robot arm grasping performance benchmark.
\newblock \emph{IEEE Robotics and Automation Letters}, 5\penalty0 (2):\penalty0
836--843, 2020.
\bibitem[Calli et~al.(2015)Calli, Singh, Walsman, Srinivasa, Abbeel, and
Dollar]{calli2015ycb}
B.~Calli, A.~Singh, A.~Walsman, S.~Srinivasa, P.~Abbeel, and A.~M. Dollar.
\newblock The ycb object and model set: Towards common benchmarks for
manipulation research.
\newblock In \emph{2015 international conference on advanced robotics (ICAR)},
pages 510--517. IEEE, 2015.
\bibitem[Bekiroglu et~al.(2019)Bekiroglu, Marturi, Roa, Adjigble, Pardi, Grimm,
Balasubramanian, Hang, and Stolkin]{bekiroglu2019benchmarking}
Y.~Bekiroglu, N.~Marturi, M.~A. Roa, K.~J.~M. Adjigble, T.~Pardi, C.~Grimm,
R.~Balasubramanian, K.~Hang, and R.~Stolkin.
\newblock Benchmarking protocol for grasp planning algorithms.
\newblock \emph{IEEE Robotics and Automation Letters}, 5\penalty0 (2):\penalty0
315--322, 2019.
\bibitem[Xiang et~al.(2017)Xiang, Schmidt, Narayanan, and
Fox]{xiang2017posecnn}
Y.~Xiang, T.~Schmidt, V.~Narayanan, and D.~Fox.
\newblock Posecnn: A convolutional neural network for 6d object pose estimation
in cluttered scenes.
\newblock \emph{arXiv preprint arXiv:1711.00199}, 2017.
\bibitem[Tremblay et~al.(2018)Tremblay, To, Sundaralingam, Xiang, Fox, and
Birchfield]{tremblay2018deep}
J.~Tremblay, T.~To, B.~Sundaralingam, Y.~Xiang, D.~Fox, and S.~Birchfield.
\newblock Deep object pose estimation for semantic robotic grasping of
household objects.
\newblock \emph{arXiv preprint arXiv:1809.10790}, 2018.
\bibitem[Deng et~al.(2021)Deng, Mousavian, Xiang, Xia, Bretl, and
Fox]{deng2021poserbpf}
X.~Deng, A.~Mousavian, Y.~Xiang, F.~Xia, T.~Bretl, and D.~Fox.
\newblock Poserbpf: A rao--blackwellized particle filter for 6-d object pose
tracking.
\newblock \emph{IEEE Transactions on Robotics}, 37\penalty0 (5):\penalty0
1328--1342, 2021.
\bibitem[Miller and Allen(2004)]{miller2004graspit}
A.~T. Miller and P.~K. Allen.
\newblock Graspit! a versatile simulator for robotic grasping.
\newblock \emph{IEEE Robotics \& Automation Magazine}, 11\penalty0
(4):\penalty0 110--122, 2004.
\bibitem[Diankov and Kuffner(2008)]{diankov2008openrave}
R.~Diankov and J.~Kuffner.
\newblock Openrave: A planning architecture for autonomous robotics.
\newblock \emph{Robotics Institute, Pittsburgh, PA, Tech. Rep.
CMU-RI-TR-08-34}, 79, 2008.
\bibitem[Sucan et~al.(2012)Sucan, Moll, and Kavraki]{sucan2012open}
I.~A. Sucan, M.~Moll, and L.~E. Kavraki.
\newblock The open motion planning library.
\newblock \emph{IEEE Robotics \& Automation Magazine}, 19\penalty0
(4):\penalty0 72--82, 2012.
\bibitem[Xie et~al.(2020)Xie, Xiang, Mousavian, and Fox]{xie2020best}
C.~Xie, Y.~Xiang, A.~Mousavian, and D.~Fox.
\newblock The best of both modes: Separately leveraging rgb and depth for
unseen object instance segmentation.
\newblock In \emph{Conference on robot learning}, pages 1369--1378. PMLR, 2020.
\bibitem[Xiang et~al.(2021)Xiang, Xie, Mousavian, and Fox]{xiang2021learning}
Y.~Xiang, C.~Xie, A.~Mousavian, and D.~Fox.
\newblock Learning rgb-d feature embeddings for unseen object instance
segmentation.
\newblock In \emph{Conference on Robot Learning}, pages 461--470. PMLR, 2021.
\bibitem[Lu et~al.(2022)Lu, Chen, Ruozzi, and Xiang]{lu2022mean}
Y.~Lu, Y.~Chen, N.~Ruozzi, and Y.~Xiang.
\newblock Mean shift mask transformer for unseen object instance segmentation.
\newblock \emph{arXiv preprint arXiv:2211.11679}, 2022.
\bibitem[Mousavian et~al.(2019)Mousavian, Eppner, and Fox]{mousavian20196}
A.~Mousavian, C.~Eppner, and D.~Fox.
\newblock 6-dof graspnet: Variational grasp generation for object manipulation.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 2901--2910, 2019.
\bibitem[Sundermeyer et~al.(2021)Sundermeyer, Mousavian, Triebel, and
Fox]{sundermeyer2021contact}
M.~Sundermeyer, A.~Mousavian, R.~Triebel, and D.~Fox.
\newblock Contact-graspnet: Efficient 6-dof grasp generation in cluttered
scenes.
\newblock In \emph{2021 IEEE International Conference on Robotics and
Automation (ICRA)}, pages 13438--13444. IEEE, 2021.
\bibitem[Yu et~al.(2020)Yu, Quillen, He, Julian, Hausman, Finn, and
Levine]{yu2020meta}
T.~Yu, D.~Quillen, Z.~He, R.~Julian, K.~Hausman, C.~Finn, and S.~Levine.
\newblock Meta-world: A benchmark and evaluation for multi-task and meta
reinforcement learning.
\newblock In \emph{Conference on robot learning}, pages 1094--1100. PMLR, 2020.
\bibitem[James et~al.(2020)James, Ma, Arrojo, and Davison]{james2020rlbench}
S.~James, Z.~Ma, D.~R. Arrojo, and A.~J. Davison.
\newblock Rlbench: The robot learning benchmark \& learning environment.
\newblock \emph{IEEE Robotics and Automation Letters}, 5\penalty0 (2):\penalty0
3019--3026, 2020.
\bibitem[Zhu et~al.(2020)Zhu, Wong, Mandlekar, Mart{\'\i}n-Mart{\'\i}n, Joshi,
Nasiriany, and Zhu]{zhu2020robosuite}
Y.~Zhu, J.~Wong, A.~Mandlekar, R.~Mart{\'\i}n-Mart{\'\i}n, A.~Joshi,
S.~Nasiriany, and Y.~Zhu.
\newblock robosuite: A modular simulation framework and benchmark for robot
learning.
\newblock \emph{arXiv preprint arXiv:2009.12293}, 2020.
\bibitem[Heo et~al.(2023)Heo, Lee, Lee, and Lim]{heo2023furniturebench}
M.~Heo, Y.~Lee, D.~Lee, and J.~J. Lim.
\newblock Furniturebench: Reproducible real-world benchmark for long-horizon
complex manipulation.
\newblock \emph{arXiv preprint arXiv:2305.12821}, 2023.
\bibitem[Liu et~al.(2021)Liu, Liu, Qin, Xiang, Gou, Xin, Roa, Calli, Su, Sun,
et~al.]{liu2021ocrtoc}
Z.~Liu, W.~Liu, Y.~Qin, F.~Xiang, M.~Gou, S.~Xin, M.~A. Roa, B.~Calli, H.~Su,
Y.~Sun, et~al.
\newblock Ocrtoc: A cloud-based competition and benchmark for robotic grasping
and manipulation.
\newblock \emph{IEEE Robotics and Automation Letters}, 7\penalty0 (1):\penalty0
486--493, 2021.
\bibitem[Dasari et~al.(2022)Dasari, Wang, Hong, Bahl, Lin, Wang, Thankaraj,
Chahal, Calli, Gupta, et~al.]{dasari2022rb2}
S.~Dasari, J.~Wang, J.~Hong, S.~Bahl, Y.~Lin, A.~Wang, A.~Thankaraj, K.~Chahal,
B.~Calli, S.~Gupta, et~al.
\newblock Rb2: Robotic manipulation benchmarking with a twist.
\newblock \emph{arXiv preprint arXiv:2203.08098}, 2022.
\bibitem[Morgan et~al.(2019)Morgan, Hang, Bircher, Alladkani, Gandhi, Calli,
and Dollar]{morgan2019benchmarking}
A.~S. Morgan, K.~Hang, W.~G. Bircher, F.~M. Alladkani, A.~Gandhi, B.~Calli, and
A.~M. Dollar.
\newblock Benchmarking cluttered robot pick-and-place manipulation with the box
and blocks test.
\newblock \emph{IEEE Robotics and Automation Letters}, 5\penalty0 (2):\penalty0
454--461, 2019.
\bibitem[Todorov et~al.(2012)Todorov, Erez, and Tassa]{todorov2012mujoco}
E.~Todorov, T.~Erez, and Y.~Tassa.
\newblock Mujoco: A physics engine for model-based control.
\newblock In \emph{2012 IEEE/RSJ international conference on intelligent robots
and systems}, pages 5026--5033. IEEE, 2012.
\bibitem[{Dawson-Haggerty et al.}()]{trimesh}
{Dawson-Haggerty et al.}
\newblock trimesh.
\newblock URL \url{https://trimsh.org/}.
\bibitem[wor(2022)]{workshop}
International workshop on recovering 6d object pose.
\newblock \url{http://cmp.felk.cvut.cz/sixd/workshop_2022/}, 2022.
\end{thebibliography}