Reinforcement Learning with Parameterized Manipulation Primitives for Robotic Assembly
Extracted source text · arXiv 2306.06679
\documentclass[journal]{IEEEtran}
% The following packages can be found on http:\\www.ctan.org
\usepackage{graphics} % for pdf, bitmapped graphics files
\usepackage{epsfig} % for postscript graphics files
% \usepackage{mathptmx} % assumes new font selection scheme installed
% \usepackage{times} % assumes new font selection scheme installed
% \usepackage{amsmath} % assumes amsmath package installed
\usepackage{amssymb} % assumes amsmath package installed
% --------- PACKAGE
% for hyperlink reference
\usepackage{hyperref}
% for enumerate
% \usepackage{enumitem}
% text color
\usepackage{color}
% merge row in table
\usepackage{multirow}
% multiple figures
\usepackage[caption=false, font=footnotesize]{subfig}
% table note
\usepackage{threeparttable}
% *** MATH PACKAGES ***
%
\usepackage[cmex10]{amsmath}
% --------- COMMAND
% straight and bold character
\newcommand{\bs}[1]{\boldsymbol{\mathrm{#1}}}
% straight character
\newcommand{\s}[1]{\mathrm{#1}}
\newcommand{\TODO}[1]{{\color{red} \textbf{TODO}: #1}}
\newcommand{\bluetext}[1]{\textcolor{blue}{#1}}
% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}
\begin{document}
\title{Reinforcement Learning with Parameterized Manipulation Primitives for Robotic Assembly}
\author{Nghia Vuong$^{1}$, and Quang-Cuong
Pham$^{1,2}$% <-this % stops a space
\thanks{$^{1}$Singapore Centre for 3D Printing (SC3DP), School of
Mechanical and Aerospace Engineering, NTU, Singapore}% <-this % stops a space
\thanks{$^{2}$Eureka Robotics, Singapore}
}
% The paper headers
% \markboth{Journal of \LaTeX\ Class Files,~Vol.~6, No.~1, January~2007}%
% {Shell \MakeLowercase{\textit{et al.}}: Bare Demo of IEEEtran.cls for Journals}
% The only time the second header will appear is for the odd numbered pages
% after the title page when using the twoside option.
%
% *** Note that you probably will NOT want to include the author's ***
% *** name in the headers of peer review papers. ***
% You can use \ifCLASSOPTIONpeerreview for conditional compilation here if
% you desire.
% If you want to put a publisher's ID mark on the page you can do it like
% this:
%\IEEEpubid{0000--0000/00\$00.00~\copyright~2007 IEEE}
% Remember, if you use this you must call \IEEEpubidadjcol in the second
% column for its text to clear the IEEEpubid mark.
% make the title area
\maketitle
\begin{abstract}
A common theme in robot assembly is the adoption of Manipulation Primitives as the atomic motion to compose assembly strategy, typically in the form of a state machine or a graph. While this approach has shown great performance and robustness in increasingly complex assembly tasks, the state machine has to be engineered manually in most cases. Such hard-coded strategies will fail to handle unexpected situations that are not considered in the design. To address this issue, we propose to find dynamics sequence of manipulation primitives through Reinforcement Learning. Leveraging parameterized manipulation primitives, the proposed method greatly improves both assembly performance and sample efficiency of Reinforcement Learning compared to a previous work using non-parameterized manipulation primitives. In practice, our method achieves good zero-shot sim-to-real performance on high-precision peg insertion tasks with different geometry, clearance, and material.
\end{abstract}
% \def\abstractname{Note to Practitioners}
% \begin{abstract}
% In recent years, increasingly complex assembly tasks have been
% demonstrated on robot systems. However, in most cases, the difficult robot assembly skills, such as tight pin insertion or part mating, are still accomplished by hand-designed, hard-coded, strategies (e.g. spiral search followed by force-controlled insertion). This paper presents a novel approach to automatically find assembly skills based on Reinforcement Learning. This method uses simple motion primitives, such as ``Move down until contact'', ``Slide along x while maintaining contact with the surface'', as the atomic action, enabling fast skill discovery. Assembly skills are learned in simulation and then transferred onto a physical platform. We show that such assembly skills achieve high success rate on several high-precision peg insertion tasks with different profiles. In future research, we will further improve the performance of this method by exploiting tactile information.
% \end{abstract}
\begin{IEEEkeywords}
Robot assembly, Reinforcement learning, sim-to-real transfer
\end{IEEEkeywords}
% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle
\section{Introduction}
% \IEEEPARstart{A}{}
A common theme in robot assembly is the adoption of Manipulation Primitives (MP) as the atomic motion to compose assembly strategy, typically in the form of a state machine or a graph \cite{thomasErrortolerantExecution2003}, \cite{finkemeyerExecutingAssembly2005}, \cite{suarez-ruizFrameworkFine2016}. While this approach has shown great performance and robustness in increasingly complex assembly tasks \cite{suarez-ruizCanRobots2018a}, the state machine has to be engineered manually in most cases. Such hard-coded strategies will fail to handle unexpected situations that are not considered in the design. Furthermore, designing and fine-tuning such strategies require considerable engineering expertise and time.
\begin{figure}[!t]
\centering
% \includegraphics[width=0.8\columnwidth]{../fig/visual-id.jpg}
\includegraphics[width=\columnwidth]{fig/robotic-setup.jpg}
\caption{Robotic assembly setup}
\label{visual-id}
\end{figure}
We are interested in the automatic discovery of assembly strategy, specifically in the form of a dynamics sequence of manipulation primitives. By dynamic sequence, we mean a policy that outputs an MP online from a set of MPs. We propose to learn such policies through Reinforcement Learning (RL), which was initially explored by our group in \cite{vuongLearningSequences2021}\footnote[3]{Here we summarize the main evolutions of this paper with respect to the conference version \cite{vuongLearningSequences2021}\begin{itemize}
\item The introduction of continuously-parameterized MPs as the action space for RL.
\item More extensive experimental results showing that parameterized MPs improve RL sample efficiency and task performance.
\end{itemize}}. However, one limitation of this work is the lack of flexibility as the number of MPs in the set is large while only several MPs are useful for a particular task. In this paper we address this fundamental problem by utilizing a set of \emph{continuously-parameterized} MPs. The action space, as a consequence, becomes hybrid: the RL policy needs to output an MP and its parameters (e.g., the speed of movement, the amount of force to terminate an MP). Using parameterized MPs enables reducing the number of MPs from 91 MPs to 13 MPs and expands the range of possible robot motion realized by the set of MPs. The addition of continuous parameters, however, could introduce additional complication for RL algorithm. We show experimentally that parameterized MPs improve both RL sample efficiency and assembly performance.
Another key advantage of MPs is their additional \emph{semantics}, which make them robust in sim-to-real and against model/environment variations and uncertainties: consider how ``Move down until contact'' is inherently more robust than a sequence of several short ``Move down'' actions. In practice, our method achieves good zero-shot sim-to-real performance on high-precision peg insertion tasks with different geometry, clearance, and material.
The rest of the paper is organized as follows. In Section
\ref{sec-review}, we review works that are related to our proposed
approach. In Section \ref{sec-background}, we introduce the background for the study. In Section \ref{sec-learn-seq}, we detail the procedure for designing a set of MPs and learning assembly strategy with RL. In Section
\ref{sec-exp}, the experimental setup and quantitative
results are presented. Finally, in Section \ref{sec-sum}, we discuss the advantages and limitations of the presented approach, as well as some directions
for future work.
\section{Related Work} \label{sec-review}
\textbf{Manipulation Primitives in robotic assembly.} Manipulation primitive or skill primitive is commonly used as an interface between task specification and control \cite{krogerTaskFrame2004}, \cite{krogerManipulationPrimitives2011} in robot manipulation. Decades-long research have seen wide adoption of MPs in robot assembly \cite{thomasErrortolerantExecution2003}, \cite{finkemeyerExecutingAssembly2005}, \cite{suarez-ruizFrameworkFine2016}. In these works, the assembly strategy is represented as a sequence or a graph composing of MPs. Recently, Johannsmeier et al. \cite{johannsmeierFrameworkRobot2019} propose a method to automatically optimize the parameters for the MPs in a predefined graph to minimize the assembly time through gradient-free optimization methods. It was shown that the graph of MPs with learned parameters outperforms human on performing a sequence of cylindrical pin insertion tasks. A shared limitation of the above studies is that the sequence of graph of MPs are defined manually, thus lack the ability to generalize to different contexts. Furthermore, since the graph is generated offline, it could not adapt to environmental uncertainties that might occur during execution: one failure of any MPs might lead to the failure of the whole execution. In our method, we do not assume a fixed sequence of MPs; instead, the MP is generated at run time.
\textbf{RL for high precision robotics assembly.}
Recent studies have explored several methods to improve sample efficiency and performance of RL algorithms in high precision robotics assembly. One such method is smart action space discretization and problem
decomposition. In
\cite{inoueDeepReinforcement2017}, Inoue et al. decompose the task into a search
phase and an insertion phase, and designs different state space, action
space, and reward function for each phase. More specifically, several
meta-actions are designed to form a discrete action space. This
greatly speeds up training and achieves good performance at the same
time. However, one drawback of this method is flexibility, since a few
meta-actions limit the dynamic capabilities of the robot. We address
this problem by using a large set of such meta-actions. Although this
choice compromises the reduced training time, we adopt sim2real to
address this issue and argue that the use of MPs as meta-actions makes
the sim-to-real transfer more efficient. In the same vein, Hamaya et
al. \cite{hamayaLearningRobotic2020a} divide the peg-in-hole task into five
steps with different action spaces and state spaces. This
decomposition greatly speeds up training through dimensional reduction
of action and state spaces. The method, however, assumes a fixed
sequence of steps that might not generalize well to other
tasks.
Another class of methods exploits a model-based or a heuristic policy to improve exploration phase of RL algorithm. One such technique is Residual Reinforcement Learning \cite{johanninkResidualReinforcement2019a}, in which the RL policy is the sum of a hand-designed controller and a parametric policy. Only the latter is learned by RL algorithm, while the former quickly guides the RL agent to the region in state space with high reward. These approaches have been shown to outperform vanilla RL in assembly tasks \cite{johanninkResidualReinforcement2019a}, \cite{schoettlerMetaReinforcementLearning2020a}. Subsequently, several works have studied the use of different types of policies, such as Gaussian Mixture Model \cite{maEfficientInsertion2021}, Dynamic Movement Primitive \cite{davchevResidualLearning2021} in place of the hand-designed controller.
In \cite{luoReinforcementLearning2019a}, Luo et
al. uses a model-based method, namely iterative Linear-Quadratic-Gaussian (iLQG)
\cite{todorovGeneralizedIterative2005} to find local control policies, then train a neural network that generalizes this controller to adapt to environmental variations, taking into account the force feedback. This method is
fast, being able to find a good control policy in just a few
interactions, but relies on iLQG to find the local controllers that
might not achieve a good performance on complex tasks, e.g. tight
insertion tasks, due to the imposed linear structure on the system
dynamic.
We believe that the idea of using a model-based or heuristic policy can be applied to improve the sample efficiency of our method. For example, one can design a simple sequence of MPs and initialize the control policy to this simple sequence. However, this remains to be explored in future works.
One line of research utilizes sim-to-real techniques. Kaspar et al. \cite{kasparSim2RealTransfer2020a} perform system identification to find several simulation parameters (gravity, joint damping, etc.) to align the simulated with the physical system. However, this approach still require collecting excitation trajectories on the physical system. Domain randomization \cite{tobinDomainRandomization2017}, \cite{pengSimtorealTransfer2018} is another technique that is largely adopted for sim-to-real transfer \cite{schoettlerMetaReinforcementLearning2020a}, \cite{beltran-hernandezVariableCompliance2020}. Domain randomization aims to learn a policy that is robust to task variations, such as position or physical properties of objects. However, domain randomization requires prior knowledge to determine which parameters to randomize and their ranges in order to learn a policy that is robust to sim-to-real gap while also not being too conservative. In contrast, our method relies on the robustness of the MPs for sim-to-real transfer.
\textbf{Reinforcement learning with structured action space}
The choice of action space of RL environment is an important design factor. Common action spaces such as joint space (an action could be desired joint position), task space (an action could be desired end-effector position or robot impedance) have been explored in the literature \cite{martin-martinVariableImpedance2019}, \cite{bogdanovicLearningVariable2020}. To further improve the sample efficiency and performance of RL, recent works have investigated the use of high-level action, such as skills \cite{chitnisEfficientBimanual2020}, modular controllers \cite{sharmaLearningCompose2020}, or motion primitives \cite{vuongLearningSequences2021}, \cite{nasirianyAugmentingReinforcement2021}, \cite{zhangLearningInsertion2021}, \cite{dalalAcceleratingRobotic2021}.
Chitnis et al. \cite{chitnisEfficientBimanual2020} proposes to learn task schema composing of skills, such as grasping, goal reaching, lifting, to solve more complex tasks. However, the task schema is state-independent, which has been shown to struggle with complex manipulation tasks such as nut assembly and peg insertion \cite{nasirianyAugmentingReinforcement2021}.
Sharma et al. \cite{sharmaLearningCompose2020} defines multiple controllers that act on elementary axes of the task frame. The RL agent learns to compose multiple controller in parallel to realize more diverse behaviors. In their formulation, the controller is executed for a fixed number of steps while we consider more general MPs with an explicit condition of termination.
Recently, \cite{dalalAcceleratingRobotic2021} defines action primitive based on a goal state and an error metric. They evaluate their methods extensively in simulation on various benchmarks of robot manipulation tasks. In contrast, we aim to demonstrate our method on the real robot and focus on the high-precision peg insertion tasks.
Most related to our work is \cite{zhangLearningInsertion2021}, where they also propose to learn parameterized motion primitives for assembly tasks. In this work, we consider manipulation primitives with additional semantics with two possible termination conditions. This also provides an additional intrinsic reward to encourage the RL agent to discover consistent strategy. Furthermore, we evaluate our method on the tight peg insertion with square and triangle shape. We observe that these tasks are more challenging than the round peg insertion task.
\section{Background} \label{sec-background}
\subsection{Manipulation primitives} \label{sec-mp}
An MP represents a desired motion of the robot end-effector ($E$) with respect to a task frame ($T$) \cite{vuongLearningSequences2021}. It consists of a desired velocity command $\bs{v}_{\mathrm{des}}$, a desired force command $\bs{f}_{\mathrm{des}}$, and a stopping condition $\lambda$. The desired velocity and force commands are defined as
\begin{align}
\begin{split}
\bs{v}_{\mathrm{des}}(t) & := g_v(t, \bs{\Omega}_t;\bs{\theta}_v), \\
\bs{f}_{\mathrm{des}}(t) & := g_f(t, \bs{\Omega}_t;\bs{\theta}_f),
\end{split}
\label{equation:primitive}
\end{align}
where $g_v$ and $g_f$ are any functions parameterized respectively by
$\bs{\theta}_v$ and $\bs{\theta}_f$, and $\bs{\Omega}_t$ is the vector
of all sensor signals at time $t$. The stopping condition is
defined as
$\lambda : (t,\bs{\Omega}_t, \bs{\theta}_s) \mapsto \{\verb+SUCCESS+, \verb+FAILURE+,
\verb+CONTINUE+\}$. An MP terminates when \(\lambda\) returns either \verb+SUCCESS+ or \verb+FAILURE+. An MP is fully defined by the functions \(g_v\) \(g_f\), \(\lambda\), and their parameters \(\bs{\theta}=[\bs{\theta}_v, \bs{\theta}_f, \bs{\theta}_s]\)
Executing an MP requires controlling of desired end-effector velocity/position and force, which can be realized by the hybrid motion/force control scheme. In this scheme, the desired velocity and desired force is achieved by a position control loop and a force control loop, respectively. In this paper, we use the inverse dynamics control in the operational space approach \cite{khatibUnifiedApproach1987} for the position control loop and a simple feedforward force controller for the force control loop.
\subsection{Reinforcement learning} \label{sec-rl}
We consider here the discounted episodic RL problem. In this setting,
the problem is described as a Markov Decision Process (MDP)
\cite{suttonReinforcementLearning2018a}. At each time step $t$, the agent
observes current state $\bs{s}_t \in \mathcal{S}$,
executes an action $\bs{a}_t \in \mathcal{A}$, and receives an immediate
reward $r_t$. The environment evolves through the state transition
probability $p(\bs{s}_{t+1} | \bs{s}_t, \bs{a}_t)$. The goal in RL is to learn a
policy $\pi (\bs{a}_t|\bs{s}_t)$ that maximizes the expected discounted return
$R = \sum_{t=1}^{T} \gamma^t r_t$, where $\gamma$ is the discount
factor.
\subsection{Reinforcement learning with parameterized action space} \label{sec-pamdp}
A parameterized action space consists of a finite set \(\mathcal{A}_d=\{a_1,a_2\dots a_n\}\) with cardinality \(n\) and \(n\) sets \(\mathcal{X}_i, \; i=\overline{1,n}\). An action is a tuple \((a_i, x)\), where \(a_i\in \mathcal{A}_d\) and \(x\in \mathcal{X}_i\). Problems with parameterized action space can be formally defined as parameterized action MDP (PAMDP) \cite{massonReinforcementLearning2016}. In a PAMDP, the RL agent makes a decision in two steps: it first s \(a_i\) and then choose \(x\). One can think of the variable \(x\) as some parameters of the discrete action \(a_i\). For example, consider the manipulation primitive defined in \ref{sec-mp}, \(x\) can be the parameters of an MP. Finally, the policy can be written as
\begin{equation}
\pi(a_i,x|s)=\pi^d(a_i|s_t)\pi^c_i(x|s_t)
\label{eq-policy}
\end{equation}
where \(\pi^d(a_i|s_t)\) is denoted as the discrete-action policy and \(\pi^c_i(x|s_t)\) is denoted as the action-parameter policy
\section{Learning sequences of manipulation primitives for peg insertion tasks} \label{sec-learn-seq}
Manipulation primitives have long been used as the atomic actions to compose manipulation skills \cite{finkemeyerExecutingAssembly2005}, \cite{krogerManipulationPrimitives2011}, \cite{suarez-ruizFrameworkFine2016}. Despite using slightly different definitions of MP, the main idea is to provide a simple and unified interface to task specification and robot programming. Based on this idea, we propose a method to automatically find the sequences of MPs and their parameters for a class of high-precision peg insertion tasks. Note that the methodology detailed in this section could be applied for tasks other than peg insertion.
We make the following assumptions regarding the peg insertion task
\begin{itemize}
\item The peg is firmly grasped or rigidly attached to the end-effector
\item The hole position (defined as the point T in Fig.~\ref{fig-primitive-a}) and axis of insertion can be estimated with position error less than 1\,mm and orientation error less than 1\,degree. This can be achieved by, for example, using visual servoing technique \cite{yuSiameseConvolutional2019}.
\item The task frame is chosen such that its origin coincides with the estimated hole position and the direction of insertion is along the -z axis. This assumption simplifies the design of MPs without the loss of generality.
\end{itemize}
\begin{figure}[!t]
\centering
%% -- 1st subfigure --
\subfloat[Free-space, translate until contact]{
\label{fig-primitive-a}
\includegraphics[width=0.4\columnwidth]{fig/mv2cn.pdf}}
\hspace{0.1\columnwidth}
%% -- 2st subfigure --
\subfloat[Free-space, translate a predefined distance]{
\label{fig-primitive-b}
\includegraphics[width=0.4\columnwidth]{fig/displace.pdf}}\\
%% -- 3st subfigure --
\subfloat[In-contact, rotate until next contact]{
\label{fig-primitive-c}
\includegraphics[width=0.27\columnwidth]{fig/rot2con.pdf}}
\hspace{0.1\columnwidth}
%% -- 4th subfigure --
\subfloat[In-contact, insert]{
\label{fig-primitive-d}
\includegraphics[width=0.27\columnwidth]{fig/insert.pdf}}
\caption{Examples of Manipulation Primitives for insertion task}
\label{fig-primitive}
\end{figure}
We consider two families of MPs, free-space MPs and in-contact MPs:
\begin{itemize}
\item Free-space MPs are to be executed when the robot is not in
contact with the environment, i.e., when all external forces/torques
are zero. MPs in this family are then associated with zero desired
force/torque command.
\item In-contact MPs are to be executed when the robot is in contact
with the environment, i.e. when some of the external force/torque
components are non-zero. In addition to other objectives, MPs in
this family have some of the components of their desired
force/torque command to be non-zero in order to maintain the same
contact state during the execution. Given the assumption that the direction of insertion is along the -z axis, the desired force/torque command has the form \(\bs{f}_{des}=[0, 0, -f_d, 0, 0, 0]\)
\end{itemize}
Each family of MPs is subdivided into three types: (i) move until
(next) contact, (ii) move a predefined amount, (iii) insert. Figure
\ref{fig-primitive} illustrates some examples of MPs, which are
further detailed below.
\textbf{Move until contact.} Translate the end-effector
along a direction \(\bs{u}\), or rotate the end-effector about a direction \(\bs{u}\), until contact is detected. The stopping condition is \texttt{SUCCESS} if \(\bs{f}_{\mathrm{ext}}^T\bs{u}>f_\mathrm{thr}\), \texttt{FAILURE} if the execution time is larger than \(T\) secs. The parameters of this MP is \((\bs{u},v,f_\mathrm{thr},T)\), where \(v\) denotes motion speed. The parameters include the desired force \(f_d\) if the MP is an in-contact MP.
\textbf{Move fixed distance.} Translate the
end-effector along a direction \(\bs{u}\) over a predefined distance~$d$, or
rotate about a direction \(\bs{u}\) over a predefined angle~$d$. The stopping condition is \texttt{SUCCESS} if robot reaches the desired distance/angle, \texttt{FAILURE} if interaction force/torque exceeds \(f_\mathrm{max}\). The parameters of this MP is \((\bs{u},d,v,f_\mathrm{max})\) where \(v\) denotes the motion speed. The parameters include the desired force \(f_d\) if the MP is an in-contact MP.
\textbf{Insert.} Track a non-zero force in the direction
of insertion and regulate the external torques to zero in all the
directions. To regulate torques to zero, we set desired velocity as \(\bs{v}_{\mathrm{des}}(t) = -K_d\bs{f}_{\mathrm{ext}}\) where \(K_d\) is a diagonal matrix and its diagonal is \(diag(K_d)=[0,0,0,k,k,k],\;k\geq0\). The stopping condition is \texttt{SUCCESS} if the end-effector reach a goal position within 2\,mm, \texttt{FAILURE} if the execution time is larger than \(T\) secs. The parameters of this MP is \((f_d,k,T)\)
Instead of manually design a sequence of MPs and tune their parameters for a particular tasks, we propose to automatically find the sequence of MPs and their parameters through RL. For this purpose, we first design a set of MPs and define bounds (or values) for all parameters. An example is shown in table~\ref{tab-mp-hybrid}, which consists of 13 MPs parameterized by 18 parameters. The axis of movement \(\bs{u}\) is chosen from the set of elementary axes \(x,y,z\) of the task frame. Note that other design choices are possible. For example, one can consider the axis of movement as a continuous parameter as in \cite{zhangLearningInsertion2021}.
\begin{table*}[!ht]
\vspace*{5pt}
\caption[caption]{The set of 13 Manipulation Primitives used in experiment. Ranges correspond to learnable parameters}
\label{tab-mp-hybrid}
\begin{center}
\begin{threeparttable}
\begin{tabular}{|c|c|c|c|c|c|}
\hline
Family & Type & Axis & Parameters & Value/Range & N \\
\hline
%%%%%%%%%% FREE
% Translate until contact
\multirow{8}{*}{Free space} & \multirow{3}{*}{Translate until contact (T$^\textrm{c}$)} & \multirow{3}{*}{$-z$} & $v$ & 10\,mm/s & \multirow{3}{*}{1} \\
& & & $f_\mathrm{thr}$ & 5\,N & \\
& & & $T$ & 2\,s & \\
\cline{2-6}
% Translate
& \multirow{3}{*}{Translate (T)} & \multirow{3}{*}{$x,y$} &
$v$ & 10\,mm/s & \multirow{3}{*}{2} \\
& & & $f_\mathrm{thr}$ & 20\,N & \\
& & & $d$ & [-10, 10]\,mm & \\
\cline{2-6}
% Rotate
& \multirow{3}{*}{Rotate (R)} & \multirow{3}{*}{$ x,
y$} &
$v$
& 0.1\,rad/s & \multirow{3}{*}{2} \\
& & & $f_\mathrm{thr}$ & 2\,Nm & \\
& & & $d$ & [-0.1, 0.1]\,rad & \\
\hline
%%%%%%%%%% CONTACT
% move to contact
% translate
\multirow{17}{*}{In contact} & \multirow{4}{*}{Translate
until next
contact (T$^\textrm{c}$)}& \multirow{4}{*}{$ x, y$} & $v$ & [-10, 10]\,mm/s & \multirow{4}{*}{2}\\
& & & $f_\mathrm{thr}$ & [5, 12]\,N & \\
& & & $f_d$ & -8\,N & \\
& & & $T$ & [0.1, 2]\,s & \\
\cline{2-6}
% rotate
% & \multirow{3}{*}{Rotate
% until next
% contact (R$^\textrm{c}$)}& \multirow{3}{*}{$ x, y, z$} & $v$ & 4 or 7\,deg/s & \multirow{3}{*}{24}\\
% & & & $f_\mathrm{thr}$ & $0.1$ or $0.5$\,Nm & \\
% & & & $f_d$ & $-3$\,N & \\
% \cline{2-6}
% displace
& \multirow{4}{*}{Translate (T)} & \multirow{4}{*}{$ x, y$} & $v$ & [5, 10]\,mm/s & \multirow{4}{*}{2} \\
& & & $f_\mathrm{thr}$ & 20\,N & \\
& & & $d$ & [-10, 10]\,mm & \\
& & & $f_d$ & $-8$\,N & \\
\cline{2-6}
& \multirow{4}{*}{Rotate (R)} & \multirow{4}{*}{$ x, y, z$} & $v$ & 0.1\,rad/s & \multirow{4}{*}{3} \\
& & & $f_\mathrm{thr}$ & 2\,Nm & \\
& & & $d$ & [-0.1, 0.1]\,rad & \\
& & & $f_d$ & $-8\,$N & \\
\cline{2-6}
% insert
& \multirow{5}{*}{Insert (I)} & \multirow{4}{*}{$-z$} & $\epsilon$ & 2\,mm & \multirow{5}{*}{1} \\
% & & & $k_{dt}$ & 0.0 & \\
& & & $k$ & [0.01, 0.2] & \\
& & & $f_d$ & [6, 15]\,N & \\
& & & $T$ & [0.1, 2]\,s & \\
\hline
\end{tabular}
\end{threeparttable}
\end{center}
\vspace*{-10pt}
\end{table*}
Given a set of MPs, the problem of finding a sequence of MPs can be naturally formulated as a PAMDP where an action is \((a_i,x)\); the discrete action \(a_i\) is an MP and \(x\) is its parameter. The state is defined by
$\bs{s}_t = [\bs{p}_t, \bs{f}_{ext, t}]$, where $\bs{p}_t$
is the pose of the peg relative to the hole (position and orientation of frame E with respect to frame T in Fig.~\ref{fig-primitive-a});
$\bs{f}_{ext, t}$ is the external force and torque acting on the
end-effector. Note that at a particular state, not all MPs are feasible. For example, in-contact MPs shouldn't be executed while the robot is in free space. Denote $\mathcal{A}_\mathrm{free}$ the set of free-space MPs and $\mathcal{A}_\mathrm{con}$ the set of in-contact MPs, the set of feasible actions at each state is either $\mathcal{A}_\mathrm{free}$ if $f_\mathrm{ext,z}=\bs{0}$, or $\mathcal{A}_\mathrm{con}$ if $f_\mathrm{ext,z}\neq\bs{0}$.
The reward function is defined as
\begin{eqnarray}
r_t:= c_1\left(e^{\frac{-||\bs{p}_{t+1} - \bs{p}_{goal}||_2^2}{k_1}} -1\right)+ c_2s(\bs{a}_t)
\label{eq-rew}
\end{eqnarray}
where \(c_1, c_2, k_1\) are positive constants. The first term rewards the RL agent for moving closer to the goal, while second term $s(\bs{a}_t)=0$ if
a \verb+SUCCESS+ status is returned, $s(\bs{a}_t)=-1$ if a \verb+FAILURE+
status is returned. This term can be thought as an intrinsic reward to maintain the semantics of the MPs.
The episode terminates when the number of MPs exceed 15 or the task is successful. In the latter case, the RL agent receives an additional termination reward of 5. The task is considered successful if the end-effector reach the goal position within 2\,mm.
\textbf{Policy architecture} As mentioned in \ref{sec-pamdp}, the policy in a PAMDP is composed of a discrete-action policy and an action-parameter policy. The discrete-action policy outputs the logits that define a categorical distribution over all MPs, while the action-parameter policy outputs the mean of a Gaussian distribution over the parameters of the current MP. To realize the state-dependent action space, the discrete-action policy is represented by two Neural Networks (NNs) which output the logits for the in-contact and free-space MPs, respectively. Given an input, the policy performs forward propagation only on one network depending on the measured external force. The action-parameter policy consists of multiple heads, one for each MP. The architecture of NNs are shown in Fig.~\ref{fig-policy}.
\begin{figure}[!t]
\centering
\includegraphics[width=\columnwidth]{fig/fig-policy.pdf}
\caption{Discrete-action policy network (left of dash line) and action-parameter policy network (right). The networks share a layer of batch normalization. The discrete-action policy contains two separate networks, one for each action subspace.}
\label{fig-policy}
\end{figure}
The policy is trained with PPO \cite{schulmanProximalPolicy2017}. In addition to the policy network, PPO requires a value network to approximate the value function. The value network has similar network architecture to the action-parameter networks and has independent weights.
\section{EXPERIMENTS} \label{sec-exp}
We conduct experiments to (1) validate the proposed method in simulation on various peg insertion with different shape (2) evaluate the sim-to-real performance of the learned policy.
\subsection{Experimental setups}
\textbf{Task description.} We evaluate the proposed approach on high-precision pin insertion tasks with different shapes and different clearances. More details are shown in Table \ref{tab-dim}.
\begin{table}[!t]
\vspace*{5pt}
\caption[caption]{Dimensions and material of pegs and holes. Size is diameter for round profile and side length for square and triangle one}
\label{tab-dim}
\begin{center}
\begin{tabular}{|c|c|c|c|c|}
\hline
Profile & Hole size (mm) & Peg size (mm) & Material \\
\hline
% Round & 30\,mm & Aluminum & 0.1\,mm \\
% \hline
Round & 30.03 & 29.96 & Aluminum \\
\hline
Round & 30.03 & 29.9 & Aluminum \\
\hline
Square & 19.98 & 19.96 & Aluminum \\
\hline
Square & 19.98 & 19.72 & Plastic \\
\hline % Triangle & 25.1 & Plastic & 1 \\
% \hline
Triangle & 25 & 24.9 & Aluminum \\
\hline
Triangle & 25 & 24.2 & Plastic \\
\hline
\end{tabular}
\end{center}
\end{table}
\textbf{Robot system setup.} A 7-DOF Franka Emika Panda robot is
used in our experiment. We additionally attach a Gamma IP60 force torque sensor to the flange of the robot to measure
the external force and torque acting on the end-effector. The measurement from FT sensor is needed to implement the insert primitive, as we observe that the external torque
estimation provided by \verb+libfranka+ is not precise enough to perform this
motion. Controller is implemented based on Robot Operating System and runs at 1000Hz. Execution of manipulation primitives is implemented based on ROS service-client framework.
\textbf{Simulation environment} We use the Mujoco physics engine
\cite{todorovMuJoCoPhysics2012} and adapt an open-source Panda robot model \footnote{available online at \href{https://github.com/vikashplus/franka_sim}{franka\_sim}}. The controller is simulated with a control frequency of 500Hz, similar to the simulation step. We use a lower frequency in simulation than that in real world to increase computational speed, while still maintaining a stable simulation. Three simulation environments are created for each of the peg shape shown in Fig.~\ref{tab-dim}. The \emph{perception error} is simulated by adding positional and rotational noise to the actual hole pose, this ``estimated'' hole pose is then used to compute the task frame for manipulation primitives, observation and reward for RL.
\textbf{RL implementation details.} We use \texttt{gym} \cite{brockmanOpenAIGym2016} to design the RL environment and \texttt{garage} \cite{garage}, an RL framework based on PyTorch for the implementation of PPO algorithm. For the policy architecture, each fully-connected layer has 128 nodes in the discrete-action policy network and 24 nodes in the action-parameter policy networks. The hyperparameters for PPO are listed as follows: clip ratio is 0.1, minibatch size is 64, discount factor is 0.99, learning rate is 0.0005, and policy entropy coefficient is 0.001, 2048 samples are collected for each policy update.
At the start of each episode, the robot's end-effector is reset to a position 10\,mm along the z axis of the task frame. During training, the perception error is varied at the beginning of each episode by sampling the positional uncertainty in \([-1,1]\)\,mm uniformly for all axes. To generate random rotational uncertainty, we sample a random unit vector and a random angle in \([-1, 1]\)\,degree uniformly, which together define an axis-angle representation of the uncertainty rotation.
\textbf{Baselines} We compare our method with three baselines. The first baseline learns in a purely continuous action space. Specifically, the control policy outputs the desired end-effector displacement at a rate of 40\,Hz. Since the OSC runs at a higher frequency, the input to the OSC is interpolated between 0 and the desired end-effector displacement during one policy step. We refer to the first baseline \texttt{ee-pose}. The second baseline learns in a purely discrete action space. The RL agent decides at each step which MP to execute on the robot, while its parameters are pre-defined. We manually pick two values within the range of parameters shown in Fig.~\ref{tab-mp-hybrid} to form a set of 81 MPs. We refer to this baseline \texttt{discrete} and our method \texttt{hybrid}.
The third baseline is based on \cite{johannsmeierFrameworkRobot2019}, where a fix sequence of MPs is manually defined and MPs parameters are optimized through gradient-free optimization techniques. Following \cite{johannsmeierFrameworkRobot2019}, the strategy has five steps (1) approach, (2) contact, (3) fit, (4) align, (5) insertion except for some slight differences. We use OSC as the low-level controller similar to our method and other baselines and use our insert MP in step 5 instead of sinusoidal motion. These modifications reduce the number of parameters included for optimization to four. The parameters are optimized by the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) algorithm \cite{hansenCompletelyDerandomized2001}, which shows the best performance in \cite{johannsmeierFrameworkRobot2019} among the considered algorithm. We refer to this baseline \texttt{fix-seq}.
\subsection{Simulation result}
We train three instances of each method (exclude \texttt{fix-seq}) with three different seeds. The training curves are shown in Fig~\ref{fig-training}. In all three tasks, our method is much more sample efficient and achieve higher final success rate than the other baselines. The number of samples required to reaches 60\% success rate for \texttt{hybrid} is about 2 times less than that of \texttt{discrete}. We also found that \texttt{ee-pose} has trouble in exploration at start of learning. This could be explained by the fact that in high-precision assembly task, the region in state space that leads to the goal is very small. Furthermore, during insertion phase, random exploration noise also causes large interaction force between peg and hole along x and y axes, which prevents the peg to move along the z axis.
\begin{figure}[!t]
\vspace*{5pt}
\centering
\subfloat[Round]{\includegraphics[scale=0.25]{fig/training-curve-round.pdf}} \hspace{0.0\columnwidth}
\subfloat[Square]{\includegraphics[scale=0.25]{fig/training-curve-square.pdf}}\\
\subfloat[Triangle]{\includegraphics[scale=0.25]{fig/training-curve-triangle.pdf}}
\caption{Training curve showing success rate over cumulative simulation steps. One simulation step corresponding to 2\,ms. The solid curve depicts the mean and the shaded region depicts the standard deviation over three different random seeds.}
\label{fig-training}
\end{figure}
\begin{figure}[!t]
\vspace*{5pt}
\centering
\subfloat[Success rate]{\includegraphics[scale=0.3]{fig/chart-success-sim.pdf}} \hspace{0.0\columnwidth}
\subfloat[Execution time (s)]{\includegraphics[scale=0.3]{fig/chart-time-sim.pdf}}
\caption{Quantitative evaluation in simulation. The final success rate and execution time is averaged over 100 trials.}
\label{fig-eval-sim}
\end{figure}
After training, we obtain the quantitative performance of the trained policies by executing each policy on the corresponding tasks for 100 trials. The average success rate and average execution time is shown in Fig.~\ref{fig-eval-sim}. Overall, \texttt{hybrid} achieves the best performance in terms of success rate, which is consistent with the training curve. Our method also achieves better execution time than \texttt{discrete} as the manually chosen parameters in \texttt{discrete} could be suboptimal. Another observation is that the average execution time for \texttt{ee-pose} is only 1.23\,s, much less than \texttt{hybrid} and \texttt{ee-pose}. There are two reasons for this result. First, the strategy learned by \texttt{hybrid} and \texttt{discrete} has a clear ``search'' phase whose purpose is to align the peg with the hole. The time required for this search phase depends on the environmental uncertainty (the hole pose in this case) causing larger execution time. Second, in \texttt{ee-pose}, robot motion is not constrained as in the other two baselines, i.e. the robot is free to move in any direction, thus the method can explore more diverse behavior. This is a limitation of our method which could be addressed by adding a dummy primitive that moves the robot end-effector to a desired position \cite{dalalAcceleratingRobotic2021}.
\subsection{Sim2real policy transfer on physical robot}
We evaluate the trained policies in simulation directly on six tasks corresponding to six peg-hole pairs shown in Table~\ref{tab-dim} without any further fine-tuning. In this experiment, we also report result for \texttt{fix-seq} baseline. For each task, an optimization is carried out to optimize the parameters of MPs. We then execute the final sequence for 20 trials and report the average success rate and execution time together with other methods. The results are shown in Fig.~\ref{fig-eval-real}. We omit the result for \texttt{ee-pose} because it always fails due to large contact force.
In general, we observe the same pattern as in the simulation: \texttt{hybrid} achieves higher success rate and shorter execution time than \texttt{discrete}. An exception is for the triangle-hard task, \texttt{hybrid} has longer execution time than \texttt{discrete}, but the success rate almost doubles that of \texttt{discrete}. The \texttt{fix-seq} baseline achieves the best performance in terms of execution time. The reason is that the fix sequence always execute four MPs per trial, while RL policies learned by other methods attempt a longer sequence to search for the hole.
We observe two common failure modes during evaluation of the policy learned by our method. First the policy repeatedly choose the ``move until contact'' MP even after the peg has already made contact, causing the robot to be locked in the current contact state. The second failure mode is when the policy fails to fit and align with the hole. We hypothesize that this is because the policy makes decision based solely on the relative pose between peg and hole, which is amenable to sim2real gap in contact modeling (i.e. overlap between two bodies in contact) and difference in object's size. We believe that the first problem could be addressed by incorporating contact force to RL observation, as contact force is more natural signal to infer contact state than pose. However, this remains to be investigated in the future.
\begin{figure}[!t]
\vspace*{5pt}
\centering
\subfloat[Success rate]{\includegraphics[scale=0.3]{fig/chart-success-real.pdf}} \hspace{0.0\columnwidth}
\subfloat[Execution time]{\includegraphics[scale=0.3]{fig/chart-time-real.pdf}}\\
\caption{Quantitative evaluation on the physical robot across six different pin insertion tasks. The "hard" suffix denotes tasks with smaller clearance. The final success rate/execution time is averaged over 20 trials}
\label{fig-eval-real}
\end{figure}
% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation Results}
%\label{fig_sim}
%\end{figure}
% Note that IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.
% An example of a floating table. Note that, for IEEE style tables, the
% \caption command should come BEFORE the table. Table text will default to
% \footnotesize as IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}
\section{Conclusion} \label{sec-sum}
In this paper, we have proposed a method to find dynamics sequence of manipulation primitives through Reinforcement Learning. Leveraging parameterized manipulation primitives, the proposed method was shown to greatly improve both assembly performance and sample efficiency of Reinforcement Learning. The experimental results showed that policies learned purely in simulation were able to consistently solve peg insertion tasks with different geometry and very small clearance.
A limitation of the approach is that the policy depends only on the kinematics information, while interaction force is only used to separate the free and in-contact action subspace. We hypothesize that this is the main causes for sim-to-real transfer failure. Integration of tactile information is thus an interesting future works.
In tasks that have complex dynamics, where instability is a key
consideration, the choice of the low-level control law in each MP
define the upper limit of the overall system performance. Hence,
incorporating advanced robust control laws~\cite{phamConvexController2020} is a
promising direction.
% \appendices
% \section{Proof of the First Zonklar Equation}
% Appendix one text goes here.
% % you can choose not to have a title for an appendix
% % if you want by leaving the argument blank
% \section{}
% Appendix two text goes here.
% use section* for acknowledgement
\section*{Acknowledgment}
This research was supported by the National Research Foundation, Prime
Minister’s Office, Singapore under its Medium Sized Centre funding
scheme, Singapore Centre for 3D Printing, CES\_SDC Pte Ltd, and Chip
Eng Seng Corporation Ltd.
% Can use something like this to put references on a page
% by themselves when using endfloat and the captionsoff option.
\ifCLASSOPTIONcaptionsoff
\newpage
\fi
% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}
% references section
\bibliographystyle{IEEEtran}
\bibliography{ref}
% biography section
%
% If you have an EPS/PDF photo (graphicx package needed) extra braces are
% needed around the contents of the optional argument to biography to prevent
% the LaTeX parser from getting confused when it sees the complicated
% \includegraphics command within an optional argument. (You could create
% your own custom macro containing the \includegraphics command to make things
% simpler here.)
%\begin{biography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{mshell}}]{Michael Shell}
% or if you just want to reserve a space for a photo:
% \begin{IEEEbiography}{Nghia Vuong}
% Nghia Vuong is currently a Ph.D student at Nanyang Technological University, School of Mechanical and Aerospace Engineering. His interest is on reinforcement learning and application for robot assembly.
% \end{IEEEbiography}
% % if you will not have a photo at all:
% \begin{IEEEbiographynophoto}{Quang-Cuong Pham}
% \TODO{fix mark}
% Quang-Cuong Pham was born in Hanoi, Vietnam and grew up in Vietnam and then in France. He graduated from École Normale Supérieure, France, in 2007. He obtained a PhD in Neuroscience from Université Paris 6 and Collège de France in 2009. In 2010, he was a Visiting Researcher at the University of São Paulo, Brazil. From 2011 to 2013, he was as a Fellow of the Japan Society for the Promotion of Science (JSPS), carrying out research in Robotics at the University of Tokyo. He joined NTU, Singapore as an Assistant Professor in 2013 and is currently an Associate Professor in the School of Mechanical and Aerospace Engineering.
% \end{IEEEbiographynophoto}
% \vfill
% insert where needed to balance the two columns on the last page with
% biographies
%\newpage
% You can push biographies down or up by placing
% a \vfill before or after them. The appropriate
% use of \vfill depends on what kind of text is
% on the last page and whether or not the columns
% are being equalized.
%\vfill
% Can be used to pull up biographies so that the bottom of the last one
% is flush with the other column.
%\enlargethispage{-5in}
% that's all folks
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{thomasErrortolerantExecution2003}
U.~Thomas, B.~Finkemeyer, T.~Kroger, and F.~Wahl, ``Error-tolerant execution of
complex robot tasks based on skill primitives,'' in \emph{2003 {{IEEE
International Conference}} on {{Robotics}} and {{Automation}} ({{Cat}}.
{{No}}.{{03CH37422}})}, vol.~3, Sep. 2003, pp. 3069--3075 vol.3.
\bibitem{finkemeyerExecutingAssembly2005}
B.~Finkemeyer, T.~Kr{\"o}ger, and F.~M. Wahl, ``Executing assembly tasks
specified by manipulation primitive nets,'' \emph{Advanced Robotics},
vol.~19, no.~5, pp. 591--611, Jan. 2005.
\bibitem{suarez-ruizFrameworkFine2016}
F.~{Su{\'a}rez-Ruiz} and Q.-C. Pham, ``A framework for fine robotic assembly,''
in \emph{2016 {{IEEE International Conference}} on {{Robotics}} and
{{Automation}} ({{ICRA}})}, May 2016, pp. 421--426.
\bibitem{suarez-ruizCanRobots2018a}
F.~{Su{\'a}rez-Ruiz}, X.~Zhou, and Q.-C. Pham, ``Can robots assemble an
{{IKEA}} chair?'' \emph{Science Robotics}, vol.~3, no.~17, p. eaat6385, Apr.
2018.
\bibitem{vuongLearningSequences2021}
N.~Vuong, H.~Pham, and Q.-C. Pham, ``Learning {{Sequences}} of {{Manipulation
Primitives}} for {{Robotic Assembly}},'' in \emph{2021 {{IEEE International
Conference}} on {{Robotics}} and {{Automation}} ({{ICRA}})}, May 2021, pp.
4086--4092.
\bibitem{krogerTaskFrame2004}
T.~Kroger, B.~Finkemeyer, and F.~Wahl, ``A task frame formalism for practical
implementations,'' in \emph{{{IEEE International Conference}} on {{Robotics}}
and {{Automation}}, 2004. {{Proceedings}}. {{ICRA}} '04. 2004}, vol.~5, Apr.
2004, pp. 5218--5223.
\bibitem{krogerManipulationPrimitives2011}
T.~Kr{\"o}ger, B.~Finkemeyer, and F.~M. Wahl, ``Manipulation {{Primitives}}
\textemdash{} {{A Universal Interface}} between {{Sensor-Based Motion
Control}} and {{Robot Programming}},'' in \emph{Robotic {{Systems}} for
{{Handling}} and {{Assembly}}}, ser. Springer {{Tracts}} in {{Advanced
Robotics}}, D.~Sch{\"u}tz and F.~M. Wahl, Eds.\hskip 1em plus 0.5em minus
0.4em\relax {Berlin, Heidelberg}: {Springer}, 2011, pp. 293--313.
\bibitem{johannsmeierFrameworkRobot2019}
L.~Johannsmeier, M.~Gerchow, and S.~Haddadin, ``A {{Framework}} for {{Robot
Manipulation}}: {{Skill Formalism}}, {{Meta Learning}} and {{Adaptive
Control}},'' in \emph{2019 {{International Conference}} on {{Robotics}} and
{{Automation}} ({{ICRA}})}, May 2019, pp. 5844--5850.
\bibitem{inoueDeepReinforcement2017}
T.~Inoue, G.~De~Magistris, A.~Munawar, T.~Yokoya, and R.~Tachibana, ``Deep
reinforcement learning for high precision assembly tasks,'' in \emph{2017
{{IEEE}}/{{RSJ International Conference}} on {{Intelligent Robots}} and
{{Systems}} ({{IROS}})}.\hskip 1em plus 0.5em minus 0.4em\relax {Vancouver,
BC}: {IEEE}, Sep. 2017, pp. 819--825.
\bibitem{hamayaLearningRobotic2020a}
M.~Hamaya, R.~Lee, K.~Tanaka, F.~{von Drigalski}, C.~Nakashima, Y.~Shibata, and
Y.~Ijiri, ``Learning {{Robotic Assembly Tasks}} with {{Lower Dimensional
Systems}} by {{Leveraging Physical Softness}} and {{Environmental
Constraints}},'' in \emph{2020 {{IEEE International Conference}} on
{{Robotics}} and {{Automation}} ({{ICRA}})}, May 2020, pp. 7747--7753.
\bibitem{johanninkResidualReinforcement2019a}
T.~Johannink, S.~Bahl, A.~Nair, J.~Luo, A.~Kumar, M.~Loskyll, J.~A. Ojea,
E.~Solowjow, and S.~Levine, ``Residual {{Reinforcement Learning}} for {{Robot
Control}},'' in \emph{2019 {{International Conference}} on {{Robotics}} and
{{Automation}} ({{ICRA}})}, May 2019, pp. 6023--6029.
\bibitem{schoettlerMetaReinforcementLearning2020a}
G.~Schoettler, A.~Nair, J.~A. Ojea, S.~Levine, and E.~Solowjow,
``Meta-{{Reinforcement Learning}} for {{Robotic Industrial Insertion
Tasks}},'' in \emph{2020 {{IEEE}}/{{RSJ International Conference}} on
{{Intelligent Robots}} and {{Systems}} ({{IROS}})}, Oct. 2020, pp.
9728--9735.
\bibitem{maEfficientInsertion2021}
Y.~Ma, D.~Xu, and F.~Qin, ``Efficient {{Insertion Control}} for {{Precision
Assembly Based}} on {{Demonstration Learning}} and {{Reinforcement
Learning}},'' \emph{IEEE Transactions on Industrial Informatics}, vol.~17,
no.~7, pp. 4492--4502, Jul. 2021.
\bibitem{davchevResidualLearning2021}
T.~Davchev, K.~S. Luck, M.~Burke, F.~Meier, S.~Schaal, and S.~Ramamoorthy,
``Residual {{Learning}} from {{Demonstration}}: {{Adapting DMPs}} for
{{Contact-rich Manipulation}},'' \emph{arXiv:2008.07682 [cs]}, Sep. 2021.
\bibitem{luoReinforcementLearning2019a}
J.~Luo, E.~Solowjow, C.~Wen, J.~A. Ojea, A.~M. Agogino, A.~Tamar, and
P.~Abbeel, ``Reinforcement {{Learning}} on {{Variable Impedance Controller}}
for {{High-Precision Robotic Assembly}},'' in \emph{2019 {{International
Conference}} on {{Robotics}} and {{Automation}} ({{ICRA}})}, May 2019, pp.
3080--3087.
\bibitem{todorovGeneralizedIterative2005}
E.~Todorov and W.~Li, ``A generalized iterative {{LQG}} method for
locally-optimal feedback control of constrained nonlinear stochastic
systems,'' in \emph{Proceedings of the 2005, {{American Control Conference}},
2005.}, Jun. 2005, pp. 300--306 vol. 1.
\bibitem{kasparSim2RealTransfer2020a}
M.~Kaspar, J.~D. Mu{\~n}oz~Osorio, and J.~Bock, ``{{Sim2Real Transfer}} for
{{Reinforcement Learning}} without {{Dynamics Randomization}},'' in
\emph{2020 {{IEEE}}/{{RSJ International Conference}} on {{Intelligent
Robots}} and {{Systems}} ({{IROS}})}, Oct. 2020, pp. 4383--4388.
\bibitem{tobinDomainRandomization2017}
J.~Tobin, R.~Fong, A.~Ray, J.~Schneider, W.~Zaremba, and P.~Abbeel, ``Domain
{{Randomization}} for {{Transferring Deep Neural Networks}} from
{{Simulation}} to the {{Real World}},'' \emph{arXiv:1703.06907 [cs]}, Mar.
2017.
\bibitem{pengSimtorealTransfer2018}
X.~B. Peng, M.~Andrychowicz, W.~Zaremba, and P.~Abbeel, ``Sim-to-real transfer
of robotic control with dynamics randomization,'' in \emph{2018 {{IEEE}}
International Conference on Robotics and Automation ({{ICRA}})}.\hskip 1em
plus 0.5em minus 0.4em\relax {IEEE}, 2018, pp. 3803--3810.
\bibitem{beltran-hernandezVariableCompliance2020}
C.~C. {Beltran-Hernandez}, D.~Petit, I.~G. {Ramirez-Alpizar}, and K.~Harada,
``Variable {{Compliance Control}} for {{Robotic Peg-in-Hole Assembly}}: {{A
Deep-Reinforcement-Learning Approach}},'' \emph{Applied Sciences}, vol.~10,
no.~19, p. 6923, Jan. 2020.
\bibitem{martin-martinVariableImpedance2019}
R.~{Mart{\'i}n-Mart{\'i}n}, M.~A. Lee, R.~Gardner, S.~Savarese, J.~Bohg, and
A.~Garg, ``Variable impedance control in end-effector space: {{An}} action
space for reinforcement learning in contact-rich tasks,'' \emph{arXiv
preprint arXiv:1906.08880}, 2019.
\bibitem{bogdanovicLearningVariable2020}
M.~Bogdanovic, M.~Khadiv, and L.~Righetti, ``Learning {{Variable Impedance
Control}} for {{Contact Sensitive Tasks}},'' \emph{arXiv:1907.07500 [cs]},
Jul. 2020.
\bibitem{chitnisEfficientBimanual2020}
R.~Chitnis, S.~Tulsiani, S.~Gupta, and A.~Gupta, ``Efficient bimanual
manipulation using learned task schemas,'' in \emph{2020 {{IEEE International
Conference}} on {{Robotics}} and {{Automation}} ({{ICRA}})}.\hskip 1em plus
0.5em minus 0.4em\relax {IEEE}, 2020, pp. 1149--1155.
\bibitem{sharmaLearningCompose2020}
M.~Sharma, J.~Liang, J.~Zhao, A.~LaGrassa, and O.~Kroemer, ``Learning to
{{Compose Hierarchical Object-Centric Controllers}} for {{Robotic
Manipulation}},'' \emph{arXiv:2011.04627 [cs]}, Nov. 2020.
\bibitem{nasirianyAugmentingReinforcement2021}
S.~Nasiriany, H.~Liu, and Y.~Zhu, ``Augmenting {{Reinforcement Learning}} with
{{Behavior Primitives}} for {{Diverse Manipulation Tasks}},''
\emph{arXiv:2110.03655 [cs]}, Oct. 2021.
\bibitem{zhangLearningInsertion2021}
X.~Zhang, S.~Jin, C.~Wang, X.~Zhu, and M.~Tomizuka, ``Learning {{Insertion
Primitives}} with {{Discrete-Continuous Hybrid Action Space}} for {{Robotic
Assembly Tasks}},'' Oct. 2021.
\bibitem{dalalAcceleratingRobotic2021}
M.~Dalal, D.~Pathak, and R.~Salakhutdinov, ``Accelerating {{Robotic
Reinforcement Learning}} via {{Parameterized Action Primitives}},''
\emph{arXiv:2110.15360 [cs]}, Oct. 2021.
\bibitem{khatibUnifiedApproach1987}
O.~Khatib, ``A unified approach for motion and force control of robot
manipulators: {{The}} operational space formulation,'' \emph{IEEE Journal on
Robotics and Automation}, vol.~3, no.~1, pp. 43--53, Feb. 1987.
\bibitem{suttonReinforcementLearning2018a}
R.~S. Sutton and A.~G. Barto, \emph{Reinforcement {{Learning}}, Second Edition:
{{An Introduction}}}.\hskip 1em plus 0.5em minus 0.4em\relax {MIT Press},
Nov. 2018.
\bibitem{massonReinforcementLearning2016}
W.~Masson, P.~Ranchod, and G.~Konidaris, ``Reinforcement learning with
parameterized actions,'' in \emph{Proceedings of the {{AAAI Conference}} on
{{Artificial Intelligence}}}, vol.~30, 2016.
\bibitem{yuSiameseConvolutional2019}
C.~Yu, Z.~Cai, H.~Pham, and Q.-C. Pham, ``Siamese {{Convolutional Neural
Network}} for {{Sub-millimeter-accurate Camera Pose Estimation}} and {{Visual
Servoing}},'' in \emph{2019 {{IEEE}}/{{RSJ International Conference}} on
{{Intelligent Robots}} and {{Systems}} ({{IROS}})}, Nov. 2019, pp. 935--941.
\bibitem{schulmanProximalPolicy2017}
J.~Schulman, F.~Wolski, P.~Dhariwal, A.~Radford, and O.~Klimov, ``Proximal
{{Policy Optimization Algorithms}},'' \emph{arXiv:1707.06347 [cs]}, Aug.
2017.
\bibitem{todorovMuJoCoPhysics2012}
E.~Todorov, T.~Erez, and Y.~Tassa, ``{{MuJoCo}}: {{A}} physics engine for
model-based control,'' in \emph{2012 {{IEEE}}/{{RSJ International
Conference}} on {{Intelligent Robots}} and {{Systems}}}, Oct. 2012, pp.
5026--5033.
\bibitem{brockmanOpenAIGym2016}
G.~Brockman, V.~Cheung, L.~Pettersson, J.~Schneider, J.~Schulman, J.~Tang, and
W.~Zaremba, ``{{OpenAI Gym}},'' \emph{arXiv:1606.01540 [cs]}, Jun. 2016.
\bibitem{hansenCompletelyDerandomized2001}
N.~Hansen and A.~Ostermeier, ``Completely {{Derandomized Self-Adaptation}} in
{{Evolution Strategies}},'' \emph{Evolutionary Computation}, vol.~9, pp.
159--195, Jun. 2001.
\bibitem{phamConvexController2020}
H.~Pham and Q.-C. Pham, ``Convex {{Controller Synthesis}} for {{Robot
Contact}},'' \emph{arXiv:1909.04313 [cs]}, Jan. 2020.
\end{thebibliography}