Learning from Trajectories via Subgoal Discovery
Extracted source text ยท arXiv 1911.07224
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2019
% ready for submission
% \usepackage{neurips_2019}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2019}
% to compile a camera-ready version, add the [final] option, e.g.:
%\usepackage[final]{neurips_2019}
%\usepackage{neurips_2019}
% to avoid loading the natbib package, add option nonatbib:
\usepackage[nonatbib,final]{neurips_2019}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{amsmath,amssymb,mathtools}
\usepackage{enumitem}
\usepackage{wrapfig}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
%\usepackage{wrapfig}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
%\newcommand{\theHalgorithm}{\arabic{algorithm}}
\title{Learning from Trajectories via Subgoal Discovery}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{
Sujoy Paul$^1$ \\
\texttt{supaul@ece.ucr.edu}
\And Jeroen van Baar$^2$ \\
\texttt{jeroen@merl.com}
\And Amit K. Roy-Chowdhury$^1$ \\
\texttt{amitrc@ece.ucr.edu} \\
\AND
{\normalfont $^1$University of California-Riverside \hspace{4mm}
$^2$Mitsubishi Electric Research Laboratories (MERL)}
}
\begin{document}
\maketitle
\begin{abstract}
Learning to solve complex goal-oriented tasks with sparse terminal-only rewards often requires an enormous number of samples. In such cases, using a set of expert trajectories could help to learn faster. However, Imitation Learning (IL) via supervised pre-training with these trajectories may not perform as well and generally requires additional finetuning with expert-in-the-loop. In this paper, we propose an approach which uses the expert trajectories and learns to decompose the complex main task into smaller sub-goals. We learn a function which partitions the state-space into sub-goals, which can then be used to design an extrinsic reward function. We follow a strategy where the agent first learns from the trajectories using IL and then switches to Reinforcement Learning (RL) using the identified sub-goals, to alleviate the errors in the IL step. To deal with states which are under-represented by the trajectory set, we also learn a function to modulate the sub-goal predictions. We show that our method is able to solve complex goal-oriented tasks, which other RL, IL or their combinations in literature are not able to solve.
\end{abstract}
\input{introduction}
\input{related}
\input{methodology}
\input{experiments}
\input{conclusion}
\bibliography{bib}
\bibliographystyle{unsrt}
\end{document}
\section{Introduction}
\label{intro}
Reinforcement Learning (RL) aims to take sequential actions so as to maximize, by interacting with an environment, a certain pre-specified reward function, designed for the purpose of solving a task. RL using Deep Neural Networks (DNNs) has shown tremendous success in several tasks such as playing games \cite{mnih2015human,silver2016mastering}, solving complex robotics tasks \cite{levine2016end,duan2016benchmarking}, etc. However, with sparse rewards, these algorithms often require a huge number of interactions with the environment, which is costly in real-world applications such as self-driving cars \cite{bojarski2016end}, and manipulations using real robots \cite{levine2016end}. Manually designed dense reward functions could mitigate such issues, however, in general, it is difficult to design detailed reward functions for complex real-world tasks.
Imitation Learning (IL) using trajectories generated by an expert can potentially be used to learn the policies faster \cite{argall2009survey}. But, the performance of IL algorithms \cite{ross2011reduction} are not only dependent on the performance of the expert providing the trajectories, but also on the state-space distribution represented by the trajectories, especially in case of high dimensional states. In order to avoid such dependencies on the expert, some methods proposed in the literature \cite{sun2017deeply,cheng2018fast} take the path of combining RL and IL. However, these methods assume access to the expert value function, which may become impractical in real-world scenarios.
In this paper, we follow a strategy which starts with IL and then switches to RL. In the IL step, our framework performs supervised pre-training which aims at learning a policy which best describes the expert trajectories. However, due to limited availability of expert trajectories, the policy trained with IL will have errors, which can then be alleviated using RL. Similar approaches are taken in \cite{cheng2018fast} and \cite{nagabandi2018neural}, where the authors show that supervised pre-training does help to speed-up learning. However, note that the reward function in RL is still sparse, making it difficult to learn. With this in mind, we pose the following question: \textit{can we make more efficient use of the expert trajectories, instead of just supervised pre-training?}
Given a set of trajectories, humans can quickly identify waypoints, which need to be completed in order to achieve the goal. We tend to break down the entire complex task into sub-goals and try to achieve them in the best order possible. Prior knowledge of humans helps to achieve tasks much faster \cite{ Andreas2017policysketch,dubey2018investigating} than using only the trajectories for learning. The human psychology of divide-and-conquer has been crucial in several applications and it serves as a motivation behind our algorithm which learns to partition the state-space into sub-goals using expert trajectories. The learned sub-goals provide a discrete reward signal, unlike value based continuous reward \cite{ng1999policy,sun2018truncated}, which can be erroneous, especially with a limited number of trajectories in long time horizon tasks. As the expert trajectories set may not contain all the states where the agent may visit during exploration in the RL step, we augment the sub-goal predictor via one-class classification to deal with such under-represented states. We perform experiments on three goal-oriented tasks on MuJoCo \cite{todorov2014convex} with sparse terminal-only reward, which state-of-the-art RL, IL or their combinations are not able to solve.
%To learn the sub-goals in a structured manner, we impose certain natural temporal ordering constraints on the sub-goals, which are not provided by the expert, but are often seen in several real-world tasks. We develop an iterative learn-predict-correct framework with the correction being performed using Dynamic Time Warping (DTW) to impose such ordering constraints. We develop an extrinsic reward function based on state transitions between such sub-goal partitions. In case of limited trajectories, the sub-goal partition can be erroneous at states not well represented in the trajectory set. We develop a mechanism to avoid such errors by using 1-class classification. Experimental evaluations show that our method is able to solve complex tasks and achieve better cumulative rewards than the expert, which other RL, IL or their combinations in literature are not able to solve.
\section{Related Works}
\label{related}
Our work is closely related to learning from demonstrations or expert trajectories as well as discovering sub-goals in complex tasks. We first discuss works on imitation learning using expert trajectories or reward-to-go. We also discuss the methods which aim to discover sub-goals, in an online manner during the RL stage from its past experience.
\textbf{Imitation Learning.}
Imitation Learning \cite{schaal1999imitation, silver2008high, chernova2009interactive,rajeswaran2017learning,hester2018deep} uses a set of expert trajectories or demonstrations to guide the policy learning process. A naive approach to use such trajectories is to train a policy in a supervised learning manner. However, such a policy would probably produce errors which grow quadratically with increasing steps. This can be alleviated using Behavioral Cloning (BC) algorithms \cite{ross2011reduction, ross2014reinforcement, torabi2018behavioral}, which queries expert action at states visited by the agent, after the initial supervised learning phase. However, such query actions may be costly or difficult to obtain in many applications. Trajectories are also used by \cite{levine2013guided}, to guide the policy search, with the main goal of optimizing the return of the policy rather than mimicking the expert. Recently, some works \cite{sun2017deeply,chang2015learning, sun2018truncated} aim to combine IL with RL by assuming access to experts reward-to-go at the states visited by the RL agent. \cite{cheng2018fast} take a moderately different approach where they switch from IL to RL and show that randomizing the switch point can help to learn faster. The authors in \cite{ranchod2015nonparametric} use demonstration trajectories to perform skill segmentation in an Inverse Reinforcement Learning (IRL) framework. The authors in \cite{murali2016tsc} also perform expert trajectory segmentation, but do not show results on learning the task, which is our main goal. SWIRL \cite{krishnan2019swirl} make certain assumptions on the expert trajectories to learn the reward function and their method is dependent on the discriminability of the state features, which we on the other hand learn end-to-end.
%In this paper, we take a similar approach where we first obtain a policy by supervised pre-training using the trajectories. Thereafter we use RL to interact with the environment and alleviate the error at the states not well-represented in the trajectory set, by using a set of sub-goals discovered automatically from the trajectory set.
\textbf{Learning with Options.}
Discovering and learning options have been studied in the literature \cite{sutton1999between,precup2000temporal,stolle2002learning} which can be used to speed-up the policy learning process. \cite{silver2012compositional} developed a framework for planning based on options in a hierarchical manner, such that low level options can be used to build higher level options. \cite{Florensa2017StochasticNN} propose to learn a set of options, or skills, by augmenting the state space with a latent categorical skill vector. A separate network is then trained to learn a policy over options. The Option-Critic architecture \cite{bacon2017option} developed a gradient based framework to learn the options along with learning the policy. This framework is extended in \cite{riemerlearning} to handle a hierarchy of options. \cite{held2017automatic} proposed a framework where the goals are generated using Generative Adversarial Networks (GAN) in a curriculum learning manner with increasingly difficult goals. Researchers have shown that an important way of identifying sub-goals in several tasks is identifying bottle-neck regions in tasks. Diverse Density \cite{mcgovern2001automatic}, Relative Novelty \cite{csimcsek2004using}, Graph Partitioning \cite{csimcsek2005identifying}, clustering \cite{mannor2004dynamic} can be used to identify such sub-goals. However, unlike our method, these algorithms do not use a set of expert trajectories, and thus would still be difficult to identify useful sub-goals for complex tasks.
\section{Methodology}
\label{sec:method}
We first provide a formal definition of the problem we are addressing in this paper, followed by a brief overall methodology, and then present a detailed description of our framework.
\textbf{Problem Definition.}
Consider a standard RL setting where an agent interacts with an environment which can be modeled by a Markov Decision Process (MDP) $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, r, \gamma, \mathcal{P}_0)$, where $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, $r$ is a scalar reward function, $\gamma \in [0,1]$ is the discount factor and $\mathcal{P}_0$ is the initial state distribution. Our goal is to learn a policy $\pi_\theta(\boldsymbol{a}|\boldsymbol{s})$, with $\boldsymbol{a} \in \mathcal{A}$, which optimizes the expected discounted reward $\mathbb{E}_\tau[\sum_{t=0}^\infty \gamma^t r(\boldsymbol{s}_t, \boldsymbol{a}_t)]$, where $\tau=(\dots, \boldsymbol{s}_t,\boldsymbol{a}_t,r_t,\dots)$ and $\boldsymbol{s}_0 \sim \mathcal{P}_0$, $\boldsymbol{a}_t \sim \pi_\theta(\boldsymbol{a}|\boldsymbol{s}_t)$ and $\boldsymbol{s}_{t+1} \sim \mathcal{P}(\boldsymbol{s}_{t+1}|\boldsymbol{s}_t,a_t)$.
With sparse rewards, optimizing the expected discounted reward using RL may be difficult. In such cases, it may be beneficial to use a set of state-action trajectories $\mathcal{D}=\{\{(\boldsymbol{s}_{ti},\boldsymbol{a}^*_{ti})\}_{t=1}^{n_i}\}_{i=1}^{n_d}$ generated by an expert to guide the learning process. $n_d$ is the number of trajectories in the dataset and $n_i$ is the length of the $i^{th}$ trajectory. We propose a methodology to efficiently use $\mathcal{D}$ by discovering sub-goals from these trajectories and use them to develop an extrinsic reward function.
\begin{figure}
\centering
\begin{subfigure}{0.48\textwidth}
\centering
\includegraphics[scale=0.4]{images/ours.pdf}
\caption{}
\label{fig:ours}
\end{subfigure}
\hfill
\begin{subfigure}{0.48\textwidth}
\centering
\includegraphics[width=2.4in,height=1.38in]{images/trajectories.pdf}
\caption{}
\label{fig:trajectories}
\end{subfigure}
\caption{(a) This shows an overview of our proposed framework to train the policy network along with sub-goal based reward function with out-of-set augmentation.
(b) An example state-partition with two independent trajectories in black and red. Note that the terminal state is shown as a separate state partition because we assume it to be indicated by the environment and not learned.}
\label{fig:im}
\end{figure}
% \begin{figure}
% \centering
% \includegraphics[scale=0.44]{images/ours.pdf}
% \caption{This figure presents the proposed framework to obtain the extrinsic reward function based on sub-goal prediction and out-of-sample estimation using 1-class classification.}
% \label{fig:ours}
% \end{figure}
\textbf{Overall Methodology.} %\label{sec:overall}
Several complex, goal-oriented, real-world tasks can often be broken down into sub-goals with some natural ordering. Providing positive rewards after completing these sub-goals can help to learn much faster compared to sparse, terminal-only rewards. In this paper, we advocate that such sub-goals can be learned directly from a set of expert demonstration trajectories, rather than manually designing them.
%For e.g., consider the task of a robot pouring water from a jug to a glass. If a reward is given only after task completion, it will take a long time to learn the task. However, if we assign positive rewards for sub-goals such as grasping the jug, moving to top of the glass and tilting the jug towards the glass, then the learning process may be faster. In this paper, we advocate that such sub-goals can be determined directly from a set of oracle demonstration trajectories, rather than requiring to design the reward function by the oracle.
A pictorial description of our method is presented in Fig. \ref{fig:ours}. We use the set $\mathcal{D}$ to first train a policy by applying supervised learning. This serves a good initial point for policy search using RL. However, with sparse rewards, the search can still be difficult and the network may forget the learned parameters in the first step if it does not receive sufficiently useful rewards.
%We advocate that decomposing the main task into smaller sub-goals and obtaining rewards for completing each sub-goal, will make the reward function denser and help to achieve faster and better convergence.
To avoid this, we use $\mathcal{D}$ to learn a function $\pi_\phi(g|\boldsymbol{s})$, which given a state, predicts sub-goals. We use this function to obtain a new reward function, which intuitively informs the RL agent whenever it moves from one sub-goal to another. We also learn a utility function $u_\psi(\boldsymbol{s})$ to modulate the sub-goal predictions over the states which are not well-represented in the set $\mathcal{D}$. We approximate the functions $\pi_\theta$, $\pi_\phi$, and $u_\psi$ using neural networks. We next describe our meaning of sub-goals followed by an algorithm to learn them.
\subsection{Sub-goal Definition} \label{sec:definition}
\textbf{Definition 1.} Consider that the state-space $\mathcal{S}$ is partitioned into sets of states as $\{\mathcal{S}_1,\mathcal{S}_2,\dots,\mathcal{S}_{n_g}\}$, s.t., $\mathcal{S}=\cup_{i=1}^{n_g} \mathcal{S}_i$ and $\cap_{i=1}^{n_g} \mathcal{S}_i =\emptyset$ and $n_g$ is the number of sub-goals specified by the user. For each $(\boldsymbol{s},\boldsymbol{a},\boldsymbol{s}')$, we say that the particular action takes the agent from one sub-goal to another iff $\boldsymbol{s} \in \mathcal{S}_i$, $\boldsymbol{s}' \in \mathcal{S}_j$ for some ${i,j} \in G = \{1,2, \dots, n_g\}$ and $i \neq j$.
We assume that there is an ordering in which groups of states appear in the trajectories as shown in Fig. \ref{fig:trajectories}. However, the states within these groups of states may appear in any random order in the trajectories. These groups of states are not defined a priori and our algorithm aims at estimating these partitions. Note that such orderings are natural in several real-world applications where a certain sub-goal can only be reached after completing one or more previous sub-goals. We show (empirically in the supplementary) that our assumption is soft rather than being strict, i.e., the degree by which the trajectories deviate from the assumption determines the granularity of the discovered sub-goals. We may consider that states in the trajectories of $\mathcal{D}$ appear in increasing order of sub-goal indices, i.e., achieving sub-goal $j$ is harder than achieving sub-goal $i$ $(i<j)$. This gives us a natural way of defining an extrinsic reward function, which would help towards faster policy search. Also, all the trajectories in $\mathcal{D}$ should start from the initial state distribution and end at the terminal states.
%We next discuss the assumptions over the trajectories in $\mathcal{D}$.
% \begin{figure}
% \includegraphics[scale=0.4]{images/trajectories.pdf}
% \caption{This figure shows an example state-partition with two independent trajectories in black and red. It may be noted that the terminal state is drawn as a separate state partition because we assume it to be indicated by the environment and not learned.}
% \label{fig:trajectories}
% \end{figure}
% \textbf{Assumptions.}
% There are certain assumptions that we make about $\mathcal{D}$, which we use in Section \ref{sec:learningsubgoal} and \ref{sec:reward}, and they are as follows:
% \begin{itemize}[leftmargin=*]
% \item \textbf{Completeness.} We assume that all the trajectories are complete, i.e., there exists no trajectory in $\mathcal{D}$ which is a temporal subpart of another trajectory in the dataset. In other words, we assume that all the trajectories start from the possible initial states only and continue to the possible terminal states. If there are a lot of trajectories which are sub-parts of the otherwise complete trajectories, e.g., starting from $\mathcal{S}_2$ in Fig. \ref{fig:trajectories} and ending to the terminal states, then the sub-goal learning algorithm would assume during the learning process that both $\mathcal{S}_1$ and $\mathcal{S}_2$ to belong to the same sub-goal, leading to irregular sub-goals partitions.
% \item \textbf{Causality.} There is a certain ordering in the states that appear in the trajectories, i.e., if a certain group of states appears before another group of states in a trajectory, then there should not be any trajectory where the occurrence of these groups of states is reversed. For example, if there are trajectories which move from $\dots \rightarrow \mathcal{S}_1 \rightarrow \mathcal{S}_2 \rightarrow \mathcal{S}_1 \dots \rightarrow$, then $\mathcal{S}_1$ and $\mathcal{S}_2$ may be predicted to be the same sub-goal. It may be noted that the states within a certain group of states, i.e., sub-goals, should not follow any order. In other words, a better ordering of states in the trajectories, will lead to more regular sub-goal partitions.
% \end{itemize}
% We also assume that the number of sub-goals are specified by the user, potentially based on domain knowledge.
\subsection{Learning Sub-Goal Prediction} \label{sec:learningsubgoal}
We use $\mathcal{D}$ to partition the state-space into $n_g$ sub-goals, with $n_g$ being a hyperparameter. We learn a neural network to approximate $\pi_\phi(g|\boldsymbol{s})$, which given a state $\boldsymbol{s} \in \mathcal{S}$ predicts a probability mass function (p.m.f.) over the possible sub-goal partitions $g \in G$. The order in which the sub-goals occur in the trajectories, i.e., $\mathcal{S}_1 < \mathcal{S}_2 < \dots < \mathcal{S}_{n_g}$, acts as a supervisory signal, which can be derived from our assumption mentioned above.
We propose an iterative framework to learn $\pi_\phi(g|\boldsymbol{s})$ using these ordered constraints. In the first step, we learn a mapping from states to sub-goals using equipartition labels among the sub-goals. Then we infer the labels of the states in the trajectories and correct them by imposing ordering constraints. We use the new labels to again train the network and follow the same procedure until convergence. These two steps are as follows.
\textbf{Learning Step.} %\label{sec:learningstep}
In this step we consider that we have a set of tuples $(\boldsymbol{s}, g)$, which we use to learn the function $\pi_\phi$, which can be posed as a multi-class classification problem with $n_g$ categories. We optimize the following cross-entropy loss function,
\begin{equation}
\pi_\phi^* = \argmin_{\pi_\phi} \frac{1}{N} \sum_{i=1}^{n_d} \sum_{t=1}^{n_i} \sum_{k=1}^{n_g} -\boldsymbol{1}\{g_{ti}=k\} \log \pi_\phi(g=j|\boldsymbol{s}_{ti})
\label{eq:cent}
\end{equation}
where $\boldsymbol{1}$ is the indicator function and $N$ is the number of states in the dataset $\mathcal{D}$. To begin with, we do not have any labels $g$, and thus we consider equipartition of all the sub-goals in $G$ along each trajectory. That is, given a trajectory of states $\{\boldsymbol{s}_{1i}, \boldsymbol{s}_{2i}, \dots, \boldsymbol{s}_{n_ii}\}$ for some $i \in \{1, 2, \dots, n_d\}$, the initial sub-goals are,\begin{equation}
g_{ti}=j, \ \ \ \forall \ \floor{\frac{(j-1)n_i}{n_g}} < t <= \floor{\frac{jn_i}{n_g}}, \ j \in G
\label{eq:initgoal}
\end{equation}
Using this initial labeling scheme, similar states across trajectories may have different labels, but the network is expected to converge at the Maximum Likelihood Estimate (MLE) of the entire dataset. We also optimize CASL \cite{paul2018w} for stable learning as the initial labels can be erroneous. In the next iteration of the learning step, we use the inferred sub-goal labels, which we obtain as follows.
%Once we perform an initial training step using these equipartition labels, we use the learned network to predict the labels of the states, which we then correct using ordering constraints.
\textbf{Inference Step.}
Although the equipartition labels in Eqn. \ref{eq:initgoal} may have similar states across different trajectories mapped to dissimilar sub-goals, the learned network modeling $\pi_\phi$ provides maps similar states to the same sub-goal. But, Eqn. \ref{eq:cent}, and thus the predictions of $\pi_\phi$ does not account for the natural temporal ordering of the sub-goals. Even with architectures such as Recurrent Neural Networks (RNN), it may be better to impose such temporal order constraints explicitly rather than relying on the network to learn them. We inject such order constraints using Dynamic Time Warping (DTW).% \cite{keogh2001derivative}.
Formally, for the $i^{th}$ trajectory in $\mathcal{D}$, we obtain the following set: $\{(\boldsymbol{s}_{ti}, \boldsymbol{\pi_\phi}(g|\boldsymbol{s}_{ti})\}_{t=1}^{n_i}$, where $\boldsymbol{\pi_\phi}$ is a vector representing the p.m.f. over the sub-goals $G$. However, as the predictions do not consider temporal ordering, the constraint that sub-goal $j$ occurs after sub-goal $i$, for $i<j$, is not preserved. To impose such constraints, we use DTW between the two sequences $\{\boldsymbol{e}_1, \boldsymbol{e}_2, \dots, \boldsymbol{e}_{n_g}\}$, which are the standard basis vectors in the $n_g$ dimensional Euclidean space and $\{ \boldsymbol{\pi_\phi}(g|\boldsymbol{s}_{1i}), \boldsymbol{\pi_\phi}(g|\boldsymbol{s}_{2i}), \dots, \boldsymbol{\pi_\phi}(g|\boldsymbol{s}_{n_ii}) \}$. We use the $l1$-norm of the difference between two vectors as the similarity measure in DTW. In this process, we obtain a sub-goal assignment for each state in the trajectories, which become the new labels for training in the learning step.
We then invoke the learning step using the new labels (instead of Eqn. \ref{eq:initgoal}), followed by the inference step to obtain the next sub-goal labels. We continue this process until the number of sub-goal labels changed between iterations is less than a certain threshold. This method is presented in Algorithm \ref{alg:subgoal}, where the superscript $k$ represents the iteration number in learning-inference alternates.
\textbf{Reward Using Sub-Goals.}
\label{sec:reward}
The ordering of the sub-goals, as discussed before, provides a natural way of designing a reward function as follows:
\begin{equation}
r'(\boldsymbol{s}, a, \boldsymbol{s}') =
\gamma*\argmax_{j \in G} \pi_\phi(g=j|\boldsymbol{s}') - \argmax_{k \in G} \pi_\phi(g=k|\boldsymbol{s})
\label{eq:extreward}
\end{equation}
where the agent in state $\boldsymbol{s}$ takes action $a$ and reaches state $\boldsymbol{s}'$. The augmented reward function would become $r+r'$. Considering that we have a function of the form $\Phi_\phi(\boldsymbol{s})=\argmax_{j \in G} \pi_\phi(g=j|\boldsymbol{s})$, and without loss of generality that $G = \{0,1, \dots, n_g-1\}$, so that for the initial state $\Phi_\phi(\boldsymbol{s}_0)=0$, it follows from \cite{ng1999policy} that every optimal policy in $\mathcal{M}' = (\mathcal{S}, \mathcal{A}, \mathcal{P}, r+r', \gamma, \mathcal{P}_0)$, will also be optimal in $\mathcal{M}$. However, the new reward function may help to learn the task faster.
%It is interesting to note that the RL agent would prefer a route from $\mathcal{S}_i$ to $\mathcal{S}_j$, with $j-i \geq 2$, rather than traversing $i \rightarrow i+1 \rightarrow \dots \rightarrow j$, as the former would result in a higher discounted reward compared to the later. This shows that in case of sub-optimal trajectories, the RL agent may be able to surpass it. We do see such behavior in experiments.
\textbf{Out-of-Set Augmentation.}
In several applications, it might be the case that the trajectories only cover a small subset of the state space, while the agent, during the RL step, may visit states outside of the states in $\mathcal{D}$. The sub-goals estimated at these out-of-set states may be erroneous.
%In case of high dimensional state-space, the number of trajectories required to learn an appropriate partition of state-space into sub-goals may be enormous, obtaining which may be infeasible in practice. In this section, we propose a method to cope with this problem. We consider that we have access to only a small set of expert trajectories. In that case, the sub-goal predictor $\pi_\phi$ may not generalize well to all states, and thus may not be accurate at states which are not well represented in the expert trajectory set.
To alleviate this problem, we use a logical assertion on the potential function $\Phi_\phi(\boldsymbol{s})$ that the sub-goal predictor is confident only for states which are well-represented in $\mathcal{D}$, and not elsewhere. We learn a neural network to model a utility function $u_\psi: \mathcal{S} \rightarrow \mathbb{R}$, which given a state, predicts the degree by which it is seen in the dataset $\mathcal{D}$. To do this, we build upon Deep One-Class Classification \cite{ruff2018deep}, which performs well on the task of anomaly detection. The idea is derived from Support Vector Data Description (SVDD) \cite{tax2004support}, which aims to find the smallest hypersphere enclosing the given data points with minimum error. Data points outside the sphere are then deemed as anomalous. We learn the parameters of $u_\psi$ by optimizing the following function:
\begin{equation*}
\psi^* = \argmin_{\psi} \frac{1}{N} \sum_{i=1}^{n_d} \sum_{t=1}^{n_i} ||f_\psi(\boldsymbol{s}_{ti}) - \boldsymbol{c}||^2 + \lambda ||\psi||_2^2,
\label{eq:oneclass}
\end{equation*}
where $\boldsymbol{c} \in \mathbb{R}^m$ is a vector determined a priori \cite{ruff2018deep}, $f$ is modeled by a neural network with parameters $\psi$, s.t. $f_\psi(\boldsymbol{s}) \in \mathbb{R}^m$. The second part is the $l2$ regularization loss with all the parameters of the network lumped to $\psi$. The utility function $u_\psi$ can be expressed as follows:
\begin{equation}
u_\psi(\boldsymbol{s}) = ||f_\psi(\boldsymbol{s}) - \boldsymbol{c}||_2^2
\end{equation}
A lower value of $u_\psi(\boldsymbol{s})$ indicates that the state has been seen in $\mathcal{D}$. We modify the potential function $\Phi_\phi(\boldsymbol{s})$ and thus the extrinsic reward function, to incorporate the utility score as follows:
\begin{align}
\Phi_{\phi,\psi}(\boldsymbol{s})=\boldsymbol{1}\{u_\psi(\boldsymbol{s}) \leq \delta\} & * \argmax_{j \in G} \pi_\phi(g=j|\boldsymbol{s}) \nonumber, \\
r'(\boldsymbol{s}, a, \boldsymbol{s}') =& \gamma \Phi_{\phi, \psi}(\boldsymbol{s}') - \Phi_{\phi, \psi}(\boldsymbol{s}),
\label{eq:1classextreward}
\end{align}
where $\Phi_{\phi, \psi}$ denotes the modified potential function. It may be noted that as the extrinsic reward function is still a potential-based function \cite{ng1999policy}, the optimality conditions between the MDP $\mathcal{M}$ and $\mathcal{M}'$ still hold as discussed previously.
%Intuitively speaking, such a modification to the reward function provides a higher reward function for transitions to higher sub-goals through the distribution of states within the trajectory set. Consider that the RL agent moves from sub-goal $j$ to a set of states for $n$ time-steps outside the distribution as in the dataset and then returns to sub-goal $k (>j)$. This would result in a cumulative reward of $\gamma^{n+1}k - j$. However, if the RL agent moves always through the distribution of states within the trajectory set, then it would atleast obtain a cumulative reward of $\gamma^{n+1}k-\gamma^{n+1}j$, if it completes the transition with the same $n$ steps.
\begin{wrapfigure}{l}{0.6\textwidth}
\begin{minipage}{0.6\textwidth}
\begin{algorithm}[H]
\caption{Learning Sub-Goal Prediction}
\label{alg:subgoal}
\begin{algorithmic}
\STATE {\bfseries Input:} Expert trajectory set $\mathcal{D}$
\STATE {\bfseries Output:} Sub-goal predictor $\pi_\phi(g|\boldsymbol{s})$
\STATE $k \leftarrow 0$
\STATE Obtain $g^k$ for each $\boldsymbol{s} \in \mathcal{D}$ using Eqn. \ref{eq:initgoal}
\REPEAT
\STATE Optimize Eqn. \ref{eq:cent} to obtain $\pi_\phi^k$
\STATE Predict p.m.f of $G$ for each $\boldsymbol{s} \in \mathcal{D}$ using $\pi_\phi^k$
\STATE Obtain new sub-goals $g^{k+1}$ using the p.m.f in DTW
\STATE done = True, if $|g^k - g^{k+1}| < \epsilon$, else False
\STATE $k \leftarrow k + 1$
\UNTIL{done is True}
\end{algorithmic}
\end{algorithm}
\end{minipage}
\end{wrapfigure}
\textbf{Supervised Pre-Training.}
We first pre-train the policy network using the trajectories $\mathcal{D}$ (details in supplementary). The performance of the pre-trained policy network is generally quite poor and is upper bounded by the expert performance from which the trajectories are drawn. We then employ RL, which starts from the pre-trained policy, to learn from the subgoal based reward function. Unlike standard imitation learning algorithms, e.g., DAgger, which finetune the pre-trained policy with the expert in the loop, our algorithm only uses the initial set of expert trajectories and does not invoke the expert otherwise.
\section{Experiments}
\label{exp}
\begin{figure}[t]
\begin{minipage}[c]{0.54\textwidth}
\begin{subfigure}{0.32\textwidth}
\centering
\includegraphics[height=2cm]{images/bimgame.png}
\caption{BiMGame}
\label{fig:task1}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\centering
\includegraphics[height=2cm]{images/anttarget.png}
\caption{AntTarget}
\label{fig:task2}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\centering
\includegraphics[height=2cm]{images/antmaze.png}
\caption{AntMaze}
\label{fig:task3}
\end{subfigure}
\end{minipage}
\begin{minipage}[c]{0.45\textwidth}
\caption{This figure presents the three environments used in this paper - (a) Ball-in-Maze Game (BiMGame) (b) Ant locomotion in an open environment with an end goal (AntTarget) (c) Ant locomotion in a maze with an end goal (AntMaze) }
\label{fig:tasks}
\end{minipage}
%\vspace{-1mm}
\end{figure}
In this section, we perform experimental evaluation of the proposed method of learning from trajectories and compare it with other state-of-the-art methods. We also perform ablation of different modules of our framework.
% \begin{wrapfigure}{r}{0.57\textwidth}
% \centering
% \begin{subfigure}{0.18\textwidth}
% \centering
% \includegraphics[height=2cm]{images/maze_sim.png}
% \caption{}
% \label{fig:task1}
% \end{subfigure}
% \begin{subfigure}{0.18\textwidth}
% \centering
% \includegraphics[height=2cm]{images/AntTarget.png}
% \caption{}
% \label{fig:task2}
% \end{subfigure}
% \begin{subfigure}{0.18\textwidth}
% \centering
% \includegraphics[height=2cm]{images/AntMaze.png}
% \caption{}
% \label{fig:task3}
% \end{subfigure}
% \caption{This figure presents the three environments used in this paper - (a) Ball-in-Maze Game (BiMGame) (b) Ant locomotion in an open environment with an end goal (AntTarget) (c) Ant locomotion in a maze with an end goal (AntMaze) }
% \label{fig:task}
% \end{wrapfigure}
\textbf{Tasks.} We perform experiments on three challenging environments as shown in Fig. \ref{fig:tasks}. First is Ball-in-Maze Game (BiMGame) introduced in \cite{van2018sim2}, where the task is to move a ball from the outermost to the innermost ring using a set of five discrete actions - clock-wise and anti-clockwise rotation by $1^\circ$ along the two principal dimensions of the board and ``no-op" where the current orientation of the board is maintained. The states are images of size $84 \times 84$. The second environment is AntTarget which involves the Ant \cite{schulman2015high}. The task is to reach the center of a circle of radius $5$m with the Ant being initialized on a $45^\circ$ arc of the circle. The state and action are continuous with $41$ and $8$ dimensions respectively. The third environment, AntMaze, uses the same Ant, but in a U-shaped maze used in \cite{held2017automatic}. The Ant is initialized on one end of the maze with the goal being the other end indicated as red in Fig. \ref{fig:task3}. Details about the network architectures we use for $\pi_\theta$, $\pi_\phi$ and $f_\psi(s)$ can be found in the supplementary material.
\textbf{Reward.} For all tasks, we use sparse terminal-only reward, i.e., $+1$ only after reaching the goal state and $0$ otherwise. Standard RL methods such as A3C \cite{mnih2016asynchronous} are not able to solve these tasks with such sparse rewards.
\textbf{Trajectory Generation.}
We generate trajectories from A3C \cite{mnih2016asynchronous} policies trained with dense reward, which we do not use in any other experiments. We also generate sub-optimal trajectories for BiMGame and AntMaze. To do so for BiMGame, we use the simulator via Model Predictive Control (MPC) as in \cite{paul2018trajectory} (details in the supplementary). For AntMaze, we generate sub-optimal trajectories from an A3C policy stopped much before convergence. We generate around $400$ trajectories for BiMGame and AntMaze, and $250$ for AntTarget. As we generate two separate sets of trajectories for BiMGame and AntTarget, we use the sub-optimal set for all experiments, unless otherwise mentioned. % where we use the optimal set of trajectories for ablation studies.
\begin{figure}
\centering
\hspace{-10mm}
\begin{subfigure}{0.329\textwidth}
\centering
\includegraphics[height=3cm]{images/bimgame_bl_4sg.pdf}
\caption{BiMGame}
\label{fig:bimgame_bl}
\end{subfigure}
\begin{subfigure}{0.329\textwidth}
\centering
\includegraphics[height=3cm]{images/anttarget_bl_10sg_noerr.pdf}
\caption{AntTarget}
\label{fig:anttarget_bl}
\end{subfigure}
\begin{subfigure}{0.329\textwidth}
\centering
\includegraphics[height=3cm]{images/antmaze_bl_20sg.pdf}
\caption{AntMaze}
\label{fig:antmaze_bl}
\end{subfigure}
\caption{This figure shows the comparison of our proposed method with the baselines. Some lines may not be visible as they overlap. For tasks (a) and (c) our method clearly outperforms others. For task (b), although value reward initially performs better, our method eventually achieves the same performance. For a fair comparison, we do not use the out-of-set augmentation to generate this plot.}
\label{fig:baselines}
%\vspace{-1mm}
\end{figure}
%% (a) This plot presents the reward for the BiMGame task with a terminal reward of $+1$. (b) To zoom into the progress of the baselines, this plots presents the cumulative reward of $\pm 1$ for moving to a ring towards or away from the goal (used only for plotting), for a single run, while using the same $+1$ terminal reward to learn.
\textbf{Baselines.}
We primarily compare our method with RL methods which utilize trajectory or expert information - AggreVaTeD \cite{sun2017deeply} and value based reward shaping \cite{ng1999policy}, equivalent to the $K=\infty$ in THOR \cite{sun2018truncated}. For these methods, we use $\mathcal{D}$ to fit a value function to the sparse terminal-only reward of the original MDP $\mathcal{M}$ and use it as the expert value function. We also compare with standard A3C, but pre-trained using $\mathcal{D}$. It may be noted that we pre-train all the methods using the trajectory set to have a fair comparison. We report results with mean cumulative reward and $\pm \sigma$ over $3$ independent runs. %We use A3C with $6-10$ parallel agents in the RL stage of all the methods.
% \begin{figure}[!htb]
% \begin{center}
% \begin{subfigure}{0.48\textwidth}
% \includegraphics[scale=0.18]{images/Figure_4.pdf}
% \caption{}
% \label{fig:fig3}
% \end{subfigure}
% \hfill
% \begin{subfigure}{0.48\textwidth}
% \includegraphics[scale=0.18]{images/Figure_3.pdf}
% \caption{}
% \label{fig:fig4}
% \end{subfigure}
% \caption{(a) This plot presents the effectiveness of using Eqn. \ref{eq:extreward} (green) vs. \ref{eq:1classextreward} (red) for reward shaping. (b) This figure presents the variations in rate of convergence with changes in number of sub-goals $n_g$.}
% \label{fig:im2}
% \end{center}
% \end{figure}
\begin{figure}
\centering
\hspace{-10mm}
\begin{subfigure}{0.33\textwidth}
\centering
\includegraphics[height=3cm]{images/bimgame_sg.pdf}
\caption{BiMGame}
\label{fig:bimgame_sg}
\end{subfigure}
\begin{subfigure}{0.33\textwidth}
\centering
\includegraphics[height=3cm]{images/anttarget_sg.pdf}
\caption{AntTarget}
\label{fig:anttarget_sg}
\end{subfigure}
\begin{subfigure}{0.33\textwidth}
\centering
\includegraphics[height=3cm]{images/antmaze_sg.pdf}
\caption{AntMaze}
\label{fig:antmaze_sg}
\end{subfigure}
\caption{(a) This plot presents the learning curves associated with different number of learned sub-goals for the three tasks. For BiMGame and AntTarget, the number of sub-goals hardly matters. However, due to the inherently longer length of the task for AntMaze , lower number of sub-goals such as $n_g=5$ perform much worse than with higher $n_g$.}
\label{fig:subgoals}
\vspace{-4mm}
\end{figure}
\textbf{Comparison with Baselines.}
First, we compare our method with other baselines in Fig \ref{fig:baselines}. Note that as out-of-set augmentation using $u_\psi$ can be applied for other methods which learn from trajectories, such as value-based reward shaping, we present the results for comparison with baselines without using $u_\psi$, i.e., Eqn. \ref{eq:extreward}. Later, we perform an ablation study with and without using $u_\psi$. As may be observed, none of the baselines show any sign of learning for the tasks, except for ValueReward, which performs comparably with the proposed method for AntTarget only. Our method, on the other hand, is able to learn and solve the tasks consistently over multiple runs. The expert cumulative rewards are also drawn as straight lines in the plots and imitation learning methods like DAgger \cite{ross2011reduction} can only reach that mark. %Note that we use the sub-optimal trajectory set for BiMGame and AntMaze for this comparison.
Our method is able to surpass the expert for all the tasks. In fact, for AntMaze, even with a rather sub-optimal expert (an average cumulative reward of only $0.0002$), our algorithm achieves about $0.012$ cumulative reward at $100$ million steps.
The poor performance of the ValueReward and AggreVaTeD can be attributed to the imperfect value function learned with a limited number of trajectories. Specifically, with an increase in the trajectory length, the variations in cumulative reward in the initial set of states are quite high. This introduces a considerable amount of error in the estimated value function in the initial states, which in turn traps the agent in some local optima when such value functions are used to guide the learning process.
%Due to the terminal reward of $+1$, even if the RL agent moves through some of the rings, it does not receive any reward. Thus, to zoom into the learning curve of the baselines, we plot in Fig. \ref{fig:fig2} a more dense reward function of $\pm 1$ whenever the ball moves to a ring towards or away from the goal. However, note that we still use only the terminal reward for learning and use the dense reward only for plotting. The learning curves for a single run are presented in Fig. \ref{fig:fig2}. It shows that although the RL agents move into rings towards the goal, it is not able to solve the task.
\textbf{Variations in Sub-Goals.}
The number of sub-goals $n_g$ is specified by the user, based on domain knowledge. For example, in the BiMGame, the task has four bottle-necks, which are states to be visited to complete the task and they can be considered as sub-goals. We perform experiments with different sub-goals and present the plots in Fig. \ref{fig:subgoals}. It may be observed that for BiMGame and AntTarget, our method performs well over a large variety of sub-goals. On the other hand for AntMaze, as the length of the task is much longer than AntTarget (12m vs 5m), $n_g\geq 10$ learn much faster than $n_g=5$, as higher number of sub-goals provides more frequent rewards. Note that the variations in speed of learning with number of sub-goals is also dependent on the number of expert trajectories. If the pre-training is good, then less frequent sub-goals might work fine, whereas if we have a small number of expert trajectories, the RL agent may need more frequent reward (see the supplementary material for more experiments).
%this only holds when the networks are pre-trained using $\mathcal{D}$. Omitting pre-training is equivalent to standard RL with sub-goal rewards, and will perform worse compared to if we start from model pre-trained on the expert trajectories. For AntMaze, we see such behavior even with pre-trained weights, as $n_g=5$ performs much worse compared to $n_g>=10$. This is because, AntTarget involves traversing a shortest path of $5$m, while it is $12$m for AntMaze. (see the supplementary material for more experiments).
%We would expect that with an increasing number of sub-goals, the rate of convergence increases. We do see such behavior, where with $n_g=2$, initial phase of the is somewhat delayed compared to cases with higher $n_g$. Also, with higher $n_g$, the agents reach a higher cumulative reward compared to $n_g=2$. However, note that with very high $n_g$ is tasks where the same goal can be reached using multiple paths, some of the state visitation assumptions of the trajectories in Section \ref{sec:assume} may tend to fail leading to haphazard sub-goal partition. Thus, extreme values of $n_g$ should be avoided and domain knowledge may be invoked to design the parameter.
\begin{figure}
\centering
\hspace{-10mm}
\begin{minipage}[c]{0.70\textwidth}
\begin{subfigure}{0.49\textwidth}
\centering
\includegraphics[scale=0.12]{images/bimgame_oncl_comp_4sg.pdf}
\caption{BiMGame}
\label{fig:bimgame_oncl}
\end{subfigure}
\begin{subfigure}{0.49\textwidth}
\centering
\includegraphics[scale=0.12]{images/anttarget_oncl_comp_10sg.pdf}
\caption{AntTarget}
\label{fig:anttarget_oncl}
\end{subfigure}
\end{minipage}
\begin{minipage}[c]{0.31\textwidth}
\caption{This plot presents the comparison of our proposed method for with and without using the one-class classification method for out-of-set augmentation.}
\label{fig:oncl}
\end{minipage}
\vspace{-2mm}
\end{figure}
\textbf{Effect of Out-of-Set Augmentation.}
The set $\mathcal{D}$ may not cover the entire state-space. To deal with this situation we developed the extrinsic reward function in Eqn. \ref{eq:1classextreward} using $u_\psi$. To evaluate its effectiveness we execute our algorithm using Eqn. \ref{eq:extreward} and Eqn. \ref{eq:1classextreward}, and show the results in Fig. \ref{fig:oncl}, with legends showing without and with $u_\psi$ respectively. For BiMGame, we used the optimal A3C trajectories, for this evaluation. This is because, using MPC trajectories with Eqn.~\ref{eq:extreward} can still solve the task with similar reward plots, since MPC trajectories visit a lot more states due to its short-tem planning. The (optimal) A3C trajectories on the other hand, rarely visit some states, due to its long-term planning. In this case, using Eqn. \ref{eq:extreward} actually traps the agents to a local optimum (in the outermost ring), whereas using $u_\psi$ as in Eqn. \ref{eq:1classextreward}, learns to solve the task consistently (Fig. \ref{fig:bimgame_oncl}).
%%For BiMGame, we used the A3C generated trajectories, as the MPC trajectories, due to its short-term planning, visit a lot more states compared to the A3C trajectories. Using the MPC trajectories with Eqn. \ref{eq:extreward} can still solve the task with similar reward plots. However, the optimal A3C trajectories, due to its long-term planning, rarely visit some states and we found that using Eqn. \ref{eq:extreward} actually traps the agents to a local optimum (in the outermost ring). Using $u_\psi$ as in Eqn. \ref{eq:1classextreward}, learns to solve the task consistently (Fig. \ref{fig:bimgame_oncl}).
For AntTarget in Fig. \ref{fig:anttarget_oncl}, using $u_\psi$ performs better than without using $u_\psi$ (and also surpasses Value based Reward Shaping). This is because the trajectories only span a small sector of the circle (Fig. \ref{fig:anttarget_vis}) while the Ant is allowed to visit states outside of it in the RL step. Thus, $u_\psi$ avoids incorrect sub-goal assignments to states not well-represented in $\mathcal{D}$ and helps in the overall learning.
\begin{figure}[!t]
\centering
\hspace{-10mm}
\begin{minipage}[c]{0.7\textwidth}
\begin{subfigure}{0.49\textwidth}
\centering
\includegraphics[scale=0.12]{images/bimgame_expert_comp.pdf}
\caption{BiMGame}
\label{fig:a3c_vs_mpc}
\end{subfigure}
\begin{subfigure}{0.49\textwidth}
\centering
\includegraphics[scale=0.12]{images/antmaze_expert_comp.pdf}
\caption{AntMaze}
\label{fig:pretraining}
\end{subfigure}
\end{minipage}
\begin{minipage}[c]{0.3\textwidth}
\caption{This plot presents a comparison of our proposed method for two different types of expert trajectories. The corresponding expert rewards are also plotted as horizontal lines.}
\label{fig:expert_comp}
\end{minipage}
\vspace{-2mm}
\end{figure}
\begin{figure}
\centering
\hspace{-10mm}
\begin{minipage}[c]{0.7\textwidth}
\begin{subfigure}{0.32\textwidth}
\centering
\includegraphics[height=2.6cm]{images/bimgame_vis_4sg.png}
\caption{BiMGame \\ $n_g=4$}
\label{fig:bimgame_vis}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\centering
\includegraphics[height=2.6cm, width=3.5cm]{images/anttarget_vis_10sg_250_show120.png}
\caption{AntTarget \\ $n_g=10$}
\label{fig:anttarget_vis}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\centering
\includegraphics[height=2.6cm]{images/antmaze_vis_15sg.png}
\caption{AntMaze \\ $n_g=15$}
\label{fig:antmaze_vis}
\end{subfigure}
\end{minipage}
\begin{minipage}[c]{0.3\textwidth}
\caption{This figure presents the learned sub-goals for the three tasks which are color coded. Note that for (b) and (c), multiple sub-goals are assigned the same color, but they can be distinguished by their spatial locations. }
\label{fig:visualization}
\end{minipage}
\vspace{-2mm}
\end{figure}
\textbf{Effect of Sub-Optimal Expert.}
In general, the optimality of the expert may have an effect on performance. The comparison of our algorithm with optimal vs. sub-optimal expert trajectories are shown in Fig. \ref{fig:expert_comp}. As may be observed, the learning curve for both the tasks is better for the optimal expert trajectories. However, in spite of using such sub-optimal experts, our method is able to surpass and perform much better than the experts. We also see that our method performs better than even the optimal expert (as it is only \emph{optimal} w.r.t. some cost function) used in AntMaze.
\textbf{Visualization.}
We visualize the sub-goals discovered by our algorithm and plot it on the x-y plane in Fig. \ref{fig:visualization}. As can be seen in BiMGame, with $4$ sub-goals, our method is able to discover the bottle-neck regions of the board as different sub-goals. For AntTarget and AntMaze, the path to the goal is more or less equally divided into sub-goals. This shows that our method of sub-goal discovery can work for both environments with and without bottle-neck regions. The supplementary material has more visualizations and discussion.
%%We visualize the sub-goals discovered by our algorithm and plot it on the x-y plane in Fig. \ref{fig:visualization}. As can be seen in BiMGame, with $4$ sub-goals, our method is able to discover the bottle-neck regions of the board as different sub-goals. For AntTarget and AntMaze, we see as expected, i.e., the path to the goal is divided into the sub-goals (see supplementary for more visualizations and discussions). This shows that our method of sub-goal discovery can work both for environments which contain bottle-neck regions or some sort of ordering in the group of states.
\section{Discussions}
\label{discussions}
The experimental analysis we presented in the previous section contain the following key observations:\vspace{-8pt}
\begin{itemize}[leftmargin=*]
\item Our method for sub-goal discovery works both for tasks with inherent bottlenecks (e.g. BiMGame) and for tasks without any bottlenecks (e.g. AntTarget and AntMaze), but with temporal orderings between groups of states in the expert trajectories, which is the case for many applications.\vspace{-4pt}
\item Experiments show, that our assumption on the temporal ordering of groups of states in expert trajectories is soft, and determines the granularity of the discovered sub-goals (see supplementary).\vspace{-4pt}
\item Discrete rewards using sub-goals performs much better than value function based continuous rewards. Moreover, value functions learned from long and limited number of trajectories may be erroneous, whereas segmenting the trajectories based on temporal ordering may still work well.\vspace{-4pt}
\item As the expert trajectories may not cover the entire state-space regions the agent visits during exploration in the RL step, augmenting the sub-goal based reward function using out-of-set augmentation performs better compared to not using it.
\end{itemize}
%In several real-world tasks, there often exists some waypoints or bottleneck states which need to be achieved in order to achieve a bigger goal. Our assumptions in Section \ref{sec:assume} on the trajectories are often satisfied in such goal-oriented tasks and it becomes quite straight-forward to use our method in such cases.
%In one of our experiments for the BiMGame we note that a reward based on the geodesic distance to the goal, performs only as well as a $\pm 1$ reward when the ball gets through one of the openings, towards or away from the goal. The value based reward in \cite{sun2018truncated}, learned from the trajectories has a similar notion to such a distance based reward, but with errors. AggreVaTeD \cite{sun2017deeply} on the other hand can be detrimental if due to errors, the expert value function decreases for a state transition to a bottleneck state. In both cases, we see that the agent stagnates at a local optima as most of the time, the ball hovers near the openings in the outer-most ring, and rarely going through it. This is because there are a lot of possible ways to approach the gates, but the space of action sequences required to move through the gates is limited, and using the estimated value function only makes it worse in such cases.
\section{Conclusion}
\label{conclusion}
In this paper, we presented a framework to utilize the demonstration trajectories in an efficient manner by discovering sub-goals, which are waypoints that need to be completed in order to achieve a certain complex goal-oriented task. We use these sub-goals to augment the reward function of the task, without affecting the optimality of the learned policy. Experiments on three complex task show that unlike state-of-the-art RL, IL or methods which combines them, our method is able to solve the tasks consistently. We also show that our method is able to perform much better than sub-optimal experts used to obtain the expert trajectories and at least as good as the optimal experts. Our future work will concentrate on extending our method for repetitive non-goal oriented tasks.
\begin{thebibliography}{10}
\bibitem{mnih2015human}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei~A Rusu, Joel Veness,
Marc~G Bellemare, Alex Graves, Martin Riedmiller, Andreas~K Fidjeland, Georg
Ostrovski, et~al.
\newblock Human-level control through deep reinforcement learning.
\newblock {\em Nature}, 518(7540):529, 2015.
\bibitem{silver2016mastering}
David Silver, Aja Huang, Chris~J Maddison, Arthur Guez, Laurent Sifre, George
Van Den~Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda
Panneershelvam, Marc Lanctot, et~al.
\newblock Mastering the game of go with deep neural networks and tree search.
\newblock {\em nature}, 529(7587):484, 2016.
\bibitem{levine2016end}
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.
\newblock End-to-end training of deep visuomotor policies.
\newblock {\em JMLR}, 17(1):1334--1373, 2016.
\bibitem{duan2016benchmarking}
Yan Duan, Xi~Chen, Rein Houthooft, John Schulman, and Pieter Abbeel.
\newblock Benchmarking deep reinforcement learning for continuous control.
\newblock In {\em ICML}, pages 1329--1338, 2016.
\bibitem{bojarski2016end}
Mariusz Bojarski, Davide Del~Testa, Daniel Dworakowski, Bernhard Firner, Beat
Flepp, Prasoon Goyal, Lawrence~D Jackel, Mathew Monfort, Urs Muller, Jiakai
Zhang, et~al.
\newblock End to end learning for self-driving cars.
\newblock {\em arXiv preprint arXiv:1604.07316}, 2016.
\bibitem{argall2009survey}
Brenna~D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning.
\newblock A survey of robot learning from demonstration.
\newblock {\em Robotics and autonomous systems}, 57(5):469--483, 2009.
\bibitem{ross2011reduction}
St{\'e}phane Ross, Geoffrey Gordon, and Drew Bagnell.
\newblock A reduction of imitation learning and structured prediction to
no-regret online learning.
\newblock In {\em AISTATS}, pages 627--635, 2011.
\bibitem{sun2017deeply}
Wen Sun, Arun Venkatraman, Geoffrey~J Gordon, Byron Boots, and J~Andrew
Bagnell.
\newblock Deeply aggrevated: Differentiable imitation learning for sequential
prediction.
\newblock In {\em ICML}, pages 3309--3318, 2017.
\bibitem{cheng2018fast}
Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots.
\newblock Fast policy learning through imitation and reinforcement.
\newblock {\em UAI}, 2018.
\bibitem{nagabandi2018neural}
Anusha Nagabandi, Gregory Kahn, Ronald~S Fearing, and Sergey Levine.
\newblock Neural network dynamics for model-based deep reinforcement learning
with model-free fine-tuning.
\newblock In {\em ICRA}, pages 7559--7566. IEEE, 2018.
\bibitem{Andreas2017policysketch}
Jacob Andreas, Dan Klein, and Sergey Levine.
\newblock Modular multitask reinforcement learning with policy sketches.
\newblock In {\em ICML}, pages 166--175, 2017.
\bibitem{dubey2018investigating}
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas~L Griffiths, and Alexei~A
Efros.
\newblock Investigating human priors for playing video games.
\newblock {\em arXiv preprint arXiv:1802.10217}, 2018.
\bibitem{ng1999policy}
Andrew~Y Ng, Daishi Harada, and Stuart Russell.
\newblock Policy invariance under reward transformations: Theory and
application to reward shaping.
\newblock In {\em ICML}, volume~99, pages 278--287, 1999.
\bibitem{sun2018truncated}
Wen Sun, J~Andrew Bagnell, and Byron Boots.
\newblock Truncated horizon policy search: Combining reinforcement learning \&
imitation learning.
\newblock {\em arXiv preprint arXiv:1805.11240}, 2018.
\bibitem{todorov2014convex}
Emanuel Todorov.
\newblock Convex and analytically-invertible dynamics with contacts and
constraints: Theory and implementation in mujoco.
\newblock In {\em ICRA}, pages 6054--6061, 2014.
\bibitem{schaal1999imitation}
Stefan Schaal.
\newblock Is imitation learning the route to humanoid robots?
\newblock {\em Trends in cognitive sciences}, 3(6):233--242, 1999.
\bibitem{silver2008high}
David Silver, James Bagnell, and Anthony Stentz.
\newblock High performance outdoor navigation from overhead data using
imitation learning.
\newblock {\em RSS}, 2008.
\bibitem{chernova2009interactive}
Sonia Chernova and Manuela Veloso.
\newblock Interactive policy learning through confidence-based autonomy.
\newblock {\em JAIR}, 34:1--25, 2009.
\bibitem{rajeswaran2017learning}
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John
Schulman, Emanuel Todorov, and Sergey Levine.
\newblock Learning complex dexterous manipulation with deep reinforcement
learning and demonstrations.
\newblock {\em RSS}, 2017.
\bibitem{hester2018deep}
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal
Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et~al.
\newblock Deep q-learning from demonstrations.
\newblock In {\em Thirty-Second AAAI Conference on Artificial Intelligence},
2018.
\bibitem{ross2014reinforcement}
Stephane Ross and J~Andrew Bagnell.
\newblock Reinforcement and imitation learning via interactive no-regret
learning.
\newblock {\em arXiv preprint arXiv:1406.5979}, 2014.
\bibitem{torabi2018behavioral}
Faraz Torabi, Garrett Warnell, and Peter Stone.
\newblock Behavioral cloning from observation.
\newblock {\em IJCAI}, 2018.
\bibitem{levine2013guided}
Sergey Levine and Vladlen Koltun.
\newblock Guided policy search.
\newblock In {\em ICML}, pages 1--9, 2013.
\bibitem{chang2015learning}
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daume~III, and John
Langford.
\newblock Learning to search better than your teacher.
\newblock {\em ICML}, 2015.
\bibitem{ranchod2015nonparametric}
Pravesh Ranchod, Benjamin Rosman, and George Konidaris.
\newblock Nonparametric bayesian reward segmentation for skill discovery using
inverse reinforcement learning.
\newblock In {\em IROS}, pages 471--477. IEEE, 2015.
\bibitem{murali2016tsc}
Adithyavairavan Murali, Animesh Garg, Sanjay Krishnan, Florian~T Pokorny,
Pieter Abbeel, Trevor Darrell, and Ken Goldberg.
\newblock Tsc-dl: Unsupervised trajectory segmentation of multi-modal surgical
demonstrations with deep learning.
\newblock In {\em ICRA}, pages 4150--4157, 2016.
\bibitem{krishnan2019swirl}
Sanjay Krishnan, Animesh Garg, Richard Liaw, Brijen Thananjeyan, Lauren Miller,
Florian~T Pokorny, and Ken Goldberg.
\newblock Swirl: A sequential windowed inverse reinforcement learning algorithm
for robot tasks with delayed rewards.
\newblock {\em IJRR}, 38(2-3):126--145, 2019.
\bibitem{sutton1999between}
Richard~S Sutton, Doina Precup, and Satinder Singh.
\newblock Between mdps and semi-mdps: A framework for temporal abstraction in
reinforcement learning.
\newblock {\em Artificial intelligence}, 112(1-2):181--211, 1999.
\bibitem{precup2000temporal}
Doina Precup.
\newblock {\em Temporal abstraction in reinforcement learning}.
\newblock University of Massachusetts Amherst, 2000.
\bibitem{stolle2002learning}
Martin Stolle and Doina Precup.
\newblock Learning options in reinforcement learning.
\newblock In {\em SARA}, pages 212--223. Springer, 2002.
\bibitem{silver2012compositional}
David Silver and Kamil Ciosek.
\newblock Compositional planning using optimal option models.
\newblock {\em arXiv preprint arXiv:1206.6473}, 2012.
\bibitem{Florensa2017StochasticNN}
Carlos Florensa, Yan Duan, and Pieter Abbeel.
\newblock Stochastic neural networks for hierarchical reinforcement learning.
\newblock {\em ICLR}, 2017.
\bibitem{bacon2017option}
Pierre-Luc Bacon, Jean Harb, and Doina Precup.
\newblock The option-critic architecture.
\newblock In {\em AAAI}, pages 1726--1734, 2017.
\bibitem{riemerlearning}
Matthew Riemer, Miao Liu, and Gerald Tesauro.
\newblock Learning abstract options.
\newblock {\em NIPS}, 2018.
\bibitem{held2017automatic}
David Held, Xinyang Geng, Carlos Florensa, and Pieter Abbeel.
\newblock Automatic goal generation for reinforcement learning agents.
\newblock {\em ICML}, 2017.
\bibitem{mcgovern2001automatic}
Amy McGovern and Andrew~G Barto.
\newblock Automatic discovery of subgoals in reinforcement learning using
diverse density.
\newblock {\em ICML}, 2001.
\bibitem{csimcsek2004using}
{\"O}zg{\"u}r {\c{S}}im{\c{s}}ek and Andrew~G Barto.
\newblock Using relative novelty to identify useful temporal abstractions in
reinforcement learning.
\newblock In {\em ICML}, page~95. ACM, 2004.
\bibitem{csimcsek2005identifying}
{\"O}zg{\"u}r {\c{S}}im{\c{s}}ek, Alicia~P Wolfe, and Andrew~G Barto.
\newblock Identifying useful subgoals in reinforcement learning by local graph
partitioning.
\newblock In {\em ICML}, pages 816--823. ACM, 2005.
\bibitem{mannor2004dynamic}
Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein.
\newblock Dynamic abstraction in reinforcement learning via clustering.
\newblock In {\em ICML}, page~71. ACM, 2004.
\bibitem{paul2018w}
Sujoy Paul, Sourya Roy, and Amit~K Roy-Chowdhury.
\newblock W-talc: Weakly-supervised temporal activity localization and
classification.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}, pages 563--579, 2018.
\bibitem{ruff2018deep}
Lukas Ruff, Nico G{\"o}rnitz, Lucas Deecke, Shoaib~Ahmed Siddiqui, Robert
Vandermeulen, Alexander Binder, Emmanuel M{\"u}ller, and Marius Kloft.
\newblock Deep one-class classification.
\newblock In {\em ICML}, pages 4390--4399, 2018.
\bibitem{tax2004support}
David~MJ Tax and Robert~PW Duin.
\newblock Support vector data description.
\newblock {\em Machine learning}, 54(1):45--66, 2004.
\bibitem{van2018sim2}
Jeroen van Baar, Alan Sullivan, Radu Cordorel, Devesh Jha, Diego Romeres, and
Daniel Nikovski.
\newblock Sim-to-real transfer learning using robustified controllers in
robotic tasks involving complex dynamics.
\newblock {\em ICRA}, 2018.
\bibitem{schulman2015high}
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter
Abbeel.
\newblock High-dimensional continuous control using generalized advantage
estimation.
\newblock {\em ICLR}, 2015.
\bibitem{mnih2016asynchronous}
Volodymyr Mnih, Adria~Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy
Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
\newblock Asynchronous methods for deep reinforcement learning.
\newblock In {\em ICML}, pages 1928--1937, 2016.
\bibitem{paul2018trajectory}
Sujoy Paul and Jeroen van Baar.
\newblock Trajectory-based learning for ball-in-maze games.
\newblock {\em arXiv preprint arXiv:1811.11441}, 2018.
\end{thebibliography}