Acquiring Target Stacking Skills by Goal-Parameterized Deep Reinforcement Learning
Extracted source text ยท arXiv 1711.00267
\documentclass{article} % For LaTeX2e
\usepackage{iclr2018_conference,times}
\usepackage{hyperref}
\usepackage{url}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{array}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
\newcommand{\ra}[1]{\renewcommand{\arraystretch}{#1}}
\title{Acquiring Target Stacking Skills by Goal-Parameterized Deep Reinforcement Learning}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Wenbin Li \\
Max Planck Institute for Informatics\\
Saarland Informatics Campus\\
Germany \\
\texttt{wenbinli@mpi-inf.mpg.de} \\
\And
Jeannette Bohg \\
Department of Computer Science \\
Stanford University \\
USA \\
\texttt{bohg@stanford.edu} \\
\AND
Mario Fritz \\
Max Planck Institute for Informatics \\
Saarland Informatics Campus\\
Germany \\
\texttt{mfritz@mpi-inf.mpg.de} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
Understanding physical phenomena is a key component of human intelligence and enables physical interaction with previously unseen environments. In this paper, we study how an artificial agent can autonomously acquire this intuition through interaction with the environment. We created a synthetic block stacking environment with physics simulation in which the agent can learn a policy end-to-end through trial and error. Thereby, we bypass to explicitly model physical knowledge within the policy. We are specifically interested in tasks that require the agent to reach a given goal state that may be different for every new trial. To this end, we propose a deep reinforcement learning framework that learns policies which are parametrized by a goal. We validated the model on a toy example navigating in a grid world with different target positions and in a block stacking task with different target structures of the final tower. In contrast to prior work, our policies show better generalization across different goals.
\end{abstract}
\input{intro}
\input{related}
\input{learning}
\input{experiment}
\input{conclusion}
%\subsubsection*{Acknowledgments}
%
%Use unnumbered third level headings for the acknowledgments. All
%acknowledgments, including those to funding agencies, go at the end of the paper.
\bibliography{mybib}
\bibliographystyle{iclr2018_conference}
\end{document}
\section{Introduction}
Understanding and predicting physical phenomena in daily life is an important component of human intelligence. This ability enables us to effortlessly manipulate objects in unseen conditions. It is an open question how this kind of knowledge can be represented and what kind of models could explain human manipulation behavior~\citep{yildirim2017physical}. In this paper we are concerned with the question of how an artificial agent can autonomously acquire physical interaction skills through trial and error.
Until recently, researcher have attempted to build computational models for capturing the essence of physical events via machine learning methods from sensory inputs \citep{mottaghi2015newtonian,wu2015galileo,fragkiadaki2015learning,apratim16ariv,fergus16blocsarxiv,li2016fall}. Yet there is little work to investigate how the knowledge captured by such a model can be directly applied for manipulation.
In this work, we aim to learn block stacking through trial-and-error, bypassing to explicitly model the corresponding physics knowledge. For this purpose, we build a synthetic environment with physics simulation, where the agent can move and stack blocks and observe the different outcomes of its actions. We apply deep reinforcement learning to directly acquire the block stacking skill in an end-to-end fashion.
While previous work focuses on learning policies for a fixed task, we introduce goal-parameterized policies that facilitate generalization of the learned skill to different targets.
We study this problem in the aforementioned block stacking task in which the agent has to reproduce a tower as shown in an image. The agent has to stack blocks into the same shape while retaining physical stability and avoiding pre-mature collisions with the existing structure.
In particular, we aim to learn a single model to guide the agent to build different shapes on request. This is generally not intended in conventional reinforcement learning formulations where the policy is typically optimized to reach a specific goal. In our learning framework, the varying goals are given to the agent as input. We first validated this extended model on a toy example where the agent has to navigate in a gridworld. Both, the location of the start and end point are randomized for each episode. We observed good generalization performance.
Then we apply the framework to the block stacking task. We show that execution depends on the desired target structure and observe promising results for generalization across different goals.
\section{Related Work}
Humans possess the amazing ability to perceive and understand ubiquitous physical phenomena occurring in their daily life. There is research in psychology that seeks to understand how this ability develops. \cite{baillargeon2002acquisition} suggest that infants acquire the knowledge of physical events at a very young age by observing those events, including support events and others.
Interestingly, in a recent work \cite{denil2016learning}, the authors introduce a basic set of tasks that require the learning agent to estimate physical properties (mass and cohesion combinations) of objects in an interactive simulated environment and find that it can learn to perform the experiments strategically to discover such hidden properties in analogy to human's development of physics knowledge.
\cite{battaglia2013simulation} proposes an intuitive physics simulation engine as an internal mechanism for such type of ability and found close correlation between its behavior patterns and human subjects' on several psychological tasks.
More recently, there is an increasing interest in equipping artificial agents with such an ability by letting them learn physical concepts from visual data.
\cite{mottaghi2015newtonian} aim at understanding dynamic events governed by laws of Newtonian physics and use proto-typical motion scenarios as exemplars. \cite{fragkiadaki2015learning} analyze billiard table scenarios and learn dynamics from observation with explicit object notion.
An alternative approach based on boundary extrapolation \cite{apratim16ariv} addresses similar settings without imposing any object notion.
\cite{wu2015galileo} aims to understand physical properties of objects based on explicit physical simulation.
\cite{mottaghi2017see} proposes to reason about containers and the behavior of the liquids inside them from a single RGB image.
Moreover, \cite{fergus16blocsarxiv} propose using a visual model to predict stability and falling trajectories for simple 4 block scenes. \cite{li2016fall} investigate if and how the prediction performance of such image-based models changes when trained on block stacking scenes with larger variety. They further examine how the human's prediction adapts to the variation in the generated scenes and compare to the learned visual model. Each work requires significant amounts of simulated, physically-realistic data to train the large-capacity, deep models.
Another interesting question that has been explored in psychology is how knowledge about physical events affects and guides human's actual interaction with objects \cite{yildirim2017physical}.
Yet it is not clear how machine model trained for physics understanding can be directly applied into real-world interactions with object and accomplish manipulation tasks.
\cite{li2017visual} makes a first attempt along this direction by extending their previous work \citep{li2016fall} on stability classification. They task a robot to place a wooden block on an existing structure while maintaining stability. Placement candidates are first generated and then evaluated through the visual stability classifier, so that only predicted stable placements are executed on the robot.
In this paper, reinforcement learning is used to learn an end-to-end model directly from the experience collected during interaction with a physically-realistic environment.
The majority of work in reinforcement learning focuses on solving task with a single goal. However, there are also tasks where the goal may change for every trial. It is not obvious how to directly apply the model learned towards a specific goal to a different one. An early idea has been proposed by \cite{kaelbling1993learning} for a maze navigation problem in which the goal changes. The author introduces an analogous formulation to the Q-learning by using shortest path in replacement of the value functions.
Yet there are two major limitations for the framework: 1) it is only formulated in tabular form which is not practical for application with complex states 2) the introduced shortest path is very specific to the maze navigation setting and hence cannot be easily adapt to handle task like different targets stacking, serving as a general solution to this type of problem.
In contrast, we propose a goal-parameterized model to integrate goal information into a general learning-based framework that facilitates generalization across different goals. The model has been shown to work on both a navigation task and target stacking.
Notably, \cite{schaul2015universal} also propose to integrate goal information into learning. However, they learn an embedding for state and goal to allow generalization in a reinforcement learning setup. The process is then incorporated into the Horde framework \citet{sutton2011horde}, where each agent learns towards different goals. In our work, we do not introduce a dedicated embedding learning but instead resort to an end-to-end approach where the function approximator will learn a direct mapping from sensory observations to actions that allows generalization across different goals.
\section{Target Stacking Task}
We introduce a new manipulation task: {\em target stacking\/}. In this task, an image of a target structure made of stacked blocks is provided. Given the same number of blocks as in the target structure, the goal is to reproduce the structure shown in the image. The manipulation primitives in this task include moving and placing blocks. This is inspired by the scenario where young children learn to stack blocks to different shapes given an example structure. We want to explore how an artificial agent can acquire such a skill through trial and error.
\subsection{Task Description}
For each task instance, a target structure is generated and its image is provided to the agent along with the number of blocks. Each of these blocks has a fixed orientation. The sequence of block orientations is such that reproducing the target is feasible. The agent attempts to construct the target structure by placing the blocks in the given sequence. The spawning location for each block is randomized along the top boundary of the environment. A sample task instance is shown in Figure~\ref{fig:tstack_demo}.
\begin{figure}
\centering
\includegraphics[width=1.\linewidth]{./figs/t_stack_eg.pdf}
\caption{Target stacking: Given a target shape image, the agent is required to move and stack blocks to reproduce it.}
\label{fig:tstack_demo}
\end{figure}
\subsection{Task Distinction}
The following characteristics distinguish this task from other tasks commonly used in the literature.
\paragraph{Goal-Specific}
A widely-used benchmark for deep reinforcement learning algorithm are the Atari games \citep{bellemare13arcade} that were made popular by \cite{mnih2013playing}. While this game collection has a large variety, the games are defined by a single goal or no specific goal is enforced at a particular point in time. For example in Breakout, the player tries to bounce off as many bricks as possible. In Enduro, the player tries to pass as many cars as possible while simultaneously avoiding cars.
In the target stacking task, each task instance differs in the specific goal (the target structure), and all the moves are planned towards this goal. Given the same state, moves that were optimal in one task instance are unlikely to be optimal in another task instance with a different target structure. This is in contrast to games where one type of move will most likely work in similar scenes. This argument also applies to AI research platforms with richer visuals like VizDoom \citep{kempka2016vizdoom}.
\paragraph{Longer sequences}
Target stacking requires looking ahead over a longer time horizon to simultaneously ensure stability and similarity to the target structure. This is different from learning to poke \citep{agrawal2016learning} where the objective is to select a motion primitive that is the optimal next action. It is also different from the work by \cite{li2017visual} that reasons about the placement of one block.
\paragraph{Rich Physics Bounded}
Besides stacking to the assigned target shape the agent needs to learn to move the block without colliding with the environment and existing structure and to choose the block's placement wisely not to collapse the current structure. The agent has no prior knowledge of this. It needs to learn everything from scratch by observing the consequence (collision, collapse) of its actions.
\subsection{Environment Implementation}
A deep reinforcement learning agent requires to learn from a larger number of samples. To enable this, we build a simulated environment for the agent to interact with physical-realistic task instances. While we keep the essential parts of the task, at its current stage the simulated environment remains an abstraction of a real-world robotics scenario. This generally requires an integration of multiple modules for a full-fledged working system, such as \cite{toussaint2010integrated}, which is out of scope of this paper.
In detail, the simulated stacking environment is implemented in Panda3D \citep{goslin2004panda3d} with bullet \citep{coumans2010bullet} as physics engine.
The block size follows a ratio of $l:w:h = 5:2:1$, where $l$,$w$,$h$ denote length, width and height respectively. We ignore the impact during block placement and focus on the resulting stability of the entire structure. Once the block makes contact with the existing structure, it is treated as releasing the block for a placement. In each episode, if the moving block collides with the environment boundary or existing structure, it will terminate the current episode. Further, if the block placement causes the resulting new structure to collapse, it will also end the episode. Stability is simulated similar to \cite{li2017visual} by comparing the change of displacement across all the blocks to a pre-selected small threshold within a fraction of time. If all of the blocks' displacements are below this threshold, the structure is deemed stable, otherwise unstable. To simplify the setting, we further constrain the action to be $\{\text{left},\text{right},\text{down}\}$.
The physics simulation runs at $60Hz$. However considering the cost of simulation we only use it when there is contact between the moving block and the boundary or the existing structure. Otherwise, the current block is moving without actual physics simulation. To further reduce the appearance difference caused by varying perspective, the environment is rendered using orthographic projection. Figure~\ref{fig:scene_sample} shows example images. The environment provides a user-friendly Python interface (similar to Gym\citep{brockman2016openai}) so that it can be used to test different reinforcement learning agents. At time of publication we will release our implementation of the environment.
\begin{figure}
\begin{center}$
\begin{tabular}{cccc}
\label{fig:hstack1}\includegraphics[width=0.2\linewidth]{./figs/hstack_1.png}&
\label{fig:hstack2}\includegraphics[width=0.2\linewidth]{./figs/hstack_2.png}&
\label{fig:hstack3}\includegraphics[width=0.2\linewidth]{./figs/hstack_3.png}&
\label{fig:hstack4}\includegraphics[width=0.2\linewidth]{./figs/hstack_4.png}\\
\end{tabular}$
\end{center}
\caption{Example scenes constructed by the learned agent.}
\label{fig:scene_sample}
\end{figure}
\section{Goal-Parameterized Deep Q Networks (GDQN)}
As one major characteristic of this task is that it requires goal-specific planning: given the same or similar states under different objectives, the optimal move can be different. To this end, we extend the typical reinforcement learning formulation to incorporate additional goal information.
\subsection{Learning Framework}
In a typical reinforcement learning setting, the agent interacts with the environment at time $t$, observes the state $s_t$, takes action $a_t$, receives reward $r_t$ and transits to a new state $s_{t+1}$. A common goal for a reinforcement learning agent is to maximize the cumulative reward. This is commonly formalized in form of a value function as the expected sum of rewards from a state $s$, $\mathbf{E}[\sum\limits_{i=0}^\infty \gamma^i r_{t+i+1}|s_t = s,\pi]$ when actions are taken with respect to a policy $\pi(a|s)$, with $0\leq\gamma\leq 1$ being the discount factor. The alternative formulation to this is the action-value function $Q^\pi(s,a)=\mathbf{E}[\sum\limits_{i=0}^\infty \gamma^i r_{t+i+1}|s_t=s,a_t=a]$.
Value-based reinforcement learning algorithms, such as Q-learning \citep{watkins1992q} directly search for optimal Q-value function. Recently by incorporating deep neural network as a function approximator for $Q$-function, the DQN \citep{mnih2015human} has shown impressive results across a variety of Atari games.
\paragraph{DQN}
For our task, we apply a {\em Deep Q Network\/} (DQN) which uses a deep neural network for approximating the action-value function $Q(s,a;\theta)$, mapping from an input state $s$ and action $a$ to Q values. In particular, two important improvements have been proposed by \cite{mnih2015human} for the learning process, including (1) experience replay, the agent stores observed transitions in a memory buffer for some time, and uniformly samples from the memory to update the network (2) the target network, agent maintains two networks for the loss function --- one for the current estimator of Q function and one for the surrogate of the true Q function. For the current estimator, the parameters are constantly updated. For the surrogate, the parameters are only updated for every certain number of steps from the current estimator network otherwise kept fixed.
\paragraph{Learning Goal-Parameterized Policies}
To plan with respect to the specific goal, we can parametrize the agent's policy $\pi$ by the goal $g$:
\begin{equation}
\pi(s,g,a)
\end{equation}
Since in this work, we applies DQN as value-based method, this corresponds to the update to original Q function with the additional goal information. The new Q-value function is hence defined as:
\begin{equation}
Q^\pi(s,g,a) = \mathbf{E}[\sum\limits_{i=0}^\infty \gamma^i r_{t+i+1}|s_t=s,g,a_t=a]
\end{equation}
As shown in Figure~\ref{fig:goal_net}, in contrast to the original DQN model, where state and action are used to estimate Q-value, the new model further include the current goal into the network to produce the estimate. We call this model as Goal-Parametrized Q Network (GDQN).
The resulted loss function is as:
\begin{equation}
L_Q = \mathbf{E}[(R + \gamma \text{max}_a' Q^\pi(s',g,a';\theta^-) - Q(s,g,a;\theta))^2 ]
\end{equation}
where $\theta^-$ are the previous parameters and the optimization is with respect to $\theta$.
\begin{figure}[h]
\centering
\includegraphics[width=0.69\textwidth]{./figs/qg-net_rev.pdf}
\caption{Our proposed model GDQN which extends the $Q$-function approximator to integrate goal information.}
\label{fig:goal_net}
\end{figure}
\subsection{Implementation Details}
The DQN agent is implemented in Theano and Keras to adapt to the settings in our experiment, while we use a $2$ hidden layer (each with $64$ hidden units and rectified linear activation) multilayer perceptron (MLP) for most cases, we additionally swap the MLP with the CNN and follow the reported parameter settings as in the original paper \citep{mnih2015human} to ensure our implementation can reach similar performance.
Note we don't apply the frame-skipping technique \citep{bellemare2012investigating} used for Atari games \citep{mnih2015human} allowing the agent sees and selects actions on every $k$th frame where its last action is repeated on skipped frames. It does not suit our task, in particular when the moving block is getting close to the existing structure, simply repeating action decided from previous frame can cause unintended collision or collapse.
\paragraph{Reward}
In the target stacking task, the agent gets reward $+1$ when the episode ends with complete reproduction of the target structure, otherwise $0$ reward.
Further, we explore reward shaping \citep{ng1999policy} in the task providing more prompt intermediate reward. Two types of reward shaping are included: overlap ratio and distance transform.
For the overlap ratio, for each state $s_t$ under the same target $g_i$, an overlap ratio is counted as the ratio between the intersected foreground region (of the current state and the target state) and the target foreground region (shown in Figure~\ref{fig:overlap_ratio}):
\begin{equation}\label{eq:ovl}
o(s_t,g_i) = \frac{s_t \cap g_i}{g_i}
\end{equation}
For each transition $(s_t, a_t, s_{t+1})$, the reward is defined by the change of overlap ratio before and after the action:
\begin{equation}
r_t =
\begin{cases}
1, & \text{if}\ \Delta o_{t\rightarrow t+1} = o(s_{t+1}) - o(s_t) > 0 \\
-1, & \text{if}\ \Delta o_{t\rightarrow t+1} = o(s_{t+1}) - o(s_t) < 0\\
0, & \text{otherwise}
\end{cases}
\end{equation}
The intuition is that actions increasing the current state to become more overlapped with the target scene should be encouraged.
For the distance transform \citep{fabbri20082d}, it generates a map $D$ whose value in each pixel $p$ is the smallest distance from it to a target object $O$:
\begin{equation}
D(p) = \text{min}\{\text{dist}(p,q)|q\in O\}
\end{equation}
where $\text{dist}$ can be any valid distance metric, like Euclidean or Manhattan distance.
For each state $s_t$ under the same target $g_i$, a distance to the goal is the sum of all the element-wise distance in $s_t$ to $g_i$ under $D_{g_i}$ (shown in Figure~\ref{fig:dist_transform}) as:
\begin{equation}
d(s_t,g_i) = \sum_j D_{g_i}(s_t^j), s_t^j \in s_t
\end{equation}
For each transition $(s_t, a_t, s_{t+1})$, the reward is defined as:
\begin{equation}
r_t =
\begin{cases}
1, & \text{if}\ \Delta d_{t\rightarrow t+1} = d(s_{t+1}) - d(s_t) < 0 \\
-1, & \text{if}\ \Delta d_{t\rightarrow t+1} = d(s_{t+1}) - d(s_t) > 0\\
0, & \text{otherwise}
\end{cases}
\end{equation}
The intuition behind this is that action decreasing the distance between the current state and the target scene should be encouraged.
\begin{figure}
\centering
\begin{subfigure}{0.47\linewidth}
\includegraphics[width=\textwidth]{./figs/ovl_ratio.pdf}
\caption{}
\label{fig:overlap_ratio}
\end{subfigure}
~
\begin{subfigure}{0.47\linewidth}
\includegraphics[width=\textwidth]{./figs/dist_transform.pdf}
\caption{}
\label{fig:dist_transform}
\end{subfigure}
\caption{Reward shaping used in target stacking. (\protect\subref{fig:overlap_ratio}): overlap ratio to the target. The gray area in the middle figure denotes the intersected foreground region between current and target scene, and the overlap ratio is the ratio between the areas of the two. (\protect\subref{fig:dist_transform}): distance under the distance transform of the target. The middle figure denotes the distance transform under the target shown in the left. The distance from current scene to the target is the sum of distances masked by the current scene in the distance transform.}
\label{fig:reward_shape}
\end{figure}
\section{Experiments}
We evaluate the proposed GDQN model on both a navigation task and target stacking and compare it to the base DQN model which does not integrate goal information. In addition, we include the result from GDQN model with different ways of reward shaping in the target stacking task.
\subsection{Toy Example with Goal Integration}
As a toy example, we introduce a type of navigation task in the classic gridworld environment. The locations for the starting point and goal are randomized for each episode. The agent needs to reach the goal with four possible actions $\{\text{left},\text{right},\text{up},\text{down}\}$. Action that will make the agent go off the grid will leave it stay in the same location. The episode only terminates once the agent reaches the goal. The agent only receive reward $+1$ when reaching the current goal. Two different sizes of gridworld are tested at $5\times5$ and $7\times7$.
The training epoch size is $1000$ in steps for the smaller gridworld and $3000$ for the larger one, the test sizes are the same for both at $100$. All the agents run for $100$ epochs and the $\epsilon$ for $\epsilon$-greedy anneals linearly from $1.0$ to $0.1$ over the first $20$ epochs, and fixed at $0.1$ thereafter. The memory buffer size is set the same to the annealing length, i.e. for the smaller gridworld, the buffer size equals to the length $20$ epochs in training with $20000$ steps whereas for the larger one, the buffer size is $30000$ steps.
We measure the proportion of episodes in the test epoch that reaches the goal in shortest distance as the success ratio. The results are shown in Table~\ref{tab:gridworld_result} for the best agents throughout the training process.
As in this simple task with relative small state space, DQN gets some performance due to running an average policy across all the the goals, but this is not addressing the task we set out to do. In contrast, GDQN parametrized specifically to include goal information achieves significant better results on both sizes of the environment.
\begin{table}
\centering
\begin{tabular}{c|cc}
\toprule
Grid Size & DQN & GDQN \\
\midrule
$5\times 5$ & 0.67 & 0.97 \\
$7\times 7$ & 0.67 & 0.95 \\
\bottomrule
\end{tabular}
\caption{Results from navigation task.}
\label{tab:gridworld_result}
\end{table}
\subsection{Target Stacking}
We set up $3$ groups of target structures consisted of different number of blocks $\{2,3,4\}$ in the scene as shown in Figure~\ref{fig:tgt_stack}. Within each group of target shapes, a random target (with the accompanied orientation order) is picked at the very beginning for individual episode. Each training epoch consists of $10000$ steps and each test epoch with $1000$ steps. Similar to the setting in the toy example, all the agents run for $100$ epochs and the $\epsilon$ anneals for the first $20$ epochs, and the memory buffer size is set as long as the annealing steps at $200K$ steps.
We computed both average overlap ratio (OR) and success rate (SR) for the finished stacking episodes in each test epoch. Here overlap ratio is the same as defined in the reward shaping in Equation~\ref{eq:ovl}, but simply measures the end scene over the assigned target scene. This tells the relative completion of the stacked structure in comparison to the assigned target structure, the higher the value is, the better completion it is to the target. At the maximum of $1$, it suggests completely reproduction of the target. The success rate counts the ratio how many episodes complete the exact same shape as assigned over the total number of episodes finished in the test epoch. This is the absolute metric counting overall successful stacking. The results are shown in Table~\ref{tab:tgt_stack} for the best agents throughout the training process.
Over all groups on both metrics, we observe GDQN outperforms DQN, showing the importance of integrating goal information. In general, the more blocks in the task, the more difficult it becomes. When there are only small number of blocks ($2$ blocks and $3$ blocks) in the scene, the single policy learned by DQN averages over the few target shapes can still work to some extent. However when introducing more blocks into the scene, it becomes more and more difficult for this averaged model to handle. As we can see from the result, there is already a significant decrease of performance (success rate drops from $0.70$ to $0.43$) when increasing the blocks number from $2$ to $3$, whereas GDQN's performance only decreases slightly from $0.72$ to $0.67$. In $4$ blocks scene, the DQN can no longer reproduce any single target ($0.0$ for success rate, $0.03$ for overlap ratio) while GDQN parametrized specifically to include goal information can still do. Though the success rate (absolute completion to the target) for the basic GDQN is relatively low at $0.17$ but the average overlap ratio (relative completion to the target) still holds up pretty well at $0.41$. Also we see reward shaping can further improves GDQN model, in particular distance transform can boost the performance more than overlap ratio.
\begin{figure}
\centering
\begin{subfigure}[b]{0.32\linewidth}
\includegraphics[width=\textwidth]{./figs/2blks_stack_targets.pdf}
\caption{}
\label{fig:2blks_tgt_stack}
\end{subfigure}
~
\begin{subfigure}[b]{0.32\linewidth}
\includegraphics[width=\textwidth]{./figs/3blks_stack_targets.pdf}
\caption{}
\label{fig:3blks_tgt_stack}
\end{subfigure}
~
\begin{subfigure}[b]{0.32\linewidth}
\includegraphics[width=\textwidth]{./figs/4blks_stack_targets.pdf}
\caption{}
\label{fig:4blks_tgt_stack}
\end{subfigure}
\caption{\protect\subref{fig:2blks_tgt_stack}: Targets for $2$ blocks.\protect\subref{fig:3blks_tgt_stack}: Targets for $3$ blocks. \protect\subref{fig:4blks_tgt_stack}: Targets for $4$ blocks.}
\label{fig:tgt_stack}
\end{figure}
\begin{table}
\centering
\begin{tabular}{@{}clccccccccccc@{}}
\toprule
\multicolumn{2}{c}{\multirow{2}{*}{Num. of Blks.}} & \multicolumn{2}{c}{DQN} & & \multicolumn{2}{c}{GDQN} & & \multicolumn{2}{c}{GDQN + OR} & & \multicolumn{2}{c}{GDQN + DT} \\ \cmidrule(lr){3-4} \cmidrule(lr){6-7} \cmidrule(lr){9-10} \cmidrule(l){12-13}
\multicolumn{2}{c}{} & OR & SR & & OR & SR & & OR & SR & & OR & SR \\ \midrule
\multicolumn{2}{c}{2} & 0.70 & 0.70 & & 0.82 & 0.72 & & 0.84 & 0.77 & & {\bf 0.88} & {\bf 0.78} \\
\multicolumn{2}{c}{3} & 0.43 & 0.43 & & 0.76 & {\bf 0.67} & & {\bf 0.86} & 0.63 & & 0.83 & 0.65 \\
\multicolumn{2}{c}{4} & 0.03 & 0.0 & & 0.41 & 0.17 & & 0.73 & 0.55 & & {\bf 0.79} & {\bf 0.56} \\ \bottomrule
\end{tabular}
\caption{Results for target stacking. For ``GDQN + X'', X denotes different ways for reward shaping as described in previous section, OR for overlap ratio, DT for distance transform. For metrics, OR stands for average overlap ratio, SR for average success rate.}
\label{tab:tgt_stack}
\end{table}
\section{Conclusion}
We create a synthetic block stacking environment with physics simulation in which the agent can learn block stacking end-to-end through trial and error, bypassing to explicitly model the corresponding physics knowledge.
We introduce a target stacking task where the agent stacks blocks to reproduces a tower shown in an image. The task presents a distinct type of challenge requiring the agent to reach a given goal state that may be different for every new trial.
Therefore we propose a goal-parametrized GDQN model to plan with respect to the specific goal, allowing better generalization across different goals.
We validate the model on both a navigation task in a classic gridworld environment with different start and goal positions and the block stacking task itself with different target structures. Our proposed model shows good performance on both tasks.
\begin{thebibliography}{28}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Agrawal et~al.(2016)Agrawal, Nair, Abbeel, Malik, and
Levine]{agrawal2016learning}
Pulkit Agrawal, Ashvin~V Nair, Pieter Abbeel, Jitendra Malik, and Sergey
Levine.
\newblock Learning to poke by poking: Experiential learning of intuitive
physics.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
5074--5082, 2016.
\bibitem[Baillargeon(2002)]{baillargeon2002acquisition}
Renee Baillargeon.
\newblock The acquisition of physical knowledge in infancy: A summary in eight
lessons.
\newblock \emph{Blackwell handbook of childhood cognitive development}, 2002.
\bibitem[Battaglia et~al.(2013)Battaglia, Hamrick, and
Tenenbaum]{battaglia2013simulation}
Peter~W Battaglia, Jessica~B Hamrick, and Joshua~B Tenenbaum.
\newblock Simulation as an engine of physical scene understanding.
\newblock \emph{Proceedings of the National Academy of Sciences}, 2013.
\bibitem[{Bellemare} et~al.(2013){Bellemare}, {Naddaf}, {Veness}, and
{Bowling}]{bellemare13arcade}
M.~G. {Bellemare}, Y.~{Naddaf}, J.~{Veness}, and M.~{Bowling}.
\newblock The arcade learning environment: An evaluation platform for general
agents.
\newblock \emph{Journal of Artificial Intelligence Research}, 47:\penalty0
253--279, jun 2013.
\bibitem[Bellemare et~al.(2012)Bellemare, Veness, and
Bowling]{bellemare2012investigating}
Marc~G Bellemare, Joel Veness, and Michael Bowling.
\newblock Investigating contingency awareness using atari 2600 games.
\newblock In \emph{AAAI}, 2012.
\bibitem[Bhattacharyya et~al.(2016)Bhattacharyya, Malinowski, Schiele, and
Fritz]{apratim16ariv}
Apratim Bhattacharyya, Mateusz Malinowski, Bernt Schiele, and Mario Fritz.
\newblock Long-term image boundary extrapolation.
\newblock arxiv:1611.08841 [cs.cv], 2016.
\bibitem[Brockman et~al.(2016)Brockman, Cheung, Pettersson, Schneider,
Schulman, Tang, and Zaremba]{brockman2016openai}
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman,
Jie Tang, and Wojciech Zaremba.
\newblock Openai gym.
\newblock \emph{arXiv preprint arXiv:1606.01540}, 2016.
\bibitem[Coumans(2010)]{coumans2010bullet}
Erwin Coumans.
\newblock Bullet physics engine.
\newblock \emph{Open Source Software: http://bulletphysics. org}, 1, 2010.
\bibitem[Denil et~al.(2016)Denil, Agrawal, Kulkarni, Erez, Battaglia, and
de~Freitas]{denil2016learning}
Misha Denil, Pulkit Agrawal, Tejas~D Kulkarni, Tom Erez, Peter Battaglia, and
Nando de~Freitas.
\newblock Learning to perform physics experiments via deep reinforcement
learning.
\newblock \emph{arXiv preprint arXiv:1611.01843}, 2016.
\bibitem[Fabbri et~al.(2008)Fabbri, Costa, Torelli, and Bruno]{fabbri20082d}
Ricardo Fabbri, Luciano Da~F Costa, Julio~C Torelli, and Odemir~M Bruno.
\newblock 2d euclidean distance transform algorithms: A comparative survey.
\newblock \emph{ACM Computing Surveys (CSUR)}, 40\penalty0 (1):\penalty0 2,
2008.
\bibitem[Fragkiadaki et~al.(2015)Fragkiadaki, Agrawal, Levine, and
Malik]{fragkiadaki2015learning}
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik.
\newblock Learning visual predictive models of physics for playing billiards.
\newblock \emph{arXiv preprint arXiv:1511.07404}, 2015.
\bibitem[Goslin \& Mine(2004)Goslin and Mine]{goslin2004panda3d}
Mike Goslin and Mark~R Mine.
\newblock The panda3d graphics engine.
\newblock \emph{Computer}, 37\penalty0 (10):\penalty0 112--114, 2004.
\bibitem[Kaelbling(1993)]{kaelbling1993learning}
Leslie~Pack Kaelbling.
\newblock Learning to achieve goals.
\newblock In \emph{IJCAI}, pp.\ 1094--1099, 1993.
\bibitem[Kempka et~al.(2016)Kempka, Wydmuch, Runc, Toczek, and
Ja{\'s}kowski]{kempka2016vizdoom}
Micha{\l} Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech
Ja{\'s}kowski.
\newblock Vizdoom: A doom-based ai research platform for visual reinforcement
learning.
\newblock In \emph{Computational Intelligence and Games (CIG), 2016 IEEE
Conference on}, pp.\ 1--8. IEEE, 2016.
\bibitem[Lerer et~al.(2016)Lerer, Gross, and Fergus]{fergus16blocsarxiv}
Adam Lerer, Sam Gross, and Rob Fergus.
\newblock Learning physical intuition of block towers by example.
\newblock \emph{arXiv preprint arXiv:1603.01312}, 2016.
\bibitem[Li et~al.(2016)Li, Azimi, Leonardis, and Fritz]{li2016fall}
Wenbin Li, Seyedmajid Azimi, Ale{\v{s}} Leonardis, and Mario Fritz.
\newblock To fall or not to fall: A visual approach to physical stability
prediction.
\newblock \emph{arXiv preprint arXiv:1604.00066}, 2016.
\bibitem[Li et~al.(2017)Li, Leonardis, and Fritz]{li2017visual}
Wenbin Li, Ale{\v{s}} Leonardis, and Mario Fritz.
\newblock Visual stability prediction for robotic manipulation.
\newblock In \emph{IEEE International Conference on Robotics and Automation}.
IEEE, 2017.
\bibitem[Mnih et~al.(2013)Mnih, Kavukcuoglu, Silver, Graves, Antonoglou,
Wierstra, and Riedmiller]{mnih2013playing}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis
Antonoglou, Daan Wierstra, and Martin Riedmiller.
\newblock Playing atari with deep reinforcement learning.
\newblock \emph{arXiv preprint arXiv:1312.5602}, 2013.
\bibitem[Mnih et~al.(2015)Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare,
Graves, Riedmiller, Fidjeland, Ostrovski, et~al.]{mnih2015human}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei~A Rusu, Joel Veness,
Marc~G Bellemare, Alex Graves, Martin Riedmiller, Andreas~K Fidjeland, Georg
Ostrovski, et~al.
\newblock Human-level control through deep reinforcement learning.
\newblock \emph{Nature}, 2015.
\bibitem[Mottaghi et~al.(2015)Mottaghi, Bagherinezhad, Rastegari, and
Farhadi]{mottaghi2015newtonian}
Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi.
\newblock Newtonian image understanding: Unfolding the dynamics of objects in
static images.
\newblock \emph{arXiv preprint arXiv:1511.04048}, 2015.
\bibitem[Mottaghi et~al.(2017)Mottaghi, Schenck, Fox, and
Farhadi]{mottaghi2017see}
Roozbeh Mottaghi, Connor Schenck, Dieter Fox, and Ali Farhadi.
\newblock See the glass half full: Reasoning about liquid containers, their
volume and content.
\newblock \emph{arXiv preprint arXiv:1701.02718}, 2017.
\bibitem[Ng et~al.(1999)Ng, Harada, and Russell]{ng1999policy}
Andrew~Y Ng, Daishi Harada, and Stuart Russell.
\newblock Policy invariance under reward transformations: Theory and
application to reward shaping.
\newblock In \emph{ICML}, volume~99, pp.\ 278--287, 1999.
\bibitem[Schaul et~al.(2015)Schaul, Horgan, Gregor, and
Silver]{schaul2015universal}
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver.
\newblock Universal value function approximators.
\newblock In \emph{Proceedings of the 32nd International Conference on Machine
Learning (ICML-15)}, pp.\ 1312--1320, 2015.
\bibitem[Sutton et~al.(2011)Sutton, Modayil, Delp, Degris, Pilarski, White, and
Precup]{sutton2011horde}
Richard~S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick~M
Pilarski, Adam White, and Doina Precup.
\newblock Horde: A scalable real-time architecture for learning knowledge from
unsupervised sensorimotor interaction.
\newblock In \emph{The 10th International Conference on Autonomous Agents and
Multiagent Systems-Volume 2}, pp.\ 761--768. International Foundation for
Autonomous Agents and Multiagent Systems, 2011.
\bibitem[Toussaint et~al.(2010)Toussaint, Plath, Lang, and
Jetchev]{toussaint2010integrated}
Marc Toussaint, Nils Plath, Tobias Lang, and Nikolay Jetchev.
\newblock Integrated motor control, planning, grasping and high-level reasoning
in a blocks world using probabilistic inference.
\newblock In \emph{Robotics and Automation (ICRA), 2010 IEEE International
Conference on}, pp.\ 385--391. IEEE, 2010.
\bibitem[Watkins \& Dayan(1992)Watkins and Dayan]{watkins1992q}
Christopher~JCH Watkins and Peter Dayan.
\newblock Q-learning.
\newblock \emph{Machine learning}, 8\penalty0 (3-4):\penalty0 279--292, 1992.
\bibitem[Wu et~al.(2015)Wu, Yildirim, Lim, Freeman, and
Tenenbaum]{wu2015galileo}
Jiajun Wu, Ilker Yildirim, Joseph~J Lim, Bill Freeman, and Josh Tenenbaum.
\newblock Galileo: Perceiving physical object properties by integrating a
physics engine with deep learning.
\newblock In \emph{NIPS}, 2015.
\bibitem[Yildirim et~al.(2017)Yildirim, Gerstenberg, Saeed, Toussaint, and
Tenenbaum]{yildirim2017physical}
Ilker Yildirim, Tobias Gerstenberg, Basil Saeed, Marc Toussaint, and Josh
Tenenbaum.
\newblock Physical problem solving: Joint planning with symbolic, geometric,
and dynamic constraints.
\newblock \emph{arXiv preprint arXiv:1707.08212}, 2017.
\end{thebibliography}