Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions
Extracted source text ยท arXiv 2405.16184
%# !TEX program = latexmk
%# !TEX options = -f -pdf -outdir="%OUTDIR%" -pdflatex="pdflatex -synctex=1 -interaction=nonstopmode" "%DOC%"
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2020
\PassOptionsToPackage{numbers}{natbib}
% ready for submission
% \usepackage{neurips_2020}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{cs285}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{cs285}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2020}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % micro-typography
% ##########################################
% ########## CUSTOM PACKAGES #############
% ##########################################
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{float}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{enumitem}
% ##########################################
% ########## CUSTOM PACKAGES #############
% ##########################################
\title{Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions}
\author{%
% {\ Huang Huang$^*$}\\
% UC Berkeley\\
% \texttt{hh19971229@berkeley.edu}\\
% \And
% {\ Bobby Yan$^*$}\\
% UC Berkeley\\
% \texttt{bobbyy@berkeley.edu}\\
% \And
{\ Harry Zhang}\\
Course Project for 16.332\\
}
\newcommand\mymathop[1]{\mathop{\operatorname{#1}}}
\begin{document}
\maketitle
\begingroup
\addtolength\leftmargini{-0.3in}
% \begin{abstract}
% learning control policies and neural network Lyapunov functions for safe deep model-based reinforcement learning pipelines,
\textbf{Summary: }Reinforcement learning (RL) for control systems with unknown nonlinear dynamics and safety constraints has been difficult due to unintended behavior from imperfect cost function and dynamic uncertainty, which could lead to instability breaking down the system. We propose a new RL framework to enable efficient policy learning with unknown dynamics based on the recently proposed learning model predictive control framework \cite{rosolia2017learning,rosolia2017robust,rosolia2019sample, pan2022tax, zhang2016health}, with provable guarantees of stability. The new stability-augmented framework consists of a neural-network-based learner that learns a Lyapunov function, and a model-based RL agent to consistently complete the tasks while satisfying user-specified constraints given only sub-optimal demonstrations and sparse-cost feedback.
Our work is based on Safety Augmented Value Estimation from Demonstrations (SAVED) \cite{thananjeyan2020safety, sim2019personalization, elmquist2022art, zhang2020dex, zhang2021robots}. SAVED is a sample-efficient method that can consistently complete tasks with complex safety constraints using only non-expert, sub-optimal demonstrations and sparse rewards from simple supervision identifying task completion. By adding safety constraints during the training of policy learning, SAVED ensures a high probability of being able to plan back to the safe region. However, it has no stability guarantees as the learned value function is an arbitrary neural network that is not constrained to be a Lyapunov function. To address the stability issues which could cause states to blow up and catastrophic consequences in practice, we use the learned Lyapunov network for the model-based RL agent as the value function, which will act in the environment in tandem with the safety-augmented agent in order to achieve both stability and safety.
To construct and learn a Lyapunov-constrained value function approximator, we train a Bayesian Neural Network with a loss function adapted from \citet{mittal2020neural, zhang2023flowbot++, eisner2022flowbot3d, avigal20206, avigal2021avplug}. The output from the network is defined as $V_{Net}(x)$, a $n_V \times n_x$ matrix, and the actual Lyapunov function used as the value function can be derived as $V(x) = x^T(l_lI+V_{net}(x)^TV_{net}(x))x$, where $x$ is the state and $l_l$ is a tunable parameter.
We train a 3-layer neural network with 500 hidden units per layers and swish activations, as in \cite{thananjeyan2020safety, pan2023tax, lim2021planar, lim2022real2sim2real} and evaluate the performance of the SAVED algorithm with and without Lyapunov-constrained value functions in four simulated environments and point-mass navigation tasks. The effectiveness of our method and its effects on the stability during training as well as the optimality of the learned policy are shown and summarized in \autoref{fig:iteration-cost}. From the results, we found that the Lyapunov value function significantly improved the controller performance by training a value function that was monotonically decreasing along the planning trajectories, which prevented the agent from being stuck in local minima around the initial states and resulted in more optimal trajectories with less variance across runs as well as higher task completion and constraint satisfaction rates.
Our contribution with this study is twofold: first, the implementation of learning a Lyapunov function as the value function of SAVED; second, the empirical validation of the theoretical performance improvement in terms of stability, task completion rates, and constraint satisfaction rates. Some future work of this study includes the physical experiments on a robot and the study of stronger stability guarantees such as asymptotic stability.
% \end{abstract}
\endgroup
\clearpage
\begin{abstract}
% learning control policies and neural network Lyapunov functions for safe deep model-based reinforcement learning pipelines,
Model-based Reinforcement Learning (MBRL) has shown many desirable properties for intelligent control tasks. However, satisfying safety and stability constraints during training and rollout remains an open question. We propose a new Model-based RL framework to enable efficient policy learning with unknown dynamics based on learning model predictive control (LMPC) framework with mathematically provable guarantees of stability. We introduce and explore a novel method for adding safety constraints for model-based RL during training and policy learning. The new stability-augmented framework consists of a neural-network-based learner that learns to construct a Lyapunov function, and a model-based RL agent to consistently complete the tasks while satisfying user-specified constraints given only sub-optimal demonstrations and sparse-cost feedback. We demonstrate the capability of the proposed framework through simulated experiments. Code and data are available \href{https://github.com/harryzhangOG/salved}{\textcolor{blue}{here}}.
\end{abstract}
\section{Introduction}
Modern control problems require a proper understanding of the system. However, the dynamic model of some tasks could be very complicated with thousands of parameters which makes it impossible to deploy system identification. Furthermore, some tasks require additional safety constraints such as the collision avoidance in navigation and manipulation, velocity and position limitation in human robot interaction, which makes the problem harder. In addition, a poorly designed controller could lead to unstable behavior resulting in catastrophic consequences.
Deep model-based reinforcement learning (MBRL) has the advantage of using a deep neural network to express the dynamic model with sample efficiency. However, the performance of deep model-based RL usually depends on the design of the cost function which could lead to unintended behavior \cite{thananjeyan2020safety, devgon2020orienting, yao2023apla, shen2024diffclip}. Prior work has been done to relax the requirement of the cost function design by using the task completion as the supervision with some sub-optimal demonstrations~\cite{thananjeyan2020abc,thananjeyan2020safety}. To ensure the agent staying in the safety region, safety constraints are added during the training the RL agent.
One concern of deep model-based RL is the stability of the closed-loop system with the learned policy. Since the dynamic model is not perfectly modeled and the outputs from the learned policy are not explicitly constrained, there is no stability guarantee. A common verification of stability of the dynamic system is the existence of the Lyapunov function, which implies the convergence to the equilibrium point of the resulting trajectory. A Lyapunov function for a system with unknown dynamics can be estimated using a neural network \cite{berkenkamp2017safe}. \cite{bobiti2016samplingbased,bobiti2017,gallieri2019safe} have shown that the learned Lyapunov neural network can provide stability guarantees and improve the performance.
In this study, we focus on policy learning for unknown systems with safety constraints and stability guarantees. We use a learned Lyapunov function as the value function for Safety Augmented Value Estimation from Demonstrations (SAVED)~\cite{thananjeyan2020safety}, a learning method which is able to complete the tasks efficiently with safety constraints using simple task completion supervision and sub-optimal demonstrations. The Lyapunov function is learned using a neural network with a loss function proposed in \cite{mittal2020neural}. Our experiments using the Safety Augmented Lyapunov Value Estimation from Demonstrations (SALVED) show the performance improvement brought by the Lyapunov value function, which helps to increase the stability of the MPC controller and lower the probability of the controller getting stuck in sub-optimal trajectories by forcing the trained value function to be monotonically decreasing along the closed-loop trajectories and reducing the number of local minima in the value function. The results from SALVED also show higher task completion and constraint satisfaction rates and more optimal trajectories with less variance.
The contributions of this study consist of two parts: (1) the implementation of the Lyapunov value function in SAVED and (2) the experiments validating the benefits and stability of the new framework SALVED.
\section{Related Work}
Significant interest in model-based planning and
data-driven model-based RL \cite{chua2018deep, deisenroth2011pilco, fu2016one, lowrey2018plan, nagabandi2018neural} has been sparked in research due to the improvements in sample efficiency when planning over learned dynamics compared to model-free methods for continuous control \cite{fujimoto2018addressing, haarnoja2018soft}. Moreover, model predictive control (MPC) with learned model through data-driven approach has been shown to be promising as it has been applied in various tasks such as deformable objects manipulation \cite{yan2020learning}, and car racing \cite{rosolia2017learning}.
The safety of a control system implies the ability of the system to stay in the safe region and converge to the region of attraction. In reinforcement learning, the safety of the closed loop system with agent is a challenging problem due to unknown system dynamics. Safe RL has risen broad interests in the past few decades \cite{garcia2015comprehensive}. \cite{thananjeyan2020abc,thananjeyan2020safety} focuses on the problem of ensuring the states lie in safety set with high probabilities for system with unknown dynamics. \cite{thananjeyan2020abc,thananjeyan2020safety} solve the problem using sparse cost and probabilistic constraints during exploration. \cite{achiam2017constrained,li2018safe} study the problem of exploration with explicit constraints satisfied.
The study on the stability of closed loop system with RL agents mostly adapts the idea of using Lyapunov stability criterion. To our knowledge, \cite{perkins2002Lyapunov} first used Lyapunov function in reinforcement learning to guarantee the closed loop system stability. The construction of the Lyapunov function for an unknown system is challenging. \cite{berkenkamp2017safe} used a neural network to estimate the Lyapunov function. \cite{chow2018Lyapunov} proposed a Lyapunov based approach for constrained Markov decision process (CMDP) by constructing the Lyapunov function using an LP-based algorithm. \cite{mittal2020neural} proposed a method using a learned Lyapunov Neural Network as the value function estimation to be the terminal cost of the Model predictive control (MPC) and showed an increased stable horizon compared to the baseline MPC. \cite{gallieri2019safe} learned the safety set from the Lyapunov function. The Lyapunov network proposed in \cite{gallieri2019safe} is used in \cite{mittal2020neural}. In this paper, we use the same Lyapunov network from \cite{gallieri2019safe} to learn a Lyapunov function as the value function.
\section{Preliminaries}
In this section, we introduce some key concepts that this paper builds upon.
\subsection{Control Lyapunov Functions}
A Lyapunov function $V(x)$ is a positive scalar function that takes as input a state and outputs a positive scalar value:
\[
V(x):\; \mathbb{R}^n_x\to\mathbb{R}^+
\],
and it should satisfy the following constraint that it monotonically decreases along the closed-loop trajectory of a controlled system until a target point (or set) is reached. The existence of such a function is a necessary and sufficient condition for stability and convergence of a dynamical system~\cite{khalil2014nonlinear}. Control Lyapunov functions can be used to design control policies.
A control Lyapunov function should be upper and lower bounded by strictly increasing, unbounded,
positive functions $\mathcal{K}_\infty$~\cite{khalil2014nonlinear}. Consider the optimal control problem with cost-to-go function defined as:
\[
c(x, u) = x(t)^TQx(t) + u(t)^TRu(t)
\],
where $x$ is the state and $u$ is the input. The $\mathcal{K}_\infty$ can be designed as the scaled, squared $l_2$ norm of the state:
\[
l_l||x(t)||_2^2\leq V(x(t))\leq L_V||x(t)||_2^2
\]
where $l_l$ is the minimum eigenvalue of $Q$ and $L_V$ is a local Lipschitz constant for $V$.
Furthermore, we want the Lyapunov function values to decrease monotonically along the closed-loop system's trajectory. The amount of the decrease is usually chosen to be quantified by the stage cost (cost-to-go) value incurred at the current time step. Mathematically, this condition can be written as:
\[
V (x(t + 1)) - V (x(t)) \leq -c(x(t), u(t)).
\]
\subsection{Model Predictive Control}
Model predictive control (MPC) is a control strategy that iteratively plans ahead and solves for a sequence of actions to take that incurs the minimal cost-to-go values in the plan horizon. The agent will only take the first action in the planned sequence in order to minimize the deviation caused by model inaccuracy. The iterative optimization problem that MPC tries to solve is described as follows:
\begin{align*}
J^*(x(t)) &= \min_{u_t\dots u_{t+N-1}} \sum_{k=0}^{N-1}c(x_{t+k}, u_{t+k})\\
\text{s.t. }x_{t+k+1} &= f(x_{t+k}, u_{t+k})\\
\end{align*},
where $N$ is the planning horizon of the MPC and we only take the first element in the planned the sequence of inputs $u_t\dots u_{t+N-1}$. After taking this step, we result in the next state and re-plan according to the above optimization problem.
\section{Methods}
The main novelty of our work is that we are trying to design a Lyapunov-constrained value function which the model-based RL agent can leverage in order to achieve stable performance. Specifically, we plan to improve the SAVED algorithm, which can perform poorly if the learned value function contains many local minima as shown in \autoref{fig:pb1-local-minima}, using a Lyapunov neural network that can guide the agent to correctly plan the action to take at each time step.
\begin{figure}
\begin{subfigure}{.24\textwidth}
\includegraphics[width=\linewidth]{assets/pb1-mpc-traj/time_001.png}
\end{subfigure}\hfil
\begin{subfigure}{.24\textwidth}
\includegraphics[width=\linewidth]{assets/pb1-mpc-traj/time_010.png}
\end{subfigure}\hfil
\begin{subfigure}{.24\textwidth}
\includegraphics[width=\linewidth]{assets/pb1-mpc-traj/time_050.png}
\end{subfigure}\hfil
\begin{subfigure}{.24\textwidth}
\includegraphics[width=\linewidth]{assets/pb1-mpc-traj/time_090.png}
\end{subfigure}
\caption{Planning trajectories of the MPC policy in SAVED in Pointbot 1 (point mass navigation) environment, where the value of the heat map is the predicted cost-to-go (negative returns) according to the value function at each iteration. In the plots, the $x$-axis is [-130, -80] and the $y$-axis is [-10, 10]. The starting position $x_0$ is $(-100, 0)$ and the goal is at the origin, $(0, 0)$. We see that even in an environment with no obstacles or constraints, SAVED struggles to complete the task due to local minima.}
\label{fig:pb1-local-minima}
\end{figure}
\subsection{SAVED Algorithm}
At each time step, SAVED solves the optimization problem in \autoref{eq:saved}, where $C(x_{t+i},u_{t+i})$ is the sparse cost function indicating task completion, $V_\phi^\pi(x_{i+H})$ is the value function and $f_\theta$ is the learned dynamic model. $\rho_\alpha$ is the safety density model constraining the exploration by requiring $x_{t+H}$ to be in the region with high task completion probability. And $\beta$ is the probability of $x_{t:t+H}$ falling in the feasible region $\mathcal{X}^{H+1}$, where $\mathcal{X}^{H+1}$ is the set of $H+1$ length state sequences \cite{berkenkamp2017safe}.
\begin{equation}
\begin{aligned}
& u^*_{t:t+H-1} =\displaystyle\mymathop {argmin}_{u_{t:t+H-1}\in\mathcal{U}^H}\mathbb{E}_{x_{t:t+H}}[\sum_{i=0}^{H-1}C(x_{t+i},u_{t+i})+V_\phi^\pi(x_{i+H})],\\
&\text{s.t.}\quad x_{t+i+1}\sim f_\theta(x_{t+i,u+i}) \forall i \in \{0,\dots,H-1\},\\
&\quad\quad\rho_\alpha(x_{t+H})>\delta,\mathbb{P}(x_{t:t+H}\in \mathcal{X}^{H+1})\geq \beta
\label{eq:saved}
\end{aligned}
\end{equation}
\subsection{Lyapunov Value Function}
We plan to constrain the value function used by the model-based RL to be a control Lyapunov function. To achieve this, we use the function introduced in \cite{gallieri2019safe}:
\begin{equation}
s_v V (x) = s_v \cdot x^T\left(l_lI + V_\text{net}(x)^TV_\text{net}\right)x,
\label{eq:lyv}
\end{equation}
where $V_\text{net}$ is a neural network that
produces a $n_V \times n_x$ matrix. Here $n_V$, $s_v$, and $l_l>0$ are hyper-parameters. In the original introduction of the value function in \cite{gallieri2019safe}, $s_v$ is fixed to be 1, but in our experiments, we have found that with $s_v = 1$, $V(x)$ tends to be numerically unstable, causing the loss to blow up, so we scale down the value function output significantly (e.g. $s_v = 0.001$).
We use the same loss function as described in \cite{mittal2020neural}, which enforces the value function to be control Lyapunov. The objective of the agent is to minimize the following loss in order to train the Lyapunov neural network:
\[
\min_{V_\text{net}, l_s} \mathbb{E}_{(x, u, x')}\left[J(x, u, x')\right]
\]
where $J(x, u, x')$ is defined as
\[
J(x, u, x') =\frac{\mathcal{I}(x)}{\rho}J_s(x, u, x') + J_\text{vol}(x, u, x')
\]
and
\begin{align*}
\mathcal{I}(x) &= 0.5(\text{sign}[l_s-V(x)] + 1)\\
J_s(x, u, x') &= \frac{\text{ReLU}[\Delta V(x)]}{V(x) +\epsilon}\\
J_\text{vol}(x, u, x') &=\text{sign}[\Delta V(x)](l_s - V(x))\\
\Delta V(x) &= V(x') - \lambda V(x) + v\mathcal{I}(x)
\end{align*}.
In the definitions above, $\mathcal{I}(x)$ is an indicator function representing the step cost that the agent incurs at a given state $x$. $J_\text{vol}$ is a classification loss that computes the correct boundary between the stable and unstable points, which also helps to increase the safe set volume \cite{mittal2020neural}. The scalars $\epsilon > 0$, $\lambda\in[0, 1)$, $v\in[0, 1]$, and $\rho > 0$ are hyper-parameters. $l_s$ is a trainable variable, and the role of $l_s$ is to provide an upper bound for $V(s)$ such that the states can remain in an invariant set.
\subsection{Safety-Augmented Lyapunov Value Estimation from Demonstration (SALVED)}
By substituting the Lyapunov value function $V$ calculated from \autoref{eq:lyv} in \autoref{eq:saved}, we have SALVED to be SAVED with a Lyapunov value function, which provides an estimate for the terminal cost for the current policy at the end of the planning horizon:
\begin{equation}
u^*_{t:t+H-1} =\displaystyle\mymathop {argmin}_{u_{t:t+H-1}\in\mathcal{U}^H}\mathbb{E}_{x_{t:t+H}}\left[\sum_{i=0}^{H-1}C(x_{t+i},u_{t+i})+s_v V(x_{i+H})\right],\\
\end{equation}
According to Lemma 1 from \cite{mittal2020neural}, for a given horizon length $N$ and contraction factor $\lambda$, there is minimum scaling of Lyapunov function $V$, which is calculated from \autoref{eq:lyv}, and a lower bound on the discount factor $s_v$ such that the system under MPC is stable.
When the learned dynamic model $f_\theta$ is perfect, the state would converge to the equilibrium point. In practice, however, the learned dynamics model may be imperfect for various reasons, such as due to aleatoric uncertainty in the actual dynamics and the epistemic uncertainty as a result of insufficient information about the environment, but the system can still stable when error between the learned dynamic model and the actual model is bounded, in which case the state would converge to a bounded area around the equilibrium point, whose size depends on the model error, $N$ and $s_v$ \cite{mittal2020neural}.
\section{Experiments}
\subsection{Experimental Setup}
For all of our SALVED experiments, we used Safety-Augmented Value Estimation from Demonstrations (SAVED) as our baseline. In order to test whether the new Lyapunov-based value functions can provide additional stability to model-based RL methods that are based on the learning MPC framework, such as SAVED, we benchmarked our proposed method, SALVED, and compared its performance with that of the baseline on four 4-dimensional simulated navigation tasks in which a point-mass navigates from a random starting position to a goal set, which is defined as a unit ball centered around the origin:
\begin{itemize}[leftmargin=*, itemsep=0pt, topsep=0pt]
\item \textbf{Pointbot 1}: long navigation task with no obstacles
\item \textbf{Pointbot 2}: medium-length navigation task with a single large obstacle along the $y$-axis
\item \textbf{Pointbot 3}: medium-length navigation task with two medium-sized obstacles and an open channel along the $x$-axis
\item \textbf{Pointbot 4}: long navigation task with obstacles of varying sizes surrounding the goal and a small opening for entry
\end{itemize}
\subsection{Implementation Details}
The dynamics models for the baseline and SALVE have the same network architecture, which is a probabilistic ensemble of five 3-layer neural networks with 500 hidden units per layer and swish activations. For SAVED, the value function has the same network architecture as its dynamics model, whereas for SALVED, the value function is a neural network with the same number of layers, same number of hidden units per layer, and the same activations but constrained to be a Lyapunov function.
\subsection{Results}
\begin{figure}[H]
\centering
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb1-iteration-cost.png}
\caption{Pointbot 1}
\label{fig:pb1-iter-cost}
\end{subfigure}\hfil
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb2-iteration-cost.png}
\caption{Pointbot 2}
\label{fig:pb2-iter-cost}
\end{subfigure}
% \end{figure}
% \begin{figure}
\centering
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb3-iteration-cost.png}
\caption{Pointbot 3}
\label{fig:pb3-iter-cost}
\end{subfigure}\hfil
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb4-iteration-cost.png}
\caption{Pointbot 4}
\label{fig:pb4-iter-cost}
\end{subfigure}
\caption{\textbf{Pointbot Navigation Tasks}: SALVED is evaluated on four Pointbot navigation tasks. Pointbot 1 has no obstacles or safety constraints, Pointbot 2-4 contain obstacles with increasingly more complex constraints. We observe that SALVED has a lower iteration cost than baselines throughout all four tasks and shows considerably more stability during training and evaluation.}
\label{fig:iteration-cost}
\end{figure}
\begin{figure}[H]
\centering
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb1-saved-traj.png}
\caption{Pointbot 1 (SAVED)}
\label{fig:pb1-saved-traj}
\end{subfigure}\hfil
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb1-salved-traj.png}
\caption{Pointbot 1 (SALVED)}
\label{fig:pb1-salved-traj}
\end{subfigure}
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb2-saved-traj.png}
\caption{Pointbot 2 (SAVED)}
\label{fig:pb2-saved-traj}
\end{subfigure}\hfil
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb2-salved-traj.png}
\caption{Pointbot 2 (SALVED)}
\label{fig:pb2-salved-traj}
\end{subfigure}
\end{figure}
\begin{figure}\ContinuedFloat
\centering
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb3-saved-traj.png}
\caption{Pointbot 3 (SAVED)}
\label{fig:pb3-saved-traj}
\end{subfigure}\hfil
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb3-salved-traj.png}
\caption{Pointbot 3 (SALVED)}
\label{fig:pb3-salved-traj}
\end{subfigure}
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb4-saved-traj.png}
\caption{Pointbot 4 (SAVED)}
\label{fig:pb4-saved-traj}
\end{subfigure}\hfil
\begin{subfigure}{0.5\textwidth}
\includegraphics[width=\linewidth]{assets/pb4-salved-traj.png}
\caption{Pointbot 4 (SALVED)}
\label{fig:pb4-salved-traj}
\end{subfigure}
\caption{The plots show evaluation trajectories combined across 100 iterations. The left plots show that the control policies learned by SAVED are considerably less optimal and have lower task completion rates across the experiments. The right plots demonstrate higher task completion rates and suggest that the policies learned by SALVED are significantly more optimal and consistent as the resulting trajectories are also more dense.}
\label{fig:pointbot-eval-traj}
\end{figure}
\section{Conclusion}
% In this paper, we proposed the method of using a Lyapunov value function for a model based reinforcement learning algorithm to provide stability guarantees. We tested our method by implementing a learned Lyapunov value function \cite{mittal2020neural} to SAVED \cite{gallieri2019safe} as SALVED. The results for four different environments show significant performance improvement of SALVED compared to SAVED. The resulting trajectories are more dense and do not suffer from the problem of sticking near the initial states due to the additional Lyapunov stability constraints.
In this paper, we present SALVED, a model-based reinforcement learning algorithm with provable stability guarantees by learning Lyapunov neural network value functions. We empirically evaluated SALVED on four simulated benchmarks and found that SALVED provided significant performance improvements over the baseline in all four environments. SALVED is able to maintain the sample efficiency of model-based RL algorithms such as SAVED and the iterative improvement guarantees of the Learning MPC framework, while providing additional Lyapunov stability guarantees. Results not only show higher task completion and constraint satisfaction rates, but also better-performing learned control policies with more optimal trajectories with less variance across runs and controllers that are less prone to local minima and catastrophic failures from constraint violations due to the stability guarantees from the Lyapunov-constrained value function approximation.
% \bibliographystyle{plainnat}
\bibliographystyle{IEEEtranN}
\bibliography{reference}
\end{document}
% Generated by IEEEtranN.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{40}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtranN.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem[Rosolia and Borrelli(2017)]{rosolia2017learning}
U.~Rosolia and F.~Borrelli, ``Learning model predictive control for iterative tasks. a data-driven control framework,'' \emph{IEEE Transactions on Automatic Control}, vol.~63, no.~7, pp. 1883--1896, 2017.
\bibitem[Rosolia et~al.(2017)Rosolia, Zhang, and Borrelli]{rosolia2017robust}
U.~Rosolia, X.~Zhang, and F.~Borrelli, ``Robust learning model predictive control for iterative tasks: Learning from experience,'' in \emph{2017 IEEE 56th Annual Conference on Decision and Control (CDC)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2017, pp. 1157--1162.
\bibitem[Rosolia and Borrelli(2019)]{rosolia2019sample}
U.~Rosolia and F.~Borrelli, ``Sample-based learning model predictive control for linear uncertain systems,'' in \emph{2019 IEEE 58th Conference on Decision and Control (CDC)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019, pp. 2702--2707.
\bibitem[Pan et~al.(2022)Pan, Okorn, Zhang, Eisner, and Held]{pan2022tax}
C.~Pan, B.~Okorn, H.~Zhang, B.~Eisner, and D.~Held, ``Tax-pose: Task-specific cross-pose estimation for robot manipulation,'' \emph{arXiv preprint arXiv:2211.09325}, 2022.
\bibitem[Zhang(2016)]{zhang2016health}
H.~Zhang, ``Health diagnosis based on analysis of data captured by wearable technology devices,'' \emph{International Journal of Advanced Science and Technology}, vol.~95, pp. 89--96, 2016.
\bibitem[Thananjeyan et~al.(2020{\natexlab{a}})Thananjeyan, Balakrishna, Rosolia, Li, McAllister, Gonzalez, Levine, Borrelli, and Goldberg]{thananjeyan2020safety}
B.~Thananjeyan, A.~Balakrishna, U.~Rosolia, F.~Li, R.~McAllister, J.~E. Gonzalez, S.~Levine, F.~Borrelli, and K.~Goldberg, ``Safety augmented value estimation from demonstrations ({SAVED}): Safe deep model-based {RL} for sparse cost robotic tasks,'' \emph{IEEE Robotics and Automation Letters}, vol.~5, no.~2, pp. 3612--3619, 2020.
\bibitem[Sim et~al.(2019)Sim, Beaufays, Benard, Guliani, Kabel, Khare, Lucassen, Zadrazil, Zhang, Johnson, et~al.]{sim2019personalization}
K.~C. Sim, F.~Beaufays, A.~Benard, D.~Guliani, A.~Kabel, N.~Khare, T.~Lucassen, P.~Zadrazil, H.~Zhang, L.~Johnson \emph{et~al.}, ``Personalization of end-to-end speech recognition on mobile devices for named entities,'' in \emph{2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019, pp. 23--30.
\bibitem[Elmquist et~al.(2022)Elmquist, Young, Hansen, Ashokkumar, Caldararu, Dashora, Mahajan, Zhang, Fang, Shen, et~al.]{elmquist2022art}
A.~Elmquist, A.~Young, T.~Hansen, S.~Ashokkumar, S.~Caldararu, A.~Dashora, I.~Mahajan, H.~Zhang, L.~Fang, H.~Shen \emph{et~al.}, ``Art/atk: A research platform for assessing and mitigating the sim-to-real gap in robotics and autonomous vehicle engineering,'' \emph{arXiv preprint arXiv:2211.04886}, 2022.
\bibitem[Zhang et~al.(2020)Zhang, Ichnowski, Avigal, Gonzales, Stoica, and Goldberg]{zhang2020dex}
H.~Zhang, J.~Ichnowski, Y.~Avigal, J.~Gonzales, I.~Stoica, and K.~Goldberg, ``Dex-net ar: Distributed deep grasp planning using a commodity cellphone and augmented reality app,'' in \emph{2020 IEEE International Conference on Robotics and Automation (ICRA)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 552--558.
\bibitem[Zhang et~al.(2021)Zhang, Ichnowski, Seita, Wang, Huang, and Goldberg]{zhang2021robots}
H.~Zhang, J.~Ichnowski, D.~Seita, J.~Wang, H.~Huang, and K.~Goldberg, ``Robots of the lost arc: Self-supervised learning to dynamically manipulate fixed-endpoint cables,'' in \emph{2021 IEEE International Conference on Robotics and Automation (ICRA)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 4560--4567.
\bibitem[Mittal et~al.(2020)Mittal, Gallieri, Quaglino, Salehian, and Koutn{\'\i}k]{mittal2020neural}
M.~Mittal, M.~Gallieri, A.~Quaglino, S.~S.~M. Salehian, and J.~Koutn{\'\i}k, ``Neural lyapunov model predictive control,'' \emph{arXiv preprint arXiv:2002.10451}, 2020.
\bibitem[Zhang et~al.(2023)Zhang, Eisner, and Held]{zhang2023flowbot++}
H.~Zhang, B.~Eisner, and D.~Held, ``Flowbot++: Learning generalized articulated objects manipulation via articulation projection,'' \emph{arXiv preprint arXiv:2306.12893}, 2023.
\bibitem[Eisner et~al.(2022)Eisner, Zhang, and Held]{eisner2022flowbot3d}
B.~Eisner, H.~Zhang, and D.~Held, ``Flowbot3d: Learning 3d articulation flow to manipulate articulated objects,'' \emph{arXiv preprint arXiv:2205.04382}, 2022.
\bibitem[Avigal et~al.(2020)Avigal, Paradis, and Zhang]{avigal20206}
Y.~Avigal, S.~Paradis, and H.~Zhang, ``6-dof grasp planning using fast 3d reconstruction and grasp quality cnn,'' \emph{arXiv preprint arXiv:2009.08618}, 2020.
\bibitem[Avigal et~al.(2021)Avigal, Satish, Tam, Huang, Zhang, Danielczuk, Ichnowski, and Goldberg]{avigal2021avplug}
Y.~Avigal, V.~Satish, Z.~Tam, H.~Huang, H.~Zhang, M.~Danielczuk, J.~Ichnowski, and K.~Goldberg, ``Avplug: Approach vector planning for unicontact grasping amid clutter,'' in \emph{2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 1140--1147.
\bibitem[Pan et~al.(2023)Pan, Okorn, Zhang, Eisner, and Held]{pan2023tax}
C.~Pan, B.~Okorn, H.~Zhang, B.~Eisner, and D.~Held, ``Tax-pose: Task-specific cross-pose estimation for robot manipulation,'' in \emph{Conference on Robot Learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2023, pp. 1783--1792.
\bibitem[Lim et~al.(2021)Lim, Huang, Chen, Wang, Ichnowski, Seita, Laskey, and Goldberg]{lim2021planar}
V.~Lim, H.~Huang, L.~Y. Chen, J.~Wang, J.~Ichnowski, D.~Seita, M.~Laskey, and K.~Goldberg, ``Planar robot casting with real2sim2real self-supervised learning,'' \emph{arXiv preprint arXiv:2111.04814}, 2021.
\bibitem[Lim et~al.(2022)Lim, Huang, Chen, Wang, Ichnowski, Seita, Laskey, and Goldberg]{lim2022real2sim2real}
------, ``Real2sim2real: Self-supervised learning of physical single-step dynamic actions for planar robot casting,'' in \emph{2022 International Conference on Robotics and Automation (ICRA)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2022, pp. 8282--8289.
\bibitem[Devgon et~al.(2020)Devgon, Ichnowski, Balakrishna, Zhang, and Goldberg]{devgon2020orienting}
S.~Devgon, J.~Ichnowski, A.~Balakrishna, H.~Zhang, and K.~Goldberg, ``Orienting novel 3d objects using self-supervised learning of rotation transforms,'' in \emph{2020 IEEE 16th International Conference on Automation Science and Engineering (CASE)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 1453--1460.
\bibitem[Yao et~al.(2023)Yao, Deng, Cao, Zhang, and Deng]{yao2023apla}
Y.~Yao, S.~Deng, Z.~Cao, H.~Zhang, and L.-J. Deng, ``Apla: Additional perturbation for latent noise with adversarial training enables consistency,'' \emph{arXiv preprint arXiv:2308.12605}, 2023.
\bibitem[Shen et~al.(2024)Shen, Zhu, Fan, Zhang, and Wu]{shen2024diffclip}
S.~Shen, Z.~Zhu, L.~Fan, H.~Zhang, and X.~Wu, ``Diffclip: Leveraging stable diffusion for language grounded 3d classification,'' in \emph{Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}, 2024, pp. 3596--3605.
\bibitem[Thananjeyan et~al.(2020{\natexlab{b}})Thananjeyan, Balakrishna, Rosolia, Gonzalez, Ames, and Goldberg]{thananjeyan2020abc}
B.~Thananjeyan, A.~Balakrishna, U.~Rosolia, J.~E. Gonzalez, A.~Ames, and K.~Goldberg, ``{ABC-LMPC:} safe sample-based learning {MPC} for stochastic nonlinear dynamical systems with adjustable boundary conditions,'' \emph{arXiv preprint arXiv:2003.01410}, 2020.
\bibitem[Berkenkamp et~al.(2017)Berkenkamp, Turchetta, Schoellig, and Krause]{berkenkamp2017safe}
F.~Berkenkamp, M.~Turchetta, A.~Schoellig, and A.~Krause, ``Safe model-based reinforcement learning with stability guarantees,'' in \emph{Advances in neural information processing systems}, 2017, pp. 908--918.
\bibitem[Bobiti and Lazar(2016)]{bobiti2016samplingbased}
R.~Bobiti and M.~Lazar, ``Sampling-based verification of lyapunov's inequality for piecewise continuous nonlinear systems,'' \emph{arXiv preprint arXiv:1609.00302}, 2016.
\bibitem[Bobiti(2017)]{bobiti2017}
R.~Bobiti, ``\BIBforeignlanguage{English}{Sampling--driven stability domains computation and predictive control of constrained nonlinear systems},'' Ph.D. dissertation, Department of Electrical Engineering, Oct. 2017, proefschrift.
\bibitem[Gallieri et~al.(2019)Gallieri, Salehian, Toklu, Quaglino, Masci, Koutn{\'\i}k, and Gomez]{gallieri2019safe}
M.~Gallieri, S.~S.~M. Salehian, N.~E. Toklu, A.~Quaglino, J.~Masci, J.~Koutn{\'\i}k, and F.~Gomez, ``Safe interactive model-based learning,'' \emph{arXiv preprint arXiv:1911.06556}, 2019.
\bibitem[Chua et~al.(2018)Chua, Calandra, McAllister, and Levine]{chua2018deep}
K.~Chua, R.~Calandra, R.~McAllister, and S.~Levine, ``Deep reinforcement learning in a handful of trials using probabilistic dynamics models,'' in \emph{Advances in Neural Information Processing Systems}, 2018, pp. 4754--4765.
\bibitem[Deisenroth and Rasmussen(2011)]{deisenroth2011pilco}
M.~Deisenroth and C.~E. Rasmussen, ``{PILCO:} {A} model-based and data-efficient approach to policy search,'' in \emph{Proceedings of the 28th International Conference on Machine Learning, {ICML}}, 2011, pp. 465--472.
\bibitem[Fu et~al.(2016)Fu, Levine, and Abbeel]{fu2016one}
J.~Fu, S.~Levine, and P.~Abbeel, ``One-shot learning of manipulation skills with online dynamics adaptation and neural network priors,'' in \emph{2016 {IEEE/RSJ} International Conference on Intelligent Robots and Systems, {IROS}}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2016, pp. 4019--4026.
\bibitem[Lowrey et~al.(2018)Lowrey, Rajeswaran, Kakade, Todorov, and Mordatch]{lowrey2018plan}
K.~Lowrey, A.~Rajeswaran, S.~Kakade, E.~Todorov, and I.~Mordatch, ``Plan online, learn offline: Efficient learning and exploration via model-based control,'' \emph{arXiv preprint arXiv:1811.01848}, 2018.
\bibitem[Nagabandi et~al.(2018)Nagabandi, Kahn, Fearing, and Levine]{nagabandi2018neural}
A.~Nagabandi, G.~Kahn, R.~S. Fearing, and S.~Levine, ``Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning,'' in \emph{2018 IEEE International Conference on Robotics and Automation (ICRA)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2018, pp. 7559--7566.
\bibitem[Fujimoto et~al.(2018)Fujimoto, Van~Hoof, and Meger]{fujimoto2018addressing}
S.~Fujimoto, H.~Van~Hoof, and D.~Meger, ``Addressing function approximation error in actor-critic methods,'' \emph{arXiv preprint arXiv:1802.09477}, 2018.
\bibitem[Haarnoja et~al.(2018)Haarnoja, Zhou, Abbeel, and Levine]{haarnoja2018soft}
T.~Haarnoja, A.~Zhou, P.~Abbeel, and S.~Levine, ``Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,'' \emph{arXiv preprint arXiv:1801.01290}, 2018.
\bibitem[Yan et~al.(2020)Yan, Vangipuram, Abbeel, and Pinto]{yan2020learning}
W.~Yan, A.~Vangipuram, P.~Abbeel, and L.~Pinto, ``Learning predictive representations for deformable objects using contrastive estimation,'' \emph{arXiv preprint arXiv:2003.05436}, 2020.
\bibitem[Garc{\i}a and Fern{\'a}ndez(2015)]{garcia2015comprehensive}
J.~Garc{\i}a and F.~Fern{\'a}ndez, ``A comprehensive survey on safe reinforcement learning,'' \emph{Journal of Machine Learning Research}, vol.~16, no.~1, pp. 1437--1480, 2015.
\bibitem[Achiam et~al.(2017)Achiam, Held, Tamar, and Abbeel]{achiam2017constrained}
J.~Achiam, D.~Held, A.~Tamar, and P.~Abbeel, ``Constrained policy optimization,'' \emph{arXiv preprint arXiv:1705.10528}, 2017.
\bibitem[Li et~al.(2018)Li, Kalabi{\'c}, and Chu]{li2018safe}
Z.~Li, U.~Kalabi{\'c}, and T.~Chu, ``Safe reinforcement learning: Learning with supervision using a constraint-admissible set,'' in \emph{2018 Annual American Control Conference (ACC)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2018, pp. 6390--6395.
\bibitem[Perkins and Barto(2002)]{perkins2002Lyapunov}
T.~J. Perkins and A.~G. Barto, ``Lyapunov design for safe reinforcement learning,'' \emph{Journal of Machine Learning Research}, vol.~3, no. Dec, pp. 803--832, 2002.
\bibitem[Chow et~al.(2018)Chow, Nachum, Duenez-Guzman, and Ghavamzadeh]{chow2018Lyapunov}
Y.~Chow, O.~Nachum, E.~Duenez-Guzman, and M.~Ghavamzadeh, ``A lyapunov-based approach to safe reinforcement learning,'' in \emph{Advances in Neural Information Processing Systems}, 2018, pp. 8092--8101.
\bibitem[Khalil(2014)]{khalil2014nonlinear}
H.~K. Khalil, \emph{Nonlinear Control}.\hskip 1em plus 0.5em minus 0.4em\relax Pearson Higher Ed, 2014.
\end{thebibliography}