Robust Learning-based Predictive Control for Discrete-time Nonlinear Systems with Unknown Dynamics and State Constraints
Extracted source text · arXiv 1911.09827
\documentclass{article}
\usepackage{arxiv}
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[journal]{../sty/IEEEtran}
%\usepackage{algorithm}%,algpseudocode}
%\usepackage{algorithmicx}
%\usepackage[algo2e]{algorithm2e}
\usepackage{arydshln}
\usepackage{mathtools}
\usepackage{comment}
\usepackage{graphicx}
\usepackage{graphics} % for pdf, bitmapped graphics files
\usepackage{epsfig} % for postscript graphics files
\usepackage{mathptmx} % assumes new font selection scheme installed
\usepackage{times} % assumes new font selection scheme installed
\usepackage{amsmath} % assumes amsmath package installed
\usepackage{amssymb} % assumes amsmath package installed
\usepackage[mathscr]{eucal}
\usepackage{enumerate}
\usepackage{mathrsfs}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{subfigure}
\usepackage{latexsym}
\usepackage{bm}
\usepackage{dsfont}
\usepackage{multicol}
\usepackage{multirow}
\usepackage{color}
\usepackage{url}
\usepackage{algorithmic}
%\usepackage{algorithm}
\usepackage[ruled]{algorithm2e}
%\usepackage[linesnumbered,boxed]{algorithm2e}
\usepackage{comment}
\usepackage{cite}
%\usepackage{indentfirst}
%\usepackage{flushend,cuted}
%
%% Centrare le figure
%\usepackage{adjustbox}
%\adjustboxset*{center}
\newtheorem{remark}{Remark}
\newtheorem{assumption}{Assumption}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem{definition}{Definition}
\newtheorem{corollary}{Corollary}
\newtheorem{theorem}{Theorem}
\newtheorem{property}{Property}
\newtheorem{problem}{Problem}
%\newtheorem{proof}{Proof}
% \newtheorem{algorithm}{Algorithm}
% Short for ^{\scriptscriptstyle[x]}
\newcommand{\ap}[1]{^{\scriptscriptstyle[#1]}}
% Calligraphic E
\newcommand{\E}{\mathcal{E}}
% Calligraphic N
\newcommand{\N}{\mathcal{N}}
% Environments
\newenvironment{bbmatrix}%
{\renewcommand{\arraystretch}{2.0}
\begin{bmatrix}}%
{\end{bmatrix}
\renewcommand{\arraystretch}{1.0}}
\renewcommand{\baselinestretch}{1}
\setlength{\parindent}{1em}
\def\build#1_#2^#3{\mathrel{\mathop{\kern0pt#1}\limits_{#2}^{#3}}}%
\def\argmin#1{\build{\rm argmin}_{#1}^{}}
\def\build#1_#2^#3{\mathrel{\mathop{\kern0pt#1}\limits_{#2}^{#3}}}%
\def\argmax#1{\build{\rm argmax}_{#1}^{}}
\makeatletter
\renewcommand*\env@matrix[1][*\c@MaxMatrixCols c]{%
\hskip -\arraycolsep
\let\@ifnextchar\new@ifnextchar
\array{#1}}
\makeatother
%
\title{Robust Learning-based Predictive Control for
Discrete-time Nonlinear Systems with Unknown
Dynamics and State Constraints\thanks{This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. }}
\author{
Xinglong~Zhang,~Jiahang Liu, ~Xin~Xu,~Senior member,~IEEE\\
College of Intelligence Science and Technology\\
National University of Defense Technology\\
Changsha 410073, China \\
\texttt{zhangxinglong18@nudt.edu.cn, liujiahang1992@foxmail.com, xuxin\_mail@263.net} \\
\And
Shuyou Yu, Hong~Chen,~Senior member,~IEEE\\
State Key Laboratory of Automotive Simulation and
Control and the Department of Control Science and Engineering\\
Jilin University at NanLing\\
Changchun 130025, China\\
\texttt{e-mail: yushuyou@126.com, chenh@jlu.edu.cn}\\% <-this % stops a space
}\label{key}
%
\begin{document}
\maketitle
% \IEEEtitleabstractindextext{%
\begin{abstract}
Model predictive control (MPC) has been widely employed as an effective method for model-based constrained control. For systems with unknown dynamics, reinforcement learning (RL) and adaptive dynamic programming (ADP) have received notable attention to solve the adaptive optimal control problems. Recently, works on the use of RL %for solving optimal control problems
in the framework of MPC have emerged, which can enhance the ability of MPC for data-driven control.
However, the safety under state constraints and the closed-loop robustness are difficult to be verified due to approximation errors of RL with function approximation structures.
Aiming at the above problem, we propose a data-driven robust MPC solution based on incremental RL, called data-driven robust learning-based predictive control (dr-LPC), for perturbed unknown nonlinear systems subject to safety constraints.
A data-driven robust MPC (dr-MPC) is firstly formulated with a learned predictor. The incremental Dual Heuristic Programming (DHP) algorithm using an actor-critic architecture is then utilized to solve the online optimization problem of dr-MPC. In each prediction horizon, the actor and critic learn time-varying laws for approximating the optimal control policy and costate respectively, which is different from classical MPCs. %The resultant control of dr-LPC consists of two terms that are both output-dependent.
The state and control constraints are enforced in the learning process via building a Hamilton–Jacobi–Bellman (HJB) equation and a regularized actor-critic learning structure using logarithmic barrier functions.
The closed-loop robustness and safety of the dr-LPC are proven under function approximation errors.
Simulation results on two control examples have been reported, which show that the dr-LPC can outperform the DHP and dr-MPC in terms of state regulation, and its average computational time is much smaller than that with the dr-MPC in both examples.
\end{abstract}
% Note that keywords are not normally used for peerreview papers.
\keywords {Reinforcement learning, model predictive control, safety constraints, robustness, nonlinear systems.}
% make the title area
\maketitle
%\IEEEdisplaynontitleabstractindextext
% \IEEEdisplaynontitleabstractindextext has no effect when using
% compsoc or transmag under a non-conference mode.
% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
%\IEEEpeerreviewmaketitle
%%
%
\section{Introduction}
Model predictive control (MPC), also known as receding horizon or moving horizon control, has received notable attention due to its theoretical developments and wide-spreading applications in industrial plants, see~\cite{mayne2000constrained,qin2003survey}. With MPC, usually, the control problem is transformed into an online optimization problem that penalizes the error of the state and control to the origin or steady-state value in a predefined finite prediction horizon with suitably chosen weights subject to model and variable constraints. %Typically, the optimization problem belongs to quadratic programming (QP) if all the compulsory constraints are linear, or quadratically constrained quadratic programming (QCQP) if there exist quadratic ones alongside with linear constraints.
In MPC, the optimal control sequence is computed via solving the underlying optimization problem at any adopted sampling time instant, and only the first control action is applied. Then at the subsequent time instant, the optimization problem is solved repeatedly, according to the receding horizon strategy.
As it is a model-based approach, a mathematical description of the original model is required for the design and implementation of MPC. %so as to
Most of the classic MPC algorithms assume that the prescribed model has been generated a priori, in which case, the identification process can be disregarded.
Robust MPC such as min-max MPC in~\cite{bemporad2003min} or tube-based MPC in~\cite{mayne2005robust} can be used, to account for the modeling uncertainty from identification.
A class of learning framework of MPC relying on an online update of the parameters of the controller, such as model description and system constraints, has drawn increasing interest for the capability of reducing conservativity and improving control performance. Many works have been developed in this new direction. Among them, from the theoretical perspective, a unitary learning-based predictive controller for linear systems has been addressed in~\cite{terzi2018learning}, where the identification process based on set membership is adopted to obtain a multi-step linear prediction and then used with robust MPC. Similarly, with resorting to the set membership identification, adaptive MPC algorithms for uncertain time-varying systems have been proposed in~\cite{lorenzen2017adaptive,fagiano2015scenario} for reducing the conservativity caused by robust MPC. Relying on the main idea from iterative leaning control, a data-driven learning MPC for iterative tasks has been studied in~\cite{rosolia2017learning} with terminal constraints updated in an iterative fashion. In~\cite{limon2017learning}, a learning nonlinear model predictive control algorithm with a machine learning-based model estimation, has been proposed with stability property guarantees. In the prescribed approaches, a robust (or stabilizing in~\cite{limon2017learning}) MPC problem is still to be solved online at each time instant, which might lead to an intensive computational load, especially for large-scale systems and nonlinear dynamics. It could preclude the applications of these approaches to nonlinear systems that must exhibit fast closed-loop dynamics. An attempt has been contributed in~\cite{korda2018linear} with the scope of learning predictors for nonlinear dynamics using Koopman operator, which is recently noted to be very useful for representing nonlinear dynamics by a linear one, see~\cite{arbabi2018introduction}. This approach paves the way to linear MPC formulation for nonlinear systems with a global linear predictor that represents the whole operation range, but it still leaves the theoretical property unresolved in this respect. It is also worth mentioning that a supervised machine learning algorithm has been used for approximation of nonlinear MPC in~\cite{hertneck2018learning}. The robustness is guaranteed under bounded control approximate errors with verified statistic empirical risk.
As an alternative to solve optimal control problems with the infinite or finite horizon, reinforcement learning (RL) and adaptive dynamic programming (ADP) have also received notable attention in the past decades, see~\cite{wangadp2017,lewis2009reinforcement,ni2014grdhp,lewis2013reinforcement,wang2009adaptive,liu-2015} and the references therein. Instead of solving online optimization problems, RL and ADP are interested in finding approximate solutions via value function and policy iteration in trial-and-error manner and are suitable for
complex nonlinear control tasks that are hard to be solved by optimal control techniques such as, exact dynamic programming, due to the non-linearity in Hamilton-Jacobi-Bellman equation. Similar to MPC, for control problems with high dimensions, RL and ADP might face the issues of computational complexity and learning efficiency. To cope with this problem, adaptive critic designs have been proposed, see for instance~\cite{xu2013online,li2018actor,xu2016manifold}, where the value function and policy iteration are replaced with the actor-critic approximate network structure.
Along this direction, various notable algorithms have been studied in~\cite{zhang2008novel,liu2013policy,zhang2009neural} for known dynamics, and in~\cite{wang2012optimal,jiang2012computational,liu2012neural} for unknown dynamics. These methods are designed for optimal control orientation of infinite-horizon. \cite{Xu-finitehorizonadp} has proposed a finite horizon ADP solution for nonlinear systems with input constraints, and the stability is guaranteed under the stable assumption of the open-loop system. In~\cite{zhao2014neural}, a finite-horizon near-optimal control algorithm has been presented for affine nonlinear systems with unknown dynamics, where an identifier designed with neural networks is used as the predictor.
The relations between the MPC and RL, and their comparison studies have been presented in~\cite{GORGES20174920} and~\cite{mpcrl_com}. Recently, works on the perspective of integration of MPC and RL have emerged. \cite{zanon2019safe} and~\cite{koller2019learningbased} have proposed safe controllers based on Robust MPC for learning-based control relying on an event-triggered mechanism. In~\cite{wabersich2018linear}, a safe learning controller has been presented based on set-membership recursion. From a different perspective, the RL with the actor-critic structure has also been used to solve nonlinear MPC problems in an efficient way. Among them, a learning-based receding horizon controller for perturbed discrete-time systems with control constraints has been studied in~\cite{xu2018learning}. The algorithm is in an iterative batch-mode learning way, and the convergence under the null approximate error is discussed. In~\cite{dong2018functional}, a Heuristic Dynamic Programming (HDP) based functional nonlinear MPC has been proposed for nonlinear discrete-time systems, where the control saturation is considered. The uniform ultimate boundedness of the approximation network in each prediction horizon is proven. It is highlighted that the use of RL with the actor-critic structure will bring safety issues for systems with hard state constraints due to the trial-and-error learning manner and to the existence of function approximation errors. Also, the robustness of the closed-loop system under approximation errors with receding horizon strategy might not be easily verified.
Motivated by prescribed problems, a robust MPC solution based on the incremental RL with an actor-critic structure, i.e., dr-LPC, is proposed for perturbed nonlinear systems subject to safety constraints. Note that the satisfaction of state constraints has been considered in the adaptive control community, see~\cite{liu-2016,Chen-2017,LIU201883}, via constructing barrier Lyapunov functions, while the proposed solution and the adopted technique in this paper are highly different. %A data-driven robust MPC (dr-MPC) is initially formulated with a learned predictor using the Koopman operator. Different from~\cite{wabersich2018linear}, we use the incremental DHP to solve the online optimization problem of the dr-MPC in order to reduce computational complexity.
The main contributions of this paper are summarized in two aspects.
\begin{enumerate}[{$\bullet$}]
\item A dr-LPC scheme is proposed for unknown perturbed dynamics subject to safety constraints.
A data-driven robust MPC (dr-MPC) is initially formulated with a learned predictor using Koopman operator. The online optimization problem of the dr-MPC is solved using the incremental DHP with an actor-critic structure that learns time-varying laws for approximating the optimal control policy and costate. The resultant control of the dr-LPC consists of a successive time-varying functional and a static feedback term that are both output-dependent.
\item The safety and robustness of the closed-loop system with the dr-LPC are verified. The state and control constraints are enforced in the learning process via building a constraint-inclusive (Hamilton–Jacobi–Bellman) HJB equation and a regularized actor-critic learning structure with logarithmic barrier functions. The boundedness property of the actor-critic network is analyzed and the convergence condition is given. The safety of the controller as well as the robustness of the closed-loop system are proven under approximation errors.
\end{enumerate}
Two simulation studies for regulating a Van der Pol oscillator and an Inverted Pendulum are reported, which show that the dr-LPC outperforms the infinite horizon DHP with real dynamics, and shows an advantageous point than the dr-MPC in terms of state regulation in both cases, but requires more input energy for controlling Inverted Pendulum. The computational complexity with dr-LPC is much smaller than that with dr-MPC.
The rest of the paper is organized as follows. Section II firstly introduces the considered control problem and the dr-MPC solution. In Section III the design and main algorithm of the proposed dr-LPC are described. Section VI shows the simulation results, while some conclusions are drawn in Section VII. Proofs to the main results and off-line computation details are given in the Appendix.
\textbf{Notation:} Given the variable $r$, we use $\overrightarrow{r}(k:k+N-1)$ to denote the sequence $r (k)\ldots r (k+N-1)$, where $k$ is the discrete time index and $N$ is a positive integer. For a vector $x\in\mathbb{R}^{n\times 1}$, we use $\|x\|_Q^2$ to stand for $x^{\top}Qx$, $\|x\|$ to denote its Euclidean norm, while for a matrix $x\in\mathbb{R}^{n\times m}$, we denote $\|x\|$ as the Frobenius norm. Given a matrix $A\in\mathbb{R}^{n\times m}$, we use $\sigma_{max}(A)$ and $\sigma_{min}$ to denote the maximum and minimal single values. Given two sets $\mathcal{Z}$ and $\mathcal{V}$, their Minkowski sum is represented by $\mathcal{Z}\oplus \mathcal{V}=\{z+v|z\in \mathcal{Z}, v\in \mathcal{V}\}$. Given a set $\mathcal Z$, we use $Int(\mathcal{Z})$ to denote its interior and $\partial \mathcal{Z}$ to represent its boundary. %We use the notation $\mathcal{N}$ to denote non-negative integer.
For a given set of variables $z_{i}\in{\mathbb{R}}^{q_{i}}$,
$i=1,2,\dots,M$, we define the vector whose vector-components are $z_{i}$ in the following compact form: $(z_{1}, z_{2}, \cdots, z_{\rm\scriptscriptstyle M})=[\,z_{1}^{\top}\ z_{2}^{\top}\ \cdots\ z_{\rm\scriptscriptstyle M}^{\top}\,]^{\top}\in{\mathbb{R}}^{q}$, where $q= \sum_{i=1}^{M}q_{i}$. Finally, a ball with radius $\rho_{\varepsilon_{i}}$ and centered at $\bar{x}$ in the ${\mathbb{R}}^{dim}$
space is defined as follows
\[
\mathcal{B}_{\rho_{\varepsilon_{i}}}(\bar{x}):=\{x\in{\mathbb{R}}^{dim}:||x-\bar{x}||\leq\rho_{\varepsilon_{i}}\}.
\]
\section{Problem formulation and a dr-MPC solution}
\subsection{Control problem and data-driven model}\label{sec:model-mpc}
Consider a class of discrete-time nonlinear perturbed systems described by
\begin{equation}\label{Eqn:non-model}
x(k+1)=f(x(k),u(k))+w_o(k)
\end{equation}
where $k$ is the discrete-time index, $x\in \mathcal{X}\subseteq \mathbb{R}^{n}$, $u\in \mathcal{U}\subseteq \mathbb{R}^m$ are the state and control variables, $w_o\in\mathcal{W}_o$ is an additive noise, $\mathcal{X}$ and $\mathcal{U}$ are convex and compact sets containing the origin in their interiors, $f$ is the state transition function and is assumed to be smooth but unknown. Starting from any initial condition $x(0)\in\mathcal{X}$, the control objective is to drive the pair $(x(k),u(k))$ to the origin as $k$ goes to infinity.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In line with~\cite{arbabi2018introduction,korda2018linear}, we adopt a data-driven linear predictor based on Koopman operator to represent~\eqref{Eqn:non-model}:
\begin{equation}\label{Eqn:linear_p-residual}
\Sigma:\ \left\{\begin{array}{l}
{s}(k+1)=A{s}(k)+Bu(k)+Dw_o(k)+w(k)\\%\delta_w(k)\\
[0.2cm] x(k)=%C{s}(k)
\bar{\Phi}(s(k),n)+v(k),%\delta_v(k),
\end{array}\right. \qquad
\end{equation}
where ${s}(k)\in \mathbb{R}^{\bar n}$, $\bar n>n$, is a lifted state variable, mapping from the original state $x(k)$:
\begin{equation}\label{Eqn:initial-con}
{s}(k)=\Phi\left(x(k),\bar n\right) :=( {\phi_{1}\left(x(k)\right)},{\cdots}, {\phi_{\bar n}\left(x(k)\right)})
\end{equation}
where $\phi_i$, $i=1,\cdots,\bar n$, can be chosen as basis or some regular nonlinear functions. $A$, $B$, $D$ are linear model parameters, $\bar{\Phi}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{\bar n}$ %$C\in \mathbb{R}^{n\times \bar n}$
is the output mapping function from the feature to original state space, $w(k)$ and $v(k)$ are the modeling uncertainties. It is assumed that $w(k)\in\mathcal{W}$, $v(k)\in \mathcal{V}$, where $\mathcal{W}$ and $\mathcal{V}$ are compact sets containing the origin. %Given the structure of the predictor, and under certain assumptions, possible choices of $\mathcal{W}$ and $\mathcal{V}$ can be given and are deferred in Appendix~\ref{sec:app-b}.%=\bar{\delta}_w({s}(k),u(k))$, $\bar \delta_w$ is the minimal residual caused by~\eqref{Eqn:appro_K}; while $\delta_v=\bar{\delta}_v{s}(k)$, $\bar \delta_v$ is the minimal residual of~\eqref{Eqn:appro_C}.
For prediction purpose, we define the unperturbed system of $\Sigma$ as
\begin{equation}\label{Eqn:unpert}
\hat\Sigma:\ \left\{\begin{array}{l}
\hat{s}(k+1)=A\hat{s}(k)+B\hat u(k)\\
[0.2cm] \hat x(k)=\bar{\Phi}(\hat s(k),n),
\end{array}\right. \qquad
\end{equation}
with $\hat {s}(k)=\Phi\left(\hat x(k),\bar n\right)$.
The control action to $\Sigma$ in robust MPC relies on the feedback term of the output error correction:
\begin{equation}\label{Eqn:real_u}
u(k)=\hat u^o(k)+Ke_{\scriptscriptstyle{s}}(k)
\end{equation}
where $e_{\scriptscriptstyle {s}}:={s}-\hat {s}$, $\hat u^o$ can be computed with a standard MPC (deferred in~\eqref{Eqn:optimiz1}) with respect to $\hat\Sigma$, $K$ is a feedback gain matrix such that $F=A+BK$. The error $e_{\scriptscriptstyle {s}}$ involves in the following unforced system by subtracting $\Sigma$ with~\eqref{Eqn:real_u} and $\hat\Sigma$:
\begin{equation}\label{Eqn:e}
\Delta\Sigma:\ \left\{\begin{array}{l}
e_{\scriptscriptstyle{s}}(k+1)=Fe_{\scriptscriptstyle{s}}(k)+Dw_o(k)+w(k)\\
[0.2cm] e_x(k)=%C e_{\scriptscriptstyle{s}}(k)
\bar{\Phi}(s(k),n)-\bar{\Phi}(\hat s(k),n)+v(k),
\end{array}\right. \qquad
\end{equation}
where $e_x=x-\hat x$.
\begin{assumption}\label{assu:system}
\begin{enumerate}[1.]
% \item\label{assu:ac} The pair $(A, C)$ is observable. %and the pair $(A, B)$ is stabilizable.
\item\label{assu:mea} The variable $x$ is measurable;
\item\label{assu:ab} $F=A+BK$ is Schur stable;
\item\label{assu:a_inverse} The matrix $A$ is full rank, i.e., $\text{rank}(A)=\bar n$;
\item\label{assu:phi-ori} The lifted function satisfies $\Phi(0,\bar n)=\bm 0$.
\end{enumerate}
\end{assumption}
The former three points are standard assumptions, while the fourth point is required for the reason that there are multiple choices of the basis function vector $\Phi$, some of them might result $\Phi(0,\bar n)\neq0$. In this case, a coordinate transformation can be used to ensure $\Phi(0,\bar n)=0$.
Let $\bar{\mathcal{Z}}$ be the (possibly minimal)-robust invariant set of $e_{\scriptscriptstyle{s}}$, such that $ F\bar{\mathcal{Z}}\oplus(D\mathcal{W}_o)\oplus\mathcal{W}\subseteq\bar{\mathcal{Z}}$. Assuming that $\bar {\Phi}$ is Lipschitz continuous, i.e., $\|\bar {\Phi}(s,n)-\bar {\Phi}(\hat s,n)\|\leq L_s\|s-\hat s\|$, it is possible to compute the robust ``output" invariant set of $e_x$ defined as $\bar{\mathcal{O}}$. As a special case, $\bar {\Phi}(s,n)=Cs$ leads to $\bar{\mathcal{O}}$ being $C\bar{\mathcal{Z}}\oplus \mathcal{V}$. %=\{\bar{\Phi}(z,n)-\bar{\Phi}(\hat z,n)+v|z\in\mathcal{X}C\mathcal{Z}\oplus\mathcal{V}$.
For the sake of simplicity, in the rest of the paper, we choose $\bar{\Phi}(s,n)=Cs$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Data-driven output-feedback robust MPC}\label{sec:dr-MPC}
In~\cite{korda2018linear}, a data-driven MPC solution has been presented with the predictor obtained with the Koopman operator, where additive disturbance and modeling error are not considered. Hence, we firstly extend the work~\cite{korda2018linear} to a rigorous dr-MPC formulation with~\eqref{Eqn:unpert} based on the tubed-based MPC framework. This results in an output-feedback robust MPC problem stated in the following. At any time instant $k$, an online optimization problem is to be solved:
\begin{equation}\label{Eqn:optimiz1}
\begin{array}{l}
\min_{\hat s(k),\overrightarrow {\hat u}(k:k+N-1)} {V\big(\hat{s}(k)\big)}\\
\mbox{subject to:}\\
\bullet\,\,\mbox{the dynamics}~\eqref{Eqn:unpert}\\
\bullet\,\,\mbox{initial constraint}~\eqref{Eqn:initial-con} \\
\bullet\,\,\hat {s}(k+i)\in \bar {\mathcal{S}},\,\,i=0,\dots,N-1\\
\bullet\,\,\hat u(k+i)\in \bar{\hat {\mathcal{U}}},\,\,i=0,\dots,N-1\\
\bullet\,\,s(k)-\hat {s}(k)\in\bar{\mathcal{Z}}\\
%\bullet\,\,{s}(k)-\hat {s}(k)\in \mathcal{Z}\\
\bullet\,\,\hat {s}(k+N)\in \bar{{\mathcal{S}}_f},
\end{array}
\end{equation}
where
\begin{equation}\label{Eqn:V_b}
\begin{array}{ll}
V\big(\hat{s}(k)\big)=\\
\sum_{i=0}^{N-1}(\|\hat {x}(k+i)\|_{ Q}^2+\|\hat u(k+i)\|_{R}^2)
+V_f(\hat{s}(k+N)),
\end{array}
\end{equation}
$V_f(\hat{s})=\hat{s}^{\top}P\hat{s}$, $\bar{\mathcal{S}}=\{\hat{s}|C\hat s\in {\mathcal{X}}\ominus\bar{\mathcal{O}}\}$, $\bar{\hat {\mathcal{U}}}={\mathcal{U}}\ominus K\bar{\mathcal{O}}$, the invariant set is given as $\bar{\mathcal{S}}_f=\{\hat {s}| \hat{s}^\top \bar Z\hat{s}\leq 1\}$, where $\bar Z$ is a symmetric positive-definite matrix such that $\bar Z>F^{\top}\bar ZF$. The terminal penalty matrix $P$ is a symmetric positive-definite matrix computed with the following Lyapunov function
\begin{equation}\label{Eqn:LYA}
F^{\top}PF-P=-\bar Q-K^{\top}RK.
\end{equation}
where $\bar Q=C^{\top}QC$.
Assume at any time instant $k$ that, the optimal control sequence $\overrightarrow {\hat u^o}(k:k+N-1|k)$ can be found, then the final control applied to the system~\eqref{Eqn:non-model} is given as
\begin{equation}\label{Eqn:real_u_k}
u(k)=\hat u^o(k|k)+K({x}(k)-\hat{x}(k|k))
\end{equation}
As a further step to~\cite{korda2018linear}, we prove the feasibility and robustness of the dr-MPC with resorting to the theoretical arguments of the classical tube-based MPC.
\begin{theorem}[Recursive feasibility and robustness of the dr-MPC]\label{the:rmpc}
Let assume the sets $\mathcal{W}$ and $\mathcal{V}$ are finite. Under Assumption~\ref{assu:system}, if the initial condition $x(0)\in\mathcal{X}$ is such that the optimization problem~\eqref{Eqn:optimiz2} is feasible at $k=0$, then
\begin{enumerate}[1.]
\item the optimization problem~\eqref{Eqn:optimiz2} is recursively feasible at all $k>0$;
\item under~\eqref{Eqn:optimiz2}, $\hat x(k),\,\hat u(k)\rightarrow 0$, $ x(k)\rightarrow \bar{\mathcal{O}}$; consequently, $\hat s(k)\rightarrow 0$, $s(k)\rightarrow\bar{\mathcal{Z}}$, as $k\rightarrow+\infty$.
\end{enumerate}
\end{theorem}
Proof: please refer to Appendix~\ref{appen-the1}. \hfill$\square$
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Design of the dr-LPC}
In this section, the proposed dr-LPC is designed. The main idea is to design a DHP with an actor-critic network in order to learn a near-optimal successive time-varying control policy for ~\eqref{Eqn:optimiz1} in a trial-and-error way. In this way, exact online optimization is not required and computational complexity is expected to be reduced. However, the following two problems are to be concerned: i) safety constraints might not be verified due to the use of actor-critic approximation network, so how to explicitly cope with state and control constraints in the learning process is a crucial issue; ii) the existence of approximating errors of the actor could introduce new uncertainties in the control channel, hence the impact must be considered in the control design to guarantee constraint satisfaction and robustness.%; iii) to reduce computational complexity, the actor-critic is performed forward-in-time, the initial state as the decision variable.
In that follows, we first present the predictor and the core learning control problem by taking into consideration of the approximating errors, then the value function reconstruction is presented with soft barrier functions.
Finally, the main algorithm of the regularized actor-critic design with receding horizon strategy, its theoretical property, and the implementing details of the dr-LPC are described.
%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Learning control problem with approximation errors}
As the actor-critic approximating network might introduce uncertainty in the control channel, the real dynamics of~\eqref{Eqn:non-model} with dr-LPC can be rewritten as
\begin{equation}\label{Eqn:all-residual}
\Sigma:\ \left\{\begin{array}{l}
{s}(k+1)=A{s}(k)+Bu(k)+d(k)\\
[0.2cm] x(k)=C{s}(k)+v(k),
\end{array}\right. \qquad
\end{equation}
where $d(k)=Dw_o(k)+w(k)+\delta_{N}(k)$, $\delta_{N}$ is the approximation error of the control action by the actor-critic network. It is assumed that $d(k)\in\mathcal{D}$, where $\mathcal{D}$ is a compact set containing the origin. Given the structure of the linear predictor and the actor-critic network, and under certain assumptions, (possibly conservative) choice of $\mathcal{D}$ can be given and are deferred in Appendix~\ref{sec:app-b}.
Similarly to Section~\ref{sec:model-mpc}, let the robust invariant set of $e_{\scriptscriptstyle{s}}$ in this case be $\mathcal{Z}$, such that $ F\mathcal{Z}\oplus\mathcal{D}\subseteq\mathcal{Z}$. The corresponding robust ``output" invariant set of $e_x$ is defined as $\mathcal{O}$.
At any time $k$, the dr-LPC aims to solve~\eqref{Eqn:optimiz1} with the actor-critic structure. Note that, in~\eqref{Eqn:optimiz1} the initial state $\hat s$ is served as the decision variable, which is difficult to be optimized in the forward-in-time iterative manner. Indeed, a near-optimal solution of $\hat s$ can be found using Monte Carlo methods with sufficient samples but leading to an intensive computation load. Hence, we only allow two typical choices of $\hat s$ to be considered, i.e., $\hat s(k)\in\{s(k),\hat s(k|k-1)\}$.
Hence the new optimization problem becomes
\begin{equation}\label{Eqn:optimizL}
\begin{array}{l}
\min_{\hat s(k),\overrightarrow {\hat u}(k:k+N-1)} {V\big(\hat{s}(k)\big)}\\
\mbox{subject to:}\\
\bullet\,\,\mbox{the dynamics}~\eqref{Eqn:unpert}\\
\bullet\,\,\mbox{initial constraint}~\eqref{Eqn:initial-con} \\
\bullet\,\,\hat {s}(k+i)\in \mathcal{S},\,\,i=0,\dots,N-1\\
\bullet\,\,\hat u(k+i)\in \hat {\mathcal{U}},\,\,i=0,\dots,N-1\\
\bullet\,\,\hat s(k)\in\{s(k),\hat s(k|k-1)\}\\
\bullet\,\,\hat {s}(k+N)\in {{\mathcal{S}}}_{f},
\end{array}
\end{equation}
where $\mathcal{S}=\{\hat{s}|C\hat {s}\in {\mathcal{X}}\ominus\mathcal{O}\}$, $\hat {\mathcal{U}}={\mathcal{U}}\ominus K\mathcal{Z}$, the invariant set is given as ${\mathcal{S}}_{f}=\{\hat {s}| \hat{s}^\top Z\hat{s}\leq \varrho\}$, where $0\leq\rho\leq1$. %$Z$ is a symmetric positive-definite matrix such that $Z>F^{\top}ZF$.
%%%%%%%%%%%%%%%%%%
\subsection{Cost function reconstruction with barrier functions}
%%%%%%%%%%%%%%%%%%%%%%%%%
Hard safety constraints are difficult to be coped with in the actor-critic design. Hence we propose to transform the hard state and control constraints into soft ones with barrier functions. In this way and along the line with~\cite{feller2017relaxed}, the value function can be reconstructed as
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{equation}\label{Eqn:V_bz}
\begin{array}{ll}
V_b\big(\hat{s}(k)\big)=&V\big(\hat{s}(k)\big)+
\sum_{i=0}^{N-1}
\mu B(\hat{s}(k+i))+\mu B(\hat u(k+i))\\ &+\mu B_f(\hat {s}(k+N))
\end{array}
\end{equation}
where $\mu$ is a weighting scalar, $B(\hat{s})$, $B(\hat u)$, and $B_f(\hat {s})$ are the state, control, and terminal-state barrier functions respectively. $B(\hat{s})=0$ for $\hat {s}\in Int(\mathcal{S})$, and $B(\hat{s})\rightarrow +\infty$ for $\hat {s}\in\partial\mathcal{S}$. $B(\hat u)=0$ for $\hat u\in Int(\hat{\mathcal{U}})$, and $B(\hat u)\rightarrow +\infty$ for $\hat u\in\partial\hat{\mathcal{U}}$. $B_{f}(\hat {s})=0$ for $\hat {s}\in Int(\mathcal{S}_f)$, and $B_{f}(\hat {s})\rightarrow +\infty$ for $\hat {s}\in\partial\mathcal{S}_f$.
If the value function in~\eqref{Eqn:V_bz} is finite, the system variables stay in their restrictive interiors and result in almost null values of the barrier functions.
Specifically in this case, we use continuous and differentiable logarithmic barrier functions to represent the soft constraints. The definitions for polyhedral and ellipsoidal constraints are introduced.
\begin{definition}\label{defi:polytopic}
For any variable $z\in\mathbb{Z}$, where $\mathbb{Z}=\{z| a_i^\top z\leq b_i,\forall i=1,\cdots,p\}$ is a polyhedron, the barrier function is defined as
\begin{equation*}\label{Eqn:barrier_in}
\bar B(z)=\left\{\begin{array}{l}
-\sum_{i=1}^p{\rm{log}}\big(b_i- a_i^\top z\big)\ \ z\in Int(\mathbb{Z})\\
[0.2cm]+\infty\ \ \rm{otherwise}.
\end{array}\right. \qquad
\end{equation*}
\end{definition}
\begin{definition}
For any variable $z\in\mathbb{Z}$, where $\mathbb{Z}=\{z| z^\top Zz\leq 1 \}$ is an ellipsoid, and where $Z$ is a symmetric positive-definite matrix with suitable dimensions, the barrier function is defined as
\begin{equation*}\label{Eqn:barrier_eli}
\bar B(z)=\left\{\begin{array}{l}
-{\rm{log}}\big(1- z^\top Zz \big)\ \ z\in Int(\mathbb{Z})\\
[0.2cm]+\infty\ \ \rm{otherwise}.
\end{array}\right. \qquad
\end{equation*}
\end{definition}
\begin{remark}
It is highlighted that the logarithmic barrier function has been widely used in interior-point methods to solve convex constrained optimization problem, see~\cite{boyd2004convex}. In our case, the barrier function is not only used in the optimization formulation but also in the learning process where the safety restriction on the network weights is enforced, see Section~\ref{sec:actor-critic}.
\end{remark}
%%%%%%%%%%%%%%%%%%%%%%
Note however that $\bar B(0)$ is not guaranteed to be zero, which results in the optimal value function $V_b^o\big(0,\, 0\big)$ probably being nonzero. This impedes the stability arguments by selecting $V_b^o$ as a Lyapunov function due to null of the value in the origin being violated. For this reason, we introduce the following Lemma about barrier functions~\cite{wills2004barrier}.
\begin{lemma}\label{lemma:recentered}\hfill
\begin{enumerate}[1.]
\item\label{lemma:recentered_1} Let $B_c(z)=\bar B(z)-\bar B(0)-\bigtriangledown\bar B(0)^{\top}z$ be a gradient re-centered barrier function of $\bar B(z)$, then $B_c(z)$ is differentiable and convex for all $z\in Int(\mathbb{Z})$, and $B_c(0)=0$;
\item \label{lemma:relax} Let the relaxed barrier function for polyhedral constraint be defined as
\begin{equation}\label{Eqn:relaxed_B}
B(z)=\left\{\begin{array}{ll}
B_c(z)&\bar\sigma\geq \kappa\\
\gamma(z,\bar\sigma)&\bar\sigma <\kappa
\end{array}\right.
\end{equation}
where the small positive scalar $\kappa$ is the relaxing factor, $\sigma_i=b_i-a_i^{\top}z$, $i=1,\cdots,p$, $\bar\sigma=\min\{\sigma_1,\cdots,\sigma_p\}$, the function $\gamma(z,\bar\sigma):(-\infty,\kappa)$ is strictly monotone and differentiable such that $B(z)$ is differentiable at any $z$ that $\bar\sigma=\kappa$, and $\bigtriangledown^2\gamma(z,\bar\sigma)$ is smaller than $\sum_{i=1}^p\|a_i\|_{\kappa^{-2}}$, then there exists a positive-definite matrix $H\geq\frac{1}{2}\sum_{i=1}^p\|a_i\|_{\kappa^{-2}}^2$ such that $B(z)\leq z^{\top}Hz\leq B_{max}(z)$, where $B_{max}(z)=\max_{z\in\mathbb{Z}}z^{\top}Hz$.
\end{enumerate}
\end{lemma}
Proof: please refer to Appendix~\ref{appen-lamma}. \hfill$\square$
\begin{remark}
The control, state, and terminal state constraints can be easily transformed into the corresponding gradient re-centered barrier functions, while due to space limitations the computation procedures are omitted. It is highlighted that in view of Lemma~\ref{lemma:recentered}.\ref{lemma:recentered_1}, the optimal value function at the origin is zero, which allows us to choose $V_b^o$ as a Lyapunov function candidate.
\end{remark}
With~\eqref{Eqn:relaxed_B} and $B_f(z)=B_c(z)$ being included in~\eqref{Eqn:V_bz}, the terminal penalty matrix $P$ is modified as
\begin{equation}\label{Eqn:qn_ph}
F^{\top}PF-P=-\bar Q-K^{\top}RK-\mu H,
\end{equation}
where $H=H_{\hat{s}}+K^{\top}H_{\hat u}K$ and $H_{\hat{s}}$ and $H_{\hat u}$ are computed respectively according to Lemma~\ref{lemma:recentered}.\ref{lemma:relax} with a presumed value of $\kappa$ in~\eqref{Eqn:relaxed_B} for $\mathcal{S}$ and $\hat {\mathcal{U}}$.
%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Constraint-inclusive HJB equation and iterative DHP}
Let at any time instant $k$, the index $\tau$ belong to the prediction horizon $[k,k+N-1]$ and,
\begin{equation*}\label{Eqn:r_bzk}
\begin{array}{l}
\begin{array}{ll}%\|\hat{s}(k)\|_{\bar Q}^2+\|\hat u(k)\|_{R}^2+\mu B(\hat u(k)),\ \mbox{for}\ \tau=k\\
r_b(\tau)=\|\hat{s}(\tau)\|_{\bar Q}^2+
\|\hat u(\tau)\|_{R}^2+\mu B(\hat {s}(\tau))+\mu B(\hat u(\tau)),\ %\mbox{for}\ \tau\in[k,k+N-1]\\
\end{array}%\right.
\end{array}
\end{equation*}
We define
\begin{equation}\label{Eqn:V_bzk}
\begin{array}{ll} V_{b}\big(\hat {s}(\tau)\big)=\\
=\sum_{i=\tau-k}^{N-1}r_b(k+i)+V_f(\hat {s}(k+N))+\mu B_f(\hat{s}(k+N))\\
=r_b(\tau)+ V_{b,\tau+1}\big(\hat{s}(\tau+1)\big)
\end{array}
\end{equation}
where $V_{b,k+N}\big(\hat{s}(k+N)\big)=V_f(\hat {s}(k+N))+\mu B_f(\hat{s}(k+N))$.
Different from~\eqref{Eqn:optimiz1} where the control sequence is regarded as a whole and computed by solving once the underlying optimization problem, for any initial condition $\hat s(k)\in\{s(k),\hat s(k|k-1)\}$, the proposed controller computes the control $u(\tau)$ at any time instant $\tau\in[k,k+N-1]$, i.e.,
\begin{equation}\label{Eqn:optimiz2}
\begin{array}{l}
\min_{\hat u(\tau)} r_b(\tau)+ V_{b}\big(\hat{s}(\tau+1)\big)\\
\mbox{subject to:}\\
\bullet\,\,\mbox{the dynamics}~\eqref{Eqn:unpert}%~\mbox{with}~\eqref{Eqn:initial-con}\\
\end{array}
\end{equation}
Assume that the optimal solution $\hat u^{\ast}(\tau)$ $\forall$ $\tau\in[k,k+N-1]$ can be found for \eqref{Eqn:optimiz2}, that solves the discrete-time HJB equation
%\begin{equation}\label{Eqn:u_optimal}
$V_{b}^{*}(\hat{s}(\tau))= \min_{\hat u(\tau)} r_b(\tau)+ V_{b}\big(\hat{s}(\tau+1)\big)$.
%\end{equation}
According to optimal principle, and in view of~\eqref{Eqn:V_bzk} and~\eqref{Eqn:optimiz2}, minimizing $V_{b}\big(\hat {s}(\tau)\big)$ is equivalent to solving
\begin{equation}\label{Eqn:hjb}
\begin{array}{ll}
\frac{\partial V_{b}^{\ast}(\hat{s}(\tau))}{\partial \hat u^{\ast}(\tau)}=
\mu\frac{\partial B^{\ast}(\hat u(\hat{s}(\tau)))}{\partial \hat u^{\ast}(\tau)}+2R\hat u^{\ast}(\tau)+B^\top \lambda^{\ast}(\hat{s}(\tau+1))=0,
\end{array}
\end{equation}
$\forall \tau\in[k,k+N-1]$, where the costate $\lambda^{\ast}(\hat{s}(\tau))$ is the partial derivative of optimal value function $V^{\ast}_{b}\big(\hat{s}(\tau)\big)$
with respect to $\hat {s}^{\ast}(\tau)$:
\begin{equation}\label{Eqn:lambda}
\begin{array}{ll}
\lambda^{\ast}(\hat{s}(\tau))
=\mu\frac{\partial B(\hat{s}(\tau))}{\partial \hat{s}(\tau)}+2\bar Q\hat{s}(\tau)+A^\top \lambda^{\ast}(\hat{s}(\tau+1))
\end{array}
\end{equation}
$\forall$ $\tau\in[k, k+N-1]$.
%where $\star$ represents ${s}$ for $\tau\in[k,k+N-2]$ and $f$ for $\tau=k+N-1$.
In principle, one can resort to the underlying discrete-time Riccati Algebraic Equation to compute the optimal policy for an unconstrained linear system. But this is not the case as the HJB equation is nonlinear-function inclusive. %policy or value iteration algorithms to solve the above problem.
The iterative DHP can be used to solve the above HJB equation. Starting from the initial choice $\lambda^{0}(\hat{s}(\tau+1))=0$, $\tau\in[k,k+N-1]$, the algorithm consists of two main iterative steps.
\begin{enumerate}
\item Policy improvement: at the $i$-th iterative step, an improved control policy can be obtained as
\begin{equation}\label{Eqn:policy-im}
\begin{array}{ll}
\hat u^{i+1}(\tau)=
{\rm arg}\min_{\hat u(\tau)}\{\mu\frac{\partial B(\hat u(\hat{s}(\tau)))}{\partial \hat u(\tau)}+2R\hat u(\tau)+B^\top \lambda^i(\hat{s}(\tau+1))=0\}
\end{array}
\end{equation}
\item Value function update: given a control policy $\hat u^i(\tau)$, the costate function can be updated as
\begin{equation}\label{Eqn:value-up}
\lambda^{i+1}(\hat{s}(\tau))
=\mu\frac{\partial B(\hat{s}(\tau))}{\partial \hat{s}{(\tau)}}+2\bar Q\hat{s}(\tau)+A^\top \lambda^i(\hat{s}(\tau+1))
\end{equation}
\end{enumerate}
\begin{remark}
The convergence proof of $\hat u^{i}$ and $\lambda^{i}$ to the optimal values of the above value iteration algorithm~\eqref{Eqn:policy-im} and~\eqref{Eqn:value-up} can be easily verified as $i\rightarrow +\infty$, according to the results in~\cite{xu2018learning} and~\cite{heydari2012finite}.
\end{remark}
\subsection{Regularized actor-critic design with logarithmic barrier functions}\label{sec:actor-critic}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In this section we present the main idea of using a regularized actor-critic design with logarithmic barrier functions to implement the learning-based predictive controller. % with the scope to obtain a near-optimal control policy.
Different from~\cite{xu2018learning} and~\cite{heydari2012finite}, an efficient incremental actor-critic learning algorithm is proposed, where the iteration step $i$ in~\eqref{Eqn:policy-im} and~\eqref{Eqn:value-up} coincides with the time $\tau$. In the prediction horizon starting from any time $\tau=k$, the critic network is in charge of approximating the costate $\lambda^{\ast}(\tau)$ for all $\tau\in[k,k+N]$. The actor is for estimating the control vector $\overrightarrow{\hat u^{\ast}} (k:k+N-1)$.
%%%%%%%%%%%%%%%%%%%
The critic network is firstly designed. In view of the universal capability of neural network in terms of function approximation, $\lambda^{\ast}(\hat{s}(\tau))$ can be represented as $$\lambda^{\ast}(\hat{s}(\tau))=W_{c}^{\top}h_c(\hat{s}(\tau),\tau)+\bar{\epsilon}_c(\tau)$$
where $W_c\in\mathbb{R}^{N_c\times \bar n}$ is the weighting matrix, $h_c$ is a vector whose entries are basis functions. It is assumed that $\|W_c\|\leq W_{c,m}$, $\|h_c\|\leq h_{c,m}$, $\|\bar \epsilon_c\|\leq \bar \epsilon_{c,m}$. $\bar{\epsilon}_c\rightarrow 0$ as $N_c\rightarrow +\infty$.
The estimating critic network is given as
\begin{equation}\label{eqn:critic}
\hat\lambda(\hat{s}(\tau))=\hat W_c^{\top}h_c(\hat{s}(\tau),\tau),
\end{equation}
where $\tau\in[k,k+N]$, $\hat W_c$ is the estimate of $W_c$. %For the sake of clarity, we use $h_c(\tau)$ to represent $h(\hat{s}(\tau))$.
The critic network wishes to minimize the residual of the optimal costate $\lambda^{\ast}$ and $\hat\lambda$. However, as $\lambda^{\ast}$ is not available, we define $\lambda_d(\tau)$ as the target, that is
\begin{equation}\label{Eqn:lam_d}
\begin{array}{l}
\lambda_d(\hat {s}(\tau))=
\left\{\begin{array}{l}\mu \frac{\partial B(\hat {s}(\tau))}{\partial \hat {s}(\tau)}+
2\bar Q\hat {s}(\tau)+
A^\top \hat\lambda(\hat {s}(\tau+1)),\mbox{for} \ \tau\in[k,k+N-1]\\
\mu \frac{\partial B_f(\hat {s}(\tau))}{\partial \hat {s}(\tau)}+
2P\hat{s}(\tau),\mbox{for} \ \tau=k+N
\end{array}\right.
\end{array}
\end{equation}
where $\hat{s}(k+N)$ is randomly chosen from ${\mathcal{S}}_{f}$.
The target $\lambda_d(\hat{s}(\tau))$ can also be represented as the output of the critic network plus a residual term, i.e., $ \lambda_d(\hat{s}(\tau))=\hat\lambda(\hat{s}(\tau))+\epsilon_{c}(\tau)$, where $\epsilon_c(\tau)$ is the approximation residual to be minimized.
Due to the time-varying nature of $\lambda_d$ in~\eqref{Eqn:lam_d}, we adopt the following quadratic performance index $\|\epsilon_{c}(\tau)\|^2+\|\epsilon_{c,N}\|^2$,
$\tau\in[k,k+N-1]$, $\epsilon_{c,N}=\lambda_d(\hat {s}(k+N))-\hat\lambda(\hat {s}(k+N))$. The weighting matrix is usually updated according to the gradient descend method, however this might lead to the constraint nonsatisfaction of $\hat s(\tau)$ in the learning process. Notice that, it is almost impossible to exactly design a barrier function that can represent the state constraint. Instead, we use the barrier function on $\hat\lambda$ to guide the weight update for $\hat W_c$. The problem left now is how to find an upper bound for $\lambda(\hat {s}(\tau))$ induced by the safety constraints for $\tau\in[k,k+N-1]$. %As
To proceed, in view of Assumption~\ref{assu:system}.\ref{assu:a_inverse}, to ensure the constraint on~$\hat \lambda$, the corresponding feasible state region $\mathcal{S}^j\in \mathcal{S}$, $j\leq N$ can be found at $\tau=k+j$ starting from the terminal set $\mathcal{S}^N=\mathcal{S}_f$, see~\cite{scibilia2011feasible}. In view of~\eqref{Eqn:lam_d}, also considering $B(0)=0$ and $B(\hat {s}(\tau))\leq \hat {s}(\tau)^{\top}H_{\hat{s}}\hat {s}(\tau)$, it holds that $\|\frac{\partial B(\hat {s}(\tau))}{\partial \hat {s}(\tau)}\|_{\infty}\leq\|2H_{\hat {s}}\hat {s}(\tau)\|_{\infty}$. Denoting $\Gamma^j=\max_{\hat {s}(\tau)\in\mathcal{S}^j}\|2H_{\hat{s}}\hat {s}(\tau)\|_{\infty}$, one promptly has $\frac{\partial B(\hat {s}(\tau))}{\partial \hat {s}(\tau)}\in\mathcal{B}_{\Gamma^j}(0)$, the region which $\lambda(\hat {s}(k+j))$, $j<N$ lies in, can be defined and represented as $\Lambda^j=\mathcal{B}_{\Gamma^j}(0)\oplus 2\bar Q\mathcal{S}^j\oplus A^\top \Lambda^{j+1}$. As in fact $\frac{\partial B(\hat {s}(\tau))}{\partial \hat {s}(\tau)}\rightarrow 0$ for $\hat {s}(\tau)\in Int(\mathcal{S}^j)$, the effect caused by the barrier function $B(\hat {s}(\tau))$ to the scale of $\Lambda^j$ is limited. Therefore, we simplify the calculation formula of $\Lambda^j$ as $\Lambda^j=2\bar Q\mathcal{S}^j\oplus A^\top \Lambda^{j+1}$. The details on how to compute $\Lambda^j$ is described in Algorithm~\ref{Atm:1}. Thus, the regularized cost to be minimized by the critic network is
\begin{equation}\label{Eqn:ec}
\delta_c(\tau)=\|\epsilon_{c}(\tau)\|^2+\|\epsilon_{c,N}\|^2+\bar\mu (B(\hat\lambda(\hat {s}(\tau)))+B_f(\hat\lambda(\hat s(k+N)))),
\end{equation}
where for each $j=1,\cdots,N$, $B(\cdot)$ and $B_f(\cdot)$ is the relaxed re-centered barrier functions for constraints $\hat\lambda(\hat{s}(k+j))\in\Lambda^j$ and $\hat\lambda(\hat{s}(k+N))\in\Lambda^N$.
At any time instant $\tau\in[k,k+N-1]$, the weight update is given as
\begin{equation}\label{Eqn:wc}
\hat W_{c}(\tau+1)={\hat W_{c}(\tau)} - {\beta}\frac{\partial\delta_c(\tau)}{\partial \hat W_{c}(\tau)}.
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
where $\beta$ is the learning rate.
Along the same line, $\hat u^{\ast}$ can be represented as $$\hat u^{\ast}(\tau)=W_{a}^{\top}h_a(\hat{s}(\tau),\tau)+\bar{\epsilon}_a(\tau)$$
where $W_{a}\in\mathbb{R}^{N_u\times m}$ is the weighting matrix, $h_a$ is a vector whose entries are basis functions. It is assumed that $\|W_a\|\leq W_{a,m}$, $\|h_a\|\leq h_{a,m}$, $\|\bar \epsilon_a\|\leq \bar \epsilon_{a,m}$. $\bar{\epsilon}_a\rightarrow 0$ as $N_u\rightarrow +\infty$.
To define the actor network, in view of ~\eqref{Eqn:hjb}, we define the desired control action $ u_d(\tau)$, $\tau\in[k,k+N-1]$ as
\begin{equation}\label{Eqn:act-d}
%\begin{align}
\mu\frac{\partial B( u_d(\tau))}{\partial u_d(\tau)}+2Ru_d(\hat{s}(\tau))=-B^\top \hat\lambda(\hat{s}(\tau+1))
%&\mu\frac{\partial B(e_{{s}_d}(k))}{\partial \hat{s}_d(k)}+2\bar Q\hat{s}_d(k)=-A^\top \hat{\lambda}(\hat{s}(k+1)),\label{Eqn:xi-d}
%\end{align}
\end{equation}
%The situation is similar in that of~\eqref{Eqn:xi-d} for $\hat {s}_d(k)$.
The estimate of $u_d$ is then defined as
\begin{equation}\label{Eqn:actor}
%\begin{align}
\hat u_d(\tau)=\hat W_{a}^{\top}h_a(\hat{s}(\tau),\tau), \tau\in[k,k+N-1]
%&\hat {s}(k)=W_{a{s}}(k)^{\top}h(k),\label{Eqn:actor2}
%\end{align}
\end{equation}
where $\hat W_{a}$ is the estimate of $W_{a}$.
Note that, usually the actor network is used for approximating the desired decision variables $u_d(\tau)$ via minimizing the error of the outputs with respect to their desired ones in quadratic cost form, which is however not suitable in this case due to for instance, the left-side of~\eqref{Eqn:act-d} being composed by $u_d$ itself multiplied with a constant matrix and the partial gradient of the barrier function $B(u_d)$ with respect to $u_d$, where the entries of $u_d$ show explicitly in the denominator term. For this reason, provided the structure of the actor network like~\eqref{Eqn:actor}, different from that of the classic actor, the objective here concerned is to regard the left-side term of~\eqref{Eqn:act-d} as a whole to be estimated. To this end, denoting $G_u(u_d(\hat{s}(\tau)))=\mu\frac{\partial B(u_d(\hat{s}(\tau)))}{\partial u_d(\hat{s}(\tau))}+2Ru_d(\hat{s}(\tau))$, $ \tau\in[k,k+N-1]$, %$G_{{s}}(\hat{s}_d(k))=\mu\frac{\partial B(e_{{s}_d}(k))}{\partial \hat{s}_d(k)}+2\bar Q\hat{s}_d(k)$,
the estimated value $\hat G_u(\hat u_d(\tau))$ is generated by the output of the actor network such that $ G_u(u_d(\hat{s}(\tau)))=\hat G_u(\hat u_d(\hat{s}(\tau)))+\epsilon_{a}(\tau)$, %$ G_{{s}}(\hat{s}_d(k))=\hat G_{{s}}(\hat{s}(k))+\epsilon_{a{s}}$
where $\epsilon_{a}$ is the approximation residual.
%that can be reduced by increase respectively $N_u$ and $\bar n$ and approaches null when $N_{u}$, $\bar n$ go to infinity.
At each time instant $\tau\in[k,k+N-1]$, the residual $\epsilon_{a}(\tau)$ needs to be minimized, typically formed in the quadratic cost function as $\|\epsilon_{a}(\tau)\|^2$, where $Q_a$ is a positive-definite matrix.
Also, a barrier function of $\hat u_d(\tau)$ is included in the cost function for control constraint satisfaction. The regularized loss function is given as
\begin{equation}\label{Eqn:loss_a}
%\begin{align}
\delta_{a}(\tau)=\|\epsilon_{a}(\tau)\|^2+\bar\mu B(\hat u_d(\tau))\\
%&\delta_{a{s}}(k)=\|\epsilon_{a{s}}(k)\|_{Q_{a{s}}}^2+\mu B(e_{{s}}(k)).
%\end{align}
\end{equation}
%where $B(\hat u_d)=0$ for $\hat u_d\in \hat{\mathcal{U}}$, and $B(\hat u_d)\rightarrow \infty$ for $\hat u_d\in \partial{\hat{\mathcal{U}}}$.
At any time instant $\tau\in[k,k+N-1]$, the weight is updated according to the following rule
\begin{equation}\label{Eqn:wa}
%\begin{align}
\hat W_{a}(\tau+1)={\hat W_{a}(\tau)} - {\gamma}\frac{\partial\delta_{a}(\tau)}{\partial \hat W_{a}(\tau)}
%&W_{a{s},i+1}(k)={W_{a{s},i}(k)} - {\gamma_i^{{s}}}\frac{\partial \hat {s}(k)}{\partial W_{a{s},i}(k)}\left(\frac{\partial \delta_{a{s}}(k)}{\partial \hat {s}(k)}\right)^{\top},\label{Eqn:wxia}
%\end{align}
\end{equation}
where $\gamma$ is the learning rate.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{algorithm}[h]
\caption{Computation of the feasible sets $\Lambda^j$, $\forall j=1,\cdots,N$. }\label{Atm:1}
%\KwData{this text}
%\KwResult{how to write algorithm with \LaTeX2e }
\For{ $j\leftarrow N$ \KwTo $1$}{
\If {$j=N$}
{${\mathcal{S}}^j={\mathcal{S}_f}$\;}
${\mathcal{S}}^j=A^{-1}(\mathcal{S}^{j+1}\oplus B(-\hat{\mathcal{U}}))$\;
${\mathcal{S}}^j={\mathcal{S}}^j\cap\mathcal{S}$\;
}
\For{ $j\leftarrow N$ \KwTo $1$}{
\If {$j=N$}
{ $\Lambda^j=2P\bar {\mathcal{S}}_f$\;}
$\Lambda^j=2\bar Q\mathcal{S}^j\oplus A^\top \Lambda^{j+1}$\;
}
\end{algorithm}
\subsection{Safety and robustness of the closed-loop system}\label{sec:con-actor-critic}
Different from classical actor-critic learning, in the proposed approach, the new type of regularization using the logarithmic barrier function is built, also the cost function to be optimized in the actor is constraint-inclusive. Hence, standard convergence proof, see~\cite{dong2018functional,Xu-finitehorizonadp}, is not applicable in this case. In that follows, we prove the boundedness of the actor-critic network and give the point-wise convergence condition in each prediction horizon.
\begin{theorem} [Boundedness and convergence conditions of the regularized actor-critic learning in each prediction horizon]\label{the:ac-conver}\hfill
Under the learning laws~\eqref{Eqn:wc} and~\eqref{Eqn:wa}, if the learning rates $\gamma$ and $\beta$ %and tuning parameters $\alpha_c$ and $\alpha_a$
are selected such that the matrices defined in Appendix \ref{sec:proof_2} satisfy
$\bar G_{c1}, \bar G_{a1}>0,$ the following results can be stated.
\begin{enumerate}[1.]
\item The variables
$\epsilon_c(\tau)$, $\epsilon_{c,N}$, and $\hat u^{\ast}(\tau)-\hat u_d(\tau)$ converge to the corresponding finite regions i.e.,
\begin{equation}\label{Eqn:ac-uub}
\begin{array}{ll}
\|\epsilon_c(\tau)\|,\|\epsilon_{c,N}\|\leq \eta_{c}\\
\|\hat u^{\ast}(\tau)-\hat u_d(\tau)\|\leq \eta_{a},
\end{array}
\end{equation}
$\tau \in[k,k+N-1]$, as $N\rightarrow+\infty$, where $\eta_{c}$ and $\eta_{a}$ will be defined in~\eqref{Eqn:est-uub}.
\item As an additional result, under~\eqref{Eqn:ac-uub}, if $\bar\epsilon_{a}$ and $\bar \epsilon_{c}$ steer to zero, $\bar \mu=0$, and the matrix $\sigma_{max}(A)< 1$, then the weights of the actor and critic converge to their optimal ones, i.e.,
\begin{equation}\label{Eqn:ac-con}
\begin{array}{ll}
\hat W_c(\tau)\rightarrow W_c\\
\hat W_a(\tau)\rightarrow W_a,
\end{array}
\end{equation}
as $\tau\rightarrow N$ and $N\rightarrow+\infty$.
\end{enumerate}
\end{theorem}
Proof: please refer to Appendix~\ref{sec:proof_2}. \hfill$\square$
\begin{remark}
Theorem~\ref{the:ac-conver} implies a sufficient large choice of $N$, denoted as $N=N_l$, required for verifying condition~\eqref{Eqn:ac-uub}. However, an outer iterative loop with a length of $\bar i$ can be adopted, which allows to choose a reasonable value $N=\bar n$. Condition~\eqref{Eqn:ac-uub} can be fulfilled as long as $\bar i \bar n\geq N_l$.
\end{remark}
\begin{definition} [Safe control policy] For a generic time $k$, let $u(k)={\hat u}_d(k|k)+K(x(k)-\hat s(k|k))$, associated with a solution of~\eqref{Eqn:optimizL}, i.e., $\hat s(k|k),{\hat u_d}(k|k),\cdots,{\hat u_d}(k+N-1|k)$, being such that $\hat s(k+N|k)\in\mathcal{S}_f$, $\hat s(k+j|k)\in\mathcal{S}$, $\hat u_d(k+j|k)\in\hat {\mathcal{U}}$ for all $j\in[k,k+N-1]$. We call $u(k)$ a safe control policy such that $(x(k),u(k))\in\mathcal{X}\times\mathcal{U}$.
\end{definition}
\begin{definition}[Monotonic decreasing of value function]
For a generic time $k$, we say the learned value function $V_b(k|k)$ associated with the sequence $\hat s(k|k),{\hat u_d}(k|k),\cdots,{\hat u_d}(k+N-1|k)$ satisfying monotonic decreasing property if $V_b(k|k)\leq V_b(k-1|k-1)$, where $V_b(k-1|k-1)$ is associated with the adopted solution $\hat s(k-1|k-1),{\hat u_d}(k-1|k-1),\cdots,{\hat u_d}(k+N-2|k-1)$.
\end{definition}
Under Theorem~\ref{the:ac-conver}, it is possible to prove step-by-step the safety and robustness property of the learning-based closed-loop control system. As a first step, we give the safety condition of the dr-LPC at the initial time, then we prove the recursive safety property for any future time instant. With this result, we can finally prove the robustness of the closed-loop system.
\begin{theorem}[Safety and robustness of the closed-loop system]\label{the:2}\hfill
Under Assumption~\ref{assu:system}, given any finite initial condition $x(0)\in\mathcal{X}$, it can be stated as follows.
\begin{enumerate}[1.]
\item The terminal state $\hat s(N|0)$ with the dr-LPC is capable to reach the terminal region ${\mathcal{S}}_{f}$ at time $k=0$, if $N$ is such that optimization problem~\eqref{Eqn:optimizL} is feasible at $k=0$ and
\begin{equation}\label{Eqn:feasi} \sqrt{N}\|\sum_{j=1}^{N-j-1}A^jB\|\eta_a\leq\frac{1-\rho}{\sigma_{max}(Z)}
\end{equation}
\item If at any time $k$, $u(k)={\hat u}_d(k|k)+K(x(k)-\hat s(k|k))$ is a safe control policy associated with the sequence $\hat s(k|k),\hat u_d(k|k),\cdots,\hat u_d(k+N-1|k)$. Then $u(k+1)={\hat u}_d(k+1|k)+K(x(k+1)-\hat s(k+1|k))$ is a backup safe one at time $k+1$ associated with the sequence $\hat s(k+1|k),\hat u_d(k+1|k),\cdots,\hat u_d(k+N-1|k),K\hat s(k+N)$.
\item Moreover, let at time $k$ the applied control policy be defined as
\begin{equation}\label{Eqn:safe-u}
u(k)=\left\{\begin{array}{lll}
{\hat u}_d(k|k)+K(x(k)-\hat s(k|k))& {\rm case \ 1}\\
{\hat u}_d(k|k-1)+K(x(k)-\hat s(k-1|k))& {\rm otherwise},
\end{array}\right.
\end{equation}
where case 1 corresponds to the safety and monotonic decreasing property of the learned control and value function being verified. Then, the state $x(k)$ of the nonlinear system~\eqref{Eqn:non-model} with dr-LPC converges to the set $\mathcal{O}$ (see~\eqref{Eqn:all-residual}) as $k\rightarrow +\infty$, i.e., $$x(k)\rightarrow {\mathcal{O}}\ \text{as} \ k\rightarrow\infty.$$
\end{enumerate}
\end{theorem}
Proof: please refer to Appendix~\ref{appen:the3}.
\hfill$\square$
\begin{remark}
The first point in Theorem~\ref{the:2} implies a trade-off between $N$ and $N_u$. To ensure safety at the initial time instant, one can choose a larger value of $N_u$ to obtain a smaller approximating error bound $\eta_a$ or shrink the prediction horizon $N$. The second point suggests a safe backup policy to be applied in case when online learning failures may occur, while the last point gives the suitable control policy design for guaranteeing the closed-loop robustness.
\end{remark}
\begin{algorithm}[h]
%\KwData{this text}
% \KwResult{how to write algorithm with \LaTeX2e }
Given $x(0)\in\mathcal{X}$, compute ${s}(0)$ with~\eqref{Eqn:initial-con}\;
Set $\bar i$, %$B_{max}(\hat u_d)$, $B_{max}(\hat u_d)$, $B_{max}(\hat{s})$,
initialize $\gamma$, $\beta$, $\kappa$, select $W_{a}(\tau)$ and $W_{c}(\tau)$ such that $\delta_{a}<B_{max}(\hat u_d)$, and $\delta_c<B_{max}(\hat\lambda)$, $k=1$, $i=0$\;
\While{ $k\leq N_{sim}$}{
\While{$i\leq \bar i$}{
\For{$\tau=k$ \KwTo $k+N-1$}{
obtain $\hat u_d(\hat{s}(\tau))$ with~\eqref{Eqn:actor} and calculate the state evolution $\hat {s}(\tau+1)$ with~\eqref{Eqn:unpert}\;
compute $\hat\lambda(\hat{s}(\tau+1))$ with~\eqref{eqn:critic} and
$\lambda_d(\hat {s}(\tau))$ with~\eqref{Eqn:lam_d} for $\tau\leq k+N-1$\;
compute $u_d(\hat{s}(\tau))$ with~\eqref{Eqn:act-d}\;
update $W_{c}(\tau+1|k)$ with~\eqref{Eqn:wc} and $W_{a}(\tau+1|k)$ with~\eqref{Eqn:wa};
}
\eIf{$\{k=1\}$ $\vee$ \{safety verified and value function monotonicity detected\}}{generate $\overrightarrow {\hat u_d}(k:k+N-1|k)$ and break;}{ the sequence $\overrightarrow {\hat u_d}(k:k+N-1|k)=[\overrightarrow {\hat u_d}(k-1:k+N-2|k-1), K\hat{s}(k+N-1)]$ and break\;}%}
\If{$\sum_{i=k+1}^N\|\hat W_{\ast}(i|k)-\hat W_{\ast}(i-1|k)\|\leq\varepsilon_W $, $\ast=a,c$ }{break;}
$i\leftarrow i+1$
}
generate $u(k)=\hat u_d(k|k)+K({s}(k)-\hat{s}(k|k))$\;
update $\hat{s}(k+1)$, ${s}(k+1)$, $\hat x(k+1)$, and $x(k+1)$\;
$k\leftarrow k+1$.
}
\caption{Implementation of the dr-LPC. }\label{Atm:2}
\end{algorithm}
\subsection{Implementing steps of the dr-LPC}
The implementation of the dr-LPC requires a number of steps of off-line designs as follows:
\begin{enumerate}[a.]
\item compute the model matrices $A$, $B$, $C$, and $D$ with~\eqref{Eqn:appro_K} and~\eqref{Eqn:appro_C} in~Appendix~\ref{app-model};
\item determine the sets $\mathcal{W}$ and $\mathcal{V}$ according to~Appendix~\ref{sec:app-b};
\item compute the terminal constraint ${\bar{\mathcal{S}}_f}$ and the set $\bar{\mathcal{O}}$;
\item compute $ {\mathcal{S}_f}$ under the light of~\eqref{Eqn:feasi}, validate $\mathcal{D}$ and ${\mathcal{O}}$ according to~Appendix~\ref{sec:app-b};
\item obtain $\Lambda^i$ for $i=1,\cdots,N$ using Algorithm~\ref{Atm:1}.
\end{enumerate}
The main online implementation steps of the dr-LPC are described in Algorithm~\ref{Atm:2}.
\begin{remark}
The computational complexity of the proposed dr-LPC is roughly $O(N(pmN_u^2+N_c\bar n^2+N_c^2\bar n))$. In contrast, the interior-point method takes $O(N(\bar n + m)^3
)$ operations for systems with block-diagonal structure and $O(N^3(\bar n+ m)^3
)$ operations if the block-diagonal structure is not exploited.
\end{remark}
\section{Simulation results}\label{sec:simulation}
\subsection{Learning control on a Van der Pol oscillator}
Consider the problem of regulating a Van der Pol oscillator. The continuous-time system model is given as
\begin{equation}\label{Eqn:ct}
\begin{bmatrix}
\dot x_1\\
\dot x_2
\end{bmatrix}=\begin{bmatrix}
x_2\\
1-x_1^2x_2-x_1+u
\end{bmatrix}+w_o
\end{equation}
where $x_1$ and $x_2$ represent the position and speed respectively, while $u$ is the force input, the disturbance $\|w_o\|_{\infty}\leq 0.08$. Denoting $x=(x_1,x_2)$, the state and control are limited as
$
-(2.5m,2.5m/s)\leq x\leq(2.5m,2.5m/s),\
-10m^2/s\leq u\leq 10m^2/s.
$ The control objective is to drive $x$ to the origin. In order to implement the proposed learning predictive controller, first model~\eqref{Eqn:ct} has been discretized with a sampling period $T=0.025s$ to obtain~\eqref{Eqn:non-model}.
\begin{comment}
, that is
\begin{equation}\label{Eqn:dt}
\left\{\begin{array}{l}
x_1(k+1)=x_1(k)+T (x_2(k)+w(k))\\
x_2(k+1)=x_2(k)+T((1-x_1(k)^2x_2(k)-x_1(k)+u(k))+w(k))
\end{array}\right.
\end{equation}
\end{comment}
The data sets of $(z,z^+)$ in~\eqref{Eqn:appro_K} with $M=4\cdot 10^5$ are collected with random policy, and the samples that violate the state and control constraints have been removed. The polynomial kernel functions are selected to construct the lifted observable state variables, i.e., $\Phi\left(x,\bar n\right)=(x,\,x_1^2,\,x_1^2x_2)$, where $\bar n=4$. The weighting scalar $\theta$ has been chosen as $100$. The parameters of the linear predictor has been computed according to~\eqref{Eqn:appro_K} and~\eqref{Eqn:appro_C}.
The penalty matrices $Q$ and $R$ have been selected as $Q=I_2$, $R=0.01$. The penalty scalar on the barrier function is $\mu=\bar \mu=0.001$. The relaxed factor $\kappa$ has been set as 0.1. With this value, the matrix $H$ can be obtained, according to Lemma~\ref{lemma:recentered}.\ref{lemma:relax}. The terminal penalty matrix can be calculated with~\eqref{Eqn:qn_ph}.
The set $\mathcal{W}$, $\mathcal{V}$, and $\mathcal{D}$ have be empirically obtained according to Appendix~\ref{sec:app-b}. % as $\mathcal{W}=\emptyset$, $\mathcal{V}=\{v|\|v\|\leq0.001\}$, and $\mathcal{D}=\{d|\|d\|\leq0.01\}$.
Then, a possible conservative choice of the robust invariant set has been obtained with~\eqref{Eqn:e} according to ~\cite{rawlings2009model}. The constraint on the costate in Algorithm~\ref{Atm:1} has been computed using Ellipsoidal toolbox described in~\cite{kurzhanskiy2006ellipsoidal}. The terminal constraint $\bar {\mathcal{S}}_f$ has been computed according to the algorithm described in~\cite{farina2018hierarchical}.
The prediction horizon $N=10$. The basis functions have been chosen as
\begin{equation}\label{Eqn:h_ac-time}
h_a=h_c=(\Phi\left(x,\bar n\right),\nu \tau,\nu \tau^2)
\end{equation}
where $\nu=0.01$.
For comparison, the dr-MPC presented in Section~\ref{sec:dr-MPC}, which is an extension of~\cite{korda2018linear}, and an incremental infinite-horizon DHP described in~\cite{Werbos1992ApproximateDP} using the real dynamics have been designed to control the Van der Pol oscillator~\eqref{Eqn:ct}. In dr-MPC, the stage cost $r_c=x^{\top}Qx+Ru^2$, the terminal penalty matrix $P_c= P$, and the terminal constraint has been computed using the original state and control constraints according to~\cite{farina2018hierarchical}. The DHP has adopted the same stage cost as the proposed controller. All the three controllers have been implemented with initial condition $x(0)=(0.1,-0.1)$. %All the simulation experiments have been performed on a
Also, the weighting matrices $\hat W_a$ and $\hat W_c$ in the proposed controller and the DHP have been initialized as uniformly distributed random values. The simulations with the dr-LPC and the dr-MPC have been performed within Yalmip toolbox~\cite{Lofberg2004} installed in Matlab 2019a environment, in a Laptop with Intel Core i7-8550U CPU @1.8GHz running with Windows 10 operating system. The simulation results computed with the dr-LPC, dr-MPC, and DHP are reported in Figure~\ref{fig:pro_mpc_com}. It can be seen that the dr-LPC significantly outperforms the DHP. The control performance with the dr-LPC is slightly better than that with the dr-LPC in terms of state regulation but experiences a larger variance due to the random initialization of dr-LPC.
To clearly compare the control performance, the average cumulative square errors of the regulating cost and the input energy of the fifty tests $J=1/50\sum_{i=1}^{50}\sum_{j=1}^{N_{sim}}\|x_i(j)\|_Q^2+\|u_i(j)\|^2_R$, $J_x=1/50\sum_{i=1}^{50}\sum_{j=1}^{N_{sim}}x_i(j)^{\top}x_i(j)$, and $J_u=1/50\sum_{i=1}^{50}\sum_{j=1}^{N_{sim}}u_i(j)^2$, where the simulation length $N_{sim}=320$, have been collected and listed in Table~\ref{tab:Tab_com1}. It can be shown that the average cost $J_x$ with the dr-LPC is smaller than that with the dr-MPC, but the average energy consumed by the former being $10.6$ is higher than that by the latter, which is $2.37$. It is noted in Table~\ref{tab:Tab_com1} that, the average online computational time of the dr-MPC, i.e., $0.128$$s$, is greater than the sampling period $T_s$. For this reason, the MPC might not be applicable in the adopted environment. The dr-LPC exhibits advantageous point in this respect, since the online computational time is greatly smaller than that with the MPC. The average costs are greatly larger than that with the dr-LPC and dr-MPC. Moreover, the success rates computed by 50 trials and with different initial conditions are listed in Table~\ref{tab:Tab_com1}, which shows that the average success rate with the dr-LPC is $50$ over $50$, which implies the verification of control safety. However, the average success rate with the DHP is $9$ over $50$ with $x(0)=(0.1,-0.1)$ and $5$ over $50$ collected with 10 different initial conditions. Specifically, a display of fifty tests with dr-LPC using $x(0)=(2.4,-2)$ (that is close to the set boundary) is in Figure~\ref{fig:pro_mpc_con}, where it can be seen that the control constraint is activated and the state constraint can be verified.
\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{"figure/LPCandMPCandDHP"}
\caption{The values of the state and control of the Van der Pol oscillator with dr-LPC, dr-MPC, and DHP: the filled areas are the corresponding variance intervals of fifty tests. The data sets of successful trails with DHP are used, while those of failed trials are disregarded.}
\label{fig:pro_mpc_com}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.48\textwidth]{"figure/LPC_con"}
\caption{Safety tests of the Van der Pol oscillator with dr-LPC using initial condition $x(0)=(2.4,-2)$: the filled areas are the corresponding variance intervals of fifty tests.}
\label{fig:pro_mpc_con}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{"figure/j"}
\caption{The variation of $J_x$ and $J_u$ of the Van der Pol oscillator with dr-LPC and dr-MPC: the filled areas are the corresponding variance intervals of fifty tests.}
\label{fig:trend_cost}
\end{figure}
\begin{table*}[h!tb]
\centering \caption{Comparison with fifty tests of the Van der Pol oscillator using random weights: A.c.t. stands for ``average computational time", S.r.w represents for ``Success rate with".}
\label{tab:Tab_com1}
\vskip 0.2cm
%
\scalebox{1}{
% The {|c|c|c|c|c|} define the number of columns.
% c means centered
% | defines a vertical line between two columns
\begin{tabular}{ccccccc}
\toprule
Algorithm & $J_x$ & $J_u$& $J$& A.c.t & S.r.w $x(0)=(0.1,-0.1)$&S.r.w (10 different $x(0)$s) \\
\midrule
dr-LPC ($1^{st}$ run) & 0.48& 10.6&$0.586$&{0.004}$s$ &{50/50}&{50/50}\\
\midrule
dr-LPC ($5^{th}$ run) & \textbf{0.40}& \textbf{1.95}&\textbf{0.42}&{0.004}$s$ &50/50&50/50\\
\midrule
dr-MPC & 0.56 & 2.37&$0.584$&0.128$s$&$-$&$-$\\
\midrule
DHP & $38.7$ & $6.9\cdot 10^3$ &$107.7$ &\textbf{0.0001}$s$&9/50&5/50\\
\bottomrule
\end{tabular}
}
\end{table*}
The control performance with the dr-LPC can be persistingly improved in an iterative way, where the converged weights of $\hat W_a$ and $\hat W_c$ in the previous run is used for the initialization in the subsequent one. To show the persisting learning capability, starting with a random initialization condition, the online control process has been repeated 5 times with the previous converged weights used for the initialization. The variation of the values of $J_x$ and $J_u$ are reported in Figure~\ref{fig:trend_cost}. The results indicate that in view of the initialization strategy adopted, the values of $J_x$ as well as $J_u$ have been reduced significantly in the second episode and then remain almost constant for the rest of episodes. The reason for the fast learning ability of the proposed dr-LPC is the closed-loop nature due to the receding horizon strategy used. Indeed, the improvement of weights has proceeded in the iterative steps of each prediction horizon and between different predictions and multiple runs. Taking the fifth run as an example, the average values of the fifty tests obtained as $J_x=0.4$ and $J_u=1.95$, are both smaller than that that with MPC, which are $0.56$ and $ 2.37$, see Table~\ref{tab:Tab_com1}. This reveals that, after a small number of runs, the proposed dr-LPC can exhibit better control performance to the dr-MPC.
The comparison of the dr-LPC using the result of the fifth run's with the MPC is presented in Figure~\ref{fig:pro_mpc_com}, which shows that both control systems experience a similar transient period, except for the control action with the dr-LPC being slightly overshot. %\emph{Impacts of $\nu$ in~\eqref{Eqn:h_ac-time} on the performance of dr-LPC}
In addition, the impact of $\nu$ in~\eqref{Eqn:h_ac-time} on the performance of dr-LPC is also studied. Note that, the basis function in~\eqref{Eqn:h_ac-time} is defined to capture the time-varying nature of the value function in each prediction horizon. The detailed impact of time on the value function approximation is not clear to us. Hence, the parameter $\nu$, which scales the size of time in $h_c$ and $h_a$, has been chosen orderly from $1$ to $0.01$ so as to show how $\nu$ impacts the control performance. The cost comparison with different values of $\nu$, each choice being repeatedly performed under fifty tests, are listed in Table~\ref{tab:Tab_com2_rho}. The result shows that the choice $\nu=0.01$ results in the lowest average cost of $J_x$ and $J_u$, while the worst performance is achieved with $\nu=1$. An almost positive correlation can be concluded between the average costs and the choice of $\nu$.
\begin{table}[h!tb]
\centering \caption{Cost comparison with different choices of $\nu$ using random weights in dr-LPC.}
\label{tab:Tab_com2_rho}
\vskip 0.2cm
%
\scalebox{0.9}{
% The {|c|c|c|c|c|} define the number of columns.
% c means centered
% | defines a vertical line between two columns
\begin{tabular}{ccccccc}
\toprule
Choice of $\nu$ & 0.01& 0.02& 0.1 & 0.2& 0.5&1\\
\midrule
$J$ & 0.59 & 0.65&0.62 &0.65&0.67&1.31\\
\midrule
$J_x$ & 0.48 & 0.52&0.50 &0.52&0.59&1.2\\
\midrule
$J_u$ & 10.6 & 12.8&12.3&12.8&8.1&10.9\\
\bottomrule
\end{tabular}
}
\end{table}
\subsection{Learning control for an Inverted Pendulum}
\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{"figure/LPCandMPCandDHP_IP"}
\caption{The values of the state and control of the Inverted Pendulum with dr-LPC, dr-LPC, and DHP: the filled areas are the corresponding variance intervals of fifty tests. The data sets of successful trails with DHP are used, while those of failed trials are disregarded.}
\label{fig:pro_mpc_com_IP}
\end{figure}
\begin{table*}[h!tb]
\centering \caption{Comparison with fifty tests of the Inverted Pendulum with random initialization: A.c.t. stands for ``average computational time", S.r.w represents for ``Success rate with".}
\label{tab:Tab_com2}
\vskip 0.2cm
%
\scalebox{1}{
% The {|c|c|c|c|c|} define the number of columns.
% c means centered
% | defines a vertical line between two columns
\begin{tabular}{ccccccc}
\toprule
Algorithm & $J_x$ & $J_u$&$J$& A.c.t & S.r.w $x(0)=(0.1,0,0,0)$&S.r.w (10 different $x(0)$s) \\
\midrule
% dr-LPC ($1^{st}$ run) & \textbf{35.7}& 140&38.5&{0.006}$s$ &{50/50}&{50/50}\\
% \hline
dr-LPC & \textbf{39.6}& 134.5&\textbf{42.4}&{0.006}$s$ &50/50&50/50\\
\midrule
dr-MPC & 85.2 & \textbf{85.5}&86.9&0.182$s$&$-$&$-$\\
\midrule
DHP & $411$ & $254$ &416.1&\textbf{0.005}$s$&2/50&2/50\\
\bottomrule
\end{tabular}
}
\end{table*}
Consider also the problem of regulating an Inverted Pendulum. The continuous-time system model is:
\begin{equation}\label{Eqn:ct-ip}
\begin{bmatrix}
\dot x_1\\
\dot x_2\\
\dot x_3\\
\dot x_4
\end{bmatrix}=\begin{bmatrix}
x_2\\
\frac{3}{2l}(gsinx_1+ucosx_1)\\
x_4\\
u
\end{bmatrix}+w_o
%\left\{\begin{array}{l}
%\end{array}\right.
\end{equation}
where $x_1$, $x_2$, $x_3$, $x_4$ are the displacement and velocity of the rod, position and velocity of the car, $l=0.5$ $m$, $g=9.8$ $m/s^2$, $\|w_o\|_{\infty}\leq 0.5$. Denoting $x=(x_1,x_2,x_3,x_4)$, the state and control are limited as $-(0.25 rad,2 rad/s,1 m,2 m/s)\leq x\leq (0.25 rad,2 rad/s,1 m,2 m/s)$, $|u|\leq 10$$m/s^2$. The control objective is to drive $x$ to the origin. Similarly, the model~\eqref{Eqn:ct-ip} has been discretized with a sampling period $T=0.01s$ to obtain~\eqref{Eqn:non-model}. The data sets of $(z,z^+)$ in~\eqref{Eqn:appro_K} with $M=2\cdot 10^4$ are collected with random policy. The Gaussian kernel functions are selected according to~\cite{xu-skernel} to construct the lifted observable state variables with $\bar n=7$ and Assumption~\ref{assu:system}.\ref{assu:phi-ori}. The weighting scalar $\theta$ has been chosen as $1$. The parameters of the linear predictor has been computed according to~\eqref{Eqn:appro_K} and~\eqref{Eqn:appro_C}. The penalty matrices $Q$ and $R$ have been selected as $Q=I_2$, $R=0.02$. $\mu=\bar \mu=0.001$, $\kappa=0.1$. All the parameters have been computed similar to the previous simulation case. The prediction horizon $N=20$. The basis functions have been chosen as
$h_a=h_c=(\Phi,\nu \tau,\nu \tau^2)$
where $\nu=0.01$.
Similarly to the previous case, the comparison results with the dr-MPC and the DHP with the learned dynamics have been reported in Figure~\ref{fig:pro_mpc_com_IP} and~Table~\ref{tab:Tab_com2}.
It can be seen that the control performance with the learned dr-LPC %using random initialization and the $5^{th}$ run of dr-LPC
is better than that with the dr-MPC in terms of state regulation, but the input energy consumed is larger than that with the dr-MPC. The proposed dr-LPC still exhibits a significantly advantageous point than that with the dr-MPC in the respect of computational efficiency, since its online computational time is greatly smaller. In spite of the high computational efficiency with DHP, the control performance is the worst among all the controllers. Moreover, the average success rate computed by 50 trials and with different initial conditions are listed in Table~\ref{tab:Tab_com1}, which show that the average success rate with the dr-LPC is $50$ over $50$, while that with the DHP is $2$ over $50$ with 10 different initial conditions.
\section{Conclusions}
In this paper, we have proposed a data-driven robust learning-based predictive control scheme for unknown nonlinear systems with additive disturbance subject to safety constraints. Different from classical MPCs, the dr-LPC has been fully data-driven and in each prediction horizon, instead of computing a control sequence via solving the online optimization problem, we use an incremental DHP with an actor-critic structure to learn a near-optimal control policy. The resultant control consists of a successive time-varying functional and a static feedback term that are both output-dependent. As another contribution, the safety constraints have been explicitly dealt with in the dr-LPC. It is a standard capability in classic MPCs but a nontrivial task when the actor-critic learning network is used. To ensure safety constraints satisfaction, we utilize logarithmic barrier functions to construct a constraint-inclusive HJB equation and a regularized actor-critic learning structure.
The theoretical property of the dr-LPC has been analyzed. It has been founded that, the online safety can be verified as long as a feasible solution is detected at the initial time step. The conditions on the convergence of the actor-critic network as well as the robustness of the closed-loop system have been derived. Two simulation studies are reported, which shows the dr-LPC significantly outperforms the infinite-horizon DHP in terms of regulation and energy consumption in both cases. The state regulation cost with the dr-LPC is better than the dr-MPC in both cases, while requires more energy for controlling the Inverted Pendulum. Moreover, the online computational complexity is greatly smaller than that with the dr-MPC in both cases, which is a big advantageous point of the proposed dr-LPC. Future works will focus on the extension of the dr-LPC to continuous-time nonlinear systems.% possiby perturbed by exogenous distrubance.
\appendix
%
%
\section{}\label{appen-a}
\subsection{Proof of Theorem~\ref{the:rmpc}}\label{appen-the1}
Assume at any time instant $k$, an optimal decision sequence has been generated with~\eqref{Eqn:optimiz2}, i.e., $\hat s(k|k)$, $\overrightarrow{\hat u^o}(k:k+N-1|k)=\hat u^o(k|k),\cdots,\hat u^o(k+N-1|k)$,
associated with the optimal cost given as $V^o(k)$ such that $\hat {s}(k+N|k)\in\bar{\mathcal{S}}_f$. At the next time instant $k+1$, choose $\hat{s}(k+1|k+1)=\hat{s}(k+1|k)$, $\overrightarrow{\hat u^s}(k+1:k+N|k+1)=\hat u^o(k+1|k),\cdots,\hat u^o(k+N-1|k),K\hat{s}(k+N|k)$ as the solution. It can be seen that $C{s}(k+1)-C\hat{s}(k+1|k)\in\bar{\mathcal{O}}\ominus\mathcal{V}$, also $\hat s(k+j|k+1)\in\bar {\mathcal{S}}$ for $j\geq 1$ and $\hat{s}(k+N+1|k+1)\in\bar{\mathcal{S}}_f$. Hence, ~\eqref{Eqn:optimiz2} is feasible at time $k+1$ associated with a sub-optimal cost $V^s(k+1)$. The recursive feasibility of the robust MPC holds.
As for the convergence proof, note $V^o(k)\geq \|\hat {x}(k)\|_{ Q}^2$ for $\hat s\in \bar{\mathcal{S}}$. Also, iterating~\eqref{Eqn:LYA} leads to $V^o(k)\leq \hat s^{\top}P\hat s$ for $\hat s\in\bar{\mathcal{S}}_f$. Considering the fact that $V^o(k+1)\leq V^s(k+1)$, and in view of~\eqref{Eqn:LYA}, one has the monotonic property:
\begin{equation}\label{Eqn:V-MONO-r}
\begin{array}{ll}
V^o(k+1)- V^o(k)\leq-(\|\hat {x}(k)\|_{Q}^2+\|\hat u(k)\|_{R}^2)\\
\end{array}
\end{equation}
Then, from~\eqref{Eqn:V-MONO-r}, $V^o(k+1)- V^o(k)\rightarrow0$ as $k\rightarrow\infty$. Considering also $ V^o(k)-V^o(k+1)\geq (\|\hat {x}(k)\|_{ Q}^2+\|\hat u(k)\|_{R}^2$, hence $\hat u(k)\rightarrow 0$, $\hat x(k)\rightarrow 0$, as $k\rightarrow \infty$, in view of the positive-definiteness of $R$ and $Q$. Then it holds that, $x(k)\rightarrow\bar {\mathcal{O}}$, as $k\rightarrow \infty$. Consequently, $\hat{s}(k)\rightarrow 0$ due to $\Phi(0,\bar n)=0$ in view of Assumption~\ref{assu:system}.\ref{assu:phi-ori}, and $s(k)\rightarrow\bar{\mathcal{Z}}$ as $k\rightarrow \infty$.\hfill$\square$
\subsection{Proof of Lemma~\ref{lemma:recentered}}\label{appen-lamma}
Considering $\bar B(z)$ is differentiable and convex, and in view of theorem 2.1 in~\cite{nocedal2006numerical} that $\bar B(z)=\bar B(y)+\bigtriangledown\bar B(y)^{\top}(z-y)+\frac{1}{2}(z-y)^{\top}\bigtriangledown^2\bar B(y+tz-ty)(z-y)$ for some $t\in(0,1]$, the first condition follows via setting $y=0$. Also, one can write $B_c(z)=\frac{1}{2}z^{\top}\bigtriangledown^2\bar B(tz)z$. In the case that $\mathbb{Z}$ is polyhedral, in view of definition~\ref{defi:polytopic}, for $\bar\sigma\geq \kappa$, $\bigtriangledown^2\bar B(z)=\sum_{i=1}^p\|a_i\|_{\sigma_i^{-2}}\leq \sum_{i=1}^p\|a_i\|_{\kappa^{-2}}^2$. Considering also $\bigtriangledown^2\gamma(z,\bar\sigma)\leq\sum_{i=1}^p\|a_i\|_{\kappa^{-2}}$, then $B(z)\leq z^{\top}Hz$ is verified.\hfill $\square$
\subsection{Proof of Theorem~\ref{the:ac-conver}}\label{sec:proof_2}
The proof of Theorem~\ref{the:ac-conver} lies in the following intermediate results.
\begin{proposition}\label{prop:w}
Any feasible choices of $\hat W_c(\tau)$ and $\hat W_a(\tau)$ such that $\delta_{a}(\tau)<B_{max}(\hat u_d)$, and $\delta_{c}(\tau)<B_{max}(\hat\lambda)$
imply that $\|\hat W_{a}(\tau)\|_{\infty}<\infty$, $\|\hat W_{c}(\tau)\|_{\infty}<\infty$, $\tau\in[k,k+N]$.
\end{proposition}
Proof of Proposition~\ref{prop:w}
%We first show that %$\forall \tau\in[k,k+N-1]$,
%$\|\hat W_{\ast}(\tau)^{\top}\|_{\infty}<\infty$, where $\ast=a,c$.
As $\delta_{a}(\tau)<B_{max}(\hat u_d)$ implies that $\hat u_d(\tau)=\hat W_{a}(\tau)^{\top}h(\tau)\leq \rho_{\hat u}$, where $\rho_{\hat {u}}$ is such that
${\mathcal{\hat{U}}}\subseteq\mathcal{B}_{\rho_{\hat{u}}}(0)$. Then $h(\tau)^{\top}\hat W_{a}(\tau)\hat W_{a}(\tau)^{\top}h(\tau)\leq \sigma_{max}^2\left(\sqrt{\hat W_{a}(\tau)\hat W_{a}(\tau)^{\top}}\right) h(\tau)^{\top}h(\tau)\leq\rho_{\hat {u}}^2$, which leads to $\|\hat W_{a}(\tau)^{\top}\|\leq\frac{\rho_{\hat {u}}}{\sigma_{min}\left(h(\tau)^{\top}h(\tau)\right)}<\infty$. Along the same line, letting $\rho_{\hat {\lambda}(j)}$ be such that
$\Lambda^j\subseteq\mathcal{B}_{\rho_{\hat{\lambda}(j)}}(0)$, one has $\|\hat W_{c}(\tau)^{\top}\|\leq\frac{\rho_{\hat{\lambda}(j)}}{\sigma_{min}\left(h(k+j)^{\top}h(k+j)\right)}<\infty$, $j=1,\cdots,N-1$.
In view of norm equivalence property, it holds that $\|W_{c}(\tau)\|,\ \|W_{a}(\tau)\|< +\infty$.
%Recalling the fact that, given any matrix $\Gamma\in\mathbb{R}^{m\times n}$, $\|\Gamma\|_{1}\leq \sqrt{m}\|\Gamma\|$ and $\|\Gamma\|_{1}= \|\Gamma^{\top}\|_{\infty}$, it follows that $\|\hat W_{a}(\tau)\|_{\infty}<\infty$, as well as $\hat W_{c}(\tau)$.
\hfill $\square$
Proof of Theorem~\ref{the:ac-conver}
1) Consider the Lyapunov function
$$
\begin{array}{ll}
L(\tau)=L_a(\tau)+L_c(\tau)\\
\end{array}
$$
where $L_a=\frac{1}{2}tr(\tilde W_a^{\top}\tilde W_a)$, $\tilde W_a=W_a-\hat W_a$, $L_c=\frac{1}{2}tr(\tilde W_c^{\top}\tilde W_c)$, $\tilde W_c=W_c-\hat W_c$. In view of~\eqref{Eqn:wa} and~\eqref{Eqn:wc}, the difference of $L$ writes
\begin{equation}\label{Eqn:deltaL}
\Delta L(\tau+1)=\Delta L_a(\tau)+\Delta L_c(\tau)
\end{equation}
where
\begin{subequations}\label{eq:output_r}
\begin{align}
\Delta L_c(\tau)=\frac{1}{2}tr(2\beta\nabla\delta_{c}(\tau)^{\top}\tilde W_c(\tau))+{\beta}^2\nabla\delta_{c}(\tau)^{\top}\nabla\delta_{c}(\tau))\label{Eqn:deltaL_c}\\
\Delta L_a(\tau)=\frac{1}{2}tr(2\gamma\nabla\delta_{a}(\tau)^{\top}\tilde W_a(\tau))+{\gamma}^2\nabla\delta_{a}(\tau)^{\top}\nabla\delta_{a}(\tau)),\label{Eqn:deltaL_a}
\end{align}
\end{subequations}
where $\nabla\delta_{c}=\partial\delta_{c}/\partial \hat W_{c}$ and $\nabla\delta_{a}=\partial\delta_{a}/\partial \hat W_{a}$.
To compute $\Delta L_c$, in view of~\eqref{Eqn:ec}, one writes
\begin{equation}\label{Eqn:partial error c}
\nabla\delta_{c}(\tau)=\frac{\partial tr(\|\epsilon_{c}(\tau)\|^2+\|\epsilon_{c,N}\|^2+\bar\mu (B(\hat\lambda(\tau))+B_f(\hat\lambda))}{\partial \hat W_{c}}
\end{equation}
In view of the definition of $\lambda_d$ and $\lambda^{\ast}$, it holds that
$$\begin{array}{ll}
\epsilon_{c}(\tau)=\lambda_d(\tau)-\lambda^{\ast}(\tau)+\lambda^{\ast}(\tau)-\hat \lambda(\tau)\\
%=-A^{\top}\tilde W_c^{\top}h_c(\tau+1)+\tilde W_c(\tau)^{\top}h(\tau)+\Delta \bar \epsilon_c\\
=\xi_c(\tau)+\Delta \bar \epsilon_c(\tau)
\end{array}$$
where $\xi_c(\tau)=-A^{\top}\tilde W_c(\tau)^{\top}h_c(\tau+1)+\tilde W_c(\tau)^{\top}h_c(\tau)$, $\Delta \bar \epsilon_c(\tau)=\bar \epsilon_c(\tau)-A^{\top}\bar \epsilon_c(\tau+1)$. For the sake of simplicity, in the rest of the Appendix we use $q$ to denote a generic variable $q$ with time index $\tau$, i.e., $q(\tau)$, and $q^{+}$ to represent $q(\tau+1)$.
The former two terms of~\eqref{Eqn:partial error c} can be computed as
\begin{equation}\label{Eqn:Wc-fac}\begin{array}{lll}
\frac{\partial tr(\|\epsilon_{c}\|^2+\|\epsilon_{c,N}\|^2)}{\partial \hat W_{c}}&=&\frac{<\partial\epsilon_{c},\epsilon_{c}>}{\partial \hat W_{c}}+\frac{<\partial\epsilon_{cN},\epsilon_{cN}>}{\partial \hat W_{c}}\\
%=-2h_c\epsilon_{c}^{\top}-2h_{cN}\epsilon_{cN}^{\top}\\
%=-2h_c(h_c^{\top}\tilde{W}_c+\bar\epsilon_{c}^{\top})-2h_{cN}(h_{cN}^{\top}\tilde{W}_c+\bar\epsilon_{cN}^{\top})\\
%=-\bar h_c^{\top}\tilde W_c-\bar\epsilon_{\lambda}^{\top}
&=&-2(h_c(\xi_c+\Delta\bar\epsilon_c)^{\top}-h_c^{+}(\xi_c+\Delta\bar\epsilon_c)^{\top}A^{\top}\\
&&+h_{cN}(\xi_{cN}+\bar\epsilon_{cN})^{\top})%:=\varUpsilon
%=-2h_c(\tau)\xi(\tau)^{\top}+2h_c(\tau+1)\xi(\tau)^{\top}A^{\top}-2h_{cN}\xi_{cN}^{\top}+\phi_c(\tau)
\end{array}
\end{equation}
where $\xi_{cN}=\tilde{W}_c^{\top}h_{cN}$, %$\phi_c(\tau)=-2h_c(\tau)\Delta\bar\epsilon(\tau)^{\top}+2h_c(\tau+1)\Delta\bar\epsilon(\tau)^{\top}A^{\top}+2h_{cN}\bar\epsilon_{cN}^{\top}$, %$\bar h_c=2h_ch_c^{\top}+2h_{cN}h_{cN}^{\top}$, %$\bar\epsilon_{\lambda}=2\bar\epsilon_{c}h_c^{\top}+2\bar\epsilon_{cN}h_{cN}^{\top}$
$h_{cN}=h_c(k+N)$, $\bar\epsilon_{cN}=\bar\epsilon_{c}(k+N)$.
Consider the constraint on $\hat \lambda$ of type ${{\Lambda}}^{i}=\{\hat \lambda|\hat \lambda^{\top}Z_{i}\hat\lambda\leq 1\}$, then one can compute the last two terms of~\eqref{Eqn:partial error c} as %It is easy to compute $B(\hat \lambda)=-\sum_{i=1}^plog(b_{\lambda,i}-a_{\lambda,i}^{\top}\hat \lambda)+b_{\lambda,i}-{a_{\lambda,i}^{\top}}{b_{\lambda,i}}^{-1}\hat \lambda$, then $\frac{\partial B(\hat \lambda)}{\partial \hat \lambda}=\sum_{i=1}^p{a_{\lambda,i}}\kappa_{\lambda}^{-1}{b_{\lambda,i}}^{-1}-{a_{\lambda,i}^{\top}}{b_{\lambda,i}}^{-1}$,
\begin{equation}\label{Eqn:Wc-bar-fac}\begin{array}{ll}
\frac{\bar\mu\partial tr( B(\hat\lambda))+B_f(\hat\lambda_N))}
{\partial \hat W_{c}}
&=2\bar\mu (\kappa_{\lambda}^{-1}h_ch_c^{\top}\hat W_cZ_{i}+\kappa_{\lambda N}^{-1}h_{cN}h_{cN}^{\top}\hat W_cZ_{N})\\
&:=2\bar\mu\mathcal{\omega}
\end{array}
\end{equation}
where %$l_{\lambda}=\sum_{i=1}^l{a_{\lambda,i}^{\top}}({\kappa_{\hat\lambda}}^{-1}-{b_{\lambda,i}}^{-1})$,
%$\bar\kappa_{\lambda}=\kappa_{\lambda}^{-1}h_ch_c^{\top}$,
$\kappa_{\lambda}=1-\hat \lambda^{\top}Z_{i}\hat\lambda$,
$\kappa_{\lambda N}=1-\hat \lambda^{\top}Z_{N}\hat\lambda$, $\hat\lambda_N=\hat\lambda(k+N)$.
In view of~\eqref{Eqn:Wc-fac} and \eqref{Eqn:Wc-bar-fac}, one can write the first term in~\eqref{Eqn:deltaL_c} as
\begin{equation}\label{Eqn:DELTA LC}
\begin{array}{lll}
tr(\beta\nabla\delta_{c}^{\top}\tilde W_c)&=&-2\beta tr(\xi_c(\xi_c+\Delta\bar \epsilon_c)^{\top}+\xi_{cN}(\xi_{cN}+\bar \epsilon_{cN})^{\top}-\bar\mu\tilde{W}_c^{\top}\omega)\\
&=&-tr(\|\bar \xi_c\|^2_{M}+\bar \xi_c^{\top}M\psi_1-2\beta\bar\mu\tilde{W}_c^{\top}\omega)
\end{array}
\end{equation}
where $\bar \xi_c=(
\xi_c,\xi_{cN})$, $\psi_1=(
\Delta\bar \epsilon_c,\bar \epsilon_{cN})$, $M=2\beta\text{diag}\{1,1\}$.
%where $\bar\chi_c={W}_c^{\top}h_c$, $\chi_c=\tilde{W}_c^{\top}h_c$, $\bar\xi_{cN}={W}_c^{\top}h_{cN}$.
Also, the last term in~\eqref{Eqn:deltaL_c} can be written as
\begin{comment}
\begin{equation}\label{Eqn:DELTA LC1}
\begin{array}{ll}
tr(\delta_{c}^{\top}\delta_{c})=&
2\beta^2tr((h_c^{\top}h_c+(h_c^{+})^{\top}h_c^{+}-2A^{\top}h_c^{\top}h_c^{+})\xi_c\xi_c^{\top})+\\
&h_{cN}^{\top}h_{cN}\xi_{cN}\xi_{cN}^{\top}+(2h_c^{\top}h_{cN}-2A(h_c^{+})^{\top}h_{cN})\xi_c\xi_{cN}^{\top}\\
&+\varUpsilon^{\top}\phi_c+\phi_c^{\top}\phi_c+\frac{1}{2}\beta^2\varUpsilon^{\top}\bar\mu\omega\hat W_c+\frac{1}{4}\bar\mu^2\hat W_c^{\top}\mathcal{\omega}^{\top}\mathcal{\omega}\hat W_c)
\end{array}
\end{equation}
\end{comment}
\begin{equation}\label{Eqn:DELTA LC1}
\begin{array}{ll}
tr(\frac{\beta^2}{2}\nabla\delta_{c}^{\top}\nabla\delta_{c})=tr(-\|\bar \xi_c\|_{G_1}^2+\bar \xi_c^{\top}(G_2\psi_1+G_3\psi_2)+\bar g_{\bar \epsilon_c}
)\end{array}
\end{equation}
where $\psi_2=
(\hat W_c^{\top}h_c,\hat W_c^{\top}h_{cN})
$,
$\bar g_{\bar \epsilon_c}=2\beta^2(\phi_c^{\top}\phi_c+2\bar\mu\phi_c^{\top}\omega+\bar\mu^2\omega^{\top}\omega))$, $\phi_c=-h_c\Delta\bar\epsilon_c^{\top}+h_c^{+}\Delta\bar\epsilon_c^{\top}A^{\top}-h_{cN}\bar\epsilon_{cN}^{\top}$, $$ G_{1}=-\begin{bmatrix}
g_1&g_{12}\\g_{12}& g_2
\end{bmatrix},$$
and where $g_1=2\beta^{2}(\bar h_c+\bar h_c^{+}A^{\top}A-2A^{\top}h_c^{\top}h_c^{+})$, $g_{2}=2\beta^{2}\bar h_{cN}$, $g_{12}=2\beta^{2}(h_c^{\top}h_{cN}-A(h_c^{+})^{\top}h_{cN})$, $\bar h_c=h_c^{\top}h_c$,
$\bar h_{cN}=h_{cN}^{\top}h_{cN}$, $\bar h_c^{+}=(h_c^{+})^{\top}h_c^{+}$,
$$G_{2}=\begin{bmatrix}
g_3&g_{4}\\g_5&g_{6}
\end{bmatrix},$$
and where $g_3=4\beta^2(\bar h_c-h_c^{\top}h_c^{+}A-(h_c^{+})^{\top}h_cA^{\top}+\bar h_c^{+}AA^{\top})$, $g_4=4\beta^2(-(h_c^{+})^{\top}h_{cN}A+h_c^{\top}h_{cN})$, $g_5=4\beta^2(h_{cN}^{\top}h_c-h_{cN}^{\top}h_c^{+}A)$, $g_6=4\beta^2h_c^{\top}h_{cN}$,
$$G_{3}=\begin{bmatrix}
g_{7}& g_8
\\g_{9}&
g_{10}
\end{bmatrix},$$
and where $g_7=4\bar\mu\beta^2\kappa_{\lambda}^{-1}(-\bar h_c+(h_c^{+})^{\top}h_cA)Z_i$,
$g_8=4\bar\mu\beta^2\kappa_{\lambda N}^{-1}(-h_c^{\top}h_{cN}+(h_c^{+})^{\top}h_{cN})A)Z_N$,
$g_9=-4\bar\mu\beta^2\kappa_{\lambda}^{-1}h_{cN}^{\top}h_cZ_i$, $g_{10}=-4\bar\mu\beta^2\kappa_{\lambda N}^{-1}\bar h_{cN}Z_N$.
Taking~\eqref{Eqn:DELTA LC} and~\eqref{Eqn:DELTA LC1} into consideration, $\Delta L_c$ results
\begin{equation}\label{Eqn:delta Lc-expan}
\begin{array}{ll}
\Delta L_c=tr(-\|\bar \xi_c\|_{G_{c1}}^2+\bar \xi_c^{\top}(G_{c2}\psi_1+G_{c3}\psi_2)+ g_{\bar \epsilon_c}
)\end{array}
\end{equation}
where $G_{c1}=G_1+M$, $ G_{c2}=G_2-M$, $G_{c3}=G_3$, $g_{\bar \epsilon_c}=\bar g_{\bar \epsilon_c}+2\beta\bar\mu\tilde{W}_c^{\top}\omega$.
\begin{comment}
where $g_1=-2\beta+2\beta^{2}(\bar h_c+\bar h_c^{+}-2A^{\top}h_c^{\top}h_c^{+})$, $g_{2}=-2\beta+2\beta^{2}\bar h_{cN}$, $g_{12}=2\beta^{2}(h_c^{\top}h_{cN}-A(h_c^{+})^{\top}h_{cN})$,
$g_3=4\beta^2(\bar h_c-h_c^{\top}h_c^{+}A-(h_c^{+})^{\top}h_cA^{\top}+\bar h_c^{+}A^{\top}A)-2\beta$, $g_4=4\beta^2(-(h_c^{+})^{\top}h_{cN}A^{\top}+h_c^{\top}h_{cN})$, $g_5=4\beta^2(\bar h_c+h_c^{\top}h_c^{+}A)$, $g_6=4\beta^2h_c^{\top}h_{cN}-2\beta$,
$g_7=4\bar\mu^2\beta^2\kappa_{\lambda}^{-1}(-\bar h_c+(h_c^{+})^{\top}h_c(\tau)A)Z_i$,
$g_8=4\bar\mu^2\beta^2\kappa_{\lambda N}^{-1}(-h_c^{\top}h_{cN}+(h_c^{+})^{\top}h_{cN})A)Z_N$,
$g_9=-4\bar\mu^2\beta^2\kappa_{\lambda}^{-1}h_{cN}^{\top}h_cZ_i$, $g_{10}=-4\bar\mu^2\beta^2\kappa_{\lambda N}^{-1}\bar h_{cN}Z_N$,
$\bar h_c=h_c^{\top}h_c$,
$\bar h_{cN}=h_{cN}^{\top}h_{cN}$, $\bar h_c^{+}=(h_c^{+})^{\top}h_c^{+}$, $g_{\bar \epsilon_c}=1/2\beta^2(\phi_c^{\top}\phi_c+2\bar\mu\phi_c^{\top}\omega\hat W_c+\bar\mu^2\hat{W}_c^{\top}\omega^{\top}\omega\hat W_c))-2\bar\mu\beta\tilde{W}_c^{\top}\omega\hat W_c$.
\end{comment}
Applying the Young's inequality and the Cauthy–Schwartz inequality, it holds that
$$\begin{array}{ll}
tr(\bar \xi_c^{\top}(G_{c2}\psi_1+G_{c3}\psi_2))\leq
\frac{\alpha_c}{2}(\|G_{c2}\|^2+\|G_{c3}\|^2)\|\bar \xi_c\|^2+
\frac{1}{2\alpha_c}(\|\psi_1\|^2+\|\psi_2\|^2)
\end{array}$$
where $\alpha_c>0$ is a tuning parameter.
{\color{black}
As $\hat W_c$ is bounded in view of~Lemma.\ref{prop:w}, we assume $\|\hat {W}_c\|\leq W_{c,m}$, then $\|\tilde {W}_c\|\leq 2W_{c,m}$. It holds that $\|\omega\|\leq
\kappa^{-1}h_{c,m}^2W_{c,m}(\|Z_{i}\|+\|Z_{N}\|):=\bar \omega$, $\|\phi_c\|\leq h_{c,m}\bar\epsilon_{c,m}(\sqrt{2}+1+\|A\|):=\bar \phi_c$, $\| g_{\bar \epsilon_c}\|\leq 2\beta^2(\|\bar\phi_c\|^2+2\bar\mu\bar\phi_c\bar\omega+\bar\mu^2\bar\omega^2))+4\beta\bar\mu{W}_{c,m}^2\bar\omega:= g_{c,m}$.
}
Hence,~\eqref{Eqn:delta Lc-expan} leads to
\begin{equation}\label{Eqn:delta Lc-bound}
\begin{array}{ll}
\Delta L_c(\tau)\leq -\|\bar \xi_c(\tau)\|_{\bar G_{c1}}^2+{\rm res_c}
\end{array}
\end{equation}
%
where $\bar G_{c1}=G_{c1}+\frac{\alpha_c}{2}(\|G_{c2}\|^2+\|G_{c3}\|^2)$, ${\rm res_c}= g_{c,m}+\frac{1}{2\alpha_c}(2\bar\epsilon_{c,m}^2+2W_{c,m}^2h_{c,m}^2)$.
%$g_4=4\beta^2(\xi_c(\tau)h_c(\tau)^{\top}+A\xi_c(\tau)h_c(\tau+1)^{\top}+\xi_{cN}h_{cN}^{\top})\phi_c(\tau)$, $g_5=4\mu\beta^2(\xi_c(\tau)h_c(\tau)^{\top}+A\xi_c(\tau)h_c(\tau+1)^{\top}+\xi_{cN}h_{cN}^{\top})\omega\hat W_c(\tau)$,
% $g_6=-2\beta(\xi_c(\tau)\Delta\bar \epsilon(\tau)^{\top}+\xi_{cN}(\tau)\bar \epsilon_{cN}^{\top})$,
%The last inequality in~\eqref{Eqn:DELTA LC} comes from the fact that
%$
%tr(\betah_c^{\top}h_{cN}\tilde W_c^{\top}h_{c}h_{cN}^{\top}W_c)\leq tr(2\beta(h_c^{\top}h_{c}\tilde W_c^{\top}h_{c}h_{c}^{\top}W_c+h_{cN}^{\top}h_{cN}\tilde W_c^{\top}h_{cN}h_{cN}^{\top}W_c))
%$, and $\|W_c\|_F\leq W_{c,max}$.
As a second step, we compute the $\Delta L_a$ in~\eqref{Eqn:deltaL_a}.
Note that, in view of~\eqref{Eqn:loss_a}, one has $$
\frac{\partial\delta_{a}}{\partial \hat W_{a}}=\frac{\partial tr(\|\epsilon_{a}\|^2+\bar\mu B(\hat u_d))}{\partial \hat W_{a}}$$
Consider the input constraint of type $\hat{\mathcal{U}}=\{\hat u|\sum_{i=1}^p a_{u,i}^\top \hat u\leq b_{u,i},\forall i=1,\cdots,p\}$. It is easy to compute $B(\hat u_d)=-\sum_{i=1}^plog(b_{u,i}-a_{u,i}^{\top}\hat u_d)+b_{u,i}-{a_{u,i}^{\top}}{b_{u,i}}^{-1}\hat u_d$, then $\frac{\partial B(\hat u)}{\partial \hat u}=\sum_{i=1}^p{a_{u,i}}^{\top}\kappa_{\hat u,i}^{-1}-{a_{u,i}}^{\top}{b_{u,i}}^{-1}$, where $\kappa_{\hat u,i}=b_{u,i}-a_{u,i}^{\top}\hat u_d$. One has
\begin{equation}\label{Eqn:Wa-fac}
\begin{array}{ll}
\frac{\partial tr(\|\epsilon_{a}\|^2)}{\partial \hat W_{a}}&=\frac{<-\partial \hat G_u(\hat u_d),\epsilon_{a}>}{\partial \hat W_{a}}\\
%=\frac{<-\partial 2R\hat W_a^{\top}h_a,\epsilon_{a}>}{\partial \hat W_{a}}+\frac{<-\partial \mu\frac{\partial B( u_d(\tau))}{\partial u_d(\tau)},\epsilon_{a}>}{\partial \hat W_{a}}\\
%=-2h_a\epsilon_{a}^{\top}R-<\frac{\partial \mu\sum_{i=1}^p \kappa_{\hat u,i}^{-2}a_{u,i}a_{u,i}^{\top}\partial W_a^{\top}h_a,\epsilon_{a}>}{\partial \hat W_{a}}\\
%=-2h_a\epsilon_{a}^{\top}\bar R_1\\
&=-2h_a(\phi_a+\bar R_2\tilde{W}_a^{\top}h_a)^{\top}\bar R_1
\end{array}
\end{equation}
where $\kappa_{u,i}=b_{u,i}-a_{u,i}^{\top} u_d$, $\phi_a=(a_{u,i}a_{u,i}^{\top}+I)\bar\epsilon_{a}+B^{\top}\tilde W_c^{\top}h_c^{+}+B^{\top}\bar\epsilon_c^{+}$, $ \bar R_1=2R+\hat\kappa_{\hat u,i}^{-2}a_{u,i}a_{u,i}^{\top}$, $\bar R_2=2R+\sum_{i=1}^pa_{u,i}a_{u,i}^{\top}\kappa_{u,i}^{-1} \kappa_{\hat u,i}^{-1}$.
Recalling the fact that $\frac{\partial a^{\top }X^{\top}b}{\partial X}=ba^{\top}$ where $X$ is a matrix, $a$ and $b$ are two vectors, one has
\begin{equation}\label{Eqn:Wa-bar-fac}\begin{array}{ll}
\frac{\partial tr(\bar\mu B(\hat u_d(\tau)))}{\partial \hat W_{a}(\tau)}=\bar\mu h_al_{u}
\end{array}
\end{equation}
where $l_{u}=\sum_{i=1}^p{a_{u,i}^{\top}}({\hat\kappa_{u,i}}^{-1}-{b_{u,i}}^{-1})$.
Hence, denoting $\xi_a=\tilde{W}_a^{\top}h_a$, in view of \eqref{Eqn:Wa-fac}, and~\eqref{Eqn:Wa-bar-fac},~\eqref{Eqn:deltaL_a} can be written as
$$\begin{array}{ll}
\Delta L_a=tr(-\|\xi_a\|_{G_{a1}}^2+G_{a2}\xi_a+g_{\bar \epsilon_a})\\
%\leq -c_{R}\|\xi_a\|^2+c_{W_a}\|\xi_a\|+c_{\bar\epsilon_a,\mu}\\
%\leq -c_{R}\|\xi_a\|^2+c_{W_a}\|\xi_a\|+c_{\bar\epsilon_a,\mu}
\end{array}$$
where $G_{a1}=(2\gamma I-2\gamma^2\bar h_a\bar R_2\bar R_1)\bar R_1\bar R_2$, $\bar h_a=h_a^{\top}h_a$, $G_{a2}=(4\gamma^2\bar R_2\bar R_1\bar R_1\phi_a\bar h_a-2\gamma\bar R_1\phi_a+\bar\mu\gamma lu^{\top}+2\gamma^2\bar\mu\bar R_2\bar R_1 lu^{\top}\bar h_a)^{\top}$, $g_{\bar \epsilon_a}=2\gamma^2\bar R_1\phi_a \bar h_a\phi_a^{\top}\bar R_1-2\gamma^2\bar\mu lu^{\top}\bar h_a\phi_{a}\bar R_1+1/2\gamma^2\bar\mu^2lu^{\top}\bar h_alu$.
Note that, $\|\phi_a\|\leq\|a_{u,i}a_{u,i}^{\top}+I\|\bar\epsilon_{a,m}+\sqrt{2N_c}\|B^{\top}\| W_{c,m}h_{c,m}+\|B^{\top}\|\bar\epsilon_{c,m}:=\bar {\phi}_a$, $\|l_{u}\|=\sum_{i=1}^p\|{a_{u,i}^{\top}}\|({\kappa}^{-1}+|{b_{u,i}}^{-1}|):=\bar l_u$, then $\|g_{\bar \epsilon_a}\|=2\gamma^2\|\bar R_1\|^2\bar\phi_a^2 \bar h_{a,m}^2+2\gamma^2\bar\mu\sqrt{m} \|lu\| h_{a,m}^2\bar\phi_{a}\|\bar R_1\|+1/2\gamma^2\bar\mu^2\sqrt{m}\|lu\|^2h_{a,m}:= g_{a,m}$. Similarly, one can easily compute a bound such that $\|G_{a2}\|\leq G_{am}$.
Applying the Young's inequality and the Cauthy–Schwartz inequality leads to
\begin{equation}\label{Eqn:delta La-bound}
\begin{array}{ll}
\Delta L_a
\leq -\|\xi_a\|_{\bar G_{a1}}^2+{\rm res_a}
\end{array}
\end{equation}
where $\bar G_{a1}= G_{a1}-\frac{\alpha_a\gamma}{2}$, $\alpha_a>0$ is a tuning parameter, ${\rm res_a}=\frac{2}{\alpha_a\gamma}G_{am}^2+ g_{a,m}$. %, $\eta=\frac{1}{\alpha}c_{W_a}+c_{\bar\epsilon_a,\mu}$ $c_R=\underline{\sigma}(\bar R_1\bar R_2)\underline{\sigma}(2\gamma I-4\gamma^2h_a^{\top}h_a\bar R_2\bar R_1)$, $c_{W_a}=8\gamma^2\|\bar R_1\phi_a\bar h_a^{\top}\|\|\bar R_2\bar R_1\|+\gamma\|\bar R_1\phi_a\|+\mu\gamma\| lu^{\top}\|+4\gamma^2\mu \|lu^{\top}\bar h_a\|\|\bar R_2\bar R_1\|$, $c_{\bar\epsilon_a,\mu}=\rho_{\bar\epsilon_a}+\rho_{lu}$.
With~\eqref{Eqn:delta Lc-bound} and~\eqref{Eqn:delta La-bound}, one can write
\begin{equation}
\Delta L\leq -\|\bar \xi_c(\tau)\|_{\bar G_{c1}}^2-\|\xi_a(\tau)\|_{\bar G_{a1}}^2+{\rm res_{tot}}
\end{equation}
where ${\rm res_{tot}}={\rm res_{a}}+{\rm res_{c}}$, leading to the variables $\bar \xi_c(\tau)$ and $\xi_a(\tau)$ converging to the corresponding sets as, $\tau\rightarrow N$ and $N\rightarrow +\infty$, i.e.,
\begin{equation}\label{Eqn:uub}
\begin{array}{ll}
\|\bar \xi_c(\tau)\|\leq \frac{\rm res_{tot}}{\sigma_{min}(\bar G_{c1})},\\
\| \xi_a(\tau)\|\leq \frac{\rm res_{tot}}{\sigma_{min}(\bar G_{a1})}.
\end{array}
\end{equation}
In view of the definition of $\bar \xi_c(\tau)$, $ \xi_a(\tau)$, one has $$\begin{array}{ll}
\bar \xi_c=\epsilon_c^{t}-\bar \epsilon_{c,cN},\
\xi_a=\epsilon_a-\bar \epsilon_{a}
\end{array}$$
where $\epsilon_c^{t}=(\epsilon_c,\epsilon_{cN})$, $\bar \epsilon_{{c,cN}}=(\Delta \bar \epsilon_c,\bar \epsilon_{cN})$.
Hence,~\eqref{Eqn:uub} results
\begin{subequations}\label{Eqn:est-uub}
\begin{align}
\|\epsilon_c^{t}\|\leq \frac{\rm res_{tot}}{\sigma_{min}(\bar G_{c1})}+\pi_{A} \epsilon_{c,m}:=\eta_c,\label{Eqn:epi_ct}\\
\| \xi_a(\tau)\|\leq \frac{\rm res_{tot}}{\sigma_{min}(\bar G_{a1})}+\bar \epsilon_{a,m}:=\eta_a.
\end{align}
\end{subequations}
where the inequality in~\eqref{Eqn:epi_ct} comes from that $\|\bar \epsilon_{{c,cN}}\|=\sqrt{\Delta \bar \epsilon_c^{\top}\Delta \bar \epsilon_c+\bar \epsilon_{cN}^{\top}\bar \epsilon_{cN}}\leq\pi_{A} \epsilon_{c,m} $, $\pi_{A}=\sqrt{2+2\|A\|+\|A^2\|}$.
2) Provided that $\bar \epsilon_{c}=0$, $\bar \epsilon_{{cN}}=0$, $\bar\mu=0$, one has
$$\Delta L_c(\tau)\leq -\|\bar \xi_c(\tau)\|_{\bar G_{c1}}^2$$
which implies that $\bar \xi_c(\tau)\rightarrow 0$ exponentially. That is to say, $\hat\lambda(\tau)$ converges to $\lambda_d(\tau)$ at an exponential rate. In view of this, one can rewrite~\eqref{Eqn:lam_d} as
\begin{equation}\label{}
\begin{array}{l}
\lambda_d^{i+1}(\hat {s}(\tau))=\\
\mu \frac{\partial B(\hat {s}(\tau))}{\partial \hat {s}(\tau)}+
2\bar Q\hat {s}(\tau)+
A^\top \lambda_d^i(\hat {s}(\tau+1)),\, \mbox{for} \ \tau\in[k,k+N-1]
\end{array}
\end{equation}
where the superscript $i$ is the iterative step corresponding to $\hat W_c(\tau+i)$, i.e., $\lambda_d^{i}(\hat {s}(\tau))=\hat W_c(\tau+i)^{\top}h_c(\tau)$.
In view of~\eqref{Eqn:lambda} and denoting $\tilde \lambda$ as the subtraction of $\lambda^{\ast}$ and $\lambda_d$, one promptly has
$$\tilde \lambda^{i+1}(\tau)=A^{\top}\tilde \lambda^{i}(\tau+1)$$
By induction, then
$$\tilde \lambda^{i}(\tau)=(A^{\top})^i\tilde \lambda^0(\tau+i)\rightarrow 0$$
as $i\rightarrow+\infty$, due to the fact $\|A^{\top}\|\rightarrow 0$ for $i\rightarrow +\infty$ in view of $\sigma_{max}(A)$ being smaller than 1.
The above result also implies $\hat \lambda(\tau)\rightarrow\lambda^{\ast}(\tau)$ for $\tau\in[k,k+N-1]$ as $N\rightarrow +\infty$. With this result, the term $\hat W_c(\tau)^{\top}h_c^{+}$ in~\eqref{Eqn:Wc-fac} is vanishing, then one has
\begin{equation*}
\begin{array}{ll}
\Delta L_a
\leq -\|\xi_a\|_{\bar G_{a1}}^2,
\end{array}
\end{equation*}
leading to the result that $\xi_a$ converges to $\xi^{\ast}$. That is to say, $\hat W_c(\tau)\rightarrow W_c$ and $\hat W_a(\tau)\rightarrow W_a$ as $\tau\rightarrow N$ and $N\rightarrow+\infty$.
\hfill$\square$
\subsection{Proof of Theorem~\ref{the:2}}\label{appen:the3}
%At time $k=0$, if $N$ is such that problem~\eqref{Eqn:optimizL} with $\hat s(N)=0$ is feasible at $k=0$ if the the predicted terminal state$x(N|0)$ will reach nearly the origin with the optimal control policy computed with~\eqref{Eqn:hjb}, as long as the prediction horizon $N$ has been chosen sufficiently large. This result was proven in~\cite{chmielewski1996constrained}, where interested readers can refer to the computation steps of a finite value of $N$.
1) Under Theorem~\ref{the:ac-conver}, assume that condition~\eqref{Eqn:ac-uub} has been verified with sufficient outer-loop iterative steps, one has $\|\hat u^{\ast}(\tau)-\hat u_d(\tau)\|\leq \eta_{a}$.
Denoting the error $y:=\hat s^{\ast}-\hat s_d$, where $\hat s^{\ast}$ and $\hat s_d$ are the lifted state variables associated with $\hat u^{\ast}$ and $\hat u_d$ respectively. Starting with $x(0)=\hat x(0)=x_0$ at time $0$, one has $s(0)=\hat s(0)=\Phi\left(x_0,\bar n\right)$, then it holds that
$$
\begin{array}{ll}
y_1=B(\hat u^{\ast}(0)-\hat u_d(0))
\end{array}$$
For any $k\geq 1$,
$$
\begin{array}{ll}
&y_{k+1}=Ay_k+B(\hat u^{\ast}(k)-\hat u_d(k))\\
\end{array}$$
By induction and norm operation, one promptly has
$$\|y(N)\|\leq\sqrt{N}\|\sum_{j=1}^{N-j-1}A^jB\|\eta_a.$$
Recall that $N$ is such that MPC problem~\eqref{Eqn:optimizL} is feasible at $k=0$, then $\hat s^{\ast}(N)\in\bar{\mathcal{S}}_f$. With control policy $\hat u_d$ obtained with the actor network, the terminal lifted state can reach the terminal region if $y(N)+\hat s^{\ast}(N)\in\mathcal{X}_f$ for any $\hat s^{\ast}(N)\in\bar{\mathcal{S}}_f$, leading to a sufficient condition $$
\sqrt{N}\|\sum_{j=1}^{N-j-1}A^jB\|\eta_a\leq\frac{1-\rho}{\sigma_{max}(Z)}
$$
that is condition \eqref{Eqn:feasi}.
%Therefore, the learning control problem can be feasible at time $t=0$ for a sufficiently large value of $N$ and small value of $T$ with a suitable weight initialization.
2) The proof of the second point is similar to that of MPC.
Due to the existence of approximation errors of the actor-critic, it is not respected to obtain the optimal control sequence in each prediction horizon. Assumed that, at any time instant $k$, %$\hat {s}(k|k)=\hat {s}(k|k-1)$, and
all the parameters are properly chosen, a feasible control sequence can be generated with~\eqref{Eqn:actor} by solving the optimization problem~\eqref{Eqn:optimiz2} in an incremental learning way in the prediction horizon $[k,k+N-1]$. We denote the near-optimal choice as $\hat s(k|k)$,
$\overrightarrow{\hat u}(k:k+N-1|k)$, %=\hat u^s(k|k),\cdots,\hat u^s(k+N-1|k)$
where $\hat u(k+j|k)=\hat W_{a}(k+j|k)^{\top}h_a(k+j|k)$, $j=0,\cdots,N-1$, associated with a near-optimal cost given as $V_b(k)$ such that $x(k+i|k)\in\mathcal{X}$, $u(k+i|k)\in \mathcal{U}$, $\forall i\in[0,N-1]$, $\hat{s}(k+N|k)\in\mathcal{S}_f$. At the next time instant $k+1$, $\hat s(k+1|k)$, $\hat u(k+1|k),\cdots,\hat u(k+N-1|k),K\hat{s}(k+N|k)$ can be proven to be a safe choice such that $x(k+i|k+1)\in\mathcal{X}$, $u(k+i|k+1)\in \mathcal{U}$, $\forall i\in[0,N-1]$, $\hat{s}(k+N+1|k+1)\in\mathcal{S}_f$, This is due to the standard recursive feasibility argument of MPC. We denote the corresponding cost as $V_b^s(k+1)$. Hence, it can concluded that with the backup safe control policy the real state and control constraints $x\in\mathcal{X}$, $u\in \mathcal{U}$ are verified for all time instants.
3) At time instant $k+1$, the learned cost $V_b^f(k+1|k+1)$ might not be smaller than $V_b^s(k+1|k+1)$ computed with the backup control policy, i.e., the monotonic decreasing property of the learned cost is not verified. In this case and in the case safety is not guaranteed, the backup control policy is activated and applied, see~\eqref{Eqn:safe-u}. %. As the learning process is based on trial-and-errors, it is reasonable to enforce a decision condition that the computed control sequence is to be applied if $V_b^f(k+1|k+1)\leq V_b^s(k+1|k+1)$, otherwise the backup safe policy is adopted.
With this policy, denoting $V_b(k+1)=min\{V_b^f(k+1|k+1),V_b^s(k+1|k+1)\}$, one has
\begin{equation}\label{Eqn:V-MONO}
\begin{array}{ll}
V_b(k+1|k+1)- V_b(k|k)\leq
-(\|\hat {s}(k)\|_{\bar Q}^2+\|\hat u(k)\|_{R}^2\\
+\mu B(\hat s(k))+\mu B(\hat u(k)))-\mu B_f(\hat {s}(k+N))\\
%\|K\hat{s}(k+N)\|_{R}^2+\|\hat {s}(k+N)\|_{\bar Q}^2+\mu B(K\hat{s}(k+N))+\mu B(\hat{s}(k+N))+V_f(\hat {s}(k+N+1))-V_f(\hat {s}(k+N))+\\\mu B_f(\hat {s}(k+N+1))-\mu B_f(\hat {s}(k+N))
+\mu B_f(\hat {s}(k+N+1))+\|\hat {s}(k+N)\|_{F^{\top}PF-P+\bar Q+K^{\top}RK+\mu H}\\
\end{array}
\end{equation}
Recalling that
\begin{equation}
\begin{array}{ll}
B_f(\hat {s}(k+N+1))- B_f(\hat {s}(k+N))\\
=\frac{1}{1-{\hat {s}(k+N)^{\top}}F^{\top}ZF\hat {s}(k+N)}-\frac{1}{1-{\hat {s}(k+N)^{\top}}Z\hat {s}(k+N)}\\
<0
%\leq\frac{1}{\alpha-{\hat {s}(k+N)^{\top}}\left(F^{\top}PF+\bar Q+K^{\top}RK+\mu H\right)\hat {s}(k+N)}-\frac{1}{\alpha-{\hat {s}(k+N)^{\top}}P\hat {s}(k+N)}\\
%=\frac{1}{\alpha-{\hat {s}(k+N)^{\top}}P\hat {s}(k+N)}-\frac{1}{\alpha-{\hat {s}(k+N)^{\top}}P\hat {s}(k+N)}=0
\end{array}
\end{equation}
In view of~\eqref{Eqn:qn_ph}, from~\eqref{Eqn:V-MONO}, the monotonic property of the value function is obtained, i.e., $$
\begin{array}{ll}
V_b(k+1)- V_b(k)\leq\\
-(\|\hat {s}(k)\|_{\bar Q}^2+\|\hat u(k)\|_{R}^2
+\mu B(\hat s(k))+\mu B(\hat u(k)))
\end{array} $$
which leads to $V_b(k+1)- V_b(k)\rightarrow0$ as, $k\rightarrow\infty$. Hence $\hat {x}(k)$, $\hat u(k)\rightarrow0$ as $k\rightarrow \infty$.
Recall that the real state remains inside the tube, i.e., $\hat s\in\mathcal{X}\ominus{\mathcal{Z}}$, then the robustness is obtained, i.e.,
$
{\rm lim}_{k\rightarrow\infty} \,{s}(k)\rightarrow\mathcal{Z}.
$ Consequently,
$${\rm lim}_{k\rightarrow\infty} x(k)\rightarrow {\mathcal{O}}.$$
\hfill$\square$
\section{}
\subsection{Computation of the data-driven model parameters}\label{app-model}
The data-driven modeling process relies on the Koopman operator, but we will skip its definition for space limitation. Interested readers can refer to~\cite{arbabi2018introduction,korda2018linear} for details. In this paper we extend the Koopman operator to systems with additive disturbances.
Given a nonlinear dynamics described in~\eqref{Eqn:non-model}, the main idea is to use a set of scalar observables of the original states in order to define a new lifted state or feature space and estimate their evolution using linear transition matrices.
We first rewrite~\eqref{Eqn:non-model} with redefinition of the state space as $z(k)=(x(k),\boldsymbol{u}_w(k))$, where $\boldsymbol{u}_w(k)=u_w(k),u_w(k+1),\cdots$, $u_w=\begin{bmatrix}
u^{\top}&w^{\top}
\end{bmatrix}^{\top}$ and the extended dynamic is
$$z(k+1)=f(z(k))=\begin{bmatrix}f(x(k),u(k))\\ \Gamma \boldsymbol{u}_w(k)\end{bmatrix},$$ where $\Gamma$ is a left shift operator such that $\boldsymbol{u}_w(k+1)=\Gamma\boldsymbol{u}_w(k)$. In this way, a koopman operator can be defined as
\begin{equation}\label{Eqn:koopman_ex}
(\mathcal{K} \phi)(z)=(\phi \circ f)({z})=\phi(f({z}))
\end{equation}
The objective of interest is to find a finite (possibly minimal)-dimensional approximation for the Koopman operator~\eqref{Eqn:koopman_ex}, which can readily be used for generating the model parameters of the linear predictor. Note however that the extended state $z$ is of infinite-dimension, which is impossible for practical computations. Hence, we select a special form of $\Phi$ as \begin{equation*}
\Phi(z,N_k)=\begin{bmatrix}
\Phi(x,\bar n)\\
u
\end{bmatrix}\end{equation*}
where $N_k=\bar n+m$. Let $K\in\mathbb{R}^{N_k\times N_k}$ be a finite approximate of $\mathcal{K}$, such that $(\phi \circ f)({z})=\mathscr{K}\Phi(z)+\delta$, where $\delta$ is the approximate residual. We use extended dynamic mode decomposition (EDMD) to compute the model parameters. Assume to have $M$ data sets of $(z,z^+)$, the approximate objective is to minimize $\delta$ by solving the corresponding optimization problem with $\ell_2$ regularization
\begin{equation}\label{Eqn:appro_K}
\min_{\mathscr{K}} \sum_{i=1}^{M}\|\mathscr{K}\Phi(z_i,N_k)-\Phi(z_i^+,N_k)\|^2+\theta\|\mathscr{K}\|^2
\end{equation}
where $\theta$ is a positive scalar, $z_i$, $z_i^+$ are the samples belonging to the $i$-th data sets.
%Note however that the extended state $z$ is of infinite-dimension, which is impossible for practical computations. Hence, we select a special form of $\Phi$ as \begin{equation*}
%\Phi(z,N_k)=\begin{bmatrix}
%\Phi(x,\bar n)\\
%u
%\end{bmatrix}\end{equation*}
%where $N_k=\bar n+m$.
%In this way,~\eqref{Eqn:appro_K} is ready to be solved.
%As $\Phi$ is of finite-dimension that might be not invariant under $\mathscr{K}$, the optimal value of residual $\delta$ is not identically zero, but it can be regarded as additive disturbance to be rejected in the robust control framework.
%%%%%%%%%%%%%%%%%%%%%
%Since we are only interested in the future evolution of the state, the last $m$ components of the computed $\mathscr{K}$ being the transition mapping from $(x,u)$ to $u$ can be disregarded.
The matrix group $[A\ B\ D]$ coincides the first $\bar n$ rows of the computed $\mathscr{K}$.
\begin{comment}
in optimization problem~\eqref{Eqn:appro_K}, which leads to the following minimization problem
\begin{equation}\label{Eqn:appro_AB}
\min_{A,B} \sum_{i=1}^{M}\|A\Phi(x_i,N)+Bu_i-\Phi(x_i^+,N)\|
\end{equation}
where $x_i$ is the state data sample in $z_i$, $i=1,\cdots,M$, the matrix group $[A\ B]$ coincides the first $N$ rows of $K$.
\end{comment}
In order to find a linear approximation of $\bar \Phi$ that maps from the observable to the original state space, the following optimization is to be solved:
\begin{equation}\label{Eqn:appro_C}
\min_{C} \sum_{i=1}^{M}\|C{\Phi(x_i,\bar n)}-x_i\|.
\end{equation}
\subsection{Statistical validations of sets $\mathcal{W}$, $\mathcal{V}$, and $\mathcal{D}$}\label{sec:app-b}
\begin{assumption}\label{assu:bound_dis}
For given positive scalars, $\rho_w$, $\rho_v$, and $\rho_d$, $\rho_w\leq\rho_d$, the EDMD and the actor-critic are constructed such that $\mathcal{W}=\mathcal{B}_{\rho_w}(0)$, $\mathcal{V}=\mathcal{B}_{\rho_v}(0)$, and $\mathcal{D}=\mathcal{B}_{\rho_d}(0)$.
\end{assumption}
In the following we show how these bounds specified in Assumption~\ref{assu:bound_dis} can be guaranteed using statistic learning theory for supervised learning. As described in \cite{ hertneck2018learning} and~\cite{von2011statistical}. if the distribution of sample collections is unknown, the true risk of the learning problem is impossible to compute, but can be estimated given sufficient number of training data sets according to the law of large numbers. A relaxed risk evaluation condition is developed by Hoeffding $(1963)$:
\begin{equation}\label{Eqn:risk-eva}
P\left(\left|R_{\mathrm{emp}}(f_l)-R(f_l)\right| \geq \epsilon\right) \leq \delta_r\end{equation}
where $\delta_r:=2 \exp \left(-2 n \epsilon^{2}\right)$, $R_{\mathrm{emp}}(f_l)$ and $R(f_l)$ are the empirical and true risk respectively, $f_l$ is the learned function, and $n$ is the number of training samples. For fixed $f_l$, provided a large number of data sets, the empirical risk is probably a good estimate of the true one.
{\emph {Validation of $\rho_w$ and $\rho_v$ for EDMD}:}
Given $M$ sets of pair $(z_i,z_i^+)$ being independent and identically distributed sampling, where the subscript $i$ denotes the $i$-th sample. We define the loss function as $$\ell(z_i,z_i^+, \Sigma)=\left\{\begin{array}{ll}{0} & {\text { if } \delta_w\leq\rho_w\ \&\ \delta_v\leq\rho_v} \\ {1} & {\text { otherwise }}\end{array}\right.$$ The empirical risk of the learned linear predictor $R_{\mathrm{emp}}=\frac{1}{M}\sum_{i=1}^M\ell(z_i,z_i^+, \Sigma)$. In view of~\eqref{Eqn:risk-eva}, $P(\ell=0)=R\leq R_{\mathrm{emp}}+\epsilon$. For the validation, we choose a fixed value of $\bar R\geq R$ and $\delta_r$, then we check if for the $M$ samples, $\bar R\geq R_{\mathrm{emp}}+\epsilon=R_{\mathrm{emp}}+\sqrt{-{\log \left({0.5\delta_{r}}\right)}/{2 M}}$ with a confidence level of $1-\delta_r$. The validation step is as follows. Set $\bar R\geq R$ and $\delta_r$, we calculate the $R_{\mathrm{emp}}$ and $\epsilon$ for the $M$ samples. If the above condition is satisfied, then the validation is successful; otherwise, the number of samples has to be increased. If the condition is still not verified when the number of samples has reached its maximum value, the learning process has to be modified, e.g., via increasing $N_k$, and repeated.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
{\emph {Validation of $\rho_d$ via off-line closed-loop training}:}
Note that the actor-critic learning are not belonging to supervised learning. Despite of this, given any initial condition $\hat {s}(0)= {s}(0)\in\mathcal{S}$, if the obtained control policy is such that all the predicted states verifying the corresponding constraints, then for any the subsequent time instant a safe solution can always be computed with constraint satisfaction, see Theorem~\ref{the:2}. Therefore, for $S$ sets of ${x}_i(0)$, $i=1,\cdots,S$, the loss function is defined as
$$\begin{array}{l}\ell({x}_i(0),\mathcal{C})=\\
\left\{\begin{array}{ll}{0} & {\text { if a safe learning policy can be found at time}\ 0} \\ {1} & {\text { otherwise }}\end{array}\right.
\end{array}$$
where $\mathcal{C}$ represents the mapping of the closed-loop controller. $\ell({x}_i(0),\mathcal{C})=0$ implies that $d\leq\rho_d$. Naturally, we can define $R_{\mathrm{emp}}=\frac{1}{M}\sum_{i=1}^S\ell({x}_i(0),\mathcal{C})$. The validation process is similar to the that in the previous case, hence it is omitted for saving space.%Along the same line, with fixed choice of $\bar R\geq R$ and $\delta_r$, we check if for the $S$ samples, $\bar R\geq R_{\mathrm{emp}}+\sqrt{-{\log \left({0.5\delta_{r}}\right)}/{2 S}}$ with a confidence level of $1-\delta_r$. The validation step is similar to that of the EDMD, hence here it is neglected due to the limitation of space.
\hfill $\square$
%\bibliographystyle{plain}
%\bibliography{ref}
%\bibliographystyle{IEEEtran}
%\bibliography{IEEEabrv,ref}
\bibliographystyle{unsrt}
\bibliography{ref}
\end{document}
\begin{thebibliography}{10}
\bibitem{mayne2000constrained}
David~Q Mayne, James~B Rawlings, Christopher~V Rao, and Pierre~OM Scokaert.
\newblock Constrained model predictive control: Stability and optimality.
\newblock {\em Automatica}, 36(6):789--814, 2000.
\bibitem{qin2003survey}
S~Joe Qin and Thomas~A Badgwell.
\newblock A survey of industrial model predictive control technology.
\newblock {\em Control engineering practice}, 11(7):733--764, 2003.
\bibitem{bemporad2003min}
Alberto Bemporad, Francesco Borrelli, and Manfred Morari.
\newblock Min-max control of constrained uncertain discrete-time linear
systems.
\newblock {\em IEEE Transactions on automatic control}, 48(9):1600--1606, 2003.
\bibitem{mayne2005robust}
David~Q Mayne, Mar{\'\i}a~M Seron, and SV~Rakovi{\'c}.
\newblock Robust model predictive control of constrained linear systems with
bounded disturbances.
\newblock {\em Automatica}, 41(2):219--224, 2005.
\bibitem{terzi2018learning}
Enrico Terzi, Lorenzo Fagiano, Marcello Farina, and Riccardo Scattolini.
\newblock Learning-based predictive control for linear systems: a unitary
approach.
\newblock {\em arXiv preprint arXiv:1810.12584}, 2018.
\bibitem{lorenzen2017adaptive}
Matthias Lorenzen, Frank Allg{\"o}wer, and Mark Cannon.
\newblock Adaptive model predictive control with robust constraint
satisfaction.
\newblock {\em IFAC-PapersOnLine}, 50(1):3313--3318, 2017.
\bibitem{fagiano2015scenario}
Lorenzo Fagiano, Georg Schildbach, Marko Tanaskovic, and Manfred Morari.
\newblock Scenario and adaptive model predictive control of uncertain systems.
\newblock {\em IFAC-PapersOnLine}, 48(23):352--359, 2015.
\bibitem{rosolia2017learning}
Ugo Rosolia and Francesco Borrelli.
\newblock Learning model predictive control for iterative tasks. a data-driven
control framework.
\newblock {\em IEEE Transactions on Automatic Control}, 63(7):1883--1896, 2017.
\bibitem{limon2017learning}
D~Limon, J~Calliess, and Jan~Marian Maciejowski.
\newblock Learning-based nonlinear model predictive control.
\newblock {\em IFAC-PapersOnLine}, 50(1):7769--7776, 2017.
\bibitem{korda2018linear}
Milan Korda and Igor Mezi{\'c}.
\newblock Linear predictors for nonlinear dynamical systems: Koopman operator
meets model predictive control.
\newblock {\em Automatica}, 93:149--160, 2018.
\bibitem{arbabi2018introduction}
Hassan Arbabi.
\newblock Introduction to koopman operator theory of dynamical systems.
\newblock 2018.
\bibitem{hertneck2018learning}
Michael Hertneck, Johannes K{\"o}hler, Sebastian Trimpe, and Frank
Allg{\"o}wer.
\newblock Learning an approximate model predictive controller with guarantees.
\newblock {\em IEEE Control Systems Letters}, 2(3):543--548, 2018.
\bibitem{wangadp2017}
D.~{Wang}, H.~{He}, and D.~{Liu}.
\newblock Adaptive critic nonlinear robust control: A survey.
\newblock {\em IEEE Transactions on Cybernetics}, 47(10):3429--3451, Oct 2017.
\bibitem{lewis2009reinforcement}
Frank~L Lewis and Draguna Vrabie.
\newblock Reinforcement learning and adaptive dynamic programming for feedback
control.
\newblock {\em IEEE circuits and systems magazine}, 9(3):32--50.
\bibitem{ni2014grdhp}
Zhen Ni, Haibo He, Dongbin Zhao, Xin Xu, and Danil~V Prokhorov.
\newblock {GrDHP}: A general utility function representation for dual heuristic
dynamic programming.
\newblock {\em IEEE transactions on neural networks and learning systems},
26(3):614--627, 2014.
\bibitem{lewis2013reinforcement}
Frank~L Lewis and Derong Liu.
\newblock {\em Reinforcement learning and approximate dynamic programming for
feedback control}, volume~17.
\newblock John Wiley \& Sons, 2013.
\bibitem{wang2009adaptive}
Fei-Yue Wang, Huaguang Zhang, Derong Liu, et~al.
\newblock Adaptive dynamic programming: An introduction.
\newblock 2009.
\bibitem{liu-2015}
Y.~{Liu}, L.~{Tang}, S.~{Tong}, C.~L.~P. {Chen}, and D.~{Li}.
\newblock Reinforcement learning design-based adaptive tracking control with
less learning parameters for nonlinear discrete-time mimo systems.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
26(1):165--176, 2015.
\bibitem{xu2013online}
Xin Xu, Zhongsheng Hou, Chuanqiang Lian, and Haibo He.
\newblock Online learning control using adaptive critic designs with sparse
kernel machines.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
24(5):762--775, 2013.
\bibitem{li2018actor}
Luntong Li, Dazi Li, Tianheng Song, and Xin Xu.
\newblock Actor-critic learning control based on $l_2$-regularized
temporal-difference prediction with gradient correction.
\newblock {\em IEEE transactions on neural networks and learning systems},
(99):1--11, 2018.
\bibitem{xu2016manifold}
Xin Xu, Zhenhua Huang, Lei Zuo, and Haibo He.
\newblock Manifold-based reinforcement learning via locally linear
reconstruction.
\newblock {\em IEEE transactions on neural networks and learning systems},
28(4):934--947, 2016.
\bibitem{zhang2008novel}
Huaguang Zhang, Qinglai Wei, and Yanhong Luo.
\newblock A novel infinite-time optimal tracking control scheme for a class of
discrete-time nonlinear systems via the greedy {HDP} iteration algorithm.
\newblock {\em IEEE Transactions on Systems, Man, and Cybernetics, Part B
(Cybernetics)}, 38(4):937--942, 2008.
\bibitem{liu2013policy}
Derong Liu and Qinglai Wei.
\newblock Policy iteration adaptive dynamic programming algorithm for
discrete-time nonlinear systems.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
25(3):621--634, 2013.
\bibitem{zhang2009neural}
Huaguang Zhang, Yanhong Luo, and Derong Liu.
\newblock Neural-network-based near-optimal control for a class of
discrete-time affine nonlinear systems with control constraints.
\newblock {\em IEEE Transactions on Neural Networks}, 20(9):1490--1503, 2009.
\bibitem{wang2012optimal}
Ding Wang, Derong Liu, Qinglai Wei, Dongbin Zhao, and Ning Jin.
\newblock Optimal control of unknown nonaffine nonlinear discrete-time systems
based on adaptive dynamic programming.
\newblock {\em Automatica}, 48(8):1825--1832, 2012.
\bibitem{jiang2012computational}
Yu~Jiang and Zhong-Ping Jiang.
\newblock Computational adaptive optimal control for continuous-time linear
systems with completely unknown dynamics.
\newblock {\em Automatica}, 48(10):2699--2704, 2012.
\bibitem{liu2012neural}
Derong Liu, Ding Wang, Dongbin Zhao, Qinglai Wei, and Ning Jin.
\newblock Neural-network-based optimal control for a class of unknown
discrete-time nonlinear systems using globalized dual heuristic programming.
\newblock {\em IEEE Transactions on Automation Science and Engineering},
9(3):628--634, 2012.
\bibitem{Xu-finitehorizonadp}
H.~{Xu}, Q.~{Zhao}, and S.~{Jagannathan}.
\newblock Finite-horizon near-optimal output feedback neural network control of
quantized nonlinear discrete-time systems with input constraint.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
26(8):1776--1788, Aug 2015.
\bibitem{zhao2014neural}
Qiming Zhao, Hao Xu, and Sarangapani Jagannathan.
\newblock Neural network-based finite-horizon optimal control of uncertain
affine nonlinear discrete-time systems.
\newblock {\em IEEE transactions on neural networks and learning systems},
26(3):486--499, 2014.
\bibitem{GORGES20174920}
Daniel Görges.
\newblock Relations between model predictive control and reinforcement
learning.
\newblock {\em IFAC-PapersOnLine}, 50(1):4920 -- 4928, 2017.
\newblock 20th IFAC World Congress.
\bibitem{mpcrl_com}
D.~{Ernst}, M.~{Glavic}, F.~{Capitanescu}, and L.~{Wehenkel}.
\newblock Reinforcement learning versus model predictive control: A comparison
on a power system problem.
\newblock {\em IEEE Transactions on Systems, Man, and Cybernetics, Part B
(Cybernetics)}, 39(2):517--529, April 2009.
\bibitem{zanon2019safe}
Mario Zanon and Sébastien Gros.
\newblock Safe reinforcement learning using robust mpc, 2019.
\bibitem{koller2019learningbased}
Torsten Koller, Felix Berkenkamp, Matteo Turchetta, Joschka Boedecker, and
Andreas Krause.
\newblock Learning-based model predictive control for safe exploration and
reinforcement learning, 2019.
\bibitem{wabersich2018linear}
Kim~P Wabersich and Melanie~N Zeilinger.
\newblock Linear model predictive safety certification for learning-based
control.
\newblock In {\em 2018 IEEE Conference on Decision and Control (CDC)}, pages
7130--7135. IEEE, 2018.
\bibitem{xu2018learning}
Xin Xu, Hong Chen, Chuanqiang Lian, and Dazi Li.
\newblock Learning-based predictive control for discrete-time nonlinear systems
with stochastic disturbances.
\newblock {\em IEEE transactions on neural networks and learning systems},
(99):1--12, 2018.
\bibitem{dong2018functional}
Lu~Dong, Jun Yan, Xin Yuan, Haibo He, and Changyin Sun.
\newblock Functional nonlinear model predictive control based on adaptive
dynamic programming.
\newblock {\em IEEE transactions on cybernetics}, 2018.
\bibitem{liu-2016}
Y.~{Liu}, J.~{Li}, S.~{Tong}, and C.~L.~P. {Chen}.
\newblock Neural network control-based adaptive learning design for nonlinear
systems with full-state constraints.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
27(7):1562--1571, 2016.
\bibitem{Chen-2017}
Z.~{Chen}, Z.~{Li}, and C.~L.~P. {Chen}.
\newblock Adaptive neural control of uncertain mimo nonlinear systems with
state and input constraints.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
28(6):1318--1330, 2017.
\bibitem{LIU201883}
Yan-Jun Liu, Shumin Lu, Shaocheng Tong, Xinkai Chen, C.L.~Philip Chen, and
Dong-Juan Li.
\newblock Adaptive control-based barrier lyapunov functions for a class of
stochastic nonlinear systems with full state constraints.
\newblock {\em Automatica}, 87:83 -- 93, 2018.
\bibitem{feller2017relaxed}
Christian Feller and Christian Ebenbauer.
\newblock Relaxed logarithmic barrier function based model predictive control
of linear systems.
\newblock {\em IEEE Transactions on Automatic Control}, 62(3):1223--1238, 2017.
\bibitem{boyd2004convex}
Stephen Boyd, Stephen~P Boyd, and Lieven Vandenberghe.
\newblock {\em Convex optimization}.
\newblock Cambridge university press, 2004.
\bibitem{wills2004barrier}
Adrian~G Wills and William~P Heath.
\newblock Barrier function based model predictive control.
\newblock {\em Automatica}, 40(8):1415--1422, 2004.
\bibitem{heydari2012finite}
Ali Heydari and Sivasubramanya~N Balakrishnan.
\newblock Finite-horizon control-constrained nonlinear optimal control using
single network adaptive critics.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
24(1):145--157, 2012.
\bibitem{scibilia2011feasible}
Francesco Scibilia, Sorin Olaru, and Morten Hovd.
\newblock On feasible sets for {MPC} and their approximations.
\newblock {\em Automatica}, 47(1):133--139, 2011.
\bibitem{rawlings2009model}
James~Blake Rawlings and David~Q Mayne.
\newblock {\em Model predictive control: Theory and design}.
\newblock Nob Hill Pub., 2009.
\bibitem{kurzhanskiy2006ellipsoidal}
Alex~A Kurzhanskiy and Pravin Varaiya.
\newblock Ellipsoidal toolbox (et).
\newblock In {\em Proceedings of the 45th IEEE Conference on Decision and
Control}, pages 1498--1503. IEEE, 2006.
\bibitem{farina2018hierarchical}
Marcello Farina, Xinglong Zhang, and Riccardo Scattolini.
\newblock A hierarchical multi-rate mpc scheme for interconnected systems.
\newblock {\em Automatica}, 90:38--46, 2018.
\bibitem{Werbos1992ApproximateDP}
Paul~J. Werbos.
\newblock Approximate dynamic programming for real-time control and neural
modeling.
\newblock 1992.
\bibitem{Lofberg2004}
J.~L{\"{o}}fberg.
\newblock Yalmip : A toolbox for modeling and optimization in matlab.
\newblock In {\em In Proceedings of the CACSD Conference}, Taipei, Taiwan,
2004.
\bibitem{xu-skernel}
X.~{Xu}, Z.~{Hou}, C.~{Lian}, and H.~{He}.
\newblock Online learning control using adaptive critic designs with sparse
kernel machines.
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
24(5):762--775, May 2013.
\bibitem{nocedal2006numerical}
Jorge Nocedal and Stephen Wright.
\newblock {\em Numerical optimization}.
\newblock Springer Science \& Business Media, 2006.
\bibitem{von2011statistical}
Ulrike Von~Luxburg and Bernhard Sch{\"o}lkopf.
\newblock Statistical learning theory: Models, concepts, and results.
\newblock In {\em Handbook of the History of Logic}, volume~10, pages 651--706.
Elsevier, 2011.
\end{thebibliography}