Continual Predictive Learning from Videos
Extracted source text · arXiv 2204.05624
% CVPR 2022 Paper Template
% based on the CVPR template provided by Ming-Ming Cheng (https://github.com/MCG-NKU/CVPR_Template)
% modified and extended by Stefan Roth (stefan.roth@NOSPAMtu-darmstadt.de)
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
% \usepackage[review]{cvpr} % To produce the REVIEW version
\usepackage{latex/cvpr} % To produce the CAMERA-READY version
% \usepackage[pagenumbers]{cvpr} % To force page numbers, e.g. for an arXiv version
% Include other packages here, before hyperref.
\usepackage{graphicx}
\usepackage{amsmath}
\DeclareMathOperator*{\argmin}{arg\,min}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{bbding}
\usepackage{color}
\usepackage{algorithm}
% \usepackage{algorithm}
\usepackage{algorithmicx}
\usepackage{algpseudocode}
\usepackage{makecell}
\usepackage{bbding}
\usepackage[misc]{ifsym}
\usepackage{times}
\usepackage{epsfig}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
\usepackage{multirow}
\usepackage{enumitem}
\usepackage[accsupp]{axessibility}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}
\definecolor{MyDarkBlue}{rgb}{0,0.08,1}
\definecolor{MyDarkGreen}{rgb}{0.02,0.6,0.02}
\definecolor{MyDarkRed}{rgb}{0.8,0.02,0.02}
\definecolor{MyDarkOrange}{rgb}{0.40,0.2,0.02}
\definecolor{MyPurple}{RGB}{111,0,255}
\definecolor{MyRed}{rgb}{1.0,0.0,0.0}
\definecolor{MyGold}{rgb}{0.75,0.6,0.12}
\definecolor{MyDarkgray}{rgb}{0.66, 0.66, 0.66}
\newcommand{\eqn}[1]{Eq.~(\ref{#1})}
\newcommand{\sect}[1]{Section~\ref{#1}}
\newcommand{\tab}[1]{Table~\ref{#1}}
\newcommand{\fig}[1]{Fig.~\ref{#1}}
\newcommand{\alg}[1]{Alg.~\ref{#1}}
% \newcommand{\yb}[1]{{\textcolor{MyDarkGreen}{[wyb: #1]}}}
\newcommand{\myparagraph}[1]{\vspace{-10pt}\paragraph{#1}}
\newcommand{\myitem}{\vspace{-5pt}\item}
\newcommand\blfootnote[1]{%
\begingroup
\renewcommand\thefootnote{}\footnote{#1}%
\addtocounter{footnote}{-1}%
\endgroup
}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\cvprPaperID{6337} % *** Enter the CVPR Paper ID here
\def\confName{CVPR}
\def\confYear{2022}
\begin{document}
%%%%%%%%% TITLE - PLEASE UPDATE
% \title{Replay-based Continual Video Prediction with Test Time Adaptation}
\title{Continual Predictive Learning from Videos}
\author{
Geng Chen$^{1*}$\quad
Wendong Zhang$^{1*}$\quad
Han Lu$^{1}$\quad
Siyu Gao$^{1}$\quad
Yunbo Wang$^{1\dagger}$\\
Mingsheng Long$^{2}$\quad
Xiaokang Yang$^{1}$\\
$^1$MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University\\
$^2$School of Software, BNRist, Tsinghua University\\
{\tt\small \{chengeng, diergent, yunbow\}@sjtu.edu.cn}
}
% \author{First Author\\
% Institution1\\
% Institution1 address\\
% {\tt\small firstauthor@i1.org}
% % For a paper whose authors are all at the same institution,
% % omit the following lines up until the closing ``}''.
% % Additional authors and addresses can be added with ``\and'',
% % just like the second author.
% % To save space, use either the email address or home page, not both
% \and
% Second Author\\
% Institution2\\
% First line of institution2 address\\
% {\tt\small secondauthor@i2.org}
% }
\maketitle
\input{latex/text/abs}
\input{latex/text/intro}
\input{latex/text/prelim}
\input{latex/text/method}
\input{latex/text/expri}
\input{latex/text/related}
\input{latex/text/concl}
%%%%%%%%% REFERENCES
{\small
\bibliographystyle{latex/ieee_fullname}
\bibliography{latex/egbib}
}
\end{document}
%%%%%%%%% ABSTRACT
\begin{abstract}
Predictive learning ideally builds the world model of physical processes in one or more given environments. Typical setups assume that we can collect data from all environments at all times. In practice, however, different prediction tasks may arrive sequentially so that the environments may change persistently throughout the training procedure. Can we develop predictive learning algorithms that can deal with more realistic, non-stationary physical environments? In this paper, we study a new continual learning problem in the context of video prediction, and observe that most existing methods suffer from severe catastrophic forgetting in this setup. To tackle this problem, we propose the continual predictive learning (CPL) approach, which learns a mixture world model via predictive experience replay and performs test-time adaptation with non-parametric task inference. We construct two new benchmarks based on RoboNet and KTH, in which different tasks correspond to different physical robotic environments or human actions. Our approach is shown to effectively mitigate forgetting and remarkably outperform the na\"ive combinations of previous art in video prediction and continual learning.
\end{abstract}
%%%%%%%%% BODY TEXT
\section{Introduction}
\label{sec:intro}
\blfootnote{$^*$\ Equal contribution.}
\blfootnote{$^{\dagger}$\ Corresponding author: Yunbo Wang.}
\vspace{-10pt}
Predictive learning is an unsupervised learning technique to build a world model of the environment by learning the consequences from historical observations, sequences of actions, and corresponding future observation frames.
%
The standard predictive learning setup is assumed to operate the model in a stationary environment with relatively fixed physical dynamics~\cite{srivastava2015unsupervised,shi2015convolutional,denton2018stochastic,guen2020disentangling}.
%
However, the assumption of stationarity does not always hold in more realistic scenarios, such as in the settings of continual learning (CL), where the model is learned through tasks that arrive sequentially.
%
For example, in robotics (see \fig{fig:intro}), world models often serve as the representation learners of model-based control systems~\cite{ha2018recurrent,finn2017deep,hafner2019learning,hafner2020dream}, while the agent may be subjected to non-stationary environments in different training periods.
%
Under these circumstances, it is not practical to maintain a single model for each environment or each task, nor is it practical to collect data from all environments at all times.
%
A primary finding of this paper is that most existing predictive networks~\cite{srivastava2015unsupervised,shi2015convolutional,denton2018stochastic,guen2020disentangling} cannot perform well when trained in non-stationary environments, suffering from a phenomenon known as catastrophic forgetting~\cite{goodfellow2013empirical}.
We formalize this problem setup as \textit{continual predictive learning}, in which the world model is trained in time-varying environments (\ie, ``tasks'' in the context of continual learning) with non-stationary physical dynamics. The model is expected to handle both newer tasks and older ones after the entire training phase (see \sect{sec:problem} for detailed setups). There are two major challenges.
\begin{figure}[t]
\centering
\includegraphics[width=0.9\columnwidth]{latex/fig/intro.pdf}
\vspace{-5pt}
\caption{The new problem of continual predictive learning and the general framework of our approach at test time.}
\label{fig:intro}
\vspace{-10pt}
\end{figure}
% \myparagraph{Challenge 1:}
\subsection{Covariate--Dynamics--Target Shift}
Unlike in the settings of \textit{domain-incremental} or \textit{class-incremental} CL for deterministic models, the world model, which can be viewed as a conditioned generative model, cannot assume a stationary distribution of training targets or fixed target space.
%
Therefore, different from all previous CL problems, the unique challenge of continual predictive learning is due to \textit{the co-existence of three types of distribution shift, including the covariate shift in $P_X$, the target shift in $P_{Y}$, and the dynamics shift $P_{{Y}|{X}}$\footnote{In predictive learning settings, the input $X$ is in forms of sequential observation frames $\mathcal{X}_{1:T}$ and the training target $Y$ corresponds to future frames $\mathcal{X}_{T+1:T+H}$. We here skip the input action signals for simplicity.}.}
%
Notably, the covariate shift~\cite{shimodaira2000improving,sugiyama2007direct,gretton2009covariate,long2017deep,long2015learning,long2017conditional,liu2019transferable} and target shift~\cite{zhang2013domain,iyer2014maximum,lipton2018detecting,azizzadenesheli2019regularized,guo2020ltf} have been widely considered by existing methods, whereas the conditional distribution is typically assumed to be invariant.
%
In our setup, however, the conditional distribution $P_{{Y}|{X}}$ corresponding to the spatiotemporal dynamics also changes over training periods.
%
It significantly increases the probability of catastrophic forgetting in the world model.
To combat the dynamics shift, we first present a new world model that learns multi-modal visual dynamics of different tasks on top of task-specific latent variables.
%
Future frames are generated by drawing samples from learned mixture-of-Gaussian priors conditioned on a set of categorical task variables, and combining them with a deterministic component of future prediction (see \sect{sec:arch}).
Second, we specifically design a novel training scheme named \textit{predictive experience replay}. Like deep generative replay (DGR)~\cite{shin2017continual}, the proposed training method leverages a learned generative model to produce samples of previous tasks. Yet, in our approach, these samples are fed into the world model as the first frames to generate entire sequences, which can be reused as model inputs for rehearsal.
%
The world model alternates between (i) generating rehearsal data without backpropagating the gradients, (ii) regressing the facilitate future frames of previous tasks produced by the world model itself, and (iii) generating future frames from real data of the current task.
%
Another advantage of this training scheme is about the memory efficiency, as it only retains parts of low-dimensional action vectors in the buffer for action-conditioned predictive replay (see \sect{sec:replay}).
\subsection{Task Inference: Coupled Forgetting Issues}
The second challenge in continual predictive learning is the task ambiguity at test time, which can greatly affect the prediction results.
%
Unlike existing CL methods for fully generative models~\cite{shin2017continual,rao2019continual}, in our setup, the models are required not only to solve each task seen so far, but also to infer which task they are presented with.
%
A na\"ive solution is to infer the task using another neural network.
%
However, due to the inevitable forgetting issue of the task inference model itself, coupled with that of the world model, this method is unlikely to perform well.
%
In \sect{sec:inference}, we propose the non-parametric task inference strategy, which overcomes the intrinsic nature of forgetting of a deterministic model.
%
We also present a self-supervised, test-time training process that recalls the pre-learned knowledge of the inferred task through one or several online adaptation steps.
We construct two new benchmarks for continual predictive learning based on real-world datasets, RoboNet~\cite{DBLP:conf/corl/DasariETNBSSLF19} and KTH~\cite{DBLP:conf/icpr/SchuldtLC04}, in which different tasks correspond to different physical robotic environments or human actions. Our CPL approach is shown to effectively avoid forgetting and remarkably outperform the straightforward combinations of previous art in video prediction and continual learning.
\section{Problem Setup}
\label{sec:problem}
Unlike existing predictive learning approaches, we consider to learn a world model ($\mathcal{M}$) in non-stationary environments (\ie, the evolution of tasks), such that
\begin{equation}
\label{eq:setup}
\widehat{\mathcal X}_{T+1:T+H} \sim \mathcal{M}(\mathcal X_{1:T}, a_{T:T+H-1}, \hat{k}),
\end{equation}
where $\mathcal X_{1:T}$ and $\mathcal X_{T+1:T+H}$ are respectively the observed frames and future frames to be predicted.
%
The task index $k$ is known at training, but not observed at test. It requires our approach not only to solve each task seen so far, but also to infer which task it is presented with, denoted as $\mathcal{T}_{\hat{k}}$.
%
Here, $a_{T:T+H-1}$ is the optional inputs of action signals when $\mathcal{M}$ is learned for vision-based robot control, as in the action-conditioned video prediction experiments in this paper.
%
Formally, continual predictive learning assumes that:
\begin{equation}
\small
\begin{split}
\label{eq:shift}
\text{Covariate shift:} & \ P(\mathcal X_{1:T}^k) \neq P(\mathcal X_{1:T}^{k+1}) \\
\text{Dynamics shift:} & \ P(\mathcal X_{T+1:T+H}^k | \mathcal X_{1:T}^k) \neq P(\mathcal X_{T+1:T+H}^{k+1} | \mathcal X_{1:T}^{k+1}) \\
\text{Target shift:} & \ P(\mathcal X_{T+1:T+H}^k) \neq P(\mathcal X_{T+1:T+H}^{k+1}),
\end{split}
\end{equation}
where we leave out $a_{T:T+H-1}$ for simplicity in the conditional distribution of visual dynamics.
%
The setup is in part similar to class-incremental CL for supervised tasks that assumes $P(\mathcal X^k) \neq P(\mathcal X^{k+1})$, $\{\mathcal Y^k\} = \{\mathcal Y^{k+1}\}$, $P(\mathcal Y^k) \neq P(\mathcal Y^{k+1})$.
%
$\{\mathcal Y^k\}$ denotes a constant label set for discriminative models.
%
In contrast, continual predictive learning does not assume a fixed target space, and therefore may have more severe catastrophic forgetting issues.
\section{Approach}
\newcommand{\x}{{\mathcal{X}}}
\newcommand{\z}{\textbf{z}}
\newcommand{\M}{{\mathcal{M}}}
\newcommand{\G}{{\mathcal{G}}}
\begin{figure*}[t]
\centering
\includegraphics[width=1.0\linewidth]{latex/fig/model_final.pdf}
\vspace{-20pt}
\caption{The overall network architecture of the \textit{mixture world model} and the \textit{predictive experience replay} training scheme in the proposed CPL method. (a) The world model learns representations in the forms of mixture-of-Gaussians based on categorical task variables. (b) As for the predictive experience replay, the world model ($\M$) interacts with the initial-frame generative model ($\G$).
%
In this replay stage, we first use $\G$ to generate the first frames of previous tasks without backpropagating the gradients, then use $\M$ to predict the corresponding future frames, and finally combine the rehearsal data and real data to jointly train $\M$ and $\G$.}
\label{fig:method}
\vspace{-5pt}
\end{figure*}
In this section, we present the new continual predictive learning (CPL) approach, which first mitigates catastrophic forgetting within the world model from two aspects:
\begin{itemize}[leftmargin=*]
\myitem \textbf{Mixture world model:} A recurrent network that captures multi-modal visual dynamics. Unlike existing models \cite{denton2018stochastic,hafner2020dream}, the learned task-specific priors are in forms of mixture-of-Gaussians to overcome dynamics shift.
\myitem \textbf{Predictive experience replay:} A new rehearsal-based training scheme that combats the forgetting within the world model and is efficient in memory usage.
\end{itemize}
\vspace{-5pt}
To cope with the challenge of task ambiguity when testing the world model in an unknown task, we propose:
\begin{itemize}[leftmargin=*]
\myitem \textbf{Non-parametric task inference:} Instead of using any parametric task inference model that may introduce extra forgetting issues, we use a trial-and-error strategy over the task label set to determine the present task.
%Before making predictions, we then perform several steps of self-supervised test-time adaptation to recall the pre-learned knowledge of the inferred task.
\end{itemize}
% The overall network architecture of the world model and the training scheme are shown in \fig{fig:method}.
\subsection{Mixture World Model}
\label{sec:arch}
The world model $\M$ considers a new remedy to catastrophic forgetting from the perspective of spatiotemporal representation.
%
As mentioned above, the forgetting problem within existing world models \cite{denton2018stochastic,hafner2020dream} is mainly caused by the covariate-dynamics-target shift in time-varying environments.
%
Therefore, the key idea of the proposed world model in CPL is to use mixture-of-Gaussian variables to capture the multi-modal distribution of visual dynamics in the latent space, as well as that of spatial appearance in the input/output observation space.
%
Accordingly, as shown in \fig{fig:method}\textcolor{red}{(a)}, the world model consists of three components:
\begin{equation}
% \small
\begin{split}
\text{Representation module:} & \quad \z_{t} \sim q_{\phi}({\x}_{1:t}^k, k) \\
\text{Encoding module:} & \quad \hat{\z}_{t} \sim p_{\psi}({\x}_{1:t-1}^k, k) \\
\text{Dynamics module:} & \quad \widehat{\x}_{t} = p_{\theta}(\x_{1:t-1}^k,a_{1:t-1},\z_{1:t}, k).\\
\end{split}
\end{equation}
The representation module infers the latent state $\z_{t}$ from the target frames. It takes as input the categorical task variable $k \in
\{1,\ldots,K\}$ to cope with the target shift in continual predictive learning scenarios.
%
The encoding module corresponds to the covariate shift and dynamically maps the input frames to $\hat{\z}_{t}$ in the same latent subspace as $\z_{t}$.
%
The dynamics module learns the deterministic transition component from inputs to prediction targets.
%
It responds to multi-modal spatiotemporal dynamics by taking as input the task-specific latent variable $\z_{t}$.
%
All components are implemented as neural networks, in which the dynamics module is particularly composed of stacked ST-LSTM layers~\cite{wang2017predrnn}.
The task-specific latent representation $\z_{t}$ is drawn from a mixture-of-Gaussian distribution, inspired by existing unsupervised learning methods that use Gaussian Mixture priors for variational autoencoders \cite{dilokthanakul2016deep,jiang2017variational,rao2019continual}.
%
Our mixture world model is an early work that uses this representation form to model the multi-modal priors in spatiotemporal dynamics.
%
Specifically, for each task, the representation module and the encoding module are both conditioned on the present task label.
%
They are jointly trained to learn the posterior and prior distribution of $\z_{t}$ by optimizing the Kullback-Leibler divergence.
%They are jointly trained to learn task-specific visual priors by optimizing the Kullback-Leibler divergence between posterior and prior distributions.
%
At task $\mathcal T_k$, the objective function ${\mathcal{L}}_{\M}^k(\x_{1:T+H}^k,a_{1:T+H-1}^k)$ combines the KL loss with the reconstruction loss:
\begin{equation}
\begin{split}
\sum\limits_{t=2}^{T+H} & \big[ \mathbb{E}_{q(\z_{1:t} \, | \, {{\mathcal X}_{1:t}^k,k})}\log p({\mathcal X}_{t}^k \, | \, {\mathcal X}_{1:t-1}^k,a_{1:t-1}^k,\z_{1:t},k)\\
&-\alpha D_{KL}(q(\z_{t} \, | \, {\mathcal X}_{1:t}^k,k) \, || \, p(\hat{\z}_{t} \, | \, {\mathcal X}_{1:t-1}^k,k))\big], \\
\end{split}
\label{eq:KL}
\end{equation}
where $\alpha$ is set to $10^{-4}$ in our experiments. In the test phase, we discard the representation module $q_{\phi}$ and only use the encoding module $p_{\psi}$ to sample task-specific latent variables for frames generation.
\subsection{Predictive Experience Replay}
\label{sec:replay}
The two main challenges in typical CL setups are catastrophic forgetting and memory limitation.
%
Due to the co-existence of covariate shift, target shift, and dynamics shift, these challenges become even more urgent in the context of continual predictive learning based on video data.
%
One common way to tackle these challenges is generative replay \cite{shin2017continual,rao2019continual}, which considers using a generative model to produce samples of previous tasks.
%
However, the generative replay method cannot be used directly in our setup, as it is extremely difficult to generate a valid video sequence using a generative model alone.
Therefore, we propose the predictive experience replay, which firmly combines an initial-frame generative model ($\G$), which learns to generate the first frame of videos at previous tasks given the task labels, with the world model ($\M$).
%
To counter the coviariate shift of image appearance in non-stationary environments, $\G$ also uses learnable mixture-of-Gaussian latent priors, denoted by $\boldsymbol{e}$.
%
As shown in \fig{fig:method}\textcolor{red}{(b)}, for each previous task $\mathcal T_{\tilde{k}}$, we first use $\G$ to generate the first frames of the rehearsal video sequences, and then use $\M$ to predict the corresponding future frames.
%
Finally, we mix the rehearsal sequences at previous tasks and real sequences at the present task $\mathcal T_k$ to train $\G$ and $\M$ in turn. We summarize the training procedure in \alg{alg:train}.
%
The predictive experience replay is different from all existing generative replay methods because the world model plays a key role in the rehearsal process.
\begin{algorithm}[t]
\caption{Predictive experience replay}
\label{alg:train}
\begin{algorithmic}[1]
\Require
Training data $\{ \x_{1:T+H}^{k}\}_{k=1}^K $, $\{a_{1:T+H-1}^{k}\}_{k=1}^K $
\Ensure World model $\M$, generative model $\G$
\State Train $\M$ at $\mathcal{T}_1$ according to \eqn{eq:KL}
\State Train $\G$ at $\mathcal{T}_1$ according to \eqn{eq:G_loss} with $k=1$
\For{$k=2, \ldots, K$}
\State \textit{\# Replay video sequences (skip the batch size)}
\For{$\tilde{k}=1, \ldots, k-1$}
% \State \textit{\# Generate first frames of $\mathcal{T}_i$}
\State $\widehat{\x}^{\tilde{k}}_{1} \leftarrow \G(a^{\tilde{k}}_{1},{\tilde{k}})$
% \State \textit{\# generate sequences of task $i$}
\State $\widehat{\x}_{2:T+H}^{\tilde{k}} \leftarrow \M(\widehat{\x}^{\tilde{k}}_{1},a_{2:T+H-1}^{\tilde{k}},{\tilde{k}})$
\EndFor
\State \textit{\# Mix replayed data at $\mathcal{T}_{1:k-1}$ and real data at $\mathcal{T}_k$}
\State $ (\widehat{\x}_{1:T+H}^{1:k-1},a_{1:T+H-1}^{1:k-1}) \cup (\x_{1:T+H}^k,a_{1:T+H-1}^k)$
\State Train $\M$ according to \eqn{eq:M_loss}
\State Train $\G$ according to \eqn{eq:G_loss}
\EndFor
\end{algorithmic}
\end{algorithm}
%In \alg{alg:train}, we first perform $\G$ and $\M$ successively to generate the fake data for each previous task $\mathcal T_{\tilde{k}}$. We then mix the replayed sequences and real sequences at the present task $\mathcal T_k$ to train $\G$ and $\M$ in turn.
%
%As shown in \fig{fig:method}\textcolor{red}{(b)}, $\M$ alternates between (i) generating rehearsal data without backpropagating the gradients, (ii) regressing the facilitate future frames of previous tasks produced by the world model itself, and (iii) generating future frames from real data of the current task.
%
In particular, for action-conditioned predictive learning scenarios, we maintain a buffer to keep parts (${\tiny{\sim}}7\%$) of the low-dimensional action sequences from previous tasks.
%
During predictive experience replay, we first sample an action sequence from the buffer $a_{1:T+H-1}^{\tilde{k}}$ at a previous task $\mathcal T_{\tilde{k}}$.
%
We feed the initial action $a_1^{\tilde{k}}$ and the task label $\tilde{k}$ into $\G$ to ensure that the generated first frame $\widehat{\x}_1^{\tilde{k}}$ is valid for robot control, and perform $\M$ to produce predictive replay results $\widehat{\x}_{2:T+H}^{\tilde{k}}$ given $\widehat{\x}_1^{\tilde{k}}$ and $a_{2:T+H-1}^{\tilde{k}}$.
%
In predictive experience replay, we train the world model $\M$ at $\mathcal{T}_k$ by minimizing
\begin{equation}
\begin{split}
{\mathcal{L}}_{\M} = & \sum\limits_{\tilde{k}=1}^{k-1} {\mathcal{L}}_{\M}^{\tilde{k}}(\widehat{\x}_{1:T+H}^{\tilde{k}},a_{1:T+H-1}^{\tilde{k}}) \\
& + {\mathcal{L}}_{\M}^k(\x_{1:T+H}^k,a_{1:T+H-1}^k). \\
\end{split}
\label{eq:M_loss}
\end{equation}
The objective function of the initial-frame generative model $\G$ can be written as
\begin{equation}
% \small
\begin{split}
{\mathcal L}_{\G} & = \ \mathbb{E}_{q(\boldsymbol{e} \, | \, {{\x}_{1}^k,k})} \log p({\x}_{1}^k \, | \, \boldsymbol{e},a_1^k,k) \\
%
& \quad - \beta D_{KL}(q(\boldsymbol{e} \, | \, {\x}_{1}^k, k) \, || \, p(\hat{\boldsymbol{e}} \, | \, k)) \\
%
& + \sum\limits_{\tilde{k}=1}^{k-1} \big[ \mathbb{E}_{q(\boldsymbol{e} \, | \, {\widehat{{\x}}_{1}^{\tilde{k}},{\tilde{k}}})} \log p(\widehat{{\x}}_{1}^{\tilde{k}} \, | \, \boldsymbol{e},a_1^{\tilde{k}},{\tilde{k}}) \\
%
& \quad - \beta D_{KL}(q(\boldsymbol{e} \, | \, \widehat{{\x}}_{1}^{\tilde{k}}, {\tilde{k}}) \, || \, p(\hat{\boldsymbol{e}} \, | \, {\tilde{k}})) \big],\\
\end{split}
\label{eq:G_loss}
\end{equation}
where the reconstruction loss is in an $\ell_2$ form and $\beta$ is set to $10^{-4}$ through empirical grid search.
\subsection{Non-Parametric Task Inference}
\label{sec:inference}
\begin{algorithm}[t]
\caption{Testing procedure}
\label{alg:test}
\begin{algorithmic}[1]
\Require
Observation frames $\x_{1:T}$, optional actions $a_{1:T+H}$
\Ensure Predicted future frames $\widehat{\x}_{T+1:T+H}$
\State \textit{\# Non-parametric task inference}
\For{$k=1, \ldots, K$}
\State $\widehat{\x}_{T/2+1:T}^k \leftarrow \M( \x_{1:T/2}, a_{1:T-1}, k)$
\EndFor
\State $\hat{k} = \argmin_{k \in \{1,\ldots,K \}} \sum_{t=T/2+1}^{T}(\x_{t}-\widehat{\x}_{t}^k)^2$
\State \textit{\# Test-time adaptation (optional)}
\State Optimize $\mathcal M$ with ${\mathcal L}_{\M}^{\hat{k}}({\mathcal X}_{1:T},a_{1:T-1})$
\State \textit{\# Model deployment}
\State $\widehat{\mathcal X}_{T+1:T+H} \leftarrow \M(\x_{1:T}, a_{1:T+H-1}, \hat{k})$
\end{algorithmic}
\end{algorithm}
In the mixture world model, the task label has a significant impact on the learned priors and corresponding prediction results.
%
Since it is unknown at test time, it can only be inferred from the input observation sequences, \ie, video classification.
%
However, existing video classification models tend to underperform in the domain-incremental CL setting, which will magnify the catastrophic forgetting problem jointly trained with the world model.
%
To avoid the inherent forgetting issue of model-based task inference, we propose a new non-parametric method that only exploits the learned mixture world model to make task inference.
\begin{table*}[t]
\centering
\small
\begin{tabular}{|l|cc|cc|}
\hline
\multirow{2}{0cm}{Method} & \multicolumn{2}{c|}{Action-conditioned} & \multicolumn{2}{c|}{Action-free} \\
& PSNR$^{\uparrow}$ & SSIM$^{\uparrow}$ ($\times 10^{-2}$) & PSNR$^{\uparrow}$ & SSIM$^{\uparrow}$ ($\times 10^{-2}$) \\
\hline
\hline
SVG~\cite{denton2018stochastic} &18.72 $\pm$ 0.61 & 68.59 $\pm$ 2.22 & 18.92 $\pm $0.51 & 68.08 $\pm$ 2.20 \\
PredRNN~\cite{wang2017predrnn} & 19.45 & 66.38 & 19.56 & 69.92 \\
PhyDNet~\cite{guen2020disentangling} & 19.60 & 68.68 & 21.00 & 75.47 \\
\hline
PredRNN + LwF~\cite{li2017learning}& 19.10 & 64.73 & 19.79 & 71.43 \\
PredRNN + EWC~\cite{kirkpatrick2017overcoming} &21.15&74.72&21.15&78.02\\
CPL-base + EWC~\cite{kirkpatrick2017overcoming} & 21.29 $\pm$ 0.30 & 75.16 $\pm$ 0.98 & 21.38 $\pm$ 0.18 & 76.68 $\pm$ 0.69 \\
\hline
CPL-base & 19.36 $\pm$ 0.00 & 63.57 $\pm$ 0.00 & 20.15 $\pm$ 0.02 & 71.15 $\pm$ 0.08 \\
CPL-full & \textbf{23.26 $\pm$ 0.10} & \textbf{80.72 $\pm$ 0.23} & \textbf{22.48 $\pm$ 0.03} & \textbf{78.84 $\pm$ 0.07} \\
\hline
CPL-base (Joint training) & 24.64 $\pm$ 0.01 & 83.73 $\pm$ 0.00 & 22.56 $\pm$ 0.01 & 79.57 $\pm$ 0.02 \\
\hline
\end{tabular}
% \vspace{-2pt}
\caption{Quantitative results of continual predictive learning on the RoboNet benchmark in both action-conditioned and action-free setups. (\textbf{Lines 1-3}) Existing video prediction models with i.i.d. assumption.
(\textbf{Lines 4-6}) Combinations of predictive models and continual learning approaches.
(\textbf{Lines 7-8}) Our predictive model based on learned mixture-of-Gaussian priors, and the the entire CPL with predictive experience replay and non-parametric task inference.
(\textbf{Line 9}) A baseline model jointly trained on all tasks throughout the training procedure, whose results can be roughly viewed as the upper bound of our approach.
}
\label{tab:robonet}
\vspace{-5pt}
\end{table*}
More precisely, as shown in \alg{alg:test}, we feed the first half of each input sequence into the world model $\x_{1:T/2}$, along with a hypothetic task label $k$.
%
We then enumerate each task label $k \in \{1,\ldots,K\}$ and evaluate the outputs of the world model on the remaining frames of the input sequence $\x_{T/2+1:T}$.
%
Finally, we choose the task label $\hat{k}$ that leads to the best prediction quality.
In addition to using $P(\x_{T/2+1:T} | \x_{1:T/2})$ to perform task inference, we also use this self-supervision for test-time adaptation, which allows the model to continue training after deployment.
%
Test-time adaptation effectively recalls the pre-learned knowledge in the inferred task ${\mathcal{T}}_{\hat{k}}$ through one-step (or few-steps) online optimization, thus further alleviating the forgetting problem.
\section{Experiment}
% In this section, we validate the effectiveness of CPL on two real-world datasets that are widely used by previous video prediction approaches. We develop continual learning setups on these datasets, according to different robotic environments or different action categories.
% %
% We mainly compare CPL with representative video prediction models (\eg, PredRNN \cite{wang2017predrnn}, SVG \cite{denton2018stochastic}, and PhyDNet \cite{guen2020disentangling}) and the combinations with continual learning algorithms (\eg, EWC \cite{kirkpatrick2017overcoming} and LwF \cite{li2017learning}).
\subsection{Experimental Setup}
\paragraph{Benchmarks.}
We quantitatively and qualitatively evaluate CPL on the following two real-world datasets:
\begin{itemize}[leftmargin=*]
\myitem \textbf{RoboNet} \cite{DBLP:conf/corl/DasariETNBSSLF19}. The RoboNet dataset contains action-conditioned videos of robotic arms interacting with a variety of objects in various environments. We divide the whole dataset into four continual learning tasks according to the environments (\ie, \textit{Berkeley} $\rightarrow$ \textit{Google} $\rightarrow$ \textit{Penn} $\rightarrow$ \textit{Stanford}). For each task, we collect about $3{,}840$ training sequences and $960$ testing sequences.
\myitem \textbf{KTH action}\cite{DBLP:conf/icpr/SchuldtLC04}. This dataset contains gray-scale videos which include $6$ types of human actions. We directly use the action labels to divide the dataset into $6$ tasks (\ie, \textit{Boxing} $\rightarrow$ \textit{Clapping} $\rightarrow$ \textit{Waving} $\rightarrow$ \textit{Walking} $\rightarrow$ \textit{Jogging} $\rightarrow$ \textit{Running}). For each task, we collect about $1{,}500$ training sequences and $800$ testing sequences in average.
\end{itemize}
\vspace{-5pt}
We define the task orders by random sampling, and without loss of generality, our approach is effective to any task orders (see \sect{sec:ablation}). More experimental configurations and the implementation details can be found in the Supplementary Material.
% \vspace{-2pt}
\myparagraph{Evaluation criteria.}
We adopt SSIM and PSNR from previous literature~\cite{denton2018stochastic,wang2017predrnn} to evaluate the prediction results. We run the continual learning procedure $10$ times and report the mean results and standard deviations in the two metrics.
% \vspace{-2pt}
\myparagraph{Compared methods.}
We compare CPL with the following baselines and existing approaches:
\begin{itemize}[leftmargin=*]
\myitem \textbf{CPL-base}: A baseline model that excludes the new components of Gaussian mixtures, predictive replay, and task inference.
%
\myitem \textbf{PredRNN} \cite{wang2017predrnn}, \textbf{SVG} \cite{denton2018stochastic}, \textbf{PhyDNet} \cite{guen2020disentangling}: Video prediction models focused on stochastic, deterministic, and disentangled dynamics modeling respectively.
%
\myitem \textbf{LwF} \cite{li2017learning}: It is a distillation-based CL method built on the memory state of PredRNN \cite{wang2017predrnn}.
%
\myitem \textbf{EWC} \cite{kirkpatrick2017overcoming}: It constrains the parameters of PredRNN and CPL-base on new tasks with additional loss terms.
\end{itemize}
\vspace{-5pt}
% \myparagraph{Implementation details.}
% We provide some common settings of our approach on two datasets. For the mixture world model, we use four ST-LSTM cells~\cite{wang2017predrnn} in the dynamic modules and the dimension of the prior and posterior are set to $128$.
% In the predictive experience replay scheme,
% the number of rehearsal video sequences from all previous tasks is about one-third of those used in the current task. All models are trained using the ADAM optimizer and the learning rate is set to $0.0005$ for the KTH dataset and $0.0001$ for the RoboNet dataset. Besides, the mini-batch size is set to $32$ for the KTH dataset and $16$ for the RoboNet dataset, and the input frames are pre-resized to $64\times 64$ for all datasets. We optimize the whole model by $30{,}000$ iterations for each task during the continual learning process.
\subsection{RoboNet Benchmark}
We first evaluate CPL on the real-world RoboNet benchmark, in which different continual learning tasks are divided by laboratory environments.
%
We conduct both action-conditioned and action-free video prediction on RoboNet.
%
The former follows the common practice~\cite{babaeizadeh2018stochastic,DBLP:conf/cvpr/WuNM0F21} to train the world model to predict $10$ frames into the future from $2$ observations and corresponding action sequence at the $11$ time steps.
%
For the action-free setup, we use the first $5$ frames as input to predict the next $10$ frames.
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{latex/fig/robonet_psnr_compare.pdf}
\vspace{-15pt}
\caption{Results on the action-conditioned RoboNet benchmark. The horizontal axis represents the sequential training process, and the vertical axes represent test results on particular tasks after each training period. The purple dashed line indicates the results of the baseline model jointly trained on all tasks.
}
\vspace{-5pt}
\label{fig:robonet_action_cond}
\end{figure}
\myparagraph{Quantitative comparison.}
\tab{tab:robonet} gives the quantitative results on RoboNet, in which the models are evaluated on the test sets of all $4$ tasks after the training period on the last task.
%
We have the following findings here.
%
\textbf{First}, CPL outperforms existing video prediction models by a large margin.
%
For instance, in the action-conditioned setup, it improves SVG in PSNR by $24.3\%$, PredRNN by $19.6\%$, and PhyDNet by $18.7\%$.
%
\textbf{Second}, CPL generally performs better than previous continual learning methods (\ie, LwF and EWC) combined with video prediction backbones.
%
Note that a na\"ive implementation of LwF on top of PredRNN even leads to a negative effect on the final results.
%
\textbf{Third}, by comparing CPL-full (our final approach) with CPL-base (w/o Gaussian mixture latents, predictive experience replay, or non-parametric task inference), we can see that the new technical contributions have a great impact on the performance gain.
%
We provide more detailed ablation studies in \sect{sec:ablation}.
%
\textbf{Finally}, CPL is shown to effectively ease catastrophic forgetting by approaching the results of jointly training the world model on all tasks in the i.i.d. setting ($23.26$ vs. $24.64$ in PSNR).
%
Apart from the average scores for all tasks, in \fig{fig:robonet_action_cond}, we provide the test results on particular tasks after individual training periods.
%
As shown in the bar charts right to the main diagonal, CPL performs particularly well on previous tasks, effectively alleviating the forgetting issue.
%
Please refer to the Supplementary Material for detailed comparison results.
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{latex/fig/robonet_qualititave.pdf}
\vspace{-25pt}
\caption{Showcases of action-conditioned video prediction in the first environment of RoboNet (\ie, \textit{Berkeley}) after training the models in the last environment (\ie, \textit{Stanford}).}
% \vspace{-5pt}
\label{fig:robonet_qualitative}
\end{figure}
\myparagraph{Qualitative comparison.}
\fig{fig:robonet_qualitative} provides the qualitative comparisons on the action-conditioned RoboNet benchmark.
%
Specifically, we use the final models after the training period of the last task to make predictions on the first task.
%
We can see from these demonstrations that our approach is more accurate in predicting both future dynamics of the objects as well as the static information of the scene.
%
In contrast, the predicted frames by PredRNN+LwF and CPL-base+EWC suffer from severe blur effect in the moving object or the static (but complex) background, indicating that directly combining existing CL algorithms with the world models cannot effectively cope with the dynamics shift in highly non-stationary environments.
\subsection{KTH Benchmark}
% We further conduct experiments on the KTH dataset to show the effectiveness of our CPL approach on human action prediction which focuses more on body part dynamics. In this case, we use different human action categories to divide the whole dataset into different continual learning tasks where each task contains video sequences from a particular action category. The learning sequence is set as Boxing, Hand-clapping, Hand-waving, Walking, Jogging, and Running. For each video sequence, we use the first ten frames as input and predict the next ten frames. We provide the quantitative results in \tab{tab:Quantitative on kth dataset} and \fig{fig:kth_psnr_compare}, and qualitative results are shown in \fig{fig:kth_qualitative}.
\begin{table}[t]
\small
\centering
\begin{tabular}{|l|cc|}
\hline
Method & PSNR & SSIM ($\times 10^{-2}$) \\
\hline
\hline
SVG~\cite{denton2018stochastic} & 22.20 $\pm$ 0.02 & 69.23 $\pm$ 0.01 \\
% \hline
PredRNN~\cite{wang2017predrnn} & 23.27 & 70.47 \\
% \hline
PhyDNet~\cite{guen2020disentangling} & 23.68 & 72.97 \\
\hline
PredRNN + LwF~\cite{li2017learning} & 24.25 & 70.93 \\
CPL-base + EWC~\cite{kirkpatrick2017overcoming} & 24.32 $\pm$ 0.15 & 69.02 $\pm$ 0.48 \\
\hline
CPL-base & 22.96 $\pm$ 0.05 & 68.98 $\pm$ 0.02 \\
CPL-full & \textbf{29.12 $\pm$ 0.03} & \textbf{84.50 $\pm$ 0.04} \\
\hline
CPL-base (Joint train) & 28.12 $\pm$ 0.01 & 82.16 $\pm$ 0.00 \\
\hline
\end{tabular}
\vspace{-5pt}
\caption{Quantitative results on the KTH benchmark.
}
\label{tab:kth}
\vspace{-5pt}
\end{table}
\paragraph{Quantitative comparison.}
\tab{tab:kth} shows the quantitative results on the test sets of all $6$ tasks after the last training period of the models on the last task.
%
We can observe that CPL significantly outperforms the compared video prediction methods and continual learning methods in both PSNR and SSIM.
%
Furthermore, an interesting result is that our approach even outperforms the joint training model, as shown in the bottom line in \tab{tab:kth}.
%
While we do not know the exact reasons, we state two hypotheses that can be investigated in future work.
%
First, the Gaussian mixture priors enable the world model to better disentangle the representations of visual dynamics learned in different continual learning tasks.
%
Second, the predictive experience replay allows the pre-learned knowledge on previous tasks to facilitate the learning process on new tasks.
%
\fig{fig:kth_psnr_compare} provides the intermediate test results on particular tasks after each training period, which confirm the above conclusions.
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{latex/fig/kth_psnr_compare.pdf}
\vspace{-15pt}
\caption{Results on the KTH benchmark. The horizontal axis represents the sequential training process, and the vertical axes represent test results on particular tasks after each training period.}
% \vspace{5pt}
\label{fig:kth_psnr_compare}
\end{figure}
\newcommand{\yes}{{\scriptsize\Checkmark}}
\newcommand{\no}{{\scriptsize\XSolidBrush}}
\begin{table}
\small
\centering
\begin{tabular}{|c|c|c|c|cc|}
\hline
Replay & Infer ${k}$ & Random ${k}$ & Adapt & PSNR & SSIM\\
\hline
\hline
\no & \no & \no & \no & 22.96 & 68.98 \\
% \hline
\yes & \no & \no & \no & 27.21 & 79.99 \\
% \hline
\yes & \yes & \no & \no & 27.82 & 81.51 \\
% \hline
\yes & \no & \yes & \no & 26.56 & 78.64 \\
% \hline
\yes & \yes & \no & \yes & \textbf{29.12} & \textbf{84.50} \\
\hline
\end{tabular}
\vspace{-5pt}
\caption{Ablation study for each component of CPL on the KTH benchmark. ``Replay'' denotes the use of predictive experience replay. ``Infer $k$'' indicates the use of non-parametric task inference. ``Random $k$'' means that the world model takes as input a random task label at test time. ``Adapt'' means test-time adaptation.}
\label{tab:Ablations_kth}
\vspace{-5pt}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{latex/fig/kth_qualititave.pdf}
\vspace{-15pt}
\caption{Showcases of predicted frames of the first task (\ie, \textit{Boxing}) after the training period of the last task (\ie, \textit{Running}).}
\vspace{-10pt}
\label{fig:kth_qualitative}
\end{figure}
\myparagraph{Qualitative comparison.}
We visualize a sequence of predicted frames on the first task of KTH in \fig{fig:kth_qualitative}.
%
As shown, all existing video prediction models and even the one with LwF generate future frames with the dynamics learned in the last task (\ie, \textit{Running}), which clearly demonstrates the influence of the dynamics shift.
%
Images generated by CPL-base+EWC suffer from a severe blur effect, indicating that the model cannot learn disentangled representations for different dynamics in the non-stationary training environments.
%
In comparison, CPL produces more reasonable results.
% Instead, our CPL method remarkably outperforms previous pure video prediction methods and other combination methods, which not only generates less blurry future frames, but also produces much more accurate prediction results according to the task.
%
To testify the necessity of task inference, we also provide incorrect task labels for CPL.
%
As shown in the third line from the bottom, the model takes as input the \textit{Boxing} frames along with an erroneous task label of \textit{Running}.
%
Interestingly, CPL combines the inherent dynamics of input frames (reflected in motion of arms) with the dynamics priors from the input task label (reflected in motion of legs).
\subsection{Ablation Study}
\label{sec:ablation}
\paragraph{Effectiveness of each component in CPL.}
We conduct ablation studies on the KTH benchmark step by step.
%
In \tab{tab:Ablations_kth}, the first line shows the results of the CPL-base model and the bottom line corresponds to our final approach.
%
In the second line, we train CPL-base with predictive experience replay and observe a significant improvement from $22.96$ to $27.21$ in PSNR.
%
In the third line, we improve the world model with mixture-of-Gaussian priors and accordingly perform non-parametric task inference at test time. We observe consistent improvements in both PSNR and SSIM upon the previous version of the model.
%
In the fourth line, we skip the non-parametric task inference during testing and use a random task label instead. We observe that the performance drops from $27.82$ to $26.56$ in PSNR, indicating the importance of task inference to predictive experience replay.
%
Finally, in the bottom line, we introduce the self-supervised test-time adaptation. It shows a remarkable performance boost compared with all the above variants.
\myparagraph{Is CPL robust to the task order?}
As shown in \tab{tab:kth_false_label}, we further conduct experiments to analyze that whether CPL can effectively alleviate catastrophic forgetting regardless of the task order. We additionally train the CPL model in $3$-$4$ random task orders.
% %
% Specifically, we train CPL with a different task order from that in the above experiments: \textit{Boxing} $\rightarrow$ \textit{Walking} $\rightarrow$ \textit{Clapping} $\rightarrow$ \textit{Jogging} $\rightarrow$ \textit{Waving} $\rightarrow$ \textit{Running}, such that the visual dynamics switch frequently between subtle hand movements and more global body movements.
% %
From the results, we find that the proposed techniques including mixture world model, predictive experience replay, and non-parametric task inference are still effective despite the change of training order.
\begin{table}[t]
\small
\centering
\begin{tabular}{|l|cc|}
\hline
Dataset & PSNR & SSIM ($\times 10^{-2}$) \\
\hline
\hline
RoboNet & 23.58 $\pm$ 0.28 & 79.67 $\pm$ 3.75 \\
KTH & 28.93 $\pm$ 0.14 & 83.99 $\pm$ 0.40 \\
\hline
\end{tabular}
\vspace{-5pt}
\caption{Robustness of CPL on random task orders.}
\label{tab:kth_false_label}
\vspace{-10pt}
\end{table}
%(PSNR/SSIM)
\section{Related Work}
\paragraph{Continual learning of supervised tasks.}
Continual learning is designed to cope with the continuous information flow, retaining or even optimizing old knowledge while absorbing new knowledge.
%
Mainstream paradigms include regularization, replay, and parameter isolation~\cite{delange2021continual}.
%
The regularization approaches typically tackle catastrophic forgetting~\cite{goodfellow2013empirical} by constraining the learned parameters on new tasks with additional loss terms, \eg, EWC~\cite{kirkpatrick2017overcoming}, or distilling knowledge from old tasks, \eg, LwF~\cite{li2017learning}.
%
For replay-based approaches, a typical solution is to retain a buffer on earlier tasks of representative data or feature exemplars~\cite{rebuffi2017icarl,riemer2019learning,ayub2021eec}. Some approaches also use generative networks to encode the previous data distribution and synthesize fictitious data for experience replay, \eg, DGR~\cite{shin2017continual} and CURL~\cite{rao2019continual}.
%
The parameter isolation approaches allow the neural networks to dynamically expand when new tasks arrive~\cite{rusu2016progressive} or encourage the new tasks to use previously ``unused'' parameter subspaces~\cite{he2018overcoming}.
\myparagraph{Continual learning of unsupervised tasks.}
Most existing approaches are mainly focused on supervised tasks of image data.
%
Despite the previous literature that discussed unsupervised CL \cite{rao2019continual,cha2021co2l,ke2021classic}, our approach is significantly different from these methods as it explores the specific challenges of continual predictive learning for video data, especially the covariate-dynamics-target shift.
%
The most related method to CPL is CURL \cite{rao2019continual}, which introduces a mixture-of-Gaussian latent space for class-incremental CL and combat forgetting via generative replay.
%
There are three major differences between CPL and CURL. First, CURL cannot be directly used in our setup as it does not handle the dynamics shift in non-stationary spacetime, while CPL tackles it through a new world model.
%
Second, CPL greatly benefits from the carefully-designed predictive replay algorithm, while it is extremely difficult for CURL to replay valid video frames using a fully generative model alone.
%
Third, CPL provides a non-parametric task inference method as opposed to the model-based inference method in CURL.
\myparagraph{Video prediction.}
RNN-based models have been widely used for deterministic video prediction \cite{ranzato2014video, srivastava2015unsupervised,shi2015convolutional,de2016dynamic,wang2017predrnn,wang2019eidetic,su2020convolutional, yao2020unsupervised}.
%
Shi \etal \cite{shi2015convolutional} proposed ConvLSTM to improve the learning ability of spatial information by combining convolutions with LSTM transitions.
%
Following this line, Wang \etal \cite{wang2017predrnn} proposed PredRNN, modeling memory cells in a unified spatial and temporal representation.
%
Stochastic video prediction models assume that different plausible outcomes would be equally probable for the same input, and thus incorporate uncertainty in the models using GANs \cite{vondrick2016generating,tulyakov2018mocogan} or VAEs \cite{denton2018stochastic,babaeizadeh2018stochastic, lee2018stochastic, castrejon2019improved, franceschi2020stochastic}.
%
Particularly, Yao \etal proposed to adapt video prediction models from multiple source domains to a target domain via distillation~\cite{yao2020unsupervised}.
%
However, it cannot be easily used as a solution to continual predictive learning, as the number of retained model parameters increases linearly with the number of tasks.
\section{Discussion}
In this paper, we explored a new research problem of continual predictive learning, which is challenging due to the co-existence of the covariate, dynamics, and target shift.
%
We proposed an approach named CPL, whose major contributions of CPL can be viewed in three aspects.
%
First, it presents a new world model to capture task-specific visual dynamics in a Gaussian mixture latent space.
%
Second, it introduces the predictive experience replay method to overcome the forgetting issue in the world model.
%
Third. it leverages a non-parametric task inference strategy to avoid coupling the forgetting issues caused by the introduction of a task inference model.
%
Our approach has shown competitive results on RoboNet and KTH benchmarks, achieving remarkable improvements over the na\"ive combinations of existing world models and CL algorithms.
Although CPL can be easily extended to more complex tasks, the potential limitation is that it has not been evaluated in the entire pipeline of vision-based robot control, which includes the processes of predictive learning and decision making.
%
In future work, we plan to integrate CPL in a model-based reinforcement learning framework to further validate its effectiveness for downstream tasks.
\section{Acknowledgement}
This work was supported by NSFC grants (U19B2035, 62106144, 62021002, 62022050), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), and Shanghai Sailing Program (21Z510202133).
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{ayub2021eec}
Ali Ayub and Alan~R Wagner.
\newblock {EEC}: Learning to encode and regenerate images for continual
learning.
\newblock In {\em ICLR}, 2021.
\bibitem{azizzadenesheli2019regularized}
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar.
\newblock Regularized learning for domain adaptation under label shifts.
\newblock {\em arXiv preprint arXiv:1903.09734}, 2019.
\bibitem{babaeizadeh2018stochastic}
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy~H Campbell, and Sergey
Levine.
\newblock Stochastic variational video prediction.
\newblock In {\em ICLR}, 2018.
\bibitem{castrejon2019improved}
Lluis Castrejon, Nicolas Ballas, and Aaron Courville.
\newblock Improved conditional {VRNNs} for video prediction.
\newblock In {\em CVPR}, pages 7608--7617, 2019.
\bibitem{cha2021co2l}
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin.
\newblock Co2l: Contrastive continual learning.
\newblock In {\em ICCV}, pages 9516--9525, 2021.
\bibitem{DBLP:conf/corl/DasariETNBSSLF19}
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher,
Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn.
\newblock Robonet: Large-scale multi-robot learning.
\newblock In {\em CoRL}, pages 885--897, 2019.
\bibitem{de2016dynamic}
Bert De~Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van~Gool.
\newblock Dynamic filter networks.
\newblock In {\em NeurIPS}, pages 667--675, 2016.
\bibitem{delange2021continual}
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales
Leonardis, Greg Slabaugh, and Tinne Tuytelaars.
\newblock A continual learning survey: Defying forgetting in classification
tasks.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
2021.
\bibitem{denton2018stochastic}
Emily Denton and Rob Fergus.
\newblock Stochastic video generation with a learned prior.
\newblock In {\em ICML}, pages 1182--1191, 2018.
\bibitem{dilokthanakul2016deep}
Nat Dilokthanakul, Pedro~AM Mediano, Marta Garnelo, Matthew~CH Lee, Hugh
Salimbeni, Kai Arulkumaran, and Murray Shanahan.
\newblock Deep unsupervised clustering with gaussian mixture variational
autoencoders.
\newblock {\em arXiv preprint arXiv:1611.02648}, 2016.
\bibitem{finn2017deep}
Chelsea Finn and Sergey Levine.
\newblock Deep visual foresight for planning robot motion.
\newblock In {\em ICRA}, pages 2786--2793, 2017.
\bibitem{franceschi2020stochastic}
Jean-Yves Franceschi, Edouard Delasalles, Micka{\"e}l Chen, Sylvain Lamprier,
and Patrick Gallinari.
\newblock Stochastic latent residual video prediction.
\newblock In {\em ICML}, pages 3233--3246, 2020.
\bibitem{goodfellow2013empirical}
Ian~J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.
\newblock An empirical investigation of catastrophic forgetting in
gradient-based neural networks.
\newblock {\em arXiv preprint arXiv:1312.6211}, 2013.
\bibitem{gretton2009covariate}
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten
Borgwardt, and Bernhard Sch{\"o}lkopf.
\newblock Covariate shift by kernel mean matching.
\newblock {\em Dataset shift in machine learning}, 3(4):5, 2009.
\bibitem{guen2020disentangling}
Vincent~Le Guen and Nicolas Thome.
\newblock Disentangling physical dynamics from unknown factors for unsupervised
video prediction.
\newblock In {\em CVPR}, pages 11474--11484, 2020.
\bibitem{guo2020ltf}
Jiaxian Guo, Mingming Gong, Tongliang Liu, Kun Zhang, and Dacheng Tao.
\newblock {LTF}: A label transformation framework for correcting label shift.
\newblock In {\em ICML}, pages 3843--3853, 2020.
\bibitem{ha2018recurrent}
David Ha and J{\"u}rgen Schmidhuber.
\newblock Recurrent world models facilitate policy evolution.
\newblock In {\em NeurIPS}, 2018.
\bibitem{hafner2020dream}
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.
\newblock Dream to control: Learning behaviors by latent imagination.
\newblock In {\em ICLR}, 2020.
\bibitem{hafner2019learning}
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha,
Honglak Lee, and James Davidson.
\newblock Learning latent dynamics for planning from pixels.
\newblock In {\em ICML}, pages 2555--2565, 2019.
\bibitem{he2018overcoming}
Xu He and Herbert Jaeger.
\newblock Overcoming catastrophic interference using conceptor-aided
backpropagation.
\newblock In {\em ICLR}, 2018.
\bibitem{iyer2014maximum}
Arun Iyer, Saketha Nath, and Sunita Sarawagi.
\newblock Maximum mean discrepancy for class ratio estimation: Convergence
bounds and kernel selection.
\newblock In {\em ICML}, pages 530--538, 2014.
\bibitem{jiang2017variational}
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou.
\newblock Variational deep embedding: An unsupervised and generative approach
to clustering.
\newblock In {\em IJCAI}, pages 1965–--1972, 2017.
\bibitem{ke2021classic}
Zixuan Ke, Bing Liu, Hu Xu, and Lei Shu.
\newblock Classic: Continual and contrastive learning of aspect sentiment
classification tasks.
\newblock In {\em EMNLP}, pages 6871--6883, 2021.
\bibitem{kirkpatrick2017overcoming}
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume
Desjardins, Andrei~A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
Grabska-Barwinska, et~al.
\newblock Overcoming catastrophic forgetting in neural networks.
\newblock {\em Proceedings of the national academy of sciences},
114(13):3521--3526, 2017.
\bibitem{lee2018stochastic}
Alex~X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and
Sergey Levine.
\newblock Stochastic adversarial video prediction.
\newblock {\em arXiv preprint arXiv:1804.01523}, 2018.
\bibitem{li2017learning}
Zhizhong Li and Derek Hoiem.
\newblock Learning without forgetting.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
40(12):2935--2947, 2017.
\bibitem{lipton2018detecting}
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola.
\newblock Detecting and correcting for label shift with black box predictors.
\newblock In {\em ICML}, pages 3122--3130, 2018.
\bibitem{liu2019transferable}
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan.
\newblock Transferable adversarial training: A general approach to adapting
deep classifiers.
\newblock In {\em ICML}, pages 4013--4022, 2019.
\bibitem{long2015learning}
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.
\newblock Learning transferable features with deep adaptation networks.
\newblock In {\em ICML}, pages 97--105, 2015.
\bibitem{long2017conditional}
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael~I Jordan.
\newblock Conditional adversarial domain adaptation.
\newblock In {\em NeurIPS}, 2017.
\bibitem{long2017deep}
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael~I Jordan.
\newblock Deep transfer learning with joint adaptation networks.
\newblock In {\em ICML}, pages 2208--2217, 2017.
\bibitem{ranzato2014video}
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan
Collobert, and Sumit Chopra.
\newblock Video (language) modeling: a baseline for generative models of
natural videos.
\newblock {\em arXiv preprint arXiv:1412.6604}, 2014.
\bibitem{rao2019continual}
Dushyant Rao, Francesco Visin, Andrei~A Rusu, Yee~Whye Teh, Razvan Pascanu, and
Raia Hadsell.
\newblock Continual unsupervised representation learning.
\newblock In {\em NeurIPS}, 2019.
\bibitem{rebuffi2017icarl}
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph~H
Lampert.
\newblock i{CaRL}: Incremental classifier and representation learning.
\newblock In {\em CVPR}, pages 2001--2010, 2017.
\bibitem{riemer2019learning}
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu,
and Gerald Tesauro.
\newblock Learning to learn without forgetting by maximizing transfer and
minimizing interference.
\newblock In {\em ICLR}, 2019.
\bibitem{rusu2016progressive}
Andrei~A Rusu, Neil~C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James
Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.
\newblock Progressive neural networks.
\newblock {\em arXiv preprint arXiv:1606.04671}, 2016.
\bibitem{DBLP:conf/icpr/SchuldtLC04}
Christian Sch{\"{u}}ldt, Ivan Laptev, and Barbara Caputo.
\newblock Recognizing human actions: {A} local {SVM} approach.
\newblock In {\em ICPR}, pages 32--36, 2004.
\bibitem{shi2015convolutional}
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and
Wang-chun Woo.
\newblock Convolutional {LSTM} network: A machine learning approach for
precipitation nowcasting.
\newblock In {\em NeurIPS}, pages 802--810, 2015.
\bibitem{shimodaira2000improving}
Hidetoshi Shimodaira.
\newblock Improving predictive inference under covariate shift by weighting the
log-likelihood function.
\newblock {\em Journal of statistical planning and inference}, 90(2):227--244,
2000.
\bibitem{shin2017continual}
Hanul Shin, Jung~Kwon Lee, Jaehong Kim, and Jiwon Kim.
\newblock Continual learning with deep generative replay.
\newblock In {\em NeurIPS}, pages 2990--2999, 2017.
\bibitem{srivastava2015unsupervised}
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.
\newblock Unsupervised learning of video representations using {LSTM}s.
\newblock In {\em ICML}, pages 843--852, 2015.
\bibitem{su2020convolutional}
Jiahao Su, Wonmin Byeon, Furong Huang, Jan Kautz, and Animashree Anandkumar.
\newblock Convolutional tensor-train lstm for spatio-temporal learning.
\newblock In {\em NeurIPS}, 2020.
\bibitem{sugiyama2007direct}
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Von~Buenau, and
Motoaki Kawanabe.
\newblock Direct importance estimation with model selection and its application
to covariate shift adaptation.
\newblock In {\em NeurIPS}, volume~7, pages 1433--1440, 2007.
\bibitem{tulyakov2018mocogan}
Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz.
\newblock {M}o{C}o{GAN}: Decomposing motion and content for video generation.
\newblock In {\em CVPR}, pages 1526--1535, 2018.
\bibitem{vondrick2016generating}
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.
\newblock Generating videos with scene dynamics.
\newblock In {\em NeurIPS}, pages 613--621, 2016.
\bibitem{wang2019eidetic}
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li.
Fei-Fei.
\newblock Eidetic 3{D} {LSTM}: A model for video prediction and beyond.
\newblock In {\em ICLR}, 2019.
\bibitem{wang2017predrnn}
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S~Yu Philip.
\newblock Pred{RNN}: Recurrent neural networks for predictive learning using
spatiotemporal lstms.
\newblock In {\em NeurIPS}, pages 879--888, 2017.
\bibitem{DBLP:conf/cvpr/WuNM0F21}
Bohan Wu, Suraj Nair, Roberto Mart{\'{\i}}n{-}Mart{\'{\i}}n, Li Fei{-}Fei, and
Chelsea Finn.
\newblock Greedy hierarchical variational autoencoders for large-scale video
prediction.
\newblock In {\em CVPR}, pages 2318--2328, 2021.
\bibitem{yao2020unsupervised}
Zhiyu Yao, Yunbo Wang, Mingsheng Long, and Jianmin Wang.
\newblock Unsupervised transfer learning for spatiotemporal predictive
networks.
\newblock In {\em ICML}, pages 10778--10788, 2020.
\bibitem{zhang2013domain}
Kun Zhang, Bernhard Sch{\"o}lkopf, Krikamol Muandet, and Zhikun Wang.
\newblock Domain adaptation under target and conditional shift.
\newblock In {\em ICML}, pages 819--827, 2013.
\end{thebibliography}