Efficient Continuous Video Flow Model for Video Prediction
Extracted source text · arXiv 2412.05633
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{mathtools,amssymb}
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{pgfplots,pgfplotstable}
\pgfplotsset{compat=1.14}
\usepackage{array,colortbl}
\usepackage{xcolor}
\usepackage{algorithm,algorithmicx,algpseudocode}
\usepackage[capitalise]{cleveref}
\usepackage{caption}
\usepackage{graphbox}
\usepackage{placeins}
\usepackage{wrapfig}
\usepackage{subcaption}
\usepackage{etoolbox}
\usepackage{amsmath}
% \DeclareMathOperator*{\argmax}{arg\,max}
% \DeclareMathOperator*{\argmin}{arg\,min}
\newtoggle{hqfigures}
\togglefalse{hqfigures} % use this for low quality figures
\newcommand{\vardbtilde}[1]{\tilde{\raisebox{0pt}[0.85\height]{$\tilde{#1}$}}}
\newcommand{\defeq}{\coloneqq}
\newcommand{\grad}{\nabla}
% \newcommand{\E}{\mathbb{E}}
% \newcommand{\Var}{\mathrm{Var}}
% \newcommand{\Cov}{\mathrm{Cov}}
\newcommand{\Ea}[1]{\E\left[#1\right]}
\newcommand{\Eb}[2]{\E_{#1}\!\left[#2\right]}
\newcommand{\Vara}[1]{\Var\left[#1\right]}
\newcommand{\Varb}[2]{\Var_{#1}\left[#2\right]}
\newcommand{\kl}[2]{D_{\mathrm{KL}}\!\left(#1 ~ \| ~ #2\right)}
% \newcommand{\pdata}{{p_\mathrm{data}}}
\newcommand{\bA}{\mathbf{A}}
\newcommand{\bI}{\mathbf{I}}
\newcommand{\bJ}{\mathbf{J}}
\newcommand{\bH}{\mathbf{H}}
\newcommand{\bL}{\mathbf{L}}
\newcommand{\bM}{\mathbf{M}}
\newcommand{\bQ}{\mathbf{Q}}
\newcommand{\bR}{\mathbf{R}}
\newcommand{\bzero}{\mathbf{0}}
\newcommand{\bone}{\mathbf{1}}
\newcommand{\bb}{\mathbf{b}}
\newcommand{\bu}{\mathbf{u}}
\newcommand{\bv}{\mathbf{v}}
\newcommand{\bw}{\mathbf{w}}
\newcommand{\bx}{\mathbf{z}^{j}}
\newcommand{\by}{\mathbf{z}^{j+1}}
\newcommand{\bz}{\mathbf{z}}
\newcommand{\bxh}{\hat{\mathbf{x}}}
\newcommand{\btheta}{{\boldsymbol{\theta}}}
\newcommand{\bphi}{{\boldsymbol{\phi}}}
\newcommand{\bepsilon}{{\boldsymbol{\epsilon}}}
\newcommand{\bmu}{{\boldsymbol{\mu}}}
\newcommand{\bnu}{{\boldsymbol{\nu}}}
\newcommand{\bSigma}{{\boldsymbol{\Sigma}}}
\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}
\title{Efficient Continuous Video Flow Model\\ for Video Prediction}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{
Gaurav Shrivastava \\
University of Maryland, College Park \\
\texttt{gauravsh@umd.edu} \\
\And
Abhinav Shrivastava \\
University of Maryland, College Park \\
\texttt{abhinav@cs.umd.edu} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
Multi-step prediction models, such as diffusion and rectified flow models, have emerged as state-of-the-art solutions for generation tasks. However, these models exhibit higher latency in sampling new frames compared to single-step methods. This latency issue becomes a significant bottleneck when adapting such methods for video prediction tasks, given that a typical 60-second video comprises approximately 1.5K frames. In this paper, we propose a novel approach to modeling the multi-step process, aimed at alleviating latency constraints and facilitating the adaptation of such processes for video prediction tasks. Our approach not only reduces the number of sample steps required to predict the next frame but also minimizes computational demands by reducing the model size to one-third of the original size. We evaluate our method on standard video prediction datasets, including KTH, BAIR action robot, Human3.6M and UCF101, demonstrating its efficacy in achieving state-of-the-art performance on these benchmarks.
\end{abstract}
\section{Introduction}
Videos serve as digital representations of the continuous real world. However, due to inherent camera limitations, particularly fixed framerate constraints, these continuous signals are discretized in the temporal domain when captured. This temporal discretization, while often overlooked in generative video modeling, presents significant challenges. Current approaches, such as diffusion-based models, generally focus on generating the next frame in a sequence based on a given context frame, neglecting intermediate moments between frames (e.g., predicting frames at $T+0.5$ or $T+0.25$ given a context frame at $T$) as represented by Figure~\ref{fig:diff_compare}. In this work, we aim to address the limitations imposed by such discretization during the generative modeling of videos.
Diffusion models have emerged as a state-of-the-art technique for video generation, but they face computational challenges, particularly in inference time. High fidelity inferences from diffusion models require hundreds or even thousands of sampling steps, which may be feasible for single-image generation but becomes prohibitive for video generation, where thousands of frames must be generated for even a short video. The naive adaptation of diffusion models for video tasks results in significant computational bottlenecks during inference, especially in production settings. To address this issue, we propose a method that reduces the need for extensive sampling by starting the process from previous context frames rather than from an analytical distribution. This continuous modeling approach not only reduces the number of sampling steps but also enhances fidelity and reduces the model's parameter count.
Starting with two consecutive frames, we pass them through an encoder network to obtain their latent embeddings. We then interpolate between these endpoints using a noise schedule that applies zero noise at the boundaries, thus ensuring the existence of $p(\mathbf{x}_t)$ at all points. This continuous framework, builds on existing diffusion models for images while extending their applicability to videos.
In summary, our contributions are as follows:
\begin{itemize}
\item We introduce a novel approach for representing videos as multi-dimensional continuous processes in latent space.
\item We empirically demonstrate that our model requires fewer sampling steps, reducing inference-time computational overhead without compromising result fidelity.
\item By reducing the memory footprint of frames in latent space, our model can predict future frames over a longer context sequence.
\item Our approach requires fewer parameters compared to state-of-the-art video prediction models.
\item We demonstrate state-of-the-art performance across several video prediction benchmarks, including KTH action recognition, BAIR robot push, Human3.6M, and UCF101 datasets.
\end{itemize}
\begin{figure}
\centering
\includegraphics[width=0.95\linewidth]{figure/Comparison.png}
% \vspace{-.4in}
\caption{Fig. (a) represents a naive adaptation of the diffusion model for the video prediction task. Here, the sampling process always starts from a Gaussian distribution, and sampling steps are taken in the direction of conditional distribution given by $X_\text{j+1}|X_j$. Here, $X_\text{j+1}$ denotes frame at time $j+1$. In contrast, Fig. (b) introduces our Continuous Video Flow (CVF) approach, which reimagines the problem by treating video not as a discrete sequence of frames but as a continuously evolving process. Instead of starting from a static Gaussian distribution for each sampling step, CVF models the underlying dynamics of the entire video, learning to predict changes smoothly over time. This continuous framework allows the model to better capture temporal coherence and evolution, leading to more accurate and fluid video predictions. }
% \vspace{-.05in}
\label{fig:diff_compare}
\end{figure}
\section{Related Works}
\label{sec:related}
Understanding and predicting future states based on observed past data~\citenum{shrivastava2021diverse,bodla2021hierarchical,shrivastava2023video,saini2022recognizing,shrivastava2024video,shrivastava2021diversethesis,shrivastava2024thesis,IonescuSminchisescu11,imagenet_cvpr09,1334462,soomro2012ucf101,fize2017geodict,roche2017valorcarn,ebert2017selfsupervised} is a cornerstone challenge in the domain of machine learning. It is crucial for video-based applications where capturing the inherent multi-modality of future states is vital, such as in autonomous vehicles. Early approaches, such as those by Yuen et al. \citep{yuen2010data} and Walker et al. \citep{walker2014patch}, tackled this problem by matching observed frames to predict future states, primarily relying on symbolic trajectories or directly retrieved future frames from datasets. While these methods laid important groundwork, their predictions were constrained by the deterministic nature of their models, limiting their ability to capture the inherent uncertainty of future frames.
The introduction of deep learning marked a significant advancement in this area. Srivastava et al. \citep{Srivastava:2015:ULV:3045118.3045209} pioneered the use of multi-layer LSTM networks, which focused on deterministic representation learning for video sequences. Building upon this, subsequent work explored more complex and stochastic methods. For instance, studies such as \citep{Oliu:2017,Cricri:2016,VillegasYHLL17,Elsayed:2019,villegas2019high,wang2018eidetic,Castrejon:2019,bodla2021hierarchical} shifted towards incorporating stochastic processes to better model the uncertain nature of future frame predictions. This transition represents a growing recognition in the field: capturing uncertainty is essential for more accurate and robust future frame prediction.
Research in video prediction has advanced along two primary directions: implicit and explicit probabilistic models. Implicit models, particularly those rooted in Generative Adversarial Networks (GANs)~\citep{goodfellow2014generative}, have been widely explored but frequently encounter difficulties with training stability and mode collapse—where the model disproportionately focuses on a limited subset of data modes~\citep{lee2018savp,clark2019adversarial,luc2020transformation}. These limitations have motivated increased interest in explicit probabilistic methods, which offer a more controlled approach to video prediction.
Explicit methods cover a spectrum of techniques, such as Variational Autoencoders (VAEs)~\citep{Kingma2013AutoEncodingVB}, Gaussian processes, and diffusion models. VAE-based approaches to video prediction~\citep{denton2018stochastic,castrejon2019improved,lee2018savp} often produce results that average multiple potential outcomes, resulting in lower-quality predictions. Gaussian process-based models~\citep{shrivastava2021diverse,bhagat2020disentangling}, while effective on small datasets, struggle with scalability due to the computational expense of matrix inversions required for training likelihood estimations. Attempts to circumvent this limitation usually result in reduced predictive accuracy.
Diffusion models~\citep{voleti2022mcvd,davtyan2023efficient,ho2022video,höppe2022diffusion} have emerged as a powerful alternative, providing high-quality samples and mitigating issues like mode collapse. These models leverage multi-step denoising, ensuring more consistent predictions. However, maintaining temporal coherence across frames requires additional constraints, such as temporal attention blocks, which can significantly increase computational demands.
An emerging method, the continuous video process~\citep{shrivastava2025video}, conceptualizes videos as continuously evolving data streams. This approach efficiently utilizes redundant information in video frames to reduce the time needed for inference. Despite its promise, this method relies on pixel-space blending between consecutive frames, leading to suboptimal redundancy exploitation. Our approach instead performs blending in the latent space, facilitating improved semantic interpolation between frames, as evidenced by previous work~\citep{shrivastava2025video}.
Finally, recent advancements such as InDI~\citep{delbracio2023inversion}, rectified flow methods~\citep{liu2022flowstraightfastlearning,esser2024scalingrectifiedflowtransformers}, and Cold Diffusion~\citep{bansal2022cold} propose alternative diffusion-based strategies, though their focus has predominantly been on image generation and computational photography. In contrast, our approach extends these ideas to video prediction, achieving a balance between computational efficiency and temporal consistency.
\begin{figure}
\centering
\vspace{-.05in}
\includegraphics[width=0.95\linewidth]{figure/pipeline1.png}
\caption{
\textbf{Overview of the Continuous Video Flow (CVF) framework.}
(a) Stage 1 depicts the auto-encoding process where input video frames $X_j$ are passed through an encoder (Enc) and decoder (Dec) to reconstruct $\hat{X_j}$.
(b) Stage 2 illustrates the forward and reverse processes in the latent space. In the forward process, latent embeddings $z_T, \dots, z_1$ are generated through a fixed process given by Eqn~\ref{eqn:fwdprocess}. The reverse process involves sampling $z_\text{j+1}$ from $z_\text{j}$ through the learned process $p_\theta$ for continuous video frame prediction. (c) The full pipeline of CVF, showing how frame $X_j$ is passed through the encoder to obtain latent embedding $z_j$ and $\hat{z}_{j+1}$, which is then used for decoding to obtain $\hat{X}_{j+1}$ frame.
}
\vspace{-.2in}
\label{fig:pipeline}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=0.98\linewidth]{figure/latent_flow_practise.pdf}
\caption{
(a) Illustration of the single-step estimation process for $\mathbf{z}_t$, where a pre-trained Encoder encodes a block of $k$ frames, highlighting the computational methodology employed.
(b) The training pipeline of the Continuous Video Process (CVP) model, where $\mathbf{z}_t$ and $t$ serve as inputs to the U-Net architecture (details in Appendix), producing the predicted output $\hat{\mathbf{z}}^{1:k+1}$.
(c) Overview of the sampling pipeline used in our approach, demonstrating the sequential prediction of the next frame in the video sequence. Given context frames in latent space$\hat{\mathbf{z}}^{0:k}$, the predicted latent $\hat{\mathbf{z}}^{k+1}$ is decoded to generate the subsequent frame $\hat{\mathbf{x}}^{k+1}$.
}
\label{fig:practical}
\end{figure}
\section{Method}
\label{sec:method}
Videos contain significant redundancy at the pixel level, and directly interpolating between frames in pixel space often results in blurry intermediate frames. To address both the redundancy and interpolation issues, we encode the frames into a latent space. This encoding offers two key benefits - First, \textbf{Compression}: Reduces irrelevant pixel information. Second, \textbf{Semantic Interpolation}: Ensures that interpolation in the latent space corresponds to more meaningful, semantic transitions between frames~\cite{shrivastava2023video}. In addition, encoding frames into the latent space allows for a larger context window when performing video prediction tasks.
\subsection{Video Prediction Framework}
Let $\mathcal{V} = \{\mathbf{x}^t\}_{t=1}^{N}$ be a video sequence where each frame $\mathbf{x}^j \in \mathbb{R}^{c \times h \times w}$ is a tensor representing the frame at timestep $j$. Our video prediction framework consists of two main stages:
\paragraph{Stage 1: Encoding in Latent Space.}
Each frame in the sequence is encoded into the latent space using a pre-trained autoencoder~\citep{rombach2022high}. The encoder processes the video sequence $\mathcal{V}$ and produces a corresponding sequence of latent representations, frame by frame.
\paragraph{Stage 2: Continuous Process in Latent Space.}
After encoding the frames $\{\mathbf{x}^j, \mathbf{x}^{j+1}\}$ into the latent space, we model a continuous video process that transitions from the latent embedding of frame $\mathbf{x}^j$ to that of frame $\mathbf{x}^{j+1}$, following the framework proposed in \cite{shrivastava2025video}. For clarity, $\mathbf{z}^j$ denotes the latent embedding of the frame at the discrete video timestep $j$, while $\mathbf{z}_t$ refers to the latent embedding at a continuous timestep $t$ during the interpolation between $\mathbf{x}^j$ and $\mathbf{x}^{j+1}$. In this context, $j$ indexes discrete video timesteps, whereas $t$ parameterizes the continuous video process. The interpolation between consecutive latent frames $\mathbf{z}^j$ and $\mathbf{z}^{j+1}$ is given by:
\begin{align}
\bz_t = (1-t)\bx + t\by -\frac{t\log(t)}{\sqrt{2}}\bepsilon
\label{eqn:intermediate}
\end{align}
where $\epsilon \sim \mathcal{N}(0,I)$ denotes white noise. This process ensures smooth transitions between frames in latent space. At $t=0$, we retrieve the latent $\bz^j$, and at $t=1$, we retrieve the latent $\bz^{j+1}$. For better understanding, refer to Fig.~\ref{fig:practical}.
\subsection{Forward Process}
We utilize a forward process to transition from latent $\bx$ to latent $\by$ over time $t$. The forward process starts at $t = T$ with latent $\by$ and moves to $t = 0$ for latent $\bx$, as described by Eqn.~\ref{eqn:intermediate} and further can be written as equation below:
\begin{align}
\bz_{t+\Delta t} = \bz_t + (\by-\bx)\Delta t - t\log(t)\bepsilon
\label{eqn:fwdprocess}
\end{align}
From this, the forward process posterior is:
\begin{align}
q(\bz_{t+1}|\bz_{t},\bx,\by) = \mathcal{N}(\bz_{t+1}; \tilde{\mu}(\bz_t, \bx, \by), g^2(t)I)
\end{align}
where $g(t) = -t\log(t)$ and $\tilde{\mu}(\bz_t, \bx, \by) = \bz_t + (\by-\bx)$.
\subsection{Likelihood and Variational Bound}
We model the continuous video process in latent space through the following likelihood function:
\begin{align}
p_\theta(\bz_T) \defeq \int p_\theta(\bz_{0:T}) \,d\bz_{0:T-1}
\end{align}
To train the model, we minimize the negative log-likelihood. The training objective involves minimizing the following variational bound:
\begin{align}
\mathbb{E}[-\log p_\theta(\bz_T)] \leq \mathbb{E}_q\left[ -\log \frac{p_\theta(\bz_{0:T})}{q(\bz_{0:T-1}|\bz_T)}\right]
\end{align}
which can be simplified following the paper~\citep{shrivastava2025video}:
\begin{align}
L(\theta) = \sum_{t \geq 1} \text{KL}(q(\bz_{t}|\bz_{t-1}, \bx, \by) || p_\theta(\bz_{t}|\bz_{t-1}, \bx))
\end{align}
Here, the KL divergence compares the forward process posterior with the reverse process model.
\subsection{Reverse Process}
We assume a Markov chain structure for the reverse process, meaning that the current state depends only on the previous timestep:
\begin{align}
p_\theta(\bz_{0:T}) = p(\bz_0) \prod_{t=1}^{T} p_\theta(\bz_{t}|\bz_{t-1})
\end{align}
The reverse process is modeled as a Gaussian distribution:
\begin{align}
p_\theta(\bz_{t}|\bz_{t-1}) = \mathcal{N}(\bz_{t}; \bmu_\theta(\bz_{t-1}, t-1), g^2(t)\bI)
\end{align}
\subsection{Training Objective}
The training objective simplifies to:
\begin{align}
L_\mathrm{simple}(\theta) \defeq \mathbb{E}_{t, \bz_t} \left[ \frac{1}{2g^2(t)} \left\| \by - \bz_\theta(\bz_t, t) \right\|^2 \right]
\label{eq:training_objective}
\end{align}
where $g(t) = -t\log(t)$ controls the noise added at each timestep. We use the interpolation function in Eqn.~\ref{eqn:intermediate} to model the intermediate frames during training.
\paragraph{Final Loss Function.}
The final training objective to optimize the video prediction model is given by:
\begin{align}
\argmin_\theta \mathbb{E}_{t, \bx, \by} \left[ \frac{1}{2g^2(t)} \left\| \by - \bz_\theta((1-t)\bx+t\by + \frac{g(t)}{\sqrt{2}}\bepsilon, t) \right\|^2 \right]
\label{eq:training_objective_simple}
\end{align}
The entire training and sampling pipeline is described in Algorithm~\ref{alg:training} and Algorithm~\ref{alg:sampling}, and visualized in Figure~\ref{fig:pipeline}.
\algrenewcommand\algorithmicindent{0.5em}%
\begin{figure}[t]
\begin{minipage}[t]{0.49\textwidth}
\begin{algorithm}[H]
\caption{Sampling Algorithm} \label{alg:sampling}
\small
\begin{algorithmic}[1]
\vspace{.04in}
\State $\bx \sim q_\text{data}(\bx)$
\State $\bz_0 = \text{Enc}(\mathbf{x})$
\State $d = \frac{1}{N},\qquad\quad$ Here $N$ denotes number of steps.
\For{$t=1, \dotsc, N$}
\State $\bepsilon \sim \mathcal{N}(\bzero, \bI d)$ if $t > 1$, else $\bepsilon = \bzero$
\State $\bz_{t+1} = \bz_{t} + (\hat{y}(\bz_t,t)-\bx)d - t\log(t) \bepsilon$
\EndFor
\State \textbf{return} \text{Dec}($\bz_T)$
% \vspace{.04in}
\end{algorithmic}
\end{algorithm}
\end{minipage}
% \end{figure}
\hfill
\begin{minipage}[t]{0.495\textwidth}
\begin{algorithm}[H]
\caption{Training of CVF model} \label{alg:training}
\small
\begin{algorithmic}[1]
\Repeat
\State $\mathbf{x},\mathbf{y} \sim q_\text{data}(\mathbf{x},\mathbf{y})$
\State $\bx, \by \sim \text{Enc}(\mathbf{x}),\text{Enc}(\mathbf{y})$
\State $t \sim \mathrm{Uniform}(\{1, \dotsc, T\})$
% \State $\bepsilon\sim\mathcal{N}(\bzero,\bI)$
\State $\bepsilon \sim \mathcal{N}(\bzero, \bI)$
\State Take gradient descent step on
\Statex $\grad_\theta \frac{1}{2g^2(t)}\left\| \by - \bz_\theta\left((1-t)\bx+t\by -(t\log(t)/\sqrt{2})\bepsilon, t\right) \right\|^2$
\Until{converged}
\end{algorithmic}
\end{algorithm}
\end{minipage}
% \vspace{-1em}
\end{figure}
\section{Experiments}
The video prediction task is defined as predicting future frames from a given sequence of context frames. In this section, we empirically validate the performance of our proposed method, demonstrating its superior results in modeling video prediction tasks across diverse datasets.
\subsection{Datasets}
We employ four standard benchmarks to evaluate the efficacy of our approach: KTH Action Recognition, BAIR Robot Pushing, Human3.6M, and UCF101. Each of these datasets presents unique challenges for video prediction, allowing for a comprehensive assessment of our method’s robustness. Training protocols and architecture specifics are provided in the appendix.
\begin{figure}[t]
\centering
\includegraphics[width = 0.85\linewidth]{figure/kth.png}
\caption{Figure represents qualitative results of our CVF model on the KTH dataset. The number of context frames used in the above setting is 4 for all three sequences. Every $4^{th}$ predicted future frame is shown in the figure. }
\label{fig:kthresults}
\end{figure}
\begin{table}[!h]%[13]{h}{9cm}
% \vspace{-.45cm}
\centering
\small
\caption{Video prediction results on KTH ($64\times64$), predicting 30 and 40 frames using models trained to predict $k$ frames at a time. All models condition on 10 past frames on 256 test videos.}
% \vspace{-.5em}
% \resizebox{.8\textwidth}{!}{
\setlength{\tabcolsep}{2pt}
\begin{tabular}{l|rc|lcl}
\toprule
\textbf{KTH} [10 $\rightarrow$ $\# \text{pred}$; trained on $k$] & $k$ & $\# \text{pred}$ & FVD$\downarrow$ & PSNR$\uparrow$ & SSIM$\uparrow$ \\ \hline
SVG-LP~\citep{denton2018stochastic} & 10 & 30 & 377 & 28.1 & 0.844 \\
SAVP~\citep{lee2018savp} & 10 & 30 & 374 & 26.5 & 0.756 \\
MCVD~\citep{voleti2022mcvd} & 5 & 30 & 323 & 27.5 & 0.835 \\
SLAMP~\citep{akan2021slamp} & 10 & 30 & 228 & 29.4 & 0.865 \\
SRVP~\citep{franceschi2020stochastic} & 10 & 30 & 222 & 29.7 & 0.870 \\
RIVER~\citep{davtyan2023efficient}& 10&30& 180& 30.4 & 0.86\\
CVP~\citep{shrivastava2025video} & 1 & 30 & 140.6& 29.8 & 0.872 \\
\textbf{CVF (Ours) } & \textbf{1} & 30 & \textbf{108.6}& 30.6 & \textbf{0.891} \\
\midrule
Struct-vRNN~\citep{minderer2019unsupervised} & 10 & 40 & 395.0 & 24.29 & 0.766 \\
SVG-LP~\citep{denton2018stochastic} & 10 & 40 & 157.9 & 23.91 & 0.800 \\
MCVD~\citep{voleti2022mcvd}& 5 & 40 & 276.7 & 26.40 & 0.812 \\
SAVP-VAE~\citep{lee2018savp} & 10 & 40 & 145.7 & 26.00 & 0.806 \\
Grid-keypoints~\citep{gao2021accurate}& 10 & 40 & 144.2 & 27.11 & 0.837 \\
RIVER~\citep{davtyan2023efficient}& 10&40& 170.5& 29.0 & 0.82\\
CVP~\citep{shrivastava2025video} & 1 & 40 & 120.1 & 29.2 & 0.841 \\
\textbf{CVF (Ours)} & \textbf{1} & 40 & \textbf{100.8} & \textbf{29.7} & \textbf{0.852} \\
\bottomrule
\end{tabular}
% }
\label{tab:KTH}
\end{table}
\noindent\textbf{KTH Action Recognition Dataset:} This dataset~\citep{1334462} contains video sequences of 25 individuals performing six actions—walking, jogging, running, boxing, hand-waving, and hand-clapping. The videos feature a uniform background with a single person performing the action in the foreground, offering relatively regular motion patterns. Each video frame is downsampled to a spatial resolution of $64\times64$ and is represented as a single-channel image.
\noindent\textbf{BAIR Robot Pushing Dataset:} The BAIR dataset~\citep{ebert2017selfsupervised} captures a Sawyer robotic arm pushing various objects on a table. It includes different robotic actions, providing a dynamic and controlled environment. The resolution of the video frames is also $64\times64$.
\noindent\textbf{Human3.6M Dataset:} Human3.6M~\citep{IonescuSminchisescu11} features 10 subjects performing 15 different actions. Unlike other works, we exclude pose information from the prediction task, focusing solely on video frame data. The dataset offers regular foreground motion against a uniform background, with video frames in RGB format and downsampled to $64\times64$.
\noindent\textbf{UCF101 Dataset:} UCF101~\citep{soomro2012ucf101} consists of 13,320 videos spanning 101 action classes. This dataset introduces a diverse range of backgrounds and actions. Each frame is resized from the original resolution of $320\times240$ to $128\times128$ using bicubic downsampling, retaining all three RGB channels.
\begin{figure}[t]
\centering
% \vspace{-.5cm}
\includegraphics[width = 0.85\linewidth]{figure/bair.png}
\caption{Figure represents qualitative results of our CVF model on the BAIR dataset. The number of context frames used in the above setting is two for both sequences. Every $6^{th}$ predicted future frame is shown in the figure.}
\label{fig:bair}
\end{figure}
\begin{table}[h]
%\vspace{-.5cm}
\small
\caption{
\emph{BAIR} dataset evaluation. Video prediction results on BAIR ($64\times64$) conditioning on $p$ past frames and predicting $pred$ frames in the future, using models trained to predict $k$ frames at at time.The common way to compute the FVD is to compare 100$\times$256 generated sequences to 256 randomly sampled test videos. Best results are marked in \textit{bold}. %(past $\rightarrow$ pred)
}
\label{tab:bair_pred}
\centering
\begin{tabular}{l|crc|c}
\toprule
\textbf{BAIR} ($64\times64$)& $p$ & $k$ & $\# \text{pred}$ & FVD$\downarrow$ \\
\cmidrule(){1-5}
LVT \citep{rakhimov2020latent} & 1 & 15 & 15 & 125.8 \\
DVD-GAN-FP \citep{clark2019adversarial} & 1 & 15 & 15 & 109.8\\
TrIVD-GAN-FP~\citep{luc2020transformation} & 1 & 15 & 15 & 103.3\\
VideoGPT~\citep{yan2021videogpt} & 1 & 15 & 15 & 103.3\\
CCVS \citep{le2021ccvs} & 1 & 15 & 15 & 99.0\\
FitVid \citep{babaeizadeh2021fitvid} & 1 & 15 & 15 & 93.6 \\
MCVD~\citep{voleti2022mcvd}& 1 & 5 & 15 & 89.5\\
NÜWA~\citep{liang2022nuwa}& 1& 15& 15 & 86.9 \\
RaMViD~\citep{höppe2022diffusion}& 1& 15& 15 & 84.2 \\
VDM~\citep{ho2022video}& 1& 15& 15 & 66.9 \\
RIVER~\citep{davtyan2023efficient} & 1& 15& 15 &73.5 \\
CVP~\citep{shrivastava2025video}&1&1&15&70.1\\
CVF (Ours) & 1& 1&15&65.8\\
\cmidrule(){1-5}
DVG~\citep{shrivastava2021diverse}& 2& 14& 14 & 120.0\\
SAVP~\citep{lee2018savp} & 2 & 14 & 14 & 116.4\\
MCVD~\citep{voleti2022mcvd} & 2 & 5 & 14 & 87.9 \\
CVP~\citep{shrivastava2025video}&2&1&14&65.1\\
\textbf{CVF(Ours)}&2&\textbf{1}&14&\textbf{61.2}\\
\cmidrule(){1-5}
SAVP~\citep{lee2018savp} & 2 & 10 & 28 & 143.4 \\
Hier-vRNN~\citep{castrejon2019improved}& 2 & 10 & 28 & 143.4 \\
MCVD~\citep{voleti2022mcvd}& 2 & 5 & 28 & 118.4 \\
CVP~\citep{shrivastava2025video}&2&\textbf{1}&28&85.1\\
\textbf{CVF (Ours)}&2&\textbf{1}&28&\textbf{78.5}\\
\bottomrule
\end{tabular}
\end{table}
\begin{table}%[13]{h}{9cm}
% \vspace{-.45cm}
\centering
\small
\caption{Comparison with baselines on number of parameters, sampling steps and sampling time required for BAIR robot push dataset.}
% \vspace{-.5em}
% \resizebox{0.8\textwidth}{!}{
\setlength{\tabcolsep}{2pt}
\begin{tabular}{l|c|cc}
\toprule
\textbf{BAIR} & $\#$ Params & $\#$ Sampling(Steps/Frame) & Time Taken(in hrs) \\ \hline
MCVD~\citep{voleti2022mcvd}& 251M & 100 & 2 \\
RaMViD~\citep{höppe2022diffusion}& 235M&500& 7.2\\
CVP~\citep{shrivastava2025video} & 118M & 25 & 0.45 \\
\textbf{CVF (Ours)} & \textbf{40M} & \textbf{5} & \textbf{0.112} \\
\bottomrule
\end{tabular}
% }
\label{tab:BAIR_eff}
\end{table}
\subsection{Metrics}
For performance evaluation, we use the Fréchet Video Distance (FVD)~\citep{unterthiner2018accurate} metric, which assesses both reconstruction quality and diversity in generated samples. The FVD computes the Fréchet distance between I3D embeddings of generated and real video samples. The I3D network is pretrained on the Kinetics-400 dataset to provide reliable embeddings for this metric.
\section{Setup and Results}
In this section, we detail the experimental setup, comparing our method to existing baselines. We then present the performance of our approach across all datasets, analyzing both quantitative metrics and qualitative results.
\noindent\textbf{KTH Action Recognition Dataset:}
For this dataset, we followed the baseline setup from \citep{shrivastava2025video}, which uses the first 10 frames as context to predict the subsequent 30 or 40 future frames. A key distinction in our experiment is that, instead of utilizing all 10 context frames, we only use the last 4 frames as context in our CVF model, deliberately discarding the first 6 frames. This choice aligns with prior methodologies and allows a fair comparison. The results of our evaluation are summarized in Table \ref{tab:KTH}.
From Table \ref{tab:KTH}, it is evident that our model achieves superior performance while requiring fewer training frames. Unlike other approaches that rely on a larger number of frames for both context and future predictions (e.g., 10 context frames plus $k$ future frames), our model operates effectively with just 4 context frames and 1 future frame. Specifically, we predict the immediate next frame using 4 context frames and then autoregressively generate 30 or 40 future frames, depending on the evaluation setting. This efficiency stems from our model’s continuous sequence processing capabilities, which avoid the need for explicit temporal attention mechanisms or artificial constraints.
The results, as presented in Table \ref{tab:KTH}, confirm that our method delivers state-of-the-art performance relative to baseline models. Qualitative results of our CVF model on the KTH dataset are shown in Fig.~\ref{fig:kthresults}.
\noindent\textbf{BAIR Robot Push Dataset:}
The BAIR Robot Push dataset is known for its highly stochastic video sequences. In line with previous studies~\citep{shrivastava2025video}, we experimented with three different setups: 1) using one context frame to predict the next 15 frames, 2) using two context frames to predict 14 future frames, and 3) using two context frames to predict 28 future frames. The results for these settings are detailed in Table \ref{tab:bair_pred}.
As shown in Table \ref{tab:bair_pred}, there is a clear trend where increasing the number of predicted frames leads to a gradual decline in prediction quality. This performance drop is likely due to an increasing mismatch between the context frames and the predicted future frames. For instance, when using two context frames, denoted as $\mathbf{x}^{0:2}$, to predict a single future frame, the prediction block is represented as $\mathbf{z}^{1:3}$, aligning with the setup in Eqn.\ref{eqn:intermediate}. In the second scenario, where two frames are predicted, the future block extends to $\mathbf{x}^{2:4}$. This setup means that in the first condition, interpolation occurs between adjacent frames (i.e., from $\bz^0\rightarrow\bz^1$ and $\bz^1\rightarrow\bz^2$), while in the second condition, interpolation spans a larger gap (e.g., $\bz^0\rightarrow\bz^2$ and $\bz^1\rightarrow\bz^3$). This extended gap likely contributes to the observed decrease in predictive performance, particularly when $k = 2$ and $p = 2$.
As indicated in Table~\ref{tab:bair_pred}, our method consistently outperforms baseline models. Qualitative results of our CVF model on the BAIR dataset can be seen in Fig.~\ref{fig:bair}.
Table~\ref{tab:BAIR_eff} highlights the superior efficiency of the proposed CVF model compared to diffusion based baselines across key metrics. CVF has the fewest parameters and requires only 5 sampling steps per frame. This makes CVF highly efficient and practical for video prediction tasks, where speed and resource efficiency are critical.
\noindent\textbf{Human3.6M Dataset:}
Like the KTH dataset, the Human3.6M dataset features actors performing distinct actions against a static background. However, Human3.6M provides a broader range of actions and includes three-channel RGB video frames, whereas KTH offers single-channel frames. For our evaluation, we adopted a setup similar to that used for KTH, providing 5 context frames and predicting the subsequent 30 future frames. The results of this evaluation are summarized in Table \ref{tab:human}.
From Table \ref{tab:human}, it is clear that our model requires significantly fewer training frames to achieve superior results. Specifically, it only uses 6 frames per block (5 context frames and 1 future frame), yet delivers performance that surpasses the baseline methods.
The results, as shown in Table \ref{tab:human}, demonstrate that our approach outperforms existing models, establishing a new state-of-the-art on the Human3.6M dataset. Additionally, the qualitative results in Fig.~\ref{fig:human} highlight our CVF model’s ability to accurately capture and predict the diverse actions within the dataset, further showcasing its effectiveness.
\begin{figure}[t]
\centering
% \vspace{-.45cm}
\includegraphics[width = 0.85\linewidth]{figure/human.png}
\caption{Figure represents qualitative results of our CVF model on the Human3.6M dataset. The number of context frames used in the above setting is 4 for all three sequences. Every $4^{th}$ predicted future frame is shown in the figure.}
\label{fig:human}
\end{figure}
\begin{table}[t]
\centering
\vspace{-.25cm}
\caption{Quantitative comparisons on the Human3.6M dataset. The best results under each metric are marked in bold.
}
\label{tab:human}
\setlength{\tabcolsep}{2pt}
% \resizebox{0.5\textwidth}{!}{
\begin{tabular}{l|rcc|c}
\toprule
\textbf{Human3.6M} &p& $k$ & $\# \text{pred}$ & FVD$\downarrow$ \\
\midrule
SVG-LP~\citep{denton2018stochastic} &5 & 10 & 30 &718 \\
Struct-VRNN~\citep{minderer2019unsupervised} &5& 10 & 30 &523.4 \\
DVG~\citep{shrivastava2021diverse}&5&10&30& 479.5 \\
SRVP~\citep{franceschi2020stochastic} &5& 10 & 30 &416.5 \\
Grid keypoint~\citep{gao2021accurate} &8& 8 & 30 & 166.1\\
CVP~\citep{shrivastava2025video} &5& 1 & 30 &144.5 \\
\textbf{CVF(Ours)} &5&1&30&\textbf{120.2} \\
\bottomrule
\end{tabular}
% }
\end{table}
\noindent\textbf{UCF101 Dataset:}
The UCF101 dataset introduces a significantly higher level of complexity compared to the KTH and Human3.6M datasets, primarily due to its large variety of action categories, diverse backgrounds, and pronounced camera movements. For our frame-conditional generation task, we strictly utilize information from the context frames, without leveraging any additional data, such as class labels, for the prediction task. Our evaluation setup mirrors that used for the Human3.6M dataset, where 5 context frames are provided, and the model is tasked with predicting the subsequent 16 frames. The results of this evaluation are presented in Table \ref{tab:UCF}.
Upon analyzing Table \ref{tab:UCF}, it is evident that our CVF model outperforms all baseline models, setting a new state-of-the-art benchmark for the UCF101 dataset. Furthermore, the qualitative results shown in Fig.~\ref{fig:ucf} demonstrate the model's ability to effectively capture and predict the diverse actions present in this challenging dataset, even without the use of external information like class labels.
\begin{figure}[t]
\centering
\vspace{-.45cm}
\includegraphics[width = 0.85\linewidth]{figure/ucf.png}
\caption{Figure represents qualitative results of our CVF model on the UCF dataset. The number of context frames used in the above setting is 5 for all three sequences. Every $4^{th}$ predicted future frame is shown in the figure.}
% \vspace{-0.1in}
\label{fig:ucf}
\end{figure}
\begin{table}%[13]{h}{9cm}
\vspace{-.25cm}
\centering
\small
\caption{Video prediction results on UCF ($128\times128$), predicting 16 frames. All models are conditioned on 5 past frames.}
% \vspace{-.5em}
\setlength{\tabcolsep}{2pt}
\begin{tabular}{l|ccc|c}
\toprule
\textbf{UCF101} [$5 \rightarrow 16$]&$p$& $k$&$\# \text{pred}$& FVD$\downarrow$ \\
\midrule
SVG-LP~\citep{denton2018stochastic} &5&10&16& 1248 \\
CCVS~\citep{le2021ccvs} &5&16&16& 409 \\
MCVD~\citep{voleti2022mcvd} &5&5&16 & 387 \\
RaMViD~\citep{höppe2022diffusion}&5&4&16& 356\\
CVP~\citep{shrivastava2025video} &5&1&16& 245.2\\
\textbf{CVF(Ours)}& 5&1&16&\textbf{221.7}\\
\bottomrule
\end{tabular}
\label{tab:UCF}
\end{table}
\section{Conclusion}
\label{sec:conclusion}
In this work, we introduced a novel model architecture designed to more effectively leverage video representations, marking a significant contribution to video prediction tasks. Through extensive experimental evaluation on diverse datasets, including KTH, BAIR, Human3.6M, and UCF101, our approach consistently demonstrated superior performance, setting new benchmarks in state-of-the-art video prediction.
A key strength of our model is its efficient utilization of parameters and significantly lower number of sampling steps during inference compared to existing methods. Notably, our model’s ability to treat video as a continuous process eliminates the need for additional constraints, such as temporal attention mechanisms, which are often used to enforce temporal consistency. This highlights the model’s inherent capacity to maintain temporal coherence naturally, streamlining the video prediction process while improving both efficiency and predictive accuracy.
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
\clearpage
\appendix
% \section{Appendix}
\begin{figure}
\centering
\begin{verbatim}
model = Gpt(
parameter_sizes=[z_dim*n], # z_dim is latent space dimension
parameter_names=['weight'],
predict_xstart=True,
absolute_loss_conditioning=False,
chunk_size=64, # cfg.transformer.chunk_size,
max_freq_log2=20,
num_frequencies=128,
n_embd=64, # cfg.transformer.n_embd,
encoder_depth=2, # cfg.transformer.encoder_depth,
decoder_depth=2, # cfg.transformer.decoder_depth,
n_layer=768, # cfg.transformer.n_layer,
n_head=16, # cfg.transformer.n_head,
attn_pdrop=0.1, # cfg.transformer.dropout_prob,
resid_pdrop=0.1, # cfg.transformer.dropout_prob,
embd_pdrop=0.1 # cfg.transformer.dropout_prob
)
\end{verbatim}
\caption{\textbf{GPT modified transformer for diffusion~\cite{Peebles2022} in latent space:} The latent embedding dimension for KTH, BAIR and Human3.6M is kept at $64$ and $128$ for the UCF101 dataset. Additionally, we keep the number of timesteps $T$ as 100 given our compute resources. $n$ is the number of initial context frames based on which next frame is predicted,i.e., $\bz^{0:n}\rightarrow\bz^{1:n+1}$. Also, for KTH, Human3.6M, and BAIR datasets we used vgg-based autoencoders~\cite{shrivastava2021diverse}. For UCF101 we used pretrained autoencoder~\cite{rombach2022high}.}
\end{figure}
\section{Training Details}
For the optimization of our model, we harnessed the compute of two Nvidia A6000 GPUs, each equipped with 48GB of memory, to train our \texttt{CVF} model effectively. We adopted a batch size of 64 and conducted training for a total of 500,000 iterations. To optimize the model parameters, we employed the AdamW optimizer. Additionally, we incorporated a cosine decay schedule for learning rate adjustment, with warm-up steps set at 10,000 iterations. The maximum learning rate (Max LR) utilized during training was 5e-5.
\section{Limitation}
\label{sec:limit}
While our method demonstrates strong performance in video prediction, it is essential to acknowledge certain limitations that point toward avenues for future work.
First, a key limitation lies in computational efficiency. Although our approach requires fewer sampling steps compared to traditional diffusion-based models, generating each frame still demands a sequential process that can become a bottleneck when scaling to longer video sequences or real-time applications. Further optimization, particularly in reducing the number of sampling steps and computational overhead, remains an open challenge.
Second, our experiments were constrained by computational resources, utilizing only two A6000 GPUs. With access to more powerful hardware or distributed computing, there may be potential for significant gains in both model complexity and performance. We encourage future research to investigate the model's behavior on larger datasets and with more substantial computational resources, as these factors could reveal additional improvements in video prediction quality.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{53}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Akan et~al.(2021)Akan, Erdem, Erdem, and G{\"u}ney]{akan2021slamp}
Adil~Kaan Akan, Erkut Erdem, Aykut Erdem, and Fatma G{\"u}ney.
\newblock Slamp: Stochastic latent appearance and motion prediction.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 14728--14737, 2021.
\bibitem[Babaeizadeh et~al.(2021)Babaeizadeh, Saffar, Nair, Levine, Finn, and Erhan]{babaeizadeh2021fitvid}
Mohammad Babaeizadeh, Mohammad~Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan.
\newblock Fitvid: Overfitting in pixel-level video prediction.
\newblock \emph{arXiv preprint arXiv:2106.13195}, 2021.
\bibitem[Bansal et~al.(2022)Bansal, Borgnia, Chu, Li, Kazemi, Huang, Goldblum, Geiping, and Goldstein]{bansal2022cold}
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie~S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein.
\newblock Cold diffusion: Inverting arbitrary image transforms without noise.
\newblock \emph{arXiv preprint arXiv:2208.09392}, 2022.
\bibitem[Bhagat et~al.(2020)Bhagat, Uppal, Yin, and Lim]{bhagat2020disentangling}
Sarthak Bhagat, Shagun Uppal, Zhuyun Yin, and Nengli Lim.
\newblock Disentangling multiple features in video sequences using gaussian processes in variational autoencoders, 2020.
\bibitem[Bodla et~al.(2021)Bodla, Shrivastava, Chellappa, and Shrivastava]{bodla2021hierarchical}
Navaneeth Bodla, Gaurav Shrivastava, Rama Chellappa, and Abhinav Shrivastava.
\newblock Hierarchical video prediction using relational layouts for human-object interactions.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 12146--12155, 2021.
\bibitem[Castrejon et~al.(2019)Castrejon, Ballas, and Courville]{castrejon2019improved}
Lluis Castrejon, Nicolas Ballas, and Aaron Courville.
\newblock Improved conditional vrnns for video prediction.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on computer vision}, pp.\ 7608--7617, 2019.
\bibitem[Castrej{\'{o}}n et~al.(2019)Castrej{\'{o}}n, Ballas, and Courville]{Castrejon:2019}
Llu{\'{\i}}s Castrej{\'{o}}n, Nicolas Ballas, and Aaron~C. Courville.
\newblock Improved conditional vrnns for video prediction.
\newblock \emph{CoRR}, abs/1904.12165, 2019.
\newblock URL \url{http://arxiv.org/abs/1904.12165}.
\bibitem[Catalin~Ionescu(2011)]{IonescuSminchisescu11}
Cristian~Sminchisescu Catalin~Ionescu, Fuxin~Li.
\newblock Latent structured models for human pose estimation.
\newblock In \emph{International Conference on Computer Vision}, 2011.
\bibitem[Clark et~al.(2019)Clark, Donahue, and Simonyan]{clark2019adversarial}
Aidan Clark, Jeff Donahue, and Karen Simonyan.
\newblock Adversarial video generation on complex datasets.
\newblock \emph{arXiv preprint arXiv:1907.06571}, 2019.
\bibitem[Cricri et~al.(2016)Cricri, Ni, Honkala, Aksu, and Gabbouj]{Cricri:2016}
Francesco Cricri, Xingyang Ni, Mikko Honkala, Emre Aksu, and Moncef Gabbouj.
\newblock Video ladder networks.
\newblock \emph{CoRR}, abs/1612.01756, 2016.
\newblock URL \url{http://arxiv.org/abs/1612.01756}.
\bibitem[Davtyan et~al.(2023)Davtyan, Sameni, and Favaro]{davtyan2023efficient}
Aram Davtyan, Sepehr Sameni, and Paolo Favaro.
\newblock Efficient video prediction via sparsely conditioned flow matching.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 23263--23274, 2023.
\bibitem[Delbracio \& Milanfar(2023)Delbracio and Milanfar]{delbracio2023inversion}
Mauricio Delbracio and Peyman Milanfar.
\newblock Inversion by direct iteration: An alternative to denoising diffusion for image restoration.
\newblock \emph{arXiv preprint arXiv:2303.11435}, 2023.
\bibitem[Deng et~al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei]{imagenet_cvpr09}
J.~Deng, W.~Dong, R.~Socher, L.-J. Li, K.~Li, and L.~Fei-Fei.
\newblock {ImageNet: A Large-Scale Hierarchical Image Database}.
\newblock In \emph{CVPR09}, 2009.
\bibitem[Denton \& Fergus(2018)Denton and Fergus]{denton2018stochastic}
Emily Denton and Rob Fergus.
\newblock Stochastic video generation with a learned prior, 2018.
\bibitem[Ebert et~al.(2017)Ebert, Finn, Lee, and Levine]{ebert2017selfsupervised}
Frederik Ebert, Chelsea Finn, Alex~X. Lee, and Sergey Levine.
\newblock Self-supervised visual planning with temporal skip connections, 2017.
\bibitem[{Elsayed} et~al.(2019){Elsayed}, {Maida}, and {Bayoumi}]{Elsayed:2019}
N.~{Elsayed}, A.~S. {Maida}, and M.~{Bayoumi}.
\newblock Reduced-gate convolutional lstm architecture for next-frame video prediction using predictive coding.
\newblock In \emph{2019 International Joint Conference on Neural Networks (IJCNN)}, pp.\ 1--9, July 2019.
\newblock \doi{10.1109/IJCNN.2019.8852480}.
\bibitem[Esser et~al.(2024)Esser, Kulal, Blattmann, Entezari, Müller, Saini, Levi, Lorenz, Sauer, Boesel, Podell, Dockhorn, English, Lacey, Goodwin, Marek, and Rombach]{esser2024scalingrectifiedflowtransformers}
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach.
\newblock Scaling rectified flow transformers for high-resolution image synthesis, 2024.
\newblock URL \url{https://arxiv.org/abs/2403.03206}.
\bibitem[Fize et~al.(2017)Fize, Shrivastava, and M{\'e}nard]{fize2017geodict}
Jacques Fize, Gaurav Shrivastava, and Pierre~Andr{\'e} M{\'e}nard.
\newblock Geodict: an integrated gazetteer.
\newblock In \emph{Proceedings of Language, Ontology, Terminology and Knowledge Structures Workshop (LOTKS 2017)}, 2017.
\bibitem[Franceschi et~al.(2020)Franceschi, Delasalles, Chen, Lamprier, and Gallinari]{franceschi2020stochastic}
Jean-Yves Franceschi, Edouard Delasalles, Micka{\"e}l Chen, Sylvain Lamprier, and Patrick Gallinari.
\newblock Stochastic latent residual video prediction.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 3233--3246. PMLR, 2020.
\bibitem[Gao et~al.(2021)Gao, Jin, Dou, Fu, and Heng]{gao2021accurate}
Xiaojie Gao, Yueming Jin, Qi~Dou, Chi-Wing Fu, and Pheng-Ann Heng.
\newblock Accurate grid keypoint learning for efficient video prediction.
\newblock In \emph{2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pp.\ 5908--5915. IEEE, 2021.
\bibitem[Goodfellow et~al.(2014)Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, and Bengio]{goodfellow2014generative}
Ian~J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
\newblock Generative adversarial networks, 2014.
\bibitem[Ho et~al.(2022)Ho, Salimans, Gritsenko, Chan, Norouzi, and Fleet]{ho2022video}
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David~J. Fleet.
\newblock Video diffusion models.
\newblock 2022.
\bibitem[Höppe et~al.(2022)Höppe, Mehrjou, Bauer, Nielsen, and Dittadi]{höppe2022diffusion}
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi.
\newblock Diffusion models for video prediction and infilling, 2022.
\bibitem[Kingma \& Welling(2013)Kingma and Welling]{Kingma2013AutoEncodingVB}
Diederik~P. Kingma and Max Welling.
\newblock Auto-encoding variational bayes.
\newblock \emph{CoRR}, abs/1312.6114, 2013.
\bibitem[Le~Moing et~al.(2021)Le~Moing, Ponce, and Schmid]{le2021ccvs}
Guillaume Le~Moing, Jean Ponce, and Cordelia Schmid.
\newblock Ccvs: context-aware controllable video synthesis.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 14042--14055, 2021.
\bibitem[Lee et~al.(2018)Lee, Zhang, Ebert, Abbeel, Finn, and Levine]{lee2018savp}
Alex~X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine.
\newblock Stochastic adversarial video prediction.
\newblock \emph{arXiv preprint arXiv:1804.01523}, 2018.
\bibitem[Liang et~al.(2022)Liang, Wu, Hu, Gan, Wang, Wang, Liu, Fang, and Duan]{liang2022nuwa}
Jian Liang, Chenfei Wu, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan.
\newblock Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 15420--15432, 2022.
\bibitem[Liu et~al.(2022)Liu, Gong, and Liu]{liu2022flowstraightfastlearning}
Xingchao Liu, Chengyue Gong, and Qiang Liu.
\newblock Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022.
\newblock URL \url{https://arxiv.org/abs/2209.03003}.
\bibitem[Luc et~al.(2020)Luc, Clark, Dieleman, Casas, Doron, Cassirer, and Simonyan]{luc2020transformation}
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de~Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan.
\newblock Transformation-based adversarial video prediction on large-scale data.
\newblock \emph{arXiv preprint arXiv:2003.04035}, 2020.
\bibitem[Minderer et~al.(2019)Minderer, Sun, Villegas, Cole, Murphy, and Lee]{minderer2019unsupervised}
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin~P Murphy, and Honglak Lee.
\newblock Unsupervised learning of object structure and dynamics from videos.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Oliu et~al.(2017)Oliu, Selva, and Escalera]{Oliu:2017}
Marc Oliu, Javier Selva, and Sergio Escalera.
\newblock Folded recurrent neural networks for future video prediction.
\newblock \emph{CoRR}, abs/1712.00311, 2017.
\newblock URL \url{http://arxiv.org/abs/1712.00311}.
\bibitem[Peebles et~al.(2022)Peebles, Radosavovic, Brooks, Efros, and Malik]{Peebles2022}
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei Efros, and Jitendra Malik.
\newblock Learning to learn with generative models of neural network checkpoints.
\newblock \emph{arXiv preprint arXiv:2209.12892}, 2022.
\bibitem[Rakhimov et~al.(2020)Rakhimov, Volkhonskiy, Artemov, Zorin, and Burnaev]{rakhimov2020latent}
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev.
\newblock Latent video transformer.
\newblock \emph{arXiv preprint arXiv:2006.10704}, 2020.
\bibitem[Roche et~al.(2017)Roche, Teisseire, and Shrivastava]{roche2017valorcarn}
Mathieu Roche, Maguelonne Teisseire, and Gaurav Shrivastava.
\newblock Valorcarn-tetis: Terms extracted with biotex.
\newblock 2017.
\bibitem[Rombach et~al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer]{rombach2022high}
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj{\"o}rn Ommer.
\newblock High-resolution image synthesis with latent diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pp.\ 10684--10695, 2022.
\bibitem[Saini et~al.(2022)Saini, He, Shrivastava, Rambhatla, and Shrivastava]{saini2022recognizing}
Nirat Saini, Bo~He, Gaurav Shrivastava, Sai~Saketh Rambhatla, and Abhinav Shrivastava.
\newblock Recognizing actions using object states.
\newblock In \emph{ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality}, 2022.
\bibitem[{Schuldt} et~al.(2004){Schuldt}, {Laptev}, and {Caputo}]{1334462}
C.~{Schuldt}, I.~{Laptev}, and B.~{Caputo}.
\newblock Recognizing human actions: a local svm approach.
\newblock In \emph{Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004.}, volume~3, pp.\ 32--36 Vol.3, Aug 2004.
\newblock \doi{10.1109/ICPR.2004.1334462}.
\bibitem[Shrivastava(2021)]{shrivastava2021diversethesis}
Gaurav Shrivastava.
\newblock \emph{Diverse Video Generation}.
\newblock PhD thesis, University of Maryland, College Park, 2021.
\bibitem[Shrivastava(2024)]{shrivastava2024thesis}
Gaurav Shrivastava.
\newblock \emph{Advanced video modeling techniques for generation and enhancement tasks}.
\newblock PhD thesis, University of Maryland, College Park, 2024.
\bibitem[Shrivastava \& Shrivastava(2021)Shrivastava and Shrivastava]{shrivastava2021diverse}
Gaurav Shrivastava and Abhinav Shrivastava.
\newblock Diverse video generation using a gaussian process trigger.
\newblock \emph{arXiv preprint arXiv:2107.04619}, 2021.
\bibitem[Shrivastava \& Shrivastava(2024)Shrivastava and Shrivastava]{shrivastava2025video}
Gaurav Shrivastava and Abhinav Shrivastava.
\newblock Video prediction by modeling videos as continuous multi-dimensional processes.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 7236--7245, 2024.
\bibitem[Shrivastava et~al.(2023)Shrivastava, Lim, and Shrivastava]{shrivastava2023video}
Gaurav Shrivastava, Ser-Nam Lim, and Abhinav Shrivastava.
\newblock Video dynamics prior: An internal learning approach for robust video enhancements.
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\newblock URL \url{https://openreview.net/forum?id=CCq73CGMyV}.
\bibitem[Shrivastava et~al.(2024)Shrivastava, Lim, and Shrivastava]{shrivastava2024video}
Gaurav Shrivastava, Ser-Nam Lim, and Abhinav Shrivastava.
\newblock Video decomposition prior: Editing videos layer by layer.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=nfMyERXNru}.
\bibitem[Soomro et~al.(2012)Soomro, Zamir, and Shah]{soomro2012ucf101}
Khurram Soomro, Amir~Roshan Zamir, and Mubarak Shah.
\newblock Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
\bibitem[Srivastava et~al.(2015)Srivastava, Mansimov, and Salakhutdinov]{Srivastava:2015:ULV:3045118.3045209}
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.
\newblock Unsupervised learning of video representations using lstms.
\newblock In \emph{Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37}, ICML'15, pp.\ 843--852. JMLR.org, 2015.
\newblock URL \url{http://dl.acm.org/citation.cfm?id=3045118.3045209}.
\bibitem[Unterthiner et~al.(2018)Unterthiner, van Steenkiste, Kurach, Marinier, Michalski, and Gelly]{unterthiner2018accurate}
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.
\newblock Towards accurate generative models of video: A new metric and challenges, 2018.
\bibitem[Villegas et~al.(2017)Villegas, Yang, Hong, Lin, and Lee]{VillegasYHLL17}
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee.
\newblock Decomposing motion and content for natural video sequence prediction.
\newblock \emph{CoRR}, abs/1706.08033, 2017.
\newblock URL \url{http://arxiv.org/abs/1706.08033}.
\bibitem[Villegas et~al.(2019)Villegas, Pathak, Kannan, Erhan, Le, and Lee]{villegas2019high}
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc~V. Le, and Honglak Lee.
\newblock High fidelity video prediction with large stochastic recurrent neural networks, 2019.
\bibitem[Voleti et~al.(2022)Voleti, Jolicoeur-Martineau, and Pal]{voleti2022mcvd}
Vikram Voleti, Alexia Jolicoeur-Martineau, and Chris Pal.
\newblock Mcvd-masked conditional video diffusion for prediction, generation, and interpolation.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 23371--23385, 2022.
\bibitem[Walker et~al.(2014)Walker, Gupta, and Hebert]{walker2014patch}
Jacob Walker, Abhinav Gupta, and Martial Hebert.
\newblock Patch to the future: Unsupervised visual prediction.
\newblock In \emph{Proceedings of the IEEE conference on Computer Vision and Pattern Recognition}, pp.\ 3302--3309, 2014.
\bibitem[Wang et~al.(2019)Wang, Jiang, Yang, Li, Long, and Fei-Fei]{wang2018eidetic}
Yunbo Wang, Lu~Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li~Fei-Fei.
\newblock Eidetic 3d {LSTM}: A model for video prediction and beyond.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\newblock URL \url{https://openreview.net/forum?id=B1lKS2AqtX}.
\bibitem[Yan et~al.(2021)Yan, Zhang, Abbeel, and Srinivas]{yan2021videogpt}
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.
\newblock Videogpt: Video generation using vq-vae and transformers.
\newblock \emph{arXiv preprint arXiv:2104.10157}, 2021.
\bibitem[Yuen \& Torralba(2010)Yuen and Torralba]{yuen2010data}
Jenny Yuen and Antonio Torralba.
\newblock A data-driven approach for event prediction.
\newblock In \emph{European Conference on Computer Vision}, pp.\ 707--720. Springer, 2010.
\end{thebibliography}