ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos
Extracted source text · arXiv 2505.18650
% CVPR 2025 Paper Template; see https://github.com/cvpr-org/author-kit
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
% \usepackage{cvpr} % To produce the CAMERA-READY version
% \usepackage[review]{cvpr} % To produce the REVIEW version
\usepackage[pagenumbers]{cvpr} % To force page numbers, e.g. for an arXiv version
% Import additional packages in the preamble file, before hyperref
\input{preamble}
% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues,
% e.g. with the file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete *.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you should be clear).
\definecolor{cvprblue}{rgb}{0.21,0.49,0.74}
\usepackage[pagebackref,breaklinks,colorlinks,allcolors=cvprblue]{hyperref}
\usepackage{xcolor}
%%%%%%%%% PAPER ID - PLEASE UPDATE
% \def\paperID{16271} % *** Enter the Paper ID here
\def\confName{CVPR}
\def\confYear{2025}
%%%%%%%%% TITLE - PLEASE UPDATE
\title{ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos}
%%%%%%%%% AUTHORS - PLEASE UPDATE
% \author{Xiaodong Wang\\
% Institution1\\
% Institution1 address\\
% {\tt\small firstauthor@i1.org}
% % For a paper whose authors are all at the same institution,
% % omit the following lines up until the closing ``}''.
% % Additional authors and addresses can be added with ``\and'',
% % just like the second author.
% % To save space, use either the email address or home page, not both
% \and
% Second Author\\
% Institution2\\
% First line of institution2 address\\
% {\tt\small secondauthor@i2.org}
% }
\author{Xiaodong Wang$^{1,2}$ \quad Peixi Peng$^{1,2}$\thanks{Corresponding author.} \\
{$^{1}$Peking University \quad $^{2}$Peng Cheng Laboratory} \\
{\tt\small\{wangxiaodong21s@stu., pxpeng\}@pku.edu.cn}
}
\begin{document}
\maketitle
\input{sec/0_abstract}
\input{sec/1_intro}
\input{sec/2_related}
\input{sec/3_method}
\input{sec/4_exp}
\input{sec/5_conclusion}
\input{sec/X_suppl}
{
\small
\bibliographystyle{ieeenat_fullname}
\bibliography{main}
}
% WARNING: do not forget to delete the supplementary pages from your submission
\end{document}
%
% --- inline annotations
%
\newcommand{\red}[1]{{\color{red}#1}}
\newcommand{\todo}[1]{{\color{red}#1}}
\newcommand{\TODO}[1]{\textbf{\color{red}[TODO: #1]}}
% --- disable by uncommenting
% \renewcommand{\TODO}[1]{}
% \renewcommand{\todo}[1]{#1}
\begin{abstract}
Real-world driving requires people to observe the current environment, anticipate the future, and make appropriate driving decisions. This requirement is aligned well with the capabilities of world models, which understand the environment and predict the future. However, recent world models in autonomous driving are built explicitly, where they could predict the future by controllable driving video generation. We argue that driving world models should have two additional abilities: action control and action prediction. Following this line, previous methods are limited because they predict the video requires given actions of the same length as the video and ignore the dynamical action laws. To address these issues, we propose ProphetDWM, a novel end-to-end driving world model that jointly predicts future videos and actions. Our world model has an action module to learn latent action from the present to the future period by giving the action sequence and observations. And a diffusion-model-based transition module to learn the state distribution. The model is jointly trained by learning latent actions given finite states and predicting action and video. The joint learning connects the action dynamics and states and enables long-term future prediction. We evaluate our method in video generation and action prediction tasks on the Nuscenes dataset. Compared to the state-of-the-art methods, our method achieves the best video consistency and best action prediction accuracy, while also enabling high-quality long-term video and action generation.
%Currently, video diffusion models make imagining the future easier. But making synchronous driving decisions is intricate. Existing methods usually use a two-stage pipeline to accomplish this goal. They first generate future frames or features and then make action predictions based on the history and future frames or features. However, they rely on complete future actions to generate the corresponding future frames and can only predict future actions after the video prediction, lacking synchronization. To address these issues, we propose ProphetDWM, a novel end-to-end driving world model that can synchronously predict the corresponding future frames and actions. We design an action n-gram prediction module based on the Large Language Model architecture and a diffusion-based video generation module. Given a reference frame and a few reference actions, the action module encodes the actions and generates an n-gram representation, which synchronously guides the generation of future frames and the prediction of actions. We evaluate our method in video generation and action prediction tasks on the NuScenes dataset. Compared to the state-of-the-art methods, our method achieves the best video consistency and comparable action prediction, while also enabling long-term video and action generation.
%Real-world driving requires people to observe the current environment, anticipate the future, and make appropriate driving decisions, with synchronous anticipation and decision-making being crucial. Currently, video diffusion models make imagining the future easier. But making synchronous driving decisions is intricate. Existing methods usually use a two-stage pipeline to accomplish this goal. They first generate future frames or features and then make action predictions based on the history and future frames or features. However, they rely on complete future actions to generate the corresponding future frames and can only predict future actions after the video prediction, lacking synchronization. To address these issues, we propose ProphetDWM, a novel end-to-end driving world model that can synchronously predict the corresponding future frames and actions. We design an action n-gram prediction module based on the Large Language Model architecture and a diffusion-based video generation module. Given a reference frame and a few reference actions, the action module encodes the actions and generates an n-gram representation, which synchronously guides the generation of future frames and the prediction of actions. We evaluate our method in video generation and action prediction tasks on the NuScenes dataset. Compared to the state-of-the-art methods, our method achieves the best video consistency and comparable action prediction, while also enabling long-term video and action generation. %An anonymous link showing video examples is at \url{https://prophetdwm.github.io/}.
\end{abstract}
\section{Introduction}
\label{sec:intro}
% World models have recently gained huge attraction, especially after the success of large-scale video generation training.
World models are expected to understand the environment and predict the future conditioned on given actions \cite{lecun2022worldmodel,ha2018world}.
Early works often model the world models in latent feature space and have been validated on games or simulators \cite{ebert2018visual,dosovitskiy2017carla}. Recently, inspired by the success of video generation ~\cite{openai2023sora}, several cutting-edge approaches \cite{hu2023gaia,wang2023driveWM,gao2024vista,yang2024genad} aim to build driving world models in an explicit way. That is, they could predict the future by controllable driving video generation and have shown promising results.
% which has been explored early in latent feature space. and validated on game
% simulation robotic~\cite{ebert2018visual} and driving environments~\cite{dosovitskiy2017carla}.
% Recently,
% a kind of definition of a world model is to purely predict the future. % especially developed in video generation tasks.
% For example, a recent cutting-edge technology Sora~\cite{openai2023sora} can be regarded as a world model, which can simulate the physical world. Moreover, world models for autonomous driving have shown promising results~\cite{hu2023gaia,wang2023driveWM}.% some works also explored future action prediction~\cite{wang2023drivedreamer,jia2023adriver}. %Equipping with the diverse video generation ability and action prediction ability, driving world models can work as a simulator and help to make decisions in some real-world driving scenarios.
\begin{table*}[]
\centering
\begin{tabular}{lcccc}
\toprule
Method & Video Prediction & Action Control & Action Prediction & One-stage \\
\midrule
DriveGAN~\cite{kim2021drivegan} & \checkmark & & \\
DriveDreamer~\cite{wang2023drivedreamer} & \checkmark & \checkmark & \checkmark & \\
Drive-WM~\cite{wang2023driveWM} & \checkmark & & & \\
ADriver-I~\cite{jia2023adriver} & \checkmark & & \checkmark & \\
GenAD~\cite{yang2024genad} & \checkmark & & & \\
Vista~\cite{gao2024vista} & \checkmark & \checkmark & & \\
\midrule
Ours & \checkmark & \checkmark & \checkmark & \checkmark \\
\bottomrule
\end{tabular}
\caption{Driving world model comparison. }
\label{tab:dwm}
\end{table*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{sec/intro.pdf}
\caption{The comparison of video-action generation paradigms. We compare our method with previous methods that can perform action prediction. ADriver-I~\cite{jia2023adriver} and DriveDreamer~\cite{wang2023drivedreamer} belong to two-stage methods, because they need first predict the video, and then predict the subsequent actions, and their video prediction are conditioned on corresponding actions. In contrast, our method learns from the given action sequence and current observation and jointly predicts future actions and video in a one-stage manner. }
\label{fig:intro}
\end{figure*}
%Early work~\cite{kim2021drivegan} proposed a GAN-based method to control the driving action for each timestep and autoregressively generate future videos frame by frame.
Thanks to the development of diffusion models~\cite{ho2020ddpm,rombach2022sdm}, video prediction for driving scenarios is not very hard~\cite{blattmann2023align,kim2021drivegan,wang2023driveWM,jia2023adriver,yang2024genad}. Besides, we argue that the driving world model should have two additional abilities: 1) Action control: different from pure video generation, world models are more concerned with the counterfactual reasoning of actions. That is, the generated videos should exactly reflect the future differences of various given actions; 2) Action prediction: The world model is not only used to visualize the future as video generation methods do, but also plan the near future action simultaneously. The accurate action prediction is even more important in autonomous driving tasks.
Following this line, a comparison of recent driving world models is summarized in Table~\ref{tab:dwm}, where the action refers to low-level driving action (such as steering angle and speed). Specifically, models~\cite{gao2024vista,wang2023driveWM,kim2021drivegan} only focus on video generation, i.e., a kind of conditional prediction, given commands, trajectories, or low-level actions of every frame in the future, and ignore the action prediction. DriveDreamer~\cite{wang2023drivedreamer} and ADrive-I~\cite{jia2023adriver} are more similar to this work. However, they separate the predictions of video and action, and both predict video first and then use the predicted video to predict action. We argue this paradigm should be improved for two reasons: Firstly, the length of the first video generation must be the same as the length of the given actions. Once the latter is limited, the long prediction must generated by multiple turns, and error accumulation may occur. Secondly, these methods only utilize the action sequence as a condition and ignore the dynamical action laws, which exhibit why such an action sequence is produced. Hence, as shown in Figure~\ref{fig:intro}, we propose a novel method that enables the driving world model to jointly learn to predict future actions and videos from given actions and observations.
% which is not in line with the idea of the world model learning from historical sequences and states to plan the future. Therefore, how to plan and predict the future directly from historical sequences and states has not been systematically explored in the field of driving world models.
% . However, learning to plan is a key difference between world models and video prediction models. A comparison of recent driving world models is summarized in Table~\ref{tab:dwm}, where the action refers to low-level driving action (such as steering angle and speed). Existing driving world models are all equipped with video prediction ability, and some can control future video generation by action inputs. DriveDreamer~\cite{wang2023drivedreamer} and ADrive-I~\cite{jia2023adriver} are more similar to this work since they can predict future actions. However, they separate the prediction of video and the prediction of action, and they both predict video first and then use video to predict action, which is not in line with the idea of the world model learning from historical sequences and states to plan the future. Therefore, how to plan and predict the future directly from historical sequences and states has not been systematically explored in the field of driving world models.
% That is, the video generation should have action controllable ability. To achieve this goal,
% To develop a driving world model by video generation, a natural way is to utilize diffusion models~\cite{ho2020ddpm,rombach2022sdm}.
% and introduce took text, 3Dbox, HDmap, or driving actions as conditions to make the generated video controllable. Specifically, these methods ~\cite{gao2024vista,wang2023driveWM,kim2021drivegan}
% To develop a driving world model, most works delve into the more controllable driving video generation.
% Most works utilized diffusion models~\cite{ho2020ddpm,rombach2022sdm} as the video generator and took text, 3Dbox, HDmap, or driving actions as conditions to improve the visual quality. Thanks to the development of diffusion models, video prediction for driving scenarios is not very hard~\cite{blattmann2023align}. Moreover, given such fine-grained layout conditions (3Dbox, HDmap), existing methods can predict more consistent future videos, and given future actions, the generated videos can align with those actions~\cite{wang2023drivedreamer}.
%and some works can also predict future actions~\cite{wang2023drivedreamer,jia2023adriver}. Similar to a human driver, a driving world model should have two abilities: \emph{foresee} and \emph{plan}, foresee the near future and plan the driving actions.
% Most previous driving world models~\cite{gao2024vista,wang2023driveWM,kim2021drivegan} only focus on video generation, i.e., a kind of conditional prediction, given commands, trajectories, or low-level actions of every frame in the future. However, learning to plan is a key difference between world models and video prediction models. A comparison of recent driving world models is summarized in Table~\ref{tab:dwm}, where the action refers to low-level driving action (such as steering angle and speed). Existing driving world models are all equipped with video prediction ability, and some can control future video generation by action inputs. DriveDreamer~\cite{wang2023drivedreamer} and ADrive-I~\cite{jia2023adriver} are more similar to this work since they can predict future actions. However, they separate the prediction of video and the prediction of action, and they both predict video first and then use video to predict action, which is not in line with the idea of the world model learning from historical sequences and states to plan the future. Therefore, how to plan and predict the future directly from historical sequences and states has not been systematically explored in the field of driving world models.
% To clarify the difference, we first compare these methods ADriver-I~\cite{jia2023adriver} and DriveDreamer~\cite{wang2023drivedreamer} with our method as shown in Figure~\ref{fig:intro}. The first difference lies in whether all future frames are given corresponding actions, if so, then the video prediction part is only conditional generation. So, video prediction tasks in ADriver-I and DriveDreamer belong to a conditional generation task, where ADriver-I utilizes GPT-3.5 to convert the future low-level actions to a motion prompt and feeds it in the diffusion model to generate future frames, and DriveDreamer directly encodes all the future actions as prior features in the diffusion model to generate corresponding future frames. In contrast, our model only receives very few future actions that indicate the dynamics because using only a single image cannot reason with the current driving state. Our model is encouraged to learn from the action sequences and observation to estimate a latent action, which is the dynamics prior to the future frame generation. The second difference is whether the video and actions are jointly learned, which is similar to the policy learning in the traditional world models that predict the actions, future states, and rewards and then maximize imagined rewards~\cite{lin2023dynalang,hafner2023dreamerv3,hafner2020dreamerv2,hafner2019dreamer}. Although ADriver-I and DriveDreamer can predict future actions, they both predict the video first and then, based on the dynamic information of the video, they further predict actions. In our work, we propose a novel method that enables the driving world model to jointly learn to predict future actions and videos from given actions and observations. We want to transfer the idea of joint prediction from the policy learning of traditional world models to generative models such as diffusion models. So, we build a driving world model in a one-stage prediction manner.
%In this work, we delve into driving world models that can predict both videos and actions. There are some drawbacks to the mainstream paradigms. As shown in Fig.~\ref{fig:intro}, we compare two methods, DriveDreamer~\cite{wang2023drivedreamer} and ADriver-I~\cite{jia2023adriver}, that can generate videos and predict actions. The first issue is that they both rely on future ground-truth actions to generate corresponding future frames. DriveDreamer receives a reference image and future actions and only generates the future frames of the same time sequence. ADriver-I receives multiple frames and future actions. It first generates future frames and then utilizes total history actions and frames to predict the next action. Since the action input during the diffusion model training is the same length as the fitted video frames, they can not infer future frames if no future action is given. The second issue is that they only predict future actions after the video generation is done. DriveDreamer first needs to obtain video latent features and then concatenate them with actions to predict future actions. ADriver-I requires decoding latent features into future frames and then inputting them back into itself to output a future action each timestep. In that way, they need to pass through the model multiple times in inference. %they need to pass through the model multiple times to infer multiple frames and multiple actions.
To this end, we consider the controllable driving task to be a finite-horizon, partially observable Markov decision process (POMDP) \cite{smallwood1973pomdp}. Video in the driving process is considered to be observation, and low-level driving actions are considered to be actions performed by agents. To model the POMDP, we construct a driving world model named ProphetDWM that predicts future states and actions based on a sequence of given states and actions. Similar to most world models in latent feature space ~\cite{lin2023dynalang,hafner2023dreamerv3,hafner2020dreamerv2}, we also design two important components to learn the states representation and latent actions. Firstly, we propose an action module that aims to learn the latent action from the present moment to the future period. The action module is lightweight and only stacked with MLP (Multilayer Perceptron). It could not only learn the dynamics information from given actions but also incorporate the states of current observation to learn a sequence of latent actions, and the latent actions are decoded into low-level actions, which are supervised by the ground-truth actions. Second, we leverage the state-of-the-art diffusion models as a transition model to learn the state distribution, where the states are converted from image observations by a VAE (Variational Autoencoder) encoder. Besides the self-attention among the sequence states in diffusion models, we design a short pathway that injects the state context to the hidden states inside the diffusion model, i.e., the latent action extracted from the action module, which aims to promote the quality and consistency of predictions. Finally, the two components are trained jointly to predict the actions and videos.
To sum up, the major contributions of this paper are:
\begin{itemize}
\item We propose ProphetDWM, an end-to-end driving world model that jointly learns to predict future actions and video. It is not a conditional low-level action-to-video model, but is trained by learning latent actions given finite states and predicting action and video.
\item The world model has an action module to learn latent actions from the present moment to the future period by giving the action sequence and observations and a diffusion-model-based transition model to learn the state distribution. The world model is trained to predict the actions and videos jointly.
%We designed an action module based on LLM (Large Language Model) architecture, which encodes the given actions and extrapolates n-gram future representations. A tailored diffusion module that equips fusion layers to perceive the reference frame and temporal cross-attention layers that query the information from future action representations.
\item We evaluate ProphetDWM in video generation and action prediction tasks on Nuscenes. Compared with state-of-the-art methods, our method achieves the best video consistency and action accuracy and can roll out high-quality long-term videos and actions.
\end{itemize}
% \begin{figure*}
% \centering
% \begin{subfigure}{0.68\linewidth}
% \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
% \caption{An example of a subfigure.}
% \label{fig:short-a}
% \end{subfigure}
% \hfill
% \begin{subfigure}{0.28\linewidth}
% \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
% \caption{Another example of a subfigure.}
% \label{fig:short-b}
% \end{subfigure}
% \caption{Example of a short caption, which should be centered.}
% \label{fig:short}
% \end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{sec/main.pdf}
\caption{Schematic diagram of proposed ProphetDWM. Our world model has an action module to learn the latent action and a video prediction module to learn state distribution. All modules are optimized together to jointly predict future actions and videos.}
\label{fig:main}
\end{figure*}
\section{Related Work}
\label{sec:relat}
\subsection{World Model}
The typical world model~\cite{lecun2022worldmodel} learns a general representation of the world and predicts future world states from history actions. Learning typical world models in the game~\cite{ha2018recurrent,hafner2019dreamer}, lab environments~\cite{ebert2018visual,tassa2018dmc}, or multimodal environments~\cite{lin2023learninglanguage} has been widely studied. Dreamer~\cite{wu2023daydreamer} and DreamerV2~\cite{hafner2020dreamerv2} learned a latent dynamic model to predict state values and actions in a latent space, which effectively handles the visual control tasks such as DeepMind Control Suite~\cite{tassa2018dmc}. MILE~\cite{hu2022mile} utilized imitation learning to learn a dynamic model and driving behavior in Carla~\cite{dosovitskiy2017carla}. Recently, a kind of definition of a world model is to purely predict the future~\cite{hu2023gaia,wang2024worlddreamer}. Moreover, driving world models in the real driving datasets have been explored~\cite{wang2023drivedreamer,jia2023adriver,wang2023driveWM,zhao2024drivedreamer2}. DriveDreamer~\cite{wang2023drivedreamer} trained a video diffusion model to receive conditions such as HDmap, 3Dbox, and actions to generate future driving videos. GAIA-1~\cite{hu2023gaia} designed a versatile diffusion model that learns image generation, video generation, and video interpolation. ADriver-I~\cite{jia2023adriver} leveraged a video diffusion model and a multi-modal language model that first generates a sequence of frames and then predicts the next action based on history action and all frames. Panacea~\cite{wen2023panacea} introduced BEV layouts guided video generation by integrating a 4D attention in U-Net. DriveWM~\cite{wang2023driveWM} took an innovative step towards generating multi-view videos. However, these previous works rely on future conditions to generate corresponding frames and failed to predict video and actions synchronously. In this work, we stepped towards joint action and video prediction.
\subsection{Video Generation}
Recent works in video generation focus on text-to-video generation, and the mainstream method is to pre-train a diffusion model on large-scale video-text datasets. Video-LDM~\cite{blattmann2023align} turned a pre-trained image LDM into a video generator by introducing temporal layers, which can also handle the generation of driving scenes. VideoPoet~\cite{kondratyuk2023videopoet} utilized a decoder-only transformer capable of processing multi-modal inputs, enabling zero-shot video generation. MagicVideo2~\cite{wang2024magicvideo} utilized a cascaded diffusion model that separates image and video generation. Lumiere~\cite{bar2024lumiere} designed a space-time U-Net that can directly generate the entire temporal duration of the video through a single pass in the model. MagicTime~\cite{yuan2024magictime} incorporated more physics knowledge and able to generate metamorphic videos. In this work, we designed a transition model based on the diffusion model that receives the learned prior from an action sequence and observations.
\section{Method}
\label{sec:method}
\subsection{World Model Learning}
We consider the controllable driving task to be a finite-horizon, partially observable Markov decision process (POMDP~\cite{smallwood1973pomdp}). Visual observation, action space, and time horizon are denoted as $\mathcal{O}$, $\mathcal{A}$, and $\mathcal{T}$, respectively. \footnote{We don't consider reward here because it is correlated with specific task, and the reward often could be calculated by observations.}
% We define the state as the historical observations and actions, $s_t=\{o_\tau, a_\tau\}_{\tau< t}$.
In this setting, the collected data in real-world autonomous driving include an agent (ego vehicle) that performs continuous actions $a_t$ on $o_t$ and receives $o_{t+1} \sim p(o_{t+1}|o_t, a_t)$.
% Here, $(o\leq t, a < t)$ are used to represent $\{o_\tau, a_\tau\}_{\tau< t}$ for simplification.
In many realistic tasks such as driving, the POMDP could be observed only, and just $\{o_\tau, a_\tau\}_{\tau< t}$ is known. To evaluate the subsequent effects of different policies (i.e., action sequence), the world model is proposed to predict the future observations $o_{t+1:t+\Delta T}$ based on given action sequence $a_{t+1,t+\eta}$. In our setting, the length of the given action sequence is always much shorter than the length of required observation prediction ($\eta<\Delta T$), which is the main difference compared to previous methods \cite{wang2023drivedreamer,gao2024vista}.
%\sim p(a_t|o\leq t, a < t)
% Considering the length of given action sequence is always much shorter than the length of required observation prediction ($\eta<k$), we design an action model to predict $\{\tilde a_{t+\eta},...a_{t+n}\}$, and
% Hence, we design a predictive transition model $s_t \sim q(s_t|s_{t-1}, a_{t-1})$
% To model the POMDP, we aim to lean the transition model $o_t \sim p(o_t|o< t, a< t)$ to prediction, and predictive action model $a_t \sim q(a_t|o\leq t, a_{< t})$
% we design a predictive transition model $s_t \sim q(s_t|s_{t-1}, a_{t-1})$, and the predictive action model $a_t \sim q(a_t|s_{t-1}, a_{< t})$, where $s$ denotes the state of the environment, and the estimated state $\hat{s}$ from an observation $o$ can be denoted as $\hat{s}=p_{\phi}(o_t)$ using a vision encoder. This task aims to build a promising transition model to predict the future and a decision model to predict the action.
% Our main contribution lies in constructing a world model for a real-world driving environment. The goal of this world model is different from traditional world models~\cite{hafner2019dreamer,hafner2020dreamerv2,hafner2023dreamerv3}, instead of maximizing some predefined rewards, our goal aims to better predict the future scenes and predict more precise future actions when driving, because measuring rewards of the open-loop driving environment is non-trivial.
% and a simple light-weight MLP (Multilayer Perceptron) to directly predict actions
Since the goal of this model is expected to be a predictive task, and there are some benchmarks (such as Nuscenes~\cite{caesar2020nuscenes}, Waymo~\cite{waymo}) that collect multi-modal data from real-world autonomous driving environments, we can use supervised training and state-of-the-art models to model the distribution of the driving scenes and actions, and then easily to predict the future using the learned distribution. We choose the diffusion model~\cite{ho2020ddpm,rombach2022sdm} as the backbone to model the visual feature distribution.
Our framework is shown in Figure~\ref{fig:main}.
Given the image observation $o_t$ or historical observations $\{o_\tau\}_{\tau< t}$ , the latent state could be extracted by a VAE (Variational Autoencoder) encoder, and our model is trained on a sequence of latent features of videos, also, the predicted latent features can be decoded into an image by a VAE decoder. Note we only consider $o_t$ in our method, and it could be extended to $\{o_\tau\}_{\tau< t}$ directly.
\begin{gather}
\begin{aligned}
&\text{VAE encoder:} &&s_t \sim p_{\Phi}(s_t|o_t)
\\
& \text{VAE decoder:} &&\hat{o}_t \sim p_{\Phi}(\hat{o}_t|s_t)
\label{eq:vae}
\end{aligned}
\end{gather}
The future observations $o_{t+1:t+\Delta T}$ is closely related with the given future actions $a_{t+1:t+\Delta T}$. Considering $\eta<\Delta T$, a light-weight MLP as a multi-modal action sequence model as follows:
\begin{equation}
\text{Predicted action: } \tilde a_{t+\eta+1:t+\Delta T} \sim q_{\theta}(o_{t},a_{t+1:t+\eta}).
\end{equation}
The action model not only predicts the future actions, but also extracts the latent action feature (See Sec.~\ref{sec:action}), which can be formulated as follows:
\begin{equation}
\text{Latent action feature: } \hat{h} \sim q_{\theta}(o_{t}, a_{t+1:t+\eta})
\end{equation}
The predicted $\tilde a_{t+\eta+1:t+\Delta T}$ provide the future actions for ego vehicle, while $\hat{h}$ could act as the condition to predict latent state $s_{t+1:t+\Delta T}$ because $\hat{h}$ contains the information of $\{a_{t+1:t+\eta}, \tilde a_{t+\eta+1:t+\Delta T}\}\}$:
\begin{equation}
\text{Transition model: } q_{\phi}(\tilde{s}_{t+1:t+\Delta T} |s_t, \hat{h}).
\end{equation}
Details are explained in Sec.~\ref{sec:video-trans}. After the transition model output a sequence of latent features $\tilde{s}_{t+1:t+\Delta T}$, these features are decoded into frames $\tilde{o}_{t+1:t+\Delta T}$ by a VAE decoder:
\begin{equation}
\text{Predicted video: } \tilde{o}_{t+1:t+\Delta T} \sim q_{\phi}(s_t).
\label{eq:predict}
\end{equation}
Besides, by iteratively rolling out the model prediction, we can simply obtain long-term future actions and videos.
% At each time step, the model receives observation $o_t$ or historical observations $\{o_\tau\}_{\tau< t}$, and given actions $a_{t:t+k}(k>0)$.
% The know actions $a_{t:tk}$ aims to prompt the world model to understand the dynamics and predict the future with a length $\Delta t$ in a desired direction, because only one frame has very few context to determine the driving intention.
% The design of latent action feature $\Tilde{h}$ aims to model the temporal dynamics, and prompt the transition model to make more correct predictions. Because the observation is image or video,
% \begin{equation}
% \text{Transition model: } && q_{\phi}(s_{t:t+\Delta t} |s_t, a_{t+1:t+\eta}, \hat{h})
% \end{equation}
% \begin{gather}
% \begin{aligned}
% &\text{Predicted actions:} &&a_{t+\eta+1, tk+\Delta t} \sim q_{\theta}(o_t, a\leq tk)
% \\
% & \text{Predicted video:} &&\hat{o}_{t\eta:\ t\eta+\Delta t} \sim q_{\phi}(s_t, a\leq tk)
% \label{eq:predict}
% \end{aligned}
% \end{gather}
\subsection{Light-weight Action Model}
\label{sec:action}
Instead of only predicting one action one time, which is low-efficiency, the action model here aims to predict multiple future actions with one step. Forecasting more future actions enables the application to become more flexible, and can specify the trade-off between speed and accuracy.
The design motivation is that if the model can predict future actions well, then the high-dimensional features obtained during the prediction process have smaller losses than encoding the predicted actions again, which is also inefficient using re-encoding. Learning latent actions is
also validated to be effective for predicting the future~\cite{bruce2024genie}
We carefully designed the action sequence model, the details are shown in Figure~\ref{fig:action}. Given actions $a\leq t+\eta $, the action model estimates the current action latent features and also the future action latent features by only feeding the known actions into the action model once:
\begin{equation}
h_{\leq t+\eta}, \Tilde{h}_{t+\eta+1:t+\Delta T} \sim q_{\theta}(a_{t+\eta+1:t+\Delta T}|o_t, a_{t+1:t+\eta})
\end{equation}
The action model receives multi-modal input including current states (observations) $o_t$ and known actions $a_{t+1:t+\eta}$. We use an embedding layer to lift the low-level actions to high-dimensional features and use an MLP to extract the temporal information, which can be regarded as the current latent action $h$. Then, a transform layer projects the visual features into action feature space, and the multi-modal features pass through an MLP to obtain the future latent action $\Tilde{h}$. We simply concatenate $h$ and $\Tilde{h}$ with the time dimension, resulting in final latent action $\hat{h}=[h, \Tilde{h}]$, which can prompt the following transition model to predict the consistency future frames. At the same time, the future latent action $\Tilde{h}_{t+\eta+1:t+\Delta T}$ are transformed into the future low-level actions $\Tilde{a}_{t+\eta+1:t+\Delta T}$. The action model is optimized by an L1 regularizer loss as follows:
\begin{equation}
\min_{\theta} \mathcal{L}_a = \beta_a \| a_{t+\eta+1:t+\Delta T} - \Tilde{a}_{t+\eta+1:t+\Delta T} \|
\end{equation}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{sec/action.pdf}
\caption{Light-weight action model. This model learns the latent action features given an action sequence and observations, }
\label{fig:action}
\end{figure}
\subsection{Video Transition Model}
\label{sec:video-trans}
% motivation
The world model learns rules from historical observations and action sequences to predict the future. We divide this process into two steps. The first step is written in Section~\ref{sec:action}, which is to learn the rules of action transition and obtain high-dimensional latent actions. For predicting the future, we can use state-of-the-art generative models, such as diffusion models~\cite{ho2020ddpm,rombach2022sdm}. We only need to design this model to receive latent actions and current observations. Therefore, in this section, we will introduce how to design and train a video transition model.
Given current observation $o_t$ and the latent action $\hat{h}$, we need to design a transition model that can predict future states. Intuitively, we can leverage diffusion models to achieve this purpose, as diffusion models are widely used in video generation~\cite{yuan2024magictime,wang2024magicvideo,yin2023nuwaxl} and prediction~\cite{yang2024cogvideox,ren2024consisti2v,blattmann2023svd}. However, most pre-trained diffusion models are conditioned on text, such as the text describing the scene, or depicting the dynamics and motion, where our target dataset Nuscenes~\cite{caesar2020nuscenes} is lacking. To this end, we utilized a state-of-the-art multi-modal language model (LLaVA-NeXT-Video~\cite{zhang2024llavanextvideo}) to annotate scenes in Nuscenes.
Based on annotations, we first fine-tuned a base model that adapts the general model~\cite{rombach2022sdm} to the driving scenario. Then, we design a spatiotemporal model following previous work~\cite{blattmann2023align,blattmann2023svd}. As shown in Figure~\ref{fig:main}, the model is built on the U-Net architecture. In order to satisfy the training of the U-Net and diffusion process, that is, the input and output feature dimensions are the same, and to learn from noise. We add noise to the state $s_t=p_\Phi(o_t)$ and ground-truth future states $s_{t+1:t+\Delta T}$, if we make a $\Delta T$-step prediction. The noisy state sequence is:
\begin{equation}
\Tilde{s}_{t:t+\Delta T} = \sqrt{\overline{\alpha}_\tau}s_{t:t+\Delta T}+ (1-\overline{\alpha}_\tau)\epsilon, \ \epsilon \sim \mathcal{N}(\textbf{0}, \textbf{I})
\label{eq:noise_state}
\end{equation}
where $\alpha_\tau$ and $\overline{\alpha}_\tau$ are predefined hyper-parameters, $\tau$ denotes the noise timestep.
\paragraph{State context short pathway}To satisfy observations of varying lengths, because in general, more observations make more precise predictions, we designed a flexible fusion module to make better use of the observations $o_{t: (t+\Delta \tau)}$, where $\Delta \tau\ \geq 0$ is the length of additional observations. First, $o_{t: (t+\Delta \tau)}$ is fed into VAE to obtain latent states $s_{t: (t+\Delta \tau)\sim P_\Phi(o_{t: (t+\Delta \tau)})}$, then we repeat the last state to obtain the state context:
\begin{equation}
s_c = [\underbrace{s_{t},s_{t+\Delta \tau}, \cdots, s_{t+\Delta \tau}}_{\Delta T \ \text{terms}}]
\end{equation}
Since subsequent observations are transitioned from the last observation $s_t$, the key of the fusion module is to provide a short pathway for the frame transition. The fusion module is designed with a multi-scale architecture that satisfies the multi-scale hidden states of the U-Net.
Specifically, as for a short pathway at $d$-th block in the U-Net, the state context at $d$-th block $s_c^d$ is transferred to scale $w^d_c$ and shift $b^d_c$ via zero-initialized convolution layers. The scale $w^d_c$ and shift $b^d_c$ are fused into the hidden state $h^d$ of $d$-th block as follows:
\begin{equation}
h^d= w^d_c \cdot h^d + b^d_c + h^d
\end{equation}
To support the classifier-free guidance sampling, we randomly drop the state context by a certain ratio in the training process.
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{sec/result1.pdf}
\caption{World model prediction by our proposed method. (Resolution of video is 256$\times$448. The orange \textcolor{orange}{\textcircled{ }} indicates the key frame with Ground-truth actions. The blue \textcolor{blue}{\textcircled{ }} indicates the key frame with the predicted actions, and the rest are non-key frames.) }
\label{fig:re1}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{sec/result2.pdf}
\caption{Video prediction comparison. Although different models have different resolutions for inference, CogVideoX-2b-nus(CogVX-nus) uses 480$\times$720, Vista uses 576$\times$1024, our model uses 256$\times$448, our model shows better quality and clarity, avoiding misaligned starting frame and corrupted results. }
\label{fig:re2}
\end{figure*}
\paragraph{Future prediction}The video transition model learns to predict future frames. The action model in Sec.~\ref{sec:action} produces latent action $\hat{h}$ from the current observations $o_t$ and the current actions $a\leq t+\eta$, which provides the estimated dynamics and benefits the future frame prediction. The future prediction objective is:
\begin{equation}
\min_{\phi} \mathcal{L}_v = \beta_v \mathbf{E}_{\tau,s,\epsilon} \| \epsilon - q_{\phi}(\Tilde{s}_{t:t+\Delta T}, s_c, \hat{h}, \tau) \|^2
\end{equation}
When training, the noise is added to known states $s_t$ and ground-truth states $s_{t:t+\Delta T}$, and this diffusion objective learns to predict the future states from the noise. Using the cross-attention mechanism, latent action $\hat{h}$ acts on the time dimension of the state sequence.
The world model is trained to optimize the overall loss $\mathcal{L}_a+\mathcal{L}_v$ with respect to all parameters.
\section{Experiment}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{sec/long-horizon.pdf}
\caption{Long-term video and action rollout. Given four input frames, our model can predict long-term video and actions, up to 100 frames. The video quality is still well-maintained and the predicted actions are accurate. The upper sample shows that our model generates a future in which the ego car is slowing down and then slowly stopping. The bottom sample shows a future in which the ego car stays driving, then deviates slightly to the left (consistent with predicted steering angles), and then returns to a straight line.}
\label{fig:long}
\end{figure*}
\begin{table*}[]
\centering
\caption{Quantitative comparison on driving video generation}
\label{tab:fvd}
\begin{tabular}{lccc}
\toprule
Method & Condition & FID↓ & FVD↓ \\
\hline
DriveGAN~\cite{kim2021drivegan} & 1st image & 73.4 & 502.3 \\
DriveDreamer~\cite{wang2023drivedreamer} & 1st image+HDmap+3Dbox & 52.6 & 452.0 \\
DrivingDiffusion~\cite{li2023drivingdiffusion} & HDmap+3Dbox & 15.9 & 335.0 \\
Panacea~\cite{wen2023panacea} & 1st image+HDmap+3Dbox & 20.4 & 247.0 \\
GenAD-nus~\cite{yang2024genad} & 1st image & \underline{15.4} & 244.0 \\
Drive-WM~\cite{wang2023driveWM} & 1st image+HDmap+3Dbox & 23.3 & \underline{228.5} \\
\hline
Ours & 1st image & \textbf{6.9} & \textbf{190.5} \\
\bottomrule
\end{tabular}
\vspace{-0.5cm}
\end{table*}
\begin{table*}[t]
% \vspace{-1cm}
\caption{Quantitative comparison on driving action prediction. $^\dagger$ denotes action-only.}
% \renewcommand\tabcolsep{3pt}
% \small
\label{tab:action}
\centering
\begin{tabular}{lccc}
\toprule
Method & L1↓ Speed & L1↓ Steering angle & Average L1↓ \\
\midrule
DriverDreamer$^\dagger$ \cite{wang2023drivedreamer}& 0.150 & - & - \\
DriverDreamer\cite{wang2023drivedreamer} & 0.140 & - & - \\
ADriver-I$^\dagger$ ~\cite{jia2023adriver} & 0.122 & 0.101 & 0.1115 \\
ADriver-I ~\cite{jia2023adriver} & 0.103 & \textbf{0.092} & 0.0975 \\
\midrule
Ours & \textbf{0.090} & 0.093 & \textbf{0.0915} \\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Setting}
All experiments are conducted on the Nuscenes dataset~\cite{caesar2020nuscenes}. Following the setting in~\cite{wang2023driveWM}, to make a fair comparison, we use the 192$\times$384 resolution for video quality evaluation. We use FID (Fréchet Inception Distance)~\cite{heusel2017fid} and FVD (Fréchet Video Distance)~\cite{unterthiner2018fvd} to evaluate images and video quality, respectively. Besides, for video visualization and long-horizon rollouts, we train a 256$\times$448 video model. For action prediction, we follow~\cite{jia2023adriver} to report the L1 error of driving speed and steering angle. More details can be found in the Appendix.
\subsection{Qualitative Results}
\paragraph{Joint video-action prediction} Since we using joint training combining action prediction and future prediction, our trained model is capable of joint prediction, where predicted videos and actions are well aligned. We show two examples in Figure~\ref{fig:re1}. For each scene, we input four reference frames and two actions to prompt the dynamic information and then roll out future frames and actions iteratively. We present the first two non-key frames to indicate that our model can maintain the scenes well, and subsequent frames marked with blue circles are keyframes. The jointly generated actions are shown in the last column. For the upper sampler, the predicted future is a slow acceleration of the ego car. The bottom sampler presents a future with a faster acceleration. The predicted frames and actions are well aligned, validating the joint-prediction ability inside our world model.
%We show that ProphetDWM can jointly predict future frames and corresponding actions. For each scene, we input a reference frame and 4 reference actions into ProphetDWM. In Fig.~\ref{fig:re1}, the top depicts a scene where the ego car is turning left. The future speeds and steering angles, transitioning from a left turn to a straight path, are well predicted. The bottom shows a scene where the ego car is driving straight. Future steering angles are accurately predicted. Future speed is gradually increased and maintained. Although it differs from the ground truth, these speeds are reasonable in real-world driving scenarios. ProphetDWM can also roll out long-term videos and actions. More results and roll-out prediction refer to Appendix.
\paragraph{Video quality comparison}
As there are limited open-sourced pre-trained models, DriverDreamer~\cite{wang2023drivedreamer}, ADrive-I~\cite{jia2023adriver}, and Drive-WM~\cite{wang2023driveWM} did not release the pre-trained models. We compare our method with Vista~\cite{gao2024vista}, the latest video prediction model in the driving scenario. We also fine-tune an open-source state-of-the-art model CogVideoX-2b~\cite{yang2024cogvideox} on Nuscenes as a video prediction model to be compared, denoted as CogVX-2b-nus. Different methods use different resolutions in the inference process. CogVX-2b-nus uses 480$\times$720, Vista uses 576$\times$1024, and our model uses 256$\times$448 resolution. An example is shown in Figure~\ref{fig:re2}, we input the first frame ($T=0$) to three models, and all the models will output the reconstructed first frame, but CogVX-2b-nus outputs a misaligned frame, which also leads to more corrupted results in the subsequent frames. Vista and our model can reconstruct the first frame well. Although Vista has a high resolution of 576$\times$1024, its video content is not very clear, even a little blurred, which led to some subsequent corrupted results. In contrast, our model does not experience the accumulation of errors, and subsequent frames do not appear to be corrupted. The comparison validates the advantages of our method.
\paragraph{Long-term rollout}The proposed world model learns from the history sequence states and predicts the future sequence states, which enables it to plan and forecast the future. One of the strengths of our world model is the ability to make long-term future predictions. Two long-term future predictions are shown in Figure~\ref{fig:long}. Our model can predit diverse futures (such as slowdowns and deviations to the left), and the long-term videos' quality is well-maintained.
\begin{figure*}
\centering
\includegraphics[width=0.7\linewidth]{sec/ab1.pdf}
\caption{Comparison between the proposed multi-scale fusion and widely-used concatenation~\cite{yang2024cogvideox,blattmann2023svd}. Our method can converge faster and improve the quality of prediction.}
\label{fig:ab1}
\end{figure*}
\subsection{Quantitative Evaluation}
\paragraph{Driving video generation}We compare with the state-of-the-art methods that can generate single-view driving videos. The results of ADriver-I~\cite{jia2023adriver} are not included since it generates 4 frames based on the previous 4 frames, which is unfair for the listed methods. GenAD~\cite{yang2024genad} and Vista~\cite{gao2024vista} results are also not included here, because they trained their models on extra large-scale video datasets. As shown in Table~\ref{tab:fvd}, even without the fine-grained conditions such as HDmap and 3Dbox as input priors, our method achieves the best results compared to other methods, with an FID of 6.9 and an FVD of 190.5, showing significant improvements.
\paragraph{Driving action prediction}We compare the driving action prediction with the previous work that has action prediction ability, including DriveDreamer~\cite{wang2023drivedreamer} and ADriver-I~\cite{jia2023adriver}. As shown in Tab.~\ref{tab:action}, our method achieves the lowest L1 error in speed, outperforming both DriveDreamer and ADriver-I. For the steering angle, our method also surpasses ADriver-I when using only actions, and comparable with ADriver-I using multi-modal inputs. Because all the methods finally use an MLP to project high-dimensional features into two low-level signals, we can directly report the average L1 error, and our method achieves the best average L1 error.
\subsection{Ablation Study}
\paragraph{Multi-scale fusion vs. Concatenation}We design a multi-scale fusion module to construct a state context short pathway. We compare a widely-used method (concatenation), which concatenates the context with the noisy latent features on the input layer~\cite{yang2024cogvideox,blattmann2023svd}. We modify the channels of $\texttt{conv\_in}$ to 8, to adapt the concatenated features. When training at the same steps, the model equipped with multi-scale fusion predicts better video than with concatenation in a large margin. The visualization results are in Figure~\ref{fig:ab1}.
\paragraph{Benefits of joint training}We ablate the different training strategies for video prediction, including no latent actions, frozen action module, and joint training. As shown in Table~\ref{tab:abaction}, joint training with the latent actions shows the best video quality, validating the effectiveness of our model.
\paragraph{Different reference frame}
Our multi-scale fusion module can tackle different reference frames, so we investigate the impact of the number of reference frames. As shown in Table~\ref{tab:ab2}, more frames can prompt the video consistency, i.e., lower FVD, and the impact on the image quality of each frame is comparable.
\begin{table}[]
% \vspace{-1cm}
\caption{Video quality comparison under different training}
\label{tab:abaction}
\centering
\begin{tabular}{cccc}
\toprule
Latent action & Joint train. & FID↓ & FVD↓ \\
\midrule
& & 7.2 & 265 \\
\checkmark & & 7.1 & 227 \\
\checkmark & \checkmark & 6.9 & 191 \\
\bottomrule
\end{tabular}
% \vspace{-0.5cm}
\end{table}
\begin{table}[]
% \vspace{-1cm}
% \vspace{-0.5cm}
\caption{Video quality comparison using different reference frames in training}
\label{tab:ab2}
\centering
\begin{tabular}{lcc}
\toprule
Frame Num. & FID↓ & FVD↓ \\
\midrule
1 & 6.9 & 190.5 \\
2 & 6.9 & 188.3 \\
4 & 7.1 & 182.5 \\
\bottomrule
\end{tabular}
\vspace{-0.3cm}
\end{table}
\paragraph{Different resolution} We investigate the impact of training resolution on some metrics such as FID and FVD. As shown in Table~\ref{tab:ab3}, higher resolution can promote the FID, because the clarity of each frame is better. But higher resolution leads to worse FVD. One reason may be that dynamic information and motion at high resolution are difficult to learn, so the temporal consistency becomes worse.
\begin{table}[]
% \vspace{-1cm}
\caption{Video quality comparison using different resolution.}
\label{tab:ab3}
\centering
\begin{tabular}{lcc}
\toprule
Train. resolution & FID↓ & FVD↓ \\
\midrule
192$\times$384 & 6.9 & 190.5 \\
256$\times$448 & 6.2 & 238.0 \\
\bottomrule
\end{tabular}
\end{table}
\section{Conclusion}
we proposed ProphetDWM, a novel end-to-end driving world model that can jointly predict future videos and actions. We construct a world model that is trained by learning latent actions given finite states and predicting actions and videos, where two crucial modules, the action module and diffusion-model-based transition model, are jointly optimized. Experiments in video generation and action prediction validate the effectiveness of our world model.
%We propose an action module to learn latent actions from the present moment to the future and a diffusion-model-based transition model to the state distribution, and these two modules are jointly optimized. Experiments in video generation and action prediction validate the effectiveness of our world model.
%In this work, we proposed ProphetDWM, a novel end-to-end driving world model that can synchronously predict the corresponding future frames and actions. Due to the action module's n-gram prediction, our method can predict long-term action representations, which guide the video generation module to predict long-term future frames and generate the corresponding actions. ProphetDWM achieves the best single-view video quality and comparable action prediction accuracy on NuScenes.
\clearpage
\setcounter{page}{1}
\maketitlesupplementary
\appendix
\renewcommand{\thetable}{A\arabic{table}}
\renewcommand{\thefigure}{A\arabic{figure}}
\section{Experiment details}
\label{sec:detail}
%
\begin{table}[]
\centering
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{lccc}
\hline
\textbf{Setting} & \textbf{1} & \textbf{2} & \textbf{3} \\ \hline
\textbf{Resolution} & 192$\times$384 & 256$\times$448 & 256$\times$448 \\ \hline
\textbf{Refer. Frames} & 1 & 1 & 4 \\ \hline
\textbf{Sequence Length} & 8 & 10 & 10 \\ \hline
\textbf{Fps} & 2 & 4 & 4 \\ \hline
\textbf{LR} & 1e-4 & 1e-4 & 1e-4 \\ \hline
\end{tabular}
\caption{Driving World Model experimental settings}
\label{tab:detail}
\end{table}
In this paper, we conduct various experiment results. All experiments are conducted on A100 GPUs. The settings are shown in Table~\ref{tab:detail}.
Following the previous setting~\cite{wang2023driveWM}, we conduct the first setting-1 for a fair quantitative comparison. The resolution is 192$\times$ 384, the sequence length is 8, the reference frame number is 1, and for simplicity, we do not utilize additional HDmap or 3Dbox as ~\cite{wang2023driveWM} did. The training dataset only includes keyframes, i.e., fps=2. In this setting, we report the FID and FVD results.
For the qualitative comparison, we train two models based on resolution 256$\times$448 and use one reference frame and four reference frames, respectively. The sequence length is 10. The training sequence is visualized in Figure~\ref{fig:A3}. We use 4Hz data, and there are two non-keyframes between two keyframes, and fps=4. The orange marks are the inputs to our world model. The known future actions indicate the dynamic information in the future. Our model jointly predicts nine subsequent frames and two actions. The reason is that in Nuscenes~\cite{caesar2020nuscenes}, the actions are only recorded for keyframes.
When training the image model, we finetune Stable-diffusion v1.4 on Nuscenes~\cite{caesar2020nuscenes} images for 50 epochs. When training the video model, we use a total batch size of 16 and train the video model for 30k steps in total. The learning rate is 1e-4, and the scheduler is a cosine scheduler. The video model's training took 40 hours on 4 A100 GPUs.
When inferring future videos and actions, we start from the reference frame(s) and reference actions, and the reference frame(s) and reference actions will be updated when a rollout is done. We roll out the future 16 times to produce a long-term future, with more than 100 frames.
\section{Quality comparison}
We found that using more references would increase the video quality. As shown in Figure~\ref{fig:A1}, using more reference frames enhances content consistency and avoids excessive content changes. However, using only one frame produces more frames when using the same rollout times. For example, if we roll out the future 16 times, using four frames can produce 100 frames, while using one frame can produce 145 frames. As shown in Figure~\ref{fig:A1}, although these two predictions start from the same scene, the results of different predictions are still diverse.
\section{Action Control}
In Figure~\ref{fig:A4}, we show the prediction results from different low-level control actions, such as going straight and turning left. We use low-level actions including speeds and steering angles to execute the commands. The results show that different commands can predict different futures while maintaining a high quality of the predicted results.
\section{More results}
We present more results using one frame in Figure~\ref{fig:A2}. For each sample, we roll out 16 times to produce 145 frames.
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{sec/A3.pdf}
\caption{Training sequence visualization. The \textcolor{orange}{orange} marks are the inputs to our world model. The known future actions indicate the dynamic information in the future. Our model jointly predicts nine subsequent frames and two actions. }
\label{fig:A3}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{sec/A1.pdf}
\caption{The comparison between using four reference frames and using one reference frame. Using the same 16 rollout times. Using four frames produces a total of 100 frames, and using one frame produces a total of 145 frames. Starting from the same scene, the results of different generations are still diverse.}
\label{fig:A1}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{sec/A4.pdf}
\caption{The prediction results from different low-level control actions.}
\label{fig:A4}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{sec/A2.pdf}
\caption{More results using one reference frame.}
\label{fig:A2}
\end{figure*}
\begin{thebibliography}{43}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bar-Tal et~al.(2024)Bar-Tal, Chefer, Tov, Herrmann, Paiss, Zada, Ephrat, Hur, Li, Michaeli, et~al.]{bar2024lumiere}
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et~al.
\newblock Lumiere: A space-time diffusion model for video generation.
\newblock \emph{arXiv preprint arXiv:2401.12945}, 2024.
\bibitem[Blattmann et~al.(2023{\natexlab{a}})Blattmann, Dockhorn, Kulal, Mendelevitch, Kilian, Lorenz, Levi, English, Voleti, Letts, et~al.]{blattmann2023svd}
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et~al.
\newblock Stable video diffusion: Scaling latent video diffusion models to large datasets.
\newblock \emph{arXiv preprint arXiv:2311.15127}, 2023{\natexlab{a}}.
\bibitem[Blattmann et~al.(2023{\natexlab{b}})Blattmann, Rombach, Ling, Dockhorn, Kim, Fidler, and Kreis]{blattmann2023align}
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung~Wook Kim, Sanja Fidler, and Karsten Kreis.
\newblock Align your latents: High-resolution video synthesis with latent diffusion models.
\newblock In \emph{CVPR}, pages 22563--22575, 2023{\natexlab{b}}.
\bibitem[Bruce et~al.(2024)Bruce, Dennis, Edwards, Parker-Holder, Shi, Hughes, Lai, Mavalankar, Steigerwald, Apps, et~al.]{bruce2024genie}
Jake Bruce, Michael~D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et~al.
\newblock Genie: Generative interactive environments.
\newblock In \emph{Forty-first International Conference on Machine Learning}, 2024.
\bibitem[Caesar et~al.(2020)Caesar, Bankiti, Lang, Vora, Liong, Xu, Krishnan, Pan, Baldan, and Beijbom]{caesar2020nuscenes}
Holger Caesar, Varun Bankiti, Alex~H Lang, Sourabh Vora, Venice~Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.
\newblock nuscenes: A multimodal dataset for autonomous driving.
\newblock In \emph{CVPR}, pages 11621--11631, 2020.
\bibitem[Dosovitskiy et~al.(2017)Dosovitskiy, Ros, Codevilla, Lopez, and Koltun]{dosovitskiy2017carla}
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.
\newblock Carla: An open urban driving simulator.
\newblock In \emph{Conference on robot learning}, pages 1--16. PMLR, 2017.
\bibitem[Ebert et~al.(2018)Ebert, Finn, Dasari, Xie, Lee, and Levine]{ebert2018visual}
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine.
\newblock Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.
\newblock \emph{arXiv preprint arXiv:1812.00568}, 2018.
\bibitem[Gao et~al.(2024)Gao, Yang, Chen, Chitta, Qiu, Geiger, Zhang, and Li]{gao2024vista}
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li.
\newblock Vista: A generalizable driving world model with high fidelity and versatile controllability.
\newblock \emph{arXiv preprint arXiv:2405.17398}, 2024.
\bibitem[Ha and Schmidhuber(2018{\natexlab{a}})]{ha2018recurrent}
David Ha and J{\"u}rgen Schmidhuber.
\newblock Recurrent world models facilitate policy evolution.
\newblock \emph{NIPS}, 31, 2018{\natexlab{a}}.
\bibitem[Ha and Schmidhuber(2018{\natexlab{b}})]{ha2018world}
David Ha and J{\"u}rgen Schmidhuber.
\newblock World models.
\newblock \emph{arXiv preprint arXiv:1803.10122}, 2018{\natexlab{b}}.
\bibitem[Hafner et~al.(2019)Hafner, Lillicrap, Ba, and Norouzi]{hafner2019dreamer}
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.
\newblock Dream to control: Learning behaviors by latent imagination.
\newblock \emph{arXiv preprint arXiv:1912.01603}, 2019.
\bibitem[Hafner et~al.(2020)Hafner, Lillicrap, Norouzi, and Ba]{hafner2020dreamerv2}
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba.
\newblock Mastering atari with discrete world models.
\newblock \emph{arXiv preprint arXiv:2010.02193}, 2020.
\bibitem[Hafner et~al.(2023)Hafner, Pasukonis, Ba, and Lillicrap]{hafner2023dreamerv3}
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.
\newblock Mastering diverse domains through world models.
\newblock \emph{arXiv preprint arXiv:2301.04104}, 2023.
\bibitem[Heusel et~al.(2017)Heusel, Ramsauer, Unterthiner, Nessler, and Hochreiter]{heusel2017fid}
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
\newblock Gans trained by a two time-scale update rule converge to a local nash equilibrium.
\newblock \emph{NIPS}, 30, 2017.
\bibitem[Ho et~al.(2020)Ho, Jain, and Abbeel]{ho2020ddpm}
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock \emph{NIPS}, 33:\penalty0 6840--6851, 2020.
\bibitem[Hu et~al.(2022)Hu, Corrado, Griffiths, Murez, Gurau, Yeo, Kendall, Cipolla, and Shotton]{hu2022mile}
Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton.
\newblock Model-based imitation learning for urban driving.
\newblock \emph{NIPS}, 35:\penalty0 20703--20716, 2022.
\bibitem[Hu et~al.(2023)Hu, Russell, Yeo, Murez, Fedoseev, Kendall, Shotton, and Corrado]{hu2023gaia}
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado.
\newblock Gaia-1: A generative world model for autonomous driving.
\newblock \emph{arXiv preprint arXiv:2309.17080}, 2023.
\bibitem[Jia et~al.(2023)Jia, Mao, Liu, Zhao, Wen, Zhang, Zhang, and Wang]{jia2023adriver}
Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang.
\newblock Adriver-i: A general world model for autonomous driving.
\newblock \emph{arXiv preprint arXiv:2311.13549}, 2023.
\bibitem[Kim et~al.(2021)Kim, Philion, Torralba, and Fidler]{kim2021drivegan}
Seung~Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler.
\newblock Drivegan: Towards a controllable high-quality neural simulation.
\newblock In \emph{CVPR}, pages 5820--5829, 2021.
\bibitem[Kondratyuk et~al.(2023)Kondratyuk, Yu, Gu, Lezama, Huang, Hornung, Adam, Akbari, Alon, Birodkar, et~al.]{kondratyuk2023videopoet}
Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos{\'e} Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et~al.
\newblock Videopoet: A large language model for zero-shot video generation.
\newblock \emph{arXiv preprint arXiv:2312.14125}, 2023.
\bibitem[LeCun(2022)]{lecun2022worldmodel}
Yann LeCun.
\newblock A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.
\newblock \emph{Open Review}, 62\penalty0 (1), 2022.
\bibitem[Li et~al.(2023)Li, Zhang, and Ye]{li2023drivingdiffusion}
Xiaofan Li, Yifu Zhang, and Xiaoqing Ye.
\newblock Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model.
\newblock \emph{arXiv preprint arXiv:2310.07771}, 2023.
\bibitem[Lin et~al.(2023{\natexlab{a}})Lin, Du, Watkins, Hafner, Abbeel, Klein, and Dragan]{lin2023dynalang}
Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan.
\newblock Learning to model the world with language.
\newblock 2023{\natexlab{a}}.
\bibitem[Lin et~al.(2023{\natexlab{b}})Lin, Du, Watkins, Hafner, Abbeel, Klein, and Dragan]{lin2023learninglanguage}
Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan.
\newblock Learning to model the world with language.
\newblock 2023{\natexlab{b}}.
\bibitem[OpenAI(2024)]{openai2023sora}
OpenAI.
\newblock Sora technical report.
\newblock 2024.
\bibitem[Ren et~al.(2024)Ren, Yang, Zhang, Wei, Du, Huang, and Chen]{ren2024consisti2v}
Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen.
\newblock Consisti2v: Enhancing visual consistency for image-to-video generation.
\newblock \emph{arXiv preprint arXiv:2402.04324}, 2024.
\bibitem[Rombach et~al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer]{rombach2022sdm}
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj{\"o}rn Ommer.
\newblock High-resolution image synthesis with latent diffusion models.
\newblock In \emph{CVPR}, pages 10684--10695, 2022.
\bibitem[Smallwood and Sondik(1973)]{smallwood1973pomdp}
Richard~D Smallwood and Edward~J Sondik.
\newblock The optimal control of partially observable markov processes over a finite horizon.
\newblock \emph{Operations research}, 21\penalty0 (5):\penalty0 1071--1088, 1973.
\bibitem[Sun et~al.(2020)Sun, Kretzschmar, Dotiwalla, Chouard, Patnaik, Tsui, Guo, Zhou, Chai, Caine, et~al.]{waymo}
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et~al.
\newblock Scalability in perception for autonomous driving: Waymo open dataset.
\newblock In \emph{CVPR}, pages 2446--2454, 2020.
\bibitem[Tassa et~al.(2018)Tassa, Doron, Muldal, Erez, Li, Casas, Budden, Abdolmaleki, Merel, Lefrancq, et~al.]{tassa2018dmc}
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de~Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et~al.
\newblock Deepmind control suite.
\newblock \emph{arXiv preprint arXiv:1801.00690}, 2018.
\bibitem[Unterthiner et~al.(2018)Unterthiner, Van~Steenkiste, Kurach, Marinier, Michalski, and Gelly]{unterthiner2018fvd}
Thomas Unterthiner, Sjoerd Van~Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.
\newblock Towards accurate generative models of video: A new metric \& challenges.
\newblock \emph{arXiv preprint arXiv:1812.01717}, 2018.
\bibitem[Wang et~al.(2024{\natexlab{a}})Wang, Liu, Lin, Yan, Chen, Low, Hoang, Wu, Liew, Yan, et~al.]{wang2024magicvideo}
Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun~Hao Liew, Hanshu Yan, et~al.
\newblock Magicvideo-v2: Multi-stage high-aesthetic video generation.
\newblock \emph{arXiv preprint arXiv:2401.04468}, 2024{\natexlab{a}}.
\bibitem[Wang et~al.(2023{\natexlab{a}})Wang, Zhu, Huang, Chen, and Lu]{wang2023drivedreamer}
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu.
\newblock Drivedreamer: Towards real-world-driven world models for autonomous driving.
\newblock \emph{arXiv preprint arXiv:2309.09777}, 2023{\natexlab{a}}.
\bibitem[Wang et~al.(2024{\natexlab{b}})Wang, Zhu, Huang, Wang, Chen, and Lu]{wang2024worlddreamer}
Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, and Jiwen Lu.
\newblock Worlddreamer: Towards general world models for video generation via predicting masked tokens.
\newblock \emph{arXiv preprint arXiv:2401.09985}, 2024{\natexlab{b}}.
\bibitem[Wang et~al.(2023{\natexlab{b}})Wang, He, Fan, Li, Chen, and Zhang]{wang2023driveWM}
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang.
\newblock Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving.
\newblock \emph{arXiv preprint arXiv:2311.17918}, 2023{\natexlab{b}}.
\bibitem[Wen et~al.(2023)Wen, Zhao, Liu, Jia, Wang, Luo, Zhang, Wang, Sun, and Zhang]{wen2023panacea}
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang.
\newblock Panacea: Panoramic and controllable video generation for autonomous driving.
\newblock \emph{arXiv preprint arXiv:2311.16813}, 2023.
\bibitem[Wu et~al.(2023)Wu, Escontrela, Hafner, Abbeel, and Goldberg]{wu2023daydreamer}
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg.
\newblock Daydreamer: World models for physical robot learning.
\newblock In \emph{Conference on Robot Learning}, pages 2226--2240. PMLR, 2023.
\bibitem[Yang et~al.(2024{\natexlab{a}})Yang, Gao, Qiu, Chen, Li, Dai, Chitta, Wu, Zeng, Luo, et~al.]{yang2024genad}
Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et~al.
\newblock Generalized predictive model for autonomous driving.
\newblock In \emph{CVPR}, pages 14662--14672, 2024{\natexlab{a}}.
\bibitem[Yang et~al.(2024{\natexlab{b}})Yang, Teng, Zheng, Ding, Huang, Xu, Yang, Hong, Zhang, Feng, et~al.]{yang2024cogvideox}
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et~al.
\newblock Cogvideox: Text-to-video diffusion models with an expert transformer.
\newblock \emph{arXiv preprint arXiv:2408.06072}, 2024{\natexlab{b}}.
\bibitem[Yin et~al.(2023)Yin, Wu, Yang, Wang, Wang, Ni, Yang, Li, Liu, Yang, et~al.]{yin2023nuwaxl}
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et~al.
\newblock Nuwa-xl: Diffusion over diffusion for extremely long video generation.
\newblock \emph{arXiv preprint arXiv:2303.12346}, 2023.
\bibitem[Yuan et~al.(2024)Yuan, Huang, Shi, Xu, Zhu, Lin, Cheng, Yuan, and Luo]{yuan2024magictime}
Shenghai Yuan, Jinfa Huang, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, and Jiebo Luo.
\newblock Magictime: Time-lapse video generation models as metamorphic simulators.
\newblock \emph{arXiv preprint arXiv:2404.05014}, 2024.
\bibitem[Zhang et~al.(2024)Zhang, Li, Liu, Lee, Gui, Fu, Feng, Liu, and Li]{zhang2024llavanextvideo}
Yuanhan Zhang, Bo Li, haotian Liu, Yong~jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li.
\newblock Llava-next: A strong zero-shot video understanding model, 2024.
\bibitem[Zhao et~al.(2024)Zhao, Wang, Zhu, Chen, Huang, Bao, and Wang]{zhao2024drivedreamer2}
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang.
\newblock Drivedreamer-2: Llm-enhanced world models for diverse driving video generation.
\newblock \emph{arXiv preprint arXiv:2403.06845}, 2024.
\end{thebibliography}