Training-free Camera Control for Video Generation
Extracted source text · arXiv 2406.10126
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{amsmath}
\usepackage{booktabs}
\usepackage[linesnumbered, boxed, ruled]{algorithm2e}
\newcommand{\ours}{{CamTrol}}
\title{Training-free Camera Control for Video Generation}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Chen Hou, Zhibo Chen \\
University of Science and Technology of China\\
\texttt{\{houchen@mail.,chenzhibo@\}ustc.edu.cn} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
We propose a training-free and robust solution to offer camera movement control for off-the-shelf video diffusion models. Unlike previous work, our method does not require any supervised finetuning on camera-annotated datasets or self-supervised training via data augmentation. Instead, it can be plug-and-play with most pretrained video diffusion models and generate camera-controllable videos with a single image or text prompt as input.
The inspiration for our work comes from the layout prior that intermediate latents encode for the generated results, thus rearranging noisy pixels in them will cause the output content to relocate as well.
As camera moving could also be seen as a type of pixel rearrangement caused by perspective change, videos can be reorganized following specific camera motion if their noisy latents change accordingly.
Building on this, we propose \textbf{CamTrol}, which enables robust camera control for video diffusion models. It is achieved by a two-stage process. First, we model image layout rearrangement through explicit camera movement in 3D point cloud space. Second, we generate videos with camera motion by leveraging the layout prior of noisy latents formed by a series of rearranged images.
Extensive experiments have demonstrated its superior performance in both video generation and camera motion alignment compared with other finetuned methods.
Furthermore, we show the capability of CamTrol to generalize to various base models, as well as its impressive applications in scalable motion control, dealing with complicated trajectories and unsupervised 3D video generation.
Videos available at \textcolor{cyan}{\href{https://lifedecoder.github.io/CamTrol/}{\textbf{https://lifedecoder.github.io/CamTrol/}}}.
\end{abstract}
\section{Introduction}
As a more appealing and content-richer modality, videos differ from images by including an extra temporal dimension. This temporal aspect provides increased versatility for depicting diverse and dynamic movements, which can be decomposed into object motion, background transitions, and perspective changes.
Recent years have witnessed the rapid development and splendid breakthroughs of video generation with text prompts or images as input instructions \citep{li2023videogen,hong2022cogvideo,ho2022imagenvideo,luo2023videofusion,zeng2023PixelDance,SVD,sora,ge2023preserve,fei2023DysenVDM},
demonstrating the immense potential of diffusion models to synthesize realistic videos.
While these video generation models have made progress in generating videos with highly dynamic objects and backgrounds \citep{zeng2023PixelDance,SVD,li2023videogen}, most of them fail to provide camera control for the generated videos.
The difficulty of controlling camera motion in videos arises primarily from two aspects. The initial challenge lies in the inadequacy of annotated data, as most video annotations lack detailed descriptions, particularly about the camera movements. As a result, video generation models trained on these data often fail to interpret text prompts related to camera motions and generate correct outputs. One solution to mitigate the data insufficiency problem is to mimic videos with camera movements through simple data augmentation \citep{yang2024direct}. However, these methods can only handle simple camera motions like \textit{zoom} or \textit{truck}, and have trouble dealing with more complicated ones.
The second challenge is the extra finetuning required for camera control and its inherent limitations. As camera trajectories could be sophisticated, they sometimes cannot be accurately elaborated using na\"ive text prompts alone.
Common solutions \citep{wang2023motionctrl, he2024cameractrl} proposed to embed camera parameters into diffusion models through learnable encoders and perform extensive finetuning on large-scale datasets with detailed camera trajectories. However, such datasets as RealEstate10k \citep{realestate10k} and MVImageNet \citep{yu2023mvimgnet} are highly limited in scale and diversity due to the difficulty associated with data collection; in this way, these finetuning methods demand substantial resources but exhibit limited generalizability to other types of scenes.
\textit{Lack of annotations and the constraints of finetuning make camera control a challenging task in video generations}.
In this work, we attempt to address these issues through a training-free solution to offer camera control for off-the-shelf video diffusion models. We begin by introducing two core observations underpinning the idea that video diffusion models can achieve camera movement control in a \textit{training-free} manner.
First, we find that base video models could produce results with rough camera moves by integrating specific camera-related text into input prompts, such as \textit{camera zooms in} or \textit{camera pans right}. This simple implementation, though not very accurate and always leading to static or wrong motions, shows the natural prior knowledge learned by pretrained models about following different camera trajectories.
The other observation is the effectiveness of video models in adapting to 3D generation tasks. Recent works \citep{sv3d, im3d, shi2023mvdream} find that leveraging pretrained video models as initialization helps drastically improve the performance of multi-view generations, demonstrating their strong ability to handle perspective change.
The two crucial observations reveal the hidden power of video models for camera motion control. Therefore, we seek to find a way to invoke this innate ability, as it already exists in the model itself.
We propose \textbf{\ours}, which offers camera control for off-the-shelf video diffusion models in a training-free but robust manner.
\ours~is inspired by the layout prior that noisy latents possess for the generation results: As pixels in noisy latents change their positions, corresponding rearrangement will also occur in the output, leading to layout modification. Considering camera moves can also be seen as a type of layout rearrangement, this prior can serve as an effective hint, providing the video model with information about specific camera motions. Specifically, \ours~consists of a two-stage procedure. In stage I, explicit camera movements are modeled in 3D point cloud representation and produce a series of rendered images indicating specific camera movements. In stage II, the layout prior of noisy latents is utilized to guide video generations with camera movements. Compared with previous works, \ours~requires no additional finetuning on camera-annotated datasets, nor does it need self-supervised training based on data augmentation. Extensive experiments have demonstrated its superior performance in both video generation quality and camera motion alignment against other finetuned methods. Furthermore, we show \ours's capability to generalize to various base models, as well as its impressive applications in scalable motion control, dealing with complicated trajectories and unsupervised 3D video generation.
% controlling camera motion of generated videos, while simultaneously maintaining their dynamic, diversity and visual quality.
% We conduct extensive experiments to validate the effectiveness of our proposed \ours. Both quantitative and qualitative results demonstrate the robustness of \ours~as a useful tool of controlling camera motion for video diffusion models. Furthermore, we show that \ours~ produces impressive results of dynamic 3D rotation videos in various styles.
\section{Related Work}
\paragraph{Camera Control for Video Generation}
While methods aiming to control video foundation models continue to emerge \citep{ma2024magic, liu2023video, feng2023ccedit}, there are few works exploring how to manipulate the camera motion of generated videos. Earlier works \citep{hao2018controllable} control motion trajectory via warping images through densified sparse flow and pixel fusion. Similar ideas also appear later in \citet{chen2023motion} and \citet{yin2023dragnuwa}. Apart from utilizing optical flow, two main techniques for implementing video camera control are self-supervised augmentation or additional finetuning.
\citet{yang2024direct} disentangles object motion with camera movement and incorporates extra layers to embed camera motions, where the model is trained in a self-supervised manner by augmenting input videos to simulate simple camera movements. \citet{he2024cameractrl} and \citet{wang2023motionctrl} train additional camera encoders and integrate the outputs into the temporal attention layers of U-Net. \citet{guo2023animatediff} learns new motion patterns via LoRA \citep{hu2021lora} and finetuning with multiple reference videos.
\paragraph{Noise Prior of Latents in Diffusion Model}
One of the most natural advantages of diffusion models comes from its pixel-wise noisy latents formed during the denoising process.
These latents have a strong causal effect on the output and directly determine what the result looks like, meanwhile having robust error resilience as they are perturbed by Gaussian noise across different scales.
Numerous works have exploited the convenience of this noise prior to attain controllable generation, such as image-to-image translation \citep{meng2021sdedit}, pixel-level manipulation \citep{nichol2022glide}, image inpainting \citep{lugmayr2022repaint} and semantic editing \citep{choi2021ilvr, hou2024high}. Recent research has shown that, although sampled from a Gaussian distribution, the initial noise of the diffusion process still has a significant influence on the layout of generated content \citep{mao2023guided}.
In other works, noise prior is used to guarantee temporal consistency between video frames \citep{luo2023videofusion}, or to trade off between fidelity and diversity of image editing \citep{kim2022diffusionclip}.
\paragraph{Video Model for 3D Generation}
Similar to how most video generation models use the groundwork laid by image foundation models \citep{blattmann2023alignVideo-LDM, esser2023GEN1, singer2022make, tuneavideo}, the training of 3D generation models also relies heavily on pretrained 2D video models \citep{sv3d, im3d, shi2023mvdream, chen2024v3d, han2024vfusion3d}. These methods either finetune with rendered videos directly \citep{SVD,chen2024v3d, im3d, han2024vfusion3d}, or add camera embedding as an extra condition for each view \citep{sv3d, shi2023mvdream}. Video foundation models have been shown to be particularly beneficial in generating consistent multi-view rendering of 3D objects, demonstrating their inherent and abundant prior knowledge for handling camera pose change.
\section{Training-free Camera Control for Video Generation}
\begin{figure}
\centering
\includegraphics[width=1.0\linewidth]{figs/fig2.pdf}
% \vspace{-0.3cm}
\caption{\textbf{Pipeline of \ours}.
In stage I, camera movements are modeled through explicit 3D point cloud. In stage II, the layout prior of noisy latents is utilized to guide video generation.
}
\label{fig2}
\end{figure}
CamTrol takes two stages to invoke the innate camera control ability hidden in foundation models. In Sec.~\ref{sec: camera motion modeling}, we will describe how to model explicit camera movement for video generation. In Sec.~\ref{sec: noise prior}, we will elaborate on video motion control with the guidance of noise layout prior.
\subsection{Camera Motion Modeling}
\label{sec: camera motion modeling}
To invoke pretrained video diffusion models’ ability to deal with camera perspective changes, hints of camera motion should be injected into the diffusion model in a proper way. While simply concatenating camera trajectories with text prompts is incomprehensible to the original model, previous works \citep{wang2023motionctrl, he2024cameractrl} introduce additional embedders to encode camera parameters and finetune with limited annotated data \citep{realestate10k, yu2023mvimgnet}, which are data-hungry yet lack generalization ability. Other methods \citep{yang2024direct} construct camera motions by self-supervised augmentations, but can only handle a few easy camera controls. Thus, we seek a more efficient and robust way to guide the model towards being camera-controllable.
Considering that video perspective changes are originally caused by camera movements in 3D space, we leverage 3D representation to provide generation models with explicit motion hints. Specifically, we choose point cloud as the intermediate representation, in which we can easily manipulate camera poses to simulate diverse movements. An additional benefit that point cloud brings is its data efficiency: By utilizing inpainting techniques, only one single input image is required for the entire point cloud reconstruction. This eliminates the need of large-scale finetuning.
% Besides explicit camera modeling, introducing point cloud brings extra benefits: The first is its data-efficiency. By utilizing inpainting techniques, only one single input image is required for the whole point cloud reconstruction, sidestepping the effort of large-scale finetuning. Second, consistency between multi-view renderings can be easily guaranteed, as the known points will remain unchanged once the reconstruction finished.
\paragraph{Point Cloud Initialization}
We start by lifting pixels in the input image plane into 3D point cloud space. In practice, the input image can be either user-defined or created by image generators like Stable Diffusion \citep{rombach2022high}. Given an input image $\mathbf{I}_0 \in \mathbb{R}^{3 \times H \times W}$, we first estimate its depth map $\mathbf{D}_0$ using the off-the-shelf monocular depth estimator ZeoDepth \citep{bhat2023zoedepth}. By combining the image and its depth map, we initialize the point cloud $\mathcal{P}_0$ as:
\begin{equation}
\mathcal{P}_0 = \phi ([\mathbf{I}_0, \mathbf{D}_0], \mathbf{K}, \mathbf{P}_0),
\end{equation}
where $\phi$ denotes the mapping function from RGBD to 3D point cloud, $\mathbf{K}$ and $\mathbf{P}_0$ represent camera's intrinsic and extrinsic matrices set by convention \citep{chung2023luciddreamer} as they're usually intractable.
\paragraph{Camera Trajectories}
To get consistent images from multiple views, we model the camera motion as a trajectory of extrinsic matrices $\left \{\mathbf{P}_1, ... , \mathbf{P}_{N-1} \right \}$, each including a rotation matrix and a translation matrix representing camera pose and position.
At each step $i$, we project the point cloud back to the camera plane using $\psi$ and get a rendered image with perspective change: $\mathbf{I}_i = \psi (\mathcal{P}_i, \mathbf{K}, \mathbf{P}_i)$. By computing the extrinsic matrices of different movements, we obtain a series of camera motions including zoom, tilt, pan, pedestal, truck, roll, and rotation, enabling flexible camera movements. Detailed definitions of these movements are provided in Appendix~\ref{apx: definitions}. Basic trajectories are combined to produce hybrid camera movements, adding cinematic appeal to the generated videos. Furthermore, benefiting from explicit camera motion modeling, our method can support trajectories with precise extrinsics, which means it can generate videos with arbitrarily complex camera motions.
\paragraph{Multi-view Consistency}
When perspective changes, holes can appear as some areas are unoccupied within the point cloud. To obtain more reasonable results, we employ image inpainting \citep{rombach2022high} to fill the gaps in new renderings, with a mask distinguishing the known points from the nonexistent ones. This operation ensures coherence between the known and novel views in 2D space. After inpainting, the image is lifted back into 3D space and gradually completes the whole point cloud. During this process, points between adjacent views may become misaligned since the depth estimator only estimates relative depth, leading to inconsistencies in 3D point cloud and rendered images. To avoid this situation, we adopt depth coefficient optimization \citep{chung2023luciddreamer} at each step of the camera movement, formed as:
\begin{equation}
d_i = \operatorname*{argmin}_{d} \left(\sum_{M} \left \| \phi ([\tilde{\mathbf{I}}_i, d \tilde{\mathbf{D}}_i], \mathbf{K}, \mathbf{P}_i) - \mathcal{P}_{i-1} \right \| \right),
\end{equation}
where $\tilde{\mathbf{I}}_i$ and $\tilde{\mathbf{D}}_i$ refer to the inpainted image and the corresponding depth map, respectively, $d_i$ denotes the depth coefficient to be optimized, and $M$ refers to the overlapping region between $\mathcal{P}_{i}$ and $\mathcal{P}_{i-1}$, as other areas are not shared for calculating $\ell_1$ loss.
Thus, we get a set of images that correspond to the input and indicate specific camera movements:
\begin{equation}
\left \{ \mathbf{I}_0, ... , \mathbf{I}_{N-1} \right \} = \left \{ \psi (\mathcal{P}_i, \mathbf{K}, \mathbf{P}_i) | i \in \left [ 0, N-1 \right ] \right \} .
\end{equation}
\subsection{Layout Prior of Noise}
\begin{figure}
\centering
\includegraphics[width=1.0\linewidth]{figs/explain-v3.pdf}
% \vspace{-0.3cm}
\caption{\textbf{Transition of samples between two distinct distributions.}
% The blue and yellow shaded areas represent distribution of images and video, respectively, with the dots indicating samples within them.
As the layout-arranged images are inverted by adding random noise, the distribution of their noisy latents will gradually converge to that of their video counterpart (green area), eventually forming a nearly Gaussian latent with specific layout change. This information is then inherited during the generation process. The inversion step determines the trade-off between video diversity and motion fidelity.}
\label{explain}
\end{figure}
\label{sec: noise prior}
With camera motion modeling, we obtain a sequence $\mathbf{V}_0 = \left \{ \mathbf{I}_0, ... ,\mathbf{I}_{N-1} \right \} \in \mathbb{R} ^{N \times 3 \times H \times W} $ of rendered images adhering to a specific camera trajectory. Note that the quality of rendered images is not perfect since a single input image only leads to sparse point cloud reconstruction. Moreover, these renderings are static, thus they cannot be used directly as video frames. To form an ideal video, we need to find a way that satisfies three requirements: 1) camera motions should be maintained; 2) the video should be enriched with more dynamics; and 3) quality imperfection should be compensated.
\paragraph{Camera Motion Inversion}
Recent work on diffusion models has demonstrated the strong controllability of their noisy latents \citep{meng2021sdedit, mao2023guided}, the causality and error-resilience they possess with respect to the final output make them a convenient yet powerful tool for controllable generation in diffusion models.
Particularly for initial noise, even when sampled from a Gaussian distribution, it still has significant influence on the layout of the generated image \citep{mao2023guided}. For instance, if all pixels in initial noise shift to the right by a certain distance, it is likely that the generated output reflects a similar shift. This reminds us that the impact of camera movement on images can also be interpreted as a type of layout rearrangement, where pixels change their positions due to viewpoint change. In a similar way, videos can be reorganized following camera motion if their noisy latents change accordingly.
Inspired by this, we first construct a series of noisy latents indicating specific camera movements.
It can be done intuitively by employing diffusion's inversion process on the rendered image sequence $\mathbf{V}_0$. Latent at timestep $t_0$ can be calculated as follows, where $\bar{\alpha}_t$ is the variance of the scheduler:
\begin{equation}
\mathbf{V}_{t_0} = \sqrt{\bar{\alpha}_{t_0}} \mathbf{V}_0 + \sqrt{1-\bar{\alpha}_{t_0}} \epsilon, \quad \epsilon \sim \mathcal{N} (\mathbf{0},\mathbf{I}),
\end{equation}
Because the rendered images $\mathbf{V}_0$ share common pixels in certain regions, their latents are also correlated to each other in a way reflecting pixel movement.
Moreover, by introducing random noise, blank spaces and flawed regions in $\mathbf{V}_0$ can be adaptively completed, providing the video model with more possibilities to generate and correct them.
\paragraph{Video Generation}
After camera motion inversion, noisy latents representing camera movements are then passed through the backward process of the video diffusion model, leveraging their layout controllability to guide video generation. Using prior knowledge of the base model, the generation process also infuses the video with rational dynamic information. In this way, explicit camera movements are injected into the video diffusion model in an appropriate, training-free fashion.
Starting from noisy motion latents at timestep $t_0$, the generation step can be represented as:
\begin{equation}
\hat{\mathbf{V}}_{t-1} = \sqrt{\alpha_{t-1}}\left ( \frac{\mathbf{V}_t-\sqrt{1-\alpha_t}\epsilon^{(t)}_{\theta}(\mathbf{V}_t)}{\sqrt{\alpha_t}}\right ) +\sqrt{1-\alpha_{t-1}-\sigma^2_t}\epsilon^{(t)}_{\theta}(\mathbf{V}_t) + \sigma_t \epsilon, \quad t \in \left [ 1, t_0 \right ].
\label{eq: ddim}
\end{equation}
Here $\epsilon_{\theta}$ denotes the video model for noise prediction and $\sigma_t$ determines whether the denoising process is deterministic or probabilistic. We set $\sigma = 1$ to encourage diversity in generation process.
\paragraph{Trade-off Between Fidelity and Diversity}
Leveraging noise prior guidance in diffusion models could lead to a trade-off between generation fidelity and diversity \citep{meng2021sdedit, hou2024high}, where results that are more faithful to the guidance tend to degrade in generation quality. In this task, similar circumstances also occur, as the model is required to be guided by some imperfect renderings while generating a reasonable video. The key factor in balancing the trade-off problem lies in the choice of $t_0$. When a smaller $t_0$ is applied, the generated output bears more resemblance to original guidance $\mathbf{V}_0$, but lacks rationality and dynamics to be an appealing video. Instead, a larger $t_0$ leads to better-generated videos but makes them less aligned with the desired camera motion. Empirically, we find that a smaller $t_0$ works better for motions with moderate intensity, and for those with relatively drastic movements, a larger $t_0$ shows preferable performance. The process of stage II is illustrated in Fig.~\ref{explain}.
\section{Experiments}
\begin{table}[t]
\caption{\textbf{Quantitative comparisons.} Our method attains comparable performance with finetuned methods in both video generation quality and camera motion alignment.
}
\label{tab1}
\centering
\begin{tabular}{lccccccc}
\toprule
\multirow{2}{*}{Method} &
\multicolumn{4}{c}{\textbf{Video Quality}}&
\multicolumn{3}{c}{\textbf{Motion Accuracy}}
\\
% \cmidrule(r)
& FVD
% $\left ( \downarrow \right ) $
$\downarrow$
& FID
% $\left ( \downarrow \right ) $
$\downarrow$
& IS
% $\left ( \uparrow \right ) $
$\uparrow$
& CLIP-SIM
% $\left ( \uparrow \right ) $
$\uparrow$
& ATE
% $\left ( \downarrow \right ) $
$\downarrow$
& RPE-T
% $\left ( \downarrow \right ) $
$\downarrow$
& RPE-R
% $\left ( \downarrow \right ) $
$\downarrow$
\\
\midrule
\textit{SVD} & 1107.93 & 68.51 & 7.21 & 0.3095 & 4.23 & 1.79 & 0.021 \\
MotionCtrl+SVD & \underline{810.59} & 69.03 & \textbf{7.17} & 0.3076 & \underline{4.19} & \textbf{1.07} & \underline{0.012} \\
CameraCtrl+SVD & 951.80 & \textbf{67.59} & 6.82 & \textbf{0.3138} & 4.22 & 1.17 & 0.013 \\
\textbf{CamTrol+SVD} & \textbf{778.46} & \underline{68.06} & \underline{7.05} & \underline{0.3110} & \textbf{4.17} & \textbf{1.07}
& \textbf{0.010} \\
\textit{Reference} & - & - & - & - & \textit{3.60} & \textit{0.89}
& \textit{0.008} \\
\bottomrule
\end{tabular}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figs/comparison-new.pdf}
\caption{\textbf{Qualitative comparisons with finetuned methods.} CamTrol's outputs align well with complex trajectories from the reference video, while others fail to capture subtle changes in camera pose. We provide videos in the supplementary materials for clearer comparison.}
\label{comparison}
\end{figure}
\begin{table}[t]
\centering
\caption{\textbf{Computational analysis of inference process,} evaluated under unified settings.}
\label{computational320}
\begin{tabular}{lccccc}
\toprule
\multicolumn{2}{c}{ } & SVD & MotionCtrl & CameraCtrl & CamTrol ($t_0=15$) \\ \midrule
\multicolumn{2}{c}{Max GPU memory(MB)} & 11542 & 31702 & 26208 & 11542 \\
\midrule
\multirow{2}{*}{Time (s)} & pre-process & - & - & - & 56
\\
& inference & 11 & 32 & 42 & 8
\\
\bottomrule
\end{tabular}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figs/generic.pdf}
\caption{\textbf{Generalization comparisons.} CamTrol can avoid domain collapse that arises from overfitting on certain datasets and adapt to more general scenes (\textit{Left}), meanwhile preserving video's dynamics while adhering to desired camera movements (\textit{Right}).}
\label{generic}
\end{figure}
\subsection{Experimental Settings}
\paragraph{Implementation Details}
We compare our method with state-of-the-art works including MotionCtrl \citep{wang2023motionctrl} and CameraCtrl \citep{he2024cameractrl}. To ensure a fair comparison, we employ SVD \citep{SVD} as base model for all methods. SVD is originally trained at a resolution of $576\times1024$, but SVD-based CameraCtrl only supports $320\times576$. Since changing the original resolution results in suboptimal generation quality, we use $576\times1024$ for MotionCtrl and CamTrol, then resize the outputs to $320\times576$ to calculate metrics. For all methods, the number of frames and the decoding size of SVD are set to 14. We use 25 steps for both the inversion and generation processes.
\paragraph{Evaluation Details}
In the quantitative evaluation, FVD \citep{unterthiner2018FVD}, FID \citep{heusel2017FID} and IS \citep{saito2020IS} are used to assess video generation quality, and CLIPSIM \citep{wu2021CLIPSIM} quantifies the similarity between the generated video and the input prompt. For camera motion accuracy, we adopt ParticleSFM\citep{zhao2022particlesfm} to estimate camera trajectories from generated videos, with the use of Absolute Trajectory Error (ATE) to measure their differences from the ground truth. Relative Pose Error (RPE) is calculated to assess how well the relative motions between consecutive frames match the expected ones, including their translation (RPE-T) and rotation component (RPE-R). Specifically, we randomly sample 500 prompt-trajectory pairs from RealEstate10k \citep{realestate10k}, and use them as references for calculating FVD and FID. Since SVD is an image-to-video model, we generate the first frames using Stable Diffusion \citep{rombach2022high} based on text prompts. We also provide the results produced by vanilla SVD as a reference. For camera motion accuracy, we provide the evaluations on ground truth videos as a lower bound of these metrics.
\subsection{Comparisons with State-of-the-art Methods}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figs/ori_compare_row.pdf}
\vspace{-0.3cm}
\caption{\textbf{Comparison with base model.} Controlling camera motion via prompt engineering rarely succeeds. Instead, CamTrol offers robust control to video's camera movements.}
\label{ori}
\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figs/warp_compare_row.pdf}
\vspace{-0.3cm}
\caption{\textbf{Effectiveness of layout prior. } Layout prior guidance compensates for the voids (\textit{Left}) and static elements (\textit{Right}) in point cloud.}
\label{warp}
\end{figure}
\paragraph{Quantitative Evaluation}
Quantitative evaluations are shown in Table ~\ref{tab1}. In the table, the best results are in bold, and the second best are underlined. The performance of vanilla SVD (without motion control) is denoted as \textit{SVD}, while the lower bound for motion metrics, provided by ground truth videos, is denoted as \textit{Reference}.
In terms of video quality, CamTrol attains performance comparable to methods finetuned on the RealEstate10k dataset. For motion accuracy, CamTrol also achieves the lowest score in ATE and RPE-T/R. The quantitative evaluation demonstrates CamTrol's ability to generate videos with both accurate camera motion and high visual quality.
\paragraph{Qualitative Analysis}
Qualitative comparisons are illustrated in Fig.~\ref{comparison}.
The reference trajectory includes zoom, pan, and roll. While MotionCtrl and CameraCtrl fail to capture subtle camera motions, resulting in simple pan movements, CamTrol is able to follow the complex trajectory and generate videos with correct motion.
We also evaluate the generalizability of different methods in generating more general scenes and dynamic content. The results are shown in Fig.~\ref{generic}. Since both MotionCtrl and CameraCtrl are finetuned on limited scenes (i.e., real estate videos) with static content, they have difficulty generalizing to other scenes, such as videos in non-realistic styles or videos that include people. As illustrated in Fig.~\ref{generic}, their motion controls in both situations are misaligned with the intended movements. Furthermore, finetuning on such datasets leads to a loss of dynamics, which is a crucial element in video generation. In comparison, CamTrol preserves most of the prior knowledge from video base models, enabling it to handle general scenes as well as generate dynamic content. Relevant videos are available in the supplementary materials.
% All methods generate videos align well with desired camera motion, however, the constraints imposed by camera trajectories are sometimes too strong, that output videos have to compromise their quality and rationality to satisfy these movements. Some examples that demonstrate this are results of \textit{Zoom Out} and \textit{Roll Anticlockwise}, which looks more like flat images being zoomed or rolled on 2D plane instead of videos recorded in 3D spaces. The phenomenon primarily results from the limitation of the proprietary dataset used for finetuning \citep{realestate10k, yu2023mvimgnet}, where most scenes are static, leading to a diminished understanding of dynamic contents in pretrained model. In contrast, \ours~makes full use of the powerful dynamics in base model and generate plausible videos with camera perspective change utilizing layout prior guidance, handles these circumstances gracefully.
\paragraph{Computational Analysis}
We provide the computational analysis in Table \ref{computational320}, including the maximum GPU memory required and the processing time for all methods during inference. Evaluations are conducted under unified settings. We test at a resolution of $576\times320$ with 25 generation steps. The number of frames and the decoding size of SVD are set to 14. As a training-free method, CamTrol requires no extra GPU memory during the inference process compared to the base model. This saves 10-20GB of GPU memory compared to other methods under the same circumstances, allowing it to run on a single RTX 3090. The major consumption of CamTrol comes from rendering multi-view images. Since this process is temporally sequential, i.e., $t_0 \to t_1 \to t_2$, a more parallelized approach may improve time efficiency. The results for $576\times1024$ resolution and more detailed settings can be found in the Appendix.
\subsection{Ablation Study}
% \begin{table}[t]
% \centering
% \caption{\textbf{Ablation on depth coefficient optimization.} Depth optimization offers better consistency of point cloud, which benefits both perceptual quality and camera motion accuracy.}
% \label{ablation}
% \begin{tabular}{cccccccc}
% \toprule
% \textbf{CamTrol+SVD} & FVD$\downarrow$& FID$\downarrow$ & IS$\uparrow$ & CLIP-SIM$\uparrow$ & ATE$\downarrow$ & RPE-T$\downarrow$ & RPE-R$\downarrow$ \\ \midrule
% w/o optimization & 5033.44 & 219.01 & 5.67 & 0.2949 & 4.093 & 1.131 & 0.046 \\
% w/ optimization & \textbf{4977.88} & \textbf{218.36} & \textbf{5.72} & 0.2947 & \textbf{3.906} & \textbf{0.978} & 0.049 \\
% \bottomrule
% \end{tabular}
% \end{table}
\begin{table}[t]
\caption{\textbf{Quantitative effect of $t_0$.}}
\label{table_t0}
\centering
\begin{tabular}{lccccccc}
\toprule
\multirow{2}{*}{$t_0$} &
\multicolumn{4}{c}{\textbf{Video Quality}}&
\multicolumn{3}{c}{\textbf{Motion Accuracy}}
\\
% \cmidrule(r)
& FVD
% $\left ( \downarrow \right ) $
$\downarrow$
& FID
% $\left ( \downarrow \right ) $
$\downarrow$
& IS
% $\left ( \uparrow \right ) $
$\uparrow$
& CLIP-SIM
% $\left ( \uparrow \right ) $
$\uparrow$
& ATE
% $\left ( \downarrow \right ) $
$\downarrow$
& RPE-T
% $\left ( \downarrow \right ) $
$\downarrow$
& RPE-R
% $\left ( \downarrow \right ) $
$\downarrow$
\\
\midrule
$t_0=20$ & 1079.88 & 68.52 & 7.14 & 0.3100 & 4.17 & 1.09 & 0.012 \\
$t_0=15$ & 778.46 & 68.06 & 7.05 & 0.3110 & 4.17 & 1.07 & 0.010 \\
$t_0=10$ & 754.14 & 67.98 & 7.00 & 0.3107 & 4.13 & 1.02 & 0.008 \\
\bottomrule
\end{tabular}
\end{table}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.49\textwidth}
\centering
\includegraphics[scale=0.38]{figs/t0.pdf}
\caption{\textbf{Effect of $t_0$.} Larger $t_0$ encourages\\ dynamics while smaller $t_0$ preserves camera \\movements (\textit{Pedestal Down}).}
\label{t_0}
\end{minipage}
\begin{minipage}[t]{0.49\textwidth}
\centering
\includegraphics[scale=0.38]{figs/scale.pdf}
\caption{\textbf{3D rotation videos at different motion scales.}
CamTrol supports camera movements across multiple scales.
}
\label{scale}
\end{minipage}
\end{figure}
\paragraph{Comparison to Base Model}
To demonstrate that the changes in camera motion are attributed to our method rather than the innate capability of the video model, we conduct an ablation study to evaluate its effectiveness. We add prompts describing specific camera movements (e.g. \textit{zooms out}) and let the video model interpret them on its own. The results are presented in Fig.~\ref{ori}. It can be observed that, even when provided with prompts indicating how the camera should move, the base model fails to produce correct results. In contrast, \ours~is able to implement the designated motion control without any instructions from text prompts. In Table \ref{tab1}, the comparisons with vanilla SVD also demonstrate CamTrol's effectiveness.
\paragraph{Effectiveness of Layout Prior}
We conduct an ablation study to validate the effectiveness of layout prior guidance, demonstrating its necessity in two aspects: the completeness of missing areas and the dynamics of the generated video. In Fig.~\ref{warp}, we showcase frames before and after applying noise prior guidance. As camera pose changes, there are regions left unfilled in the point cloud, leading to blank spaces in rendered images (left part); Additionally, due to the static nature of point cloud, the rendered images remain stationary (right part). Layout prior guidance compensates for these flaws, ultimately producing videos with inpainted gaps and natural dynamics.
% \paragraph{Ablation on Depth Optimization}
% Ablation results relevant to the effectiveness of depth optimization are presented in Table~\ref{ablation}. Experiments are conducted on complicated 10 trajectories extracted from RealEstate10k, each with 10 prompts mentioned in MotionCtrl. It could be validated that depth optimization offers better alignment between adjacent views rendered from point cloud, which benefits both perceptual quality and camera motion accuracy of generated videos.
\paragraph{Effect of Timestep $t_0$}
$t_0$ is a crucial factor that influences the trade-off between generated video's diversity and its faithfulness to camera motion requirements. To investigate its effect on the output, we conduct experiments with various $t_0$ values, the relevant results are shown in Fig.~\ref{t_0}. As illustrated, videos generated with smaller $t_0$ tend to conform better to camera motion requirements, but suffer from decreased dynamics; On the other hand, larger $t_0$ leads to more plausible generations but fails to meet camera's requirements, since the latents at these timesteps carry more randomness. We also provide quantitative evaluations for different $t_0$ values in Table \ref{table_t0}. As $t_0$ decreases, CamTrol produces videos resembling static, camera-moving scenes (which have lower FVD as we use RealEstate10k as the reference videos) with higher accuracy in motion control.
\paragraph{Generalization to Diverse Situations}
Our proposed \ours~can be seamlessly integrated into various scenarios, accommodating different base models, resolutions, and generation lengths, all in a training-free style. We present visual results of its applications under different settings, including CogVideoX \citep{yang2024cogvideox} (diffusion transformer model, $720\times480$ resolution, 49 frames) and VideoFusion \citep{luo2023videofusion} (decomposed diffusion process, $128\times128$ resolution, 16 frames), in Fig.~\ref{other}. Our approach remains effective when applied to alternative video base models, resolutions, and generation lengths, demonstrating its strong robustness and generalizability.
\subsection{Further Applications}
\begin{figure}[t]
\centering \includegraphics[width=1.0\linewidth]{figs/multi-traj.pdf}
% \vspace{-0.3cm}
\caption{\textbf{Multi-trajectory video generation.} CamTrol is able to generate videos with different camera motions for the same scene.}
\label{multi-traj}
\end{figure}
\begin{figure}[t]
\centering \includegraphics[width=1.0\linewidth]{figs/other_backbones.pdf}
% \vspace{-0.3cm}
\caption{\textbf{Applied to CogVideoX \citep{yang2024cogvideox} and VideoFusion \citep{luo2023videofusion}.} CamTrol can be plug-and-play under various situations, accommodating different base models, different resolutions, different generation lengths, all in a training-free manner.}
\label{other}
\end{figure}
\paragraph{3D Rotation Videos}
One of the main advantages of our method is that it can generate videos with rotating movements and produce outputs similar to those of 3D generation models \citep{sv3d, im3d}. While these 3D models require large-scale training on 3D datasets and can only handle inputs in specific styles, our approach is capable of dealing with any type of image and achieving this in a completely zero-shot manner. An example is shown in Fig.~\ref{scale}.
% more results of unsupervised 3D video generation could be found on the \href{https://anonymous.4open.science/w/CamTrol-anonymous-5509/}{\textbf{anonymous demo page}}.
\paragraph{Hybrid and Complex Camera Movements}
By combining basic camera trajectories, \ours~can support hybrid camera movements and produce videos with cinematic appeal. In addition, explicit motion modeling equips CamTrol with the ability to handle trajectories of precise extrinsics, allowing it to generate videos with arbitrarily complicated camera movements. Relevant results can be found at \textcolor{cyan}{\href{https://lifedecoder.github.io/CamTrol/}{\textbf{https://lifedecoder.github.io/CamTrol/}}}.
% Videos of hybrid and complex motions are exhibited on the \href{https://anonymous.4open.science/w/CamTrol-anonymous-5509/}{\textbf{anonymous demo page}}.
\paragraph{Multi-trajectory Video Generation}
One natural application of our method is to generate multi-trajectory videos for a single scene. Since the point cloud remains fixed once the reconstruction has finished, the spatial consistency between different trajectories is guaranteed, making it easy to produce multiple camera-moving videos of the same scene. We demonstrate this application in Fig. \ref{multi-traj}. More results on multi-trajectory video generation can be found in the supplementary materials.
\paragraph{Camera Motion at Different Scales}
\ours~supports camera movements at controllable scales. By specifying different magnitudes of the camera's extrinsic matrix, the rendered images will exhibit varying degrees of motion, resulting in videos with distinct scales of camera movements. This further demonstrates the powerful controllability of \ours~and provides a new pathway for customized camera control of video generation. We provide relevant visualization in Fig.~\ref{scale}.
% , more video results could be found on the \href{https://anonymous.4open.science/w/CamTrol-anonymous-5509/}{\textbf{anonymous demo page}}.
\section{Conclusion}
In this paper, we propose a training-free and robust method \textbf{\ours} to offer camera control for off-the-shelf video diffusion models. It consists of a two-stage procedure, including explicit camera motion modeling in 3D point cloud space and video generation utilizing the layout prior of noisy latents.
Compared to previous works, \ours~does not require additional finetuning on camera-annotated datasets or self-supervised training via data augmentation. Comprehensive experiments demonstrate its superior performance in both generation quality and motion alignment against other state-of-the-art methods.
Furthermore, we show the ability of \ours~to generalize to various scenarios, as well as its impressive applications including unsupervised generation of 3D video, scalable motion control, and dealing with complicated camera trajectories.
% instead, it could be plugged and played with most video diffusion models to generate camera controllable videos with single image or text prompt as input.
% Comprehensive experimental results demonstrate the effectiveness and robustness of \ours~for controlling camera motion for videos.
% Besides, we show that \ours~produces impressive results in generating videos with 3D rotations, complicated trajectories, and different moving scales.
\newpage
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
\appendix
% \section{Appendix}
\section{More Results on Camera Control }
\label{apx: camtrol}
We present additional qualitative results of CamTrol on our demo page:
\textcolor{cyan}{\textbf{\href{https://lifedecoder.github.io/CamTrol/}{https://lifedecoder.github.io/CamTrol/}}}
This demo page includes CamTrol-generated videos including basic camera motions (including \textit{Zoom, Tilt, Pan, Pedestal, Truck, Roll, Rotate, Hybrid, Complicated}, detailed definitions are in Appendix \ref{apx: definitions}), hybrid motions (\textit{Zoom In first, then Pedestal Up}, \textit{Zoom Out + Pedestal Up + Truck Left + Tilt Down + Pan Right}), and complicated motions generated from precise camera extrinsics (extracted from RealEstate10k~\citep{realestate10k}).
In addition, it contains 3D rotation videos generated unsupervised from video base models (both objects and scenes). These outputs share similarities with those of 3D generation models, as they all exhibit in a turning-table-like way, with the camera rotating around objects. The difference here is that the 3D model, as trained on specific datasets, can only generate outputs in certain styles, \textit{e.g.}, single static object with no background. Instead, our model can handle arbitrary images as input and generate rotating videos with appropriate dynamics. From this perspective, our method could be seen as an infinite source of 3D data. By employing our method with stronger backbones, video foundation models could truly become the largest source of 3D data, as they should be.
Furthermore, it contains additional results discussed in ablation studies, e.g., controlling camera motions at different scales, and the effectiveness of using layout prior compared with the raw output given by the video base model.
\section{Definitions of Basic Camera Motions}
\label{apx: definitions}
We refer to the terminology in cinematography to describe different camera motions, the definitions of each type are detailed in Table \ref{tab2}.
To obtain consistent images from multiple views, we set camera motion as a trajectory of extrinsic matrices $\left \{\mathbf{P}_1, ... , \mathbf{P}_{N-1} \right \}$, each consisting of a rotation matrix and a translation matrix, representing the camera pose and position. For hybrid motions, CamTrol supports both spatial (i.e., performing several basic motions simultaneously) and temporal (i.e., concatenating basic motions sequentially) combinations. In the case of complicated trajectories, one can directly use precise parameters for camera's extrinsic matrix as input to control video's motion. Additionally, if these parameters are not available, a reference video with corresponding move can also serve as input. With the help of SFM algorithms (e.g., COLMAP \footnote{https://github.com/colmap/colmap}, ParticleSFM \citep{zhao2022particlesfm}), camera motion can be easily estimated and used for imitation. In this sense, CamTrol can also be seen as an unsupervised method to achieve video motion transfer.
\section{Visual Comparisons with State-of-the-arts}
\label{apx: comparison}
Sec.~\ref{comparison} showcases some qualitative comparisons between CamTrol and different state-of-the-art approaches. For better visualization and comparison, we provide more video results in the supplementary materials.
\section{Implementation Details}
For text prompt input, we use Stable Diffusion v2-1 \footnote{https://huggingface.co/stabilityai/stable-diffusion-2-1}
or Stable Diffusion XL \footnote{https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0} to generate the initial image. The inpainting model we apply is Stable Diffusion inpainting model proposed by Runway \footnote{https://huggingface.co/runwayml/stable-diffusion-inpainting}, and the backward step of inpainting is set to 25. We use ZeoDepth \footnote{https://github.com/isl-org/ZoeDepth} as depth estimation model. The classifier-free guidance scale follows the default setting of the base models themselves. We use SVD's default setting of 6 fps for video generation, and process reference videos to 6 fps for FVD and FID calculations. The complete procedure of CamTrol is elucidated in Algorithm~\ref{algo}.
For computational analysis, we set both the number of frames and the decoding size to 14, and the generation steps to 25. We do not apply xformers in any of the approaches. The computational analysis at $576\times1024$ resolution is shown in Table \ref{computation1024}. SVD-based CameraCtrl only supports a resolution at $320\times576$.
\begin{table}[t]
\centering
\caption{\textbf{Computational analysis of on $\textbf{576}\times\textbf{1024}$.}}
\label{computation1024}
\begin{tabular}{lccccc}
\toprule
\multicolumn{2}{c}{ } & SVD & MotionCtrl & CameraCtrl & CamTrol ($t_0=15$) \\ \midrule
\multicolumn{2}{c}{Max GPU memory(MB)} & 34236 & 71096 & - & 34236 \\
\midrule
\multirow{2}{*}{Time (s)} & pre-process & - & - & - & 149
\\
& inference & 32 & 54 & - & 22
\\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{\textbf{Definitions of camera motions.} We adopt the terminology from cinematography to describe different camera movements.}
\label{tab2}
\centering
\begin{tabular}{ccc}
\toprule
\textbf{Camera Motion} &
\textbf{Directions} &
\textbf{Definition}
\\
\midrule
\multirow{2}{*}{Zoom} & In & \multirow{2}{*}{Camera moves towards or away from a subject.} \\
& Out \\
\midrule
\multirow{2}{*}{Tilt} & Up & \multirow{2}{*}{Rotating the camera vertically from a fixed position.} \\
& Down \\
\midrule
\multirow{2}{*}{Pan} & Left & \multirow{2}{*}{Rotating the camera horizontally from a fixed position.} \\
& Right \\
\midrule
\multirow{2}{*}{Pedestal} & Up & \multirow{2}{*}{Moving the camera vertically in its entirety.} \\
& Down \\
\midrule
\multirow{2}{*}{Truck} & Left & \multirow{2}{*}{Moving the camera horizontally in its entirety.} \\
& Right \\
\midrule
\multirow{2}{*}{Roll} & Clockwise & \multirow{2}{*}{Rotating the camera in its entirety in a horizontal manner.} \\
& Anticlockwise \\
\midrule
\multirow{2}{*}{Rotate} & Clockwise & \multirow{2}{*}{Moving the camera around a subject.} \\
& Anticlockwise \\
\midrule
\multirow{2}{*}{Hybrid} & \multirow{2}{*}{Combinations} & \multirow{2}{*}{Spatial and temporal combinations of basic motions.} \\
\\
\midrule
\multirow{2}{*}{Complicated} & \multirow{2}{*}{Arbitrary} & \multirow{2}{*}{Camera extrinsic matrices or reference videos as input.} \\
\\
\bottomrule
\end{tabular}
\end{table}
\section{Choice of 3D Representation}
\begin{algorithm}[t]
\caption{Training-free camera control for video generation}
\label{algo}
\SetAlgoLined
\BlankLine
\KwIn{Text prompt $p$, camera motion $\mathbf{P}$, input image $\mathbf{I}_0$ (\textit{optional}).}
\BlankLine
\tcp{Stage I: Camera Motion Modeling:}
\BlankLine
\For{i=1,...,N-1}{
$\tilde{\mathbf{I}}_i = \text{inpainting}(\mathbf{I}_{i-1}, \mathbf{P}_i, p)$ \;
$\tilde{\mathbf{D}}_i = \text{depth}(\tilde{\mathbf{I}}_i)$ \;
\While{not converged}{
$d_i = \operatorname*{argmin}_{d} \left(\sum_{M} \left \| \phi ([\tilde{\mathbf{I}}_i, d \tilde{\mathbf{D}}_i], \mathbf{K}, \mathbf{P}_i) - \mathcal{P}_{i-1} \right \| \right)$
}
$\mathcal{P}_i = \phi ([\tilde{\mathbf{I}}_i, d_i \tilde{\mathbf{D}}_i], \mathbf{K}, \mathbf{P}_i)$ \;
$\mathbf{I}_i = \psi (\mathcal{P}_i, \mathbf{K}, \mathbf{P}_i)$ \;
}
\BlankLine
\tcp{Stage II: Layout Prior Generation:}
\BlankLine
$\mathbf{V}_0 \gets \left \{ \mathbf{I}_i \right \}_{i=0}^{N-1} $ \;
Sample random noise $\epsilon \sim \mathcal{N} (\mathbf{0},\mathbf{I})$ \;
Motion inversion $\mathbf{V}_{t_0} \gets \sqrt{\bar{\alpha}_{t_0}} \mathbf{V}_0 + \sqrt{1-\bar{\alpha}_{t_0}} \epsilon$ \;
\For{t=$t_0$,...,1}{
$\mathbf{V}_{t-1} \gets \sqrt{\alpha_{t-1}}\left ( \frac{\mathbf{V}_t-\sqrt{1-\alpha_t}\epsilon^{(t)}_{\theta}(\mathbf{V}_t)}{\sqrt{\alpha_t}}\right ) +\sqrt{1-\alpha_{t-1}-\sigma^2_t}\epsilon^{(t)}_{\theta}(\mathbf{V}_t) + \sigma_t \epsilon$
}
\BlankLine
\end{algorithm}
In Sec.~\ref{sec: camera motion modeling}, we designate point cloud as the intermediate 3D representation for explicit camera motion modeling. Doubts may arise as to why we do not use a more complex 3D representation which might be more useful. Here we take the most prevalent 3D representation--3D Gaussian Splatting \footnote{https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}, as an example to elaborate on the benefits of using point cloud in three aspects:
Firstly, concerning the input, point cloud is able to construct the entire scene from one single input image combining techniques of depth estimation and inpainting. GS, though also an explicit 3D representation, requires optimization based on images from different views, which means it is neither capable of handling single input image, nor can it leverage 2D inpainting to gradually complete the scene.
Secondly, from the aspect of time, point cloud can directly lift 2D points onto 3D spaces, while 3DGS demands optimization for each scenario. As a training-free approach, our method takes almost no time to generate a video after multi-view images are acquired, but would need more time if 3DGS were applied.
Lastly, from the task itself, the goal of stage I is not to precisely reconstruct the 3D scene but rather to offer a rough layout guidance. In this context, rendered images from point cloud are sufficiently qualified, and no further refinement of the 3D reconstruction is necessary.
From the analysis, point cloud is quite qualified serving as a rough layout guidance in a relatively quick speed, without any further optimization or redundant multi-view images as input. The use of point cloud allows our algorithm to be completely training-free, while still being able to produce camera-moving videos quickly with only one single image or text prompt as input.
% Negative prompt has quite influential impact on the quality of inpainting, we employ unified negative prompt in all settings:
% \quad \textit{out of frame, lowres, text, error, cropped, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, out of frame, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, username, watermark, signature.}
% \begin{algorithm}
% \caption{Training-free camera control for video generation}
% \label{algo}
% \begin{algorithmic}
% \REQUIRE{Text prompt $p$, camera motion $\mathbf{P}$, input image $\mathbf{I}_0$ (\textit{optional}).}
% \FOR{i=1,...,N-1}
% \STATE $\tilde{\mathbf{I}}_i = \text{inpainting}(\mathbf{I}_{i-1}, \mathbf{P}_i, p)$
% \STATE $\tilde{\mathbf{D}}_i = \text{depth}(\tilde{\mathbf{I}}_i)$
% \REPEAT
% \STATE $d_i = \operatorname*{argmin}_{d} \left(\sum_{M} \left \| \phi ([\tilde{\mathbf{I}}_i, d \tilde{\mathbf{D}}_i], \mathbf{K}, \mathbf{P}_i) - \mathcal{P}_{i-1} \right \| \right)$
% \UNTIL{converged}
% \STATE $\mathcal{P}_i = \phi ([\tilde{\mathbf{I}}_i, d_i \tilde{\mathbf{D}}_i], \mathbf{K}, \mathbf{P}_i)$
% \STATE $\mathbf{I}_i = \psi (\mathcal{P}_i, \mathbf{K}, \mathbf{P}_i)$
% \ENDFOR
% \STATE $\mathbf{V}_0 = \left [ \mathbf{I}_0, ... ,\mathbf{I}_{N-1} \right ]$
% \STATE $\epsilon \sim \mathcal{N} (\mathbf{0},\mathbf{I})$
% \STATE $\mathbf{V}_{t_0} = \sqrt{\bar{\alpha}_{t_0}} \mathbf{V}_0 + \sqrt{1-\bar{\alpha}_{t_0}} \epsilon$
% \FOR{t=$t_0$,...,1}
% \STATE $\mathbf{V}_{t-1} = \sqrt{\alpha_{t-1}}\left ( \frac{\mathbf{V}_t-\sqrt{1-\alpha_t}\epsilon^{(t)}_{\theta}(\mathbf{V}_t)}{\sqrt{\alpha_t}}\right ) +\sqrt{1-\alpha_{t-1}-\sigma^2_t}\epsilon^{(t)}_{\theta}(\mathbf{V}_t) + \sigma_t \epsilon$
% \ENDFOR
% \end{algorithmic}
% \end{algorithm}
% \subsection{Techniques of Inpainting}
\section{Details about Generating 3D Rotation Videos}
3D generation models~\citep{sv3d, im3d, shi2023mvdream, chen2024v3d, han2024vfusion3d} are trained on highly-regulated 3D datasets, these datasets are hard to collect, and include only a limited variety of data types (e.g., single static objects with no background). As a consequence, 3D generation models exhibit a very constrained output distribution and can only produce results in certain styles. CamTrol avoids these problems of arduous data collection, laborious finetuning, and output collapse by utilizing the layout prior hidden in video foundation models. Not only does it require no training, but this advantage also benefits CamTrol from inheriting most of the prior knowledge inside video foundation models, such as the diversity of scenarios, the dynamics of moving objects, temporal consistency, etc. Thus, CamTrol is able to generate dynamic 3D content, both objects and scenes, in a totally unsupervised and training-free manner, this is what other methods cannot achieve yet.
Compared with regulated datasets, the problem of processing wild pictures in 3D is that some of the parameters are unknown. In Sec.~\ref{sec: camera motion modeling}, we mentioned that the camera intrinsic matrix $\mathbf{K}$ and the initial extrinsic matrix $\mathbf{P}_0$ are set by convention, as they are usually intractable. Another crucial parameter concerning 3D video generation is the distance between the camera and the content of input image (denoted as $f$), note that input image could be synthetic or real. Considering that most camera rotations are performed around the center point, we extract a patch from the very center of the input image, conduct depth estimations on it, and define the distance $f$ as the average depth. The rotation and translation matrices of camera movement can be formed as:
\begin{equation}
\begin{aligned}
\mathcal{R}_y&=
\begin{bmatrix}
\cos \theta_i & 0 & -\sin \theta_i \\
0 & 1 & 0 \\
\sin \theta_i & 0 & \cos \theta_i
\end{bmatrix}, \quad
t =
\begin{bmatrix}
f \sin \theta_i \\
0 \\
f - f \cos \theta_i
\end{bmatrix},
\\
&\text{where} \quad f = \frac{1}{|P|} \sum_{(j,k)\in P} D(x_0+j, y_0+k).
\end{aligned}
\end{equation}
Here, $i \in [0, N-1]$ and $\theta_i$ refer to the rotation angle around the $y$ axis at step $i$, $D$ denotes the depth estimation of the image, and $P$ represents the patch around the central point $(x_0, y_0)$. In our experiment, we choose $(j, k) \in [-10, 10]$ as the size of the patch.
\section{Motion Blur, Problems and Solutions}
Videos produced by CamTrol need to satisfy certain camera movements, and some drastic perspective changes might cause visible trails, recognized as motion blur of objects or scenes. This phenomenon will appear to be more indispensable when the video base model has a relatively small generation length (e.g., 16 frames) as well as the motion scale becomes larger (e.g., tilt up by 60 degrees or more). To avoid blur issues when controlling video camera motion, we propose several solutions as follows:
\begin{enumerate}
\item According to the analysis above, the blur issue is caused by limited generation frames and large camera movements, thus the most intuitive solution is to either reduce motion scale or use a more capable generation model. For severe perspective changes, the optimal approach would involve employing video foundation models that support larger generation lengths (e.g., CogVideoX~\citep{yang2024cogvideox} supports generating videos with 49 frames). This allows the model to manage motions of equivalent magnitude while experiencing a smaller moving range between adjacent frames, thereby bringing effective alleviation to the blur problem.
\item One can also stack the results of multiple generations to form a complete output, i.e., treating the last frame of the previous generation as the starting frame for the next and integrating them as a whole. This approach is more suitable when using an image-to-video (I2V) base model. Since most open-source video foundation models are text-to-video (T2V), one may consider increasing the step of camera motion inversion $t_0$, which guarantees more fidelity towards input images' content (and motion).
\item Besides the above two approaches, applying frame interpolation to the output is also a common and convenient choice. Many off-the-shelf frame interpolation models are open-source and can be found on GitHub.
\item Lastly, if the video length cannot be altered, it may be necessary to increase the fps of the generated videos for better visual quality. Although a larger frame rate leads to a shorter video duration, it simultaneously makes the visual persistence brought by motions less pronounced, which reduces blur visually.
\end{enumerate}
In our experiments, we take the raw output in all settings.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{47}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bhat et~al.(2023)Bhat, Birkl, Wofk, Wonka, and M{\"u}ller]{bhat2023zoedepth}
Shariq~Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M{\"u}ller.
\newblock Zoedepth: Zero-shot transfer by combining relative and metric depth.
\newblock \emph{arXiv preprint arXiv:2302.12288}, 2023.
\bibitem[Blattmann et~al.(2023{\natexlab{a}})Blattmann, Dockhorn, Kulal, Mendelevitch, Kilian, Lorenz, Levi, English, Voleti, Letts, et~al.]{SVD}
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et~al.
\newblock Stable video diffusion: Scaling latent video diffusion models to large datasets.
\newblock \emph{arXiv preprint arXiv:2311.15127}, 2023{\natexlab{a}}.
\bibitem[Blattmann et~al.(2023{\natexlab{b}})Blattmann, Rombach, Ling, Dockhorn, Kim, Fidler, and Kreis]{blattmann2023alignVideo-LDM}
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung~Wook Kim, Sanja Fidler, and Karsten Kreis.
\newblock Align your latents: High-resolution video synthesis with latent diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 22563--22575, 2023{\natexlab{b}}.
\bibitem[Brooks et~al.(2024)Brooks, Peebles, Homes, DePue, Guo, Jing, Schnurr, Taylor, Luhman, Luhman, et~al.]{sora}
T~Brooks, B~Peebles, C~Homes, W~DePue, Y~Guo, L~Jing, D~Schnurr, J~Taylor, T~Luhman, E~Luhman, et~al.
\newblock Video generation models as world simulators, 2024.
\bibitem[Chen et~al.(2023)Chen, Lin, Tseng, Lin, and Yang]{chen2023motion}
Tsai-Shien Chen, Chieh~Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang.
\newblock Motion-conditioned diffusion model for controllable video synthesis.
\newblock \emph{arXiv preprint arXiv:2304.14404}, 2023.
\bibitem[Chen et~al.(2024)Chen, Wang, Wang, Wang, and Liu]{chen2024v3d}
Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu.
\newblock V3d: Video diffusion models are effective 3d generators.
\newblock \emph{arXiv preprint arXiv:2403.06738}, 2024.
\bibitem[Choi et~al.(2021)Choi, Kim, Jeong, Gwon, and Yoon]{choi2021ilvr}
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon.
\newblock Ilvr: Conditioning method for denoising diffusion probabilistic models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 14367--14376, 2021.
\bibitem[Chung et~al.(2023)Chung, Lee, Nam, Lee, and Lee]{chung2023luciddreamer}
Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung~Mu Lee.
\newblock Luciddreamer: Domain-free generation of 3d gaussian splatting scenes.
\newblock \emph{arXiv preprint arXiv:2311.13384}, 2023.
\bibitem[Esser et~al.(2023)Esser, Chiu, Atighehchian, Granskog, and Germanidis]{esser2023GEN1}
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.
\newblock Structure and content-guided video synthesis with diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 7346--7356, 2023.
\bibitem[Fei et~al.(2023)Fei, Wu, Ji, Zhang, and Chua]{fei2023DysenVDM}
Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, and Tat-Seng Chua.
\newblock Empowering dynamics-aware text-to-video diffusion with large language models.
\newblock \emph{arXiv preprint arXiv:2308.13812}, 2023.
\bibitem[Feng et~al.(2023)Feng, Weng, Wang, Yuan, Bao, Luo, Chen, and Guo]{feng2023ccedit}
Ruoyu Feng, Wenming Weng, Yanhui Wang, Yuhui Yuan, Jianmin Bao, Chong Luo, Zhibo Chen, and Baining Guo.
\newblock Ccedit: Creative and controllable video editing via diffusion models.
\newblock \emph{arXiv preprint arXiv:2309.16496}, 2023.
\bibitem[Ge et~al.(2023)Ge, Nah, Liu, Poon, Tao, Catanzaro, Jacobs, Huang, Liu, and Balaji]{ge2023preserve}
Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji.
\newblock Preserve your own correlation: A noise prior for video diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 22930--22941, 2023.
\bibitem[Guo et~al.(2023)Guo, Yang, Rao, Liang, Wang, Qiao, Agrawala, Lin, and Dai]{guo2023animatediff}
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu~Qiao, Maneesh Agrawala, Dahua Lin, and Bo~Dai.
\newblock Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2023.
\bibitem[Han et~al.(2024)Han, Kokkinos, and Torr]{han2024vfusion3d}
Junlin Han, Filippos Kokkinos, and Philip Torr.
\newblock Vfusion3d: Learning scalable 3d generative models from video diffusion models.
\newblock \emph{arXiv preprint arXiv:2403.12034}, 2024.
\bibitem[Hao et~al.(2018)Hao, Huang, and Belongie]{hao2018controllable}
Zekun Hao, Xun Huang, and Serge Belongie.
\newblock Controllable video generation with sparse trajectories.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, pp.\ 7854--7863, 2018.
\bibitem[He et~al.(2024)He, Xu, Guo, Wetzstein, Dai, Li, and Yang]{he2024cameractrl}
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo~Dai, Hongsheng Li, and Ceyuan Yang.
\newblock Cameractrl: Enabling camera control for text-to-video generation.
\newblock \emph{arXiv preprint arXiv:2404.02101}, 2024.
\bibitem[Heusel et~al.(2017)Heusel, Ramsauer, Unterthiner, Nessler, and Hochreiter]{heusel2017FID}
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
\newblock Gans trained by a two time-scale update rule converge to a local nash equilibrium.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Ho et~al.(2022)Ho, Chan, Saharia, Whang, Gao, Gritsenko, Kingma, Poole, Norouzi, Fleet, et~al.]{ho2022imagenvideo}
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik~P Kingma, Ben Poole, Mohammad Norouzi, David~J Fleet, et~al.
\newblock Imagen video: High definition video generation with diffusion models.
\newblock \emph{arXiv preprint arXiv:2210.02303}, 2022.
\bibitem[Hong et~al.(2022)Hong, Ding, Zheng, Liu, and Tang]{hong2022cogvideo}
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.
\newblock Cogvideo: Large-scale pretraining for text-to-video generation via transformers.
\newblock \emph{arXiv preprint arXiv:2205.15868}, 2022.
\bibitem[Hou et~al.(2024)Hou, Wei, and Chen]{hou2024high}
Chen Hou, Guoqiang Wei, and Zhibo Chen.
\newblock High-fidelity diffusion-based image editing.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~38, pp.\ 2184--2192, 2024.
\bibitem[Hu et~al.(2021)Hu, Wallis, Allen-Zhu, Li, Wang, Wang, Chen, et~al.]{hu2021lora}
Edward~J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang, Weizhu Chen, et~al.
\newblock Lora: Low-rank adaptation of large language models.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Kim et~al.(2022)Kim, Kwon, and Ye]{kim2022diffusionclip}
Gwanghyun Kim, Taesung Kwon, and Jong~Chul Ye.
\newblock Diffusionclip: Text-guided diffusion models for robust image manipulation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 2426--2435, 2022.
\bibitem[Li et~al.(2023)Li, Chu, Wu, Yuan, Liu, Zhang, Li, Feng, Ding, and Wang]{li2023videogen}
Xin Li, Wenqing Chu, Ye~Wu, Weihang Yuan, Fanglong Liu, Qi~Zhang, Fu~Li, Haocheng Feng, Errui Ding, and Jingdong Wang.
\newblock Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation.
\newblock \emph{arXiv preprint arXiv:2309.00398}, 2023.
\bibitem[Liu et~al.(2023)Liu, Zhang, Li, Lin, and Jia]{liu2023video}
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.
\newblock Video-p2p: Video editing with cross-attention control.
\newblock \emph{arXiv preprint arXiv:2303.04761}, 2023.
\bibitem[Lugmayr et~al.(2022)Lugmayr, Danelljan, Romero, Yu, Timofte, and Van~Gool]{lugmayr2022repaint}
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van~Gool.
\newblock Repaint: Inpainting using denoising diffusion probabilistic models.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pp.\ 11461--11471, 2022.
\bibitem[Luo et~al.(2023)Luo, Chen, Zhang, Huang, Wang, Shen, Zhao, Zhou, and Tan]{luo2023videofusion}
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan.
\newblock Videofusion: Decomposed diffusion models for high-quality video generation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 10209--10218, 2023.
\bibitem[Ma et~al.(2024)Ma, Zhou, Yeh, Wang, Li, Yang, Dong, Keutzer, and Feng]{ma2024magic}
Ze~Ma, Daquan Zhou, Chun-Hsiao Yeh, Xue-She Wang, Xiuyu Li, Huanrui Yang, Zhen Dong, Kurt Keutzer, and Jiashi Feng.
\newblock Magic-me: Identity-specific video customized diffusion.
\newblock \emph{arXiv preprint arXiv:2402.09368}, 2024.
\bibitem[Mao et~al.(2023)Mao, Wang, and Aizawa]{mao2023guided}
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa.
\newblock Guided image synthesis via initial image editing in diffusion model.
\newblock In \emph{Proceedings of the 31st ACM International Conference on Multimedia}, pp.\ 5321--5329, 2023.
\bibitem[Melas-Kyriazi et~al.(2024)Melas-Kyriazi, Laina, Rupprecht, Neverova, Vedaldi, Gafni, and Kokkinos]{im3d}
Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos.
\newblock Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation.
\newblock \emph{arXiv preprint arXiv:2402.08682}, 2024.
\bibitem[Meng et~al.(2021)Meng, He, Song, Song, Wu, Zhu, and Ermon]{meng2021sdedit}
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
\newblock Sdedit: Guided image synthesis and editing with stochastic differential equations.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Nichol et~al.(2022)Nichol, Dhariwal, Ramesh, Shyam, Mishkin, Mcgrew, Sutskever, and Chen]{nichol2022glide}
Alexander~Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.
\newblock Glide: Towards photorealistic image generation and editing with text-guided diffusion models.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 16784--16804. PMLR, 2022.
\bibitem[Rombach et~al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer]{rombach2022high}
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj{\"o}rn Ommer.
\newblock High-resolution image synthesis with latent diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pp.\ 10684--10695, 2022.
\bibitem[Saito et~al.(2020)Saito, Saito, Koyama, and Kobayashi]{saito2020IS}
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi.
\newblock Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan.
\newblock \emph{International Journal of Computer Vision}, 128\penalty0 (10-11):\penalty0 2586--2606, 2020.
\bibitem[Shi et~al.(2023)Shi, Wang, Ye, Long, Li, and Yang]{shi2023mvdream}
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.
\newblock Mvdream: Multi-view diffusion for 3d generation.
\newblock \emph{arXiv preprint arXiv:2308.16512}, 2023.
\bibitem[Singer et~al.(2022)Singer, Polyak, Hayes, Yin, An, Zhang, Hu, Yang, Ashual, Gafni, et~al.]{singer2022make}
Uriel Singer, Adam Polyak, Thomas Hayes, Xi~Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et~al.
\newblock Make-a-video: Text-to-video generation without text-video data.
\newblock \emph{arXiv preprint arXiv:2209.14792}, 2022.
\bibitem[Unterthiner et~al.(2018)Unterthiner, Van~Steenkiste, Kurach, Marinier, Michalski, and Gelly]{unterthiner2018FVD}
Thomas Unterthiner, Sjoerd Van~Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.
\newblock Towards accurate generative models of video: A new metric \& challenges.
\newblock \emph{arXiv preprint arXiv:1812.01717}, 2018.
\bibitem[Voleti et~al.(2024)Voleti, Yao, Boss, Letts, Pankratz, Tochilkin, Laforte, Rombach, and Jampani]{sv3d}
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani.
\newblock Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion.
\newblock \emph{arXiv preprint arXiv:2403.12008}, 2024.
\bibitem[Wang et~al.(2023)Wang, Yuan, Wang, Chen, Xia, Luo, and Shan]{wang2023motionctrl}
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan.
\newblock Motionctrl: A unified and flexible motion controller for video generation.
\newblock \emph{arXiv preprint arXiv:2312.03641}, 2023.
\bibitem[Wu et~al.(2021)Wu, Huang, Zhang, Li, Ji, Yang, Sapiro, and Duan]{wu2021CLIPSIM}
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan.
\newblock Godiva: Generating open-domain videos from natural descriptions.
\newblock \emph{arXiv preprint arXiv:2104.14806}, 2021.
\bibitem[Wu et~al.(2023)Wu, Ge, Wang, Lei, Gu, Shi, Hsu, Shan, Qie, and Shou]{tuneavideo}
Jay~Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan~Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike~Zheng Shou.
\newblock Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, pp.\ 7623--7633, October 2023.
\bibitem[Yang et~al.(2024{\natexlab{a}})Yang, Hou, Huang, Ma, Wan, Zhang, Chen, and Liao]{yang2024direct}
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di~Zhang, Xiaodong Chen, and Jing Liao.
\newblock Direct-a-video: Customized video generation with user-directed camera movement and object motion.
\newblock \emph{arXiv preprint arXiv:2402.03162}, 2024{\natexlab{a}}.
\bibitem[Yang et~al.(2024{\natexlab{b}})Yang, Teng, Zheng, Ding, Huang, Xu, Yang, Hong, Zhang, Feng, et~al.]{yang2024cogvideox}
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et~al.
\newblock Cogvideox: Text-to-video diffusion models with an expert transformer.
\newblock \emph{arXiv preprint arXiv:2408.06072}, 2024{\natexlab{b}}.
\bibitem[Yin et~al.(2023)Yin, Wu, Liang, Shi, Li, Ming, and Duan]{yin2023dragnuwa}
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan.
\newblock Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.
\newblock \emph{arXiv preprint arXiv:2308.08089}, 2023.
\bibitem[Yu et~al.(2023)Yu, Xu, Zhang, Liu, Ye, Wu, Yan, Zhu, Xiong, Liang, et~al.]{yu2023mvimgnet}
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et~al.
\newblock Mvimgnet: A large-scale dataset of multi-view images.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pp.\ 9150--9161, 2023.
\bibitem[Zeng et~al.(2024)Zeng, Wei, Zheng, Zou, Wei, Zhang, and Li]{zeng2023PixelDance}
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li.
\newblock Make pixels dance: High-dynamic video generation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2024.
\bibitem[Zhao et~al.(2022)Zhao, Liu, Guo, Wang, and Liu]{zhao2022particlesfm}
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu.
\newblock Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild.
\newblock In \emph{European Conference on Computer Vision}, pp.\ 523--542. Springer, 2022.
\bibitem[Zhou et~al.(2018)Zhou, Tucker, Flynn, Fyffe, and Snavely]{realestate10k}
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.
\newblock Stereo magnification: Learning view synthesis using multiplane images.
\newblock \emph{arXiv preprint arXiv:1805.09817}, 2018.
\end{thebibliography}