Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing
Extracted source text · arXiv 2410.12526
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
\usepackage{amsmath}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{graphicx}
\usepackage{float}
\usepackage{booktabs}
\usepackage{color}
\usepackage{placeins}
\usepackage{authblk}
%\title{Stylizing Concept Guided Videos for Text-Guided Editing with Dual Prior Supervision}
%\title{Stylizing Stabilized Text-Guided Video based on Textual Inversion }
\title{Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing}
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\iclrfinalcopy
\pagestyle{empty}
\author[1]{Mingce Guo$^{\textbf{\#}}$}
\author[2]{Jingxuan He$^{\textbf{\#}}$}
\author[2]{Shenggeng Tang}
\author[1]{Zhangye Wang}
\author[2]{Lechao Cheng$^{\textbf{*}}$}
% \author{Mingce Guo$^{\textbf{\#}, 1}$, Jingxuan He$^{\textbf{\#}, 2}$, Shenggeng Tang$^2$, Zhangye Wang$^1$, Lechao Cheng$^2$\thanks{Corresponding author} \\
% %\overset{\text{1}}{Zhejiang University}\\
% Zhejiang University\\
% Hefei University of Technology \\
% \And
% Jingxuan He \\
% %Computer Science and Information Engineering \\
% Hefei University of Technology \\
% \AND
% Shenggeng Tang \\
% Hefei University of Technology \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email}
%}
\affil[1]{Zhejiang University}
\affil[2]{Hefei University of Technology}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
%\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
%% author list
%% Mingce Guo, Jingxuan He, Lechao Cheng, Shenggeng Tang, Zhangye Wang, Chunhua Shen ???
\maketitle
\footnotetext{\textbf{*}Corresponding author}
%\input{sections/0.abstract}
\begin{abstract}
Text-driven video editing utilizing generative diffusion models has garnered significant attention due to their potential applications. However, existing approaches are constrained by the limited word embeddings provided in pre-training, which hinders nuanced editing targeting open concepts with specific attributes. Directly altering the keywords in target prompts often results in unintended disruptions to the attention mechanisms. To achieve more flexible editing easily, this work proposes an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs.
Specifically, the framework involves concept-augmented textual inversion and a dual prior supervision mechanism. The former enables plug-and-play guidance of stable diffusion for video editing, effectively capturing target attributes for more stylized results. The dual prior supervision mechanism significantly enhances video stability and fidelity. Comprehensive evaluations demonstrate that our approach generates more stable and lifelike videos, outperforming state-of-the-art methods.
% The anonymous code is available at \href{https://anonymous.4open.science/w/STIVE-PAGE-B4D4/}{https://anonymous.4open.science/w/STIVE-PAGE-B4D4/}.
\end{abstract}
\input{sections/1.introduction}
\input{sections/2.related_works}
\input{sections/3.method}
\input{sections/4.experiments}
\input{sections/5.limitations}
\input{sections/6.conclusion}
% \section{Submission of conference papers to ICLR 2025}
% ICLR requires electronic submissions, processed by
% \url{https://openreview.net/}. See ICLR's website for more instructions.
% If your paper is ultimately accepted, the statement {\tt
% {\textbackslash}iclrfinalcopy} should be inserted to adjust the
% format to the camera ready requirements.
% The format for the submissions is a variant of the NeurIPS format.
% Please read carefully the instructions below, and follow them
% faithfully.
% \subsection{Style}
% Papers to be submitted to ICLR 2025 must be prepared according to the
% instructions presented here.
% %% Please note that we have introduced automatic line number generation
% %% into the style file for \LaTeXe. This is to help reviewers
% %% refer to specific lines of the paper when they make their comments. Please do
% %% NOT refer to these line numbers in your paper as they will be removed from the
% %% style file for the final version of accepted papers.
% Authors are required to use the ICLR \LaTeX{} style files obtainable at the
% ICLR website. Please make sure you use the current files and
% not previous versions. Tweaking the style files may be grounds for rejection.
% \subsection{Retrieval of style files}
% The style files for ICLR and other conference information are available online at:
% \begin{center}
% \url{http://www.iclr.cc/}
% \end{center}
% The file \verb+iclr2025_conference.pdf+ contains these
% instructions and illustrates the
% various formatting requirements your ICLR paper must satisfy.
% Submissions must be made using \LaTeX{} and the style files
% \verb+iclr2025_conference.sty+ and \verb+iclr2025_conference.bst+ (to be used with \LaTeX{}2e). The file
% \verb+iclr2025_conference.tex+ may be used as a ``shell'' for writing your paper. All you
% have to do is replace the author, title, abstract, and text of the paper with
% your own.
% The formatting instructions contained in these style files are summarized in
% sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.
% \section{General formatting instructions}
% \label{gen_inst}
% The text must be confined within a rectangle 5.5~inches (33~picas) wide and
% 9~inches (54~picas) long. The left margin is 1.5~inch (9~picas).
% Use 10~point type with a vertical spacing of 11~points. Times New Roman is the
% preferred typeface throughout. Paragraphs are separated by 1/2~line space,
% with no indentation.
% Paper title is 17~point, in small caps and left-aligned.
% All pages should start at 1~inch (6~picas) from the top of the page.
% Authors' names are
% set in boldface, and each name is placed above its corresponding
% address. The lead author's name is to be listed first, and
% the co-authors' names are set to follow. Authors sharing the
% same address can be on the same line.
% Please pay special attention to the instructions in section \ref{others}
% regarding figures, tables, acknowledgments, and references.
% There will be a strict upper limit of 10 pages for the main text of the initial submission, with unlimited additional pages for citations.
% \section{Headings: first level}
% \label{headings}
% First level headings are in small caps,
% flush left and in point size 12. One line space before the first level
% heading and 1/2~line space after the first level heading.
% \subsection{Headings: second level}
% Second level headings are in small caps,
% flush left and in point size 10. One line space before the second level
% heading and 1/2~line space after the second level heading.
% \subsubsection{Headings: third level}
% Third level headings are in small caps,
% flush left and in point size 10. One line space before the third level
% heading and 1/2~line space after the third level heading.
% \section{Citations, figures, tables, references}
% \label{others}
% These instructions apply to everyone, regardless of the formatter being used.
% \subsection{Citations within the text}
% Citations within the text should be based on the \texttt{natbib} package
% and include the authors' last names and year (with the ``et~al.'' construct
% for more than two authors). When the authors or the publication are
% included in the sentence, the citation should not be in parenthesis using \verb|\citet{}| (as
% in ``See \citet{} for more information.''). Otherwise, the citation
% should be in parenthesis using \verb|\citep{}| (as in ``Deep learning shows promise to make progress
% towards AI~\citep{ho2020denoising}.'').
% The corresponding references are to be listed in alphabetical order of
% authors, in the \textsc{References} section. As to the format of the
% references themselves, any style is acceptable as long as it is used
% consistently.
% \subsection{Footnotes}
% Indicate footnotes with a number\footnote{Sample of the first footnote} in the
% text. Place the footnotes at the bottom of the page on which they appear.
% Precede the footnote with a horizontal rule of 2~inches
% (12~picas).\footnote{Sample of the second footnote}
% \subsection{Figures}
% All artwork must be neat, clean, and legible. Lines should be dark
% enough for purposes of reproduction; art work should not be
% hand-drawn. The figure number and caption always appear after the
% figure. Place one line space before the figure caption, and one line
% space after the figure. The figure caption is lower case (except for
% first word and proper nouns); figures are numbered consecutively.
% Make sure the figure caption does not get separated from the figure.
% Leave sufficient space to avoid splitting the figure and figure caption.
% You may use color figures.
% However, it is best for the
% figure captions and the paper body to make sense if the paper is printed
% either in black/white or in color.
% \begin{figure}[h]
% \begin{center}
% %\framebox[4.0in]{$\;$}
% \fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
% \end{center}
% \caption{Sample figure caption.}
% \end{figure}
% \subsection{Tables}
% All tables must be centered, neat, clean and legible. Do not use hand-drawn
% tables. The table number and title always appear before the table. See
% Table~\ref{sample-table}.
% Place one line space before the table title, one line space after the table
% title, and one line space after the table. The table title must be lower case
% (except for first word and proper nouns); tables are numbered consecutively.
% \begin{table}[t]
% \caption{Sample table title}
% \label{sample-table}
% \begin{center}
% \begin{tabular}{ll}
% \multicolumn{1}{c}{\bf PART} &\multicolumn{1}{c}{\bf DESCRIPTION}
% \\ \hline \\
% Dendrite &Input terminal \\
% Axon &Output terminal \\
% Soma &Cell body (contains cell nucleus) \\
% \end{tabular}
% \end{center}
% \end{table}
% \section{Default Notation}
% In an attempt to encourage standardized notation, we have included the
% notation file from the textbook, \textit{Deep Learning}
% \cite{goodfellow2016deep} available at
% \url{https://github.com/goodfeli/dlbook_notation/}. Use of this style
% is not required and can be disabled by commenting out
% \texttt{math\_commands.tex}.
% \centerline{\bf Numbers and Arrays}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1in}p{3.25in}}
% $\displaystyle a$ & A scalar (integer or real)\\
% $\displaystyle \va$ & A vector\\
% $\displaystyle \mA$ & A matrix\\
% $\displaystyle \tA$ & A tensor\\
% $\displaystyle \mI_n$ & Identity matrix with $n$ rows and $n$ columns\\
% $\displaystyle \mI$ & Identity matrix with dimensionality implied by context\\
% $\displaystyle \ve^{(i)}$ & Standard basis vector $[0,\dots,0,1,0,\dots,0]$ with a 1 at position $i$\\
% $\displaystyle \text{diag}(\va)$ & A square, diagonal matrix with diagonal entries given by $\va$\\
% $\displaystyle \ra$ & A scalar random variable\\
% $\displaystyle \rva$ & A vector-valued random variable\\
% $\displaystyle \rmA$ & A matrix-valued random variable\\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Sets and Graphs}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle \sA$ & A set\\
% $\displaystyle \R$ & The set of real numbers \\
% $\displaystyle \{0, 1\}$ & The set containing 0 and 1 \\
% $\displaystyle \{0, 1, \dots, n \}$ & The set of all integers between $0$ and $n$\\
% $\displaystyle [a, b]$ & The real interval including $a$ and $b$\\
% $\displaystyle (a, b]$ & The real interval excluding $a$ but including $b$\\
% $\displaystyle \sA \backslash \sB$ & Set subtraction, i.e., the set containing the elements of $\sA$ that are not in $\sB$\\
% $\displaystyle \gG$ & A graph\\
% $\displaystyle \parents_\gG(\ervx_i)$ & The parents of $\ervx_i$ in $\gG$
% \end{tabular}
% \vspace{0.25cm}
% \centerline{\bf Indexing}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle \eva_i$ & Element $i$ of vector $\va$, with indexing starting at 1 \\
% $\displaystyle \eva_{-i}$ & All elements of vector $\va$ except for element $i$ \\
% $\displaystyle \emA_{i,j}$ & Element $i, j$ of matrix $\mA$ \\
% $\displaystyle \mA_{i, :}$ & Row $i$ of matrix $\mA$ \\
% $\displaystyle \mA_{:, i}$ & Column $i$ of matrix $\mA$ \\
% $\displaystyle \etA_{i, j, k}$ & Element $(i, j, k)$ of a 3-D tensor $\tA$\\
% $\displaystyle \tA_{:, :, i}$ & 2-D slice of a 3-D tensor\\
% $\displaystyle \erva_i$ & Element $i$ of the random vector $\rva$ \\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Calculus}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% % NOTE: the [2ex] on the next line adds extra height to that row of the table.
% % Without that command, the fraction on the first line is too tall and collides
% % with the fraction on the second line.
% $\displaystyle\frac{d y} {d x}$ & Derivative of $y$ with respect to $x$\\ [2ex]
% $\displaystyle \frac{\partial y} {\partial x} $ & Partial derivative of $y$ with respect to $x$ \\
% $\displaystyle \nabla_\vx y $ & Gradient of $y$ with respect to $\vx$ \\
% $\displaystyle \nabla_\mX y $ & Matrix derivatives of $y$ with respect to $\mX$ \\
% $\displaystyle \nabla_\tX y $ & Tensor containing derivatives of $y$ with respect to $\tX$ \\
% $\displaystyle \frac{\partial f}{\partial \vx} $ & Jacobian matrix $\mJ \in \R^{m\times n}$ of $f: \R^n \rightarrow \R^m$\\
% $\displaystyle \nabla_\vx^2 f(\vx)\text{ or }\mH( f)(\vx)$ & The Hessian matrix of $f$ at input point $\vx$\\
% $\displaystyle \int f(\vx) d\vx $ & Definite integral over the entire domain of $\vx$ \\
% $\displaystyle \int_\sS f(\vx) d\vx$ & Definite integral with respect to $\vx$ over the set $\sS$ \\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Probability and Information Theory}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle P(\ra)$ & A probability distribution over a discrete variable\\
% $\displaystyle p(\ra)$ & A probability distribution over a continuous variable, or over
% a variable whose type has not been specified\\
% $\displaystyle \ra \sim P$ & Random variable $\ra$ has distribution $P$\\% so thing on left of \sim should always be a random variable, with name beginning with \r
% $\displaystyle \E_{\rx\sim P} [ f(x) ]\text{ or } \E f(x)$ & Expectation of $f(x)$ with respect to $P(\rx)$ \\
% $\displaystyle \Var(f(x)) $ & Variance of $f(x)$ under $P(\rx)$ \\
% $\displaystyle \Cov(f(x),g(x)) $ & Covariance of $f(x)$ and $g(x)$ under $P(\rx)$\\
% $\displaystyle H(\rx) $ & Shannon entropy of the random variable $\rx$\\
% $\displaystyle \KL ( P \Vert Q ) $ & Kullback-Leibler divergence of P and Q \\
% $\displaystyle \mathcal{N} ( \vx ; \vmu , \mSigma)$ & Gaussian distribution %
% over $\vx$ with mean $\vmu$ and covariance $\mSigma$ \\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \centerline{\bf Functions}
% \bgroup
% \def\arraystretch{1.5}
% \begin{tabular}{p{1.25in}p{3.25in}}
% $\displaystyle f: \sA \rightarrow \sB$ & The function $f$ with domain $\sA$ and range $\sB$\\
% $\displaystyle f \circ g $ & Composition of the functions $f$ and $g$ \\
% $\displaystyle f(\vx ; \vtheta) $ & A function of $\vx$ parametrized by $\vtheta$.
% (Sometimes we write $f(\vx)$ and omit the argument $\vtheta$ to lighten notation) \\
% $\displaystyle \log x$ & Natural logarithm of $x$ \\
% $\displaystyle \sigma(x)$ & Logistic sigmoid, $\displaystyle \frac{1} {1 + \exp(-x)}$ \\
% $\displaystyle \zeta(x)$ & Softplus, $\log(1 + \exp(x))$ \\
% $\displaystyle || \vx ||_p $ & $\normlp$ norm of $\vx$ \\
% $\displaystyle || \vx || $ & $\normltwo$ norm of $\vx$ \\
% $\displaystyle x^+$ & Positive part of $x$, i.e., $\max(0,x)$\\
% $\displaystyle \1_\mathrm{condition}$ & is 1 if the condition is true, 0 otherwise\\
% \end{tabular}
% \egroup
% \vspace{0.25cm}
% \section{Final instructions}
% Do not change any aspects of the formatting parameters in the style files.
% In particular, do not modify the width or length of the rectangle the text
% should fit into, and do not change font sizes (except perhaps in the
% \textsc{References} section; see below). Please note that pages should be
% numbered.
% \section{Preparing PostScript or PDF files}
% Please prepare PostScript or PDF files with paper size ``US Letter'', and
% not, for example, ``A4''. The -t
% letter option on dvips will produce US Letter files.
% Consider directly generating PDF files using \verb+pdflatex+
% (especially if you are a MiKTeX user).
% PDF figures must be substituted for EPS figures, however.
% Otherwise, please generate your PostScript and PDF files with the following commands:
% \begin{verbatim}
% dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps
% ps2pdf mypaper.ps mypaper.pdf
% \end{verbatim}
% \subsection{Margins in LaTeX}
% Most of the margin problems come from figures positioned by hand using
% \verb+\special+ or other commands. We suggest using the command
% \verb+\includegraphics+
% from the graphicx package. Always specify the figure width as a multiple of
% the line width as in the example below using .eps graphics
% \begin{verbatim}
% \usepackage[dvips]{graphicx} ...
% \includegraphics[width=0.8\linewidth]{myfile.eps}
% \end{verbatim}
% or % Apr 2009 addition
% \begin{verbatim}
% \usepackage[pdftex]{graphicx} ...
% \includegraphics[width=0.8\linewidth]{myfile.pdf}
% \end{verbatim}
% for .pdf graphics.
% See section~4.4 in the graphics bundle documentation (\url{http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps})
% A number of width problems arise when LaTeX cannot properly hyphenate a
% line. Please give LaTeX hyphenation hints using the \verb+\-+ command.
% \subsubsection*{Author Contributions}
% If you'd like to, you may include a section for author contributions as is done
% in many journals. This is optional and at the discretion of the authors.
% \subsubsection*{Acknowledgments}
% Use unnumbered third level headings for the acknowledgments. All
% acknowledgments, including those to funding agencies, go at the end of the paper.
%\clearpage
% \section{Ethics Statement}
% This research adheres to ethical guidelines and standards. No human subjects were involved, and the dataset used was publicly available and anonymized to ensure privacy. The methodologies employed were carefully chosen to avoid introducing bias or unfair outcomes. There are no conflicts of interest or sponsorships influencing the findings. All legal and institutional regulations, including IRB approval where necessary, were strictly followed.
% \section{Reproducibility Statement}
% This work provides a clear link to the anonymized code: \href{https://anonymous.4open.science/r/STIVE-79D5/README.md}{https://anonymous.4open.science/r/STIVE-79D5/README.md}. The details of the data used in the experiments have been clearly outlined in the main text, and additional results in the Appendix demonstrate convincing superiority.
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
\newpage
\appendix
\input{sections/7.appendix}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\section{introduction}
Text-driven video editing using generative diffusion models~\citep{ho2020denoising, song2020denoising, rombach2021highresolution} has garnered significant attention due to its potential applications in various fields, including film production, art, and advertising~\citep{ho2022imagen, hong2022cogvideo, blattmann2023stable}.
Existing text-driven video editing methods based on diffusion models, such as Tune-A-Video~\citep{wu2023tune}, FateZero~\citep{qi2023fatezero}, Zeroscope~\citep{zeroscope}, and VideoComposer~\citep{wang2024videocomposer}, have significantly improved the ability to edit objects, backgrounds, and styles in video scenes while maintaining overall scene consistency through the optimization of attention mechanisms and spatiotemporal continuity. These approaches have demonstrated notable success in video generation. However, they are often limited by the restricted word embeddings provided by CLIP~\citep{radford2021learning} during the text-driven encoding process, which restricts their ability to perform diverse and nuanced edits on targets with specific attributes. Modifying words in the target prompt can disrupt the attention mechanisms, leading to inconsistencies in non-target areas before and after the editing process.
To address these limitations, recent methods such as~\citep{bar2022text2live, lee2023shape, chai2023stablevideo} have explored the use of Neural Layered Atlases (NLA)~\citep{kasten2021layered}. These methods primarily focus on extracting layered atlases from video frames, editing these atlases using text-driven image-editing diffusion models~\citep{rombach2021highresolution,zhang2023adding}, and then synthesizing the final edited video through post-processing. While this approach is effective at preserving non-edited background areas, it exhibits poor performance in maintaining spatio-temporal continuity. Moreover, the processing of individual image frames makes the generation of neural atlases extremely time-consuming.
To achieve more diverse editing results easily, one feasible approach is to draw inspiration from the Textual Inversion~\citep{gal2022textual} method used in image generation by incorporating external concept word embeddings. In text-to-image generation, external word embeddings (referred to as "concepts") are optimized within the CLIP text encoder~\citep{radford2021learning} while freezing the diffusion model’s parameters. This allows the model to address the need for user-provided custom images to guide image editing. However, using a pre-trained diffusion model for self-supervised textual inversion presents limitations in the expressive power of these external embeddings, which are constrained by the size of the latent space and the number of training iterations, often leading to under-fitting and restricted expressive capacity.
In this paper, we propose an improved concept-augmented video editing method. This approach flexibly generates diverse and stable target videos by defining abstract conceptual pairs (concept prompt and concept video) that describe the target scene. Specifically, we propose the \textbf{Concept-Augmented Textual Inversion} method, which reliably and accurately captures the target attributes in the user's custom video. In addition, we also introduce a \textbf{Dual Prior Supervision} mechanism that stabilizes the generated video by crossing the attention between the source and target, preventing attention dispersion caused by modifications to the target prompt. This mechanism effectively improves the consistency of non-target areas before and after video editing, while also enriching the fidelity of the concepts in the edited results. Our key contributions are as follows:
\begin{itemize}
\item We orchestrate a framework that allows users to extract concepts from custom videos to efficiently generate diverse edited videos through concept templates.
\item We propose a concept-augmented textual inversion method, which efficiently and stably extracts detailed attributes of the target in the user's custom concept video and supports plug-and-play guiding of stable diffusion for video editing.
\item We present a dual prior supervision mechanism, which effectively improves the consistency and stability of video editing results.
\end{itemize}
% Recent methods \citep{wu2023tune, kara2024rave} primarily rely on diffusion models enhanced with spatial-temporal or conditional networks. These models facilitate the denoising process of the source video based on user-provided target text, allowing for edits in shape, background, and style. For example, \citep{qi2023fatezero} builds upon a temporally extended stable diffusion model and proposes attention blending and fusing algorithms to better support user prompts for stable video editing. However, altering words in the target prompt can disrupt the cross-attention mechanism, resulting in inconsistent outcomes in non-target regions before and after editing. These approaches support text-driven video editing with limited expressive capability, which hinders its application in many scenarios. The aforementioned algorithms primarily support text-driven video editing but have limited expressive capabilities. Recent work proposed by \citet{zhao2023motiondirector} employs a pre-trained text-to-video generation model, allowing users to input custom images to fine-tune the embedded spatial attention modules, enabling target-specific editing in the source video. However, during the diffusion process, the model lacks practical guidance from the latent variables of the source video, and fine-tuning the spatial attention parameters carries the risk of overfitting, which can lead to instability in the editing results.
% In the context of text-driven image editing tasks, \citet{gal2022textual} addresses the need for user-provided custom images to guide image editing by freezing the diffusion model parameters and learning the user's custom image representation through additional external word embeddings, termed \textbf{\textit{Concept}}. These concepts are used to enhance image editing or style transfer. However, the expressive capacity of these external embeddings is constrained by the size of the latent space and the number of training iterations, potentially leading to underfitting and limited expressive power.
% To overcome these limitations, we draw inspiration from \citet{gal2022textual} and propose a \textbf{Concept-Augmented Textual Inversion} method, which reliably and accurately captures the attributes of the target in the user's custom input video. Furthermore, we introduce a \textbf{Dual Prior Supervision} strategy, which stabilizes the generated video by crossing source and target attention to avoid dispersion caused by altering words in the target prompt. This mechanism effectively improves the consistency of non-target regions before and after video editing while enhancing the fidelity of the concept in the edited result. Our contributions are as follows:
%To address challenges such as cross-attention dispersion caused by modifying or adding words in the target prompt, as well as noise introduced during concept optimization from custom concept videos,
% Text-driven video editing based on generative diffusion models~\citep{ho2020denoising, song2020denoising, rombach2021highresolution} has been receiving increasing attention due to its potential value in a wide range of applications, such as film production, art and advertising industries \citep{ho2022imagen, hong2022cogvideo, blattmann2023stable}.
% Recent methods \citep{wu2023tune, kara2024rave}, primarily rely on diffusion models extended with spatial-temporal or conditional networks. These models participate in the denoising process of the source video according to user-input target text, enabling the editing of shape, background, and style. \citep{qi2023fatezero} building upon a temporally extended Stable Diffusion model, proposes Attention Blending and Fusing algorithms to support user-provided target prompts for more stable video editing. However, the replacement or addition of words in the target prompt can cause dispersion in the cross-attention mechanism, leading to weak consistency in non-target regions before and after editing.
% Aforementioned algorithms only support text-driven video editing with limited expressive capability. \citep{zhao2023motiondirector} adopts a pre-trained text-to-video generation model and allows users to input custom images to fine-tune embedded spatial attention modules, achieving target editing in the source video. However, during the diffusion process, it lacks effective guidance from the latent variables of the source video, and fine-tuning the spatial attention parameters carries a risk of over-fitting, resulting in instability in the editing results.
% Moreover, in text-driven image editing tasks, to support user-input custom images for guiding image editing, \citep{gal2022textual} freezes the parameters of the diffusion model and learns the user's custom image representation by training additional external word embeddings, called concept, which are used to enhance image editing or style transfer. However, the expressive capability of these external word embeddings is related to the size of the latent space and the number of training iterations; the concepts may suffer from under-fitting and inherently limited expressive power issues.
% In consideration of these constraints in existing methodologies, we draw inspiration from \citep{gal2022textual} and propose a Concept-Augmented Textual Inversion method, which can reliably and meticulously capture the attributes of the target in the user's custom input video. To address the issues of cross-attention dispersion caused by replacing or adding words in the target prompt, as well as the noise introduction problem during concept optimization from custom concept videos, we propose a source and target cross-attention supervision mechanism. This mechanism effectively improves the consistency of non-target regions before and after video editing and enhances the fidelity of the concept in edited result.
% \begin{itemize}
% \item We propose a concept-augmented textual inversion method, which efficiently and stably extracts detailed attributes of the target in the user's custom concept video and supports plug-and-play guiding of Stable Diffusion for shape editing.
% \item We present dual prior supervision mechanisms, which effectively improve the consistency and stability of video editing results.
% \item We orchestrate a framework that allows users to extract concepts from custom videos, utilizing a pre-trained OWL-ViT model \citep{minderer2022simple} to generate pseudo-labels for attention supervision. Additionally, the framework supports video editing guidance through integrated attention control mechanisms.
% \end{itemize}
% \textbf{(i)} We propose a Concept-Augmented Textual Inversion method, which efficiently and stably extracts detailed attributes of the target in the user's custom concept video, and supports plug-and-play guiding of Stable Diffusion for shape editing.
% \textbf{(ii)} We present dual prior supervision mechanisms, which effectively improve the consistency and stability of video editing results.
% \textbf{(iii)} We release a framework that allows users to extract concepts from custom videos, uses a pre-trained OWL-ViT \citep{minderer2022simple} to generate pseudo-labels for attention supervision, and supports guiding video editing in conjunction with attention control mechanisms.
\section{related work}
\textbf{Text-Driven Video Synthesis.}
A series of works based on diffusion models~\citep{ho2020denoising, song2020denoising, rombach2021highresolution} has made significant progress in text-driven image generation.
Subsequent efforts~\citep{esser2023structure, wang2023modelscope, zeroscope, blattmann2023stable} further achieve text-driven video generation by extending existing image generation diffusion models.
These works commonly inherit the spatial parameters of UNet and fine-tune the newly added temporal modules with large-scale video-text pair datasets to improve inter-frame stability for video synthesis.
These works laid a good foundation for video editing with textual descriptions.
\textbf{Text-Driven Video Editing.}
Current approaches for text-driven video editing mainly fall into three categories: fine-tuning video generation models~\citep{zhao2023motiondirector, wang2024videocomposer}, fine-tuning image generation models extended with temporal modules~\citep{wu2023tune, qi2023fatezero}, and combining NLA~\citep{kasten2021layered} with pre-trained image generation models~\citep{bar2022text2live, lee2023shape, chai2023stablevideo}.
MotionDirector~\citep{zhao2023motiondirector} improves the ability of editing camera and object motions by adding LoRA~\citep{hu2022lora} to the attention modules of the pre-trained Zeroscope~\citep{zeroscope}, strengthening the connection between texts and motions in edited videos.
VideoComposer~\citep{wang2024videocomposer} enhances inter-frame consistency by introducing a condition fusion module with spatial and temporal conditions such as motion vectors, depth maps, and sketches.
% enhances the stability of objects and actions by incorporating conditional fusion networks for motion vectors, contours, and depth.
\citep{bar2022text2live, lee2023shape, chai2023stablevideo} extract layered neural atlases from video frames to edit atlases which are further processed to synthesize videos; however, generating a neural atlas demands considerable computational time.
% NLA~\citep{kasten2021layered} method,
Recently, Tune-A-Video~\citep{wu2023tune} achieves one-shot video editing with improved inter-frame coherency by updating self-attention with sparse causal attention.
% builds upon stable-diffusion-v1-4, replacing self-attention with inter-frame sparse causal attention and adding temporal attention mechanisms to enhance the coherence of generated video frames.
FateZero~\citep{qi2023fatezero} further proposes self-attention blending and incorporates attention control~\citep{hertz2023prompttoprompt} to enhance the ability of editing objects, background, and styles while maintaining scene consistency.
% FateZero~\citep{qi2023fatezero} further adds temporal convolution layers, proposes self-attention blending, and incorporates attention control methods~\citep{hertz2023prompttoprompt} to enhance the ability to edit scene objects, backgrounds, and styles while maintaining scene consistency.
However, these methods are constrained by the finite word embeddings of CLIP~\citep{radford2021learning}.
For editing targets with specific attributes, it becomes necessary to introduce external word embeddings.
Our method supports the incorporation of external concept word embeddings. Furthermore, inspired by Tune-A-Video~\citep{wu2023tune} and FateZero~\citep{qi2023fatezero}, we introduce a dual prior supervision mechanism between video frame latents and word embeddings to enhance scene consistency before and after video editing based on the prompt-to-prompt attention control method.
\textbf{Textual Inversion.}
\citep{gal2022textual} proposes a textual inversion method that optimizes newly added concept word embeddings in the CLIP~\citep{radford2021learning} text encoder, supervised by the latent variable distribution of specific images in the diffusion model. However, using a pre-trained diffusion model for self-supervised text inversion may lead to under-fitting for some specific images due to the finite latent variable space. Although it's feasible to optimize concept word embeddings with a smaller learning rate simultaneously, or to train the diffusion model with frozen concept embeddings in the next stage, this process faces issues of easy over-fitting and high storage costs. Our method, building upon textual inversion~\citep{gal2022textual}, attempts to add LoRA~\citep{hu2022lora} modules to the diffusion model, optimizing them simultaneously with concept words at a smaller learning rate, to enhance the text editing capabilities of concept words.
\textbf{Cross Attention Control and Supervision.}
Prompt-to-Prompt~\citep{hertz2023prompttoprompt} proposes three attention control methods for stable text-driven image editing based on diffusion models: word swap, refinement, and reweighting. By applying the cross-attention probability map recorded from the original text and original image latent variables to the denoising process of edited text and original image latent variables, it has achieved significant success in stable text-driven image editing~\citep{Avrahami_2022_CVPR, avrahami2023blendedlatent}. Additionally,~\citep{qi2023fatezero} proposed self-attention blend effectively transfers the stability of text-driven image editing to the video editing domain. Our method, building upon this foundation, introduces external concept words to support editing with higher degrees of freedom. However, when performing text-driven editing, whether using existing word embeddings or external concept word embeddings as editing words, there exists a problem of attention dispersion. This means that editing words have non-negligible effects on latent variables other than the editing target. Inspired by the work of~\citep{yang2022supervised}, we introduce an attention supervision mechanism to address the issue of dispersed attention in editing words.
\section{Method}
\subsection{Preliminaries}
\textbf{Textual Inversion.}
% Clarification:
% You should give the definition of textual inversion at the beginning of this paragraph
% A Latent Diffusion Model typically comprises an autoencoder and a denoising network, and perhaps a conditioning encoding module/network
% autoencoder != encoder
% For diffusion models, the noise adding process is usually called 'forward/diffusion' process, and the noise removing process is usually called 'backward/denoising' process. Please pay attention to the use of 'diffusion process'.
Textual inversion~\citep{gal2022textual} learns new embeddings that represent user-provided visual concepts within the textual embedding space.
These learned embeddings are then associated with pseudo-words that can be incorporated into novel sentences to achieve text-to-vision editing.
The learning process of textual inversion relies on a latent variable diffusion model, which typically consists of an autoencoder and a noise prediction network.
For an image $x$, the autoencoder is pretrained such that the encoder $\mathcal{E}$ maps the image to a latent variable $z = \mathcal{E}(x)$, and the decoder $\mathcal{D}$ reconstructs the original image from the latent variable $x \approx \mathcal{D}(z)$.
Particularly, textual inversion leverages a CLIP~\citep{radford2021learning} text encoder $c_\theta$ with additional concept words to encode conditional text input $y$.
The optimization objective is defined as:
\begin{equation}
\label{eq:noise_pred_loss}
\mathcal{L}_{noise}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0, 1),t}[\|\epsilon-\epsilon_\theta(z_t, t, c_\theta(y))\|_2^2],
\end{equation}
where $z_t$ is the noised latent at time step $t$, $\epsilon$ is the noise, and $\epsilon_\theta$ is the noise prediction network.
% For a specific image $x$, a variational autoencoder $\mathcal{E}$ is used to map it to a spatial latent variable $z = \mathcal{E}(x)$. The latent variable diffusion model includes a noise prediction network $\epsilon_\theta$, and a CLIP~\citep{radford2021learning} text encoder $c_\theta$ with additional concept words used for encoding text conditional input $y$. Optimization goal at $t$ time step in diffusion process can then be defined as:
% \begin{equation}
% \mathcal{L}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0, 1),t}[\|\epsilon-\epsilon_\theta(z_t, t, c_\theta(y))\|_2^2].
% \end{equation}
\textbf{Low-Rank Adaption.}
\citep{hu2022lora} proposes an efficient fine-tuning scheme based on matrix low-rank decomposition. For the pre-trained weight $\mW_0 \in \mathbb{R}^{d\times k}$ in the original model, it applies low-rank decomposition to update the weight as $\mW = \mW_0 + \Delta \mW$, where $\Delta \mW = \mB\mA$, $\mB \in \mathbb{R}^{d\times r}$, $\mA \in \mathbb{R}^{r\times k}$, and $r \ll min(d, k)$. During the fine-tuning process, the pre-trained weight $\mW_0$ is frozen, while $\mA$ and $\mB$ are trainable parameters. For the forward computation of the original weight $\vh = \mW_0 \vx$, the updated forward computation becomes:
\begin{equation}
\label{eq:lora}
LoRA(\vh) = \mW_0 \vx + \Delta \mW \vx.
\end{equation}
\textbf{Video Diffusion Models with Temporal Extensions.}
Tune-A-Video~\citep{wu2023tune} introduces Spatio-Temporal Attention (ST-Attn) to replace the original Self-Attention~\citep{vaswani2017attention} in the 2D UNet.
When calculating the keys $\mK$ and values $\mV$, ST-Attn concatenates latent variables of the first and former frames of the video, leading to the attention result where the current frame attends to both the first and former frames.
% thereby outputting the attention result of the current frame in relation to both the first and former frames.
% The specific operations for replacing $\mK, \mV$ in Self-Attention mechanism $Attention(\mQ, \mK, \mV)$ are as follows:
The specific operations for replacing $\mK, \mV$ in Self-Attention are as follows:
\begin{equation}
\label{eq:kv}
\mK = \mW^K [\vz_{v_1}; \vz_{v_{i - 1}}], \mV = \mW^V [\vz_{v_1}; \vz_{v_{i - 1}}],
\end{equation}
where $W^K$ and $W^V$ are projection matrices for key and value respectively, $z_{v_i}$ denotes the latent variable of the $i$-th frame of the video to the current attention layer, and $[\cdot]$ denotes concatenation.
%
% NOTE: I rearrange this part to the overview of our framework.
%
% While inheriting the pre-trained 2D UNet network parameters from Stable-Diffusion, we add LoRA~\citep{houlsby2019parameter, hu2022lora, yu2023visual} structure temporal convolution layers after the spatial convolution layers, and temporal attention modules with a zero-initialized linear output layer after the cross-attention modules.The outputs of these newly added modules are then residually connected with the outputs of the original modules.
% These extensions result in a temporally extended 3D UNet~\citep{qi2023fatezero} noise prediction network $\epsilon_{\theta_T}$ for video latents. Optimization goal will be updated to:
% \begin{equation}
% \mathcal{L}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0, 1),t}[\|\epsilon-\epsilon_{\theta_T}(z_t, t, c_\theta(y))\|_2^2].
% \end{equation}
% \textbf{Attention Control.}
% In the diffusion process of the source video and source prompt pair, works like Prompt-to-Prompt~\citep{hertz2023prompttoprompt} and FateZero~\citep{qi2023fatezero} save the set of self and cross attention probability matrices $\sM_{src}$. By employing algorithms such as Word Swap, Phase Adding, and Attention Blending to manipulate the attention probability matrix set $\sM_{tgt}$ during the diffusion process of the source video and target prompt, they achieve stable image and video editing.
\subsection{Stabilized Text-Driven Video Editing}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figures/framework.pdf}
\vspace{-5mm}
\caption{\textbf{Overview of our training and inference pipelines.} During the training stage, we first adapt the diffusion model to new visual concepts using our introduced Concept-Augmented Textual Inversion (CATI), and then we tune the temporally extended diffusion model with our proposed Dual Prior Supervision (DPS) mechanism to prevent unintended changes in edited videos. During the inference stage, we blend self-attention matrices to retain semantic layout (Self-Attention Blending) and swap cross-attention matrices to achieve text-driven video editing (Cross-Attention Swap). }
%\vspace{-3mm}
\label{fig:overview}
\end{figure}
Our training and inference pipelines are visualized in Fig.~\ref{fig:overview}.
We adopt a UNet that inherits the pre-trained 2D UNet parameters from Stable Diffusion~\citep{rombach2021highresolution} as the noise prediction network.
The original 2D UNet consists of a series of spatial convolution layers and cross-attention layers.
To adapt it for 3D video inputs, we replace the original spatial self-attention layers with spatio-temporal attention as explained in Eq.~(\ref{eq:kv}).
Following FateZero~\citep{qi2023fatezero}, we also incorporate LoRA-structured temporal convolution layers after the spatial convolution layers, and temporal self-attention layers with zero-initialized linear output layers after the cross-attention layers.
The outputs of these newly added modules are residually connected to the outputs of the original modules.
% While inheriting the pre-trained 2D UNet network parameters from Stable-Diffusion, we add LoRA~\citep{houlsby2019parameter, hu2022lora, yu2023visual} structure temporal convolution layers after the spatial convolution layers, and temporal attention modules with a zero-initialized linear output layer after the cross-attention modules.The outputs of these newly added modules are then residually connected with the outputs of the original modules.
Our approach for stabilized text-driven video editing has two learning phases.
In the first phase, we introduce Concept-Augmented Textual Inversion (CATI) to adapt the diffusion model to new visual concepts.
In the second phase, we tune partial parameters of the temporally extended diffusion model to suppress unintended changes in edited videos by calibrating cross-attention results.
\textbf{Concept-Augmented Textual Inversion.}
Textual inversion~\citep{gal2022textual} learns to represent a specific set of user-provided images with pseudo-words in the latent space, offering an intuitive way for natural language-guided image editing.
We incorporate this technique into our framework to facilitate video editing.
However, due to the self-supervised nature within the limited latent space of the pre-trained diffusion model, the vanilla textual inversion often results in varied performance in terms of quality and efficiency for different image sets, requiring meticulous adjustments for learning rates and iteration counts.
% When performing textual inversion~\citep{gal2022textual} on a specific set of user-input images, and then using pre-trained concept words to participate in text-driven diffusion models for image editing,
% there are limitations due to the self-supervised nature within the limited latent space of the pre-trained diffusion model. This results in differences in quality and efficiency of textual inversion for different image sets, requiring adjustments to learning rates and iteration counts to adapt, or fine-tuning the diffusion model with a smaller learning rate to reduce iteration time costs.
To alleviate this issue, we draw inspiration from existing parameter-efficient fine-tuning techniques and propose adding LoRA modules~\citep{hu2022lora} to the value projection parameters in the cross-attention layers of the UNet.
Consequently, the values $\mV$ are updated to $LoRA(\mV)$ according to Eq.~(\ref{eq:lora}).
The rationale behind our approach is that we aim to enhance the expressiveness of the pre-trained diffusion model by slightly adjusting its capacity to accommodate new visual concepts while preserving its original generation capability.
Besides, inserting LoRA modules not only augments textual inversion with low storage overhead but also maintains a plug-and-play characteristic during inference.
We train the concept-word embeddings of textual inversion and the weight parameters of LoRA modules in an end-to-end manner (see orange blocks in Fig.~\ref{fig:overview}), where the learning rate for LoRA parameters is relatively smaller than that for concept-word embeddings to avoid over-fitting.
Denote the noise prediction network with LoRA modules loaded on value projection parameters as $\epsilon_{\theta_L}$, the optimization objective of concept-augmented textual inversion is then updated from Eq.~(\ref{eq:noise_pred_loss}) to the following:
\begin{equation}
\mathcal{L}_{noise}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0, 1),t}[\|\epsilon-\epsilon_{\theta_L}(z_t, t, c_\theta(y))\|_2^2].
\end{equation}
% To alleviate this issue, we combine existing fine-tuning techniques and attempt to add LoRA~\citep{hu2022lora} modules to the Value weight parameters in the cross-attention layers of the UNet in the diffusion model, specifically replacing the output $\mV$ of the Value module with $LoRA(\mV)$. We control the concept word embeddings and the associated learnable LoRA weight parameters to iterate at different learning rates. This approach improves the efficiency of textual inversion with low additional storage overhead while maintaining plug-and-play characteristics during inference. After inserting the LoRA module, the noise prediction function $\epsilon_\theta$ is updated to $\epsilon_{\theta_L}$, and the optimization objective is then replaced to:
% \begin{equation}
% \mathcal{L}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0, 1),t}[\|\epsilon-\epsilon_{\theta_L}(z_t, t, c_\theta(y))\|_2^2].
% \end{equation}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figures/losses.pdf}
\vspace{-8mm}
\caption{\textbf{Visualization of the dual prior supervision mechanism.} Each row displays a video frame, a set of cross-attention maps between this video frame and prompt words, and a pseudo ground truth mask. The \textit{scam} loss and \textit{tcam} loss are computed between relevant words and pseudo masks to prevent unintended changes for stabilized video editing. }
\vspace{-3mm}
\label{fig:losses}
\end{figure}
% \textbf{Tune Video Diffusion Model with Dual Prior Supervision.}
\textbf{Model Tuning with Dual Prior Supervision.}\label{sec:dps}
After learning concept-augmented textual inversion, we adapt and tune the video diffusion model for text-driven video editing in line with the paradigm of few-shot learning.
Specifically, we learn the LoRA-structured temporal convolution layers, the query projection weights within spatio-temporal attention layers and cross-attention layers, and the temporal self-attention layers (see red blocks in Fig.~\ref{fig:overview}).
These parameters are selected for updates during training due to their strong relevance to the temporal modeling of 3D videos.
To attain more stable and higher quality editing results, we tried directly integrating existing attention control techniques~\citep{hertz2023prompttoprompt} in an early attempt; however, we found that when applying text-driven video editing types such as word swap, the dispersion phenomenon of cross-attention between text embeddings and video latents leads to reduced stability in editing results.
To address this challenge, we propose a dual prior supervision mechanism, which includes a source cross-attention mask (\textit{scam}) loss and a target cross-attention mask (\textit{tcam}) loss.
% % Question: why cite a zero-shot work when mentioning fine-tuning the model???
% For text-driven editing of few-shot video data, fine-tuning the text-driven video generation model~\citep{qi2023fatezero}, which is a temporal extension of Stable-diffusion, can be used to achieve this.
% Combined with attention control techniques like~\citep{hertz2023prompttoprompt}, more stable and higher quality editing effects can be obtained.
% However, we found that when using text-driven video editing types such as word swap, the cross-attention dispersion of editing words on video latent variables leads to reduced stability in editing results. Therefore, we propose a method of supervising cross-attention using masks generated from pseudo-labels.
The \textit{scam} loss is designed to reduce the attention influence of the words to be replaced in the source prompt on irrelevant frame areas (see the first row in Fig.~\ref{fig:losses}).
It is also applied to modulate attention between concept words and concept videos (see the second row in Fig.~\ref{fig:losses}).
Specifically, for $K$ cross-attention layers in the UNet, we record cross-attention matrices $\mathbf{M}_{s}$ between the words and the video frame latents in each cross-attention layer.
To obtain ground truth for optimization, we use an off-the-shelf object detection network OWL-ViT~\citep{minderer2022simple} to localize objects in video frames and generate corresponding binary pseudo-labels ${\mathbf{M}}^{\text{gt}}_{s}$.
We further apply max pooling to generate $K$ pseudo-labels, each with a designated resolution $P_k$.
The loss is then calculated as the mean absolute loss on irrelevant areas:
\begin{equation}
\mathcal{L}_{scam} = \frac{1}{K} \sum_{k=1}^{K} \sum_{i=1}^{P_k} \left [ \| \mathbf{M}^{\text{gt}}_{s,k,i} - \mathbf{M}_{s,k,i} \| \cdot ( 1 - \mathbf{M}^{\text{gt}}_{s,k,i} ) \right ].
\end{equation}
% We then apply max pooling to the generated pseudo-labels to obtain a set of masks $\sM_s^{gt}$ with a set of \textcolor{red}{different resolutions $\sP$}, of length $K$. To supervise the attention distribution of the prompt words to be replaced in the source video and reduce their attention influence on irrelevant areas, we propose the calculation method for the scam (source cross attention mask) loss as follows, where $\overline{\vm}_{s,k}\in \sM_s^{gt}$, $\vm_{s,k}\in\sM_s$ and $p_k\in\sP$.
% \begin{equation}
% \mathcal{L}_{scam}=\frac{1}{K} \textcolor{red}{\sum_{k=0}^{K}\sum_{i=0}^{p_k}} [\| \overline{m}_{s,k,i}-m_{s,k,i}\| \cdot (1-\overline{m}_{s,k,i})].
% \end{equation}
The \textit{tcam} loss is introduced to diminish the attention influence of the target words in the edited prompt to further promote the consistency of irrelevant areas before and after video editing (see the third row in Fig.~\ref{fig:losses}).
Similar to the \textit{scam} loss, we obtain cross-attention matrices $\mathbf{M}_{t}$ and pseudo-labels $\mathbf{M}^{\text{gt}}_{t}$ between the target words in the edited prompt and the video frame latents.
The loss is computed as:
\begin{equation}
\mathcal{L}_{tcam} = \frac{1}{K} \sum_{k=1}^{K} \sum_{i=1}^{P_k} \left [ \| \mathbf{M}^{\text{gt}}_{t,k,i}-\mathbf{M}_{t,k,i} \| \cdot ( 1 - \mathbf{M}^{\text{gt}}_{t,k,i} ) \right].
\end{equation}
% To supervise the attention distribution between the target words in the edited prompt and the video to be edited, it is also necessary to calculate the tcam(target cross attention mask) loss in each iteration to reduce their attention influence on irrelevant areas. Similarly, we obtain the cross-attention probability set $\sM_t$ and mask set $\sM_t^{gt}$ between the target words in the edited text and the latent variables of the target video. The calculation method is as follows, where $\overline{\vm}_{t,k}\in \sM_t^{gt}$, $\vm_{t,k}\in\sM_t$ and $p_k\in\sP$.
% \begin{equation}
% \mathcal{L}_{tcam}=\frac{1}{K}\sum_{k=0}^{K}\sum_{i=0}^{p_k} [\| \overline{m}_{t,k,i}-m_{t,k,i}\| \cdot (1-\overline{m}_{t,k,i})].
% \end{equation}
Let the trainable parameters during the model tuning phase be denoted as $\epsilon_{\theta_{T}}$.
The noise prediction loss $\mathcal{L}_{noise}$ is then obtained by substituting $\epsilon_{\theta}$ in Eq.~(\ref{eq:noise_pred_loss}) with $\epsilon_{\theta_{T}}$.
Given $\alpha$ and $\beta$ as the weighting coefficients for our proposed \textit{scam} loss and \textit{tcam} loss, respectively, the total loss for model tuning with dual prior supervision is formulated as:
\begin{equation}
\mathcal{L}=\mathcal{L}_{noise}+\alpha \mathcal{L}_{scam}+\beta \mathcal{L}_{tcam}.
\label{eq:supvis}
\end{equation}
% Additionally, to complement the concept-augmented textual inversion for user-input custom target videos, we insert pre-trained LoRA parameters into the noise prediction network $\epsilon_{\theta_{T}}$ of the original temporally extended video generation model to obtain $\epsilon_{\theta_{TL}}$. We then \textcolor{red}{use masks} to supervise the input training source videos.
% \begin{equation}
% \mathcal{L}_{noise}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0, 1),t}[\|\epsilon-\epsilon_{\theta_{TL}}(z_t, t, c_\theta(y))\|_2^2].
% \end{equation}
% For the trainable parameters in the noise prediction network $\epsilon_{\theta_{TL}}$, the loss function combining the noise prediction loss and the attention supervision loss is as follows, where $\alpha$ and $\beta$ are the weight parameters for the scam loss and tcam loss respectively.
% \begin{equation}
% \mathcal{L}=\mathcal{L}_{noise}+\alpha \mathcal{L}_{scam}+\beta \mathcal{L}_{tcam}.
% \label{eq:supvis}
% \end{equation}
\textbf{Inference.}
As shown in Fig.~\ref{fig:overview}, the inference pipeline involves an inversion stage using the source text prompt, and an editing stage using the modified text prompt.
We cache the intermediate self-attention matrices and cross-attention matrices at each time step during inversion.
These matrices are then leveraged to manipulate attention during editing.
Specifically, we blend self-attention matrices to retain the semantic layout following FateZero~\cite{qi2023fatezero} (Self-Attention Blending), and swap cross-attention matrices between the changed words and video latents similar to Prompt-to-Prompt~\cite{hertz2023prompttoprompt} (Cross-Attention Swap).
\section{Experiments}
\subsection{Settings and Datasets}
Our experiments are conducted on a machine equipped with an NVIDIA GeForce RTX 4090. During the concept augmented textual inversion stage, we set the learning rate for CLIP~\citep{radford2021learning} word embeddings to $1\times 10^{-3}$, and the learning rate for LoRA modules inserted into the UNet to $1\times 10^{-5}$, with the number of training steps set to $5000$. %Additionally, in this stage, we augment the length of training frames from the concept video to prevent the inversion process from over-fitting to a fixed frames length. Specifically, We set the length of training frames per iteration within the range $[4, 8]$, and randomly sample sequences of frames of this length from the concept videos to obtain training frames.
Additionally, we randomly sample frame numbers within the range $[4, 8]$ from the concept video during training, to prevent the inversion process from over-fitting to a fixed frame number.
For the video diffusion model fine-tuning stage, we empirically set $\alpha=0.1$ and $\beta=0.1$ in Eq.~(\ref{eq:supvis}). The training steps above all use AdamW~\citep{loshchilov2017decoupled} optimizer. In the inference stage of video editing, the guidance scale is set to $12.5$, the number of DDIM Inversion steps is $T=50$, and the self-attention blending and cross-attention swap steps are within the interval $[0, 0.7 T]$. To evaluate our proposed method, we used a portion of the DAVIS~\citep{Caelles_arXiv_2019} dataset and clip videos from the internet to construct video editing pairs, either with or without concept videos.
% useless
% We compare the video editing results of Tune-A-Video~\citep{wu2023tune}, FateZero~\citep{qi2023fatezero}, MotionDirector~\citep{zhao2023motiondirector}, and RAVE~\citep{kara2024rave} methods in scenarios both including and excluding the concept. For MotionDirector, we utilized its supported temporal path and spatial path, training them using the source video and the concept video, respectively.
% \subsection{Datasets}
\subsection{Metrics}
\textbf{Frame Consistency.}
% To compare the coherence of the video frames $\sF$, we refer to the metric used in~\citep{wu2023tune, hessel2021clipscore}, which construct a set $\sD$ from the vector pairs $(\vv_i, \vv_j)$ encoded by the CLIP visual encoder~\citep{radford2021learning} for any two different frames $\vf_i, \vf_j\in \sF, \vf_i \ne \vf_j$ and calculates the average cosine distance $d$ in Eq.~(\ref{eq:fc}).
To compare the coherence of the video frames $\sF$, we refer to the metric used in~\citep{wu2023tune, hessel2021clipscore}, which calculates the average cosine distance $d$ between features $(\vv_i, \vv_j)$ of each two different frames $(\vf_i, \vf_j)$ encoded by the CLIP visual encoder~\citep{radford2021learning}, as Eq.~(\ref{eq:fc}). Here, $\vf_i, \vf_j\in \sF$, $\vf_i \ne \vf_j$, and $\sD$ denotes the set of the vector pairs $(\vv_i, \vv_j)$.
\begin{equation}
d=\frac{1}{|\sD|}\sum_{(\vv_i,\vv_j)\in \sD}\frac{\vv_i \cdot \vv_j}{\|\vv_i\| \|\vv_j\|}.
\label{eq:fc}
\end{equation}
\begin{table}[b]
\small
\centering
\begin{tabular}{lccc}
\toprule
% \hline
Method & Frame Consistency $\uparrow$ & M-PSNR $\uparrow$ & Concept Consistency $\uparrow$ \\
\midrule
% \hline
Tune-A-Video~\citep{wu2023tune} & 0.9399 & 14.88 & 0.6982\\
FateZero~\citep{qi2023fatezero} & 0.9361 & 17.33 & 0.6822\\
MotionDirector~\citep{zhao2023motiondirector} & 0.9446 & 13.48 & 0.7222\\
RAVE~\citep{kara2024rave} & 0.9339 & 17.67 & 0.6990 \\
\noalign{\vskip 0.4ex}
\hline
\noalign{\vskip 0.4ex}
Ours & \textbf{0.9464} & \textbf{19.90} & \textbf{0.7642}\\
\bottomrule
\end{tabular}\
\caption{\textbf{Quantitative comparison of video editing results.}}
\vspace{-5mm}
\label{tab:quan}
\end{table}
\textbf{Masked Peek-Signal-Noise Ratio.}
To compare the stability of the video non-target areas before and after target editing, we design a Masked Peak Signal-to-Noise Ratio (\textbf{M-PSNR}) metric. We use the OWL-ViT~\citep{minderer2022simple} open-vocabulary object detection model with text pseudo-labels to estimate the bounding box mask $\mM$ of the edited target. We then compare the average peek-signal-noise ratio of the original video frames and the edited video frames after applying this mask. The calculation formula for the specific function $f$ for the Mean Squared Error (MSE) used as input is as follows, where $\mM\in \mathbb{R}^{H\times W}$, $\mI^{s}\in \mathbb{R}^{H\times W \times C}$, and $\mI^{e}\in \mathbb{R}^{H\times W \times C}$ refer to the mask value, the frame pixel value of video before and after editing, respectively.
\begin{equation}
% f(\mI^{s}, \mI^{e}, \mM)=\frac{1}{C}\sum_{k=0}^{C}\frac{\sum\limits_{i=0}^{H}\sum\limits_{j=0}^{W}(I_{i,j,k}^{s}-I_{i,j,k}^{e})^2(1-M_{i,j})}{\sum\limits_{i=0}^{H}\sum\limits_{j=0}^{W}(1-M_{i,j})}.
% f(\mI^{s}, \mI^{e}, \mM)=\frac{1}{C}\sum_{k=0}^{C}\frac{\sum_{i=0}^{H}\sum_{j=0}^{W}(I_{i,j,k}^{s}-I_{i,j,k}^{e})^2(1-M_{i,j})}{\sum_{i=0}^{H}\sum_{j=0}^{W}(1-M_{i,j})}.
% f(\mI^{s}, \mI^{e}, \mM)=\frac{1}{C}\sum_{k\in C}\frac{\sum_{i\in H}\sum_{j\in W}(I_{i,j,k}^{s}-I_{i,j,k}^{e})^2(1-M_{i,j})}{\sum_{i\in H}\sum_{j\in W}(1-M_{i,j})}.
f(\mI^{s}, \mI^{e}, \mM)=\frac{1}{C}\frac{\sum_{k\in C}\sum_{i\in H}\sum_{j\in W}(I_{i,j,k}^{s}-I_{i,j,k}^{e})^2(1-M_{i,j})}{\sum_{i\in H}\sum_{j\in W}(1-M_{i,j})}.
\end{equation}
\textbf{Concept Consistency.}
To evaluate the correlation between the video editing results guided by the concept video and the concept video itself, while minimizing non-target areas interference, we employ a multi-step approach. First, we utilize a pre-trained OWL-ViT~\citep{minderer2022simple} model in conjunction with pseudo-label prediction to generate object masks for both videos. Using these masks, we then extract pixel segments of the target objects from both the edited video and the concept video. Finally, we leverage the CLIP model to predict visual encoding vectors for these extracted segments and calculate the average cosine similarity between them.
\begin{figure}[t]
\centering
%\vspace{-3mm}
\includegraphics[width=1.0\textwidth]{figures/qualitative.pdf}
\vspace{-5mm}
\caption{\textbf{Video generation with (\textbf{Setting I}) and without (\textbf{Setting II}) concept pairs.} The top row of the figure contains the concept video with its prompt in the left part, and comparison settings in the right part. The second row is the source video frames to be edited and its prompt. The rows below show the editing results of the source video using the editing prompt, for \citep{wu2023tune}, \citep{qi2023fatezero}, \citep{zhao2023motiondirector}, \citep{kara2024rave} and our method, respectively, in which words with `\$' ahead mean concept words, and the same applies to subsequent results.}
\label{fig:qual}
%\vspace{-5mm}
\end{figure}
\subsection{Comparisons with Existing Methods}
\textbf{Quantitative Evaluation.}
As illustrated in Tab.~\ref{tab:quan}, we quantitatively assess text-driven video editing results in three aspects.
Compared with existing methods that extend and fine-tune the Stable Diffusion model, including Tune-A-Video~\citep{wu2023tune}, FateZero~\citep{qi2023fatezero}, RAVE~\citep{kara2024rave}, and MotionDirector~\citep{zhao2023motiondirector}, our approach demonstrates superior inter-frame coherence in terms of the Frame Consistency Metric.
% compared to other methods that extend and fine-tune the Stable-Diffusion model
% Specifically, we utilize the Frame Consistency metric to evaluate the frame consistency of edited videos.
To evaluate the consistency of unrelated areas before and after video editing, we employ M-PSNR as a reference metric, and our method achieves the highest score by a large margin.
Concretely, our method outperforms MotionDirector~\citep{zhao2023motiondirector} by a noticeable 6.42 M-PSNR.
This is attributed to our proposed prior supervision mechanism, which effectively reduces the editing noise in non-target areas for both source and concept videos.
% As evidenced in Fig.~\ref{fig:qual} indicate that the attention supervision mechanism we proposed effectively reduces noise from both the source video and the concept video, thereby enhancing the consistency of non-target regions in the edited results.
Furthermore, to evaluate the target fidelity in concept and edited videos,
% in the edited results and the target in the concept video,
we utilize Concept Consistency as a reference metric, and our method demonstrates greater fidelity compared to others.
% As illustrated in Table~\ref{tab:quan}, to compare the frame consistency of the video editing results, we use the Frame Consistency metric as a numerical reference. MotionDirector~\citep{zhao2023motiondirector}, based on a pre-trained text-driven video generation model, has an advantage in inter-frame coherence. Our shows an advantage in inter-frame coherence compared to other methods that extend and fine-tune the Stable-Diffusion model, such as Tune-A-Video~\citep{wu2023tune} and FateZero ~\citep{qi2023fatezero}, as well as the RAVE~\citep{kara2024rave} that uses ControlNet \citep{zhang2023adding} to guide Stable-Diffusion for video editing. To compare the consistency of unrelated regions before and after video editing, we use M-PSNR as a reference metric, our method achieves higher scores compared to other methods. As indicated by the results in Figure~\ref{fig:qual}, the attention supervision we proposed can effectively reduce noise from the source video and concept video, and improve the consistency of non-target regions in the edited results. To compare the fidelity between the replaced target in the edited results and the target in the concept video, we use concept consistency as a reference metric, our method achieves higher faithfulness compared to other methods.
\begin{figure}[ht]
\centering
\includegraphics[width=0.9\textwidth]{figures/ablation-lora.pdf}
\vspace{-3mm}
\caption{\textbf{Comparison of whether to use Concept Augmentation (CA) for textual inversion.} Compared the text inversion results without and with concept augmentation for pairs (a), (b): `jeep' $\to$ \textcolor{red}{`\$LAMBO'}; and (c), (d): `jeep' $\to$ \textcolor{red}{`\$CYBERTRUCK'}, respectively, from the same source prompt ``a jeep driving down a curvy road in the countryside".}
\label{fig:abla-lora}
%\vspace{-5mm}
\end{figure}
\textbf{Qualitative Evaluation.}
The visual comparison results of video editing with and without concept video guidance are shown in Fig.~\ref{fig:qual}.
As can be seen, our method can maintain content consistency in non-target areas before and after video editing with and without concept videos.
Particularly, when using concept videos, our method can effectively introduce the visual concept from the concept video into the edited video.
For example, in Fig.~\ref{fig:qual} (Setting I), our method successfully replaces `man' with the concept `SOPTIMUS', while others either fail to preserve the background or cannot transfer the integral target shape.
% The visual comparison of our proposed method with Tune-A-Video~\citep{wu2023tune} and FateZero~\citep{qi2023fatezero} for video editing with and without concept video guidance is shown in Fig.~\ref{fig:qual}.
On the other hand, other approaches commonly face instability in non-target areas of their edited videos.
Tune-A-Video~\citep{wu2023tune} encounters the issue of dispersed cross-attention between word embeddings and video latents as it fine-tunes the model using only one video-text pair.
% This leads to instability in non-target areas of edited videos, such as noticeable changes in background color or even content.
While FateZero~\citep{qi2023fatezero} and RAVE~\citep{kara2024rave} mitigate this issue by manipulating cross-attention matrices or shuffling noise in a zero-shot manner, these methods directly use concepts to drive video editing, which results in compromised non-target area consistency and degraded concept fidelity.
% mitigates this issue to some extent by partially replacing cross-attention probability matrices in a zero-shot manner, it
% directly using the concept to drive video editing may affect the consistency between the video before and after editing,
% RAVE~\citep{kara2024rave} as well, since the concept words may introduce noise information from unrelated regions during inversion.
MotionDirector~\citep{zhao2023motiondirector} naturally supports extracting targets from concept videos via its trainable spatial path; however, the coupled spatial and temporal paths struggle to provide stable guidance, leading to noticeable inconsistencies in non-target areas.
% Despite MotionDirector~\citep{zhao2023motiondirector} natively supports introducing targets from concept videos through its trainable spatial path, when using the spatial path to intervene in the temporal path to drive video editing, the lack of effective and stable guidance causes the editing results to be uncontrollably influenced by the spatial path, resulting in low consistency of non-target areas.
In contrast, our proposed concept-augmented textual inversion and dual prior supervision can effectively maintain content consistency in non-target areas before and after video editing while accurately capturing specific attributes of user-provided concepts.
% In contrast, the attention supervision we proposed can effectively address both of these issues, maintaining consistency in non-target areas of the video before and after editing while effectively embodying the specific attributes of the target concept.
% \begin{figure}[!htb]
% \centering
% \includegraphics[width=1.0\textwidth]{figures/qualitative.pdf}
% \caption{\textbf{Qualitative comparison of text-driven video editing with and without concept video.} At the top of the figure are the source video frames to be edited and its corresponding descriptive prompt. The rows below show the editing results of the source video using the editing prompt, for \citep{wu2023tune}, \citep{qi2023fatezero}, \citep{zhao2023motiondirector}, \citep{kara2024rave} and our method, respectively.}
% \label{fig:qual}
% \end{figure}
\subsection{Ablation Study}
\begin{figure}[t]
\centering
\includegraphics[width=0.98\textwidth]{figures/ablation-full-supvis-with-attn-prob-3.pdf}
%\vspace{-3mm}
\caption{\textbf{The impact of dual prior supervision.} From the first to the last row, using the editing example in Fig.~\ref{fig:overview}, we compare the average cross-attention maps and the editing results with and without the supervision mechanism of \textit{scam} and \textit{tcam}. Each case contains three pairs, and each pair consists of an average cross-attention map on the left and an edited frame on the right. The full comparisons are put in Fig.~\ref{fig:abla-full-supvis-6} and Fig.~\ref{fig:abla-full-supvis-attn-prob-6}.}
%\vspace{-3mm}
\label{fig:abla-full-supvis}
\end{figure}
\textbf{Concept Augmentation Alleviates Under-Fitting of Textual Inversion.}
In this work, we draw on the idea of Textual Inversion (TI) from text-to-image generation and apply it to text-driven video editing to address the embedding of external concept words. However, simply applying TI may lead to under-fitting, resulting in a lack of realism. For instance, in the results shown in Fig.~\ref{fig:abla-lora}(a) and Fig.~\ref{fig:abla-lora}(c), where the keywords `jeep' are altered to `LAMBO' and `CYBERTRUCK', although some attributes (e.g., shape) of the target concepts are partially retained, the results appear to "drift" due to insufficient inductive bias. In contrast, the concept-augmented textual inversion can effectively capture the color, shape, and other attributes, as demonstrated in Fig.~\ref{fig:abla-lora}(b) and Fig.~\ref{fig:abla-lora}(d). The concept augmentation provides more detailed features to the target, significantly improving the fidelity of details in the inversion results.
% As shown in Fig.~\ref{fig:abla-lora}, using Concept Augmented Textual-Inversion can more accurately reflect the color, shape, and other attributes of the target concept in the editing results. Due to the limited latent space of diffusion models and the differences in latent space distribution among various concept videos, Textual Inversion~\citep{gal2022textual} results face the risk of under-fitting, leading to insufficient reflection of the detailed characteristics of the inversion target. However, Textual Inversion augmented with LoRA can effectively enhance the fidelity of details in the inversion results.
\textbf{Dual Prior Supervision Improves Stability and Fidelity.}
In this work, we propose a Dual Prior Supervision strategy, which consists of two main components (See Sec.~\ref{sec:dps}): \textit{scam} loss and \textit{tcam} loss. Both components play crucial roles in maintaining the stability of the target generation. By comparing the attention regions in Fig.~\ref{fig:abla-full-supvis} (a) (w/o \textit{tcam}, w/o \textit{scam}), Fig.~\ref{fig:abla-full-supvis} (b) (w/o \textit{tcam}, w/ \textit{scam}), and Fig.~\ref{fig:abla-full-supvis} (c) (w/ \textit{tcam}, w/o \textit{scam}), we can conclude that both \textit{scam} and \textit{tcam} (Fig.~\ref{fig:abla-full-supvis} (d)) significantly reduce background disturbances and improve stability. However, the generated video results reveal that using either component alone cannot effectively reproduce the target object's attributes, such as the car's color. The proposed \textbf{Dual Prior Supervision}, which combines both components, not only enhances the stability of the background in the target results, but also captures the target object's attributes more accurately, thereby improving the fidelity of the edited concept target.
% As shown in Fig.~\ref{fig:abla-full-supvis}, the editing results using both scam and tcam attention supervision mechanisms demonstrate more stable backgrounds and more faithful target concept attributes compared to other supervision mechanisms. Compared to using only scam supervision, our method effectively improves the stability of the background in the editing results by constraining the attention between the concept and irrelevant areas of the source video through tcam. Compared to using only the tcam supervision mechanism, the scam mechanism effectively reduces the possibility of introducing noise from irrelevant areas during tuning with target words, thereby improving the fidelity of the edited concept target.
\textbf{Tuning w/ Concept Video Produces Stylized Results.}
Recall that we construct the target videos in this work by templating the concept pairs to make the editing process more flexible. To explore the impact of the concept video in \textbf{Setting I} (Fig.~\ref{fig:qual}), we conduct a simple experiment as shown in Fig.~\ref{fig:abla-tune-concept}. As shown in Fig.~\ref{fig:abla-tune-concept}(a) and Fig.~\ref{fig:abla-tune-concept}(b), tuning models with both concept video and concept prompt produce more stylized videos. The possible explanation for this is that the introduction of the concept video alleviates the over-fitting issue. Note that tuning with only the concept video (without the concept prompt) is not viable here, as we cannot analyze the intent without any textual guidance.
% As illustrated in Fig.~\ref{fig:abla-tune-concept}, fine-tuning the video diffusion model with concept videos can effectively improve the faithfulness of consistency between the replacement target and the concept. When fine-tuning the video diffusion model using only a single sample of the source video, the newly added trainable parameters for temporal extension are prone to over-fitting the content in the source video. However, incorporating concept videos into the tuning process effectively mitigates this over-fitting problem.
\begin{figure}[!htb]
\centering
\includegraphics[width=0.98\textwidth]{figures/ablation-tune-concept.pdf}
\caption{\textbf{Comparison of whether to tune with the concept video.} Compared the video editing results without and with tuning concept video for the left part: `car' $\to$ \textcolor{red}{`\$GT3'}; and the right part: `car' $\to$ \textcolor{red}{`\$LAMBO'}, from the source prompt ``a car is drifting around a curve road with the background of a forest" and ``a car is drifting in the snow", respectively.}
\label{fig:abla-tune-concept}
%\vspace{-5mm}
\end{figure}
\section{Limitations and Future Work}
\textbf{Mismatch when Significant Deformation.}
Although our proposed method effectively mitigates the inconsistency in non-target areas caused by attention dispersion in video editing methods using attention replacement mechanisms, it may struggle when a single concept video guides target replacement in cases of significant deformation in the source video, such as running people. For instance, there may be insufficient detailed correspondences between the internal parts of the replacing and replaced targets during deformation, such as moving arms and legs. Potential solutions include ControlNet~\citep{zhang2023adding} and OpenPose~\citep{8765346}, which utilize motion conditions, like human pose, to guide the video editing process.
% Although our proposed method effectively alleviates the problem of inconsistency in non-target regions caused by attention dispersion in video editing methods based on attention replacement mechanisms, it may fail when using a single concept video to guide target replacement in scenarios where the replaced target undergoes significant deformation in the source video, such as running people. For example, lack of more detailed correspondences between the internal parts in the deformation progress of the replacing and replaced targets, like moving arms and legs. Possible solutions like ControlNet~\citep{zhang2023adding} and OpenPose~\citep{8765346}, utilizing motion conditions such as human pose to guide the video editing process.
\section{Conclusion}
% In this paper, we introduce a framework that extracts concept words from user-customized concept videos and employs an attention supervision mechanism to fine-tune an extended Stable Diffusion model for text-driven video editing. By utilizing concept-augmented textual inversion to simultaneously fine-tune a set of plug-and-play learnable parameters in both the CLIP and UNet sides with different learning rates, our method more efficiently captures the detailed attributes of the target in the concept video. By applying attention masks estimated by a pre-trained model to supervise the cross-attention during the fine-tuning of both the concept video and the source video, we effectively reduce the influence of concept words and normal words on unrelated regions. Experimental results demonstrate that our method significantly improves the consistency of non-target regions in the video editing results and the fidelity of the edited target.
In this paper, we propose an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs. Specifically, the framework introduces a Concept-Augmented Textual Inversion (CATI), which extracts the target concept from user-customized videos. In practice, CATI enables plug-and-play guidance of stable diffusion for video shaping, effectively capturing target attributes for more stylized editing results. In addition, a Dual Prior Supervision (DPS) mechanism is designed to prevent unintended changes in non-target visual areas by crossing the attention between the sources and targets. Experimental results demonstrate that our method significantly improves the flexibility, consistency, and stability of text-guided video editing.
\section{Appendix}
\subsection{Training and Inference Details}
\textbf{Details of Concept Augmented Textual Inversion.} During the stage of concept augmented textual inversion, we use AdamW optimizer~\citep{loshchilov2017decoupled}, with the default betas set to $0.9$ and $0.999$, the weight decay set to $1\times 10^{-2}$ and the epsilon set to $1\times 10^{-8}$. Besides, LoRA modules inserted to UNet are without bias trainable parameters, in which the LoRA rank is set to $16$, the weight coefficient to scale LoRA output is set to $1.0$, and the dropout parameter is set to $0.1$. Additionally, we add a prefix to the head of concept prompts and use one embedding to represent a concept word as the same as in Textual Inversion~\citep{gal2022textual}.
\begin{figure}[!htb]
\centering
\includegraphics[width=1.0\textwidth]{figures/qualitative-no-concept.pdf}
\vspace{-5mm}
\caption{\textbf{Additional qualitative comparison of text-driven video editing without concept video.} At the top of the figure are the source video frames to be edited and its corresponding descriptive prompt. The rows below show the editing results of the source video using the editing prompt, for \citep{wu2023tune}, \citep{qi2023fatezero}, \citep{zhao2023motiondirector}, \citep{kara2024rave}, and our method, respectively.}
\label{fig:qual-no-concept}
\end{figure}
\textbf{Details of Dual Prior Supervision.} During the stage of tuning the temporally extended diffusion model with dual prior supervision, we use the same optimizer settings in concept augmented textual inversion stage. For each training epoch, we sequentially use the pairs of source video with source prompt and concept video with concept prompt as input for one training iteration, predicting noise and computing the noise prediction loss. During one training iteration, we retain all the cross-attention probability matrices from the cross attention layers in the UNet and compute \textit{scam} based on the pseudo ground-truth mask corresponding to each video. Before the end of each iteration, we then input the pair of source video and target prompt, also retaining the cross-attention probability matrices, and compute \textit{tcam} based on the pseudo ground-truth mask. The number of training epochs are less than $250$. The input data frames have a resolution of $512\times 512$ pixels and a length of $6$ frames. The training time does not exceed $30$ minutes on an NVIDIA GeForce RTX 4090 device.
\begin{figure}[t]
\centering
\includegraphics[width=1.0\textwidth]{figures/qualitative-concept-2.pdf}
\vspace{-5mm}
\caption{\textbf{Additional qualitative comparison of text-driven video editing with concept video.} At the top of the figure are the concept videos and the prompt containing the concept word. Below are the source video frames to be edited along with its descriptive prompt. The rows below show the editing results of the source video using the prompt containing the concept word, for \citep{wu2023tune}, \citep{qi2023fatezero}, \citep{zhao2023motiondirector}, \citep{kara2024rave}, and our method, respectively.}
\label{fig:qual-concept-2}
\end{figure}
\textbf{Details of Inference.} During the stage of inference, we use the device equipped with an NVIDIA GeForce RTX 4090 and store the model parameters and data inputs in fp16 format. During the denoising of the source video, we store intermediate variables including the self-attention and cross-attention probability matrices and source latents at each step in RAM, occupying approximately $100$ GB space. The attention control processes of self-attention blending and cross-attention swap consume most of the inference time. Depending on different attention control configurations, the inference time generally ranges between $1$ to $3$ minutes.
Additionally, to reduce the host memory overhead of intermediate variables during inference, we have designed and implemented a memory-saving inference scheme. This scheme requires about twice the inference time compared to the original but reduces the overhead of RAM to $5$ GB. The key difference between this scheme and the original is that it does not store the self-attention and cross-attention probability matrices; instead, it only stores the source latents at each denoising step and recalculates the self-attention and cross-attention probability matrices during attention control.
\begin{figure}[t]
\centering
\includegraphics[width=1.0\textwidth]{figures/qualitative-concept-3.pdf}
\vspace{-5mm}
\caption{\textbf{Additional qualitative comparison of text-driven video editing with concept video.} The form is the same as Fig.~\ref{fig:qual-concept-2}.}
\label{fig:qual-concept-3}
\end{figure}
\subsection{Additional Comparisons with Existing Methods.}
\textbf{Text-driven Video Editing without Concept Video.} We provide an additional set of comparison examples without concept video guidance for video editing, as shown in Fig.~\ref{fig:qual-no-concept}.
% \begin{figure}[!htb]
% \centering
% \includegraphics[width=1.0\textwidth]{figures/qualitative-no-concept-2.pdf}
% \vspace{-5mm}
% \caption{\textbf{Additional qualitative comparison of text-driven video editing without concept video.} At the top of the figure are the source video frames to be edited and its corresponding descriptive prompt. The rows below show the editing results of the source video using the editing prompt, for \citep{wu2023tune}, \citep{qi2023fatezero}, \citep{zhao2023motiondirector}, \citep{kara2024rave}, and our method, respectively.}
% \label{fig:qual-no-concept}
% \end{figure}
\textbf{Text-driven Video Editing with Concept Video.} We provide six additional comparison examples with concept video guidance for video editing, as shown in Fig.~\ref{fig:qual-concept-2}, Fig.~\ref{fig:qual-concept-3}, and Fig.~\ref{fig:qual-concept-4}.
% \begin{figure}[!htb]
% \centering
% \includegraphics[width=1.0\textwidth]{figures/qualitative-concept-2.pdf}
% \vspace{-5mm}
% \caption{\textbf{Additional qualitative comparison of text-driven video editing with concept video.} At the top of the figure are the concept videos and the prompt containing the concept word. Below are the source video frames to be edited along with its descriptive prompt. The rows below show the editing results of the source video using the prompt containing the concept word, for \citep{wu2023tune}, \citep{qi2023fatezero}, \citep{zhao2023motiondirector}, \citep{kara2024rave}, and our method, respectively.}
% \label{fig:qual-concept-2}
% \end{figure}
% \begin{figure}[!htb]
% \centering
% \includegraphics[width=1.0\textwidth]{figures/qualitative-concept-3.pdf}
% \vspace{-5mm}
% \caption{\textbf{Additional qualitative comparison of text-driven video editing with concept video.} The form is the same as Figure~\ref{fig:qual-concept-2}.}
% \label{fig:qual-concept-3}
% \end{figure}
\begin{figure}[!htb]
\centering
\includegraphics[width=1.0\textwidth]{figures/qualitative-concept-4.pdf}
\vspace{-5mm}
\caption{\textbf{Additional qualitative comparison of text-driven video editing with concept video.} The form is the same as Fig.~\ref{fig:qual-concept-2}.}
\label{fig:qual-concept-4}
\end{figure}
\FloatBarrier
\subsection{Additional Ablation Study Results}
\textbf{Additional Results for Dual Prior Supervision.} We provide the full comparison of the editing results for Fig.~\ref{fig:abla-full-supvis}, as shown in Figure~\ref{fig:abla-full-supvis-6}, and the comparison of the full average cross attention map in Fig.~\ref{fig:abla-full-supvis-attn-prob-6}.
\begin{figure}[!htb]
\centering
\includegraphics[width=0.9\textwidth]{figures/ablation-full-supvis-6.pdf}
\vspace{-2mm}
\caption{\textbf{Full comparison of whether to tune with attention supervision.} The first column shows frames from the source video to be edited. From the second column to the last column, using the editing pair in Fig.~\ref{fig:overview} as the example, we compare the editing results with and without the attention supervision mechanism of scam and tcam.}
\label{fig:abla-full-supvis-6}
\end{figure}
\begin{figure}[!htb]
\centering
\includegraphics[width=0.9\textwidth]{figures/ablation-full-supvis-attn-prob-6.pdf}
\vspace{-5mm}
\caption{\textbf{Average cross attention probability map explorations.} The first column shows frames from the source video to be edited. From the second column to the last column, using the editing pair in Fig.~\ref{fig:overview} as the example, we compare the average cross attention maps with and without the attention supervision mechanism of scam and tcam, visualization of plasma colormap in matplotlib~\citep{Hunter:2007}.}
\label{fig:abla-full-supvis-attn-prob-6}
\end{figure}
\FloatBarrier
\textbf{Impact of Attention Supervision Weights.}
Considering the impact of different attention supervision weights, Fig.~\ref{fig:abla-weight} shows that as the weight increases, there is a higher degree of overlap between the area of the replacement target and the edited target. Simultaneously, the consistency in non-target areas improves to a certain extent.
% \begin{figure}[htbp]
\begin{figure}[!htb]
\centering
\includegraphics[width=1.0\textwidth]{figures/ablation-weight.pdf}
\vspace{-5mm}
\caption{\textbf{Hyperparameters of different attention supervision weights.} The first column shows frames from the source video to be edited. From the second column to the last column, using the editing pair in Fig.~\ref{fig:abla-lora}(c) as the example, we compare the editing results with attention supervision weights from 0.05, 0.1 and 0.3.}
\label{fig:abla-weight}
\end{figure}
\begin{thebibliography}{33}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Avrahami et~al.(2022)Avrahami, Lischinski, and Fried]{Avrahami_2022_CVPR}
Omri Avrahami, Dani Lischinski, and Ohad Fried.
\newblock Blended diffusion for text-driven editing of natural images.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pp.\ 18208--18218, June 2022.
\bibitem[Avrahami et~al.(2023)Avrahami, Fried, and Lischinski]{avrahami2023blendedlatent}
Omri Avrahami, Ohad Fried, and Dani Lischinski.
\newblock Blended latent diffusion.
\newblock \emph{ACM Trans. Graph.}, 42\penalty0 (4), jul 2023.
\newblock ISSN 0730-0301.
\newblock \doi{10.1145/3592450}.
\newblock URL \url{https://doi.org/10.1145/3592450}.
\bibitem[Bar-Tal et~al.(2022)Bar-Tal, Ofri-Amar, Fridman, Kasten, and Dekel]{bar2022text2live}
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.
\newblock Text2live: Text-driven layered image and video editing.
\newblock In \emph{European conference on computer vision}, pp.\ 707--723. Springer, 2022.
\bibitem[Blattmann et~al.(2023)Blattmann, Dockhorn, Kulal, Mendelevitch, Kilian, Lorenz, Levi, English, Voleti, Letts, et~al.]{blattmann2023stable}
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et~al.
\newblock Stable video diffusion: Scaling latent video diffusion models to large datasets.
\newblock \emph{arXiv preprint arXiv:2311.15127}, 2023.
\bibitem[Caelles et~al.(2019)Caelles, Pont-Tuset, Perazzi, Montes, Maninis, and {Van Gool}]{Caelles_arXiv_2019}
Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokitsi Maninis, and Luc {Van Gool}.
\newblock The 2019 davis challenge on vos: Unsupervised multi-object segmentation.
\newblock \emph{arXiv:1905.00737}, 2019.
\bibitem[{Cao} et~al.(2019){Cao}, {Hidalgo Martinez}, {Simon}, {Wei}, and {Sheikh}]{8765346}
Z.~{Cao}, G.~{Hidalgo Martinez}, T.~{Simon}, S.~{Wei}, and Y.~A. {Sheikh}.
\newblock Openpose: Realtime multi-person 2d pose estimation using part affinity fields.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine Intelligence}, 2019.
\bibitem[Chai et~al.(2023)Chai, Guo, Wang, and Lu]{chai2023stablevideo}
Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.
\newblock Stablevideo: Text-driven consistency-aware diffusion video editing.
\newblock \emph{arXiv preprint arXiv:2308.09592}, 2023.
\bibitem[Esser et~al.(2023)Esser, Chiu, Atighehchian, Granskog, and Germanidis]{esser2023structure}
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.
\newblock Structure and content-guided video synthesis with diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 7346--7356, 2023.
\bibitem[Gal et~al.(2022)Gal, Alaluf, Atzmon, Patashnik, Bermano, Chechik, and Cohen-Or]{gal2022textual}
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or~Patashnik, Amit~H. Bermano, Gal Chechik, and Daniel Cohen-Or.
\newblock An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
\newblock URL \url{https://arxiv.org/abs/2208.01618}.
\bibitem[Hertz et~al.(2023)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-or]{hertz2023prompttoprompt}
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or.
\newblock Prompt-to-prompt image editing with cross-attention control.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\newblock URL \url{https://openreview.net/forum?id=_CDixzkzeyb}.
\bibitem[Hessel et~al.(2021)Hessel, Holtzman, Forbes, Bras, and Choi]{hessel2021clipscore}
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan~Le Bras, and Yejin Choi.
\newblock {CLIPScore:} a reference-free evaluation metric for image captioning.
\newblock In \emph{EMNLP}, 2021.
\bibitem[Ho et~al.(2020)Ho, Jain, and Abbeel]{ho2020denoising}
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 6840--6851, 2020.
\bibitem[Ho et~al.(2022)Ho, Chan, Saharia, Whang, Gao, Gritsenko, Kingma, Poole, Norouzi, Fleet, et~al.]{ho2022imagen}
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik~P Kingma, Ben Poole, Mohammad Norouzi, David~J Fleet, et~al.
\newblock Imagen video: High definition video generation with diffusion models.
\newblock \emph{arXiv preprint arXiv:2210.02303}, 2022.
\bibitem[Hong et~al.(2022)Hong, Ding, Zheng, Liu, and Tang]{hong2022cogvideo}
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.
\newblock Cogvideo: Large-scale pretraining for text-to-video generation via transformers.
\newblock \emph{arXiv preprint arXiv:2205.15868}, 2022.
\bibitem[Hu et~al.(2022)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen]{hu2022lora}
Edward~J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang, and Weizhu Chen.
\newblock Lo{RA}: Low-rank adaptation of large language models.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\newblock URL \url{https://openreview.net/forum?id=nZeVKeeFYf9}.
\bibitem[Hunter(2007)]{Hunter:2007}
J.~D. Hunter.
\newblock Matplotlib: A 2d graphics environment.
\newblock \emph{Computing in Science \& Engineering}, 9\penalty0 (3):\penalty0 90--95, 2007.
\newblock \doi{10.1109/MCSE.2007.55}.
\bibitem[Kara et~al.(2024)Kara, Kurtkaya, Yesiltepe, Rehg, and Yanardag]{kara2024rave}
Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James~M. Rehg, and Pinar Yanardag.
\newblock Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2024.
\bibitem[Kasten et~al.(2021)Kasten, Ofri, Wang, and Dekel]{kasten2021layered}
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel.
\newblock Layered neural atlases for consistent video editing.
\newblock \emph{ACM Transactions on Graphics (TOG)}, 40\penalty0 (6):\penalty0 1--12, 2021.
\bibitem[Lee et~al.(2023)Lee, Jang, Chen, Qiu, and Huang]{lee2023shape}
Yao-Chih Lee, Ji-Ze~Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang.
\newblock Shape-aware text-driven layered video editing.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 14317--14326, 2023.
\bibitem[Loshchilov \& Hutter(2017)Loshchilov and Hutter]{loshchilov2017decoupled}
Ilya Loshchilov and Frank Hutter.
\newblock Decoupled weight decay regularization.
\newblock \emph{arXiv preprint arXiv:1711.05101}, 2017.
\bibitem[Minderer et~al.(2022)Minderer, Gritsenko, Stone, Neumann, Weissenborn, Dosovitskiy, Mahendran, Arnab, Dehghani, Shen, et~al.]{minderer2022simple}
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et~al.
\newblock Simple open-vocabulary object detection.
\newblock In \emph{European Conference on Computer Vision}, pp.\ 728--755. Springer, 2022.
\bibitem[Qi et~al.(2023)Qi, Cun, Zhang, Lei, Wang, Shan, and Chen]{qi2023fatezero}
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.
\newblock Fatezero: Fusing attentions for zero-shot text-based video editing.
\newblock \emph{arXiv:2303.09535}, 2023.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et~al.]{radford2021learning}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et~al.
\newblock Learning transferable visual models from natural language supervision.
\newblock In \emph{International conference on machine learning}, pp.\ 8748--8763. PMLR, 2021.
\bibitem[Rombach et~al.(2021)Rombach, Blattmann, Lorenz, Esser, and Ommer]{rombach2021highresolution}
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
\newblock High-resolution image synthesis with latent diffusion models, 2021.
\bibitem[Song et~al.(2020)Song, Meng, and Ermon]{song2020denoising}
Jiaming Song, Chenlin Meng, and Stefano Ermon.
\newblock Denoising diffusion implicit models.
\newblock \emph{arXiv preprint arXiv:2010.02502}, 2020.
\bibitem[Sterling(2023)]{zeroscope}
Spencer Sterling.
\newblock Zeroscope, 2023.
\newblock URL \url{https://huggingface.co/cerspense/zeroscope_v2_576w}.
\bibitem[Vaswani(2017)]{vaswani2017attention}
A~Vaswani.
\newblock Attention is all you need.
\newblock \emph{Advances in Neural Information Processing Systems}, 2017.
\bibitem[Wang et~al.(2023)Wang, Yuan, Chen, Zhang, Wang, and Zhang]{wang2023modelscope}
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.
\newblock Modelscope text-to-video technical report.
\newblock \emph{arXiv preprint arXiv:2308.06571}, 2023.
\bibitem[Wang et~al.(2024)Wang, Yuan, Zhang, Chen, Wang, Zhang, Shen, Zhao, and Zhou]{wang2024videocomposer}
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.
\newblock Videocomposer: Compositional video synthesis with motion controllability.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Wu et~al.(2023)Wu, Ge, Wang, Lei, Gu, Shi, Hsu, Shan, Qie, and Shou]{wu2023tune}
Jay~Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan~Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike~Zheng Shou.
\newblock Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 7623--7633, 2023.
\bibitem[Yang \& Tang(2022)Yang and Tang]{yang2022supervised}
Gene-Ping Yang and Hao Tang.
\newblock Supervised attention in sequence-to-sequence models for speech recognition.
\newblock In \emph{ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pp.\ 7222--7226. IEEE, 2022.
\bibitem[Zhang et~al.(2023)Zhang, Rao, and Agrawala]{zhang2023adding}
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.
\newblock Adding conditional control to text-to-image diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 3836--3847, 2023.
\bibitem[Zhao et~al.(2023)Zhao, Gu, Wu, Zhang, Liu, Wu, Keppo, and Shou]{zhao2023motiondirector}
Rui Zhao, Yuchao Gu, Jay~Zhangjie Wu, David~Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike~Zheng Shou.
\newblock Motiondirector: Motion customization of text-to-video diffusion models.
\newblock \emph{arXiv preprint arXiv:2310.08465}, 2023.
\end{thebibliography}