Pathways on the Image Manifold: Image Editing via Video Generation
Extracted source text · arXiv 2411.16819
% CVPR 2025 Paper Template; see https://github.com/cvpr-org/author-kit
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
\usepackage{cvpr} % To produce the CAMERA-READY version
% \usepackage[review]{cvpr} % To produce the REVIEW version
% \usepackage[pagenumbers]{cvpr} % To force page numbers, e.g. for an arXiv version
\usepackage[accsupp]{axessibility}
\usepackage{ulem}
\usepackage{placeins}
\usepackage{float}
% \usepackage{nicematrix}
\usepackage{tabularray}
% Import additional packages in the preamble file, before hyperref
\input{preamble}
% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues,
% e.g. with the file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete *.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you should be clear).
\definecolor{cvprblue}{rgb}{0.21,0.49,0.74}
\usepackage[pagebackref,breaklinks,colorlinks,allcolors=cvprblue]{hyperref}
\usepackage{xcolor} % Allows you to use colored text
\usepackage{colortbl}
\newcommand{\noam}[1]{\textcolor{green}{#1}}
% In the preamble (beginning of the document)
\newcommand{\gal}[1]{\textcolor{blue}{#1}}
\newcommand{\silver}[1]{\textcolor{purple}{#1}}
\newcommand{\db}[1]{\textcolor{red}{#1}}
\newcommand{\ron}[1]{\textcolor{violet}{#1}}
\newcommand{\roy}[1]{\textcolor{cyan}{#1}}
\newcommand{\yell}[1]{\textcolor{yellow}{#1}}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\paperID{15470} % *** Enter the Paper ID here
\def\confName{CVPR}
\def\confYear{2025}
%%%%%%%%% TITLE - PLEASE UPDATE
\title{
% Frame2Frame: Image Editing Through Video Generation \\
% Frame2Frame: Video generation is all you need for Image Editing \\
Pathways on the Image Manifold:
Image Editing via Video Generation
}
% Generative Video Models as Image Editors\\
% Temporal Image-to-Image Translation Through Video Generation \\
% Frame2Frame: Image-to-Image Translation Through Video Generation
%%%%%%%%% AUTHORS - PLEASE UPDATE
% \author{First Author\\
% Institution1\\
% Institution1 address\\
% {\tt\small firstauthor@i1.org}
% % For a paper whose authors are all at the same institution,
% % omit the following lines up until the closing ``}''.
% % Additional authors and addresses can be added with ``\and'',
% % just like the second author.
% % To save space, use either the email address or home page, not both
% \and
% Second Author\\
% Institution2\\
% First line of institution2 address\\
% {\tt\small secondauthor@i2.org}
% }
\author{
Noam Rotstein \qquad
Gal Yona \qquad
Daniel Silver \qquad
\\
Roy Velich \qquad
David Bensa\"id \qquad
Ron Kimmel
\\ \\
Technion - Israel Institute of Technology
}
\begin{document}
\maketitle
\input{sec/0_abstract}
\input{sec/1_zintro-david}
\input{sec/2_related}
\input{sec/3_method}
\input{sec/4_0_manifold}
\input{sec/4_experiments}
\input{sec/5_limitations}
\input{sec/6_conclusions}
\FloatBarrier
% \input{sec/limitations}
\newpage
{
\small
\bibliographystyle{ieeenat_fullname}
\bibliography{main}
}
\input{sec/7_appendix}
% WARNING: do not forget to delete the supplementary pages from your submission
% \input{sec/X_suppl}
\end{document}
%
% --- inline annotations
%
\newcommand{\red}[1]{{\color{red}#1}}
\newcommand{\todo}[1]{{\color{red}#1}}
\newcommand{\TODO}[1]{\textbf{\color{red}[TODO: #1]}}
% --- disable by uncommenting
% \renewcommand{\TODO}[1]{}
% \renewcommand{\todo}[1]{#1}
\begin{abstract}
% Image editing techniques have advanced significantly in recent years, particularly with the emergence of image diffusion models.
% Simultaneously, video generation has made remarkable strides, producing models that effectively function as world simulators.
% In this paper, we propose merging these two fields by utilizing image-to-video models for image editing.
% We frame the image editing task as simulating real-world changes and leverage generative video models in a zero-shot manner to represent and apply these transformations, effectively performing an image-to-video-to-image process.
% Our approach achieves state-of-the-art results in image editing tasks across various benchmarks, evaluated using several models.
% In addition, we highlight the limitations of such approaches and suggest potential future solutions.
%In recent years the field of generative image editing has seen substantial progress with the emergence of image diffusion models, though critical challenges remain. Models tend to lack fidelity, altering essential aspects of original images, and struggling to accurately follow edit instructions.
%Simultaneously, video generation has made remarkable strides, producing models that effectively function as consistent and continues world simulators.
%In this paper, we propose merging these two fields by utilizing image-to-video models for image editing.
%We frame the image editing task as simulating real-world changes \noam{not sure real world is good} \db{realistic modifications} and leverage generative video models in a zero-shot manner \db{with pretrained generative video models} \db{to smoothly interpolate bewteen the original picture and the edited one.} to represent and apply these \db{to simulate these} transformations, effectively performing an image-to-video-to-image process \db{maybe remove the last sentence}. Our approach traverses the image manifold through continuous video, ensuring consistent edits and preserving key aspects of the original image.
%Our approach achieves state-of-the-art results in image editing tasks across various benchmarks \roy{"various" is misleading, as we're only using TedBench, maybe change to "various metrics" in the context of TedBench}, evaluated using several models.
%In addition, we highlight the limitations of such approaches and suggest potential future solutions. \db{Not sure that it should be in the abstract.}
%\noam{maybe add specifics}
%\noam{maybe change all paper to img2img translation? Time to decide! call it Frame2Frame, if translation than its not only generative.}
%\noam{kimmel:traversing on the images manifold which makes the transition more sensible in most cases}
%\noam{add here "Frame2Frame"}
%\noam{add here simple?}
Recent advances in image editing, driven by image diffusion models, have shown remarkable progress.
However, significant challenges remain, as these models often struggle to follow complex edit instructions accurately and frequently compromise fidelity by altering key elements of the original image.
Simultaneously, video generation has made remarkable strides, with models that effectively function as consistent and continuous world simulators.
In this paper, we propose merging these two fields by utilizing image-to-video models for image editing.
We reformulate image editing as a temporal process, using pretrained video models to create smooth transitions from the original image to the desired edit.
This approach traverses the image manifold continuously, ensuring consistent edits while preserving the original image's key aspects.
Our approach achieves state-of-the-art results on text-based image editing, demonstrating significant improvements in both edit accuracy and image preservation.
Visit our \href{https://rotsteinnoam.github.io/Frame2Frame}{project page}.
\vspace{-10pt}
\begin{figure}
\centering
\includegraphics[width=0.98\linewidth]{figures/teaser_v2.pdf}
\vspace{-6pt}
\caption{\textbf{Visualization of Frame2Frame's editing process.}
Temporal progression of our video-based approach.
Starting from the source image (leftmost), frames illustrate the natural evolution toward the target edit (rightmost).
Our method produces temporally coherent intermediate states while preserving fidelity to both the source content and the editing intent.
}
% \vspace{-3pt}
% \roy{small: add "time" caption next to the time axis.}
% \noam{I think its better like this so they don't understand what's are doing in the first second}
\label{fig:teaser}
\vspace{-0.46cm}
\end{figure}
\end{abstract}
\section{Introduction}
% Each paragraph:
% - About editing- (two different paragraphs)- talk about limitation- its not robust and does not work good 20%
% - Video generation- mention its world simulation 15%
% - in this paper we merge this two subjects- a new thing + explanation how we do it. Why this approach answers the editing limitations. 40%
% - we add methods for frame selection and prompt writing + experiments and results. 15 %
% - main contributions. 10%
% * light that video generators can serve for editing
% * technical + sota results
%Image editing has witnessed remarkable advancements through deep learning and text-guided diffusion networks.
%These developments have set a new benchmark for image manipulations, enhancing both control and quality.
%However, current approaches still face challenges in real-world scenarios, often struggling to achieve precise edits while maintaining fidelity to the original image content.
%\db{Is it one or two limitations?}
%\roy{I agree, maybe change "while" to "as well as" or "and"}
%Text-guided diffusion models begin with random noise that is iteratively refined to produce a natural image aligned with the target text prompt.
%Leading editing methods follow a similar process but add the input image as an additional condition.
%While conditioning is applied in various forms across methods \noam{cite}, all approaches ultimately generate a single natural edited image. \roy{I don't really understand the context and value of this sentence, what's between image conditioning and generating a single natural image? Why wouldn't I get a single image at any case? It should be clarified or rephrased.}
%\noam{check this is true, for example, instruct pix2pix does have images as the number of denoised ones, inversion}.
% In this paper, we propose an alternative approach by leveraging video generation, a rapidly advancing field with demonstrated success. \noam{add cites} \cite{videoworldsimulators2024, BarTal2024LumiereAS, yang2023lvdm}
% Instead of producing a single edited image, we generate a video sequence that begins with the original image and progresses through frames depicting the desired edit.
% \roy{I would suggest to rewrite it as follows - "
%In this paper, we introduce a paradigm shift in image editing by harnessing the power of video generation, a field that has recently achieved breakthrough results in temporal coherence and visual quality. Instead of attempting to generate a single edited image, an approach that has proven challenging for maintaining global consistency, we generate a video sequence that begins with the source image and evolves naturally through time to achieve the desired edit. This temporal approach allows us to leverage the sophisticated world understanding embedded in video models to guide the editing process.
%Trained on large-scale data, video generators act as powerful world simulators, allowing the sequence to represent a plausible scenario where the edit unfolds naturally over time.
%We argue that by generating a realistic sequence from the source to the target image, our method leads to a more accurate and consistent transformation compared to "single image" generation. \roy{This paragraph should precede the previous paragraph. As I see it, in this paragraph we introduce the strengths of video-models, and in the previous paragraph, we introduce how we exploit it in our paper.}
% From a geometrical perspective, conventional editing approaches involve projecting noise onto the natural image manifold, aiming to "hit" a single point on this manifold where the image remains consistent with both the source and the edit request.
% From a geometrical perspective, conventional editing approaches involve extracting the image from the plausible image manifold by adding noise to it, then aiming to "hit" a single point on this manifold where the image remains consistent with both the source and the edit request.
Image editing has witnessed remarkable advancements through deep learning and text-guided diffusion networks.
These developments have set a new benchmark for image manipulations, enhancing both control and quality.
% However, current approaches still face challenges in real-world scenarios, often struggling to achieve precise edits while \roy{the "While" is not accurate IMHO. It should be something in the fashion of "and/or"} maintaining fidelity to the original image content.
However, current approaches continue to face significant limitations in real-world scenarios. These methods often struggle with two key challenges: achieving precise edits that accurately reflect the intended modifications, and preserving the essential characteristics of the original image content.
% \noam{add examples and a ref to the figure.}
% Leading image editing methods typically rely on text-guided diffusion models that generate images aligned with edit prompts, while being conditioned on the original image in various ways.
% Leading image editing methods typically rely on text-guided diffusion models that generate images aligned with edit prompts, conditioned on the original image through techniques like latent space inversion \cite{} or model fine-tuning \cite{}.
% \noam{add cites}
State-of-the-art techniques predominantly rely on text-guided diffusion models, which iteratively denoise random latent representations to generate edited images.
Such methods condition the generation process on both the source image—using techniques such as latent inversion \cite{mokady2023null} or model fine-tuning \cite{kawar2023imagic}—and the target edit description.
However, these approaches require the model to generate a single output image that preserves source image fidelity while implementing complex edits, often compromising edit accuracy and content preservation.
In this paper, we propose a paradigm shift in image editing by reformulating it as a video generation task.
Rather than a single-state transition, our approach harnesses temporal coherence: the source image serves as the initial frame of a video that progressively and naturally transforms toward the target edit.
This temporal evolution allows the editing process to unfold through physically plausible intermediate states, providing a continuous path between source and target images, as illustrated in \Cref{fig:teaser} and \Cref{fig:f2f_videos} in the appendix.
This temporal approach leverages the sophisticated world understanding embedded in recent video generation models, which have achieved breakthrough results in temporal coherence and visual quality through training on large-scale internet data \cite{yang2024cogvideox, blattmann2023stable}.
From a geometrical perspective, conventional editing approaches project initial noise onto the natural image manifold, targeting a single point where the image aligns with both the source and the edit request.
In contrast, our approach generates a continuous path along the manifold between the original and edited image, producing a smooth realistic transition across different image states, as thoroughly discussed in \Cref{sec:manifold}.
% \noam{after finishing experiment maybe more information added here}
% We illustrate how the proposed editing method progresses along a continuous path within a reduced-dimensional representation of the CLIP latent space.
% \noam{after finishing experiment maybe more information added here}
% \roy{it is still not clear what's the motivation behind this paragraph. if we give a geometric interpretation, we should also mentioned why and how it help us to get better results. I still think this section should come up last, after we've briefly explained what we're doing. I even think it should not appear in the intro, but in its relevant section down below.}
% We implement the proposed paradigm through a structured pipeline named \textbf{Frame2Frame}.
% We first transform the edit instruction into a \textit{Temporal Editing Caption} - a scenario that guides video generation by specifying how the edit should naturally unfold over time using a pretrained Vision-Language Model (VLM).
% A state-of-the-art image-to-video model then generates a temporally coherent transformation across frames, starting from the initial image to edit and following the edit path \roy{I personally don't like this text "starting from the initial image to edit and following the edit path" it is a bit awkward} dictated by the \textit{Temporal Editing caption} \roy{We use "Temporal Editing caption" in italics twice. We should use an acronym here instead, like TEC}.
% In the final stage, we identify the frame that best represents the desired edit with a pretrained VLM.
We implement the proposed approach through a structured pipeline called \textbf{Frame2Frame} (F2F). First, we transform the edit instruction into a \textit{Temporal Editing Caption} — a scenario describing how the edit should naturally evolve over time—using a pretrained Vision-Language Model (VLM). Next, a state-of-the-art image-to-video model generates a temporally coherent sequence guided by the temporal caption.
Finally, we identify the frame that best realizes the desired edit with the assistance of a VLM.
Extensive experiments demonstrate improvements over existing image-to-image approaches.
We evaluate on TEdBench \cite{kawar2023imagic} and PosEdit, a newly curated dataset derived from UTD-MHAD \cite{chen2015utd}, which focuses on human pose transformations.
PosEdit pairs source images with ground-truth targets of the same subject in different poses, enabling rigorous evaluation of both edit accuracy and source fidelity.
Beyond commonly defined editing tasks, our framework shows promising results in more classical computer vision problems such as de-blurring, de-noising, and relighting by recasting them as temporal progressions, suggesting broader applications for video-based image transformations.
% To summarize, our main contributions include:
% %\begin{enumerate}
% % \item Reformulating image editing as a video generation task, creating a world-simulated edit path on the image manifold.
% % \item Introducing Frame2Frame, a framework capable of executing the approach.
% % \item Achieve state-of-the-art performance in image editing on the TEdBench benchmark, with encouraging preliminary results for additional vision tasks.
% % \noam{}
% %\end{enumerate}
% \begin{enumerate}
% \item A novel formulation of image editing as a video generation task, creating world-simulated edit paths on the image manifold that maintain both edit accuracy and image fidelity.
% \item Frame2Frame, a practical framework that implements our approach through temporal editing captions, state-of-the-art video generation, and a frame selection technique.
% \item State-of-the-art performance on various benchmark including TEdBench and EditMotion, a novel editing dataset focusing on human motions.
% \end{enumerate}
% \vspace{-2pt}
\textbf{Our main contributions include}:
\begin{enumerate}
\item Reformulating image editing as a generative video task--leveraging temporal coherence to create edit paths on the natural image manifold, enabling high-fidelity manipulations while preserving source characteristics.
\item Frame2Frame: an end-to-end framework that realizes the reformulation through three key components: (1) temporal editing captions, (2) generated video-based editing, and (3) automated frame selection.
\item Comprehensive evaluation showing state-of-the-art performance on TEdBench and PosEdit, a new dataset for evaluating human pose edits.
\end{enumerate}
% \begin{enumerate}
% \item A novel paradigm that reformulates image editing as a video generation task, leveraging temporal coherence to create physically plausible edit paths on the natural image manifold. This approach enables high-fidelity transformations while preserving source image characteristics.
% \item Frame2Frame: an end-to-end framework that realizes this paradigm through three key components: (1) automatic generation of temporal editing captions, (2) video-based edit synthesis using a state-of-the-art generative model, and (3) vision-language guided frame selection.
% \item Comprehensive empirical validation demonstrating state-of-the-art performance on TEdBench and PosEdit — a new dataset we introduce for evaluating human pose transformations.
% \end{enumerate}
% \item Extensive empirical analysis of edit paths in latent space, providing insights into the advantages of temporal editing over direct image-to-image translation.
%\roy{This is not enough. Yes, we do implicitly create a continuous path on the image manifold, but eventually we just take the last frame. So what makes us different? How do we make sure that the last frame is aligned with the image and prompt input conditions? (as we do state it as a positive trait of image2image models). How do we benefit from the path we take? We have to explain it clearly. Currently, the text just states an observation with no clear practical/theoretical justification.}
% We demonstrate both that edits are performed along a continuous path within the image manifold \noam{refer to section} and that this approach achieves superior editing results with greater fidelity to the source image’s attributes compared to previous methods \noam{refer to section}.
% In contrast, our approach, using the first frame as a fidelity anchor, generates a continuous path on the manifold, representing a realistic sequence of images that transitions smoothly between different states.
% We demonstrate both that edits are performed along a continuous path within the image manifold \noam{refer to section} and that this approach achieves superior editing results with greater fidelity to the source image’s attributes compared to previous methods \noam{refer to section}.
%To accommodate video sequence generation rather than a single target image, we reformulate the conventional edit target caption into a "temporal editing caption"—a video prompt that defines a coherent progression along the desired edit path. \roy{This part might give a relevant answer to my previous comment. I think that the geometric interpretation paragraph should come last, after we've introduced the technicalities of our method. It will be easier to concisely "justify" the geometric motivation.}
%\roy{Not sure about it. Eventually, we just take the last frame. I think that we should write it as it really is, and also mention that we support integration with VLMs in order to select the best target frame among all candidates.}
%In contrast also demonstrate that our entire pipeline can be executed in a fully automated manner. \roy{We should clarify what "automated" means in this context}
% \noam{new dataset? say TEdBench?}
\begin{figure*}
\centering
\includegraphics[width=0.73\textwidth]{figures/manifold_v9.pdf}
\vspace{-7pt}
\caption{\textbf{Editing Manifold Pathway.}
Given an input image and target caption "A happy person making a heart shape with their hands", our method generates a continuous path on the natural image manifold.
Each generated frame (indicated by black arrows) represents a plausible intermediate state between the source and target, maintaining temporal consistency throughout the transformation.
As a result, in contrast to the competing approach, F2F achieves the desired edit while preserving the "AI" text on the person’s shirt.
}
\vspace{-8pt}
\label{fig:manifold_walk}
\end{figure*}
% However, current approaches face persistent challenges in real-world scenarios, frequently failing to achieve the desired edits or struggling to maintain fidelity to the original image content.
% These limitations manifest particularly in complex edits requiring a semantic understanding of scene composition, lighting, and object interactions.
% Recent works \cite{} attribute these shortcomings to a fundamental limitation: the scarcity of large-scale, high-quality image editing datasets, which forces methods to rely on unsupervised learning.
% \noam{when we have results we need to rewrite}
% Video generation has recently emerged as a powerful paradigm for visual synthesis, with state-of-the-art models like Sora effectively functioning as general-purpose world simulators.
% These models leverage massive collections of video data to learn complex temporal dynamics and physical interactions, enabling the generation of highly realistic and temporally consistent sequences that capture intricate real-world behaviors and transformations.
% In this paper, we present a novel approach that bridges image editing and video generation by leveraging image-to-video models for zero-shot image manipulation. Our key insight is that video generation models, trained on vast amounts of real-world data, inherently capture the manifold of natural image transformations.
% We exploit this through a carefully designed pipeline that: (1) utilizes Large Language Models to translate editing instructions into video scenarios, (2) employs image-to-video generation to simulate the edit as a temporal transformation, and (3) leverages a Vision-Language Models to extract the optimal edited frame.
% \noam{I am not sure this steps should be in this place, they are more technical steps, before we established why our approach is awesome}
% This approach naturally addresses the limitations of traditional editing methods by grounding transformations in learned real-world dynamics, while our frame selection mechanism ensures high fidelity to both the original content and editing instructions.
% Our extensive experiments demonstrate significant improvements over existing approaches across various editing tasks, both in terms of edit accuracy and image fidelity. Our main contributions include:
% \begin{enumerate}
% \item A novel paradigm that reimagines image editing through the lens of video generation, addressing fundamental limitations of current approaches.
% \item A robust framework combining that achieves state-of-the-art performance on image editing benchmarks without task-specific training.
% \end{enumerate}
%Image editing has attracted considerable interest in recent years. Remarkable advancements in deep learning, and more recently in text-guided diffusion networks, have fueled the development of promising editing methods that demonstrate improved quality and realism. However, current leading methods still suffer from significant drawbacks: (i) they frequently fail in real-world scenarios, and (ii) they struggle to balance fidelity to the original image with the implementation of the editing instructions. Figure \ref{} illustrate those limitations.
%Recent research \cite{} has attributed these limitations to two fundamental issues. First, the lack of large, high-quality datasets of image editing examples restricts methods to unsupervised regimes. Second, there is an absence of reliable metrics for training and evaluation that align well with both the given editing instructions and the original image.
%In parallel to the advancements in image editing, video generation has recently made extraordinary progress, with state-of-the-art models now producing highly realistic and temporally consistent video sequences. These models ֿ\noam{cite} leverage massive collections of online video data \noam{not sure} to effectively act as sophisticated real-world simulators \cite{}.
%However, despite these impressive capabilities, the potential of video generation techniques to enhance image editing remains largely unexplored.
%\noam{I don't think we should call this "unexplored", this sounds like it already exists and not explored, we point it before anybody else!}
%In this paper, we propose to leverage image-to-video models for zero-shot \noam{zero shot in the video model not here} image editing. Our approach first uses a Large Multimodal Model (e.g., GPT-4o) to generate a scenario from the input image and editing instruction. Subsequently, an image-to-video model is used to generate a short video sequence of the scenario. A Vision-Language Model finally selects the most suitable frame from this edited sequence.
%Relying on video generation models trained on massive amounts of data, our approach addresses the lack of training data in image editing that leads to frequent failures in real-world scenarios. [Moreover] [Furthermore], by leveraging the temporal consistency inherent in video generation and utilizing VLM-based frame selection, our method naturally maintains fidelity to the original image content. Extensive quantitative and qualitative experiments demonstrate the benefits of the proposed approach over existing editing-methods. To summarize our main contributions are:
% \vspace{-2pt}
\section{Related Efforts}
% \vspace{-2pt}
\subsection{Image Editing}
% \vspace{-2pt}
Text-based image editing has advanced significantly with the success of generative diffusion models \cite{ho2020denoising}.
These models, in their text-to-image version, generate images through a denoising diffusion process conditioned on input text, relying on ground truth text-image pairs for training \cite{saharia2022photorealistic, rombach2022high}.
In contrast, image editing often lacks predefined ground truth data for source and target images, posing unique challenges.
This limitation has led researchers to explore diverse editing methodologies \cite{huang2024diffusion}.
For example, SDEdit \cite{meng2021sdedit} injects noise into an image and then denoises it based on an editing target prompt.
Imagic \cite{kawar2023imagic} fine-tunes a text-to-image model on a single image, subsequently interpolating between input and target text embeddings to produce edits.
% \noam{a must add:
% Say imagic is a bit similar maybe? But its not a real path but just an interpolation with a known last point. Here all the path is generated simultaneously.
% }
Other methods first invert the input image into a diffusion model’s latent space \cite{mokady2023null, Huberman-Spiegelglas_2024_CVPR} and then generate the edited image from that latent representation using various techniques for structure preservation and manipulation \cite{hertz2022prompt, parmar2023zero}.
InstructPix2Pix \cite{brooks2023instructpix2pix} synthesizes an editing dataset using the approach in \cite{hertz2022prompt}, filters it, and employs this dataset to train a diffusion model in a supervised fashion.
Paint-by-Inpaint \cite{wasserman2024paint} further explores this supervised approach, generating a real-image dataset for object insertion.
% \noam{add a bit about ti instructions vs prompts}
\subsection{Generative Video Models}
Recent years have seen remarkable advancements in video generation, evolving from domain-specific systems \cite{gao2023high} and brief clips to models capable of generating diverse, high-fidelity content.
This progress has stemmed from paradigm shifts and the large-scale expansion of datasets \cite{chen2024panda} and architectures.
Approaches have evolved from recurrent networks \cite{ha2018world, srivastava2015unsupervised} and generative adversarial networks (GANs) \cite{vondrick2016generating, tulyakov2018mocogan, clark2019adversarial} to latent diffusion models (LDMs) \cite{blattmann2023align, blattmann2023stable, danier2024ldmvfi}, which leverage large U-Net or transformer-based architectures alongside vast internet-sourced datasets.
Notable recent efforts include Stable Video Diffusion \cite{blattmann2023stable}, which trains an LDM on large curated datasets, and OpenAI's Sora \cite{liu2024sora},
which achieves impressive results by integrating large architectures with extensive public and private datasets.
These models, often termed "world simulators" due to their emergent understanding of physical dynamics and temporal coherence.
Our work builds upon CogVideoX \cite{yang2024cogvideox}, a transformer-based latent diffusion model that employs a 3D Variational Autoencoder to compress videos across both spatial and temporal dimensions, enhancing coherence.
To improve text-video alignment, CogVideoX also integrates an expert transformer with adaptive LayerNorm, enabling deep fusion of visual and textual modalities.
Close to our work, several recent efforts have utilized the world simulation capabilities of video diffusion models for various computer vision tasks.
Make-A-Video3D \cite{singer2023text} temporally extends static NeRFs using Score Distillation Sampling from video models.
ViVid-1-to-3 \cite{kwak2024vivid} generates images along a camera trajectory around objects to enable novel view synthesis.
PhysDreamer \cite{zhang2025physdreamer} models rigid object properties through 3D Gaussians and material fields, trained via distillation from pretrained video generators.
\subsection{Image Editing and Video}
\vspace{-3pt}
The intersection of image editing and video has received limited attention.
Existing methods focus on sampling pairs of random frames from videos to build image pair datasets that capture the same subject under varying conditions.
For example, AnyDoor \cite{chen2024anydoor} uses the paired frames as an augmentation method, segmenting foreground objects in each frame and assigning one masked object as the target edited appearance of the subject.
MagicFixup \cite{alzayer2024magic} employs these frames to build a dataset focused on refining user-made subject coarse 2D edits.
Recently, drag-based editing approaches \cite{shi2024instadrag, luo2024readout}, used frames extracted from video and computed optical flow to collect a dataset aimed at editing by spatially dragging points within the image.
In contrast to these approaches, which rely on frame sampling for image data collection, we are the first to directly perform image editing using generative video models.
\vspace{-3pt}
\section{Frame2Frame}
\vspace{-4pt}
% Our approach reformulates text-based image editing as a temporal transformation process, introducing a novel generative framework termed Frame2Frame.
% This framework leverages image-to-video generation models to achieve consistent and realistic image transformations.
% % Text-based image editing usually uses a source image $I_s$ and a target caption $c$, which is supposed to describe the image after editing.
% Our method can be divided into three different steps, as exhibited in \Cref{fig:manifold_image}.
We present Frame2Frame, a framework that reformulates image editing as a temporal transformation process. Our approach leverages video generation models to create natural transitions between source and target images, achieving consistent and realistic edits. The proposed method has three main steps, as illustrated in \Cref{fig:method_image}.
% Traditional methods \cite{TODO} often fail to preserve key aspects of the source image, are limited in editing capabilities, or produce undesirable results. By embedding the editing process within a temporal framework, we utilize video generation to simulate gradual changes that maintain the original image's integrity.
% Text-based image editing usually uses a source image $I_s$ and a target caption $c$, which is supposed to describe the image after editing.
% we transform these inputs into a temporal sequence $V$ that guides the video generation process. The core components of our method are: \begin{enumerate}
% \item transforming the editing prompt $c$ into a coherent temporal editing caption $\tilde{c}$.
% \item generating the video sequence $V$ using an image-to-video model
% \item Selecting the optimal frame that best realizes the intended modification.
% \end{enumerate}
% Formally, let $V = \{f_1, ..., f_T\}$ be the generated video sequence of $T$ frames, where $f_1$ corresponds to the source image $I_s$.
% Our goal is to find a frame $f_t \in V$ that best satisfies the editing instruction while preserving the contextual integrity of the original image:
% \begin{equation}
% f_{opt} = \arg\max_{f_t \in V} \mathcal{Q}(f_t, I_s, e)
% \end{equation}
% where $\mathcal{Q}$ represents our quality metric based on semantic similarity. While a baseline approach would simply select the last frame $f_T$, we employ a more sophisticated frame selection mechanism using CLIP scores to evaluate the alignment between each frame and the editing instruction. This framework circumvents common limitations of existing image editing techniques by leveraging the temporal coherence inherent in video generation models, resulting in more consistent and realistic transformations.
% \vspace{-2pt}
\vspace{-2pt}
\subsection{Temporal Editing Captions}\label{temporal_captions}
\label{sec:tec}
\vspace{-2pt}
% Modern image editing models perform edits on an input image $I_s$ using either an editing instruction $c$ \cite{brooks2023instructpix2pix, wasserman2024paint} or a target image prompt $p$ \cite{mokady2023null, kawar2023imagic}.
Text-based image editing methods typically operate on two inputs: a source image $I_s$ and a target caption $c$, where $c$ specifies the desired modifications to $I_s$.
Our approach differs fundamentally by modeling editing as a temporal process. This requires a novel type of prompt—the \textit{Temporal Editing Caption}, denoted by $\tilde{c}$—that describes the sequential transformation from source to target image.
We construct $\tilde{c}$ by combining information from $I_s$ and $c$ to create a description of how the desired edit unfolds over time.
To automate this process, we leverage recent advances in Vision-Language Models (VLMs) \cite{dai2023instructblipgeneralpurposevisionlanguagemodels, wang2024cogvlmvisualexpertpretrained, wang2023cogvlm, ganz2024question, rotstein2024fusecap}. The VLM, specifically ChatGPT-4o \cite{openai2024chatgpt4o}, is instructed to produce a concise video scenario that highlights how elements within the image change or move over time. The generated caption captures the essential transformations while maintaining a static camera perspective unless movement is necessary.
To improve generation quality, we utilize in-context learning (ICL) \cite{dong2022survey}, providing the VLM with nine exemplar prompt-caption pairs.
The complete prompt template, ICL examples, and an ablation study comparing this approach with directly using the target captions are included in Section \ref{supp:temporal_captions} of the Appendix.
% Text-based image editing methods typically operate on two inputs: a source image $I_s$ and a target caption $c$, where $c$ specifies the desired modifications to $I_s$.
% Our approach differs fundamentally by modeling editing as a temporal process. This shift requires a novel type of prompt, which we term the \textit{Temporal Editing Caption}, denoted by $\tilde{c}$, that describes the sequential transformation from source to target image.
% We derive $\tilde{c}$ by combining information from $I_s$, which establishes the initial image state, and $c$, which describes the requested edit, and use it to create a description of a scenario in which the desired edit occurs over time.
% To automate this process, we leverage recent advances in Vision-Language Models (VLMs) \cite{dai2023instructblipgeneralpurposevisionlanguagemodels, wang2024cogvlmvisualexpertpretrained, wang2023cogvlm, ganz2024question, rotstein2024fusecap}, generating $\tilde{c}$ by prompting a VLM with both $I_s$ and $p$.
% The VLM, e.g. ChatGPT-4o \cite{openai2024chatgpt4o}, is instructed to produce a concise video caption that highlights how elements within the image change or move over time. \db{I don't like the sentence. I understand from it that we give to the VLM a video}
% The caption should emphasize significant transformations while omitting non-essential details or unrelated elements, and assume a static camera perspective unless a change is necessary.
% We further employ in-context learning \cite{dong2022survey}, providing the VLM with nine examples of such prompt-caption pairs. The complete prompt and examples can be found in the Appendix.
% In Section \noam{???} we evaluate the impact of using $\tilde{c}$ compared to the conventional $c$.
% \subsection{Video Generation}
% To generate the video, we employ CogVideoX \cite{yang2024cogvideox}, a pretrained open-source latent diffusion model with a transformer-based architecture.
% We use an image-to-video variant, a fine-tuned adaptation of a text-to-video generator, conditioned on the input image $I_s$, which is encoded and concatenated with noise in latent space.
% During training, the model learns to generate video sequences that start from an input-conditioned image.
% Leveraging this, during inference, the model generates videos that begin with $I_s$ and continue within the image manifold, producing realistic and coherent visual transitions.
% Formally, given a video generator $G$, where \textit{V} denotes the generated video sequence and $T$ represents the total number of frames, and where $f_t$ represents the individual frames generated at each timestep $t$, we define the process as follows:
% \begin{eqnarray}
% \text{G}(I_s, \tilde{c}) = V = \{f_1, \dots, f_T\}.
% \end{eqnarray}
\subsection{Video Generation}
We employ CogVideoX (I2V-5B) \cite{yang2024cogvideox}, a pretrained generative video latent diffusion model utilizing a transformer-based architecture.
Specifically, we use its image-to-video variant, which has been fine-tuned to generate videos starting from an input image $I_s$.
During generation, $I_s$ is encoded and concatenated with noise in latent space, where the model applies a denoising process guided by the temporal caption $\tilde{c}$.
As elaborated in \Cref{sec:manifold}, this conditioning allows generated videos to start from $I_s$ and evolve naturally along the image manifold, maintaining temporal coherence and consistency.
Additionally, the model's transformer architecture enables effective fusion of visual and textual information, allowing precise control over the editing process through our temporal captions. Formally, given the video generator $G$, we define the generation process as:
\vspace{-3pt}
\begin{eqnarray*}
\text{G}(I_s, \tilde{c}) = V = \{f_1, \dots, f_T\}
\end{eqnarray*}
\vspace{-3pt}
where $V$ denotes the generated video with $T$ frames, and $f_t$ represents the frame at timestep $t$.
\subsection{Frame Selection}\label{sec:frame_selection}
We observed that the optimal number of frames required for an edit can vary—small changes may be completed in fewer frames, while more extensive transformations often necessitate additional ones.
Additionally, later frames tend to deviate further from the source image.
Thus, even though $V$ serves as an editing path originating from $I_s$, there is no guarantee that $f_T$ is the optimal edited image in $V$.
For instance, in \Cref{fig:method_image}, the edited image of the cat might be expected to capture the midpoint of its jump, but the last frame shows the jump already completed.
Therefore, we aim to identify the optimal edited frame, denoted $f_{t^*}$, which corresponds to the earliest timestep $t$ that achieves the desired edit.
The transition from the initial frame $f_1$ (or $I_s$) to the final edited frame $f_{t^*}$ motivates our method's name, \textit{Frame2Frame}.
To automate the selection of $t^*$ and avoid manual frame-by-frame review, we employ an automated approach.
After generating the sequence $V$, we sample every fourth frame, imprinting each with a unique identifier and assembling them into an image collage alongside $I_s$.
Inspired by \cite{kim2024image}, which introduces a novel approach to video comprehension by transforming videos into image grids, we use a VLM, specifically GPT-4o, to assist in selecting $t^*$ by providing it with the collage and the editing prompt $c$.
The VLM is tasked with identifying the frame that best fulfills the editing intent, evaluating each frame’s alignment with $c$ and fidelity to $I_s$.
The model is instructed to select the optimal frame with the lowest index that completes the edit.
% An ablation study comparing this automated approach with simply selecting the last frame of the video is provided in the appendix.
An ablation study, detailed in Section \ref{supp:frame_seletion} of the appendix, evaluates the effectiveness of our automated approach against the baseline of selecting the final frame of the video.
Note that while we refer to an optimal $t^*$, the definition of the required edit can be subjective and dependent on user preferences.
For example, in \Cref{fig:method_image}, different users may select frames of the cat jumping based on variations in its altitude.
Thus, frame selection could serve as a flexible and customizable advantage in certain scenarios.
% ֿ\noam{add that we check this-
% e evaluate the effectiveness of this automated process}
% Our frame selection prompt examples are provided in the supplementary materials.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/method.pdf}
\vspace{-8pt}
\caption{\textbf{Frame2Frame Overview}. Given a source image and editing prompt, our pipeline proceeds in three steps. First, a Vision-Language Model generates a temporal caption describing the transformation. Next, this caption guides a video generator to create a natural progression of the edit. Finally, our frame selection strategy identifies the optimal frame that best realizes the desired edit, producing the final image of the cat mid-leap.
% \noam{tmp image} \roy{Consider adding an LLM block.}
}
\vspace{-8pt}
\label{fig:method_image}
\end{figure}
% ultimately yielding an optimal frame that captures the edited transformation in a cohesive and contextually accurate manner.
% Silver and Gal:
% \noam{maybe add related papers + talk about it being an advantage}
% % A critical component of our image-to-video-to-image pipeline is the selection of the optimal frame that best represents the desired edit.
% We develop a robust frame selection mechanism that leverages CLIP embeddings \cite{radford2021learning} and incorporates multiple augmentation strategies to ensure reliable results.
% Following a similar approach as \cite{parmar2023zero} \noam{they do it only for text}, to enhance the robustness of this selection process, we implement two key augmentation strategies:
% \begin{enumerate}
% \item \textbf{Text Augmentation:} We utilize a large language model \cite{brown2020language} to generate M semantically equivalent variations of both source and target descriptions, preserving meaning while varying linguistic structure and phrasing. We then take the average CLIP embedding of those augmentations $E_{text}$, This helps mitigate the sensitivity of CLIP to specific formulations \cite{gao2021clip}.
% \item \textbf{Frame Augmentation:} Each video frame undergoes K distinct augmentations, including crops, flips, and color variations, inspired by data augmentation techniques shown to improve robustness in vision-language models \cite{radford2021learning, zhou2021augmentations}. We then take the average CLIP embedding of those augmentations $E_{frame}$, This approach helps ensure that the selection process is robust to minor visual variations.
% \end{enumerate}
% Our frame selection criterion is based on measuring the semantic alignment between the editing instruction and the video frames. Specifically, for each frame $f_t$ in the generated video sequence, we compute a similarity score:
% \begin{equation}
% S(f_t) = \cos(\Delta_{text}, \Delta_{frame})
% \end{equation}
% where $\Delta_{text} = E_{text\_source} - E_{text\_target}$ represents the difference between source and target text embeddings, and $\Delta_{frame} = E_{source\_image} - E_{curr\_frame}$ captures the visual transformation from the source image to the current frame. In cases where the source description does not exist, we create one using a VLM.
% Additionally, we incorporate a temporal regularization term that slightly biases the selection toward later frames in the sequence, where the edit has had more time to stabilize.
% % @TODO: need to write exactly which regularization equation we used, currently we use f(t)=exp(1.2*t/num_of_frames) s.t. t is the index of the current frame.
% Temporal regularization techniques, often used in video processing, help avoid intermediate states that may only partially capture the intended transformation \cite{huang2021temporally}.
% ֿ\noam{maybe we should perform an experiment checking this works}
% \\\
\section{Editing Manifold Pathway}\label{sec:manifold}
To illustrate the advantages of our method over conventional image-to-image approaches, we aim to visualize the editing process within the natural image manifold, where realistic images reside.
Given the high dimensionality of this space, we project it into a lower-dimensional representation suitable for interpretation.
To achieve this, we first generate three sets of images, each containing 200 samples, using the text-to-image generator FLUX.1-dev \cite{flux_model}.
The sets are based on the following prompts:
\begin{enumerate}
\item \textbf{AI}: \textit{``Full-body portrait of a happy person wearing a shirt with the word 'AI' on it''}.
\item \textbf{AI + Heart Hands}: \textit{``Full-body portrait of a happy person wearing a shirt with the word 'AI' on it and making a heart shape with their hands''}.
\item \textbf{Heart Hands}: \textit{``Full-body portrait of a happy person making a heart shape with their hands''}.
\end{enumerate}
We manually filter the outcomes to ensure alignment with descriptions, with examples provided in the supplementary material in Section \ref{supp:manifold}.
Then, we use a CLIP ViT-B/32 model \cite{radford2021learning} to extract a 512-dimensional feature vector for each image.
Using these features, along with 25 feature vectors extracted from random noise images, we perform Principal Component Analysis (PCA) to reduce the dimensionality into a two-dimensional subspace.
The resulting subspace, now suitable for visualization, is depicted in~\Cref{fig:manifold_walk}.
As illustrated in the figure, the natural images form a smooth manifold, distinctly separated from the distribution of noise images.
Within this manifold, there is a clear semantic progression: images of people with 'AI' shirts (green cluster) are close to images of people with 'AI' shirts making a heart shape (purple cluster), which are adjacent to images of people only making a heart shape (red cluster).
Thus, transitioning smoothly along the manifold allows a person with an 'AI' shirt to perform a heart shape with their hands while preserving the shirt's text.
Consider an original image from the AI group that we wish to edit using the target prompt: ``A happy person making a heart shape with their hands''.
Current editing methods generate a single image, which may cause an abrupt transition to the red cluster, effectively removing the 'AI' on the shirt.
This behavior is illustrated in the figure, where the edit by LEDITS++ is positioned near the red cluster, and the 'AI' text on the shirt disappears.
In contrast, our method leverages video generation to perform the edit smoothly, moving along the manifold with incremental changes until reaching the required edit in the purple cluster, as indicated by black arrows in \Cref{fig:manifold_walk}.
The temporal editing caption guiding this process is: ``A happy person very slowly raising their hands to form a heart shape''.
This gradual progression lets us reach the purple cluster, resulting in an edited image where the person maintains the `AI' on their shirt while making a heart shape—a faithful preservation of the original image's key attributes.
This experiment illustrates that the proposed paradigm enables smooth traversal across the image manifold, enabling consistent edits while preserving the essential characteristics of the original image.
% By approximating the geodesic path on the manifold through temporal interpolation in the video generation model, each frame in the sequence represents a subtle adjustment aligned with the manifold's structure. This ensures that transitions are realistic and preserve essential image features. By progressing smoothly along this estimated geodesic, our approach avoids common pitfalls in conventional editing, such as unnatural artifacts and the loss of important semantic characteristics, delivering transformations that are cohesive and faithful to the original content.
% In this manifold, images with similar semantic content are located close to each other, exhibiting short geodesic distances. Traditional image editing methods typically operate by directly jumping from the original image to the edited image—effectively moving between two distant points on the manifold. This abrupt transition often leads to edits that sacrifice key features, compromising fidelity to the original image.
% In our framework, we redefine image editing as traversing a continuous path on this manifold by leveraging the temporal consistency of video generation. Here, each frame in the generated video sequence corresponds to a point on the image manifold. Collectively, these frames form a smooth trajectory transitioning from the original to the edited image. We model this trajectory as an approximation of a geodesic path on the manifold, facilitating gradual and coherent transformations that preserve both the semantics and structural coherence of the source image.
% my points for starting
% - manifold of images
% - when it comes to editing methods generate a single image, which basically means they jumpt from one point on the manifold to another
% - closely semantic images should be close and have short geodesic distance between them
% - we leverage this fact in our approach by generating a video which is the editing path
% - to examine and show this we perform the experiment presented in \Cref{fig:manifold_image}.
% then something like this:
% In this experiment we generate three types of images using a text to image generator, using the following primtes
% - AI "Full-body portrait of a happy person wearing an shirt with the word 'AI' on it", green in figure
% - AI -heart hands"Full-body portrait of a happy person wearing an shirt with the word 'AI' on it and making a heart shape with their hands"- purple in figure
% - heart hands- Full body portrait of a happy person making a heart shape with their hands- red in figure.
% we generate 200 of each type which we filter manually to verify they follow the prompt.
% To simulate the manifold, we use a CLIP-b/32, which כםר each image $I$ extracts a $f \in R^512$ vector.
% We extract for each of the groups such that I^ai, I^H and I^(H+AI).
% add equations to this:
% we perform pca to a 2d subspace to the extracted features f^ai, f^H and f^(H+AI) along with 100 features extracted from random noise images.
% As can be seen in the figure the natural images form what appears to be smoothed manifold compared to the noise.
% Moreover it is easy to see that the re is a continuou. semantic meaning where in the manifold people with ai shirts, lie closer to people with shirts doing a heart shape, which lay closer to people only doing heart shape.
% We use this representation to demonstrate how an editing work.
% For example a person with an ai shirt which is supposed to be edited using the target prompt "a happy person making a heart shape with their hands", is supposed to move from the green cluster to the purple cluster because his shirt should not change.
% We see for example in the gprah the results of LEFITS++ for such image, which is in the border between the purple and the red clusters.
% In contrast, our method which applies video generation for editing smoothly performs the edit moving in the manifold with the black arrows, and slowly progressing until the edit.
% In this example we can see how wee manage to land o the purple cluser- and in the image we can see out person maintains both the ai on his shirt and both performs the edit doing a heart shape.
% you can use this information:
% The space of all-natural images can be conceptualized as a high-dimensional manifold, where each point represents a unique image configuration.
% Traditional editing methods typically operate by directly jumping between two points on this manifold: the original image and the target edit. However, this often leads to abrupt transitions that stray far from the origin, resulting in edits that may sacrifice key features and compromise fidelity to the original image.
% In our framework, we redefine image editing as traversing a continuous path on this manifold by leveraging the temporal consistency of video generation. Here, each frame in the generated video sequence corresponds to a point on the image manifold. Collectively, the frames form a smooth trajectory transitioning from the original to the edited image. We model of this trajectory as a geodesic path on the manifold.
% % —an optimal path that balances fidelity to the original while achieving the desired edit, preserving both the semantics and structural coherence of the source image.
% To approximate this geodesic path, we harness the video generation model’s capability to simulate gradual, real-world changes. Each frame within this temporal sequence represents a subtle adjustment aligned with the manifold’s structure, ensuring that transitions are realistic and preserve key image features. By progressing smoothly along this estimated geodesic, our approach avoids common pitfalls in conventional editing, such as unnatural artifacts and the loss of essential image characteristics, delivering transformations that are cohesive and faithful.
% embed the frames into a lower-dimensional representation of the manifold, enabling visualization and assessment of the path's smoothness and alignment with the intended edits.
% highlights how our method aligns with the inherent geometry of the image manifold, resulting in edits that are both visually consistent and semantically accurate.
% \section{Experiments}
% We compare our method with respect to various leading image-editing methods LEDITS++ \cite{leitspp}, SDEdit \cite{sdedit}, Pix2Pix-Zero \cite{pix2pix_zero}, DDPM \cite {ddpm_inversion} and Imagic \cite{kawar2023imagic} on two benchmarks, TEdBench \cite{kawar2023imagic} and PosEdit, a novel benchmark for human movement editing.
% \subsection{TEdBench Quantative}
% We evaluate each method on every image from the TEdBench benchmark, using 9 different seeds (we use the same 9 seeds for all methods), by conditioning the methods on the source image and edit prompt. Note that some methods, like DDPM \cite{ddpm_inversion} and Pix2Pix-Zero \cite{pix2pix_zero} require not only a target prompt, but also a source prompt that the describe the source image. In those case, we use BLIP \cite{li2022blip} for generating the source prompt. Moreover, note that our video generator of choice - CogVideoX \cite{yang2024cogvideox}, works only in a fixed resolution of $720 \times 480$ pixels, for both input images and the generated output video. Therefore, we had to resize and pad source images that do not fit those requirements, as well as crop the chosen video frame in order to remove the padding that was applied for the source image.
\begin{table}
\centering
\begin{tabular}{l|cc|c}
\toprule
& \multicolumn{2}{c|}{\textbf{Source}} & \textbf{Target} \\
\raisebox{1.2em}{Model} &
\raisebox{1.2em}{LPIPS$_\downarrow$} &
\raisebox{1.2em}{CLIP-I$_\uparrow$} &
\raisebox{1.2em}{CLIP$_\uparrow$} \\
\noalign{\vskip -0.6em}
\midrule
SDEdit & 0.30 & 0.85 & 0.60 \\
Pix2Pix-ZERO & 0.29 & 0.84 & 0.62 \\
% \\ EF-DDPM & 22.84 & .204 & .877 & .627 \\
Imagic & 0.52 & 0.86 & \textbf{0.63} \\
LEDITS++ & 0.23 & 0.87 & \textbf{0.63} \\
FlowEdit & \textbf{0.22} & \textbf{0.89} & 0.61 \\
\midrule
% F2F$^A$-Prompts & \textbf{17.30} & .369 & .857 & .622 \\
% F2F$^A$ & ֿ16.63 & \textbf{.329} & .867 & \textbf{.634} \\
% F2F & 14.94 & .345 & \textbf{.875} & .630 \\
% F2F$^A$-Prompts & ? & ? & ? \\
% F2f$^A$-Frames & ? & ? & ? \\
F2F & \textbf{0.22} & \textbf{0.89} & \textbf{0.63} \\
\bottomrule
\end{tabular}
\caption{
\textbf{TEdBench Results.} Quantitative comparison on TEdBench benchmark. Source metrics (LPIPS and CLIP-I) measure content preservation, while Target metric (CLIP) measures edit accuracy.
% Lower LPIPS is better ($\downarrow$), higher CLIP-I and CLIP scores are better ($\uparrow$).
Our Frame2Frame (F2F) method achieves better or comparable performance across all metrics.
% \noam{
% blabla, explain F2F and F2F$^A$, talk about resolution? somewhere explain the EF is edited-friendly. add ledits to related?
% rerun other with more seeds?
% }
}
\label{tbl:tedbench_quant}
\label{tedench_results}
\vspace{-8pt}
\end{table}
\vspace{-3pt}
\section{Experiments}
\vspace{-3pt}
We evaluate Frame2Frame against state-of-the-art image editing methods, including LEdits++ \cite{leitspp}, SDEdit \cite{sdedit}, Pix2Pix-Zero \cite{pix2pix_zero} Imagic \cite{kawar2023imagic}, and FLUX-based FlowEdit \cite{kulikov2024flowedit}.
Our evaluation spans two benchmarks: the established TEdBench \cite{kawar2023imagic} for general image editing, and our newly introduced PosEdit, specifically designed for human pose editing.
Finally, we conduct a human evaluation to assess our method based on real user preferences.
\subsection{Evaluation Protocol}
We conduct our experiments following a consistent evaluation protocol across all methods and benchmarks.
Following common practice \cite{kawar2023imagic, leitspp}, for each method and source image, we manually select the best result from fifteen random seeds based on visual quality and edit accuracy, ensuring the same seed set is used across all methods.
For all methods, we use the default hyperparameters and settings as provided in their official implementations or official Hugging Face repositories
\footnote{
\href{https://github.com/ml-research/ledits_pp}{LEdits++}, \href{https://huggingface.co/docs/diffusers/main/en/api/pipelines/pix2pix_zero}{Pix2Pix-Zero},
\href{https://huggingface.co/docs/diffusers/en/api/pipelines/stable_diffusion/img2img}{SDEdit}
% \href{https://huggingface.co/DongKyung/Edit-friendly-DDPM-inversion/blob/main/DDPM_Inversion.ipynb}{DDPM}
}.
% Our evaluation pipeline processes each source image and editing prompt as follows:
\vspace{-4pt}
\paragraph{Image Preprocessing and Generation Details.}
We use CogVideoX \cite{yang2024cogvideox} as our video generation backbone, which operates at a fixed resolution of $720 \times 480$ pixels. Both TEdBench and PosEdit benchmarks consist of images with 1:1 aspect ratio. To accommodate CogVideoX's resolution requirements while preserving image content, we first resize all source images to $480 \times 480$ pixels, then horizontally pad them with black pixels to reach the required $720 \times 480$ resolution (adding 120 pixels on each side). After generating the video sequence, we crop the central $480 \times 480$ region of the selected frame to remove the padding and resize it to the final evaluation resolution of $512 \times 512$ pixels, matching the standard resolution used in prior work.
For video generation, we adopt the default hyperparameters proposed by CogVideoX: a guidance scale of 6, 49 generated frames per sequence, and 50 denoising inference steps.
This configuration generates videos approximately 6 seconds in duration at a frame rate of 8 frames per second.
% \noam{this is written great but a bit too long}
\begin{table}
\centering
\begin{tabular}{@{}l@{\hspace{3pt}}|@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}|@{\hspace{3pt}}c@{\hspace{3pt}}c@{}c@{}}
\toprule
& \multicolumn{2}{c|@{\hspace{3pt}}}{\textbf{Source}} & \multicolumn{3}{c@{}}{\textbf{Target}} \\
% \cline{2-6}
\raisebox{0.8em}{Model} &
\raisebox{0.8em}{LPIPS$_\downarrow$} &
\raisebox{0.8em}{CLIP-I$_\uparrow$} &
\raisebox{0.8em}{LPIPS$_\downarrow$} &
\raisebox{0.8em}{CLIP-I$_\uparrow$} &
\raisebox{0.8em}{CLIP$_\uparrow$} \\
\noalign{\vskip -0.4em}
\midrule
\midrule
\textcolor{gray}{
GT (Reference)} &
\textcolor{gray}{
0.08} &
\textcolor{gray}{
0.91 }
&
\textcolor{gray}{0} &
\textcolor{gray}{1} &
\textcolor{gray}{0.61} \\
\midrule
SDEdit & 0.39 & 0.61 & 0.39 & 0.64 & 0.57 \\
Pix2Pix-ZERO & 0.39 & 0.57 & 0.40 & 0.60 & 0.56 \\
% EF-DDPM & ? & ? & ? & ? & ? \\
LEDITS++ & 0.26 & 0.65 & 0.28 & 0.69 & \textbf{0.64} \\
\midrule
F2F & \textbf{0.14} & \textbf{0.82} & \textbf{0.15}
&\textbf{0.84} & \textbf{0.64} \\
\bottomrule
% "GT" & 0.08 & 0.91 & 0 & 1 & 0.61 \\
% \bottomrule
\end{tabular}
\caption{
\textbf{PosEdit Results.}
Quantitative evaluation on PosEdit.
Source metrics assess similarity to the original image, while Target metrics include LPIPS and CLIP-I comparisons to the ground-truth target image, along with the CLIP score for edit accuracy.
“GT” provides ground-truth target image metrics for context.
}
\label{tbl:PosEdit_results}
\vspace{-8pt}
\end{table}
\begin{figure*}[!htb]
\centering
\includegraphics[width=0.9\linewidth]{figures/comparisons_v3.pdf}
\vspace{-2pt}
\caption{
\textbf{Qualitative Results on TEdBench.} Comparison with other methods across various editing tasks. Our approach consistently produces edits that better align with the target prompt while preserving the source image's content and structure. For instance, in the teddy bear example, our method uniquely achieves complex structural modifications while maintaining high visual quality.
}
\vspace{-8pt}
\label{fig:tedbench_qualitative}
\end{figure*}
\paragraph{Method-Specific Requirements.}
Pix2Pix-Zero \cite{pix2pix_zero} requires additional source image descriptions along with the standard inputs.
We generate these automatically using BLIP-2 \cite{li2023blip}, a state-of-the-art image captioning model, ensuring consistent source descriptions across all experiments.
For our method, we transform the original editing prompts from both benchmarks into temporal editing captions, as described in \Cref{sec:tec}.
% These captions serve as input prompts to our video generation pipeline, providing temporal guidance for the editing process.
% For Imagic \cite{kawar2023imagic}, we use the evaluation metrics reported in their original paper, as their implementation is not publicly available.
% \noam{add that we use original magic image model results in their git + footnote link}
% \roy{I think it is better to mention it in the caption of the relevant table}
% \noam{lets add how much time it takes maybe? or maybe not?}
\vspace{-3pt}
\subsection{TEdBench Evaluation Results}\label{sec:tedbench_results}
\vspace{-3pt}
We quantitatively evaluate our method and baselines on the TEdBench benchmark using three complementary metrics. For each method, we compute the metrics between the source image and its corresponding best edited version. LPIPS \cite{zhang2018perceptual} measures perceptual similarity to assess how much the edit preserves the source image's content, with lower values indicating better preservation. CLIP-I evaluates the similarity between source and edited images in CLIP's \cite{radford2021learning} feature space, where higher values indicate better preservation of semantic content. Finally, CLIP score measures alignment between the edited image and the target prompt, with higher values indicating better adherence to the editing instruction. As shown in \Cref{tbl:tedbench_quant}, our method achieves strong performance across all metrics, demonstrating effective balance between preserving source content and achieving the desired edit.
The qualitative advantages of our method are visually evident in the comparisons presented in \Cref{fig:tedbench_qualitative}.
The figure demonstrates our method's superior performance across diverse editing scenarios, producing results that both faithfully execute the intended edits while maintaining strong alignment with the source image.
Additional examples showcasing these capabilities are provided in the appendix in Section \ref{supp:more_examples}.
% \roy{add a paragraph about the qualitative evaluation once the figure will be ready}
% ֿ\noam{we haven't discussed our results yet!}
% Maybe experiments on ted bench, they simply choose the best manually!:
% https://arxiv.org/html/2309.10556v2
% https://arxiv.org/html/2311.16711v2
% think how to talk about captions
% remember we used blip
% \begin{table}
% \centering
% \begin{tabular}{l|cc|c}
% \toprule
% & \multicolumn{2}{c|}{\textbf{Source}} & \textbf{Target} \\
% \raisebox{1.2em}{Model} &
% \raisebox{1.2em}{LPIPS$_\downarrow$} &
% \raisebox{1.2em}{CLIP-I$_\uparrow$} &
% \raisebox{1.2em}{CLIP$_\uparrow$} \\
% \noalign{\vskip -0.6em}
% \midrule
% SDEdit & 0.3 & 0.85 & 0.60 \\
% Pix2Pix-ZERO & 0.29 & 0.84 & 0.62 \\
% % \\ EF-DDPM & 22.84 & .204 & .877 & .627 \\
% LEDITS++ & 0.23 & 0.87 & \textbf{0.63} \\
% Imagic & 0.52 & 0.86 & \textbf{0.63} \\
% \midrule
% % F2F$^A$-Prompts & \textbf{17.30} & .369 & .857 & .622 \\
% % F2F$^A$ & ֿ16.63 & \textbf{.329} & .867 & \textbf{.634} \\
% % F2F & 14.94 & .345 & \textbf{.875} & .630 \\
% % F2F$^A$-Prompts & ? & ? & ? \\
% % F2f$^A$-Frames & ? & ? & ? \\
% F2F & \textbf{0.22} & \textbf{0.89} & \textbf{0.63} \\
% \bottomrule
% \end{tabular}
% \caption{
% \textbf{TEdBench Results.} Quantitative comparison on TEdBench benchmark. Source metrics (LPIPS and CLIP-I) measure content preservation, while Target metric (CLIP) measures edit accuracy.
% % Lower LPIPS is better ($\downarrow$), higher CLIP-I and CLIP scores are better ($\uparrow$).
% Our Frame2Frame (F2F) method achieves better or comparable performance across all metrics.
% % \noam{
% % blabla, explain F2F and F2F$^A$, talk about resolution? somewhere explain the EF is edited-friendly. add ledits to related?
% % rerun other with more seeds?
% % }
% }
% \label{tbl:tedbench_quant}
% \label{tedench_results}
% \end{table}
% Average LPIPS Distance: 0.36936812786734663
% Average SSIM Score: 0.5432913222163915
% Average CLIP Image Similarity (Original vs Edited): 0.857294921875
% Average CLIP Similarity (Edited vs Text): 0.622064208984375
% \begin{table}
% \centering
% \begin{tabular}{@{}l@{\hspace{3pt}}|@{\hspace{3pt}}c@{\hspace{3pt}}c@{\hspace{3pt}}|@{\hspace{3pt}}c@{\hspace{3pt}}c@{}c@{}}
% \toprule
% & \multicolumn{2}{c|@{\hspace{3pt}}}{\textbf{Source}} & \multicolumn{3}{c@{}}{\textbf{Target}} \\
% % \cline{2-6}
% \raisebox{0.8em}{Model} &
% \raisebox{0.8em}{LPIPS$_\downarrow$} &
% \raisebox{0.8em}{CLIP-I$_\uparrow$} &
% \raisebox{0.8em}{LPIPS$_\downarrow$} &
% \raisebox{0.8em}{CLIP-I$_\uparrow$} &
% \raisebox{0.8em}{CLIP$_\uparrow$} \\
% \noalign{\vskip -0.4em}
% \midrule
% SDEdit & 0.39 & 0.61 & 0.39 & 0.64 & 0.57 \\
% Pix2Pix-ZERO & 0.39 & 0.57 & 0.40 & 0.60 & 0.56 \\
% % EF-DDPM & ? & ? & ? & ? & ? \\
% LEDITS++ & 0.26 & 0.65 & 0.28 & 0.69 & \textbf{0.64} \\
% \midrule
% F2F & \textbf{0.14} & \textbf{0.82} & \textbf{0.15} &\textbf{0.84} & \textbf{0.64} \\
% \bottomrule
% \end{tabular}
% \caption{
% \textbf{PosEdit Results.}
% Quantitive evaluation on PosEdit. It is similar to TEdBench (Table \ref{tbl:tedbench_quant}), enhanced with additional target metrics made possible by PosEdit’s ground-truth pairs. Source metrics assess similarity to the original image, while Target metrics now include both LPIPS and CLIP-I comparisons to the ground-truth target image, along with the CLIP score for edit accuracy. These additional metrics enable a rigorous assessment of the preservation of key features from the original image throughout the editing process.
% }
% \label{tbl:PosEdit_results}
% \end{table}
% \begin{table}
% \centering
% \begin{tabular}{@{}l|c@{\hspace{10pt}}c|c@{\hspace{10pt}}c@{}}
% \toprule
% & \multicolumn{2}{c|}{\textbf{Edit Accuracy}} & \multicolumn{2}{c}{\textbf{Edit Quality}} \\
% % \cline{2-5}
% \raisebox{0.8em}{Method} &
% \raisebox{0.8em}{Overall} &
% \raisebox{0.8em}{Per-Image} &
% \raisebox{0.8em}{Overall} &
% \raisebox{0.8em}{Per-Image} \\
% \noalign{\vskip -0.4em}
% \midrule
% F2F & \textbf{54.1\%} & \textbf{53} & \textbf{65.6\%} & \textbf{67} \\
% LEDITS++ & 45.9\% & 47 & 34.4\% & 33 \\
% \bottomrule
% \end{tabular}
% \caption{
% \textbf{Survey Results.}
% Human comparison of F2F and LEDITS++. The results showcase overall and per-image preferences, highlighting differences in performance.
% }
% \label{tbl:survey_results}
% \end{table}
% \begin{figure*}[!htb]
% \centering
% \includegraphics[width=0.85\linewidth]{figures/comparisons.pdf}
% \caption{
% \textbf{Qualitative Results on TEdBench.} Comparison with baseline methods across various editing tasks. Our approach consistently produces edits that better align with the target prompt while preserving the source image's content and structure. Note the teddy bear example, where our method uniquely achieves the complex structural modification while maintaining visual quality.
% }
% \label{fig:tedbench_qualitative}
% \end{figure*}
% \noam{
% evaluation on resolution 512. option to stay with the best image...
% + talk about seeds
% }
% \subsection{Human Evaluation Survey}
% To further evaluate our method, we conducted a human survey.
% As indicated in \cref{tbl:tedbench_quant}, LEDITS++ emerges as the most competitive method compared to ours, making it a natural choice for comparison.
% Participants were presented with 20 TEdBench samples, including the source image, the target edit prompt, and the corresponding edited outputs from both methods.
% Inspired by the methodology proposed by \cite{zhang2024magicbrush}, participants evaluated each comparison based on two criteria: (1) edit accuracy relative to the prompt, and (2) edit quality—defined as the preservation of visual fidelity to the source image, seamless integration of edited elements, and the overall natural appearance of modifications.
% % We collected 1,059 individual responses from 59 randomly selected online evaluators.
% We collected around 1,000 individual responses from 59 randomly selected online evaluators.
% % ֿ\noam{use numbers that fit to not raise questions}
% To ensure an unbiased assessment, evaluators were not informed of the study’s objectives.
% The results were quantified using two metrics: (i) overall global preference, expressed as a percentage, and (ii) aggregated per-image preference, where a tie resulted in each method receiving 0.5 points.
% Our results, summarized in Table \ref{tbl:survey_results}, demonstrate that F2F outperforms LEdits++ on both metrics.
% Specifically:
% In terms of edit quality, F2F achieved a 65.6\% global preference compared to LEdits++'s 34.4\%.
% % , highlighting its unparalleled ability to integrate edits seamlessly and maintain visual fidelity.
% For edit accuracy, F2F obtained a 53\% per-image preference, marginally outperforming LEdits++ (47\%).
% % On average, the global preference metric, balancing both edit quality and accuracy, favored F2F in 59.9\% of cases.
% These findings reinforce our claim that smooth temporal editing—using video as a medium—preserves essential scene characteristics while successfully performing the edit.
% Our findings suggest that the gap between F2F and LEdits++ in source preservation is larger than reflected by the LPIPS scores in \ref{tbl:tedbench_quant}.
% The full survey details are provided in the supplementary materials.
% On average, the global preference metric, balancing both edit quality and accuracy, favored F2F in 59.9\% of cases, highlighting its robust performance across both evaluation criteria.
% We conducted a Two-Alternative Forced Choice (2AFC)r, study to evaluate our method against LEdits++.
% ֿ\noam{cite or examplain?}
% Participants were presented with the source image, target edit prompt, and corresponding edited versions from both methods.
% For each comparison, participants assessed two aspects: (1) edit accuracy relative to the prompt, and (2) edit quality—defined as the preservation of visual fidelity to the source image, seamless integration of edited elements, and the overall natural appearance of modifications.
% We gathered 1059 individual responses from 59 human evaluators. To ensure an unbiased assessment, evaluators were not informed of the study’s specific objectives or the underlying research goals. The results were quantified using two metrics: (i) overall global preference, expressed as a percentage, and (ii) aggregated per-image preference, where a tie resulted in each method receiving 0.5 points, with a total of 100, which is the number of images in the survey.
% Our results, summarized in Table \ref{tbl:survey_results}, demonstrate that F2F outperforms LEdits++ on both metrics. Specifically:
% In terms of edit quality, F2F achieved a 65.6\% global preference compared to LEdits++'s 34.4\%, highlighting its unparalleled ability to integrate edits seamlessly and maintain visual fidelity.
% For edit accuracy, F2F obtained a 53\% per-image preference, marginally outperforming LEdits++ (47\%).
% These findings reinforce our claim that smooth temporal editing—using video as a medium—preserves essential scene characteristics while traversing the image manifold in a continuous, natural manner. Our findings suggest that the gap between F2F and LEdits++ in source preservation is larger then reflected by the LPIPS scores, as human feedback indicated that F2F consistently generated higher-quality outputs, particularly in preserving fidelity to the original image.
% On average, the global preference metric, balancing both edit quality and accuracy, favored F2F in 59.9\% of cases, highlighting its robust performance across both evaluation criteria.
% Further details on the evaluation protocol and survey design are provided in the supplementary materials.
% These findings reinforce the effectiveness of our video-based editing framework, particularly in preserving the scene’s key characteristics while delivering precise, high-quality edits. Evaluators consistently noted smoother transitions and more natural results with F2F compared to the abrupt modifications produced by LEdits++.
% The findings, detailed in Table \ref{tbl:survey_results}, strengthening out claim that smooth temporal editing, using video as a medium, indeed menage to preserve scene characteristics. We observed that in terms of fidelity to the original image, our evaluators strongly preferred F2F over LEDITS, one of the leading methods and main comparators in the terms of both source LPIPS (corresponding to fidelity) and Target CLIP (corresponding to preposition). This means that by traveling on the image manifold, F2F managed to highly
% while not compromising on editing precision
% reveal a strong human preference for our model's output in terms of image quality and fidelity to the original image, with a slight preference for its adherence to the instructions. On average, the global preference metric (balancing both evaluation criteria) indicates a preference for our model 59.9\% of the time.
% Further details on this survey are provided in the supplementary materials.
% \noam{add cite to magicbrush and that we adopted their methodology, their data is only for local editing- don't refer to what they do, only methodology}
% \silver{In the end we didn't do exactly as them (Three-Alternative Forced Choice vs Two-Alternative Forced Choice), lets leave the citation for the appendix maybe? when we show the phrasing of the questions}
% \noam{add that we used magicbrush (a datset for local editing) methodology because blalabla- from their paper}
% \subsection{Temporal Editing Captions}
% \subsection{Automated approach}
% \noam{we need to decide who to compare to}
\vspace{-3pt}
\subsection{PosEdit Benchmark}\label{sec:posedit}
\vspace{-3pt}
We introduce PosEdit, a benchmark for human pose editing derived from the UTD-MHAD dataset \cite{chen2015utd}.
UTD-MHAD includes RGB videos of 8 subjects (4 females and 4 males) performing predefined actions in a controlled indoor environment. We carefully curated 58 editing tasks encompassing 8 distinct action categories, ranging from simple poses like a raised hand to complex athletic poses such as basketball shooting and lunging.
The proposed benchmark specifically focuses on human pose manipulation.
Unlike TEdBench, each editing task in PosEdit includes a ground truth frame extracted from the same subject in the target pose.
The availability of reference images allows for a more comprehensive evaluation of both editing accuracy and identity preservation.
% This feature allows to supplement to the metrics reported in Subsection \ref{tedbench_results}, CLIP-I and LPIPS with the target image.
In the dataset, each editing task consists of two images: a source image showing the subject in a neutral standing pose with arms relaxed at their sides, and a ground-truth edited target image capturing the subject performing in a specific pose. The benchmark also provides a prompt for each task that describes the target pose the subject should achieve (e.g., "A person in a basketball shooting posture.").
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/PoseEdit_v2.pdf}
\vspace{-8pt}
\caption{\textbf{Qualitative Results on PosEdit.} Comparison between our Frame2Frame method and LEDITS++ on human motion editing tasks. For each example, we show the source image, edited results from both methods, and the ground-truth target image. Our method better preserves subject identity while achieving more natural pose transitions.
The evaluation metrics for each image are provided in Section \ref{supp:posedit} of the appendix.
}
\vspace{-10pt}
\label{fig:posedit_qualitative}
\end{figure}
\vspace{-6pt}
\paragraph{Benchmark Evaluation.}
The evaluation on PosEdit follows the same protocol established for TEdBench: generating multiple variants using nine different seeds and manually selecting the best result for each method. However, PosEdit's ground-truth target images enable additional evaluation metrics. Beyond measuring preservation of source content (via LPIPS and CLIP-I between source and edited images) and edit accuracy (via CLIP score with the prompt), we compute LPIPS and CLIP-I metrics between the edited image and its corresponding ground-truth target. This enhanced evaluation directly assesses both the accuracy of the transformation and preservation of identity features.
Additionally, we include the evaluation metrics of the ground-truth images as a reference to the rest of the results.
% As shown in Tables \ref{tbl:tedbench_quant} and \ref{tbl:PosEdit_results}, our method achieves strong performance across all metrics on both benchmarks, demonstrating effective balance between preserving source content and achieving accurate edits.
\Cref{tbl:PosEdit_results} demonstrates that our method consistently outperforms competitive methods across all metrics, with particularly strong performance in similarity measures with the target ground-truth image. This advantage highlights our method's superior ability to preserve key features while achieving the desired edit, as illustrated by the qualitative comparisons in \Cref{fig:posedit_qualitative}. The figure shows that our method generates more natural pose transitions while maintaining crucial identity attributes such as facial features, body proportions, and clothing details.
% Table \ref{tbl:PosEdit_results} demonstrates that our method consistently outperforms competitive methods across all metrics.
% Figure \ref{fig:posedit_qualitative} presents a qualitative analysis.
% Our approach displays a particularly strong advantage in similarity metrics with the target ground-truth image, highlighting our method’s superior ability to preserve key features of the original image.
% % For each method, the best results across 9 seeds were manually selected.
% Figure \ref{fig:posedit_qualitative} presents a qualitative analysis, displaying that Frame2Frame is superior in simulating realistic edits while preserving identity features.
% \noam{did we explain about the experiment and results. we can say here no blip was used just : "A person standing naturally with his arms relaxed at his sides.".}
% Those results showcase the superiority of our method in realistic transformations that are effectively rendered as general-purpose world simulators. Note that Frame2Frame is also capable of handling unrealistic transformations, as illustrated by the magically filled vase in Figure \ref{fig:teaser}.
Two important implementation notes:
First, for Pix2Pix-Zero, which requires source image descriptions, we use the same static prompt across all tasks: "\textit{A person standing naturally with his arms relaxed at his sides}." Second, we exclude Imagic \cite{kawar2023imagic} from this comparison as their official implementation is not publicly available and their results are only reported on TEdBench.
% \begin{table}
% \centering
% \begin{tabular}{@{}l|c@{\hspace{10pt}}c|c@{\hspace{10pt}}c@{}}
% \toprule
% & \multicolumn{2}{c|}{\textbf{Edit Accuracy}} & \multicolumn{2}{c}{\textbf{Edit Quality}} \\
% % \cline{2-5}
% \raisebox{0.8em}{Method} &
% \raisebox{0.8em}{Overall} &
% \raisebox{0.8em}{Per-Image} &
% \raisebox{0.8em}{Overall} &
% \raisebox{0.8em}{Per-Image} \\
% \noalign{\vskip -0.4em}
% \midrule
% F2F & \textbf{54.1\%} & \textbf{53.0\%} & \textbf{65.6\%} & \textbf{67.0\%} \\
% LEDITS++ & 45.9\% & 47.0\% & 34.4\% & 33.0\% \\
% \bottomrule
% \end{tabular}
% \vspace{-2pt}
% \caption{
% \textbf{Human Survey Results.}
% Human evaluation on TEdBench shows that F2F surpasses LEDITS++ in edit accuracy while offering a significant advantage in preserving the original image.
% }
% \vspace{-9pt}
% \label{tbl:survey_results}
% \end{table}
\subsection{Human Evaluation Survey}\label{sec:survey}
To further evaluate our method, we conducted a human survey.
As indicated in \Cref{tbl:tedbench_quant}, LEDITS++ emerges as the most competitive method compared to ours, making it a natural choice for comparison.
Each participant was presented with 20 randomly sampled TEdBench samples, including the source image, the target edit prompt, and the corresponding edited outputs from both methods.
Inspired by the methodology proposed by \cite{zhang2024magicbrush}, participants evaluated each comparison based on two criteria: (1) edit accuracy relative to the prompt, and (2) edit quality—defined as the preservation of visual fidelity to the source image, seamless integration of edited elements, and the overall natural appearance of modifications.
% We collected 1,059 individual responses from 59 randomly selected online evaluators.
We collected responses from 59 randomly selected online evaluators.
% ֿ\noam{use numbers that fit to not raise questions}
To ensure an unbiased assessment, evaluators were not informed of the study’s objectives.
The results were quantified using two metrics: (i) overall global preference, expressed as a percentage, and (ii) aggregated per-image preference, where a tie resulted in each method receiving 0.5 points.
\begin{table}
\centering
\begin{tabular}{@{}l|c@{\hspace{10pt}}c|c@{\hspace{10pt}}c@{}}
\toprule
& \multicolumn{2}{c|}{\textbf{Edit Accuracy}} & \multicolumn{2}{c}{\textbf{Edit Quality}} \\
% \cline{2-5}
\raisebox{0.8em}{Method} &
\raisebox{0.8em}{Overall} &
\raisebox{0.8em}{Per-Image} &
\raisebox{0.8em}{Overall} &
\raisebox{0.8em}{Per-Image} \\
\noalign{\vskip -0.4em}
\midrule
F2F & \textbf{54.1\%} & \textbf{53.0\%} & \textbf{65.6\%} & \textbf{67.0\%} \\
LEDITS++ & 45.9\% & 47.0\% & 34.4\% & 33.0\% \\
\bottomrule
\end{tabular}
\vspace{-2pt}
\caption{
\textbf{Human Survey Results.}
Human evaluation on TEdBench shows that F2F surpasses LEDITS++ in edit accuracy while offering a significant advantage in preserving the original image.
}
\vspace{-9pt}
\label{tbl:survey_results}
\end{table}
Our results, summarized in \Cref{tbl:survey_results}, demonstrate that F2F outperforms LEdits++ on both metrics.
Specifically:
For edit quality, F2F achieved a global preference score of 53\%, slightly surpassing the 47\% obtained by LEdits++.
In terms of edit accuracy, F2F achieved a 65.6\% global preference compared to LEdits++'s 34.4\%.
The per-image preference results mirrored this trend, indicating robustness across individual examples and no significant influence of outliers.
These findings reinforce our claim that smooth temporal editing—using video as a medium—preserves essential scene characteristics while successfully performing the edit.
Our results suggest that the gap between F2F and LEdits++ in source preservation is larger than reflected by the LPIPS scores in \Cref{tbl:tedbench_quant}.
The full survey details are provided in the supplementary materials in Section \ref{supp:human_survey}.
\begin{figure*}[!htb]
\centering
\includegraphics[width=1.0\linewidth]{figures/silver_v3.pdf}
\caption{
\textbf{Additional Vision Tasks. }
Qualitative results of our image-to-video-to-image editing approach on selected traditional tasks.
% Deblurring was achieved by imitating camera focusing, outpainting by simulating zooming out, and relighting by moving a light source in a time-lapse.
% Full prompts for each scenario are provided in the supplementary material.
}
% ֿ\vspace{-3pt}
\label{fig:generatior_example}
\end{figure*}
\subsection{Additional Vision Tasks}\label{sec:silver_tasks}
We demonstrate our framework's applicability beyond traditional editing by applying it to fundamental image manipulation tasks: denoising, deblurring, outpainting, and relighting.
For these experiments, we utilize Runway Gen-3 \cite{runway-gen1} as our video generation backbone, as it showed superior performance on these specific tasks compared to other generative video models.
Since these tasks serve as extensions and potential applications of our method, we focus on qualitative results, leaving quantitative evaluation for future work.
Our results, shown in \Cref{fig:generatior_example}, demonstrate strong performance across all tasks.
We associate this success to the natural alignment between these operations and common video scenarios: deblurring maps to camera focusing, outpainting to camera motion, and relighting to time-lapse lighting changes.
This alignment enables our method to leverage the video model's learned temporal dynamics, achieving high-quality results without task-specific training.
Full prompts are provided in Section \ref{supp:vision_tasks} of the supplementary material.
\vspace{8pt}
\noindent
\textbf{Extended Results.} Additional examples illustrating our method's editing performance across diverse backgrounds are provided in Section \ref{supp:more_ablations} of the Appendix.
% \subsection{Additional Vision Tasks}
% \noam{we have to say this is with runway which costs money (cog wasn't goo enough.}
% In our experiments, we applied our generative image-to-video-to-image editing approach to several traditional image editing tasks, including denoising, deblurring, outpainting, and relighting. Our quantitative results, as shown in Figure \ref{fig:generatior_example}, demonstrate that the method performs well on these tasks. Notably, many of these image editing actions align closely with scenarios encountered in the training data for video generation models. For instance, deblurring resembles the refocusing process common in video footage, while outpainting can be likened to zooming out. Similarly, relighting is inherently related to moving light sources and time-lapse video. By leveraging these implicit tasks within the video model's training data, our method achieves consistency and high quality across various editing tasks.
% \noam{add that full prompts are provided in the supplementary}
% \subsection{More Models}
% \noam{
% - how we chose our video model including experiment on TEdBench}
% Probably need to change and modify, but this subsection seem important for the overall coherence of explaining the method.\\
% We used the following Video models in this paper:
% }
% \begin{enumerate}
% % add citations to them, maybe add a little bit more meat to this section.
% \item Luma AI
% \item RunWay
% \item KlingAI
% \item COG
% \end{enumerate}
% ֿ\noam{
% Do not delete!!
% some of the prompts are not good enough do the lame instruction.
% Maybe fix them manually and rerun only them, its mostly details missing.
% }
% \begin{figure*}[!htb]
% \centering
% \includegraphics[width=1.0\linewidth]{figures/silver.pdf}
% \caption{
% Quantitative results of our image-to-video-to-image editing approach on selected traditional tasks. Deblurring was achieved by imitating camera focusing, outpainting by simulating zooming out, and relighting by moving a light source in a time-lapse. Full prompts for each case are provided in the supplementary material.
% }
% \label{fig:generatior_example}
% \end{figure*}
% \begin{figure*}[!htb]
% \centering
% \includegraphics[width=0.25\linewidth]{figures/silver.pdf}
% \caption{
% Qualitative results of our image-to-video-to-image editing approach on selected traditional tasks. Deblurring was achieved by imitating camera focusing, outpainting by simulating zooming out, and relighting by moving a light source in a time-lapse. Full prompts for each case are provided in the supplementary material.
% }
% \label{fig:generatior_example}
% \end{figure*}
\vspace{-3pt}
\section{Limitations}\label{sec:limitations}
While our approach addresses key shortcomings in current image editing methods, it also introduces unique challenges.
For example, natural camera motion sometimes appears in video sequences, which, when replicated in generated content, can lead to unintended perspective shifts.
As with all generative models, video models are trained on specific data domains, making it challenging to produce results that deviate significantly from the model's training data, which predominantly includes real-world transformations. Despite these challenges, F2F demonstrates success in several cases. For example, in the middle row of \Cref{fig:teaser}, the vase is "magically" filled with green water, showcasing the model's ability to perform imaginative edits, with additional examples discussed in Section \ref{supp:more_ablations} of the supplementary.
Additionally, our method is computationally intensive, as transforming an image into a video sequence is resource-heavy and often slower than other image editing methods.
However, video generation efficiency is advancing rapidly, as demonstrated by models like Runway Turbo\footnote{\href{https://runwayml.com/news/introducing-the-runway-api}{Runway Turbo API}} and LTX-Video \cite{hacohen2024ltx}, which can execute the process in seconds, making this approach increasingly less resource-intensive. Additionally, optimizing the number of frames per edit—currently fixed at 49 for CogVideoX—could enable faster editing possibilities.
% We anticipate that advancements in video generation technology will reduce computational demands and improve processing speed, addressing current runtime limitations.
%Our approach faces three key limitations. First, video generation models can introduce unintended perspective shifts due to natural camera motions inherent in training data. Second, while our method handles unrealistic transformations (e.g., magically filling a vase in Figure \ref{fig:teaser}), video models' training data bias toward real-world scenarios limits extensive stylistic editing. Finally, video generation is computationally intensive compared to traditional editing methods, though recent advances like Runway Turbo demonstrate significant speed improvements. We expect ongoing progress in video generation to address these constraints.
% \section{Limitations}
% While our approach addresses key shortcomings in current image editing methods, it also introduces unique challenges. For instance, natural camera motions sometimes appear in video sequences, which, when replicated in generated content, can lead to unintended perspective shifts. As illustrated by the magically filled vase in Figure 1, Frame2Frame is capable of handling unrealistic transformations. However, like all generative models, video models are trained on specific data domains, making it challenging to produce results that significantly deviate from the model's training data, which predominantly includes real-world transformations. This limitation affects the ability to achieve extensive stylistic editing.
% Additionally, transforming an image into a video sequence is computationally demanding and often slower than traditional image editing methods. Nonetheless, video generation efficiency is advancing rapidly, as shown by models like Runway Turbo \noam{add cite like}, which can complete the process in seconds.
% We anticipate that future advancements in video generation technology will address current limitations in runtime and scope, making more complex and flexible edits increasingly feasible.
% \noam{add that in the supplementary we provide further examples}
%Our method aims to address key shortcomings in existing generative image editing approaches, but as a novel approach based on video generation, it also introduces unique limitations stemming from the nature of video models.
%Video models, like most generative models, are trained within certain data domains, making it challenging to produce out-of-distribution results. Consequently, our method may struggle with edits that deviate significantly from real-world transformations, such as edits that involve extensive style transformations. Even so, there are instances where the model successfully executes highly unreal transformations. For example, The vase that is magically filled with green water in Figure \ref{fig:generatior_example2}.
% \noam{cite that its not perfect, maybe from cogvideox itself or some survey:
% https://arxiv.org/pdf/2408.06072} \silver{they really don't talk about limitations in cogvideox...}
% \noam{refer to teaser and maybe more images to show how we fill wit green water}
%Additionally, natural video sequences usually exhibit some degree of camera movement, a trait that is often replicated in generated videos.
%This can lead to unintended perspective shifts, which, while generally negligible for standard editing tasks as the scene do not change, could reduce precision in scenarios requiring high accuracy. However, our chosen generator, cog, is usually successful in preserving perspective when prompted to do so.
% \noam{lets write that cog is relatively ok in this + example where this happened but not severly}
%On a different note, the process of transforming an image into a video sequence is computationally demanding and can be slower than traditional image editing methods. It is important to note runtime can vary significantly based on the employed method; for example, while Cog may require few minutes to execute, Runway Turbo is capable of completing the process in seconds.
% \noam{needs to sound more positive}
%On a different note, while generating videos is computationally demanding, it can vary significantly based on the employed method. For example, while Cog may take a few minutes, Runway Turbo can achieve the generation in seconds, demonstrating the efficiency of current video generation technologies.
%The potential of our approach evolves with video generation technology, which currently imposes some limitations. As the field is rapidly advancing \cite{yang2024cogvideox}, we anticipate future improvements will address our current limitations in runtime and scope, enhancing the feasibility of more complex edits.
% The success and flexibility of our approach are closely tied to advancements in video generation: our method’s performance and scope are limited those of the current video models. However, as this is a nascent field, those are likely to improve in quickly. Future developments may mitigate our current runtime and out-of-domain limitations, and solve cases where the video model struggles to follow the prompt, potentially expanding the types of edits that are feasible.
% \noam{cite things about the developing field. i see cogvx talk about it:
% https://arxiv.org/pdf/2408.06072}
% poits:
% - in training data domain
% - camera is not always static.
% - I think this is one of the most important sections- we need to deeply evaluate when it works or not
% - the control with prompt over video is not great yet, as this improve the method will improve
% - slow + compute
% - say that it cannot edit everything but as video models improve this will improve- for example.... and point to an image
% \section{Conclusions}
% We propose to reformulate image editing as a video generation task. By leveraging the temporal coherence and world simulation capabilities of state-of-the-art video models, our approach creates smooth and physically plausible transitions between the original image and the desired edit. Extensive experiments demonstrate the effectiveness of our Frame2Frame framework, with substantial improvements over existing methods on both standard and custom benchmarks.
\section{Conclusions}
\vspace{-2pt}
We introduced Frame2Frame, a novel approach that reformulates image editing through video generation. By leveraging video models' inherent understanding of temporal transformations, our method achieves state-of-the-art editing results while maintaining high fidelity to source images. We demonstrated our framework's effectiveness on standard benchmarks, introduced PosEdit for human pose editing, and showed promising results on more classical vision tasks. As video generation technology advances, we expect our approach to enable increasingly sophisticated image manipulations while maintaining natural and physically plausible results.
\vspace{-2pt}
\paragraph{Future Research.}
Our work opens several promising directions.
First, fine-tuning video generators specifically for image editing presents an exciting opportunity, including straightforward solutions like enforcing static camera scenarios or using datasets curated for editing tasks, alongside more complex approaches yet to emerge.
Secondly, a key direction could involve reducing the overhead of full video generation while preserving the benefits of gradual, temporally coherent transformations, potentially enhancing both the efficiency and speed of editing.
The speed can be affected both in terms of per-frame average generation speed or the number of frames generated per edit, which is currently fixed at 49 for CogVideoX.
% \subsection{Future Research}
% Our work opens several promising research directions at the intersection of video generation and image editing. First, addressing the challenge of non-static cameras in video generation by fine-tuning the video generator could enable more precise editing control while maintaining the benefits of our manifold-based approach. Second, we look forward to anticipated advancements in video generation technologies that will enable models to better understand and execute complex prompts. Such developments would enhance our method by allowing for more nuanced and accurate edits. %, expanding the range of what is correctly possible.
% % Second, expanding the range of supported editing operations beyond the training domain of current video models represents an important frontier - particularly for specialized technical edits that rarely appear in natural video data.
% A critical direction for future research is developing methods to achieve the benefits of our video-based approach without the computational overhead of full video generation. One promising avenue is finding ways to directly compute the latent embedding of the edited image by learning to predict the temporal transformation in the model's latent space. This could potentially allow us to "fast-forward" to the desired edit frame without generating the intermediate video sequence. Such an approach could dramatically improve computational efficiency while preserving the advantages of our manifold-based editing paradigm.
\clearpage
% \setcounter{page}{1}
\appendix
\renewcommand{\thefigure}{S\arabic{figure}}
\renewcommand{\thetable}{S\arabic{table}}
% Correct command to center section titles
% \section*{\centering Appendix}
\twocolumn[
\begin{@twocolumnfalse}
\begin{center}
{\LARGE \textbf{Appendix}} % Adjust size and formatting as needed
\end{center}
% \vspace{1em} % Add space after the title
\end{@twocolumnfalse}
]
\section{Additional Editing Examples}\label{supp:more_examples}
We provide additional qualitative examples of the editing results on the TEdBench benchmark, generated based on the experimental setup detailed in \Cref{sec:tedbench_results}.
These examples further supplement those presented in Figures \ref{fig:teaser} and \ref{fig:tedbench_qualitative}.
Each example is accompanied by the temporal editing caption used to perform the edit, which was generated using the method described in \Cref{sec:tec}.
Furthermore, in \Cref{fig:f2f_videos}, we supplement \Cref{fig:teaser} with additional video examples generated by our method, illustrating the transition from the source image (left) to the target edit (right).
\begin{figure}[H]
\centering
\includegraphics[width=0.95\linewidth]{figures/editing_sup_examples_1.pdf}
\caption{\textbf{TEdBench Editing Examples.}}
\label{fig:f2f_examples_1}
\end{figure}
\begin{figure}[H]
\centering
% \vspace{4pt}
\includegraphics[width=1.04\linewidth]{figures/editing_sup_examples_2.pdf}
\caption{\textbf{TEdBench Editing Examples.}}
\label{fig:f2f_examples_2}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=1.03\linewidth]{figures/sequences.pdf}
\caption{\textbf{Generated Video Examples.} Additional video sequences illustrating the temporal evolution from the source image to the target edit.}
\label{fig:f2f_videos}
\end{figure}
\section{Temporal Editing Captions}\label{supp:temporal_captions}
\subsection{VLM Instruction}
As outlined in \Cref{temporal_captions}, we propose a framework for automatically generating the temporal editing caption by leveraging the original target editing prompt in conjunction with the source image.
The instruction given to the VLM, along with the source image is:
\textit{
'Write a one-sentence description of a short video that begins with the provided image and smoothly transitions into a scene of a "CAPTION",
highlighting how elements in the image undergo changes or movement over time.
Keep the description simple, concise and short, focusing only on essential changes and actions without altering unnecessary details.
Avoid mentioning elements that do not contribute to the main change needed, and focus the description on the main transitions.
Do not add objects that are not in the original image or described in the final scene.
The camera should remain static unless movement is absolutely necessary.
Ensure all transitions happen within a few second duration without mentioning the length or using the word "video".'
}
Here, "CAPTION" is replaced with the target caption specific to the image.
Additionally, as explained, in-context learning is employed to provide the VLM with examples alongside the instruction. Before processing the desired source image and edit prompt, the instruction is presented to the VLM nine times, each paired with a distinct example consisting of a source image, target caption, and corresponding temporal editing caption.
Examples of these are illustrated in \Cref{fig:icl}.
\begin{figure}[H]
\centering
\includegraphics[width=1.03\linewidth]{figures/icl.pdf}
\caption{\textbf{In Context Learning Examples.}}
\label{fig:icl}
\end{figure}
\subsection{Ablation}
To assess the impact of the Temporal Editing Caption, we conduct an ablation experiment comparing its use against directly using the target editing captions from the TEdBench benchmark.
Apart from this modification, we adhere to the same protocols as described in the original experiment in \Cref{sec:tedbench_results}.
As shown in \Cref{tab:tedench_results_ablation}, this setup preserves a similar resemblance to the source image but underperforms in terms of image editing performance.
\begin{table}[H]
\centering
\begin{tabular}{l|cc|c}
\toprule
& \multicolumn{2}{c|}{\textbf{Source}} & \textbf{Target} \\
\raisebox{1.2em}{Model} &
\raisebox{1.2em}{LPIPS$_\downarrow$} &
\raisebox{1.2em}{CLIP-I$_\uparrow$} &
\raisebox{1.2em}{CLIP$_\uparrow$} \\
\noalign{\vskip -0.6em}
\midrule
Original Captions & \textbf{0.21} & \textbf{0.89} & 0.60 \\
Temporal Captions & 0.22 & \textbf{0.89} & \textbf{0.63} \\
\bottomrule
\end{tabular}
\caption{
\textbf{Temporal Editing Captions Ablation.}
}
\label{tab:tedench_results_ablation}
% \vspace{-5pt}
\end{table}
\section{Frame Selection}\label{supp:frame_seletion}
\subsection{VLM Instruction}
As detailed in \Cref{sec:frame_selection}, our method selects the frame that best aligns with the intended edit from each generated video.
To automate this process, inspired by \cite{kim2024image}, we create a collage of uniformly sampled frames from the video, along with the source image and target editing caption, and prompt a VLM to identify the optimal frame.
The model is instructed to select the earliest frame (i.e., with the lowest index) that satisfies the editing intent, minimizing deviation from the original image.
In both this process and the best seed selection process (applied across all methods), if none of the edited frames successfully fulfill the desired edit, the original image is retained as the final output.
The instruction provided to the VLM is:
\textit{
`The image displays the source photo at the top, with a collage of 12 edited versions beneath it.
The target edit image caption was: ``CAPTION".
Your task is to choose the image from 1 to 12 that best follows this edit fully and naturally.
If none of the images follows the edit, select image 0.
If multiple images follow the edit equally, prioritize the one with the lowest number possible.
Avoid selecting images that appear to follow the edit but are not edits of the original image.
Additionally, avoid images where camera motion, zoom, or image quality differs significantly, or where the content does not appear stable relative to the original source.
Respond with: ``The selected edit is:x" where x is the number of your chosen edit.'
}
Here, "CAPTION" refers to the target editing caption. Examples of the collages can be found in \Cref{fig:frame_collage}.
\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{figures/frames_collage.pdf}
\caption{\textbf{Frame Selection Collage.} The target editing caption for this example is: ``A photo of a cat yawning.".}
\label{fig:frame_collage}
\end{figure}
\subsection{Ablation}
To validate the effectiveness of our approach, we compare it to the naive solution of using the last frame of the generated video as the edited output.
The evaluation follows the same protocol described in \Cref{sec:tedbench_results}.
As can be seen in \Cref{tab:tedench_frame_ablation}, this naive approach results in a lower target CLIP score for the edited outputs, highlighting the advantages of our method.
\begin{table}[H]
\centering
\begin{tabular}{l|cc|c}
\toprule
& \multicolumn{2}{c|}{\textbf{Source}} & \textbf{Target} \\
\raisebox{1.2em}{Model} &
\raisebox{1.2em}{LPIPS$_\downarrow$} &
\raisebox{1.2em}{CLIP-I$_\uparrow$} &
\raisebox{1.2em}{CLIP$_\uparrow$} \\
\noalign{\vskip -0.6em}
\midrule
Last Frame & 0.24 & \textbf{0.9} & 0.61 \\
Selected Frame & \textbf{0.22} & 0.89 & \textbf{0.63} \\
\bottomrule
\end{tabular}
\caption{
\textbf{Frame Selection Ablation.}
}
\label{tab:tedench_frame_ablation}
% \vspace{-5pt}
\end{table}
\section{Editing Manifold Pathway}\label{supp:manifold}
As elaborated in \Cref{sec:manifold}, to simulate the image manifold, we generated 200 images across three distinct categories using FLUX.1-dev.
Examples of these generated images are shown in~\Cref{fig:flux}.
\begin{figure}[H]
\centering
\includegraphics[width=0.85\linewidth]{figures/flux.pdf}
\caption{\textbf{Flux.1-dev Generations}}
\label{fig:flux}
\end{figure}
\section{Inference Hyperparameters}
We use the 5B-parameter image-to-video version of CogVideoX, ensuring consistency by applying the same inference hyperparameters across all experiments.
The model generates a fixed 49 frames per sample. During inference, we use the default denoising scheduler in CogVideoX, which is based on DDIM with V-prediction. We perform 50 denoising steps and set the classifier-free guidance scale for text conditioning to 6.0.
\section{PosEdit}\label{supp:posedit}
\vspace{-2pt}
In \Cref{sec:posedit}, we outline the construction of a human pose editing dataset.
This dataset encompasses 58 editing tasks, distributed across 8 distinct action categories, featuring 8 different subjects.
The source images consistently depict a neutral standing pose with arms relaxed at the sides, while the target poses vary according to the edit category.
Each editing category is paired with a target caption and a temporal caption.
\Cref{fig:davidata_examples_1} and \Cref{fig:davidata_examples_2} illustrates examples for each action category.
Additionally, in \Cref{fig:davidata_examples_metrics}, we complement \Cref{fig:posedit_qualitative} with quantitative results, demonstrating how the numerical evaluation aligns with its intended measurement objectives.
\begin{figure}[H]
\centering
\includegraphics[width=0.86\linewidth]{figures/davidata_examples_1.pdf}
\caption{\textbf{PosEdit Examples.}}
\label{fig:davidata_examples_1}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.86\linewidth]{figures/davidata_examples_2.pdf}
\vspace{-5pt}
\caption{\textbf{PosEdit Examples.}}
\label{fig:davidata_examples_2}
\vspace{-5pt}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=1\linewidth]{figures/figure_5_nums.pdf}
\vspace{-5pt}
\caption{\textbf{PosEdit Quantitative Evaluation Examples.}}
\label{fig:davidata_examples_metrics}
\vspace{-5pt}
\end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=0.9\linewidth]{figures/davidata_examples_full.pdf}
% \label{fig:davidata_examples}
% \caption{\textbf{PosEdit Examples.}}
% \end{figure}
\vspace{-5pt}
\section{Human Survey}\label{supp:human_survey}
\vspace{-5pt}
As detailed in \Cref{sec:survey}, we conducted a human evaluation survey to assess our method's performance based on real user preferences.
Following the framework in \cite{zhang2024magicbrush}, the survey questions evaluated (1) the accuracy of the edit relative to the prompt and (2) the quality of the edit, defined as the preservation of visual fidelity to the source image.
Each participant reviewed 20 edits, comparing our method with LEDITS++.
Examples of the pages shown to the evaluators are provided in Figures \ref{fig:survey_1} and \ref{fig:survey_2}.
\begin{figure}[H]
\centering
\includegraphics[width=1.0\linewidth]{figures/survey1.pdf}
\caption{\textbf{Survey Example.}}
\label{fig:survey_1}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=1.0\linewidth]{figures/survey2.pdf}
\caption{\textbf{Survey Example.}}
\label{fig:survey_2}
\end{figure}
\section{Further Ablations}\label{supp:more_ablations}
We present additional ablation results demonstrating the ability of our method to handle two distinct editing challenges: (1) diverse backgrounds and (2) out-of-video-distribution edits.
\noindent \textbf{Backgrounds.} Similar to our image generation approach in \Cref{sec:manifold}, we generate eight images of two objects with varying backgrounds.
Each image generation prompt follows one of the two templates below, where LOCATION is replaced with one of the following settings:
colorful amusement park,
sandy beach,
magical fairy tale forest,
futuristic cityscape,
old library,
snowy mountain peak,
bustling train station,
and quaint village square.
\begin{enumerate}
\item \textit{``A small brown teddy bear sitting in} LOCATION\textit{.''}
\item \textit{``A tennis ball lies in} LOCATION\textit{.''}
\end{enumerate}
\noindent \textbf{Out-of-video-distribution editing.} As discussed in \Cref{sec:limitations}, one might assume that our method would fail to perform edits requiring temporal transformations that deviate significantly from typical real-world videos (the model's training set).
To show that this is not necessarily the case, we selected two unreal editing processes, using the following temporal captions:
\begin{enumerate}
\item \textit{``The teddy bear slowly rips, revealing stuffing coming out.''}
\item \textit{``The tennis ball gradually transforms into a ripe red tomato.''}
\end{enumerate}
These edits depict unrealistic events that do not ordinarily occur in real-life footage. However, as shown in \cref{fig:tennis_tomato_backgrounds}, our method successfully handles both the background variations and the out-of-distribution nature of these transformations. Moreover, the full temporal sequences in \cref{fig:f2f_videos} demonstrate that these edits occur seamlessly without any external interaction (the teddy bear rips apart spontaneously, and the tennis ball morphs magically into a tomato).
\begin{figure}[H]
\centering
\includegraphics[width=\linewidth]{figures/tennis_tomato.pdf}
\caption{\textbf{Ablation Examples.}}
\label{fig:tennis_tomato_backgrounds}
\end{figure}
% \begin{figure}[H]
% \centering
% \includegraphics[width=\linewidth]{figures/ablation_seq.pdf}
% \caption{\textbf{Ablation Sequences.}}
% \label{fig:tennis_tomato_seqs}
% \end{figure}
\section{Additional Vision Tasks Captions}\label{supp:vision_tasks}
As outlined in \Cref{sec:silver_tasks} and demonstrated in \Cref{fig:generatior_example}, we showcase our framework's applicability for additional, more classic vision tasks that are not typically classified as image editing.
For these tasks, we employ Runway Gen-3 as our video generator. Empirically, these tasks required longer and more descriptive captions.
The temporal editing captions used for each task are as follows:
\begin{enumerate}
\item \textbf{Relighting}:
\textit{
'The scene's lighting shifts gradually, changing to night. The sun is setting, and artificial lights replace it.
The camera is static. Time-lapse. Cinematic.'
}
\item \textbf{Outpainting}:
\textit{
'The image expands, adding new surroundings seamlessly beyond the original frame.'
}
\item \textbf{Denoising}:
\textit{
'The image clears up as noise fades away, revealing smoother, cleaner details.'
}
\item \textbf{Debluring}:
\textit{
'The camera comes into focus, revealing sharp details and enhanced clarity, as though a camera lens has adjusted perfectly. Nothing moves. Static image.'
}
\end{enumerate}
\begin{thebibliography}{51}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Alzayer et~al.(2024)Alzayer, Xia, Zhang, Shechtman, Huang, and Gharbi]{alzayer2024magic}
Hadi Alzayer, Zhihao Xia, Xuaner Zhang, Eli Shechtman, Jia-Bin Huang, and Michael Gharbi.
\newblock Magic fixup: Streamlining photo editing by watching dynamic videos.
\newblock \emph{arXiv preprint arXiv:2403.13044}, 2024.
\bibitem[Blattmann et~al.(2023{\natexlab{a}})Blattmann, Dockhorn, Kulal, Mendelevitch, Kilian, Lorenz, Levi, English, Voleti, Letts, et~al.]{blattmann2023stable}
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et~al.
\newblock Stable video diffusion: Scaling latent video diffusion models to large datasets.
\newblock \emph{arXiv preprint arXiv:2311.15127}, 2023{\natexlab{a}}.
\bibitem[Blattmann et~al.(2023{\natexlab{b}})Blattmann, Rombach, Ling, Dockhorn, Kim, Fidler, and Kreis]{blattmann2023align}
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung~Wook Kim, Sanja Fidler, and Karsten Kreis.
\newblock Align your latents: High-resolution video synthesis with latent diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 22563--22575, 2023{\natexlab{b}}.
\bibitem[Brack et~al.(2023)Brack, Friedrich, Kornmeier, Tsaban, Schramowski, Kersting, and Passos]{leitspp}
Manuel Brack, Felix Friedrich, Katharina Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinário Passos.
\newblock Ledits++: Limitless image editing using text-to-image models.
\newblock 2023.
\bibitem[Brooks et~al.(2023)Brooks, Holynski, and Efros]{brooks2023instructpix2pix}
Tim Brooks, Aleksander Holynski, and Alexei~A Efros.
\newblock Instructpix2pix: Learning to follow image editing instructions.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 18392--18402, 2023.
\bibitem[Chen et~al.(2015)Chen, Jafari, and Kehtarnavaz]{chen2015utd}
Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz.
\newblock Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor.
\newblock In \emph{2015 IEEE International conference on image processing (ICIP)}, pages 168--172. IEEE, 2015.
\bibitem[Chen et~al.(2024{\natexlab{a}})Chen, Siarohin, Menapace, Deyneka, Chao, Jeon, Fang, Lee, Ren, Yang, et~al.]{chen2024panda}
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung~Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et~al.
\newblock Panda-70m: Captioning 70m videos with multiple cross-modality teachers.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 13320--13331, 2024{\natexlab{a}}.
\bibitem[Chen et~al.(2024{\natexlab{b}})Chen, Huang, Liu, Shen, Zhao, and Zhao]{chen2024anydoor}
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.
\newblock Anydoor: Zero-shot object-level image customization.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 6593--6602, 2024{\natexlab{b}}.
\bibitem[Clark et~al.(2019)Clark, Donahue, and Simonyan]{clark2019adversarial}
Aidan Clark, Jeff Donahue, and Karen Simonyan.
\newblock Adversarial video generation on complex datasets.
\newblock In \emph{ICCV}, 2019.
\bibitem[Dai et~al.(2023)Dai, Li, Li, Tiong, Zhao, Wang, Li, Fung, and Hoi]{dai2023instructblipgeneralpurposevisionlanguagemodels}
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng~Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
\newblock Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
\bibitem[Danier et~al.(2024)Danier, Zhang, and Bull]{danier2024ldmvfi}
Duolikun Danier, Fan Zhang, and David Bull.
\newblock Ldmvfi: Video frame interpolation with latent diffusion models.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, 2024.
\bibitem[Dong et~al.(2022)Dong, Li, Dai, Zheng, Ma, Li, Xia, Xu, Wu, Liu, et~al.]{dong2022survey}
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et~al.
\newblock A survey on in-context learning.
\newblock \emph{arXiv preprint arXiv:2301.00234}, 2022.
\bibitem[Esser et~al.(2023)Esser, Chiu, Atighehchian, Granskog, and Germanidis]{runway-gen1}
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.
\newblock Structure and content-guided video synthesis with diffusion models, 2023.
\bibitem[Ganz et~al.(2024)Ganz, Kittenplon, Aberdam, Ben~Avraham, Nuriel, Mazor, and Litman]{ganz2024question}
Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben~Avraham, Oren Nuriel, Shai Mazor, and Ron Litman.
\newblock Question aware vision transformer for multimodal reasoning.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 13861--13871, 2024.
\bibitem[Gao et~al.(2023)Gao, Zhou, Wang, Li, Ming, and Lu]{gao2023high}
Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu.
\newblock High-fidelity and freely controllable talking head video generation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 5609--5619, 2023.
\bibitem[Ha and Schmidhuber(2018)]{ha2018world}
David Ha and J{\"u}rgen Schmidhuber.
\newblock World models.
\newblock \emph{arXiv preprint arXiv:1803.10122}, 2018.
\bibitem[HaCohen et~al.(2024)HaCohen, Chiprut, Brazowski, Shalem, Moshe, Richardson, Levin, Shiran, Zabari, Gordon, et~al.]{hacohen2024ltx}
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et~al.
\newblock Ltx-video: Realtime video latent diffusion.
\newblock \emph{arXiv preprint arXiv:2501.00103}, 2024.
\bibitem[Hertz et~al.(2022)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or]{hertz2022prompt}
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.
\newblock Prompt-to-prompt image editing with cross attention control.
\newblock \emph{arXiv preprint arXiv:2208.01626}, 2022.
\bibitem[Ho et~al.(2020)Ho, Jain, and Abbeel]{ho2020denoising}
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 6840--6851, 2020.
\bibitem[Huang et~al.(2024)Huang, Huang, Liu, Yan, Lv, Liu, Xiong, Zhang, Chen, and Cao]{huang2024diffusion}
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao.
\newblock Diffusion model-based image editing: A survey.
\newblock \emph{arXiv preprint arXiv:2402.17525}, 2024.
\bibitem[Huberman-Spiegelglas et~al.(2024)Huberman-Spiegelglas, Kulikov, and Michaeli]{Huberman-Spiegelglas_2024_CVPR}
Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli.
\newblock An edit friendly ddpm noise space: Inversion and manipulations.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pages 12469--12478, 2024.
\bibitem[Kawar et~al.(2023)Kawar, Zada, Lang, Tov, Chang, Dekel, Mosseri, and Irani]{kawar2023imagic}
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.
\newblock Imagic: Text-based real image editing with diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 6007--6017, 2023.
\bibitem[Kim et~al.(2024)Kim, Choi, Lee, and Rhee]{kim2024image}
Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee.
\newblock An image grid can be worth a video: Zero-shot video question answering using a vlm.
\newblock \emph{arXiv preprint arXiv:2403.18406}, 2024.
\bibitem[Kulikov et~al.(2024)Kulikov, Kleiner, Huberman-Spiegelglas, and Michaeli]{kulikov2024flowedit}
Vladimir Kulikov, Matan Kleiner, Inbar Huberman-Spiegelglas, and Tomer Michaeli.
\newblock Flowedit: Inversion-free text-based editing using pre-trained flow models.
\newblock \emph{arXiv preprint arXiv:2412.08629}, 2024.
\bibitem[Kwak et~al.(2024)Kwak, Dong, Jin, Ko, Mahajan, and Yi]{kwak2024vivid}
Jeong-gi Kwak, Erqun Dong, Yuhe Jin, Hanseok Ko, Shweta Mahajan, and Kwang~Moo Yi.
\newblock Vivid-1-to-3: Novel view synthesis with video diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 6775--6785, 2024.
\bibitem[Labs(2024)]{flux_model}
Black~Forest Labs.
\newblock Flux.1-dev.
\newblock \url{https://github.com/black-forest-labs/FLUX.1-dev}, 2024.
\bibitem[Li et~al.(2023)Li, Li, Savarese, and Hoi]{li2023blip}
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
\newblock Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
\newblock In \emph{International conference on machine learning}, pages 19730--19742. PMLR, 2023.
\bibitem[Liu et~al.(2024)Liu, Zhang, Li, Yan, Gao, Chen, Yuan, Huang, Sun, Gao, et~al.]{liu2024sora}
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et~al.
\newblock Sora: A review on background, technology, limitations, and opportunities of large vision models.
\newblock \emph{arXiv preprint arXiv:2402.17177}, 2024.
\bibitem[Luo et~al.(2024)Luo, Darrell, Wang, Goldman, and Holynski]{luo2024readout}
Grace Luo, Trevor Darrell, Oliver Wang, Dan~B Goldman, and Aleksander Holynski.
\newblock Readout guidance: Learning control from diffusion features.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 8217--8227, 2024.
\bibitem[Meng et~al.(2021)Meng, He, Song, Song, Wu, Zhu, and Ermon]{meng2021sdedit}
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
\newblock Sdedit: Guided image synthesis and editing with stochastic differential equations.
\newblock \emph{arXiv preprint arXiv:2108.01073}, 2021.
\bibitem[Meng et~al.(2022)Meng, He, Song, Song, Wu, Zhu, and Ermon]{sdedit}
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
\newblock {SDE}dit: Guided image synthesis and editing with stochastic differential equations.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\bibitem[Mokady et~al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or]{mokady2023null}
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.
\newblock Null-text inversion for editing real images using guided diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 6038--6047, 2023.
\bibitem[OpenAI(2024)]{openai2024chatgpt4o}
OpenAI.
\newblock Chatgpt-4o (october 2024 version).
\newblock \url{https://chat.openai.com/chat}, 2024.
\bibitem[Parmar et~al.(2023{\natexlab{a}})Parmar, Kumar~Singh, Zhang, Li, Lu, and Zhu]{parmar2023zero}
Gaurav Parmar, Krishna Kumar~Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.
\newblock Zero-shot image-to-image translation.
\newblock In \emph{ACM SIGGRAPH 2023 Conference Proceedings}, pages 1--11, 2023{\natexlab{a}}.
\bibitem[Parmar et~al.(2023{\natexlab{b}})Parmar, Kumar~Singh, Zhang, Li, Lu, and Zhu]{pix2pix_zero}
Gaurav Parmar, Krishna Kumar~Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.
\newblock Zero-shot image-to-image translation.
\newblock In \emph{ACM SIGGRAPH 2023 Conference Proceedings}, New York, NY, USA, 2023{\natexlab{b}}. Association for Computing Machinery.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et~al.]{radford2021learning}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et~al.
\newblock Learning transferable visual models from natural language supervision.
\newblock In \emph{International conference on machine learning}, pages 8748--8763. PMLR, 2021.
\bibitem[Rombach et~al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer]{rombach2022high}
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj{\"o}rn Ommer.
\newblock High-resolution image synthesis with latent diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 10684--10695, 2022.
\bibitem[Rotstein et~al.(2024)Rotstein, Bensa{\"\i}d, Brody, Ganz, and Kimmel]{rotstein2024fusecap}
Noam Rotstein, David Bensa{\"\i}d, Shaked Brody, Roy Ganz, and Ron Kimmel.
\newblock Fusecap: Leveraging large language models for enriched fused image captions.
\newblock In \emph{Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}, pages 5689--5700, 2024.
\bibitem[Saharia et~al.(2022)Saharia, Chan, Saxena, Li, Whang, Denton, Ghasemipour, Gontijo~Lopes, Karagol~Ayan, Salimans, et~al.]{saharia2022photorealistic}
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily~L Denton, Kamyar Ghasemipour, Raphael Gontijo~Lopes, Burcu Karagol~Ayan, Tim Salimans, et~al.
\newblock Photorealistic text-to-image diffusion models with deep language understanding.
\newblock \emph{Advances in neural information processing systems}, 35:\penalty0 36479--36494, 2022.
\bibitem[Shi et~al.(2024)Shi, Liew, Yan, Tan, and Feng]{shi2024instadrag}
Yujun Shi, Jun~Hao Liew, Hanshu Yan, Vincent~YF Tan, and Jiashi Feng.
\newblock Instadrag: Lightning fast and accurate drag-based image editing emerging from videos.
\newblock \emph{arXiv preprint arXiv:2405.13722}, 2024.
\bibitem[Singer et~al.(2023)Singer, Sheynin, Polyak, Ashual, Makarov, Kokkinos, Goyal, Vedaldi, Parikh, Johnson, et~al.]{singer2023text}
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et~al.
\newblock Text-to-4d dynamic scene generation.
\newblock \emph{arXiv preprint arXiv:2301.11280}, 2023.
\bibitem[Srivastava et~al.(2015)Srivastava, Mansimov, and Salakhudinov]{srivastava2015unsupervised}
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.
\newblock Unsupervised learning of video representations using lstms.
\newblock In \emph{International conference on machine learning}. PMLR, 2015.
\bibitem[Tulyakov et~al.(2018)Tulyakov, Liu, Yang, and Kautz]{tulyakov2018mocogan}
Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz.
\newblock Mocogan: Decomposing motion and content for video generation.
\newblock In \emph{CVPR}, 2018.
\bibitem[Vondrick et~al.(2016)Vondrick, Pirsiavash, and Torralba]{vondrick2016generating}
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.
\newblock Generating videos with scene dynamics.
\newblock In \emph{NeurIPS}, 2016.
\bibitem[Wang et~al.(2023)Wang, Lv, Yu, Hong, Qi, Wang, Ji, Yang, Zhao, Song, et~al.]{wang2023cogvlm}
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et~al.
\newblock Cogvlm: Visual expert for pretrained language models.
\newblock \emph{arXiv preprint arXiv:2311.03079}, 2023.
\bibitem[Wang et~al.(2024)Wang, Lv, Yu, Hong, Qi, Wang, Ji, Yang, Zhao, Song, Xu, Xu, Li, Dong, Ding, and Tang]{wang2024cogvlmvisualexpertpretrained}
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang.
\newblock Cogvlm: Visual expert for pretrained language models, 2024.
\bibitem[Wasserman et~al.(2024)Wasserman, Rotstein, Ganz, and Kimmel]{wasserman2024paint}
Navve Wasserman, Noam Rotstein, Roy Ganz, and Ron Kimmel.
\newblock Paint by inpaint: Learning to add image objects by removing them first.
\newblock \emph{arXiv preprint arXiv:2404.18212}, 2024.
\bibitem[Yang et~al.(2024)Yang, Teng, Zheng, Ding, Huang, Xu, Yang, Hong, Zhang, Feng, et~al.]{yang2024cogvideox}
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et~al.
\newblock Cogvideox: Text-to-video diffusion models with an expert transformer.
\newblock \emph{arXiv preprint arXiv:2408.06072}, 2024.
\bibitem[Zhang et~al.(2024)Zhang, Mo, Chen, Sun, and Su]{zhang2024magicbrush}
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su.
\newblock Magicbrush: A manually annotated dataset for instruction-guided image editing.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Zhang et~al.(2018)Zhang, Isola, Efros, Shechtman, and Wang]{zhang2018perceptual}
Richard Zhang, Phillip Isola, Alexei~A Efros, Eli Shechtman, and Oliver Wang.
\newblock The unreasonable effectiveness of deep features as a perceptual metric.
\newblock In \emph{CVPR}, 2018.
\bibitem[Zhang et~al.(2025)Zhang, Yu, Wu, Feng, Zheng, Snavely, Wu, and Freeman]{zhang2025physdreamer}
Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon~Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William~T Freeman.
\newblock Physdreamer: Physics-based interaction with 3d objects via video generation.
\newblock In \emph{European Conference on Computer Vision}. Springer, 2025.
\end{thebibliography}