Boost Video Frame Interpolation via Motion Adaptation
Extracted source text · arXiv 2306.13933
\documentclass{bmvc2k}
\usepackage{booktabs, multirow, makecell}
\usepackage{wrapfig}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{bbding}
\usepackage{etoolbox}
\definecolor{bmvc_blue}{RGB}{0, 0, 102}
\usepackage{caption}
\captionsetup{margin=3pt, font=small, labelfont={color=bmvc_blue, bf}, labelsep=period, skip=5pt}
\title{Boost Video Frame Interpolation via \\ Motion Adaptation}
% Enter the paper's authors in order
% \addauthor{Name}{email/homepage}{INSTITUTION_CODE}
\addauthor{Haoning Wu}{whn15698781666@sjtu.edu.cn}{1}
\addauthor{Xiaoyun Zhang$^\dagger$}{xiaoyun.zhang@sjtu.edu.cn}{1}
\addauthor{Weidi Xie}{weidi@sjtu.edu.cn}{1,2}
\addauthor{Ya Zhang}{ya_zhang@sjtu.edu.cn}{1,2}
\addauthor{Yanfeng Wang$^\dagger$}{wangyanfeng622@sjtu.edu.cn}{1,2}
% Enter the institutions
% \addinstitution{Name\\Address}
\addinstitution{
Coop. Medianet Innovation Center,\\
Shanghai Jiao Tong University, China\\
}
\addinstitution{
Shanghai AI Laboratory, China\\
}
\runninghead{WU, ET.AL}{Boost Video Frame Interpolation via Motion Adaptation}
\def\eg{\emph{e.g}\bmvaOneDot}
\def\Eg{\emph{E.g}\bmvaOneDot}
\def\etal{\emph{et al}\bmvaOneDot}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\weidi}[1]{{\textcolor{magenta}{Weidi: #1} }}
\newcommand{\haoning}[1]{{\textcolor{blue}{Haoning: #1} }}
\newcommand\blfootnote[1]{%
\begingroup
\renewcommand\thefootnote{}\footnote{#1}%
\addtocounter{footnote}{-1}%
\endgroup
}
\begin{document}
\maketitle
\blfootnote{$\dagger$: Corresponding author.}
\begin{abstract}
%-------------------------------------------------------------------------
\input{section/0_abstract}
%-------------------------------------------------------------------------
\end{abstract}
\begin{figure}[t]
\footnotesize
\centering
\includegraphics[width=.99\textwidth]{images/teaser.pdf}
\vspace{-0.1cm}
\caption{\textbf{High-level idea overview.}
(a) To address the generalisation challenge of VFI models due to domain gap on unseen data, we propose the optimisation-based video frame interpolation. By performing test-time motion adaptation on our proposed lightweight {\em adapter}, we achieve the generalization of VFI models across different video scenarios and subsequently boost their performance.
(b) Visual comparison on the cases with complex and large-scale motions from DAVIS~\cite{perazzi2016benchmark} dataset. Our method assists VFI models in generalising to diverse scenarios and synthesizing high-quality frames with clearer structures and fewer distortions.
}
\label{fig:teaser}
\vspace{-0.4cm}
\end{figure}
\section{Introduction}
\label{sec:intro}
%-------------------------------------------------------------------------
\input{section/1_introduction}
%-------------------------------------------------------------------------
\section{Related Work}
%-------------------------------------------------------------------------
\input{section/2_relatedwork}
%-------------------------------------------------------------------------
\section{Methods}
%-------------------------------------------------------------------------
\input{section/3_methods}
%-------------------------------------------------------------------------
\section{Experiments}
%-------------------------------------------------------------------------
\input{section/4_experiments}
%-------------------------------------------------------------------------
%-------------------------------------------------------------------------
\section{Results}
%-------------------------------------------------------------------------
\input{section/5_results}
%-------------------------------------------------------------------------
\section{Conclusion}
%-------------------------------------------------------------------------
\input{section/6_conclusion}
%-------------------------------------------------------------------------
\input{section/8_acknowledgement}
\bibliography{egbib}
\clearpage
\appendix
\noindent{\bf \color{bmvc_blue} \Large Appendix}
\vspace{0.3cm}
\input{section/7_supplementary}
\end{document}
%Existing learning-based video frame interpolation (VFI) methods have investigated various ways to ameliorate performance, such as refining the model architecture or incorporating extra input information. However, due to the limited motion distribution of training datasets, it is still challenging for VFI models to generalise to data with diverse motions. In order to tackle this issue, we explore an alternative direction and propose optimisation-based video frame interpolation via test-time motion adaptation. To this end, we present a strategy denoted as {\em cycle-consistency adaptation} that is suitable for VFI task. By fully leveraging the motion characteristics among video sequence during test time, we can significantly boost the generalisation ability and performance of existing VFI models. To further improve the efficiency of adaptation, we introduce a lightweight \textbf{Adapter} that can be inserted into the motion estimation module of existing pre-trained VFI models, achieving twofold improvement in efficiency and similar performance gain compared with end-to-end finetuning. Extensive experiments demonstrate that our approach can boost two-frame VFI models to perform favorably against the existing state-of-the-art methods on various benchmarks, and even outperform those with extra input frames.
Video frame interpolation (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video.
Existing learning-based VFI methods have achieved great success,
but they still suffer from limited generalization ability due to the limited motion distribution of training datasets.
In this paper, we propose a novel optimization-based VFI method that can adapt to unseen motions at test time. Our method is based on a {\em cycle-consistency adaptation} strategy that leverages the motion characteristics among video frames.
We also introduce a lightweight {\em adapter} that can be inserted into the motion estimation module of existing pre-trained VFI models to improve the efficiency of adaptation. Extensive experiments on various benchmarks demonstrate that our method can boost the performance of two-frame VFI models,
outperforming the existing state-of-the-art methods,
even those that use extra input frames.
Project page: \href{https://haoningwu3639.github.io/VFI_Adapter_Webpage/}{https://haoningwu3639.github.io/VFI\_Adapter\_Webpage/}
% \haoning{TODO: 1. Abstract, Conclusion; POLISH;}
%\weidi{consider to give a teaser.}
%\haoning{DONE}
%\haoning{The jounal version (TPAMI) of SAVFI (CVPR)~\cite{choi2020scene} (Scene-Adaptive Video Frame Interpolation via Meta-Learning) used the name (Test-Time Adaptation for Video Frame Interpolation via Meta-Learning). So, I want to avoid or replace this keyword.}
% Video Frame Interpolation (VFI) has been a long-standing fundamental computer vision task, which helps increase the temporal resolution of the given video. However, due to the limitations imposed by the motion distribution of the training dataset, it is challenging for a network to generalize to different video data with diverse motions. In order to tackle this issue, we attempt to leverage test-time adaptation to boost the generalization ability of VFI and propose two adaptation strategies that are suitable for VFI, termed Direct Adaptation and Cycle-Consistency Adaptation. To further improve the efficiency of adaptation, we introduce a lightweight and simple Adapter module that can be inserted into the motion estimation module of pre-trained VFI models to adjust the estimated optical flow according to the motion characteristics of different video data, synthesizing higher-quality intermediate frame. Extensive experiments demonstrate that our approach performs favorably against the existing state-of-the-art VFI methods on various video datasets, and even outperforms those with extra input frames.
%Video Frame Interpolation (VFI) aims to increase the temporal resolution of a video, by synthesizing intermediate frames for smoother transition. Such technique has a wide range of applications, {\em e.g.}, video compression~\cite{wu2018video, peleg2019imnet}, slow-motion generation~\cite{liu2017dvf} and novel-view synthesis~\cite{flynn2016deepstereo, kalantari2016learning}, \textit{etc}. In the literature, existing approaches can be roughly divided into two categories, namely flow-agnostic and flow-based frame synthesis, depending on whether motion compensation between frames is coupled with optical flow. Recently, thanks to the rapid progress on flow estimation~\cite{sun2018pwc, teed2020raft}, flow-based VFI approaches~\cite{niklaus2018context, jiang2018super, bao2019memc, hu2022many, jin2022unified} have evolved into a popular framework and exhibits promising performance.
Video frame interpolation (VFI) is a technique that increases the temporal resolution of a video by synthesizing intermediate frames between existing frames.
This results in smoother transitions between frames, which can improve the overall quality of the video. VFI has a wide range of applications,
including video compression~\cite{wu2018video, peleg2019imnet},
slow-motion generation~\cite{liu2017dvf} and novel-view synthesis~\cite{flynn2016deepstereo, kalantari2016learning}, \textit{etc}.
In the literature, existing VFI approaches can be roughly divided into two categories:
flow-agnostic and flow-based frame synthesis.
Flow-agnostic approaches do not use optical flow to compensate for motion between frames, instead, they typically use a combination of adaptive convolution kernel and interpolation techniques.
Flow-based approaches, on the other hand, do use optical flow to estimate the motion between frames. This information is then used to synthesize the intermediate frames.
Recently, thanks to the rapid progress in optical flow estimation~\cite{sun2018pwc, teed2020raft}, flow-based VFI approaches~\cite{niklaus2018context, jiang2018super, bao2019memc, hu2022many, jin2022unified} have become the dominant approach in the field.
Existing flow-based frame interpolation models are learning-based,
that usually follow a similar pipeline: extract visual features from the input frames,
estimate optical flows between the reference and target frame to be synthesized, warp the input frames and their contextual features based on the estimated flows, then finally synthesize the intermediate frame based on the aligned visual features.
Under such design, various model architectures have been explored,
such as constructing multi-scale pyramids~\cite{kong2022ifrnet, huang2022real, jin2022unified} and designing vision transformer~\cite{shi2022vfit, lu2022vfiformer}.
Additionally, the incorporation of extra input information, such as depth~\cite{bao2019depth}, or more adjacent frames~\cite{xu2019quadratic, liu2020enhanced, kalluri2023flavr, shi2022vfit}, have also been explored as potential solutions.
Despite training such meticulously-designed models on a large amount of videos,
generalisation towards real videos with complex and large-scale motions still remains challenging.
Unlike existing approaches on architecture design,
we explore an alternative direction for boosting models' performance at inference time,
{\em i.e.} optimisation-based frame interpolation via test-time motion adaptation.
Test-time adaptation has been proven effective in enhancing the performance of models in specific scenarios, as demonstrated in numerous computer vision tasks,
such as image super-resolution~\cite{michaeli2013nonparametric, shocher2018zero, huang2015single} and image deblurring~\cite{chi2021testdeblur},
{\em etc}, it remains unexplored in the domain of video frame interpolation.
To this end, we devise a novel strategy suitable for video frame interpolation,
namely {\em cycle-consistency adaptation}.
The key idea is to construct triplet samples with consecutive frames from low frame-rate videos during test time and optimise model parameters on each video sequence by leveraging the inter-frame consistency. Considering that the adaptation process necessitates considerable inference time, we further propose a simple, yet effective {\em adapter} that can be injected into the existing motion estimation module of VFI, enabling motion refinement with less than $4\%$ trainable parameters comparing to the original model.
% \weidi{summary of contribution.}
To summarise, we make the following contributions in this paper:
(i) to improve the generalisation ability of existing VFI models and boost their performance, we propose an optimisation-based motion adaptation strategy based on cycle-consistency,
that allows to tune the model on each test video sequence at inference time;
(ii) to address the drawback of high time cost associated with test-time adaptation,
we design a simple, yet effective plug-in {\em adapter} to refine the motion flow estimated by VFI models, with minimal tuning cost;
(iii) we experiment on various models and benchmarks,
demonstrating that our optimisation-based method can always boost the performance of existing two-frame VFI models, even outperforming approaches with extra inputs.
%In this section, we review literature on video frame interpolation, especially learning-based methods.
%Furthermore, various computer vision algorithms that have applied cycle-consistency and test-time adaptation have also been described.
\vspace{3pt}
\noindent {\textbf{Video Frame Interpolation.}}
Video frame interpolation (VFI) is a long-standing computer vision research topic and has been widely studied.
The recent literature on training deep neural networks has demonstrated extraordinary performance for video frame interpolation.
Depending on whether optical flow is used,
VFI methods can be broadly classified into two categories:
flow-agnostic~\cite{niklaus2017adaconv, niklaus2017sepconv, choi2020channel, cheng2021multiple, kalluri2023flavr, lee2020adacof, shi2021video} and flow-based ones~\cite{niklaus2018context, jiang2018super, niklaus2020softmax, park2020bmbc, park2021asymmetric, bao2019memc, hu2022many, jin2022unified, jin2023enhanced}.
With the rapid development of optical flow estimation algorithms~\cite{ilg2017flownet, hui2018liteflownet, sun2018pwc, teed2020raft}, flow-based approaches have taken the dominant position,
%\weidi{the following sentence is repeating Intro.}
which typically employ optical flow to warp visual features of adjacent frames to synthesize intermediate frame, hence the quality of the generated frame is highly affected by the accuracy of motion estimation.
% which typically follow a similar paradigm: extracting features from adjacent frames for motion estimation, warping pixels or features based on the estimated optical flow, and merging aligned information from the adjacent frames for middle frame synthesis.
Various strategies have been explored to improve the performance of flow-based methods. These include exploring depth information for occlusion reasoning~\cite{bao2019depth}, guiding the learning of motion estimation via knowledge distillation~\cite{huang2022real, kong2022ifrnet}, designing efficient architectures for high-resolution videos with relatively large motion~\cite{sim2021xvfi, reda2022film}, utilizing the long-range dependency modeling capability of transformer for processing extensive motion~\cite{lu2022vfiformer, shi2022vfit}, and making full use of multiple adjacent frames for complex motion modeling~\cite{xu2019quadratic, liu2020enhanced, kalluri2023flavr, shi2022vfit}.
In contrast to the aforementioned learning-based methods that aim to enhance VFI model generalisation and performance by modifying model architecture or incorporating extra input information, we consider an optimisation-based video frame interpolation that adapts pre-trained VFI models to the motion patterns in different video sequences at inference time.
% Considering that only linear motion can be modeled with two adjacent frames, several methods have attempted to break this prior constraint by utilizing multiple adjacent frames to synthesize more high-quality middle frames. QVI~\cite{xu2019quadratic} exploits the acceleration information in videos for high-order motion modeling, which allows prediction with curvilinear trajectory and variable velocity. EQVI~\cite{liu2020enhanced} further adopts a rectified quadratic flow formulation for accurate motion estimation. FLAVR~\cite{kalluri2023flavr} proposed using 3D spatio-temporal convolution to implicitly learn reasoning the nonlinear motion and complex occlusion among multiple frames. VFIT~\cite{shi2022vfit} constructs a spatial-temporal Transformer that allows modeling long-range dependencies among pixels.
% Many existing works have already explored the use of cycle constraints to regularize the loss functions. This has been widely used in many self-supervised methods, such as BYOL~\cite{grill2020bootstrap} and DINO~\cite{caron2021emerging}.
\vspace{5pt}
\noindent {\textbf{Cycle Consistency.}}
The idea of cycle consistency has been widely explored in various self-supervised methods, such as image representation and correspondence learning~\cite{wu2021contrastive, wang2019learning, lai2019selfsupervised, zhou2016learning, Dwibeditemporal}.
For low-level vision tasks, CycleGAN~\cite{zhu2017unpaired} utilizes cycle-consistency loss to constrain the training process of generative adversarial networks on image2image translation task. ARIS~\cite{zhou2022aris} exploits cycle-consistency constraint to augment the models' ability for arbitrary super-resolution. CyclicGen~\cite{liu2019cyclicgen} and Reda \etal \cite{reda2019unsupervised} learn video frame interpolation in an unsupervised manner with the proposed cycle-consistency loss on general video data.
\vspace{5pt}
\noindent {\textbf{Test-time Adaptation.}} Test-time adaptation manages to adapt the trained model to test data distribution for performance improvement, and has been successful in tasks such as classification~\cite{sun2020test} and pose estimation~\cite{li2021test}, {\em etc}.
Moreover, the idea has also been widely employed in low-level vision, such as super-resolution~\cite{michaeli2013nonparametric, shocher2018zero, huang2015single}, in order to improve the generalisation ability of models on various data. For video frame interpolation, SAVFI~\cite{choi2020scene} utilizes a meta-learning framework for training models to achieve scene-adaptive frame synthesis.
In this paper, we exploit {\em cycle-consistency} for test-time motion adaptation in video frame interpolation task, optimising motion for test video sequences on-the-fly, thus improving performance steadily. Moreover, a lightweight yet effective plug-in {\em adapter} has been proposed to improve the efficiency of VFI test-time adaptation.
%\weidi{Cycle consistency and test-time adaptation can be separated into two paragraphs, in cycle consistency, can also mention cycleGAN.
%in test-time adaptation, this was originally used for classification or discriminative tasks, to bridge the domain gap, check papers from Alexei Efros, Xiaolong Wang.} \haoning{DONE.}
%\weidi{reword the above sentence}
% For video frame interpolation, SAVFI~\cite{choi2020scene} presents implementing scene-adaptive frame generation using a meta-learning framework based on Model-Agnostic Meta-Learning (MAML).
%To further address the efficiency issue, we propose a lightweight and effective plug-in \textbf{Adapter} module, which not only maintains performance improvement, but also significantly raises efficiency.
In this section, we start by introducing the considered problem scenario,
{\em e.g.}, learning-based and our proposed optimisation-based video frame interpolation (Sec.~\ref{sec3.1}). Subsequently, we detail our proposed motion adaptation strategy that is suitable for video frame interpolation, namely {\em cycle-consistency adaptation} (Sec.~\ref{sec:3.2}). Lastly, to further improve the efficiency of test-time adaptation,
we present a lightweight {\em adapter} that can serve as a plug-in module for better motion estimation in VFI models (Sec.~\ref{sec:3.3}).
\subsection{Problem Scenario}
\label{sec3.1}
Given a low frame-rate input video, the goal of video frame interpolation is to synthesize intermediate frame between two or multiple adjacent frames, ending up of high-frame videos with smoother motion.
In a learning-based VFI model, it typically takes two consecutive frames ($\mathcal{I}_{i-1}$ and $\mathcal{I}_{i+1}$) as input and outputs one single intermediate frame ($\mathcal{I}_{i}$) between them, the model's parameters are learnt by minimizing the empirical risk on a training set:
\begin{align}
\begin{split}
&\mathcal{L}_{\mathcal{D}}(\Theta) = \mathbb{E}_{\mathcal{D}} (||\hat{\mathcal{I}}_{i} - \mathcal{I}_i||) \qquad
\text{where \hspace{0.06cm}} \hat{\mathcal{I}}_{i} = \Phi(\mathcal{I}_{i-1}, \mathcal{I}_{i+1}; \Theta)
\end{split}
\end{align}
$\Phi(\cdot)$ refers to the video frame interpolation model,
$\Theta$ denotes the parameters to be learnt on a large-scale training set~($\mathcal{D}$), and $\{\hat{\mathcal{I}}_{i}\}$ denote the predicted intermediate frames.
At inference time, the model is expected to generalise towards unseen videos.
However, in practise, these models can sometimes be fragile on cases with diverse and complex motions.
%However, even with a massive training set such as Vimeo90K~\cite{xue2019toflow}, the diversity of motion included therein remains limited. This makes it difficult for the model to generalise well on out-of-distribution data with complex and large-scale motion.
% usefulness
In this paper, we consider to improve the model's efficacy, with optimisation-based video frame interpolation via test-time motion adaptation:
\begin{equation}
\hat{\Theta}_{\mathcal{V}} = \arg \min_{\Theta} \mathcal{L}_{\mathcal{V}}(\Phi, \mathcal{V}; \Theta)
\end{equation}
where $\Phi(\cdot)$ denotes a pre-trained VFI model with parameters $\Theta$, and we aim to further optimise its parameters to boost the performance on one given test video sequence~($\mathcal{V}$).
In the following section, we aim to answer the core question:
\textbf{how can we design the objective function~($\mathcal{L}_{\mathcal{V}}$),
given only low frame-rate videos are presented at inference time ?}
\subsection{Cycle-Consistency Adaptation}
\label{sec:3.2}
At inference time, we construct a series of triplet samples from the given test video sequence, each consists of three consecutive frames,
for example, $\mathcal{D}_1 = \{\mathcal{I}_1, \mathcal{I}_3, \mathcal{I}_5\}$,
$\mathcal{D}_2 = \{\mathcal{I}_3, \mathcal{I}_5, \mathcal{I}_7\}$, {\em etc}.
Our goal is to optimise the model's parameters using $\mathcal{D}_1$,
and to boost the performance on synthesizing intermediate frames $\{\hat{{\mathcal{I}}}_{2}, \hat{{\mathcal{I}}}_{4}\}$ by exploiting the cycle-consistency constraint.
As detailed in Figure~\ref{fig:method} (a),
taking the triplet $\mathcal{D}_1 = \{\mathcal{I}_1, \mathcal{I}_3, \mathcal{I}_5\}$ for demonstration, we can generate the intermediate frames with a pre-trained off-the-shelf VFI model on input consecutive video frames,
\begin{equation}
\begin{split}
\hat{\mathcal{I}}_2 = \Phi(\mathcal{I}_1, \mathcal{I}_3; \Theta) \qquad
\hat{\mathcal{I}}_4 = \Phi(\mathcal{I}_3, \mathcal{I}_5; \Theta)
\end{split}
\end{equation}
and reuse the synthesized frames to predict the target frame:
\begin{equation}
\hat{\mathcal{I}}_3 = \Phi(\hat{\mathcal{I}}_2, \hat{\mathcal{I}}_4; \Theta)
\end{equation}
The model's parameters $\Theta$ can be updated according to:
\begin{equation}
\begin{split}
&\Theta \leftarrow \Theta - \eta \nabla_{\Theta} \mathcal{L}_{\mathcal{V}}(\hat{\mathcal{I}}_3, \mathcal{I}_3) \quad \text{where \hspace{0.04cm}} \mathcal{L}_{\mathcal{V}}(\Theta) = ||\hat{\mathcal{I}}_{3} - \mathcal{I}_3||
\end{split}
\end{equation}
$\eta$ denotes the adaptation learning rate.
This cycle-consistency adaptation strategy enables the model to make full use of inter-frame consistency, thereby acquiring motion patterns that are more suitable for testing scenarios and achieving stable performance improvements.
\begin{figure}[t]
\footnotesize
\centering
\includegraphics[width=\textwidth]{images/method.pdf}
% \begin{tabular}{cc} \bmvaHangBox{\includegraphics[width=.4\textwidth]{images/adap_strategy.pdf}} &
% \bmvaHangBox{\includegraphics[width=.52\textwidth]{images/arch.pdf}} \\
% (a) Cycle-Consistency Adaptation Strategy & (b) Plug-in Adapter module
% \end{tabular}
\vspace{-0.4cm}
\caption{\textbf{Proposed {\em cycle-consistency adaptation} strategy and plug-in {\em adapter} module for efficient test-time adaptation.} (a) {\em Cycle-consistency adaptation} first synthesizes intermediate frames between each two input frames and reuses them to interpolate the target frame to calculate cycle-loss, which fully utilizes the consistency within video sequences.
(b) To improve efficiency, we freeze all the parameters of pre-trained VFI models and solely optimise the proposed plug-in {\em adapter}, which predicts a set of parameters $\{\alpha, \beta\}$ based on the extracted visual features. The pixel-wise weights $\alpha$ and biases $\beta$ are used for rectifying the estimated flow to fit each video sequence.}
\label{fig:method}
\vspace{-0.2cm}
\end{figure}
\subsection{Lightweight Motion Adaptation with Plug-in Adapter}
\label{sec:3.3}
Optimising the entire model for test-time motion adaptation incurs computation overhead, here, we further propose a simple, lightweight plug-in {\em adapter} module that can be inserted into the motion estimation module of existing pre-trained VFI models, requiring minimal tuning to boost the performance.
As depicted in Figure \ref{fig:method} (b), we freeze all parameters of the pre-trained VFI model, {\em e.g.}, feature extraction, motion estimation and frame synthesis modules, then incorporate the proposed adapter into the motion estimation module, which takes visual features extracted from adjacent frames as input, and predicts a set of parameters $\{\alpha, \beta\}$ to adjust motion estimation per video sequence.
%the efficiency of fine-tuning the whole model is unsatisfactory. In order to boost the performance of VFI model as efficiently as possible through test-time adaptation, we propose utilizing extracted neighboring frame features to predict correction parameters for the estimated motion flow.
To be specific, the adapter reuses the convolutional features from motion estimation module, and subsequently employs a $1\times 1$ Convolution layer to transform the feature map into pixel-wise weights $\alpha$ and biases $\beta$:
\begin{equation}
\hat{F} = \alpha F + \beta \text{,\hspace{0.1cm} where \hspace{0.04cm}}\{\alpha, \beta\} = \text{Conv}(\Psi(M))
\end{equation}
$M$ represents the extracted visual feature map and $\Psi$ denotes the resused convolutional layers, then the predicted $\{\alpha, \beta\}$ are used to modify the estimated motion flow $F$. During test-time adaptation, we only finetune the parameters in adapter,
effectively refining the estimated motion flow and boosting the performance of video frame interpolation model.
%perform optimization-based video frame interpolation by
%Adapter module on the original train set and fitting it to the VFI model,
% on specific test data to adapt to unique motions in different scenes, and subsequently
% to guarantee cycle-consistency among frames of the video sequence. This necessitates multiple iterations of inference to optimize the model parameters, but this efficiency trade-off enables the model to make full use of inter-frame consistency, thereby acquiring motion patterns that are more suitable for testing scenarios and achieving stable performance improvements.
% Considering the influence of temporal distance between frames on the motion estimation prior learned by the video frame interpolation, we aim to optimize model parameters to better align with the test scenario by fully leveraging consistency among frames, rather than naively synthesizing intermediate $\hat{I}_3$ with $I_1$ and $I_5$ as inputs, which would lead to a discrepancy between the adapted motion prior and the testing scene.
% Regarding the optimization-based video frame interpolation mentioned above, we present two test-time adaptation strategies, namely direct adaptation and cycle-consistency adaptation. As depicted in Figure~\ref{fig:adap_strategy}, direct adaptation simply utilizes $I_1$ and $I_5$ as inputs for the VFI model to synthesize the intermediate frame $\hat{I}_3 = \Phi(I_1, I_5; \theta)$. Conversely, cycle-consistency adaptation first employs the input frame sequence to generate intermediate frames: $\hat{I}_2 = \Phi(I_1, I_3; \theta)$ and $\hat{I}_4 = \Phi(I_3, I_5; \theta)$, and then predicts $\hat{I}_3 = \Phi(\hat{I}_2, \hat{I}_4; \theta)$ to guarantee cycle-consistency among frames of the video sequence.
% In order to fine-tune the model using $4$ input frames during the test time, we can construct $2$ triplet training samples with each sequence of three consecutive video frames, denoted as $\mathcal{D}_1 = \{I_1, I_3, I_5\}$ and $\mathcal{D}_2 = \{I_3, I_5, I_7\}$. The ultimate goal of adaptation on $\mathcal{D}_1$ and $\mathcal{D}_2$ is to boost the test performance on the testing triplet $\mathcal{T} = \{I_3, I_4, I_5\}$, that is, to synthesize higher-quality target frame $\hat{I}_4$ with $I_3$ and $I_5$ as input frames. \weidi{$I_3$ and $I_5$ are given, they don't need to be tested, right ?}
% there are two distinct adaptation strategies that can be employed during testing: Direct Adaptation and Cycle-Consistency Adaptation. As depicted in Figure~\ref{fig:adap_strategy}, direct adaptation involves utilizing $I_1$ and $I_5$ as inputs for the VFI model to synthesize the intermediate frame $\hat{I}_3 = f_{\theta}(I_1, I_5)$, and computing the loss $\mathcal{L}$ between the interpolated frame $\hat{I}_3$ and $I_3$ in the input sequence via $\mathcal{L}_{D} = ||\hat{I}_3 - I_3||_1$ to update the model's parameters $\theta$ by gradient descent:
% \begin{equation}
% \theta \leftarrow \theta - \eta \nabla_{\theta} \mathcal{L}_{D}
% \end{equation}
% Conversely, cycle-consistency adaptation employs the input frame sequence to generate intermediate frames: $\hat{I}_2 = f_{\theta}(I_1, I_3)$ and $\hat{I}_4 = f_{\theta}(I_3, I_5)$, and iteratively predicting $\hat{I}_3 = f_{\theta}(\hat{I}_2, \hat{I}_4)$, then calculating the cycle loss $\mathcal{L}_{CC} = ||\hat{I}_3 - I_3||_1$ to update the parameters:
% \begin{equation}
% \theta \leftarrow \theta - \eta \nabla_{\theta} \mathcal{L}_{CC}
% \end{equation}
% \input{table/table1}
% Both of the aforementioned adaptation methods can be applied to the input sequence of video frame interpolation task. The process of direct adaptation is more succinct, thus exhibiting higher efficiency. Nevertheless, excessive direct adaptation may lead to a deterioration in performance due to the noticeably larger temporal distance between frames in the adaptation sample in comparison to the target testing sample. In comparison, to ensure cycle-consistency within the video sequence, the cycle-consistency adaptation necessitates multiple iterations of inference to optimize the model parameters. This efficiency trade-off enables the model to fully leverage inter-frame consistency, thereby acquiring motion patterns that are more suitable for testing scenarios and achieving stable performance improvements. To quantitatively compare the feasibility of the aforementioned two adaptation strategies, we conducted a rudimentary test on the Vimeo90K-Septuplet~\cite{xue2019toflow} dataset using both IFRNet~\cite{kong2022ifrnet} and UPRNet~\cite{jin2022unified}, and present the results in Table~\ref{tab:table1}.
% Direct Adaptation requires fewer inference times, thus exhibiting higher efficiency. However, its time stride for fine-tuning on $\mathcal{D}_i$ greater than that of testing on $\mathcal{T}$, and excessive adaptation may lead to a performance deterioration. In comparison, Cycle-Consistency Adaptation demands more inference times to calculate adaptation loss and update parameters. Nevertheless, sufficiently leveraging inter-frame consistency allows it to learn motions that better correspond to testing scenarios.
%This optimization-based video frame interpolation can be formulated as:
%by constructing triplets samples within the video sequence to optimize the loss function $\mathcal{L}_{\mathcal{V}}$, thus
% The standard setting for most VFI models involves taking two consecutive frames as input and outputting a single intermediate frame. Specifically, if we denote the two consecutive input frames as ${I}_1$ and ${I}_3$, our goal is to synthesize the middle frame $\hat{{I}}_{2}$.
% \weidi{give some math notations that can be used in the following sections:}
% As for real video data, the VFI model is capable of obtaining more than two input frames, therefore, some methods leverage more neighboring frames $\{{I}_1, {I}_3, {I}_5, {I}_7\}$ to interpolate the middle frame ${I}_4$.
% \weidi{unify the two scenarios by feeding a frame set as input, can be 2 or 4 frames, don't say for real video data.}
% \weidi{what existing approach do, for example, the $\Theta$ is obtained by optimising what, empirical risk ? encourage generalisation on a held-out test set, blablabla...}
% \weidi{The following paragraph is not satisfactory, it should be something like: ``In this paper, we consider an alternative approach,
% what is that, optimisation based, what is the formulation,
% for example, for one test video, how do we compute the $\hat{\Theta} = \argmin \mathcal{L}_{\text{video}} $.....''}
% In order to better explore the effectiveness of test-time adaptation in VFI tasks, we align our experimental set-up with those multi-frame methods. Specifically, we construct TTA samples using the input sequence $\{{I}_1, {I}_3, {I}_5, {I}_7\}$ and improve the performance of various two-frame input models in generating ${I}_4$ with $\{{I}_3, {I}_5\}$. This not only fully utilizes the multi-frame data in the video, but also offers more flexibility compared to the original multi-frame methods.
% \begin{figure}[!hbt]
% \centering
% \includegraphics[width=\textwidth]{images/adaptation.pdf} \\
% \caption{\textbf{Proposed Cycle-Consistency Adaptation strategy and plug-in Adapter module}. (a) Cycle-Consistency Adaptation fully utilizes the consistency among video sequence and optimizes the parameters via cycle loss.}
% \label{fig:adap_strategy}
% %\vspace{-10pt}
% \end{figure}
%Given a pre-trained two-frame VFI model $\Phi$ with parameters $\theta$, we aim to perform per-sequence adaptation on each test video to fine-tune the model parameters for scenarios with different motions, thereby unleashing the potential of the VFI model and boosting its performance. As described in Section~\ref{sec3.1},
In this paper, we start from a series of VFI models that have been pre-trained on Vimeo90K-Triplet dataset~\cite{xue2019toflow}, then train our proposed {\em adapter} module on the same dataset.
For test-time motion adaptation, the model is further optimised on each testing sequence from three benchmark datasets, including Vimeo90K-Septuplet~\cite{xue2019toflow}, DAVIS~\cite{perazzi2016benchmark}, and SNU-FILM~\cite{choi2020channel}.
\vspace{5pt}
\noindent{\bf Training Set.} Vimeo90K-Triplet~\cite{xue2019toflow} training set comprises 51,312 triplets, wherein each triplet contains three consecutive video frames with a spatial resolution of $448 \times 256$ pixels.
\vspace{5pt}
\noindent{\bf Testing Set.}
Vimeo90K-Septuplet~\cite{xue2019toflow} dataset encompasses 7,824 seven-frame sequences for testing, with a fixed spatial resolution of $448 \times 256$ pixels.
DAVIS~\cite{perazzi2016benchmark} is a typically high-quality video segmentation dataset with a fixed resolution of $854 \times 480$ pixels.
Following FLAVR~\cite{kalluri2023flavr} and VFIT~\cite{shi2022vfit},
we also report performance on 2,847 septuplet test samples from DAVIS.
SNU-FILM~\cite{choi2020channel} dataset contains 1,240 triplets,
with a predominant resolution of approximately $1280 \times 720$ pixels.
It comprises of four categories with ascending motion scales: easy, medium, hard, and extreme. We further sample the surrounding frames of the ground truth and extend each triplet into a septuplet.
As a result, the easy, medium, and hard categories contain 310 sequences,
while the extreme category contains 234 sequences. Among these septuplets in the above three benchmarks, the four odd frames compose the input video sequence of the VFI model, as described in Sec.~\ref{sec3.1}. And the intermediate frame is regarded as the ground truth for the frame to be interpolated in our experiments.
\vspace{5pt}
\noindent{\bf Evaluation Metrics.}
Following common practise, we report Peak Signal-to-Noise Ratio (PSNR) and Structural-Similarity-Image-Metric (SSIM) on the RGB channel of the target interpolated frame for the three benchmark datasets.
\vspace{5pt}
\noindent{\bf Training Details. }
To start with, we freeze the parameters of the pre-trained VFI models and only
train our proposed plug-in adapter on Vimeo90K for 30 epochs with a batch size of 16. We randomly crop $256 \times 256$ patches and augment the training data using horizontal and vertical flipping, temporal order reversing and RGB channel flipping. We use the AdamW~\cite{loshchilov2017fixing} optimizer with $\beta_1 = 0.9$ and $\beta_2 = 0.99$, and the learning rate is gradually reduced from $3 \times 10^{-4}$ to $3 \times 10^{-5}$ using cosine annealing during the whole training process. Considering the high resolution of some video data, optimisation-based VFI is performed on one 40G NVIDIA A100 GPU. During test-time adaptation for the full model, we use a fixed learning rate of $1 \times 10^{-5}$ and calculate L1 loss to fine-tune parameters of VFI models. As for adapter-boosted models, since other parameters have been frozen, we adapt the plug-in adapter module to each test sequence with a larger learning rate of $1 \times 10^{-4}$ for IFRNet~\cite{kong2022ifrnet}, $3 \times 10^{-4}$ for UPRNet~\cite{jin2022unified}, and $1 \times 10^{-3}$ for RIFE~\cite{huang2022real}, respectively.
In this section, we start by providing experimental results for comparison with existing state-of-the-art approaches~(Sec.~\ref{sec:comp}),
showing the effectiveness of our proposed {\em cycle-consistency adaptation}, in both end-to-end and plug-in adapter finetuning scenarios.
After that, we conduct a series of ablation studies on the critical design choices on our adaptation strategy and the plug-in adapter module~(Sec.~\ref{sec:abl}).
\subsection{Comparison to state-of-the-art}
\label{sec:comp}
\noindent {\bf Quantitative Results.}
We compare our boosted models with 9 representative learning-based models trained on Vimeo90K-Triplet~\cite{xue2019toflow}, including flow-free ones: SepConv~\cite{niklaus2017sepconv}, EDSC~\cite{cheng2021multiple} and FLAVR~\cite{kalluri2023flavr} and flow-based ones: RIFE~\cite{huang2022real}, UPRNet~\cite{jin2022unified} and \textit{etc}. Among them, FLAVR~\cite{kalluri2023flavr} and VFIT~\cite{shi2022vfit} take four frames as input, while others only use two adjacent frames.
Specifically, we consider two scenarios, namely, end-to-end finetuning (e2e),
or plug-in adapter finetuning (plugin), the former optimises all parameters in the model,
denoted as [model-ours-e2e],
while the latter only updates adapters, denoted as [model-ours-plugin].
By default, all test-time motion adaptations are only conducted for 10-step updates,
with one exception on [model-ours-e2e++], which has performed 30-step adaptation, aiming to show the performance variation with more optimisation steps.
% with only one exception on UPRNet-ours-e2e++, which has performed 30-step adaptation, aiming to show the performance variation with more optimisation steps.
\input{table/table1}
As shown in Table~\ref{tab:results}, we can draw the following three observations:
(i) comparing with the off-the-shelf VFI models, our proposed {\em cycle-consistency adaptation} strategy with end-to-end finetuning can always bring significant PSNR performance gain on all benchmarks, that confirms the universality of our approach;
(ii) the end-to-end adapted IFRNet-ours-e2e and UPRNet-ours-e2e have exhibited comparable performance to state-of-the-art methods, such as EMA-VFI~\cite{Zhang2023ExtractingMA} and {\em etc}. And notably, UPRNet-ours-e2e++ with further adaptation improves \textbf{0.83dB} (36.90dB vs 36.07dB) on Vimeo90K testset and has consistently exhibited a performance gain of over \textbf{0.47dB} on other benchmarks, even outperforming the methods that take multiple frames as input, showing the effectiveness of test-time motion adaptation for unleashing the potential of pre-trained two-frame VFI models; (iii) models with the proposed plug-in {\em adapter} module have exhibited similar performance improvement to end-to-end finetuning, simultaneously incurring efficiency and efficacy.
\vspace{3pt}
\noindent {\bf Qualitative Results.}
We demonstrate the qualitative results in Figure~\ref{fig:results}, with the following observations: (i) comparing with existing state-of-the-art methods, the images generated by the models boosted via end-to-end adaptation and plug-in adapter present more details and have higher fidelity; (ii) our test-time optimised models generate less motion blur in the synthesized intermediate frame, indicating that the model has better adapted to the special motion characteristics in each scenario, thus improving the quality of synthesized frames.
\begin{figure}[!htb]
\centering
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=\textwidth]{images/result1.pdf} \\
\vspace{-0.2cm}
\small
\caption*{(a) Qualitative Comparison on Vimeo90K~\cite{xue2019toflow}}
\vspace{0.1cm}
\end{minipage}
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=\textwidth]{images/result2.pdf} \\
\vspace{-0.2cm}
\small
\caption*{(b) Qualitative Comparison on SNU-FILM~\cite{choi2020channel} and DAVIS~\cite{perazzi2016benchmark}}
\end{minipage}
\vspace{-0.1cm}
\caption{\textbf{Qualitative comparison against the state-of-the-art VFI algorithms.} We show visualization on Vimeo90K~\cite{xue2019toflow}, SNU-FILM~\cite{choi2020channel} and DAVIS~\cite{perazzi2016benchmark} benchmarks for comparison. The patches for careful comparison are marked with \textcolor{red}{red} in the original images. Our boosted models can generate higher-quality results with clearer structures and fewer distortions.}
\label{fig:results}
\vspace{-0.4cm}
\end{figure}
% \begin{figure}[!htb]
% \centering
% \includegraphics[width=\textwidth]{images/result2.pdf} \\
% \caption{\textbf{Qualitative Comparison} against the state-of-the-art frame interpolation algorithms on SNU-FILM-Extreme~\cite{choi2020channel} and DAVIS~\cite{perazzi2016benchmark}. The patches for careful comparison are marked with \textcolor{red}{red} in the original images. On datasets with various motions, our optimization-based VFI demonstrates significantly stronger generalization ability.}
% \label{fig:results_SNU}
% %\vspace{-10pt}
% \end{figure}
\subsection{Ablation Studies}
\label{sec:abl}
In this section, we have conducted thorough ablation studies to quantitatively and qualitatively demonstrate the effectiveness of the proposed {\em cycle-consistency adaptation} strategy and {\em adapter} module from the perspectives of stability and efficiency.
\vspace{2.5pt}
\noindent{\bf Adaptation Approach.}
In addition to the {\em cycle-consistency adaptation} proposed by us,
we here consider a baseline approach for test-time adaptation,
which is to directly optimise the distance between $\mathcal{I}_3$ and $\hat{\mathcal{I}}_3$ synthesized with $\mathcal{I}_1$ and $\mathcal{I}_5$ as input, denoted as na\"ive optimisation. It is noteworthy that the inter-frame temporal distance during such adaptation is larger than that of test scenario.
As shown in Table \ref{tab:table2}, we compare the two adaption strategies on five VFI methods, and have the following observations:
(i) under the same adaptation steps, the performance gain of our proposed cycle-consistency adaptation is significantly higher than that of na\"ive adaptation;
(ii) as the steps of adaptation increase,
na\"ive adaptation may lead to a drop in performance improvement and even result in inferior performance compared to the original pre-trained models,
whereas cycle-consistency adaptation can steadily boost VFI models,
as it fully utilizes the inter-frame consistency to learn motion characteristics within the test video sequence.
\input{table/table2}
% As stated above, the time stride of fine-tuning samples constructed during Direct Adaptation is larger than that of the real testing samples, resulting in insufficient improvement in performance. Moreover, excessive fine-tuning may lead to decreased performance gain, while Cycle-Consistency Adaptation and plugin Adapter do not present such side effects.
% First, as mentioned in Section~\ref{sec:3.2}, we demonstrate the superiority of proposed Cycle-Consistency Adaptation over naive adaptation through experiments. The results of fine-tuning using both adaptation strategies on the Vimeo90K~\cite{xue2019toflow} testset are presented in Table~\ref{tab:table2}.
% \vspace{3pt}
% \noindent{\bf Trainable Parameters.}
%\haoning{I think the Trainable Parameters and Time Cost can be merged into one entry, namely, Adaptation Cost.}
\vspace{2.5pt}
\noindent{\bf Adaptation Cost.} As mentioned in Sec.~\ref{sec:3.3}, the proposed plug-in adapter is designed to improve the efficiency of test-time motion adaptation.
Here, we conduct end-to-end and plug-in adapter finetuning on three VFI models,
and compare the number of parameters to be optimised and the time required for each step of adaptation. The results in Table~\ref{table3} have illustrated that with the support of our proposed plug-in adapter, we can achieve a 2 times acceleration with less than 4\% parameters to be optimised, while maintaining inference efficiency and similar quantitative performance improvement comparing to end-to-end finetuning. This confirms the efficiency and feasibility of our proposed plug-in adapter.
\input{table/table3}
% We perform ablation studies on the effectiveness and efficiency of Cycle-Consistency Adaptation and our proposed plug-in Adapter module. As shown in Table \ref{tab:ablation}, we use the Vimeo90K~\cite{xue2019toflow} test set to ablate the performance and efficiency of two different adaptation strategies: Cycle-Consistency Adaptation and plugin Adapter with pre-trained RIFE~\cite{huang2022real}, IFRNet~\cite{kong2022ifrnet} and UPRNet~\cite{jin2022unified}. Compared to fine-tuning the entire model parameters, only adapting the plugin Adapter module significantly reduces time consumption, making it more efficient and practical.
\vspace{2.5pt}
\noindent{\bf Motion Field Visualization.} As stated in Sec.~\ref{sec:3.2}, our proposed cycle-consistency adaptation strategy enables VFI models to fully use inter-frame consistency, and thus acquire motion patterns that are more suitable for testing scenarios.
To qualitatively verify this idea, we visualize the motion fields estimated by the VFI model before and after adaptation.
Specifically, we compute the optical flow between the target frame and the reference frame by RAFT~\cite{teed2020raft} as motion ground truth,
and compare the motion fields estimated by UPRNet~\cite{jin2022unified} before and after adaptation.
As shown in Figure~\ref{fig:flow_adaptation}, the model boosted by our proposed motion adaptation can output \textbf{smoother motion fields} and \textbf{more precise motion edges}, leading to steady improvement in the quality of synthesized images.
\begin{figure}[t]
\footnotesize
\centering
\includegraphics[width=.99\textwidth]{images/flow_adaptation.pdf}
\vspace{-0.1cm}
\caption{\textbf{Motion field visualization.} The VFI model boosted by our proposed motion adaptation can estimate more precise motion fields, thereby producing synthesized frames with higher quality.
}
\label{fig:flow_adaptation}
\vspace{-0.4cm}
\end{figure}
% \begin{table}[!htb]
\begin{table}[t]
\scriptsize
\begin{center}
\tabcolsep=0.06cm
\begin{tabular}{lcccccccc}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{Methods}} & \multicolumn{2}{c}{Adaptation} & \multirow{2}{*}{Vimeo90K~\cite{xue2019toflow}} & \multirow{2}{*}{DAVIS~\cite{perazzi2016benchmark}} & \multicolumn{4}{c}{SNU-FILM~\cite{choi2020channel}} \\
\cmidrule{2-3} \cmidrule{6-9}
\multicolumn{1}{c}{} & e2e & plugin & & & Easy & Medium & Hard & Extreme \\
\midrule
% \XSolidBrush
SepConv~\cite{niklaus2017sepconv} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{33.72 / 0.9639} & \multicolumn{1}{c|}{26.65 / 0.8611} & \multicolumn{1}{c|}{40.21 / 0.9909} & \multicolumn{1}{c|}{35.45 / 0.9785} & \multicolumn{1}{c|}{29.62 / 0.9302} & \multicolumn{1}{c}{24.16 / 0.8457} \\
SepConv-ours-e2e & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{33.96 / 0.9650} & \multicolumn{1}{c|}{26.83 / 0.8639} & \multicolumn{1}{c|}{40.41 / 0.9911} & \multicolumn{1}{c|}{35.71 / 0.9794} & \multicolumn{1}{c|}{29.80 / 0.9313} & \multicolumn{1}{c}{24.26 / 0.8479} \\
EDSC~\cite{cheng2021multiple} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{34.55 / 0.9677} & \multicolumn{1}{c|}{26.83 / 0.8578} & \multicolumn{1}{c|}{40.66 / 0.9915} & \multicolumn{1}{c|}{35.77 / 0.9795} & \multicolumn{1}{c|}{29.75 / 0.9301} & \multicolumn{1}{c}{24.12 / 0.8420} \\
EDSC-ours-e2e & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{34.73 / 0.9685} & \multicolumn{1}{c|}{26.96 / 0.8600} & \multicolumn{1}{c|}{40.88 / 0.9917} & \multicolumn{1}{c|}{35.98 / 0.9803} & \multicolumn{1}{c|}{29.85 / 0.9313} & \multicolumn{1}{c}{24.19 / 0.8436} \\
\midrule
RIFE~\cite{huang2022real} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{35.28 / 0.9704} & \multicolumn{1}{c|}{27.61 / 0.8760} & \multicolumn{1}{c|}{40.74 / 0.9916} & \multicolumn{1}{c|}{36.18 / 0.9808} & \multicolumn{1}{c|}{30.30 / 0.9368} & \multicolumn{1}{c}{24.62 / 0.8531} \\
RIFE-ours-e2e & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{35.57 / 0.9717} & \multicolumn{1}{c|}{27.81 / 0.8798} & \multicolumn{1}{c|}{40.95 / 0.9918} & \multicolumn{1}{c|}{36.58 / 0.9816} & \multicolumn{1}{c|}{30.49 / 0.9386} & \multicolumn{1}{c}{24.71 / 0.8549} \\
RIFE-ours-e2e++ & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{35.93 / 0.9733} & \multicolumn{1}{c|}{28.10 / 0.8850} & \multicolumn{1}{c|}{41.20 / 0.9924} & \multicolumn{1}{c|}{36.94 / 0.9835} & \multicolumn{1}{c|}{30.83 / 0.9430} & \multicolumn{1}{c}{24.87 / 0.8589} \\
RIFE-ours-plugin & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\Checkmark} & \multicolumn{1}{c|}{35.56 / 0.9714} & \multicolumn{1}{c|}{27.76 / 0.8771} & \multicolumn{1}{c|}{40.99 / 0.9918} & \multicolumn{1}{c|}{36.55 / 0.9825} & \multicolumn{1}{c|}{30.48 / 0.9387} & \multicolumn{1}{c}{24.64 / 0.8533} \\
\midrule
IFRNet~\cite{kong2022ifrnet} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{35.86 / 0.9729} & \multicolumn{1}{c|}{28.03 / 0.8851} & \multicolumn{1}{c|}{40.91 / 0.9918} & \multicolumn{1}{c|}{36.58 / 0.9816} & \multicolumn{1}{c|}{30.75 / 0.9403} & \multicolumn{1}{c}{24.85 / 0.8590} \\
IFRNet-ours-e2e & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.38 / 0.9753} & \multicolumn{1}{c|}{28.45 / 0.8936} & \multicolumn{1}{c|}{41.21 / 0.9921} & \multicolumn{1}{c|}{37.03 / 0.9832} & \multicolumn{1}{c|}{31.10 / 0.9440} & \multicolumn{1}{c}{25.03 / 0.8634} \\
IFRNet-ours-e2e++ & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.68 / 0.9760} & \multicolumn{1}{c|}{28.78 / 0.8995} & \multicolumn{1}{c|}{41.48 / 0.9923} & \multicolumn{1}{c|}{37.57 / 0.9850} & \multicolumn{1}{c|}{31.45 / 0.9482} & \multicolumn{1}{c}{25.22 / 0.8694} \\
IFRNet-ours-plugin & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\Checkmark} & \multicolumn{1}{c|}{36.01 / 0.9734} & \multicolumn{1}{c|}{28.16 / 0.8825} & \multicolumn{1}{c|}{41.06 / 0.9920} & \multicolumn{1}{c|}{36.92 / 0.9834} & \multicolumn{1}{c|}{30.88 / 0.9404} & \multicolumn{1}{c}{24.93 / 0.8599} \\
\midrule
UPRNet~\cite{jin2022unified} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.07 / 0.9735} & \multicolumn{1}{c|}{28.38 / 0.8914} & \multicolumn{1}{c|}{41.01 / 0.9919} & \multicolumn{1}{c|}{36.80 / 0.9819} & \multicolumn{1}{c|}{31.22 / 0.9422} & \multicolumn{1}{c}{25.39 / 0.8648} \\
UPRNet-ours-e2e & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.68 / 0.9758} & \multicolumn{1}{c|}{\textcolor{blue}{\underline{28.84}} / \textcolor{blue}{\underline{0.8997}}} & \multicolumn{1}{c|}{41.31 / \textcolor{blue}{\underline{0.9923}}} & \multicolumn{1}{c|}{37.24 / 0.9836} & \multicolumn{1}{c|}{\textcolor{blue}{\underline{31.66}} / 0.9464} & \multicolumn{1}{c}{25.64 / 0.8699} \\
UPRNet-ours-e2e++ & \multicolumn{1}{c}{\Checkmark} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{\textcolor{red}{\textbf{36.90}} / \textcolor{blue}{\underline{0.9768}}} & \multicolumn{1}{c|}{\textcolor{red}{\textbf{29.15}} / \textcolor{red}{\textbf{0.9062}}} & \multicolumn{1}{c|}{\textcolor{red}{\textbf{41.48}} / \textcolor{red}{\textbf{0.9925}}} & \multicolumn{1}{c|}{\textcolor{red}{\textbf{37.66}} / \textcolor{red}{\textbf{0.9855}}} & \multicolumn{1}{c|}{\textcolor{red}{\textbf{32.00}} / \textcolor{red}{\textbf{0.9519}}} & \multicolumn{1}{c}{\textcolor{red}{\textbf{25.99}} / \textcolor{red}{\textbf{0.8798}}} \\
UPRNet-ours-plugin & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\Checkmark} & \multicolumn{1}{c|}{36.44 / 0.9751} & \multicolumn{1}{c|}{28.69 / 0.8945} & \multicolumn{1}{c|}{\textcolor{blue}{\underline{41.32}} / \textcolor{blue}{\underline{0.9923}}} & \multicolumn{1}{c|}{\textcolor{blue}{\underline{37.38}} / 0.9843} & \multicolumn{1}{c|}{31.64 / 0.9448} & \multicolumn{1}{c}{\textcolor{blue}{\underline{25.69}} / 0.8705} \\
\midrule
VFIformer~\cite{lu2022vfiformer} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.14 / 0.9738} & \multicolumn{1}{c|}{28.33 / 0.8898} & \multicolumn{1}{c|}{40.93 / 0.9918} & \multicolumn{1}{c|}{36.53 / 0.9815} & \multicolumn{1}{c|}{30.52 / 0.9392} & \multicolumn{1}{c}{24.92 / 0.8580} \\
EMA-VFI~\cite{Zhang2023ExtractingMA} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.23 / 0.9740} & \multicolumn{1}{c|}{28.07 / 0.8826} & \multicolumn{1}{c|}{41.04 / 0.9921} & \multicolumn{1}{c|}{36.73 / 0.9821} & \multicolumn{1}{c|}{30.88 / 0.9400} & \multicolumn{1}{c}{24.92 / 0.8580} \\
FLAVR~\cite{kalluri2023flavr} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.22 / 0.9746} & \multicolumn{1}{c|}{27.97 / 0.8806} & \multicolumn{1}{c|}{41.09 / 0.9918} & \multicolumn{1}{c|}{36.85 / 0.9830} & \multicolumn{1}{c|}{31.10 / 0.9456} & \multicolumn{1}{c}{25.23 / 0.8676} \\
VFIT-S~\cite{shi2022vfit} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{36.42 / 0.9760} & \multicolumn{1}{c|}{28.46 / 0.8926} & \multicolumn{1}{c|}{41.15 / 0.9920} & \multicolumn{1}{c|}{37.07 / \textcolor{blue}{\underline{0.9845}}} & \multicolumn{1}{c|}{31.39 / \textcolor{blue}{\underline{0.9501}}} & \multicolumn{1}{c}{25.52 / 0.8717} \\
VFIT-B~\cite{shi2022vfit} & \multicolumn{1}{c}{\XSolidBrush} & \multicolumn{1}{c|}{\XSolidBrush} & \multicolumn{1}{c|}{\textcolor{blue}{\underline{36.89}} / \textcolor{red}{\textbf{0.9775}}} & \multicolumn{1}{c|}{28.60 / 0.8945} & \multicolumn{1}{c|}{41.24 / 0.9921} & \multicolumn{1}{c|}{37.06 / 0.9839} & \multicolumn{1}{c|}{31.39 / \textcolor{blue}{\underline{0.9501}}} & \multicolumn{1}{c}{25.61 / \textcolor{blue}{\underline{0.8731}}} \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-0.2cm}
\caption{\textbf{Quantitative (PSNR/SSIM) comparison}. We compare our boosted models to representative state-of-the-art methods on Vimeo90K~\cite{xue2019toflow}, DAVIS~\cite{perazzi2016benchmark} and SNU-FILM~\cite{choi2020channel} benchmarks. Both of the optimisation approaches exhibit a substantial improvement in performance. \textbf{Note that} FLAVR~\cite{kalluri2023flavr} and VFIT~\cite{shi2022vfit} take \textbf{multiple frames} as input, but our boosted models can still outperform them. \textcolor{red}{\textbf{RED}}: best performance, \textcolor{blue}{\underline{BLUE}}: second best performance.
%\weidi{put space before and after `/'.}
}
% \weidi{RIFE, RIFE-ours-A: End-to-end adaptation, RIFE-ours-P: plugin adapter.}
% `model-Adap' refers to the model boosted by proposed Cycle-Consistency adaptation, while `model-plugin' indicates model boosted by our proposed plug-in Adapter module.
\label{tab:results}
\vspace{-0.4cm}
\end{table}
% \begin{table}[!htb]
\begin{table}[t]
\footnotesize
\tabcolsep=0.1cm
\centering
\begin{tabular}{ccccccc}
\toprule
Strategies & \multicolumn{1}{c|}{\#Adaptations} & \multicolumn{1}{c|}{SepConv~\cite{niklaus2017sepconv}} & \multicolumn{1}{c|}{EDSC~\cite{cheng2021multiple}} & \multicolumn{1}{c|}{RIFE~\cite{huang2022real}} & \multicolumn{1}{c|}{IFRNet~\cite{kong2022ifrnet}} & UPRNet~\cite{jin2022unified} \\ \midrule
Original & \multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{33.72 / 0.9639} & \multicolumn{1}{c|}{34.55 / 0.9677} & \multicolumn{1}{c|}{35.28 / 0.9704} & \multicolumn{1}{c|}{35.86 / 0.9729} & 36.07 / 0.9735 \\
\midrule
\multirow{4}{*}{Na\"ive} & \multicolumn{1}{c|}{5} & \multicolumn{1}{c|}{33.77 / 0.9641} & \multicolumn{1}{c|}{34.62 / 0.9679} & \multicolumn{1}{c|}{35.36 / 0.9708} & \multicolumn{1}{c|}{35.95 / 0.9734} & 36.23 / 0.9744 \\
& \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{33.83 / 0.9644} & \multicolumn{1}{c|}{34.69 / 0.9683} & \multicolumn{1}{c|}{35.45 / 0.9713} & \multicolumn{1}{c|}{35.81 / 0.9731} & 36.16 / 0.9747 \\
& \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{33.91 / 0.9647} & \multicolumn{1}{c|}{34.80 / 0.9687} & \multicolumn{1}{c|}{35.45 / 0.9715} & \multicolumn{1}{c|}{35.03 / 0.9685} & 35.79 / 0.9737 \\
& \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{33.95 / 0.9648} & \multicolumn{1}{c|}{34.85 / 0.9688} & \multicolumn{1}{c|}{35.33 / 0.9710} & \multicolumn{1}{c|}{34.09 / 0.9615} & 35.51 / 0.9721 \\
\midrule
\multirow{4}{*}{Cycle} & \multicolumn{1}{c|}{5} & \multicolumn{1}{c|}{33.83 / 0.9644} & \multicolumn{1}{c|}{34.63 / 0.9680} & \multicolumn{1}{c|}{35.41 / 0.9710} & \multicolumn{1}{c|}{36.14 / 0.9741} & 36.49 / 0.9750 \\
& \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{33.96 / 0.9650} & \multicolumn{1}{c|}{34.73 / 0.9685} & \multicolumn{1}{c|}{35.57 / 0.9717} & \multicolumn{1}{c|}{36.38 / 0.9753} & 36.68 / 0.9758 \\
& \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{34.17 / 0.9659} & \multicolumn{1}{c|}{34.94 / 0.9693} & \multicolumn{1}{c|}{35.80 / 0.9728} & \multicolumn{1}{c|}{36.60 / 0.9759} & 36.84 / 0.9766 \\
& \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{34.29} / \textbf{0.9662}} & \multicolumn{1}{c|}{\textbf{35.06} / \textbf{0.9699}} & \multicolumn{1}{c|}{\textbf{35.93} / \textbf{0.9733}} & \multicolumn{1}{c|}{\textbf{36.68} / \textbf{0.9760}} & \textbf{36.90} / \textbf{0.9768} \\
\bottomrule
\end{tabular}
\vspace{5pt}
\caption{\textbf{Quantitative (PSNR/SSIM) comparison of adaptation strategies.} The experiments on Vimeo90K~\cite{xue2019toflow} dataset have shown that cycle-consistency adaptation steadily boosts VFI models by fully leveraging the inter-frame consistency to learn motion characteristics within the test sequence.
}
\label{tab:table2}
\vspace{-0.4cm}
\end{table}
% Cycle-consistency adaptation achieves stable performance improvement by fully leveraging the consistency within the video sequence.
\begin{table}[!htb]
\footnotesize
% \scriptsize
\begin{center}
\tabcolsep=0.15cm
\begin{tabular}{lccccccc}
% \toprule
% \multicolumn{1}{c}{\multirow{2}{*}{Methods}} & \#Finetuning & \multicolumn{2}{c}{Vimeo90K~\cite{xue2019toflow}} & \multicolumn{2}{c}{DAVIS~\cite{perazzi2016benchmark}} \\ \cline{3-6}
% & Parameters & AdapTime (ms) & PSNR/SSIM & AdapTime (ms) & PSNR/SSIM \\
% \midrule
% \multicolumn{1}{l|}{RIFE-ours-e2e} & \multicolumn{1}{c|}{10.21M} & \multicolumn{1}{c}{145.6} & \multicolumn{1}{c|}{35.57/0.9717} & \multicolumn{1}{c}{162.7} & \multicolumn{1}{c}{27.81/0.8798} \\
% \multicolumn{1}{l|}{RIFE-ours-plugin} & \multicolumn{1}{c|}{0.087M} & \multicolumn{1}{c}{83.13} & \multicolumn{1}{c|}{35.56/0.9714} & \multicolumn{1}{c}{86.84} & \multicolumn{1}{c}{27.76/0.8771} \\
% \midrule
% \multicolumn{1}{l|}{IFRNet-ours-e2e} & \multicolumn{1}{c|}{18.79M} & \multicolumn{1}{c}{107.7} & \multicolumn{1}{c|}{36.38/0.9753} & \multicolumn{1}{c}{196.2} & \multicolumn{1}{c}{28.45/0.8936} \\
% \multicolumn{1}{l|}{IFRNet-ours-plugin} & \multicolumn{1}{c|}{0.676M} & \multicolumn{1}{c}{39.08} & \multicolumn{1}{c|}{36.01/0.9734} & \multicolumn{1}{c}{73.79} & \multicolumn{1}{c}{28.16/0.8825} \\
% \midrule
% \multicolumn{1}{l|}{UPRNet-ours-e2e} & \multicolumn{1}{c|}{6.260M} & \multicolumn{1}{c}{285.5} & \multicolumn{1}{c|}{36.68/0.9758} & \multicolumn{1}{c}{507.0} & \multicolumn{1}{c}{28.84/0.8997} \\
% \multicolumn{1}{l|}{UPRNet-ours-plugin} & \multicolumn{1}{c|}{0.009M} & \multicolumn{1}{c}{162.0} & \multicolumn{1}{c|}{36.44/0.9751} & \multicolumn{1}{c}{237.6} & \multicolumn{1}{c}{28.69/0.8945} \\
% \bottomrule
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{Methods}} & \#Finetuning & \multicolumn{3}{c}{Adaptation Time (ms)} & \multicolumn{3}{c}{Inference Time (ms)} \\ \cline{3-8}
& Parameters & Vimeo90K & DAVIS & SNU-FILM & Vimeo90K & DAVIS & SNU-FILM \\
\midrule
\multicolumn{1}{l|}{RIFE-ours-e2e} & \multicolumn{1}{c|}{10.21M} & \multicolumn{1}{c}{145.6} & \multicolumn{1}{c}{162.7} & \multicolumn{1}{c|}{260.8} & \multicolumn{1}{c}{10.94} & \multicolumn{1}{c}{12.74} & \multicolumn{1}{c}{23.61} \\
\multicolumn{1}{l|}{RIFE-ours-plugin} & \multicolumn{1}{c|}{0.087M} & \multicolumn{1}{c}{83.13} & \multicolumn{1}{c}{86.84} & \multicolumn{1}{c|}{125.4} & \multicolumn{1}{c}{11.79} & \multicolumn{1}{c}{14.67} & \multicolumn{1}{c}{24.79} \\
\midrule
\multicolumn{1}{l|}{IFRNet-ours-e2e} & \multicolumn{1}{c|}{18.79M} & \multicolumn{1}{c}{107.7} & \multicolumn{1}{c}{196.2} & \multicolumn{1}{c|}{403.3} & \multicolumn{1}{c}{18.61} & \multicolumn{1}{c}{25.94} & \multicolumn{1}{c}{55.54} \\
\multicolumn{1}{l|}{IFRNet-ours-plugin} & \multicolumn{1}{c|}{0.676M} & \multicolumn{1}{c}{39.08} & \multicolumn{1}{c}{73.79} & \multicolumn{1}{c|}{158.1} & \multicolumn{1}{c}{19.11} & \multicolumn{1}{c}{29.32} & \multicolumn{1}{c}{61.58} \\
\midrule
\multicolumn{1}{l|}{UPRNet-ours-e2e} & \multicolumn{1}{c|}{6.260M} & \multicolumn{1}{c}{285.5} & \multicolumn{1}{c}{507.0} & \multicolumn{1}{c|}{1487.8} & \multicolumn{1}{c}{28.33} & \multicolumn{1}{c}{49.90} & \multicolumn{1}{c}{90.85} \\
\multicolumn{1}{l|}{UPRNet-ours-plugin} & \multicolumn{1}{c|}{0.009M} & \multicolumn{1}{c}{162.0} & \multicolumn{1}{c}{237.6} & \multicolumn{1}{c|}{872.7} & \multicolumn{1}{c}{29.20} & \multicolumn{1}{c}{50.72} & \multicolumn{1}{c}{92.60} \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-0.2cm}
\caption{\textbf{Ablation Study on end-to-end and plug-in adapter adaptation.} Models boosted by our proposed plug-in adapter require minimal finetuning parameters for adaptation, resulting in a 2 times improvement in efficiency while maintaining comparable inference efficiency and performance.}
\label{table3}
% \vspace{-5pt}
\vspace{-0.4cm}
\end{table}
In this paper, we present optimisation-based video frame interpolation to tackle the generalisation challenge of VFI models and boost their performance at inference time.
To this end, a test-time motion adaptation strategy that is suitable for video frame interpolation has been introduced, namely {\em cycle-consistency adaptation}.
In order to address the efficiency drawback of motion adaptation,
we further propose a lightweight yet effective plug-in {\em adapter} module which can be injected into the motion estimation module of existing pre-trained VFI models to refine the estimated motion flow, thus synthesizing higher-quality intermediate frames.
Extensive experiments on various models and benchmarks have demonstrated the effectiveness of the proposed cycle-consistency adaptation strategy on VFI task and confirmed that the proposed plug-in adapter module can efficiently and steadily boost the performance of VFI models, even outperforming approaches with extra inputs.
% In this paper, we present two test-time adaptation strategies suitable for video frame interpolation task, termed Direct Adaptation and Cycle-Consistency Adaptation respectively. Then we demonstrated the effectiveness of the proposed adaptation strategies on VFI task via experiments on multiple existing methods and compared their pros and cons. Taking the extra inference time cost during test-time adaptation into consideration, we proposed a lightweight plugin Adapter module which can be injected into the flow estimation module of the pre-trained VFI models to predict a couple of hyper-parameters for the correction of estimated motion flow, thus synthesizing higher-quality intermediate frame. Extensive experiments on several benchmarks confirm that the proposed plugin Adapter module can effectively and efficiently boost the performance of VFI models and even outperform existing models with extra frame inputs.
\section*{Acknowledgments}
This work is supported by National Natural Science Foundation of China (62271308), STCSM (22511105700, 22DZ2229005), 111 plan (BP0719010), and State Key Laboratory of UHD Video and Audio Production and Presentation.
In this supplementary document, we start by giving more details on the implementation details of our proposed cycle-consistency motion adaptation strategy and plug-in adapter module in Section~\ref{implementation}.
Then, we demonstrate the ablation study on the influence of more adaptation steps and how cycle-consistency adaptation steadily works on several video frame interpolation (VFI) models in Section~\ref{ablation}.
% Finally, we supplement more qualitative comparison results on several benchmarks in Section~\ref{qualitative}.
Next, we supplement more qualitative comparison results on several benchmarks in Section~\ref{qualitative}.
Finally, we illustrate the limitation of our method and our future work in Section~\ref{limitation}.
\section{Implementation Details}
\label{implementation}
\noindent{\textbf{Cycle-consistency adaptation.}} Considering that each sample in common datasets~\cite{xue2019toflow, perazzi2016benchmark, choi2020channel} typically comprises a septuplet, with odd frames as input, denoted as $\{\mathcal{I}_1, \mathcal{I}_3, \mathcal{I}_5, \mathcal{I}_7\}$.
We divide it into two triplets, {\em i.e.,} $\{\mathcal{I}_1, \mathcal{I}_3, \mathcal{I}_5\}$ and $\{\mathcal{I}_3, \mathcal{I}_5, \mathcal{I}_7\}$ and perform motion adaptation on these two triplets to adapt the model to the motion characteristics of the current video sequence.
To be specific, we take the middle frame of each triplet as the ground truth and calculate loss between the generated frames and the target frames, denoted as $\mathcal{L}_3 = ||\hat{\mathcal{I}}_3 - \mathcal{I}_3||$ and $\mathcal{L}_5 = ||\hat{\mathcal{I}}_5 - \mathcal{I}_5||$.
Then we take their average $\mathcal{L} = \frac{1}{2}(\mathcal{L}_3 + \mathcal{L}_5)$ to backward gradients for parameters updates, and regard this entire process as one adaptation step. This cycle-consistency adaptation strategy has been employed in end-to-end finetuning as well as plug-in adapter finetuning, and theoretically, it can be extended to longer video sequences.
\vspace{0.2cm}
\noindent{\textbf{Plug-in adapter.}} For flow-based VFI methods~\cite{huang2022real, kong2022ifrnet, jin2022unified}, we can freeze the pre-trained parameters and simply use our proposed lightweight plug-in adapter to adjust the estimated motion flow. Concretely, we reuse the convolutional layers from the pre-trained model's motion estimation module to align the channel dimensions of the extracted visual features with that of motion flow. Then a simple $1 \times 1$ convolution layer is utilized to predict pixel-wise weights $\alpha$ and biases $\beta$ for motion adaptation, without introducing any additional activation layers.
\input{table/supp}
\section{Additional Ablation Study}
\label{ablation}
In this section, we present more ablation results for our proposed optimisation-based VFI, including the motion adaptation strategy and the adaptation steps.
Specifically, we conduct test-time motion adaptation experiments on three benchmarks, namely Vimeo90K~\cite{xue2019toflow}, DAVIS~\cite{perazzi2016benchmark}, and SNU-FILM~\cite{choi2020channel} with five VFI models, SepConv~\cite{niklaus2017sepconv}, EDSC~\cite{cheng2021multiple}, RIFE~\cite{huang2022real}, IFRNet~\cite{kong2022ifrnet} and UPRNet~\cite{jin2022unified}, in two manners, end-to-end (e2e) and plug-in adapter (plugin) boosted. We further investigate the impact of different adaptation steps on performance.
\vspace{0.2cm}
\noindent{\textbf{Adaptation Strategy.}} The results in Table~\ref{tab:supp} further demonstrate the conclusions we have drawn in our paper: The performance of pre-trained VFI models can be enhanced via end-to-end motion adaptation during test-time.
Furthermore, for flow-based methods such as RIFE~\cite{huang2022real}, IFRNet~\cite{kong2022ifrnet} and UPRNet~\cite{jin2022unified}, the proposed plug-in adapter module can effectively boost the performance of pre-trained models.
Moreover, it achieves comparable performance improvements to that of end-to-end finetuning, with the same number of adaptation steps.
\vspace{0.2cm}
\noindent{\textbf{Adaptation Steps.}} As indicated in Table~\ref{tab:supp}, with adaptation steps increasing, both end-to-end finetuning and plug-in adapter finetuning effectively boost the performance of VFI models, thus confirming the effectiveness of our proposed optimisation-based VFI.
Furthermore, while cycle-consistency adaptation can steadily boost performance, models exhibit significant improvements within the first $10$ adaptation steps, with performance gains gradually approaching saturation after more steps. Therefore, taking both efficiency and performance into consideration, we have chosen $10$ steps of adaptation as our default setting.
% \input{table/supp}
\section{More Qualitative Results}
\label{qualitative}
We provide more qualitative results in Figure~\ref{fig:supp_figure1}, Figure~\ref{fig:supp_figure2} and Figure~\ref{fig:supp_figure3}.
Concretely, we compare UPRNet-ours-plugin (10-step-adaptation) and UPRNet-ours-e2e++ (30-step-adaptation) with other state-of-the-art VFI methods on benchmark datasets with different motion characteristics.
We can observe that other VFI methods tend to generate images with motion blur or missing object details, while our method can consistently achieve better perceptual quality, {\em i.e.,} maintaining the object shape and textual information, and synthesizing shaper details.
In summary, our proposed motion adaptation can steadily boost VFI models.
\section{Limitation \& Future Work}
\label{limitation}
In this paper, we have demonstrated the effectiveness of our proposed cycle-consistency motion adaptation and lightweight adapter as a plug-in module for VFI.
However, some limitations remain: it still suffers from the well-known problem of using test-time adaptation, that is, time consumption is still non-negligible, which poses limitations in increasing adaptation steps to pursue higher performance.
Besides, due to the limitation of data and computing resources, our motion adaptation strategy is currently restricted to 2-frame VFI models and has not been extended to frame synthesis at arbitrary temporal positions.
Moreover, in general, our proposed method is beneficial for better handling motions that are regular, such as rotations and large-scale motions, however, it remains challenging for irregular motions, such as sudden camera shaking, or illumination changes, due to the difficulty of inferring priors.
Our future work will investigate more efficient plug-in adapter architecture and further extend it to flexible VFI models with multiple frames as inputs, which has the potential to further boost the performance and generalisation ability of video frame interpolation models.
\begin{figure}[!htbp]
\centering
\includegraphics[width=\textwidth]{images/supp_figure1.pdf} \\
\vspace{-0.1cm}
\caption{\textbf{Qualitative comparison against the state-of-the-art VFI algorithms.} We show visualization on Vimeo90K~\cite{xue2019toflow} benchmark for comparison. The patches for careful comparison are marked with \textcolor{red}{red} in the original images. Our boosted models can generate higher-quality results with clearer structures and fewer distortions.}
\label{fig:supp_figure1}
\vspace{-0.4cm}
\end{figure}
\begin{figure}[!htbp]
\centering
\includegraphics[width=\textwidth]{images/supp_figure2.pdf} \\
\vspace{-0.1cm}
\caption{\textbf{Qualitative comparison against the state-of-the-art VFI algorithms.} We show visualization on DAVIS~\cite{perazzi2016benchmark} benchmark for comparison. The patches for careful comparison are marked with \textcolor{red}{red} in the original images. Our boosted models can generate higher-quality results with clearer structures and fewer distortions.}
\label{fig:supp_figure2}
\vspace{-0.4cm}
\end{figure}
\begin{figure}[!htbp]
\centering
\includegraphics[width=\textwidth]{images/supp_figure3.pdf} \\
\vspace{-0.1cm}
\caption{\textbf{Qualitative comparison against the state-of-the-art VFI algorithms.} We show visualization on SNU-FILM~\cite{choi2020channel} benchmark for comparison. The patches for careful comparison are marked with \textcolor{red}{red} in the original images. Our boosted models can generate higher-quality results with clearer structures and fewer distortions.}
\label{fig:supp_figure3}
\vspace{-0.4cm}
\end{figure}
% \begin{table}[!htb]
\begin{table}[t]
\scriptsize
\begin{center}
\tabcolsep=0.06cm
\begin{tabular}{lccccccc}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{Methods}} & \multicolumn{1}{c}{\#Adapt} & \multirow{2}{*}{Vimeo90K~\cite{xue2019toflow}} & \multirow{2}{*}{DAVIS~\cite{perazzi2016benchmark}} & \multicolumn{4}{c}{SNU-FILM~\cite{choi2020channel}} \\
\cmidrule{5-8}
% \cline{5-8}
\multicolumn{1}{c}{} & Steps & & & Easy & Medium & Hard & Extreme \\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{SepConv-ours-e2e}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{33.72 / 0.9639} & \multicolumn{1}{c|}{26.65 / 0.8611} & \multicolumn{1}{c|}{40.21 / 0.9909} & \multicolumn{1}{c|}{35.45 / 0.9785} & \multicolumn{1}{c|}{29.62 / 0.9302} & 24.16 / 0.8457 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{33.96 / 0.9650} & \multicolumn{1}{c|}{26.83 / 0.8639} & \multicolumn{1}{c|}{40.41 / 0.9911} & \multicolumn{1}{c|}{35.71 / 0.9794} & \multicolumn{1}{c|}{29.80 / 0.9313} & 24.26 / 0.8479 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{34.17 / 0.9659} & \multicolumn{1}{c|}{26.93 / 0.8649} & \multicolumn{1}{c|}{40.56 / 0.9913} & \multicolumn{1}{c|}{35.91 / 0.9800} & \multicolumn{1}{c|}{29.88 / 0.9313} & 24.32 / 0.8484 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{34.29} / \textbf{0.9662}} & \multicolumn{1}{c|}{\textbf{26.98} / \textbf{0.8651}} & \multicolumn{1}{c|}{\textbf{40.66} / \textbf{0.9914}} & \multicolumn{1}{c|}{\textbf{36.04} / \textbf{0.9804}} & \multicolumn{1}{c|}{\textbf{29.93} / \textbf{0.9318}} & \textbf{24.36} / \textbf{0.8491} \\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{34.29 / 0.9662} & \multicolumn{1}{c|}{26.98 / 0.8651} & \multicolumn{1}{c|}{40.66 / 0.9914} & \multicolumn{1}{c|}{36.04 / 0.9804} & \multicolumn{1}{c|}{29.93 / 0.9318} & 24.36 / 0.8491 \\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{EDSC-ours-e2e}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{34.55 / 0.9677} & \multicolumn{1}{c|}{26.83 / 0.8578} & \multicolumn{1}{c|}{40.66 / 0.9915} & \multicolumn{1}{c|}{35.77 / 0.9795} & \multicolumn{1}{c|}{29.75 / 0.9301} & 24.12 / 0.8420 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{34.73 / 0.9685} & \multicolumn{1}{c|}{26.96 / 0.8600} & \multicolumn{1}{c|}{40.88 / 0.9917} & \multicolumn{1}{c|}{35.98 / 0.9803} & \multicolumn{1}{c|}{29.85 / 0.9313} & 24.19 / 0.8436 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{34.94 / 0.9693} & \multicolumn{1}{c|}{27.07 / 0.8618} & \multicolumn{1}{c|}{40.98 / 0.9919} & \multicolumn{1}{c|}{36.18 / 0.9811} & \multicolumn{1}{c|}{29.95 / 0.9322} & 24.23 / 0.8445 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{35.06} / \textbf{0.9699}} & \multicolumn{1}{c|}{\textbf{27.14} / \textbf{0.8630}} & \multicolumn{1}{c|}{\textbf{41.09} / \textbf{0.9920}} & \multicolumn{1}{c|}{\textbf{36.33} / \textbf{0.9817}} & \multicolumn{1}{c|}{\textbf{30.02} / \textbf{0.9334}} & \textbf{24.28} / \textbf{0.8455} \\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{35.06 / 0.9699} & \multicolumn{1}{c|}{27.14 / 0.8630} & \multicolumn{1}{c|}{41.09 / 0.9920} & \multicolumn{1}{c|}{36.33 / 0.9817} & \multicolumn{1}{c|}{30.02 / 0.9334} & 24.28 / 0.8455 \\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{RIFE-ours-e2e}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{35.28 / 0.9704} & \multicolumn{1}{c|}{27.61 / 0.8760} & \multicolumn{1}{c|}{40.75 / 0.9916} & \multicolumn{1}{c|}{36.18 / 0.9808} & \multicolumn{1}{c|}{30.30 / 0.9368} & 24.62 / 0.8531 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{35.57 / 0.9717} & \multicolumn{1}{c|}{27.81 / 0.8798} & \multicolumn{1}{c|}{40.95 / 0.9918} & \multicolumn{1}{c|}{36.42 / 0.9815} & \multicolumn{1}{c|}{30.49 / 0.9386} & 24.71 / 0.8549 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{35.80 / 0.9728} & \multicolumn{1}{c|}{27.99 / 0.8830} & \multicolumn{1}{c|}{41.10 / 0.9920} & \multicolumn{1}{c|}{36.71 / 0.9827} & \multicolumn{1}{c|}{30.67 / 0.9408} & 24.79 / 0.8570 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{35.93} / \textbf{0.9733}} & \multicolumn{1}{c|}{\textbf{28.10} / \textbf{0.8850}} & \multicolumn{1}{c|}{\textbf{41.20} / \textbf{0.9924}} & \multicolumn{1}{c|}{\textbf{36.94} / \textbf{0.9835}} & \multicolumn{1}{c|}{\textbf{30.83} / \textbf{0.9430}} & \textbf{24.87} / \textbf{0.8589} \\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{35.93 / 0.9733} & \multicolumn{1}{c|}{28.10 / 0.8850} & \multicolumn{1}{c|}{41.20 / 0.9924} & \multicolumn{1}{c|}{36.94 / 0.9835} & \multicolumn{1}{c|}{30.83 / 0.9430} & 24.87 / 0.8589 \\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{RIFE-ours-plugin}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{35.33 / 0.9706} & \multicolumn{1}{c|}{27.64 / 0.8765} & \multicolumn{1}{c|}{40.66 / 0.9915} & \multicolumn{1}{c|}{36.12 / 0.9807} & \multicolumn{1}{c|}{30.32 / 0.9371} & 24.67 / 0.8539 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{35.56 / 0.9714} & \multicolumn{1}{c|}{27.76 / 0.8871} & \multicolumn{1}{c|}{40.99 / 0.9918} & \multicolumn{1}{c|}{36.55 / 0.9825} & \multicolumn{1}{c|}{30.48 / 0.9387} & 24.64 / 0.8533 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{35.61 / 0.9719} & \multicolumn{1}{c|}{27.79 / 0.8786} & \multicolumn{1}{c|}{41.01 / 0.9919} & \multicolumn{1}{c|}{36.55 / 0.9824} & \multicolumn{1}{c|}{30.65 / 0.9404} & 24.79 / 0.8555 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{35.71} / \textbf{0.9722}} & \multicolumn{1}{c|}{\textbf{27.88} / \textbf{0.8799}} & \multicolumn{1}{c|}{\textbf{41.12} / \textbf{0.9920}} & \multicolumn{1}{c|}{\textbf{36.77} / \textbf{0.9832}} & \multicolumn{1}{c|}{\textbf{30.74} / \textbf{0.9404}} & \textbf{24.84} / \textbf{0.8590}\\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{35.71 / 0.9722} & \multicolumn{1}{c|}{27.88 / 0.8799} & \multicolumn{1}{c|}{41.12 / 0.9920} & \multicolumn{1}{c|}{36.77 / 0.9832} & \multicolumn{1}{c|}{30.74 / 0.9404} & 24.84 / 0.8590\\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{IFRNet-ours-e2e}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{35.86 / 0.9729} & \multicolumn{1}{c|}{28.03 / 0.8851} & \multicolumn{1}{c|}{40.91 / 0.9916} & \multicolumn{1}{c|}{36.18 / 0.9808} & \multicolumn{1}{c|}{30.30 / 0.9368} & 24.62 / 0.8531 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{36.38 / 0.9753} & \multicolumn{1}{c|}{28.45 / 0.8936} & \multicolumn{1}{c|}{41.21 / 0.9921} & \multicolumn{1}{c|}{37.03 / 0.9832} & \multicolumn{1}{c|}{31.10 / 0.9440} & 25.03 / 0.8634 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{36.60 / 0.9759} & \multicolumn{1}{c|}{28.69 / 0.8979} & \multicolumn{1}{c|}{41.40 / 0.9923} & \multicolumn{1}{c|}{37.36 / 0.9844} & \multicolumn{1}{c|}{31.37 / 0.9476} & 25.18 / 0.8676 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{36.68} / \textbf{0.9760}} & \multicolumn{1}{c|}{\textbf{28.78} / \textbf{0.8995}} & \multicolumn{1}{c|}{\textbf{41.48} / \textbf{0.9923}} & \multicolumn{1}{c|}{\textbf{37.57} / \textbf{0.9850}} & \multicolumn{1}{c|}{\textbf{31.45} / \textbf{0.9482}} & \textbf{25.22} / \textbf{0.8694} \\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{36.68 / 0.9760} & \multicolumn{1}{c|}{28.78 / 0.8995} & \multicolumn{1}{c|}{41.48 / 0.9923} & \multicolumn{1}{c|}{37.57 / 0.9850} & \multicolumn{1}{c|}{31.45 / 0.9482} & 25.22 / 0.8694 \\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{IFRNet-ours-plugin}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{35.86 / 0.9729} & \multicolumn{1}{c|}{28.02 / 0.8850} & \multicolumn{1}{c|}{40.91 / 0.9918} & \multicolumn{1}{c|}{36.58 / 0.9816} & \multicolumn{1}{c|}{30.74 / 0.9404} & 24.84 / 0.8590 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{36.01 / 0.9734} & \multicolumn{1}{c|}{28.16 / 0.8825} & \multicolumn{1}{c|}{41.06 / 0.9920} & \multicolumn{1}{c|}{36.92 / 0.9834} & \multicolumn{1}{c|}{30.88 / 0.9404} & 24.93 / 0.8599 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{36.11 / 0.9738} & \multicolumn{1}{c|}{28.26 / 0.8875} & \multicolumn{1}{c|}{41.11 / 0.9921} & \multicolumn{1}{c|}{37.01 / 0.9837} & \multicolumn{1}{c|}{30.95 / 0.9414} & 24.96 / 0.8599 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{36.14} / \textbf{0.9742}} & \multicolumn{1}{c|}{\textbf{28.33} / \textbf{0.8888}} & \multicolumn{1}{c|}{\textbf{41.18} / \textbf{0.9923}} & \multicolumn{1}{c|}{\textbf{37.14} / \textbf{0.9844}} & \multicolumn{1}{c|}{\textbf{31.03} / \textbf{0.9419}} & \textbf{24.97} / \textbf{0.8600} \\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{36.14 / 0.9742} & \multicolumn{1}{c|}{28.33 / 0.8888} & \multicolumn{1}{c|}{41.18 / 0.9923} & \multicolumn{1}{c|}{37.14 / 0.9844} & \multicolumn{1}{c|}{31.03 / 0.9419} & 24.97 / 0.8600 \\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{UPRNet-ours-e2e}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{36.07 / 0.9735} & \multicolumn{1}{c|}{28.38 / 0.8914} & \multicolumn{1}{c|}{41.01 / 0.9919} & \multicolumn{1}{c|}{36.80 / 0.9819} & \multicolumn{1}{c|}{31.22 / 0.9422} & 25.39 / 0.8648 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{36.68 / 0.9758} & \multicolumn{1}{c|}{28.84 / 0.8997} & \multicolumn{1}{c|}{41.31 / 0.9923} & \multicolumn{1}{c|}{37.24 / 0.9836} & \multicolumn{1}{c|}{31.66 / 0.9464} & 25.64 / 0.8699 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{36.84 / 0.9766} & \multicolumn{1}{c|}{29.07 / 0.9043} & \multicolumn{1}{c|}{41.42 / 0.9924} & \multicolumn{1}{c|}{37.52 / 0.9849} & \multicolumn{1}{c|}{31.89 / 0.9500} & 25.85 / 0.8755 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{36.90} / \textbf{0.9768}} & \multicolumn{1}{c|}{\textbf{29.15} / \textbf{0.9062}} & \multicolumn{1}{c|}{\textbf{41.48} / \textbf{0.9925}} & \multicolumn{1}{c|}{\textbf{37.66} / \textbf{0.9855}} & \multicolumn{1}{c|}{\textbf{32.00} / \textbf{0.9519}} & \textbf{25.99} / \textbf{0.8798} \\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{36.90 / 0.9768} & \multicolumn{1}{c|}{29.15 / 0.9062} & \multicolumn{1}{c|}{41.48 / 0.9925} & \multicolumn{1}{c|}{37.66 / 0.9855} & \multicolumn{1}{c|}{32.00 / 0.9519} & 25.99 / 0.8798 \\
\midrule
\multicolumn{1}{l|}{\multirow{4}{*}{UPRNet-ours-plugin}} &
\multicolumn{1}{c|}{0} & \multicolumn{1}{c|}{36.04 / 0.9734} & \multicolumn{1}{c|}{28.31 / 0.8896} & \multicolumn{1}{c|}{41.00 / 0.9919} & \multicolumn{1}{c|}{36.77 / 0.9818} & \multicolumn{1}{c|}{31.18 / 0.9418} & 25.38 / 0.8645 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{10} & \multicolumn{1}{c|}{36.44 / 0.9751} & \multicolumn{1}{c|}{28.69 / 0.8945} & \multicolumn{1}{c|}{41.32 / 0.9923} & \multicolumn{1}{c|}{37.38 / 0.9843} & \multicolumn{1}{c|}{31.64 / 0.9448} & \textbf{25.69} / 0.8705 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{20} & \multicolumn{1}{c|}{36.52 / 0.9754} & \multicolumn{1}{c|}{28.78 / 0.8963} & \multicolumn{1}{c|}{41.37 / 0.9923} & \multicolumn{1}{c|}{\textbf{37.59} / 0.9847} & \multicolumn{1}{c|}{31.70 / 0.9461} & 25.68 / 0.8705 \\
\multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{\textbf{36.57} / \textbf{0.9756}} & \multicolumn{1}{c|}{\textbf{28.90} / \textbf{0.8989}} & \multicolumn{1}{c|}{\textbf{41.40} / \textbf{0.9924}} & \multicolumn{1}{c|}{37.55 / \textbf{0.9849}} & \multicolumn{1}{c|}{\textbf{31.75} / \textbf{0.9470}} & \textbf{25.69} / \textbf{0.8706} \\
% \multicolumn{1}{l|}{} & \multicolumn{1}{c|}{30} & \multicolumn{1}{c|}{36.57 / 0.9756} & \multicolumn{1}{c|}{28.90 / 0.8989} & \multicolumn{1}{c|}{41.40 / 0.9924} & \multicolumn{1}{c|}{37.55 / 0.9849} & \multicolumn{1}{c|}{31.75 / 0.9470} & 25.69 / 0.8706 \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-0.2cm}
\caption{\textbf{Additional ablation study on adaptation strategies and steps}.
We inherit pre-trained VFI models~\cite{niklaus2017sepconv, cheng2021multiple, huang2022real, kong2022ifrnet, jin2022unified}, then perform end-to-end (e2e) and plug-in adapter (plugin) adaptation with different adaptation steps, comparing their performances on three commonly used benchmarks.
}
\label{tab:supp}
\vspace{-0.4cm}
\end{table}
\begin{thebibliography}{54}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bao et~al.(2019{\natexlab{a}})Bao, Lai, Ma, Zhang, Gao, and
Yang]{bao2019depth}
Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan
Yang.
\newblock Depth-aware video frame interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 3703--3712, 2019{\natexlab{a}}.
\bibitem[Bao et~al.(2019{\natexlab{b}})Bao, Lai, Zhang, Gao, and
Yang]{bao2019memc}
Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.
\newblock Memc-net: Motion estimation and motion compensation driven neural
network for video interpolation and enhancement.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine
Intelligence}, 43\penalty0 (3):\penalty0 933--948, 2019{\natexlab{b}}.
\bibitem[Cheng and Chen(2021)]{cheng2021multiple}
Xianhang Cheng and Zhenzhong Chen.
\newblock Multiple video frame interpolation via enhanced deformable separable
convolution.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine
Intelligence}, 44\penalty0 (10):\penalty0 7029--7045, 2021.
\bibitem[Chi et~al.(2021)Chi, Wang, Yu, and Tang]{chi2021testdeblur}
Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang.
\newblock Test-time fast adaptation for dynamic scene deblurring via
meta-auxiliary learning.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 9137--9146, 2021.
\bibitem[Choi et~al.(2020{\natexlab{a}})Choi, Choi, Baik, Kim, and
Lee]{choi2020scene}
Myungsub Choi, Janghoon Choi, Sungyong Baik, Tae~Hyun Kim, and Kyoung~Mu Lee.
\newblock Scene-adaptive video frame interpolation via meta-learning.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 9444--9453, 2020{\natexlab{a}}.
\bibitem[Choi et~al.(2020{\natexlab{b}})Choi, Kim, Han, Xu, and
Lee]{choi2020channel}
Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung~Mu Lee.
\newblock Channel attention is all you need for video frame interpolation.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~34, pages 10663--10671, 2020{\natexlab{b}}.
\bibitem[Dwibedi et~al.(2019)Dwibedi, Aytar, Tompson, Sermanet, and
Zisserman]{Dwibeditemporal}
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew
Zisserman.
\newblock Temporal cycle-consistency learning.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 1801--1810, 2019.
\bibitem[Flynn et~al.(2016)Flynn, Neulander, Philbin, and
Snavely]{flynn2016deepstereo}
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely.
\newblock Deepstereo: Learning to predict new views from the world's imagery.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 5515--5524, 2016.
\bibitem[Hu et~al.(2022)Hu, Niklaus, Sclaroff, and Saenko]{hu2022many}
Ping Hu, Simon Niklaus, Stan Sclaroff, and Kate Saenko.
\newblock Many-to-many splatting for efficient video frame interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 3553--3562, 2022.
\bibitem[Huang et~al.(2015)Huang, Singh, and Ahuja]{huang2015single}
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.
\newblock Single image super-resolution from transformed self-exemplars.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 5197--5206, 2015.
\bibitem[Huang et~al.(2022)Huang, Zhang, Heng, Shi, and Zhou]{huang2022real}
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.
\newblock Real-time intermediate flow estimation for video frame interpolation.
\newblock In \emph{Proceedings of the European Conference on Computer Vision},
pages 624--642, 2022.
\bibitem[Hui et~al.(2018)Hui, Tang, and Loy]{hui2018liteflownet}
Tak-Wai Hui, Xiaoou Tang, and Chen~Change Loy.
\newblock Liteflownet: A lightweight convolutional neural network for optical
flow estimation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 8981--8989, 2018.
\bibitem[Ilg et~al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and
Brox]{ilg2017flownet}
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy,
and Thomas Brox.
\newblock Flownet 2.0: Evolution of optical flow estimation with deep networks.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 2462--2470, 2017.
\bibitem[Jiang et~al.(2018)Jiang, Sun, Jampani, Yang, Learned-Miller, and
Kautz]{jiang2018super}
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller,
and Jan Kautz.
\newblock Super slomo: High quality estimation of multiple intermediate frames
for video interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 9000--9008, 2018.
\bibitem[Jin et~al.(2023{\natexlab{a}})Jin, Wu, Chen, Chen, Koo, and
Hahm]{jin2022unified}
Xin Jin, Longhai Wu, Jie Chen, Youxin Chen, Jayoon Koo, and Cheul-hee Hahm.
\newblock A unified pyramid recurrent network for video frame interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, 2023{\natexlab{a}}.
\bibitem[Jin et~al.(2023{\natexlab{b}})Jin, Wu, Shen, Chen, Chen, Koo, and
Hahm]{jin2023enhanced}
Xin Jin, Longhai Wu, Guotao Shen, Youxin Chen, Jie Chen, Jayoon Koo, and
Cheul-hee Hahm.
\newblock Enhanced bi-directional motion estimation for video frame
interpolation.
\newblock In \emph{Proceedings of the IEEE Winter Conference on Applications of
Computer Vision}, pages 5049--5057, 2023{\natexlab{b}}.
\bibitem[Kalantari et~al.(2016)Kalantari, Wang, and
Ramamoorthi]{kalantari2016learning}
Nima~Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi.
\newblock Learning-based view synthesis for light field cameras.
\newblock \emph{ACM Transactions on Graphics (TOG)}, 35\penalty0 (6):\penalty0
1--10, 2016.
\bibitem[Kalluri et~al.(2023)Kalluri, Pathak, Chandraker, and
Tran]{kalluri2023flavr}
Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du~Tran.
\newblock Flavr: Flow-agnostic video representations for fast frame
interpolation.
\newblock In \emph{Proceedings of the IEEE Winter Conference on Applications of
Computer Vision}, pages 2071--2082, 2023.
\bibitem[Kong et~al.(2022)Kong, Jiang, Luo, Chu, Huang, Tai, Wang, and
Yang]{kong2022ifrnet}
Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying
Tai, Chengjie Wang, and Jie Yang.
\newblock Ifrnet: Intermediate feature refine network for efficient frame
interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 1969--1978, 2022.
\bibitem[Lai and Xie(2019)]{lai2019selfsupervised}
Zihang Lai and Weidi Xie.
\newblock Self-supervised video representation learning for correspondence
flow.
\newblock In \emph{Proceedings of the British Machine Vision Conference}, 2019.
\bibitem[Lee et~al.(2020)Lee, Kim, Chung, Pak, Ban, and Lee]{lee2020adacof}
Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, and
Sangyoun Lee.
\newblock Adacof: Adaptive collaboration of flows for video frame
interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 5316--5325, 2020.
\bibitem[Li et~al.(2021)Li, Hao, Di, Gundavarapu, and Wang]{li2021test}
Yizhuo Li, Miao Hao, Zonglin Di, Nitesh~Bharadwaj Gundavarapu, and Xiaolong
Wang.
\newblock Test-time personalization with a transformer for human pose
estimation.
\newblock \emph{Advances in Neural Information Processing Systems},
34:\penalty0 2583--2597, 2021.
\bibitem[Liu et~al.(2020)Liu, Xie, Siyao, Sun, Qiao, and Dong]{liu2020enhanced}
Yihao Liu, Liangbin Xie, Li~Siyao, Wenxiu Sun, Yu~Qiao, and Chao Dong.
\newblock Enhanced quadratic video interpolation.
\newblock In \emph{Proceedings of the European Conference on Computer Vision
Workshops}, pages 41--56, 2020.
\bibitem[Liu et~al.(2019)Liu, Liao, Lin, and Chuang]{liu2019cyclicgen}
Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang.
\newblock Deep video frame interpolation using cyclic frame generation.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, pages 8794--8802, 2019.
\bibitem[Liu et~al.(2017)Liu, Yeh, Tang, Liu, and Agarwala]{liu2017dvf}
Ziwei Liu, Raymond~A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.
\newblock Video frame synthesis using deep voxel flow.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 4463--4471, 2017.
\bibitem[Loshchilov and Hutter(2019)]{loshchilov2017fixing}
Ilya Loshchilov and Frank Hutter.
\newblock Decoupled weight decay regularization.
\newblock In \emph{Proceedings of the International Conference on Learning
Representations}, 2019.
\bibitem[Lu et~al.(2022)Lu, Wu, Lin, Lu, and Jia]{lu2022vfiformer}
Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia.
\newblock Video frame interpolation with transformer.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 3532--3542, 2022.
\bibitem[Michaeli and Irani(2013)]{michaeli2013nonparametric}
Tomer Michaeli and Michal Irani.
\newblock Nonparametric blind super-resolution.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 945--952, 2013.
\bibitem[Niklaus and Liu(2018)]{niklaus2018context}
Simon Niklaus and Feng Liu.
\newblock Context-aware synthesis for video frame interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 1701--1710, 2018.
\bibitem[Niklaus and Liu(2020)]{niklaus2020softmax}
Simon Niklaus and Feng Liu.
\newblock Softmax splatting for video frame interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 5437--5446, 2020.
\bibitem[Niklaus et~al.(2017{\natexlab{a}})Niklaus, Mai, and
Liu]{niklaus2017adaconv}
Simon Niklaus, Long Mai, and Feng Liu.
\newblock Video frame interpolation via adaptive convolution.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 670--679, 2017{\natexlab{a}}.
\bibitem[Niklaus et~al.(2017{\natexlab{b}})Niklaus, Mai, and
Liu]{niklaus2017sepconv}
Simon Niklaus, Long Mai, and Feng Liu.
\newblock Video frame interpolation via adaptive separable convolution.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 261--270, 2017{\natexlab{b}}.
\bibitem[Park et~al.(2020)Park, Ko, Lee, and Kim]{park2020bmbc}
Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim.
\newblock Bmbc: Bilateral motion estimation with bilateral cost volume for
video interpolation.
\newblock In \emph{Proceedings of the European Conference on Computer Vision},
pages 109--125, 2020.
\bibitem[Park et~al.(2021)Park, Lee, and Kim]{park2021asymmetric}
Junheum Park, Chul Lee, and Chang-Su Kim.
\newblock Asymmetric bilateral motion estimation for video frame interpolation.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 14539--14548, 2021.
\bibitem[Peleg et~al.(2019)Peleg, Szekely, Sabo, and Sendik]{peleg2019imnet}
Tomer Peleg, Pablo Szekely, Doron Sabo, and Omry Sendik.
\newblock Im-net for high resolution video frame interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 2398--2407, 2019.
\bibitem[Perazzi et~al.(2016)Perazzi, Pont-Tuset, McWilliams, Van~Gool, Gross,
and Sorkine-Hornung]{perazzi2016benchmark}
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van~Gool, Markus
Gross, and Alexander Sorkine-Hornung.
\newblock A benchmark dataset and evaluation methodology for video object
segmentation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 724--732, 2016.
\bibitem[Reda et~al.(2022)Reda, Kontkanen, Tabellion, Sun, Pantofaru, and
Curless]{reda2022film}
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru,
and Brian Curless.
\newblock Film: Frame interpolation for large motion.
\newblock In \emph{Proceedings of the European Conference on Computer Vision},
2022.
\bibitem[Reda et~al.(2019)Reda, Sun, Dundar, Shoeybi, Liu, Shih, Tao, Kautz,
and Catanzaro]{reda2019unsupervised}
Fitsum~A Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu,
Kevin~J Shih, Andrew Tao, Jan Kautz, and Bryan Catanzaro.
\newblock Unsupervised video interpolation using cycle consistency.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 892--900, 2019.
\bibitem[Shi et~al.(2021)Shi, Liu, Shi, Dai, and Chen]{shi2021video}
Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai, and Jun Chen.
\newblock Video frame interpolation via generalized deformable convolution.
\newblock \emph{IEEE Transactions on Multimedia}, 24:\penalty0 426--439, 2021.
\bibitem[Shi et~al.(2022)Shi, Xu, Liu, Chen, and Yang]{shi2022vfit}
Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming-Hsuan Yang.
\newblock Video frame interpolation transformer.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 17482--17491, 2022.
\bibitem[Shocher et~al.(2018)Shocher, Cohen, and Irani]{shocher2018zero}
Assaf Shocher, Nadav Cohen, and Michal Irani.
\newblock “zero-shot” super-resolution using deep internal learning.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 3118--3126, 2018.
\bibitem[Sim et~al.(2021)Sim, Oh, and Kim]{sim2021xvfi}
Hyeonjun Sim, Jihyong Oh, and Munchurl Kim.
\newblock Xvfi: extreme video frame interpolation.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 14489--14498, 2021.
\bibitem[Sun et~al.(2018)Sun, Yang, Liu, and Kautz]{sun2018pwc}
Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.
\newblock Pwc-net: Cnns for optical flow using pyramid, warping, and cost
volume.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 8934--8943, 2018.
\bibitem[Sun et~al.(2020)Sun, Wang, Liu, Miller, Efros, and Hardt]{sun2020test}
Yu~Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.
\newblock Test-time training with self-supervision for generalization under
distribution shifts.
\newblock In \emph{Proceedings of the International Conference on Machine
Learning}, pages 9229--9248, 2020.
\bibitem[Teed and Deng(2020)]{teed2020raft}
Zachary Teed and Jia Deng.
\newblock Raft: Recurrent all-pairs field transforms for optical flow.
\newblock In \emph{Proceedings of the European Conference on Computer Vision},
pages 402--419, 2020.
\bibitem[Wang et~al.(2019)Wang, Jabri, and Efros]{wang2019learning}
Xiaolong Wang, Allan Jabri, and Alexei~A Efros.
\newblock Learning correspondence from the cycle-consistency of time.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 2566--2576, 2019.
\bibitem[Wu et~al.(2018)Wu, Singhal, and Krahenbuhl]{wu2018video}
Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl.
\newblock Video compression through image interpolation.
\newblock In \emph{Proceedings of the European Conference on Computer Vision},
pages 416--431, 2018.
\bibitem[Wu and Wang(2021)]{wu2021contrastive}
Haiping Wu and Xiaolong Wang.
\newblock Contrastive learning of image representations with cross-video
cycle-consistency.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 10149--10159, 2021.
\bibitem[Xu et~al.(2019)Xu, Siyao, Sun, Yin, and Yang]{xu2019quadratic}
Xiangyu Xu, Li~Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang.
\newblock Quadratic video interpolation.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Xue et~al.(2019)Xue, Chen, Wu, Wei, and Freeman]{xue2019toflow}
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William~T Freeman.
\newblock Video enhancement with task-oriented flow.
\newblock \emph{International Journal of Computer Vision}, 127:\penalty0
1106--1125, 2019.
\bibitem[Zhang et~al.(2023)Zhang, Zhu, Wang, Chen, Wu, and
Wang]{Zhang2023ExtractingMA}
Guozhen Zhang, Yuhan Zhu, Hongya Wang, Youxin Chen, Gangshan Wu, and Limin
Wang.
\newblock Extracting motion and appearance via inter-frame attention for
efficient video frame interpolation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, 2023.
\bibitem[Zhou et~al.(2022)Zhou, Li, Xie, Zhang, Wang, and Zhang]{zhou2022aris}
Qinye Zhou, Ziyi Li, Weidi Xie, Xiaoyun Zhang, Yan-Feng Wang, and Ya~Zhang.
\newblock A simple plugin for transforming images to arbitrary scales.
\newblock In \emph{Proceedings of the British Machine Vision Conference}, 2022.
\bibitem[Zhou et~al.(2016)Zhou, Krahenbuhl, Aubry, Huang, and
Efros]{zhou2016learning}
Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei~A
Efros.
\newblock Learning dense correspondence via 3d-guided cycle consistency.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 117--126, 2016.
\bibitem[Zhu et~al.(2017)Zhu, Park, Isola, and Efros]{zhu2017unpaired}
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei~A Efros.
\newblock Unpaired image-to-image translation using cycle-consistent
adversarial networks.
\newblock In \emph{Proceedings of the International Conference on Computer
Vision}, pages 2223--2232, 2017.
\end{thebibliography}