Robust Ego-Exo Correspondence with Long-Term Memory
Extracted source text · arXiv 2510.11417
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
% \usepackage{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
\usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage{arydshln}
\usepackage{booktabs}
% \usepackage{hyperref}
\usepackage{subcaption}
\usepackage{caption}
\usepackage{wrapfig}
\usepackage{amssymb} % for checkmark
\usepackage[table]{xcolor} % for row color
\usepackage[table]{xcolor}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
% \usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\usepackage{tikz}
\usepackage{amsmath}
\usepackage{pifont}
\definecolor{cvprblue}{rgb}{0.21,0.49,0.74}
\usepackage[breaklinks,colorlinks,citecolor=cvprblue,urlcolor=black]{hyperref}
\title{Robust Ego-Exo Correspondence with \\ Long-Term Memory}
\author{
Yijun Hu$^{1*}$\;\;\;\;\; Bing Fan$^{2*}$ \;\;\;\;\; Xin Gu$^{1}$ \;\;\;\;\; Haiqing Ren$^{3,1}$ \;\;\;\;\; Dongfang Liu$^{4}$ \\ \textbf{Heng Fan}$^{2\dag}$ \;\;\;\;\; \textbf{Libo Zhang}$^{3,1\dag}$ \\
$^{1}$University of Chinese Academy of Sciences \;\;\; $^{2}$University of North Texas \\
$^{3}$Institute of Software Chinese Academy of Sciences \;\;\;
$^{4}$Rochester Institute of Technology \\
$^*$Equal contribution and co-first authors \;\;\; $^\dag$Equal advising and corresponding authors \\
% examples of more authors
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\makeatletter
\renewcommand{\@notice}{}
\makeatother
\begin{document}
\maketitle
\begin{abstract}
Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint variations, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM 2 encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a dual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Experts (MoE). Compared to SAM 2, our approach features \textbf{(i)} a Memory-View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and \textbf{(ii)} a dual-memory bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed \textbf{\emph{LM-EEC}}, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at \url{https://github.com/juneyeeHu/LM-EEC}.
\end{abstract}
\section{Introduction}
\label{sec:intro}
Aligning observations of objects in egocentric and exocentric viewpoints can facilitate many applications. In augmented reality (AR), a person wearing smart glasses could quickly pick up new skills with a virtual intelligent coach that provides real-time guidance. In robotics, a robot watching people in its environment is able to acquire new dexterous manipulation skills with less physical experience. The recent work EgoExo4D~\cite{EgoExo4D} makes significant contributions to establishing object-level correspondence between these ego-exo views by introducing a benchmark featuring annotated, temporally synchronized egocentric and exocentric videos along with object segmentation masks.
The first-person (ego) perspective captures fine-grained details of hand-object interactions and the camera wearer’s attention, while the third-person (exo) perspective provides a broader view of full-body poses and the surrounding environment. Therefore, this task presents significant challenges, including extreme viewpoint variations, substantial object occlusions, and the presence of numerous small objects. Previous approaches, such as XSegTx~\cite{EgoExo4D}, which formulates the task as a matching problem, and XView-XMem~\cite{EgoExo4D}, which adapts XMem~\cite{XMem} to track objects across different views, struggle to effectively address these challenges as depicted on the left of Figure~\ref{fig:intro}, highlighting the need for more robust solutions. Recently, a unified model for both video and image segmentation, called Segment Anything Model 2 (SAM 2)~\cite{SAM2}, has been introduced. Trained on the SA-V dataset~\cite{SAM2}, which contains 50.9K videos with 642.6K masklets, SAM 2 demonstrates strong generalization capabilities and adaptability across a wide range of scenarios and tasks~\cite{sam2unet,samurai}. The right panel of Figure~\ref{fig:intro} shows the strong performance of SAM 2 on our task. ``Ego-Exo'' refers to segmenting objects in the exocentric videos, given the corresponding egocentric videos and annotations, while ``VOS'' refers to segmenting objects in the exocentric videos using only the first-frame annotations.
However, SAM 2 still faces challenges in establishing object-level correspondence between temporally synchronized egocentric and exocentric videos. A key issue arises when using annotations from one view as prompts to segment the other. Effective fusion between two views is crucial, yet SAM 2 simply adds memory-aware features and prompt embeddings together, completely overlooking the gap in different features and distribution differences between two views. Another major challenge is the scale of the task, which involves processing large volumes of long videos. Yet SAM 2 retains only a few frames closest to the current frame in its memory bank. As a result, long-term information is not effectively preserved, leading to degradation in complicated scenarios.%forget earlier target states and scene context.
To overcome these limitations, we first introduce an adaptive Mixture-of-Experts (MoE) strategy that dynamically integrates diverse knowledge-fused features based on their distinct characteristics, enabling them to complement each other effectively. Additionally, optimizing SAM 2’s memory management is crucial. Current approaches~\cite{SAM2,samurai,sam2long,medicalsam2}, which directly store recent or selected frames in the memory bank, introduce redundant features while failing to retain long-term information. Addressing these challenges is essential to fully harness SAM 2’s capabilities for robust object segmentation in ego-exo correspondence scenarios.
\begin{figure}[!t]
\centering
\includegraphics[width=1\linewidth]{images/intro12.png}
\caption{
\textbf{Left:} Comparison of segmentation results between XView-XMem and our model, using exocentric videos as an example.
\textbf{Right:} Quantitative results on the EgoExo4D validation set.
}
\label{fig:intro}
\vspace{-2mm}
\end{figure}
we propose LM-EEC, a SAM-based robust segmentation model with dual compressed long-term memory and an adaptive fusion module. Our method introduces two key advancements: \textbf{(i)} a flexible Memory-View Mixture-of-Experts module that dynamically reweights expert features based on their characteristics, enabling more effective integration of complementary information, and \textbf{(ii)} a dual-memory bank system that separates ego and exo memory banks and leverages a specially designed compression strategy to efficiently preserve critical long-term information. In our extensive evaluation on the challenging EgoExo4D benchmark, LM-EEC achieves new state-of-the-art results and significantly outperforms other models including SAM 2.
In summary, our main \textbf{\emph{contributions}} are as follows: \ding{171} We enhance SAM 2’s segmentation capabilities for the ego-exo correspondence task by introducing a specialized MoE mechanism that effectively integrates features from both egocentric and exocentric views; \ding{170} To efficiently manage long-term dependencies, we design a dual-memory bank system that incorporates a view-specific compression strategy, effectively reducing redundancy while retaining critical long-term information; \ding{168} Extensive experiments on EgoExo4D demonstrate that our model achieves state-of-the-art performance, showcasing its strong generalization ability across diverse real-world scenarios.
\section{Related work}
% \label{sec:formatting}
\textbf{Video object segmentation}. Video Object Segmentation (VOS) involves tracking an object throughout a video given its mask in the first frame~\cite{davis}. This task is classified as ``semi-supervised VOS'' since the initial object mask serves as a supervision signal available only in the first frame. VOS has attracted significant attention due to its broad applications in areas such as video editing and robotics.
Early deep learning-based models often rely on online fine-tuning~\cite{online1,online2,online3,online4,online5,online6,online7,online8}, either on the first frame or across all frames, to adapt the model to the target object. To improve inference speed, offline-trained models were introduced, leveraging conditioning on either the first frame alone~\cite{first_frame1,first_frame2} or incorporating information from previous frames~\cite{previous_frame1,previous_frame2}. These conditioning strategies have since been extended to all frames using RNNs~\cite{rnn} and transformers~\cite{transformer1,transformer2,transformer3,transformer4,XMem,transformer5,transformer6}.
Recently, the Segment Anything Model 2 (SAM 2)~\cite{SAM2} has emerged as a unified foundational model for promptable object segmentation in both images and videos. Notably, SAM 2~\cite{SAM2} has established new state-of-the-art benchmarks across various VOS tasks, significantly outperforming previous methods. For instance, SAM2-UNet~\cite{sam2unet} introduces a simple yet effective U-shaped architecture for versatile segmentation across both natural and medical domains, while SAMURAI~\cite{samurai}, an enhanced adaptation of SAM 2 tailored for visual object tracking, achieves substantial improvements in both success rate and precision over existing tracking models. Building upon this foundation, we develop our model based on SAM 2~\cite{SAM2} to further enhance segmentation performance in our task.
%\subsection{Long-term video models}
\textbf{Long-term video models}. Long-term video understanding focuses on capturing long-range dependencies in the videos. The primary challenge lies in balancing the retention of crucial information while maintaining computational efficiency. To address this, a common strategy involves utilizing pre-extracted features, eliminating the need for joint training of backbone architectures~\cite{joint_train1,joint_train2,joint_train3,joint_train4,joint_train5}. Alternatively, some studies have explored sparse video sampling techniques~\cite{sampling1,sampling2}, which reduce the number of input frames by selectively preserving salient content.
Another widely adopted approach is the use of memory banks to retain relevant features, particularly in video object segmentation. For instance, XMem~\cite{XMem} employs multiple independent yet highly interconnected feature memory stores: a rapidly updated sensory memory, a high-resolution working memory, and a compact but persistent long-term memory. Similarly, RMem~\cite{RMem} and SAM 2~\cite{SAM2} constrain memory banks to a limited set of essential frames, striking a balance between relevance and freshness when updating feature representations.
\section{The Proposed Method}
\subsection{Task definition}
Given a pair of synchronized ego-exo videos and a sequence of query masks for an object of interest in one of the videos, the objective is to identify the corresponding masks of the same object in each synchronized frame of the other view, if visible. This task consists of two scenarios: one where the ego view serves as the query to segment the object in the exo view, and the other where the exo view serves as the query to segment the object in the ego view. These two settings are referred to as \textit{\textbf{Ego-to-Exo correspondence}} and \textit{\textbf{Exo-to-Ego correspondence}}, denoted as \textit{\textbf{Ego2Exo}} and \textit{\textbf{Exo2Ego}}, respectively, in the following for brevity.
Formally, taking \textit{\textbf{Ego2Exo}} as an example, let an ego-view video with $T$ frames and a corresponding exo-view video with $T$ frames be represented as $\lbrace I_{t}^{ego}, G_{t}^{ego} \rbrace_{t=1}^{T}$ and $\lbrace I_{t}^{exo} \rbrace_{t=1}^{T}$, respectively. Here, $G_{t}^{ego}$ denotes the ground-truth object mask of frame $I_{t}^{ego}$. Given these synchronized frames, the goal is to segment the object masks $M_{t}^{exo}$ in each frame of the exo-view video.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/overview5.png}
\caption{Overview of our proposed model, which consists of three key components: multi-view encoding, dual memory compression, and object mask prediction. The multi-view encoder extracts features from egocentric and exocentric videos, using a Memory-View MoE module to adaptively combine memory-aware and view-specific representations. The object mask prediction module then generates segmentation masks for objects in the exocentric view. To capture long-term dependencies efficiently, we apply dual memory compression on the memory banks from both viewpoints.}
\label{fig:model}
\vspace{-2mm}
\end{figure}
\subsection{Preliminary on SAM 2 }
SAM 2~\cite{SAM2} begins with an image encoder that encodes each input frame into embeddings. In contrast to SAM~\cite{SAM}, where frame embeddings are directly fed into the mask decoder, SAM 2 designs a memory module that conditions features of the current frame on both previous and prompted frames.
Specifically, for semi-supervised video object segmentation tasks, SAM 2 maintains a memory bank at each time step $t\ge1$:
\begin{equation}
\mathcal{M}_t = \left\{ \mathbf{M}_\tau \in \mathbb{R}^{K \times C} \right\}_{\tau \in \mathcal{I}}
\end{equation}
where K is the number of memory tokens per frame, C is the channel dimension, and $\mathcal{I}$ is the set of frame indices included in the memory. In SAM 2, the memory bank stores up to N of the most recent frames, along with the initial mask, using a First-In-First-Out (FIFO) queue mechanism.
The encoded frame feature would be fused with the memory bank through the memory attention module, which is stacked by L transformer blocks, facilitating full interaction between the two. After this memory fusion, if mask prompts are present, the fused feature is summed with the dense prompt embedding, which is then processed by the convolutions of the prompt encoder.
\subsection{Overview}
Figure~\ref{fig:model} provides an overview of our proposed model LM-EEC. Taking \textit{\textbf{Ego2Exo}} as an example, given the frames and target object masks from the ego view along with the corresponding frames from the exo view (left side of Figure~\ref{fig:model}), the workflow then unfolds as follows:
First, the video frames from both views are encoded using the image encoder. The feature representations of the current exo and ego frames are then passed through the dual attention module, where the exo feature is integrated separately by the attention module with the stored features from both the ego and exo memory banks, as well as the corresponding annotated ego feature which is encoded by the off-the-shelf memory encoder. Once the memory-aware and view-specific features are obtained, the Memory-View Mixture-of-Experts module(sec~\ref{sec:3.4}) adaptively integrates the two expert features. The fused feature is subsequently processed by the mask decoder to generate the predicted object mask.
Finally, the predicted mask, along with the given ego-view mask and their corresponding encoded features, are fed into the memory encoder to generate memory frames, which are stored in the ego and exo memory banks. As the dual memory banks have a fixed capacity, the tailored memory compression mechanism(sec~\ref{sec:3.5.1}) is triggered when the stored frames exceed this limit, ensuring efficient memory management.
\begin{figure}[t]
\centering
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
\includegraphics[width=0.98\linewidth]{images/MV-MoE1.png}
\caption{Overview of the proposed Memory-View Mixture-of-Experts (MV-MoE) module. Channel- and spatial-wise routers generate dynamic weights to recalibrate memory-aware and view-specific features, enabling adaptive and context-aware fusion of complementary information.}
\label{fig:gated fusion}
\vspace{-2mm}
\end{figure}
\subsection{Memory-View Mixture-of-Experts module}
\label{sec:3.4}
SAM 2 simply adds together the prompt embedding generated by the prompt encoder and the memory-aware feature. In our task, the memory-aware feature and the view-specific feature, which serves as prompt information, highlight different regions and have different feature distributions. This may cause the prompt information to overwhelm the representation of the other branch, leading to less discriminative and robust target representations.
To alleviate this issue, we propose the \textbf{Memory-View Mixture-of-Experts (MV-MoE) module}, which treats the memory-aware and view-specific features as two complementary experts. Rather than performing direct fusion, we design a lightweight dual-branch routing mechanism to adaptively assign contribution weights to each expert along both channel and spatial dimensions, allowing the network to determine the relative importance of each expert in a data-dependent manner.
Specifically, as shown in Figure~\ref{fig:gated fusion}, given the memory-aware feature $F_{{mem}} \in \mathbb{R}^{h \times w \times c}$ and view-specific feature $F_{{view}} \in \mathbb{R}^{h \times w \times c}$, channel-wise routing is performed firstly. We concatenate the two features along the channel dimension and apply global average pooling to obtain a compact descriptor. This descriptor is then passed through two separate MLPs—each consisting of two linear layers, with ReLU and sigmoid activations—to generate dynamic channel-wise importance weights $\mathbf{w}_{{mem}}^c, \mathbf{w}_{{view}}^c \in \mathbb{R}^{1 \times 1 \times c}$. These weights are used to recalibrate each expert through residual modulation:
\begin{equation}
\mathbf{w}_{mem/view}^c = MLP_{1/2} (Avg(Concat(F_{mem},F_{view}))),
\end{equation}
\begin{equation}
\dot{F}_{mem/view} = \mathbf{w}_{mem/view}^c \otimes F_{mem/view} + F_{mem/view},
\end{equation}
where ${Concat}(\cdot)$ and ${Avg}(\cdot)$ denote channel-wise concatenation and global average pooling, respectively, and $\otimes$ represents element-wise multiplication. This operation adaptively emphasizes the more informative channels in each expert while preserving the original feature content.
Subsequently, spatial-wise routing is applied to the channel-enhanced features $\dot{F}_{mem}$ and $\dot{F}_{view}$. The two features are concatenated and fed into two parallel convolutional branches—each composed of a Conv–ReLU–Conv–Sigmoid sequence—to generate spatial attention maps $\mathbf{w}_{mem}^s, \mathbf{w}_{view}^s \in \mathbb{R}^{h \times w \times 1}$. These spatial weights are then used to further refine each feature spatially, again using residual modulation:
\begin{equation}
\mathbf{w}_{{mem/view}}^s = Conv_{1/2} (Concat(\dot{F}_{mem},\dot{F}_{view})),
\end{equation}
\begin{equation}
\ddot{F}_{mem/view} = \mathbf{w}_{mem/view}^c \otimes \dot{F}_{mem/view} + \dot{F}_{mem/view},
\end{equation}
Finally, the refined memory-aware and view-specific features are summed to produce the fused target feature which is then forwarded to the decoder for subsequent segmentation:
\begin{equation}
{F}_{tar} = \ddot{F}_{mem} + \ddot{F}_{view}.
\end{equation}
These routing strategies enable the model to adaptively modulate the contributions of memory-aware and view-specific features across both spatial and channel dimensions, aligning with the core intuition of the Mixture-of-Experts (MoE) paradigm—selectively leveraging complementary sources of knowledge based on the input context. Unlike conventional MoE architectures that rely on multiple sub-networks and sparse expert activation, our design performs \textbf{dense, feature-level expert routing}, which preserves the adaptive expert weighting behavior of MoE while avoiding the complexity of network-level sparsity. This fine-grained and lightweight formulation enables context-aware fusion and enhances the model’s flexibility in dynamic video scenarios.
\begin{figure}[!t]
\centering
\includegraphics[width=0.98\linewidth]{images/compression.png}
\caption{An illustration of our memory bank compression strategy, which preserves a fixed memory size in both ego-view and exo-view memory banks by aggregating temporally redundant information.}
\label{fig:temporal_com}
\vspace{-2mm}
\end{figure}
\subsection{Dual Memory Compression}
\label{sec:3.5.1}
Previous work~\cite{EgoExo4D} constructs a single memory bank that stores both egocentric and exocentric features uniformly. However, egocentric and exocentric videos differ substantially in terms of viewpoints, motion patterns, and visual appearance. To better leverage their complementary characteristics, we first design a dual memory bank based on SAM 2, in which ego and exo features are stored separately.
SAM 2 maintains the most recent N frame features in its memory bank. Although effective in the short-term context, this strategy suffers from several limitations. As segmentation progresses, the memory bank gradually loses alignment with earlier frames, reducing its temporal consistency. Meanwhile, since the target object typically occupies only a small region of each frame, the memory bank becomes saturated with redundant spatial information. More critically, this unified memory design fails to differentiate between egocentric and exocentric views, thereby undermining our dual-memory objective.
To address these challenges, we propose a view-specific memory compression strategy that separately compresses the ego and exo memory banks. This approach reduces redundancy, preserves discriminative long-term features, and improves both efficiency and robustness.
Figure~\ref{fig:temporal_com} illustrates our memory compression process. The core idea is to temporally aggregate and compress video features by exploiting similarities between adjacent frames, while retaining informative long-term content. This enables us to maintain compact and discriminative memory representations for both ego and exo views.
The compression algorithm is applied at each auto-regressive iteration whenever the length of the ego/exo memory bank exceeds a pre-defined threshold \(M\). Formally, given an ego/exo memory bank consisting of the following sequence:
\begin{equation}
[f^{1}, f^{2}, \dots, f^{M}], \quad f^{t} \in \mathbb{R}^{P \times C},
\end{equation}
where \(P = h \times w\), representing the spatial dimensions of each memory frame feature. When a new frame feature \(f^{\text{M+1}}\) arrives, the memory bank must be compressed by reducing its length by one.
For each spatial location \(i\), we first compute the Euclidean distance between the corresponding feature points in the temporally adjacent memory frames:
\begin{equation}
d_{i}^{t} = \text{Eucli}(f_{i}^{t}, f_{i}^{\text{t+1}}), \quad t \in [1, M], \quad i \in [1, P],
\end{equation}
Next, we identify the most temporally redundant features by selecting the feature with the minimum Euclidean distance, indicating the highest similarity across time:
\begin{equation}
k = \arg\min(d_{i}^{t}), \quad t \in [1, M].
\end{equation}
Finally, to reduce the memory bank length by one, we perform a simple feature averaging at each spatial location:
\begin{equation}
f_{i}^{k} = \frac{f_{i}^{k} + f_{i}^{\text{k+1}}}{2}.
\end{equation}
This three-in-one approach not only reduces redundancy but also preserves long-term information. Furthermore, it adaptively updates the memory bank based on scene variations in each viewpoint, ensuring a compact and informative memory representation.
\section{Experiments}
\subsection{Dataset}
We conduct experiments on the challenging EgoExo4D benchmark~\cite{EgoExo4D}. EgoExo4D is a dataset specifically designed for the ego-exo correspondence task. It contains approximately 5.5K annotated objects across 1.3K temporally synchronized egocentric and exocentric video pairs. In total, around 4 million frames have been annotated, resulting in 742K ego-view and 1.1M exo-view paired segmentation masks. We adopt the official dataset split in our experiment, where 756 videos are used for training, 202 for validation, and 291 for testing.
\subsection{Metrics}
\label{sec:metrics}
Following~\cite{EgoExo4D}, we employ four evaluation metrics, including Intersection Over Union (\textbf{IoU}) between the predicted and ground-truth masks, Location Error (\textbf{LE}), which is defined as the normalized distance between the centroids of the predicted and ground-truth masks, Contour Accuracy (\textbf{CA}), which measures how well the predicted masks match the ground-truth masks on the boundary, and Existence Balance Accuracy (\textbf{BA}), which evaluates the method's ability to estimate object visibility in the target view, since, in practice, objects may often be occluded or fall outside the field of view.
\subsection{Implementation Details}
Our model is built upon the official SAM 2 base~\cite{SAM2}, with key modifications including the introduction of Memory-View MoE module and a refined memory bank construction strategy. Following the training protocol of SAM 2, we sample 8 consecutive frames for each object from ego-exo video pairs, and set the memory bank size to 6. To reduce computational overhead due to the large resolution of original video frames, we resize all frames to $480 \times 480$, following the practice in~\cite{EgoExo4D}. Given the large-scale nature of EgoExo4D, we train all modules jointly based on the pre-trained SAM 2 checkpoint, without freezing any components. The model is trained for 60 epochs on 8 NVIDIA A100 GPUs with a batch size of 32 and evaluated on a single V100 GPU, achieving an inference speed of approximately 8.4 FPS. Memory compression is applied only during inference. Our code and model are available at \url{https://github.com/juneyeeHu/LM-EEC}.
\subsection{Comparison to State-of-the-art Methods}
\label{sec:comparison}
\textbf{Table~\ref{tab:ego_exo_benchmark}} presents the quantitative results in comparison with previous methods on the \textbf{test set}. We experiment with two settings: providing the ground-truth object track in the exo view (exo query mask) and predicting it in the ego view \textit{\textbf{Exo2Ego}}), and vice versa. Given the limited availability of existing models tailored for this task, we selected several VOS models from 2021 onward~\cite{stcn,qdmn,gsfm,simvos,cutie} and adjusted their architectures in a manner similar to XView-XMem~\cite{EgoExo4D}, namely, integrating features from the other view with annotations before segmenting the current frame. All models are trained or fine-tuned on the EgoExo4D dataset to ensure a fair comparison. To provide a more comprehensive comparison, we additionally report the performance of our \textbf{base model}, which comprises only a standard dual-memory bank and simple prompt addition, to intuitively highlight the effectiveness of our proposed method. As shown in Figure~\ref{fig:obj_size_all}, we also evaluate and compare the segmentation performance of our method across varying object sizes under the \textbf{\textit{Ego2Exo}} task. Specifically, we divide the \textbf{test set} based on the proportion of object pixels within each frame. Furthermore, we provide visual results, conduct qualitative comparisons with other baselines, and analyze the model’s performance across diverse activity scenarios in the EgoExo4D dataset. For more details, please kindly refer to the \textbf{\emph{appendix}}.
\begin{table}[t]
\centering
\caption{Comparison for the ego-exo correspondence benchmark on \textbf{test set}. Best results are reported in \textbf{bold}, whereas our results are highlighted in \colorbox{cyan!10}{cyan} and the results of the base model are \underline{underlined}.}
% \small
\vspace{4pt}
\footnotesize
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{12pt}
\begin{tabular}{llccccc}
\toprule
Query Mask & Method & IoU$\uparrow$ & LE$\downarrow$ & CA$\uparrow$ & BA$\uparrow$ \\
\midrule
Ego & XSegTx~\cite{EgoExo4D} & 18.99 & 0.070 & 0.386 & 66.31 \\
Ego & XView-Xmem (w/ finetuning)~\cite{EgoExo4D} & 14.84 & 0.115 & 0.242 & 61.24 \\
Ego & XView-Xmem (+ XSegTx)~\cite{EgoExo4D} & 34.90 & 0.038 & 0.559 & \textbf{66.79} \\
Ego & STCN~\cite{stcn} & 27.39 & 0.109 & 0.378 & 61.97 \\
Ego & QDMN~\cite{qdmn} & 27.03 & 0.108 & 0.346 & 63.16 \\
Ego & GSFM~\cite{gsfm} & 3.98 & 0.146 & 0.057 & 52.39 \\
Ego & SimVOS~\cite{simvos} & 38.26 & 0.090 & 0.481 & 57.21 \\
Ego & Cutie~\cite{cutie} & 27.03 & 0.108 & 0.346 & 59.18 \\
Ego & Base model & \underline{52.13} & \underline{0.024} & \underline{0.734} & \underline{61.56} \\
\rowcolor{cyan!10}
Ego & LM-EEC(Ours) & \textbf{54.98} & \textbf{0.017} & \textbf{0.778} & 64.22 \\
\midrule
Exo & XSegTx~\cite{EgoExo4D} & 27.14 & 0.104 & 0.358 & \textbf{82.01} \\
Exo & XView-Xmem (w/ finetuning)~\cite{EgoExo4D} & 21.37 & 0.139 & 0.269 & 61.72 \\
Exo & XView-Xmem (+ XSegTx)~\cite{EgoExo4D} & 25.00 & 0.117 & 0.327 & 59.71 \\
Exo & STCN~\cite{stcn} & 27.61 & 0.122 & 0.348 & 65.98 \\
Exo & QDMN~\cite{qdmn} & 17.56 & 0.158 & 0.213 & 58.94\\
Exo & GSFM~\cite{gsfm} & 13.78 & 0.164 & 0.169 & 59.53 \\
Exo & SimVOS~\cite{simvos} & 40.67 & 0.099 & 0.481& 66.62 \\
Exo & Cutie~\cite{cutie} & 47.52 & 0.070 & 0.579 & 70.71 \\
Exo & Base model & \underline{57.27} & \underline{0.047} & \underline{0.677} & \underline{57.11} \\
\rowcolor{cyan!10}
Exo & LM-EEC(Ours) & \textbf{65.77} & \textbf{0.031} & \textbf{0.774} & 58.14 \\
\bottomrule
\end{tabular}
\label{tab:ego_exo_benchmark}
\vspace{-2mm}
\end{table}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.32\linewidth}
\centering
\includegraphics[width=\linewidth]{images/obj_size_iou.png}
% \caption{IoU evaluation.}
\label{fig:obj_size_iou}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.32\linewidth}
\centering
\includegraphics[width=\linewidth]{images/obj_size_shape_acc.png}
% \caption{Shape accuracy evaluation.}
\label{fig:obj_size_shapeacc}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.32\linewidth}
\centering
\includegraphics[width=\linewidth]{images/obj_size_location_score.png}
% \caption{Location score evaluation.}
\label{fig:obj_size_location_score}
\end{subfigure}
\caption{Performance evaluation across different object sizes in the target (exo) view, including IoU, shape accuracy, and location score.}
\label{fig:obj_size_all}
\vspace{-1mm}
\end{figure*}
Overall, our model consistently outperforms all baselines across different object sizes and delivers substantial improvements over the base model, particularly in the \textit{\textbf{Exo2Ego}} task. XSegTx~\cite{EgoExo4D} achieves a high level of balanced accuracy because it is a co-segmentation model and employs specialized data augmentation techniques during training.
\subsection{Ablations}
We conduct ablation studies on the \textbf{validation set} of EgoExo4D for the \textit{\textbf{Ego2Exo}} task. Evaluation metrics are reported, with our final configuration highlighted in \colorbox{cyan!10}{cyan}.
\begin{figure*}[t]
\centering
% 第一行：1 和 2
\begin{minipage}[t]{0.5\textwidth}
\centering
\captionof{table}{Ablation on our fusion mechanism.}
\footnotesize
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{5pt}
\begin{tabular}{lccc}
\toprule
\textbf{Setting} & IoU$\uparrow$ & LE$\downarrow$ & CA$\uparrow$ \\
\midrule
w/o other view & 0.5691 & 0.0276 & 0.7660 \\
simply add (base model) & 0.5673 & 0.0258 & 0.7643 \\
\rowcolor{cyan!10}
MV-MoE & \textbf{0.5925} & \textbf{0.0198} & \textbf{0.8006} \\
\bottomrule
\end{tabular}
\label{tab:prompt_ablation}
\end{minipage}
\hfill
\begin{minipage}[t]{0.46\textwidth}
\centering
\captionof{table}{Ablation on training frame number.}
\footnotesize
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{5pt}
\begin{tabular}{lccc}
\toprule
\textbf{Frame} & IoU$\uparrow$ & LE$\downarrow$ & CA$\uparrow$ \\
\midrule
4 & 0.5857 & 0.0204 & 0.7931 \\
\rowcolor{cyan!10}
8 & 0.5925 & 0.0198 & 0.8006 \\
10 & 0.5927 & 0.0201 & 0.8003 \\
\bottomrule
\end{tabular}
\label{tab:frame_num}
\end{minipage}
\vspace{0.4cm} % 间距
% 第二行：3 和 4
\begin{minipage}[t]{0.5\textwidth}
\centering
\captionof{table}{Ablation on our memory stores.}
\footnotesize
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{7pt}
\begin{tabular}{lccccc}
\toprule
\multicolumn{1}{l}{\textbf{Setting}} & IoU$\uparrow$ & LE$\downarrow$ & CA$\uparrow$ \\
\midrule
\rowcolor{cyan!10}
All memory stores & \textbf{0.5925} & \textbf{0.0198} & \textbf{0.8006} \\
No ego mem. & 0.5748 & 0.0220 & 0.7837 \\
No exo mem. & 0.5420 & 0.0293 & 0.7495 \\
\bottomrule
\end{tabular}
\label{tab:memory_ablation}
\end{minipage}
\hfill
\begin{minipage}[t]{0.46\textwidth}
\centering
\captionof{table}{Ablation on memory size.}
\footnotesize
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{lccc}
\toprule
\textbf{Memory} & IoU$\uparrow$ & LE$\downarrow$ & CA$\uparrow$ \\
\midrule
4 & 0.5921 & 0.0205 & 0.8004 \\
\rowcolor{cyan!10}
6 & 0.5925 & 0.0198 & 0.8006 \\
8 & 0.5963 & 0.0196 & 0.8025 \\
\bottomrule
\end{tabular}
\label{tab:memory_size}
\end{minipage}
\end{figure*}
\noindent \textbf{MV-MoE module}
Table~\ref{tab:prompt_ablation} presents the ablation study on our MV-MoE module. We compare two different experimental settings: (1) excluding the prompt from the other view of the current frame, just using the memory-aware feature for segmentation, (2) directly adding the prompt and the memory-aware feature, as done in SAM 2. Our results indicate that simple prompt addition fails to fully exploit the effectiveness of the prompts. In contrast, our proposed module leverages a routing mechanism to selectively emphasize and integrate key information from both sources, leading to more effective fusion. In \textbf{\emph{appendix}}, we also demonstrate the modularity of MV-MoE by integrating it into other
backbones.
\noindent \textbf{Memory stores.}
Table~\ref{tab:memory_ablation} summarizes the performance of our model when each memory bank is ablated individually. The results clearly demonstrate the effectiveness of our dual-memory design and underscore the importance of integrating both egocentric and exocentric views to achieve robust perception in complex scenarios.
\begin{wraptable}{r}{0.4\textwidth} % r=右侧放置；l=左侧；宽度=表格宽度
\centering
\footnotesize
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{4pt}
\caption{Ablation on our memory compression strategy.}
\begin{tabular}{lccc}
\toprule
\textbf{Setting} & IoU$\uparrow$ & LE$\downarrow$ & CA$\uparrow$ \\
\midrule
FIFO & 0.5823 & 0.0214 & 0.7874 \\
Cluster & 0.5867 & 0.0208 & 0.7942 \\
IoU Sel. & 0.5880 & 0.0204 & 0.7948 \\
\rowcolor{cyan!10}
Ours & \textbf{0.5925} & \textbf{0.0198} & \textbf{0.8006} \\
\bottomrule
\end{tabular}
\label{tab:temporal_compression_ablation}
\end{wraptable}
\noindent \textbf{Memory compression strategies.} Table~\ref{tab:temporal_compression_ablation} compares different memory compression strategies. The FIFO (first-in, first-out) approach stores the most recent N frames in the memory bank. The IoU selection strategy~\cite{samurai} follows a conditional FIFO approach based on IoU scores predicted by SAM 2. In contrast, the cluster selection method first applies average pooling to all frames, followed by clustering to identify and retain the most representative frames. Notably, these methods operate at the frame level, whereas our proposed method performs compression at the feature point level. The results demonstrate that our strategy more effectively aids the model in segmenting objects.
\noindent \textbf{Frame number and memory size.}
As show in Table~\ref{tab:frame_num}, increasing the number of frames brings gains although there could be some variance and we use a default of 8 to balance speed and accuracy. Meanwhile, increasing the (maximum) number of memories \(M\), generally helps the performance, as in Table~\ref{tab:memory_size}. We use a default value of 6 past frames to strike a balance between temporal context length and computational cost.
Due to space limitation, we include more results, analysis, and discussion in the \textbf{\emph{appendix}}. Please kindly refer to our \textbf{\emph{appendix}} for details.
\section{Conclusion}
In this paper, we present LM-EEC, a novel framework that builds upon the strong generalization capabilities of SAM 2 to effectively tackle challenges such as severe object occlusions and the presence of numerous small objects. To better exploit multi-view information, LM-EEC introduces a Memory-View Mixture-of-Experts module that dynamically integrates egocentric and exocentric features. In addition, we enhance the memory management mechanism of SAM 2 by constructing a dual long-term memory bank using a view-specific compression strategy, which preserves discriminative information while reducing redundancy. Extensive experiments on the EgoExo4D benchmark show that LM-EEC consistently outperforms prior approaches, especially in complex and cluttered scenes.
\vspace{0.5em}
\textbf{Acknowledgment.} Libo Zhang was supported by National Natural Science Foundation of China (No. 62476266). Heng Fan was not supported by any fund for this work.
% \bibliographystyle{plainnat}
\bibliographystyle{unsrt}
\bibliography{main}
\appendix
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
% \section*{\centering \hypertarget{appendix_anchor}{Appendix}}
\begin{center}
{\LARGE \textbf{\hypertarget{appendix_anchor}{Appendix}}}
\end{center}
The appendix is structured as follows:
\begin{itemize}
\item \textbf{Section~\ref{sec:B}}: We first analyze model performance across diverse activity scenarios to evaluate its generalization capability.
\item \textbf{Section~\ref{sec:C}}: We then provide qualitative results, comparing the proposed LM-EEC to baselines.
\item \textbf{Section~\ref{sec:D}}: We discuss the limitation of our model and reflect on its broader impact.
\item \textbf{Section~\ref{sec:E}}: We also visualize attention maps to demonstrate how our model establishes precise object-level correspondence between egocentric and exocentric views, enabling cross-view alignment.
\item \textbf{Section~\ref{sec:F}}: We validate long-term ego-exo correspondence using association accuracy over videos of varying lengths.
\item \textbf{Section~\ref{sec:G}}: We demonstrate the modularity of MV-MoE by integrating it into alternative backbones, showing the architectural transferability.
\item \textbf{Section~\ref{sec:H}}: We also provide basic training configurations for all compared methods.
\end{itemize}
\input{appendix}
\end{document}
\begin{figure*}[ht]
\centering
% 第一行
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=\linewidth]{images/IoU_scenario.png}
\label{fig:short-a}
\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=\linewidth]{images/Contour_Acc_scenario.png}
\label{fig:short-b}
\end{subfigure}
% 添加行间距
\vskip 0.5em
% 第二行
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=\linewidth]{images/Location_Score_scenario.png}
\label{fig:short-c}
\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=\linewidth]{images/BalAcc_scenario.png}
\label{fig:short-d}
\end{subfigure}
\caption{Comparison across different activity scenarios for each model.}
\label{fig:scenario_comparison}
\end{figure*}
\section{Analysis on Different Scenarios}
\label{sec:B}
Beyond object size, we further analyze model performance across different activity scenarios in EgoExo4D, which consists of Cooking, Basketball, Bike Repair, Health, and Music. We compare the results of various models across these scenarios to assess their robustness and adaptability.
As shown in Figure~\ref{fig:scenario_comparison}, some activities, such as Basketball, are generally easier to model due to limited variation in object shape and appearance. In contrast, activities like Cooking and Bike Repair pose greater challenges, as objects exhibit more diverse appearances and shapes across different views.
Notably, in these more challenging scenarios, our model demonstrates a clear advantage. The performance gap between our method and other baselines is more pronounced in these cases, highlighting the effectiveness of our approach in handling complex and highly variable environments.
\begin{figure*}[!t]
\centering
\begin{subfigure}{0.98\linewidth} % 改为 \linewidth
\centering
\includegraphics[width=\linewidth]{images/appendix_egoexo_vis.pdf}
\label{fig:short-a1}
\end{subfigure}
\vspace{-3mm}
\caption{Ego to Exo results of different approaches.}
\label{fig:egoexo_vis}
\vspace{-3mm}
\end{figure*}
% \vspace{-1.8em}
\section{Qualitative Results Comparison}
\label{sec:C}
We further compare the qualitative results of our model with other baselines. As shown in Figure~\ref{fig:egoexo_vis} and ~\ref{fig:exoego_vis}, while some models struggle to fully segment the object (e.g., the stainless steel spatula) or to distinguish it from similar objects in the background, our model accurately segments the object and effectively handles occlusion and variations in its appearance.
\begin{figure*}[!t]
\centering
\begin{subfigure}{\linewidth} % 改为 \linewidth
\centering
\includegraphics[width=\linewidth]{images/appendix_exoego_vis.pdf}
\label{fig:short-a2}
\end{subfigure}
\caption{Exo to Ego results of different approaches.}
\label{fig:exoego_vis}
\vspace{-6mm}
\end{figure*}
\section{Limitation and Broader Impact}
\label{sec:D}
\noindent\textbf{Discussion of Limitation.} As observed, our model does not achieve the highest performance in terms of balanced accuracy. This indicates that when the target object disappears from the scene, the model may mistakenly segment visually similar background objects. Representative cases of such failures are illustrated in Figure~\ref{fig:failure_case}.
This limitation primarily stems from the fact that existing baselines do not explicitly consider object disappearance during training. As a result, the model tends to associate and segment visually similar objects in the current frame, even when the target object is no longer present. This observation reveals an inherent trade-off among IoU, contour accuracy, location score, and balanced accuracy.
In this work, we build upon SAM~2 as the baseline architecture to develop our model. While SAM~2 demonstrates strong matching capability and performs effectively in video segmentation, it lacks explicit mechanisms to handle object disappearance. Consequently, our model inevitably inherits this limitation to some extent. We believe that incorporating targeted data augmentation during training or introducing an additional branch to explicitly predict object presence or absence could be promising directions to address this issue in future work.
\noindent{\textbf{Discussion of Broader Impact.}} The proposed LM-EEC which focuses on establishing object-level correspondence between egocentric and exocentric views has the potential to benefit a wide range of real-world applications, including intelligent robotics, augmented reality (AR), and assistive technologies. By enhancing the ability of AI systems to reason about objects from multiple viewpoints, LM-EEC could contribute to more accurate and intuitive human-AI interaction, such as guiding users through complex tasks or supporting remote collaboration.
It's worth noting that our work carries potential risks, particularly concerning privacy and surveillance, as the ability to associate first-person and third-person views could be misused for unauthorized tracking or monitoring of individuals beyond intended applications. Therefore, we emphasize that our work must be strictly applied within ethical and privacy-compliant contexts.
\begin{figure*}[!t]
\centering
\begin{subfigure}{\linewidth} % 改为 \linewidth
\centering
\includegraphics[width=0.8\linewidth]{images/failure_case.pdf}
\label{fig:short-a3}
\end{subfigure}
\caption{Failure cases.}
\label{fig:failure_case}
\vspace{-6mm}
\end{figure*}
\begin{figure}[!t]
\centering
% \begin{subfigure}{\linewidth} % 改为 \linewidth
% \centering
\includegraphics[width=0.75\linewidth]{images/attention_vis.pdf}
\label{fig:short-a4}
% \end{subfigure}
\caption{Attention map visualization.}
\label{fig:attention map}
\vspace{-3mm}
\end{figure}
\section{Attention Map Visualization}
\label{sec:E}
In addition, we visualize the attention map of both the current frame and the corresponding annotated frame. Specifically, we display the attention map related to the current target to be segmented from an alternate view. As shown in Figure~\ref{fig:attention map}, our model accurately matches objects from both ego and exo views (e.g., the guitar).
\section{Long-term Correspondence Evidence}
\label{sec:F}
To verify the effectiveness of our method in modeling long-term ego-exo correspondence, we conduct an experiment to evaluate the \textbf{association accuracy} across videos of different lengths. Specifically, we compare our model with the baseline (i.e., SAM~2) that excludes the core MV-MoE and memory compression modules.
In this experiment, the \textbf{association accuracy} is defined as the ratio of frames where the Intersection over Union (IoU) between the predicted and ground truth masks exceeds a threshold of 0.5. The comparison results are summarized in Table~\ref{tab:association_accuracy}.
As shown in Table~\ref{tab:association_accuracy}, our model consistently outperforms the baseline across all video lengths. Notably, for longer videos with more than 600 frames, our method achieves a significant improvement of approximately \textbf{14\%} in association accuracy compared with the baseline model. This result effectively validates the capability of our approach to maintain robust \textbf{long-term ego-exo correspondence} over extended temporal durations.
\begin{table}[!t]
\centering
\renewcommand{\arraystretch}{1.1} % 默认是1.0，1.2~1.4效果不错
\caption{Association accuracy of our method and the baseline under different video lengths.}
\begin{tabular}{lccc}
\toprule
Video Length & $<$200 frames & 200--600 frames & $>$600 frames \\
\midrule
Baseline Model & 0.5580 & 0.6260 & 0.4724 \\
\rowcolor{cyan!10}
Our Model & \textbf{0.6054} & \textbf{0.6732} & \textbf{0.6150} \\
\bottomrule
\end{tabular}
\label{tab:association_accuracy}
\end{table}
\section{Modularity of MV-MoE}
\label{sec:G}
While the proposed MV-MoE is initially designed on top of the SAM 2 architecture, its core component, i.e., the dual-branch routing mechanism that adaptively assigns contribution weights to each expert feature along both channel and spatial dimensions, is conceptually modular and can be readily integrated into other architectures that involve multi-source feature fusion. To demonstrate the generality of MV-MoE, we conduct additional experiments by integrating it into two alternative frameworks with different segmentation bones, including STCN and QDMN, for the Ego-Exo Correspondence task. The results on the Ego2Exo test split of the EgoExo4D dataset are shown in Table~\ref{tab:MoE1}.
From Tab. B, we can clearly observe that, incorporating MV-MoE consistently enhances performance across metrics, indicating that the module is not only effective but also transferable to different backbone architectures. This suggests that MV-MoE can be easily adapted to other segmentation architectures/backbones and future versions of SAM 2-like frameworks.
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.1} % 默认是1.0，1.2~1.4效果不错
\caption{Application of MV-MoE on other segmentation backbones.}
\begin{tabular}{l c c c c}
\toprule
\textbf{Model} & \textbf{IoU} $\uparrow$ & \textbf{LE} $\downarrow$ & \textbf{CA} $\uparrow$ & \textbf{BA} $\uparrow$ \\
\midrule
STCN & 27.39 & 0.109 & 0.378 & 61.97 \\
\rowcolor{cyan!10}
STCN + MV-MoE & \textbf{30.51} & \textbf{0.083} & \textbf{0.436} & \textbf{62.81} \\
% \hdashline
\midrule
QDMN & 27.03 & 0.108 & 0.346 & 63.16 \\
\rowcolor{cyan!10}
QDMN + MV-MoE & \textbf{28.75} & \textbf{0.100} & \textbf{0.376} & \textbf{64.02} \\
\bottomrule
\end{tabular}
\label{tab:MoE1}
\end{table}
\section{Basic Training Configurations}
\label{sec:H}
All baselines were implemented from official code with minimal changes for EgoExo4D, following original training settings. Table~\ref{tab:training_config} summarizes the training configurations of our method and other approaches. Notably, all the compared method are trained for more epochs/iterations than our proposed model to ensure sufficient optimization. Our goal is to ensure fair and optimal performance for all methods on EgoExo4D.
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2} % 默认是1.0，1.2~1.4效果不错
\caption{Training configurations of compared methods.}
\begin{tabular}{l c l}
\toprule
\textbf{Model} & \textbf{Epochs (Iterations)} & \textbf{Optimizer} \\
\midrule
XSegTx & 250 (250K iter.) & Adam \\
XView-Xmem (w/ finetuning) & 358 (100K iter.) & AdamW \\
XView-Xmem (+ XSegTx) & 358 (100K iter.) & AdamW \\
STCN & 258 (50K iter.) & Adam \\
QDMN & 207 (80K iter.) & Adam \\
GSFM & 213 (55K iter.) & Adam \\
SimVOS & 193 (300K iter.) & Adam \\
Cutie & 167 (125K iter.) & AdamW \\
\rowcolor{cyan!10}
LM-EEC(Ours) & 60 (12K iter.) & AdamW \\
\bottomrule
\end{tabular}
\label{tab:training_config}
\end{table}
\begin{thebibliography}{10}
\bibitem{EgoExo4D}
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zachary Chavis, Joya Chen, Feng Cheng, Fu{-}Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Mar{\'{\i}}a Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md~Mohaiminul Islam, Suyog~Dutt Jain, Rawal Khirodkar, Devansh Kukreja, Kevin~J. Liang, Jia{-}Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh~Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonz{\'{a}}lez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi~Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova,
Fiona Ryan, Wei Shan, Kiran~K. Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin~Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbel{\'{a}}ez, Gedas Bertasius, Dima Damen, Jakob~J. Engel, Giovanni~Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C.~V. Jawahar, Richard~A. Newcombe, Hyun~Soo Park, James~M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike~Zheng Shout, and Michael Wray.
\newblock Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 19383--19400, 2024.
\bibitem{XMem}
Ho~Kei Cheng and Alexander~G Schwing.
\newblock Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model.
\newblock In {\em European conference on computer vision}, pages 640--658. Springer, 2022.
\bibitem{SAM2}
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R{\"a}dle, Chloe Rolland, Laura Gustafson, et~al.
\newblock Sam 2: Segment anything in images and videos.
\newblock {\em arXiv preprint arXiv:2408.00714}, 2024.
\bibitem{sam2unet}
Xinyu Xiong, Zihuang Wu, Shuangyi Tan, Wenxue Li, Feilong Tang, Ying Chen, Siying Li, Jie Ma, and Guanbin Li.
\newblock Sam2-unet: Segment anything 2 makes strong encoder for natural and medical image seg- mentation.
\newblock {\em arXiv preprint arXiv:2408.08870}, 2024.
\bibitem{samurai}
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc~Van Gool.
\newblock The 2017 davis challenge on video object segmentation.
\newblock {\em arXiv preprint arXiv:2411.11922}, 2017.
\bibitem{sam2long}
Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang.
\newblock Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree.
\newblock {\em arXiv preprint arXiv:2410.16268}, 2024.
\bibitem{medicalsam2}
Jiayuan Zhu, Abdullah Hamdi, Yunli Qi, Yueming Jin, and Junde Wu.
\newblock Medical sam 2: Segment medical images as video via segment anything model 2.
\newblock {\em arXiv preprint arXiv:2408.00874v2}, 2024.
\bibitem{davis}
Cheng-Yen Yang, Hsiang-Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang.
\newblock Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory.
\newblock {\em arXiv preprint arXiv:1704.00675}, 2024.
\bibitem{online1}
Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taix{\'e}, Daniel Cremers, and Luc Van~Gool.
\newblock One-shot video object segmentation.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 221--230, 2017.
\bibitem{online2}
Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine-Hornung.
\newblock Learning video object segmentation from static images.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 2663--2672, 2017.
\bibitem{online3}
Jae Shin~Yoon, Francois Rameau, Junsik Kim, Seokju Lee, Seunghak Shin, and In~So~Kweon.
\newblock Pixel-level matching for video object segmentation using convolutional neural networks.
\newblock In {\em Proceedings of the IEEE international conference on computer vision}, pages 2167--2176, 2017.
\bibitem{online4}
K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taix{\'e}, Daniel Cremers, and Luc Van~Gool.
\newblock Video object segmentation without temporal information.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence}, 41(6):1515--1530, 2018.
\bibitem{online5}
Yuan-Ting Hu, Jia-Bin Huang, and Alexander Schwing.
\newblock Maskrnn: Instance level video object segmentation.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{online6}
Goutam Bhat, Felix~J{\"a}remo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg, Luc Van~Gool, and Radu Timofte.
\newblock Learning what to learn for video object segmentation.
\newblock In {\em European Conference on Computer Vision}, pages 777--794. Springer, 2020.
\bibitem{online7}
Andreas Robinson, Felix~Jaremo Lawin, Martin Danelljan, Fahad~Shahbaz Khan, and Michael Felsberg.
\newblock Learning fast and robust target models for video object segmentation.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 7406--7415, 2020.
\bibitem{online8}
Paul Voigtlaender and Bastian Leibe.
\newblock Online adaptation of convolutional neural networks for video object segmentation.
\newblock {\em arXiv preprint arXiv:1706.09364}, 2017.
\bibitem{first_frame1}
Yuan-Ting Hu, Jia-Bin Huang, and Alexander~G Schwing.
\newblock Videomatch: Matching based video object segmentation.
\newblock In {\em Proceedings of the European conference on computer vision (ECCV)}, pages 54--70, 2018.
\bibitem{first_frame2}
Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van~Gool.
\newblock Blazingly fast video object segmentation with pixel-wise metric learning.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 1189--1198, 2018.
\bibitem{previous_frame1}
Seoung~Wug Oh, Joon-Young Lee, Kalyan Sunkavalli, and Seon~Joo Kim.
\newblock Fast video object segmentation by reference-guided mask propagation.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 7376--7385, 2018.
\bibitem{previous_frame2}
Zongxin Yang, Yunchao Wei, and Yi~Yang.
\newblock Collaborative video object segmentation by foreground-background integration.
\newblock In {\em European Conference on Computer Vision}, pages 332--348. Springer, 2020.
\bibitem{rnn}
Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang.
\newblock Youtube-vos: Sequence-to-sequence video object segmentation.
\newblock In {\em Proceedings of the European conference on computer vision (ECCV)}, pages 585--601, 2018.
\bibitem{transformer1}
Seoung~Wug Oh, Joon-Young Lee, Ning Xu, and Seon~Joo Kim.
\newblock Video object segmentation using space-time memory networks.
\newblock In {\em Proceedings of the IEEE/CVF international conference on computer vision}, pages 9226--9235, 2019.
\bibitem{transformer2}
Ho~Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang.
\newblock Rethinking space-time networks with improved memory coverage for efficient video object segmentation.
\newblock {\em Advances in neural information processing systems}, 34:11781--11794, 2021.
\bibitem{transformer3}
Mingxing Li, Li~Hu, Zhiwei Xiong, Bang Zhang, Pan Pan, and Dong Liu.
\newblock Recurrent dynamic embedding for video object segmentation.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 1332--1341, 2022.
\bibitem{transformer4}
Zongxin Yang, Yunchao Wei, and Yi~Yang.
\newblock Associating objects with transformers for video object segmentation.
\newblock {\em Advances in Neural Information Processing Systems}, 34:2491--2502, 2021.
\bibitem{transformer5}
Zongxin Yang and Yi~Yang.
\newblock Decoupling features in hierarchical propagation for video object segmentation.
\newblock {\em Advances in Neural Information Processing Systems}, 35:36324--36336, 2022.
\bibitem{transformer6}
Qiangqiang Wu, Tianyu Yang, Wei Wu, and Antoni~B Chan.
\newblock Scalable video object segmentation with simplified framework.
\newblock In {\em Proceedings of the IEEE/CVF international conference on computer vision}, pages 13879--13889, 2023.
\bibitem{joint_train1}
Jeffrey Donahue, Lisa Anne~Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell.
\newblock Long-term recurrent convolutional networks for visual recognition and description.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 2625--2634, 2015.
\bibitem{joint_train2}
Joe Yue-Hei~Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici.
\newblock Beyond short snippets: Deep networks for video classification.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 4694--4702, 2015.
\bibitem{joint_train3}
Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell.
\newblock Actionvlad: Learning spatio-temporal aggregation for action classification.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 971--980, 2017.
\bibitem{joint_train4}
Noureldien Hussein, Efstratios Gavves, and Arnold~WM Smeulders.
\newblock Timeception for complex action recognition.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 254--263, 2019.
\bibitem{joint_train5}
Chao-Yuan Wu and Philipp Krahenbuhl.
\newblock Towards long-form video understanding.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 1884--1894, 2021.
\bibitem{sampling1}
Bruno Korbar, Du~Tran, and Lorenzo Torresani.
\newblock Scsampler: Sampling salient clips from video for efficient action recognition.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 6232--6242, 2019.
\bibitem{sampling2}
Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry~S Davis.
\newblock Adaframe: Adaptive frame selection for fast video recognition.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 1278--1287, 2019.
\bibitem{RMem}
Junbao Zhou, Ziqi Pang, and Yu-Xiong Wang.
\newblock Rmem: Restricted memory banks improve video object segmentation.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 18602--18611, 2024.
\bibitem{SAM}
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander~C Berg, Wan-Yen Lo, et~al.
\newblock Segment anything.
\newblock In {\em Proceedings of the IEEE/CVF international conference on computer vision}, pages 4015--4026, 2023.
\bibitem{stcn}
Ho~Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang.
\newblock Rethinking space-time networks with improved memory coverage for efficient video object segmentation.
\newblock {\em Advances in neural information processing systems}, 34:11781--11794, 2021.
\bibitem{qdmn}
Yong Liu, Ran Yu, Fei Yin, Xinyuan Zhao, Wei Zhao, Weihao Xia, and Yujiu Yang.
\newblock Learning quality-aware dynamic memory for video object segmentation.
\newblock In {\em European Conference on Computer Vision}, pages 468--486. Springer, 2022.
\bibitem{gsfm}
Yong Liu, Ran Yu, Jiahao Wang, Xinyuan Zhao, Yitong Wang, Yansong Tang, and Yujiu Yang.
\newblock Global spectral filter memory network for video object segmentation.
\newblock In {\em European Conference on Computer Vision}, pages 648--665. Springer, 2022.
\bibitem{simvos}
Qiangqiang Wu, Tianyu Yang, Wei Wu, and Antoni~B Chan.
\newblock Scalable video object segmentation with simplified framework.
\newblock In {\em Proceedings of the IEEE/CVF international conference on computer vision}, pages 13879--13889, 2023.
\bibitem{cutie}
Ho~Kei Cheng, Seoung~Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing.
\newblock Putting the object back into video object segmentation.
\newblock In {\em CVPR}, 2024.
\end{thebibliography}