The Solution for Single Object Tracking Task of Perception Test Challenge 2024
Extracted source text · arXiv 2410.16329
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage{iccv}
\usepackage{authblk}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{enumitem}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{cite}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage[autostyle, english=american]{csquotes}
\MakeOuterQuote{"}
% Include other packages here, before hyperref.
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[breaklinks=true,bookmarks=false]{hyperref}
\iccvfinalcopy % *** Uncomment this line for the final submission
\def\iccvPaperID{****} % *** Enter the ICCV Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
% Pages are numbered in submission mode, and unnumbered in camera-ready
\ificcvfinal\pagestyle{empty}\fi
\begin{document}
%%%%%%%%% TITLE
\title{The Solution for Single Object Tracking Task of Perception Test Challenge 2024 }
%\author{Dian Chao\\
%Nanjing University of Science and Technology\\
%{\tt\small firstauthor@i1.org}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
%\and
%%Shupeng Zhong\\
%Nanjing University of Science and Technology\\
%{\tt\small secondauthor@i2.org}
%}
\author{
Zhiqiang Zhong$^1$,
Yang Yang\textsuperscript{1, $\thanks{Corresponding Author}$},
Fengqiang Wan$^1$,
Henglu Wei$^2$,
Xiangyang Ji$^2$,
}
\affil{
$^1$Nanjing University of Science and Technology\\
$^2$Tsinghua University\\
}
\maketitle
% Remove page # from the first page of camera-ready.
\ificcvfinal\thispagestyle{empty}\fi
\begin{abstract}
\quad This report presents our method for Single Object Tracking (SOT), which aims to track a specified object throughout a video sequence. We employ the LoRAT method. The essence of the work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. We train our model using the extensive LaSOT and GOT-10k datasets, which provide a solid foundation for robust performance. Additionally, we implement the alpha-refine technique for post-processing the bounding box outputs. Although the alpha-refine method does not yield the anticipated results, our overall approach achieves a score of 0.813, securing first place in the competition.
\end{abstract}
%%
%% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
%% Please copy and paste the code instead of the example below.
%%
% \begin{CCSXML}
% <ccs2012>
% <concept>
% <concept_id>00000000.0000000.0000000</concept_id>
% <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
% <concept_significance>500</concept_significance>
% </concept>
% <concept>
% <concept_id>00000000.00000000.00000000</concept_id>
% <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
% <concept_significance>300</concept_significance>
% </concept>
% <concept>
% <concept_id>00000000.00000000.00000000</concept_id>
% <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
% <concept_significance>100</concept_significance>
% </concept>
% <concept>
% <concept_id>00000000.00000000.00000000</concept_id>
% <concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
% <concept_significance>100</concept_significance>
% </concept>
% </ccs2012>
% \end{CCSXML}
% \ccsdesc[500]{Do Not Use This Code~Generate the Correct Terms for Your Paper}
% \ccsdesc[300]{Do Not Use This Code~Generate the Correct Terms for Your Paper}
% \ccsdesc{Do Not Use This Code~Generate the Correct Terms for Your Paper}
% \ccsdesc[100]{Do Not Use This Code~Generate the Correct Terms for Your Paper}
%%
%% Keywords. The author(s) should pick words that accurately describe
%% the work being presented. Separate the keywords with commas.
% \keywords{Do, Not, Us, This, Code, Put, the, Correct, Terms, for,
% Your, Paper}
% \received{20 February 2007}
% \received[revised]{12 March 2009}
% \received[accepted]{5 June 2009}
%%
%% This command processes the author and affiliation and title
%% information and builds the first part of the formatted document.
\maketitle
\section{Introduction.}
\textbf{Team Introduction.} We have established ourselves as a hub for innovative studies in machine learning, data mining, and pattern recognition~\cite{DBLP:conf/mm/0074ZGGZ22,DBLP:journals/titb/ZhangYYGLZYR22,DBLP:conf/aaai/YangHGXX23,DBLP:journals/corr/abs-2401-07551,DBLP:journals/corr/abs-2404-08347}. Our primary research interests encompass foundational, cutting-edge, and transformative approaches aimed at advancing these disciplines.
We focus on several key areas, including multimodal machine learning, open-environment machine learning, and incremental learning. Additionally, we are dedicated to practical applications, exploring real-world scenarios such as image retrieval, video segmentation, and novel class detection~\cite{DBLP:conf/mm/0074ZGGZ22,DBLP:journals/tkde/YangYBZZGXY23,DBLP:conf/kdd/YangZZX019,DBLP:journals/titb/ZhangYYGLZYR22}. Our commitment to both theoretical advancements and their implementation in diverse contexts positions us at the forefront of knowledge-mining research, fostering an environment where collaboration and innovation can thrive~\cite{DBLP:journals/fcsc/YangGLLLY24,DBLP:conf/ijcai/YangZXYZY21,DBLP:conf/pakdd/Zhou0Z21,DBLP:conf/aaai/YangHGXX23}.
\textbf{Competition Introduction.} Benchmarking models is crucial for selecting the most optimal model and determining the most suitable development path for tracking tasks. While existing benchmarks have driven significant advancements in computational performance, many primarily focus on computational aspects rather than perceptual capabilities.
To address this gap, DeepMind has introduced the Perception Test, a novel multimodal video benchmark designed to evaluate the capabilities of multimodal perception models more comprehensively. Unlike traditional benchmarks, the Perception Test emphasizes perceptual tasks by leveraging real-world video data that are purposefully designed, filmed, and annotated. This benchmark aims to assess a wide range of skills, types of reasoning, and modalities in multimodal perception models~\cite{patraucean2024perception}. The Perception Test features an extensive dataset comprising 11,600 real-world videos, each with an average length of 23 seconds. These videos capture perceptually interesting scenarios and are filmed by approximately 100 participants from around the globe. The dataset is densely annotated with six types of labels, including multiple-choice questions, grounded video question-answers, object and point tracks, temporal action segments, and sound segments. This rich annotation allows for a thorough evaluation of both language-based and non-language-based models, providing a more holistic view of model performance across various perceptual tasks.
In the Perception Test Challenge 2024, our team selects single-object tracking. In this task, the model receives a video and a bounding box representing an object, and it tracks the object throughout the video sequence. Figure \ref{fig1} shows examples of object tracking annotations. We evaluate several prominent SOT models, including SiamFC~\cite{bertinetto2016fully}, SAM2~\cite{ravi2024sam}, and LoRAT~\cite{lin2024tracking}. After a thorough assessment, we choose LoRAT as the base model for our solution due to its superior performance and alignment with our tracking objectives. To enhance the model's capabilities, we incorporate additional strategies during the training phase.
\begin{figure}[h]
\centering
\includegraphics[width=0.47\textwidth]{images/1.png}
\caption{the examples of object tracking annotations.}
\label{fig1}
\end{figure}
We utilize supplementary datasets such as LaSOT~\cite{fan2019lasot} and GOT-10k~\cite{huang2019got} to broaden the model's exposure and improve its generalization across different scenarios. This expanded training dataset significantly contributes to the model's robustness and adaptability. Additionally, we implement the Alpha-Refine method to post-process the bounding box output generated by the model. Although Alpha-Refine aims to improve tracking performance through accurate bounding box estimation, its performance is not good enough.
Despite the limited effect of the Alpha-Refine method, our approach, combining diverse training data and advanced tracking techniques, leads to significant performance improvements. These efforts ultimately result in our team achieving first place in the test phase of this track.
\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth]{images/3.jpg}
\caption{Architecture of LoRAT. }
\label{fig2}
\end{figure*}
\section{Method}
This section provides an overview of the LoRAT approach~\cite{DBLP:journals/corr/abs-2403-05231}, highlighting its architectural components. Key elements include adjustments to the model’s input embedding for enhanced adaptability and integrating an MLP-only head network. These modifications are crucial for optimizing performance while minimizing computational overhead. Figure \ref{fig2} illustrates an overview of the tracker.
\subsection{LoRA-friendly Model Design}
\textbf{Decoupled Input Embedding.} ViT models require positional encodings to inject positional information~\cite{vaswani2017attention,yang2023towards}. Existing one-stream trackers utilize separate positional embeddings for template and search region tokens, which disrupts the structure of the pre-trained ViT model and leads to ineffectiveness in parameter-efficient fine-tuning (PEFT). To address this issue, adaptations for multi-resolution positional embeddings are explored, along with the integration of token-type embeddings.
\textit{Token type embedding.} Token type embedding, also known as segment embedding, is incorporated into the tracker by assigning unique embeddings to template and search region tokens. This approach decouples token type identification from positional embeddings, thereby enhancing parameter efficiency and flexibility.
The use of token-type embedding is further extended to address specific challenges associated with the cropping strategy commonly employed in Siamese trackers. This cropping strategy, used to generate tracking templates, presents several issues that may hinder the tracker's performance: the aspect ratio of target objects in the templates can vary, some target objects may lack clear semantic meaning, and certain target objects may be indistinguishable from the background. Token-type embedding helps mitigate these issues by explicitly annotating foreground and background regions within the template.
\textit{Positional embedding.} Positional embeddings~\cite{dosovitskiy2020image,huo2018learning} typically function for fixed resolutions. However, one-stream trackers often use image pairs of different sizes as inputs. According to the parameter-efficient fine-tuning (PEFT) design principle, sharing the positional embedding across these two images is crucial~\cite{yang2022exploiting}. Strategies for effectively reusing standard 1D absolute positional embeddings for varying input sizes are explored. Both strategies consider the original 1D absolute embedding within a 2D context. Let \(Q\) denote the original 1D absolute positional embedding; the 2D representation of \(Q\) can be expressed as \(Q_{2d} = [q_{1,1},...,q_{1,w}; q_{2,1},...,q_{2,w}; q_{h,1},...,q_{h,w}]\). For convenience, it is assumed that the resolution of the search region image matches the native resolution of the ViT model, allowing \(Q_{2d}\) to be directly applied to search region tokens, i.e., the search region positional embedding \(Q_x = Q_{2d}\).
\textbf{MLP-only Head Network.} To mitigate potential inductive biases inherent in convolutional heads, a multi-layer perceptron (MLP) only head is employed. The head network generates a redundant set of bounding boxes, each accompanied by a classification score. These classification scores rank the bounding boxes, with the highest score determining the final output. The head network is structured into two branches: one for classification and another for bounding box regression, each consisting of a three-layer MLP. Both branches process the feature map of the search area \(\mathcal{S}\) outputted from the Transformer encoder. The head network adopts a center-based, anchor-free style ~\cite{tian2019fcos} to accelerate training coverage and enhance tracking performance.
\section{Experiments}
\textbf{Dataset.} We use a fine-tuned evaluation method. The experimental evaluation utilizes three datasets: Perception Test, LaSOT~\cite{fan2019lasot}, and GOT-10k~\cite{huang2019got}. The Perception Test dataset is divided into training, validation, and test subsets, with the training and validation sets used to fine-tune the model, and the test set reserved for performance evaluation. See details about the Perception Test dataset in Table \ref{tab:1}. Additionally, the LaSOT and GOT-10k datasets are exclusively employed as training data.
\begin{table}[h]
\caption{Perception Test dataset}
\label{tab:1}
\centering
\begin{tabular}{ccc}
\toprule
split&videos&object tracks\\
\midrule
Train & 2184 & 35373\\
Validation & 1000 & 16501\\
Test & 1000 & 16339\\
\bottomrule
\end{tabular}
\end{table}
\textbf{Metric.} The evaluation metric is the average Intersection over Union (IoU). It is calculated as the average IoU between the predicted bounding boxes and the ground truth bounding boxes for each tracked object.
\textbf{Comparison Methods Result.} To select the most suitable model for the foundational component of the solution, comparative experiments are conducted on the test set. Table \ref{tab:2} illustrates the evaluation of four model approaches: Dummy Static, SiamFC, SAM2, and LoRAT.
\begin{table}[h]
\caption{Experimental results of different methods}
\label{tab:2}
\centering
\begin{tabular}{cc}
\toprule
Method&average IoU \\
\midrule
Dummy static & 0.640\\
SiamFC & 0.658\\
SAM2 & 0.693\\
LoRAT & 0.802\\
\bottomrule
\end{tabular}
\end{table}
Through experiments on the test set, we can see that the LoRAT method is the best on the indicator of IoU. According to this result, we select the LoRAT model as the basic model for our solution. Subsequent experiments are also based on this model.
\textbf{Ablation Study.} To analyze the Contribution of each component to LoRAT, we conduct more ablation studies on the test set of the competition. The average Intersection
over Union after adding components is demonstrated in Table \ref{tab:3}. We implement the Alpha-Refine method to post-process the bounding box output generated by the model. Although Alpha-Refine aims to improve tracking performance through accurate bounding box estimation, its performance is not good enough.
\begin{table}[h]
\caption{Ablation experiment. The B in B-224 stands for ViT-B, and 224 means the search region size is set to [224 × 224]. The L in L-378 stands for ViT-L, and 378 means the search region size is set to [378 × 378].}
\label{tab:3}
\centering
\begin{tabular}{cc}
\toprule
Method&average IoU\\
\midrule
LoRAT-B-224 & 0.802\\
LoRAT-L-378 & 0.810 \\
LoRAT-L-378+Alpha-Refine & 0.785\\
LoRAT-L-378+LaSOT+GOT-10k & 0.813\\
\bottomrule
\end{tabular}
\end{table}
\section{Conclusion}
This report presents a comprehensive evaluation of single-object tracking methods, highlighting the selection and application of the LoRAT model as the core tracking algorithm. By leveraging diverse datasets, including Perception Test, LaSOT, and GOT-10k, and employing advanced techniques such as interpolation-based positional embedding adaptation, the study demonstrates significant improvements in tracking performance. Despite the limited impact of the Alpha-Refine method, the overall approach effectively addresses the challenges of single-object tracking, leading to a top ranking in the final test phase.
{\small
\bibliographystyle{unsrt}
\bibliography{main}
}
\end{document}
\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
\caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}
\begin{thebibliography}{10}
\bibitem{DBLP:conf/mm/0074ZGGZ22}
Yang Yang, Jingshuai Zhang, Fan Gao, Xiaoru Gao, and Hengshu Zhu.
\newblock {DOMFN:} {A} divergence-orientated multi-modal fusion network for resume assessment.
\newblock In {\em {MM} '22: The 30th {ACM} International Conference on Multimedia}, pages 1612--1620, Lisboa,Portugal, 2022.
\bibitem{DBLP:journals/titb/ZhangYYGLZYR22}
Xiao Zhang, Hongzheng Yu, Yang Yang, Jingjing Gu, Yujun Li, Fuzhen Zhuang, Dongxiao Yu, and Zhaochun Ren.
\newblock Harmi: Human activity recognition via multi-modality incremental learning.
\newblock {\em {IEEE} J. Biomed. Health Informatics}, 2022.
\bibitem{DBLP:conf/aaai/YangHGXX23}
Yang Yang, Yurui Huang, Weili Guo, Baohua Xu, and Dingyin Xia.
\newblock Towards global video scene segmentation with context-aware transformer.
\newblock In {\em AAAI}, pages 3206--3213, Washington, DC, USA, 2023.
\bibitem{DBLP:journals/corr/abs-2401-07551}
Wenjuan Xi, Xin Song, Weili Guo, and Yang Yang.
\newblock Robust semi-supervised learning for self-learning open-world classes.
\newblock {\em CoRR}, 2024.
\bibitem{DBLP:journals/corr/abs-2404-08347}
Yang Yang, Hongpeng Pan, Qing{-}Yuan Jiang, Yi~Xu, and Jinghui Tang.
\newblock Learning to rebalance multi-modal optimization by adaptively masking subnetworks.
\newblock {\em CoRR}, 2024.
\bibitem{DBLP:journals/tkde/YangYBZZGXY23}
Yang Yang, Jia{-}Qi Yang, Ran Bao, De{-}Chuan Zhan, Hengshu Zhu, Xiaoru Gao, Hui Xiong, and Jian Yang.
\newblock Corporate relative valuation using heterogeneous multi-modal graph neural network.
\newblock {\em {IEEE} Trans. Knowl. Data Eng.}, 2023.
\bibitem{DBLP:conf/kdd/YangZZX019}
Yang Yang, Da{-}Wei Zhou, De{-}Chuan Zhan, Hui Xiong, and Yuan Jiang.
\newblock Adaptive deep models for incremental learning: Considering capacity scalability and sustainability.
\newblock In {\em KDD}, pages 74--82, Anchorage, AK, USA, 2019.
\bibitem{DBLP:journals/fcsc/YangGLLLY24}
Yang Yang, Jinyi Guo, Guangyu Li, Lanyu Li, Wenjie Li, and Jian Yang.
\newblock Alignment efficient image-sentence retrieval considering transferable cross-modal representation learning.
\newblock {\em Frontiers Comput. Sci.}, 2024.
\bibitem{DBLP:conf/ijcai/YangZXYZY21}
Yang Yang, Chubing Zhang, Yi{-}Chu Xu, Dianhai Yu, De{-}Chuan Zhan, and Jian Yang.
\newblock Rethinking label-wise cross-modal retrieval from {A} semantic sharing perspective.
\newblock In {\em IJCAI}, pages 3300--3306, Canada, 2021.
\bibitem{DBLP:conf/pakdd/Zhou0Z21}
Da{-}Wei Zhou, Yang Yang, and De{-}Chuan Zhan.
\newblock Detecting sequentially novel classes with stable generalization ability.
\newblock In {\em PAKDD}, pages 371--382, 2021.
\bibitem{patraucean2024perception}
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi~Yang, Carl Doersch, et~al.
\newblock Perception test: A diagnostic benchmark for multimodal video models.
\newblock {\em NeurIPS}, 2024.
\bibitem{bertinetto2016fully}
Luca Bertinetto, Jack Valmadre, Joao~F Henriques, Andrea Vedaldi, and Philip~HS Torr.
\newblock Fully-convolutional siamese networks for object tracking.
\newblock In {\em ECCV}, pages 850--865, Amsterdam, The Netherlands, 2016.
\bibitem{ravi2024sam}
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R{\"a}dle, Chloe Rolland, Laura Gustafson, et~al.
\newblock Sam 2: Segment anything in images and videos.
\newblock {\em CoRR}, 2024.
\bibitem{lin2024tracking}
Liting Lin, Heng Fan, Zhipeng Zhang, Yaowei Wang, Yong Xu, and Haibin Ling.
\newblock Tracking meets lora: Faster training, larger model, stronger performance.
\newblock {\em CoRR}, 2024.
\bibitem{fan2019lasot}
Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge~Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.
\newblock Lasot: A high-quality benchmark for large-scale single object tracking.
\newblock In {\em CVPR}, pages 5374--5383, Long Beach, CA, USA, 2019.
\bibitem{huang2019got}
Lianghua Huang, Xin Zhao, and Kaiqi Huang.
\newblock Got-10k: A large high-diversity benchmark for generic object tracking in the wild.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence}, 2019.
\bibitem{DBLP:journals/corr/abs-2403-05231}
Liting Lin, Heng Fan, Zhipeng Zhang, Yaowei Wang, Yong Xu, and Haibin Ling.
\newblock Tracking meets lora: Faster training, larger model, stronger performance.
\newblock {\em CoRR}, 2024.
\bibitem{vaswani2017attention}
A~Vaswani.
\newblock Attention is all you need.
\newblock {\em NeurIPS}, 2017.
\bibitem{yang2023towards}
Yang Yang, Yurui Huang, Weili Guo, Baohua Xu, and Dingyin Xia.
\newblock Towards global video scene segmentation with context-aware transformer.
\newblock In {\em AAAI}, pages 3206--3213, Washington, DC, USA, 2023.
\bibitem{dosovitskiy2020image}
Alexey Dosovitskiy.
\newblock An image is worth 16x16 words: Transformers for image recognition at scale.
\newblock {\em CoRR}, 2020.
\bibitem{huo2018learning}
Xuan Huo, Yang Yang, Ming Li, and De-Chuan Zhan.
\newblock Learning semantic features for software defect prediction by code comments embedding.
\newblock In {\em 2018 IEEE international conference on data mining (ICDM)}, pages 1049--1054, Singapore, 2018.
\bibitem{yang2022exploiting}
Yang Yang, Hongchen Wei, Hengshu Zhu, Dianhai Yu, Hui Xiong, and Jian Yang.
\newblock Exploiting cross-modal prediction and relation consistency for semisupervised image captioning.
\newblock {\em IEEE Transactions on Cybernetics}, 2022.
\bibitem{tian2019fcos}
Z~Tian, C~Shen, H~Chen, and T~He.
\newblock Fcos: Fully convolutional one-stage object detection.
\newblock {\em CoRR}, 2019.
\end{thebibliography}