Robust Tracking with Particle Filtering for Fluorescent Cardiac Imaging
Extracted source text · arXiv 2508.05262
\documentclass[USenglish,twocolumn]{article}
\usepackage[utf8]{inputenc}
\usepackage[big,online]{dgruyter}
\usepackage[sort&compress,square,numbers]{natbib}
\usepackage{times}
\usepackage{subfig}
\captionsetup[subfigure]{font=footnotesize,
labelfont=up,
justification=centering
}
\usepackage{xcolor}
\usepackage[detect-all=true,separate-uncertainty=true, multi-part-units=single,range-phrase={-},product-units = single]{siunitx}
\DeclareSIUnit{\pixel}{px}
\DeclareSIUnit{\fps}{fps}
\usepackage{pgfplots}
\usepgfplotslibrary{groupplots}
\pgfplotsset{width=10cm,compat=1.9}
\usepackage{verbatim}
\usepackage{tikz}
\usetikzlibrary{arrows,positioning,shapes.geometric}
\usetikzlibrary{intersections}
\begin{document}
%%%--------------------------------------------%%%
%%% Please do not alter the following lines: %%%
%%%--------------------------------------------%%%
%\articletype{Proceedings}
%\aop
\DOI{10.1515/}
\openaccess
\pagenumbering{gobble}
%%%--------------------------------------------%%%
\title{Robust Tracking with Particle Filtering for Fluorescent Cardiac Imaging}
\runningtitle{Robust Tracking with Particle Filtering for Fluorescent Cardiac Imaging}
%\subtitle{Insert subtitle if needed}
\author*[1]{S.~Guttikonda}
\author[1]{M.~Neidhardt}
\author[1]{J.~Sprenger}
\author[2]{J.~Petersen}
\author[2]{C.~Detter}
\author[1,5]{A.~Schlaefer}
\runningauthor{S.~Guttikonda et al.}
\affil[1]{\protect\raggedright
Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Am Schwarzenberg-Campus 3, Hamburg, Germany, e-mail: suresh.guttikonda@tuhh.de}
\affil[2]{\protect\raggedright
Department of Cardiovascular Surgery, University Heart and Vascular Center
Hamburg, Hamburg, Germany}
\affil[5]{\protect\raggedright
SustAInLivWork Center of Excellence, Kaunas, Lithuania}
\abstract{Intraoperative fluorescent cardiac imaging enables quality control following coronary bypass grafting surgery. We can estimate local quantitative indicators, such as cardiac perfusion, by tracking local feature points. However, heart motion and significant fluctuations in image characteristics caused by vessel structural enrichment limit traditional tracking methods. We propose a particle filtering tracker based on cyclic-consistency checks to robustly track particles sampled to follow target landmarks. Our method tracks $117$ targets simultaneously at \SI{25.4}{\fps}, allowing real-time estimates during interventions. It achieves a tracking error of (\SI{5.00(2.2)}{\pixel}) and outperforms other deep learning trackers (\SI{22.3(11)}{\pixel}) and conventional trackers (\SI{58.1(27.1)}{\pixel}).}
%\abstract{Fluorescent cardiac imaging can be utilized intraoperatively for quality control following coronary bypass grafting surgery. Local quantitative metrics, e.g., myocardial perfusion, can be estimated if tracking of local feature points is applicable. However, heart motion and strong variations in image features due to enrichment of the vessel structures limit conventional tracking approaches. Hence, we propose a deep learning based tracker with particle filtering. To enable robust tracking, we define a set of particles in proximity to the tracking target. We estimate tracking accuracy, derived by forward-backward estimates, of each particle and following remove and add new particles. Tracking metrics are estimated on 14 videos recorded during fluorescent cardiac interventions on pig hearts. We report a mean tracking error of \SI{5.00(2.2)}{\pixel} on image sequence lengths of roughly \SI{90}{\second}. We can track up to 117 targets simultaneously with \SI{25.4}{\fps}, allowing real-time estimates during interventions. Our tracker outperforms other deep learning trackers (\SI{22.3(11)}{\pixel}) and conventional trackers (\SI{58.1(27.1)}{\pixel}). Overall, we present a tracker that is robust towards image feature changes during enrichment and rapid movements.}
% 3-5 Keywords
\keywords{Motion Estimation, Heart Surface Tracking, Fluorescent Cardiac Imaging, Deep Learning}
\maketitle
% \vspace{-5mm}
\section{Introduction}
Fluorescent cardiac imaging (FCI) offers a non-invasive technique for visualizing coronary vessels. FCI provides real-time, dynamic information about cardiac function to diagnose pathological conditions, such as coronary bypass stenosis~\cite{detter2018qualitative} or graft assessment~\cite{ohmes2017techniques}. Additionally, clinicians can estimate quantitative values based on fluorescence intensity; for example, to determine myocardial perfusion~\cite{detter2018qualitative}. To achieve this, robustly tracking local features on the deforming and moving heart surface presents a significant challenge and remains highly desirable.
Conventional markerless object tracking methods, such as Minimum Output Sum of Squared Error (MOSSE)~\cite{DBLP:conf/cvpr/BolmeBDL10} and Kernelized Correlation Filter (KCF)~\cite{DBLP:conf/eccv/HenriquesCMB12}, operate with computational efficiency and enable real-time tracking. However, these methods deliver limited tracking accuracy when objects experience partial or full occlusion, images contain artifacts (e.g., motion blur caused by rapid movements), or image features change (e.g., during contrast enrichment).
%If possible, please provide figures in the grayscale or CMYK color mode rather than the RGB mode%
\begin{figure}
\centering
%\fbox{\includegraphics[options]{image}}
\includegraphics[width=\linewidth]{graphics/heart_tracking_data.drawio.png}
\caption{\textbf{Fluorescent Cardiac Imaging:} Video sequences illustrate different stages of the heart enrichment procedure. Large motions and varying image features challenge the effectiveness of conventional tracking approaches.}
\label{img:heart_tracking_data}
\end{figure}
Recently, tracking any point (TAP) approaches~\cite{DBLP:conf/iccv/DoerschYVG0ACZ23, DBLP:journals/corr/abs-2410-11831} have gained attention due to its ability to estimate both dense and long-range pixel-level trajectories in video sequences, even with occlusions. These techniques fundamentally use a coarse-to-fine tracking design that iteratively predicts an initial "coarse" point trajectories using low-resolution features and then refines it using local, spatiotemporal information at higher resolution. Researchers train these methods either with unlabeled raw data combined with unsupervised or semi-supervised learning~\cite{DBLP:journals/corr/abs-2410-11831}, or with large-scale synthetic data through supervised learning~\cite{DBLP:conf/iccv/DoerschYVG0ACZ23}.
\begin{figure*}
\centering
\subfloat[Amid Enrichment] {
\begin{tikzpicture}
\node[anchor=south west,inner sep=0] (img) at (0, 0){\includegraphics[trim=0 0 0 0, clip, width=0.495\linewidth]{graphics/cost_plot_FA___100004_233_610_s425_AMID_ENRICHMENT.pdf}};
\node[] at (0.75,0.4) {\scriptsize FBE [px]};
\end{tikzpicture}
\label{img:tracking_map_amid_enrichment}
}
\subfloat[Post Enrichment] {
\begin{tikzpicture}
\node[anchor=south west,inner sep=0] (img) at (0, 0){\includegraphics[trim=0 0 0 0, clip, width=0.495\linewidth]{graphics/cost_plot_FA___100004_233_610_s425_POST_ENRICHMENT.pdf}};
\node[] at (0.75,0.4) {\scriptsize FBE [px]};
\end{tikzpicture}
\label{img:tracking_map_post_enrichment}
}
\caption{\textbf{Tracking Error Dissemination:} We present the distribution of the FBE for all evaluated trackers. Tracking occurs (a) during contrast agent injection - amid enrichment (AE), and (b) post enrichment (PE), on video sequences with a duration of \SI{15}{\second}, respectively.}
\end{figure*}
\begin{comment}
\def\xw{2.8}
\def\top{2.2}
\def\cbDistance{0.6}
\begin{figure*}
\centering
\subfloat[Amid Enrichment] {
\begin{tikzpicture}
% Top Row
\node[anchor=south west,inner sep=0] (img) at (0*\xw,\top+\cbDistance){\includegraphics[trim=0 0 0 0, clip, width=2.8cm]{graphics/cost_plot_FA___100004_233_610_s425_AMID_ENRICHMENT_OPENCV_KCF.pdf}};
\node[anchor=south west,inner sep=0] (img) at (1*\xw,\top+\cbDistance){\includegraphics[trim=0 0 0 0, clip, width=2.8cm]{graphics/cost_plot_FA___100004_233_610_s425_AMID_ENRICHMENT_OPENCV_MOSSE.pdf}};
\node[anchor=south west,inner sep=0] (img) at (2*\xw,\top+\cbDistance){\includegraphics[trim=0 0 0 0, clip, width=2.8cm]{graphics/cost_plot_FA___100004_233_610_s425_AMID_ENRICHMENT_OPENCV_MEDIANFLOW.pdf}};
\node[] at (0*\xw+1.4,\top+\cbDistance+2) {\scriptsize KCF};
\node[] at (1*\xw+1.4,\top+\cbDistance+2) {\scriptsize MOSSE};
\node[] at (2*\xw+1.4,\top+\cbDistance+2) {\scriptsize Median Flow};
% Bottom Row
\node[anchor=south west,inner sep=0] (img) at (0*\xw,0+\cbDistance){\includegraphics[trim=0 0 0 0, clip, width=2.8cm]{graphics/cost_plot_FA___100004_233_610_s425_AMID_ENRICHMENT_COTRACKER_ONLINE.pdf}};
\node[anchor=south west,inner sep=0] (img) at (1*\xw,0+\cbDistance){\includegraphics[trim=0 0 0 0, clip, width=2.8cm]{graphics/cost_plot_FA___100004_233_610_s425_AMID_ENRICHMENT_COTRACKER_OFFLINE.pdf}};
\node[anchor=south west,inner sep=0] (img) at (2*\xw,0+\cbDistance){\includegraphics[trim=0 0 0 0, clip, width=2.8cm]{graphics/cost_plot_FA___100004_233_610_s425_AMID_ENRICHMENT_COTRACKER_PARTICLE_3.pdf}};
\node[] at (0*\xw+1.4,0+2+\cbDistance) {\scriptsize CoTracker (Online)};
\node[] at (1*\xw+1.4,0+2+\cbDistance) {\scriptsize CoTracker (Offline)};
\node[] at (2*\xw+1.4,0+2+\cbDistance) {\scriptsize Ours (3 Particles)};
% Colorbar
\node[anchor=center,inner sep=0] (img) at (1.5*\xw,0.35){\includegraphics[trim=0 12cm 0 2mm, clip, width=5cm, height=3mm]{graphics/cost_plot_colorbar.pdf}};
\node[] at (1.0,0.35) {\scriptsize FBE [px]};
\node[] at (1.5*\xw - 2.5 + 0*1/5*5,0) {\scriptsize 0};
\node[] at (1.5*\xw - 2.5 + 1*1/5*5,0) {\scriptsize 20};
\node[] at (1.5*\xw - 2.5 + 2*1/5*5,0) {\scriptsize 40};
\node[] at (1.5*\xw - 2.5 + 3*1/5*5,0) {\scriptsize 60};
\node[] at (1.5*\xw - 2.5 + 4*1/5*5,0) {\scriptsize 80};
\node[] at (1.5*\xw - 2.5 + 5*1/5*5,0) {\scriptsize 100};
\end{tikzpicture}
\label{img:tracking_map_amid_enrichment}
}
\subfloat[Post Enrichment] {
\begin{tikzpicture}
% Top Row
\node[anchor=south west,inner sep=0] (img) at (0*\xw,\top+\cbDistance){\includegraphics[trim=0 0 0 0mm, clip, width=2.8cm]{graphics/cost_plot_FA___100004_610_987_s425_POST_ENRICHMENT_OPENCV_KCF.pdf}};
\node[anchor=south west,inner sep=0] (img) at (1*\xw,\top+\cbDistance){\includegraphics[trim=0 0 0 0mm, clip, width=2.8cm]{graphics/cost_plot_FA___100004_610_987_s425_POST_ENRICHMENT_OPENCV_MOSSE.pdf}};
\node[anchor=south west,inner sep=0] (img) at (2*\xw,\top+\cbDistance){\includegraphics[trim=0 0 0 0mm, clip, width=2.8cm]{graphics/cost_plot_FA___100004_610_987_s425_POST_ENRICHMENT_OPENCV_MEDIANFLOW.pdf}};
\node[] at (0*\xw+1.4,\top+\cbDistance+2) {\scriptsize KCF};
\node[] at (1*\xw+1.4,\top+\cbDistance+2) {\scriptsize MOSSE};
\node[] at (2*\xw+1.4,\top+\cbDistance+2) {\scriptsize Median Flow};
% Bottom Row
\node[anchor=south west,inner sep=0] (img) at (0*\xw,0+\cbDistance){\includegraphics[trim=0 0 0 0mm, clip, width=2.8cm]{graphics/cost_plot_FA___100004_610_987_s425_POST_ENRICHMENT_COTRACKER_ONLINE.pdf}};
\node[anchor=south west,inner sep=0] (img) at (1*\xw,0+\cbDistance){\includegraphics[trim=0 0 0 0mm, clip, width=2.8cm]{graphics/cost_plot_FA___100004_610_987_s425_POST_ENRICHMENT_COTRACKER_OFFLINE.pdf}};
\node[anchor=south west,inner sep=0] (img) at (2*\xw,0+\cbDistance){\includegraphics[trim=0 0 0 0mm, clip, width=2.8cm]{graphics/cost_plot_FA___100004_610_987_s425_POST_ENRICHMENT_COTRACKER_PARTICLE_3.pdf}};
\node[] at (0*\xw+1.4,0+2+\cbDistance) {\scriptsize CoTracker (Online)};
\node[] at (1*\xw+1.4,0+2+\cbDistance) {\scriptsize CoTracker (Offline)};
\node[] at (2*\xw+1.4,0+2+\cbDistance) {\scriptsize Ours (3 Particles)};
% Colorbar
\node[anchor=center,inner sep=0] (img) at (1.5*\xw,0.35){\includegraphics[trim=0 12cm 0 2mm, clip, width=5cm, height=3mm]{graphics/cost_plot_colorbar.pdf}};
\node[] at (1.0,0.35) {\scriptsize FBE [px]};
\node[] at (1.5*\xw - 2.5 + 0*1/5*5,0) {\scriptsize 0};
\node[] at (1.5*\xw - 2.5 + 1*1/5*5,0) {\scriptsize 20};
\node[] at (1.5*\xw - 2.5 + 2*1/5*5,0) {\scriptsize 40};
\node[] at (1.5*\xw - 2.5 + 3*1/5*5,0) {\scriptsize 60};
\node[] at (1.5*\xw - 2.5 + 4*1/5*5,0) {\scriptsize 80};
\node[] at (1.5*\xw - 2.5 + 5*1/5*5,0) {\scriptsize 100};
\end{tikzpicture}
\label{img:tracking_map_post_enrichment}
}
\caption{\textbf{Tracking Error Dissemination:} Distribution of the FBE of all evaluated trackers. Tracking was performed (a) during contrast agent injection, amid enrichment (AE), and (b) post enrichment (PE), on a video sequence length of \SI{15}{\second}, respectively.}
\end{figure*}
\end{comment}
% RESULTS!
%When used out-of-the box, both conventional method and the recent $(\textbf{TAP})$ algorithm (CoTracker3~\cite{DBLP:journals/corr/abs-2410-11831}) has shown poor tracking performance in tracking the points on the regions on the heart surface where there is a large appearance and scale change between frame t=\SI{9.3}{\second} and t=\SI{24.4}{\second}, i.e., anchor points sampled before enrichment, refer Figure~\ref{img:tracking_map_amid_enrichment}. However, CoTracker3~\cite{DBLP:journals/corr/abs-2410-11831} performed much better to track the surface across the frames with post enrichment, refer Figure~\ref{img:tracking_map_post_enrichment}.
In this work, we propose to retain the benefits of the TAP algorithm, trained on a large-scale dataset, while achieving real-time tracking performance without fine-tuning by refining its predictions using a particle filter–style filtering mechanism. Our study shows that the proposed method outperforms both conventional and deep learning trackers and exhibits greater robustness to large appearance and scale changes during both amid enrichment (AE) and post enrichment (PE) scenarios.
\vspace{-5mm}
\section{Method}
\subsection{Notations} \label{sec:formulation}
Given a video $\begin{Bmatrix}\boldsymbol{\mathcal{I}}_{t}\end{Bmatrix}^{T}_{t=1}$, which consists of a sequence of $\text{T}$ frames $\boldsymbol{\mathcal{I}}_{t} \in \mathbb{R}^{H \times W \times 3}$, and an anchor point $\boldsymbol{\mathcal{Q}} = (t^{q}, x^{q}, y^{q}) \in \mathbb{R}^{3}$, where $t^{q}$ denoting the query frame index, and $(x^{q}, y^{q})$ representing the corresponding spatial location of the anchor point, the point tracking problem aims to estimate the trajectory of this anchor point across the video. Specifically, the forward point track comprises predictions on subsequent frames of the image sequence and is formally denoted as $\boldsymbol{\mathcal{P}}^{f}_{t} = (x_{t}, y_{t}) \in \mathbb{R}^{2}$, $t \in [1, T]$, with $(x_{t}, y_{t}) = (x^{q}, y^{q})$.
Correspondingly, we define a backward point track consisting of predictions performed from the last to the first frame of the sequence. It is formally denoted as $\boldsymbol{\mathcal{P}}^{b}_{t} = (\hat{x}_{t}, \hat{y}_{t}) \in \mathbb{R}^{2}$, $t \in [1, T]$, with $(\hat{x}_{T}, \hat{y}_{T}) = (x_{T}, y_{T})$.
\subsection{Our Tracking Algorithm} \label{sec:our_approach}
We propose to track the center of a bounding box with size $w \times w$~pixels. First, we sample $M$ points, referred to as particles, $\begin{Bmatrix}\boldsymbol{X}^{m}_{0}\end{Bmatrix}^{M}_{m=1} \in \mathbb{R}^{2}$, from a Gaussian distribution $ \mathcal{N}\begin{pmatrix}\boldsymbol{X}_{0} = [{x}^{q}, {y}^{q}]^{\top}; {\sigma}_{0} = 5.0\end{pmatrix} $ centered at the bounding box center. Each particle receives an initial weight, defined by forward-backward tracking consistency~\cite{DBLP:conf/icpr/KalalMM10, DBLP:conf/cvpr/WangJE19}, such that $\begin{Bmatrix}{w}^{m}_{0}\end{Bmatrix}^{M}_{m=1} = \frac{1}{M}$, where ${\sum}^{M}_{m=1} w^{m}_{t} = 1$ and $t \in [0, T]$.
Next, we compute forward prediction tracks for the sampled particles across a sequence of $L$ frames. We recognize that particles experience varying tracking accuracy due to the highly deformable heart surface, which causes occlusions and changes in image features from contrast enrichment. To maintain robustness, we remove particles based on their corresponding weights. Following this, we estimate the Gaussian likelihood ($\mathcal{L}$) for each particle as,
\begin{multline}
\mathcal{L}(\boldsymbol{\mathcal{P}}^{b}_{t}, \boldsymbol{\Sigma} | \boldsymbol{\mathcal{P}}^{f}_{t}) = \\ \frac{1}{2 \pi \sqrt{|\boldsymbol{\Sigma}|}} exp\begin{pmatrix} - \frac{1}{2}
(\boldsymbol{\mathcal{P}}^{b}_{t} - \boldsymbol{\mathcal{P}}^{f}_{t})^{\top}\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mathcal{P}}^{b}_{t} - \boldsymbol{\mathcal{P}}^{f}_{t})\end{pmatrix}
\label{eq:neg_loglikelihood_error}
\end{multline}
with the covariance matrix $\boldsymbol{\Sigma} = \begin{bmatrix} \sigma^2 & 0 \\ 0 & \sigma^2 \end{bmatrix}$, the standard deviation $\sigma = 3.0$, $d$ represent the data dimensions, and $|\boldsymbol{\Sigma}|$ denote the determinant of the covariance matrix.
Finally, we perform stochastic universal soft-resampling\footnote[1]{\url{https://github.com/stanford-iprl-lab/torchfilter}}~\cite{DBLP:books/daglib/0014221, DBLP:conf/corl/KarkusHL18} with a trade-off resampling parameter $\alpha = 0.5$. This process retains or replaces particles with lower weights by generating new particles sampled near those with higher weights at the end of each filtering window, thereby enhancing tracking robustness and accuracy.
\vspace{-5mm}
\subsection{Experimental Evaluation}
We evaluate different tracking approaches using $14$ videos recorded during pig heart fluorescent cardiac interventions. Following anesthesia, the pigs undergo coronary revascularization on their beating hearts and median sternotomy. During the procedure, we inject a contrast agent (indocyanine green) and expose the heart to near-infrared light at \SI{785}{\nano\meter}. The fluorescence signal is captured using an FSI device (LLS GmbH, Ulm, Germany) equipped with band-pass filters that block excitation light while selectively transmitting fluorescent light at \SI{830}{\nano\meter}. Each recording lasts approximately \SI{100}{\second}, with a temporal resolution of \SI{25}{\Hz} and a spatial resolution of \SI{512}{\pixel} $\times$ \SI{384}{\pixel}. Figure~\ref{img:heart_tracking_data} illustrates a complete enrichment cycle, including both amid and post enrichment sequences. We conduct all training and evaluations using a GPU (Titan RTX, Nvidia, California, USA).
To evaluate tracking performance, we perform cyclic-consistency checks between the forward and backward predicted point tracks~\cite{DBLP:conf/icpr/KalalMM10}. We define the forward-backward tracking error (FBE) for each anchor point as follows:
\begin{equation}
FBE = \sum^{T}_{t=0} || \boldsymbol{\mathcal{P}}^{b}_{t} - \boldsymbol{\mathcal{P}}^{f}_{t} ||_2
\label{eq:forward_backward_error}
\end{equation}
where $||\cdot||_2$ is the Euclidean norm.
As shown in Figure~\ref{img:dense_grid_anchors_bboxes_particles}, we estimate tracking performance on a grid of anchor points spaced \SI{32}{\pixel} apart along both axes. We place the sample anchor points at the center of bounding boxes with size w=\SI{100}{\pixel}, as smaller sizes result in immediate tracking failures. For comparison, we select conventional bounding box–based trackers\footnote[2]{\url{https://docs.opencv.org/3.4.20/d0/d0a/classcv_1_1Tracker.html}} including MOSSE~\cite{DBLP:conf/cvpr/BolmeBDL10}, KCF~\cite{DBLP:conf/eccv/HenriquesCMB12}, and Median Flow~\cite{DBLP:conf/icpr/KalalMM10}. For our proposed tracker, we sample a collection of particles from a Gaussian distribution around each anchor point. We ensure that all sampled particles lie within the bounding box to provide a fair comparison with other methods. We evaluate our tracker using $M\in{[3,5,25]}$ sampled particles and set the filtering window length to $L = 16$ frames for all experiments.
%If possible, please provide figures in the grayscale or CMYK color mode rather than the RGB mode%
\begin{figure}
\centering
%\fbox{\includegraphics[options]{image}}
\begin{tikzpicture}
\node[anchor=south west,inner sep=0] (img) at (0,0){\includegraphics[trim=0 16cm 7cm 0, clip, width=\linewidth]{graphics/DENSE_GRID_m64_g32_s117_p5_particles.pdf}};
\node[] at (1.3cm,3.0cm) {\textcolor{white}{\scriptsize\SI{100}{\pixel} $\times$ \SI{100}{\pixel}}};
\draw [|<->|,>=stealth, thick, white] (3.2,3.7-1.0) -- (3.8,3.7-1.0) node [midway,above] {\textcolor{white}{\scriptsize $\SI{32}{\pixel}$}};
\end{tikzpicture}
\caption{\textbf{Evaluation:} We evaluate tracking performance using bounding boxes (blue) arranged in a grid structure. The red dots represent anchor points positioned at the center of each bounding box. For our proposed tracker, we sample particles (green dots) located within the bounding boxes.}
\label{img:dense_grid_anchors_bboxes_particles}
\end{figure}
%In all our experiments, we sampled a dense grid of anchor points to tracks on the heart's surface during the stages of enrichment, refer Figure~\ref{img:dense_grid_anchors_bboxes_particles}. These anchors are used directly for tracking using CoTracker~\cite{DBLP:journals/corr/abs-2410-11831}. We used bounding boxes of size $100 \times 100$ pixels centered around the sampled anchor points to obtain the region of interest (ROI) to be used with conventional trackers, i.e., MOSSE~\cite{DBLP:conf/cvpr/BolmeBDL10}, KCF~\cite{DBLP:conf/eccv/HenriquesCMB12}, and Median Flow~\cite{DBLP:conf/icpr/KalalMM10}. We explicitly used an offset margin to avoid sampling the bounding boxes outside the image boundaries.
%For our proposed approach we use the sampled anchor points as the mean of Gaussian distribution to sample our tracking particles, as discussed in Section~\ref{sec:our_approach}. We use CoTracker3~\cite{DBLP:journals/corr/abs-2410-11831} as our baseline model for tracking. CoTracker3 (Online) is used to compute the forward point track $\boldsymbol{\mathcal{P}}^{f}_t$ for the filtering window length $L = 16$ followed by forward backward point track $\boldsymbol{\mathcal{P}}^{b}_{t}$ using CoTracker3 (Offline).
\vspace{-5mm}
\section{Results} \label{sec:our_results}
First, as shown in Figures~\ref{img:tracking_map_amid_enrichment} and~\ref{img:tracking_map_post_enrichment}, we assess the distribution of the FBE, defined in Equation~\ref{eq:forward_backward_error}, on the surface of the heart during amid-enrichment (AE) and post-enrichment (PE) sequences, respectively. The results reveal that traditional trackers (top row) exhibit a non-homogeneous FBE distribution; for example, MOSSE tracks only discrete features on the heart surface with low FBE. In PE tracking, deep learning–based trackers provide more consistent estimates, whereas they only partially succeed during AE tracking (see CoTracker online). Our approach delivers robust estimates that remain consistent regardless of tracking location, achieving a mean FBE of \SI{4.9(4.8)}{\pixel} and \SI{4.4(3.6)}{\pixel} for AE and PE sequences, respectively.
%\subsection{Tracker's Performance}
%If possible, please provide figures in the grayscale or CMYK color mode rather than the RGB mode%
\begin{figure*}
\centering
%\fbox{\includegraphics[options]{image}}
\includegraphics[width=\linewidth]{graphics/tracking_performance_FA___100004.pdf}
\caption{\textbf{Tracking Performance of All Trackers:} (a) Tracking accuracy vs sequence length, (b) Inference time vs the number of anchor points, and (c) Peak GPU memory usage vs sequence length.}
\label{img:Performance_Trackers}
\end{figure*}
\begin{comment}
\begin{figure*}
\centering
\subfloat[] {
%\fbox{\includegraphics[options]{image}}
{\includegraphics[width=0.32\linewidth]{graphics/rmse_plot_FA___100004_s117_frame_trackers.pdf}}
\label{img:Performance_FBE}
}
\subfloat[] {
%\fbox{\includegraphics[options]{image}}
{\includegraphics[width=0.32\linewidth]{graphics/time_plot_FA___100004_89_144_anchors_trackers.pdf}}
\label{img:Performance_InferenceTime}
}
\subfloat[] {
%\fbox{\includegraphics[options]{image}}
{\includegraphics[width=0.32\linewidth]{graphics/gpu_plot_FA___100004_s117_frame_trackers.pdf}}
\label{img:Performance_GPU}
}
\caption{\textbf{Tracking Performance for all trackers:} (a) tracking accuracy vs. sequence length, (b) inference time vs. number of anchor points, and (c) peak GPU usage vs. sequence length. }
\end{figure*}
\end{comment}
%We first performed a systematic evaluation and comparison of our proposed approach with conventional~\cite{DBLP:conf/cvpr/BolmeBDL10, DBLP:conf/eccv/HenriquesCMB12, DBLP:conf/icpr/KalalMM10} algorithms and CoTracker~\cite{DBLP:journals/corr/abs-2410-11831} for varying number of anchor points and tracking frames. Figure~\ref{img:time_plot_ablation_anchors} shows the performance of different tracking approaches for a fixed number of $55$ image frames and anchor points of sizes $\{6, 15, 35, 45, 117, 153, 425, 561, 1025, 1617, 2337, 3969\}$ by varying the spacing between anchors. Intitially, MOSSE tracker has shown better superior inference time per frame performance for sparse anchor points of size upto $153$. However, for a dense tracking CoTracker (Online) outperforms all other approaches. In Table~\ref{tab:dense_grid_tracking_rmse}, both MOSSE and CoTracker (Online) achieved tracking at $\sim 50$ fps, CoTracker (Offline) failed to track beyond $500$ frames due to higher GPU requirement while our proposed approach (with 5 particles) achieved $\sim 18.7$ fps. For the same study we also evaluate the tracker's accuracy by computing the forward-backward error, refer Equation~\ref{eq:forward_backward_error}, as shown in Figure~\ref{img:rmse_plot_ablation_flames}. Our approach achieved better tracking performance compared to others in our study. Furthermore, we performed the evaluation of GPU utilization of CoTracker (Online) and (Offline) and compared with the proposed approach in Figure~\ref{img:gpu_plot_ablation_anchors}. Overall CoTracker (Online) achieves lower GPU utilization when varying number of anchors with frames to track.
%\subsection{Tracker's Accuracy}
%Table~\ref{tab:dense_grid_tracking_rmse} summarizes the tracking accuracy of different trackers for high enrichment and lower enrichment scenarios using $117$ dense anchor points. For amid enrichment scenario, we densly sampled anchors in frame $t=9.3$ and tracked until frame t = \SI{24.4}{\second} and calculated the forward-backward tracking error~\ref{eq:forward_backward_error}. Similarly, for post enrichment scenario, we densely sampled anchors in frame t = \SI{24.4}{\second} and tracked until frame t = \SI{39.5}{\second}. Overall, our proposed approach achieved better tracking accuracy thus displaying robustness to larger appearance and scale changes during both amid and post enrichment scenarios of heart's surface tracking.
\begin{table}[!ht]
\footnotesize
\caption{\textbf{Tracking Performance:} We report the mean FBE and frames per second (fps) for tracking $117$ anchor points across $14$ videos during contrast agent injection, amid enrichment (AE), and post enrichment (PE), each with a tracking duration of \SI{15}{\second}. We refer to the total tracking duration of \SI{90}{\second}, which includes both AE and PE sequences, as "Total".}
\renewcommand\arraystretch{1.3}
\renewcommand\tabcolsep{0pt}
\begin{tabular*}{\linewidth}{@{\extracolsep{\fill}}lccccc}
\hline
\text{} & & \multicolumn{3}{c}{FBE} \\ \cmidrule{3-5}
Tracker & fps & AE & PE & Total \\ \midrule
KCF & 5.2 &\SI{24.3(13.7)}{} & \SI{26.0(17.3)}{} &\SI{58.1(27.1)}{} \\
MOSSE & \textbf{50.8} &\SI{38.3(17.9)}{} &\SI{43.8(21.0)}{} &\SI{70.8(17.8)}{} \\
Median Flow & 13.9 &\SI{84.7(49.9)}{} &\SI{83.5(47.9)}{} &\SI{159.9(59.2)}{} \\
CoTracker (Online) & 42.5 &\SI{25.8(19.9)} {} &\SI{6.4(3.4)}{} &\SI{22.3(11.0)}{} \\
CoTracker (Offline) & 31.3 &\SI{15.4(10.6)}{} &\SI{5.7(3.4)}{} & -- \\ \midrule
Ours (3 particles) & \textbf{25.4} &\SI{4.9(4.8)} {} &\SI{4.4( 3.6)}{} &\SI{5.0(2.2)}{} \\
Ours (5 particles) & 17.9 &\SI{4.8(4.6)} {} &\SI{4.3( 3.4)}{} &\SI{4.8(2.1)}{} \\
Ours (25 particles) & 4.2 &\textbf{\SI{4.5(4.7)}{}} &\textbf{\SI{3.9(3.3)}{}} &\textbf{\SI{4.3(2.2)}{}} \\
\hline
\end{tabular*}\par\medskip
\label{tab:dense_grid_tracking_rmse}
\end{table}
% Needs more adapting still
Additionally, we calculate the average FBE for image sequences of varying durations, as shown in Figure~\ref{img:Performance_Trackers}a. Compared to other trackers, our approach achieves a notably lower mean FBE for tracking sequences longer than \SI{10}{\second}. Figure~\ref{img:Performance_Trackers}a demonstrates that while our tracking error remains consistently below \SI{7}{\pixel} during AE, the mean FBE significantly increases for all other trackers. During PE, the error varies only marginally across all methods. A similar trend appears in Table~\ref{tab:dense_grid_tracking_rmse}, where conventional trackers show overall higher tracking errors compared to our method. Notably, unlike our approach, CoTracker’s tracking error increases only slightly during PE sequences but nearly triples during AE sequences. Furthermore, increasing the number of particles reduces the FBE in our approach. With just three particles, our method outperforms all other trackers, achieving an overall FBE of \SI{5.0(2.2)}{\pixel}.
Compared to CPU-based traditional trackers, as shown in Figure~\ref{img:Performance_Trackers}b, our approach exhibits a longer inference time. However, the inference time per anchor point decreases as we reduce the number of particles. Achieving a frame rate of up to \SI{25.4}{\fps} with three particles—matching the fluorescent cardiac image stream—we can track $117$ anchor points in parallel while maintaining a lower FBE compared to other trackers. Figure~\ref{img:Performance_Trackers}c presents the GPU peak consumption for various tracking sequence lengths by deep learning–based trackers. While our method remains stable over longer sequences, it incurs slightly higher computational costs than CoTracker (online) due to the use of $M$ particles per anchor point, although it remains more efficient than CoTracker (offline).
% Further, we evaluate the mean FBE for different lengths of image sequences, as depicted in Figure~\ref{img:Performance_Trackers}a. It stands out that our method provides a lower mean FBE for tracking sequences longer than approximately \SI{10}{\second} in comparison to other trackers. Further, we investigate the FBE for AE and PE tracking sequences. It can be seen in Figure~\ref{img:Performance_Trackers}a that AE the mean FBE increases drastically for other trackers while our tracking error remains below \SI{7}{\pixel}. Subsequently, during PE tracking, the mean FBE only slightly changes. A similar trend can be observed in Table~\ref{tab:dense_grid_tracking_rmse}. Only minimal deviations in FBE are observed for our tracker between AE and PE sequences. Overall conventional trackers have an increased FBE in comparison to our method. It can be noted that during PE sequences the FBE is only slightly increased for CoTracker in comparison to our method. However, during AE sequences the FBE is almost threefold increased. It can also be noted that the FBE decreases for our method if more particles are utilized. Still, even for 3 particles, we outperform other trackers with an overall FBE of \SI{5.0(2.2)}{\pixel}.
% Considering the computational performance of our tracker, we can achieve a framerate of up to \SI{25.4}{\fps} with 3 particles. Our deep learning based tracker has an increased inference time in contrast to CPU based trackers, depicted in Figure~\ref{img:Performance_Trackers}b. However, by reducing the number of particles, the inference time per anchor point also reduces. Following, we can track 117 anchor points in parallel with up to \SI{25.4}{fps}, which matches the framerate of the fluorescent cardiac image stream. Hence, real-time estimates with a lower FBE than comparative trackers are feasible. The GPU peak usage for tracking different sequence lengths is given in Figure~\ref{img:Performance_Trackers}c. As expected, our approach is computationally more expensive in comparison to the online CoTracker but is stable across all sequence lengths.
\vspace{-5mm}
\section{Discussion and Conclusion}
Real-time capture of rapid movements and changes on the heart’s surface is essential for estimating quantitative metrics and assessing heart functions. While conventional trackers offer computational efficiency, they exhibit poor tracking accuracy during fluorescent cardiac imaging (FCI). Advanced deep learning–based trackers, such as CoTracker, demonstrate improved tracking performance with robustness to occlusions; however, they still struggle to overcome domain gaps and face trade-offs between accuracy and GPU computational constraints over longer sequences. In this work, we employ a particle filter mechanism to refine CoTracker’s predictions, enabling robust real-time estimates even when image features change over time, such as during contrast agent injection.
Our experiments indicate that, when used out-of-the-box, CoTracker (offline) outperforms conventional trackers but is limited to video sequences of up to \SI{21}{\second}. In contrast, CoTracker (online) can handle longer sequences but achieves lower overall performance. By applying our proposed approach, we achieve a fourfold improvement in tracking performance on \SI{90.0}{\second}-long video sequences while simultaneously tracking 117 anchor points at a temporal sampling rate of \SI{25.0}{\fps}.
% To get a deeper understanding of cardiac mechanics and the pathological conditions we need to capture the rapid movements and changes in the heart's surfaces in real-time. While conventional trackers have shown to be able to computationally efficient, however, they result in poor tracking accuracy when there is large appearance and scale changes such as during heart's enrichment. Recent Track Any Point (TAP) algorithms such as CoTracker has shown to be more robust in tracking objects however we need to bridge domain gap and trade-off accuracy with real-time requirements and GPU computation limitations. In this regard, by combining tracking with filtering, we propose to refine the predictions from the CoTracker using particle filter mechanism to achieve more robustness while meeting real-time requirements.
% In our experiments, we achieve better tracking performance both in high and low heart enrichment with dense surface points tracking. Using $3$ particles, we achieved three times better tracking performance and robustness in high enrichment scenario and an inference speed of $\sim$\SI{25.0}{\fps} when tracking $117$ anchor points across $2400$ frames. However, our proposed approach's computational requirements is heavily dependent on the number of particles per anchors been used and has room for further improvements in the future.
%\begin{acknowledgement}
%\end{acknowledgement}
\noindent
\textsf{\textbf{Author Statement}}\\
This research was co-funded by the MARLOC project (DFG, grant SCHL 1844-10-1) and by the European Union under Horizon Europe programme grant agreement No. 101059903; and by the European Union funds for the period 2021-2027. Conflict of interest: none. Informed consent: obtained. Ethical approval: not applicable.
% This research was partially funded by the MARLOC DFG project (SCHL 1844-10-1) and by the European Union under Horizon Europe programme grant agreement No. 101059903; and by the European Union funds for the period 2021-2027 and the state budget of the Republic of Lithuanian financial agreement Nr. 10-042-P-0001. Conflict of interest: Authors state no conflict of interest. Informed consent: Informed consent has been obtained from all individuals included in this study. Ethical approval: not applicable.
\vspace{-7mm}
\bibliographystyle{elsevier3}
\bibliography{Proceedings_template_LaTeX}
\newpage
\begin{comment}
%If possible, please provide figures in the grayscale or CMYK color mode rather than the RGB mode%
\begin{figure}
\centering
%\fbox{\includegraphics[options]{image}}
{\includegraphics[width=\linewidth]{graphics/tracker_filter_framework.drawio.png}}
\caption{Our proposed tracker with particle filtering framework.}
\label{img:tracker_filter_framework}
\end{figure}
%If possible, please provide figures in the grayscale or CMYK color mode rather than the RGB mode%
\begin{figure}
\centering
%\fbox{\includegraphics[options]{image}}
{\includegraphics[width=\linewidth]{graphics/rmse_plot_FA___100004_s117_frame_trackers.pdf}}
\caption{Comparison of different tracker's tracking accuracy across different tracking frame lengths.}
\label{img:rmse_plot_ablation_flames}
\end{figure}
%If possible, please provide figures in the grayscale or CMYK color mode rather than the RGB mode%
\begin{figure}
\centering
%\fbox{\includegraphics[options]{image}}
{\includegraphics[width=\linewidth]{graphics/gpu_plot_FA___100004_s117_frame_trackers.pdf}}
\caption{Comparison of different tracker's GPU usage across different tracking frame lengths.}
\label{img:gpu_plot_ablation_anchors}
\end{figure}
%If possible, please provide figures in the grayscale or CMYK color mode rather than the RGB mode%
\begin{figure}
\centering
%\fbox{\includegraphics[options]{image}}
{\includegraphics[width=\linewidth]{graphics/time_plot_FA___100004_89_144_anchors_trackers.pdf}}
\caption{Comparison of different tracker's inference time using different tracking anchor points.}
\label{img:time_plot_ablation_anchors}
\end{figure}
\end{comment}
\end{document}
\begin{thebibliography}{10}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\eprint}[1]{\href{http://arxiv.org/abs/#1}{\path{#1}}}
\providecommand{\DOIprefix}{doi:}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{http://dx.doi.org/#1}{\path{#1}}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\BIBand}{and}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\undefined \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Article
\bibitem[{Detter et~al.(2018)Detter, Russ, Kersten, Reichenspurner and
Wipper}]{detter2018qualitative}
\bibinfo{author}{Detter\xfnm[ C.]}, \bibinfo{author}{Russ\xfnm[ D.]},
\bibinfo{author}{Kersten\xfnm[ J.F.]}, \bibinfo{author}{Reichenspurner\xfnm[
H.]}, \bibinfo{author}{Wipper\xfnm[ S.]}.
\newblock \bibinfo{title}{Qualitative angiographic and quantitative myocardial
perfusion assessment using fluorescent cardiac imaging during graded coronary
artery bypass stenosis}.
\newblock \bibinfo{journal}{Int J Cardiovasc Imaging} \bibinfo{year}{2018};.
%Type = Article
\bibitem[{Ohmes et~al.(2017)Ohmes, Di~Franco, Di~Giammarco, Rosati, Lau,
Girardi et~al.}]{ohmes2017techniques}
\bibinfo{author}{Ohmes\xfnm[ L.B.]}, \bibinfo{author}{Di~Franco\xfnm[ A.]},
\bibinfo{author}{Di~Giammarco\xfnm[ G.]}, \bibinfo{author}{Rosati\xfnm[
C.M.]}, \bibinfo{author}{Lau\xfnm[ C.]}, \bibinfo{author}{Girardi\xfnm[
L.N.]}, et~al.
\newblock \bibinfo{title}{Techniques for intraoperative graft assessment in
coronary artery bypass surgery}.
\newblock \bibinfo{journal}{Journal of Thoracic Disease} \bibinfo{year}{2017};.
%Type = Inproceedings
\bibitem[{Bolme et~al.(2010)Bolme, Beveridge, Draper and
Lui}]{DBLP:conf/cvpr/BolmeBDL10}
\bibinfo{author}{Bolme\xfnm[ D.S.]}, \bibinfo{author}{Beveridge\xfnm[ J.R.]},
\bibinfo{author}{Draper\xfnm[ B.A.]}, \bibinfo{author}{Lui\xfnm[ Y.M.]}.
\newblock \bibinfo{title}{Visual object tracking using adaptive correlation
filters}.
\newblock In: \bibinfo{booktitle}{{CVPR}}. \bibinfo{publisher}{{IEEE}};
\bibinfo{year}{2010},.
%Type = Inproceedings
\bibitem[{Henriques et~al.(2012)Henriques, Caseiro, Martins and
Batista}]{DBLP:conf/eccv/HenriquesCMB12}
\bibinfo{author}{Henriques\xfnm[ J.F.]}, \bibinfo{author}{Caseiro\xfnm[ R.]},
\bibinfo{author}{Martins\xfnm[ P.]}, \bibinfo{author}{Batista\xfnm[ J.P.]}.
\newblock \bibinfo{title}{Exploiting the circulant structure of
tracking-by-detection with kernels}.
\newblock In: \bibinfo{booktitle}{{ECCV}}. \bibinfo{publisher}{Springer};
\bibinfo{year}{2012},.
%Type = Inproceedings
\bibitem[{Doersch et~al.(2023)Doersch, Yang, Vecer{\'{\i}}k, Gokay, Gupta,
Aytar et~al.}]{DBLP:conf/iccv/DoerschYVG0ACZ23}
\bibinfo{author}{Doersch\xfnm[ C.]}, \bibinfo{author}{Yang\xfnm[ Y.]},
\bibinfo{author}{Vecer{\'{\i}}k\xfnm[ M.]}, \bibinfo{author}{Gokay\xfnm[
D.]}, \bibinfo{author}{Gupta\xfnm[ A.]}, \bibinfo{author}{Aytar\xfnm[ Y.]},
et~al.
\newblock \bibinfo{title}{{TAPIR:} tracking any point with per-frame
initialization and temporal refinement}.
\newblock In: \bibinfo{booktitle}{{ICCV}}. \bibinfo{publisher}{{IEEE}};
\bibinfo{year}{2023},.
%Type = Article
\bibitem[{Karaev et~al.(2024)Karaev, Makarov, Wang, Neverova, Vedaldi and
Rupprecht}]{DBLP:journals/corr/abs-2410-11831}
\bibinfo{author}{Karaev\xfnm[ N.]}, \bibinfo{author}{Makarov\xfnm[ I.]},
\bibinfo{author}{Wang\xfnm[ J.]}, \bibinfo{author}{Neverova\xfnm[ N.]},
\bibinfo{author}{Vedaldi\xfnm[ A.]}, \bibinfo{author}{Rupprecht\xfnm[ C.]}.
\newblock \bibinfo{title}{Cotracker3: Simpler and better point tracking by
pseudo-labelling real videos}.
\newblock \bibinfo{journal}{CoRR} \bibinfo{year}{2024};.
%Type = Inproceedings
\bibitem[{Kalal et~al.(2010)Kalal, Mikolajczyk and
Matas}]{DBLP:conf/icpr/KalalMM10}
\bibinfo{author}{Kalal\xfnm[ Z.]}, \bibinfo{author}{Mikolajczyk\xfnm[ K.]},
\bibinfo{author}{Matas\xfnm[ J.]}.
\newblock \bibinfo{title}{Forward-backward error: Automatic detection of
tracking failures}.
\newblock In: \bibinfo{booktitle}{{ICPR}}. \bibinfo{publisher}{{IEEE}};
\bibinfo{year}{2010},.
%Type = Inproceedings
\bibitem[{Wang et~al.(2019)Wang, Jabri and Efros}]{DBLP:conf/cvpr/WangJE19}
\bibinfo{author}{Wang\xfnm[ X.]}, \bibinfo{author}{Jabri\xfnm[ A.]},
\bibinfo{author}{Efros\xfnm[ A.A.]}.
\newblock \bibinfo{title}{Learning correspondence from the cycle-consistency of
time}.
\newblock In: \bibinfo{booktitle}{{CVPR}}. \bibinfo{publisher}{{IEEE}};
\bibinfo{year}{2019},.
%Type = Book
\bibitem[{Thrun et~al.(2005)Thrun, Burgard and Fox}]{DBLP:books/daglib/0014221}
\bibinfo{author}{Thrun\xfnm[ S.]}, \bibinfo{author}{Burgard\xfnm[ W.]},
\bibinfo{author}{Fox\xfnm[ D.]}.
\newblock \bibinfo{title}{Probabilistic robotics}.
\newblock Intelligent robotics and autonomous agents;
\bibinfo{publisher}{{MIT}}; \bibinfo{year}{2005}.
%Type = Inproceedings
\bibitem[{Karkus et~al.(2018)Karkus, Hsu and Lee}]{DBLP:conf/corl/KarkusHL18}
\bibinfo{author}{Karkus\xfnm[ P.]}, \bibinfo{author}{Hsu\xfnm[ D.]},
\bibinfo{author}{Lee\xfnm[ W.S.]}.
\newblock \bibinfo{title}{Particle filter networks with application to visual
localization}.
\newblock In: \bibinfo{booktitle}{CoRL}. \bibinfo{publisher}{{PMLR}};
\bibinfo{year}{2018},.
\end{thebibliography}