HighlightNet: Highlighting Low-Light Potential Features for Real-Time UAV Tracking
Extracted source text · arXiv 2208.06818
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
% 1 2 3 4 5 6 7 8
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4 paper
\IEEEoverridecommandlockouts % This command is only needed if
% you want to use the \thanks command
\overrideIEEEmargins % Needed to meet printer requirements.
%In case you encounter the following error:
%Error 1010 The PDF file may be corrupt (unable to open PDF file) OR
%Error 1000 An error occurred while parsing a contents stream. Unable to analyze the PDF file.
%This is a known problem with pdfLaTeX conversion filter. The file cannot be opened with acrobat reader
%Please use one of the alternatives below to circumvent this error by uncommenting one or the other
%\pdfobjcompresslevel=0
%\pdfminorversion=4
% See the \addtolength command later in the file to balance the column lengths
% on the last page of the document
% The following packages can be found on http:\\www.ctan.org
\usepackage{graphics} % for pdf, bitmapped graphics files
\usepackage{epsfig} % for postscript graphics files
%\usepackage{mathptmx} % assumes new font selection scheme installed
\usepackage{times} % assumes new font selection scheme installed
\usepackage{amsmath} % assumes amsmath package installed
\usepackage{amssymb} % assumes amsmath package installed
\usepackage{bm}
\usepackage{color}
\usepackage{cite}
\usepackage[linesnumbered,ruled]{algorithm2e}
\usepackage{ulem} %to strike the words
\usepackage[colorlinks=true,linkcolor=black,citecolor=blue,urlcolor=blue,]{hyperref}
\usepackage[doipre={doi:~}]{uri}
\usepackage{float}
\usepackage{booktabs}
\usepackage{multirow}
% \usepackage{ bbold }
\usepackage{mathrsfs}
\usepackage[utf8]{inputenc}
\usepackage[caption=false, font=footnotesize]{subfig}
%\usepackage{ulem}
%\usepackage{subfig}
\usepackage{graphicx}
\usepackage{pifont}
\newcommand{\xmark}{\text{\ding{55}}}
%\usepackage{subfigure}
\newcounter{RNum}
\renewcommand{\theRNum}{\arabic{RNum}}
\newcommand{\Remark}{\noindent\textit{\textbf{Remark}~\refstepcounter{RNum}\textbf{\theRNum}: }}
\usepackage{soul}
\newcommand{\NoOne}[1]{\textcolor{red}{#1}}
\newcommand{\NoTwo}[1]{\textcolor{green}{#1}}
\newcommand{\NoThree}[1]{\textcolor{blue}{#1}}
\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}
\soulregister\NoOne7
\soulregister\NoTwo7
\soulregister\NoThree7
\soulregister\Remark7
\usepackage{flushend}
\title{\LARGE \bf
HighlightNet: Highlighting Low-Light Potential Features \\ for Real-Time UAV Tracking
}
\author{Changhong Fu$^{*}$, Haolin Dong, Junjie Ye, Guangze Zheng, Sihang Li, and Jilin Zhao% <-this % stops a space
\thanks{$^{*}$Corresponding author}
\thanks{The authors are with the School of Mechanical Engineering, Tongji University, Shanghai 201804, China.
{\tt\small changhongfu@tongji.edu.cn}}%
}
%\setlength{\textfloatsep}{6pt}
%\setlength{\belowcaptionskip}{2.5pt} %???????????
%\setlength{\abovecaptionskip}{2.5pt} %???????????
%\setlength\abovedisplayskip{2pt}
%\setlength\belowdisplayskip{2pt}
\begin{document}
\maketitle
\thispagestyle{empty}
\pagestyle{empty}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
%UAV Tracking has so far made great promotion. Designed by imitating biological neural networks, state-of-the-art (SOTA) tracking methods based on convolutional networks, exemplified by Siamese-based trackers, towards tracking like human eyes. Recent developments show that their tracking performance almost surpasses humans in common scenarios. Nevertheless, such satisfactory performance can hardly maintain in low-light conditions where the objects can hardly be distinguished by naked eyes.
Low-light environments have posed a formidable challenge for robust unmanned aerial vehicle (UAV) tracking even with state-of-the-art (SOTA) trackers since the potential image features are hard to extract under adverse light conditions. Besides, due to the low visibility, accurate online selection of the object also becomes extremely difficult for human monitors to initialize UAV tracking in ground control stations. To solve these problems, this work proposes a novel enhancer, \textit{i.e.}, HighlightNet, to light up potential objects for both human operators and UAV trackers. By employing Transformer, HighlightNet can adjust enhancement parameters according to global features and is thus adaptive for the illumination variation. Pixel-level range mask is introduced to make HighlightNet more focused on the enhancement of the tracking object and regions without light sources. Furthermore, a soft truncation mechanism is built to prevent background noise from being mistaken for crucial features. Evaluations on image enhancement benchmarks demonstrate HighlightNet has advantages in facilitating human perception. Experiments on the public UAVDark135 benchmark show that HightlightNet is more suitable for UAV tracking tasks than other state-of-the-art (SOTA) low-light enhancers. In addition, real-world tests on a typical UAV platform verify HightlightNet's practicability and efficiency in nighttime aerial tracking-related applications. The code and demo videos are available at \url{https://github.com/vision4robotics/HighlightNet}.
%A UAV tracking task can be divided into two steps, \textit{e.g.}, target object selection and auto tracking. In low-light conditions, human operator can hardly make out contour of the target and make accurate selection. Moreover, even the current top-ranked trackers can hardly keep their performance in nighttime scenes. To address these problems, a Transformer-based adaptive pixel-level enhancer (HighlightNet) is proposed. HighlightNet consists of three novel modules, each one serves for several concrete issues of UAV nighttime tasks. To deal with small target and artificial light sources ,HighlightNet includes a pixel-level range mask to realize small object enhancement and improve brightness uniformity. To cope with the image noise in low-light conditions, a soft truncation is introduced. Furthermore, to deal with the high-speed scene switching and occlusion problems in UAV tracking applications, network parameters can be adjusted by a Transformer-based adaptive branch, due to its advantage on processing global information. Experiments on image enhancement data set demonstrate HighlightNet have advantages over state-of-the-arts methods quantitatively. Evaluations on the public UAVDark135 benchmark show that HighlightNet is more suitable for UAV tracking tasks than other top-ranked low-light enhancers. Real-world tests on a typical UAV platform have further verify the practicability of HighlightNet.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction} \label{sec:intro}
Object tracking has been widely used in a variety of sectors, most representative in UAV applications for target following~\cite{Cheng2017IROS}, autonomous landing~\cite{LAND}, and self-localization~\cite{Ye_2021_TIE}. SOTA tracking approaches~\cite{Li_2018_CVPR, Cao2021ICCV} have achieved outstanding performance under favorable illumination. However, the degradation of these trackers' performance in low-light conditions has been discussed in recent studies~\cite{li2021allday,Ye2021IROS}. On the one hand, as shown in Fig.~\ref{fig:fig1}, human operators can hardly make an accurate initial annotation for trackers. On the other hand, due to the quick mobility of both the UAV and the tracked object, as well as numerous challenges such as occlusion, noise, illumination variation, robust and precise tracking in nighttime conditions has remained a difficult task. An effective approach to light up the target object for both human operators and trackers is in dire need.
%As previously mentioned, one key issue of current trackers is the practicability during the night or in other low-light environments. existing studies generally dedicate to designing tracking approaches for normal-illumination conditions while neglecting tracking robustness in the dark.
%The shortcomings of nighttime tracking can be mitigated by performing a low-light enhancer before visual tracking, which prompted us to propose a plug-and-play enhancer HighlightNet for UAV tracking. Moreover, HighlightNet will also light up the night for human operators to achieve higher accuracy of initial target selection. In consideration of the complexity of visual information captured by UAVs and the scarce computational resources, the enhancer must be lightweight, adaptive, and most significantly, capable of highlighting useful features hiding in the nighttime shadow.
A low-light enhancer can be a solution to this problem. However, enhancement of objects in dark areas may lead to overexposure problems in regions under favorable illumination, which may cause the destruction of potential features. Besides, artificial light sources are omnipresent in low-light conditions, which divided the image into regions with different illumination. In UAV object tracking, this problem is especially crucial because of UAVs' wide vision field. When the target enters over-exposed or extreme dark regions, it is more difficult for the tracker to recognize the object. In this case, this work proposes a range mask to distinguish the enhancement of regions with or without artificial light sources. Moreover, experiments on typical nighttime UAV scenes show that the enhancer can focus on the enhancement of tracking objects according to the mask.
%To improve the robustness of trackers in dark scenes, DarkLighter
%visibility of low-light conditions,
%to light up the darkness for CNN-based trackers
\begin{figure}[!t]
\centering
\includegraphics[width=0.98\linewidth]{images/fig1.pdf}
\setlength{\abovecaptionskip}{-12pt}
\caption
{
% Tracking performance comparison in a typical dark scene with the proposed DarkLighter module activated (in \textcolor{red}{red}) or not (in \textcolor[RGB]{255, 105, 180}{pink}). The curves between estimated locations and ground truth bounding boxes are also exhibited. DarkLighter raises tracking robustness in low-light conditions remarkably.
Comparison of the overall tracking results from the baseline tracker with HighlightNet (in \textcolor{red}{red}) or not (in \textcolor{blue}{blue}). Operators may make an inaccurate selection due to the low illumination. With the help of HighlightNet, human operators can conduct an accurate selection of targets, and the SOTA tracker can achieve promising results, as also shown in center location error (CLE) curves.
}
\vspace{-12pt}
\label{fig:fig1}
\end{figure}
Only highlighting tracking objects and other effective local features is not enough. In nighttime UAV applications, it is inevitable that noises will damage the structural details of images and even be mistaken for tracking objects. Moreover, with excessive background information in UAVs' wide vision field, more background noises are also introduced. However, existing low-light enhancers can hardly distinguish the noise from effective features. To avoid over-enhancement of noise, we consequently design a dark area truncation function, which produces an anti-noise mask to filter unwanted noise.
Apart from effective local features, global information is also essential for low-light enhancers. Due to the high mobility of UAVs, rapid changes of scenarios can hardly be avoided, which may cause severe global illumination variation. Nevertheless, most low-light enhancers~\cite{LIME,RetinexNet,Zhang2019ACM,EnlightenGAN,Guo_2020_CVPR} are designed for stable photography scenes. Without adjustment mechanisms, their effectiveness is unsatisfying while dealing with illumination variation. To address this problem, Transformer~\cite{vaswani2017nips} is introduced to dynamically regulate parameters according to global illumination information.
%In the nighttime environment, the images captured by UAV cameras are usually of low brightness, so the object outline is not clear or even invisible. This may not only prevent the operator from selecting templates correctly, but also makes the tracker's feature extractors trained with daytime images ineffective. Without discriminative object features, the robustness of the tracker will be significantly reduced. Moreover, in low-light condition, the high-level noise will further damage the structural details of images and artificial light sources will lead to obvious difference in brightness of each part of the image.
%In UAV nighttime tracking tasks, artificial light sources and small target object are avoidless. To improve brightness uniformity and realize pixel-level enhancement, HighlightNet includes an enhancement range mask obtained by a lightweight convolutional neural network (CNN). The mask can distinguish the brightness increase of regions with or without artificial light sources. As the mask corresponds to the original image pixel by pixel, it can easily cope with small size enhancement. Moreover, to get rid of the high cost involved in collecting sufficient paired data for UAV conditions, HighlightNet can be trained with unpaired training data.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Another difficulty of UAV image enhancement is the limitation of computational resources. In consideration of the take-off weight, UAVs can hardly carry a high-power computing platform. Approaches with a high computation complexity have difficulty being processed in real-time on UAVs. To improve the computing efficiency, the main workflow of HighlightNet is based on the gray channel rather than the RGB channels. In addition, for supervised training, most low-light enhancers use paired data. In order to reduce the high expense of gathering enough paired data for UAV conditions, HighlightNet is designed to be trained with unpaired data.
The contributions of this work are summarized as follows:
\begin{itemize}
\item An adaptive enhancer HighlightNet is constructed to facilitate both online object selection and UAV tracking in low-light conditions. %As a result, trackers equipped with the proposed enhancer can track with the reflectance of the object, which is assumed to be invariant to illumination.
\item The range mask and the anti-noise mask are designed to highlight the object and filter unwanted noise.%Trained with general captured images, the proposed enhancer generalized well to various illumination conditions in aerial tracking.
\item A dynamic parameter adjustment method based on an efficient Transformer is proposed to process global features for illumination retouching.
%are conducted with several SOTA trackers
\item Comprehensive evaluation on image enhancement test sets, UAV nighttime tracking benchmarks, and real-world experiments demonstrate that HighlightNet has advantages in facilitating human operator perception and improving trackers' nighttime performance.
\end{itemize}
\section{Related Works}
\subsection{UAV Tracking}
%\subsection{CNN for Low-light Image Enhancement}
%conventional data-driven
Object tracking methods include correlation filter-based approaches~\cite{Danelljan2017CVPR,zheng2021ICRA} and convolutional neural network (CNN)-based approaches~\cite{LiBo_2019_CVPR, Xu_2020_AAAI, Cao2021ICCV}. Among them, due to the promising tracking performance, methods based on Siamese networks have been widely used in UAV tracking. SiamAPN~\cite{Fu_2021_TGRS} is a Siamese-based no-prior two-stage method for adaptive anchor proposing. \cite{Li_2018_CVPR,LiBo_2019_CVPR,Ren2017tpami} employ the region proposal network (RPN) as anchor-based trackers to further improve tracking accuracy. HiFT~\cite{Cao2021ICCV} introduces a feature Transformer network to obtain hierarchical similarity maps. Ad$^2$Attack~\cite{AD2} proposes a novel adaptive adversarial attack approach to help increase awareness of the potential risk. TCTrack~\cite{TCT} designs a comprehensive framework to fully utilize temporal contexts for UAV tracking. In spite of their high accuracy in daytime conditions, their performance is unsatisfying in low-light conditions. Damage to potential features in nighttime conditions makes their robustness drop greatly. What's worse, precise online target selection is nearly impossible in extreme low-light conditions. Therefore, image enhancement technology before tracking is urgently needed.
\subsection{Low-light Image Enhancement}
The theory of Retinex~\cite{Edwin_1977_retinex} has evolved into a variety of low-light enhancement approaches. LIME~\cite{LIME} estimates a coarse illumination map by extracting the maximum of each pixel in RGB channels, which is subsequently refined by a structure prior. Based on Retinex, convolutional neural networks (CNNs) are introduced and remarkably improve low-light performance~\cite{RetinexNet, Zhang2019ACM}. The first CNN model LL-Net \cite{lore2017llnet} employs an autoencoder to learn denoising and light enhancement simultaneously. DeepUPE \cite{DUPE} introduces intermediate illumination to connect the input and the anticipated enhancement result. The method in~\cite{LEARN} takes noise into consideration and realizes satisfying performance in enhancing practical noisy low-light images. However, because of the internal locality of convolution operations, CNNs are not suitable for processing global features.
Another disadvantage of most CNN-based methods is the requirement of paired data for supervised training. In consideration of the high investment caused by collecting sufficient paired data for UAV conditions, low-light enhancement by paired data is impractical. Therefore, advanced methods turn to unsupervised training. EnlightenGAN~\cite{EnlightenGAN} employs a generative adversarial network (GAN) so it can be trained using adversarial loss as an attention-based U-Net. Zero-DCE~\cite{Guo_2020_CVPR} employs image-specific curve estimation and has high efficiency, also trained with unpaired data. RUAS~\cite{RUAS} can also be trained with unpaired data by employing a cooperative reference-free learning strategy. Nevertheless, in low-light conditions, the performance of SOTA enhancers in target object selection and target tracking can hardly meet our satisfaction. This can be partly attributed to the neglect of potential effective features.
\subsection{Low-light UAV Tracking}
Though UAV tracking tasks in nighttime scenes are crucial, only several approaches~\cite{Ye2021IROS, SCT, li2021allday,Ye2022CVPR} to improve UAV tracking performance in low-light conditions have been proposed. ADTrack~\cite{li2021allday} serves for correlation filter (CF) trackers, while Darklighter~\cite{Ye2021IROS} and SCT~\cite{SCT} are mainly designed for trackers based on Siamese network. Darklighter~\cite{Ye2021IROS} constructs a lightweight map estimation network to cope with poor illumination and noise. Applying task-tailored training and a Transformer structure, SCT~\cite{SCT} realizes stable nighttime tracking. The basic idea of these three methods is to employ a plug-and-play enhancer as a preprocessing step for UAV trackers. UDAT~\cite{Ye2022CVPR} instead considers the nighttime tracking problem as an unsupervised domain adaptation task. Despite the progress, existing approaches generally ignore the improvement of human perception, which plays a vital role in the step of online object selection. Moreover, without adjustment mechanisms and denoising modules, these methods can hardly deal with illumination variation and background noises.
\begin{figure*}[!t]
%\centering
\includegraphics[width=0.98\linewidth]{images/fig2.pdf}
\setlength{\abovecaptionskip}{-2pt}
\caption
{
Overview of our HighlightNet pipeline. Given a low-light image patch, HighlightNet includes three novel modules, \textit{i.e.}, per-pixel low-light enhancement, Transform-based parameter adjustment, and soft truncation, to highlight potential features for both human operators and trackers with slight computational consumption. The bottom row serves for template selection by humans.
}
% \vspace*{-12pt}
\label{fig:main}
\end{figure*}
\section{Methodology}
This work proposes a Transformer-based adaptive enhancer called HighlightNet to highlight potential features for low-light conditions. As shown in Fig.~\ref{fig:main}, to serve both human operators and UAV trackers, the template patch and search patch are processed separately on ground control stations and UAVs. The proposed HighlightNet includes three novel modules. The Transformer-based parameter adjustment module obtains the constraint $\alpha$ and the truncation threshold $\beta$ by processing the downsampled gray-scale image. In the enhancement module, a range mask is constructed by a convolutional neural network (CNN). The mask is fused with constraint $\alpha$ by a fusion function to adapt the enhanced curve. As the mask corresponds to the original image pixel by pixel, each pixel has a unique enhancement range. The truncation threshold $\beta$ acts on the soft truncation module. By the truncation function, the gray-scale image is translated to the anti-noise image. To improve computational efficiency, the main enhancement workflow is based on the gray-scale image. Therefore, the color information is first translated to channel proportion information and stored.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Per-Pixel Low-Light Enhancement}
To focus on the enhancement of objects, this module is mainly designed for predicting the accurate enhancement for each pixel. Therefore, the range mask $\mathbf{M}_{i} \in \mathbb{R}^{1 \times H_{i} \times W_{i}}$ is with the same resolution as the input gray-scale image $\mathbf{G}_{i} \in \mathbb{R}^{1 \times H_{i} \times W_{i}}$. We employ a CNN with symmetrical concatenation to obtain the range mask $\mathbf{M}_{i}$. It is constructed by seven convolutional layers. In each layer, the ReLU activation function is followed by 4 convolutional kernels of size $3\times 3$. The sigmoid activation function is then applied to the final layer and produces the range mask $\mathbf{M}_{i}$. The range mask $\mathbf{M}_{i}$ is then fused with constraint $\alpha$ by the fusion function to fit the enhanced curve.
As a pixel-level operation, the enhanced curve has a great influence on the computational efficiency of HighlightNet. Considering the scarce computational resources of UAV platforms, gamma transform is employed as the enhanced curve. It can realize large-scale enhancement without iteration. A standard gamma transform can be expressed as:
\begin{equation}\label{equ:gamma}
\mathbf{O}_{i}\left ( \textit{r,c} \right )= \mathbf{G}_{i}\left ( \textit{r,c} \right )^{\gamma_{i} \left ( \textit{r,c} \right ) }, \, 0\leqslant \textit{r}< H_{i} \, ,\, 0\leqslant \textit{c}< W_{i}
\quad,
\end{equation}
where $\mathbf{G}_{i}\left ( \textit{r,c} \right )$ is the value of each pixel in gray-scale image $\mathbf{G}_{i}$, $\gamma \left ( \textit{r,c} \right )$ is the output of the fusion function, $\mathbf{O}_{i}\left ( \textit{r,c} \right )$ is the output of this enhancement module, $H_{i}$ and $W_{i}$ are the height and width of the image. Each pixel of the gray-scale image $\mathbf{G}_{i}$ is normalized to [0,1] before calculation. After designating the enhanced curve, the fusion function can be designed.
The fusion function can change the codomain of each pixel's value in the range mask $\mathbf{M}_{i}$. Because of the sigmoid activation function, the value of pixels in the range mask $\mathbf{M}_{i}$ is limited between 0 and 1. However, for the enhanced curve defined in Eq.~(\ref{equ:gamma}), the input $\gamma \left ( \textit{r,c} \right )$ should be between the constraint $\alpha$ and 1. Therefore, the fusion function can be expressed as:
\begin{equation}\label{equ:fusion}
\gamma_{i}\left ( \textit{r,c} \right )= \alpha_{i} ^{\mathbf{M}_{i}\left ( \textit{r,c} \right ) }, \, 0\leqslant \textit{r}< H_{i} \, ,\, 0\leqslant \textit{c}< W_{i}
\quad,
\end{equation}
where $\mathbf{M}_{i}\left ( \textit{r,c} \right )$ are the value of each pixel in the range mask $\mathbf{M}_{i}$, $\alpha$ is the constraint obtained by the Transformer-based branch, $\gamma \left ( \textit{r,c} \right )$ is the input of the enhanced curve Eq.~(\ref{equ:gamma}). The fusion function and the enhanced curve are not fixed, they can be changed to adapt to different tasks. The reason why we choose gamma transform is mainly for computational efficiency.
\begin{figure}[!b]
\centering
\includegraphics[width=0.98\linewidth]{images/fig3.pdf}
\setlength{\abovecaptionskip}{-2pt}
\caption
{
Three examples of the pixel-wise range mask. (a) denotes original image. (b) denotes range mask, represented by heatmaps. For visualization, we select a particular region of the image and normalize the values to the range of [0, 1]. The range mask conducts HighlightNet to focus on the enhancement of tracking objects and regions without light resources.
}
\label{fig:mask}
\vspace{-4pt}
\end{figure}
Three examples of the range mask are presented in Fig.~\ref{fig:mask}. As shown, the range mask makes the enhancer focus on the enhancement of the potential objects. Moreover, the enhanced result exposes the features in shadow and maintains the areas with artificial light sources, which improves the brightness uniformity. Since the shadow is illuminated by HighlightNet, the tracker won't lose the object when it enters the dark areas without light resources.
\Remark The three masks in Fig.~\ref{fig:mask} are small regions cut from the high-resolution original mask. Therefore, the objects in these small masks can be considered as tiny targets. As shown in Fig.~\ref{fig:mask}, with the help of the range mask, HighlightNet is able to adjust the enhancement range at pixel level and realize the targeted enhancement of small objects. Moreover, as shown in Fig.~\ref{fig:fig4}, HighlightNet can help the base tracker recognize small objects in low-light conditions.
\subsection{Transformer-Based Parameter Adjustment}
The main function of this module is producing constraint $\alpha$ and truncation threshold $\beta$ by processing global features. As mentioned in the last section, constraint $\alpha$ is fused with the enhanced range mask to obtain the actual enhancement of each pixel. Therefore, this branch has a similar influence on all pixels. Since global information plays a more important role than local context, Transformer~\cite{vaswani2017nips} is introduced. DETR~\cite{DETR} has shown the ideal efficiency of Transformer in processing global information in computer vision tasks. With the advantage of global processing, this module can dynamically regulate parameters to adapt to illumination variation caused by the high mobility of UAVs.
In this global processing module, the resolution of input does not play an important role. We find that the computation resource can be greatly saved without performance degradation by setting the resolution of input to $32 \times 32$. Therefore, as shown in Fig.~\ref{fig:main}, the input gray-scale image $\mathbf{G}_{i} \in \mathbb{R}^{1 \times H_{i} \times W_{i}}$ is first identically downsampled to a low-resolution image $\mathbf{L}_{i} \in \mathbb{R}^{1 \times 32 \times 32}$. After processing by a convolutional layer, it is in the size of ${16 \times 16 \times 16}$ and then reshaped to ${16 \times (16 \times 16)}$ to enter the encoder of Transformer. Several identical layers make up the encoder and then two sublayers make up each layer. Each layer consists of two sub-layers. The first layer uses the concept of Multi-Head Attention \cite{dosovitskiy2020image} (MHA) to drive the model to concentrate on information from various positions and representation subspaces. MHA is defined as:
\begin{equation}
\label{mha}
{\mathrm{MHA}}( {\mathbf{Q}}, {\mathbf{K}}, {\mathbf{V}}) = {\mathbf{Cat}}( \mathrm{head_1}, \mathrm{head_2}, \cdots, \mathrm{head}_h){\mathbf{W}}^{\mathbf{O}}\ ,
\end{equation}
where ${\mathbf{Q}}, {\mathbf{K}}, {\mathbf{V}} \in \mathbb{R}^{n \times d_m}$ are input embedding matrices, $n$ is sequence length, $d_m$ is the embedding dimension, and $h$ is the number of heads. Each head is defined as:
\begin{equation}
\label{att}
\begin{split}
\mathrm{head}_j = {\mathrm{softmax}} \left[\frac{\hat{\mathbf{Q}} \hat{\mathbf{K}}^{\rm T}}{\sqrt{{\mathbf{d_K}}}} \right]\hat{\mathbf{V}},\\
\end{split}
\end{equation}
where $\mathbf{d_K}, \mathbf{d_V}$ are the hidden dimensions of the projection subspaces. $\hat{\mathbf{Q}},\hat{\mathbf{K}},\hat{\mathbf{V}}$ are query, key ,and value projected into the subspaces respectively, which can be defined as:
\begin{equation}
\label{att1}
\hat{\mathbf{Q}}={\mathbf{QW}}_j^{\mathbf{Q}},
\hat{\mathbf{K}}={\mathbf{KW}}_j^{\mathbf{K}},
\hat{\mathbf{V}}={\mathbf{VW}}_j^{\mathbf{V}},
\end{equation}
where ${\mathbf{W}}_j^{\mathbf{Q}},{\mathbf{W}}_j^{\mathbf{QK}} \in \mathbb{R}^{\mathbf{d_m} \times \mathbf{d_K}},{\mathbf{VW}}_j^{\mathbf{V}} \in \mathbb{R}^{\mathbf{d_m} \times \mathbf{d_V}}$ are learned matrices.
The second sub-layer is a fully connected feed-forward network (FFN). The connection between each sub-layer is a residual module followed by layer normalization. Therefore, the output of each sub-layer can be expressed as:
\begin{equation}\label{equ:transformer}
\begin{split}
\mathbf{F}'= \mathrm{LN}\left ( \mathrm{MHA}\left ( \mathbf{F} \right )+\mathbf{F} \right )\quad,\\
\mathbf{F}''= \mathrm{LN}\left ( \mathrm{FFN}\left ( \mathbf{F}' \right )+\mathbf{F}' \right )\quad,\\
\end{split}
\end{equation}
where $\mathbf{F}'$ and $\mathbf{F}''$ denote the output of MHA and FFN, respectively. LN is layer normalization. The output of Transfomer is in the same size as input ${16 \times (16 \times 16)}$. In order to obtain two adaptive parameters, a simple fully connected layer(FC) followed by the sigmoid activation function is introduced to reshape the output to a bivector [$\alpha$,$\beta$]. Each dimensionality in this bivector is then translated to constraint $\alpha$ and truncation threshold $\beta$ by the parameter transform module, which is a simple linear conversion.
\begin{figure}[!t]
\centering
\includegraphics[width=0.98\linewidth]{images/fig4.pdf}
\setlength{\abovecaptionskip}{-2.5pt}
\caption
{
Comparison of confidence maps generated by the SOTA tracker with and without HighlightNet. The tracked objects are shown by the \textcolor{green}{green} boxes in the first column. The sequence names are provided in the top left corner of the original frames from the public UAVDark135 benchmark. Without HighlightNet, the base tracker loses its capacity to recognize objects in low-light conditions.
}
\label{fig:fig4}
\vspace{-0.4cm}
\end{figure}
\subsection{Soft Truncation}
In this module, a soft truncation function $\mathbf{T}\left ( \mathbf{G} \right )$ is proposed to filter useless features brought by over-enhanced noise. The intensity of enhancement weakening is negatively correlated with the gray value. Moreover, considering the limitation of computational resources on the UAV platform, this function should be easy to compute. Therefore, a cubic function determined by two parameters is employed, which can be expressed as:
\begin{equation}\label{equ:truncation}
\begin{split}
\mathbf{T}_{i}\left (\textit{r,c} \right )= -\tau \times \left ( \beta _{i}\, - \mathbf{G}_{i}\left ( \textit{r,c}\right )\right )^{3},\; \tau \beta _{i}^{3}\mathbf{T}_{i}\left (\textit{r,c} \right )< 0 ,
\\ 0\leqslant \textit{r}< H_{i} \, ,\, 0\leqslant \textit{c}< W_{i},
\end{split}
\end{equation}
where $\mathbf{G}_{i}\left ( \textit{r,c} \right )$ is the value of each pixel in gray-scale image $\mathbf{G}_{i} \in \mathbb{R}^{1 \times H_{i} \times W_{i}}$, $ \beta _{i}$ is the threshold of truncation, $\tau$ determines the reducing range. $\mathbf{T}_{i}\left (\textit{r,c} \right )$ is the output of the truncation function with the codomain of $\left [\tau \beta _{i}^{3},0 \right )$. Therefore, the value of the output anti-noise mask is subtractive, which will apply to the pixel-wise addition module to reduce the enhancement range of dark area noise.
\Remark The threshold $ \beta _{i}$ of truncation is an adjustable parameter produced by the parameter adjustment module. Therefore, the result of soft truncation is not only based on the input grey-scale image but also influenced by the global features acting on Transformer. Threshold $ \beta _{i}$ increases in extreme low-light conditions to filter more useless features brought by noise.
For the same purpose, a non-reference loss function of dark area noise $\mathcal{L}_{\mathrm{dan}}$ is introduced. Since severe noise problems always appear in dark regions of the image, this loss function limits the enhancement range of shadows and night sky. The enhanced image is first divided into local regions in the size of ${16 \times16}$. The average enhancement range of pixels in the dark area of each region is then calculated. Therefore, $\mathcal{L}_{\mathrm{dan}}$ can be expressed as:
\begin{equation}\label{equ:lossdan}
\begin{split}
\mathcal{L}_{\mathrm{dan}}= \frac{1}{N}\sum_{i}^{N}\mathbf{A}_{i}\quad,
\end{split}
\end{equation}
where $N$ is the number of regions, $\mathbf{A}_{i} $ is the average enhancement range of pixels in the dark area of each region. The threshold to determine whether the pixel is in a dark area is set to 0.04. This threshold is half of the average of $ \beta _{i}$. Because the impact of soft truncation becomes obvious from this threshold. Except for the $\mathcal{L}_{dan}$, other three non-reference losses are employed to further improve the performance of HighlightNet. The total loss can be expressed as:
\begin{equation}\label{equ:loss}
\begin{split}
\mathcal{L}_{\mathrm{total}}= \lambda _{1}\mathcal{L}_{\mathrm{dan}}+\mathcal{L}_{\mathrm{spa}}+\lambda _{2}\mathcal{L}_{\mathrm{exp}}+\lambda _{3}\mathcal{L}_{\mathrm{tv}}\quad,
\end{split}
\end{equation}
where $\mathcal{L}_{\mathrm{dan}}$ is dark area noise loss, $\mathcal{L}_{\mathrm{spa}}$ is spatial consistency loss, $\mathcal{L}_{\mathrm{exp}}$ is exposure control loss, $\mathcal{L}_{\mathrm{tv}}$ is the illumination smoothness loss, $\lambda_1$, $\lambda_2$, and $\lambda_3$ are the weights of losses. $\mathcal{L}_{\mathrm{spa}}$, $\mathcal{L}_{\mathrm{exp}}$, and $\mathcal{L}_{\mathrm{tv}}$ are first proposed in~\cite{Guo_2020_CVPR} as non-reference losses. Therefore, HighlightNet can be trained with unpaired data, which avoid the high cost of preparing a paired dataset for UAV tracking tasks.
% \begin{figure*}[!t]
%\centering
% \includegraphics[width=0.45\linewidth]{images/IV.pdf}
% \includegraphics[width=0.45\linewidth]{images/IV.pdf}
% \caption
% {
% Attribute-based analysis.
% }
% \label{fig:IV}
%\vspace{-0.4cm}
%\end{figure*}
% % Table generated by Excel2LaTeX from sheet 'Sheet1'
% \begin{table*}[!t]
% \centering
% \scriptsize
% \caption{Efficacy of DarkLighter on SOTA trackers. Success rate and precision of different trackers with DarkLighter enabled or not is represented in the third row. Gains after DarkLighter activated are reported in the last row.}
% %\renewcommand\tabcolsep{2.pt}
% %\resizebox{0.98\linewidth}{!}{
% \begin{tabular}{ccccccccccccccccccccccccc}
% \toprule
% Trackers & \multicolumn{2}{c}{SiamAPN} & \multicolumn{2}{c}{SiamRPN++} & \multicolumn{2}{c}{DiMP50} & \multicolumn{2}{c}{PrDiMP50} & \multicolumn{2}{c}{PrDiMP18}\\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9} \cmidrule(lr){10-11}
% DarkLighter & \xmark & \checkmark & \xmark & \checkmark & \xmark & \checkmark & \xmark & \checkmark & \xmark & \checkmark \\
% \cmidrule(lr){2-2} \cmidrule(lr){3-3} \cmidrule(lr){4-4} \cmidrule(lr){5-5} \cmidrule(lr){6-6} \cmidrule(lr){7-7} \cmidrule(lr){8-8} \cmidrule(lr){9-9} \cmidrule(lr){10-10} \cmidrule(lr){11-11}
% Succ./Prec. & 0.290/0.413 & 0.394/0.521 & 0.372/0.474 & 0.385/0.495 & 0.526/0.672 & \textbf{0.544/0.700} & 0.501/0.640 & 0.507/0.657 & 0.469/0.595 & 0.469/0.601 \\
% Gain (\%) & \multicolumn{2}{c}{\textbf{+35.86/+26.15}} & \multicolumn{2}{c}{+3.49/+4.43} & \multicolumn{2}{c}{+3.42/+4.17} & \multicolumn{2}{c}{+1.20/+2.66} & \multicolumn{2}{c}{0.00/+1.01} \\
% \bottomrule
% \end{tabular}%}
% \label{tab:overall}%
% \end{table*}
\section{Experiment}
To testify the effectiveness and robustness of HighlightNet for both online object selection and UAV object tracking, our experiments can be divided into two parts. For the template selection task, we quantitatively testify the performance of HighlightNet on the Part2 subset of SICE dataset~\cite{Cai2018TIP} and compare HighlightNet with SOTA low-light enhancers, respectively Darklighter~\cite{Ye2021IROS}, EnlightenGAN~\cite{EnlightenGAN}, Zero-DCE~\cite{Guo_2020_CVPR}, and LIME~\cite{LIME}. As a preprocessing step for UAV target tracking, its success rate and precision on UAVDark135 benchmark with baseline are compared with other SOTA low-light enhancers to testify its advantage on UAV nighttime tracking tasks. Performance on sequences with different attributes in UAVDark135 is tested to verify its ability to deal with specific UAV challenges. Moreover, to demonstrate its universality for different UAV trackers, it has been implemented on 4 SOTA trackers, \textit{i.e.}, SiamAPN++~\cite{Cao2021IROS}, SiamAPN~\cite{Fu_2021_TGRS}, HiFT~\cite{Cao2021ICCV}, and SiamRPN++~\cite{LiBo_2019_CVPR}. To show the function of each module, an ablation study is also introduced. Finally, we conduct the real-world test by adopting HightlightNet on a typical UAV platform to verify its applicability.
%To testify the effectiveness and robustness of DarkLighter, it is implemented on 4 SOTA trackers, \textit{i.e.}, DiMP50~\cite{Bhat_2019_ICCV}, PrDiMP50~\cite{Danelljan_2020_CVPR}, SiamAPN~\cite{SiamAPN}, and SiamRPN++~\cite{LiBo_2019_CVPR}. Comprehensive experiments are conducted on UAVDark135~\cite{2021arXiv210108446L}---the only UAV dark tracking benchmark so far. In UAVDark135, there are 135 challenging sequences of various objects (person, car, building, \textit{etc.}) captured in poor illumination, with over 125K frames in total, where most dark tracking scenes are involved.
%Moreover, DarkLighter is further compared with SOTA low-light enhancers, respectively EnlightenGAN~\cite{EnlightenGAN}, Zero-DCE~\cite{Guo_2020_CVPR}, and LIME~\cite{LIME}, to demonstrate its advantages for UAV dark tracking. In the end, DarkLighter is further ported to a typical UAV platform to evaluate its applicability and efficiency.
%PrDiMP18,
%\Remark All parameters and models of the approaches involved in the experiment remain their official releases without any tuning.
%DiMP50 and PrDiMP50 denote the corresponding approaches employ ResNet-50~\cite{He_2016_CVPR} as backbone, while the backbone in SiamAPN and SiamRPN++ is AlexNet~\cite{Krizhevsky2012nips}.
\subsection{Implementation Details}
To bring the capability of the illumination-based adaptive parameter adjustment into full play, the unpaired training set includes both low-light and over-exposed images. Therefore, images from Part1 of the SCIE dataset~\cite{Cai2018TIP} are employed to train our network. The training images are resized to 512×512. The weights $\lambda_1$, $\lambda_2$, and $\lambda_3$ are set to 200, 50, and 20 respectively, to balance the scale of losses. We implement our framework with PyTorch on a PC with an Intel i9-9920X CPU, an NVIDIA TITAN RTX GPU, and 32GB RAM. The batch size is set to 8, with a total of 100 epochs. We employ the ADAM optimizer and set the learning rate to 0.001. Other parameters of the optimizer are default. Finally, to confirm the viability of HighlightNet in nighttime UAV tracking, the real-world test uses an NVIDIA Jetson AGX Xavier, which is widely applied on UAV platforms.
%\Remark To further alleviate the impact of various illumination conditions, the dataset consists of multi-exposure images is employed to train ME-Net, which further ensuring the generalization performance of DarkLighter in variation illuminations and promoting the ability of trackers to meet illumination variation challenges.
%Different from current evaluation metrics in image enhancement,
\subsection{Evaluation Metrics}
To verify HighlightNet’s advantage on the nighttime online object selection task, we perform quantitative experiments on the Part2 subset of SCIE dataset~\cite{Cai2018TIP}, which includes 229 multi-exposure sequences. To testify its performance in low-light conditions, we choose the first low-light image in all 229 sequences and resize them to a size of 960×640×3. Finally, we obtain 229 paired low/normal light images. We choose the Part2 subset of the SCIE dataset for it contains numerous outdoor sequences and is primarily designed for low-light enhancement.
We also use visual tracking evaluation measures to rate the performance because HighlightNet is also targeting nighttime UAV tracking. The studies follow the one-pass evaluation method~\cite{Mueller2016ECCV}, which uses two metrics: precision and success rate. The precision is calculated by measuring the CLE between the estimated position and the ground truth position. And the success rate is calculated using the intersection over union (IoU) between the estimated bounding box and the ground truth.
\subsection{Efficacy of HighlightNet for Online Target Selection}
Since target selection is mainly executed by human operators, human perception should be evaluated quantificationally to testify the efficacy of HighlightNet. Therefore, the peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) are employed. TABLE~\ref{tab:psnr} reports the comparison result of HighlightNet with other SOTA low-light enhancers. HighlightNet achieves the highest PSNR and SSIM score on the Part2 subset of the SICE dataset~\cite{Cai2018TIP}. Darklighter is specially designed for trackers and thus its performance is unsatisfying in the test for human perception. Since initial target selection is inevitable, HightlightNet is more practical in nighttime UAV applications.
\begin{table}[htbp]
\centering
\vspace*{-2pt}
\caption{Comparison of HighlightNet with other SOTA enhancers in terms of full-reference image quality assessment metrics. The best result is in \NoOne{red} whereas the second best one is in \textcolor{green}{green}. HighlightNet has an obvious advantage on facilitate human perception.}
\resizebox{0.95\linewidth}{!}{
\begin{tabular}{cccccc}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{Method} }& \multicolumn{1}{c}{Darklighter} & \multicolumn{1}{c}{LIME} & \multicolumn{1}{c}{Zero-DCE} & \multicolumn{1}{c}{EnlightenGAN} & \multicolumn{1}{c}{\textbf{HighlightNet}} \\
\multicolumn{1}{c}{ } &\multicolumn{1}{c}{~\cite{Ye2021IROS}}&\multicolumn{1}{c}{~\cite{LIME}}&\multicolumn{1}{c}{~\cite{Guo_2020_CVPR}}&\multicolumn{1}{c}{~\cite{EnlightenGAN}}&\multicolumn{1}{c}{\textbf{(ours)}} \\
\midrule
PSNR$\uparrow$ & 8.3 & \textcolor{green}{\textbf{11.4}} & 8.9 & 10.8 & \textcolor[rgb]{ 1, 0, 0}{\textbf{11.8}} \\
SSIM$\uparrow$ & 0.57 & 0.70 & 0.63 & \textcolor{green}{\textbf{0.72}} & \textcolor[rgb]{ 1, 0, 0}{\textbf{0.75}} \\
\bottomrule
\end{tabular}}%
\vspace{-12pt}
\label{tab:psnr}%
\end{table}%
\begin{figure*}[!t]
\centering
\includegraphics[width=0.49\linewidth]{images/overlap_OPE.pdf}
\includegraphics[width=0.49\linewidth]{images/error_OPE.pdf}
\setlength{\abovecaptionskip}{-4pt}
\caption
{
Precision and success plots of SOTA trackers with HighlightNet enabled or not. The results with HightlightNet are in \textbf{bold} type. HighlightNet improves the performance of all involved trackers.
}
\label{fig:all}
\vspace{-0.4cm}
\end{figure*}
\Remark HighlightNet not only has a better performance on the object selection task but also be system-friendly in terms of computational complexity, for the design of HighlightNet takes the computational cost into account heavily.
\begin{table}[b!]
\scriptsize
\centering
\vspace{-12pt}
\caption{Comparison of HighlightNet with SOTA enhancers on UAVDark135. The first ,second, and third best results are in \NoOne{red}, \textcolor{green}{green}, and \textcolor{blue}{blue} font, respectively. $\Delta_s$ and $\Delta_p$ represent the percentages of success rate and precision exceeding the baseline, \textit{i.e.}, original tracker.}
\begin{tabular}{ccccc}
\toprule
Method & \multicolumn{1}{c}{\, Succ. \,} & \multicolumn{1}{c}{\, Prec. \,} &$\Delta_s$ (\%) & $\Delta_p$ (\%) \\
\midrule
Origin & 0.336 & 0.428 & - & - \\
\midrule
+LIME~\cite{LIME} & 0.415 & 0.518 & 23.5 & 21.0 \\
\midrule
+Darklighter~\cite{Ye2021IROS} & 0.415 & \textcolor{blue}{\textbf{0.529}} &23.5 & \textcolor{blue}{\textbf{23.8}} \\
\midrule
+EnlightenGAN~\cite{EnlightenGAN} & \textcolor{blue}{\textbf{0.418}} & 0.522 & \textcolor{blue}{\textbf{24.4}} & 22.0 \\
\midrule
+Zero-DCE~\cite{Guo_2020_CVPR} & \textcolor{green}{\textbf{0.419}} & \textcolor{green}{\textbf{0.530}} & \textcolor{green}{\textbf{24.7}} & \textcolor{green}{\textbf{23.8}} \\
\midrule
{\textbf{+HighlightNet (ours)}} &\textcolor[rgb]{ 1, 0, 0}{\textbf{0.424}} & \textcolor[rgb]{ 1, 0, 0}{\textbf{0.539}} &\textcolor[rgb]{ 1, 0, 0}{\textbf{26.2}} & \textcolor[rgb]{ 1, 0, 0}{\textbf{25.9}} \\
\bottomrule
\end{tabular}
\vspace{-12pt}
\label{tab:compare1}
\end{table}
% Table generated by Excel2LaTeX from sheet 'Sheet1'
\subsection{Efficacy of HighlightNet for UAV Dark Tracking}
\subsubsection{Comparison of HighlightNet and SOTA enhancers}
To demonstrate the superiority of HighlightNet for low-light UAV tracking, the performance of HighlightNet and other SOTA low-light enhancers---Darklighter~\cite{Ye2021IROS}, EnlightenGAN~\cite{EnlightenGAN}, Zero-DCE~\cite{Guo_2020_CVPR}, and LIME~\cite{LIME}, on UAVDark135 benchmark with tracker SiamAPN++~\cite{Cao2021IROS} is analyzed. SiamAPN++ is employed because it's designed specifically for UAV tracking. TABLE~\ref{tab:compare1} shows the tracking results with images enhanced by various enhancers. Since HighlightNet is mainly designed for UAV nighttime applications, it achieves the highest improvement in success rate and precision among the four enhancers. HighlightNet brings an increase of more than \textbf{26\%} and \textbf{25\%} in success rate and precision, surpassing the second-best Zero-DCE by \textbf{6.07\%} and \textbf{8.8\%} in the increase of success rate and precision, respectively.
\subsubsection{HighlightNet on different UAV challenges}
Sequences in benchmark UAVDark135 with different attributes are employed to testify the tracking performance of specific challenges in UAV applications. The improvement of tracking performance in typical low-light aerial challenges, \textit{i.e.}, fast motion (FM), illumination variation (IV), low-resolution (LR), visual occlusion (OCC), and viewpoint change (VC) is reported in TABLE~\ref{tab:att}. As shown, in the term of IV and FM, it achieves an enormous improvement of over \textbf{25\%}, which is credited to the design of dynamic parameter adjustment according to global illumination. Furthermore, thanks to the pixel-level illumination enhancement brought by HightlightNet, the ability of trackers to cope with LR is also revived markedly. Though HighlightNet is not specially designed to deal with OCC and VC, it still brings an increase of over \textbf{13\%}, which is due in large part to the capacity of HighlightNet to highlight potential features for the tracker.
\begin{table}[b!]
\scriptsize
\centering
\caption{Evaluation of tracking performance in different UAV challenges, results are shown as success/precision. $\Delta$ denotes the percentages of improvement brought by HighlightNet. In all typical challenges of dark tracking, HighlightNet boosts tracking performance obviously.}
\resizebox{0.95\linewidth}{!}{
\begin{tabular}{cccccc}
\toprule
Attributes & \multicolumn{1}{c}{ FM } & \multicolumn{1}{c}{ IV } & \multicolumn{1}{c}{LR }& \multicolumn{1}{c}{ OCC} & \multicolumn{1}{c}{VC} \\
\midrule
Original & 0.315/0.404 & 0.312/0.408 & 0.323/0.464 & 0.332/0.416 & 0.353/0.426 \\
\midrule
+HighlightNet (ours) & 0.399/0.507 & 0.398/0.515 & 0.402/0.571 & 0.392/0.497 & 0.408/0.485 \\
\midrule
$\Delta$ (\%) &{\textbf{26.7/25.5}} &{\textbf{27.6/26.2}} &{\textbf{24.4/23.1}} &{\textbf{18.1/19.5}} &{\textbf{15.6/13.8}} \\
\bottomrule
\end{tabular}}
\label{tab:att}
\end{table}
\subsubsection{HighlightNet on different trackers}
As shown in Fig.~\ref{fig:all}, HighlightNet facilitates the low-light tracking performance significantly, with obvious improvement for all trackers in both precision and success rate. Among these four trackers, HighlightNet improves the dark tracking performance of SiamAPN and SiamAPN++ obviously, with an increase of more than \textbf{42.6\%} and \textbf{26.2\%} in success rate, as well as an increase of \textbf{31.8\%} and \textbf{25.9\%} in precision.
Some tracking screenshots of the trackers with or without HighlightNet are exhibited in Figure~\ref{fig:qua}. As shown, with the help of HighlightNet, trackers are able to retrieve the target missing in the shadow. We can conclude that HighlightNet boosts the accuracy and reliability of the trackers in these low-light conditions.
\Remark For visualization, images in Fig.~\ref{fig:qua} are enhanced by HighlightNet except for the first column. The enhanced results demonstrate that HighlightNet can markedly facilitate the perception of human operators in nighttime conditions, benefiting the initial annotation task.
\begin{figure}[!t]
\centering
\includegraphics[width=0.99\linewidth]{images/fig6.pdf}
\setlength{\abovecaptionskip}{-4pt}
\caption
{
Qualitative results of trackers with HighlightNet enabled (solid boxes) or not (dashed boxes). For visualization, HighlightNet is employed to enhance the images in the last three columns. The sequences, from top to bottom, are $bike3$, $bike9$, and $group1$ from the public benchmark UAVDark135. HighlightNet dramatically improves tracker nighttime tracking performance.
}
\label{fig:qua}
\vspace{-0.4cm}
\end{figure}
\begin{table}[!b]
\scriptsize
\centering
\caption{Comparison of HighlightNet with different modules enabled. Success rate and precision (Succ./Prec.) represent tracking performance while PSNR and SSIM represent human perception. The best results are in \NoOne{red} font.}
% \footnotesize
% \resizebox{0.7\linewidth}{!}{
\begin{tabular}{cccccc}
\toprule
\multicolumn{1}{c}{RM} & \multicolumn{1}{c}{TPA} & \multicolumn{1}{c}{ST} & \multicolumn{1}{c}{$\mathcal{L}_{\mathrm{dan}}$} & Succ./Prec. & PSNR/SSIM \\
\midrule
& & & & 0.336/0.428 & -/- \\
\midrule
\multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & & & 0.391/0.501 & 11.6/0.67 \\
\multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & & 0.419/0.530 & 11.3/0.73 \\
\multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & & \multicolumn{1}{c}{\checkmark} & 0.416/0.527 & 11.6/0.69 \\
\midrule
\multicolumn{1}{c}{\checkmark} & & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & 0.419/0.526 & 10.6/0.73 \\
\midrule
& \multicolumn{1}{c}{\checkmark}& \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & 0.418/0.524 & 11.2/0.71 \\
\midrule
\multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \textcolor[rgb]{ 1, 0, 0}{\textbf{0.424/0.539}} & \textcolor[rgb]{ 1, 0, 0}{\textbf{11.8/0.75}} \\
\bottomrule
\end{tabular}
\label{tab:abla}%
\end{table}%
\subsection{Ablation Study}
The performance of different variants of HighlightNet is investigated in this subsection. Tracking results on UAVDark135~\cite{li2021allday} are exhibited in TABLE~\ref{tab:abla}. RM, TPA, and ST denote the range mask, the Transformer-based parameter adjustment, and the soft truncation, respectively. We trained all these variations in the same way. The bottom row shows the unsatisfying performance without HighlightNet, while the top row shows the completely enhanced result. Since the soft truncation and $\mathcal{L}_{\mathrm{dan}}$ both target filtering of useless features, their effects are discussed in the same sector.
\subsubsection{Range mask} To ablate the range mask without impacting other modules, it is replaced with another mask. Each pixel in this mask is the same as the average value of pixels in the range mask. As shown in the sixth line of TABLE~\ref{tab:abla}, without the range mask, the improvement of the tracking performance is reduced to \textbf{8.0\%} in success rate and \textbf{8.5\%} in precision. Moreover, with a decrease of \textbf{5.0\%} and \textbf{6.0\%}, PSNR and SSIM also degrade obviously. The function of the range mask to make HighlightNet focus on the tracking target and improve illumination uniformity can be verified.
\subsubsection{TPA} Constraint $\alpha$ and threshold $\beta$ are set to constant values, which are their average values in a complete HighlightNet. Since the parameters are invariable, the adjustment branch is consequently disabled. As shown in the fifth line of TABLE~\ref{tab:abla}, the gains of HighlightNet bringing to both human perception and tracking performance degraded significantly. Therefore, the Transformer-based parameter adjustment module is of clear benefit to UAV tracking tasks.
\subsubsection{ST \& $\mathcal{L}_{\mathrm{dan}}$} Ablating ST and $\mathcal{L}_{\mathrm{dan}}$ respectively, the benefit of introducing HighlightNet decreases to some extent, validating the effectiveness of both anti-noise mask and dark area noise loss. Furthermore, as shown in the second line of TABLE~\ref{tab:abla}, simultaneously disabling ST and $\mathcal{L}_{\mathrm{dan}}$ brings a more obvious reduction in success rate, precision, PSNR, and SSIM. Therefore, it is testified that ST should be employed along with the $\mathcal{L}_{\mathrm{dan}}$ to minimize the over-enhancement of noise.
\begin{figure}[!t]
% \setlength{\abovecaptionskip}{-4pt}
\centering
\includegraphics[width=0.99\linewidth]{images/real-world_2.pdf}
\setlength{\abovecaptionskip}{-4pt}
\caption
{
Evaluations on a standard UAV platform in the real world. The estimated positions are indicated by \NoOne{red} bounding boxes. Below are CLE curves between predictions and ground truth. The \NoTwo{green} dashed line marks a 20-pixel threshold, within which tracking mistakes are usually considered acceptable. The base tracker achieves good nighttime tracking with the help of HighlightNet.
}
\label{fig:real}
\vspace{-0.4cm}
\end{figure}
\subsection{Real-World Tests}
To testify HighlightNet's practicability in real-world low-light UAV tracking applications, we employ it on a typical UAV platform with embedded devices, \textit{i.e.}, the NVIDIA Jetson AGX Xavier. HighlightNet offers a conspicuous real-time performance of $\textbf{32.2}$ FPS even without TensorRT acceleration. In addition, Fig.~\ref{fig:real} shows CLE curves and various real-world nighttime tracking experiments. The main obstacles in the testing are illumination variation, low brightness, tiny target, and partial occlusion. The prediction errors are less than 20 pixels on the CLE curves, showing that the tracking is accurate. With the help of HighlightNet, the basic tracker can provide reliable object tracking in nighttime conditions.
%The frame-rate of image acquisition of UAV is 10 FPS, which leads to great inter-frame variation and considerable challenges for tracking.
% The tracking speed ($\sim$11.23 frames/s) surpass the frame-rate of image acquisition of UAV (10 frames/s), meeting the real-time requirement.
\section{Conclusion}
This work proposes a low-light enhancer HightlightNet for facilitating both online target selection step and tracking step in UAV nighttime tracking. Therefore, HighlightNet takes both human perception and nighttime challenges for UAVs into consideration. By three novel modules, it highlights the potential features and consequently reduces the influence of rapid illumination variation, artificial light sources, small objects, and image noise. Experiments on various tracking approaches confirm its compatibility and effectiveness in both online target selection and tracking. Comparison with other SOTA low-light enhancers verifies the advantages of HighlightNet for facilitating human perception and tracking performance in nighttime conditions. Real-world experiments on a typical UAV platform confirm its applicability and dependability while consuming little computational resources. To summarize, we are confident that this research will aid in the expansion of UAV tracking applications to nighttime environments.
%This work constructs a general low-light enhancer towards facilitating high-level vision tasks, exemplified by UAV tracking. To this concern, DarkLighter dedicates to alleviate the influence of poor illumination on UAV tracking. Experiments verify its compatibility and effectiveness on different tracking approaches. Comparison with current SOTA low-light enhancers demonstrates the superiority of DarkLighter for promoting UAV tracking robustness in low-illumination conditions. Real-world tests onboard a typical UAV system further substantiate its applicability and dependability, with low computational consumption. To sum up, we strongly believe this study can be of assistance to broadening UAV tracking applications to low-light conditions.
%substantiate
\section*{Acknowledgment}
This work is supported by the National Natural Science Foundation of China (No. 62173249), the Natural Science Foundation of Shanghai (No. 20ZR1460100).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Reference
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{IEEEtran}
\normalem
\bibliography{IROS2021}
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{Cheng2017IROS}
H.~{Cheng}, L.~{Lin}, Z.~{Zheng}, Y.~{Guan}, and Z.~{Liu}, ``{An Autonomous
Vision-Based Target Tracking System for Rotorcraft Unmanned Aerial
Vehicles},'' in \emph{Proceedings of the IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS)}, 2017, pp. 1732--1738.
\bibitem{LAND}
C.~Meng, B.~He, H.~Lin, and L.~Wu, ``{Research on UAV Point Landing Based on
Visual Navigation},'' in \emph{Proceedings of the IEEE CSAA Guidance,
Navigation and Control Conference (CGNCC)}, 2018, pp. 1--6.
\bibitem{Ye_2021_TIE}
J.~Ye, C.~Fu, F.~Lin, F.~Ding, S.~An, and G.~Lu, ``{Multi-Regularized
Correlation Filter for UAV Tracking and Self-Localization},'' \emph{IEEE
Transactions on Industrial Electronics}, vol.~69, no.~6, pp. 6004--6014,
2022.
\bibitem{Li_2018_CVPR}
B.~Li, J.~Yan, W.~Wu, Z.~Zhu, and X.~Hu, ``{High Performance Visual Tracking
with Siamese Region Proposal Network},'' in \emph{Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR)}, 2018, pp.
8971--8980.
\bibitem{Cao2021ICCV}
Z.~Cao, C.~Fu, J.~Ye, B.~Li, and Y.~Li, ``{HiFT: Hierarchical Feature
Transformer for Aerial Tracking},'' in \emph{Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV)}, 2021, pp.
15\,437--15\,446.
\bibitem{li2021allday}
B.~Li, C.~Fu, F.~Ding, J.~Ye, and F.~Lin, ``{All-Day Object Tracking for
Unmanned Aerial Vehicle},'' \emph{IEEE Transactions on Mobile Computing}, pp.
1--13, 2022.
\bibitem{Ye2021IROS}
J.~Ye, C.~Fu, G.~Zheng, Z.~Cao, and B.~Li, ``{DarkLighter: Light Up the
Darkness for UAV Tracking},'' in \emph{Proceedings of the IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS)}, 2021, pp.
3079--3085.
\bibitem{LIME}
X.~{Guo}, Y.~{Li}, and H.~{Ling}, ``{LIME: Low-Light Image Enhancement via
Illumination Map Estimation},'' \emph{IEEE Transactions on Image Processing},
vol.~26, no.~2, pp. 982--993, 2017.
\bibitem{RetinexNet}
C.~Wei, W.~Wang, W.~Yang, and J.~Liu, ``{Deep Retinex Decomposition for
Low-light Enhancement},'' in \emph{Proceedings of the British Machine Vision
Conference (BMVC)}, 2018, pp. 127--136.
\bibitem{Zhang2019ACM}
Y.~Zhang, J.~Zhang, and X.~Guo, ``{Kindling the Darkness: A Practical Low-Light
Image Enhancer},'' in \emph{Proceedings of the ACM International Conference
on Multimedia (MM)}, 2019, pp. 1632--1640.
\bibitem{EnlightenGAN}
Y.~{Jiang}, X.~{Gong}, D.~{Liu}, Y.~{Cheng}, C.~{Fang}, X.~{Shen}, J.~{Yang},
P.~{Zhou}, and Z.~{Wang}, ``{EnlightenGAN: Deep Light Enhancement Without
Paired Supervision},'' \emph{IEEE Transactions on Image Processing}, vol.~30,
pp. 2340--2349, 2021.
\bibitem{Guo_2020_CVPR}
C.~Guo, C.~Li, J.~Guo, C.~C. Loy, J.~Hou, S.~Kwong, and R.~Cong,
``{Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement},''
in \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2020, pp. 1777--1786.
\bibitem{vaswani2017nips}
A.~Vaswani, N.~Shazeer, N.~Parmar \emph{et~al.}, ``{Attention Is All You
Need},'' in \emph{Proceedings of the Advances in Neural Information
Processing Systems (NIPS)}, 2017, pp. 6000--6010.
\bibitem{Danelljan2017CVPR}
M.~Danelljan, G.~Bhat, F.~S. Khan, and M.~Felsberg, ``{ECO: Efficient
Convolution Operators for Tracking},'' in \emph{Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR)}, 2017, pp.
6638--6646.
\bibitem{zheng2021ICRA}
G.~Zheng, C.~Fu, J.~Ye, F.~Lin, and F.~Ding, ``{Mutation Sensitive Correlation
Filter for Real-Time UAV Tracking with Adaptive Hybrid Label},'' in
\emph{Proceedings of the IEEE International Conference on Robotics and
Automation (ICRA)}, 2021, pp. 503--509.
\bibitem{LiBo_2019_CVPR}
B.~Li, W.~Wu, Q.~Wang, F.~Zhang, J.~Xing, and J.~Yan, ``{SiamRPN++: Evolution
of Siamese Visual Tracking With Very Deep Networks},'' in \emph{Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR)}, 2019, pp. 4277--4286.
\bibitem{Xu_2020_AAAI}
Y.~{Xu}, Z.~{Wang}, Z.~{Li}, Y.~{Yuan}, and G.~{Yu}, ``{SiamFC++: Towards
Robust and Accurate Visual Tracking with Target Estimation Guidelines},'' in
\emph{Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
vol.~34, no.~07, 2020, pp. 12\,549--12\,556.
\bibitem{Fu_2021_TGRS}
C.~Fu, Z.~Cao, Y.~Li, J.~Ye, and C.~Feng, ``Onboard real-time aerial tracking
with efficient siamese anchor proposal network,'' \emph{IEEE Transactions on
Geoscience and Remote Sensing}, vol.~60, pp. 1--13, 2022.
\bibitem{Ren2017tpami}
S.~{Ren}, K.~{He}, R.~{Girshick}, and J.~{Sun}, ``{Faster R-CNN: Towards
Real-Time Object Detection with Region Proposal Networks},'' \emph{IEEE
Transactions on Pattern Analysis and Machine Intelligence}, vol.~39, no.~6,
pp. 1137--1149, 2017.
\bibitem{AD2}
C.~Fu, S.~Li, X.~Yuan, J.~Ye, Z.~Cao, and F.~Ding, ``{Ad$^2$Attack: Adaptive
Adversarial Attack on Real-Time UAV Tracking},'' in \emph{Proceedings of the
International Conference on Robotics and Automation (ICRA)}, 2022, pp.
5893--5899.
\bibitem{TCT}
Z.~Cao, Z.~Huang, L.~Pan, S.~Zhang, Z.~Liu, and C.~Fu, ``{TCTrack: Temporal
Contexts for Aerial Tracking},'' in \emph{Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR)}, 2022, pp.
14\,798--14\,808.
\bibitem{Edwin_1977_retinex}
E.~H. Land, ``{The Retinex Theory of Color Vision},'' \emph{Scientific
American}, vol. 237, no.~6, pp. 108--129, 1977.
\bibitem{lore2017llnet}
K.~G. Lore, A.~Akintayo, and S.~Sarkar, ``{LLNet: A deep autoencoder approach
to natural low-light image enhancement},'' \emph{Pattern Recognition},
vol.~61, pp. 650--662, 2017.
\bibitem{DUPE}
R.~Wang, Q.~Zhang, C.-W. Fu, X.~Shen, W.-S. Zheng, and J.~Jia, ``{Underexposed
Photo Enhancement Using Deep Illumination Estimation},'' in \emph{Proceedings
of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR)}, 2019, pp. 6842--6850.
\bibitem{LEARN}
K.~Xu, X.~Yang, B.~Yin, and R.~W. Lau, ``{Learning to Restore Low-Light Images
via Decomposition-and-Enhancement},'' in \emph{Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR)}, 2020, pp.
2278--2287.
\bibitem{RUAS}
R.~Liu, L.~Ma, J.~Zhang, X.~Fan, and Z.~Luo, ``{Retinex-inspired Unrolling with
Cooperative Prior Architecture Search for Low-light Image Enhancement},'' in
\emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2021, pp. 10\,556--10\,565.
\bibitem{SCT}
J.~Ye, C.~Fu, Z.~Cao, S.~An, G.~Zheng, and B.~Li, ``{Tracker Meets Night: A
Transformer Enhancer for UAV Tracking},'' \emph{IEEE Robotics and Automation
Letters}, vol.~7, no.~2, pp. 3866--3873, 2022.
\bibitem{Ye2022CVPR}
J.~Ye, C.~Fu, G.~Zheng, D.~P. Paudel, and G.~Chen, ``{Unsupervised Domain
Adaptation for Nighttime Aerial Tracking},'' in \emph{Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 2022,
pp. 8896--8905.
\bibitem{DETR}
N.~Carion, F.~Massa, G.~Synnaeve, N.~Usunier, A.~Kirillov, and S.~Zagoruyko,
``{End-to-End Object Detection with Transformers},'' in \emph{Proceedings of
the European Conference on Computer Vision (ECCV)}, 2020, pp. 213--229.
\bibitem{dosovitskiy2020image}
A.~Dosovitskiy, L.~Beyer, A.~Kolesnikov, D.~Weissenborn, X.~Zhai \emph{et~al.},
``{An Image is Worth 16x16 Words: Transformers for Image Recognition at
Scale},'' in \emph{Proceedings of the International Conference on Learning
Representations (ICLR)}, 2021, pp. 1--12.
\bibitem{Cai2018TIP}
J.~{Cai}, S.~{Gu}, and L.~{Zhang}, ``{Learning a Deep Single Image Contrast
Enhancer from Multi-Exposure Images},'' \emph{IEEE Transactions on Image
Processing}, vol.~27, no.~4, pp. 2049--2062, 2018.
\bibitem{Cao2021IROS}
Z.~Cao, C.~Fu, J.~Ye, B.~Li, and Y.~Li, ``{SiamAPN++: Siamese Attentional
Aggregation Network for Real-Time UAV Tracking},'' in \emph{Proceedings of
the IEEE/RSJ International Conference on Intelligent Robots and Systems
(IROS)}, 2021, pp. 3086--3092.
\bibitem{Mueller2016ECCV}
M.~Mueller, N.~Smith, and B.~Ghanem, ``{A Benchmark and Simulator for UAV
Tracking},'' in \emph{Proceedings of the European Conference on Computer
Vision (ECCV)}, 2016, pp. 445--461.
\end{thebibliography}