Benchmarking person re-identification datasets and approaches for practical real-world implementations
Extracted source text · arXiv 2212.09981
\documentclass[a4paper,twoside]{article}
\usepackage{epsfig}
\usepackage{subcaption}
\usepackage{calc}
\usepackage{amssymb}
\usepackage{amstext}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{multicol}
\usepackage{pslatex}
\usepackage{apalike}
\usepackage[bottom]{footmisc}
\usepackage{SCITEPRESS} % Please add other packages that you may need BEFORE the SCITEPRESS.sty package.
\usepackage{multirow}
\usepackage{xcolor}
\usepackage{url}
%\usepackage{hyperref}
\begin{document}
\title{Benchmarking person re-identification datasets and approaches for practical real-world implementations}
%\author{Anonymous submission}
\author{
\authorname{Jose Huaman,Felix O. Sumari, Luigy Machaca, Esteban Clua and Joris Guerin}
\affiliation{\sup{1} Instituto de Computação, Universidade Federal Fluminense, Niteroi-RJ , Brazil}
\affiliation{\sup{2}LAAS-CNRS, Université de Toulouse, Toulouse, France}
\email{\{jmhcruz, fsumari, luigyarcana\}@id.uff.br,esteban@ic.uff.br, jorisguerin.research@gmail.com}
}
\keywords{Person Re-Identification, Practical Deployment, Benchmark Study}
\abstract{Recently, Person Re-Identification (Re-ID) has received a lot of attention. Large datasets containing labeled images of various individuals have been released, allowing researchers to develop and test many successful approaches. However, when such Re-ID models are deployed in new cities or environments, the task of searching for people within a network of security cameras is likely to face an important domain shift, thus resulting in decreased performance. Indeed, while most public datasets were collected in a limited geographic area, images from a new city present different features (e.g., people's ethnicity and clothing style, weather, architecture, etc.). In addition, the whole frames of the video streams must be converted into cropped images of people using pedestrian detection models, which behave differently from the human annotators who created the dataset used for training. To better understand the extent of this issue, this paper introduces a complete methodology to evaluate Re-ID approaches and training datasets with respect to their suitability for unsupervised deployment for live operations. This method is used to benchmark four Re-ID approaches on three datasets, providing insight and guidelines that can help to design better Re-ID pipelines in the future.}
\onecolumn \maketitle \normalsize \setcounter{footnote}{0} \vfill
This paper is the extended version of our short paper accepted in VISAPP - 2023.
\section{\uppercase{Introduction}}
\label{sec:introduction}
% Paragraph 1 -- What is ReID ? why do we use it ? How good can it be for society? -- If you find some papers discussing this, good to cite it and base the discussion on their studies.
As many cameras are being deployed in public places (airports, malls, etc.), monitoring of video streams by security agents becomes impractical. Automated processing is a promising perspective to analyze the whole network in real-time, and select only relevant sequences for verification by human operators. This paper deals with person Re-Identification (Re-ID), a computer vision problem to find an individual in a network of non-overlapping cameras~\cite{old_reID_survey}. It has diverse potential applications such as suspect searching~\cite{open_set_first_paper}, identifying owners of abandoned luggage~\cite{abandoned_luggage}, or recovering missing children~\cite{missing_children}.
In the literature, the problem of Re-ID is studied under different settings (see %The different Re-ID paradigms are presented in details in
Section~\ref{sec:related_paradigms}). On the one hand, the most studied Re-ID paradigm, which we refer to as \emph{standard Re-ID}, tries to find images representing the query person within a gallery of pre-cropped images of persons, containing at least one correct match~\cite{standard_reID_survey}. %person search~\cite{person_search_first_paper} extends the standard Re-ID setting by considering a search gallery of whole scene images instead of pre-cropped images of humans.
On the other hand, we recently introduced a setting considering specifically the constraints to implement Re-ID for live operations, which we call as \emph{live Re-ID}~\cite{application_level_ReID}. The first contribution of this paper is to formalize the definition and constraints associated with live Re-ID and to extend the evaluation metrics presented in~\cite{application_level_ReID} to facilitate interpretation.
\begin{figure*}[t]
\centering
\includegraphics[width=0.75\textwidth]{figure/overview.png}
\caption{\textbf{Conceptual overview}. Representation of the objectives of our benchmark study. This work aims at evaluating how different standard Re-ID approaches and training datasets behave for practical deployment in new environments (live Re-ID).}
\label{fig:overview}
\end{figure*}
Standard Re-ID is not the best-suited paradigm for practical implementations, as it does not consider the influence of domain shift due to pedestrian detection errors or deployment in a city with different characteristics. Indeed, in our previous experiments~\cite{application_level_ReID}, we showed that training a successful Re-ID model with respect to standard Re-ID metrics does not guarantee good performance when evaluated in a specific live Re-ID context.
% Paragraph 3 -- A lot of ReID methods are being developed, different datasets are being used for training these methods. The goal of this paper is to propose a way to benchmark different methods and different training datasets to see how well they perform for the FF setting. Questions we want to answer: What method allow better generalization to FF PRID on a different domain ? What training dataset allow better generalization to FF PRID on a different domain ? Is there interactions between the method used and the training dataset used (a given methods performs better with a given dataset) ? Can we use a standard cross-domain evaluation methodology to assess the FF prid generalization of a given method-dataset pair?
Nevertheless, most publicly available large-scale datasets for Re-ID focus on the standard Re-ID setting, and many successful approaches have been developed for this specific purpose. For this reason, we believe that it is essential to study if these datasets and approaches can be used to implement and deploy practical applications in different contexts. %Recent research demonstrated that a good OD model and a good Re-ID model does not necessarily produce good results for DL Re-ID.
%To do this, this paper presents a structured benchmark experiment
More specifically, the objective of this paper is to answer the following questions: \begin{enumerate}
\item Which characteristics of a standard Re-ID dataset (diversity, size) are most important to train standard Re-ID models for the live Re-ID setting?
\item Which standard Re-ID approaches can be successfully deployed for practical implementations in the live Re-ID setting?
\item Do different Re-ID approaches have different optimal datasets for deployment?
\item Can we use a simple cross-dataset evaluation methodology to assess the deployability of a given approach-dataset pair?
\end{enumerate}
To answer these questions, we conducted a study using three standard Re-ID training datasets and four recent standard Re-ID approaches. For each approach-dataset pair, the Re-ID model obtained was evaluated against the other two datasets and against another one configured for the live Re-ID setting. We also combine training datasets to investigate how dataset size and diversity influence the generalization of the obtained standard Re-ID model. A conceptual overview of our objectives is represented in Figure~\ref{fig:overview}.
% Paragraph 4 -- Explain BRIEFLY the experiments, the methodology employed to answer these questions. Give hints towards the results obtained. (Can leave this part for later if you want, after we write the result section)
In this paper, we consider the evaluation of Re-ID models without additional training on images from the target domain. More sophisticated approaches have been proposed for domain adaptation of standard Re-ID models. On the one hand, the unsupervised domain adaptation problem consists in leveraging unlabeled data from the target domain to improve the performance of the standard Re-ID model~\cite{zhao2020unsupervised,mekhazni2020unsupervised}. On the other, methods from the transfer learning field~\cite{zhao2020unsupervised} have been applied to fine-tune standard Re-ID models for new contexts where a small amount of labeled data is available~\cite{chen2018deep}. Such domain adaptation approaches are not tested in this work, but we believe standard Re-ID models performing well without target domain training (our experiments) are likely to be good initialization for more sophisticated fine-tuning approaches. On another note, it was shown that considering bounding box extraction and Re-ID separately is not as good as end-to-end approaches for person search, i.e., galleries of whole scene images~\cite{person_search_explain}. However, our results show that this two-step approach can perform well on the live Re-ID setting for some configurations. Likewise, we believe that the results from our study can be useful to pre-train successful initial live Re-ID models and to guide the development of more complex end-to-end architectures for live Re-ID.
This paper is organized as follows: Section~\ref{sec:related} discusses the relevant related literature. The proposed benchmark methodology is detailed in Section~\ref{sec:methodology}. The results are presented in Section~\ref{sec:results} and discussed in Section~\ref{sec:discussion}. Finally, Section~\ref{sec:conclusion} presents our conclusions.
\section{\uppercase{Related work}}
\label{sec:related}
A complete literature review of Re-ID approaches is not the purpose of this paper. Instead, we present clear definitions of the different existing Re-ID settings and discuss existing benchmark studies about Re-ID.
\subsection{Person re-identification settings}\label{sec:related_paradigms}
The field of Re-ID consists in retrieving instances of a given individual, called the \emph{query}, within a complex set of multimedia content called the \emph{gallery}~\cite{standard_reID_first_paper}. Different settings are defined by how they represent the query and the gallery items, the constraints on the gallery content, the boundaries of the Re-ID system, and the evaluation methodology.
\subsubsection{Popular settings}
\paragraph{Standard Re-ID} In this setting, both the query, and all items in the gallery are well-cropped images representing entire human bodies. It is sometimes called closed-set Re-ID as it assumes that the query has at least one representative in the gallery. According to the statistics in Papers with Code~\cite{papers_with_code} it is the most studied Re-ID setting by a large margin, in terms of number of papers, datasets and benchmarks published. Some standard Re-ID datasets and successful methods are used for our benchmark study and presented in Section~\ref{sec:methodology}. For a more complete overview of standard Re-ID approaches, see~\cite{standard_reID_survey2}.
\begin{figure*}[t]
\centering
\includegraphics[width=0.9\textwidth]{figure/liveReID.png}
\caption{\textbf{The live Re-ID setting}. When deploying Re-ID models in practice, the galleries are composed of whole scene video sequences. When an alert is raised, the data are verified by a security agent to decide whether actions should be triggered.}
\label{fig:liveReId}
\end{figure*}
\paragraph{Person search} This setting consists in replacing the gallery items by whole scene images~\cite{person_search_explain}. In other words, a person search model must return the index of the gallery image where the query is present and its location in terms of Bounding Box (BB) coordinates. A survey about person search approaches was proposed in~\cite{person_search_survey}.
\paragraph{Open-set Re-ID} This setting differs from standard Re-ID in that there is no guarantee that the query is represented in the gallery, i.e., an open-set Re-ID model should be able to answer whether the gallery contains the query. The reader can refer to~\cite{open_set_survey} for an overview of recent approaches.
\paragraph{Video-based Re-ID} In this setting, images (query and gallery) are replaced by image sequences extracted from consecutive video frames. Sequences are composed of well-cropped entire body images representing the same person. A complete survey of video-based Re-ID was proposed in~\cite{standard_reID_survey2}.
\paragraph{Others} For completeness, we mention other Re-ID variants, namely unsupervised Re-ID\cite{unsupervised_reid}, semi-supervised Re-ID\cite{semisupervised_reid}, human-in-the-loop Re-ID\cite{human_in_the_loop_reid}, or federated Re-ID\cite{federated_ReID}. However, their specificity lie in Re-ID models training while the other settings above focus on constraints at inference.
\subsubsection{Live Re-ID setting} \label{sec:related_live}
In this section, we clearly define and formalize the \emph{live Re-ID} setting, which is inspired by our previous work~\cite{application_level_ReID}. It takes into account all relevant aspects for deploying Re-ID in practical real-world applications. An overview of the live Re-ID workflow can be seen in Figure~\ref{fig:liveReId}.
When looking for a query person during live operations, whole scene videos need to be processed in near real-time, hence the galleries for live Re-ID are composed of the consecutive \emph{whole scene frames} from \emph{short video sequences}. The live Re-ID context is also highly \emph{open-set} as the probability to have the query in a short video sequence from a given camera is low. Hence, this setting combines elements from several of the Re-ID settings mentioned above. Using these live Re-ID characteristics, it was recently shown that using tracking and anomaly detection to reduce the size of the generated gallery improves live Re-ID results~\cite{machaca2022trade}.
Another key characteristic of live Re-ID is that the training context is different from the deployment context. Indeed, building new specialized datasets for deployment in every shopping mall or small city is unrealistic from the perspective of future advances in the field. This highlights the importance of studying \emph{cross domain} transfer of Re-ID, which was first discussed and highlighted in~\cite{luo_strong_2020}.
Finally, this setting also takes into account that Re-ID model predictions need to be \emph{processed by a human agent}, who triggers appropriate actions. This way, very high rank-1 accuracy is not mandatory for live Re-ID, as the operator can find the query in later ranks. On the other hand, false alarm rates must be kept low to avoid overloading human operators. To evaluate these two objectives, we introduced two evaluation metrics representing both dimensions of the problem in~\cite{application_level_ReID}. The experiments conducted in this paper aim at studying the transferability of standard Re-ID approaches and datasets for deployment in the live Re-ID setting.
% Present the different Re-ID paradigms and papers associated. Also present different approaches BRIEFLY for each paradigm.
% In this paper, we focus on FF-PRID paradigm. It should be explained very well. In particular, explain why it is important (it corresponds to what happens in practice when trying to implement Re-ID in smart cities).
\subsection{Person re-identification benchmarks}\label{sec:related_eval_methods}
This section presents several benchmark studies considering different aspects of the Re-ID pipeline. %We discuss these works from the perspective of the evaluation methodology employed.
A large scale benchmark experiment was conducted to compare various approaches for standard and video-based Re-ID~\cite{benchmark_reID_single_multi}. They evaluated more than 30 approaches on 16 public datasets, and produced the largest Re-ID benchmark to date. In addition, they built a new dataset to represent several constraints for real-world implementations, such as pedestrian detection errors and illumination variations, among others. However, they do not consider cross domain performance % to be able to deploy Re-ID algorithms in new cities without having to develop context dependent specialized datasets.
% Furthermore, %Another drawback is that
and all evaluations are conducted in the closed-set setting, which are major limitations regarding future deployments. %To address these limitations, all important requirements for future deployments of Re-ID algorithms are taken into account in our live Re-ID evaluation experiment.
In addition, a smaller systematic evaluation of video-based Re-ID approaches was proposed in~\cite{mars}.
Another extensive set of experiments was conducted to evaluate different pedestrian detection models on a two-step person search pipeline~\cite{benchmark_OD_reID}. They demonstrated that the best models on standard object detection metrics are not necessarily the best suited for Re-ID from whole scene frames. In addition, a first benchmark regarding cross-domain transfer of Re-ID approaches was proposed in~\cite{benchmark_reID_cross}. Their experiments consisted in training an approach on one standard Re-ID dataset and evaluating on another. Finally, on another note, different approaches for federated Re-ID where compared in~\cite{federated_ReID}.%, i.e. learning Re-ID across decentralized clients to preserve privacy.
The studies presented above have brought valuable insights to the Re-ID community. However, none of them allows to assess the performance of a Re-ID model against all the challenges involved during deployment in a new environment for practical use in security applications. Our paper contributes to bridging this gap by conducting experiments within the live Re-ID setting, which was designed to take into account all these challenges. In particular, we consider the influence of different standard Re-ID approaches and training datasets on live Re-ID results.
\section{\uppercase{Benchmark methodology}}\label{sec:methodology}
The objective of this paper is to study if different standard Re-ID approaches and training datasets can be used to build efficient live Re-ID pipelines, ready for practical deployment. %In particular, we aim to assess the quality of different Re-ID approaches within the context of practical implementation of FF-PRID. In addition, another goal is to understand if training these approaches on different publicly available Re-ID datasets will lead to different results.
This section presents the different components of the proposed benchmarking evaluation, i.e., the compared datasets and approaches, metrics used, and experiments conducted.
\begin{figure}[t]
\centering
\begin{subfigure}{0.48\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figure/repre_market.png}
\caption{Market-1501.}
\label{fig:datasets_market}
\end{subfigure}
~
\begin{subfigure}{0.48\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figure/repre_duke.png}
\caption{DukeMTMC.}
\label{fig:datasets_duke}
\end{subfigure}
~
\begin{subfigure}{0.48\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figure/repre_cuhk.png}
\caption{CUHK03.}
\label{fig:datasets_cuhk}
\end{subfigure}
~
\begin{subfigure}{0.48\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figure/repre_prid_image.png}
\caption{Original PRID-2011.}
\label{fig:datasets_prid}
\end{subfigure}
~
\begin{subfigure}{0.48\textwidth}
\centering
\includegraphics[width=0.95\textwidth]{figure/repre_prid_video.png}
\caption{{m-PRID}. Extracted from PRID-2011 videos using YOLO-V3. Blue indicates good images for standard Re-ID, while red BB are likely to generate Re-ID errors.}
\label{fig:datasets_mprid}
\end{subfigure}
~
\caption{\textbf{Benchmarking datasets}. Example images from the datasets used in our experimental study.}
\label{fig:repredata}
\end{figure}
\subsection{Datasets}
In our experiments, we used three public datasets to train and evaluate standard Re-ID models, and a live Re-ID dataset to evaluate the trained Re-ID models within the context of live operations. Figure~\ref{fig:repredata} shows example images from the datasets, where we can see that they represent people from different geographic regions, under different resolutions, lighting conditions, and camera angles.
\subsubsection{Standard Re-ID datasets}
\begin{table*}[t]
\centering
\caption{\textbf{Standard Re-ID training datasets}. Characteristics of the standard Re-ID datasets used in this paper.}
\label{tab:data_table}
\begin{center}
\begin{tabular}{cccccc}
%\toprule[1.5pt]
\cline{1-6}
Dataset &
\# Cameras &
%\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Image cropping \\ method\end{tabular}} &
Split &
Input type &
\# IDs & \# Images \\
%\midrule[1.5pt] %\textbf{Place}
\cline{1-6}
\multirow{3}{*}{CUHK03} & \multirow{3}{*}{2} & %\multirow{3}{*}{Hand} &
Train & -- & 767 & 7368 \\
% \cline{4-7}
& & \multirow{2}{*}{Test} & Query & 700 & 1400 \\
& & & Gallery & 700 & 5328 \\
%\midrule
\cline{1-6}
\multirow{3}{*}{DukeMTMC} & \multirow{3}{*}{8} & %\multirow{4}{*}{Hand + Detected} &
Train & -- & 702 & 16522 \\ %\multirow{2}{*}{University of Duke} \\
% \cline{4-7}
& & \multirow{2}{*}{Test} & Query & 702 & 2228 \\ %\multirow{2}{*}{University of Duke} \\
& & & Gallery & 1110 & 17661 \\
%\midrule
\cline{1-6}
\multirow{3}{*}{Market-1501} & \multirow{3}{*}{6} & %\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}}Pedestrian \\ detection \\ + \\ Manual filtering\end{tabular}} &
Train & -- & 751 & 12936\\%& \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Supermarket in \\ Tsinghua University\end{tabular}} \\
% \cline{3-6}
& & \multirow{2}{*}{Test} & Query & 750 & 3368 \\%& \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Supermarket in \\ Tsinghua University\end{tabular}} \\
& & & Gallery & 751 & 15913 \\
%\bottomrule[1.5pt]
\cline{1-6}
\end{tabular}
\end{center}
\end{table*}
This section presents briefly the standard Re-ID datasets used in this study. Table~\ref{tab:data_table} summarizes relevant statistics.
%\begin{figure}
% \centering
% \includegraphics[scale=0.4]{datasets_sample2.png}
% \caption{Representing images of datasets used in this study.}
% \label{fig:repredata}
%\end{figure}
\paragraph{Market-1501} This dataset was collected at a supermarket in Tsinghua University, Beijing, China~\cite{market}. %The training set is composed of $12,936$ images of $751$ different identities, and the testing set contains $3368$ query images and $15,931$ gallery images of $750$ identities.
The cropped images are detected automatically using a Deformable Part Model~%for pedestrian detection
\cite{DPM}, which outputs are filtered manually to keep only good BB representing human bodies. This automated way of extracting BB is closer to realistic settings, which might improve live Re-ID results for models trained on Market-1501. As illustrated by Figure~\ref{fig:datasets_market}, cropped images appear to present a high level of details about the people represented (i.e., images are taken from a close perspective or videos are high resolutions). Lighting conditions in this dataset are also good to distinguish specific features.
% contains 14,097 images of 1,467 identities. Each identity is captured by two cameras on the CUHK campus. This dataset contains two image sets. One is anno- tated by hand-drawn BB, and the other is pro- duced by the DPM detector [9]. We use the detected set in this paper. There are 9.6 images per identity in the training set. We report the averaged result after training/testing 20 times. We use the single shot setting.
\paragraph{DukeMTMC}
This dataset was collected at the Duke University campus, Durham, North Carolina, USA~\cite{duke}. %The training set is composed of $16,522$ images of $702$ identities, and the testing set contains $2,228$ queries and $17,661$ gallery images of $702$ other identities. In addition, $408$ distractor identities are included in the test gallery.
%The size of all images inside duke
The BB in DukeMTMC are hand drawn, lighting conditions are good but resolution of the BB is relatively low (Figure~\ref{fig:datasets_duke}).
\paragraph{CUHK03}
This dataset was built using video footage collected at the campus of the Chinese University of Hong Kong~\cite{cuhk}. In our work, we used the manually labeled version of the BB. Cropped images are high resolution but illumination is dark, which reduces images quality (Figure~\ref{fig:datasets_cuhk}).
%is a subset of the newly-released multi-target, multi-camera pedestrian tracking dataset [30]. The original dataset contains eight 85-minute high- resolution videos from eight different cameras. Hand- drawn pedestrian BB are available. In this work, we use a subset of [30] for image-based re-ID, in the format of the Market-1501 dataset [49]. We crop pedes- trian images from the videos every 120 frames, yielding 36,411 total BB with IDs annotated by [30]. The DukeMTMC-reID dataset for re-ID has 1,812 identi- ties from eight cameras. There are 1,404 identities appear- ing in more than two cameras and 408 identities (distractor ID) who appear in only one camera. We randomly select 702 IDs as the training set and the remaining 702 IDs as the testing set. In the testing set, we pick one query image for each ID in each camera and put the remaining images in the gallery. As a result, we get 16,522 training images with 702 identities, 2,228 query images of the other 702 identi- ties and 17,661 gallery images. The evaluation protocol is available on our website [2]. Some example re-ID results from the DukeMTMC-reID are sho
\subsubsection{Live Re-ID dataset}
To evaluate the different standard Re-ID models for the live Re-ID setting, we used the same dataset as our previous work~\cite{application_level_ReID}, which we call \emph{m-PRID}. It is a modified version of PRID-2011~\cite{prid2011}, built from the raw video footage and the original annotations that were used to create the official curated version of PRID-2011\footnote{We thank the authors of PRID-2011 for their help.}. The videos were collected from two non-overlapping cameras (A and B), located in Graz, Austria. This way, compared to the training datasets above, evaluation on m-PRID represents a geographic domain shift. In total PRID-2011 contains 385 different identities for camera A and 749 for camera B, of which 200 identities appear in both cameras. The m-PRID dataset is composed of several two minutes videos (30 from A and 33 from B). For each short video sample, a ground truth file gather information about each person it contains (identifier, frames where it appears, bounding box coordinates). For evaluation, a total of 73 queries are considered.
To better grasp the influence of the pedestrian detection model, we also evaluate our different models on the original PRID-2011 dataset for standard Re-ID. Figure~\ref{fig:datasets_prid} shows cropped images of poor resolution, taken from relatively high camera angle compared to other datasets. This way, we can see if performance decrease on the live Re-ID setting are due to the domain shift of PRID or to the pedestrian detector inaccuracies (Figure~\ref{fig:datasets_mprid}).
\subsection{Re-ID approaches evaluated} \label{sec:reid_approaches}
This work studies the performance of four successful standard Re-ID approaches. To complement previous benchmark studies (Section~\ref{sec:related_eval_methods}), only recent approaches are selected for this work. %Our experimental results should help understanding which design choices are most important for domain adaptation and generalization to the live Re-ID setting.
\subsubsection{Bag of Tricks (BoT)} The \emph{Bag of Tricks} approach resulted from the observation that most improvements for Re-ID baselines come from neural network training tricks rather than Re-ID approaches themselves~\cite{luo_bag_2019}. As a result, they came up with a simple recipe to successfully train standard Re-ID models on top of a ResNet-50 backbone~\cite{he_deep_2015}. In particular:
\begin{enumerate}
\item the model is pretrained on ImageNet,
\item the dimension of the fully connected layer is set to the number of training identities,
\item the batch size is set to 64,
\item images are resized to 256 X 128,
\item both %the model output features and prediction logits are used respectively to compute
triplet and cross entropy loss are used, and
\item Adam is adopted for optimization.% (initial learning rate of 0.00035 and 120 training epochs).
\end{enumerate}
% Main motivations that authors found to publish their work:
% \begin{itemize}
% \item Previous works were expanded on poor baselines Only two in 23 baselines surpassed 90\% rank-1 accuracy on
% Market1501 Dataset.
% \item Improvements of some works were mainly from training tricks rather than methods themselves.
% \item The industry prefers simple but effective models over concatenating many local features in the inference stage.
% \end{itemize}
% Luo et al. followed a standard baseline which the backbone is ResNet50\cite{he_deep_2015}, this baseline during the training stage follow this steps:
% \begin{enumerate}
% \item They initialize the model with pre-trained parameters on ImageNet and change the dimension of fully connected layer to N(number of identities in training dataset).
% \item They set batch size of 64.
% \item They apply a resize each image into 256 x 128 pixels
% \item The model outputs ReID features $f$ and ID prediction logits $p$. Triplet loss is calculated using $f$ and cross entropy loss using prediction logits $p$.
% \item They adopted Adam method to optimize the model, initial learning rate is set to be 0.00035 and there are 120 training epochs.
% \end{enumerate}
\subsubsection{Strong Baseline and Batch Normalization Neck (SBS)} This approach~\cite{luo_strong_2020} extended BoT by adding the following tricks:
\begin{enumerate}
\item a warm-up strategy~\cite{Fan_2019} is applied to bootstrap the network,
\item random erasing augmentation~\cite{zhong_random_2017} is used to account for potential occlusion,
\item label smoothing~\cite{szegedy_rethinking_2015} is used to reduce overfitting,
\item the last stride of the ResNet-50 backbone is set to 1 to increase spatial resolution,
\item batch normalization layers are added, and
\item a new center loss is introduced to account for the clustering effect of tracking.
\end{enumerate}
% After explore and define a strong baseline in their previous work, they select six tricks to introduce in this study and add the tricks into a widely used baseline to obtain their strong baseline:
% % \begin{enumerate}
% \item Warmup Learning Rate: They applied a warmup strategy to bootstrap the network for enhanced performance.
% \item Random Erasing Augmentation: In real scenarios of person Re-ID there is a occlusion in the person images, so the address this problem \cite{zhong_random_2017} proposed a new data augmentation approach, namely, random erasing augmentation (REA).
% \item Label Smoothing: some IDs of of the testing set doesn't appear in training set to prevent an over fitting in training step they apply Label Smoothing (LS) proposed by \cite{szegedy_rethinking_2015}.
% \item Last Stride: They change the last stride of backbone ResNet50 from 2 to 1, this slightly increase computational cost but the final output increase spatial resolution and gets significant improvement without any extra training parameters.
% \item BNNeck: They add a BN layers to the baseline and test if this can smothen the feature distribution in the embedding space.
% \item Center Loss: Classic Triplet loss can be calculated with feature distances of positive and negative pairs, but ignores the clustering effect that can be useful for tracking and to determine a threshold to separate positive and negative objects. The authors introduce here the center loss which learn center for deep features and an corresponding class centers.
% \end{enumerate}
\subsubsection{Attention Generalized mean pooling with Weighted triplet loss (AGW)} This technique~\cite{standard_reID_survey2} was also designed on top of BoT with three major improved components:
\begin{enumerate}
\item %Non-local Attention (Att) Block: after the authors did experiments with nineteen methods over four datasets. They understood that part/global and attention feature learning had influence in the final re-id discriminative process. So they adopted
a powerful non-local attention block is developed to mix the part and global attention features,
\item a learnable pooling layer replaces max and average pooling to better capture the domain-specific discriminative features, %That's why they adopted a , named generalized-mean (GeM) pooling. This layer is learned on Back propagation process.
\item the use of weighted regularization triplet loss inherits the advantages of relative distance optimization between positive and negative pairs without introducing additional parameters.
\end{enumerate}
\subsubsection{Multiple Granularity Network (MGN)} This approach~\cite{wang_learning_2018} was designed
% Some Deep Learning approaches focus only on the main body of an image but it's possible to
%to focus on different partial regions of the image and search for local features. %This idea was proposed by Wang et al. and designed a Multi Granularity Network(MGN).
combines local and global information in different image granularity. %It combines local and global information in different image granularities. The feature maps are split in N horizontal stripes to learn local feature representations. Then, the ResNet-50 backbone is divided into a multi-branch network after the fourth residual stage.% they propose a multi-branch network architecture divide into two branches:
%The global branch learns global features using down-sampling with a stride-2 convolution layer and representations with 256 dimension features. The part-N branch has a similar architecture without down-sampling.
Its specificities are:
\begin{enumerate}
\item a strategy to learn local feature representations by splitting the input image in horizontal stripes. Then, the ResNet-50 backbone is divided into a multiple branches after the fourth residual stage.
\item a strategy to learn global features using a global branch using down-sampling with a stride-2 convolution layer and representations with 256 dimension features. The part-N branch has a similar architecture without down-sampling.
\item a new Loss functions, combining Softmax for classification and tripĺet loss for metric learning.
\end{enumerate}
\subsection{Proposed experiments}
To compare the Re-ID datasets and approaches presented above, several experiments are conducted.
\subsubsection{Single dataset evaluation}
We first evaluate each approach and dataset pair individually. The standard Re-ID approach is simply fitted to the training split of the dataset, and evaluated on the testing split. The quality of the Re-ID model's predictions on the testing set is assessed using standard Re-ID metrics, coming from the field of information retrieval:
\paragraph{Rank-n}
This metric represents the proportion of queries for which at least one correct match was predicted within the $n$ highest ranked gallery images~\cite{cmc_curve}. In practice, we report results for $n\in\{1,5,10\}$. This metric represents the Re-ID model's ability to retrieve the easiest match.
\paragraph{mAP} The \emph{mean average precision} for Re-ID takes into account the predicted ranks of all existing matches~\cite{benchmark1}. To have a perfect mAP, all the gallery images corresponding to the query need to be ranked in the first places. It is the average performance across all instances of the query.
\paragraph{mINP} The \emph{mean inverse negative penalty} reflects the position of the worst ranked match from the gallery~\cite{standard_reID_survey2}. It represents the capacity of a Re-ID model to find all instances of the query in the gallery.
These three metrics represent different skills of a Re-ID model. Computing them might help understand which of these skill is important regarding generalization to new contexts and to more complex real-world scenarios, i.e., live Re-ID in different cities.
\subsubsection{Cross-dataset evaluation}
A simple cross-dataset experiment is also conducted~\cite{benchmark_reID_cross}. It consists in training an approach on one of the three standard Re-ID datasets, and evaluating it on the other two. The same metrics are used (rank-n, mAP and mINP). As the datasets were built in different geographic areas, these results can give first insights about domain generalization of the different training datasets and approaches. Conducting such cross-dataset evaluation is also much easier than evaluating the system in the live Re-ID setting. Hence, another objective of this experiment is to discover if simple cross-dataset evaluation can be used as a proxy to quickly test new datasets and approaches for live implementations. In other words, we want to know if there is correlation between cross-dataset results and live Re-ID results of dataset-approach pairs.
For the cross-datasets experiments, we also try to combine training datasets to see if it improves test performance. In the COMBINED$_{\text{all}}$ experiments, training is conducted on all training sets available (Market-1501, DukeMTMC and CUHK03), including the one corresponding to the test set of interest. This allows to evaluate if adding data from other sources can help improving standard Re-ID in the traditional supervised setting. In the COMBINED$_{\text{others}}$ experiments, the training set corresponding to the test dataset is excluded. For example, when evaluating on CUHK03, the standard Re-ID models are trained on Market-1501 and DukeMTMC. Finally, the COMBINED$_{\text{scaled}}$ setting is similar to COMBINED$_{\text{others}}$, but we ensure that the total number of training data is equal to the number of data in the largest dataset.
For example, when combining datasets $A$ and $B$, respectively of size $N_A$ and $N_B$, we only take fractions $N^*_A$ and $N^*_B$ of each datasets such that $N^*_A + N^*_B = \text{max}(N_A, N_B)$ and $N^*_A = N^*_B$. Comparing COMBINED$_{\text{scaled}}$, with COMBINED$_{\text{others}}$ allows to evaluate how size and diversity affect the generalization power of a dataset. For evaluations on PRID-2011, which is not among the training datasets, COMBINED$_{\text{all}}$ and COMBINED$_{\text{others}}$ are identical and referred to as COMBINED. When evaluating on CUHK03, the COMBINED$_{\text{scaled}}$ dataset is composed of 8261 images from DukeMTMC and 8261 from Market-1501. When evaluating on DukeMTMC it contains 6468 images from both CUHK03 and Market-1501, and when evaluating on Market-1501 it contains 9754 images from DukeMTMC and 7368 images from CUHK03. Finally, when evaluating on PRID-2011, COMBINED$_{\text{scaled}}$ is composed of 5507 images from CUHK03, 5508 from DukeMTMC and 5507 from Market-1501.
\subsubsection{Live Re-ID evaluation}\label{sec:methodology_livereID_eval}
Finally, each standard Re-ID approach and dataset pairs are evaluated in the live Re-ID setting using the m-PRID dataset. We apply the evaluation methodology from~\cite{application_level_ReID}. For each short video sequence, Bounding Boxes (BB) of pedestrians are extracted using a YOLO-V3 object detector~\cite{redmon2018yolov3}, trained on COCO~\cite{lin2015microsoft} and available in TensorFlow~\cite{tensorflow2015-whitepaper}. The score threshold used to decide which predicted BB to keep is set to $0.5$. Then, the trained standard Re-ID approaches are applied to the gallery composed of these BB. Following the notations of~\cite{application_level_ReID}, the length of video sequences evaluated $\tau$ is set to $1000$ frames and the number of candidates shown to the monitoring agent $\eta$ is set to $20$. These values generated best results by a large margin in their experiments. For the threshold $\beta$ on Re-ID scores used to generate alerts, we test all values between $0$ and $1$ with a step size of $0.02$.
To compare the different models, we use the live Re-ID metrics introduced in~\cite{application_level_ReID}. On the one hand, the \emph{Finding Rate} (FR) represents the proportion of videos where the query was present, such that an alert was shown to the agent and where the query was among the selected candidates. A low FR means that the query was missed frequently. On the other, the \emph{True Validation Rate} (TVR) represents the proportion of alert shown to the monitoring agent in which the query was present among the candidates. A low TVR means that the agent was frequently disturbed for no reason, which can be problematic when many cameras need to be monitored simultaneously.
In this paper we also propose two new metrics to represent the performance of a live Re-ID approach with a single number, to facilitate comparisons and interpretation. The first one is based on the observation that %, although they are not strictly identical,
the meanings of FR and TVR are respectively very close to recall and precision. This way, similarly to object detection evaluation, we can plot \emph{TVR vs FR} curves and compute the \emph{mean Average Precision} ({mAP}) as the area under the curve. The second unified metric %is more similar to the F-score. The two metrics to evaluate live Re-ID approaches are ratios. Hence,
consists in computing a weighted harmonic mean of FR and TVR, similarly to the F-score computation for precision and recall. We call the resulting metric $F_{\gamma}$, which is defined as follows:
\begin{equation}
F_{\gamma} = (1+\gamma^2).\frac{\text{FR}.\text{TVR}}{(\gamma^2.\text{FR})+\text{TVR}}.
\end{equation}
In practice, we compute $F_{\gamma}$ for $\gamma \in \{0.5, 1, 2\}$. In $F_{0.5}$, we consider that having a high TVR is two times more important than a high FR. In $F_{2}$, we consider FR two times more important than TVR, and in $F_{1}$ FR and TVR contribute equally to the results. However, for each value of the threshold $\beta$, there is a different corresponding value of $F_{\gamma}$. To solve this issue, we use the same approach as Guerin et al.,\cite{icmla_joris} consisting in evaluating a model by its performance at the optimal configuration. The result is called \emph{optimal $F_{\gamma}$} ($F^*_{\gamma}$), and corresponds to the highest $F_{\gamma}$ across values of $\beta$. The value of $\beta$ corresponding to $F^*_{\gamma}$ can be viewed as the operating point of the Re-ID model, which can be obtain by quick experiments in the practical implementation context. An $F^*_{\gamma}$ score of 1 means that there exist a Re-ID threshold $\beta$ such that it always find the query when it is in the video sequence, but never raises alerts when it is not.
\begin{table*}[t]
\caption{\textbf{Single dataset evaluations}. Results obtained by training and evaluating Re-ID approaches with the train and test splits of the same dataset. For each dataset, the best Re-ID approach is in bold.}
\label{tab:classic_reid_eval}
\begin{center}
\scalebox{0.9}{
\begin{tabular}{ccccccc}%
%\toprule[1.5pt]
\hline
Dataset & Approach & Rank-1 & Rank-5 & Rank-10 & mAP & mINP \\ %\toprule[1.5pt]%\hline
\hline
\multirow{4}{*}{CUHK03} & AGW & 0.73 & 0.88 & 0.92 & 0.72 & 0.63 \\
& MGN & \textbf{0.78} & \textbf{0.91} & \textbf{0.95} & \textbf{0.76} & \textbf{0.66} \\
& SBS & 0.74 & 0.89 & 0.93 & 0.73 & 0.62 \\
& BoT & 0.69 & 0.86 & 0.92 & 0.67 & 0.55 \\ \hline
\multirow{4}{*}{DukeMTMC} & AGW & 0.89 & 0.95 & \textbf{0.97} & 0.80 & 0.46 \\
& MGN & \textbf{0.91} & \textbf{0.96} & 0.97 & \textbf{0.82} & \textbf{0.47} \\
& SBS & 0.89 & 0.95 & 0.96 & 0.79 & 0.44 \\
& BoT & 0.87 & 0.94 & 0.96 & 0.77 & 0.41 \\ \hline
\multirow{4}{*}{Market-1501} & AGW & 0.95 & 0.99 & 0.99 & 0.88 & 0.66 \\
& MGN & \textbf{0.96} & \textbf{0.99} & \textbf{0.99} & \textbf{0.89} & \textbf{0.66} \\
& SBS & 0.95 & 0.98 & 0.99 & 0.88 & 0.66 \\
& BoT & 0.94 & 0.98 & 0.99 & 0.86 & 0.61 \\
%\toprule[1.5pt]%\hline
\hline
\end{tabular}
}
\end{center}
\end{table*}
The objective of this experiments is to see if the best approaches and datasets from previous experiments are also the best ones from the perspective of practical implementation in new cities.
Combined datasets experiments are also conducted for the live Re-ID setting. However, as PRID-2011 is not one of the training datasets used in our experiments, COMBINED$_{\text{all}}$ and COMBINED$_{\text{others}}$ are actually equivalent here and simply referred to as COMBINED. They are also compared against COMBINED$_{\text{scaled}}$ results to study the impact of dataset size and diversity. The COMBINED$_{\text{scaled}}$ training dataset for live Re-ID experiments on m-PRID are composed of 5507 images from CUHK03, 5508 from DukeMTMC and 5507 from Market-1501.
% \begin{enumerate}
% % \item For COMBINED$_{\text{SCALED-all}}$ we extracted for Train: For Query: 742 images from CUHK, 743 from Duke and 743 from Market.
% % Finally for Gallery: 5328 from CUHK, 6167 from Duke and 6167 from Market.
% \item For Market-1501 \& DukeMTMC $_{\text{SCALED}}$ we extracted for Train: 8261 from Duke and 8261 from Market. For Query: 1114 from Duke and 1114 from Market.
% Finally for Gallery: 8831 from Duke and 8830 from Market.
% \item For CUHK03 \& Market-1501 $_{\text{SCALED}}$ we extracted for Train: 6468 images from CUHK and 6468 from Market. For Query: 1400 images from CUHK and 1968 from Market.
% Finally for Gallery: 5328 from CUHK and 10585 from Market.
% \item For DukeMTMC \& CUHK03 $_{\text{SCALED}}$ we extracted for Train: 9754 images from Duke and 7368 from CUHK. For Query: 1114 images from Duke and 1114 from CUHK.
% Finally for Gallery: 12333 from Duke and 5328 from CUHK.
% \end{enumerate}
\section{\uppercase{Results}} \label{sec:results}
%In order to improve clarity, only a condensed version of the results is presented here. The complete results can be found in the appendix: \ref{sec:appendix_full_cross_results} contains all the results from cross-dataset evaluation, \ref{sec:appendix_full_liveReID_results} contains the missing metrics and the TVR vs FR curves for live Re-ID evaluations. Overall, the curated results presented in the core paper are representative of the complete results and are sufficient to draw our conclusions.
In order to improve clarity, only a condensed version of the results is presented. The complete results can be found at: \url{https://github.com/josemiki/benchmarking_person_Re_ID}. It contains all the results from cross-dataset evaluation, contains the missing metrics and the TVR vs FR curves for live Re-ID evaluations. Overall, the curated results presented in the core paper are representative of the complete results and are sufficient to draw our conclusions.
The results for single dataset evaluation are reported in Table~\ref{tab:classic_reid_eval}. The results obtained are good: rank-1 and mAP are around 70\% for the worst approach on the most difficult dataset. They are also relatively homogeneous: for each dataset-metric pairs, all four methods perform similarly (less than 10\% difference). The results obtained show that the tested approaches generalize differently to new contexts. For instance training MGN on Market-1501 leads to 47\% rank-10 accuracy on CUHK03, while the same experiment using BoT only reaches 15\%. For comparison, when training was conducted on CUHK03 itself, only a 3\% difference was observed between the approaches (Table~\ref{tab:classic_reid_eval}). The choice of the training dataset is also important, e.g., when training MGN for CUHK03, Market-1501 is 13\% better than DukeMTMC.
\begin{table*}[t]
\caption{\textbf{Cross-dataset evaluations}. Results obtained by training Re-ID approaches on one dataset and evaluating on another. For each evaluation dataset: the best Re-ID approach for a given dataset is in bold; the best training dataset for a given approach is in blue. R10 means Rank-10.}
\label{tab:cross_eval}
\begin{center}
%\renewcommand{\arraystretch}{0.8}%For line spacing 0.7
\scalebox{0.9}{
\begin{tabular}{cccc|cc|cc|cc}
% \Xhline{3\arrayrulewidth}
\hline
%\toprule[1.5pt]
\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Evaluation \\ dataset\end{tabular}} & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Training \\ dataset\end{tabular}} & \multicolumn{2}{c}{AGW} & \multicolumn{2}{c}{MGN} & \multicolumn{2}{c}{SBS} & \multicolumn{2}{c}{BoT} \\
& & R10 & mAP & R10 & mAP & R10 & mAP & R10 & mAP \\ %\midrule[1.5pt]%\Xhline{3\arrayrulewidth}
\hline
\multirow{5}{*}{CUHK03} & Market-1501 & 0.21 & 0.08 & \textbf{0.47} & \textbf{0.22} & 0.40 & 0.18 & 0.15 & 0.04 \\
& DukeMTMC & 0.18 & 0.06 & 0.34 & \textbf{0.14} & \textbf{0.35} & 0.13 & 0.15 & 0.05 \\
%\midrule
& COMBINED$_{\text{all}}$ & \textcolor{blue}{0.94} & \textcolor{blue}{0.71} & \textcolor{blue}{\textbf{0.96}} & \textcolor{blue}{\textbf{0.82}} & \textcolor{blue}{0.94} & \textcolor{blue}{0.76} & \textcolor{blue}{0.92} & \textcolor{blue}{0.68} \\
& COMBINED$_{\text{others}}$ & 0.32 & 0.14 & \textbf{0.55} & \textbf{0.27} & 0.52 & 0.24 & 0.28 & 0.11 \\
& COMBINED$_{\text{scaled}}$ & 0.31 & 0.13 & \textbf{0.52} & \textbf{0.23} & 0.46 & 0.20 & 0.23 & 0.09 \\
\hline
\multirow{5}{*}{DukeMTMC} & Market-1501 & 0.58 & 0.22 & \textbf{0.77} & \textbf{0.39} & 0.74 & 0.34 & 0.49 & 0.15 \\
& CUHK03 & 0.50 & 0.17 & \textbf{0.70} & \textbf{0.31} & 0.60 & 0.21 & 0.36 & 0.10 \\
& COMBINED$_{\text{all}}$ & \textcolor{blue}{0.96} & \textcolor{blue}{0.79} & \textcolor{blue}{\textbf{0.97}} & \textcolor{blue}{\textbf{0.82}} & \textcolor{blue}{0.96} & \textcolor{blue}{0.78} & \textcolor{blue}{0.96} & \textcolor{blue}{0.77}\\
& COMBINED$_{\text{others}}$ & 0.65 & 0.29 & \textbf{0.81} & \textbf{0.44} & 0.79 & 0.41 & 0.55 & 0.21 \\ %\midrule
& COMBINED$_{\text{scaled}}$ & 0.62 & 0.26 & \textbf{0.78} & \textbf{0.40} & 0.75 & 0.35 & 0.51 & 0.18 \\
\hline
\multirow{5}{*}{Market-1501} & DukeMTMC & 0.75 & 0.26 & \textbf{0.87} & \textbf{0.37} & 0.82 & 0.31 & 0.71 & 0.22 \\
& CUHK03 & 0.73 & 0.29 & \textbf{0.86} & \textbf{0.39} & 0.80 & 0.34 & 0.66 & 0.22 \\
& COMBINED$_{\text{all}}$ & \textcolor{blue}{0.99} & \textcolor{blue}{0.88} & \textcolor{blue}{\textbf{0.99}}& \textcolor{blue}{\textbf{0.91}} & \textcolor{blue}{0.99} & \textcolor{blue}{0.88} & \textcolor{blue}{0.99}& \textcolor{blue}{0.86} \\
& COMBINED$_{\text{others}}$ & 0.83 & 0.38 & \textbf{0.93} & \textbf{0.52}
& 0.91 & 0.47 & 0.80 & 0.34\\
& COMBINED$_{\text{scaled}}$ & 0.83 & 0.38 & \textbf{0.92} & \textbf{0.52} & 0.89 & 0.46 & 0.78 & 0.32 \\
%\midrule
\hline
\multirow{5}{*}{PRID-2011} & CUHK03 & 0.18 & 0.11 & \textbf{0.35} & \textbf{0.26} & 0.29 & 0.20 & 0.13 & 0.09 \\
& DukeMTMC & 0.20 & 0.12 & \textbf{0.42} & \textbf{0.30} & 0.26 & 0.17 & 0.16 & 0.07 \\
& Market-1501 & 0.26 & 0.19 & \textbf{0.40} & \textbf{0.28} & 0.30 & 0.20 & 0.23 & 0.13 \\
& COMBINED & \textcolor{blue}{0.32} & \textcolor{blue}{0.20} & \textbf{0.45} & \textbf{0.35} & 0.33 & 0.23 & \textcolor{blue}{0.24} & \textcolor{blue}{0.15} \\
& COMBINED$_{\text{scaled}}$ & 0.24 & 0.18 & \textcolor{blue}{\textbf{0.46}} & \textcolor{blue}{\textbf{0.36}} & \textcolor{blue}{0.36} & \textcolor{blue}{0.26} & 0.22 & 0.15 \\
%\bottomrule[1.5pt]%\Xhline{3\arrayrulewidth}
\hline
\end{tabular}
}
\end{center}
\end{table*}
The cross-dataset evaluation results are presented in Table~\ref{tab:cross_eval}. We only report Rank-10 for two reasons. First, the complete results show that the ranking of approaches is stable under different values of $n$. Second, having a high Rank-10 is more important than lower ranks for live Re-ID, as explained in Section~\ref{sec:related_live}.
%Also we combined two datasets for training this is COMBINED$_{\text{others}}$, for example when we are performing evaluation on CUHK03 we train on Market-1501, DukeMTMC and Combining both. As we can see in Table~\ref{tab:cross_eval} combining dataset improves the R10 and mAP in all approach's in the evaluation. As we explained before we used PRID\cite{prid2011} only for evaluation, as we can see in Table~\ref{tab:cross_eval} the best approach is MGN and there an influence in the dataset used for training. The best option is COMBINED$_{\text{all}}$ datasets and the second one is only using DukeMTMC. With this results we think that when we perform live Re-ID experiments the best results will be combining all datasets for training and with MGN approach.
% We conducted a complete pair cross-dataset experiments, are presented in Table~\ref{tab:cross_full_res_pair}. In this experiments we select two of the three datasets for training and evaluate in the other one and also over PRID. Here we found a first clue about the best option for training datasets, as we observe the results after train with DukeMTMC and CUHK03 and evaluate over PRID the approach MGN gets similar results as COMBINED$_{\text{all}}$ presented in Table~\ref{tab:cross_full_res_pair}. Both results leads over 45\% in Rank-10, we also observe that MGN evaluated on Market-1501 had the best result in Rank-10 with 93\%. It's possible that the best evaluation results on Market-1501 is due the quality of images provided by this dataset.
% This results are presented in Table~\ref{tab:cross_scal_full_res}, here we observe that train on COMBINED$_{\text{SCALED-all}}$ dataset and evaluate in PRID had better results in Rank-10 over 46\% and mAP over 36\% than same dataset without scaling as shown in Table~\ref{tab:cross_full_res_pair}. There is a same behavior when we train on DukeMTMC \& CUHK03 $_{\text{SCALED}}$ dataset and evaluate over Market-1501 had similar score in Rank-10 over 92\% than same dataset without scaling as shown in Table~\ref{tab:cross_full_res_pair}.
%Change columns per Row as table 3
Finally, the live Re-ID evaluation results are presented in Table~\ref{tab:live-reid}. They also illustrate that it is crucial to properly select the training dataset and approach for such task transfer. Overall, MGN appears to generalize much better for use in a live Re-ID setting. For training, Market-1501 appear to work best for most approaches except MGN. The best combination using a single dataset is MGN trained on DukeMTMC, reaching a mAP of 0.72 and an optimal F1 of 0.76.
\section{\uppercase{Discussion}} \label{sec:discussion}
In this section, we propose to discuss the results obtained in our study, highlighting several key insights regarding training standard Re-ID models for cross-domain live Re-ID.
\subsection{Impact of the training dataset}
Proper selection of the training dataset clearly influences the results in a different evaluation domain. However, there is no clear winner between Market-1501 and DukeMTMC to know which individual dataset should be used for any context. In addition, the cross-dataset results do not allow to choose the best individual dataset for training models for the live Re-ID setting. Indeed, Table~\ref{tab:cross_eval} suggested that the best dataset for MGN should be Market-1501, whereas it is outperformed by DukeMTMC for live Re-ID (Table~\ref{tab:live-reid}). In the remaining of this section, we discuss the results obtained on the combined datasets settings to gain new insights regarding building standard Re-ID datasets for efficient training of live Re-ID models.
\subsubsection{Can data from a different domain improve results in the standard Re-ID scenario ?}
To answer this question, we compare results from Table~\ref{tab:classic_reid_eval} and the COMBINED$_{\text{all}}$ rows in Table~\ref{tab:cross_eval}. Overall, for both Rank-10 and mAP, the results for COMBINED$_{\text{all}}$ appear slightly better than the results obtained when learning only on the training set of the evaluated dataset. To illustrate this, we computed the mean and standard deviation across all evaluation dataset and approaches. When using only the training set we obtain the following results: $R10 = 0.962 \pm 0.026$ and $mAP = 0.798 \pm 0.068$. When combining the three available datasets for training we have: $R10 = 0.964 \pm 0.022$ and $mAP = 0.805 \pm 0.067$.
\begin{table*}[t]
\begin{center}
\caption{\textbf{Live Re-ID evaluation}. Results obtained by training Re-ID approaches on one standard Re-ID dataset and evaluating on m-PRID for the live Re-ID setting. For each training dataset, the best approach is in bold and for each approach, the best dataset is in blue.}
\label{tab:live-reid}
%\renewcommand{\arraystretch}{0.8}%For line spacing 0.7
\scalebox{0.9}{%For scale this table 0.8
\begin{tabular}{ccc|cc|cc|cc|cc}
%\toprule[1.5pt]%\cline{2-7}
\hline
\multirow{2}{*}{Approach}& \multicolumn{2}{c}{CUHK03} & \multicolumn{2}{c}{DukeMTMC} & \multicolumn{2}{c}{Market-1501} & \multicolumn{2}{c}{COMBINED} & \multicolumn{2}{c}{COMBINED$_{\text{scaled}}$} \\
& $F_{\text{1}}^*$ & mAP & $F_{\text{1}}^*$ & mAP & $F_{\text{1}}^*$ & mAP & $F_{\text{1}}^*$ & mAP & $F_{\text{1}}^*$ & mAP\\
%\midrule[1.5pt]
\hline
AGW & 0.39 & 0.23 & 0.40 & 0.25 & 0.46 & 0.33 & \textcolor{blue}{0.56} & \textcolor{blue}{0.49} & 0.49 & 0.39 \\
BoT & 0.27 & 0.10 & 0.40 & 0.22 & \textcolor{blue}{0.47} & \textcolor{blue}{0.32} & 0.45 & 0.30 & 0.44 & 0.31\\
SBS & 0.51 & 0.43 & 0.58 & 0.54 & 0.60 & 0.50 & \textcolor{blue}{0.71} & \textcolor{blue}{0.71} & 0.68 & 0.72\\
MGN & \textbf{0.66} & \textbf{0.60} & \textbf{0.76} & \textbf{0.72} & \textbf{0.69} & \textbf{0.63} & \textcolor{blue}{\textbf{0.81}} & \textcolor{blue}{\textbf{0.80}} & \textbf{0.77} & \textbf{0.75}\\
\hline
%\bottomrule[1.5pt]
\end{tabular}
}
\end{center}
\end{table*}
To confirm this intuition, we conduct a Paired Sample T-Test to determine whether the mean difference between the results obtained using the single in-domain training set and the COMBINED$_{\text{all}}$ are statistically significant. The p-values obtained are $0.2750$ for R10 and $0.2313$ for mAP, suggesting that the Null Hypothesis cannot be rejected, i.e., \emph{we cannot conclude that using more data from a different domain is beneficial to the standard Re-ID training process}.
% Which standard Re-ID metrics are best suited to predict the performance of a given model for cross-domain scenarios (including live Re-ID)} ?
% Across our large experiments conducted in this study we think that \textit{Rank-10} and \textit{mAP} are the best metrics for predict performance for cross-domain scenarios. Because when you take the train dataset with highest values from the evaluation over classic PRID for example in Table~\ref{tab:cross_full_res} is COMBINED$_{\text{all}}$ and compare with same dataset used for evaluation of m-PRID in Table~\ref{tab:live_full_res} had good results too and in this case it's the best option for train.
\subsubsection{Between dataset size and diversity, which is most important for cross-domain generalization ?}
The first question we want to answer is whether combining datasets from different domains can help cross domain generalization. To evaluate this, we can compare the results for COMBINED$_{\text{others}}$ (COMBINED for PRID-2011) against the results from the best individual dataset in Table~\ref{tab:cross_eval}. The mean and standard deviation across all evaluation dataset and approaches are $R10 = 0.509 \pm 0.242$ and $mAP = 0.224 \pm 0.099$ for the best individual dataset, and $R10 = 0.580 \pm 0.241$ and $mAP = 0.297 \pm 0.124$ when combining all the available training datasets (except the one corresponding to the evaluated test set). The Paired Sample T-Test gives p-values of $0.0001$ for both R10 and mAP, which is extremely statistically significant. In other words, our experiments confirm that \emph{combining several training datasets from different domains allows to train Re-ID models that generalize better to new unknown domains}.
We then want to know if simply increasing the diversity in the training dataset without increasing its size also help for cross domain generalization. To evaluate this, we can compare the results for COMBINED$_{\text{scaled}}$ against the results from the best individual dataset in Table~\ref{tab:cross_eval}. As a reminder, COMBINED$_{\text{scaled}}$ consists in building a training dataset by taking data from all available training sets (except the one corresponding to the evaluated test set) in such a way that the total number of training data does not exceed that size of the largest individual training set. The mean and standard deviation across all evaluation dataset and approaches are $R10 = 0.509 \pm 0.242$ and $mAP = 0.224 \pm 0.099$ for the best individual dataset, and $R10 = 0.555 \pm 0.245$ and $mAP = 0.279 \pm 0.125$ for COMBINED$_{\text{scaled}}$. The Paired Sample T-Test gives p-values of $0.0001$ for R10 and $0.0005$ for mAP, which is extremely statistically significant. In other words, our experiments confirm that \emph{increasing diversity in training dataset, even without increasing its size, allows to train Re-ID models that generalize better to new domains}.
In view of the two encouraging results presented above, we now want to know whether the size of the training dataset is actually helping cross-domain generalization or if adding diversity is actually sufficient. To evaluate this, we can compare the results for COMBINED$_{\text{others}}$ against the results for COMBINED$_{\text{scaled}}$ in Table~\ref{tab:cross_eval}. The mean and standard deviation across all evaluation dataset and approaches are $R10 = 0.580 \pm 0.241$ and $mAP = 0.297 \pm 0.124$ for COMBINED$_{\text{others}}$, and $R10 = 0.555 \pm 0.245$ and $mAP = 0.279 \pm 0.125$ for COMBINED$_{\text{scaled}}$. The Paired Sample T-Test gives p-values of $0.0020$ for R10 and $0.0075$ for mAP, which is very statistically significant. In other words, our experiments confirm that \emph{adding more data from domains that are already present in the training set help generalization to new unknown domains}.
\subsubsection{live Re-ID results}
The live ReID results on m-PRID (Table~\ref{tab:live-reid}) confirm the conclusions drawn from the cross-dataset experiments. In particular, the COMBINED$_{\text{scaled}}$ results appear better than the results with a single training set, suggesting the importance of training data diversity for practical live Re-ID implementation in a new context. The COMBINED results are themselves better than COMBINED$_{\text{scaled}}$, which suggest that one should use all the available data to train a good standard Re-ID model for live Re-ID implementation. Finally, we emphasize the good results obtained by training MGN on the COMBINED training dataset. These results are very encouraging after the pessimistic results reported by Sumari et al.\cite{application_level_ReID} for live Re-ID.
\subsection{Impact of the standard Re-ID approaches}
All the approaches tested in this study perform well in the single dataset scenario. However, when it comes to generalization to live operation contexts, MGN has a clear advantage against the other three techniques. This conclusion could already be intuited from the cross-dataset experiments, which suggests a simple yet powerful approach to test future standard Re-ID approaches before live deployment. MGN is the only approach involving a specific image splitting, forcing the network to focus on different body part. In view of our results, this property appears to be desirable for generalization to the live Re-ID setting.
Besides MGN, the SBS approach also appear to present much better generalization than its competitors (Table~\ref{tab:cross_eval} and \ref{tab:live-reid}). Hence, a promising research direction for live Re-ID research would be to design a new standard Re-ID architecture combining features from MGN and SBS, as described in Section~\ref{sec:reid_approaches}.
% \begin{itemize}
% \item We have conducted a large quantity of experiments to determine if there is an influence when we train an approach with some benchmark datasets. Unfortunately due size of this table with all metrics proposed in most of the state-of-the-art we put at disposal this results as an Appendix \ref{sec:appendix_full_cross_results}. From the table \ref{tab:cross_full_res} we can conclude that the best approach with highest Rank-1 was MGN which was trained with DukeMTMC dataset and evaluated in Market-1501. Also, when the same approach was evaluated in CUHK03 dataset we see that Rank-1 decrease significantly even that has the highest score compared with the other three approaches. %Make a large result table in the appendix with ALL the results we have.
% \item Before start making cross-evaluation, we want to be sure if the four approaches from the state-of-the-art had good results when they are trained and evaluated in the same domain. As you can see in table \ref{tab:classic_reid_eval}, all approaches obtain more than 77\% in Rank-1 metric and with two datasets that are the larger ones the results are over 86\%. The highest Rank-1 and mAP are obtained when we train all approaches with Market-1501. We can derive that this is a good dataset for training. %Make a table for standard evaluation results. Comment these results, which method is the best regarding different metrics ?
% \item After single evaluation, we conduct a Cross-dataset evaluation. So we train our approaches in one dataset and evaluate it with other two. in table \ref{tab:cross_eval}, we found that the best datasets for training were CUHK03 and DukeMTMC when are avaluated over Market-1501 because they obtain highest Rank 10. This was a surprise because we thought that if we train on Market-1501 as single evaluation, we would have the highest Rank-10 scores on evaluation over the other two datasets. That means that when you want to apply a Re-Id approach in live scenarios you need to make an evaluation for all the approaches that you want to use, because single and cross validation aren't enough to take a decision. But here we have a confirmation that MGN approach once more shows that is strong and have the highest Rank-10 and mAP over all the experiments. %Make a table for cross-dataset results. Explain why you evaluate them under the particular metrics chosen (rank1 and rank 5). Do the results correspond with what was seen in the standard evaluation ? Is there surprises in the results ?
% \item Finally, in table \ref{tab:live-reid} we evaluate the four approaches with different dataset for training but all evaluated over FF PRID pipeline. We can conlude here that MGN approach is the best one to choose as Re-ID approach and for this case is better to train it with DukeMTMC because you get the highest $F_1$ and mAP results. %Make a table for FF PRID evaluation results. Compare the results from cross dataset. Any thing surprising? Can we choose a good dataset - approach pair for future implementation?
% \end{itemize}
% \begin{itemize}
% \item Is the proposed evaluation approach useful (what knowledge did we gain from this evaluation approach compared to the standard evaluation or the simple cross-dataset evaluation) ?
% \item How the results can help to design better Re-ID pipelines in the future? To choose datasets and approaches in a better/more informed way.
% \end{itemize}
\section{\uppercase{Conclusion}} \label{sec:conclusion}
\subsection{Overview}
This paper presents a comprehensive evaluation methodology to benchmark different standard Re-ID approaches and training datasets with respect to their ability to be deployed in practical applications from a different context. To do so, we first formalized the new live Re-ID setting, and define new unified evaluation metrics to facilitate interpretation. The performance of different standard Re-ID models is evaluated on this setting. We also conduct simple cross-dataset experiments to see if it can be used to predict which datasets and approaches will generalize better to the live Re-ID setting.
The main conclusions from this study are:
\begin{enumerate}
\item Although previous work reported very pessimistic results,\cite{application_level_ReID} our experiments showed that it is possible to design good live Re-ID pipelines by properly choosing the standard Re-ID model and combining publicly available training dataset.
\item Proper choice of the standard re-ID approach and training dataset can influence greatly the results when transferring the model to the cross-domain live Re-ID setting.
\item Increasing training dataset diversity helps generalization to the cross-domain live Re-ID setting.
\item Increasing training dataset size allows to improve cross-domain generalization even further.
\item Simple cross-dataset evaluation can be used to quickly assess the generalization performance of future standard Re-ID techniques for live Re-ID.
\end{enumerate}
Although we only studied the straightforward transfer strategy without fine-tuning, we believe that the results presented here can serve as a good starting point to develop better live Re-ID models in the future.
% The results obtained suggest that it is possible to build a good live Re-ID pipeline by simply concatenating a pedestrian detection model and a standard Re-ID model. However, the proper choice of a good Re-ID approach and the appropriate corresponding training dataset is paramount to obtain satisfying results. The simple cross-dataset experiments presented in this paper allow to quickly evaluate the transferability of a given approach, but it fails to assess the best training dataset. To evaluate other training datasets for practical Re-ID implementations, one need to go through the steps of evaluating the learned models on the live Re-ID setting. %The proposed benchmarking methodology can be used straightforwardly to evaluate new datasets and approaches in the future.
% \begin{itemize}
% \item Recap the entire benchmark + results presented in this paper.
% \item Guidelines obtained from these results.
% \item The proposed benchmarking methodology can be used straightforwardly to evaluate new datasets and new re-id approaches in the future.
% \end{itemize}
\subsection{Future work}
The outputs of this study suggest several interesting research directions. First, it would be very valuable to build new live Re-ID datasets, allowing not only to confirm the results obtained in this study, but also to see if good live Re-ID performance is consistent across different scenarios. Then, this benchmark experiment can be extended to account for different pedestrian detectors, another important component of the live Re-ID pipeline. In particular, it would be interesting to study if specific Re-ID approaches combine better with specific object detectors.
The evaluation methodology proposed in this paper could help to answer this question. %In this paper we used only YOLO to extract people BB. Could this have an influence on the final results? Do some Re-ID approaches combine better with some OD models? The evaluation methodology proposed in this paper could be mimicked to answer this question).
Another valuable contribution would be to create a ready-to-use website implementing the proposed benchmarking methodology for researchers to test their new approaches easily. % In view of the results obtained, it could also be interesting to try to combine public datasets to train Re-ID models that are better suited for live Re-ID.
It would also be interesting to see if existing unsupervised cross-dataset adaptation methods could help generalization to the live Re-ID setting. Finally, it would be interesting to study how the good design choices identified in this study can be leveraged to develop successful end-to-end approaches for live Re-ID.
\bibliographystyle{apalike}
{\footnotesize
\bibliography{example}}
%\section*{\uppercase{Appendix}}
%%If any, the appendix should appear directly after the
%references without numbering, and not on a new page. To do so please use the following command:
%\textit{$\backslash$section*\{APPENDIX\}}
\end{document}
\begin{thebibliography}{}
\bibitem[Abadi et~al., 2015]{tensorflow2015-whitepaper}
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado,
G.~S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp,
A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M.,
Levenberg, J., Man\'{e}, D., Monga, R., Moore, S., Murray, D., Olah, C.,
Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P.,
Vanhoucke, V., Vasudevan, V., Vi\'{e}gas, F., Vinyals, O., Warden, P.,
Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. (2015).
\newblock {TensorFlow}: Large-scale machine learning on heterogeneous systems.
\newblock Software available from tensorflow.org.
\bibitem[Altunay et~al., 2018]{abandoned_luggage}
Altunay, D.~G., Karademir, N., Top{\c{c}}u, O., and Direko{\u{g}}lu, C. (2018).
\newblock Intelligent surveillance system for abandoned luggage.
\newblock In {\em 26th Signal Processing and Communications Applications Conf.
(SIU)}, pages 1--4. IEEE.
\bibitem[Bedagkar-Gala and Shah, 2014]{old_reID_survey}
Bedagkar-Gala, A. and Shah, S.~K. (2014).
\newblock A survey of approaches and trends in person re-identification.
\newblock {\em Image and vision computing}, 32(4):270--286.
\bibitem[Chen et~al., 2018]{chen2018deep}
Chen, H., Wang, Y., Shi, Y., Yan, K., Geng, M., Tian, Y., and Xiang, T. (2018).
\newblock Deep transfer learning for person re-identification.
\newblock In {\em 2018 IEEE Fourth International Conference on Multimedia Big
Data (BigMM)}, pages 1--5. IEEE.
\bibitem[Deb et~al., 2021]{missing_children}
Deb, D., Aggarwal, D., and Jain, A.~K. (2021).
\newblock Identifying missing children: Face age-progression via deep feature
aging.
\newblock In {\em 25th International Conference on Pattern Recognition (ICPR)},
pages 10540--10547. IEEE.
\bibitem[Fan et~al., 2019]{Fan_2019}
Fan, X., Jiang, W., Luo, H., and Fei, M. (2019).
\newblock Spherereid: Deep hypersphere manifold embedding for person
re-identification.
\newblock {\em Journal of Visual Communication and Image Representation},
60:51–58.
\bibitem[Felzenszwalb et~al., 2009]{DPM}
Felzenszwalb, P.~F., Girshick, R.~B., McAllester, D., and Ramanan, D. (2009).
\newblock Object detection with discriminatively trained part-based models.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
32(9):1627--1645.
\bibitem[Gheissari et~al., 2006]{standard_reID_first_paper}
Gheissari, N., Sebastian, T.~B., and Hartley, R. (2006).
\newblock Person reidentification using spatiotemporal appearance.
\newblock In {\em 2006 IEEE computer society conference on computer vision and
pattern recognition (CVPR'06)}, volume~2, pages 1528--1535. IEEE.
\bibitem[Gou et~al., 2018]{benchmark_reID_single_multi}
Gou, M., Wu, Z., Rates-Borras, A., Camps, O., Radke, R.~J., et~al. (2018).
\newblock A systematic evaluation and benchmark for person re-identification:
Features, metrics, and datasets.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
41(3):523--536.
\bibitem[Gu{\'e}rin et~al., 2020]{icmla_joris}
Gu{\'e}rin, J., de~Paula~Canuto, A.~M., and Goncalves, L. M.~G. (2020).
\newblock Robust detection of objects under periodic motion with gaussian
process filtering.
\newblock In {\em 2020 19th IEEE International Conference on Machine Learning
and Applications (ICMLA)}, pages 685--692. IEEE.
\bibitem[He et~al., 2015]{he_deep_2015}
He, K., Zhang, X., Ren, S., and Sun, J. (2015).
\newblock Deep {Residual} {Learning} for {Image} {Recognition}.
\newblock {\em arXiv:1512.03385 [cs]}.
\newblock arXiv: 1512.03385.
\bibitem[He et~al., 2020]{benchmark_reID_cross}
He, L., Liao, X., Liu, W., Liu, X., Cheng, P., and Mei, T. (2020).
\newblock Fastreid: A pytorch toolbox for general instance re-identification.
\newblock {\em arXiv preprint arXiv:2006.02631}.
\bibitem[Hirzer et~al., 2011]{prid2011}
Hirzer, M., Beleznai, C., Roth, P.~M., and Bischof, H. (2011).
\newblock Person re-identification by descriptive and discriminative
classification.
\newblock In {\em Scandinavian conference on Image analysis}, pages 91--102.
Springer.
\bibitem[Islam, 2020]{person_search_survey}
Islam, K. (2020).
\newblock Person search: New paradigm of person re-identification: A survey and
outlook of recent works.
\newblock {\em Image and Vision Computing}, 101:103970.
\bibitem[Lavi et~al., 2020]{standard_reID_survey}
Lavi, B., Ullah, I., Fatan, M., and Rocha, A. (2020).
\newblock Survey on reliable deep learning-based person re-identification
models: Are we there yet?
\newblock {\em arXiv preprint arXiv:2005.00355}.
\bibitem[Leng et~al., 2019]{open_set_survey}
Leng, Q., Ye, M., and Tian, Q. (2019).
\newblock A survey of open-world person re-identification.
\newblock {\em IEEE Transactions on Circuits and Systems for Video Technology},
30(4):1092--1108.
\bibitem[Li et~al., 2014]{cuhk}
Li, W., Zhao, R., Xiao, T., and Wang, X. (2014).
\newblock Deepreid: Deep filter pairing neural network for person
re-identification.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 152--159.
\bibitem[Liao et~al., 2014]{open_set_first_paper}
Liao, S., Mo, Z., Zhu, J., Hu, Y., and Li, S.~Z. (2014).
\newblock Open-set person re-identification.
\newblock {\em arXiv preprint arXiv:1408.0872}.
\bibitem[Lin et~al., 2015]{lin2015microsoft}
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J.,
Perona, P., Ramanan, D., Zitnick, C.~L., and Dollár, P. (2015).
\newblock Microsoft coco: Common objects in context.
\bibitem[Luo et~al., 2019]{luo_bag_2019}
Luo, H., Gu, Y., Liao, X., Lai, S., and Jiang, W. (2019).
\newblock Bag of {Tricks} and a {Strong} {Baseline} for {Deep} {Person}
{Re}-{Identification}.
\newblock In {\em 2019 {IEEE}/{CVF} {Conference} on {Computer} {Vision} and
{Pattern} {Recognition} {Workshops} ({CVPRW})}, pages 1487--1495, Long Beach,
CA, USA. IEEE.
\bibitem[Luo et~al., 2020]{luo_strong_2020}
Luo, H., Jiang, W., Gu, Y., Liu, F., Liao, X., Lai, S., and Gu, J. (2020).
\newblock A {Strong} {Baseline} and {Batch} {Normalization} {Neck} for {Deep}
{Person} {Re}-identification.
\newblock {\em IEEE Transactions on Multimedia}, 22(10):2597--2609.
\newblock arXiv: 1906.08332.
\bibitem[Machaca et~al., 2022]{machaca2022trade}
Machaca, L., Huaman, J., Clua, E., Guerin, J., et~al. (2022).
\newblock {TrADe Re-ID}--live person re-identification using tracking and
anomaly detection.
\newblock {\em 21st IEEE International Conference on Machine Learning and
Applications (to appear)}.
\bibitem[Mekhazni et~al., 2020]{mekhazni2020unsupervised}
Mekhazni, D., Bhuiyan, A., Ekladious, G., and Granger, E. (2020).
\newblock Unsupervised domain adaptation in the dissimilarity space for person
re-identification.
\newblock In {\em European Conference on Computer Vision}, pages 159--174.
Springer.
\bibitem[Moon and Phillips, 2001]{cmc_curve}
Moon, H. and Phillips, P.~J. (2001).
\newblock Computational and performance aspects of pca-based face-recognition
algorithms.
\newblock {\em Perception}, 30(3):303--321.
\bibitem[Moskvyak et~al., 2021]{semisupervised_reid}
Moskvyak, O., Maire, F., Dayoub, F., and Baktashmotlagh, M. (2021).
\newblock Going deeper into semi-supervised person re-identification.
\newblock {\em arXiv preprint arXiv:2107.11566}.
\bibitem[{Papers with Code}, 2021]{papers_with_code}
{Papers with Code} (2021).
\newblock person {Re-ID}.
\newblock \url{paperswithcode.com/task/person-re-identification}.
\bibitem[Redmon and Farhadi, 2018]{redmon2018yolov3}
Redmon, J. and Farhadi, A. (2018).
\newblock Yolov3: An incremental improvement.
\newblock {\em arXiv preprint arXiv:1804.02767}.
\bibitem[Ristani et~al., 2016]{duke}
Ristani, E., Solera, F., Zou, R., Cucchiara, R., and Tomasi, C. (2016).
\newblock Performance measures and a data set for multi-target, multi-camera
tracking.
\newblock In {\em European conference on computer vision}, pages 17--35.
Springer.
\bibitem[Sumari et~al., 2020]{application_level_ReID}
Sumari, F.~O., Machaca, L., Huaman, J., Clua, E.~W., and Gu{\'e}rin, J. (2020).
\newblock Towards practical implementations of person re-identification from
full video frames.
\newblock {\em Pattern Recognition Letters}, 138:513--519.
\bibitem[Szegedy et~al., 2015]{szegedy_rethinking_2015}
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015).
\newblock Rethinking the {Inception} {Architecture} for {Computer} {Vision}.
\newblock {\em arXiv:1512.00567 [cs]}.
\newblock arXiv: 1512.00567.
\bibitem[Wang et~al., 2018]{wang_learning_2018}
Wang, G., Yuan, Y., Chen, X., Li, J., and Zhou, X. (2018).
\newblock Learning {Discriminative} {Features} with {Multiple} {Granularities}
for {Person} {Re}-{Identification}.
\newblock {\em Proceedings of the 26th ACM international conference on
Multimedia}, pages 274--282.
\newblock arXiv: 1804.01438 version: 1.
\bibitem[Wang et~al., 2016]{human_in_the_loop_reid}
Wang, H., Gong, S., Zhu, X., and Xiang, T. (2016).
\newblock Human-in-the-loop person re-identification.
\newblock In {\em European conference on computer vision}, pages 405--422.
Springer.
\bibitem[Xiao et~al., 2017]{person_search_explain}
Xiao, T., Li, S., Wang, B., Lin, L., and Wang, X. (2017).
\newblock Joint detection and identification feature learning for person
search.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 3415--3424.
\bibitem[Yang et~al., 2021]{unsupervised_reid}
Yang, C., Qi, F., and Jia, H. (2021).
\newblock Survey on unsupervised techniques for person re-identification.
\newblock In {\em 2021 2nd International Conference on Computing and Data
Science (CDS)}, pages 161--164. IEEE.
\bibitem[Ye et~al., 2021]{standard_reID_survey2}
Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., and Hoi, S.~C. (2021).
\newblock Deep learning for person re-identification: A survey and outlook.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence}.
\bibitem[Zhao et~al., 2020]{zhao2020unsupervised}
Zhao, F., Liao, S., Xie, G.-S., Zhao, J., Zhang, K., and Shao, L. (2020).
\newblock Unsupervised domain adaptation with noise resistible mutual-training
for person re-identification.
\newblock In {\em European Conference on Computer Vision}, pages 526--544.
Springer.
\bibitem[Zheng et~al., 2016]{mars}
Zheng, L., Bie, Z., Sun, Y., Wang, J., Su, C., Wang, S., and Tian, Q. (2016).
\newblock Mars: A video benchmark for large-scale person re-identification.
\newblock In {\em European Conference on Computer Vision}, pages 868--884.
Springer.
\bibitem[Zheng et~al., 2015a]{market}
Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., and Tian, Q. (2015a).
\newblock Scalable person re-identification: A benchmark.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision}, pages 1116--1124.
\bibitem[Zheng et~al., 2015b]{benchmark1}
Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., and Tian, Q. (2015b).
\newblock Scalable person re-identification: A benchmark.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision}, pages 1116--1124.
\bibitem[Zheng et~al., 2017]{benchmark_OD_reID}
Zheng, L., Zhang, H., Sun, S., Chandraker, M., Yang, Y., and Tian, Q. (2017).
\newblock Person re-identification in the wild.
\newblock In {\em Proceedings IEEE Conference on Computer Vision and Pattern
Recognition}, pages 1367--1376.
\bibitem[Zhong et~al., 2017]{zhong_random_2017}
Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y. (2017).
\newblock Random {Erasing} {Data} {Augmentation}.
\newblock {\em arXiv:1708.04896 [cs]}.
\newblock arXiv: 1708.04896.
\bibitem[Zhuang et~al., 2020]{federated_ReID}
Zhuang, W., Wen, Y., Zhang, X., Gan, X., Yin, D., Zhou, D., Zhang, S., and Yi,
S. (2020).
\newblock Performance optimization of federated person re-identification via
benchmark analysis.
\newblock In {\em Proceedings of the 28th ACM International Conference on
Multimedia}, pages 955--963.
\end{thebibliography}