Morphable Detector for Object Detection on Demand
Extracted source text · arXiv 2110.04917
Many emerging applications of intelligent robots need to explore and understand new environments, where it is desirable to detect objects of novel classes on the fly with minimum online efforts. This is an object detection on demand (ODOD) task. It is challenging, because it is impossible to annotate a large number of data on the fly, and the embedded systems are usually unable to perform back-propagation which is essential for training. Most existing few-shot detection methods are confronted here as they need extra training. We propose a novel morphable detector (MD), that simply ``morphs" some of its changeable parameters online estimated from the few samples, so as to detect novel classes without any extra training. The MD has two sets of parameters, one for the feature embedding and the other for class representation (called ``prototypes"). Each class is associated with a hidden prototype to be learned by integrating the visual and semantic embeddings. The learning of the MD is based on the alternate learning of the feature embedding and the prototypes in an EM-like approach which allows the recovery of an unknown prototype from a few samples of a novel class.
Once an MD is learned, it is able to use a few samples of a novel class to directly compute its prototype to fulfill the online morphing process. We have shown the superiority of the MD in Pascal~\cite{everingham2010pascal}, COCO~\cite{lin2014microsoft} and FSOD~\cite{fan2020fsod} datasets. The code is available https://github.com/Zhaoxiangyun/Morphable-Detector.
\vspace{-3mm}
\section{Conclusion}
In this paper, we focus on a very challenging task: object detection on demand (ODOD) task. The prevailing FSD methods can not solve this problem well as ODOD requires no extra training. We propose a novel morphable detector (MD), that simply ``morphs" some of its changeable parameters online estimated from the few samples, so as to detect novel categories without any extra training. The learning of the MD is based on the alternative learning of the feature embedding and the prototypes in an EM-like approach which allows better recovery of an unknown prototype from a few samples of a novel category. Extensive experiments are performed to demonstrate the superiority of the MD.
\paragraph{Acknowledgements} We thank Pengbo Zhao for the suggestions on the writing. This work was supported in part by National Science Foundation grant IIS-1619078, IIS-1815561, and IIS-2007613.
% Object Detection on Demand which is defined as detecting the novel categories without extra training. We present a morphable detector which is trained using the seen category data and can be online morphed to detect novel categories given a few novel samples. In the MD, each category is assoticated with a prototype. The morphable detector learns a feature embedding where the classification can be performed by computing the similarity with different prototypes. In training, we use the semantic vectors as initial prototypes and propose to recursively optimize the network parameters and prototypes alternately. %Extensive experiments has shown %to improving the prototypes from the network training recursively. The prototypes and network parameters are optimized alternatively to benefit each other mutually. In each iteration, prototypes are refined by a joint visual-semantic embedding that uses the visual and semantic information. % Extensive experiments are performed to verify the effectiveness of the proposed method.
\section{Experiments}
%We explore the use of our meta-learning based approach
%“MetaDet” for few-shot detection tasks. We begin with extensive evaluation on the PASCAL VOC dataset benchmarking the performance of CNN based detectors in
%the small-sample size regime and addressing variants and
%different design choices of MetaDet. We then evaluate on
%challenging, large-scale MS-COCO [36], ImageNet [47],
To evaluate our morphable detector (MD) on Object Detection on Demand (ODOD), we begin with an evaluation on a challenging large-scale dataset FSOD~\cite{fan2020fsod} which benchmarks the performance of detectors on few-shot detection setting. Then we evaluate on two widely used datasets MS COCO~\cite{lin2014microsoft} and Pascal VOC datasets~\cite{everingham2010pascal}, and compare against state-of-the-arts on few-shot detection setting. Finally, as a by-product, we compare against state-of-the-arts on zero-shot detection setting. %We build our MD on Faster R-CNN~\cite{ren2015faster} detection framework. %We select 15 and 70 classes as predefined classes in Pascal VOC and COCO separately to stimulate the ODOD. The remaining 5 and 10 classes are treated as online specified novel classes. The detector is first trained using all predefined class data where the novel classes samples are excluded so that the model does not see the novel classes. When the novel classes are online specified, the morpable detector is morphed to detect all novel and predefined classes.
%\subsection{Experiment Setup}
%We used ResNet-101~\cite{he2016deep} backbone as our baseline model. The Region Proposal Network and box regression are from Faster RCNN~\cite{ren2015faster}. We implement all our models with PyTorch\footnote{We use the following code base as the starting point for our implementation: \url{https://github.com/facebookresearch/maskrcnn-benchmark}}. We use class-agnostic box regression, so we do not consider getting separate box regression branches for the novel classes. The model is pretrained from ImageNet~\cite{deng2009imagenet}. The weights of the mapping $f_a$ and $f_b$ (i.e. $\Theta_a$ and $\Theta_b$ in Eq 3.) are initialized from Gaussian distributions with mean value 0 and std 0.01. When training the mophable detector, we train 100,000 and 50,000 iterations for VOC and COCO with batch size 4 with a learning rate 0.002. The size of the attribute embedding is 200. In evaluation, we mostly use standard $mAP^{0.5}$ as our evaluation metric following most detection frameworks~\cite{ren2015faster,lin2017focal}. The experiments are performed under the condition that one example is given unless indicated. Since the performance varies when choosing different samples of the novel classes. We randomly select the different samples five times and report the average performance. %For zero-shot detection on MS-COCO, we follow~\cite{bansal2018zero,li2019zero} that recall@100 is adopted as the evaluation metric.
%We used VOC 2007, VOC 2012, and MS-COCO to validate the proposed model.
%Note that since indoor and outdoor classes are less semantic related, we do not select classes from them. We evaluate the detector in two different source/target splits.
%\subsection{Different choices of the one example effects}
%In ODOD, one example is given when the novel classes are specified. In this section, we study how the choice of the one example given influence the performance. We randomly sample different examples of the novel
%classes three times. In table, three difference choices lead to performance difference. This indicate that the ODOD is sensitive to the sample choosing. This is . Regarding the predefined classes, the performance slightly changes, this indicates that the performance of the predefined classes is not sensitive to the choice of the example.
\subsection{Experiments on FSOD dataset}
\paragraph{Dataset and implementation} Few Shot Object Detection (FSOD)~\cite{fan2020fsod} dataset is proposed to evaluate the detector which is trained using base class data and evaluated on the novel classes. This dataset contains 1000 classes
with 800/200 split for training and test set respectively. There is no overlap between the training and test classes. There are 52350 images with 147489 annotated boxes in the training set and 14152 images with 35102 annotated boxes in the test set. We use the Region Proposal Network and class-agnostic regression from Faster-RCNN in our model. Following the training strategies in ~\cite{fan2020fsod}, we use ResNet-50 as our backbone and pretrain the model on the COCO dataset. Then we train the model using the base class data which contains 800 classes and test on the test set which contains 200 novel classes. We randomly select 5 samples for each novel class as known samples for the novel classes as ~\cite{fan2020fsod}. We train the model with batch size 4 for 50k iterations with a learning rate 0.002 and another 20k iterations with a learning rate 0.0002. All models are evaluated using standard ${\rm AP_{0.5}}$ and ${\rm AP}$. The dimension size of the feature embedding and semantic vectors~\footnote{We use the extracted semantic vectors from \url{https://github.com/agnusmaximus/Word2Bits}} is 200.
\vspace{-3mm}
\paragraph{Comparison with baselines}
We first compare against different baselines which use different prototype initializations. Then, we evaluate the MD model after each iteration.
\begin{itemize}
\item \textbf{Visual (ImageNet/FRCNN)}. In our MD, the prototypes are initialized using semantic vectors. In the experiments, we compare against MD variants using visual features for prototype initialization. To get the visual features for each class, we forward the base class ground truth samples through a trained Faster-RCNN model or ImageNet pre-trained model to get the visual features whose dimension is 1024 in our experiment. Then we use the mean feature vectors for each class as the prototype for that class to train the MD. %After the MD is trained, we follow the same online morphing strategy given a few novel samples.%used as the prototype for that class. Then we train our MD using visual prototypes.
% \item \textbf{Semantic prototype}. We use the semantic vectors as prototypes to train the model.
\item \textbf{Iterative results}. We evaluate the MD model's performance after each iteration.
\end{itemize}
Table~\ref{tab:fsod} summarizes the comparisons against the baselines. Directly using ImageNet~\cite{deng2009imagenet} or FRCNN~\cite{ren2015faster} features as prototypes does not work well. The results show that the model learned only using visual features can not be well generalized to novel classes. The semantic information learned from text data provides useful information about the relationship between different classes. So, the MD (iter 1) which uses semantic vectors as prototypes obtains significant improvements over the model only using visual features. This verifies that semantic vectors can help improve the model's generalizability to novel classes. Note that the semantic information does not examine the visual appearances. As a result, the overall performance is still limited. To overcome this limitation, our MD is learned by integrating the visual and semantic embeddings. The results show that the model's performance can be significantly improved by the joint visual and semantic embedding. We empirically find that the performance can not be further improved after 2-3 iterations so we set the number of iterations to be 3.
%It confirms the conjecture that only using semantic vectors does not work well and shows the advantage of our joint visual semantic embedding in detecting the novel classes. Compared on the predefined classes, our morphable detector works better(i.e., it achieves slightly better in Pascal VOC and 2 points improvement in COCO). The reason is that better prototypes represent the class better, so the better prototypes can facilitate the predefined class learning. The conclusion is consistent in other splits as shown in Table~\ref{tab:voc_2}, \ref{tab:voc_3}, \ref{tab:coco_2}. % than the Using joint visual-semantic embedding as the prototype also slightly improves the predefined classes' performance. This proves the predefined classes still benefit from better prototypes.
%Comparing the predefined class performance, although most few-shot detection and one shot detection models obtains worse performance than their baseline, our detector obtains comparable performance with Faster-RCNN framework. This illustrate that our morphable detector is able to detect the novel classes without sacrificing the predefined class performance.
%Table~\ref{tab:voc_1} shows that the visual baseline generally fail to detect the novel classes. Although this detector obtains reasonable performance on the predefined classes, it can not generalize well to the novel classes. The reason is that the visual baseline does not use the semantic vectors, so the mean visual features do not capture the expected attributes which can be shared among predefined and novel classes. Compared on the predefined classes, our morphable detector obtains much better performance since the mean features do not capture the expected attributes for the predefined classes.
\vspace{-3mm}
\paragraph{Ablation study}
To study the best way to combine the mean feature vector and the prototypes to compute the new prototypes, we compare ``element-wise sum" and ``concatenate". After the initial training, we concatenate or element-wise sum the mean feature vectors of the ground truth samples and the prototypes (i.e., the first item and second item in Eq.~\ref{eq.update}). The experimental results show that the element-wise sum is a better combination way, so the MD uses element-wise sum in the remaining experiments. Then, we compare the MDs using different $\lambda$ in Eq.~\ref{eq.update}, and we find that $\lambda=0.5$ works the best.
\paragraph{Comparison with state-of-the-arts}
We compare against three state-of-the-arts: FSOD method~\cite{fan2020fsod}, FRCNN~\cite{ren2015faster}, LSTD~\cite{chen2018lstd}. FRCNN and LSTD results are reimplemented by ~\cite{fan2020fsod}. Our MD obtains $\rm AP_{0.5}$ 10 points improvements over FSOD and 13 points improvement over LSTD. Unlike FSOD method~\cite{fan2020fsod} which takes 'query-target' pairs as input to train the model, our morphable detector learns more discriminative features and leverages useful external textual information. Therefore, our detector has better generalizability to novel classes. To verify that the MD can be well generalized to novel classes, we randomly selected 50 classes from the test set and visualize the feature vectors by t-SNE tool~\cite{van2008visualizing}. Fig~\ref{fig.embedding} shows that objects from most novel classes are clustered together on the learned feature space.
\vspace{-3mm}
\paragraph{Computation time} It takes around 0.04
seconds in ResNet 50 and 0.09 seconds in ResNet 101 to
add a new class using a single RTX 3090 GPU.
%This shows that effectiveness of using useful external semantic information and the proposed iterative EM algorithm.
\subsection{Experiments on Pascal and COCO}
\paragraph{Datasets and implementation}On Pascal VOC~\cite{Everingham15}, VOC 07 and 12 train/val sets are used for training and VOC 2007 test set is used for testing. In order to fairly compare with the state-of-the-art methods~\cite{kang2019few,chen2018lstd,yan2019meta}, we follow \cite{kang2019few} to use the same novel splits. %For the 20 classes in VOC, we first divide them into several semantic clusters: animals, traffic, furniture, and kitchen. To make sure the source and target classes have some semantic relationship, out of the 20 classes, we randomly select 1-2 classes from the above semantic clusters, in total five classes as the target classes. We evaluate the detector in three different source/target splits. During source training, only annotations of source classes are used. For zero-shot detection, no target data is used. For few shot detection, we used few images with only k objects existing where k equals 10, 30. Following [], we used the same k-shot images.
On MS COCO~\cite{lin2014microsoft}, we follow~\cite{lin2017feature} and train the model using the union of 80k train
images and a 35k subset of val images (trainval35k~\cite{bell2016inside}), and report the testing performed on a 5k subset of val images
(minival). We group the 80 classes in COCO dataset into 5 different semantic clusters and randomly select two classes from each cluster as novel classes(i.e. 10 classes in total). We randomly select two splits using this strategy. In the split 1, we select ``\textit{bicycle}", ``\textit{car}", ``\textit{dog}", ``\textit{sheep}", ``\textit{frisbee}", ``\textit{surfboard}", ``\textit{pizza}", ``\textit{laptop}", ``\textit{microwave}" and ``\textit{refrigerator}" as the novel classes. In the split 2, we select ``\textit{car}", ``\textit{train}", ``\textit{boat}", `` \textit{dog}", `` \textit{horse}", ``\textit{skateboard}", ``\textit{sandwich}", ``\textit{pizza}", ``\textit{keyboard}" and ``\textit{microwave}" as the novel classes. To fairly compare with most few-shot detectors~\cite{kang2019few, chen2018lstd,yan2019meta,wang2019meta,wang2020frustratingly, xiao2020few}, we also perform experiments using the same split as them (i.e., using 20 Pascal VOC classes as novel classes). Following these few-shot detectors, we ignore the novel classes annotations in training and randomly select one example as a given example for each novel class in testing. So, we perform one-shot detection experiments. We use ResNet-101~\cite{he2016deep} as the backbone and train the model with batch size 4 for 50k iterations and 160k iterations for Pascal VOC and COCO separately. The initial learning rate is 0.002 and it is decreased to 0.0002 after 40k and 120k iterations for Pascal and COCO. We use the same embedding size and semantic vectors used in FSOD experiments.
%In Table~\ref{tab:voc_1}, our detector obtains better performance than the visual-sem baseline on predefined and novel classes. Although visual sem-baseline uses the semantic vectors to do the initial training, it does not use the semantic vectors to update the prototypes. It shows the advantage of the joint-visual semantic embedding.
%Comparing the novel class performance after each iteration, our morphable detector performance steadily improves from 22.2 to 25 in Table~\ref{tab:voc_1} and from 23.6 to 26.3 in \ref{tab:coco_1} over the iterations. The same trend is observed in other splits in Table~\ref{tab:voc_2}, \ref{tab:voc_3}, \ref{tab:coco_2}. Our morphable detector benefits from the iterative training. It indicates that the proposed EM-like algorithm can help generate better prototypes to detect the novel classes. The overall performance converges after 2-3 iterations.
\begin{figure*}[t]
\centering
\includegraphics[scale=0.55]{example.pdf}
\caption{Some qualitative results of our proposed Morphable Detector on FSOD test set.}
\label{fig.example}
\end{figure*}
\begin{figure}[t]
\centering
\includegraphics[scale=0.55]{single.png}
\caption{ t-SNE visualization of feature embeddings of objects
from randomly selected 50 novel classes in FSOD test set on the learned MD.}
\label{fig.embedding}
\end{figure}
\subsection{Comparison on novel classes}
Table~\ref{tab:coco1} summarizes the comparisons on COCO dataset for split 1 and 2. Over iterations, the MD's performance consistently improves on the two splits. This verifies the effectiveness of the proposed EM-like approach. We also use the average performance to compare against one-shot-detector~\cite{hsieh2019one} which perform their experiments on another four random splits. Same with ~\cite{fan2020fsod}, OSOD~\cite{hsieh2019one} takes "query-target" pairs as input. So, our MD has better generalizability to the novel classes than~\cite{hsieh2019one}. Table~\ref{tab:coco2} summarizes the comparison against several state-of-the-arts~\cite{kang2019few, chen2018lstd, yan2019meta,wang2019meta,wang2020frustratingly, xiao2020few} on COCO dataset for split 3. Among them, only FSView~\cite{xiao2020few} reported their performance on one-shot detection settings. For others, they reported their 10-shot performance in their paper. Our method outperforms FSView by a large margin in the 1-shot detection setting and outperforms most of the others even though we only use 1-shot data. More importantly, as all these methods need extra training, they can not be deployed on the embedded systems as our method. Note that the performance may vary for different splits. The reason is that as the prototypes are initialized with semantic vectors, the relationship between base and
novel classes can influence the MD’s performance. In
the first two splits, the base and novel classes are split based
on semantic clusters of the classes, so the results of them
can be obviously better than those of split 3. Table~\ref{tab:voc} summarizes the comparison against the state-of-the-arts on three splits of the Pascal VOC dataset. Our MD outperforms state-of-the-arts by a large margin on the three splits. Our MD leverages semantic and visual information to help generalize the trained model to novel classes. % so the overall performance is related to the relationship between the base and novel classes.% As in split 1, the relationship does not help much as the other two splits, the performance on split 1 is not as good as the other two splits. % are far from the base classes in semantic space. Our MD leverages the semantic information so in the MD, a base class may help its semantic relevant classes in novel classes. while our method mainly leverages the semantic information to generalize the model trained on base classes to novel classes. %Our split 1 and 2 are the same splits used in them. %we used the same choice of target classes and low-shot data as~\cite{kang2019few} which are publicly available. %Both of them rely on few shot data to learn some unknown relationship between source and target categroeis.
%The results are shown in Table~\ref{tab:voc_2} and ~\ref{tab:voc_3} . Our method significantly outperforms them in all splits on novel and predefined classes. Furthermore, their predefined class performance decreases compared with their baseline due to the few-shot finetuning. In contrast, our method does not do any finetuning but generates a detector for the novel classes. So our method performance on the predefined classes does not decrease when the novel classes are specified. More importantly, Feature-Reweight~\cite{kang2019few}, LSTD\cite{chen2018lstd}, and Meta-RCNN~\cite{yan2019meta} rely on the finetuning to detect the novel classes, so they can not satisfy the real-time constraints in ODOD. %, In 3-shot detection, our SAN obtains 37 points and 20 points gains over LSTD and RW in split 1, and 31 points and 18 points improvement in split 2. This is because LSTD and RW rely on few shot data to learn the unknown relationships between source and target classes, and they would overfit to the data when the training shot is pretty low. This shows that the semantic relationship we used can alleviate the overfitting issue and bring significant gains. Furthermore, RW and LSTD fail for zero-shot detection.
%In MS-COCO, we perform the experiments on two different splits. Since the splits are different from Feature-Reweight~\cite{kang2019few}, and Meta-RCNN~\cite{yan2019meta}, the performance can not be directly compared. But we can compare the overall performance for a rough comparison. Feature-Reweight~\cite{kang2019few} reported average 12.1 and Meta-RCNN~\cite{yan2019meta} reported average 19.1 $AP^{0.5}$ for ten shot detection . Our morphable without further training obtain ~25 using only one shot. Furthermore, we perform a rough comparison with the incremental shot detection method ~\cite{hsieh2019one}. They obtain 0.7 and 17.9 AP on the novel class and base class separately. Our method obtains 16.1 and 33.2 AP on the novel class and base class separately.
% In ODOD, the novel class prototype can be generated using one example or a few examples, as shown in Eq~\ref{eq.prototype}. In this section, we compare our detector performance when the number of examples increases to be 3. Same with experiments using one example, we perform five experiments and report the average performance to exclude the data variation effects. Using only one example, our detector obtains comparable performance with the detector using three examples on the novel classes. It indicates that increasing the number of objects does not necessarily further improve the overall performance.
\subsection{Comparison on base classes}
Table~\ref{tab:coco_seen} and~\ref{tab:voc_seen} summarize the comparison against our baselines and state-of-the-arts on Pascal and COCO datasets for base classes. Our MD obtains obviously a bit better performance on the base classes over iterations. This verifies the proposed EM-like algorithm can help improve the model's generalizability to unseen samples of base classes. %This is because the semantic vectors for the base classes do not examine the visual appearance, the models trained with semantic vectors have limited generalizability to the unseen samples of the seen classes.
Our MD performs much better than OSOD~\cite{hsieh2019one} which takes ``query-target" pairs as input. This verifies that OSOD~\cite{hsieh2019one} learns much less discriminative features for base classes. Compared with state-of-the-arts few-shot detectors, our MD performs the best for the base classes. The reason is that these FSD models take a small number of base class data to further train the model so these models can easily overfit to the small data. Compared with FRCNN~\cite{ren2015faster}, our model can still outperform it. This shows the advantage of our MD over FRCNN on the base classes. Note that FRCNN can not be generalized to novel classes without extra training. %This is because such ``query-target" training models learn less discriminative features, so it is hard for them to be well generalized to the unseen samples of the seen classes.
% Given the prototypes, we compare our contrastive loss with margin-based loss, such as triplet loss or N-negative pair loss~\cite{sohn2016improved}. The margin-based loss encourages the sample of the one class to be closer to the corresponding prototype than that of other classes by a margin. We use the following loss to replace the contrastive loss in Eq~\ref{eq.cl},
% \begin{equation}
% \sum_{i=1}^{N} \sum_{j \in \mathcal{L}_s, j \neq y_i}^{} max(0, m - Cos(v_i, P_{y_i}) + Cos(v_i , P_j) )
% \label{eq.loss}
% \end{equation}
% where $Cos$ is the cosine similarity between the features and prototypes. For the background, we generate a random vector as its prototype. The difference with our contrastive loss is that our contrastive loss normalized the distances to learn better discriminability. In this experiment, our method and the margin-based method both use semantic vectors as prototypes. During testing, the proposals are classified by selecting the best match prototype. In Table~\ref{tab:voc_1}, our morphable detector works much better than margin-based loss on both novel and predefined classes, especially on the predefined classes. It verifies that our contrastive loss is better at learning the discriminative features for detection. %For the novel classes, ours still outperforms the margin based loss. This indicates effectiveness of the prototype based contrastive loss.
\begin{table}[t]
% \begin{tabular*}
\centering
\footnotesize
% \setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{0.49\textwidth}{@{\extracolsep{\fill}}p{0.3cm}<{\centering}|p{0.3cm}<{\centering}p{0.6cm}<{\centering}|p{0.3cm}<{\centering}p{0.6cm}<{\centering}|p{0.3cm}<{\centering}p{0.6cm}<{\centering}|p{0.7cm}<{\centering}}
%\setlength{\tabcolsep}{1mm}{
%\begin{tabular*}{c|c|c|c|c|c|c|c}
\hline
\multicolumn{1}{c|}{\multirow{2}{*}{Method}} & \multicolumn{2}{c|}{Split 1} & \multicolumn{2}{c|}{Split 2} & \multicolumn{2}{c|}{Split 3} & Ave \\
& $\rm AP$ & $\rm AP_{0.5}$ & $\rm AP$ & $\rm AP_{0.5}$ & $\rm AP$ & $\rm AP_{0.5}$& $\rm AP_{0.5}$\\
\hline
\hline
\multicolumn{1}{c|}{FRCNN~\cite{ren2015faster}} &
37.3&
59.9&
36.9&
58.7&
37.0&
59.1 & 59.2
\\
\multicolumn{1}{c|}{OSOD~\cite{hsieh2019one}} & - & - & - & -& - & - & 40.9
\\
\multicolumn{1}{c|}{MD (iter1)} & 37.5&
60.6
&\textbf{37.2}
&\textbf{59.3}
&37.2
&59.0 & 59.6
\\
\multicolumn{1}{c|}{MD} &
\textbf{37.8}&
\textbf{60.7}&
36.9&
59.0&
\textbf{37.5}&
\textbf{59.2} & \textbf{59.7}
\\
\hline
\end{tabular*}
}
\vspace{-2mm}
\caption{Performance comparison on the COCO dataset for base classes.}
\label{tab:coco_seen}
\end{table}
\begin{table}[t]
% \begin{tabular*}
\centering
% \scriptsize
% \setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{0.49\textwidth}{@{\extracolsep{\fill}}p{1cm}<{\centering}|p{1cm}<{\centering}|p{1cm}<{\centering}|p{1cm}<{\centering}|p{1cm}<{\centering}}
\hline
\multicolumn{1}{c|}{{Method}} & {Split 1} & Split 2 & Split 3 & Ave \\
\hline
\hline
\multicolumn{1}{c|}{MetaRCNN~\cite{yan2019meta}}
&64.8& - & - &64.8\\
\multicolumn{1}{c|}{TFA w/cos~\cite{wang2020frustratingly}}
&79.2& - & - &79.1\\
\multicolumn{1}{c|}{OSOD~\cite{hsieh2019one}} & - & - & -& 60.1\\
\multicolumn{1}{c|}{MD (iter1)}
&80.2&81.6&
78.4&80.1 \\
\multicolumn{1}{c|}{MD}
&\textbf{80.7}&\textbf{82.1} &\textbf{79.2}&\textbf{80.7}\\
\hline
\end{tabular*}
}
\caption{Performance comparison on the PASCAL VOC dataset for base classes. }
\label{tab:voc_seen}
\vspace{-5mm}
\end{table}
\begin{table}[t]
% \begin{tabular*}
\centering
% \scriptsize
% \setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{0.44\textwidth}{@{\extracolsep{\fill}}p{3cm}<{\centering}|p{2cm}<{\centering}|p{2cm}<{\centering}}
\hline
\multicolumn{1}{c|}{{Method}} & Recall@100 & $\rm AP_{0.5}$ \\
\hline
\hline
\multicolumn{1}{c|}{SB~\cite{bansal2018zero}}
&22.4& 0.7 \\
\multicolumn{1}{c|}{DSES~\cite{bansal2018zero}}
&27.2& 0.54 \\
\multicolumn{1}{c|}{TD~\cite{li2019zero}}
&34.3 & - \\
\multicolumn{1}{c|}{DELO~\cite{zhu2020don}}
&33.5& 7.6\\
\multicolumn{1}{c|}{MD (split 1)}
&44.8& 9.4 \\
\multicolumn{1}{c|}{MD (split 2)}
&\textbf{47.2}& \textbf{9.8} \\
\multicolumn{1}{c|}{MD (split 3)}
&27.0& 4.2 \\
\hline
\end{tabular*}
}
\caption{Comparison under zero shot detection setting on COCO dataset for novel classes. }
\label{tab:zero}
\vspace{-2mm}
\end{table}
\subsection{Comparison with zero shot detectors}
As a by-product, we also perform the MD under zero-shot detection setting. In training, we use the semantic vectors for the base classes as prototypes to train the MD. In testing, we use the semantic vectors for the novel classes as novel prototypes. Table~\ref{tab:zero} summarizes the comparison against state-of-the-art zero shot detectors using standard evaluation metrics recall@100 and $\rm AP_{0.5}$. Our MD obtains very impressive results on the first two splits. The performance on the split 3 is not as good as the other two splits because the relationship between base and novel classes on split 3 does not help much as the other splits. Our MD obtains an average of 39.7 Recall@100 which is better than other zero-shot detectors.
% For our morphable detector, the novel classes can be online specified over time stream. First, we perform experiments without specifying any novel classes in Table~\ref{tab:voc_1}, this experiment is named as \textit{only seen}. Comparing the detector performance with and without novel classes to detect on the predefined classes, the performance only slightly decreases after specifying the novel classes. To further verify this, we incrementally specify \textit{bicycle}, \textit{car} and \textit{train}, \textit{dog}, the detector performance on the predefined classes just very slightly decrease, like 0.1-0.2 points. By further adding the other three novel classes(i.e., our model results), our detector performance on the predefined classes and the last novel classes' performance do not decrease. This indicates that our detector can handle incremental detection without forgetting.
%\subsection{Comparison with One-Shot Detection SOTA}
\section{Introduction}
%Recent years have witnessed significant progress in object detection since the deep convolutional neural network was first used in R-CNN~\cite{girshick2014rich}. These detectors defined the classes to detect offline so that people can annotate these classes in the dataset and train the model using the annotated data. However, only detecting the predefined classes makes these detectors unscalable for the real-time scene understanding applications, like robotics navigation~\cite{nurmaini2017intelligent}. In these applications, the system is often exposed to different new environments in which the objects of novel classes are present. So to detect a novel class which is not defined before when the system is exposed to a new environment is required. We also call this problem as Object Detection on Demand. % Furthermore, although online collecting one or a few examples is feasible in these applications, collecting much data is very difficult due to the expensive annotation and real-time constraint. It poses another issue, notorious \textit{long-tail} issue in recognition. Most current detection models are trained using many annotated data for the predefined classes. It is difficult for them to handle the \textit{long-tail} issue. Last but not least, in these real-time applications, the detectors expect to detect the novel classes as long as they are specified. So real-time generating the new detector for the novel classes is required.
%To meet the requirements above, we study a practical detection setting, \textit{Object Detection on Demand}(ODOD). ODOD is defined by 1) there are much-annotated data for the predefined classes available to train a base model. 2) The detector can detect the novel classes as long as they are online specified by giving one or a few examples, also known as real-time constraints. 3) Once the base model is trained, the trained model can not be updated with further training due to the real-time constraint.
%There are several challenges for Object Detection on Demand: 1) It is hard to collect large number of data for the novel classes in short time. But people can quickly annotate a few samples of the new classes. 2) In real-time applications, the model can not be updated by a long time training. 3) incremental learning.
%To handle the first challenge, few-shot detection models are proposed. In these models, they first train a base model using the base classes data which of offline defined, then finetune the model using few novel class data. However, few shot detection models depends on long time of fine-tuning process. Furthermore, after finetuning, the performance for novel classes decrease which is known as "forgetting" issue. So in incremental learning, the performance for the base classes decrease as the number of novel classes incrementally increases. Therefore, few-shot detection is not a appropriate solution for Object Detection on Demand. %In this setting, many data for base classes, but only a few data for novel types are available for training.
%Most few-shot detection models first train a base model using base class data, then finetune the base model using few data for the novel classes to detect the novel classes. So they mostly rely on the finetuning process, making these methods fail to satisfy the real-time constraint in ODOD. Furthermore, these methods lack the capability of incremental learning of novel classes from a data stream over time in the open-ended problem~\cite{perez2020incremental}. So Perez-Rua el al. ~\cite{perez2020incremental} recently proposed an incremental few shot detection model that can do incremental learning of the novel classes over time. However, it still relies on the supervised learning process to detect the novel classes. So it still can not satisfy the real-time constraint.
%ODOD is very challenging due to multiple issues. First, generalizing the detection model trained from predefined classes to online specified classes is difficult since the relationship between predefined and novel classes is hard to model. Second, with few examples of the novel classes, it is difficult to generate a detector that generalizes well to detect general examples of these classes since it is hard to capture the common properties of novel classes from a few examples. Third, the real-time constraint is hard to satisfy.
%Recent years have witnessed significant progress~\cite{} in object detection since the deep convolutional neural network was first used in R-CNN~\cite{girshick2014rich}. These detectors offline defined the classes to detect, so they follow \textit{train and detect} framework in which to detect one class, the model has to be trained first. We argue that \textit{train and detect} is not able to handle the real-time scene applications like robotics navigation. Because first, in these applications, the system is often exposed to different new environments in which the objects of novel classes are present. So the detectors are required to detect a novel class without further training due to time and resource constraints. Second, incremental adding novel classes is required in these applications. Further training would make former known class performance drop which is also known as learning with forgetting issue. %To detect a novel class without further training is also called as object detection on demand.
%In general detection setting, the class to detect are offline defined so people annotate enough data for each class offline and follow the \textit{train and detect} framework. In few shot detection setting, a large number of known class data and few examples of novel classes are collected. In this setting, a base model is first trained using the known classes, then the base model is then tuned using the few examples of novel class. Overall, they still follow \textit{train and detect} framework.
%Different with above two detection settings, we study a new detection setting, \textit{Object Detection on Demand}(ODOD). ODOD is defined by 1) there are a number of data for the known classes available to train a base detector. 2) The trained detector is able to detect the novel class without further training. 3) We assume manually annotating one example of the novel class is easy. ODOD is very challenging since 1) It is hard to generate a new detector for the novel class without further training. 2) Modeling the relationship between the known and novel classes is difficult.
%Zero-shot detection models~\cite{zhu2020don,bansal2018zero} do not need extra data of the novel classes and further training. They usually use the semantic relationship obtained from the language models~\cite{mikolov2013distributed} to model the semantic relationship between the base classes and novel classes so that the model trained using base classes can be transferred to detect the novel classes. However, the semantic relationship is not always good relationship for such transfer. For example, source classes arecar,busandtrain, and target class isair-plane. The semantic embedding is learned by the semantic relationships amongsource classes as shown in Fig. 1. Therefore, in such embedding,airplaneimage is expected to match theairplanesemantic vector. %Each concept/class is assigned one semantic vector, which represents the semantic properties of that class. %A mapping from the visual features to the semantic embedding is learned in these detection models so that within the learned semantic embedding space, the best match among
%semantic vectors of novel classes is selected for the visual features of any given proposal of the novel classes.
%However, since the semantic vectors fail to consider these classes' visual properties, such semantic vectors are not accurate to gap the predefined and novel classes. Therefore, these detection models are hard to generalize well to the novel classes in visual data. Recent Hsieh et al. ~\cite{hsieh2019one} proposed a one-shot detection model, but this model easily overfits the training data. From the reported results in ~\cite{hsieh2019one}, the training classes' performance is even worse than the novel classes'.
%We propose to learn a morphable detector for ODOD. Our morphable detector is trained with the predefined class data. When the novel classes are specified, and a few examples are given, it can be morphed to be a new detector to detect these novel classes without further training. The morphable detector is built on the Faster-RCNN framework, and it replaces the final classifier in Faster-RCNN with an attribute embedding, as shown in Fig~\ref{fig.netork}. To learn the attribute embedding, we present a prototype-based contrastive loss to train the network. A prototype is defined as a representative embedding for a group of semantically similar instances, as all instances of one class. This loss is designed to bring samples closer to their corresponding prototypes and push samples from other prototypes further apart in the attribute embedding. The morphable detector treats the prototypes as hidden variables as they are unknown, which are gradually refined by jointly using the visual and semantic information, called as \textit{joint visual-semantic embedding}. The prototypes and the network parameters are optimized alternatively in an EM-like way, as shown in Fig~\ref{fig.framework}. %To make the alternative learning effective, we propose to progressively refine the prototypes between each iteration by the \textit{joint visual-semantic embedding}.
%Specially, we present prototype based contrastive learning to learn the morphable detector. To get the prototype, we present a joint visual-semantic embedding an EM algorithm in which the prototypes are treated as latent variables. The ideal prototype represents the common attributes of the class. The predefined and novel classes share the attribute space. % jointly encode the visual properties of the visual data and semantic properties of each concept into the prototype by an EM algorithm in which the prototypes are treated as latent variables. The ideal prototype represents the common attributes of the class. The predefined and novel classes share the attribute space. %Each class is assigned one prototype.
%To better represent the concept, we propose to encode the visual properties of the visual data and semantic properties of each concept into the prototype of one class. By doing so, the prototypes which joint encode visual-semantic properties represent these classes well so that the detector can be well generalized to the online specified novel classes.
% When the novel class is specified, the morphable detector generates the corresponding prototype by \textit{joint visual-semantic embedding}. When proposals come, the best match among the prototypes of novel classes in the attribute embedding is selected for the visual features of any given proposal of the novel classes.
%Recent years have witnessed significant progress~\cite{} in object detection since the deep convolutional neural network was used in R-CNN~\cite{girshick2014rich}. Most of these works are focused on a general detection setting where all classes to detect are offline defined so the detector does not need to detect novel classes.
In applications, like robotics exploration and autonomous driving, the systems need to explore and understand new environments, where it is desirable to detect objects of novel classes on the fly with minimum online human supervision and interaction. This is an object detection on demand (ODOD) task. %In many real world applications, such as robotics exploration and autonomous driving, the system is often exposed to a new environment where objects of novel classes are present. Thus, the system is required to detect the objects of novel classes.
ODOD is very challenging because it is impossible to collect a large amount of data on the fly, and the computing resources are generally not powerful enough for computationally intensive and time-consuming training on board, not to mention that many embedded systems are unable to perform back-propagation which is essential for training. In embedded systems, the detection task is usually carried out on a computationally limited platform where the neural networks are locked after the system is built due to resource limits~\cite{alibaba2020mnn}. The prevailing few-shot detection (FSD)~\cite{kang2019few,yan2019meta,wang2019meta,wang2020frustratingly, xiao2020few} methods are confronted here as they generally need to perform extra training %making them unfeasible but to train the detector on a resource-constrained mobile system is not feasible. The reason is that in these systems, the detection task is usually carried out on a computationally limited platform where the neural networks are locked after the system is built due to resource limits~\cite{alibaba2020mnn}. So, they only support feed-forward inference but not back-propagation operation which is essential for training.
for objects from novel classes. % zx 0318
%However, for applications like robotics exploration, the system is often exposed to a new environment where the objects of novel classes are present. Thus, the detector is required to detect the novel classes in timely sense. But it is hard to collect enough data to train a new detector in short time. %An idea system can be fast adapted to detect the novel classes when it is exposed to a new environment. %Collecting new data and re-training the model are very expensive and time-consuming % Another constraint for these applications is that it requires real-time adaptation %Furthermor, these applications require the trained detector be real-time adapted to detect the novel classes without forgetting the old classes since the old classes may exist in the new environment.
% In real-time scene application, there is no time to further train the model for the novel class. Second, incremental adding novel classes is required in these applications. Further training would make former known class performance drop which is also known as learning with forgetting issue. So \textit{train and detect} framework is not suitable for these applications. %To detect a novel class without further training is also called as object detection on demand.
\begin{figure}[t]
\centering
\includegraphics[scale=0.3]{intro.pdf}
\caption{Comparison with other detection tasks. Different from other few-shot detection tasks, Object Detection on Demand requires no extra training.}
\label{fig.task}
\vspace{-3mm}
\end{figure}
%In general detection setting, the class to detect are offline defined so people annotate enough data for each class offline and follow the \textit{train and detect} framework. In few shot detection setting, a large number of known class data and few examples of novel classes are collected. In this setting, a base model is first trained using the known classes, then the base model is then tuned using the few examples of novel class. %Over follow \textit{train and detect} framework.
To this end, % zx 0318
we define Object Detection on Demand (ODOD) specifically as detecting the novel classes without extra training while preserving the existing knowledge, given (1) a detector offline trained using base class data, (2) no access to base class data (3) a few samples for novel classes. The ODOD can be regarded as a special few-shot detection task and the differences of ODOD from other detection tasks are listed in Fig~\ref{fig.task} .
%ODOD is very challenging as no further training is allowed. %Compared with general detection setting in which all classes to detect are offline defined,
%ODOD is very challenging because it is difficult to detect objects of novel classes without extra training.
%no prior knowledge about the novel classes is provided, it is difficult to generalize an offline-trained detector to unknown novel classes. So ODOD is very challenging.%So ODOD can also be regarded as special few-shot detection.% with some constraints which are motivated by the problems in real applications. %Different with general detection settings which follow \textit{train and detect} framework, we study a new detection setting, \textit{Object Detection on Demand}(ODOD). ODOD is defined by 1) there are a number of data for the known classes available to train a base detector. 2) The trained detector is able to detect the novel class without further training. 3) We assume manually annotating one example of the novel class is easy. Can we design a detector which is able to detect the novel class without further training? Yes, we design a morphable detector which is first trained using the known class data and is able to be morphed to be a new detector to detect the novel class without further training. To train a morphable detector is very challenging due to following difficulties. First, modeling the relationship between the known and novel classes is difficult. Second, without further training, it is hard to make use of the one example of the novel class to generate a new detector for the novel class.
The prevailing few-shot detectors (FSD) aim to train a detector using base class data and further train it with a few samples from novel classes. However, extra training is unfeasible in the ODOD task. Furthermore, to keep the performance for the base classes, these FSDs have to use the base class data in extra training, otherwise, they suffer from catastrophic forgetting~\cite{mccloskey1989catastrophic} - a significant performance degradation, when the
past data are not available. Other few-shot detectors~\cite{fan2020fsod, hsieh2019one} use a siamese network and take ``query-target" pairs as input so as to detect all instances of the ``target" object appearing in the ``query" image. However, as the target representation is always changing during the training, the model learns less discriminative representations. As a result, the model's generalizability to unseen samples of the base classes is limited. % and its performance on the seen classes is obviously worse than many existing discriminative detectors which learn discriminative representations to discriminate classes.
%In the morphable detector, no further training is required, so the relationship between known and novel classes needs to be modeled. Zero-shot detection models~\cite{zhu2020don,bansal2018zero} is a specific case of morphable detector when no example of the novel class is present. They make use of the semantic relationship obtained from the language models~\cite{mikolov2013distributed} to model the relationship between the known classes and novel classes. However, the semantic relationship is not always accurate for visual models. For example, suppose we have known classes car, bus and train, and the novel class airplane. All of these classes are close in semantic space, but airplane has very different visual appearances with other classes. Without seeing the airplane, the model would think the airplane would have similar appearances with the known classes. We argue that such inaccurate relationship from semantic relationship can be adjusted using one example of the novel class. %In this paper, we focus on a more general morphable detector when one example of the novel class is present.
In this paper, we present a novel morphable detector (MD) that simply ``morphs" some of its changeable parameters online estimated from the few samples, so as to detect novel classes without any extra training. Different from most existing object detectors, this novel MD has two sets of parameters, one for the feature embedding (i.e. the network parameters), and the other for class representation (called ``prototypes") as illustrated in Fig~\ref{fig.framework}. We view the MD for recognizing visual samples of different classes as if they live in a common space called feature space. Each class is associated with a prototype which is the targeted coordinate of each class in the feature space. Therefore, for each object proposal, the MD learns a feature vector whose similarity with prototypes is regarded as the foreground classification score. As it is hard to assign one prototype to the background, the MD directly regresses a background score from the visual features as shown in Fig~\ref{fig.framework}. Once an MD is learned, it is able to use a few samples of a novel class to directly compute its prototype to fulfill the online morphing process (details are in \ref{sec.morphine}).% Regarding the background, since it is hard to assign one prototype to it, the MD directly regress a background score from the visual features as shown in Fig~\ref{fig.framework}.
% Each class is associated with a hidden prototype to be learnt by integrating the visual and semantic embeddings.
%we propose to train a novel morphable detector(MD) which is trained using seen class data and can be online morphed to detect the novel classes given a few samples. We view the MD for recognizing visual samples of different classes as if they live in a common space called foreground space. In a MD, each class is associated with a prototype which is the targeted coordinate of each class in the foreground space. Therefore, for each object proposal, the MD learns a foreground embedding whose similarity with prototypes are regarded as the classification scores. So during the online morphing, the MD only needs generate the prototypes for the novel classes by forwarding the few samples through network(details are in Sec.~\ref{sec.morphine}). Without back-propagation computation, our MD can be easily deployed on the mobile systems. As the network model is not changed during online morphing, the MD does not have the forgetting problem as other few-shot detectors. Regarding the background, since it is hard to assign one prototype to it, the MD directly regress a background score from the visual features as shown in Fig.
The learning of the MD is based on the alternate learning of the feature embedding and the prototypes in an EM-like approach as shown in Fig~\ref{fig.trianing}. The prototype is regarded as a hidden variable to be learned by integrating the visual and semantic embeddings. In "E"-step, we fix the network parameters and update the prototypes by combining the current prototypes and the feature vectors of the training samples on the trained model (details are in Sec.~\ref{sec.update}). In "M"-step, the prototypes are fixed and the network is trained using the updated prototypes. The prototypes are initialized with semantic vectors which bring useful external information from textual data. But note that directly using semantic vectors as prototypes without the proposed EM-like algorithm still suffers from limited generalizability (to novel classes) because the external information does not directly examine visual appearances while the model concerns itself with recognizing visual features. Therefore, the joint learning of the feature embedding and prototypes allows better recovery of an unknown prototype from a few samples of a novel class. Our approach is different from the existing approach, such as RepNet~\cite{karlinsky2019repmet} which learns representatives for each class from the visual data in an end-to-end training. The proposed MD learns the representatives (prototypes) by an EM-like approach where the visual and semantic information are integrated to improve the model's generalizability (to novel classes).
\begin{figure*}[t]
\centering
\includegraphics[scale=0.46]{MD.pdf}
\caption{The proposed morphable detector (MD) structure. Given a trained MD, it is able to compute the representations (prototypes) for the novel classes using a few samples from novel classes (see sec.~\ref{sec.morphine}). Given a test image and proposals generated by RPN, the MD outputs the feature vectors, box regression, and background score for each proposal. The similarity between the feature vectors with the prototypes associated with each novel class is used to estimate the novel class posterior probability (see sec.~\ref{sec.morphine}).
}
\label{fig.framework}
\vspace{-3mm}
\end{figure*}
% The reason is that we leverage semantic vectors~\cite{mikolov2013distributed} So, we propose to first train the MD using semantic vectors as prototypes, and recursively optimize the prototypes and network parameters alternately like an EM algorithm. Thus, the visual information is combined with the semantic information. Over iterations, the prototypes becomes better since better foreground embedding is extracted. As a result, the model's generalizability to novel classes improves.
% In the network training,% the seen class prototypes are the target coordinates of the seen classes in the foreground space. %The training goal is to push the training samples close to its corresponding prototype while far away from other prototypes in the foreground space. Obviously,
% %So,
% using better prototypes to
%Training the model with better prototypes leads to a model with better generalizability to novel classes. When the prototypes are one-hot vectors where each class is associated with a one-hot vector, the MD degenerates to a discriminative detection model\cite{ren2015faster, lin2017feature, cai2018cascade, cai2016unified, dai2016r}. As one-hot vectors are orthogonal and independent to each other, it is hard to generalize the model to the novel classes without extra training. % a few samples. %When the prototypes are semantic vectors~\cite{} which bring external information from textual data to improve the model's generalization power, however it is still limited since the external information does not directly examine visual appearances while the model concerns itself with recognizing visual features. Thus, the performance for zero-shot detectors is limited.
%To address this issue, we propose to leverage semantic vectors~\cite{mikolov2013distributed} which bring useful external information from textual data. But directly using semantic vectors as prototypes still suffers from limited generalizability (to novel classes) because the external information does not directly examine visual appearances while the model concerns itself with recognizing visual features. So, we propose to first train the MD using semantic vectors as prototypes, and recursively optimize the prototypes and network parameters alternately like an EM algorithm. In "E"-step, we fix the network parameters and update the prototypes by combining the current prototypes and the foreground embedding of the training samples on the current trained model(details are in Sec.~\ref{sec.update}). Thus, the visual information is combined with the semantic information. In "M"-step, the prototypes are fixed and the network is trained using the prototypes. Over iterations, the prototypes becomes better since better foreground embedding is extracted. As a result, the model's generalizability to novel classes improves.
%Why we still use the semantic vectors to update the prototypes? Because the semantic vectors contains useful information which help learn some class-independent attributes which are transferable to the novel classes. E.g, many animals tend to have two or four legs, such attribute is useful to detect a novel class which belongs to animals. %So, the resulting prototypes not only contain useful external information but also examine the visual appearances.
%Our MD has several advantages over existing FSDs: 1) our MD can online morphed with a few samples for novel classes without extra training. 2) our MD makes use of external useful textual information which helps learn class-invariant attributes
%In MD, the prototypes are very To train the morphable detector, we treat the prototypes as hidden variables and design an EM-like algorithm to train the model in which we use semantic vectors as initial prototypes to train the network model in "M-step", and use the trained model, visual data and semantic vectors to update the prototypes in the "E-step". So the network training and prototype update happen alternately to improve the learned foreground embedding's generalizability to the novel class. %, we assign each class a prototype, a representative embedding of that class， then train a embedding using a prototype based contrastive loss which push the samples closer to its corresponding prototype while far away from other prototypes. After the embedding is trained, the detector can detect the novel class by generating the prototype of the novel class. The relationship among the prototypes represents the relationship among classes. The more accurate the relationship is, the better performance on the novel classes. In this work, we present a joint visual-semantic embedding algorithm as illustrated in Section 3. We first use semantic vectors as prototypes to train the model. After the model is trained, we obtain better prototypes by joint visual-semantic optimization which takes the semantic vectors and visual embedding of the examples on the trained model to generate a better prototype (the details are in Section 3). We can recursively train the model and obtain better prototypes in an EM-like way until the model converges.
Overall, the contributions of this work are four-fold:
\vspace{-2mm}
\begin{itemize}
\item We study a special few-shot detection task, Object Detection on Demand, which is rarely discussed in the literature and can not be solved by many existing Few-shot detection methods. \vspace{-2mm}
\item We present a novel morphable detector (MD) which can be online morphed to detect the novel classes without extra training. \vspace{-2mm}
\item We propose to learn the MD by joint visual and semantic embedding in an EM-like approach. \vspace{-2mm}
\item Extensive experiments are performed on different datasets to demonstrate the superiority of the MD over other methods.\vspace{-2mm}
\end{itemize}
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage{iccv}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bbm}
\usepackage{adjustbox}
\usepackage{bbding}
\usepackage{multirow}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{array}
\usepackage{multirow}
\usepackage{float}
%\usepackage{graphicx}
\usepackage{color}
\usepackage{amsfonts,amssymb}
%\usepackage{lmodern}
\usepackage{bm}
\usepackage{subfig}
\usepackage{algorithm,algorithmic}
\usepackage{lipsum}
\usepackage{pifont}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\usepackage{authblk}
\usepackage{setspace}
% \usepackage[accsupp]{axessibility}
% \usepackage{multicol}
% Include other packages here, before hyperref.
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage{hyperref}
\hypersetup{breaklinks=true,letterpaper=true,colorlinks,bookmarks=false}
\iccvfinalcopy % *** Uncomment this line for the final submission
\def\iccvPaperID{2865} % *** Enter the ICCV Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
% Pages are numbered in submission mode, and unnumbered in camera-ready
\ificcvfinal\pagestyle{empty}\fi
\begin{document}
%%%%%%%%% TITLE
\title{Morphable Detector for Object Detection on Demand}
\author[1]{Xiangyun Zhao}
\author[2]{Xu Zou}
\author[1]{Ying Wu}
\affil[1]{Northwestern University}
\affil[2]{Huazhong University of Science and Technology}
%\institute{$^1$Northwestern University \hspace{5mm} $^2$NEC Labs America \hspace{5mm} $^3$UC San Diego}
% \author{First Author\\
% Institution1\\
% Institution1 address\\
% {\tt\small firstauthor@i1.org}
% % For a paper whose authors are all at the same institution,
% % omit the following lines up until the closing ``}''.
% % Additional authors and addresses can be added with ``\and'',
% % just like the second author.
% % To save space, use either the email address or home page, not both
% \and
% Second Author\\
% Institution2\\
% First line of institution2 address\\
% {\tt\small secondauthor@i2.org}
% }
\maketitle
% Remove page # from the first page of camera-ready.
\ificcvfinal\thispagestyle{empty}\fi
%%%%%%%%% ABSTRACT
\begin{abstract}
\input{abstract}
\end{abstract}
\input{introduction}
\input{related}
\input{method}
\input{experiments}
\input{conclution}
\clearpage
{\small
\bibliographystyle{ieee_fullname}
\bibliography{egbib}
}
\end{document}
\section{Morphable Detector}
\vspace{-2mm}
In this paper, we present a novel Morphable Detector (MD) which can be online morphed to detect the novel classes without extra training. As illustrated in Fig~\ref{fig.trianing}, we propose to learn the prototypes and network parameters alternately in an EM-like approach, with the other fixed in each iteration. Fig~\ref{fig.framework} illustrates how the morphable detector(MD) is morphed to detect the novel classes given a few samples from novel classes. Once the MD is trained, the MD only needs to forward a few samples of a novel class through the network to compute its prototype (details are in sec.~\ref{sec.morphine}) to detect the novel classes.%In "E"-step, the network parameters are fixed, the prototypes are updated based on the
%Our morphable detector is built on the Faster-RCNN framework~\cite{ren2015faster} in which the region proposals are first generated by Region Proposal Network(RPN), then the proposals are forwarded to do classification and bounding box regression. The morphable detector extract the region proposals as Faster RCNN and learns a class-agnostic bounding box regression, a background score for the background and a visual-semantic embedding space for the foreground classes in which the classification is performed by computing the similarity between the embedding and prototypes which are representative embedding for each class. The choice of the prototype is critical for the morphable detector to be morphed to the new detector %Regarding the box regression, ~\cite{wang2019meta} show that using class-agnostic regressions in object detection only minorly harms the performance. So we do not focus on generating good bounding box regression for the novel classes, and we build the whole system with class-agnostic bounding box regression.
%Regarding the classification, the proposals' visual features are first extracted by ROI pooling, then are mapped to the attribute embedding, as shown in Fig~\ref{fig.netork}. In our morphable detector, each class is assigned one prototype, a representative embedding of that class in the attribute embedding. In inference, the proposal is classified as one class if it matches that class's prototype in the attribute embedding. To detect the novel class, the morphable detector only needs to generate the novel class prototype. %Then, the detector can detect that class. So the morphable detector can be morphed to detect the novel class without further training.
%We present an EM-like algorithm to learn the morphable detector for ODOD. The prototype represents the attributes of one class. Since the optimal prototypes are unknown, we treat them as hidden variables, which are first initialized with semantic vectors, and then recursively refined in our approach. Given prototypes, the network is trained using a prototype-based contrastive loss. This loss brings samples from the same class closer and features of samples from different classes further apart in the attribute embedding. As the learned attribute embedding depends on the prototypes, the morphable detector's performance heavily depends on the prototypes' quality. With better prototypes, the morphable detector can be better generalized to the novel classes. The proposed algorithm progressively improves the prototypes' quality from the semantic vectors by jointly using visual and semantic information. The detection network parameters and prototypes are alternatively optimized following an EM algorithm in Fig~\ref{fig.framework}, with the other fixed in each iteration.
%So in this work we build the whole system based on Faster-RCNN with class-agnostic bounding box regression. In the following sections, we mainly discuss how to classify the proposals when novel classes are specified.
%Given the prototypes, the attribute space embedding is learned. The prototype based contrastive learning is formulated as an EM algorithm in which the prototypes and network parameters to learn the attribute embedding are alternately optimized. In the EM algorithm, the prototypes are optimized by encoding the visual and semantic properties so that the learned morphable detector can be well generalized to the novel class.
\subsection{Basic Morphable Detector}
%In this work, we extract the proposals using Region Proposal Network(RPN) and do class-agnostic bounding box regression for each proposal.
We have base class set $\mathcal{C}_{base}$ and novel class set $\mathcal{C}_{\rm novel}$, in which $\mathcal{C}_{\rm base} \cap \mathcal{C}_{novel} = \phi$.
We denote the base class dataset as $D_{\rm base}$ which consists of the training images and the corresponding box annotations. The MD framework applies to a variety of CNN-based detectors~\cite{ren2015faster, dai2016r, lin2017feature, cai2018cascade}. Here we instantiate the framework with Faster R-CNN(FRCNN)~\cite{ren2015faster} because it is a simple and widely used framework. The MD uses Region Proposal Network (RPN)~\cite{ren2015faster} to generate proposals and ROI pooling to extract the proposal features as illustrated in Fig.~\ref{fig.framework}. The MD has two sets of parameters: the network parameters and the class representation (called ``prototypes"). We denote the prototypes for base and novel classes as $\mathcal{P}_{\rm base}$ and $\mathcal{P}_{\rm novel}$ respectively. $\mathcal{P}_{\rm base}$ is learned by joint visual and semantic embeddings in the EM-like approach. Once the MD is trained, $\mathcal{P}_{\rm novel}$ can be computed by forwarding the samples from novel classes through the trained network. Specifically, a trained MD's parameters consist of the network parameter $\Theta$ and prototypes $\mathcal{P}_{\rm base}$. Once $\mathcal{P}_{\rm novel}$ is computed, the MD can be online morphed to a new detector whose parameters consist of $\Theta$, $\mathcal{P}_{\rm base}$ and $\mathcal{P}_{\rm novel}$. As a result, the new detector can detect novel classes.
%Given an image $x_i$ in the training data, $n_i$ object proposals are generated by a Region Proposal Network(RPN) and the ROI features are extracted as $\{z_{i,j}\}^{n_i}_{j=1}$ whose corresponding labels are $\{y_{i,j} \}^{n_i}_{j=1}$, where $y_{i,j} \in \left\{0, 1, ... , n \right\}$. When $y_{i,j}$ is 0, it indicates the proposal belongs to the background. We denote the prototypes for the base classes as $ \mathcal{P}_n = =\left\{P_1, P_2, ..., P_n \right\}$. For each proposal $\{z_{i,j}\}$, the corresponding prototype is $P_{y_{i,j}}$.
%We denote the a few samples for the novel classes as $\mathcal{X}^{novel}$.
%prototypes for the novel classes as $\mathcal{P}_m = =\left\{P_i \right\}^{m}_{i=1}$ where m is the number of novel classes and $P_i$ is the prototype for i-th class. For each sample $x_i$, we denote the corresponding prototype as $P_{y_i}$. We denote the ground truth object features and the corresponding labels as $\mathcal{g}^s , \mathcal{Y}_g^s$. We denote the support samples for the novel classes as $\mathcal{G}^{novel} , \mathcal{Y}^{novel}$.
%Now, we define the Object Detection on Demand(ODOD). The detector model $D_n$ is first be trained using $\left\{ \mathcal{X}^s , \mathcal{Y}^s \right\}$. After that, the $\{ \mathcal{X}^s , \mathcal{Y}^s\}$ is not available anymore. Given the trained model $D_n$ and novel class support samples $\mathcal{G}^{novel} , \mathcal{Y}^{novel}$, we aim to detect the novel classes without decreasing the performance for the base classes. ODOD can be regarded as special few shot detection. %To detect the novel classes, we have $\left\{ \mathcal{G}^{novel} , \mathcal{Y}^{novel} \right\}$. The data samples of the unbase data consists of a few samples of the novel classes. Let $l_u$ denote the number of novel classes specified. %The novel classes can be incrementally specified several times over the time stream. For simplicity, we only discuss the case that the novel classes are specified one time.
%Let $\mathcal{X}^u = \left\{x_i \right\}^{m}_{i=1}$ where $m$ is the number of novel class samples and their corresponding labels $\mathcal{Y}^u$, where $y_i \in \mathcal{L}_u=\left\{l_s+1, l_s+2, ..., l_s+l_u \right\}$. Given the trained detector $D$, and $\left\{ \mathcal{X}^u , \mathcal{Y}^u \right\}$, the detector $D$ is able to be morphed to detector $D'$ to detect all predefined and novel classes without further training. %Object Detection on Demand is to train such morphable detector.
%\subsection{Learning Morphable Detector}
%The morphable detector is trained using the predefined class data and is able to be morphed to a new detector to detect novel class without further training.
%To learn the morphable detector, the attribute space embedding is learned and it is shared among the predefined and novel classes. Each class is assigned one prototype which is the representative embedding of each class in the attribute space embedding.
%The ideal prototypes represents class attributes which are invariant to training samples so that the prototype generated by one or a few samples can generalize well to other samples of that class. We present an EM algorithm to learn the prototypes and network parameters. In E-step, given the trained attribute space embedding, we present an joint visual-semantic objective to optimize the prototypes. In M-step, with the prototypes fixed, we present an prototype based contrastive loss to learn the attribute space embedding. % are optimized by joint encoding the semantic and visual properties of one class. Given the attribute space embedding, the visual properties of each sample is the attribute embedding of it. The joint visual-semantic encoding not only overcome the limitations of the semantic vectors which fail to consider the visual properties but also address the overfiting issue when only visual properties are used.
%Once the novel class is online specified, the prototype of the novel class is generated. The novel class can be detected by finding objects which match the novel class prototype in the attribute embedding. Given the prototypes, the prototype based contrastive loss is used to learn the attribute space embedding.
% \subsection{Basic Morphable Detector Model}
% The Morphable Detector(MD) is trained using base class data $D_{\rm base}$ and the MD can be online morphed to detect the novel classes without extra training. Our MD framework applies to a variety of CNN-based detectors~\cite{ren2015faster, dai2016r, lin2017feature, cai2018cascade}. Here we instantiate the framework with Faster R-CNN(FRCNN)~\cite{ren2015faster} because it is a simple and widely used framework. FRCNN uses a region proposal network (RPN) for generating region proposals and a detection network to do classification and bounding box regression for each proposal. The MD uses RPN to generate proposals and ROI pooling to extract the proposal features as FRCNN as illustrated in Fig.~\ref{fig.framework}. %Our MD learns a background score, feature embedding and class-agnostic bounding box regression for each proposal.
% The MD has two sets of parameters: the network parameters and the class representation (called ``prototypes"). The network parameters are shared by base and novel classes and trained using base class data $D_{\rm base}$. %The MD recognizes visual samples of different classes as if they live in a common space, called feature space. Each class is associated with a prototype which is the target coordinates of each class in the feature space.
% For each proposal, the MD performs class-agnostic box regression and learns a feature embedding whose similarity with prototypes is regarded as the foreground classification score. As it is hard to assign a prototype to the background, the MD directly regresses a background score from the visual features. Once an MD is learned, it is able to use a few samples from a novel class to directly compute its prototype to fulfill the online morphing process. Specifically, a trained MD model is denoted as $\mathcal{D}_{\rm base} = \{\mathcal{N}, \mathcal{P}_{\rm base} \}$ which consists of the Network model $\mathcal{N}$ and the prototypes $\mathcal{P}_{\rm base}$ for base classes. The network model structure is shown in Fig~\ref{fig.framework}. %The network model extracts feature for each proposal and output its class-agnostic bounding box regression, background score, and foreground embedding.
% It computes the prototypes $\mathcal{P}_{\rm novel}$ for novel classes by forwarding the novel samples $D_{\rm novel}$ through the network model $\mathcal{N}$ (the details are in sec.~\ref{sec.morphine}). Then the MD becomes a new detector $\mathcal{D}_{\rm base+novel} = \{\mathcal{N}, \mathcal{P}_{\rm base} \cup \mathcal{P}_{\rm novel}\}$ which is able to detect the base and novel classes.
%$\mathcal{P}_{\rm novel}$ is computed %As the MD does not need back-propagation during the morphing, it does not need extra training and can be easily deployed on the embedded systems. %, like the self-driving car and robotics exploration systems. % so that the morphable detector can satisfy all constraints and requirements for ODOD. %To morph $\mathcal{D}_n$ to be a new detector for the novel classes, the class-agnostic bounding box regression and background score trained using base data can be directly used. The training goal is to achieve an embedding in which the the novel class has good intra-class compactness so that the few samples of the novel class can generate a good prototype for the novel class. The moprhable detector $\mathcal{D}_n = \{\mathcal{N}, \mathcal{P}_n \}$ is morphed to a new detector $\mathcal{D}_{n+m} = \{\mathcal{N}, \mathcal{P}_n, \mathcal{P}_m\}$ which is able to detect the novel classes by adding the prototypes of the novel classes. $\mathcal{P}_m$ is generated by forwarding the support samples $\mathcal{G}^{novel} , \mathcal{Y}^{novel}$ through the network model $\mathcal{N}$ and getting the average embedding. Since we do not change the trained network model during the morphing so that the morphable detector can satisfy all constraints and requirements for ODOD.
\subsection{Training of Morphable Detector}
% In most discriminative detection models~\cite{ren2015faster, dai2016r}, given the proposal $\{x_i, y_i \in {\rm ROI} \}$, the training objective is to train a discriminative model to maximize the likelihood:
% \begin{equation}
% L(D_{\rm base}) = \sum_{x_i, y_i \in {\rm ROI}}P(y_i|x_i)
% \label{eq.mapping1}
% \end{equation}
In the Morphable Detector (MD), each class is associated with one prototype. Suppose we have prototypes for base classes $\mathcal{P_{\rm base}} = \{p_{j}\}$ where $j$ indicates the class. MD generates the proposals $\{x_i,y_i\} \in {\rm ROI}$ by the RPN~\cite{ren2015faster}. The MD learns prototypes for the base classes and a feature space where the feature vector of a given sample is expected to be close to the corresponding prototype while far away from prototypes of other classes. The objective is to maximize the likelihood:
\begin{equation}
\hspace{-4mm}\sum_{x_i, y_i \in {\rm ROI}, y_i >0}P(y_i|p_i)P(p_i|x_i) + \hspace{-6mm} \sum_{x_i, y_i \in {\rm ROI}, y_i = 0}P(y_i|x_i)
\label{eq.likelihood}
\end{equation}
where $P(p_i|x_i)$ or $P(y_i|x_i)$ is determined by the network output and $P(y_i|p_i)$ is determined by the prototype associated with each class.
%Training the model with different prototypes $\mathcal{P}_{base}$ obviously results in a model with different generalizability to the novel classes. %Here we discuss several choices of $P$. We denote the one-hot vectors as $\mathcal{H} \in \mathbb{R}^n = \{ h_1, h_2, ..., h_n\}$ where n is the number of classes and each vector in $\mathcal{H}$ is one hot vector with i-th element is one. We denote the semantic vectors as $\mathcal{S} \in \mathbb{R}^d = \{s_1, s_2, ..., s_n \}$ where the $d$ is the size of the semantic vector. In $\mathcal{S}$, each vector is the semantic vector from natural language model and $s_i$ denotes the semantic vector of the i-th class. We denote the visual vectors as $\mathcal{V} \in \mathbb{R}^d = \{v_1, v_2, ..., v_n\}$ where each vector $v_i$ is the average embedding of the ground truth samples in a given network model $\mathcal{N}$. For visual vectors, in this paper, we use the same embedding size as semantic vectors.
%If the prototypes are one-hot vectors where different classes are represented by different one-hot vectors, the model degenerates to be a discriminative detector model. The discriminative detector is a special case of the morphable detector when the prototypes are one-hot vectors. Since the one-hot vectors are orthogonal and independent to each other, it is difficult to generalize such model to novel classes with a few novel samples. If the prototypes are semantic vectors which is denoted as $\mathcal{S}_{\rm base} \in \mathbb{R}^d = \{s_i\}^{n}_{i=1}$ where the $d$ is the dimension of the semantic vector and $n$ is the number of base classes. %the morphable detector is a zero-shot detector if we use the semantic vectors $\mathcal{S}_{\rm novel}$ for the novel classes as novel prototypes $\mathcal{P}_{\rm novel}$.
%Although the semantic vectors bring useful external information from textual data, the model's generalization power is still limited because the external textual information does not directly examine visual appearances while the model concerns itself with recognizing visual features.
To maximize the above likelihood, we regard the prototype as a hidden variable and propose to learn it by integrating the visual and semantic embedding. The feature embedding and the prototypes are alternately learned in an EM-like approach where the prototypes are initialized with semantic vectors in the initial training and recursively updated over iterations. % which are denoted as $\mathcal{S}_{\rm base} \in \mathbb{R}^d = \{s_i\}^{n}_{i=1}$ where the $d$ is the dimension of the semantic vector and $n$ is the number of base classes.
In ``E"-step, the network parameters $\Theta$ are fixed so the $P(p_{i}|x_i)$ is a constant in Eq~\ref{eq.likelihood}. It aims to update the prototypes for next iterative training. So, the ``E" step takes provided base class data $D_{\rm base}$, the trained model $\mathcal{N}^t$ and the current prototypes $\mathcal{P}_{base}^t$ as inputs, then outputs updated prototypes $\mathcal{P}^{t+1}_{\rm base}$.
\begin{equation}
\mathcal{P}^{t+1}_{\rm base}= E(D_{\rm base}, \mathcal{N}^t, \mathcal{P}_{base}^t).
\label{eq.mapping1}
\end{equation}
In ``M" step, with the prototypes fixed, $P(y_i|p_i)$ is a constant. The optimization is essentially equivalent to the maximum-likelihood estimation of the network parameters. Therefore, the "M" step takes the training data $D_{\rm base}$ and the prototypes $\mathcal{P}_{\rm base}^{t+1}$ for base classes as input, and outputs a newly trained network model $\mathcal{N}^{t+1}$.
\begin{equation}
\mathcal{N}^{t+1}= M(D_{\rm base}, \mathcal{P}_{\rm base}^{t+1}).
\label{eq.mapping1}
\end{equation}
The model usually converges after several iterations.
%More details are in Sec. The above two optimizations happen alternatively and the overall model converges after several iterations.
\begin{figure*}[t]
\centering
\includegraphics[scale=0.55]{MD_train_ver2.pdf}
\caption{Training framework (EM-like approach). The learning of the MD is based on the alternate learning of the feature embedding and the prototypes in an EM-like approach. In ``E" step, with the network fixed, we compute the mean feature vector for each base class on the feature space to update the prototypes associated with that class (see sec. ~\ref{sec.update}). In ``M" step, with the prototypes fixed, we use the prototypes computed in ``E" step to train the network (see sec. ~\ref{sec.train}). }
\label{fig.trianing}
\vspace{-3mm}
\end{figure*}
\vspace{-3mm}
\subsubsection{Network Training}
\label{sec.train}
Given the extracted proposals $\{x_i,y_i\} \in {\rm ROI}$, the deep visual features for each proposal $x_i$ are extracted as $\phi(x_i)$. As it is hard to assign a prototype to the background, the MD directly regresses a background score from the visual features. $\phi(x_i)$ is forwarded through two separate fully connected layers to obtain the background score $b_i \in \mathbb{R}^1$ and the feature vector $f_i \in \mathbb{R}^d$, where $d$ is the dimension of the feature vector. The network is trained with a prototype-based contrastive loss which consists of two terms: the foreground loss and background loss. The foreground loss encourages the feature vector of the proposal $\{x_i, y_i \in {\rm ROI} \}$ to be close to the corresponding prototype while far away from other prototypes if the proposal belongs to the foreground (i.e. $y_i >0$). So, the foreground loss is defined as
% Since the background proposals can not be matched with any prototype, we consider it separately. So we define the losses for the foreground and background separately.
% Given one source data sample $\left\{ x_i , y_i \right\}$ from $\left\{ \mathcal{X}^s , \mathcal{Y}^s \right\}$, the corresponding prototype is denoted as $P_{y_i} \in \mathcal{P}_s = \left\{P_1, P_2, .., P_{l_s} \right\}$. The visual feature is
% \begin{equation}
% v_i = f_v(x_i;\Theta_v)
% \label{eq.loss}
% \end{equation}
% We define the following term,
% \begin{equation}
% e(v_i, P_{y_i}) = \exp(f(v_i;\Theta) \cdot P_{y_i}/\tau)
% \label{eq.loss}
% \end{equation}
%Then we define the contrastive loss for the foreground,
% \begin{small}
% \begin{equation}
% L_{FG} = \sum_{\mbox{\tiny$\begin{array}{c}
% x_i \in \mathcal{X}^s\\
% y_i \in \mathcal{Y}^s\\
% y_i >0 \end{array}$} }^{}-log(\frac{f_i \cdot P_{y_i}}{\exp(b_i) + \sum_{P_j\in \mathcal{P}_n}( f_i \cdot P_{j}) })
% \label{eq.loss}
% \end{equation}
% \end{small}
\begin{small}
\begin{equation}
L_{FG} = \sum_{y_i >0}^{}-{\rm log}(\frac{\exp(f_i \cdot p_{y_i})}{\exp(b_i) + \sum_{p_m\in \mathcal{P}_{\rm base}}\exp( f_i \cdot p_{m}) }),
\label{eq.loss}
\end{equation}
\end{small} where $p_{y_i}$ is the prototype corresponding to class $y_i$. When the proposal belongs to the background, the contrastive loss encourages the background score to be high. The background loss is defined as
\begin{small}
\begin{equation}
L_{BG} = \sum_{y_i =0}^{}-{\rm log}(\frac{\exp{(b_i)}}{\exp(b_i) + \sum_{p_m\in \mathcal{P}_{\rm base}}\exp( f_i \cdot p_{m}) }).
\label{eq.loss}
\end{equation}
\end{small}
The overall loss is sum of the foreground loss, background loss, and a class-agnostic bounding box regression loss~\cite{ren2015faster}
\begin{equation}
L = L_{\rm BG} + L_{\rm FG} + L_{\rm bbox}.
\label{eq.cl}
\end{equation}
\subsubsection{Prototype update}
\label{sec.update}
We denote one ground truth box for class $j$ as $x_i \in g_j$ and the feature vectors of it on the network $\mathcal{N}$ is $\mathcal{N}(x_i)$. Then we compute the mean feature vector $v_j$ of all samples from each class $j$ as \begin{equation}
v_{j} = \frac{1}{|g_{j}|}\sum_{x_i \in \rm g_{j}}\mathcal{N}^t(x_i).
\end{equation}
Then we use the mean feature vector $v_{j}$ to compute the new prototype $p^{t+1}_{j}$ by fusing it with the current prototypes $p^t_{j}$ (associated with class $j$) by weighted element-wise sum,
\begin{equation}
p_{j}^{t+1} = (1-\lambda)v_{j} + \lambda p_{j}^t,
\label{eq.update}
\end{equation}
where $\lambda$ is a constant between 0 and 1. Note that before the element-wise sum, both prototypes and mean feature vectors are normalized.
%An accurate prototype is critical to bridge the predefined and novel classes. Directly using semantic vectors as prototypes is not accurate enough to bridge them. Since the semantic vectors are from the language models which do not consider the visual data, the detector using semantic vectors as prototypes is hard to be generalized well to the novel classes. To overcome this limitation, we propose to make use of the visual information of the visual samples and semantic information of these concepts. The attribute embedding of these samples contains the visual information. %The detector trained only using the visual properties as the prototypes is easy to overfit to a few samples used to extract the visual properties.
%The objective of the EM algorithm is to optimize the prototypes and network parameters. In the EM algorithm, the prototypes are treated as hidden variables. The prototypes and network parameters are optimized alternately. That is, with one fixed, the other one is optimized. The prototype is first initialized from the semantic vectors. An attribute space embedding is learned by using the prototype based
%In order to overcome the issues when only using the visual properties or semantic vectors as the prototype,
% We present \textit{joint visual-semantic embedding}, which makes use of the visual and semantic information to refine the prototypes. Since we need the representative attribute embedding of the visual samples, we use the ground truth samples instead of the proposals used in network training. Let $\mathcal{X}_g^s , \mathcal{Y}_g^s $ represent source data with ground truth as samples. To be more specific, given the trained network with parameter $\Theta$, semantic embedding $\mathcal{S}$ and source data $\mathcal{X}_g^s , \mathcal{Y}_g^s $, \textit{joint visual-semantic embedding} has the following objective
% %where $S_{y_i}$ is the semantic embedding corresponding to class $y_i$, $L_p$ is the objective function of the joint visual-semantic embedding.
% We formulate the $L_p$ as a objective which requires the prototype not only close to its corresponding visual vector but also close to its semantic vector. $S_j$ is the semantic vector corresponding the class j. % The visual properties is the attribute embedding of the visual samples.
% Given the samples $\left\{ x_i , y_i \right\}$ from $\left\{ \mathcal{X}_g^s , \mathcal{Y}_g^s \right\}$, the visual features $v_i$ are extracted. Then the $L_p$ is
% \begin{small}
% \begin{equation}
% L_p = \sum_{j=1}^{l_s} \frac{1}{N_j}\sum_{i=1}^N \mathbbm{1}{(y_i=j)} \norm{ \biggl(f_a(v_i;\Theta_a)-P_{y_i}\biggr) }^2+ \sum_{j=1}^{l_s}\lambda\norm{ \biggl(S_{j}-P{j}\biggr) }^2
% \label{eq.Estep}
% \end{equation}
% \end{small}
% where $N_j$ is the number of ground truth samples belonging to class $j$, $l_s$ is number of predefined classes, $\lambda$ is a weighting parameter which is set to 1 in this paper. To eliminate the scale effects, we normalize the $f(x_i; \Theta)$ and $S_i$. The joint visual-semantic encoding not only overcomes the limitations of the semantic vectors which fail to consider the visual properties but also addresses the overfiting issue when only visual properties are used.
% For this function, we derive the closed-form solution of the prototype for class j,
% \begin{equation}
% P_j = \frac{\frac{1}{N_j}\sum_{i=1}^N \mathbbm{1}{(y_i=j) } f_a(v_i;\Theta_a)+\lambda S_{j}}{1+\lambda}
% \label{eq.prototype}
% \end{equation}
% This optimized prototype is the linear combination of the semantic vector and the mean visual vectors of that class.
\subsection{Online Morphing}
\label{sec.morphine}
The online morphing is to compute the new prototypes for the novel classes as shown in Fig~\ref{fig.framework}. Suppose we have a ground truth box $x_i \in g_{j}$ from novel class $j$, and forward the samples through the network to get the feature vectors $\mathcal{N}(x_i)$. The mean feature vector of all samples belonging to a novel class $j$ is used as the new prototype for that class,
\begin{equation}
p_{j} = \frac{1}{|g_{j}|}\sum_{x_i \in \rm g_{j}}\mathcal{N}(x_i) .
\end{equation}
Now, we have the $\mathcal{P}_{\rm novel} = \{p_j\}$ where $j$ is the class index for a novel class, so the novel class can be detected. Given a test image, the RPN first generates the proposals $x_i \in {\rm ROI}$, and get the bbox score $b_i$ and feature vector $f_i$. The class posterior probability for class $j$ is,
\begin{small}
\begin{equation}
\frac{\exp(f_i \cdot p_{j})}{\exp(b_i) + \sum_{p_m\in \mathcal{P}_{\rm base} \cup \mathcal{P}_{\rm novel}}\exp( f_i \cdot p_{m}) }.
\label{eq.loss}
\end{equation}
\end{small}
Then, the detected boxes are obtained by setting a threshold for the class score like other detectors~\cite{ren2015faster}.
% \begin{table}[t]
% \centering
% % \scriptsize
% % \setlength{\tabcolsep}{1.2mm}
% {
% \begin{tabular}{|l|ccccc|}
% \hline
% \multirow{2}{*}{Method} & \multirow{1}{*}{Average Precision} \\
% & 0.5::0.95 & 0.5 & 0.75 \\ \hline
% \multicolumn{1}{|l|}{MRD} & 31.8 & 10.4 & 14.8 \\
% \end{tabular}
% }
% \caption{Comparison of Our method with different choice of the prior-focus regularization on ADE20K under 100-10 setting.}
% \label{tab:100-10-justify}
% \end{table}
\begin{table}[t]
% \begin{tabular*}
\centering
% \scriptsize
% \setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{0.49\textwidth}{@{\extracolsep{\fill}}p{2cm}<{\centering}|p{1.6cm}<{\centering}|p{1.5cm}<{\centering}|p{1.5cm}<{\centering}}
\hline
\multicolumn{1}{l|}{{Method}} & {\footnotesize Extra training} & $\rm AP$ & $\rm AP_{0.5}$ \\
\hline
\hline
\multicolumn{1}{c|}{FRCNN~\cite{ren2015faster}} & \cmark & - & 23.0 \\
\multicolumn{1}{c|}{LSTD~\cite{chen2018lstd}} & \cmark & - & 24.2 \\
\multicolumn{1}{c|}{FSOD method~\cite{fan2020fsod}} & \xmark & - & 27.5 \\
\hline
\multicolumn{1}{c|}{Visual (ImageNet)} & \xmark & 10.1 & 16.3 \\
\multicolumn{1}{c|}{Visual (FRCNN)} & \xmark & 15.5 & 22.6 \\
\multicolumn{1}{c|}{MD (concat)} & \xmark & 17.3 &29.9 \\
\multicolumn{1}{c|}{MD ($\lambda=0$)} & \xmark & 21.5 & 36.2 \\
\multicolumn{1}{c|}{MD ($\lambda=0.3$)} & \xmark & 21.3 & 35.9 \\
\multicolumn{1}{c|}{MD ($\lambda=0.7$)} & \xmark & 21.6 & 36.3 \\
\multicolumn{1}{c|}{MD (iter1)} & \xmark & 18.2 & 31.2 \\
\multicolumn{1}{c|}{MD (iter2)} & \xmark & 21.9 & 36.7 \\
\multicolumn{1}{c|}{MD} & \xmark & \textbf{22.2} & \textbf{37.1} \\
\hline
\end{tabular*}
}
\caption{Comparison of our method with different baselines and different variants on FSOD Dataset}
\label{tab:fsod}
\end{table}
\begin{table}[t]
% \begin{tabular*}
\centering
% \scriptsize
% \setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{0.49\textwidth}{@{\extracolsep{\fill}}p{0.7cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}|p{0.7cm}<{\centering}}
\hline
\multicolumn{1}{c|}{\multirow{2}{*}{Method}} & \multicolumn{2}{c|}{Split 1} & \multicolumn{2}{c|}{Split 2} & Ave \\
& $\rm AP$ & $\rm AP_{0.5}$ & $\rm AP$ & $\rm AP_{0.5}$ & $\rm AP_{0.5}$\\
\hline
\hline
\multicolumn{1}{c|}{OSOD~\cite{hsieh2019one}} & - & - & - & -& 22.0
\\
\multicolumn{1}{c|}{MD (iter1)} & 20.2& 32.9 &21.1 &32.6& 32.8
\\
\multicolumn{1}{c|}{MD} &
\textbf{21.5}&
\textbf{33.0}&
\textbf{24.9}&
\textbf{36.1}& \textbf{34.6}\\
\hline
\end{tabular*}
}
\caption{One-shot detection performance comparison on the first two splits of COCO dataset for the novel classes.}
\label{tab:coco1}
\end{table}
\begin{table*}[t]
% \begin{tabular*}
\centering
% \scriptsize
% \setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}p{1.5cm}<{\centering}|p{1.5cm}<{\centering}|p{1.5cm}<{\centering}|p{1.5cm}<{\centering}|p{1.7cm}<{\centering}|p{1.5cm}<{\centering}|p{1.7cm}<{\centering}|p{1.7cm}<{\centering}|p{2cm}<{\centering}}
\hline
\multicolumn{1}{c|}{\multirow{2}{*}{\footnotesize Method}} & {\footnotesize Ours (1-shot)} &{\footnotesize FSView~\cite{xiao2020few} (1-shot)}& {\footnotesize LSTD~\cite{chen2018lstd} (10-shot)} & {\footnotesize MetaYOLO~\cite{kang2019few} (10-shot)}& {\footnotesize MetaDet~\cite{wang2019meta} (10-shot)}& { \footnotesize MetaRCNN~\cite{yan2019meta} (10-shot)} & {\footnotesize TFA w/fc~\cite{wang2020frustratingly} (10-shot)}&{\footnotesize TFA-w/cos~\cite{wang2020frustratingly} (10-shot)} \\
\hline
\hline
\multicolumn{1}{c|}{$\rm AP$} & \underline{9.7} & 4.5 &3.2 &5.6&7.1&8.7& \textbf{10.0}&\textbf{10.0}\\
\multicolumn{1}{c|}{$\rm AP_{0.5}$} & \underline{15.0} & 12.4&8.1&12.3&14.6 &\textbf{19.1}&-&- \\
\multicolumn{1}{c|}{$\rm AP_{0.75}$} & \textbf{9.9} & 2.2&2.1 &4.6&6.1 &6.6&9.2& \underline{9.3} \\
\hline
\end{tabular*}
}
\vspace{-2mm}
\caption{Performance comparison with state-of-the-arts on COCO dataset for novel classes in split 3. The best one and the second-best one are highlighted in \textbf{bold} and \underline{underlined} respectively.}
\label{tab:coco2}
\end{table*}
\vspace{-2mm}
\begin{table*}[t]
% \begin{tabular*}
\centering
% \scriptsize
% \setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}p{1.5cm}<{\centering}|p{1.5cm}<{\centering}|p{1.5cm}<{\centering}|p{1.5cm}<{\centering}|p{1.7cm}<{\centering}|p{1.5cm}<{\centering}|p{1.7cm}<{\centering}|p{1.7cm}<{\centering}|p{2cm}<{\centering}}
\hline
\multicolumn{1}{c|}{{Method}} & {\footnotesize Ours} &{\footnotesize FSView~\cite{xiao2020few}}&{\footnotesize LSTD~\cite{chen2018lstd}} & {\footnotesize MetaYOLO~\cite{kang2019few}}& {\footnotesize MetaDet~\cite{wang2019meta}}&{\footnotesize MetaRCNN~\cite{yan2019meta}}& {\footnotesize TFA w/fc~\cite{wang2020frustratingly}}&{\footnotesize TFA w/cos~\cite{wang2020frustratingly}} \\
\hline
\hline
\multicolumn{1}{c|}{Split 1} & \textbf{53.2} & 24.2 &8.4 &14.8 & 18.9&19.9&22.9&25.3\\
\multicolumn{1}{c|}{Split 2} & \textbf{41.6} & 21.6 & 11.4 &15.7 & 21.8 &10.4 & 16.9 & 18.3\\
\multicolumn{1}{c|}{Split 3} & \textbf{38.6} & 21.2 &12.6 &21.3 & 20.6 &14.3&15.7&17.9 \\
\hline
\end{tabular*}
}
\caption{Performance comparison with state-of-the-arts PASCAL VOC dataset for novel classes in three splits.}
\vspace{-0.2cm}
\label{tab:voc}
\end{table*}
% \begin{table}
% % \begin{center}
% %\begin{adjustbox}{width=19cm, center}
% \begin{tabular}{ccc}
% & AP & mAP\_0.5 \\
% MD(iter 0)/Semantic & 16.1 & 29.2 \\
% MD(iter 1) & 16.7 & 30.3 \\
% MD(iter 2) & 17.0 & 30.5 \\
% One hot vector & & \\
% Visual Vector & & \\
% MD(λ = 0.3) & 16.5 & 29.7 \\
% MD(λ = 0.7) & 16.7 & 30.1 \\
% MD(λ = 1.0) & 16.8 & 30.2 \\
% MD(concat) & & \\
% SOTA 2 & - & 27.5
% \end{tabular}
% %\end{adjustbox}
% \caption{Performance comparison on COCO 2014 minival in split 1.}
% \label{tab:coco_1}
% \end{table}
\section{Related Work}
\paragraph{Zero / few Shot Learning}
Zero Shot Learning (ZSL)~\cite{elhoseiny2017link,elhoseiny2013write,qiao2016less,reed2016learning,akata2015label,lampert2009learning,lampert2013attribute, zhao2019recognizing} has been widely studied for image recognition. It aims to
recognize unseen classes without training samples. People usually leverage semantic information~\cite{elhoseiny2017link,elhoseiny2013write,qiao2016less,reed2016learning} or attributes representation~\cite{akata2015label,lampert2009learning,lampert2013attribute} for ZSL. Few Shot Learning aims to recognize a class with a few annotated samples. People try to address this problem by metric learning based approaches~\cite{vinyals2016matching,snell2017prototypical,oreshkin2018tadam,triantafillou2017few,hariharan2017low} or meta-learning based approaches~\cite{ravi2016optimization, zhang2018metagan, munkhdalai2017meta, munkhdalai2018rapid}. Different from them, in this work, we focus a more challenging object detection task.
\vspace{-1mm}
% try to address this problem by meta-learning based approach and use metric
\paragraph{Object Detection tasks} %In recent years, significant progresses~\cite{cai2016unified,cai2018cascade,lin2017feature,liu2016ssd,lin2017focal,redmon2016you,redmon2017yolo9000,redmon2018yolov3,ren2015faster} has been made in object detection since deep convolutional neutral network has
%been successfully used in R-CNN~\cite{girshick2014rich, girshick2015fast}.
Most of existing detection methods~\cite{cai2016unified,cai2018cascade,lin2017feature,liu2016ssd,lin2017focal,redmon2016you,redmon2017yolo9000,redmon2018yolov3, zhao2020object, zhao2018pseudo, ren2015faster,girshick2014rich, girshick2015fast} focus on general object detection task where each category has large number of annotated data. However, when the labeled data are
scarce or not available, the models can overfit or fail to generalize. So people start to focus on zero-shot / few-shot detection~\cite{li2019zero, bansal2018zero, zhu2020don, kang2019few,wang2020frustratingly,wang2019meta,yan2019meta,xiao2020few, chen2018lstd} tasks where no example or a few examples for novel category are given. But the models can suffer from catastrophic forgetting~\cite{mccloskey1989catastrophic} when the
past data are not available. People start to work on incremental few-shot detection~\cite{perez2020incremental} task. Different from the above tasks, we focus on a more challenging task, Object Detection on Demand, where only a few samples for novel categories are given and no extra training is required.
%\paragraph{Object Detection}Recently, the deep convolutional neutral network has been widely used in object detection after the first trial in R-CNN~\cite{girshick2014rich}. Object Detection has been improved in different aspects. Faster R-CNN~\cite{ren2015faster} first unifies the two stages in a unified end-to-end manner so that the detection accuracy is significantly improved. MS-CNN~\cite{cai2016unified} and FPN~\cite{lin2017feature} detect proposals at multiple output layers so as to alleviate the scale mismatch between the feature receptive field size and object size. Cascade R-CNN~\cite{cai2018cascade} trains a sequence of the detectors with increasing IoU thresholds to be sequentially more selective against close false positives. To improve the model efficiency, RFCN~\cite{dai2016r} proposes position-sensitive score maps through fully-convolutional layers to avoid using a fully connected layer, and YOLO~\cite{redmon2016you,redmon2017yolo9000,redmon2018yolov3} and SSD~\cite{liu2016ssd} propose a one-stage proposal-free framework by directly classifying the pre-defined anchors and regressing the bounding boxes. RetinaNet~\cite{lin2017focal} proposes a new focal loss to address the severe foreground-background imbalance problem, which is critical in one-stage detectors. Although these methods achieve great success, they still have some limitations. First, they rely on a large number of labeled data and can not handle the long-tail issue. Second, the categories to detect are offline defined, so it can not be scaled to solve open-ended detection problem.
\vspace{-6mm}
\paragraph{Non-morphable Detector}
Recent works~\cite{zhu2020don, kang2019few,wang2020frustratingly,wang2019meta,yan2019meta,xiao2020few, chen2018lstd} have made significant progresses on few-shot detection tasks. They aim to leverage fully labeled seen category data to train a base model and adapt this model to novel classes using a meta feature
learner(i.e. extra training). However, extra training is unfeasible for emerging applications of robots. Different from them, we propose to train a morphable detector that can be online morphed to detect novel categories without any extra training.
\vspace{-5mm}
\paragraph{Morphable Detector} One basic morphable detector is a zero-shot detector~\cite{li2019zero,zhu2019zero,bansal2018zero} which can detect the novel categories by leveraging semantic information without any annotated examples. However, as the semantic information does not directly examine the visual appearances, zero-shot detectors have limited generalization capability and the overall performance is far from satisfactory. Another morphable detector is some few-shot detector which takes ``query-target" pairs as input to detect all instances of the ``target" object appearing in the ``query" image. But this model learns less discriminative representations as the target representations are always changing in training. RepNet~\cite{karlinsky2019repmet} propose to learn a few representatives for each category from the visual data. But learning from visual data is not enough for the model generalization. Different from them, we propose to leverage external semantic information~\cite{mikolov2013distributed} and present an EM-like approach to integrate the visual and semantic embeddings.
\vspace{-2mm}
%The few-shot detection models~\cite{kim2020few,wang2019meta,yan2019meta,kang2019few,wang2020frustratingly} are proposed to address the \textit{long-tail} issue. In a few shot detection settings, the detector is first trained with the source categories for which there is much data available, then finetune the model using a few novel category data. These kind of approaches mostly rely on the finetuning process, so most of them design a novel finetuning procedure to achieve better performance on the novel categories. However, finetuning takes some time, and it is hard for them to train a detector in real-time due to resource limitations. Recently, Hsieh et al.~\cite{hsieh2019one} proposed a one-shot detection model that does not need further finetuning, but this model easily overfits to the training data. From the reported results, the training categories' performance is even worse than the novel categories’.
%Zero shot detection aims to detect the novel category even without seeing it. [][][] aims to model the semantic relationship among the categories
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage{iccv}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bbm}
\usepackage{adjustbox}
\usepackage{bbding}
\usepackage{multirow}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{array}
\usepackage{multirow}
\usepackage{float}
%\usepackage{graphicx}
\usepackage{color}
\usepackage{amsfonts,amssymb}
%\usepackage{lmodern}
\usepackage{bm}
\usepackage{subfig}
\usepackage{algorithm,algorithmic}
\usepackage{lipsum}
\usepackage{pifont}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
% \usepackage{multicol}
% Include other packages here, before hyperref.
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}
\iccvfinalcopy % *** Uncomment this line for the final submission
\def\iccvPaperID{2865} % *** Enter the ICCV Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
% Pages are numbered in submission mode, and unnumbered in camera-ready
\ificcvfinal\pagestyle{empty}\fi
\begin{document}
%%%%%%%%% TITLE
\title{Supplementary Material - Morphable Detector for Object Detection on Demand}
\author{First Author\\
Institution1\\
Institution1 address\\
{\tt\small firstauthor@i1.org}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
\and
Second Author\\
Institution2\\
First line of institution2 address\\
{\tt\small secondauthor@i2.org}
}
\maketitle
% Remove page # from the first page of camera-ready.
\ificcvfinal\thispagestyle{empty}\fi
\section{Split details}
\begin{figure*}
\centering
\includegraphics[scale=0.7]{LaTeX/figure/visual_coco.pdf}
\caption{Qualitative results of our proposed Morphable Detector on COCO 2017 validation set. The model is trained in split 1 and we only visualize the qualitative results for the novel categories}
\label{fig.coco_visual}
\vspace{-3mm}
\end{figure*}
On the VOC dataset, we follow most few shot detectors~\cite{yan2019meta,kang2019few,wang2020frustratingly} and use the same splits as them. The results details are listed in Table~\ref{tab:voc}. On the COCO dataset, we group the 80 categories into 5 different clusters and randomly select two categories from each cluster as the novel categories. In the split 1, we select \textit{bicycle} \textit{car} \textit{dog} \textit{sheep} \textit{apple} \textit{frisbee} \textit{surfboard} \textit{pizza} \textit{laptop} \textit{microwave} \textit{refrigerator} as the novel categories. The results details are in Table~\ref{tab:coco1}. In the split 2, we select car \textit{train} \textit{boat} \textit{dog} \textit{horse} \textit{skateboard} \textit{sandwich} \textit{pizza} \textit{keyboard} \textit{microwave} as the novel categories. The results details are in Table~\ref{tab:coco2}. In the split 3, we use all PASCAL VOC categories as the novel categories. The results details are in Table~\ref{tab:coco3}.
\section{Qualitative Results}
More qualitative results on FSOD test set~\cite{fan2020fsod} are in Figure~\ref{fig.fsod}. Qualitative results on COCO validation set are in Figure~\ref{fig.coco_visual}. The model is trained in split 1 and we only visualize the qualitative results for the novel categories.
%%%%%%%%% ABSTRACT
\begin{figure*}[t]
\centering
\includegraphics[scale=0.7]{LaTeX/figure/visual_fsod_2.pdf}
\caption{Qualitative results of our proposed Morphable Detector on FSOD test set.}
\label{fig.fsod}
\vspace{-3mm}
\end{figure*}
\begin{table*}[t]
% \begin{tabular*}
\centering
\scriptsize
\setlength{\tabcolsep}{0.7mm}
{
\center
\begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}p{0.7cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}}
\hline
\multicolumn{1}{l|}{\multirow{2}{*}{Method}} & \multicolumn{6}{c|}{Split 1} & \multicolumn{6}{c|}{Split 2} & \multicolumn{6}{c}{Split 3} \\
&bird & bus & cow& mbike& sofa& mean& aero& bottle& cow &horse &sofa &mean& boat& cat& mbike& sheep& sofa &mean\\
\hline
\hline
\multicolumn{1}{l|}{Meta-YOLO~\cite{kang2019few}} &13.5 & 10.6 &30.5 &13.8& 4.3& 14.8 &11.8 &9.1 &15.6& 23.7& 18.2& 15.7& 10.8& 44.0& 17.8& 18.1& 5.3& 19.2 \\
\multicolumn{1}{l|}{Meta-RCNN~\cite{yan2019meta}} &6.10 & 32.8 &15.0 &35.4 &0.2 &19.9 &23.9& 0.8& 23.6& 3.1& 0.7& 10.4& 0.6& 31.1& 28.9& 11.0& 0.1& 14.3 \\
\multicolumn{1}{l|}{MD} &10.9
&42.0&24.9 &55.0 &5.4 &27.6 &62.2& 58.7& 58.6&65.8&20.5& 53.2&
25.9
&71.3
&59.9
&60.0
&13.5
&46.1
\\
\hline
\end{tabular*}
}
\caption{The results on the PASCAL VOC dataset for novel categories. }
\label{tab:voc}
\end{table*}
\begin{table*}[t]
% \begin{tabular*}
\centering
% \scriptsize
\setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}p{1cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{1.2cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{1.2cm}<{\centering}p{1.2cm}<{\centering}p{0.5cm}<{\centering}}
\hline
& {\footnotesize bicycle} &{\footnotesize car}&{\footnotesize dog} & {\footnotesize sheep}&{\footnotesize apple}& {\footnotesize frisbee}&{\footnotesize surfboard}& {\footnotesize pizza}&{\footnotesize laptop}&{\footnotesize microwave} &{\footnotesize refrigerator} &{\footnotesize ave} \\
\hline
\hline
$\rm AP$ & 9.70
&15.5
&26.1
&24.5
&3.1
&14.0
&17.3
&18.3
&27.1
&17.0
&31.0
&18.5 \\
$\rm AP_{50}$&
22.7
&33.1
&41.1
&39.2
&5.4
&31.6
&24.6
&29.0
&41.9
&22.8
&45.1
&30.6 \\
$\rm AP_{75}$&
6.60
&12.4
&28.3
&26.2
&3.1
&10.8
&24.2
&20.0
&30.5
&20.0
&37.8
&20.0 \\
\hline
\end{tabular*}
}
\caption{The results in the split 1 on the COCO dataset for the novel categories.}
\vspace{-0.2cm}
\label{tab:coco1}
\end{table*}
\begin{table*}[t]
% \begin{tabular*}
\centering
% \scriptsize
\setlength{\tabcolsep}{0.7mm}
{
\center
\begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}p{1cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{1.2cm}<{\centering}p{1.2cm}<{\centering}p{0.7cm}<{\centering}p{1.2cm}<{\centering}p{1.2cm}<{\centering}p{0.7cm}<{\centering}}
\hline
& {\footnotesize car} &{\footnotesize train}&{\footnotesize boat} & {\footnotesize dog}&{\footnotesize horse}& {\footnotesize skateboard}&{\footnotesize sandwich}& {\footnotesize pizza}&{\footnotesize keyboard}&{\footnotesize microwave} &{\footnotesize ave} \\
\hline
\hline
$\rm AP$ &
19.2
&19.5
&31.0
&23.4
&17.3
&10.5
&3.35
&21.8
&10.1
&7.62
&16.4
\\
$\rm AP_{50}$&
38.4
&29.8
&48.9
&39.9
&30.6
&18.0
&6.18
&29.1
&16.4
&12.3
&27.0 \\
$\rm AP_{75}$&
17.0
&22.0
&34.1
&24.1
&18.3
&10.8
&2.93
&28.2
&10.1
&9.16
&17.7 \\
\hline
\end{tabular*}
}
\caption{The results in the split 2 on the COCO dataset for the novel categories.}
\vspace{-0.2cm}
\label{tab:coco2}
\end{table*}
\begin{table*}[t]
% \begin{tabular*}
\centering
\scriptsize
\setlength{\tabcolsep}{1.2mm}
{
\center
\begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}p{0.7cm}<{\centering}|p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.5cm}<{\centering}p{1.0cm}<{\centering}p{1.0cm}<{\centering}p{0.4cm}<{\centering}p{0.5cm}<{\centering}p{0.5cm}<{\centering}p{0.5cm}<{\centering}p{0.4cm}<{\centering}p{0.5cm}<{\centering}p{0.6cm}<{\centering}p{0.6cm}<{\centering}p{0.4cm}<{\centering}p{0.5cm}<{\centering}p{0.5cm}<{\centering}p{0.5cm}<{\centering}p{0.5cm}<{\centering}p{0.5cm}<{\centering}p{0.3cm}<{\centering}p{0.5cm}<{\centering}}
\hline
& {\footnotesize person} &{\footnotesize bicycle}&{\footnotesize car} & {\footnotesize motorbike}&{\footnotesize airplane}& {\footnotesize bus}&{\footnotesize train}& {\footnotesize boat}&{\footnotesize bird}&{\footnotesize cat} &{\footnotesize dog}&{\footnotesize horse}&{\footnotesize sheep} & {\footnotesize cow}&{\footnotesize bottle}& {\footnotesize chair}&{\footnotesize sofa}& {\footnotesize plant}&{\footnotesize table}&{\footnotesize tv} &{\footnotesize ave} \\
\hline
\hline
$\rm AP$ &
13.0
&4.2
&18.5
&3.7
&11.8
&8.7
&8.9
&3.1
&1.7
&24.7
&9.25
&14.3
&7.85
&12.5
&8.4
&3.4
&10.7
&3.4
&3.4
&23.2
&9.7
\\
$\rm AP_{50}$
&20.1
&9.5
&33.9
&7.9
&16.6
&12.5
&13.8
&7.01
&2.71
&39.6
&13.8
&21.9
&11.6
&18.9
&13.0
&7.33
&19.5
&6.45
&5.87
&34.2
&15.8
\\
$\rm AP_{75}$
&14.9
&2.7
&17.7
&3.5
&14.1
&9.9
&9.5
&2.4
&1.7
&28.4
&10.1
&16.5
&8.8
&13.9
&9.2
&2.7
&9.8
&3.4
&3.1
&24.5
&10.3
\\
\hline
\end{tabular*}
}
\caption{Performance details in the split 3 on the COCO dataset for the novel categories.}
\vspace{-0.2cm}
\label{tab:coco3}
\end{table*}
{\small
\bibliographystyle{ieee_fullname}
\bibliography{egbib}
}
\end{document}
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{akata2015label}
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid.
\newblock Label-embedding for image classification.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
38(7):1425--1438, 2015.
\bibitem{bansal2018zero}
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran.
\newblock Zero-shot object detection.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}, pages 384--400, 2018.
\bibitem{bell2016inside}
Sean Bell, C Lawrence~Zitnick, Kavita Bala, and Ross Girshick.
\newblock Inside-outside net: Detecting objects in context with skip pooling
and recurrent neural networks.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 2874--2883, 2016.
\bibitem{cai2016unified}
Zhaowei Cai, Quanfu Fan, Rogerio~S Feris, and Nuno Vasconcelos.
\newblock A unified multi-scale deep convolutional neural network for fast
object detection.
\newblock In {\em European conference on computer vision}, pages 354--370.
Springer, 2016.
\bibitem{cai2018cascade}
Zhaowei Cai and Nuno Vasconcelos.
\newblock Cascade r-cnn: Delving into high quality object detection.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 6154--6162, 2018.
\bibitem{chen2018lstd}
Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao.
\newblock Lstd: A low-shot transfer detector for object detection.
\newblock In {\em Thirty-Second AAAI Conference on Artificial Intelligence},
2018.
\bibitem{dai2016r}
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun.
\newblock R-fcn: Object detection via region-based fully convolutional
networks.
\newblock In {\em Advances in neural information processing systems}, pages
379--387, 2016.
\bibitem{deng2009imagenet}
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
\newblock Imagenet: A large-scale hierarchical image database.
\newblock In {\em 2009 IEEE conference on computer vision and pattern
recognition}, pages 248--255. Ieee, 2009.
\bibitem{elhoseiny2013write}
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal.
\newblock Write a classifier: Zero-shot learning using purely textual
descriptions.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 2584--2591, 2013.
\bibitem{elhoseiny2017link}
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal.
\newblock Link the head to the" beak": Zero shot learning from noisy text
description at part precision.
\newblock In {\em 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, pages 6288--6297. IEEE, 2017.
\bibitem{Everingham15}
M. Everingham, S.~M.~A. Eslami, L. Van~Gool, C.~K.~I. Williams, J. Winn, and A.
Zisserman.
\newblock The pascal visual object classes challenge: A retrospective.
\newblock {\em International Journal of Computer Vision}, 111(1):98--136, Jan.
2015.
\bibitem{everingham2010pascal}
Mark Everingham, Luc Van~Gool, Christopher~KI Williams, John Winn, and Andrew
Zisserman.
\newblock The pascal visual object classes (voc) challenge.
\newblock {\em International journal of computer vision}, 88(2):303--338, 2010.
\bibitem{fan2020fsod}
Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai.
\newblock Few-shot object detection with attention-rpn and multi-relation
detector.
\newblock In {\em CVPR}, 2020.
\bibitem{girshick2015fast}
Ross Girshick.
\newblock Fast r-cnn.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision}, pages 1440--1448, 2015.
\bibitem{girshick2014rich}
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.
\newblock Rich feature hierarchies for accurate object detection and semantic
segmentation.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 580--587, 2014.
\bibitem{hariharan2017low}
Bharath Hariharan and Ross Girshick.
\newblock Low-shot visual recognition by shrinking and hallucinating features.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 3018--3027, 2017.
\bibitem{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 770--778, 2016.
\bibitem{hsieh2019one}
Ting-I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, and Tyng-Luh Liu.
\newblock One-shot object detection with co-attention and co-excitation.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
2725--2734, 2019.
\bibitem{alibaba2020mnn}
Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng
Yang, Zongyang Cui, Yu Cai, Tianhang Yu, Chengfei Lv, and Zhihua Wu.
\newblock Mnn: A universal and efficient inference engine.
\newblock In {\em MLSys}, 2020.
\bibitem{kang2019few}
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell.
\newblock Few-shot object detection via feature reweighting.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 8420--8429, 2019.
\bibitem{karlinsky2019repmet}
Leonid Karlinsky, Joseph Shtok, Sivan Harary, Eli Schwartz, Amit Aides, Rogerio
Feris, Raja Giryes, and Alex~M Bronstein.
\newblock Repmet: Representative-based metric learning for classification and
few-shot object detection.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 5197--5206, 2019.
\bibitem{lampert2009learning}
Christoph~H Lampert, Hannes Nickisch, and Stefan Harmeling.
\newblock Learning to detect unseen object classes by between-class attribute
transfer.
\newblock In {\em 2009 IEEE Conference on Computer Vision and Pattern
Recognition}, pages 951--958. IEEE, 2009.
\bibitem{lampert2013attribute}
Christoph~H Lampert, Hannes Nickisch, and Stefan Harmeling.
\newblock Attribute-based classification for zero-shot visual object
categorization.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
36(3):453--465, 2013.
\bibitem{li2019zero}
Zhihui Li, Lina Yao, Xiaoqin Zhang, Xianzhi Wang, Salil Kanhere, and Huaxiang
Zhang.
\newblock Zero-shot object detection with textual descriptions.
\newblock In {\em Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~33, pages 8690--8697, 2019.
\bibitem{lin2017feature}
Tsung-Yi Lin, Piotr Doll{\'a}r, Ross Girshick, Kaiming He, Bharath Hariharan,
and Serge Belongie.
\newblock Feature pyramid networks for object detection.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 2117--2125, 2017.
\bibitem{lin2017focal}
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll{\'a}r.
\newblock Focal loss for dense object detection.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision}, pages 2980--2988, 2017.
\bibitem{lin2014microsoft}
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Doll{\'a}r, and C~Lawrence Zitnick.
\newblock Microsoft coco: Common objects in context.
\newblock In {\em European conference on computer vision}, pages 740--755.
Springer, 2014.
\bibitem{liu2016ssd}
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed,
Cheng-Yang Fu, and Alexander~C Berg.
\newblock Ssd: Single shot multibox detector.
\newblock In {\em European conference on computer vision}, pages 21--37.
Springer, 2016.
\bibitem{mccloskey1989catastrophic}
Michael McCloskey and Neal~J Cohen.
\newblock Catastrophic interference in connectionist networks: The sequential
learning problem.
\newblock In {\em Psychology of learning and motivation}, volume~24, pages
109--165. Elsevier, 1989.
\bibitem{mikolov2013distributed}
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg~S Corrado, and Jeff Dean.
\newblock Distributed representations of words and phrases and their
compositionality.
\newblock In {\em Advances in neural information processing systems}, pages
3111--3119, 2013.
\bibitem{munkhdalai2017meta}
Tsendsuren Munkhdalai and Hong Yu.
\newblock Meta networks.
\newblock In {\em International Conference on Machine Learning}, pages
2554--2563. PMLR, 2017.
\bibitem{munkhdalai2018rapid}
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler.
\newblock Rapid adaptation with conditionally shifted neurons.
\newblock In {\em International Conference on Machine Learning}, pages
3664--3673. PMLR, 2018.
\bibitem{oreshkin2018tadam}
Boris~N Oreshkin, Pau~Rodr{\'\i}guez L{\'o}pez, and Alexandre Lacoste.
\newblock Tadam: Task dependent adaptive metric for improved few-shot learning.
\newblock In {\em NeurIPS}, 2018.
\bibitem{perez2020incremental}
Juan-Manuel Perez-Rua, Xiatian Zhu, Timothy~M Hospedales, and Tao Xiang.
\newblock Incremental few-shot object detection.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 13846--13855, 2020.
\bibitem{qiao2016less}
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton Van Den~Hengel.
\newblock Less is more: zero-shot learning from online textual documents with
noise suppression.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 2249--2257, 2016.
\bibitem{ravi2016optimization}
Sachin Ravi and Hugo Larochelle.
\newblock Optimization as a model for few-shot learning.
\newblock 2016.
\bibitem{redmon2016you}
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.
\newblock You only look once: Unified, real-time object detection.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 779--788, 2016.
\bibitem{redmon2017yolo9000}
Joseph Redmon and Ali Farhadi.
\newblock Yolo9000: better, faster, stronger.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 7263--7271, 2017.
\bibitem{redmon2018yolov3}
Joseph Redmon and Ali Farhadi.
\newblock Yolov3: An incremental improvement.
\newblock {\em arXiv preprint arXiv:1804.02767}, 2018.
\bibitem{reed2016learning}
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele.
\newblock Learning deep representations of fine-grained visual descriptions.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 49--58, 2016.
\bibitem{ren2015faster}
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
\newblock Faster r-cnn: Towards real-time object detection with region proposal
networks.
\newblock In {\em Advances in neural information processing systems}, pages
91--99, 2015.
\bibitem{snell2017prototypical}
Jake Snell, Kevin Swersky, and Richard Zemel.
\newblock Prototypical networks for few-shot learning.
\newblock In {\em Advances in neural information processing systems}, pages
4077--4087, 2017.
\bibitem{triantafillou2017few}
Eleni Triantafillou, Richard~S Zemel, and Raquel Urtasun.
\newblock Few-shot learning through an information retrieval lens.
\newblock In {\em NIPS}, 2017.
\bibitem{van2008visualizing}
Laurens Van~der Maaten and Geoffrey Hinton.
\newblock Visualizing data using t-sne.
\newblock {\em Journal of machine learning research}, 9(11), 2008.
\bibitem{vinyals2016matching}
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et~al.
\newblock Matching networks for one shot learning.
\newblock In {\em Advances in neural information processing systems}, pages
3630--3638, 2016.
\bibitem{wang2020frustratingly}
Xin Wang, Thomas~E Huang, Trevor Darrell, Joseph~E Gonzalez, and Fisher Yu.
\newblock Frustratingly simple few-shot object detection.
\newblock {\em arXiv preprint arXiv:2003.06957}, 2020.
\bibitem{wang2019meta}
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.
\newblock Meta-learning to detect rare objects.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 9925--9934, 2019.
\bibitem{xiao2020few}
Yang Xiao and Renaud Marlet.
\newblock Few-shot object detection and viewpoint estimation for objects in the
wild.
\newblock In {\em European Conference on Computer Vision}, pages 192--210.
Springer, 2020.
\bibitem{yan2019meta}
Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin.
\newblock Meta r-cnn: Towards general solver for instance-level low-shot
learning.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 9577--9586, 2019.
\bibitem{zhang2018metagan}
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song.
\newblock Metagan: An adversarial approach to few-shot learning.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
2365--2374, 2018.
\bibitem{zhao2018pseudo}
Xiangyun Zhao, Shuang Liang, and Yichen Wei.
\newblock Pseudo mask augmented object detection.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 4061--4070, 2018.
\bibitem{zhao2020object}
Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan
Chandraker, and Ying Wu.
\newblock Object detection with a unified label space from multiple datasets.
\newblock In {\em European Conference on Computer Vision}, pages 178--193.
Springer, 2020.
\bibitem{zhao2019recognizing}
Xiangyun Zhao, Yi Yang, Feng Zhou, Xiao Tan, Yuchen Yuan, Yingze Bao, and Ying
Wu.
\newblock Recognizing part attributes with insufficient data.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 350--360, 2019.
\bibitem{zhu2019zero}
Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama.
\newblock Zero shot detection.
\newblock {\em IEEE Transactions on Circuits and Systems for Video Technology},
2019.
\bibitem{zhu2020don}
Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama.
\newblock Don't even look once: Synthesizing features for zero-shot detection.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 11693--11702, 2020.
\end{thebibliography}