Novel Categories Discovery Via Constraints on Empirical Prediction Statistics
Extracted source text · arXiv 2307.03856
\documentclass{article}
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{spconf,amsmath,graphicx}
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{bbm}
\usepackage{comment}
\usepackage{multirow}
\usepackage{booktabs}
% ============================================
% Macro for reclaiming a lot of space
% --------------------------------------------
% START BLOCK
\usepackage[compact,center]{titlesec}
% \titlespacing{\section}{0pt}{2pt}{2pt}
\titleformat*{\section}{\centering\large\scshape\bfseries}
\titleformat*{\subsection}{\bfseries}
% \titleformat*{\subsubsection}{}
\titlelabel{\thetitle.\quad}
\AtBeginDocument{
\setlength\abovedisplayskip{0pt}
\setlength\belowdisplayskip{0pt}}
\setlength{\textfloatsep}{2pt plus 0pt minus 0pt}
\setlength{\floatsep}{2pt plus 0pt minus 0pt}
\setlength{\intextsep}{2pt plus 0pt minus 0pt}
\usepackage[margin=0pt, font=small]{caption}
\setlength{\belowcaptionskip}{1pt}
\captionsetup{belowskip=2pt,aboveskip=3pt}
\usepackage[ruled,vlined]{algorithm2e}
\definecolor{commentcolor}{RGB}{110,154,155} % define comment color
\newcommand{\PyComment}[1]{\ttfamily\textcolor{commentcolor}{\# #1}} % add a "#" before the input text "#1"
\newcommand{\PyCode}[1]{\ttfamily\textcolor{black}{#1}} % \ttfamily is the code font
%\usepackage{mathptmx}
\usepackage[activate={true,nocompatibility},final,tracking=false,kerning=true,spacing=true,factor=1100,stretch=10,shrink=20]{microtype}
\usepackage[moderate]{savetrees}
% END BLOCK
% % ============================================
\title{Novel Categories Discovery from probability matrix perspective}
%
% Single address.
% ---------------
% \name{, Heesung Kwon\textsuperscript{\$}, Hyungtae Lee\textsuperscript{\$}, Nirmalya Roy\textsuperscript{*}
\name{\begin{tabular}{c}Zahid Hasan\textsuperscript{*}, Abu Zaher Md Faridee\textsuperscript{+}, Masud Ahmed\textsuperscript{*}, Sanjay Purushotham\textsuperscript{*} \\Heesung Kwon\textsuperscript{\$}, Hyungtae Lee\textsuperscript{\$}, Nirmalya Roy\textsuperscript{*}\end{tabular}
\thanks{This research is supported by the NSF CAREER grant \#1750936, REU Site grant \#2050999 and U.S. Army grant \#W911NF2120076.}
}
% \address{Author Affiliation(s)}
% \name{Zahid Hasan\textsuperscript{*}, Masud Ahmed\textsuperscript{*}, Abu Zaher Md Faridee\textsuperscript{+}, Sanjay Purushotham\textsuperscript{*}, Heesung Kwon\textsuperscript{\$}, Hyungtae Lee\textsuperscript{\$}, Nirmalya Roy\textsuperscript{*}}
\address{
\textsuperscript{*}Department of Information Systems, University of Maryland, Baltimore County, USA\\
\textsuperscript{+}Amazon, USA;
\textsuperscript{\$}DEVCOM Army Research Laboratory (ARL), USA}
%
% For example:
% ------------
%\address{School\\
% Department\\
% Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
% {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
% {School A-B\\
% Department A-B\\
% Address A-B}
% {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
% while at ...}}
% {School C-D\\
% Department C-D\\
% Address C-D}
% Abu Zaher Md Faridee, Amazon USA <abufari@amazon.com>
\begin{document}
\maketitle
\begin{abstract}
% Typical NCD objectives aim to classify the available label space and separately cluster the unlabeled instances under disjoint label-space and equipartition assumptions. % In this research, we leverage the assumptions and
\textit{Novel Categories Discovery} (NCD) tackles the open-world problem of classifying known and clustering novel categories based on the class semantics using partial class space annotated data. % Research Gap & what we propose
Unlike traditional pseudo-label and retraining, we investigate NCD from the novel data probability matrix perspective.
We leverage the connection between NCD novel data sampling with provided novel class Multinoulli (categorical) distribution and hypothesize to implicitly achieve semantic-based novel data clustering by learning their class distribution.
We propose novel constraints on first-order (mean) and second-order (covariance) statistics of probability matrix features while applying instance-wise information constraints.
In particular, we align the neuron distribution (activation patterns) under a large batch of Monte-Carlo novel data sampling by matching their empirical features mean and covariance with the provided Multinoulli-distribution.
Simultaneously, we minimize entropy and enforce prediction consistency for each instance.
Our simple approach successfully realizes semantic-based novel data clustering provided the semantic similarity between label-unlabeled classes. We demonstrate the discriminative capacity of our approaches in image and video modalities. %two modalities (maybe more)
Moreover, we perform extensive ablation studies regarding data, networks, and our framework components to provide better insights.
Our approach maintains $\approx~94\%$, $\approx~93\%$, and $\approx~85\%$, classification accuracy in labeled data while achieving $\approx~90\%$, $\approx~84\%$ and $\approx~72\%$ clustering accuracy for novel categories for Cifar10, UCF101, and MPSC-ARL datasets that matches state-of-the-art approaches without any external clustering.
%[temperature control using the prior classes' knowledge]
%Neural statistics constraint: Bernoulli constraint and statistics matching, avoid collapse while learning semantics from the labeled data.
%Distribution matching [remove negative learning ], 0 activations for the novel classes.
%detailed empirical analysis of different loss function selection, and f-divergence loss functions.
%Swapped prediction: Consistency losses
%information-based augmentations.
%demonstrate the connection with prototype learning. and probability matrix is a special case of prototype learning.
%hyper-spherical prototype learning: with neuron statistics.
%Curricular learning (not forcing the cluster selection):
% Multiple networks together.
\end{abstract}
\begin{keywords}
NCD, Distribution Learning, Clustering.
\end{keywords}
\section{Introduction}
%=====================
% NCD background
\textit{Novel Categories Discovery} (NCD) offers a learning framework to develop robust machine-learning models that can exploit partially annotated/labeled datasets to detect the novel classes from unlabeled data in an open world setting~\cite{han2019learning, han2021autonovel, han2020automatically}.
NCD is a specialized case of \textit{Generalized Categories Discovery} (GCD) that relaxes the requirement of distinct labels for all underlying classes in the data with two specific assumptions: (i) a disjoint set condition between labeled and unlabeled classes, and (ii) prior knowledge of the number and distribution of novel classes. NCD aims to develop a data-driven model capable of understanding the general class formation conditions to simultaneously classify labeled classes and cluster novel unlabeled classes separately utilizing labeled and unlabeled novel data.
The popular NCD methods leverage the class notions and semantics from the labeled data and then cluster the unlabeled data based on those. Hence, the NCD approaches can be distinguished from others by the constraints and methodology of achieving this clustering. Existing NCD literature has proposed and demonstrated the clustering of novel data via pairwise similarity losses \cite{hsu2017learning}, pseudo-labeling \cite{jia2021joint}, prototype learning \cite{zhang2022automatically}, meta-learning, contrastive learning, kernel network, Sinkhorn-Knopp clustering \cite{fini2021unified}, variance regularization, and self-training \cite{asano2019self} using the novel class uniformity assumption.
In contrast to the previous approaches, we explore the non-contrastive self-supervised learning framework and its adaptation in the NCD framework to cluster novel data to a fixed number of groups with a prior distribution following the class semantics.
%Prototype-based learning: hyperspherical inference: Addressing the pseudo-label problem: EMA and KD
%motivation and background:
We observe the labeled data provides a deterministic target class for each labeled instance. Alternatively, for randomly sampled unlabeled (novel) data instances the probability of the target class follows a parametric Multinoulli (categorical) distribution with known novel class probabilities. We exploit this connection between the unlabeled data samples and their underlying Multinoulli distribution knowledge to develop a novel distribution learning-based NCD method utilizing the predicted probability matrix of Monte Carlo sampled unlabeled data instances. We rethink NCD from a probability matrix perspective and hypothesize achieving semantic-based novel data clustering by learning their class distribution while simultaneously learning semantics notions from labeled data in a supervised fashion.
In this research, we study a novel NCD clustering from the perspective of the probability matrix of the unlabeled data samples.
We propose to enforce learning novel data statistics by applying statistical constraints on the predicted probability matrix to achieve the implicit objective of novel data clustering.
In particular, we leverage the similarity between NCD and Multinoulli distribution and provide statistics-inspired instance-wise and feature-wise regularization to learn data class distribution.
Our approach is closely related to canonical correlation analysis and non-negative self-supervised methods as we impose constraints on class-representative neuron activation patterns for a probability matrix consisting of a large batch of Monte Carlo sampled instances to cluster class groups together. We pose our contributions as below,
%% will add in the next version.
%We demonstrate the extension towards prototype-based learning and offer general prototype-based methods to discover NCD. We propose contrastive prototype learning in NCD clustering applications.
%derive the probability matrix-based approach is a specialized case of prototype learning and propose generalized distance-based hyperspherical prototype NCD solutions.
(1) We demonstrate that simultaneous novel data Multinoulli distribution learning and labeled data classifying entail semantic-based novel data clustering. We propose a distribution learning-based novel NCD method that imposes constraints on each instance (entropy and consistency) and collective feature statistics (first order: mean and second order: covariance) of novel data probability matrices. Our instance-wise losses bound the network to reach a unique and invariant class decision for each instance. Meanwhile, the feature-wise loss oversees the collective neuron activation pattern (statistics) under diverse samples from multiple classes to align with the underlying class distribution.
(2) We experiment with image, video, and time series sensor data to evaluate our novel NCD approach. Further, we perform extensive ablation studies from (i) \textit{data} (labeled class set, augmentations, distribution), (ii) \textit{network architecture} (shallow, deep), and (iii) \textit{proposed learning framework} (individual losses, optimization) perspectives to provide insight regarding their individual and mutual impacts on learning robust NCD models. We open-source our codes and data \footnote{\href{https://github.com/mxahan/NEV-NCD}{Codes}} and experiment settings for further research.
\section{Related works}
%NCD works:
Typically NCD approaches follow either a two-stage or a single-stage joint optimization \cite{joseph2022spacing} to facilitate knowledge transfer from labeled to novel data.
The two-stage approaches can be grouped into learned-similarity-based and latent-space-based methods \cite{troisemaine2023novel}. The similarity-based NCD solution such as CCN \cite{hsu2017learning} and MCL \cite{hsu2019multi} seek to develop a binary classifier that identifies similarity/dissimilarity on unique pairs of instances.
The latent-space-based approaches such as DTC \cite{han2019learning}, DEC \cite{xie2016unsupervised}, and MM \cite{chi2021meta} aim to learn the latent representation of the high label features from the labeled data and apply a clustering algorithm to cluster novel data.
On the other hand, single-stage methods such as AutoNovel~\cite{han2021autonovel}, UNO~\cite{fini2021unified}, OpenMix~\cite{zhong2021openmix}, \cite{zhang2022automatically}, NEV-NCD \cite{hasan2023nev} enable NCD to utilize both the labeled and unlabeled data simultaneously with/without self-supervised pretraining. They jointly minimize a combination of supervised (labeled data) and unsupervised losses (unlabeled data). The unsupervised losses are the key to novel data clustering and often rely on clustering-based pseudo-labeling \cite{arazo2020pseudo} and data equipartitioning \cite{fini2021unified}.
Our approaches are closely related to and improve generalization upon NEV-NCD \cite{hasan2023nev} addressing the non-uniform class distributions and negative learning redundancy. Alternatively, we take inspiration from canonical relationship analysis \cite{hotelling1992relations} based on negative sample-free self-supervised learning \cite{balestriero2023cookbook} such as DINO \cite{caron2021emerging}, Barlow-twin \cite{zbontar2021barlow}, W-MSE \cite{ermolov2021whitening}, SwAV \cite{caron2020unsupervised}.
% Motivation from: cL from affinity matrix, and matrix formulation, canonical correlation analysis
% primary motivated by: NEV-NCD, Barlow-twin, vicreg, w-mse loss. address the NEV-NCD fixed issue, and generalize by matching the multinoulli distribution distribution directly. learning to cluster by learning the data class distribution, with the help of supervised counterparts.
% Closely related to NEV-NCD but the difference in generalization. broader picture., Barlow-twin.
% non-uniform cases
% NEV-NCD and its limitation.
% connection to DINO
\section{Learning from Probability Matrix}
\subsection{Notation and problem formulation}
% Vector as bold
% Scaler as normal notation
The dataset, $\mathcal{D} = \mathcal{D}^l \cup \mathcal{D}^u$, consists of labeled data $\mathcal{D}^l = \{(\mathbf{x}_i^l, y_{i})\}_{i = 1}^{N^l}$ where $y_{i} \in C_L=\{c_1, c_2, ..., c_L\}$ from $L = |C_L|$ unique ordinary categories and unlabeled data $\mathcal{D}^u = \{(\mathbf{x}_i^u)\}_{i = N^l+1}^{N^l + N^u}$ from $U = |C_U|$ latent novel classes $y_i \in C_U=\{c_{L+1}, c_{L+2}, ..., c_{L+U}\}$ with non-overlapping categories between labeled and novel classes $C_L \cap C_U = \emptyset$. The novel classes are from a known Multinoulli distribution parameterized by $\mathbf{p}_U = [p_{u, L+1}, ..., p_{u, L+U}]^T$, where the elements $p_{u,j}$ denote the probability corresponding to the $U$ unlabeled novel classes in $\mathcal{D}^u$.
We represent $K$-dimensional vector with all $0/1$ elements as $\mathbf{0}_K/\mathbf{1}_K \in \mathbb{R}^K$.
We define $\mathbf{y}$ as $L+U$ dimensional Multinoulli (categorical) variable with $\{y_j\in\{0,1\}^{L+U} \wedge \sum_{j = 1}^{L+U}y_{j} = 1\}$. The target label for the corresponding $i$-th sample is represented by one-hot encoding Multinoulli vector $\mathbf{y}_{i}$ with $y_{ji} = 1$ at $j=y_i$-th position.
We denote parameterized neural network encoder $f_\theta$, embedding representation layer $h_\theta$, and classification head $g_\theta$. The $g_\theta$ consists of $L+U$ neurons ($L$ labels and $U$ novels) with softmax activation to represent each of the $L+U$ classes and their corresponding probabilities.
The network predicted class probability distribution for a single input instance is denoted by $(L+U) \times 1$-dimensional vector $\hat{\mathbf{y}} = g_\theta(h_\theta(f_\theta(\mathbf{x}_i)))$, where each of $L+U$ elements (neurons) value represents one of the unique $L+U$ classes probability.
We form the probability matrix $\mathbf{P}_{\mathbf{p}} = [\mathbf{p}_1, \mathbf{p}_2, ..., \mathbf{p}_B] \in \mathbb{R}^{K \times B}$ for a batch of $B$ instances by placing $B$ vectors $\{\mathbf{p}_i \in \mathbb{R}^K, i = 1, 2, ..., B\}$ as column, where $i$-th column, $\mathbf{P}_{y, *i}$, represents the class probability for a $i$-th instance of batch $B$.
The NCD objectives aim to develop DL models to classify the $\mathcal{D}^l$ by comprehending the labeling semantics and clustering the $\mathcal{D}^u$ counterpart based on their underlying latent class distribution.
%A notation Table: Maybe
\subsection{Novel-data Clustering using Multinoulli distribution}
The parameterized discrete $K$-dimensional Multinoulli (Categorical) probability distribution \cite{murphy2012machine} is the generalization of the Bernoulli distribution for a categorical random variable and is a special case of Multinomial (sum of Multinoulli) distribution.
It describes the possible outcome of a random variable that takes on one of $K$ mutually exclusive categories with a prior probability (probabilities of potential outcomes of a single rolling of $K$-face dice) and is the most general distribution over a $K$-way event.
\textbf{Definition}: Let $\mathbf{Y}= [Y_1, ..., Y_K]^T$ be a discrete random vector with support $R_\mathbf{Y}=\{Y_j\in\{0,1\}^K : \sum_{j = 1}^{K}Y_{j} = 1\}$ i.e. only one entry equal to $1$ and others are $0$. For $K>0$ categories with strictly positive event probabilities $p_1, p_2, ..., p_K$ with $\sum_{i=1}^K p_i = 1$. We define $\mathbf{Y}$ with a Multinoulli distribution with parameter $\mathbf{p} = [p_1, ..., p_K]^T \in \mathbb{R}^K$ if its joint probability mass function is
\begin{equation}
Mu(\mathbf{Y}|1, \mathbf{p})= p_{\mathbf{Y}}({\mathbf{Y}})=
\begin{cases}
\Pi_{j=1}^K p_j^{Y_j} & \text{if } \mathbf{Y} \in R_{\mathbf{Y}}\\
0 & \text{otherwise}
\end{cases}
\label{eq:multi_def}
\end{equation}
The expected value of $\mathbf{Y}$ vector is $\mathbb{E}[\mathbf{Y}] =\mathbf{p}$ and the covariance matrix of $\mathbf{Y}$ is $\text{Var}(\mathbf{Y})= \Sigma: \Sigma \in \mathbb{R}^{K \times K}$ with the following element entry
\begin{equation}
\Sigma_{i,j}=
\begin{cases}
p_i(1-p_i) & \text{if }j=i\\
-p_ip_j & \text{if } j\neq i
\end{cases}
\end{equation}
%Formation of probability matrix. Known class cases: Fixed target and supervised losses. we know the corresponding target matrix and optimize instance-wise supervised objective.
We sample a batch of $B$ input data instances from the dataset using Monte Carlo sampling methods and column-wise stack their corresponding network output vectors $\{\mathbf{P}_{\hat{\mathbf{y}},*i} = \hat{\mathbf{y}}_i: i = 1,..., B\}$ that represents predicted probability distribution over $L+U$ classes for $i$-th instance to form probability matrix $\mathbf{P}_{\hat{\mathbf{y}}}\in \mathbb{R}^{K\times B}$ with $K = L+U$ in this context.
We will avoid using $\hat{\mathbf{y}}$ in the subscript to use $\mathbf{P}$ notation instead of $\mathbf{P}_{\hat{\mathbf{y}}}$ unless otherwise stated to avoid symbol overload in our subsequent discussion.
Under the assumption of one class per data instance, the objecting discriminative model aims to predict the correct class from the input instances by providing concentrated probability on the representative ground-truth class neuron.
We construct a probability matrix $\mathbf{P}^l$ for labeled data by sampling from $\mathcal{D}^l$. For $\mathbf{P}^l$, we have a ground truth matrix $\mathbf{P}_{\mathbf{y}}^l$ that provides an exact one-hot Multinoulli target variable $\mathbf{y}_i, i= \{1, ..., B\}$ for each column ($\mathbf{P}_{*i}^l$) and apply data instance-wise supervised objective such as cross-entropy loss to update the network parameters towards developing the accurate model.
However, the probability matrix $\mathbf{P}^u$ for novel data samples from $\mathcal{D}^u$ have a probabilistic target that sample belongs to one of the novel classes with certain probability rather than a specific target neuron $\mathbf{y}_i$, for the columns $\mathbf{P}_{*i}^u = g_\theta(h_\theta(f_\theta(\mathbf{x}_i))) | \mathbf{x}_i\sim \mathcal{D}^u$ containing predicted instance probabilities. Traditional approaches utilize pseudo-label generation methods to generate Multinoulli pseudo-label targets for the instances by clustering their low-dimensional embedding and re-train the network to match $\mathbf{P}_{*i}^u$ with the pseudo-target using a supervised objective.
Alternatively, We leverage the knowledge about associated uncertainty for an individual novel sample and expected collective behavior under a large batch of Monte Carlo sampling samples and model $\mathbf{P}^u$ from the probabilistic perspective. We rethink the ground truth $\mathbf{y}$ of each column $\mathbf{P}_{*i}^u$ as a probabilistic sample from a Multinoulli distribution parameterized by $\mathbf{p}_U$.
The empirical distribution (collective property) of labels for Monte Carlo sampling from $\mathcal{D}^u$ follows the underlying Multinoulli distribution defined by $\mathbf{p}_U$. The empirical distribution of ideal NCD discriminative model probability prediction $\mathbf{P}_{*i}^u$ over the novel classes for novel samples would also follow the same Multinoulli distribution, i.e., the mean and covariance would match the distribution parameters.
Alternative to pseudo-labeling and re-training, we hypothesize that by learning to match the novel class Multinoulli distribution defined by $\mathbf{p}_U$, the model would implicitly learn to cluster the unlabeled novel data based on similar semantics from the labeled data.
%emerging property!!
To achieve such learning objectives, we propose to apply statistical constraints (enforce collective feature behavior) and individual constraints on the $\mathbf{P}^u_{*i}$.
As statistical constraints, we minimize the distance (divergence) between the empirical distribution of the model class prediction for the large batch samples from $\mathcal{D}^u$, and the novel class distribution prior. In terms of $\mathbf{P}^u$ for a large batch, we condition that the empirical expectation, $\mathbb{E}[\mathbf{P}^u_{*i}]$, and covariance, $\Sigma^{\mathbf{P}^u} = Var(\mathbf{P}^u) = \mathbb{E}[(\mathbf{P}^u_{*i} - \mathbb{E}[\mathbf{P}^u_{*i}])(\mathbf{P}^u_{*i} - \mathbb{E}[\mathbf{P}^u_{*i}])^T]$ with $\Sigma^{\mathbf{P}^u} \in \mathbb{R}^{K \times K}$, of the model prediction ($\mathbf{P}^u_{*i}$) would align with the Multinoulli distribution defined by $\mathbf{p}_U$.
\begin{equation}
\mathbb{E}[\mathbf{P}_{*i}^u] =
\begin{bmatrix}
\mathbf{0}_L \\
\mathbf{p}_U
\end{bmatrix}
\label{eq:true_mean}
\end{equation}
\begin{equation}
\Sigma_{i,j}^{\mathbf{P}^u}=
\begin{cases}
0 & \text{if }i<L+1 \vee j<L+1\\
p_{i}(1-p_i) & \text{if }j=i \wedge i,j>L\\
-p_ip_j & \text{if } j\neq i \wedge i,j>L
\end{cases}
\label{eq:true_cov}
\end{equation}
Besides feature-wise optimizations (statistics of $\mathbf{P}_{*i}^u$) for the unlabeled data, we propose constraints on individual prediction $\mathbf{P}_{*i}^u$ and impose instance-wise regularization to maintain the data properties that each instance belongs to a single class (Multinoulli variable) and level-preserving augmentation would not change the predicted label. In terms of $\mathbf{P}^u$, we put constraints that the individual predicted probability of each column ($\mathbf{P}_{*i}^u$) maintain a Multinoulli variable property (concentrated probability in the latent class representative neuron as in equation \ref{eq:multi_def}) and consistency of prediction under label preserving data-augmentation.
% swap prediction: higher version of consistency learning
% prototype idea
% everything is from a probability matrix perspective.
% \subsection{Algorithm}
% alternative optimization.
% Provide PyTorch-style pseudo code.
\begin{comment}
\begin{algorithm}[h]
\SetAlgoLined
\PyComment{this is a comment} \\
\PyComment{this is a comment} \\
\PyComment{} \\
\PyComment{going to have indentation} \\
\PyCode{While Converge:} \\
\Indp % start indent
\PyCode{$\mathbf{x}_i \sim \mathcal{D}^l$; $i = 1,2,...,B$} \\
\PyCode{your code} \PyComment{inline comment} \\
\Indm % end indent, must end with this, else all the below text will be indented
\PyComment{this is a comment} \\
\PyCode{your code}
\caption{PyTorch-style pseudocode for your-algo}
\label{algo:your-algo}
\end{algorithm}
\end{comment}
\begin{comment}
\subsection{Connection to generalized prototype-based Learning}
in the classification layer, each neuron represents a class.
one neuron represents one class, and a vector represents one class.
The probability matrix is a specialized case of prototype learning. where each prototype is one hot encoding and orthogonal.
the issue with such orthogonal settings!
Instead, prototype matching. embedding. hyperspherical prototypes. k-prototypes and form a probability matrix based on the prototype similarity.
orthogonal prototypes: extend to generalized hyperspherical prototypes.
Hyperspherical embedding and uniform hyperspherical prototypes extension.
w2 normalization for the embedding vector.
\end{comment}
\section{Methodology}
%negative learning of log loss causes issues when the final layer weights are not l2 normalized!
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{figures/Overview_diagram.pdf}
\caption{Overview of our NCD approach based on conditions on probability matrix}
\label{fig:overview_diag}
\end{figure*}
% Novel categories from the probability matrix
% Novel categories from hyperspherical prototypes: prototype selection, dimension.
% Aim: Classification of the labeled data and Clustering of the novel data based on the semantics.
\subsection{NCD from Probability matrix}
%How to form both matrices.
We randomly sample a batch of $B$ data instances of labeled/unlabeled data samples $\mathbf{x}_i^{l/u}; i =\{1,2...., B\}$ from $\mathcal{D}^{l/u}$ and randomly augment the samples using augmentation operation from a set of label-preserving augmentation operators $\mathcal{A}$. The parameterized neural network model predicts the class distribution for input data samples over $K\;(=L+U)$ classes $\hat{\mathbf{y}}_i^{l/u}= g_\theta(h_\theta (f_\theta (\mathbf{x}_i^{l/u} | \mathcal{D}^{l/u}))): \hat{\mathbf{y}}_i^{l/u} \in \mathbb{R}^K$, where, the $L+U$ classification head neurons represent corresponding probability for $L+U$ classes in a bijection manner. We form probability matrix $\mathbf{P}^{l/u} \in \mathbb{R}^{K \times B}$ for $B$ labeled/novel data samples where the column of the matrix are defined as $\mathbf{P}_{*i}^{l/u} = \hat{\mathbf{y}}_i^{l/u}$ where $i =\{1,2, ..., B\}$.
\subsubsection{Learning from $\mathbf{P}^l$:}
The dataset provides corresponding exact ground truth class label $y_i \in C_L$ for each data instance $\mathbf{x}_i\in \mathcal{D}^u$.
We utilize the $y_i$ to create a one-hot encoding (Multinoulli variable) target vector $\mathbf{y}_i \in \mathbb{R}^{K}$ with element $y_{i, j} = \mathrm{1}_{[j = y_i]}, j =\{1, ..., K\}, y_j \in C_L$ for the $i$-th column of $\mathbf{P}^l$, where, $\mathrm{1}_{[j = y_i]}$ refers identity operators. We construct the target matrix $\mathbf{P}_{\mathbf{y}}^l$ for predicted probability matrix $\mathbf{P}^l$ with $\mathbf{P}_{\mathbf{y}, *i}^{l} = \mathbf{y}_i$ that is the $i$-th column contains the one-hot encoding target for the $i$-th sample of the corresponding $\mathbf{P}^l$. We minimize traditional instance-wise supervised loss objective for classification i.e. categorical cross-entropy loss as defined in \ref{eq:ce_sup} to align $\mathbf{P}^l$ with corresponding $\mathbf{P}^l_{\mathbf{y}}$ by learning the class semantics from $\mathcal{D}^l$.
\begin{align}
\mathcal{L}_{ce} = - \underset{(\mathbf{x}_i, \mathbf{y}_i) \in \mathcal{D}^l}{\mathbb{E}} [\frac{1}{B}\sum_{i = 1}^B{\mathbf{P}_{\mathbf{y},*i}^l}^T \log{\mathbf{P}_{*i}^l} ] \\
= - \underset{(\mathbf{x}_i, \mathbf{y}_i) \in \mathcal{D}^l}{\mathbb{E}} [\frac{1}{B}\textbf{tr} ({\mathbf{P}_{\mathbf{y}}^l}^T \log \mathbf{P}^l)]
\label{eq:ce_sup}
\end{align}
\subsubsection{Learning from $\mathbf{P}^u$}
The ground truth for random samples $\mathbf{x}^u$ drawn from $\mathcal{D}^u$ is a random variable that follows a distribution of $\mathbf{p}_{U}$ over the novel classes instead of having an exact Multinoulli variable. In particular, the probabilistic target $\mathbf{y}_i|\mathbf{x}^u_i$ for network prediction $\mathbf{P}^u_{*i} $ is a sample from $K$-dimensional Multinoulli distribution parameterized by $\mathbf{y}^U = [\mathbf{0}_L^T \; \mathbf{p}_U^T]^T$ over labeled and novel classes ($\mathbf{0}_L^T$ represents $0$ probability over the $L$ labeled classes due to disjoint classes) i.e. $\mathbf{y}|\mathbf{x}^u \sim Mu(\mathbf{Y}|1, [\mathbf{0}_L^T \; \mathbf{p}_U^T]^T)$.
We train the parameterized neural network to learn the underlying Multinoulli distribution over novel data samples to implicitly cluster unlabeled data separately based on the semantics of $\mathcal{D}^u$ by applying both instance-wise and feature-wise constraints for the predicted $\mathbf{P}^u$.
\textbf{Instance-wise Constraints:}
Our instance-wise serves two objectives. Firstly, we enforce the Multinoulli variable property of Dirac-delta distribution for each prediction, i.e., $\mathbf{P}_{*i}^u \in R_\mathbf{Y}$ by applying the entropy minimization objective \cite{grandvalet2004semi}.
\begin{equation}
\mathcal{L}_{H} = - \underset{\textbf{x}_i \in \mathcal{D}^u}{\mathbb{E}}[\frac{1}{B}\textbf{tr} ({\mathbf{P}^u}^T\log{\mathbf{P}^u})]
\label{eq:ent_reg}
\end{equation}
% \begin{equation}
% \mathcal{L}_{H} = - \underset{\textbf{x}_i \in \mathcal{D}^u}{\mathbb{E}}[\frac{1}{B}\sum_{i = 1}^B {\mathbf{P}_{*i}^{u}}^T\log{\mathbf{P}_{*i}^u}]
% \label{eq:ent_reg}
% \end{equation}
% alternative notion by pointwise multiplication.
Secondly, we enforce prediction consistency \cite{yang2022divide} for unlabeled data under different label-preserving data augmentations.
We form two prediction matrices $\mathbf{P}^u$ and $\mathbf{P}^{u\prime}$ where their columns $\mathbf{P}_{*i}^u$ and ${\mathbf{P}^{u}_{*i}}^\prime$ represents the network prediction for two differently augmented versions of the same data instances.
We minimize the Frobenius norm of the difference matrix between $\mathbf{P}^u$ and $\mathbf{P}^{u\prime}$. The matrix Frobenius norm is defined by $||\mathbf{A}||_F = \sqrt{\mathbf{tr}(\mathbf{A}^T\mathbf{A})}$.
%(two different augmentations) Consistency for the augmentations.
%swapped prediction: Consistency losses.
\begin{equation}
\mathcal{L}_{mse} = \underset{\mathbf{x}_i \in \mathcal{D}^{u/l}}{\mathbb{E}} [\frac{1}{B}||\mathbf{P}^{u} - {\mathbf{P}^{u}}^\prime ||_F]
\label{eq:mse}
\end{equation}
\textbf{Feature-wise Constraints}
The feature-wise constraints enforce the statistical behaviors (collective property) of the predicted probability features of $\mathbf{P}^u$ columns to follow the prior Multinoulli distribution with $\mathbf{y}^U = [\mathbf{0}_L^T \; \mathbf{p}_U^T]^T$. In particular, we aim to align the empirical mean and covariance of the predictions with the objective Multinoulli distribution's mean and covariance.
The sample mean of a Multinoulli variable represents the probability distribution over the classes. We apply the distribution minimizing $f$-divergence ($KL$-divergence) objective as in equation \ref{eq:f-div-loss-mean} to align the true mean (distribution) defined in equation \ref{eq:true_mean} and empirical mean (distribution) calculated using equation \ref{eq:emp_mean}.
To avoid the trivial solution of $\mathbf{P}^u$ by predicting fixed mean values and emerging clustering behavior, we enforce the empirical feature covariance matrix estimated using \ref{eq:cov-matrix}. Here, we compute the difference matrix between estimation covariance and expected covariance of equation \ref{eq:true_cov} and minimize the matrix Frobenius norm as in equation \ref{eq:cov-loss}.
\begin{equation}
\hat{\mathbb{E}}[\hat{\mathbf{y}}] = \hat{\mathbb{E}}[\mathbf{P}^u_{*i}]= \frac{1}{B}\sum_{i = 1}^B\mathbf{P}_{*i}^u= \frac{1}{B}\mathbf{P}^u\mathbf{1}_B
\label{eq:emp_mean}
\end{equation}
\begin{equation}
\mathcal{L}_{kl-div} = \underset{\mathbf{x}_i \in \mathcal{D}^u}{\mathbb{E}} {\mathbf{y}^U}^T[\log{\mathbf{y}^U} - \log{(\frac{1}{B}\mathbf{P}^u\mathbf{1}_B]}
\label{eq:f-div-loss-mean}
\end{equation}
\begin{align}
\text{Zero centered } \mathbf{P}^u = \{\mathbf{\Bar{P}}^u: {\mathbf{\Bar{P}}^u_{*i}} = {\mathbf{P}^u_{*i} - \hat{\mathbb{E}}[\hat{\mathbf{y}}]}\} \\
\hat{\Sigma}^{\mathbf{P}^u} = \frac{1}{B} \mathbf{\Bar{P}}^{u}\Bar{\mathbf{P}}^{uT} \;\text{ where } \hat{\Sigma}^{\mathbf{P}^u} \in \mathbb{R}^{K \times K}
\label{eq:cov-matrix}
\end{align}
\begin{equation}
\mathcal{L}_{var} = || \hat{\Sigma}^{\mathbf{P}^u} - \Sigma^{\mathbf{P}^u}||_F
\label{eq:cov-loss}
\end{equation}
\textbf{Joint Optimization:}
We jointly optimize the learning objective \ref{eq:joint_op} of a weighted ($\lambda$) sum of the discussed losses alternatively by utilizing $\mathbf{P}^l$ and $\mathbf{P}^u$.
%Covariance constraint over the representation embedding???
\begin{equation}
\mathcal{L}=
\begin{cases}
\lambda_{ce}\mathcal{L}_{ce} & \text{if } \mathbf{x} \in \mathcal{D}^l\\ \lambda_u\mathcal{L}_{u} & \text{if } \mathbf{x} \in \mathcal{D}^u
\end{cases}
\label{eq:joint_op1}
\end{equation}
Where, $\mathcal{L}_{u}$ represents loss over $\mathbf{P}^u$
\begin{equation}
\mathcal{L}_{u}= \lambda_{H}\mathcal{L}_{H} + \lambda_{m}\mathcal{L}_{mse} +
\lambda_{kl}\mathcal{L}_{kl-div} + \lambda_{v}\mathcal{L}_{var}
\label{eq:joint_op}
\end{equation}
\begin{comment}
\subsection{Learning from embedding and prototypes}
\textit{Category Discrimination:}
We utilize the known categories of the $\mathcal{D}^l$ and disjoint criterion to perform categorical contrastive learning. Here, we sample query and positive instances from $c_j \in C_L$ and take negative instances distributed from remaining classes and unlabeled data $C_L \slash \{c_j\} \cup C_U$ to minimize the contrastive objective of \ref{eq:l_info} where $s_{i,j} = cosine(z_i, z_j)$, $\tau$ denotes cosine similarity between embedding and temperature.
%Two types of data samples.
\begin{equation}
\mathcal{L}_{cl} = \underset{c_l \sim C_L}{\mathbb{E}}\underset{x_{k,n}| c_k\neq c_l}{\underset{x_i, x_j| c_l}{\mathbb{E}}}-\log\frac{e^{(s_{i,j}/\tau)}}{e^{(s_{i,j}/\tau)} + \sum_{n =0}^N e^{(s_{i,kn}/\tau)}}
\label{eq:l_info}
\end{equation}
\textit{Instance Discrimination:} Although there lacks information about the intra-class similarity between unlabeled instances, we have their complementary labels. For the query of unlabeled instances, we sample negatives from $\mathcal{D}^l$ to avoid sampling false negative keys and minimize \ref{eq:u_info}. We also enforce consistency \cite{yang2022divide} between the representation of the instance $x_i$ and their augmented version $x_i^\prime$ by equation \ref{eq:mse}. Here, we rely on variance regularization \ref{var_reg} instead of contrastive learning to discriminate between unlabeled latent classes.
\begin{equation}
\mathcal{L}_{cl} =\underset{x_{k,n}| c_l\in C_L}{\underset{x_i \in \mathcal{D}^u}{\mathbb{E}}}-\log\frac{e^{(s_{i,i^\prime}/\tau)}}{e^{(s_{i,i^\prime}/\tau)} + \sum_{n =0}^N e^{(s_{i,kn}/\tau)}}
\label{eq:u_info}
\end{equation}
probability matrix as orthogonal prototypes.
orthogonal one-hot encoding is a class prototype.
Prototype-based loss function,
enforce the embedding to match some predefined class prototypes in the low-dimensional hypersphere.
contrastive learning losses, and entropy minimization, distribution matching.
sk-clustering and retraining.
Our proposed NEV-NCD method falls under the single-stage approach requiring the availability of both $\mathcal{D}^l$ and $\mathcal{D}^u$ concurrently. NEV-NCD jointly optimizes both the supervised and clustering objectives (Figure \ref{fig:overview_diag}) as discussed in the following.
Learning using probability matrix.
Extension towards hyperspherical prototype learning.
Embedding, Prototypes.(hierarchical classes: super-classes and sub-classes)
constructive learning, prototypes. over-clustering.
quick adaptation to special InfoNCE contrastive learning.
%\subsection{Clustering Losses}
\textbf{Contrastive and Consistency Loss:}
We adopt the SupCon \cite{khosla2020supervised} framework with NCD to enhance the representation quality by modifying the data sampling using the underlying data conditions.
We optimize probabilistic InfoNCE contrastive objectives to learn both categorical and instance discrimination in the representation layer output denoted by $z_i = h_\theta(f_\theta(x_i))$.
%Two contrastive learning if the query samples are from L (distributed negatives)and query samples are from U. Put external Consistency loss
%both supervised and data assumption. SupCon
%Distributed negatives instance segmentation with mixed negatives When instances are from negative the positives are from augmentation and the negatives are from the positive classes.
\textbf{Variance Regularization:} \label{var_reg} Uniform random sampling from a balanced data of $k$ classes is equivalent to roll a fair $k-$face dice with probabilities, $P = \frac{1}{k}$, of each class. We can define a binary random variable, $X_j \in \{0,1\}$, representing the events of $j-$th face, $j \in {1, 2, .., k}$, outcome success and failure with probabilities, $P(X_j =1) = \frac{1}{k}$, mean, $\mathbb{E}(X_j) = \frac{1}{k}$, and variance $\frac{k-1}{k^2}$.
We explicitly apply this constraint as regularization \ref{eq:var_reg} for the empirical distribution of the sharpened (sharpening hyper-parameter $Sr$) unlabeled prediction head $\Tilde{y}_i = \sigma_k(\hat{y}_i/Sr)$ for relatively large batch size.
%self-labeling and training,
%Enforcing cluster among the classification head
%Sharpening and training.
% VicReg idea, the clustering for the output.\cite{bardes2021vicreg}
%Variance of sharpening version of the prediction
%output probability tuple for each of the output node = $(0, 1) = (\frac{k-1}{k}, \frac{1}{k})$
% \begin{align}
% \mathbb{E}(node) = \frac{1}{k}\\
% Obj_{var} = \mathbb{E}[x - \mathbb{E}(x)]^2 \\
% Obj_{var} = \frac{k-1}{k^2}
% \end{align}
\begin{equation}
\mathcal{L}_{var} = \underset{x_i \in \mathcal{D}^u}{\mathbb{E}} |var(\Tilde{y}_{i}) - \frac{k-1}{k^2}|_2^2
\label{eq:var_reg}
\end{equation}
\end{comment}
\section{Experiments}
We demonstrate the applicability of our NCD framework by experimenting with both image and video modalities and performing extensive ablation studies.
\subsection{Dataset}
\textbf{CIFAR10}\cite{krizhevsky2009learning} dataset contains $32 \times 32 \times 3$ images of $10$ classes (\textit{plane, car,} bird, cat, deer, dog, frog, horse, \textit{ship, truck}) with $5000$ training and $1000$ testing images per class. Besides performing ablation studies on CIFAR10, we follow the label/unlabeled class setting of \cite{zhang2022automatically} to compare with the previous approaches.
% \textbf{UCF101 \cite{soomro2012ucf101}:} We perform multiple experiments with the UCF101 dataset with randomly selected $90$ labeled actions and the remaining $11$ as unlabeled novel actions.
\textbf{In-house dataset:} We perform numerous controlled experiments to study the parameters effects, data quality, and action selection using our in-house dataset. This dataset consists of three synchronous views from three realistic camera angular positions with varying camera settings. It contains class-balanced ten regular micro-actions with static (sitting, standing, lying with face up and down) and dynamic patterns (walking, push up, waving hand, leg exercise, object carrying, and pick/drop) from $11$ volunteers under varying realistic environments.
\textbf{UCF101 \cite{soomro2012ucf101}:} We perform multiple experiments with the UCF101 dataset with randomly selected $90$ labeled actions and the remaining $11$ as unlabeled novel actions.
\textbf{In-house dataset:} We perform numerous controlled experiments to study the parameters effects, data quality, and action selection using our in-house dataset. This dataset consists of three synchronous views from three realistic camera angular positions with varying camera settings. It contains class-balanced ten regular micro-actions with static (sitting, standing, lying with face up and down) and dynamic patterns (walking, push up, waving hand, leg exercise, object carrying, and pick/drop) from $11$ volunteers under varying realistic environments.
\begin{table}[]
\centering
\footnotesize
\caption{Labeled/Unlabeled splits for NCD experiments}
\label{table:data_split}
\begin{tabular}{@{}ccc@{}}
\toprule
Dataset & \# Labeled & \#Unlabeled \\ \midrule
Cifar10 & 4/5/6 & 6/5/4 \\
UCF101 & 90 & 11 \\
Mpsc-ARL & 4/5/6 & 6/5/4 \\ \bottomrule
\end{tabular}
\end{table}
%audio
%sensor data
\subsection{Implementation Details}
\textbf{Network Architectures:} We experimented with three different CNN architectures for the experimented data modalities that have demonstrated their capabilities in the respective domains. We have experimented with ResNet architectures \cite{he2016deep} (ResNet18, ResNet34, ResNet50) and MobilenetV2 \cite{sandler2018mobilenetv2} as image classifiers and SlowFast \cite{feichtenhofer2019slowfast}, R2+1D \cite{tran2015learning}, and P3D \cite{qiu2017learning} as video action classifiers with different weight initialization.
\textbf{Optimization}
We applied fixed weighting and adaptive weights \cite{liu2022residual} for labeled data and unlabeled novel data corresponding loss components.
To apply loss weight scheduling with the epoch number, $n_{ep}$, during the learning phase as shown in equation \ref{eq:var_weight} and \ref{eq:ce_weight} to implement adaptive weights. In our experiments, we fixed entropy loss weights ($\lambda_{H} = 1.0$) and other hyper-parameters ($\tau = 0.05$, $Sr = 0.1$).
To calculate the empirical Multinoulli distribution expectation and covariance, we traded off between computational cost and the law of large numbers and applied the batch size with a minimum of $10 \times U$ instances (batch size of $256$, $128$, and $64$ for Cifar10, UCF101, and MPSC-ARL datasets).
Here, we concatenate the $\mathbf{P}^u$ and $\mathbf{P}^{u\prime}$ to calculate the class activation order statistics (feature means and covariance matrix) performed best instead of considering a single matrix.
We simultaneously optimize our loss objectives using the Pytorch stochastic gradient descent (SGD) optimizer with a learning rate scheduler.
%Loss Weights: PyTorch
%Temperature to balance between losses and samples. (will discuss on a later iteration)
\begin{equation}
\lambda_{kl} = \lambda_{var} = 0.2 +0.5 \times n_{ep}
\label{eq:var_weight}
\end{equation}
\begin{equation}
\lambda_{ce} = \max (0, 1-0.01 \times n_{ep}) + 0.5
\label{eq:ce_weight}
\end{equation}
\subsection{Baseline Comparison and Ablation Studies}
The NCD baselines are tricky to compare because of the variation in settings and objectives due to different backbone architectures, self-supervised pre-training, label-unlabeled class selection, data assumptions, data augmentations, transfer learning, training complexity, and training schemes.
We primarily compare our methods with pre-training free methods with convolutional backbone architectures and demonstrate our method's effectiveness in NCD with a simplistic training scheme.
We compare our performance with the naive baseline of labeled data supervision, single-stage state-of-the-art NCD method Sinkhorn-Knopp clustering \cite{asano2019self} based pseudo-labeling UNO~\cite{fini2021unified} (both image and video setting) and two earlier two-stages benchmark methods KCL \cite{hsu2017learning} and MCL \cite{hsu2019multi} on identical Cifar10 setting.
%\textbf{BaseLine1:} Fully supervised setting with partial classes \textbf{BaseLine2:} CL pretraining with \textbf{Improved BaseLine2 (approach 1)}: SupCon with some negatives from the data.
% memory bank with variance loss
We ablate over different design choices of NCD settings. We utilize heuristics data knowledge to design and evaluate numerous experiments toward a better understanding of our proposed method.
Our ablation experiments can be broadly categorized into three main groups.
Firstly, from the \textbf{Dataset Perspective}, we perform three sets of experiments: (i) Augmentation $\mathcal{A}$: We vary the numbers (different types) and strength of data augmentations across multiple experiments. (ii) Label/Unlabeled classes $C_L/C_U$: We consider different numbers and classes of labeled and unlabeled class sets by randomly selecting classes. We have hand-picked the class groups with varying notions of semantics to observe the impacts of labeled sets. In this setting, we considered animal classes as labeled data (bird, cat, deer, dog, frog, horse) and attempted to discover machine classes \textit{plane, car, ship, truck}) for the Cifar10 dataset. In the MPSC-ARL dataset, we considered static classes for labeled and dynamic classes for unlabeled classes and vice-versa.
(iii) Distribution $\mathbf{p}_U$: We varied the unlabeled class distributions beyond the uniform assumption to evaluate the distribution learning performance.
%Augmentation, class selections (animal and machine classes), (dynamic and steady classes), Random class selection, number of classes and class selection
% dataset and their setting: for comparison, and further experiments e.g. distribution variation, class selection variation, weight loss sweeping.
Secondly, from the \textbf{network architecture} perspective, we analyze the NCD performances concerning the different convolutional architecture-based function classes with varying strengths, as discussed above.
Finally, from the \textbf{framework} perspective, we investigate the importance of each component of our proposed method and the framework's robustness by varying the hyperparameters, e.g., loss weights, optimization approaches, and batch size.
%adversarial/disconnected class supervision.
%parameter sweeping and control of the separation.
%Types of Labeled data notions Multi-view conditions of the data: We experiment with different types of labeled class and unlabeled class combinations as ablation studies. Multi-view condition on the data. \textbf{Working with Views only for some activities}
%Data augmentation.
%\textbf{Omitted Experiments:}
% Image dataset.
% time-series: opportunity dataset
% learning to cluster by learning data distribution.
%Extension to prototype, contrastive learning [memory bank]
% f-divergence: later
%novel contrastive loss for NCD: Prototypes motivations.
%Data-driven curriculum learning.
\section{Results}
We evaluated our distribution learning methods in left-out latest labeled and novel instances by both quantitative and qualitative metrics. To scrutinize the quality (i.e., separability between classes representation) of the embeddings, we utilize low-dimensional t-SNE visualization.
We quantified the performance over the labeled and unlabeled data by deploying standard accuracy (top-1) and average clustering accuracy (ACC) (equation \ref{eq:acc_mat}), respectively. We report the average accuracy and ACC over the three runs from different initialization.
\begin{equation}
ACC = \underset{p \in Perm}{\max} \frac{1}{N} \sum_{i = 1}^N \mathbbm{1} \{y_i = p({\hat{y}_i})\}
\label{eq:acc_mat}
\end{equation}
\noindent Here $ y_i \in C_U $ and $\hat{y}_i$ denote the label and predicted label of $\mathbf{x}_i \in \mathcal{D}^u$. $Perm$ set contains all permutations for labels of $\forall \mathbf{x}_i$ computed efficiently using \cite{kuhn1955hungarian}.
\subsection{Main Results}
We perform quantitative analysis for labeled and unlabeled novel instances over the left-out test data of the image (Cifar10) and video (UCF101 and MPSC-ARL). Our distribution learning-based approach successfully enables the respective model to learn semantic-based clustering for novel categories of image and video while $\mathcal{D}^u$ (high ACC) while classifying the $\mathcal{D}^l$ (high accuracy). We report our model performance for image data in Table ~\ref{table:cifar10_main_res} and video data in Table ~\ref{table:result_ucf101}. In both cases, our simplistic method performs on par with state-of-the-art NCD approaches that significantly outperform the traditional close-set learning settings in semantic-based novel pattern discovery.
\begin{table}[!htb]
\centering
\footnotesize
\caption{Comparative performance for classification of labeled and clustering novel categories on CIFAR10 test dataset with our approach (ResNet50 network) with 5/5 labeled/unlabeled setting.}
\label{table:cifar10_main_res}
\begin{tabular}{@{}ccc@{}}
\toprule
Methods & $\mathcal{D}^l$ (Accu \%) & $\mathcal{D}^l$ (ACC \%) \\ \midrule
KCL & 79.4 $\pm$ 0.6 & 60.1 $\pm$ 0.6 \\
MCL & 81.4 $\pm$ 0.4 & 64.8 $\pm$ 0.4 \\
DTC & 58.7 $\pm$ 0.6 & 78.6 $\pm$ 0.2 \\
AutoNovel & 90.6 $\pm$ 0.2 & 88.8 $\pm$ 0.2 \\
Ours & 90.2 $\pm$ 0.5 & 85.5 $\pm$ 0.6 \\ \bottomrule
\end{tabular}
\end{table}
\begin{table}[!htb]
\centering
\footnotesize
\caption{Quantative Result and comparative analysis on UCF101 Dataset (90/11) and MPSC-ARL dataset (5/5) with R2+1D networks.}
\label{table:result_ucf101}
\begin{tabular}{@{}ccccccc@{}}
\toprule
& Methods & Ours & NEV-NCD & UNO & SpCn & Supervise \\ \midrule
\multirow{2}{*}{UCF101} & Accc. \% & 92.3 & 92.4 & 92.1 & 93.2 & 92.6 \\
& ACC \% & 86.5 & 82.7 & 79.8 & 24.1 & 16.4 \\ \midrule
\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}MPSC-\\ ARL\end{tabular}} & Accc. \% & 82.3 & 82.9 & 83.4 & 85.2 & 84.6 \\
& ACC \% & 70.2 & 67.3 & 62.1 & 25.8 & 26.3 \\ \bottomrule
\end{tabular}
\end{table}
We also report a sample confusion matrix at Figure~\ref{fig:ts_cf_cifar} (a). it demonstrates that the model assigns one-to-one mapping between class representative neurons to novel classes that avoid class collapse for the unlabeled novel data.
We further analyze the embedding representation by the trained model and observe that model successfully learns to distinguish and cluster the novel classes from labeled class clusters in the embedding space as shown in Figure~\ref{fig:ts_cf_cifar} (b).
\begin{figure*}[!htb]
\centering
\includegraphics[width = \linewidth]{figures/tsne_plot.pdf}
\caption{(a) Confusion matrix for Cifar10 where each neuron probability represents one of the novel classes. (b) Model embedding separately clusters the label and novel data groups. The ResNet50 model trained on 5/5 split cifar10 data.}
\label{fig:ts_cf_cifar}
\end{figure*}
\subsection{Ablation Study}
Here, we elaborately discuss the dataset properties in discovering novel classes by discussing the results of the ablation experiments.
\textbf{Data Perspective:}
Data augmentation provides the missing inductive bias to identify class-specific features making the model invariant to label-preserving noises like rotation, partial occlusion, color jitter, histogram equalization, and frame rate by providing multiple views of data points.
In an NCD setting, learning consistency across multiple augmented versions of the same instance enforces the network to understand the class representative semantics information and prevents learning random data properties.
The NCD model suffers from performance drops in the image and video modality under the absence of strong augmentation schemes as reported in Table \ref{table:data_ablate_cifar} and \ref{table:data_ablate_mpsc-arl} as they provide a vast range of missing inductive biases compared to weaker augmentations.
The nature of labeled class semantics and the relative number of label/unlabeled classes play a vital role in clustering novel data shown in Table \ref{table:data_ablate_cifar} and \ref{table:data_ablate_mpsc-arl}.
When the class semantic varies across labeled and novel data (animal/machine separation in Cifar10 and static/dynamic separation in MPSC-ARL dataset) the NCD performance drops as the labeled class fails to provide the accurate class notion for unlabeled novel categories.
Further, NCD clustering performance degrades as the proportional number of unlabeled novel classes increases in general.
Finally, our proposed method performs robust clustering of novel data clustering performance and tackles various class distributions beyond uniform class distribution. However, the model performance degrades under extremely skewed tail class distribution due to overfitting issues as the network biases towards the head novel classes.
\begin{table}[!htb]
\centering
\footnotesize
\caption{Ablation studies on Cifar10 test dataset with our approach ResNet18 setting: Dataset Perspective}
\label{table:data_ablate_cifar}
\begin{tabular}{@{}cccc@{}}
\toprule
& Description & $\mathcal{D}^l$ (Accu \%) & $\mathcal{D}^l$ (ACC \%) \\ \midrule
\multirow{2}{*}{$\mathcal{A}$} & Strong & 85.2 & 81.82 \\
& Weak & 84.6 & 72.1 \\ \midrule
\multirow{2}{*}{\# L/U Classes} & 6/4 & 85.1 & 84.4 \\
& 4/6 & 86.5 & 76.2 \\ \midrule
\multirow{2}{*}{\begin{tabular}[c]{@{}l@{}} L/U classes\end{tabular}} & \begin{tabular}[c]{@{}l@{}}Animal/machine\end{tabular} & 86.2 & 62.5 \\
& \begin{tabular}[c]{@{}l@{}}Machine/Animal\end{tabular} & 85.7 & 60.3 \\ \midrule
\multirow{3}{*}{Distribution $\mathbf{p}_U$} & $[1/3, \{1/6\}^4]$ & 85.5 & 81.4 \\
& $[3/7, \{1/7\}^4] $ & 85.4 & 80.7 \\
& $[1/2, \{1/8\}^4] $ & 85.7 & 75.1 \\ \bottomrule
\end{tabular}
\end{table}
\begin{table}[!htb]
\centering
\footnotesize
\caption{Ablation studies on MPSC-ARL test person dataset with our approach R2+1D setting: Dataset Perspective}
\label{table:data_ablate_mpsc-arl}
\begin{tabular}{@{}cccc@{}}
\toprule
& Description & $\mathcal{D}^l$ (Accu \%) & $\mathcal{D}^l$ (ACC \%) \\ \midrule
\multirow{2}{*}{$\mathcal{A}$} & Strong & 82.3 & 70.2 \\
& Weak & 80.9 & 60.5 \\ \midrule
\multirow{2}{*}{\# L/U Classes} & 6/4 & 82.1 & 72.3 \\
& 4/6 & 82.5 & 65.2 \\ \midrule
\multirow{2}{*}{L/U Classes} & Dynamic/Static & 82.1 & 65.1 \\
& Static/Dynamic & 83.5 & 58.6 \\ \midrule
\multirow{3}{*} {Distribution $\mathbf{p}_U $} & $[1/3, \{1/6\}^4]$ & 82.1 & 67.6 \\
& $[3/7, \{1/7\}^4] $ & 82.4 & 65.8 \\
& $[1/2, \{1/8\}^4 ] $ & 82.4 & 62.4 \\ \bottomrule
\end{tabular}
\end{table}
% Please add the following required packages to your document preamble:
% \usepackage{booktabs}
\textbf{Model Perspective}
We observe that the deeper CNN model improves the classification and clustering accuracy simultaneously in both labeled and unlabeled data \ref{table:model_ablate}.
However, after certain strengths compared to the considered dataset, CNN-based image and video recognition models' performance reaches a plateau.
\begin{table}[!htb]
\centering
\footnotesize
\caption{Ablation studies for classification of labeled and clustering novel categories on test dataset with our approach with 5/5 labeled/unlabeled setting: Model Variation}
\label{table:model_ablate}
\begin{tabular}{@{}cccc@{}}
\toprule
Dataset & Network & $\mathcal{D}^l$ (Accu \%) & $\mathcal{D}^l$ (ACC \%) \\ \midrule
\multirow{4}{*}{Cifar10} & ResNet50 & 90.2 $\pm$ 0.5 & 85.5 $\pm$ 0.6 \\
& ResNet34 & 87.51 & 81.93 \\
& ResNet18 & 85.20 & 81.82 \\
& MobileNetV2 & 84.77 & 80.14 \\ \midrule
\multirow{2}{*}{UCF101} & SlowFast & 92.8 & 88.7 \\
& R2 +1D & 92.2 & 86.5 \\
\midrule
\multirow{3}{*}{\begin{tabular}[c]{@{}l@{}}MPSC-\\ ARL\end{tabular}} & SlowFast & 85.2 & 72.3 \\
& R2+1D & 82.3 & 70.2 \\
& P3D & 74.2 & 61.2 \\ \bottomrule
\end{tabular}
\end{table}
\textbf{Framework Perspective:}
In both modalities, we observe similar effects and impacts of the individual loss objects as reported in Table~\ref{table:framework_ablate_cm}. The NCD performance relies on the cross entropy loss $\mathcal{L}_{ce}$ to learn the semantic notion of class formation and achieve clustering based on random features with their presence. The $\mathcal{L}_{mse}$ loss ensures consistency by incorporating key-semantic information in clustering. Presence of $\mathcal{L}_{mse}$ prevents the model to cluster the data based on the noise properties. It also helps the model to identify and differentiate key-semantic features across multiple augmented versions.
Our distribution learning-based approach avoids trivial solutions by balancing between two extreme collapses: the model output predicts the $\mathbf{p}_U$ along all the novel class representative neurons or predicts one fixed class regardless of the input.
The first-order statistics-based distribution learning loss $\mathcal{L}_{kl}$ prevents one class prediction to dominate but encourages the collapse to the fixed Multnioulli distribution probability prediction, meanwhile the information constraint entropy loss $\mathcal{L}_{H}$ has the opposite effect that collapses all novel classes into a single cluster.
Finally, the $\mathcal{L}_{kl}$ and $\mathcal{L}_{var}$ complement each other to some extent. However, their simultaneous presence provides the best distribution and cluster learning performance across datasets.
\begin{table}[]
\centering
\footnotesize
\caption{Ablation studies on Cifar10 and MPSC-ARL test dataset with our approach ResNet18 setting: Loss function Perspective}
\label{table:framework_ablate_cm}
\begin{tabular}{@{}ccccccc@{}}
$\mathcal{L}_{kl}$ & $\mathcal{L}_{var}$ & $\mathcal{L}_{mse}$ & $\mathcal{L}_{H}$ & $\mathcal{L}_{ce}$ & \begin{tabular}[c]{@{}l@{}}Cifar10\\ ACC \%\end{tabular} & \begin{tabular}[c]{@{}l@{}}MPSC\\ ACC \%\end{tabular} \\ \midrule
& & & & x & 43.5 & 37.8 \\
& & & x & & 25.5 & 31.2 \\
& & x & & & 53.9 & 48.5 \\
& x & & & & 75.8 & 62.7 \\
x & & & & & 66.1 & 57.3 \\
x & x & & & & 26.7 & 29.5 \\ \bottomrule
\end{tabular}
\end{table}
\section{Discussion and future works}
\textit{Training mechanism}: During the joint optimization, we observe the behavior of individual loss components and monitor their convergence. We observe that each loss has a similar scale and converges simultaneously without any adversarial impact on each other during the optimization process. Moreover, our covariance and consistency loss is upper bounded as the $\mathbf{P}^u_{*i}$ are upper bounded by the SoftMax operation.
%Numerical stability: hyperspherical normalization, the temperature is very crucial: entropy, softmax, log loss, mean, etc.
%Feature-wise statistical loss (large batch): matches the distribution multinoulli distribution. collective properties.
%Importance of Data Augmentation:
%adaptive weights
%numerical stability analysis:
%temperature values, normalized final layers (w2 normalization),
Our distribution learning-based approach underperforms in various cases. Firstly, the model underperforms when the class semantics varies between labeled and unlabeled classes, as reported in the ablation study section aligning with previous work in the NCD field. Secondly, our approach fails to discover tail classes with smaller class probabilities under extreme distribution mismatches between different novel classes.
Finally, our approaches also require considering the large data imbalance for robust performance due to equal sampling from labeled and novel instances, e.g., $90/11$ labeled/unlabeled separation in the UCF101 created an imbalance between their corresponding classes. Under such cases, training with an equal number of samples from both groups at each gradient descent step tends to bias and overfit the model towards the majority sampled group. The data imbalance between label/unlabeled can be tackled by making proportionate larger batch sizes for the larger group or learning more frequently from the majority group. We applied the earlier setting in our experimentation.
% broader connection:
% canonical correlation analysis of probability matrix.
% enforcing class semantic learning: data augmentation. ??
% Importance of Augmentations.
% Connection to mix-match augmentation.
% connected to DINO to avoid collapse: the entropy and distribution loss balances to avoid trivial solutions.
% Learning to cluster without negative pair.
% Future works:
In the future, we aim to investigate distribution learning frameworks toward a theoretical understanding of our approach. We plan to bridge its connection to canonical correlation analysis-based self-supervision learning by developing a generalized formulation as our method seeks to cluster the novel data without using specific negative samples similar to alignment-uniform learning of self-supervision.
Further, we plan to improve our system performance to enhance semantic-based clustering in the future by tackling the problem of cluster corruption during the learning phase. We aim to apply label correction mechanisms to fine-tune the model performance.
We also plan to expand our approach towards generalized categories discovery \cite{vaze2022generalized} problems by relaxing different NCD assumptions. We also seek to identify the detailed impacts of individual components, e.g., models, data, and frameworks in discovering novel categories.
% and explore over-clustering, self-training, and distribution aligning solutions. We also plan to integrate and investigate the impact of attention-based learners (vision transformers) in NCD VAR applications.
% fixing label corruption to improve performance.
% number of clustierng.
% Connection to self-supervised contrastive learning. : Uniformity-alignment, canonical correlation-based method.
% we will investigate future approaches to solve such problems.
% making sure to learn semantic-based clustering.
% Importance of data augmentation.
% requires theoretical understanding
% very important in data augmentation.
% the labeled and unlabeled classes should contain similar semantics and notion of classes.
% Unknown class numbers.
%potential solutions: over clustering, data augmentation, label correction, and label corruption.
%prototype selection
%Optimizing multiple losses together.
%Our current method leverages three typical NCD data assumptions: i) prior knowledge about the number of latent classes for the unlabeled data (unknown target cluster quantity), ii) disjoint sets between the $C_L$ and $C_U$ (negative learning), and iii) the uniform distribution among unlabeled classes (variance regularization). Moreover, the empirical variance estimation requires a large batch size relative to $U$ randomly sampled from unlabeled data as per the strong law of large numbers.
% \textbf{Miscellaneous}
% alternative optimization of two matrices and simultaneous
% batch normalization issues during alternative optimization.
% adaptive loss weights work best.
% More powerful networks may overfit irrelevant features.
\section{Conclusion}
We propose a novel NCD approach by enforcing Multinoulli distribution learning from the class activation patterns of the probability matrix from a large batch of unlabeled data samples. We apply joint constraints on first-order and second-statistics of class representative neurons, information of individual instances, and classification loss to learn a single network capable of classifying labeled data and clustering novel data based on class semantics. We evaluate and analyze our approach by performing extensive experiments with images and video classification using three partial classes-annotated datasets and convolutional networks. Our analysis demonstrates the importance of class notions, augmentations, sampling, and individual losses toward realizing semantic-based clustering and compatible performance with state-of-the-art NCD works without requiring any pertaining and pseudo-labeling.
%MEMORY bank in NCD setting
% \newpage
% \section{Supplimentary Material}
% \subsection{Experiments}
\bibliographystyle{IEEEbib}
\bibliography{refs}
\end{document}
\begin{thebibliography}{10}
\bibitem{han2019learning}
Kai Han, Andrea Vedaldi, and Andrew Zisserman,
\newblock ``Learning to discover novel visual categories via deep transfer
clustering,''
\newblock in {\em ICCV}, 2019, pp. 8401--8409.
\bibitem{han2021autonovel}
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and
Andrew Zisserman,
\newblock ``Autonovel: Automatically discovering and learning novel visual
categories,''
\newblock {\em IEEE TPAMI}, 2021.
\bibitem{han2020automatically}
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and
Andrew Zisserman,
\newblock ``Automatically discovering and learning new visual categories with
ranking statistics,''
\newblock {\em arXiv:2002.05714}, 2020.
\bibitem{hsu2017learning}
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira,
\newblock ``Learning to cluster in order to transfer across domains and
tasks,''
\newblock {\em arXiv:1711.10125}, 2017.
\bibitem{jia2021joint}
Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green,
\newblock ``Joint representation learning and novel category discovery on
single-and multi-modal data,''
\newblock in {\em ICCV}, 2021, pp. 610--619.
\bibitem{zhang2022automatically}
Lu~Zhang, Lu~Qi, Xu~Yang, Hong Qiao, Ming-Hsuan Yang, and Zhiyong Liu,
\newblock ``Automatically discovering novel visual categories with
self-supervised prototype learning,''
\newblock {\em arXiv:2208.00979}, 2022.
\bibitem{fini2021unified}
Enrico Fini, Enver Sangineto, St{\'e}phane Lathuili{\`e}re, Zhun Zhong, Moin
Nabi, and Elisa Ricci,
\newblock ``A unified objective for novel class discovery,''
\newblock in {\em ICCV}, 2021, pp. 9284--9292.
\bibitem{asano2019self}
Yuki~Markus Asano, Christian Rupprecht, and Andrea Vedaldi,
\newblock ``Self-labelling via simultaneous clustering and representation
learning,''
\newblock {\em arXiv:1911.05371}, 2019.
\bibitem{joseph2022spacing}
KJ~Joseph, Sujoy Paul, Gaurav Aggarwal, Soma Biswas, Piyush Rai, Kai Han, and
Vineeth~N Balasubramanian,
\newblock ``Spacing loss for discovering novel categories,''
\newblock in {\em CVPR}, 2022, pp. 3761--3766.
\bibitem{troisemaine2023novel}
Colin Troisemaine, Vincent Lemaire, St{\'e}phane Gosselin, Alexandre
Reiffers-Masson, Joachim Flocon-Cholet, and Sandrine Vaton,
\newblock ``Novel class discovery: an introduction and key concepts,''
\newblock {\em arXiv preprint arXiv:2302.12028}, 2023.
\bibitem{hsu2019multi}
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira,
\newblock ``Multi-class classification without multi-class labels,''
\newblock {\em arXiv preprint arXiv:1901.00544}, 2019.
\bibitem{xie2016unsupervised}
Junyuan Xie, Ross Girshick, and Ali Farhadi,
\newblock ``Unsupervised deep embedding for clustering analysis,''
\newblock in {\em International conference on machine learning}. PMLR, 2016,
pp. 478--487.
\bibitem{chi2021meta}
Haoang Chi, Feng Liu, Bo~Han, Wenjing Yang, Long Lan, Tongliang Liu, Gang Niu,
Mingyuan Zhou, and Masashi Sugiyama,
\newblock ``Meta discovery: Learning to discover novel classes given very
limited data,''
\newblock {\em arXiv preprint arXiv:2102.04002}, 2021.
\bibitem{zhong2021openmix}
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi~Yang, and Nicu Sebe,
\newblock ``Openmix: Reviving known knowledge for discovering novel visual
categories in an open world,''
\newblock in {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, 2021, pp. 9462--9470.
\bibitem{hasan2023nev}
Zahid Hasan, Masud Ahmed, Abu Zaher~Md Faridee, Sanjay Purushotham, Heesung
Kwon, Hyungtae Lee, and Nirmalya Roy,
\newblock ``Nev-ncd: Negative learning, entropy, and variance regularization
based novel action categories discovery,''
\newblock {\em arXiv preprint arXiv:2304.07354}, 2023.
\bibitem{arazo2020pseudo}
Eric Arazo, Diego Ortego, Paul Albert, Noel~E O’Connor, and Kevin McGuinness,
\newblock ``Pseudo-labeling and confirmation bias in deep semi-supervised
learning,''
\newblock in {\em 2020 International Joint Conference on Neural Networks
(IJCNN)}. IEEE, 2020, pp. 1--8.
\bibitem{hotelling1992relations}
Harold Hotelling,
\newblock ``Relations between two sets of variates,''
\newblock {\em Breakthroughs in statistics: methodology and distribution}, pp.
162--190, 1992.
\bibitem{balestriero2023cookbook}
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar,
Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian,
et~al.,
\newblock ``A cookbook of self-supervised learning,''
\newblock {\em arXiv preprint arXiv:2304.12210}, 2023.
\bibitem{caron2021emerging}
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv{\'e} J{\'e}gou, Julien Mairal,
Piotr Bojanowski, and Armand Joulin,
\newblock ``Emerging properties in self-supervised vision transformers,''
\newblock in {\em Proceedings of the IEEE/CVF international conference on
computer vision}, 2021, pp. 9650--9660.
\bibitem{zbontar2021barlow}
Jure Zbontar, Li~Jing, Ishan Misra, Yann LeCun, and St{\'e}phane Deny,
\newblock ``Barlow twins: Self-supervised learning via redundancy reduction,''
\newblock in {\em International Conference on Machine Learning}. PMLR, 2021,
pp. 12310--12320.
\bibitem{ermolov2021whitening}
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe,
\newblock ``Whitening for self-supervised representation learning,''
\newblock in {\em International Conference on Machine Learning}. PMLR, 2021,
pp. 3015--3024.
\bibitem{caron2020unsupervised}
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and
Armand Joulin,
\newblock ``Unsupervised learning of visual features by contrasting cluster
assignments,''
\newblock {\em Advances in neural information processing systems}, vol. 33, pp.
9912--9924, 2020.
\bibitem{murphy2012machine}
Kevin~P Murphy,
\newblock {\em Machine learning: a probabilistic perspective},
\newblock MIT press, 2012.
\bibitem{grandvalet2004semi}
Yves Grandvalet and Yoshua Bengio,
\newblock ``Semi-supervised learning by entropy minimization,''
\newblock {\em NeurIPS}, vol. 17, 2004.
\bibitem{yang2022divide}
Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng,
\newblock ``Divide and conquer: Compositional experts for generalized novel
class discovery,''
\newblock in {\em CVPR}, 2022, pp. 14268--14277.
\bibitem{krizhevsky2009learning}
Alex Krizhevsky, Geoffrey Hinton, et~al.,
\newblock ``Learning multiple layers of features from tiny images,''
\newblock 2009.
\bibitem{soomro2012ucf101}
Khurram Soomro, Amir~Roshan Zamir, and Mubarak Shah,
\newblock ``Ucf101: A dataset of 101 human actions classes from videos in the
wild,''
\newblock {\em arXiv preprint arXiv:1212.0402}, 2012.
\bibitem{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
\newblock ``Deep residual learning for image recognition,''
\newblock in {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, 2016, pp. 770--778.
\bibitem{sandler2018mobilenetv2}
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh
Chen,
\newblock ``Mobilenetv2: Inverted residuals and linear bottlenecks,''
\newblock in {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, 2018, pp. 4510--4520.
\bibitem{feichtenhofer2019slowfast}
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He,
\newblock ``Slowfast networks for video recognition,''
\newblock in {\em ICCV}, 2019, pp. 6202--6211.
\bibitem{tran2015learning}
Du~Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri,
\newblock ``Learning spatiotemporal features with 3d convolutional networks,''
\newblock in {\em ICCV}, 2015, pp. 4489--4497.
\bibitem{qiu2017learning}
Zhaofan Qiu, Ting Yao, and Tao Mei,
\newblock ``Learning spatio-temporal representation with pseudo-3d residual
networks,''
\newblock in {\em ICCV}, 2017, pp. 5533--5541.
\bibitem{liu2022residual}
Yu~Liu and Tinne Tuytelaars,
\newblock ``Residual tuning: Toward novel category discovery without labels,''
\newblock {\em IEEE Transactions on Neural Networks and Learning Systems},
2022.
\bibitem{kuhn1955hungarian}
Harold~W Kuhn,
\newblock ``The hungarian method for the assignment problem,''
\newblock {\em Naval research logistics quarterly}, vol. 2, no. 1-2, pp.
83--97, 1955.
\bibitem{vaze2022generalized}
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman,
\newblock ``Generalized category discovery,''
\newblock in {\em CVPR}, 2022, pp. 7492--7501.
\end{thebibliography}