Measuring Self-Supervised Representation Quality for Downstream Classification using Discriminative Features
Extracted source text · arXiv 2203.01881
\documentclass{article}
\usepackage{arxiv}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{lipsum}
\usepackage{fancyhdr} % header
\usepackage{graphicx} % graphics
\graphicspath{{media/}} % organize your images and other figures under media/ folder
\usepackage{amsmath, bbm}
\usepackage{subfigure}
\usepackage{multirow}
\newcommand{\bff}{\mathbf{f}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\bh}{\mathbf{h}}
\newcommand{\bz}{\mathbf{z}}
\newcommand{\bbR}{\mathbb{R}}
\newcommand{\bH}{\mathbf{H}}
%% Title
\title{Understanding Failure Modes of \\
Self-Supervised Learning}
\author{
Neha Mukund Kalibhat \\
Department of Computer Science \\
University of Maryland, College Park \\
\texttt{nehamk@cs.umd.edu} \\
\And
Kanika Narang \\
Meta AI \\
\texttt{kanika13@fb.com} \\
\And
Liang Tan \\
Meta AI \\
\texttt{liangtan@fb.com} \\
\And
Hamed Firooz \\
Meta AI \\
\texttt{mhfirooz@fb.com} \\
\And
Maziar Sanjabi \\
Meta AI \\
\texttt{maziars@fb.com} \\
\And
Soheil Feizi \\
Department of Computer Science \\
University of Maryland, College Park \\
\texttt{sfeizi@cs.umd.edu} \\
}
\begin{document}
\maketitle
\input{abstract}
\input{tables_and_figures/correct_vs_incorrect_latent}
\input{tables_and_figures/correct_vs_incorrect_latent_reg}
\input{tables_and_figures/grad_cam_noisy}
\input{introduction}
\input{tables_and_figures/metrics_pr_roc_curves}
\input{related_work}
\input{tables_and_figures/metrics_auc}
\input{failure_modes}
\input{tables_and_figures/zscore_l1norm_scatter}
\input{metrics}
\input{tables_and_figures/class_acc}
\input{qscore}
\input{results}
\input{discussion}
\clearpage
\bibliography{references}
\bibliographystyle{apalike}
\newpage
\input{appendix}
\end{document}
\begin{abstract}
Self-supervised learning methods have shown impressive results in downstream classification tasks. However, there is limited work in understanding their failure models and interpreting the learned representations of these models. In this paper, we tackle these issues and study the representation space of self-supervised models by understanding the underlying reasons for misclassifications in a downstream task. Over several state-of-the-art self-supervised models including SimCLR, SwaV, MoCo V2 and BYOL, we observe that representations of {\it correctly} classified samples have few discriminative features with highly deviated values compared to other features. This is in a clear contrast with representations of {\it misclassified} samples. We also observe that noisy features in the representation space often correspond to spurious attributes in images making the models less interpretable. Building on these observations, we propose a sample-wise Self-Supervised Representation Quality Score (or, Q-Score) that, without access to any label information, is able to predict if a given sample is likely to be misclassified in the downstream task, achieving an AUPRC of up to $0.90$. Q-Score can also be used as a regularization to remedy low-quality representations leading to $3.26\%$ relative improvement in accuracy of SimCLR on ImageNet-100. Moreover, we show that Q-Score regularization increases representation sparsity, thus reducing noise and improving interpretability through gradient heatmaps.
%Self-supervised representation learning methods have shown impressive results in downstream classification tasks. However, there is limited work in understanding and interpreting the learned representations of these models. In this paper, we study the representation space of self-supervised models by understanding the reasons for misclassification in downstream tasks. We observe that correctly-classified representations obey certain properties that misclassifications do not. Noisy features in the representation space, often corresponding to spurious attributes, make these models less interpretable. Building on our observations, we discover various quality metrics and define a new score called Self-Supervised Representation Quality Score (or Q-Score). Q-Score is computed using the mean, standard deviation and L1 norm of any given representation \KN{Do we really want to say this, it kind of trivializes the metric imo.} and serves as a reliable metric to assess its quality in an unsupervised manner. We show that Q-Score is effective in determining which sample is likely to be correctly or incorrectly classified for various state-of-the-art self-supervised models with an AUPRC of up to $0.90$. Q-Score can also be used as a regularization to remedy low-quality representations leading to $3.26\%$ relative improvement in accuracy of SimCLR on ImageNet-100. Moreover, we show that Q-Score regularization increases representation sparsity, therefore reducing noise and improving interpretability through gradient heat maps.
% Self-supervised methods have shown impressive results in downstream classification tasks. However, there is limited work in understanding and interpreting the learned representations of these models. In this paper, we study the representation space of self-supervised models and understand the reasons for misclassification in downstream tasks. We discover various properties of extracted features that make them useful and informative, using which we define a new score called Self-Supervised Model Quality Score (or Q-Score). Q-Score is computed using the mean, standard deviation and L1 norm of any given representation and serves as a reliable metric to assess its quality in an unsupervised manner\maz{I think we have niot shown this? Have we shown that the better the score is for a feature extractor, the better it is at downstream classification?}. We show that Q-Score is effective in determining which sample is likely to be correctly or incorrectly classified for various state-of-the-art self-supervised models. Q-Score can also be used as a regularization to remedy low-quality representations of pre-trained models. Moreover, we show that our regularization improves representation sparsity and interpretability through gradient heat maps.
%Self-supervised methods have shown impressive results in downstream classification tasks. However, there is limited work in understanding and interpreting learned representations of these models. In this paper, we observe what causes self-supervised representations to be mis-classified. We discover various properties of features that make them useful and informative. Using these properties, we define a new score (SSL Q-Score) to assess the quality of each representation generated by a self-supervised model in an unsupervised manner. We show that Q-Score is effective in determining which sample is likely to be correctly or incorrectly classified for various state-of-the-art self-supervised models. Q-Score can also be used as a regularization to remedy low-quality representations of pre-trained models. Moreover, we show that our regularization improves representation sparsity and interpretability through gradient heat maps.
\end{abstract}
\begin{figure*}[ht]
\centering
\subfigure{\includegraphics[width=0.48\textwidth, trim={1cm 0cm 3cm
0cm}, clip]{figures/imagenet100_top_features_correct.pdf}}
\subfigure{\includegraphics[width=0.48\textwidth, trim={1cm 0cm 3cm
0cm}, clip]{figures/imagenet100_top_features_incorrect.pdf}}
\caption{\textbf{Representation space of SimCLR:} In this heatmap, we visualize the top features of class-averages representations of correct (left) and incorrect (right) classifications. We observe that, the correct classifications contain features that are significantly deviated from the rest of the features, whereas misclassifications do not. The set of discriminative features for each class may be unique indicating that they capture distinguishable characteristics of each class. }
\label{fig:correct_vs_incorrect_latent}
\end{figure*}
\begin{figure*}[ht]
\centering
\subfigure{\includegraphics[width=0.48\textwidth, trim={1cm 0cm 3cm
0cm}, clip]{figures/imagenet100_top_features_correct_reg.pdf}}
\subfigure{\includegraphics[width=0.48\textwidth, trim={1cm 0cm 3cm
0cm}, clip]{figures/imagenet100_top_features_incorrect_reg.pdf}}
\caption{\textbf{Representation space of SimCLR (Q-Score regularized):} In this heatmap, we visualize the top features of class-averages representations of correct (left) and incorrect (right) classifications. We observe that, the regularization improves the discriminative features, compared to our baseline in Figure \ref{fig:correct_vs_incorrect_latent}. Moreover, misclassifications also contain visibly discriminative features, unlike the baseline.}
\label{fig:correct_vs_incorrect_latent_reg}
\end{figure*}
\begin{figure*}[h]
\centering
\includegraphics[width=0.78\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/grad_cam_noisy.png}
\caption{\textbf{Heatmaps of significant and noisy features of SimCLR:} We observe the heatmaps of significant and noisy features of 3 classes with SimCLR. We observe that significant features are informative in correct classifications and can be spurious or irrelevant in mis-classifications. We identify this as a failure mode of self-supervised models. The representations contain many noisy features that are close to 0. These features highlight uninformative parts of the image, without contributing much to classification.}
\label{fig:grad_cam_noisy}
\end{figure*}
\section{Introduction}
Self-supervised models learn to extract useful representations from data without relying on human supervision. These models \cite{simclr, swav, mocov2, byol, simsiam, deepcluster, khosla2020supervised} have shown comparable results to supervised models in downstream classification tasks. By means of data augmentation, these models are trained to encode semantically relevant information from images while ignoring {\it nuisance} %\KN{Is that the right word, can we cite some paper?}
aspects. Therefore, the representations ultimately should only contain the information required to define a given sample. However, in practice, learned representations are often quite noisy and not interpretable causing difficulties in understanding and debugging their failure models \cite{anonymous2022understanding, huang2021generalization, ericsson2021selfsupervised}.
% \SF{This paragraph is vague and not clear. Instead, I'd put some specific examples of failure modes of SSL or its lack of interpretability in the latent space, coupled with some heatmaps. Expand.} The \textit{informative} features should also, ideally, be unique to a given class \SF{can't classes classes informative features? I think you mean there should be informative discriminatory features for each class}. In practice, however, this is not the case.
In this paper, we first study the cause of downstream misclassifications in self-supervised learning (SSL) \cite{simclr, swav, mocov2, byol}. We make several empirical observations: In Figure \ref{fig:correct_vs_incorrect_latent}, we observe that each class has an informative discriminatory set of features that contribute to correct classifications. Misclassifications are caused when such visual patterns are not present in the representation space. We also show that the learned representations end up being mostly noisy, encoding a lot of information that is irrelevant to the class label. In Figure \ref{fig:grad_cam_noisy}, we observe that discriminative features highlight useful attributes whereas noisy features often correspond to spurious attributes.
Building on these observations, we study properties of \textit{proper} representations with the help of several statistical metrics. Over several state-of-the-art self-supervised models including SimCLR \cite{simclr}, SwaV \cite{swav}, MoCoV2 \cite{mocov2} and BYOL \cite{byol}, we observe that representations of {\it correctly} classified samples, in contrast to misclassified ones, have few discriminative features with highly deviated values compared to other features.
Therefore, we propose a {\bf Self-Supervised Quality Score (Q-Score)}, which is an unsupervised score for measuring the representation quality of each sample in self-supervised models (without requiring the label information); i.e., to what extent a given sample satisfies the favorable representation properties.
A high Q-Score for a sample implies that its representation is sparse with at least one discriminative feature that is highly deviated compared to other features. We empirically observe that the higher the Q-Score, the more likely that the sample will be correctly classified. We confirm this on SimCLR, SwaV, MoCoV2 and BYOL by computing their ROC and Precision-Recall curves (Figure \ref{fig:metrics_pr_roc_curves}). Q-Score achieves an AUPRC of up to $0.90$ and AUROC of up to $0.74$ for these models.
%Building on these observations, we study properties of \textit{proper} representations with the help of several statistical metrics like mean, standard deviation, sparsity, etc. Using these metrics, we propose a {\bf Self-Supervised Quality Score (Q-Score)}, which is an unsupervised score for measuring the representation quality of each sample in self-supervised models (without requiring the label information); i.e., to what extent does a given sample satisfy the favorable representation properties. Q-Score is a combination of L1-norm, mean and standard deviation of a given representation such that it encapsulates all the properties we have identified, including, sparsity and discriminative features. We empirically observe that the higher the Q-Score, the more likely that the sample will be correctly classified. We confirm this on SimCLR \cite{simclr}, SwaV \cite{swav}, MoCoV2 \cite{mocov2} and BYOL \cite{byol} by observing ROC and Precision-Recall curves. Q-Score achieves an AUPRC of up to $0.90$ and AUROC of up to $0.74$ for these models.
We next use Q-Score as a regularizer during the training of self-supervised models to increase the Q-score of samples with low-quality representations, pushing their representations to be less noisy and more informative. With Q-Score regularization, we achieve $3.26\%$ improvement in test accuracy of SimCLR trained on ImageNet-100. This regularization improves class-wise accuracy and Q-Score of the trained encoder. The representations show an improved structure, less noise and more more interpretability.
%The Q-Score, when applied as a regularizer to the self-supervised loss, can be used to maximize the score of \textit{low quality} samples in the dataset, pushing them to be less noisy and more informative.
We summarize our contributions as follows:
\begin{itemize}
\item We study failure modes of self-supervised models in downstream classification tasks and identify several properties of correctly and incorrectly classified samples across classes and common to different self-supervised models.
%\item We compute statistical metrics over each sample to identify how likely it is to be correctly classified. We observe up to $0.90$ AUPRC and $0.74$ AUROC of these metrics in distinguishing between correct and incorrect classifications.
\item We introduce Self-Supervised Quality Score (Q-Score) to measure the quality of each learned representation. We empirically observe that the higher the Q-Score, the more likely that the sample will be correctly classified, achieving an AUPRC of up to $0.90$.
%\item We define Self-Supervised Quality Score (Q-Score), that measures the quality of each learned representation
\item We apply Q-Score as a regularizer to the self-supervised loss and show that, by improving the quality of low-score samples, we can improve downstream classification task accuracy by $3.26\%$.
\item We show that using the Q-Score regularization improves the interpretability of self-supervised representations by removing noise and highlighting useful information in each representation.
\end{itemize}
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/simclr_roc.pdf}}
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/swav_roc.pdf}}
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/mocov2_roc.pdf}}
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/byol_roc.pdf}}
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/simclr_pr.pdf}}
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/swav_pr.pdf}}
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/mocov2_pr.pdf}}
\subfigure{\includegraphics[width = 0.24\textwidth, trim={1cm 1cm 0.9cm
1cm}, clip]{figures/byol_pr.pdf}}
\caption{\textbf{ROC and Precision-Recall curves of various quality metrics:} We plot the ROC and Precision-Recall curves of the metrics we identify in Section \ref{sec:metrics} for 4 self-supervised models. We observe that Q-Score is the most effective among all models in identifying how likely a given sample will be classified correctly or incorrectly.}
\label{fig:metrics_pr_roc_curves}
\end{figure*}
\section{Related Work}
Unsupervised methods for classification has been a long-standing area of research, generally involving the use of clustering techniques \cite{pmlr-v70-bojanowski17a, NIPS2014_07563a3f, YM.2020Self-labelling, NIPS2016_65fc52ed, Caron_2018_ECCV, Caron_2019_ICCV, pmlr-v97-huang19b}. Self-supervised learning, is a recent approach to learn without human supervision by training models to prepare their own labels for each example \cite{pmlr-v70-bojanowski17a, NIPS2014_07563a3f, Wu_2018_CVPR, 7312476} usually with the help of a contrastive loss. Contrastive learning \cite{arora2019theoretical, tosh2021contrastive, NEURIPS2019_ddf35421} usually uses a temperature-controlled cross-entropy loss between positive pairs of \textit{similar} samples and negative pairs of \textit{dissimilar} samples. Positive pairs are usually considered as multiple transformations (views) \cite{NEURIPS2020_4c2e5eaa} of a given sample using stochastic data augmentation. Through this approach, several state of the art self-supervised techniques \cite{simclr, swav, simsiam, byol, mocov2, khosla2020supervised} have produced representations that give linear classification accuracy comparable to a supervised approach.
Understanding these learned representations is relatively less explored. \cite{anonymous2022understanding}, observes that self-supervised representations collapse to a lower dimensional space instead of the entire embedding space. Other methods \cite{Kugelenetal21, xiao2021what}, propose to separate the representation space into variant and invariant information so that augmentations are not task-specific. \cite{grigg2021selfsupervised}, observe representations across layers of the encoder and compare it to supervised setups. In this work, we focus more on thoroughly studying the representation space, its properties and the causes of mis-classifications in self-supervised models when they are used in down-stream classification.
% \KN{Say 1-2 lines about what are we doing and how is it different from others?}
\begin{table*}[h]
\centering
\resizebox{0.78\textwidth}{!}{
\begin{tabular}{l|c|c|c|c||c|c|c|c}
\toprule
\multirow{2}{*}{\textbf{Metric}} & \multicolumn{4}{c}{\textbf{AUROC}} & \multicolumn{4}{c}{\textbf{AUPRC}} \\
& \textbf{SimCLR} & \textbf{SwaV} & \textbf{MoCo V2} & \textbf{BYOL} & \textbf{SimCLR} & \textbf{SwaV} & \textbf{MoCo V2} & \textbf{BYOL} \\
\midrule
\midrule
$\mu_i$ (Mean) & 0.73 & 0.62 & 0.57 & 0.70 & 0.90 & 0.79 & 0.73 & 0.84 \\
$\sigma_i$ (Standard Deviation) & 0.71 & 0.63 & 0.58 & 0.67 & 0.88 & 0.78 & 0.74 & 0.83 \\
$\bh_i$ (Sparsity) & 0.54 & 0.58 & 0.58 & 0.58 & 0.89 & 0.77 & 0.80 & 0.84 \\
$\|\bh_i\|_1$ (L1 Norm) & 0.71 & 0.63 & 0.58 & 0.67 & 0.88 & 0.78 & 0.74 & 0.83\\
$\dfrac{\max(\bh_i) - \mu_i}{\sigma_i}$ (Z-Score) & 0.74 & 0.63 & 0.58 & 0.70 & 0.90 & 0.80 & 0.74 & 0.84 \\
SSL Q-Score & 0.74 & 0.63 & 0.58 & 0.70 & 0.90 & 0.79 & 0.74 & 0.84 \\
\bottomrule
\end{tabular}
}
\caption{\textbf{AUPRC and AUROC of quality metrics:} We compute the AUPRC and AUROC of each quality metric in Figure \ref{fig:metrics_pr_roc_curves}. These scores determine how effective each metric is in identifying correct or incorrect classifications in an unsupervised manner. We observe that our Q-Score, is consistently effective across 4 self-supervised models in measuring quality of a given representation.}
\label{tab:metrics_auc}
\end{table*}
\section{Failure Modes of Self-Supervised Learning}
In this section, we study some properties of the latent space that drive representations to be correctly classified by a downstream linear classifier. We use SimCLR \cite{simclr} pre-trained over ImageNet-100 \cite{imagenet} as our baseline and train a linear classifier over the learned representations. In Figure \ref{fig:correct_vs_incorrect_latent}, we visualize the mean representations of each class in ImageNet-100 (sorted in decreasing order of accuracy). We selectively display the top features (in terms of magnitude) for each of these classes. On the left, we show the mean representations of correctly classified samples in each class while on the right, we show mean representations of misclassifications within each class.
The most compelling observation in Figure \ref{fig:correct_vs_incorrect_latent} is that there is a clear difference between representations of correctly and incorrectly classified examples. Through some visual analysis, we can draw several conclusions about the useful properties of latent representations. First, the representation space is {\it nearly} sparse, i.e., most features of each class are close to 0. Since we are observing class-averaged representations, these low-magnitude features can be noisy and they may be activated only for a small percentage of samples within the class \cite{anonymous2022understanding}. Second, correctly classified examples for each class have a select few \textit{discriminating} features that are highly deviated from the remaining features in the representation space. In most cases, these features are unique to a given class and are not dominant in other classes. Incorrectly classified examples are also sparse, but do not satisfy the latter property. Third, the features that are activated across all samples can be considered as noisy features since they are unlikely to encode any class-specific, useful information. These visual observations imply that we can potentially classify representations in an unsupervised manner (i.e. without requiring label information) by leveraging the structural properties of representations.
To better study the effect of individual features, we visualize gradients of both dominant and noisy features with respect to the inputs (referred to as heatmaps) in Figure \ref{fig:grad_cam_noisy}. In each of the correctly classified images, we observe that the heatmap of the most discriminative feature of that class, captures a relevant and defining characteristic of the image, therefore is highly correlated with the ground-truth. If we observe the heatmap of the same feature in a mis-classified image, it often highlights noisy and spurious parts of the image. Noisy features (i.e., features that are less significant by magnitude) also show noisy heatmaps which focus on aspects that are uninformative. These observations imply that representations can be riddled with noisy features that often do not contribute to correct classifications. Misclassifications can be attributed to two reasons i.e., missing class-specific dominant features as shown in Figure \ref{fig:correct_vs_incorrect_latent}, and dominant features mapping to spurious attributes in images. In the next section we build metrics to quantify and predict these failure modes of self-supervised models. For more visualized heatmaps of other classes with similar observations see Appendix Section \ref{sec:more_grad_cams}.
% \SF{this sentence is vague; expand/explain.}.
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width=0.26\textwidth, trim={1cm 1cm 1cm
0.9cm}, clip]{figures/z_score.pdf}}
\subfigure{\includegraphics[width=0.26\textwidth, trim={1cm 1cm 1cm
0.9cm}, clip]{figures/l1_norm.pdf}}
\subfigure{\includegraphics[width=0.26\textwidth, trim={1cm 1cm 1cm
0.9cm}, clip]{figures/qscore.pdf}}
\caption{\textbf{Scatter plots of correct and incorrect classifications:} In these plots, we visualize 3 quality metrics for correctly and incorrectly classified samples. In the first plot, we calculate Z-Score($\max(\bh_i)$) and show that correct classifications have a higher value than incorrect classifications. In the second plot, we calculate $\|\bh_i\|_i$ and show that the L1 norm of incorrectly classified representations is higher than correctly classified representations. Combining both these metrics, in the third plot, we calculate the Q-Score, see \eqref{eq:q-score}, showing that Q-Score of correct classifications is higher than incorrect classifications.}
\label{fig:zscore_l1norm_scatter}
\end{figure*}
\section{Self-Supervised Representation Quality Metrics}
\label{sec:metrics}
Building on our observations, in this section, we define quality metrics on the representation space of self-supervised models. Let us consider a SimCLR model with a ResNet \cite{resnet} base encoder $f(.)$ and an MLP projection head $g(.)$. We define $\bx_i \in \bbR^r$ and $\Tilde{\bx}_i \in \bbR^r$ as two transformed views of the $i^{th}$ sample in a given input dataset containing $N$ samples. The data transformation we apply, is a combination of \textit{random crop}, \textit{random horizontal flip}, \textit{random color distortion} and \textit{random Gaussian blur}. Similar to SimCLR, we pass the input samples through the base encoder to get self-supervised representations denoted by $f(\bx_i) = \bh_i \in \bbR^l$ and $f(\Tilde{\bx}_i) = \Tilde{\bh}_i \in \bbR^l$ where $l$ is the size of the representation space. For contrastive training, we use the output of the projection head $g(\bh_i) = \bz_i \in \bbR^m$ and $g(\Tilde{\bh}_i) = \Tilde{\bz}_i \in \bbR^m$ where $m$ is the size of the projection space.
The SimCLR optimization for the set of model parameters $\theta$, is as follows,
\begin{align}
\max_\theta \frac{1}{2N} \sum_{i=1}^{2N} \frac{\exp(sim(\bz_i, \Tilde{\bz}_i))}{\sum_{j=1}^{2N} \mathbbm{1}_{j \ne i} \exp(sim(\bz_i, \bz_j))}
\end{align}
where $sim(\bz_i, \bz_j) = \frac{1}{\tau} \frac{\bz_i^T \bz_j}{\|\bz_i\|\|\bz_j\|}$. Since the representations $\bh_i \in \bbR^l$ are used in downstream classification tasks, we calculate the following quality metrics over this space for each sample:
\begin{itemize}
\item Mean ($\mu_i$) - This metric calculates the mean of each representation $\bh_i$.
\item Standard Deviation ($\sigma_i$) - This metric calculates the standard deviation of each representation $\bh_i$.
\item Soft Sparsity - This metric calculates the \% of features in $\bh_i$ that are $< \eta$, where $0 < \eta \ll 1$.
\item L1 Norm ($\|\bh_i\|_1$) - This metric calculates the L1 norm of each representation, $\bh_i$.
\item Z-Score of $\max(\bh_i)$ $\left(=\frac{\max(\bh_i) - \mu_i}{\sigma_i}\right)$ - This metric calculates the element-wise maximum of $\bh_i$ and computes its z-score using $\mu_i$ and $\sigma_i$.
\end{itemize}
It is worth noting that calculating these values do {\it not} require any labels and only require access to the latent representation of samples. Our objective with these metrics is to understand how effective they are in differentiating between correctly and incorrectly classified representations in an unsupervised manner. To achieve this, we demonstrate ROC (receiver operating characteristic) and precision-recall curves in Figure \ref{fig:metrics_pr_roc_curves} of each metric computed over representations of state-of-the-art self-supervised models including SimCLR \cite{simclr}, SwaV \cite{swav}, MoCo V2 \cite{mocov2} and BYOL \cite{byol}. We also tabulate the corresponding AUROC (area under ROC curve) and AUPRC (area under precision-recall curve) scores for each metric in Table \ref{tab:metrics_auc}.
We observe that the L1 and Z-Score are consistently reliable metrics across models in identifying correctly from incorrectly classified representations. In Figure \ref{fig:zscore_l1norm_scatter}, we visualize these metrics across $5,000$ test samples in ImageNet-100 for a SimCLR encoder. We can observe that representations with a higher value of Z-Score are more likely to be correctly classified. Similarly, representations with a lower value of L1 score are more likely to be correctly classified. Therefore, we conclude that these are useful metrics to measure the \textit{quality} of latent representations.
% \SF{these are strong observations; I'd add the highlights to abstract/intro}
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width=0.8\textwidth, trim={1cm 2cm 0.5cm
0.5cm}, clip]{figures/imagenet100_class_acc_annotated.png}}
\caption{\textbf{Class-wise accuracy of SimCLR with and without Q-Score regularization):} We plot the class-wise accuracy of SimCLR in increasing order with SimCLR (Q-Score regularized) accuracy in red on top. We observe that $53$ out of $100$ classes improve in accuracy with regularization. When grouped into 3 sections, we observe the most improvement ($5.86\%$) in the classes of lowest accuracy.}
\label{fig:class_acc}
\end{figure*}
\section{Self-Supervised Q-Score}
\label{sec:qscore}
We use the metrics identified in Section \ref{sec:metrics} to design our own Self-Supervised Quality Score which we refer to as the Q-Score. The Q-Score for sample $i$ is defined as,
\begin{align}
\label{eq:q-score}
Q_i := \frac{\text{Z-Score}(\max({\bh_i}))}{\|\bh_i\|_1}
\end{align}
Intuitively, higher $Q_i$ implies that the representation is sparse with at least one discriminative feature that is highly deviated. In Figure \ref{fig:metrics_pr_roc_curves} and Table \ref{tab:metrics_auc}, we show that the Q-Score performs well across all self-supervised models in identifying correct and incorrectly classified representations. It combines two favorable properties of representations i.e., sparsity (or feature noise, computed by L1 norm) and highly-deviant features (computed by Z-Score($\max\bh_i$)). Although Table \ref{tab:metrics_auc} shows comparable scores while using either one metric, we prepare our metric such that it captures the intuition behind both metrics.
% \SF{expand on it; is it better than just Z score or L1 score? how much better? explain.}
Q-Score can also be used to improve the quality of representations of pre-trained models by using it as a regularization term. For example, we can apply this regularizer to the SimCLR optimization as follows,
\begin{align}
\max_\theta \frac{1}{2N} \sum_{i=1}^{2N} &\Big[\frac{\exp(sim(\bz_i, \Tilde{\bz}_i))}{\sum_{j=1}^{2N} \mathbbm{1}_{j \ne i} \exp(sim(\bz_i, \bz_j))} \nonumber \\
&+ \lambda_1 \mathbbm{1}_{Q_i < \alpha}( Q_i) \Big]
\end{align}
where, $\alpha$ is a threshold with which we select the samples whose $Q$ scores should be maximized and $\lambda_1$ is the regularization coefficient. The goal of this regularization is to improve low-quality representations, similar to the ones shown in Figure \ref{fig:correct_vs_incorrect_latent}, by maximizing their Q-Score to prevent them from being misclassified in downstream tasks.
In practice, however, directly applying this regularization leads to a trivial solution where a small set of features gets activated for all samples as shown in Figure \ref{fig:trivial}. This is not a favorable situation because these representations become harder to classify accurately and more importantly, the discriminative features are no longer \textit{informative} because they are activated for all samples. Such features have significantly large L1 norms {\it across} samples compared to the remaining features. Therefore, in our revised optimization, we penalize features that have large L1 norms across samples. Let us denote all representations in our dataset by $\bH \in \bbR^{2N \times l}$ where $\|\bH_{*,k}\|_1$ represents the L1 norm of the $k^{\text{th}}$ column (corresponding to the $k^{\text{th}}$ feature). Our regularized objective would then be,
\begin{align}
\max_\theta \frac{1}{2N} &\sum_{i=1}^{2N} \Bigg[\frac{\exp(sim(\bz_i, \Tilde{\bz}_i))}{\sum_{j=1}^{2N} \mathbbm{1}_{j \ne i} \exp(sim(\bz_i, \bz_j))} \nonumber \\
&+ \lambda_1 \mathbbm{1}_{Q_i < \alpha}( Q_i) \Bigg] \nonumber \\
&- \lambda_2 \sum_{k = 1}^{l} \mathbbm{1}_{\|\bH_{*,k}\|_1 > \beta} (\|\bH_{*,k}\|_1)
\end{align}
where the threshold $\beta$ helps us select the uninformative features whose L1 norms should be minimized.
\section{Results}
\input{tables_and_figures/sorted_features}
\subsection{Experimental Setup}
Our setup consists of a SimCLR encoder ($f(.)$) and projection head ($g(.)$) pre-trained on ImageNet-100. We use 224x224 images, batch size of $128$ and $\tau = 0.2$. We optimize our loss using LARS \cite{You2017LargeBT}, with a learning rate of $0.01$ and warmup-anneal scheduling. We train the regularized encoder until both the SimCLR loss and the regularizer loss converge, therefore, the number of iterations depends on the selection of $\lambda_1$ and $\lambda_2$ which are selected empirically. The higher the regularizer weights, the longer the encoder takes to converge. We get the best result with $\lambda_1 = 0.1, \lambda_2 = 0.1$ trained for $30,000$ iterations.
\input{tables_and_figures/grad_cam_noisy_reg}
\input{tables_and_figures/acc_baseline_vs_reg}
\input{tables_and_figures/sparsity}
\subsection{Q-Score Regularization}
\subsubsection{Accuracy}
We perform Q-Score regularization as discussed in Section \ref{sec:qscore} on ImageNet-100 pre-trained SimCLR. As shown in Table \ref{tab:acc_baseline_vs_reg}, we observe a $3.26\%$ relative improvement in accuracy. If we plot the class-wise accuracy as shown in Figure \ref{fig:class_acc}, we can see improvements for most classes. Among the previously lower performing classes, we observe the highest overall improvement of almost $6\%$. In some classes, however, we under-perform compared to the baseline. Most of these classes are of animal super-classes such as dog breeds, reptiles, primates etc. ImageNet-100 contains several such classes which share a lot of common features in the representation space and often tend to be misclassified. The Q-Score regularization, amplifies, features that are already dominant. Due to this, samples that are ambiguous and borderline between classes, can get misclassified. Applying Q-score regularization earlier during training, can prevent the effect of spurious dominant features.
% \maz{We need some positive spin here. Do we think this could have been improved if we used regularization earlier in training before those dominant features are formed? If so, maybe we should mention it to say there is a way to get away from this downside.}
\subsubsection{Latent Representation}
In Figure \ref{fig:correct_vs_incorrect_latent_reg}, we visualize the latent representations after Q-score regularization similar to Figure \ref{fig:correct_vs_incorrect_latent}. Among the correct classifications, we observe that the dominant features are more deviated from the remaining features compared to the baseline in Figure \ref{fig:correct_vs_incorrect_latent}. The misclassified samples in the regularized model also obey the favorable properties we have identified (which is not true in Figure \ref{fig:correct_vs_incorrect_latent}) i.e, their representations are sparse and contain highly deviant features that are class-specific and informative. These results indicate that the Q-Score regularization is effective in improving the quality of all representations.
% \KN{What is the point? The last sentence seems incomplete.}
In Figure \ref{fig:sorted_features}, we visualize the class-averaged representations (combining correct and incorrect classifications), sorting each class-average representation by magnitude of its features in descending order. On the vertical axis we also sort the classes based on their accuracy. In the baseline, we observe 1-2 discriminative features for several classes. However, not all classes, especially the ones of lower accuracy, contain such features. With Q-Score regularization, we improve the score of each sample leading to clearly visible discriminative features for even classes that show low performance. Note that the scales are different for both the heatmaps, meaning that feature magnitudes are much higher after regularization.
In Table \ref{tab:acc_baseline_vs_reg}, we also tabulate the AUROC and AUPRC of how effective Q-Score is in differentiating between correctly and incorrectly classified samples. We observe a drop in both the metrics, implying that the regularization works. The Q-Score regularization, improves the quality of wrongly classified samples, therefore, making it harder to distinguish between correct and incorrect classifications in an unsupervised manner.
% \maz{Maybe we should note that there seems to still be a lot of room for improvement based on the power of Q-score in detecting falsely classified samples even after regularization.}
% \KN{which score are you talking about here? and what is its connection to the second part of the sentence?}.
\subsubsection{Interpretability}
We observed in Figure \ref{fig:correct_vs_incorrect_latent}, that representations are \textit{nearly} sparse, i.e., most features are close to zero. These features are noisy and do not generally contribute to an informative attribute in the sample. We confirm this in Figure \ref{fig:grad_cam_noisy}, where noisy features show spurious and irrelevant heatmaps in images. In Figure \ref{fig:sparsity}, we plot the actual sparsity (\% of zeros) of each of the representations of the baseline model and the Q-score regularized model. We observe that the average sparsity increases from $35\%$ to $52\%$.
With latent representations being more sparse, we reduce a lot of feature noise. In Figure \ref{fig:grad_cam_noisy_reg}, we visualize heatmaps of several samples of 4 classes in the Q-Score regularized model. The discriminative features of each class activate relevant attributes for each example. However, the noisy features (features that are close to 0), do not get activated for the majority of the examples. This shows that applying Q-score regularization over representations to improve sparsity, reduces noisy features to a large extent, making representations more interpretable.
% \maz{We might not have time for ICML but One more thing we can try is to do adversarial attack on the model but instead of using the features as is, use them with a sparsity filter. I think in this case, the regularized model should work much better as its original features are way sparser.}
\begin{figure}[h]
\centering
\subfigure{\includegraphics[width=0.23\textwidth, trim={1cm 1cm 0.8cm
0.9cm}, clip]{figures/imagenet100_sorted_features.pdf}}
\subfigure{\includegraphics[width=0.23\textwidth, trim={1cm 1cm 0.8cm
0.9cm}, clip]{figures/imagenet100_reg_sorted_features.pdf}}
\caption{\textbf{Class-averaged representations sorted by feature magnitude.} In these heatmaps, we plot the class-averaged representations (rows) where each representation is sorted by its features (columns) magnitude. We observe that in SimCLR (left), the representations of higher class accuracy have significant (highly-deviant) features whereas classes of lower accuracy do not. In Q-Score regularized SimCLR, we observe that all classes (higher and lower accuracy) have significant features.}
\label{fig:sorted_features}
\end{figure}
\begin{figure*}[h]
\centering
\includegraphics[width=0.78\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/grad_cam_noisy_reg.png}
\caption{\textbf{Heatmaps of significant and noisy features of SimCLR (Q-Score Regularized):} We observe the heatmaps of significant and noisy features of 4 classes with Q-score regularized SimCLR. We observe that significant features thoroughly capture meaningful attributes of each image. The noisy features (unlike Figure \ref{fig:grad_cam_noisy}), do not highlight images in most cases indicating that the regularization reduces noise in the representation space, thereby improving model interpretability.}
\label{fig:grad_cam_noisy_reg}
\end{figure*}
\begin{table}[]
\centering
\resizebox{0.45\textwidth}{!}{
\begin{tabular}{c|c|c}
\toprule
\multirow{2}{*}{\textbf{Metric}} & \textbf{SimCLR} & \textbf{SimCLR} \\
& \textbf{(Baseline)} & \textbf{(Q-Score Regularized)} \\
\midrule
\midrule
Accuracy & 71.71 & \textbf{74.05} \\
Q-Score (AUROC) & 0.74 & 0.70 \\
Q-Score (AUPRC) & 0.90 & 0.86 \\
\bottomrule
\end{tabular}
}
\caption{\textbf{Comparing Accuracy and Q-Score between SimCLR and SimCLR (Q-Score Regularized):} We observe that the test accuracy of ImageNet-100 shows a $3.26\%$ relative improvement with Q-Score regularization. The AUROC and AUPRC of Q-Score in differentiating between correct and incorrect classifications drops, indicating that the regularization is effective in improving Q-Score of bad-quality samples. }
% \maz{train accuracy improvement should be 4.32? why train is lower than test? what does train and test mean here?}}
\label{tab:acc_baseline_vs_reg}
\end{table}
\begin{figure}[h]
\centering
\includegraphics[width=0.35\textwidth, trim={1cm 1cm 0.5cm
0.5cm}, clip]{figures/sparsity.pdf}
\caption{\textbf{Sparsity of representations:} In this scatter plot, we show that Q-Score regularization significantly increases the sparsity of representations compared to the SimCLR baseline.}
\label{fig:sparsity}
\end{figure}
\section{Conclusion}
In this paper, we studied failure modes of self-supervised models in downstream classification tasks and discovered significant differences in feature representations between correctly and incorrectly classified samples. Building on these observations, we defined quality metrics, including Self-Supervised Quality Score (Q-Score) that are effective in determining how likely samples are to be correctly or incorrectly classified. Our proposed score can be computed per sample in an unsupervised manner (without the label information). Thus, it can be used to identify class-specific patterns in the representation space of self-supervised models associated with their failure modes. Moreover, we can also identify low-quality representations that are likely to be misclassified. With the help of Q-Score regularization, we remedied these low-quality samples by improving their Q-Scores, thereby, improving the overall accuracy of the model. We also observed that regularization improves model interpretability by reducing feature noise and improving discriminative feature correlations with the ground truth. %In conclusion, our study provides a deeper understanding of why self-supervised models fail by exploring the learned representations.
%In this paper, we discover a significant difference in correct and incorrectly classified representations of self-supervised models. We define quality metrics, including Self-Supervised Quality Score (Q-Score) that are effective in determining how likely samples are to be correctly or incorrectly classified. The advantage of our metric, is that it is unsupervised, so we can identify class-specific patterns without requiring a linear classifier. Moreover, we can also identify bad-quality representations that are very likely to be misclassified. With the help of Q-Score regularization, we remedy these bad-quality samples by improving their Q-Scores, thereby, improving the overall accuracy by $3.26\%$. We also observe that regularization improves model interpretability by reducing feature noise and improving discriminative feature correlations to ground truth. In conclusion, our study provides a deeper understanding of why self-supervised models fail by exploring the learned representations.
\appendix
\onecolumn
\renewcommand\thefigure{\thesection.\arabic{figure}}
\setcounter{figure}{0}
\renewcommand\thetable{\thesection.\arabic{table}}
\setcounter{table}{0}
\section{Appendix}
\subsection{Q-Score Regularization Without Feature Regularization}
In Section \ref{sec:qscore}, we discuss that Q-Score regularization cannot be directly applied since it can lead to trivial solutions. In Figure \ref{fig:trivial}, we visualize an example of such a trivial solution. To prevent such unfavorable situations, we perform L1 norm regularization on features (columns) as well.
\input{tables_and_figures/trivial}
\subsection{Class-wise Accuracy}
In Figure \ref{fig:class_acc_diff}, we visualize the accuracy change in each class from the SimCLR baseline to Q-Score regularized SimCLR. We observe that $53$ out of $100$ classes show improvement in accuracy. The drops are seen in mostly animal classes, especially dog breeds, which are harder to classify.
\input{tables_and_figures/class_acc_diff}
\subsection{More Gradient Heatmaps of SimCLR}
\label{sec:more_grad_cams}
In Figures \ref{fig:more_grad_cams_ski_mask}, \ref{fig:more_grad_cams_park_bench}, \ref{fig:more_grad_cams_head_cabbage} and \ref{fig:more_grad_cams_dutch_oven}, we plot more heatmaps of discriminative and noisy features of SimCLR. We observe that, in correctly classified samples, discriminative features are highly correlated with the ground truth, whereas, in misclassifications, they are irrelevant. We also plot noisy features, to show that they map to spurious portions that do not contribute to useful information.
\input{tables_and_figures/more_grad_cams}
\begin{figure}[h]
\centering
\subfigure{\includegraphics[width=0.4\textwidth, trim={1cm 0cm 5cm
0cm}, clip]{figures/imagenet100_trivial.pdf}}
\caption{\textbf{Trivial features with Q-Score Regularization:} In this heatmap, we show that Q-Score regularization can lead to a trivial solution without applying L1 norms across features (columns).}
\label{fig:trivial}
\end{figure}
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width=\textwidth, trim={0cm 1cm 0.5cm
0.5cm}, clip]{figures/imagenet100_acc_diff.pdf}}
\caption{\textbf{Change in accuracy from SimCLR to SimCLR (Q-Score Regularized):} We plot the difference between accuracy of SimCLR (Q-Score Regularized) and SimCLR in different classes. We observe that we improve the accuracy in $53$ out of $100$ classes with regularization.}
\label{fig:class_acc_diff}
\end{figure*}
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/ski_mask_rgb.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/ski_mask_vis.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/ski_mask_vis_noise.png}}
\caption{\textbf{Heatmaps of Discriminative and Noisy Features of SimCLR (Class - Ski Mask)}: We plot the gradient heat maps of the most discriminative feature (by magnitude) of the given class and a noisy feature of the same class. We observe that discriminative features are more correlated with ground truth labels in correct classifications but are not correlated in misclassifications. Noisy features map to spurious portions of the images.}
\label{fig:more_grad_cams_ski_mask}
\end{figure*}
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/park_bench_rgb.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/park_bench_vis.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/park_bench_vis_noise.png}}
\caption{\textbf{Heatmaps of Discriminative and Noisy Features of SimCLR (Class - Park Bench)}: We plot the gradient heat maps of the most discriminative feature (by magnitude) of the given class and a noisy feature of the same class. We observe that discriminative features are more correlated with ground truth labels in correct classifications but are not correlated in misclassifications. Noisy features map to spurious portions of the images.}
\label{fig:more_grad_cams_park_bench}
\end{figure*}
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/head_cabbage_rgb.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/head_cabbage_vis.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/head_cabbage_vis_noise.png}}
\caption{\textbf{Heatmaps of Discriminative and Noisy Features of SimCLR (Class - Head Cabbage)}: We plot the gradient heat maps of the most discriminative feature (by magnitude) of the given class and a noisy feature of the same class. We observe that discriminative features are more correlated with ground truth labels in correct classifications but are not correlated in misclassifications. Noisy features map to spurious portions of the images.}
\label{fig:more_grad_cams_head_cabbage}
\end{figure*}
\begin{figure*}[h]
\centering
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/dutch_oven_rgb.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/dutch_oven_vis.png}}
\subfigure{\includegraphics[width = 0.8\textwidth, trim={0cm 0cm 0cm
0cm}, clip]{figures/dutch_oven_vis_noise.png}}
\caption{\textbf{Heatmaps of Discriminative and Noisy Features of SimCLR (Class - Dutch Oven)}: We plot the gradient heat maps of the most discriminative feature (by magnitude) of the given class and a noisy feature of the same class. We observe that discriminative features are more correlated with ground truth labels of correct classifications but are not correlated in misclassifications. Noisy features map to spurious portions of the images.}
\label{fig:more_grad_cams_dutch_oven}
\end{figure*}
\begin{thebibliography}{}
\bibitem[Anonymous, 2022]{anonymous2022understanding}
Anonymous (2022).
\newblock Understanding dimensional collapse in contrastive self-supervised
learning.
\newblock In {\em Submitted to The Tenth International Conference on Learning
Representations}.
\newblock under review.
\bibitem[Arora et~al., 2019]{arora2019theoretical}
Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. (2019).
\newblock A theoretical analysis of contrastive unsupervised representation
learning.
\bibitem[Bachman et~al., 2019]{NEURIPS2019_ddf35421}
Bachman, P., Hjelm, R.~D., and Buchwalter, W. (2019).
\newblock Learning representations by maximizing mutual information across
views.
\newblock In Wallach, H., Larochelle, H., Beygelzimer, A., d\textquotesingle
Alch\'{e}-Buc, F., Fox, E., and Garnett, R., editors, {\em Advances in Neural
Information Processing Systems}, volume~32. Curran Associates, Inc.
\bibitem[Bautista et~al., 2016]{NIPS2016_65fc52ed}
Bautista, M.~A., Sanakoyeu, A., Tikhoncheva, E., and Ommer, B. (2016).
\newblock Cliquecnn: Deep unsupervised exemplar learning.
\newblock In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R.,
editors, {\em Advances in Neural Information Processing Systems}, volume~29.
Curran Associates, Inc.
\bibitem[Bojanowski and Joulin, 2017]{pmlr-v70-bojanowski17a}
Bojanowski, P. and Joulin, A. (2017).
\newblock Unsupervised learning by predicting noise.
\newblock In Precup, D. and Teh, Y.~W., editors, {\em Proceedings of the 34th
International Conference on Machine Learning}, volume~70 of {\em Proceedings
of Machine Learning Research}, pages 517--526. PMLR.
\bibitem[Caron et~al., 2018a]{deepcluster}
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. (2018a).
\newblock Deep clustering for unsupervised learning of visual features.
\newblock {\em Lecture Notes in Computer Science}, page 139–156.
\bibitem[Caron et~al., 2018b]{Caron_2018_ECCV}
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. (2018b).
\newblock Deep clustering for unsupervised learning of visual features.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}.
\bibitem[Caron et~al., 2019]{Caron_2019_ICCV}
Caron, M., Bojanowski, P., Mairal, J., and Joulin, A. (2019).
\newblock Unsupervised pre-training of image features on non-curated data.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV)}.
\bibitem[Caron et~al., 2020]{swav}
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A.
(2020).
\newblock Unsupervised learning of visual features by contrasting cluster
assignments.
\newblock In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.~F., and Lin,
H., editors, {\em Advances in Neural Information Processing Systems},
volume~33, pages 9912--9924. Curran Associates, Inc.
\bibitem[Chen et~al., 2020a]{simclr}
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020a).
\newblock A simple framework for contrastive learning of visual
representations.
\newblock In III, H.~D. and Singh, A., editors, {\em Proceedings of the 37th
International Conference on Machine Learning}, volume 119 of {\em Proceedings
of Machine Learning Research}, pages 1597--1607. PMLR.
\bibitem[Chen et~al., 2020b]{mocov2}
Chen, X., Fan, H., Girshick, R., and He, K. (2020b).
\newblock Improved baselines with momentum contrastive learning.
\bibitem[Chen and He, 2021]{simsiam}
Chen, X. and He, K. (2021).
\newblock Exploring simple siamese representation learning.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR)}, pages 15750--15758.
\bibitem[Dosovitskiy et~al., 2016]{7312476}
Dosovitskiy, A., Fischer, P., Springenberg, J.~T., Riedmiller, M., and Brox, T.
(2016).
\newblock Discriminative unsupervised feature learning with exemplar
convolutional neural networks.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
38(9):1734--1747.
\bibitem[Dosovitskiy et~al., 2014]{NIPS2014_07563a3f}
Dosovitskiy, A., Springenberg, J.~T., Riedmiller, M., and Brox, T. (2014).
\newblock Discriminative unsupervised feature learning with convolutional
neural networks.
\newblock In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and
Weinberger, K.~Q., editors, {\em Advances in Neural Information Processing
Systems}, volume~27. Curran Associates, Inc.
\bibitem[Ericsson et~al., 2021]{ericsson2021selfsupervised}
Ericsson, L., Gouk, H., and Hospedales, T.~M. (2021).
\newblock Why do self-supervised models transfer? investigating the impact of
invariance on downstream tasks.
\bibitem[Grigg et~al., 2021]{grigg2021selfsupervised}
Grigg, T.~G., Busbridge, D., Ramapuram, J., and Webb, R. (2021).
\newblock Do self-supervised and supervised methods learn similar visual
representations?
\bibitem[Grill et~al., 2020]{byol}
Grill, J.-B., Strub, F., Altch\'{e}, F., Tallec, C., Richemond, P.,
Buchatskaya, E., Doersch, C., Avila~Pires, B., Guo, Z., Gheshlaghi~Azar, M.,
Piot, B., kavukcuoglu, k., Munos, R., and Valko, M. (2020).
\newblock Bootstrap your own latent - a new approach to self-supervised
learning.
\newblock In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.~F., and Lin,
H., editors, {\em Advances in Neural Information Processing Systems},
volume~33, pages 21271--21284. Curran Associates, Inc.
\bibitem[He et~al., 2016]{resnet}
He, K., Zhang, X., Ren, S., and Sun, J. (2016).
\newblock Deep residual learning for image recognition.
\newblock In {\em 2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, pages 770--778.
\bibitem[Huang et~al., 2019]{pmlr-v97-huang19b}
Huang, J., Dong, Q., Gong, S., and Zhu, X. (2019).
\newblock Unsupervised deep learning by neighbourhood discovery.
\newblock In Chaudhuri, K. and Salakhutdinov, R., editors, {\em Proceedings of
the 36th International Conference on Machine Learning}, volume~97 of {\em
Proceedings of Machine Learning Research}, pages 2849--2858. PMLR.
\bibitem[Huang et~al., 2021]{huang2021generalization}
Huang, W., Yi, M., and Zhao, X. (2021).
\newblock Towards the generalization of contrastive self-supervised learning.
\bibitem[Khosla et~al., 2020]{khosla2020supervised}
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot,
A., Liu, C., and Krishnan, D. (2020).
\newblock Supervised contrastive learning.
\bibitem[Russakovsky et~al., 2015]{imagenet}
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., Berg, A.~C., and Fei-Fei, L. (2015).
\newblock {ImageNet Large Scale Visual Recognition Challenge}.
\newblock {\em International Journal of Computer Vision (IJCV)},
115(3):211--252.
\bibitem[Tian et~al., 2020]{NEURIPS2020_4c2e5eaa}
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. (2020).
\newblock What makes for good views for contrastive learning?
\newblock In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.~F., and Lin,
H., editors, {\em Advances in Neural Information Processing Systems},
volume~33, pages 6827--6839. Curran Associates, Inc.
\bibitem[Tosh et~al., 2021]{tosh2021contrastive}
Tosh, C., Krishnamurthy, A., and Hsu, D. (2021).
\newblock Contrastive learning, multi-view redundancy, and linear models.
\bibitem[von K{\"u}gelgen* et~al., 2021]{Kugelenetal21}
von K{\"u}gelgen*, J., Sharma*, Y., Gresele*, L., Brendel, W., Sch{\"o}lkopf,
B., Besserve, M., and Locatello, F. (2021).
\newblock Self-supervised learning with data augmentations provably isolates
content from style.
\newblock In {\em Advances in Neural Information Processing Systems 34 (NeurIPS
2021)}.
\newblock *equal contribution.
\bibitem[Wu et~al., 2018]{Wu_2018_CVPR}
Wu, Z., Xiong, Y., Yu, S.~X., and Lin, D. (2018).
\newblock Unsupervised feature learning via non-parametric instance
discrimination.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}.
\bibitem[Xiao et~al., 2021]{xiao2021what}
Xiao, T., Wang, X., Efros, A.~A., and Darrell, T. (2021).
\newblock What should not be contrastive in contrastive learning.
\newblock In {\em International Conference on Learning Representations}.
\bibitem[YM. et~al., 2020]{YM.2020Self-labelling}
YM., A., C., R., and A., V. (2020).
\newblock Self-labelling via simultaneous clustering and representation
learning.
\newblock In {\em International Conference on Learning Representations}.
\bibitem[You et~al., 2017]{You2017LargeBT}
You, Y., Gitman, I., and Ginsburg, B. (2017).
\newblock Large batch training of convolutional networks.
\newblock {\em arXiv: Computer Vision and Pattern Recognition}.
\end{thebibliography}