Augmentation Component Analysis: Modeling Similarity via the Augmentation Overlaps
Extracted source text · arXiv 2206.00471
\documentclass{article} % For LaTeX2e
\usepackage{iclr2023_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref} % hyperlinks
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{url}
\usepackage{xspace}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{multirow}
\usepackage{makecell}
%\usepackage{ulem}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2022} with \usepackage[nohyperref]{icml2022} above.
\usepackage{wrapfig}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{enumitem}
\usepackage{algorithm}
\usepackage{algorithmic}
\title{Augmentation Component Analysis:\\ Modeling Similarity via the Augmentation Overlaps}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Lu Han, Han-Jia Ye, De-Chuan Zhan \\
%\thanks{ Use footnote for providing further information about author (webpage, alternative address)---\emph{not} for acknowledging funding agencies. Funding acknowledgements go at the end of the paper.} \\
State Key Laboratory for Novel Software Technology, Nanjing University\\
\texttt{\{hanlu,yehj\}@lamda.nju.edu.cn, zhandc@nju.edu.cn}
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\def\Eg{\emph{E.g.}\xspace}
\def\eg{\emph{e.g.}\xspace}
\def\ie{\emph{i.e.}\xspace}
\def\Ie{\emph{I.e.}\xspace}
\newcommand{\cs}[1]{\texttt{\char`\\#1}}
\newcommand{\med}{\mathop{\textrm{med}}}
\newcommand{\x}{{\boldsymbol{x} }}
\newcommand{\w}{{\boldsymbol{w}}}
\newcommand{\bcX}{\bar{\mathcal{X}}}
\newcommand{\bx}{\bar{\x}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\aaa}{{\bf a}}
\newcommand{\bb}{{\bf b}}
\newcommand{\s}{{\bf s}}
\newcommand{\p}{{\bf p}}
\newcommand{\bml}{{\bm l}}
\newcommand{\cc}{{\bf c}}
\newcommand{\e}{{\bf e}}
\newcommand{\z}{{\bf z}}
\newcommand{\h}{{\bf h}}
\newcommand{\oo}{{\bf o}}
\newcommand{\m}{{\bf m}}
\newcommand{\y}{{\bf y}}
\newcommand{\A}{\mathcal{A}}
\newcommand{\ud}{\mathrm{d}}
\newcommand{\Y}{\mathcal{Y}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\N}{\mathcal{N}}
\newcommand{\bmu}{\boldsymbol{\mu}}
\newcommand{\cL}{\mathcal{L}}
\newcommand{\expect}{\mathbb{E}}
\newcommand{\mathI}{\mathcal{I}}
\newcommand{\ta}{$\mathcal{T}_{\rm Ref}$}
\newcommand{\defeq}{\stackrel{\text{def}}{=}}
\newcommand{\cev}[1]{\reflectbox{\ensuremath{\vec{\reflectbox{\ensuremath{#1}}}}}}
\newcommand{\HL}[1]{{\textbf{\color{blue} (Lu: #1)}}}
\newcommand{\appallingunderline}[1]{%
\underline{\smash{#1}\vphantom{T}}\vphantom{#1}%
}
\usepackage[capitalize,noabbrev]{cleveref}
\def\month{02} % Insert correct month for camera-ready version
\def\year{2023} % Insert correct year for camera-ready version
\def\openreview{\url{https://openreview.net/forum?id=XXXX}} % Insert correct link to OpenReview for camera-ready version
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
Self-supervised learning aims to learn a embedding space where semantically similar samples are close. Contrastive learning methods pull views of samples together and push different samples away, which utilizes semantic invariance of augmentation but ignores the relationship between samples. To better exploit the power of augmentation, we observe that semantically similar samples are more likely to have similar augmented views. Therefore, we can take the augmented views as a special description of a sample. In this paper, we model such a description as the augmentation distribution and we call it augmentation feature. The similarity in augmentation feature reflects how much the views of two samples overlap and is related to their semantical similarity.
%So the augmentation feature, composed of the distribution of augmentations, can act as the ideal embedding, and similarity over them reveals how much the augmentations of two samples overlap.
Without computational burdens to explicitly estimate values of the augmentation feature, we propose Augmentation Component Analysis (ACA) with a contrastive-like loss to learn principal components
and an on-the-fly projection loss to embed data. ACA equals an efficient dimension reduction by PCA and extracts low-dimensional embeddings, theoretically preserving the similarity of augmentation distribution between samples.
Empirical results show our method can achieve competitive results against various traditional contrastive learning methods on different benchmarks. Code available at \url{https://github.com/hanlu-nju/AugCA}.
\end{abstract}
\vspace{-2mm}
\section{Introduction}
\label{sec:intro}
\vspace{-2mm}
The rapid development of contrastive learning has pushed self-supervised representation learning to unprecedented success. Many contrastive learning methods surpass traditional pretext-based methods by a large margin and even outperform representation learned by supervised learning~\citep{wu2018unsupervised,oord2018representation,Tian20Contrastive,he2020momentum,Chen20Simple,Chen20Improved}.
The key idea of self-supervised contrastive learning is to construct views of samples via modern data augmentations~\citep{Chen20Simple}. Then discriminative embeddings are learned by pulling together views of the same sample in the embedding space while pushing apart views of others.
Contrastive learning methods utilize the semantic invariance between views of the same sample, but the semantic relationship between samples is ignored.
%and ``false negative'' samples~\citep{Chuang20Debiased} are pushed away.
%For example, two crops of the same deer image, one with an antler and another containing a forest background, are treated as similar, while two antler crops in different deer images are labeled dissimilar.
Instead of measuring the similarity between certain augmented views of samples,
we claim that the similarity between the {\em augmentation distributions} of samples can reveal the sample-wise similarity better. In another word, semantically similar samples have similar sets of views.
%we claim that it is the similarity between the {\em augmentation distributions} of samples, \ie, the set of all possible images to be augmented to, that reveals the sample-wise similarity better.
As is shown in~\cref{fig:motivation} left, two images of deer create many similar crops, and sets of their augmentation results, \ie, their distributions, overlap much. In contrast, a car image will rarely be augmented to the same crop as a deer, and their augmentation distributions overlap little.
In~\cref{fig:motivation} right, we verify the motivation numerically. We approximate the overlaps between image augmentations with a classical image matching algorithm~\citep{zitova2003image}, which counts the portion of matched key points of raw images. We find samples of the same class overlap more than different classes on average, supporting our motivation.
Therefore, we establish the semantic relationship between samples in an unsupervised manner based on the similarity of augmentation distributions, \ie, how much they overlap.
% We observe that modern augmentation methods tend to have similar augmentation results if two samples are semantically similar.
\begin{figure}[t]
\hspace{2mm}
\includegraphics[width=0.5\linewidth]{figures/motivation}
\includegraphics[width=0.45\linewidth]{figures/figure_overlap_bar.pdf}
\vspace{-2mm}
\caption{Left: semantically similar samples (\eg, those in the same class) usually create similar augmentations.
The right figure indicates the same class images have higher averaged augmentation overlaps than those from different classes on four common datasets.
For this reason, we learn embeddings by preserving the similarity between augmentation distributions of samples.}
\vspace{-2mm}
\label{fig:motivation}
\end{figure}
% The augmentation overlap naturally acts as the self-supervision to learn embeddings.
% Following this idea, we can manually construct an ``ideal'' (but possibly infinite-dimensional) \textit{augmentation feature} for a sample --- whose elements reveal the probability of augmenting that sample to another and encode the augmentation information comprehensively.
In this paper, we propose to describe data directly by their augmentation distributions. We call the feature of this kind the \textit{augmentation feature}. The elements of the augmentation feature represent the probability of getting a certain view by augmenting the sample as shown in the left of \cref{fig:method}. The augmentation feature serves as an ``ideal'' representation since it encodes the augmentation information without any loss and we can easily obtain the overlap of two samples from it.
%It seems that we can obtain the solution directly without further learning.
However, not only its elements are hard to calculate, but also such high-dimensional embeddings are impractical to use.
Inspired by the classical strategy to deal with high-dimensional data, we propose Augmentation Component Analysis (ACA), which employs the idea of PCA~\citep{hotelling1933analysis} to perform dimension reduction on augmentation features previously mentioned.
ACA reformulates the steps of extracting principal components of the augmentation features with a contrastive-like loss. With the learned principal components, another on-the-fly loss embeds samples effectively. ACA learns operable low-dimensional embeddings theoretically preserving the augmentation distribution distances.
In addition, the resemblance between the objectives of ACA and traditional contrastive loss may explain why the contrastive learning can learn semantic-related embeddings -- they embed samples into spaces that partially preserve augmentation distributions.
Experiments on synthetic and real-world datasets demonstrate that our ACA achieves competitive results against various traditional contrastive learning methods.
%In addition, we conduct experiments to reveal the requirements for a good augmentation method, which validates our motivation.
Our contributions are:
\begin{itemize}[leftmargin=6mm, itemsep=0mm]
\vspace{-2mm}
\item We propose a new self-supervised strategy, which measures the sample-wise similarity via the similarity of augmentation distributions. This new aspect facilitates learning embeddings.
\item We propose ACA method that implicitly employs the dimension reduction over the augmentation feature, and the learned embeddings preserve augmentation similarity between samples.
\item Benefiting from the resemblance to the contrastive loss, our ACA helps explain the functionality of contrastive learning and why they can learn semantically meaningful embeddings.
% \item We empirically show two requirements for a good augmentation: (1) existence of overlap among samples; (2) intra-class samples overlap more than inter-class.
\vspace{-2mm}
\end{itemize}
\section{Related Work}
\paragraph{Self-Supervised Learning.} Learning effective visual representations without human
supervision is a long-standing problem. Self-supervised learning methods solve this problem by creating supervision from the data itself instead of human labelers. The model needs to solve a pretext task before it is used for the downstream tasks. For example, in computer vision, the pretext tasks include colorizing grayscale images~\citep{Zhang16Colorful}, inpainting images~\citep{Pathak2016Context}, predicting relative patch~\citep{Doersch2015Unsupervised}, solving jigsaw puzzles~\citep{Noroozi2016Unsupervised}, predicting rotations~\citep{Gidaris2018Unsupervised} and exploiting generative models~\citep{Goodfellow14Generative,Kingma13Auto,Donahue19Large}. Self-supervised learning also achieves great success in natural language processing~\citep{Mikolov13Distributed,Devlin19BERT}.
\vspace{-2mm}
\paragraph{Contrastive Learning and Non-Contrastive Methods.} Contrastive approaches have been one of the most prominent representation learning strategies in self-supervised learning. The idea behind these approaches is to maximize the agreement between positive pairs and minimize the agreement between negative pairs. Positive pairs are commonly constructed by co-occurrence~\citep{oord2018representation,Tian20Contrastive,Bachman19Learning} or augmentation of the same sample~\citep{he2020momentum,Chen20Simple,Chen20Improved,Li21Prototypical}, while all the other samples are taken as negatives. Most of these methods employ the InfoNCE loss~\citep{oord2018representation}, which acts as a lower bound of mutual information between views. Based on this idea, there are several methods that attempts to improve contrastive learning, including mining nearest neighbour~\citep{Dwibedi2021Little,Koohpayegani2021Mean,Azabou2021Mine} and creating extra views by mixing up~\citep{kalantidis2020hard} or adversarial training~\citep{hu2021adco}. Another stream of methods employ the similar idea of contrastive learning to pull views of a sample together without using negative samples~\citep{Grill20Bootstrap,Chen21Exploring}. Barlow Twins~\citep{zbontar2021barlow} minimizes the redundancy within the representation vector. \citet{Tsai2021Note} reveals the relationship among Barlow Twins, contrastive and non-contrastive methods.
%Some methods do not involve negative samples~\citep{Grill20Bootstrap,Chen21Exploring}.
Most of these methods only utilize the semantic invariance of augmentation and ignore the relationship between samples.
%their learned embeddings somehow form semantically meaningful clusters~\citep{he2020momentum,Chen20Simple,Chen20Improved}.
%Some of the previous works explain this phenomenon with the strong assumption that two views are conditionally independent given the label or a hidden variable~\citep{Saunshi19Theoretical,Tosh21Contrastive}.
%\citet{Chen2021Provable} turn the assumption towards how views of different samples are connected by the augmentation process. \citet{Wang2022Chaos} notice that the augmentation overlap provides a ladder for gradually learning class-separated representations.
%These analyses motivate us to rethink the role of augmentation and emphasize augmentation distribution as a better self-supervised way of exploiting semantic similarity.
Different from them, we propose a new way to perform self-supervised learning by perserving the similarity of augmentation distribution, based on the observation that a strong correlation exists between the similarity of {\em augmentation distributions} and the similarity of semantics.
%between samples
\vspace{-2mm}
\paragraph{Explanation of Contrastive Learning.} Several works provide empirical or theoretical results for explaining the behavior of contrastive learning.~\citet{Tian20What,Xiao21What} explore the role of augmentation and show contrastive model can extract useful information from views but also can be affected by nuisance information.~\citet{Zhao21What} empirically shows that contrastive learning preserves low-level or middle-level instance information. In theoretical studies, ~\citet{Saunshi19Theoretical} provide guarantees of downstream linear classification tasks under conditionally independence assumption. Other works weaken the assumption
but are still unrealistic~\citep{Lee21Predicting,Tosh21Contrastive}. ~\citet{Chen2021Provable} focus on how views of different samples are connected by the augmentation process and provide guarantees with certain connectivity assumptions. \citet{Wang2022Chaos} notice that the augmentation overlap provides a ladder for gradually learning class-separated representations.
In addition to the alignment and uniformity as shown by \citet{Wang20Understanding}, \citet{Huang21Generalization} develop theories on the crucial effect of data augmentation on the generalization of contrastive learning. \citet{Hu2022Your} explain that the contrastive loss is implicitly doing SNE with ``positive'' pairs constructed from data augmentation. Being aware of the important role of augmentation, we provide a novel contrastive method that ensures preserving augmentation overlap.
%However, ~\citet{Wang2022Chaos} provides their analysis based on the assumption that common contrastive loss can somehow learn to align overlapped samples. In contrast, we provide a novel contrastive method that ensures preserving augmentation overlap.
% \paragraph{Classical Dimension Reduction.} Several works provide
\vspace{-2mm}
\section{Notations}
\vspace{-2mm}
The set of all the natural data (the data without augmentation) is denoted by $\bcX$, with size $\lvert \bcX \rvert = N$. We assume the natural data follows a uniform distribution $p(\bx)$ on $\bcX$, \ie, $p(\bx)=\frac{1}{N}, \forall \bx \in \bcX$.
By applying an augmentation method $\A$, a natural sample $\bx \in \bcX$ could be augmented to another sample $\x$ with probability $p_{\A}(\x\mid\bx)$, so we use $p(\cdot\mid\bx)$ to encode the augmentation distribution.
\footnote{Note that $p(\cdot\mid\bx)$ is usually difficult to compute and we can only sample from it. We omit the subscript $\A$ and directly use $p(\cdot\mid\bx)$ in the following content for convenient}
For example, if $\bx$ is an image, then $\A$ can be common augmentations like Gaussian blur, color distortion and random cropping~\citep{Chen20Simple}.
Denote the set of all possible augmented data as $\X$. We assume $\X$ has finite size $\lvert \X \rvert=L$ and $L > N$ for ease of exposition. Note that $N$ and $L$ are finite but they can be arbitrarily large. We denote the encoder as $f_\theta$, parameterized by $\theta$, which projects a sample $\x$ to an embedding vector in $\R^k$.
% Self-Supervised Representation Learning (SSL) extracts features with an encoder $f_\theta$, parameterized by $\theta$, which projects a sample $\x$ to an embedding vector in $\R^k$.
% By training on a pretext task, SSL aims to obtain an encoder that provides semantically meaningful representations of any sample and performs well on various downstream tasks.
% The most popular way of constructing pretext tasks is to utilize the information provided by data augmentation and contrastive learning belongs to this stream.
% Traditional contrastive learning tasks use augmentation to create views, taking advantage of its semantic invariance property. They pull views of the same samples together and push away views of different samples.
% %Assume $\x^+$ is the positive view augmented by $\A$ from $\x$ and $\x^-$ denote a view from different sample.
% A common practice is to use a contrastive loss, \eg, the InfoNCE loss~\citep{oord2018representation}.
% %\begin{equation}
% % \label{eq:infonce}
% % \setlength{\abovedisplayskip}{3pt}
% % \cL_\text{InfoNCE} = - \expect_{\x,\x^+} f(\x)^\top f(\x^+) + \expect_{\x,\{\x^-\}} \log \sum_{\x^-} e^{f(\x)^\top f(\x^-)} \;.
% % \setlength{\belowdisplayskip}{3pt}
% %\end{equation}
% %The expectations over $\x$ and $\x^-$ in \cref{eq:infonce} follows the uniform distribution, and we sample $\vx^+$ follows $p(\vx^+\mid\x)$.
% %The similarity between views of samples is measured based on augmentation, which tends to make two semantically similar pairs $\x$ and $\x^-$ away from each other and two semantically dissimilar augmentations $\x$ and $\x^+$ close.
% However, utilizing the semantic invariance property of augmentation only exploits the relationship between views of the same sample, and ignores the semantic relationships between samples.
% Different from them, we propose a new way to perform contrastive learning with the observation that a strong correlation exists between the similarity of {\em augmentation distributions} and the similarity of semantics.
\vspace{-2mm}
\section{Learning via Augmentation Overlaps}
\vspace{-2mm}
As we mentioned in~\cref{sec:intro}, measuring the similarity between augmentation distributions, \ie, the overlap of all two sample's augmented results, reveals their {\em semantic} relationship well.
For example, in natural language processing, we usually generate augmented sentences by dropping out some words. Then different sentences with similar meanings are likely to contain the same set of words and thus have a high probability of creating similar augmented data.
With the help of this self-supervision, \appallingunderline{we formulate the embedding learning task to meet the following similarity preserving condition}:
\begin{equation}
\label{eq:framework}
\setlength\abovedisplayskip{3pt}
\setlength\belowdisplayskip{3pt}
\operatorname{d_{\R^k}} \left(f_{\theta^\star}\left(\bx_1\right),\; f_{\theta^\star}\left(\bx_2\right)\right) \propto \operatorname{d_{\A}}(p(\cdot\mid\bx_1),\;
p(\cdot\mid\bx_2))\;.
\end{equation}
$\operatorname{d_{\R^k}}$ is a distance measure in the embedding space $\R^{k}$, and $\operatorname{d_{\A}}$ measures the distance between two augmentation distributions.
\cref{eq:framework} requires the learned embedding with the optimal parameter $\theta^\star$ has the {\em same similarity comparison} with that measured by the augmentation distributions.
In this section, we first introduce the {\em augmentation feature} for each sample, \textit{which is a manually designed embedding satisfying the condition in \cref{eq:framework}}. To handle the high dimensionality and complexity of the augmentation feature, we further propose our Augmentation Component Analysis (ACA) that learns to reduce the dimensionality and preserve the similarity.
\vspace{-2mm}
\subsection{Augmentation Feature}
\label{subsec:augmentatoin_feature}
To reach the goal of similarity preserving in~\cref{eq:framework}, a direct way is to manually construct the feature by the augmentation distributions of each natural sample, \ie, $f(\bx) = [p(\x_1\mid\bx),\ldots,p(\x_L\mid\bx)]^\top$, where each element $p(\x_i\mid\bx)$ represents the probability of getting a certain element $\x_i$ in space $\mathcal{X}$ by augmenting $\bx$.
We omit $\theta$ in $f(\bx)$ since such \textit{augmentation feature}\footnote{Following the common knowledge in dimension reduction, we call the raw high dimensional representation as ``feature'', and learned low-dimensional representation as ``embedding''.} does not rely on any learnable parameters.
In this case, any distance $\operatorname{d_{\R^{L}}}$ defined in space of $f$ is exactly a valid distribution distance, which reveals the augmentation overlaps and is related to the semantic similarity.
Though the constructive augmentation feature naturally satisfies the similarity preserving condition (\cref{eq:framework}) (because it directly use the augmentation distribution without loss of information), it is impractical for the following reasons.
First, \appallingunderline{its dimensionality is exponentially high}, which is up to $L$, the number of possible augmented results. For example, even on CIFAR10, the small-scale dataset with image size $32\times32\times3$, $L$ is up to $256^{3072}$ (3072 pixels and 256 possible pixel values).
Second, \appallingunderline{the computation of each element is intractable}. We may need an exponentially large number of samples to accurately estimate each $p(\x\mid\bx)$. The dimensionality and computation problems make the augmentation feature impractical both at inference and training time. Such inconvenience motivates us to (1) conduct certain dimension reduction to preserve the information in low dimensional space (\cref{subsec:DR_Augmentation}) and (2) develop an efficient algorithm for dimension reduction (\cref{subsec:ACA}).
% In this paper, we propose Augmentation Component Analysis (ACA) that can efficiently perform dimension reduction on a transformed version of augmentation feature.
\vspace{-2mm}
\subsection{Dimension Reduction on Augmentation Features}
\label{subsec:DR_Augmentation}
\begin{figure}[t]
\includegraphics[width=0.95\linewidth]{figures/method}
\vspace{-2mm}
\caption{The idea of learning embeddings via Augmentation Component Analysis (ACA). The upper right figure demonstrates the process of PCA. It learns PCs and projects the input feature to get embeddings of data. Similarly, ACA performs PCA on the augmentation feature, which encodes all the information about the augmentation distribution. To overcome the dimensional and computational complexity, ACA employs ACA-PC loss and projection loss to tractably learn PCs and embeddings. Via ACA, our model can learn embeddings that preserve augmentation similarity for natural data.}
\label{fig:method}
\vspace{-2mm}
\end{figure}
To deal with the high-dimensional property,
%of augmentation features.
we employ the idea of PCA~\citep{hotelling1933analysis}, which reconstructs the data with principal components.\footnote{In this paper, we use the non-centered version~\citep{reyment1996applied}, which is more appropriate for observations than for variables, where the origin matters more.
}
For convenience, we denote the design matrix of augmentation feature by $A$, where $A\in \R^{N\times L}$, $A_{\bx,\x} = p(\x\mid\bx)$ (see~\cref{fig:method}). We perform PCA on a transformed augmentation feature called \textit{normalized augmentation feature}:
\begin{equation}
\setlength\abovedisplayskip{3pt}
\setlength\belowdisplayskip{3pt}
\hat{A} = A D^{-\frac{1}{2}},
\end{equation}
where $D = \operatorname{diag}([d_{\x_1},d_{\x_2}, \ldots , d_{\x_L}])$, $d_\x = \sum_{\bx} p(\x\mid\bx)$.
%Using the transformed augmentation feature leads to two benefits. First, $d_{\x}$ measures the relative frequent of getting each augmented data when we augment randomly sampled natural data. The background is usually ubiquitous and unimportant, with large $d_\x$. Normalized augmentation feature will down-weight the elements at augments from background. So it focuses on the main objects, which contribute more with small $d_\x$.
%More importantly,
Based on normalized augmentation feature, we can develop an efficient algorithm for similarity preserving embeddings.
Assume the SVD of $\hat{A} = U\Sigma V^\top$ with $U\in \R^{N\times N}$, $\Sigma \in \R^{N\times L}$, $V\in \R^{L \times L}$ , PCA first learns the projection matrix consisting of the top-$k$ right singular vectors, which can be denoted as $\tilde{V}\in \R^{L \times k}$. The vectors in $\tilde{V}$ are called Principal Components (PCs). Then, it projects the feature by $\hat{A}\tilde{V}$ to get the embeddings for each sample. The overall procedure is illustrated at the top-right of \cref{fig:method}. But performing PCA on the augmentation feature will encounter many obstacles. The element of augmentation feature is not possible to accurately estimate, not to mention its high dimensionality. Even if we can somehow get the projection matrix $\tilde{V}$, it is also impractical to project high dimensional matrix $\hat{A}$. For this reason, \textit{we propose ACA to make PC learning and projection process efficient without explicitly calculating elements of augmentation feature}.
%All of the obstacles lie in the high dimensionality of augmentation distribution.
\vspace{-2mm}
\subsection{Augmentation Component Analysis}
\label{subsec:ACA}
\vspace{-1mm}
Although there are several obstacles when performing PCA on the augmentation features directly, fortunately, \appallingunderline{it is efficient to sample from the augmentation distribution $p(\x \mid \bx)$}, \ie, by performing augmentation on the natural data $\bx$ and get an augmented sample $\x$. Being aware of this, our ACA uses two practical losses to simulate the PCA process efficiently by sampling. The first contrastive-like loss leads the encoder to learn principal components of $\hat{A}$, which can be efficiently optimized by sampling like traditional contrastive methods. The second loss performs on-the-fly projection of $\hat{A}$ through the training trajectory, which solves the difficulty of high dimensional projection.
\vspace{-2mm}
\paragraph{Learning principal components.} ACA learns the principal components by an efficient contrastive-like loss. Besides its projection functionality, these learned principal components can also serve as embeddings that preserve a kind of posterior distribution similarity, as we will show later.
In the SVD view,
$U\Sigma$ serves as the PCA projection results for samples and $V$ contains the principal components~\citep{jolliffe2002principal}. However, if changing our view, $V\Sigma$ can be seen as the representation of each column. Since each column of $\hat{A}$ encodes probability on augmented data given natural data, $V\Sigma$
%can serve as the embeddings for augmented data and also
preserves certain augmentation relationships, as we will show in~\cref{prop:isometry} later.
To leverage the extrapolation power of encoders like deep neural networks,
we choose to design a loss that can guide the parameterized encoder $f_\theta$ to learn similar embeddings as PCA. Inspired by the rank minimization view of PCA~\citep{Vidal16Generalized}, we employ the low-rank approximation objective with matrix factorization, similar to ~\citet{Chen2021Provable}:
\begin{equation}
\vspace{-1mm}
\label{eq:Lmf}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
\min_{F\in \R^{L\times k}} \mathcal{L}_{mf} = \lVert \hat{A}^\top\hat{A} - FF^\top \rVert^2_F\;,
\end{equation}
where columns of $F$ store the scaled version of top-$k$ right singular vectors, and each row can be seen as the embedding of augmented data as will show in~\cref{lemma:ACAa}. According to Eckart–Young–Mirsky theorem~\citep{Eckart1936Approximation}, by optimizing $\mathcal{L}_{mf}$, we can get the optimal $\hat{F}$, which has the form $\tilde{V}\tilde{\Sigma}Q$, $Q\in \R^{k\times k}$ is an orthonormal matrix. $\tilde{\Sigma}$ and $\tilde{V}$ contains the top-$k$ singular values and right singular vectors. By expanding \cref{eq:Lmf}, we get Augmentation Component Analysis Loss for learning Principal Components (ACA-PC) in the following lemma:
\begin{lemma}[ACA-PC loss]
\label{lemma:ACAa}
Let $F_{\x,:} = \sqrt{d_{\x}} f^\top_\theta(\x),\forall \x\in \X$. Minimizing $\mathcal{L}_{mf}$ is equivalent to minimizing the following objective:
\begin{equation}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
\begin{aligned}
\cL_\text{ACA-PC} = &-2\expect_{\bx \sim p(\bx),\substack{\substack{\x_i\sim p(\x_i \mid \bx) \\
\x_j \sim p\left(\x_j \mid \bx\right)}}} f_\theta(\x_i)^\top f_\theta(\x_j)\\
&+ N\expect_{\x_1\sim p_\A(\x_1),\x_2\sim p_\A(\x_2)}\left[\left(f_\theta(\x_1)^\top f_\theta(\x_2)\right)^2\right].
\end{aligned}
\label{eq:ACAa}
\end{equation}
\end{lemma}
The proof can be found in~\cref{sec:proof_ACAa}.
In ACA-PC, the first term is the common alignment loss for augmented data and the second term is a form of uniformity loss ~\citep{Wang20Understanding}. Both of the two terms can be estimated by Monte-Carlo sampling. ACA-PC is a kind of contrastive loss. But unlike most of the others, it has theoretical meanings. We note that the form of ACA-PC differs from spectral loss~\citep{Chen2021Provable} by adding a constant $N$ before the uniformity term. This term is similar to the noise strength in NCE~\citep{Gutmann10Noise} or the number of negative samples in InfoNCE~\citep{oord2018representation}. It can be proved that the learned embeddings by ACA-PC preserve the posterior distribution distances between augmented data:
\begin{theorem}[Almost isometry for posterior distances]
\label{prop:isometry}
Assume $f_\theta$ is a universal encoder, $\sigma_{k+1}$ is the $(k+1)$-th largest singular value of $\hat{A}$, $d_{\min} = \min_\x d_\x$, and $\delta_{\x_1 \x_2} = \mathbb{I}(\x_1 = \x_2)$, \appallingunderline{the minimizer $\theta^*$ of $\mathcal{L}_{ACA-PC}$ satisfies}:
$$
\operatorname{d_{post}^2} (\x_1,\x_2) -\frac{2 \sigma_{k+1}^{2}}{d_{\min}}\left(1-\delta_{\x_1 \x_2}\right) \le \lVert f_{\theta^*}(\x_1) - f_{\theta^*}(\x_2) \rVert^2_2 \;\le\; \operatorname{d^2_{post}}(\x_1,\x_2)\;,\quad \forall \x_1,\x_2 \in \X
$$
where the posterior distance
\begin{equation}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
\operatorname{d_{post}^2} (\x_1,\x_2) = \sum_{\bx\in \bcX} \left(p_{\A}(\bx\mid\x_1)- p_{\A}(\bx\mid\x_2)\right)^{2}
\label{eq:post_dist}
\end{equation}
measures the squared Euclidean distance between the posterior distribution $p_{\A}(\bx\mid\x) = \frac{p(\x\mid\bx)p(\bx)}{p_\A(\x)}$.
\end{theorem}
\vspace{-2mm}
We give the proof in~\cref{sec:proof_isometry}.
\cref{prop:isometry} states that the optimal encoder for ACA-PC preserves the distance of posterior distributions between augmented data within an error related to embedding size $k$. As $k$ increase to $N$, the error decrease to $0$. It corresponds to the phenomenon that a larger embedding size leads to better contrastive performance~\citep{Chen20Simple}. The posterior distribution $p_\A(\bx\mid\x)$ represents the probability that a given augmented sample $\x$ is created by a natural sample $\bx$. Augmented data that are only produced by the same natural sample will have the smallest distance, and embeddings of those in overlapped areas will be pulled together by ACA-PC. Since the overlapped area are usually created by two same-class samples, ACA-PC can form semantically meaningful embedding space.
It is also noticeable that the optimal encoder meets the similarity preserving condition (\cref{eq:framework}) but concerning the posterior distribution for augmented data not the augmentation distribution for natural data. Since what we care about is the distribution of natural data, we further propose a projection loss that helps learn good embeddings for all the natural data.
\vspace{-2mm}
\paragraph{On-the-fly Projection.} As stated in the previous part, the learned embeddings by ACA-PC not only serve as certain embeddings for augmented data but also contain principal components of normalized augmentation feature. Based on this, we propose to use these embeddings to act as a projection operator to ensure meaningful embeddings for all the natural data. To be specific, denote the embedding matrix for all augmented data as $F^{aug}$($\in \R^{L\times k}$), where each row $F^{aug}_{\x,:}=f_{\theta^*}^\top(\x)$. From~\cref{eq:Lmf} and $\hat{F}_{\x,:} = \sqrt{d_{\x}} f^\top_{\theta^*}(\x)$, it can be easily seen that:
$$
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
F^{aug} =D^{-\frac{1}{2}} \hat{F} = D^{-\frac{1}{2}}\tilde{V}\tilde{\Sigma}Q
$$
Similar to PCA~\citep{hotelling1933analysis} that projects the original feature by the principal components $V$, we propose to use $F^{aug}$ to project the augmentation feature to get the embeddings for each natural sample. Denote the embedding matrix for natural data as $F^{nat}$($\in \R^{N\times k}$), where each row $F^{nat}_{\bx,:}$ represents the embeddings of $\bx$. We compute $F^{nat}$ as follows:
\begin{equation}
\label{eq:natural_embedding}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
F^{nat} = A F^{aug} = \hat{A}D^{\frac{1}{2}}D^{-\frac{1}{2}}\tilde{V}\tilde{\Sigma}Q =(\tilde{U} \tilde{\Sigma}) \tilde{\Sigma} Q,
\end{equation}
where $\tilde{\Sigma}$,$\tilde{U}$ contain the top-$k$ singular values and corresponding left singular vectors. It is noticeable that $F^{nat}$ is exactly the PCA projection result multiplied by an additional matrix $\tilde{\Sigma} Q$. Fortunately, such additional linear transformation does not affect the linear probe performance~\citep{Chen2021Provable}.
With~\cref{eq:natural_embedding}, the embedding of each natural sample can be computed as follows:
\vspace{-1mm}
\begin{equation}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
F^{nat}_{\bx,:} = A_{\bx,:}F^{aug} = \sum_{\x} p(\x\mid\bx) f^\top_{\theta^*}(\x) = \expect_{\x \sim p(\x\mid\bx)} f^\top_{\theta^*}(\x)
\label{eq:mean_feature}
\end{equation}
which is exactly the expected feature over the augmentation distribution. Similar to~\cref{prop:isometry}, the embeddings calculated by \cref{eq:mean_feature} also present a certain isometry property
%for the augmentation distribution distances
:
\begin{theorem}[Almost isometry for weighted augmentation distances]
\label{prop:isometry_natural}
Assume $f_\theta$ is a universal encoder, $\sigma_{k+1}$ is the $(k+1)$-th largest sigular value of $\hat{A}$,$\delta_{\bx_1 \bx_2} = \mathbb{I}(\bx_1 = \bx_2)$, let the minimizer of $\mathcal{L}_{ACA-PC}$ be $\theta^*$ and $g(\bx) = \expect_{\x \sim p(\x\mid\bx)} f_{\theta^*}(\x)$ as in~\cref{eq:mean_feature}, then:
$$
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
\operatorname{d_{w-aug}^2} (\bx_1,\bx_2) -2 \sigma_{k+1}^{2}\left(1-\delta_{\bx_1 \bx_2}\right) \le \lVert g(\bx_1) - g(\bx_2) \rVert^2_{\Sigma_k^{-2}} \;\le\; \operatorname{d^2_{w-aug}}(\bx_1,\bx_2)\;,\quad \forall \x_1,\x_2 \in \X
$$
where $\lVert \cdot \rVert_{\Sigma_k^{-2}}$ represent the Mahalanobis distance with matrix $\Sigma_k^{-2}$,$\Sigma_k = \operatorname{diag}([\sigma_1,\sigma_2,\ldots, \sigma_k])$ is the diagonal matrix containing top-$k$ sigular values and the weighted augmentation distance
\begin{equation}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
\operatorname{d_{w-aug}^2} (\bx_1,\bx_2) = \frac{1}{N} \sum_{\x\in \X} \frac{\left(p(\x\mid\bx_1)- p(\x\mid\bx_2)\right)^{2}}{p_\A(\x)}
\label{eq:w_aug_dist}
\end{equation}
measures the weighted squared Euclidean distance between the augmentation distribution $p(\x\mid\bx)$.
\end{theorem}
Different from~\cref{prop:isometry}, which presents isometry between Euclidean distances in embeddings and augmentation distribution, \cref{prop:isometry_natural} presents isometry between Mahalanobis distances. The weighted augmentation distances weight the Euclidean distances by $p_\A(\x)$. $\operatorname{d_{w-aug}}$ can be regarded as a valid augmentation distance measure $\operatorname{d_{\A}}$ as in~\cref{eq:framework} and $F^{nat}$ preserve such distance. So our goal is to make embeddings of $\bx$ approaches $\expect_{p(\x\mid\bx)} f_{\theta^\star}(\x)$. However, as stated before, the additional projection process is not efficient, \ie, we need exponentially many samples from $p(\x\mid\bx)$. We notice that samples during the training process of ACA-PC can be reused.
For this reason, we propose an on-the-fly projection loss that directly uses the current encoder for projection:
\begin{equation}
\label{eq:proj_loss}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
\cL_\text{proj} = \expect_{\bx\sim p(\bx)} \left[\lVert f_\theta(\bx) - \expect_{p(\x\mid\bx)} f_{\theta}(\x) \rVert^2_2 \right]
\end{equation}
\paragraph{Full objective of ACA.} Based on the discussion of the above parts, ACA simultaneously learns the principal components by ACA-PC and projects natural data by an on-the-fly projection loss. The full objective of ACA has the following form:
\begin{equation}
\label{eq:final_loss}
\setlength\abovedisplayskip{4pt}
\setlength\belowdisplayskip{4pt}
\cL_\text{ACA-Full} = \cL_\text{ACA-PC} + \alpha \cL_\text{proj}
\end{equation}
where $\alpha$ is a trade-off hyperparameter. We also find $N$ in~\cref{eq:ACAa} too large for stable training, so we replace it with a tunable hyperparameter $K$. Here, we only display the loss in expectation forms. Details of implementation are described in~\cref{app:impl}.
% In practice, we find the constant $N$ lead to numerical instability. So we turn the $N$ into a tunable noise strength $K$. We describe the empirical loss in~\cref{sec:empirical_loss}, but the algorithm is summarized here in~\cref{alg:model} for clarification.
\section{A Pilot Study}
\label{sec:synthetic_exp}
\vspace{-2mm}
In this section, we experiment with our Augmentation Component Analysis method on a synthetic mixture component data with a Gaussian augmentation method. In this example, we aim to show the relationship between semantic similarity and posterior/weighted augmentation distances. We also show the effectiveness of our method compared to traditional contrastive learning. In this example, the natural data $\bx$ are sampled from a mixture gaussian with $c$ component:
$$
\setlength{\abovedisplayskip}{3pt}
p(\bx) = \sum_{i=1}^c \pi_i \N (\bmu_i, s_i I)
\setlength{\belowdisplayskip}{3pt}
$$
We use Gaussian noise as the data augmentation of a natural data sample, \ie, $\A(\bx) = \bx + \xi$ where $\xi \sim \N (0, s_a I)$. Concretely, we conduct our experiment on 2-D data with $c= 4 , \pi_i= \frac{1}{c}, s_i = 1$ and the $\mu_i$ uniformly distributed on a circle with radius $2$ . For each component, we sample 200 natural data with the index of the component as their label. For each natural datum, we augment it 2 times with $s_a = 4$, which results in totally 1600 augmented data. We compute the augmentation probability for between $\x$ and $\bx$ by $p(\x\mid\bx)$ and we normalize the probability for each $\bx$.
First, we plot the distribution of posterior distances (\cref{eq:post_dist}) for pairs of augmented data and weighted augmentation distances (\cref{eq:w_aug_dist}) for pairs of natural data in~\cref{fig:synthetic} left. The two distances appear similar distributions because the synthetic data are Gaussian. It can be seen that data from the same component tend to have small distances, while from different components, their distances are large. In low-distance areas, there are same-class pairs, meaning that the two distances are reliable metrics for judging semantic similarity. In all, this picture reveals the correlation between semantic similarity and posterior/weighted augmentation distances.
Second, we compare our methods with SimCLR~\citep{Chen20Simple}, the traditional contrastive method and Spectral~\citep{Chen2021Provable}, which similarly learns embeddings with spectral theory. We test the learned embeddings using a Logistic Regression classifier and report the error rate of the prediction in~\cref{fig:synthetic} right. We also report performance when directly using \textit{augmentation feature} (AF). First, AF has the discriminability for simple linear classifiers. As the embedding size increase, SimCLR and Spectral tend to underperform AF, while our methods consistently outperform. It may be confusing since our method performs dimension reduction on this feature. But we note that as the embedding size increase, the complexity of the linear model also increases, which affect the generalizability. All of the methods in ~\cref{fig:synthetic} right show degradation of this kind. However, our methods consistently outperform others, which shows the superiority of ACA. Also, by adding projection loss, ACA-Full improves ACA-PC by a margin. Additionally, traditional contrastive learning like SimCLR achieves similar performance as our methods. We think it reveals that traditional contrastive learning has the same functionality as our methods.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.43\linewidth]{figures/distrib_density.pdf}
\hspace{4mm}
\includegraphics[width=0.51\linewidth]{figures/systhetic_performance.pdf}
\vspace{-4mm}
\caption{Synthetic experiments on mixture Gaussian data with Gaussian noise as augmentation. (a) The posterior distance and weighted augmentation distances among data that is sampled from the same component and different components. It reveals the correlation between semantic similarity and the two distances, especially when the distance is small. (b) Comparison of linear classification performance among SimCLR, Spectral and our methods with various embedding dimensions ranging from 4 to 200. The dashed line represents the result when directly using \textit{Augmentation Feature} (AF). ACA-PC outperforms SimCLR and Spectral. ACA-Full further improves.}
\label{fig:synthetic}
%\label{fig:pred_distr}
\vspace{-4mm}
\end{figure}
\vspace{-2mm}
\section{Experiments}
\label{sec:exp}
\subsection{Setup}
\vspace{-2mm}
\paragraph{Dataset.} In this paper, we conduct experiments mainly on the following datasets with RTX-3090 $\times 4$. \textbf{CIFAR-10} and \textbf{CIFAR-100}~\citep{Krizhevsky2009Learning}: two datasets containing totally 500K images of size 32 $\times$ 32 from 10 and 100 classes respectively. \textbf{STL-10}~\citep{Coates11Analysis}: derived from ImageNet~\citep{Deng09ImageNet}, with 96 $\times$ 96 resolution images with 5K labeled training data from 10 classes. Additionally, 100K unlabeled images are used for unsupervised learning. \textbf{Tiny ImageNet}: a reduced version of ImageNet~\citep{Deng09ImageNet}, composed of 100K images scaled down to 64 $\times$ 64 from 200 classes. \textbf{ImageNet-100}~\citep{Tian20Contrastive}: a subset of ImageNet, with 100-classes. \textbf{ImageNet}~\citep{Deng09ImageNet}, the large-scale dataset with 1K classes.
\vspace{-3mm}
\paragraph{Network Structure.} Following common practice~\citep{Chen20Simple,Chen20Big,Chen20Improved}, we use the encoder-projector structure during training, where the projector projects the embeddings into a low-dimensional space. For CIFAR-10 and CIFAR-100, we use the CIFAR variant of ResNet-18~\citep{He16Deep,Chen21Exploring} as the encoder. We use a two-layer MLP as the projector whose hidden dimension is half of the input dimension and output dimension is 64. For STL-10 and Tiny ImageNet, only the max-pooling layer is disabled following~\citep{Chen21Exploring,Ermolov21Whitening}. For these two datasets, we use the same projector structure except for the output dimension is 128. For ImageNet, we use ResNet-50 with the same projector as~\citet{Chen20Simple}.
\vspace{-3mm}
\paragraph{Image Transformation.} Following the common practice of contrastive learning~\citep{Chen20Simple}, we apply the following augmentations sequentially during training: (a) crops with a random size; (b) random horizontal flipping; (c) color jittering; (d) grayscaling. For ImageNet-100 and ImageNet, we use the same implementation as~\citep{Chen20Simple}.
\vspace{-3mm}
\paragraph{Optimizer and other Hyper-parameters.} For datasets except for ImageNet, adam optimizer~\citep{Kingma14Adam} is used for all the datasets. For CIFAR-10 and CIFAR-100, we use 800 epochs with a learning rate of $3 \times 10^{-3}$. For Tiny ImageNet and STL-10, we train 1,000 epochs with learning rate $2 \times 10^{-3}$. We use
%a learning rate warm-up for the first 500 iterations of the optimizer, and
a 0.1 learning rate decay at 100, 50, 20 epochs before the end. Due to hardware resource restrictions, we use a mini-batch of size $512$. The weight decay is $1 \times 10^{-6}$ if not specified. Following common practice in contrastive learning, we normalize the projected feature into a sphere. For CIFAR-10, we use $\alpha = 1$. For the rest datasets, we use $\alpha = 0.2$. By default, $K$ is set to 2. For ImageNet, we use the same hyperparameters as~\citep{Chen20Simple} except batch size being 256, $\alpha = 0.2$ and $K=2$.
\vspace{-3mm}
\paragraph{Evaluation Protocol.} We evaluate the learned representation on two most commonly used protocols -- linear classification~\citep{Zhang16Colorful,Kolesnikov19Revisiting} and k-nearest neighbors classifier~\citep{Chen21Exploring}. In all the experiments, we train the linear classifier for 100 epochs. The learning rate is exponentially decayed from $10^{-2}$ to $10^{-6}$. The weight decay is $1 \times 10^{-6}$. We report the classification accuracy on test embeddings as well as the accuracy of a 5-Nearest Neighbors classifier for datasets except for ImageNet.
\begin{table}[t]
\centering
\caption{Top-1 linear classification accuracy and 5-NN accuracy on four datasets with a ResNet-18 encoder. We use \textbf{bold} to mark the best results and \appallingunderline{underline} to mark the second best results. $^\sharp$ means the results are reproduced by our code.}
\begin{tabular}{lcccccccc}
\toprule
\multirow{2}{*}{method} & \multicolumn{2}{c}{CIFAR-10} & \multicolumn{2}{c}{CIFAR-100} & \multicolumn{2}{c}{STL-10} & \multicolumn{2}{c}{Tiny ImageNet} \\
& Linear & 5-NN & Linear & 5-NN & Linear & 5-NN & Linear & 5-NN \\
\midrule
SimCLR$^\sharp$
%~\citep{Chen20Simple}
& 90.88 & 88.25 & 65.53 & \underline{55.32} & 89.27 & 85.44 & 47.41 & 31.95 \\
BYOL$^\sharp$
%~\citep{Grill20Bootstrap}
& \underline{91.20} & \underline{89.52} & 64.85 & 54.92 & 88.64 & \textbf{86.73} & \underline{47.66} & \underline{32.96} \\
SimSiam$^\sharp$
%~\citep{Chen21Exploring}
& 91.06 & 89.43 & 65.41 & 54.84 & 90.04 & 85.48 & 45.17 & 30.41 \\
Spectral$^\sharp$
%~\citep{Chen2021Provable}
& 90.28 & 87.25 & 65.42 & 55.05 & 89.16 & 84.23 & 45.69 & 29.32 \\
\midrule
ACA-PC (ours) & 90.35 & 87.38 & \underline{65.69} & 54.57 & \underline{90.08} & 85.86 & 46.08 & 30.97 \\
ACA-Full (ours) & \textbf{92.04} & \textbf{89.79} & \textbf{67.16} & \textbf{56.52} & \textbf{90.88} & \underline{86.44} &\textbf{48.79} & \textbf{33.53} \\
\bottomrule
\end{tabular}
\label{tb:compare}
\vspace{-4mm}
\end{table}
\begin{table}[t]
\centering
\caption{Left: Top-1 classification accuracy and 5-NN accuracy on ImageNet-100 with ResNet-18. $^\dagger$: results are taken from~\citep{Wang20Understanding}. $^\star$: results are taken from~\citep{Tian20What}.$^\sharp$ means the results are reproduced by our code. Right: Top-1 classification accuracy on ImageNet with ResNet-50, results are taken from ~\citep{Chen21Exploring,Chen2021Provable}. We use \textbf{bold} to mark the best results and \underline{underline} to mark the second best results.}
\begin{tabular}{lcc}
\toprule
ImageNet-100 & Linear & 5-NN \\
\midrule
MoCo$^\dagger$ %~\citep{He20Momentum}
& 72.80 & - \\
$\cL_{align}+ \cL_{uniform}$ $^\dagger$ %~\citep{Wang20Understanding}
& 74.60 & - \\
% W-MSderline{76.00}&\textbf{67.04}\\
InfoMin$^\star$ %~\citep{Tian20What}
&74.90 & - \\
SimCLR$^\sharp$
%~\citep{Chen20Simple}
&75.62&\underline{62.70} \\
Spectral$^\sharp$
%~\citep{Chen2021Provable}
&75.52&61.80\\
\midrule
ACA-PC (ours) & \underline{75.80} & 62.54 \\
ACA-Full (ours) & \textbf{76.02} & \textbf{63.20} \\
\bottomrule
\end{tabular}
\quad \quad
\begin{tabular}{lc}
\toprule
ImageNet & Linear (100 epochs) \\
\midrule
SimCLR
%~\citep{Chen20Simple}
& 66.5 \\
MoCo v2
%~\citep{Chen20Improved}
& $67.4$ \\
BYOL
%~\citep{Grill20Bootstrap}
& 66.5 \\
% SwAV
%~\citep{Caron20Unsupervised}
% & 66.5 \\
SimSiam
%~\citep{Chen21Exploring}
& \underline{68.1} \\
Spectral
%~\citep{Chen2021Provable}
& 66.97\\
\midrule
ACA-PC (ours) &67.21 \\
ACA-Full (ours) & \textbf{68.32} \\
\bottomrule
\end{tabular}
\label{tb:in100}
\vspace{-4mm}
\end{table}
\vspace{-2mm}
\subsection{Performance Comparison}
In~\cref{tb:compare}, we compare the linear probe performance on various small-scale or mid-scale benchmarks with several methods including SimCLR~\citep{Chen20Simple}, BYOL~\citep{Grill20Bootstrap}, SimSiam~\citep{Chen21Exploring} and Spectral~\citep{Chen2021Provable}. For transfer learning benchmarks, please refer to \cref{subsec:transfer_linear} and \cref{sec:detection}. SimCLR uses is a method that uses contrastive loss. BYOL and SimSiam do not use negative samples. Spectral is a similar loss derived from the idea of spectral clustering.
From~\cref{tb:compare}, we can see that our ACA-Full method achieves competitive results on small-scale or mid-scale benchmarks, achieving either the best or the second-best results on all the benchmarks except the 5-NN evaluation on STL-10. Also, ACA-PC differs from ACA-Full in the projection loss. In all the benchmarks, we can see the projection loss improves performance.
%, in accordance with our synthetic experiment.
For large-scale benchmarks, we compare several methods on ImageNet-100 and ImageNet. On ImageNet-100, we compare our method additionally to MoCo~\citep{He20Momentum}, $\cL_{align}+ \cL_{uniform}$~\citep{Wang20Understanding} and InfoMin~\citep{Tian20What}. Note that the results of the other three methods are reported when using the ResNet-50 encoder, which has more capacity than ResNet18. Our method can also achieve state-of-the-art results among them. This means our method is also effective with relatively small encoders even on large-scale datasets. On ImageNet, we see that ACA-PC achieves competitive performance against state-of-the-art contrastive methods~\citep{Chen20Simple,Chen20Improved,Grill20Bootstrap,Chen21Exploring,Chen2021Provable} and ACA-Full achieves the best.
\section{Conclusion}
\label{sec:conclusion}
In this paper, we provide a new way of constructing self-supervised contrastive learning tasks by modeling similarity through augmentation overlap, which is motivated by the observation that semantically similar data usually creates similar augmentations. We propose Augmentation Component Analysis to perform PCA on augmentation feature efficiently. Interestingly, our methods have a similar form as the traditional contrastive loss and may explain the ability of contrastive loss. We hope our paper can inspire more thoughts about how to measure similarity in self-supervised learning and how to construct contrastive learning tasks. Currently, we limit our work to discrete augmentation distribution and we put investigation of continuous distribution for future work.
\bibliography{references}
\bibliographystyle{iclr2023_conference}
\newpage
\appendix
\section{Implementation and Further Discussion}
\label{app:impl}
In the previous section, we have presented the expected form of ACA. Thanks to its form, we can efficiently optimize ACA by Monte-Carlo sampling, making the problem \textbf{tractable}.
For convenience of illustration, we decompose the ACA-PC loss into two parts, \ie, $\cL_\text{ACA-PC} = \cL_\text{ali} + \cL_\text{uni}$. For the first part, $\cL_\text{ali}$ serves as the alignment loss in traditional contrastive learning, which maximizes the inner product similarity between augmented samples from the same natural data:
\begin{equation}
\begin{aligned}
\cL_\text{ali} = 2\expect_{\bx \sim p(\bx),\substack{\x_i\sim p(\x_i \mid \bx) \\
\x_j \sim p\left(\x_j \mid \bx\right)}} - f_\theta(\x_i)^\top f_\theta(\x_j)
\end{aligned},
\label{eq:ali}
\end{equation}
we use the mini-batch of natural sample to estimate $\expect_{\bx \sim p(\bx)}$. And we just use one sample to estimate $\expect_{\x_i\sim p(\x_i \mid \bx)}$ and $\expect_{\x_j \sim p\left(\x_j \mid \bx\right)}$ respectively. This leads to the traditional contrastive learning procedure : sample a mini-batch of natural data, augment it twice, compute and maximize the similarity of two augmented data.
For the second part, $\cL_\text{uni}$ minimize the inner product similarity of augmented data from the marginal distribution:
\begin{equation}
\cL_\text{uni} = N\expect_{\x_1\sim p_\A(\x_1),\x_2\sim p_\A(\x_2)}\left[\left(f_\theta(\x_1)^\top f_\theta(\x_2)\right)^2\right].
\label{eq:uni}
\end{equation}
We use the in-batch augmented data to estimate $\expect_{\x_1\sim p_\A(\x_1)}$. Notably, two augmented samples randomly sampled are hardly augmented by the same natural sample. Therefore, following common practice~\citep{Chen20Simple}, we use two augmented data that are created by augmenting two different natural data to compute this term. Additionally, we find that $N$ in $\cL_\text{uni}$ is too large to perform stable numerical computation. Thus in our implementation, we replace the $N$ with a tunable noise strength $K$.
For $\cL_\text{proj}$, it is not efficient to fully sample from $p(\x\mid\bx)$. However, it is notable that:
$$
\cL_\text{proj} = \expect_{\bx\sim p(\bx)} \left[\lVert f_\theta(\bx) - \expect_{p(\x\mid\bx)} f_{\theta}(\x) \rVert^2_2 \right] = \expect_{\bx \sim p(\bx),\substack{\x_i\sim p(\x_i \mid \bx) \\
\x_j \sim p\left(\x_j \mid \bx\right)}} \left[\lVert f_\theta(\bx) -
\frac{f_{\theta}(\x_i)+f_{\theta}(\x_j)}{2} \rVert^2_2 \right].
$$
It has the same expectation subscript as $\cL_\text{ali}$. So we can use the same strategy as $\cL_\text{ACA-PC}$ and reuse the samples. $\cL_\text{proj}$ is computed along with $\cL_\text{ali}$ during training, \ie, the principal component learning and projection are done simultaneously. That is why we call $\cL_\text{proj}$ ``on-the-fly projection''.
The overall implementation of ACA is illustrated in \cref{fig:imple}. And the algorithm is illustrated in \cref{alg:ACA}.
\begin{figure}[tbp]
\includegraphics[width=0.95\linewidth]{figures/implementation}
\caption{The implementation of ACA. Like traditional contrastive learning methods, ACA samples a mini-batch from the whole natural dataset, then performs two random augmentations on the mini-batch. The mini-batch sampling and augmentation create samples from $p(\bx)$ and $p(\x\mid\bx)$ respectively. Then we use the samples to estimate the values of $\cL_\text{uni}$,$\cL_\text{uni}$ and $\cL_\text{proj}$ as in the figure.}
\label{fig:imple}
\end{figure}
\paragraph{Discussion on the relation with traditional contrastive learning.} As is described in this section. ACA-PC takes a similar form as traditional contrastive learning methods. Similar to them, ACA-PC maximizes the inner product similarity of two views from the sample by~\cref{eq:ali}, and minimizes the squared inner product similarity of views from different samples. Note that we have proved that the learned embeddings by ACA-PC function as the principal components of the augmentation feature and preserve the posterior augmentation distances (\cref{prop:isometry}). We believe traditional contrastive loss also has the similar functionality as ours. Due to the strong correlation between augmentation overlap and semantic similarity, this may explain contrastive learning can learn semantically meaningful embeddings even though they ignore the semantic relationship between samples.
\paragraph{Discussion on approximation in implementations.} There are several approximations to stabilize the training. First, we replace the factor $N$ in~\cref{eq:uni} with a tunable noise strength $K$ . Usually, the number of samples is very large in common datasets. When we use a complex model like DNN, it is unstable to involve such a large number in the loss. Therefore, we tune it small and find it works well. But we also note that we use $N$ in our synthetic experiment in~\cref{sec:synthetic_exp} for the dimensionality and model is not too complex. The superior results proves effectiveness of our theory. Second, we normalize embeddings to project them into a sphere, which equals replacing the inner product with cosine similarity. We find this modification improves the performance from 81.84\% to 91.58\%.
%\section{Algorithm}
%\label{sec:empirical_loss}
%$\cL_\text{ACA-PC}$ consists of two parts, for the first part, \ie, the alignment loss:
%$$
%\begin{aligned}
% &\expect_{\bx \sim p(\bx),\substack{\x_i\sim p(\x_i \mid \bx) \\
% \x_j \sim p\left(\x_j \mid \bx\right)}} f_\theta(\x_i)^\top f_\theta(\x_j) \\
% =& \expect_{\bx \sim p(\bx)}\expect_{\x_i\sim p(\x_i \mid \bx)}\expect_{
% \x_j \sim p\left(\x_j \mid \bx\right)} f_\theta(\x_i)^\top f_\theta(\x_j)
%\end{aligned},
%$$
%we use the mini-batch of natural sample to estimate $\expect_{\bx \sim p(\bx)}$. And we just use one sample to estimate $\expect_{\x_i\sim p(\x_i \mid \bx)}$ and $\expect_{\x_j \sim p\left(\x_j \mid \bx\right)}$ respectively. This leads to the traditional contrastive learning procedure : sample a mini-batch of natural data, augment it twice, compute the similarity of two augmented data.
%
%For the second part,
%$$
%\expect_{\x_1\sim p_\A(\x_1),\x_2\sim p_\A(\x_2)}\left[\left(f_\theta(\x_1)^\top f_\theta(\x_2)\right)^2\right]
%$$
%we use the in-batch augmented data to estimate $\expect_{\x_1\sim p_\A(\x_1)}$. Following common practice~\citep{Chen20Simple}, we use two augmented data that are created by augmenting two different natural data to compute this term.
%
%For $\cL_{proj}$, it is not efficient to fully sample from $p(\x\mid\bx)$. However, it is notable that:
%$$
%\expect_{\bx\sim p(\bx)} \left[\lVert f_\theta(\bx) - \expect_{p(\x\mid\bx)} f_{\theta}(\x) \rVert^2_2 \right] = \expect_{\bx \sim p(\bx),\substack{\x_i\sim p(\x_i \mid \bx) \\
% \x_j \sim p\left(\x_j \mid \bx\right)}} \left[\lVert f_\theta(\bx) -
%\frac{f_{\theta}(\x_i)+f_{\theta}(\x_j)}{2} \rVert^2_2 \right]
%$$
%so we can use the same strategy as $\cL_\text{ACA-PC}$ and reuse the samples.
%
%
%In all, the algorithm is summarized in~\cref{alg:model} in~\cref{subsec:ACA}.
\begin{algorithm}[h]
\caption{Augmentation Component Analysis Algorithm}
\label{alg:model}
\begin{algorithmic}[1]
\REQUIRE
Unlabeled natural dataset $\{\x_i\}^N_{i=1}$; Augmentation function $\A$ ; Encoding model $f_\theta$ parameterized with $\theta$; projection parameter $\alpha$; Noise Strength $K$; Batch size $B$.
\FOR{sampled minibatch $\{\x_i\}^B_{i=1}$}
\FOR{$i= 1,2,\ldots, B$}
\STATE $\x^{(1)}_i = \A(\bx_i) ,\x^{(2)}_i = \A(\bx_i)$
\ENDFOR
\STATE $\cL_{ACA-Full} = - \frac{2}{B} \sum^B_{i=1} f^\top_\theta (\x^{(1)}_i) f_\theta (\x^{(2)}_i) + \frac{K}{B(B-1)} \sum_{i\neq j} (f^\top_\theta (\x^{(1)}_i) f_\theta (\x^{(2)}_j))^2 + \alpha \lVert f_\theta(\bx) - \frac{f_\theta(\x^{(1)}_i) + f_\theta(\x^{(2)}_i)}{2}\rVert^2$
\STATE update $\theta$ with w.r.t $\cL_{ACA-Full}$.
\ENDFOR
\STATE \textbf{return} $f_\theta$
\end{algorithmic}
\label{alg:ACA}
\end{algorithm}
\section{Effect of Augmentation Overlaps}
\label{subsec:exp_aug}
Like contrastive learning, our method relies on the quality of augmentation. Therefore, we investigate the influence of different augmentations and reveal the relationship between distribution difference and the linear probe performance on CIFAR10.
The augmentation distribution is estimated \appallingunderline{by augmenting $10^6$ times} for a subset of random 2000 pairs of samples with the number of intra-class and inter-class pairs being 1000 respectively. Note that as is stated in~\cref{subsec:augmentatoin_feature}, even on CIFAR10, the actual value of $L$ is exponentially large (up to $256^{3072}$). It is impossible to accurately estimate a distribution over so many possible values. But we notice that for neural networks, many operators can reduce the possible number of values, like convolutions and poolings. Following this observation and to make the computation efficient, we descrete the color into 8-bit for each channel and use a max pooling operation to get a $4\times 4$ picture. by this kind of approximation, the number of $L$ reduces to $8^{48}$. Seems still too large, but it can be noted that the augmentation distribution of each sample covers only a small region. It is enough to estimate the distribution by sampling. For restriction of memory, we can not fully estimate the weighted augmentation distance in~\cref{prop:isometry_natural}. Because we can not store all possible values for $p_\A(\x)$. Instead, we use the Hellinger distance as the distribution distance measure:
$$
\operatorname{d_{H}^2} (\bx_1,\bx_2) = \frac{1}{N} \sum_{\x\in \X} \left(\sqrt{p(\x\mid\bx_1)} - \sqrt{p(\x\mid\bx_2)} \right)^{2}
$$
Hellinger distance ranges $[0, 2]$, making the comparison clear.
We list the experimented augmentation here:
\begin{enumerate}
\item \textbf{Grayscale}: Randomly change the color into gray with probability of $0.1$.
\item \textbf{HorizontalFlip}: Randomly flip horizontally with probability $0.5$.
\item \textbf{Rotation}: Randomly rotate image with uniformly distributed angle in $[0,\pi]$
\item \textbf{ColorJitter}: Jitter (brightness, contrast, saturation, hue) with strength (0.4, 0.4, 0.4, 0.1) and probability 0.8.
\item \textbf{ResizedCrop}: Extract crops with a random size from 0.2 to 1.0 of the original area and a random aspect ratio from 3/4 to 4/3 of the original aspect ratio.
\item \textbf{Augs in SimCLR}: Sequential combination of 5,4,1,2.
\end{enumerate}
In~\cref{tb:aug_his}, we display the histogram (HIST) of intra- and inter-class augmentation \textit{distribution distances}.
%Distribution estimation and distance details are described in~\cref{app:exp_aug}.
ACC displays the linear probe performance on the test set. From the table, the following requirements for a good augmentation can be concluded: (1) \textbf{Existence of overlap}. For the upper three augmentations. The ``scope'' of augmentation is small. As a result, the majority of samples do not overlap. This makes the embeddings lack the discriminative ability for downstream tasks. On the contrary, the lower three create overlaps for most of the samples, leading to much better performance. (2) \textbf{Intra-class distance is lower than inter-class}. Compared to ColorJitter, ResizedCrop makes more intra-class samples have lower distance. So ResizedCrop outperforms ColorJitter. SimCLR augmentation surpasses these two for the same reason. Interestingly, we find the same phenomena appear when using other contrastive methods like SimCLR. It shows that these methods somehow utilize the augmentation overlap like our method.
\begin{table}[tb]
\centering
\caption{Histogram (HIST) of distribution distances and linear probe accuracy (ACC) when using different augmentations on CIFAR10. Note that HIST is estimated in the input space. It is property of augmentation, regardless of learning algorithm. We aims to investigate the different augmentation overlaps caused by different augmentation, and reveal its connection between learned model. ``Same'' denotes the distance between samples with the same semantic class, and ``Different" means different classes. The existence of overlap and relationship between intra/inter-class distances affects the performance.}
\begin{tabular}{cccc}
\toprule
&Grayscale & HorizontalFlip & Rotation \\
\midrule
HIST &
\raisebox{-.5\height}{\includegraphics[width=0.27\linewidth]{figures/ablation/hist_grayscale}}
&\raisebox{-.5\height}{\includegraphics[width=0.27\linewidth]{figures/ablation/hist_hflip}}&\raisebox{-.5\height}{\includegraphics[width=0.27\linewidth]{figures/ablation/hist_rotation}} \\
ACC & 27.05\% & 29.67\% & 33.50\% \\\midrule
& ColorJitter & ResizedCrop & Augs in SimCLR \\\midrule
HIST&\raisebox{-.5\height}{\includegraphics[width=0.27\linewidth]{figures/ablation/hist_colorjitter}}&\raisebox{-.5\height}{\includegraphics[width=0.27\linewidth]{figures/ablation/hist_randomcrop}} &\raisebox{-.5\height}{\includegraphics[width=0.27\linewidth]{figures/ablation/hist_simclr}} \\ACC & 59.74\% & 75.98\% & 92.04\% \\
\bottomrule
\end{tabular}
\label{tb:aug_his}
\end{table}
% \section{Details of~\cref{subsec:exp_aug}}
% \label{app:exp_aug}
\section{Performance Curve}
In this section, we illustrate the performance curve throughout training. We aim to demonstrate the functionality of projection loss and show that our ACA method leads to better performance. The compared traditional contrastive learning method is chosen to be SimCLR, for the reason that \textit{our method only differs from SimCLR in the loss, with all other things (architecture, optimizer and other shared hyperparameters) identical. Also, we do not introduce extra mechanisms like momentum encoder (BYOL, MoCo) and predictor (BYOL, SimSiam)}.
\cref{fig:performance_curve} shows the performance curve along with the projection loss on the CIFAR-10 dataset. The left figure shows the projection loss. We can see that in the early stage of training, the projection loss will increase. It reveals that the natural data will deviate from the center of augmentation distribution. It is harmful to the performance of the model. With the help of projection loss, the embeddings of natural data will be dragged back to their right position, the center. The mid and right figures illustrate the performance curve during training. With only ACA-PC loss, the model can only achieve similar performance during training. But the ACA-Full loss will help improve the performance all the training time. Also, we can see that ACA starts to outperform SimCLR and ACA-PC by a considerable margin from about 50 epochs. This happens to be the epoch that the projection loss increases to its stable level. Therefore, pulling the natural data to the center of its augmentation helps learn better embeddings.
\begin{figure}
\centering
\includegraphics[width=0.32\linewidth]{figures/projection_loss.pdf}
\includegraphics[width=0.32\linewidth]{figures/aca_simclr_compare_knn.pdf}
\includegraphics[width=0.32\linewidth]{figures/aca_simclr_compare_test.pdf}
\vspace{-3mm}
\caption{Values of projection loss and accuracy during training. Left is the projection loss. It rises in the early stage and is stable later, showing that the projection loss prevents the embeddings of natural data from deviating from the center of augmentation distribution. The mid and right figures show the performance curve measured by kNN accuracy and test accuracy. ACA-PC achieves similar performance as SimCLR but ACA-Full is better during the training process.}
\vspace{-3mm}
\label{fig:performance_curve}
\end{figure}
\vspace{-2mm}
\section{Transfer to Other Datasets}
\label{subsec:transfer_linear}
Following \citet{Chen20Simple}, we evaluate the self-supervised pre-trained models for linear classification task on 10 datasets as it is conducted in MSF paper~\citep{Koohpayegani21Mean}. The results are reported in \cref{tab:transfer_linear}. All the results other than ACA are taken from \citet{Koohpayegani21Mean}. Although our method is trained with fewer epochs, it achieves competitive results with contrastive learning methods. Notably, it surpasses 1000-epoch SimCLR which differs from our method only in loss. It shows that the embeddings learned by our method are also transferable to other downstream tasks. We think it is due to the universality of correlation between augmentation similarity and semantical similarity across these benchmarks.
\begin{table}
\centering
\caption{We compare various SSL methods on transfer tasks by training linear layers. Only a single linear layer is trained on top of features. Simple random horizontal flips are used. Results except ours are taken from~\citet{Koohpayegani21Mean}. Our method can achieve competitive results with other contrastive learning methods despite short epochs, especially that it can surpass 1000-epoch SimCLR.}
\label{tab:transfer_linear}
\scalebox{0.8}{
\begin{tabular}{cccccccccccc}
\toprule
Method & Epochs & Food & CIFAR10 & SUN & Cars & Aircraft & DTD & Pets & Caltech & Flowers & Mean \\
\midrule
Supervised & & 72.30 & 93.60 & 61.90 & 66.70 & 61.00 & 74.90 & 91.50 & 94.50 & 94.70 & 78.90 \\
\midrule
SimCLR & 1000 & 72.80 & 90.50 & 60.60 & 49.30 & 49.80 & \textbf{75.70} & 84.60 & 89.30 & 92.60 & 74.00 \\
MoCo v2 & 800 & 72.50 & \textbf{92.20} & 59.60 & 50.50 & 53.20 & 74.40 & 84.60 & 90.00 & 90.50 & 74.20 \\
BYOL & 1000 & \textbf{75.30} & 91.30 & \textbf{62.20} & \textbf{67.80} & \textbf{60.60} & 75.50 & \textbf{90.40} & \textbf{94.20} & \textbf{96.10} & \textbf{79.20} \\
\midrule
BYOL-asym & 200 & 70.20 & 91.50 & 59.00 & 54.00 & 52.10 & 73.40 & 86.20 & 90.40 & 92.10 & 74.30 \\
MoCo v2 & 200 & 70.40 & 91.00 & 57.50 & 47.70 & 51.20 & \textbf{73.90} & 81.30 & 88.70 & 91.10 & 72.60 \\
MSF & 200 & 70.70 & 92.00 & 59.00 & \textbf{60.90} & 53.50 & 72.10 & \textbf{89.20} & 92.10 & \textbf{92.40} & \textbf{75.80} \\
MSF-w/s & 200 & 71.20 & \textbf{92.60} & \textbf{59.20} & 55.60 & 53.70 & 73.20 & 88.70 & \textbf{92.70} & 92.00 & 75.50 \\
\midrule
ACA (ours) & 100 & \textbf{72.54} & 90.01 & \textbf{59.17} & 49.61 & \textbf{56.51} & \textbf{73.78} & 83.27 & 89.92 & 90.18 & 74.07 \\
\bottomrule
\end{tabular}
}
\vspace{-2mm}
\end{table}
\section{Transfer to Object Detection}
\label{sec:detection}
Following the procedure outlined in \cite{He20Momentum}, we use Faster-RCNN \cite{Ren15Faster} for the task of object detection on PASCAL-VOC \cite{Everingham15Pascal}. We use the code provided at MoCo repository\footnote{https://github.com/facebookresearch/moco} with default parameters. All the weights are finetuned on the \texttt{trainval07+12} set and evaluated on the \texttt{test07} set. We report an average over 5 runs in Table \ref{tab:detection}. Despite the shorter training epochs, our method can achieve better results than SimCLR, especially outperform by a large margin on $\text{AP}_{75} (>1\%)$.
\begin{table}
\centering
\caption{We compare our models on the transfer task of object detction. We find that given a similar computational budget, our method is better than SimCLR, with shorter training time. The models are trained on the VOC \texttt{trainval07+12} set and evaluated on the \texttt{test07} set. We report average over 5 runs.}
\label{tab:detection}
\begin{tabular}{ccccc}
\toprule
Method & Epochs & $\text{AP}_{50}$ & AP & $\text{AP}_{75}$ \\
\midrule
Sup. IN & - & 81.30 & 53.50 & 58.80 \\
Scratch & - & 60.20 & 33.80 & 33.10 \\
\midrule
SimCLR~\citep{Chen20Simple} & 200 & 81.80 & 55.50 & 61.40 \\
MoCo v2~\citep{Chen20Improved} & 200 & 82.30 & \textbf{57.00} & 63.30 \\
BYOL~\citep{Grill20Bootstrap} & 200 & 81.40 & 55.30 & 61.10 \\
SwAV~\citep{Caron20Unsupervised} & 200 & 81.50 & 55.40 & 61.40 \\
SimSiam~\citep{Chen21Exploring} & 200 & \textbf{82.40} & \textbf{57.00} & \textbf{63.70} \\
MSF \citep{Koohpayegani21Mean} & 200 & 82.20 & 56.70 & 63.40 \\
MSF w/s \citep{Koohpayegani21Mean} & 200 & 82.20 & 56.60 & 63.10 \\
\midrule
ACA-Full (ours) & 100 & 82.25 & 56.14 & 63.05 \\
\bottomrule
\end{tabular}
\end{table}
\section{Proof of~\cref{lemma:ACAa}}
\label{sec:proof_ACAa}
For convenient, we define $M := \hat{A}^\top \hat{A}$. The elements of $M$ are:
\begin{equation}
M_{\x_1\x_2} = \sum_{\bx\in \bcX} \frac{p(\x_1\mid\bx)p(\x_2\mid\bx)}{\sqrt{d_{\x_1}} \sqrt{d_{\x_2}}}, \x_1, \x_2 \in \X
\end{equation}
Expanding~\cref{eq:Lmf}, we get:
$$
\begin{aligned}
\cL_{mf} &= \sum_{\x_1, \x_2 \in \X} ( M_{\x_1 \x_2} - F^\top_{\x_1} F_{\x_2})^2 \\
&= \sum_{\x_1, \x_2 \in \X} ( M_{\x_1 \x_2} - \sqrt{d_{\x_1}}\sqrt{d_{\x_2}} f_\theta(\x_1)^\top f_\theta(\x_2))^2 \\
& = \text{const} - 2\sum_{\x_1, \x_2 \in \X} \sqrt{d_{\x_1}}\sqrt{d_{\x_2}} M_{\x_1 \x_2} f_\theta(\x_1)^\top f_\theta(\x_2) + \sum_{\x_1, \x_2 \in \X} d_{\x_1}d_{\x_2} (f_\theta(\x_1)^\top f_\theta(\x_2))^2 \\
& = \text{const} - 2\sum_{\x_1, \x_2 \in \X}\sum_{\bx\in \bcX} p(\x_1\mid\bx)p(\x_2\mid\bx) f_\theta(\x_1)^\top f_\theta(\x_2) + \sum_{\x_1, \x_2 \in \X} d_{\x_1}d_{\x_2} (f_\theta(\x_1)^\top f_\theta(\x_2))^2
\end{aligned}
$$
multiply by $p(\bx)=\frac{1}{N}$ and replace $d_\x$ with $\sum_{\bx} p(\x\mid\bx) = N p_{\A} (\x)$. The objective becomes:
$$
\begin{aligned}
\min_\theta \quad& - 2 \sum_{\x_1, \x_2 \in \X} \sum_{\bx\in \bcX} p(\x_1\mid\bx)p(\x_2\mid\bx) p(\bx) f_\theta(\x_1)^\top f_\theta(\x_2) \\
& \quad +N \sum_{\x_1, \x_2 \in \X} p_\A (\x_1) p_\A (\x_2) (f_\theta(\x_1)^\top f_\theta(\x_2))^2 \\
&= - 2\expect_{\bx\sim p(\bx),\substack{\substack{\x_i\sim \mathcal{A}(\x_i \mid \bx) \\ \x_j \sim \mathcal{A}\left(\x_j \mid \bx\right)}}}\left[ f_\theta(\x_1)^\top f_\theta(\x_2) \right] \\
& \quad + N \expect_{\x_1\sim p_\A(\x_1),\x_2\sim p_\A(\x_2)} \left[ (f_\theta(\x_1)^\top f_\theta(\x_2))^2 \right] \\
& = \cL_\text{ACA-PC}
\end{aligned}
$$
\section{Proof of~\cref{prop:isometry}}
\label{sec:proof_isometry}
As in~\cref{sec:proof_ACAa}, we define $M := \hat{A}^\top \hat{A}$. By Eckart–Young–Mirsky theorem~\citep{Eckart1936Approximation}, the minimizer $\hat{F}$ of $\lVert M - FF^\top \rVert^2_F$, must have the form $ \hat{V}\hat{\Sigma} Q$, where $\hat{V},\hat{\Sigma}$ contain the top-k singular values and corresponding right singular vectors of $\hat{A}$, $Q\in \R^{k\times k}$ is some orthonormal matrix with $Q^\top Q = I$. Since we let $F_\x = \sqrt{d_x} f_\theta(\x)$, then the minimizer $\theta^\star$ must satisfy
$$f_{\theta^\star}(\x) =Q \frac{\hat{\boldsymbol{\sigma}} \odot \hat{\boldsymbol{v}}(\x)}{\sqrt{d_\x}} = Q \frac{\left[\sigma_1 v_1(\x), \sigma_2 v_2(\x),\ldots, \sigma_k v_k(\x) \right]^\top}{\sqrt{d_\x}}.$$
where $\odot$ is the element-wise multiplication. For convenience, we use $\sigma_i$ to denote $i$-th largest singular value, $u_i(\bx)$,$v_i(\x)$ to denote the element of $i$-th left/right singular value corresponding to $\bx$/$\x$ . When $p(\bx) = \frac{1}{N}$, $d_\x = Np_\A (\x) = \frac{p_\A (\x)}{p(\bx)}$. Then the posterior distance:
\begin{equation}
\begin{aligned}
\operatorname{d_{post}^2} (\x_1,\x_2) &= \sum_{\bx\in \bcX} \left(p_{\A}(\bx\mid\x_1)- p_{\A}(\bx\mid\x_2)\right)^{2} \\
&= \sum_{\bx\in \bcX} \left(\frac{p(\x_1\mid\bx)p(\bx)}{p_\A(\x_1)}- \frac{p(\x_1\mid\bx)p(\bx)}{p_\A(\x_1)}\right)^{2} \\
&= \sum_{\bx\in \bcX} \left(\frac{p(\x_1\mid\bx)}{d_{\x_1}}- \frac{p(\x_2\mid\bx)}{d_{\x_2}}\right)^{2} \\
&= \sum_{\bx\in \bcX} \left(\frac{\hat{A}_{\bx \x_1}}{\sqrt{d_{\x_1}} }-\frac{\hat{A}_{\bx \x_2}}{\sqrt{d_{\x_2}} }\right)^{2}\\
&= \sum_{\bx\in \bcX} \left(\sum^N_{i=1} \frac{\sigma_i u_i(\bx) v_i(\x_1)}{\sqrt{d_{\x_1}}}-\frac{\sigma_i u_i(\bx) v_i(\x_2)}{\sqrt{d_{\x_2}}}\right)^{2} \\
&= \sum_{\bx\in \bcX} \left(\sum^N_{i=1} \sigma_i u_i(\bx)(\frac{ v_i(\x_1)}{\sqrt{d_{\x_1}}}-\frac{ v_i(\x_2)}{\sqrt{d_{\x_2}}})\right)^{2} \\
& = \sum_{\bx\in \bcX}\sum_{i,i^\prime} \sigma_i u_i(\bx) \sigma_{i'} u_{i'}(\bx)(\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})(\frac{v_{i'}(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i'}(\x_2)}{\sqrt{d_{\x_2}}} )\\
& = \sum_{i,i^\prime} \sigma_i \sigma_{i'} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})(\frac{v_{i'}(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i'}(\x_2)}{\sqrt{d_{\x_2}}} )\sum_{\bx\in \bcX}u_i(\bx)u_{i'}(\bx) \\
& \overset{(1)}{=} \sum_{i,i^\prime} \sigma_i \sigma_{i'} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})(\frac{v_{i'}(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i'}(\x_2)}{\sqrt{d_{\x_2}}} )\delta_{i,i'} \\
&= \sum^N_{i=1} \sigma^2_i (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2
\end{aligned}
\end{equation}
(1) is due to the orthogonality of singular vectors. Note that:
$$
\begin{aligned}
&\sum^N_{i=1} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2\\
=& \sum^L_{i=1} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2 - \sum^{L}_{i=N+1} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2 \\
\le& \sum^L_{i=1} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2 \\
=& \sum^L_{i=1} \frac{v_i^2(\x_1)}{d_{\x_1}} + \sum^L_{i=1} \frac{v_i^2(\x_2)}{d_{\x_2}} - 2 \sum^L_{i=1}
\frac{v_i(\x_1)v_i(\x_2)}{\sqrt{d_{\x_1}}\sqrt{d_{\x_2}}}\\
= & \frac{1}{d_{\x_1}} + \frac{1}{d_{\x_2}} - \frac{2\delta_{\x_1 \x_2}}{\sqrt{d_{\x_1}}\sqrt{d_{\x_2}}} \\
\overset{(2)}{\le} & (\frac{1}{d_{\x_1}} + \frac{1}{d_{\x_2}})(1 -\delta_{\x_1 \x_2} ) \\
\le & \frac{2}{d_{\min}}\left(1-\delta_{\x_1 \x_2}\right)
\end{aligned}
$$
(2) can be deduced by considering conditions whether $\x_1= \x_2$ or not. Then:
$$
\begin{aligned}
&\lVert f_{\theta^\star}(\x_1) - f_{\theta^\star}(\x_2)\rVert^2 \\
=& \sum^k_{i=1} \sigma^2_i (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2 \\
=& \operatorname{d_{post}^2} (\x_1,\x_2) - \sum^N_{i=k} \sigma^2_i (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2 \quad (\le \operatorname{d_{post}^2} (\x_1,\x_2)) \\
\ge& \operatorname{d_{post}^2} (\x_1,\x_2) - \sigma^2_{k+1} \sum^N_{i=k+1} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2\\
\ge& \operatorname{d_{post}^2} (\x_1,\x_2) - \sigma^2_{k+1} \sum^N_{i=1} (\frac{v_i(\x_1)}{\sqrt{d_{\x_1}}}- \frac{v_{i}(\x_2)}{\sqrt{d_{\x_2}}})^2 \\
\ge& \operatorname{d_{post}^2} (\x_1,\x_2) - \frac{2 \sigma_{k+1}^{2}}{d_{\min}}\left(1-\delta_{\x_1 \x_2}\right)
\end{aligned}
$$
Therefore, we have proved~\cref{prop:isometry}.
\section{Proof of~\cref{prop:isometry_natural}}
\label{sec:proof_isometry_natural}
similar to~\cref{sec:proof_isometry},
$$
\begin{aligned}
\operatorname{d_{w-aug}^2} (\bx_1,\bx_2) &= \sum_{\x\in \X} \frac{1}{N p_\A(\x)} \left(p(\x\mid\bx_1)- p(\x\mid\bx_2)\right)^{2} \\
&= \sum_{\x\in \X} \left(\frac{p(\x\mid\bx_1)}{\sqrt{Np_\A(\x)}} - \frac{p(\x\mid\bx_1)}{\sqrt{Np_\A(\x)}}\right)^{2}\\
&= \sum_{\x\in \X} \left(\frac{p(\x\mid\bx_1)}{\sqrt{d_\x}} - \frac{p(\x\mid\bx_1)}{\sqrt{d_\x}}\right)^{2} \\
&= \sum_{\x\in \X} \left(\hat{A}_{\bx_1 \x}-\hat{A}_{\bx_2 \x} \right)^{2}\\
&= \sum_{\x\in \X} \left(\sum^N_{i=1} \sigma_i u_i(\bx_1) v_i(\x)-\sigma_i u_i(\bx_2) v_i(\x)\right)^{2} \\
&= \sum_{\x\in \X} \left(\sum^N_{i=1} \sigma_i (u_i(\bx_1) -u_i(\bx_2))v_i(\x)\right)^{2} \\
& = \sum_{\x\in \X}\sum_{i,i^\prime} \sigma_i v_i(\x) \sigma_{i'} v_{i'}(\x)(u_i(\bx_1)- u_{i}(\bx_2))(u_{i'}(\bx_1)- u_{i'}(\bx_2) )\\
& = \sum_{i,i^\prime} \sigma_i \sigma_{i'} (u_i(\bx_1)- u_{i}(\x_2))(u_{i'}(\bx_1)-u_{i'}(\bx_2))\sum_{\x\in \X}v_i(\x)v_{i'}(\x) \\
& \overset{(1)}{=} \sum_{i,i^\prime} \sigma_i \sigma_{i'} (u_i(\bx_1)- u_{i}(\bx_2))(u_{i'}(\bx_1)- u_{i'}(\bx_2) )\delta_{i,i'} \\
&= \sum^N_{i=1} \sigma^2_i (u_i(\x_1)- u_{i}(\x_2))^2
\end{aligned}
$$
(1) is due to the orthogonality of singular vectors. And $g(\bx)$ takes the following form:
$$g(\bx) = Q \left[\sigma^2_1 u_1(\x), \sigma^2_2 u_2(\x),\ldots, \sigma^2_k u_k(\x) \right]^\top.$$
Thus,
$$
\begin{aligned}
\lVert g(\bx_1) - g(\bx_2) \rVert^2_{\Sigma_k^{-2}} &= \sum^k_{i=1} \sigma^2_i (u_i(\x_1)- u_{i}(\x_2))^2 \\
&= \operatorname{d_{w-aug}^2} (\bx_1,\bx_2) - \sum^N_{i=k+1} \sigma^2_i (u_i(\x_1)- u_{i}(\x_2))^2 \quad (\le \operatorname{d_{w-aug}^2} (\bx_1,\bx_2)) \\
&\ge \operatorname{d_{w-aug}^2} (\bx_1,\bx_2) - \sigma^2_{k+1} \sum^N_{i=1} (u_i(\x_1)- u_{i}(\x_2))^2 \\
& = \operatorname{d_{w-aug}^2} (\bx_1,\bx_2) - 2\sigma^2_{k+1}(1-\delta_{\bx_1 \bx_2})
\end{aligned}
$$
\section{Ablation Study on Parameter $\alpha$ and $K$}
We conduct ablation experiments on the parameter $\alpha$ and $K$. $\alpha$ is the trade-off parameter between ACA-PC loss and projection loss~\cref{eq:final_loss}. $K$ act as the noise strength for ACA-PC, which replace $N$ in~\cref{eq:ACAa}.
\cref{fig:ablation} shows the effect of $\alpha$ and $K$ on different benchmarks. It can be seen that $\alpha$ is necessary to improve the performance of ACA-PC. A certain value of $\alpha$ helps the model achieve better results. However, a too large value of $\alpha$ degrades the performance. The same phenomenon is the same on $K$.
\begin{figure}[tbp]
\centering
\subfigure[]{
\includegraphics[width=0.31\linewidth]{figures/figure_alpha_acc_cifar10_line}
}
\subfigure[]{
\includegraphics[width=0.31\linewidth]{figures/figure_alpha_acc_cifar100_line}
}
\subfigure[]{
\includegraphics[width=0.31\linewidth]{figures/figure_alpha_acc_stl10_line.pdf}
}\\
\vspace{-3mm}
\subfigure[]{
\includegraphics[width=0.31\linewidth]{figures/figure_K_acc_cifar10_line.pdf}
}
\subfigure[]{
\includegraphics[width=0.31\linewidth]{figures/figure_K_acc_cifar100_line.pdf}
}
\subfigure[]{
\includegraphics[width=0.31\linewidth]{figures/figure_K_acc_stl10_line.pdf}
}
\caption{Ablation studies on the effect of $\alpha$ and $K$. We report linear classification and 5-nearest neighbor accuracy on different datasets with the ResNet-18 encoder. The upper 3 figures illustrate the effect of $\alpha$ on 3 different datasets. The lower 3 figures illustrate the performance v.s. $K$. }
\label{fig:ablation}
%\label{fig:pred_distr}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Comparison of Nearest Neighbors}
We randomly select 8 samples from the validation set of ImageNet-100~\citep{Tian20Contrastive}. Then we use the encoder learned by our ACA method and SimCLR~\citep{Chen20Simple} to extract features and investigate their nearest neighbors of them. The left-most column displays the selected samples and the following columns show the 5 nearest neighbors. The samples that are labeled as different classes have been marked by the red box. We also annotate the distance between the samples and their nearest neighbors. First, we can see that even though utilizing the augmentation in a different way, ACA achieves similar results as traditional contrastive learning. Both of them can learn semantically meaningful embeddings. However, we can see that ACA tends to learn embeddings that pull together images that are similar in the input space, \ie, creating similar augmentation, while SimCLR sometimes has neighbors that seem different.
\begin{figure}[tb]
\centering
\fbox{\includegraphics[width=0.46\linewidth]{figures/CPCL.pdf}}
\fbox{\includegraphics[width=0.46\linewidth]{figures/SimCLR.pdf}}
\caption{The 5 nearest neighbors of selected samples in the embedding space of ACA and SimCLR. Images are taken from the validation set of ImageNet-100. Distances between selected samples and their nearest neighbors are annotated above each picture. We can see that the embeddings learned by ACA tend to pull together images that are similar in the input space, \ie, creating similar augmentation. While SimCLR sometimes has neighbors that seem different. }
\label{fig:my_label}
\end{figure}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{58}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Azabou et~al.(2021)Azabou, Azar, Liu, Lin, Johnson, Bhaskaran{-}Nair,
Dabagia, Hengen, Roncal, Valko, and Dyer]{Azabou2021Mine}
Mehdi Azabou, Mohammad~Gheshlaghi Azar, Ran Liu, Chi{-}Heng Lin, Erik~C.
Johnson, Kiran Bhaskaran{-}Nair, Max Dabagia, Keith~B. Hengen, William
R.~Gray Roncal, Michal Valko, and Eva~L. Dyer.
\newblock Mine your own view: Self-supervised learning through across-sample
prediction.
\newblock \emph{CoRR}, abs/2102.10106, 2021.
\bibitem[Bachman et~al.(2019)Bachman, Hjelm, and Buchwalter]{Bachman19Learning}
Philip Bachman, R.~Devon Hjelm, and William Buchwalter.
\newblock Learning representations by maximizing mutual information across
views.
\newblock In \emph{NeurIPS}, pp.\ 15509--15519, 2019.
\bibitem[Caron et~al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and
Joulin]{Caron20Unsupervised}
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and
Armand Joulin.
\newblock Unsupervised learning of visual features by contrasting cluster
assignments.
\newblock In \emph{NeurIPS}, 2020.
\bibitem[Chen et~al.(2020{\natexlab{a}})Chen, Kornblith, Norouzi, and
Hinton]{Chen20Simple}
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey~E. Hinton.
\newblock A simple framework for contrastive learning of visual
representations.
\newblock In \emph{ICML}, pp.\ 1597--1607, 2020{\natexlab{a}}.
\bibitem[Chen et~al.(2020{\natexlab{b}})Chen, Kornblith, Swersky, Norouzi, and
Hinton]{Chen20Big}
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey~E.
Hinton.
\newblock Big self-supervised models are strong semi-supervised learners.
\newblock In \emph{NeurIPS}, 2020{\natexlab{b}}.
\bibitem[Chen \& He(2021)Chen and He]{Chen21Exploring}
Xinlei Chen and Kaiming He.
\newblock Exploring simple siamese representation learning.
\newblock In \emph{CVPR}, pp.\ 15750--15758, 2021.
\bibitem[Chen et~al.(2020{\natexlab{c}})Chen, Fan, Girshick, and
He]{Chen20Improved}
Xinlei Chen, Haoqi Fan, Ross~B. Girshick, and Kaiming He.
\newblock Improved baselines with momentum contrastive learning.
\newblock \emph{CoRR}, abs/2003.04297, 2020{\natexlab{c}}.
\bibitem[Coates et~al.(2011)Coates, Ng, and Lee]{Coates11Analysis}
Adam Coates, Andrew~Y. Ng, and Honglak Lee.
\newblock An analysis of single-layer networks in unsupervised feature
learning.
\newblock In \emph{AISTATS}, pp.\ 215--223, 2011.
\bibitem[Deng et~al.(2009)Deng, Dong, Socher, Li, Li, and
Fei{-}Fei]{Deng09ImageNet}
Jia Deng, Wei Dong, Richard Socher, Li{-}Jia Li, Kai Li, and Li~Fei{-}Fei.
\newblock Imagenet: {A} large-scale hierarchical image database.
\newblock In \emph{CVPR}, pp.\ 248--255, 2009.
\bibitem[Devlin et~al.(2019)Devlin, Chang, Lee, and Toutanova]{Devlin19BERT}
Jacob Devlin, Ming{-}Wei Chang, Kenton Lee, and Kristina Toutanova.
\newblock {BERT:} pre-training of deep bidirectional transformers for language
understanding.
\newblock In \emph{NAACL-HLT}, pp.\ 4171--4186, 2019.
\bibitem[Doersch et~al.(2015)Doersch, Gupta, and
Efros]{Doersch2015Unsupervised}
Carl Doersch, Abhinav Gupta, and Alexei~A Efros.
\newblock Unsupervised visual representation learning by context prediction.
\newblock In \emph{ICCV}, pp.\ 1422--1430, 2015.
\bibitem[Donahue \& Simonyan(2019)Donahue and Simonyan]{Donahue19Large}
Jeff Donahue and Karen Simonyan.
\newblock Large scale adversarial representation learning.
\newblock In \emph{NeurIPS}, pp.\ 10541--10551, 2019.
\bibitem[Dwibedi et~al.(2021)Dwibedi, Aytar, Tompson, Sermanet, and
Zisserman]{Dwibedi2021Little}
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew
Zisserman.
\newblock With a little help from my friends: Nearest-neighbor contrastive
learning of visual representations.
\newblock In \emph{ICCV}, pp.\ 9588--9597, 2021.
\bibitem[Eckart \& Young(1936)Eckart and Young]{Eckart1936Approximation}
Carl Eckart and Gale Young.
\newblock The approximation of one matrix by another of lower rank.
\newblock \emph{Psychometrika}, 1\penalty0 (3):\penalty0 211--218, 1936.
\bibitem[Ermolov et~al.(2021)Ermolov, Siarohin, Sangineto, and
Sebe]{Ermolov21Whitening}
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe.
\newblock Whitening for self-supervised representation learning.
\newblock In \emph{ICML}, pp.\ 3015--3024, 2021.
\bibitem[Everingham et~al.(2015)Everingham, Eslami, Gool, Williams, Winn, and
Zisserman]{Everingham15Pascal}
Mark Everingham, S.~M.~Ali Eslami, Luc~Van Gool, Christopher K.~I. Williams,
John~M. Winn, and Andrew Zisserman.
\newblock The pascal visual object classes challenge: {A} retrospective.
\newblock \emph{IJCV}, 111\penalty0 (1):\penalty0 98--136, 2015.
\bibitem[Gidaris et~al.(2018)Gidaris, Singh, and
Komodakis]{Gidaris2018Unsupervised}
Spyros Gidaris, Praveer Singh, and Nikos Komodakis.
\newblock Unsupervised representation learning by predicting image rotations.
\newblock 2018.
\bibitem[Goodfellow et~al.(2014)Goodfellow, Pouget{-}Abadie, Mirza, Xu,
Warde{-}Farley, Ozair, Courville, and Bengio]{Goodfellow14Generative}
Ian~J. Goodfellow, Jean Pouget{-}Abadie, Mehdi Mirza, Bing Xu, David
Warde{-}Farley, Sherjil Ozair, Aaron~C. Courville, and Yoshua Bengio.
\newblock Generative adversarial nets.
\newblock In \emph{NIPS}, pp.\ 2672--2680, 2014.
\bibitem[Grill et~al.(2020)Grill, Strub, Altch{\'{e}}, Tallec, Richemond,
Buchatskaya, Doersch, Pires, Guo, Azar, Piot, Kavukcuoglu, Munos, and
Valko]{Grill20Bootstrap}
Jean{-}Bastien Grill, Florian Strub, Florent Altch{\'{e}}, Corentin Tallec,
Pierre~H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo~{\'{A}}vila
Pires, Zhaohan Guo, Mohammad~Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu,
R{\'{e}}mi Munos, and Michal Valko.
\newblock Bootstrap your own latent - {A} new approach to self-supervised
learning.
\newblock In \emph{NeurIPS}, 2020.
\bibitem[Gutmann \& Hyv{\"{a}}rinen(2010)Gutmann and
Hyv{\"{a}}rinen]{Gutmann10Noise}
Michael Gutmann and Aapo Hyv{\"{a}}rinen.
\newblock Noise-contrastive estimation: {A} new estimation principle for
unnormalized statistical models.
\newblock In \emph{AISTATS}, pp.\ 297--304, 2010.
\bibitem[HaoChen et~al.(2021)HaoChen, Wei, Gaidon, and Ma]{Chen2021Provable}
Jeff~Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma.
\newblock Provable guarantees for self-supervised deep learning with spectral
contrastive loss.
\newblock In \emph{NeurIPS}, pp.\ 5000--5011, 2021.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{He16Deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{CVPR}, pp.\ 770--778, 2016.
\bibitem[He et~al.(2020{\natexlab{a}})He, Fan, Wu, Xie, and
Girshick]{he2020momentum}
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
\newblock Momentum contrast for unsupervised visual representation learning.
\newblock In \emph{CVPR}, pp.\ 9729--9738, 2020{\natexlab{a}}.
\bibitem[He et~al.(2020{\natexlab{b}})He, Fan, Wu, Xie, and
Girshick]{He20Momentum}
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross~B. Girshick.
\newblock Momentum contrast for unsupervised visual representation learning.
\newblock In \emph{CVPR}, pp.\ 9726--9735, 2020{\natexlab{b}}.
\bibitem[Hotelling(1933)]{hotelling1933analysis}
Harold Hotelling.
\newblock Analysis of a complex of statistical variables into principal
components.
\newblock \emph{Journal of educational psychology}, 24\penalty0 (6):\penalty0
417, 1933.
\bibitem[Hu et~al.(2021)Hu, Wang, Hu, and Qi]{hu2021adco}
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi.
\newblock Adco: Adversarial contrast for efficient learning of unsupervised
representations from self-trained negative adversaries.
\newblock In \emph{CVPR}, pp.\ 1074--1083, 2021.
\bibitem[Hu et~al.(2022)Hu, Liu, Zhou, Wang, and Huang]{Hu2022Your}
Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang.
\newblock Your contrastive learning is secretly doing stochastic neighbor
embedding.
\newblock \emph{CoRR}, abs/2205.14814, 2022.
\bibitem[Huang et~al.(2021)Huang, Yi, and Zhao]{Huang21Generalization}
Weiran Huang, Mingyang Yi, and Xuyang Zhao.
\newblock Towards the generalization of contrastive self-supervised learning.
\newblock \emph{CoRR}, abs/2111.00743, 2021.
\bibitem[Jolliffe(2002)]{jolliffe2002principal}
Ian~T Jolliffe.
\newblock \emph{Principal component analysis for special types of data}.
\newblock Springer, 2002.
\bibitem[Kalantidis et~al.(2020)Kalantidis, Sariyildiz, Pion, Weinzaepfel, and
Larlus]{kalantidis2020hard}
Yannis Kalantidis, Mert~Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and
Diane Larlus.
\newblock Hard negative mixing for contrastive learning.
\newblock \emph{NeurIPS}, 33:\penalty0 21798--21809, 2020.
\bibitem[Kingma \& Ba(2015)Kingma and Ba]{Kingma14Adam}
Diederik~P. Kingma and Jimmy Ba.
\newblock Adam: {A} method for stochastic optimization.
\newblock In \emph{ICLR}, 2015.
\bibitem[Kingma \& Welling(2014)Kingma and Welling]{Kingma13Auto}
Diederik~P. Kingma and Max Welling.
\newblock Auto-encoding variational bayes.
\newblock In \emph{ICLR}, 2014.
\bibitem[Kolesnikov et~al.(2019)Kolesnikov, Zhai, and
Beyer]{Kolesnikov19Revisiting}
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer.
\newblock Revisiting self-supervised visual representation learning.
\newblock In \emph{CVPR}, pp.\ 1920--1929, 2019.
\bibitem[Koohpayegani et~al.(2021{\natexlab{a}})Koohpayegani, Tejankar, and
Pirsiavash]{Koohpayegani2021Mean}
Soroush~Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash.
\newblock Mean shift for self-supervised learning.
\newblock In \emph{ICCV}, pp.\ 10326--10335, 2021{\natexlab{a}}.
\bibitem[Koohpayegani et~al.(2021{\natexlab{b}})Koohpayegani, Tejankar, and
Pirsiavash]{Koohpayegani21Mean}
Soroush~Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash.
\newblock Mean shift for self-supervised learning.
\newblock In \emph{ICCV}, pp.\ 10306--10315. {IEEE}, 2021{\natexlab{b}}.
\bibitem[Krizhevsky et~al.(2009)Krizhevsky, Hinton,
et~al.]{Krizhevsky2009Learning}
Alex Krizhevsky, Geoffrey Hinton, et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock 2009.
\bibitem[Lee et~al.(2021)Lee, Lei, Saunshi, and Zhuo]{Lee21Predicting}
Jason~D. Lee, Qi~Lei, Nikunj Saunshi, and Jiacheng Zhuo.
\newblock Predicting what you already know helps: Provable self-supervised
learning.
\newblock In \emph{NeurIPS}, pp.\ 309--323, 2021.
\bibitem[Li et~al.(2021)Li, Zhou, Xiong, and Hoi]{Li21Prototypical}
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.~H. Hoi.
\newblock Prototypical contrastive learning of unsupervised representations.
\newblock In \emph{ICLR}, 2021.
\bibitem[Mikolov et~al.(2013)Mikolov, Sutskever, Chen, Corrado, and
Dean]{Mikolov13Distributed}
Tom{\'{a}}s Mikolov, Ilya Sutskever, Kai Chen, Gregory~S. Corrado, and Jeffrey
Dean.
\newblock Distributed representations of words and phrases and their
compositionality.
\newblock In \emph{NIPS}, pp.\ 3111--3119, 2013.
\bibitem[Noroozi \& Favaro(2016)Noroozi and Favaro]{Noroozi2016Unsupervised}
Mehdi Noroozi and Paolo Favaro.
\newblock Unsupervised learning of visual representations by solving jigsaw
puzzles.
\newblock In \emph{ECCV}, pp.\ 69--84. Springer, 2016.
\bibitem[Pathak et~al.(2016)Pathak, Krahenbuhl, Donahue, Darrell, and
Efros]{Pathak2016Context}
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei~A
Efros.
\newblock Context encoders: Feature learning by inpainting.
\newblock In \emph{CVPR}, pp.\ 2536--2544, 2016.
\bibitem[Ren et~al.(2015)Ren, He, Girshick, and Sun]{Ren15Faster}
Shaoqing Ren, Kaiming He, Ross~B. Girshick, and Jian Sun.
\newblock Faster {R-CNN:} towards real-time object detection with region
proposal networks.
\newblock In Corinna Cortes, Neil~D. Lawrence, Daniel~D. Lee, Masashi Sugiyama,
and Roman Garnett (eds.), \emph{NIPS}, pp.\ 91--99, 2015.
\bibitem[Reyment \& Jvreskog(1996)Reyment and Jvreskog]{reyment1996applied}
Richard~A Reyment and KG~Jvreskog.
\newblock \emph{Applied factor analysis in the natural sciences}.
\newblock Cambridge University Press, 1996.
\bibitem[Saunshi et~al.(2019)Saunshi, Plevrakis, Arora, Khodak, and
Khandeparkar]{Saunshi19Theoretical}
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and
Hrishikesh Khandeparkar.
\newblock A theoretical analysis of contrastive unsupervised representation
learning.
\newblock In \emph{ICML}, pp.\ 5628--5637, 2019.
\bibitem[Tian et~al.(2020{\natexlab{a}})Tian, Krishnan, and
Isola]{Tian20Contrastive}
Yonglong Tian, Dilip Krishnan, and Phillip Isola.
\newblock Contrastive multiview coding.
\newblock In \emph{ECCV}, pp.\ 776--794, 2020{\natexlab{a}}.
\bibitem[Tian et~al.(2020{\natexlab{b}})Tian, Sun, Poole, Krishnan, Schmid, and
Isola]{Tian20What}
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and
Phillip Isola.
\newblock What makes for good views for contrastive learning?
\newblock In \emph{NeurIPS}, 2020{\natexlab{b}}.
\bibitem[Tosh et~al.(2021)Tosh, Krishnamurthy, and Hsu]{Tosh21Contrastive}
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu.
\newblock Contrastive learning, multi-view redundancy, and linear models.
\newblock In \emph{Algorithmic Learning Theory}, pp.\ 1179--1206, 2021.
\bibitem[Tsai et~al.(2021)Tsai, Bai, Morency, and Salakhutdinov]{Tsai2021Note}
Yao{-}Hung~Hubert Tsai, Shaojie Bai, Louis{-}Philippe Morency, and Ruslan
Salakhutdinov.
\newblock A note on connecting barlow twins with negative-sample-free
contrastive learning.
\newblock \emph{CoRR}, abs/2104.13712, 2021.
\bibitem[van~den Oord et~al.(2018)van~den Oord, Li, and
Vinyals]{oord2018representation}
A{\"{a}}ron van~den Oord, Yazhe Li, and Oriol Vinyals.
\newblock Representation learning with contrastive predictive coding.
\newblock \emph{CoRR}, abs/1807.03748, 2018.
\bibitem[Vidal et~al.(2016)Vidal, Ma, and Sastry]{Vidal16Generalized}
Ren{\'{e}} Vidal, Yi~Ma, and S.~Shankar Sastry.
\newblock \emph{Generalized Principal Component Analysis}, volume~40 of
\emph{Interdisciplinary applied mathematics}.
\newblock Springer, 2016.
\bibitem[Wang \& Isola(2020)Wang and Isola]{Wang20Understanding}
Tongzhou Wang and Phillip Isola.
\newblock Understanding contrastive representation learning through alignment
and uniformity on the hypersphere.
\newblock In \emph{ICML}, pp.\ 9929--9939, 2020.
\bibitem[Wang et~al.(2022)Wang, Zhang, Wang, Yang, and Lin]{Wang2022Chaos}
Yifei Wang, Qi~Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin.
\newblock Chaos is a ladder: {A} new theoretical understanding of contrastive
learning via augmentation overlap.
\newblock \emph{CoRR}, abs/2203.13457, 2022.
\bibitem[Wu et~al.(2018)Wu, Xiong, Yu, and Lin]{wu2018unsupervised}
Zhirong Wu, Yuanjun Xiong, Stella~X Yu, and Dahua Lin.
\newblock Unsupervised feature learning via non-parametric instance
discrimination.
\newblock In \emph{CVPR}, pp.\ 3733--3742, 2018.
\bibitem[Xiao et~al.(2021)Xiao, Wang, Efros, and Darrell]{Xiao21What}
Tete Xiao, Xiaolong Wang, Alexei~A. Efros, and Trevor Darrell.
\newblock What should not be contrastive in contrastive learning.
\newblock In \emph{ICLR}, 2021.
\bibitem[Zbontar et~al.(2021)Zbontar, Jing, Misra, LeCun, and
Deny]{zbontar2021barlow}
Jure Zbontar, Li~Jing, Ishan Misra, Yann LeCun, and St{\'e}phane Deny.
\newblock Barlow twins: Self-supervised learning via redundancy reduction.
\newblock In \emph{ICML}, pp.\ 12310--12320. PMLR, 2021.
\bibitem[Zhang et~al.(2016)Zhang, Isola, and Efros]{Zhang16Colorful}
Richard Zhang, Phillip Isola, and Alexei~A. Efros.
\newblock Colorful image colorization.
\newblock In \emph{ECCV}, pp.\ 649--666, 2016.
\bibitem[Zhao et~al.(2021)Zhao, Wu, Lau, and Lin]{Zhao21What}
Nanxuan Zhao, Zhirong Wu, Rynson W.~H. Lau, and Stephen Lin.
\newblock What makes instance discrimination good for transfer learning?
\newblock In \emph{ICLR}, 2021.
\bibitem[Zitova \& Flusser(2003)Zitova and Flusser]{zitova2003image}
Barbara Zitova and Jan Flusser.
\newblock Image registration methods: a survey.
\newblock \emph{Image and vision computing}, 21\penalty0 (11):\penalty0
977--1000, 2003.
\end{thebibliography}