Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
Extracted source text · arXiv 2306.05268
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2022
% ready for submission
\PassOptionsToPackage{numbers, sort&compress}{natbib}
\usepackage[final]{neurips_2023}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2022}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2022}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2022}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{comment} % multiline comments
\usepackage{amsmath}
\usepackage{multirow}
\usepackage{bm}
\usepackage{amsthm}
\usepackage{hhline}
\usepackage{amssymb}
\usepackage{makecell}
\usepackage{mathtools}
\usepackage{color}
\usepackage{xcolor}
\usepackage{MnSymbol}
\usepackage{makecell}
\usepackage{graphicx}
\usepackage{arydshln}
\usepackage{booktabs}
\usepackage{framed}
\usepackage[font=small]{caption}
\usepackage[scientific-notation=true]{siunitx}
\usepackage{wrapfig}
\usepackage{lipsum}
\usepackage{sidecap}
\usepackage{pifont}% http://ctan.org/pkg/pifont
\usepackage{fixltx2e}
\usepackage[flushleft]{threeparttable}
\usepackage{diagbox}
\usepackage{algorithm}
\usepackage{algorithmic}
\newtheorem{definition}{Definition}
\newtheorem{assumption}{Assumption}
\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}
\definecolor{gg}{RGB}{15,150,15}
\definecolor{rr}{RGB}{230,45,45}
\newcommand\blfootnote[1]{%
\begingroup
\renewcommand\thefootnote{}\footnote{#1}%
\addtocounter{footnote}{-1}%
\endgroup
}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\newcommand{\todo}[1]{\textcolor{red}{TODO: #1}}
\newcommand{\martin}[1]{\textcolor{olive}{Martin: #1}}
\newcommand{\james}[1]{{\color{red} {\bf James:} #1}}
\newcommand{\camera}[1]{{\color{cyan} {\bf} #1}}
\newcommand{\namel}{\textsc{Factorized Contrastive Learning}}
\newcommand{\names}{\textsc{FactorCL}}
\newcommand{\indep}{\perp\!\!\!\perp}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
% Recommended, but optional, packages for figures and better typesetting:
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\title{\namel:\\Going Beyond Multi-view Redundancy}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Paul Pu Liang$^{1*}$, Zihao Deng$^{2*}$, Martin Q. Ma$^{1*}$\\
\textbf{James Zou$^{3}$, Louis-Philippe Morency$^{1}$, Ruslan Salakhutdinov$^{1}$}\\
$^{1}$Carnegie Mellon University, $^{2}$University of Pennsylvania, $^{3}$Stanford University\\
\texttt{pliang@cs.cmu.edu,zihaoden@cs.cmu.edu,qianlim@cs.cmu.edu} \\
}
\begin{document}
\maketitle
\vspace{-8mm}
\begin{abstract}
\vspace{-2mm}
In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data with only pairing information (e.g., image-caption or video-audio pairs). Underpinning these approaches is the assumption of \textit{multi-view redundancy} - that shared information between modalities is necessary and sufficient for downstream tasks.
However, in many real-world settings, task-relevant information is also contained in modality-unique regions: information that is only present in one modality but still relevant to the task.
How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks? This paper proposes \names, a new multimodal representation learning method to go beyond multi-view redundancy. \names\ is built from three new contributions: (1) factorizing task-relevant information into shared and unique representations, (2) capturing task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bounds, and (3) multimodal data augmentations to approximate task relevance without labels. On large-scale real-world datasets, \names\ captures both shared and unique information and achieves state-of-the-art results on six benchmarks. \blfootnote{$^*$First three authors contributed equally.}
\end{abstract}
\vspace{-6mm}
\section{Introduction}
\vspace{-2mm}
Learning representations from different modalities is a central paradigm in machine learning~\cite{liang2022foundations}.
Today, a popular learning method is to first pre-train general representations on unlabeled multimodal data before fine-tuning on task-specific labels~\cite{bugliarello2021multimodal,kenton2019bert,liang2022foundations,liang2022highmmt,lu2019vilbert}. These current multimodal pre-training approaches have largely been inherited from prior work in multi-view learning~\cite{chen2020simple,oord2018representation} that exploit a critical assumption of \textit{multi-view redundancy}: the property that shared information between modalities is almost exactly what is relevant for downstream tasks~\cite{sridharan2008information,tosh2021contrastive,tsai2020self}.
When this assumption holds, approaches based on contrastive pre-training to capture shared information~\cite{chen2020simple,khosla2020supervised,radford2021learning,tian2020makes}, followed by fine-tuning to keep task-relevant shared information~\cite{tsai2020self}, have seen successful applications in learning from images and captions~\cite{radford2021learning}, video and audio~\cite{arandjelovic2017look}, speech and transcribed text~\cite{oord2018representation}, and instructions and actions~\cite{eysenbach2022contrastive}.
However, our paper studies two fundamental limitations in the application of contrastive learning (CL) to broader real-world multimodal settings (see Figure~\ref{fig:overview} for a visual depiction and experimental results showing the performance drop of CL):
\begin{enumerate}[noitemsep,topsep=0pt,nosep,leftmargin=*,parsep=0pt,partopsep=0pt]
\item \textbf{Low \textit{shared} information} relevant to tasks: There exists a wide range of multimodal tasks involving small amounts of shared information, such as between cartoon images and figurative captions (i.e., not literal but metaphoric or idiomatic descriptions of the images~\cite{marsh2003taxonomy, yosef2023irfl}).
In these situations, standard multimodal CL will only receive a small percentage of information from the learned representations and struggle to learn the desired task-relevant information.
\item \textbf{High \textit{unique} information} relevant to tasks: Many real-world modalities can provide unique information not present in other modalities. Examples include healthcare with medical sensors or robotics with force sensors~\cite{liang2021multibench,liang2023quantifying}. Standard CL will discard task-relevant unique information, leading to poor downstream performance.
\end{enumerate}
\input{figures/overview_fig}
In light of these limitations, how can we design suitable multimodal learning objectives that work beyond multi-view redundancy? In this paper, starting from the first principles in information theory, we provide formal definitions of shared and unique information via conditional mutual information and propose an approach, \namel\ (\names\ for short), to learn these multimodal representations beyond multi-view redundancy using three key ideas. The first idea is to explicitly \textit{factorize} shared and unique representations. The second idea is to \textit{capture task-relevant} information via maximizing lower bounds on MI and \textit{remove task-irrelevant} information via minimizing upper bounds on MI, resulting in representations with sufficient and necessary information content. Finally, a notion of task relevance without explicit labels in the self-supervised setting is achieved by leveraging \textit{multimodal augmentations}.
Experimentally, we evaluate the effectiveness of \names\ on a suite of synthetic datasets and large-scale real-world multimodal benchmarks involving images and figurative language~\cite{yosef2023irfl}, prediction of human sentiment~\cite{zadeh2016mosi}, emotions~\cite{zadeh2018multimodal}, humor~\cite{hasan2019ur}, and sarcasm~\cite{castro2019towards}, as well as patient disease and mortality prediction from health indicators and sensor readings~\cite{johnson2016mimic}, achieving new state-of-the-art performance on six datasets. Overall, we summarize our key technical contributions here:
\begin{enumerate}[noitemsep,topsep=0pt,nosep,leftmargin=*,parsep=0pt,partopsep=0pt]
\item A new analysis of contrastive learning performance showing that standard multimodal CL fails to capture task-relevant unique information under low shared or high unique information cases.
\item A new contrastive learning algorithm called \names:
\begin{enumerate}[noitemsep,topsep=0pt,nosep,leftmargin=*,parsep=0pt,partopsep=0pt]
\item \names\ factorizes task-relevant information into shared and unique information, expanding contrastive learning to better handle low shared or high unique information.
\item \names\ optimizes shared and unique information separately, by removing task-irrelevant information via MI upper bounds and capturing task-relevant information via lower bounds, yielding optimal task-relevant representations.
\item \names\ leverages multimodal augmentations to approximate task-relevant information, enabling self-supervised learning from our proposed \names.
\end{enumerate}
\end{enumerate}
\vspace{-3mm}
\section{Analysis of Multi-view Contrastive Learning}
\vspace{-2mm}
We begin by formalizing definitions of four types of information: shared, unique, task-relevant, and task-irrelevant information in multimodal data. To formalize the learning setting, we assume there exist two modalities expressed as random variables $X_1$ and $X_2$ with outcomes $x_1$ and $x_2$, and a task with the random variable $Y$ and outcome $y$. We denote $X_{-i}$ as the other modality where appropriate.
\textbf{Shared and unique information}: We formalize shared and unique information by decomposing the total multimodal information $I (X_1,X_2; Y)$ into three conditional mutual information (MI) terms:
\begin{align}
\label{eq:sharedunique}
I (X_1,X_2; Y) = \underbrace{I(X_1;X_2;Y)}_\text{$S=\textrm{shared}$} + \underbrace{I(X_1;Y|X_2)}_\text{$U_1=\textrm{uniqueness in } X_1$} + \underbrace{I(X_2;Y|X_1)}_\text{$U_2=\textrm{uniqueness in } X_2$},
\end{align}
where $I (X_1,X_2; Y) = \int p(x_1,x_2,y) \log \frac{p(x_1,x_2,y)}{p(x_1,x_2) p(y)} dx_1 dx_2 dy$ is the total MI between the joint random variable $X_1,X_2$ and the task $Y$, $S=I(X_1;X_2;Y) = I(X_1;X_2) - I(X_1;X_2|Y) = \int p(x_1,x_2) \log \frac{p(x_1,x_2)}{p(x_1) p(x_2)} dx_1 dx_2 - I(X_1;X_2|Y)$ is the task-relevant shared information, $I(X_1;X_2|Y)=\int p(x_1,x_2|y) \log \frac{p(x_1,x_2|y)}{p(x_1|y) p(x_2|y)} dx_1 dx_2 dy$ is the task-irrelevant shared information, and $U_1=I(X_1;Y|X_2)$, $U_2=I(X_2;Y|X_1)$ denote unique task-relevant information.
\textbf{Limitations of CL}: Current approaches for CL maximize mutual information $I(X_1;X_2)$ (and subsequently task-relevant shared information $I(X_1;X_2;Y)$ during supervised fine-tuning), without modeling unique information. These methods generally learn a pair of representations \cite{tosh2021contrastive,tsai2020self},
\begin{align}
Z_1 = \argmax_{Z_1 := f_\theta(X_1)} I(Z_1;X_2), \quad Z_2 = \argmax_{Z_2 := f_\theta(X_2)} I(X_1;Z_2). \label{eq:standard_cl_z}
\end{align}
For example, $Z_1$ could encode images $X_1$ and $Z_2$ encodes text $X_2$ via maximizing a lower bound on $I(X_1;X_2)$ using the NCE objective~\cite{oord2018representation}. The NCE objective falls into a broader class of contrastive learning methods~\cite{chen2020simple,he2020momentum,radford2021learning, chen2021empirical, khosla2020supervised} that model the ratio between joint densities $p(x_1,x_2)$ and product of marginal densities $p(x_1)p(x_2)$ using positive and negative samples~\cite{nguyen2010estimating, poole2019variational, tschannen2019mutual, wu2020mutual, ozair2019wasserstein} or probabilistic classifiers~\cite{mukherjee2020ccmi,tsai2020neural}. It has been shown that contrastive learning works well under the assumption of multi-view redundancy~\cite{sridharan2008information, hjelm2018learning, bachman2019learning, tian2019contrastive, tsai2020self}:
\begin{definition}
\label{eq:multiview_redundancy_assump}
(Multi-view redundancy) $\exists \epsilon >0$ such that $I(X_1;Y|X_2) \le \epsilon$ and $I(X_2;Y|X_1) \le \epsilon$.
\end{definition}
In other words, the task-relevant information in data is mostly shared across both views and the unique information is at most a small $\epsilon$. From a representation perspective, \citet{tian2020makes} further introduces the assumption that the optimal representation is minimal and sufficient, where all learned task-relevant information is shared information: $I(Z_1; Y | X_2)=I(Z_2; Y | X_1)=0$. While the multi-view redundancy is certainly true for particular types of multimodal distributions, it crucially ignores settings that display \textit{multi-view non-redundancy} and unique information can be important, such as when health indicators, medical sensors, and robotic visual or force sensors each provide unique information not present in other modalities~\cite{liang2021multibench,liang2023quantifying}.
\begin{definition}
\label{eq:multiview_nonredundancy_assump}
(Multi-view non-redundancy) $\exists \epsilon >0$ such that $I(X_1;Y|X_2) > \epsilon$ or $I(X_2;Y|X_1) > \epsilon$.
\end{definition}
Under multi-view non-redundancy, we show that standard CL only receives a weak training signal since it can only maximize a lower bound on shared information $I(X_1;X_2)$, and struggles to learn task-relevant unique information. We formalize this intuition with the following statement:
\begin{theorem}
\label{thm:suboptimal_eq}
(Suboptimality of standard CL) When there is multi-view non-redundancy as in Definition \ref{eq:multiview_nonredundancy_assump}, given optimal representations $\{Z_1,Z_2\}$ that satisfy Eq.(\ref{eq:standard_cl_z} and $I(Z_1; Y | X_2)=I(Z_2; Y | X_1)=0$~\citep{tian2020makes}, we have that
{\small
\begin{align}
I(Z_1,Z_2;Y) = I(X_1,X_2;Y) - I(X_1;Y|X_2) - I(X_2;Y|X_1) = I(X_1;X_2) - I(X_1;X_2|Y) < I(X_1,X_2;Y). \label{eq:suboptimal_eq}
\end{align}}Correspondingly, the Bayes error rate $P_e(Z_1,Z_2):=1 - \mathbb{E}_{p(z_1, z_2)}\left[\max_{y\in Y} P\left(\hat{Y}=y \mid z_1, z_2 \right)\right]$ of contrastive representations $\{Z_1,Z_2\}$ for a downstream task $Y$ is given by:
\begin{align}
P_e &\le 1 - \exp \left[I(X_1, X_2; Y)- I(X_1;Y|X_2) - I(X_2;Y|X_1) - H(Y) \right] \\
&= 1 - \exp \left[I(X_1;X_2;Y) - H(Y) \right]
\label{eq:bayes_error}
\end{align}
\end{theorem}
We include proofs and a detailed discussion of the assumptions in Appendix \ref{app:discussion_assumption}.
Based on Eq.(\ref{eq:suboptimal_eq}), $I(Z_1,Z_2;Y)$ decreases with higher task-relevant unique information $I(X_1;Y|X_2)$ and $I(X_2;Y|X_1)$; we call this the difference $I(X_1, X_2 ; Y) - I(Z_1, Z_2; Y)$ the $\textit{uniqueness gap}$. The uniqueness gap measures the loss in task-relevant information between the input and encoded representation: as task-relevant unique information grows, the uniqueness gap increases. In addition, $I(Z_1,Z_2;Y)$ also drops with lower $I(X_1; X_2)$ (i.e., two modalities sharing little information to begin with), or with higher $I(X_1;X_2|Y)$ (i.e., when the shared information is mostly task-irrelevant). Similarly, in Eq.(\ref{eq:bayes_error}), the Bayes error rate of using $\{Z_1,Z_2\}$ for prediction is directly related to the task-relevant information in $\{Z_1,Z_2\}$: error on the downstream task increases with higher unique information and lower shared information.
\vspace{-2mm}
\section{\namel}
\vspace{-2mm}
We now present a suite of new CL objectives that alleviate the challenges above and work at all ranges of shared and unique information.
At a high level, we aim to learn a set of factorized representations $Z_{S_1},Z_{S_2},Z_{U_1},Z_{U_2}$ representing task-relevant information in $X_1$ shared with $X_2$, in $X_2$ shared with $X_1$, unique to $X_1$, and unique to $X_2$ respectively.
As common in practice~\cite{radford2021learning,tian2020makes}, we define neural networks $f_{\theta}$ with trainable parameters $\theta$ to extract representations from inputs $X_1$ and $X_2$. Learning these parameters requires optimizing differentiable and scalable training objectives to capture task-relevant shared and unique information (see overview in Figure~\ref{fig:method}):
\begin{align}
Z_{S_1} &= \argmax_{Z_1 = f_\theta(X_1)} I(Z_1;X_2;Y), &&Z_{S_2} = \argmax_{Z_2 = f_\theta(X_2)} I(Z_2;X_1;Y), \label{eq:final_objectives1} \\
Z_{U_1} &= \argmax_{Z_1 = f_\theta(X_1)} I(Z_1;Y|X_2), &&Z_{U_2} = \argmax_{Z_2 = f_\theta(X_2)} I(Z_2;Y|X_1). \label{eq:final_objectives2}
\end{align}
where $I(Z_1;X_2;Y) = I(Z_1; X_2) - I(Z_1; X_2 | Y)$ is the shared information and $I(Z_2;X_1;Y) = I(Z_2; X_2) - I(Z_2; X_1 | Y)$ is the unique information. One important characteristic of our framework is that when unique information is zero: $I(X_1; Y|X_2) =0$ and $I(X_2; Y|X_1) = 0$, or all shared information is task-relevant: $I(X_1;X_2;Y) = I(X_1; X_2)$, our framework recovers standard CL as in Eq.(\ref{eq:standard_cl_z}). However, as we have previously indicated and will show empirically, these assumptions can easily be violated, and our framework enlarges Eq.(\ref{eq:standard_cl_z}) to cases where unique information is present.
The learned $Z$s can then be used as input to a linear classifier and fine-tuned to predict the label for multimodal classification or retrieval tasks. However, the shared and unique MI terms above are often intractable in practice. In the next section, we will build up our method step by step, eventually showing that each term in Eqs.(\ref{eq:final_objectives1}- \ref{eq:final_objectives2}) can be approximated as follows:
\begin{align}
S = I(X_1; X_2; Y) &\ge I_\textsc{NCE}(X_1;X_2) - I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') \label{eq:shared}\\
U_i = I(X_i; Y | X_{-i}) &\ge I_\textsc{NCE}(X_i;X_i') - I_\textsc{NCE-CLUB}(X_1;X_2) + I_\textsc{NCE}(X_1;X_2|X_1',X_2') \label{eq:unique}
\end{align}
where $I_\textsc{NCE}$ and $I_\textsc{NCE-CLUB}$ are scalable contrastive estimators (Section \ref{subsec:objectives}) and $X_1',X_2'$ are suitable data augmentations (Section \ref{subsec:unique_aug}) on each modality. Overall, these equations can be interpreted as both positive and negative signals to learn representations for $S$ and $U$. For shared information $S$, the estimator maximizes task-relevant shared information via $I_\textsc{NCE}(X_1;X_2)$ while removing task-irrelevant shared information via a novel upper bound $- I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2')$. For unique information $U_i$, we capture task-relevant uniqueness via $+I_\textsc{NCE}(X_i;X_i')$ while non-unique information is removed via $- (I_\textsc{NCE-CLUB}(X_1; X_2) - I_\textsc{NCE}(X_1;X_2|X_1',X_2'))$.
In the following sections, we derive this final objective step-by-step: (1) approximating the MI objectives in $S$ and $U$ with CL estimators, (2) relaxing the dependence on labels $Y$ with self-supervised data augmentations, finally (3) discussing overall training and implementation details of end-to-end self-supervised learning.
\subsection{Supervised \names\ with shared and unique information}
\label{subsec:objectives}
\input{figures/method_fig}
To capture shared and unique information via an objective function, we will need to maximize lower bounds for all terms with a positive sign in Eq.(\ref{eq:shared}) and (\ref{eq:unique}) $\left( I\left(X_1; X_2\right), I\left(X_i; Y\right), I\left(X_1; X_2 | Y\right)\right)$ and minimize upper bounds for all terms with a negative sign $\left( I\left(X_1; X_2\right), I\left(X_1; X_2 | Y\right)\right)$. Our first theorem derives general lower and upper bounds for MI terms as variants of contrastive estimation:
\begin{theorem}
(Contrastive estimators for $I(X_1; X_2)$) Defining the NCE and NCE-CLUB estimators,
\begin{align}
I_\textsc{NCE}(X_1; X_2) &= \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2)\\x_2^- \sim p(x_2)}} \left[ \log \frac{\exp f(x_1,x_2^+)}{\sum_k \exp f(x_1, x_2^-)} \right] \label{eq:nce_original}\\
I_\textsc{NCE-CLUB}(X_1; X_2) &= \mathbb{E}_{x_1,x_2^+ \sim p(x_1,x_2)} \left[ f^*(x_1,x_2^+) \right] - \mathbb{E}_{\substack{x_1 \sim p(x_1)\\x_2^- \sim p(x_2)}} \left[f^*(x_1,x_2^-) \right] \label{eq:nceclub}
\end{align}
where $f^*(x_1,x_2)$ is the optimal critic from $I_\textsc{NCE}$ plugged into the $I_\textsc{CLUB}$ objective \cite{cheng2020club}. We call the proposed plug-in objective Eq.(\ref{eq:nceclub}) $I_\textsc{NCE-CLUB}$, and obtain lower and upper bounds on $I(X_1; X_2)$:
\begin{align}
I_\textsc{NCE}(X_1; X_2) \le I(X_1; X_2) \le I_\textsc{NCE-CLUB}(X_1; X_2).
\end{align}
\end{theorem}
\begin{proof}
The lower bound $I_\textsc{NCE}(X_1; X_2) \le I(X_1; X_2)$ follows from~\citet{oord2018representation}: optimizing the objective leads to an optimal critic \cite{poole2019variational} $f^* = \log p(x_1 | x_2) + c(x_1)$, with a deterministic function $c(\cdot)$. Plugging optimal critic $f^*$ into $I_\textsc{NCE-CLUB}(X_1; X_2)$ cancels out the $c(x_1)$ term and yields $I_\textsc{NCE-CLUB}(X_1; X_2)$ and $I(X_1; X_2) \le I_\textsc{NCE-CLUB}$. We include a detailed proof in Appendix~\ref{app:method1}.
\end{proof}
$I_\textsc{NCE-CLUB}(X_1; X_2)$ gives a desired upper bound of $I(X_1; X_2)$ ``for free'' while avoiding separately optimizing lower bound and upper bounds. In Figure~\ref{fig:bounds}, we show these two bounds in practice across two Gaussian distributions $X_1$ and $X_2$ with varying amounts of MI $I(X_1;X_2)$. We use the second formulation of $I_\textsc{CLUB}$ \cite{cheng2020club}, which assumes $p(x_1|x_2)$ to be unknown.
Our upper bound is empirically tighter (see Figure~\ref{fig:bounds}) and comes for ``free'' via jointly maximizing the lower bound $I_\textsc{NCE}$. These lower and upper bounds can be seen as new contrastive objectives over positive and negative $(x_1,x_2)$ pairs, enabling a close integration with existing pre-training paradigms.
Finally, we can similarly obtain bounds for the conditional MI $I_\textsc{NCE}(X_1;X_2|Y) \le I(X_1;X_2|Y) \le I_\textsc{NCE-CLUB}(X_1;X_2|Y)$:
{\small
\begin{align}
I_\textsc{NCE}(X_1;X_2|Y) &= \mathbb{E}_{p(y)} \left[ \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2|y)\\x_2^- \sim p(x_2|y)}} \left[ \log \frac{\exp f(x_1,x_2^+, y)}{\sum_k \exp f(x_1, x_2^-, y)} \right] \right] \label{eq:conditional_nce_supervised} \\
I_\textsc{NCE-CLUB}(X_1;X_2|Y) &= \mathbb{E}_{p(y)} \left[ \mathbb{E}_{x_1,x_2^+ \sim p(x_1,x_2|y)} \left[ f^*(x_1,x_2^+, y) \right] - \mathbb{E}_{\substack{x_1 \sim p(x_1|y)\\x_2^- \sim p(x_2|y)}} \left[ f^*(x_1,x_2^-, y) \right] \right] \label{eq:conditional_club_supervised}
\end{align}
}
These two bounds result in \textit{conditional CL} objectives \cite{tsailearning, tsai2022conditional, ma2021conditional} - they differ critically from standard CL methods since they capture task-irrelevant shared information that remains between $X_1$ and $X_2$ after observing $Y$. This task-irrelevant shared information is removed by minimizing its upper bound. Note that $f(x_1, x_2, y)$ here denotes a different function from $f(x_1, x_2)$ in Eq.(\ref{eq:nce_original}), as the general forms are different (taking in $x_1, x_2$ versus $x_1, x_2, y$). $f(x_1, x_2, y)$ can be implemented in different ways, e.g., $g([x_1, y])^Th(x_2)$ where $g(), h()$ are trainable encoders and $[x_1, y]$ denotes concatenation \citep{sordoni2021decomposed}.
\input{figures/bounds_fig}
\subsection{Self-supervised \names\ via multimodal augmentations}
\label{subsec:unique_aug}
The derivations above bring about supervised CL objectives with access to $Y$~\cite{khosla2020supervised}. For unsupervised CL~\cite{tian2020makes,oord2018representation}, we derive similar objectives without access to $Y$ by leveraging semantic augmentations on each modality. Denote $X'$ as some augmentation of $X$ (e.g., rotating, shifting, or cropping). Under the \textit{optimal augmentation} assumption from~\citet{tian2020makes} (restated below), replacing $Y$ with $X'$ in our formulations enables learning of task-relevant information without access to labels:
\begin{definition}
\label{def:optimal_single}
(Optimal unimodal augmentation)~\cite{tian2020makes} $X_1'$ is an optimal unimodal augmentation for $X_1$ when $I(X; X') = I(X; Y)$, which implies that the only information shared between $X$ and $X'$ is task-relevant with no irrelevant noise.
\end{definition}
This assumption is satisfied when all information shared between $X$ and $X'$ is task-relevant, which implies that the augmentation keeps task-relevant information constant while changing task-irrelevant information. In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background.
By performing two separate unimodal augmentations giving $X_1'$ and $X_2'$, we can substitute contrastive estimators in Eqs.(\ref{eq:conditional_nce_supervised}) and (\ref{eq:conditional_club_supervised}), by replacing $I(X_i;Y)$ terms with $I(X_i;X_i')$ and replacing $I(X_1;X_2|Y)$ terms with $I(X_1;X_2|X_1', X_2')$:
{\small
\begin{align}
I_\textsc{NCE}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1', x_2')} \left[ \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2|x_1', x_2')\\x_2^- \sim p(x_2|x_1', x_2')}} \left[ \log \frac{\exp f(x_1,x_2^+, x_1', x_2')}{\sum_k \exp f(x_1, x_2^-, x_1', x_2')} \right] \right] \label{eq:nce_final_ssl} \\
I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1', x_2')} \Big[ \mathbb{E}_{x_1,x_2^+ \sim p(x_1,x_2|x_1', x_2')} [f^*(x_1,x_2^+, x_1', x_2') ] \nonumber \\
&- \mathbb{E}_{\substack{x_1 \sim p(x_1|x_1', x_2')\\x_2^- \sim p(x_2|x_1', x_2')}} [f^*(x_1,x_2^-, x_1', x_2') ] \Big] \label{eq:nceclub_final_ssl}
\end{align}
}The objectives can be seen as conditional contrastive learning on augmentations $(X_1',X_2'$). Here again $f(x_1, x_2, x_1', x_2')$ is different from the critics in Eqs.(\ref{eq:conditional_nce_supervised} because of the different general forms. We implement $f()$ here as $g([x_1, x_1'])^Th([x_2, x_2'])$ where $g(), h()$ are trainable encoders specific for each modality and $[x_1, x_1']$ denotes concatenation. This concatenation is justified by the CMI estimators in~\citet{sordoni2021decomposed}, who show that concatenating the conditioning variable with the input in the critic $f(x_1, x_2, x_1', x_2')$ yields a Conditional InfoNCE estimator (Eq.(\ref{eq:nce_final_ssl})) that is a lower bound for CMI. However, the exact Conditional InfoNCE estimator learns a different conditional distribution $p(x_1, x_2 | x_1', x_2')$ for each augmented pair $x_1',x_2'$, which can be prohibitively expensive. We could approximate this by creating multiple augmentations of a single paired $x_1, x_2$. Our code uses one augmented pair $x_1', x_2'$ for each $x_1, x_2$ but could be extended to multiple pairs, and we find this simple approach yields consistent CMI lower and upper bounds that are empirically comparable to existing CMI estimators~\cite{mukherjee2020ccmi,sordoni2021decomposed}. We include full comparisons and implementation details in Appendix~\ref{app:discussion_conditional}, and in Appendix~\ref{app:method2} we discuss an alternative interpretation based on viewing CL as kernel learning which permits using conditional kernel estimation for our objectives.
\input{figures/unique_augment_fig}
Although we find this method to work well in practice, a more careful analysis reveals that $2$ separate unimodal augmentations $X_1'$ and $X_2'$ each satisfying $I(X_i; X_i') = I(X_i; Y)$ do not together satisfy $I(X_1;X_2|Y)=I(X_1;X_2|X_1', X_2')$ needed for the substitution in Eqs.(\ref{eq:nce_final_ssl}) and (\ref{eq:nceclub_final_ssl}) to hold with equality. To satisfy this property exactly, we define optimal multimodal augmentations:
\begin{definition}
\label{def:optimal_multi}
(Optimal multimodal augmentation) $X_1'$ and $X_2'$ are optimal multimodal augmentation for $X_1$ and $X_2$ when $I(X_1,X_2; X_1',X_2') = I(X_1,X_2; Y)$, which implies that the only information shared between $X_1,X_2$ and $X_1',X_2'$ is task-relevant with no irrelevant noise.
\end{definition}
We satisfy $I(X_1,X_2; X_1',X_2') = I(X_1,X_2; Y)$ using two steps:
\begin{align}
\textit{Unimodal aug: } &X_1' \textrm{ s.t. } I(X_1; X_1') = I(X_1; Y), \label{assump:unimodal_aug}\\
\textit{Unique aug: } &X_2' \textrm{ s.t. } I(X_2; X_2'|X_1) = I(X_2; Y|X_1). \label{assump:unique_aug}
\end{align}
We call the second step \textit{unique augmentation}: after observing $X_1$, we create augmented $X_2'$ from $X_2$ to keep task-relevant information not already in $X_1$. To empirically satisfy optimal multimodal augmentations, we avoid augmentations in one modality that will remove or strongly destroy information shared with the other modality. For example, in image captioning, we should avoid image augmentations such as cropping that destroy information from the caption (e.g., cropping object parts referred to by the caption), and instead, only augment images via flipping or color jittering which retains all caption information. Figure~\ref{fig:augs} shows an example of unique augmentation that satisfies these conditions. In our experiments, we will show that our augmentations consistently perform better than standard augmentations (Table \ref{tab:fig}), suggesting that approximately satisfying Eqs.(\ref{assump:unimodal_aug}) and (\ref{assump:unique_aug}) can be empirically sufficient, which is simple and straightforward to implement on real-world datasets.
\input{figures/alg}
\vspace{-2mm}
\subsection{Overall method and implementation}
\vspace{-2mm}
The final algorithm sketch is in Algorithm~\ref{alg:full}, which we compare against standard CL in Algorithm~\ref{alg:standardcl}. It can be shown that \names\ learns all the task-relevant information from both modalities:
\begin{theorem}
(Optimality of \names) If $Z_{S_1},Z_{S_2},Z_{U_1},Z_{U_2}$ perfectly maximize Eqs.(\ref{eq:final_objectives1}-\ref{eq:final_objectives2}) and the estimations in Eqs.(\ref{eq:shared}) and (\ref{eq:unique}) are tight, we obtain $I(X_1, X_2 ; Y) = I(Z_{S_1}; Z_{S_2}; Y) + I(Z_{U_1}; Y | Z_{S_2}) + I(Z_{U_2}; Y | Z_{S_1})$, suggesting that \names\ learns both shared and unique task-relevant information.
\end{theorem}
We include the full proof in Appendix~\ref{app:method3}. In practice, while we do not expect perfect estimation of MI quantities and maximization with respect to MI objectives, we include implementation details regarding architectures and contrastive objectives that improve empirical performance in Appendix~\ref{app:implementation}.
\textbf{Complexity}: Compared to heuristic combinations of cross-modal and single-modality CL~\cite{huang2021multilingual,jain2021mural,lee2020parameter,wang2022rethinking,yuan2021multimodal,shan2022ernievil, yang2022unified}, our approach does not significantly increase complexity: (1) upper bounds on MI can be estimated ``for free'' by directly plugging in the optimal critic from $I_\textsc{NCE}$, (2) removal of task-irrelevant information via $I(X_1;X_2|X_1',X_2')$ shares encoders with $I_\textsc{NCE}$, and (3) separate unimodal augmentations perform empirically well. We describe some extensions of other self-supervised methods in Appendix~\ref{app:method5}.
\vspace{-2mm}
\section{Experiments}
\vspace{-2mm}
We run comprehensive experiments on a suite of synthetic and large-scale real-world datasets with varying requirements of shared and unique task-relevant information, comparing our \names\ method to key baselines:
\begin{enumerate}[noitemsep,topsep=0pt,nosep,leftmargin=*,parsep=0pt,partopsep=0pt]
\item SimCLR~\cite{chen2020simple}: the straightforward method of cross-modal $(X_1,X_2)$ contrastive learning.
\item Cross+Self~\cite{yuan2021multimodal,huang2021multilingual,lee2020parameter,jain2021mural,shan2022ernievil, yang2022unified}: captures a range of methods combining cross-modal $(X_1,X_2)$ CL with additional unimodal $(X_i,X_i')$ CL objectives. This category also includes other ways of preserving unique information, such as through (variational) autoencoder reconstructions~\cite{wang2022rethinking}.
\item Cross+Self+Fact~\cite{yuan2021multimodal,yang2022visionlanguage}: A factorized extension of Cross+Self, which is approximately done in prior work that adds separate (typically pre-trained) unimodal encoders for each modality.
\item SupCon~\cite{khosla2020supervised}, which learns $I(X_1;X_2|Y)$ using CL conditioned on $Y$ from labeled data.
\end{enumerate}
We also carefully ablate each component of our method and investigate factors, including training data size and choice of augmentations. The intermediate ablations that emerge include:
\begin{enumerate}[noitemsep,topsep=0pt,nosep,leftmargin=*,parsep=0pt,partopsep=0pt]
\item \names-SUP: The supervised CL version which uses labels $Y$ in Eqs.(\ref{eq:conditional_nce_supervised}) and (\ref{eq:conditional_club_supervised}).
\item \names-SSL: The fully self-supervised version of our approach replacing $Y$ with multimodal augmentations $X_1'$ and $X_2'$ to approximate the task.
\item OurCL-SUP: \names-SUP but removing the factorization so only two features $Z_1$ is optimized for both $I(X_1;X_2;Y)$ and $I(X_1;Y|X_2)$, $Z_2$ optimized for both $I(X_1;X_2;Y)$ and $I(X_2;Y|X_1)$.
\item OurCL-SSL: \names-SSL but also removing the factorization in the self-supervised setting.
\end{enumerate}
The formulation of each ablation and implementation can be found in Appendix~\ref{app:implementation}.
\vspace{-1mm}
\subsection{Controlled experiments on synthetic datasets}
\vspace{-1mm}
\textbf{Synthetic data generation}: We begin by generating data with controllable ratios of task-relevant shared and unique information. Starting with a set of latent vectors $w_1, w_2, w_s \sim \mathcal{N}(0_d, \Sigma_d^2), d=50$ representing information unique to $X_1,X_2$ and common to both respectively, the concatenated vector $[w_1,w_s]$ is transformed into high-dimensional $x_1$ using a fixed transformation $T_1$ and likewise $[w_2,w_s]$ to $x_2$ via $T_2$. The label $y$ is generated as a function (with nonlinearity and noise) of varying ratios of $w_s$, $w_1$, and $w_2$ to represent shared and unique task-relevant information.
\textbf{Results}: In Figure~\ref{fig:overview}, we show our main result on synthetic data comparing \names\ with existing CL baselines. \names\ consistently maintains the best performance, whereas SimCLR~\cite{chen2020simple} and SupCon~\cite{khosla2020supervised} see performance drops as unique information increases. Cross+Self~\cite{yuan2021multimodal,huang2021multilingual,lee2020parameter,jain2021mural} recovers in fully unique settings (x-axis$=1.0$) but suffers at other ratios.
\input{tables/probing}
\textbf{Representation probing information}: We run a probing experiment to compute how well different contrastive representations capture shared and unique information. In Table~\ref{tab:probing}, for the $Z_i$'s learned by each method, we approximately compute $I(Z_i;w_1)$, $I(Z_i;w_2)$, and $I(Z_i;w_s)$ with respect to ground truth generative variables $w_s$, $w_1$, and $w_2$. As expected, existing methods such as SimCLR capture smaller amounts of unique information (roughly $4$ bits in $I(Z_i;w_1)$ and $I(Z_i;w_2)$), focusing instead on learning $I(Z_i;w_s)$ (12 bits). Cross+self captures slightly larger $I(Z_i;w_2)=4.26$, and SupCon with labeled data captures up to $5$ bits of unique information. Our \names\ approach captures $7$ bits of unique information and maintains $10$ bits of shared information, with total information captured higher than the other approaches. Furthermore, $\{Z_{S_1},Z_{S_2}\}$ capture more information about $w_s$, $Z_{U_1}$ about $w_1$, and $Z_{U_2}$ about $w_2$, indicating that factorization in our approach is successful.
\vspace{-1mm}
\subsection{Self-supervised multimodal learning with low redundancy and high uniqueness}
\vspace{-1mm}
\textbf{Multimodal fusion datasets}: We use a large collection of real-world datasets provided in MultiBench~\citep{liang2021multibench}, where we expect varying ratios of shared and unique information important for the task, to compare \names\ with other CL baselines:
\begin{enumerate}[noitemsep,topsep=0pt,nosep,leftmargin=*,parsep=0pt,partopsep=0pt]
\item \textsc{MIMIC}~\cite{johnson2016mimic}: mortality and disease prediction from $36,212$ medical records (tabular patient data and medical time-series sensors from ICU).
\item \textsc{MOSEI}~\cite{zadeh2018multimodal}: multimodal sentiment and emotion benchmark with $23,000$ monologue videos.
\item \textsc{MOSI}~\cite{zadeh2016mosi}: multimodal sentiment analysis from $2,199$ YouTube videos.
\item \textsc{UR-FUNNY}~\citep{hasan2019ur}: a dataset of humor detection from more than $16,000$ TED talk videos.
\item \textsc{MUsTARD}~\citep{castro2019towards}: a corpus of $690$ videos for research in sarcasm detection from TV shows.
\item \textsc{IRFL}~\cite{yosef2023irfl}: $6,697$ matching images and figurative captions (rather than literal captions).
\end{enumerate}
Together, these datasets cover seven different modalities from the healthcare, affective computing, and multimedia research areas and total more than $84,000$ data points.
For \textsc{MIMIC} with tabular and medical sensor inputs, we train self-supervised CL models on top of raw modality inputs.
For \textsc{IRFL} with image and caption inputs, we start with a pretrained CLIP model~\cite{radford2021learning} and perform continued pre-training to update CLIP weights with our \names\ objectives, before linear classifier testing. For the remaining four video datasets, we train self-supervised CL models starting from standard pre-extracted text, video, and audio features~\cite{liang2021multibench}. Please refer to Appendix~\ref{appendix:data} for experimental details. We release our code and models at \url{https://github.com/pliang279/FactorCL}.
\textbf{Multimodal fusion results}: From Table~\ref{tab:fusion}, \names\ significantly outperforms the baselines that do not capture both shared and unique information in both supervised and self-supervised settings, particularly on \textsc{MuStARD} (where unique information expresses sarcasm, such as sardonic facial expressions or ironic tone of voice), and on \textsc{MIMIC} (with unique health indicators and sensor readings).
In Table~\ref{tab:fig}, we also show that \names\ substantially improves the state-of-the-art in classifying images and figurative captions which are not literally descriptive of the image on \textsc{IRFL}, outperforming zero-shot and fine-tuned CLIP~\cite{radford2021learning} as well as continued pre-training baselines on top of CLIP.
\input{tables/fusion}
\textbf{Modeling ablations}:
In Table~\ref{tab:fusion}, we also carefully ablate each component in our method and indicate either existing baselines or newly-run ablation models.
\begin{enumerate}[noitemsep,topsep=0pt,nosep,leftmargin=*,parsep=0pt,partopsep=0pt]
\item \textbf{Factorized representations}: In comparing \names-SSL with OurCL-SSL, and also \names-SUP with OurCL-SUP, we find that factorization is critical: without it, performance drops on average $6.1\%$, with performance drop as high as $8.6\%$ for \textsc{MIMIC}.
\item \textbf{Information removal via upper bound}: By comparing \names\ with SimCLR, Cross+Self, and Cross+Self+Fact, and SupCon that only seek to capture task-relevant information via contrastive lower bounds on MI, we find that separately modeling the task-relevant information (to be captured) and task-irrelevant information (to be removed) is helpful. Without removing task-irrelevant information via the upper-bound objective, performance drops on average $13.6\%$, with performance drops as high as $23.5\%$ for the \textsc{MOSI} dataset. We also found that training was more difficult without this objective, which is expected due to overwhelming superfluous information from the dataset \cite{zadeh2018multimodal}.
\item \textbf{Multimodal augmentations}: Finally, we investigate the differences between separate unimodal augmentations (\names-IndAug in Table~\ref{tab:fig}) versus a joint multimodal augmentation (\names-SSL) on the \textsc{IRFL} dataset. We choose this dataset since its images and captions are the easiest to visualize (see Figure~\ref{fig:augs} for augmentations from both strategies).
In the self-supervised setting, we find that multimodal augmentations achieve $95\%$ performance, higher than the $92\%$ for separate unimodal augmentations, and both outperform baselines SimCLR and Cross+Self.
\end{enumerate}
\input{tables/clip}
\textbf{Ablations on $S, U_1$ and $U_2$}:
In Table~\ref{tab:ablations}, we also test \names\ when training linear classifiers on top of only shared $\{Z_{S_1},Z_{S_2}\}$ and unique $Z_{U_1}$, $Z_{U_2}$ separately. We call these models \names-$S$, \names-$U_1$, and \names-$U_2$.
Immediately, we observe that performance drops as compared to the full \names\ model, indicating that both shared and unique information are critical in real-world multimodal tasks.
As expected, the best-performing submodel is the one that captures the region with the largest amount of task-relevant information: \textsc{MOSEI} and \textsc{MOSI} are known to include a lot of redundancy and unique information since language is very important for detecting sentiment~\cite{zadeh2018multimodal}, so \names-$S$ and \names-$U_2$ perform best. For sarcasm detection on \textsc{MuStARD}, video information is most important with \names-$U_1$ performing best ($59.4\%$), and ablation models are also the furthest away from full multimodal performance ($69.9\%$). This is aligned with intuition where sarcasm is expressed through tone of voice and visual gestures (high $U_1$), as well as from contradictions between language and video (higher multimodal performance).
\input{tables/ablations}
\textbf{Additional results}: In Appendix~\ref{appendix:results}, we also verify \names\ in settings with abundant shared information, where we expect to recover the same performance as standard CL~\cite{chen2020simple,oord2018representation,tian2020makes}.
\vspace{-3mm}
\section{Related Work}
\vspace{-3mm}
\textbf{Contrastive learning} is a successful self-supervised learning paradigm for computer vision~\cite{oord2018representation,chen2020simple,he2020momentum,grill2020bootstrap,chen2021exploring,caron2020unsupervised}, natural language~\cite{gao2021simcse, meng2021coco, neelakantan2022text}, speech~\cite{oord2018representation, schneider2019wav2vec, baevski2020wav2vec}, and multimodal tasks~\cite{radford2021learning, jia2021scaling, akbari2021vatt}. Its foundational underpinnings are inspired by work in multiview information theory~\cite{federici2020learning,khosla2020supervised,sridharan2008information,tian2020makes,tsai2020self} studying the shared information between two views and whether they are necessary or sufficient in predicting the label. Recently,~\citet{wang2022rethinking} and~\citet{kahana2022contrastive} discuss the limitations of assuming multiview redundancy and propose autoencoder reconstruction or unimodal contrastive learning to retain unique information, which resembles the Cross+self baselines in our experiments. We refer the reader to~\citet{shwartz2023compress} for a comprehensive review on multiview and contrastive learning. Our work also relates to conditional contrastive learning \cite{tsai2022conditional, ma2021conditional, ye2022contrastive, chi2022conditional}, where positive or negative pairs are supposed to sample from conditional distributions.
\textbf{Multimodal contrastive learning} aims to align related data from different modalities, typically provided as positive pairs. This could be done via optimizing a contrastive objective for inter-modality pairs \cite{radford2021learning,alayrac2020self, akbari2021vatt, jia2021scaling}, or both intra- and inter-modality data pairs \cite{yuan2021multimodal, huang2021multilingual, lee2020parameter, jain2021mural, kim2022transferring}. Our work also relates to factorized representation learning, which primarily studies how to capture modality-specific information primarily in each modality and multimodal information redundant in both modalities~\cite{hsu2018disentangling,tsai2018learning}. Prior work has used disentangled latent variable models~\cite{Bengio:2013:RLR:2498740.2498889,higgins2016beta,hsu2018disentangling,tsai2018learning}, mixture-of-experts~\cite{shi2019variational}, or product-of-experts~\cite{wu2018multimodal} layer to explain factors in multimodal data.
\textbf{Information theory} \cite{cover1991information,shannon1948mathematical} has been used to study several phenomena in multimodal learning, including co-learning~\cite{zadeh2020foundations, rahate2022multimodal} and multi-view learning~\cite{tsai2020self, huang2021makes}. Due to its theoretical importance, several lower and upper bounds have been proposed for practical estimation~\cite{oord2018representation,wu2020mutual,poole2019variational,ozair2019wasserstein}. One particular upper bound is given by~\citet{cheng2020club}, which we build on to create a more accurate and stable bound.
Our characterizations of shared and unique information are also related to partial information decomposition \cite{williams2010nonnegative}, co-information \cite{bell2003co, vergara2014review}, and interaction information \cite{mcgill1954multivariate} research.
\vspace{-3mm}
\section{Conclusion}
\vspace{-3mm}
This paper studied how standard CL methods suffer when task-relevant information lies in regions unique to each modality, which is extremely common in real-world applications such as sensor placement, medical testing, and multimodal interaction. In response, we proposed \names, a new method expanding CL techniques through the use of factorized representations, removing task-irrelevant information via upper bounds on MI, and multimodal data augmentations suitable for approximating the unobserved task. Based on \names's strong performance, there are several exciting directions in extending these ideas for masked and non-contrastive pre-training; we further discuss broader impacts and limitations of this line of work in Appendix~\ref{sec:impact}.
\vspace{-2mm}
\section*{Acknowledgements}
\vspace{-2mm}
This material is based upon work partially supported by Meta, National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH125740, R01MH132225, R01MH096951 and R21MH130767.
PPL is supported in part by a Siebel Scholarship and a Waibel Presidential Fellowship.
RS is supported in part by ONR grant N000142312368 and DARPA FA87502321015.
One of the aims of this project is to understand the comfort zone of people for better privacy and integrity.
Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred. Finally, we would also like to acknowledge feedback from anonymous reviewers who significantly improved the paper and NVIDIA’s GPU support.
{\small
\bibliographystyle{plainnat}
\bibliography{refs}
}
\clearpage
\input{supp}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{eq~(\ref{#1})}
% Reference to an equation, upper case
\def\Eqref#1{Eq~(\ref{#1})}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\usepackage{hyperref}
\usepackage{booktabs} %
\usepackage{amsfonts} %
\usepackage{nicefrac} %
\usepackage{microtype} %
\usepackage{subcaption}
\usepackage{enumitem}
\setlist{nolistsep}
\setlist[itemize]{noitemsep, topsep=0pt}
\usepackage{times}
\usepackage{graphicx} %
\usepackage{color}
\usepackage{array}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{xspace}
\usepackage{fancyhdr}
\usepackage{comment}
\newcommand{\mycaptionof}[2]{\captionof{#1}{#2}}
\newcolumntype{H}{>{\setbox0=\hbox\bgroup}c<{\egroup}@{}}
\newcommand{\noaistats}[1]{} %
\definecolor{darkgreen}{rgb}{0,0.4,0.0}
\definecolor{darkblue}{rgb}{0,0.1,0.3}
\definecolor{darkred}{rgb}{0.7,0.0,0.0}
\newcommand{\lbs}{\ensuremath{B}} %
\newcommand{\all}{\infty}
\newcommand{\lepochs}{\ensuremath{E}} %
\newcommand{\clientfrac}{\ensuremath{C}} %
\newcommand{\tnn}{2NN\xspace} %
\newcommand{\xx}{\hspace{-0.01in}\ensuremath{\times}}
\newcommand{\loss}{\ell}
\newcommand{\SUB}[1]{\ENSURE \hspace{-0.15in} \textbf{#1}}
\newcommand{\algfont}[1]{\texttt{#1}}
\renewcommand{\algorithmicensure}{}
\newcommand{\eqdef}{\overset{\text{def}}{=}}
\newcommand{\T}{\rule{0pt}{2.2ex}}
\newcommand{\targetTNN}{97\%\xspace}
\newcommand{\targetCNN}{99\%\xspace}
\newcommand{\targetLSTM}{54\%\xspace}
\newcommand{\nc}{K}
\newcommand{\pp}{\mathcal{P}}
\newlength{\pw}
\newcommand{\BO}{\mathcal{O}}
\newcommand{\qqand}{\qquad \text{and} \qquad}
\newcommand{\qqwhere}{\qquad \text{where} \qquad}
\newcommand{\qqwith}{\qquad \text{with} \qquad}
\newcommand{\grad}{\triangledown}
\newcommand{\h}{\frac{1}{2}}
\newcommand\norm[1]{\left\lVert#1\right\rVert}
\begin{figure}[tbp]
\centering
\vspace{-6mm}
\includegraphics[width=\linewidth]{figures/overview.pdf}
\vspace{-2mm}
\caption{\textbf{Left}: We define $S=I(X_1;X_2;Y)$ as task-relevant shared information and $U_1=I(X_1;Y|X_2)$, $U_2=I(X_2;Y|X_1)$ as task-relevant unique information. \textbf{Right}: On controllable datasets with varying ratios of $S$, $U_1$, and $U_2$, standard CL captures $S$ but struggles when there is more $U_1$ and $U_2$. Our \names\ approach maintains best performance, whereas SimCLR~\cite{chen2020simple} and SupCon~\cite{khosla2020supervised} see performance drops as unique information increases, and Cross+Self~\cite{huang2021multilingual,jain2021mural,lee2020parameter,yuan2021multimodal} recovers in fully unique settings but suffers at other ratios.}
\label{fig:overview}
\vspace{-4mm}
\end{figure}
\begin{figure}[tbp]
\centering
\vspace{-4mm}
\includegraphics[width=\linewidth]{figures/method.pdf}
\vspace{-2mm}
\caption{\names: We propose a self-supervised CL method to learn \textit{factorized} representations $Z_{S_1}$, $Z_{S_2}$, $Z_{U_1}$, and $Z_{U_2}$ to capture task-relevant information shared in both $X_1$ and $X_2$, unique to $X_1$, and unique to $X_2$. By starting with information-theoretic first principles of shared and unique information, we design contrastive estimators to both \textit{capture task-relevant} and \textit{remove task-irrelevant} information, where a notion of task-relevance without explicit labels is afforded by a new definition of \textit{multimodal augmentations} $X_1',X_2'$. Lower bounds are in {\color{gg}green} and upper bounds are in {\color{rr}red}.}
\label{fig:method}
\vspace{-4mm}
\end{figure}
\begin{figure}[tbp]
\centering
\vspace{-2mm}
\includegraphics[width=0.94\linewidth]{figures/bounds.pdf}
\vspace{-2mm}
\caption{\small{Estimated $I_\textsc{NCE}$ lower bound~\cite{oord2018representation} and our proposed upper bound $I_\textsc{NCE-CLUB}$ on sample distributions with changing mutual information: our upper bound is tighter, more accurate, and more stable than $I_\textsc{CLUB}$ upper bound~\cite{cheng2020club}, and also comes for `free' via jointly estimating both lower and upper bounds simultaneously. We find that as dimension increases, the $I_\textsc{CLUB}$ estimator collapses to zero and no longer tracks true MI.}}
\label{fig:bounds}
\vspace{-4mm}
\end{figure}
\begin{wrapfigure}{L}{0.45\textwidth}
\vspace{-8mm}
\begin{minipage}{0.45\textwidth}
\vspace{-0mm}
\includegraphics[width=\linewidth]{figures/unique_augment.pdf}
\vspace{-4mm}
\caption{\small{Standard vs. unique augmentations for the figurative language~\cite{yosef2023irfl} dataset. After augmenting text modality $X_1$ independently (same for both augmentation types), we illustrate their differences for image augmentation: unique augmentation on images should avoid removing information referred to by $X_1$ (the text). The text mentions that the car is fast so unique augmentation for images should \textit{not} remove the highway pixels of the image which can suggest the car is fast.}}
\label{fig:augs}
\vspace{-4mm}
\end{minipage}
\vspace{-11mm}
\end{wrapfigure}
\begin{figure}[tbp]
\begin{minipage}{0.45\textwidth}
\vspace{-22mm}
\begin{algorithm}[H]
\begin{algorithmic}
\REQUIRE Multimodal dataset $\{\mathbf{X_1}, \mathbf{X_2}$\}.
\vspace{.2em}\hrule\vspace{.5em}
\STATE Initialize networks $f(\cdot)$.
\WHILE{not converged}
\FOR{sampled batch $\{\vx_1, \vx_2\}$}
\STATE Estimate $I_{\textsc{NCE}}(X_1; X_2)$ from Eq. \ref{eq:nce_original}
\STATE $\mathcal{L} = - I_{\textsc{NCE}}(X_1; X_2)$
\STATE Update $f(\cdot)$ to minimize $\mathcal{L}$
\ENDFOR
\ENDWHILE
\RETURN $f(\cdot)$
\end{algorithmic}
\caption{Standard multimodal CL.}
\label{alg:standardcl}
\end{algorithm}
\end{minipage}
\begin{minipage}{0.48\textwidth}
\vspace{-14.2mm}
\begin{algorithm}[H]
\begin{algorithmic}
\REQUIRE Multimodal dataset $\{\mathbf{X_1}, \mathbf{X_2}$\}.
\vspace{.2em}\hrule\vspace{.5em}
\STATE Initialize networks $f(\cdot)$.
\WHILE{not converged}
\FOR{sampled batch $\{\vx_1, \vx_2\}$}
\STATE $\vx_1' \leftarrow$ \textbf{{\color{rr}$\textrm{Augment}(\vx_1)$}}
\STATE $\vx_2' \leftarrow$ \textbf{{\color{rr}$\textrm{Unique-Augment}(\vx_2| \vx_1)$}}
\STATE Plug $\vx_1'$ and $\vx_2'$ into Eq. \ref{eq:nce_final_ssl} and \ref{eq:nceclub_final_ssl}
\STATE Estimate \textbf{{\color{rr} $\mathbf{S}, \mathbf{U_1}, \mathbf{U_2}$}} from Eq. \ref{eq:shared} and \ref{eq:unique}
\STATE $\mathbf{\color{rr} \mathcal{L} = -( S + U_1 + U_2)}$
\STATE Update $f(\cdot)$ to minimize $\mathcal{L}$
\ENDFOR
\ENDWHILE
\RETURN $f(\cdot)$
\end{algorithmic}
\caption{\textbf{\color{rr}\names}.}
\label{alg:full}
\end{algorithm}
\vspace{-8mm}
\end{minipage}
\end{figure}
\begin{table*}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{4pt}
\vspace{-8mm}
\caption{We probe whether contrastive representations learned by classic CL methods and \names\ contain shared $w_s$ or unique $w_1,w_2$ information. \names\ captures the most unique information.}
\centering
\begin{tabular}{l|cc|cc|cc|cccc}
\hline \hline
Model & \multicolumn{2}{c|}{SimCLR} & \multicolumn{2}{c|}{Cross+self} & \multicolumn{2}{c|}{SupCon} & \multicolumn{4}{c}{\names} \\
Representations & $Z_1$ & $Z_2$ & $Z_1$ & $Z_2$ & $Z_1$ & $Z_2$ & $Z_{U_1}$ & $Z_{U_2}$ & $Z_{S_1}$ & $Z_{S_2}$ \\
\hline
$I(Z;w_1)$ & 4.45 & 0.16 & 4.39 & 0.14 & 5.17 & 0.19 & \textbf{7.83} & 0.03 & 6.25 & 0.04\\
$I(Z;w_2)$ & 0.17 & 3.92 & 0.13 & 4.26 & 0.23 & 5.17 & 0.06 & \textbf{7.17} & 0.05 & 5.79\\
$I(Z;w_s)$ & 12.61 & 12.06 & 11.30 & 11.47 & 7.48 & 7.17 & 9.47 & 9.89 & 10.13 & 9.40\\
\hline \hline
\end{tabular}
\vspace{-4mm}
\label{tab:probing}
\end{table*}
\begin{table*}[t]
\centering
\fontsize{8}{10}\selectfont
\setlength\tabcolsep{1pt}
\vspace{-1mm}
\caption{Results on MultiBench~\citep{liang2021multibench} datasets with varying shared and unique information: \names\ achieves strong results vs self-supervised (top $5$ rows) and supervised (bottom $3$ rows) baselines that do not have unique representations, factorization, upper-bounds to remove irrelevant information, and multimodal augmentations.}
\centering
\begin{tabular}{l|ccccc|ccccc}
\hline \hline
Model & $(X_1; X_2)$ & $(X_i; X_i')$ & $(X_1; X_2|Y)$ & $(X_2'')$ & Fact & \textsc{MIMIC} & \textsc{MOSEI} & \textsc{MOSI} & \textsc{UR-FUNNY} & \textsc{MUStARD} \\
\hline
SimCLR~\cite{chen2020simple} & \textcolor{gg}\cmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & 66.67\% & 71.03\% & 46.21\% & 50.09\% & 53.48\%\\
Cross+Self~\cite{wang2022rethinking} & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & 65.20\% & 71.04\% & 46.92\% & 56.52\% & 53.91\%\\
Cross+Self+Fact~\cite{yuan2021multimodal} & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & \textcolor{gg}\cmark & 65.49\% & 71.07\% & 52.37\% & 59.91\% & 53.91\% \\
OurCL-SSL & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{rr}\xmark & 65.22\% & 71.16\% & 48.98\% & 58.79\% & 53.98\%\\
\names-SSL & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textbf{67.34}\% & \textbf{74.88\%} & \textbf{52.91\%} & \textbf{60.50\%} & \textbf{55.80}\%\\
\hline
SupCon~\cite{khosla2020supervised} & \textcolor{rr}\xmark & \textcolor{rr}\xmark & \textcolor{gg}\cmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & 67.37\% & 72.71\% & 47.23\% & 50.98\% & 52.75\%\\
OurCL-SUP & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{rr}\xmark & \textcolor{rr}\xmark & 68.16\% & 71.15\% & 65.32\% & 58.32\% & 65.05\%\\
\names-SUP & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{gg}\cmark & \textcolor{rr}\xmark & \textcolor{gg}\cmark & \textbf{76.79\%} & \textbf{77.34\%} & \textbf{70.69\%} & \textbf{63.52\%} & \textbf{69.86}\%\\
\hline \hline
\end{tabular}
\vspace{-4mm}
\label{tab:fusion}
\end{table*}
\begin{wraptable}{r}{5cm}
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{2pt}
\vspace{-4mm}
\caption{Continued pre-training on CLIP with our \names\ objectives on classifying images and figurative language.}
\centering
\begin{tabular}{l|c c}
\hline \hline
Task & \textsc{IRFL} \\
\hline
Zero-shot CLIP~\cite{radford2021learning} & 89.15\%\\
SimCLR~\cite{chen2020simple} & 91.57\%\\
Cross+Self~\cite{wang2022rethinking,yuan2021multimodal} & 95.18\%\\
\names-IndAug & 92.77\% \\
\names-SSL & \textbf{95.18}\%\\
\hline
Fine-tuned CLIP~\cite{radford2021learning} & 96.39\%\\
SupCon~\cite{khosla2020supervised} & 89.16\%\\
\names-SUP & \textbf{98.80}\%\\
\hline \hline
\end{tabular}
\label{tab:fig}
\vspace{-2mm}
\end{wraptable}
\begin{table*}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{3pt}
\vspace{-0mm}
\caption{We ablate using only shared representations $\{Z_{S_1},Z_{S_2}\}$, unique representation $Z_{U_1}$, and $Z_{U_2}$ separately for prediction. Both shared and unique information are critical in real-world multimodal tasks.}
\centering
\begin{tabular}{l|ccccc}
\hline \hline
Model & \textsc{MIMIC} & \textsc{MOSEI} & \textsc{MOSI} & \textsc{UR-FUNNY} & \textsc{MUStARD} \\
\hline
\names-$S$ & 63.77\% & 77.17\% & 70.12\% & 63.42\% & 57.25\%\\
\names-$U_1$ & 55.90\% & 77.06\% & 70.11\% & 62.00\% & 59.42\%\\
\names-$U_2$ & 69.08\% & 71.01\% & 52.33\% & 54.35\% & 53.62\%\\
\hline
\names-SUP & \textbf{76.79\%} & \textbf{77.34\%} & \textbf{70.69\%} & \textbf{63.52\%} & \textbf{69.86\%} \\
\hline \hline
\end{tabular}
\vspace{-6mm}
\label{tab:ablations}
\end{table*}
\appendix
\vspace{-2mm}
\section*{Appendix}
\vspace{-2mm}
\vspace{-2mm}
\section{Broader Impact}
\label{sec:impact}
\vspace{-2mm}
Multimodal data and self-supervised models are ubiquitous in a range of real-world applications. This paper is our attempt at broadening the applicability of self-supervised contrastive methods to a wider range of multimodal tasks beyond those that exhibit multi-view redundancy. We believe that special care must be taken to ensure that these models are safely deployed for real-world benefit:
\textbf{Time and space complexity}: Modern multimodal models are large and take up a significant amount of carbon footprint during training and testing. As compared to heuristic combinations of cross-modal and single-modality CL~\cite{huang2021multilingual,jain2021mural,lee2020parameter,wang2022rethinking,yuan2021multimodal,shan2022ernievil, yang2022unified}, we believe that \names\ does not significantly increase complexity: (1) upper bounds on MI can be estimated ``for free'' by directly plugging in the optimal critic from $I_\textsc{NCE}$, and (2) removal of task-irrelevant information via $I(X_1;X_2|X_1',X_2')$ shares encoders with $I_\textsc{NCE}$, and (3) separate unimodal augmentations perform well enough in practice. We also release our code and models so that they can be evaluated quickly on new tasks, which can amortize complexity costs.
\textbf{Privacy and security}: There may be privacy risks associated with making predictions from multimodal data of recorded human behaviors and medical data (i.e., the datasets used in our experiments for analysis of sentiment, emotions, personality, sarcasm, and humor, as well as disease prediction from medical data). We have followed best practices in maintaining the privacy and safety of these datasets: (1) the creators of these video datasets have taken the appropriate steps to only access public data that participants or content creators have consented for public release (creative commons license and following fair use guidelines of YouTube)~\citep{castro2019towards,hasan2019ur,zadeh2018multimodal}, (2) \textsc{MIMIC} has been rigorously de-identified in accordance with Health Insurance Portability and Accountability Act (HIPAA) such that all possible personal information has been removed from the dataset~\citep{johnson2016mimic}, (3) all video data was also anonymized and stripped of all personal (e.g., personally identifiable information) and protected attributes (e.g., race, gender), (4) all models trained on affect recognition datasets use only pre-extracted non-invertible features that rely on general visual or audio features such as the presence of a smile or magnitude of voice which cannot be used to identify the speaker~\citep{zadeh2016mosi,zadeh2018multimodal}, and (5) we studied the videos collected in these affective computing datasets and found no offensive words used or personal attacks recorded in the video. Finally, we only use these datasets for research purposes and emphasize that any multimodal models trained to perform prediction should only be used for scientific study and should not in any way be used for real-world harm.
\textbf{Social biases}: We acknowledge risks of social bias due to imbalanced datasets, resulting in potential biases surrounding gender, race, and ethnicity, among others~\citep{bolukbasi2016man,liang2021towards}. We note that our \names\ approach has a close link with conditional CL~\citep{ma2021conditional}, which can also be adapted to condition on sensitive attributes and therefore reduce bias. Studying these research questions is an important direction for future work.
\textbf{Future work}: We discuss more limitations and potential future work in this direction. Firstly, optimizing our objectives using better MI lower and upper bounds such as in \citet{guo2022tight} and \citet{sordoni2021decomposed}, could improve the performance for inputs of higher dimension and complex modality. Next, the current data augmentation method requires one to pick augmentations to approximately satisfy Definition \ref{def:optimal_multi}; future work could extend InfoMin \citep{tian2020makes} to automatically generate data augmentations to satisfy Definition \ref{def:optimal_multi}, or leverage future progress in multimodal generative models for data augmentation. Lastly, future work could quantify whether shared or unique information is more important for different tasks and reweight the terms in the \names\ objective to suit the tasks.
\vspace{-2mm}
\section{Analysis of Multi-view Contrastive Learning}
\label{app:analysis}
\vspace{-2mm}
\textbf{Multi-view shared information} describes the extent and dimensions in which information can be shared across different views. The presence of shared information is often in contrast to unique information that exists solely in a single modality, and can be formalized via information theory:
\begin{definition}
(Shared information) Given $X_1$ and $X_2$, $I(X_1;X_2) = \int p(x_1,x_2) \log \frac{p(x_1,x_2)}{p(x_1) p(x_2)} $ measures the degree of information-theoretic shared information between $X_1$ and $X_2$.
\end{definition}
\begin{definition}
(Task-relevant shared information) Given $X_1$, $X_2$, and a target $Y$, $I(X_1;X_2;Y) = I(X_1;X_2) - I(X_1;X_2|Y) = \int p(x_1,x_2) \log \frac{p(x_1,x_2)}{p(x_1) p(x_2)} - \int p(x_1,x_2|y) \log \frac{p(x_1,x_2|y)}{p(x_1|y) p(x_2|y)} $ measures the amount of task-relevant shared information between $X_1$ and $X_2$ for predicting $Y$. $I(X_1;X_2|Y)$ represents the task-irrelevant shared information.
\end{definition}
\textbf{Learning shared information via contrastive learning}: Current approaches for multi-view contrastive learning model shared information $I(X_1;X_2)$ (and subsequently task-relevant shared information $I(X_1;X_2;Y)$ during downstream task fine-tuning), without modeling unique information.
\begin{align}
Z_1 = \argmax_{Z_1 := f_\theta(X_1)} I(Z_1;X_2), Z_2 = \argmax_{Z_2 := f_\theta(X_2)} I(X_1;Z_2).
\label{eq:standard_cl_z_app}
\end{align}
Optimizing for $I(X_1;X_2)$ is performed via a surrogate loss during self-supervised pre-training (where we do not have access to the label $Y$) by maximizing the InfoNCE objective:
\begin{equation}
\textsc{InfoNCE} = \sup_f \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2)\\x_2^- \sim p(x_2)}} \left[ \log \frac{\exp f(x_1,x_2^+)}{\sum_k \exp f(x_1, x_2^-)} \right],
\end{equation}
\citet{oord2018representation} show that $I(X_1;X_2) \ge \log k - \mathcal{L}_{\textrm{NCE}}(X_1; X_2)$ where $\mathcal{L}_{\textrm{NCE}}(X_1; X_2)$ is negative of $\textsc{InfoNCE}$ and is the loss to minimize (rather than maximize) in training. NCE falls into a broader class of contrastive learning methods~\cite{chen2020simple, he2020momentum, radford2021learning, chen2021empirical, khosla2020supervised} that model the ratio between joint densities $p(x_1,x_2)$ and product of marginal densities $p(x_1)p(x_2)$ using positive and negative samples~\cite{nguyen2010estimating, poole2019variational, tschannen2019mutual, wu2020mutual, ozair2019wasserstein} or probabilistic classifiers~\cite{mukherjee2020ccmi,tsai2020neural}, all of which can also be used to capture shared information.
\citet{tian2019contrastive} argues that the optimal view of contrastive learning is also minimal: the minimal representations only extract relevant information of the
contrastive task (maximizing the shared part) and throw away other information. Therefore, from this minimal assumption, we have $I(Z_1; Y | X_2) = 0$ and $I(Z_2; Y | X_1) = 0$ as minimal $Z_1$ and $Z_2$ only captures task-relevant information from the shared part. By conditioning on $X_1$ or $X_2$, the shared part is removed, and $Z_1$ and $Y$ (or $Z_2$ and $Y$) do not share information.
Lastly, we restate the multi-view non-redundancy from Definition \ref{eq:multiview_nonredundancy_assump}:
\begin{definition}
\label{eq:multiview_nonredundancy_assump_app}
(Multi-view non-redundancy) $\exists \epsilon >0$ such that $I(X_1;Y|X_2) > \epsilon$ or $I(X_2;Y|X_1) > \epsilon$.
\end{definition}
\label{app:discussion_assumption} We would like to compare and clarify the differences between the multiview redundancy assumption in Eq.(\ref{eq:multiview_redundancy_assump}) and the multi-view nonredundancy in Def. \ref{eq:multiview_nonredundancy_assump_app}. The multiview redundancy assumption in Eq.(\ref{eq:multiview_redundancy_assump}) states that the task-relevant information from the unique part is minimal ($\le \epsilon$). The multiview non-redundancy states the opposite: the task-relevant information from the unique part is nonzero and nonminimal, as it is not bounded by $\epsilon$. Next we briefly clarify the relationship between these two assumptions and the InfoMin assumption: $I(Z_1; Y | X_2)=I(Z_2; Y | X_1)=0$. InfoMin is about representation $Z$ while the redundancy assumptions are only about data $X$.
InfoMin states that the optimal (sufficient and minimal) representation learns task-relevant information only from the shared part, as we discussed in the paragraph above. We empirically checked the two assumptions: Tables \ref{tab:probing} and \ref{tab:ablations} in the main text show that the multiview non-redundancy assumption holds empirically, and Table \ref{tab:probing_mi_conditional} shows that the InfoMin assumption holds empirically.
We now show the limitations of CL methods, first restating the Theorem here:
\begin{theorem}
\label{thm:suboptimal_eq_app}
(Suboptimality of standard CL) When there is multi-view non-redundancy as in Definition \ref{eq:multiview_nonredundancy_assump_app}, given optimal representations $\{Z_1,Z_2\}$ that satisfy Eq.(\ref{eq:standard_cl_z_app} and $I(Z_1; Y | X_2)=I(Z_2; Y | X_1)=0$~\citep{tian2020makes}, we have that
{\small
\begin{align}
I(Z_1,Z_2;Y) = I(X_1,X_2;Y) - I(X_1;Y|X_2) - I(X_2;Y|X_1) = I(X_1;X_2) - I(X_1;X_2|Y) < I(X_1,X_2;Y). \label{eq:suboptimal_eq_app}
\end{align}}
\end{theorem}
\begin{proof}
Since $Z_1$ and $Z_2$ maximize $I(X_1;X_2)$ we have that $I(Z_1;X_2) = I(X_1;Z_2) = I(X_1;X_2)$ so $I(Z_1;X_2;Y) = I(X_1;Z_2;Y) = I(X_1;X_2;Y)$ and $I(Z_1;X_2|Y) = I(X_1;Z_2|Y) = I(X_1;X_2|Y)$.
We now show the relationship between $I(X_1,X_2;Y)$, which is the total information $X_1,X_2$ contributes towards predicting $Y$ in classic supervised learning, with $I(Z_1,Z_2;Y)$, which is the information that our learned self-supervised representations can contribute towards $Y$ during supervised fine-tuning. We first derive the relationship between $I(Z_1;Y)$ and $I(X_1;Y)$:
\begin{align}
I(Z_1;Y) &= I(Z_1;X_2;Y) + I(Z_1;Y|X_2) \\
&= I(X_1;X_2;Y) + I(Z_1;Y|X_2) \\
&= I(X_1;Y) - I(X_1;Y|X_2) + I(Z_1;Y|X_2) \\
&= I(X_1;Y) - I(X_1;Y|X_2)
\end{align}
Given $X_1$, we further derive a relationship between $I(Z_2;Y|Z_1)$ and $I(X_2;Y|X_1)$:
\begin{align}
I(Z_2;Y|Z_1) &= I(Z_2;X_1;Y|Z_1) + I(Z_2;Y|Z_1,X_1) \label{eq:unique_proof_step1} \\
&= I(X_1;X_2;Y|Z_1) + I(Z_2;Y|Z_1,X_1) \\
&= I(X_1;X_2;Y|Z_1) + I(Z_2;Y|X_1) \label{eq:suboptimal_proof_deterministic_1}\\
&= I(X_2;Y|Z_1) - I(X_2;Y|X_1,Z_1) + I(Z_2;Y|X_1) \\
&= I(X_2;Y|Z_1) - I(X_2;Y|X_1) + I(Z_2;Y|X_1) \label{eq:suboptimal_proof_deterministic_2}\\
&= I(X_2;Y) - I(Z_1;X_2;Y) - I(X_2;Y|X_1) + I(Z_2;Y|X_1) \\
&= I(X_2;Y) - I(X_1;X_2;Y) - I(X_2;Y|X_1) + I(Z_2;Y|X_1) \label{eq:train_z} \\
&= I(X_2;Y|X_1) - I(X_2;Y|X_1) + I(Z_2;Y|X_1) = 0 \label{eq:unique_proof_step2}
\end{align}
In Eqs.(\ref{eq:suboptimal_proof_deterministic_1}) and (\ref{eq:suboptimal_proof_deterministic_2}), we use the fact that conditioning on $Z_1$ and $X_1$ jointly reduces to conditioning on $X_1$ since $Z_1$ is deterministically obtained from $X_1$, and in Eq.(\ref{eq:train_z}) we use the definition of learning $Z_s$ to maximize $I(X_1;X_2)$ so $I(Z_1;X_2;Y) = I(X_1;Z_2;Y)$.
Finally, adding both terms up,
\begin{align}
I(Z_1,Z_2;Y) &= I(Z_1;Y) + I(Z_2;Y|Z_1) \\
&= I(X_1;Y) - I(X_1;Y|X_2) \\
&= I(X_1;X_2;Y) \\
&= I(X_1,X_2;Y) - I(X_1;Y|X_2) - I(X_2;Y|X_1) \\
&= I(X_1;X_2) - I(X_1;X_2|Y)
\end{align}
gives the desired result.
\end{proof}
\textbf{Bayes error rate.} The Bayes error rate $P_e(Z_1,Z_2):=1 - \mathbb{E}_{P_{Z_1, Z_2}}\left[\max_{y\in Y} P\left(\hat{Y}=y \mid z_1, z_2 \right)\right]$ of contrastive representations $\{Z_1,Z_2\}$ is given by:
\begin{align}
P_e &\le 1 - \exp \left[I(X_1, X_2; Y)- I(X_1;Y|X_2) - I(X_2;Y|X_1) - H(Y) \right] \\
&= 1 - \exp \left[ I(X_1;X_2;Y) - H(Y)\right] \label{eq:bayes_error_app}
\end{align}
\begin{proof}
We use the inequality between $P_e$ and $H(Y | Z)$ \cite{feder1994relations, tsai2020self, wang2022rethinking}:
\begin{align}
-\ln (1 - P_e) \le H(Y | Z), \text{ or equivalently, } P_e \le 1 - \exp[-H(Y | Z)]
\end{align}
If we regard $Z$ as the joint of $Z_1$ and $Z_2$, then we have
\begin{align}
P_e &\le 1 - \exp[-H(Y | Z_1, Z_2)] \label{eq:bayes_error_rate_orig_app}
\end{align}
We further expand $H(Y | Z_1, Z_2)$ by definition of mutual information, $I(X; Y) = H(X) - H(X | Y)$, Theorem \ref{thm:suboptimal_eq_app}, and the $I(X_1; X_2; Y) = I(X_1; X_2) - I(X_1; X_2 | Y)$:
\begin{align}
H(Y | Z_1, Z_2) &= H(Y) - I(Z_1, Z_2; Y)\\
&= H(Y) - I(X_1, X_2; Y) + I(X_1; Y | X_2) + I(X_2; Y | X_1)\\
&= H(Y) - I(X_1; X_2) + I(X_1; X_2 | Y) \\
&= H(Y) - I(X_1; X_2;Y)
\end{align}
Plugging in Eq.(\ref{eq:bayes_error_rate_orig_app}), we have
\begin{align}
P_e &\le 1 - \exp[-H(Y | Z_1, Z_2)]\\
&= 1 - \exp[-(H(Y) - I(X_1, X_2; Y) + I(X_1; Y | X_2) + I(X_2; Y | X_1))]\\
&= 1 - \exp[-H(Y) + I(X_1, X_2; Y) - I(X_1; Y | X_2) - I(X_2; Y | X_1)]
\end{align}
and
\begin{align}
P_e &\le 1 - \exp[-H(Y | Z_1, Z_2)]\\
&= 1 - \exp[-(H(Y) - I(X_1; X_2;Y))]\\
&= 1 - \exp[-H(Y) + I(X_1; X_2;Y)]
\end{align}
resulting in the Bayes error rate as desired.
\end{proof}
\section{\namel}
\label{app:method}
\subsection{Contrastive estimators}
\label{app:method1}
\begin{theorem}
(Contrastive estimators for $I(X_1; X_2)$) Defining the NCE estimator and NCE-CLUB estimator as follows,
\begin{align}
I_\textsc{NCE}(X_1; X_2) &= \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2)\\x_2^- \sim p(x_2)}} \left[ \log \frac{\exp f(x_1,x_2^+)}{\sum_k \exp f(x_1, x_2^-)} \right] \label{eq:nce_original_app}\\
I_\textsc{NCE-CLUB}(X_1; X_2) &= \mathbb{E}_{x_1,x_2^+ \sim p(x_1,x_2)} \left[ f^*(x_1,x_2^+) \right] - \mathbb{E}_{\substack{x_1 \sim p(x_1)\\x_2^- \sim p(x_2)}} \left[f^*(x_1,x_2^-) \right] \label{eq:nceclub_app}
\end{align}
where $f^*(x_1,x_2)$ is the optimal critic from $I_\textsc{NCE}$ plugged into the $I_\textsc{CLUB}$ objective \cite{cheng2020club}. We call the proposed plug-in objective Eq.(\ref{eq:nceclub}) $I_\textsc{NCE-CLUB}$, and obtain lower and upper bounds on $I(X_1; X_2)$:
\begin{align}
I_\textsc{NCE}(X_1; X_2) \le I(X_1; X_2) \le I_\textsc{NCE-CLUB}(X_1; X_2).
\end{align}
\end{theorem}
\begin{proof}
The lower bound $I_\textsc{NCE}(X_1; X_2) \le I(X_1; X_2)$ follows from~\citet{oord2018representation}: optimizing the objective leads to an optimal critic $f^* = \log p(x_2|x_1) + c(x_2)$ \cite{poole2019variational} or without loss of generality $f^* = \log p(x_1|x_2) + c(x_1)$, where $c(\cdot)$ is an arbitrary deterministic function. Plugging the optimal critic into the $I_\textsc{NCE}(X_1; X_2)$ gives the result: $I_\textsc{NCE}(X_1; X_2) + \log N \le I(X_1; X_2)$ \cite{oord2018representation, poole2019variational}.
Next, the original $I_\textsc{CLUB}(X_1; X_2)$ \cite{cheng2020club} is defined as:
\begin{align}
I_\textsc{CLUB}(X_1; X_2) := \mathbb{E}_{p(x_1,x_2)} \left[ \log p(x_2|x_1) \right] - \mathbb{E}_{p(x_1)p(x_2)} \left[ \log p(x_2|x_1) \right].
\end{align}
As mutual information is symmetric w.r.t $x_1$ and $x_2$: $I(X_1; X_2) = I(X_2; X_1)$ , without loss of generality, we have:
\begin{align}
I_\textsc{CLUB}(X_1; X_2) = I_\textsc{CLUB}(X_2; X_1) = \mathbb{E}_{p(x_1,x_2)} \left[ \log p(x_1|x_2) \right] - \mathbb{E}_{p(x_1)p(x_2)} \left[ \log p(x_1|x_2) \right]
\end{align}
The formulation above assumes $p(x_1|x_2)$ is known, which is satisfied when we use the optimal critic $f^*=\log p(x_1|x_2) + c(x_1)$ from $I_\textsc{NCE}(X_1; X_2)$. Plugging the optimal critic $f^*$ into $I_\textsc{CLUB}(X_1; X_2)$, we obtain a desired upper bound $I_\textsc{NCE-CLUB}(X_1; X_2)$ of $I(X_1; X_2)$:
\begin{align}
&I_\textsc{NCE-CLUB}(X_1; X_2) = \mathbb{E}_{p(x_1,x_2)} \left[ \log p(x_1|x_2) + c(x_1) \right] - \mathbb{E}_{p(x_1)p(x_2)} \left[ \log p(x_1|x_2) + c(x_1) \right] \\
&= \mathbb{E}_{p(x_1,x_2)} \left[ \log p(x_1|x_2) \right] + \mathbb{E}_{p(x_1,x_2)} \left[ c(x_1) \right] - \mathbb{E}_{p(x_1)p(x_2)} \left[ \log p(x_1|x_2) \right] - \mathbb{E}_{p(x_1)p(x_2)} \left[ c(x_1) \right] \label{eq:club_derive_e} \\
&= \mathbb{E}_{p(x_1,x_2)} \left[ \log p(x_1|x_2) \right] - \mathbb{E}_{p(x_1)p(x_2)} \left[ \log p(x_1|x_2) \right] \label{eq:club_derive_c} \\
&\ge I(X_1;X_2). \label{eq:club_derive_upper}
\end{align}
Eq.(\ref{eq:club_derive_e}) is from the linearity of expectation, Eq.(\ref{eq:club_derive_c}) is from the fact that $c(x_1)$ is not a function of $x_2$ and therefore $\mathbb{E}_{p(x_1,x_2)} \left[ c(x_1) \right] = \mathbb{E}_{p(x_1)p(x_2)} \left[ c(x_1) \right] = \mathbb{E}_{p(x_1)} \left[ c(x_1) \right]$, and Eq.(\ref{eq:club_derive_upper}) is directly from the original $I_\textsc{CLUB}(X_1; X_2)$ paper \cite{cheng2020club}.
\end{proof}
\subsection{Unimodal and multimodal augmentations}
\label{app:method2}
We first restate the definitions of optimal single-view and multi-view augmentation:
\begin{definition}
\label{def:optimal_single_app}
(Optimal unimodal augmentation) $X_1'$ is an optimal unimodal augmentation for $X_1$ when $I(X; X') = I(X; Y)$, which implies that the only information shared between $X$ and $X'$ is task-relevant with no irrelevant noise.
\end{definition}
\begin{definition}
\label{def:optimal_multi_app}
(Optimal multimodal augmentation) $X_1'$ and $X_2'$ are optimal multimodal augmentation for $X_1$ and $X_2$ when $I(X_1,X_2; X_1',X_2') = I(X_1,X_2; Y)$, which implies that the only information shared between $X_1,X_2$ and $X_1',X_2'$ is task-relevant with no irrelevant noise.
\end{definition}
When are these assumptions satisfied? $I(X; X') = I(X; Y)$ holds when all information shared between $X$ and $X'$ is task-relevant, which implies that the augmentation keeps task-relevant information constant while changing task-irrelevant information. In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background. To satisfy $I(X_1,X_2; X_1',X_2') = I(X_1,X_2; Y)$, by the chain rule of MI, we augment in two steps:
\begin{align}
\textit{Unimodal aug: } &X_1' \textrm{ s.t. } I(X_1; X_1') = I(X_1; Y), \label{assump:unimodal_aug_app}\\
\textit{Unique aug: } &X_2' \textrm{ s.t. } I(X_2; X_2'|X_1) = I(X_2; Y|X_1). \label{assump:unique_aug_app}
\end{align}
the second step is the \textit{unique augmentation}: after observing $X_1$, we create augmented $X_2'$ from $X_2$ to keep the task-relevant information but meanwhile do not affect any information from $X_1$. In Table \ref{tab:augs}, we include some more examples of how unique augmentations could be designed across different datasets.
\input{tables/augs}
\textbf{Final objectives}: If Definitions~\ref{def:optimal_single_app} and~\ref{def:optimal_multi_app} are both satisfied, we can substitute contrastive estimators in the following equations:
{\small
\begin{align}
I_\textsc{NCE}(X_1;X_2|Y) &= \mathbb{E}_{p(y)} \left[ \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2|y)\\x_2^- \sim p(x_2|y)}} \left[ \log \frac{\exp f(x_1,x_2^+, y)}{\sum_k \exp f(x_1, x_2^-, y)} \right] \right] \label{eq:conditional_nce_supervised_app} \\
I_\textsc{NCE-CLUB}(X_1;X_2|Y) &= \mathbb{E}_{p(y)} \left[ \mathbb{E}_{x_1,x_2^+ \sim p(x_1,x_2|y)} \left[ f^*(x_1,x_2^+, y) \right] - \mathbb{E}_{\substack{x_1 \sim p(x_1|y)\\x_2^- \sim p(x_2|y)}} \left[ f^*(x_1,x_2^-, y) \right] \right] \label{eq:conditional_club_supervised_app}
\end{align}
}
by replacing $I(X_i;Y)$ terms with $I(X_i;X_i')$ and replacing $I(X_1;X_2|Y)$ terms with $I(X_1;X_2|X_1', X_2')$:
{\small
\begin{align}
I_\textsc{NCE}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1', x_2')} \left[ \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2|x_1', x_2')\\x_2^- \sim p(x_2|x_1', x_2')}} \left[ \log \frac{\exp f(x_1,x_2^+, x_1', x_2')}{\sum_k \exp f(x_1, x_2^-, x_1', x_2')} \right] \right] \label{eq:nce_final_ssl_app} \\
I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1', x_2')} \Big[ \mathbb{E}_{x_1,x_2^+ \sim p(x_1,x_2|x_1', x_2')} [f^*(x_1,x_2^+, x_1', x_2') ] \nonumber \\
&-\mathbb{E}_{\substack{x_1 \sim p(x_1|x_1', x_2')\\x_2^- \sim p(x_2|x_1', x_2')}} [f^*(x_1,x_2^-, x_1', x_2') ] \Big] \label{eq:nceclub_final_ssl_app}
\end{align}
}
\subsubsection{Implementing conditional CL via kernel}
We restate our objectives below:
{\small
\begin{align}
I_\textsc{NCE}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1', x_2')} \left[ \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2|x_1', x_2')\\x_2^- \sim p(x_2|x_1', x_2')}} \left[ \log \frac{\exp f(x_1,x_2^+, x_1', x_2')}{\sum_k \exp f(x_1, x_2^-, x_1', x_2')} \right] \right] \\
I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1', x_2')} \Big[ \mathbb{E}_{x_1,x_2^+ \sim p(x_1,x_2|x_1', x_2')} [f^*(x_1,x_2^+, x_1', x_2') ] \nonumber \\
& - \mathbb{E}_{\substack{x_1 \sim p(x_1|x_1', x_2')\\x_2^- \sim p(x_2|x_1', x_2')}} [f^*(x_1,x_2^-, x_1', x_2') ] \Big]
\end{align}}
However, sampling from $p(\cdot | x_1', x_2')$ is hard. Since $X_1', X_2'$ are continuous variables, directly sampling from the conditional distributions $p(\cdot | x_1', x_2')$ may be difficult; training a generative model $p_{\theta}(x_1, x_2 | x_1', x_2')$ from augmented data $x_1', x_2'$ to original data $x_1, x_2$ can be expensive and nontrivial in a multimodal setup. In this work, we implement the conditioning in $p(x_1, x_2 | x_1', x_2')$ through concatenation and the details are in Appendix \ref{app:implementation}. Here we discuss an alternative solution to this problem introduced by \citet{tsai2022conditional}. It leverages kernel methods for conditional sampling in contrastive learning by assigning weights to each sampled data given the kernel similarity between conditioned variables, avoiding directly sampling from the conditional distributions or training generative models. In our formulation, given multimodal input $(x_1, x_2)$ with their augmentation $(x_1', x_2')$, we can simply use the technique from \cite{tsai2022conditional} to estimate $I_\textsc{NCE}(X_1;X_2|X_1',X_2')$ and $I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2')$, where the kernel measures the similarity between different pairs $(x_1', x_2')$ of the conditional variable $X_1, X_2$. Specifically,
\begin{align}
I_\textsc{NCE}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1, x_2, x_1', x_2')} \left[ \log \frac{\exp f(x_1,x_2^+)}{\exp f(x_1,x_2) + n * \left[ K_{X_1\indep X_2|X_1', X_2'} \right]_{ii} } \right] \label{eq:nce_kernel}\\
I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') &= \mathbb{E}_{p(x_1, x_2, x_1', x_2')} \left[ f^*(x_1,x_2) - \log \left[K_{X_1\indep X_2|X_1', X_2'}\right]_{ii} \label{eq:nce_club_kernel}\right]
\end{align}
where $K_{X_1\indep X_2|X_1', X_2'} = K_{X_1X_2} (K_{X_1'X_2'} + \lambda {\bf I})^{-1} K_{X_1'X_2'}$ and $\left[K_{X_1\indep X_2|X_1', X_2'}\right]_{ii}$ is the $i$th row and $i$th column of $K_{X_1\indep X_2|X_1', X_2'}$. $K_{X_1X_2}$ is a kernel similarity matrix between $X_1$ and $X_2$, and $K_{X_1'X_2'}$ is a separate kernel similarity matrix between $X_1'$ and $X_2'$. $f^*$ is the optimal solution of Eq.(\ref{eq:nce_kernel}). By leveraging the similarity $K_{X_1'X_2'}$ between conditional variables $X_1'$ and $X_2'$, $K_{X_1\indep X_2|X_1', X_2'}$ transforms the similarity scores between $X_1$ and $X_2$ under unconditional distributions into similarity scores under conditional distributions. Note that the expectations in Eqs.(\ref{eq:nce_kernel}) and (\ref{eq:nce_club_kernel}) are taken over the joint distribution $p(x_1, x_2, x_1', x_2')$, which comes naturally after augmenting both modalities $X_1$ and $X_2$. This method could effectively alleviate the problem of sampling from conditional distributions in our formulation. We refer the reader to \citet{tsai2022conditional} for more details.
\subsection{Final estimators in \names}
\label{app:method3}
\begin{theorem}
(Contrastive estimators for shared and unique information). Under assumptions on single-view augmentations $I(X_1;Y) = I(X_1, X_1')$ (Definition \ref{def:optimal_single_app}) and optimal multi-view augmentation $X_2'$ such that $I(X_1,X_2; X_1',X_2') = I(X_1,X_2; Y)$ (Definition \ref{def:optimal_multi_app}), we can define contrastive objectives for task-relevant shared and unique information with:
\begin{align}
S = I(X_1; X_2; Y) &\ge I_\textsc{NCE}(X_1;X_2) - I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') \label{eq:shared_app}\\
U_i = I(X_i; Y | X_{-i}) &\ge I_\textsc{NCE}(X_i;X_i') - I_\textsc{NCE-CLUB}(X_1;X_2) + I_\textsc{NCE}(X_1;X_2|X_1',X_2') \label{eq:unique_app}
\end{align}
\end{theorem}
\begin{proof}
The objectives follow from the fact that $I_\textsc{NCE}(X_1;X_2)$ and $I_\textsc{NCE}(X_1;X_2|X_1',X_2')$ are lower bounds of $ I(X_1;X_2)$ and $I(X_1; X_2 | Y)$ respectively, and $I_\textsc{NCE-CLUB}(X_1;X_2)$ and $I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2')$ are upper bounds of $ I(X_1;X_2)$ and $I(X_1; X_2 | Y)$ respectively:
\begin{align}
\label{eqn:U2-def-decompose}
S &= I(X_1; X_2; Y) = I(X_1;X_2) - I(X_1; X_2 | Y) \\
&\ge I_\textsc{NCE}(X_1;X_2) - I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') \\
U_i &= I(X_i; Y | X_{-i}) = I(X_i; Y) - (I(X_1; X_2) - I(X_1; X_2 | Y)) \\
& \ge I_\textsc{NCE}(X_i;X_i') - (I_\textsc{NCE-CLUB}(X_1;X_2) - I_\textsc{NCE}(X_1;X_2|X_1',X_2'))
\end{align}
and symmetrically for $U_2$.
\end{proof}
Now we show that \names\ learns both shared and unique task-relevant information. First, we restate the definition of the factorized representations:
\begin{align}
Z_{S_1} &= \argmax_{Z_1 = f_\theta(X_1)} I(Z_1;X_2;Y), &&Z_{S_2} = \argmax_{Z_2 = f_\theta(X_2)} I(Z_2;X_1;Y), \label{eq:final_objectives1_app} \\
Z_{U_1} &= \argmax_{Z_1 = f_\theta(X_1)} I(Z_1;Y|X_2), &&Z_{U_2} = \argmax_{Z_2 = f_\theta(X_2)} I(Z_2;Y|X_1). \label{eq:final_objectives2_app}
\end{align}
where $I(Z_1;X_2;Y) = I(Z_1; X_2) - I(Z_1; X_2 | Y)$ is the shared information and $I(Z_2;X_1;Y) = I(Z_2; X_2) - I(Z_2; X_1 | Y)$ is the unique information.
\begin{theorem}
(Optimality of \names) If $Z_{S_1},Z_{S_2},Z_{U_1},Z_{U_2}$ perfectly maximize Eqs.(\ref{eq:final_objectives1_app}-\ref{eq:final_objectives2_app}) and the estimations in Eqs.(\ref{eq:conditional_nce_supervised}-\ref{eq:nceclub_final_ssl_app}) are tight, we obtain $I(X_1, X_2 ; Y) = I(Z_{S_1}; Z_{S_2}; Y) + I(Z_{U_1}; Y | Z_{S_2}) + I(Z_{U_2}; Y | Z_{S_1})$, suggesting that \names\ learns both shared and unique task-relevant information.
\end{theorem}
\begin{proof}
Because $I(X_1, X_2 ; Y) = I(X_1; X_2; Y) + I(X_1; Y | X_2) + I(X_2; Y | X_1)$, it is sufficient to show that $I(X_1; X_2; Y) = I(Z_{S_1}; Z_{S_2}; Y), I(X_1; Y | X_2) = I(Z_{U_1}; Y | Z_{S_2})$ and $I(X_2; Y | X_1) = I(Z_{U_2}; Y | Z_{S_1})$.
First we show $I(X_1; X_2; Y) = I(Z_{S_1}; Z_{S_2}; Y)$. Crucially, by definition of how $Z_{S_1}$ and $Z_{S_2}$ are optimized to maximize $I(X_1; X_2; Y)$, we have that:
\begin{align}
I(X_1; X_2; Y) = I(Z_{S_1}; X_2; Y) = I(Z_{S_2}; X_1; Y).
\end{align}
We can then obtain
\begin{align}
I(X_1; X_2; Y) &= I(X_1; Z_{S_2}; Y) \\
&= I(X_1; Z_{S_2}; Y | Z_{S_1} ) + I(Z_{S_1}; Z_{S_2} ; X_1; Y ) \\
&= I(Z_{S_2}; Y | Z_{S_1} ) - I(Z_{S_2}; Y | Z_{S_1}, X_1 ) + I(Z_{S_1}; Z_{S_2} ; X_1; Y ) \\
&= I(Z_{S_2}; Y | Z_{S_1} ) - I(Z_{S_2}; Y | X_1 ) + I(Z_{S_1}; Z_{S_2} ; X_1; Y ) \label{eq:step1} \\
&= I(Z_{S_2}; Y | Z_{S_1} ) - I(Z_{S_2}; Y | X_1 ) + I(Z_{S_1}; Z_{S_2} ; Y ) \label{eq:step2} \\
&= I(Z_{S_2}; Y | Z_{S_1} ) + I(Z_{S_1}; Z_{S_2} ; Y ) \\
&= I(Z_{S_1}; Z_{S_2} ; Y ) \label{eq:step3}
\end{align}
where Eq.(\ref{eq:step1}) is because $Z_{S_1}$ are deterministically obtained from $S_1$ and Eq.(\ref{eq:step2}) is because $Z_{S_1}$ maximizes the shared information.
Finally, we go to Eq.(\ref{eq:step3}) $I(Z_{S_2}; Y | Z_{S_1} ) = 0$ as shown in Eqs.(\ref{eq:unique_proof_step1}-\ref{eq:unique_proof_step2}) using the fact that $Z_{S_1}$ is learned to maximize $I(X_1;X_2;Y)$ and $I(Z_{S_1};X_2;Y) = I(X_1;Z_{S_2};Y)$.
Next, we show $I(X_1; Y | X_2) = I(Z_{U_1}; Y | Z_{S_2})$:
\begin{align}
&I(Z_{U_1}; Y | Z_{S_2}) = I(Z_{U_1}; Y | Z_{S_2}, Z_{U_2}) + I(Z_{U_1}; Y; Z_{U_2} | Z_{S_2}),
\end{align}
which is by the chain rule of conditional mutual information. Then we show $I(Z_{U_1}; Y; Z_{U_2} | Z_{S_2}) = 0$:
\begin{align}
I(Z_{U_1}; Y; Z_{U_2} | Z_{S_2}) = I(Z_{U_1}; Z_{U_2} | Z_{S_2}) - I(Z_{U_1}; Z_{U_2} | Y; Z_{S_2}) = 0 - 0 = 0
\end{align}
This is because Eq.(\ref{eq:final_objectives2_app}) leads to $I(Z_{U_1}; Y | X_2) = I(X_1; Y | X_2)$ and $I(Z_{U_2}; Y | X_1) = I(X_2; Y | X_1)$. If the estimations in Eqs.(\ref{eq:conditional_nce_supervised}-\ref{eq:nceclub_final_ssl_app}) are tight, by conditioning and by the previously stated $I(Z_{U_1}; Y | X_2) = I(X_1; Y | X_2)$, $Z_{U_1}$ tightly captures information from only $X_1$ and not in $X_2$. The same applies to $Z_{U_2}$. We have $I(Z_{U_1}; X_2) = I(Z_{U_2}; X_1) = I(Z_{U_1}; Z_{U_2}) = I(Z_{U_1}; Z_{U_2} | T) = 0$ with $T$ being an arbitrary random variable because no shared information exists between $Z_{U_1}$ and $Z_{U_2}$. Then we obtain:
\begin{align}
I(Z_{U_1}; Y | Z_{S_2}, Z_{U_2}) &= I(Z_{U_1}; Y | Z_{S_2}, Z_{U_2}, X_2) + I(Z_{U_1}; Y; X_2 | Z_{S_2}, Z_{U_2}) \\
&= I(Z_{U_1}; Y | X_2)
\end{align}
We use the fact that conditioning on $Z_{S_2}, Z_{U_2}$ and $X_2$ jointly reduces to conditioning on $X_2$ since $Z_{S_2}$ and $Z_{U_2}$ are deterministically obtained from $X_2$. Lastly, since Eqs.(\ref{eq:final_objectives1_app}-\ref{eq:final_objectives2_app}) are satisfied, $Z_{U_1} = \argmax_{Z_1 = f_\theta(X_1)} I(Z_1;Y|X_2)$ therefore $I(Z_{U_1};Y|X_2) = I(X_1;Y|X_2)$. We have:
\begin{align}
I(Z_{U_1}; Y | Z_{S_2}) = I(Z_{U_1}; Y | X_2) = I(X_1; Y | X_2).
\end{align}
The proof for $I(X_2; Y | X_1) = I(Z_{U_2}; Y | Z_{S_1})$ is similar. We now have shown that $I(X_1; X_2; Y) = I(Z_{S_1}; Z_{S_2}; Y)$, $I(X_1; Y | X_2) = I(Z_{U_1}; Y | Z_{S_2})$ and $I(X_2; Y | X_1) = I(Z_{U_2}; Y | Z_{S_1})$, adding up all LHS and RHS we have the theorem.
\end{proof}
\subsection{Extensions to masking and non-contrastive learning}
\label{app:method5}
We now show how similar ideas can be extended to other popular self-supervised learning objectives, such as non-contrastive learning \cite{zbontar2021barlow, bardes2021vicreg} and masked pre-training \cite{he2022masked, devlin2018bert}. Importantly, this paper provides a new principle for multimodel self-supervised learning: (1) learning task-relevant information and (2) removing task-irrelevant information from both shared and unique parts across modalities. Our paper focuses on realizing this principle via multi-view information theory and contrastive learning. Below we provide two potential alternatives to realize this principle on non-contrastive and masking methods, respectively:
\textbf{Non-contrastive learning:} Methods such as Barlow Twins \cite{zbontar2021barlow} and VICReg \cite{bardes2021vicreg} use invariance and covariance regularizations to maximally preserve shared information in the embeddings across two modalities. However, the embeddings learned may contain only contain task-relevant information from the shared part and not unique parts. To use the principle in this paper to capture more task-relevant information from unique parts, one should perform VIC-regularization on $X_1$ features, on $X_2$ features, and on $X_1,X_2$ cross-modal features. When performing VICReg on unimodal features, one should condition on the other modality when performing augmentation. Specifically, similar to the idea of multimodal augmentation in this paper, the augmentation of the second modality should not interfere with the shared part (i.e., do not augment regions referred to by the first modality), making the invariance and covariance regularization of the second modality focus on the augmented modality-unique features. This makes the model learn unique modality features that are not captured by the joint embedding from standard independent augmentations.
\textbf{Masking:} Conceptually, masking \cite{he2022masked, devlin2018bert} can be interpreted as leveraging unmasked regions in the same modality to predict masked regions or leveraging the other modality to predict the masked region. However, the learned representation may not be all task-relevant. To use the principle in this paper to exclude task-irrelevant information and capture more task-relevant information from unique parts, we can perform conditional masking, where masking is conditioned on augmented views (similar to the multimodal augmentation in the paper, where the conditioned views are approximating the labels). As a result, only unique regions in the second modality can be masked out, making the model capture more unique information from the second modality by masked prediction. Here we have only provided high-level intuitions of extensions to these methods, and future work should explore these ideas in more detail.
\section{Experimental Details}
\subsection{Implementation details}
\label{app:implementation}
\textbf{Objective Formulation and Architecture}
In Algorithm~\ref{alg:full} in the main text, we see the sketch for doing contrastive learning with our proposed objectives. To implement all algorithms used in our ablation experiments, we start with two encoders $e_1(\cdot)$ and $e_2(\cdot)$, which takes samples $x_1$ and $x_2$ from the modalities $X_1$ and $X_2$, and outputs corresponding representations $z_1$ and $z_2$. We also have a critic function $f_{\theta}(\cdot,\cdot)$ parametrized by $\theta$ which takes $z_1$ and $z_2$ as inputs and returns a scalar. A popular way to perform contrastive learning aims to maximize $I_\textsc{NCE}(X_1; X_2)$, where
\begin{align}
I_\textsc{NCE}(X_1; X_2) &= \mathbb{E}_{\substack{x_1,x_2^+ \sim p(x_1,x_2)\\x_2^- \sim p(x_2)}} \left[ \log \frac{\exp f_{\theta}(e_1(x_1),e_2(x_2^+))}{\sum_k \exp f_{\theta}(e_1(x_1), e_2(x_2^-))} \right]. \label{eq:nce_implement}
\end{align}
In our algorithms, we follow the derivations in Eqs.(\ref{eq:shared}-\ref{eq:unique}) to maximize each $I_\textsc{NCE}$ objective and minimize each $I_\textsc{NCE-CLUB}$ objective. Therefore, for each objective, we add two additional MLP heads on top of the two encoders and create a separate critic which takes in the outputs of the MLP heads instead of the encoders. In all the experiments, we adopt the concat critic design~\cite{oord2018representation, poole2019variational, DBLP:journals/corr/abs-1910-06222}, where $f_{\theta}(x,y)=h_{\theta}([x,y])$ with $h_{\theta}$ being an MLP.
\textbf{\names-SUP}: In the supervised version of CL which uses label $Y$, the objective we aim to maximize is formulated as
\begin{align}
\mathcal{L}_{\names-SUP} &= I_\textsc{NCE}(X_1;X_2) - I_\textsc{NCE-CLUB}(X_1;X_2|Y) \\
&+ I_\textsc{NCE}(X_1;Y) + I_\textsc{NCE}(X_2;Y) \\
&- I_\textsc{NCE-CLUB}(X_1;X_2) + I_\textsc{NCE}(X_1;X_2|Y). \label{eq:FactorSup_implement}
\end{align}
Each $I_\textsc{NCE}$ and $I_\textsc{NCE-CLUB}$ term in this objective is calculated using its own critic as discussed above. The conditional terms involving the label $Y$ are implicitly modeled by directly concatenating $Y$ to the outputs of both heads before feeding into the critic. To obtain the learned representations $Z_{S_1}$, we concatenate the outputs of the heads on top of the encoder $e_1$ that correspond to the terms $I_\textsc{NCE}(X_1;X_2)$ and $I_\textsc{NCE-CLUB}(X_1;X_2|Y)$. To obtain $Z_{U_1}$, we concatenate $e_1$'s head outputs from the terms $I_\textsc{NCE}(X_1;Y)$, $I_\textsc{NCE-CLUB}(X_1;X_2)$, and $I_\textsc{NCE}(X_1;X_2|Y)$. $Z_{S_2}$ and $Z_{U_2}$ are obtained in a similar fashion, except we use the outputs from $e_2$'s heads instead of $e_1$.
\textbf{\names-SSL}: In the self-supervised version of CL which uses augmentations $X_1'$ and $X_2'$ of the input modalities, the objective we aim to maximize is formulated as
\begin{align}
\mathcal{L}_{\names-SSL} &= I_\textsc{NCE}(X_1;X_2) - I_\textsc{NCE-CLUB}(X_1;X_2|X_1',X_2') \\
&+ I_\textsc{NCE}(X_1;X_1') + I_\textsc{NCE}(X_2;X_2') \\
&- I_\textsc{NCE-CLUB}(X_1;X_2) + I_\textsc{NCE}(X_1;X_2|X_1',X_2'). \label{eq:FactorSSL_implement}
\end{align}
Here the conditional terms are conditioned on the augmentations $X_1'$ and $X_2'$, and we can similarly model it by concatenating the head outputs of $X_1'$ to $X_1$ and the head outputs of $X_2'$ to $X_2$ before feeding into the critic. We use Figure \ref{fig:concat_app} to illustrate this. The way to obtain the learned representations is the same as described in \names-SUP.
\begin{figure}[t]
\centering
\includegraphics[width=0.35\linewidth]{figures/concat.pdf}
\caption{An illustration of conditioning by concatenation in the implementation of \names. Conditioning is done by concatenating $Z_1$, the encoded representation of $X_1$, and $Z_1'$, the encoded representation of $X_1’$. A similar operation is performed for $X_2$ and $X_2'$. The concatenated vectors are then fed to MI estimators, such as $I_\textsc{NCE}$ and $I_\textsc{NCE-CLUB}$ (the figure illustrates $I_\textsc{NCE}$).}
\label{fig:concat_app}
\vspace{-4mm}
\end{figure}
\textbf{Estimation of CMI}:\label{app:discussion_conditional} To estimate the conditional mutual information (CMI) $I(X_1; X_2 | X_1', X_2')$, we can estimate the lower or upper bounds of true CMI \citep{sordoni2021decomposed, mukherjee2020ccmi, ma2021conditional}. However, direct sampling from the conditional distribution $p(x_1, x_2 | x_1', x_2')$ can be expensive because we should consider a different conditional distribution $p(x_1, x_2 | x_1', x_2')$ for each data pair $x_1', x_2'$. \citet{sordoni2021decomposed} address this by concatenating the conditioning variable with the input in the critic: $\phi(x_1, x_2, c)$, and showing that Conditional InfoNCE (Eq.(\ref{eq:nce_final_ssl}) is a lower bound and estimator of CMI. This estimator can be made more exact by further importance sampling \citep{sordoni2021decomposed}. However, adding importance sampling \citep{sordoni2021decomposed} or using more accurate estimators \citep{guo2022tight} comes with a trade-off in complexity. Since we focus on capturing unique information to learn a scalable multimodal representation instead of accurately estimating the CMI, we leveraged a simpler version of the estimator from \citet{sordoni2021decomposed}: generating multiple augmented pairs from $x_1, x_2$, and concatenating the input $x_1, x_2$ and each augmented pair $x_1', x_2'$ to define samples from the conditional distribution $p(x_1, x_2 | x_1', x_2')$. We argue that since augmentations do not significantly change the semantics of images, $p(x_1, x_2 | x_1', x_2')$ could be approximated by $p(x_1'', x_2'' | x_1', x_2')$ where $x_1'', x_2''$ are other augmented pairs in addition to $x_1', x_2'$. In this submission, we use one pair of augmented samples for consistency, but our code easily supports increasing the number of augmented pairs that can improve the accuracy of CMI estimation.
Regardless, our existing one-pair implementation can already show that our estimators are empirically comparable to CMI estimators with guarantees such as \citet{mukherjee2020ccmi} (Table \ref{tab:cmi_lower_supp}), and our estimators empirically satisfy that the lower bound is smaller than the true CMI, and the true CMI smaller than the upper bound, i.e., Conditional InfoNCE $\leq$ true CMI $\leq$ Conditional InfoNCE-CLUB (also in Table \ref{tab:cmi_lower_supp}). We refer the reader to \citet{sordoni2021decomposed} for tighter bounds for CMI.
\textbf{OurCL-SUP}: For this ablation, we remove the factorization and only learn $Z_1$ for $X_1$ and $Z_2$ for $X_2$. The objective we use is the same as that of \names-SUP. The only difference is that we now take $e_1(x_1)$ and $e_2(x_2)$ as the learned representations for inputs $x_1$ and $x_2$.
\textbf{OurCL-SSL}: This is a similar ablation for \names-SSL where we remove the factorization. The objective is the same as that of \names-SSL and we use $e_1(x_1)$ and $e_2(x_2)$ as the learned representations for inputs $x_1$ and $x_2$.
\input{figures/alg_supp}
\textbf{Training Strategy}: In regular contrastive learning using $I_\textsc{NCE}$ as the only objective, we can simply perform gradient descent to minimize $I_\textsc{NCE}$, updating all parameters in the encoders, MLP heads, and critics. However, training any of the four methods above also involves the minimization of the $I_\textsc{NCE-CLUB}$ objectives, which require the optimal critic $f^*$ from $I_\textsc{NCE}$, as stated in Eq.(\ref{eq:nceclub}). Therefore, within each iteration during our training, we need to first obtain the optimal critics for the $I_\textsc{NCE-CLUB}$ terms using the $I_\textsc{NCE}$ objective. We outline the training strategy using a sampling method in Algorithm~\ref{alg:training}. In this algorithm, $\mathcal{L}_{\names}$ can be either $\mathcal{L}_{\names-SUP}$ or $\mathcal{L}_{\names-SSL}$, and $\mathcal{L_\textsc{NCE}}$ is the summation of $I_\textsc{NCE}$ objectives for the $I_\textsc{NCE-CLUB}$ terms. In particular, we have
\begin{align}
\mathcal{L}_\textsc{NCE}=
\begin{cases}
I_\textsc{NCE}(X_1;X_2|Y) + I_\textsc{NCE}(X_1;X_2),& \text{if } \mathcal{L} = \mathcal{L}_{\names-SUP} \text{ ;}\\
I_\textsc{NCE}(X_1;X_2|X_1', X_2') + I_\textsc{NCE}(X_1;X_2), & \text{if } \mathcal{L} = \mathcal{L}_{\names-SSL} \text{ .}
\end{cases}
\end{align}
We define $\phi$ to be the parameters of critics for the $I_\textsc{NCE-CLUB}$ terms, and $\theta$ corresponds to all the rest parameters in the network (parameters of encoders, heads, and critics for $I_\textsc{NCE}$ terms). In the outer loop, we update $\theta$ using the main learning objective $\mathcal{L}$. In the inner loop, we update $\phi$ using the $\mathcal{L}_\textsc{NCE}$ objective, which learns the optimal critics $f^*$ needed to compute the $I_\textsc{NCE-CLUB}$ terms. Ideally in the inner loop we would update $\phi$ until convergence so we get a good approximation to the optimal critic. In practice we found sampling just one batch by setting $k=1$ in Algorithm~\ref{alg:training} works pretty well. Using only one iteration does not have a big impact on the convergence and still produces promising results. More importantly, it significantly reduces the time required for training, and allows our algorithms to have comparable running time to existing contrastive learning methods.
\subsection{Datasets}
\label{appendix:data}
\textbf{Gaussian datasets for MI estimation}: As shown in Figure~\ref{fig:bounds} in the main text, we first demonstrate the quality of our proposed upper bounds $I_\textsc{NCE-CLUB}(X_1; X_2)$ on a toy Gaussian dataset. We obtain the samples $\{(x_i, y_i)\}$ from 4 multivariate Gaussian distribution with dimensions \{20, 50, 100, 200\}. In each dataset, we set the ground truth MI values to be \{2, 4, 6, 8, 10\}, and so we can compute the correlation $\rho$ needed for achieving these MI values using the ground truth MI formula for Multivariate Gaussian: $I(X,Y)=-\frac{d}{2}\log(1-\rho^2)$. At each true MI value we sample 4000 times using a batch size of 64.
\textbf{Synthetic dataset with controlled generation}: We generate data with controllable ratios of task-relevant shared and unique information to analyze the behavior of each contrasive learning objective in Figure~\ref{fig:overview} in the main text. Starting with a set of latent vectors $w_1, w_2, w_s \sim \mathcal{N}(0_d, \Sigma_d^2), d=50$ representing information unique to $X_1,X_2$ and common to both respectively, the concatenated vector $[w_1,w_s]$ is transformed into high-dimensional $x_1$ using a fixed full-rank transformation $T_1$ and likewise $[w_2,w_s]$ to $x_2$ via $T_2$. The label $y$ is generated as a function (with nonlinearity and noise) of varying ratios of $w_s$, $w_1$, and $w_2$ to represent shared and unique task-relevant information. For experiments, we used 1-layer MLPs with 512 hidden size as encoders, and the embedding dimensions are 128 for both modalities. The heads on top of encoders are also 1-layer MLPs with the same hidden and output dimension as the input, and all critics are 1-layer MLPs with 512 hidden size.
\textbf{Multimodal fusion datasets}: We use a collection of 5 real-world datasets provided in MultiBench~\citep{liang2021multibench} and the IRFL dataset to test our method in the context of varying ratios of shared and unique information that is important for the task. In all the datasets below, the heads added on top of the encoders are 1-Layer MLPs with ReLU activations that map the encoder outputs to the same dimensions. All critics are also MLPs with 1 hidden layer of size 512 and ReLU activation.
\input{tables/fusion_appendix}
\input{tables/clip_appendix}
\input{tables/additional}
\begin{enumerate}[topsep=0pt,nosep,leftmargin=*,parsep=6pt,partopsep=0pt]
\item \textbf{\textsc{MIMIC-III}}~\cite{johnson2016mimic} (Medical Information Mart for Intensive Care III) is a large-scale dataset for healthcare which contains records of over 40,000 ICU patients in both forms of times-series data measured by hours and static data (age, gender, ethnicity) in the tabular numerical form. We use the preprocessed data provided in MultiBench~\citep{liang2021multibench}, where the time-series data is measured every 1 hour in a 24-hour period and consists of vectors of size 12, and the tabular data consists of vectors of size 5. The task we use in the experiment is a binary classification on whether the patient fits any ICD-9 code in group 7 (460-519).
In the experiments, we use a 2-layer MLP with 10 hidden layer size for the tabular data modality, and map it to a vector of size 10. The time-series modality is encoded using a GRU with hidden size 30 and followed by a linear layer which projects the output to embeddings of size 15. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
\item \textbf{\textsc{CMU-MOSEI}}~\cite{zadeh2018multimodal} is the largest sentence-level multimodal sentiment and emotion benchmark with $23,000$ monologue videos. It contains more than 65 hours of annotated video from more than 1,000 speakers and 250 topics. Each video is labeled with a sentiment intensity ranging from -3 to 3. In our experiments, we cast the intensity values to a binary classification on whether the sentiment is positive or negative. MultiBench~\citep{liang2021multibench} provides access to the extracted features of the vision, text, and audio modalities, and in our experiments, we use the vision and text features for doing contrastive learning.
In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
\item \textbf{\textsc{CMU-MOSI}}~\cite{zadeh2016mosi} is a similar dataset for multimodal sentiment analysis created from $2,199$ YouTube videos clips. The data focuses on videos that reflect the real-world distribution of speakers expressing their opinions in the form of monologues. The sentiment intensities are labeled continuously from -3 to 3. Again we cast the label into a binary classification on whether the sentiment is positive or negative, and we used the extracted vision and text features for contrastive learning.
In our experiments we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
\item \textbf{\textsc{UR-FUNNY}}~\citep{hasan2019ur} is the first large-scale dataset for humor detection in human speech. The dataset consists of samples from more than $16,000$ TED talk videos with speakers from diverse backgrounds sharing their ideas. The laughter markup is used to filter out 8,257 humorous punchlines from the transcripts. The context is extracted from the prior sentences to the punchline. Using a similar approach, 8,257 negative samples are chosen at random intervals where the last sentence is not immediately followed by a laughter marker. The task is to classify whether there is humor or not using the vision and text modalities.
In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
\item \textbf{\textsc{MUStARD}}~\citep{castro2019towards} is a corpus of $690$ videos for research in sarcasm detection from popular TV shows including Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholics Anonymous. It contains audiovisual utterances together with the textual context. We use the preprocessed features of the vision and text modalities for doing contrastive learning and performing sarcasm detection.
In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
\item \textbf{\textsc{IRFL}}~\cite{yosef2023irfl} is a dataset for understanding multimodal figurative languages. It contains $6,697$ matching images and figurative captions (rather than literal captions) of three types of figurative languages: idiom, simile, and metaphor. The original data for the matching task is provided in the form of 1 caption, 3 distractor images, and 1 matching image. We convert it into a fusion task by only collecting the matching image and text pairs and assigning labels using the type of figurative language it belongs to.
For this dataset, we do not train from scratch. Instead, we performed continued pretraining using our proposed objectives on pretrained CLIP~\cite{radford2021learning} models. We used the CLIP-VIT-B/32 model and its pretrained image and text encoders. We performed training for 10 epochs using the Adam optimizer with a 1e-6 learning rate.
\end{enumerate}
\subsection{Additional analysis and results}
\label{appendix:results}
\input{tables/cmi_lower_appendix}
\input{tables/cmi_upper_appendix}
\input{tables/infomin_probe}
\textbf{Fusion experiments}: In Table~\ref{tab:fusion_supp} and \ref{tab:fig_supp} we present more detailed results on the Multibench~\cite{liang2021multibench} and IRFL~\cite{yosef2023irfl} datasets computed from 5 independent runs. \names\ significantly outperforms the baselines that do not capture both shared and unique information in both supervised and self-supervised settings, particularly on \textsc{MuStARD} (where unique information expresses sarcasm, such as sardonic facial expressions or ironic tone of voice), and on \textsc{MIMIC} (with unique health indicators and sensor readings). There are also big improvements on the two sentiment analysis datasets \textsc{MOSEI} and \textsc{MOSI}, with $6.8\%$ and $21.9\%$ increases respectively when compared to SupCon~\cite{khosla2020supervised}.
In Table~\ref{tab:fig_supp}, we also see that \names\ substantially improves the state-of-the-art in classifying images and figurative captions which are not literally descriptive of the image on \textsc{IRFL}, outperforming zero-shot and fine-tuned CLIP~\cite{radford2021learning} as well as continued pre-training baselines on top of CLIP. While the supervised version gives the best results overall, the self-supervised version with our proposed unique augmentations also performs better than independent augmentations, indicating that in the case without label information, we should always try to find and use unique augmentations when possible. In our experiments, we use word masking for text augmentations. For independent image augmentations, we use cropping, flipping, and color jittering. The unique augmentation simply removes the cropping operation, as illustrated in Figure~\ref{fig:augs} in the main text.
\textbf{Additional experiments on high shared information and low unique information}: In Table~\ref{tab:additional} we include additional results using our method on the CIFAR10~\cite{Krizhevsky} and MNIST~\cite{deng2012mnist} datasets. Our method outperforms the self-supervised contrastive learning on both datasets as expected, and roughly maintains the same performance as supervised contrastive learning. Therefore, in cases with abundant shared information (two modalities with high shared information or two different views generated from augmentations), our method recovers the performance of existing methods that do not capture unique information.
\textbf{Experiments on CMI estimator verification}: In Table~\ref{tab:cmi_lower_supp} and Table~\ref{tab:cmi_upper_supp}, we include experiment results which verify that computing the conditional MI lower and upper bounds via concatenation indeed yields reliable estimates. In particular, we aim to verify that the the Conditional InfoNCE objective gives a lower bound of the CMI, and the Conditional InfoNCE-CLUB objective gives an upper bound of the CMI. We follow the experiment setups in~\cite{mukherjee2020ccmi}, presenting the true CMI and results of our estimators on synthetic data with fixing dimension of representation $Z$ and varying samples $n$, and fixing samples $n$ and varying $d_z$. The specific implementations used for conditional InfoNCE and conditional InfoNCE-CLUB can be found in Equation~\ref{eq:conditional_nce_supervised} and Equation~\ref{eq:conditional_club_supervised}, respectively. The results indicate that our Conditional InfoNCE gives estimations smaller than the true CMI, and Conditional InfoNCE-CLUB gives estimations greater than the true CMI. The performances are comparable to estimators in ~\cite{mukherjee2020ccmi}, suggesting that our method yields valid and competitive lower and upper bounds for CMI.
\textbf{Empirical verification on InfoMin assumption}\label{assump:infomin_empirical_app}: To verify the InfoMin assumption \citep{tian2020makes} ($I(Z_1; Y | X_2) = I(Z_2; Y | X_1) = 0$), we use the same synthetic dataset as in Table \ref{tab:probing} and measure $I(Z_1; Y | X_2)$. The results are shown in Table \ref{tab:probing_mi_conditional}: we get $I(X_1; X_2) = 12.29$ and $I(Z_1; Y | X_2)=0.4$. $I(Z_1; Y | X_2)$ is much smaller and closer to zero than $I(Z_1; Y | X_2)$, indicating that the InfoMin assumption holds in practice.
\textbf{Compute resources}: All experiments in this paper are run on a single NVIDIA A100 GPU. It takes about 10 to 12 GPU hours to train the model on the CIFAR10~\cite{Krizhevsky} for 300 epochs, and all the other experiments can be finished within 1 GPU hour using our specified hyperparameters.
\begin{table*}[t]
\centering
\fontsize{8}{11}\selectfont
\setlength\tabcolsep{2pt}
\vspace{-0mm}
\caption{More examples of optimal single-view and multi-view augmentations.}
\centering
\begin{tabular}{l c c c c c c c }
\hline
& & & & Standard Aug & \textbf{Unique Aug} \\
Dataset & $X_1$ & $X_2$ &$X_1'$ & $X_2'$ & $X_2''$ \\
\hline
Cartoon & Caption & Image & Word Masking & Crop + Flip + Resize & Flip + Resize \\
MIMIC & Signals & Tables & Time Warping & CutMix \cite{yun2019cutmix} on All Features & CutMix on Nonclinical Features \\
MOSEI & Transcripts & Video+Audio & Word Masking & Noise Injection on Any Frames & Noise Injection on Silent Frames\\
UR-FUNNY & Transcripts & Video+Audio & Word Masking & Noise Injection on Any Frames & Noise Injection on Silent Frames \\
MUsTARD & Transcripts & Video+Audio & Word Masking & Noise Injection on Any Frames & Noise Injection on Silent Frames \\
\hline \hline
\end{tabular}
\vspace{-0mm}
\label{tab:augs}
\end{table*}
\begin{figure*}[tbp]
\centering
\begin{minipage}{0.55\textwidth}
\vspace{-0mm}
\begin{algorithm}[H]
\begin{algorithmic}
\REQUIRE Multimodal dataset $\{\mathbf{X_1}, \mathbf{X_2}$\}.
\vspace{.2em}\hrule\vspace{.5em}
\STATE $\theta$, $\phi$ $\leftarrow$ Initialize network parameters.
\WHILE{not converged}
\STATE $\{\vx_1, \vx_2\}$ $\leftarrow$ Sampled batch from $\{\mathbf{X_1}, \mathbf{X_2}$\}
\STATE $\theta$ $\leftarrow$ Update parameters by maximizing $\mathcal{L}_{\textsc{\names}}$
\FOR{$i=1$ to $k$}
\STATE $\{\vx_1', \vx_2'\}$ $\leftarrow$ Sampled batch from $\{\mathbf{X_1}, \mathbf{X_2}$\}
\STATE $\phi$ $\leftarrow$ Update parameters by maximizing $\mathcal{L}_{\textsc{NCE}}$
\ENDFOR
\ENDWHILE
\RETURN $\theta$, $\phi$
\end{algorithmic}
\caption{CL training with sampling}
\label{alg:training}
\end{algorithm}
\end{minipage}
\end{figure*}
\begin{table*}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{4pt}
\vspace{-4mm}
\caption{Results on MultiBench~\citep{liang2021multibench} datasets with varying shared and unique information: \names\ achieves strong results vs self-supervised (top $5$ rows) and supervised (bottom $3$ rows) baselines that do not have unique representations, factorization, upper-bounds to remove irrelevant information, and multimodal augmentations.}
\centering
\begin{tabular}{l|ccccc}
\hline \hline
Model & \textsc{MIMIC} & \textsc{MOSEI} & \textsc{MOSI} & \textsc{UR-FUNNY} & \textsc{MUStARD} \\
\hline
SimCLR~\cite{chen2020simple} & 66.7 $\pm$ 0.0\% & 71.9 $\pm$ 0.3\% & 47.8 $\pm$ 1.8\% & 50.1 $\pm$ 1.9 \% & 53.5 $\pm$ 2.9\%\\
Cross+Self~\cite{wang2022rethinking} & 65.2 $\pm$ 0.0\% & 71.1 $\pm$ 0.2\% & 48.6 $\pm$ 1.2\% & 56.5 $\pm$ 0.7\% & 53.9 $\pm$ 4.5\%\\
Cross+Self+Fact~\cite{yuan2021multimodal} & 65.5 $\pm$ 0.0\% & 71.9 $\pm$ 0.2\% & 49.0 $\pm$ 1.1\% & 59.9 $\pm$ 0.9\% & 53.9 $\pm$ 4.0\% \\
OurCL-SSL & 65.2 $\pm$ 0.0\% & 71.2 $\pm$ 0.2\% & 49.0 $\pm$ 0.8\% & 58.8 $\pm$ 1.3\% & 54.0 $\pm$ 2.5\%\\
\names-SSL & \textbf{67.3 $\pm$ 0.0}\% & \textbf{74.5 $\pm$ 0.1\%} & \textbf{51.2 $\pm$ 1.6\%} & \textbf{60.5 $\pm$ 0.8\%} & \textbf{55.8 $\pm$ 0.9}\%\\
\hline
SupCon~\cite{khosla2020supervised} & 67.4 $\pm$ 0.0\% & 71.0 $\pm$ 0.1\% & 47.2 $\pm$ 1.2\% & 50.1 $\pm$ 2.0\% & 52.7 $\pm$ 2.2\%\\
OurCL-SUP & 68.2 $\pm$ 0.0\% & 71.1 $\pm$ 0.2\% & 65.3 $\pm$ 0.8\% & 58.3 $\pm$ 1.1\% & 65.1 $\pm$ 1.8\%\\
\names-SUP & \textbf{76.8 $\pm$ 0.0}\% & \textbf{77.8 $\pm$ 0.3\%} & \textbf{69.1 $\pm$ 0.6\%} & \textbf{63.5 $\pm$ 0.8\%} & \textbf{69.9 $\pm$ 1.9}\%\\
\hline \hline
\end{tabular}
\vspace{-7mm}
\label{tab:fusion_supp}
\end{table*}
\begin{table*}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{2pt}
\vspace{6mm}
\caption{Continued pre-training on CLIP with our \names\ objectives on classifying images and figurative language. Our approach shows strong results as compared to standard fine-tuning and continued pre-training.}
\centering
\begin{tabular}{l|c c}
\hline \hline
Task & \textsc{IRFL} \\
\hline
Zero-shot CLIP~\cite{radford2021learning} & 89.2 $\pm$ 0.0\%\\
SimCLR~\cite{chen2020simple} & 91.6 $\pm$ 0.0\%\\
Cross+Self~\cite{wang2022rethinking,yuan2021multimodal} & 91.1 $\pm$ 1.2\%\\
\names-IndAug & 91.6 $\pm$ 1.3\% \\
\names-SSL & \textbf{93.8 $\pm$ 1.4}\%\\
\hline
Fine-tuned CLIP~\cite{radford2021learning} & 96.4 $\pm$ 0.0\%\\
SupCon~\cite{khosla2020supervised} & 87.7 $\pm$ 4.7\%\\
\names-SUP & \textbf{98.3 $\pm$ 1.2}\%\\
\hline \hline
\end{tabular}
\label{tab:fig_supp}
\vspace{-4mm}
\end{table*}
\begin{table*}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{2pt}
\caption{Additional experiments on CIFAR10~\cite{Krizhevsky} and MNIST~\cite{deng2012mnist} datasets using our \names\ objectives on image classification.}
\centering
\begin{tabular}{l|c c c}
\hline \hline
Task & \textsc{CIFAR10} & \textsc{MNIST}\\
\hline
SimCLR~\cite{chen2020simple} & 87.0\% & 98.84\%\\
SupCon~\cite{khosla2020supervised} & 92.7\% & 99.38\%\\
\names-SUP & 91.3\% & 99.21\%\\
\hline \hline
\end{tabular}
\label{tab:additional}
\end{table*}
\begin{table*}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{4pt}
% \vspace{-3mm}
\caption{We verify our conditional lower and upper bound estimators on a synthetic dataset with fixed dimension of representation $d_z$ and varying number of samples $n$.}
\centering
\begin{tabular}{l|cccc}
\hline \hline
Number of samples ($\times 10^3$), $d_z=20$ & 5 & 10 & 20 & 50 \\
\hline
CCMI (MI-Diff + Classifier) & 2.03 & 2.06 & 2.15 & 2.20\\
Conditional InfoNCE & 2.19 & 2.20 & 2.20 & 2.20 \\
Conditional InfoNCE-CLUB & 3.45 & 3.53 & 2.98 & 2.86 \\
True CMI & 2.32 & 2.32 & 2.32 & 2.32\\
\hline \hline
\end{tabular}
\vspace{-2mm}
\label{tab:cmi_lower_supp}
\end{table*}
\begin{table}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{4pt}
\caption{We verify our conditional lower and upper bound estimators on a synthetic dataset with varying dimensions of representation $d_z$ and fixed number of samples $n$.}
\centering
\begin{tabular}{l|ccccc}
\hline \hline
Dimension $d_z$, $n=2\times10^4$ & 1 & 10 & 20 & 50 & 100\\
\hline
CCMI (MI-Diff + Classifier) & 2.30 & 2.18 & 2.15 & 1.98 & 1.67\\
Conditional InfoNCE & 2.18 & 2.20 & 2.20 & 2.26 & 2.30\\
Conditional InfoNCE-CLUB & 3.70 & 2.95 & 2.98 & 2.79 & 2.86 \\
True CMI & 2.32 & 2.32 & 2.32 & 2.32 & 2.32 \\
\hline \hline
\end{tabular}
\label{tab:cmi_upper_supp}
\end{table}
\begin{table*}[t]
\centering
\fontsize{9}{11}\selectfont
\setlength\tabcolsep{4pt}
\caption{We probe whether the InfoMin assumption from Tian et al., $I(Z_1; Y | X_2) = 0$ and $I(Z_2; Y | X_1) = 0$, is reasonable for Theorem 1. Compared to the shared information $I(X_1; X_2)$, $I(Z_1; Y | X_2)$ is much smaller and closer to zero, indicating that the InfoMin assumption is reasonable, and Theorem 1 holds in practice.}
\centering
\begin{tabular}{c|c|c|c|c|c}
\hline \hline
$I(X_1, Y;X_2)$ & $I(X_1; X_2)$ & $I(Z_1; Y | X_2)$ & $I(X_2, Y;X_1)$ & $I(X_2; X_1)$ & $I(Z_2; Y | X_1)$ \\
\hline
12.69 & 12.29 & 0.40 & 11.31 & 10.92 & 0.38 \\
\hline \hline
\end{tabular}
\label{tab:probing_mi_conditional}
\end{table*}
\begin{thebibliography}{93}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Akbari et~al.(2021)Akbari, Yuan, Qian, Chuang, Chang, Cui, and
Gong]{akbari2021vatt}
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin
Cui, and Boqing Gong.
\newblock Vatt: Transformers for multimodal self-supervised learning from raw
video, audio and text.
\newblock \emph{Advances in Neural Information Processing Systems},
34:\penalty0 24206--24221, 2021.
\bibitem[Alayrac et~al.(2020)Alayrac, Recasens, Schneider, Arandjelovi{\'c},
Ramapuram, De~Fauw, Smaira, Dieleman, and Zisserman]{alayrac2020self}
Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja
Arandjelovi{\'c}, Jason Ramapuram, Jeffrey De~Fauw, Lucas Smaira, Sander
Dieleman, and Andrew Zisserman.
\newblock Self-supervised multimodal versatile networks.
\newblock \emph{Advances in Neural Information Processing Systems},
33:\penalty0 25--37, 2020.
\bibitem[Arandjelovic and Zisserman(2017)]{arandjelovic2017look}
Relja Arandjelovic and Andrew Zisserman.
\newblock Look, listen and learn.
\newblock In \emph{Proceedings of the IEEE international conference on computer
vision}, pages 609--617, 2017.
\bibitem[Bachman et~al.(2019)Bachman, Hjelm, and
Buchwalter]{bachman2019learning}
Philip Bachman, R~Devon Hjelm, and William Buchwalter.
\newblock Learning representations by maximizing mutual information across
views.
\newblock \emph{Advances in neural information processing systems}, 32, 2019.
\bibitem[Baevski et~al.(2020)Baevski, Zhou, Mohamed, and
Auli]{baevski2020wav2vec}
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.
\newblock wav2vec 2.0: A framework for self-supervised learning of speech
representations.
\newblock \emph{Advances in neural information processing systems},
33:\penalty0 12449--12460, 2020.
\bibitem[Bardes et~al.(2021)Bardes, Ponce, and LeCun]{bardes2021vicreg}
Adrien Bardes, Jean Ponce, and Yann LeCun.
\newblock Vicreg: Variance-invariance-covariance regularization for
self-supervised learning.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Bell(2003)]{bell2003co}
Anthony~J Bell.
\newblock The co-information lattice.
\newblock In \emph{Proceedings of the fifth international workshop on
independent component analysis and blind signal separation: ICA}, volume
2003, 2003.
\bibitem[Bengio et~al.(2013)Bengio, Courville, and
Vincent]{Bengio:2013:RLR:2498740.2498889}
Yoshua Bengio, Aaron Courville, and Pascal Vincent.
\newblock Representation learning: A review and new perspectives.
\newblock \emph{TPAMI}, 35\penalty0 (8), August 2013.
\bibitem[Bolukbasi et~al.(2016)Bolukbasi, Chang, Zou, Saligrama, and
Kalai]{bolukbasi2016man}
Tolga Bolukbasi, Kai-Wei Chang, James~Y Zou, Venkatesh Saligrama, and Adam~T
Kalai.
\newblock Man is to computer programmer as woman is to homemaker? debiasing
word embeddings.
\newblock \emph{Advances in neural information processing systems}, 29, 2016.
\bibitem[Bugliarello et~al.(2021)Bugliarello, Cotterell, Okazaki, and
Elliott]{bugliarello2021multimodal}
Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott.
\newblock Multimodal pretraining unmasked: A meta-analysis and a unified
framework of vision-and-language berts.
\newblock \emph{Transactions of the Association for Computational Linguistics},
9:\penalty0 978--994, 2021.
\bibitem[Caron et~al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and
Joulin]{caron2020unsupervised}
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and
Armand Joulin.
\newblock Unsupervised learning of visual features by contrasting cluster
assignments.
\newblock \emph{Advances in neural information processing systems},
33:\penalty0 9912--9924, 2020.
\bibitem[Castro et~al.(2019)Castro, Hazarika, P{\'e}rez-Rosas, Zimmermann,
Mihalcea, and Poria]{castro2019towards}
Santiago Castro, Devamanyu Hazarika, Ver{\'o}nica P{\'e}rez-Rosas, Roger
Zimmermann, Rada Mihalcea, and Soujanya Poria.
\newblock Towards multimodal sarcasm detection (an \_obviously\_ perfect
paper).
\newblock \emph{arXiv preprint arXiv:1906.01815}, 2019.
\bibitem[Chen et~al.(2020)Chen, Kornblith, Norouzi, and Hinton]{chen2020simple}
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
\newblock A simple framework for contrastive learning of visual
representations.
\newblock In \emph{International conference on machine learning}, pages
1597--1607. PMLR, 2020.
\bibitem[Chen and He(2021)]{chen2021exploring}
Xinlei Chen and Kaiming He.
\newblock Exploring simple siamese representation learning.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition}, pages 15750--15758, 2021.
\bibitem[Chen et~al.(2021)Chen, Xie, and He]{chen2021empirical}
Xinlei Chen, Saining Xie, and Kaiming He.
\newblock An empirical study of training self-supervised vision transformers.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 9640--9649, 2021.
\bibitem[Cheng et~al.(2020)Cheng, Hao, Dai, Liu, Gan, and Carin]{cheng2020club}
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence
Carin.
\newblock Club: A contrastive log-ratio upper bound of mutual information.
\newblock In \emph{International conference on machine learning}, pages
1779--1788. PMLR, 2020.
\bibitem[Chi et~al.(2022)Chi, Shand, Yu, Chang, Zhao, and
Tian]{chi2022conditional}
Jianfeng Chi, William Shand, Yaodong Yu, Kai-Wei Chang, Han Zhao, and Yuan
Tian.
\newblock Conditional supervised contrastive learning for fair text
classification.
\newblock \emph{arXiv preprint arXiv:2205.11485}, 2022.
\bibitem[Cover and Thomas(1991)]{cover1991information}
Thomas~M Cover and Joy~A Thomas.
\newblock Information theory and statistics.
\newblock \emph{Elements of information theory}, 1\penalty0 (1):\penalty0
279--335, 1991.
\bibitem[Deng(2012)]{deng2012mnist}
Li~Deng.
\newblock The mnist database of handwritten digit images for machine learning
research.
\newblock \emph{IEEE Signal Processing Magazine}, 29\penalty0 (6):\penalty0
141--142, 2012.
\bibitem[Devlin et~al.(2018)Devlin, Chang, Lee, and Toutanova]{devlin2018bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
\newblock Bert: Pre-training of deep bidirectional transformers for language
understanding.
\newblock \emph{arXiv preprint arXiv:1810.04805}, 2018.
\bibitem[Eysenbach et~al.(2022)Eysenbach, Zhang, Levine, and
Salakhutdinov]{eysenbach2022contrastive}
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ~R Salakhutdinov.
\newblock Contrastive learning as goal-conditioned reinforcement learning.
\newblock \emph{Advances in Neural Information Processing Systems},
35:\penalty0 35603--35620, 2022.
\bibitem[Feder and Merhav(1994)]{feder1994relations}
Meir Feder and Neri Merhav.
\newblock Relations between entropy and error probability.
\newblock \emph{IEEE Transactions on Information theory}, 40\penalty0
(1):\penalty0 259--266, 1994.
\bibitem[Federici et~al.(2020)Federici, Dutta, Forr{\'e}, Kushman, and
Akata]{federici2020learning}
Marco Federici, Anjan Dutta, Patrick Forr{\'e}, Nate Kushman, and Zeynep Akata.
\newblock Learning robust representations via multi-view information
bottleneck.
\newblock \emph{arXiv preprint arXiv:2002.07017}, 2020.
\bibitem[Gao et~al.(2021)Gao, Yao, and Chen]{gao2021simcse}
Tianyu Gao, Xingcheng Yao, and Danqi Chen.
\newblock Simcse: Simple contrastive learning of sentence embeddings.
\newblock \emph{arXiv preprint arXiv:2104.08821}, 2021.
\bibitem[Grill et~al.(2020)Grill, Strub, Altch{\'e}, Tallec, Richemond,
Buchatskaya, Doersch, Avila~Pires, Guo, Gheshlaghi~Azar,
et~al.]{grill2020bootstrap}
Jean-Bastien Grill, Florian Strub, Florent Altch{\'e}, Corentin Tallec, Pierre
Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila~Pires, Zhaohan
Guo, Mohammad Gheshlaghi~Azar, et~al.
\newblock Bootstrap your own latent-a new approach to self-supervised learning.
\newblock \emph{Advances in neural information processing systems},
33:\penalty0 21271--21284, 2020.
\bibitem[Guo et~al.(2022)Guo, Chen, Wang, Yang, Deng, Huang, Carin, Li, and
Tao]{guo2022tight}
Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Jing Huang, Larry
Carin, Fan Li, and Chenyang Tao.
\newblock Tight mutual information estimation with contrastive fenchel-legendre
optimization.
\newblock \emph{Advances in Neural Information Processing Systems},
35:\penalty0 28319--28334, 2022.
\bibitem[Hasan et~al.(2019)Hasan, Rahman, Zadeh, Zhong, Tanveer, Morency, and
Hoque]{hasan2019ur}
Md~Kamrul Hasan, Wasifur Rahman, AmirAli~Bagher Zadeh, Jianyuan Zhong,
Md~Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed~Ehsan Hoque.
\newblock Ur-funny: A multimodal language dataset for understanding humor.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 2046--2056, 2019.
\bibitem[He et~al.(2020)He, Fan, Wu, Xie, and Girshick]{he2020momentum}
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
\newblock Momentum contrast for unsupervised visual representation learning.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition}, pages 9729--9738, 2020.
\bibitem[He et~al.(2022)He, Chen, Xie, Li, Doll{\'a}r, and
Girshick]{he2022masked}
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll{\'a}r, and Ross
Girshick.
\newblock Masked autoencoders are scalable vision learners.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pages 16000--16009, 2022.
\bibitem[Higgins et~al.(2016)Higgins, Matthey, Pal, Burgess, Glorot, Botvinick,
Mohamed, and Lerchner]{higgins2016beta}
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot,
Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.
\newblock beta-vae: Learning basic visual concepts with a constrained
variational framework.
\newblock 2016.
\bibitem[Hjelm et~al.(2018)Hjelm, Fedorov, Lavoie-Marchildon, Grewal, Bachman,
Trischler, and Bengio]{hjelm2018learning}
R~Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil
Bachman, Adam Trischler, and Yoshua Bengio.
\newblock Learning deep representations by mutual information estimation and
maximization.
\newblock In \emph{International Conference on Learning Representations}, 2018.
\bibitem[Hsu and Glass(2018)]{hsu2018disentangling}
Wei-Ning Hsu and James Glass.
\newblock Disentangling by partitioning: A representation learning framework
for multimodal sensory data.
\newblock \emph{arXiv preprint arXiv:1805.11264}, 2018.
\bibitem[Huang et~al.(2021{\natexlab{a}})Huang, Patrick, Hu, Neubig, Metze, and
Hauptmann]{huang2021multilingual}
Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and
Alexander Hauptmann.
\newblock Multilingual multimodal pre-training for zero-shot cross-lingual
transfer of vision-language models.
\newblock \emph{arXiv preprint arXiv:2103.08849}, 2021{\natexlab{a}}.
\bibitem[Huang et~al.(2021{\natexlab{b}})Huang, Du, Xue, Chen, Zhao, and
Huang]{huang2021makes}
Yu~Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang.
\newblock What makes multi-modal learning better than single (provably).
\newblock \emph{Advances in Neural Information Processing Systems},
34:\penalty0 10944--10956, 2021{\natexlab{b}}.
\bibitem[Jain et~al.(2021)Jain, Guo, Srinivasan, Chen, Kudugunta, Jia, Yang,
and Baldridge]{jain2021mural}
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao
Jia, Yinfei Yang, and Jason Baldridge.
\newblock Mural: multimodal, multitask retrieval across languages.
\newblock \emph{arXiv preprint arXiv:2109.05125}, 2021.
\bibitem[Jia et~al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung, Li, and
Duerig]{jia2021scaling}
Chao Jia, Yinfei Yang, Ye~Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le,
Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
\newblock Scaling up visual and vision-language representation learning with
noisy text supervision.
\newblock In \emph{International Conference on Machine Learning}, pages
4904--4916. PMLR, 2021.
\bibitem[Johnson et~al.(2016)Johnson, Pollard, Shen, Lehman, Feng, Ghassemi,
Moody, Szolovits, Anthony~Celi, and Mark]{johnson2016mimic}
Alistair~EW Johnson, Tom~J Pollard, Lu~Shen, Li-wei~H Lehman, Mengling Feng,
Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony~Celi, and
Roger~G Mark.
\newblock Mimic-iii, a freely accessible critical care database.
\newblock \emph{Scientific data}, 3\penalty0 (1):\penalty0 1--9, 2016.
\bibitem[Kahana and Hoshen(2022)]{kahana2022contrastive}
Jonathan Kahana and Yedid Hoshen.
\newblock A contrastive objective for learning disentangled representations.
\newblock In \emph{Computer Vision--ECCV 2022: 17th European Conference, Tel
Aviv, Israel, October 23--27, 2022, Proceedings, Part XXVI}, pages 579--595.
Springer, 2022.
\bibitem[Kenton and Toutanova(2019)]{kenton2019bert}
Jacob Devlin Ming-Wei~Chang Kenton and Lee~Kristina Toutanova.
\newblock Bert: Pre-training of deep bidirectional transformers for language
understanding.
\newblock In \emph{Proceedings of NAACL-HLT}, pages 4171--4186, 2019.
\bibitem[Khosla et~al.(2020)Khosla, Teterwak, Wang, Sarna, Tian, Isola,
Maschinot, Liu, and Krishnan]{khosla2020supervised}
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip
Isola, Aaron Maschinot, Ce~Liu, and Dilip Krishnan.
\newblock Supervised contrastive learning.
\newblock \emph{Advances in neural information processing systems},
33:\penalty0 18661--18673, 2020.
\bibitem[Kim et~al.(2022)Kim, Choi, Hwang, Lee, and Lee]{kim2022transferring}
Byoungjip Kim, Sungik Choi, Dasol Hwang, Moontae Lee, and Honglak Lee.
\newblock Transferring pre-trained multimodal representations with cross-modal
similarity matching.
\newblock \emph{Advances in Neural Information Processing Systems},
35:\penalty0 30826--30839, 2022.
\bibitem[Krizhevsky et~al.()Krizhevsky, Nair, and Hinton]{Krizhevsky}
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.
\newblock Cifar-10 (canadian institute for advanced research).
\newblock URL \url{http://www.cs.toronto.edu/~kriz/cifar.html}.
\bibitem[Lee et~al.(2020)Lee, Yu, Kim, Breuel, Kautz, and
Song]{lee2020parameter}
Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song.
\newblock Parameter efficient multimodal transformers for video representation
learning.
\newblock \emph{arXiv preprint arXiv:2012.04124}, 2020.
\bibitem[Liang et~al.(2021{\natexlab{a}})Liang, Lyu, Fan, Wu, Cheng, Wu, Chen,
Wu, Lee, Zhu, Salakhutdinov, and Morency]{liang2021multibench}
Paul~Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie
Chen, Peter Wu, Michelle~A Lee, Yuke Zhu, Ruslan Salakhutdinov, and
Louis-Philippe Morency.
\newblock Multibench: Multiscale benchmarks for multimodal representation
learning.
\newblock \emph{NeurIPS Datasets and Benchmarks Track}, 2021{\natexlab{a}}.
\bibitem[Liang et~al.(2021{\natexlab{b}})Liang, Wu, Morency, and
Salakhutdinov]{liang2021towards}
Paul~Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov.
\newblock Towards understanding and mitigating social biases in language
models.
\newblock In \emph{International Conference on Machine Learning}, pages
6565--6576. PMLR, 2021{\natexlab{b}}.
\bibitem[Liang et~al.(2022{\natexlab{a}})Liang, Lyu, Fan, Mo, Yogatama,
et~al.]{liang2022highmmt}
Paul~Pu Liang, Yiwei Lyu, Xiang Fan, Shengtong Mo, Dani Yogatama, et~al.
\newblock Highmmt: Towards modality and task generalization for high-modality
representation learning.
\newblock \emph{arXiv preprint arXiv:2203.01311}, 2022{\natexlab{a}}.
\bibitem[Liang et~al.(2022{\natexlab{b}})Liang, Zadeh, and
Morency]{liang2022foundations}
Paul~Pu Liang, Amir Zadeh, and Louis-Philippe Morency.
\newblock Foundations and recent trends in multimodal machine learning:
Principles, challenges, and open questions.
\newblock \emph{arXiv preprint arXiv:2209.03430}, 2022{\natexlab{b}}.
\bibitem[Liang et~al.(2023)Liang, Cheng, Fan, Ling, Nie, Chen, Deng, Mahmood,
Salakhutdinov, and Morency]{liang2023quantifying}
Paul~Pu Liang, Yun Cheng, Xiang Fan, Chun~Kai Ling, Suzanne Nie, Richard Chen,
Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency.
\newblock Quantifying \& modeling feature interactions: An information
decomposition framework.
\newblock \emph{arXiv preprint arXiv:2302.12247}, 2023.
\bibitem[Lu et~al.(2019)Lu, Batra, Parikh, and Lee]{lu2019vilbert}
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.
\newblock Vilbert: pretraining task-agnostic visiolinguistic representations
for vision-and-language tasks.
\newblock In \emph{Proceedings of the 33rd International Conference on Neural
Information Processing Systems}, pages 13--23, 2019.
\bibitem[Ma et~al.(2021)Ma, Tsai, Liang, Zhao, Zhang, Salakhutdinov, and
Morency]{ma2021conditional}
Martin~Q Ma, Yao-Hung~Hubert Tsai, Paul~Pu Liang, Han Zhao, Kun Zhang, Ruslan
Salakhutdinov, and Louis-Philippe Morency.
\newblock Conditional contrastive learning for improving fairness in
self-supervised learning.
\newblock \emph{arXiv preprint arXiv:2106.02866}, 2021.
\bibitem[Marsh and Domas~White(2003)]{marsh2003taxonomy}
Emily~E Marsh and Marilyn Domas~White.
\newblock A taxonomy of relationships between images and text.
\newblock \emph{Journal of documentation}, 2003.
\bibitem[McGill(1954)]{mcgill1954multivariate}
William McGill.
\newblock Multivariate information transmission.
\newblock \emph{Transactions of the IRE Professional Group on Information
Theory}, 4\penalty0 (4):\penalty0 93--111, 1954.
\bibitem[Meng et~al.(2021)Meng, Xiong, Bajaj, Bennett, Han, Song,
et~al.]{meng2021coco}
Yu~Meng, Chenyan Xiong, Payal Bajaj, Paul Bennett, Jiawei Han, Xia Song, et~al.
\newblock Coco-lm: Correcting and contrasting text sequences for language model
pretraining.
\newblock \emph{Advances in Neural Information Processing Systems},
34:\penalty0 23102--23114, 2021.
\bibitem[Mukherjee et~al.(2020)Mukherjee, Asnani, and
Kannan]{mukherjee2020ccmi}
Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan.
\newblock Ccmi: Classifier based conditional mutual information estimation.
\newblock In \emph{Uncertainty in artificial intelligence}, pages 1083--1093.
PMLR, 2020.
\bibitem[Neelakantan et~al.(2022)Neelakantan, Xu, Puri, Radford, Han, Tworek,
Yuan, Tezak, Kim, Hallacy, et~al.]{neelakantan2022text}
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse~Michael Han, Jerry
Tworek, Qiming Yuan, Nikolas Tezak, Jong~Wook Kim, Chris Hallacy, et~al.
\newblock Text and code embeddings by contrastive pre-training.
\newblock \emph{arXiv preprint arXiv:2201.10005}, 2022.
\bibitem[Nguyen et~al.(2010)Nguyen, Wainwright, and
Jordan]{nguyen2010estimating}
XuanLong Nguyen, Martin~J Wainwright, and Michael~I Jordan.
\newblock Estimating divergence functionals and the likelihood ratio by convex
risk minimization.
\newblock \emph{IEEE Transactions on Information Theory}, 56\penalty0
(11):\penalty0 5847--5861, 2010.
\bibitem[Oord et~al.(2018)Oord, Li, and Vinyals]{oord2018representation}
Aaron van~den Oord, Yazhe Li, and Oriol Vinyals.
\newblock Representation learning with contrastive predictive coding.
\newblock \emph{arXiv preprint arXiv:1807.03748}, 2018.
\bibitem[Ozair et~al.(2019)Ozair, Lynch, Bengio, Van~den Oord, Levine, and
Sermanet]{ozair2019wasserstein}
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van~den Oord, Sergey Levine,
and Pierre Sermanet.
\newblock Wasserstein dependency measure for representation learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Poole et~al.(2019)Poole, Ozair, Van Den~Oord, Alemi, and
Tucker]{poole2019variational}
Ben Poole, Sherjil Ozair, Aaron Van Den~Oord, Alex Alemi, and George Tucker.
\newblock On variational bounds of mutual information.
\newblock In \emph{International Conference on Machine Learning}, pages
5171--5180. PMLR, 2019.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal,
Sastry, Askell, Mishkin, Clark, et~al.]{radford2021learning}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et~al.
\newblock Learning transferable visual models from natural language
supervision.
\newblock In \emph{International Conference on Machine Learning}, pages
8748--8763. PMLR, 2021.
\bibitem[Rahate et~al.(2022)Rahate, Walambe, Ramanna, and
Kotecha]{rahate2022multimodal}
Anil Rahate, Rahee Walambe, Sheela Ramanna, and Ketan Kotecha.
\newblock Multimodal co-learning: challenges, applications with datasets,
recent advances and future directions.
\newblock \emph{Information Fusion}, 81:\penalty0 203--239, 2022.
\bibitem[Schneider et~al.(2019)Schneider, Baevski, Collobert, and
Auli]{schneider2019wav2vec}
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli.
\newblock wav2vec: Unsupervised pre-training for speech recognition.
\newblock \emph{arXiv preprint arXiv:1904.05862}, 2019.
\bibitem[Shan et~al.(2022)Shan, Yin, Sun, Tian, Wu, and Wang]{shan2022ernievil}
Bin Shan, Weichong Yin, Yu~Sun, Hao Tian, Hua Wu, and Haifeng Wang.
\newblock Ernie-vil 2.0: Multi-view contrastive learning for image-text
pre-training, 2022.
\bibitem[Shannon(1948)]{shannon1948mathematical}
Claude~Elwood Shannon.
\newblock A mathematical theory of communication.
\newblock \emph{The Bell system technical journal}, 27\penalty0 (3):\penalty0
379--423, 1948.
\bibitem[Shi et~al.(2019)Shi, Paige, Torr, et~al.]{shi2019variational}
Yuge Shi, Brooks Paige, Philip Torr, et~al.
\newblock Variational mixture-of-experts autoencoders for multi-modal deep
generative models.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Shwartz-Ziv and LeCun(2023)]{shwartz2023compress}
Ravid Shwartz-Ziv and Yann LeCun.
\newblock To compress or not to compress--self-supervised learning and
information theory: A review.
\newblock \emph{arXiv preprint arXiv:2304.09355}, 2023.
\bibitem[Song and Ermon(2019)]{DBLP:journals/corr/abs-1910-06222}
Jiaming Song and Stefano Ermon.
\newblock Understanding the limitations of variational mutual information
estimators.
\newblock \emph{CoRR}, abs/1910.06222, 2019.
\newblock URL \url{http://arxiv.org/abs/1910.06222}.
\bibitem[Sordoni et~al.(2021)Sordoni, Dziri, Schulz, Gordon, Bachman, and
Des~Combes]{sordoni2021decomposed}
Alessandro Sordoni, Nouha Dziri, Hannes Schulz, Geoff Gordon, Philip Bachman,
and Remi~Tachet Des~Combes.
\newblock Decomposed mutual information estimation for contrastive
representation learning.
\newblock In \emph{International Conference on Machine Learning}, pages
9859--9869. PMLR, 2021.
\bibitem[Sridharan and Kakade(2008)]{sridharan2008information}
Karthik Sridharan and Sham~M Kakade.
\newblock An information theoretic framework for multi-view learning.
\newblock In \emph{Conference on Learning Theory}, 2008.
\bibitem[Tian et~al.(2020{\natexlab{a}})Tian, Krishnan, and
Isola]{tian2019contrastive}
Yonglong Tian, Dilip Krishnan, and Phillip Isola.
\newblock Contrastive multiview coding.
\newblock \emph{ECCV}, 2020{\natexlab{a}}.
\bibitem[Tian et~al.(2020{\natexlab{b}})Tian, Sun, Poole, Krishnan, Schmid, and
Isola]{tian2020makes}
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and
Phillip Isola.
\newblock What makes for good views for contrastive learning?
\newblock \emph{Advances in Neural Information Processing Systems},
33:\penalty0 6827--6839, 2020{\natexlab{b}}.
\bibitem[Tosh et~al.(2021)Tosh, Krishnamurthy, and Hsu]{tosh2021contrastive}
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu.
\newblock Contrastive learning, multi-view redundancy, and linear models.
\newblock In \emph{Algorithmic Learning Theory}, pages 1179--1206. PMLR, 2021.
\bibitem[Tsai et~al.()Tsai, Li, Liu, Liao, Salakhutdinov, and
Morency]{tsailearning}
Yao-Hung~Hubert Tsai, Tianqin Li, Weixin Liu, Peiyuan Liao, Ruslan
Salakhutdinov, and Louis-Philippe Morency.
\newblock Learning weakly-supervised contrastive representations.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[Tsai et~al.(2019)Tsai, Liang, Zadeh, Morency, and
Salakhutdinov]{tsai2018learning}
Yao-Hung~Hubert Tsai, Paul~Pu Liang, Amir Zadeh, Louis-Philippe Morency, and
Ruslan Salakhutdinov.
\newblock Learning factorized multimodal representations.
\newblock \emph{ICLR}, 2019.
\bibitem[Tsai et~al.(2020{\natexlab{a}})Tsai, Wu, Salakhutdinov, and
Morency]{tsai2020self}
Yao-Hung~Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency.
\newblock Self-supervised learning from a multi-view perspective.
\newblock In \emph{International Conference on Learning Representations},
2020{\natexlab{a}}.
\bibitem[Tsai et~al.(2020{\natexlab{b}})Tsai, Zhao, Yamada, Morency, and
Salakhutdinov]{tsai2020neural}
Yao-Hung~Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and
Russ~R Salakhutdinov.
\newblock Neural methods for point-wise dependency estimation.
\newblock \emph{Advances in Neural Information Processing Systems},
33:\penalty0 62--72, 2020{\natexlab{b}}.
\bibitem[Tsai et~al.(2022)Tsai, Li, Ma, Zhao, Zhang, Morency, and
Salakhutdinov]{tsai2022conditional}
Yao-Hung~Hubert Tsai, Tianqin Li, Martin~Q Ma, Han Zhao, Kun Zhang,
Louis-Philippe Morency, and Ruslan Salakhutdinov.
\newblock Conditional contrastive learning with kernel.
\newblock \emph{arXiv preprint arXiv:2202.05458}, 2022.
\bibitem[Tschannen et~al.(2019)Tschannen, Djolonga, Rubenstein, Gelly, and
Lucic]{tschannen2019mutual}
Michael Tschannen, Josip Djolonga, Paul~K Rubenstein, Sylvain Gelly, and Mario
Lucic.
\newblock On mutual information maximization for representation learning.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\bibitem[Vergara and Est{\'e}vez(2014)]{vergara2014review}
Jorge~R Vergara and Pablo~A Est{\'e}vez.
\newblock A review of feature selection methods based on mutual information.
\newblock \emph{Neural computing and applications}, 24:\penalty0 175--186,
2014.
\bibitem[Wang et~al.(2022)Wang, Guo, Deng, and Lu]{wang2022rethinking}
Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu.
\newblock Rethinking minimal sufficient representation in contrastive learning.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pages 16041--16050, 2022.
\bibitem[Williams and Beer(2010)]{williams2010nonnegative}
Paul~L Williams and Randall~D Beer.
\newblock Nonnegative decomposition of multivariate information.
\newblock \emph{arXiv preprint arXiv:1004.2515}, 2010.
\bibitem[Wu and Goodman(2018)]{wu2018multimodal}
Mike Wu and Noah Goodman.
\newblock Multimodal generative models for scalable weakly-supervised learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 31, 2018.
\bibitem[Wu et~al.(2020)Wu, Zhuang, Mosse, Yamins, and Goodman]{wu2020mutual}
Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman.
\newblock On mutual information in contrastive learning for visual
representations.
\newblock \emph{arXiv preprint arXiv:2005.13149}, 2020.
\bibitem[Yang et~al.(2022{\natexlab{a}})Yang, Li, Zhang, Xiao, Liu, Yuan, and
Gao]{yang2022unified}
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce~Liu, Lu~Yuan, and
Jianfeng Gao.
\newblock Unified contrastive learning in image-text-label space,
2022{\natexlab{a}}.
\bibitem[Yang et~al.(2022{\natexlab{b}})Yang, Duan, Tran, Xu, Chanda, Chen,
Zeng, Chilimbi, and Huang]{yang2022visionlanguage}
Jinyu Yang, Jiali Duan, Son Tran, Yi~Xu, Sampath Chanda, Liqun Chen, Belinda
Zeng, Trishul Chilimbi, and Junzhou Huang.
\newblock Vision-language pre-training with triple contrastive learning,
2022{\natexlab{b}}.
\bibitem[Ye and Yao(2022)]{ye2022contrastive}
Zesheng Ye and Lina Yao.
\newblock Contrastive conditional neural processes.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pages 9687--9696, 2022.
\bibitem[Yosef et~al.(2023)Yosef, Bitton, and Shahaf]{yosef2023irfl}
Ron Yosef, Yonatan Bitton, and Dafna Shahaf.
\newblock Irfl: Image recognition of figurative language.
\newblock \emph{arXiv preprint arXiv:2303.15445}, 2023.
\bibitem[Yuan et~al.(2021)Yuan, Lin, Kuen, Zhang, Wang, Maire, Kale, and
Faieta]{yuan2021multimodal}
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire,
Ajinkya Kale, and Baldo Faieta.
\newblock Multimodal contrastive training for visual representation learning.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pages 6995--7004, 2021.
\bibitem[Yun et~al.(2019)Yun, Han, Oh, Chun, Choe, and Yoo]{yun2019cutmix}
Sangdoo Yun, Dongyoon Han, Seong~Joon Oh, Sanghyuk Chun, Junsuk Choe, and
Youngjoon Yoo.
\newblock Cutmix: Regularization strategy to train strong classifiers with
localizable features.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on
computer vision}, pages 6023--6032, 2019.
\bibitem[Zadeh et~al.(2016)Zadeh, Zellers, Pincus, and Morency]{zadeh2016mosi}
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency.
\newblock Mosi: multimodal corpus of sentiment intensity and subjectivity
analysis in online opinion videos.
\newblock \emph{arXiv preprint arXiv:1606.06259}, 2016.
\bibitem[Zadeh et~al.(2020)Zadeh, Liang, and Morency]{zadeh2020foundations}
Amir Zadeh, Paul~Pu Liang, and Louis-Philippe Morency.
\newblock Foundations of multimodal co-learning.
\newblock \emph{Information Fusion}, 64:\penalty0 188--193, 2020.
\bibitem[Zadeh et~al.(2018)Zadeh, Liang, Poria, Cambria, and
Morency]{zadeh2018multimodal}
AmirAli~Bagher Zadeh, Paul~Pu Liang, Soujanya Poria, Erik Cambria, and
Louis-Philippe Morency.
\newblock Multimodal language analysis in the wild: Cmu-mosei dataset and
interpretable dynamic fusion graph.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 2236--2246,
2018.
\bibitem[Zbontar et~al.(2021)Zbontar, Jing, Misra, LeCun, and
Deny]{zbontar2021barlow}
Jure Zbontar, Li~Jing, Ishan Misra, Yann LeCun, and St{\'e}phane Deny.
\newblock Barlow twins: Self-supervised learning via redundancy reduction.
\newblock In \emph{International Conference on Machine Learning}, pages
12310--12320. PMLR, 2021.
\end{thebibliography}