Interpolation Consistency Training for Semi-Supervised Learning
Extracted source text ยท arXiv 1903.03825
\documentclass{article}
\pdfpagewidth=8.5in
\pdfpageheight=11in
\usepackage[preprint]{nips_2018}
\usepackage{times}
\usepackage{soul}
\usepackage{url}
%\usepackage[hidelinks]{hyperref}
\usepackage[utf8]{inputenc}
\usepackage[small]{caption}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{wrapfig,lipsum,booktabs}
\usepackage{subcaption}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{booktabs}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{makecell}
\usepackage{microtype}
\urlstyle{same}
\usepackage{tikz}
\usepackage{pgfplots}
\usepackage{setspace}
\definecolor{mygreen}{HTML}{167dde}
\definecolor{myred}{HTML}{f22835}
\colorlet{greenfill}{mygreen!20!white}
\colorlet{purplefill}{red!50!blue!30!white}
\colorlet{redfill}{myred!20!white}
\colorlet{moreredfill}{myred!40!white}
\usetikzlibrary{positioning}
\usetikzlibrary{backgrounds}
\usetikzlibrary{calc}
\usetikzlibrary{fit}
\newcommand{\mix}[2]{\text{Mix}_\lambda(#1, #2)}
\newcommand{\ict}{Interpolation Consistency Training}
\newcommand{\ICT}{ICT}
\newcommand{\inputmixup}{Input Mixup}
\DeclareMathOperator*{\expectation}{\mathbb{E}}
\title{Interpolation Consistency Training for Semi-Supervised Learning}
\author{
Vikas Verma \\
Aalto Univeristy, Finland \hfill\\
Montr\'{e}al Institute for Learning Algorithms \\
\texttt{vikas.verma@aalto.fi} \\
\And
Alex Lamb \\
Montr\'{e}al Institute for Learning Algorithms \\
\texttt{lambalex@iro.umontreal.ca} \\
\AND
Juho Kannala \\
Aalto Univeristy, Finland \hfill\\
\texttt{juho.kannala@aalto.fi} \\
\And
\hspace{0.5cm}Yoshua Bengio \\
\hspace{0.5cm}Montr\'{e}al Institute for Learning Algorithms \\
\hspace{0.5cm}CIFAR Senior Fellow \\
\hspace{0.5cm}\texttt{yoshua.umontreal@gmail.com} \\
\AND
\hspace{0.8cm}David Lopez-Paz\\
\hspace{0.8cm}Facebook AI Research\\
\hspace{0.8cm}\texttt{dlp@fb.com} \\
}
\begin{document}
\maketitle
\begin{abstract}
We introduce \ict{} (\ICT{}), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm.
%
\ICT{} encourages
%
the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points.
%
In classification problems, \ICT{} moves the decision boundary to low-density regions of the data distribution.
%
Our experiments show that \ICT{} achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark datasets.
\end{abstract}
\section{Introduction}
Deep learning achieves excellent performance in supervised learning tasks where labeled data is abundant \citep{lecun2015deep}.
%
However, labeling large amounts of data is often prohibitive due to time, financial, and expertise constraints.
%
As machine learning permeates an increasing variety of domains, the number of applications where unlabeled data is voluminous and labels are scarce increases.
%
For instance, recognizing documents in extinct languages, where a machine learning system has access to a few labels, produced by highly-skilled scholars \citep{clanuwat2018deep}.
The goal of Semi-Supervised Learning (SSL) \citep{chapple} is to leverage large amounts of unlabeled data to improve the performance of supervised learning over small datasets.
%
Often, SSL algorithms use unlabeled data to learn additional structure about the input distribution.
%
For instance, the existence of cluster structures in the input distribution could hint the separation of samples into different labels.
%
This is often called the \textit{cluster assumption}: if two samples belong to the same cluster in the input distribution, then they are likely to belong to the same class.
%
The cluster assumption is equivalent to the \textit{low-density separation assumption}: the decision boundary should lie in the low-density regions.
%
The equivalence is easy to infer:
%
%Since clusters are high-density regions, decision boundaries traversing low-density regions do not partition data from the same cluster into different labels.
A decision boundary which lies in a high-density region, will cut a cluster into two different classes, requiring that samples from different classes lie in the same cluster; which is the violation of the \textit{cluster assumption}.
%
The \textit{low-density separation assumption} has inspired many recent \textit{consistency-regularization} semi-supervised learning techniques,
including %entropy Minimization \citep{entmin},
the $\Pi$-model \citep{sajjadi,laine2016temporal}, temporal ensembling \citep{laine2016temporal}, VAT \citep{miyato2017vat}, and the Mean-Teacher \citep{meanteacher}.
\begin{figure*}[t!]
\centering
\begin{subfigure}{0.30\textwidth}
\centering
\includegraphics[scale=0.18,trim={1cm 1cm 1cm 1cm},clip]{figures/mymoon100.png}
\caption[]%
{After 100 updates}
\end{subfigure}
\begin{subfigure}{0.30\textwidth}
\centering
\includegraphics[scale=0.18,trim={1cm 1cm 1cm 1cm},clip]{figures/mymoon500.png}
\caption[]%
{After 500 updates}
\end{subfigure}
\begin{subfigure}{0.30\textwidth}
\centering
\includegraphics[scale=0.18,trim={1cm 1cm 1cm 1cm},clip]{figures/mymoon.png}
\caption[]%
{After 1000 updates}
\end{subfigure}
\caption{\ict{} (\ICT{}) applied to the ``two moons'' dataset, when three labels per class (large dots) and a large amount of unlabeled data (small dots) is available.
When compared to supervised learning (red), \ICT{} encourages a decision boundary traversing a low-density region that would better reflect the structure of
the unlabeled data.
Both methods employ a multilayer perceptron with three hidden ReLU layers of twenty neurons.}
\label{fig:two_moon}
\end{figure*}
\textit{Consistency regularization} methods for semi-supervised learning enforce the low-density separation assumption by encouraging invariant prediction $f(u) = f(u + \delta)$ for perturbations $u + \delta$ of unlabeled points $u$.
%
Such consistency and small prediction error can be satisfied simultaneously if and only if the decision boundary traverses a low-density path.
Different consistency regularization techniques vary in how they choose the unlabeled data perturbations $\delta$.
%
One simple alternative is to use random perturbations $\delta$.
%
However, random perturbations are inefficient in high dimensions, as only a tiny proportion of input perturbations are capable of pushing the decision boundary into low-density regions.
%
To alleviate this issue, Virtual Adversarial Training or VAT \citep{miyato2017vat}, searches for small perturbations $\delta$ that maximize the change in the prediction of the model.
%
This involves computing the gradient of the predictor with respect to its input, which can be expensive for large neural network models.
This additional computation makes VAT \citep{miyato2017vat} and other related methods such as \citep{vaad} less appealing in situations where unlabeled data is available in large quantities.
%
Furthermore, recent research has shown that training with adversarial perturbations can hurt generalization performance \citep{nakkiran2019adversarial,tsipras2018robustness}.
To overcome the above limitations, we propose the \ict{} (\ICT{}), an efficient consistency regularization technique for state-of-the-art semi-supervised learning.
%
In a nutshell, \ICT{} regularizes semi-supervised learning by encouraging consistent predictions $f(\alpha u_1 + (1 - \alpha) u_2) = \alpha f(u_1) + (1 - \alpha) f(u_2)$ at interpolations $\alpha u_1 + (1 - \alpha) u_2$ of unlabeled points $u_1$ and $u_2$.
%The central hypothesis motivating \ict{} is:
%\begin{center}
% \emph{Interpolating an unlabeled sample by a random amount in the direction of other randomly chosen unlabeled sample results in a better perturbation direction than what is achieved by adding random noise.}
%\end{center}
Our experimental results on the benchmark datasets CIFAR10 and SVHN and neural network architectures CNN-13 \citep{laine2016temporal,miyato2017vat,meanteacher,vaad,smooth} and WRN28-2 \citep{oliver2018ssl} outperform (or are competitive with) the state-of-the-art methods. \ICT{} is simpler and more computation efficient than several of the recent SSL algorithms, making it an appealing approach to SSL.
%
%This provides strong support to the aforementioned hypothesis.
%
Figure~\ref{fig:two_moon} illustrates how \ICT{} learns a decision boundary traversing a low density region in the ``two moons'' problem.
\section{Interpolation Consistency Training}
Given a mixup \citep{mixup} operation:
\begin{equation*}
\mix{a}{b} = \lambda \cdot a + (1 - \lambda) \cdot b,
\end{equation*}
\ict (\ICT{}) trains a prediction model $f_\theta$ to provide consistent predictions at interpolations of unlabeled points:
\begin{equation*}
f_\theta(\mix{u_j}{u_k}) \approx \mix{f_{\theta'}(u_j)}{f_{\theta'}(u_k)},
\end{equation*}
where $\theta'$ is a moving average of $\theta$ (Figure~\ref{fig:icl}).
%
But, why do interpolations between unlabeled samples provide a good consistency perturbation for semi-supervised training?
To begin with, observe that the most useful samples on which the consistency regularization should be applied are the samples near the decision boundary.
%
Adding a small perturbation $\delta$ to such low-margin unlabeled samples $u_j$ is likely to push $u_j + \delta$ over the other side of the decision boundary.
%
This would violate the \textit{low-density separation assumption}, making $u_j + \delta$ a good place to apply consistency regularization.
%
These violations do not occur at high-margin unlabeled points that lie far away from the decision boundary.
Back to low-margin unlabeled points $u_j$, how can we find a perturbation $\delta$ such that $u_j$ and $u_j+\delta$ lie on opposite sides of the decision boundary?
%
Although tempting, using random perturbations is an inefficient strategy, since the subset of directions approaching the decision boundary is a tiny fraction of the ambient space.
%
%
Instead, consider interpolations $u_j + \delta = \mix{u_j}{u_k}$ towards a second randomly selected unlabeled examples $u_k$.
%
Then, the two unlabeled samples $u_j$ and $u_k$ can either:
\begin{enumerate}
\item lie in the same cluster,
\item lie in different clusters but belong to the same class,
\item lie on different clusters and belong to the different classes.
\end{enumerate}
\begin{figure*}[t]
\centering
\begin{tikzpicture}[
scale=1.0,
black!50, text=black,
font=\small,
%every node/.append style={font=\medium},
node distance=1mm,
dnode/.style={
align=center,
% The shape:
font=\small,
rectangle,minimum size=15mm,rounded corners,
% The rest
inner sep=20pt},
rnode/.style={
align=center,
font=\small,
% The shape:
rectangle,
minimum width=5mm,
minimum height=6mm,
rounded corners,
% The rest
inner sep=3pt,
thin, draw},
tuplenode/.style={
align=center,
% The shape:
rectangle,minimum size=10mm,rounded corners,
% The rest
inner sep=15pt},
darrow/.style={
rounded corners,-latex,shorten <=5pt,shorten >=1pt,line width=2mm},
mega thick/.style={line width=2pt}
]
\matrix[row sep=3.2mm, column sep=2mm] {
\node (xL1) [rnode, minimum width=20mm, top color=green!20, bottom color=green!20] {$(x_i, y_i) \sim \mathcal{D}_{L}$ \\\includegraphics[width=.07\textwidth,trim={0cm 0.0cm 0cm 0.0cm},clip]{figures/digits3.png}}; && &&
\node (yL) [rnode, minimum width=15mm, top color=green!20, bottom color=green!20] {$f_{\theta}(x_{i})$};&&
\node (lossL) [rnode, minimum width=20mm, top color=gray!10, bottom color = gray!10] {Supervised loss \\ $(\hat{y}_i,y_i)$};\\
\node (xULi) [rnode, minimum width=20mm, top color=redfill,bottom color=redfill] {$u_j \sim \mathcal{D}_{UL}$\\\includegraphics[width=.07\textwidth,trim={0cm 5.1cm 0cm 5.2cm},clip]{figures/digits.png}};&&
&&
\node (yULi) [rnode, minimum width=15mm, top color=redfill,bottom color=redfill] {$f_{\theta'}(u_{j})$};&&&&
\node (lossT) [rnode, minimum width=15mm, top color=gray!10, bottom color = gray!10] {Supervised Loss +\\ $w_t \cdot$ Consistency\\ Loss};
\\
&&
\node (x_mix) [rnode, top color = redfill, bottom color = greenfill, minimum width=15mm] {$\mix{u_j}{u_k}$\\\includegraphics[width=0.07\textwidth,trim={0cm 10.5cm 0cm 0cm},clip]{figures/digits.png}};&&
\node (y_mix) [rnode, minimum width=15mm,top color=redfill,bottom color=greenfill] {$f_{\theta}(u_{m})$};&&
\node (lossUL) [rnode, minimum width=20mm, top color=gray!10, bottom color = gray!10] {Consistency loss \\ $(\mix{\hat{y}_j}{\hat{y}_k}, \hat{y}_{m})$};
\\
\node (xULj) [rnode, minimum width=20mm, top color=greenfill,bottom color=greenfill] {$u_k \sim \mathcal{D}_{UL}$\\\includegraphics[width=0.07\textwidth,trim={0cm 0cm 0cm 10.3cm},clip]{figures/digits.png}};&&
&&
\node (yULj) [rnode, minimum width=15mm, top color=greenfill,bottom color=greenfill] {$f_{\theta'}(u_{k})$};
\\
};
%\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=blue!70!cyan] (input) to node[label=left:Mixup $\lambda \sim p_{\alpha}(\lambda)$]{} (xm);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (xL1) to node[label=above:$x_i$]{} (yL);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (yL) to node[label=above:$\hat{y}_i$]{} (lossL);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (lossL) to node[]{} (lossT);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (xULi) to node[label=above:$u_j$]{} (yULi);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (xULi) to node[label=above:$u_j$]{} (x_mix);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (xULj) to node[label=above:$u_k$]{} (x_mix);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (x_mix) to node[label=above:$u_m$]{} (y_mix);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (xULj) to node[label=above:$u_k$]{} (yULj);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (yULi) to node[label=above:$\hat{y}_j$]{} (lossUL);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (yULj) to node[label=below:$\hat{y}_k$]{} (lossUL);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (y_mix) to node[label=above:$\hat{y}_{m}$]{} (lossUL);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan] (lossUL) to node[]{} (lossT);
\draw[-latex,shorten <=1pt,shorten >=1pt,thick,color=black!70!cyan, bend right = 15] (xL1) to node[label=above:$y_i$]{} (lossL);
\draw[decorate,decoration={brace,amplitude=10pt,mirror}]
(xL1.north west) -- (xL1.south west) node [midway,xshift=-0.5cm,yshift=1.2cm,left,rotate=90] {Labeled Sample};
\draw[decorate,decoration={brace,amplitude=10pt,mirror}]
(xULi.north west) -- (xULj.south west) node [midway,xshift=-0.5cm,yshift=1.5cm,left,rotate=90] {Unlabeled Samples};
\end{tikzpicture}
\caption{\ict{} (\ICT{}) learns a student network $f_\theta$ in a semi-supervised manner.
%
To this end, \ICT{} uses a mean-teacher $f_{\theta'}$, where the teacher parameters $\theta'$ are an exponential moving average of the student parameters $\theta$.
%
During training, the student parameters $\theta$ are updated to encourage consistent predictions $f_\theta(\mix{u_j}{u_k}) \approx \mix{f_{\theta'}(u_j)}{f_{\theta'}(u_k)}$, and correct predictions for labeled examples $x_i$.
%
}
\label{fig:icl}
\end{figure*}
Assuming the cluster assumption, the probability of (1) decreases as the number of classes increases.
%
The probability of (2) is low if we assume that the number of clusters for each class is balanced.
%
Finally, the probability of (3) is the highest.
%
Then, assuming that one of $(u_j, u_k)$ lies near the decision boundary (it is a good candidate for enforcing consistency), it is likely (because of the high probability of (3)) that the interpolation towards $u_k$ points towards a region of low density, followed by the cluster of the other class.
%
Since this is a good direction to move the decision, the interpolation is a good perturbation for consistency-based regularization.
Our exposition has argued so far that interpolations between random unlabeled samples are likely to fall in low-density regions.
%
Thus, such interpolations are good locations where consistency-based regularization could be applied.
%
But how should we label those interpolations?
%
Unlike random or adversarial perturbations of single unlabeled examples $u_j$, our scheme involves two unlabeled examples $(u_j, u_k)$.
%
Intuitively, we would like to push the decision boundary as far as possible from the class boundaries, as it is well known that decision boundaries with large margin generalize better \citep{srm}.
%
In the supervised learning setting, one method to achieve large-margin decision boundaries is mixup \citep{mixup}.
%
In mixup, the decision boundary is pushed far away from the class boundaries by enforcing the prediction model to change linearly in between samples.
%
This is done by training the model $f_\theta$ to predict $\mix{y}{y'}$ at location $\mix{x}{x'}$, for random pairs of labeled samples $((x, y), (x', y'))$.
%
Here we extend mixup to the semi-supervised learning setting by training the model $f_\theta$ to predict the ``fake label'' $\mix{f_\theta(u_j)}{f_\theta(u_k)}$ at location $\mix{u_j}{u_k}$.
%
In order to follow a more conservative consistent regularization, we encourage the model $f_\theta$ to predict the fake label $\mix{f_{\theta'}(u_j)}{f_{\theta'}(u_k)}$ at location $\mix{u_j}{u_k}$, where $\theta'$ is a moving average of $\theta$, also known as a \emph{mean-teacher} \citep{meanteacher}.
We are now ready to describe in detail the proposed \ict{} (\ICT{}).
%
Consider access to labeled samples $(x_i, y_i)\sim \mathcal{D}_{L}$, drawn from the joint distribution $P(X, Y)$.
%
Also, consider access to unlabeled samples $u_j, u_k \sim \mathcal{D}_{UL}$, drawn from the marginal distribution $P(X) = \frac{P(X, Y)}{P(Y | X)}$
%
Our learning goal is to train a model $f_\theta$, able to predict $Y$ from $X$.
%
By using stochastic gradient descent, at each iteration $t$, update the parameters $\theta$ to minimize
\begin{equation*}
L = L_S + w(t) \cdot L_{US}
\end{equation*}
where $L_S$ is the usual cross-entropy supervised learning loss over labeled samples $\mathcal{D}_L$,
and $L_{US}$ is our new interpolation consistency regularization term.
%
These two losses are computed on top of (labeled and unlabeled) minibatches, and the ramp function $w(t)$ increases the importance of the consistency regularization term $L_{US}$ after each iteration.
%
To compute $L_{US}$, sample two minibatches of unlabeled points $u_j$ and $u_k$, and compute their fake labels $\hat{y}_j = f_{\theta'}(u_j)$ and $ \hat{y}_k = f_{\theta'}(u_k)$, where $\theta'$ is an moving average of $\theta$ \citep{meanteacher}.
%
Second, compute the interpolation $u_{m}= \mix{u_j}{u_k}$, as well as the model prediction at that location, $\hat{y}_{m} = f_\theta(u_{m})$.
%
Third, update the parameters $\theta$ as to bring the prediction $\hat{y}_m$ closer to the interpolation of the fake labels $\mix{\hat{y}_j}{\hat{y}_k}$.
%
The discrepancy between the prediction $\hat{y}_m$ and $\mix{\hat{y}_j}{\hat{y}_k}$ can be measured using any loss; in our experiments, we use the mean squared error.
%
Following \citep{mixup}, on each update we sample a random $\lambda$ from $\text{Beta}(\alpha, \alpha)$.
In sum, the population version of our \ICT{} term can be written as:
\begin{align}
\label{eq: consistency loss}
\mathcal{L}_{US} = &\expectation_{u_j, u_k \sim P(X)}\,
\expectation_{\lambda \sim \text{Beta}(\alpha, \alpha)}
\ell(f_\theta(\mix{u_j}{u_k}), \mix{f_{\theta'}(u_j)}{f_{\theta'}(u_k)})
\end{align}
\ICT{} is summarized in Figure~\ref{fig:icl} and Algorithm~\ref{algo}.
\newcommand{\Req}{\textbf{Require:}\hspace*{0.5em}}
\newcommand{\X}{\hspace*{3mm}}
\newcommand{\XX}{\X\X}
\newcommand{\cm}[1]{$\triangleright$ #1}
\begin{algorithm*}[t]
\setstretch{1.45}
\caption{
The Interpolation Consistency Training (ICT) Algorithm
}
\label{algo}
\begin{tabbing}
\Req $f_\theta(x)$: neural network with trainable parameters $\theta$ \\
\Req $f_{\theta'}(x)$ mean teacher with $\theta'$ equal to moving average of $\theta$\\
\Req $\mathcal{D}_L(x,y)$: collection of the labeled samples \hspace*{25mm} \= \\
\Req $\mathcal{D}_{UL}(x)$: collection of the unlabeled samples \\
\Req $\alpha$: rate of moving average \\
\Req $w(t)$: ramp function for increasing the importance of consistency regularization\\
\Req $T$: total number of iterations \\
\Req ${Q}$: random distribution on [0,1] \\
\Req $\mix{a}{b} = \lambda a + (1-\lambda)b$. \\
\X {\bf for} $t= 1, \ldots, T$ {\bf do} \\
\XX Sample $\{(x_i,y_i)\}_{i=1}^{B}\sim \mathcal{D}_{L}(x,y)$ \quad \cm{Sample labeled minibatch} \\
\XX ${L}_{S} = \text{CrossEntropy}(\{(f_{\theta}({x_i}),{y_i})\}_{i=1}^{B})$ \quad \cm{Supervised loss (cross-entropy)} \\
\XX Sample $\{u_j\}_{j=1}^{U}, \{u_k\}_{k=1}^U \sim \mathcal{D}_{UL}(x)$ \quad \cm{Sample two unlabeled examples} \\
\XX $\{\hat{y}_j\}_{j=1}^{U}= \{f_{\theta'}(u_j)\}_{j=1}^{U}$, $\{\hat{y}_k\}_{k=1}^{U}= \{f_{\theta'}(u_k)\}_{k=1}^{U}$ \quad \cm{Compute fake labels} \\
\XX Sample $\lambda \sim {Q}$ \quad \cm{sample an interpolation coefficient} \\
\XX $(u_{m} = \mix{u_j}{u_k}, \hat{y}_{m} = \mix{\hat{y}_j}{\hat{y}_k})$\quad \cm{Compute interpolation} \\
\XX ${L}_{US} = \text{ConsistencyLoss}(\{(f_{\theta}({u_{m}}) , \hat{y}_{m})\}_{m=1}^{U})$ \quad \cm{e.g., mean squared error} \\
\XX ${L} = {L}_{S} + w(t) \cdot {L}_{US}$ \quad \cm{Total Loss}\\
\XX $ g_\theta \gets \nabla_{\theta} L$ \quad \cm {Compute Gradients} \\
\XX $\theta' = \alpha \theta' + (1-\alpha) \theta$ \quad \cm {Update moving average of parameters} \\
\XX $\theta \gets \text{Step}(\theta, g_\theta)$ \quad \cm{e.g. SGD, Adam}\\
\X {\bf end for}\\
\X {\bf return $\theta$}
\end{tabbing}
\vspace*{-1.5mm}
\end{algorithm*}
\section{Experiments}
\label{sec:sslexp}
\subsection {Datasets}
We follow the common practice in semi-supervised learning literature \citep{laine2016temporal,miyato2017vat,meanteacher,vaad,smooth} and conduct experiments using the CIFAR-10 and SVHN datasets, where only a fraction of the training data is labeled, and the remaining data is used as unlabeled data. We followed the standardized procedures laid out by \citep{oliver2018ssl} to ensure a fair comparison.
The CIFAR-10 dataset consists of 60000 color images each of size $32\times32$, split between 50K training and 10K test images. This dataset has ten classes, which include images of natural objects such as cars, horses, airplanes and deer. The SVHN dataset consists of 73257 training samples and 26032 test samples each of size $32\times32$. Each example is a close-up image of a house number (the ten classes are the digits from 0-9).
We adopt the standard data-augmentation and pre-processing scheme which has become standard practice in the semi-supervised learning literature \citep{sajjadi,laine2016temporal,meanteacher,miyato2017vat,smooth,benathi}. More specifically, for CIFAR-10, we first zero-pad each image with 2 pixels on each side. Then, the resulting image is randomly cropped to produce a new $32\times32$ image. Next, the image is horizontally flipped with probability 0.5, followed by per-channel standardization and ZCA preprocessing. For SVHN, we zero-pad each image with 2 pixels on each side and then randomly crop the resulting image to produce a new $32\times32$ image, followed by zero-mean and unit-variance image whitening.
\subsection {Models}
We conduct our experiments using CNN-13 and Wide-Resnet-28-2 architectures. The CNN-13 architecture has been adopted as the standard benchmark architecture in recent state-of-the-art SSL methods \citep{laine2016temporal,meanteacher,miyato2017vat,vaad,smooth}. We use its variant (i.e., without additive Gaussian noise in the input layer) as implemented in \citep{benathi}. We also removed the Dropout noise to isolate the improvement achieved through our method. Other SSL methods in Table ~\ref{tb:cifar_cnn} and Table ~\ref{tb:svhn_cnn} use the Dropout noise, which gives them more regularizing capabilities. Despite this, our method outperforms other methods in several experimental settings.
\citep{oliver2018ssl} performed a systematic study using Wide-Resnet-28-2 \citep{zagoruyko2016wrn}, a specific residual network architecture, with extensive hyperparameter search to compare the performance of various consistency-based semi-supervised algorithms. We evaluate \ICT{} using this same setup as a mean towards a fair comparison to these algorithms.
\subsection {Implementation details}
\label{sec:implem}
We used the SGD with nesterov momentum optimizer for all of our experiments. For the experiments in Table~\ref{tb:cifar_cnn} and Table~\ref{tb:svhn_cnn}, we run the experiments for 400 epochs. For the experiments in Table~\ref{tb:wrn}, we run experiments for 600 epochs. The initial learning rate was set to 0.1, which is then annealed using the cosine annealing technique proposed in \citep{sgdr} and used by \citep{meanteacher}. The momentum parameter was set to 0.9. We used an L2 regularization coefficient 0.0001 and a batch-size of 100 in our experiments.
In each experiment, we report mean and standard deviation across three independently run trials.
The consistency coefficient $w(t)$ is ramped up from its initial value 0.0 to its maximum value at one-fourth of the total number of epochs using the same sigmoid schedule of \citep{meanteacher}. We used MSE loss for computing the consistency loss following \citep{laine2016temporal,meanteacher}. We set the decay coefficient for the mean-teacher to 0.999 following \citep{meanteacher}.
We conduct hyperparameter search over the two hyperparameters introduced by our method: the maximum value of the consistency coefficient $w(t)$ (we searched over the values in \{1.0, 10.0, 20.0, 50.0, 100.0\}) and the parameter $\alpha$ of distribution $Beta(\alpha, \alpha)$ (we searched over the values in \{0.1, 0.2, 0.5, 1.0\}). We select the best hyperparameter using a validation set of 5000 and 1000 labeled samples for CIFAR-10 and SVHN respectively. This size of the validation set is the same as that used in the other methods compared in this work.
We note the in all our experiments with \ICT{}, to get the supervised loss, we perform the interpolation of labeled sample pair and their corresponding labels (as in \textit{mixup} \citep{mixup}). To make sure, that the improvements from \ICT{} are not only because of the supervised \textit{mixup} loss, we provide the direct comparison of \ICT{} against supervised \textit{mixup} and \textit{Manifold Mixup} training in the Table~\ref{tb:cifar_cnn} and Table~\ref{tb:svhn_cnn}.
\subsection{Results}
\label{sec:results}
\begin{table*}[t]
\caption{Error rates (\%) on CIFAR-10 using CNN-13 architecture. We ran three trials for \ICT{}.}
\label{tb:cifar_cnn}
\centering
\begin{tabular}{lrrr}
\toprule
Model &\makecell{1000 labeled\\ 50000 unlabeled} &\makecell{2000 labeled \\ 50000 unlabeled} & \makecell{4000 labeled \\ 50000 unlabeled }\\
\midrule
Supervised & $39.95 \pm 0.75$ & $31.16 \pm 0.66$ & $21.75 \pm 0.46$\\
Supervised (Mixup) & $36.48\pm0.15$ & $26.24\pm0.46$ & $19.67\pm0.16$ \\
Supervised (Manifold Mixup) & $34.58\pm 0.37$ & $25.12\pm 0.52$ & $18.59\pm 0.18$ \\
\midrule
$\Pi$ model~\citep{laine2016temporal}&$31.65\pm1.20$ &$17.57\pm0.44$&$12.36\pm0.31$ \\
TempEns~\citep{laine2016temporal} &$23.31\pm1.01$ &$15.64\pm0.39$&$12.16\pm0.24$ \\
MT~\citep{meanteacher} & $21.55\pm1.48$ & $15.73\pm0.31$ & $12.31\pm0.28$ \\
VAT~\citep{miyato2017vat} &-- & -- & $11.36\pm \,\,\text{NA}$ \\
VAT+Ent~\citep{miyato2017vat} &-- & -- & $10.55 \pm\,\,\text{NA}$ \\
% \rowcolor
VAdD~\citep{vaad} & -- & -- & $11.32\pm 0.11$ \\
SNTG~\citep{smooth} & $18.41\pm0.52$ & $13.64\pm0.32$ & $10.93\pm0.14$ \\
MT+ Fast SWA ~\citep{benathi} & $15.58 \pm \,\,\text{NA}$ & $11.02 \pm\,\,\text{NA}$ & $9.05 \pm\,\,\text{NA}$ \\
\midrule
\ICT{} & $\bf{15.48 \pm 0.78}$ & $\bf{9.26 \pm0.09}$& $\bf{7.29 \pm 0.02}$ \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table*}[t]
\caption{Error rates (\%) on SVHN using CNN-13 architecture. We ran three trials for \ICT{}.}
\label{tb:svhn_cnn}
\centering
\begin{tabular}{lrrr}
\toprule
Model & \makecell{250 labeled\\ 73257 unlabeled} & \makecell{500 labeled\\73257 unlabeled} & \makecell{1000 labeled \\ 73257 unlabeled} \\
\midrule
Supervised & $40.62\pm0.95$ & $22.93\pm0.67$ & $15.54\pm0.61$ \\
Supervised (Mixup) & $33.73\pm1.79$ & $21.08\pm0.61$ & $13.70\pm0.47$ \\
Supervised ( Manifold Mixup) & $31.75\pm1.39$ & $20.57\pm0.63$ & $13.07\pm0.53$\\
\midrule
$\Pi$ model~\citep{laine2016temporal}& $9.93\pm1.15$& $6.65\pm0.53$ & $4.82\pm0.17$\\
TempEns~\citep{laine2016temporal} &$12.62\pm2.91$ & $5.12\pm0.13$ & $4.42\pm0.16$ \\
MT~\citep{meanteacher} &$4.35\pm 0.50$ & $4.18\pm0.27$ & $3.95\pm0.19$ \\
VAT~\citep{miyato2017vat} & -- & -- & $5.42\pm \,\,\text{NA}$ \\
VAT+Ent~\citep{miyato2017vat} & -- & -- & $3.86\pm \,\,\text{NA}$ \\
VAdD~\citep{vaad} & -- & -- & $4.16\pm0.08$ \\
SNTG~\citep{smooth} & $\bf{4.29\pm0.23}$ & $\bf{3.99\pm0.24}$ & $\bf{3.86\pm0.27}$ \\
\midrule
\ICT{} & $4.78\pm0.68$ & $4.23\pm0.15$ & $3.89\pm0.04$ \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table}[!]
%\begin{wraptable}{R}{10.0cm}
%\centering
\caption{Results on CIFAR10 (4000 labels) and SVHN (1000 labels) (in test error \%). All results use the same standardized architecture (WideResNet-28-2). Each experiment was run for three trials. $\dagger$ refers to the results reported in \citep{oliver2018ssl}.
We did not conduct any hyperparameter search and used the best hyperparameters found in the experiments of Table ~\ref{tb:cifar_cnn} and ~\ref{tb:svhn_cnn} for CIFAR10(4000 labels) and SVHN(1000 labels)}
\label{tb:wrn}
\centering
\begin{tabular}{lrr} %\hline
\toprule
SSL Approach & \makecell{CIFAR10 \\ 4000 labeled \\ 50000 unlabeled} & \makecell{SVHN \\ 1000 labeled \\ 73257 unlabeled} \\ %\hline
\midrule
Supervised $\dagger$ & $20.26 \pm 0.38$ & $12.83 \pm 0.47$ \\
Mean-Teacher $\dagger$ & $15.87 \pm 0.28$ & $5.65 \pm 0.47$ \\
VAT $\dagger$ & $13.86 \pm 0.27$ & $5.63 \pm 0.20$ \\ %\hline
VAT-EM $\dagger$ & $13.13 \pm 0.39$ & $5.35 \pm 0.19$ \\ %\hline
\midrule
\ICT{} & $\bf{7.66 \pm 0.17} $ & $\bf{3.53 \pm 0.07} $ \\ %\hline
\bottomrule
\end{tabular}
\end{table}
We provide the results for CIFAR10 and SVHN datasets using CNN-13 architecture in the Table~\ref{tb:cifar_cnn} and Table~\ref{tb:svhn_cnn}, respectively.
To justify the use of a SSL algorithm, one must compare its performance against the state-of-the-art supervised learning algorithm \citep{oliver2018ssl}. To this end, we compare our method against two state-of-the-art supervised learning algorithms \citep{mixup,verma2018manifold}, denoted as Supervised(Mixup) and Supervised(Manifold Mixup), respectively in Table~\ref{tb:cifar_cnn} and ~\ref{tb:svhn_cnn}. \ICT{} method passes this test with a wide margin, often resulting in a two-fold reduction in the test error in the case of CIFAR10 (Table~\ref{tb:cifar_cnn}) and a four-fold reduction in the case of SVHN (Table~\ref{tb:svhn_cnn})
Furthermore, in Table~\ref{tb:cifar_cnn}, we see that \ICT{} improves the test error of other strong SSL methods. For example, in the case of 4000 labeled samples, it improves the test error of best-reported method by $\sim 25\%$.
%In the case of lesser labeled sample experiments (1000 and 2000 labeled samples) the improvement achieved by \ICT{} is even more profound.
The best values of the hyperparameter max-consistency coefficient for 1000, 2000 and 4000 labels experiments were found to be 10.0, 100.0 and 100.0 respectively and the best values of the hyperparameter $\alpha$ for 1000, 2000 and 4000 labels experiments were found to be 0.2, 1.0 and 1.0 respectively. In general, we observed that for less number of labeled data, lower values of max-consistency coefficient and $\alpha$ obtained better validation errors.
For SVHN, the test errors obtained by \ICT{} are competitive with other state-of-the-art SSL methods (Table~\ref{tb:svhn_cnn}). The best values of the hyperparameters max-consistency coefficient and $\alpha$ were found to be 100 and 0.1 respectively, for all the \ICT{} results reported in the Table~\ref{tb:svhn_cnn}.
\citep{oliver2018ssl} performed extensive hyperparameter search for various consistency regularization SSL algorithm using the WRN-28-2 and they report the best test errors found for each of these algorithms. For a fair comparison of \ICT{} against these SSL algorithms, we conduct experiments on WRN-28-2 architecture. The results are shown in Table~\ref{tb:wrn}. \ICT{} achieves improvement over other methods both for the CIFAR10 and SVHN datasets.
We note that unlike other SSL methods of Table~\ref{tb:cifar_cnn}, Table~\ref{tb:svhn_cnn} and Table~\ref{tb:wrn}, we do not use Dropout regularizer in our implementation of CNN-13 and WRN-28-2. Using Dropout along with the \ICT{} may further reduce the test error.
\subsection{Ablation Study}
\begin{itemize}
\item Effect of not using the mean-teacher in \ICT{}: We note that $\Pi$-model, VAT and VAdD methods in Table~\ref{tb:cifar_cnn} and Table~\ref{tb:svhn_cnn} do not use a mean-teacher to make predictions on the unlabeled data. Although the mean-teacher \citep{meanteacher} used in \ICT{} does not incur any significant computation cost, one might argue that a more direct comparison with $\Pi$-model, VAT and VAdD methods requires not using a mean-teacher. To this end, we conduct an experiment on the CIFAR10 dataset, without the mean-teacher in \ICT, i.e. the prediction on the unlabeled data comes from the network $f_{\theta}(x)$ instead of the mean-teacher network $f_{\theta'}(x)$ in Equation ~\ref{eq: consistency loss}. We obtain test errors of $19.56\pm 0.56\%$, $14.35\pm 0.15\%$ and $11.19\pm0.14\%$ for 1000, 2000, 4000 labeled samples respectively (We did not conduct any hyperparameter search for these experiments and used the best hyperparameters found in the \ICT{} experiments of Table ~\ref{tb:cifar_cnn}). This shows that even without a mean-teacher, \ICT{} has major a advantage over methods such as VAT \citep{miyato2017vat} and VAdD \citep{vaad} that it does not require an additional gradient computation yet performs on the same level of the test error.
\item Effect of not having the \textit{mixup} supervised loss: In Section ~\ref{sec:implem}, we noted that to get the supervised loss, we perform the interpolation of labeled sample pair and their corresponding labels (\textit{mixup} supervised loss as in \citep{mixup}). Will the performance of \ICT{} be significantly reduced by not having the \textit{mixup} supervised loss? We conducted experiments with \ICT{} on both CIFAR10 and SVHN with the \textit{vanilla} supervised loss. For CIFAR10, we obtained test errors of $14.86\pm 0.39$, $9.02\pm 0.12$ and $8.23\pm 0.22$ for 1000, 2000 and 4000 labeled samples respectively. We did not conduct any hyperparameter search and used the best values of hyperparameters (max-consistency coefficient and $\alpha$) found in the experiments of the Table ~\ref{tb:cifar_cnn}. We observe that in the case of 1000 and 2000 labeled samples, there is no increase in the test error (w.r.t having the \textit{mixup} supervised loss), whereas in the case of 4000 labels, the test error increases by approximately $1\%$ . This suggests that, in the low labeled data regimes, not having the \textit{mixup} supervised loss in the \ICT{} does not incur any significant increase in the test error.
%\item Sensitivity to max-consis coeff and alpha
\end{itemize}
\section{Related Work}
This work builds on two threads of research: consistency-regularization for semi-supervised learning and interpolation-based regularizers.
On the one hand, consistency-regularization semi-supervised learning methods \citep{sajjadi,laine2016temporal,meanteacher,miyato2017vat,smooth,benathi} encourage that realistic perturbations $u+\delta$ of unlabeled samples $u$ should not change the model predictions $f_{\theta}(u)$.
%
These methods are motivated by the \textit{low-density separation assumption} \citep{chapple}, and as such push the decision boundary to lie in the low-density regions of the input space, achieving larger classification margins.
%
\ICT{} differs from these approaches in two aspects. First, \ICT{} chooses perturbations in the direction of another randomly chosen unlabeled sample, avoiding expensive gradient computations.
%
When interpolating between distant points, the regularization effect of \ICT{} applies to larger regions of the input space.
On the other hand, interpolation-based regularizers \citep{mixup,betweenclass,verma2018manifold} have been recently proposed for supervised learning, achieving state-of-the-art performances across a variety of tasks and network architectures. While \citep{mixup,betweenclass} was proposed to perform interpolations in the input space, \citep{verma2018manifold} proposed to perform interpolation also in the hidden space representations. Furthermore, in the unsupervised learning setting, \citep{acai} proposes to measure the realism of latent space interpolations from an autoencoder to improve its training.
Other works have approached semi-supervised learning from the perspective of generative models. Some have approached this from a consistency point of view, such as \citep{lecouat2018manifold}, who proposed to encourage smooth changes to the predictions along the data manifold estimated by the generative model (trained on both labeled and unlabeled samples). Others have used the discriminator from a trained generative adversarial network \citep{goodfellow2014gan} as a way of extracting features for a purely supervised model \citep{radford2015unsupervised}. Still, others have used trained inference models as a way of extracting features \citep{dumoulin2016ali}.
%\newpage
\section{Conclusion}
Machine learning is having a transformative impact on diverse areas, yet its application is often limited by the amount of available labeled data.
%
Progress in semi-supervised learning techniques holds promise for those applications where labels are expensive to obtain.
%
In this paper, we have proposed a simple but efficient semi-supervised learning algorithm, \ict (\ICT{}), which has two advantages over previous approaches to semi-supervised learning.
%
First, it uses almost no additional computation, as opposed to computing adversarial perturbations or training generative models.
%
Second, it outperforms strong baselines on two benchmark datasets, even without an extensive hyperparameter tuning.
%
As for the future work, extending \ICT{} to interpolations not only at the input but at hidden representations \citep{verma2018manifold} could improve the performance even further.
%
Another direction for future work is to better understand the theoretical properties of interpolation-based regularizers in the SSL paradigm.
\section*{Acknowledgments}
Vikas Verma was supported by Academy of Finland project 13312683 / Raiko Tapani AT kulut.
We would also like to acknowledge Compute Canada for providing computing resources used in this work.
%\clearpage
\bibliographystyle{icml}
\bibliography{ijcai19}
\end{document}
\begin{thebibliography}{24}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Athiwaratkun et~al.(2019)Athiwaratkun, Finzi, Izmailov, and
Wilson]{benathi}
Athiwaratkun, B., Finzi, M., Izmailov, P., and Wilson, A.~G.
\newblock There are many consistent explanations of unlabeled data: Why you
should average.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\newblock URL \url{https://openreview.net/forum?id=rkgKBhA5Y7}.
\bibitem[Berthelot et~al.(2019)Berthelot, Raffel, Roy, and Goodfellow]{acai}
Berthelot, D., Raffel, C., Roy, A., and Goodfellow, I.
\newblock Understanding and improving interpolation in autoencoders via an
adversarial regularizer.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\newblock URL \url{https://openreview.net/forum?id=S1fQSiCcYm}.
\bibitem[Chapelle et~al.(2010)Chapelle, Schlkopf, and Zien]{chapple}
Chapelle, O., Schlkopf, B., and Zien, A.
\newblock \emph{Semi-Supervised Learning}.
\newblock The MIT Press, 1st edition, 2010.
\newblock ISBN 0262514125, 9780262514125.
\bibitem[Clanuwat et~al.(2018)Clanuwat, Bober-Irizar, Kitamoto, Lamb, Yamamoto,
and Ha]{clanuwat2018deep}
Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., and Ha,
D.
\newblock Deep learning for classical japanese literature.
\newblock \emph{arXiv preprint arXiv:1812.01718}, 2018.
\bibitem[Dumoulin et~al.(2016)Dumoulin, Belghazi, Poole, Mastropietro, Lamb,
Arjovsky, and Courville]{dumoulin2016ali}
Dumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky,
M., and Courville, A.
\newblock Adversarially learned inference.
\newblock \emph{arXiv preprint arXiv:1606.00704}, 2016.
\bibitem[Goodfellow et~al.(2014)Goodfellow, Pouget-Abadie, Mirza, Xu,
Warde-Farley, Ozair, Courville, and Bengio]{goodfellow2014gan}
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair,
S., Courville, A., and Bengio, Y.
\newblock Generative adversarial nets.
\newblock In \emph{Advances in neural information processing systems}, pp.\
2672--2680, 2014.
\bibitem[Laine \& Aila(2016)Laine and Aila]{laine2016temporal}
Laine, S. and Aila, T.
\newblock Temporal ensembling for semi-supervised learning.
\newblock \emph{CoRR}, abs/1610.02242, 2016.
\newblock URL \url{http://arxiv.org/abs/1610.02242}.
\bibitem[Lecouat et~al.(2018)Lecouat, Foo, Zenati, and
Chandrasekhar]{lecouat2018manifold}
Lecouat, B., Foo, C.-S., Zenati, H., and Chandrasekhar, V.
\newblock Manifold regularization with gans for semi-supervised learning.
\newblock \emph{arXiv preprint arXiv:1807.04307}, 2018.
\bibitem[LeCun et~al.(2015)LeCun, Bengio, and Hinton]{lecun2015deep}
LeCun, Y., Bengio, Y., and Hinton, G.
\newblock Deep learning.
\newblock \emph{nature}, 521\penalty0 (7553):\penalty0 436, 2015.
\bibitem[Loshchilov \& Hutter(2016)Loshchilov and Hutter]{sgdr}
Loshchilov, I. and Hutter, F.
\newblock {SGDR:} stochastic gradient descent with restarts.
\newblock \emph{CoRR}, abs/1608.03983, 2016.
\newblock URL \url{http://arxiv.org/abs/1608.03983}.
\bibitem[Luo et~al.(2018)Luo, Zhu, Li, Ren, and Zhang]{smooth}
Luo, Y., Zhu, J., Li, M., Ren, Y., and Zhang, B.
\newblock Smooth neighbors on teacher graphs for semi-supervised learning.
\newblock In \emph{2018 {IEEE} Conference on Computer Vision and Pattern
Recognition, {CVPR} 2018, Salt Lake City, UT, USA, June 18-22, 2018}, pp.\
8896--8905, 2018.
\bibitem[Miyato et~al.(2018)Miyato, ichi Maeda, Koyama, and
Ishii]{miyato2017vat}
Miyato, T., ichi Maeda, S., Koyama, M., and Ishii, S.
\newblock Virtual adversarial training: a regularization method for supervised
and semi-supervised learning.
\newblock \emph{IEEE transactions on pattern analysis and machine
intelligence}, 2018.
\bibitem[Nakkiran(2019)]{nakkiran2019adversarial}
Nakkiran, P.
\newblock Adversarial robustness may be at odds with simplicity.
\newblock \emph{arXiv preprint arXiv:1901.00532}, 2019.
\bibitem[{Oliver} et~al.(2018){Oliver}, {Odena}, {Raffel}, {Cubuk}, and
{Goodfellow}]{oliver2018ssl}
{Oliver}, A., {Odena}, A., {Raffel}, C., {Cubuk}, E.~D., and {Goodfellow},
I.~J.
\newblock {Realistic Evaluation of Deep Semi-Supervised Learning Algorithms}.
\newblock In \emph{Neural Information Processing Systems (NIPS)}, 2018.
\bibitem[Park et~al.(2018)Park, Park, Shin, and Moon]{vaad}
Park, S., Park, J., Shin, S.-J., and Moon, I.-C.
\newblock Adversarial dropout for supervised and semi-supervised learning.
\newblock \emph{AAAI}, 2018.
\bibitem[Radford et~al.(2015)Radford, Metz, and
Chintala]{radford2015unsupervised}
Radford, A., Metz, L., and Chintala, S.
\newblock Unsupervised representation learning with deep convolutional
generative adversarial networks.
\newblock \emph{arXiv preprint arXiv:1511.06434}, 2015.
\bibitem[Sajjadi et~al.(2016)Sajjadi, Javanmardi, and Tasdizen]{sajjadi}
Sajjadi, M., Javanmardi, M., and Tasdizen, T.
\newblock Regularization with stochastic transformations and perturbations for
deep semi-supervised learning.
\newblock In \emph{Proceedings of the 30th International Conference on Neural
Information Processing Systems}, NIPS'16, pp.\ 1171--1179, USA, 2016. Curran
Associates Inc.
\newblock ISBN 978-1-5108-3881-9.
\newblock URL \url{http://dl.acm.org/citation.cfm?id=3157096.3157227}.
\bibitem[Shawe-Taylor et~al.(1996)Shawe-Taylor, Bartlett, C.~Williamson, and
Anthony]{srm}
Shawe-Taylor, J., Bartlett, P., C.~Williamson, R., and Anthony, M.
\newblock A framework for structural risk minimisation.
\newblock pp.\ 68--76, 01 1996.
\newblock \doi{10.1145/238061.238070}.
\bibitem[Tarvainen \& Valpola(2017)Tarvainen and Valpola]{meanteacher}
Tarvainen, A. and Valpola, H.
\newblock Mean teachers are better role models: Weight-averaged consistency
targets improve semi-supervised deep learning results.
\newblock In \emph{Advances in Neural Information Processing Systems 30}, pp.\
1195--1204, 2017.
\bibitem[Tokozume et~al.(2018)Tokozume, Ushiku, and Harada]{betweenclass}
Tokozume, Y., Ushiku, Y., and Harada, T.
\newblock Between-class learning for image classification.
\newblock In \emph{The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, June 2018.
\bibitem[Tsipras et~al.(2018)Tsipras, Santurkar, Engstrom, Turner, and
Madry]{tsipras2018robustness}
Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A.
\newblock Robustness may be at odds with accuracy.
\newblock \emph{stat}, 1050:\penalty0 11, 2018.
\bibitem[{Verma} et~al.(2018){Verma}, {Lamb}, {Beckham}, {Najafi},
{Mitliagkas}, {Courville}, {Lopez-Paz}, and {Bengio}]{verma2018manifold}
{Verma}, V., {Lamb}, A., {Beckham}, C., {Najafi}, A., {Mitliagkas}, I.,
{Courville}, A., {Lopez-Paz}, D., and {Bengio}, Y.
\newblock {Manifold Mixup: Better Representations by Interpolating Hidden
States}.
\newblock \emph{arXiv e-prints}, art. arXiv:1806.05236, Jun 2018.
\bibitem[Zagoruyko \& Komodakis(2016)Zagoruyko and Komodakis]{zagoruyko2016wrn}
Zagoruyko, S. and Komodakis, N.
\newblock Wide residual networks.
\newblock In Richard C.~Wilson, E. R.~H. and Smith, W. A.~P. (eds.),
\emph{Proceedings of the British Machine Vision Conference (BMVC)}, pp.\
87.1--87.12. BMVA Press, September 2016.
\newblock ISBN 1-901725-59-6.
\newblock \doi{10.5244/C.30.87}.
\newblock URL \url{https://dx.doi.org/10.5244/C.30.87}.
\bibitem[Zhang et~al.(2018)Zhang, Cisse, Dauphin, and Lopez-Paz]{mixup}
Zhang, H., Cisse, M., Dauphin, Y.~N., and Lopez-Paz, D.
\newblock mixup: Beyond empirical risk minimization.
\newblock In \emph{International Conference on Learning Representations}, 2018.
\newblock URL \url{https://openreview.net/forum?id=r1Ddp1-Rb}.
\end{thebibliography}