Gradient penalty from a maximum margin perspective
Extracted source text · arXiv 1910.06922
\documentclass[twoside]{article}
\usepackage{aistats2020}
% If your paper is accepted, change the options for the package
% aistats2020 as follows:
%
%\usepackage[accepted]{aistats2020}
%
% This option will print headings for the title of your paper and
% headings for the authors names, plus a copyright note at the end of
% the first column of the first page.
% If you set papersize explicitly, activate the following three lines:
%\special{papersize = 8.5in, 11in}
%\setlength{\pdfpageheight}{11in}
%\setlength{\pdfpagewidth}{8.5in}
% If you use natbib package, activate the following three lines:
\usepackage[round]{natbib}
\renewcommand{\bibname}{References}
\renewcommand{\bibsection}{\subsubsection*{\bibname}}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{lmodern}
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
% Not in NIPS2018
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{algpseudocode,algorithm,algorithmicx}
\usepackage{graphicx}
\graphicspath{ {images/} }
\usepackage[usenames, dvipsnames]{color}
\def\UrlBreaks{\do\/\do-}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\usepackage{mathtools}
\definecolor{dark-gray}{gray}{.35}
\definecolor{myorange}{RGB}{246, 164, 16}
\definecolor{mygreen}{RGB}{1, 100, 3}
\usepackage{footnote}
\makesavenoteenv{table}
\makesavenoteenv{tabular}
\usepackage{cleveref}[2012/02/15]% v0.18.4;
% 0.16.1 of May 2010 would be sufficient, but what is the exact day?
\crefformat{footnote}{#2\footnotemark[#1]#3}
% !TeX spellcheck = en_US
%\title{Suport Vector Machines and Wasserstein's distance both arise from maximizing a geometric margin}
% Suport Vector Machines and Wasserstein's distance arise from maximizing a geometric margin
% Generalizing Support Vector Machines leads to Wasserstein's and gradient penalty GANs
\begin{document}
\twocolumn[
\aistatstitle{Connections between Support Vector Machines, Wasserstein distance and gradient-penalty GANs}
\aistatsauthor{ Alexia Jolicoeur-Martineau \And Ioannis Mitliagkas}
\aistatsaddress{ Mila, University of Montreal \And Mila, University of Montreal} ]
\begin{abstract}
We generalize the concept of maximum-margin classifiers (MMCs) to arbitrary norms and non-linear functions.
Support Vector Machines (SVMs) are a special case of MMC.
We find that MMCs can be formulated as Integral Probability Metrics (IPMs) or classifiers with some form of gradient norm penalty.
This implies a direct link to a class of Generative adversarial networks (GANs) which penalize a gradient norm.
We show that the Discriminator in Wasserstein, Standard, Least-Squares, and Hinge GAN with Gradient Penalty is an MMC.
We explain why maximizing a margin may be helpful in GANs.
We hypothesize and confirm experimentally that $L^\infty$-norm penalties with Hinge loss produce better GANs than $L^2$-norm penalties (based on common evaluation metrics). We derive the margins of Relativistic paired (Rp) and average (Ra) GANs.%given that they correspond to maximizing a $L^1$-norm margin rather than a $L^2$-norm margin.
\end{abstract}
\section{Introduction}
Support Vector Machines (SVMs) \cite{cortes1995support} are a very popular type of maximum-margin classifier (MMC). The margin can be conceptualized as the minimum $L^p$ distance between the decision boundary of the classifier and any data-point. An SVM is a linear classifier which maximizes the minimum $L^2$ margin. A significant body of work has been done on generalizing SVM beyond a simple linear classifier through the kernel trick \cite{aizerman1964theoretical}. However, until very recently, SVMs had not been generalized to arbitrary norms with non-linear classifiers (e.g., neural networks). In this paper, we describe how to train MMCs (which generalize SVMs) through different approximations of the $L^p$-norm margin and we show that this results in loss functions with a gradient norm penalty.
Generative adversarial networks (GANs) \citep{GAN} are a very successful class of generative models.
Their most common formulation involves a game played between two competing neural networks, the discriminator $D$ and the generator $G$.
$D$ is a classifier trained to distinguish real from fake examples, while $G$ is trained to generate fake examples that will confuse $D$ into recognizing them as real.
When the discriminator's objective is maximized, it yields the value of a specific divergence (i.e., a distance between probability distributions) between the distributions of real and fake examples. The generator then aims to minimize that divergence (although this interpretation is not perfect; see \citet{jolicoeur2018beyonddivergence}).
Importantly, many GANs apply some form of gradient norm penalty to the discriminator \citep{WGAN-GP,fedus2017many,mescheder2018training,karras2019style}.
This penalty is motivated by a Wasserstein distance formulation in \citet{WGAN-GP},
or by numerical stability arguments \cite{mescheder2018training,karras2019style}.
In this paper, we show that discriminator loss functions that use a gradient penalty correspond to specific types of MMCs. Our contributions are the following:
\begin{enumerate}
\item We define the concept of expected margin maximization and show that Wasserstein-, Standard-, Least-Squares-, and Hinge-GANs can be derived from this framework.
\item We derive a new method from this framework, a GAN that penalize $L^\infty$ gradient norm values above 1 (instead of penalizing all values unequal to 1 as done by \citet{WGAN-GP}).
We hypothesize and experimentally show that this method leads to better generated outputs.
\item We describe how margin maximization (and thereby gradient penalties) help reduce vanishing gradients at fake (generated) samples, a known problem in many GANs.
\item We derive the margins of Relativistic paired and average GANs \citep{jolicoeur2018relativistic}.
\end{enumerate}
It is worth noting that \cite{lim2017geometric} explore a similar connection between GANs and SVMs, which they use to propose Geometric GANs.
The main difference to our work is that they assume a linear classifier working on the {\em feature space of a neural network's output} instead of the {\em input space}.
Furthermore, that work does not exploit the duality theory of SVMs. Thereby, it does not draw a connection to gradient penalty terms. Our work explores this new connection which motivates an $L^\infty$ norm gradient penalty and shows great promise over the standard $L^2$ norm gradient penalty.
The paper is organized as follows. In Section~\ref{sec:2}, we review SVMs and GANs. In Section~\ref{sec:3}, we generalize the concept of maximum-margin classifiers (MMCs). In Section~4, we explain the connections between MMCs and GANs with gradient penalty. In Section~\ref{sec:4.1}, we mention that enforcing 1-Lipschitz is equivalent to assuming a bounded gradient; this implies that Wasserstein's distance can be approximated with an MMC formulation. In Section~\ref{sec:4.2}, we describe the benefits of using MMCs in GANs. In Section~\ref{sec:4.3}, we hypothesize that $L^1$-norm margins may lead to more robust classifiers. In Section~\ref{sec:4.4}, we derive margins for Relativistic paired and average GANs. Finally, in Section~\ref{sec:5}, we provide experiments to support the hypotheses in our contributions.
%Before attempting to generalize SVMs, we start with a small review of SVMs and GANs.
%These results show that classifiers with gradient norm penalty are maximum-margin classifiers. This type of classifier is very popular in Generative Adversarial Networks (GANs). This may explain the increase in performance from these methods in GANs. Furthermore, this provides a better explanation to how WGAN-GP works. Although WGAN-GP is defined as approximately estimating Wasserstein's distance, this not actually correct. Our results simply show that WGAN-GP maximizes the margin with $L^1$-norm, while the actual Wasserstein's distance maximizes the margin with $L^2$-norm.
\section{Review of SVMs and GANs}
\label{sec:2}
\subsection{Notation}
\label{sec:notation}
We focus on binary classifiers. Let $f$ be the classifier and $(x,y) \sim \mathbb{D}$ the distribution (of a dataset $D$) with $n$ data samples $x$ and labels $y$. As per SVM literature, $y=1$ when $x$ is sampled from class 1 and $y=-1$ when $x$ is sampled from class 2. Furthermore, we denote $x_1 = x | (y = 1) \sim \mathbb{P}$ and $x_2 = x | (y = -1) \sim \mathbb{Q}$ as the data samples from class 1 and class 2 respectively (with distributions $\mathbb{P}$ and $\mathbb{Q}$). When discussing GANs, $x_1 \sim \mathbb{P}$ (class 1) refer to real data samples and $x_2 \sim \mathbb{Q}$ (class 2) refer to fake data samples (produced by the generator). The $L^{\infty}$-norm is defined as: $|| x ||_\infty = \max(|x_1|,|x_2|,\ldots,|x_k|)$. Note that we sometimes refer to a function $F$; this is an objective function to be maximized (not to be confused with the classifier $f$).
The critic (C) is the discriminator (D) before applying any activation function (i.e., $D(x)=a(C(x))$, where $a$ is the activation function).
For consistency with existing literature, we will generally refer to the critic rather than the discriminator.
\subsection{SVMs}
In this section, we explain how to obtain a linear maximum-margin classifier (MMC) which maximizes the minimum $L^2$-norm margin (i.e., SVMs).
\subsubsection{Decision boundary and margin}
The \em decision boundary \em of a classifier is defined as the set of points $x_0$ such that $f(x_0)=0$.
The margin is either defined as
i) the minimum distance between a sample and the boundary, or
ii) the minimum distance between the \em closest sample \em to the boundary and the boundary.
The former thus corresponds to the \em margin of a sample \em and the latter corresponds to the \em margin of a dataset \em.
In order to disambiguate the two cases, we refer to the former as the \em margin \em and the latter as the \em minimum margin\em.
The first step towards obtaining a linear MMC is to define the $L^p$-norm margin:
\begin{align}\label{eq:10}
\gamma(x) = &\min_{x_0} || x_0 - x ||_p \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x_0)=0
\end{align}
With a linear classifier (i.e., $f(x) = w^T x - b$) and $p=2$, we have: $$\gamma(x) = \frac{|w^T x - b|}{||w||_2} = \frac{\alpha(x)}{\beta}$$
Our goal is to maximize this margin, but we also want to obtain a classifier. To do so, we simply replace $\alpha(x)=|w^T x - b|$ by $\widetilde{\alpha}(x,y) = y(w^T x - b)$. We call $\widetilde{\alpha}$ the \em functional margin\em. After replacement, we obtain the \em geometric margin\em:
$$\widetilde{\gamma}(x,y) = \frac{y(w^T x - b)}{||w||_2} = \frac{\widetilde{\alpha}(x,y)}{\beta}$$
The specific goal of SVMs is to find a linear classifier which maximizes the \em minimum \em $L^2$-norm geometric margin (in each class):
\begin{align}\label{eqn:1}
\max_{w,b} \min_{(x,y) \in D} \widetilde{\gamma}(x,Y).
\end{align}
\subsubsection{Formulations}
Directly solving equation \eqref{eqn:1} is an ill-posed problem for multiple reasons. Firstly, the numerator and denominator are dependent on one another; increasing the functional margin also increases the norm of the weights (and vice-versa). Thereby, there are infinite solutions which maximize the geometric margin. Secondly, maximizing $\widetilde{\alpha}$ means minimizing the denominator which can cause numerical issues (division by near zero). Thirdly, it makes for a very difficult optimization given the max-min formulation.
For these reasons, we generally prefer to
i) constrain the numerator and minimize the denominator, or
ii) constrain the denominator and maximize the numerator.
The classical approach is to minimize the denominator and constrain the numerator using the following formulation:
\begin{align}\label{eqn:2}
\min_{w,b} {||w||^2_2} \hspace{5pt} \text{ s.t. } \hspace{5pt} y(w^T x - b) \ge 1 \hspace{2pt}\forall\hspace{2pt} (x,y) \in D
\end{align}
This formulation corresponds to Hard-Margin SVM. The main limitation of this approach is that it only works when the data are separable. However, if we take the opposite approach of maximizing a function of $y(w^T x - b)$ and constraining the denominator $|| w ||_2$, we can still solve the problem with non-separable data.
For this reason, we prefer solving of the following Soft-Margin SVM:
\begin{align*}
\min_{w,b} \frac{1}{n} \sum_{(x,y)\in D}\left[\max(0,1-y(w^T x - b))\right] \hspace{1pt} \text{ s.t. } \hspace{1pt} ||w||_2 = 1
\end{align*}
This can be rewritten equivalently with a KKT multiplier $\lambda$ in the following way:
\begin{align*}
\min_{w,b} \frac{1}{n} \sum_{(x,y)\in D}\left[\max(0,1-y(w^T x - b))\right] + \lambda (||w||^2_2 - 1),
\end{align*}
Note that the Hinge function $max(0,1-y(w^T x - b)$ is simply a relaxation of the hard constraint $y(w^T x - b) \ge 1 \hspace{2pt}\forall\hspace{2pt} (x,y) \in D$. Thereby, we are not actually solving equation \eqref{eqn:1} anymore.
\subsection{GANs}\label{sec:GAN}
GANs can be formulated in the following way:
\begin{align}\label{eqn:3}
&\max_{C: \mathcal{X} \to \mathbb{R}} \mathbb{E}_{x_1 \sim \mathbb{P}}\left[ f_1(C(x_1)) \right] + \mathbb{E}_{z \sim \mathbb{Z}} \left[ f_2(C(G(z))) \right], \\
&\min_{G: Z \to \mathcal{X}} \mathbb{E}_{z \sim \mathbb{Z}} \left[ f_3(C(G(z))) \right],
\end{align}\label{eqn:4}
where $f_1, f_2, f_3:\mathbb{R} \to \mathbb{R}$, $\mathbb{P}$ is the distribution of real data with support $\mathcal{X}$, $\mathbb{Z}$ is a multivariate normal distribution with support $Z=\mathbb{R}$, $C(x)$ is the critic evaluated at $x$, $G(z)$ is the generator evaluated at $z$, and $G(z) \sim \mathbb{Q}$, where $\mathbb{Q}$ is the distribution of fake data.
Many variants exist; to name a few: Standard GAN (SGAN) \citep{GAN} corresponds to $f_1(z)=\log(\text{sigmoid}(z))$, $f_2(z)=\log(\text{sigmoid}(-z))$, and $f_3(z)=-f_1(z)$. Least-Squares GAN (LSGAN) \citep{LSGAN} corresponds to $f_1(z)=-(1-z)^2$, $f_2(z)=-(1+z)^2$, and $f_3(z)=-f_1(z)$. HingeGAN \citep{lim2017geometric} corresponds to $f_1(z)=-max(0,1-z)$, $f_2(z)=-max(0,1+z)$, and $f_3(z)=-z$.
An important class of GANs are those based on Integral probability metrics (IPMs) \citep{muller1997integral}. IPMs are statistical divergences (distances between probability distributions) defined in the following way:
\[
IPM_{F} (\mathbb{P} || \mathbb{Q}) = \sup_{C \in \mathcal{F}} \mathbb{E}_{x_1 \sim \mathbb{P}}[C(x_1)] - \mathbb{E}_{x_2 \sim \mathbb{Q}}[C(x_2)],
\]
where $\mathcal{F}$ is a class of real-valued functions. Of note, certain connections between IPMs and SVMs have been identified in \citet{sriperumbudur2009integral}.
IPM-based GANs attempt to solve the following problem $$ \min_{G} \max_{C \in \mathcal{F}} \mathbb{E}_{x_2 \sim \mathbb{P}}[C(x_1)] - \mathbb{E}_{z \sim \mathbb{Z}}[C(G(z))].$$
There are many GANs based on IPMs \citep{mroueh2017sobolev,Fisher}, but we will focus on two of them: WGAN \citep{WGAN} and WGAN-GP \citep{WGAN-GP}.
WGAN is an IPM-based GAN which uses the first-order Wasserstein's distance ($W_1$), the IPM restricted to the class of all 1-Lipschitz functions. This corresponds to the set of functions $C$ such that $\frac{C(x_1)-C(x_2)}{d(x_1,x_2)} \leq 1$ for all $x_1$,$x_2$, where $d(x_1,x_2)$ is a metric. $W_1$ also has a primal form which can be written in the following way:
$$ W_1 (\mathbb{P}, \mathbb{Q}):= \inf_{\pi \in \Pi (\mathbb{P}, \mathbb{Q})} \int_{M \times M} d(x_1, x_2) \, \mathrm{d} \pi (x_1, x_2),$$
where $\Pi (\mathbb{P}, \mathbb{Q})$ is the set of all distributions with marginals $\mathbb{P}$ and $\mathbb{Q}$ and we call $\pi$ a coupling.
The original way to enforce the 1-Lipschitz property on the critic was to clamp its weights after each update. This was later shown to be problematic \citep{WGAN-GP}. Albeit with its issues, WGAN improves the stability of GANs and reduces the incidence of mode collapse (when the generator produces less diversity than the training dataset) \citep{WGAN}.
\citet{WGAN-GP} showed that if the optimal critic $f^{*}(x)$ is differentiable everywhere and $\hat{x} = \alpha x_1 + (1-\alpha)x_2$ for $0 \leq \alpha \leq 1$, we have that $||\nabla C^{*}(\hat{x})|| = 1$ almost everywhere for all pair $(x_1,x_2)$ which comes from the optimal coupling $\pi^{*}$. Sampling from the optimal coupling is difficult so they suggest to softly penalize $\mathbb{E}_{\tilde{x}}{(||\nabla_{\tilde{x}} C(\tilde{x})||_2-1)^2}$, where $\tilde{x} = \alpha x + (1-\alpha)y$, $\alpha \sim U(0,1)$, $x \sim \mathbb{P}$, and $y \sim \mathbb{Q}$. This penalty works well in practice and is a popular way to approximate Wasserstein's distance. However, this is not equivalent to estimating Wasserstein's distance since we are not sampling from $\pi^{*}$ and $f^{*}$ does not need to be differentiable everywhere \citep{petzka2017regularization}. %Thus, calling this approach WGAN-GP is a misnomer; although it is derived based on properties of Wasserstein's distance, it is only a lower bound of $W_1$ (since it does not contain all Lipschitz functions).
Of importance, gradient norm penalties of the form $\mathbb{E}_{x}{(||\nabla_{x} D(x)||_2-\delta)^2}$, for some $\delta \in \mathbb{R}$ are very popular in GANs. Remember that $D(x)=a(C(x))$; in the case of IPM-based-GANs, we have that $D(x)=C(x)$. It has been shown that the GP-1 penalty ($\delta=1$), as in WGAN-GP, also improves the performance of non-IPM-based GANs \citep{ManyPaths}. Another successful variant is GP-0 ($\delta=0$ and $x \sim \mathbb{P}$) \citep{mescheder2018training,karras2019style}. Although there are explanations to why gradient penalties may be helpful \citep{mescheder2018training, kodali2017convergence, WGAN-GP}, the theory is still lacking.
There are other GAN variants which improve the stability of training and will be relevant to our discussion. The first one is HingeGAN \citep{lim2017geometric} which uses the Hinge loss as objective function. This corresponds to using equation \eqref{eqn:3} and \eqref{eqn:4} using $f_1(z)=-max(0,1-z)$, $f_2(z)=-max(0,1+z)$, and $f_3(z)=-z$.
Another class of GANs relevant to our discussion are Relativistic paired GANs (RpGANs) \citep{jolicoeur2018relativistic,jolicoeur2019relativistic}:
\begin{align*}
\max\limits_{C:\mathcal{X} \to \mathbb{R}} \hspace{1pt}
&\underset{ \substack{x_1 \sim \mathbb{P} \\ x_2 \sim \mathbb{Q}}}{\mathbb{E}\vphantom{p}} \left[ f_1 \left( C(x_1) - C(x_2) \right) \right], \\
\min_{G} \hspace{1pt}&
\underset{ \substack{x_1 \sim \mathbb{P} \\ x_2 \sim \mathbb{Q}}}{\mathbb{E}\vphantom{p}} \left[ f_2 \left( C(x_1) - C(x_2) \right) \right],
\end{align*}
and Relativistic average GANs (RaGANs) \citep{jolicoeur2018relativistic,jolicoeur2019relativistic}:
\begin{align*}
\max\limits_{C:\mathcal{X} \to \mathbb{R}} &\mathbb{E}_{x_1 \sim \mathbb{P}}\left[ f_1\left( C(x_1)-\mathbb{E}_{x_2 \sim \mathbb{Q}} \hspace{1pt} C(x_2) \right)) \right] + \\ &\mathbb{E}_{x_2 \sim \mathbb{Q}} \left[ f_2 \left( C(x_2)-\mathbb{E}_{x_1 \sim \mathbb{P}} \hspace{1pt} C(x_1) \right) \right], \\
\max_{G} \hspace{2pt} &\mathbb{E}_{x_2 \sim \mathbb{Q}}\left[ f_1\left( C(x_2)-\mathbb{E}_{x_1 \sim \mathbb{P}} \hspace{1pt} C(x_1) \right)) \right] + \\ &\mathbb{E}_{x_1 \sim \mathbb{P}} \left[ f_2 \left( C(x_1)-\mathbb{E}_{x_2 \sim \mathbb{Q}} \hspace{1pt} C(x_2) \right) \right].
\end{align*}
Most loss functions can be represented as RaGANs or RpGANs; SGAN, LSGAN, and HingeGAN all have relativistic counterparts.
\section{Generalizing SVMs}
\label{sec:3}
The main approach used to generalize SVMs beyond the linear classifier is to apply the kernel trick \citep{aizerman1964theoretical}. This simply consists in replacing $f(x)=w^T x - b$ by $f(x) = w^T \phi(x) - b$, where $\phi(x)$ is a kernel. %This transforms the original space into a more complex space so that we have non-linear classifier with respect to the data, but linear with respect to the kernel.
Kernels can be chosen a priori or learned \citep{goodfellow2016deep}. In this section, we generalize SVMs to arbitrary classifiers $f(x)$, $L^p$-norms and loss functions. We start by showing how to derive an $L^p$-norm geometric margin. Then, we present the concept of maximizing the \em expected \em margin, rather than the \em minimum \em margin.
\subsection{Approximating the geometric margin}
Calculating the geometric margin involves computing a projection.
For general $L^p$ norms it has no closed form.
One way to approximate it, is using a Taylor expansion.
Depending on the order of the Taylor expansion (before or after solving for the projection), we can get two different approximations:
one new and one existing.
\subsubsection{Taylor approximation (After solving)}\label{sec:3.1.1}
The formulation of the $L^p$-norm margin \eqref{eq:10} has no closed form for arbitrary non-linear classifiers. However, when $p=2$, if we use a Taylor's expansion, we can show that \begin{align*}
\gamma_2(x) &= \frac{|\nabla_{x_0} f(x_0)^T (x-x_0)|}{|| \nabla_{x_0} f(x_0) ||_2} \\
&\approx \frac{|f(x)|}{|| \nabla_{x_0} f(x_0) ||_2} \hspace*{2pt} \text{ (Taylor's expansion)} \\
&\approx \frac{|f(x)|}{|| \nabla_x f(x) ||_2} \hspace*{2pt} \text{ (if $f(x)\approx w^T x - b$)}
\end{align*}
This approximation depends on approximate linearity of the classifier. If $f(x)=w^T x - b$, we have that $|| \nabla_x f(x) ||_2 = ||w||$ (and vice-versa). This means that if we enforce $|| \nabla_x f(x) ||_2 \approx 1$ for all $x$, we have $|| \nabla_{x_0} f(x_0) ||_2 \approx 1$ for all points $x_0$ on the boundary. This may appear to bring us back to the original scenario with a linear classifier. However, in practice, we only penalize the gradient norm in expectation which means that we do not obtain a linear classifier.
Thus, we can use the following pseudo-margin:
$$ \gamma(x)_2^{+} = \frac{yf(x)}{|| \nabla_x f(x) ||_2}. $$
\subsubsection{Taylor approximation (Before solving)}
An alternative approach to derive a pseudo-margin is to use Taylor's approximation before solving the problem rather than after (as done by \citet{matyasko2017margin} and \citet{elsayed2018large}):
\begin{align*}
\gamma_p(x) &= \min_{r} || r ||_p \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x+r)=0 \\
&\approx \min_{r} || r ||_p \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x)+\nabla_x f(x)^T r=0 \\
&=\frac{|f(x)|}{|| \nabla_x f(x) ||_q},
\end{align*}
where $||\cdot||_q$ is the dual norm \citep{boyd2004convex} of $||\cdot||_p$. By Hölder's inequality \citep{holder1889ueber, rogers1888extension}, we have that $1/p + 1/q=1$. This means that if $p=2$, we still get $q=2$; if $p=\infty$, we get $q=1$; if $p=1$, we get $q=\infty$.
We can then define the geometric margin as:
$$ \gamma_p^{-} = \frac{yf(x)}{|| \nabla_x f(x) ||_q}.$$
\subsection{Maximizing the expected margin}
As previously discussed, the goal of hard-margin SVMs is to maximize the \em minimum margin \em as in equation \eqref{eqn:1}.
However, this problem is infeasible in non-linearly separable datasets.
In these cases, the soft-margin formulation of SVM is most common:
\begin{align}\label{eqn:4andahalf}
\max_f \mathbb{E}_{(x,y)\sim D}\left[ F(\gamma(x,y)) \right],
\end{align}
where $F:\mathbb{R}\to \mathbb{R}$ is an objective to be maximized (not to be confused with the classifier $f$) and the expectation represents the empirical average over a sampled dataset $D$. For large datasets, the empirical average is a good approximation of the expectation of the data distribution, $\mathbb{D}$.
This is an easier optimization problem to solve compared to equation \eqref{eqn:1}, and is also always feasible.
% However, this problem is difficult and, in practice, we solve specific instances of the following optimization problem \footnote{We replaced the empirical average by the expectation over the distribution for more generality. The empirical average is an estimate of the expectation.}:
If $F$ is chosen to be the negative hinge function (i.e., $F(z)=-max(0,1-z)$),
we ignore samples far from the boundary (as in SVMs). For general choices of $F$, every sample may influence the solution. The identity function $F(z)=z$, cross entropy with sigmoid activation $F(z)=\log(\text{sigmoid}(z)))$ and least-squares $F(z)=-(1-z)^2$ are also valid choices.
However, as before, we prefer to separate the numerator from the denominator of the margin. Furthermore, the denominator (the norm of the gradient) is now a random variable. To make things as general as possible, we use the following formulation:
\begin{align}\label{eqn:6}
\max_f \mathbb{E}_{(x,y)\sim \mathbb{D}}\left[F(yf(x)) - \lambda g(||\nabla_x f(x)||_q)\right].
\end{align}
where $F,g:\mathbb{R}\to \mathbb{R}$ and $\lambda$ is a scalar penalty term. There are many potential choices of $F$ and $g$ which we can use.
%Regarding the loss function $l$, as mentioned, we can use the Hinge function to obtain Soft-SVMs. The simplest choice of $l$ is to use the identity function $F(z)=z$ (as we will show, this corresponds to WGAN-GP formulation). %In SVMs, we use the Hinge function $F(z)=max(0,1-z)$. Of note, this function is not differentiable at zero; to make the Hinge loss function differentiable everywhere, one can also use the squared Hinge function $F(z)=max(0,1-z)^2$. Many other smooth variants of the Hinge function also exist \citep{hastie2005elements, rennie2005loss}.
The standard choice of $g$ (in SVMs) is $g(z)=(z^2-1)$. This corresponds to constraining $|| \nabla_x f(x) || = 1$ or $|| \nabla_x f(x) || \leq 1$ for all $x$ (by KKT conditions). Since the gradient norm is a random variable, we do not want it to be equal to one everywhere. For this reason, we will generally work with softer constraints of the form $g(z)=(z-1)^2$ or $g(z)=max(0,z-1)$. The first function enforces a soft equality constraint so that $z\approx 1$ while the second function enforces a soft inequality constraint so that $z \leq 1$.
Of note, under perfect separation of the data and with a linear classifier, it has been shown that the empirical version of equation \eqref{eqn:6} (integrating over a dataset $D$ drawn from distribution $\mathbb{D}$) divided by its norm is equivalent to \eqref{eqn:1} under the constraint $||w||=1$ when $\lambda \to 0$ \citep{rosset2004margin}. This is true for cross-entropy and Hinge loss functions, but not least-squares. This implies that, under strong assumptions, maximizing the expected margin could also maximize the minimum margin.
\subsection{Better approximation of the margin}
In Section~\ref{sec:3.1.1}, we showed an approximation to the $L^2$-norm geometric margin. To reach a closed form, we had to assume that the classifier was approximately linear. This approximation is problematic since samples are pushed away from the boundary so we may never minimize the gradient norm at the boundary (as needed to actually maximize the geometric margin).
Given that we separate the problem of estimating an MMC into maximizing a function of the numerator ($yf(x)$) and minimizing a function of the denominator (gradient norm), we do not need to make this approximation. Rather than finding the closest element of the decision boundary $x_0$ for a given sample $x$, we can simply apply the penalty on the decision boundary. However, working on the boundary is intractable given the infinite size of the decision boundary.
Although sampling from the decision boundary is difficult, sampling around it is easy. Rather than working on the decision boundary, we can instead apply the constraint in a bigger region encompassing all points of the decision boundary. A simple way to do so is to sample from all linear interpolations between samples from classes 1 and 2. This can be formulated as:
\begin{align}\label{eqn:7}
\max_f \mathbb{E}_{(x,y)\sim \mathbb{D}}\left[F(yf(x)) \right] - \lambda \mathbb{E}_{\tilde{x}} \left[g(||\nabla_{\tilde{x}} f(\tilde{x})||_2)\right],
\end{align}
where $\tilde{x} = \alpha x + (1-\alpha)y$, $\alpha \sim U(0,1)$, $x \sim \mathbb{P}$, and $y \sim \mathbb{Q}$. This is same interpolation as used in WGAN-GP; this provides an additional argument in favor of this practice.
\section{Connections to GANs}
Let $f(x)=C(x)$. Although not immediately clear given the different notations, the objective functions of the discriminator/critic in many penalized GANs are equivalent to the ones from MMCs based on \eqref{eqn:7}. If $g(z)=(z-1)^2$, we have that $F(z)=z$ corresponds to WGAN-GP, $F(z)=\log(\text{sigmoid}(z))$ corresponds to SGAN, $F(z)=-(1-z)^2$ corresponds to LSGAN, and $F(z)=-max(0,1-z)$ corresponds to HingeGAN with gradient penalty. Thus, all of these penalized GANs maximize an expected $L^2$-norm margin.
% Note that \cite{lim2017geometric} explored a similar connection between GANs and SVMs which they used to propose Geometric GANs. The main difference is that they assume a linear classifier working on the \em feature space of a neural network's output \em instead of the \em input space \em. Thereby, they did not make a connection to gradients penalties. This new connection motivates an $L^\infty$ norm gradient penalty which shows great promise over the standard $L^2$ norm gradient penalty.
\subsection{Equivalence between gradient norm constraints and Lipschitz functions}
\label{sec:4.1}
As stated in Section~\ref{sec:GAN}, the popular approach of softly enforcing $|| \nabla_x f(x) ||_2 \approx 1$ at all interpolations between real and fake samples does not ensure that we estimate the Wasserstein distance ($W_1$).
On the other hand, we show here that enforcing $|| \nabla_x f(x) ||_2 \leq 1$ is sufficient in order to estimate $W_1$.
Assuming $d(x_1,x_2)$ is a $L^p$-norm, $p \ge 2$ and $f(x)$ is differentiable, we have that:
\begin{align*}
|| \nabla f(x) ||_p \leq K \iff f \text{ is K-Lipschitz on $L^p$}.
\end{align*} See appendix for the proof. \citet{adler2018banach} showed a similar result on dual norms. This suggests that, in order to work on the set of Lipschitz functions, we can penalize $|| \nabla_x f(x) || \leq 1$ for all $x$.
This can be done easily through \eqref{eqn:6} by choosing $g(z)=\max(0,z-1)$. \citet{petzka2017regularization} also suggested using a similar function (the square hinge) in order to only penalize gradient norms above 1.
If we let $F(z)=z$ and $g(z)=\max(0,z-1)$, we have an IPM over all Lipschitz functions; thus, we effectively approximate $W_1$. This means that $W_1$ can be found through maximizing a geometric margin.
Importantly, most successful GANs \citep{brock2018large,karras2019style,karras2017progressive} either enforce the 1-Lipschitz property using Spectral normalization \citep{miyato2018spectral} or use some form of gradient norm penalty \citep{WGAN-GP,mescheder2018training}. Since 1-Lipschitz is equivalent to enforcing a gradient norm constraint (as shown above), we have that most successful GANs effectively train a discriminator/critic to maximize a geometric margin. This suggests that the key contributor to stable and effective GAN training may not be having a 1-Lipschitz discriminator, but may be maximizing a geometric margin.
\subsection{Why do maximum-margin classifiers make good GAN discriminators/critics?}
\label{sec:4.2}
To answer this question, we focus on a simple two-dimensional example where $x=(x_{(1)},x_{(2)})$. Let real data (class 1) be uniformly distributed on the line between $(1,-1)$ and $(1,1)$. Let fake data (class 2) be uniformly distributed on the line between $(-1,-1)$ and $(-1,1)$. This is represented by Figure~\ref{fig:fig1}. Clearly, the maximum-margin boundary is the line $x_{(1)}=0$ and any classifier should learn to ignore $x_{(2)}$. For a classifier of the form $f(x)=c + w_1 x_{(1)} + w_2 x_{(2)}$, the maximum-margin classifier is $f^{*}(x)=w_1 x_{(1)}$ for any choice of $w_1 > 0$. We can see this by looking at the expected geometric margin:
\begin{align*}
\mathbb{E}_{(z,y)\sim \mathbb{D}}\left[ \gamma(x,y) \right] &= \mathbb{E}_{(z,y)\sim \mathbb{D}}\left[\frac{y w_1 x_{(1)}}{|w_1|} \right] \\
&= \frac{2 w_1 x_{(1)}}{|w_1|} \\
&= \begin{cases}
2 &\text{if $w_1 > 0$}\\
-2 &\text{if $w_1 < 0$}.
\end{cases}
\end{align*}
\vspace*{-15pt}
\begin{figure}[!ht]
\centering
\includegraphics[scale=0.59]{fig1.pdf}
%\vspace*{-15pt}
\caption{Two-dimensional GAN example with different choices of boundaries.}
\label{fig:fig1}
\end{figure}
This means that the problem is overparameterized (there are infinitely many solutions). We will show that this is problematic in GANs.
In GANs, the dynamics of the game depends in great part on $||\nabla_{x_2} f(x_2)||$ where $x_2$'s are samples from the fake, or generated, distribution (not to be confused with $x_{(2)}$, see Section~\ref{sec:notation} for definition).
This is because the generator only learns through the discriminator/critic and it uses $\nabla_{x_2} f(x_2)$ in order to improve its objective function. Thus, for stable training, $||\nabla_{x_2} f(x_2)||$ should not be too big or too small. %In the previous example, we can end up in either situation since the gradient norm $|w_1|$ can be chosen arbitrarily.
There are two ways of ensuring this property in this example: we can either i) fix the gradient norm to 1 or ii) fix $y w_1 x_{(1)} = 1$. Both solutions lead to $w_1=1$. The former is the approach taken by soft-margin SVMs and the latter is the approach taken by hard-margin SVMs. %However, note that soft-SVMs do not normally work on the margin directly, but rather use a Hinge loss.
This means that, in order to get stable GAN training, maximizing a margin is not enough. We need to ensure that we obtain a solution with a stable non-zero gradient around fake samples. Thus, it is preferable to solve the penalized formulation from equation \eqref{eqn:7} and choose a large penalty term $\lambda$ in order to obtain a small-gradient solution.
When the gradient norm is 1 everywhere, the only solution is a linear classifier which leads to the gradient being fixed everywhere. In this case, the placement of the margin may not be particularly important for GAN stability since the gradient is the same everywhere (although we do still obtain an MMC).
When we have a non-linear classifier and we impose
$||\nabla_{x} f(x)|| \leq 1$ through $g(z)=\max(0,z-1)$,
the gradient norm will fade toward zero as we move away from the boundary. Thus, in this case, obtaining a maximum-margin solution is important because it reduces the risk of vanishing gradients at fake samples. To see this, we can consider our simple example, but assume $f(x)=\text{sigmoid}(w_1 x_{(1)}+w_0)$ (See Figure~\ref{fig:fig2}).
%\vspace*{-10pt}
\begin{figure}[!ht]
\centering
\includegraphics[scale=0.59]{fig3.pdf}
\caption{$\nabla f(x_{(1)})$ at different values of $x_{(1)}$ for the two-dimensional example assuming a sigmoid function.}
\label{fig:fig2}
\end{figure}
%\vspace*{-15pt}
We can enforce $||\nabla_{x} f(x)|| \leq 1$, by choosing $w_1 \leq 4$. We let $w_1 = 4$ because it leads to the best classifier. The maximum-margin boundary is at $x_{(1)}=0$ (which we get by taking $w_0=0$; blue curve in Figure~\ref{fig:fig2}); for this choice, we have that $f(x_1)=.02$ and $f(x_2)=.98$ for real and fake samples respectively. Meanwhile, if we take a slightly worse margin with boundary at $x_{(1)} = \frac{1}{4}$ (equivalent to choosing $w_0=-1$; red curve in Figure~\ref{fig:fig2}), we have that $f(x_1)=.01$ and $f(x_2)=.95$ for real and fake samples respectively. Thus, both solutions almost perfectly classify the samples. However, the optimal margin has gradient $.07$, while the worse margin has gradient $.03$ at fake samples. Thus, the maximum-margin provides a stronger signal for the generator. Had we not imposed a gradient penalty constraint, we could have chosen $w_1 = 8$ (green curve in Figure~\ref{fig:fig2}) and we would have ended up with vanishing gradients at fake samples while still using a maximum-margin classifier.
In summary, imposing $||\nabla_{x} f(x)|| \approx 1$, as done in WGAN-GP, may be helpful because it approximately fixes the gradient to 1 everywhere which stabilizes the generator's training. However, it imposes a strong constraint (linear discriminator) which only leads to a lower bound on Wasserstein's distance. Meanwhile, imposing $||\nabla_{x} f(x)|| \leq 1$, as we suggested to properly estimate Wasserstein's distance, may be helpful because it reduces the risk of having no gradient at fake samples.
\subsection{Are certain margins better than others?}
\label{sec:4.3}
It is well known that $L^p$-norms (with $p\ge 1$) are more sensitive to outliers as $p$ increases which is why many robust methods minimize the $L^1$-norm \citep{bloomfield1983least}. Furthermore, minimizing the $L^1$-norm loss results in a median estimator \citep{bloomfield1983least}. This suggests that penalizing the $L^2$ gradient norm penalty ($p=2$) may not lead to the most robust classifier. We hypothesize that $L^\infty$ gradient norm penalties may improve robustness in comparision to $L^2$ gradient norm penalties since they correspond to maximizing $L^1$-norm margin.
In Section~\ref{sec:experiments} we provide experimental evidence in support of our hypothesis.
\subsection{Margins in Relativistic GANs}
\label{sec:4.4}
Relativistic paired GANs (RpGANs) and Relativistic average GANs (RaGAN) \citep{jolicoeur2018relativistic} are GAN variants which tend to be more stable than their non-relativistic counterparts.
Below, we explain how we can link both approaches to MMCs.
\subsubsection{Relativistic average GANs}
From the loss function of RaGAN, we can deduce its decision boundary.
Contrary to typical classifiers, we define two boundaries, depending on the label.
The two surfaces are defined as two sets of points $(x_0,y_0)$ such that:
\begin{align*}
f(x_0) &= \mathbb{E}_{x \sim \mathbb{Q}}[f(x)] \text{, when } y_0 = 1 (\text{real}) \\
f(x_0) &= \mathbb{E}_{x \sim \mathbb{P}}[f(x)] \text{, when } y_0 = -1 (\text{fake})
\end{align*}
It can be shown that the relativistic average geometric margin is approximated as:
\begin{align*}
\gamma_p^{Ra-}(x,y) = &\frac{((y+1)/2)(f(x)-\mathbb{E}_{x \sim \mathbb{Q}}[f(x)])}{|| \nabla_x f(x) ||_q} + \\ &\frac{ ((y-1)/2)(f(x)-\mathbb{E}_{x \sim \mathbb{P}}[f(x)])}{|| \nabla_x f(x) ||_q} \\ = &\frac{\alpha_{Ra}(x,y)}{\beta(x)}.
\end{align*}
Maximizing the boundary of RaGANs can be done in the following way:
\begin{align*}
\max_f \mathbb{E}_{(x,y)\sim \mathbb{D}}\left[F(\alpha_{Ra}(x,y)) - \lambda g(||\nabla_x f(x)||_q)\right].
\end{align*}
Of note, when $F(z)=z$ (identity function), the loss is equivalent to its non-relativistic counterpart (WGAN-GP). Of all the RaGAN variants presented by \citet{jolicoeur2018relativistic}, RaHingeGAN with $F(z)=-\max(0,1-z)$ is the only one which maximizes the relativistic average geometric margin when using a gradient norm penalty.
\subsubsection{Relativistic paired GANs}
From its loss function (as described in section \ref{sec:GAN}), it is not clear what the boundary of RpGANs can be. However, through reverse engineering, it is possible to realize that the boundary is the same as the one from non-relativistic GANs, but using a different margin. We previously derived that the approximated margin (non-geometric) for any point is $\gamma_p(x) \approx \frac{|f(x)|}{|| \nabla_x f(x) ||_q}$. We define the geometric margin as the margin after replacing $|f(x)|$ by $yf(x)$ so that it depends on both $x$ and $y$. However, there is an alternative way to transform the margin in order to achieve a classifier. We call it the \em relativistic paired margin\em:
\begin{align*}
\gamma^{*}_p(x_1,x_2) &= \gamma_p(x_1) - \gamma_p(x_2) \\
& = \frac{f(x_1)}{|| \nabla_{x_1} f(x_1) ||_q} - \frac{f(x_2)}{|| \nabla_{x_2} f(x_2) ||_q}.
\end{align*}
where $x_1$ is a sample from $\mathbb{P}$ and $x_2$ is a sample from $\mathbb{Q}$. This alternate margin does not depend on the label $y$, but only ask that for any pair of class 1 (real) and class 2 (fake) samples, we maximize the relativistic paired margin. This margin is hard to work with, but if we enforce $|| \nabla_{x_1} f(x_1) ||_q \approx || \nabla_{x_2} f(x_2) ||_q$, for all $x_1 \sim \mathbb{P}$,$x_2 \sim \mathbb{Q}$, we have that:
\begin{align*}
\gamma^{*}_p(x_1,x_2) \approx \frac{f(x_1) - f(x_2)}{|| \nabla_{x} f(x) ||_q},
\end{align*}
where $x$ is any sample (from class 1 or 2).
Thus, we can train an MMC to maximize the relativistic paired margin in the following way:
\begin{align*}
\max_f \underset{ \substack{x_1 \sim \mathbb{P} \\ x_2 \sim \mathbb{Q}}}{\mathbb{E}\vphantom{p}}&\left[F(f(x_1)-f(x_2))\right] - \\ \lambda& \mathbb{E}_{(x,y)\sim \mathbb{D}}\left[g(||\nabla_x f(x)||_q)\right],
\end{align*}
where $g$ must constrain $||\nabla_x f(x)||_q$ to a constant.
This means that maximizing $F(f(x_1)-f(x_2))$ without gradient penalty can be problematic if we have different gradient norms at samples from class 1 (real) and 2 (fake). This provides an explanation as to why RpGANs do not perform very well unless using a gradient penalty \citep{jolicoeur2018relativistic}.
\section{Experiments}
\label{sec:5}
\label{sec:experiments}
Following our analysis and discussion in the previous sections, we hypothesized that $L^1$ margins, corresponding to a $L^\infty$ gradient norm penalty, would perform better than $L^2$ margins ($L^2$ gradient norm penalty).
As far as we know, researchers have not yet tried using a $L^\infty$ gradient norm penalty in GANs.
In addition, we showed that it would be more sensible to penalize violations of $||\nabla f(x)||_q \leq 1$ rather than $||\nabla f(x)||_q \approx 1$.
To test these hypotheses, we ran experiments on CIFAR-10 \citep{krizhevsky2009learning} using HingeGAN ($F(z)=-\max(0,1-z)$) and WGAN ($F(z)=z$) loss functions with $L^1$, $L^2$, $L^\infty$ gradient norm penalties. We enforce either $||\nabla f(x)||_q \approx 1$ using Least Squares (LS) $(g(z)=(z-1)^2)$ or $||\nabla f(x)||_q \leq 1$ using Hinge $(g(z)=\max(0,z-1))$. We used the ADAM optimizer \citep{Adam} with $\beta=(.5,.99)$ and a DCGAN architecture \citep{DCGAN}. As per convention, we report the Fréchet Inception Distance (FID) \citep{heusel2017gans}; lower values correspond to better generated outputs (higher quality and diversity). We ran all experiments using seed 1 and with gradient penalty $\lambda=20$. Details on the architectures are in the Appendix. Code is available on \em https://github.com/AlexiaJM/MaximumMarginGANs\em. The results are shown in Table~\ref{tab:1}.
\begin{table}[!ht]
\caption{Fréchet Inception Distance (FID) after 100k generator iterations on CIFAR-10.}
\label{tab:1}
\centering
\begin{tabular}{ccc}
\toprule
$g(||\nabla_x f(x))||_q)$ & WGAN & HingeGAN \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_1-1)^2$ & 99.7 & 88.9 \\
$\max(0,||\nabla_x f(x))||_1-1)$ & 65.6 & 77.3 \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_2-1)^2$ & 37.6 & 32.8 \\
$\max(0,||\nabla_x f(x))||_2-1)$ & 37.8 & 33.9 \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_{\infty}-1)^2$ & 33.4 & 33.6 \\
$\max(0,||\nabla_x f(x))||_{\infty}-1)$ & 36 & \fontseries{b}\selectfont 27.1 \\
\bottomrule
\end{tabular}
\end{table}
Due to space constraint, we only show the previously stated experiments in Table~\ref{tab:1}. However, we also ran additional experiments on CIFAR-10 with 1) Relativistic paired and average HingeGAN, 2) $\beta=(0,.90)$, 3) the standard CNN architecture from \citet{miyato2018spectral}. Furthermore, we ran experiments on CAT \citep{cat} with 1) Standard CNN (in 32x32), and 2) DCGAN (in 64x64). These experiments correspond to Table~\ref{tab:2},~\ref{tab:3},~\ref{tab:4},~\ref{tab:5}, and~\ref{tab:6} from the appendix.
In all sets of experiments, we generally observed that we obtain smaller FIDs by using: i) a larger $q$ (as theorized), ii) the Hinge penalty to enforce an inequality gradient norm constraint (in both WGAN and HingeGAN), and iii) HingeGAN instead of WGAN.
\section{Conclusion}
%In this paper, we explained how to generalize Maximum-Margin Classifiers (MMCs) beyond the $L^2$-norm margin with a linear classifier (which corresponds to SVMs). This lead to various loss functions with gradient norm penalties. We explained the links between GANs with gradient penalties and MMCs. WGAN-GP and Hinge-GAN are special cases of MMCs with an $L^2$-norm margin. We hypothesized that discriminators that maximized $L^1$-norm geometric margins would potentially lead to better generated samples by the generator than those that maximize the $L^2$-norm or $L^\infty$-norm geometric margins. Through experiments, we confirmed this to be true.
This work provides a framework in which to derive MMCs that results in very effective GAN loss functions. In the future, this could be used to derive new gradient norm penalties which further improve the performance of GANs. Rather than trying to devise better ways of enforcing 1-Lipschitz, researchers may instead want to focus on constructing better MMCs (possibly by devising better margins).
This research shows a strong link between GANs with gradient penalties, Wasserstein's distance, and SVMs. Maximizing the minimum $L^2$-norm geometric margin, as done in SVMs, has been shown to lower bounds on the VC dimension which implies lower generalization error \citep{vapnik1998statistical,mount2015sure}. This paper may help researchers bridge the gap needed to derive PAC bounds on Wasserstein's distance and GANs/IPMs with gradient penalty. Furthermore, it may be of interest to theoreticians whether certain margins lead to lower bounds on the VC dimension.
\section{Acknowledgements}
This work was supported by Borealis AI through the Borealis AI Global Fellowship Award. We would also like to thank Compute Canada and Calcul Québec for the GPUs which were used in this work.
This work was also partially supported by the FRQNT new researcher program (2019-NC-257943), the NSERC Discovery grant (RGPIN-2019-06512), a startup grant by IVADO, a grant by Microsoft Research and a Canada CIFAR AI chair.
%\bibliographystyle{apalike}
\bibliographystyle{unsrtnat}
\bibliography{paper}
\clearpage
\appendix
\section*{Appendices}
\addcontentsline{toc}{section}{Appendices}
\renewcommand{\thesubsection}{\Alph{subsection}}
\subsection{Additional experiments}
Note that the \em smooth maximum \em is defined as
$$\text{smax}(x_{(1)},\ldots,x_{(k)})=\frac{\sum_{i=1}^{k} x_{(i)} e^{x_i}}{\sum_{i=1}^{k} e^{x_i}}.$$
We sometime use the smooth maximum as a smooth alternative to the $L^{\infty}$-norm margin; results are worse with it.
\begin{table}[!ht]
\caption{FID after 100k generator iterations on CIFAR-10 using the same setting as Table1, but we are using Relativistic paired and average GANs.}
\label{tab:2}
\centering
\begin{tabular}{ccc}
\toprule
$g(||\nabla_x f(x))||_q)$ & RpHinge & RaHinge \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_1-1)^2$ & 64.4 & 65.0 \\
$\max(0,||\nabla_x f(x))||_1-1)$ & 60.3 & 68.5 \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_2-1)^2$ & 32.8 & 31.9 \\
$\max(0,||\nabla_x f(x))||_2-1)$ & 32.6 & 35.0 \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_{\infty}-1)^2$ & 32.5 & 33.5 \\
$\max(0,||\nabla_x f(x))||_{\infty}-1)$ & \fontseries{b}\selectfont 28.2 & 28.4 \\
\cmidrule(){1-3}
$(\text{smax}|\nabla_x f(x)|-1)^2$ & 133.7 & 124.0 \\
$\max(0,\text{smax}|\nabla_x f(x)|-1)$ & 30.5 & 30.3 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!ht]
\caption{FID after 100k generator iterations on CIFAR-10 using the same setting as Table1, but we are using Adam $\beta=(0, .90)$.}
\label{tab:3}
\centering
\begin{tabular}{ccc}
\toprule
$g(||\nabla_x f(x))||_q)$ & WGAN & HingeGAN \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_1-1)^2$ & 163.2 & 179.0 \\
$\max(0,||\nabla_x f(x))||_1-1)$ & 66.9 & 66.1 \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_2-1)^2$ & 34.6 & 33.8 \\
$\max(0,||\nabla_x f(x))||_2-1)$ & 37.0 & 34.9 \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_{\infty}-1)^2$ & 37.5 & 33.9 \\
$\max(0,||\nabla_x f(x))||_{\infty}-1)$ & 38.3 & \fontseries{b}\selectfont 28.6 \\
\cmidrule(){1-3}
$(\text{smax}|\nabla_x f(x)|-1)^2$ & 31.9 & 283.1 \\
$\max(0,\text{smax}|\nabla_x f(x)|-1)$ & 31.8 & 32.1 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!ht]
\caption{FID after 100k generator iterations on CIFAR-10 using the same setting as Table1, but we are using the standard CNN architecture.}
\label{tab:4}
\centering
\begin{tabular}{ccc}
\toprule
$g(||\nabla_x f(x))||_q)$ & WGAN & HingeGAN \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_2-1)^2$ & 30.2 & 26.1 \\
$\max(0,||\nabla_x f(x))||_2-1)$ & 31.8 & 27.3 \\
\cmidrule(){1-3}
$\max(0,||\nabla_x f(x))||_{\infty}-1)$ & 74.3 & \fontseries{b}\selectfont 21.3 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!ht]
\caption{FID after 100k generator iterations on CAT (in 32x32) using the same setting as Table 1, but we are using the standard CNN architecture. Exceptionally, this set of experiment showed convergence at around 10-40k iterations (this has not been the case in any of the other experiments). For this reason, we show also the lowest FID obtained during training (FID was measured at every 10k iterations).}
\label{tab:5}
\centering
\begin{tabular}{ccc}
\toprule
$g(||\nabla_x f(x))||_q)$ & WGAN & HingeGAN \\
\cmidrule(){1-3}
At 100k iterations & & \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_2-1)^2$ & 27.3 & \fontseries{b}\selectfont 19.5 \\
$\max(0,||\nabla_x f(x))||_2-1)$ & 21.4 & 23.9 \\
\cmidrule(){1-3}
$\max(0,||\nabla_x f(x))||_{\infty}-1)$ & 66.5 & 24.0 \\
\cmidrule(){1-3}
Lowest FID obtained & & \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_2-1)^2$ & 20.9 & 16.2 \\
$\max(0,||\nabla_x f(x))||_2-1)$ & 19.4 & 17.0 \\
\cmidrule(){1-3}
$\max(0,||\nabla_x f(x))||_{\infty}-1)$ & 32.32 & \fontseries{b}\selectfont 9.5 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!ht]
\caption{FID after 100k generator iterations on CAT (in 64x64) using the same setting as Table 1.}
\label{tab:6}
\centering
\begin{tabular}{ccc}
\toprule
$g(||\nabla_x f(x))||_q)$ & WGAN & HingeGAN \\
\cmidrule(){1-3}
$(||\nabla_x f(x))||_2-1)^2$ & 48.2 & 26.7 \\
$\max(0, ||\nabla_x f(x))||_2-1)$ & 43.7 & 29.6 \\
\cmidrule(){1-3}
$\max(0, ||\nabla_x f(x))||_{\infty}-1)$ & 18.3 & \fontseries{b}\selectfont 17.5 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!ht]
\caption{Extras from Table 1}
\label{tab:7}
\centering
\begin{tabular}{ccc}
\toprule
$g(||\nabla_x f(x))||_q)$ & WGAN & HingeGAN \\
\cmidrule(){1-3}
\cmidrule(){1-3}
$(\text{smax}|\nabla_x f(x)|-1)^2$ & 35.3 & 197.9 \\
$\max(0,\text{smax}|\nabla_x f(x)|-1)$ & 31.4 & 29.5 \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Proofs}
Note that both of the following formulations represent the margin:
\begin{align*}
\gamma(x) = &\min_{x_0} || x_0 - x || \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x_0)=0 \\
= &\min_{r} || r || \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x+r)=0
\end{align*}
\subsubsection{Bounded Gradient $\iff$ Lipschitz }
Assume that $f:X \to \mathbb{R}$ and $X$ is a convex set.
Let $\tilde{x}(\alpha) = \alpha x_1 + (1-\alpha) x_2$, where $\alpha \in [0,1]$ be the interpolation between any two points $x_1,x_2 \in X$. We know that $\tilde{x}(\alpha) \in X$ for any $\alpha \in [0,1]$ by convexity of $X$.
\begin{align*}
f(x_1)-f(x_2) &= f(\tilde{x}(1))-f(\tilde{x}(0)) \\
&= \int_{0}^{1} \frac{df(\tilde{x}(\alpha))}{d\alpha} d\alpha \\
&= \int_{0}^{1} \nabla f(\tilde{x}(\alpha)) \frac{\tilde{x}(\alpha)}{d\alpha} d\alpha \\
&= \int_{0}^{1} \nabla f(\tilde{x}(\alpha)) (x_1-x_2) d\alpha \\
&= (x_1-x_2) \int_{0}^{1} \nabla f(\tilde{x}(\alpha)) d\alpha
\end{align*}
1) We show $||\nabla f(x)||_p \leq K \implies \frac{|f(x)-f(y)|}{||x-y||_p} \leq K$ for all $x,y$:
Let $||\nabla f(x)||_p \leq K$ for all $x \in X$.
\begin{align*}
|f(x_1)-f(x_2)| &= || (x_1-x_2) \int_{0}^{1} \nabla f(\tilde{x}(\alpha)) d\alpha ||_p \\
&\leq || x_1-x_2 ||_p \cdot ||_p \int_{0}^{1} \nabla f(\tilde{x}(\alpha)) d\alpha || \\
&\leq || x_1-x_2 ||_p \int_{0}^{1} ||\nabla f(\tilde{x}(\alpha))||_p d\alpha \\
&\leq || x_1-x_2 ||_p \int_{0}^{1} K d\alpha \\
&\leq K || x_1-x_2 ||_p \\
\end{align*}
2) We show $\frac{|f(x)-f(y)|}{||x-y||_p} \leq K$ for all $x,y$ $\implies || \nabla f(x)||_p \leq K$ for $p\ge2$:
Assume $p\ge 2$ and $\frac{1}{p}+\frac{1}{q}=1$.
\begin{align*}
|| \nabla_x f(x) ||_p &\leq || \nabla_x f(x) ||_q \hspace{5pt} \text{ since $p \ge q$} \\
&= \max_v \nabla_x f(x)^T v \hspace{5pt} \text{ s.t. } \hspace{5pt} ||v||_p \leq 1 \\
&= \left| \nabla_x f(x)^T v^{*} \right| \hspace{3pt} \text{ where $v^{*}$ is the optimum} \\
&= lim_{h\to 0} \left| \frac{f(x+hv^{*})-f(x)}{h} \right| \\
&\leq lim_{h\to 0} \frac{\left|f(x+hv^{*})-f(x)\right|}{h|| v^{*} ||_p} \\
&= lim_{h\to 0} \frac{\left|f(x+hv^{*})-f(x)\right|}{||x + hv^{*} - x ||_p} \\
&\leq K
\end{align*}
\subsubsection{Taylor approximation}
Let $r = x_0 - x$; at the boundary $x_0$, we have that $f(x_0)=f(x+r)=C$, for some constant $C$. In the paper, we use generally assume $C=0$. We will make use of the following Taylor approximations:
\begin{align*}
f(x+r) \approx f(x) + \nabla_x f(x)^T r \\
\implies f(x) - C \approx -\nabla_x f(x)^T r
\end{align*}
and
\begin{align*}
f(x) \approx f(x_0) + \nabla_{x_0} f(x_0)^T (x-x_0) \\
\implies f(x) - C \approx -\nabla_{x_0} f(x_0)^T r
\end{align*}
\subsubsection{Taylor approximation (After solving)}
To make things easier, we maximize $(\gamma_2(x))^2$ instead of $\gamma_2(x)$. We also maximize with respect to $x_0$ instead of $r$:
\begin{align*}
\gamma^2_2(x) &= \min_{x_0} || x_0-x ||^2_2 \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x_0)=0 \\
&= \min_{x_0} || x_0-x ||^2_2 - \lambda f(x_0),
\end{align*}
where $\lambda$ is a scalar (Lagrange multiplier).
We can then differentiate with respect to $x_0$:
\begin{align}\label{eqn:8}
\nabla_{x_0} \gamma^2_2(x) &= (x_0-x) + \lambda \nabla_{x_0} f(x_0) = 0.
\end{align}
We will then use a inner product to be able to extract the optimal Lagrange multiplier:
\begin{align*}
\implies& -\nabla_{x_0} f(x_0)^T (x_0-x) = \lambda^{*} \nabla_{x_0} f(x_0)^T \nabla_{x_0} f(x_0) \\
\implies& \lambda^{*} = -\frac{\nabla_{x_0} f(x_0)^T (x_0-x)}{\nabla_{x_0} f(x_0)^T \nabla_{x_0} f(x_0)} \\
\implies& \lambda^{*} = -\frac{\nabla_{x_0} f(x_0)^T (x_0-x) }{|| \nabla_{x_0} f(x_0) ||^2_2}
\end{align*}
Now, we plug-in the optimal Langrange multiplier into equation \eqref{eqn:8} and we use a inner product:
\begin{align*}
\implies& x_0-x = -\lambda \nabla_{x_0} f(x_0) \\
\implies& x_0-x = \frac{\nabla_{x_0} f(x_0)^T (x_0-x) }{|| \nabla_{x_0} f(x_0) ||^2_2} \nabla_{x_0} f(x_0) \\
\implies& (x_0-x)^T (x_0-x) = \frac{\left( \nabla_{x_0} f(x_0)^T (x_0-x) \right)^2 }{|| \nabla_{x_0} f(x_0) ||^2_2} \\
\implies& \gamma^2_2(x) = || x_0 - x ||^2_2 = \frac{\left( \nabla_{x_0} f(x_0)^T (x_0-x) \right)^2 }{|| \nabla_{x_0} f(x_0) ||^2_2} \\
\implies& \gamma_2(x) = \frac{\left| \nabla_{x_0} f(x_0)^T (x_0-x) \right|}{|| \nabla_{x_0} f(x_0) ||_2} \\
\end{align*}
\subsubsection{Taylor approximation (Before solving)}
\begin{align*}
\min_r || r ||_p& \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x+r)=C \\
\approx \min_r || r ||_p& \hspace{5pt} \text{ s.t. } \hspace{5pt} f(x)+\nabla_x f(x)^T r=C \\
\implies \min_r || r ||_p& = \frac{|f(x)-C|}{\max_r \frac{|\nabla_x f(x)^T r|}{|| r ||_p}} \\
& = \frac{|f(x)-C|}{\max_r \frac{\nabla_x f(x)^T r}{|| r ||_p}} \\
& = \frac{|f(x)-C|}{\max_r \nabla_x f(x)^T \frac{r}{|| r ||_p}} \\
& = \frac{|f(x)-C|}{\max_{|| r ||_p \leq 1} \nabla_x f(x)^T r} \\
& = \frac{|f(x)-C|}{|| \nabla_x f(x) ||_q},
\end{align*}
where $\frac{1}{p} + \frac{1}{q} = 1$
This is true because of the definition of the Dual norm \citep{rudin1991functional}:
$$|| a ||_{*} = \max_{|| r ||_p \leq 1} a^T r = \max_r \frac{a^T r}{|| r ||_p} = ||a||_q$$
For a standard classifier, we have $C = 0$. For a RaGAN, we have $C= \mathbb{E}_{\mathbb{Q}}[f(x)]$ when $y=1$ (real) and $C= \mathbb{E}_{\mathbb{P}}[f(x)]$ when $y=-1$ (fake).
\subsection{Architectures}
\subsubsection{DCGAN 32x32 (as in Table 1, 2, 3)}
\begin{center}
\begin{tabular}{c}
Generator \\
\toprule\midrule
$z \in \mathbb{R}^{128} \sim N(0,I)$ \\
\midrule
ConvTranspose2d 4x4, stride 1, pad 0, 128$\to$512 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 512$\to$256 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 256$\to$128 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 128$\to$3 \\
\midrule
Tanh \\
\bottomrule
\end{tabular}
\end{center}
\begin{center}
\begin{tabular}{c}
Discriminator \\
\toprule\midrule
$x \in \mathbb{R}^{\text{3x32x32}}$ \\
\midrule
Conv2d 4x4, stride 2, pad 1, 3$\to$128 \\
\midrule
LeakyReLU 0.2 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 128$\to$256 \\
\midrule
BN and LeakyReLU 0.2 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 256$\to$512 \\
\midrule
BN and LeakyReLU 0.2 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 512$\to$1 \\
\bottomrule
\end{tabular}
\end{center}
\subsubsection{DCGAN 64x64 (as in Table 6)}
\begin{center}
\begin{tabular}{c}
Generator \\
\toprule\midrule
$z \in \mathbb{R}^{128} \sim N(0,I)$ \\
\midrule
ConvTranspose2d 4x4, stride 1, pad 0, 128$\to$512 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 512$\to$256 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 256$\to$128 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 128$\to$64 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 64$\to$3 \\
\midrule
Tanh \\
\bottomrule
\end{tabular}
\end{center}
\begin{center}
\begin{tabular}{c}
Discriminator \\
\toprule\midrule
$x \in \mathbb{R}^{\text{3x64x64}}$ \\
\midrule
Conv2d 4x4, stride 2, pad 1, 3$\to$64 \\
\midrule
LeakyReLU 0.2 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 64$\to$128 \\
\midrule
BN and LeakyReLU 0.2 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 128$\to$256 \\
\midrule
BN and LeakyReLU 0.2 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 256$\to$512 \\
\midrule
BN and LeakyReLU 0.2 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 512$\to$1 \\
\bottomrule
\end{tabular}
\end{center}
\subsubsection{Standard CNN (as in Table 4, 5)}
\begin{center}
\begin{tabular}{c}
Generator \\
\toprule\midrule
$z \in \mathbb{R}^{128} \sim N(0,I)$ \\
\midrule
linear, 128 $\to$ 512*4*4 \\
\midrule
Reshape, 512*4*4 $\to$ 512 x 4 x 4 \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 512$\to$256 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 256$\to$128 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 4x4, stride 2, pad 1, 128$\to$64 \\
\midrule
BN and ReLU \\
\midrule
ConvTranspose2d 3x3, stride 1, pad 1, 64$\to$3 \\
\midrule
Tanh \\
\bottomrule
\end{tabular}
\end{center}
\begin{center}
\begin{tabular}{c}
Discriminator \\
\toprule\midrule
$x \in \mathbb{R}^{\text{3x32x32}}$ \\
\midrule
Conv2d 3x3, stride 1, pad 1, 3$\to$64 \\
\midrule
LeakyReLU 0.1 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 64$\to$64 \\
\midrule
LeakyReLU 0.1 \\
\midrule
Conv2d 3x3, stride 1, pad 1, 64$\to$128 \\
\midrule
LeakyReLU 0.1 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 128$\to$128 \\
\midrule
LeakyReLU 0.1 \\
\midrule
Conv2d 3x3, stride 1, pad 1, 128$\to$256 \\
\midrule
LeakyReLU 0.1 \\
\midrule
Conv2d 4x4, stride 2, pad 1, 256$\to$256 \\
\midrule
LeakyReLU 0.1 \\
\midrule
Conv2d 3x3, stride 1, pad 1, 256$\to$512 \\
\midrule
Reshape, 512 x 4 x 4 $\to$ 512*4*4 \\
\midrule
linear, 512*4*4 $\to$ 1 \\
\bottomrule
\end{tabular}
\end{center}
\end{document}
\begin{thebibliography}{40}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Cortes and Vapnik(1995)]{cortes1995support}
Corinna Cortes and Vladimir Vapnik.
\newblock Support-vector networks.
\newblock \emph{Machine learning}, 20\penalty0 (3):\penalty0 273--297, 1995.
\bibitem[Aizerman(1964)]{aizerman1964theoretical}
Mark~A Aizerman.
\newblock Theoretical foundations of the potential function method in pattern
recognition learning.
\newblock \emph{Automation and remote control}, 25:\penalty0 821--837, 1964.
\bibitem[Goodfellow et~al.(2014)Goodfellow, Pouget-Abadie, Mirza, Xu,
Warde-Farley, Ozair, Courville, and Bengio]{GAN}
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
\newblock Generative adversarial nets.
\newblock In Z.~Ghahramani, M.~Welling, C.~Cortes, N.~D. Lawrence, and K.~Q.
Weinberger, editors, \emph{Advances in Neural Information Processing Systems
27}, pages 2672--2680. Curran Associates, Inc., 2014.
\newblock URL
\url{http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf}.
\bibitem[Jolicoeur-Martineau(2018{\natexlab{a}})]{jolicoeur2018beyonddivergence}
Alexia Jolicoeur-Martineau.
\newblock Gans beyond divergence minimization.
\newblock \emph{arXiv preprint arXiv:xxxx}, 2018{\natexlab{a}}.
\bibitem[Gulrajani et~al.(2017)Gulrajani, Ahmed, Arjovsky, Dumoulin, and
Courville]{WGAN-GP}
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron~C
Courville.
\newblock Improved training of wasserstein gans.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
S.~Vishwanathan, and R.~Garnett, editors, \emph{Advances in Neural
Information Processing Systems 30}, pages 5767--5777. Curran Associates,
Inc., 2017.
\newblock URL
\url{http://papers.nips.cc/paper/7159-improved-training-of-wasserstein-gans.pdf}.
\bibitem[Fedus et~al.(2017{\natexlab{a}})Fedus, Rosca, Lakshminarayanan, Dai,
Mohamed, and Goodfellow]{fedus2017many}
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew~M Dai, Shakir
Mohamed, and Ian Goodfellow.
\newblock Many paths to equilibrium: Gans do not need to decrease a divergence
at every step.
\newblock \emph{arXiv preprint arXiv:1710.08446}, 2017{\natexlab{a}}.
\bibitem[Mescheder et~al.(2018)Mescheder, Geiger, and
Nowozin]{mescheder2018training}
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.
\newblock Which training methods for gans do actually converge?
\newblock \emph{arXiv preprint arXiv:1801.04406}, 2018.
\bibitem[Karras et~al.(2019)Karras, Laine, and Aila]{karras2019style}
Tero Karras, Samuli Laine, and Timo Aila.
\newblock A style-based generator architecture for generative adversarial
networks.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 4401--4410, 2019.
\bibitem[Jolicoeur-Martineau(2018{\natexlab{b}})]{jolicoeur2018relativistic}
Alexia Jolicoeur-Martineau.
\newblock The relativistic discriminator: a key element missing from standard
gan.
\newblock \emph{arXiv preprint arXiv:1807.00734}, 2018{\natexlab{b}}.
\bibitem[Lim and Ye(2017)]{lim2017geometric}
Jae~Hyun Lim and Jong~Chul Ye.
\newblock Geometric gan.
\newblock \emph{arXiv preprint arXiv:1705.02894}, 2017.
\bibitem[Mao et~al.(2017)Mao, Li, Xie, Lau, Wang, and Smolley]{LSGAN}
Xudong Mao, Qing Li, Haoran Xie, Raymond~YK Lau, Zhen Wang, and Stephen~Paul
Smolley.
\newblock Least squares generative adversarial networks.
\newblock In \emph{2017 IEEE International Conference on Computer Vision
(ICCV)}, pages 2813--2821. IEEE, 2017.
\bibitem[M{\"u}ller(1997)]{muller1997integral}
Alfred M{\"u}ller.
\newblock Integral probability metrics and their generating classes of
functions.
\newblock \emph{Advances in Applied Probability}, 29\penalty0 (2):\penalty0
429--443, 1997.
\bibitem[Sriperumbudur et~al.(2009)Sriperumbudur, Fukumizu, Gretton,
Sch{\"o}lkopf, and Lanckriet]{sriperumbudur2009integral}
Bharath~K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard
Sch{\"o}lkopf, and Gert~RG Lanckriet.
\newblock On integral probability metrics,$\backslash$phi-divergences and
binary classification.
\newblock \emph{arXiv preprint arXiv:0901.2698}, 2009.
\bibitem[Mroueh et~al.(2017)Mroueh, Li, Sercu, Raj, and
Cheng]{mroueh2017sobolev}
Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu~Cheng.
\newblock Sobolev gan.
\newblock \emph{arXiv preprint arXiv:1711.04894}, 2017.
\bibitem[Mroueh and Sercu(2017)]{Fisher}
Youssef Mroueh and Tom Sercu.
\newblock Fisher gan.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
S.~Vishwanathan, and R.~Garnett, editors, \emph{Advances in Neural
Information Processing Systems 30}, pages 2513--2523. Curran Associates,
Inc., 2017.
\newblock URL \url{http://papers.nips.cc/paper/6845-fisher-gan.pdf}.
\bibitem[Arjovsky et~al.(2017)Arjovsky, Chintala, and Bottou]{WGAN}
Martin Arjovsky, Soumith Chintala, and L{\'e}on Bottou.
\newblock Wasserstein generative adversarial networks.
\newblock In \emph{International Conference on Machine Learning}, pages
214--223, 2017.
\bibitem[Petzka et~al.(2017)Petzka, Fischer, and
Lukovnicov]{petzka2017regularization}
Henning Petzka, Asja Fischer, and Denis Lukovnicov.
\newblock On the regularization of wasserstein gans.
\newblock \emph{arXiv preprint arXiv:1709.08894}, 2017.
\bibitem[Fedus et~al.(2017{\natexlab{b}})Fedus, Rosca, Lakshminarayanan, Dai,
Mohamed, and Goodfellow]{ManyPaths}
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew~M Dai, Shakir
Mohamed, and Ian Goodfellow.
\newblock Many paths to equilibrium: Gans do not need to decrease adivergence
at every step.
\newblock \emph{arXiv preprint arXiv:1710.08446}, 2017{\natexlab{b}}.
\bibitem[Kodali et~al.(2017)Kodali, Abernethy, Hays, and
Kira]{kodali2017convergence}
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira.
\newblock On convergence and stability of gans.
\newblock \emph{arXiv preprint arXiv:1705.07215}, 2017.
\bibitem[Jolicoeur-Martineau(2019)]{jolicoeur2019relativistic}
Alexia Jolicoeur-Martineau.
\newblock On relativistic $ f $-divergences.
\newblock \emph{arXiv preprint arXiv:1901.02474}, 2019.
\bibitem[Goodfellow et~al.(2016)Goodfellow, Bengio, and
Courville]{goodfellow2016deep}
Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
\newblock \emph{Deep learning}.
\newblock MIT press, 2016.
\bibitem[Matyasko and Chau(2017)]{matyasko2017margin}
Alexander Matyasko and Lap-Pui Chau.
\newblock Margin maximization for robust classification using deep learning.
\newblock In \emph{2017 International Joint Conference on Neural Networks
(IJCNN)}, pages 300--307. IEEE, 2017.
\bibitem[Elsayed et~al.(2018)Elsayed, Krishnan, Mobahi, Regan, and
Bengio]{elsayed2018large}
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy
Bengio.
\newblock Large margin deep networks for classification.
\newblock In \emph{Advances in neural information processing systems}, pages
842--852, 2018.
\bibitem[Boyd and Vandenberghe(2004)]{boyd2004convex}
Stephen Boyd and Lieven Vandenberghe.
\newblock \emph{Convex optimization}.
\newblock Cambridge university press, 2004.
\bibitem[Hölder(1889)]{holder1889ueber}
Otto Hölder.
\newblock via an averaging clause.
\newblock \emph{Messages from the Society of Sciences and the
Georg-Augusts-Universität zu Göttingen}, 1889:\penalty0 38--47, 1889.
\bibitem[Rogers(1888)]{rogers1888extension}
Leonhard~James Rogers.
\newblock An extension of a certain theorem in inequalities.
\newblock \emph{Messenger of Math.}, 17:\penalty0 145--150, 1888.
\bibitem[Rosset et~al.(2004)Rosset, Zhu, and Hastie]{rosset2004margin}
Saharon Rosset, Ji~Zhu, and Trevor~J Hastie.
\newblock Margin maximizing loss functions.
\newblock In \emph{Advances in neural information processing systems}, pages
1237--1244, 2004.
\bibitem[Adler and Lunz(2018)]{adler2018banach}
Jonas Adler and Sebastian Lunz.
\newblock Banach wasserstein gan.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
6754--6763, 2018.
\bibitem[Brock et~al.(2018)Brock, Donahue, and Simonyan]{brock2018large}
Andrew Brock, Jeff Donahue, and Karen Simonyan.
\newblock Large scale gan training for high fidelity natural image synthesis.
\newblock \emph{arXiv preprint arXiv:1809.11096}, 2018.
\bibitem[Karras et~al.(2017)Karras, Aila, Laine, and
Lehtinen]{karras2017progressive}
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
\newblock Progressive growing of gans for improved quality, stability, and
variation.
\newblock \emph{arXiv preprint arXiv:1710.10196}, 2017.
\bibitem[Miyato et~al.(2018)Miyato, Kataoka, Koyama, and
Yoshida]{miyato2018spectral}
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
\newblock Spectral normalization for generative adversarial networks.
\newblock \emph{arXiv preprint arXiv:1802.05957}, 2018.
\bibitem[Bloomfield and Steiger(1983)]{bloomfield1983least}
Peter Bloomfield and William~L Steiger.
\newblock \emph{Least absolute deviations: theory, applications, and
algorithms}.
\newblock Springer, 1983.
\bibitem[Krizhevsky et~al.(2009)Krizhevsky, Hinton,
et~al.]{krizhevsky2009learning}
Alex Krizhevsky, Geoffrey Hinton, et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock Technical report, Citeseer, 2009.
\bibitem[Kingma and Ba(2014)]{Adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{arXiv preprint arXiv:1412.6980}, 2014.
\bibitem[Radford et~al.(2015)Radford, Metz, and Chintala]{DCGAN}
Alec Radford, Luke Metz, and Soumith Chintala.
\newblock Unsupervised representation learning with deep convolutional
generative adversarial networks.
\newblock \emph{arXiv preprint arXiv:1511.06434}, 2015.
\bibitem[Heusel et~al.(2017)Heusel, Ramsauer, Unterthiner, Nessler, and
Hochreiter]{heusel2017gans}
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
Hochreiter.
\newblock Gans trained by a two time-scale update rule converge to a local nash
equilibrium.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
6626--6637, 2017.
\bibitem[Zhang et~al.(2008)Zhang, Sun, and Tang]{cat}
Weiwei Zhang, Jian Sun, and Xiaoou Tang.
\newblock Cat head detection-how to effectively exploit shape and texture
features.
\newblock In \emph{European Conference on Computer Vision}, pages 802--816.
Springer, 2008.
\bibitem[Vapnik and Vapnik(1998)]{vapnik1998statistical}
V~Vapnik and Vlamimir Vapnik.
\newblock Statistical learning theory.
\newblock \emph{Wiley}, 1998.
\bibitem[Mount(2015)]{mount2015sure}
John Mount.
\newblock How sure are you that large margin implies low vc dimension?
\newblock \emph{Win-Vector Blog}, 2015.
\bibitem[Rudin(1991)]{rudin1991functional}
Walter Rudin.
\newblock Functional analysis, mcgrawhill.
\newblock \emph{Inc, New York}, 1991.
\end{thebibliography}