The Cramer Distance as a Solution to Biased Wasserstein Gradients
Extracted source text · arXiv 1705.10743
\documentclass{article}
%
%
%
\usepackage[final]{arxiv}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage[utf8]{inputenc} %
\usepackage[T1]{fontenc} %
\usepackage{hyperref} %
\usepackage{url} %
\usepackage{booktabs} %
\usepackage{amsfonts} %
\usepackage{nicefrac} %
%
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{mathrsfs}
\usepackage{amsthm}
\usepackage{xcolor}
\usepackage{bm}
%
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{enumitem}
\newcommand{\beq}{\begin{equation}}
\newcommand{\eeq}{\end{equation}}
\newcommand{\beqa}{\begin{eqnarray}}
\newcommand{\eeqa}{\end{eqnarray}}
\newcommand{\beqan}{\begin{eqnarray*}}
\newcommand{\eeqan}{\end{eqnarray*}}
\renewcommand{\P}{\mathbb{P}}
\newcommand{\E}{\mathbb{E}}
%
\def \dive {\mathbf{d}}
\def \wass {w}
\def \ldive {l}
\newcommand{\ra}[1]{\renewcommand{\arraystretch}{#1}}
\include{notation}
\newcommand{\minus}{\scalebox{0.75}[1.0]{$-$}}
\newcommand{\textblue}[1]{\textcolor{blue}{#1}}
\newcommand{\highlight}[1]{\textbf{\textblue{#1}}}
\def \cramer {Cram{\'e}r}
\def \Deq {\overset{D}{=}}
\def \cTpi {\cT^\pi}
\def \dx {\textrm{d} x}
\def \du {\textrm{d} u}
\def \tx {\tilde x}
\def \sgn {\text{sgn}}
%
\def \Qtheta {Q_\theta}
\def \gradtheta {{\grad_\theta}}
\def \batch {\mathbf{X}_m}
\newcommand{\errfn}[1]{\textnormal{err}(x, y, #1)}
\def \htheta {\hat \theta}
\title{The Cramer Distance as a Solution to Biased Wasserstein Gradients}
% These metrics are making me thirsty!
\author{Marc G. Bellemare\textsuperscript{1},
Ivo Danihelka\textsuperscript{1,3},
Will Dabney\textsuperscript{1},
Shakir Mohamed\textsuperscript{1} \AND
Balaji Lakshminarayanan\textsuperscript{1},
Stephan Hoyer\textsuperscript{2},
R\'emi Munos\textsuperscript{1}\\
\textsuperscript{1}Google DeepMind, London UK,
\textsuperscript{2}Google\\
\textsuperscript{3}CoMPLEX, Computer Science, UCL\\
%
%
\texttt{\{bellemare,danihelka,wdabney,shakir,balajiln,shoyer,munos\}@google.com} \\
}
\begin{document}
\maketitle
\begin{abstract}
The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes.
The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling.
%
In this paper we describe three natural properties of probability divergences that reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients. The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third. We provide empirical evidence
suggesting that this is a serious issue in practice. Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cram\'er distance. We show that the Cram\'er distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences. To illustrate the relevance of the Cram\'er distance in practice we design a new algorithm, the Cram{\'e}r Generative Adversarial Network (GAN), and show that it performs significantly better than the related Wasserstein GAN.
\end{abstract}
\section{Introduction}
In machine learning, the Kullback-Leibler (KL) divergence is perhaps the most common way of assessing how well a probabilistic model explains observed data. Among the reasons for its popularity is that it is directly related to maximum likelihood estimation and is easily optimized. However, the KL divergence suffers from a significant limitation: it does not take into account how close two outcomes might be, but only their relative probability. This closeness can matter a great deal: in image modelling, for example, perceptual similarity is key \citep{rubner2000earth,gao2016distributionally}. Put another way, the KL divergence cannot reward a model that ``gets it almost right''.
To address this limitation, researchers have turned to the Wasserstein metric, which does incorporate the underlying geometry between outcomes. The Wasserstein metric can be applied to distributions with non-overlapping supports, and has good out-of-sample performance \citep{esfahani2015data}. Yet, practical applications of the Wasserstein distance, especially in deep learning, remain tentative. In this paper we provide a clue as to why that might be: estimating the Wasserstein metric from samples yields biased gradients, and may actually lead to the wrong minimum. This precludes using stochastic gradient descent (SGD) and SGD-like methods, whose fundamental mode of operation is sample-based, when optimizing for this metric.
%
%
%
%
%
%
%
As a replacement we propose the Cram\'er distance \citep{szekely02estatistics, rizzo2016energy}, also known as the continuous ranked probability score in the probabilistic forecasting literature \citep{gneiting07strictly}. The Cram\'er distance, like the Wasserstein metric, respects the underlying geometry but also has unbiased sample gradients. To underscore our theoretical findings, we demonstrate a significant quantitative difference between the two metrics when employed in typical machine learning scenarios: categorical distribution estimation, regression, and finally image generation.
\section{Probability Divergences and Metrics}
In this section we provide the notation to mathematically distinguish the Wasserstein metric (and later, the Cram\'er distance) from the Kullback-Leibler divergence and probability distances such as the total variation.
The experimentally-minded reader may choose to skip to Section \ref{sec:impact_wasserstein_bias}, where we show that minimizing the sample Wasserstein loss leads to significant degradation in performance.
Let $P$ be a probability distribution over $\bR$. When $P$ is continuous, we will assume it has density $\mu_P$. The expectation of a function $f : \bR \to \bR$ with respect to $P$ is
\begin{equation*}
\expect_{x \sim P} f(x) := \int_{-\infty}^\infty f(x) P(\dx) = \left \{
\begin{array}{ll}
\int f(x) \mu_P(x) \dx & \text{if $P$ is continuous, and} \vspace{0.5em}\\
\sum f(x) P(x) & \text{if $P$ is discrete}.
\end{array} \right .
\end{equation*}
We will suppose all expectations and integrals under consideration are finite. We will often associate $P$ to a random variable $X$, such that for a subset of the reals $A \subseteq \bR$, we have $\Pr \{ X \in A \} = P(A)$. The (cumulative) distribution function of $P$ is then
\begin{equation*}
F_P(x) := \Pr \{ X \le x \} = \int_{-\infty}^x P(dx) .
\end{equation*}
Finally, the inverse distribution function of $P$, defined over the interval $(0, 1]$, is
\begin{equation*}
F^{-1}_P(u) := \inf \{ x : F_P(x) = u \}.
\end{equation*}
\subsection{Divergences and Metrics}
Consider two probability distributions $P$ and $Q$ over $\bR$. A \emph{divergence} $\dive$ is a mapping $(P, Q) \mapsto \bR^+$ with $\dive(P, Q) = 0$ if and only if $P = Q$ almost everywhere. A popular choice is the Kullback-Leibler (KL) divergence
\begin{equation*}
\KL(P \cdbar Q) := \int_{-\infty}^\infty \log \frac{P(\dx)}{Q(\dx)} P(\dx),
\end{equation*}
with $\KL(P \cdbar Q) = \infty$ if $P$ is not absolutely continuous w.r.t. $Q$. The KL divergence, also called relative entropy, measures the amount of information needed to encode the change in probability from $Q$ to $P$ \citep{cover91elements}.
A \emph{probability metric} is a divergence which is also symmetric ($\dive(P, Q) = \dive(Q, P)$) and respects the triangle inequality: for any distribution $R$, $\dive(P, Q) \le \dive(P, R) + \dive(R, Q)$. We will use the term \emph{probability distance} to mean a symmetric divergence satisfying the relaxed triangle inequality $\dive(P, Q) \le c \left [ \dive(P, R) + \dive(R, Q) \right ]$ for some $c \ge 1$.
%
%
We will first study the $p$-Wasserstein metrics $\wass_p$ \citep{dudley2002real}.
For $1 \le p < \infty$, a practical definition is through the inverse distribution functions of $P$ and $Q$:
\begin{equation}
\wass_p(P, Q) := \left ( \int_0^1 \big |F^{-1}_P(u) - F^{-1}_Q(u)\big |^p \du \right )^{1/p}.\label{eqn:primal_wasserstein}
\end{equation}
We will sometimes find it convenient to deal with the $p^\mathit{th}$ power of the metric, which we will denote by $\wass_p^p$; note that $\wass_p^p$ is not a metric proper, but is a probability distance.
%
We will be chiefly concerned with the 1-Wasserstein metric, which is most commonly used in practice.
The 1-Wasserstein metric has a dual form which is theoretically convenient and which we mention here for completeness. %
Define $\bF_\infty$ to be the class of $1$-Lipschitz functions. Then
\begin{equation}\label{eqn:ipm_wasserstein}
\wass_1(P, Q) := \sup_{f \in \bF_\infty} \big | \expect_{x \sim P} f(x) - \expect_{x \sim Q} f(x) \big |.
\end{equation}
This is a special case of the celebrated Monge-Kantorovich duality \citep{rachev13methods}, and is the integral probability metric (IPM) with function class $\bF_\infty$ \citep{muller97integral}. We invite the curious reader to consult these two sources as a starting point on this rich topic.
\subsection{Properties of a Divergence}
As noted in the introduction, the fundamental difference between the KL divergence and the Wasserstein metric is that the latter is sensitive not only to change in probability but also to the geometry of possible outcomes. %
To capture this notion we now introduce the concept of an \emph{ideal divergence}.
Consider a divergence $\dive$, and for two random variables $X, Y$ with distributions $P, Q$ write $\dive(X, Y) := \dive(P, Q)$. We say that $\dive$ is \emph{scale sensitive} (of order $\beta$), i.e. it has property \textbf{(S)}, if there exists a $\beta > 0$ such that for all $X$, $Y$, and a real value $c > 0$,
\begin{equation}
\dive(cX, cY) \le |c|^\beta \dive(X, Y). \tag{\textbf{S}}
\end{equation}
A divergence $\dive$ has property \textbf{(I)}, i.e. it is \emph{sum invariant}, if whenever $A$ is independent from $X$, $Y$
\begin{equation}
\dive(A + X, A + Y) \le \dive(X, Y). \tag{\textbf{I}}
\end{equation}
Following \citet{zolotarev76metric}, an ideal divergence $\dive$ is one that possesses both (S) and (I).\footnote{Properties (S) and (I) are called \emph{regularity} and \emph{homogeneity} by \citeauthor{zolotarev76metric}; we believe our choice of terms is more machine learning-friendly.}
%
We can illustrate the sensitivity of ideal divergences to the value of outcomes by considering Dirac functions $\delta_x$ at different values of $x$. If $\dive$ is scale sensitive of order $\beta = 1$ then the divergence $\dive(\delta_0, \delta_{1/2})$ can be no more than half the divergence $\dive(\delta_0, \delta_1)$. If $\dive$ is \emph{sum invariant}, then the divergence of $\delta_0$ to $\delta_{1}$ is equal to the divergence of the same distributions shifted by a constant $c$, i.e. of $\delta_{c}$ to $\delta_{1 + c}$.
%
As a concrete example of the importance of these properties, \citet{bellemare17distributional} recently demonstrated the importance of ideal metrics in reinforcement learning, specifically their role in providing the contraction property for a distributional form of the Bellman update.
In machine learning we often view the divergence $\dive$ as a loss function. Specifically, let $Q_\theta$ be some distribution parametrized by $\theta$, and consider the loss $\theta \mapsto \dive(P, \Qtheta)$. We are interested in minimizing this loss, that is finding $\theta^* := \argmin_\theta \dive(P, Q_\theta)$. We now describe a third property based on this loss, which we call \emph{unbiased sample gradients}.
Let $\batch := X_1, X_2, \dots, X_m$ be independent samples from $P$ and define the empirical distribution $\hat P_m := \hat P_m(\batch) := \tfrac{1}{m} \sum_{i=1}^m \delta_{X_i}$ (note that $\hat P_m$ is a random quantity). From this, define the \emph{sample loss} $\theta \mapsto \dive(\hat P_m, \Qtheta)$. We say that $\dive$ has unbiased sample gradients when the expected gradient of the sample loss equals the gradient of the true loss for all $P$ and $m$:
\begin{equation}
\expect_{\batch \sim P} \gradtheta \dive(\hat P_m, \Qtheta) = \gradtheta \dive(P, \Qtheta) \tag{\textbf{U}} .
\label{eqn:unbiased_sample_gradients}
\end{equation}
The notion of unbiased sample gradients is ubiquitous in machine learning and in particular in deep learning. Specifically, if a divergence $\dive$ does not possess (U) then minimizing it with stochastic gradient descent may not converge, or it may converge to the wrong minimum. Conversely, if $\dive$ possesses (U) then we can guarantee that the distribution which minimizes the expected sample loss
is $Q = P$.
In the probabilistic forecasting literature, this makes $\dive$ a \emph{proper scoring rule} \citep{gneiting07strictly}.
%
%
We now characterize the KL divergence and the Wasserstein metric in terms of these properties. As it turns out, neither simultaneously possesses both (U) and (S).
\begin{prop}\label{prop:kl_prop}
The KL divergence has unbiased sample gradients (U), but is not scale sensitive (S).
\end{prop}
\begin{prop}\label{prop:wasserstein_prop}
The Wasserstein metric is ideal (I, S), but does not have unbiased sample gradients.
\end{prop}
We will provide a proof of the bias in the sample Wasserstein gradients just below; the proof of the rest and later results are provided in the appendix.
%
\section{Bias in the Sample Gradient Estimates of the Wasserstein Distance}
%
%
In this section we give theoretical evidence of serious issues with gradients of the sample Wasserstein loss. We will consider a simple Bernoulli distribution $P$ with parameter $\theta^*\in (0,1)$, which we would like to estimate from samples. Our model is $\Qtheta$, a Bernoulli distribution with parameter $\theta$. We study the behaviour of stochastic gradient descent w.r.t. $\theta$ over the sample Wasserstein loss, specifically using the $p^{th}$ power of the metric (as is commonly done to avoid fractional exponents).
Our results build on the example given by \citet{bellemare17distributional}, whose result is for $\theta^* = \tfrac{1}{2}$ and $m=1$.
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
Consider the estimate $\grad_\theta \wass^p_p(\hat P_m, Q_\theta)$ of the gradient $\grad_\theta \wass^p_p(P, Q_\theta)$.
%
We now show that even in this simplest of settings, this estimate is biased, and we exhibit a lower bound on the bias for any value of $m$.
Hence the Wasserstein metric does not have property (U).
More worrisome still, we show that the minimum of the expected empirical Wasserstein loss $\theta\mapsto \expects_{\batch} \big[ \wass^p_p(\hat P_m, Q_\theta)\big]$ is not the minimum of the Wasserstein loss $\theta\mapsto \wass^p_p(P, Q_\theta)$.
We then conclude that minimizing the sample Wasserstein loss by stochastic gradient descent may in general fail to converge to the minimum of the true loss.
%
\begin{thm}\label{thm:wasserstein_bias}
Let $\hat P_m=\frac 1m\sum_{i=1}^m \delta_{X_i}$ be the empirical distribution derived from $m$ independent samples $\batch = X_1,\dots, X_m$ drawn from a Bernoulli distribution $P$. Then for all $1 \le p < \infty$,
\begin{itemize}[leftmargin=4ex,topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]
\item {\bf Non-vanishing minimax bias} of the sample gradient. For any $m \ge 1$ there exists a pair of Bernoulli distributions $P$, $Q_\theta$ for which
\begin{equation*}
\Big| \expect_{\batch \sim P} \big[ \gradtheta w^p_p(\hat P_m, Q_\theta)\big] - \gradtheta w^p_p(P, Q_\theta) \Big| \ge 2e^{-2};
%
\end{equation*}
\item {\bf Wrong minimum} of the sample Wasserstein loss. The minimum of the expected sample loss $\tilde\theta = \argmin_\theta \expects_{\batch} \big[ w^p_p(\hat P_m, Q_\theta)\big]$ is in general different from the minimum of the true Wasserstein loss $\theta^*=\argmin_\theta w^p_p(P, Q_\theta)$.
\item {\bf Deterministic solutions} to stochastic problems. For any $m \ge 1$, there exists a distribution $P$ with nonzero entropy whose sample loss is minimized by a
distribution $Q_{\tilde \theta}$ with zero entropy.
\end{itemize}
\end{thm}
Taken as a whole, Theorem \ref{thm:wasserstein_bias} states that we cannot in general minimize the Wasserstein loss using naive stochastic gradient descent methods. Although our result does not imply the lack of a stochastic optimization procedure for this loss,\footnote{For example, if $P$ has finite support keeping track of the empirical distribution suffices.} we believe our result to be serious cause for concern.
We leave as an open question whether an unbiased optimization procedure exists and is practical.
%
\subsection*{Wasserstein Bias in the Literature}
Our result is surprising given the prevalence of the Wasserstein metric in empirical studies. We hypothesize that this bias exists in published results and is an underlying cause of learning instability and poor convergence often remedied to by heuristic means. For example, \citet{frogner15learning} and \citet{montavon2016wasserstein} reported the need for a mixed KL-Wasserstein loss to obtain good empirical results, with the latter explicitly discussing the issue of wrong minima when using Wasserstein gradients.
%
We remark that our result also applies to the dual \eqnref{ipm_wasserstein}, since the losses are the same. This dual was recently considered by \citet{arjovsky17wasserstein} as an alternative loss to the primal \eqnref{primal_wasserstein}.
The adversarial procedure proposed by the authors is a two time-scale process which first maximizes \eqnref{ipm_wasserstein} w.r.t $f \in \bF_\infty$ using $m$ samples, then takes a single stochastic gradient step w.r.t. $\theta$. Interestingly, this approach does seem to provide unbiased gradients as $m \to \infty$. However, the cost of a single gradient is now significantly higher, and for a fixed $m$ we conjecture that the minimax bias remains.
\section{The Cram\'er Distance}\label{sec:cramer_distance}
We are now ready to describe an alternative to the Wasserstein metric, the Cram{\'e}r distance \citep{szekely02estatistics, rizzo2016energy}. As we shall see, the Cram{\'e}r distance has the same appealing properties as the Wasserstein metric, but also provides us with unbiased sample gradients. As a result, we believe this underappreciated distance is strictly superior to the Wasserstein metric for machine learning applications.
\subsection{Definition and Analysis}
Recall that for two distributions $P$ and $Q$ over $\bR$, their (cumulative) distribution functions are respectively $F_P$ and $F_Q$. The Cram{\'e}r distance between $P$ and $Q$ is %
\begin{equation*}
\ldive^2_2(P, Q) := \int_{-\infty}^\infty (F_P(x) - F_Q(x))^2 \dx .
\end{equation*}
Note that as written, the Cram\'er distance is not a metric proper. However, its square root is, and is a member of the $\ldive_p$ family of metrics
\begin{equation*}
\ldive_p(P, Q) := \left ( \int_{-\infty}^\infty |F_P(x) - F_Q(x)|^p \dx \right )^{1/p} .
\end{equation*}
The $\ldive_p$ and Wasserstein metrics are identical at $p=1$, but are otherwise distinct. Like the Wasserstein metrics, the $\ldive_p$ metrics have dual forms as integral probability metrics \citep[see][for a proof]{dedecker07empirical}:
\begin{equation}
\ldive_p(P, Q) = \sup_{f \in \bF_q} \big | \expect_{x \sim P} f(x) - \expect_{x \sim Q} f(x) \big |, \label{eqn:cramer_ipm}
\end{equation}
where $\bF_q := \{ f : f \text{ is absolutely continuous}, \big \| \frac{\text{d} f}{\text{d}x} \big \|_q \le 1 \}$ and $q$ is the conjugate exponent of $p$, i.e. $p^{-1} + q^{-1} = 1$.\footnote{This relationship is the reason for the notation $\bF_\infty$ in the definition the dual of the 1-Wasserstein \eqnref{ipm_wasserstein}.}
It is this dual form that we use to prove that the Cram\'er distance has property (S).
\begin{thm}\label{thm:cramer_properties}
Consider two random variables $X$, $Y$, a random variable $A$ independent of $X, Y$, and a real value $c > 0$. Then for $1 \le p \le \infty$,
\begin{equation*}
\textbf{(I) } \, \ldive_p(A + X, A + Y) \le \ldive_p(X, Y) \qquad \textbf{(S) } \, \ldive_p(cX, cY) \le |c|^{1/p} \ldive_p(X, Y) .
\end{equation*}
Furthermore, the Cram\'er distance has unbiased sample gradients. That is, given $\batch := X_1, \dots, X_m$ drawn from a distribution $P$, the empirical distribution $\hat P_m := \tfrac{1}{m} \sum_{i=1}^m \delta_{X_i}$, and a distribution $\Qtheta$,
\begin{equation*}
\expect_{\batch \sim P} \gradtheta l^2_2 (\hat P_m, \Qtheta) = \gradtheta l^2_2(P, \Qtheta),
\end{equation*}
and of all the $l_p^p$ distances, only the Cram\'er ($p = 2$) has this property.
\end{thm}
We conclude that the Cram\'er distance enjoys both the benefits of the Wasserstein metric and the SGD-friendliness of the KL divergence. Given the close similarity of the Wasserstein and $l_p$ metrics, it is truly remarkable that only the Cram\'er distance has unbiased sample gradients.
The \emph{energy distance} \citep{szekely02estatistics} is a natural extension of the Cram{\'e}r distance to the multivariate case. Let $P, Q$ be probability distributions over $\bR^d$ and let $X, X'$ and $Y, Y'$ be independent random variables distributed according to $P$ and $Q$, respectively. The energy distance \citep[sometimes called the squared energy distance, see e.g.][]{rizzo2016energy} is
\begin{equation}
\cE(P, Q) := \cE(X, Y) := 2 \expect \norm{X - Y}_2 - \expect \norm{X - X'}_2 - \expect \norm{Y - Y'}_2.
\label{eqn:energy_distance}
\end{equation}
\citeauthor{szekely02estatistics} showed that, in the univariate case, $l_2^2(P, Q) = \tfrac{1}{2} \cE(P, Q)$. Interestingly enough, the energy distance can also be written in terms of a difference of expectations. For
\begin{equation*}
f^*(x) := \expect \norm{x - Y'}_2 - \expect \norm{x - X'}_2,
\end{equation*}
we find that
\begin{equation}
\cE(X, Y) = \expect f^*(X) - \expect f^*(Y).\label{eqn:energy_distance_dual}
\end{equation}
Note that this $f^*$ is \emph{not} the maximizer of the dual \eqnref{cramer_ipm}, since $\frac{1}{2} \cE$ is equal to the \emph{squared} $l_2$ metric (i.e. the Cram\'er distance).\footnote{The maximizer of \eqnref{cramer_ipm} is: $g^*(x) = \frac{f^*(x)}{\sqrt{2 (\expect f^*(X) - \expect f^*(Y))}}$ \citep[based on results by][]{gretton2012kernel}. }
Finally, we remark that $\cE$ also possesses properties (I), (S), and (U) (proof in the appendix).
\subsection{Comparison to the 1-Wasserstein Metric}\label{sec:impact_wasserstein_bias}
To illustrate how the Cram\'er distance compares to the 1-Wasserstein metric, we consider modelling the discrete distribution $P$ depicted in Figure \ref{fig:dist_comparison_toy_example} (left). Since the trade-offs between metrics are only apparent when using an approximate model, we use an underparametrized discrete distribution $\Qtheta$ which assigns the same probability to $x = 1$ and $x = 10$. That is,
\begin{equation*}
\Qtheta(0) := \Qtheta\{x = 0\} = \frac{1}{1 + 2 e^\theta} \qquad \Qtheta(1) = \Qtheta(10) = \frac{e^{\theta}}{1 + 2 e^{\theta}} .
\end{equation*}
Figure \ref{fig:dist_comparison_toy_example} depicts the distributions minimizing the various divergences under this parametrization. In particular, the Cram\'er solution is relatively close to the 1-Wasserstein solution. Furthermore, the minimizer of the sample Wasserstein loss ($m = 1$) clearly provides a bad solution (most of the mass is on 0). Note that, as implied by Theorem \ref{thm:wasserstein_bias}, the bias shown here would arise even if the distribution could be exactly represented.
\begin{figure*}
\begin{center}
\includegraphics[width=\textwidth]{figures/various-losses-toy.pdf}
\end{center}
\caption{\textbf{Leftmost.} Target distribution. One outcome ($10$) is significantly more distant than the two others ($0$, $1$). \textbf{Rest.} Distributions minimizing the divergences discussed in this paper, under the constraint $Q(1) = Q(10)$. Both Wasserstein metric and Cram\'er distance underemphasize $Q(0)$ to better match the cumulative distribution function. The sample Wasserstein loss result is for $m=1$. \label{fig:dist_comparison_toy_example}}
\end{figure*}
To further show the impact of the Wasserstein bias we used gradient descent to minimize either the true or sample losses with a fixed step-size ($\alpha = 0.001$). In the stochastic setting, at each step we construct the empirical distribution $\hat P_m$ from $m$ samples (a Dirac when $m=1$), and take a gradient step. As we are interested in finding a solution that accounts for the numerical values of outcomes, we measure the performance of each method in terms of the true 1-Wasserstein loss.
Figure \ref{fig:ordinalregression} (left) plots the resulting training curves in the 1-Wasserstein regime, with the KL and Cram\'er solutions indicated for reference (for completeness, full learning curves are provided in the appendix). We first note that, compared to the KL solution, the Cram\'er solution has significantly smaller Wasserstein distance to the target distribution. Second, for small sample sizes stochastic gradient descent fails to find reasonable solutions, and for $m=1$ even converges to a solution worse than the KL minimizer.
This small experiment highlights the cost incurred from minimizing the sample Wasserstein loss, and shows that increasing the sample size may not be sufficient to guarantee good behaviour.
%
%
%
%
%
%
\begin{figure}%
\begin{center}
%
\includegraphics[width=1\textwidth]{figures/toy_and_ordinal.pdf}
\caption{\textbf{Left.} Wasserstein distance in terms of SGD updates, minimizing the true or sample Wasserstein losses. Also shown are the distances for the KL and Cram\'er solutions. Results are averaged over $10$ random initializations, with error-bands indicating one standard deviation. \textbf{Center.} Ordinal regression on the Year Prediction MSD dataset. Learning curves report RMSE on test set. \textbf{Right.} The same in terms of sample Wasserstein loss.}
\label{fig:ordinalregression}
\end{center}
\end{figure}
\subsection*{Ordinal Regression}
We next trained a neural network in an ordinal regression task using either of the three divergences. The task we consider is the Year Prediction MSD dataset \citep{ucirepository}. In this task, the model must predict the year a song was written (from 1922 to 2011) given a 90-dimensional feature representation. In our setting, this prediction takes the form of a probability distribution. We measure each method's performance on the test set (Figure~\ref{fig:ordinalregression}) in two ways: root mean squared error (RMSE) -- the metric minimized by \citet{PB} -- and the sample Wasserstein loss. Full details on the experiment may be found in the appendix.
The results show that minimizing the sample Wasserstein loss results in significantly worse performance. By contrast, minimizing the Cram\'er distance yields the lowest RMSE and Wasserstein loss, confirming the practical importance of having unbiased sample gradients.
Naturally, minimizing for one loss trades off performance with respect to the others, and minimizing the Cram\'er distance results in slightly higher negative log likelihood than when minimizing the KL divergence (Figure \ref{fig:ordinalregression_batch_size} in appendix). We conclude that, in the context of ordinal regression where outcome similarity plays an important role, the Cram\'er distance should be preferred over either KL or the Wasserstein metric.
\section{Cram\'er GAN}\label{sec:cramer_gan}
We now consider the Generative Adversarial Networks (GAN) framework \citep{goodfellow14generative}, in particular issues arising in the Wasserstein GAN \citep{arjovsky17wasserstein},
and propose a better GAN based on the Cram\'er distance. A GAN is composed of a generative model (in our experiments, over images), called the \emph{generator}, and a trainable loss function called a discriminator or \emph{critic}.
In theory, the Wasserstein GAN algorithm requires training the critic until convergence, but this is rarely achievable: we would require a critic that is a very powerful network to approximate the Wasserstein distance well. Simultaneously, training this critic to convergence would overfit the empirical distribution of the training set, which is undesirable.
%
Our proposed loss function allows for useful learning with imperfect critics by combining the energy distance with a transformation function $h : \bR^d \to \bR^k$, where $d$ is the input dimensionality and $k = 256$ in our experiments.
The generator then seeks to minimize the energy distance of the transformed variables $\cE(h(X), h(Y))$, where $X$ is a real sample and $Y$ is a generated sample. The critic itself seeks to maximize this same distance by changing the parameters of $h$, subject to a soft constraint \citep[the gradient penalty used by][]{gulrajani2017improved}.
%
The losses used by the Cram\'er GAN algorithm is summarized in Algorithm~1, with additional design choices detailed in the appendix.
%
\begin{figure*}
\begin{center}
\begin{minipage}{0.47\textwidth}
\includegraphics[width=\linewidth]{figures/cond_celeba64_wgan.png}
\end{minipage}
\hskip 0.5cm
\begin{minipage}{0.47\textwidth}
\includegraphics[width=\linewidth]{figures/cond_celeba64_cramgan.png}
\end{minipage}
\end{center}
\caption{Generated right halves of the faces for WGAN-GP (left) and Cram\'er GAN (right). The given left halves are from CelebA 64x64 validation set \citep{liu2015faceattributes}.}
\label{fig:cond_celeba64}
\end{figure*}
\begin{figure*}
\begin{minipage}{0.60\textwidth}
\begin{algorithmic}
{\hrule height.8pt depth0pt \kern2pt}
\STATE {\bfseries Algorithm 1:} Cram\'er GAN Losses.
\vspace{0.1cm}
{\hrule height.8pt depth0pt \kern2pt}
\STATE {\bfseries Defaults:} The gradient penalty coefficient $\lambda=10$.
\STATE Sample $x_r \sim P$ a real sample.
\STATE Sample $x_g, x'_g \sim Q$ two independent generator samples.
\STATE Sample $\epsilon \sim \mathrm{Uniform}(0, 1)$ a random number.
\STATE Interpolate real and generated samples:
\STATE $\hat{x} = \epsilon x_r + (1 - \epsilon) x_g$
\STATE Define the critic:
\STATE $f(x) = \norm{h(x) - h(x'_g)}_2 - \norm{h(x)}_2$
\STATE Compute the generator loss:
\STATE $L_\mathit{g} = \norm{h(x_r) - h(x_g)}_2 + \| h(x_r) - h(x'_g) \|_2$
\STATE $\qquad \quad - \| h(x_g) - h(x'_g) \|_2$
\STATE Compute the surrogate generator loss:
\STATE $L_\mathit{surrogate} = f(x_r) - f(x_g)$
\STATE Compute the critic loss:
\STATE $L_\mathit{critic} = -L_\mathit{surrogate} + \lambda (\|\grad_{\hat{x}} f(\hat{x})\|_2 - 1)^2$
\end{algorithmic}
\vspace{0.1cm}
{\hrule height.8pt depth0pt \kern2pt}
\end{minipage}
\hskip 0.05cm
\begin{minipage}{0.38\textwidth}
\includegraphics[width=\linewidth]{figures/plot_cond_edgan_celeba64_wdistance.pdf}
\vspace{-0.5cm}
\caption{Approximate Wasserstein distances between CelebA test set and each generator. $N_u$ indicates the number critic updates per generator update.}
\label{fig:plot_cond_edgan_celeba64_wdistance}
\end{minipage}
\end{figure*}
\subsection{Cram\'er GAN Experiments}
We now show that, compared to the improved Wasserstein GAN (WGAN-GP) of \citet{gulrajani2017improved}, the Cram\'er GAN leads to more stable learning and increased diversity in the generated samples. In both cases we train generative models that predict the right half of an image given the left half; samples from unconditional models are provided in the appendix (Figure~\ref{fig:uncond_celeba64}). The dataset we use here is the CelebA $64\times64$ dataset \citep{liu2015faceattributes} of celebrity faces.
\textbf{Increased diversity.} In our first experiment, we compare the qualitative diversity of completed faces by showing three sample completions generated by either model given the left half of a validation set image (Figure \ref{fig:cond_celeba64}). We observe that the completions produced by WGAN-GP are almost deterministic. Our findings echo those of \citet{isola2016image}, who observed that \emph{``the generator simply learned to ignore the noise.''} By contrast, the completions produced by Cram\'er GAN are fairly diverse, including different hairstyles, accessories, and backgrounds. This lack of diversity in WGAN-GP is particularly concerning given that the main requirement of a generative model is that it should provide a variety of outputs.
Theorem \ref{thm:wasserstein_bias} provides a clue as to what may be happening here. We know that minimizing the sample Wasserstein loss will find the wrong minimum. In particular, when the target distribution has low entropy, the sample Wasserstein minimizer may actually be a deterministic distribution. But a good generative model of images \emph{must} lie in this ``almost deterministic'' regime, since the space of natural images makes up but a fraction of all possible pixel combinations and hence there is little per-pixel entropy. We hypothesize that the increased diversity in the Cram\'er GAN comes exactly from learning these almost deterministic predictions.
\textbf{More stable learning.}
In a second experiment, we varied the number of critic updates ($N_u$) per generator update.
To compare performance between the two architectures, we measured the loss computed by an independent WGAN-GP critic trained on the validation set, following a similar evaluation previously done by \citet{ivogan}.
%
Figure~\ref{fig:plot_cond_edgan_celeba64_wdistance} shows the independent Wasserstein critic distance between each generator and the test set during the course of training. Echoing our results with the toy experiment and ordinal regression, the plot shows that when a single critic update is used, WGAN-GP performs particularly poorly. We note that additional critic updates also improve Cram\'er GAN. This indicates that it is helpful to keep adapting the $h(x)$ transformation.
\section{Conclusion}
\label{sect:conclusion}
There are many situations in which the KL divergence, which is commonly used as a loss function in machine learning, is not suitable. The desirable alternatives, as we have explored, are the divergences that are ideal and allow for unbiased estimators: they allow geometric information to be incorporated into the optimization problem; because they are scale-sensitive and sum-invariant, they possess the convergence properties we require for efficient learning; and the correctness of their sample gradients means we can deploy them in large-scale optimization problems. Among open questions, we mention deriving an unbiased estimator that minimizes the Wasserstein distance, and variance analysis and reduction of the Cram\'er distance gradient estimate.
%
%
%
%
%
%
\bibliography{distributional_l2}
\bibliographystyle{apalike}
\clearpage
\newpage
\appendix
\section{Proofs}
\subsection{Properties of a Divergence}
\begin{proof}[Proof (Proposition \ref{prop:kl_prop} and \ref{prop:wasserstein_prop})]
The statement regarding (U) for the KL divergence is well-known, and forms the basis of most stochastic gradient algorithms for classification. \citet{chung87discounted} have shown that the total variation does not have property (S); by Pinsker's inequality, it follows that the same holds for the KL divergence. A proof of (I) and (S) for the Wasserstein metric is given by \citet{bickel81asymptotic}, while the lack of (U) is shown in the proof of Theorem~\ref{thm:wasserstein_bias}.
\end{proof}
\subsection{Biased estimate}
\begin{proof}[Proof (Theorem \ref{thm:wasserstein_bias}).]
{\bf Minimax bias:}
Consider $P={\cal B}(\theta^*)$, a Bernoulli distribution of parameter $\theta^*$ and $Q_\theta={\cal B}(\theta)$ a Bernoulli of parameter $\theta$. The empirical distribution $\hat P_m$ is a Bernoulli with parameter $\hat \theta := \tfrac{1}{m} \sum\nolimits_{i=1}^m X_i$.
Note that with $P$ and $\Qtheta$ both Bernoulli distributions, the $p^{th}$ powers of the $p$-Wasserstein metrics are equal, i.e. $w_1(P, \Qtheta) = w^p_p(P, \Qtheta)$. This gives us an easy way to prove the stronger result that all $p$-Wasserstein metrics have biased sample gradients.
The gradient of the loss $w^p_p(P,Q_\theta)$ is, for $\theta\neq\theta^*$,
\begin{equation*}
g := \grad w^p_p(P, Q_\theta) = \nabla \Big[ \big| \theta^*-\theta\big| \Big] = \sgn(\theta - \theta^*),
\end{equation*}
and similarly, the gradient of the sample loss is, for $\theta\neq\hat \theta$,
$$
\hat g := \grad w_p^p(\hat P_m, Q_\theta) = \nabla \Big[ \big| \hat \theta-\theta\big| \Big] = \sgn(\theta - \hat \theta).
$$
Notice that this estimate is biased for any $m\geq 1$ since
\begin{equation}\label{eqn:expected_sample_wasserstein_gradient}
\expect\hat g = 2 \Pr \{ \hat \theta < \theta \} - 1,
\end{equation}
which is different from $g$ for any $\theta^*\in (0,1)$. In particular for $m=1$, $\expects_P \hat g = 1 - 2 \theta^*$ does not depend on $\theta$, thus a gradient descent using a one-sample gradient estimate has no chance of minimizing the Wasserstein loss as it will converge to either $1$ or $0$ instead of $\theta^*$.
Now observe that for $m\geq 2$, and any $\theta > \tfrac{m - 1}{m}$,
\begin{equation*}
\Pr \{ \hat \theta < \theta \} = \Pr \{ \exists i \text{ s.t. } X_i = 0 \} = 1 - (\theta^*)^m,
\end{equation*}
and therefore
\begin{equation*}
\expect\hat g = 1 - 2 (\theta^*)^m.
\end{equation*}
Taking $\theta^* = \tfrac{m-1}{m}$, we find that
\begin{equation*}
g - \expect \hat g = 1 - \left [ 1 - 2(\theta^*)^m \right ] = 2\left(1 - \frac{1}{m}\right )^{m} \ge 2e^{-2} .
\end{equation*}
Thus for any $m$, there exists $P={\cal B}(\theta^*)$ and $Q_\theta={\cal B}(\theta)$ with $\theta^*=\frac{m-1}{m}<\theta<1$ such that the bias $g - \expect \hat g$ is lower-bounded by a numerical constant. Thus the {\em minimax bias} does not vanish with the number of samples $m$.
Notice that a similar argument holds for $\theta^*$ and $\theta$ being close to $0$. In both situations where $\theta^*$ is close to $0$ or $1$, the bias is non vanishing when $|\theta^*-\theta|$ is of order $\tfrac{1}{m}$. However this is even worse when $\theta^*$ is away from the boundaries. For example chosing $\theta^*=\tfrac{1}{2}$, we can prove that the bias is non vanishing even when $|\theta^*-\theta|$ is (only) of order $\tfrac{1}{\sqrt{m}}$.
Indeed, using the anti-concentration result of \cite{Veraar2010} (Proposition 2), we have that for a sequence $Y_1,\dots,Y_m$ of Rademacher random variables (i.e.~$+/-1$ with equal probability),
$$\Pr\Big(\frac 1n\sum_{i=1}^m Y_i \geq \epsilon\Big) \geq (1-m\epsilon^2)^2/3.$$
This means that for samples $X_1,\dots,X_m$ drawn from a Bernoulli ${\cal B}(\theta^*=\tfrac{1}{2})$ (i.e., $Y_i=2X_i-1$ are Rademacher), we have
$$\Pr\Big(\hat \theta \geq \theta^* + \epsilon/2\Big) \geq (1-m\epsilon^2)^2/3,$$
thus for $1/2 = \theta^*<\theta<\theta^*+1/\sqrt{8m}$ we have the following lower bound on the bias:
$$g - \expect \hat g = 2 \Pr \big( \hat \theta \geq \theta\big) \geq 1/6.$$
Thus the bias is lower-bounded by a constant (independent of $m$) when $\theta^*=\tfrac{1}{2}$ and $|\theta^*-\theta|=O(1/\sqrt{m})$.
{\bf Wrong minimum:}
From \eqnref{expected_sample_wasserstein_gradient}, we deduce that a stochastic gradient descent algorithm based on the sample Wasserstein gradient will converge to a $\tilde \theta$ such that $\Pr\{\hat \theta < \tilde \theta\}=\tfrac{1}{2}$, i.e., $\tilde\theta$ is the median of the distribution over $\hat \theta$, whereas $\theta^*$ is the mean of that distribution. Since $\hat \theta$ follows a (normalized) binomial distribution with parameters $m$ and $\theta^*$, we know that the median $\tilde\theta$ and the mean $\theta^*$ do not necessarily coincide, and can actually be as far as $\frac{1}{2m}$-away from each other. For example for any odd $m$ and any $\theta^*\in \big(\tfrac{1}{2}, \tfrac{1}{2} - \tfrac{1}{2m} \big)$ the median is $\theta^*-\tfrac{1}{2m}$.
It follows that the minimum of the expected sample Wasserstein loss (the fixed point of the stochastic gradient descent using the sample Wasserstein gradient) is different from the minimum of the true Wasserstein loss:
\begin{equation*}
\argmin_{\theta} \expects [ w^p_p(\hat P_m, Q_{\theta}) ] \neq \argmin_{\theta} [w^p_p(P, Q_{\theta})].
\end{equation*}
This is illustrated in Figure~\ref{fig:empirical.W.loss}.
\begin{figure*}
\begin{center}
\includegraphics[width=\linewidth]{figures/sample-wasserstein-loss.pdf}
\end{center}
\caption{Wasserstein loss (black curve) $\theta\mapsto |\theta^*-\theta|$ versus expected sample Wasserstein loss (red curve) $\theta\mapsto \E[|\hat \theta - \theta|]$, for different values of $m$ and $\theta^*$ and $p=1$. {\bf Left:} $m=1$, $\theta^*=0.6$. A stochastic gradient using a one-sample Wasserstein gradient estimate will converge to $1$ instead of $\theta^*$. {\bf Middle:} $m=6$, $\theta^*=0.6$. The minimum of the expected sample Wasserstein loss is the median of $\hat \theta$ which is here $\tilde\theta=\tfrac{2}{3}\neq\theta^*=0.6$. {\bf Right:} $m=5$, $p=0.9$. The minimum of the expected sample Wasserstein is $\tilde\theta=1$ and not $\theta^*=0.9$.}
\label{fig:empirical.W.loss}
\end{figure*}
Notice that the fact that the minima of these losses differ is worrisome as it means that minimizing the sample Wasserstein loss using (finite) samples will not converge to the correct solution.
\textbf{Deterministic solutions:} Consider the specific case where $(1/2)^{1/n} < \theta^* <1$ (illustrated in the right plot of Figure~\ref{fig:empirical.W.loss}). Then the expected sample gradient $\nabla \expects [ w^p_p(\hat P_m, Q_{\theta^*}) ] = \E \hat g = 1- 2(\theta^*)^n < 0$ for any $\theta$, so a gradient descent algorithm will converge to $1$ instead of $\theta^*$. Notice that a symmetric argument applies for $\theta^*$ close to $0$.
In this simple example, minimizing the sample Wasserstein loss may lead to degenerate solutions (i.e., deterministic) when our target distributions have low (but not zero) entropy.
\end{proof}
\subsection{Consistency of the sample $1$-Wasserstein gradient}
We provide an additional result here showing that the sample $1$-Wasserstein gradient converges to the true gradient as $m \to \infty$.
\begin{thm}\label{thm:consistency}
Let $P$ and $Q_\theta$ be probability distributions, with $Q_\theta$ parametrized by $\theta$. Assume that the set $\big\{ x\in X, \mbox{ such that } F_{P}(x)=F_{Q_\theta}(x) \big\}$ has measure zero, and that for any $x\in X$, the map $\tilde \theta\mapsto F_{Q_{\tilde \theta}}(x)$ is differentiable in a neighborhood ${\cal V}(\theta)$ of $\theta$ with a uniformly bounded derivative (for $\tilde\theta\in {\cal V}(\theta)$ and $x\in X$).
Let $\hat P_m=\frac 1m \sum_{i=1}^m \delta_{X_i}$ be the empirical distribution derived from $m$ independent samples $X_1,\dots, X_m$ drawn from $P$. Then
\begin{equation*}
\lim_{m \to \infty} \grad w_1(\hat P_m, Q_\theta) = \grad w_1(P, Q_\theta), \mbox{ almost surely.}
\end{equation*}
\end{thm}
We note that the measure requirement is strictly to keep the proof simple, and does not subtract from the generality of the result.
\begin{proof}
Let $\grad := \gradtheta$. Since $p=1$ the Wasserstein distance $w_1(P,Q)$ measures the area between the curves defined by the distribution function of $P$ and $Q$, thus
$w_1(P,Q) = l_1(P,Q) = \int \big| F_{P}(x)-F_{Q}(x)\big| dx$ and
\beqan
\nabla w_1(P,Q_\theta)&=&\lim_{\Delta\rightarrow 0} \frac{w_1(P,Q_{\theta+\Delta})- w_1(P,Q_{\theta})}{\Delta}\\
&=& \lim_{\Delta\rightarrow 0} \int \frac 1{\Delta} \Big( \big| F_{P}(x)-F_{Q_{\theta+\Delta}}(x)\big| - \big| F_{P}(x)-F_{Q_\theta}(x)\big| \Big) dx.
\eeqan
Now since we have assumed that for any $x\in X$, the map $\theta\mapsto F_{Q_{\theta}}(x)$ is differentiable in a neighborhood ${\cal V}(\theta)$ of $\theta$ and its derivative is uniformly (over ${\cal V}(\theta)$ and $x$) bounded by $M$, we have
\beqan
\frac 1{\Delta} \Big| \big| F_{P}(x)-F_{Q_{\theta+\Delta}}(x)\big| - \big| F_{P}(x)-F_{Q_\theta}(x)\big| \Big|
&\leq & \frac 1{\Delta} \big| F_{Q_{\theta+\Delta}}(x) - F_{Q_\theta}(x)\big| \leq M.
\eeqan
Thus the dominated convergence theorem applies and
\beqan
\nabla w_1(P,Q_\theta)&=&\int \lim_{\Delta\rightarrow 0} \frac 1{\Delta} \Big( \big| F_{P}(x)-F_{Q_{\theta+\Delta}}(x)\big| - \big| F_{P}(x)-F_{Q_\theta}(x)\big| \Big) dx\\
&=& \int \nabla \big| F_{P}(x)-F_{Q_\theta}(x)\big| dx \\
&=& \int \sgn\big( F_{P}(x)-F_{Q_\theta}(x)\big) \nabla F_{Q_\theta}(x) dx,
\eeqan
since we have assumed that the set of $x\in X$ such that $F_{P}(x)=F_{Q_\theta}(x)$ has measure zero.
Now, using the same argument for $w_1(\hat P_m,Q_\theta)$ we deduce that
\beqan
\nabla w_1(\hat P_m,Q_\theta)&=&\int \underbrace{ \lim_{\Delta\rightarrow 0} \frac 1{\Delta} \Big( \big| F_{\hat P_m}(x)-F_{Q_{\theta+\Delta}}(x)\big| - \big| F_{\hat P_m}(x)-F_{Q_\theta}(x)\big| \Big)}_{A(x)} dx.
\eeqan
Let us decompose this integral over $X$ as the sum of two integrals, one over $X\setminus \Omega_m$ and the other one over $\Omega_m$, where $\Omega_m = \big\{x\in X, F_{\hat P_m}(x)=F_{Q_\theta}(x)\big\}$. We have
$$\int_{X\setminus \Omega_m} A(x) dx = \int_{X\setminus \Omega_m} \sgn\big( F_{\hat P_m}(x)-F_{Q_\theta}(x)\big) \nabla F_{Q_\theta}(x) dx,$$
and
\beqan
\Big| \int_{\Omega_m} A(x) dx \Big|
&\leq & \int_{\Omega_m} \lim_{\Delta\rightarrow 0} \frac 1{\Delta} \Big( \big| F_{Q_{\theta+\Delta}}(x)-F_{Q_\theta}(x)\big| \Big) dx \\
&\leq & M |\Omega_m |.
\eeqan
Now from the strong law of large numbers, we have that for any $x$, the empirical cumulative distribution function $F_{\hat P_m}(x)$ converges to the cumulative distribution $F_{P}(x)$ almost surely. We deduce that $\Omega_m$ converges to the set $\big\{x, F_{P}(x)=F_{Q_\theta}(x)\big\}$ which has measure zero, thus $|\Omega_m |\rightarrow 0$ and
\beqan
\lim_{m\rightarrow\infty} \nabla w_1(\hat P_m,Q_\theta)&=&\lim_{m\rightarrow\infty} \int_{X} \sgn\big( F_{\hat P_m}(x)-F_{Q_\theta}(x)\big) \nabla F_{Q_\theta}(x) dx.
\eeqan
Now, since $|\nabla F_{Q_\theta}(x)|\leq M$, we can use once more the dominated convergence theorem to deduce that
\begin{align*}
\lim_{m\rightarrow\infty} \nabla w_1(\hat P_m,Q_\theta) &=
\int_{X} \lim_{m\rightarrow\infty} \sgn\big( F_{\hat P_m}(x)-F_{Q_\theta}(x)\big) \nabla F_{Q_\theta}(x) dx\\
&= \int_{X} \sgn\big( F_{P}(x)-F_{Q_\theta}(x)\big) \nabla F_{Q_\theta}(x) dx\\
&= \nabla w_1(P,Q_\theta). \qedhere
\end{align*}
\end{proof}
The following lemma will be useful in proving that the Cram\'er distance has property (U).
\begin{lem}\label{lem:expected_empirical_cdf}
Let $\batch := X_1, \dots, X_m$ be independent samples from $P$, and let $\hat P_m := \tfrac{1}{m} \sum\nolimits_i \delta_{X_i}$. Then
\begin{equation*}
\expect_{\batch \sim P} F_{\hat P_m}(x) = F_P(x) .
\end{equation*}
\end{lem}
\begin{proof}
Because the $X_i$'s are independent,
\begin{align*}
F_{\hat P_m}(x) &= \int^x_{-\infty} \hat P_m(\dx) = \tfrac{1}{m} \sum_{i=1}^m \indic{X_i \le x} .
\end{align*}
Now, taking the expectation w.r.t. $\batch$,
\begin{align*}
\expect_{\batch \sim P} F_{\hat P_m}(x) &= \expect_{\batch \sim P} \tfrac{1}{m} \sum_{i=1}^m \indic{X_i \le x} \\
&= \tfrac{1}{m} \sum_{i=1}^m \expect_{X_i \sim P} \indic{X_i \le x} \\
&= \tfrac{1}{m} \sum_{i=1}^m \Pr \{ X_i \le x \} \\
&= F_P(x),
\end{align*}
since the $X_i$ are identically distributed according to $P$.
\end{proof}
\begin{proof}[Proof (Theorem \ref{thm:cramer_properties})]
We will prove that $\ldive_p$ has properties (I) and (S) for $p \in [1, \infty)$; the case $p = \infty$ follows by a similar argument. Begin by observing that
\begin{align*}
F_{cX}(x) &= Pr\{ cX \le x \} \\
&= Pr\left\{X \le \frac{x}{c}\right\} \\
&= F_X\left(\frac{x}{c}\right).
\end{align*}
Then we may rewrite $\ldive_p^p(cX, cY)$ as
\begin{align*}
\ldive_p^p(cX, cY) &= \int_{-\infty}^{\infty} \left | F_X\left(\frac{x}{c} \right) - F_Y \left(\frac{x}{c} \right) \right |^p \dx \\
&\overset{(a)}{=} c \int_{-\infty}^{\infty} \Big | F_X\left(z \right) - F_Y \left(z \right) \Big |^p \textrm{d} z,
%
%
\end{align*}
where ($a$) uses a change of variables $z = x/c$. Taking both sides to the power $1/p$ proves that the $\ldive_p$ metric possesses property (S) of order $1/p$. For (I), we use the IPM formulation \eqnref{cramer_ipm}:
\begin{align*}
\ldive_p(A + X, A + Y) &= \sup_{f \in \cF_q} \Big | \expect_{A + X} f(x) - \expect_{A + Y} f(y) \Big | \\
&\overset{(a)}{=} \sup_{f \in \cF_q} \Big | \expects_A \expects_{X} f(x + a) - \expects_A \expects_{Y} f(y + a) \Big | \\
&= \sup_{f \in \cF_q} \Big | \expects_A \big [ \expects_{X} f(x + a) - \expects_{Y} f(y + a) \big] \Big | \\
&\overset{(b)}{\le} \expects_A \sup_{f \in \cF_q} \Big | \expects_{X} f(x + a) - \expects_{Y} f(y + a) \Big |,
\end{align*}
where ($a$) is by independence of $A$ and $X$, $Y$, and ($b$) is by Jensen's inequality. Next, recall that $\cF_q$ is the set of absolutely continuous functions whose derivative has bounded $L_q$ norm. Hence if $f \in \cF_q$, then also for all $a$ the translate $g_a(x) := f(x + a)$ is also in $\cF_q$. Therefore,
\begin{align*}
\ldive_p(A + X, A + Y) &\le \expects_A \sup_{f \in \cF_q} \Big | \expects_{X} f(x + a) - \expects_{Y} f(y + a) \Big | \\
&= \expects_A \sup_{g \in \cF_q} \Big | \expects_{X} g(x) - \expects_{Y} g(y) \Big | \\
&= \sup_{g \in \cF_q} \Big | \expects_{X} g(x) - \expects_{Y} g(y) \Big | \\
&= \ldive_p(X, Y). \qedhere
\end{align*}
Now, to prove \textbf{(U)}. Here we make use of the introductory requirement that ``all expectations under consideration are finite.'' Specifically, we require that the mean under $P$, $\expect_{x \sim P} [x]$, is well-defined and finite, and similarly for $\Qtheta$. In this case,
\begin{equation}
\expect_{x \sim P} [x] = \int_0^\infty (1 - F_P(x)) \dx \, - \int_{-\infty}^0 F_P(x) \dx .\label{eqn:light_tails}
\end{equation}
This mild requirement guarantees that the tails of the distribution function $F_P$ are light enough to avoid infinite Cram\'er distances and expected gradients (a similar condition was set by \citet{dedecker07empirical}). Now, by definition,
\begin{align*}
\gradtheta \ldive_2^2(P, \Qtheta) &= \gradtheta \int_{-\infty}^{\infty} \left ( F_{\Qtheta}(x) - F_P(x) \right )^2 \dx \\
&\overset{(a)}{=} \int_{-\infty}^{\infty} \gradtheta \left ( F_{\Qtheta}(x) - F_P(x) \right )^2 \dx \\
&= \int_{-\infty}^{\infty} 2 \left ( F_{\Qtheta}(x) - F_P(x) \right ) \gradtheta F_{\Qtheta}(x) \dx \\
&\overset{(b)}{=} \int_{-\infty}^{\infty} 2 \left ( F_{\Qtheta}(x) - \expects_{\batch} F_{\hat P_m}(x) \right ) \gradtheta F_{\Qtheta}(x) \dx \\
&= \int_{-\infty}^{\infty} 2 \expects_{\batch} \left ( F_{\Qtheta}(x) - F_{\hat P_m}(x) \right ) \gradtheta F_{\Qtheta}(x) \dx \\
&\overset{(c)}{=} \expects_{\batch} \int_{-\infty}^{\infty} 2 \left ( F_{\Qtheta}(x) - F_{\hat P_m}(x) \right ) \gradtheta F_{\Qtheta}(x) \dx \\
&= \expects_{\batch} \gradtheta \ldive_2^2(\hat P_m, \Qtheta),
\end{align*}
where (a) follows from the hypothesis \eqnref{light_tails} (the convergence of the squares follows from the convergence of the ordinary values), (b) follows from Lemma \ref{lem:expected_empirical_cdf} and (c) follows from Fubini's theorem, again invoking \eqnref{light_tails}.
Finally, we prove that \textbf{of all the $\ldive_p^p$ distances ($1 \le p \le \infty$) only the \cramer\ distance, $l_2^2$, has the (U) property.}
Without loss of generality, let us suppose $P$ is not a Dirac, and further suppose that for any $\batch \sim P$, $F_{Q_\theta}(x) \ge F_{\hat P_m}(x)$ everywhere. For example, when ${Q_\theta}$ has bounded support we can take $P$ to be a sufficiently translated version of ${Q_\theta}$, such that the two distributions' supports do not overlap.
We have already established that the 1-Wasserstein does not have the (U) property, and is equivalent to $\ldive_p^p$ for $p = 1$. We will thus assume that $p > 1$, and also that $p < \infty$, the latter being recovered through standard limit arguments. Begin with the gradient for $\ldive_p^p(P, {Q_\theta})$,
\begin{align*}
\gradtheta \ldive_p^p(P, {Q_\theta}) &= \gradtheta \int_{-\infty}^{\infty} \big | F_{Q_\theta}(x) - F_P(x) \big |^p \dx \\
&\overset{(a)}{=} p \int_{-\infty}^{\infty} \big (F_{Q_\theta}(x) - F_P(x) \big )^{p-1} \gradtheta F_{Q_\theta}(x) \dx \\
&= p \int_{-\infty}^{\infty} \phi_p(F_{Q_\theta}(x) - F_P(x)) \gradtheta F_{Q_\theta}(x) \dx \\
&= p \int_{-\infty}^{\infty} \phi_p\Big (\expects_{\batch} \big ( F_{Q_\theta}(x) - F_{\hat P_m}(x) \big ) \Big ) \gradtheta F_{Q_\theta}(x) \dx,
\end{align*}
for $\phi_p(z) = z^{p-1}$; in (a) we used the same argument as in Theorem \ref{thm:consistency}.
Now, $\phi_p$ is convex on $[0, \infty)$ when $p \ge 2$ and concave on the same interval when $1 < p < 2$. From Jensen's inequality we know that for a convex (concave) function $\phi$ and a random variable $Z$, $\expect \phi(Z)$ is greater than (less than) or equal to $\phi(\expects Z)$, with equality if and only if $\phi$ is linear or $Z$ is deterministic. By our first assumption we have ruled out the latter. By our second assumption $F_{Q_\theta}(x) \ge F_{\hat P_m}(x)$, we can apply Jensen's inequality at every $x$ to deduce that
\begin{align*}
\expects_{\batch} \left[ \gradtheta \ldive_p^p(\hat P_m, {Q_\theta}) \right] < \gradtheta \ldive_p^p(P, {Q_\theta}),\ \quad & \text{if $1 < p < 2$},\\
\expects_{\batch} \left[ \gradtheta \ldive_p^p(\hat P_m, {Q_\theta}) \right] > \gradtheta \ldive_p^p(P, {Q_\theta}),\ \quad & \text{if $p > 2$},\\
\expects_{\batch} \left[ \gradtheta \ldive_p^p(\hat P_m, {Q_\theta}) \right] = \gradtheta \ldive_p^p(P, {Q_\theta}),\ \quad & \text{if $p = 2$}.
\end{align*}
We conclude that of the $l_p^p$ distances, only the Cram\'er distance has unbiased sample gradients.$\hfill \square$
\end{proof}
\begin{prop}
The energy distance $\cE(P, Q)$ has properties (I), (S), and (U).
\end{prop}
\begin{proof}
As before, write $\cE(X, Y) := \cE(P, Q)$. Recall that
\begin{equation*}
\cE(X, Y) = 2 \expect \norm{X - Y}_2 - \expect \norm{X - X'}_2 - \expect \norm{Y - Y'}_2 .
\end{equation*}
Consider a random variable $A$ independent of $X$ and $Y$, and let $A'$ be an independent copy of $A$. Then
\begin{align*}
\cE(A + X, A + Y) &= 2 \expect \norm{A + X - A - Y}_2 - \expect \norm{A + X - A' - X'}_2 - \expect \norm{A + Y - A' - Y'}_2 \\
&\le 2 \expect \norm{X - Y}_2 - \expect \norm{X - X'}_2 - \expect \norm{Y - Y'}_2 - 2 \expect \norm{A - A'}_2 \\
&\le 2 \expect \norm{X - Y}_2 - \expect \norm{X - X'}_2 - \expect \norm{Y - Y'}_2 \\
&= \cE(X, Y).
\end{align*}
This proves (I). Next, consider a real value $c > 0$. We have
\begin{align*}
\cE(cX, cY) &= 2 \expect \norm{cX - cY}_2 - \expect \norm{cX - cX'}_2 - \expect \norm{cY - cY'}_2 \\
&= 2 c \expect \norm{X - Y}_2 - c \expect \norm{X - X'}_2 - c \expect \norm{Y - Y'}_2 \\
&= c \cE(X, Y) .
\end{align*}
This proves (S). Finally, suppose that $Y$ is distributed according to $\Qtheta$ parametrized by $\theta$. Let $\batch = X_1, \dots, X_m$ be drawn from $P$, and let $\hat P_m := \tfrac{1}{m} \sum_{i=1}^m \delta_{X_i}$. Let $\hat X$ be the random variable distributed according to $\hat P_m$, and $\hat X'$ an independent copy of $\hat X$. Then
\begin{equation*}
\cE(\hat P_m, \Qtheta) = \cE(\hat X, Y) = 2 \expect \bignorm{\hat X - Y}_2 - \expect \bignorm{\hat X - \hat X'}_2 - \expect \bignorm{Y - Y'}_2 .
\end{equation*}
The gradient of the true loss w.r.t. $\theta$ is
\begin{equation}
\grad_\theta \cE(X, Y) = 2 \grad_\theta \expect \bignorm{X - Y}_2 - \grad_\theta \expect \bignorm{Y - Y'}_2 . \label{eqn:energy_distance_gradient}
\end{equation}
Now, taking the gradient of the sample loss w.r.t. $\theta$,
\begin{equation}
\grad_\theta \cE(\hat X, Y) = 2 \grad_\theta \expect \bignorm{\hat X - Y}_2 - \grad_\theta \expect \bignorm{Y - Y'}_2 . \label{eqn:energy_distance_sample_gradient}
\end{equation}
Since the second terms of the gradients match, all we need to show is that the first terms are equal, in expectation. Assuming that $\gradtheta$ and the expectation over $\batch$ commute, we write
\begin{align*}
\expect_{\batch} \grad_\theta \expect \bignorm{\hat X - Y}_2 &= \grad_\theta \expect_{\batch} \expect \bignorm{\hat X - Y}_2 \\
&= \grad_\theta \expect_{\batch} \expect \expect_{x \sim \hat P_m} \bignorm{x - Y}_2,
\end{align*}
by independence of $X$ and $Y$. But now we know that the expected empirical distribution is $P$, that is
\begin{equation*}
\expect_{\batch} \expect_{x \sim \hat P_m} \expect \bignorm{x - Y}_2 = \expect_{x \sim P} \expect \bignorm{x - Y}_2 = \expect \bignorm{X - Y}_2 .
\end{equation*}
It follows that the first terms of \eqnref{energy_distance_gradient} and \eqnref{energy_distance_sample_gradient} are also equal, in expectation w.r.t. $\batch$. Hence we conclude that the energy distance has property (U), that is
\begin{equation*}
\expect_{\batch \sim P} \grad_\theta \cE(\hat P_m, \Qtheta) = \grad_\theta \cE(P, \Qtheta) .
\end{equation*}
\end{proof}
\section{Comparison with the Wasserstein Distance}
Figure \ref{fig:ordinalregression} (left) provides learning curves for the toy experiment described in Section \ref{sec:impact_wasserstein_bias}.
\begin{figure*}
\begin{center}
\includegraphics[width=\textwidth]{figures/loss_comparison_toy.pdf}
\end{center}
\caption{Wasserstein while training to minimize different loss functions (Wasserstein, KL, Cram\'er). Averaged over $10$ random initializations. Error-bands indicate one standard deviation. Note the different y-axes.
\label{fig:loss_comparison_toy_example_appendix}}
\end{figure*}
\subsection{Ordinal Regression}
We compare the different losses on an ordinal regression task using the Year Prediction MSD dataset from \citep{ucirepository}. The task is to predict the year of a song (taking on values from 1922 to 2011), from 90-dimensional feature representation of the song.\footnote{We refer to \url{https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD} for the details.} Previous work has used this dataset for benchmarking regression performance \citep{PB}, treating the target as a continuous value. Following \citet{PB}, we train a network with a single hidden layer with 100 units and ReLU non-linearity, using SGD with 40 passes through the training data, using the standard train-test split for this dataset \citep{ucirepository}. Unlike \citep{PB}, the network outputs a probability distribution over the years (90 possible years from 1922-2011).
We train models using either the 1-Wasserstein loss, the Cram\'er loss, or the KL loss, the latter of which reduces the ordinal regression problem to a classification problem. In all cases, we compare performance for three different minibatch sizes, i.e. the number of input-target pairs per gradient step.
Note that the minibatch size only affects the gradient estimation, but has otherwise no direct relation to the number of samples $m$ previously discussed, since each sample corresponds to a different input vector.
We report results as a function of number of passes over the training data so that our results are comparable with previous work, but note that smaller batch sizes get more updates.
The results are shown in Figure~\ref{fig:ordinalregression}. Training using the Cram\'er loss results in the lowest root mean squared error (RMSE) and the final RMSE value of 8.89 is comparable to regression \citep{PB} which directly optimizes for MSE.
We further observe that minimizing the Wasserstein loss trains relatively slowly and leads to significantly higher KL loss. Interestingly, larger minibatch sizes do seem to improve the performance of the Wasserstein-based method somewhat, suggesting that there might be some beneficial bias reduction from combining similar inputs. By contrast, using with the Cram\'er loss trains significantly faster and is more robust to choice of minibatch size.
\begin{figure}%
\begin{center}
\includegraphics[width=1\textwidth]{figures/ordinal-regression-results.pdf}
\caption{Ordinal regression on the year prediction MSD dataset. Each loss function trained with various minibatch sizes. Training progress shown in terms of: \textbf{Left.} RMSE, \textbf{Middle.} Wasserstein distance, \textbf{Right.} Negative log-likelihood.}
\label{fig:ordinalregression_batch_size}
\end{center}
\end{figure}
\subsection{Image Modelling with PixelCNN}
As additional supporting material, we provide here the results of experiments on learning a probabilistic generative model on images using either the 1-Wasserstein, Cram{\'e}r, or KL loss. We trained a PixelCNN model \citep{vandenoord16pixel} on the CelebA 32x32 dataset \citep{liu2015faceattributes}, which is constituted of 202,599 images of celebrity faces.
At a high level, probabilistic image modelling involves defining a joint probability $Q_\theta$ over the space of images. PixelCNN forms this joint probability autoregressively, by predicting each pixel using a histogram distribution conditional on a probability-respecting subset of its neighbours. This kind of modelling task is a perfect setting to study Wasserstein-type losses, as there is a natural ordering on pixel intensities. This is also a setting in which full distributions are almost never available, because each prediction is conditioned on very different context; and hence we require a loss that can be optimized from single samples. Here the true losses are not available. Instead we report the sample Wasserstein loss, which is an upper bounds on the true loss \citet[proof is provided by][]{bellemare17distributional}. For the KL divergence we report the cross-entropy loss, as is typically done; the KL divergence itself corresponds to the expected cross-entropy loss minus the real distribution's (unknown) entropy.
\begin{figure*}
\begin{minipage}{0.63\textwidth}
%
\includegraphics[width=\textwidth]{figures/plot_scorecnn_celeba32.pdf}
\end{minipage}
\begin{minipage}{0.2\textwidth}
\ra{1.2}
\begin{tabular}{@{}llll@{}} \toprule
min. & \multicolumn{3}{c}{test loss} \\
\cmidrule{2-4}
& KL & Cram\'er & Wass. \\
\midrule
KL & \highlight{3.76} & 3.55 & 7.10 \\
Cram\'er & 10.09 & \highlight{3.51} & \highlight{7.02} \\
Wass. & 4016 & 15.99 & 16.00 \\
\bottomrule
\end{tabular}
\end{minipage}
\caption{\textbf{Left, middle.} Sample Wasserstein and cross-entropy loss curves on the CelebA validation data set. \textbf{Right.} Test loss at the end of training, in function of loss minimized (see text for details).\label{fig:pixelcnn_celeba_losses}}
\end{figure*}
Figure \ref{fig:pixelcnn_celeba_losses} shows, as in the toy example, that minimizing the Wasserstein distance by means of stochastic gradient fails. The Cram{\'e}r distance, on the other hand, is as easily minimized as the KL and in fact achieves lower Wasserstein and Cram{\'e}r loss. We note that the resulting KL loss is higher than when directly minimizing the KL, reflecting the very real trade-off of using one loss over another. We conclude that in the context of learning an autoregressive image model, the Cram{\'e}r should be preferred to the Wasserstein metric.
\section{Cram\'er GAN}
\begin{figure*}[t]
\begin{center}
\begin{minipage}{0.47\textwidth}
\includegraphics[width=\linewidth]{figures/imagenet64_wgan_mask.png}
\end{minipage}
\hskip 0.5cm
\begin{minipage}{0.47\textwidth}
\includegraphics[width=\linewidth]{figures/imagenet64_cramgan_mask.png}
\end{minipage}
\end{center}
\caption{Generated right halves for WGAN-GP (left) and Cram\'er GAN (right) for left halves from the validation set of Downsampled ImageNet 64x64 \citep{vandenoord16pixel}. The low diversity in WGAN-GP samples is consistent with the observations of \citet{isola2016image}: \emph{``the generator simply learned to ignore
the noise.''}}
\label{fig:cond_imagenet64}
\end{figure*}
\subsection{Loss Function Details}
Our critic has a special form:
\begin{align*}
f(x) = \expect_{Y' \sim Q} \|h(x) - h(Y')\|_2 - \expect_{X' \sim P} \|h(x) - h(X')\|_2
\end{align*}
where $Q$ is the generator and $P$ is the target distribution.
The critic has trainable parameters only inside the deep network used for the transformation $h$.
From \eqnref{energy_distance_dual}, we define the generator loss to be
\begin{align}
L_\mathit{g}(X, Y) = \expect_{X \sim P} [f(X)] - \expect_{Y \sim Q} [f(Y)],
\label{eqn:cramer_generator_loss}
\end{align}
as in Wasserstein GAN, except that no $\max_f$ operator is present and we can obtain unbiased sample gradients.
At the same time, to provide helpful gradients for the generator,
we train the transformation $h$ to maximize the generator loss.
Concretely, the critic seeks to maximize the generator loss while minimizing a gradient penalty:
\begin{align}
L_\mathit{critic}(X, Y) &= -L_\mathit{g}(X, Y) + \lambda \mathrm{GP} \label{eqn:cramer_critic_loss}
\end{align}
where $\mathrm{GP}$ is the gradient penalty from the original WGAN-GP algorithm \citep{gulrajani2017improved} (the penalty is given in Algorithm~1). The gradient penalty bounds the critic's outputs without using a saturating function. We chose $\lambda = 10$ from a short parameter sweep. Our training is otherwise similar to the improved training of Wasserstein GAN \citep{gulrajani2017improved}.
\subsection{Gradient Estimates for the Generator}
Recall that the energy distance is:
\begin{align*}
\cE(X, Y) &= 2 \expect_{\substack{X \sim P \\ Y \sim Q}} \norm{X - Y}_2
- \expect_{\substack{X \sim P \\ X' \sim P}} \norm{X - X'}_2
- \expect_{\substack{Y \sim Q \\ Y' \sim Q}} \norm{Y - Y'}_2
\end{align*}
If $Y$ is generated from the standard normal noise $Z \sim N(0, 1)$
by a continuously differentiable generator $Y = G(Z)$, we can use
the reparametrization trick \citep{kingma2013auto}
to compute the gradient with respect to the generator parameters:
\begin{align*}
\grad_{\theta_G} \cE(X, Y) &= 2 \expect_{\substack{X \sim P \\ Z \sim N(0, 1)}} \grad_{\theta_G} \norm{X - G(Z)}_2
- \expect_{\substack{Z \sim N(0, 1) \\ Z' \sim N(0, 1)}} \grad_{\theta_G} \norm{G(Z) - G(Z)'}_2 .
\end{align*}
\subsection{Gradient Estimates for the Transformation}
As shown in the previous section, we can obtain an unbiased gradient estimate of the generator loss \eqnref{cramer_generator_loss} from three samples: two from the generator, and one from the target distribution. However, to estimate the gradient of the \cramer~GAN loss with respect to the transformation parameters we need \emph{four} independent samples: two from the generator and two from the target distribution. In many circumstances, for example when learning conditional densities, we do not have access to two independent target samples. As an approximation, we can instead assume that $h(x) \approx 0$ for $x \sim P$. This leads to the approximate critic
\begin{align*}
f_\mathit{surrogate}(x) = \expect_{Y \sim Q} \|h(x) - h(Y)\|_2 - \|h(x)\|_2
\end{align*}
which we use to define a loss $L_\mathit{surrogate}(X, Y)$ similar to \eqnref{cramer_generator_loss} with which we train the critic. In practice, it works well to train also the generator
to minimize the $L_\mathit{surrogate}$ loss.
The whole training procedure is summarized as Algorithm~1.
\subsection{Generator Architecture}
The generator architecture is the U-Net \citep{ronneberger2015u} previously used for Image-to-Image translation \citep{isola2016image}.
We used no batch normalization and no dropout in the generator and in the critic. The network conditioned on the left half of the image and on extra 12 channels with Gaussian noise.
We generated two independent samples for a given image to compute the Cram\'er GAN loss. To be computationally fair to WGAN-GP, we trained WGAN-GP with twice the minibatch size (i.e., the Cram\'er GAN minibatch size was 64, while the WGAN-GP minibatch size was 128).
\subsection{Critic Architecture}
Our $h(x)$ transformation is a deep network with $256$ outputs (more is better). The network has the traditional deep convolutional architecture \citep{radford2015unsupervised}.
We do not use batch normalization, as it would conflict with the gradient penalty.
\section{Relation of the Energy Distance to Maximum Mean Discrepancy}
\begin{figure*}
\begin{center}
\begin{minipage}{0.47\textwidth}
\includegraphics[width=\linewidth]{figures/celeba64_flat.png}
\end{minipage}
\hskip 0.5cm
\begin{minipage}{0.47\textwidth}
\includegraphics[width=\linewidth]{figures/celeba64_edgan.png}
\end{minipage}
\end{center}
\caption{
\textbf{Left.} Generated images from a generator trained to minimize the energy distance of raw images, $\cE(X, Y)$.
\textbf{Right.} Generated images if minimizing the Cram\'er GAN loss, $\cE(h(X), h(Y))$. Both generators had the same DCGAN architecture \citep{radford2015unsupervised}.}
\label{fig:uncond_celeba64}
\end{figure*}
%
The energy distance can be generalized by replacing the 2-norms $\|x - y\|_2$ with a different distance.
\citet{szekely02estatistics} discusses the usage of $\|x - y\|_2^\alpha$. For $\alpha \in (0, 2)$, the energy distance will be zero iff $X$ and $Y$ are identically distributed. For $\alpha=2$, the energy distance degenerates to the difference of expectations: $2 \norm{\expect(X) - \expect(Y)}_2^2$.
\citet{sejdinovic2013equivalence} showed the equivalence of the energy distance and the squared maximum mean discrepancy \citep[MMD;][]{gretton2012kernel} with kernel $\minus\norm{x - y}_2$. In that sense, our loss can be viewed as squared MMD, computed for transformed variables.
Interestingly, in our experiments, $\|x - y\|_2^\alpha$ distances were more stable than distances generated by Gaussian or Laplacian kernels.
Generative Moment Matching Networks \citep{li2015generative,dziugaite2015training} were previously used to train a generator to minimize the squared MMD.
The minimization of the energy distance of raw images does not work well. We found the transformation $h(x)$ very helpful. Figure~\ref{fig:uncond_celeba64} shows the obtained samples, if minimizing $\cE(X, Y)$ of untransformed images.
Recently, \citet{mroueh2017mcgan} reported good results with mean and covariance feature matching GAN (McGan). McGan is also training a transformation to produce better features for the loss. An unbiased estimator exists for the squared MMD and \citet{dziugaite2015training} used the unbiased estimator.
An independent work by \citet{mmdgan} combine Generative Moment Matching Networks with a learned transformation. The transformation is driven to be injective by an additional auto-encoder loss.
\end{document}
% New in this paper.
\newcommand{\pnorm}[1]{\| #1 \|_p}
\newcommand{\panorm}[1]{| #1 |^p}
\newcommand{\dP}{\textrm{d} P}
% End new.
\newcommand{\cX}{\mathcal{X}}
\newcommand{\cS}{\mathcal{S}}
\newcommand{\cM}{\mathcal{M}}
\newcommand{\cF}{\mathcal{F}}
\newcommand{\cA}{\mathcal{A}}
\newcommand{\cE}{\mathcal{E}}
\newcommand{\cB}{\mathcal{B}}
\newcommand{\cZ}{\mathcal{Z}}
\newcommand{\cP}{\mathcal{P}}
\newcommand{\cR}{\mathcal{R}}
\newcommand{\cT}{\mathcal{T}}
\newcommand{\cG}{\mathcal{G}}
\newcommand{\cQ}{\mathcal{Q}}
\newcommand{\cL}{\mathcal{L}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bT}{\mathbb{T}}
\newcommand{\bG}{\mathbb{G}}
\newcommand{\bF}{\mathbb{F}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\lpar}{\left (}
\newcommand{\rpar}{\right )}
\newcommand{\lbar}{\left \vert}
\newcommand{\rbar}{\right \vert}
\newcommand{\zerovec}{\mathbf{0}}
\newcommand{\intset}[1]{[#1]}
% Indicator function
\newcommand{\indic}[1]{\mathbb{I}\left [ #1 \right ]}
%\newcommand{\indic}[1]{\left \llbracket #1 \right \rrbracket}
% Dot product
\newcommand{\dotp}[2]{#1 \cdot #2}
% Expectation
\DeclareMathOperator*{\expect}{{\huge \mathbb{E}}}
\newcommand{\expects}{\expect\nolimits}
\newcommand{\bigexpect}[1]{\bE \left [ #1 \right ]}
\newcommand{\Var}{\mathbb{V}}
% Norms
\newcommand{\norm}[1]{\left \| #1 \right \|}
\newcommand{\bignorm}[1]{\big \| #1 \big \|}
\newcommand{\infnorm}[1]{\left \| #1 \right \|_{\infty}}
\newtheorem{defn}{Definition}
\newtheorem{prop}{Proposition}
\newtheorem{lem}{Lemma}
\newtheorem{rem}{Remark}
\newtheorem{assum}{Assumption}
\newtheorem{cor}{Corollary}
\newtheorem{thm}{Theorem}
\newtheorem{example}{Example}
\newtheorem{notation}{Notation}
\newtheorem{obs}{Observation}
\newcommand{\etal}{\textit{et al.}}
% AIXI shorthand
\newcommand{\xn}{x_{1:n}}
\newcommand{\xln}{x_{<n}}
\newcommand{\xt}{x_{1:t}}
\newcommand{\at}{a_{1:t}}
\newcommand{\logl}{\mathcal{L}}
% Misc.
\newcommand{\floor}[1]{\left \lfloor {#1} \right \rfloor}
\newcommand{\ceil}[1]{\left \lceil {#1} \right \rceil}
\newcommand{\cbar}{\, | \,}
\newcommand{\cdbar}{\, \| \,}
\newcommand{\csemi}{\, ; \,}
\newcommand{\KL}{\textrm{KL}}
\newcommand{\TV}{\textrm{TV}}
\newcommand{\K}{\textrm{K}}
\def \grad {\nabla}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
\newcommand{\eqnref}[1]{(\ref{eqn:#1})}
\begin{thebibliography}{}
\bibitem[Arjovsky et~al., 2017]{arjovsky17wasserstein}
Arjovsky, M., Chintala, S., and Bottou, L. (2017).
\newblock Wasserstein {GAN}.
\newblock {\em arXiv preprint arXiv:1701.07875}.
\bibitem[Bellemare et~al., 2017]{bellemare17distributional}
Bellemare, M.~G., Dabney, W., and Munos, R. (2017).
\newblock A distributional perspective on reinforcement learning.
\newblock In {\em Proceedings of the International Conference on Machine
Learning, to appear}.
\bibitem[Bickel and Freedman, 1981]{bickel81asymptotic}
Bickel, P.~J. and Freedman, D.~A. (1981).
\newblock Some asymptotic theory for the bootstrap.
\newblock {\em The Annals of Statistics}, pages 1196--1217.
\bibitem[Chung and Sobel, 1987]{chung87discounted}
Chung, K.-J. and Sobel, M.~J. (1987).
\newblock Discounted {MDP}’s: Distribution functions and exponential utility
maximization.
\newblock {\em SIAM Journal on Control and Optimization}, 25(1):49--62.
\bibitem[Cover and Thomas, 1991]{cover91elements}
Cover, T.~M. and Thomas, J.~A. (1991).
\newblock {\em Elements of information theory}.
\newblock John Wiley \& Sons.
\bibitem[Danihelka et~al., 2017]{ivogan}
Danihelka, I., Lakshminarayanan, B., Uria, B., Wierstra, D., and Dayan, P.
(2017).
\newblock {Comparison of Maximum Likelihood and GAN-based training of Real
NVPs}.
\newblock {\em arXiv preprint arXiv:1705.05263}.
\bibitem[Dedecker and Merlev{\`e}de, 2007]{dedecker07empirical}
Dedecker, J. and Merlev{\`e}de, F. (2007).
\newblock The empirical distribution function for dependent variables:
asymptotic and nonasymptotic results in {L}p.
\newblock {\em ESAIM: Probability and Statistics}, 11:102--114.
\bibitem[Dudley, 2002]{dudley2002real}
Dudley, R.~M. (2002).
\newblock {\em Real analysis and probability}, volume~74.
\newblock Cambridge University Press.
\bibitem[Dziugaite et~al., 2015]{dziugaite2015training}
Dziugaite, G.~K., Roy, D.~M., and Ghahramani, Z. (2015).
\newblock Training generative neural networks via maximum mean discrepancy
optimization.
\newblock In {\em Proceedings of the Conference on Uncertainty in Artificial
Intelligence}.
\bibitem[Esfahani and Kuhn, 2015]{esfahani2015data}
Esfahani, P.~M. and Kuhn, D. (2015).
\newblock Data-driven distributionally robust optimization using the
{W}asserstein metric: Performance guarantees and tractable reformulations.
\newblock {\em arXiv preprint arXiv:1505.05116}.
\bibitem[Frogner et~al., 2015]{frogner15learning}
Frogner, C., Zhang, C., Mobahi, H., Araya, M., and Poggio, T.~A. (2015).
\newblock Learning with a {W}asserstein loss.
\newblock In {\em Advances in Neural Information Processing Systems}.
\bibitem[Gao and Kleywegt, 2016]{gao2016distributionally}
Gao, R. and Kleywegt, A.~J. (2016).
\newblock Distributionally robust stochastic optimization with {W}asserstein
distance.
\newblock {\em arXiv preprint arXiv:1604.02199}.
\bibitem[Gneiting and Raftery, 2007]{gneiting07strictly}
Gneiting, T. and Raftery, A.~E. (2007).
\newblock Strictly proper scoring rules, prediction, and estimation.
\newblock {\em Journal of the American Statistical Association},
102(477):359--378.
\bibitem[Goodfellow et~al., 2014]{goodfellow14generative}
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair,
S., Courville, A., and Bengio, Y. (2014).
\newblock Generative adversarial nets.
\newblock In {\em Advances in Neural Information Processing Systems}.
\bibitem[Gretton et~al., 2012]{gretton2012kernel}
Gretton, A., Borgwardt, K.~M., Rasch, M.~J., Sch{\"o}lkopf, B., and Smola, A.
(2012).
\newblock A kernel two-sample test.
\newblock {\em Journal of Machine Learning Research}, 13:723--773.
\bibitem[Gulrajani et~al., 2017]{gulrajani2017improved}
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017).
\newblock Improved training of {W}asserstein {GAN}s.
\newblock {\em arXiv preprint arXiv:1704.00028}.
\bibitem[Hern{\'a}ndez-Lobato and Adams, 2015]{PB}
Hern{\'a}ndez-Lobato, J.~M. and Adams, R.~P. (2015).
\newblock Probabilistic backpropagation for scalable learning of {B}ayesian
neural networks.
\newblock In {\em Proceedings of the International Conference on Machine
Learning}.
\bibitem[Isola et~al., 2016]{isola2016image}
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A.~A. (2016).
\newblock Image-to-image translation with conditional adversarial networks.
\newblock {\em arXiv preprint arXiv:1611.07004}.
\bibitem[Kingma and Welling, 2014]{kingma2013auto}
Kingma, D.~P. and Welling, M. (2014).
\newblock Auto-encoding variational {B}ayes.
\newblock {\em Proceedings of the International Conference on Learning
Representations}.
\bibitem[{Li} et~al., 2017]{mmdgan}
{Li}, C.-L., {Chang}, W.-C., {Cheng}, Y., {Yang}, Y., and {P{\'o}czos}, B.
(2017).
\newblock {MMD GAN}: Towards deeper understanding of moment matching network.
\newblock {\em arXiv preprint arXiv:1705.08584}.
\bibitem[Li et~al., 2015]{li2015generative}
Li, Y., Swersky, K., and Zemel, R. (2015).
\newblock Generative moment matching networks.
\newblock In {\em Proceedings of the International Conference on Machine
Learning}.
\bibitem[Lichman, 2013]{ucirepository}
Lichman, M. (2013).
\newblock {UCI} machine learning repository.
\bibitem[Liu et~al., 2015]{liu2015faceattributes}
Liu, Z., Luo, P., Wang, X., and Tang, X. (2015).
\newblock Deep learning face attributes in the wild.
\newblock In {\em Proceedings of International Conference on Computer Vision}.
\bibitem[Montavon et~al., 2016]{montavon2016wasserstein}
Montavon, G., M{\"u}ller, K.-R., and Cuturi, M. (2016).
\newblock {W}asserstein training of restricted {B}oltzmann machines.
\newblock In {\em Advances in Neural Information Processing Systems}.
\bibitem[Mroueh et~al., 2017]{mroueh2017mcgan}
Mroueh, Y., Sercu, T., and Goel, V. (2017).
\newblock Mc{G}an: Mean and covariance feature matching {GAN}.
\newblock {\em arXiv preprint arXiv:1702.08398}.
\bibitem[M{\"u}ller, 1997]{muller97integral}
M{\"u}ller, A. (1997).
\newblock Integral probability metrics and their generating classes of
functions.
\newblock {\em Advances in Applied Probability}, 29(2):429--443.
\bibitem[Rachev et~al., 2013]{rachev13methods}
Rachev, S.~T., Klebanov, L., Stoyanov, S.~V., and Fabozzi, F. (2013).
\newblock {\em The methods of distances in the theory of probability and
statistics}.
\newblock Springer.
\bibitem[Radford et~al., 2015]{radford2015unsupervised}
Radford, A., Metz, L., and Chintala, S. (2015).
\newblock Unsupervised representation learning with deep convolutional
generative adversarial networks.
\newblock {\em arXiv preprint arXiv:1511.06434}.
\bibitem[Rizzo and Sz{\'e}kely, 2016]{rizzo2016energy}
Rizzo, M.~L. and Sz{\'e}kely, G.~J. (2016).
\newblock Energy distance.
\newblock {\em Wiley Interdisciplinary Reviews: Computational Statistics},
8(1):27--38.
\bibitem[Ronneberger et~al., 2015]{ronneberger2015u}
Ronneberger, O., Fischer, P., and Brox, T. (2015).
\newblock U-net: Convolutional networks for biomedical image segmentation.
\newblock In {\em International Conference on Medical Image Computing and
Computer-Assisted Intervention}.
\bibitem[Rubner et~al., 2000]{rubner2000earth}
Rubner, Y., Tomasi, C., and Guibas, L.~J. (2000).
\newblock The earth mover's distance as a metric for image retrieval.
\newblock {\em International journal of computer vision}, 40(2):99--121.
\bibitem[Sejdinovic et~al., 2013]{sejdinovic2013equivalence}
Sejdinovic, D., Sriperumbudur, B., Gretton, A., Fukumizu, K., et~al. (2013).
\newblock Equivalence of distance-based and {RKHS}-based statistics in
hypothesis testing.
\newblock {\em The Annals of Statistics}, 41(5):2263--2291.
\bibitem[Sz{\'e}kely, 2002]{szekely02estatistics}
Sz{\'e}kely, G.~J. (2002).
\newblock E-statistics: The energy of statistical samples.
\newblock Technical Report 02-16, Bowling Green State University, Department of
Mathematics and Statistics.
\bibitem[{Van den Oord} et~al., 2016]{vandenoord16pixel}
{Van den Oord}, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016).
\newblock Pixel recurrent neural networks.
\newblock In {\em Proceedings of the International Conference on Machine
Learning}.
\bibitem[Veraar, 2010]{Veraar2010}
Veraar, M. (2010).
\newblock On {K}hintchine inequalities with a weight.
\newblock {\em Proceedings of the American Mathematical Society},
138(11):4119--4121.
\bibitem[Zolotarev, 1976]{zolotarev76metric}
Zolotarev, V.~M. (1976).
\newblock Metric distances in spaces of random variables and their
distributions.
\newblock {\em Sbornik: Mathematics}, 30(3):373--401.
\end{thebibliography}