A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model
Extracted source text · arXiv 2205.06924
\documentclass{article} % For LaTeX2e
\usepackage{iclr2022_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath, bm}
\usepackage{amssymb}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{siunitx}
\usepackage{wrapfig}
\usepackage{lipsum}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{mathtools}
\usepackage{algorithm}
\usepackage{algorithmicx}
\usepackage{algpseudocode}
\newcommand\norm[1]{\left\lVert#1\right\rVert}
\hypersetup{
colorlinks=true,
citecolor = {blue}
}
\title{\centering A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model}
\author{\hspace{3.6cm} Jianwen Xie, Yaxuan Zhu, Jun Li, Ping Li \\\\
\hspace{3.8cm}Cognitive Computing Lab, Baidu Research\\
%\hspace{5cm}Cognitive Computing Lab\\
%\hspace{5.7cm}Baidu Research\\
\hspace{3.65cm}10900 NE 8th St. Bellevue, WA 98004, USA\\
\hspace{0cm} \texttt{\texttt{\{jianwen.kenny, yaxuanzhu.alvin, junli.sh04, pingli98\}@gmail.com}}
}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\setlength{\textfloatsep}{6.0 pt plus 1.0pt minus 3.0pt}
\setlength{\intextsep}{0.3cm}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}\vspace{-0.05in}
This paper studies the cooperative learning of two generative flow models, in which the two models are iteratively updated based on the jointly synthesized examples. The first flow model is a normalizing flow that transforms an initial simple density into a target density by applying a sequence of invertible transformations. The second flow model is a Langevin flow that runs finite steps of gradient-based MCMC toward an energy-based model. We start from proposing a generative framework that trains an energy-based model with a normalizing flow as an amortized sampler to initialize the MCMC chains of the energy-based model. In each learning iteration, we generate synthesized examples by using a normalizing flow initialization followed by a short-run Langevin flow revision toward the current energy-based model. Then we treat the synthesized examples as fair samples from the energy-based model and update the model parameters with the maximum likelihood learning gradient, while the normalizing flow directly learns from the synthesized examples by maximizing the tractable likelihood. Under the short-run non-mixing MCMC scenario, the estimation of the energy-based model is shown to follow the perturbation of maximum likelihood, and the short-run Langevin flow and the normalizing flow form a two-flow generator that we call CoopFlow. We provide an understating of the CoopFlow algorithm by information geometry and show that it is a valid generator as it converges to a moment matching estimator. We demonstrate that the trained CoopFlow is capable of synthesizing realistic images, reconstructing images, and interpolating between images.
\end{abstract}
\section{Introduction and Motivation}
\vspace{-0.05in}
Normalizing flows \citep{DinhKB14,DinhSB17,kingma2018glow} are a family of generative models that construct a complex distribution by transforming a simple probability density, such as Gaussian distribution, through a sequence of invertible and differentiable mappings. Due to the tractability of the exact log-likelihood and the efficiency of the inference and synthesis, normalizing flows have gained popularity in density estimation \citep{kingma2018glow, ho2019flow++, YangHH0BH19, PrengerVC19, KumarBEFLDK20} and variational inference \citep{RezendeM15, KingmaSJCCSW16}. However, for the sake of ensuring the favorable property of closed-form density evaluation, the normalizing flows typically require special designs of the sequence of transformations, which, in general, constrain the expressive power of the models.
Energy-based models (EBMs) \citep{ZhuWM98,lecun2006tutorial,hinton2012practical} define an unnormalized probability density function of data, which is the exponential of the negative energy function. The energy function is directly defined on data domain, and assigns each input configuration with a scalar energy value, with lower energy values indicating more likely configurations. Recently, with the energy function parameterized by a modern deep network such as a ConvNet, the ConvNet-EBMs \citep{XieLZW16} have gained unprecedented success in modeling large-scale data sets \citep{GaoLZZW18,nijkamp2019learning,du2019implicit,GrathwohlWJD0S20,ruiqi21,zhao2020learning,ZhengXL21} and exhibited stunning performance in synthesizing different modalities of data, e.g., videos \citep{XieZW17,XieZW21}, volumetric shapes \citep{XieZGWZW18,xie2020generative}, point clouds \citep{XieXZZW21} and molecules \citep{DuMMFR20}. The parameters of the energy function can be trained by maximum likelihood estimation (MLE). However, due to the intractable integral in computing the normalizing constant, the evaluation of the gradient of the log-likelihood typically requires Markov chain Monte Carlo (MCMC) sampling \citep{barbu2020monte}, e.g., Langevin dynamics \citep{neal2011mcmc}, to generate samples from the current model. However, the Langevin sampling on a highly multi-modal energy function, due to the use of deep network parameterization, is generally not mixing. When sampling from a density with a multi-modal landscape, the Langevin dynamics, which follows the gradient information, is apt to get trapped by local modes of the density and is unlikely to jump out and explore other isolated modes. Relying on non-mixing MCMC samples, the estimated gradient of the likelihood is biased, and the resulting learned EBM may become an invalid model, which is unable to approximate the data distribution~as~expected.
Recently, \citet{nijkamp2019learning} propose to train an EBM with a short-run non-convergent Langevin dynamics, and show that even though the energy function is invalid, the short-run MCMC can be~treated as a valid flow-like model that generates realistic examples. This not only provides an explanation of why an EBM with a non-convergent MCMC is still capable of synthesizing realistic examples, but also suggests a more practical computationally-affordable way to learn useful generative models under the existing energy-based frameworks. Although EBMs have been widely applied to different domains, learning short-run MCMC in the context of EBM is still~underexplored.
In this paper, we accept the fact that MCMC sampling is not mixing in practice, and decide~to~give~up the goal of training a valid EBM of data. Instead, we treat the short-run non-convergent Langevin dynamics, which shares parameters with the energy function, as a flow-like transformation that we call the Langevin flow because it can be considered a noise-injected residual network. %with the initial noise serving as the latent variables.
Even though implementing a short-run Langevin flow is surprisingly simple, which is nothing but a design of a bottom-up ConvNet for the energy function, it might still require a sufficiently large number of Langevin steps (each Langevin step consists of one step of gradient descent and one step of diffusion) to construct an effective Langevin flow, so that it can be expressive enough to represent the data distribution. Motivated by reducing the number of Langevin steps in the Langevin flow for computational efficiency, we propose the CoopFlow model that trains a Langevin flow jointly with a normalizing flow in a cooperative learning scheme \citep{XieLGZW20}, in which the normalizing flow learns to serve as a rapid sampler to initialize the Langevin flow so that the Langevin flow can be shorter,
while the Langevin flow teaches the normalizing flow by short-run MCMC transition toward the EBM so that the normalizing flow can accumulate the temporal difference in the transition to provide better initial samples.
Compared to the original cooperative learning~framework~\citep{XieLGZW20} that incorporates the MLE algorithm of an EBM and the MLE algorithm of a generator, the CoopFlow benefits from using a normalizing flow instead of a generic generator because the MLE of a normalizing flow generator is much more tractable than the MLE of any other generic generator.
The latter might resort to either MCMC-based inference to evaluate the posterior distribution or another encoder network for variational inference. Besides, in the CoopFlow, the~Langevin~flow can overcome the expressivity limitation of the normalizing flow caused by invertibility constraint, which also motivates us to study the CoopFlow~model. We further understand the dynamics of cooperative learning with short-run non-mixing MCMC by information geometry. We provide a justification that the CoopFlow trained in the context of EBM with non-mixing MCMC is a valid generator because it converges to a moment matching estimator. Experiments, including image generation, image reconstruction, and latent space interpolation, are conducted to~support~our~justification.
\vspace{-0.05in}
\section{Contributions and Related Work}
\vspace{-0.05in}
Our paper studies amortized sampling for training a short-run non-mixing Langevin sampler toward an EBM. We propose a novel framework, the CoopFlow, which cooperatively trains a short-run Langevin flow as a valid generator and a normalizing flow as an amortized sampler for image representation. We provide both theoretical and empirical justifications for the proposed CoopFlow algorithm, which has been implemented in the PaddlePaddle deep learning platform. The following are some closely related work. We will point out the differences between our work and the prior arts to further highlight the contributions and novelty of our paper.
\textbf{Learning short-run MCMC as a generator.} \ Recently, \citet{nijkamp2019learning} propose to learn an EBM with short-run non-convergent MCMC that samples from the model, and they eventually keep the short-run MCMC as a valid generator and discard the biased EBM. \citet{NijkampP0ZZW20} use short-run MCMC to sample the latent space of a top-down generative model in a variational learning framework. \citet{AnXL21} propose to correct the bias of the short-run MCMC inference by optimal transport in training latent variable models. \citet{pang2020learning} adopt short-run MCMC to sample from both the EBM prior and the posterior of the latent variables. Our work studies learning a normalizing flow to amortize the sampling cost of a short-run non-mixing MCMC sampler (i.e., Langevin flow) in data space, which makes a further step forward in this underexplored theme.
%\vspace{0.07in}
\textbf{Cooperative learning with MCMC teaching.} \ Our learning algorithm is inspired by the cooperative learning or the CoopNets \citep{XieLGW18,XieLGZW20}, in which an ConvNet-EBM \citep{XieLZW16} and a top-down generator \citep{HanLZW17} are jointly trained by jump-starting their maximum learning algorithms. \citet{XieZL21} replace the generator in the original CoopNets by a variational autoencoder (VAE) \citep{kingma2013auto} for efficient inference. The proposed CoopFlow algorithm is different from the above prior work in the following two aspects. (i) In the idealized long-run mixing MCMC scenario, our algorithm is a cooperative learning framework that trains an unbiased EBM and a normalizing flow via MCMC teaching, where updating a normalizing flow with a tractable density is more efficient and less biased than updating a generic generator via variational inference as in \cite{XieZL21} or MCMC-based inference as in \cite{XieLGZW20}. (ii) Our paper has a novel emphasis on studying cooperative learning with short-run non-mixing MCMC, which is more practical and common in reality. Our algorithm actually trains a short-run Langevin flow and a normalizing flow together toward a biased EBM for image generation. We use information geometry to understand the learning dynamics and show that the learned two-flow generator (i.e., CoopFlow) is a valid generative model, even though the learned EBM is biased.
%\vspace{0.07in}
\textbf{Joint training of EBM and normalizing flow.} \ There are other two works studying an EBM and a normalizing flow together as well. To avoid MCMC, \citet{gao2020flow} propose to train an EBM using noise contrastive estimation \citep{GutmannH10}, where the noise distribution is a normalizing flow. \citet{nijkamp2020learning} propose to learn an EBM as an exponential tilting of a pretrained normalizing flow, so that neural transport MCMC sampling in the latent space of the normalizing flow can mix well. Our paper proposes to train an EBM and a normalizing flow via short-run MCMC teaching. More specifically, we focus on short-run non-mixing MCMC and treat it as a valid flow-like model (i.e., short-run Langevin flow) that is guided by the EBM. Disregarding the biased EBM, the resulting valid generator is the combination of the short-run Langevin flow and the normalizing flow, where the latter serves as a rapid initializer of the former. The form~of~this two-flow generator shares a similar fashion with the Stochastic Normalizing Flow \citep{0035KN20}, which consists of a sequence of deterministic invertible transformations and stochastic sampling~blocks.
\vspace{-0.05in}
\section{Two Flow Models}
\vspace{-0.05in}
\subsection{Langevin Flow}
\textbf{Energy-based model.} \ Let $x\in \mathbb{R}^D$ be the observed signal such as an image. An energy-based model defines an unnormalized probability distribution of $x$ as follows:
\begin{equation}
p_{\theta}(x)=\frac{1}{Z(\theta)} \exp[f_{\theta}(x)],
\label{eq:ebm}
\end{equation}
where $f_{\theta}: \mathbb{R}^D \rightarrow \mathbb{R}$ is the negative energy function and defined by a bottom-up neural network whose parameters are denoted by $\theta$. The normalizing constant or partition function $Z(\theta)=\int \exp[f_{\theta}(x)]dx$ is analytically intractable and difficult to compute due to high dimensionality of $x$.
%\vspace{0.07in}
\textbf{Maximum likelihood learning.} \ Suppose we observe unlabeled training examples $\{x_i, i=1,...,n\}$ from unknown data distribution $p_{\text{data}}(x)$, the energy-based model in Eq.~(\ref{eq:ebm}) can be trained from $\{x_i\}$ by Markov chain Monte Carlo (MCMC)-based maximum likelihood estimation, in which MCMC samples are drawn from the model $p_{\theta}(x)$ to approximate the gradient of the log-likelihood function for updating the model parameters $\theta$. Specifically, the log-likelihood is $L(\theta)=\frac{1}{n}\sum_{i=1}^{n}\log p_{\theta}(x_i)$. For a large $n$, maximizing $L(\theta)$ is equivalent to minimizing the Kullback-Leibler (KL) divergence $\mathbb{D}_{\text{KL}}(p_{\text{data}}||p_{\theta})$. The learning gradient is given by
\begin{equation}
L'(\theta)=\mathbb{E}_{p_{\text{data}}}[\nabla_{\theta}f_{\theta}(x)]-\mathbb{E}_{p_{\theta}}[\nabla_{\theta}f_{\theta}(x)]\approx \frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}f_{\theta}(x_i)-\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}f_{\theta}(\tilde{x}_i),
\label{eq:ebm_gradient}
\end{equation}
where the expectations are approximated by averaging over the observed examples $\{x_i\}$ and the synthesized examples $\{\tilde{x}_i\}$ generated from the current model $p_{\theta}(x)$, respectively.
%\vspace{0.07in}
\textbf{Langevin dynamics as MCMC.} \ Generating synthesized examples from $p_{\theta}(x)$ can be accomplished with a gradient-based MCMC such as Langevin dynamics, which is applied as follows
\begin{equation}
x_{t}=x_{t-1}+ \frac{\delta^2}{2} \nabla_{x} f_{\theta}(x_{t-1})+\delta \epsilon_{t}; \hspace{3mm} x_0 \sim p_0(x),\hspace{3mm} \epsilon_{t} \sim \mathcal{N}(0, I_D), \hspace{3mm} t=1, \cdots,T,
\label{eq:MCMC}
\end{equation}
where $t$ indexes the Langevin time step, $\delta$ denotes the Langevin step size, $\epsilon_t$ is a Brownian motion that explores different modes, and $p_0(x)$ is a uniform distribution that initializes MCMC chains.
%\vspace{0.07in}
\textbf{Langevin flow.} \ As $T \rightarrow \infty$ and $\delta \rightarrow 0$, $x_T$ becomes an exact sample from $p_{\theta}(x)$ under some regularity conditions. However, it is impractical to run infinite steps with infinitesimal step
size to generate fair examples from the target distribution. Additionally, convergence of MCMC chains in many cases is hopeless because $p_{\theta}(x)$ can be very complex and highly multi-modal, then the gradient-based Langevin dynamics has no way to escape from local modes, so that different MCMC chains with different starting points are unable to mix. Let $\tilde{p}_{\theta}(x)$ be the distribution of $x_T$, which is the resulting distribution of $x$ after $T$ steps of Langevin updates starting from $x_0 \sim p_0(x)$. Due~to~the fixed $p_0(x)$, $T$ and $\delta$, the distribution $\tilde{p}_{\theta}(x)$ is well defined, which can be implicitly expressed by
\begin{equation}
\tilde{p}_{\theta}(x)=(\mathcal{K}_{\theta}p_0)(x)=\int p_{0}(z)\mathcal{K}_{\theta}(x|z) dz,
\label{eq:flow_density}
\end{equation}
where $\mathcal{K}_{\theta}$ denotes the transition kernel of $T$ steps of Langevin dynamics that samples $p_{\theta}$. Generally, $\tilde{p}_{\theta}(x)$ is not necessarily equal to $p_{\theta}(x)$. $\tilde{p}_{\theta}(x)$ is dependent on $T$ and $\delta$, which are omitted in the notation for simplicity. The KL-divergence $\mathbb{D}_{\text{KL}}(\tilde{p}_{\theta}(x)||p_{\theta}(x))=-{\text{entropy}}(\tilde{p}_{\theta}(x))-\mathbb{E}_{\tilde{p}_{\theta}(x)}[f(x)] +\log Z$. The gradient ascent part of Langevin dynamics, $x_{t}=x_{t-1}+ \delta^2/2 \nabla_{x} f_{\theta}(x_{t-1})$, increases the negative energy $f_{\theta}(x)$ by moving $x$ to the local modes of $f_{\theta}(x)$, and the diffusion part $\delta \epsilon_t$ increases the entropy of $\tilde{p}_{\theta}(x)$ by jumping out of the local modes. According to the law of thermodynamics \citep{cover2006elements}, $\mathbb{D}_{\text{KL}}(\tilde{p}_{\theta}(x)||p_{\theta}(x))$ decreases to zero monotonically as $T \rightarrow \infty$.
\subsection{Normalizing Flow}
Let $z \in \mathbb{R}^{D}$ be the latent vector of the same dimensionality as $x$.
A normalizing flow is of the form
\begin{equation}
x=g_{\alpha}(z); z \sim q_{0}(z),
\label{eq:flow}
\end{equation}
where $q_{0}(z)$ is a known prior distribution such as Gaussian white noise distribution $\mathcal{N}(0,I_D)$, and $g_{\alpha}: \mathbb{R}^D \rightarrow \mathbb{R}^D$ is a mapping that consists of a sequence of $L$ invertible transformations, i.e., $g(z)=g_{L} \circ \cdots \circ g_{2} \circ g_{1}(z)$, whose inversion $z=g_{\alpha}^{-1}(x)$ and log-determinants of the Jacobians can be computed in closed form. $\alpha$ are the parameters of $g_{\alpha}$. The mapping is used to transform a random vector $z$ that follows a simple distribution $q_0$ into a flexible distribution. Under the change-of-variables law, the resulting random vector $x=g_{\alpha}(z)$ has a probability density $q_{\alpha}(x)=q_{0}(g_{\alpha}^{-1}(x))|\det (\partial g_{\alpha}^{-1}(x)/\partial x)|$.
Let $h_l=g_{l}(h_{l-1})$. The successive transformations between $x$ and $z$ can be expressed as a flow
%$z \xrightleftharpoons[g_1^{-1}]{g_1} h_1 \xrightleftharpoons[g_2^{-1}]{g_2} h_2 \cdots \xrightleftharpoons[g_L^{-1}]{g_L} x$,
$z \stackrel{g_1}{\longleftrightarrow} h_1 \stackrel{g_2}{\longleftrightarrow} h_2 \cdots \stackrel{g_L}{\longleftrightarrow} x$, where we define $z:=h_0$ and $x:=h_L$ for succinctness. Then the determinant becomes $|\det (\partial g_{\alpha}^{-1}(x)/\partial x)|=\prod_{l=1}^{L}|\det (\partial h_{l-1}/\partial h_{l})|$.
The log-likelihood of a datapoint $x$ can be easily computed by
\begin{equation}
\log q_{\alpha}(x)=\log q_{0}(z)+ \sum_{l=1}^{L}\log \left|\det \left(\frac{\partial h_{l-1}}{\partial h_l}\right)\right|.
\label{eq:flow_log}
\end{equation}
With some smart designs of the sequence of transformations $g_{\alpha}=\{g_{l},l=1,...,L\}$, the log-determinant in Eq.~(\ref{eq:flow_log}) can be easily computed, then the normalizing flow $q_{\alpha}(x)$ can be trained by maximizing the exact data log-likelihood $L(\alpha)=\frac{1}{n}\sum_{i=1}^{n} \log q_{\alpha}(x_i)$ via gradient ascent algorithm.
\vspace{-0.05in}
\section{CoopFlow: Cooperative Training of Two Flows}
\vspace{-0.05in}
\subsection{CoopFlow Algorithm}
We have moved from trying to use a convergent Langevin dynamics to train a valid EBM. Instead, we accept the fact that the short-run non-convergent MCMC is inevitable and more affordable in practice, and we treat a non-convergent short-run Langevin flow as a generator and propose to jointly train it with a normalizing flow as a rapid initializer for more efficient generation. The resulting generator is called CoopFlow, which consists of both the Langevin flow and the normalizing flow.
Specifically, at each iteration, for $i = 1,...,m$, we first generate $z_i \sim \mathcal{N}(0,I_D)$, and then transform $z_i$ by a normalizing flow to obtain $\hat{x}_i = g_{\alpha}(z_i)$.
Next, starting from each $\hat{x}_i$, we run a Langevin flow (i.e., a finite number of Langevin steps toward an EBM $p_{\theta}(x)$) to obtain $\tilde{x}_i$. ${\tilde{x}_i}$ are considered synthesized examples that are generated by the CoopFlow model. We then update $\alpha$ of the normalizing flow by treating ${\tilde{x}_i}$ as training data, and update $\theta$ of the Langevin flow according to the learning gradient of the EBM, which is computed with the synthesized examples $\{\tilde{x}_i\}$ and the observed examples $\{x_i\}$. Algorithm~\ref{alg1} presents a description of the proposed CoopFlow algorithm. The advantage of this training scheme is that we only need to minimally modify the existing codes for the MLE training of the EBM $p_{\theta}$ and the normalizing flow $q_{\alpha}$. The probability density of the CoopFlow ${\pi}_{(\theta,\alpha)}(x)$ is well
defined, which can be implicitly expressed by
\begin{equation}
\pi_{(\theta,\alpha)}(x)=(\mathcal{K}_{\theta}q_{\alpha})(x)=\int q_{\alpha}(x')\mathcal{K}_{\theta}(x|x') dx'.
\label{eq:coop_flow_density}
\end{equation}
$\mathcal{K}_{\theta}$ is the transition kernel of the Langevin flow. If we increase the length $T$ of the Langevin flow, $\pi_{(\theta,\alpha)}$ will converge to the EBM $p_{\theta}(x)$. The network $f_\theta(x)$ in the Langevin flow is scalar valued and is of free form, whereas the network $g_{\alpha}(x)$ in the normalizing flow has high-dimensional output and is of a severely constrained form. Thus the Langevin flow can potentially provide a tighter fit to $p_{\text{data}}(x)$ than the normalizing flow. The Langevin flow may also be potentially more data efficient as it tends to have a smaller network than the normalizing flow. On the flip side, sampling from the Langevin flow requires multiple iterations, whereas the normalizing flow can synthesize examples via a direct mapping. It is thus desirable to train these two flows simultaneously, where the normalizing flow serves as an approximate sampler to amortize the iterative sampling of the Langevin flow.
%amortize the MCMC sampling of $p_{\theta}(x)$.
Meanwhile, the normalizing flow is updated by a temporal difference MCMC teaching provided by the Langevin flow, to further amortize the short-run Langevin flow.
\begin{algorithm}[H]
%\small
\caption{CoopFlow Algorithm}
\textbf{Input}: (1) Observed images $\{x_i\}_{i}^{n}$; (2) Number of Langevin steps $T$; (3) Langevin step size $\delta$; (4) Learning rate $\eta_{\theta}$ for Langevin flow; (5) Learning rate $\eta_\alpha$ for normalizing flow; (6) batch size $m$. \\
\textbf{Output}:
Parameters $\{\theta, \alpha\}$
\begin{algorithmic}[1]
\State Randomly initialize $\theta$ and $\alpha$.
%\For{$t \leftarrow 1$ to $T$}
\Repeat
\State Sample observed examples $\{x_i\}_i^m \sim p_{\text{data}}(x)$.
\State Sample noise examples $\{z_i\}_{i=1}^{m} \sim q_0(z)$,
\State
%\textbf{Normalizing Flow Initialization}:
Starting from $z_i$, generate $\hat{x}_i=g_{\alpha}(z_i)$ via normalizing flow.
\State
%\textbf{Langevin Flow Evolution}:
Starting from $\hat{x}_i$, run a $T$-step Langevin flow to obtain $\tilde{x}_i$ by following Eq.~(\ref{eq:MCMC}).
\State %\textbf{Update Langevin flow}:
Given $\{\tilde{x}_i\}$, update $\alpha$ by maximizing $\frac{1}{m}\sum_{i=1}^{m} \log q_{\alpha}(\tilde{x}_i)$ with Eq.~(\ref{eq:flow_log}).
\State %\textbf{Update normalizing flow}:
Given $\{x_i\}$ and $\{\tilde{x}_i\}$, update $\theta$ by following the gradient $\nabla{\theta}$ in Eq.~(\ref{eq:ebm_gradient}).
\Until{converged}
%\EndFor
\end{algorithmic} \label{alg1}
% \vspace{5mm}
\end{algorithm}
\subsection{Understanding the Learned Two Flows} \label{theory_understand}
\textbf{Convergence equations.} \ In the traditional contrastive divergence (CD) \citep{Hinton02} algorithm, MCMC chains are initialized with observed data so that the CD learning seeks to minimize $\mathbb{D}_{\text{KL}}(p_{\text{data}}(x)||p_{\theta}(x))-\mathbb{D}_{\text{KL}}((\mathcal{K}_{\theta} p_{\text{data}})(x)|| p_{\theta}(x))$, where $(\mathcal{K}_{\theta} p_{\text{data}})(x)$ denotes the marginal distribution obtained by running the Markov transition $\mathcal{K}_{\theta}$, which is specified by the Langevin flow, from the data distribution $p_{\text{data}}$. In the CoopFlow algorithm, the learning of the EBM (or the Langevin flow model) follows a modified contrastive divergence, where the initial distribution of the Langevin flow is modified to be a normalizing flow $q_{\alpha}$. Thus, at iteration $t$, the update of $\theta$ follows the gradient of
$\mathbb{D}_{\text{KL}}(p_{\text{data}}||p_{\theta})-\mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{(t)}} q_{\alpha^{(t)}}|| p_{\theta} )$
with respect to $\theta$. Compared to the traditional CD loss, the modified one replaces $p_{\text{data}}$ by $q_{\alpha}$ in the second KL divergence term. At iteration $t$, the update of the normalizing flow $q_{\alpha}$ follows the gradient of
$\mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{(t)}} q_{\alpha^{(t)}}|| q_{\alpha} )
\label{eq:flow_KL}$
with respect to $\alpha$. Let $(\theta^*, \alpha^*)$ be a fixed point the learning algorithm converges to, then we have the following convergence equations%the convergent point satisfies the following equations:
\begin{align}
\theta^{*} &= \arg \min_{\theta}
\mathbb{D}_{\text{KL}}(p_{\text{data}}||p_{\theta})-\mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{*}} q_{\alpha^{*}}|| p_{\theta} ),\label{eq:fix_points1}\\
\alpha^{*} &= \arg \min_{\alpha} \mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{*}} q_{\alpha^{*}}|| q_{\alpha} ).
\label{eq:fix_points2}
\end{align}
\textbf{Ideal case analysis.} \ In the idealized scenario where the normalizing flow $q_{\alpha}$ has infinite capacity and the Langevin sampling can mix and converge to the sampled EBM, Eq.~(\ref{eq:fix_points2}) means that $q_{\alpha^{*}}$ attempts to be the stationary distribution of $\mathcal{K}_{\theta^{*}}$, which is $p_{\theta^{*}}$ because $\mathcal{K}_{\theta^{*}}$ is the Markov transition kernel of a convergent MCMC sampled from $p_{\theta^{*}}$. That is, $\min_{\alpha} \mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{*}} q_{\alpha^{*}}|| q_{\alpha} )=0$, thus $q_{\alpha^{*}}=p_{\theta^{*}}$. With the second term becoming 0 and vanishing, Eq.~(\ref{eq:fix_points1}) degrades to $\min_{\theta}
\mathbb{D}_{\text{KL}}(p_{\text{data}}||p_{\theta})$ and thus $\theta^{*}$ is the maximum likelihood estimate. On the other hand, the normalizing flow $q_{\alpha}$ chases the EBM $p_{\theta}$ toward $p_{\text{data}}$, thus $\alpha^{*}$ is also a maximum likelihood estimate.
%\vspace{0.07in}
\textbf{Moment matching estimator.} \ In the practical scenario where the Langevin sampling is not mixing, the CoopFlow model $\pi_t = \mathcal{K}_{\theta^{(t)}} q_{\alpha^{(t)}}$ is an interpolation between the learned $q_{\alpha^{(t)}}$ and $p_{\theta^{(t)}}$, and it converges to $\pi^* = \mathcal{K}_{\theta^*} q_{\alpha^*}$, which is an interpolation between $q_{\alpha^*}$ and $p_{\theta^*}$. ${\pi}^{*}$ is the short-run Langevin flow starting from $q_{\alpha^*}$ towards EBM $p_{\theta*}$. $\pi^{*}$ is a legitimate generator because $\mathbb{E}_{p_{\text{data}}}[\nabla_\theta f_{\theta^{*}}(x)] = \mathbb{E}_{\pi^{*}}[\nabla_{\theta} f_{\theta^{*}}(x)]$ at convergence. That is, $\pi^{*}$ leads to moment matching in the feature statistics $\nabla_\theta f_{\theta^{*}}(x)$. In other words, $\pi^{*}$ satisfies the above estimating equation.
\textbf{Understanding via information geometry}. Consider a simple EBM with $f_\theta(x)=\langle\theta, h(x) \rangle$, where $h(x)$ is the feature statistics.
Since $\nabla_{\theta} f_{\theta}(x)=h(x)$, the MLE of the EBM $p_{\theta_{\text{MLE}}}$ is a moment matching estimator due to $\mathbb{E}_{p_{\text{data}}}[h(x)] = \mathbb{E}_{p_{\theta_{\text{MLE}}}}[h(x)]$. The CoopFlow $\pi^{*}$ also converges to a moment matching estimator, i.e., $\mathbb{E}_{p_{\text{data}}}[h(x)] = \mathbb{E}_{\pi^{*}}[h(x)]$.
Figure~\ref{fig:illustration} is an illustration of model distributions that correspond to different parameters at convergence. We first introduce three families of distributions: $\Omega=\{p: \mathbb{E}_{p}[h(x)]=\mathbb{E}_{p_{\text{data}}}[h(x)]\}$, $\Theta=\{p_{\theta}(x)=\exp(\langle \theta,h(x)\rangle)/Z(\theta), \forall \theta\}$, and $A=\{q_{\alpha}, \forall \alpha\}$, which are shown by the red, blue and green curves respectively in Figure~\ref{fig:illustration}.
\begin{wrapfigure}{r}{0.37\textwidth}
\begin{minipage}{0.36\textwidth}
\vspace{-0.05in}
\begin{center}
\includegraphics[width=0.99\textwidth]{images/illus_flow.jpg}
\end{center}
\vspace{-0.1in}
\caption{An illustration of convergence of the CoopFlow algorithm.}
\label{fig:illustration}\vspace{-0.1in}
\end{minipage}
\end{wrapfigure}
$\Omega$ is the set of distributions that reproduce statistical property $h(x)$ of the data distribution. Obviously, $p_{\theta_{\text{MLE}}}$, $p_{\text{data}}$, and $\pi^{*}=\mathcal{K}_{\theta^{*}}q_{\alpha^{*}}$ are included in $\Omega$. $\Theta$ is the set of EBMs with different values of $\theta$, thus $p_{\theta_{\text{MLE}}}$ and $p_{\theta^{*}}$ belong to $\Theta$. Because of the short-run Langevin flow, $p_{\theta^{*}}$ is not a valid model that matches the data distribution in terms of $h(x)$, and thus $p_{\theta^{*}}$ is not in $\Omega$. $A$ is the set of normalizing flow models with different values of $\alpha$, thus $q_{\alpha^{*}}$ and $q_{\alpha_{\text{MLE}}}$ belong to $A$.
The yellow line shows the MCMC trajectory. The solid segment of the yellow line, starting from $q_{\alpha^{*}}$ to $\mathcal{K}_{\theta^{*}}q_{\alpha^{*}}$, illustrates the short-run non-mixing MCMC that is initialized by the normalizing flow $q_{\alpha^{*}}$ in $A$ and arrives at $\pi^{*}=\mathcal{K}_{\theta^{*}}q_{\alpha^{*}}$ in $\Omega$. The dotted segment of the yellow line, starting from $\pi^{*}=\mathcal{K}_{\theta^{*}}q_{\alpha^{*}}$ in $\Omega$ to $p_{\theta^{*}}$ in $\Theta$, shows the potential long-run MCMC trajectory, which is not really realized because we stop short in MCMC. If we increase the number of steps of short-run Langevin flow, $\mathbb{D}_{\text{KL}}(\pi^*|| p_{\theta^{*}})$ will be monotonically decreasing to 0. Though $\pi^{*}$ stops midway in the path toward $p_{\theta*}$, $\pi^{*}$ is still a valid generator because it is in $\Omega$. %with the fact that $\mathbb{E}_{p_{\text{data}}}[\nabla_\theta f_{\theta^{*}}(x)] = \mathbb{E}_{\pi^{*}}[\nabla_{\theta} f_{\theta^{*}}(x)]$.
\textbf{Perturbation of MLE}. $p_{\theta_{\text{MLE}}}$ is the intersection between $\Theta$ and $\Omega$. It is the projection of $p_{\text{data}}$ onto $\Theta$ because $\theta_{\text{MLE}}=\arg \min_{\theta} \mathbb{D}_{\text{KL}}(p_{\text{data}}||p_\theta)$. $q_{\alpha_{\text{MLE}}}$ is also a projection of $p_{\text{data}}$ onto $A$ because $\alpha_{\text{MLE}}=\arg \min_{\alpha} \mathbb{D}_{\text{KL}}(p_{\text{data}}||q_\alpha)$. Generally, $q_{\alpha_{\text{MLE}}}$ is far from $p_{\text{data}}$ due to its restricted form network, whereas $p_{\theta_{\text{MLE}}}$ is very close to $p_{\text{data}}$ due to its scalar-valued free form network. Because the short-run MCMC is not mixing, $\theta^{*}\neq\theta_{\text{MLE}}$ and $\alpha^{*}\neq \alpha_{\text{MLE}}$. $\theta^{*}$ and $\alpha^{*}$ are perturbations of $\theta_{\text{MLE}}$ and $\alpha_{\text{MLE}}$, respectively. $p_{\theta_{\text{MLE}}}$ shown by an empty dot is not attainable unfortunately. We want to point out that, as $T$ goes to infinity, $p_{\theta^{*}} = p_{\theta_{\text{MLE}}}$, and $q_{\alpha^{*}} = q_{\alpha_{\text{MLE}}}$.
Note that $\Omega$, $\Theta$ and $A$ are high-dimensional manifolds instead of 1D curves as depicted in Figure \ref{fig:illustration}, and $\pi^{*}$ may be farther away from $p_{\text{data}}$ than~$p_{\theta_{\text{MLE}}}$~is. During learning, $q_{\alpha^{(t+1)}}$ is the projection of $\mathcal{K}_{\theta^{(t)}}q_{\alpha^{(t)}}$ on $A$. At convergence, $q_{\alpha^{*}}$ is the projection of $\pi^{*}=\mathcal{K}_{\theta^{*}}q_{\alpha^{*}}$ on $A$. There is an infinite looping between $q_{\alpha^{*}}$ and $\pi^{*}=\mathcal{K}_{\theta^{*}}q_{\alpha^{*}}$ at convergence of the CoopFlow algorithm, i.e., $\pi^{*}$ lifts $q_{\alpha^{*}}$ off the ground $A$, and the projection~drops~$\pi^{*}$~back~to~$q_{\alpha^{*}}$. Although $p_{\theta^{*}}$ and $q_{\alpha^{*}}$ are biased, they are not wrong. Many useful models and algorithms, e.g., variational autoencoder and contrastive divergence are also biased to MLE. Their learning objectives also follow perturbation of MLE.
\vspace{-0.05in}
\section{Experiments}
\vspace{-0.05in}
We showcase experiment results on various tasks. We start from a toy example to illustrate the basic idea of the CoopFlow in Section~\ref{toy}. We show image generation results in Section~\ref{generation}. Section~\ref{reconstruction} demonstrates the learned CoopFlow is useful for image reconstruction and inpainting, while Section~\ref{interpolation} shows that the learned latent space is meaningful so that it can be used for interpolation.
\subsection{Toy Example Study} \label{toy}
We first demonstrate our idea using a two-dimensional toy example where data lie on a spiral. We train three CoopFlow models with different lengths of Langevin flows. As shown in Figure~\ref{fig:toy_example}, the rightmost box shows the results obtained %long-run MCMC setting where we use
with 10,000 Langevin steps. We can see that both the normalizing flow $q_{\alpha}$ and the EBM $p_{\theta}$ can fit the ground truth density, which is displayed in the red box, perfectly. This validates that, with a sufficiently long Langevin flow, the CoopFlow algorithm can learn both a valid $q_{\alpha}$ and a valid $p_{\theta}$. The leftmost green box represents the model trained with 100 Langevin steps. This is a typical non-convergent short-run MCMC setting. In this case, neither $q_{\alpha}$ nor $p_{\theta}$ is valid, but their cooperation is. The short-run Langevin dynamics toward the EBM actually works as a flow-like generator that modifies the initial proposal by the normalizing flow. We can see that the samples from $q_{\alpha}$ are not perfect, but after the modification, the samples from $\pi_{(\theta, \alpha)}$ fit the ground truth distribution very well. The third box shows the results obtained with 500 Langevin steps. This is still a short-run setting, even though it uses more Langevin steps. $p_{\theta}$ becomes better than that with 100 steps, but it is still invalid. With an increased number of Langevin steps, samples from both $q_{\alpha}$ and $\pi_{(\theta, \alpha)}$ are improved and comparable to those in the long-run setting with 10,000 steps. The results verify that the CoopFlow might learn a biased BEM and a biased normalizing flow if the Langevin flow is non-convergent. However, the non-convergent Langevin flow together with the biased normalizing flow can still form a valid generator that synthesizes~valid~examples.
\begin{figure*}[ht]
\centering
%\includegraphics[width=1.\textwidth]{images/toy_example/toy_cam_ready.png}
\includegraphics[width=1\textwidth]{images/final_edit/toy_newpng.png}
\caption{Learning CoopFlows on two-dimensional data. The ground truth data distribution is~shown in the red box and the models trained with different Langevin steps are in the green boxes. In each green box, the first row shows the learned distributions of the normalizing flow and the EBM, and the second row shows the samples from the learned normalizing flow and the~learned~CoopFlow.}
\label{fig:toy_example}\vspace{-0.1in}
\end{figure*}
\subsection{Image Generation} \label{generation}
We test our model on three image datasets for image synthesis. (i) CIFAR-10 \citep{krizhevsky2009learning} is a dataset containing 50k training images and 10k testing images in 10 classes; (ii) SVHN \citep{netzer2011reading} is a dataset containing over 70k training images and over 20k testing images of house numbers; (iii) CelebA \citep{liu2015faceattributes} is a celebrity facial dataset containing over 200k images. We downsample all the images to the resolution of $32 \times 32$. For our model, we show results of three different settings. \textit{CoopFlow(T=30)} denotes the setting where we train a normalizing flow and a Langevin flow together from scratch and use 30 Langevin steps. \textit{CoopFlow(T=200)} denotes the setting where we increase the number of Langevin steps to 200. In the \textit{CoopFlow(Pre)} setting, we first pretrain a normalizing flow from observed data, and then train the CoopFlow with the parameters of the normalizing flow being initialized by the pretrained one.
%flow as initialization and train normalizing flow and EBM together using Algorithm {alg1}.
We use a 30-step Langevin flow in this setting. For all the three settings, we slightly increase the Langevin step size at the testing stage for better performance. %, which corresponds to cooling-down the energy term during sampling.
We show both qualitative results in Figure~\ref{fig:gen_samples} and quantitative results in Table~\ref{tab:fid}. To calculate FID \citep{heusel2017gans} scores, we generate 50,000 samples on each dataset. Our models outperform most of the baseline algorithms. We get lower FID scores comparing to the individual normalizing flows and the prior works that jointly train a normalizing flow with an EBM, e.g., \citet{gao2020flow} and \citet{nijkamp2020learning}. We also~achieve comparable results with the state-of-the-art EBMs. %Among our different settings,
We can see using more Langevin steps or a pretrained normalizing flow can help improve the performance of the CoopFlow. The former enhances the expressive power, while the latter stabilizes the training. %, and using can stabilize the training of the CoopFlow. %provide the lowest FID scores.
% among all the three settings.
More experimental details and results can be found in Appendix.
\begin{figure*}[t]
\centering
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/load_pretrain/cifar_ebm.png}
\caption{CIFAR-10}\label{fig:image21}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/load_pretrain/SVHN_ebm.png}
\caption{SVHN}\label{fig:image22}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/load_pretrain/celeba_ebm.png}
\caption{CelebA}\label{fig:image23}
\end{subfigure}
\caption{Generated examples ($32 \times 32$ pixels) by the CoopFlow models trained on the CIFAR-10, SVHN and CelebA datasets respectively. Samples are obtained from the setting of \textit{CoopFlow(pre)}.}
\label{fig:gen_samples}
\end{figure*}
\begin{table}[h]
\centering
\begin{subtable}{0.49\textwidth}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{lll}
\toprule
\textbf{ } & \textbf{Models} & \textbf{FID} $\downarrow$ \\
\midrule
VAE & VAE \citep{kingma2013auto} & 78.41 \\
\midrule
\multirow{2}{*}{Autoregressive} & PixelCNN \citep{salimans2017pixelcnn++} & 65.93 \\
& PixelIQN \citep{ostrovski2018autoregressive} & 49.46 \\
\midrule
\multirow{3}{*}{GAN} & WGAN-GP\citep{gulrajani2017improved} & 36.40 \\
& SN-GAN \citep{miyato2018spectral} & 21.70\\
& StyleGAN2-ADA \citep{karras2020training} & \; 2.92\\
\midrule
\multirow{3}{*}{Score-Based} & NCSN \citep{song2019generative} & 25.32 \\
& NCSN-v2 \citep{song2020improved} & 31.75 \\
%& DDPM \citep{ho2020denoising} & \;3.17\\
& NCSN++ \citep{YangSong21} & \; 2.20 \\
\midrule
\multirow{2}{*}{Flow} & Glow \citep{kingma2018glow} & 45.99\\
& Residual Flow \citep{chen2019residual} & 46.37\\
\midrule
\multirow{12}{*}{EBM} & LP-EBM \citep{pang2020learning} & 70.15\\
& EBM-SR \citep{nijkamp2019learning} & 44.50\\
& EBM-IG \citep{du2019implicit} & 38.20 \\
& CoopVAEBM \citep{XieZL21} & 36.20 \\
& CoopNets \citep{XieLGZW20} & 33.61 \\
& Divergence Triangle \citep{han2020joint} & 30.10 \\
& VARA \citep{Will21} & 27.50 \\
& EBM-CD \citep{Du21} & 25.10 \\
& GEBM \citep{arbel2020generalized} & 19.31 \\
& CF-EBM \cite{zhao2020learning} & 16.71 \\
& VAEBM \citep{xiao2020vaebm} & 12.16 \\
& EBM-Diffusion \citep{ruiqi21} & \; 9.60\\
\midrule
\multirow{2}{*}{Flow+EBM} & NT-EBM \citep{nijkamp2020learning} & 78.12 \\
& EBM-FCE \citep{gao2020flow} & 37.30 \\
\midrule
\multirow{3}{*}{Ours} & CoopFlow(T=30) & 21.16 \\
& CoopFlow(T=200) & 18.89 \\
& CoopFlow(Pre) & 15.80 \\
\bottomrule
\end{tabular}}
\caption{CIFAR-10}
\end{subtable}
\hspace{-0.1em}
\begin{minipage}{0.36\textwidth}
\begin{subtable}{\textwidth}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{ll}
\toprule
\textbf{Models} & \textbf{FID} $\downarrow$ \\
\midrule
ABP \citep{HanLZW17} & 49.71 \\
ABP-SRI \citep{NijkampP0ZZW20} & 35.23 \\
ABP-OT \citep{AnXL21} & 19.48 \\
VAE \citep{kingma2013auto} & 46.78 \\
2sVAE \citep{dai2019diagnosing} & 42.81 \\
RAE \citep{ghosh2019variational} & 40.02\\
Glow \citep{kingma2018glow} & 41.70 \\
DCGAN \citep{radford2015unsupervised} & 21.40 \\
NT-EBM \citep{nijkamp2020learning} & 48.01 \\
LP-EBM \citep{pang2020learning} & 29.44 \\
EBM-FCE \citep{gao2020flow} & 20.19 \\
\midrule
CoopFlow(T=30) & 18.11 \\
CoopFlow(T=200) & 16.97 \\
CoopFlow(Pre) & 15.32 \\
\bottomrule
\end{tabular}}
\caption{SVHN}
\end{subtable}
\begin{subtable} {\textwidth}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{ll}
\toprule
\textbf{Models} & \textbf{FID} $\downarrow$ \\
\midrule
ABP \citep{HanLZW17} & 51.50 \\
ABP-SRI \citep{NijkampP0ZZW20} & 36.84 \\
VAE \citep{kingma2013auto} & 38.76 \\
Glow \citep{kingma2018glow} & 23.32 \\
DCGAN \citep{radford2015unsupervised} & 12.50 \\
EBM-FCE \citep{gao2020flow} & 12.21 \\
GEBM \citep{arbel2020generalized} & \; 5.21\\
\midrule
CoopFlow(T=30) & \; 6.44 \\
CoopFlow(T=200) & \; 4.90 \\
CoopFlow(Pre) & \; 4.15 \\
\bottomrule
\end{tabular}}
\caption{CelebA}
\end{subtable}
\end{minipage}
\caption{FID scores on the CIFAR-10, SVHN and CelebA datasets ($32\times32$~pixels).
}
\label{tab:fid}
\end{table}
\newpage
\subsection{Image Reconstruction and Inpainting} \label{reconstruction}
We show that the learned CoopFlow model is able to reconstruct observed images. We may consider the CoopFlow model $\pi_{(\theta,\alpha)}(x)$ a latent variable generative model: $z \sim q_0(z);\hat{x}=g_{\alpha}(z); x=F_{\theta}(\hat{x}, e)$, where $z$ denotes the latent variables, $e$ denotes all the injected noises in the Langevin flow, and $F_{\theta}$ denotes the mapping realized by a $T$-step Langevin flow that is actually a $T$-layer noise-injected residual network. Since the Langevin flow is not mixing, $x$ is dependent on $\hat{x}$ in the Langevin flow, thus also dependent on $z$. The CoopFlow is a generator $x=F_{\theta}(g_{\alpha}(z),e)$, so we can reconstruct any $x$ by inferring the corresponding latent variables $z$ using gradient descent on $L(z)=||x-F_{\theta}(g_{\alpha}(z),e)||^2$, with $z$ being initialized by $q_0$. However, $g$ is an invertible transformation, so we can infer $z$ by an efficient way, i.e., we first find $\hat{x}$ by gradient descent on $L(\hat{x})=||x-F_{\theta}(\hat{x},e)||^2$, with $\hat{x}$ being initialized by $\hat{x}_0=g_{\alpha}(z)$ where $z \sim q_{0}(z)$ and $e$ being set to be 0, and then use $z=g_{\alpha}^{-1}(\hat{x})$ to get the latent variables. These two methods are equivalent, but the latter one is computationally efficient, since computing the gradient on the whole two-flow generator $F_{\theta}(g_{\alpha}(z),e)$ is difficult and time-consuming. %Given an image $x$, we can find its $\hat{x}$ by using gradient descent on $L(\hat{x})=||x-F_{\theta}(\hat{x},e)||^2$, with $\hat{x}$ being initialized by $\hat{x}_0=g_{\alpha}(z)$ where $z \sim p_{0}(z)$ and $e$ being set to be 0.
Let $\hat{x}^{*}=\arg \min_{\hat{x}} L(\hat{x})$. The reconstruction is given by $F_{\theta}(\hat{x}^{*})$.
The optimization is done using 200 steps of gradient descent over $\hat{x}$.
\begin{wrapfigure}{R}{0.19\textwidth}
\vspace{-0.05in}
\begin{minipage}{0.18\textwidth}
\begin{center}
\includegraphics[width=0.85\textwidth]{images/recons_interp/recons2_long.png}
\end{center}
\caption{Image reconstruction on the CIFAR-10.}
\label{fig:recons}
\end{minipage}
\end{wrapfigure}
The reconstruction results are shown in Figure~\ref{fig:recons}. We use the CIFAR-10 testing set. The right column displays the original images $x$ that need to be reconstructed. The left and the middle columns display $\hat{x}^{*}$ and $F_{\theta}(\hat{x}^{*})$, respectively.
%The left column is the final reconstructed input to Langevin Flow ($x_0^*$) and the middle column is the reconstructed image output after the Langevin updates ($x_T^*$).
Our model can successfully reconstruct the observed images, %The final optimized $x_0^*$ tends to be a blurred and noisy version of the target image while $x_T^*$ becomes much clear.
verifying that the CoopFlow with a non-mixing MCMC is indeed a valid latent variable model.
We further show that our model is also capable of doing image inpainting. Similar to image reconstruction, given a masked observation $x_{\text{mask}}$ along with a binary matrix $M$ indicating the positions of the unmasked pixels, we optimize $\hat{x}$ to minimize the reconstruction error between $F_{\theta}(\hat{x})$ and $x_{\text{mask}}$ in the unmasked area, i.e.,
$L(\hat{x})=||M\odot(x_{\text{mask}}-F_{\theta}(\hat{x}))||^2$, where $\odot$ is the element-wise multiplication operator. $\hat{x}$ is still initialized by the normalizing flow. We do experiments on the CelebA training set. In Figure~\ref{fig:inpaint}, each row shows one example of inpainting with a different initialization of $\hat{x}$ provided by the normalizing flow and the first 17 columns display the inpainting results at different optimization iterations. The last two columns show the masked images and the original images respectively. We can see that our model can reconstruct the unmasked areas faithfully and simultaneously fill in the blank areas of the input images. With different initializations, our model can inpaint diversified and meaningful patterns.
\begin{figure}[ht!]
\centering
%\includegraphics[width=0.85\linewidth]{images/recons_interp/Impaint3.png}
\includegraphics[width=0.85\linewidth]{images/final_edit/inpaint_new.png}
\caption{Image inpainting on the CelebA dataset ($32 \times 32$ pixels). %Here we show the results of reconstructing a masked image using different initializations.
Each row represents one different initialization. %The rightmost column is the original image. The second column to the right is the input masked image.
The last two columns display the masked and original images respectively. From the 1st column to the 17th column, we show the inpainted images at different optimization iterations.}
\label{fig:inpaint}
\end{figure}
\subsection{Interpolation in the Latent Space} \label{interpolation}
The CoopFlow model is capable of doing interpolation in the latent space $z$. Given an image $x$, we first find its corresponding $\hat{x}^{*}$ using the reconstruction method described in Section~\ref{reconstruction}. We then infer $z$ by the inversion of the normalizing flow $z^{*}=g_{\alpha}^{-1}(\hat{x}^{*})$.
Figure \ref{fig:interploation} shows two examples of interpolation between two latent vectors inferred from observed images. For each row, the images at the two ends
are observed. Each image in the middle is obtained by first interpolating the
latent vectors of the two end images, and then generating the image using the CoopFlow generator, %which involves a normalizing flow mapping followed by 30 steps of Langevin updates.
This experiment shows that the CoopFlow can learn a smooth latent space that
traces the data manifold.
\begin{figure}[h!]
\centering
%\includegraphics[width=0.85\linewidth]{images/recons_interp/Interpolate2.png}
\includegraphics[width=0.85\linewidth]{images/final_edit/interpolation_new.png}
\caption{Image interpolation results on the CelebA dataset ($32 \times 32$ pixels). The leftmost and rightmost columns display the images we observed. The columns in the middle represent the interploation results between the inferred latent vectors of the two end observed images.}
\label{fig:interploation}
\end{figure}
\section{Conclusion}
This paper studies an interesting problem of learning two types of deep flow models in the context of energy-based framework for image representation and generation. One is the normalizing flow that generates synthesized examples by transforming Gaussian noise examples through a sequence of invertible transformations, while the other is the Langevin flow that generates synthesized examples by running a non-mixing, non-convergent short-run MCMC toward an EBM. We propose the CoopFlow algorithm to train the short-run Langevin flow model jointly with the normalizing flow serving as a rapid initializer in a cooperative manner. The experiments show that the CoopFlow is a valid generative model that can be useful for image generation, reconstruction, and interpolation.
\subsubsection*{Acknowledgments}
The authors sincerely thank Dr. Ying Nian Wu for the helpful discussion on the information geometry part and thank Dr. Yifei Xu for his assistance with some extra experiments during the rebuttal. The authors would also like to thank the anonymous reviewers of ICLR’22 program committee for providing
constructive comments and suggestions to improve the work.
%\clearpage\newpage
\bibliography{iclr2022_conference}
\bibliographystyle{iclr2022_conference}
\clearpage\newpage
\appendix
\section{Appendix}
% You may include other additional sections here.
\subsection{Network Architecture of the CoopFlow}
We use the same network architecture for all the experiments. For the normalizing flow $g_{\alpha}(z)$ in our CoopFlow framework, we use the Flow++ \citep{ho2019flow++} network architecture that was originally designed for the CIFAR-10 ($32 \times 32$ pixels) dataset in \citet{ho2019flow++}. As to the EBM in our CoopFlow, we use the architecture shown in Table~\ref{tab:model arch} to design the negative energy function $f_{\theta}(x)$.
%Our code is based on the Pytorch implementation online \citep{Flow++}.
\begin{table}[h!]
\resizebox{0.44\textwidth}{!}{
\begin{subtable}{0.44\textwidth}
\begin{tabular}{l}
\toprule
$3\times3$ Conv2d, \textit{str}=1, \textit{pad}=1, \textit{ch}=128; Swish \\
\midrule
Residual Block, \textit{ch}=256\\
\midrule
Residual Block, \textit{ch}=512\\
\midrule
Residual Block, \textit{ch}=1,024\\
\midrule
$3\times3$ Conv2d, \textit{str}=4, \textit{pad}=0, \textit{ch}=100; Swish \\
\midrule
Sum over channel dimension \\
\bottomrule
\end{tabular}
\caption{EBM architecture}
\end{subtable}
}
%\hfill
\hspace{0.75cm}
\resizebox{0.34\textwidth}{!}{
\begin{subtable}{0.34\textwidth}
\centering
\begin{tabular}{l}
\toprule
$3\times3$ Conv2d, \textit{str}=1, \textit{pad}=1; Swish \\
\midrule
$3\times3$ Conv2d, \textit{str}=1, \textit{pad}=1; Swish \\
\midrule
+ $3\times3$ Conv2d, \textit{str}=1, \textit{pad}=1; Swish (input) \\
\midrule
$2\times2$ Average pooling \\
\bottomrule
\end{tabular}
\caption{Residual Block architecture}
\end{subtable}}
\caption{Network architecture of the EBM in the CoopFlow (\textit{str}: stride, \textit{pad}: padding, \textit{ch}: channel).}
\label{tab:model arch}
\end{table}
\subsection{Experimental Details}
We have three different settings for the CoopFlow model. In the \textit{CoopFlow(T=30)} setting and the \textit{CoopFlow(T=200)} setting, we train both the normalizing flow and the Langevin flow from scratch. The difference between them are only the number of the Langevin steps. The \textit{CoopFlow(T=200)} uses a longer Langevin flow than the \textit{CoopFlow(T=30)}. We follow \cite{ho2019flow++} and use the data-dependent parameter initialization method \citep{salimans2016weight} for our normalizing flow in both settings \textit{CoopFlow(T=30)} and \textit{CoopFlow(T=200)}. On the other hand, as to the \textit{CoopFlow(Pre)} setting, we first pretrain a normalizing flow on training examples, and then train a 30-step Langevin flow, whose parameters are initialized randomly, together with the pretrained normalizing flow by following Algorithm~\ref{alg1}. The cooperation between the pretrained normalizing flow and the untrained Langevin flow would be difficult and unstable because the Langevin flow is not knowledgeable at all to teach the normalizing flow. To stabilize the cooperative training and make a smooth transition for the normalizing flow, we include a warm-up phase in the CoopFlow algorithm. During this phase, instead of updating both the normalizing flow and the Langevin flow, we fix the parameters of the pretrained normalizing flow and only update the parameters of the Langevin flow. After a certain number of learning epochs, the Langevin flow may get used to the normalizing flow initialization, and learn to cooperate with it. Then we begin to update both two flows as described in Algorithm~\ref{alg1}. This strategy is effective in preventing the Langevin flow from generating bad synthesized examples at the beginning of the CoopFlow algorithm to ruin the pretrained normalizing~flow.
We use the Adam optimizer \citep{kingma2014adam} for training. We set learning rates $\eta_{\alpha}=0.0001$ and $\eta_{\theta}=0.0001$ for the normalizing flow and the Langevin flow, respectively. We use $\beta_1=0.9$ and $\beta_2=0.999$ for the normalizing flow and $\beta_1=0.5$ and $\beta_2=0.5$ for the Langevin flow. In the Adam optimizer, $\beta_1$ is the exponential decay rate for the first moment estimates, and $\beta_2$ is the exponential decay rate for the second-moment estimates. %We keep monitoring the FID scores during training and pick the best one over the training trajectory.
We adopt random horizontal flip as data augmentation only for the CIFAR-10 dataset. We remove the noise term in each Langevin update by following \citet{zhao2020learning}. We also propose an alternative strategy to gradually decay the effect of the noise term in Section \ref{sec:decay}. The batch sizes for settings \textit{CoopFlow(T=30)}, \textit{CoopFlow(T=200)}, and \textit{CoopFlow(Pre)} are 28, 32 and 28. The values of other hyperparameters can be found in Table~\ref{tab:hyperparam}.
\begin{table}[h!]
\centering
\begin{adjustbox}{width=.99\columnwidth, center}
\begin{tabular}{cccccccc}
\toprule
\multirow{3}{*}{Model} & \multirow{3}{*}{Dataset} & \# of epochs & \# of warm-up & \# of epochs & \# of & Langevin & Langevin \\
& & to pretrain & epochs for & for & Langevin & step size & step size \\
& & Normal. Flow & Lang. Flow & CoopFlow & steps & (train) & (test) \\
%& & flow & flow & training & & &\\
\midrule
\multirow{3}{*}{\shortstack{CoopFlow\\(T=30)}} & CIFAR-10 & 0 & 0 & 100 & 30 & 0.03 & 0.04\\
& SVHN & 0 & 0 & 100 & 30 & 0.03 & 0.035\\
& CelebA & 0 & 0 & 100 & 30 & 0.03 & 0.035\\
\midrule
\multirow{3}{*}{\shortstack{CoopFlow\\(T=200)}} & CIFAR-10 & 0 & 0 & 100 & 200 & 0.01 & 0.012\\
& SVHN & 0 & 0 & 100 & 200 & 0.011 & 0.0125\\
& CelebA & 0 & 0 & 100 & 200 & 0.011 & 0.013\\
\midrule
\multirow{3}{*}{\shortstack{CoopFlow\\(Pre)}} & CIFAR-10 & 300 & 25 & 75 & 30 & 0.03 & 0.04\\
& SVHN & 200 & 10 & 90 & 30 & 0.03 & 0.035\\
& CelebA & 80 & 10 & 90 & 30 & 0.03 & 0.035\\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Hyperparameter setting of the CoopFlow in our experiments.}
\label{tab:hyperparam}
\end{table}
\subsection{Analysis of Hyperparameters of Langevin Flow}
We investigate the influence of the Langevin step size $\delta$ and the number of Langevin steps $T$ on the CIFAR-10 dataset using the \textit{CoopFlow(Pre)} setting. We first fix the Langevin step size to be 0.03 and vary the number of Langevin steps from 10 to 50. The results are shown in Table~\ref{tab:Num_Lange}. On the other hand, we show the influence of the Langevin step size in Table~\ref{tab:step_size}, where we fix the number of Langevin steps to be 30 and vary the Langevin step size used in training. When synthesizing examples from the learned models in testing, we slightly increase the Langevin step size by a ratio of $4/3$ for better performance. We can see that our choices of 30 as the number of Langevin steps and 0.03 as the Langevin step size are reasonable.
Increasing the number of Langevin steps can improve the performance in terms of FID, but also be computationally expensive. The choice of $T=30$ is a trade-off between the synthesis performance and the computation efficiency.
\begin{table}[h!]
\centering
%\begin{adjustbox}{width=.53\columnwidth, center}
\begin{tabular}{cccccc}
\toprule
\# of Langevin steps & 10 & 20 & 30 & 40 & 50 \\
\midrule
FID $\downarrow$ & 16.46 & 15.20 & 15.80 & 16.80 & 15.64 \\
\bottomrule
\end{tabular}
%\end{adjustbox}
\caption{FID scores over the numbers of Langevin steps of \textit{CoopFlow(Pre)} on the CIFAR-10 dataset.}
\label{tab:Num_Lange}
\end{table}
\begin{table}[h!]
\centering
%\begin{adjustbox}{width=.6\columnwidth, center}
\begin{tabular}{ccccccc}
\toprule
Langevin step size (train) & 0.01 & 0.02 & 0.03 & 0.04 & 0.05 & 0.10\\
\midrule
Langevin step size (test) & 0.013 & 0.026 & 0.04 & 0.053 & 0.067 & 0.13\\
\midrule
FID $\downarrow$ & 15.99 & 16.32 & 15.80 & 16.52 & 18.17 & 19.82\\
\bottomrule
\end{tabular}
%\end{adjustbox}
\caption{FID scores of \textit{CoopFlow(Pre)} on the CIFAR-10 dataset under different Langevin step sizes.}
\label{tab:step_size}
\end{table}
\subsection{Ablation Study}
To show the effect of the cooperative training, we compare a CoopFlow model with an individual normalizing flow and an individual Langevin flow. For fair comparison, the normalizing flow component in the CoopFlow has the same network architecture as that in the individual normalizing flow, while the Langevin flow component in the CoopFlow also uses the same network architecture as that in the individual Langevin flow.
%Here we compare the results of training normalizing flow and Langevin flow using or Cooperative algorithm with training them separately.
We train the individual normalizing flow by following \citet{ho2019flow++}
%the original normalizing flow training algorithm (we use the flow++ \citep{ho2019flow++} structure here, which is the same as the one used in our CoopFlow model).
and train the individual Langevin flow by following \citet{nijkamp2019learning}. %Through our experiments, we find that removing the noise injection to both observed data and Langevin updates can give us better result.
All three models are trained on the CIFAR-10 dataset. We present a comparison of these three models in terms of FID in Table~\ref{tab:fid_ablation}, and also show generated samples in Figure~\ref{fig:gen_samples_ablation}. From Table ~\ref{tab:fid_ablation}, we can see that the CoopFlow model outperforms both the normalizing flow and the Langevin flow by a large margin, which verifies the effectiveness of the proposed CoopFlow algorithm.
\begin{table}[h!]
\centering
%\begin{adjustbox}{width=.4\columnwidth, center}
\begin{tabular}{cccc}
\toprule
Model & Normalizing flow & Langevin flow & CoopFlow\\
\midrule
FID $\downarrow$ & 92.10 & 49.51 & 21.16\\
\bottomrule
\end{tabular}
%\end{adjustbox}
\caption{A FID comparison among the normalizing flow, the Langevin flow and the CoopFlow.
}
\label{tab:fid_ablation}
\end{table}
\begin{figure}[h!]
%\vspace{-0.15in}
\centering
%\begin{adjustbox}{width=.9\columnwidth, center}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/Ablation/ablation_flow_out.png}
\caption{Normalizing flow}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/Ablation/short_run_ablation.png}
\caption{Langevin flow}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/scratch/Cifar10.png}
\caption{CoopFlow(T=30)}
\end{subfigure}
%\end{adjustbox}
%\vspace{-0.1in}
\caption{Generated examples by (a) the individual normalizing flow, (b) the individual Langevin flow, and (c) the CoopFlow, which are trained on the CIFAR-10 dataset.}
\label{fig:gen_samples_ablation}%\vspace{-0.4in}
\end{figure}
%\newpage\clearpage
\subsection{More Image Generation Results}
In Section~\ref{generation}, we have shown generated examples from the \textit{CoopFlow(Pre)} model. In this section, we show examples generated by the \textit{CoopFlow(T=30)} model in Figure~\ref{fig:gen_samples_scrath} and examples generated by the \textit{CoopFlow(T=200)} in Figure~\ref{fig:gen_samples_long}. We can see that all the generated examples are meaningful. On the other hand, we show the examples generated from the normalizing flow component of the \textit{CoopFlow(T=30)} model in Figure~\ref{fig:flow_ebm}. By comparing the synthesized images shown in Figure~\ref{fig:gen_samples_scrath} and those in Figure~\ref{fig:flow_ebm}, we observe that there is an obvious visual gap between the normalizing flow and the CoopFlow. The samples from the normalizing flow look blurred but become sharp and clear after the Langevin flow revision. This supports our claim made in Section~\ref{theory_understand}.
\begin{figure}[h!]
\centering
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/scratch/Cifar10.png}
\caption{CIFAR-10}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/scratch/SVHN.png}
\caption{SVHN}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/scratch/Celeba32.png}
\caption{CelebA}
\end{subfigure}
\caption{Generated examples ($32 \times 32$ pixels) by the CoopFlow models trained on the CIFAR-10, SVHN and CelebA datasets respectively. Samples are obtained from the \textit{CoopFlow(T=30)} setting.}
\label{fig:gen_samples_scrath}
\end{figure}
\begin{figure}[h!]
\centering
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/long_run/cifar10_ebm87_12_eval.png}
\caption{CIFAR-10}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/long_run/SVHN_ebm46_125.png}
\caption{SVHN}
\end{subfigure}
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/long_run/celeba_ebm30_13.png}
\caption{CelebA}
\end{subfigure}
\caption{Generated examples ($32 \times 32$ pixels) by the CoopFlow models trained on the CIFAR-10, SVHN and CelebA datasets respectively. Samples are obtained from the \textit{CoopFlow(T=200)} setting.}
\label{fig:gen_samples_long}
\end{figure}
\begin{figure}[h!]
\centering
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/scratch/cifar_flow.png}
\caption{CIFAR-10}
\end{subfigure}
%\hfill
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/scratch/SVHN_flow35.png}
\caption{SVHN}
\end{subfigure}
%\hfill
\hspace{-0.1em}
\begin{subfigure}{.29\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/scratch/Celeba_flow44.png}
\caption{CelebA}
\end{subfigure}
\caption{Generated examples by the normalizing flow components in the CoopFlow models trained on the CIFAR-10, SVHN, and CelebA datasets respectively. The CoopFlow models are trained in the \textit{CoopFlow(T=30)} setting.}
\label{fig:flow_ebm}
\end{figure}
\subsection{More Image Inpainting Results}
We show more image inpainting results in Figure~\ref{fig:more_inpaint}. Each panel corresponds to one inpainting task. Each row in each panel represents one inpainting result with a different initialization. In each panel, the last two columns show the masked and the ground truth images respectively, and images from the 1st column to the 20th column are the inpainted images at different optimization iterations.
\begin{figure}[h]
\centering
\begin{subfigure}{.9\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/recons_interp/impainted_diversity15.png}
\end{subfigure}
\begin{subfigure}{.9\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/recons_interp/impainted_diversity19.png}
\end{subfigure}
\caption{More results on image inpainting.}
\label{fig:more_inpaint}
\end{figure}
\newpage\clearpage
\subsection{FID Curve over Training Epochs}
Figure~\ref{fig:fid_curve} shows the FID trends during the training of the CoopFlow models on the CIFAR-10 dataset. Each curve represents the FID scores over training epochs. We train the CoopFlow models using the settings of \textit{CoopFlow(T=30)} and \textit{CoopFlow(pre)}. For each of them, we can observe that, as the cooperative learning proceeds, the FID keeps decreasing and converges to a low value. This is an empirical evidence to show that the proposed CoopFlow algorithm is a convergent algorithm.
\begin{figure}[ht]
\centering
\begin{subfigure}{.45\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/Fig12a.pdf}
\caption{FID curve for \textit{CoopFlow(T=30)}.}
\label{fig:FID_coop}
\end{subfigure}
\hspace{-0.2cm}
\begin{subfigure}{.45\linewidth}
\centering
\includegraphics[width=0.99\textwidth]{images/Fig12b.pdf}
\caption{FID curve for \textit{CoopFlow(Pre)}.}
\label{fig:FID_coop_pre}
\end{subfigure}
\caption{FID curves on the CIFAR-10 dataset. The FID score is reported every 5 epochs.}
\label{fig:fid_curve}
\end{figure}
\subsection{Quantitative Results For Image Reconstruction}
We provide additional quantitative results for the image reconstruction experiment in Section \ref{reconstruction}. Following \cite{nijkamp2019learning}, we calculate the per-pixel mean squared error (MSE) on 1,000 examples in the testing set of the CIFAR-10 dataset. We use a 200-step gradient descent to minimize the reconstruction loss. We plot the reconstruction error curve showing the MSEs over iterations in Figure~\ref{fig:error_curve} and report the final per-pixel MSE in Table~\ref{tab:recons_err}. For a baseline, we train an EBM using a 100-step short-run MCMC and the resulting model is the short-run Langevin flow. We then apply it to the reconstruction task of the same 1,000 images by following \cite{nijkamp2019learning}. The baseline EBM has the same network architecture as that of the EBM component in our CoopFlow model for fair comparison. The experiment results show that the CoopFlow works better than the individual short-run Langevin flow in this image reconstruction task.
\begin{table}[h!]
\centering
\begin{tabular}{ll}
\toprule
Model & MSE \\
\midrule
Langevin Flow / EBM with short-run MCMC & 0.1083 \\
CoopFlow (ours) & 0.0254 \\
\bottomrule
\end{tabular}
\caption{Reconstruction error (MSE per pixel).}
\label{tab:recons_err}
\end{table}
\begin{figure}[h!]
\centering
\includegraphics[width=0.45\textwidth]{images/Fig13.pdf}
\caption{Reconstruction errors (MSE per pixel) over iterations.}
\label{fig:error_curve}
\end{figure}
%\color{black}
\newpage
\subsection{Model Complexity}
%\color{purple}
In Table~\ref{tab:model_size}, we present a comparison of different models in terms of model size and FID score. Here we mainly compare those models that have a normalizing flow component, e.g., EBM-FCE, NT-EBM, GLOW, Flow++, as well as an EBM jointly trained with a VAE generator, e.g., VAEBM. We can see the CoopFlow model has a good balance between model complexity and performance. It is noteworthy that both the CoopFlow and the EBM-FCE consist of an EBM and a normalizing flow, and their model sizes are also similar, but the CoopFlow achieves a much lower FID than the EBM-FCE. Note that the Flow++ baseline uses the same structure as that in our CoopFlow. By comparing the Flow++ and the CoopFlow, we can find that recruiting an extra Langevin flow can help improve the performance of the normalizing flow in terms of FID. On the other hand, although the VAEBM model achieves a better FID than ours, but it relies on a much larger pretrained NVAE model \citep{NVAE} that significantly increases its model complexity. %, which is much larger than others.
\begin{table}[ht]
\centering
\begin{tabular}{lll}
\toprule
Model & \# of Parameters & FID $\downarrow$\\
\midrule
NT-EBM \citep{nijkamp2020learning} & 23.8M & 78.12 \\
GLOW \citep{kingma2018glow} & 44.2M & 45.99 \\
EBM-FCE \citep{gao2020flow} & 44.9M & 37.30 \\
Flow++ \citep{ho2019flow++} & 28.8M & 92.10 \\
VAEBM \citep{xiao2020vaebm} & 135.1M & 12.16 \\
CoopFlow(T=30) (ours) & 45.9M & 21.16 \\
CoopFlow(T=200) (ours) & 45.9M & 18.89 \\
CoopFlow(pre) (ours) & 45.9M & 15.80 \\
\bottomrule
\end{tabular}
\caption{A comparison of model sizes and FID scores among different models. FID scores are reported on the CIFAR-10 dataset.}
\label{tab:model_size}
\end{table}
%\color{black}
\subsection{Comparison with Models Using Short-Run MCMC}
Our method is relevant to the short-run MCMC.
In this section, we compare the CoopFlow model with other models that use a short-run MCMC as a flow-like generator. The baselines include (1) the single EBM with short-run MCMC starting from the noise distribution \citep{nijkamp2019learning}, and (ii) cooperative training of an EBM and a generic generator \citep{XieLGZW20}. In Table~\ref{tab:MCMC_2}, we report the FID scores of different methods over different numbers of MCMC steps. With the same number of Langevin steps, the CoopFlow can generate much more realistic image patterns than the other two baselines. Further, the results show that the CoopFlow can use less Langevin steps (i.e., a shorter Langevin flow) to achieve better performance.
\begin{table}[h!]
\centering
\begin{tabular}{llllllll}
\toprule
\backslashbox{Model}{\# of MCMC steps} & 10 & 20 & 30 & 40 & 50 & 200 \\
\midrule
Short-run EBM & 421.3 & 194.88 & 117.02 & 140.79 & 198.09 & 54.23 \\
CoopNets & 33.74 & 33.48 & 34.12 & 33.85 & 42.99 & 38.88 \\
CoopFlow(Pre) (ours)& 16.46 & 15.20 & 15.80 & 16.80 & 15.64 & 17.94 \\
\bottomrule
\end{tabular}
\caption{A comparison of FID scores of the short-run EBM, the CoopNets and the CoopFlow under different numbers of Langevin steps on the CIFAR-10 dataset.}
\label{tab:MCMC_2}
\end{table}
\subsection{Noise Term in the Langevin Dynamics}\label{sec:decay}
While for the experiments shown in the main text, we completely disable the noise term $\delta \epsilon$ of the Langevin equation presented in Eq.~(\ref{eq:MCMC}) by following \citet{zhao2020learning} to achieve better results, here we try an alternative way where we gradually decay the effect of the noise term toward zero during the training process.
The decay ratio for the noise term can be computed by the following:
\begin{equation}
\textrm{decay ratio} = \max ({(1.0 - \frac{\textrm{epoch}}{K})}^{20}, 0.0)
\end{equation}
where $K$ is a hyper-parameter controlling the decay speed of the noise term. Such a noise decay strategy enables the model to do more exploration in the sampling space at the beginning of the training and then gradually focus on the basins of the reachable local modes for better synthesis quality when the model is about to converge. Note that we only decay the noise term during the training stage and still remove the noise term during the testing stage, including image generation and FID calculation. We carry out experiments on the CIFAR-10 and SVHN datasets using the \textit{CoopFlow(Pre)} setting. The results are shown in Table~\ref{tab:reduce_noise}. %We can see that models trained with reducing noise achieve similar or slightly better results than models trained with no noise.
\begin{table}[h]
\centering
\begin{tabular}{llll}
\toprule
Dataset & $K$ & FID \textrm{(no noise)} & FID \textrm{(decreasing noise)}\\
\midrule
CIFAR-10 & 30 & 15.80 & 14.55 \\
SVHN & 15 & 15.32 & 15.74 \\
\bottomrule
\end{tabular}
\caption{FID scores for the CoopFlow models trained with gradually reducing noise term in the Langevin dynamics.}
\label{tab:reduce_noise}
\end{table}
\subsection{Comparison Between CoopFlow and Short-Run EBM via Information Geometry}
Figure~\ref{fig:illustration2} illustrates the convergences of both the CoopFlow and the EBM with a short-run MCMC starting from an initial noise distribution $q_0$. We define the short-run EBM in a more generic form \citep{XieLZW16} as follows
\begin{align}
p_{\theta}(x)=\frac{1}{Z(\theta)}\exp[f_{\theta}(x)]q_0(x), \label{eq:ebm_ref}
\end{align}
which is an exponential tilting of a known reference distribution $q_0(x)$. In general, the reference distribution can be either the Gaussian distribution or the uniform distribution. When the reference distribution is the uniform distribution, $q_0$ can be removed in Eq.~(\ref{eq:ebm_ref}). Since the initial distribution of the CoopFlow is the Gaussian distribution $q_0$, which is actually the prior distribution of the normalizing flow. For a convenient and fair comparison, we will use the Gaussian distribution as the reference distribution of the EBM in Eq.~(\ref{eq:ebm_ref}). And the CoopFlow and the baseline short-run EBM will use the same EBM defined in Eq.~(\ref{eq:ebm_ref}) in their frameworks. We will use $\bar{p}_{\bar{\theta}}$ to denote the baseline short-run EBM and keep using $p_{\theta}$ to denote the EBM component in the CoopFlow.
There are three families of distributions: $\Omega=\{p: \mathbb{E}_{p}[h(x)]=\mathbb{E}_{p_{\text{data}}}[h(x)]\}$, $\Theta=\{p_{\theta}(x)=\exp(\langle \theta,h(x)\rangle)q_{0}(x)/Z(\theta), \forall \theta\}$, and $A=\{q_{\alpha}, \forall \alpha\}$, which are shown by the red, blue and green curves respectively in Figure~\ref{fig:illustration2}, which is an extension of Figure~\ref{fig:illustration} by adding the following elements:
\begin{itemize}
\item $q_0$, which is the initial distribution for both the CoopFlow and the short-run EBM. It belongs to $\Theta$ because it corresponds to $\theta = 0$. $q_0$ is just a noise distribution thus it is far under the green curve. That is, it is very far from $q_{\alpha*}$ because $q_{\alpha*}$ has been already a good approximation of $p_{\theta^{*}}$.
\item $g_{\alpha^*}$, which is the learned transformation of the normalizing flow $q_{\alpha}$, and is visualized as a mapping from $q_0$ to $q_{\alpha^*}$ by a directed brown line segment.
\item The MCMC trajectory of the baseline short-run EBM $\bar{p}_{\bar{\theta}}$, which is shown by the yellow line on the right hand side of the blue curve. The solid part of the yellow line, starting from $q_0$ to $\bar{\pi}^*=\mathcal{K}_{\bar{\theta}^*}q_0$, shows the short-run non-mixing MCMC starting from the initial Gaussian distribution $q_0$ in $\Theta$ and arriving at $\bar{\pi}^*$ in $\Omega$. The dotted part of the yellow line is the potential long-run MCMC trajectory that is unrealized.
\end{itemize}
By comparing the MCMC trajectories of the CoopFlow and the short-run EBM in Figure \ref{fig:illustration2}, we can find that the CoopFlow has a much shorter MCMC trajectory than that of the short-run EBM, since the normalizing flow $g_{\alpha^{*}}$ in the CoopFlow amortizes the sampling workload for the Langevin flow in the CoopFlow model.
%(i) (ii) (iii)
\begin{figure}[t]
\centering
\includegraphics[width=0.5\textwidth]{images/CoopFlow_comp.jpg}
\caption{A comparison between the CoopFlow and the short-run EBM.}
\label{fig:illustration2}\vspace{0.2in}
\end{figure}
\subsection{More Convergence Analysis}
The CoopFlow algorithm simply involves two MLE learning algorithms: (i) the MLE learning of the EBM $p_{\theta}$ and (ii) the MLE learning of the normalizing flow $q_{\alpha}$. The convergence of each of the two learning algorithms has been well studied and verified in the existing literature, e.g., \citet{younes1999convergence,XieLZW16,kingma2018glow}. That is, each of them has a fixed point. The only interaction between these two MLE algorithms in the proposed CoopFlow algorithm is that, in each learning iteration, they feed each other with their synthesized examples and use the cooperatively synthesized examples in their parameter update formulas. To be specific, the normalizing flow uses its synthesized examples to initialize the MCMC of the EBM, while the EBM feeds the normalizing flow with its synthesized examples as training examples. The synthesized examples from the Langevin flow are considered the cooperatively synthesized examples by the two models, and are used to compute their learning gradients. Unlike other amortized sampling methods (e.g., \citet{han2019divergence,Will21}) that uses variational learning, the EBM and normalizing flow in our framework do not back-propagate each other through the cooperatively synthesized examples. They just feed each other with some input data for their own training algorithms. That is, each learning algorithm will still converge to a fixed point.
Now let us analyze the convergence of the whole CoopFlow algorithm that alternates two maximum likelihood learning algorithms. We first analyze the convergence of the objective function at each learning step, and then we conclude the convergence of the whole algorithm.
\textbf{The convergence of CD learning of EBM.} \ The learning objective of the EBM is to minimize the KL divergence between the EBM $p_{\theta}$ and the data distribution $p_{\text{data}}$. Since the MCMC of the EBM in our model is initialized by the normalizing flow $q_{\alpha}$, it follows a modified contrastive divergence algorithm. That is, at iteration $t$, it has the following objective,
\begin{align}
\theta^{(t+1)} = \arg \min_{\theta}
\mathbb{D}_{\text{KL}}(p_{\text{data}}||p_{\theta})-\mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{(t)}} q_{\alpha^{(t)}}|| p_{\theta}).
\end{align}
No matter what kind of distribution is used to initialize the MCMC, it will has a fixed point when the learning gradient of $\theta$ equals to 0, i.e., $L'(\theta)= \frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}f_{\theta}(x_i)-\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}f_{\theta}(\tilde{x}_i)=0$. The initialization of the MCMC only affects the location of the fixed point of the learning algorithm. The convergence and the analysis of the fixed point of the contrastive divergence algorithm has been studied by \citet{Hinton02,Carreira-Perpinan05}.
%\newpage\clearpage
\textbf{The convergence of MLE learning of normalizing flow.} \ The objective of the normalizing flow is to learn to minimize the KL divergence between the normalizing flow and the Langevin flow (or the EBM) because, in each learning iteration, the normalizing flow uses the synthesized examples generated from the Langevin dynamics as training data. At iteration $t$, it has the following objective
\begin{align}
\alpha^{(t+1)} = \arg \min_{\alpha} \mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{(t)}} q_{\alpha^{(t)}}|| q_{\alpha} ),
\end{align}
which is a convergent algorithm at each $t$. The convergence has been studied by \cite{younes1999convergence}.
\textbf{The convergence of CoopFlow.} \ The CoopFlow alternates the above two learning algorithms. The EBM learning seeks to reduce the KL divergence between the EBM and the data, i.e., $p_{\theta} \rightarrow p_{data}$; while the MLE learning of normalizing flow seeks to reduce the KL divergence between the normalizing flow and the EBM, i.e., $q_{\alpha} \rightarrow p_{\theta}$. Therefore, the normalizing flow will chase the EBM toward the data distribution gradually. Because the process $p_{\theta} \rightarrow p_{\text{data}}$ will stop at a fixed point, therefore $q_{\alpha} \rightarrow p_{\theta}$ will also stop at a fixed point obviously. Such a chasing game is a contraction algorithm, therefore the fixed point of the CoopFlow exists. Empirical evidence also support our claim. If we use $(\theta^*,\alpha^*)$ to denote the fixed point of the CoopFlow, according to the definition of a fixed point, $(\theta^*,\alpha^*)$ will satisfy
\begin{align}
\theta^{*} &= \arg \min_{\theta}
\mathbb{D}_{\text{KL}}(p_{\text{data}}||p_{\theta})-\mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{*}} q_{\alpha^{*}}|| p_{\theta} ),\label{eq:fix_points1_supp}\\
\alpha^{*} &= \arg \min_{\alpha} \mathbb{D}_{\text{KL}}(\mathcal{K}_{\theta^{*}} q_{\alpha^{*}}|| q_{\alpha} ).
\label{eq:fix_points2_supp}
\end{align}
The convergence of the cooperative learning framework (CoopNets) that integrates the MLE algorithm of an EBM and the MLE algorithm of a generic generator has been verified and discussed in \citet{XieLGZW20}. The CoopFlow that uses a normalizing flow instead of a generic generator has the same convergence property as that of the original CoopNets. The major contribution in our paper is to start from the above fixed point equation to analyze where the fixed point will be in our learning algorithm, especially when the MCMC is non-mixing and non-convergent. This goes beyond all the prior works about cooperative learning.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{69}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[An et~al.(2021)An, Xie, and Li]{AnXL21}
Dongsheng An, Jianwen Xie, and Ping Li.
\newblock Learning deep latent variable models by short-run {MCMC} inference
with optimal transport correction.
\newblock In \emph{Proceedings of the {IEEE} Conference on Computer Vision and
Pattern Recognition (CVPR)}, pp.\ 15415--15424, Virtual Event, 2021.
\bibitem[Arbel et~al.(2021)Arbel, Zhou, and Gretton]{arbel2020generalized}
Michael Arbel, Liang Zhou, and Arthur Gretton.
\newblock Generalized energy based models.
\newblock In \emph{Proceedings of the 9th International Conference on Learning
Representations (ICLR)}, Virtual Event, 2021.
\bibitem[Barbu \& Zhu(2020)Barbu and Zhu]{barbu2020monte}
Adrian Barbu and Song-Chun Zhu.
\newblock \emph{Monte Carlo Methods}.
\newblock Springer Nature, 2020.
\bibitem[Carreira{-}Perpi{\~{n}}{\'{a}}n \&
Hinton(2005)Carreira{-}Perpi{\~{n}}{\'{a}}n and Hinton]{Carreira-Perpinan05}
Miguel~{\'{A}}. Carreira{-}Perpi{\~{n}}{\'{a}}n and Geoffrey~E. Hinton.
\newblock On contrastive divergence learning.
\newblock In \emph{Proceedings of the 10th International Workshop on Artificial
Intelligence and Statistics (AISTATS)}, Bridgetown, Barbados, 2005.
\bibitem[Chen et~al.(2019)Chen, Behrmann, Duvenaud, and
Jacobsen]{chen2019residual}
Tian~Qi Chen, Jens Behrmann, David Duvenaud, and J{\"{o}}rn{-}Henrik Jacobsen.
\newblock Residual flows for invertible generative modeling.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, pp.\ 9913--9923, Vancouver, Canada, 2019.
\bibitem[Cover \& Thomas(2006)Cover and Thomas]{cover2006elements}
Thomas~M. Cover and Joy~A. Thomas.
\newblock \emph{Elements of information theory, Second Edition}.
\newblock Wiley, 2006.
\bibitem[Dai \& Wipf(2019)Dai and Wipf]{dai2019diagnosing}
Bin Dai and David~P. Wipf.
\newblock Diagnosing and enhancing {VAE} models.
\newblock In \emph{Proceeding of the 7th International Conference on Learning
Representations (ICLR)}, New Orleans, LA, 2019.
\bibitem[Dinh et~al.(2015)Dinh, Krueger, and Bengio]{DinhKB14}
Laurent Dinh, David Krueger, and Yoshua Bengio.
\newblock {NICE:} non-linear independent components estimation.
\newblock In \emph{Workshop Proceedings of the 3rd International Conference on
Learning Representations (ICLR Workshop)}, San Diego, CA, 2015.
\bibitem[Dinh et~al.(2017)Dinh, Sohl{-}Dickstein, and Bengio]{DinhSB17}
Laurent Dinh, Jascha Sohl{-}Dickstein, and Samy Bengio.
\newblock Density estimation using real {NVP}.
\newblock In \emph{Proceedings of the 5th International Conference on Learning
Representations (ICLR)}, Toulon, France, 2017.
\bibitem[Du \& Mordatch(2019)Du and Mordatch]{du2019implicit}
Yilun Du and Igor Mordatch.
\newblock Implicit generation and modeling with energy based models.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, pp.\ 3603--3613, Vancouver, Canada, 2019.
\bibitem[Du et~al.(2020)Du, Meier, Ma, Fergus, and Rives]{DuMMFR20}
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives.
\newblock Energy-based models for atomic-resolution protein conformations.
\newblock In \emph{Proceedings of the 8th International Conference on Learning
Representations (ICLR)}, Addis Ababa, Ethiopia, 2020.
\bibitem[Du et~al.(2021)Du, Li, Tenenbaum, and Mordatch]{Du21}
Yilun Du, Shuang Li, Joshua~B. Tenenbaum, and Igor Mordatch.
\newblock Improved contrastive divergence training of energy-based models.
\newblock In \emph{Proceedings of the 38th International Conference on Machine
Learning (ICML)}, pp.\ 2837--2848, Virtual Event, 2021.
\bibitem[Gao et~al.(2018)Gao, Lu, Zhou, Zhu, and Wu]{GaoLZZW18}
Ruiqi Gao, Yang Lu, Junpei Zhou, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Learning generative convnets via multi-grid modeling and sampling.
\newblock In \emph{Proceedings of the 2018 {IEEE} Conference on Computer Vision
and Pattern Recognition (CVPR)}, pp.\ 9155--9164, Salt Lake City, UT, 2018.
\bibitem[Gao et~al.(2020)Gao, Nijkamp, Kingma, Xu, Dai, and Wu]{gao2020flow}
Ruiqi Gao, Erik Nijkamp, Diederik~P. Kingma, Zhen Xu, Andrew~M. Dai, and
Ying~Nian Wu.
\newblock Flow contrastive estimation of energy-based models.
\newblock In \emph{Proceedings of the 2020 {IEEE} Conference on Computer Vision
and Pattern Recognition (CVPR)}, pp.\ 7515--7525, Seattle, WA, 2020.
\bibitem[Gao et~al.(2021)Gao, Song, Poole, Wu, and Kingma]{ruiqi21}
Ruiqi Gao, Yang Song, Ben Poole, Ying~Nian Wu, and Diederik~P. Kingma.
\newblock Learning energy-based models by diffusion recovery likelihood.
\newblock In \emph{Proceeding of the 9th International Conference on Learning
Representations (ICLR)}, Virtual Event, 2021.
\bibitem[Ghosh et~al.(2020)Ghosh, Sajjadi, Vergari, Black, and
Sch{\"{o}}lkopf]{ghosh2019variational}
Partha Ghosh, Mehdi S.~M. Sajjadi, Antonio Vergari, Michael~J. Black, and
Bernhard Sch{\"{o}}lkopf.
\newblock From variational to deterministic autoencoders.
\newblock In \emph{Proceeding of the 8th International Conference on Learning
Representations (ICLR)}, Addis Ababa, Ethiopia, 2020.
\bibitem[Grathwohl et~al.(2020)Grathwohl, Wang, Jacobsen, Duvenaud, Norouzi,
and Swersky]{GrathwohlWJD0S20}
Will Grathwohl, Kuan{-}Chieh Wang, J{\"{o}}rn{-}Henrik Jacobsen, David
Duvenaud, Mohammad Norouzi, and Kevin Swersky.
\newblock Your classifier is secretly an energy based model and you should
treat it like one.
\newblock In \emph{Proceedings of the 8th International Conference on Learning
Representations (ICLR)}, Addis Ababa, Ethiopia, 2020.
\bibitem[Grathwohl et~al.(2021)Grathwohl, Kelly, Hashemi, Norouzi, Swersky, and
Duvenaud]{Will21}
Will~Sussman Grathwohl, Jacob~Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin
Swersky, and David Duvenaud.
\newblock No {MCMC} for me: Amortized sampling for fast and stable training of
energy-based models.
\newblock In \emph{Proceedings of the 9th International Conference on Learning
Representations (ICLR)}, Virtual Event, 2021.
\bibitem[Gulrajani et~al.(2017)Gulrajani, Ahmed, Arjovsky, Dumoulin, and
Courville]{gulrajani2017improved}
Ishaan Gulrajani, Faruk Ahmed, Mart{\'{\i}}n Arjovsky, Vincent Dumoulin, and
Aaron~C. Courville.
\newblock Improved training of wasserstein gans.
\newblock In \emph{Advances in Neural Information Processing Systems (NIPS)},
pp.\ 5767--5777, Long Beach, CA, 2017.
\bibitem[Gutmann \& Hyv{\"{a}}rinen(2010)Gutmann and
Hyv{\"{a}}rinen]{GutmannH10}
Michael Gutmann and Aapo Hyv{\"{a}}rinen.
\newblock Noise-contrastive estimation: {A} new estimation principle for
unnormalized statistical models.
\newblock In \emph{Proceedings of the Thirteenth International Conference on
Artificial Intelligence and Statistics (AISTATS)}, pp.\ 297--304, Chia
Laguna Resort, Sardinia, Italy, 2010.
\bibitem[Han et~al.(2017)Han, Lu, Zhu, and Wu]{HanLZW17}
Tian Han, Yang Lu, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Alternating back-propagation for generator network.
\newblock In \emph{Proceedings of the Thirty-First {AAAI} Conference on
Artificial Intelligence (AAAI)}, pp.\ 1976--1984, San Francisco, CA, 2017.
\bibitem[Han et~al.(2019)Han, Nijkamp, Fang, Hill, Zhu, and
Wu]{han2019divergence}
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song{-}Chun Zhu, and
Ying~Nian Wu.
\newblock Divergence triangle for joint training of generator model,
energy-based model, and inferential model.
\newblock In \emph{Proceedings of the {IEEE} Conference on Computer Vision and
Pattern Recognition (CVPR)}, pp.\ 8670--8679, Long Beach, CA, 2019.
\bibitem[Han et~al.(2020)Han, Nijkamp, Zhou, Pang, Zhu, and Wu]{han2020joint}
Tian Han, Erik Nijkamp, Linqi Zhou, Bo~Pang, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Joint training of variational auto-encoder and latent energy-based
model.
\newblock In \emph{Proceedings of the 2020 {IEEE} Conference on Computer Vision
and Pattern Recognition (CVPR)}, pp.\ 7975--7984, Seattle, WA, 2020.
\bibitem[Heusel et~al.(2017)Heusel, Ramsauer, Unterthiner, Nessler, and
Hochreiter]{heusel2017gans}
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
Hochreiter.
\newblock {GANs Trained by a Two Time-Scale Update Rule Converge to a Local
Nash Equilibrium}.
\newblock In \emph{Advances in Neural Information Processing Systems (NIPS)},
pp.\ 6626--6637, Long Beach, CA, 2017.
\bibitem[Hinton(2002)]{Hinton02}
Geoffrey~E. Hinton.
\newblock Training products of experts by minimizing contrastive divergence.
\newblock \emph{Neural Computation}, 14\penalty0 (8):\penalty0 1771--1800,
2002.
\bibitem[Hinton(2012)]{hinton2012practical}
Geoffrey~E Hinton.
\newblock A practical guide to training restricted boltzmann machines.
\newblock In \emph{Neural networks: Tricks of the trade}, pp.\ 599--619.
Springer, 2012.
\bibitem[Ho et~al.(2019)Ho, Chen, Srinivas, Duan, and Abbeel]{ho2019flow++}
Jonathan Ho, Xi~Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel.
\newblock Flow++: Improving flow-based generative models with variational
dequantization and architecture design.
\newblock In \emph{Proceedings of the 36th International Conference on Machine
Learning (ICML)}, pp.\ 2722--2730, Long Beach, CA, 2019.
\bibitem[Karras et~al.(2020)Karras, Aittala, Hellsten, Laine, Lehtinen, and
Aila]{karras2020training}
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and
Timo Aila.
\newblock Training generative adversarial networks with limited data.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, Virtual Event, 2020.
\bibitem[Kingma \& Ba(2015)Kingma and Ba]{kingma2014adam}
Diederik~P. Kingma and Jimmy Ba.
\newblock Adam: {A} method for stochastic optimization.
\newblock In \emph{Proceedings of the 3rd International Conference on Learning
Representations (ICLR)}, San Diego, CA, 2015.
\bibitem[Kingma \& Dhariwal(2018)Kingma and Dhariwal]{kingma2018glow}
Diederik~P. Kingma and Prafulla Dhariwal.
\newblock Glow: Generative flow with invertible 1x1 convolutions.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, pp.\ 10236--10245, Montr{\'{e}}al, Canada, 2018.
\bibitem[Kingma \& Welling(2014)Kingma and Welling]{kingma2013auto}
Diederik~P. Kingma and Max Welling.
\newblock Auto-encoding variational bayes.
\newblock In \emph{Proceedings of the 2nd International Conference on Learning
Representations (ICLR)}, Banff, Canada, 2014.
\bibitem[Kingma et~al.(2016)Kingma, Salimans, J{\'{o}}zefowicz, Chen,
Sutskever, and Welling]{KingmaSJCCSW16}
Diederik~P. Kingma, Tim Salimans, Rafal J{\'{o}}zefowicz, Xi~Chen, Ilya
Sutskever, and Max Welling.
\newblock Improving variational autoencoders with inverse autoregressive flow.
\newblock In \emph{Advances in Neural Information Processing Systems (NIPS)},
pp.\ 4736--4744, Barcelona, Spain, 2016.
\bibitem[Krizhevsky \& Hinton(2009)Krizhevsky and
Hinton]{krizhevsky2009learning}
Alex Krizhevsky and Geoffrey Hinton.
\newblock Learning multiple layers of features from tiny images.
\newblock 2009.
\bibitem[Kumar et~al.(2020)Kumar, Babaeizadeh, Erhan, Finn, Levine, Dinh, and
Kingma]{KumarBEFLDK20}
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine,
Laurent Dinh, and Durk Kingma.
\newblock Videoflow: {A} conditional flow-based model for stochastic video
generation.
\newblock In \emph{Proceeding of the 8th International Conference on Learning
Representations (ICLR)}, Addis Ababa, Ethiopia, 2020.
\bibitem[LeCun et~al.(2006)LeCun, Chopra, Hadsell, Ranzato, and
Huang]{lecun2006tutorial}
Yann LeCun, Sumit Chopra, Raia Hadsell, M~Ranzato, and F~Huang.
\newblock A tutorial on energy-based learning.
\newblock \emph{Predicting structured data}, 1\penalty0 (0), 2006.
\bibitem[Liu et~al.(2015)Liu, Luo, Wang, and Tang]{liu2015faceattributes}
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
\newblock Deep learning face attributes in the wild.
\newblock In \emph{Proceedings of the 2015 International Conference on Computer
Vision (ICCV)}, Santiago, Chile, 2015.
\bibitem[Miyato et~al.(2018)Miyato, Kataoka, Koyama, and
Yoshida]{miyato2018spectral}
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
\newblock Spectral normalization for generative adversarial networks.
\newblock In \emph{Proceeding of the 6th International Conference on Learning
Representations (ICLR)}, Vancouver, Canada, 2018.
\bibitem[Neal(2011)]{neal2011mcmc}
Radford~M Neal.
\newblock {MCMC} using hamiltonian dynamics.
\newblock \emph{Handbook of markov chain monte carlo}, 2\penalty0
(11):\penalty0 2, 2011.
\bibitem[Netzer et~al.(2011)Netzer, Wang, Coates, Bissacco, Wu, and
Ng]{netzer2011reading}
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo~Wu, and Andrew~Y
Ng.
\newblock Reading digits in natural images with unsupervised feature learning.
\newblock In \emph{NIPS Workshop on Deep Learning and Unsupervised Feature
Learning}, 2011.
\bibitem[Nijkamp et~al.(2019)Nijkamp, Hill, Zhu, and Wu]{nijkamp2019learning}
Erik Nijkamp, Mitch Hill, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Learning non-convergent non-persistent short-run {MCMC} toward
energy-based model.
\newblock In \emph{Advances in Neural Information Processing (NeurIPS)}, pp.\
5233--5243, Vancouver, Canada, 2019.
\bibitem[Nijkamp et~al.(2020{\natexlab{a}})Nijkamp, Gao, Sountsov, Vasudevan,
Pang, Zhu, and Wu]{nijkamp2020learning}
Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo~Pang,
Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Learning energy-based model with flow-based backbone by neural
transport {MCMC}.
\newblock \emph{arXiv preprint arXiv:2006.06897}, 2020{\natexlab{a}}.
\bibitem[Nijkamp et~al.(2020{\natexlab{b}})Nijkamp, Pang, Han, Zhou, Zhu, and
Wu]{NijkampP0ZZW20}
Erik Nijkamp, Bo~Pang, Tian Han, Linqi Zhou, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Learning multi-layer latent variable model via variational
optimization of short run {MCMC} for approximate inference.
\newblock In \emph{Proceedings of the 16th European Conference on Computer
Vision (ECCV, Part {VI})}, pp.\ 361--378, Glasgow, UK, 2020{\natexlab{b}}.
\bibitem[Ostrovski et~al.(2018)Ostrovski, Dabney, and
Munos]{ostrovski2018autoregressive}
Georg Ostrovski, Will Dabney, and R{\'{e}}mi Munos.
\newblock Autoregressive quantile networks for generative modeling.
\newblock In \emph{Proceedings of the 35th International Conference on Machine
Learning (ICML)}, pp.\ 3933--3942, Stockholmsm{\"{a}}ssan, Stockholm,
Sweden, 2018.
\bibitem[Pang et~al.(2020)Pang, Han, Nijkamp, Zhu, and Wu]{pang2020learning}
Bo~Pang, Tian Han, Erik Nijkamp, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Learning latent space energy-based prior model.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, Virtual Event, 2020.
\bibitem[Prenger et~al.(2019)Prenger, Valle, and Catanzaro]{PrengerVC19}
Ryan Prenger, Rafael Valle, and Bryan Catanzaro.
\newblock Waveglow: {A} flow-based generative network for speech synthesis.
\newblock In \emph{Proceeding of the International Conference on Acoustics,
Speech and Signal Processing (ICASSP)}, pp.\ 3617--3621, Brighton, United
Kingdom, 2019.
\bibitem[Radford et~al.(2016)Radford, Metz, and
Chintala]{radford2015unsupervised}
Alec Radford, Luke Metz, and Soumith Chintala.
\newblock Unsupervised representation learning with deep convolutional
generative adversarial networks.
\newblock In \emph{Proceedings of the 4th International Conference on Learning
Representations (ICLR)}, San Juan, Puerto Rico, 2016.
\bibitem[Rezende \& Mohamed(2015)Rezende and Mohamed]{RezendeM15}
Danilo~Jimenez Rezende and Shakir Mohamed.
\newblock Variational inference with normalizing flows.
\newblock In \emph{Proceedings of the 32nd International Conference on Machine
Learning (ICML)}, pp.\ 1530--1538, Lille, France, 2015.
\bibitem[Salimans \& Kingma(2016)Salimans and Kingma]{salimans2016weight}
Tim Salimans and Diederik~P. Kingma.
\newblock Weight normalization: {A} simple reparameterization to accelerate
training of deep neural networks.
\newblock In \emph{Advances in Neural Information Processing Systems 29
(NIPS)}, pp.\ 901, Barcelona, Spain, 2016.
\bibitem[Salimans et~al.(2017)Salimans, Karpathy, Chen, and
Kingma]{salimans2017pixelcnn++}
Tim Salimans, Andrej Karpathy, Xi~Chen, and Diederik~P. Kingma.
\newblock {PixelCNN++}: Improving the pixelcnn with discretized logistic
mixture likelihood and other modifications.
\newblock In \emph{Proceeding of the 5th International Conference on Learning
Representations (ICLR)}, Toulon, France, 2017.
\bibitem[Song \& Ermon(2019)Song and Ermon]{song2019generative}
Yang Song and Stefano Ermon.
\newblock Generative modeling by estimating gradients of the data distribution.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, pp.\ 11895--11907, Vancouver, Canada, 2019.
\bibitem[Song \& Ermon(2020)Song and Ermon]{song2020improved}
Yang Song and Stefano Ermon.
\newblock Improved techniques for training score-based generative models.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, Virtual Event, 2020.
\bibitem[Song et~al.(2021)Song, Sohl{-}Dickstein, Kingma, Kumar, Ermon, and
Poole]{YangSong21}
Yang Song, Jascha Sohl{-}Dickstein, Diederik~P. Kingma, Abhishek Kumar, Stefano
Ermon, and Ben Poole.
\newblock Score-based generative modeling through stochastic differential
equations.
\newblock In \emph{Proceeding of the 9th International Conference on Learning
Representations (ICLR)}, Virtual Event, 2021.
\bibitem[Vahdat \& Kautz(2020)Vahdat and Kautz]{NVAE}
Arash Vahdat and Jan Kautz.
\newblock {NVAE:} {A} deep hierarchical variational autoencoder.
\newblock In \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, Virtual Event, 2020.
\bibitem[Wu et~al.(2020)Wu, K{\"{o}}hler, and No{\'{e}}]{0035KN20}
Hao Wu, Jonas K{\"{o}}hler, and Frank No{\'{e}}.
\newblock Stochastic normalizing flows.
\newblock In \emph{Advances in Neural Information Processing (NeurIPS)},
Virtual Event, 2020.
\bibitem[Xiao et~al.(2021)Xiao, Kreis, Kautz, and Vahdat]{xiao2020vaebm}
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat.
\newblock {VAEBM:} {A} symbiosis between variational autoencoders and
energy-based models.
\newblock In \emph{Proceeding of the 9th International Conference on Learning
Representations (ICLR)}, Virtual Event, 2021.
\bibitem[Xie et~al.(2016)Xie, Lu, Zhu, and Wu]{XieLZW16}
Jianwen Xie, Yang Lu, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock A theory of generative convnet.
\newblock In \emph{Proceedings of the 33nd International Conference on Machine
Learning (ICML)}, pp.\ 2635--2644, New York City, NY, 2016.
\bibitem[Xie et~al.(2017)Xie, Zhu, and Wu]{XieZW17}
Jianwen Xie, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Synthesizing dynamic patterns by spatial-temporal generative convnet.
\newblock In \emph{Proceeding of the {IEEE} Conference on Computer Vision and
Pattern Recognition (CVPR)}, pp.\ 1061--1069, Honolulu, HI, 2017.
\bibitem[Xie et~al.(2018{\natexlab{a}})Xie, Lu, Gao, and Wu]{XieLGW18}
Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying~Nian Wu.
\newblock Cooperative learning of energy-based model and latent variable model
via {MCMC} teaching.
\newblock In \emph{Proceedings of the 32nd {AAAI} Conference on Artificial
Intelligence, (AAAI)}, pp.\ 4292--4301, New Orleans, Louisiana, USA,
2018{\natexlab{a}}.
\bibitem[Xie et~al.(2018{\natexlab{b}})Xie, Zheng, Gao, Wang, Zhu, and
Wu]{XieZGWZW18}
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song{-}Chun Zhu, and
Ying~Nian Wu.
\newblock Learning descriptor networks for 3d shape synthesis and analysis.
\newblock In \emph{Proceeding of the {IEEE} Conference on Computer Vision and
Pattern Recognition (CVPR)}, pp.\ 8629--8638, Salt Lake City, UT,
2018{\natexlab{b}}.
\bibitem[Xie et~al.(2020{\natexlab{a}})Xie, Lu, Gao, Zhu, and Wu]{XieLGZW20}
Jianwen Xie, Yang Lu, Ruiqi Gao, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Cooperative training of descriptor and generator networks.
\newblock \emph{{IEEE} Transactions on Pattern Analysis and Machine
Intelligence (TPAMI)}, 42\penalty0 (1):\penalty0 27--45, 2020{\natexlab{a}}.
\bibitem[Xie et~al.(2020{\natexlab{b}})Xie, Zheng, Gao, Wang, Zhu, and
Wu]{xie2020generative}
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and
Ying~Nian Wu.
\newblock Generative voxelnet: learning energy-based models for 3d shape
synthesis and analysis.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine Intelligence
(TPAMI)}, 2020{\natexlab{b}}.
\bibitem[Xie et~al.(2021{\natexlab{a}})Xie, Xu, Zheng, Zhu, and Wu]{XieXZZW21}
Jianwen Xie, Yifei Xu, Zilong Zheng, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Generative pointnet: Deep energy-based learning on unordered point
sets for 3d generation, reconstruction and classification.
\newblock In \emph{Proceeding of the {IEEE} Conference on Computer Vision and
Pattern Recognition (CVPR)}, pp.\ 14976--14985, Virtual Event,
2021{\natexlab{a}}.
\bibitem[Xie et~al.(2021{\natexlab{b}})Xie, Zheng, and Li]{XieZL21}
Jianwen Xie, Zilong Zheng, and Ping Li.
\newblock Learning energy-based model with variational auto-encoder as
amortized sampler.
\newblock In \emph{Proceeding of the Thirty-Fifth {AAAI} Conference on
Artificial Intelligence (AAAI)}, pp.\ 10441--10451, Virtual Event,
2021{\natexlab{b}}.
\bibitem[Xie et~al.(2021{\natexlab{c}})Xie, Zhu, and Wu]{XieZW21}
Jianwen Xie, Song{-}Chun Zhu, and Ying~Nian Wu.
\newblock Learning energy-based spatial-temporal generative convnets for
dynamic patterns.
\newblock \emph{{IEEE} Transactions on Pattern Analysis and Machine
Intelligence (TPAMI)}, 43\penalty0 (2):\penalty0 516--531,
2021{\natexlab{c}}.
\bibitem[Yang et~al.(2019)Yang, Huang, Hao, Liu, Belongie, and
Hariharan]{YangHH0BH19}
Guandao Yang, Xun Huang, Zekun Hao, Ming{-}Yu Liu, Serge~J. Belongie, and
Bharath Hariharan.
\newblock Pointflow: 3d point cloud generation with continuous normalizing
flows.
\newblock In \emph{Proceeding of the {IEEE} International Conference on
Computer Vision (ICCV)}, pp.\ 4540--4549, Seoul, Korea, 2019.
\bibitem[Younes(1999)]{younes1999convergence}
Laurent Younes.
\newblock On the convergence of markovian stochastic algorithms with rapidly
decreasing ergodicity rates.
\newblock \emph{Stochastics: An International Journal of Probability and
Stochastic Processes}, 65\penalty0 (3-4):\penalty0 177--228, 1999.
\bibitem[Zhao et~al.(2021)Zhao, Xie, and Li]{zhao2020learning}
Yang Zhao, Jianwen Xie, and Ping Li.
\newblock Learning energy-based generative models via coarse-to-fine expanding
and sampling.
\newblock In \emph{Proceeding of the 9th International Conference on Learning
Representations (ICLR)}, Virtual Event, 2021.
\bibitem[Zheng et~al.(2021)Zheng, Xie, and Li]{ZhengXL21}
Zilong Zheng, Jianwen Xie, and Ping Li.
\newblock Patchwise generative convnet: Training energy-based models from a
single natural image for internal learning.
\newblock In \emph{Proceedings of the {IEEE} Conference on Computer Vision and
Pattern Recognition (CVPR)}, pp.\ 2961--2970, virtual Event, 2021.
\bibitem[Zhu et~al.(1998)Zhu, Wu, and Mumford]{ZhuWM98}
Song-Chun Zhu, Ying~Nian Wu, and David Mumford.
\newblock Filters, random fields and maximum entropy {(FRAME):} towards a
unified theory for texture modeling.
\newblock \emph{International Journal of Computer Vision (IJCV)}, 27\penalty0
(2):\penalty0 107--126, 1998.
\end{thebibliography}