Scalable Generative Models for Graphs with Graph Attention Mechanism
Extracted source text · arXiv 1906.01861
\documentclass{article} % For LaTeX2e
\usepackage{iclr2020_conference_arxiv,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{svg}
\usepackage[export]{adjustbox}
\usepackage{lscape}
\graphicspath{{./figures/}{./figures/visualization/}}
% \graphicspath{}
% \iclrfinalcopy
\title{Scalable Generative Models for Graphs with Graph Attention Mechanism}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Wataru Kawai$^{1}$, Yusuke Mukuta$^{1, 2}$ \& Tatsuya Harada$^{1,2}$ \\
The University of Tokyo$^1$, RIKEN$^2$ \\
\texttt{\{w-kawai, mukuta, harada\}@mi.t.u-tokyo.ac.jp} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
%\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
Graphs are ubiquitous real-world data structures, and generative models that approximate distributions over graphs and derive new samples from them have significant importance.
Among the known challenges in graph generation tasks, \textit{scalability} handling of large graphs and datasets is one of the most important for practical applications.
Recently, an increasing number of graph generative models have been proposed and have demonstrated impressive results.
However, scalability is still an unresolved problem due to the complex generation process or difficulty in training parallelization.
In this paper, we first define scalability from three different perspectives: number of nodes, data, and node/edge labels.
Then, we propose GRAM, a generative model for graphs that is scalable in all three contexts, especially in training.
We aim to achieve scalability by employing a novel graph attention mechanism, formulating the likelihood of graphs in a simple and general manner.
Also, we apply two techniques to reduce computational complexity.
Furthermore, we construct a unified and non-domain-specific evaluation metric in node/edge-labeled graph generation tasks by combining a graph kernel and Maximum Mean Discrepancy.
Our experiments on synthetic and real-world graphs demonstrated the scalability of our models and their superior performance compared with baseline methods.
\end{abstract}
\section{Introduction}
Graphs are ubiquitous and fundamental data structures in the real world, appearing in various fields, such as social science, chemistry, and biology.
% For example, knowledge bases, molecular structures, and relations between objects in an image can be represented as graphs, which allows us to efficiently capture the essence of these data.
Moreover, in applications such as drug discovery and network simulation, graph generative models that can approximate distributions over graphs on a specific domain and derive new samples from them are very important.
% Recently, generation tasks of images, sentences, and sounds have been gaining great interest\cite{NIPS2014_5423, VanDenOord:2016:PRN:3045390.3045575, DBLP:journals/corr/KingmaW13, AAAI1714344, NIPS2014_5346, DBLP:journals/corr/abs-1802-04208}.
Compared with generation tasks for images or natural language, graph generation tasks are significantly more difficult;
this is due to the necessity of modeling complex local/global dependencies among nodes and edges as well as the intractable properties of graphs themselves, such as discreteness, variable number of nodes and edges, and uncertainty of node ordering.
Among the several challenges involved in graph generation tasks described above, \textit{scalability} is one of the most important for applications in a wide range of real-world domains.
In this paper, we define scalability from three perspectives: \textit{graph scalability}, \textit{data scalability}, and \textit{label scalability}.
Graph scalability denotes scalability to large graphs with many nodes within the limit of practical time/space complexity, especially in training.
Data scalability denotes scalability to large datasets containing many data.
Label scalability denotes scalability to graphs with many node/edge labels.
Besides these, it is possible to consider other viewpoints, such as edge number, graph diameter, and the total number of nodes in a dataset.
However, because these are closely related to the above three defined perspectives, we consider them as a mutually exclusive and collectively exhaustive division.
In recent years, an increasing number of graph generative models based on machine learning have been proposed.
They have demonstrated great performances in several tasks, such as link prediction, molecular property optimization, and network structure optimization \citep{2018arXiv180303324L, DBLP:conf/icml/YouYRHL18, 2018arXiv180310459G, NIPS2018_8007, NIPS2018_7877, NIPS2018_8005, Simonovsky2018GraphVAETG, 2017arXiv171108267W, Li2018}.
However, to the best of our knowledge, no proposed model is scalable in all three contexts.
For example, DeepGMG \citep{2018arXiv180303324L} can generate only small graphs, and GraphRNN \citep{DBLP:conf/icml/YouYRHL18} cannot consider node/edge labels.
Besides, both models have weak compatibility with parallel training, which is key to efficient training on a large dataset.
In this work, we propose the \textit{Graph Generative Model with Graph Attention Mechanism} (GRAM) for generating graphs that is scalable in all three contexts, especially during training.
Given a set of graphs, our model approximates their distribution in an unsupervised manner.
To achieve graph scalability, we employ an autoregressive sequential generation process that is flexible to variable nodes and edges, and
formulate the likelihood of graphs in a simple manner to simplify the generation process.
Besides, we apply two techniques to reduce computational cost: breadth-first search (BFS) and zeroing attention weights in edge estimation.
Regarding data scalability, we apply a novel graph attention mechanism that simply extends the attention mechanism used in natural language processing \citep{NIPS2017_7181} to graphs.
% More precisely, by considering the relative position between two nodes in multi-head attention, it avoids the problem that we can't define absolute coordinate on graphs, which is one of the significant differences from sentences.
This graph attention mechanism does not include sequentially dependent hidden states as the recurrent neural network (RNN) does, which improves the parallelizability of training significantly.
Compared with other graph attention mechanisms \citep{velickovic2018graph, NIPS2018_8131, 2019arXiv190201020I},
ours is architecturally simple, computationally lightweight, and general, i.e., its applicability is not limited to generation tasks.
Finally, for label scalability, we formulate the likelihood of graphs assuming multiple node/edge labels.
Also, we use graph convolution/attention layers, whose numbers of parameters do not depend directly on the number of labels.
Moreover, we introduce a unified non-domain-specific evaluation metric for generation tasks of node/edge-labeled graphs.
Because such a method does not currently exist, prior works relied on domain-specific metrics or visual inspection, which made unified and objective evaluations difficult.
Thus, we construct a unified evaluation metric that dispenses with domain-specific knowledge and considers not only the topology but also node/edge labels by combining a graph kernel and Maximum Mean Discrepancy (MMD) \citep{Gretton:2012:KTT:2188385.2188410}.
Although a similar statistical test based on MMD via a graph kernel was used for schema matching of protein graphs in \citet{10.1093/bioinformatics/btl242}, to the best of our knowledge, this work is the first to use as an evaluation metric for graph generation tasks.
Our experiments on synthetic and real-world graphs demonstrated that our models can scale up to handle large graphs and datasets that previous methods had difficulty with; our models demonstrated superior results to those of baseline methods.
To summarize, the contributions of this work are as follows:
\begin{itemize}
\item We propose GRAM, a graph generative model that is scalable, especially in training.
\item We propose a novel graph attention mechanism that is general and architecturally simple as a key portion of the generative model.
\item We define scalability in graph generation from three perspectives: number of nodes, data, and labels.
\item We construct a unified non-domain-specific evaluation metric for node/edge-labeled graph generation tasks by combining a graph kernel and MMD.
\end{itemize}
\section{Related Work}
Although there are several traditional graph generative models \citep{erdos1959random, RevModPhys.74.47, Leskovec:2010:KGA:1756006.1756039, ROBINS2007173, NIPS2008_3578}, we focus here on recent machine learning-based models, which have outperformed traditional models in various tasks.
In terms of their generation process, existing graph generative models can be classified into at least two types: tensor generation models and sequential generation models.
Tensor generation models \citep{Simonovsky2018GraphVAETG, 2018arXiv180511973D, 2018arXiv180310459G} generate a graph by outputting tensors that correspond to the graph.
These models are architecturally simple and easy to optimize for small graphs.
However, they face difficulty in generating large graphs owing to the non-unique correspondence between a graph and tensors, limitations in the pre-defined maximum number of nodes, and the increasing number of parameters depending on the maximum graph size.
In contrast, sequential generation models, such as DeepGMG \citep{2018arXiv180303324L}, GraphRNN\citep{DBLP:conf/icml/YouYRHL18}, generate graphs by adding nodes and edges one-by-one, which alleviates the above problems and generates larger graphs.
To achieve graph scalability, we employ the latter.
However, to generate a graph with $n$ nodes and $m$ edges, DeepGMG requires at least $\mathcal{O}(m n^2)$ operations because of its complex generation process.
GraphRNN reduces this time complexity to $\mathcal{O}(n M)$ with a dataset-specific value $M$ by utilizing the BFS.
However, it cannot handle node/edge labels, mainly because it omits the feature extraction process, and relies mainly on the information in the hidden states of the RNN.
% and $M $ can be large value for large graphs with $100\sim 200$ nodes because this strategy is based on the search algorithm, not on the properties of graphs.
Moreover, these models include sequentially dependent hidden states, which make training parallelization difficult.
In contrast, our model employs a graph attention mechanism without sequentially dependent hidden states.
This improves the parallelizability of training significantly.
Also, by applying two approximation methods, we reduce the complexity to almost a linear order of $n$ while conducting a rich feature extraction.
A comparison of our models’ and the baselines’ complexities is given in Table \ref{tab:complexity-comparison} in Section \ref{sec:A-complexity-analysis}.
As another division, there are unsupervised learning approaches and reinforcement learning approaches.
Given samples of graphs, unsupervised learning models \citep{2018arXiv180303324L, DBLP:conf/icml/YouYRHL18, Simonovsky2018GraphVAETG, 2018arXiv180310459G} approximate the distribution of them in an unsupervised manner.
Reinforcement learning models \citep{NIPS2018_8007, NIPS2018_7877, 2018arXiv180511973D, NIPS2018_8005, Li2018} learn to generate graphs to maximize a given return, such as QED \citep{QED}.
Although reinforcement learning approaches have demonstrated promising results on several tasks, such as molecular generation and network architecture optimization, we employ an unsupervised approach;
this is because it is considered to be more advantageous when new samples that share certain properties with training samples are required or when the reward functions cannot be designed (e.g., a family of pharmaceutical molecules against a certain disease is known, but the mechanism by which they work is not entirely known).
\section{Proposed Method}
An outline of this section is as follows:
In Section \ref{sec:notations}, we first describe notations and define the considered graph generation task.
In Section \ref{sec:likelihood-formulation}, we formulate the likelihood of graphs.
In Section \ref{sec:graph-attention}, we present our proposed graph attention mechanism, which is the key portion of our model and is used in feature extraction and edge estimation.
In Section \ref{sec:gram}, we present our scalable graph generative model, GRAM.
We describe two techniques to reduce the computational cost of edge estimation: BFS in Section \ref{sec:bfs} and zeroing attention weights in Section \ref{sec:attention_zeroing}.
In Section \ref{sec:training}, we describe the training and propose an evaluation metric.
\subsection{Notations and Problem Definition}
\label{sec:notations}
In this work, we define a graph $G=(V, E)$ as a data structure that consists of a node set $V=\{v_1, ..., v_n\}$ and an edge set $E=\{e_{i, j}|i, j \in \{1, ..., n\}\}$.
For simplicity, we focus on undirected graphs.
We assume nodes and edges are associated with node/edge labels and denote the number of them as $a$ and $b$, respectively.
For graph representation, we employ a tensor representation.
Specifically, given node ordering $\pi$, we represent a pair of a graph and node ordering $(G, \pi)$ as a pair of tensors $(X^\pi, A^\pi)$.
Note that we assume the existence of node ordering in graphs, and $\pi$ is a permutation function over $\{1, ..., n\}$.
$X^\pi\in \{0, 1\}^{n \times a}$, whose $i$-th row is a one-hot vector corresponding to the label of $v_{\pi(i)}$, stores information about nodes.
$A^\pi\in \{0, 1\}^{n \times n \times b}$, whose $(i, j)$-th element is a one-hot vector corresponding to the label of $e_{\pi(i), \pi(j)}$, stores information about edges.
If no edge exists between $v_{\pi(i)}$ and $v_{\pi(j)}$, we replace it with a zero vector.
An example is illustrated in Figure \ref{fig:tensor-representation} in Section \ref{sec:A-tensor-representation}.
Note that $(G, \pi)$ and $(X^\pi, A^\pi)$ are in unique correspondence.
Finally, for simplicity, we do not consider self-looping or multiple edges.
% and we assume that independent elements of $A^\pi$ are limited to its upper triangular elements $A^\pi_{i, j}~(1\le i \le n, i < j \le n)$.
A possible extension to self-looping edges would involve adding a step to estimate them.
For multiple or directed edges, we can prepare additional labels that correspond to them.
In the remainder of this paper, for clear notation, we assume $\pi (i) = i$ and omit the superscript $\pi$.
The target graph generation task is as follows: given samples from a distribution of graphs $p(G)$, approximate the distribution ($\hat{p} \approx p$) such that we can derive new samples from it ($G \sim \hat{p}(G)$).
\subsection{Likelihood Formulation of Graphs}
\label{sec:likelihood-formulation}
To approximate distributions over graphs in an autoregressive manner, we formulate the likelihood of a pair of a graph and node ordering $(G, \pi)$ and decompose it into a product of conditional probabilities.
Because $(G, \pi)$ uniquely defines $(X, A)$ and vice versa, we have $p(G, \pi)=p(X, A)$,
which is decomposed into a product of conditional probabilities as
\begin{align}
p(X, A)
% = p(X^\pi_{1}, A^\pi_{1, 1}) \prod _{s=2}^{n} {p(X^\pi_{s}, A^\pi_{s, \cdot}|X^\pi_{<s}, A^\pi_{<s, <s})} \nonumber \\
% &= p(X^\pi_{1}, A^\pi_{1, 1}) \prod _{s=2}^{n} {p(X^\pi_{s}|X^\pi_{<s}, A^\pi_{<s, <s}) p(A^\pi_{s, \cdot}|X^\pi_{s}, X^\pi_{<s}, A^\pi_{<s, <s}) } \nonumber \\
= p(X_{1}, A_{1, 1}) \textstyle \prod _{s=2}^{n} {p(X_{s}|X_{<s}, A_{<s, <s}) \textstyle \prod _{t=1}^{s-1} p(A_{t, s}|A_{<t, s}, X_{s}, X_{<s}, A_{<s, <s}) } \label{eq:ll_seq}
\end{align}
where $A_{<s, <s} \in \mathbb{R}^{s-1\times s-1\times b}$ represents a partial tensor $A_{i, j}~(1 \le i, j < s)$, and other notations follow this.
For simplicity, we omit the slice notation for the last dimension of each tensor.
In addition, $(X_{<s}, A_{<s, <s})$ uniquely defines a subgraph of $G$, which we denote as $G_{<s}$.
With this formulation, the likelihood of a graph $G$ is defined as marginal, $p(G) = \sum _\pi p(G, \pi) = \sum _\pi p(X, A)$.
Also, we represent $(X_{<s}, A_{<s, <s})$ as $G_{<s}$ in the following equations.
During training, given samples from $p(G)$, our model approximates the joint probability $p(X, A)$.
More precisely, it approximates $p(X, A)$ by approximating the conditional probabilities $p(X_{s}|G_{<s})$ and $p(A_{t, s}|A_{<t, s}, X_{s}, G_{<s})$.
As for $p(X_{1}, A_{1, 1})$, we use a sampling distribution from the training data.
Although the choice of $\pi$ provides room for discussion, we adopt the BFS used in \citet{DBLP:conf/icml/YouYRHL18}.
On inference, we sequentially sample $X_s$ and $A_{t, s} (s=2, ..., n, t=1, ..., s-1)$ from the approximated distribution,
and we get $(X, A)$ when the EOS is output.
This can be viewed as sampling from the approximated distribution $(G, \pi) \sim \hat{p}(G, \pi)$.
In particular, by focusing only on $G$, we can view it as sampling from the marginal $G \sim \hat{p}(G)$.
\subsection{Graph Attention Mechanism}
\label{sec:graph-attention}
The aim of employing a graph attention mechanism is to efficiently take in the information of distant nodes by attention.
Our graph attention mechanism extends the attention mechanism used in natural language processing \citep{NIPS2017_7181} to graphs.
However, one of the significant differences between graphs and sentences is that we cannot define absolute coordinates in a graph,
which makes it difficult to embed positional information in nodes as in \citet{NIPS2017_7181}.
To alleviate this problem, we focus on multi-head attention as in \citet{NIPS2017_7181}, and we use relative positional information between nodes:
we introduce bias terms in projection to subspaces, which are functions of the shortest path length between two nodes.
The operation of graph attention is illustrated in Figures \ref{fig:graph-attention-self-attention} and \ref{fig:graph-attention-source-target-attention} in Section \ref{sec:A-graph-attention-illustration}.
Following \citet{NIPS2017_7181}, we denote matrices into which $n$ query, key, and value vectors are stacked
as $Q = (\bm{q}_{1}, ..., \bm{q}_{n})^T (\in \mathbb{R}^{n \times d_Q}), K = (\bm{k}_{1}, ..., \bm{k}_{n})^T (\in \mathbb{R}^{n \times d_K})$, and $V = (\bm{v}_{1}, ..., \bm{v}_{n})^T (\in \mathbb{R}^{n \times d_{V}})$, respectively.
Note that we use $V$ to represent the value matrix, not node set.
Also, we assume that the $i$-th row of each matrix corresponds to the feature vector of node $v_i$.
Note that the operation of graph attention is permutation invariant.
Denoting the dimensions of the output vector and the subspace as $d_O$ and $d_S$, respectively, the graph attention mechanism is defined as
\begin{align}
\mathrm{GMultiHead}(Q, K, V) &= \mathrm{Concat}(\mathrm{head}_1, ..., \mathrm{head}_H) W^O &&(\in \mathbb{R}^{n \times d_{O}}) & \label{eq:gatt1}\\
\text{where } \mathrm{head}_h &= \mathrm{GAttention}(Q, K, V) &&(\in \mathbb{R}^{n \times d_S}) & ( h=1,...,H)\nonumber
\end{align}
where $H$ represents the number of projections to $d_S$-dimensional subspaces.
The operation of $\mathrm{GAttention}(\cdot, \cdot, \cdot)$ is defined as
\begin{align}
\mathrm{GAttention}(Q, K, V) &= (\bm{o}_{1}, ..., \bm{o}_{n})^T &&(\in \mathbb{R}^{n \times d_S}) & \\
\text{where } \bm{o}_i &= \textstyle \sum _{j=1}^n {(\bm{c}_i)}_j (W^V \bm{v}_j + \bm{b}^V(v_i, v_j)) && (\in \mathbb{R}^{d_S}) & (i=1,...,n) \nonumber
\end{align}
Each attention weight vector $\bm{c}_i$ is calculated as
\begin{align}
\bm{c}_i &= \mathrm{softmax}(\bm{s}_i) && (\in \mathbb{R}^{n}) & (i = 1, ..., n) \\
\text{where }({\bm{s}_i)}_j &= {d_K}^{-\frac{1}{2}} (W^Q \bm{q}_i + \bm{b}^Q(v_i, v_j))^T (W^K \bm{k}_j + \bm{b}^K(v_i, v_j)) && (\in \mathbb{R}) & (j=1,...,n) \nonumber
\end{align}
The parameters to be learned are four weight matrices -- $W^Q (\in \mathbb{R}^{d_S \times d_Q}), W^K (\in \mathbb{R}^{d_S \times d_K}), W^V (\in \mathbb{R}^{d_S \times d_V}), W^O (\in \mathbb{R}^{H d_S \times d_O})$ -- and three bias terms -- $\bm{b}^Q, \bm{b}^K, \bm{b}^V (\in \mathbb{R}^{d_S})$.
We used different parameters for each head.
To consider the geometric relation between two nodes, we used functions of the shortest path length between $v_i$ and $v_j$
as $\bm{b}^Q(v_i, v_j), \bm{b}^K(v_i, v_j)$, and $\bm{b}^V(v_i, v_j)$.
Furthermore, because path length is discrete, we used different weight parameters for each path length.
Examples of other possible approaches include using network flow or functional approximation.
Specifically, setting $\bm{b}^Q = \bm{b}^K = \bm{b}^V = \bm{0}$ yields the original multi-head attention.
As in \citet{NIPS2017_7181}, we added a two-layer feedforward neural network (FNN), which is applied after the above operations.
We denote these operations, including the FNN, as the graph attention mechanism in the following sections.
\subsection{GRAM: Scalable Generative Models for Graphs}
\label{sec:gram}
GRAM approximates a distribution over graphs in an autoregressive manner.
It utilizes the graph attention mechanism to improve the parallelizability of training.
The $s$-th step of the generation process is illustrated in Figure \ref{fig:pipeline}; a larger and more detailed version is given in Figure \ref{fig:pipeline-large} in Section \ref{sec:A-pipeline-large}.
A detailed analysis of computational complexity is included in Section \ref{sec:A-complexity-analysis}.
\subsubsection{Model Overview}
In the following sections, we focus on the $s$-th step, i.e., the generation of $v_s$ and $\{e_{1, s}, ..., e_{s-1, s}\}$.
The architecture of GRAM consists of three networks: feature extractor, node estimator, and edge estimator.
The feature extractor calculates node feature vectors and a graph feature vector by summing them.
The node estimator predicts the label of the new node $v_s$.
The edge estimator predicts the labels of new edges $\{e_{1, s},...,e_{s-1, s}\}$ between previously generated nodes $\{v_1,...,v_{s-1}\}$ and the new node $v_s$.
In other words, the node estimator approximates $p(X_{s}|G_{<s})$, and the edge estimator approximates $p(A_{t, s}|A_{<t, s}, X_{s}, G_{<s})$ in Equation \ref{eq:ll_seq}.
\begin{figure}
\begin{minipage}{0.72\hsize}
\begin{flushleft}
\includegraphics[width=0.96\linewidth]{pipeline_small-crop.pdf}
\vspace{-10pt}
\caption{$s$-th step of the generation process.}
\label{fig:pipeline}
\end{flushleft}
\end{minipage}
\begin{minipage}{0.27\hsize}
\vspace{7pt}
\begin{flushright}
\includegraphics[width=0.96\linewidth]{feature_extraction_block-crop.pdf}
\caption{$l$-th feature extraction block.}
\label{fig:fe_block}
\end{flushright}
\end{minipage}
\vspace{-12pt}
\end{figure}
%
\subsubsection{Feature Extractor}
Given a pair $(X_{<s}, A_{<s, <s})$, the feature extractor calculates $s-1$ node feature vectors $H^v = (\bm{h}^v_1, ..., \bm{h}^v_{s-1})^T$ and a graph feature vector $\bm{h}^G$ of the corresponding subgraph $G_{<s}$.
Here, $\bm{h}^v_i$ denotes the feature vector of node $v_i$, and we use $\bm{h}^{v (l)}_i$ to represent the output vector of the $l$-th block.
This consists of $L$ feature extraction blocks and a graph pooling layer.
In this work, we used $L=3$.
A feature extraction block is composed of a graph convolution layer and a graph attention layer stacked in parallel, as illustrated in Figure \ref{fig:fe_block}.
We aim to extract local information by graph convolution layers and global information by graph attention layers.
Although there are various types of graph convolutions, we employed the one used in \citet{johnson2018image}.
\footnote{Because \citet{johnson2018image} assume directed graphs as input, we ignore $g_o$ in Figure 3 of that paper.}
Roughly speaking, this convolutes the features of neighboring nodes and edges into each node and edge in a graph.
A graph attention layer operates self-attention, where query/key/value vectors are all node feature vectors.
To reduce computational cost, we restricted the range of attention to path length $r$.
Also, to exploit low-level features, we stacked degree and clustering coefficients, which are non-domain-specific statistics in graphs, into each input node vector.
Calculation of these graph statistics and BFS is required only once before training and is relatively fast compared with training time.
The graph pooling layer computes a graph feature vector $\bm{h}^G$
by summing all node feature vectors in the subgraph.
To improve its expressive power in aggregation, we used a gating network as in \citet{2018arXiv180303324L}.
Specifically, the operation in the graph pooling layer is defined as
\begin{align}
\bm{h}^G = \textstyle \sum _{i = 1}^{s-1} \sigma (g_{pool}(\bm{h}^v_i)) \bm{h}^v_i \label{eq:graph_pooling}
\end{align}
where $g_{pool}$ is a two-layer FNN, and $\sigma$ is a sigmoid function.
\subsubsection{Node Estimator}
The node estimator predicts the label of new node $v_s$.
More specifically, it predicts $X_s$ from $\bm{h}^G$ as
\begin{align}
\hspace{80pt} & X_s = \mathrm{softmax}(g_{NE}(\bm{h}^G)) && (\in \mathbb{R}^{a+1})
\end{align}
where $g_{NE}$ is a three-layer FNN, and the output dimension, including the EOS, is $a+1$.
We terminate the generation process when the EOS is output.
\subsubsection{Edge Estimator}
\label{sec:ee}
The edge estimator predicts the labels of the new edges $\{e_{1, s}, ..., e_{s-1, s}\}$ between previously generated nodes $\{v_1, ..., v_{s-1}\}$ and the new node $v_s$.
More precisely, it predicts $A_{t, s} ~ (t=1, ..., s-1)$ from $\bm{h}^v_t, \bm{h}^G, \bm{h}^v_s$ and previously predicted edge labels in this step as
\begin{align}
\hspace{80pt} A_{t, s} = \mathrm{softmax}(g_{EE}(\bm{h}^v_t, \bm{h}^G, \bm{h}^v_s, \bm{h}^e_{<t})) && (\in \mathbb{R}^{b+1}) && (t = 1, ..., s-1)
\end{align}
where $\bm{h}^v_s$ is the embedded vector of the predicted label of $v_s$, i.e., $X_s$.
The vector $\bm{h}^e_{<t}$ is calculated by a source-target attention in which the query vector is $\mathrm{Concat}(\bm{h}^v_t, \bm{h}^v_s)$ and the key/value vectors are $\{\mathrm{Concat}(\bm{h}^v_\tau, \bm{h}^v_s, \bm{h}^e_{\tau, s})|\tau=1, ..., t-1\}$, where $\bm{h}^e_{\tau, s}$ is an embedded vector of the predicted label of $e_{\tau, s}$, i.e., $A_{\tau, s}$.
Thereby, we aim to express the dependency on $A_{<t, s}$ in Equation \ref{eq:ll_seq}.
The operation is illustrated in Figure \ref{fig:graph-attention-source-target-attention} in Section \ref{sec:A-graph-attention-illustration}.
We use a three-layer FNN as $g_{EE}$, where the output dimension is $b+1$, including the ``no edge'' label.
When the ``no edge'' label is output, we do not add an edge and we set $A_{t, s} = \bm{0}$.
\subsection{Reducing Computational Complexity in Edge Estimation}
\label{sec:complexity-reduction}
To generate a graph with $n$ nodes, the edge estimation process requires $\mathcal{O}(n)$ operations for one edge, resulting in $\mathcal{O}(n^2)$ operations for one step and $\mathcal{O}(n^3)$ operations in total.
This is a significant obstacle to achieving graph scalability.
Here, we present two methods to reduce complexity in edge estimation: BFS and zeroing attention weights;
these can be combined independently.
\subsubsection{Complexity Reduction via Breadth Frist Search}
\label{sec:bfs}
As in \citet{DBLP:conf/icml/YouYRHL18}, we utilize the BFS node ordering, but our implementation gives a stricter upper bound.
More precisely, when estimating new edge $e_{t, s} ~ (t=1, ..., s-1)$, nodes that can be connected to the new node $v_s$ are restricted to ``frontier nodes'' $V_{f} = \{v_\tau | \min (\{i | v_i \text{ is connected to } v_{s-1} \}) \le \tau \le s-1 \}$ under $\pi = \mathrm{BFS}$ .
With this property, we need only consider $V_{f}$ instead of the whole $\{v_1, ..., v_{s-1}\}$ when estimating $e_{t, s}$.
This reduces the complexity of edge estimation to $\mathcal{O}(n\beta^2)$, where $\beta = |V_f|$.
A proof and example are included in Section \ref{sec:A-bfs}.
Finally, we denote this variant as GRAM-B.
\subsubsection{Complexity Reduction via Attention Weights Zeroing}
\label{sec:attention_zeroing}
As another method, we use an approximation inspired by empirical observation.
Specifically, our inspection of each attention weight during estimation of new edge $e_{t, s} ~ (t=1, ..., s-1)$ revealed that the nodes predicted to have no connection to $v_s$ (i.e., $V_{\text{noedge}} = \{v_\tau|A_{\tau, s} = \bm{0}, \tau=1, ..., t-1\}$) have near-zero weights,
while the nodes predicted to have connection (i.e., $V_{\text{haveedge}} = \{v_\tau|A_{\tau, s} \neq \bm{0}, \tau=1, ..., t-1\}$) have larger weights.
This suggests that among $\{v_1, ..., v_{t-1}\}$, $V_{\text{haveedge}}$ are important to predict the new edge $e_{t, s}$, while $V_{\text{noedge}}$ are not.
Hence, we deterministically set the weights of the latter to zero, which means we do not consider them in graph attention.
With this approximation,
the complexity is reduced to $\mathcal{O}(n^2 \alpha)$, where $\alpha = |V_{\text{haveedge}}|$.
In addition, because $\alpha \le \mathrm{deg}(v_s)$ holds, we can assume that $\alpha \ll n$ when $\mathrm{deg}(v_s) \ll n$, which is often the case in many real-world graphs.
A more detailed discussion and an example are included in Section \ref{sec:A-attention-zeroing}.
Finally, we denote this variant as GRAM-A and the combination of the two as GRAM-AB, which has complexity $\mathcal{O}(n \alpha \beta )$.
\subsection{Training and Evaluation}
\label{sec:training}
The loss function is the negative logarithm of Equation \ref{eq:ll_seq}, and the training is done in the usual autoregressive manner.
In particular, for one graph, forward/backward propagation of $n$ node estimations and at most $n (n-1) / 2$ edge estimations can be processed parallelly (i.e., in $\mathcal{O}(1)$ sequential operations) by using ground-truth labels and proper masking.
To evaluate the quality of generation, evaluation metrics based on MMD with graph statistics were used in \citet{DBLP:conf/icml/YouYRHL18}.
However, these metrics cannot consider node/edge labels, and it is difficult to determine which one is important or not.
Therefore, we construct a unified evaluation metric \textit{GK-MMD} that can consider not only topology but also node/edge labels by combining a graph kernel and MMD.
MMD is a test statistic to determine whether two sets of samples from the distribution $p$ and $q$ are derived from the same distribution (i.e., whether $p = q$).
When its function class $\mathcal{F}$ is a unit ball in a reproducing kernel Hilbert space (RKHS) $\mathcal{H}$,
we can derive the squared MMD as
\begin{align}
\mathrm{MMD}^2[\mathcal{F}, p, q] = \mathbb{E}_{x, x' \sim p}[k(x, x')] - 2 \mathbb{E}_{x\sim p, y \sim q}[k(x, y)] + \mathbb{E}_{y, y' \sim q}[k(y, y')] \label{eq:mmd_kernel}
\end{align}
where $k(\cdot , \cdot)$ is the associated kernel function \citep{Gretton:2012:KTT:2188385.2188410}.
Graph kernels are kernel functions over graphs, and we used the neighborhood subgraph pairwise distance kernel (NSPDK) \citep{CostaFabrizio2010Fnsp},
which measures the similarity of two graphs by matching pairs of subgraphs with different radii and distances.
Because NSPDK is a positive-definite kernel \citep{CostaFabrizio2010Fnsp}, it follows that it defines a unique RKHS $\mathcal{H}$ \citep{10.2307/1990404}.
This allows us to calculate the squared MMD using Equation \ref{eq:mmd_kernel}.
Because NSPDK considers low/high-level topological features and node/edge labels, GK-MMD can be used as a unified evaluation metric for node/edge-labeled graph generation tasks.
\section{Experiments}
We performed experiments on four types of synthetic graphs and three types of real-world graphs with various topologies, graph sizes, dataset sizes, and numbers of node/edge labels.
We evaluated the quality of generation and the actual computational cost required.
As synthetic graphs, we used four types of random graphs: grid, lobster, community \citep{fortunato2010community}, and Barab\'asi-Albert (B-A) \citep{barabasi1999emergence}.
We generated 700 graphs with numbers of nodes in the range $50 \le |V| \le 100$. We split them into training/test/validation data at 500:100:100.
To make the problem more realistic, we deterministically labeled nodes and edges according to their properties, such as degree or distance from backbone.
Detailed configurations, including model-specific parameters and node/edge labeling, are described in Section \ref{sec:A-experiment}.
We used three types of real-world graphs: molecular, ego, and protein.
For the molecular graphs, we used 250k samples from the ZINC database \citep{doi:10.1021/acs.jcim.5b00559} provided by \citet{2017arXiv170301925K}.
The ego graphs are ego networks from the Citeseer networks \citep{Sen_Namata_Bilgic_Getoor_Galligher_Eliassi-Rad_2008}.
The protein graphs represent protein structures with amino acids as nodes \citep{DOBSON2003771}.
For the ego and protein datasets, we used data provided by \citet{DBLP:conf/icml/YouYRHL18}.
We compared our models with DeepGMG \citep{2018arXiv180303324L}, GraphRNN, and GraphRNN-S \citep{DBLP:conf/icml/YouYRHL18} as recent deep learning-based baselines.
We modified the GraphRNN and GraphRNN-S models so that they can output multiple node/edge labels.
Because we target the unsupervised learning graph generation task and most of the datasets we used are node/edge-labeled, we did not conduct a comparison with reinforcement learning-based methods or traditional models, which are incapable of handling node/edge labels, except for GraphRNN and its variant.
To evaluate the quality of generation, we calculated squared GK-MMD score between generated graphs and test graphs.
We also used the squared MMD score of the three types of graph statistics used in \citet{DBLP:conf/icml/YouYRHL18}: distributions of degree, clustering coefficient, and orbit count; however, these metrics do not consider node/edge labels.
A lower MMD score indicates a better quality of approximation in terms of the given statistic.
For the molecular graphs, we generated 10k samples and calculated the ratios of generated graphs that are valid as molecules (valid [\%]),
unique samples (unique [\%]),
and novel samples, i.e., valid, unique, and unseen in the training data (novel [\%]).
We also reported training time [hour].
The results are listed in Table \ref{tab:synthetic-graphs}.
We did not train DeepGMG on the ego, protein and molecular datasets due to its impractical training time (estimated to take over 90 hours).
Instead, for the molecular dataset, we used generated graphs provided by the author for evaluation.
Although DeepGMG achieved a high valid/unique score and low GK-MMD score in the molecular dataset, it failed for larger graphs.
This is likely because its feature extraction method is based only on graph convolution, which gives only local information, so it works well only on small graphs, such as those of molecules.
Moreover, its training is slow due to its sequentially dependent hidden states.
GraphRNN and its variant achieved low topological MMD scores for simple topology graphs like grid.
However, their performances were poor on other graphs with more complex topologies or rules.
% This is likely because they do not calculate node/edge features, and their expressive power are limited.
This is likely because they do not calculate node/edge features, which limited their expressive power.
In other words, the feature extraction process is inevitable for the generation of complex graphs.
In contrast, our models achieved overall lower MMD scores on most of the datasets, which means that the distance between the approximated distribution and the test set is relatively small in terms of topological and node/edge label features.
Also, from the results on the molecular dataset, we can see that the generation is diverse while capturing the strict valence rule via unsupervised learning.
This is likely due to the bias term of graph attention in edge estimation: it enables models to control the valence and type of rings.
In terms of training time, there is a significant acceleration compared with the baselines.
This is mainly owing to the graph attention mechanism, which reduces the number of sequential operations during training to almost constant and enables efficient parallelized training.
Besides, we can observe the impact of the two computational reduction techniques on the actual computational time.
They were also more efficient in terms of memory usage, requiring less than half that of the original model.
Surprisingly, although there is generally some trade-off between training speed and performance, GRAM-B performed better than the original model on the grid and ego datasets, and GRAM-A performed better on the molecular dataset.
This is likely because each approximated model has its own strong topology for which both performance is boosted and computational cost is reduced.
On average, the proposed models performed better on large graphs.
More detailed discussions are included in Sections \ref{sec:A-bfs} and \ref{sec:A-attention-zeroing}.
From the above results, we can see the scalability of our models for large graphs and datasets and multiple node/edge labels, which the baseline methods have difficulty in handling.
Note that although our approximated models have some trade-off between performance and training speed, they still achieved superior or competitive results over the baselines in most cases.
Also, we found an interesting correlation between GK-MMD score and topological MMD scores or visual similarity, which supports the validity of this metric.
The detailed setting of the experiments, additional results, visualization, and an ablation study are included in Section \ref{sec:A-experiment}.
% \setlength{\tabcolsep}{3.3pt}
\begin{table}[t]
\setlength{\tabcolsep}{2.3pt}
\caption{Results on synthetic graphs (top) and real-world graphs (bottom). The dataset size $N$, maximum graph size $|V|_{\max}$, and the number of node/edge labels $a, b$ are shown as $(N, |V|_{\max}, a, b)$.}
\vspace{7pt}
\label{tab:synthetic-graphs}
\centering
% \scriptsize
% \footnotesize
\fontsize{8.4pt}{8.4pt}\selectfont
\begin{tabular}{lcccccccccccccccccccc}
\toprule
& \multicolumn{5}{c}{Grid (500, 100, 3, 2)} & \multicolumn{5}{c}{Community (500, 100, 4, 2)} & \multicolumn{5}{c}{B-A (500, 100 , 2, 3)} \\
\cmidrule(lr){2-6} \cmidrule(lr){7-11} \cmidrule(lr){12-16}
& deg. & clus. & orbit & GK & time & deg. & clus. & orbit & GK & time & deg. & clus. & orbit & GK & time \\
\midrule
DeepGMG & 1.315 & 1e-5 & 0.682 & 0.409 & 15.9 & 1.515 & 1.074 & 1.118 & 0.233 & 9.2 & 1.839 & 1.568 & 1.075 & 0.257 & 9.1 \\
GraphRNN & \bf{5e-4} & \bf{0} & \bf{2e-5} & 0.289 & 1.9 & \bf{0.011} & 0.121 & 0.024 & 0.061 & 3.9 & 0.359 & 0.096 & 0.130 & 0.025 & 3.4 \\
GraphRNN-S & 0.040 & 8e-4 & 0.001 & \bf{0.287} & 1.1 & 0.633 & 0.195 & 0.427 & 0.062 & 2.5 & 0.581 & 0.159 & 0.158 & 0.037 & 2.6 \\
\midrule
GRAM & 0.105 & 0.002 & 0.013 & \bf{0.287} & 0.8 & \bf{0.011} & 0.025 & \bf{0.009} & \bf{0.016} & 1.0 & \bf{0.034} & 0.009 & \bf{0.024} & \bf{0.015} & 1.0 \\
GRAM-A & 0.314 & 0.007 & 0.099 & 0.295 & 0.7 & 0.031 & \bf{0.004} & 0.038 & \bf{0.016} & 0.9 & 0.061 & 0.094 & 0.045 & \bf{0.015} & 1.0 \\
GRAM-B & 0.029 & 0.002 & 0.011 & \bf{0.287} & 0.7 & 0.029 & 0.056 & 0.015 & 0.019 & 0.9 & 0.063 & \bf{0.005} & 0.031 & \bf{0.015} & 1.0 \\
GRAM-AB & 0.357 & 0.015 & 0.126 & 0.309 & 0.7 & 0.164 & 0.080 & 0.080 & 0.020 & 0.9 & 0.044 & 0.015 & 0.031 & 0.020 & 0.9 \\
\bottomrule
\end{tabular}
\centering
% \scriptsize
\fontsize{8.4pt}{8.4pt}\selectfont
\begin{tabular}{lcccccccccccccccccc}
\toprule
& \multicolumn{5}{c}{Ego (605, 399, 1, 1)} & \multicolumn{5}{c}{Protein (734, 500, 89, 1)} & \multicolumn{5}{c}{Molecular (250k, 38, 9, 3)} \\
\cmidrule(lr){2-6} \cmidrule(lr){7-11} \cmidrule(lr){12-16}
& deg. & clus. & orbit & GK & time & deg. & clus. & orbit & GK & time & valid & unique & novel & GK & time \\
\midrule
DeepGMG & - & - & - & - & - & - & - & - & - & - & 84.5 & 99.1 & 83.8 & 0.0201 & - \\
GraphRNN & 0.117 & 0.615 & 0.043 & 0.028 & 16.0 & \bf{0.040} & 1.256 & 0.506 & 0.011 & 22.4 & 11.1 & \bf{100.0} & 11.1 & 0.0284 & 20.5 \\
GraphRNN-S & 0.306 & 0.337 & 0.314 & 0.067 & 11.4 & 0.523 & 1.247 & 0.123 & 0.019 & 13.7 & 26.2 & 99.9 & 26.2 & 0.0551 & 12.4 \\
\midrule
GRAM & 0.141 & 0.160 & 0.058 & 0.034 & 2.5 & 0.154 & 0.232 & 0.257 & 0.013 & 6.9 & 92.7 & \bf{100.0} & 92.7 & \bf{0.0198}& 9.5 \\
GRAM-A & 0.139 & 0.268 & 0.062 & 0.031 & 2.0 & 0.224 & 0.569 & 0.425 & 0.011 & 4.9 & \bf{95.0} & \bf{100.0} & \bf{95.0} & \bf{0.0198}& 9.4 \\
GRAM-B & \bf{0.108} & \bf{0.083} & \bf{0.033} & \bf{0.017} & 2.0 & 0.154 & 0.247 & \bf{0.040} & \bf{0.010} & 4.6 & 94.3 & \bf{100.0} & 94.2 & 0.0206 & 8.5 \\
GRAM-AB & 0.269 & 0.182 & 0.249 & \bf{0.017} & 2.0 & 0.046 & \bf{0.049} & 0.111 & \bf{0.010} & 4.7 & 94.9 & 99.8 & 94.8 & 0.0206 & 8.4 \\
\bottomrule
\end{tabular}
\vspace{-18pt}
\end{table}
\section{Conclusion}
In this work, we tackled the problem of scalability as it is one of the most important challenges in graph generation tasks.
We first defined scalability from three perspectives, and then we proposed a scalable graph generative model named GRAM along with some variants.
Also, we proposed a novel graph attention mechanism as a key portion of the model and constructed GK-MMD as a unified evaluation metric for node/edge-labeled graph generation tasks.
In an experiment using synthetic and real-world graphs, we verified the scalability and superior performances of our models.
\subsubsection*{Acknowledgments}
This work was partially supported by JST CREST Grant Number JPMJCR1403, and partially supported by JSPS KAKENHI Grant Number JP19H01115.
We would like to thank Atsuhiro Noguchi for his helpful advice on graphs.
We would also like to show our gratitude to Yujia Li for providing data of generated molecular graphs and giving permission to use it for comparison in the experiment.
Finally, we would like to appreciate every member of the lab for beneficial discussions, which inspired our work greatly.
\bibliography{iclr2020_conference}
\bibliographystyle{iclr2020_conference}
\clearpage
\appendix
\section{Appendix}
\subsection{Complexity Analysis}
\label{sec:A-complexity-analysis}
In this section, we compare the computational complexity of the training of our models with that of existing models.
Following \citet{NIPS2017_7181}, we consider two factors: total computational complexity and minimum sequential operations with parallelization.
Note that we assume all matrix-vector products can be computed in constant time.
Table \ref{tab:complexity-comparison} summarizes the results.
To generate a graph with $n$ nodes and $m$ edges,
DeepGMG \citep{2018arXiv180303324L} requires at least $\mathcal{O}(m n^2)$ operations.
Because the initialization of the state of the new node depends on the state in the previous step, DeepGMG’s minimum required sequential operations is $\mathcal{O}(n^2)$.
On the other hand, GraphRNN \citep{DBLP:conf/icml/YouYRHL18} requires $\mathcal{O}(n M)$ operations with a dataset-specific constant value $M$,
utilizing the properties of the BFS and relying mainly on the hidden states of the RNN.
However, its minimum number of sequential operations is $\mathcal{O}(n + M)$ because of the sequential dependency of the hidden states of the RNN; moreover, it cannot generate node/edge-labeled graphs.
Next, we evaluate GRAM and its variants.
We denote the average number of nodes/edges within distance $r$ of one node as $N_r$/$M_r$ and reuse the $\alpha$ and $\beta$ defined in Section \ref{sec:complexity-reduction}.
In one step, the feature extractor requires $\mathcal{O}(m + n N_r)$ operations,
considering a graph convolution layer requires $\mathcal{O}(m)$, and
a graph attention layer requires $\mathcal{O}(n N_r)$.
However, comparing the node features in $G_{<s}$ with those in $G_{<s+1} (=G_s \cup v_s \cup \{e_{1, s}, ..., e_{s-1, s}\})$, only nodes within distance $r L$ of $v_s$ change their features.
Utilizing this property, we conduct feature extraction only for $r L$-neighboring nodes of the previously added node and reuse features for the other nodes, by which the number of feature extraction operations is reduced to $\mathcal{O}(n (M_{rL} + N_{rL} N_r))$.
Also, the node estimator requires $\mathcal{O}(n)$ operations in total.
In edge estimation, GRAM requires $\mathcal{O}(n^3)$ operations, GRAM-B requires $\mathcal{O}(n \beta ^2)$, GRAM-A requires $\mathcal{O}(n^2 \alpha)$, and GRAM-AB requires $\mathcal{O}(n \alpha \beta)$.
Therefore, the total complexity of GRAM is $\mathcal{O}(n^3 + n (M_{rL} + N_{rL} N_r))$, that of GRAM-B is $\mathcal{O}(n \beta ^2 + n (M_{rL} + N_{rL} N_r))$, that of GRAM-A is $\mathcal{O}(n^2 \alpha + n (M_{rL} + N_{rL} N_r))$, and that of GRAM-AB is $\mathcal{O}(n \alpha \beta + n (M_{rL} + N_{rL} N_r))$.
Note that all the estimation operations of the node estimator and the edge estimator can be parallelized in training because our model has no sequentially dependent hidden states like the RNN does.
From the above analysis, we can see that our models require fewer operations than DeepGMG when $N_r \ll n$ holds, which is the often case, while keeping rich feature extraction.
In addition, assuming $N_{rL}$ and $M_{rL}$ are nearly constant and sufficiently smaller than $n$ and $m$, respectively,
the total complexities of GRAM-B and GRAM-AB can be regarded as almost linear in $n$, which is competitive with GraphRNN, but ours keep the feature extraction.
More importantly, all estimation operations can be parallelized, which facilitates training using multiple computing nodes.
Therefore, we can expect graph scalability and data scalability of our models.
We also expect label scalability because our models are flexible to a variable number of labels by modifying only the dimensions of the input and output layers.
\begin{table}[h]
\caption{Total computational complexity and minimum sequential operations during training.}
\vspace{9pt}
\label{tab:complexity-comparison}
\centering
\begin{tabular}{lcc}
\toprule
Method & Total Complexity & Minimum Sequential Operations \\
\midrule
DeepGMG & $\mathcal{O}(m n^2 )$ & $\mathcal{O}(n^2) $ \\
GraphRNN & $\mathcal{O}(n M)$ & $\mathcal{O}(n + M)$ \\
GRAM & $\mathcal{O}(n^3 + n (M_{rL} + N_{rL} N_r))$ & $\mathcal{O}(1)$ \\
GRAM-B & $\mathcal{O}(n \beta ^2 + n (M_{rL} + N_{rL} N_r))$ & $\mathcal{O}(1)$ \\
GRAM-A & $\mathcal{O}(n^2 \alpha + n (M_{rL} + N_{rL} N_r))$ & $\mathcal{O}(1)$ \\
GRAM-AB & $\mathcal{O}(n \alpha \beta + n (M_{rL} + N_{rL} N_r))$ & $\mathcal{O}(1)$ \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Example of Tensor Representation}
\label{sec:A-tensor-representation}
An example of tensor representation is illustrated in Figure \ref{fig:tensor-representation}.
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{tensor_representation-crop.pdf}
\caption{Example of tensor representation.
$n, a$, and $b$ denote the numbers of nodes, node labels, and edge labels, respectively.
$X^\pi\in \{0, 1\}^{n \times a}$, whose $i$-th row is a one-hot vector corresponding to the label of $v_{\pi(i)}$, stores information about nodes.
$A^\pi\in \{0, 1\}^{n \times n \times b}$, whose $(i, j)$-th element is a one-hot vector corresponding to the label of $e_{\pi(i), \pi(j)}$, stores information about edges.
If no edge exists between $v_{\pi(i)}$ and $v_{\pi(j)}$, we replace it with a zero vector.
Note that $(G, \pi)$ and $(X^\pi, A^\pi)$ correspond uniquely.
}
\label{fig:tensor-representation}
\end{figure}
\subsection{Detailed Model Pipeline}
\label{sec:A-pipeline-large}
A detailed pipeline of GRAM is illustrated in Figure \ref{fig:pipeline-large}.
\begin{figure}[h]
\centering
\includegraphics[width=0.9\linewidth]{pipeline-crop.pdf}
\caption{$s$-th step of the generation process ($s = 6, t = 3$).
In its $s$-th step, given a subgraph $G_{<s}$, our model estimates the new node $v_s$ and new edges $\{e_{1, s}, ..., e_{s-1, s}\}$ to add next.
Specifically, it takes a tensor pair $(X_{<s}, A_{<s, <s})$ as input and predicts the label of the new node $X_s$ and the labels of the new edges $A_{t, s} ~ (t=1, ..., s-1)$.
In detail, first, the feature extractor calculates $s-1$ node feature vectors $H^v = (\bm{h}^v_1, ..., \bm{h}^v_{s-1})^T$ and a graph feature vector $\bm{h}^G$ from a tensor pair $(X_{<s}, A_{<s, <s})$.
Next, the node estimator predicts the label of the new node $X_s$ from the graph feature vector.
When the EOS is output, the generation is terminated.
Finally, the edge estimator predicts the labels of the new edges $A_{t, s} ~ (t=1, ..., s-1)$ from the node feature vectors, the graph feature vector, the embedded vector of the predicted node label, and the embedded vectors of previously predicted edge labels in this step.
}
\label{fig:pipeline-large}
\end{figure}
\subsection{Illustration of Graph Attention}
\label{sec:A-graph-attention-illustration}
The graph attention mechanism in feature extraction (self attention) is illustrated in Figure \ref{fig:graph-attention-self-attention} and that in edge estimation (source target attention) is illustrated in Figure \ref{fig:graph-attention-source-target-attention}.
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{self_attention-crop.pdf}
\caption{Operation of graph attention in the $l$-th feature extraction block, where query, key, and value vectors are all node feature vectors (i.e., $Q=K=V=H^v = (\bm{h}^{v (l-1)}_1, ..., \bm{h}^{v (l-1)}_{s-1})^T$) (self attention).
For simplicity, we focused on one query/output vector and one subspace.
}
\label{fig:graph-attention-self-attention}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.8\linewidth]{source_target_attention-crop.pdf}
\caption{Operation of graph attention in the estimation of the new edge $e_{t, s}$ ($s = 6, t = 3$).
The query vector is $\mathrm{Concat}(\bm{h}^v_t, \bm{h}^v_s)$, and the value and key vectors are $\{\mathrm{Concat}(\bm{h}^v_\tau, \bm{h}^v_s, \bm{h}^e_{\tau, s})|\tau=1, ..., t-1\}$, where $\bm{h}^v_t$ is the feature vector of the node $v_t$, $\bm{h}^v_s$ is the embedded vector of the predicted label of the new node $v_s$, $X_s$, and $\bm{h}^e_{\tau, s}$ is the embedded vector of the predicted label of the new edge $e_{\tau, s}$, $A_{\tau, s}$.
In other words, $Q = \mathrm{Concat}(\bm{h}^v_t, \bm{h}^v_s)^T$, $K = V = (\mathrm{Concat}(\bm{h}^v_1, \bm{h}^v_s, \bm{h}^e_{1, s}), ..., \mathrm{Concat}(\bm{h}^v_{t-1}, \bm{h}^v_s, \bm{h}^e_{t-1, s}))^T$ (source target attention).
}
\label{fig:graph-attention-source-target-attention}
\end{figure}
\clearpage
\subsection{GRAM-B: Utilizing Breadth-First Search Node Ordering}
\label{sec:A-bfs}
In this section, a brief and intuitive proof for the proposition in Section \ref{sec:bfs} is given along with additional discussion on how much the computational complexity is actually reduced by this approximation and its strong topology.
\subsubsection{Proof}
Consider the BFS tree in the left of Figure \ref{fig:bfs-tree}.
Suppose there is at least one connection between the new node $v_s$ and the non-frontier nodes $V \setminus V_f$, where $V_{f} = \{v_\tau | \min (\{i | v_i \text{ is connected to } v_{s-1} \}) \le \tau \le s-1 \}$ (right of Figure \ref{fig:bfs-tree}).
Then, under the assumption of BFS node ordering, $v_s$ must be visited/generated before $v_{s-1}$, which is a contradiction.
Therefore, the nodes that can have connections between the new node $v_s$ are limited to the frontier nodes $V_f$ (middle of Figure \ref{fig:bfs-tree}).
\begin{figure}[h]
\centering
\includegraphics[width=0.8\linewidth]{bfs_tree-crop.pdf}
\caption{BFS tree example of $G_{<s}$ (left), possible connections to the new node $v_s$ (middle), and impossible connections to the new node $v_s$ (right).}
\label{fig:bfs-tree}
\end{figure}
\subsubsection{Discussion}
The average values of $\beta = |V_f|$ for each dataset are listed in Table \ref{tab:alhpa-beta}.
In both the synthetic and real-world graphs, we can see that $\beta$ is relatively small compared to $n$ (approximately $1/10 \sim 1/3$).
Therefore, considering the results in Table \ref{tab:complexity-comparison}, the average computational cost of GRAM-B is estimated to be approximately $1/100 \sim 1/9$ that of GRAM, which is a significant reduction.
In the experiment, GRAM-B performed better than its original model on the grid and ego datasets.
This is likely because these graphs have simple topologies and their BFS trees also have typical simple patterns.
Therefore, restricting the range of edge estimation to the frontier nodes $V_f$ made the problem easier for the model.
Thus, we expect GRAM-B’s performance to be boosted when the topologies of the graphs and their BFS trees have simple patterns.
\begin{table}[h]
\caption{Average values of $\alpha$ and $\beta$ and the number of nodes $n$ per graph for each dataset.}
\vspace{9pt}
\label{tab:alhpa-beta}
\centering
\begin{tabular}{lccc}
\toprule
& $\alpha$ & $\beta$ & $n$ \\
\midrule
Grid & 1.8 & 9.0 & 72.2 \\
Lobster & 1.0 & 4.2 & 76.0 \\
Community & 2.7 & 23.5 & 74.2 \\
B-A & 3.8 & 31.9 & 72.7 \\
Molecular & 1.1 & 3.6 & 23.2 \\
Ego & 2.1 & 45.8 & 141.1 \\
Protein & 2.5 & 25.7 & 260.2 \\
\bottomrule
\end{tabular}
\end{table}
\subsection{GRAM-A: Zeroing Attention Weights}
\label{sec:A-attention-zeroing}
Here, we provide an additional discussion on the actual computational complexity reduction by this approximation and its strong graph types.
Figure \ref{fig:attention-weight-zeroing} illustrates the attention weight zeroing in edge estimation.
Figure \ref{fig:attention-weight-hist} shows the two distributions of attention weights in one step of edge estimation (i.e., in the calculation of $\bm{h}^e_{<t}$) on the grid dataset.
The blue represents the distribution of weights for nodes that are predicted to have no connection to the new node $v_s$ (i.e., $\{v_\tau|{A}^\pi_{\tau, s} = \bm{0}, \tau = 1, ..., t-1\}$),
and the orange represents the distribution of weights for those predicted to have connection to the new node (i.e., $\{v_\tau|{A}^\pi_{\tau, s} \neq \bm{0}, \tau = 1, ..., t-1\}$).
From this figure, we can see that the former distribution is sharper around zero compared with the latter, most of whose weights take near-zero values.
Similar results were observed for the other datasets.
The average values of $\alpha$ for each dataset are listed in Talbe \ref{tab:alhpa-beta}.
In both the synthetic and real-world graphs, we can see that $\alpha$ is less than 4 and approximately $1/100 \sim 1/20$ of $n$.
Thus, considering the results in Table \ref{tab:complexity-comparison}, the computational cost of GRAM-A is estimated to be approximately $1/100 \sim 1/20$ that of GRAM on average, which is a dramatic reduction.
In the experiment, GRAM-A performed better than GRAM on the molecular dataset.
This is likely because these graphs have strict rules or distinct patterns in incident edges to a node, such as the valence rule, and focusing only on the actual generated edges is both sufficient and makes the problem simpler.
Thus, we expect that GRAM-A will demonstrate better performance when the graphs follow a certain strict rule on incident edges of their nodes.
\begin{figure}[h]
\centering
\includegraphics[width=0.6\linewidth]{attention_weight_hist-crop.pdf}
\caption{Distributions of attention weights in edge estimation on the grid dataset.
The blue represents the distribution of nodes predicted to have no connection to the new node, and the orange represents that of nodes predicted to have connection.
More than 90 \% of attention weights are less than 0.05 for the former distribution, whereas approximately 70 \% of attention weights are less than 0.05 and more than 15 \% are larger than 0.95 for the latter.
Note that the scale of the y-axis is different between the top and halves.
}
\label{fig:attention-weight-hist}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.5\linewidth]{attention_weight_zeroing-crop.pdf}
\caption{Illustration of attention weight zeroing. In the estimation of $e_{t, s}$, we ignore nodes that are predicted to have no connection with $v_s$ ($v_2$ or $e_{2, s}$ in this case) and set their attention weights to zero deterministically.}
\label{fig:attention-weight-zeroing}
\vspace{-8pt}
\end{figure}
\subsection{Experimental Setting Details}
\label{sec:A-experiment}
\subsubsection{Dataset Configuration}
Here, the detailed configurations of the synthetic graphs are described.
Samples from each synthetic graph dataset are illustrated in Figures \ref{fig:grid-generated-graphs}, \ref{fig:lobster-generated-graphs}, \ref{fig:community-generated-graphs}, and \ref{fig:BA-generated-graphs} (color is best).
Different color corresponds to different node/edge labels on each dataset.
\paragraph{Grid}
We used 2D grid graphs with the number of nodes in the range $50 \le |V| \le 100$.
Each node was labeled according to its degree, yielding 3 types of node labels: ``corner'', ``edge'', and ``inside''.
Each edge was labeled according to its geometric direction, yielding 2 types of edge labels: ``horizontal'' and ``vertical''.
\paragraph{Lobster}
We used lobster graphs with the number of nodes in the range $50 \le |V| \le 100$.
We set probability of adding an edge to the backbone $p_1 = 0.7$, and probability of adding an edge one level beyond the backbone $p_2 = 0.3$.
Each node was labeled according to its distance from backbone, yielding 3 types of node labels: ``leaf'', ``branch'', and ``backbone''.
Each edge was labeled according to its incident nodes, yielding 2 types of edge labels: ``leaf-branch'' and ``branch-backbone''.
\paragraph{Community}
We used 4-community graphs \citep{fortunato2010community} with the number of nodes in the range $50 \le |V| \le 100$.
We set number of communities $l = 4$, number of nodes in one community $k = |V| / 4$, probability of intra-connection $p_{in} = 0.23$, and probability of inter-connection $p_{out} = 0.023$.
Each node was labeled according to the community it belongs to, yielding 4 types of node labels: ``community1'', ``community2'', ``community3'', and ``community4''.
Each edge was labeled according to its incident nodes, yielding 2 types of edge labels: ``intra-community connection'' and ``inter-community connection''.
\paragraph{B-A}
We used Barab\'asi-Albert (B-A) graphs \citep{barabasi1999emergence} with the number of nodes in the range $50 \le |V| \le 100$.
We set number of edges to connect a new node and existing nodes as $m = 4$.
Each node was labeled according to whether its degree falls in the top 50\% or bottom 50\%, yielding 2 types of node labels: ``hub'' and ``exterior''.
Each edge was labeled according to its incident nodes, yielding 3 types of edge labels: ``hub-hub'', ``hub-exterior'', and ``exterior-exterior''.
\subsubsection{Training Configuration}
Here, we give details of the training configurations.
We essentially used the default settings for the baseline methods.
For all datasets, DeepGMG was trained using a single GPU for 10 epochs with batch size 1 because its generation process is so complex that batch implementation is difficult.
GraphRNN and GraphRNN-S were trained using a single GPU for 96000 iterations with batch size 32.
For the molecular dataset, considering its relatively large dataset size, we increased the number of iterations to 1536000.
For the convenience of implementation, we started the generation process from seed subgraphs with the number of nodes $N_{\min}$.
For the synthetic graph datasets, we trained our models using a single GPU for 100 epochs with batch size 32, and we used $r=2$ and $N_{\min} = 10$.
For the molecular dataset, we trained our models using 2 GPUs for 20 epochs with batch size 256, and we used $r=7$ and $N_{\min} = 5$.
For the ego dataset, we trained our models using 2 GPUs for 25 epochs, and we used $r=2$ and $N_{\min} = 12$.
We used batch size 4 for GRAM and batch size 8 for GRAM-A, GRAM-B, and GRAM-AB.
For the protein dataset, we trained our models using 2 GPUs for 25 epochs, and we used $r=2$ and $N_{\min} = 12$.
We used batch size 4 for GRAM and batch size 32 for GRAM-A, GRAM-B, and GRAM-AB.
Due to the code dependencies, Tesla P100 was used for DeepGMG, GraphRNN, and GraphRNN-S, and Tesla V100 was used for GRAM and its variants.
Empirically, the choice between the two gave no significant difference in computation time for our cases.
In evaluation, we reported an average of 3 runs for each score.
To calculate GK-MMD score for the molecular dataset, we used 100 samples from the generated graphs and 100 samples from the test set for fast evaluation.
We reported an average of 10 runs.
\subsubsection{Additional Experiment Result}
The results on the lobster dataset are listed in Table \ref{tab:synthetic-graphs2}.
As on other datasets, GRAM and its variants demonstrated competitive results in terms of topological MMDs, and better performance in terms of GK-MMD.
\begin{table}[h]
\caption{Results on lobster dataset. Dataset size $N$, maximum graph size $|V|_{\max}$, and the number of node/edge labels $a, b$ are shown as $(N, |V|_{\max}, a, b)$.}
\vspace{9pt}
\centering
\label{tab:synthetic-graphs2}
\begin{tabular}{lcccccccccccccccccccc}
\toprule
& \multicolumn{5}{c}{Lobster (500, 100, 3, 4)} \\
\cmidrule(lr){2-6}
& deg. & clus. & orbit & GK & time \\
\midrule
DeepGMG & 0.958 & 8e-5 & 0.035 & 0.527 & 13.5 \\
GraphRNN & \bf{1e-4} & \bf{0} & \bf{0} & 0.203 & 2.2 \\
GraphRNN-S & 0.020 & 0.023 & 4e-4 & 0.221 & 1.4 \\
\midrule
GRAM & 0.001 & 5e-4 & 6e-5 & 0.049 & 0.9 \\
GRAM-A & 0.001 & 0.001 & 8e-4 & \bf{0.043} & 0.8 \\
GRAM-B & 0.001 & 8e-5 & 2e-4 & 0.048 & 0.7 \\
GRAM-AB & 0.001 & 2e-5 & 7e-5 & 0.047 & 0.7 \\
\bottomrule
\end{tabular}
\end{table}
In addition, considering that the molecular dataset is a subset of the druglike category in ZINC database \citep{doi:10.1021/acs.jcim.5b00559}, we reported the average QED scores \citep{QED} of the training samples and the generated samples.
QED score measures the drug likeliness of the molecular.
Table \ref{tab:qed} summarizes the results.
We can see the generated molecular graphs of our models have higher QED compared with those of baselines, and near to that of the training set.
Note that our models do not rely on reinforcement learning methods and utilize neither the information of QED nor valence rule;
they learned only through unsupervised learning.
\begin{table}[h]
\caption{Average QED score.}
\vspace{9pt}
\centering
\label{tab:qed}
\begin{tabular}{lc}
\toprule
& QED \\
\midrule
Train & 0.732 \\
\midrule
DeepGMG & 0.644 \\
GraphRNN & 0.563 \\
GraphRNN-S & 0.496 \\
\midrule
GRAM & \bf{0.722} \\
GRAM-A & \bf{0.722} \\
GRAM-B & 0.691 \\
GRAM-AB & 0.694 \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Ablation Study}
To examine the effectiveness of the bias terms in the graph attention mechanism, we conducted an ablation study.
Specifically, we evaluated two variants: GRAM without bias terms of graph attention in edge estimation, and GRAM without bias terms of graph attention in feature extraction.
The results are listed in Table \ref{tab:ablation-study}.
On all the synthetic graph datasets, removing bias terms in edge estimation degraded its performance.
From this, we can see the importance of the bias terms in edge estimation for the better quality of generation.
Also, removing bias terms in feature extraction degraded the performance on the B-A dataset.
This is likely because its expressive power is limited.
On the other hand, the drop in the performance by removing bias terms in feature extraction is minor on the grid, lobster and community dataset.
Considering the properties of these random graphs, this is likely because only local features were enough for these cases.
\begin{table}[h]
\caption{Results of ablation study.}
\vspace{9pt}
% \setlength{\tabcolsep}{2.5pt}
\label{tab:ablation-study}
\centering
% \scriptsize
% \footnotesize
% \fontsize{8.8pt}{8.8pt}\selectfont
\begin{tabular}{lcccccccccccccccccccc}
\toprule
& \multicolumn{4}{c}{Grid} & \multicolumn{4}{c}{Community } \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9}
& deg. & clus. & orbit & GK & deg. & clus. & orbit & GK \\
\midrule
GRAM & \bf{0.105} & 0.002 & 0.013 & \bf{0.287} & 0.011 & 0.025 & \bf{0.009} & \bf{0.016} \\
w/o bias in feature extraction & 0.111 & \bf{0.001} & \bf{0.009} & \bf{0.287} & \bf{0.006} & \bf{0.007} & 0.016 & \bf{0.016} \\
w/o bias in edge estimation & 0.247 & 0.272 & 0.190 & 0.306 & 0.066 & 0.037 & 0.038 & 0.017 \\
\bottomrule
\end{tabular}
\centering
\begin{tabular}{lccccccccccccccccccc}
\toprule
& \multicolumn{4}{c}{B-A} & \multicolumn{4}{c}{Lobster} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9}
& deg. & clus. & orbit & GK & deg. & clus. & orbit & GK \\
\midrule
GRAM & \bf{0.034} & \bf{0.009} & \bf{0.024} & \bf{0.015} & \bf{0.001} & 5e-4 & \bf{6e-5} & \bf{0.049} \\
w/o bias in feature extraction & \bf{0.034} & 0.020 & 0.033 & 0.016 & 0.003 & \bf{1e-4} & 2e-4 & 0.054 \\
w/o bias in edge estimation & 0.050 & 0.035 & 0.042 & 0.017 & 0.002 & 3e-4 & 8e-5 & 0.054 \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Visualization of Generated Graphs}
Visualizations of generated graphs on each dataset are shown in Figures \ref{fig:grid-generated-graphs}, \ref{fig:lobster-generated-graphs}, \ref{fig:community-generated-graphs}, \ref{fig:BA-generated-graphs} and \ref{fig:molecule-generated-graphs} (color is best).
We used the same visualization method for all graphs.
Different color corresponds to different node/edge labels on each dataset.
The graphs generated by GraphRNN are quite similar to those in the training set in terms of their topology.
However, they completely failed to capture node/edge label patterns in most cases.
In contrast, the graphs generated by GRAM are similar to training samples in terms of both topology and node/edge label pattens.
\begin{figure}[h]
\centering
\begin{minipage}{0.7\linewidth}
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize Train}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_train_1.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_train_4.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_train_5.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GraphRNN}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_GraphRNN_RNN_0.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_GraphRNN_RNN_30.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_GraphRNN_RNN_3.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GRAM}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_GRAM_10.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_GRAM_12.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{grid_medium_GRAM_5.pdf}
\end{minipage}
\caption{Generated graphs (grid). }
\label{fig:grid-generated-graphs}
\end{minipage}
\end{figure}
\begin{figure}[h]
\centering
\begin{minipage}{0.7\linewidth}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize Train}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_train_0.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_train_14.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_train_2.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GraphRNN}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_GraphRNN_RNN_0.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_GraphRNN_RNN_3.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_GraphRNN_RNN_2.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GRAM}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_GRAM_0.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_GRAM_10.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{lobster_medium_GRAM_8.pdf}
\end{minipage}
\caption{Generated graphs (lobster).}
\label{fig:lobster-generated-graphs}
\end{minipage}
\end{figure}
\begin{figure}[h]
\centering
\begin{minipage}{0.7\linewidth}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize Train}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_train_1.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_train_4.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_train_5.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GraphRNN}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_GraphRNN_RNN_1.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_GraphRNN_RNN_4.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_GraphRNN_RNN_5.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GRAM}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_GRAM_1.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_GRAM_4.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{community_medium_GRAM_5.pdf}
\end{minipage}
\caption{Generated graphs (community).}
\label{fig:community-generated-graphs}
\end{minipage}
\end{figure}
\begin{figure}[h]
\centering
\begin{minipage}{0.7\linewidth}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize Train}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_train_0.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_train_14.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_train_2.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GraphRNN}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_GraphRNN_RNN_9.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_GraphRNN_RNN_10.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_GraphRNN_RNN_2.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GRAM}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_GRAM_0.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_GRAM_14.pdf}
\end{minipage}
\begin{minipage}{0.31\linewidth}
\includegraphics[width=1.1\linewidth]{BA_medium_GRAM_2.pdf}
\end{minipage}
\caption{Generated graphs (B-A).}
\label{fig:BA-generated-graphs}
\end{minipage}
\end{figure}
\begin{figure}[h]
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize Train}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_train_0.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_train_1.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_train_2.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_train_13.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_train_4.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_train_5.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GraphRNN}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GraphRNN_RNN_0.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GraphRNN_RNN_1.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GraphRNN_RNN_2.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GraphRNN_RNN_3.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GraphRNN_RNN_4.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GraphRNN_RNN_5.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize DeepGMG}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_Baseline_DGMG_0.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_Baseline_DGMG_1.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_Baseline_DGMG_2.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_Baseline_DGMG_3.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_Baseline_DGMG_4.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_Baseline_DGMG_5.pdf}
\end{minipage}
\centering
\begin{minipage}{0.01\linewidth}
\rotatebox{90}{\footnotesize GRAM}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GRAM_0.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GRAM_1.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GRAM_6.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GRAM_3.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GRAM_7.pdf}
\end{minipage}
\begin{minipage}{0.15\linewidth}
\includegraphics[width=\linewidth]{zinc_GRAM_5.pdf}
\end{minipage}
\caption{Generated molecular graphs.}
\label{fig:molecule-generated-graphs}
\end{figure}
\subsection{Limitations and Future Work}
While our models demonstrated superior performance on most of the datasets, they have some limitations.
For example, on inference, one wrong prediction of node/edge label may collapse all the following generation processes, resulting in a bizarre graph generated.
This is a deficit of its rich feature extraction process.
A possible solution to this problem is, for example, to employ beam search.
By applying beam search on inference, we can get rid of samples with a quite lower likelihood and avoid the above undesirable situations.
Additionally, although we reduced the computational complexity of the models by a significant amount, the feature extraction process still requires a large amount of computation.
To this challenge, some methods proposed in the field of image processing and natural language processing would help to alleviate it.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{31}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abu-El-Haija et~al.(2018)Abu-El-Haija, Perozzi, Al-Rfou, and
Alemi]{NIPS2018_8131}
Sami Abu-El-Haija, Bryan Perozzi, Rami Al-Rfou, and Alexander~A Alemi.
\newblock Watch your step: Learning node embeddings via graph attention.
\newblock In \emph{Advances in Neural Information Processing Systems 31}. 2018.
\bibitem[Airoldi et~al.(2009)Airoldi, Blei, Fienberg, and Xing]{NIPS2008_3578}
Edo~M Airoldi, David~M. Blei, Stephen~E. Fienberg, and Eric~P. Xing.
\newblock Mixed membership stochastic blockmodels.
\newblock In \emph{Advances in Neural Information Processing Systems 21}. 2009.
\bibitem[Albert \& Barab\'asi(2002)Albert and Barab\'asi]{RevModPhys.74.47}
R\'eka Albert and Albert-L\'aszl\'o Barab\'asi.
\newblock Statistical mechanics of complex networks.
\newblock \emph{Rev. Mod. Phys.}, 74:\penalty0 47--97, Jan 2002.
\bibitem[Aronszajn(1950)]{10.2307/1990404}
N.~Aronszajn.
\newblock Theory of reproducing kernels.
\newblock \emph{Transactions of the American Mathematical Society}, 68\penalty0
(3):\penalty0 337--404, 1950.
\bibitem[Barab{\'a}si \& Albert(1999)Barab{\'a}si and
Albert]{barabasi1999emergence}
Albert-L{\'a}szl{\'o} Barab{\'a}si and R{\'e}ka Albert.
\newblock Emergence of scaling in random networks.
\newblock \emph{science}, 286\penalty0 (5439):\penalty0 509--512, 1999.
\bibitem[Bickerton et~al.(2012)Bickerton, Paolini, Besnard, Muresan, and
Hopkins]{QED}
G.~Richard Bickerton, Gaia~V. Paolini, J{\'e}r{\'e}my Besnard, Sorel Muresan,
and Andrew~L. Hopkins.
\newblock Quantifying the chemical beauty of drugs.
\newblock \emph{Nature Chemistry}, 4:\penalty0 90 EP --, 01 2012.
\bibitem[Costa \& Grave(2010)Costa and Grave]{CostaFabrizio2010Fnsp}
Fabrizio Costa and Kurt~De Grave.
\newblock Fast neighborhood subgraph pairwise distance kernel.
\newblock In \emph{Proceedings of the 27th International Conference on
International Conference on Machine Learning}, 2010.
\bibitem[{De Cao} \& {Kipf}(2018){De Cao} and {Kipf}]{2018arXiv180511973D}
Nicola {De Cao} and Thomas {Kipf}.
\newblock {MolGAN: An implicit generative model for small molecular graphs}.
\newblock \emph{arXiv e-prints}, art. arXiv:1805.11973, 2018.
\bibitem[Dobson \& Doig(2003)Dobson and Doig]{DOBSON2003771}
Paul~D. Dobson and Andrew~J. Doig.
\newblock Distinguishing enzyme structures from non-enzymes without alignments.
\newblock \emph{Journal of Molecular Biology}, 330\penalty0 (4):\penalty0 771
-- 783, 2003.
\newblock ISSN 0022-2836.
\bibitem[Erd{\"o}s \& R{\'e}nyi(1959)Erd{\"o}s and R{\'e}nyi]{erdos1959random}
Paul Erd{\"o}s and Alfr{\'e}d R{\'e}nyi.
\newblock On random graphs, i.
\newblock \emph{Publicationes Mathematicae (Debrecen)}, 6:\penalty0 290--297,
1959.
\bibitem[Fortunato(2010)]{fortunato2010community}
Santo Fortunato.
\newblock Community detection in graphs.
\newblock \emph{Physics reports}, 486\penalty0 (3-5):\penalty0 75--174, 2010.
\bibitem[Gretton et~al.(2012)Gretton, Borgwardt, Rasch, Sch\"{o}lkopf, and
Smola]{Gretton:2012:KTT:2188385.2188410}
Arthur Gretton, Karsten~M. Borgwardt, Malte~J. Rasch, Bernhard Sch\"{o}lkopf,
and Alexander Smola.
\newblock A kernel two-sample test.
\newblock \emph{J. Mach. Learn. Res.}, 13:\penalty0 723--773, 2012.
\bibitem[{Grover} et~al.(2018){Grover}, {Zweig}, and
{Ermon}]{2018arXiv180310459G}
Aditya {Grover}, Aaron {Zweig}, and Stefano {Ermon}.
\newblock {Graphite: Iterative Generative Modeling of Graphs}.
\newblock \emph{arXiv e-prints}, art. arXiv:1803.10459, 2018.
\bibitem[{Ishiguro} et~al.(2019){Ishiguro}, {Maeda}, and
{Koyama}]{2019arXiv190201020I}
Katsuhiko {Ishiguro}, Shin-ichi {Maeda}, and Masanori {Koyama}.
\newblock {Graph Warp Module: an Auxiliary Module for Boosting the Power of
Graph Neural Networks}.
\newblock \emph{arXiv e-prints}, art. arXiv:1902.01020, 2019.
\bibitem[Johnson et~al.(2018)Johnson, Gupta, and Fei-Fei]{johnson2018image}
Justin Johnson, Agrim Gupta, and Li~Fei-Fei.
\newblock Image generation from scene graphs.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, 2018.
\bibitem[Kriegel et~al.(2006)Kriegel, Borgwardt, Gretton, Sch^^c3^^b6lkopf,
Rasch, and Smola]{10.1093/bioinformatics/btl242}
Hans-Peter Kriegel, Karsten~M. Borgwardt, Arthur Gretton, Bernhard
Sch^^c3^^b6lkopf, Malte~J. Rasch, and Alex~J. Smola.
\newblock {Integrating structured biological data by Kernel Maximum Mean
Discrepancy}.
\newblock \emph{Bioinformatics}, 22\penalty0 (14):\penalty0 e49--e57, 07 2006.
\bibitem[{Kusner} et~al.(2017){Kusner}, {Paige}, and
{Hern{\'a}ndez-Lobato}]{2017arXiv170301925K}
Matt~J. {Kusner}, Brooks {Paige}, and Jos{\'e}~Miguel {Hern{\'a}ndez-Lobato}.
\newblock {Grammar Variational Autoencoder}.
\newblock \emph{arXiv e-prints}, art. arXiv:1703.01925, 2017.
\bibitem[Leskovec et~al.(2010)Leskovec, Chakrabarti, Kleinberg, Faloutsos, and
Ghahramani]{Leskovec:2010:KGA:1756006.1756039}
Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos, and
Zoubin Ghahramani.
\newblock Kronecker graphs: An approach to modeling networks.
\newblock \emph{J. Mach. Learn. Res.}, 11:\penalty0 985--1042, March 2010.
\bibitem[Li et~al.(2018)Li, Zhang, and Liu]{Li2018}
Yibo Li, Liangren Zhang, and Zhenming Liu.
\newblock Multi-objective de novo drug design with conditional graph generative
model.
\newblock \emph{Journal of Cheminformatics}, 10\penalty0 (1):\penalty0 33,
2018.
\bibitem[{Li} et~al.(2018){Li}, {Vinyals}, {Dyer}, {Pascanu}, and
{Battaglia}]{2018arXiv180303324L}
Yujia {Li}, Oriol {Vinyals}, Chris {Dyer}, Razvan {Pascanu}, and Peter
{Battaglia}.
\newblock {Learning Deep Generative Models of Graphs}.
\newblock \emph{arXiv e-prints}, art. arXiv:1803.03324, 2018.
\bibitem[Liu et~al.(2018)Liu, Allamanis, Brockschmidt, and
Gaunt]{NIPS2018_8005}
Qi~Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander Gaunt.
\newblock Constrained graph variational autoencoders for molecule design.
\newblock In \emph{Advances in Neural Information Processing Systems 31}. 2018.
\bibitem[Luo et~al.(2018)Luo, Tian, Qin, Chen, and Liu]{NIPS2018_8007}
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu.
\newblock Neural architecture optimization.
\newblock In \emph{Advances in Neural Information Processing Systems 31}. 2018.
\bibitem[Robins et~al.(2007)Robins, Pattison, Kalish, and
Lusher]{ROBINS2007173}
Garry Robins, Pip Pattison, Yuval Kalish, and Dean Lusher.
\newblock An introduction to exponential random graph (p*) models for social
networks.
\newblock \emph{Social Networks}, 29\penalty0 (2):\penalty0 173 -- 191, 2007.
\bibitem[Sen et~al.(2008)Sen, Namata, Bilgic, Getoor, Galligher, and
Eliassi-Rad]{Sen_Namata_Bilgic_Getoor_Galligher_Eliassi-Rad_2008}
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher,
and Tina Eliassi-Rad.
\newblock Collective classification in network data.
\newblock \emph{AI Magazine}, 29\penalty0 (3):\penalty0 93, Sep. 2008.
\bibitem[Simonovsky \& Komodakis(2018)Simonovsky and
Komodakis]{Simonovsky2018GraphVAETG}
Martin Simonovsky and Nikos Komodakis.
\newblock Graphvae: Towards generation of small graphs using variational
autoencoders.
\newblock In \emph{ICANN}, 2018.
\bibitem[Sterling \& Irwin(2015)Sterling and
Irwin]{doi:10.1021/acs.jcim.5b00559}
Teague Sterling and John~J. Irwin.
\newblock Zinc 15 ^^e2^^80^^93 ligand discovery for everyone.
\newblock \emph{Journal of Chemical Information and Modeling}, 55\penalty0
(11):\penalty0 2324--2337, 2015.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
Kaiser, and Polosukhin]{NIPS2017_7181}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, \L~ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock In \emph{Advances in Neural Information Processing Systems 30}. 2017.
\bibitem[Veli{\v{c}}kovi{\'{c}} et~al.(2018)Veli{\v{c}}kovi{\'{c}}, Cucurull,
Casanova, Romero, Li{\`{o}}, and Bengio]{velickovic2018graph}
Petar Veli{\v{c}}kovi{\'{c}}, Guillem Cucurull, Arantxa Casanova, Adriana
Romero, Pietro Li{\`{o}}, and Yoshua Bengio.
\newblock {Graph Attention Networks}.
\newblock \emph{International Conference on Learning Representations}, 2018.
\bibitem[{Wang} et~al.(2017){Wang}, {Wang}, {Wang}, {Zhao}, {Zhang}, {Zhang},
{Xie}, and {Guo}]{2017arXiv171108267W}
Hongwei {Wang}, Jia {Wang}, Jialin {Wang}, Miao {Zhao}, Weinan {Zhang}, Fuzheng
{Zhang}, Xing {Xie}, and Minyi {Guo}.
\newblock {GraphGAN: Graph Representation Learning with Generative Adversarial
Nets}.
\newblock \emph{arXiv e-prints}, art. arXiv:1711.08267, 2017.
\bibitem[You et~al.(2018{\natexlab{a}})You, Liu, Ying, Pande, and
Leskovec]{NIPS2018_7877}
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec.
\newblock Graph convolutional policy network for goal-directed molecular graph
generation.
\newblock In \emph{Advances in Neural Information Processing Systems 31}.
2018{\natexlab{a}}.
\bibitem[You et~al.(2018{\natexlab{b}})You, Ying, Ren, Hamilton, and
Leskovec]{DBLP:conf/icml/YouYRHL18}
Jiaxuan You, Rex Ying, Xiang Ren, William~L. Hamilton, and Jure Leskovec.
\newblock Graphrnn: Generating realistic graphs with deep auto-regressive
models.
\newblock In \emph{Proceedings of the 35th International Conference on Machine
Learning, {ICML} 2018}, 2018{\natexlab{b}}.
\end{thebibliography}