Universal approximations of permutation invariant/equivariant functions by deep neural networks
Extracted source text · arXiv 1903.01939
\documentclass{article} % For LaTeX2e
\usepackage{iclr2020_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\title{Universal approximations of permutation invariant/equivariant functions by deep neural networks}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Akiyoshi Sannai \& Yuuki Takai \\
RIKEN Center for Advanced Intelligence Project/Keio University\\
Tokyo, Japan \\
\texttt{\{akiyoshi.sannai, yuuki.takai\}@riken.jp} \\
\And
Matthieu Cordonnier \\
\'{E}cole Normale Sup\'{e}rieure Paris-Saclay, \\
Cachan, France\\
\texttt{matthieu.cordonnier@ens-paris-saclay.fr} \\
%\AND
%Coauthor \\
%Affiliation \\
%Address \\
%\texttt{email}
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% commented out (previous version, 2019/09/23)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% In this paper, we develop a theory about the relationship between permutation invariant/equivariant functions and deep neural networks.
% Our main contribution is to prove an invariant/equivariant version of the universal approximation theorem concerning what we call $G$-invariant/equivariant deep neural networks for a finite group $G$.
% A $G$-equivariant deep neural network consists of stacking standard single-layer neural networks $F_i\colon X_i \to X_{i+1}$ for which every $F_i$ is equivariant with respect to the actions of $S_n$.
% On the other hand, a $G$-invariant deep neural network is made up by stacking equivariant neural networks plus some standard single-layer neural networks $Z_i\colon X_i \to X_{i+1}$ for which every $Z_i$ is invariant with respect to the actions of $G$. We establish the following theorem: $G$-invariant/equivariant deep neural networks are universal approximators for permutation invariant/equivariant functions. %Our models are natural generalizations of the ones introduced by Zaheer et al. %\cite{deepsets}.
% Moreover, we show that the number of free parameters appearing in these models turns out to be exponentially fewer than the number of the ones in the usual models.
% By combining these results, we conclude that although the number of free parameters is much smaller than the one of the usual model, the invariant/equivariant models can approximate invariant/equivariant functions with arbitrary accuracy.
% %This justifies why our models are appropriate for invariant/equivariant problems.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In this paper, we develop a theory about the relationship between
$G$-invariant/equivariant functions and deep neural networks for finite group $G$.
% Our main contribution is to prove an invariant/equivariant version of the universal approximation theorem concerning what we call $G$-invariant/equivariant deep neural networks for a finite group $G$.
% A $G$-equivariant deep neural network consists of stacking standard single-layer neural networks $F_i\colon X_i \to X_{i+1}$ for which every $F_i$ is equivariant with respect to the actions of $S_n$.
% On the other hand, a $G$-invariant deep neural network is made up by stacking equivariant neural networks plus some standard single-layer neural networks $Z_i\colon X_i \to X_{i+1}$ for which every $Z_i$ is invariant with respect to the actions of $G$. We establish the following theorem:
Especially, for a given $G$-invariant/equivariant function, we construct
its universal approximator by deep neural network whose layers
equip $G$-actions and each affine transformations are $G$-equivariant/invariant.
% $G$-invariant/equivariant deep neural networks are universal approximators for permutation invariant/equivariant functions. %Our models are natural generalizations of the ones introduced by Zaheer et al. %\cite{deepsets}.
Due to representation theory, we can show that this approximator has exponentially fewer free parameters than usual models.
% Moreover, we show that the number of free parameters appearing in these models turns out to be exponentially fewer than the number of the ones in the usual models.
% By combining these results, we conclude that although the number of free parameters is much smaller than the one of the usual model, the invariant/equivariant models can approximate invariant/equivariant functions with arbitrary accuracy.
%This justifies why our models are appropriate for invariant/equivariant problems.
\end{abstract}
\section{Introduction}
Deep neural networks have great success in many applications such as image recognition, speech recognition, natural language process and others as \cite{krizhevsky2012imagenet}, \cite{goodfellow2013multi}, \cite{wan2013regularization},
and \cite{silver2017mastering}.
A common strategy in their works is to construct larger and deeper networks. %When we use larger and deeper networks, one of the main obstructions to learn is the so called {\it curse of dimensionality}, i.e. if the parameter, dimension of the models increase, then the required sample size and hence computational complexity for approximation increases exponentially. One idea to overcome the curse of dimensionality is to design models which fit to objectives.
However, one of the main obstructions about using very deep and large networks for learning tasks is the so-called {\it curse of dimensionality}. Namely, if the parameters' dimension increase, so does the required sample size. Then, the computational complexity becomes exponentially higher. An idea to overcome this
%{curse of dimensionality}
is to design models with respect to the objective structure.
\cite{deepsets} designed a model adapted to machine learning tasks defined on sets, which are, from a mathematical point of view, permutation invariant or equivariant tasks. They demonstrate surprisingly good applicability on their method on population statistic estimation, point cloud classification, set expansion, and outliers detection. Empirically speaking, their results are really significant. Many researchers studied invariant/equivariant networks as \cite{qi2017pointnet}, \cite{hartford2018deep}, \cite{kondor2018covariant}, \cite{maron2018invariant},
%\cite{maron2019universality},
\cite{bloem2019probabilistic}, \cite{pmlr-v80-kondor18a}, and so on.
%However, its theoretical guarantee is not considered sufficiently in their paper. Hence, because of this motivation, we generalize their models and give {\it invariant/equivariant versions} of the universal approximation theorem in this paper. Our theorem is not only for the classical {\it depth-bounded} version but also the {\it width-bounded} version.
Nevertheless, theoretical guarantee of their methods is not sufficiently considered. One of our motivations is to establish a theoretical guarantee.
In this paper, we prove an {\it invariant/equivariant version} of the universal approximation theorem by constructing an approximator.
% In our theorem, we take into consideration the {\it depth-bounded} version as well as the {\it width-bounded} version of the universal approximation theorem.
%Furthermore, we also calculate the number of free parameters appeared in our invariant/equivariant models. As a result, the number of free parameters appeared in our models is exponentially smaller than the one of the usual models.
%Hence, we conclude that although the free parameters of the invariant/equivarint models are exponentially fewer than the one of the usual models, the invariant/equivariant models can approximate the invariant/equivariant functions to arbitrary accuracy.https://www.overleaf.com/project/5c5aa8c7238b3326afbeb8be
%This gives us an understanding of why the invariant/equivariant models designed in \cite{deepsets} work well.
%To show this, we construct an approximator. %Unfortunately, this approximator is different from the model of \citet{deepsets}.
For the symmetric group, our approximator is close to the equivariant model of \citet{deepsets} in a sense (see a remark after Theorem \ref{theo:main-theorem}).
We can calculate the number of the free parameters appearing in our invariant/equivariant model, and show that this number is exponentially smaller than one of the usual models.
%Despite this, our universal approximation theorem implies that our invariant/equivariant models of networks are still able to approximate any invariant/equivariant functions defined on a compact set with arbitrary accuracy.
For usual deep neural networks, a {\it universal approximation theorem} was first proved by \cite{cyb}. It states that, when the width goes to infinity, a (usual) neural network with a single hidden layer can, with arbitrary accuracy, approximate any continuous function with compact support. Though his theorem was only for sigmoid activation functions, there are further versions of this theorem which allows some wider classes of activation functions. In the recent literature, the most commonly used activation function is the {\it ReLU (Rectified Linear Unit)} function, which is the one we focus on in this paper. Some important previous works
on universal approximation theorem for {\it ReLU} activation function are by
\cite{barron1994approximation},
\cite{hornik1989multilayer},
\cite{funahashi1989approximate},
\cite{kurkova2002comparison}, and
\cite{sonoda2017neural}.
In particular, for a part of the proof of our main theorem, we borrow the results
of \cite{sonoda2017neural} and \cite{approxrelu}.
% [Barron, 1994; Hornik et al., 1989; Funahashi, 1989; Kurkov$\acute{\mbox{a}}$].
The interest of the universal approximation theorem in learning theory is to guarantee that {\it we can search in the space which contains the solutions}. The universal approximation theorem states the existence of the model which approximates the target function in arbitrary accuracy. This means that if we use the suitable algorithm, we have the desired solutions. We cannot guarantee such situations without the universal approximation theorem.
Our universal approximation theorem allows us to apply
representation theory. By this point of view, we can calculate
the number of free parameters of our approximator.
%Hence, our invariant/equivariant version of the universal approximation theorem is important.
%However, the invariant models they construct have huge width in general and they asked whether there exist more efficient $G$-invariant models. As we see later, our universal approximation theorem gives an answer to their question in the case of $G=S_n$.
In the equivariant case, a technical key point of the proof is {\it the one to one correspondence} between $G$-equivariant functions and $\Stab_G(i)$-invariant functions. Here, $G$ is a finite group and $\Stab_G(i)$ is the subgroup of $G$ consisting of the elements which fix $i$. We first confirm this correspondence at the function level. After that, we rephrase it by deep neural networks. This correspondence enables us to reduce the equivariant case to the invariant case.
The invariant case has already established by some researchers \cite{deepsets},
\cite{yarotsky2018universal}, \cite{maron2019universality}.
For $G=S_n$, here $S_n$ is the symmetric group of degree $n$,
\cite{deepsets} showed that
{ a representation theorem} of $S_n$-invariant function
which is famous as a solution for the Hilbert's 13th problem by \cite{kolmogorovrepresentation} and \cite{arnoldrepresentation} %[Kolmogorov, 1961 Arnold, 1963].
gives us an explicit description.
Due to this theorem and the usual universal approximation theorem, we can construct a concrete deep neural network of the invariant model.
% Combining this with the reduction to the invariant case, we complete the proof of the equivariant case.
Recently, \cite{maron2019universality} proved an invariant version of the universal approximation theorem for any finite group $G$ using tensor structures.
We borrow their results to obtain our main results.
% % There are two important invariants in deep neural networks. the first one is the width and the second one is the depth.
% To calculate the number of the free parameters of our approximator, we need to consider a depth-bound or a weight-bound of our model.
% Considering these invariants, the universal approximation theorem of \cite{sonoda2017neural} can be viewed as a {\it depth-bounded} universal approximation theorem. A {\it width-bounded} universal approximation theorem is proved by \cite{approxrelu}, which states that even if the maximal width of deep neural networks is bounded, if the minimal width is bigger than the input dimension, then the universal approximation theorem holds. This kind of the universal approximation theorem is also important to question of neural network expressivity, i.e., how the architectural properties of a neural network (depth, width, layer type) affect the resulting functions it can compute, and its ensuing performance.
% % We provide both of invariant/equivariant versions of {\it depth-bounded} universal approximation theorem and {\it width-bounded} universal approximation theorem in this paper. \\[-20pt]
% We give both of the
% depth-bounded and width-bounded versions of the invariant/equivariant universal approximation theorem in this paper.
% %The {\it width-bounded} version of our invariant universal approximation theorem is an answer to the question asked by \cite{maron2019universality}\\[-20pt]
\subsection{Contributions}
\vspace{-10pt}
Our contributions are summarized as follows:
$\bullet$ We prove an invariant/equivariant version of the approximation theorems, which is a one step to understand the behavior of deep neural networks with permutations or more generally group actions.
%Furthermore, we give a depth/width bound of our approximators.
% both of the depth-bounded and width-bounded versions of the invariant/equivariant universal approximation theorem.
$\bullet$ Using representation theory, we calculate the number of free parameters appearing in our models. As a result, the number of parameter in our models is exponentially smaller than the one of the usual models. This means that our models are easier to train than the usual models.
$\bullet$ Although our model is slightly different from the equivariant model of \cite{deepsets} for $G=S_n$,
our theorem guarantees that our model for finite group $G$ can approximate any $G$-invariant/equivariant functions.
%$\bullet$ As an application of our results, these two results above give an understanding of the models designed in \cite{deepsets}, i.e., our approximation theorem and the calculation of the free parameters are evidences for the models in \cite{deepsets} to work well.
\subsection{Related works}
Group theory, or symmetry is an important concept in mathematics, physics, and machine learning. In machine learning, deep symmetry networks (symnets) is designed by %[Gans, et al., 2014]
\cite{Gens2014} as a generalization
of convnets that forms feature maps over arbitrary symmetry groups. Group equivariant Convolutional
Neural Networks (G-CNNs) is designed by %[Cohen, Weiling, 2016]
\cite{Cohen-Welling2016}, as a natural generalization of convolutional neural networks that reduces sample complexity by exploiting symmetries. The models for permutation invariant/equivariant tasks are designed by \cite{deepsets} to give great results on population statistic estimation, point cloud classification, set expansion, and outlier detection.
The universal approximation theorem is one of the most classical mathematical theorems of neural networks. %The universal approximation theorem states that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $\mathbb{R}^n$.
As we saw in the introduction, \cite{cyb} proved this theorem in 1989 for sigmoid activation functions. After his achievement, some researchers showed similar results to generalize the sigmoid function to a larger class of activation functions %[Barron, 1994; Hornik et al., 1989; Funahashi, 1989; Kurkov$\acute{\mbox{a}}$]
as
\cite{barron1994approximation}, \cite{hornik1989multilayer}, \cite{funahashi1989approximate}, \cite{kurkova1992} and \cite{sonoda2017neural}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% commented out (about depth efficiency, 2019/09/24)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% %Recently, with much success of deep learning, the interest in the expressive power increased.
% Depth efficiency is one of the main problems in expressive power. %Though feed-forward networks with a single hidden layer are universal approximators, these need exponentially many neurons for approximation.
% There exists a family of functions which can be represented much more efficiently with deep networks than with shallow ones as well %[Delalleau et al., 2011]
% \cite{delalleau2011}. There exists a 3-layer network, which cannot be realized by any 2-layer to more than a constant accuracy if the size grows subexponentially in the dimension %[Eldan et al., 2016]
% \cite{Ronen2016}. There exist some classes of deep convolutional ReLU networks that cannot be realized by shallow ones if the size is no more than an exponential bound %[Cohen et al,. 2016].
% \cite{Cohen2016}.
% A width-bounded version of the universal approximation theorem is proved by %[Lu et al., 2017]
% \cite{Z.Lu2017}. They showed that if the width is bigger than $n+3$, where $n$ is the input dimension, deep neural networks with ReLU activation functions are universal approximators. \cite{approxrelu} generalized their results, i.e., if the width is bigger than the input dimension, neural networks are universal approximators.
% Our approximation theorems rely on the results of \cite{sonoda2017neural} and \cite{approxrelu}. Thereby, although we only focus on $ReLU$ activation function, our conclusions still hold as long as those of \cite{sonoda2017neural} and \cite{approxrelu} do.
% %\todonum{Add the relation with our works}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As mentioned above, the invariant case has been established.
For $G=S_n$, \cite{deepsets} essentially proved an invariant version of an
universal approximation theorem.
\cite{yarotsky2018universal} gave a more explicit $S_n$-invariant approximator by a shallow deep neural network.
%which focused on the action of compact groups and provided network's architectures based on polynomial layer.
\cite{maron2019universality} considered a $G$-invariant model with some
tensor structures for any finite group $G$.
An equivariant version
%of universal approximation theorem
for finite group $G$ by
shallow (hyper-)graph neural networks is proved by \cite{keriven2019universal}.
Our architecture of approximator is different from theirs. Moreover,
although they proved only for ``squashing functions'' which exclude ReLU functions, our theorem allows us to use the ReLU functions. We also remark that
our setting in this paper is quite general. In particular, ours include tensor structures, hence graph neural networks.
It must be interesting to compare the numbers of free parameters of models of us and \cite{keriven2019universal}.
% Thereby, they proved \textit{universal approximation theorem of invariant networks} for compact and finite group actions. Our present work contains three main results. Despite than our result about \textit{universality of invariant networks} overlaps with the contributions of \cite{maron2019universality} and \cite{yarotsky2018universal}, these latter don't provide any bound to control the width of their network's architectures. In our present work, we give such bounds for width as well as for depth of our network's architecture.
% %\todonum{Add about papers pointed out}
\section{Preliminaries and main results}\label{sec:preliminaries}
In this paper, we treat fully connected deep neural networks.
We mainly consider ReLU activation functions. Here,
the ReLU activation function is defined by
\[
\mbox{ReLU}(x)=\mbox{max}(0,x). %=[x]_+.
\]
We remark that our argument during this paper works for any activation functions
which satisfy a usual universal approximation theorem.
% We remark that if the usual universal approximation theorem holds for some other activation functions,
% then our proof works for these functions.
%as [Barron, 1994; Hornik et al., 1989; Funahashi, 1989; Kurkov$\acute{\mbox{a}}$].
A deep neural network is built by stacking the blocks which consist of a linear map and a ReLU activation. More formally, it is a function $Z_i$ from $\mathbb{R}^{d_i}$ to $\mathbb{R}^{d_{i+1}}$ defined by $Z_i(\bm{x}) = \mbox{ReLU}(W_i\bm{x}+ \bm{b}_i)$, where $W_i \in \mathbb{R}^{d_{i+1}\times d_i}, \bm{b}_i \in \mathbb{R}^{d_i+1}$. In this case, $d_i$ is called the width of the $i$-th layer. The output of the deep neural networks is
\begin{align}
Y(\bm{x})= Z_H \circ Z_{H-1} \ldots Z_2 \circ Z_1(\bm{x}), \label{eq:DNN}
\end{align}
where $H$ is called the depth of the deep neural network.
We define the width of a deep neural network as the maximum of
the widths of all layers.
% Our models, which are generalization of the models in \cite{deepsets}, are defined by the invariant/equivariant property of $Z_i$.
% Before defining the invariant/equivariant models, we define
% the invariance/equivariance for functions.
%Only two groups will be considered in this paper.
Our main objects are deep neural networks which are invariant/equivariant
with actions by a finite group $G$.
We review some facts about groups and these actions here.
Some more details are written in Appendix \ref{sec:appendix-groups}.
Let $S_n$ be the group consisting of permutations of
$n$ elements $\{1, 2, \dots, n\}$.
This $S_n$ is called the symmetric group of degree $n$.
The symmetric group $S_n$ acts on
$\{1, 2, \dots, n \}$ by the permutation $i \mapsto \sigma^{-1}(i)$
for $\sigma \in S_n$.
By Proposition \ref{prop:embedding-to-symmetric-group} that
any finite group $G$ can be regarded as a subgroup of $S_n$
for a positive integer $n$.
% For a finite group $G$ which is a subgroup
% of $S_n$,
Then, $G$ also acts on $\{1, \dots, n \}$ by the action as
an element of $S_n$. For $i \in \{1, 2, \dots, n \}$, we define the
orbit of $i$ as $O_i = G(i) = \{ \sigma(i) \mid \sigma \in G\}$.
Then, the set $\{1,2, \dots, n \}$ can be divided to a disjoint union of
the orbits:
$\{1,2,\dots, n \} = \bigsqcup_{j=1}^m O_{i_j}$.
Let $G$ be a finite group action on $\{1,2,\dots,n\}$.
For $i =1,2,\dots, n$, we define the stabilizer subgroup $\Stab_G(i)$ of $G$ associated
with $x$ by the subgroup of elements of $G$ fixing $i$.
%\[
%\Stab_G(x) = \left\{\sigma\in G \mid \sigma\cdot x = x \right\}.
%\]
Then, by Proposition \ref{prop:orbit-stab-bijectivity},
the orbit $O_i$ and the set of cosets
$G/\Stab_G(i)$ are bijective.
When $G=S_n$, $X = \{1,2,\dots, n \}$, and $x = 1$,
we set $\Stab_n(1) = \Stab(1) := \Stab_G(x)$.
We next consider an action of $S_n$ on the vector space $\RR^n$.
% esspecially, each layers of deep neural networks.
%the natural permutation of nodes in the fixed layer.
The left action ``$\cdot$'' of $S_n$ on $\RR^n$ is defined by
\[
\sigma\cdot \bm{x} = \sigma\cdot (x_1, x_2, \dots, x_n)^\top
= (x_{\sigma^{-1}(1)}, x_{\sigma^{-1}(2)}, \dots, x_{\sigma^{-1}(n)})^\top
\]
for $\sigma \in S_n$ and $\bm{x} = (x_1, \dots, x_n)^\top \in \RR^n$.
We also call this the permutation action of $S_n$ on $\RR^n$.
If there is an injective group homomorphism
$\varphi\colon G \hookrightarrow S_n$,
$G$ acts on $\RR^n$ by the permutation action as an element of
$S_n$ by $\varphi$: For $\sigma \in G$ and $\bm{x}\in \RR^n$,
we define $ \sigma\cdot \bm{x} := \varphi(\sigma)\cdot \bm{x}$.
Then, we simply say that $G$ acts on $\RR^n$.
%When we say that $G$ acts on $\RR^n$, we give an injective
\begin{exmp}
The finite group $G = S_2$ is embedded into $S_3$ by
$\varphi\colon (1\ 2) \mapsto (1 \ 2)$,
where $(i \ j)$ is transposition between $i$ and $j$.
In this case, the orbit decomposition of $\{1,2,3 \}$ is
$\{ 1,2 \} \sqcup \{3\}$.
By this embedding $\varphi$, an $S_2$-action on $\RR^3$ is defined:
$\varphi((1 \ 2))\cdot(x_1, x_2, x_3)^\top = (x_2, x_1, x_3)^\top$.
\end{exmp}
\begin{exmp}[Tensors]\label{ex:tensor}
The group action of $G$ which is a subgroup of $S_n$ on tensors as in \cite{maron2019universality} is realized as follows:
An $S_n$-action on $\RR^{n^k\times a}$ is defined by the following
injective homomorphism
$\varphi\colon G \hookrightarrow S_{an^k}$:
We fix a bijection from $\{1, \dots, a n^k \}$ to
$\{ 1, \dots, n\}^k \times \{1,2,\dots, a \}$, and for $\sigma \in S_n$, $\varphi(\sigma) \in S_{an^k}$
is defined by
\[
\varphi(\sigma)\cdot (i_1, \dots, i_k, j) =
(\sigma^{-1}(i_1), \dots, \sigma^{-1}(i_k), j)
\]
for $(i_1, \dots, i_k) \in \{1, \dots, n \}^k$ and $j \in \{1,2,\dots, a \}$.
Then, for a tensor
$X = (x_{i_1,\dots,i_k, j})_{i_1, \dots, i_k = 1\dots, n, \ j = 1,2,\dots,a } \in \RR^{n^k\times a}$,
$\sigma \in S_n$ acts on $\RR^{n^k\times a}$ by
$\varphi(\sigma)\cdot X = (x_{\sigma^{-1}(i_1),\dots,\sigma^{-1}(i_k),j})$. This action is same as one of \cite{maron2019universality}.
\end{exmp}
\begin{exmp}[$n$-tuple of $D$-dimensional vectors]
We identify $\{1, 2, \dots, nD \}$ with
$\{ (i,j) \mid i=1,\dots, n, j= 1, \dots, D\}$ and
define $\varphi\colon S_n \hookrightarrow S_{nD}$ as
$\varphi(\sigma)\cdot(i,j) = (\sigma^{-1}(i), j)$.
Let $(\bm{x}_1, \dots, \bm{x}_n) \in (\RR^D)^n$ be an $n$-tuple of
$D$-dimensional vectors.
Then, for $\sigma \in S_n$,
$\phi(\sigma)\cdot(\bm{x}_1, \dots, \bm{x}_n) = (\bm{x}_{\sigma^{-1}(1)}, \dots, \bm{x}_{\sigma^{-1}(n)})$. This means a permutation of $n$ vectors.
\end{exmp}
\begin{defn}
Let $G$ be a finite group.
We assume that an injective homomorphisms $\varphi\colon G \hookrightarrow S_m$ are given. Then, $G$ acts on $\RR^m$.
%We assume that $G$ acts on $X$ (resp. $Y$) by $g\cdot x$ (resp. $g\ast y$) for $g\in G$ and $x \in X$ (resp. $y \in Y$) .
We say that a map $f\colon \RR^m \to \RR^n$ is {\em $G$-invariant} if $f(\varphi(\sigma) \cdot \bm{x})=f(\bm{x})$ for any $ \sigma \in G$ and any $\bm{x} \in \RR^m$. We also assume that $\psi\colon G \hookrightarrow S_n$.
Then, we say that a map $f\colon \RR^m \to \RR^n$
{\em $G$-equivariant} if $f(\varphi(\sigma)\cdot \bm{x})= \psi(\sigma)\cdot f(\bm{x})$ for any $ \sigma\in G$.
% \begin{itemize}
% \item {\em $G$-invariant} if $f(\varphi(\sigma) \cdot \bm{x})=f(\bm{x})$
% for any $ \sigma \in G$ and any $\bm{x} \in \RR^m$,
% \item {\em $G$-equivariant} if $f(\varphi(\sigma)\cdot \bm{x})= \psi(\sigma)\cdot f(\bm{x})$ for any $ \sigma\in G$
% and any $\bm{x} \in \RR^m$.
% \end{itemize}
% $\psi\colon G \hookrightarrow S_n$
When $G = S_n$ and the actions are induced by permutation,
we call $G$-invariant (resp. $G$-equivariant) functions as
{\em permutation invariant} (resp. {\em permutation equivariant}) functions.
\end{defn}
% \begin{defn}
% Let $S_n$ be the group of permutations of $n$ elements. Let $X$ and $Y$ be two sets. We assume that $S_n$ acts on $X$ (resp. $Y$) by $\sigma\cdot x$ (resp. $\sigma\ast y$) for $\sigma\in S_n$ and $x \in X$ (resp. $y \in Y$) .
% We say that a map $f\colon X \to Y$ is
% \begin{itemize}
% \item {\em $S_n$-invariant} if $f(\sigma\cdot x)=f(x)$
% for any $ \sigma \in S_n$ and any $x \in X$,
% \item {\em $S_n$-equivariant} if $f(\sigma\cdot x)=\sigma*f(x)$ for any $ \sigma \in S_n$
% and any $x \in X$.
% \end{itemize}
% \end{defn}
%Taking the inverse of $\sigma$ is the reason why we consider only the left action.
% Throughout this paper, we abbreviate the vector $\bm{x} = (x_1, \dots, x_n)^\top \in \RR^n$ as $\bm{x} = (x_i)_i$.
% Using this notation, we can rewrite the above induced action
% as $\sigma\cdot (x_i)_i = (x_{\sigma^{-1}\cdot i})_i$.
%\label{eq:natural-action}$
% Like this action, if the action is caused by the permutation of the index set, i.e.,
% $ \sigma\cdot (x_i)_i = (x_{\sigma \cdot i})_i \label{eq:natural-action}$ holds , we say that $\sigma$ is the $S_n$-action.
We define $G$-invariance and $G$-equivariance for deep neural networks.
We can easily confirm that the models in \cite{deepsets} satisfies these properties.
\begin{defn}
We say that a deep neural network
$Z_H \circ Z_{H-1} \ldots Z_2 \circ Z_1$ as (\ref{eq:DNN}) is
$G$-{\it equivariant}
%(resp. {\it $S_n$-preinvariant})
if an action of $G$ on each of layers $\mathbb{R}^{d_i}$ is given by embedding $G\hookrightarrow S_{d_i}$ and the all corresponding map $Z_i\colon \mathbb{R}^{d_i} \to \mathbb{R}^{d_{i+1}}$ is $G$-equivariant.
%(resp. $S_n$-invariant).
We say that a deep neural network is $G$-{\it invariant}
if there is a positive integer $c\leq H$ such that $G$-actions on each layer $\mathbb{R}^{d_i}$ for $1 \leq i \leq c+1$ are given and the corresponding map $Z_i\colon \mathbb{R}^{d_i} \to \mathbb{R}^{d_{i+1}}$ is $G$-equivariant for $1 \leq i \leq c-1$ and the map $Z_{c}:\RR^{d_{c}} \to \RR^{d_{c+1}}$ is $G$-invariant.
%and $Z_{i}$ is a perceptron for $c+2 \leq i \leq H$.
% Let $G$ be the group and $X$ and $Y$ two sets.
% We assume that $G$ acts on $X$ (resp. $Y$) by $g\cdot x$ (resp. $g\ast y$) for $g\in G$ and $x \in X$ (resp. $y \in Y$) .
% We say that a map $f\colon X \to Y$ is
% \begin{itemize}
% \item {\em $G$-invariant} if $f(g\cdot x)=f(x)$
% for any $ g \in G$ and any $x \in X$,
% \item {\em $G$-equivariant} if $f(g\cdot x)= g*f(x)$ for any $ g\in G$
% and any $x \in X$.
% \end{itemize}
\end{defn}
%In this paper, we give three main theorems about invariant/equivariant networks. The first one is the invariant version of universal approximation theorem.
Some approximation theorems for invariant functions have been
already known:
\begin{prop}[$G$-invariant version of universal approximation theorem]\label{theo:inv-theorem}
Let %\todo{$K$ should be $G$-stable? Is it too nervous?}
$G$ be a finite group which is a subgroup of $S_n$.
Let $K$ be a compact set in $\mathbb{R}^n$ which is stable for the corresponding $G$-action in $\mathbb{R}^n$. Then, for any $f\colon K\rightarrow \mathbb{R}^m$ which is continuous and $G$-invariant and for any $\epsilon>0$, the following $G$-invariant ReLU neural networks
$\mathcal{N}^{\inv}_G\colon \RR^n \to \RR^m$ satisfy that
these represented functions $R_{\mathcal{N}^\inv_G}$ satisfy
$\|f-R_{\mathcal{N}^\inv_G}\|_\infty < \epsilon$:
\begin{itemize}
\item $\mathcal{N}^\inv_G = \Sigma \circ L_H\circ \ReLU \circ \cdots \circ L_1$, where $\Sigma$ is the summation ($G$-invariant part) and
$L_i\colon\RR^{{n^{d_i}}\times a_i}\to \RR^{n^{d_{i+1}}\times a_{i+1}}$
is a linear map such that $L_i(g\cdot X) = g\cdot L_i(X)$ for any
$g\in G$ and any $X \in \RR^{{n^{d_i}}\times a_i}$. Here, the actions on each layers except for the output layer are same as Example \ref{ex:tensor}.
\item For $G = S_n$.
$\mathcal{N}^\inv_G = \mathcal{N}_\rho \circ
\Sigma \circ (\mathcal{N}_\phi, \dots, \mathcal{N}_\phi)^\top$, where
$\mathcal{N}_\rho$ (resp. $\mathcal{N}_\phi$) is a deep neural network approximating $\rho$ (resp. $\phi$) defined below.
\end{itemize}
% Furthermore, we can take $\mathcal{N}$ as either of the following:
% $\bullet$ $\mathcal{N}$ has two hidden layers and the width is not bounded, or
% $\bullet$ The width is of $\mathcal{N}$ is bounded above by $n(n+2)$ and the depth is not bounded.
\end{prop}
\begin{figure}[t]
\begin{center}
\scalebox{1}[1]{
\begin{tikzpicture}
\node[functions] (rho) {$\rho$};
\node[output,right=2em of rho] (output) {};
\path[draw,->] (rho) -- (output);
\node[node,left=1.5em of rho] (a) {}
node[sum,left=1.5em of a] (sum) {$\sum$};
\path[draw,->] (sum) -- (a);
\path[draw,->] (a) -- (rho);
\node[left=1.5em of sum] (2) {$\vdots$} -- (2) node[left=3em of 2] (l2) {$\vdots$} node[left=3em of l2] (ll2) {$\vdots$}; %node[below of=l2] (ldots) {$\vdots$} node[below of=ll2] (lldots) {$\vdots$} ;
%\node[node,left=3em of sum] (2) {} -- (2) node[functions,left=2em of 2] (l2) {$\phi$} node[input,left=2em of l2] (ll2) {$x_2$} node[below of=l2] (ldots) {$\vdots$} node[below of=ll2] (lldots) {$\vdots$} ;
% \path[draw,->] (ll2)--(l2);
% \path[draw,->] (l2) -- (2);
%\path[draw,->] (2) -- (sum);
% \node[below of=2] (dots) {$\vdots$} ;
\node[node,below of=2] (n) {} -- (n) node[functions,left=2em of n] (ln) {$\phi$} node[input,left=2em of ln] (lln) {$x_n$} ;
\path[draw,->] (lln)--(ln);
\path[draw,->] (ln) -- (n);
\path[draw,->] (n) -- (sum);
\node[node,above =1em of 2] (1) {} -- (1) node[functions,left =2em of 1] (l1) {$\phi$}node[input,left=2em of l1] (ll1) {$x_2$} ;
\path[draw,->] (ll1)--(l1);
\path[draw,->] (l1) -- (1);
\path[draw,->] (1) -- (sum);
\node[node,above =1em of 1] (0) {} -- (0) node[functions,left=2em of 0] (l0) {$\phi$}node[input,left=2em of l0] (ll0) {$x_1$} ;
\path[draw,->] (ll0)--(l0);
\path[draw,->] (l0) -- (0);
\path[draw,->] (0) -- (sum);
%\node[above of=2 ,font=\scriptsize] {inputs};
%\node[below of=n,font=\scriptsize] {weights};
\end{tikzpicture}
}
\end{center}
\caption{A neural network approximating
$S_n$-invariant function $f$.
In blue: the inputs, in red: the output, in green: $\rho$ and $\phi$ who have to be learned.}
\label{fig:invariant}
\end{figure}
This proposition for $G = S_n$ is proven by \cite{deepsets}, \cite{yarotsky2018universal}. For general finite group $G$, \cite{maron2019universality} proved it.
Diagram \ref{fig:invariant} illustrates the $S_n$-invariant ReLU neural network appeared in Proposition \ref{theo:inv-theorem}.
%In Maron's paper, they
%\cite{maron2019universality} also showed this theorem.
%However, we give the bound of the width and the depth.
The key ingredient of the proof by \cite{deepsets} is the following
Kolmogorov-Arnold representation theorem:
\begin{theo}[\cite{deepsets} Kolmogorov-Arnold's representation theorem for permutation actions]\label{thm:representation-of-invariant}
Let $K\subset\mathbb{R}^n$ be a compact set.
Then, any continuous $S_n$-invariant function $f\colon K\to\mathbb{R}$ can be represented as \vspace{-12pt}
\begin{align}
f(x_1,\dots,x_n)=\rho\left(\sum_{i=1}^{n}\phi(x_i)\right) \label{eq:representation}
\end{align}
\vspace{-4pt}
for some continuous function $\rho\colon\mathbb{R}^{n+1}\rightarrow\mathbb{R}$.
Here, $\phi\colon \mathbb{R} \to \mathbb{R}^{n+1}; x \mapsto
(1, x, x^2, \dots, x^n)^\top$.
\end{theo}
%to give a presentation $f=\rho\left(\sum_{i=1}^{n}\phi(x_i)\right)$, where $f$ is the objective function.
Since $\phi (x)$ has only one variable, we line up the copies of the network which approximates $\phi(x)$. Then, by combining $\Sigma$ and the network which approximates $\rho$, we obtain the network which approximates $f$. By using the theorem of \cite{approxrelu} (resp. \cite{sonoda2017neural}), we obtain the bound of the width (resp. the depth) for approximation of $\phi$ and $\rho$. \cite{maron2019universality} proved this proposition using a tensor structure.
The main theorem is a $G$-equivariant version of universal approximation
theorem. To state the main theorem, we need some notation.
Let $G$ be a finite group acting on $\RR^n$. We set the orbit decomposition of $\{1, 2, \dots, n \}$ as $\{1, 2, \dots, n \} = \bigsqcup_{j = 1}^m O_{i_j}$,
and let $O_{i_j} = \{ i_{j1}=i_j, i_{j2}, \dots, i_{jl_j} \}$. Without loss of generality, we may reorder $\{1, 2, \dots, n \}$ as
\[
O_{i_1} = \{1, 2, \dots, l_1 \}, O_{i_2} = \{ l_1 + 1, l_1 +2, \dots l_1 + l_2 \}, \dots
\]
and $i_j = \sum_{k = 1}^{j-1} l_k + 1$ for $j = 1,2, \dots, m$.
For each $j$, let
$
G = \bigsqcup_{k=1}^{l_j}\Stab_G(i_j) \tau_{j,k}
$
be the coset decomposition by $\Stab_G(i_j)$> Then, by ... we may assume that
$\tau_{j,k}\in G$ satisfies $\tau_{j,k}^{-1}(i_j) = i_j + k$ for
$k=1,2,\dots, l_j$. Then, the main theorem is the following:
\begin{theo}[$G$-equivariant version of universal approximation theorem]\label{theo:main-theorem}
Let %\todo{$K$ should be $G$-stable? Is it too nervous?}
$G$ be a finite group which is a subgroup of $S_n$.
Let $K$ be a compact set in $\mathbb{R}^n$ which is stable for the corresponding $G$-action in $\mathbb{R}^n$. Then, for any
$F\colon K\rightarrow \mathbb{R}^n; \bm{x} \mapsto (f_1, \dots, f_n)$
which is continuous and $G$-equivariant
and for any $\epsilon>0$, the following $G$-invariant ReLU neural network
$\mathcal{N}^{\equi}_G\colon \RR^n \to \RR^n$ satisfies that
these represented functions $R_{\mathcal{N}^\equi_G}$ satisfy
$\|F-R_{\mathcal{N}^\equi_G}\|_\infty < \epsilon$:
\[
\mathcal{N}^\equi_G = (\mathcal{N}^\inv_{\Stab_G(i_1)}\circ \tau_{1,1}, \
\mathcal{N}^\inv_{\Stab_G(i_1)}\circ \tau_{1,2}, \ \dots, \
\mathcal{N}^\inv_{\Stab_G(i_m)}\circ \tau_{m,l_m})^\top.
\]
Here, $\mathcal{N}^\inv_{\Stab_G(i_j)}$ is the $\Stab_G(i_j)$-invariant deep neural network approximating $f_{i_j}$
as in Proposition \ref{theo:inv-theorem}, and the actions on each layers are defined as follows: Each of hidden layers are written by $\RR^{n^2}\otimes_\RR V$ for a vector space $V$. On this space, $\sigma \in G$ acts on
$(\bm{x}_1, \dots, \bm{x}_n)\otimes v \in \RR^{n^2}\otimes_\RR V$ ($\bm{x}_i\in \RR^n$) by
\[
\sigma\cdot ((\bm{x}_1, \dots, \bm{x}_n)\otimes v) =
(\wt{\sigma}_{1,1}\cdot \bm{x}_{\sigma^{-1}(1)}, \dots, \wt{\sigma}_{m,l_m}\cdot\bm{x}_{\sigma^{-1}(n)})\otimes v,
\]
where $\wt{\sigma}_{j,k}$ is the element of $\Stab_G(i_j)$ satisfying
$\wt{\sigma}_{j,k} = \tau_{j,k'}\sigma \tau_{j,k}^{-1}$ for some $k'= 1,2,\dots, l_j$.
%$\bullet$ $\mathcal{N}$ has two hidden layers and the width is not bounded, or
%
%$\bullet$ The width is of $\mathcal{N}$ is bounded above by $n^3$ and the depth is not bounded.
\end{theo}
%\begin{textblock*}{0.4\linewidth}(300pt,180pt)
\begin{figure}[t]
% \centering
\begin{center}
\scalebox{1}[1]{
\begin{tikzpicture}%[baseline=20cm]
\tikzset{basic/.style={draw,fill=blue!80,text width=0.75em,text badly centered}}
\node[functions] (rho) {$\rho$};
\node[output,right=1em of rho] (output) {};
\path[draw,->] (rho) -- (output);
\node[node,left=1em of rho] (node1) {};
\path[draw,->] (node1)--(rho);
\node[sum,left=1em of node1] (sum) {$\!\sum$};
\path[draw,->] (sum) -- (node1);
\node[left=1em of sum] (2) {$\vdots$} -- (2) node[left=1.5em of 2] (l2) {$\vdots$} node[left=1.5em of l2] (ll2) {$\vdots$} node[above=1em of ll2] (a) {} node[below=1em of ll2] (b) {} node[above=1em of 2] (a') {} node[below=1em of 2] (b') {};
%\path[draw,->] (a)--(a');
% \path[draw,->] (b)--(b');
%\path[draw,->] (ll2)--(2);
% \path[draw,->] (a')--(sum);
% \path[draw,->] (b')--(sum);
%\path[draw,->] (2)--(sum);
\node[node,below=1.5em of 2] (n) {} -- (n) node[functions,left=1em of n] (ln) {$\phi$} node[node,left=1em of ln] (lln) {} node[node, left=1em of lln] (llln) {};
\path[draw,->] (llln)--(lln);
\path[draw,->] (lln)--(ln);
\path[draw,->] (ln) -- (n);
\path[draw,->] (n) -- (sum);
\node[node,above =1em of 2] (1) {} -- (1) node[functions,left =1em of 1] (l1) {$\phi$}node[node,left=1em of l1] (ll1) {} node[node, left=1em of ll1] (lll1) {};
\path[draw,->] (lll1)--(ll1);
\path[draw,->] (ll1)--(l1);
\path[draw,->] (l1) -- (1);
\path[draw,->] (1) -- (sum);
\node[node,above =1em of 1] (0) {} -- (0) node[sum,left=1em of 0] (l0) {$\mathrm{id}$}node[node,left=1em of l0] (ll0) {} node[node, left=1em of ll0] (lll0) {};
\path[draw,->] (lll0)--(ll0);
\path[draw,->] (ll0)--(l0);
\path[draw,->] (l0) -- (0);
\node[above=1.5em of llln] (f) {$\vdots$};
\path[draw,->] (f)--(ll2);
\node[sum, above=3em of sum] (id) {$\mathrm{id}$};
\node[node, below right=1em of id] (a) {};
\path[draw,->] (0)--(id);
\path[draw,->] (id)--(a);
\path[draw,->] (a)--(rho);
\node[below=1.5em of ln] (dot) {$\vdots$};
%\node[functions, below=14em of rho] (rho2) {$\rho$};
% \node[output,right=1.5em of rho2] (output2) {};
% \path[draw,->] (rho2) -- (output2);
% \node[sum,left=1.5em of rho2] (sum2) {$\sum$};
% \path[draw,->] (sum2) -- (rho2);
% \node[left=1em of sum2] (22) {$\vdots$} -- (22) node[left=2em of 22] (l22) {$\vdots$} node[left=2em of l22] (ll22) {$\vdots$} node[above=1em of ll22] (a2) {} node[below=1em of ll22] (b2) {} node[above=1em of 22] (a'2) {} node[below=1em of 22] (b'2) {};
%\path[draw,->] (a2)--(a'2);
%\path[draw,->] (b2)--(b'2);
%\path[draw,->] (ll22)--(22);
%\path[draw,->] (a'2)--(sum2);
%\path[draw,->] (b'2)--(sum2);
%\path[draw,->] (22)--(sum2);
% \node[node,below=1.5em of 22] (n2) {} -- (n2) node[functions,left=1.5em of n2] (ln2) {$\phi$} node[node,left=1.5em of ln2] (lln2) {} node[node, left=1.5em of lln2] (llln2) {};
% \path[draw,->] (llln2)--(lln2);
% \path[draw,->] (lln2)--(ln2);
% \path[draw,->] (ln2) -- (n2);
% \path[draw,->] (n2) -- (sum2);
% \node[node,above =1.5em of 22] (12) {} -- (12) node[functions,left =1.5em of 12] (l12) {$\phi$}node[node,left=1.5em of l12] (ll12) {} node[node, left=1.5em of ll12] (lll12) {};
% \path[draw,->] (ll12)--(l12);
% \path[draw,->] (l12) -- (12);
% \path[draw,->] (12) -- (sum2);
%\node[node,above =1.5em of 12] (02) {} -- (02) node[sum,left=1.5em of 02] (l02) {$id$}node[node,left=1.5em of l02] (ll02) {} node[node, left=1.5em of ll02] (lll02) {};
% \path[draw,->] (lll02)--(ll12);
% \path[draw,->] (lll12)--(ll02);
%
% \path[draw,->] (ll02)--(l02);
% \path[draw,->] (l02) -- (02);
%\node[above=1.5em of llln2] (e) {$\vdots$};
% \path[draw,->] (e)--(ll22);
%\node[sum, above=3em of sum2] (id2) {$id$};
%\path[draw,->] (02)--(id2);
%\path[draw,->] (id2)--(rho2);
% \node[above of=ln,font=\scriptsize] {inputs};
%\node[below of=n,font=\scriptsize] {weights};
\node[functions, below=15em of rho] (rhon) {$\rho$};
\node[output,right=1em of rhon] (outputn) {};
\path[draw,->] (rhon) -- (outputn);
\node[node,left=1em of rhon] (noden) {};
\path[draw,->] (noden)--(rhon);
\node[sum,left=1em of noden] (sumn) {$\!\sum$};
\path[draw,->] (sumn) -- (noden);
\node[left=1em of sumn] (2n) {$\vdots$} -- (2n) node[left=1.5em of 2n] (l2n) {$\vdots$} node[left=1.5em of l2n] (ll2n) {$\vdots$} node[above=1em of ll2n] (an) {} node[below=1em of ll2n] (bn) {} node[above=1em of 2n] (a'n) {} node[below=1em of 2n] (b'n) {};
%\path[draw,->] (an)--(a'n);
%\path[draw,->] (bn)--(b'n);
%\path[draw,->] (ll2n)--(2n);
%\path[draw,->] (a'n)--(sumn);
%\path[draw,->] (b'n)--(sumn);
%\path[draw,->] (2n)--(sumn);
\node[node,below=1.5em of 2n] (nn) {} -- (nn) node[functions,left=1em of nn] (lnn) {$\phi$} node[node,left=1em of lnn] (llnn) {} node[node, left=1em of llnn] (lllnn) {};
\path[draw,->] (llnn)--(lnn);
\path[draw,->] (lnn) -- (nn);
\path[draw,->] (nn) -- (sumn);
\node[node,above =1.5em of 2n] (1n) {} -- (1n) node[functions,left =1em of 1n] (l1n) {$\phi$}node[node,left=1em of l1n] (ll1n) {} node[node, left=1em of ll1n] (lll1n) {};
\path[draw,->] (lll1n)--(ll1n);
\path[draw,->] (ll1n)--(l1n);
\path[draw,->] (l1n) -- (1n);
\path[draw,->] (1n) -- (sumn);
\node[node,above =1em of 1n] (0n) {} -- (0n) node[sum,left=1em of 0n] (l0n) {$\mathrm{id}$}node[node,left=1em of l0n] (ll0n) {} node[node, left=1em of ll0n] (lll0n) {};
\path[draw,->] (lllnn)--(ll0n);
\path[draw,->] (lll0n)--(llnn);
\path[draw,->] (ll0n)--(l0n);
\path[draw,->] (l0n) -- (0n);
\node[above=1.5em of lllnn] (d) {$\vdots$};
\path[draw,->] (d)--(ll2n);
\node[sum, above=3em of sumn] (idn) {$\mathrm{id}$};
\node[node, below right=1em of idn] (an) {};
\path[draw,->] (0n)--(idn);
\path[draw,->] (idn)--(an);
\path[draw,->] (an)--(rhon);
\node[below=1.7 of rho] (c) {$\vdots$};
\node[input, below left =3em of lll1] (x1) {$x_1$} node[input,below=8em of x1] (xn) {$x_n$} node[below=3.7em of x1] (g) {$\vdots$} ;
\path[draw,->] (x1)--(lll0);
% \path[draw,->] (x1)--(lll02);
\path[draw,->] (x1)--(lll0n);
\path[draw,->] (xn)--(llln);
%\path[draw,->] (xn)--(llln2);
\path[draw,->] (xn)--(lllnn);
%\node[below of= lllnn , font=\scriptsize] {$\bigoplus_{i=1}^{n}\mathbb{R}^{n}$};
%\node[below =22.5em of xn , font=\scriptsize] {$\mathbb{R}^{n}$};
%\node[below of= llnn , font=\scriptsize] (h) {$\bigoplus_{i=1}^{n}\mathbb{R}^{n}$ };
%\node[below of= h , font=\scriptsize] {and the layer just after, some space $\bigoplus_{i=1}^{n}V$ };
\end{tikzpicture}
}
\end{center}
%\end{textblock*}
\caption{A neural network approximating $S_n$-equivariant map $F$}
\label{fig:Total-DNN}
\end{figure}
For $G=S_n$, when $\mathcal{N}^\equi_G$ is
represented by $Z_H\circ Z_{H-1} \circ \cdots \circ Z_1$ as in (\ref{eq:DNN}), we can consider that $Z_2, \dots, Z_H$ are
$S_n$-equivariant by usual permutation. In this sense, our model is
close to the $S_n$-equivariant model in \cite{deepsets}.
%Although invariant versions are proved by \cite{maron2019universality} and \cite{yarotsky2018universal}, the equivariant version of universal approximation theorem for $S_n$ is first in the literature.
Our strategy for the proof is the following: At first, we establish the correspondence between $\Stab_G(i_j)$-invariant functions and $G$-equivariant functions. By this correspondence, we take $\Stab_G(i_j)$-invariant function $f$ corresponding to the objective function $F$. By Proposition \ref{theo:inv-theorem}, we can approximate $f$ by a $\Stab_G(i_j)$-invariant network $\mathcal{N}^\inv_{\Stab_G(i_j)}$. Using
$\mathcal{N}^\inv_{\Stab_G(i_j)}$, we construct the $G$-equivariant network which approximates $F$. Diagram \ref{fig:Total-DNN} illustrates the $S_n$-equivariant ReLU neural network appeared in Theorem~\ref{theo:main-theorem}.
%\todo{Revise this paragraph}
Due to our universal approximation theorems, if the free parameters of the invariant/equivariant models are fewer than the ones of the usual models, we have a guarantee for using the invariant/equivariant models. The following definition illustrates the swapping of nodes.
\begin{defn}
Let $X=\{ 1, \dots, M\}$ be an index set of nodes in a layer. We say an $S_n$-action on $X$ is a union of permutations if $X = \bigsqcup X_i$, where each $X_i$ has $n$ elements and $S_n$ acts on $X_i$ by permutation.%$S_n$ acts on each $X_i$ and the action is the permutation of $S_n$.
%(Hence, $X_i$ has $n$ points.)
\end{defn}
\begin{theo}\label{thm:number-of-params}
Let $\mathcal{N}$ be an $S_n$-invariant model of depth $D$ and width $M$ whose number of the equivariant layers is $d$ (resp. an $S_n$-equivariant model of depth $D$ and width $M$). Assume that the action is a union of permutations on nodes in each equivariant layer.
Then, the number of free parameters in this model is bounded by $M^{2D}\cdot (2/n^{2})^d$ (resp. $M^{2D}\cdot (2/n^2)^D$).
\end{theo}
Note that the number of free parameters in the usual model is $M^{2D}$. Hence, this theorem implies that {\it the free parameters of the invariant/equivariant models are exponentially fewer than the ones of the usual models}.
\label{sec:headings}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% comment out (section:invarinat case 2019/09/24)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \section{Invariant case}\label{sec:invariant-case}
% In this section, we discuss the invariant case.
% An important result about the structures of invariant functions is that these have exact representations as follows:
% %Indeed, \cite{deepsets} have shown the following theorem:
% \begin{theo}[\cite{deepsets} Kolmogorov-Arnold's representation theorem for permutation actions]\label{thm:representation-of-invariant}
% Let $K\subset\mathbb{R}^n$ be a compact set.
% Then, any continuous $S_n$-invariant function $f\colon K\to\mathbb{R}$ can be represented as \vspace{-12pt}
% \begin{align}
% f(x_1,\dots,x_n)=\rho\left(\sum_{i=1}^{n}\phi(x_i)\right) \label{eq:representation}
% \end{align}
% \vspace{-4pt}
% for some continuous function $\rho\colon\mathbb{R}^{n+1}\rightarrow\mathbb{R}$.
% Here, $\phi\colon \mathbb{R} \to \mathbb{R}^{n+1}; x \mapsto
% (1, x, x^2, \dots, x^n)^\top$.
% \end{theo}
% % %An important points on this theorem is
% % We should remark that $\phi$ is independent of $f$.
% % This representation \eqref{eq:representation}
% % divides $f$ to separated two parts $\rho$ and $\phi$.
% % %looks like a feed-forward network with two hidden layers.
% % The strategy of \cite{deepsets} for learning invariant functions is to replace $\rho$ and $\phi$ by universal approximators and to learn each of these approximators.
% % Diagram \ref{fig:invariant} illustrates this idea.
% Theorem \ref{thm:representation-of-invariant} implies
% Proposition \ref{theo:inv-theorem}, i.e., an invariant version of universal approximation theorem.
% We give a proof of Proposition \ref{theo:inv-theorem} to Appendix \ref{sec:appendix-inv-theorem}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\begin{proof}[Proof of Proposition \ref{theo:inv-theorem}]
%%We first reduce to the case of $N=1$.
%We may assume $N=1$. In fact,
%since we consider the $L^{\infty}$-norm, if all components of $\|f-R_\mathcal{N}\|$ is bounded by $\epsilon$, then $\|f-R_\mathcal{N}\|_\infty \leq \epsilon$ holds. By Theorem 3.1, we have $f(x_1,\dots,x_n)=\rho\left(\sum_{i=1}^{n}\phi(x_i)\right) $. Then, $\phi$ and $\Sigma$ are explicit maps. Hence, we can %\todonum{Too rough?}
%write them down by a feed forward network. We give $S_n$-actions on this network.
%%We give the natural $S_n$-action on the input layer.
%The $S_n$-action on the input layer is permutation.
%Then, since $\Sigma$ is apparently an invariant linear
%function, it remains to approximate $\phi$ and $\rho$. %\todo{?}
%Since $\phi$ depends on a single variable, we can extend $S_n$-action on arbitrary approximations. By \cite{sonoda2017neural}, we can approximate $\phi$ and $\rho$ by shallow networks. Hence, we have deep neural network
%$\mathcal{N}$ which has two hidden layers and the width is not bounded.
%By \cite{approxrelu}, we can respectively approximate each of $\phi$ and $\rho$ by some neural networks whose width are bounded by $n+2$ and the depth is not bounded. Hence, we have deep neural network $\mathcal{N}$ whose width is bounded by $n(n+2)$ and the depth is not bounded. The detail of the proof is in Appendix \ref{subsec:inv-theorem}.
%\end{proof} %\todonum{Revise proof more readable.}
% \begin{figure}[t]
% \begin{center}
% \begin{tikzpicture}
% \node[functions] (rho) {$\rho$};
% \node[output,right=2em of rho] (output) {};
% \path[draw,->] (rho) -- (output);
% \node[node,left=1.5em of rho] (a) {}
% node[sum,left=1.5em of a] (sum) {$\sum$};
% \path[draw,->] (sum) -- (a);
% \path[draw,->] (a) -- (rho);
% \node[left=1.5em of sum] (2) {$\vdots$} -- (2) node[left=3em of 2] (l2) {$\vdots$} node[left=3em of l2] (ll2) {$\vdots$} node[below of=l2] (ldots) {$\vdots$} node[below of=ll2] (lldots) {$\vdots$} ;
% %\node[node,left=3em of sum] (2) {} -- (2) node[functions,left=2em of 2] (l2) {$\phi$} node[input,left=2em of l2] (ll2) {$x_2$} node[below of=l2] (ldots) {$\vdots$} node[below of=ll2] (lldots) {$\vdots$} ;
% % \path[draw,->] (ll2)--(l2);
% % \path[draw,->] (l2) -- (2);
% %\path[draw,->] (2) -- (sum);
% \node[below of=2] (dots) {$\vdots$} ;
% \node[node,below of=dots] (n) {} -- (n) node[functions,left=2em of n] (ln) {$\phi$} node[input,left=2em of ln] (lln) {$x_n$} ;
% \path[draw,->] (lln)--(ln);
% \path[draw,->] (ln) -- (n);
% \path[draw,->] (n) -- (sum);
% \node[node,above =1em of 2] (1) {} -- (1) node[functions,left =2em of 1] (l1) {$\phi$}node[input,left=2em of l1] (ll1) {$x_2$} ;
% \path[draw,->] (ll1)--(l1);
% \path[draw,->] (l1) -- (1);
% \path[draw,->] (1) -- (sum);
% \node[node,above =1em of 1] (0) {} -- (0) node[functions,left=2em of 0] (l0) {$\phi$}node[input,left=2em of l0] (ll0) {$x_1$} ;
% \path[draw,->] (ll0)--(l0);
% \path[draw,->] (l0) -- (0);
% \path[draw,->] (0) -- (sum);
% %\node[above of=2 ,font=\scriptsize] {inputs};
% %\node[below of=n,font=\scriptsize] {weights};
% \end{tikzpicture}
% \end{center}
% \caption{A neural network approximating
% $S_n$-invariant function $f$.
% In blue : the inputs, in red : the output, in green : $\rho$ and $\phi$ who have to be learned.}
% \label{fig:invariant}
% \end{figure}
% \begin{theo}[Permutation invariant version of universal approximation theorem]\label{theo:inv-theorem}
% Let $K$ be a compact set in $\mathbb{R}^n$. Then, for any $f:K\rightarrow \mathbb{R}^N$ which is continuous and $S_n$-invariant and for any $\epsilon>0$, there is an $S_n$-invariant ReLU neural network $\mathcal{N}$ such that its represented function $R_\mathcal{N}$ satisfies $\|f-R_\mathcal{N}\|_\infty \leq \epsilon$. Furthermore, we can take $\mathcal{N}$ as either of the following:
% $\bullet$ $\mathcal{N}$ has two hidden layers and the width is not bounded.
% or
% $\bullet$ The width is of $\mathcal{N}$ is bounded above by $n(n+2)$ and the depth is not bounded.
% \end{theo}
%\todonum{Add a toy example here? (said by 4th reviewer.)}
% \subsection{equivariant case}
% In the case of equivariant function, we do not have representation theorem. Then, we introduce the object that we study in this paper, namely equivariant neural networks.
% \section{Relation between invariant and equivariant functions}\label{sec:relation-inv-equ}
% Here, we present a relation between $S_n$-equivariant function from $\RR^n$
% to $\RR^n$ and invariant function from $\RR^n$ to $\RR$ by the action of the stabilizer subgroup $\Stab_n(1)$ of $S_n$ with respect to $\{ 1 \}$.
% Using this relation, we can reduce an approximation problem of $S_n$-equivariant
% functions to one of $\Stab_n(1)$-invariant functions. In principle, this relation comes
% from the coset decomposition of $S_n$ by $\Stab_n(1)$. We give a precise description of this relation below.
% For the details about the permutation group $S_n$, see \cite[Section~4.3]{kurzweil2006theory}.
% During this section, as an action of $S_n$,
% we only consider the action on $\RR^n$ induced from permutation $\sigma\cdot (x_i)_i = (x_{\sigma^{-1}(i)})_i$.
% % The stabilizer subgroup of $S_n$ with
% % respect to $\{1 \}$ is defined by
% % \[
% % \Stab_n(1) =\left\{\sigma\in S_n \mid \sigma\cdot 1 =1\right\}.
% % \]
% % When there is no ambiguity, we denote it by $\Stab(1)$.
% % Then, for any $\Stab(1)$-invariant function $f:\RR^n \to \RR$,
% % $F= (f, f\circ(1 \ 2), \dots, f\circ (1 \ n))^\top : \RR^n \to
% % \RR^n$ is $S_n$-equivariant.
% We define the action of $\Stab(1)$ on $\RR^n$
% by regarding $\Stab(1)$ as a subgroup of $S_n$.
% Then, the following shows a relation between $S_n$-equivariance
% and $\Stab(1)$-invariance.
% \begin{prop}\label{prop:relation-invariant-equivariant}
% A map $F\colon\RR^n \to \RR^n$ is $S_n$-equivariant if and only if there is a $\Stab(1)$-invariant function
% $f\colon\RR^n \to \RR$ satisfying
% $F = (f, f\circ(1\ 2), \dots, f\circ (1 \ n))^\top$. Here,
% $(1 \ i)\in S_n$ is the transposition between $1$ and $i$.
% \end{prop}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% The full proof of Prop. 4.1 commented out 2019/5/22
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \begin{proof}
% %[Proof of Proposition \ref{prop:relation-invariant-equivariant}]
% First, we show that for $\Stab(1)$-invariant function $f$,
% the map $F = (f\circ (1 \ i))_i \in \RR^n$ is $S_n$-equivariant.
% It is well-known that the transpositions $(i \ j)$ for $i,j = 1, \dots, n $ generate $S_n$. Any transposition $(i \ j)$ can be written as
% $(i\ j)=(1\ i)(1\ j)(1\ i)$, hence $(1\ i), i= 1, \dots, n$ also generate $S_n$.
% Therefore, it suffices to show that
% $F((1 \ i) \cdot \bm{x}) = (1 \ i) \cdot F(\bm{x})$ for any $i$ and any $\bm{x}\in\RR^n$.
% If $j$ is not equal to $i$, we have
% \begin{align*}
% F((1\ i)\cdot \bm{x})_j &= f((1\ j)((1 \ i)\cdot \bm{x})) \\
% %& = f((j \ i)(1 \ j)\cdot \bm{x}) \\
% %& = f((1 \ j) \cdot \bm{x}) = F(\bm{x})_j.
% \end{align*}
% Here, $F(\bm{x})_j$ means the $j$-th entry of
% $F(\bm{x})\in \RR^n$.
% The last equation is due to $\Stab(1)$-invariance of $f$.
% Similarly, we have $F((1\ i)\cdot \bm{x})_i = f(\bm{x}) = F(\bm{x})_1$ and
% $F((1\ i)\cdot \bm{x})_1 = f((1\ i) \cdot \bm{x}) = F(\bm{x})_i$.
% Hence $F((1\ i)\cdot \bm{x}) = (1\ i)\cdot F(\bm{x})$ holds.
% Next, we assume that $F$ is $S_n$-equivariant, { i.e.}
% $F(\sigma\cdot \bm{x}) = \sigma \cdot F(\bm{x})$ for any $\bm{x}\in \RR^n$ and any $\sigma\in S_n$.
% We set $F = (f_1, \dots, f_n)^\top$, $f_i:\RR^n \to \RR$.
% Then, by the definition, the $S_n$-equivariance of $F$ is equivalent to
% \begin{equation}
% f_i (\sigma \cdot \bm{x}) = f_{\sigma^{-1}(i)}(\bm{x}) \label{eq:S_n-equivariance_for_f_i}
% \end{equation}
% for any $\sigma\in S_n$ and $\bm{x} \in \RR^n$.
% By this equation (\ref{eq:S_n-equivariance_for_f_i}) with $i=1$, we have
% \[
% f_1 (\sigma \cdot\bm{x}) = f_{\sigma^{-1}(1)}(\bm{x})
% \]
% for any $\sigma \in S_n$ and any $\bm{x}\in\RR^n$.
% Because $\sigma \in \Stab(1)$ if and only if
% $\sigma^{-1} \in \Stab(1)$, we have $f_{\sigma^{-1}(1)}(\bm{x}) = f_1(\bm{x})$ if $\sigma \in \Stab(1)$. Thus, $f_1$ is $\Stab(1)$-invariant.
% We consider the equation (\ref{eq:S_n-equivariance_for_f_i})
% for $i = 1$ and the transposition $\sigma = (1 \ j)$ for $j \neq 1$,
% we have
% \[
% f_1 ( (1 \ j)\cdot \bm{x}) = f_{(1\ j)^{-1}(1)}(\bm{x}) = f_j(\bm{x}).
% \]
% This means $F(\bm{x}) = (f_1((1\ i)\cdot \bm{x}))_i$.
% %The unicity of $f=f_1$ for $F$ is trivial by the above argument.
% \end{proof}
% %\fi
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% %%% Sketch of the proof of Prop. 4.1 commented out 2019/5/21 (Return 2019/5/22)
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% The proof of this proposition is based on
% the coset decomposition of $S_n$ by $\Stab(1)$ as
% $S_n = \bigsqcup_{i=1}^n \Stab(1)(1 \ i)$ and
% the fact that transposition $(1 \ i)$, $i=1, \dots, n$
% generate $S_n$.
% Using these facts and some calculation, it is not hard to
% show the equivalence.
% %Due to the limitaion of space, we omit the proof.
% The detail of the proof is in Appendix \ref{sec:appendix-proof-of-prop}.
%\begin{proof}[Proof]
% First, we show that $\psi$ is well defined ie thar for
%First, we show that for $\Stab(1)$-invariant function $f$,
%the map $F = (f\circ (1 \ i))_i \in \RR^n$ is $S_n$-equivariant.
% $f\in\left\{\Stab(1)-invariant function : \mathbb{R}^n\longmapsto\mathbb{R}\right\}$, we have $\psi(f)\in\left\{S_n-equivariant function : \mathbb{R}^n\longmapsto\mathbb{R}^n\right\}$.\\
%It is well-known that the transpositions $(i \ j)$ for $i,j = 1, \dots, n $ generate $S_n$. Any transposition $(i \ j)$ can be written as
%$(i\ j)=(1\ i)(1\ j)(1\ i)$, hence $(1\ i), i= 1, \dots, n$ also generate $S_n$.
%Therefore, it suffices to show that
%$F((1 \ i) \cdot \bm{x}) = (1 \ i) \cdot F(\bm{x})$ for any $i$ and any $\bm{x}\in\RR^n$.
%\todo{vector should be bold?}
% $\forall 1\leq i\leq n$, $\forall f\in\left\{\Stab(1)-inv : \mathbb{R}^n\longmapsto\mathbb{R}\right\}$ , we have $\psi(f)((1\ i)\cdot x)=(1\ i)\cdot\psi(f)(x)$.
%If $j$ is not equal to $i$, we have
%\begin{align*}
% F((1\ i)\cdot \bm{x})_j &= f((1\ j)((1 \ i)\cdot \bm{x})) \\
%& = f((j \ i)(1 \ j)\cdot \bm{x}) \\
%& = f((1 \ j) \cdot \bm{x}) = F(\bm{x})_j.
%\end{align*}
%Here, $F(\bm{x})_j$ means the $j$-th entry of
%$F(\bm{x})\in \RR^n$.
%The last equation is due to $\Stab(1)$-invariance of $f$.
%Similarly, we have $F((1\ i)\cdot \bm{x})_i = f(\bm{x}) = F(\bm{x})_1$ and
%$F((1\ i)\cdot \bm{x})_1 = f((1\ i) \cdot \bm{x}) = F(\bm{x})_i$.
%Hence $F((1\ i)\cdot \bm{x}) = (1\ i)\cdot F(\bm{x})$ holds.
% Indeed,
% $$\psi(f)((1\ i)\cdot x)=\begin{pmatrix}
% f((1\ i)\cdot x)\\f((1\ 2)(1\ i)\cdot x)\\ \vdots \\f((1\ i)(1\ i)\cdot x)\\ \vdots\\f((1\ n)(1\ i)\cdot x)
% \end{pmatrix}
% =\begin{pmatrix}
% f((1\ i)\cdot x)\\f((2\ i)(1\ 2)\cdot x)\\ \vdots \\f( x)\\ \vdots\\f((n\ i)(1\ n)\cdot x)
% \end{pmatrix}=\begin{pmatrix}
% f((1\ i)\cdot x)\\f((1\ 2)\cdot x)\\ \vdots \\f( x)\\ \vdots\\f((1\ n)\cdot x)
% \end{pmatrix}
% =(1\ i)\cdot\psi(f)(x)$$
% Because $f$ is $\Stab(1)$-invariant. So $\psi(f)$ is $S_n$-equivariant and $\psi$ is well defined.
% \\
%Next, we assume that $F$ is $S_n$-equivariant, { i.e.}
%$F(\sigma\cdot \bm{x}) = \sigma \cdot F(\bm{x})$ for any $\bm{x}\in \RR^n$ and any $\sigma\in S_n$.
%We set $F = (f_1, \dots, f_n)^\top$, $f_i:\RR^n \to \RR$.
%Then, by the definition, the $S_n$-equivariance of $F$ is equivalent to
%\begin{equation}
%f_i (\sigma \cdot \bm{x}) = f_{\sigma^{-1}(i)}(\bm{x}) \label{eq:S_n-equivariance_for_f_i}
%\end{equation}
%for any $\sigma\in S_n$ and $\bm{x} \in \RR^n$.
%By this equation (\ref{eq:S_n-equivariance_for_f_i}) with $i=1$, we have
%\[
% f_1 (\sigma \cdot\bm{x}) = f_{\sigma^{-1}(1)}(\bm{x})
%\]
%for any $\sigma \in S_n$ and any $\bm{x}\in\RR^n$.
%Because $\sigma \in \Stab(1)$ if a d onlynif
%$\sigma^{-1} \in \Stab(1)$, we have $f_{\sigma^{-1}(1)}(\bm{x}) = f_1(\bm{x})$ if $\sigma \in \Stab(1)$. Thus, $f_1$ is $\Stab(1)$-invariant.
% $$F\circ\sigma=\sigma\circ F$$
% $$\Leftrightarrow \begin{pmatrix}
% f_1\circ\sigma\\ \vdots \\f_n\circ\sigma\end{pmatrix}
% =\begin{pmatrix}
% f_{\sigma^{-1}(1)}\\ \vdots \\f_{\sigma{-1}(n)}\end{pmatrix}
% $$
%We consider the equation (\ref{eq:S_n-equivariance_for_f_i})
%for $i = 1$ and the transposition $\sigma = (1 \ j)$ for $j \neq 1$,
%we have
%\[
%f_1 ( (1 \ j)\cdot \bm{x}) = f_{(1\ j)^{-1}(1)}(\bm{x}) = f_j(\bm{x}).
%\]
%This means $F(\bm{x}) = (f_1((1\ i)\cdot \bm{x}))_i$.
%The unicity of $f=f_1$ for $F$ is trivial by the above argument.
% And for $\sigma=(1\ i)$, it gives $ \begin{pmatrix}
% f_1\circ(1\ i)\\ \vdots \\f_i\circ(1\ i)\\ \vdots\\f_n\circ(1\ i)\end{pmatrix}
% =\begin{pmatrix}
% f_i\\ \vdots\\ f_1 \\ \vdots\\f_n\end{pmatrix}
% \Rightarrow f_i=f_1\circ(1\ i) $
% \\
% \\
% Conclusion is $F=\psi(f_1)$ and $\psi$ is surjective.
%\end{proof}
\section{Equivariant case}\label{sec:equivariant}
In this section, we prove Theorem \ref{theo:main-theorem}, namely, the equivariant version of the universal approximation theorem.
The key ingredient is the following proposition (proof is in Appendix \ref{sec:appendix-proof-of-prop}):
\begin{prop}\label{prop:relation-invariant-equivariant}
Notations are same as Theorem \ref{theo:main-theorem}.
Then, a map $F\colon\RR^n \to \RR^n$ is $G$-equivariant if and only if $F$ can be
represented by $F = (f_1\circ \tau_{1,1}, f_1\circ \tau_{1,2}, \dots, f_m\circ \tau_{m,l_m})^\top$ for some $\Stab_G(i_j)$-invariant functions
$f_j\colon\RR^n \to \RR$.
Here, $\tau_{j,k} \in G$ is regarded as a linear map $\RR^n\to \RR^n$.
\end{prop}
%We prove this proposition in Appendix~\ref{sec:appendix-proof-of-prop}.
For simplicity, we prove Theorem \ref{theo:main-theorem} only for $G=S_n$.
We can show the general case by a similar argument.
More precisely, we construct an $S_n$-equivariant deep neural network
%made from stacking $S_n$-equivariant single layers
approximating the given $S_n$-equivariant function.
Similarly, we can prove this theorem for any finite group $G$.
To show Theorem \ref{theo:main-theorem} for $G=S_n$, we divide the proof
to four steps as follows:\\[-14pt]
\begin{enumerate}
\item By Proposition \ref{prop:relation-invariant-equivariant} proved below, we reduce
the argument on $S_n$-equivariant map $F$ to the one of $\Stab(1)$-invariant function $f$.
\item Modifying Theorem \ref{thm:representation-of-invariant},
we have a representation of $\Stab(1)$-invariant function $f$.
\item Using the above representation, we have a $\Stab(1)$-invariant
deep neural net which approximates $f$ and construct
a deep neural network approximating $F$.
\item We introduce a certain action of $S_n$ on $(\RR^n)^n$ which appears the first hidden layer naturally and show the $S_n$-equivariance between the input layer and
the first hidden layer.
\end{enumerate}
% Because step~1 has already been explained in Section~\ref{sec:relation-inv-equ}, we start from step~2 in
% the next section.
%%%%%%%%%%%%%%
% Should be replace as below?
%%%%%%%%%%%%%%
% Because step~1 has already been explained in Section~\ref{sec:relation-inv-equ}, we start from step~2 in
% Section \ref{subsec:Rep-of-Stab(1)-inv}.
% In Section~\ref{subsec:approx-Stab(1)-inv}, we explain step~3. Section~\ref{subsec:another-action} is devoted to step~4.
% In Section~\ref{subsec:equivariance-of-first-gap}, we show step~5
% and conclude the proof of Theorem \ref{theo:main-theorem}.
%\todo{Should be more simple.}
% We have already explained about step 1 in Section~\ref{sec:relation-inv-equ}.
%Let $f$ be the
%$\Stab(1)$-function on $\RR^n$ to $\RR$ corresponding to $F$
%by Propsition \ref{prop:relation-invariant-equivariant}.
% \subsection{Representation of Stab(1)-invariant functions}\label{subsec:Rep-of-Stab(1)-inv}
We first investigate step 1.
We recall that, during this section,
we only consider the action of $S_n$ on $\RR^n$
induced from permutation $\sigma\cdot (x_1,\dots, x_n)^\top = (x_{\sigma^{-1}(1)}, \dots, x_{\sigma^{-1}(n)})^\top$.
Then, we remark that the orbit of $1$ by this action is the
total set $O_1 = \{1,2,\dots, n \}$ and the coset decomposition of
$S_n$ by $\Stab(1)$ is $S_n = \bigsqcup_{i=1}^n\Stab(1)(1 \ i)$.
Thus, we have the following:
\begin{cor}\label{cor:relation-invariant-equivariant}
A map $F\colon\RR^n \to \RR^n$ is $S_n$-equivariant if and only if there is $\Stab(1)$-invariant function
$f\colon\RR^n \to \RR$ satisfying
$F = (f\circ (1 \ 1), f_1\circ (1 \ 2), \dots, f_m\circ (1 \ n))^\top$. Here, $(i \ j) \in S_n$ is the transposition between
$i$ and $j$ and is regarded as
a linear map $\RR^n \to \RR^n$.
\end{cor}
% The stabilizer subgroup of $S_n$ with
% respect to $\{1 \}$ is defined by
% \[
% \Stab_n(1) =\left\{\sigma\in S_n \mid \sigma\cdot 1 =1\right\}.
% \]
% When there is no ambiguity, we denote it by $\Stab(1)$.
% Then, for any $\Stab(1)$-invariant function $f:\RR^n \to \RR$,
% $F= (f, f\circ(1 \ 2), \dots, f\circ (1 \ n))^\top : \RR^n \to
% \RR^n$ is $S_n$-equivariant.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% Sketch of the proof of Prop. 4.1 commented out 2019/5/21 (Return 2019/5/22)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%The proof of this proposition is based on
%the coset decomposition of $S_n$ by $\Stab(1)$ as
%$S_n = \bigsqcup_{i=1}^n \Stab(1)(1 \ i)$ and
%the fact that transposition $(1 \ i)$, $i=1, \dots, n$
%generate $S_n$.
%Using these facts and some calculation, it is not hard to
%show the equivalence.
%%Due to the limitaion of space, we omit the proof.
% The detail of the proof is in Appendix \ref{sec:appendix-proof-of-prop}.
Next, we consider step 2.
% In this section, we establish step 2, namely,
% we show a representation theorem of $\Stab(1)$-invariant functions similar to Theorem~\ref{thm:representation-of-invariant}.
The stabilizer subgroup $\Stab_n(1)$ is isomorphic to $S_{n-1}$ as a group by Lemma. %\todo{Add Lemma in Appendix}.
Hence, we can regard the $\Stab(1)$-invariant function $f\colon\RR^n \to \RR$ as
an $S_{n-1}$-invariant function.
This point of view allows us to apply
Theorem \ref{thm:representation-of-invariant} to $f$.
Hence, we have the following representation theorem of $\Stab(1)$-invariant functions as a corollary of Theorem \ref{thm:representation-of-invariant}.
\begin{cor}[Representation of $\Stab(1)$-invariant function]\label{cor:representation-of-Stab(1)-inv}
Let $K\subset\mathbb{R}^n$ be a compact set, let $f\colon K \longrightarrow\mathbb{R}$ be a continuous and $\Stab(1)$-invariant function. Then, $f(\bm{x})$ can be represented
as
\[
%\forall (x_1,\dots,x_n)\in K,
f(\bm{x}) = f(x_1,\dots,x_n)=\rho\left(x_1,\sum_{i=2}^{n}\phi(x_i)\right),
\]
for some continuous function
$\rho\colon \mathbb{R}^{n+1}\longrightarrow\mathbb{R}$. Here,
$\phi\colon\RR \to \RR^n$ is similar as in Theorem \ref{thm:representation-of-invariant}.
% $\begin{array}[t]{l|rcl}
% \phi : & \mathbb{R} & \longrightarrow & \mathbb{R}^{n} \\
% & x & \longmapsto & \begin{pmatrix}
% 1\\x\\x^2\\ \vdots \\x^{n-1}
% \end{pmatrix} \end{array}
% $.
\end{cor}
By this corollary, %~\ref{cor:representation-of-Stab(1)-inv},
we can represent the $\Stab(1)$-invariant function $f:\mathbb{R}^n\longmapsto\mathbb{R}$ as $f=\rho\circ L \circ\Phi$, where
$\Phi: \RR^n \to \RR \times (\mathbb{R}^{n})^{n-1}$ and
$L: \RR\times (\RR^n)^{n-1} \to \RR\times \RR^n$ are \\[-10pt]
\[
\Phi(x_1, \dots, x_n) = (x_1, \phi(x_2), \dots, \phi(x_n)), \ \
L(x, \ (\bm{y}_1, \dots, \bm{y}_{n-1})) = \left(x, \sum_{i=1}^{n-1} \bm{y}_i \right).
\]
% $\Phi: \RR^n \to \RR \times (\mathbb{R}^{n})^{n-1}$ is
% \[
% \Phi(x_1, \dots, x_n) = (x_1, \phi(x_2), \dots, \phi(x_n)),
% \]
% and $L: \RR\times (\RR^n)^{n-1} \to \RR\times \RR^n$ is
% \[
% L(x, \ (\bm{y}_1, \dots, \bm{y}_{n-1})) = \left(x, \sum_{i=1}^{n-1} \bm{y}_i \right).
% \]
%\todo{to be more readable.}
% \begin{align*}
% \begin{array}[t]{l|rcl}
% \Phi : \mathbb{R}^n & \longrightarrow & \mathbb{R}\times(\mathbb{R}^{n})^{n-1} \\
% & \begin{pmatrix}
% x_1\\ \vdots\\x_n
% \end{pmatrix} & \longmapsto & \begin{pmatrix}
% x_1\\\phi(x_2)\\ \vdots \\\phi(x_n)
% \end{pmatrix} \end{array} \\
% \begin{array}[t]{l|rcl}
% L : & \mathbb{R}\times(\mathbb{R}^{n})^{n-1} & \longrightarrow & \mathbb{R}\times\mathbb{R}^{n} \\
% & \begin{pmatrix}x\\y\end{pmatrix} & \longmapsto & \begin{pmatrix}
% x\\ \sum_{i=1}^{n-1}y_i
% \end{pmatrix} \end{array}
% \end{align*}
%is linear. And $\rho$ is as in Corollary 3.1 .
%\\
%\\
% \subsection{Universal approximation theorem for Stab(1)-invariant functions}
% \label{subsec:approx-Stab(1)-inv}
Then, we consider step 3, namely, the existence of $\Stab(1)$-invariant deep neural network approximating the function $f$.
After that, using this approximator, we construct a deep neural network approximating $S_n$-equivariant function $F$.
%By the result on the universal approximation theorem by ReLU neural network in [Sonoda-Murata, 2017], \cite{approxrelu},
%\todo{what is the precise citation?},
By a slight modification of the invariant version of Proposition \ref{theo:inv-theorem} for $\Stab(1)$-invariant case,
there exists a sequence of deep neural networks $\{ A_m \}_m$
(resp. $\{ B_m\}_m$) which converges to $\Phi$ (resp. $\rho$) uniformly.
Then, the sequence of deep neural networks $\{ B_m\circ L \circ A_m\}_m$ converges to $f=\rho\circ L \circ\Phi$ uniformly.
% \begin{prop}
% Notation is same as above.
% The sequence of DNN $B_m\circ L\circ A_m$ tends to $f=\rho\circ L \circ\Phi$ for uniform convergence.
% \end{prop}
% Indeed, $L$ is Lipschitz continuous, hence uniformly continuous. On the other hand, $\rho$ is continuous on compact set $K$, thus uniformly continuous.
% Therefore, the following well-known lemma implies the uniform convergence of
% $\{ B_m\circ L \circ A_m\}_m$:
% \begin{lem}\label{lem:convergence-of-composition}
% If $(f_n)$ tends uniformly to $f$, $(g_n)$ tends uniformly to $g$ and $f$ is uniformly continuous, then $f_n\circ g_n$ tends uniformly to $f\circ g$.
% \end{lem}
% \begin{proof}[Proof of Lemma \ref{lem:convergence-of-composition}]
% For any $n\in \ZZ_{\geq 0}$, we have
% \begin{align*}
% &|f_n\circ g_n(x)-f\circ g(x)|= \\
% &|f_n\circ g_n(x)-f\circ g_n(x)+f\circ g_n(x)-f\circ g(x)| \\
% &\leq \underbrace{|f_n\circ g_n(x)-f\circ g_n(x)|}_{\leq ||f_n-f||_{\infty}} + |f\circ g_n(x)-f\circ g(x)|
% \end{align*}
% Because $f$ is uniformly continuous, for any $\epsilon > 0$,
% there exists $\delta >0$ such that $|x-y|\leq \delta \Rightarrow |f(x)-f(y)|<\epsilon$.
% Then, for sufficiently large $n$, $\|g_n-g\|_{\infty}\leq\delta \Rightarrow |f\circ g_n(x)-f\circ g(x)|\leq \epsilon$. And for sufficiently large $n$,
% $||f_n-f||_{\infty}\leq\epsilon $.
% Hence, for sufficiently large $n$, $||f_n\circ g_n-f\circ g||\leq\epsilon$.
% \end{proof}
% Lemma \ref{lem:convergence-of-composition} implies that
% $L\circ A_m$ tends to $L\circ \Phi$ because L is Lipschitz, hence uniformly continuous. Moreover, $B_m\circ L\circ A_m$ tends to $\rho\circ L\circ\Phi$, since $\rho$ is continuous on compact set so uniformly continuous.
% \end{proof}
\begin{figure}[t]
\begin{center}
\scalebox{1}{
\begin{tikzpicture}
\node[functions] (rho) {$\rho$};
\node[node,left=2em of rho] (i) {};
\node[output,right=2em of rho] (output) {};
\path[draw,->] (rho) -- (output);
\node[sum,left=2em of i] (sum) {$\sum$};
\path[draw,->] (sum) -- (i);
\path[draw,->] (i)--(rho);
\node[left=1em of sum] (2) {$\vdots$} -- (2) node[left=3em of 2] (l2) {$\vdots$} node[left=3em of l2] (ll2) {$\vdots$} node[above=1em of ll2] (a) {} node[below=1em of ll2] (b) {} node[above=1em of 2] (a') {} node[below=1em of 2] (b') {};
%\path[draw,->] (a)--(a');
%\path[draw,->] (b)--(b');
%\path[draw,->] (ll2)--(2);
%\path[draw,->] (a')--(sum);
% \path[draw,->] (b')--(sum);
% \path[draw,->] (2)--(sum);
\node[node,below=2em of 2] (n) {} -- (n) node[functions,left=2em of n] (ln) {$\phi$} node[input,left=2em of ln] (lln) {$x_n$} ;
\path[draw,->] (lln)--(ln);
\path[draw,->] (ln) -- (n);
\path[draw,->] (n) -- (sum);
\node[node,above =1em of 2] (1) {} -- (1) node[functions,left =2em of 1] (l1) {$\phi$}node[input,left=2em of l1] (ll1) {$x_2$} ;
\path[draw,->] (ll1)--(l1);
\path[draw,->] (l1) -- (1);
\path[draw,->] (1) -- (sum);
\node[node,above =2em of 1] (0) {} -- (0) node[sum,left=2em of 0] (l0) {$\mathrm{id}$}node[input,left=2em of l0] (ll0) {$x_1$} ;
\path[draw,->] (ll0)--(l0);
\path[draw,->] (l0) -- (0);
\node[sum, above=4em of sum] (id) {$\mathrm{id}$};
\node[node, below right=2em of id] (a) {};
\path[draw,->] (0)--(id);
\path[draw,->] (id)--(a);
\path[draw,->] (a)--(rho);
% \node[above of=ln,font=\scriptsize] {inputs};
%\node[below of=n,font=\scriptsize] {weights};
\end{tikzpicture}
}
\end{center}
\caption{A neural network approximating the $\Stab(1)$-invariant function $f$}
\label{fig:Stab(1)-invariant}
\end{figure}
Now, $f$ can be approached by the following deep neural network by replacing $\rho$ and $\Phi$ by universal approximators as
Diagram \ref{fig:Stab(1)-invariant}.
We remark that the left part (the part of before taking sum) of this deep neural network is naturally equivariant for the action of $\Stab(1)$.
% : $ \sigma\in \Stab(1), \ \sigma\cdot\begin{pmatrix}
% x_1\\
% \vdots \\
% x_n
% \end{pmatrix} =
% \begin{pmatrix}
% x_{1}\\
% x_{\sigma^{-1}(2)}\\
% \vdots \\
% x_{\sigma^{-1}(n)}
% \end{pmatrix}$.
For an $S_n$-equivariant map $F:\mathbb{R}^n \to \mathbb{R}^n$ with the natural action, by Proposition \ref{prop:relation-invariant-equivariant}, there is a unique $\Stab(1)$-invariant function $f$ such that
$F(\bm{x})_i=(f\circ(1 \ i))(\bm{x}).$
% \[
% F(\bm{x})_i=(f\circ(1 \ i))(\bm{x}).
% % F(\bm{x})=\begin{pmatrix}
% % f(\bm{x})\\f\circ(1\ 2)(\bm{x})\\ \vdots \\f\circ(1\ n)(\bm{x})
% % \end{pmatrix}=
% % \begin{pmatrix}
% % f\\f\\ \vdots\\f
% % \end{pmatrix}
% % \circ
% % \begin{pmatrix}
% % (1\ 1) \\ (1\ 2)\\ \vdots\\ (1\ n)
% % \end{pmatrix}\circ
% % \begin{pmatrix}
% % I_n \\ I_n \\ \vdots\\ I_n
% % \end{pmatrix}
% % (\bm{x}).
% \]
Here, $F(\bm{x}) = (F(\bm{x})_1, \dots, F(\bm{x})_n)^\top$ and we regard any element of $S_n$ as a map from $\RR^n$ to $\RR^n$.
%$\circ$ means the composition of maps,
%and $I_n$ is the $n\times n$ unit matrix.
By the argument in this section, %\ref{subsec:approx-Stab(1)-inv},
we can approximate $f$ by the previous deep neural network $\{ B_m \circ L \circ A_m\}_m$.
Substituting $B_m \circ L \circ A_m$ for $f$, we construct a deep neural network approximating $F$ as Diagram \ref{fig:Total-DNN}.
The represented function of this neural network of $F_i$ is
$B_m\circ L \circ A_m \circ (1 \ i)$.
% \begin{align}
% B_m\circ L \circ A_m \circ (1 \ i). \label{eq:def-of-DNN}
% % \begin{pmatrix}
% % B_m \circ L \circ A_m\\B_m \circ L \circ A_m\\
% % \vdots\\B_m \circ L \circ A_m
% % \end{pmatrix}
% % \circ
% % \begin{pmatrix}
% % (1\ 1) \\ (1\ 2)\\ \vdots\\ (1\ n)
% % \end{pmatrix}\circ
% % \begin{pmatrix}
% % I_n \\ I_n \\ \vdots\\ I_n
% % \end{pmatrix}.
% %\label{eq:def-of-DNN}
% \end{align}
%\vspace{-10pt}
The map $F$ splits into two parts, the part of transpositions and part of $(f, f, \dots, f)^\top$.
On the deep neural network corresponding to $F$ as above,
the latter part corresponds to the layers from the first hidden layer to the output layer. This part is the $n$ copies of same $\Stab(1)$-invariant deep neural network (an approximation of $(f, f, \dots, f)^\top$).
Thus, this part is clearly made of equivariant stacking layers for the permutation action of $S_n$.
%, hence is trivially $S_n$-equivariant.
Hence, it is remained to show that the former part is also $S_n$-equivariant.
%By returning it to appropriate part, we can construct the following deep neural network to approximate $F$:
%\todonum{This sentence is correct in grammar?}
We here investigate bounds of the width and the depth of approximators.
%the width and the depth when we limit either.
By Theorem~\ref{theo:inv-theorem}, each of $\phi$ and $\rho$ can be approximated by a shallow neural network. Hence, if we do not bound the width, we can obtain deep neural network approximating $F$ with depth three.
On the other hand, by Theorem~\ref{theo:inv-theorem} again,
if we do not bound the depth, $\phi$ (resp. $\rho$) can be approximated by
a deep neural network with width $n+1$ (resp. $n+2$). Thus, we can obtain
a deep neural network approximating $F$ with width bounded from above by $n^3$.
% we can obtain a deep neural network approximating
%\todo{explain each layer and actions on it}
%\subsection{Another $S_n$-action $\ast$ and equivariance}\label{subsec:another-action}
Finally, as step 4, we show that our deep neural network
is an $S_n$-equivariant deep neural network.
%The right part of the above deep neural network consisting of $n$ copies of the same deep neural networks (an approximation of $(f, f, \dots, f)^\top$)
%is clearly made of equivariant stacking layers for the natural action of $S_n$.
%\todonum{not clear}
The most difficult part is to show the equivariance between
the input layer $\RR^n$ and the first hidden layer $\RR^{n^2}$
presented by a function $g\colon\RR^n \to \RR^{n^2}$ as $g=(g_1,\dots, g_n)^\top$ for
$g_i = \ReLU\circ l \circ (1 \ i)$
% \[
% g =
% \begin{pmatrix}
% \ReLU \\\ReLU \\ \vdots\\ \ReLU
% \end{pmatrix}
% \circ \begin{pmatrix}
% l\\ l \\ \vdots\\l
% \end{pmatrix}
% \circ
% \begin{pmatrix}
% (1 \ 1) \\(1\ 2)\\\vdots\\(1\ n)
% \end{pmatrix}\circ
% \begin{pmatrix}
% I_n \\ I_n \\\vdots\\ I_n
% \end{pmatrix}
% \]
for a certain $\Stab(1)$-invariant linear function
$l\colon\RR^n \to V$. (Although $V$ is equal to $\RR^n$, we distinguish them to stress the difference.) Unfortunately, the permutation action on the latter space $\RR^{n^2}$ does not make the function $g$ equivariant. For this reason, we need to define another action of $S_n$ on $\RR^{n^2}$ exploiting the $\Stab(1)$-equivariance among each copies.
%%%%%
% I hesitated over which $\RR^{n^2}$ of $\bigoplus_{i=1}^n V$.
%%%%%
\begin{defn}\label{defn:another-action}
We suppose that $\Stab(1)$ acts on $\RR^n$ by permutation, denoted
by $\sigma\cdot \bm{x}$ (i.e., by regarding $\Stab(1)$ as a subgroup of $S_n$). Then, we define the action ``$\ast$'' of $S_n$ on $\RR^{n^2}$ as follows:
\[
% \sigma\ast\begin{pmatrix}
% \bm{x}_1 \\ \vdots\\ \bm{x}_n
% \end{pmatrix}
% =
% \begin{pmatrix}
% \tilde{\sigma}_1\cdot \bm{x}_{\sigma^{-1}(1)} \\ \vdots\\ \tilde{\sigma}_n\cdot \bm{x}_{\sigma^{-1}(n)}
% \end{pmatrix}
\sigma\ast(
\bm{x}_1, \dots, \bm{x}_n)
=
(
\tilde{\sigma}_1\cdot \bm{x}_{\sigma^{-1}(1)}, \dots, \tilde{\sigma}_n\cdot \bm{x}_{\sigma^{-1}(n)}
)
= (x_{\wt{\sigma}_j^{-1}(i),\sigma^{-1}(j)})_{i,j=1,\dots,n}
\]
for any $\sigma\in S_n$, and for any
$(\bm{x}_1, \bm{x}_2, \dots, \bm{x}_n) = (x_{i,j})_{i,j=1,\dots,n} \in (\RR^n)^n$.
Here, for any $i$, $\tilde{\sigma}_i$ is an element of $\Stab(1)$ defined as $(1\ i)\sigma=\tilde{\sigma_i}(1\ \sigma^{-1}(i)) $.
\end{defn}
% \begin{prop}
% This action on the left is well defined
% \end{prop}
We will prove the well-definedness of this action ``$\ast$'' in Appendix \ref{sec:appendix-well-definedness}.
This action is obtained by the injective
homomorphism
\begin{align*}
\varphi\colon S_n \hookrightarrow S_{n^2}; \ \sigma \mapsto \varphi(\sigma), \ \
\varphi(\sigma)\cdot(i,j) = (\wt{\sigma}_j^{-1}(i), \sigma^{-1}(j))
\end{align*}
for $(i,j)\in \{1,2, \dots, n \}^2$.
% {i.e.} for any $\sigma, \tau \in S_n$ and any
% $X\in \RR^{n^2}$, we have $\tau\ast(\sigma\ast X)=(\tau\sigma)\ast X$.
% Indeed, this action ``$\ast$'' is based on the coset decomposition of $S_n$ as $\bigsqcup_{i=1}^{n}\Stab(1)(1\ i)$.
% By this decomposition, $\tilde{\sigma}_i$ is uniquely determined as an element of $\Stab(1)$ for $\sigma$ and $i$. By some tedious calculation with the use of this relation, we can check the well-definedness of ``$\ast$''.
We remark that this action ``$\ast$'' naturally appears in representation theory as the induced representation, which is an operation to construct a representation of group $S_n$ from a representation of the subgroup $\Stab(1)$ of $S_n$.
% In this case, the action ``$\ast$'' is the induced representation of $S_n$ from
% the representation of $\Stab(1)$ defined by the action ``$\cdot$''.
% \begin{proof}[Proof]
% % We need to check that $\forall \sigma, \tau \in S_n$, $\forall X\in \oplus_{i=1}^{n}V_i$, we have $\tau\ast(\sigma\ast X)=(\tau\sigma)\ast X$.\\
% %We have $S_n=\bigsqcup_{i=1}^{n}\Stab(1)(1\ i)$\\
% %\\
% More precisely, $\forall\sigma\in S_n,\ \sigma=\underbrace{\sigma(1\ \sigma^{-1}(1))}_{\in \Stab(1)}(1\ \sigma^{-1}(1))$. So $\forall i$ , $\tilde{\sigma_i}=(1\ i)\sigma(1\ \sigma^{-1}(i)) \in \Stab(1)$.\\
% Let $\sigma, \tau \in S_n$ let $X\in \oplus_{i=1}^{n}V_i$ :
% $$\tau\ast(\sigma\ast X)=\tau\ast \begin{pmatrix}
% \tilde{\sigma_1}\cdot x_{\sigma^{-1}(1)} \\ \vdots\\ \tilde{\sigma_n}\cdot x_{\sigma^{-1}(n)}
% \end{pmatrix}=\begin{pmatrix}
% \tilde{\tau_1}\tilde{\sigma_{\tau^{-1}(1)}}\cdot x_{\sigma^{-1}(\tau^{-1}(1))} \\ \vdots\\ \tilde{\tau_1}\tilde{\sigma_{\tau^{-1}(n)}}\cdot x_{\sigma^{-1}(\tau^{-1}(n))}
% \end{pmatrix}$$
% $$=\begin{pmatrix}
% \tilde{\tau_1}\tilde{\sigma_{\tau^{-1}(1)}}\cdot x_{(\tau\sigma)^{-1}(1)} \\ \vdots\\ \tilde{\tau_1}\tilde{\sigma_{\tau^{-1}(n)}}\cdot x_{(\tau\sigma)^{-1}(n)}
% \end{pmatrix}$$
% \\
% \\
% And $\tilde{\tau_i}\tilde{\sigma_{\tau^{-1}(i)}}=(1\ i)\tau(1\ \tau^{-1}(i))(1\ \tau{-1}(i))\sigma(1\ \sigma^{-1}(\tau^{-1}(i)))=(1\ i)\tau\sigma(1 (\tau\sigma)^{-1}(i))=\tilde{\tau\sigma}_i $
% So :
% $$\tau\ast(\sigma\ast X)=\begin{pmatrix}
% \tilde{\tau_1}\tilde{\sigma_{\tau^{-1}(1)}}\cdot x_{(\tau\sigma)^{-1}(1)} \\ \vdots\\ \tilde{\tau_1}\tilde{\sigma_{\tau^{-1}(n)}}\cdot x_{(\tau\sigma)^{-1}(n)}
% \end{pmatrix}=\begin{pmatrix}
% \tilde{\tau\sigma}_1\cdot x_{(\tau\sigma)^{-1}(1)} \\ \vdots\\ \tilde{\tau\sigma}_n\cdot x_{(\tau\sigma)^{-1}(n)}
% \end{pmatrix}=(\tau\sigma)\ast X$$
% So the action is well defined.
% \end{proof}
% \subsection{Equivariance between the input layer and the first hidden layer}\label{subsec:equivariance-of-first-gap}
We conclude the proof of Theorem \ref{theo:main-theorem} by
showing the $S_n$-equivariance of $g$:
%between the input layer $\RR^n and the first hidden layer $\RR^{n^2}$ of our deep neural network. %(\ref{eq:def-of-DNN}).
% The situation is as follows:
% $S_n$ acts on $\RR^n$ and $V$ by
% the action ``$\cdot$'' induced from permutation,
% and $\Stab(1)$ acts on $V$ by the same action ``$\cdot$''.
% The action on $\RR^{n^2}$ is by ``$\ast$'' in Definition \ref{defn:another-action}.
% In this setting, it suffices to show $S_n$-equivariance of the function $g$ defined as follows:
% For a $\Stab(1)$-equivariant linear function $l:\mathbb{R}^n\to V$, the map $g:\RR^n\to \RR^{n^2}$
% is defined by
% \[
% g =
% \begin{pmatrix}
% \ReLU \\\ReLU \\ \vdots\\ \ReLU
% \end{pmatrix}
% \circ \begin{pmatrix}
% l\\ l \\ \vdots\\l
% \end{pmatrix}
% \circ
% \begin{pmatrix}
% (1 \ 1) \\(1\ 2)\\\vdots\\(1\ n)
% \end{pmatrix}\circ
% \begin{pmatrix}
% I_n \\ I_n \\\vdots\\ I_n
% \end{pmatrix}.
% \]
%Then, the following lemma holds:
\begin{lem}\label{lem:equivariance-first-part}
The function $g\colon \RR^n \to \RR^{n^2}$ is $S_n$-equivariant.
\end{lem}
\vspace{-6pt}
%The proof of this lemma is only calculation.
The detail of proof is in Appendix \ref{subsec:equivariance-of-first-part}.
By this lemma, we conclude the proof of Theorem \ref{theo:main-theorem}.
We remark that the affine transformation $g$ is corresponding to $Z_1$ in the notation (\ref{eq:DNN}). By a representation theoretic aspect,
this is an intertwining operator between these representation spaces.
This affine transformation has $n^3$ free parameters a priori. However, by $S_n$-equivariance and a representation theoretic argument, in principle, $g$ can be written by only five parameters.
By a similar argument, for the other hidden layers, the affine
transformation $Z_i: \RR^{n^2}\otimes V_i \to \RR^{n^2}\otimes V_{i+1}$ has
$15\dim V_i\dim V_{i+1}$ parameters (though $n^4\dim V_i\dim V_{i+1}$ a priori).
% then,
% $$=\begin{pmatrix}
% l\\\vdots\\l
% \end{pmatrix}
% \circ
% \begin{pmatrix}
% \tilde{\sigma_1}\cdot(1\ \sigma^{-1}(1)\cdot x\\\tilde{\sigma_2}\cdot(1\ \sigma^{-1}(2)\cdot x\\\vdots\\\tilde{\sigma_n}\cdot(1\ \sigma^{-1}(n)\cdot x
% \end{pmatrix}=
% \begin{pmatrix}
% l\circ (\tilde{\sigma_1}\cdot(1\ \sigma^{-1}(1)\cdot x)\\\vdots\\l\circ (\tilde{\sigma_n}\cdot(1\ \sigma^{-1}(n)\cdot x)
% \end{pmatrix}
% =\begin{pmatrix}
% \tilde{\sigma_1}\cdot l\circ (1\ \sigma^{-1}(1)\cdot x)\\\vdots\\\tilde{\sigma_n}\cdot l\circ (1\ \sigma^{-1}(n)\cdot x)
% \end{pmatrix}$$
% Because $l$ is $\Stab(1)$-equivariant
% $$=\sigma\ast \begin{pmatrix}
% l\circ ((1\ 1)\cdot x)\\\vdots\\l\circ\ ((1\ n)\cdot x)
% \end{pmatrix}=
% \sigma\ast\left(\begin{pmatrix}
% l\\\vdots\\l
% \end{pmatrix}
% \circ
% \begin{pmatrix}
% \tilde{\sigma_1}\cdot(1\ \sigma^{-1}(1))\cdot x\\\tilde{\sigma_2}\cdot(1\ \sigma^{-1}(2))\cdot x\\\vdots\\\tilde{\sigma_n}\cdot(1\ \sigma^{-1}(n))\cdot x
% \end{pmatrix}\right)$$
%Which means the equivariance of $g$.
\section{Dimension reduction}\label{sec:dimension}
In this section, we give the proof of Theorem \ref{thm:number-of-params}.
% we discuss about some advantages of our invariant/equivariant models. We first point out that our invariant/equivariant models have fewer parameters than the usual models for a same number of of layers and nodes. This is because they have the constraint conditions induced by the invariance/equivariance with respect to actions of $S_n$. Indeed, the following proposition calculates the number of parameters in our equivariant models.
\begin{prop}\label{layer}
Let $Z_l=\ReLU\circ W_l\colon\mathbb{R}^M \to \mathbb{R}^N$ be an $S_n$-equivariant map. Assume that the $S_n$-action on $\mathbb{R}^M$ and $\mathbb{R}^N$ is a union of permutations. Then, $n$ divides $M$ and $N$, and the number of the free parameters in $W_l$ is equal to $2MN/n^2$.
\end{prop}
%\vspace{-6pt}
\begin{proof}
Since $\mathbb{R}^M$ and $\mathbb{R}^N$ have union of permutation actions, by considering the orbit of the cordinates, we see that $n$ devides $M$ and $N$. Let us write $\mathbb{R}^M=(\mathbb{R}^n)^
{M^{\prime}}$ and $\mathbb{R}^N=(\mathbb{R}^n)^{N^{\prime}}$.
In this case, $W_l$ is written by sum of $n\times n$ matrices $V_{ij}$, namely $W_l = (V_{ij})^n_{i,j=1}$.
% \[
% W_l = \left(
% \begin{array}{cccc}
% V_{11} & V_{12} & \ldots & V_{1M'} \\
% V_{21} & V_{22} & \ldots & V_{2M'} \\
% \vdots & \vdots & \ddots & \vdots \\
% V_{M'1} & V_{M'2} & \ldots & V_{ N'M'}
% \end{array}
% \right).
% \]
Here, each $n\times n$ matrix $V_{ij}$ corresponds to the linear map:
\[
\mathbb{R}^n \hookrightarrow (\mathbb{R}^n)^
{M^{\prime}} \xrightarrow{Z_l} (\mathbb{R}^n)^{N^{\prime}} \twoheadrightarrow \mathbb{R}^n,
\]
where the first map is the inclusion to coordinates of $(\mathbb{R}^n)^
{M^{\prime}}$ and the last map is projection to coordinates of $(\mathbb{R}^n)^{N^{\prime}}$.
Since these constructions are taken to be compatible with $S_n$-action, we see that $\mbox{ReLU}\circ V_{ij}$ is $S_n$-equivariant. If the activation functions are bijective, we are done because of the same discussion as in the proof of Lemma 3 in \cite{deepsets}. But in our case, we need more discussion, which is in Appendix \ref{subsec:dim}.
%But since ReLU functions are not bijective, we need more discussion.
%Let us take a transposition $(p,q)$. We consider the condition under which the $p$-th coordinate of $V_{ij}(\bm{x})$ is positive. Since the $p$-th coordinate of $V_{ij}(\bm{x})$ is written as an inner product of the $p$-th row vector of $V_{ij}$ and $\bm{x}$, the points in the upper half of the hyperplane defined by $p$-th row vector give positive values. Hence, if $\bm x$ is in the intersection of two hyperplane associated to $p$ and $q$, the $p$-th coordinate and $q$-th coordinate of $V_{ij}(\bm{x})$ is positive. This means that if $\bm x$ is in the intersection of two hyperplane and if we consider the transition $(p, q)$, we may assume $V_{ij}=W_l$. Combining with the discussion in the discussion in the proof of Lemma 3 in \cite{deepsets}, we have
%$$V_{ij}=\lambda \bm I + \gamma (\bm 1 \bm 1^\top).$$
%Hence, for each $n\times n$ matrix, we have two free parameters. Since the number of $n\times n$ matrices appeared in $W_i$ is $MN/n^2$, we have the desired result.
\end{proof}
\begin{proof}[Proof of Theorem \ref{thm:number-of-params}]
By Proposition \ref{layer}, the number of the free parameter in each equivariant layer is bounded by $2M^2/n^2$. Hence we obtain the desired bound.
\end{proof}
%By this observation, we conclude the number of free parameters in the equivariant models.
%\begin{theo}\label{number}
%Let $\mathcal{N}$ be a deep neural network of the invariant model, consist of the equivariant part of depth $d$ and width $M$ and the preinvariant part of depth $e$ and width $N$ (resp. the equivariant model of depth $d$ and width $M$).
%Then, the number of free parameters in the model is $(2^dM^{2d}/n^{2d})\cdot N^{2e}$ (resp. $2^dM^{2d}/n^{2d}$).
%\end{theo}
%The number of the free parameters in the usual model is $M^{2d}$.
%Hence, Theorem \ref{thm:number-of-params} implies that {\it the free parameters of the invariant/equivarint models are exponentially fewer than the ones of the usual models}.
%Data augmentation is a common technique in empirical learning.
%In the case of the invariant/equivariant tasks, a possible augmentation is to make new samples by the acting permutation on samples. New samples are effective to the usual models, but not effective to our invariant/equivariant models. This is because in our models, all weights are symmetric under permutation actions. This means that our models learn augmented samples from a sample. By acting permutations, we can make $n!$ new samples from a sample. Hence, computational complexity is reduced to $1/n!$ times.
\section{Conclusion}
We introduced some invariant/equivariant models of deep neural networks which are universal approximators for invariant/equivariant functions. The universal approximation theorems in this paper and the discussion in
Section \ref{sec:dimension} show that
%give a guarantee for the success of the models by \cite{deepsets}, i.e.,
although the free parameters of our invariant/equivariant models are exponentially fewer than the ones of the usual models, the invariant/equivariant models can approximate the invariant/equivariant functions to arbitrary accuracy.
Our theory also implies that there is much possibility that the group models behave as the usual models for the tasks related to groups and representation theory can be powerful tool for theory of machine learning.
This must be a good perspective to develop the models in deep learning.
%\subsubsection*{Author Contributions}
%If you'd like to, you may include a section for author contributions as is done
%in many journals. This is optional and at the discretion of the authors.
%
%\subsubsection*{Acknowledgments}
%Use unnumbered third level headings for the acknowledgments. All
%acknowledgments, including those to funding agencies, go at the end of the paper.
%\bibliography{iclr2020_conference}
\bibliographystyle{iclr2020_conference}
\input{iclr2020_conference.bbl}
\newpage
\appendix
%\section{Appendix}
\section{Review of groups and group actions}\label{sec:appendix-groups}
Let $G$ be a set with product, i.e., for any
$g,h \in G$, $gh$ is defined an element of $G$.
Then, $G$ is called { a group} if $G$ satisfies
the following conditions:
\begin{enumerate}
\item There is an element $e \in G$ such that $eg=ge = x$ for all $g \in G$.
\item For any $g \in G$, there is an element $g^{-1} \in G$ such that $g g^{-1} = g^{-1} g = e$.
\item For any $g, h, i \in G$, $(gh)i = g(hi)$ holds.
\end{enumerate}
Let $G, G'$ be two finite groups. We say a map $\varphi\colon G \to G'$ is a (group) homomorphism if $\varphi(gh) = \varphi(g)\varphi(h)$. This means that
the map $\varphi$ preserves the group structures of $G$ in $G'$.
Next, we review actions of groups. Let $X$ be a set. {
An action of $G$} (or { $G$-action}) on $X$ is defined as a map $G\times X \to X; (g,x) \mapsto g\cdot x$ satisfying the following:
\begin{enumerate}
\item For any $x \in X$, $e\cdot x = x$.
\item For any $g,h \in G$ and $x\in X$, $(gh)\cdot x = g\cdot (h\cdot x)$.
\end{enumerate}
%Then, we also say that $G$ acts on $X$ by a left action.
%A typical example of groups is the permutation group $S_n$ of
% $n$ elements.
% The $S_n$ and its stabilizer subgroup $\Stab(1)$ with respect to $\{1 \}$ defined below only appear in this paper:
%\[
% \Stab_n(1) = \Stab(1) =\left\{\sigma\in S_n \mid \sigma\cdot 1 =1\right\}.
%\]
%When there is no ambiguity, we denote it by $\Stab(1)$.
% Although $S_n$ acts on the set $X = \{1, 2, \dots, n \}$ by
% permutation, we remark that
% we can define some other actions of $S_n$ on $X$, e.g., the trivial action $\sigma\cdot x = x$ for any $\sigma\in S_n, x \in X$. When we stress the difference of some actions,
% we use some distinguished letters for each actions as
%``$\cdot$'' or ``$\ast$'' etc. In particular, in what follow, \todonum{English here is strange?}we set the permutation action of $S_n$ by $\sigma(i)$ for $\sigma \in S_n$ and $i \in \{1,2, \dots, n \}$.
If these conditions are satisfied, we say that $G$ acts on $X$ by a left action.
\begin{exmp}
An example which we mainly consider in
this paper is the permutation group $S_n$ of $n$ elements:
\[
S_n = \{ \sigma\colon \{1, \dots, n \} \to
\{1, \dots, n \}; \mathrm{bijective} \}
\]
and the product of $\sigma, \tau \in S_n$ is given
by the composition $\sigma\circ \tau$ as maps.
$S_n$ acts on the set $\{ 1, 2, \dots, n \}$ by
the permutation $\sigma\cdot i = \sigma^{-1}(i)$.
\end{exmp}
% A typical example which we mainly consider in
% this paper is the permutation group $S_n$ of $n$ elements acting on $X = \{1, 2, \dots, n \}$ by permutation, i.e., $\sigma\cdot i =\sigma(i)$ for $\sigma \in S_n$ and $i \in X$.
We remark that actions of $S_n$ on $X$ is not unique,
for example, the trivial action $\sigma\cdot x = x$ for any $\sigma\in S_n, x \in X$ is also one of action.
Hence, when we stress the difference of some actions,
we use some distinguished notation for each actions as ``$\cdot$'' or ``$\ast$'' etc.
Let $G$ be a group and $H$ a subset of $G$.
We call $H$ a subgroup of $G$ if $H$ is a group with the same product as $G$.
\begin{exmp}
Let $G$ be a finite group acting on a set $X$. For an element $x\in X$,
the stabilizer subgroup $\Stab_G(x)$ of G with respects to $\{x \}$ is defined by
\[
\Stab_G(x) = \{ g \in G \mid
g\cdot x = x \}.
\]
\end{exmp}
When $G= S_n$ and $x = 1$, we use the following notation: $\Stab_n(1)= \Stab(1):=\Stab_G(x)$.
Let $G, G'$ be two groups. If there is an injective homomorphism $\varphi\colon G \to G'$, the image $\varphi(G)\subset G'$ can be a subgroup of $G'$. Then, we say that $\varphi$ is an embedding of group $G$ to $G'$.
Moreover, if $G'$ acts on a set $X$, then $G$ also acts on $G'$ through $\varphi$, i.e., by $\varphi(\sigma)\cdot x$ for $\sigma\in G$ and $x\in X$.
Then, the following proposition holds:
\begin{prop}\label{prop:embedding-to-symmetric-group}
Any finite group $G$ can be embedded into $S_n$ for some $n$.
\end{prop}
\begin{proof}
Let $n:= |G|$ and $G = \{ g_1, g_2, \dots, g_n \}$. For any $g\in G$,
$gg_i = g_j$ for some $j \in \{ 1,2, \dots, n\}$ as $g g_i \in G$.
We set $\sigma_g^{-1} (i) = j$. Then, we define
$\sigma\colon G \to S_n; g \mapsto \sigma_g$. It is easy to show that
this $\sigma$ is an injective homomorphism.
\end{proof}
This proposition implies that any finite group $G$ can be realized as
a permutation action on $R^n$ for some $n$.
Let $G$ be a finite group acting on $X$. Then, for $x\in X$, we define the
($G$-)orbit $O_x$ of $x$ as
\[
O_x = G\cdot x = \{ g\cdot x \mid g\in G \}.
\]
Then, for $x, y \in X$, the relation that $x$ and $y$ are in a same orbit
is an equivalent relation. Hence, $X$ can be divided to a disjoint union of the equivalent classes of this equivalent relation:
\[
X = \bigsqcup_{j = 1}^m O_{x_j}.
\]
We call this the ($G$-)orbit decomposition of $X$.
Let $H$ be a subgroup of a finite group $G$. Then, for $g \in G$,
the set
\[
gH = \{ gh \in G \mid h \in H\}
\]
is called the left coset of $H$ with respect to $g$. The relation that two elements
$g$ and $g'$ are in a same coset is also an equivalent relation.
Hence, we can divide $G$ to a disjoint union of equivalent classes of this
relation:
\[
G = \bigsqcup_{i=1}^l g_iH.
\]
We call this decomposition the right coset decomposition of $G$ by $H$.
We set $G/H$ as the set of the left cosets of $G$ by H:
\[
G/H = \{ g_1H, g_2H, \dots, g_lH \}.
\]
Then, there is a relation between an orbit and a set of cosets:
\begin{prop}\label{prop:orbit-stab-bijectivity}
Let $G$ be a finite group acting on a set $X$. For $x\in X$,
the map
\[
G/\Stab_G(x) \to O_x; \ g\Stab_G(x) \to g \cdot x
\]
is bijective.
\end{prop}
\begin{proof}
It is easy to check well-definedness and bijectivity.
\end{proof}
For $G = S_n$ acting on $\{1,2, \dots, n \}$,
the $G$-orbit of $1$ is only one, i.e., $O_1 = \{ 1,2, \dots, n \}$.
Hence, the following holds (the permutation action of $S_n$ is defined by taking inverse $\sigma^{-1}$, hence we need to consider the set of right cosets):
\begin{cor}
The map
\[
\Stab_n(1)\!\setminus \! S_n \to \{1,2,\dots, n \}; \Stab_n(1)\sigma \mapsto
\sigma^{-1}(1)
\]
is bijective.
\end{cor}
% To prove Proposition \ref{prop:relation-invariant-equivariant} in
% Section \ref{sec:equivariant}, we use this relation.
\section{Proof of Proposition \ref{theo:inv-theorem}}
\label{sec:appendix-inv-theorem}
\begin{proof}[Proof of Proposition \ref{theo:inv-theorem}]
We may assume $N=1$. In fact,
since we consider the $L^{\infty}$-norm, if all components of $\|f-R_\mathcal{N}\|$ is bounded by $\epsilon$, then $\|f-R_\mathcal{N}\|_\infty \leq \epsilon$ holds.
For $N=1$, we have $f\colon K\to \mathbb{R}$. Then, by
Theorem~\ref{thm:representation-of-invariant}, we obtain the representation of $f$ as
\[
f(x_1,\dots,x_n)=\rho\left(\sum_{i=1}^{n}\phi(x_i)\right).
\]
By the universal approximation property of ReLU deep neural network, we can find two sequences of ReLU deep neural network $\{\mathcal{N}^{\rho}_{k}\}_k$ and $\{\mathcal{N}^{\phi}_{k}\}_k$ such that their corresponding functions $\{F_{\mathcal{N}^{\rho}_{k}}\}_k$
and $\{F_{\mathcal{N}^{\phi}_{k}}\}_k$ tend to $\rho$ and $\phi$ for the $L^{\infty}$-norm when $k$ tends to infinity. Let $\{\mathcal{N}_k\}_k$ be the sequence of networks whose corresponding functions are $F_{\mathcal{N}_k} : (x_1,\dots,x_n)\mapsto \left(F_{\mathcal{N}^{\rho}_{k}}\left(\sum_{i=1}^{n}F_{\mathcal{N}^{\phi}_{k}}(x_i)\right) \right)$. To show that $\{F_{\mathcal{N}_k}\}_k$ uniformly tends to $f$, we use the following lemma:
\begin{lem}\label{lem:approximation-composite}
If $\{f_k\}_k$ tends uniformly to $f$, $\{g_k\}_k$ tends uniformly to $g$ and $f$ is uniformly continuous, then $\{f_k\circ g_k\}_k$ tends uniformly to $f\circ g$.
\end{lem}
\begin{proof}[proof of Lemma~\ref{lem:approximation-composite}]
For any $k$, we have
\begin{align*}
|f_k\circ g_k(x)-f\circ g(x)|&=|f_k\circ g_k(x)-f\circ g_k(x)+f\circ g_k(x)-f\circ g(x)| \\
&\leq \underbrace{|f_k\circ g_k(x)-f\circ g_k(x)|}_{\leq \| f_k-f\|_{\infty}} + |f\circ g_k(x)-f\circ g(x)|.
%& \leq |f_k\circ g_k(x)-f\circ g_k(x)| + |f\circ g_k(x)-f\circ g(x)|.
\end{align*}
Let $\epsilon > 0$.
By the uniform continuity of $f$, there is a $\delta >0$
such that for all $x,y$ satisfying
$|x-y|\leq \delta$, $|f(x)-f(y)|\leq \epsilon/2$ holds.
Then for large enough $k$, we have both
$\|g_k-g \|_{\infty}\leq\delta $ which implies that
for any $x$, $|f\circ g_k(x)-f\circ g(x)|\leq \epsilon/2$, and $||f_k-f||_{\infty}\leq \epsilon/2$.
Hence, for any $k$ large enough, $||f_k\circ g_k-f\circ g||\leq\epsilon$ holds.
\end{proof}
Now using Lemma \ref{lem:approximation-composite}, we have that $(x_1,\dots,x_n)\mapsto\sum_{i=1}^{n}F_{\mathcal{N}^{\phi}_{k}}(x_i)$ tends to $(x_1,\dots,x_n)\mapsto\sum_{i=1}^{n}\phi(x_i)$, because $(x_1,\dots,x_n)\mapsto\sum_{i=1}^{n}x_i$ is Lipschitz (by triangular inequality)
so uniformly continuous. Then, using Lemma \ref{lem:approximation-composite} again, we obtain the result, since $\rho$ is continuous on a compact set so uniformly continuous.
Moreover, by \cite{sonoda2017neural}, we can approximate $\phi$ and $\rho$ by shallow networks. Hence, we have an approximation by a deep neural network
$\mathcal{N}$ which has two hidden layers and the width is not bounded.
By \cite{approxrelu}, we can respectively approximate each of $\phi$ and $\rho$ by some neural networks whose width are bounded by $n+2$ and the depth is not bounded. Hence, we have an approximation by a deep neural network $\mathcal{N}$ whose width is bounded by $n(n+2)$ and the depth is not bounded.
The invariant function $f$ is approached by a deep neural network $\mathcal{N}$ having the following diagram:
\begin{center}
\begin{tikzpicture}
\node[functions] (rho) {$\mathcal{N}_{\rho}$};
\node[output,right=2em of rho] (output) {};
\path[draw,->] (rho) -- (output);
\node[node,left=1.5em of rho] (a) {}
node[sum,left=1.5em of a] (sum) {$\sum$};
\path[draw,->] (sum) -- (a);
\path[draw,->] (a) -- (rho);
\node[left=1.5em of sum] (2) {$\vdots$} -- (2) node[left=3em of 2] (l2) {$\vdots$} node[left=3em of l2] (ll2) {$\vdots$}; %node[below of=l2] (ldots) {$\vdots$} node[below of=ll2] (lldots) {$\vdots$} ;
%\node[below of=2] (dots) {$\vdots$} ;
\node[node,below of=2] (n) {} -- (n) node[functions,left=2em of n] (ln) {$\mathcal{N}_{\phi}$} node[input,left=2em of ln] (lln) {$x_n$} ;
\path[draw,->] (lln)--(ln);
\path[draw,->] (ln) -- (n);
\path[draw,->] (n) -- (sum);
\node[node,above =1em of 2] (1) {} -- (1) node[functions,left =2em of 1] (l1) {$\mathcal{N}_{\phi}$}node[input,left=2em of l1] (ll1) {$x_2$} ;
\path[draw,->] (ll1)--(l1);
\path[draw,->] (l1) -- (1);
\path[draw,->] (1) -- (sum);
\node[node,above =1em of 1] (0) {} -- (0) node[functions,left=2em of 0] (l0) {$\mathcal{N}_{\phi}$}node[input,left=2em of l0] (ll0) {$x_1$} ;
\path[draw,->] (ll0)--(l0);
\path[draw,->] (l0) -- (0);
\path[draw,->] (0) -- (sum);
\end{tikzpicture}
\end{center}
Let us show that this is a $S_n$-{\it invariant} deep neural network. Since the sum ($\Sigma$) is an invariant function, we can divide this neural network in two parts : the fist part on the left of the symbol $\Sigma$ and the second on the right. For each layer $\mathbb{R}^{d_i}$ of $\mathcal{N}_{\phi}$ there is a corresponding map $Z_i: \mathbb{R}^{d_i} \to \mathbb{R}^{d_{i+1}}$. Then for each layer $(\mathbb{R}^{d_i})^{n}$ of the first part of $\mathcal{N}$, there is a $S_n$ action $ \sigma\cdot (x_i)_i = (x_{\sigma^{-1}(i)})_i$ for
$\bm{x} = (x_i)_i \in \mathbb{R})^{d_i}$, and the corresponding map $(Z_1,\cdots,Z_n): (\mathbb{R}^{d_i})^n \to (\mathbb{R}^{d_{i+1}})^n$ is $S_n$-equivariant. On the second part of $\mathcal{N}$, there is no $S_n$ actions on the layers $\mathbb{R}^{d_j}$ except the trivial action. Hence, for this action, each corresponding map $Z_i: \mathbb{R}^{d_j} \to \mathbb{R}^{d_{i+j}}$ is invariant. This shows that the network $\mathcal{N}$ is $S_n$-{\it invariant}.
\end{proof}
\section{Proof of Proposition \ref{prop:relation-invariant-equivariant}}
\label{sec:appendix-proof-of-prop}
\begin{proof}[Proof of Proposition
\ref{prop:relation-invariant-equivariant}]
%\todonum{Revise for any finite group}
First, we show that for $\Stab_G(i_j)$-invariant function $f_j$,
the map $F = (f_1\circ\sigma_{1,1}, f_1\circ\sigma_{1,2}, \dots, f_m\circ \sigma_{m,l_m})^\top \in \RR^n$ is $G$-equivariant.
Without loss of generality, we may assume that
\begin{align*}
O_{i_1} = \{ 1, 2, \dots, l_1 \}, \
O_{i_2} = \{ l_1 + 1, l_1 + 2, \dots, l_1 + l_2 \}, \dots.
\end{align*}
In particular, $i_j = \sum_{k = 1}^{j-1} l_k + 1$. For each $j$, we set the coset decomposition by
\[
G = \bigsqcup_{k = 1}^{l_j} \Stab_G(i_j) \tau_{j, k},
\]
where we may assume that $\tau_{j,k}$ satisfies
$\tau_{j,k}^{-1}(i_j) = i_j + k$.
Now, for $\sigma\in G$, there is a unique $k'$ such that
$\tau_{j,k}\sigma = \wt{\sigma}_{j,k}\tau_{j,k'}$. Hence, the
$(ij + k)$-th entry becomes
\[
f_j\circ \tau_{j,k}\circ \sigma = f_j\circ \wt{\sigma}_{j,k} \circ
\tau_{j,k'} = f_j \circ \tau_{j,k'}.
\]
On the other hand, we have
\begin{align*}
\sigma^{-1}( i_j + k) = \tau_{j,k'}^{-1} \wt{\sigma}_{j,k}^{-1}
\tau_{j,k} (i_j + k) = i_j + k'.
\end{align*}
This shows $F$ is $G$-equivariant.
Conversely, we assume that $F\colon \RR^n \to \RR^n$ is $G$-equivariant.
Let $F = (g_1, g_2, \dots, g_n)^\top$. The orbit decomposition
$O_{i_1}, \dots, O_{i_m}$ is same as above.
For $\sigma \in G$, by $\sigma\cdot F(\bm{x}) = F(\sigma\cdot \bm{x})$ for any $\bm{x} \in \RR^n$, we have
\begin{align}
g_i(\sigma\cdot \bm{x}) = g_{\sigma^{-1}(i)}(\bm{x}).\label{eq:entry-wise}
\end{align}
For simplicity, we consider only for $i = 1$. Then, $\sigma^{-1}(1)\in O_{i_1}$, thus $\sigma^{-1}(1) = 1, \dots, l_1$.
By the equation (\ref{eq:entry-wise}), for $\sigma \in \Stab_G(1)$, we have
\[
g_1(\sigma\cdot \bm{x}) = g_1(\bm{x}),
\]
as $\sigma \in \Stab_G(1)$ if and only if $\sigma^{-1} \in \Stab_G(1)$.
Hence, $g_1$ is $G$-invariant.
The equation (\ref{eq:entry-wise}) for $\tau_{1,k}$ ($k = 1, 2, \dots, l_1$)
implies
\[
g_1(\tau_{1,k}\cdot \bm{x}) = g_{\tau_{1,k}^{-1}}(1)(\bm{x}) = g_k(\bm{x}).
\]
This completes the proof for the orbit $O_{i_1}$.
By same arguments, we can prove the similar result for the other orbits.
% It is well-known that the transpositions $(i \ j)$ for $i,j = 1, \dots, n $ generate $S_n$. Any transposition $(i \ j)$ can be written as
% $(i\ j)=(1\ i)(1\ j)(1\ i)$, hence $(1\ i), i= 1, \dots, n$ also generate $S_n$.
% Therefore, it suffices to show that
% $F((1 \ i) \cdot \bm{x}) = (1 \ i) \cdot F(\bm{x})$ for any $i$ and any $\bm{x}\in\RR^n$.
% If $j$ is not equal to $i$, we have
% \begin{align*}
% F((1\ i)\cdot \bm{x})_j &= f((1\ j)((1 \ i)\cdot \bm{x})) \\
% %& = f((j \ i)(1 \ j)\cdot \bm{x}) \\
% %& = f((1 \ j) \cdot \bm{x}) = F(\bm{x})_j.
% \end{align*}
% Here, $F(\bm{x})_j$ means the $j$-th entry of
% $F(\bm{x})\in \RR^n$.
% The last equation is due to $\Stab(1)$-invariance of $f$.
% Similarly, we have $F((1\ i)\cdot \bm{x})_i = f(\bm{x}) = F(\bm{x})_1$ and
% $F((1\ i)\cdot \bm{x})_1 = f((1\ i) \cdot \bm{x}) = F(\bm{x})_i$.
% Hence $F((1\ i)\cdot \bm{x}) = (1\ i)\cdot F(\bm{x})$ holds.
% Next, we assume that $F$ is $S_n$-equivariant, { i.e.},
% $F(\sigma\cdot \bm{x}) = \sigma \cdot F(\bm{x})$ for any $\bm{x}\in \RR^n$ and any $\sigma\in S_n$.
% We set $F = (f_1, \dots, f_n)^\top$, $f_i:\RR^n \to \RR$.
% Then, by the definition, the $S_n$-equivariance of $F$ is equivalent to
% \begin{equation}
% f_i (\sigma \cdot \bm{x}) = f_{\sigma^{-1}(i)}(\bm{x}) \label{eq:S_n-equivariance_for_f_i}
% \end{equation}
% for any $\sigma\in S_n$ and $\bm{x} \in \RR^n$.
% By this equation (\ref{eq:S_n-equivariance_for_f_i}) with $i=1$, we have
% \[
% f_1 (\sigma \cdot\bm{x}) = f_{\sigma^{-1}(1)}(\bm{x})
% \]
% for any $\sigma \in S_n$ and any $\bm{x}\in\RR^n$.
% Because $\sigma \in \Stab(1)$ if and only if
% $\sigma^{-1} \in \Stab(1)$, we have $f_{\sigma^{-1}(1)}(\bm{x}) = f_1(\bm{x})$ if $\sigma \in \Stab(1)$. Thus, $f_1$ is $\Stab(1)$-invariant.
% We consider the equation (\ref{eq:S_n-equivariance_for_f_i})
% for $i = 1$ and the transposition $\sigma = (1 \ j)$ for $j \neq 1$,
% we have
% \[
% f_1 ( (1 \ j)\cdot \bm{x}) = f_{(1\ j)^{-1}(1)}(\bm{x}) = f_j(\bm{x}).
% \]
% This means $F(\bm{x}) = (f_1((1\ i)\cdot \bm{x}))_i$.
% %The unicity of $f=f_1$ for $F$ is trivial by the above argument.
\end{proof}
%\fi
\section{Well-definedness of the action ``$\ast$''.}
\label{sec:appendix-well-definedness}
We here show that the left action ``$\ast$'' of $S_n$ on $\RR^{n^2}$ defined in Section \ref{sec:equivariant} is
well-defined, {i.e.}, for any $\sigma, \tau \in S_n$ and any
$X = (\bm{x}_1^\top, \dots, \bm{x}_n^\top)^\top \in \RR^{n^2}$, we have $\tau\ast(\sigma\ast X)=(\tau\sigma)\ast X$.
% We need to check that $\forall \sigma, \tau \in S_n$, $\forall X\in \oplus_{i=1}^{n}V_i$, we have $\tau\ast(\sigma\ast X)=(\tau\sigma)\ast X$.\\
%We have $S_n=\bigsqcup_{i=1}^{n}\Stab(1)(1\ i)$\\
%\\
The permutation group $S_n$ is decomposed as
\[
S_n = \bigsqcup_{i=1}^n \Stab(1) (1\ i).
\]
For any $\sigma\in S_n$, because $\sigma(1\ \sigma^{-1}(1))$ is in $\Stab(1)$,
this $\sigma$ is in the coset $\Stab(1)(1 \ \sigma^{-1}(1))$.
Apply this for $(1\ i)\sigma$ for $i$,
$(1 \ i)\sigma$ is in the coset
\[
\Stab(1)(1\ ((1\ i)\sigma)^{-1}(1)) =
\Stab(1)(1\ \sigma^{-1}(i)),
\]
thus, we have a unique element
$\tilde{\sigma_i} \in \Stab(1)$ such that
\begin{align}\label{eq:sigma-tilde}
(1\ i)\sigma = \tilde{\sigma}_i (1\ \sigma^{-1}(i)).
\end{align}
For $\sigma, \tau \in S_n$ and
$X= (\bm{x}_1^\top, \dots, \bm{x}_n^\top)^\top\in \RR^{n^2}$,
we have
\begin{align*}
\tau\ast(\sigma\ast X)
&=\tau\ast \begin{pmatrix}
\tilde{\sigma}_1\cdot x_{\sigma^{-1}(1)} \\ \vdots\\ \tilde{\sigma}_n\cdot x_{\sigma^{-1}(n)}
\end{pmatrix}
=\begin{pmatrix}
\tilde{\tau_1}\tilde{\sigma}_{\tau^{-1}(1)}\cdot x_{\sigma^{-1}(\tau^{-1}(1))} \\ \vdots\\ \tilde{\tau_1}\tilde{\sigma}_{\tau^{-1}(n)}\cdot x_{\sigma^{-1}(\tau^{-1}(n))}
\end{pmatrix}\\
&=\begin{pmatrix}
\tilde{\tau_1}\tilde{\sigma}_{\tau^{-1}(1)}\cdot x_{(\tau\sigma)^{-1}(1)} \\ \vdots\\ \tilde{\tau_1}\tilde{\sigma}_{\tau^{-1}(n)}\cdot x_{(\tau\sigma)^{-1}(n)}
\end{pmatrix}
\end{align*}
Then, by the equation (\ref{eq:sigma-tilde}),
$\tilde{\sigma}_i = (1\ i) \sigma (1\ \sigma^{-1}(i))$.
Hence we have
\begin{align*}
\tilde{\tau_i}\tilde{\sigma}_{\tau^{-1}(i)}
&=(1\ i)\tau(1\ \tau^{-1}(i))(1\ \tau^{-1}(i))\sigma(1\ \sigma^{-1}(\tau^{-1}(i))) \\
&=(1\ i)\tau\sigma(1\ (\tau\sigma)^{-1}(i))
=\wt{(\tau\sigma)}_i.
\end{align*}
This relation implies
\begin{align*}
\tau\ast(\sigma\ast X)=\begin{pmatrix}
\tilde{\tau_1}\tilde{\sigma}_{\tau^{-1}(1)}\cdot x_{(\tau\sigma)^{-1}(1)} \\ \vdots\\ \tilde{\tau_1}\tilde{\sigma}_{\tau^{-1}(n)}\cdot x_{(\tau\sigma)^{-1}(n)}
\end{pmatrix}
=\begin{pmatrix}
\widetilde{(\tau\sigma)}_1\cdot x_{(\tau\sigma)^{-1}(1)} \\ \vdots\\ \widetilde{(\tau\sigma)}_n\cdot x_{(\tau\sigma)^{-1}(n)}
\end{pmatrix}=(\tau\sigma)\ast X
\end{align*}
Thus, the action is well-defined.
To satisfy the hypothesis of Theorem 2.3, we need to check that the "$\ast$" action is free.
\section{Proof of Lemma \ref{lem:equivariance-first-part}}
\label{subsec:equivariance-of-first-part}
\begin{proof}[Proof of Lemma \ref{lem:equivariance-first-part}]
For any $i$, any $\sigma\in S_n$ and any $\bm{x}\in\mathbb{R}^{n}$, we have
\[
(l\circ (1 \ i) \circ I_n)(\sigma\cdot \bm{x}) =
l (((1 \ i)\sigma) \cdot \bm{x}).
\]
% $$\begin{pmatrix}
% l\\\vdots\\l
% \end{pmatrix}
% \circ
% \begin{pmatrix}
% id\\(1\ 2)\\\vdots\\(1\ n)
% \end{pmatrix}
% (\sigma\cdot x)=\begin{pmatrix}
% l\\\vdots\\l
% \end{pmatrix}
% \circ
% \begin{pmatrix}
% \sigma\cdot x\\(1\ 2)\sigma\cdot x\\\vdots\\(1\ n)\sigma\cdot x
% \end{pmatrix}$$
Because for any $i$, there is a unique $\tilde{\sigma}_i \in \Stab(1)$ such that $(1\ i)\sigma=\tilde{\sigma_i}(1\ \sigma^{-1}(i))$ as
in Definition \ref{defn:another-action}, we have
\[
l ( ((1 \ i)\sigma) \cdot \bm{x}) =
l (\tilde{\sigma_i}(1\ \sigma^{-1}(i)) \cdot \bm{x})
= \tilde{\sigma_i} \cdot l ((1\ \sigma^{-1}(i)) \cdot \bm{x}).
\]
The last equality is due to $\Stab(1)$-equivariance of $l$.
On the other hand, by Definition \ref{defn:another-action},
$i$-th entry of $\sigma\ast g(\bm{x})$ becomes
\begin{align*}
(\sigma\ast g(\bm{x}))_i &= \tilde{\sigma}_i \cdot g(\bm{x})_{\sigma^{-1}(i)} \\
& = \tilde{\sigma}_i\cdot (\ReLU ( l ((1 \ \sigma^{-1}(i))\cdot \bm{x}))).
\end{align*}
Because $\tilde{\sigma}_i \circ \ReLU = \ReLU \circ\tilde{\sigma}_i$ holds, $g$ is $S_n$-equivariant.
\end{proof}
\section{Data augmentation}
Data augmentation is a common technique in empirical learning.
In the case of the invariant/equivariant tasks, a possible augmentation is to make new samples by the acting permutation on samples. New samples are effective to the usual models, but not effective to our invariant/equivariant models. This is because in our models, all weights are symmetric under permutation actions. This means that our models learn augmented samples from a sample. By acting permutations, we can make $n!$ new samples from a sample. Hence, computational complexity is reduced to $1/n!$ times.
\section{Dimension reduction}\label{subsec:dim}
In this section, we give the proof of Proposition \ref{layer}.
\begin{proof}[Proof of Proposition \ref{layer}]
%We firstly see that $n$ devides $M$ and $N$. By induction on $L$, we claim that if $\mathbb{R}^L$ has $S_n$-action, $n$ devides $L$.
%If $L$ is smaller than $n$, it is easy to see that $\mathbb{R}^L$ does not have $S_n$-action. Let $(x_i)_{i\in I}$ be the coordinate of $\mathbb{R}^L$. Since $\mathbb{R}^L$ have $S_n$-action, for each $\sigma \in S_n$, there is a $j_{\sigma}$ such that $\sigma \circ x_1 = x_{j_{\sigma}}$ holds. We can see that the set $J_1=\{j_{\sigma}|\sigma \in S_n\}$ is consist of $n$ elements. Let $I_1=I-J_1$, then $(x_i)_{i\in I_1}$ is the coordinate of $\mathbb{R}^{L-n}$ and $\mathbb{R}^{L-n}$ has $S_n$-action induced by the one of $\mathbb{R}^L$. By the inductive hypothesis, $n$ devides $L-n$, hence $n$ devides $L$.
Since the action on $\mathbb{R}^M, \mathbb{R}^N$ is a union of permutations on nodes in each equivariant layer,
we can write $\mathbb{R}^M=(\mathbb{R}^n)^
{M^{\prime}}$ and $\mathbb{R}^N=(\mathbb{R}^n)^{N^{\prime}}$.
In this case, $W_i$ can be written as the following form;
\[
W_i = \left(
\begin{array}{cccc}
V_{11} & V_{12} & \ldots & V_{1M'} \\
V_{21} & V_{22} & \ldots & V_{2M'} \\
\vdots & \vdots & \ddots & \vdots \\
V_{M'1} & V_{M'2} & \ldots & V_{ N'M'}
\end{array}
\right)
\]
, where $Vij$ are $n\times n$ matrices.
Let us consider the following maps: $$\mathbb{R}^n \xrightarrow{i} (\mathbb{R}^n)^
{M^{\prime}} \xrightarrow{Z_i} (\mathbb{R}^n)^{N^{\prime}} \xrightarrow{p} \mathbb{R}^n \\
$$
, where the first map is the inclusion to the coordinates started from the $(j-1)n$ th coordinate of $(\mathbb{R}^n)^
{M^{\prime}}$ ended at the $jn-1$ th coordinate of $(\mathbb{R}^n)^
{M^{\prime}}$ and the last map is the projection to the coordinates started from the $(i-1)n$ th coordinate of $(\mathbb{R}^n)^
{N^{\prime}}$ ended at the $in-1$ th coordinate of $(\mathbb{R}^n)^
{N^{\prime}}$ . We chase the elements of these maps as follows;
\begin{align*}
p\circ Z_i \circ i \left(\left[
\begin{array}{c}
a_1 \\
a_2 \\
\vdots \\
a_n
\end{array}
\right]\right)
&=p\circ Z_i
\left(\left[
\begin{array}{c}
0 \\
\vdots \\
0\\
a_1 \\
a_2 \\
\vdots \\
a_n\\
0 \\
\vdots \\
0
\end{array}
\right]
\right)
=p \left( \mbox{ReLU}\circ \left(
\begin{array}{cccc}
V_{11} & V_{12} & \ldots & V_{1M'} \\
V_{21} & V_{22} & \ldots & V_{2M'} \\
\vdots & \vdots & \ddots & \vdots \\
V_{N'1} & V_{N'2} & \ldots & V_{ N'M'}
\end{array}
\right)
\left[
\begin{array}{c}
0 \\
\vdots \\
0\\
a_1 \\
a_2 \\
\vdots \\
a_n\\
0 \\
\vdots \\
0
\end{array}
\right]
\right)\\
&=p\left(\begin{array}{c}
\mbox{ReLU}\circ V_{1j}
\left[
\begin{array}{c}
a_1 \\
a_2 \\
\vdots \\
a_n\\
\end{array}
\right]
\\
\vdots\\
\mbox{ReLU}\circ V_{M'j}
\left[
\begin{array}{c}
a_1 \\
a_2 \\
\vdots \\
a_n\\
\end{array}
\right]
\end{array}
\right)
=
\mbox{ReLU}\circ V_{ij}
\left[
\begin{array}{c}
a_1 \\
a_2 \\
\vdots \\
a_n\\
\end{array}
\right]
\end{align*}
Hence, each $n\times n$ matrix $V_{ij}$ induces a subneural network.
Since these constructions are taken to be compatible with $S_n$-action, we see that $f=\mbox{ReLU}\circ V_{ij}: \mathbb{R^n} \to \mathbb{R^n}$ is $S_n$-equivariant. If the activation functions are bijective, we are done because of the same discussion as in the proof of Lemma 3 in \cite{deepsets}. But since ReLU functions are not bijective, we need more discussion.
Let us take a transposition $\sigma=(p \ q)$. Since $f$ is $S_n$ -equivariant, $\sigma\circ f(\bm{x}) = f(\sigma\circ\bm{x})$ holds for any $\bm{x}$. We have
\begin{align*}
%\begin{eqnarray*}
\sigma\circ f(\bm{x})
&= \sigma \left( \mbox{ReLU}\circ \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{p1} &\ldots & a_{pn} \\
\vdots & \ddots & \vdots \\
a_{q1} &\ldots & a_{qn}\\
\vdots &\ddots & \vdots \\
a_{n1} & \ldots & a_{nn}
\end{array}
\right)
\left[
\begin{array}{c}
x_1 \\
x_2 \\
\vdots \\
x_n\\
\end{array}
\right]
\right)
=\mbox{ReLU} \left( \sigma\circ \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{p1} &\ldots & a_{pn} \\
\vdots & \ddots & \vdots \\
a_{q1} &\ldots & a_{qn}\\
\vdots &\ddots & \vdots \\
a_{n1} & \ldots & a_{nn}
\end{array}
\right)
\left[
\begin{array}{c}
x_1 \\
x_2 \\
\vdots \\
x_n\\
\end{array}
\right]
\right)\\
&=\mbox{ReLU}\left( \sigma \circ
\left[ \begin{array}{c}
\sum_{j=1}^n a_{1j}x_j \\
\vdots\\
\sum_{j=1}^n a_{pj}x_j \\
\vdots \\
\sum_{j=1}^n a_{qj}x_j \\
\vdots \\
\sum_{j=1}^n a_{nj}x_j \\
\end{array}
\right]
\right)
=\mbox{ReLU}\left(
\left[ \begin{array}{c}
\sum_{j=1}^n a_{1j}x_j \\
\vdots\\
\sum_{j=1}^n a_{qj}x_j \\
\vdots \\
\sum_{j=1}^n a_{pj}x_j \\
\vdots \\
\sum_{j=1}^n a_{nj}x_j \\
\end{array}
\right]
\right)
= \mbox{ReLU} \left( \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{q1} &\ldots & a_{qn} \\
\vdots & \ddots & \vdots \\
a_{p1} &\ldots & a_{pn}\\
\vdots &\ddots & \vdots \\
a_{n1} & \ldots & a_{nn}
\end{array}
\right)
\left[
\begin{array}{c}
x_1 \\
x_2 \\
\vdots \\
x_n\\
\end{array}
\right]
\right).
%\end{eqnarray*}
\end{align*}
On the other hand, we have
\begin{align*}
f(\sigma \circ \bm{x}) = f\left( \left[
\begin{array}{c}
x_1 \\
\vdots \\
x_q \\
\vdots \\
x_p \\
\vdots\\
x_n\\
\end{array}
\right]
\right)
&= \mbox{ReLU} \left( \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1p} &\ldots & a_{1q} & \ldots & a_{1n} \\
a_{21} &\ldots & a_{2p} &\ldots & a_{2q} & \ldots & a_{2n} \\
\vdots & \ddots & \vdots & \ddots & \vdots & \ddots & \vdots \\
a_{n1} &\ldots & a_{np} & \ldots & a_{nq} & \ldots & a_{nn}
\end{array}
\right)
\left[
\begin{array}{c}
x_1 \\
\vdots \\
x_q \\
\vdots \\
x_p \\
\vdots\\
x_n\\
\end{array}
\right]
\right)\\
&= \mbox{ReLU} \left( \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1q} &\ldots & a_{1p} & \ldots & a_{1n} \\
a_{21} &\ldots & a_{2q} &\ldots & a_{2p} & \ldots & a_{2n} \\
\vdots & \ddots & \vdots & \ddots & \vdots & \ddots & \vdots \\
a_{n1} &\ldots & a_{nq} & \ldots & a_{np} & \ldots & a_{nn}
\end{array}
\right)
\left[
\begin{array}{c}
x_1 \\
\vdots \\
x_p \\
\vdots \\
x_q \\
\vdots\\
x_n\\
\end{array}
\right]
\right).
\end{align*}
\begin{claim}
$ \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1q} &\ldots & a_{1p} & \ldots & a_{1n} \\
a_{21} &\ldots & a_{2q} &\ldots & a_{2p} & \ldots & a_{2n} \\
\vdots & \ddots & \vdots & \ddots & \vdots & \ddots & \vdots \\
a_{n1} &\ldots & a_{nq} & \ldots & a_{np} & \ldots & a_{nn}
\end{array}
\right)
=
\left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{q1} &\ldots & a_{qn} \\
\vdots & \ddots & \vdots \\
a_{p1} &\ldots & a_{pn}\\
\vdots &\ddots & \vdots \\
a_{n1} & \ldots & a_{nn}
\end{array}
\right)
$
\end{claim}
\begin{proof}[Proof of Claim 1]
We have
\begin{align*}
&\mbox{ReLU} \left( \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{q1} &\ldots & a_{qn} \\
\vdots & \ddots & \vdots \\
a_{p1} &\ldots & a_{pn}\\
\vdots &\ddots & \vdots \\
a_{n1} & \ldots & a_{nn}
\end{array}
\right)
\left[
\begin{array}{c}
x_1 \\
x_2 \\
\vdots \\
x_n\\
\end{array}
\right]
\right)=
&\mbox{ReLU} \left( \left(
\begin{array}{ccccccc}
a_{11} &\ldots & a_{1q} &\ldots & a_{1p} & \ldots & a_{1n} \\
a_{21} &\ldots & a_{2q} &\ldots & a_{2p} & \ldots & a_{2n} \\
\vdots & \ddots & \vdots & \ddots & \vdots & \ddots & \vdots \\
a_{n1} &\ldots & a_{nq} & \ldots & a_{np} & \ldots & a_{nn}
\end{array}
\right)
\left[
\begin{array}{c}
x_1 \\
\vdots \\
x_p \\
\vdots \\
x_q \\
\vdots\\
x_n\\
\end{array}
\right]
\right)
\end{align*}
for any $\bm{x}$.
Hence, the the $l$ -th coordinate of the left hand side is positive if and only if the one of the right hand side is positive.It is clear that each equation is positive for infinitely many $\bm{x}$. This implies that the coefficients of each equations coincide. Hence, we have the desired result.
\end{proof}
We show that $a_{pp}=a_{qq}$ holds for any $p,q$.
We can see that the $(p \ q)$ entry of the matrix of the left hand side in the claim is equal to $a_{pp}$. Similarly, the
$(p \ q)$ entry of the matrix of the right hand side in the claim is equal to $a_{qq}$. Hence, by the claim, we have $a_{pp}=a_{qq}$.
We show that if $i \neq j$ and $s\neq t$, $a_{ij}=a_{st}$ holds. Consider the $(i \ q)$ entry of each matrices, the one of the left hand side is equal to $a_{ip}$ and the one of the right hand side is equal to $a_{iq}$.
Hence, we have $a_{ip}=a_{iq}$, where $i \neq p$ ane $i \neq q$. By the symmetry, $a_{pi}=a_{qi}$ holds for any $i\neq p$ and $i\neq q$.
Hence, we have
$$
a_{ij}=a_{sj}=a_{st}
$$
for any $i \neq j$ and $s \neq t$.
Hence, we can write $V_{ij}=\lambda \bm I + \gamma (\bm 1 \bm 1^\top)$.
\end{proof}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage[textsize=small]{todonotes}
\newcounter{todocounter}
\newcommand{\todonum}[2][]
{\stepcounter{todocounter}\todo[#1]{\thetodocounter: #2}}
\newcounter{mycomment}
\newcommand{\mycomment}[2][]{%
% initials of the author (optional) + note in the margin]
\refstepcounter{mycomment}%
{%
%\setstretch{0.7}% spacing
\todo[color={red!100!green!33},size=\small]{%
\textbf{Comment [\uppercase{#1}\themycomment]:}~#2}%
}}
%%%%%%%%%%%%%%%% added option by us
\usepackage{amsmath, amsfonts, amssymb, amsthm, array}
\usepackage{mathtools, bm}
\usepackage{color}
\usepackage{xcolor}
%\usepackage[absolute,overlay]{textpos}
\renewcommand{\figurename}{Diagram}
%
%\usepackage{graphicx}
%\usepackage{nicefrac} % compact symbols for 1/2, etc.
%\usepackage{microtype} % microtypography
%\usepackage{subfigure}
%\usepackage[colorinlistoftodos]{todonotes}
\usepackage{cases}
\usepackage{xr}
%\usepackage{hyperref}
%\usepackage{lipsum}
\usepackage{tikz}
\usetikzlibrary{positioning}
\theoremstyle{plain}
\newtheorem{theo}{Theorem}[section]
\newtheorem{claim}{Claim}
\newtheorem{lem}{Lemma}[section]
\newtheorem{prop}{Proposition}[section]
\newtheorem{cor}{Corollary}[section]
\theoremstyle{definition}
\newtheorem{defn}{Definition}[section]
\newtheorem{exmp}{Example}[section]
\renewcommand*{\proofname}{Proof}
\tikzset{basic/.style={draw,fill=blue!80,text width=1em,text badly centered}}
\tikzset{sum/.style={basic,rectangle,fill=white}}
\definecolor{mygray}{gray}{0.8}
\tikzset{node/.style={basic,circle,fill=mygray}}
%\tikzset{node/.style={basic,circle,fill=grey!30}}
\tikzset{input/.style={basic,circle}}
\tikzset{output/.style={basic,circle,fill=red}}
\tikzset{weights/.style={basic,rectangle}}
\tikzset{functions/.style={basic,rectangle,fill=green}}
\newcounter{num}
\setcounter{num}{0}
%\newcommand{\todo}[1]{\refstepcounter{num}\textcolor{red}{[[ToDo, No.\thenum: \ #1]]}} % Todo
\newcommand{\RR}{\mathbb{R}} % real number field
\newcommand{\ZZ}{\mathbb{Z}} % ring of integers
%\newcommand{\ul}[1]{\underline{#1}} % underline
\newcommand{\ol}[1]{\overline{#1}} % overline
\newcommand{\wt}[1]{\widetilde{#1}} % widetilde
\newcommand{\wh}[1]{\widehat{#1}} % widetilde
\newcommand{\Stab}{\mathrm{Stab}}
\newcommand{\ReLU}{\mathrm{ReLU}}
\newcommand{\inv}{\mathrm{inv}}
\newcommand{\equi}{\mathrm{equiv}}
\begin{thebibliography}{26}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Alex et~al.(2012)Alex, Sutskever, and
Geoffrey~E]{krizhevsky2012imagenet}
Krizhevsky Alex, Sutskever Sutskever, Ilya, and Hinton Geoffrey~E.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In \emph{Advances in neural information processing systems}, pp.\
1097--1105, 2012.
\bibitem[Arnold(1957)]{arnoldrepresentation}
Vladimir~I. Arnold.
\newblock On functions of three variables.
\newblock \emph{Proceeding of the USSR Academy of Sciences}, 114:\penalty0
679--681, 1957.
\newblock English translation: Amer. Math. Soc. Transl., 28 (1963), pp.
51–54.
\bibitem[Barron(1994)]{barron1994approximation}
Andrew~R Barron.
\newblock Approximation and estimation bounds for artificial neural networks.
\newblock \emph{Machine learning}, 14\penalty0 (1):\penalty0 115--133, 1994.
\bibitem[Bloem-Reddy \& Teh(2019)Bloem-Reddy and Teh]{bloem2019probabilistic}
Benjamin Bloem-Reddy and Yee~Whye Teh.
\newblock Probabilistic symmetry and invariant neural networks.
\newblock \emph{arXiv preprint arXiv:1901.06082}, 2019.
\newblock URL \url{https://arxiv.org/abs/1901.06082}.
\bibitem[Cohen \& Welling(2016)Cohen and Welling]{Cohen-Welling2016}
Taco~S Cohen and Max Welling.
\newblock Group equivariant convolutional networks.
\newblock \emph{Proceedings of the 33rd International Conference on Machine
Learning}, 4, 2016.
\newblock URL: \url{arXiv preprint arXiv:1602.07576, 2016. }.
\bibitem[Cybenko(1989)]{cyb}
George Cybenko.
\newblock Approximation by superpositions of a sigmoidal function.
\newblock \emph{Mathematics of control, signals and systems}, 2\penalty0
(4):\penalty0 303--314, 1989.
\bibitem[Funahashi(1989)]{funahashi1989approximate}
Ken-Ichi Funahashi.
\newblock On the approximate realization of continuous mappings by neural
networks.
\newblock \emph{Neural networks}, 2\penalty0 (3):\penalty0 183--192, 1989.
\bibitem[Gens \& Domingos(2014)Gens and Domingos]{Gens2014}
Robert Gens and Pedro~M Domingos.
\newblock Deep symmetry networks.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
2537--2545, 2014.
\bibitem[Goodfellow et~al.(2013)Goodfellow, Bulatov, Ibarz, Arnoud, and
Shet]{goodfellow2013multi}
Ian~J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet.
\newblock Multi-digit number recognition from street view imagery using deep
convolutional neural networks.
\newblock \emph{arXiv preprint arXiv:1312.6082}, 2013.
\bibitem[Hanin \& Sellke(2017)Hanin and Sellke]{approxrelu}
Boris Hanin and Mark Sellke.
\newblock Approximating continuous functions by relu nets of minimal width.
\newblock \emph{arXiv preprint arXiv:1710.11278}, 2017.
\bibitem[Hartford et~al.(2018)Hartford, Graham, Leyton-Brown, and
Ravanbakhsh]{hartford2018deep}
Jason Hartford, Devon Graham, Kevin Leyton-Brown, and Siamak Ravanbakhsh.
\newblock Deep models of interactions across sets.
\newblock In \emph{International Conference on Machine Learning}, pp.\
1914--1923, 2018.
\bibitem[Hornik et~al.(1989)Hornik, Stinchcombe, and
White]{hornik1989multilayer}
Kurt Hornik, Maxwell Stinchcombe, and Halbert White.
\newblock Multilayer feedforward networks are universal approximators.
\newblock \emph{Neural networks}, 2\penalty0 (5):\penalty0 359--366, 1989.
\bibitem[Keriven \& Peyr{\'e}(2019)Keriven and Peyr{\'e}]{keriven2019universal}
Nicolas Keriven and Gabriel Peyr{\'e}.
\newblock Universal invariant and equivariant graph neural networks.
\newblock \emph{arXiv preprint arXiv:1905.04943}, 2019.
\bibitem[Kolmogorov(1956)]{kolmogorovrepresentation}
Andrey~Nikolaevich Kolmogorov.
\newblock On the representation of continuous functions of several variables by
superpositions of continuous functions of a smaller number of variables.
\newblock \emph{Proceeding of the USSR Academy of Sciences}, 108:\penalty0
176--182, 1956.
\newblock English translation: Amer. Math. Soc. Transl., 17 (1961), pp.
369–373.
\bibitem[Kondor \& Trivedi(2018)Kondor and Trivedi]{pmlr-v80-kondor18a}
Risi Kondor and Shubhendu Trivedi.
\newblock On the generalization of equivariance and convolution in neural
networks to the action of compact groups.
\newblock In Jennifer Dy and Andreas Krause (eds.), \emph{Proceedings of the
35th International Conference on Machine Learning}, volume~80 of
\emph{Proceedings of Machine Learning Research}, pp.\ 2747--2755,
Stockholmsmässan, Stockholm Sweden, 10--15 Jul 2018. PMLR.
\bibitem[K\r{u}rkov{\'a}(1992)]{kurkova1992}
V\v{e}ra K\r{u}rkov{\'a}.
\newblock Kolmogorov’s theorem and multilayer neural networks.
\newblock \emph{Neural networks}, 5(3):\penalty0 501–506, 1992.
\bibitem[K\r{u}rkov{\'a} \& Sanguineti(2002)K\r{u}rkov{\'a} and
Sanguineti]{kurkova2002comparison}
V\v{e}ra K\r{u}rkov{\'a} and Marcello Sanguineti.
\newblock Comparison of worst case errors in linear and neural network
approximation.
\newblock \emph{IEEE Transactions on Information Theory}, 48\penalty0
(1):\penalty0 264--275, 2002.
\bibitem[Maron et~al.(2019{\natexlab{a}})Maron, Ben-Hamu, Shamir, and
Lipman]{maron2018invariant}
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman.
\newblock Invariant and equivariant graph networks.
\newblock In \emph{International Conference on Learning Representations},
2019{\natexlab{a}}.
\newblock URL \url{https://openreview.net/forum?id=Syx72jC9tm}.
\bibitem[Maron et~al.(2019{\natexlab{b}})Maron, Fetaya, Segol, and
Lipman]{maron2019universality}
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman.
\newblock On the universality of invariant networks.
\newblock \emph{Proceedings of the 36th International Conference on Machine
Learning}, 97, 2019{\natexlab{b}}.
\bibitem[Qi et~al.(2017)Qi, Su, Mo, and Guibas]{qi2017pointnet}
Charles~R Qi, Hao Su, Kaichun Mo, and Leonidas~J Guibas.
\newblock Pointnet: Deep learning on point sets for 3d classification and
segmentation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pp.\ 652--660, 2017.
\bibitem[Risi~Kondor(2018)]{kondor2018covariant}
Horace Pan Shubhendu Trivedi Brandon~Anderson Risi~Kondor, Hy Truong~Son.
\newblock Covariant compositional networks for learning graphs, 2018.
\newblock URL \url{https://openreview.net/forum?id=S1TgE7WR-}.
\bibitem[Silver et~al.(2017)Silver, Schrittwieser, Simonyan, Antonoglou, Huang,
Guez, Hubert, Baker, Lai, Bolton, et~al.]{silver2017mastering}
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja
Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton,
et~al.
\newblock Mastering the game of go without human knowledge.
\newblock \emph{Nature}, 550\penalty0 (7676):\penalty0 354, 2017.
\bibitem[{Sonoda} \& {Murata}(2017){Sonoda} and {Murata}]{sonoda2017neural}
Sho {Sonoda} and Noboru {Murata}.
\newblock Neural network with unbounded activation functions is universal
approximator.
\newblock \emph{Applied and Computational Harmonic Analysis}, 43\penalty0
(2):\penalty0 233--268, 2017.
\bibitem[Wan et~al.(2013)Wan, Zeiler, Zhang, Le~Cun, and
Fergus]{wan2013regularization}
Li~Wan, Matthew Zeiler, Sixin Zhang, Yann Le~Cun, and Rob Fergus.
\newblock Regularization of neural networks using dropconnect.
\newblock In \emph{International conference on machine learning}, pp.\
1058--1066, 2013.
\bibitem[Yarotsky(2018)]{yarotsky2018universal}
Dmitry Yarotsky.
\newblock Universal approximations of invariant maps by neural networks.
\newblock \emph{arXiv preprint arXiv:1804.10306}, 2018.
\newblock URL: \url{https://arxiv.org/abs/1804.10306}.
\bibitem[Zaheer et~al.(2017)Zaheer, Kottur, Ravanbakhsh, Poczos, Salakhutdinov,
and Smola]{deepsets}
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan~R
Salakhutdinov, and Alexander~J Smola.
\newblock Deep sets.
\newblock In \emph{Advances in neural information processing systems}, pp.\
3391--3401, 2017.
\end{thebibliography}