On the Expressive Power of Sparse Geometric MPNNs
Extracted source text · arXiv 2407.02025
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
\iclrfinalcopy
\input{imports} % Import common packages
\usepackage{authblk}
\usepackage{hyperref}
\usepackage{url}
\usepackage[symbol]{footmisc}
\definecolor{myColor}{RGB}{255, 0, 0} % Define a custom color
\title{On the Expressive Power of Sparse Geometric MPNNs}
\author{\raggedright
\textbf{Yonatan Sverdlov\textsuperscript{1}}, \textbf{Nadav Dym\textsuperscript{1,2}
}}
\begin{document}
\maketitle
\vspace{-2em} % Reduce space after title
%\noindent
\textsuperscript{1} Faculty of Mathematics \\
\textsuperscript{2} Faculty of Computer Science \\
Technion -- Israel Institute of Technology \\
\texttt{yonatans@campus.technion.ac.il} \\ \texttt{nadavdym@technion.ac.il}
\iclrfinalcopy
\begin{abstract}
Motivated by applications in chemistry and other sciences, we study the expressive power of message-passing neural networks for geometric graphs, whose node features correspond to 3-dimensional positions. Recent work has shown that such models can separate \emph{generic} pairs of non-isomorphic geometric graphs, though they may fail to separate some rare and complicated instances. However, these results assume a fully connected graph, where each node possesses complete knowledge of all other nodes. In contrast, often, in application, every node only possesses knowledge of a small number of nearest neighbors.
This paper shows that generic pairs of non-isomorphic geometric graphs can be separated by message-passing networks with rotation equivariant features as long as the underlying graph is connected. When only invariant intermediate features are allowed, generic separation is guaranteed for generically globally rigid graphs. We introduce a simple architecture, $\us$, which achieves our theoretical guarantees and compares favorably with alternative architectures on synthetic and chemical benchmarks. Our code is available at
\href{https://github.com/yonatansverdlov/E-GenNet}{GitHub}.
\end{abstract}
\section{Introduction}
Geometric graphs are graphs whose nodes are a 'position' vector in $\RR^d$ (typically $d=3$) and whose symmetries include the permutation symmetries of combinatorial graphs and translation and rotation of the node positions. Geometric graphs arise naturally in learning applications for chemistry, physical dynamics, and computer vision as natural models for molecules, particle systems, and 3D point clouds. These applications motivated the introduction of many learning models for geometric graphs, which were often inspired by 'standard' \emph{graph neural networks} (GNNs) for combinatorial graphs \citep{egnn,han2024survey,gemnet}. Subsequently, several theoretical works aimed primarily at understanding the expressive power and limitations of GNNs for geometric graphs.
GNNs for geometric graphs produce global graph features invariant to geometric graph symmetries. The expressive power of GNNs is typically assessed primarily by their ability to assign different global features to pairs of geometric graphs that are not geometrically isomorphic.
Recent research on this problem has uncovered several interesting results, mainly assuming that the graphs are full (an edge connects each pair of nodes). Under the full graph assumptions, several models for geometric GNNs are \emph{complete}, that is, capable of separating \emph{all} pairs of non-isomorphic graphs.
However, this typically comes at a relatively high computational price. Examples of complete models include $(d-1)$-WL-based GNNs \citep{three_iterations,gramnet,disgnn}, GNNs based on sub-graph aggregation \citep{Invariant_models}, and message passing models based on arbitrarily high-dimensional irreducible representations \citep{TFN,gemnet,dym2020universality,finkelshtein2022simple}.
This paper focuses on more efficient geometric GNNs: message-passing neural networks (MPNN) based on simple invariant and equivariant features. In \citep{gcnn,disgnn}, it was shown that, even under the full graph assumption, invariant MPNN-based networks are not complete. In contrast, \citet{Invariant_models} and \citet{gramnet} proved that these models are 'generically complete,' which means that, under the full graph assumption, they are capable of separating most geometric graphs and only fail on a Lebesgue measure zero set of corner cases, which are difficult to separate.
In practical applications, the full graph assumption is often not fulfilled: to improve computational efficiency, geometric GNNs often operate on sparse graphs, where each node is only connected to a few nearest neighbors.
Accordingly, the first question we address in this paper is:
\textbf{Question 1.} For which graphs, other than the full graphs, are geometric message-passing neural networks generically complete?
Our answer to this question is inspired by the results of \cite{on_the_expressive_power}. They divided Message-passing-based networks for geometric graphs into \emph{Equivariant Geometric Graph Neural Networks} (E-GGNNs), which use rotation \emph{equivariant} hidden features, and \emph{Invariant Geometric Graph Neural Networks} (I-GGNNs), which use rotation \emph{invariant} hidden features, and showed that E-GGNNs have substantial advantages over I-GGNNs in terms of their expressive power.
Our answer to Question 1 is also strongly related to this dichotomy. We prove in \autoref{thm:ggnn} that E-GGNNs are generically complete for a given graph if and only if the graph is connected. This is visualized in Figure \ref{fig:teaser}. E-GGNNs will not be able to differentiate between the pair of geometric graphs in Figure \ref{fig:teaser}(A) as the underlying graph is disconnected but will be able to differentiate between the pairs in (B) and (C).
\begin{figure}[t]
\centering
\includegraphics[width=13.0cm]{figs/example_pairs.png}
\caption{(A) and (B) are pairs of non-isomorphic geometric graphs with the same distances along edges. I-GGNN cannot distinguish such pairs, while E-GGNN can (generically) if the graph is connected, as in (B). Subplot (C) depicts an example of globally rigid graphs, where non-isomorphic geometric graphs do not share the same distances across edges (in the figure, the edge between pink and light green). Such examples can be separated by both I-GGNN and E-GGNN.}
\centering
\label{fig:teaser}
\end{figure}
In contrast, I-GGNNs will not be able to differentiate between the non-isomorphic pairs in Figure \ref{fig:teaser}(A)-(B), as the distances across all graph edges are the same. On the other hand, in (C), the distances along graph edges determine the positions of all nodes up to rotation and translation. In rigidity theory, such graphs are called 'generically globally rigid.' In \autoref{thm:iggnn}, we prove that I-GGNNs are generically complete if and only if the underlying graph is generically globally rigid. These two theorems give a complete answer to Question 1.
% Here figs.
To guarantee generic separation, our theorems make the standard assumption \citep{on_the_expressive_power} that the I-GGNN and E-GGNN we consider are maximally expressive, meaning that all aggregation and readout functions used throughout the networks are injective. Recent work has shown that relatively standard and simple I-GGNN can fulfill this assumption \citep{Neural_Injective}. However, for E-GGNN, it is unclear how this assumption can be fulfilled. Accordingly, our second question is
\textbf{Question 2.} Can we devise a simple E-GGNN architecture that is maximally expressive hence, generically complete on connected graphs?
We answer this question affirmatively by devising a simple architecture, which we name $\us$, and proving in \autoref{thm:maximally} that it is maximally expressive (for generic graphs). This architecture is based on a simple message-passing architecture that resembles EGNN \citep{egnn} but uses multiple equivariant channels as in \citep{levy2023using}.
We experimentally find that $\us$ is highly successful in synthetic separation tasks and on several learning tasks involving medium-sized molecules, with a performance comparable to, or better than, state-of-the-art methods. These results indicate that simple, generically complete methods like $\us$ are sufficient for the tasks we considered. However, we acknowledge that there may be other tasks, perhaps involving molecules with non-trivial symmetry groups, where more complex geometric graph neural networks that offer full separation could be beneficial.
\paragraph{Contributions} To summarize, our main results are
\begin{itemize}
\item Showing generic separation of E-GGNNs depends on graph connectivity.
\item Showing generic separation of I-GGNNs depends on graph rigidity.
\item Proposing a simple E-GGNN architecture that is provably (generically) maximally expressive and performs well in practical tasks.
\end{itemize}
\paragraph{Appendix} See the appendix for all full proofs and related work.
\section{Setup: Geometric Graph Neural Networks and Rigidity Theory}
\textbf{Geometric Graphs. }
In our discussion of geometric graphs and GNNs, we loosely follow the definitions of \cite{on_the_expressive_power}. We define a geometric graph as a pair $\G = (\mA, \mX)$. The matrix $\mA \in \{0,1 \}^{n \times n}$ is an adjacency matrix, which fully defines a combinatorial graph. The neighborhood of a node $i$ is denoted by $\N_{i}$. Accordingly, we sometimes use statements like 'the graph $\mA$.' We assume for simplicity that the graph nodes are the integers $1,\ldots,n$. The matrix $\mX=(\vx_1,\ldots,\vx_n)\in \RR^{n \times d}$ denotes node positions.
We denote the set of all geometric graphs with coordinates in $\mathbb{R}^{d}$ by $\Gd$, and the subset of graphs with $\leq N$ nodes by $\GdN$. In Appendix \ref{app:features}, we discuss incorporating additional node and edge features into our model and theoretical results.
The symmetry group of geometric graphs is the product of the permutation group $S_n$ and the group of rigid motions $\mathcal{E}(d)$. The action of a permutation matrix $\mP$ on a geometric graph is given by $\mP \gG := (\mP\mA\mP^\top, \mP\mX)$. A rigid motion in $\mathcal{E}(d)$ is a rotation $\mQ \in \O(d)$ and a translation vector $\vt \in \RR^d$. They act on $\mX$ by applying the rotation and translation to all its coordinates, $\vx_i\mapsto \mQ \vx_i+\vt$.
Two geometric graphs $\G$ and $\H$ are \emph{geometrically isomorphic} if there exists a transformation in $S_n \times \mathcal{E}(d) $ which can be applied to $\mathcal{G}$ to obtain $\mathcal{H}$. We will say that a function $F$ defined on $\Gd$ is \emph{invariant} if $F(\G)=F(\H) $ for all $\G$ and $\H$ which are geometrically isomorphic.
The expressive power of an invariant model $F$ is closely related to its ability to separate pairs $\G,\H$ that are not geometrically isomorphic:
clearly, if $F$ cannot separate $\G$ from $\H$, then
$F$ will not be a good approximation for functions $f$ for which $f(\G)\neq f(\H)$. Conversely, if $F$ can separate all pairs of geometric graphs, then it can be used to approximate all continuous invariant \citep{gortler_and_dym,gramnet} and equivariant \citep{villar2021scalars,hordan2} functions. In Appendix \ref{sec:approx}, we discuss the connection between approximation and separation in the context of our results.
Constructing an \( F \) which can separate all geometric graphs is at least as challenging as distinguishing all combinatorial graphs (by setting all node features to zero). Without any assumptions on the point cloud, achieving this is infeasible. To address this, we employ the concept of generic geometric graphs which originates from rigidity theory.
\paragraph{Generic separation and Rigidity Theory}
As discussed in the introduction, our paper focuses on generic separation. The results we present in this section are primarily influenced by rigidity theory, so we also use this field's common notion of 'generic'.
We will say that $\mX\in \RR^{d\times n}$ is \emph{generic} if it is not the root of any multivariate polynomial $p:\RR^{d\times n} \to \RR$ with rational coefficients. The genericity assumption guarantees that $\mX$ avoids many degeneracies, making separation easier. For example, for generic $\mX$ all pairs of points have different norms, and every $d$ tuple of points $\vx_{i_1},\ldots,\vx_{i_d}$ is full rank because $\|\vx_i\|^2-\|\vx_j\|^2 $ and $\det[\vx_{i_1},\ldots,\vx_{i_d}] $ are polynomials with natural coefficients.
We note that Lebesgue almost all point clouds in $\RR^{d\times n}$ are generic \citep{mityagin2020zero}.
A globally rigid graph is a central topic in rigidity theory:
\begin{definition}[Globally rigid]( \cite{reconstructible}) A geometric graph $\mathcal{G} = (\mA,\mX)$ is \emph{globally rigid} in $\RR^d$, if the only $\mX' \in \mathbb{R}^{n \times d}$ which satisfy that $\| \vx_i-\vx_j\|=\|\vx_i'-\vx_j'\|$ for all edges $(i,j)$ in $\mA$, are those that are related to $\mX$ by a rigid motion.\\
A combinatorial graph $\mA$ is \emph{generically globally rigid} in $\RR^d$, if the geometric graph $\mathcal{G} = (\mA,\mX)$ is \emph{globally rigid} in $\RR^d$ for every generic $\mX\in \mathbb{R}^{n \times d}$.
\end{definition}
The geometric graphs in \autoref{fig:teaser} (A)-(B) are not globally rigid, as they have the same distances along edges but are not related by a rigid motion. The underlying combinatorial graph, disconnected in (A) or line graph in (B), is not generically globally rigid.
In contrast, the underlying combinatorial graph in (C) is generically globally rigid. Another simple example of a generically globally rigid graph is the full graph. Additional non-trivial examples can be found in \citep{power-graph,power-graph2}.
A necessary condition for generic global rigidity of a graph with $n \geq d + 2$ nodes in $\mathbb{R}^{d}$ is to be $d+1$ connected \footnotemark, which explains why the graphs in (A)-(B) are not generically globally rigid.
\footnotetext{We say a graph $G$ is $k$ connected if the graph remains connected after removing any $k$ vertices.}
Note that global rigidity definitions focus on reconstructing $\mX$ when $\mA$ is known. In the study of GGNNs, we are interested in identifying both $\mX$ and $\mA$:
\begin{definition}
Let $d$ be a natural number. Following \cite{Invariant_models}, we will say that an invariant function $F$ defined on $\Gd$, \emph{identifies} a geometric graph $\G$, if for every $\hat \G \in \Gd$, we have that $F(\G)=F(\hat{\G})$ if and only if $\G$ and $\hat \G$ are geometrically isomorphic.\\
We say that $F$ \emph{generically identifies} a \emph{combinatorial graph} $\mA$ in $\mathbb{R}^{d}$, if $F$ can identify $\G=(\mA,\mX)$ for every generic $\mX\in \mathbb{R}^{n \times d}$. \\
We say that $F$ \emph{generically fails to identify} a \emph{combinatorial graph} $\mA$ in $\mathbb{R}^{d}$, if $F$ does not identify $\G=(\mA,\mX)$ for every generic $\mX \in \mathbb{R}^{n \times d}$.
\end{definition}
Note that the fact that $F$ does not generically identify $\mA$ only guarantees that there exists a generic $\mX$ so that $F$ fails to identify $(\mA,\mX)$. The statement ‘$F$ generically fails to identify' is, apriori, significantly stronger. Also note that invariant $F$ separates all non-isomorphic graphs (is \emph{complete}) if and only if it identifies all graphs.
As discussed in the introduction, our goal will be to classify which graphs are generically identifiable by standard message-passing-based networks for geometric graphs, which can be classified as either I-GGNN or E-GGNN. We will now define these concepts.
\paragraph{E-GGNN}
In the context of combinatorial graphs (with no geometric information), \citet{gilmer2017neural,jogl2023expressivitypreserving} and \citet{velivckovic2022message} showed that many famous graph neural networks are instantiations of MPNNs. The goal of \cite{on_the_expressive_power} was to show, analogously, that many neural networks for \emph{geometric} graphs follow a generalized MPNN framework. Loosely following \cite{on_the_expressive_power}, we define \emph{Equivariant Geometric Graph Neural Networks (E-GGNN)} to be functions defined by a sequence of layers that propagates vector features from iteration $t$ to $t+1$ via a learnable function $f_{(t)}$ (at initialization we set $v_j^{(0)}=0$).
\begin{equation}
\label{eq:GGNN}
\vv_i^{(t+1)}=f_{(t)} \left( \lms \vv_j^{(t)}, \vx_i-\vx_j , \mid j \in \mathcal{N}_i \rms \right).
\end{equation}
To ensure the equivariance of the construction, the function $f_{(t)}$ is required to be rotation equivariant. Translation invariance is implicitly guaranteed by the translation invariance of $\vx_{i}-\vx_j$, and the requirement that $f_{(t)}$ is defined on multi-sets implicitly enforces permutation equivariance. We will often refer to $f_{(t)}$ as an \emph{aggregation function}.
We note that for the sake of simplicity, we don't require a \emph{combine} step that incorporates $\vv_i^{(t)}$ in the process of computing $\vv_i^{(t+1)} $ since we find that this step has little influence on our theoretical and empirical results.
We also note that the vector $\vv_i^{(t)}$ is not necessarily $d$ dimensional, and the action of $O(d)$ on $\vv_i^{(t)}$ is allowed to change from layer to layer. For example, in the $\us$ architecture we will introduce later on, each $\vv_i^{(t)}$ will be a $d\times m$ matrix, and the action of $O(d)$ will be multiplication on each of the $m$ coordinates.
After $T$ iterations of E-GGNNs, a permutation, rotation, and translation \emph{invariant} feature vector can be obtained via a two-step process involving a rotation-invariant function $\finv$ and a multi-set \textsc{ReadOut} function:
\begin{equation}
\vs_i =\finv(\vvv_i^{(T)}), \quad
\sglobal = \textsc{ReadOut}\lms\vs_{1},..,\vs_{n}\rms
\label{eq:readout}
\end{equation}
As shown by \cite{on_the_expressive_power}, many popular geometric graph neural networks can be seen as instances of E-GGNNs with a specific choice of functions $f_{(t)}$ and readout functions. Examples include EGNN \citep{egnn}, TFN \citep{TFN}, GVP \citep{gvp-egnn}, and MACE \citep{mace}.
In Section \ref{architecture}, we present $\us$, our instantiation of this framework.
\paragraph{I-GGNNs}
An important subset of E-GGNNs are \emph{invariant} GGNNs (I-GGNNs). These networks only maintain scalar features $\vs_i^{(t)}$ and replace $\vx_i-\vx_j$ with its norm to obtain
\begin{equation}\label{eq:IGGNN} \vs_i^{(t+1)}=f_{(t)} \left( \lms \vs_j^{(t)}, \|\vx_i-\vx_j\| , \mid j \in \mathcal{N}_i \rms \right)
\end{equation}
A global invariant feature is obtained via a permutation invariant readout function as in \eqref{eq:readout}.
An advantage of I-GGNNs is that there are no rotation equivariance constraints on $f_{(t)}$. Therefore, any standard message-passing neural network that can process continuous edge weights can be employed as an I-GGNN by setting the edge weights to $\|{\vx}_{ij}\|$. Examples of I-GGNNs in the literature include a variant of EGNN used for invariant tasks \citep{egnn}, SchNet \citep{SchNet}, DimeNet \citep{DimeNet} and SphereNet \citep{SphereNet}\footnotemark.
Our definition of I-GGNN follows the definition from \citep{gcnn,disgnn}.
In Appendix \ref{I-GGNN types}, we explain that the definition of I-GGNN \cite{on_the_expressive_power} has a seemingly minor difference that renders it significantly stronger.
\footnotetext{\label{invariant models} DimeNet \citep{DimeNet} and SphereNet \citep{SphereNet} have access to the power graph $\G^2$ (defined later in the paper) and thus are invariant models on $\G^{2}$ and have potential expressive power stronger than invariant I-GGNN on $\G$.}
\paragraph{Maximally expressive E-GGNN and I-GGNNs}
The separation power of E-GGNN and I-GGNN architectures depends on the specific instance used. For example, if we choose the final readout function as the zero function, all geometric graphs will be assigned the same value. To address this, it is customary to assume that all functions $f_{(t)}$, as well as $\finv$ and the readout function, are injective, up to the ambiguities these functions have by construction. Following \cite{on_the_expressive_power}, we will call E-GGNN (respectively I-GGNN) architectures, which are composed of injective functions, maximally expressive E-GGNN (respectively I-GGNN) architectures.
We note that the existence of the injective functions necessary for maximal expressivity follows from set-theoretic considerations, similar to those used by \cite{wang2024rethinking}. The question of whether practical differentiable aggregation functions are injective is discussed in Section \ref{sec:maximally}.
\section{Expressive power of I-GGNN}
\label{sec:inv}
Our next goal is to analyze the separation abilities of I-GGNN.
\begin{restatable}{theorem}{IGGNN}
[expressive power of I-GGNN]
\label{thm:iggnn}
Let $d$ be a natural number. Let $F$ be an I-GGNN. Let $\mA$ be a graph not generically globally rigid on $\RR^d$. Then, $F$ generically fails to identify $\mA$. \\ Conversely, if $\mA$ is generically globally rigid on $\RR^d$ and $F$ is a maximally expressive I-GGNN with depth $T=1$, then $F$ generically identifies $\mA$.
\end{restatable}
\begin{proof}[Proof idea]
if $\mA$ is not globally rigid, then according to \cite{gortler2010characterizing}, for \emph{every} generic $\mX$ there exists $\mX'$ for which distances along edges are preserved, but $\mX$, $\mX'$ are not related by a rigid motion.
It's easy to see any I-GGNN, which is based only on distances along edges, can't separate such graphs.
On the other hand, generic geometric graphs
have distinct pairwise distances. The full proof (see Appendix \ref{Proof of I-GGNN}) shows that this can be used to reconstruct the combinatorial graph $\mA$. By global rigidity, we can then reconstruct $\mX$, up to rigid motion.
\end{proof}
Note that \autoref{thm:iggnn} is a generalization of \cite{gramnet} that showed maximally expressive I-GGNN generically identify full graphs. We now showed this is true for all generically globally rigid graphs.
\paragraph{I-GGNN generic separation by power graph preprocessing}
A natural strategy to overcome the lack of generic separation for graphs that are not generically globally rigid, as proven in Theorem \ref{thm:iggnn}, is to replace these graphs with graphs that \emph{are} generically globally rigid. This replacement can be done using power graphs:
For a given graph $\mathcal{G}$ and natural number $k$, the power graph $\mathcal{G}^{k}$ is defined to have the same nodes as the original graph $\mathcal{G}$. A pair of nodes are connected by an edge in the power graph if and only if there is a path between them in the original graph of length $\leq k$.
In \citep{power-graph, power-graph2}, it was shown that if the original graph $\G$ is connected, then the power graph $\G^{d+1}$ is generically globally rigid in $\mathbb{R}^{d}$. Accordingly, when the input graph $\G$ is connected but not generically globally rigid, our theory suggests that applying I-GGNN not to the original graph but to its $d+1$ power may benefit separation. An experiment illustrating this idea is presented in \autoref{tab:power_graph}.
\section{Expressive power of E-GGNN}
When using E-GGNN rather than I-GGNN, we will have generic separation if and only if the graph is connected.
\begin{restatable}{theorem}{GGNN}
[expressive power of E-GGNN]\label{thm:ggnn}
Let $d$ be a natural number. Let $F$ be an E-GGNN. Let $\mA$ be a disconnected graph. Then, $F$ generically fails to identify $\mA$.\\ Conversely, if $\mA$ is connected and $F$ is a maximally expressive E-GGNN with depth $T\geq d+1$, then $F$ generically identifies $\mA$.
\end{restatable}
\begin{proof}[Proof idea]
Suppose a geometric graph $\mathcal{G}$ isn't connected. We can apply two distinct rigid motions to two connected components and obtain a new geometric graph, $\mathcal{G}'$, which cannot be separated from $\mathcal{G}$ using E-GGNN. This is illustrated in \autoref{fig:teaser}(A).
Conversely, in the appendix \autoref{Identifiable}, we show that the features $s_i$ obtained after $d+1$ iterations of a maximally expressive E-GGNN can encode the multi-set of all distances $\|x_i-x_j\| $ for all $j$ which are $d+1$-hop neighbors of $i$ (a similar claim was shown by \cite{on_the_expressive_power}). Thus, $d+1$ iterations of maximally expressive E-GGNN can simulate the application of a single iteration of an I-GGNN on the $d+1$ power graph. The proof then follows from the fact that the $(d+1)$ power graph is generically globally rigid \citep{power-graph, power-graph2}.
\end{proof}
An immediate result of this theorem is that generic graph separation is a local problem with a problem radius of $d+1$ \citep{alon2020bottleneck} .\footnote{A task's problem radius denoted by $r$ is the minimal integer where the task becomes solvable upon each node observing its neighborhood within a $r$-hop radius and aggregating all node's information.} We note that the generic assumption allows us to reconstruct the coordinates of the nodes and the combinatorial graph. Without the generic assumption, even models based on $2$-WL that are complete on full graphs cannot reconstruct the combinatorial graph. Setting all positions to zero nullifies the geometric information, and $2$-WL is insufficient to distinguish all combinatorial graphs.
\section{Building a generically maximally expressive E-GGNN}\label{sec:maximally}
To obtain generic separation guarantees as in Theorems \ref{thm:iggnn}, \ref{thm:ggnn} for a practical instantiation of E-GGNN (or I-GGNN), we will need to
show that the instantiation is fully expressive. This type of question was studied rather extensively for combinatorial graphs \citep{Neural_Injective,aamand2022exponentially}, and these results can be used to show that some simple realizations of I-GGNN are maximally expressive. We discuss this in more detail in Appendix \ref{building i-ggnn}.
For E-GGNN, the requirement that $f_{(t)}$ is simultaneously injective (up to permutation) and equivariant to rotations seems to be challenging to attain with a simple E-GGNN construction, and indeed, \cite{on_the_expressive_power} who suggested the notions of maximally expressive E-GGNN, did not give any indication of how they could be constructed in practice. Moreover, in Appendix \ref{maximally_expressive_egnn}, we show that a maximally expressive E-GGNN can separate all pairs of full graphs and separate geometrically all pairs of connected graphs\footnote{We define the geometric separation of a geometric graph if we can separate the point clouds up to group action, without necessarily reconstructing the combinatorial graph.}, indicating constructing such a network may be computationally expensive, as currently, such guarantees are only achieved by networks of relatively high complexity based on $2$-WL subgraph aggregation techniques.
In our setting, however, the problem is less severe since, from the outset, we are only interested in analyzing network behavior on generic graphs. Accordingly, we would like to require our E-GGNN to be maximally expressive only for generic graphs. It turns out that this goal can be achieved with a straightforward E-GGNN architecture, which we name $\us$. Our next step will be to present this architecture.
\subsection{The $\us$ architecture}
\label{architecture}
Our goal in this section is to build a maximally expressive EGNN. For this, we need to implement injective readout and aggregation functions $f_{(t)},\finv, \textsc{ReadOut}$ (see equations \ref{eq:GGNN}-\ref{eq:readout}.
The $\us$ architecture we suggest resembles an EGNN architecture \citep{egnn} with multiple equivariant channels as in \citep{levy2023using}.
The $\us$ architecture depends on two hyper-parameters: depth $T$ (the term ‘depth’ refers to the number of MPNN iterations) and channel number $C$. The input to the architecture is a geometric graph $(\mA,\mX)$.
\paragraph{Aggregation}
The architectures then maintains, at each iteration $t$ and node $i$, an 'equivariant' node feature with $C$ channels
\begin{equation}
\vv^{(t)}_i=(v^{(t)}_{i,1},\ldots,v^{(t)}_{i,C}), \quad v^{(t)}_{i,c} \in \RR^{d}, c=1,\ldots,C
\end{equation}
These feature values are initialized to zero and are updated via the rotation and permutation equivariant aggregation formula
\begin{equation}
v^{(t+1)}_{i,q}=\sum_{j\in \gN_i}\left( \phi^{(t,q,0)}(\|\vx_i-\vx_j\|,\|\vv_j\|)(\vx_i-\vx_j)+\sum_{c=1}^C \phi^{(t,q,c)}(\|\vx_i-\vx_j\|,\|\vv_j\|)v_{jc}^{(t)} \right)
\end{equation}
where $\|\vv_j\|$ is the $C$ dimensional vector containing the norms of the vectors $v_{jc}, c=1,\ldots,C$.
\paragraph{Construction of $\finv$ and \textsc{ReadOut}}
After $T$ iterations, we obtain rotation and permutation invariant node features $\vs_i\in \RR^C$ and global feature $\sglobal \in \RR^C $ via
\begin{align}
s_{i,q}=\|\sum_{c=1}^C \theta_{c,q}v^{(T)}_{i,c}\|^2 , \quad
\sglobal_q=\sum_{i=1}^n \phi_{\mathrm{global}}^{(q)}(\vs_i), \quad q=1,\ldots,C
\end{align}
Where $\theta$ is a learnable matrix, and
the functions $\phi_{\mathrm{global}}^{(q)},\phi^{(t,q,c)}$ are
all fully connected neural networks with a single layer, an output dimension of $1$, and an analytic non-polynomial activation (in our implementation, TanH activation).
The complexity of $\us$ is linear in $N\cdot C$, as we discuss in Appendix \ref{complexity}.
The next theorem shows that $\us$ is generically maximally expressive: \begin{restatable}{theorem}{express}
\label{thm:maximally}
Let $d, N, T$ be natural numbers. Let $F$ be a maximally expressive E-GGNN of depth $T$, and let $F_\theta$ denote the $\us$ architecture with depth $T$ and $C=2Nd+1$ channels. Then, for Lebesgue almost every choice of network parameters $\theta$, we have that for all generic $\G,\hat \G \in \GdN$,
$$ F(\G)=F(\hat \G) \iff F_\theta(\G)=F_\theta(\hat \G) $$
\end{restatable}
\begin{proof}[Proof idea]
The proof relies on the methodology developed by \citet{gortler_and_dym, Neural_Injective} and \citet{gramnet}. We note that each step in $\us$ employs $C$ identical analytic functions indexed by $q=1,\ldots, C$, with independent parameters. \citet{gortler_and_dym} and \citet{Neural_Injective} showed that in this setting, if every pair of non-isomorphic inputs can be separated by some parameter of the original function, then $C=2Nd+1$ copies of this function with random parameters will suffice to separate \emph{all pairs} uniformly. The proof is then reduced to showing that any pair of non-isomorphic inputs can be separated by some parameter vector, which we prove in the appendix \ref{sub:maximal}.
\end{proof}
Joining this theorem with our theorem on maximally expressive E-GGNN, we deduce
\begin{corollary}\label{cor:us}
Let $d, N$ be natural numbers.
let $F_\theta$ denote the $\us$ architecture with depth $T=d+1$ and $C=2Nd+1$ channels. Then for Lebesgue almost every $\theta$, for every pair of graphs $\G, \G'\in \GdN $ which are generic and connected,
$$F_{\theta}(\G)=F_{\theta}(\hat \G) \iff \G, \hat \G \text{ are geometrically isomorphic}$$
\end{corollary}
Note that by \cite{on_the_expressive_power}, theorems 20, 22, the dimension of any injective embedding must be at least $nd-\frac{d(d-1)}{2}$, showing our embedding dimension $2nd+1$ is optimal up to factor of $2$ (when $n\gg d$).
\section{Experiments}
In the experiments section, we have three main objectives. First, we aim to demonstrate that invariant models are indeed strictly less expressive than equivariant models and that power graphs help to mitigate this gap. Second, we seek to show that our model can distinguish challenging graph examples, including those requiring multiple layers for separation, thereby avoiding the bottleneck phenomenon encountered in other geometric models. Finally, we aim to demonstrate the improved performance of our model on real-world benchmarks.
\subsection{Separation experiments}
\label{sepearion exps}
\paragraph{Enhancing Model Expressiveness: Power Graphs in Invariant vs. Equivariant Comparisons}
We begin with experiments to corroborate our theoretical findings. We consider examples where the graph $\mA$ is a line graph, which is connected but not generically globally rigid. We choose two pairs of geometric line graphs, Pair a and Pair b, depicted in Figure \ref{fig:Two pairs for power Graphs}. These pairs have identical distances along graph edges but are not isomorphic. Therefore, they cannot be distinguished by any I-GGNN but (at least when generic) can be distinguished by E-GGNN and our $\us$ architectures.
Following the protocol from \cite{on_the_expressive_power}, we construct binary classification problems from Pair a and Pair b and apply our E-GGNN architecture $\us$, as well as three popular I-GGNN architectures: SchNet \citep{SchNet}, DimeNet \citep{DimeNet}, and SphereNet \citep{SphereNet}.
The results are shown in the first column in Table \ref{tab:power_graph}. As expected, $\us$ perfectly separates pairs a and b, while SchNet fails to separate both. Note that DimeNet and SphereNet separate Pair a but not Pair b. This is because DimeNet and SphereNet compute angles at edges, which utilizes information not contained in the graph $\G$ but only in the power graph $\G^{2}$. Indeed, distances between two-hop neighbors are sufficient for discriminating between the two geometric graphs in Pair a but not in Pair b, where the distance between the three-hop red and orange nodes is needed for separation.
\begin{figure}[t]
\centering
\includegraphics[width=1.0\textwidth]{figs/three_pairs.png}
\caption{From Left to Right: $k$-chain pair graphs that are non-generic and require $\sim k/2$ blocks for separation, Pair a in which $2$-power graph is enough for I-GGNN to distinguish and Pair b in which $3$-power graph is necessary and sufficient for separation.}
\label{fig:Two pairs for power Graphs}
\end{figure}
Next, as we suggested in Section \ref{sec:inv}, we apply the same algorithms to the powers of the original graph. As shown in Table \ref{tab:power_graph}, this improves the separation capabilities of the I-GGNN models. All invariant models succeed when the third power is taken, as expected since a connected graph's $d+1$ power (here $d=2$) is generically globally rigid.
\begin{table}
\begin{center}
\caption{E-GGNN models like $\us$ can separate the two pairs of connected graph in Figure \ref{fig:Two pairs for power Graphs}. I-GGNN models cannot separate unless power graphs are considered.}
\begin{tabular}{clcccc}
\toprule
& \multicolumn{5}{c}{\textbf{Power graph}} \\
& \textbf{GNN Layer} & $1$ & $2$ & $3$ & \\
\midrule
\multirow{4}{*}{\rotatebox[origin=c]{90}{Pair a}} &
\gray{Maximal I-GGNN} & \gray{50\%} & \gray{100\%} & \gray{100\%} \\
& SchNet \cite{SchNet} & \cellcolor{red!10} 50.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} \\
& DimeNet \cite{DimeNet} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} \\
& SphereNet \cite{SphereNet} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} \\
\bottomrule
& SchNet $_{\text{full graph}}$ \footref{schnet models} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
& SchNet$_{\text{global feat}}$ \footref{schnet models} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
\bottomrule
& $\us$(us) & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
\bottomrule
\multirow{4}{*}{\rotatebox[origin=c]{90}{Pair b}} &
\gray{Maximal I-GGNN} & \gray{50\%} & \gray{50\%} & \gray{100\%} \\
& SchNet \cite{SchNet} & \cellcolor{red!10} 50.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{red!10} 50.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} \\
& DimeNet \cite{DimeNet} & \cellcolor{red!10} 50.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} \\
& SphereNet \cite{SphereNet} & \cellcolor{red!10} 50.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} & \cellcolor{green!10} 100.0 {\scriptsize {\scriptsize ± 0.0}} \\
\bottomrule
& SchNet $_{\text{full graph}}$ \footref{schnet models} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
& SchNet$_{\text{global feat}}$ \footref{schnet models} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
\bottomrule
& $\us$ (us) & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
\bottomrule
\end{tabular}
\label{tab:power_graph}
\end{center}
\end{table}
%\end{wraptable}
\paragraph{Unpacking the Bottleneck: Layer Requirements for Complex Graph Separation}
\paragraph{k-chains}
Next, we consider the $k$ chain pairs suggested in \cite{on_the_expressive_power} and illustrated in the left of Figure \ref{fig:Two pairs for power Graphs}. By design, this is a pair of graphs with $k$ nodes, which can only be separated by $k-1$ hop distances. In \cite{on_the_expressive_power}, it was shown that a depth of at least $\lfloor k/2 \rfloor+1 $ is necessary to achieve separation by E-GGNN in this setting. Note that as most nodes in the chain graphs are on the same line, this point cloud is not generic, so the fact that for large $k$, more than $d+1$ iterations are needed does not contradict our theory.
In our experiments, we include two variations of SchNet \citep{SchNet}: full graph and global feature. Those variations use all pairwise distances, thus showing better results than other invariant models.
In the first experiment, we set $k = 4$.
Theoretically, $3$ blocks are needed for propagating the information, and $2$ blocks are insufficient.
As shown in Appendix \autoref{tab:4chains}, we succeed with a probability of $1$ over $10$ trials with $3$ blocks.
For checking long-term dependencies, we choose $k=12$ again; $7$ blocks are needed, and $6$ is insufficient. We examined all models with $7,8,9,10,11$ blocks, each $10$ times. As shown in \autoref{tab:12chains}, we are the only ones succeeding with $7$ blocks, and all other models need a much higher number of blocks. This may suggest our model is less influenced by over-squashing \cite{alon2020bottleneck}. In the appendix, \autoref{tab:time_com}, we add a time comparison table for all models using the $128$ dimensional feature.
% Here fig.
\footnotetext{\label{schnet models} Here, SchNet global and full are invariant models utilizing the full graph structure (unlike other models utilizing the given combinatorial structure).}
\paragraph{Hard examples}
In \autoref{tab:hard-pc results} in the appendix, we show that our model can also perfectly separate challenging examples from \citep{PhysRevLett.125.166001,gcnn}, using the experiment setup taken from \cite{gramnet}. These geometric graph pairs cannot be separated by I-GGNN, even on full graphs, and are also difficult to separate using equivariant methods like EGNN.
\paragraph{Chemical property experiments}
Finally, we check our model's ability to learn regression-invariant real-world chemical properties, a critical task in computational chemistry and material science. Learning chemical properties accurately is essential for predicting the behaviors of compounds, enabling advancements in drug discovery, material design, and environmental science. By effectively capturing these properties, the model demonstrates its potential to assist in solving complex chemical problems, where precise property prediction can reduce experimental costs and guide targeted research.
As in \citep{baselines}, we compare to the following models: Random Forest \citep{Random_forest}, LSTM \citep{LSTM}, Transformer \citep{Transformer}, GIN \citep{GIN}, GIN+VN \citep{GIN-VN}, ChemProp \citep{ChemProp}, GraphGPS \citep{GraphGPS}, SchNet \citep{SchNet}, DimeNet++ \citep{DimeNet++}, GemNet \citep{gemnet}, PaiNN \citep{painn}, ClofNet \citep{clofnet}, LEFTNet \citep{leftnet}, and Marcel \citep{baselines}.
Our task is learning chemical property prediction on three datasets: Drugs \citep{Drugs}, Kraken \citep{Kraken}, and BDE \citep{BDE}. Drugs are the largest and most challenging dataset, containing 560K samples; BDE is a medium-sized dataset with 80K datasets; and Kraken is the smallest, with 22K samples.
The baseline model performance, data split, and training procedure are taken from \cite{baselines}. In particular, we ran each experiment three times and reported the model's test error with the best validation performance.
In Table \ref{mean} in the appendix, we report the mean and standard deviation.
\begin{table}[t]
\centering
\caption{Performance of 1D, 2D, and 3D baseline MRL models and the best results from ensemble learning strategies on 3D GNNs. The metric used is the Mean Absolute Error (MAE, \(\downarrow\)). The \textbf{bold} indicates the best-performing model, while \underline{underlined} denotes the second-best.}
\small % Reduce font size
\setlength{\tabcolsep}{4pt} % Adjust column spacing
\renewcommand{\arraystretch}{1.2} % Adjust row height
\begin{tabular}{@{}lccccccccccc@{}}
\toprule
\rowcolor{gray!25}
& & \multicolumn{3}{c}{\textbf{Drugs}} & \multicolumn{4}{c}{\textbf{Kraken}} & \multicolumn{2}{c}{\textbf{BDE}} \\
\cmidrule(lr){3-5} \cmidrule(lr){6-9} \cmidrule(lr){10-11}
\textbf{Category} & \textbf{Model} & \textbf{IP} & \textbf{EA} & $\boldsymbol{\chi}$ & $\mathbf{B_5}$ & \textbf{L} & $\mathbf{BurB_5}$ & $\mathbf{BurL}$ & \textbf{BE} \\
\midrule
\multirow{3}{*}{1D}
& Random Forest & 0.4987 & 0.4747 & 0.2732 & 0.4760 & 0.4303 & 0.2758 & 0.1521 & 3.0335 \\
& LSTM & 0.4788 & 0.4648 & 0.2505 & 0.4879 & 0.5142 & 0.2813 & 0.1924 & 2.8279 \\
& Transformer & 0.6617 & 0.5850 & 0.4073 & 0.9611 & 0.8389 & 0.4929 & 0.2781 & 10.0771 \\
\midrule
\multirow{4}{*}{2D}
& GIN & 0.4354 & 0.4169 & 0.2260 & 0.3128 & 0.4003 & 0.1719 & 0.1200 & 2.6368 \\
& GIN+VN & 0.4361 & 0.4169 & 0.2267 & 0.3567 & 0.4344 & 0.2422 & 0.1741 & 2.7417 \\
& ChemProp & 0.4595 & 0.4417 & 0.2441 & 0.4850 & 0.5452 & 0.3002 & 0.1948 & 2.6616 \\
& GraphGPS & 0.4351 & 0.4085 & 0.2212 & 0.3450 & 0.4363 & 0.2066 & 0.1500 & 2.4827 \\
\midrule
\multirow{8}{*}{3D}
& SchNet & 0.4394 & 0.4207 & 0.2243 & 0.3293 & 0.5458 & 0.2295 & 0.1861 & 2.5488 \\
& DimeNet++ & 0.4441 & 0.4233 & 0.2436 & 0.3510 & 0.4174 & 0.2097 & 0.1526 & \underline{1.4503} \\
& GemNet & 0.4069 & 0.3922 & \textbf{0.1970} & 0.2789 & 0.3754 & 0.1782 & 0.1635 & 1.6530 \\
& PaiNN & 0.4505 & 0.4495 & 0.2324 & 0.3443 & 0.4471 & 0.2395 & 0.1673 & 2.1261 \\
& ClofNet & 0.4393 & 0.4251 & 0.2378 & 0.4873 & 0.6417 & 0.2884 & 0.2529 & 2.6057 \\
& LEFTNet & 0.4174 & 0.3964 & 0.2083 & 0.3072 & 0.4493 & 0.2176 & 0.1486 & 1.5328 \\
& Marcel & \underline{0.4066} & \underline{0.3910} & \underline{0.2027} & \underline{0.2225} & \underline{0.3386} & \underline{0.1589} & \underline{0.0947} & 1.4741 \\
& $\us$(Ours) & \textbf{0.3965} & \textbf{0.3686} & 0.2068 & \textbf{0.1878} & \textbf{0.1738} & \textbf{0.1530} & \textbf{0.0626} & \textbf{0.3170} \\
\bottomrule
\end{tabular}
\label{cpe}
\end{table}
As we see in \autoref{cpe}, we show comparable or improved results in all tasks. For example, in BDE, we have an improvement of $78\%$, in Kraken property \emph{burL} $33\%$, and in the property \emph{L} $48\%$.
Remarkably, our method outperforms competing more complex methods like GemNet, which employs high-order irreducible representations. This can be explained by the generic completeness of our method, coupled with results we will show in the appendix \autoref{tab:norms_repit}, which shows that these datasets have a small number of norm-repetitions, indicating that they are ‘close to generic’ .
\section{Limitation and Future Work}
In this work, we characterized the generic expressive power of I-GGNN and E-GGNN, showed that $\us$ is a generically maximally expressive E-GGNN, and showed the effectiveness of this architecture on chemical regression tasks. A limitation of our work is that it may struggle with symmetric and non-generic input. Future work could consider assessing the expressive power of non-MPNN models like geometric 2-WL on sparse graphs to address such input efficiently.
\section{Acknowledgments}
We thank Snir Hordan for reviewing our manuscript and for the helpful discussions and suggestions. We would like to thank Idan Tankel for his technical help and support. N.D. and Y.S. are funded by Israeli Science Foundation grant no.
272/23.
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
\newpage
\appendix
\section{Related Work}
This section discusses related work that is not discussed in the main text.
\paragraph{Equivariant models}
In the context of learning on geometric graphs, models should be equivariant to the joint action of permutations, rotations, and translations. Besides the models discussed in the paper, some prominent models with this equivariant structure include Cormorant \citep{cormorant}, Vector Neurons \citep{deng2021vector}, and SE(3) transformers \citep{se3transformer}.
\paragraph{Completeness, universality and generic completeness}
The main text discusses related works on completeness focused on methods coming from the geometric learning community. Other complete methods, generally using similar ideas, appeared independently in the more chemistry-oriented community\citep{kurlin2023polynomial,widdowson2023recognizing,shapeev,nigam2020recursive}. Completeness in $2D$ was achieved in \cite{zz}, and completeness with respect to rigid motions (without permutations) was achieved by Comenet \citep{comenet}. Comenet also uses some genericity assumptions (e.g., that there is a well-defined notion of nearest neighbors).
The main text discussed how generic completeness can be achieved using I-GGNN when the graph is globally rigid. A similar result can be obtained when considering the list of all distances \citep{reconstructible}. Generic completeness for full graphs can also be obtained using more complex distance-based features \citep{widdowson2022generic} or shape principal axes \citep{puny2021frame} (under the generic assumption that the principal eigenvalues are distinct).
\paragraph{Comparison with \cite{new}}
Generic completeness without the full graph assumption, employing only the assumption that local neighborhoods are not symmetric, is discussed by \cite{new}. However, there are notable differences in key aspects of the results. Specifically, Theorem 4.2 in \cite{new} demonstrates that their proposed equivariant model can identify generic molecules using \( L \) message-passing iterations, where \( L \) is the graph's diameter. In contrast, our Theorem \ref{thm:ggnn} requires only \( d+1 \) iterations (where \( d = 3 \) in typical applications, resulting in a total of just four iterations).
Moreover, while Theorem 4.2 in \cite{new} focuses on recovering the 3D structure, it does not address the combinatorial structure. Our Theorem \ref{thm:ggnn}, however, guarantees the recovery of both combinatorial and geometric structures in the generic case.
Additionally, our Theorem \ref{thm:iggnn} precisely describes the conditions under which invariant models are, and are not, generically complete—a result not covered in \cite{new}.
\section{Implementation details}
Here, we detail the hyper-parameter choice.
For all experiments, we use AdamW optimizer \cite{AdamW}. We use a varying weight decay of $1e^{-4},1e^{-5}$ and learning rate of $1e^{-4}$. Reduce On Platue scheduler is used with a rate decrease of 0.75 with the patience of 15 epochs.
We use two blocks with $40$ channels in the challenging point cloud separation experiments.
For \emph{k-chain} experiments, we repeat each experiment $10$ times, with $10$ different seeds, and set the number of channels to be $60$, and a varying number of blocks, as detailed in the table.
For power graph experiments, we repeat each experiment $10$ times, with $10$ different seeds, and set the number of channels to be $60$ and $3$ blocks.
For chemical property experiments, all the procedure is taken from \cite{baselines}: each dataset is partitioned randomly into three subsets: 70\% for training, 10\% for validation, and
20\% for test. We set the number of channels to be $256$ and used $6$ blocks. Each model is trained over $1,500$ epochs. Experiments are repeated three times for all eight regression targets, and the results reported correspond to the model that performs best on the validation set in terms of Mean Absolute Error (MAE). In contrast to the experiments in \cite{baselines}, we didn't consider the EE dataset as it's currently not public on GitHub.
We utilize PyTorch and PyTorch-Geometric to implement all deep learning models. All
experiments are conducted on servers with NVIDIA A$40$ GPUs, each with 48GB of memory. All dataset details, including statistics, can be found in \cite{baselines}.
\section{Building perfect I-GGNN}
\label{building i-ggnn}
Here, we describe in detail how to construct a perfect I-GGNN. We will assume that we consider geometric graphs with up to $N$ nodes. The main ingredients needed are aggregation functions and \textsc{ReadOut} function that are injective on the space of all multisets of size $\leq N$. \cite{Neural_Injective} proved that the sum of projections and analytic activation can be used to construct such layers. Formally, be $\M$ semi-analytic set of dimension $D$, and $N$ vectors in $\mathbb{R}^{d}$ denoted by $\mX$. Be $A\in \mathbb{R}^{(2D+1) \times d}, b\in \mathbb{R}^{2D+1}$, $\sigma:\mathbb{R}\rightarrow \mathbb{R}$ analytic non-polynomial activation, then for Lebesgue almost every $A,b$ $$f_{A,b}(\mX) = \sum^{N}_{i=1} \sigma(A\cdot \mX_{i}+b)$$ is injective up to permutation.
Thus, the composition of such blocks and such \textsc{ReadOut} function with dimension $D = d\cdot N$ yields a maximally expressive model on all input domain. Here, $D$ is the intrinsic dimension of all features, ultimately produced by the $D$ dimensional input space.
\section{Separation Power of Fully Maximally expressive E-GGNN }
\label{maximally_expressive_egnn}
In the main text, we proposed $\us$ a simple \emph{generically} maximal expressive E-GGNN. In this section, we show that if we have a maximally expressive E-GGNN on all input domain, we can reconstruct a full graph up to group action, and if the graph is connected, we can reconstruct its point cloud. This indicates that fully maximally expressive E-GGNN is as challenging as achieving completeness, which generally requires complex methods like $2$-WL-based GNNs.
\begin{theorem}
\label{reconstruction}
Let $\G = (\mA,\mX)$ be a geometric graph, where $\mA$ is the full graph on $n$ nodes, and $\mX$ is an arbitrary $d\times n$ matrix, then one iteration of maximally expressive E-GGNN can reconstruct $\mX$.
\end{theorem}
\begin{proof}
Assume we run one iteration of E-GGNN on our geometric graph, then for each node $i$, we know up to some rotation $Q_{i}$
\begin{align*}
\lms Q_{i}\cdot (x_{i} - x_{j}) | j \in [n] \rms
\end{align*}
Choosing $i=1$, and $x_{i}=0$, we have all other positions $Q_{i} \cdot x_{j},j\neq i$, thus up to translation (as we set $x_{i}$), rotation $Q_{i}$ and node permutation, we can reconstruct our point cloud.
\end{proof}
We now show that using $n$ iterations of E-GGNN can reconstruct the point cloud of each connected graph.
\begin{theorem}
Assuming a connected geometric graph $\G = (\mA,\mX)$ with $n$ nodes, then $n$ iteration of maximally expressive E-GGNN can reconstruct the point cloud.
\end{theorem}
\begin{proof}
Assume we run $n$ iterations of a maximally expressive E-GGNN on our connected graph. Then, by \autoref{thm:ggnn}, we can reconstruct for each node $i$ its $n$-hop up to rotation $Q_{i}$. As the $n$-hop of a node is all other nodes (the graph's diameter is at most $n$), we reconstruct the full geometric graph up to rotation, and by theorem \ref{reconstruction}, we can reconstruct the point cloud.
\end{proof}
\section{I-GGNN+}
\label{I-GGNN types}
Here, we present the definition of I-GGNN presented by \cite{on_the_expressive_power}, denoted by I-GGNN+, and show that it can separate all geometric full graphs. After, we explain why this model is strictly stronger than our I-GGNN.
\cite{on_the_expressive_power} defines (neglecting the update step also used there)
\label{I-GGNN+ types}
\begin{equation}
\vv_i^{(t+1)}=f^{t}\left( \lms \vv_j^{(t)}, \vx_i-\vx_j , \mid j \in \mathcal{N}_i \rms \right).
\end{equation}
Such that $f^{t}$ holds that for all $Q\in O(d)$
\begin{align*}
f^{t}\left( \lms \vv_j^{(t)}, \vx_i-\vx_j , \mid j \in \mathcal{N}_i \rms \right)= f^{t}\left( \lms \vv_j^{(t)},Q \cdot(\vx_i-\vx_j) , \mid j \in \mathcal{N}_i \rms \right)
\end{align*}
I-GGNN+ is maximally expressive if $f^{t}$ holds:
\begin{align*}
f^{t}\left( \lms \vv_j^{(t)}, \vx_i-\vx_j , \mid j \in \mathcal{N}_i \rms \right)= f^{t}\left( \lms \vv_j^{(t)}, \hat{\vx_i}-\hat{\vx_j} , \mid j \in \mathcal{N}_i \rms \right) \iff \\
\exists Q \in O(3): \lms \vv_j^{(t)}, \vx_i-\vx_j , \mid j \in \mathcal{N}_i \rms = \lms \vv_j^{(t)}, Q \cdot(\hat{\vx_i}-\hat{\vx_j}) , \mid j \in \mathcal{N}_i \rms
\end{align*}
We now prove that if we run one iteration of maximally expressive I-GGNN+ on a full graph, we can reconstruct it.
\begin{theorem}
Let $\G = (\mA,\mX)$ be a geometric graph, where $\mA$ is the full graph on $n$ nodes, and $\mX$ is an arbitrary $d\times n$ matrix, then one iteration of maximally expressive I-GGNN+ can reconstruct $\mX$.
\end{theorem}
\begin{proof}
Assume we run one iteration of I-GGNN+. Then, for each node $i$, we have $$f^{t}\left( \lms \vv_j^{(t)}, \vx_i-\vx_j , \mid j \in [n] \rms \right)$$ then up to some rotation $Q_{i}\in O(3)$, we know $$\lms Q_{i}\cdot(\vx_i-\vx_j) , \mid j \in [n] \rms$$ Then, we can reconstruct the point cloud by \autoref{reconstruction}.
\end{proof}
It's easy to see that I-GGNN+ can simulate our I-GGNN as the norm function is invariant under multiplication by $Q\in O(3)$. But \citep{PhysRevLett.125.166001,gcnn} gave examples of full-graphs each I-GGNN can't separate, but by what we proved above, maximally expressive I-GGNN+ can separate them. Thus, I-GGNN+ is strictly more powerful than I-GGNN in its expressive power.
\section{Approximation and separation}\label{sec:approx}
It is known that a tight connection exists between the approximation of invariant functions and complete invariant models \cite{gramnet}.
In the context of our paper, we discussed generic completeness. In particular, we showed in Corollary \ref{cor:us} that $\us$ with random parameters is complete on $\GdNgeneric$, which denotes the set of all geometric graphs $\G=(\mA,\mX)$ with exactly $N$ nodes, $\mA$ connected, and $\mX\in \RR^{d\times n}$ generic. Accordingly, we can obtain the following theorem:
\begin{theorem}
Let $K \subseteq \GdNgeneric$ be a compact set, $f^{sep}:K \rightarrow R^{m}$ a continuous function invariant to rotations, translations, and permutations. Then $f^{sep}$ is injective on $K$ (up to symmetries) if and only if every continuous invariant $f:K\rightarrow R^{M}$ can be approximated uniformly on $K$ using functions of the form $\N(f^{sep})$, where $\N$ is a fully connected neural network.
\end{theorem}
\begin{proof}
Assume, on the one hand, $f^{sep}$ is injective on $K$ (up to symmetries). Given a continuous $f: K \rightarrow R^{M}$ we want to approximate, then, by \cite{gramnet}, $\forall \epsilon > 0$ $\exists$ neural network $\N$: \begin{align*}
sup_{\G \in K} |f(\G) - \N(f^{sep}(\G))| < \epsilon
\end{align*}
On the other hand, assume by contradiction there exists $\G_{1} = (\mA_{1},\mX_{1})$ and $\G_{2}=(\mA_{2},\mX_{2})$ which are not related by translation, rotation or permutation, such that $f^{sep}(\G_{1}) = f^{sep}(\G_{2})$.
Define $f(\G) = Inf_{g \in G} \|(\mA,\mX)- g \cdot (\mA_{1},\mX_{1})\|$, where $\G=(\mA,\mX$), and $G$ is the group of permutations, rotations, and translation. Note that this minimum is obtained as the product group of rotation and permutation is compact, and the translation can be bounded by the sum of the norms of the two graphs. Then, according to the Tikhonov theorem, our group is compact, and we take a minimum continuous function over a compact set. Note that $f$ is an invariant, continuous function such that $f(\G_{1})=0$, and $f(\G_{2})> 0$, but for each neural network $\N$, $\N(f^{sep}(\G_{1})) = \N(f^{sep}(\G_{1}))$, and we can't approximate our desired function $f$ with $\epsilon = \frac{f(\G_{2})}{2}$ accuracy, yielding a contradiction.
\end{proof}
\section{Proofs}
In this section, we restate and prove theorems that have not been fully proved in the main text.
\IGGNN*
\begin{proof}
\label{Proof of I-GGNN}
Let $\mathcal{G}=(\mA,\mX) $ be a geometric graph, and assume that $\mX$ is generic.
First, we assume that $\mA$ is generically globally rigid. Assume that there is some geometric graph $\mathcal{G}'$ which is assigned the same global feature as $\mathcal{G}$ after a single iteration. We need to show the two graphs are geometrically isomorphic.
The multisets of $s_i$ and $s_i'$ features in \eqref{eq:readout} are equal, and in particular, both graphs have the same number of nodes $n$, and by relabeling if necessary, we can assume that $s_i=s_i'$ for all $i=1,\ldots,n$. This equation, in turn, implies that for every node $i$,
$$\lms \|\vx_{ij}\|, j \in \N_i \rms=\lms \|\vx_{ij}'\|, j \in \N_i' \rms$$
In particular, the $i$'th node in $\G$ and $\G'$ have the same degree, so both graphs have the same number of edges.
For any fixed $(i,j)$ which is an edge of $\G$, the distance $\|x_{ij}\|'$ will appear exactly once in the multiset corresponding to $s_i'$ and $s_j'$ and will not appear in the other multisets. This observation implies that $(i,j)$ is also an edge in the graph $\G'$. Since the number of edges in both graphs is the same, we deduce that $\mA=\mA'$.
By global rigidity, we deduce that all pairwise distances are the same, and by \cite{egnn}, the two graphs are geometrically isomorphic.
We mention a theorem proved in \cite{gortler2010characterizing} for the other direction. \cite{gortler2010characterizing} proved $\mA$ is globally rigid if exists generic $\mX$ such that $(\mA,\mX)$
is globally rigid. Thus if $\mA$ is not globally rigid, for \emph{every} generic $\mX$, there exists $\mX'$ for whom distances along edges are preserved, but $\mX$ and $\mX'$ aren't geometrically isomorphic. By immediate induction on the number of layers, it can be shown that any I-GGNN cannot separate any pair of geometric graphs whose distances across edges are preserved. This concludes the proof.
\end{proof}
\GGNN*
\begin{proof}
\label{Identifiable}
Assume $\mA$ is not connected. Let $\mX \in \RR^{n\times d}$. Let $\mV_{1},\mV_{2}$ be two connected components of the graph, and let $\G'$ be the geometric graph whose adjacency matrix is $\mA'=\mA$, and where $\mX'$ is obtained from $\mX$ by applying a non-trivial rotation $Q$ not in the symmetry group of $\mV_{2}$ : $\hat{\vx_i} = Q \cdot \vx_i$ for all $i \in V_1$, and setting $\hat{\vx_j}=\vx_j$
for all $j\in V_2$. The two graphs are not geometrically isomorphic by construction. However, for any E-GGNN, we can easily see recursively that all features $\vv_i^{(t)}$
constructed from $\mX$ and $\mX'$, respectively, will be the same, and therefore, the E-GGNN will assign the same global features to $\mathcal{G}$ and $\mathcal{G}'$.
Now assume $\mA$ is connected, and we are given two geometric graphs $\G$ and $\hat{\G}$ in $(\GdNgeneric,\GdN)$. By assumption, $\mX$ is generic. Assume that the E-GGNN assigns the same value of $\G$ and $\hat{\G}$. We need to show that the two graphs are geometrically isomorphic.
First, we note that since the two graphs have the same global features, they have the same number of nodes $n$, and by applying relabeling if necessary, we can assume that
\begin{align*}
s_i=\hat{s}_i, \quad \forall i=1,\ldots,n
\end{align*}
which implies that for every node $i$ there is some rotation $R_i\in O(d)$ such that $\vv_i^{(T)}=R_i\hat{\vv}_i^{(T)}$. By rotation equivariance, we know that $R_i\hat{\vv}_i^{(T)}$ is the feature that we would have obtained by applying the maximally expressive E-GGNN to $R_i \cdot \hat\mX $. We deduce that
$$\lms \vv_j^{(T-1)}, \vx_i-\vx_j|j \in \gN_i \rms=\lms R_i\hat{\vv}_j^{(T-1)}, R_i(\hat{\vx}_i-\hat{\vx}_j)| j \in \hat{\gN}_i \rms$$
The equality of multisets above implies that the node $i$ has the same degree in $\mA$ and $\hat{\mA}$. Since this is true for all nodes, $\mA$ and $\hat{\mA}$ have the same number of edges.
Moreover
$$\lms \|\vx_i-\vx_j\|, j \in \gN_i \rms=\lms \|\vx_i-\vx_j\|, j \in \hat{\gN}_i \rms $$
Since $\mX$ is generic, for every edge $(i,j)\in \mA$ the distance $||\vx_i-\vx_j|| $ will appear only in the 1-hop multi-sets corresponding to the $i$ and $j$ nodes. Thus, they will appear only once in the 1-hop multisets corresponding to the second graph's $i$ and $j$ nodes. This implies that $(i,j)$ is an edge in $\hat{\mA}$ too, and
$$ R_i(\hat{\vx}_i-\hat{\vx}_j)=\vx_i-\vx_j, \forall j \in \gN_i.$$
Since the two graphs have the same number of edges, it follows that $\mA=\hat{\mA} $.
We now show recursively that for all $1\leq t \leq T-1$, we have $\vv_j^{T-t}=R_i \hat{\vv}_j^{T-t} $ and $\vx_i-\vx_j=R_i(\hat{\vx}_i-\hat{\vx}_j)$ for all $j$ that is a $t$-hop neighbor of $i$ (we say $j$ is an $t$-hop neighbor of $i$, if there is a path of length at most $t$ from $i$ to $j$).
We have already proved the claim for $t=1$. Now we assume correctness for $t$ and show for $t+1$. Since by the induction hypothesis we know that $\vv_j^{T-t}=R_i \hat{\vv}_j^{T-t} $ for every $j$ which is $t$ hop neighbor of $i$, we deduce the multi-set equality
$$\lms \vv_k^{T-(t+1)}, \vx_j-\vx_k | k \in \gN_j\rms=\lms R_i\hat{\vv}_k^{T-(t+1)}, R_i(\hat{\vx}_j-\hat{\vx}_k)|k \in \gN_j \rms $$
Since we know that the pairwise norms of $\mX$ are all distinct to to the genericity of $\mX$, and also that for every edge $(j,k)$ we have the norm equality $\|\vx_j-\vx_k\|=\|\hat \vx_j-\hat \vx_k\| $, we see that necessarily
$$\vv_k^{T-(t+1)}= R_i \hat{\vv}_k^{T-t+1}, \text {and } \vx_j-\vx_k=R_i(\hat{\vx}_j-\hat{\vx}_k).$$
Now, every $k$, an $t+1$ hop neighbor of $i$, is connected by an edge to a node $j$, an $t$ hop neighbor of $i$.
By the induction assumption we know that $\vx_i-\vx_j=R_i(\hat{\vx}_i-\hat{\vx}_j) $. We deduce that
$$\vx_i-\vx_k=\vx_i-\vx_j+\vx_j-\vx_k=R_i(\hat{\vx}_i-\hat{\vx}_j+\hat{\vx}_j-\hat{\vx}_k)=R_i (\hat{\vx}_i-\hat{\vx}_k) $$
for all $t+1$ hop neighbors of $i$. Continuing with this argument inductively, we see that for all $T=d+1$ hop neighbors $k$ of $i$, we obtain
$$\vx_i-\vx_k=R_i (\hat{\vx}_i-\hat{\vx}_k) $$
and in particular
$$\|\vx_i-\vx_k\|=\|\hat{\vx}_i-\hat{\vx}_k\| $$
for all $d+1$ hop neighbors. By global generic rigidity of the $d+1$ power graph of a connected graph \citep{power-graph,power-graph2}, it follows that the two graphs are geometrically isomorphic.
Thus, as previously, we reconstructed the combinatorial graph, and we now reconstructed the geometric graphs; we are done.
\end{proof}
\subsection{Proof of Maximal Expressivity of $\us$}
\label{sub:maximal}
This subsection is devoted to proving Theorem \ref{thm:maximally} on the generic maximal expressivity of $\us$. This proof will require some background on the finite witness theorem from \cite{Neural_Injective}:
\paragraph{Background on the finite witness theorem} Before we prove the theorem, we review some definitions and results from \cite{Neural_Injective}, which we will use for the proof.
The main tool we require from \cite{Neural_Injective} is Theorem A.2 from that paper, the finite witness theorem. We state it here in the special case where the parameter space is Euclidean, and the function $F$ is analytic in the parameters:
\begin{theorem}[Finite Witness Theorem, \cite{Neural_Injective}]\label{thm_fwt_full}
Let $\M \subseteq \RR^p$ be a $\sigma$-sub-analytic set of dimension $D$. Let $F:\M \times \mathbb{R}^{q} \to \mathbb{R}$ be a $\sigma$-sub-analytic function, and assume that $F(\bz;\bth)$ is analytic as a function of $\bth$ for all fixed $\bz \in \M$. Define the set
$$\N=\{\bz\in \M| \quad F(\bz;\bth)=0,\ \forall \bth\in \RR^q \}.$$
Then for Lebesgue almost every \footnotemark $\left( \bth^{(1)},\ldots,\bth^{(D+1)}\right) \in \RR^{q\times (D+1)}$,
\begin{equation}\label{eq:Nequals}
\N=\{\bz\in \M| \quad F(\bz;\bth^{(i)})=0,\ \forall i=1,\ldots D+1\} .
\end{equation}
\end{theorem}
\footnotetext{The original statement in \cite{Neural_Injective} discussed \emph{generic} $\left( \bth^{(1)},\ldots,\bth^{(D+1)}\right) \in \W^{D+1}$, in the sense defined in \cite{Neural_Injective}. In particular, this implies the "Lebesgue almost every" statement we used here. }
\textbf{Remark:} We note that this theorem also holds when the output of $F$ is a vector in $\RR^d$, which is what we will use in our proofs below.
To see the theorem holds for a function $\hat F:\M \times \mathbb{R}^q \to \mathbb{R}^{d}$, we can define a new parametric function $F$ with additional parameters $a\in \RR^d$,
$$F(\bz;\bth,a)=\langle \hat F(\bz;\bth),a \rangle $$
and use the theorem for $F$, which has scalar output, to deduce the theorem for $\hat F$.
The finite witness theorem assumes the domain is a $\sigma$ sub-analytic set. The full definition of $\sigma$-sub-analytic sets is rather involved, and we refer the reader to Appendix A in \cite{Neural_Injective} for a full definition. Rather than giving a full definition here, we will recall the properties of these sets, which we will need.
The class of $\sigma$ subanaltyic sets in $\RR^d$ is rather large. In particular, it contains all open sets in $\RR^d$, all semi-algebraic sets in $\RR^d$ (sets defined by polynomial equalities and inequalities, like spheres and linear sub-spaces), as well as sets defined by analytic equality and inequality constraints. The class of $\sigma$ subanaltyic sets is closed under countable unions and finite intersections.
$\sigma$-sub-analytic sets are always a countable union of smooth manifolds. The dimension of a $\sigma$ sub-analytic set is the maximal dimension of the manifolds composing it.
A $\sigma$-sub-analytic function is a function whose graph is a $\sigma$ sub-analytic set. Analytic functions are, in particular, $\sigma$-sub-analytic. When a $\sigma$ sub-analytic function is applied to a $\sigma$ sub-analytic set, the image is a $\sigma$ sub-analytic set whose dimension is no larger than the dimension of the $\sigma$ sub-analytic domain \cite{Neural_Injective}.
We can now restate and prove Theorem \ref{thm:maximally}:
\express*
\begin{proof}
In the proof, we consider the set of all generic geometric graphs in $\GdN$. We will prove the theorem for an even larger set: the set $\GdNdistinct$ of all geometric graphs in $\GdN$, which additionally satisfies that for any distinct edges $(i,j)$ and $(s,t)$ in the graph,
$0<\|\vx_i-\vx_j\|\neq \|\vx_s-\vx_t\|$. Since polynomial inequalities with integer coefficients define it, we note that $\GdNdistinct$ contains \emph{all generic} graphs with $\leq N$ nodes. The advantage of considering $\GdNdistinct$ is that the set of all geometric graphs in $\GdNdistinct$ of the same cardinality $n \leq N$ can be
thought of as a subset of $\RR^{n\times n}\oplus \RR^{n \times d} $. Since a constant number of polynomial inequalities defines it, it is a semi-algebraic set, and hence, in particular, $\sigma$-subanaltyic, so the finite witness theorem can be applied. In contrast, it is unclear that the set of all generic points of fixed dimension is a $\sigma$ subanaltyic set.
Our goal is to show that for the networks described in the theorem, for almost every choice of parameter values, all functions described in the procedure are injective. To be more accurate, let us introduce some notation given an arbitrary pair
of geometric graphs
$$\G=(\mA,\mX), \quad \hat \G=(\hat \mA,\hat \mX) $$
in our domain $\GdNdistinct$, we denote the features corresponding to the first graph by $\vv_{i}^{(t)},\vs_i$ and $\sglobal$ as in the main text, and the features corresponding to the second graph by $\hat{\vv}_{i}^{(t)},\hat{\vs}_i$ and $\sglobalhat$. The neighborhood of a node $k$ in $\hat \G$ is denoted by $\hat \gN_k $. The number of nodes in the two graphs is denoted by $n$ and $m$, respectively.
We need to show that for all $t=0,\ldots,T-1$ and all nodes $i$ of $\G$ and $k$ of $\hat \G$,
\begin{align}
\vv_{i}^{(t+1)}&=\hat{\vv}_{k}^{(t+1)} \text{ if and only if } \lms (\vx_i-\vx_j,\vv_{j}^{(t)}), j \in \gN_i \rms=\lms (\hat{\vx}_k-\hat{\vx}_j,\hat{\vv}_{j}^{(t)})| j \in \hat{\gN}_k \rms \label{eq:first}\\
\vs_{i}&=\hat{\vs}_k \text{ if and only if } \exists \mQ \in O(d), \mQ\vv_{i}^{(T)}=\hat{\vv}_{k}^{(T)} \label{eq:middle}\\
\sglobal&=\sglobalhat \text{ if and only if } \lms \vs_i| i=1,\ldots,n \rms= \lms \hat \vs_i| i=1,\ldots,m \rms \label{eq:last}
\end{align}
Our first step for reaching this result is using the finite witness theorem, similar to what was done in \cite{gramnet}. We note that the features we are interested in are composed of $C=2Nd+1$ coordinates, denoted by $q=1,\ldots, C$, which are of the general form
\begin{align}
v_{i,q}^{(t+1)}-\hat{v}_{i,q}^{(t+1)}&=F_t(\mX,\mV^{(t)},\hat \mX, \hat{\mV}^{(t)};\alpha_{q,t}), \quad t=0,\ldots,T-1 \label{eq:bla}\\
s_{i,q}^{(t+1)}-\hat{s}_{i,q}^{(t+1)}&=F_{T}(\mV^{(T)}, \hat{\mV}^{(t)};\alpha_{q,T})\\
\sglobal_q-\sglobalhat_g&=F_{T+1}(\vs_1,\ldots,\vs_n,\hat{\vs}_1,\ldots,\hat {\vs}_{m}; \alpha_{q,T+1})
\end{align}
where $\alpha_{q,t}$ denote the parameters of the appropriate function, $\mV^{(t)}$ is the concatenation of all $n$ features $\vv_i^{(t)}$, and $\hat \mV^{(t)}$ is the concatenation of all $m$ features $\hat{\vv}_k^{(t)}$. In essence, we claim that by the finite witness theorem, for almost every choice of network parameters, we will have the following equalities for all geometric graphs $\G,\hat \G $ satisfying the theorem's assumptions:
\begin{align}
\vv_{i}^{(t+1)}&-\hat{\vv}_{k}^{(t+1)}=0 \text{ if and only if } F_t(\mX,\mV^{(t)},\hat \mX, \hat{\mV}^{(t)};\alpha_{t})=0, \forall \alpha \label{eq:first2}\\
\vs_{i}&-\hat{\vs}_k=0 \text{ if and only if } F_{T}(\mV^{(T)}, \hat{\mV}^{(t)};\alpha), \forall \alpha \label{eq:middle2}\\
\sglobal&-\sglobalhat=0 \text{ if and only if } F_{T+1}(\vs_1,\ldots,\vs_n,\hat{\vs}_1,\ldots,\hat \vs_{m}; \alpha)=0, \quad \forall \alpha \label{eq:last2}
\end{align}
In other words, the finite witness theorem enables us to reduce the problem to the problem of showing that some choice of network parameter exists, which enables separation.
Some explanation is needed as to the details of how the finite witness theorem is applied. It is convenient to think of the network parameters being chosen recursively. For $t=0$, the finite witness theorem allows us to choose the parameters $\alpha_{q,0}, q=1,\ldots, C $ in \eqref{eq:bla} so that the theorem applies. Formally, to do this, we need to split into a finite number of cases: once the number of nodes $n$ and $m$ in the two graphs are specified, as well as the combinatorial graphs $\mA, \hat \mA$ and the nodes $i,k$, we can think of $F_0$ as a function which is analytic in the parameters, and $\sigma$-subanalytic on its domain (the norms used in the network are semi-algebraic functions, and hence $\sigma$-subnaltyic functions. All other functions used in the network are analytic). The function $F_0$ operates on the domain of pairs $\mX \in \RR^{d\times n}, \hat \mX \in \RR^{d\times m}$ which satisfy a finite number of polynomial inequalities. This domain is semi-algebraic and has dimensions $d(n+m)\leq 2Nd$. Therefore, we can apply the finite witness theorem with $C=2Nd+1$.
Next, once the parameters $\alpha_{q,t}$ are chosen for all $t<t'$, we can choose the parameters $\alpha_{q,t'}$ again using the finite witness theorem. Central to the usage here is that, once the previous parameters were fixed, the features in the $t'$ step are an image of a $\sigma$-subanaltyic function applied to a pair $(\mX,\hat \mX) $. Therefore, the intrinsic dimension of the $t'$ generation features is no larger than $2Nd+1=C$ (for more on this argument, see \cite{gramnet}). This concludes the recursive definition of parameter choice. By Fubini's theorem, one can verify that, in fact, for Lebesgue almost every parameter will be 'good.'
It remains to show that
\begin{align}
\text{The right hand side of \eqref{eq:first} holds if and only if the right hand side of \eqref{eq:first2} holds} \label{eq:first_claim}\\
\text{The right hand side of \eqref{eq:middle} holds if and only if the right hand side of \eqref{eq:middle2} holds} \label{eq:middle_claim}\\
\text{The right hand side of \eqref{eq:last} holds if and only if the right hand side of \eqref{eq:last2} holds} \label{eq:last_claim}
\end{align}
We now reformulate these three claims as three lemmas.
We use some fixed analytic non-polynomial function $\sigma$ in the following lemmas. We say that $\phi$ is a \emph{basic analytic network} if it is of the form
$$\phi(\vs;a,b)=\sigma(a \cdot \vs+b)$$
for some $a\in \RR^d, b\in \RR$.
Our first lemma will show \ref{eq:first_claim}, where $z_j$ will assume the role of $\vx_i-\vx_j$.
\begin{lemma}
\label{eq:hardest}
Let $z_1,\ldots,z_n$ and $\hat{z}_1,\ldots,\hat{z}_m$ be non-zero vectors in $\RR^d$. Additionally assume that $\|z_1\|<\|z_2\|<\ldots <\|z_n\| $ and $\|\hat{z}_1\|<\|\hat{z}_2\|<\ldots <\|\hat{z}_m\| $.
Let $\vv_1,\ldots,\vv_n $ and $\hat{\vv}_1,\ldots,\hat{\vv}_m $ be matrices in $\RR^{d \times C}$, and denote
$\vv_i=(v_{i,1},\ldots,v_{i,C})$ and $\hat \vv=(\hat{v}_{i,1},\ldots,\hat{v}_{i,C})$. If for all basic analytic neural networks $\phi^{(0)},\ldots,\phi^{(C)} $ we have that
\begin{align}\sum_{j=1}^n\left( \phi^{(0)}(\|z_j\|)(z_j)+\sum_{c=1}^C \phi^{(c)}(\|z_j\|)v_{jc} \right)-
\sum_{j=1}^m\left( \phi^{(0)}(\|\hat{z}_j\|)(\hat{z}_j)+\sum_{c=1}^C \phi^{(c)}(\|\hat{z}_j\|)\hat{v}_{jc} \right)=0 \label{eq:big}
\end{align}
then
$$\lms (z_1,\vv_1), \ldots(z_n,\vv_n) \rms
=\lms (\hat{z}_1,\hat{\vv}_1), \ldots(\hat{z}_m,\hat{\vv}_m) \rms$$
\end{lemma}
We note that this lemma is slightly stronger than we need since, in contrast with the original MPNN, we do not use the norms of $v_{i,c}$ as features.
Our second lemma will show \ref{eq:middle_claim}
\begin{lemma}\label{eq:rot}
Let $\vv=(v_1,\ldots,v_C)\in \RR^{d \times C} $ and $\hat \vv=(\hat{v}_1,\ldots,\hat{v}_C)\in \RR^{d \times C} $. If for all $\theta \in \RR^C$ we have that
$$\|\sum_{c=1}^C \theta_cv_c\|=\|\sum_{c=1}^C \theta_c\hat{v}_c\| $$
then there exists an $\mQ \in O(d)$ such that $\mQ v_c=\hat v_c $ for all $c=1,\ldots,C$.
\end{lemma}
For proof of this lemma, see Section 3.2 in \cite{gortler_and_dym}.
Our third lemma will show \ref{eq:last_claim}
\begin{lemma}
Let $\vs_1,\ldots,\vs_n $ and $\hat {\vs}_1, \ldots ,\hat {\vs}_m$ be vectors in $\RR^d$.
If for all basic analytic $\phi$, we have that
$\sum_{i=1}^n \phi(\vs_i)=\sum_{i=1}^m \phi(\hat \vs_i) $
then
$\lms \vs_1, \ldots,\vs_n \rms=\lms \hat{\vs}_1,\ldots, \hat{\vs}_m \rms $.
\end{lemma}
For proof of this lemma, see Proposition 3.2 in \cite{Neural_Injective}.
It remains to prove Lemma \ref{eq:hardest}. We note that if \eqref{eq:big} holds for all basic analytic networks $\phi^{(c)}, c=0,\ldots, C$, This equality is also true if each $\phi^{(c)}$ is a linear combination of a finite number of shallow neural networks. Next, since we are only considering the values of $\phi^{(c)}$ on a fixed finite collection of inputs, and since linear combinations of shallow neural networks with analytic (or even continuous) non-polynomial functions can approximate any continuous function uniformly on a compact set (and in particular on a finite set) \cite{approximations}, it follows that \eqref{eq:hardest} holds for all \emph{continuous} $\phi^{(c)}$.
Next, we consider the norm multisets:
$$S=\lms \|z_1\|, \ldots,\|z_n\| \rms, \quad \hat{S} =\|\hat{z}_1\|, \ldots,\|\hat{z}_m\| .$$
We claim that $S=\hat{S}$. We first show that $S\subseteq \hat S$. Fix some $s$ in $\{1,\ldots,n\}$, we need to show that $\|z_s\| \in \hat S$. We define a continuous function $f^s$ such that $f^s(\|z_s\|)=1$ and $f^s(\|z_i\|)=0=f^s(\|\hat z_j\|) $ for all $i\neq s$, and for all $j=1,\ldots,m$ such that the norms $\|\hat z_j\|$ and $\|z_s\|$ are not equal. We claim that there exists $j$ for which the norms \emph{are} equal. Otherwise, we choose the continuous functions $\phi^{(c)}, c=1,\ldots,C $ to be the zero functions, and choose $\phi^{(0)}$ to be $f^s$. This will lead to the equation $z_s=0$ in \ref{eq:hardest}, which contradicts the assumption that $z_s\neq 0$, so we see that $S \subseteq \hat S $. A symmetric argument shows that $\hat S \subseteq S $ and so $S=\hat S$. We deduce that $n=m$ and $\|z_i\|=\| \hat z_i \| $ for all $i=1,\ldots,n$.
We can now conclude the proof: fix an arbitrary $s$ in $\{1,\ldots,n\}$. Choose $\phi^{(0)}=f^s$ and take the other $\phi^{(c)}$ to be zero. Then \ref{eq:hardest} gives
$$z_s-\hat{z}_s=0 $$
Similarly, for an arbitrary index $q$ in $\{1,\ldots,C\}$, we can choose $\phi^{(q)}=f^s $ and $\phi^{(c)}=0$ for all $c\neq q$ in $\{0,1,\ldots,C\}$. In this case \ref{eq:hardest} gives
$$v_{jq}-\hat{v}_{jq}=0 .$$
This concludes the proof of the lemma and the proof of the theorem.
\end{proof}
\subsection{Complexity}
\label{complexity}
Running $\us$ for $T$ iterations with $C$ channels requires $O(C \cdot d \cdot deg_{i})$ elementary arithmetic operation for every node $i$, at every iteration $t$. Overall, this requires $O(C \cdot d \cdot N \cdot d_{avg} \cdot T)$ to run $\us$, where $d_{avg}$ denotes the average degree in the graph. This is linear in $N$, which dominates the other terms in the complexity bound. If we set $C=2Nd+1$ as required for maximal expressivity in Theorem \ref{thm:maximally}, then the complexity will be quadratic in $N$, which is less by a factor of $n$ then \citep{three_iterations,disgnn}, and by a factor of $n^{2}$ than \cite{gramnet}.
We also note that the cardinality in the finite witness theorem, the main tool for our proof, depends on the intrinsic dimension of the input and not the ambient dimension. Thus, under the standard 'manifold assumption' \cite{manifold}, which asserts that data in learning tasks typically resides in a manifold of low dimension $r$, embedded in a high dimensional Euclidean space, the proof can be adapted to show that the number of channels necessary for maximal expressivity is only $2r+1$. Thus, under the manifold assumption, the complexity of $\us$ is linear in the number of nodes $N$.
\section{Adding node and edge features}
\label{app:features}
This section explains how we incorporate node and edge features into our architecture and theory.
A featured geometric graph is a $4$-tuple $\G =(\mA_{1},\mX_{1},\mF_{1},\mE_{1})$ where as before $\mA$ is an $n$ by $n$ adjacency matrix, and $\mX$ is a $n $ by $d$ matrix denoting node positions. $\mF$ is a $n$ by $d_n$ matrix denoting node features (without the action of rigid motions), and $\mE$ is a $n\times n \times d_e$ tensor of edge features. We denote the feature at a node $i$ and edge $(i,j)$ by $f_i$ (a column of $\mF$) and $f_{i,j}$, respectively.
We incorporate edge and node features into $\us$ by slightly modifying the aggregation step to
\begin{align*}
v^{(t+1)}_{i,q}=\sum_{j\in \gN_{i}}\psi_{n}^{(t,q)}(f_{i})\cdot \psi_{e}^{(t,q)}(f_{i,j}) \cdot (\phi^{(t,q,0)}(\|\vx_{i}-\vx_{j}\|,\|\vv_{j}\|)(\vx_{i}-\vx_{j})+ \\
\sum_{c=1}^{k} \phi^{(t,q,c)}(\|\vx_{i}-\vx_{j}\|,\|\vv_{j}\|)v_{jc}^{(t)})
\end{align*}
In our analysis we will take $\psi_{n}^{(t,q)}:\RR^{d_{n}} \to \RR$ and $\psi_{e}^{(t,q)}:\RR^{d_{e}} \to \RR$ to be simple linear functions
$$\psi_{n}^{(t,q)}(f_{i})=a_{n}^{(t,q)} \cdot f_{i}+b_{n}^{(t,q)}, \quad \psi_{e}^{(t,q)}(f_{i,j})=a_{e}^{(t,q)} \cdot f_{i,j}+b_{q}^{(t,q)} $$
In practice, in our code, we use a slightly more complex function involving radial basis functions as in \cite{disgnn}.
To extend our analysis to the case of node and edge features, we first extend our definition of isomorphism of geometric graphs to include edge and node features.
\begin{definition}
Given two featured geometric graphs
\begin{align*}
\G =(\mA_{1},\mX_{1},\mF_{1},\mE_{1}), \quad \mH = (\mA_{2},\mX_{2},\mF_{2},\mE_{2})
\end{align*}
$\G$ and $\H$ are \emph{featured geometrically isomorphic} if there exists a permutation matrix $\mP$, a rotation matrix $\mQ$, and a translation ${\vt}$ such that:
\begin{enumerate}
\item $\mA_{2}$ = $ \mP \cdot \mA_{1} \cdot \mP^{T}$.
\item $\mX_{2} = \mP \cdot \mX_{1} \cdot \mQ + \vt$
\item $\mF_{2} = \mP \cdot \mF_{1}$
\item $\mE_{2}$ = $ \mP \cdot \mE_{1} \cdot \mP^{T}$
\end{enumerate}
where $ \mP \cdot \mE_{1} \cdot \mP^{T}$ should be understood as operating element-wise on the $n_e$ matrices of dimension $n \times n$ which compose the tensor $\mE $.
\end{definition}
We can then prove the following theorem: a generalization of \autoref{cor:us} to the featured graph case.
\begin{theorem}
Let $N,d,d_e,d_n$ be natural numbers. Let $F_\theta$ denote the $\us$ architecture for featured geometric graphs, with depth $T=d+1$ and $C=2Nd+d_{e}+d_{n}+1$ channels. Then for Lebesgue almost every $\theta$, we have for all featured graphs $\G, \G'\in \GdN $ which are generic and connected,
$$F_\theta(\G)=F_\theta(\hat{\G}) \iff \G, \hat{\G} \text{ are featured geometrically isomorphic}$$
\end{theorem}
We note that the assumption that the geometric graph is generic is only on the position coordinates $\mX$. The statement will hold for all node and edge features in this case.
\begin{proof}
We explain how to modify the proof of Theorem \ref{thm:maximally} to obtain this result. First, we note that the number of channels was adapted to take into account the higher dimensionality of the data now that node and edge features are included. The finite witness theorem can be applied with this adapted dimension, as explained in Theorem \ref{thm:maximally}. Thus, we only need to show that, for given featured graphs $\hat{\G},\G $ satisfying the conditions of the theorem, if for all $\theta$ we have that $F_\theta(\G)=F_\theta(\hat \G) $, then $\G $ and $\hat \G$ are featured geometrically isomorphic.
Now, let us assume that, for all $\theta$ we have $F_\theta(\G)=F_\theta(\hat \G)$. We can obtain all the results we had for the case where there were no additional features by choosing the parameters
$$a_n^{(t,q)}=0, b_n^{(t,q)}=1, a_e^{(t,q)}=0, b_e^{(t,q)}=1 $$
so that $\psi_n^{(t,q)}=1=\psi_e^{(t,q)}$, and the effect of node and edge features on the architectures is 'cancelled out'. In particular, we can deduce that, after an appropriate permutation and rigid motion are applied to the data, the graphs have the same number of nodes $n$, and for all node $i$ and iteration $t$, we have
$$\vv_i^{(t)}=\hat {\vv}_i^{(t)}, \vx_i=\hat{\vx}_i $$
Now, by setting $\phi^{(t,q,c)}=0$ for $c\geq 1$, and considering $\phi^{(t,q,0)}$ which depend only on the first coordinate, we can deduce that
\begin{align*}
\sum_{j\in \gN_i} \left[(\psi_n^{(t,q)}(f_i)\cdot \psi_e^{(t,q)}(f_{i,j})-\psi_n^{(t,q)}(\hat{f}_i)\cdot \psi_e^{(t,q)}(\hat{f}_{i,j})\right]\cdot \left( \phi^{(t,q,0)}(\|\vx_i-\vx_j\|)(\vx_i-\vx_j) \right)=0
\end{align*}
for all choices of parameters of the functions $\psi_n^{(t,q)},\psi_e^{(t,q)}$ and $\phi^{(t,q,0)}$. As we showed in the proof of \autoref{thm:maximally}, this equation implies that this equation will also hold for any continuous $\phi^{(t,q,0)}$ (and any choice of parameters of the other functions) since the span of all such basic analytic functions is dense. In particular, for fixed $i,j$, we can choose $\phi^{(t,q,0)}$ to be a continuous functions with $\phi^{(t,q,0)}(\|\vx_i-\vx_j\|)=1$ and $\phi^{(t,q,0)}(\|\vx_i-\vx_k\|)=0$ if $k \neq j$. Inserting this into the last equality we obtain
$$\left[\psi_n^{(t,q)}(f_i)\cdot \psi_e^{(t,q)}(f_{i,j})-\psi_n^{(t,q)}(\hat{f}_i)\cdot \psi_e^{(t,q)}(\hat{f}_{i,j})\right] \cdot (\vx_i-\vx_j) =0$$
by genericity $\vx_i-\vx_j \neq 0$, and so we deduce that
$$ \psi_n^{(t,q)}(f_i)\cdot \psi_e^{(t,q)}(f_{i,j})-\psi_n^{(t,q)}(\hat{f}_i)\cdot \psi_e^{(t,q)}(\hat{f}_{i,j})=0$$
for every choice of function parameters. Setting
$a_n^{(t,q)}=0, b_n^{(t,q)}=1, b_e^{(t,q)}=0 $
we obtain that
$$a_e \cdot (f_{ij}-\hat{f}_{ij})=0, \quad \forall a_e \in \RR^{d_e}$$
and therefore $f_{ij}-\hat{f}_{ij}=0 $. Similarly, setting $a_e^{(t,q)}=0, b_e^{(t,q)}=1, b_n^{(t,q)}=0 $ we obtain that
$$a_n \cdot (f_{i}-\hat{f}_{i})=0, \quad \forall a_n \in \RR^{d_n}$$
and therefore $f_{i}-\hat{f}_{i}=0 $. This concludes the proof of the theorem.
\end{proof}
\section{Additional Empirical Results}
\paragraph{k-chain results}
Here, we show the results of the $k$-chain experiment for $k=4$. As we can see, only our method $\us$ and GVP \cite{gvp-egnn} succeed using the minimal number of blocks. This experiment illustrates our model's ability to distinguish very sparse graphs. Next, we show results for $k=12$. As can be seen, we are the only ones succeeding with the minimal number of blocks, and other models need a higher number of blocks to succeed in distinguishing.
\begin{table*}[t!]
% \hfill
\caption{Separation Learning of $4$-chain paths \autoref{fig:Two pairs for power Graphs}. We show the learned accuracy over ten trials for each model, depending on the number of blocks. We show the minimal needed number of blocks (3) is sufficient for separation, and only \cite{gvp-egnn} also succeeded (among equivariant models working on the sparse graph) all $10$ trials using $3$ blocks.}
\begin{subtable}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{clccccc}
\toprule
& ($k=\mathbf{4}$-chains) & \multicolumn{5}{c}{\textbf{Number of layers}} \\
& \textbf{GNN Layer} & $\lfloor \frac{k}{2} \rfloor$ & \cellcolor{gray!10} $\lfloor \frac{k}{2} \rfloor + 1 = \mathbf{3}$ & $\lfloor \frac{k}{2} \rfloor + 2$ & $\lfloor \frac{k}{2} \rfloor + 3$ & $\lfloor \frac{k}{2} \rfloor + 4$ \\
\midrule
\multirow{5}{*}{\rotatebox[origin=c]{90}{Equiv.}} &
\gray{GWL} & \gray{50\%} & \gray{\textbf{100\%}} & \gray{\textbf{100\%}} & \gray{\textbf{100\%}} & \gray{\textbf{100\%}} \\
& E-GNN \cite{egnn} & 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
& GVP-GNN \cite{gvp-egnn} & 50.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
& TFN \cite{TFN} & 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{blue!10} 80.0 {\scriptsize ± 24.5} & \cellcolor{blue!10} 85.0 {\scriptsize ± 22.9} \\
& MACE \cite{mace} & 50.0 {\scriptsize ± 0.0} & \cellcolor{blue!10} 90.0 {\scriptsize ± 20.0} & \cellcolor{blue!10} 90.0 {\scriptsize ± 20.0} & \cellcolor{blue!10} 95.0 {\scriptsize ± 15.0} & \cellcolor{blue!10} 95.0 {\scriptsize ± 15.0} \\
& $\us$(ours) & 50.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
\midrule
\multirow{4}{*}{\rotatebox[origin=c]{90}{Inv.}} &
\gray{IGWL} & \gray{50\%} & \gray{50\%} & \gray{50\%} & \gray{50\%} & \gray{50\%} \\
& SchNet \cite{SchNet} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} \\
& DimeNet \cite{DimeNet} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} \\
& SphereNet \cite{SphereNet} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} \\
\cmidrule{2-7}
& SchNet $_{\text{full graph}} \footref{schnet models}$ \cite{SchNet} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
& SchNet$_{\text{global feat}} \footref{schnet models}$ \cite{SchNet} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} \\
\bottomrule
\end{tabular}
}
\end{subtable}
\hfill
\label{tab:4chains}
\end{table*}
\begin{table*}[b!]
\hfill
\caption{Separation Learning of $12$-chain paths \autoref{fig:Two pairs for power Graphs}. We show the learned accuracy over ten trials for each model depending on the number of blocks. As we can see, we are the only ones that succeed with the minimal number of blocks ($7$), and other models need many more layers to show high Performance.}
\begin{subtable}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{clcccccc}
\toprule
& ($k=\mathbf{12}$-chains) & \multicolumn{5}{c}{\textbf{Number of layers}} \\
& \textbf{GNN Layer} & $ \lfloor \frac{k}{2} \rfloor $ & \cellcolor{gray!10} $\lfloor \frac{k}{2} \rfloor +1$ & $ \lfloor \frac{k}{2} \rfloor 2$ & $ \lfloor \frac{k}{2} \rfloor +3$ & $ \lfloor \frac{k}{2} \rfloor +4$\\
\midrule
\multirow{5}{*}{\rotatebox[origin=c]{90}{Equiv.}} &
\gray{Maximally expressiveness E-GGNN} & \gray{50\%} & \gray{\textbf{100\%}} & \gray{100\%} & \gray{100\%} & \gray{\textbf{100\%}} & \\
& E-GNN \cite{egnn} & 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} &
\cellcolor{red!10} 50.0 {\scriptsize ± 0.0} &
\cellcolor{blue!10} 95 {\scriptsize ± 15.0} & \\
& GVP-GNN \cite{gvp-egnn} & 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{blue!10} 60.0 {\scriptsize ± 20.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{blue!10} 60.0 {\scriptsize ± 20.0} & \\
& TFN \cite{TFN} & 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \\
& MACE \cite{mace} & 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \cellcolor{red!10} 50.0 {\scriptsize ± 0.0} & \\
& $\us$(ours) & 50.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0}& \\
\midrule
\multirow{4}{*}{\rotatebox[origin=c]{90}{Inv.}} &
\gray{Maximally expressiveness I-GGNN} & \gray{50\%} & \gray{50\%} & \gray{50\%} & \gray{50\%} & \gray{50\%} & \\
& SchNet \cite{SchNet} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} \\
& DimeNet \cite{DimeNet} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}}& \\
& SphereNet \cite{SphereNet} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} & 50.0 {\scriptsize {\scriptsize ± 0.0}} \\
\cmidrule{2-7}
& $\text{SchNet}_{\text{full graph}}$\footref{schnet models} \cite{SchNet} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0}& \\
& $\text{SchNet}_{\text{global feat}}$\footref{schnet models} \cite{SchNet} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0} & \cellcolor{green!10} 100.0 {\scriptsize ± 0.0}& \\
\bottomrule
\end{tabular}
}
\end{subtable}
\hfill
\label{tab:12chains}
\end{table*}
\paragraph{Hard examples}
This section presents the results mentioned in Section \ref{sepearion exps}. We use a dataset of four difficult-to-separate pairs of point clouds. Three (Hard1,Hard2,Hard3) are taken from
\cite{PhysRevLett.125.166001}. The last pair, 'Harder,' is taken from \cite{gcnn} and is an example of a non-isomorphic pair that I-GGNN models cannot distinguish. We show we succeeded in separating all tuples with a probability of $1$.
The tested models are $\us$, GramNet \cite{gramnet}, GeoEGNN \cite{gramnet}, EGNN \cite{egnn}, LinearEGNN, MACE \cite{mace}, TFN \cite{TFN}, DimeNet \cite{DimeNet}, GVPGNN \cite{gvp-egnn} and are taken from \cite{gramnet}.
\begin{table*}[t!]
\centering
\caption{Results on challenging point clouds \cite{gcnn,PhysRevLett.125.166001}. The table shows the performance of different models on progressively more difficult tasks (Hard1 to Harder).}
\begin{subtable}
\centering
% \resizebox{\linewidth}{!}{
\begin{tabular}{clcccc}
\toprule
& \textbf{Point Clouds} & \textbf{Hard1} & \textbf{Hard2} & \textbf{Hard3} & \textbf{Harder} \\
\midrule
\multirow{8}{*}{\rotatebox[origin=c]{90}{Results}} &
GramNet & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 \\
& GeoEGNN & \cellcolor{blue!10} 0.998 & \cellcolor{blue!10} 0.97 & \cellcolor{blue!10} 0.85 & \cellcolor{blue!10} 0.899 \\
& EGNN & \cellcolor{red!10} 0.5 & \cellcolor{red!10} 0.5 & \cellcolor{red!10} 0.5 & \cellcolor{red!10} 0.5 \\
& LinearEGNN & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{red!10} 0.5 \\
& MACE & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 \\
& TFN & \cellcolor{red!10} 0.5 & \cellcolor{red!10} 0.5 & \cellcolor{blue!10} 0.55 & \cellcolor{red!10} 0.5 \\
& DimeNet & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 \\
& GVPGNN & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 \\
& $\us$ (ours) & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 & \cellcolor{green!10} 1.0 \\
\bottomrule
\end{tabular}
% }
\end{subtable}
\hfill
\label{tab:hard-pc results}
\end{table*}
\paragraph{Chemical property experiments with full statistics}
the table for the chemical property prediction tasks in the main text, Table \ref{mean}, was produced using the protocol from \cite{baselines}. In particular, each method was run with three different seeds. The seed that did best on the validation set was chosen, and its results on the test set were reported.
Table \ref{mean} presents each task's mean and standard deviation over $3$ seeds. As we can see, in most tasks, the standard deviation is rather low, and the results are qualitatively similar to those obtained in Table \ref{cpe}. In particular, in the tasks in the Kraken and BDE datasets, we outperform the competitors, often by a large margin. In the Drugs 75K, we obtain comparable, but somewhat higher, results than the best method (while in Table \ref{cpe}, in two of three Drugs-75K cases, our results were the best by a small margin).
\renewcommand{\arraystretch}{2.2} % Increase row
\begin{table}[t]
\centering
\caption{ Raw performance data (mean ± standard deviation) of 1D, 2D, 3D, and conformer ensemble MRL models all taken from \cite{baselines} in terms of absolute test error. Our model is in the 3D category, named $\us$.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{llccccccccc}
\toprule
\rowcolor{gray!25} Category & Model & IP & EA & $\chi$ & B\textsubscript{5} & L & BurB\textsubscript{5} & BurL & BDE \\
\midrule
\rowcolor{gray!10} 1D & Random Forest & 0.4987±0.0037 & 0.4747±0.0022 & 0.2732±0.0031 & 0.4760±0.0041 & 0.4303±0.0090 & 0.2758±0.0180 & 0.1521±0.0149 & 3.03±0.27 \\
\rowcolor{gray!10} 1D & LSTM & 0.4788±0.0024 & 0.4648±0.0002 & 0.2505±0.0050 & 0.4879±0.0280 & 0.5142±0.0411 & 0.2813±0.0041 & 0.1924±0.0028 & 2.82±0.07 \\
\rowcolor{gray!10} 1D & Transformer & 0.6617±0.0023 & 0.5850±0.0031 & 0.4073±0.0006 & 0.9611±0.0813 & 0.8389±0.0431 & 0.4929±0.0369 & 0.2781±0.0207 & 10.08±0.64 \\
\midrule
\rowcolor{snsblue!15} 2D & GIN & 0.4354±0.0029 & 0.4169±0.0032 & 0.2260±0.0017 & 0.3128±0.0264 & 0.4003±0.0341 & 0.1719±0.0031 & 0.1200±0.0040 & 2.63±0.22 \\
\rowcolor{snsblue!15} 2D & GIN-VN & 0.4361±0.0059 & 0.4169±0.0083 & 0.2267±0.0002 & 0.3567±0.0031 & 0.4344±0.0416 & 0.2422±0.0033 & 0.1741±0.0109 & 2.74±0.24 \\
\rowcolor{snsblue!15} 2D & ChemProp & 0.4595±0.0028 & 0.4417±0.0045 & 0.2441±0.0012 & 0.4850±0.0068 & 0.5452±0.0454 & 0.3002±0.0086 & 0.1948±0.0138 & 2.66±0.14 \\
\rowcolor{snsblue!15} 2D & GraphGPS & 0.4351±0.0049 & 0.4085±0.0055 & 0.2212±0.0054 & 0.3450±0.0324 & 0.4363±0.0133 & 0.2066±0.0115 & 0.1500±0.0138 & 2.48±0.19 \\
\midrule
\rowcolor{snsorange!15} 3D & SchNet & 0.4394±0.0062 & 0.4207±0.0021 & 0.2243±0.0089 & 0.3293±0.0068 & 0.5458±0.0341 & 0.2295±0.0111 & 0.1861±0.0095 & 2.54±0.00 \\
\rowcolor{snsorange!15} 3D & DimeNet++ & 0.4441±0.0087 & 0.4233±0.0072 & 0.2436±0.0075 & 0.3510±0.0107 & 0.4174±0.0397 & 0.2097±0.0160 & 0.1526±0.0072 & \underline{1.45±0.03} \\
\rowcolor{snsorange!15} 3D & GemNet & 0.4069±0.0007 & \textbf{0.3922±0.0024} & \textbf{0.1970±0.0039} & 0.2789±0.0125 & 0.3754±0.0086 & 0.1782±0.0099 & 0.1635±0.0063 & 1.65±0.30 \\
\rowcolor{snsorange!15} 3D & PaiNN & 0.4505±0.0041 & 0.4495±0.0054 & 0.2324±0.0040 & 0.3443±0.0388 & 0.4471±0.0324 & 0.2395±0.0176 & 0.1673±0.0088 & 2.12±0.09 \\
\rowcolor{snsorange!15} 3D & ClofNet & 0.4393±0.0084 & 0.4251±0.0066 & 0.2378±0.0020 & 0.4873±0.0093 & 0.6417±0.0362 & 0.2884±0.0166 & 0.2529±0.0052 & 2.60±0.02 \\
\rowcolor{snsorange!15} 3D & LEFTNet & 0.4174±0.0007 & 0.3964±0.0009 & 0.2083±0.0054 & 0.3072±0.0012 & 0.4493±0.0261 & 0.2176±0.0010 & 0.1486±0.0095 & 1.53±0.05 \\
\rowcolor{snsorange!15} 3D & $\us$(Ours) & 0.4251±0.0463 & \underline{0.3935±0.0361} & 0.2332±0.0248 & \textbf{0.1953±0.0068} & \textbf{0.1956±0.0264} & \textbf{0.1545±0.0019} & \textbf{0.0644±0.0017} & \textbf{0.4455±0.1305} \\
\midrule
\rowcolor{snsgreen!15} 3D + Sampling & SchNet & 0.4452±0.0080 & 0.4232±0.0042 & 0.2243±0.0022 & 0.3235±0.0147 & 0.4598±0.0041 & 0.2086±0.0111 & 0.1739±0.0142 & 1.97±0.01 \\
\rowcolor{snsgreen!15} 3D + Sampling & DimeNet++ & 0.4395±0.0032 & 0.4217±0.0040 & 0.2432±0.0048 & 0.3323±0.0320 & 0.4153±0.0208 & 0.2237±0.0122 & 0.1561±0.0241 & 1.47±0.03 \\
\rowcolor{snsgreen!15} 3D + Sampling & GemNet & \textbf{0.4066±0.0015} & 0.3910±0.0004 & \underline{0.2027±0.0013} & 0.2694±0.0221 & 0.3488±0.0252 & 0.1796±0.0098 & 0.1184±0.0033 & 1.60±0.10 \\
\rowcolor{snsgreen!15} 3D + Sampling & PaiNN & 0.4466±0.0087 & 0.4393±0.0045 & 0.2331±0.0037 & 0.3441±0.0161 & 0.4358±0.0343 & 0.2476±0.0070 & 0.1543±0.0022 & 1.92±0.01 \\
\rowcolor{snsgreen!15} 3D + Sampling & ClofNet & 0.4430±0.0074 & 0.4237±0.0005 & 0.2335±0.0090 & 0.4524±0.0935 & 0.5962±0.0074 & 0.2442±0.0109 & 0.1756±0.0112 & 2.51±0.23 \\
\rowcolor{snsgreen!15} 3D + Sampling & LEFTNet & \underline{0.4149±0.0019} & 0.3988±0.0048 & 0.2141±0.0084 & 0.2834±0.0068 & 0.4407±0.0531 & 0.2120±0.0097 & 0.1547±0.0101 & 1.52±0.00 \\
\midrule
\rowcolor{snspurple!10} SchNet & Mean & 0.4583±0.0019 & 0.4410±0.0018 & 0.2371±0.0098 & 0.3075±0.0151 & 0.4691±0.0234 & 0.2282±0.0206 & 0.1619±0.0062 & 2.53±0.02 \\
\rowcolor{snspurple!10} & DeepSet & 0.4537±0.0065 & 0.4396±0.0010 & 0.2385±0.0066 & 0.3105±0.0381 & 0.4322±0.0464 & 0.2249±0.0234 & 0.1535±0.0076 & 2.29±0.22 \\
\rowcolor{snspurple!10} & Attention & 0.4556±0.0075 & 0.4382±0.0125 & 0.2380±0.0007 & 0.2704±0.0187 & 0.4517±0.0132 & 0.2024±0.0183 & 0.1443±0.0043 & 2.64±0.00 \\
\midrule
\rowcolor{snspurple!10} DimeNet++ & Mean & 0.4488±0.0086 & 0.4340±0.0079 & 0.2425±0.0060 & 0.2630±0.0122 & 0.3828±0.0331 & 0.1960±0.0059 & 0.1268±0.0060 & 1.79±0.12 \\
\rowcolor{snspurple!10} & DeepSet & 0.4126±0.0076 & 0.3944±0.0034 & 0.2267±0.0047 & 0.2889±0.0069 & 0.3468±0.0090 & 0.1783±0.0110 & 0.1339±0.0087 & 1.75±0.01 \\
\rowcolor{snspurple!10} & Attention & 0.4188±0.0024 & 0.4030±0.0075 & 0.2325±0.0028 & 0.3718±0.0300 & 0.3628±0.0259 & 0.1899±0.0081 & 0.1185±0.0105 & 2.57±0.21 \\
\midrule
\rowcolor{snspurple!10} GemNet & Mean & 0.4505±0.0052 & 0.4334±0.0023 & 0.2289±0.0032 & 0.2635±0.0053 & 0.3753±0.0036 & 0.1671±0.0154 & 0.1587±0.0029 & 2.19±0.06 \\
\rowcolor{snspurple!10} & DeepSet & 0.4187±0.0022 & 0.4002±0.0012 & 0.2169±0.0036 & 0.2313±0.0026 & \underline{ 0.3386±0.0269} & \underline{ 0.1589±0.0068} & \underline{ 0.0947±0.0012} & 2.25±0.21 \\
\rowcolor{snspurple!10} & Attention & 0.4212±0.0017 & 0.4221±0.0097 & 0.2260±0.0056 & 0.2670±0.0026 & 0.3554±0.0147 & 0.1769±0.0153 & 0.1346±0.0075 & 2.68±0.02 \\
\midrule
\rowcolor{snspurple!10} PaiNN & Mean & 0.4591±0.0024 & 0.4425±0.0064 & 0.2360±0.0032 & 0.2877±0.0252 & 0.3950±0.0233 & 0.1817±0.0091 & 0.1472±0.0039 & 1.87±0.16 \\
\rowcolor{snspurple!10} & DeepSet & 0.4471±0.0071 & 0.4269±0.0033 & 0.2294±0.0065 & \underline{0.2225±0.0218} & 0.3619±0.0192 & 0.1693±0.0111 & 0.1324±0.0091 & 2.20±0.05 \\
\rowcolor{snspurple!10} & Attention & 0.4641±0.0016 & 0.4567±0.0094 & 0.2471±0.0049 & 0.3496±0.0140 & 0.4109±0.0167 & 0.2123±0.0005 & 0.1506±0.0029 & 2.23±0.12 \\
\midrule
\rowcolor{snspurple!10} ClofNet & Mean & 0.4536±0.0030 & 0.4301±0.0007 & 0.2365±0.0075 & 0.3555±0.0193 & 0.4485±0.0053 & 0.2473±0.0076 & 0.2022±0.0212 & 2.01±0.08 \\
\rowcolor{snspurple!10} & DeepSet & 0.4280±0.0056 & 0.4033±0.0024 & 0.2199±0.0073 & 0.3228±0.0020 & 0.4742±0.0161 & 0.2263±0.0249 & 0.1548±0.0039 & 2.35±0.04 \\
\rowcolor{snspurple!10} & Attention & 0.4330±0.0071 & 0.4107±0.0048 & 0.2220±0.0084 & 0.3734±0.0267 & 0.4963±0.0286 & 0.2178±0.0186 & 0.1690±0.0281 & 2.66±0.14 \\
\midrule
\rowcolor{snspurple!10} LEFTNet & Mean & 0.4402±0.0062 & 0.4267±0.0026 & 0.2183±0.0007 & 0.2949±0.0001 & 0.3643±0.0352 & 0.2098±0.0146 & 0.1386±0.0007 & 2.04±0.00 \\
\rowcolor{snspurple!10} & DeepSet & 0.4167±0.0043 & 0.3953±0.0000 & 0.2069±0.0022 & 0.2644±0.0130 & 0.3866±0.0270 & 0.2023±0.0026 & 0.1441±0.0042 & 2.51±0.30 \\
\rowcolor{snspurple!10} & Attention & 0.4229±0.0059 & 0.4067±0.0047 & 0.2198±0.0011 & 0.3161±0.0116 & 0.4324±0.0292 & 0.2017±0.0023 & 0.1508±0.0075 & 2.63±0.15 \\
\midrule
\bottomrule
\end{tabular}%
}
\label{mean}
\end{table}
\paragraph{Time measurements}
To compare our running time with other models, we measured the execution time for each model on the 12-chain task with all models using 128 hidden feature size. The results \ref{tab:time_com} show that our running time is on par with the competitors.
\paragraph{Norm Accuracies}
To heuristically estimate ‘how generic’ the molecules in the chemical property datasets \citep{Drugs,Kraken,BDE} we considered are, we computed all norms of atoms in each (centralized) molecule and counted the number of unique norms relative to the total number of points. We considered a norm unique if its distance from all other norms exceeded a specified threshold $\mu$, and considered various threshold values. The results \ref{tab:norms_repit} indicate that the norms exhibit considerable variation, suggesting that the molecules are close to being generic. This supports the conjecture stated in the main text that the surprising success of our simple method, compared to more expressive methods, is related to the near-genericity of the molecules in these datasets.
\begin{table*}[ht]
\centering
\caption{Running times of different methods on the 12-chain problem.}
\begin{tabular}{lc}
\toprule
\textbf{Method} & \textbf{Time (in secs)} \\
\midrule
SchNet \cite{SchNet} & 53.3 \\
\hline
DimeNet \cite{DimeNet} & 222.6 \\
\hline
Sphere-Net \cite{SphereNet} & 360.9 \\
\hline
EGNN \cite{egnn} & 98.8 \\
\hline
GVP \cite{gvp-egnn} & 222.4 \\
\hline
TFN \cite{TFN} & 268.1 \\
\hline
MACE \cite{mace} & 440.6 \\
\hline
\textbf{$\us$ (ours)} & 112.0 \\
\bottomrule
\end{tabular}
\label{tab:time_com}
\end{table*}
\begin{table*}[ht]
\centering
\caption{Norms repetition percentages at various thresholds for each dataset.}
\begin{tabular}{lccc}
\toprule
\textbf{Threshold} & \textbf{Drugs} & \textbf{Kraken} & \textbf{BDE} \\
\midrule
0.0 & 1.0 & 1.0 & 1.0 \\
\hline
1e-4 & 0.9967 & 0.9804 & 0.9950 \\
\hline
1e-3 & 0.9678 & 0.9139 & 0.9573 \\
\hline
1e-2 & 0.7230 & 0.5217 & 0.6698 \\
\bottomrule
\end{tabular}
\label{tab:norms_repit}
\end{table*}
\end{document}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2023
% ready for submission
% \usepackage{neurips_2024}
%\usepackage[natbib]{iclr2025_conference}
\usepackage{iclr2025_conference,times}
\usepackage{hyperref}
\usepackage{url}
%\usepackage[final]{neurips_2024}
%\usepackage[nonatbib]{neurips_2024}
%\usepackage{authblk}
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
%\usepackage{arxiv}
\usepackage{wrapfig} %for wrapping figures
\usepackage{thm-restate}
\usepackage{microtype}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{nicematrix} % Tal: To be used in tables
\usepackage{algpseudocode}
%\usepackage[showframe]{geometry}
\usepackage{algorithm}
\usepackage{xspace}
\usepackage{enumitem}
\usepackage{amssymb}% http://ctan.org/pkg/amssymb
\usepackage{pifont}% http://ctan.org/pkg/pifont
\usepackage{booktabs}
\usepackage{caption}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2022} with \usepackage[nohyperref]{icml2022} above.
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor}
% Attempt to make hyperref and algorithmic work together better:
% Use the following line for the initial blind version submitted for review:
%\usepackage{icml2022}
% If accepted, instead use the following line for the camera-ready submission:
\usepackage{booktabs} % For professional table lines
\usepackage{colortbl} % For cell background colors
\usepackage{xcolor} % For custom colors
% Define custom colors
\definecolor{gray}{gray}{0.9}
\definecolor{lightgray}{gray}{0.95}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{float}
\usepackage{booktabs} % For professional table lines
\usepackage{colortbl} % For cell background colors
\usepackage{tabularray} % For tblr environment
\usepackage{xcolor} % For custom colors
\usepackage[normalem]{ulem} % For underline in table
% Define custom colors
\definecolor{gray}{gray}{0.9}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%\usepackage[pdftex]{graphicx}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
%%%%%%%%%
\newtheorem{lemma}[theorem]{Lemma}
\usepackage{mathtools}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\usepackage[textsize=tiny]{todonotes}
\usepackage{float}
\DeclareMathOperator{\Ima}{Im}
\DeclareMathOperator{\rank}{rank}
\newcommand{\lms}{ \{\!\!\{ }
\newcommand{\rms}{ \}\!\!\} }
\newcommand{\sglobal}{s^{\mathrm{global}}}
\newcommand{\sglobalhat}{\hat{s}^{\mathrm{global}}}
\newcommand{\gennet}{\textsc{EGenNet}}
\newcommand{\us}{\gennet}
\newcommand{\embed}{\text{\textbf{Embed}}}
\newcommand{\hash}{\embed}
\newcommand{\SES}{\mathcal{SO}[d,n]}
\newcommand{\SE}{\mathcal{SO}(d)}
\newcommand{\SEthree}{\mathcal{SO}(3)}
\newcommand{\SESthree}{\mathcal{SO}[3,n]}
\newcommand{\ES}{\mathcal{O}[d,n]}
\newcommand{\ESthree}{\mathcal{O}[3,n]}
\newcommand{\embedomega}{\embed_{\Omega}}
\newcommand{\embedtheta}{\embed_{\Theta}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Y}{\mathcal{Y}}
\newcommand{\SO}{\mathcal{SO}}
\renewcommand{\O}{\mathcal{O}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\N}{\mathcal{N}}
\newcommand{\C}{\mathbf{C}}
\newcommand{\I}{\mathbf{i}}
\newcommand{\ggnn}{\mathrm{GNN}_\aleph}
\newcommand{\Fone}{F^{\textup{1-Geo}}}
%\newcommand{\egnn}{\mathrm{EGNN}_\aleph}%
\usepackage{fontawesome5}
\usepackage{titletoc}
\usepackage[toc,page,header]{appendix}
\usepackage{caption}
\usepackage{tikz}
\usetikzlibrary{patterns}
\newcommand{\hatchedtext}[1]{
\tikz[baseline=(word.base)] \node[anchor=base,inner sep=0.3ex,outer sep=0ex, fill=gray!5, postaction={pattern=north east lines, pattern color=gray!50}] (word) {#1};
}
\usepackage{graphicx}
\newcommand{\minus}{\scalebox{0.75}[1.0]{$-$}}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2023}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2023}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2023}
\usepackage{amsmath,amsfonts,bm}
\usepackage{mathtools}
%\urlstyle{same}
\usepackage{xcolor}
\hypersetup{
% colorlinks=true,
linkcolor=blue,
filecolor=magenta,
urlcolor=blue,
pdftitle={Overleaf Example},
pdfpagemode=FullScreen,
}
\usepackage{booktabs} % For professional-looking table lines
\usepackage{colortbl} % For cell background colors
\usepackage{graphicx} % For \resizebox
\usepackage{xcolor} % For color definitions
\usepackage{array}
\usepackage{tabularray}
% Define a custom color for links
\usepackage{booktabs} % Professional table lines
\usepackage{colortbl} % Cell background colors
\usepackage{xcolor} % For custom colors
\usepackage{graphicx} % For resizing tables
\usepackage{array} % Improved alignment in tables
% Define custom colors
\definecolor{snsblue}{RGB}{31,119,180}
\definecolor{snsorange}{RGB}{255,127,14}
\definecolor{snsgreen}{RGB}{44,160,44}
\definecolor{snsred}{RGB}{214,39,40}
\usepackage{booktabs}
\usepackage{colortbl}
\usepackage{xcolor}
\usepackage{graphicx}
\usepackage{array}
% Define custom colors
\definecolor{snspurple}{RGB}{148,103,189}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
%\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\newcommand{\CC}{\mathbb{C}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\Z}{\mathcal{Z}}
\newcommand{\B}{\mathcal{B}}
\newcommand{\bigO}{\mathcal{O}}
\newcommand{\mbf}{\mathbf}
\newcommand{\dX}{D_{\X}}
\newcommand{\Dint}{\dX}
\newcommand{\enc}{\mathrm{enc}}
\newcommand{\ReLU}{\mathrm{ReLU}}
\newcommand{\relu}{\mathrm{ReLU}}
\renewcommand{\k}{\kappa}
%mathcal commands
\newcommand{\F}{\mathcal{F}}
\newcommand{\M}{\mathbb{M}}
\newcommand{\W}{\mathbb{W}}
\newcommand{\Gnd}{\G_{\leq n}(\RR^d)}
\newcommand{\V}{\mathcal{V}}
\newcommand{\E}{\mathcal{E}}
\newcommand{\K}{\mathcal{K}}
%\newcommand{\T}{\mathcal{T}}
\newcommand{\D}{\mathcal{D}}
\renewcommand{\P}{\mathcal{P}}
\renewcommand{\H}{\mathcal{H}}
\renewcommand{\L}{\mathcal{L}}
\newcommand{\A}{\mathcal{A}}
\renewcommand{\S}{\mathcal{S}}
%mathbb commands
\newcommand{\bbm}{\boldsymbol{m}}
\newcommand{\bh}{\boldsymbol{h}}
\newcommand{\bX}{\boldsymbol{X}}
\newcommand{\bY}{\boldsymbol{Y}}
\newcommand{\bz}{\boldsymbol{z}}
\newcommand{\bC}{\boldsymbol{C}}
\newcommand{\bzero}{\boldsymbol{0}}
\newcommand{\btheta}{\boldsymbol{\theta}}
\newcommand{\bth}{\btheta}
%these should be removed
%\newcommand{\Snd}{\S_{\leq n}(\RR^d)}
%\newcommand{\SnO}{\S_{\leq n}(\Omega)}
%\newcommand{\SnnO}{\S_n(\Omega)}
%\newcommand{\MnO}{\M_{\leq n}(\Omega)}
\newcommand{\SnR}{\S_{\leq n}(\RR)}
\newcommand{\SnSig}{\S_{\leq n}(\Sigma)}
\newcommand{\MnSig}{\mathcal{M}_{\leq n}(\Sigma)}
\newcommand{\Mntheta}{\mathcal{M}_{\leq n}(\Omega(\theta_1))}
\newcommand{\PnK}{\P_{\leq n}(K)}
\newcommand{\Sneq}{\S_{\leq n}(\Omega)}
\newcommand{\Sndeq}{\S_{\leq n}(\RR^d)}
\newcommand{\Sn}{\S_{\leq n}(\Omega)}
\newcommand{\GnO}{\mathcal{G}_{\leq n}(\Omega)}
\newcommand{\GnSig}{\mathcal{G}_{\leq n}(\Sigma)}
\newcommand{\Snd}{\S_{\leq n}(\RR^d)}
%\newcommand{\Snnd}{\S_n(\RR^d)}
%\newcommand{\SnnR}{\S_n(\RR)}
\newcommand{\MnR}{\mathcal{M}_{\leq n}(\RR)}
\newcommand{\Mnd}{\mathcal{M}_{\leq n}(\RR^d)}
\newcommand{\Mn}{\mathcal{M}_{\leq n}(\Omega)}
\newcommand{\ba}{\bm{a}}
\newcommand{\bb}{\bm{b}}
\newcommand{\bpsi}{\bm{\psi}}
\newcommand{\rowsort}{\mathbf{rowsort}}
\newcommand{\columnsort}{\mathbf{columnsort}}
\newcommand{\poly}{\mathrm{poly}}
\newcommand{\Dparam}{D_{param}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
\newcommand{\innerproduct}[2]{\langle #1, #2 \rangle}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\DeclarePairedDelimiter{\norm}{\lVert}{\rVert}
% Groups and group actions
\def\fG{{\mathfrak{G}}}
\def\fg{{\mathfrak{g}}}
\newcommand{\defeq}{\vcentcolon=}
\usepackage{soul}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts % hyperlinks
\usepackage{multirow}
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
%\documentclass{article}
\usepackage[english]{babel}
\usepackage{soul, color}
\newcommand{\Note}[1]{}
\renewcommand{\Note}[1]{#1} % comment out this re-definition to suppress all Notes
\newcommand{\NoteSigned}[3]{{\textcolor{#2}{\Note{\textbf{[#1: #3]}}}}}
% Alternative with highlighter color rather than text colour
%\renewcommand{\NoteSigned}[3]{{\sethlcolor{#2}\Note{\hl{#1: #3}}}}
% Specific author comments
\newcommand{\pl}[1]{\NoteSigned{PL}{blue}{#1}} % use for notes from Pietro
\newcommand{\tc}[1]{\NoteSigned{TC}{green}{#1}}
\newcommand{\cb}[1]{\NoteSigned{CB}{brown}{#1}}
\newcommand{\cj}[1]{\NoteSigned{CJ}{purple}{#1}}
\newcommand{\sm}[1]{\NoteSigned{SM}{magenta}{#1}}
% Inline todo: Flag something in the text to be done
\newcommand{\itodo}[1]{\NoteSigned{TODO}{red}{#1}}
% Preliminary: Flag preliminary statements or results
\newcommand{\prelim}[1]{\NoteSigned{Prelim}{grey}{#1}}
% Convenient other commands
\newcommand{\sowhat}{$\rightarrow$~}
% Colour commands
\newcommand{\red}[1]{\textcolor{red}{#1}}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\newcommand{\black}[1]{\textcolor{black}{#1}}
\newcommand{\green}[1]{\textcolor{green}{#1}}
\newcommand{\teal}[1]{\textcolor{teal}{#1}}
\newcommand{\yellow}[1]{\textcolor{yellow}{#1}}
\newcommand{\magenta}[1]{\textcolor{magenta}{#1}}
\newcommand{\gray}[1]{\textcolor{gray}{#1}}
% Variant of the TODO box for addressing reviewer comments
\newcommand{\bluebox}[1]{\todo[color=blue!20!white]{#1}}
\usepackage[toc,page,header]{appendix}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{authblk}
\usepackage{tikz}
\usetikzlibrary{patterns}
\theoremstyle{remark}
\usepackage{xcolor,colortbl}
\usepackage{booktabs}
\usepackage{siunitx}
\usepackage{tabularray}
\UseTblrLibrary{booktabs,siunitx}
\usepackage{amsmath}
\usepackage{csquotes}
\usepackage{amssymb}
\newcommand{\GG}{\mathbb{G}}
\newcommand{\Gd}{\mathbb{G}(d)}
\newcommand{\GdN}{\mathbb{G}_{N}(d)}
\newcommand{\GdNdistinct}{\mathbb{G}_{N}^{\mathrm{distinct}}(d)}
\newcommand{\GdNgeneric}{\mathbb{G}_{\mathrm{generic}}(d,N)}
\newcommand{\vvv}{{\vv}}
\newcommand{\vvx}{{\vx}}
\newcommand{\finv}{f_{\mathrm{inv}}}
\newcommand{\pair}{\mathbb{G}^{\text{pair}}}
\usepackage{array}
\usepackage{booktabs}
\usepackage{tabularray}
\begin{thebibliography}{64}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Aamand et~al.(2022)Aamand, Chen, Indyk, Narayanan, Rubinfeld, Schiefer, Silwal, and Wagner]{aamand2022exponentially}
Anders Aamand, Justin Chen, Piotr Indyk, Shyam Narayanan, Ronitt Rubinfeld, Nicholas Schiefer, Sandeep Silwal, and Tal Wagner.
\newblock Exponentially improving the complexity of simulating the weisfeiler-lehman test with graph neural networks.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 27333--27346, 2022.
\bibitem[Alon \& Yahav(2020)Alon and Yahav]{alon2020bottleneck}
Uri Alon and Eran Yahav.
\newblock On the bottleneck of graph neural networks and its practical implications.
\newblock \emph{arXiv preprint arXiv:2006.05205}, 2020.
\bibitem[Amir et~al.(2023)Amir, Gortler, Avni, Ravina, and Dym]{Neural_Injective}
Tal Amir, Steven Gortler, Ilai Avni, Ravina Ravina, and Nadav Dym.
\newblock Neural injective functions for multisets, measures and graphs via a finite witness theorem.
\newblock In A.~Oh, T.~Neumann, A.~Globerson, K.~Saenko, M.~Hardt, and S.~Levine (eds.), \emph{Advances in Neural Information Processing Systems}, volume~36, pp.\ 42516--42551. Curran Associates, Inc., 2023.
\bibitem[Anderson et~al.(2019)Anderson, Hy, and Kondor]{cormorant}
Brandon Anderson, Truong~Son Hy, and Risi Kondor.
\newblock Cormorant: Covariant molecular neural networks.
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer, F.~d\textquotesingle Alch\'{e}-Buc, E.~Fox, and R.~Garnett (eds.), \emph{Advances in Neural Information Processing Systems}, volume~32. Curran Associates, Inc., 2019.
\newblock URL \url{https://proceedings.neurips.cc/paper_files/paper/2019/file/03573b32b2746e6e8ca98b9123f2249b-Paper.pdf}.
\bibitem[Axelrod \& Gomez-Bombarelli(2022)Axelrod and Gomez-Bombarelli]{Drugs}
Simon Axelrod and Rafael Gomez-Bombarelli.
\newblock Geom, energy-annotated molecular conformations for property prediction and molecular generation.
\newblock \emph{Scientific Data}, 9\penalty0 (1):\penalty0 185, 2022.
\bibitem[Batatia et~al.(2022)Batatia, Kovacs, Simm, Ortner, and Cs{\'a}nyi]{mace}
Ilyes Batatia, David~P Kovacs, Gregor Simm, Christoph Ortner, and G{\'a}bor Cs{\'a}nyi.
\newblock Mace: Higher order equivariant message passing neural networks for fast and accurate force fields.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 11423--11436, 2022.
\bibitem[B{\"o}kman et~al.(2022)B{\"o}kman, Kahl, and Flinth]{zz}
Georg B{\"o}kman, Fredrik Kahl, and Axel Flinth.
\newblock Zz-net: A universal rotation equivariant architecture for 2d point clouds.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 10976--10985, 2022.
\bibitem[Breiman(2001)]{Random_forest}
Leo Breiman.
\newblock Random forests.
\newblock \emph{Machine learning}, 45:\penalty0 5--32, 2001.
\bibitem[Cayton et~al.(2008)]{manifold}
Lawrence Cayton et~al.
\newblock \emph{Algorithms for manifold learning}.
\newblock eScholarship, University of California, 2008.
\bibitem[Cheung \& Whiteley(2005)Cheung and Whiteley]{power-graph2}
Matthew Cheung and Walter Whiteley.
\newblock Transfer of global rigidity results among dimensions: Graph powers and coning.
\newblock \emph{Preprint, York University}, 2005.
\bibitem[Delle~Rose et~al.(2023)Delle~Rose, Kozachinskiy, Rojas, Petrache, and Barcel{\'o}]{three_iterations}
Valentino Delle~Rose, Alexander Kozachinskiy, Crist{\'o}bal Rojas, Mircea Petrache, and Pablo Barcel{\'o}.
\newblock Three iterations of (d- 1)-wl test distinguish non isometric clouds of d-dimensional points.
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\bibitem[Deng et~al.(2021)Deng, Litany, Duan, Poulenard, Tagliasacchi, and Guibas]{deng2021vector}
Congyue Deng, Or~Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas~J Guibas.
\newblock Vector neurons: A general framework for so (3)-equivariant networks.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 12200--12209, 2021.
\bibitem[Du et~al.(2022)Du, Zhang, Du, Meng, Chen, Zheng, Shao, and Liu]{clofnet}
Weitao Du, He~Zhang, Yuanqi Du, Qi~Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie-Yan Liu.
\newblock Se (3) equivariant graph neural networks with complete local frames.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 5583--5608. PMLR, 2022.
\bibitem[Du et~al.(2024)Du, Wang, Feng, Wang, Ji, Gomes, Ma, et~al.]{leftnet}
Yuanqi Du, Limei Wang, Dieqiao Feng, Guifeng Wang, Shuiwang Ji, Carla~P Gomes, Zhi-Ming Ma, et~al.
\newblock A new perspective on building efficient and expressive 3d equivariant graph neural networks.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Dym \& Gortler(2024)Dym and Gortler]{gortler_and_dym}
Nadav Dym and Steven~J Gortler.
\newblock Low-dimensional invariant embeddings for universal geometric learning.
\newblock \emph{Foundations of Computational Mathematics}, pp.\ 1--41, 2024.
\bibitem[Dym \& Maron(2020)Dym and Maron]{dym2020universality}
Nadav Dym and Haggai Maron.
\newblock On the universality of rotation equivariant point cloud networks.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Finkelshtein et~al.(2022)Finkelshtein, Baskin, Maron, and Dym]{finkelshtein2022simple}
Ben Finkelshtein, Chaim Baskin, Haggai Maron, and Nadav Dym.
\newblock A simple and universal rotation equivariant point-cloud network.
\newblock In \emph{Topological, Algebraic and Geometric Learning Workshops 2022}, pp.\ 107--115. PMLR, 2022.
\bibitem[Fuchs et~al.(2020)Fuchs, Worrall, Fischer, and Welling]{se3transformer}
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling.
\newblock Se (3)-transformers: 3d roto-translation equivariant attention networks.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 1970--1981, 2020.
\bibitem[Garamv{\"o}lgyi et~al.(2022)Garamv{\"o}lgyi, Gortler, and Jord{\'a}n]{reconstructible}
D{\'a}niel Garamv{\"o}lgyi, Steven~J Gortler, and Tibor Jord{\'a}n.
\newblock Globally rigid graphs are fully reconstructible.
\newblock In \emph{Forum of Mathematics, Sigma}, volume~10, pp.\ e51. Cambridge University Press, 2022.
\bibitem[Gasteiger et~al.(2020{\natexlab{a}})Gasteiger, Giri, Margraf, and G{\"u}nnemann]{DimeNet++}
Johannes Gasteiger, Shankari Giri, Johannes~T Margraf, and Stephan G{\"u}nnemann.
\newblock Fast and uncertainty-aware directional message passing for non-equilibrium molecules.
\newblock \emph{arXiv preprint arXiv:2011.14115}, 2020{\natexlab{a}}.
\bibitem[Gasteiger et~al.(2020{\natexlab{b}})Gasteiger, Gro{\ss}, and G{\"u}nnemann]{DimeNet}
Johannes Gasteiger, Janek Gro{\ss}, and Stephan G{\"u}nnemann.
\newblock Directional message passing for molecular graphs.
\newblock \emph{arXiv preprint arXiv:2003.03123}, 2020{\natexlab{b}}.
\bibitem[Gasteiger et~al.(2021)Gasteiger, Becker, and G{\"u}nnemann]{gemnet}
Johannes Gasteiger, Florian Becker, and Stephan G{\"u}nnemann.
\newblock Gemnet: Universal directional graph neural networks for molecules.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 6790--6802, 2021.
\bibitem[Gensch et~al.(2022)Gensch, dos Passos~Gomes, Friederich, Peters, Gaudin, Pollice, Jorner, Nigam, Lindner-D’Addario, Sigman, et~al.]{Kraken}
Tobias Gensch, Gabriel dos Passos~Gomes, Pascal Friederich, Ellyn Peters, Th{\'e}ophile Gaudin, Robert Pollice, Kjell Jorner, AkshatKumar Nigam, Michael Lindner-D’Addario, Matthew~S Sigman, et~al.
\newblock A comprehensive discovery platform for organophosphorus ligands for catalysis.
\newblock \emph{Journal of the American Chemical Society}, 144\penalty0 (3):\penalty0 1205--1217, 2022.
\bibitem[Gilmer et~al.(2017)Gilmer, Schoenholz, Riley, Vinyals, and Dahl]{gilmer2017neural}
Justin Gilmer, Samuel~S Schoenholz, Patrick~F Riley, Oriol Vinyals, and George~E Dahl.
\newblock Neural message passing for quantum chemistry.
\newblock In \emph{International conference on machine learning}, pp.\ 1263--1272. PMLR, 2017.
\bibitem[Gortler et~al.(2010)Gortler, Healy, and Thurston]{gortler2010characterizing}
Steven~J Gortler, Alexander~D Healy, and Dylan~P Thurston.
\newblock Characterizing generic global rigidity.
\newblock \emph{American Journal of Mathematics}, 132\penalty0 (4):\penalty0 897--939, 2010.
\bibitem[Han et~al.(2024)Han, Cen, Wu, Li, Kong, Jiao, Yu, Xu, Wu, Wang, Xu, Wei, Liu, Rong, and Huang]{han2024survey}
Jiaqi Han, Jiacheng Cen, Liming Wu, Zongzhao Li, Xiangzhe Kong, Rui Jiao, Ziyang Yu, Tingyang Xu, Fandi Wu, Zihe Wang, Hongteng Xu, Zhewei Wei, Yang Liu, Yu~Rong, and Wenbing Huang.
\newblock A survey of geometric graph neural networks: Data structures, models and applications, 2024.
\bibitem[Heid et~al.(2023)Heid, Greenman, Chung, Li, Graff, Vermeire, Wu, Green, and McGill]{ChemProp}
Esther Heid, Kevin~P Greenman, Yunsie Chung, Shih-Cheng Li, David~E Graff, Florence~H Vermeire, Haoyang Wu, William~H Green, and Charles~J McGill.
\newblock Chemprop: A machine learning package for chemical property prediction.
\newblock \emph{Journal of Chemical Information and Modeling}, 64\penalty0 (1):\penalty0 9--17, 2023.
\bibitem[Hochreiter \& Schmidhuber(1997)Hochreiter and Schmidhuber]{LSTM}
Sepp Hochreiter and J{\"u}rgen Schmidhuber.
\newblock Long short-term memory.
\newblock \emph{Neural computation}, 9\penalty0 (8):\penalty0 1735--1780, 1997.
\bibitem[Hordan et~al.(2023)Hordan, Amir, Gortler, and Dym]{gramnet}
Snir Hordan, Tal Amir, Steven~J Gortler, and Nadav Dym.
\newblock Complete neural networks for euclidean graphs.
\newblock \emph{arXiv preprint arXiv:2301.13821}, 2023.
\bibitem[Hordan et~al.(2024)Hordan, Amir, and Dym]{hordan2}
Snir Hordan, Tal Amir, and Nadav Dym.
\newblock Weisfeiler leman for euclidean equivariant machine learning.
\newblock \emph{arXiv e-prints}, pp.\ arXiv--2402, 2024.
\bibitem[Hu et~al.(2020)Hu, Fey, Zitnik, Dong, Ren, Liu, Catasta, and Leskovec]{GIN-VN}
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.
\newblock Open graph benchmark: Datasets for machine learning on graphs.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 22118--22133, 2020.
\bibitem[Jing et~al.(2020)Jing, Eismann, Suriana, Townshend, and Dror]{gvp-egnn}
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John~Lamarre Townshend, and Ron Dror.
\newblock Learning from protein structure with geometric vector perceptrons.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Jogl et~al.(2023)Jogl, Thiessen, and G{\"a}rtner]{jogl2023expressivitypreserving}
Fabian Jogl, Maximilian Thiessen, and Thomas G{\"a}rtner.
\newblock Expressivity-preserving {GNN} simulation.
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\newblock URL \url{https://openreview.net/forum?id=ytTfonl9Wd}.
\bibitem[Jord{\'a}n \& Tanigawa(2022)Jord{\'a}n and Tanigawa]{power-graph}
Tibor Jord{\'a}n and Shin-ichi Tanigawa.
\newblock Globally rigid powers of graphs.
\newblock \emph{Journal of Combinatorial Theory, Series B}, 155:\penalty0 111--140, 2022.
\bibitem[Joshi et~al.(2023)Joshi, Bodnar, Mathis, Cohen, and Lio]{on_the_expressive_power}
Chaitanya~K Joshi, Cristian Bodnar, Simon~V Mathis, Taco Cohen, and Pietro Lio.
\newblock On the expressive power of geometric graph neural networks.
\newblock \emph{arXiv preprint arXiv:2301.09308}, 2023.
\bibitem[Kurlin(2023)]{kurlin2023polynomial}
Vitaliy Kurlin.
\newblock Polynomial-time algorithms for continuous metrics on atomic clouds of unordered points.
\newblock \emph{Match: Communications in Mathematical and in Computer Chemistry}, 2023.
\bibitem[Levy et~al.(2023)Levy, Kaba, Gonzales, Miret, and Ravanbakhsh]{levy2023using}
Daniel Levy, S{\'e}kou-Oumar Kaba, Carmelo Gonzales, Santiago Miret, and Siamak Ravanbakhsh.
\newblock Using multiple vector channels improves e (n)-equivariant graph neural networks.
\newblock \emph{arXiv preprint arXiv:2309.03139}, 2023.
\bibitem[Li et~al.(2024{\natexlab{a}})Li, Wang, Huang, and Zhang]{disgnn}
Zian Li, Xiyuan Wang, Yinan Huang, and Muhan Zhang.
\newblock Is distance matrix enough for geometric deep learning?
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024{\natexlab{a}}.
\bibitem[Li et~al.(2024{\natexlab{b}})Li, Wang, Kang, and Zhang]{Invariant_models}
Zian Li, Xiyuan Wang, Shijia Kang, and Muhan Zhang.
\newblock On the completeness of invariant geometric deep learning models.
\newblock \emph{arXiv preprint arXiv:2402.04836}, 2024{\natexlab{b}}.
\bibitem[Liu et~al.(2021)Liu, Wang, Liu, Lin, Zhang, Oztekin, and Ji]{SphereNet}
Yi~Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji.
\newblock Spherical message passing for 3d molecular graphs.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Loshchilov \& Hutter(2017)Loshchilov and Hutter]{AdamW}
Ilya Loshchilov and Frank Hutter.
\newblock Decoupled weight decay regularization.
\newblock \emph{arXiv preprint arXiv:1711.05101}, 2017.
\bibitem[Meyer et~al.(2018)Meyer, Sawatlon, Heinen, Von~Lilienfeld, and Corminboeuf]{BDE}
Benjamin Meyer, Boodsarin Sawatlon, Stefan Heinen, O~Anatole Von~Lilienfeld, and Cl{\'e}mence Corminboeuf.
\newblock Machine learning meets volcano plots: computational discovery of cross-coupling catalysts.
\newblock \emph{Chemical science}, 9\penalty0 (35):\penalty0 7069--7077, 2018.
\bibitem[Mityagin(2020)]{mityagin2020zero}
BS~Mityagin.
\newblock The zero set of a real analytic function.
\newblock \emph{Mathematical Notes}, 107:\penalty0 529--530, 2020.
\bibitem[Nigam et~al.(2020)Nigam, Pozdnyakov, and Ceriotti]{nigam2020recursive}
Jigyasa Nigam, Sergey Pozdnyakov, and Michele Ceriotti.
\newblock Recursive evaluation and iterative contraction of n-body equivariant features.
\newblock \emph{The Journal of chemical physics}, 153\penalty0 (12), 2020.
\bibitem[Pinkus(1999)]{approximations}
Allan Pinkus.
\newblock Approximation theory of the mlp model in neural networks.
\newblock \emph{Acta numerica}, 8:\penalty0 143--195, 1999.
\bibitem[Pozdnyakov \& Ceriotti(2022)Pozdnyakov and Ceriotti]{gcnn}
Sergey Pozdnyakov and Michele Ceriotti.
\newblock Incompleteness of graph convolutional neural networks for points clouds in three dimensions.
\newblock 01 2022.
\newblock URL \url{arXiv:2201.07136v4}.
\bibitem[Pozdnyakov et~al.(2020)Pozdnyakov, Willatt, Bart\'ok, Ortner, Cs\'anyi, and Ceriotti]{PhysRevLett.125.166001}
Sergey~N. Pozdnyakov, Michael~J. Willatt, Albert~P. Bart\'ok, Christoph Ortner, G\'abor Cs\'anyi, and Michele Ceriotti.
\newblock Incompleteness of atomic structure representations.
\newblock \emph{Phys. Rev. Lett.}, 125:\penalty0 166001, Oct 2020.
\newblock \doi{10.1103/PhysRevLett.125.166001}.
\newblock URL \url{https://link.aps.org/doi/10.1103/PhysRevLett.125.166001}.
\bibitem[Puny et~al.(2021)Puny, Atzmon, Smith, Misra, Grover, Ben-Hamu, and Lipman]{puny2021frame}
Omri Puny, Matan Atzmon, Edward~J Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman.
\newblock Frame averaging for invariant and equivariant network design.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Ramp{\'a}{\v{s}}ek et~al.(2022)Ramp{\'a}{\v{s}}ek, Galkin, Dwivedi, Luu, Wolf, and Beaini]{GraphGPS}
Ladislav Ramp{\'a}{\v{s}}ek, Michael Galkin, Vijay~Prakash Dwivedi, Anh~Tuan Luu, Guy Wolf, and Dominique Beaini.
\newblock Recipe for a general, powerful, scalable graph transformer.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 14501--14515, 2022.
\bibitem[Satorras et~al.(2021)Satorras, Hoogeboom, and Welling]{egnn}
V{\i}ctor~Garcia Satorras, Emiel Hoogeboom, and Max Welling.
\newblock E (n) equivariant graph neural networks.
\newblock In \emph{International conference on machine learning}, pp.\ 9323--9332. PMLR, 2021.
\bibitem[Sch{\"u}tt et~al.(2021)Sch{\"u}tt, Unke, and Gastegger]{painn}
Kristof Sch{\"u}tt, Oliver Unke, and Michael Gastegger.
\newblock Equivariant message passing for the prediction of tensorial properties and molecular spectra.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 9377--9388. PMLR, 2021.
\bibitem[Sch{\"u}tt et~al.(2018)Sch{\"u}tt, Sauceda, Kindermans, Tkatchenko, and M{\"u}ller]{SchNet}
Kristof~T Sch{\"u}tt, Huziel~E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R M{\"u}ller.
\newblock Schnet--a deep learning architecture for molecules and materials.
\newblock \emph{The Journal of Chemical Physics}, 148\penalty0 (24), 2018.
\bibitem[Shapeev(2016)]{shapeev}
Alexander~V. Shapeev.
\newblock Moment tensor potentials: A class of systematically improvable interatomic potentials.
\newblock \emph{Multiscale Modeling \& Simulation}, 14\penalty0 (3):\penalty0 1153--1173, 2016.
\newblock \doi{10.1137/15M1054183}.
\newblock URL \url{https://doi.org/10.1137/15M1054183}.
\bibitem[Thomas et~al.(2018)Thomas, Smidt, Kearnes, Yang, Li, Kohlhoff, and Riley]{TFN}
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li~Li, Kai Kohlhoff, and Patrick Riley.
\newblock Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.
\newblock \emph{arXiv preprint arXiv:1802.08219}, 2018.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]{Transformer}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan~N Gomez, Lukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Veli{\v{c}}kovi{\'c}(2022)]{velivckovic2022message}
Petar Veli{\v{c}}kovi{\'c}.
\newblock Message passing all the way up.
\newblock \emph{arXiv preprint arXiv:2202.11097}, 2022.
\bibitem[Villar et~al.(2021)Villar, Hogg, Storey-Fisher, Yao, and Blum-Smith]{villar2021scalars}
Soledad Villar, David~W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith.
\newblock Scalars are universal: Equivariant machine learning, structured like classical physics.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 28848--28863, 2021.
\bibitem[Wang et~al.(2022)Wang, Liu, Lin, Liu, and Ji]{comenet}
Limei Wang, Yi~Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji.
\newblock Comenet: Towards complete and efficient message passing for 3d molecular graphs.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 650--664, 2022.
\bibitem[Wang et~al.(2024)Wang, Hsu, Baker, Bertozzi, Xin, and Wang]{wang2024rethinking}
Shih-Hsin Wang, Yung-Chang Hsu, Justin Baker, Andrea~L. Bertozzi, Jack Xin, and Bao Wang.
\newblock Rethinking the benefits of steerable features in 3d equivariant graph neural networks.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=mGHJAyR8w0}.
\bibitem[Wang \& Zhang(2022)Wang and Zhang]{new}
Xiyuan Wang and Muhan Zhang.
\newblock Graph neural network with local frame for molecular potential energy surface.
\newblock In \emph{The First Learning on Graphs Conference}, 2022.
\newblock URL \url{https://openreview.net/forum?id=0lSm-R82jBW}.
\bibitem[Widdowson \& Kurlin(2023)Widdowson and Kurlin]{widdowson2023recognizing}
Daniel Widdowson and Vitaliy Kurlin.
\newblock Recognizing rigid patterns of unlabeled point clouds by complete and continuous isometry invariants with no false negatives and no false positives.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 1275--1284, 2023.
\bibitem[Widdowson et~al.(2022)Widdowson, Mosca, Pulido, Kurlin, and Cooper]{widdowson2022generic}
Daniel Widdowson, Marco Mosca, Angeles Pulido, Vitaliy Kurlin, and Andrew~I Cooper.
\newblock Average minimum distances of periodic point sets are fundamental invariants for mapping all periodic crystals.
\newblock \emph{MATCH Communications in Mathematical and in Computer Chemistry}, 87\penalty0 (3):\penalty0 529--559, 2022.
\bibitem[Xu et~al.(2018)Xu, Hu, Leskovec, and Jegelka]{GIN}
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.
\newblock How powerful are graph neural networks?
\newblock \emph{arXiv preprint arXiv:1810.00826}, 2018.
\bibitem[Zhu et~al.(2024)Zhu, Hwang, Adams, Liu, Nan, Stenfors, Du, Chauhan, Wiest, Isayev, Coley, Sun, and Wang]{baselines}
Yanqiao Zhu, Jeehyun Hwang, Keir Adams, Zhen Liu, Bozhao Nan, Brock Stenfors, Yuanqi Du, Jatin Chauhan, Olaf Wiest, Olexandr Isayev, Connor~W. Coley, Yizhou Sun, and Wei Wang.
\newblock Learning over molecular conformer ensembles: Datasets and benchmarks.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=NSDszJ2uIV}.
\end{thebibliography}