ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures
Extracted source text · arXiv 2510.04938
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{cleveref}
\usepackage{caption}
\usepackage[subtle]{savetrees}
\title{ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{%
Shiwen Qin \\ University of Edinburgh
\And Alexander Auras \\ University of Siegen
\And Shay B. Cohen \\ University of Edinburgh
\And Elliot J. Crowley \\ University of Edinburgh
\And Michael Moeller \\ University of Siegen
\And Linus Ericsson \\ University of Glasgow
\And Jovita Lukasik \\ University of Siegen
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
Neural architecture search (NAS) automates the design process of high-performing architectures, but remains bottlenecked by expensive performance evaluation. Most existing studies that achieve faster evaluation are mostly tied to cell-based search spaces and graph encodings tailored to those individual search spaces, limiting their flexibility and scalability when applied to more expressive search spaces.
In this work, we aim to close the gap of individual search space restrictions and search space dependent network representations. We present ONNX-Bench, a benchmark consisting of a collection of neural networks in a unified format based on ONNX files. ONNX-Bench\footnote{\url{https://huggingface.co/datasets/carlosqsw/ONNX-Bench}} includes all open-source NAS-bench-based neural networks, resulting in a total size of more than $600$k $\{$architecture, accuracy$\}$ pairs. This benchmark allows creating a shared neural network representation, ONNX-Net, able to represent any neural architecture using natural language descriptions acting as an input to a performance predictor. This text-based encoding can accommodate arbitrary layer types, operation parameters, and heterogeneous topologies, enabling a single surrogate to generalise across all neural architectures rather than being confined to cell-based search spaces. Experiments show strong zero-shot performance across disparate search spaces using only a small amount of pretraining samples, enabling the unprecedented ability to evaluate any neural network architecture instantly. Our code is available at \url{https://github.com/shiwenqin/ONNX-Net}.
\end{abstract}
\section{Introduction}
Neural Architecture Search (NAS) aims to automate the design of neural networks, with the goal of surpassing manually developed architectures and enabling the discovery of novel network types. However, NAS has largely failed to deliver on its promise of uncovering fundamentally new architectures—such as facilitating the shift from convolutional networks to transformers. One contributing factor is the use of restrictive search spaces, like cell-based search spaces, which limit exploration to a single class of network designs \citep{ying2019bench, ChenPFL21}. Recently, researchers have begun to focus on more expressive search spaces that enable the discovery of more diverse and innovative architectures \citep{schrodi2023construction, ericsson2024einspace}.
However, the design of search spaces is only one aspect of NAS. Based on the design, the actual search strategy within the search space is the biggest computational bottleneck, as the number of possible architectures increases significantly with increasing size and expressiveness of the search space.
To improve on the search cost, surrogate models \citep{dudziak2020brp, lee2021hardware} were introduced. These surrogate models learn a mapping from architecture representations to their evaluation performance, enabling search methods to explore more candidates per unit time. Early surrogate models \citep{tang2020semi, wen2020neural} typically relied on graph-based encodings of architectures using Graph Neural Networks (GNN) \citep{KipfW17}. While effective within their target settings, these designs are based on strong priors about the search space, such as fixed cell topologies and constrained node counts, making them difficult to transfer across tasks and search spaces.
More recent approaches adopt graph-based representations \citep{mills2023gennape, akhauri2024encodings} to improve generalisation across search spaces. However, this paradigm has important limitations. First, adjacency-matrix encodings scale well only in cell-based spaces, where the number of nodes is tightly controlled, and struggle to extend to more flexible search spaces \citep{ericsson2024einspace, schrodi2023construction} with sparse variable sized graphs. Second, adjacency structures mainly capture topology and are largely insensitive to operator parametrisation: two architectures with identical graphs but different hyperparameters (e.g., convolution kernel size, stride, padding or dilation) that share the same adjacency matrix are often indistinguishable. This underlines the strong need for a representation that captures both topology and rich operator-level details in a search space-agnostic manner.
\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{images/overview_big_picture.pdf}
\caption{Overview of our approach: ONNX-Bench contains $\{$architecture, accuracy$\}$ pairs from multiple search spaces in unified ONNX representation; ONNX-Net consists of a robust, universal text encoding and an LLM-based performance predictor.}
\label{fig:contrib_overview}
\end{figure}
To address this, we develop a more general and robust encoding for NAS surrogate models, \textbf{ONNX-Net} (cf. \cref{fig:contrib_overview} (right)) --- one that is agnostic to search spaces, sensitive to operator-level details, and simple to extend.
We propose representing architectures as text generated from Open Neural Network Exchange (ONNX) based computational network representations \citep{onnx} and show that training an end-to-end LLM-based predictor on the text encoding allows for instant performance prediction.
Text encodings are flexible and compositional; they can naturally capture topology, operators, and fine-grained parameters, as well as auxiliary context, without redesigning the encoder for each space. However, learning effective predictors from text also requires a sufficiently diverse collection of architectures spanning multiple search spaces. Existing NAS benchmarks fall short in this regard, as they are typically confined to a single network type and cannot support training predictors that generalise beyond it.
To overcome this limitation, we introduce \textbf{ONNX-Bench} (cf. \cref{fig:contrib_overview} (left)), a benchmark that consolidates architectures from multiple search spaces into a unified ONNX-based representation \citep{onnx} with a consistent evaluation setup. ONNX-Bench provides the diversity needed for training and testing cross-space predictors, and serves as a foundation for studying encodings that capture both structural and operator-level details, such as our proposed ONNX-Net. In our experiments, we demonstrate that a surrogate model using the novel text-based encoding trained on ONNX-Bench achieves competitive performance, especially for zero-shot transferability with minimal pretraining.
Our contributions are as follows:
\begin{itemize}
\item We release an open-source collection of neural networks in a unified ONNX format, evaluated on CIFAR-10 \citep{krizhevsky2009learning}, enabling research that goes beyond the boundaries of individual search spaces
\item We propose a novel ONNX-to-text encoding method that applies to arbitrary architectures and leverages the generalisation ability of large language models
\item We present initial experiments on surrogate modeling with this representation, showing strong performance with few pretraining examples and good zero-shot transfer across search spaces
\end{itemize}
\section{Related Work}
\subsection{Network Search Spaces}
The effectiveness of neural architecture search depends strongly on the design of the underlying search space.
Foundational work explored macro-designs using ResNet-style \citep{HeZRS16} building blocks with skip-connections \citep{ZophL17}. Building on this idea, cell-based designs became widely adopted, where a single cell structure is searched and stacked to form the network \citep{ying2019bench, dong2020bench, dong2021nats, liu2018darts, siems2020bench}. While computationally efficient, cell-based search spaces are much more restrictive, motivating the design of more expressive alternatives, such as hierarchical search spaces \citep{LiuSVFK18, RuEC20, schrodi2023construction}. More recently, grammar-based search spaces were proposed~\citep{schrodi2023construction, ericsson2024einspace} offering a more flexible and principled way of generating diverse architectures. Notably, \texttt{einspace} \citep{ericsson2024einspace} overcomes the bias of hand-designed search spaces, using a probabilistic grammar to encapsulate a wide range of architectural families.
Despite these advances, search spaces have been evaluated in isolation due to different design constraints. This highlights the need to unify the search spaces. Ideally, such a framework would combine all $\{$architecture, accuracy$\}$ pairs within those search spaces, to enable an unbiased, diverse search space for evaluation of performance predictors.
Most previous attempts to create such a search space have used python code as architecture representation \citep{Rahman2025lemonade, Gao2025llm4gnas, Zhou2025lapt, Nasir2024llmatic, Chen23evoprompting}. While very general, this form of representation comes with its own drawbacks, as code generation could produce nonsensical or syntactically incorrect results, and thus requiring either verification of the resulting code, which is non-trivial, or the use of a template, thus restricting the results to a fixed structure again.
In this paper, we take a more holistic view by introducing ONNX-Bench, a dataset that unifies multiple search spaces under a common ONNX-based network format. This enables performance prediction and search methods to operate across diverse search spaces within a consistent representation, facilitating fair comparison and more general NAS approaches. Another central advantage of structured file formats such as ONNX is the high expressivity, similar to python code, while still ensuring easy validatability by performing simple graph operations (e.g. ensuring the existence of only one connected component, checking for cycles, ...).
\subsection{Network Encodings}
Orthogonal to the design of the search space, the network encoding plays a crucial role in the search process, especially in order to facilitate speed up techniques such as performance prediction methods. Due to the popularity of cell-based search spaces, most encoding approaches define architectures as graph data and encode them as adjacency matrices processed by Graph Neural Networks (GNNs) \citep{arch2vec20, Ning_DGF, Velickovic_GAT}
To overcome the pure adjacency based structure, \cite{LukasikMK25} used zero-cost proxies as architecture encodings, and \cite{KadlecovaLPVSNH24} included additional search space specific network topology information as an input to a tabular prediction method. Recently, \cite{mills2023gennape, akhauri2024encodings} learn graph-based encodings with the focus on the ability to transfer between search spaces. While \cite{mills2023gennape} learns a graph encoder using contrastive learning, \cite{akhauri2024encodings} combines different search-space specific learned encodings, such as an unsupervised learned latent space encoding \citep{arch2vec20}, a learned cell encoding using GNNs and zero-cost proxies to learn a network representation. However, these network representations are search space specific, especially with the restriction of only being applicable in cell-based search spaces, eventually limiting the flexibility and scalability of the encoding.
\subsection{LLMs in NAS}
Recently, LLMs for NAS have become quite a common approach, fuelled by the frequent publication of ever more capable language models. Common modes of operation are LLMs as performance predictors \citep{Jawahar2024llmpp} or for generating/mutating network architectures, often in combination with evolutionary algorithms. Most approaches do apply LLMs in settings where topology or operations are confined by some prior structure (e.g. a cell-based search space \citep{Cai2025seki, Zhong2024llmo, Zheng2023genius}, supernets \citep{Ji2025rznas, Jawahar2024llmpp}, choosing parameters only in predefined ranges \citep{Qin2024flnas}, or manipulating sequences, where entries have predefined meanings \citep{Dong2023hgnas, Hu2025lmsearcher}). \cite{qin2025transferrable} proposed to use a string representation of a network, based on the grammar in einspace, with an LLM for performance prediction to overcome the lack of flexibility, showing improvements over the usage of zero-cost proxies and topology features as in \cite{KadlecovaLPVSNH24}.
However, all these methods are dependent on the search space, and cannot be transferred from one search space to another. In this work, we aim to push the boundaries of NAS and present ONNX-Net, a universal network encoding, independent of the search space, with the ability of instant performance prediction. This encoding is independent of the search space design and allows encoding any neural network that can be converted into an ONNX file.
\begin{table}[t]
\centering
\footnotesize
\captionsetup{skip=2pt}
\caption{Composition of the ONNX-Bench dataset.}
\vspace{0.1em}
\label{tab:onnx_bench}
\begin{tabular}{lllr}
% \toprule
Search Space & Type & Evaluation & Num Architectures \\ \cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-3}\cmidrule(lr){4-4} \addlinespace[0.5ex]
NAS-Bench-101 & \multirow{5}{*}{Cell-based} & \multirow{4}{*}{CIFAR-10} & 423624 \\
NAS-Bench-201 & & & 15625 \\
NATS-Bench & & & 32768 \\
NAS-Bench-301 & & & 57189 \\ \cmidrule(lr){3-3}
TransNAS-Bench-101 & & Other & 38895 \\ \cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-3}\cmidrule(lr){4-4} \addlinespace[0.5ex]
hNAS-Bench-201 & \multirow{3}{*}{Hierarchical} & \multirow{2}{*}{CIFAR-10} & 8000 \\ \cmidrule(lr){1-1}
\multirow{2}{*}{einspace} & & & 57495 \\ \cmidrule(lr){3-3}
& & UnseenNAS & 16000 \\ \cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-3}\cmidrule(lr){4-4} \addlinespace[0.5ex]
Total & & & 649596 \\ \bottomrule
\end{tabular}
\end{table}
\section{ONNX-Bench}
We introduce a new benchmark dataset for neural architecture search and performance prediction. It collates networks across several sources in a unified format and evaluation setup. This is needed to take NAS beyond the restriction of individual smaller search spaces, and will enable the training of performance predictors that can transfer across existing and future search spaces.
To achieve high compatibility with different frameworks and network formats, we chose the ONNX file format \citep{onnx} as a basis for our work, as it is a de-facto standard for neural network persistence. This allows our method, cf. \cref{sec:onnx_net}, to be applied to nearly any network found in the wild, as most frameworks and file formats support saving or conversion into ONNX.
ONNX is a binary file format that represents neural network architectures as directed graphs, where nodes are instances of a set of pre-defined operations, while edges represent tensors/arrays passed between these operations. Nodes also contain the hyperparameter values for their operations, e.g. the kernel size of a pooling operation. Additionally, every ONNX file contains a list of input tensors (and their values, in the case of learned parameters), and output tensors.
\begin{figure}[t]
\centering
\includegraphics[width=0.85\linewidth]{images/onnx_diversity_metrics_new.pdf}
\caption{Diversity within and between the search spaces in ONNX-Bench, diversity is measured using Jensen-Shannon divergence (in bits).}
\label{fig:space_diversity}
\end{figure}
The benchmark includes networks from multiple sources, spanning both cell-based and macro-level search spaces such as NAS-Bench-101 \citep{ying2019bench}, NAS-Bench-201 \citep{dong2020bench}, NATS-Bench \citep{dong2021nats}, DARTS-style cells in NAS-Bench-302 \citep{siems2020bench}, and the hierarchical spaces hNAS-Bench-201 \citep{schrodi2023construction}, and einspace \citep{ericsson2024einspace}. Standardising these diverse architectures into ONNX allows them to be easily compared, reused, and extended. Table~\ref{tab:onnx_bench} summarises the distribution of architectures across sources, along with statistics such as the number of nodes and operator types.
All architectures are evaluated on the CIFAR-10 dataset using a consistent training pipeline. CIFAR-10 is widely used in NAS research and provides a balance between computational tractability and benchmark relevance. By fixing the dataset and training settings, ONNX-Bench ensures that observed performance differences reflect the architectures themselves rather than inconsistencies in training protocols.
In total, ONNX-Bench comprises 649596 trained models, with node counts ranging from 1 to 3503 and CIFAR-10 accuracies spanning [0.0, 97.03]. This diversity includes both poorly performing and competitive architectures, making the benchmark suitable for evaluating predictors across the full performance spectrum. We expect ONNX-Bench to support the development of NAS methods that transfer across search spaces and reduce the need for repeated, costly retraining of architectures from scratch.
To further analyse the difference between each search space, we calculate diversity metrics both within and between search spaces in ONNX-Bench. For each model, we count ONNX \texttt{node.op\_type} occurrences (excluding Constant) and normalize to a probability $p$ over the op vocabulary $V$. For a sampled set $S$ of $n$ models from a search space. We compute Jensen–Shannon divergence (JSD, base-2, in bits) between all pairs:
\begin{equation}
\begin{aligned}
JSD(p_i, p_j) = \frac{1}{2} KL(p_i \,\|\, m) + \frac{1}{2} KL(p_j\,\|\,m), \quad \text{where } m = \frac{1}{2}(p_i + p_j)
\end{aligned}
\end{equation}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/onnx_statistics.pdf}
\caption{Distribution of nodes (left) in log scale and CIFAR-10 accuracy (right) of the search spaces contained in ONNX-Bench.}
\label{fig:space_stat}
\end{figure}
In terms of across-space dissimilarity, for two spaces $A$ and $B$, we pool $n$ sampled models in each space to obtain $p_{pool}^A$ and $p_{pool}^B$ over joint vocabulary $V=V_A \cup V_B$, then report $JSD(p_{pool}^A,p_{pool}^B)$ in bits.
The ideal NAS search space encompasses all well-performing neural network architectures possible. While this is infeasible to achieve, we argue that a very diverse search space comes closest to this goal. In \cref{fig:space_diversity}, we report diversity measures over 5k random samples from each search space, and show that hierarchical search spaces, such as einspace and hNAS-Bench-201, differ strongly from other spaces (and, in the case of einspace, also have a much broader variety of architectures). That these additional architectures are not only composed out of low-performing "fail-cases" can be clearly seen in \cref{fig:space_stat}. As ONNX-Bench encompasses all aforementioned search spaces, it fulfils the goal of architecture-diversity to a high degree.
\section{ONNX-Net}\label{sec:onnx_net}
As a baseline for future performance predictors developed on ONNX-Bench, we propose a presentation that allows to describe any neural network architecture in the ONNX format and can be easily coupled with an instant performance prediction on a given dataset. Figure~\ref{fig:onnx-net-overview} shows an overview of our approach.
The central part of our proposed approach is the representation of the neural architecture in the form of natural language. To represent the information contained in the ONNX files as text, we first reconstruct the graph contained in the file in memory, retaining all information.
Since context length is a limiting factor for many LLMs, we try to reduce the size of the graph by performing a variety of optimisations:
\begin{description}
\item[Node removal] We remove nodes we deem to be of low importance, such as identity operations or input nodes for parameters, which can be implicitly inferred by their context.
\item[Subgraph merging] We merge common subgraphs with known interpretation into single nodes, e.g. merging a matrix-vector multiplication with a parameter, followed by an addition with a parameter, into a single node, corresponding to a linear layer.
\end{description}
Some of the (lossless) optimisations were performed by the \textit{ONNX-Simplifier} tool \citep{onnxsim}.
The resulting, shortened graph is again shrunk by merging chains of operations without any branches, to obtain the final, condensed version of the graph.
We then convert this graph structure into text by printing each chain of nodes on one line, in the following format:
\begin{minipage}{\linewidth}
\begin{verbatim}
Operation(Input1, ...)(Parameter1=Value,...) -->
Operation(prev, ...)(Parameter1=Value,...) -->
--> ... --> Output1, ..., OutputN:Shape
\end{verbatim}
\end{minipage}
To show the applicability of ONNX-Net we train an LLM to act as an exemplary surrogate model. NAS is prohibitively expensive, due to the need of training every candidate architecture to evaluate its performance. Surrogate models circumvent this reoccurring cost by acting as a performance predictor, inferring the performance of a candidate architecture of some search space (in this case all architectures representable by ONNX) by information obtained without training the candidate. This accelerates the search, as the inference speed of many neural networks is negligible in comparison to the amount of time a (partial) training my require.
We train our model on the task of ranking input architectures with the $\{$architecture, accuracy$\}$ pairs contained in ONNX-Bench, encoded as natural language. For this purpose we finetune an of-the-shelf large lanugage model with a classic pairwise hinge loss.
\iffalse
The prediction method $f(\cdot)$ uses $\mathcal{A}$, which denotes the space of all possible ONNX-encoded neural network architectures, and $D$ the dataset of interest as input. We further define
% \begin{equation}
$
\phi(\cdot \, ; D) : \mathcal{A} \to \mathbb{R},
\label{eq:phi}
$
% \end{equation}
as the training process for any network $a \in \mathcal{A}$ on dataset $D$ which returns the validation performance.
Executing $\phi$ is typically computationally expensive --- denote this cost as $\mathrm{time}(\phi)$. To avoid repeatedly incurring this cost, we construct a surrogate model
\begin{equation}
f : \mathcal{A} \to \mathbb{R},
\label{eq:f}
\end{equation}
a performance predictor that approximates the outcome of $\phi$, such that
\begin{equation}
f(a) \approx \phi(a; D), \quad \text{with } \mathrm{time}(f) \ll \mathrm{time}(\phi).
\label{eq:approx}
\end{equation}
We train the performance predictor $f$ using a separate dataset of $\{$architecture, accuracy$\}$ pairs $\{(a_i, \phi(a_i; D))\}_{i=1}^N, \quad a_i \in \mathcal{A}$. The parameters of $f$, denoted by $\theta$, are optimised by minimising an empirical risk objective of the form
\begin{equation}
\theta^\ast = \argmin_{\theta} \; \frac{1}{N} \sum_{i=1}^N
\mathcal{L}\big(f(a_i; \theta), \, \phi(a_i; D)\big),
\label{eq:f_optimisation}
\end{equation}
where $\mathcal{L}$ is a loss function that quantifies the discrepancy between the predicted and true performances. This loss can take the form of e.g. a mean-squared error or a ranking loss.
\fi
\begin{figure}[t]
\centering
\includegraphics[width=0.7\linewidth]{images/overview2}
\caption{The ONNX-Net pipeline for performance prediction of any ONNX-encoded neural network architecture.}
\label{fig:onnx-net-overview}
\end{figure}
\section{Experiments}
ONNX-Bench can be a valuable dataset for the NAS community to investigate search space-independent surrogate models that generalise to various types of architectures. As a baseline, we evaluate ONNX-Net with respect to its ability to predict performances on new search spaces (\cref{sec:loo}), with respect to its zero shot performance (\cref{sec:zero-shot}), and its ability to generalise to new tasks (\cref{sec:unseen_task}).
\noindent\textbf{Metrics}
We report the rank correlation values (Kendall’s $\tau$, Spearman's $\rho$) between the ground-truth and predicted performances.
\noindent\textbf{Model}
We fine-tune a \texttt{ModernBERT-large} model as the performance predictor, and we also compare it with other LLMs in \cref{sec:base model}.
\subsection{How well does the surrogate perform on new search spaces?} \label{sec:loo}
We access how well surrogate trained on all but one search space generalise to the excluded search space (\cref{tab:leave_one_out}). For better comparison, we also include surrogate trained on all search space.
Given that NATS-Bench is extension of NAS-Bench-201, they are considered together as NATS-Bench in this experiment. The detailed train/val split regime is described in \cref{sec:split}.
Results in \cref{tab:leave_one_out} shows training on all search spaces yields strong but not uniformly optimal Kendall’s $\tau$ across targets. Holding out a space typically degrades performance on that space, with a notable exception for hNAS-Bench-201, where leaving it out actually improves transfer (0.533 → 0.565), suggesting negative transfer when hNAS is included. The best per-column scores are generally achieved without the full mixture, suggesting future work regarding finding the optimal data mixture for a universal surrogate model.
\begin{table}[t]
\centering
\captionsetup{skip=2pt}
\caption{Kendall's $\tau$ correlations for surrogate models trained on the full set of search spaces as well as all-but-one search space. Rows after the first show the spaces left out of the training set and columns show the evaluation spaces where we compute correlations on held-out data.}
\label{tab:leave_one_out}
\resizebox{\textwidth}{!}{%
\begin{tabular}{clccccc}
% \toprule
{} & & NAS-Bench-101 & NATS-Bench & NAS-Bench-301 & hNAS-Bench-201 & einspace \\ \cmidrule(lr){1-2}\cmidrule(lr){3-7} \addlinespace[0.5ex]
{} & Train on all & 0.772 & 0.788 & 0.691 & \underline{0.533} & 0.477 \\ \cmidrule(lr){1-2}\cmidrule(lr){3-7} \addlinespace[0.5ex]
\multirow{5}{*}{\rotatebox[origin=c]{90}{Leave-out}} & NAS-Bench-101 & 0.529 & 0.815 & 0.687 & 0.386 & \textbf{0.529} \\
{} & NATS-Bench & 0.744 & 0.390 & \textbf{0.704} & 0.382 & 0.478 \\
{} & NAS-Bench-301 & 0.777 & 0.819 & 0.508 & 0.214 & 0.474 \\
{} & hNAS-Bench-201 & \underline{0.787} & \underline{0.825} & 0.693 & \textbf{0.565} & \underline{0.524} \\
{} & einspace & \textbf{0.794} & \textbf{0.843} & \underline{0.694} & 0.456 & 0.301 \\ \bottomrule
\end{tabular}%
}
\end{table}
\subsection{Zero-shot performance across search spaces} \label{sec:zero-shot}
To enable comparison with prior work~\citep{akhauri2024encodings, mills2023gennape}, we evaluate the zero-shot transfer from \texttt{NAS-Bench-101} to \texttt{NAS-Bench-201}. Concretely, we train the surrogate on 50k random $\{$architecture, accuracy$\}$ pairs from \texttt{NAS-Bench-101} and evaluate on the full \texttt{NAS-Bench-201} set without any adaptation.
We also study data scaling by varying the \texttt{NAS-Bench-101} training set size: 200, 1k, and 5k samples. We replicate the FLAN setup using their released code under our protocol; we are unable to replicate GENNAPE due to the lack of reproducible codes and therefore we report only results from their paper.
\begin{table}[t]
\centering
\footnotesize
\captionsetup{skip=2pt}
\caption{Zero-shot predictor trained on 50k samples from NAS-Bench-101 and evaluated on NAS-Bench-201. Avg. Spearman's $\rho$ over 5 random seeds is reported.}
\label{tab:zero-shot}
\begin{tabular}{cccccccc}
% \toprule
\multirow{2}{*}{Transfer} & \multirow{2}{*}{GENNAPE} & \multicolumn{5}{c}{FLAN} & \multirow{2}{*}{ONNX-Net} \\
& & - & CATE & Arch2Vec & ZCP & CAZ & \\ \cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-7}\cmidrule(lr){8-8} \addlinespace[0.5ex]
Zero-Shot & \textbf{0.815} & 0.697 & 0.697 & 0.741 & 0.646 & 0.685 & \underline{0.747} \\ \bottomrule
\end{tabular}
\end{table}
Results in \cref{tab:zero-shot} and \cref{fig:zero-shot} show that GENNAPE achieves the strongest zero-shot transfer overall ($\rho{=}0.815$), noting that it utilises an ensemble combining multiple predictors with two pairwise classifiers. Relative to FLAN, our surrogate consistently achieves higher zero-shot performance across all training-set sizes, including FLAN variants that incorporate additional encodings such as CATE, Arch2Vec, or zero-cost proxies (ZCP); the gains are largest in the low-data regime. Our surrogate reaches its peak zero-shot correlation with 5k training samples and exhibits substantially lower seed-to-seed variance than FLAN.
\begin{figure}[t]
\centering
\includegraphics[width=0.65\linewidth]{images/spr_vs_data_size.pdf}
\caption{Zero-shot predictor trained on different number of \texttt{NAS-Bench-101}, evaluated on NAS-Bench-201. Avg. Spearman's $\rho$ and standard error over 5 random seeds is reported.}
\label{fig:zero-shot}
\end{figure}
\subsection{How well does the surrogate do on new datasets?}\label{sec:unseen_task}
We further assess the ability of the surrogate model to generalise to classification tasks other than CIFAR10.
\paragraph{CIFAR-10 to Unseen NAS on einspace (cross-dataset).}We train a surrogate using CIFAR-10 labels and evaluate zero-shot on eight Unseen NAS \citep{geada2024insights} datasets within the einspace search space. Results in \cref{tab:unseennas} show that including einspace during training (\emph{Full}) markedly improves zero-shot transfer to UnseenNAS. \emph{Full} outperforms \emph{w/o einspace} on 6 of 8 datasets. Overall, including einspace training data is critical for cross-dataset generalisation within the einspace search space, though the optimal training mix may be task-dependent.
\begin{table}[t]
\centering
\captionsetup{skip=2pt}
\caption{Kendall's $\tau$ correlations for zero-shot Unseen NAS tasks}
\label{tab:unseennas}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}cllllllll@{}}
% \toprule
& AddNIST & Language & MultNIST & CIFARTile & Gutenberg & Isabella & GeoClassing & Chesseract \\ \cmidrule(lr){1-1}\cmidrule(lr){2-9} \addlinespace[0.5ex]
Full & 0.364 & 0.338 & 0.449 & 0.351 & 0.582 & 0.248 & 0.095 & 0.294 \\ \cmidrule(lr){1-1}\cmidrule(lr){2-9} \addlinespace[0.5ex]
w/o einspace & 0.156 & 0.131 & 0.245 & 0.058 & 0.317 & 0.135 & 0.249 & 0.302 \\ \bottomrule
\end{tabular}%
}
\end{table}
\section{Ablation Study}
\subsection{Text Encoding} \label{sec:encoding ablation}
\begin{figure}[t]
\centering
\includegraphics[width=0.8\linewidth]{images/ablation}
\caption{Information to include in text encoding}
\label{fig:encoding ablation}
\end{figure}
We decompose the architecture-to-text encoding into four components (\cref{fig:encoding ablation}):
(i) \emph{Base information}: operation name and output index;
(ii) \emph{Input information}: weight/bias shapes and names of the inputs to each operation;
(iii) \emph{Parameter information}: operation-specific parameters (e.g., kernel shape and padding for convolutions);
(iv) \emph{Output shape information}: the tensor shape of operation’s output.
To assess the importance of each component, we compare:
(a) a \emph{base} variant using only the Base information;
(b) the \emph{full} encoding (all four components);
(c) three variants where we add one component to the base variant.
All models are trained and evaluated under the same setting as \cref{sec:zero-shot}, we additionally add zero-shot experiments from NAS-Bench-201 to NAS-Bench-101, excluding the 50k variant due to data limitations for NAS-Bench-201.
\begin{table}[t]
\centering
\footnotesize
\captionsetup{skip=2pt}
\caption{Zero-shot transfer learning between NAS-Bench-101 and NAS-Bench-201 using different text encodings. }
\label{tab:encoding ablation}
\begin{tabular}{lccccccc}
% \toprule
\multirow{2}{*}{Encoding} & \multicolumn{4}{c}{NB101 → NB201} & \multicolumn{3}{c}{NB201 → NB101} \\
& 200 & 1000 & 5000 & 50000 & 200 & 1000 & 5000 \\ \cmidrule(lr){1-1}\cmidrule(lr){2-5}\cmidrule(lr){6-8} \addlinespace[0.5ex]
Base & 0.618 & 0.644 & 0.691 & 0.666 & 0.599 & 0.667 & 0.691 \\ \cmidrule(lr){1-1}\cmidrule(lr){2-5}\cmidrule(lr){6-8} \addlinespace[0.5ex]
+Inputs & \textbf{0.746} & \underline{0.752} & \underline{0.756} & \underline{0.743} & 0.682 & \textbf{0.755} & \underline{0.780} \\
+Parameters & 0.726 & 0.724 & 0.715 & 0.718 & \underline{0.713} & 0.720 & 0.762 \\
+Out Shape & 0.660 & 0.679 & 0.693 & 0.682 & 0.658 & 0.686 & 0.675 \\ \cmidrule(lr){1-1}\cmidrule(lr){2-5}\cmidrule(lr){6-8} \addlinespace[0.5ex]
Full & \underline{0.739} & \textbf{0.754} & \textbf{0.760} & \textbf{0.747} & \textbf{0.715} & \underline{0.740} & \textbf{0.781} \\ \bottomrule
\end{tabular}
\end{table}
Table~\ref{tab:encoding ablation} shows that enriching the encoding with \emph{Input information} yields the largest single-step gain over the \emph{Base} variant across both settings, highlighting the importance of explicit connectivity and weight shape cues at the inputs. Adding only \emph{Parameter information} helps when data size is small but offers diminishing gains as the train size grows. \emph{Output shape} alone provides only marginal improvements over \emph{Base}, which matches the expectation as it adds least amount of information. The \emph{Full} version lags slightly behind +Inputs for some cases, likely due to the longer sequences reducing sample efficiency, but best overall. We also observe a mild dip at 50k relative to 5k for most variants, consistent with the observation in \cref{sec:zero-shot}; this points to potential overfitting to the source domain.
\subsection{Base Model Choice} \label{sec:base model}
We evaluate the effect of the LM backbone by fine-tuning two families:
an encoder-based \texttt{ModernBERT} and a decoder-based \texttt{Qwen3}.
For each family, we use the same fine-tuning recipe, data, and evaluation protocol as in \cref{sec:zero-shot}, results listed in \cref{tab:model ablation}.
\begin{table}[t]
\centering
\footnotesize
\captionsetup{skip=2pt}
\caption{Zero-shot transfer learning from NAS-Bench-101 to NAS-Bench-201 using different LM backbone.}
\label{tab:model ablation}
\begin{tabular}{lccccc}
% \toprule
\multirow{2}{*}{Model} & \multirow{2}{*}{Model Size} & \multicolumn{4}{c}{NB101 → NB201} \\
& & 200 & 1000 & 5000 & 50000 \\ \cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-6} \addlinespace[0.5ex]
ModernBERT-base & 150M & \underline{0.725} & 0.730 & 0.744 & 0.737 \\
ModernBERT-large & 396M & \textbf{0.739} & \textbf{0.754} & \textbf{0.760} & \textbf{0.747} \\ \cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-6} \addlinespace[0.5ex]
Qwen3 & 752M & 0.620 & 0.696 & \underline{0.747} & \underline{0.745} \\
Qwen3 & 2.03B & 0.660 & \underline{0.735} & 0.734 & 0.728 \\ \bottomrule
\end{tabular}
\end{table}
Across all data regimes, the encoder-based LM outperforms the decoder-based ones for zero-shot transfer from NAS-Bench-101 to NAS-Bench-201, with \texttt{ModernBERT-large} being consistently best. Scaling helps within the encoder family: \texttt{ModernBERT-large} surpasses \texttt{ModernBERT-base} at every data size. For \texttt{Qwen3}, the larger variant is better only when data size is small. The superiority of encoder-based LM matches the observation in \cite{qin2025transferrable}.
\section{Conclusion}
We have introduced ONNX-Bench, a unified collection of NAS benchmarks containing architectures in a shared ONNX format and performance scores on a common dataset, CIFAR-10. This benchmark can be used to evaluate performance predictors on a general suite of architectural styles, going beyond the existing narrow cell-based benchmarks. In the future we hope to use it for developing general search methods as well.
Using this novel benchmark, we develop and evaluate a novel surrogate model we call ONNX-Net. It uses our condensed string encoding of the ONNX representation as the input to an LLM, fine-tuned towards the performance prediction task. It shows very strong zero-shot performance using only a small amount of training data. Compared to previous methods it can handle more general and flexible architecture inputs, though it also becomes clear that the problem is more difficult and more restricted graph-based approaches can outperform our more generally applicable method. We hope that the release of this benchmark, encoding and surrogate can spur more research into search space-agnostic NAS.
Future work can build upon the benchmark to create search methods in a general architecture format such as ONNX. Continuing in the direction of LLM and string representations can lead to guided generation of architecture candidates. Furthermore, we acknowledge the limitation of this work to mainly focus on performances on the CIFAR-10 dataset, and hope that future work will expand the capabilities of surrogates to take dataset as context. Finally, due to its general scope, we hope to continue expanding ONNX-Bench as a living benchmark, to make it more diverse e.g.~with attention-based archiectures.
\subsubsection*{Acknowledgments}
This work was supported by the Engineering and Physical Sciences Research
Council [EP/X020703/1], a studentship from the School of Engineering at the University of Edinburgh and SVV project number 260 821.
JL and MM acknowledge support by the German Research Foundation research unit 5336 Learning to Sense and by the Lamarr Institute for Machine Learning and Artificial Intelligence.
%\subsubsection*{Reproducibility Statement}
%To ensure the reproducibility of our work, we submitted anonymous downloadable source code as supplementary materials, along with text encodings used for ONNX-Net experiments and some examples from ONNX-Bench due to the size limit. We also lists hyperparameters used in our experiments in \cref{tab:hyperparam}.
%\subsubsection*{LLM Usage}
%Within the scope of this paper, LLMs are used only to aid and polish writing, as well as auto-complete code fragments.
\bibliography{iclr2026_conference}
\bibliographystyle{iclr2026_conference}
\newpage
\appendix
\section{Unseen NAS Datasets}
In the following we will provide an overview of the 8 Unseen NAS tasks \citep{geada2024insights} used in \cref{tab:unseennas}. All tasks are image classification tasks, comprising different difficulties. AddNIST is based on the MNIST dataset \citep{lecun-mnisthandwrittendigit-2010} using three channels laying in top of each other, with the label being the sum of the label of each channel.The language dataset aims to classify one of 10 languages. The language image is generated by randomly selecting four words from the target language, which are concatenated into a string. This string is then encoded into a $24 \times 24$ grid, where a black pixel indicates the presence of a letter at that grid position.
MultNNIST is similar to AddNIST but uses the multiplication of the three channels as a target. CIFARTile is a combination of four CIFAR-10 images in a $2\times2$ grid. The target here is the number of distinct CIFAR-10 classes shown in the grid. Gutenberg aims at classifying authors based on three words of consecutive sequences encoded similar to the Language dataset. The Isabella dataset classifies four music eras using recordings of these eras which are converted into 64-band spectrograms. GeoClassing uses patches from the BigEarthNet \citep{SumbulCDM19} dataset to identify the corresponding country. Lastly, Chesseract contains images of the chess boards, of the final $15\%$ of the board state, of public games from eight grandmasters. The target here is to depict the classes: white wins, draw, black wins.
\section{Splitting method per space} \label{sec:split}
\begin{itemize}
\item NAS-Bench-201 and NATS-Bench: Randomly sample 20\% of architectures as validation. Due to the high similarity between these two spaces, we merge them together.
\item NAS-Bench-101: We adopt the validation indices provided by the paper \cite{akhauri2024encodings}.
\item HNAS-Bench-201 and einspace: One search seed is reserved as validation split. For einspace, the validation mirrors the setup in \cite{qin2025transferrable}.
\item NAS-Bench-301: Three sources are randomly picked as validation split.
\end{itemize}
Table~\ref{tab:data_stats} summarises the number of training and validation instances used from each search space.
\begin{table}[h]
\centering
\captionsetup{skip=2pt}
\caption{Dataset sizes per search space. For NAS-Bench-201 and NATS-Bench, we merge their respective train/validation partitions due to space similarity. All labels are CIFAR-10 top-1 accuracy.}
\begin{tabular}{lrr}
% \toprule
Search space & Train & Validation \\
\midrule
NAS-Bench-101 & 40{,}000 & 7{,}290 \\
NAS-Bench-201 + NATS-Bench & 38{,}714 & 9{,}679 \\
NAS-Bench-301 & 40{,}000 & 5{,}892 \\
hNAS-Bench-201 & 6{,}403 & 1{,}000 \\
einspace & 37{,}416 & 1{,}582 \\
\bottomrule
\end{tabular}
\label{tab:data_stats}
\end{table}
\section{Implementation Details}
\subsection{Random Seeds}
We use random seed 42 through 46 for all our experiment on multiple seeds. The random seed is used for both training sample selection and training itself.
\subsection{Training Hyperparameters}
The hyperparameters used for training is listed in \cref{tab:hyperparam}.
\section{Graph optimisations}
\begin{figure}[h]
\centering
\begin{tabular}{cc}
\includegraphics[height=8cm]{images/graph_large}
&
\includegraphics[height=8cm]{images/graph_small}
\end{tabular}
\caption{Visualisation of the graph optimisation for a simple neural network. The ONNX graph on the left is simplified in a lossless fashion into the graph on the right.}
\label{fig:graph_opt_viz}
\end{figure}
\begin{figure}[h]
\centering
%\begin{tabular}{cc}
% \includegraphics[height=5cm]{images/iden_pre}
% &
% \includegraphics[height=5cm]{images/iden_post}
% \\
% \includegraphics[height=4cm]{images/linear_pre}
% &
% \includegraphics[height=3cm]{images/linear_post}
%\end{tabular}
\includegraphics[width=0.8\linewidth]{images/optis}
\caption{Visualisation of different graph optimisation steps.}
\label{fig:graph_opt_steps_viz}
\end{figure}
\section{String encoding example}
\begin{figure}[h]
\centering
\begin{minipage}{\linewidth}
\begin{verbatim}
Conv(1x3x32x32, 128x3x3x3, 128)(dilations=1,kernel_shape=3,pads=1,strides=1) --> Value1:1x128x32x32
Relu(Value1) --> Conv(prev, 32x128x1x1, 32)(dilations=1,kernel_shape=1,pads=0,
strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value2:1x32x32x32
Concat(Value2, Value2, Value2, Value2) --> Value3:1x128x32x32
Conv(Value3, 32x128x1x1, 32)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value4:1x32x32x32
Concat(Value4, Value4, Value4, Value4) --> Value5:1x128x32x32
Conv(Value5, 32x128x1x1, 32)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value6:1x32x32x32
Concat(Value6, Value6, Value6, Value6) --> Value7:1x128x32x32
MaxPool(Value7)(kernel_shape=2,pads=0,strides=2) --> Conv(prev, 64x128x1x1, 64)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value8:1x64x16x16
Concat(Value8, Value8, Value8, Value8) --> Value9:1x256x16x16
Conv(Value9, 64x256x1x1, 64)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value10:1x64x16x16
Concat(Value10, Value10, Value10, Value10) --> Value11:1x256x16x16
Conv(Value11, 64x256x1x1, 64)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value12:1x64x16x16
Concat(Value12, Value12, Value12, Value12) --> Value13:1x256x16x16
MaxPool(Value13)(kernel_shape=2,pads=0,strides=2) --> Conv(prev, 128x256x1x1, 128)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value14:1x128x8x8
Concat(Value14, Value14, Value14, Value14) --> Value15:1x512x8x8
Conv(Value15, 128x512x1x1, 128)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value16:1x128x8x8
Concat(Value16, Value16, Value16, Value16) --> Value17:1x512x8x8
Conv(Value17, 128x512x1x1, 128)(dilations=1,kernel_shape=1,pads=0,strides=1) --> Relu(prev) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> MaxPool(prev)(kernel_shape=3,pads=1,strides=1) --> Value18:1x128x8x8
Concat(Value18, Value18, Value18, Value18) --> Value19:1x512x8x8
ReduceMean(Value19)(axes=[2,3]) --> Gemm(prev, 10x512, 10) --> Out
\end{verbatim}
\end{minipage}
\caption{A simple neural network in the text representation of ONNX-Net.}
\label{fig:onnx-net-repr}
\end{figure}
\begin{table}[h]
\centering
\captionsetup{skip=8pt}
\caption{Hyperparameters used in main table experiments.}
\begin{tabular}{llll}
% \toprule
Hyperparameter & Value & Hyperparameter & Value \\ \midrule
Learning Rate & 5e-5 & Weight Decay & 0.1 \\
Number of Epochs & 5 & Batch Size & 16 \\
Learning Rate Scheduling & Polynomial & End Learning Rate & 5e-6 \\
Gradient Accumulation & 1 & Warm-up Ratio & 0.06 \\
Loss Type & Pairwise Hinge Loss & BF16 & True \\ \bottomrule
\end{tabular}
\label{tab:hyperparam}
\end{table}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{44}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Akhauri \& Abdelfattah(2024)Akhauri and Abdelfattah]{akhauri2024encodings}
Yash Akhauri and Mohamed~S Abdelfattah.
\newblock Encodings for prediction-based neural architecture search.
\newblock \emph{arXiv preprint arXiv:2403.02484}, 2024.
\bibitem[Bai et~al.(2019)Bai, Lu, Zhang, et~al.]{onnx}
Junjie Bai, Fang Lu, Ke~Zhang, et~al.
\newblock Onnx: Open neural network exchange.
\newblock \url{https://github.com/onnx/onnx}, 2019.
\newblock v1.19.0.
\bibitem[Cai et~al.(2025)Cai, Tang, Lai, Wang, Chen, and Chen]{Cai2025seki}
Zicheng Cai, Yaohua Tang, Yutao Lai, Hua Wang, Zhi Chen, and Hao Chen.
\newblock Seki: Self-evolution and knowledge inspiration based neural architecture search via large language models.
\newblock \emph{arXiv preprint arXiv:2502.20422}, 2025.
\bibitem[Chen et~al.(2023)Chen, Dohan, and So]{Chen23evoprompting}
Angelica Chen, David Dohan, and David So.
\newblock Evoprompting: Language models for code-level neural architecture search.
\newblock \emph{Advances in neural information processing systems}, 36:\penalty0 7787--7817, 2023.
\bibitem[Chen et~al.(2021)Chen, Peng, Fu, and Ling]{ChenPFL21}
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling.
\newblock Autoformer: Searching transformers for visual recognition.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on computer vision}, pp.\ 12270--12280, 2021.
\bibitem[{@daquexian} et~al.(2019)]{onnxsim}
{@daquexian} et~al.
\newblock Onnx simplifier.
\newblock \url{https://github.com/daquexian/onnx-simplifier}, 2019.
\newblock v0.4.39.
\bibitem[Dong et~al.(2023)Dong, Gao, Wang, Yang, and Zhang]{Dong2023hgnas}
Haoyuan Dong, Yang Gao, Haishuai Wang, Hong Yang, and Peng Zhang.
\newblock Heterogeneous graph neural architecture search with gpt-4.
\newblock \emph{arXiv preprint arXiv:2312.08680}, 2023.
\bibitem[Dong \& Yang(2020)Dong and Yang]{dong2020bench}
Xuanyi Dong and Yi~Yang.
\newblock Nas-bench-201: Extending the scope of reproducible neural architecture search.
\newblock \emph{arXiv preprint arXiv:2001.00326}, 2020.
\bibitem[Dong et~al.(2021)Dong, Liu, Musial, and Gabrys]{dong2021nats}
Xuanyi Dong, Lu~Liu, Katarzyna Musial, and Bogdan Gabrys.
\newblock Nats-bench: Benchmarking nas algorithms for architecture topology and size.
\newblock \emph{IEEE transactions on pattern analysis and machine intelligence}, 44\penalty0 (7):\penalty0 3634--3646, 2021.
\bibitem[Dudziak et~al.(2020)Dudziak, Chau, Abdelfattah, Lee, Kim, and Lane]{dudziak2020brp}
Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane.
\newblock Brp-nas: Prediction-based nas using gcns.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 10480--10490, 2020.
\bibitem[Ericsson et~al.(2024)Ericsson, Espinosa~Minano, Yang, Antoniou, Storkey, Cohen, McDonagh, and Crowley]{ericsson2024einspace}
Linus Ericsson, Miguel Espinosa~Minano, Chenhongyi Yang, Antreas Antoniou, Amos~J Storkey, Shay Cohen, Steven McDonagh, and Elliot~J Crowley.
\newblock einspace: Searching for neural architectures from fundamental operations.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 1919--1953, 2024.
\bibitem[Gao et~al.(2025)Gao, Yang, Chen, Wu, Zhang, and Wang]{Gao2025llm4gnas}
Yang Gao, Hong Yang, Yizhi Chen, Junxian Wu, Peng Zhang, and Haishuai Wang.
\newblock Llm4gnas: A large language model based toolkit for graph neural architecture search.
\newblock \emph{arXiv preprint arXiv:2502.10459}, 2025.
\bibitem[Geada et~al.(2024)Geada, Towers, Forshaw, Atapour-Abarghouei, and McGough]{geada2024insights}
Rob Geada, David Towers, Matthew Forshaw, Amir Atapour-Abarghouei, and A~Stephen McGough.
\newblock Insights from the use of previously unseen neural architecture search datasets.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 22541--22550, 2024.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{HeZRS16}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pp.\ 770--778, 2016.
\bibitem[Hu et~al.(2025)Hu, Liu, Wang, Zhen, Shi, Zhang, Dou, Liu, Zhou, and Li]{Hu2025lmsearcher}
Yuxuan Hu, Jihao Liu, Ke~Wang, Jinliang Zhen, Weikang Shi, Manyuan Zhang, Qi~Dou, Rui Liu, Aojun Zhou, and Hongsheng Li.
\newblock Lm-searcher: Cross-domain neural architecture search with llms via unified numerical encoding.
\newblock \emph{arXiv preprint arXiv:2509.05657}, 2025.
\bibitem[Jawahar et~al.(2023)Jawahar, Abdul-Mageed, Lakshmanan, and Ding]{Jawahar2024llmpp}
Ganesh Jawahar, Muhammad Abdul-Mageed, Laks~VS Lakshmanan, and Dujian Ding.
\newblock Llm performance predictors are good initializers for architecture search.
\newblock \emph{arXiv preprint arXiv:2310.16712}, 2023.
\bibitem[Ji et~al.(2025)Ji, Zhu, Yuan, and Huang]{Ji2025rznas}
Zipeng Ji, Guanghui Zhu, Chunfeng Yuan, and Yihua Huang.
\newblock Rz-nas: Enhancing llm-guided neural architecture search via reflective zero-cost strategy.
\newblock In \emph{Forty-second International Conference on Machine Learning}, 2025.
\bibitem[Kadlecov{\'a} et~al.(2024)Kadlecov{\'a}, Lukasik, Pil{\'a}t, Vidnerov{\'a}, Safari, Neruda, and Hutter]{KadlecovaLPVSNH24}
Gabriela Kadlecov{\'a}, Jovita Lukasik, Martin Pil{\'a}t, Petra Vidnerov{\'a}, Mahmoud Safari, Roman Neruda, and Frank Hutter.
\newblock Surprisingly strong performance prediction with neural graph features.
\newblock \emph{arXiv preprint arXiv:2404.16551}, 2024.
\bibitem[Kipf \& Welling(2017)Kipf and Welling]{KipfW17}
Thomas~N. Kipf and Max Welling.
\newblock Semi-supervised classification with graph convolutional networks.
\newblock In \emph{5th International Conference on Learning Representations}. OpenReview.net, 2017.
\bibitem[Krizhevsky(2009)]{krizhevsky2009learning}
Alex Krizhevsky.
\newblock Learning multiple layers of features from tiny images.
\newblock Technical report, University of Toronto, 2009.
\newblock URL \url{https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf}.
\bibitem[LeCun \& Cortes(2010)LeCun and Cortes]{lecun-mnisthandwrittendigit-2010}
Yann LeCun and Corinna Cortes.
\newblock {MNIST} handwritten digit database.
\newblock http://yann.lecun.com/exdb/mnist/, 2010.
\bibitem[Lee et~al.(2021)Lee, Lee, Chong, and Hwang]{lee2021hardware}
Hayeon Lee, Sewoong Lee, Song Chong, and Sung~Ju Hwang.
\newblock Hardware-adaptive efficient latency prediction for nas via meta-learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 27016--27028, 2021.
\bibitem[Liu et~al.(2017)Liu, Simonyan, Vinyals, Fernando, and Kavukcuoglu]{LiuSVFK18}
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu.
\newblock Hierarchical representations for efficient architecture search.
\newblock \emph{arXiv preprint arXiv:1711.00436}, 2017.
\bibitem[Liu et~al.(2018)Liu, Simonyan, and Yang]{liu2018darts}
Hanxiao Liu, Karen Simonyan, and Yiming Yang.
\newblock Darts: Differentiable architecture search.
\newblock \emph{arXiv preprint arXiv:1806.09055}, 2018.
\bibitem[Lukasik et~al.(2025)Lukasik, Moeller, and Keuper]{LukasikMK25}
Jovita Lukasik, Michael Moeller, and Margret Keuper.
\newblock An evaluation of zero-cost proxies-from neural architecture performance prediction to model robustness.
\newblock \emph{International Journal of Computer Vision}, 133\penalty0 (5):\penalty0 2635--2652, 2025.
\bibitem[Mills et~al.(2023)Mills, Han, Zhang, Chudak, Mamaghani, Salameh, Lu, Jui, and Niu]{mills2023gennape}
Keith~G Mills, Fred~X Han, Jialin Zhang, Fabian Chudak, Ali~Safari Mamaghani, Mohammad Salameh, Wei Lu, Shangling Jui, and Di~Niu.
\newblock Gennape: Towards generalized neural architecture performance estimators.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, pp.\ 9190--9199, 2023.
\bibitem[Nasir et~al.(2024)Nasir, Earle, Togelius, James, and Cleghorn]{Nasir2024llmatic}
Muhammad~Umair Nasir, Sam Earle, Julian Togelius, Steven James, and Christopher Cleghorn.
\newblock Llmatic: neural architecture search via large language models and quality diversity optimization.
\newblock In \emph{proceedings of the Genetic and Evolutionary Computation Conference}, pp.\ 1110--1118, 2024.
\bibitem[Ning et~al.(2022)Ning, Zheng, Zhou, Zhao, Yang, and Wang]{Ning_DGF}
Xuefei Ning, Yin Zheng, Zixuan Zhou, Tianchen Zhao, Huazhong Yang, and Yu~Wang.
\newblock A generic graph-based neural architecture encoding scheme with multifaceted information.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine Intelligence}, 45\penalty0 (7):\penalty0 7955--7969, 2022.
\bibitem[Qin et~al.(2024)Qin, Hu, Yan, Xiong, Abbasi, and Shi]{Qin2024flnas}
Ruiyang Qin, Yuting Hu, Zheyu Yan, Jinjun Xiong, Ahmed Abbasi, and Yiyu Shi.
\newblock Fl-nas: Towards fairness of nas for resource constrained devices via large language models.
\newblock In \emph{2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC)}, pp.\ 429--434. IEEE, 2024.
\bibitem[Qin et~al.(2025)Qin, Kadlecov{\'a}, Pil{\'a}t, Cohen, Neruda, Crowley, Lukasik, and Ericsson]{qin2025transferrable}
Shiwen Qin, Gabriela Kadlecov{\'a}, Martin Pil{\'a}t, Shay~B Cohen, Roman Neruda, Elliot~J Crowley, Jovita Lukasik, and Linus Ericsson.
\newblock Transferrable surrogates in expressive neural architecture search spaces.
\newblock \emph{arXiv preprint arXiv:2504.12971}, 2025.
\bibitem[Rahman et~al.(2025)Rahman, Haider, and Chakraborty]{Rahman2025lemonade}
Md~Hafizur Rahman, Zafaryab Haider, and Prabuddha Chakraborty.
\newblock An automated multi parameter neural architecture discovery framework using chatgpt in the backend.
\newblock \emph{Scientific Reports}, 15\penalty0 (1):\penalty0 16871, 2025.
\bibitem[Ru et~al.(2020)Ru, Esperanca, and Carlucci]{RuEC20}
Robin Ru, Pedro Esperanca, and Fabio~Maria Carlucci.
\newblock Neural architecture generator optimization.
\newblock \emph{Advances in Neural Information Processing Systems}, 33:\penalty0 12057--12069, 2020.
\bibitem[Schrodi et~al.(2023)Schrodi, Stoll, Ru, Sukthanker, Brox, and Hutter]{schrodi2023construction}
Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sukthanker, Thomas Brox, and Frank Hutter.
\newblock Construction of hierarchical neural architecture search spaces based on context-free grammars.
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 23172--23223, 2023.
\bibitem[Sumbul et~al.(2019)Sumbul, Charfuelan, Demir, and Markl]{SumbulCDM19}
Gencer Sumbul, Marcela Charfuelan, Beg{\"u}m Demir, and Volker Markl.
\newblock Bigearthnet: A large-scale benchmark archive for remote sensing image understanding.
\newblock In \emph{IGARSS 2019-2019 IEEE international geoscience and remote sensing symposium}, pp.\ 5901--5904. IEEE, 2019.
\bibitem[Tang et~al.(2020)Tang, Wang, Xu, Chen, Shi, Xu, Xu, Tian, and Xu]{tang2020semi}
Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi~Tian, and Chang Xu.
\newblock A semi-supervised assessor of neural architectures.
\newblock In \emph{proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pp.\ 1810--1819, 2020.
\bibitem[Velickovic et~al.(2017)Velickovic, Cucurull, Casanova, Romero, Lio, Bengio, et~al.]{Velickovic_GAT}
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et~al.
\newblock Graph attention networks.
\newblock \emph{stat}, 1050\penalty0 (20):\penalty0 10--48550, 2017.
\bibitem[Wen et~al.(2020)Wen, Liu, Chen, Li, Bender, and Kindermans]{wen2020neural}
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans.
\newblock Neural predictor for neural architecture search.
\newblock In \emph{European conference on computer vision}, pp.\ 660--676. Springer, 2020.
\bibitem[Yan et~al.(2020)Yan, Zheng, Ao, Zeng, and Zhang]{arch2vec20}
Shen Yan, Yu~Zheng, Wei Ao, Xiao Zeng, and Mi~Zhang.
\newblock Does unsupervised architecture representation learning help neural architecture search?
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 12486--12498, 2020.
\bibitem[Ying et~al.(2019)Ying, Klein, Christiansen, Real, Murphy, and Hutter]{ying2019bench}
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter.
\newblock Nas-bench-101: Towards reproducible neural architecture search.
\newblock In \emph{International conference on machine learning}, pp.\ 7105--7114. PMLR, 2019.
\bibitem[Zela et~al.(2020)Zela, Siems, Zimmer, Lukasik, Keuper, and Hutter]{siems2020bench}
Arber Zela, Julien Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter.
\newblock Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks.
\newblock \emph{arXiv preprint arXiv:2008.09777}, 2020.
\bibitem[Zheng et~al.(2023)Zheng, Su, You, Wang, Qian, Xu, and Albanie]{Zheng2023genius}
Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie.
\newblock Can gpt-4 perform neural architecture search?
\newblock \emph{arXiv preprint arXiv:2304.10970}, 2023.
\bibitem[Zhong et~al.(2024)Zhong, Cao, Yu, and Munetomo]{Zhong2024llmo}
Rui Zhong, Yang Cao, Jun Yu, and Masaharu Munetomo.
\newblock Large language model assisted adversarial robustness neural architecture search.
\newblock In \emph{2024 6th International Conference on Data-driven Optimization of Complex Systems (DOCS)}, pp.\ 433--437. IEEE, 2024.
\bibitem[Zhou et~al.(2025)Zhou, Wu, Feng, Lu, and Tan]{Zhou2025lapt}
Xun Zhou, Xingyu Wu, Liang Feng, Zhichao Lu, and Kay~Chen Tan.
\newblock Design principle transfer in neural architecture search via large language models.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~39, pp.\ 23000--23008, 2025.
\bibitem[Zoph \& Le(2016)Zoph and Le]{ZophL17}
Barret Zoph and Quoc~V Le.
\newblock Neural architecture search with reinforcement learning.
\newblock \emph{arXiv preprint arXiv:1611.01578}, 2016.
\end{thebibliography}