Robust and Generalizable GNN Fine-Tuning via Uncertainty-aware Adapter Learning
Extracted source text · arXiv 2511.18859
\documentclass[lettersize,journal]{IEEEtran}
\usepackage{amsmath,amsfonts}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{array}
\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage{textcomp}
\usepackage{stfloats}
\usepackage{url}
\usepackage{verbatim}
\usepackage{graphicx}
\usepackage{cite}
\hyphenation{op-tical net-works semi-conduc-tor IEEE-Xplore}
% updated with editorial comments 8/9/2021
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{xcolor}
\begin{document}
\title{Robust and Generalizable GNN Fine-Tuning via Uncertainty-aware Adapter Learning}
\author{Bo Jiang, Weijun Zhao, Beibei Wang, Xiao Wang, Jin Tang %Technology,~\IEEEmembership{Staff,~IEEE,}
% % <-this % stops a space
% \thanks{This paper was produced by the IEEE Publication Technology Group. They are in Piscataway, NJ.}% <-this % stops a space
\thanks{
The authors are with School of Computer Science and Technology, Anhui University.
Manuscript received April 19, 2021; revised August 16, 2021.}
}
% The paper headers
\markboth{Journal of \LaTeX\ Class Files,~Vol.~14, No.~8, August~2021}%
{Shell \MakeLowercase{\textit{et al.}}: A Sample Article Using IEEEtran.cls for IEEE Journals}
% \IEEEpubid{0000--0000/00\$00.00~\copyright~2021 IEEE}
% Remember, if you use this you must call \IEEEpubidadjcol in the second
% column for its text to clear the IEEEpubid mark.
\maketitle
\begin{abstract}
Recently, fine-tuning large-scale pre-trained GNNs has yielded remarkable attention in adapting pre-trained GNN models for downstream graph learning tasks.
One representative fine-tuning method is to exploit adapter (termed AdapterGNN) which aims to `augment' the pre-trained model by inserting
a lightweight module to make the `augmented' model better adapt to the downstream tasks.
However, graph data may contain various types of noise in downstream tasks, such as noisy edges and ambiguous node attributes.
Existing AdapterGNNs are often prone to graph noise and exhibit limited generalizability.
%This clearlydemonstrates the effectiveness of our proposed GDCNet in enhancing the generalization ability.
\emph{How to enhance the robustness and generalization ability of GNNs' fine-tuning remains an open problem. }
In this paper, we show that the above problem can be well addressed by integrating uncertainty learning into the GNN adapter.
We propose the Uncertainty-aware Adapter (UAdapterGNN) that fortifies pre-trained GNN models against noisy graph data in the fine-tuning process.
Specifically, in contrast to regular AdapterGNN, our UAdapterGNN
%instead of designing deterministic adapter, we
exploits Gaussian probabilistic adapter to augment the pre-trained GNN model. In this way, when the graph contains various noises, our method can automatically absorb the effects of changes in the variances of the Gaussian distribution, thereby significantly enhancing the model's robustness. Also, UAdapterGNN can further improve the generalization ability of the model on the downstream tasks. Extensive experiments on several benchmarks demonstrate the effectiveness, robustness and high generalization ability of the proposed UAdapterGNN method.
\end{abstract}
\begin{IEEEkeywords}
Graph neural networks, Pre-training, Adapter-tuning, Uncertainty-aware adapter.
\end{IEEEkeywords}
\section{Introduction}
Recently, fine-tuning paradigms have emerged as pivotal
techniques to achieve knowledge transfer from pre-trained GNN models to the downstream graph learning tasks~\cite{wu2020comprehensive}.
Current research in this field can be broadly categorized into three types, i.e., full fine-tuning, prompt-based fine-tuning, and adapter-based fine-tuning.
% Below, we briefly review representative works in each category and highlight their respective strengths and limitations.
% pre-train, fine-tuning
%- Motivation
Full fine-tuning~\cite{xie2022self, hu2019strategies} aims to first leverage transferable representations learned from unlabeled graphs
and
%. Here, GNNs are first pre-trained on abundant unlabeled data and
subsequently adapt them to the downstream tasks by updating (often all) model parameters.
%- Example Works
% For instance, GROVER~\cite{rong2020grover} leverages self-supervised pre-training followed by fine-tuning, achieving enhanced sample efficiency and performance gains.
%G-tuning~\cite{sun2024fine} proposes a technique that preserves downstream graph generative patterns during fine-tuning, demonstrating significant performance improvements across multiple in-domain and cross-domain transfer experiments.
%- Challenges
However, this paradigm
is usually computationally expensive in terms of memory and time.
%faces critical challenges: (1) structural and semantic misalignment between pre-training and downstream graphs can lead to suboptimal adaptation or negative transfer, degrading generalization~\cite{huang2024measuring}; and (2) full-parameter fine-tuning of large models is computationally expensive in terms of memory and time.
% pre-train, prompt-tuning
%- Motivation
% To address these issues,
%For prompt based learning methods
%Inspired by prompt tuning in natural language processing (NLP)~\cite{liu2023pre}, researchers have started exploring the prompt based fine-tuning in the GNN field.
The prompt-based fine-tuning approach freezes the pre-trained GNN model and designs some learnable graph prompts on each downstream task, which are directly appended to the input to enable adaptation. %~\cite{gpf, sun2022gppt, liu2023graphprompt, yu2024multigprompt, yu2024hgprompt}.
%- Example Works
For example, GPF~\cite{gpf} adds a global learnable prompt to downstream graph data, achieving performance comparable to full fine-tuning. % by using less than 0.1\% of tunable parameters. while keeping all pretrained model weights frozen.
Similarly, EdgePrompt~\cite{Fu2025EdgePT} introduces learnable prompt vectors on edges to better aggregate with node representations during message passing.
GPPT~\cite{sun2022gppt} incorporates a `label-node' pair prompt into the input, unifying downstream node classification with the pre-trained link prediction task. %This enables the pre-trained GNN to be applied directly to downstream tasks without fine-tuning.
GraphPrompt~\cite{liu2023graphprompt} further unifies the pre-training and downstream stages by using a single task template.
% Furthermore,~\cite{YuLFLCZ24} enhances GraphPrompt by modifying hidden representations at each layer of pre-trained graph learning models.
%- Challenges
% However, designing more effective prompts for graph data and maintaining strong performance with limited labeled samples remain open challenges for this paradigm.
% pre-train, adapter-tuning
%- Motivation
In addition to prompt learning, adapter-based fine-tuning offers an alternative PEFT strategy, which aims to insert lightweight trainable modules (\textit{adapters}) into the frozen backbone of the pre-trained model, enabling adaptation with minimal parameter updates.
%- Example Works
Recent advancements in graph adapters have introduced diverse strategies to align pre-trained models with downstream tasks.
For example, G-Adapter~\cite{g-adapter} encodes topology via Bregman divergence to mitigate feature distribution shifts caused by domain discrepancies.
%For heterogeneous graphs, HG-Adapter~\cite{mo2024hg} employs dual structure-aware adapters to model homogeneous and heterogeneous interactions.
%In knowledge graph applications, KG-Adapter~\cite{tian2024kg} addresses semantic conflicts by injecting relation-specific projection layers, dynamically aligning external knowledge with pre-trained model parameters.
GConv-Adapter~\cite{papageorgiougraph} combines dual-normalized convolution and low-rank matrices to capture higher-order information efficiently.
GraphLoRA~\cite{yang2024graphlora} injects low-rank adaptation modules, enabling efficient cross-graph domain transfer.
AdapterGNN~\cite{adaptergnn} proposes to adapt to downstream tasks effectively by leveraging expressive adapter modules with minimal trainable parameter, simultaneously significantly enhancing the model's generalization capabilities.
%- Challenges
However, in many downstream learning tasks, graphs may contain various kinds of noises, such as noisy edges and ambiguous node attributes.
After reviewing the above adapters, we observe that existing works mainly focus on efficient knowledge transfer from pre-trained GNN models to the downstream tasks.
However, \textbf{little attention has been devoted to robust learning on downstream tasks with respect to graph noises}.
In this paper, we show that existing adapter approaches generally adopt deterministic adapter modules, which are prone to being sensitive to graph noise and also have limited generalization ability.
\textbf{Therefore, how to enhance the robustness and generalization ability of GNNs' fine-tuning remains an open problem in recent years.}
%This clearlydemonstrates the effectiveness of our proposed GDCNet in enhancing the generalization ability.
In this paper, we show that the above problem can be well addressed by integrating uncertainty learning into GNN adapter.
Specifically, we propose the Uncertainty-aware Adapter (UAdapterGNN), a novel framework that enhances large pre-trained GNNs against noisy graph data in the downstream tasks.
%
%To our best knowledge,
%this work is the first attempt to uncertainty learning model into graph fine-tuning problem.
Instead of designing a deterministic adapter, our UAdapterGNN method exploits Gaussian random adapter to augment the pre-trained GNN model.
%It can adaptively balance reliable patterns with ambiguous signals during message passing.
%
In this way, when the graphs contain noises, the proposed UAdapterGNN can automatically absorb the effects of changes in the variances of the Gaussian distributions, thereby significantly enhancing the pre-trained model’s robustness on the downstream tasks.
Moreover, it can improve the model’s generalization ability on downstream tasks when compared to traditional deterministic adapter models.
%Also, it can improve the model’s generalization ability on downstream tasks when compared with traditional determinic adapters.
Uncertainty learning methods have been usually studied in machine learning field~\cite{gal2016dropout,kendall2017uncertainties}. We note that it has also been exploited in fine-tuning pre-trained LLM/VLM tasks in recent years~\cite{WangZ0ZLT24,JiWGZZWZSY23}.
% Differently,
However, to our best knowledge, this work is the first attempt to exploit uncertainty learning model into GNN's fine-tuning problem.
More importantly, we show that the uncertainty adapter learning can well improve the robustness and generalization of pre-trained GNN model w.r.t noisy data on downstream tasks.
Note that, the proposed UAdapterGNN method offers a general \emph{plug-and-play adapter} architecture
which can be integrated with any pre-trained GNN models to enable \textbf{robust learning} on downstream tasks.
Overall, the main contributions of this paper are summarized as follows,
\begin{itemize}
\item We propose to leverage
uncertainty learning model into
GNN fine-tuning problem to enhance the robustness and generalization ability of pre-trained GNN models w.r.t noisy data in the downstream tasks.
\item We design a lightweight
architecture to implement our
UAdapterGNN by extending
traditional AdapterGNN model into uncertainty learning manner, as demonstrated in Figure~\ref{fig:framework}.
\item
% To validate the effectiveness and advantage of the proposed UAdapterGNN method,
We evaluate UAdapterGNN on several benchmark datasets. Experimental results show the effectiveness and advantages (robustness, generalization ability) of our method under different GNN pre-training strategies.
\end{itemize}
%Experimental results on many benchmark datasets demonstrate the effectiveness and advantages of the proposed UAdapterGNN method under different pre-training models.
%Despite the progress in parameter-efficient adapter designs, existing methods predominantly rely on deterministic feature propagation, generating fixed node embeddings that conflate intrinsic semantic patterns with graph noise. Deterministic adapters treat such noisy patterns as equally reliable as clean data, resulting in overconfident predictions that degrade robustness in low-data or noisy scenarios~\cite{wang2024uncertainty}.
%To bridge this gap, we propose uncertainty-aware adapter-tuning, which introduces input-dependent probabilistic modeling to dynamically adjust both node representations and their confidence levels.
%By characterizing each node as a Gaussian distribution in the latent space whose mean captures semantic identity and variance reflects predictive uncertainty~\cite{song2021uncertain}, the model adaptively balances reliable patterns with ambiguous signals during message passing. Unlike deterministic approaches, this framework allows the mean embeddings to shift based on task-specific inputs while simultaneously quantifying uncertainty through variance estimates derived from the data itself.
%Overall, the main contributions of this paper are summarized as follows,
%\begin{itemize}
%\item We propose an uncertainty-aware adapter-tuning framework for GNNs that leverages input-dependent probabilistic modeling to adjust node representations and their confidence levels dynamically.
%\item We design a dual parallel adapter architecture enhanced by learnable scaling factors that ensure a balanced integration of pre-trained knowledge and task-specific adaptation.
%\item
%\end{itemize}
% This work bridges the critical gap between uncertainty-aware learning and efficient adaptation, offering a practical solution for deploying GNNs in scenarios with noisy graph structures.
% The remainder of this paper is structured as follows: In Section II, we introduce some related works. Section III presents the detailed methodology of our uncertainty-aware adapter-tuning framework. Section IV describes the experimental setup and discusses the empirical results. Finally, Section V concludes the paper. % and outlines potential directions for future research.
\section{Related Work}
% \subsection{Graph Neural Networks}
% Graph Neural Networks (GNNs) have emerged as a dominant paradigm for modeling non-Euclidean graph data, leveraging message-passing mechanisms to iteratively aggregate neighborhood information and generate low-dimensional node representations. These embeddings inherently fuse semantic attributes and structural dependencies, serving as critical inputs for downstream tasks such as node classification, link prediction, and graph-level regression. The core architectures reflect an evolutionary trajectory: GCN~\cite{GCN} implements spectral-based feature propagation through eigendecomposition of the graph Laplacian matrix, subsequently enhancing computational efficiency via spatially localized operations with symmetrically normalized adjacency matrices; GraphSAGE~\cite{GraphSAGE} overcomes GCN's reliance on full-graph structures by adopting stochastic neighbor sampling and aggregation functions (e.g., LSTM, max-pooling) to enable inductive learning on dynamic graphs; GAT~\cite{GAT} further incorporates multi-head attention mechanisms to dynamically assign weights to neighboring nodes, thereby strengthening discriminative aggregation; GIN (Graph Isomorphism Network)~\cite{xu2018powerful} establishes theoretical foundations by employing multi-layer perceptrons to construct injective aggregation functions, achieving maximally discriminative power aligned with the Weisfeiler-Lehman graph isomorphism test~\cite{shervashidze2011weisfeiler}. This progression underscores a shift from spectral theory to spatial practicality, from static global operations to dynamic local sampling, and from homogeneous aggregation to attention-driven refinement. As an effective technology of graph structured data mining, GNNs are an important foundation of our framework.
\textbf{Pre-training on Graphs.}
To mitigate reliance on labeled data in supervised learning, pre-training GNNs has become increasingly popular, inspired by pre-training techniques in language and vision applications~\cite{HuLZHNL24,cv}.
These methods train graph encoders via self-supervised tasks~\cite{LiuJPZZXY23} to learn transferable representations, thereby improving cross-task generalization capabilities.
For example, in work~\cite{hu2019strategies}, it proposes a pre-training strategy for Graph Neural Networks (GNNs), which effectively enhances the model's generalization ability and accuracy across various downstream tasks by combining node-level and graph-level pre-training approaches. Graph Contrastive Learning~\cite{you2020graph} employs graph augmentation-based contrastive learning to pre-train GNNs, facilitating the extraction of invariant representations against structural perturbations. GPT-GNN~\cite{GPT-GNN} proposes to pre-train GNNs through self-supervised generative tasks, enabling them to jointly capture structural patterns and node semantic attributes in graphs. %PGCL~\cite{PGCL} improves the quality of graph representations by learning prototype vectors and aligning node embeddings with their corresponding prototypes. G-Zoom~\cite{G-zoom} enhances self-supervised graph representation learning by capturing both local and global semantics through a contrastive multi-scale zooming strategy.
GCC~\cite{GCC} introduces contrastive graph context prediction as its pre-training task, empowering GNNs to capture transferable structural motifs across diverse network domains.
\textbf{Graph Adapter-Tuning.}
Pre-trained graph models leverage self-supervised learning to capture universal graph structural knowledge, yet their full-parameter fine-tuning suffers from high computational overhead and overfitting risks on downstream tasks. To address this, the `pre-train, adapter-tuning' paradigm has emerged, which inserts adapter modules with lightweight neural network architecture to bridge the knowledge transfer between pre-trained models and downstream tasks~\cite{adapter}.
For example, G-Adapter~\cite{g-adapter} addresses the critical challenge of feature distribution shift in graph neural network transfer learning by integrating graph topological relation encoding and Bregman divergence-based optimization strategies. GConv-Adapter~\cite{papageorgiougraph} combines dual-normalized graph convolution and trainable low-rank weight matrices. It effectively captures second-order adjacency information which enhances structural awareness and enables parameter-efficient fine-tuning.
% UA-SAM~\cite{jiang2024uncertainty} proposes a parameter-efficient uncertainty-aware adapter that injects stochastic latent samples into SAM adapters to generate diverse uncertainty-aware segmentation hypotheses.
VRGAdapter~\cite{jiang2025beyond} proposes a parameter-efficient adapter that generates multiple textual descriptions per class, encodes them as Gaussian-distributed nodes in a random knowledge graph, and uses reparameterization to sample the adapter representations. AdapterGNN~\cite{adaptergnn} proposes a parameter-efficient fine-tuning adapter architecture to address the problem of insufficient generalization capability and excessive computational resource consumption existed in traditional full-parameter fine-tuning methods. % during transfer learning.
\section{Preliminaries}
% \subsection{Graph Neural Networks}
% Given a graph \(\mathcal{G}=(\mathcal{V, E}, \mathbf{X}_{\mathcal{V}}, \mathbf{X}_{\mathcal{E}})\), we use \(\mathcal{V}\) and \(\mathcal{E}\) to denote the set of nodes and edges, respectively. The node feature matrix \(\mathbf{X}_{\mathcal{V}} \in \mathbb{R}^{N \times \mathbb{F}_{\mathcal{V}}}\) is associated with each node \(v \in \mathcal{V}\), where \(N\) is the number of nodes, and \(\mathbb{F}_{\mathcal{V}}\) denotes the feature dimension. Each edge \((v_i, v_j) \in \mathcal{E}\) may carry a feature vector \(\mathbf{x}_{ij} \in \mathbb{R}^{\mathbb{F}_{\mathcal{E}}}\), forming the edge feature matrix \(\mathbf{X}_{\mathcal{E}} \in \mathbb{R}^{M \times \mathbb{F}_{\mathcal{E}}}\) with \(M\) edges. The graph structure is encoded by the adjacency matrix \(\mathbf{A} \in \{0,1\}^{N \times N}\), where \(\mathbf{A}_{ij} = 1\) iff \((v_i, v_j) \in \mathcal{E}\). GNNs iteratively update node embeddings through \textbf{message-passing (MP)} operations. At layer \(l\), the embedding \(\mathbf{h}_v^{(l)}\) of node \(v\) is computed as:
% \begin{equation}
% \resizebox{0.5\textwidth}{!}{
% \begin{math}
% \mathbf{h}_v^{(l)} = \text{UPDATE}\left( \mathbf{h}_v^{(l-1)}, \text{AGGREGATE}\left( \{\mathbf{h}_u^{(l-1)} \mid u \in \mathcal{N}(v)\} \right) \right),
% \end{math}
% }
% \end{equation}
% where \(\mathcal{N}(v)\) is the neighborhood of node \(v\). The GNN framework first applies an \text{AGGREGATE} function to propagate and integrate embeddings from neighboring nodes into the central node $v_i$. This aggregation can be implemented through basic operations like mean pooling~\cite{GraphSAGE,GCN} or advanced neural architectures such as attention-based mechanisms~\cite{GAT} and multi-layer perceptrons~\cite{xu2018powerful}. The \text{UPDATE} function then transforms the aggregated result into the new representation of $v_i$.
% \subsection{Adapter Architecture}
Let \(f_\Omega:\mathcal{G} \mapsto \mathbb{R}^d\) denote a pre-trained GNN model with frozen parameters \(\Omega\) and \(\mathcal{G}=(\mathcal{V, E}, \mathbf{X},\mathbf{A})\) denote an input downstream graph with \(\mathcal{V}\) and \(\mathcal{E}\) representing the set of nodes and edges, respectively. Here, \(\mathbf{X}=(\mathbf{x}_1, \mathbf{x}_2\cdots \mathbf{x}_N)\in \mathbb{R}^{N \times d}\) represents the collection of node features and \(\mathbf{A} \in \{0,1\}^{N \times N}\) represents the adjacency matrix. %where \(\mathbf{A}_{ij} = 1\) iff \((v_i, v_j) \in \mathcal{E}\).
%is associated with each node \(v \in \mathcal{V}\), where \(N\) is the number of nodes, and $d$ denotes the feature dimension.
%Given a pre-trained GNN encoder $f_{\Omega}$ with frozen parameters $\Omega$ and input graph $\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathbf{X}, \mathbf{A})$,
The message passing in the $l$-th layer of pre-trained GNN model can be usually formulated as follows,
% \begin{equation}
% g(\mathbf{x}_l)
% \;=\;
% \mathrm{BN}\bigl(\mathrm{MLP}\bigl(\mathrm{MP}(\mathbf{x}_l)\bigr)\bigr).
% \end{equation}
\begin{equation}
{\mathbf{y}}_i^{(l)} =\mathrm{MLP} \Big( \bigoplus_{j \in \mathcal{N}(i)} m^{(l)} \big( {\mathbf{x}}_i^{(l)}, {\mathbf{x}}_j^{(l)}, {e}_{ji} \big) \Big)
\label{eq: massage_passing}
\end{equation}
where % ${h}_i^{(1)} = {x}_i$,
%${\mathbf{\hat{x}}}_i^{(l)}$ denote the GNN representation and ${\mathbf{x}}_i^{(l)}$ denotes the input representation at $l$-th layer,
$\mathcal{N}(i)$ indicates the neighborhood of node $v_i$ and $\bigoplus$ denotes a permutation-invariant aggregation operator (typically summation or mean). $m^{(l)}$ represents the message function. All parameters $\Omega$ in above message passing operation are frozen during fine-tuning.
%Given the $l$-th layer input \(\mathbf{x}^{(l)}_i \in \mathbb{R}^{d_{\mathrm{in}}}\), %adapter employs a bottleneck architecture to reduce computational overhead.
The Adapter module in GNN projects
the $l$-th layer input \(\mathbf{x}^{(l)}_i \in \mathbb{R}^{d_{\mathrm{in}}}\)
into a low-dimensional space via a down-projection matrix \(\mathbf{W}_{\mathrm{down}}\in\mathbb{R}^{d_{\text{mid}}\times d_{\text{in}}}\), followed by a $\mathrm{ReLU}$ activation and up-projection matrix \(\mathbf{W}_{\mathrm{up}}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{mid}}}\), where \(d_{\text{mid}} \ll d_{\text{in}}\).
To stabilize training and mitigate distribution shifts, it also incorporates trainable Batch Normalization (BN) after the up-projection. Formally, adapter outputs $\mathbf{z}^{(l)}_i$ as,
\begin{equation}
% \mathbf{A}(\mathbf{x}_l) =
\mathbf{z}^{(l)}_i =\mathrm{BN}\left(\mathbf{W}^{(l)}_{\text{up}} \cdot \mathrm{ReLU}\left(\mathbf{W}^{(l)}_{\text{down}} \cdot \mathbf{x}^{(l)}_i\right)\right),
\label{eq:adapter_basic}
\end{equation}
%\subsection{Parameter-Efficient Adaptation of Pre-trained GNNs}
%Each edge \((v_i, v_j) \in \mathcal{E}\) may carry a feature vector \(\mathbf{x}_{ij} \in \mathbb{R}^{d}\), forming the edge feature matrix \(\mathbf{X}_{\mathcal{E}} \in \mathbb{R}^{M \times \mathbb{F}_{\mathcal{E}}}\) with \(M\) edges.
%The graph structure is encoded by the adjacency matrix \(\mathbf{A} \in \{0,1\}^{N \times N}\), where \(\mathbf{A}_{ij} = 1\) iff \((v_i, v_j) \in \mathcal{E}\).
%Traditional full fine-tuning optimizes all parameters via:
%\begin{equation}
% \Omega^* = \arg\min_{\Omega} \sum_{(G_i, \gamma_t) \in \mathcal{D}_T} \mathcal{L}\left( f_{\Omega}(G_i), \gamma_t \right),
%\end{equation}
%where \(\mathcal{D}_T\) is the downstream dataset with graph-label pairs \((G_i, \gamma_t)\). However, this approach suffers from high computational costs and overfitting risks due to the large parameter space of \(\Omega\).
%
Then, as shown in Fig.~\ref{fig:framework}(a), the `augmented/adapted' representation $\hat{\mathbf{x}}^{(l)}_i$
can be obtained
by integrating the pre-trained GNN's representation ${\mathbf{y}}^{(l)}_i$
and adapter's output $\mathbf{z}^{(l)}_i$ (Eq.~(\ref{eq:adapter_basic})) together as
\begin{equation}
{\mathbf{\hat{x}}}^{(l)}_i = {\mathrm{BN}(\mathbf{y}}^{(l)}_i) +\mathbf{z}^{(l)}_i.
\end{equation}
%
To preserve the task-agnostic knowledge in pre-trained model \(f_{\Omega}\), one can freeze the pre-trained model and optimize the above lightweight adapter module (Eq.~(\ref{eq:adapter_basic})) parameterized by $\Theta^{(l)}=\{\mathbf{W}^{(l)}_{\mathrm{down}},\mathbf{W}^{(l)}_{\mathrm{up}}\}$, as well as a task-specific classifier based on downstream
labeled graph data.
%into the pre-trained model. The adapter based fine-tuning can be formulated as,
%\begin{equation}
% \min_{\Theta, \omega} \sum_{(\mathcal{G}_i, y_t) \in \mathcal{D}_T} \mathcal{L}\left( f_{\omega}\left( f_{\Omega, \Theta}(\mathcal{G}_i) \right), y_t \right),
%\end{equation}
%where \(\mathcal{D}_T\) denotes the downstream dataset with graph-label pairs \((\mathcal{G}_i, y_t)\). % and $\Theta=\{\mathbf{W}_{\mathrm{down}},\mathbf{W}_{\mathrm{up}}\}$ denotes the learnable adapter parameters.
%This paradigm preserves task-agnostic knowledge in \(\Omega\) while adapting to downstream tasks via minimal parameters in \(\Theta\)~\cite{g-adapter,adapter}.
% \begin{figure*}
% \centering
% \includegraphics[width=0.81\linewidth]{UadapterGNN_heng.pdf}
% \caption{Overall framework of UAdapterGNN. In each GNN layer of the pre-trained model, UAdapterGNN freezes the MLP module and inserts two probabilistic-modeling adapters before the message-passing stage. These adapters estimate the mean and variance of the latent variables. After sampling from these latent distributions, the sampled vectors are combined with the original layer outputs through a learnable scaling coefficient, yielding the final node embedding representations.}
% \label{fig:framework}
% \end{figure*}
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{UadapterGNN_1025.pdf}
% \caption{Overall framework of UAdapterGNN. In each GNN layer of the pre-trained model, UAdapterGNN freezes the MLP module and inserts two probabilistic-modeling adapters before the message-passing stage. These adapters estimate the mean and variance of the latent variables. After sampling from these latent distributions, the sampled vectors are combined with the original layer outputs through a learnable scaling coefficient, yielding the final node embedding representations.}
% \label{fig:framework}
% \end{figure}
\begin{figure*}[!htbp]
\centering
\subfloat[\footnotesize Regular Adapter in GNN]{
\includegraphics[width=0.42\linewidth]{UadapterGNN_1030_adapterGNN.pdf}
}
\hspace{1cm}
\subfloat[\footnotesize Our proposed UAdapterGNN]{
\includegraphics[width=0.42\linewidth]{UadapterGNN_1030_UadapterGNN.pdf}
}
\caption{Comparison of regular Adapter in GNN and our UAdapterGNN. In each layer of the pre-trained GNN model, UAdapterGNN learns a Gaussian module to argument the pre-trained GNN backbone on downstream tasks. %After sampling from the learned distribution, the sampled vectors are combined with the original layer's output through a learnable scaling coefficient, yielding the final node representations.
}
\label{fig:framework}
\end{figure*}
\section{Methodology}
In this section,
we propose our Uncertainty-aware Adapter (UAdapterGNN) framework to enhance the robustness and generalization of pre-trained GNNs on the downstream tasks. % during fine-tuning by explicitly modeling uncertainty inherent in graph-structured data.
The whole fine-tuning process is illustrated in Fig.~\ref{fig:framework}(b). It mainly comprises three main components: 1) message passing via a frozen pre-trained GNN backbone, 2) uncertainty representation via Gaussian random adapter and 3) feature augmentation via re-parameterized operator.
Below, we introduce them in detail.
\subsection{Message Passing via Frozen Pre-trained GNN}
% Beyond enabling task-specific adaptation, UAdapterGNN first leverages task-agnostic knowledge from pre-trained GNNs. Therefore, we freeze the parameters of the pre-trained GNN backbone and utilize its message-passing mechanism to extract initial graph representations.
Given a pre-trained GNN encoder $f_{\Omega}$ with frozen parameters $\Omega$ and input graph $\mathcal{G} = (\mathcal{V}, \mathcal{E}, \mathbf{X}, \mathbf{A})$, similar to the above message passing Eq.(\ref{eq: massage_passing}) in standard AdapterGNN~\cite{adaptergnn}, the message passing at $l$-th layer can be generally formulated as follows,
%
\begin{equation}
{\mathbf{y}}_i^{(l)} =\mathrm{MLP} \Big( \bigoplus_{j \in \mathcal{N}(i)} m^{(l)} \big( {\mathbf{x}}_i^{(l)}, {\mathbf{x}}_j^{(l)}, {e}_{ji} \big) \Big)
\end{equation}
%
% \begin{equation}
% g(\mathbf{x}_l)
% \;=\;
% \mathrm{BN}\bigl(\mathrm{MLP}\bigl(\mathrm{MP}(\mathbf{x}_l)\bigr)\bigr).
% \end{equation}
%\begin{equation}
%{\mathbf{\hat{x}}}_i^{(l)} = BN\bigg(MLP \Big( \bigoplus_{j \in \mathcal{N}(i)} \psi^{(l)} \big( {\mathbf{x}}_i^{(l)}, {\mathbf{x}}_j^{(l)}, {e}_{ji} \big) \Big) \bigg)
%\end{equation}
where % ${h}_i^{(1)} = {x}_i$,
%${\mathbf{\hat{x}}}_i^{(l)}$ denote the GNN representation and ${\mathbf{x}}_i^{(l)}$ denotes the input representation at $l$-th layer,
$\mathcal{N}(i)$ indicates the neighborhood of node $v_i$ and $\bigoplus$ denotes a permutation-invariant aggregation operator (typically summation or mean operator). $m^{(l)}$ % and $\psi^{(l)}$
represents the message function. All parameters in $\Omega$ remain frozen during the fine-tuning process.
\subsection{Uncertain Modeling via Gaussian Random Adapter}
%Adapter modules have emerged as parameter-efficient solutions to bridge pre-trained knowledge and downstream tasks~\cite{adapter, adaptergnn}. However,
%Existing adapter methods adopt \textit{deterministic} transformation, which ignore the inherent uncertainty in graph data and thus leading to limited generalization ability and robustness to graph noise.
%
%To overcome these limitations, we propose the Gaussian Random Adapter to extend the deterministic graph adaptation to probabilistic adaptation.
%To be specific,
Unlike traditional AdapterGNN that outputs
task-related node representation $\mathbf{z}^{(l)}_i$ (Eq.~(\ref{eq:adapter_basic})) via a lightweight bottleneck layer,
%map input $\mathbf{x}^{(l)}_i$ to the fixed output $z^{(l)}_i$ via bottleneck layers,
our UAdapterGNN models task-related node representation $\mathbf{z}^{(l)}_i$ as a Gaussian distribution that allows adaptive adjustment to noise-induced variations.
%
To obtain $\mathbf{z}^{(l)}_i$, % the above probabilistic representation learning,
we adopt two parameterized modules to estimate the mean $\mathbf{\mu}^{(l)}_i$ and variance $\mathbf{\sigma}^{(l)}_i$, respectively.
To be specific,
for the input ${\mathbf{x}}_i^{(l)}$, we obtain the \emph{mean} vector via
a bottleneck layer as,
%based on mean adapter which can be defined as follows,
\begin{equation}
\mu^{(l)}_i = \mathrm{BN}\left( \mathbf{W}^{(l)}_{\text{up}} \cdot \mathrm{ReLU}( \mathbf{W}^{(l)}_{\text{down}} \cdot {\mathbf{x}}_i^{(l)} ) \right)
\end{equation}
where $\mathbf{W}^{(l)}_{\text{down}}$ and $\mathbf{W}^{(l)}_{\text{up}}$ denote the layer-specific trainable parameters. Similarly, the \emph{variance} vector can be computed via another bottleneck layer as
\begin{equation}
{\sigma}^{(l)}_i = \mathrm{BN}\left( \mathbf{\tilde{W}}^{(l)}_{\text{up}} \cdot \mathrm{ReLU}\big( \mathbf{\tilde{W}}^{(l)}_{\text{down}} \cdot {\mathbf{x}}_i^{(l)} \big) \right)
\end{equation}
where $\mathbf{\tilde{W}}^{(l)}_{\text{down}}$ and $\mathbf{\tilde{W}}^{(l)}_{\text{up}}$ are learnable parameters.
%To enhance numerical stability, we estimate $\log(\sigma)$ rather than $\sigma$ directly, as suggested in works~\cite{neculai2022probabilistic,wei2024robust}.
Based on learned ${\mu}^{(l)}_i$ and ${\sigma}^{(l)}_i$, we can obtain the Gaussian distribution representation for node $i$
as
%
\begin{equation}
\mathbf{z}^{(l)}_i \sim \mathcal{N}\big(\mathbf{\mu}^{(l)}_i, {\mathrm{diag}(\mathbf{\sigma}}^{(l)}_i)\big)
\end{equation}
%where $\mu^{(l)}_i$ and ${\sigma}^{(l)}_i$ denote the mean and variance of the Gaussian distribution, respectively.
% Then, we obtain the final Gaussian distribution $z^{(l)}_i \sim \mathcal{N}(\mu^{(l)}_i, {\sigma^2}^{(l)}_i\textbf{I})$.
%Let $\Theta=\{ \mathbf{{W}}^{(l)}_{\text{down}},\mathbf{{W}}^{(l)}_{\text{up}},\mathbf{\tilde{W}}^{(l)}_{\text{down}},\mathbf{\tilde{W}}^{(l)}_{\text{up}}\}_{l=1}^{L}$ be parameter set,
%we call the above parallel mean and variance adapters as \textbf{Gaussian Random Adapter} parameterized by $\Theta$.
%
%Note that, our Gaussian random adapter can capture uncertain information by allowing node representations to vary within the distribution. The mean vector $\mu^{(l)}_i$ can represent the deterministic component of node representation and preserve task-specific knowledge while variance vector ${\sigma^2}^{(l)}_i$ captures the predictive uncertainty of node representation.
% \paragraph{Reparameterized Sampling}
\subsection{Re-parameterized Operator}
After obtaining the above Gaussian random adapter representation $\mathbf{z}^{(l)}_i$, we need to integrate it %the uncertainty-aware representation $\mathbf{z}^{(l)}_i$
with the original pre-trained GNN layer's output $\mathbf{y}^{(l)}_i$ for feature augmentation.
One simple way is to conduct sampling based on the learned Gaussian distribution $\mathcal{N}(\mu^{(l)}_i,\mathrm{diag}(\sigma^{(l)}_i))$.
However, direct sampling from the distribution leads to non-differentiable operation that prevents gradient-based optimization.
Therefore, we employ a re-parameterization trick~\cite{vae} to enable end-to-end training.
%
Specifically, we first sample a random noise vector $\epsilon$ from the standard normal distribution and then compute the equivalent sampled representation as follows,
\begin{equation}
\mathbf{{z}}^{(l)}_i = \mu^{(l)}_i+\epsilon \odot \sigma^{(l)}_i,\quad \epsilon \sim \mathcal{N}(\textbf{0},\;\textbf{I}),
\end{equation}
After that,
%Besides, a key challenge in adapter-based fine-tuning is balancing the preservation of pre-trained knowledge and task-specific adaptation. Therefore, % beyond the existing fixed fusion strategies (e.g., simple addition or concatenation),
we introduce an adaptive integration mechanism based on learnable scaling factor to dynamically balance the pre-trained GNN layer's output and the uncertainty-aware adaptation $\mathbf{{z}}^{(l)}_i$. Therefore, the final node feature augmentation is obtained as follows,
%Let $\mathbf{{x}}^{(l+1)}_i$ and $z^{(l)}_i$ denote the pre-trained GNN representation and the uncertainty-aware representation, respectively,
%the final node representation of layer $l+1$ can be obtained as,
\begin{equation}
\mathbf{\hat{x}}^{(l)}_i
= {\mathrm{BN}(\mathbf{y}}^{(l)}_i)
\;+\; s^{(l)} \cdot \mathbf{z}^{(l)}_i
\label{eq:final_integration}
\end{equation}
The scaling factor $s^{(l)}$ is defined as the trainable parameter during the fine-tuning process.
%This adaptive fusion ensures that UAdapterGNN retains robust pre-trained knowledge while flexibly adapting to downstream tasks, striking an optimal balance between generalization and task-specific performance.
Finally, the augmentation representation
${\mathbf{\hat{x}}}^{(l)}_i$ is passed to the next GNN layer and iteratively refined by the next uncertainty-aware adapter.
\textbf{Remark.} Figure~\ref{fig:framework}(b) shows the architecture of our UAdapterGNN module. Note that, similar to traditional AdapterGNN~\cite{adaptergnn} (Fig.~\ref{fig:framework}(a)), it offers a general plug-and-play adapter architecture which can be integrated with
any pre-trained GNN models to enable robust learning on the
downstream learning tasks, as validated in Experiments below.
%Note that, we sample ${z}^{(l)}_i$ from Gaussian distribution to \textcolor{red}{allow adaptively down-weight high-uncertainty nodes} during training while we use mean vector $\mu^{(l)}_i$ as task-related adaptation ${z}^{(l)}_i$ \textcolor{red}{to marginalize out the uncertainty, ensuring computational efficiency and numerical stability} during inference.
%\subsection{Optimization Objective}
\begin{table*}[!htbp]
\centering
\caption{Test ROC-AUC (\%) performence on molecular prediction benchmarks using various pre-training strategies and tuning methods.}
\label{tab:benchmark}
\renewcommand{\arraystretch}{1.1} % 设置行间距
\definecolor{lightgray}{gray}{0.5} % 定义浅灰色
\resizebox{\linewidth}{!}{
\begin{tabular}{c c c c c c c c c c | c}
\toprule
\multicolumn{1}{c}{Tuning} & \multicolumn{1}{c}{Pre-training} & \multicolumn{9}{c}{Dataset} \\
Method & Method & BACE & BBBP & ClinTox & HIV & SIDER & Tox21 & MUV & ToxCast & Avg. \\
\midrule
% Full Fine-tune (100%)
\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}Full Fine-tune\\(100\%)\end{tabular}}
& EdgePred & 79.9\(_{\textcolor{lightgray}{\pm0.9}}\) & 67.3\(_{\textcolor{lightgray}{\pm2.4}}\) & 64.1\(_{\textcolor{lightgray}{\pm3.7}}\) & 76.3\(_{\textcolor{lightgray}{\pm1.0}}\) & 60.4\(_{\textcolor{lightgray}{\pm0.7}}\) & \textbf{76.0\(_{\textcolor{lightgray}{\pm0.6}}\)} & 74.1\(_{\textcolor{lightgray}{\pm2.1}}\) & 64.1\(_{\textcolor{lightgray}{\pm0.6}}\) & 70.3 \\
& ContextPred & 79.6\(_{\textcolor{lightgray}{\pm1.2}}\) & 68.0\(_{\textcolor{lightgray}{\pm2.0}}\) & 65.9\(_{\textcolor{lightgray}{\pm3.8}}\) & \textbf{77.3\(_{\textcolor{lightgray}{\pm1.0}}\)} & 60.9\(_{\textcolor{lightgray}{\pm0.6}}\) & \textbf{75.7\(_{\textcolor{lightgray}{\pm0.7}}\)} & 75.8\(_{\textcolor{lightgray}{\pm1.7}}\) & \textbf{63.9\(_{\textcolor{lightgray}{\pm0.6}}\)} & 70.9 \\
& AttrMasking & 79.3\(_{\textcolor{lightgray}{\pm1.6}}\) & 64.3\(_{\textcolor{lightgray}{\pm2.8}}\) & 71.8\(_{\textcolor{lightgray}{\pm4.1}}\) & \textbf{77.2\(_{\textcolor{lightgray}{\pm1.1}}\)} & 61.0\(_{\textcolor{lightgray}{\pm0.7}}\) & \textbf{76.7\(_{\textcolor{lightgray}{\pm0.4}}\)} & 74.7\(_{\textcolor{lightgray}{\pm1.4}}\) & \textbf{64.2\(_{\textcolor{lightgray}{\pm0.5}}\)} & 71.1 \\
& GraphCL & 74.6\(_{\textcolor{lightgray}{\pm2.2}}\) & 68.6\(_{\textcolor{lightgray}{\pm2.3}}\) & 69.8\(_{\textcolor{lightgray}{\pm7.2}}\) & 78.5\(_{\textcolor{lightgray}{\pm1.2}}\) & 59.6\(_{\textcolor{lightgray}{\pm0.7}}\) & 74.4\(_{\textcolor{lightgray}{\pm0.5}}\) & 73.7\(_{\textcolor{lightgray}{\pm2.7}}\) & 62.9\(_{\textcolor{lightgray}{\pm0.4}}\) & 70.3 \\
& SimGRACE & 74.7\(_{\textcolor{lightgray}{\pm1.0}}\) & 69.0\(_{\textcolor{lightgray}{\pm1.0}}\) & 59.9\(_{\textcolor{lightgray}{\pm2.3}}\) & 74.6\(_{\textcolor{lightgray}{\pm1.2}}\) & 59.1\(_{\textcolor{lightgray}{\pm0.6}}\) & 73.9\(_{\textcolor{lightgray}{\pm0.4}}\) & 71.0\(_{\textcolor{lightgray}{\pm1.9}}\) & 61.8\(_{\textcolor{lightgray}{\pm0.4}}\) & 68.0 \\
\midrule
% Adapter (5.2%)
\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}Adapter\\(5.2\%)\end{tabular}}
& EdgePred & 78.5\(_{\textcolor{lightgray}{\pm1.7}}\) & 65.9\(_{\textcolor{lightgray}{\pm2.8}}\) & 66.6\(_{\textcolor{lightgray}{\pm5.4}}\) & 73.5\(_{\textcolor{lightgray}{\pm0.2}}\) & 60.9\(_{\textcolor{lightgray}{\pm1.3}}\) & 75.4\(_{\textcolor{lightgray}{\pm0.5}}\) & 73.0\(_{\textcolor{lightgray}{\pm1.0}}\) & 63.0\(_{\textcolor{lightgray}{\pm0.7}}\) & 69.6 \\
& ContextPred & 75.0\(_{\textcolor{lightgray}{\pm3.3}}\) & 68.2\(_{\textcolor{lightgray}{\pm3.0}}\) & 57.6\(_{\textcolor{lightgray}{\pm3.6}}\) & 75.4\(_{\textcolor{lightgray}{\pm0.6}}\) & \textbf{62.4\(_{\textcolor{lightgray}{\pm1.2}}\)} & 74.7\(_{\textcolor{lightgray}{\pm0.7}}\) & 73.3\(_{\textcolor{lightgray}{\pm0.8}}\) & 62.2\(_{\textcolor{lightgray}{\pm0.4}}\) & 68.6 \\
& AttrMasking & 76.1\(_{\textcolor{lightgray}{\pm1.4}}\) & 68.7\(_{\textcolor{lightgray}{\pm1.7}}\) & 65.8\(_{\textcolor{lightgray}{\pm4.4}}\) & 75.6\(_{\textcolor{lightgray}{\pm0.7}}\) & 59.8\(_{\textcolor{lightgray}{\pm1.7}}\) & 74.4\(_{\textcolor{lightgray}{\pm0.9}}\) & 75.8\(_{\textcolor{lightgray}{\pm2.4}}\) & 62.6\(_{\textcolor{lightgray}{\pm0.8}}\) & 69.9 \\
& GraphCL & 72.5\(_{\textcolor{lightgray}{\pm3.0}}\) & 69.3\(_{\textcolor{lightgray}{\pm0.6}}\) & 67.3\(_{\textcolor{lightgray}{\pm7.4}}\) & 75.0\(_{\textcolor{lightgray}{\pm0.4}}\) & \textbf{59.7\(_{\textcolor{lightgray}{\pm1.2}}\)} & 74.7\(_{\textcolor{lightgray}{\pm0.4}}\) & 72.9\(_{\textcolor{lightgray}{\pm1.7}}\) & 62.9\(_{\textcolor{lightgray}{\pm0.4}}\) & 69.3 \\
& SimGRACE & 73.4\(_{\textcolor{lightgray}{\pm1.1}}\) & 64.8\(_{\textcolor{lightgray}{\pm0.7}}\) & 63.5\(_{\textcolor{lightgray}{\pm4.4}}\) & 73.9\(_{\textcolor{lightgray}{\pm1.0}}\) & \textbf{59.9\(_{\textcolor{lightgray}{\pm0.9}}\)} & 73.1\(_{\textcolor{lightgray}{\pm0.9}}\) & 70.1\(_{\textcolor{lightgray}{\pm4.6}}\) & 61.7\(_{\textcolor{lightgray}{\pm0.8}}\) & 67.6 \\
\midrule
% LoRA (5.0%)
\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}LoRA\\(5.0\%)\end{tabular}}
& EdgePred & \textbf{81.0\(_{\textcolor{lightgray}{\pm0.8}}\)} & 64.8\(_{\textcolor{lightgray}{\pm1.6}}\) & 67.7\(_{\textcolor{lightgray}{\pm1.2}}\) & 74.8\(_{\textcolor{lightgray}{\pm1.2}}\) & 60.8\(_{\textcolor{lightgray}{\pm1.1}}\) & 74.6\(_{\textcolor{lightgray}{\pm0.4}}\) & 75.0\(_{\textcolor{lightgray}{\pm1.5}}\) & 62.2\(_{\textcolor{lightgray}{\pm1.0}}\) & 70.1 \\
& ContextPred & 78.5\(_{\textcolor{lightgray}{\pm1.1}}\) & 65.3\(_{\textcolor{lightgray}{\pm2.4}}\) & 61.3\(_{\textcolor{lightgray}{\pm1.9}}\) & 74.7\(_{\textcolor{lightgray}{\pm1.6}}\) & 60.8\(_{\textcolor{lightgray}{\pm0.4}}\) & 72.9\(_{\textcolor{lightgray}{\pm0.4}}\) & 75.4\(_{\textcolor{lightgray}{\pm0.9}}\) & 63.4\(_{\textcolor{lightgray}{\pm0.2}}\) & 69.0 \\
& AttrMasking & 79.8\(_{\textcolor{lightgray}{\pm0.7}}\) & 64.2\(_{\textcolor{lightgray}{\pm1.1}}\) & 70.1\(_{\textcolor{lightgray}{\pm2.9}}\) & 76.1\(_{\textcolor{lightgray}{\pm1.4}}\) & 59.7\(_{\textcolor{lightgray}{\pm0.5}}\) & 74.6\(_{\textcolor{lightgray}{\pm0.5}}\) & 76.6\(_{\textcolor{lightgray}{\pm1.6}}\) & 61.7\(_{\textcolor{lightgray}{\pm0.4}}\) & 70.4 \\
& GraphCL & 75.1\(_{\textcolor{lightgray}{\pm0.7}}\) & 67.8\(_{\textcolor{lightgray}{\pm1.1}}\) & 65.1\(_{\textcolor{lightgray}{\pm3.5}}\) & \textbf{78.9\(_{\textcolor{lightgray}{\pm0.6}}\)} & 57.6\(_{\textcolor{lightgray}{\pm0.7}}\) & 73.9\(_{\textcolor{lightgray}{\pm0.9}}\) & 72.8\(_{\textcolor{lightgray}{\pm1.2}}\) & 62.7\(_{\textcolor{lightgray}{\pm0.6}}\) & 69.2 \\
& SimGRACE & 73.2\(_{\textcolor{lightgray}{\pm1.0}}\) & 67.5\(_{\textcolor{lightgray}{\pm0.4}}\) & 60.7\(_{\textcolor{lightgray}{\pm0.4}}\) & 74.1\(_{\textcolor{lightgray}{\pm0.5}}\) & 57.6\(_{\textcolor{lightgray}{\pm2.6}}\) & 72.2\(_{\textcolor{lightgray}{\pm0.2}}\) & 67.9\(_{\textcolor{lightgray}{\pm0.9}}\) & 61.8\(_{\textcolor{lightgray}{\pm0.2}}\) & 66.9 \\
\midrule
\midrule
% GPF (0.1%)
\multirow{4}{*}{\begin{tabular}[c]{@{}c@{}}GPF\\(0.1\%)\end{tabular}}
& EdgePred & 68.0\(_{\textcolor{lightgray}{\pm0.4}}\) & 55.9\(_{\textcolor{lightgray}{\pm0.2}}\) & 50.8\(_{\textcolor{lightgray}{\pm0.1}}\) & 66.0\(_{\textcolor{lightgray}{\pm0.7}}\) & 51.5\(_{\textcolor{lightgray}{\pm0.7}}\) & 63.1\(_{\textcolor{lightgray}{\pm0.5}}\) & 63.1\(_{\textcolor{lightgray}{\pm0.1}}\) & 55.7\(_{\textcolor{lightgray}{\pm0.5}}\) & 59.3 \\
& ContextPred & 58.7\(_{\textcolor{lightgray}{\pm0.6}}\) & 58.6\(_{\textcolor{lightgray}{\pm0.6}}\) & 39.8\(_{\textcolor{lightgray}{\pm0.8}}\) & 68.0\(_{\textcolor{lightgray}{\pm0.4}}\) & 59.4\(_{\textcolor{lightgray}{\pm0.2}}\) & 67.8\(_{\textcolor{lightgray}{\pm0.9}}\) & 71.8\(_{\textcolor{lightgray}{\pm0.8}}\) & 58.8\(_{\textcolor{lightgray}{\pm0.5}}\) & 60.4 \\
& AttrMasking & 61.7\(_{\textcolor{lightgray}{\pm0.3}}\) & 54.3\(_{\textcolor{lightgray}{\pm0.3}}\) & 56.4\(_{\textcolor{lightgray}{\pm0.2}}\) & 64.0\(_{\textcolor{lightgray}{\pm0.2}}\) & 52.0\(_{\textcolor{lightgray}{\pm0.2}}\) & 69.2\(_{\textcolor{lightgray}{\pm0.3}}\) & 62.9\(_{\textcolor{lightgray}{\pm0.9}}\) & 58.1\(_{\textcolor{lightgray}{\pm0.3}}\) & 59.8 \\
& GraphCL & 71.5\(_{\textcolor{lightgray}{\pm0.6}}\) & 63.7\(_{\textcolor{lightgray}{\pm0.4}}\) & 64.5\(_{\textcolor{lightgray}{\pm0.6}}\) & 70.3\(_{\textcolor{lightgray}{\pm0.5}}\) & 55.3\(_{\textcolor{lightgray}{\pm0.6}}\) & 65.5\(_{\textcolor{lightgray}{\pm0.5}}\) & 70.1\(_{\textcolor{lightgray}{\pm0.7}}\) & 58.5\(_{\textcolor{lightgray}{\pm0.5}}\) & 64.9 \\
\midrule
MolCPT (40.0\%) & GraphCL & 74.1\(_{\textcolor{lightgray}{\pm0.5}}\) & 60.5\(_{\textcolor{lightgray}{\pm0.8}}\) & 73.4\(_{\textcolor{lightgray}{\pm0.8}}\) & 64.5\(_{\textcolor{lightgray}{\pm0.8}}\) & 55.9\(_{\textcolor{lightgray}{\pm0.3}}\) & 67.4\(_{\textcolor{lightgray}{\pm0.7}}\) & 65.7\(_{\textcolor{lightgray}{\pm2.2}}\) & 57.9\(_{\textcolor{lightgray}{\pm0.3}}\) & 64.9 \\
\midrule
\midrule
% AdapterGNN (5.2%)
\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}AdapterGNN\\(5.2\%)\end{tabular}}
& EdgePred & 79.0\(_{\textcolor{lightgray}{\pm1.5}}\) & 69.7\(_{\textcolor{lightgray}{\pm1.4}}\) & 67.7\(_{\textcolor{lightgray}{\pm3.0}}\) & 76.4\(_{\textcolor{lightgray}{\pm0.7}}\) & 61.2\(_{\textcolor{lightgray}{\pm0.9}}\) & 75.9\(_{\textcolor{lightgray}{\pm0.9}}\) & 75.8\(_{\textcolor{lightgray}{\pm2.1}}\) & \textbf{64.2\(_{\textcolor{lightgray}{\pm0.5}}\)} & 71.2 \\
& ContextPred & 78.7\(_{\textcolor{lightgray}{\pm2.0}}\) & 68.2\(_{\textcolor{lightgray}{\pm2.9}}\) & 68.7\(_{\textcolor{lightgray}{\pm5.3}}\) & 76.1\(_{\textcolor{lightgray}{\pm0.5}}\) & 61.1\(_{\textcolor{lightgray}{\pm1.0}}\) & 75.4\(_{\textcolor{lightgray}{\pm0.6}}\) & 76.3\(_{\textcolor{lightgray}{\pm1.0}}\) & 63.2\(_{\textcolor{lightgray}{\pm0.3}}\) & 71.0 \\
& AttrMasking & 79.7\(_{\textcolor{lightgray}{\pm1.3}}\) & 67.5\(_{\textcolor{lightgray}{\pm2.2}}\) & \textbf{78.3\(_{\textcolor{lightgray}{\pm2.6}}\)} & 76.7\(_{\textcolor{lightgray}{\pm1.2}}\) & 61.3\(_{\textcolor{lightgray}{\pm1.1}}\) & 76.6\(_{\textcolor{lightgray}{\pm0.5}}\) & 78.4\(_{\textcolor{lightgray}{\pm0.7}}\) & 63.6\(_{\textcolor{lightgray}{\pm0.5}}\) & 72.8 \\
& GraphCL & 76.1\(_{\textcolor{lightgray}{\pm2.2}}\) & 67.8\(_{\textcolor{lightgray}{\pm1.4}}\) & 72.0\(_{\textcolor{lightgray}{\pm3.8}}\) & 77.8\(_{\textcolor{lightgray}{\pm1.3}}\) & 59.6\(_{\textcolor{lightgray}{\pm1.3}}\) & 74.9\(_{\textcolor{lightgray}{\pm0.9}}\) & 75.1\(_{\textcolor{lightgray}{\pm2.1}}\) & \textbf{63.1\(_{\textcolor{lightgray}{\pm0.4}}\)} & 70.7 \\
& SimGRACE & 77.7\(_{\textcolor{lightgray}{\pm1.7}}\) & 68.1\(_{\textcolor{lightgray}{\pm1.3}}\) & 73.9\(_{\textcolor{lightgray}{\pm7.0}}\) & 75.1\(_{\textcolor{lightgray}{\pm1.2}}\) & 58.9\(_{\textcolor{lightgray}{\pm0.9}}\) & 74.4\(_{\textcolor{lightgray}{\pm0.6}}\) & 71.8\(_{\textcolor{lightgray}{\pm1.4}}\) & 62.6\(_{\textcolor{lightgray}{\pm0.6}}\) & 70.3 \\
\midrule
% UAdapterGNN (ours) (XX%)
\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}\textbf{UAdapterGNN}\\(5.2\%)\end{tabular}}
& EdgePred & 80.7\(_{\textcolor{lightgray}{\pm1.0}}\) & \textbf{71.1\(_{\textcolor{lightgray}{\pm1.0}}\)} & \textbf{70.9\(_{\textcolor{lightgray}{\pm3.6}}\)} & \textbf{76.6\(_{\textcolor{lightgray}{\pm0.7}}\)} & \textbf{62.2\(_{\textcolor{lightgray}{\pm0.9}}\)} & 75.6\(_{\textcolor{lightgray}{\pm0.5}}\) & \textbf{78.7\(_{\textcolor{lightgray}{\pm0.7}}\)} & \textbf{64.2\(_{\textcolor{lightgray}{\pm0.5}}\)} & \textbf{72.5} \\
& ContextPred & \textbf{80.3\(_{\textcolor{lightgray}{\pm0.6}}\)} & \textbf{69.6\(_{\textcolor{lightgray}{\pm1.8}}\)} & \textbf{74.3\(_{\textcolor{lightgray}{\pm1.7}}\)} & 76.8\(_{\textcolor{lightgray}{\pm0.8}}\) & 62.3\(_{\textcolor{lightgray}{\pm0.8}}\) & \textbf{75.7\(_{\textcolor{lightgray}{\pm0.5}}\)} & \textbf{79.4\(_{\textcolor{lightgray}{\pm1.3}}\)} & \textbf{63.9\(_{\textcolor{lightgray}{\pm0.4}}\)} & \textbf{72.8} \\
& AttrMasking & \textbf{80.4\(_{\textcolor{lightgray}{\pm0.9}}\)} & \textbf{69.6\(_{\textcolor{lightgray}{\pm2.7}}\)} & 77.2\(_{\textcolor{lightgray}{\pm1.2}}\) & 76.8\(_{\textcolor{lightgray}{\pm0.7}}\) & \textbf{62.8\(_{\textcolor{lightgray}{\pm0.8}}\)} & \textbf{76.7\(_{\textcolor{lightgray}{\pm0.5}}\)} & \textbf{80.1\(_{\textcolor{lightgray}{\pm1.6}}\)} & 63.9\(_{\textcolor{lightgray}{\pm0.5}}\) & \textbf{73.4} \\
& GraphCL & \textbf{77.3\(_{\textcolor{lightgray}{\pm2.9}}\)} & \textbf{70.2\(_{\textcolor{lightgray}{\pm0.7}}\)} & \textbf{74.7\(_{\textcolor{lightgray}{\pm2.4}}\)} & 77.3\(_{\textcolor{lightgray}{\pm0.5}}\) & \textbf{59.7\(_{\textcolor{lightgray}{\pm0.7}}\)} & \textbf{75.5\(_{\textcolor{lightgray}{\pm0.5}}\)} & \textbf{76.9\(_{\textcolor{lightgray}{\pm1.6}}\)} & \textbf{62.9\(_{\textcolor{lightgray}{\pm0.4}}\)} & \textbf{71.8} \\
& SimGRACE & \textbf{80.4\(_{\textcolor{lightgray}{\pm1.7}}\)} & \textbf{70.6\(_{\textcolor{lightgray}{\pm0.6}}\)} & \textbf{76.4\(_{\textcolor{lightgray}{\pm1.2}}\)} & \textbf{75.2\(_{\textcolor{lightgray}{\pm0.6}}\)} & 59.6\(_{\textcolor{lightgray}{\pm0.5}}\) & \textbf{74.9\(_{\textcolor{lightgray}{\pm0.4}}\)} & \textbf{74.5\(_{\textcolor{lightgray}{\pm2.3}}\)} & \textbf{63.1\(_{\textcolor{lightgray}{\pm0.3}}\)} & \textbf{71.8} \\
\bottomrule
\end{tabular}}
\end{table*}
\section{Experiments}
In this section, we conduct experiments to assess the performance and benefits of the proposed UAdapterGNN.
\subsection{Experimental Setup}
\textbf{Datasets.}
In experiments, we utilize the chemistry datasets used in previous works~\cite{hu2019strategies,adaptergnn} for pre-training and downstream tasks. The pre-training dataset includes 2 million unlabeled molecules sampled from the ZINC15 database~\cite{sterling2015zinc}, along with 456K labeled molecules from the preprocessed ChEMBL dataset~\cite{mayr2018large, gaulton2012chembl}. For downstream tasks, we evaluate the proposed UAdapterGNN on eight molecular classification datasets from MoleculeNet~\cite{wu2018moleculenet}.
% Below, we will also present the statistical details of the downstream task datasets.
% \begin{table}[!htbp]
% \caption{Statistics of datasets for downstream tasks.
% \label{tab:dataset}}
% \centering
% \setlength{\tabcolsep}{2pt}
% \renewcommand{\arraystretch}{1.2}
% \resizebox{\linewidth}{!}{
% \begin{tabular}{ccccccccc}
% \toprule
% \textbf{Dataset} & BACE & BBBP & SIDER & ClinTox & HIV & MUV & Tox21 & Toxcast \\ \midrule
% \textbf{\# Molecules} & 1513 & 2039 & 1427 & 1478 & 41127 & 93087 & 7831 & 8575 \\
% \textbf{\# Binary prediction tasks} & 1 & 1 & 27 & 2 & 1 & 17 & 12 & 617 \\ \bottomrule
% \end{tabular}
% }
% \end{table}
\textbf{Pre-training Strategies. }
We adopt five widely used strategies to pre-train the model, which are listed as below:
\begin{itemize}
% Masking
\item \textit{AttrMasking~\cite{hu2019strategies}} aims to mask node or edge attributes and then train GNNs by predicting these attributes.
% Contextpred
\item \textit{ContextPred~\cite{hu2019strategies}} proposes to perform node or subgraph context prediction for GNNs pre-training.
% Edgepred
\item \textit{EdgePred~\cite{GraphSAGE}} predicts the existence of edge between node pairs by structure reconstruction task.
% GraphCL
\item \textit{GraphCL~\cite{you2020graph}} contrasts embeddings between different augmented graph views.
% SimGRACE
\item \textit{SimGRACE~\cite{xia2022simgrace}} contrasts embeddings between original and perturbed GNN encoders without data augmentation.
\end{itemize}
% We adopt five widely used strategies (tasks) to pre-train the GNN models, which are listed as below: AttrMasking~\cite{hu2019strategies} masks node or edge attributes and trains GNNs to predict these attributes based on the surrounding graph structure. ContextPred ~\cite{hu2019strategies} uses subgraphs to predict their surrounding graph structures, aiming to map nodes appearing in similar structural contexts to nearby embeddings in the latent space. EdgePred is a regular graph reconstruction task used by many models, where the objective is to predict the existence of edges between node pairs, typically implemented through negative sampling and pairwise scoring to enhance the model's ability to discern structural dependencies. GraphCL encourages embeddings of augmented versions of an anchor graph (positive samples) to be close to each other, while pushing embeddings of other samples (negatives) apart. The augmentation strategies proposed by~\cite{you2020graph} are used to generate positive and negative samples. SimGRACE~\cite{xia2022simgrace} takes the GNN model and its perturbed version as two encoders to generate two correlated views for contrastive learning, without requiring data augmentations.
\textbf{Implementation Details. }
We adopt 5-layer Graph Isomorphism Network (GIN)~\cite{xu2018powerful} as the GNN backbone with the hidden dimension of $300$.
In our experiments, the bottleneck dimension of adapter module is tuned from $\{15, 20, 30\}$ to balance efficiency and representation capacity. We select the learning rate from $\{0.001, 0.005\}$ and then train the model using Adam optimizer~\cite{Kingma2014AdamAM} with epochs of $100$ and batch size of $32$. The number of sampling instances is varied from $\{1, 3, 5, 7\}$. The scaling coefficient with initialization of $0.01$ is dynamically learned to balance task-specific adaptation and pre-trained knowledge retention. All experiments are run five times with different random seeds.
\subsection{Comparison with Related Works}
\textbf{Comparison Methods. }
To validate the effectiveness of our UAdapterGNN, we compare it with some graph fine-tuning methods as follows. First, we consider the full fine-tuning method, which updates all parameters of the pre-trained GNN. Then, we select several parameter-efficient fine-tuning (PEFT) methods adapted to graph-structured data for comparison: (1) Adapter method: Adapter~\cite{adapter} inserts lightweight adapter modules into the model layers and AdapterGNN~\cite{adaptergnn} incorporates dual parallel adapters in each GNN layer. (2) LoRA method: LoRA~\cite{hu2022lora} optimizes a subset of parameters via low-rank decomposition. (3) Prompting method: GPF~\cite{gpf} appends learnable features to input embeddings with minimal parameter overhead and MolCPT~\cite{molcpt} injects molecular motif prompts to augment molecular graphs in latent space. For fairness, all comparison baselines utilize identical pre-trained weights from established strategies which are reproduced by using their provided implementations.
\begin{table*}[!htbp]
\centering
\normalsize
\caption{Robustness analysis with random structural noise injection on Tox21 and ToxCast datasets.}
\label{tab:robustness}
\definecolor{lightgray}{gray}{0.5} % 定义浅灰色
\resizebox{0.85\linewidth}{!}{
\setlength{\tabcolsep}{3pt} % 压缩列间距
\renewcommand{\arraystretch}{1.25} % 调整行高
\begin{tabular}{@{} c| c| *{5}{c} |*{5}{c} @{}} % 新增左列
\toprule
% \multirow{2}{*}{Dataset} & \multirow{2}{*}{Methods}
& & \multicolumn{5}{c|}{Edge Deletion Level}
& \multicolumn{5}{c}{Edge Addition Level} \\
% \cmidrule(lr){3-7} \cmidrule(lr){8-12}
\midrule
Dataset & Methods & 0\% & 20\% & 40\% & 60\% & 80\% & 0\% & 20\% & 40\% & 60\% & 80\% \\
\midrule
% Tox21 数据集部分
\multirow{3}{*}{Tox21}
& Full Fine-tune & 76.25 & 71.07 & 67.24 & 65.68 & 65.26 & 76.25 & 72.41 & 68.26 & 66.69 & 63.00 \\
& AdapterGNN & 76.41 & 71.41 & 67.59 & 66.15 & 65.20 & 76.41 & 72.60 & 68.81 & 66.62 & 64.00 \\
& UAdapterGNN(Ours) & 76.58 & 71.93 & 68.45 & 66.90 & 67.21 & 76.58 & 72.98 & 69.54 & 68.08 & 66.61 \\
% & Imp (\%) & 0.33\% & 0.97\% & 1.54\% & 1.50\% & 3.04\% & 0.33\% & 0.66\% & 1.47\% & 2.14\% & 4.90\% \\
\midrule
% Toxcast 数据集部分
\multirow{3}{*}{Toxcast}
& Full Fine-tune & 62.96 & 60.88 & 58.89 & 56.82 & 55.49 & 62.96 & 61.03 & 58.02 & 56.70 & 55.43 \\
& AdapterGNN & 63.32 & 61.55 & 59.25 & 57.11 & 55.66 & 63.32 & 60.75 & 57.71 & 56.72 & 55.61 \\
& UAdapterGNN(Ours) & 63.88 & 61.64 & 60.02 & 57.92 & 57.21 & 63.88 & 61.38 & 58.36 & 57.67 & 56.78 \\
% & Imp (\%) & 1.17\% & 0.70\% & 1.61\% & 1.68\% & 2.94\% & 1.17\% & 0.81\% & 0.86\% & 1.69\% & 2.27\% \\
\bottomrule
\end{tabular}}
\end{table*}
\begin{figure*}[!htbp]
\centering
\includegraphics[width=0.8\linewidth]{generalization_curve.pdf}
\caption{Generalization result on SIDER dataset. As shown in the figure, the variant without the Gaussion random adapter exhibits a notably larger gap, demonstrating the effectiveness of our uncertainty-aware adapter in enhancing the model’s generalization performance.}
\label{fig:sider_generalization}
\end{figure*}
\textbf{Comparison Results. }
Table~\ref{tab:benchmark} reports the comparison results of all methods on eight datasets. Here, we can observe that:
% The experimental results demonstrate that our proposed UAdapterGNN framework significantly outperforms existing methods across most pre-training strategies and datasets.
(1) Compared to conventional fine-tuning, parameter-efficient methods obtain competitive performance with only a small fraction of trainable parameters, demonstrating the effectiveness and efficiency of PEFT in enhancing model adaptability and downstream performance.
(2) Compared to LoRA~\cite{hu2022lora} and prompt based PEFT methods, our UAdapterGNN achieves better performance on almost all datasets, which indicates the advantages of adapter tuning in model knowledge transferring.
(3) Moreover, compared to other adapter-style methods including {Adapter}~\cite{adapter}, {LoRA}~\cite{hu2022lora}, {AdapterGNN}~\cite{adaptergnn} with comparable parameter budgets, UAdapterGNN consistently attains the best results on all datasets, achieving an average ROC-AUC of 72.46\%, corresponding to an approximate 1.3\% –3.5\% improvements. This clearly validates the effectiveness of introducing uncertainty modeling via Gaussian random adapter in graph adaptation fine-tuning.
\begin{table}[!htbp]
\centering
% \scriptsize
\caption{Comparison of our learnable scaling and fixed scaling.}
\label{tab:scaling_ablation}
\resizebox{\linewidth}{!}{
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.25} % 此处将行高设置为原来的1.5倍
\definecolor{lightgray}{gray}{0.5} % 定义浅灰色
\begin{tabular}{c c c c c c c c}
\toprule
& BACE & BBBP & ClinTox & SIDER & Tox21 & ToxCast & Avg. \\ \midrule
0.01 & 79.6\(_{\textcolor{lightgray}{\pm0.8}}\) & 67.9\(_{\textcolor{lightgray}{\pm2.4}}\) & 75.3\(_{\textcolor{lightgray}{\pm1.0}}\) & 59.0\(_{\textcolor{lightgray}{\pm0.6}}\) & 74.7\(_{\textcolor{lightgray}{\pm0.5}}\) & 62.4\(_{\textcolor{lightgray}{\pm0.5}}\) & 69.8 \\
0.1 & 79.8\(_{\textcolor{lightgray}{\pm1.6}}\) & 69.4\(_{\textcolor{lightgray}{\pm1.9}}\) & 74.8\(_{\textcolor{lightgray}{\pm1.7}}\) & 62.1\(_{\textcolor{lightgray}{\pm0.5}}\) & 75.4\(_{\textcolor{lightgray}{\pm0.4}}\) & 63.1\(_{\textcolor{lightgray}{\pm1.0}}\) & 70.8 \\
0.5 & 79.3\(_{\textcolor{lightgray}{\pm1.7}}\) & 62.0\(_{\textcolor{lightgray}{\pm3.3}}\) & 72.9\(_{\textcolor{lightgray}{\pm2.3}}\) & 61.8\(_{\textcolor{lightgray}{\pm0.7}}\) & 75.4\(_{\textcolor{lightgray}{\pm0.5}}\) & 63.5\(_{\textcolor{lightgray}{\pm0.3}}\)& 69.2 \\
1 & 79.4\(_{\textcolor{lightgray}{\pm1.0}}\) & 66.6\(_{\textcolor{lightgray}{\pm1.5}}\) & 68.7\(_{\textcolor{lightgray}{\pm5.3}}\) & 61.0\(_{\textcolor{lightgray}{\pm0.4}}\) & 75.2\(_{\textcolor{lightgray}{\pm0.4}}\) & 63.4\(_{\textcolor{lightgray}{\pm0.4}}\) & 69.1 \\
5 & 68.3\(_{\textcolor{lightgray}{\pm9.5}}\) & 62.7\(_{\textcolor{lightgray}{\pm1.0}}\) & 49.6\(_{\textcolor{lightgray}{\pm4.3}}\) & 59.8\(_{\textcolor{lightgray}{\pm1.1}}\) & 73.5\(_{\textcolor{lightgray}{\pm0.7}}\) & 62.3\(_{\textcolor{lightgray}{\pm0.6}}\) & 62.7 \\
\textbf{Ours} & \textbf{80.4\(_{\textcolor{lightgray}{\pm0.9}}\)} & \textbf{69.6\(_{\textcolor{lightgray}{\pm2.7}}\)} & \textbf{77.2\(_{\textcolor{lightgray}{\pm1.2}}\)} & \textbf{62.8\(_{\textcolor{lightgray}{\pm0.8}}\)} & \textbf{76.7\(_{\textcolor{lightgray}{\pm0.5}}\)} & \textbf{63.9\(_{\textcolor{lightgray}{\pm0.5}}\)} & \textbf{71.8} \\ \bottomrule
\end{tabular}}
\end{table}
%Specifically, while \textit{AdapterGNN}~\cite{adaptergnn} is currently the only approach that consistently surpasses full fine-tuning under all pre-training strategies, UAdapterGNN further enhances this advantage with a similar parameter proportion (5.2\%). On molecular benchmarks, UAdapterGNN achieves an average ROC-AUC of 72.46\%, representing a 2.36\% improvement over full fine-tuning (70.10\%) and a 1.26\% gain compared to AdapterGNN (71.20\%)~\cite{adaptergnn}.
%This improvement validates the critical role of uncertainty modeling in adapter design: by characterizing node representations as Gaussian distributions, UAdapterGNN dynamically distinguishes reliable semantic patterns from noisy signals, thereby enhancing robustness against structural perturbations in graphs.
%Compared to other parameter-efficient tuning methods, UAdapterGNN exhibits comprehensive superiority. UAdapterGNN achieves a 3.46\% average improvement over traditional adapter methods~\cite{adapter} (69.00\%) under identical parameter constraints, demonstrating that its dual-adapter architecture and learnable scaling mechanism more effectively balance pre-trained knowledge retention with task-specific adaptation. Furthermore, although \textit{GPF}~\cite{gpf} utilizes only 0.1\% parameters, its performance remains constrained, achieving an average ROC-AUC of 61.00\%. In stark contrast, UAdapterGNN outperforms GPF by 11.46\%, underscoring that simply appending prompts at the input level fails to capture the intricate uncertainty patterns inherent in graph-structured data.
%These results highlight UAdapterGNN's capability to integrate uncertainty-aware mechanisms into adapter-tuning frameworks, offering both parameter efficiency and enhanced generalization for real-world graph learning scenarios.
\subsection{Model Analyses}
\textbf{Robustness Analysis. }
We evaluate UAdapterGNN and full fine-tuning method under different noise levels to show the robustness of the proposed uncertainty-aware modeling. We randomly add or remove different proportions of edges to generate noisy graphs and then report the comparison results in Table~\ref{tab:robustness}. We can observe that: (1) UAdapterGNN consistently achieves better performance across all noise levels. (2) As the noise level increases, the relative improvement increases significantly. For example, in the random edge addition scenario, UAdapterGNN obtains a 4.90\% improvement at 80\% edge perturbation, substantially higher than the 0.66\% improvement at 20\% edge addition. These results demonstrate that incorporating uncertainty-aware modeling in adapter module can effectively enhance the robustness against graph noises.
\begin{figure*}[!htbp]
\centering
\includegraphics[width=0.81\linewidth]{performance_curve.pdf}
\caption{Model performances with different bottleneck dimensions. Tuning the adapter’s bottleneck dimension restricts the tunable parameter space and enables knowledge transfer with very few parameters. If the bottleneck is too small, the model can under-fit and its performance will suffer.}
\label{performance_curve}
\end{figure*}
\begin{figure*}[!htbp]
\centering
\includegraphics[width=\linewidth]{train_ratio.pdf}
\caption{ROC-AUC (\%) performances with training data scaling up, compared to full fine-tuning. When data is scarce, UAdapterGNN with fewer tunable parameters achieves better results.}
\label{fig:train_ratio}
\end{figure*}
\textbf{Generalization Analysis. }
In this section, we investigate the generalization ability of UAdapterGNN on SIDER dataset. The gap between training and validation loss is commonly used to assess a model’s generalization ability~\cite{FengZDHLXYK020}, with a smaller gap indicating stronger generalization performance. We then derive a variant as a compared baseline by removing the Gaussion Random Adapter. As shown in Fig.~\ref{fig:sider_generalization}, the variant without the uncertainty module exhibits a notably larger gap, demonstrating the effectiveness of our uncertainty-aware adapter in enhancing the model’s generalization performance.
\textbf{Learnable Scaling Strategies. }
In our experiments, we adopt a learnable scaling strategy and compare it with fixed scaling factors $\{0.01, 0.1, 0.5, 1, 5\}$. As shown in Table~\ref{tab:scaling_ablation}, our method consistently outperforms all fixed settings across six datasets. We further observe that smaller scaling factors tend to better preserve pre-trained knowledge, whereas larger ones may lead to catastrophic forgetting. These results demonstrate that our learnable mechanism can dynamically achieve a balance between knowledge retention and parameter adaptation.
\textbf{Bottleneck Dimension Analysis. } It is noted that smaller bottleneck dimensions may restrict the model’s learning capacity while larger ones inevitably increase the parameter count.
To evaluate their effect in UAdapterGNN, we show the results across different dimensions, as shown in Fig.~\ref{performance_curve}. Note that the model achieves full fine-tuning performance with only 2.5\% of parameters at a dimension of $5$, while peak performance stabilizes within the range $[15, 30]$. These results suggest that the knowledge transfer from pre-trained model can be well achieved with minimal parameters by constraining the bottleneck dimension of the adapter.
\textbf{Training Size Analysis. }
Here, we validate the performance of UAdapterGNN with different sizes of training data. Fig.~\ref{fig:train_ratio} demonstrates the comparison results of UAdapterGNN and full fine-tuned model. We can observe that our UAdapterGNN consistently outperforms the traditional full fine-tuning approach in all training settings. This indicates that the Gaussian random adapter in our UAdapterGNN can mitigate the overfitting issue in scenarios with limited training samples.
\section{Conclusion}
This paper presents a novel fine-tuning approach, termed UAdapterGNN, for Graph Neural Networks (GNNs) by
exploiting Gaussian random adapter to augment the
pre-trained GNN model.
For the first time, the proposed UAdapterGNN effectively captures and utilizes uncertainty in graph data, leading to the improved performance on fine-tuning various pre-trained GNN models.
The integration of adaptive tuning allows the model to dynamically adjust based on data uncertainty, improving robustness and generalization on the downstream tasks. Our experiments demonstrate that the proposed UAdapterGNN method consistently outperforms traditional GNN fine-tuning methods, validating its effectiveness in predictive accuracy, robustness and generalization ability.
% \section*{Acknowledgments}
% This should be a simple paragraph before the References to thank those individuals and institutions who have supported your work on this article.
\bibliographystyle{IEEEtran}
\bibliography{reference}
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{wu2020comprehensive}
Z.~Wu, S.~Pan, F.~Chen, G.~Long, C.~Zhang, and P.~S. Yu, ``A comprehensive
survey on graph neural networks,'' \emph{{IEEE} Trans. Neural Networks Learn.
Syst.}, vol.~32, no.~1, pp. 4--24, 2021.
\bibitem{xie2022self}
Y.~Xie, Z.~Xu, J.~Zhang, Z.~Wang, and S.~Ji, ``Self-supervised learning of
graph neural networks: A unified review,'' \emph{IEEE transactions on pattern
analysis and machine intelligence}, vol.~45, no.~2, pp. 2412--2429, 2022.
\bibitem{hu2019strategies}
W.~Hu, B.~Liu, J.~Gomes, M.~Zitnik, P.~Liang, V.~S. Pande, and J.~Leskovec,
``Strategies for pre-training graph neural networks,'' in \emph{8th
International Conference on Learning Representations, {ICLR} 2020, Addis
Ababa, Ethiopia, April 26-30, 2020}.\hskip 1em plus 0.5em minus 0.4em\relax
OpenReview.net, 2020.
\bibitem{gpf}
T.~Fang, Y.~Zhang, Y.~Yang, C.~Wang, and L.~Chen, ``Universal prompt tuning for
graph neural networks,'' \emph{Advances in Neural Information Processing
Systems}, vol.~36, pp. 52\,464--52\,489, 2023.
\bibitem{Fu2025EdgePT}
X.~Fu, Y.~He, and J.~Li, ``Edge prompt tuning for graph neural networks,'' in
\emph{The Thirteenth International Conference on Learning Representations,
{ICLR} 2025, Singapore, April 24-28, 2025}, 2025.
\bibitem{sun2022gppt}
M.~Sun, K.~Zhou, X.~He, Y.~Wang, and X.~Wang, ``Gppt: Graph pre-training and
prompt tuning to generalize graph neural networks,'' in \emph{Proceedings of
the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}, 2022,
pp. 1717--1727.
\bibitem{liu2023graphprompt}
Z.~Liu, X.~Yu, Y.~Fang, and X.~Zhang, ``Graphprompt: Unifying pre-training and
downstream tasks for graph neural networks,'' in \emph{Proceedings of the ACM
web conference 2023}, 2023, pp. 417--428.
\bibitem{g-adapter}
A.~Gui, J.~Ye, and H.~Xiao, ``G-adapter: Towards structure-aware
parameter-efficient transfer learning for graph transformer networks,'' in
\emph{Proceedings of the AAAI Conference on Artificial Intelligence},
vol.~38, no.~11, 2024, pp. 12\,226--12\,234.
\bibitem{papageorgiougraph}
P.~Papageorgiou, H.~S. de~Oc{\'a}riz~Borde, A.~Kratsios, and M.~M. Bronstein,
``Graph low-rank adapters of high regularity for graph neural networks and
graph transformers,'' in \emph{First Workshop on Scalable Optimization for
Efficient and Adaptive Foundation Models}, 2025.
\bibitem{yang2024graphlora}
Z.~Yang, J.~Han, C.~Wang, and H.~Liu, ``Graphlora: Structure-aware contrastive
low-rank adaptation for cross-graph transfer learning,'' in \emph{Proceedings
of the 31st {ACM} {SIGKDD} Conference on Knowledge Discovery and Data Mining,
V.1, {KDD} 2025, Toronto, ON, Canada, August 3-7, 2025}.\hskip 1em plus 0.5em
minus 0.4em\relax {ACM}, 2025, pp. 1785--1796.
\bibitem{adaptergnn}
S.~Li, X.~Han, and J.~Bai, ``Adaptergnn: Parameter-efficient fine-tuning
improves generalization in gnns,'' in \emph{Proceedings of the AAAI
Conference on Artificial Intelligence}, vol.~38, no.~12, 2024, pp.
13\,600--13\,608.
\bibitem{gal2016dropout}
Y.~Gal and Z.~Ghahramani, ``Dropout as a bayesian approximation: Representing
model uncertainty in deep learning,'' in \emph{international conference on
machine learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2016, pp.
1050--1059.
\bibitem{kendall2017uncertainties}
A.~Kendall and Y.~Gal, ``What uncertainties do we need in bayesian deep
learning for computer vision?'' \emph{Advances in neural information
processing systems}, vol.~30, 2017.
\bibitem{WangZ0ZLT24}
Y.~Wang, R.~Zheng, L.~Ding, Q.~Zhang, D.~Lin, and D.~Tao, ``Uncertainty aware
learning for language model alignment,'' in \emph{Proceedings of the 62nd
Annual Meeting of the Association for Computational Linguistics, {ACL} 2024,
Bangkok, Thailand, August 11-16, 2024}.\hskip 1em plus 0.5em minus
0.4em\relax Association for Computational Linguistics, 2024, pp.
11\,087--11\,099.
\bibitem{JiWGZZWZSY23}
Y.~Ji, J.~Wang, Y.~Gong, L.~Zhang, Y.~Zhu, H.~Wang, J.~Zhang, T.~Sakai, and
Y.~Yang, ``{MAP:} multimodal uncertainty-aware vision-language pre-training
model,'' in \emph{{IEEE/CVF} Conference on Computer Vision and Pattern
Recognition, {CVPR} 2023, Vancouver, BC, Canada, June 17-24, 2023}.\hskip 1em
plus 0.5em minus 0.4em\relax {IEEE}, 2023, pp. 23\,262--23\,271.
\bibitem{HuLZHNL24}
L.~Hu, Z.~Liu, Z.~Zhao, L.~Hou, L.~Nie, and J.~Li, ``A survey of knowledge
enhanced pre-trained language models,'' \emph{{IEEE} Trans. Knowl. Data
Eng.}, vol.~36, no.~4, pp. 1413--1430, 2024.
\bibitem{cv}
K.~Zhou, J.~Yang, C.~C. Loy, and Z.~Liu, ``Conditional prompt learning for
vision-language models,'' in \emph{2022 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR)}.\hskip 1em plus 0.5em minus
0.4em\relax IEEE, 2022, pp. 16\,795--16\,804.
\bibitem{LiuJPZZXY23}
Y.~Liu, M.~Jin, S.~Pan, C.~Zhou, Y.~Zheng, F.~Xia, and P.~S. Yu, ``Graph
self-supervised learning: {A} survey,'' \emph{{IEEE} Trans. Knowl. Data
Eng.}, vol.~35, no.~6, pp. 5879--5900, 2023.
\bibitem{you2020graph}
Y.~You, T.~Chen, Y.~Sui, T.~Chen, Z.~Wang, and Y.~Shen, ``Graph contrastive
learning with augmentations,'' \emph{Advances in neural information
processing systems}, vol.~33, pp. 5812--5823, 2020.
\bibitem{GPT-GNN}
Z.~Hu, Y.~Dong, K.~Wang, K.-W. Chang, and Y.~Sun, ``Gpt-gnn: Generative
pre-training of graph neural networks,'' in \emph{Proceedings of the 26th ACM
SIGKDD international conference on knowledge discovery \& data mining}, 2020,
pp. 1857--1867.
\bibitem{GCC}
J.~Qiu, Q.~Chen, Y.~Dong, J.~Zhang, H.~Yang, M.~Ding, K.~Wang, and J.~Tang,
``Gcc: Graph contrastive coding for graph neural network pre-training,'' in
\emph{Proceedings of the 26th ACM SIGKDD international conference on
knowledge discovery \& data mining}, 2020, pp. 1150--1160.
\bibitem{adapter}
N.~Houlsby, A.~Giurgiu, S.~Jastrzebski, B.~Morrone, Q.~De~Laroussilhe,
A.~Gesmundo, M.~Attariyan, and S.~Gelly, ``Parameter-efficient transfer
learning for nlp,'' in \emph{International conference on machine
learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2019, pp. 2790--2799.
\bibitem{jiang2025beyond}
B.~Jiang, X.~Ze, B.~Wang, X.~Wang, X.~Wan, and B.~Luo, ``Beyond graph model:
Reliable vlm fine-tuning via random graph adapter,'' \emph{arXiv preprint
arXiv:2507.10355}, 2025.
\bibitem{vae}
D.~P. Kingma, M.~Welling \emph{et~al.}, ``Auto-encoding variational bayes,''
2014.
\bibitem{sterling2015zinc}
T.~Sterling and J.~J. Irwin, ``{ZINC} 15 - ligand discovery for everyone,''
\emph{J. Chem. Inf. Model.}, vol.~55, no.~11, pp. 2324--2337, 2015.
\bibitem{mayr2018large}
A.~Mayr, G.~Klambauer, T.~Unterthiner, M.~Steijaert, J.~K. Wegner,
H.~Ceulemans, D.-A. Clevert, and S.~Hochreiter, ``Large-scale comparison of
machine learning methods for drug target prediction on chembl,''
\emph{Chemical science}, vol.~9, no.~24, pp. 5441--5451, 2018.
\bibitem{gaulton2012chembl}
A.~Gaulton, L.~J. Bellis, A.~P. Bento, J.~Chambers, M.~Davies, A.~Hersey,
Y.~Light, S.~McGlinchey, D.~Michalovich, B.~Al-Lazikani \emph{et~al.},
``Chembl: a large-scale bioactivity database for drug discovery,''
\emph{Nucleic acids research}, vol.~40, no.~D1, pp. D1100--D1107, 2012.
\bibitem{wu2018moleculenet}
Z.~Wu, B.~Ramsundar, E.~N. Feinberg, J.~Gomes, C.~Geniesse, A.~S. Pappu,
K.~Leswing, and V.~Pande, ``Moleculenet: a benchmark for molecular machine
learning,'' \emph{Chemical science}, vol.~9, no.~2, pp. 513--530, 2018.
\bibitem{GraphSAGE}
W.~Hamilton, Z.~Ying, and J.~Leskovec, ``Inductive representation learning on
large graphs,'' \emph{Advances in neural information processing systems},
vol.~30, 2017.
\bibitem{xia2022simgrace}
J.~Xia, L.~Wu, J.~Chen, B.~Hu, and S.~Z. Li, ``Simgrace: A simple framework for
graph contrastive learning without data augmentation,'' in \emph{Proceedings
of the ACM web conference 2022}, 2022, pp. 1070--1079.
\bibitem{xu2018powerful}
K.~Xu, W.~Hu, J.~Leskovec, and S.~Jegelka, ``How powerful are graph neural
networks?'' in \emph{7th International Conference on Learning
Representations, {ICLR} 2019, New Orleans, LA, USA, May 6-9, 2019}, 2019.
\bibitem{Kingma2014AdamAM}
D.~P. Kingma and J.~Ba, ``Adam: {A} method for stochastic optimization,'' in
\emph{3rd International Conference on Learning Representations, {ICLR} 2015,
San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings}, 2015.
\bibitem{hu2022lora}
E.~J. Hu, Y.~Shen, P.~Wallis, Z.~Allen-Zhu, Y.~Li, S.~Wang, L.~Wang, W.~Chen
\emph{et~al.}, ``Lora: Low-rank adaptation of large language models.''
\emph{ICLR}, vol.~1, no.~2, p.~3, 2022.
\bibitem{molcpt}
C.~Diao, K.~Zhou, Z.~Liu, X.~Huang, and X.~Hu, ``Molcpt: Molecule continuous
prompt tuning to generalize molecular representation learning,'' \emph{arXiv
preprint arXiv:2212.10614}, 2022.
\bibitem{FengZDHLXYK020}
W.~Feng, J.~Zhang, Y.~Dong, Y.~Han, H.~Luan, Q.~Xu, Q.~Yang, E.~Kharlamov, and
J.~Tang, ``Graph random neural networks for semi-supervised learning on
graphs,'' in \emph{Advances in Neural Information Processing Systems 33:
Annual Conference on Neural Information Processing Systems 2020, NeurIPS
2020, December 6-12, 2020, virtual}, 2020.
\end{thebibliography}