SIGMA: Semantic Identifier Grouping for Molecular Autoregression
Extracted source text · arXiv 2603.25062
%%%%%%%% ICML 2026 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2026} with \usepackage[nohyperref]{icml2026} above.
\usepackage{hyperref}
\usepackage{silence}
\WarningFilter{hyperref}{Ignoring empty anchor}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
% Use the following line for the initial blind version submitted for review:
% \usepackage{icml2026}
% For preprint, use
\usepackage[preprint]{icml2026}
% If accepted, instead use the following line for the camera-ready submission:
% \usepackage[accepted]{icml2026}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage[table]{xcolor}
\usepackage{colortbl}
\definecolor{lightbluecell}{RGB}{220,235,255}
\definecolor{lightpinkcell}{RGB}{255,225,235}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
% % Use the postscript times font!
\usepackage{times}
\usepackage{soul}
\usepackage{url}
% \usepackage[hidelinks]{hyperref}
% \usepackage[utf8]{inputenc}
% \usepackage[small]{caption}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{booktabs}
% \usepackage{algorithm}
% \usepackage{algorithmic}
\usepackage[switch]{lineno}
\usepackage{multirow}
\usepackage{amsmath, amssymb}
\usepackage{pifont}
\newcommand{\cmark}{\ding{51}} % ✓
\newcommand{\xmark}{\ding{55}} % ✗
% \usepackage{algpseudocode}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{SIGMA: Structure-Invariant Generative Molecular Alignment via Autoregressive Contrastive Learning}
\begin{document}
\raggedbottom
\twocolumn[
\icmltitle{SIGMA: Structure-Invariant Generative Molecular Alignment for Chemical Language Models via Autoregressive Contrastive Learning}
% Autoregressive Contrastive Learning for Generative Molecular Alignment via Isomorphic Trajectory Alignment on Chemical Language Model}
% It is OKAY to include author information, even for blind submissions: the
% style file will automatically remove it for you unless you've provided
% the [accepted] option to the icml2026 package.
% List of affiliations: The first argument should be a (short) identifier you
% will use later to specify author affiliations Academic affiliations
% should list Department, University, City, Region, Country Industry
% affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order. Ideally, you
% should not use this facility. Affiliations will be numbered in order of
% appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
% Xinyu Wang, Fei Dou, Jinbo Bi, minghu song
\begin{icmlauthorlist}
\icmlauthor{Xinyu Wang}{uconn}
\icmlauthor{Fei Dou}{uga}
\icmlauthor{Jinbo Bi}{uconn}
\icmlauthor{Minghu Song}{ihb}
% \icmlauthor{Firstname5 Lastname5}{yyy}
% \icmlauthor{Firstname6 Lastname6}{sch,yyy,comp}
% \icmlauthor{Firstname7 Lastname7}{comp}
% %\icmlauthor{}{sch}
% \icmlauthor{Firstname8 Lastname8}{sch}
% \icmlauthor{Firstname8 Lastname8}{yyy,comp}
%\icmlauthor{}{sch}
%\icmlauthor{}{sch}
\end{icmlauthorlist}
\icmlaffiliation{uconn}{School of Computing, University of Connecticut, Storrs, CT, USA}
\icmlaffiliation{uga}{Institute for Artificial Intelligence, University of Georgia, Athens, GA, USA}
\icmlaffiliation{ihb}{Institute of Hydrobiology, Chinese Academy of Sciences, Wuhan, China}
% \icmlaffiliation{uconn}{Department of XXX, University of Connecticut, Location, Country}
% \icmlaffiliation{comp}{Company Name, Location, Country}
% \icmlaffiliation{sch}{School of ZZZ, Institute of WWW, Location, Country}
% \icmlcorrespondingauthor{Xinyu Wang}{xinyu.wang@uconn.edu}
% \icmlcorrespondingauthor{Firstname2 Lastname2}{first2.last2@www.uk}
% You may provide any keywords that you find helpful for describing your
% paper; these are used to populate the "keywords" metadata in the PDF but
% will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column listing the
% affiliations and the copyright notice. The command takes one argument, which
% is text to display at the start of the footnote. The \icmlEqualContribution
% command is standard text for equal contribution. Remove it (just {}) if you
% do not need this facility.
% Use ONE of the following lines. DO NOT remove the command.
% If you have no special notice, KEEP empty braces:
\printAffiliationsAndNotice{} % no special notice (required even if empty)
% Or, if applicable, use the standard equal contribution text:
% \printAffiliationsAndNotice{\icmlEqualContribution}
\begin{abstract}
Linearized string representations serve as the foundation of scalable autoregressive molecular generation; however, they introduce a fundamental modality mismatch where a single molecular graph maps to multiple distinct sequences.
This ambiguity leads to \textit{trajectory divergence}, where the latent representations of structurally equivalent partial graphs drift apart due to differences in linearization history.
To resolve this without abandoning the efficient string formulation, we propose \textbf{S}tructure-\textbf{I}nvariant \textbf{G}enerative \textbf{M}olecular \textbf{A}lignment (\textbf{SIGMA}).
Rather than altering the linear representation, SIGMA enables the model to strictly recognize geometric symmetries via a token-level contrastive objective, which explicitly aligns the latent states of prefixes that share identical suffixes.
Furthermore, we introduce \textbf{Iso}morphic \textbf{Beam} Search (\textbf{IsoBeam}) to eliminate isomorphic redundancy during inference by dynamically pruning equivalent paths.
Empirical evaluations on standard benchmarks demonstrate that SIGMA bridges the gap between sequence scalability and graph fidelity, yielding superior sample efficiency and structural diversity in multi-parameter optimization compared to strong baselines.
\end{abstract}
\begin{figure*}[t]
\centering
\includegraphics[width=1.0\textwidth]{SIGMA_T4.png}
\caption{Conceptual comparison between Global Contrastive Learning and our proposed Structure-Invariant Autoregressive Contrastive Learning (SIGMA). Top: Traditional methods treat different SMILES permutations (Views A, B, C) of the same molecule as distinct sequences, and only contrast the complete SMILES strings, causing Manifold Fragmentation where functionally equivalent states (substrings) map to disparate points in the latent space. Bottom: SIGMA enforces Trajectory Alignment. By identifying that Views A, B, and C share a topologically identical suffix, SIGMA explicitly aligns their generation paths into a unified trajectory, while pushing away structurally distinct negative samples (View D).}
\vspace{-1em}
\label{fig:SIGMA_teaser}
\end{figure*}
\section{Introduction}
\label{sec:intro}
Drug discovery has been fundamentally reshaped by Chemical Language Models (ChemLMs), which cast molecular design as a sequence modeling problem~\cite{chithrananda2020chemberta,bran2023chemcrow,bagal2021molgpt,li2023druggpt}. This paradigm parallels the success of Large Language Models (LLMs), where scaling Transformer architectures~\cite{vaswani2017attention} enables modeling of complex linguistic dependencies~\cite{devlin2019bert,radford2019language,liu2019roberta,jiang2023mistral}. Inspired by this progress, cheminformatics has adopted self-attention mechanisms under the assumption that chemical connectivity rules can be learned analogously to linguistic grammar.
To enable this linguistic processing, molecular graphs are typically serialized into 1D string representations, with the Simplified Molecular Input Line Entry System (SMILES) being the \textit{de facto} standard~\cite{weininger1988smiles}. This linearization allows models to leverage vast, unlabelled chemical repositories for pre-training, drawing from massive databases such as PubChem~\cite{wang2009pubchem}, ChEMBL~\cite{gaulton2012chembl}, and ZINC~\cite{irwin2005zinc,sterling2015zinc}. By estimating probability distributions over these molecular strings, ChemLMs have achieved significant breakthroughs across a spectrum of downstream applications. These range from the de novo generation of novel drug candidates~\cite{xu2024triple,lee2024molecule} to the precise prediction of molecular properties~\cite{wang2019smiles,honda2019smiles} and complex bioactivity modeling~\cite{seal2022integrating,seal2023merging}.
However, treating chemistry as language introduces a fundamental modality mismatch between 1D sequences and the underlying 2D/3D molecular graph. Unlike natural language, where word order is semantic, molecular serialization order is arbitrary. A single graph can correspond to factorially many valid SMILES strings depending on traversal choices. Standard autoregressive models, lacking geometric inductive biases of Message Passing Neural Networks (MPNNs)~\cite{gilmer2017neural,hu2019strategies}, treat these permutations as distinct sequences. Alternative encodings such as SELFIES~\cite{krenn2020self} and DeepSMILES~\cite{o2018deepsmiles} improve robustness but do not eliminate the one-to-many mapping issue~\cite{scalia2020evaluating,gao2022sample}. As illustrated in Figure~\ref{fig:SIGMA_teaser} (and Appendix A), this causes \textit{trajectory divergence}: two equivalent partial molecular graphs can follow different linearization paths and drift apart in latent space.
This divergence leads to \textbf{manifold fragmentation}, where chemical space splits into clusters defined by syntax rather than structure. For optimization, this is harmful: reinforcement learning agents may remain confined to one syntactic region, limiting exploration and inducing mode collapse. Existing remedies rely on Randomized SMILES augmentation~\cite{bjerrum2017smiles} or auxiliary graph encoders~\cite{zhu2023dual}, assuming invariance emerges from exposure. In practice, models often memorize frequent permutations instead of learning structural equivalence, yielding poor sample efficiency~\cite{tetko2020state}. Graph-based generators~\cite{wang2025learning} encode geometry explicitly but sacrifice the scalability and stability of sequence Transformers. A framework is needed that preserves sequence efficiency while enforcing strict geometric invariance.
Therefore, we propose \textbf{Structural-Invariant Generative Molecular Alignment (SIGMA)} via autoregressive contrastive learning, a framework designed to resolve trajectory divergence by embedding geometric consistency directly into the autoregressive backbone. Instead of relying on passive data augmentation, SIGMA employs a rigorous token-level contrastive objective (shown in Figure \ref{fig:architecture}).
We construct pairs of randomized prefix trajectories that describe the same partial graph and share identical valid future completions.
By aligning their latent representations, SIGMA reconnects fragmented regions of the latent manifold, enabling geometry-invariant understanding regardless of molecular syntax.
\begin{figure*}[t]
\centering
\includegraphics[width=1.0\textwidth]{SIGMA_Arch.png}
\caption{Token-level contrastive supervision across equivalent SMILES sequences. Two structurally equivalent SMILES (top: constrained random, bottom: original) are decoded independently using an autoregressive model. Positive alignment losses (suffix-align) are applied over matched suffix tokens (shown in blue and green), while negative repulsion (prefix-repel) is enforced between mismatched prefix tokens (in red). (Left) Token distribution alignment is computed using output logits; (Right) attention alignment compares cross-attention weights across matched positions.}
\vspace{-1em}
\label{fig:architecture}
\end{figure*}
Furthermore, {\em trajectory invariance} can be incorporated into model training but should also be recognized during inference. Commonly used sampling methods, such as random sampling and beam search, frequently suffer from isomorphic redundancy, wasting computational resources by generating multiple SMILES strings that decode to the same molecule. To address this, we introduce \textbf{Iso}morphic \textbf{Beam} Search (IsoBeam), a structure-aware decoding strategy shown in Figure~\ref{fig:isobeam}. IsoBeam dynamically detects when different search paths converge to identical subgraphs and prunes the redundant trajectories, reallocating the search budget to explore structurally distinct scaffolds.
Our main contributions are summarized as follows:
\begin{itemize}
\item We identify Manifold Fragmentation caused by trajectory divergence as the root cause of inefficiency in ChemLMs. We propose SIGMA to enforce latent isotropy via dense trajectory alignment, effectively making the sequence model behave like a graph model in latent space. \item We introduce a novel Structure-Invariant Contrastive Loss that operates at the token level during the generation of the SMILES. By maximizing the mutual information between equivalent generation paths, we decouple chemical semantics from syntactic variation.
\item We propose IsoBeam, an inference algorithm that eliminates isomorphic redundancy. Empirical results demonstrate that IsoBeam significantly increases the structural diversity of generated molecules.
% \item Extensive experiments on standard benchmarks (GuacaMol) and challenging Multi-Parameter Optimization (MPO) tasks demonstrate that SIGMA achieves state-of-the-art sample efficiency. Crucially, in Reinforcement Learning settings, SIGMA effectively mitigates mode collapse, generating highly diverse and optimized drug candidates compared to strong baselines.
\end{itemize}
\section{Related Work}
\label{sec:related_work}
%\subsection{Generative Models for Molecular Design}
{\bf Generative Models for Molecular Design.}
Deep generative models for molecules broadly fall into two categories: graph-based and sequence-based approaches. Graph generative models, such as GCPN~\cite{you2018graph}, GraphAF~\cite{shi2020graphaf}, and LO-ARM~\cite{wang2025learning}, construct molecules node-by-node or via motif assembly. These models possess intrinsic permutation invariance but typically rely on complex graph matching operations or expensive validity masking, limiting their scalability.
In contrast, ChemLMs leverage the efficiency of Transformer architectures to process molecules as linearized SMILES strings~\cite{bagal2021molgpt,chithrananda2020chemberta}. While ChemLMs scale effectively, they suffer from a modality mismatch: the linearization process destroys topological locality. Recent benchmarks indicate that this lack of geometric inductive bias leads to brittle representations, where semantically equivalent but syntactically distinct SMILES elicit inconsistent behaviors~\cite{ganeeva2024lost}. Our work aims to fill this gap, inducing the geometric rigor of graph models into the scalable architecture of ChemLMs via structural alignment.
%\subsection{Canonicalization Bias and Data Augmentation}
{\bf Canonicalization Bias and Data Augmentation.}
The many-to-one mapping from molecular graphs to SMILES strings creates isomorphic ambiguity~\cite{weininger1988smiles}. This canonicalization bias arises when models memorize the rigid syntax of canonical SMILES rather than the underlying molecular geometry. The prevalent mitigation strategy is data augmentation via Randomized SMILES~\cite{bjerrum2017smiles,arus2019randomized}. While augmentation improves robustness, naive enumeration fails to cover the full syntactic manifold due to combinatorial explosion~\cite{tetko2020state}. As a result, models often memorize specific fragments rather than learning a generalized invariant mapping. Unlike these implicit approaches, SIGMA explicitly enforces invariance through a structured contrastive objective, decoupling geometric learning from the combinatorial explosion of string enumeration.
%\subsection{Contrastive Learning in Molecular Representation}
{\bf {Contrastive Learning in Molecular Representation}.}
\textit{Global Alignment.} Early methods aligned global representations. Graph-based approaches (e.g., MolCLR~\cite{wang2022molecular}) maximize agreement between perturbed graphs. In the sequence domain, SimSon~\cite{pinheiro2022smiclr} and CONSMI~\cite{qian2024consmi} adapt SimCLR to align global SMILES embeddings (e.g., [CLS] token). While CONSMI conditions generation on invariant latents, its coarse-grained alignment is insufficient for guiding the fine-grained, step-by-step decisions required in autoregressive decoding~\cite{fabian2020molecular}.
\textit{Token-level Alignment.} To enable fine-grained control, token-wise approaches have emerged. In NLP, SimCTG~\cite{su2021tacl} uses self-contrastive loss to reduce anisotropy, forcing intra-sequence token distinctiveness to prevent degeneration. However, it lacks inter-sequence structural alignment. Molecular adaptations like FineMolTex~\cite{zhou2024token} align tokens with motifs but often necessitate complex multi-modal architectures.
\textbf{SIGMA} bridges these paradigms via \textbf{Dense Trajectory Alignment}. Unlike SimCTG (intra-sequence) or SimSon (global), SIGMA aligns token-level trajectories \textit{across} structurally equivalent views. By contrasting shared suffixes against structurally distinct negatives, it ensures generative dynamics strictly adhere to the underlying chemical topology.
\begin{figure*}[t]
\centering
\includegraphics[width=1.0\textwidth]{IsoBeam.png}
\caption{Resolving Isomorphic Redundancy with Isomorphic Beam (IsoBeam) Search. (Top) Standard Beam Search: Beam capacity is wasted on multiple linearizations of the same molecule (e.g., variants of acetophenone), causing isomorphic redundancy where topologically identical duplicates crowd the search space. (Bottom) IsoBeam Search: A Partial Graph Check verifies if prefixes share identical substructures and consistent attachment points. Lower-probability isomorphic paths are dynamically pruned (red cross), reallocating the budget to explore distinct trajectories (e.g., a pyridine-based scaffold), thereby maximizing structural diversity.}
\vspace{-1em}
\label{fig:isobeam}
\end{figure*}
\section{Methodology}
\label{sec:methodology}
%{Structural-Invariant Autoregressive Contrastive Learning
The proposed SIGMA is a framework designed to resolve the modality mismatch in autoregressive ChemLMs. Unlike standard approaches that treat SMILES permutations as independent sequences, SIGMA enforces geometric isotropy in the latent space by explicitly aligning the generative trajectories of structurally equivalent molecular views.
\subsection{Problem Formulation: Generative Modeling on Chemical Manifolds}
\label{sec:formulation}
Let $\mathcal{G} \in \mathbb{G}$ be a molecular graph defined by a set of atoms and bonds. In the context of autoregressive modeling, $\mathcal{G}$ is serialized into a sequence of tokens $S = (x_1, \dots, x_L)$ from a vocabulary $\mathcal{V}$. A fundamental challenge in this domain is the \textit{one-to-many} mapping: a single graph $\mathcal{G}$ corresponds to a factorial set of valid SMILES permutations $\Pi(\mathcal{G}) = \{S^{(1)}, \dots, S^{(K)}\}$, determined by the traversal order.
Standard ChemLMs maximize the likelihood of a sequence $S$ via the autoregressive factorization:
\begin{equation}
\log p_\theta(S) = \sum\nolimits_{t=1}^{L} \log p_\theta(x_t | x_{<t})
\end{equation}
While effective for unique sequences (e.g., natural language), this objective treats $S^{(i)}$ and $S^{(j)}$ from the same equivalence class $\Pi(\mathcal{G})$ as distinct, unrelated training examples. This leads to \textit{Trajectory Divergence}: the hidden states $h_t(S^{(i)})$ and $h_t(S^{(j)})$ corresponding to the same chemical substructure are mapped to orthogonal regions in the latent manifold. Our goal is to learn a parameterized encoder $f_\theta(\cdot)$ such that the latent representations remain invariant under the permutation group of the graph:
\begin{equation}
f_\theta(x_{<t}^{(i)}) \approx f_\theta(x_{<t}^{(j)}) \quad \text{if} \quad \text{Mol}(x_{<t}^{(i)}) \cong \text{Mol}(x_{<t}^{(j)})
\end{equation}
where $\cong$ denotes subgraph isomorphism.
\subsection{Constructing Functionally Equivalent Views}
\label{sec:view_construction}
A critical prerequisite for contrastive learning is the rigorous identification of positive pairs. Naive augmentation often leads to \textit{syntactic false positives}—prefixes that map to isomorphic subgraphs but are syntactically incompatible (e.g., one expecting a ring-closure digit, the other a branch opening). To address this, we propose a view construction pipeline rooted in \textit{Functional Equivalence}.
We construct a positive pair $(S^u, S^v)$ by sampling two randomized traversals that diverge in their history but converge to a common future trajectory. Formally, we decompose the generation process into a prefix $p$ and a suffix $s$. We aim to find two distinct prefixes $p_u$ and $p_v$ that are compatible with the \textit{exact same} suffix string $s$. The validity of the pair $(p_u, p_v)$ is conditioned on the \textbf{Dual Consistency Condition}:
\begin{enumerate}
\setlength{\itemsep}{0pt}
\setlength{\parskip}{0pt}
\setlength{\topsep}{2pt}
\setlength{\parsep}{0pt}
\item \textbf{Syntactic Divergence ($p_u \neq p_v$):} The prefixes must represent distinct graph traversal histories (e.g., starting from different atoms or traversing branches in reverse orders), ensuring the contrastive task is non-trivial.
\item \textbf{Structural Equivalence via Oracle Verification:} The concatenation of both prefixes with the shared suffix $s$ must reconstruct the identical molecular graph $\mathcal{G}$. We verify this using the InChIKey hashing oracle $\mathcal{H}$:
\begin{equation}
\mathcal{H}(\text{Mol}(p_u \oplus s)) \equiv \mathcal{H}(\text{Mol}(p_v \oplus s)) \equiv \mathcal{H}(\mathcal{G})
\end{equation}
\end{enumerate}
\paragraph{The ``Probe Suffix" Protocol.}
A practical engineering challenge is that incomplete SMILES prefixes are often chemically invalid and cannot be processed by standard informatics tools. To resolve this, we employ a \textbf{Probe Suffix Protocol}. During training view construction, if a random split point leaves open rings or unclosed branches, we temporarily append a chemically stable completion fragment $s_{probe}$ (e.g., a methyl cap or ring closure) solely for the purpose of structure verification. This ensures that the equivalence is based on the rigid topological structure rather than unstable transient states.
\paragraph{Structural Negatives.}
Standard contrastive methods often rely on random in-batch negatives. However, this is insufficient for fine-grained chemical discrimination. We introduce \textbf{Structural Negatives}: for a given anchor prefix $p_u$, we explicitly sample a negative prefix $p_{neg}$ from the batch such that $\mathcal{H}(\text{Mol}(p_{neg} \oplus s)) \neq \mathcal{H}(\mathcal{G})$. This forces the model to distinguish between truly isomorphic trajectories and those representing distinct chemical entities (e.g., stereoisomers or scaffold hops), sharpening the decision boundary in the latent space.
\subsection{Architecture and Parameterization}
\label{sec:architecture}
We instantiate our framework using a autoregressive architecture. Let $f_\theta$ denote the backbone network parameterized by $\theta$. Given a SMILES sequence $S$, the model computes a sequence of contextual hidden states $\mathbf{H} = [\mathbf{h}_1, \dots, \mathbf{h}_L]$, where $\mathbf{h}_t \in \mathbb{R}^{d_{model}}$.
\paragraph{Projection-Decoupled Mechanism.}
Directly applying contrastive loss on the backbone representations $\mathbf{H}$ restricts the expressivity required for the primary generative task. The MLE objective relies on precise syntactic features (e.g., specific ring indices `1` vs `2`) to predict the next token, whereas contrastive learning seeks to abstract away from these superficial variations. To resolve this \textbf{Syntax-Semantics Conflict}, we introduce a dedicated non-linear \textbf{Projection Head} $g_\phi$ mapping $\mathbf{H}$ to a lower-dimensional metric space $\mathcal{Z}$:
\begin{equation}
\mathbf{z}_t = g_\phi(\mathbf{h}_t) = W^{(2)}\sigma(W^{(1)}\mathbf{h}_t + b^{(1)}) + b^{(2)}
\end{equation}
where $\sigma$ is a ReLU activation, $W^{(1)} \in \mathbb{R}^{d_{model} \times d_{model}}$, and $W^{(2)} \in \mathbb{R}^{d_{proj} \times d_{model}}$. In our implementation, we set $d_{model}=768$ and $d_{proj}=128$. This bottleneck architecture acts as an information filter, preserving only the essential structural semantics required for the invariance objective, while the backbone $\mathbf{h}_t$ retains the full syntactic context for autoregressive prediction.
\paragraph{Siamese Forward Pass.}
During training, the model operates in a Siamese manner with shared weights. For a given mini-batch of $N$ molecules, we generate $2N$ augmented views (functionally equivalent pairs). These are processed in parallel to obtain the representations $\mathbf{Z}_{u}$ and $\mathbf{Z}_{v}$. The autoregressive loss $\mathcal{L}_{MLE}$ is computed on the logits derived from $\mathbf{H}$ (ensuring syntactic validity), while the structural regularization $\mathcal{L}_{SIGMA}$ is computed on the normalized projections $\hat{\mathbf{z}} = \mathbf{z} / \|\mathbf{z}\|_2$ (ensuring geometric consistency).
\subsection{Dense Trajectory Alignment Objective}
\label{sec:dense_alignment}
Previous works in molecular contrastive learning typically employ \textit{Global Alignment} (e.g., aligning [CLS] tokens). However, autoregressive generation is a sequential decision process where errors can accumulate. To enforce fine-grained consistency, we propose a \textbf{Dense Trajectory Alignment} objective.
\paragraph{Formulation.}
Consider the projected embeddings of the suffix tokens for a positive pair, denoted as $\mathbf{Z}^u_{suf}$ and $\mathbf{Z}^v_{suf}$, and a set of structural negatives. We compute the token-wise similarity matrix between the trajectories. The goal is to maximize the likelihood that the $t$-th token in the suffix of view $u$ aligns with the corresponding $t$-th token in view $v$. The loss for a specific time step $t$ in the suffix is formulated as:
\begin{equation}
\resizebox{0.98\linewidth}{!}{$
\displaystyle
\mathcal{L}_{SIGMA}^{(t)} = - \log \frac{\exp(\text{sim}(\mathbf{z}_{u,t}, \mathbf{z}_{v,t}) / \tau)}{
\exp(\text{sim}(\mathbf{z}_{u,t}, \mathbf{z}_{v,t}) / \tau) +
\sum_{k \in \mathcal{N}_{neg}} \exp(\text{sim}(\mathbf{z}_{u,t}, \mathbf{z}_{k,t}) / \tau)
}
$}
\end{equation}
% \begin{equation}
% \begin{aligned}
% \mathcal{L}_{SIGMA}^{(t)} = & \log \left( e^{\text{sim}(\mathbf{z}_{u,t}, \mathbf{z}_{v,t}) / \tau} + \sum_{k \in \mathcal{N}_{neg}} e^{\text{sim}(\mathbf{z}_{u,t}, \mathbf{z}_{k,t}) / \tau} \right) \\
% & - \text{sim}(\mathbf{z}_{u,t}, \mathbf{z}_{v,t}) / \tau
% \end{aligned}
% \end{equation}
where $\text{sim}(\cdot, \cdot)$ denotes cosine similarity and $\tau$ is a temperature parameter.
Crucially, rather than aligning a single pivot point, we average this objective over the entire suffix length $L_{suf}$:
\begin{equation}
\mathcal{L}_{SIGMA} = \tfrac{1}{L_{suf}} \sum\nolimits_{t=1}^{L_{suf}} \mathcal{L}_{SIGMA}^{(t)}
\end{equation}
% 对应的简化版算法伪代码
\begin{algorithm}[t]
\small
\caption{Isomorphic Beam Search (IsoBeam)}
\label{alg:isobeam}
\begin{algorithmic}[1]
\REQUIRE Beam size $K$, Model $\mathcal{M}$, Oracle $\mathcal{H}$ (e.g., RDKit to InChIKey)
\STATE Initialize Beam $\mathcal{B} \leftarrow \{[\text{BOS}]\}$
\FOR{$t = 1 \dots T_{\max}$}
\STATE Expand $\mathcal{B}$ to candidates via top-$k$ sampling
\STATE $\mathcal{B}_{new} \leftarrow \emptyset$, $\mathcal{S}_{seen} \leftarrow \emptyset$
\STATE Sort candidates by cumulative log-probability
\FOR{each candidate $c$ in candidates}
\STATE $M \leftarrow \text{MolFromSmiles}(c)$
\IF{$M$ is \textbf{None}}
\STATE \textit{// Keep invalid intermediates for completion}
\STATE $\mathcal{B}_{new} \leftarrow \mathcal{B}_{new} \cup \{c\}$
\ELSE
\STATE \textit{// Check structural uniqueness for valid graphs}
\STATE $id \leftarrow \mathcal{H}(M)$
\IF{$id \notin \mathcal{S}_{seen}$}
\STATE $\mathcal{B}_{new} \leftarrow \mathcal{B}_{new} \cup \{c\}$
\STATE $\mathcal{S}_{seen} \leftarrow \mathcal{S}_{seen} \cup \{id\}$
\ENDIF
\ENDIF
\IF{$|\mathcal{B}_{new}| = K$}
\STATE \textbf{break}
\ENDIF
\ENDFOR
\STATE $\mathcal{B} \leftarrow \mathcal{B}_{new}$
\ENDFOR
\STATE \textbf{Output:} $\mathcal{B}$
\end{algorithmic}
\end{algorithm}
\paragraph{Theoretical Insight: Implicit Gradient Alignment.}
A key theoretical question is: why does aligning the \textit{suffix} representations enforce invariance in the \textit{prefix} representations? We analyze this through the lens of gradient flow.
In a causal Transformer, the hidden state of a suffix token $s_t$ is a function of the prefix context $p$ and the preceding suffix tokens $s_{<t}$: $\mathbf{h}_{s_t} = \text{Transformer}(s_t | p, s_{<t})$.
In our construction, for a positive pair, the suffix inputs $s_{<t}$ are identical. The only variable is the prefix $p$. Minimizing the contrastive loss $\mathcal{L}_{SIGMA}$ between $\mathbf{h}_{s_t}^u$ and $\mathbf{h}_{s_t}^v$ necessitates minimizing the distance between the conditional distributions induced by $p_u$ and $p_v$.
Since the suffix parameters are shared, the gradient $\nabla \mathcal{L}_{SIGMA}$ backpropagates through the suffix attention layers to update the prefix encoder, effectively enforcing:
\begin{equation}
\text{Encoder}(p_u) \approx \text{Encoder}(p_v)
\end{equation}
Thus, dense trajectory alignment acts as a robust proxy for aligning the prefix representations, leveraging the causal structure of the model to propagate invariance backwards from the future (suffix) to the past (prefix).
The final training objective combines the generative and contrastive tasks: $\mathcal{L}_{total} = \mathcal{L}_{NLL} + \lambda \mathcal{L}_{SIGMA}$, where $\lambda$ is a hyperparameter balancing generation quality and structural regularization.
\begin{figure*}[t]
\centering
\includegraphics[width=0.95\linewidth]{Mols50.png}
\caption{\textbf{Latent Space Visualization of Geometric Invariance.} We utilize t-SNE to project the terminal hidden states of 50 randomized SMILES views for 10 distinct molecules (color-coded by molecular identity).
\textbf{SIGMA (Ours)} achieves superior geometric invariance, where isomorphic views collapse into tight, well-separated clusters.
In contrast, the \textbf{Baseline (MLE)} exhibits severe \textit{manifold fragmentation}, with isomorphic trajectories scattered and entangled.
\textbf{Random SMILES Augmentation} improves separation but suffers from disconnected islands, where a single molecule maps to multiple disjoint clusters.
\textbf{Last Token Contrastive Learning} fails to cleanly disentangle structural identities, demonstrating that dense token-level alignment is essential for learning isotropic representations.}
\label{fig:mols50}
\end{figure*}
\begin{figure*}[t]
\centering
\includegraphics[width=0.95\linewidth]{heatmap.png}
\caption{\textbf{Token-wise Latent Alignment Heatmaps.} We visualize the cosine similarity matrices of hidden states between two syntactically distinct SMILES linearizations of Acetophenone (y-axis: \texttt{CC(=O)...}, x-axis: \texttt{O=C(C)...}).
\textbf{SIGMA (Left)} exhibits a distinct \textbf{semantic alignment block} in the top-left region. Despite the disjoint syntax of the prefix (acetyl group), the model aligns the latent representations of \texttt{CC(=O)} and \texttt{O=C(C)}, confirming it captures the underlying structural equivalence.
In contrast, the \textbf{Baseline} and \textbf{GCL} models show near-zero similarity (dark blue) in this off-diagonal region, indicating they rely primarily on surface-level token matching and fail to recognize the structural identity of the permuted prefixes. All models successfully align the identical suffix (\texttt{c1ccccc1}, bottom-right diagonal).}
\label{fig:heatmap}
\end{figure*}
% \begin{table}[t]
% \centering
% \caption{\textbf{Distributional Consistency of Properties.} Comparison against Ground Truth (ZINC Test Set). \textbf{Sim.} denotes pairwise Tanimoto similarity.}
% \label{tab:chem_metrics}
% \setlength{\tabcolsep}{15pt}
% \renewcommand{\arraystretch}{0.9}
% \resizebox{0.9\linewidth}{!}{%
% \begin{tabular}{lccc}
% \toprule
% \textbf{Method} & \textbf{QED} & \textbf{SAS} & \textbf{Sim.} \\ \midrule
% \textit{Ground Truth} & 0.75 & 2.76 & 0.35 \\ \midrule
% LO-ARM (Graph) \cite{wang2025learning} & 0.75 & 3.08 & 0.34 \\
% GPT-2 (Canonical) & 0.68 & 3.55 & 0.42 \\
% \textbf{SIGMA-GPT (Ours)} & \textbf{0.75} & \textbf{2.80} & \textbf{0.35} \\ \bottomrule
% \end{tabular}%
% }
% \end{table}
% \subsection{Generative Modeling and Distribution Alignment}
% \begin{table}[h]
% \centering
% \caption{Evaluation of IsoBeam Search}
% \label{tab:ablation}
% \resizebox{0.85\linewidth}{!}{%
% \begin{tabular}{lccc}
% \toprule
% \textbf{Decoding Strategy} & \textbf{Validity} & \textbf{Unique Scaffolds} & \textbf{Diversity} \\
% \midrule
% Standard Beam Search & 99.8\% & 8,420 & 0.88 \\
% \textbf{IsoBeam (Ours)} & \textbf{99.8\%} & \textbf{9,650} & \textbf{0.92} \\
% \bottomrule
% \end{tabular}%
% }
% \end{table}
\subsection{Isomorphic Beam Search (IsoBeam)}
\label{sec:isobeam_method}
While SIGMA aligns the latent space during training, standard Beam Search allows the generation of redundant syntactic variations during inference. To address this, we introduce \textbf{IsoBeam}, which enforces structural uniqueness dynamically.
The core mechanism of IsoBeam is to prune candidates that correspond to already-visited molecular graphs. At each decoding step $t$, for every candidate sequence $c$ in the beam:
\begin{enumerate}
\item \textbf{Validity Check:} We first check if the current partial sequence $c$ forms a syntactically valid molecule (e.g., using RDKit).
\item \textbf{Conditional Pruning:} If $c$ is \textbf{invalid} (e.g., open rings or branches), it is retained in the beam to allow for future completion. If $c$ is \textbf{valid}, we compute its structural identifier (InChIKey). If this identifier has already been encountered in the current beam (meaning it is an isomorphic variant of a higher-probability candidate), $c$ is pruned. Otherwise, it is retained and its identifier is added to the visited set.
\end{enumerate}
This strategy ensures that the beam does not waste capacity on multiple permutations of the same valid substructure or molecule, effectively maximizing the exploration of unique chemical space.
\paragraph{Complexity vs. Diversity Trade-off.}
Although IsoBeam introduces an overhead for RDKit validation at each step, it dramatically increases the \textit{effective beam size}. In standard beam search, a beam of size $K=10$ might contain only 2 unique molecular structures (with 5 permutations each). In IsoBeam, a beam of size $K=10$ is guaranteed to contain 10 topologically distinct structures. This trade-off is highly favorable for molecular optimization tasks where sample efficiency and scaffold diversity are paramount.
\begin{table*}[t]
\centering
\caption{\textbf{PMO Benchmark Performance.} (mean $\pm$ s.d. over 3 runs). \textcolor{blue}{Blue} and \textcolor{pink}{pink} shading indicate superior performance by SIGMA and the baseline, respectively. While peak scores (Top-1, Avg-100) are comparable due to saturation, \textbf{SIGMA significantly excels in exploration (\#Scaf)}. In tasks like Osimertinib, SIGMA discovers 20--40\% more unique scaffolds, confirming that geometric invariance facilitates effective \textit{scaffold hopping} rather than local exploitation.}
\label{tab:pmo_results}
% \label{tab:optimization_full}
% 调整字体大小以适应更多列
\small
\setlength{\tabcolsep}{10pt} % 微调列间距确保不溢出
\renewcommand{\arraystretch}{1.1} % 稍微增加行高，避免拥挤
\resizebox{\linewidth}{!}{%
\begin{tabular}{l|c|ccc|ccc|ccc}
\toprule
& \textbf{Use} & \multicolumn{3}{c|}{\textbf{Optimization Quality}} & \multicolumn{3}{c|}{\textbf{Efficiency}} & \multicolumn{3}{c}{\textbf{Exploration}} \\
\multirow{-2}{*}{\textbf{Task}} & \textbf{SIGMA} & \textbf{Top-Avg 1 ($\uparrow$)} & \textbf{Tio-Avg 10 ($\uparrow$)} & \textbf{Top-Avg 100 ($\uparrow$)} & \textbf{Top-AUC 1 ($\uparrow$)} & {\textbf{Top-AUC 10} ($\uparrow$)} & \textbf{Top-AUC 100 ($\uparrow$)} & \textbf{SA ($\uparrow$)}& \textbf{Div ($\uparrow$)} & \textbf{\# Scaf ($\uparrow$)} \\
\midrule
\multicolumn{11}{c}{\textbf{MPO (Multiproperty Objective)}}\\
\midrule
% === Sitagliptin MPO ===
\multirow{2}{*}{\textbf{Sitagliptin}}
& -
& 0.552 \scriptsize{$\pm$ 0.029}
& 0.540 \scriptsize{$\pm$ 0.035}
& 0.498 \scriptsize{$\pm$ 0.032}
& 0.502 \scriptsize{$\pm$ 0.050}
& 0.473 \scriptsize{$\pm$ 0.034}
& 0.404 \scriptsize{$\pm$ 0.040}
& 2.823 \scriptsize{$\pm$ 0.356}
& 0.603 \scriptsize{$\pm$ 0.091}
& 2670 \scriptsize{$\pm$ 489} \\
& \checkmark
& \cellcolor{lightbluecell} 0.607 \scriptsize{$\pm$ 0.013}
& \cellcolor{lightbluecell} 0.596 \scriptsize{$\pm$ 0.020}
& \cellcolor{lightbluecell} 0.541 \scriptsize{$\pm$ 0.014}
& \cellcolor{lightbluecell} 0.544 \scriptsize{$\pm$ 0.011}
& \cellcolor{lightbluecell} 0.508 \scriptsize{$\pm$ 0.020}
& \cellcolor{lightbluecell} 0.423 \scriptsize{$\pm$ 0.016}
& \cellcolor{lightbluecell} 3.628 \scriptsize{$\pm$ 0.487}
& \cellcolor{lightbluecell} 0.656 \scriptsize{$\pm$ 0.073}
& \cellcolor{lightbluecell} \textbf{3143 \scriptsize{$\pm$ \phantom{0}89}} \\
\midrule
% === Zaleplon MPO ===
\multirow{2}{*}{\textbf{Zaleplon}}
& -
& 0.616 \scriptsize{$\pm$ 0.040}
& 0.611 \scriptsize{$\pm$ 0.037}
& 0.593 \scriptsize{$\pm$ 0.031}
& 0.557 \scriptsize{$\pm$ 0.011}
& 0.535 \scriptsize{$\pm$ 0.006}
& 0.493 \scriptsize{$\pm$ 0.007}
& 2.400 \scriptsize{$\pm$ 0.247}
& 0.479 \scriptsize{$\pm$ 0.083}
& 3518 \scriptsize{$\pm$ 441} \\
& \checkmark
& \cellcolor{lightbluecell} 0.635 \scriptsize{$\pm$ 0.015}
& \cellcolor{lightbluecell} 0.627 \scriptsize{$\pm$ 0.016}
& \cellcolor{lightbluecell} 0.610 \scriptsize{$\pm$ 0.013}
& \cellcolor{lightbluecell} 0.578 \scriptsize{$\pm$ 0.007}
& \cellcolor{lightbluecell} 0.558 \scriptsize{$\pm$ 0.005}
& \cellcolor{lightbluecell} 0.517 \scriptsize{$\pm$ 0.002}
& \cellcolor{lightbluecell} 2.833 \scriptsize{$\pm$ 0.046}
& \cellcolor{lightbluecell} 0.531 \scriptsize{$\pm$ 0.035}
& \cellcolor{lightbluecell}\textbf{3902 \scriptsize{$\pm$ 298}} \\
\midrule
% === Fexofenadine MPO ===
\multirow{2}{*}{\textbf{Fexofenadine}}
& -
& 0.893 \scriptsize{$\pm$ 0.012}
& 0.889 \scriptsize{$\pm$ 0.014}
& 0.877 \scriptsize{$\pm$ 0.020}
& 0.794 \scriptsize{$\pm$ 0.017}
& 0.776 \scriptsize{$\pm$ 0.016}
& 0.745 \scriptsize{$\pm$ 0.015}
& \cellcolor{lightpinkcell} 4.525 \scriptsize{$\pm$ 0.211}
& \cellcolor{lightpinkcell} 0.444 \scriptsize{$\pm$ 0.011}
& 6211 \scriptsize{$\pm$ 412} \\
& \checkmark
& \cellcolor{lightbluecell} 0.957 \scriptsize{$\pm$ 0.023}
& \cellcolor{lightbluecell} 0.945 \scriptsize{$\pm$ 0.023}
& \cellcolor{lightbluecell} 0.932 \scriptsize{$\pm$ 0.020}
& \cellcolor{lightbluecell} 0.807 \scriptsize{$\pm$ 0.006}
& \cellcolor{lightbluecell} 0.790 \scriptsize{$\pm$ 0.005}
& \cellcolor{lightbluecell} 0.758 \scriptsize{$\pm$ 0.004}
& 4.360 \scriptsize{$\pm$ 0.423}
& 0.376 \scriptsize{$\pm$ 0.059}
& \cellcolor{lightbluecell} \textbf{6272 \scriptsize{$\pm$ 467}} \\
\midrule
% === Osimertinib MPO ===
\multirow{2}{*}{\textbf{Osimertinib}}
& -
& \cellcolor{lightpinkcell} 0.910 \scriptsize{$\pm$ 0.021}
& \cellcolor{lightpinkcell} 0.906 \scriptsize{$\pm$ 0.022}
& \cellcolor{lightpinkcell} 0.896 \scriptsize{$\pm$ 0.019}
& \cellcolor{lightpinkcell} 0.847 \scriptsize{$\pm$ 0.008}
& 0.835 \scriptsize{$\pm$ 0.008}
& 0.807 \scriptsize{$\pm$ 0.007}
& \cellcolor{lightpinkcell} 3.476 \scriptsize{$\pm$ 0.257}
& 0.463 \scriptsize{$\pm$ 0.063}
& 5667 \scriptsize{$\pm$ 941} \\
& \checkmark
& 0.902 \scriptsize{$\pm$ 0.003}
& 0.899 \scriptsize{$\pm$ 0.004}
& 0.892 \scriptsize{$\pm$ 0.002}
& 0.846 \scriptsize{$\pm$ 0.004}
& \cellcolor{lightbluecell} 0.836 \scriptsize{$\pm$ 0.003}
& \cellcolor{lightbluecell} 0.808 \scriptsize{$\pm$ 0.003}
& 3.460 \scriptsize{$\pm$ 0.154}
& \cellcolor{lightbluecell} 0.532 \scriptsize{$\pm$ 0.039}
& \cellcolor{lightbluecell} \textbf{7731 \scriptsize{$\pm$ 190}} \\
\midrule
% === Perindopril MPO ===
\multirow{2}{*}{\textbf{Perindopril}}
& -
& 0.642 \scriptsize{$\pm$ 0.047}
& 0.642 \scriptsize{$\pm$ 0.047}
& 0.639 \scriptsize{$\pm$ 0.057}
& 0.550 \scriptsize{$\pm$ 0.026}
& 0.536 \scriptsize{$\pm$ 0.023}
& 0.512 \scriptsize{$\pm$ 0.017}
& 4.245 \scriptsize{$\pm$ 0.249}
& 0.340 \scriptsize{$\pm$ 0.084}
& 5295 \scriptsize{$\pm$ 317} \\
& \checkmark
& \cellcolor{lightbluecell} 0.648 \scriptsize{$\pm$ 0.005}
& \cellcolor{lightbluecell} 0.647 \scriptsize{$\pm$ 0.006}
& \cellcolor{lightbluecell} 0.643 \scriptsize{$\pm$ 0.006}
& \cellcolor{lightbluecell} 0.552 \scriptsize{$\pm$ 0.003}
& \cellcolor{lightbluecell} 0.539 \scriptsize{$\pm$ 0.002}
& \cellcolor{lightbluecell} 0.514 \scriptsize{$\pm$ 0.001}
& \cellcolor{lightbluecell} 4.383 \scriptsize{$\pm$ 0.163}
& \cellcolor{lightbluecell} 0.405 \scriptsize{$\pm$ 0.093}
& \cellcolor{lightbluecell} \textbf{6188 \scriptsize{$\pm$ 263}} \\
\midrule
\multicolumn{11}{c}{\textbf{Median Molecule}}\\
\midrule
% === Median 1 ===
\multirow{2}{*}{\textbf{Median 1}}
& -
& 0.417 \scriptsize{$\pm$ 0.024}
& 0.417 \scriptsize{$\pm$ 0.024}
& 0.407 \scriptsize{$\pm$ 0.029}
& 0.372 \scriptsize{$\pm$ 0.016}
& 0.365 \scriptsize{$\pm$ 0.016}
& 0.341 \scriptsize{$\pm$ 0.018}
& 4.748 \scriptsize{$\pm$ 0.428}
& 0.298 \scriptsize{$\pm$ 0.115}
& 1618 \scriptsize{$\pm$ 334} \\
& \checkmark
& \cellcolor{lightbluecell} 0.431 \scriptsize{$\pm$ 0.018}
& \cellcolor{lightbluecell} 0.431 \scriptsize{$\pm$ 0.018}
& \cellcolor{lightbluecell} 0.415 \scriptsize{$\pm$ 0.011}
& \cellcolor{lightbluecell} 0.393 \scriptsize{$\pm$ 0.014}
& \cellcolor{lightbluecell} 0.385 \scriptsize{$\pm$ 0.014}
& \cellcolor{lightbluecell} 0.359 \scriptsize{$\pm$ 0.012}
& \cellcolor{lightbluecell} 4.922 \scriptsize{$\pm$ 0.144}
& \cellcolor{lightbluecell} 0.458 \scriptsize{$\pm$ 0.047}
& \cellcolor{lightbluecell} \textbf{1692 \scriptsize{$\pm$ 161}} \\
\midrule
% === Median 2 ===
\multirow{2}{*}{\textbf{Median 2}}
& -
& \cellcolor{lightpinkcell} 0.349 \scriptsize{$\pm$ 0.012}
& \cellcolor{lightpinkcell} 0.342 \scriptsize{$\pm$ 0.011}
& \cellcolor{lightpinkcell} 0.329 \scriptsize{$\pm$ 0.007}
& 0.299 \scriptsize{$\pm$ 0.009}
& 0.285 \scriptsize{$\pm$ 0.004}
& \cellcolor{lightpinkcell} 0.265 \scriptsize{$\pm$ 0.003}
& \cellcolor{lightpinkcell} 2.953 \scriptsize{$\pm$ 0.314}
& \cellcolor{lightpinkcell} 0.412 \scriptsize{$\pm$ 0.014}
& 5654 \scriptsize{$\pm$ 201} \\
& \checkmark
& 0.346 \scriptsize{$\pm$ 0.002}
& 0.335 \scriptsize{$\pm$ 0.004}
& 0.323 \scriptsize{$\pm$ 0.003}
& \cellcolor{lightbluecell} 0.304 \scriptsize{$\pm$ 0.003}
& \cellcolor{lightbluecell} 0.287 \scriptsize{$\pm$ 0.002}
& \cellcolor{lightbluecell} 0.265 \scriptsize{$\pm$ 0.001}
& 2.785 \scriptsize{$\pm$ 0.138}
& 0.397 \scriptsize{$\pm$ 0.047}
& \cellcolor{lightbluecell} \textbf{5911 \scriptsize{$\pm$ 138}} \\
\midrule
\bottomrule
\end{tabular}
}
\end{table*}
\begin{table}[t]
\centering
\caption{\textbf{Unconditional Generation on ZINC250k.} Baselines marked with $\dag$ and $\ddag$ are retrieved from~\cite{wang2025learning} and~\cite{wu2024t}, respectively. \textbf{TIS} (Trajectory Invariance Score) measures latent divergence between isomorphic prefixes (lower is better). SIGMA achieves state-of-the-art FCD among sequence models, effectively bridging the gap with graph-based approaches.}
\label{tab:main_results}
\resizebox{\linewidth}{!}{%
\begin{tabular}{l|cccccc}
\toprule
\textbf{Model} & \textbf{TIS}($\downarrow$) & \textbf{Validity} ($\uparrow$) & \textbf{Uniqueness} ($\uparrow$) & \textbf{Novelty} ($\uparrow$) & \textbf{IntDiv} ($\uparrow$) & \textbf{FCD} ($\downarrow$) \\
\midrule
\multicolumn{6}{c}{\textit{Graph-based}} \\
\midrule
GraphAF$^{\dag}$ & - & 0.685 & 0.986 & - & - & 16.02 \\
MoFlow$^{\dag}$ & - & 0.631 & 0.999 & - & - & 20.93 \\
LO-ARM$^{\dag}$ & - & 0.961 & 1.000 & - & - & 3.229 \\
\midrule
\multicolumn{6}{c}{\textit{Sequence-based}} \\
\midrule
CharRNN$^{\ddag}$ & - & 0.899 & 0.690 & 0.182 & \textbf{0.992} & 0.965 \\
mGPT2$^{\ddag}$ & - & 0.853 & 0.674 & 0.672 & 0.960 & 0.830 \\
MolGPT$^{\ddag}$ & - & 0.980 & 0.770 & 0.766 & 0.989 & 0.923 \\
RandSMILES & 2.677 & 0.986 & 0.756 & 0.689 & 0.889 & 0.892 \\
LTCL & 1.698 & 0.988 & 0.798 & 0.711 & 0.910 & 0.834 \\
\midrule
\textbf{SIGMA (Ours)} & \textbf{0.041} & \textbf{0.998} & \textbf{0.814} & \textbf{0.798} & 0.910 & \textbf{0.752} \\
\bottomrule
\end{tabular}%
}
\end{table}
\section{Experiments}
\textbf{Experimental Setup}
We evaluate SIGMA on unsupervised molecular generation and goal-directed optimization. The backbone is a GPT-2 (Small) causal language model trained on \textbf{ZINC-250k}~\cite{irwin2005zinc}. Baselines include: (1) \textit{Sequence Models}, trained on canonical SMILES or randomized SMILES augmentation (\textit{RandAugment}); (2) \textit{Graph-based SOTA}, namely LO-ARM~\cite{wang2025learning}; and (3) \textit{Ablations} without the dense contrastive objective. Full implementation and hyperparameters are in Appendix~\ref{app:implementation}.
\subsection{Analysis of Geometric Invariance}
\label{sec:invariance}
A central hypothesis of this work is that standard autoregressive objectives lead to \textit{manifold fragmentation}, whereas SIGMA enforces a smooth, geometry-invariant latent space. We verify this through macroscopic visualization, microscopic alignment analysis, and quantitative consistency metrics.
\paragraph{Macroscopic: Resolving Manifold Fragmentation.}
We sample 10 molecules and generate 50 randomized SMILES views each. Terminal hidden states are projected via t-SNE. As shown in Fig.~\ref{fig:mols50}, the baseline latent space is entangled, with isomorphic views scattered. Data augmentation improves separation but still produces fragmented clusters. SIGMA collapses all permutations of a molecule into compact, well-separated clusters, indicating successful abstraction from syntactic variation.
\paragraph{Microscopic: Isomorphic Semantic Alignment.}
We analyze token-level representations using cosine similarity between hidden states of two equivalent but differently serialized trajectories of Acetophenone (Fig.~\ref{fig:heatmap}). Baselines show alignment only along identical suffix tokens. SIGMA exhibits a clear off-diagonal semantic block, demonstrating alignment of equivalent chemical substructures despite different character sequences. This shows SIGMA learns structural semantics rather than memorizing syntax.
% \paragraph{Quantitative Metric: Isomorphic Consistency.}
% To quantify these observations, we report the Mean Isomorphic Similarity (MIS), defined as the average cosine similarity between the representations of randomized views of the same molecule. As shown in Table~\ref{tab:invariance}, SIGMA achieves a near-perfect MIS of 0.98, significantly outperforming the Baseline (0.62) and RandAugment (0.85), validating that the contrastive objective effectively resolves trajectory divergence.
\subsection{Unconditional Generation: Bridging Sequence and Graph Models}
\label{sec:unconditional}
We next investigate whether the learned geometric invariance translates to superior generative quality on the ZINC-250k benchmark. Historically, a trade-off exists in generative chemistry: sequence models achieve high validity but suffer from distribution shift (high Fréchet ChemNet Distance, FCD), while graph models capture the distribution well but often struggle with validity constraints.
As presented in Table~\ref{tab:main_results}, SIGMA effectively bridges this gap.
\begin{itemize}
\item \textbf{Validity \& Uniqueness:} SIGMA maintains the near-perfect validity ($>99.8\%$) characteristic of autoregressive sequence models, significantly outperforming graph-based baselines like GraphAF and MoFlow.
\item \textbf{Distributional Fidelity (FCD):} Crucially, SIGMA achieves a Fréchet ChemNet Distance (FCD) of \textbf{0.752}. This is not only a substantial improvement over the standard sequence baselines (e.g., CharRNN: 0.965, RandSMILES: 0.892) but also establishes a new state-of-the-art among sequence-based methods.
\end{itemize}
This result confirms that by resolving manifold fragmentation, SIGMA learns a chemical probability distribution that is both syntactically robust (high validity) and chemically realistic (low FCD), effectively competing with computationally expensive graph autoregressive models (e.g., LO-ARM).
\subsection{Optimization Efficiency in Reinforcement Learning}
\label{sec:optimization}
To rigorously assess if SIGMA's geometric invariance improves optimizability, we adopt a \textbf{plug-and-play protocol} using the standard REINVENT algorithm~\cite{olivecrona2017molecular} as a fixed backbone. This design isolates the quality of the generative prior as the sole variable affecting performance. We evaluate on the PMO Benchmark (MPO and exploration tasks), with results summarized in Table~\ref{tab:pmo_results}.
Regarding raw optimization, SIGMA performs comparably to the baseline, an expected outcome due to RL-driven score saturation. However, a critical divergence appears in structural diversity. SIGMA generates significantly more unique scaffolds (\#Scaf) across all tasks. For instance, in the Osimertinib task, SIGMA identifies over \textbf{7,700 unique scaffolds} a $\sim$40\% increase over the baseline (5,600) despite achieving similar peak scores.
This highlights distinct exploration dynamics. While the baseline suffers from mode collapse (generating trivial variations of local optima), SIGMA leverages its isotropic latent space to facilitate effective \textbf{scaffold hopping}. By traversing the connected manifold of isomorphic views, SIGMA avoids syntactic minima, covering a significantly broader chemical space without sacrificing optimization success.
\subsection{Breaking the Redundancy Ceiling: Isomorphic Beam Search}
\label{sec:isobeam}
We address a decoding bottleneck: standard beam search wastes capacity on redundant isomorphic SMILES. Scaling beam width $K$ from 100 to 50,000 (Fig.~\ref{fig:isobeam_unique}) shows both methods generate similar numbers of valid SMILES, confirming robust modeling. However, scaffold discovery diverges sharply. Standard beam search saturates near $K=5{,}000$, producing mainly redundant variants. \textbf{IsoBeam} maintains near-linear growth by pruning isomorphic paths and reallocating computation to distinct regions of chemical space. At $K=50{,}000$, IsoBeam finds over \textbf{11,000 scaffolds}, roughly \textbf{2$\times$} the baseline.
\begin{figure}
\centering
\includegraphics[width=1.0\linewidth]{isobeam_compare.png}
\caption{\textbf{Inference Scalability Analysis ($K=100 \to 50,000$).}
\textbf{Left:} Both strategies yield comparable valid SMILES, confirming SIGMA's robustness.
\textbf{Right:} A critical divergence in scaffold diversity. Standard Beam Search (Grey) \textit{saturates} at $K \approx 5,000$ due to isomorphic redundancy. In contrast, \textbf{IsoBeam (Red)} mines the chemical long-tail, breaking the saturation ceiling to discover \textbf{$>2\times$} unique scaffolds at $K=50,000$.}
\label{fig:isobeam_unique}
\label{fig:isobeam_unique}
\end{figure}
\section{Conclusion}
\label{sec:conclusion}
This paper introduces SIGMA, a framework designed to resolve the topological misalignment in autoregressive chemical language models. By enforcing latent isotropy across isomorphic SMILES views, SIGMA effectively eliminates canonicalization bias without relying on computationally expensive graph encoders. Experiments on ZINC-250k show that this geometric regularization effectively bridges the gap with graph-based models, achieving state-of-the-art distributional alignment (FCD) while maintaining superior validity. Furthermore, we show that this invariance is critical for goal-directed optimization, enabling RL agents to explore the chemical space efficiently with significantly reduced mode collapse. These findings suggest that incorporating explicit geometric constraints offers a scalable and robust direction for next-generation molecular generative models.
\section*{Impact Statement}
Our work aims to advance the field of machine learning for molecular design. The potential societal impact of this research is largely positive, offering tools to accelerate drug discovery and material science. As with any generative technology in chemistry, there is a remote risk of misuse for designing harmful substances. We believe that standard safety protocols in the pharmaceutical industry and responsible release of research code are sufficient to mitigate these risks. We do not foresee any immediate negative societal consequences that require specific highlighting beyond standard dual-use considerations.
% \bibliographystyle{unsrtnat}
% \bibliographystyle{unsrt}
% \bibliography{references}
% \bibliographystyle{neurips2025}
\bibliographystyle{icml2026}
\bibliography{references}
\clearpage
\appendix
\onecolumn
\section{Visualization of Trajectory Dynamics}
\label{app:viz_dynamics}
To provide an intuitive understanding of the "manifold fragmentation" problem discussed in Section~\ref{sec:invariance} and the redundancy issue addressed in Section~\ref{sec:isobeam}, we visualize the generation trajectories in Figure~\ref{fig:constrained_viz}.
Figure~\ref{fig:constrained_viz}(b) illustrates the \textbf{unconstrained} scenario (characteristic of standard sampling or naive beam search), where the stochastic nature of sequential decoding leads to divergent paths for the same molecular graph. This divergence is the root cause of the "saturation" phenomenon observed in our beam width experiments.
In contrast, Figure~\ref{fig:constrained_viz}(c) demonstrates the \textbf{constrained} alignment principle. Although visually depicted using randomized training views, this same mechanism underpins our \textbf{IsoBeam} decoding strategy: by dynamically recognizing and merging structurally equivalent partial paths (indicated by the converging arrows), the model is forced to explore topologically distinct regions rather than syntactic permutations.
\begin{figure}[h]
\centering
\includegraphics[width=0.95\linewidth]{ConstrainedRandomSMILES.png}
\caption{\textbf{Conceptual Visualization of Trajectory Dynamics.}
\textbf{(a)} The canonical reference path.
\textbf{(b)} Unconstrained trajectories diverge into isomorphic variations (Redundancy).
\textbf{(c)} Constrained trajectories align structurally equivalent paths. This alignment mechanism is central to SIGMA's training objective and is explicitly enforced during inference via \textbf{IsoBeam} to prune redundant search branches.
\textit{*Note: START/END markers denote sequence boundaries; arrows indicate traversal direction.}}
\label{fig:constrained_viz}
\end{figure}
% ==================================================
\section{Implementation Details}
\label{app:implementation}
In this section, we provide detailed specifications regarding the model architecture, training hyperparameters, computing infrastructure, and baseline configurations to ensure the reproducibility of our results. Our code and pre-trained models will be made publicly available upon acceptance.
\subsection{Model Architecture}
We utilize the \textbf{GPT-2 Small} architecture as the backbone for the SIGMA model. The model consists of a 12-layer Transformer decoder with 12 attention heads and a hidden dimension of $d_{model} = 768$.
\begin{itemize}
\item \textbf{Embedding Layer:} We use a learned positional embedding with a maximum sequence length of 128 (covering $>99\%$ of SMILES strings in ZINC-250k).
\item \textbf{Projection Head:} For the contrastive learning objective (SIGMA alignment), we attach a 2-layer MLP projection head (Hidden dim: $768 \to 256$) to the terminal hidden state of the Transformer. This head is used only during training to compute the contrastive loss and is discarded during inference.
\item \textbf{Parameter Count:} The total number of trainable parameters is approximately 124M.
\end{itemize}
% % ==================================================
% \begin{figure*}[t]
% \centering
% \includegraphics[width=1.0\textwidth]{ConstrainedRandomSMILES.png}
% \caption{\textbf{Visualization of Trajectory Dynamics.}}
% \vspace{-1.6em}
% \label{fig:constrainedrandomsmiles}
% \end{figure*}
\subsection{Training Configuration}
All models, including SIGMA and sequence-based baselines, were trained for \textbf{50 epochs} on the ZINC-250k dataset. We use the AdamW optimizer with a linear learning rate warmup for the first 5\% of training steps, followed by a cosine decay schedule. To stabilize training, we apply gradient clipping.
We specifically note that for SIGMA, the batch size refers to the number of unique molecules; since we generate 2 views per molecule for contrastive learning, the effective number of sequences processed per step is $2 \times \text{Batch Size}$.
Detailed hyperparameters are listed in Table~\ref{tab:hyperparameters}.
\begin{table}[h]
\centering
\caption{\textbf{Hyperparameter Settings for SIGMA Training.}}
\label{tab:hyperparameters}
\begin{tabular}{l|c}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
\multicolumn{2}{c}{\textit{Optimization}} \\
\midrule
Optimizer & AdamW \\
Peak Learning Rate & $5 \times 10^{-4}$ \\
Weight Decay & $0.01$ \\
Beta parameters & $(0.9, 0.999)$ \\
Gradient Clipping (Norm) & $1.0$ \\
Batch Size & 64 \\
Training Epochs & 50 \\
Warmup Steps & 2000 \\
\midrule
\multicolumn{2}{c}{\textit{Contrastive Learning}} \\
\midrule
Temperature ($\tau$) & 0.1 \\
Augmentation Views ($N$) & 2 \\
Projection Head Dim & 256 \\
\midrule
\multicolumn{2}{c}{\textit{Architecture (GPT-2 Small)}} \\
\midrule
Layers & 12 \\
Attention Heads & 12 \\
Hidden Dimension & 768 \\
Dropout & 0.1 \\
Max Sequence Length & 128 \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Data Preprocessing and Tokenization}
\begin{itemize}
\item \textbf{Dataset:} We use the ZINC-250k dataset~\cite{irwin2005zinc}, comprising 250,000 commercially available drug-like molecules. We follow the standard random split: 220k for training, 20k for validation, and 10k for testing.
\item \textbf{Tokenization:} We employ a character-level tokenizer derived from the training set. The vocabulary consists of \textbf{69 unique tokens}, including standard atomic symbols (e.g., C, O, N), bond types, branching/ring indicators, and special control tokens (e.g., \texttt{[BOS]}, \texttt{[EOS]}, \texttt{[PAD]}).
\item \textbf{Augmentation:} For the contrastive objective, randomized SMILES are generated on-the-fly using RDKit by traversing the molecular graph with random starting atoms and traversal orders.
\end{itemize}
\subsection{Computing Infrastructure}
All experiments were conducted on a single \textbf{NVIDIA A100 Tensor Core GPU (40GB VRAM)}.
The software environment includes:
\begin{itemize}
\item \textbf{Framework:} PyTorch 2.1.0
\item \textbf{Chemistry Library:} RDKit 2023.09.5 (for SMILES handling, augmentation, and validity checks).
\item \textbf{Transformers Library:} HuggingFace Transformers 4.35.0.
\end{itemize}
Inference with IsoBeam at $K=50,000$ was optimized using batch processing on the GPU to maximize throughput.
\subsection{Inference Details (IsoBeam)}
For the asymptotic analysis in Section~\ref{sec:isobeam}, we scaled the beam width $K$ across the set $\{100, 500, 1000, 5000, 10000, 50000\}$.
\begin{itemize}
\item \textbf{Standard Beam Search:} Implemented using the default HuggingFace \texttt{generate} function with \texttt{num\_beams=K} and \texttt{num\_return\_sequences=K}.
\item \textbf{IsoBeam:} Implemented as a custom decoding loop. At each step, valid partial SMILES are cached. We utilize a hash map to store InChIKeys of visited structures to perform $O(1)$ redundancy checks.
\end{itemize}
% ==================================================
\section{Data Construction and View Generation}
\label{app:data_details}
To ensure the reproducibility of our topology-aware view construction, we detail the graph partitioning and string generation process utilized in Section~\ref{sec:view_construction}.
\subsection{Graph Partitioning Algorithm}
For a given molecule $G=(V, E)$, we identify the set of \textit{cuttable bonds} $\mathcal{E}_{cut} \subset E$. A bond $e=(u,v)$ is included in $\mathcal{E}_{cut}$ if and only if:
\begin{enumerate}
\item \textbf{Acyclicity:} $e$ does not belong to any ring system.
\item \textbf{Non-Terminal (Optional):} To avoid creating trivial fragments (e.g., single methyl groups), we typically require that both resulting components have at least \textbf{2} heavy atoms. ]
\end{enumerate}
During training, for each molecule in a batch, we uniformly sample one bond $e \in \mathcal{E}_{cut}$ to dissolve. This partitions the graph into two connected components: the prefix subgraph $G_{pre}$ and the suffix subgraph $G_{suf}$.
\subsection{Handling Attachment Points}
Crucially, simply severing the bond would lose topological information regarding the connection point. To resolve this, we introduce **dummy atoms** (anchors) at the cut site.
\begin{itemize}
\item The atom $u$ in $G_{pre}$ that was connected to $v$ is marked with a wildcard token (e.g., \texttt{*}) or a specific isotope tag to indicate the growth direction.
\item This ensures that the SMILES string for $G_{pre}$ explicitly encodes an "open" valency, distinguishing, for example, an ethyl group (\texttt{CC*}) from an ethane molecule (\texttt{CC}).
\end{itemize}
\subsection{Contrastive View Generation}
The core of SIGMA is strictly aligning syntactically distinct strings that represent the \textit{same} geometric subgraph. Once $G_{pre}$ is fixed:
\begin{itemize}
\item \textbf{View 1:} We generate a canonical SMILES string $s_{pre}^{(1)}$ of $G_{pre}$ (rooted at the cut point or canonically ordered).
\item \textbf{View 2:} We generate a randomized SMILES string $s_{pre}^{(2)}$ of the \textit{same} subgraph $G_{pre}$ by performing a random DFS traversal (using RDKit's \texttt{doRandom=True}).
\end{itemize}
This pair $(s_{pre}^{(1)}, s_{pre}^{(2)})$ serves as a positive sample for the contrastive objective, as they are guaranteed to map to the identical topological structure despite their sequence differences.
% ==================================================
\section{Dense Trajectory Alignment Objective}
\label{app:dense_alignment}
Given equivalent prefix pairs $(p_i, p_j)$, SIGMA applies a token-level contrastive objective over their valid suffix trajectories.
Let $h_t$ denote the hidden state at position $t$.
We minimize an InfoNCE-style loss:
\begin{equation}
\mathcal{L}_{\text{SIGMA}} = -\sum_{t}\log
\frac{\exp(\text{sim}(h_t^{(i)}, h_t^{(j)})/\tau)}
{\sum_{k}\exp(\text{sim}(h_t^{(i)}, h_t^{(k)})/\tau)}.
\end{equation}
This aligns representations across equivalent syntactic trajectories, restoring manifold connectivity.
% ==================================================
\section{Additional Evidence of Geometric Invariance}
\label{app:extra_experiments}
To verify that the geometric invariance observed in Section~\ref{sec:invariance} generalizes to complex pharmaceutical compounds, we conduct an additional microscopic alignment analysis on **Acetylsalicylic acid (Aspirin)**.
We inspect the attention patterns between two synonymous SMILES representations of Aspirin that differ significantly in their starting syntax:
\begin{itemize}
\item \textbf{View A:} \texttt{CC(=O)Oc1ccccc1C(=O)O} (Standard canonical start)
\item \textbf{View B:} \texttt{O=C(C)Oc1ccccc1C(=O)O} (Alternative start with inverted acetyl group)
\end{itemize}
As visualized in Figure~\ref{fig:heat2}, SIGMA exhibits strong **off-diagonal semantic alignment**. specifically, the representations of the acetyl group tokens—\texttt{CC(=O)} in View A and \texttt{O=C(C)} in View B—show high cosine similarity (indicated by the bright block in the top-left corner), despite being composed of different character sequences.
This confirms that SIGMA has successfully learned to abstract the underlying functional group (the acetyl ester) independent of its specific linearization order. The continued high similarity along the diagonal for the shared benzene ring substructure (\texttt{Oc1ccccc1...}) further validates that the model maintains trajectory consistency once the syntax converges.
\begin{figure}[h]
\centering
% 建议裁剪一下图片白边，或者用 width=0.8\linewidth 让它别太大
\includegraphics[width=0.9\linewidth]{heat2.png}
\caption{\textbf{Geometric Alignment on Acetylsalicylic Acid (Aspirin).}
We visualize the token-level cosine similarity between two isomorphic views of Aspirin: \texttt{CC(=O)...} vs. \texttt{O=C(C)...}.
The high-similarity region (red block) aligning the distinct prefixes confirms that SIGMA correctly identifies the topological equivalence of the acetyl group despite the variation in linearization direction.}
\label{fig:heat2}
\end{figure}
\section{Metric Definitions}
\label{app:metrics}
To ensure reproducibility, we provide formal definitions for all evaluation metrics used in our experiments.
\subsection{Generative Quality Metrics}
Let $\mathcal{G}$ be the set of generated molecules and $\mathcal{D}_{train}$ be the training dataset.
\begin{itemize}
\item \textbf{Validity:} The fraction of generated SMILES strings that can be parsed into valid molecular graphs by RDKit.
\item \textbf{Uniqueness:} The fraction of valid molecules that are unique within the generated set $\mathcal{G}$.
\[ \text{Uniqueness} = \frac{|\text{unique}(\mathcal{G}_{valid})|}{|\mathcal{G}_{valid}|} \]
\item \textbf{Novelty:} The fraction of valid, unique generated molecules that do not appear in the training set.
\[ \text{Novelty} = \frac{|\{m \in \mathcal{G}_{valid} \mid m \notin \mathcal{D}_{train}\}|}{|\mathcal{G}_{valid}|} \]
\item \textbf{Internal Diversity (IntDiv):} Measures the chemical diversity within the generated set using the average pairwise Tanimoto distance ($T_d$) of Morgan fingerprints ($M$).
\[ \text{IntDiv}(\mathcal{G}) = 1 - \frac{1}{|\mathcal{G}|^2} \sum_{m_i, m_j \in \mathcal{G}} \text{Tanimoto}(M(m_i), M(m_j)) \]
\end{itemize}
\subsection{Distribution Learning Metrics}
\begin{itemize}
\item \textbf{Fréchet ChemNet Distance (FCD):} Measures the distance between the distribution of generated molecules and the test set in the activation space of ChemNet (a pre-trained GRU). Lower FCD indicates that the generated molecules capture the chemical and biological properties of the real data distribution more accurately.
\item \textbf{Scaffold Diversity (\#Scaf):} The number of unique Bemis-Murcko scaffolds extracted from the generated set. A scaffold is obtained by removing all side chain atoms and keeping only the ring systems and linkers. Higher \#Scaf indicates better exploration of the structural topology.
\end{itemize}
\subsection{Geometric Invariance Metrics}
\begin{itemize}
\item \textbf{Trajectory Invariance Score (TIS):} A custom metric to quantify the stability of the latent representation against isomorphic permutations.
For a given molecule $M$, let $s_1$ and $s_2$ be two distinct SMILES prefixes representing the same partial graph $G_{sub}$. Let $\mathbf{h}(s)$ denote the normalized terminal hidden state of the model given prefix $s$. TIS is defined as the expected cosine distance over the test set:
\[ \text{TIS} = \mathbb{E}_{M \sim \mathcal{D}_{test}} \left[ 1 - \text{sim}(\mathbf{h}(s_1), \mathbf{h}(s_2)) \right] \]
Lower TIS values (closer to 0) indicate stronger geometric invariance, where the model maps syntactically different but topologically equivalent sequences to nearby points in the latent space.
\end{itemize}
\subsection{Optimization Metrics (PMO)}
For the Reinforcement Learning tasks, we report:
\begin{itemize}
\item \textbf{Top-1 / Top-10 / Top-100:} The average reward score of the top $K$ molecules found during the optimization process.
\item \textbf{Success Rate:} The percentage of generated molecules that satisfy all hard constraints (if applicable) and exceed a predefined reward threshold.
\end{itemize}
% \begin{figure*}[t]
% \centering
% \includegraphics[width=1.\textwidth]{RadarMetrics.png}
% \caption{Retrieval performance of ZINC-GPT and DrugGPT variants across four SMILES perturbation types in the AMORES benchmark.Comparison of ZINC-GPT and TACL-augmented variants across four structural consistency tasks (Cano, Cycle, Kekul, Hydro), each with three metrics (Top-1, Top-5, and MRR). Left two: Top-1 classification accuracy on CHEBI-20 using ZINC-GPT and DrugGPT, with and without TACL. Right two: Mean Absolute Error (MAE) on QM9 regression using ZINC-GPT and DrugGPT. In all settings, token-level contrastive supervision leads to improved performance.}
% \label{fig:radar_metrics}
% \end{figure*}
\end{document}