LASA: Language-and-Source-Anchored Alignment for Domain Generalized Semantic Segmentation

To translate textual semantics into fine-grained visual guidance, we propose Language-Anchored Distribution Alignment (LADA). The core motivation is to effectively use the joint semantics (category and domain) embedded in textual embeddings to organize the feature space into a structured intra-class inter-domain topology. Let $\mathcal{E}\in\mathbb{R}^{K\times C\times D_{t}}$ denote the domain-specific textual embeddings, where $K$ and $C$ represent the number of domains and categories, and each $D_{t}$-dimensional vector encodes both categorical identity and domain context. For each branch $m\in\{1,2\}$, we first randomly sample a domain index $k^{(m)}\in\{1,\ldots,K\}$, which is kept consistent across all hierarchical levels $i\in\{2,3,4,5\}$ within the same branch to ensure stylistic coherence across scales. A crucial component of LADA is the cross-modal projector $\phi(\cdot):\mathbb{R}^{D_{t}}\rightarrow\mathbb{R}^{D}$, a learnable mapping that transforms textual embeddings into the visual latent space. Specifically, for each hierarchical feature $F_{i}\in\mathbb{R}^{\frac{H}{2^{i}}\times\frac{W}{2^{i}}\times D}$, we construct a language-anchored style map $M_{i}^{(m)}\in\mathbb{R}^{\frac{H}{2^{i}}\times\frac{W}{2^{i}}\times D}$ by projecting the sampled embedding $E_{k^{(m)},c}$ (e.g., the embedding for ”a car in rain” when $c$ is the car category and $k^{(m)}$ is the rain domain), where $c=Y(h,w)$:

To transfer the textual style while maintaining semantic boundaries, LADA performs a channel-wise distribution alignment via a sort-matching operation:

where $d\in\{1,\ldots,D\}$ is the channel index. Specifically, $\operatorname{Sort}(\cdot)$ extracts the target value set by sorting the elements of $M_{i}^{(d)}$ in ascending order to define the channel’s intensity distribution. Concurrently, $\operatorname{Rank}(\cdot)$ determines the spatial ordinal ranking of pixels in $F_{i}^{(d)}$ to preserve the spatial topology. The composition operator $\circ$ then assigns the $k$-th smallest value from the sorted target distribution to the exact spatial coordinate of the $k$-th smallest pixel in the source feature. This mechanism ensures that $F_{\text{aligned},i}$ inherits the statistical characteristics of the category-domain textual priors while remaining strictly anchored to the structural information provided by the source content.

To ensure end-to-end differentiability, we decouple the gradient paths: the gradient flows naturally to the mapping layer $\phi$ through the continuous values of $F_{\text{aligned},i}$, while for the other trainable modules, we adopt a straight-through estimator (STE) to bypass the non-differentiable sorting operations:

where $\text{sg}(\cdot)$ denotes the stop-gradient operation.

To preserve the structural stability of the latent space and prevent the mapping $\phi$ from producing distorted or over-fitted representations, we introduce Source-Anchored Manifold Stabilization (SAMS). Since the CLIP backbone is frozen, its original features provide definitive structural anchors. SAMS capitalizes on this by using source features as immutable reference points, providing a critical manifold regularization for the projector $\phi$. This compels the model to generate stylized maps that are not only texturally consistent with the text prompts but also geometrically compatible with the intrinsic visual manifold of CLIP. Specifically, within the TSGST module, SAMS stochastically selects between the original source feature $F_{i}$ and the stylized feature $F_{\text{LADA},i}$ using a binary selector $s\in\{0,1\}$:

where $s$ follows a Bernoulli distribution with $P(s=0)=0.5$. This ensures that the output features remain anchored to the robust semantic structure of CLIP’s original space for half the training instances, preventing representation collapse during style transfer.

To enforce categorical consistency, we adopt a dual-path optimization scheme generating two parallel multi-level feature sets, $\hat{\mathcal{F}}^{(1)}$ and $\hat{\mathcal{F}}^{(2)}$, from the shared input $\mathcal{F}_{s}$. By independently sampling the stochastic selectors $s^{(1)}$ and $s^{(2)}$ within their respective TSGST modules, the framework reconciles diverse alignment scenarios, including source-to-augmented and augmented-to-augmented pairs. The cross-branch discrepancy is regularized via a unidirectional consistency loss:

where $P^{(m)}$ is the prediction from the $m$-th feature set and $\text{sg}(\cdot)$ denotes the stop-gradient operation. This strategy effectively minimizes the distance between diverse domain-specific sub-clusters of the same category, pulling stochastic stylistic fluctuations toward a unified categorical manifold to ensure stable decision responses regardless of environmental shifts.

The joint action of LADA, SAMS, and $\mathcal{L}_{\text{con}}$ regularizes the latent space into an orderly intra-class inter-domain topology. In this framework, the source feature space provides the structural anchors, while the textual embeddings—containing both domain and category information—act as the shaping force. By explicitly tying visual variations to these textual anchors and minimizing the divergence between branches, TSGST ensures that even domain-dominant features remain clustered within their respective categorical boundaries. This structural refinement ensures that domain-specific variations are logically arranged by the textual guidance rather than being mixed chaotically, effectively alleviating the inter-class confusion prevalent in prior alignment techniques. Consequently, the model maintains distinct inter-class boundaries while accommodating vast intra-class domain shifts, leading to superior generalization in unseen domains.

To inform the queries of their stylistic context, we first derive a robust domain signature $S$. Let $F_{\text{mask}}\in\mathbb{R}^{H\times W\times D}$ denote the mask feature produced by the pixel decoder. As a dense visual representation, $F_{\text{mask}}$ aggregates hierarchical contextual information from the frozen backbone while preserving the spatial layout essential for mask prediction.

Inspired by previous work in style encoding and feature aggregation (Huang and Belongie, 2017; Woo et al., 2018), we characterize the domain signature by extracting first-order, second-order, and salient statistics directly from $F_{\text{mask}}$. Specifically, we compute the channel-wise mean $\mu(\cdot)$, standard deviation $\sigma(\cdot)$, and maximum values $\max(\cdot)$ over the spatial dimensions. These statistics are concatenated and projected through a multi-layer perceptron (MLP) to yield the domain signature $S\in\mathbb{R}^{D}$:

In our framework, $S$ provides the necessary context to identify the specific domain region inhabited by the current input on the latent map, providing the baseline for subsequent query modulation.

In the decoder pipeline, queries $Q$ enter DAQA after the initial cross-attention layers, having already pre-aggregated certain spatial and visual contexts. To determine the categorical identity of these queries and facilitate precise localization, we extract categorical semantic guidance $E_{\text{sem}}$ by allowing the queries $Q$ to interact with fixed textual embeddings $\mathcal{E}_{cls}\in\mathbb{R}^{C\times D}$ through a cross-attention mechanism:

By distilling domain-agnostic knowledge from the stable textual space, $E_{\text{sem}}$ filters out environmental noise and identifies the categorical center, serving as a reliable semantic reference for the subsequent domain-aware shift.

The core of DAQA lies in jointly conditioning the queries on both the categorical guidance $E_{\text{sem}}$ and the domain signature $S$ to perform a fine-grained refinement. We concatenate $E_{\text{sem}}$ and $S$ to predict instance-specific modulation parameters via a specialized MLP, denoted as $\text{MLP}_{\text{aff}}$:

where $\alpha$ and $\beta$ denote the scaling and shifting factors, respectively. These parameters reflect the combined influence of categorical identity and stylistic context. The final domain-refined queries $Q^{\prime}$ are then generated by applying this modulation as a residual correction to the original queries:

Here, $\odot$ denotes the Hadamard product, and $\text{LN}(\cdot)$ represents Layer Normalization. By retaining the original queries $Q$ through a residual connection, the visual context already aggregated in the queries is preserved. In contrast to previous methods that rely on rigid normalization to force domain-invariance—which often overlooks local cluster variances—our fine-grained recalibration adaptively steers each query toward its corresponding domain-specific sub-cluster. This operation aligns the query’s latent position with the actual feature distribution of the current domain, effectively preserving the discriminative semantic details.

In summary, DAQA solidifies the orderly topology of the latent space by transitioning from static prior matching to dynamic localization. By conditioning queries on categorical semantics and domain signature, DAQA ensures that each query and feature ”meet” at the most accurate point in the manifold—specifically, at the intra-categorical sub-region corresponding to the current domain.

DADO implements a strategic decoupling of the representation space via branch-specific normalization, ensuring each prediction head operates on a distribution optimized for its respective objective:

1) Categorical Branch (Alignment for Shared Classifier): To ensure the shared linear classifier reliably distinguishes categories across varying environments, DADO employs Adaptive Layer Normalization (AdaLN) to project the domain-aware queries $Q^{\prime}$ into a unified canonical distribution:

The scaling factor $\gamma$ and shifting factor $\delta$ are dynamically generated from the domain signature S via a lightweight MLP. This rectification compensates for systematic distribution offsets, effectively ”pulling” adjacent domain-specific sub-clusters toward a stabilized categorical center to mitigate the risk of inter-class confusion.

2) Mask Branch (Structural Preservation): In contrast, the mask prediction head requires queries to retain their active domain-aware characteristics to accurately interact with the shifted visual features in $F_{\text{mask}}$. To preserve the localized coordinates established by DAQA, the mask embeddings $Q_{\text{mask}}^{\prime\prime}$ are derived using standard Layer Normalization:

The final prediction $M$ is generated via the inner product: $M=Q_{\text{mask}}^{\prime\prime}\times F_{\text{mask}}$. This ensures that the spatial segmentation remains strictly anchored to the domain-specific manifestation of objects.

The design of DADO acknowledges that domain-aware navigation and shared-classifier compatibility are divergent functional requirements. If a unified normalization were applied to both branches, the benefits of active domain perception gained in DAQA—essential for masking—would be neutralized by the global alignment required for stable categorization. By decoupling these operations, DADO allows the queries to function simultaneously as domain-sensitive probes for precise localization and domain-invariant representations for robust categorization. This two-stage refinement solidifies the orderly topology of the manifold, ensuring superior generalization in diverse and unseen domains.

The evaluation incorporates six semantic segmentation datasets. Synthetic domains include GTAV (G) (Richter et al., 2016) and SYNTHIA (S) (Ros et al., 2016). Real-world domains consist of Cityscapes (C) (Cordts et al., 2016), BDD100K (B) (Yu et al., 2020), Mapillary (M) (Neuhold et al., 2017), and the ACDC (A) (Sakaridis et al., 2021) dataset which covers four adverse conditions: Fog (AF), Night (AN), Rain (AR), and Snow (AS). Performance is measured using mIoU over 19 categories, except for SYNTHIA, which uses 16 categories (Tsai et al., 2018).

Three zero-shot generalization protocols are employed (Lee et al., 2022; Niu et al., 2025): (1) Single-Source: Train on G, test on {C, B, M, S}. (2) Real-to-Mixed: Train on C, test on {B, M, G, S}. (3) Multi-Source: Train on G+S, test on {C, B, M}. Robustness is further validated on all ACDC subsets across these settings.

The framework uses a ResNet-50 backbone initialized with CLIP pre-trained weights (Radford et al., 2021; Li et al., 2023; Fahes et al., 2024). Notably, both the CLIP vision backbone and the text encoder are kept frozen throughout all training stages to provide stable semantic priors. The segmentation architecture includes a 6-layer multi-scale deformable attention pixel decoder and a 10-layer mask decoder with 100 object queries (Ding et al., 2023; Niu et al., 2025). To perform fine-grained query recalibration and global distribution alignment, we deploy the DAQA and DADO modules in layers $[2,3,4,5]$ of the mask decoder.

Optimization is performed using AdamW (Loshchilov and Hutter, 2017) for 225,000 iterations with a batch size of 4 and a learning rate of $1\times 10^{-4}$ (PolyLR decay, power 0.9 (Chen et al., 2017)). A decoupled two-phase training scheme is adopted: (1) In the foundation stage (initial 200,000 iterations), the adaptation modules (DAQA and DADO) are frozen, while the pixel decoder, mask decoder, and the text-to-image feature space projector are optimized. (2) In the adaptation stage (remaining 25,000 iterations), we freeze all previously trained segmentation components and exclusively update the parameters of DAQA and DADO. This staged optimization ensures that the adaptation modules perform alignment based on a finalized feature manifold. During training, images are randomly cropped to $768\times 768$. The total loss comprises weighted cross-entropy (3.0), dice (5.0), mask (5.0), and consistency (10.0) components. All experiments are conducted on 4 $\times$ NVIDIA GeForce RTX 3090 GPUs.

As summarized in Table 1, LASA consistently achieves superior performance in the challenging GTAV $\rightarrow$ Real scenarios. On the GTAV $\rightarrow$ Cityscapes task, LASA reaches an mIoU of 55.93%, outperforming the recent SOTA method SCSD by a significant margin of $+$4.21%. Across all target domains, LASA achieves an average mIoU of 52.48%, which is $+$3.37% higher than SCSD. These substantial gains demonstrate that by stabilizing the feature manifold with structural anchors and fine-grained VLM guidance, LASA effectively addresses manifold distortion. This allows the model to preserve the discriminative semantic nuances that are typically suppressed by rigid designs or coarse-grained randomization.

When trained on the real-world Cityscapes dataset, LASA continues to lead the competition with an average mIoU of 52.90%. It surpasses all ResNet-50-based methods, confirming its ability to adapt to mixed environmental variations while maintaining high categorical discriminability.

Table 2 reports the GTAV + SYNTHIA $\rightarrow$ Real results. LASA achieves 55.11% mIoU, significantly surpassing SCSD (51.42%). This performance confirms LASA’s efficacy in synergistically aggregating multi-source knowledge without distribution interference.

To further stress-test the model’s adaptability in extreme environments, we evaluate LASA on the ACDC dataset. As summarized in Table 3, LASA sets a new state-of-the-art for overall robustness with an average mIoU of 42.14%, representing a significant $+$6.48% increase over the previous methods. The superiority of our framework is most pronounced in the most challenging scenarios: in low-illumination Night (AN) environments where visual cues are severely degraded, LASA improves the mIoU from 15.06% (SCSD) to 23.70% (a remarkable $+$8.64% increase). In visibility-impaired Snow (AS) conditions, it reaches 50.28% mIoU, outperforming SCSD (41.37%) by +8.91%.

To quantitatively validate the structural integrity of the latent space, we evaluate the feature manifold using high-dimensional mask features extracted from the decoder, with four literature-backed metrics summarized in Table 5. The Silhouette Score (Rousseeuw, 1987) ($[-1,1]$, $\uparrow$) measures intra-class compactness versus inter-class separation; the Davies-Bouldin Index (DBI) (Davies and Bouldin, 1979) ($[0,+\infty)$, $\downarrow$) quantifies average similarity to the most confusable neighbor. LASA achieves a Silhouette Score of 0.66 ($+$0.23) and reduces DBI from 0.91 to 0.45, confirming tighter clusters and sharper boundaries. The Intra-Class Proxy $\mathcal{A}$-distance (IC-PAD) (Ben-David et al., 2010) ($[-1,1]$, $\uparrow$) adapts domain divergence theory to an intra-class setting by applying the Proxy $\mathcal{A}$-distance within each semantic class independently and reporting the mean score across all classes; a higher score indicates well-organized intra-class domain structure rather than an undifferentiated collapse. The prior state-of-the-art yields IC-PAD of $-$0.50, confirming that coarse-grained randomization destroys intra-class domain structure; LASA raises this to $+$0.39 ($+$0.89). Fisher’s $J$ (Fisher, 1936; Wen et al., 2016) ($[0,+\infty)$, $\uparrow$) jointly captures within-class scatter and between-class separation, improving from 8.01 to 50.88. These findings are corroborated by the t-SNE visualizations in Figure 2, where LASA exhibits well-separated class clusters with visible domain sub-structure, in contrast to the inter-class overlap and domain-collapsed distributions of prior methods.

As the foundation of our framework, the TSGST module yields the most substantial performance gain. The single-path configuration—incorporating Language-Anchored Distribution Alignment (LADA) and Source-Anchored Manifold Stabilization (SAMS)—improves the average mIoU from 44.18% to 50.57% (+6.39%). This significant improvement demonstrates that providing fine-grained guidance via textual priors while maintaining structural anchors effectively prevents representation collapse and addresses manifold distortion.

Building upon the stabilization provided by the TSGST module, the dual-path mechanism further enforces semantic consistency via $\mathcal{L}_{\text{con}}$. This configuration increases the average mIoU to 51.98% and notably lifts performance on the adverse ACDC dataset from 38.30% to 41.66%. These results confirm that probabilistic consistency regularization effectively filters stylistic noise, forcing the model to focus on domain-invariant semantic cues rather than transient environmental characteristics.

Final performance refinements are provided by the adaptation modules. The DAQA module recalibrates object queries using domain-aware signatures to restore suppressed discriminative details, reaching 52.11% mIoU. Finally, the complete LASA framework, incorporating the DADO global distribution rectifier, achieves a peak performance of 52.48%. Their synergy is most pronounced in extreme conditions, pushing the ACDC average to 42.14%, as they ensure the resulting query distributions remain aligned with the shared classifier for consistent categorical responses.