Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learning

During the pre-training stage, CLIP (Radford et al., 2021) learns aligned visual and textual representations by maximizing the cosine similarity between image and text embeddings through a contrastive objective on large-scale image–text pairs. Unlike conventional classifiers trained on closed-set labels, CLIP exploits open-set semantic supervision from natural language, enabling it to capture a broad spectrum of visual concepts. As a result, CLIP supports zero-shot classification at test time without any additional training.

Specifically, given a test image $x$ belonging to one of $C$ candidate categories, let $\bm{v}(x)$ denote the visual embedding produced by the image encoder. For each class $c$, a text embedding $\bm{t}(c)$ is obtained by encoding a prompt such as “a photo of {class c}”, where the placeholder is replaced with the corresponding category name. The zero-shot predictive probability for class $c$ is then computed as

where $\tau$ is CLIP’s learned temperature parameter and $\cos(\cdot,\cdot)$ denotes cosine similarity.

Recent studies have shown that CLIP representations exhibit pronounced class-conditional clustering behavior in the embedding space, and can be well approximated by multivariate Gaussian distributions (Zanella et al., 2024; Wang et al., ; Zhu et al., 2024). Motivated by these findings, we shift the perspective from evaluating a sample in isolation to assessing it within a collective statistical context. This motivates us to adopt a generative view inspired by Gaussian Discriminant Analysis (GDA), explicitly modeling the feature distribution to enable distribution-aware inference.

Specifically, we assume that the embedding distribution of each semantic class follows a multivariate Gaussian. Given an image $x\in\mathbb{R}^{d}$, the class-conditional likelihood for class $c$ is defined as:

where $\bm{\mu}_{c}$ and $\bm{\Sigma}_{c}$ denote the mean vector and covariance matrix of class $c$, respectively. Consequently, the marginal distribution of $x$ is formulated as a Gaussian mixture over all classes:

where $P(c)$ denotes the class prior. Under the uniform prior assumption $P(c)=1/C$, the posterior distribution is fully determined by the relative geometry between the embedding and the class-conditional Gaussian component:

Consequently, the posterior can be reformulated into a normalized exponential (softmax) form

where the discriminant function is

Intuitively, $f_{c}(x)$ quantifies the compatibility between the embedding $x$ and class $c$ by balancing two geometric factors: the quadratic term represents the Mahalanobis distance to the class mean, while the log-determinant term penalizes class-wise dispersion. This formulation ensures that the posterior is governed not just by proximity to a prototype, but by the intrinsic shape of the class distribution. It is worth noting that, in our actual implementation, we adopt a shared covariance matrix $\bm{\Sigma}\in\mathbb{R}^{d\times d}$ across all semantic classes, instead of the class-specific covariance matrices $\bm{\Sigma}_{c}$ described above. This modification simplifies the model while maintaining the core distribution-aware inference capability. For explanations and reasoning regarding this part, see Appendix A.1 for details.

Building on the Gaussian modeling in Sec. 3.2, we extract two complementary distributional evidences to characterize uncertainty: distributional typicality and semantic ambiguity.

We first quantify the support of a sample within the global embedding space using the negative log-likelihood (NLL) of the mixture density:

This metric assesses how well a sample conforms to the learned target manifold. Low NLL values indicate distributionally typical samples, whereas high values flag outliers in low-density regions, signaling potential out-of-distribution inputs. Crucially, $E_{\text{den}}$ captures global geometric support independent of specific class assignments.

While density assesses representativeness, it fails to capture separability; a sample can be distributionally typical yet ambiguous if it resides in the intersection of multiple Gaussian components. To isolate this, we compute the entropy of the classification posterior:

Elevated entropy indicates that the sample lies near decision boundaries where class prototypes overlap, reflecting high aleatoric uncertainty. Conversely, minimized entropy corresponds to samples deeply embedded within a single class-specific cluster. Finally, we standardize these metrics to ensure scale consistency and aggregate them into a compact evidence vector $\mathbf{e}(x)=[\tilde{E}_{\text{den}}(x),\tilde{E}_{\text{amb}}(x)]^{\top}$.

After extracting evidence from the distribution, we combine the evidence with the representation vectors. Specifically, the visual embedding $\bm{v}(x)$ is first mapped to a hidden representation $\mathbf{h}(x)\in\mathbb{R}^{d}$ via a linear projection. Simultaneously, a gating network processes the evidence vector $\mathbf{e}(x)$ to generate sample-specific affine transformation parameters—namely, a scaling factor $\bm{\gamma}$ and a shift term $\bm{\beta}$:

where $\mathbf{W}_{\gamma},\mathbf{W}_{\beta}\in\mathbb{R}^{d\times 2}$ are learnable projection matrices and $\sigma$ denotes the sigmoid function. The visual features $\mathbf{h}(x)$ are then modulated as $\bm{\gamma}\odot\mathbf{h}(x)+\bm{\beta}$. The gating branch introduces only a negligible number of parameters, and the increased expressiveness comes primarily from the structural prior: the uncertainty quantification is forced to be a function of visual features conditioned on their distributional validity. This mechanism enables the distributional evidence to perform channel-wise recalibration of visual features. For instance, evidence of distributional anomaly (e.g., a high $E_{\text{den}}$ value) can suppress activations that lack support from the target distribution, thereby guiding the predictor to rely on more reliable signals. Then, a two-layer MLP predicts the scalar uncertainty score from the modulated representation:

where $\phi(\cdot)$ is the ReLU activation function, and $\sigma(\cdot)$ ensures the output lies in $(0,1)$.

We optimize the predicted score $\hat{y}$ using a standard binary cross-entropy objective. Departing from conventional hard label, which assigns 0 to correct predictions and 1 to errors (Lafon et al., 2025), we construct a continuous soft label $\tilde{y}\in[0,1]$ theoretically derived from the Gaussian discriminant distribution. This soft label integrates both the semantic severity of misclassifications and the predictive fragility of correct predictions. Specifically, for misclassified samples (hard label: 1), the risk is scaled by semantic severity; this design penalizes overconfident and semantically implausible errors more heavily while attenuating the risk for confusions between related concepts. Conversely, for correctly classified samples (hard label 0), we characterize fragility using the probability margin of top-2 predictions (detailed derivation in Appendix A.3). Given this soft label $\tilde{y}$ and the predicted score $\hat{y}$, we achieve calibrated uncertainty estimation by minimizing the following loss function:

where $s$ denotes the batch size.

Algorithm 1 outlines the training procedure of our dynamic uncertainty estimation framework from a batch perspective. In Step 1, the model updates the Gaussian Mixture Model via a momentum-based Exponential Moving Average (EMA) (Klinker, 2011). This procedure is mathematically equivalent to a streaming Expectation-Maximization (EM) (Greff et al., 2017; Moon, 1996) process (see Appendix A.2). By continuously synchronizing statistical estimates with the underlying feature manifold, this mechanism fosters robust evidence extraction from partial distributional observations, thereby enabling the framework to resiliently accommodate diverse distributional shifts. To unify supervised training and unsupervised adaptation, we employ the discriminative mechanism from Eq. (4) to define a gate responsibility $r_{c}(x)$:

where $\mathbb{I}\{\cdot\}$ is the indicator function and $c_{t}$ denotes the target class, and $\lambda\geq 0$ controls the strength of supervision. These responsibilities serve as dynamic weights for updating the GMM statistics: a sample $x$ of batch $\mathcal{S}$ contributes to the sufficient statistics of class $c$ proportional to $r_{c}(x)$. During training ($\lambda>0$), this formulation softly biases the responsibilities toward the ground-truth label, driving the GMM parameters $\{\bm{\mu}_{c},\bm{\Sigma}_{c}\}$ to converge to accurate class-conditional densities.

Subsequently, Step 2 extracts and standardizes distributional evidence based on density and semantic ambiguity. In Step 3, a gating network conditions the visual features on this evidence, followed by a two-layer MLP predictor that outputs calibrated uncertainty scores aligned with the current distributional perspective.

During inference ($\lambda=0$), the supervision term vanishes and $r_{c}(x)$ reduces to the zero-shot posterior, enabling the model to self-update its parameters based on its own predictions in a fully unsupervised manner. This design naturally endows the model with test-time adaptation capabilities. By exclusively updating the GMM statistics while keeping the prediction head frozen, the model efficiently calibrates distributional evidence to the target distribution, thereby avoiding computationally expensive gradient-based parameter fine-tuning.

We focus on uncertainty quantification for VLMs in zero-shot classification tasks, and consider five experimental scenarios: standard benchmark settings, distribution shift, comparison of test-time adaptation scenarios, multi-label classification, and OOD scenarios. In addition, we provide ablation studies and qualitative visualizations to further demonstrate the efficacy of the proposed uncertainty quantification approach.

Datasets. We employ two types of benchmark datasets: (i) image–label datasets and (ii) image–caption datasets.

Baselines. To comprehensively evaluate our approach, we compare it against state-of-the-art methods discussed in the Related Work section, including: (i) output-based post-hoc methods (MCM, TS+MCM, Entropy, Doctor, Rel-U), (ii) probabilistic modeling approaches (ProbVLM, BayesVLM), and (iii) training a lightweight module (ViLU).

Implementation Details. In the main experiments, we adopt CLIP ViT-B/32 as the default backbone network. We evaluate two variants of the framework: the Base version, which accesses the entire test set offline to estimate distribution statistics, and the Adaptation version, which corresponds to an online deployment scenario where samples arrive sequentially and the statistics are updated incrementally on a per-sample basis. We employ five-fold cross-validation and report the mean value along with the standard deviation. In our method, the covariance is shared, which avoids the computational and storage overhead on high-dimensional datasets such as ImageNet-1K.

Metrics. We evaluate our model using two widely adopted metrics in uncertainty quantification and failure detection: (i) the False Positive Rate at 95% True Positive Rate (FPR95), and (ii) the Area Under the Receiver Operating Characteristic Curve (AUC). These metrics measure the model’s ability to identify high-uncertainty (or potentially erroneous) predictions.

Q1: Does uncertainty depend on the model performance itself?

To some extent, this conclusion applies to all uncertainty quantification methods. A reliable backbone is a prerequisite for effective confidence estimation: no downstream uncertainty quantification module can correct the poor discriminative features generated by a weak backbone. However, as shown in Figure 3, our method exhibits excellent generalization performance across five vision backbones. Our two proposed methods achieve top results on nearly all backbones. Notably, Ours (Adaptation) achieves the highest AUC and lowest FPR95 across all five architectures, fully demonstrating its good robustness to different backbone capacities.

Q2: Is the Gaussian assumption for high-dimensional CLIP embeddings reasonable?

We fully agree that no strict multivariate Gaussian exists in high dimensions, and we do not claim CLIP embeddings strictly follow Gaussianity—we only use it as a reasonable approximation for class-conditional embeddings, supported by (Zanella et al., 2024). These works verify that CLIP embeddings present clear class-clustering on semantic manifolds, where Gaussian serves as an optimal and concise approximation. Our GMM only relies on first/second-order statistics (mean, covariance) to model distribution geometry, free from strict Gaussian constraints. Moreover, CLIP embeddings reside on low-dimensional semantic manifolds with effective dimensionality far lower than the apparent dimension, thus avoiding modeling failure caused by the curse of dimensionality.

We examine the impact of each core component in Table 6. Starting from a visual-only Baseline (a), we observe that: (i) Impact of Distributional Context: Incorporating GMM-derived evidence (b) provides a substantial performance leap. This confirms that density and ambiguity signals offer critical context for identifying failure modes that raw visual features alone cannot capture. (ii) Conditioned Feature Fusion: Moving from simple concatenation to our adaptive modulation (c) significantly enhances results. This demonstrates the superiority of conditioning the visual embeddings on distributional evidence, which enables the model to actively recalibrate visual activations based on their distributional validity rather than merely treating evidence as auxiliary input. (iii) Benefit of Dynamic Adaptation: The inclusion of Soft Risk Labels (d) and dynamic GMM Updates (e) yields the best performance. These components allow the model to learn fine-grained risk nuances and maintain alignment with the target distribution’s evolving manifold, which is essential for robust uncertainty quantification.

Figure 4 highlights the vulnerability of ViLU under distribution shift: its dependence on static semantic embeddings leads to a misalignment between the frozen text anchors and the shifted visual features. Consequently, it fails to distinguish between true semantic novelty and mere stylistic variations (e.g., in ImageNet-R), causing miscalibrated risk scores. Conversely, our framework shifts the paradigm from static matching to dynamic adaptation. By modeling the visual manifold via a dynamic GMM and explicitly accounting for semantic ambiguity, we enable the uncertainty estimation to actively synchronize with the target distribution, ensuring robust calibration even in unsupervised environments.

As illustrated in Figure 5, our framework demonstrates remarkable efficiency in both training and inference phases. First, regarding training efficiency (Figure 5a), our method rapidly converges, surpassing the strong zero-shot baseline (MCM) using as little as 20% of the training data. We attribute this to our statistical GMM modeling: unlike discriminative methods (e.g., ViLU) that require abundant samples to delineate complex boundaries, estimating sufficient statistics (means and covariances) is inherently sample-efficient. Furthermore, the semantic-aware soft target accelerates convergence by providing dense, continuous gradient signals from each sample. Second, regarding inference stability (Figure 5b), our method maintains robust performance across varying batch sizes. Notably, even at a batch size of 1 (online streaming setting), the model achieves competitive FPR95, which further improves and saturates around a batch size of 32. This confirms that our EMA update mechanism is stable and does not strictly require large buffers, making it highly suitable for real-time, resource-constrained deployment.

Additional qualitative and parameter analyses are provided in Appendix A.4. The case study under distribution shift shows that, although CLIP can remain robust or even improve its recognition performance on shifted data, existing uncertainty estimation methods such as ViLU may still suffer substantial degradation due to their reliance on static source-domain representations. In contrast, our distribution-aware framework consistently yields more reliable uncertainty estimates by adapting to the evolving representation structure of the target distribution. We further analyze the sensitivity to the number of GMM components $K$ and observe that performance improves as $K$ increases and becomes stable when $K$ approaches the number of classes $C$. Accordingly, we set $K=C$ as the default configuration in all experiments.