Unleashing the Potential of Vision-Language Models for Generalizable AI-Generated Image Detection

Prior work (Zhou et al. 2026, 2025b; Lee et al. 2026) has shown that modern visual encoders already contain the image-provenance information needed for AIGI detection. Unlike content semantics, provenance information describes how an image was produced rather than what it depicts, making it more difficult to learn from visual data alone. As shown in Figure 1(a), DINOv3 (Siméoni et al. 2025) is pretrained on images alone, whereas PE (Bolya et al. 2025) is pretrained on image-text pairs whose text may explicitly describe image provenance. In Figure 1(b), PE’s frozen features exhibit a clearer local forensic structure than DINOv3’s across diverse test sources. Appendix E reports the corresponding distributions for each In-the-Wild dataset separately.

Language alignment also enables a direct textual interpretation of the forensic information encoded in PE’s image features. Simplicity Prevails (Zhou et al. 2026) demonstrates this by comparing image features with a pool of text concepts, finding that generated images align with forgery-related concepts in PE’s shared embedding space. Because DINOv3 lacks a text encoder, it can not provide the same text-based interpretation of its forensic information.

Despite its clearer local forensic structure, PE still underperforms DINOv3 under conventional linear probing. As shown in Figure 1(c), when both linear heads are trained on GenImage SD1.4 (Zhu et al. 2023), PE-Linear achieves an average accuracy of $89.9$ on In-the-Wild, compared with $94.0$ for DINOv3-Linear. This discrepancy indicates that conventional linear probing fails to fully exploit the forensic structure already present in PE’s features.

Conventional linear-probing baselines (Ojha, Li, and Lee 2023; Zhou et al. 2026) typically freeze the image encoder, and train a randomly initialized linear head on a single source. When this setup is applied to a vision-language model, the text encoder remains unused. Consequently, the two category prototypes are learned solely from that source, and cannot benefit from the provenance semantics acquired through language alignment.

Therefore, better exploiting the AIGI detection potential of vision-language models does not require a more complex detector. The key is to change how the two forensic category prototypes are learned. Rather than estimating them entirely from a single training source, we use the model’s existing semantic information to provide a task-relevant starting point and then calibrate the prototypes on the training data.

Let $f_{I}$ denote the frozen image encoder, $\mathcal{N}$ the $\ell_{2}$-normalization operator, and $W$ the trainable binary linear head. Given an input image $I$, standard linear probing can be written as:

$$\mathbf{x}=\mathcal{N}(f_{I}(I)),\quad\mathbf{z}=\mathbf{x}W.$$

(1)

In a binary linear head, each column of the weight matrix $W$ corresponds to one output class. We denote these columns by $p_{1}$ and $p_{2}$, corresponding to generated and real images, respectively:

$$W=\left[p_{1},p_{2}\right],\quad\mathbf{z}=\left[\mathbf{x}p_{1},\mathbf{x}p_{2}\right].$$

(2)

As illustrated in Figure 2, the binary linear head can therefore be viewed as a pair of learnable forensic category prototypes.

This prototype interpretation reveals a limitation of conventional linear probing. Without task-relevant starting points, $p_{1}$ and $p_{2}$ are learned solely from a single training source, and may therefore become biased toward patterns specific to that source. As a result, these prototypes may fail to capture the general forensic differences between generated and real images, limiting generalization to unseen generators and diverse real-image sources.

Unlike vision-only models, vision-language models align image and text representations in a shared embedding space (Radford et al. 2021). We therefore derive semantic starting points for $p_{1}$ and $p_{2}$ by encoding descriptions of generated and real images. Using the text encoder $f_{T}$, we compute and normalize the resulting text embeddings.

$$\begin{array}[]{l}p_{1}\leftarrow\mathcal{N}(f_{T}(\mbox{``AI art''})),\\ p_{2}\leftarrow\mathcal{N}(f_{T}(\mbox{``a real photo''})).\end{array}$$

(3)

We then calibrate the two prototypes on the training set by minimizing the cross-entropy loss $\mathcal{L}_{\mathrm{CE}}([\mathbf{x}p_{1},\mathbf{x}p_{2}],y)$, while keeping the image encoder frozen.

We refer to this process as prototype calibration because training refines prototypes that already encode class semantics. The text-derived prototypes lie near task-optimal directions in PE’s embedding space, so calibration requires only limited adjustment. Such limited adjustment reduces the classifier’s reliance on training-source-specific cues, thereby improving cross-distribution generalization.

Once the semantic starting points are obtained, neither text input nor the text encoder is required for prototype calibration or inference. Consequently, PE-SPC retains the same inference architecture as PE-Linear, comprising a frozen image encoder and a binary linear head, and incurs no additional memory or computational cost at inference time.

Figure 3 shows that SPC improves PE-Linear across all five benchmark groups. The largest gain occurs on AIGI-Now (+6.1%), while In-the-Wild and Blur-and-JPEG improve by 4.1% and 4.2%, respectively. These benchmarks differ substantially from the SDv1.4 training distribution, making cues tied to the training generator less reliable. The consistent gains under these shifts indicate that SPC enables the classification head to make better use of the transferable forensic information in PE’s representation. SPC also improves GenImage by 4.3%, demonstrating stronger cross-generator generalization within a controlled benchmark. Even on the near-saturated AIGI-Holmes benchmark, SPC delivers a further improvement of 0.9%, showing that it remains effective when the baseline performance is already high.

Across the detailed benchmark results, SPC’s gains are concentrated on the datasets and conditions where PE-Linear performs poorly. SPC improves overall performance primarily by substantially raising the minimum accuracy in every benchmark group, with the largest increases on GenImage (+21.1%), Blur-and-JPEG (+16.0%), and AIGI-Now (+10.2%). This results in more stable performance across diverse datasets. Overall, PE-SPC matches or exceeds DINOv3-Linear in average accuracy across all benchmarks.

PE-SPC achieves these results with a 1.9B-parameter ViT-G/14 backbone, compared with the 6.7B-parameter ViT-7B/16 backbone used by DINOv3-Linear. This favorable trade-off between accuracy and model scale highlights vision-language models as a promising direction for efficient AIGI detection, while leaving substantial room to explore larger backbones. Appendix B provides detailed model-cost and inference-efficiency measurements.

Table 1 evaluates cross-generator generalization under the standard GenImage (Zhu et al. 2023) protocol. PE-Linear already performs nearly perfectly on the Stable Diffusion and Wukong subsets, but transfers less effectively to ADM and Midjourney. SPC preserves PE’s strong performance on the former subsets while substantially improving the latter, indicates that SPC primarily improves transfer beyond the single training source. PE-SPC substantially improves upon PE-Linear and surpasses DINOv3-Linear, achieving the highest average accuracy.

Table 2 evaluates robustness across four in-the-wild collections with heterogeneous sources, unknown generators, and uncontrolled processing histories. PE-Linear performs strongly on Chameleon and CommunityAI but transfers less effectively to SocialRF and WildRF, revealing substantial variation across real-world sources. SPC delivers its largest gains on the two weaker collections while retaining strong performance on Chameleon and CommunityAI, resulting in more balanced performance and matching DINOv3-Linear on average. This improved stability makes PE-SPC more reliable in practical settings, where image provenance and processing history are usually unknown.

To test whether PE-SPC exploits transferable forensic information rather than generator-specific patterns memorized during image-encoder pretraining, we evaluate it on AIGI-Now (Chen et al. 2025a) and AIGI-Holmes (Zhou et al. 2025c), both of which include generators unseen during pretraining. AIGI-Now includes closed-source generators unseen during image-encoder pretraining, such as GPT-4o and FLUX-Pro, and separates evaluation into a “pix” split that isolates low-level generative traces by aligning image formats and a “sem” split that suppresses these traces through strong degradations to emphasize high-level semantic anomalies. In Tables 3, PE-SPC achieves the best overall performance on AIGI-Now, clearly surpassing DINOv3-Linear. Relative to PE-Linear, its improvement is substantially larger on “sem” than on “pix”, indicates that SPC primarily strengthens the use of high-level forensic semantics.

AIGI-Holmes extends this evaluation to recent generators whose architectures differ from the UNet-based SD1.4 training source, including autoregressive models such as LlamaGen and VAR, as well as diffusion transformers such as FLUX and SD3.5-L. On the near-saturated AIGI-Holmes benchmark in table 4, PE-SPC again ranks first and outperforms both PE-Linear and DINOv3-Linear. Compared with PE-Linear, its gains are concentrated on VAR, SD3.5-L, and FLUX, spanning both autoregressive and diffusion-transformer generators. Together with the AIGI-Now results, this transfer to generators unseen during pretraining and to architectures distinct from SD1.4 provides evidence that SPC exploits generalizable forensic information rather than memorized generator-specific patterns.

Table 5 evaluates robustness to JPEG compression and Gaussian blur, two common post-processing operations encountered during image dissemination. Both perturbations weaken low-level forensic cues, as JPEG compression alters local image statistics through quantization and Gaussian blur progressively removes high-frequency information. PE-SPC achieves the highest average performance and outperforms DINOv3-Linear across all perturbation settings. PE-Linear remains strong under JPEG compression but deteriorates rapidly as the blur strength increases, whereas PE-SPC remains stable even under severe blur. This contrast indicates that SPC reduces reliance on post-processing-sensitive cues and makes better use of forensic information preserved under common perturbations.

Table 6 examines whether SPC’s gains arise from exact wording, text-derived starting points alone, or forensic semantics correctly aligned with the class labels. Our Prompt and Similar Prompt use semantically equivalent provenance descriptions and achieve comparable gains across all five benchmark groups, showing that SPC does not depend on exact wording. Opposite Prompt reverses the class assignments and sharply degrades performance, demonstrating that the forensic semantics must be aligned with the correct classes. Unrelated Prompt I and Unrelated Prompt II replace forensic descriptions with unrelated content semantics and both perform worse than No SPC, ruling out a generic benefit from text-derived starting points. Together, these results show that SPC works through class-aligned forensic semantics rather than prompt wording or text-derived starting points alone.

This section analyzes how SPC improves AIGI detection in vision-language models and why it benefits some models but not others. Figure 4 shows that SPC’s effect depends on whether a model has learned forensic information that distinguishes generated from real images during pretraining. Models that recognize synthetic images as AI-generated improve under SPC, whereas those that associate them with content concepts or incorrectly regard them as genuine deteriorate. This contrast demonstrates that SPC’s gains come not from text embeddings alone but from forensic information learned during pretraining. Appendix F details the candidate concepts and aggregation protocol used in this analysis.

The OpenCLIP checkpoint used in our experiments was pretrained on LAION-2B, which was collected in 2021–2022 before the rapid growth of synthetic content on the web (Schuhmann et al. 2022). Despite its recent release, SigLIP2 exhibits the same negative response to SPC as OpenCLIP, likely because it still relies on WebLI, which also predates this growth (Tschannen et al. 2025; Zhou et al. 2026). By contrast, MetaCLIP2 and PE were pretrained on newer web corpora containing large volumes of AI-generated images together with captions or metadata that reveal their origin (Chuang et al. 2025; Bolya et al. 2025).

The usefulness of SPC’s semantic starting points depends on whether the model learned relevant forensic information during pretraining. When synthetic images are paired with captions or metadata that identify them as AI-generated, the model learns to associate visual evidence of generation with forensic concepts. The resulting text-derived prototypes already lie near task-optimal directions, shortening the optimization path required for calibration. Without this association, the text-derived prototypes are misaligned with the model’s image representations and can instead steer calibration in the wrong direction.