ColSNAP : Spatial Matryoshka Training for Multi-Granularity Visual Document Retrieval

Trishan Singha Roy ^†^†thanks: Work done as an intern at IBM Arkadeep Acharya Vishwajeet Kumar Jaydeep Sen Sachindra Joshi

Abstract

Multi-modal late-interaction retrievers achieve strong retrieval on visually rich documents by representing each page as per-patch embeddings and matching at the token level. However, this approach incurs high storage costs. Existing compression methods typically fix a single compression level at indexing time, limiting flexibility. We present ColSNAP (Spatial Nested Average Pooling)¹¹ 1 Code will be released upon publication., a training method that generates a nested hierarchy of compression levels directly from a backbone’s patch grid. By spatially pooling patch embeddings into progressively coarser tiers and training all tiers simultaneously, a single model learns to support retrieval at multiple compression levels without architectural changes. Crucially, a single encoding pass yields every tier, enabling the accuracy-storage trade-off to be configured at indexing time to match available storage budgets, rather than being fixed during training. We demonstrate that models trained using ColSNAP maintain near full-resolution retrieval performance under substantial compression and that ColSNAP transfers effectively across multiple late-interaction backbones, and achieves most of its improvements via a lightweight adaptation stage applied to a pre-trained retriever.

¹IIT Delhi

²IBM

ee1230783@iitd.ac.in, acharyarka17@ibm.com, vishk024@in.ibm.com, jaydesen@in.ibm.com, jsachind@in.ibm.com

1 Introduction

Visual Document Retrieval (VDR) retrieves document pages relevant to a natural language query using both the visual layout and textual content of a page (Yan et al. 2026a; Faysse et al. 2025). Relevance cannot be determined from text alone: it depends on visual elements such as tables, charts, and figures, and on their spatial organization. Early approaches applied optical character recognition (OCR) (Hegghammer 2022; Zhang et al. 2025) to extract text for a conventional retriever, but such pipelines are brittle, as recognition errors propagate downstream and layout structure is lost when pages are verbalized. Vision-language models (VLMs) instead embed the rendered page image directly, eliminating the extraction stage.

The emergence of multi-modal embedding models has led to two dominant approaches. 1) Single-vector methods encode each page and query into one dense embedding, scored by cosine similarity (Ma et al. 2024; Meng et al. 2025). These are compact and support efficient approximate nearest-neighbour search, but compressing a page into one vector discards the fine-grained evidence that complex documents require, where relevance may hinge on a single table row or figure annotation (Weller et al. 2026). 2) Multi-vector methods adopt the late-interaction paradigm of ColBERT (Khattab and Zaharia 2020), extending it to visual documents. Systems such as ColQwen and ColPali (Faysse et al. 2025) represent a page as per-patch embeddings and score queries with a MaxSim operator that aligns each query token to its most relevant page region, preserving the localized evidence that pooling discards. This has established multi-vector representations as the dominant paradigm in VDR (de Souza P. Moreira et al. 2026; Günther et al. 2025). The cost, however, is substantial: a single page yields hundreds to thousands of stored vectors, so index size and query-time scoring both scale with the number of patches per page, and higher-resolution inputs push this further, making storage a limiting factor in large-scale deployment. This cost–quality tradeoff motivates two research questions:

•

RQ1: Can a method maintain the retrieval performance of an existing retriever while substantially reducing the number of vectors needed to encode each document page?
•

RQ2: Does such a method generalize across retrievers of differing size and retrieval performance?

We address these questions with ColSNAP (Spatial Nested Average Pooling), a training recipe that builds a nested hierarchy of compression levels directly from the backbone’s patch grid. The key insight is that pretrained patch grids exhibit natural hierarchy-spatially adjacent patches are highly redundant, so averaging them yields meaningful coarser embeddings. In a single encoding pass, we spatially pool patch embeddings into a pyramid of granularities ( $1024\to 128\to 16\to 1$ ), adding no tokens and no change to the encoder architecture. All tiers are trained jointly via Spatial Matryoshka learning with two distillation signals: (1) a frozen-teacher term anchoring all tiers to the original pretrained model, and (2) a cross-level term aligning coarse tiers to the live model’s finest tier. At indexing time, users select a tier matching their storage budget, achieving up to $\sim\!70\times$ smaller storage footprint on Nemotron ColEmbed 4B (de Souza P. Moreira et al. 2026) and $\sim\!130\times$ smaller on ColPali v1.3 (Faysse et al. 2025).

Figure 1: Overview of ColSNAP. A shared vision–language encoder maps a document page to a row of patch embeddings (blue), flanked by a few auxiliary template tokens (grey), and the query to a set of query embeddings (purple). Spatial average-pooling compresses the patch embeddings into a nested pyramid of tiers (L0–L3) while leaving the auxiliary tokens unchanged; any tier can then be scored against the query with late-interaction MaxSim. A single encoding pass produces every tier, so the compression level is selected at index time.

We summarize our contributions as follows:

•

We propose ColSNAP , a method that reduces storage costs of multi-vector retrievers by learning multiple compression levels from a single patch grid, without changing the encoder architecture.
•

ColSNAP achieves up to $\sim\!70\times$ compression while retaining over $90\%$ of full-resolution quality, with compression levels configurable at indexing time.
•

The method generalizes across retrievers of differing size and retrieval performance (demonstrated on Nemotron ColEmbed 4B (de Souza P. Moreira et al. 2026) and ColPaLI v1.3 (Faysse et al. 2025)) and requires only lightweight adaptation of pre-trained models.
•

We evaluate on ViDoRe v1 (Faysse et al. 2025), v2 (Macé et al. 2025), and the public splits of v3 (Loison et al. 2026) benchmarks and show that multi-vector retrieval is practical at scale with ColSNAP .

The remainder of this paper is organized as follows. Section 2 reviews related work in neural retrieval and representation learning. Section 3 presents the ColSNAP method in detail. Section 4 describes our experimental setup and benchmarks. Section 5 presents results and analysis. Finally, Section 6 concludes the paper.

2 Related Work

Late-interaction retrieval, introduced by ColBERT (Khattab and Zaharia 2020), encodes queries and documents into sequences of per-token embeddings and scores a pair with MaxSim, preserving fine-grained matching that a single pooled vector discards, at the cost of one vector per token. ColPali (Faysse et al. 2025) extended this to visually rich documents by encoding rendered page images into multi-vector patch embeddings, sidestepping brittle OCR pipelines, and the same recipe has since carried over to stronger backbones such as Nemotron ColEmbed (de Souza P. Moreira et al. 2026). However, this accuracy comes at a cost: each page is stored as hundreds to thousands of patch vectors, so index size and retrieval latency grow with the number of tokens per page.

A large body of work reduces this cost for text. ColBERTv2 (Santhanam et al. 2022b) quantizes embeddings against learned centroids, later pruned by PLAID (Santhanam et al. 2022a); other methods reduce the number of stored vectors directly, via token pooling (Clavié et al. 2024), learned token retention (Lee et al. 2024), clustering (Veneroso et al. 2025), or a single fixed-dimensional encoding whose inner product approximates MaxSim (Dhulipala et al. 2026). Several of the strongest methods, such as CITADEL’s lexical-key routing (Li et al. 2023), exploit discrete, term-level structure that has no analogue for continuous image patches.

Compression for visual document retrieval is more recent, from training-free pruning and merging of patch embeddings (Yan et al. 2025; Liu et al. 2026; Yan et al. 2026c; Yan et al. 2026b) to methods that finetune a module or the retriever itself (Ma et al. 2025; Cha et al. 2026; Park et al. 2026). Most of these commit to a single compression level fixed at indexing time. A few instead serve several levels from one model: MM-Matryoshka nests along embedding width and encoder depth (Xiang et al. 2026), MetaEmbed appends learnable Meta Tokens trained end to end in nested coarse-to-fine groups (Xiao et al. 2026), and jina-embeddings-v4 trains adapters to emit both a truncatable single vector and a multi-vector representation (Günther et al. 2025). The spatial layout of the patch grid, the one form of structure visual tokens reliably possess, remains largely unused as an axis for such a hierarchy.

Matryoshka Representation Learning (MRL) (Kusupati et al. 2024) encodes multiple granularities within a single vector by ordering dimensions by importance. This principle has been extended from dimension to vector count:M3 (Cai et al. 2024) learns visual tokens for multimodal generation, while MetaEmbed (Xiao et al. 2026) applies it to late-interaction retrieval via Meta Tokens. To the best of our knowledge, for visual document retrieval, ColSNAP is the first method to build a nested multi-vector hierarchy from the backbone’s patch grid. We spatially average-pool per-patch embeddings into a granularity pyramid, training each tier for retrieval. Since this spatial axis is orthogonal to embedding dimensionality, it composes with dimension-side compression without changing the encoder.

3 ColSNAP

3.1 Overview

ColSNAP addresses the storage challenge of multi-vector retrievers by constructing a nested hierarchy of token compression levels from a single forward pass. The core insight is simple: a pretrained $32\times 32$ patch grid exhibits natural spatial hierarchy. Adjacent patches encode redundant information about the same document region, so spatially averaging them substantially reduces the vector count while retaining much of the retrieval quality. ColSNAP trains all compression tiers jointly using a shared contrastive objective, together with two distillation signals. This lets the accuracy-storage trade-off be configured at indexing time rather than fixed during training. This allows a single model to serve diverse deployment scenarios without retraining.

3.2 Setting and Background

Problem:

Given a textual query and a corpus of documents whose pages are represented as rendered images, Visual Document Retrieval ranks pages by their relevance to the query without OCR or layout parsing.

Encoding:

A vision-language model encodes both sides into a shared $d$ -dimensional space. The query is tokenized and encoded into a set of query embeddings $Q=\{q_{1},\dots,q_{n_{q}}\}$ , $q_{t}\in\mathbb{R}^{d}$ . Each page image is processed by a vision encoder into a set of document embeddings. The encoder emits a $32\times 32$ grid of patch embeddings, denoted by $p_{1},\dots,p_{n_{p}}$ where $n_{p}=1024$ and $p_{i}\in\mathbb{R}^{d}$ , as well as a small number of auxiliary tokens (e.g., special tokens before and after the patch grid), denoted $a_{1},\dots,a_{k}$ where $a_{j}\in\mathbb{R}^{d}$ . The full document representation is:

D=\{a_{1},\dots,a_{k},\;p_{1},\dots,p_{n_{p}}\}.

(1)

Late Interaction Scoring:

Late-interaction retrieval (Khattab and Zaharia 2020) scores a query-document pair by summing, over each query embedding, its maximum similarity to any document embedding:

S(Q,D)\;=\;\sum_{t=1}^{n_{q}}\;\max_{v\in D}\;\langle q_{t},v\rangle,

(2)

where $|D|=k+n_{p}$ . This preserves fine-grained detail by matching each query token to its most relevant document region.

3.3 Spatial Pooling Pyramid

Figure 2: Spatial pooling pyramid. The

32\times 32

patch grid is average-pooled over increasingly large windows to form four tiers: L0 (

1\times 1

blocks,

1024

embeddings), L1 (

2\times 4

blocks,

16\times 8=128

), L2 (

8\times 8

blocks,

4\times 4=16

), and L3 (a single

32\times 32

block,

1

). Each shaded block is averaged into one pooled embedding.

ColSNAP builds a nested pyramid of coarser views of the $32\times 32$ patch grid by spatially average-pooling patch embeddings over progressively larger windows. Because each pooling window is nested inside the next, the resulting tiers are hierarchically consistent: every coarse embedding summarizes a contiguous spatial block of the finer embeddings beneath it.

This construction produces four tiers, denoted $L_{0},L_{1},L_{2},L_{3}$ (indexed by $\ell\in\{0,1,2,3\}$ ), where tier $L_{\ell}$ contains $n_{\ell}$ patch embeddings arranged as a grid:

	$\displaystyle L_{0}$	$\displaystyle:32\times 32=1024,$	$\displaystyle\quad L_{1}$	$\displaystyle:16\times 8=128,$		(3)
	$\displaystyle L_{2}$	$\displaystyle:4\times 4=16,$	$\displaystyle\quad L_{3}$	$\displaystyle:1\times 1=1.$		(3)

Each tier is a complete, coarser summary of the same underlying patches. We denote the pooled patch embeddings at tier $\ell$ as $p_{1}^{(\ell)},\dots,p_{n_{\ell}}^{(\ell)}$ . Importantly, the coarser tiers introduce no additional parameters and reuse the same underlying embeddings; Figure 2 illustrates the construction.

3.4 Multi-Tier Scoring

ColSNAP scores identically across all tiers using the MaxSim operator. At tier $L_{\ell}$ , we replace the $n_{p}=1024$ patch embeddings with their pooled counterparts $p_{1}^{(\ell)},\dots,p_{n_{\ell}}^{(\ell)}$ , while the auxiliary tokens $a_{1},\dots,a_{k}$ are carried over unchanged. The document representation at tier $\ell$ is:

D^{(\ell)}=\{a_{1},\dots,a_{k},\;p_{1}^{(\ell)},\dots,p_{n_{\ell}}^{(\ell)}\}.

(4)

The relevance score at tier $\ell$ , written $S^{(\ell)}(Q,D)$ , is then Eq. 2 with $D$ replaced by $D^{(\ell)}$ , where $|D^{(\ell)}|=k+n_{\ell}$ . $\ell=0$ applies no pooling, so $D^{(0)}=D$ : the finest tier is identical to the original, unpooled retriever. We compress only the patch embeddings, whose count varies across tiers through $1024,128,16,1$ , leaving the auxiliary tokens untouched.

3.5 Joint Training via Spatial Matryoshka Learning

To optimize all tiers simultaneously without degrading the original model’s performance, we employ a joint training objective combining three loss terms.

Spatial Matryoshka Ranking Loss.

Each tier is trained to rank matched documents above in-batch negatives. During training, each step samples a batch of $B$ matched query–document pairs. For every tier $L_{\ell}$ (where $\ell\in\{0,1,2,3\}$ ), we compute MaxSim scores between all $B$ queries and $B$ documents, forming a score matrix:

S^{(\ell)}\in\mathbb{R}^{B\times B},\quad S^{(\ell)}_{ij}=S^{(\ell)}(Q_{i},D_{j}^{(\ell)}),

(5)

where each entry is the MaxSim score of Eq. 2, evaluated for query $i$ against document $j$ at tier $\ell$ . The diagonal entries $S^{(\ell)}_{ii}$ contain scores for matched (positive) pairs, and off-diagonal entries $S^{(\ell)}_{ij}$ (where $i\neq j$ ) are in-batch negatives. For query $i$ at tier $\ell$ , the positive score is the diagonal entry $s_{+}^{(i,\ell)}=S^{(\ell)}_{ii}$ , and the hardest negative is $s_{-}^{(i,\ell)}=\max_{j\neq i}S^{(\ell)}_{ij}$ . A softplus penalty pushes the positive above this hardest negative, and the Spatial Matryoshka ranking loss sums this across all tiers:

\mathcal{L}_{\text{SMRL}}=\sum_{\ell=0}^{3}\frac{1}{B}\sum_{i=1}^{B}\log\!\bigl(1+\exp\!\bigl(s_{-}^{(i,\ell)}-s_{+}^{(i,\ell)}\bigr)\bigr).

(6)

In order to preserve the performance of the model at its original resolution, we add a frozen-teacher term $\mathcal{L}^{(\ell)}_{\text{frozen}}$ , which matches the student’s distribution over the $B$ in-batch documents to that of a frozen copy of the pretrained backbone through a temperature-scaled KL divergence. It is applied at every level, anchoring each tier to the original model at full resolution.

Additionally, in order to align the coarser tiers with the model’s own finest tier, we introduce a live cross-tier term $\mathcal{L}^{(\ell)}_{\text{live}}$ , which matches the student’s distribution at tier $\ell$ to its current L0 distribution, detached from the gradient, and applies at $\ell\in\{1,2,3\}$ . In both terms the per-query scores are z-scored before the softmax, so that differences in score scale across tiers do not dominate the distributions being matched.

Finally, we obtain the overall objective as

\mathcal{L}=\mathcal{L}_{\text{SMRL}}\;+\;\lambda_{f}\sum_{\ell=0}^{3}\mathcal{L}^{(\ell)}_{\text{frozen}}\;+\;\lambda_{l}\sum_{\ell=1}^{3}\mathcal{L}^{(\ell)}_{\text{live}}.

(7)

3.6 Flexible Deployment via Granularity Selection

The nested design decouples the compression level from training. Because all tiers are optimized jointly and remain hierarchically consistent, a corpus encoded at $L_{0}$ can be pooled down to any coarser tier on demand. At indexing time, each page is stored at the tier $L_{\ell}$ that fits the available storage budget. At query time, retrieval cost is bounded by that choice: MaxSim operates over $k+n_{\ell}$ embeddings, where the auxiliary count $k$ is fixed and only $n_{\ell}$ varies. A single trained model therefore serves multiple operating points without retraining or maintaining separate checkpoints.

4 Experimental Setup

In this section we describe the backbone models adapted with ColSNAP , the datasets and training configurations used in our experiments, and the benchmarks and evaluation metrics used for evaluation.

4.1 Model Backbones

We apply ColSNAP to two late-interaction retrieval backbones with different scales and pretraining. Our primary backbone is Nemotron ColEmbed VL 4B (de Souza P. Moreira et al. 2026), which produces 2560-dimensional embeddings and ranks among the top models on the ViDoRe V3 leaderboard. We also apply ColSNAP to ColPali v1.3 (Faysse et al. 2025), built on PaliGemma-3B (Beyer et al. 2024), which produces 128-dimensional embeddings. As a smaller, widely used backbone, ColPali helps to assess the generalizability of ColSNAP across model scales.

For both backbones, input images are resized to produce a $32\times 32$ grid of patch tokens, enabling the same pooling pyramid to be used without modification. In addition to the patch tokens, Nemotron and ColPali produce $14$ and $7$ auxiliary tokens respectively, which are preserved unchanged across all compression tiers.

4.2 Training Setup

Training data:

We use the ColPali-Train dataset (Faysse et al. 2025) with 95:5 train:validation split. In our primary experiments, each training step samples 128 matched query–document pairs (positives). To explore the effect of hard negatives, we additionally conduct experiments using mined hard negatives released by Nomic AI (Nomic AI 2025)²² 2 https://huggingface.co/datasets/nomic-ai/colpali-queries-mined-20250321-by-source, where each query is paired with one hard negative sampled uniformly from its top-10 mined candidates. In this setting, the effective batch size is 128 documents (64 positives + 64 negatives).

Optimization and hyperparameters:

We jointly optimize all the four tiers using the objective in Eq. 7 for 5 epochs, selecting the best checkpoint on the validation split. Training uses LoRA (Hu et al. 2021) ( $\mathrm{rank}=32$ , $\alpha=32$ ) with AdamW (Loshchilov and Hutter 2019) and a linear warmup-decay schedule. Models are trained in bfloat16 with distributed data parallelism on four NVIDIA A100 GPUs using an effective batch size of 128. Unless otherwise stated, we set $\lambda_{f}=2$ for Nemotron and $\lambda_{f}=0.5$ for ColPali, with $\lambda_{l}=0.3$ and a distillation temperature of $\tau=1.5$ for both backbones.

4.3 Evaluation Benchmarks and Metrics

Benchmarks.

We evaluate the performance of models adapted using ColSNAP on ViDoRe v1 (Faysse et al. 2025), ViDoRe v2 (Macé et al. 2025), and the public splits of ViDoRe V3 (Loison et al. 2026). ViDoRe V1 is an in-distribution benchmark, while ViDoRe v2 and v3 evaluate out-of-distribution across document types and domains.

Metrics.

Following the official leaderboards, we report nDCG@5 on ViDoRe v1/v2 and nDCG@10 on ViDoRe v3. Each pyramid level (L0–L3) is evaluated independently using tiered MaxSim (Eq. 2) within the MTEB framework (Muennighoff et al. 2023; Enevoldsen et al. 2025). We also report retention, the ratio of a tier’s retrieval score to that of the corresponding backbone at full resolution (L0).

5 Results and Analysis

Our analysis proceeds by evaluating ColSNAP across two late-interaction backbones with different scales and architectures: (i) Nemotron ColEmbed 4B and (ii) ColPali v1.3. We first discuss retrieval quality and compression behavior across both backbones, and then highlight gains ColSNAP delivers in training efficiency, storage requirements, and retrieval cost.

5.1 Performance of ColSNAP on Nemotron ColEmbed backbone

Refer to caption — Figure 3: Compression–quality tradeoff on Nemotron (ViDoRe V3, nDCG@10). Compression on a log scale; each marker is a tier (L0/L1/L2/L3 at $1\times$ , $7.3\times$ , $34.6\times$ , $69.2\times$ ). Curves show the untrained baseline, a full-resolution-only control, ColSNAP, and ColSNAP with mined hard negatives.

We evaluate a model trained with ColSNAP on the Nemotron ColEmbed VL 4B backbone and report results in Table 1 alongside existing late-interaction baselines of similar parameter size on ViDoRe v1, v2, and v3. In Figure 3, we further illustrate the compression–quality trade-off on this backbone, comparing four settings: (i) the untrained model with our pooling applied at inference time, (ii) the model finetuned with only the softplus loss at full resolution, (iii) the model trained with ColSNAP using our proposed objective in Equation 7, and (iv) the model trained with ColSNAP using Equation 7 and data containing mined hard negatives. Based on these results, we highlight three key observations:

1) Retrieval quality degrades gracefully with compression:

Figure 3 shows that model trained with ColSNAP achieves a graceful compression–quality trade-off. On ViDoRe v3, retrieval quality decreases from 61.7 at full resolution (L0) to 59.7, 57.4, and 56.7 for the L1, L2, and L3 tiers, respectively. Notably, the L1 tier stores roughly seven times fewer vectors while retaining over 96% of the full-resolution performance. In contrast, both the untrained baseline and the full-resolution control exhibit a much steeper degradation from L0 to L3, highlighting that the proposed training objective in ColSNAP is key to learning compressed representations that retain retrieval quality across all the tiers.

2) Compressed tiers remain competitive with full-resolution retrievers.

Table 1 shows that L1 tier surpasses several full-resolution retrievers on ViDoRe v3, including jina-embeddings-v4 (Günther et al. 2025) and colnomic-embed-multimodal-3b (Team 2025) across the ViDoRe benchmarks. This holds even for more compressed tiers: on ViDoRe v3, L2 (57.4) nearly matches jina-embeddings-v4 (57.5) and outperforms colnomic-embed-multimodal-3b (56.4). Even single patch-embedding L3 (56.7) stays competitive on this benchmark, beating colnomic-embed-multimodal-3b and trailing llama-nemoretriever-colembed-3b-v1 (57.1) by just 0.4 nDCG. A similar trend can be observed across ViDoRe V1 and ViDoRe V2, thus underscoring that ColSNAP achieves substantial compression with minimal quality loss versus strong full-resolution baselines.

3) Compression remains robust under distribution shift.

As shown by the retention values in Table 1, on the in-distribution ViDoRe v1 benchmark, even the single patch-embedding L3 tier retains 97.2% of the full-resolution performance. This retention remains high on the out-of-distribution ViDoRe v2 and v3 benchmarks, with L3 preserving 92.5% and 91.5% of full-resolution performance, respectively. Across all benchmarks, the L1 tier is particularly robust, retaining at least 95.4% of the full-resolution score. These results indicate that ColSNAP helps in learning compressed representations that generalize well beyond the training data distribution.

Together, these observations address RQ1: a retriever adapted with ColSNAP preserves over $91\%$ of its full-resolution quality while storing up to 69× fewer embeddings per page.

Model		ViDoRe v1/v2 (nDCG@5, %)		ViDoRe v3 public (nDCG@10, %)
Model		v1	v2	CS	Ene	FiEn	FiFr	HR	Ind	Pha	Phy	Mean
ColQwen3.5-4.5B		91.5	64.3	78.7	68.0	64.1	48.6	62.1	55.2	65.6	50.3	61.6
jina-embeddings-v4		90.4	58.2	71.8	63.5	59.3	46.1	59.5	50.4	63.1	46.6	57.5
llama-nemoretriever-colembed-3b-v1		91.0	63.3	75.2	62.1	60.9	43.8	58.7	47.1	63.7	45.1	57.1
colnomic-embed-multimodal-3b		89.9	55.7	72.7	64.5	56.3	44.3	57.3	47.4	61.1	47.6	56.4
Nemotron ColEmbed 4B		91.7_(100.0)	65.0_(100.0)	78.6	68.4	65.9	49.5	62.8	54.6	66.6	49.2	62.0_(100.0)
ColSNAP-Nemotron-ColEmbed-4B	L0	91.7_(100.0)	63.2_(97.2)	79.0	67.6	65.0	49.4	62.9	53.9	66.0	49.5	61.7_(99.6)
	L1	90.5_(98.7)	62.0_(95.4)	77.8	66.2	60.9	48.0	60.2	50.8	64.6	49.4	59.7_(96.4)
	L2	89.7_(97.8)	61.0_(93.7)	76.7	64.4	58.5	45.3	56.6	47.0	63.1	48.0	57.4_(92.7)
	L3	89.1_(97.2)	60.2_(92.5)	75.6	63.3	57.7	44.8	56.0	46.1	62.0	48.1	56.7_(91.5)
ColSNAP-Nemotron-ColEmbed-4B-hard-negs	L0	91.9_(100.2)	63.4_(97.5)	78.5	68.2	64.0	49.3	62.8	53.8	65.7	50.2	61.6_(99.4)
	L1	91.2_(99.4)	62.2_(95.7)	77.7	66.8	61.2	47.7	60.4	51.2	65.0	50.3	60.0_(96.9)
	L2	90.4_(98.6)	60.5_(93.0)	77.0	65.0	57.0	44.5	57.9	48.0	62.8	49.2	57.7_(93.1)
	L3	90.1_(98.3)	58.6_(90.1)	76.1	64.2	55.9	43.6	57.6	47.1	62.3	49.4	57.0_(92.0)

Table 1: Comparison across ViDoRe generations. Mean nDCG@5 on ViDoRe v1 and v2, and per-dataset nDCG@10 on the eight public ViDoRe v3 tasks (computer science, energy, finance-en, finance-fr, HR, industrial, pharmaceuticals, physics), with their mean. Parenthesised subscripts denote retention, computed relative to the untrained Nemotron ColEmbed 4B baseline. Top: representative models (retention not applicable). Bottom: the baseline backbone trained with ColSNAP at each tier (L0–L3), without and with mined hard negatives.

5.2 Transferability of ColSNAP to ColPali v1.3 backbone

We evaluate the model trained with ColSNAP on ColPali v1.3 to assess the transferability of the method to a smaller and structurally different backbone under the same experimental setting as described earlier for Nemotron ColEmbed backbone in subsection 5.1. The key metrics have been highlighted in Table 2 and Figure 4. We make three key observations:

1) ColSNAP transfers effectively to smaller backbones.

Similar to Nemotron, ColPali exhibits a graceful compression–quality trade-off under ColSNAP . The L1 tier reduces the number of embedding vectors by $7.6\times$ while remaining nearly lossless, retaining $100.2\%$ , $98.0\%$ , and $99.8\%$ of the full-resolution quality on ViDoRe v1, v2, and v3, respectively. This demonstrates that ColSNAP generalizes across backbones and preserves retrieval quality under moderate compression. As on Nemotron, the full-resolution control leaves the coarse tiers near their untrained values, indicating that the improvements at the compressed tiers likewise come from the nested objective rather than additional training alone.

2) Extreme compression is limited by backbone capacity.

At more aggressive compression levels, ColPali shows a larger degradation than Nemotron. Without hard negatives, the L2 and L3 tiers retain $88.5\%$ and $76.2\%$ of full-resolution performance on ViDoRe v3, compared to $92.7\%$ and $91.5\%$ for Nemotron. We attribute this gap to ColPali’s lower-dimensional embeddings ( $128$ dimensions versus $2560$ for Nemotron), which provide less capacity to represent page-level information after heavy pooling.

3) Hard negatives improve robustness under aggressive compression.

Training with mined hard negatives improves the compressed tiers, particularly at higher compression levels. On ViDoRe v3, hard negatives improve L2 and L3 retention from $88.5\%$ to $89.4\%$ and from $76.2\%$ to $80.7\%$ , respectively. The gains are greater in ViDoRe v2, where L3 retention improves from $68.8\%$ to $83.4\%$ . These results suggest that hard negatives help ColSNAP learn more discriminative compressed representations when the available representation capacity is limited.

These observations address RQ2: ColSNAP transfers to a smaller, lower-dimensional backbone, maintaining lossless performance up to 7.6× compression despite the limited capacity of its 128-dimensional embeddings.

		nDCG (%)			Retention (%)
Model	Level	v1@5	v2@5	v3@10	v1	v2	v3
ColPali v1.3	L0	84.0	54.5	44.6	100.0	100.0	100.0
ColSNAP-ColPali v1.3	L0	86.1	54.8	46.9	102.5	100.6	105.2
	L1	84.1	53.5	44.5	100.2	98.0	99.8
	L2	80.6	46.4	39.5	96.0	85.0	88.5
	L3	74.7	37.5	34.0	89.0	68.8	76.2
ColSNAP-ColPali v1.3-hard negs	L0	86.4	54.7	47.1	102.8	100.3	105.6
	L1	84.7	52.8	44.7	100.9	96.9	100.1
	L2	81.9	49.1	39.9	97.5	90.1	89.4
	L3	77.3	45.4	36.0	92.0	83.4	80.7

Table 2: Performance of ColSNAP on ColPali v1.3 backbone on ViDoRe Benchmarks (V1/V2/V3). Per-tier retrieval quality and retention (%), relative to the untrained baseline’s L0. ViDoRe v1 and v2 are reported as nDCG@5, and ViDoRe v3 as nDCG@10. The top row is the untrained ColPali v1.3 baseline’s full-resolution (L0) score.

5.3 ColSNAP Achieves Rapid Convergence

Figure 5 highlights the efficiency of ColSNAP as a post-training adaptation method. We plot per-tier nDCG@10 on ViDoRe v3 across training epochs for both backbones. The curves show that ColSNAP converges quickly across all four tiers, with most gains achieved within the first 1–3 epochs, indicating that the model rapidly learns effective representations for coarse-grained retrieval.

On applying ColSNAP on Nemotron ColEmbed 4B backbone, most of the improvement at compressed tiers is obtained within the first half epoch where the L1 tier reaches $59.36$ , within $0.3$ nDCG points of its final score ( $59.66$ ), while L2 and L3 achieve $83\%$ and $80\%$ of their total improvement over the untrained baseline, respectively. Further training mainly improves the coarser tiers, reaching final (epoch 5) scores of $56.94$ and $56.10$ , while L0 remains stable throughout (varying by less than $0.5$ nDCG points).

ColSNAP on ColPali v1.3 backbone follows a similar trend, converging even faster at finer tiers. After one epoch, L1 and L2 already match their final five-epoch performance, while L3 reaches $33.36$ , within $0.8$ nDCG points of its final score ( $34.08$ ). These results show that ColSNAP can efficiently adapt existing late-interaction retrievers into multi-granularity models with only a short post-training phase.

Backbone	Level	MB/1k	GFLOPs	nDCG@10
ColSNAP-Nemotron ColEmbed 4B-hard negs	L0	5314.56	187.07	61.6
	L1	727.04	25.59	60.0
	L2	153.60	5.41	57.7
	L3	76.80	2.70	57.0
ColSNAP-ColPali v1.3-hard negs	L0	263.94	8.89	47.1
	L1	34.56	1.16	44.7
	L2	5.89	0.20	39.9
	L3	2.05	0.07	36.0

Table 3: Index size (MB per

1{,}000

pages) and query-time scoring cost (GFLOPs per query) per tier, for each backbone trained with ColSNAP with mined hard negatives, alongside retrieval quality (nDCG@10, %) on ViDoRe v3. Documents are embedded in bf16 precision; GFLOPs assume a mean query length of

35.2

tokens for Nemotron and

33.7

for ColPali (ViDoRe v3 FinanceEn).

5.4 ColSNAP Reduces Storage and Retrieval Cost

Table 3 summarizes the storage, scoring cost, and retrieval quality trade-offs for each backbone and tier on a 1,000-page corpus from ViDoRe V3 with 309 queries from the finance_en split of ViDoRe V3. On Nemotron, the L1 tier provides the best balance, reducing index size and per-query scoring cost by $7.3\times$ while retaining $96.9\%$ of full-resolution retrieval quality. The coarser tiers deliver further reductions in storage and scoring cost, with the most aggressive setting still retaining over $92\%$ of full-resolution quality.

ColPali exhibits a similar trend: the finer tiers achieve substantial efficiency gains with negligible impact on retrieval quality, with both L0 and L1 exceeding the untrained baseline. Quality degradation emerges only at the most aggressive compression levels, reflecting the limited capacity of highly compressed representations. Overall, ColSNAP enables significant reductions in retrieval overhead while preserving strong retrieval performance across backbones.

6 Conclusion

We presented ColSNAP, a training recipe that turns a pretrained late-interaction retrieval model’s own patch grid into a nested hierarchy of retrieval granularities, without adding tokens or changing the encoder architecture. It gives a single retriever a range of vector cardinalities to choose from at deployment time, letting users trade compression for accuracy on demand rather than committing to one fixed representation.

On a strong retrieval model, Nemotron ColEmbed 4B, we retained $96.9\%$ of full-resolution retrieval quality at a $7.3\times$ reduction in stored embeddings, and still held $93.1\%$ and $92.0\%$ at the more extreme $34.6\times$ and $69.2\times$ reductions. On a smaller ColPali v1.3 model, the same recipe retained lossless performance at a $7.6\times$ compression, while holding $89.4\%$ and $80.7\%$ at $44.8\times$ and $128.9\times$ . In both cases, most of this benefit is available from a short, targeted training run, making ColSNAP a practical option as a lightweight adaptation step, one that, with minimal additional training, turns an existing late-interaction retriever into a multi-granularity one whose compression level is chosen at index time rather than fixed in advance.

References

Beyer et al. (2024) L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Grycner, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, R. Kabra, M. Bauer, M. Bošnjak, X. Chen, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Papalampidi, O. Henaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai PaliGemma: a versatile 3b vlm for transfer. External Links: 2407.07726, Link Cited by: §4.1.
Cai et al. (2024) M. Cai, J. Yang, J. Gao, and Y. J. Lee Matryoshka multimodal models. External Links: 2405.17430, Link Cited by: §2.
Cha et al. (2026) S. Cha, D. Kim, M. Kim, Y. Han, B. Jeon, and S. Lee ReinPool: reinforcement learning pooling multi-vector embeddings for retrieval system. External Links: 2601.07125, Link Cited by: §2.
Clavié et al. (2024) B. Clavié, A. Chaffin, and G. Adams Reducing the footprint of multi-vector retrieval with minimal performance impact via token pooling. External Links: 2409.14683, Link Cited by: §2.
de Souza P. Moreira et al. (2026) G. de Souza P. Moreira, R. Ak, M. Xu, O. Holworthy, B. Schifferer, Z. Yu, Y. Babakhin, R. Osmulski, J. Cai, R. Chesler, B. Liu, and E. Oldridge Nemotron colembed v2: top-performing late interaction embedding models for visual document retrieval. External Links: 2602.03992, Link Cited by: 3rd item, §1, §1, §2, §4.1.
Dhulipala et al. (2026) L. Dhulipala, M. Hadian, R. Jayaram, J. Lee, and V. Mirrokni MUVERA: multi-vector retrieval via fixed dimensional encodings. External Links: 2405.19504, Link Cited by: §2.
Enevoldsen et al. (2025) K. Enevoldsen, I. Chung, I. Kerboua, M. Kardos, A. Mathur, D. Stap, J. Gala, W. Siblini, D. Krzemiński, G. I. Winata, S. Sturua, S. Utpala, M. Ciancone, M. Schaeffer, G. Sequeira, D. Misra, S. Dhakal, J. Rystrøm, R. Solomatin, Ö. Çağatan, A. Kundu, M. Bernstorff, S. Xiao, A. Sukhlecha, B. Pahwa, R. Poświata, K. K. GV, S. Ashraf, D. Auras, B. Plüster, J. P. Harries, L. Magne, I. Mohr, M. Hendriksen, D. Zhu, H. Gisserot-Boukhlef, T. Aarsen, J. Kostkan, K. Wojtasik, T. Lee, M. Šuppa, C. Zhang, R. Rocca, M. Hamdy, A. Michail, J. Yang, M. Faysse, A. Vatolin, N. Thakur, M. Dey, D. Vasani, P. Chitale, S. Tedeschi, N. Tai, A. Snegirev, M. Günther, M. Xia, W. Shi, X. H. Lù, J. Clive, G. Krishnakumar, A. Maksimova, S. Wehrli, M. Tikhonova, H. Panchal, A. Abramov, M. Ostendorff, Z. Liu, S. Clematide, L. J. Miranda, A. Fenogenova, G. Song, R. B. Safi, W. Li, A. Borghini, F. Cassano, H. Su, J. Lin, H. Yen, L. Hansen, S. Hooker, C. Xiao, V. Adlakha, O. Weller, S. Reddy, and N. Muennighoff MMTEB: massive multilingual text embedding benchmark. External Links: 2502.13595, Link Cited by: §4.3.
Faysse et al. (2025) M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P. Colombo ColPali: efficient document retrieval with vision language models. External Links: 2407.01449, Link Cited by: 3rd item, 4th item, §1, §1, §1, §2, §4.1, §4.2, §4.3.
Günther et al. (2025) M. Günther, S. Sturua, M. K. Akram, I. Mohr, A. Ungureanu, B. Wang, S. Eslami, S. Martens, M. Werk, N. Wang, and H. Xiao Jina-embeddings-v4: universal embeddings for multimodal multilingual retrieval. External Links: 2506.18902, Link Cited by: §1, §2, §5.1.
Hegghammer (2022) T. Hegghammer OCR with Tesseract, Amazon Textract, and Google Document AI: a benchmarking experiment. Journal of Computational Social Science 5 (1), pp. 861–882. External Links: ISSN 2432-2725, Link, Document Cited by: §1.
Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §4.2.
Khattab and Zaharia (2020) O. Khattab and M. Zaharia ColBERT: efficient and effective passage search via contextualized late interaction over bert. External Links: 2004.12832, Link Cited by: §1, §2, §3.2.
Kusupati et al. (2024) A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, and A. Farhadi Matryoshka representation learning. External Links: 2205.13147, Link Cited by: §2.
Lee et al. (2024) J. Lee, Z. Dai, S. M. K. Duddu, T. Lei, I. Naim, M. Chang, and V. Y. Zhao Rethinking the role of token retrieval in multi-vector retrieval. External Links: 2304.01982, Link Cited by: §2.
Li et al. (2023) M. Li, S. Lin, B. Oguz, A. Ghoshal, J. Lin, Y. Mehdad, W. Yih, and X. Chen CITADEL: conditional token interaction via dynamic lexical routing for efficient and effective multi-vector retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 11891–11907. External Links: Link, Document Cited by: §2.
Liu et al. (2026) Z. Liu, Z. Hu, Y. Zhang, and Y. Xiao Structural anchor pruning: training-free multi-vector compression for visual document retrieval. External Links: 2601.20107, Link Cited by: §2.
Loison et al. (2026) A. Loison, Q. Macé, A. Edy, V. Xing, T. Balough, G. Moreira, B. Liu, M. Faysse, C. Hudelot, and G. Viaud ViDoRe v3: a comprehensive evaluation of retrieval augmented generation in complex real-world scenarios. External Links: 2601.08620, Link Cited by: 4th item, §4.3.
Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. External Links: 1711.05101, Link Cited by: §4.2.
Ma et al. (2024) X. Ma, S. Lin, M. Li, W. Chen, and J. Lin Unifying multimodal retrieval via document screenshot embedding. External Links: 2406.11251, Link Cited by: §1.
Ma et al. (2025) Y. Ma, J. Li, Y. Zang, X. Wu, X. Dong, P. Zhang, Y. Cao, H. Duan, J. Wang, Y. Cao, and A. Sun Towards storage-efficient visual document retrieval: an empirical study on reducing patch-level embeddings. External Links: 2506.04997, Link Cited by: §2.
Macé et al. (2025) Q. Macé, A. Loison, and M. Faysse ViDoRe benchmark v2: raising the bar for visual retrieval. External Links: 2505.17166, Link Cited by: 4th item, §4.3.
Meng et al. (2025) R. Meng, Z. Jiang, Y. Liu, M. Su, X. Yang, Y. Fu, C. Qin, Z. Chen, R. Xu, C. Xiong, Y. Zhou, W. Chen, and S. Yavuz VLM2Vec-v2: advancing multimodal embedding for videos, images, and visual documents. External Links: 2507.04590, Link Cited by: §1.
Muennighoff et al. (2023) N. Muennighoff, N. Tazi, L. Magne, and N. Reimers MTEB: massive text embedding benchmark. External Links: 2210.07316, Link Cited by: §4.3.
Nomic AI (2025) Nomic AI Colpali-queries-mined-20250321-by-source. Hugging Face. Note: https://huggingface.co/datasets/nomic-ai/colpali-queries-mined-20250321-by-sourceDataset Cited by: §4.2.
Park et al. (2026) S. Park, J. Jung, J. Park, and J. Kang Do all visual tokens matter equally? object-evidence preserving token merging for vision-language retrieval. External Links: 2607.04605, Link Cited by: §2.
Santhanam et al. (2022a) K. Santhanam, O. Khattab, C. Potts, and M. Zaharia PLAID: an efficient engine for late interaction retrieval. External Links: 2205.09707, Link Cited by: §2.
Santhanam et al. (2022b) K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia ColBERTv2: effective and efficient retrieval via lightweight late interaction. External Links: 2112.01488, Link Cited by: §2.
Team (2025) N. Team Nomic embed multimodal: interleaved text, image, and screenshots for visual document retrieval. Nomic AI. External Links: Link Cited by: §5.1.
Veneroso et al. (2025) J. Veneroso, R. Jayaram, J. Rao, G. H. Ábrego, M. Hadian, and D. Cer CRISP: clustering multi-vector representations for denoising and pruning. External Links: 2505.11471, Link Cited by: §2.
Weller et al. (2026) O. Weller, M. Boratko, I. Naim, and J. Lee On the theoretical limitations of embedding-based retrieval. External Links: 2508.21038, Link Cited by: §1.
Xiang et al. (2026) H. Xiang, Y. Yan, J. Huo, Y. Huang, Y. Cao, M. Ou, and X. Hu MM-matryoshka: towards budget-elastic visual document retrieval via a 2d multimodal matryoshka training framework. External Links: 2606.07654, Link Cited by: §2.
Xiao et al. (2026) Z. Xiao, Q. Ma, M. Gu, C. J. Chen, X. Chen, V. Ordonez, and V. Mohan MetaEmbed: scaling multimodal retrieval at test-time with flexible late interaction. External Links: 2509.18095, Link Cited by: §2, §2.
Yan et al. (2026a) Y. Yan, J. Huo, G. Feng, M. Ou, Y. Cao, X. Zou, S. Liu, Y. Lyu, Y. Huang, J. Li, K. Zheng, X. Zheng, P. S. Yu, J. Kwok, and X. Hu Unlocking multimodal document intelligence: from current triumphs to future frontiers of visual document retrieval. External Links: 2602.19961, Link Cited by: §1.
Yan et al. (2026b) Y. Yan, M. Ou, Y. Cao, J. Huo, X. Zou, S. Liu, J. Kwok, and X. Hu Visual late chunking: an empirical study of contextual chunking for efficient visual document retrieval. External Links: 2604.10167, Link Cited by: §2.
Yan et al. (2026c) Y. Yan, M. Ou, Y. Cao, X. Zou, J. Huo, S. Liu, J. Kwok, and X. Hu Sculpting the vector space: towards efficient multi-vector visual document retrieval via prune-then-merge framework. External Links: 2602.19549, Link Cited by: §2.
Yan et al. (2025) Y. Yan, G. Xu, X. Zou, S. Liu, J. Kwok, and X. Hu DocPruner: a storage-efficient framework for multi-vector visual document retrieval via adaptive patch-level embedding pruning. External Links: 2509.23883, Link Cited by: §2.
Zhang et al. (2025) J. Zhang, Q. Zhang, B. Wang, L. Ouyang, Z. Wen, Y. Li, K. Chow, C. He, and W. Zhang OCR hinders rag: evaluating the cascading impact of ocr on retrieval-augmented generation. External Links: 2412.02592, Link Cited by: §1.