Optimal VC Dimension of Contrastive Learning with Margin

Dionysis Arvanitakis Affiliation: Northwestern University Vaggos Chatziafratis Affiliation: UC Santa Cruz Yiyuan Luo Affiliation: UC Santa Cruz Konstantin Makarychev Affiliation: Northwestern University

Abstract

Contrastive learning is a successful paradigm for learning $d$ -dimensional geometric representations from a collection of “anchor–positive–negative” triplets $(i,j^{+},k^{-})$ , indicating that “item $i$ is closer to $j$ than to $k$ .” Despite its success, understanding why contrastive learning leads to representations of high generalization quality—beyond the often pessimistic predictions from PAC-learning—remains a central question. Recently, Alon, Avdiukhin, Elboim, Fischer and Yaroslavtsev (2024) proved that, for PAC-learning $d$ -dimensional Euclidean representations of $n$ -point datasets, $\Theta(\min(nd,n^{2}))$ triplets are necessary and sufficient, while they posed as an open question whether their VC dimension bounds for the more realistic setting of contrastive learning with a margin can be improved. For a margin parameter $\alpha>0$ , a triplet $(i,j^{+},k^{-})_{\alpha}$ is satisfied by the embedding $\phi:[n]\rightarrow\mathbb{R}^{d}$ , if $\|\phi(i)-\phi(k)\|_{2}>(1+\alpha)\cdot\|\phi(i)-\phi(j)\|_{2}$ . In this work, we resolve their question by proving that the VC dimension of contrastive learning under any margin $\alpha\in(0,1)$ is in fact $O(n/\alpha^{2})$ , improving on the previous bound of $O(n\log(n)/\alpha^{2})$ . We also establish that the bounds are optimal up to constant factors, by providing a matching lower bound of $\Omega(\frac{n}{\alpha^{2}})$ (the previously known lower bound was $\Omega(\frac{n}{\alpha})$ ), for $\alpha\geq\max(n^{-1/2},d^{-1/2})$ .

1 Introduction

Contrastive learning is a powerful paradigm in representation learning, where the goal is to find meaningful representations, encoded as features in $d$ -dimensional Euclidean space, allowing us to replace data point $x$ by its feature vector $\phi(x)\in\mathbb{R}^{d}$ in various downstream tasks. At the heart of contrastive learning lies the notion of a triplet comparison $(i,j^{+},k^{-})$ indicating that “item $i$ is closer to $j$ than to $k$ ,” also known as “anchor–positive–negative” (Schroff et al., 2015). For example, modern pipelines are trained with a contrastive loss, e.g., Triplet or InfoNCE (with one or more negatives), mapping data into $d$ -dimensional vectors by pulling anchor-positive pairs closer, while pushing anchor-negative pairs further apart, enabling a unified approach for diverse types of data, including text, audio, graph-based tasks, and images (Schroff et al., 2015; Grover and Leskovec, 2016; Oord et al., 2018; Saunshi et al., 2019; Chen et al., 2020).

Unfortunately, despite its empirical success, many theoretical aspects of contrastive learning are not well-understood. In this work, we focus on sample complexity and generalization bounds—a central question in PAC-learning—making progress towards understanding when a representation learned from anchor–positive–negative samples generalizes well to unseen (test) samples.

1.1 Sample Complexity of Contrastive Learning

Following the framework by Alon, Avdiukhin, Elboim, Fischer and Yaroslavtsev (2024), let $\mathcal{X}$ be an $n$ -point dataset and let $\phi:\mathcal{X}\to\mathbb{R}^{d}$ be an embedding that maps points into $d$ -dimensional Euclidean space (we do not make any assumptions on $\phi$ ). Motivated by the anchor–positive–negative paradigm, typically we assume access to a collection of $m$ triplets; these are usually generated from data, such as nearby words in a document, or neighboring video frames, that serve as the anchor-positive pair, and then a random other data point plays the role of the negative example. For more on how to pick negative samples, and their role in downstream performance, please refer to Saunshi et al. (2019); Robinson et al. (2020); Awasthi et al. (2022).

Definition 1.1.

We say that the embedding $\phi$ satisfies or is consistent with input triplet $(i,j^{+},k^{-})$ if:

||\phi(i)-\phi(k)||_{2}>||\phi(i)-\phi(j)||_{2}.

More generally, it is common to enforce a small margin in the above distance comparison (Vankadara et al., 2023), i.e., for a margin parameter $\alpha>0$ , the embedding $\phi$ satisfies the triplet $(i,j^{+},k^{-})_{\alpha}$ if:

||\phi(i)-\phi(k)||_{2}>(1+\alpha)||\phi(i)-\phi(j)||_{2}.

The basic question motivating our work concerns the sample complexity of contrastive learning: for a set of $n$ items, what is the smallest number $m^{*}(\varepsilon,\delta)$ of anchor–positive–negative samples required to PAC-learn an underlying ground-truth embedding $\phi(x)\in\mathbb{R}^{d}$ ?

In particular, we focus on the following problem in PAC-learning studied by Alon, Avdiukhin, Elboim, Fischer and Yaroslavtsev (2024):

Problem 1.2 (PAC-Learning an embedding $\phi(\cdot)$ from triplets Alon et al. (2024)).

Fix error parameters $\varepsilon,\delta>0$ . Given i.i.d. samples $(i,j,k)$ drawn from a distribution $\mathcal{D}$ over $[n]^{3}$ , and labeled according to embedding $\phi$ as either $(i,j^{+},k^{-})$ or $(i,k^{+},j^{-})$ , based on whether $||\phi(i)-\phi(j)||_{2}\lessgtr||\phi(i)-\phi(k)||_{2}$ , the goal is to find an embedding $\hat{\phi}$ ¹¹ 1 In the contrastive learning with margin setting studied below, learning might require an improper predictor. that generalizes well to unseen triplets, i.e., with probability $1-\delta$ , for a subsequent sample $(i,j,k)$ sampled from $\mathcal{D}$ :

\mathbb{P}_{(i,j,k)\sim\mathcal{D}}[\text{ $\hat{\phi}$ disagrees with $\phi$ on $(i,j,k)$}]\leq\varepsilon.

In other words, for a test triplet query $(i,j,k)$ , the learned embedding $\hat{\phi}$ must classify whether the anchor $i$ is closer to $j$ or to $k$ according to the (unknown) $\phi$ , i.e., whether the triplet is $(i,j^{+},k^{-})$ or $(i,k^{+},j^{-})$ . Similarly, Problem 1.2 can be studied in the agnostic case (where samples are not necessarily consistent with some embedding), or with quadruplet samples $(i,j,k,l)$ indicating whether $||\phi(i)-\phi(j)||_{2}\lessgtr||\phi(k)-\phi(l)||_{2}$ from the literature of ordinal embeddings (Bilu and Linial, 2005; Alon et al., 2008; Terada and Luxburg, 2014; Vankadara et al., 2023).

Dimensionality-vs-Generalization.

Recently, Alon, Avdiukhin, Elboim, Fischer and Yaroslavtsev (Alon et al., 2024) showed that the VC-dimension of the learning task in Problem 1.2 is $O(\min(nd,n^{2}))$ , i.e., they proved that $m^{*}(\varepsilon,\delta)=O_{\varepsilon,\delta}(\min(nd,n^{2}))$ labeled triplets are sufficient²² 2 The bound is $O(nd)$ , since $d\leq n$ without affecting pairwise distance comparisons (Bilu and Linial, 2005). for PAC-learning the embedding $\phi:\mathcal{X}\rightarrow\mathbb{R}^{d}$ . Typically, $\phi$ stems from the output layer of a neural network, so they focused on the representation dimensionality $d$ , a key parameter of interest often ranging in the thousands Radford et al. (2018); Kusupati et al. (2022); Comanici et al. (2025).

1.2 Our Contribution: Generalization from $O(n)$ samples, under any margin

For the special case when the margin $\alpha=0$ between distances $||\phi(i)-\phi(j)||_{2},||\phi(i)-\phi(k)||_{2}$ , Alon et al. (2024) managed to show a matching lower bound of $\Omega(\min(nd,n^{2}))$ for Problem 1.2. However, it is unclear what happens for the well-motivated case of contrastive learning with margin $\alpha>0$ . Observe that $\alpha>0$ provides some gap between distances, so the dimension $d$ can be directly replaced by $O(\log n/\alpha^{2})$ using standard tools, e.g., Johnson-Lindenstrauss (JL) lemma: as noted in (Alon et al., 2024), JL yields $d=O(\log n/\varepsilon^{2})$ , where $\varepsilon$ is the distortion (Dasgupta and Gupta, 2003), so $O(\tfrac{n\log n}{\alpha^{2}})$ triplets suffice for generalization (set distortion $\varepsilon\approx\alpha$ ). Improving this bound was raised as an open question by Alon et al. (2024):

For contrastive learning with large enough values of margin $\alpha>0$ ,
can the VC-dimension bound be improved beyond $O(\tfrac{n\log n}{\alpha^{2}})$ ?

Our Main Result.

We go beyond the above approach based on JL, and we obtain optimal $O(n)$ bounds in the VC dimension, thus resolving the above question for all values of margin $\alpha>0$ (not necessarily large values as originally stated in Alon et al. (2024)).

Our main result is a stronger, linear in $n$ VC-bound of $O(\max\left\{n/\alpha^{2},n\right\})$ for contrastive learning. This is coupled with a matching lower bound of $\Omega(\frac{n}{\alpha^{2}})$ (for $\alpha=\Omega(\max\left\{n^{-1/2},d^{-1/2}\right\})$ ), so when put together the two results give a tight characterization of the VC dimension in this setting. To make things formal, we need the following definition related to the notion of partial concept classes from Alon et al. (2022):

Definition 1.3 (Concept class $\mathcal{H}(n,d,\alpha)$ for embeddings with margin).

Fix $\alpha>0$ , and let $\phi:[n]\to\mathbb{R}^{d}$ be an embedding. Associated with embedding $\phi(\cdot)$ is its triplet classifier $h_{\phi}:[n]^{3}\to\left\{+1,-1,\star\right\}$ , labeling³³ 3 Symbol $\star$ is arbitrary, meaning classifiers may not answer certain queries (e.g., if distances are not separated). an input triplet as follows:

h_{\phi}(i,j,k)=\begin{cases}+1,&\text{if }\|\phi(i)-\phi(k)\|_{2}>(1+\alpha)\|\phi(i)-\phi(j)\|_{2},\\[6.0pt] -1,&\text{if }\|\phi(i)-\phi(j)\|_{2}>(1+\alpha)\|\phi(i)-\phi(k)\|_{2},\\[6.0pt] \phantom{-}\star,&\text{otherwise.}\end{cases}

The concept class containing all possible triplet classifiers is denoted⁴⁴ 4 For the case of no margin, we simply write $\mathcal{H}(n,d)$ . Note that for embeddings $\phi$ , as in the definition, $\mathcal{H}(n,d)$ is a total concept class. as $\mathcal{H}(n,d,\alpha)$ . We only consider embeddings such that for every $i,j,k$ with $j\neq k$ , $\|\phi(i)-\phi(j)\|\neq\|\phi(i)-\phi(k)\|$ .

Theorem 1.4 (Linear VC-dimension for any margin).

The VC-dimension of $\mathcal{H}(n,d,\alpha)$ is $O(\frac{n}{\alpha^{2}})$ for $\alpha\in(0,1)$ and $O(n)$ for $\alpha\geq 1$ . Specifically, there exists absolute constant $C$ such that:

\displaystyle\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))\leq C\cdot\max\bigl\{\frac{n}{\alpha^{2}},n\bigr\}.

Furthermore, assuming that $\alpha\geq\max(\frac{1}{\sqrt{n}},\frac{1}{\sqrt{d}})$ , the VC dimension is $\Omega(\frac{n}{\alpha^{2}})$ , for $\alpha\leq 1$ and $\Omega(n)$ for $\alpha>1$ . In particular there exists an absolute constant $c$ such that:

\displaystyle\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))\geq c\cdot\max\{\frac{n}{\alpha^{2}},n\}.

We prove the upper bound of Theorem 1.4 in Section 3, the proof of the lower bound follows the construction for linear classification with margin and is proven in Section 4.

A few remarks are in order. First, our result characterizes the VC-dimension of concept class $\mathcal{H}(n,d,\alpha)$ in the regime where $\alpha\geq\max(n^{-1/2},d^{-1/2})$ as $\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))=\Theta(\max(n,\frac{n}{\alpha^{2}}))$ . Second, it is surprising that under a margin $\alpha$ , dimensionality does not play a role: the larger the margin, the easier the learning task becomes, since the sought-after embedding $\phi$ is less refined, and a priori, one might expect that dimension $d$ plays a role on the samples required to avoid misclassification of test samples (perhaps this is why Alon et al. (2024) originally asked the question for large margins as a simplification).

Learning Bounds via Geometric Approximation.

An interesting aspect in our analysis that allows us to obtain the $O(n)$ VC-bound, is the use of geometric approximation algorithms aiming to satisfy as many of the input triplets as possible. In contrast to the approach taken in Alon et al. (2024), where they used combinatorial bounds on the number of sign-patterns of a system of polynomials, here we essentially reduce the $d$ -dimensional learning problem to a $1$ -dimensional setting, whose sample complexity we can bound. On the one hand, we exploit the margin $\alpha>0$ to show that a simple one-dimensional projection satisfies $\tfrac{1}{2}+\Omega(\alpha)$ fraction of triplets, independently of dimension $d$ , where $\tfrac{1}{2}$ -approximation is a trivial baseline achieved by a random embedding. On the other hand, we use the probabilistic method to show that no set of size $m=\Omega(\frac{n}{\varepsilon^{2}})$ can be shattered on the line $d=1$ , and more precisely, there exists a labeling of the triplet samples such that no embedding satisfies more than $\frac{1}{2}+\varepsilon$ of the triplets; since $\varepsilon$ can be taken to be sufficiently smaller than $\alpha$ this leads to a contradiction, giving us the upper bound of Theorem 1.4. Our approach is similar in spirit to the algorithm of Blum (2005) (see also Balcan and Blum (2005)) for linear classification with margin which combines a random projection coupled with boosting. For formal definitions and a detailed technical overview see Section 2 below.

1.3 Further Related Work

There are intense efforts to understand various aspects of contrastive learning, both from a theoretical and a practical perspective. For example, many well-known works study relations between transfer learning, inductive biases and contrastive learning (Saunshi et al., 2019; Saunshi et al., 2022), posing specific assumptions on latent classes or how contrastive triplets are generated, in order to try to explain the observed success of such methods (Schroff et al., 2015; Oord et al., 2018; Chen et al., 2020). Other works focus more on optimization loss functions such as spectral contrastive loss with good downstream performance (HaoChen et al., 2021), understanding the role of negative samples (Robinson et al., 2020; Awasthi et al., 2022; Lei et al., 2023), adversarial training (Zou and Liu, 2023) and more. Here, the work closest to ours is Alon et al. (2024), which served as the main inspiration towards our sample complexity results.

Moreover, the literature of non-metric embeddings, often called ordinal embeddings or monotone maps (Bilu and Linial, 2005; Alon et al., 2008; Terada and Luxburg, 2014; Emamjomeh-Zadeh and Kempe, 2018; Ghosh et al., 2019; Ghoshdastidar et al., 2019; Fan et al., 2020; Avdiukhin et al., 2023; Chatziafratis and Indyk, 2024; Avdiukhin et al., 2024), is also closely related to our setting, as their main algorithmic task is, given a collection of comparisons between pairwise distances, to find an embedding into a suitable geometric space, e.g., tree metric, ultrametric, or Euclidean space, so that as many of the given pairwise comparisons are satisfied. We note that, because humans are much faster at comparisons between objects, in contrast to providing numerical scores, such triplet comparisons are also widely-deployed in metric learning and crowdsourcing (Schultz and Joachims, 2003; Tamuz et al., 2011; Kulis, 2013). Other types of comparisons include the quadruplet setting $(i,j,k,l)$ mentioned earlier, where comparisons are of the form “item $i$ is closer to $j$ , than item $k$ is to $l$ ,” see survey Vankadara et al. (2023).

Finally, there are several results on the computational complexity and approximation algorithms literature associated with the empirical risk minimizer on a given sample. Given $m$ (non-contradictory) triplet constraints on $n$ items, Bilu and Linial (2005) show that $n-1$ dimensions always suffice to satisfy them, while $\Omega(n)$ are necessary. In Avdiukhin et al. (2024), they present an algorithm that achieves dimension $\sqrt{m}$ and further improve it via the notion of graph arboricity. For the case of non-realizable instances, recent work by Arvanitakis et al. (2026) showed that it is computational intractable to obtain a better than $\tfrac{1}{2}$ -approximation (assuming the Unique Games Conjecture (Khot, 2002)), showing that the problem of maximizing satisfied triplets is approximation resistant (Hstad, 2001; Hast, 2005; Guruswami et al., 2008).

Regarding more practical approaches based on local search, including gradient-based methods, it is known that in worst-case, finding even a locally optimum solution is intractable, as captured by PLS-completeness⁵⁵ 5 Both PLS and CLS are complexity classes related to finding local optima via local search methods, see Johnson, Papadimitriou and Yannakakis (1988); Daskalakis and Papadimitriou (2011). for discrete settings, and CLS-completeness for continuous embedding settings (Yan et al., 2026). However, it is reported in Vankadara et al. (2023), that popular methods usually obtain near-zero training error for a variety of triplet loss objectives (see also (Bower et al., 2018) on spurious local optima vs global optima).

2 Technical Overview

We present the main proof strategy and technical ideas behind our improved $O(n/\alpha^{2})$ bound on the VC-dimension for the concept class $\mathcal{H}(n,d,\alpha)$ of triplet classifiers with margin.

2.1 Our approach for $\alpha>0$ : Algorithmic Advantage over Random

To give some intuition and set notation, we first describe the approach taken in Alon et al. (2024) for the case of zero margin, and why it fails in our case when margin $\alpha>0$ .

Sign-Patterns of Polynomials vs. margin $\alpha>0$ .

Given that each triplet $(i,j^{+},k^{-})$ imposes a geometric constraint on how the representation $\phi$ embeds the $n$ data points, a first attempt would be to analyze how $\phi$ partitions $d$ -dimensional Euclidean space, and count the regions wherein triplet classifiers do not switch their label. Indeed, Alon et al. (2024) observed that each triplet is satisfied, if and only if the following polynomial $Q_{(i,j,k)}$ on $3\cdot d$ number of variables, and of degree $2$ is positive:

Q_{(i,j,k)}(\phi(i),\phi(j),\phi(k))=\|\phi(i)-\phi(k)\|_{2}^{2}-\|\phi(i)-\phi(j)\|_{2}^{2}

In total, this leads to $m$ such polynomials (as there are $m$ input triplets), with the total number of variables being $n\cdot d$ . Then, Alon et al. (2024) used known results from algebraic combinatorics to upper bound the different sign-patterns of the resulting system $\left\{Q_{(i_{t},j_{t},k_{t})}\right\}_{t=1}^{m}$ of $m$ polynomials on $n\cdot d$ variables (Warren, 1968; Alon et al., 1985). This limits the size of a set that can be shattered giving an upper bound on the VC-dimension (see definitions below).

While the above approach worked for the case of no margin $\alpha=0$ , and yielded the bound $\Theta(nd)$ , a similar approach for $\alpha>0$ would still yield the same (dimension-dependent) bound $\Theta(nd)$ : notice that if instead we defined the polynomial

Q^{\prime}_{(i,j,k)}(\phi(i),\phi(j),\phi(k))=\|\phi(i)-\phi(k)\|_{2}^{2}-(1+\alpha)^{2}\|\phi(i)-\phi(j)\|_{2}^{2},

the number of sign-patterns does not change, as those depend solely on $m$ , the degree of the polynomials and the number of variables.

We take a different approach to exploit the margin through geometric approximation algorithms that estimate triplet labels induced by $\phi(i),\phi(j),\phi(k)\in\mathbb{R}^{d}$ . On the one hand, the margin allows a $1$ -dimensional random projection algorithm to get slightly better-than-random approximations, specifically satisfying $\approx\tfrac{1}{2}+\Omega(\alpha)$ fraction of the triplets. This is crucially independent of what the dimension $d$ is, and is the reason why our final bound in Theorem 1.4 does not have a dependence on $d$ . On the other hand, if the VC-dimension was substantially larger than $O(n)$ , and the labels of the training set were chosen at random to be either $+1$ or $-1$ , then no embedding could fit the labels of the input triplets, better than random chance, which leads to a contradiction.

2.2 Rounding-vs-Noisy Labels in 1D

To make the above precise, we remind the reader of the standard definitions of VC-dimension and Shattering of a set (Shalev-Shwartz and Ben-David, 2014), adapted to our setting. The appropriate framework for our results is that of Alon et al. (2022), where hypothesis functions can be undefined on certain inputs (this corresponds to the asterisk case in Definition 1.3).

Definition 2.1 (Shattering).

We say that a set $S=\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ is shattered by $\mathcal{H}(n,d,\alpha)$ if for every $y:[m]\to\left\{-1,1\right\}$ there exists a concept function $h_{\phi}\in\mathcal{H}(n,d,\alpha)$ such that for every $t\in[m]$ , $h_{\phi}(i_{t},j_{t},k_{t})=y(t)$ .

Notice that in the definition, the concept is not allowed to output the special symbol $\star$ on any of the points. We can now give the definition of the VC dimension of the hypothesis class $\mathcal{H}(n,d,\alpha)$ .

Definition 2.2 (VC-dimension).

The VC-dimension of hypothesis class $\mathcal{H}$ is the largest number $m$ such that there exists a set of size $m$ that is shattered. We will use $\mathrm{VCdim}(\mathcal{H})$ to denote the VC dimension of hypothesis class $\mathcal{H}$ .

We directly apply the definition of the VC dimension and shattering to prove our main theorem. What we need to show is that for $m\approx\frac{n}{\alpha^{2}}$ , no set of $m$ unlabeled triplets $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ is shattered. Equivalently, starting with $m$ arbitrary triplets $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ (one can think of them as being selected by an adversary) we need to show that there is a labeling of the triplets such that no embedding can satisfy all of them. We let $\varepsilon=\Theta(\alpha)$ be a sufficiently small constant fraction of $\alpha$ and let $m=\frac{n}{\varepsilon^{2}}=\Theta(\frac{n}{\alpha^{2}})$ : we show that for a set $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ of unlabeled triplets there exists a labeling (i.e., every unlabeled triplet is labeled as either $(i_{t},j^{+}_{t},k_{t}^{-})_{\alpha}$ or $(i_{t},k_{t}^{+},j_{t}^{-})_{\alpha}$ ) such that no embedding $\phi:[n]\to\mathbb{R}^{n}$ is consistent with all the labeled triplets.

Rounding to 1D.

Our proof, perhaps surprisingly, begins by an observation drawing ideas from rounding semidefinite programs in the approximation algorithms literature. Suppose that we have a set of triplet constraints with margin $\alpha$ , $\left\{(i_{t},j_{t}^{+},k_{t}^{-})_{\alpha}\right\}_{t=1}^{m}$ and an embedding $\phi:[n]\to\mathbb{R}^{n}$ that satisfies all the constraints. Then, as we show, one can round the embedding $\phi$ to a one-dimensional embedding $\hat{\phi}:[n]\to\mathbb{R}$ so that a fraction of at least $\frac{1}{2}+\Omega({\alpha})$ of the triplet constraints without margin $\left\{(i_{t},j_{t}^{+},k_{t}^{-})\right\}_{t=1}^{m}$ are satisfied by $\hat{\phi}$ . A priori this result seems weak: not only the constraints that $\hat{\phi}$ satisfies are without margin, but also only a fraction of $\frac{1}{2}+\Omega(\alpha)$ of them are satisfied.⁶⁶ 6 For comparison, a trivial one-dimensional embedding that maps every item to a uniform number in $[0,1]$ satisfies half of the constraints in expectation. The fact, however, that there always exists such an embedding in one dimension that is strictly better than the random solution ends up being strong enough to give us the result. Our rounding algorithm projects all the embedded vectors along a random direction in a way similar to the MAX-CUT rounding algorithm of Goemans and Williamson (1995), see Lemma 3.2 for the formal statement and proof.

No 1D embedding can fit noise on $n/\varepsilon^{2}$ triplets.

We now go back to our setting where we are given a set of $m=\frac{n}{\varepsilon^{2}}$ unlabeled triplets, adversarially selected, and we want to show that there is a labeling of the triplets such that no embedding, even in $n$ dimensions, can satisfy all of them. The key idea is that our rounding algorithm allows us to focus on the real line and forget about the margin parameter. Recall that Alon et al. (2024) show that the VC-dimension of embeddings in dimension $d$ is $\Theta(nd)$ , and so for $d=1$ , we get that the VC dimension is $\approx n$ . What this means in particular, is that no embedding on the real line is able to fit $\approx\frac{n}{\varepsilon^{2}}$ triplets labeled by random noise: if we select the labels of a set of $m={n}/{\varepsilon^{2}}$ triplets randomly then with constant probability, no embedding will be able to satisfy more than $\frac{1}{2}+\varepsilon$ of them. This is formalized by the notion of Rademacher complexity and its bound in terms of the VC-dimension of a hypothesis class (Shalev-Shwartz and Ben-David, 2014; Bousquet et al., 2003). Using the probabilistic method, we get that for every set of $m=\frac{n}{\varepsilon^{2}}$ unlabeled triplets, there exists a labeling such that no embedding on the real line satisfies more than $(\frac{1}{2}+\varepsilon)m$ of them. The formal statement and proof can be found in Lemma 3.4.

2.3 Putting things together

We now discuss how to combine the ideas from the above paragraphs to get the result. Let $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ be the set of adversarially selected triplets and suppose, for the sake of contradiction, that the set is shattered by the hypothesis class $\mathcal{H}(n,d,\alpha)$ . On the one hand, we get that since $m$ is large enough and linear in $n$ , there exists a labeling of the triplets, such that no embedding on the real line (no $h\in\mathcal{H}(n,1)$ ) satisfies more than a fraction of $\frac{1}{2}+\varepsilon$ of them. Consider that labeling for the triplets. By our assumption that the set is shattered there exists an embedding (potentially in $n$ dimensions) such that all the triplets (with margin) are satisfied. Applying our rounding algorithm on that embedding we get that there exists an embedding on the real line satisfying a fraction of $\frac{1}{2}+\Omega(\alpha)$ of them (without margin). This leads to a contradiction that:

\displaystyle\frac{1}{2}+\Omega(\alpha)\leq\frac{1}{2}+\varepsilon,

yet we can choose $\varepsilon=\Theta(\alpha)$ to be sufficiently small compared to $\alpha$ . The above are made formal in our proof of Theorem 3.1 in Section 3.

3 Improved VC-dimension: Upper bound of Theorem 1.4

In this section, we prove the following theorem.

Theorem 3.1 (Upper bound for $\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))$ ).

The VC-dimension of $\mathcal{H}(n,d,\alpha)$ is linear in $n$ and independent of the dimensionality $d$ . In particular, there exists an absolute constant $C$ such that:

\displaystyle\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))\leq C\max\left\{n,\frac{n}{\alpha^{2}}\right\}.

Note that the VC dimension is monotonically decreasing in $\alpha$ , thus it suffices to show that for $\alpha\in(0,1]$ it is bounded by $C\frac{n}{\alpha^{2}}$ . From now on we assume that $\alpha\in(0,1]$ . Our proof uses the following lemma which, roughly speaking, states that if a set of contrastive learning constraints with margin $\alpha$ , $\left\{(i_{t},j_{t}^{+},k_{t}^{-})_{\alpha}\right\}_{t=1}^{m}$ are satisfiable (potentially in $n$ dimensions) then there exists an embedding on the real line satisfying a $\frac{1}{2}+\Omega(\alpha)$ fraction of the contrastive constraints $\left\{(i_{t},j_{t}^{+},k_{t}^{-})\right\}_{t=1}^{m}$ . Note here that the constraints that $\hat{\phi}$ satisfies are without margin.

Lemma 3.2 (Rounding to 1D).

Let $\phi:[n]\to\mathbb{R}^{n}$ be such that the constraints $\left\{(i_{t},j_{t}^{+},k_{t}^{-})_{\alpha}\right\}_{t=1}^{m}$ are satisfied by $\phi$ . Then there exists a $\hat{\phi}:[n]\to\mathbb{R}$ such that at least $\left(\frac{1}{2}+\frac{\alpha}{5}\right)\cdot m$ constraints of $\left\{(i_{t},j_{t}^{+},k_{t}^{-})\right\}_{t=1}^{m}$ are satisfied by $\hat{\phi}$ .

Proof.

Let $g\sim\mathcal{N}(0,I_{n})$ be a standard Gaussian vector. We will consider the random embedding $\hat{\phi}_{g}:[n]\to\mathbb{R}$ given by $\hat{\phi}(i)=\left\langle\phi(i),g\right\rangle$ and show that on expectation, the embedding satisfies a fraction of $\frac{1}{2}+\frac{\alpha}{5}$ of constraints in $\left\{(i_{t},j_{t}^{+},k_{t}^{-})\right\}_{t=1}^{m}$ . Note that by virtue of the fact that $\|\phi(i)-\phi(k)\|\neq\|\phi(i)-\phi(j)\|$ for every $i,j,k$ with $j\neq k$ , we get that, the with probability one, $|\hat{\phi}_{g}(i)-\hat{\phi}_{g}(j)|\neq|\hat{\phi}_{g}(i)-\hat{\phi}_{g}(k)|$ . The above combined imply that there exists a $v$ such that the embedding $\hat{\phi}$ given by $\hat{\phi}(i)=\left\langle\phi(i),v\right\rangle$ satisfies a fraction of at least $\frac{1}{2}+\frac{\alpha}{5}$ of constraints and furthermore $h_{\hat{\phi}}\in\mathcal{H}(n,1)$ .

For $i\in[n]$ let $\bar{i}=\phi(i)$ . Observe that the embedding $\hat{\phi}$ satisfies a constraint $(i,j^{+},k^{-})$ if:

\left|\left\langle\bar{i}-\bar{k},g\right\rangle\right|>\left|\left\langle\bar{i}-\bar{j},g\right\rangle\right|.

By linearity of expectation, the expected number of constraints that $\hat{\phi}_{g}$ satisfies is

\sum_{t=1}^{m}\ProbOp\left(\left|\left\langle\bar{i}_{t}-\bar{k}_{t},g\right\rangle\right|>\left|\left\langle\bar{i}_{t}-\bar{j}_{t},g\right\rangle\right|\right),

where the probability is over the randomness in $g$ . It therefore suffices to show that for every $t\in[m]$ :

\displaystyle\ProbOp\left(\left|\left\langle\bar{i}_{t}-\bar{k}_{t},g\right\rangle\right|>\left|\left\langle\bar{i}_{t}-\bar{j}_{t},g\right\rangle\right|\right)\geq\frac{1}{2}+\frac{\alpha}{5},

where for vectors $u=\bar{i}_{t}-\bar{k}_{t}$ and $v=\bar{i}_{t}-\bar{j}_{t}$ it holds that $\|u\|>(1+\alpha)\|v\|.$ To that end, we prove the following claim.

Claim 3.3.

Let $u,v\in\mathbb{R}^{d}$ and let $g$ be a standard Gaussian vector. If $\|u\|>(1+\alpha)\|v\|$ for some $\alpha>0$ , then

\ProbOp\left(|\langle u,g\rangle|>|\langle v,g\rangle|\right)\;\geq\;\frac{2}{\pi}\arctan(1+\alpha).

In particular, for $\alpha\in[0,1]$ :

\displaystyle\ProbOp\left(|\langle u,g\rangle|>|\langle v,g\rangle|\right)\;\geq\;\frac{1}{2}+\frac{\alpha}{5}.

Claim 3.3, which we prove in Appendix A, gives us the result.⁷⁷ 7 Observe this claim is for two vectors $u,v$ as long as they have a gap $\alpha$ , so it also allows us to capture the case of quadruplet comparisons $(i,j,k,l)$ (Bilu and Linial, 2005; Vankadara et al., 2023), where we compare the distance between $i,j$ to the distance between $k,l$ . ∎

In the next lemma we use the upper bound on the VC dimension of contrastive learning on the real line, proven in Alon et al. (2024), to prove that for every set of $\approx\frac{n}{\varepsilon^{2}}$ unlabeled triplets $(i_{t},j_{t},k_{t})$ , there exists a labeling of them such that no embedding is consistent with more than a fraction of $\frac{1}{2}+\varepsilon$ of them.

Lemma 3.4.

There exists an absolute constant $c$ such that for every $\varepsilon>0$ , the following holds. For every $m\geq\frac{c}{\varepsilon^{2}}n$ and set $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ there exists a labeling $y:[m]\to\left\{-1,1\right\}$ such that no embedding $\phi:[n]\to\mathbb{R}$ satisfies more than a fraction of $\frac{1}{2}+\varepsilon$ of the geometric triplet constraints induced by the labeling. In particular, for every $h\in\mathcal{H}(n,1)$ it holds that:

\displaystyle\frac{1}{m}\sum_{t=1}^{m}\mathbf{1}\left\{h(i_{t},j_{t},k_{t})=y(t)\right\}\leq\frac{1}{2}+\varepsilon

Proof.

We will use the probabilistic method. In particular, we will pick the label $y$ randomly and show that

\displaystyle\mathbb{E}_{y}\left[\sup_{h\in\mathcal{H}(n,1)}\frac{1}{m}\sum_{t=1}^{m}\mathbf{1}\left\{h(i_{t},j_{t},k_{t})=y(t)\right\}\right]\leq\frac{1}{2}+\varepsilon,

(1)

using bounds for the worst case empirical Rademacher complexity of a hypothesis class in terms of the VC-dimension, and then the result directly follows. In particular, we let $y(t)=1$ with probability $1/2$ and $y(t)=-1$ with the remaining probability, independently of everything else. We can equivalently write Equation 1 as:

\displaystyle\mathbb{E}_{y}\left[\sup_{h\in\mathcal{H}(n,1)}\frac{1}{m}\sum_{t=1}^{m}h(i_{t},j_{t},k_{t})\cdot y(t)\right]\leq 2{\varepsilon},

(2)

and observe that the left hand side exactly corresponds to the empirical Rademacher complexity of hypothesis class $\mathcal{H}(n,1)$ on points $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ . We now use (see e.g. Bousquet et al. (2003)) that there exists an absolute constant $c^{\prime}$ such that the empirical Rademacher complexity of a hypothesis class $\mathcal{H}$ on $m$ points can be upper bounded in terms of the VC dimension of the class. In particular, if $\mathcal{R}_{m}$ is the worst-case⁸⁸ 8 Supremum over sets of size $m$ . empirical Rademacher complexity on $m$ points then

\displaystyle\mathcal{R}_{m}\leq c^{\prime}\sqrt{\frac{\textrm{VCdim}(\mathcal{H})}{m}}.

(3)

We apply this bound to the left hand side of Equation 2 and use that $\mathrm{VCdim}(\mathcal{H}(n,1))=O(n)$ to get that for $c$ being a sufficiently large constant:

\displaystyle\mathbb{E}_{y}\left[\sup_{h\in\mathcal{H}(n,1)}\frac{1}{m}\sum_{t=1}^{m}h(i_{t},j_{t},k_{t})\cdot y(t)\right]

\displaystyle\leq c^{\prime}\sqrt{\frac{c_{0}n\varepsilon^{2}}{cn}}\leq 2{\varepsilon},

which gives us the result. ∎

We can now prove our main Theorem 3.1.

Proof of Theorem 3.1.

Let $\varepsilon<\frac{\alpha}{5}$ and let $c$ be a large enough constant. We will show that no set of size $m\geq c\frac{n}{\varepsilon^{2}}=\Omega(\frac{n}{\alpha^{2}})$ can be shattered, which gives the result. Consider a set of triplets $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ and assume, for the sake of contradiction, that the set is shattered by $\mathcal{H}(n,d,\alpha)$ , meaning that for every labeling of the triplets, there is an embedding $\phi:[n]\to\mathbb{R}^{n}$ that is consistent with all the corresponding triplet constraints.

Formally, by Lemma 3.4, taking $c$ bigger than the constant in the lemma we get that there exists a labeling $y^{*}$ such that for every $h\in\mathcal{H}(n,1)$ :

\displaystyle\frac{1}{m}\sum_{t=1}^{m}\mathbf{1}\left\{h(i_{t},j_{t},k_{t})=y^{*}_{t}\right\}\leq\frac{1}{2}+\varepsilon.

(4)

On the other hand, by our assumption that the set $\left\{(i_{t},j_{t},k_{t})\right\}_{t=1}^{m}$ is shattered, we get that there exists an embedding $\phi$ satisfying all the corresponding constraints: if $y^{*}_{t}=1$ then $\|\phi(i_{t})-\phi(k_{t})\|>(1+\alpha)\|\phi(i_{t})-\phi(j_{t})\|$ , and if $y^{*}_{t}=-1$ then $\|\phi(i_{t})-\phi(j_{t})\|>(1+\alpha)\|\phi(i_{t})-\phi(k_{t})\|$ . Without loss of generality and for notational convenience, we can assume that the labeled triplets are $\left\{(i_{t},j_{t}^{+},k_{t}^{-})_{\alpha}\right\}_{t=1}^{m}$ ; by Lemma 3.2, there exists a one dimensional embedding $\hat{\phi}:[n]\to\mathbb{R}$ such that a fraction of at least $\frac{1}{2}+\frac{\alpha}{5}$ constraints in $\left\{(i_{t},j_{t}^{+},k_{t}^{-})\right\}$ are satisfied.⁹⁹ 9 These are constraints without margin. For the function $h_{\hat{\phi}}\in\mathcal{H}(n,1)$ this implies that:

\displaystyle\frac{1}{m}\sum_{t=1}^{m}\mathbf{1}\left\{h_{\hat{\phi}}(i_{t},j_{t},k_{t})=y^{*}_{t}\right\}\geq\frac{1}{2}+\frac{\alpha}{5}.

(5)

Combining Equations 4 and 5 we get that:

\displaystyle\frac{1}{2}+\frac{\alpha}{5}\leq\frac{1}{2}+\varepsilon,

which is a contradiction since we have taken $\varepsilon<\frac{\alpha}{5}.$ ∎

4 A Matching Lower Bound

In this section we prove a matching lower bound for the VC dimension of contrastive learning with margin parameter $\alpha$ . In particular we show that as long as $\alpha\geq\Omega(\max(\frac{1}{\sqrt{n}},\frac{1}{\sqrt{d}})),$ then $\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))\geq\Omega(\max\{\frac{n}{\alpha^{2}},n\})$ .

Theorem 4.1.

Suppose that $\alpha\geq\max(n^{-1/2},d^{-1/2})$ . Then, there exists an absolute constant $c$ such that:

\displaystyle\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))\geq c\frac{n}{\alpha^{2}}.

Proof.

We divide into two cases. We first consider the case where $\alpha\geq\frac{1}{4}$ , in which case we show that the VC dimension is $\Omega(n)$ . We make a distinction between the elements in $[n]$ . We use $i,j$ to denote the first two elements and $\left\{k_{t}\right\}_{t=1}^{n-2}$ for the rest. We consider the set of $\Theta(n)$ triplets given by $\left\{(i,j,k_{t})\right\}_{t=1}^{n-2}$ and show that they are shattered. For a labeling of the triplets, let $A\subseteq[n-2]$ be the $t\in[n]$ such that $(i,j,k_{t})$ is labeled as $(i,j^{+},k_{t}^{-})$ and $B=[n-2]\setminus A$ . We define the embedding to be $\phi(i)=0$ , $\phi(j)=1$ , furthermore for $t\in A$ we let $\phi(k_{t})=\kappa$ where $\kappa>1+\alpha$ and for $t\in B$ , $\phi(k_{t})=\tau$ where $\tau<\frac{1}{1+\alpha}$ . Consider $t\in A$ and the triplet $(i,j^{+},k_{t}^{-})_{\alpha}$ , we have that $|\phi(i)-\phi(k_{t})|=\kappa>(1+\alpha)|\phi(i)-\phi(j)|$ , and the triplet $(i,j^{+},k_{t}^{-})_{\alpha}$ is satisfied. On the other hand for $t\in B$ , we have that $|\phi(i)-\phi(j)|=\frac{1}{\tau}|\phi(i)-\phi(k_{t})|>(1+\alpha)|\phi(i)-\phi(k_{t})|$ and the triplet $(i,k_{t}^{+},j^{-})_{\alpha}$ is satisfied. In the above construction there exist $i,j,k$ with $j\neq k$ such that $\|\phi(i)-\phi(j)\|=\|\phi(i)-\phi(k)\|$ . A small perturbation of all points resolves this issue while maintaining the conclusion.

In the remaining case, we consider $\alpha<\frac{1}{4}$ . We proceed to describe an instance of $\approx\frac{n}{\alpha^{2}}$ triplets that is shattered. Let for convenience $\gamma=4\alpha$ . We first make a distinction between the elements in $[n]$ . We will use $k_{1},\ldots,k_{\frac{1}{\gamma^{2}}}$ to denote the first $\frac{1}{\gamma^{2}}$ elements of $[n]$ and $i_{t},j_{t}$ with $t\in\left[\frac{n-\frac{1}{\gamma^{2}}}{2}\right]$ for the remaining elements. The set of triplets that we propose is inspired by the construction of Alon et al. (2022) for the lower bound on the VC dimension of linear classifiers with margin.

We consider the set of triplets $\left\{(i_{t},j_{t},k_{l}):t\in\left[\frac{n-\frac{1}{\gamma^{2}}}{2}\right],l\in[\frac{1}{\gamma^{2}}]\right\}$ and show that the set is shattered. Note that the number of triplets is $\frac{1}{2\gamma^{2}}(n-\frac{1}{\gamma^{2}})=\Theta(\frac{n}{\alpha^{2}})$ (here, we have used that $\frac{1}{\alpha^{2}}\leq n$ ), thus showing that they can be shattered gives us the result. For every labeling of the triplets, the embedding for points $k_{l}$ will be the same. In particular, we will always map $k_{l}$ to $e_{l},$ the $l$ -th standard basis vector (here we have used that $\frac{1}{\alpha^{2}}\leq d$ ). Now consider a labeling of the triplets and observe that we can decompose the triplets as $\cup_{t}\left\{(i_{t},j_{t},k_{l})\right\}$ , where having fixed the mappings of $k_{l}$ , the instances are decoupled for different $t$ . It therefore suffices to show that for every labeling of triplets $\left\{(i,j,k_{l})\right\}$ there is an embedding of points $i$ and $j$ that satisfies all the triplets.

Consider a labeling of the triplets $\left\{(i,j,k_{l})\right\}_{l=1}^{\frac{1}{\gamma^{2}}}$ and let $A$ be the subset of $l$ in $[\frac{1}{\gamma^{2}}]$ such that the triplet $(i,j,k_{l})$ is labeled as $(i,k_{l}^{+},j^{-})_{\alpha}$ and $B=[\frac{1}{\gamma^{2}}]\setminus A$ . Let $w=\sum_{l\in A}e_{l}-\sum_{l\in B}e_{l}$ , $w$ is exactly the vector that defines the hyperplane separating the points $\left\{e_{l},l\in A\right\}$ from the points $\left\{e_{l},l\in B\right\}$ with margin $\gamma$ , see Proposition 17 in Alon et al. (2022). We let $\phi(i)=\gamma\cdot w$ . We also select $j$ to be a point at distance $\sqrt{2}$ from $\phi(i)$ . We now consider $l\in A$ , we have that

	$\displaystyle\\|\phi(i)-\phi(k_{l})\\|$	$\displaystyle=\\|\phi(i)-e_{l}\\|$
		$\displaystyle=\left((1-\gamma)^{2}+\gamma^{2}\left(\frac{1}{\gamma^{2}}-1\right)\right)^{\frac{1}{2}}$
		$\displaystyle=\sqrt{2-2\gamma}.$

This in turn means that:

	$\displaystyle\\|\phi(i)-\phi(j)\\|$	$\displaystyle=\sqrt{2}$
		$\displaystyle=\frac{1}{\sqrt{1-\gamma}}\cdot\\|\phi(i)-e_{l}\\|$
		$\displaystyle\geq(1+\frac{1}{2}\gamma)\cdot\\|\phi(i)-e_{l}\\|$
		$\displaystyle=(1+2\alpha)\cdot\\|\phi(i)-\phi(k_{l})\\|$

where we have used that the function $f(x)=\frac{1}{\sqrt{1-x}}$ is convex and $f^{\prime}(0)=\frac{1}{2}$ . We therefore get that $(i,k_{l}^{+},j^{-})_{\alpha}$ is satisfied by the embedding.

On the other hand, for $l\in B$ , we have that

	$\displaystyle\\|\phi(i)-\phi(k_{l})\\|$	$\displaystyle=\sqrt{1+\gamma}\cdot\sqrt{2}$
		$\displaystyle=\sqrt{1+\gamma}\cdot\\|\phi(i)-\phi(j)\\|$
		$\displaystyle\geq(1+\frac{\gamma}{4})\cdot\\|\phi(i)-\phi(j)\\|$
		$\displaystyle=(1+\alpha)\cdot\\|\phi(i)-\phi(j)\\|,$

meaning that the triplet $(i,j^{+},k_{l}^{-})_{\alpha}$ is satisfied. In the above construction there exist $i,j,k$ with $j\neq k$ such that $\|\phi(i)-\phi(j)\|=\|\phi(i)-\phi(k)\|$ . A small perturbation of all points resolves this issue while maintaining the conclusion.

∎

5 Experiments

To complement our theoretical bounds, we empirically explore how the generalization error $\hat{\varepsilon}$ is affected as we vary the dimension $d$ , the number of points $n$ , the margin $\alpha$ and the number $m$ of given triplet samples. We test both on synthetic data generated from a Gaussian distribution and on real datasets (CIFAR-100). Our goal is to test the dimension-independence of our $O(n/\alpha^{2})$ bound.

Experimental Setting.

For each configuration $(n,d,\alpha)$ , we sample $m$ training triplets from $[n]^{3}$ uniformly at random, reject any that fail the $(1+\alpha)$ -margin condition under a ground-truth metric $\phi^{*}$ , and fit an embedding $\hat{\phi}:[n]\to\mathbb{R}^{d}$ by minimizing the multiplicative-margin triplet loss

\mathcal{L}(\hat{\phi})\;=\;\frac{1}{m}\sum_{t=1}^{m}\left[(1+\alpha)^{2}\big\lVert\hat{\phi}(i_{t})-\hat{\phi}(j_{t})\big\rVert^{2}_{2}-\big\lVert\hat{\phi}(i_{t})-\hat{\phi}(k_{t})\big\rVert^{2}_{2}\right]_{+},

which is the natural surrogate for the $(1+\alpha)$ -separation defining $\mathcal{H}(n,d,\alpha)$ (see Schroff et al. (2015) and survey Vankadara et al. (2023)). We optimize with Adam, with learning rate $0.01$ and $1000$ steps, and report the empirical generalization error $\hat{\varepsilon}$ on $5{,}000$ held-out $(1+\alpha)$ -separated test triplets.

We perform experiments for two types of ground-truth embeddings. In the synthetic setting, $\phi^{*}$ is the undelying embedding of $n$ points drawn i.i.d. from $\mathcal{N}(0,I_{50})$ and the learner directly parameterizes the $n$ embedding rows in $\mathbb{R}^{d}$ . In the real-data setting, $\phi^{*}$ is the L2-normalized penultimate feature (this provides a vector in $\mathbb{R}^{512}$ ) of an ImageNet-pretrained ResNet18 (He et al., 2016) on $5{,}000$ CIFAR-100 test images (Krizhevsky, 2009), frozen during training, and the learner is a linear head $\mathbb{R}^{512}\to\mathbb{R}^{d}$ on top of these features.

Figure 1: Dimension-independence: empirical generalization error

\hat{\varepsilon}

against

m\alpha^{2}/n

\alpha=0.2

. (a) Synthetic setup with underlying embedding

\phi^{\ast}\in\mathbb{R}^{50}

. (b) CIFAR-100 / ResNet18 setup with a linear head over 512-dimensional features.

(a) Synthetic,

\alpha=0.2

(b) CIFAR-100 / ResNet18,

\alpha=0.2

Dimension-independence.

As Theorem 3.1 does not depend on the embedding dimension $d$ , for fixed $m\alpha^{2}/n$ , the empirical error should not change as we increase $d$ . To test this we fix $\alpha=0.2$ and vary $d\in\{16,64,256\}$ . On the synthetic side we keep the ground-truth ambient dimension at $50$ so that the data distribution itself is unchanged, and only the learner’s capacity changes. We sweep $n\in\{100,300,1000\}$ and $m\in\{100,300,10^{3},3\!\cdot\!10^{3},10^{4},3\!\cdot\!10^{4},10^{5}\}$ on the synthetic setup, and $n\in\{500,2000,5000\}$ and $m\in\{300,10^{3},3\!\cdot\!10^{3},10^{4},3\!\cdot\!10^{4},10^{5}\}$ on the real-data setup, over 5 random seeds. Figure 1 agrees with the theoretical predictions on both setups, as it shows the three curves corresponding to different dimensions $d=16,64,256$ (encoded by different colors) lie on top of each other within each $n$ -trajectory. Interestingly, the plots show that on real-data (Fig. 1 (b)), dimension-independence is quite prominent, since as $d$ varies, the error $\hat{\varepsilon}$ changes only slightly (notice the small values of $\hat{\varepsilon}$ on the $y$ -axis). Appendix B provides further experiments.

Conclusion

We resolved an open question from Alon et al. (2024) related to provable generalization guarantees and the VC-dimension of contrastive learning, in the presence of a margin parameter separating the distance between an anchor-positive pair from an anchor-negative pair. Enforcing a margin is not only a practically well-motivated approach in the literature of contrastive learning and ordinal embeddings, but as we show it also leads to interesting theoretical insights. Our main result improves upon the direct application of standard low-distortion metric embedding techniques, such as the Johnson-Lindenstrauss lemma, and provides an optimal $\Theta(\max\left\{n/\alpha^{2},n\right\})$ bound for the VC-dimension for learning a ground-truth embedding on $n$ data points of interest (with the lower bound holding under the assumption that $\alpha\geq\max\left\{n^{-1/2},d^{-1/2}\right\}$ ). Surprisingly, this is independent of the underlying dimensionality $d$ of the ground-truth embedding, which is often large. Given that our bound is optimal in $n$ , our work makes steps towards understanding the empirical success of contrastive learning. Technically, our approach uses ideas from approximation algorithms and geometric projections, coupled with learning theory bounds for learning triplet comparisons.

AI Disclosure

Generative AI tools were used for polishing and for feedback on exposition as well as literature search. In particular, the free version of ChatGPT helped the authors find a reference for the stronger bound on Rademacher complexity in terms of VC-dimension. The main result of the paper was obtained by the authors in April 2026. All the proofs and technical ideas come from the authors. Generative AI tools were used to assist with the experiments.

References

Alon et al. (2024) N. Alon, D. Avdiukhin, D. Elboim, O. Fischer, and G. Yaroslavtsev OPTIMAL sample complexity of contrastive learning. In 12th International Conference on Learning Representations, ICLR 2024, Cited by: §1.1, §1.1, §1.1, §1.2, §1.2, §1.2, §1.2, §1.3, Problem 1.2, §2.1, §2.1, §2.1, §2.2, §3, Conclusion, Abstract.
Alon et al. (2008) N. Alon, M. Bădoiu, E. D. Demaine, M. Farach-Colton, M. Hajiaghayi, and A. Sidiropoulos Ordinal embeddings of minimum relaxation: general properties, trees, and ultrametrics. ACM Transactions on Algorithms (TALG) 4 (4), pp. 1–21. Cited by: §1.1, §1.3.
Alon et al. (1985) N. Alon, P. Frankl, and V. Rodl Geometrical realization of set systems and probabilistic communication complexity. In 26th Annual Symposium on Foundations of Computer Science (sfcs 1985), pp. 277–280. Cited by: §2.1.
Alon et al. (2022) N. Alon, S. Hanneke, R. Holzman, and S. Moran A theory of pac learnability of partial concept classes. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pp. 658–671. Cited by: §1.2, §2.2, §4, §4.
Arvanitakis et al. (2026) D. Arvanitakis, V. Chatziafratis, and Y. Luo Provable accuracy collapse in embedding-based representations under dimensionality mismatch. 43rd International Conference on Machine Learning (ICML). Cited by: §1.3.
Avdiukhin et al. (2024) D. Avdiukhin, V. Chatziafratis, O. Fischer, and G. Yaroslavtsev Embedding dimension of contrastive learning and $k$ -nearest neighbors. Advances in Neural Information Processing Systems 37, pp. 41359–41393. Cited by: §1.3, §1.3.
Avdiukhin et al. (2023) D. Avdiukhin, G. Yaroslavtsev, D. Vainstein, O. Fischer, S. Das, and F. Mirza Tree learning: optimal sample complexity and algorithms. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, pp. 6701–6708. Cited by: §1.3.
Awasthi et al. (2022) P. Awasthi, N. Dikkala, and P. Kamath Do more negative samples necessarily hurt in contrastive learning?. In International conference on machine learning, pp. 1101–1116. Cited by: §1.1, §1.3.
Balcan and Blum (2005) M. Balcan and A. Blum A pac-style model for learning from labeled and unlabeled data. In International Conference on Computational Learning Theory, pp. 111–126. Cited by: §1.2.
Bilu and Linial (2005) Y. Bilu and N. Linial Monotone maps, sphericity and bounded second eigenvalue. Journal of Combinatorial Theory, Series B 95 (2), pp. 283–299. Cited by: §1.1, §1.3, §1.3, footnote 2, footnote 7.
Blum (2005) A. Blum Random projection, margins, kernels, and feature-selection. In International Statistical and Optimization Perspectives Workshop” Subspace, Latent Structure and Feature Selection”, pp. 52–68. Cited by: §1.2.
Bousquet et al. (2003) O. Bousquet, S. Boucheron, and G. Lugosi Introduction to statistical learning theory. In Summer school on machine learning, pp. 169–207. Cited by: §2.2, §3.
Bower et al. (2018) A. Bower, L. Jain, and L. Balzano The landscape of non-convex quadratic feasibility. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3974–3978. Cited by: §1.3.
Chatziafratis and Indyk (2024) V. Chatziafratis and P. Indyk Dimension-accuracy tradeoffs in contrastive embeddings for triplets, terminals & top-k nearest neighbors. In 2024 Symposium on Simplicity in Algorithms (SOSA), pp. 230–243. Cited by: §1.3.
Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. Cited by: §1.3, §1.
Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §1.1.
Dasgupta and Gupta (2003) S. Dasgupta and A. Gupta An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms 22 (1), pp. 60–65. Cited by: §1.2.
Daskalakis and Papadimitriou (2011) C. Daskalakis and C. Papadimitriou Continuous local search. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms, pp. 790–804. Cited by: footnote 5.
Emamjomeh-Zadeh and Kempe (2018) E. Emamjomeh-Zadeh and D. Kempe Adaptive hierarchical clustering using ordinal queries. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 415–429. Cited by: §1.3.
Fan et al. (2020) B. Fan, D. I. Centurion, N. Mohammadi, F. Sgherzi, A. Sidiropoulos, and M. Valizadeh Learning lines with ordinal constraints. arXiv preprint arXiv:2004.13202. Cited by: §1.3.
Ghosh et al. (2019) N. Ghosh, Y. Chen, and Y. Yue Landmark ordinal embedding. Advances in neural information processing systems 32. Cited by: §1.3.
Ghoshdastidar et al. (2019) D. Ghoshdastidar, M. Perrot, and U. von Luxburg Foundations of comparison-based hierarchical clustering. Advances in neural information processing systems 32. Cited by: §1.3.
Goemans and Williamson (1995) M. X. Goemans and D. P. Williamson Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM) 42 (6), pp. 1115–1145. Cited by: §2.2.
Grover and Leskovec (2016) A. Grover and J. Leskovec Node2vec: scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855–864. Cited by: §1.
Guruswami et al. (2008) V. Guruswami, R. Manokaran, and P. Raghavendra Beating the random ordering is hard: inapproximability of maximum acyclic subgraph. In 2008 49th Annual IEEE Symposium on Foundations of Computer Science, pp. 573–582. Cited by: §1.3.
HaoChen et al. (2021) J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in neural information processing systems 34, pp. 5000–5011. Cited by: §1.3.
Hast (2005) G. Hast Beating a random assignment. In Approximation, Randomization and Combinatorial Optimization. Algorithms and Techniques, pp. 134–145. Cited by: §1.3.
He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: §5.
Hstad (2001) J. Hstad Some optimal inapproximability results. Journal of the ACM (JACM) 48 (4), pp. 798–859. Cited by: §1.3.
Johnson et al. (1988) D. S. Johnson, C. H. Papadimitriou, and M. Yannakakis How easy is local search?. Journal of computer and system sciences 37 (1), pp. 79–100. Cited by: footnote 5.
Khot (2002) S. Khot On the power of unique 2-prover 1-round games. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pp. 767–775. Cited by: §1.3.
Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: §5.
Kulis (2013) B. Kulis Metric learning: a survey. Foundations and Trends® in Machine Learning 5 (4), pp. 287–364. Cited by: §1.3.
Kusupati et al. (2022) A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems 35, pp. 30233–30249. Cited by: §1.1.
Lei et al. (2023) Y. Lei, T. Yang, Y. Ying, and D. Zhou Generalization analysis for contrastive representation learning. In International Conference on Machine Learning, pp. 19200–19227. Cited by: §1.3.
Oord et al. (2018) A. v. d. Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §1.3, §1.
Radford et al. (2018) A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever Improving language understanding by generative pre-training. Technical report OpenAI. Cited by: §1.1.
Robinson et al. (2020) J. Robinson, C. Chuang, S. Sra, and S. Jegelka Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592. Cited by: §1.1, §1.3.
Saunshi et al. (2022) N. Saunshi, J. Ash, S. Goel, D. Misra, C. Zhang, S. Arora, S. Kakade, and A. Krishnamurthy Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning, pp. 19250–19286. Cited by: §1.3.
Saunshi et al. (2019) N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar A theoretical analysis of contrastive unsupervised representation learning. In International conference on machine learning, pp. 5628–5637. Cited by: §1.1, §1.3, §1.
Schroff et al. (2015) F. Schroff, D. Kalenichenko, and J. Philbin Facenet: a unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823. Cited by: §1.3, §1, §5.
Schultz and Joachims (2003) M. Schultz and T. Joachims Learning a distance metric from relative comparisons. Advances in neural information processing systems 16. Cited by: §1.3.
Shalev-Shwartz and Ben-David (2014) S. Shalev-Shwartz and S. Ben-David Understanding machine learning: from theory to algorithms. Cambridge university press. Cited by: §2.2, §2.2.
Tamuz et al. (2011) O. Tamuz, C. Liu, S. Belongie, O. Shamir, and A. T. Kalai Adaptively learning the crowd kernel. 28th International Conference on Machine Learning (ICML). Cited by: §1.3.
Terada and Luxburg (2014) Y. Terada and U. Luxburg Local ordinal embedding. In International Conference on Machine Learning, pp. 847–855. Cited by: §1.1, §1.3.
Vankadara et al. (2023) L. C. Vankadara, M. Lohaus, S. Haghiri, F. U. Wahab, and U. Von Luxburg Insights into ordinal embedding algorithms: a systematic evaluation. Journal of Machine Learning Research 24 (191), pp. 1–83. Cited by: §1.1, §1.3, §1.3, Definition 1.1, §5, footnote 7.
Warren (1968) H. E. Warren Lower bounds for approximation by nonlinear manifolds. Transactions of the American Mathematical Society 133 (1), pp. 167–178. Cited by: §2.1.
Yan et al. (2026) J. Yan, Y. Luo, V. Chatziafratis, I. Panageas, P. Shahkar, and S. Stavroulakis The complexity of finding local optima in contrastive learning. Advances in Neural Information Processing Systems 38, pp. 52758–52785. Cited by: §1.3.
Zou and Liu (2023) X. Zou and W. Liu Generalization bounds for adversarial contrastive learning. Journal of Machine Learning Research 24 (114), pp. 1–54. Cited by: §1.3.

Appendix A Reduction to 1D

In this section we prove Claim 3.3.

Proof of Claim 3.3.

If $v=0$ , the claim is immediate. Assume $v\neq 0$ , which also implies $u\neq 0$ . Let $\bar{u}=u/\|u\|$ and $\bar{v}=v/\|v\|$ . Define $\beta=1+\alpha$ . Then the inequality $|\langle u,g\rangle|>|\langle v,g\rangle|$ holds if and only if

|\langle\bar{u},g\rangle|>\frac{\|v\|}{\|u\|}\cdot|\langle\bar{v},g\rangle|.

Since $\|u\|>\beta\|v\|$ , it suffices to show

\ProbOp\left(|\langle\bar{v},g\rangle|<\beta|\langle\bar{u},g\rangle|\right)\;\geq\;\frac{2}{\pi}\arctan(\beta).

If $\bar{v}=\pm\bar{u}$ , then the event holds with probability $1$ , so we may assume that $\bar{u}$ and $\bar{v}$ are not collinear. Let $S$ be the two-dimensional subspace spanned by $\bar{u}$ and $\bar{v}$ . We decompose $g$ as

g=\tilde{g}+g^{\perp},

where $\tilde{g}$ is the orthogonal projection of $g$ onto $S$ . Since both $\bar{u}$ and $\bar{v}$ lie in $S$ , we have

\langle\bar{u},g\rangle=\langle\bar{u},\tilde{g}\rangle,\quad\langle\bar{v},g\rangle=\langle\bar{v},\tilde{g}\rangle.

Let $\theta\in(0,\pi)$ be the angle between $\bar{u}$ and $\bar{v}$ , measured counterclockwise, and let $\varphi$ be the angle from $\bar{u}$ to $\tilde{g}$ , also measured counterclockwise. To visualize the configuration, align the first basis vector $e_{1}$ of $S$ with $\bar{u}$ . Then $\bar{u}$ has coordinates $(1,0)$ , $\bar{v}$ has coordinates $(\cos\theta,\sin\theta)$ , and $\tilde{g}$ has coordinates $(\cos\varphi,\sin\varphi)$ . See Figure 2. We exclude the events $\tilde{g}=0$ and $\varphi\in\{\pi/2,3\pi/2\}$ , which occur with probability $0$ . Then $\varphi$ is uniformly distributed on

A=(-\pi/2,\pi/2)\cup(\pi/2,3\pi/2).

Figure 2: Left: vectors

\bar{u}

\bar{v}

, and

\tilde{g}

in the plane, with angles

\theta

and

\varphi

. Right: the function

F(\theta)

for

\beta=1.5

, illustrating that the minimum is attained at

\theta=\pi/2

We have

\langle\tilde{g},\bar{u}\rangle=\|\tilde{g}\|\cos\varphi,\quad\langle\tilde{g},\bar{v}\rangle=\|\tilde{g}\|\cos(\varphi-\theta).

Therefore,

|\langle\bar{v},g\rangle|<\beta|\langle\bar{u},g\rangle|\;\Longleftrightarrow\;|\cos(\varphi-\theta)|<\beta|\cos\varphi|.

Define

\mathcal{E}=\left\{\varphi\in A:|\cos(\varphi-\theta)|<\beta|\cos\varphi|\right\}.

Since $\cos\varphi\neq 0$ , this is equivalent to

\mathcal{E}=\Big\{\varphi\in A:-\beta<\frac{\cos(\varphi-\theta)}{\cos\varphi}<\beta\Big\}.

Using $\cos(\varphi-\theta)=\cos\varphi\cos\theta+\sin\varphi\sin\theta$ , we get

\mathcal{E}=\Big\{\varphi\in A:-\beta<\cos\theta+\tan(\varphi)\sin\theta<\beta\Big\}.

Since $\theta\in(0,\pi)$ , we have $\sin\theta>0$ , and hence

\mathcal{E}=\Big\{\varphi\in A:\frac{-\beta-\cos\theta}{\sin\theta}<\tan(\varphi)<\frac{\beta-\cos\theta}{\sin\theta}\Big\}.

Therefore, $\mathcal{E}$ consists of two intervals $I$ and $I+\pi$ , where

I=\Bigl(\;\arctan\Big(\frac{-\beta-\cos\theta}{\sin\theta}\Big),\arctan\Big(\frac{\beta-\cos\theta}{\sin\theta}\Big)\;\Bigr).

The probability of $\varphi\in\mathcal{E}$ equals

\frac{|\mathcal{E}|}{2\pi}=\frac{|I|}{\pi}=\frac{1}{\pi}\Big[\arctan\Big(\frac{\beta-\cos\theta}{\sin\theta}\Big)-\arctan\Big(\frac{-\beta-\cos\theta}{\sin\theta}\Big)\Big].

Denote the expression in the square brackets on the right hand side by $F(\theta)$ (see Figure 2):

F(\theta)=\arctan\Big(\frac{\beta-\cos\theta}{\sin\theta}\Big)-\arctan\Big(\frac{-\beta-\cos\theta}{\sin\theta}\Big).

Then the probability above is $F(\theta)/\pi$ . The derivative of $F(\theta)$ equals

F^{\prime}(\theta)=\frac{1-\beta\cos\theta}{1-2\beta\cos\theta+\beta^{2}}-\frac{1+\beta\cos\theta}{1+2\beta\cos\theta+\beta^{2}}.

Write it as

F^{\prime}(\theta)=G(-\beta\cos\theta)-G(\beta\cos\theta),\quad\text{where }G(x)=\frac{x+1}{2x+1+\beta^{2}}.

Function $G(x)$ is increasing for $x>-(\beta^{2}+1)/2$ because

G(x)=\frac{1}{2}\Bigl[1-\frac{\beta^{2}-1}{2x+1+\beta^{2}}\Bigr]

and $\beta^{2}-1>0$ . Observe that $\beta\cos\theta>-(\beta^{2}+1)/2$ and $-\beta\cos\theta>-(\beta^{2}+1)/2$ . Hence $F^{\prime}(\theta)<0$ when $\cos\theta>0$ and $F^{\prime}(\theta)>0$ when $\cos\theta<0$ . Therefore, $F$ is minimized at $\theta=\pi/2$ . Consequently,

F(\theta)\geq F(\pi/2)=\arctan(\beta)-\arctan(-\beta)=2\arctan(\beta).

Thus

\ProbOp\left(\varphi\in\mathcal{E}\right)=\frac{F(\theta)}{\pi}\geq\frac{2}{\pi}\arctan(\beta).

This proves the first part of the claim. We now use that the function $\arctan(x)$ is concave for $x\geq 0$ meaning that in any interval, it can be lower bounded by the secant line between the endpoints of the interval. In particular, we get that for $\alpha\in[0,1]$ :

	$\displaystyle\frac{2}{\pi}\arctan(\beta)$	$\displaystyle\geq\frac{2}{\pi}\arctan(1+\alpha)$
		$\displaystyle\geq\frac{2}{\pi}\left((\arctan(2)-\frac{\pi}{4})\alpha+\frac{\pi}{4}\right)$
		$\displaystyle=\frac{1}{2}+\frac{2}{\pi}\left(\arctan(2)-\frac{\pi}{4}\right)\alpha,$

where $\frac{2}{\pi}\left(\arctan(2)-\frac{\pi}{4}\right)\approx 0.204833\geq\frac{1}{5}$ . ∎

Appendix B Additional Experiments

In this section we provide the additional experiments referenced from Section 5. All experiments run on a standard CPU and can be completed within an hour.

B.1 Synthetic $\alpha$ -sweep

In Section 5, we fixed $\alpha$ and varied $d$ . Here we do the opposite: on the synthetic setup, we fix $d=50$ to match the ground-truth ambient dimension, and sweep $\alpha\in\{0.1,0.2,0.4\}$ , $n\in\{100,300,1000\}$ , $m\in\{100,300,10^{3},{3\!\cdot\!10^{3}},10^{4},{3\!\cdot\!10^{4}},10^{5}\}$ over 5 seeds. In Figure 3, we plot the empirical error $\hat{\varepsilon}$ against the effective sample size $m\alpha^{2}/n$ predicted by Theorem 3.1. All nine $(n,\alpha)$ curves collapse onto a single trajectory and drop from random-guess error to near zero once $m\alpha^{2}/n\gtrsim 1$ . If we plot against $m/n$ instead, the curves separate by margin, so the $\alpha^{2}$ factor in our obtained bound is quite precise.

Figure 3: Synthetic

\alpha

-sweep. Left: all nine

(n,\alpha)

curves collapse when plotted against the predicted effective sample size

m\alpha^{2}/n

. Right: without the

\alpha^{2}

factor the curves separate by margin

\alpha

B.2 Real data $\alpha$ -sweep

On the real-data setup, we fix $d=64$ , and sweep $\alpha\in\{0.1,0.2,0.4\}$ , $n\in\{500,2000,5000\}$ , $m\in\{300,10^{3},{3\!\cdot\!10^{3}},10^{4},{3\!\cdot\!10^{4}},10^{5}\}$ over 5 seeds. Figure 4 reproduces both diagnostics: plotting against $m\alpha^{2}/n$ orders the curves and they decay together to zero error, while removing $\alpha^{2}$ separates them by $\alpha$ . Compared to the synthetic data, two things look different: the empirical errors are well below their synthetic counterparts at the same $m\alpha^{2}/n$ , and at fixed $m\alpha^{2}/n$ larger $n$ gives lower rather than higher error. Both behaviors are what we would expect when the linear head explores a more structured subspace of $\mathcal{H}(n,d,\alpha)$ , rather than randomly generated data (it has only $O(d\cdot 512)$ parameters, independent of $n$ ). A smaller class generalizes better than our worst-case bound predicts, and at fixed $m\alpha^{2}/n$ , a larger value of $n$ means more triplets constraining the same fixed-size parameter matrix.

Figure 4: Real-data

\alpha

-sweep. Left: collapse against

m\alpha^{2}/n

. Right: with the

\alpha^{2}

factor removed.

B.3 Dimension-independence across $\alpha$

The dimension-independence experiment in Section 5 uses a single representative margin $\alpha=0.2$ . We repeat this experiment at $\alpha\in\{0.05,0.1,0.4\}$ on the synthetic and real-data setups (Figure 5). The three $d$ -curves overlap within each $n$ -trajectory across the full range in both cases. The one exception is mild underfitting at $d=16$ when $\alpha$ is small and $m\alpha^{2}/n$ is large: 16 dimensions cannot faithfully embed the ground truth ( $d^{\star}=50$ synthetic, up to 512 on CIFAR-100), and tight margins are more sensitive to the resulting distortion.

Figure 5: Dimension-independence across

\alpha

. Top: synthetic setup. Bottom: CIFAR-100 / ResNet18.

B.4 Margin sanity check

The hypothesis class $\mathcal{H}(n,d,\alpha)$ is defined by embeddings whose output itself satisfies the $(1+\alpha)$ -separation; the multiplicative-margin loss is a continuous surrogate for this constraint. As a post-hoc check that the learned $\hat{\phi}$ actually lives in the hypothesis class, we measure the fraction of held-out triplets on which the learned embedding satisfies $\lVert\hat{\phi}(i)-\hat{\phi}(k^{-})\rVert\geq(1+\alpha)\lVert\hat{\phi}(i)-\hat{\phi}(j^{+})\rVert$ (Figure 6). The fraction climbs from around $0.5$ at the smallest $m$ to above $0.95$ once $m\alpha^{2}/n\gtrsim 1$ . So whenever the empirical error is small, the learned embedding really is $(1+\alpha)$ -separated on most test triplets.

Figure 6: Margin sanity check. Once

m\alpha^{2}/n\gtrsim 1

, the learned embedding satisfies the hypothesis-class constraint on

>\!95\%

of test triplets.

	$\displaystyle\\|\phi(i)-\phi(j)\\|$	$\displaystyle=\sqrt{2}$
		$\displaystyle=\frac{1}{\sqrt{1-\gamma}}\cdot\\|\phi(i)-e_{l}\\|$
		$\displaystyle\geq(1+\frac{1}{2}\gamma)\cdot\\|\phi(i)-e_{l}\\|$
		$\displaystyle=(1+2\alpha)\cdot\\|\phi(i)-\phi(k_{l})\\|$

	$\displaystyle\\|\phi(i)-\phi(k_{l})\\|$	$\displaystyle=\sqrt{1+\gamma}\cdot\sqrt{2}$
		$\displaystyle=\sqrt{1+\gamma}\cdot\\|\phi(i)-\phi(j)\\|$
		$\displaystyle\geq(1+\frac{\gamma}{4})\cdot\\|\phi(i)-\phi(j)\\|$
		$\displaystyle=(1+\alpha)\cdot\\|\phi(i)-\phi(j)\\|,$

Optimal VC Dimension of Contrastive Learning with Margin

Abstract

1 Introduction

1.1 Sample Complexity of Contrastive Learning

Definition 1.1.

Problem 1.2 (PAC-Learning an embedding ϕ⁡(⋅)\phi(\cdot) from triplets Alon et al. (2024)).

Dimensionality-vs-Generalization.

1.2 Our Contribution: Generalization from O⁡(n)O(n) samples, under any margin

Our Main Result.

Definition 1.3 (Concept class ℋ⁡(n,d,α)\mathcal{H}(n,d,\alpha) for embeddings with margin).

Theorem 1.4 (Linear VC-dimension for any margin).

Learning Bounds via Geometric Approximation.

1.3 Further Related Work

2 Technical Overview

2.1 Our approach for α>0\alpha>0: Algorithmic Advantage over Random

Sign-Patterns of Polynomials vs. margin α>0\alpha>0.

2.2 Rounding-vs-Noisy Labels in 1D

Definition 2.1 (Shattering).

Definition 2.2 (VC-dimension).

Rounding to 1D.

No 1D embedding can fit noise on n/ε2n/\varepsilon^{2} triplets.

2.3 Putting things together

3 Improved VC-dimension: Upper bound of Theorem 1.4

Theorem 3.1 (Upper bound for VCdim⁡(ℋ⁡(n,d,α))\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))).

Lemma 3.2 (Rounding to 1D).

Proof.

Claim 3.3.

Lemma 3.4.

Proof.

Proof of Theorem 3.1.

4 A Matching Lower Bound

Theorem 4.1.

Proof.

5 Experiments

Experimental Setting.

Dimension-independence.

Conclusion

AI Disclosure

References

Appendix A Reduction to 1D

Proof of Claim 3.3.

Appendix B Additional Experiments

B.1 Synthetic α\alpha-sweep

B.2 Real data α\alpha-sweep

B.3 Dimension-independence across α\alpha

B.4 Margin sanity check

Problem 1.2 (PAC-Learning an embedding $\phi(\cdot)$ from triplets Alon et al. (2024)).

1.2 Our Contribution: Generalization from $O(n)$ samples, under any margin

Definition 1.3 (Concept class $\mathcal{H}(n,d,\alpha)$ for embeddings with margin).

2.1 Our approach for $\alpha>0$ : Algorithmic Advantage over Random

Sign-Patterns of Polynomials vs. margin $\alpha>0$ .

No 1D embedding can fit noise on $n/\varepsilon^{2}$ triplets.

Theorem 3.1 (Upper bound for $\mathrm{VCdim}(\mathcal{H}(n,d,\alpha))$ ).

B.1 Synthetic $\alpha$ -sweep

B.2 Real data $\alpha$ -sweep

B.3 Dimension-independence across $\alpha$