Guaranteed Adaptive Modality Acquisition:
When the Policy Chooses Its Own Calibration Group

Let calibration scores $s_{1},\dots,s_{n}$ and a test score $s_{\ast}$ be i.i.d. $U[0,1]$, and let $\tau$ be the $\lceil(1-\alpha)(n+1)\rceil$-th smallest calibration score. Route the test point into group $S$ exactly when $s_{\ast}>\tau$, and inside $S$ emit the answer set $\mathcal{C}=\{y:s(y)\leq\tau\}$. Conditional on $\{S_{\mathrm{fin}}=S\}$ we have $s_{\ast}>\tau$ by definition of the routing rule, so the target is excluded with probability one and $\Pr[\,t\in\mathcal{C}\mid S_{\mathrm{fin}}=S\,]=0$. Every other hypothesis of theorem 1 holds: the points are i.i.d., the models are fit on disjoint data, the score is a fixed measurable function, and the routing is a deterministic measurable map of the input and the calibration set. Since coverage is $0$ rather than merely below $1-\alpha$, the example rules out any nontrivial distribution-free guarantee under unrestricted calibration-dependent routing: no bound survives if the dependence is left unconstrained. It does not rule out approximate validity under an explicit quantitative restriction on that dependence, such as a bound on mutual information, a coupling or total-variation budget, algorithmic stability, or differential privacy could each buy a degraded but nonvacuous guarantee, and 7 is exactly such a route, bounding the loss by a computable coupling term that our construction drives to its maximum. The correct reading is therefore that arbitrary calibration dependence admits no distribution-free guarantee, so approximate validity requires the dependence to be quantified; theorem 2 takes the different route of removing the dependence from the certified object instead of bounding it. Two further remarks delimit the claim. First, it concerns worst cases over data laws, as distribution-free statements always do; a particular calibration-dependent policy may be harmless on a particular law. Second, it does not say routing must ignore all data; theorem 2 identifies which data it must ignore. The measured decay between the two regimes is table S3.

Let $D_{A}$ and $D_{B}$ be disjoint and let $\pi=\pi(\cdot\,;D_{A})$ be any measurable function of $D_{A}$. Condition on $D_{A}$: then $\pi$ is a fixed measurable map from the augmented pair $(x,\omega)$ to a terminal pattern, exactly the object 1 requires, and $D_{B}$ is independent of $D_{A}$ and identically distributed with the test point. So 1 holds verbatim with $D_{B}$ as the calibration set, theorems 1 and 2 give their conclusions conditionally on $D_{A}$, and since the bounds do not depend on the value of $D_{A}$, averaging preserves them. A $K$-fold version routes each test point by the policy fitted on $\bigcup_{j\neq k}D_{j}$ for an independently drawn $k$ and calibrates on $D_{k}$. This is standard, and it is what the frozen-confidence arm does. Its limits are equally standard: the policy is graded on data it never saw, so it can be learned but it still cannot consult the deployed guarantee, and every fold spent on routing is a fold not spent on calibration. Theorem 2 gives up neither.

Fix the grid $\Lambda$ and the pattern set $\mathcal{S}$ before looking at the calibration data, and let $d_{\lambda}$ be the acquisition-and-answer policy described in the main text. For each $\lambda$, $d_{\lambda}$ is a deterministic measurable map from $(x,\omega)$ to a purchase sequence, a terminal pattern, and an answer-or-decline decision; nothing about it depends on the calibration set. Consequently $R_{S}(\lambda)$, the selective risk among the cases $d_{\lambda}$ answers at terminal pattern $S$, is a fixed population quantity, one per pair $(\lambda,S)\in\Lambda\times\mathcal{S}$, and the null $H_{\lambda,S}:R_{S}(\lambda)>\alpha$ is a fixed hypothesis.

Let $n_{\lambda,S}$ and $k_{\lambda,S}$ be the number of calibration cases $d_{\lambda}$ answers at $S$ and how many it gets wrong. Because the calibration points are i.i.d. and $d_{\lambda}$ does not read them, these errors are binomial at rate $R_{S}(\lambda)$ conditional on $n_{\lambda,S}$, so $p_{\lambda,S}=\Pr[\mathrm{Bin}(n_{\lambda,S},\alpha)\leq k_{\lambda,S}]$ is super-uniform under $H_{\lambda,S}$. Apply any procedure controlling the family-wise error rate at $\delta$ over these hypotheses and let $\widehat{\mathcal{W}}$ be its rejected set. We use Holm (23), which is valid under arbitrary dependence. That matters: the independence established above is across patterns, whereas within one pattern the $p$-values for different $\lambda$ are computed from overlapping answered sets and are strongly dependent, so a step-up procedure such as Hochberg is not licensed over $\Lambda\times\mathcal{S}$ and we do not use one there. By definition of family-wise control, $\Pr[\widehat{\mathcal{W}}\text{ contains a true null}]\leq\delta$, which is precisely $\Pr[\,R_{S}(\lambda)\leq\alpha\ \forall(\lambda,S)\in\widehat{\mathcal{W}}\,]\geq 1-\delta$.

The deployment step is where the assumption is actually relaxed. Choose $\hat{\lambda}$ by any rule, including one that reads $\widehat{\mathcal{W}}$, the estimated risks, the answer rates, or the whole calibration set, and deploy $d_{\hat{\lambda}}$ with its answers restricted to the patterns certified for $\hat{\lambda}$. On the $1-\delta$ event every deployed pair satisfies $R_{S}(\lambda)\leq\alpha$ simultaneously, so whichever pair the selection lands on is covered. No independence between the selection and the calibration set is used, because the guarantee being transferred is uniform over the family rather than pointwise at one pair. That is exactly what fails in proposition 1: there the guarantee holds at the realized threshold marginally over calibration draws, and selecting on that same draw does not preserve it.

The family $\{d_{\lambda}\}$ of the main text carries one cutoff at every pattern, a poorer class than the per-pattern cutoffs a calibration-blind baseline fits, and that class, not the certificate, is what costs answer rate. A richer family is available at no cost in validity. Split the calibration half into a selection part and a disjoint certification part. On the selection part compute, for each mask $S$ and each level $\eta$ in a pre-registered grid $\mathcal{R}$, the most permissive cutoff whose empirical risk at that mask is at most $\eta$; call the resulting vector $\lambda(\eta)$. Each $d_{\eta}$ buys while confidence at the current pattern is below $\lambda_{S}(\eta)$ and answers when it clears, so the family $\{d_{\eta}:\eta\in\mathcal{R}\}$ is a deterministic function of the selection part alone. Conditional on that part it is pre-registered, so theorem 2 applies with the certification part as its calibration set, and averaging over the selection draw preserves the bound. We use $\mathcal{R}=\{0.01,\dots,0.20\}$ in steps of $0.01$ and Holm over $\mathcal{R}\times\mathcal{S}$.

Empirically this recovers most of what the single-cutoff family gives up: answered fraction rises from $0.371$ to $0.879$ on CMU-MOSEI, $0.487$ to $0.780$ on IEMOCAP, closing $85\%$ and $74\%$ and $89\%$ of the distance to calibration-blind routing, with no breach on any of the $50$ resplits of any benchmark. It does not overtake the blind baseline, which remains $6$–$11$ points ahead.

On MHEALTH the arm abstains outright, and the arithmetic of remark 1 says why. Splitting the calibration half again leaves a certification quarter; Holm’s leading threshold over $|\mathcal{R}|\times|\mathcal{S}|=20\times 8$ hypotheses is $\delta/160=6.25\times 10^{-4}$, and clearing it with no observed errors needs $n\geq\ln(6.25\times 10^{-4})/\ln(0.9)\approx 70$ answered cases in whichever pattern is rejected first. MHEALTH’s $174$ certification cases spread across seven initial patterns leave roughly $25$ apiece, so no pair is ever rejected and no $\eta$ is certified. MHEALTH survives on its larger pools only because its answers concentrate in fewer patterns. The binding constraint on calibration-aware routing is thus the size of the certification pool relative to the family, which is the same floor that governs how fine a calibration partition may be.

The certified quantity $R_{S}(\lambda)$ is the risk among the cases that $d_{\lambda}$ itself routes to $S$ and answers there. A deployment that, on reaching an uncertified pattern, keeps buying instead of declining is a different policy, say $d_{\lambda}^{\prime}$, and for a later pattern $S_{2}$ one has in general

so the certificate computed for $(\lambda,S_{2})$ under $d_{\lambda}$ says nothing about $d_{\lambda}^{\prime}$. A two-type example makes the gap concrete. Type A cases are confidently wrong at $S_{1}$, so $d_{\lambda}$ stops and answers there and $(\lambda,S_{1})$ is correctly refused certification; type B cases pass through $S_{1}$ to $S_{2}$, where they are always right, so $(\lambda,S_{2})$ certifies on a population of pure B. Redirect the type A cases onward and $S_{2}$ receives A${}+{}$B at deployment, with risk bounded only by the type A error rate. Nothing about the calibration computation detects this, because the calibration never saw the modified policy.

Declining is safe for the complementary reason: it changes no case’s purchase sequence, so every pattern receives exactly the population it was certified on, and restricting to a subset of answered pairs can only remove answered cases from patterns that are no longer used. The alternative, if one genuinely wants the continue-behavior, is to make it part of the policy: index candidates by $(\lambda,A)$ where $A$ is a pre-registered answer/continue/decline map, and control the family-wise error over $\Lambda\times\mathcal{A}\times\mathcal{S}$. That is valid for any $\mathcal{A}$ fixed in advance, at a multiplicity cost of $|\mathcal{A}|$; we do not pursue it, and the arm reported in section 6.5 declines.

Fix a group and suppose its validation half yields $n$ answered cases with $k$ errors. The deployed rule accepts iff the one-sided Clopper–Pearson upper confidence bound at level $1-\delta$, the $(1-\delta)$-quantile of $\mathrm{Beta}(k+1,n-k)$, is at most $\alpha$. That quantile is nondecreasing in $k$, so it is minimized at $k=0$, where $\mathrm{Beta}(1,n)$ has quantile function $1-(1-p)^{1/n}$ and the bound equals $1-\delta^{1/n}$. Acceptance therefore requires $1-\delta^{1/n}\leq\alpha$, i.e. $\delta^{1/n}\geq 1-\alpha$, i.e. $\tfrac{1}{n}\ln\delta\geq\ln(1-\alpha)$. Both logarithms are negative, so this rearranges to $n\geq\ln\delta/\ln(1-\alpha)$. At $\alpha=\delta=0.1$ the right-hand side is $\ln(0.1)/\ln(0.9)=21.85$, giving $n_{\min}=22$. The bound is attained: $n=22$ error-free answered cases accept. Nothing about the score, model, or data law enters, so a group whose answered validation count falls below $n_{\min}$ cannot deploy however good its predictor is, and a partition whose cells cannot be expected to clear the floor cannot be used. Selecting the finest partition that clears it on a held-out selection split is exactly the data-adaptive backoff of corollary S3, which is valid because the selection part is disjoint from the guarantee part.

Condition on the routing assignments (which fix every pattern count $n_{S}$) and on every pattern’s selection half (which fixes every selected cutoff $\lambda_{S}$, hence which patterns have true selective risk above $\alpha$, i.e. which nulls are true). Two facts follow. First, each pattern’s validation half is a fresh sample never used to choose its own cutoff, so under its null the exact one-sided binomial $p$-value $p_{S}=\Pr[\mathrm{Bin}(n_{S}^{\mathrm{ans}},\alpha)\leq k_{S}]$ is super-uniform, which is what any family-wise procedure requires. Second, the patterns partition the calibration set, so the validation halves are disjoint subsets of an i.i.d. sample and the $\{p_{S}\}$ are mutually independent under the conditioning. Holm’s step-down procedure controls the family-wise error rate at $\delta$ for super-uniform $p$-values under arbitrary dependence, and Hochberg’s step-up procedure does so under independence; both therefore apply conditionally, and since the bound $\delta$ does not depend on the conditioning variables, averaging preserves it.

Domination is immediate at a matched selection rule. Bonferroni deploys exactly the patterns with $p_{S}\leq\delta/|\mathcal{S}|$. If there are $k$ such patterns, the $k$ smallest $p$-values are all at most $\delta/|\mathcal{S}|$, and Holm’s thresholds $\delta/(|\mathcal{S}|-j)$ at steps $j=0,\dots,k-1$ are each at least $\delta/|\mathcal{S}|$, so Holm passes through all $k$ steps and deploys at least the same patterns; Hochberg, being a step-up procedure with the same threshold sequence, deploys a superset of Holm’s. Neither can therefore answer less than Bonferroni, and both can answer more.

The qualifier matters in practice, because “Bonferroni” names two different procedures here. One corrects only the test, comparing $p_{S}$ to $\delta/|\mathcal{S}|$ while choosing each candidate cutoff at $\delta$; the other, which the earlier version of this paper reported, also tightens selection, choosing the candidate cutoff at $\delta/|\mathcal{S}|$ as well. The second selects more conservative cutoffs that clear their test more easily, so it is not dominated by the step procedures and can answer more than they do. Table S9 reports both, and the domination above is the like-for-like comparison against the first.

For contrast we also implement a fixed-sequence arm: order the patterns by a routing-measurable statistic (decreasing pool size $n_{S}$), test each at the full level $\delta$, and stop at the first failure. That is also valid, since a family-wise error requires the first bad pattern in the order to pass its own single test, which happens with probability at most $\delta$, and it splits no level at all. It is nonetheless the weakest arm in table S9, because with patterns of comparable size the ordering carries almost no information about which will pass, so the sequence truncates after $0.1$–$0.9$ patterns on average and forfeits the rest. Spending the error budget adaptively, as Holm and Hochberg do, beats spending it in a fixed order.

The bound $\alpha/\Pr[\textsc{answer}\mid S]$ is weak on patterns that answer rarely (on CMU-MOSEI the text-missing pattern abstains entirely under the deployed rule, table S6). We therefore do not rely on the divided form in the risk-capped experiments. There, each per-pattern threshold is set by the split-select/validate rule of proposition 2: the pattern’s calibration cases are halved uniformly at random. A candidate confidence cutoff is chosen on the selection half (by the Clopper–Pearson scan, though any selector measurable in that half is allowed). The single chosen cutoff is then tested once on the held-out validation half with an exact one-sided Clopper–Pearson bound at confidence $1-\delta$, deployed iff the bound is $\leq\alpha$ and abstaining otherwise.

Patterns whose validation half cannot guarantee (a Clopper–Pearson bound at $\delta{=}0.1$ needs at least $22$ error-free answered cases to reach $\alpha{=}0.10$) abstain entirely; abstention on hard patterns is thus the correct output of risk control, not a loophole in it. The divided bound is the clean population statement, and the deployed calibration is the finite-sample tool. This rule controls the selective risk directly rather than through the coverage bound of theorem 1: it deploys the scalar cutoff $\lambda_{S}$ in place of the coverage quantile $\tau_{S}$ of 1, on the same threshold-free calibration split, so the exchangeability that licenses it is unchanged.

The statement and proof are for the population residual set $\mathcal{L}_{\infty}(x_{S})=\{c:\Pr_{z\sim q_{\phi}(\cdot\mid x_{S})}[\argmax_{y}g_{\theta}(x_{S},z)_{y}=k]>0\}$, the support of the completion-induced decision distribution; the deployed $\mathcal{L}$ is its $K$-sample estimate, and the transfer to the deployed loop is quantified below. Consider the exclusive-or construction with two acquirable sources $a$ and $b$, each a single bit, and label $y=a\oplus b$. Conditioned only on the observed sources at a state, $\mathcal{L}_{\infty}$ is the set of labels still consistent with what has been read. With neither $a$ nor $b$ observed, both labels remain possible, so $|\mathcal{L}_{\infty}|=2$. Observing $a$ alone leaves $y$ equal to either $a\oplus 0$ or $a\oplus 1$ as $b$ ranges over its two values, so both labels remain and $|\mathcal{L}_{\infty}|=2$; by symmetry observing $b$ alone also leaves $|\mathcal{L}_{\infty}|=2$. Observing the pair $\{a,b\}$ fixes $y=a\oplus b$, so $|\mathcal{L}_{\infty}|=1$. Call an input on which this holds a synergy-only input, and suppose such inputs occur with probability $p>0$. A greedy one-at-a-time policy buys a source only while some single source strictly shrinks $\mathcal{L}$. On a synergy-only input no single source shrinks $\mathcal{L}$, so the greedy policy makes no purchase and stops with $|\mathcal{L}|=2$, hence never resolves the input (never collapses $\mathcal{L}$ to a singleton) regardless of how it scores individual sources. Take the pair look-ahead policy to make the same purchases as the greedy policy on every input some single source can settle, and on the synergy-only inputs to evaluate the residual shrinkage of $\{a,b\}$, find it collapses $|\mathcal{L}|$ to $1$, and buy the pair at any budget $\geq 2$. By construction the two policies then agree off the synergy event, so the resolved fractions differ only on the synergy-only inputs, where the greedy policy resolves none and the look-ahead policy resolves all. At any budget $\geq 2$ the greedy policy therefore resolves a fraction smaller by at least $p>0$ (the mass of the synergy event). (Resolution, $|\mathcal{L}|{=}1$, is the Phase-1 screen; the Phase-2 guarantee then applies to the resolved cases as in the coverage and risk-control results.) A policy forced to exhaust its budget can reach the pair by exhaustion; the race in section 6.5 includes that forced variant as a control, and it still falls short.

Finite-$K$ transfer. The deployed loop computes the $K$-sample estimate $\mathcal{L}$, not $\mathcal{L}_{\infty}$, so with probability $2(1/2)^{K}=2^{1-K}$ per evaluation (independence of the $K$ completion draws) all $K$ draws of a balanced binary state land on one class and the sampled set is spuriously a singleton. That bounds a single evaluation, and a route makes $N_{\mathrm{eval}}$ of them (one per visited state, candidate group, and outer draw: at budget $B$ with $r_{\max}{=}2$, $N_{\mathrm{eval}}\leq B\,(1+D(M+\binom{M}{2}))$), and a union bound over the $N_{\mathrm{eval}}$ evaluations gives recovery of every population mode of mass $\geq\rho_{\mathrm{mode}}$ along the whole route with probability at least $1-N_{\mathrm{eval}}\,C(1-\rho_{\mathrm{mode}})^{K}$. At $K{=}15$, $C{=}2$, $\rho_{\mathrm{mode}}{=}1/2$, and $N_{\mathrm{eval}}\leq 200$ this failure probability is below $1.3\%$ per input, and the observed $100\%$ XOR resolution is consistent with it; the population statement of proposition S2 therefore transfers to the deployed loop up to this margin, and the calibration guarantees are unaffected either way, since calibration and test run the identical finite-$K$ procedure.