Ask Which, Not How Good:
Sizing Benchmarks Scored by an LLM

$E\rho^{2}$ is defined relative to a universe of admissible judges, and published work never states one. Recomputing the ceiling under four choices moves MT-Bench from $0.600$ (all nine) to $0.649$ (drop legacy), $0.691$ (drop weak) and $0.798$ (current-generation six); our screened set moves $0.746\to 0.804\to 0.840\to 0.929$, Arena-Hard $0.792\to 0.918$, AlpacaEval 2 $0.706\to 0.829$. The ordering is stable, the level is not, so a reliability figure reported without its judge universe is uninterpretable. Section 5.5 explains which one we take as primary.

The route replica serves the same weights as anth-cheap through a different provider endpoint, so the variance of their interaction difference isolates route. It is $0.0010$–$0.0074$ across the four item sets, or $1.4\%$–$5.0\%$ of the disagreement distinct current-generation models produce on the same systems. $\sigma^{2}_{sj}$ is about which model judges, not where it is served; a panel of one model behind several endpoints buys almost none of the independence a panel of distinct models buys.

Five open-weight judges on identical material disagree $2.17\times$ more across model families than within them once scale usage is standardised away ($2.92\times$ on raw scores, before standardisation, the difference is a scale-compression artefact and we report the standardised figure). Mixing families buys independence but not a smaller panel.

Published comparisons are between closely matched frontier systems, not across a 2024–2026 capability range. Restricted to ten current frontier models, system variance falls to $1.9\%$ on our screened items and $2.8\%$ on MT-Bench. Arena-Hard is the exception at $15.7\%$, consistent with its design intent and the one benchmark still separating frontier systems.

Our items were selected for between-system variance, so finding it afterwards would be guaranteed. Two checks. The screener (gpt-4.1) is deliberately outside the evaluation panel, and splitting that panel by vendor family shows the screened set lifts the system share $3.16\times$ over MT-Bench for OpenAI-family judges and $3.82\times$ for Anthropic-family, the gain is larger on the family least like the screener, the opposite of selection on one model’s taste. And ranking items by discrimination within random halves of the system panel and correlating the two rankings gives a split-half Spearman $\rho=0.757$ (95% range $0.413$–$0.893$), the highest of the four item sets, against $0.597$ for MT-Bench and $0.551$ for Arena-Hard. Screening captured a transferable property, not the screener’s preferences.

The 42 systems are 14 models $\times$ 3 prompt configurations, so configurations are nested in models rather than crossed. Fitting the nested model, configuration-given-model carries $55.7\%$ of system variance on MT-Bench, $46.7\%$ on AlpacaEval 2, $26.1\%$ on Arena-Hard and $0\%$ on our screened items. Treating the model as the object of measurement lowers MT-Bench’s ceiling from $0.600$ to $0.404$ and AlpacaEval 2’s from $0.706$ to $0.567$. Whether a benchmark can separate models is a harder question than whether it can separate system configurations, and the results in the body answer the easier one.

Almost nobody reports $n_{j}$ separate scores; they average the panel into one. G-theory says the mean of $k$ judges carries $\sigma^{2}_{sj}/k$, which is falsifiable. A single ensemble judge cannot be fitted—at $n_{j}=1$ the judge facet is unidentified—so we form disjoint ensembles of size $k$, refit on those $\lfloor n_{j}/k\rfloor$ ensemble judges, and compare the resulting $\sigma^{2}_{sj}$ against the prediction. Over 60 random partitions per cell at $k=2$ and $k=3$ on all four item sets, observed over predicted ranges $0.93$–$1.11$ with median $1.01$. Every judge count in this paper may be read as an ensemble size.

All judges run at a fixed decoding temperature, so what the replicate facet measures is run-to-run variation at that temperature, which is $\sigma^{2}_{e}$ and not a temperature effect. Separating it needs an arm crossing temperature with judge, which we have not run. This is the one place where “judge” remains a bundle.

$E\rho^{2}$ asks whether a panel reproduces itself, not whether it is right. On SummEval (9), moving from $(n_{i}{=}10,n_{j}{=}1)$ to $(n_{i}{=}50,n_{j}{=}9)$ raises $E\rho^{2}$ from $0.711$ to $0.959$ ($+0.248$) while Kendall $\tau$ against the three-expert consensus rises only from $0.596$ to $0.710$ ($+0.114$) and is flat over the last four configurations. Recovery of the human top three sits at $23\%$–$32\%$ across every configuration against a $18.8\%$ chance baseline, and does not improve on either axis. Reliability is a precondition for a believable comparison, not a substitute for one.

On SummEval’s fully crossed human design (16 systems $\times$ 100 articles, 3 experts and 5 crowd workers), rater-by-system variance per unit of system signal is $0.07$–$0.13$ for LLM judges against $0.34$ for experts and $0.67$ for crowd workers. Two judges match three experts; crowd workers produce essentially no system-level signal ($\sigma^{2}_{s}\approx 0.002$ at five raters). Substituting humans makes the resolution worse at far greater cost.

Type I error is correctly calibrated: on pairs with a true difference of exactly zero—two independent generations of the same system—the modal protocol’s false-positive rate is $2.5\%$, below nominal. The problem is power, not calibration. Self-preference is real but small, explaining $0.8\%$–$4.8\%$ of $\sigma^{2}_{sj}$. Length explains more: within-item correlation between response length and score is $0.21$–$0.25$ on the public benchmarks and length explains $13\%$–$17\%$ of between-system variance, so part of what those benchmarks measure as quality is verbosity. On our screened items, which carry explicit format and length constraints, the correlation is $-0.004$, item design can remove this.

Three consequences a practitioner can act on. The naive bootstrap understates the interval. Resampling items only—what a careful author does today—gives a half-width of $0.34$–$0.49$ points against a true single-judge MDD of $0.58$–$1.24$, understating by $1.3\times$ to $2.8\times$ because it ignores the judge facet entirely. Rank inversions are near chance for adjacent systems. For two systems adjacent in the true top ten, whose mean separation is $0.010$–$0.013$ points, the probability of returning them in the wrong order is $45\%$–$49\%$ at one judge and barely moves at three. Under a call budget the optimum is interior. Minimising $\sigma^{2}_{\delta}$ subject to cost $\propto n_{i}n_{j}n_{r}$ gives, on MT-Bench, $30\times 4\times 1$ at 120 calls, $60\times 6\times 1$ at 360 and $108\times 10\times 1$ at 1080; on our screened items $20\times 6\times 1$, $36\times 10\times 1$ and $60\times 18\times 1$. The optimal judge count grows with budget rather than saturating, and a single run is optimal at every budget examined.

The four sets use different amounts of the scale, so a raw MDD is measured against a different ruler in each. System means span $2.63$ points on our screened set against $1.09$ on MT-Bench and $1.18$ on AlpacaEval 2. In units of each set’s own between-system spread the ordering changes: $1.57$ SD for Arena-Hard, $1.77$ for the screened set, $2.31$ for AlpacaEval 2, $2.67$ for MT-Bench, so the screened set has the largest raw MDD and the second smallest standardised one. This is the same scale-compression artefact we document for judge panels in §A: any absolute-scale comparison of noise rewards an instrument for compressing its output. Raw points are the right unit for comparing a benchmark’s floor against improvements reported on that benchmark (§5.6), and the wrong unit for ranking benchmarks against each other.

Screening raised $\sigma^{2}_{s}$ by an order of magnitude over MT-Bench ($0.456$ against $0.044$) and roughly doubled the usable scale range, but raised the interaction terms alongside it ($\sigma^{2}_{si}$ $0.474$ against $0.286$; $\sigma^{2}_{sij}$ $0.683$ against $0.267$). That is unsurprising on reflection: items selected for separating systems will also separate them in different orders, which is system-by-item interaction by definition. The net is a single-judge ceiling of $0.746$ and a standardised MDD of $1.77$ SD, in both cases second to Arena-Hard’s $0.792$ and $1.57$. Screening buys a better instrument than the two weakest public sets and does not beat the strongest: a lever worth pulling, not a solved problem.

MDD@30 ($0.58$–$1.24$) evaluates every set at a common thirty items, which is what makes them comparable. MDD at native counts ($0.41$–$1.24$) evaluates each at the count its users deploy, the right figure against a published result. MDD at infinite items ($0.40$–$1.09$) is the floor no item budget can beat under one judge. All three are single-judge, single-run, and none moves much, because items saturate.

$\sigma^{2}_{s}$ is defined over a population of systems, exactly as $E\rho^{2}$ is defined over a universe of judges. “MT-Bench has 4.5% system variance” is shorthand for “across the 42 systems studied here”. We report the near-frontier subset separately (§A) precisely because the choice of system population changes the answer: discriminating power is a property of the pair, not of the benchmark alone. Neither facet should be reported without its universe.

Coding reliability was established in two stages. The sample was first coded a second time by a different model family (GPT-5.6) under an independently worded prompt. A 30-paper subsample was then coded by a human working blind—no automated answers, no evidence quotes—so that agreement measures coding difficulty rather than suggestibility.

Against the human, overall agreement is 88.0% ($\kappa=0.73$). Crucially, the field the headline depends on validates almost perfectly: whether the evaluation was repeated reaches $\kappa=0.89$ with a single disagreement in thirty papers, the human coding 16.7% and the automated coder 20.0%. Whether the scorer was named is perfect ($\kappa=1.00$). Uncertainty reporting is substantial ($\kappa=0.65$).

The human coding also resolves an ambiguity the two automated coders could not. Between themselves they agreed only moderately on replication ($\kappa=0.47$, estimating 15% and 29%), which we had provisionally reported as a range. The human agrees closely with the first coder and not the second, indicating that the disagreement was one coder in error rather than genuine ambiguity in the papers. We therefore report the lower, validated figure.

One field remains genuinely ambiguous and is reported as a range: version pinning ($\kappa=0.45$; 53% automated, 73% human), where the human applied a more generous standard for what counts as a pinned version. Item counts show 86.7% agreement, though $\kappa$ is undefined there because the human coded every paper affirmatively.

The audit so far shows that disclosure is absent, not that any conclusion was wrong, and those are different complaints. The link we can actually test is whether the papers that do report enough to be checked are also the ones whose claims clear a floor. Of the 227 papers reporting a judged comparison, $146$ ($64.3\%$) state both a judge model and an item count, which is the minimum needed to compute an implied floor at all; the remaining third cannot be checked by anyone, including their own authors.

Among the 60 recovered deltas, 35 come from papers stating both and 25 do not. Applying the same MT-Bench floor to both groups at each paper’s own stated item count, $40\%$ of the fully-disclosing claims exceed it against $24\%$ of the rest. The absolute level here is not trustworthy—it transfers one instrument’s floor to claims made on others, which is the error this paper argues against—but the contrast is, because both groups are priced against the identical floor, so whatever bias the transfer introduces is common to them. The direction is worth stating plainly: better-disclosed claims are not the fragile ones. Disclosure is not what makes a result survive, but it does travel with results that do, most likely because papers that report an item count tend to report a larger one.

This is the causal link between Section 6 and Section 5.1, and it cuts against the most cynical reading of the audit. The problem is not that the field is making claims it knows to be unsupported; it is that two-thirds of a corpus can be checked and one-third cannot, and no reader can tell from the outside which third a given paper is in. That is a reporting failure with a cheap fix, and it is the one recommendation in this paper that costs nothing to adopt.

Two systems measured on the same items and judges share those facets, so the item and judge main effects cancel in the difference and only the interactions with system survive. That is why $\sigma^{2}_{\delta}$ carries $\sigma^{2}_{si}$, $\sigma^{2}_{sj}$, $\sigma^{2}_{sij}$ and the residual but not $\sigma^{2}_{i}$, $\sigma^{2}_{j}$ or $\sigma^{2}_{ij}$: this is relative error, appropriate for ranking or comparing systems within a study, and not absolute error, which is what one would need to compare a score against a fixed external standard. The variance of the difference is $2\sigma^{2}_{\delta}$ because the two systems are independent draws from the system population given the shared design.

$\mathrm{MDD}_{95}\approx 1.96\sqrt{2\sigma^{2}_{\delta}}$ assumes the difference of two system means is approximately normal. With $n_{i}\geq 20$ items the central limit theorem is doing the work and the approximation is unremarkable; at very small $n_{i}$ it will be optimistic, and a $t$ quantile on the appropriate error degrees of freedom should be substituted.

We estimate components by expected mean squares on the complete crossed block. The estimator is unbiased and closed form, which matters here because we refit under many judge universes, system populations and item subsets, and a closed form makes those refits cheap and deterministic. Its known cost is that individual component estimates can go negative when the true value is near zero; we truncate at zero when reporting variance shares and leave the raw value in place when computing $\sigma^{2}_{\delta}$, so that truncation never flatters a floor. REML would avoid negative estimates and is the better choice for a single definitive fit, but it is iterative and would make the bootstrap over 600–800 resamples materially more expensive. Where we report intervals we use a parametric bootstrap rather than Wald intervals, precisely because the sampling distribution of a variance component near the boundary is not symmetric.

The system facet has $n_{s}-1$ degrees of freedom, so with 42 systems $\sigma^{2}_{s}$ is the least precisely estimated component in the design, and it is the numerator of every reliability quantity. This is the main reason we report ceilings with bootstrap intervals rather than as point estimates, and why Arena-Hard—which carries the fewest items and therefore the widest interval—is the cell we are least willing to make claims about.

The screened row throughout this paper is a releasable benchmark, named here so it can be cited and criticised separately from our results. Sieve is 38 open-ended instruction items selected for between-system discriminating power, constructed as follows. Pool: 71 items across four strata designed to force systems apart rather than to sample a task distribution, pushback (24 items, a false premise or unsafe-but-plausible ask), compose (18, interacting constraints), constraint (16, a checkable format or length rule), verifiable (13, a checkable factual or arithmetic core). Screening design: $5$ systems $\times$ 71 items $\times$ 1 judge $\times$ 2 replicates, 720 calls, on the same holistic 0–5 rubric used in the main study; a checklist variant discriminated roughly $4\times$ worse and was dropped. Screening judge: openai/gpt-4.1-2025-04-14, deliberately not a member of the nine-judge evaluation panel, so no item is selected using a judge that later scores it. Selection rule: retain items with between-system variance $\geq 1.0$ on the 0–5 scale; pool mean was $3.99$ (sd $1.36$) and 38 of 71 cleared it. The main study harvests 30 of the 38. Sieve is a measurement instrument, not a capability benchmark: items were chosen because systems answer them differently, not because they represent any task distribution.

The crossed judgment dataset (373,019 calls across four item sets, nine judges, three replicates, with per-call model, prompt configuration, replicate index, raw completion and parsed score); Sieve, its stratum labels and per-item discrimination scores, plus the 33 screened-out items; the pairwise arm (4,608 preference judgments in both orders); a D-study calculator that takes variance components and returns item and judge requirements; and the audit, 628-paper frame, both automated codings, the blind human coding, and the codebook.

Data and item sets under CC BY 4.0, code under MIT. Judgment records carry the dated model identifier requested rather than the alias, since the proxy echoes the alias back; readers reproducing against another endpoint should expect different snapshots behind the same names. The endpoint used here enforces a client allowlist at the edge, so an external reader cannot replay the harvest against it and must supply their own provider credentials.

The datasets and code described above are available from the author on request.