A Persistent State for Auditable Mixture-of-Experts Routing

We sparse-upcycle two pretrained dense language models: HuggingFaceTB/SmolLM2-135M and google/gemma-3-270m. The local model and tokenizer bytes used by the reported experiments are reproducible from the following public repository revisions:

For each family, the tokenizer is taken from the same repository revision as the model; no separate tokenizer repository or tokenizer revision is used. Verification against these revisions reproduces the frozen local model and tokenizer bytes. We therefore describe them as byte-compatible reproducible revisions, rather than as recovered records of the original download event. The accompanying artifact manifests record the corresponding model, tokenizer, configuration, and source-code SHA-256 hashes.

Every dense feed-forward block is replaced with an MoE block containing 16 routed experts and one always-active shared expert. All experts retain the MLP geometry of the corresponding dense source layer. Each token executes the shared expert and the Top-2 routed experts.

For router logits $z_{\ell}$, the implementation first computes a softmax over all 16 routed experts, selects the Top-2 entries, and renormalizes their probabilities:

The two selected routed-expert outputs are weighted by $\widetilde{p}_{\ell,i}$ and summed; the always-active shared-expert output is then added to this routed output. There is no capacity clipping or dropped-token branch.

Ordinary MoE routes from $x_{\ell}$ only. Persistent and latest-write-only route from the concatenation of $x_{\ell}$ and $s_{\ell}$. The implemented router is a bias-free linear map; equivalently, the bias term in the general notation of Section 3 is $b_{\ell}=0$ for the models reported here.

Shared and routed experts are initialized as deep copies of the pretrained dense MLP at the corresponding layer. We then partially reinitialize each routed expert independently. For every routed-expert weight tensor with at least two dimensions, each entry is selected independently with probability $0.55$; selected entries are replaced by draws from a Gaussian whose mean and standard deviation equal those of the original tensor. Routed-expert biases are not reinitialized. The shared expert remains an unchanged copy of the pretrained dense MLP.

Router weights use Kaiming-uniform initialization with parameter $a=0.01$. Routers contain no bias. The state write head is initialized to produce an exact zero write: both its weights and bias are initialized to zero. The training seed is set before sparse upcycling and data construction.

The principal experiments use three independent training seeds, $42$, $43$, and $44$, for every model family and architecture variant. Architecture comparisons are paired by seed. The same construction algorithm and dense source checkpoint are used for matched variants, but we do not claim that historical initialized parameter tensors were retained byte-for-byte across variants.

At the beginning of each model forward pass, the routing state is initialized to zero in the hidden-stream dtype. It is then threaded through successive MoE layers for each token. The write head reads the always-active shared-expert representation $h^{\mathrm{shared}}_{\ell}$ and computes

The main text suppresses the write-head bias from the notation; the implementation includes it and initializes it to zero.

For Persistent,

For latest-write-only,

There is no detached or separately retained accumulator in the latest-write-only model. Because $u_{\ell}$ is produced by a $\tanh$, its native replacement state lies in $[-1,1]$ even though the common outer clamp is $[-5,5]$. The resulting state-geometry difference from Persistent is discussed further in Appendix C.5.

The model state is stored in the forward dtype; the reported training runs use bf16. Attribution calculations that require state differences or router-weight dot products convert the relevant quantities to float32, as detailed in Appendix C.

Training examples are sampled from FineWeb-Edu (Penedo et al., 2024), The Stack Dedup Python (Kocetkov et al., 2023), and FineMath-4+ (Ben Allal et al., 2025) with probabilities $0.50$, $0.25$, and $0.25$, respectively. These are sampling probabilities before packing and are not asserted to equal the realized proportions of the packed token stream.

All reported runs use 30,000 optimizer steps and a global batch of 512 length-2048 sequences, corresponding to $1{,}048{,}576$ packed input tokens per optimizer step and a nominal total of

input tokens per training run.

Routed experts and remaining base-model parameters use the base learning rate. The shared expert uses $0.1\eta$ and is held at zero learning rate for the first 3,000 optimizer steps. Router and write-head parameters use $2\eta$. After a 1,500-step linear warmup, each parameter group follows a cosine decay to $0.1$ times its peak learning rate. Weight decay is applied to eligible matrix weights but not to biases or normalization parameters.

Let $N=16$ denote the number of routed experts, $f_{i}$ the fraction of hard Top-2 assignments to expert $i$, and $P_{i}$ its mean pre-Top-2 softmax probability. The load-balancing term is

where both selected Top-2 slots contribute to $f_{i}$. Each MoE layer also uses the router z-loss (Zoph et al., 2022)

Auxiliary losses are averaged across MoE layers and added to the language-model cross-entropy objective.

The Gemma latest-write-only runs use microbatch 4 with gradient accumulation 8 rather than the microbatch-2, accumulation-16 shape used by the Ordinary/Persistent Gemma runs. This changes the per-device execution shape but preserves the world size, global batch, sequences per optimizer step, and nominal token budget.

Table 4 reports the analytical model-size and active-compute quantities used in Section 5.1. Active parameters count all non-routed parameters, the always-active shared expert, and the two selected routed experts. Forward FLOPs count the reported analytical matrix-multiplication work per token and should not be interpreted as measured wall-clock latency.

Relative to Ordinary MoE, Persistent therefore increases the analytical forward FLOP count by approximately $0.62\%$ for SmolLM2 and $0.34\%$ for Gemma. Persistent and latest-write-only have identical parameter and analytical active-compute counts within a family.

We additionally measured full-sequence runtime on a common NVIDIA GH200 120GB GPU. These measurements are descriptive implementation diagnostics rather than training-seed replications. Inference measurements are full length-2048 forward passes rather than autoregressive cached decoding. The training measurement is a single-GPU forward–backward pass and excludes optimizer updates and distributed communication.

The direction of the measured runtime difference is not consistent across the two model families. We therefore use the analytical FLOP counts as the architecture-level compute accounting and report the wall-clock measurements only as implementation-specific descriptive results.

The canonical Persistent and Ordinary-MoE implementation freeze is tied to Git commit 984f88b7f1aac9373a55d221154308cf82e1a2b8, with an empty tracked-diff hash in the corresponding freeze manifest. The latest-write-only runs are accompanied by launch manifests that bind the model source, baseline source, configuration, upcycling code, trainer, software environment, and run configuration to cryptographic hashes.

Result and configuration artifacts use SHA-256 identities over their exact serialized bytes. The training-seed registry additionally records a structured content hash computed from compact canonical JSON with sorted keys, comma/colon separators, UTF-8 encoding, and no trailing newline; the human-readable on-disk registry is stored as pretty, sorted JSON with a trailing newline. These identities separate the logical checkpoint record from filenames or storage locations.

The main replicated architecture-level results use training seeds $42$, $43$, and $44$, paired by seed within each model family. Analyses that use a fixed checkpoint rather than all three training realizations are labeled explicitly at the point of use in the later appendices.

We separate analysis construction from final evaluation. Construction data are used to choose layer sets, representations, controls, and aggregation rules; where model selection is required, a separate calibration role is used. Corresponding final-evaluation examples are not inspected until those choices are frozen. Throughout the paper, “held out” refers to this analysis-role separation. It does not imply that the underlying documents were absent from the pretraining corpus of the original dense source model.

The language-model and local routing-state analyses use a balanced three-domain final-evaluation pool. For each model family, it contains 96 sequences of length 2048: 32 prose sequences from the local FineWeb-Edu sample, 32 code sequences from The Stack Dedup Python, and 32 mathematics sequences from FineMath-4+. Stable sequence identities are retained across matched architecture variants so that paired comparisons operate on the same source material.

The full-forward attribution experiment uses a smaller, independently fixed subset of this evaluation material. Its sequence and target-selection procedure is specified in Appendix D.

The validation-perplexity quantity reported in Section 5.1 is computed on the 96-sequence balanced-domain pool described above; it is not perplexity on a standard external validation corpus.

Packed examples contain pre-aligned next-token labels. No additional label shift is applied during evaluation. Each length-2048 packed sequence contains one aligned label for each of its 2048 input positions, including the final input position, so each domain contributes exactly

valid labels and the full evaluation contains

valid labels per model evaluation.

Let $\mathcal{T}$ denote the set of valid token positions and $\ell_{t}$ the next-token negative log likelihood at position $t$. We compute

Thus the point estimate is token weighted, both within and across domains. We do not compare absolute perplexities between SmolLM2 and Gemma because the two model families use different tokenizers.

For each architecture contrast and training seed, the same 96 source sequences are evaluated under both matched variants. Relative perplexity is formed as

where $s\in\{42,43,44\}$ is the training seed. Cross-seed inference is performed on $\log r_{s}$; the mean and Student-$t$ interval are transformed back to the ratio scale. Percentage changes reported in the main text are $100(r-1)$. Training seed, rather than token count or bootstrap replicate, is the replication unit for these architecture-level claims.

We evaluate HellaSwag, ARC-Easy, PIQA, and WinoGrande zero-shot (num_fewshot=0), without a system instruction, chat template, or free-form generation. Request construction follows the corresponding EleutherAI lm-evaluation-harness task definitions at Git commit b954108c9baaaa934b4ad842033b31a97ee30816. Our local request construction and per-example metric implementation were checked against these pinned task definitions before the canonical evaluations.

For HellaSwag, ARC-Easy, and PIQA, each candidate answer is scored by its conditional continuation log likelihood. Raw accuracy selects the candidate with largest log likelihood. For normalized accuracy, the conditional log likelihood is divided by the Python character length of the corresponding task choice before selection. Ties are resolved by the lowest candidate index.

WinoGrande uses its task-specific harness construction rather than treating the candidate word itself as an independently scored continuation. Each candidate is inserted into the task prefix and the conditional likelihood of the remaining suffix is scored; the higher-scoring completed construction is selected.

The primary task metrics are

• •

  normalized accuracy for HellaSwag;

• •

  normalized accuracy for ARC-Easy;

• •

  normalized accuracy for PIQA; and

• •

  raw accuracy for WinoGrande.

The reported macro score is their unweighted arithmetic mean:

The evaluated splits and complete example counts are shown in Table 6. Inputs are limited to 2048 tokens. When truncation is required, context is left-truncated while retaining the complete scored continuation and at least one context token.

Table 7 gives the three-seed mean task-level changes relative to the matched Ordinary MoE. Values are absolute percentage-point changes in the corresponding primary metric. Full seed-by-seed values and Student-$t$ intervals are given in Appendix E.

The state-use experiment in Section 5.2 is a local same-state counterfactual rather than a full-forward intervention. For each natural Persistent forward pass, we retain the hidden representation $x_{\ell}$ and routing state $s_{\ell}$ entering a router and recompute only that router under

As noted in Appendix A.2, the implemented routers are bias-free, so $b^{\ell}=0$ in the reported models.

Neither $x_{\ell}$ nor $s_{\ell}$ is recomputed after attenuation, and the modified expert selection is not propagated through later model layers. The primary metric is

where Top-2 is treated as an unordered selected-expert set. The no-op endpoint $\lambda=1$ is zero by definition, and $\lambda=0$ removes the direct state term from the current router while leaving the naturally produced hidden representation and state fixed.

The analysis uses the 96-sequence final-evaluation pool from Appendix B.1. We first reduce the routing decisions within a checkpoint to one checkpoint-level rate. Cross-checkpoint summaries then treat training seeds 42, 43, and 44 as the three replicates and report Student-$t$ 95% intervals with two degrees of freedom. The full-suppression Persistent seed-level rates are

for SmolLM2 and

for Gemma, giving the main-text means of $87.6\%$ and $69.9\%$, respectively.

For each frozen source–target layer pair, a probe predicts the clean Top-1 expert index of the target router from a representation captured at the same token position at an earlier source layer. The source representation is the actual representation available at the source router: the general hidden stream for hidden-state probes and the 128-dimensional native routing state for native-state probes.

Training, calibration, and final-evaluation roles are disjoint. The frozen probe-training budgets are

association-role examples. Calibration and final evaluation each contain 98,304 examples per source layer, balanced as 32,768 examples from each of the prose, code, and mathematics domains. Each domain contribution comprises 64 selected token positions from each of 512 sequences. The final-evaluation role is not loaded until representation construction, hyperparameter selection, and fitted-model identities are frozen.

The frozen source and target layers are shown in Table 8.

We evaluate the native 128-dimensional routing state and five representations of the model’s ordinary hidden stream:

1. 1.

   a full-hidden multinomial linear predictor;

2. 2.

   a one-hidden-layer MLP with a 128-unit ReLU bottleneck;

3. 3.

   a supervised 128-dimensional hidden subspace;

4. 4.

   a 128-dimensional PCA representation; and

5. 5.

   a 128-dimensional random projection, averaged over five independently fixed projections.

Unless explicitly stated otherwise, a hidden-state representation is taken from the same architectural variant being evaluated. In particular, the “Persistent hidden subspace-128” curve compared with the Persistent native state in Figure 3 is learned from the Persistent model’s ordinary hidden stream; it is not an Ordinary-MoE baseline.

Captured hidden and native-state features are stored from the natural forward pass and converted to float32 before fitting. A StandardScaler is fit on probe-training examples only and then applied to calibration and final examples. Learned projected representations receive a second scaler fit only on their projected training features.

For the linear predictors, the candidate regularization values are

Fitting uses scikit-learn 1.9.0 multinomial logistic regression with solver=lbfgs, max_iter=1000, and tol=1e-4. Calibration macro-F1 selects the candidate; ties are resolved in favor of smaller $C$.

For the nonlinear bottleneck, the candidate pairs are

The classifier has one 128-unit ReLU hidden layer and uses Adam with batch size 256, max_iter=1000, tol=1e-4, n_iter_no_change=20, early_stopping=False, and deterministic shuffling. Calibration macro-F1 again selects the candidate. The frozen candidate order implements the tie rule of larger $\alpha$ first and then smaller learning rate.

PCA uses randomized SVD with 128 components and a deterministic seed. The supervised hidden subspace is constructed from the row space of the fitted full-hidden linear classifier: the classifier coefficient matrix is converted to float64, decomposed by SVD, and deterministically orthogonally completed to 128 dimensions where necessary. The resulting 128-dimensional representation is then used as the input to the final probe. Random-projection results average five independently fixed 128-dimensional projections.

All frozen source-layer fits used for the reported analyses converged before final evaluation. Hyperparameter selection uses calibration data only; final evaluation is not used to choose a representation or fitting candidate.

Figure 3 reports a diagnostic subset called transition-only. This term refers only to a transition in numeric expert-index sets. Let

be the clean Top-2 numeric expert indices at source and target layers for one example. After sorting each pair, the example is retained exactly when

Top-2 order is therefore ignored.

Because experts at different layers are distinct modules, equality of a numeric index across layers does not imply persistence of a physical or semantic expert identity. Conversely, the mask in Equation 20 should not be interpreted as a semantic expert-change label. It is a fixed numeric-index diagnostic intended to remove examples on which the source and target Top-2 index sets happen to be identical.

For retained examples, transition-only token accuracy is ordinary Top-1 classification accuracy:

where $\mathcal{M}$ is the mask defined above, $e_{i}$ is the clean target Top-1 numeric expert index, and $\hat{e}_{i}$ is the probe prediction. The metric is undefined only if $\mathcal{M}$ is empty; no reported frozen pair has an empty final-evaluation mask.

Two related but intentionally different probe aggregates appear in the main paper. We define them separately here to avoid conflating their numerical values.

The full representation study additionally evaluates the full hidden-state linear predictor, 128-unit nonlinear bottleneck, PCA-128, and five-replicate random-projection-128 controls defined in Appendix B.6. These controls address different possible explanations for the performance of the supervised hidden subspace: unrestricted linear accessibility, nonlinear probe capacity, unsupervised low-dimensional variance, and arbitrary low-dimensional projection.

The representation comparison is not parameter matched to the native routing state: a learned predictor applied to a hidden representation may contain more fitted parameters than the model’s native 128-dimensional state pathway. Accordingly, we use these probes to characterize accessibility rather than to claim an equal-resource competition between representations. The complete per-seed numerical probe record is reported in Appendix E.

The multi-write routing ledger is defined only for Persistent. For consecutive natural states entering successive routers, the realized source write is

This is the state change that remains after the source layer’s $\tanh$ write proposal, accumulation with the incoming state, and elementwise clamp. We never substitute the pre-clamp proposal $u_{\ell}$ for $\Delta s_{\ell}$.

Consider a target layer $t$, with clean Top-1 expert $e_{1}$, clean second-selected expert $e_{2}$, and clean best-excluded expert $e_{3}$. Let $W^{t}_{S}$ denote the columns of the target router acting on the 128-dimensional state. We retain two margin decompositions:

The second quantity is the contribution to the clean Top-2 membership boundary used throughout the attribution analyses.

Because $s_{0}=0$,

These equalities concern the direct state-mediated component of the target router logits. Influences transmitted through the ordinary hidden stream are outside this decomposition.

The model forward pass uses bf16. Natural states and router logits are captured at the actual router hook and converted to float32 for attribution; the relevant router weights are likewise copied to float32. The attribution cache stores float32 realized-write vectors, target logits, and contribution tensors. Reconstruction is therefore exact algebraically but need not be bit-exact after finite-precision capture, subtraction, and summation.

Table 10 reports the frozen numerical validation on the construction checkpoints. The acceptance tolerance was $10^{-4}$ absolute error. Both families remain more than an order of magnitude below this threshold.

The distributed-history analysis uses the selected-set contribution $c^{\mathrm{set}}_{\ell,t}$ from Equation 26. For brevity, write $c_{\ell,t}=c^{\mathrm{set}}_{\ell,t}$ and define the absolute attribution mass for target $t$ as

For $A_{t}>0$, define normalized absolute source weights

Let $r(t)$ denote the most recent eligible source before target $t$. The non-recent absolute contribution share is

Thus “non-recent” means every eligible source except the immediately preceding source; it does not denote a depth tercile or a fixed number of layers.

We quantify concentration with inverse Herfindahl concentration:

We refer to $N_{\mathrm{eff},t}$ as the effective source support. It equals one when all absolute mass is assigned to one source and increases as the mass becomes distributed across sources. This is an inverse-concentration quantity, not an entropy-based effective support. We additionally record the largest absolute source share

If $A_{t}=0$, the normalized quantities in Equations 30–32 are undefined and are omitted from the corresponding Persistent summary. All frozen Persistent targets used for the reported exact-provenance analysis have nonzero attribution mass, so this rule does not remove any reported Persistent target.

The direct-provenance analysis uses five frozen target layers per family:

Each frozen target-layer summary uses the same 96-sequence evaluation pool described in Appendix B.1. Target-layer summaries are averaged with equal weight within a checkpoint. Cross-checkpoint summaries then treat training seeds 42, 43, and 44 as the replication units and use Student-$t$ 95% intervals with two degrees of freedom.

Persistent assigns most absolute direct contribution away from the latest write. The non-recent share is $91.7\%$ on SmolLM2 (95% CI $[90.5,92.9]\%$) and $90.3\%$ on Gemma ($[89.2,91.4]\%$). The corresponding effective supports are $9.49$ writes ($[9.36,9.62]$) and $5.82$ writes ($[5.61,6.04]$). The largest individual source accounts for only $20.4\%$ ($[19.8,21.0]\%$) of absolute mass on SmolLM2 and $31.5\%$ ($[29.5,33.4]\%$) on Gemma.

These quantities use absolute contribution and therefore do not measure net signed support. As a complementary signed summary, the direct state-mediated Top-2-boundary contribution, $\sum_{\ell<t}c_{\ell,t}$, has three-seed means $0.408$ on SmolLM2 and $0.254$ on Gemma, with 95% intervals $[0.239,0.577]$ and $[0.175,0.333]$, respectively. Large positive and negative source terms may therefore cancel even when absolute provenance is distributed. The distribution statistics should not be interpreted as downstream causal effects; those are tested separately in Appendix D.

Latest-write-only does not use the Persistent multi-write ledger as its native audit object. Its recurrence is

so the state entering a later router directly contains only the most recent replacement value.

For target layer $t$, let $s_{t}$ be the state entering the target router and let $e_{2},e_{3}$ be the clean second-selected and best-excluded experts. The direct latest-state score assigned to the immediately preceding eligible source is

Every earlier eligible source receives direct native score zero because its replacement value has already been overwritten. The state and router-weight slice are converted to float32 before this dot product. Source-score ties are resolved by descending score and then ascending source-layer index.

The zero scores in this construction have a narrow interpretation: they mean that an earlier source has zero direct contribution through the state currently entering the target router. They do not imply zero indirect influence through the ordinary hidden stream or through consequences of earlier routing decisions.

This distinction is important for Figure 5. If one instead mechanically formed

under the latest-write-only recurrence, earlier differences would generally be nonzero and would telescope through cancellation. That is not the audit object used for the latest-write-only comparison. Under the implemented structural direct-state definition, the complete direct absolute mass belongs to the most recent replacement source; hence non-recent share is zero, effective source support is one, and largest-source share is one by construction.

Persistent and latest-write-only should therefore be understood as exposing different native direct-state provenance structures: Persistent retains a sequence of realized additive state changes, whereas latest-write-only retains one direct replacement source.

The full-forward source intervention uses the same hook operation for both state architectures but has a different recurrence-level interpretation.

For a selected token and source layer $\ell$, a pre-hook first saves the state $s_{\ell}$ entering the source block. The source block is then allowed to run naturally, including its normal routing decision, expert computation, write proposal, and state update. A post-hook verifies the natural state transition against the clean recorded transition and then replaces the selected token’s post-source state by the saved pre-source value:

All subsequent model computation is then rerun naturally from this modified trajectory. Later hidden states, routers, expert selections, write proposals, states, and output logits are not held fixed.

For Persistent, the natural transition is

whereas the intervention uses

Thus the intervention removes the realized post-clamp state transition at that source. This formulation is preferable to setting the proposal $u_{\ell}$ to zero because saturation can make the realized write $\Delta s_{\ell}$ differ from the proposal.

For latest-write-only, the natural transition is

while the same hook gives

This intervention therefore skips the current replacement and restores the previous state; it does not set the current replacement value to zero. At the next latest-write-only update the explicit state is again replaced by the newly recomputed proposal. Indirect consequences can nevertheless persist, because the intervened source may alter subsequent routing, hidden representations, and therefore later write proposals.

For the scalar audited boundary effect, the identities of the clean $e_{2}$ and $e_{3}$ experts are held fixed so that the same clean margin is evaluated before and after intervention. Expert selections and other downstream quantities themselves are allowed to change. Detailed target selection and downstream metrics are specified in Appendix D.

Persistent and latest-write-only match the routing-state width, write-head architecture, router input width, parameter count, and analytical active compute, but they do not induce the same state geometry. Because

the latest-write-only state is intrinsically confined to $[-1,1]$ coordinate wise; its common outer $[-5,5]$ clamp is therefore inactive under exact arithmetic. Persistent instead accumulates successive proposals and can reach the $[-5,5]$ clamp.

Consequently, the comparison does not hold state magnitude, variance, or saturation frequency fixed. It tests persistent accumulation against replacement under the state distributions learned by those two mechanisms. We therefore do not interpret the latest-write-only comparison as isolating accumulation while controlling away every associated geometric or optimization difference.

An older fixed-checkpoint SmolLM2 diagnostic illustrates the magnitude of this difference. At source layer 14, the mean state norm was approximately $42.52$ for Persistent and $9.74$ for latest-write-only, while approximately $32.96\%$ of Persistent state coordinates were at the clamp boundary. These values are single-checkpoint diagnostics rather than three-seed estimates and are not used in the replicated statistical claims of Section 5. Probe preprocessing, including train-only standardization of native state features, is specified in Appendix B.6.

The full-forward intervention study uses 24 fixed final-evaluation sequences and 96 target routing decisions per model family. Sequence and position selection are deterministic and outcome independent.

For each of the prose, code, and mathematics domains, the 32 candidate final-evaluation sequences from Appendix B.1 are ranked by

using salt routing-ledger-confirmatory-e4-sequences-v1. Here architecture denotes the model-family identifier, smollm or gemma, rather than the Persistent, latest-write-only, or Ordinary architectural variant. Fields are UTF-8 serialized and separated by a single NUL byte. The first eight sequences in ascending hash order are retained in each domain, giving 24 sequences per family.

The fixed target layers are

Within every selected sequence–target-layer pair, candidate positions are those with a valid aligned next-token label. They are ranked by

using salt routing-ledger-confirmatory-e4-events-v1. The first two positions are retained. Each sequence therefore contributes four target decisions, for

targets per family.

The resulting sequence and token identities are reused across matched architectural variants. Selection does not use routing margins, ledger contributions, source rankings, intervention effects, output changes, or any other downstream outcome.

For every retained target decision $i$, let $e_{1,i}$ and $e_{2,i}$ be the clean first- and second-selected experts and let $e_{3,i}$ be the clean highest-logit expert excluded from the Top-2 set. The primary audited quantity is the clean Top-2 membership boundary

The identities $e_{2,i}$ and $e_{3,i}$ are fixed from the clean trajectory when this scalar margin is evaluated after intervention. Routing decisions themselves are not frozen.

Every MoE layer $\ell<L_{i}$ is an eligible source. This yields 1,776 source interventions over the 96 SmolLM2 targets and 1,344 over the 96 Gemma targets in the exhaustive SA-MoE boundary audit.

All source rankings are computed from the clean trajectory before any intervention outcome is observed.

State-source suppression follows the architecture-specific semantics in Appendix C.4. In both Persistent and latest-write-only, the selected token’s natural post-source state is replaced by its pre-source state. The model then continues normally from the modified trajectory. Later hidden representations, writes, routing decisions, experts, and output logits may all change.

For target $i$ and source $\ell$, define the primary absolute fixed-boundary effect

We also retain the corresponding signed difference.

For clean and intervened Top-2 expert sets $S$ and $\widetilde{S}$, the routing-set distance is the Jaccard distance

Target Top-2 Jaccard distance applies Equation 49 at the audited target router. Target Top-1 change is the indicator that the clean and intervened Top-1 target experts differ. The reported downstream Top-2 divergence averages the corresponding Top-2 Jaccard distance over routing decisions strictly downstream of the audited target for the selected token.

At the selected token’s language-model output, let $p_{\mathrm{clean}}$ and $p_{\mathrm{int}}$ denote the clean and intervened softmax distributions. Output KL is reported in the clean-to-intervention direction,

using natural logarithms. Output Top-1 change records whether the most likely output token changes.

For the aligned next-token label $y_{i}$, the NLL effect is

so positive values indicate increased loss after intervention.

Within a checkpoint, target-level quantities are first grouped by their 24 source sequences. Confidence intervals use 2,000 paired sequence-level bootstrap resamples; multiple target layers and positions from one source sequence are not treated as independent replicates. For architecture-level comparisons across checkpoints, training seed is the replication unit and 95% Student-$t$ intervals use $n=3$ and two degrees of freedom.

The exhaustive seed-42 audit intervenes on every eligible source for every fixed target. Let

denote the oracle fixed-boundary effect, and let $\widehat{\ell}_{iq}$ be the top-ranked source under method $q$.

Raw oracle regret is

and normalized oracle regret is

Normalized regret is defined for targets with positive oracle effect. A zero-oracle target would be excluded from this normalized quantity and reported separately rather than used as a numerical denominator.

Top-source accuracy is one when the method-selected source belongs to the set of oracle-maximal sources and zero otherwise; hence ties among genuinely oracle-maximal sources all count as correct. Within-target Spearman is the rank correlation between the complete clean-trajectory source-score vector and the complete intervention-effect vector $\{E_{i\ell}\}_{\ell<L_{i}}$. It is reported only when both rankings are defined; a constant or unavailable score vector contributes no Spearman value rather than an artificial zero correlation.

Table 11 reports the exhaustive Persistent seed-42 diagnostic. These are sequence-bootstrap summaries within one trained checkpoint, not three-seed architecture-level estimates.