A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL
Extracted source text · arXiv 2606.02398
\documentclass{article}
% NeurIPS 2026 submission format. For camera-ready, use \usepackage[main, final]{neurips_2026}; for arXiv/preprint, use \usepackage[preprint]{neurips_2026}.
% \usepackage{neurips_2026}
\usepackage[preprint]{neurips_2026}
% \usepackage[main, final]{neurips_2026}
% Standard package includes
\usepackage{latexsym}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{microtype}
\usepackage{inconsolata}
% Packages used by the original paper
\usepackage{url}
% Fallback for existing BibTeX entries generated with hyperref links.
\providecommand{\href}[2]{#2}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{float}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{bm}
\usepackage{nicefrac}
\usepackage{xcolor}
\usepackage{tcolorbox}
\usepackage{array}
\newcommand{\vtheta}{\bm{\theta}}
\newcommand{\vdelta}{\bm{\delta}}
\newcommand{\ve}{\bm{e}}
\newcommand{\vg}{\bm{g}}
\newcommand{\vu}{\bm{u}}
\newcommand{\vd}{\bm{d}}
\newcommand{\vr}{\bm{r}}
\newcommand{\vx}{\bm{x}}
\newcommand{\vy}{\bm{y}}
\newcommand{\mH}{\bm{H}}
\newcommand{\mP}{\bm{P}}
\newcommand{\mI}{\bm{I}}
\newcommand{\mW}{\bm{W}}
\newcommand{\vv}{\bm{v}}
\newtcolorbox{summarybox}{
colback=blue!5,
colframe=blue!60!black,
% title=Section Summary,
fonttitle=\bfseries,
arc=2mm,
boxrule=0.8pt,
left=2mm,
right=2mm,
top=1mm,
bottom=1mm
}
\title{A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL}
\author{%
Lei Yang\textsuperscript{1} \quad Siyu Ding\textsuperscript{2}\thanks{Project leader} \quad Deyi Xiong\textsuperscript{1}\thanks{Corresponding author} \\
\textsuperscript{1}TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China \\
\textsuperscript{2}Baidu Inc., Beijing, China \\
\texttt{yanglei\_9@tju.edu.cn}
}
\begin{document}
\maketitle
% Reinforcement learning (RL) post-training improves large language models (LLMs) across domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others. Existing explanations appeal to catastrophic forgetting or global gradient conflict, yet substantial cross-domain damage can arise even when full-model gradients are nearly orthogonal. This raises a key question: where does cross-domain interference reside inside LLMs?
\begin{abstract}
Reinforcement learning (RL) post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others. Existing explanations based on catastrophic forgetting or global gradient conflict are incomplete: substantial interference can occur even when full-model gradients are nearly orthogonal. We show that single-domain RL produces sparse, small-magnitude parameter edits with weak overlap among top-changed neurons, while different domains still share substantial active computation routes on which update directions determine whether they act synergistically or conflict. Guided by this observation, we prove under a local perturbation model of multi-domain RL that later-domain training harms an earlier domain mainly through a second-order damage term, which under the observed sparse route structure concentrates in a low-dimensional shared conflict subspace. Moreover, a short domain refresh contracts the harmful component on this subspace, enabling selective recovery with limited collateral damage. Consistent with the theory, a brief Re-Math refresh after Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW recovers Math from 57.66 to 66.04 while largely preserving performance on the other domains, yielding the best average score of 66.39. Beyond refresh, a training-free rollback on a sparse proxy conflict coordinate set for the Math-QA pair partially restores Math, providing direct proxy-level evidence for localized damage. These results provide a localized mechanistic account of interference and recovery in multi-domain RL.
\end{abstract}
% We show that cross-domain interference in sequential multi-domain RL can arise even when full-model gradients are nearly orthogonal. A local perturbation analysis reveals that the damage concentrates in a low-dimensional shared conflict subspace, which further shows that a brief domain refresh can selectively recover degraded performance with minimal harm to other domains.
\section{Introduction}
% RL has become a primary technique for improving LLMs across reasoning, coding, question answering, and open-ended generation. In real-world settings, post-training should improve a model across multiple heterogeneous domains rather than produce isolated single-domain experts. Multi-domain RL therefore poses a fundamental challenge: updates from one domain can transfer positively, but they can also selectively damage capabilities acquired from earlier domains~\citep{li2026omnithinker}.
% Existing explanations often frame such degradation as catastrophic forgetting~\citep{kirkpatrick2017overcoming,shi2024continual} or global gradient conflict~\citep{sener2018multi,liang2026cgpo}. While useful, these views are too coarse for RL: full-model gradient cosines between domains can be close to zero even when later-domain training substantially degrades an earlier domain. Interference can therefore remain largely invisible at the whole-model level. This leads to our central question: where does cross-domain interference reside if not in global gradient antagonism?
RL has become a primary technique for improving large language models across reasoning, coding, question answering, and open-ended generation. In real-world settings, post-training should improve a model across multiple heterogeneous domains rather than produce single-domain experts. A natural approach is sequential multi-domain RL, where the model is trained on one domain after another.
However, a simple sequential curriculum already exposes a puzzling behavior. As shown in Table~\ref{tab:intro_motivation}, following the Omni-Thinker~\citep{li2026omnithinker} order, Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW, Math performance rises to 66.49 after Math training but drops to 57.66 after subsequent QA and CW training, while Code and QA remain relatively stable. This suggests that cross-domain interference is not uniform forgetting, but selective and asymmetric degradation. Existing explanations such as catastrophic forgetting~\citep{kirkpatrick2017overcoming,shi2024continual} or global gradient conflict~\citep{sener2018multi,liang2026cgpo} are too coarse for this behavior. In our experiments, full-model gradient cosines can be close to zero even when substantial degradation occurs. Interference can therefore remain invisible at the whole-model level. This leads to our central question: where does cross-domain interference reside if not in global gradient antagonism?
% The present work shifts the analysis from full-model objective conflict to the local active routes through which domain updates are expressed.
\begin{wraptable}{r}{0.48\textwidth}
% \vspace{-1.2em}
\centering
\small
\caption{A motivating example under the Omni-Thinker curriculum. Sequential RL improves the current domain, but later stages selectively damage Math.}
\label{tab:intro_motivation}
\resizebox{\linewidth}{!}{
\begin{tabular}{l|cccc}
\toprule
& $\mathrm{Code}_o$ & $\mathrm{Math}_o$ & $\mathrm{QA}_o$ & $\mathrm{CW}_o$ \\
\midrule
Code & \textbf{52.67} & 50.69 & 50.99 & 50.47 \\
Math & \textcolor{gray!45}{59.63} & \textbf{66.49} & \textcolor{red}{59.90} & \textcolor{red}{57.66} \\
QA & \textcolor{gray!45}{60.89} & \textcolor{gray!45}{60.52} & \textbf{62.34} & 62.34 \\
CW & \textcolor{gray!45}{82.40} & \textcolor{gray!45}{81.44} & \textcolor{gray!45}{81.79} & \textbf{86.52} \\
\bottomrule
\end{tabular}
}
\vspace{-1.0em}
\end{wraptable}
We answer this question by shifting the analysis from full-model objective conflict to the local active routes through which domain updates are expressed. This route-level view links three factors: where RL changes the model, which routes different domains activate at inference time, and whether update directions agree or conflict on shared routes. Although full-model gradient cosines are nearly orthogonal, layer- and module-level analysis reveals localized conflict and synergy. Domain RL induces sparse, small-magnitude edits with weak top-changed neuron overlap, ruling out the idea that domains mainly interfere by rewriting the same neurons. Yet reasoning-oriented domains still share substantial active-neuron overlap, so even sparse edits can interact through shared routes. On these shared edited routes, update direction separates positive transfer from interference.
Guided by these observations, we develop a local perturbation model of multi-domain RL.
After training on a domain $A$, the selected checkpoint is approximately stationary for the local objective.
Therefore, the leading effect of a later-domain update is a second-order damage term determined by whether the update enters curvature-sensitive directions of the earlier-domain objective.
Together with the empirically observed sparse parameter updates and highly overlapping active routes, this suggests that degradation is largely concentrated in \textit{a sensitive, low-dimensional shared active conflict subspace}.
Therefore, near-orthogonal full-model gradients, sparse parameter changes, and low cross-domain edit overlap are insufficient to prevent cross-domain interference.
Based on this theory, we further show that a short refresh on domain $A$ geometrically contracts the harmful component in this shared conflict subspace, thereby restoring the damaged domain while keeping collateral damage to other domains bounded.
We validate this prediction by retraining Math from $\mathrm{CW}_o$: Math performance recovers to 66.04, close to the Math-domain expert, while Code, QA, and CW are largely preserved, leading to the best overall average score of 66.39. We further probe the localization claim by a training-free rollback on a sparse coordinate proxy for the conflict subspace in the Math–QA pair.
% Guided by these observations, we develop a local perturbation model of multi-domain RL. After training on a domain $A$, the selected checkpoint is approximately stationary for the local objective, so a later update from another domain need not oppose $A$ in full parameter space to cause degradation. Its leading effect is instead a second-order damage term: what matters is whether the later update enters curvature-sensitive directions of the earlier-domain objective. Under the empirically observed sparse route structure, this damage concentrates in a low-dimensional shared active conflict subspace rather than across the full model. This explains why near-orthogonal full-model gradients and low edit overlap do not preclude interference: even a small update can still move along the few shared routes to which the earlier domain is sensitive. Building on this view, the theory further explains fast recovery: a short refresh on domain $A$ contracts the harmful component on this shared conflict subspace, yielding selective degradation, efficient recovery, and limited collateral damage. Consistent with this, in the Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW sequence, Math rises to 66.49 after Math training, drops to 57.66 after QA and CW, and returns to 66.04 after a short Re-Math refresh while Code, QA, and CW remain largely preserved, yielding the best average score of 66.39.
Our contributions are threefold:
\begin{itemize}
\item We show that cross-domain degradation is not explained by global gradient conflict or direct overlap among edited neurons, but by sparse RL edits interacting along shared active computation routes.
\item We formalize this mechanism by showing that degradation is driven by second-order damage on a low-dimensional shared conflict subspace, and that short domain refresh yields selective recovery with limited damage.
\item We provide two complementary validations: short-refresh recovery at the task level, and a training-free targeted rollback on proxy conflict coordinates for a pairwise interaction.
\end{itemize}
\section{Related Work}
RL post-training and multi-domain RL mainly optimize aggregate performance, leaving cross-domain interactions inside the model poorly understood. Early RLHF systems build on human-preference RL~\citep{DBLP:conf/nips/ChristianoLBMLA17}, summarization with feedback~\citep{DBLP:conf/nips/StiennonO0ZLVRA20}, and PPO-style optimization~\citep{schulman2017proximal}. Later work extends this setting to process supervision~\citep{DBLP:conf/iclr/LightmanKBEBLLS24}, mathematical reasoning~\citep{shao2024deepseekmath}, and large-scale reasoning RL~\citep{guo2025deepseekr1,team2025kimi}. Recent multi-domain RL methods include Omni-Thinker~\citep{li2026omnithinker} and CGPO~\citep{liang2026cgpo}. The present work instead asks where cross-domain interference resides, why it is selective, and why a short domain refresh restores performance.
Cross-domain degradation is often discussed in terms of continual learning (CL), forgetting, or gradient conflict. Recent studies show that apparent forgetting can reflect shifts in alignment or policy behavior~\citep{zheng2025spurious}. SFT--RL comparisons likewise suggest that RL may yield milder drift~\citep{chu2025sft,lai2025rft,shenfeld2025rlrazor,chen2025retaining}, with additional evidence for reasoning transfer~\citep{huan2025mathtransfer}. Multi-task optimization offers a complementary global view through gradient balancing~\citep{DBLP:conf/icml/ChenBLR18,sener2018multi} and gradient surgery~\citep{yu2020gradient,DBLP:conf/nips/LiuLJSL21}. Our results suggest that full-model gradients can appear nearly orthogonal even when localized conflict harms earlier domains.
Task-vector and mechanistic work suggests that model changes can often be localized. In weight space, model soups~\citep{wortsman2022model} and TIES-style merging~\citep{yadav2023ties} attribute transfer and interference to redundant or sign-conflicting updates~\citep{DBLP:conf/nips/MatenaR22}. Mechanistic studies localize knowledge and skills to compact subsets of parameters~\citep{DBLP:conf/iclr/BauBSDDG19} or neurons~\citep{dai2022knowledge,wang2022finding,leng2024towards}. Recent feature-level analyses suggest that RL mostly preserves the base representation while strengthening compact task-relevant features~\citep{shi2026why}. We build on this locality view to study changed neurons, active routes, and direction-dependent interference.
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.32\textwidth}
\centering
\includegraphics[width=\linewidth]{grad_conflict_cosine_SuperGPQA_vs_math.pdf}
\caption{Global gradient cosine.}
\label{fig:grad_conflict_cosine}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.32\textwidth}
\centering
\includegraphics[width=\linewidth]{attn_mlp_conflict_top6_top1.pdf}
\caption{Most conflicting module.}
\label{fig:attn_mlp_conflict}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.32\textwidth}
\centering
\includegraphics[width=\linewidth]{attn_mlp_synergy_top6_top1.pdf}
\caption{Most synergistic module.}
\label{fig:attn_mlp_synergy}
\end{subfigure}
\caption{Gradient relations between Math and QA at the global, attention and MLP levels.}
\label{fig:grad_conflict_localized}
\end{figure*}
\section{Empirical Study Setup}
\label{sec:setup}
To investigate where cross-domain interference arises and how recovery later occurs, we construct a controlled multi-domain RL setting. We use Qwen3-4B-Thinking-2507~\citep{DBLP:journals/corr/abs-2505-09388} as the initial model and consider four domains: mathematical reasoning, code generation, question answering, and creative writing, denoted as Math, Code, QA, and CW, respectively.
% 我们以Qwen3-4B-Thinking-2507作为初始模型，研究多领域强化学习后训练中的跨领域相互作用。本文选取四个研究领域：数学推理、代码生成、问答以及创意写作，分别简记为Math、Code、QA和CW。
\subsection{Data Construction}
The Math training data is randomly sampled from OpenR1-math~\citep{openr1}. The Code training data is from KlearReasoner-CodeSub-15K~\citep{su2025cegppocontrollingentropygradientpreserving}. The QA training data is sampled from SuperGPQA~\citep{DBLP:journals/corr/abs-2502-14739} by subfield and difficulty. The CW training data is from the crownelius/Creative-Writing series.\footnote{\url{https://huggingface.co/Crownelius}} Detailed data construction is described in Appendix~\ref{app:data_const}.
% 数学领域训练数据从OpenR1-math中随机采样获得。代码领域训练数据取自KlearReasoner-CodeSub-15K数据集。问答领域训练数据按子领域和难度层级从SuperGPQA中采样。创意写作领域训练数据源自crownelius/Creative-Writing系列数据集，具体构造见附录。
% 每个领域均采样5120条训练样本。验证集由对应数据源中未用作训练的样本构建而成。数学领域选用AIME25作为验证集，包含30道题目。代码、问答、创意写作三个领域的验证集各包含50条样本。
\subsection{Training Setup}
All training runs are based on GRPO~\citep{shao2024deepseekmath} implemented in VeRL~\citep{DBLP:conf/eurosys/ShengZYWZZPL025}. Except for the training domain and the initialization checkpoint, all experiments share the same hyperparameters, reported in Appendix~\ref{app:train_hyper}. In addition, distinct reward functions are used for each domain, with detailed specifications in Appendix~\ref{app:reward_func}.
We first train four single-domain expert models from the base model, denoted as $\mathrm{Math}_s$, $\mathrm{Code}_s$, $\mathrm{QA}_s$, and $\mathrm{CW}_s$. We then perform sequential multi-domain training. Following the Omni-Thinker curriculum~\citep{li2026omnithinker}, we use the fixed order Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW. The resulting checkpoints after the four stages are denoted as $\mathrm{Code}_o$, $\mathrm{Math}_o$, $\mathrm{QA}_o$, and $\mathrm{CW}_o$, respectively. In sequential training, each stage continues from the checkpoint obtained in the previous stage.
In addition, we include two baselines for four-domain mixed training in Section~\ref{sec:exp}, namely $\mathrm{JT}$ and $\mathrm{CGPO}$~\citep{liang2026cgpo}. For $\mathrm{JT}$, we use naive joint training, in which each batch contains equal amounts of data from all domains and parameters are updated in the standard way. In contrast, $\mathrm{CGPO}$ follows the official implementation: it computes domain-wise updates within each batch and then updates the affected parameters with a specific learning rate.
% 所有训练流程均基于VeRL框架中实现的GRPO算法。除训练领域和初始化检查点外，所有实验均采用相同的训练超参数，具体数值详见附录。此外，我们对每个领域使用不同的奖励函数，具体见附录。
% 我们首先基于基础模型训练得到四个单领域专家模型，Math_s,Code_s,QA_s,CW_s。随后开展顺序多领域训练，遵循Omni-Thinker的训练课程设定，采用固定训练顺序：代码→数学→问答→创意写作，得到Math_o,Code_o,QA_o,CW_o。在顺序训练过程中，每个训练阶段均从上一阶段得到的检查点继续训练。
% 此外，我们在第六节同样包含两个四域混合训练的baseline，\mathrm{JT}和\mathrm{CGPO}。对于JT，我们采用朴素联合训练，即在一个batch内包含等量各领域数据，但更新时正常更新。而CGPO则按领域遵循作者源码，在一个batch内按领域更新后以一定学习率更新该batch变化的参数。
\subsection{Checkpoint Selection and Evaluation}
For each training stage, we train until convergence and select the checkpoint with the best validation performance on the current training domain. Final evaluation is conducted on independent benchmarks. Math is evaluated on AIME24/25/26~\citep{aops_aime}, OlympiadBench~\citep{DBLP:conf/acl/HeLBHTSHHHZLQL024}, and HMMT~\citep{dekoninck2026matharena}. Code is evaluated on LiveCodeBench-v6~\citep{DBLP:conf/iclr/JainHGLYZWSSS25}. QA is evaluated on SuperGPQA-test~\citep{DBLP:journals/corr/abs-2502-14739} and MMLU-Pro~\citep{DBLP:conf/nips/WangMZNCGRAHJLK24}. CW is evaluated on WritingBench~\citep{DBLP:journals/corr/abs-2503-05244}. See Appendix~\ref{app:data_const} for details.
% 在每个训练阶段，我们将模型训练至收敛状态，并选取在当前训练领域验证效果最优的检查点，即单域训练选择单域最优，而多域混合则是多域平均最优。最终评估在独立基准数据集上完成。数学领域在AIME24、AIME25、AIME26、OlympiadBench和HMMT上进行评测；代码领域在LiveCodeBench-v6上进行评测；问答领域在SuperGPQA-test和MMLU-Pro上进行评测；创意写作领域在WritingBench上进行评测。细节见附录
\section{Structural Evidence for Localized Cross-Domain Interference}
\label{sec:ana}
This section identifies where cross-domain interference arises in the model. Full-model gradients give only a coarse picture; the analysis then focuses on sparse parameter edits, shared active routes, and directional alignment along those routes.
% \subsection{Global Gradients Miss Local Cross-Domain Conflict}
\subsection{Global Gradients}
A natural first question is whether cross-domain interference in domain RL can be explained by global gradient conflict. During $\mathrm{JT}$ training, domain-specific gradients are periodically computed and the cosine similarity between domain pairs is measured, $\cos(\vg_{d_i}, \vg_{d_j}) = \frac{\vg_{d_i}^\top \vg_{d_j}}{\|\vg_{d_i}\|_2\,\|\vg_{d_j}\|_2}$, where $\vg_d$ is the gradient for domain $d$. Positive, negative, and near-zero values indicate alignment, conflict, and approximate orthogonality, respectively.
% 我们首先研究领域强化学习中的跨领域干扰是否能够通过全局梯度冲突进行解释。在联合训练过程中，我们周期性计算各领域专属梯度，并计算不同领域梯度间的余弦相似度：$\cos(g_i, g_j)=\frac{g_i^\top g_j}{|g_i|_2|g_j|_2}$，其中$g_d$代表第$d$个领域的梯度。余弦相似度为正值时，表示两个领域的参数更新方向一致，即协同效应；为负值时，表示更新方向存在冲突；数值趋近于零时，则代表两个梯度近似正交。
Figure~\ref{fig:grad_conflict_cosine} shows that the global gradient cosine between Math and QA stays close to zero. This means that the later drop in Math after QA training cannot be attributed to strong full-model gradient antagonism alone. However, near-orthogonality at the global level does not mean interference is absent. Decomposing gradients by layer and module reveals a clear local structure: Figure~\ref{fig:attn_mlp_conflict} highlights the strongest conflict locations in attention and MLP modules, while Figure~\ref{fig:attn_mlp_synergy} highlights the strongest synergy locations. Thus, cross-domain interaction is not uniformly antagonistic across the whole model; it is localized, with both conflict and synergy appearing in different layers and modules. Complete results are provided in Appendix~\ref{app:grad_conf}.
% 如图X(a)所示，数学任务与问答任务之间的全局梯度余弦值始终接近于零，这表明经过问答任务训练后数学任务性能后续出现的下降，无法单纯用全模型强梯度拮抗效应来解释。不过，全局近似正交并不意味着不存在干扰结构。当我们进一步按网络层和模块对梯度进行分解后，呈现出了更具局部化的分布规律。图X(b)展示了注意力模块与多层感知机模块内部梯度冲突最强的位置，图X(c)则展示了梯度协同性最强的位置。上述结果表明，跨任务领域间的关联并非全局拮抗关系，而是在不同网络层与模块中同时存在局部冲突和局部协同两种特性。我们在附录中提供了完整的结果。
% \begin{summarybox}
% Global gradients are orthogonal, while layer-/module-level reveal localized conflict and synergy.
% \end{summarybox}
% 全局梯度几乎相互正交，而层级与模块级则体现出局部的冲突与协同关系。
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{01_abs_per_task.pdf}
\caption{Absolute parameter changes.}
\label{fig:param_abs_per_task}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{02_rel_per_task.pdf}
\caption{Relative parameter changes.}
\label{fig:param_rel_per_task}
\end{subfigure}
\caption{Parameter-change distributions of the four single-domain experts relative to the base model.}
\label{fig:param_change_per_task}
\end{figure*}
% \subsection{Single-Domain RL Induces Sparse and Small-Magnitude Updates}
\subsection{Sparse and Small-Magnitude Updates}
% The next analysis examines how single-domain RL modifies model parameters.
The gradient analysis above shows that cross-domain interaction is localized. We next examine whether the updates actually written into the model by domain RL are also localized.
Each single-domain expert is compared with the base model, with parameter changes measured at the element level across the full model: $
\Delta \mW = \mW_{\mathrm{expert}} - \mW_{\mathrm{base}},\ r(\mW)=\frac{|\Delta \mW|}{|\mW_{\mathrm{base}}|}$, where $\mW$ denotes the model parameter, $\Delta \mW$ is the absolute change, and $r(\mW)$ is the relative change.
As shown in Figure~\ref{fig:param_change_per_task}, across the four single-domain experts, approximately 77\%--89\% of parameters have absolute changes below $10^{-7}$ and relative changes below $10^{-3}$. Moreover, even among parameters with non-negligible changes, the update magnitudes remain small. These results suggest that domain-specific RL acts as a mild perturbation to the base model rather than a global parameter rewrite. The parameter analysis of sequential training is presented in Appendix~\ref{app:para}.
% \begin{summarybox}
% Domain RL modifies the base model mainly through sparse, small-magnitude parameter updates.
% \end{summarybox}
% 接下来我们分析单域强化学习如何修改模型参数。具体而言，我们将每个单域专家模型与基础模型进行对比，并在整个模型范围内以元素粒度量化参数变化：$\Delta W = W_{\mathrm{专家}} - W_{\mathrm{基础}},\ r(W)=\frac{|\Delta W|}{|W_{\mathrm{基础}}|}$，其中W为模型参数，\Delta W是参数绝对变化量，r(W)是参数相对变化量。
% 如图X所示，在四个单域专家模型中，约77%~89%的参数绝对变化量低于$10^{-7}$，相对变化量低于$10^{-3}$。这表明在强化学习后训练过程中，绝大多数参数几乎未发生改动。此外，即便是存在明显变化的参数，其更新幅度也保持在较小水平。该结果说明，域强化学习并非对模型参数进行全局重写，而是在基础模型的参数基础上施加轻微扰动。我们在附录进一步展示了对于顺序训练的参数分析。
% \subsection{Domain Edits Target Weakly Overlapping Neuron Sets}
\subsection{Weak Edit Overlap}
% Beyond parameter-level sparsity, the analysis further examines which neurons are modified by domain RL. Each MLP intermediate channel is treated as a neuron.
\begin{wrapfigure}{r}{0.37\textwidth}
\vspace{-2.0em}
\centering
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=\linewidth]{a_jaccard_heatmap.pdf}
\caption{Changed-neuron overlap.}
\label{fig:jaccard_heatmap}
\end{subfigure}
\vspace{0.2em}
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=\linewidth]{c_competition_heatmap.pdf}
\caption{Active-neuron overlap.}
\label{fig:competition_heatmap}
\end{subfigure}
\caption{Neuron-overlap rates under different settings.}
\label{fig:overlap_comparison}
\vspace{-3.5em}
\end{wrapfigure}
Since domain RL updates are sparse, a natural explanation for strong interference is direct co-editing: different domains may concentrate their large updates on the same set of functional units. We test this explanation by lifting the analysis from parameters to MLP neurons and measuring the overlap among the most strongly changed neurons across domain experts.
We treat each MLP intermediate channel as a neuron and score its change by aggregating the parameter differences associated with its gate, up, and down projections. For each layer, we select the top 10\% most changed neurons for each domain expert and compute the pairwise Jaccard overlap between these sets. Full definitions are provided in Appendix~\ref{app:neuron_definition}.
% For the $i$-th MLP channel in layer $\ell$, we define its associated parameters as the corresponding row of the gate projection, the corresponding row of the up projection, and the corresponding column of the down projection, $\vg_i^{(\ell)} = \mW_{\mathrm{gate}}^{(\ell)}[i,:],\ \vu_i^{(\ell)} = \mW_{\mathrm{up}}^{(\ell)}[i,:],\ \vd_i^{(\ell)} = \mW_{\mathrm{down}}^{(\ell)}[:,i]$. Let $\Delta \vg_i^{(\ell)}$, $\Delta \vu_i^{(\ell)}$, and $\Delta \vd_i^{(\ell)}$ denote the parameter changes of these three components between the expert and base models. We define the overall change magnitude of neuron $i$ in layer $\ell$ as $m_i^{(\ell)} = \sqrt{\|\Delta \vg_i^{(\ell)}\|_2^2 + \|\Delta \vu_i^{(\ell)}\|_2^2 + \|\Delta \vd_i^{(\ell)}\|_2^2}$. For each domain $d$, let $S_d^{(\ell)}$ be the Top-10\% neurons ranked by $m_i^{(\ell)}$ within layer $\ell$. We then measure cross-domain overlap using the layer-wise Jaccard coefficient $J_{d_i,d_j}^{(\ell)} = \frac{|S_{d_i}^{(\ell)} \cap S_{d_j}^{(\ell)}|}{|S_{d_i}^{(\ell)} \cup S_{d_j}^{(\ell)}|}$, averaged over layers: $J_{d_i,d_j} = \frac{1}{L}\sum_{\ell=1}^{L} J_{d_i,d_j}^{(\ell)}$.
Figure~\ref{fig:jaccard_heatmap} shows that top-changed neurons overlap only weakly across domains, with average Jaccard coefficients below 0.19 for all domain pairs. This suggests that domain RL updates target distinct local neuron subsets rather than a common large set of neurons. Therefore, cross-domain interference is unlikely to arise simply from large-scale co-editing of the same neurons.
This leads to a key question: if different domains edit largely non-overlapping neurons, why does later-domain training still produce clear cross-domain effects?
% We address this in the next section by analyzing task activation routes.
% \begin{summarybox}
% Top-changed neurons overlap weakly across domains.
% \end{summarybox}
% 除参数层级稀疏性外，我们进一步分析域强化学习会对哪些神经元产生修改。我们将多层感知机的每一个中间通道视作一个神经元，量化计算每一层中各个神经元的参数变化幅度。针对每个域，我们在每一层中筛选出参数变化幅度排名前10%的神经元作为高变化神经元，并采用杰卡德系数衡量不同域之间这类神经元的重合程度。
% 如图X所示，不同域间的高变化神经元重合度很低。所有域组合的平均杰卡德系数均低于0.19。这表明域强化学习的参数更新并不会集中作用于同一大批神经元，而是倾向于修改各自相对独立的局部神经元子集。换言之，跨域干扰并不能简单归因于对相同神经元的大规模共同调整。
% 由此引出一个关键问题：既然不同域仅对重合度极低的神经元进行针对性调整，为何后续的域训练仍会产生明显的跨域影响？我们将在下一节通过分析任务激活路径来解答这一问题。
% \subsection{Sparse Edits Can Act on Shared Computation Routes}
\subsection{Act on Shared Computation Routes}
The previous section shows weak overlap among top-changed neurons across domains. But low edit overlap does not imply functional independence, so top-active neurons are examined to test whether domains share active routes during inference.
To test this possibility, we measure inference-time activation overlap. Using the same MLP-neuron definition as above, we rank neurons in each layer by their average activation magnitude on each domain and select the top 5\% most active neurons. We then compute the pairwise Jaccard overlap of these active-neuron sets across domains. Full metric definitions are provided in Appendix~\ref{app:active_route_definition}.
% Using the same neuron definition as above, for domain $d$ and layer $\ell$, let $h_{i,t}^{(\ell)}(x)$ denote the activation of neuron $i$ at token position $t$ for sample $x \in \mathcal{D}_d$. The sample-level average activation magnitude of neuron $i$ in layer $\ell$ is defined as $\bar{a}_{d,i}^{(\ell)}(x) = \frac{1}{T_x}\sum_{t=1}^{T_x} |h_{i,t}^{(\ell)}(x)|$.
% We define its dataset-level activation score as $a_{d,i}^{(\ell)} = \frac{1}{|\mathcal{D}_d|}\sum_{x \in \mathcal{D}_d} \bar{a}_{d,i}^{(\ell)}(x)$. For each domain $d$, let $A_d^{(\ell)}$ be the Top-5\% neurons ranked by $a_{d,i}^{(\ell)}$ within layer $\ell$. Cross-domain overlap is then measured using the layer-wise Jaccard coefficient $J_{d_i,d_j}^{\mathrm{act},(\ell)} = \frac{|A_{d_i}^{(\ell)} \cap A_{d_j}^{(\ell)}|}{|A_{d_i}^{(\ell)} \cup A_{d_j}^{(\ell)}|}$, averaged over layers: $J_{d_i,d_j}^{\mathrm{act}} = \frac{1}{L}\sum_{\ell=1}^{L} J_{d_i,d_j}^{\mathrm{act},(\ell)}$.
Figure~\ref{fig:competition_heatmap} shows that, among reasoning-oriented domains, active-neuron overlap is much higher than changed-neuron overlap. Math, Code, and QA exhibit higher mutual overlap and therefore share more active computation routes, whereas CW remains relatively independent.
Thus, low edit overlap does not preclude functional coupling: sparse updates can still affect other domains through shared active routes.
% \begin{summarybox}
% Sparse updates can generate cross-domain impacts by sharing active computational pathways.
% \end{summarybox}
% 上一节表明，跨领域之间变化最显著的神经元重叠度较低。然而，编辑重叠度低并不意味着各领域在功能上相互独立。为探究不同领域在推理过程中是否共享计算通路，我们对激活度最高的神经元展开分析。具体而言，针对每个专家模型，在其对应领域数据上执行前向传播，收集多层感知机的中间激活值，并在每一层中选取平均激活幅度排名前5%的神经元。
% 如图X所示，侧重推理的领域之间，激活神经元的重叠度远高于变化神经元的重叠度。数学问答领域的雅卡尔系数达到0.515，代码-数学领域与代码-问答领域分别为0.449和0.440。相比之下，创意写作领域与其他领域的激活重叠度更低，数值区间为0.263至0.296。
% 这一结果说明，数学、代码与问答领域共享大量高激活计算通路，而创意写作领域则更多依赖差异化的专属计算通路。因此，编辑重叠度低并不代表功能耦合性弱。若某一领域的稀疏更新作用于两个领域共用的激活计算通路，该更新仍可对另一领域产生影响。
% \subsection{Directional Alignment Determines Synergy or Conflict on Shared Routes}
\subsection{Directionality on Shared Routes}
\begin{figure*}[t]
\centering
\includegraphics[width=0.9\linewidth]{b_all_pairs_mean_cosine.pdf}
\caption{Layer-wise average directional cosine on shared top-changed neurons across domain pairs.}
\label{fig:all_pairs_mean_cosine}
\end{figure*}
% The previous section shows that different domains often share highly active computation routes. But shared routes do not automatically cause interference.
% When two domains update these shared neurons in similar directions, the routes can support positive transfer; when the directions conflict, the same routes can drive cross-domain interference.
% To analyze this, neuron-level update directions on shared edited neurons are examined. Using the same neuron definition as in the previous subsection, for domain $d$ and layer $\ell$, let $S_d^{(\ell)}$ denote the Top-10\% changed neurons defined above. For neuron $i \in S_d^{(\ell)}$, its update vector is defined as the concatenation of the three parameter changes associated with that neuron, $\Delta \vv_{d,i}^{(\ell)} = \bigl[\Delta \vg_i^{(\ell)};\, \Delta \vu_i^{(\ell)};\, (\Delta \vd_i^{(\ell)})^\top\bigr]$. For a pair of domains $d_i$ and $d_j$, let $Q_{d_i,d_j}^{(\ell)} = S_{d_i}^{(\ell)} \cap S_{d_j}^{(\ell)}$ be the set of shared top-changed neurons in layer $\ell$. The layer-wise average directional cosine on shared neurons is then $C_{d_i,d_j}^{(\ell)} = \frac{1}{|Q_{d_i,d_j}^{(\ell)}|}\sum_{k \in Q_{d_i,d_j}^{(\ell)}} \frac{\langle \Delta \vv_{d_i,k}^{(\ell)}, \Delta \vv_{d_j,k}^{(\ell)} \rangle}{\|\Delta \vv_{d_i,k}^{(\ell)}\|_2\,\|\Delta \vv_{d_j,k}^{(\ell)}\|_2}$. A positive value indicates that the two domains edit shared neurons in aligned directions on average, while a negative value indicates conflicting directions.
The previous section shows that different domains often share highly active computation routes. But shared routes do not automatically cause interference. We therefore examine the directional alignment of edits on shared route components. For each pair of domain experts, we select the top 10\% most changed neurons per domain, take their intersection, and compute the cosine similarity between the corresponding neuron-level update vectors. Full metric definitions are provided in Appendix~\ref{app:directional_alignment_metric}.
Figure~\ref{fig:all_pairs_mean_cosine} shows that domain pairs exhibit distinct directional patterns on shared neurons. Code-Math is predominantly aligned, with the average cosine staying clearly positive. By contrast, Math-QA is not globally antagonistic but splits by layer: the average cosine is negative in layers L3--6 and positive in layers L14--21.
% CW and the other domains stay closer to zero, indicating neither stable synergy nor conflict.
% \begin{summarybox}
% Directional alignment within shared routes distinguishes synergy from conflict.
% \end{summarybox}
% 上一节表明，不同领域可以共享大量高活跃计算路径。然而，仅存在共享并不一定意味着会产生干扰。若两个领域以相近的方向更新共享神经元，共享路径可能会产生协同效应，反之则可能成为跨领域干扰的来源。
% 为对此展开分析，我们研究了共享神经元上的任务向量方向。具体而言，对于两个领域共享的变化幅度最大的神经元，我们从对应的领域专家模块中提取其参数变化向量，并计算余弦相似度。相似度为正值，表示共享神经元的参数更新方向一致；相似度为负值，则代表参数更新存在冲突。
% 如图X所示，不同领域组合在共享神经元上呈现出截然不同的方向分布规律。代码-数学领域组合的方向一致性更强，尤其在第6至18中间层，约70%–80%的共享神经元呈现同向更新。相比之下，数学-问答领域组合不存在全局对立特征，但呈现明显的分层差异：在第3至5浅层网络中，大部分共享神经元更新方向相反，而第14至21中间层则表现出较强的方向一致性。常识写作领域与其他领域的相似度分布接近随机水平，说明其与推理类领域之间既无稳定的协同关系，也不存在固定的冲突关系。
\textbf{Taken together}, these analyses give a clear picture of cross-domain RL interference. Global gradients are nearly orthogonal, domain-specific RL updates are sparse and small, and different domains affect largely distinct neuron sets. However, sparse edits can produce cross-domain effects through these routes. The direction of updates on shared routes then determines whether the effect is synergistic or conflicting. These findings motivate the theory developed next: cross-domain interference is better modeled as a localized conflict on shared active computation routes. These signals also motivate a later direct rollback experiment based on shared activation, update magnitude, and directional conflict.
% 综合来看，这些分析揭示了领域强化学习干扰的结构化特征。全局梯度几乎相互正交，这表明跨领域性能退化并非由全模型梯度拮抗所导致。领域强化学习的更新具有稀疏性且幅度较小，不同领域所调控的神经元集合仅有微弱重叠。然而，编辑重叠度低并不意味着功能相互孤立：以推理为导向的各个领域共享高活跃度的计算通路，稀疏的参数编辑仍可通过这些通路产生跨领域影响。最后，这些共享通路内部的作用方向性，决定了此类影响是协同增益还是相互冲突。以上研究发现为后续构建理论框架奠定了基础：跨领域干扰更适合建模为**共享活跃计算通路内部的局部冲突分量**，而非全局性的灾难性遗忘。
\section{Theory: Local Recoverability under Sparse Low-Dimensional Interference}
\label{sec:theory}
% This section gives a local explanation of the empirical pattern in Section~\ref{sec:ana}. First, later-domain updates in sequential training are shown to mainly cause second-order damage to an earlier domain. Next, this damage is argued to concentrate in a low-dimensional shared conflict subspace. Finally, a short refresh is shown to contract this harmful component. Only the main assumptions and proof sketches appear here; longer discussion and complementary extensions are deferred to Appendix~\ref{app:det_proof}.
The structural evidence above suggests three constraints that any explanation of cross-domain interference should satisfy. First, interference is not well explained by full-model gradient conflict, since domain gradients can be nearly orthogonal. Second, domain RL writes sparse and small-magnitude edits, and different domains modify weakly overlapping neuron sets. Third, sparse edits can still affect other domains because different domains reuse shared active routes, and the direction of edits on these routes determines synergy or conflict.
We formalize these observations with a local perturbation model. Later-domain training can damage an earlier domain by moving along curvature-sensitive conflict directions, while a short refresh contracts this harmful component and enables fast, selective recovery.
\subsection{Notation and Structural Assumptions}
We write $L_d(\vtheta)$ as the local objective for domain $d$, with gradient $\vg_d(\vtheta)$ and Hessian $\mH_d(\vtheta)$. Suppose training on domain $A$ selects a checkpoint $\vtheta_A^*$, and later training on domain $B$ induces a local update $\vdelta_B$, producing $\vtheta_A^*+\vdelta_B$. We measure interference from $B$ to $A$ by the increase in the earlier-domain objective: $\Delta_{A\leftarrow B} = L_A(\vtheta_A^*+\vdelta_B)-L_A(\vtheta_A^*)$. The goal is to characterize when this quantity is large, why it is selective across domains, and why a short refresh on $A$ can reduce it.
% Motivated by the structural evidence in Section~\ref{sec:ana}, sequential multi-domain RL is modeled as a sequence of local perturbations in parameter space. Let $\vtheta\in\mathbb{R}^p$ denote the model parameters, and let $\mathcal{D}=\{\mathrm{Math}, \mathrm{Code}, \mathrm{QA}, \mathrm{CW}\}$ be the set of domains. For each domain $d\in\mathcal{D}$, $L_d(\vtheta)$ denotes a local domain objective around the current policy. Its gradient and Hessian are $\vg_d(\vtheta)=\nabla L_d(\vtheta),\ \mH_d(\vtheta)=\nabla^2 L_d(\vtheta)$. Suppose that training on domain $A$ reaches a validation-selected checkpoint $\vtheta_A^*$. Continuing RL training on a later domain $B$ induces an update $\vdelta_B=\vtheta_{A\rightarrow B}-\vtheta_A^{*}$, so that the new parameter vector is $\vtheta_A^*+\vdelta_B$. The interference from domain $B$ to domain $A$ is measured by the increase in the previous-domain objective: $\Delta_{A\leftarrow B} = L_A(\vtheta_A^*+\vdelta_B)-L_A(\vtheta_A^*)$. The goal is to characterize when $\Delta_{A\leftarrow B}$ is large, why it is selective across domains, and why a short refresh on domain $A$ can recover the lost performance.
Under standard local smoothness, we use three structural conditions motivated by Section~\ref{sec:ana}:
(i) the selected domain-$A$ checkpoint is approximately stationary for $L_A$;
(ii) later-domain updates are local and effectively sparse;
and (iii) the curvature-sensitive part of the later update is concentrated in a low-dimensional shared active conflict subspace $S_{A,B}$.
For the refresh analysis, we additionally assume positive curvature of $L_A$ restricted to $S_{A,B}$ and weak coupling from the orthogonal complement back into this subspace.
Formal statements of these assumptions are provided in Appendix~\ref{app:theory_assumptions}.
% \subsection{Sequential Updates Induce Second-Order Local Damage}
\subsection{Second-Order Local Damage}
We first explain why later-domain training can harm an earlier domain even without strong global gradient opposition. After training on domain $A$, the selected checkpoint $\vtheta_A^*$ is approximately stationary for the local objective $L_A$. Therefore, when a later domain $B$ induces a local update $\vdelta_B$, the first-order change in $L_A$ is small, and the leading effect is governed by second-order local curvature.
\paragraph{Proposition 1.}
\label{prop:second_order_damage}
\textit{Under local smoothness and approximate stationarity of $\vtheta_A^*$, for a later-domain update $\vdelta_B$ with $\|\vdelta_B\|_2\le r$, the interference from $B$ to $A$ satisfies}
\begin{equation}
\Delta_{A\leftarrow B} = \frac12\vdelta_B^\top\mH_A(\vtheta_A^*)\vdelta_B +O\!\left(\varepsilon_A\|\vdelta_B\|_2+\|\vdelta_B\|_2^3\right).
\end{equation}
This result shows that earlier-domain degradation depends mainly on whether the later-domain parameter update moves along high-curvature directions of the earlier-domain objective. In other words, a sparse and small-magnitude update can still cause substantial degradation if it moves along curvature-sensitive directions of $L_A$. This explains why near-orthogonal full-model gradients do not rule out cross-domain interference: the harmful effect can appear as a localized second-order displacement rather than a global first-order gradient conflict.
The full proof, along with the equivalent sensitivity-based form, is provided in Appendix~\ref{app:proof_second_order}.
% \subsection{Selective Interference in a Low-Dimensional Conflict Subspace}
\subsection{Low-Dimensional Conflict Subspace}
Proposition~1 shows that later-domain damage is governed by the local curvature of the earlier-domain objective. The next question is where these curvature-sensitive harmful directions lie. The structural evidence in Section~\ref{sec:ana} suggests that they are not spread across the full parameter space: domain RL updates are sparse, different domains edit weakly overlapping neuron sets, but they can still interact through shared active routes. We therefore model the harmful component of a later-domain update as concentrated in a low-dimensional shared active conflict subspace $S_{A,B}$.
\paragraph{Proposition 2.}
\label{prop:conflict_subspace}
\textit{Let $\mP_S$ denote the projection onto the shared active conflict subspace $S_{A,B}$. Under the local structural conditions above, interference from domain $B$ to domain $A$ satisfies}
\begin{equation}
\Delta_{A\leftarrow B} = \frac{1}{2}(\mP_S\vdelta_B)^\top \mH_A(\vtheta_A^*) (\mP_S\vdelta_B) + O\left( \varepsilon_A\|\vdelta_B\|_2 + \gamma_A\|\vdelta_B\|_2^2 + \|\vdelta_B\|_2^3 \right).
\end{equation}
This result localizes the second-order damage identified in Proposition~1. The leading harmful term depends only on the projection of the later-domain update onto $S_{A,B}$, up to controlled residual terms. Thus, low edit overlap does not imply low interference: even if two domains modify largely different neuron sets, a later-domain update can still harm an earlier domain if its projection onto the shared active conflict subspace is large. This also explains why full-model near-orthogonality can coexist with selective degradation. When global first-order conflict is weak, the main damage can still arise from localized second-order displacement along shared curvature-sensitive directions. It also motivates the weaker empirical test used later in Section~\ref{sec:proxy_conflict_subspace}: if a sparse coordinate proxy captures a nontrivial portion of this harmful projection, then reverting the corresponding part of the later-domain update should partially reduce the damage, even when the proxy is only basis-aligned rather than a direct estimate of $S_{A,B}$. We provide the full decomposition and proof in Appendix~\ref{app:proof_conflict_subspace}.
\subsection{Short Refresh Geometrically Contracts the Conflict Component}
Proposition~2 shows that earlier-domain degradation is mainly controlled by the projection of the later-domain update onto the shared active conflict subspace $S_{A,B}$. This suggests that recovery does not require undoing the entire later-domain update. Instead, a short refresh on domain $A$ only needs to reduce the harmful component inside $S_{A,B}$.
Starting from the degraded checkpoint $\vtheta_0=\vtheta_A^*+\vdelta_B$, we consider a short refresh on domain $A$ with gradient descent: $\vtheta_{t+1}=\vtheta_t-\alpha \vg_A(\vtheta_t)$.
% Let $\ve_t=\vtheta_t-\vtheta_A^*$ denote the displacement from the original domain-$A$ checkpoint, and let $\mP_S\ve_t$ be its component in the shared conflict subspace.
Under positive curvature of $L_A$ restricted to $S_{A,B}$ and weak coupling from $S_{A,B}^{\perp}$ back into $S_{A,B}$, this component contracts geometrically. The formal assumption is provided in Appendix~\ref{app:theory_assumptions}.
\paragraph{Theorem 1.}
\label{thm:refresh_contraction}
\textit{Starting from $\vtheta_0=\vtheta_A^*+\vdelta_B$, a short refresh on domain $A$ satisfies}
\begin{equation}
\|\mP_S(\vtheta_t-\vtheta_A^*)\|_2 \le (1-\alpha\mu_A)^t \|\mP_S\vdelta_B\|_2,
\end{equation}
where $\mu_A>0$ is the local curvature lower bound on $S_{A,B}$. See Appendix~\ref{app:proof_refresh} for the proof.
The theorem shows that the harmful component decays geometrically with the number of refresh steps. This gives a local explanation for fast recovery: early refresh steps can rapidly remove the component responsible for earlier-domain degradation, without requiring full retraining or reversing the whole later-domain update. It also explains why recovery can be selective. Since the refresh mainly contracts the component to which domain $A$ is sensitive, it can restore domain $A$ while causing only bounded collateral damage to other domains under global near-orthogonality; the detailed proof is provided in Appendix~\ref{app:proof_other_domains}.
\textbf{Taken together}, Proposition~1, Proposition~2, and Theorem~1 give a local explanation for selective interference and recovery. Later-domain training harms an earlier domain mainly through a localized second-order displacement in shared conflict directions, while a short refresh recovers the domain by geometrically contracting this harmful component.
This view yields two complementary empirical predictions: a short refresh should selectively recover the damaged domain, and a sparse rollback on a coordinate proxy for the harmful component should partially reduce the damage without full retraining. We further discuss an alternating-refresh extension in Appendix~\ref{app:proof_alt_refresh}. The extension shows that one cycle of alternating refresh is, to first order, equivalent to a descent step on a weighted multi-domain objective. Under the standard weighted-sum view of multi-objective optimization, this suggests that repeated refresh moves toward a local Pareto-stationary compromise among domains.
% Taken together, Proposition~1, Proposition~2, and Theorem~1 show that sequential training harms an earlier domain mainly through localized second-order displacement, while refresh helps by contracting this displacement along shared conflict directions. This matches the fast Re-Math recovery.
% Taken together, Proposition~1, Proposition~2, and Theorem~1 give a local explanation for selective interference and recovery. Later-domain training harms an earlier domain mainly through a localized second-order displacement in shared conflict directions, while a short refresh recovers the domain by geometrically contracting this harmful component.
% We further discuss an alternating-refresh extension in Appendix~\ref{app:proof_alt_refresh}. The extension shows that one cycle of alternating refresh is, to first order, equivalent to a descent step on a weighted multi-domain objective. Under the standard weighted-sum view of multi-objective optimization, this suggests that repeated refresh moves toward a local Pareto-stationary compromise among domains.
% We defer two extension results to the appendix. Appendix~\ref{app:proof_other_domains} shows that small refresh steps can recover the target domain while only mildly affecting other domains. Appendix~\ref{app:proof_alt_refresh} shows that alternating refresh behaves, to first order, like optimizing a weighted mixture of domain objectives, leading to a locally balanced trade-off rather than a single-domain optimum.
% Two complementary extensions are deferred to the appendix. First, Appendix~\ref{app:proof_other_domains} shows that under global near-orthogonality, a small refresh step can recover the target domain while causing only limited degradation to other domains, both for a single step and over a short trajectory. Second, Appendix~\ref{app:proof_alt_refresh} shows that alternating refresh across domains is first-order equivalent to optimizing a weighted sum of domain objectives, which clarifies why repeated refresh can move the model toward a locally balanced trade-off rather than back to any single-domain optimum.
% These appendix results extend the main theorem without interrupting the core argument in the main text.
\begin{table*}[t]
\centering
\small
\caption{Performance of single-domain experts, sequential training, mixed-domain baselines, and Re-Math refresh.}
\label{tab:task_level_main}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|c|cccc|cccc|ccc}
\toprule
Task
& Base
& $\mathrm{Code}_s$ & $\mathrm{Math}_s$ & $\mathrm{QA}_s$ & $\mathrm{CW}_s$
& $\mathrm{Code}_o$ & $\mathrm{Math}_o$ & $\mathrm{QA}_o$ & $\mathrm{CW}_o$
& CGPO & JT & Re-Math \\
\midrule
Math
& 43.19 & 59.63 & \textbf{66.84} & 55.31 & 39.78
& 59.63 & 66.49 & 59.90 & 57.66
& 61.93 & 64.80 & 66.04 \\
Code
& 29.57 & \textbf{52.67} & 34.65 & 32.07 & 28.15
& \textbf{52.67} & 50.69 & 50.99 & 50.47
& 50.05 & 48.61 & 51.05 \\
QA
& 60.64 & 60.89 & 60.76 & \textbf{63.31} & 60.50
& 60.89 & 60.52 & 62.34 & 62.34
& 62.48 & 62.11 & 62.49 \\
CW
& 82.44 & 82.40 & 81.38 & 81.76 & 86.24
& 82.40 & 81.44 & 81.79 & 86.52
& 86.73 & \textbf{86.97} & 85.96 \\
\midrule
AVG
& 53.96 & 63.90 & 60.91 & 58.11 & 53.67
& 63.90 & 64.79 & 63.76 & 64.25
& 65.30 & 65.62 & \textbf{66.39} \\
\bottomrule
\end{tabular}
}
\end{table*}
\section{Task-Level Validation and Direct Intervention}
\label{sec:exp}
% This section validates the phenomena and theoretical implications developed above at the task level.
% This section validates the theory at two levels: task-level recovery under short refresh, and direct weight-space intervention on a proxy conflict coordinates for the Math$\rightarrow$QA pair.
This section validates the theory at two levels: task-level recovery under short refresh, and a direct weight-space rollback on a coordinate proxy for the conflict subspace in the fixed Math$\rightarrow$QA pair.
% \subsection{Selective Interference and Local Recovery}
\subsection{Recovery by Short Refresh}
\textbf{Main result.} Table~\ref{tab:task_level_main} shows the domain-level results of single-domain experts, sequential checkpoints, mixed-training baselines, and $\mathrm{Re\text{-}Math}$, obtained by a short Math refresh from $\mathrm{CW}_o$. Full benchmark-level results are in Appendix~\ref{app:full_results}.
\textbf{Selective interference.} Sequential training does not cause uniform forgetting. After Code $\rightarrow$ Math, $\mathrm{Math}_o$ reaches 66.49 on Math, close to the single-domain Math expert (66.84). After later QA and CW training, however, Math drops to 57.66, while the other domains remain largely stable or improve primarily during their own stages. This shows that interference is domain-specific rather than uniform.
\textbf{Fast recovery.} A short Math refresh from $\mathrm{CW}_o$ raises Math from 57.66 to 66.04, recovering most of the loss from later-domain training and bringing performance close to both $\mathrm{Math}_o$ and the single-domain math expert. This suggests that $\mathrm{Re\text{-}Math}$ acts as a local correction.
\textbf{Limited side effects.} While recovering Math, $\mathrm{Re\text{-}Math}$ leaves the other domains nearly unchanged: Code rises slightly to 51.05, while QA and CW remain essentially unchanged. This pattern also highlights asymmetric sensitivity: Math is more sensitive to the later QA/CW updates than the other domains are to the short Re-Math correction. As a result, $\mathrm{Re\text{-}Math}$ attains the best average score, 66.39. See Appendices~\ref{app:refresh_dynamics} and~\ref{app:asym_directionality} for further analysis and validation.
% Overall, the task-level results support our theory: multi-domain RL causes selective interference, and a short domain refresh largely reverses it with little effect on other domains.
\subsection{Conflict Subspace Proxy}
\label{sec:proxy_conflict_subspace}
The refresh results above validate recoverability, but they do not by themselves directly probe the localization claim in Proposition~2, which attributes the dominant damage from a later domain to the projection of its update onto a shared active conflict subspace. We perform a training-free weight-space rollback on the checkpoint pair $\mathrm{Math}_o \rightarrow \mathrm{QA}_o$ to isolate the QA$\rightarrow$Math interaction.
We first study an MLP-only coordinate proxy based on three signals: shared activation under Math and QA (A), the magnitude of the QA update (M), and directional conflict between the Math and QA task vectors (C). Treating checkpoints as parameter vectors, define the QA-induced displacement as $\vdelta_{Q\mid M}=\mathrm{QA}_o-\mathrm{Math}_o$. We then revert only the QA increment on the selected neurons: $\vtheta_{\mathrm{rev}}=\mathrm{QA}_o-\mP_{\hat S}\vdelta_{Q\mid M}$, where $\mP_{\hat S}$ denotes the selected neurons. This rollback removes a proxy for part of the harmful component in Proposition~2, without identifying the latent subspace.
Table~\ref{tab:proxy_conflict_revert} shows that the rollback is selective. Reverting only $2\%$ of MLP neurons selected by the composite score $A\times M\times C$ raises Math Avg from $59.90$ to $61.25$, recovering $20.4\%$ of the QA-induced Math loss while changing QA Avg by only $-0.06$; by contrast, reverting the same number of randomly selected neurons decreases Math Avg to $59.49$. The additional ablations in Appendix~\ref{app:proxy_conflict_appendix} indicate partial redundancy among the localization signals. Under the fixed-budget protocol, A-only matches the full selector on Math recovery, whereas recomputing the layer budget lowers recovery; among the two-factor variants, $M\times C$ remains the closest approximation to the full selector. We also extend the intervention to attention coordinates. As Figure~\ref{fig:proxy_conflict_beta} shows, as the budget increases, the joint MLP+Attn selector recovers more damage beyond MLP coordinates, reaching $73.6\%$ recovery at a $32\%$ budget. Together, these results suggest that, for this fixed Math-QA interaction, the harmful QA-induced displacement is localized enough that a sparse coordinate-proxy rollback can recover a substantial portion of the Math damage without retraining.
\textbf{Overall}, the task-level and intervention results support our theory: multi-domain RL causes selective interference, short refresh largely reverses it with limited side effects, and a small targeted rollback on a coordinate proxy for the conflict subspace recovers a substantial portion of the QA-induced Math loss without retraining.
\begin{table}[t]
\small
\centering
\caption{Selective rollback on a coordinate proxy for the conflict subspace from $\mathrm{QA}_o$ toward $\mathrm{Math}_o$. The $\mathrm{Math}_o$ row gives the pre-QA reference upper bound. Joint MLP+Attn includes attention layers; all other non-baseline selectors use MLP neurons only.}
\label{tab:proxy_conflict_revert}
\begin{tabular}{lccccc}
\toprule
Selector & Budget & Math Avg & $\Delta$ vs $\mathrm{QA}_o$ & Recovery & $\Delta$ QA Avg \\
\midrule
$\mathrm{Math}_o$ & -- & 66.49 & +6.59 & 100.0\% & -1.81 \\
$\mathrm{QA}_o$ & -- & 59.90 & -- & 0.0\% & 0.00 \\
Random & 2\% & 59.49 & -0.42 & -6.3\% & +0.01 \\
A$\times$M & 2\% & 60.53 & +0.63 & 9.6\% & +0.04 \\
A$\times$C & 2\% & 60.55 & +0.65 & 9.9\% & -0.05 \\
M$\times$C & 2\% & 61.11 & +1.21 & 18.3\% & +0.04 \\
A$\times$M$\times$C & 2\% & \textbf{61.25} & \textbf{+1.35} & \textbf{20.4\%} & -0.06 \\
\midrule
Joint MLP+Attn & 32\% & \textbf{64.75} & \textbf{+4.85} & \textbf{73.6\%} & -0.45 \\
\bottomrule
\end{tabular}
\end{table}
\section{Conclusion}
\label{sec:conclusion}
This work studies cross-domain interference in multi-domain RL and shows that it is not a full-model phenomenon: global domain gradients can be nearly orthogonal, yet conflict arises locally on shared active computation routes. Although RL updates are sparse, small, and have little overlap across domains, reasoning domains still reuse common routes, and directional misalignment on those routes determines whether later training helps or harms earlier-domain performance.
The theoretical analysis further shows that this damage is mainly a local second-order effect concentrated in a low-dimensional shared conflict subspace, which means a short refresh on the damaged domain can contract the harmful component while preserving other domains.
Empirically, both short refresh and sparse proxy rollback can restore damaged performance while largely preserving other domains. These results suggest that controlling localized route-level interference is a promising direction toward more stable and scalable multi-domain RL.
% \begin{ack}
% Use unnumbered first level headings for the acknowledgments. All acknowledgments
% go at the end of the paper before the list of references. Moreover, you are required to declare
% funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
% More information about this disclosure can be found at: \url{https://neurips.cc/Conferences/2026/PaperInformation/FundingDisclosure}.
% Do {\bf not} include this section in the anonymized submission, only in the final paper. You can use the \texttt{ack} environment provided in the style file to automatically hide this section in the anonymized submission.
% \end{ack}
% \nocite{*}
%
\section*{Limitations}
This work has several limitations and also suggests natural future directions. First, although our results indicate a simple practical route for multi-domain RL, we have not yet developed it into an automatic training algorithm. A key implication is that stable multi-domain improvement may not require delicate data-mixture tuning or hand-designed replay schedules: after sequential domain training, a short targeted refresh on the most degraded domains can quickly recover performance with limited side effects. Future work should automate the detection of degraded domains, the selection of refresh order and budget, and examine whether repeated targeted refresh consistently approaches a local Pareto-stationary compromise.
Second, our conflict-subspace intervention is still based on a coarse proxy. Although the 32\% rollback budget is relatively large, it should be interpreted primarily as a causal sufficiency test rather than as an estimate of the minimal conflict subspace. Because the proxy coordinates are likely redundant, comparable recovery may be achievable with a smaller and more precisely identified intervention set. Moreover, the current proxy is basis-aligned and does not directly estimate a latent rotated subspace, nor does it cover all potentially relevant factors such as normalization parameters, residual-stream couplings, or higher-order cross-module interactions. A natural next step is to identify the conflict subspace more precisely and use it for projection-based training, constrained updates, or route-aware regularization, so that harmful cross-domain components can be suppressed with a smaller and less redundant intervention budget.
Finally, our analysis focuses on multi-domain RL, while similar training dynamics may also arise in other post-training paradigms. For example, on-policy distillation also involves evolving training distributions and policy-dependent data, where interference, instability, or route-level specialization may emerge over training. Extending our diagnostic framework to such settings could help explain broader post-training dynamics beyond the specific multi-domain RL setup studied here.
% \bibliographystyle{plainnat}
\bibliographystyle{abbrv}
\bibliography{references}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \newpage
\clearpage
\appendix
\begin{table*}[!t]
\centering
\footnotesize
\caption{Training hyperparameters.}
\label{tab:selected_hyperparams}
\resizebox{\textwidth}{!}{%
\begin{tabular}{cccccc}
\toprule
\textbf{train batch size} & \textbf{ppo mini batch size} & \textbf{max prompt length} & \textbf{max response length} & \textbf{adv estimator} & \textbf{filter overlong prompts} \\
\midrule
256 & 256 & 2048 & 16384 & grpo & True \\
\midrule
\textbf{lr} & \textbf{use dynamic bsz} & \textbf{use kl loss} & \textbf{kl loss coef} & \textbf{kl loss type} & \textbf{clip ratio} \\
\midrule
1e-6 & True & False & 0.0 & low\_var\_kl & 0.2 \\
\midrule
\textbf{entropy coeff} & \textbf{rollout n} & \textbf{rollout temperature} & \textbf{rollout top k} & \textbf{rollout top p} & \textbf{val top k} \\
\midrule
0 & 8 & 1.0 & -1 & 1.0 & -1 \\
\midrule
\textbf{val top p} & \textbf{val temperature} & \textbf{val n} & \textbf{val do sample} & \textbf{use kl in reward} & \\
\midrule
0.9 & 0.7 & 8 & True & False & \\
\bottomrule
\end{tabular}
}
\end{table*}
\section{Implementation Details}
\subsection{Data Construction Details}
\label{app:data_const}
We ensure that the prompt length of all training data is within 2,048. Each domain uses 5,120 training examples. Validation sets are constructed from non-training examples in the corresponding data sources. For Math, we use AIME25~\citep{aops_aime} as the validation set, which contains 30 problems. For Code, QA, and CW, each validation set contains 50 examples.
For CW, we sample 5,120 examples from the crownelius/Creative-Writing series.\footnote{\url{https://huggingface.co/Crownelius}} The data come from four subsets: Sonnet4.6-800x,\footnote{\url{https://huggingface.co/datasets/Crownelius/Creative-Writing-Sonnet4.6-800x}} Gemini3Pro-2700x,\footnote{\url{https://huggingface.co/datasets/Crownelius/Creative-Writing-Gemini3Pro-2700x}} Reasoning-KimiK2.5-600x,\footnote{\url{https://huggingface.co/datasets/Crownelius/Creative-Writing-Reasoning-KimiK2.5-600x}} and Qwen3.5Plus-2000x.\footnote{\url{https://huggingface.co/datasets/Crownelius/Creative-Writing-Qwen3.5Plus-2000x}} For 2,560 of these examples, we resample the reference responses using Qwen3-235B-A22B-Instruct-2507 and use the newly sampled responses as references.
For evaluation, LiveCodeBench-v6 contains 175 problems spanning January 2025 to April 2025. For QA evaluation, after excluding the training and validation data from SuperGPQA, we sample 10\% of the remaining data, resulting in 2,141 test examples. Similarly, we sample 10\% of MMLU-Pro, resulting in 1,203 test examples.
% 我们保证所有训练数据的prompt长度在2048以内。
% 针对创意写作（CW）任务，我们从 crownelius/Creative-Writing 系列数据集中采样5120条样本，包含 Creative-Writing-Sonnet4.6-800x、Creative-Writing-Gemini3Pro-2700x、Creative-Writing-Reasoning-KimiK2.5-600x 以及 Creative-Writing-Qwen3.5Plus-2000x。我们从中选取2560条样本，通过 Qwen3-235B-A22B-Instruct-2507 模型重新采样参考回复，并将新生成的采样回复作为参考标准。
% 在评测环节中，LiveCodeBench-v6 数据集包含2025年1月至2025年4月期间的175道评测题目。问答（QA）评测方面，剔除 SuperGPQA 中的训练集与验证集数据后，我们对剩余数据抽取10%作为测试样本，最终得到2141条测试用例。同理，从 MMLU-Pro 数据集中抽取10%的数据，得到1203条测试样本。
\subsection{Training Hyperparameters}
\label{app:train_hyper}
All experiments use the same GRPO training configuration. The full hyperparameters are summarized in Table~\ref{tab:selected_hyperparams}.
At each training stage, we train the model until convergence and select the checkpoint with the optimal validation performance in the current training domain. Specifically, single-domain training adopts the best-performing checkpoint within the single domain, while multi-domain mixed training selects the checkpoint with the optimal average performance across all domains.
% 在每个训练阶段，我们将模型训练至收敛状态，并选取在当前训练领域验证效果最优的检查点，即单域训练选择单域最优，而多域混合则是多域平均最优。
\subsection{Reward Functions}
\label{app:reward_func}
Math and QA use answer-correctness rewards. The model is required to put the final answer in boxed. We use math\_verify to parse the answer and match it against the reference answer. The reward is binary: 1 for a correct answer and 0 otherwise.
Code uses execution-based rewards, computed from the fraction of test cases passed.
CW uses an LLM-as-a-judge preference reward. For crownelius/Creative-Writing series, given a prompt, a reference response, and a model response, the judge compares their quality. If the model response is better, the reward is 1; if the two responses are comparable, the reward is 0.5; if the reference response is better, the reward is 0. For each query of WritingBench, the model generates one response, which is scored along multiple WritingBench evaluation dimensions. We use Qwen3-235B-A22B-Instruct-2507~\citep{DBLP:journals/corr/abs-2505-09388} as the judge for training and evaluation.
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{cosine_heatmap_attn.pdf}
\caption{Attention-module heatmap of pairwise gradient cosine.}
\label{fig:cosine_heatmap_attn}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{cosine_heatmap_mlp.pdf}
\caption{MLP-module heatmap of pairwise gradient cosine.}
\label{fig:cosine_heatmap_mlp}
\end{subfigure}
\caption{Layer-wise heatmaps of pairwise gradient cosine in attention and MLP modules.}
\label{fig:grad_conflict_heatmaps}
\end{figure*}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{attn_mlp_conflict_top6_trend.pdf}
\caption{Top six attention and MLP modules with the strongest conflict.}
\label{fig:grad_conflict_top6}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{attn_mlp_synergy_top6_trend.pdf}
\caption{Top six attention and MLP modules with the strongest synergy.}
\label{fig:grad_synergy_top6}
\end{subfigure}
\caption{Module-level conflict and synergy trends across the most prominent attention and MLP modules.}
\label{fig:grad_conflict_trends}
\end{figure*}
\section{Additional Structural Analysis}
\subsection{Extended Module-Level Gradient Conflict and Synergy}
\label{app:grad_conf}
Figure~\ref{fig:grad_conflict_heatmaps} shows the six attention and MLP modules with the most prominent conflict and synergy, respectively. Figure~\ref{fig:grad_conflict_trends} presents the layer-wise heatmaps of pairwise gradient cosine in attention and MLP modules. These results show that conflicts are not uniformly distributed across all parameters but are concentrated in specific regions of the network. In particular, local conflicts are most significant for the Math-QA domain pair. In addition, certain local conflicts also exist between Code-CW. Nevertheless, despite relatively clear module interactions, the corresponding cosine values remain small and are largely orthogonal.
% 图Z展示了完整冲突、协同最强的6个attention和MLp模块，进一步地，图W展示了每层不同领域对冲突的热力图。可以看到，冲突并不是均匀分布在所有参数中，而是集中出现在网络的特定区域。尤其是 Math–QA 领域对中，局部冲突最为明显；此外，Code–CW 之间也存在一定局部冲突。不过即使存在较明显的模块作用，但其作用关系表现出来的cos值较小，依旧偏向正交。
\subsection{Sequential Parameter Changes Remain Sparse}
\label{app:para}
In addition to single-domain experts, we also measure parameter changes along the sequential training chain: Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW. We analyze both cumulative changes relative to the base model and incremental changes relative to the previous checkpoint.
As shown in Figure~\ref{fig:param_seq_vs_base}, cumulative deviation from the base model gradually increases as more domains are added. The fraction of parameters with $|\Delta W| < 10^{-7}$ and $r(W) < 10^{-3}$ decreases from 77.6\% after Code training to 73.4\% after Code-Math-QA-CW training.
However, as shown in Figure~\ref{fig:param_seq_incremental}, each individual stage remains sparse. When comparing each checkpoint with the immediately preceding one, later domain stages modify only a small fraction of parameters: 87.2\%, 84.0\%, and 88.3\% of parameters remain below the $10^{-7}$ threshold for the Math, QA, and CW stages, respectively.
These results show that sequential domain RL accumulates parameter shifts over domains, but each stage still acts as a sparse incremental update rather than a global parameter rewrite.
% 除单领域专家模型外，我们还沿着**代码→数学→问答→创意写作**的序贯训练链路测量参数变化。我们既分析相对于基础模型的累积变化，也分析相较于前一检查点的增量变化。
% 如图X所示，随着新增领域数量增多，模型参数相对基础模型的累积偏差会逐步增大。满足$|\Delta W| < 10^{-7}$且$r(W) < 10^{-3}$的参数占比，从代码训练后的77.6%下降至代码-数学-问答-创意写作全链路训练后的73.4%。
% 但每个独立训练阶段仍保持稀疏性。将每个训练检查点与其紧邻的前一检查点对比可以发现，后续领域训练仅会改动极小一部分参数：数学、问答、创意写作阶段分别有87.2%、84.0%、88.3%的参数仍低于$10^{-7}$阈值。
% 上述结果表明，领域序贯强化学习会跨领域累积参数偏移，但每一训练阶段仍属于**稀疏增量更新**，而非对全局参数进行重写。
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{01_abs_vs_base.pdf}
\caption{Absolute cumulative parameter changes relative to the base model.}
\label{fig:param_abs_vs_base}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{02_rel_vs_base.pdf}
\caption{Relative cumulative parameter changes relative to the base model.}
\label{fig:param_rel_vs_base}
\end{subfigure}
\caption{Cumulative parameter-change distributions along the sequential domain RL chain relative to the base model.}
\label{fig:param_seq_vs_base}
\end{figure*}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{03_abs_incremental.pdf}
\caption{Absolute incremental parameter changes relative to the previous checkpoint.}
\label{fig:param_abs_incremental}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{04_rel_incremental.pdf}
\caption{Relative incremental parameter changes relative to the previous checkpoint.}
\label{fig:param_rel_incremental}
\end{subfigure}
\caption{Incremental parameter-change distributions at each stage of the sequential domain RL chain.}
\label{fig:param_seq_incremental}
\end{figure*}
\subsection{Neuron-Level Edit Definition}
\label{app:neuron_definition}
For each MLP layer $\ell$, we define the $i$-th intermediate channel as one neuron. Its associated parameters consist of the corresponding row of the gate projection, the corresponding row of the up projection, and the corresponding column of the down projection:
\begin{equation}
\vg_i^{(\ell)} = \mW_{\mathrm{gate}}^{(\ell)}[i,:], \quad \vu_i^{(\ell)} = \mW_{\mathrm{up}}^{(\ell)}[i,:], \quad \vd_i^{(\ell)} = \mW_{\mathrm{down}}^{(\ell)}[:,i].
\end{equation}
For domain $d$, let $\Delta \vg_{d,i}^{(\ell)}$, $\Delta \vu_{d,i}^{(\ell)}$, and $\Delta \vd_{d,i}^{(\ell)}$ denote the corresponding parameter changes between the domain expert and the base model. We measure the edit magnitude of neuron $i$ in layer $\ell$ by aggregating the squared parameter changes over these three components:
\begin{equation}
s_{d,i}^{(\ell)} = \left\| \Delta \vg_{d,i}^{(\ell)} \right\|_2^2 + \left\| \Delta \vu_{d,i}^{(\ell)} \right\|_2^2 + \left\| \Delta \vd_{d,i}^{(\ell)} \right\|_2^2.
\end{equation}
For each layer and each domain, we select the top 10\% neurons with the largest edit magnitude:
\begin{equation}
\mathcal{N}_d^{(\ell)} = \operatorname{Top}_{10\%} \left\{s_{d,i}^{(\ell)}\right\}_{i}.
\end{equation}
The layer-wise overlap between two domains $A$ and $B$ is measured by the Jaccard coefficient:
\begin{equation}
J^{(\ell)}(A,B) = \frac{ |\mathcal{N}_A^{(\ell)} \cap \mathcal{N}_B^{(\ell)}| }{ |\mathcal{N}_A^{(\ell)} \cup \mathcal{N}_B^{(\ell)}| }.
\end{equation}
Finally, we report the average overlap across the $L$ MLP layers:
\begin{equation}
J(A,B) = \frac{1}{L} \sum_{\ell=1}^{L} J^{(\ell)}(A,B).
\end{equation}
\subsection{Active-Route Overlap Metric}
\label{app:active_route_definition}
Using the same MLP-neuron definition as in Appendix~\ref{app:neuron_definition}, we measure whether two domains activate overlapping active routes during inference. For domain $d$ and layer $\ell$, let $h_{d,i,t}^{(\ell)}(x)$ denote the activation of neuron $i$ at token position $t$ for sample $x \in \mathcal{D}_d$, and let $T_x$ be the number of tokens in $x$. We define the sample-level average activation magnitude as
\begin{equation}
\bar{a}_{d,i}^{(\ell)}(x) = \frac{1}{T_x} \sum_{t=1}^{T_x} \left|h_{d,i,t}^{(\ell)}(x)\right|.
\end{equation}
The dataset-level activation score is
\begin{equation}
a_{d,i}^{(\ell)} = \frac{1}{|\mathcal{D}_d|} \sum_{x\in \mathcal{D}_d} \bar{a}_{d,i}^{(\ell)}(x).
\end{equation}
For each layer and each domain, we select the top 5\% neurons with the largest dataset-level activation score:
\begin{equation}
\mathcal{A}_d^{(\ell)} = \operatorname{Top}_{5\%} \left\{a_{d,i}^{(\ell)}\right\}_{i}.
\end{equation}
The layer-wise active-route overlap between two domains $A$ and $B$ is measured by the Jaccard coefficient:
\begin{equation}
J_{\mathrm{act}}^{(\ell)}(A,B) = \frac{ |\mathcal{A}_{A}^{(\ell)} \cap \mathcal{A}_{B}^{(\ell)}| }{ |\mathcal{A}_{A}^{(\ell)} \cup \mathcal{A}_{B}^{(\ell)}| }.
\end{equation}
Finally, we report the average active-route overlap across the $L$ MLP layers:
\begin{equation}
J_{\mathrm{act}}(A,B) = \frac{1}{L} \sum_{\ell=1}^{L} J_{\mathrm{act}}^{(\ell)}(A,B).
\end{equation}
\subsection{Directional Alignment on Shared Edited Neurons}
\label{app:directional_alignment_metric}
For each domain $d$, layer $\ell$, and MLP neuron $i$, we define the neuron-level update vector by concatenating the parameter changes associated with its gate, up, and down projections:
\begin{equation}
\Delta \vv_{d,i}^{(\ell)} = \left[ \Delta \vg_{d,i}^{(\ell)}; \Delta \vu_{d,i}^{(\ell)}; \Delta \vd_{d,i}^{(\ell)} \right].
\end{equation}
For two domains $A$ and $B$, we define the shared edited neuron set in layer $\ell$ as
\begin{equation}
\mathcal{Q}_{A,B}^{(\ell)} = \mathcal{N}_{A}^{(\ell)} \cap \mathcal{N}_{B}^{(\ell)},
\end{equation}
where $\mathcal{N}_{d}^{(\ell)}$ is the Top-10\% changed-neuron set defined in Appendix~\ref{app:neuron_definition}. For each shared edited neuron, we compute the cosine similarity between the two domain update vectors:
\begin{equation}
c_{A,B,i}^{(\ell)} = \frac{ \langle \Delta \vv_{A,i}^{(\ell)}, \Delta \vv_{B,i}^{(\ell)} \rangle }{ \|\Delta \vv_{A,i}^{(\ell)}\|_2 \|\Delta \vv_{B,i}^{(\ell)}\|_2 }.
\end{equation}
The layer-wise average directional alignment on shared edited neurons is
\begin{equation}
C^{(\ell)}(A,B) = \frac{1}{|\mathcal{Q}_{A,B}^{(\ell)}|} \sum_{i\in\mathcal{Q}_{A,B}^{(\ell)}} c_{A,B,i}^{(\ell)}.
\end{equation}
Positive values indicate that two domains edit their shared neurons in aligned directions on average, while negative values indicate conflicting directions.
\section{Detailed Proof of Our Theory}
\label{app:det_proof}
\subsection{Structural Assumptions}
\label{app:theory_assumptions}
For readability, the main text only summarizes the structural conditions used by the local perturbation analysis. We state them formally here while directly connecting each condition to the empirical setting.
\paragraph{Assumption 1: local smoothness.}
For each domain $d$, $L_d$ is twice continuously differentiable in a local neighborhood $\mathcal{B}(\bar\vtheta,r)$ containing the sequential RL trajectory, with
\begin{equation}
\|\mH_d(\vtheta)\|_2\le \beta_d, \quad \forall\,\vtheta\in\mathcal{B}(\bar\vtheta,r).
\end{equation}
When a third-order remainder is needed, we further assume local Hessian Lipschitzness:
\begin{equation}
\|\mH_d(\vtheta)-\mH_d(\vtheta')\|_2 \le \rho_d\|\vtheta-\vtheta'\|_2, \quad \forall\,\vtheta,\vtheta' \in\mathcal{B}(\bar\vtheta,r).
\end{equation}
This standard condition is appropriate because Section~\ref{sec:ana} shows that RL post-training behaves as a sequence of small incremental updates rather than a global parameter rewrite; bounded Hessians control second-order sensitivity, and Hessian Lipschitzness keeps the Taylor remainder cubic in the update magnitude.
\paragraph{Assumption 2: approximate stationarity.}
After training on domain $A$, the selected checkpoint is approximately stationary for its objective:
\begin{equation}
\|\vg_A(\vtheta_A^*)\|_2 = \|\nabla L_A(\vtheta_A^*)\|_2 \le \varepsilon_A.
\end{equation}
This does not require $\vtheta_A^*$ to be a global optimum; it only requires the checkpoint to lie near a locally stable region for its own domain, matching our validation-based checkpoint selection protocol.
\paragraph{Assumption 3: small and effectively sparse updates.}
Later-domain training induces a local perturbation $\|\vdelta_B\|_2\le r$, and there exists a low-dimensional update subspace $U_B\subset\mathbb{R}^p$ such that
\begin{equation}
\|\mP_{U_B^\perp}\vdelta_B\|_2 \le \tau_B\|\vdelta_B\|_2, \quad \tau_B\ll 1.
\end{equation}
The norm bound formalizes locality around the previous checkpoint, while the effective sparsity condition is motivated by the parameter- and neuron-level evidence in Section~\ref{sec:ana}: updates are not literally zero outside a tiny support, but their dominant mass is concentrated in relatively few directions.
\paragraph{Assumption 4: low-dimensional shared conflict subspace.}
For a pair of domains $A$ and $B$, there exists a low-dimensional subspace $S_{A,B}\subset\mathbb{R}^p$ with projection $\mP_S=\mP_{S_{A,B}}$ such that
\begin{equation}
\left|\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B - (\mP_S\vdelta_B)^\top \mH_A(\vtheta_A^*)(\mP_S\vdelta_B)\right| \le \gamma_A\|\vdelta_B\|_2^2.
\end{equation}
This is the key structural abstraction of our empirical findings: different domains often edit different parameters, yet their active routes overlap, so $S_{A,B}$ captures the shared active directions where later-domain updates can affect the earlier-domain objective. The assumption does not claim that all interference lies exactly in $S_{A,B}$, only that off-subspace contributions remain perturbative.
\paragraph{Assumption 5: global near-orthogonality.}
Although local conflict may exist inside $S_{A,B}$, full gradients remain nearly orthogonal at the global scale:
\begin{equation}
|\langle \vg_A(\vtheta),\vg_B(\vtheta)\rangle| \le \kappa_{A,B} \|\vg_A(\vtheta)\|_2 \|\vg_B(\vtheta)\|_2, \quad \kappa_{A,B}\ll 1.
\end{equation}
This separates local conflict from full-model antagonism: as shown in Section~\ref{sec:ana}, global gradient cosine can be close to zero even when localized conflict exists, so the theory treats interference as a structured local phenomenon inside a small shared subspace rather than as uniformly conflicting gradients over the whole parameter space.
\paragraph{Assumption 6: positive curvature on $S_{A,B}$ and weak cross-subspace coupling.}
There is a local neighborhood $\mathcal N_A$ of $\vtheta_A^*$ containing the refresh trajectory such that, for all $\vtheta\in\mathcal N_A$ and $\vv\in S_{A,B}$,
\begin{equation}
\mu_A\|\vv\|_2^2 \le \vv^\top \mH_A(\vtheta)\vv \le \bar\beta_A\|\vv\|_2^2, \quad 0<\mu_A\le \bar\beta_A.
\end{equation}
Equivalently, the restriction $\mH_A^S(\vtheta):=\mP_S \mH_A(\vtheta)\mP_S\vert_{S_{A,B}}$ satisfies
\begin{equation}
\mu_A \mI_S \preceq \mH_A^S(\vtheta) \preceq \bar\beta_A \mI_S.
\end{equation}
In addition, the coupling from $S_{A,B}^\perp$ back into $S_{A,B}$ is weak throughout $\mathcal N_A$:
\begin{equation}
\|\mP_S \mH_A(\vtheta)(\mI-\mP_S)\|_2 \le \xi_A,
\end{equation}
where $\xi_A$ is small. We choose the refresh step size such that
\begin{equation}
0<\alpha\le \frac{1}{\bar\beta_A}.
\end{equation}
This assumption requires positive curvature only along the shared conflict directions, not local strong convexity in the full parameter space, and ensures that gradient descent on $L_A$ contracts the harmful component inside $S_{A,B}$ while the influence of off-subspace coordinates enters only as a controlled perturbation.
\subsection{Proof of Proposition 1}
\label{app:proof_second_order}
We provide the full derivation for Proposition~1 and make explicit its sensitivity-based interpretation. Recall that training first reaches a domain-$A$ checkpoint $\vtheta_A^*$ and later domain-$B$ training induces a local displacement $\vdelta_B$, so the degraded checkpoint is $\vtheta_A^*+\vdelta_B$. The interference from $B$ to $A$ is
\begin{equation}
\Delta_{A\leftarrow B} = L_A(\vtheta_A^*+\vdelta_B)-L_A(\vtheta_A^*).
\end{equation}
By Taylor expansion of $L_A$ around $\vtheta_A^*$,
\begin{equation}
L_A(\vtheta_A^*+\vdelta_B) ={} L_A(\vtheta_A^*) + \vg_A(\vtheta_A^*)^\top \vdelta_B + \frac{1}{2}\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B + R_A(\vdelta_B).
\end{equation}
Subtracting $L_A(\vtheta_A^*)$ gives
\begin{equation}
\Delta_{A\leftarrow B} ={} \vg_A(\vtheta_A^*)^\top \vdelta_B + \frac{1}{2}\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B + R_A(\vdelta_B).
\end{equation}
The first term is the possible first-order drift of the domain-$A$ objective, while the second term is the local curvature cost incurred by moving from $\vtheta_A^*$ in the direction $\vdelta_B$. Here $R_A(\vdelta_B)$ is the third-order Taylor remainder. Under Assumption~1, the Hessian of $L_A$ is locally Lipschitz in a neighborhood containing the segment from $\vtheta_A^*$ to $\vtheta_A^*+\vdelta_B$, so the standard remainder bound gives
\begin{equation}
|R_A(\vdelta_B)| \le \frac{\rho_A}{6}\|\vdelta_B\|_2^3.
\end{equation}
Under Assumption~2, the checkpoint $\vtheta_A^*$ is approximately stationary for domain $A$, which gives
\begin{equation}
|\vg_A(\vtheta_A^*)^\top \vdelta_B| \le \|\vg_A(\vtheta_A^*)\|_2\|\vdelta_B\|_2 \le \varepsilon_A\|\vdelta_B\|_2.
\end{equation}
Substituting these two estimates into the expression for $\Delta_{A\leftarrow B}$ and applying the triangle inequality yields
\begin{equation}
\left| \Delta_{A\leftarrow B} - \frac{1}{2}\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B \right| \le \varepsilon_A\|\vdelta_B\|_2 + \frac{\rho_A}{6}\|\vdelta_B\|_2^3,
\end{equation}
which implies
\begin{equation}
\Delta_{A\leftarrow B} ={} \frac{1}{2}\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B + O\left(\varepsilon_A\|\vdelta_B\|_2+\|\vdelta_B\|_2^3\right).
\end{equation}
This proves Proposition~1.
The statement can also be written in a form that separates update magnitude from directional sensitivity. For any nonzero direction $\vv$, define the local second-order sensitivity of domain $A$ as
\begin{equation}
\mathcal{S}^{(2)}_A(\vtheta;\vv) := \frac{\vv^\top \mH_A(\vtheta)\vv}{\|\vv\|_2^2}.
\end{equation}
Taking $\vv=\vdelta_B$ yields the equivalent expression
\begin{equation}
\Delta_{A\leftarrow B} ={} \frac{1}{2}\|\vdelta_B\|_2^2\, \mathcal{S}^{(2)}_A(\vtheta_A^*;\vdelta_B) + O\left(\varepsilon_A\|\vdelta_B\|_2+\|\vdelta_B\|_2^3\right).
\end{equation}
Thus, once $\vtheta_A^*$ is approximately stationary and $\vdelta_B$ is local, forgetting is not determined by update size alone. A small or sparse later-domain update can still produce visible degradation if its direction has large second-order sensitivity under $L_A$, whereas a larger update in low-curvature directions may cause little damage. In particular, if $\varepsilon_A$ is small enough that the linear term is negligible compared with the quadratic term, for example, when $\varepsilon_A=o(\|\vdelta_B\|_2)$ as $\|\vdelta_B\|_2\to 0$, then the dominant contribution to interference is the curvature term $\frac{1}{2}\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B$ rather than first-order drift. This is the precise sense in which sequential updates induce second-order local damage.
\subsection{Proof of Proposition 2}
\label{app:proof_conflict_subspace}
Proposition~2 sharpens Proposition~1 by showing that the dominant second-order damage is concentrated in the shared active conflict subspace $S_{A,B}$ rather than spread across the whole parameter space. Let $\mP_S$ be the projection onto $S_{A,B}$. Starting from Proposition~1,
\begin{equation}
\Delta_{A\leftarrow B} ={} \frac{1}{2}\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B + O\left(\varepsilon_A\|\vdelta_B\|_2+\|\vdelta_B\|_2^3\right).
\end{equation}
Under Assumption 4,
\begin{equation}
\left| (\vdelta_B)^\top \mH_A(\vtheta_A^*)\vdelta_B - (\mP_S\vdelta_B)^\top \mH_A(\vtheta_A^*)(\mP_S\vdelta_B) \right| \le \gamma_A\|\vdelta_B\|_2^2.
\end{equation}
That is, the full quadratic term differs from its restriction to the shared conflict subspace by at most a quadratic residual. Multiplying this bound by $\frac{1}{2}$ and combining it with Proposition~1 yields
\begin{equation}
\Delta_{A\leftarrow B} = {} \frac{1}{2}(\mP_S\vdelta_B)^\top \mH_A(\vtheta_A^*)(\mP_S\vdelta_B) + O\left(\varepsilon_A\|\vdelta_B\|_2 +\gamma_A\|\vdelta_B\|_2^2 +\|\vdelta_B\|_2^3\right).
\end{equation}
Equivalently,
\begin{equation}
\left| \Delta_{A\leftarrow B} - \frac{1}{2}(\mP_S\vdelta_B)^\top \mH_A(\vtheta_A^*)(\mP_S\vdelta_B) \right| \le \varepsilon_A\|\vdelta_B\|_2 + \frac{\gamma_A}{2}\|\vdelta_B\|_2^2 + \frac{\rho_A}{6}\|\vdelta_B\|_2^3.
\end{equation}
This proves Proposition~2. In particular, when $\varepsilon_A$, $\gamma_A$, and $\|\vdelta_B\|_2$ are sufficiently small, the leading contribution to cross-domain degradation is the component of the later-domain update inside the low-dimensional shared active conflict subspace.
The decomposition underlying this bound makes the localization explicit. Write
\begin{equation}
\vdelta_B = \mP_S\vdelta_B + (\mI-\mP_S)\vdelta_B.
\end{equation}
Write
\begin{equation}
\vx := \mP_S\vdelta_B, \vy := (\mI-\mP_S)\vdelta_B,
\end{equation}
so that $\vdelta_B=\vx+\vy$. The quadratic term in Proposition~1 can then be expanded as
\begin{equation}
\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B = {} (\vx+\vy)^\top \mH_A(\vtheta_A^*)(\vx+\vy).
\end{equation}
Using bilinearity of the quadratic form,
\begin{equation}
\begin{split}
(\vx+\vy)^\top \mH_A(\vtheta_A^*)(\vx+\vy)
={}& \vx^\top \mH_A(\vtheta_A^*)\vx
+ \vx^\top \mH_A(\vtheta_A^*)\vy \\
&+ \vy^\top \mH_A(\vtheta_A^*)\vx
+ \vy^\top \mH_A(\vtheta_A^*)\vy.
\end{split}
\end{equation}
Since $\mH_A(\vtheta_A^*)$ is the Hessian of a twice-differentiable scalar objective, it is symmetric, so
\begin{equation}
\vx^\top \mH_A(\vtheta_A^*)\vy = \vy^\top \mH_A(\vtheta_A^*)\vx.
\end{equation}
Therefore the two cross terms combine, giving
\begin{equation}
\begin{split}
\vdelta_B^\top \mH_A(\vtheta_A^*)\vdelta_B
={}& (\mP_S\vdelta_B)^\top \mH_A(\vtheta_A^*)(\mP_S\vdelta_B) \\
&+ 2(\mP_S\vdelta_B)^\top \mH_A(\vtheta_A^*)(\mI-\mP_S)\vdelta_B \\
&+ ((\mI-\mP_S)\vdelta_B)^\top \mH_A(\vtheta_A^*)(\mI-\mP_S)\vdelta_B.
\end{split}
\end{equation}
The first term is the in-subspace contribution isolated in Proposition~2. It measures the part of the second-order damage caused by the component of the later-domain update that lies directly inside $S_{A,B}$. The second term measures leakage between the conflict subspace and its orthogonal complement: even if part of the update lies outside $S_{A,B}$, curvature can couple that off-subspace motion back into the sensitive directions of domain $A$. The third term captures purely off-subspace curvature effects.
Assumption~4 is precisely what makes this decomposition useful. It states that the latter two terms are bounded by a quadratic residual, so the leading contribution to forgetting is carried by the projection $\mP_S\vdelta_B$. In this sense, Proposition~2 refines Proposition~1: Proposition~1 says that forgetting is controlled by a second-order quadratic form, while Proposition~2 identifies where the dominant part of that quadratic damage is located.
This decomposition clarifies an important point: low parameter-overlap across domains does not imply low interference. A later-domain update may modify only a small or apparently unrelated subset of parameters, yet still cause substantial forgetting if its projection enters the small set of directions along which the previous-domain objective is locally sensitive. Conversely, a larger update can remain relatively benign when most of it lies outside these sensitive shared directions.
This also connects back to the near-orthogonality result in Section~\ref{sec:ana}. If global gradients are nearly orthogonal, then immediate first-order cross-domain conflict is weak, but localized second-order displacement inside $S_{A,B}$ can still dominate. Thus, full-model near-orthogonality, sparse edits, and low edit overlap are compatible with selective degradation, because the relevant conflict is carried by a small shared active subspace rather than by global gradient antagonism.
\subsection{Proof of Theorem 1}
\label{app:proof_refresh}
Let
\begin{equation}
\ve_t=\vtheta_t-\vtheta_A^*, \quad \ve_t^S=\mP_S\ve_t.
\end{equation}
The refresh iteration is
\begin{equation}
\vtheta_{t+1}=\vtheta_t-\alpha \vg_A(\vtheta_t),
\end{equation}
which implies
\begin{equation}
\ve_{t+1}=\ve_t-\alpha \vg_A(\vtheta_t).
\end{equation}
Projecting onto the shared conflict subspace gives
\begin{equation}
\ve_{t+1}^S = \mP_S\ve_t-\alpha \mP_S \vg_A(\vtheta_t) = \ve_t^S-\alpha \mP_S \vg_A(\vtheta_t).
\end{equation}
By Assumption 1, the Hessian of $L_A$ is locally Lipschitz near $\vtheta_A^*$. Since Assumption 6 ensures that the refresh trajectory remains inside a local neighborhood $\mathcal N_A$ of $\vtheta_A^*$, we may write, for each iterate along this trajectory,
\begin{equation}
\vg_A(\vtheta_t) = \vg_A(\vtheta_A^*) + \mH_A(\vtheta_A^*)\ve_t + \vr_t,
\end{equation}
where the remainder satisfies $\|\vr_t\|_2\le c_A\|\ve_t\|_2^2$ for some local constant $c_A>0$. Substituting this into the projected recursion gives
\begin{equation}
\ve_{t+1}^S ={} \ve_t^S - \alpha \mP_S \vg_A(\vtheta_A^*) - \alpha \mP_S \mH_A(\vtheta_A^*)\ve_t - \alpha \mP_S \vr_t.
\end{equation}
Decompose $\ve_t=\ve_t^S+\ve_t^{\perp}$ with $\ve_t^{\perp}=(\mI-\mP_S)\ve_t$. Then
\begin{equation}
\mP_S \mH_A(\vtheta_A^*)\ve_t ={} \mP_S \mH_A(\vtheta_A^*)\mP_S \ve_t + \mP_S \mH_A(\vtheta_A^*)(\mI-\mP_S)\ve_t.
\end{equation}
By Assumption 6, the second term is controlled by weak cross-subspace coupling:
\begin{equation}
\left\|\mP_S \mH_A(\vtheta_A^*)(\mI-\mP_S)\ve_t\right\|_2 \le \xi_A\|\ve_t^{\perp}\|_2.
\end{equation}
Thus the projected recursion becomes
\begin{equation}
\ve_{t+1}^S ={} \left(\mI- \alpha \mP_S \mH_A(\vtheta_A^*)\mP_S\right)\ve_t^S - \alpha \mP_S \vg_A(\vtheta_A^*) + \zeta_t,
\end{equation}
where $\zeta_t=-\alpha \mP_S \mH_A(\vtheta_A^*)(\mI-\mP_S)\ve_t-\alpha \mP_S \vr_t$ collects off-subspace and higher-order local errors. Consequently,
\begin{equation}
\|\zeta_t\|_2 \le \alpha\xi_A\|\ve_t^{\perp}\|_2+c_A\alpha\|\ve_t\|_2^2
\end{equation}
along the refresh trajectory.
Under Assumption 6, $\mu_A \mI_S\preceq \mH_A^S(\vtheta)\preceq \bar\beta_A \mI_S$ and $0<\alpha\le \frac{1}{\bar\beta_A}$ for all $\vtheta\in \mathcal N_A$. Hence every eigenvalue of $\mI-\alpha \mP_S \mH_A(\vtheta_A^*)\mP_S$ on $S_{A,B}$ lies in $[0,1-\alpha\mu_A]$, so
\begin{equation}
\left\|\mI-\alpha \mP_S \mH_A(\vtheta_A^*)\mP_S\right\|_2 \le 1-\alpha\mu_A.
\end{equation}
Moreover, by Assumption 2,
\begin{equation}
\|\mP_S \vg_A(\vtheta_A^*)\|_2\le \|\vg_A(\vtheta_A^*)\|_2\le \varepsilon_A.
\end{equation}
Taking norms in the projected recursion yields
\begin{equation}
\|\ve_{t+1}^S\|_2 \le{} (1-\alpha\mu_A)\|\ve_t^S\|_2 + \alpha\varepsilon_A + \alpha\xi_A\|\ve_t^{\perp}\|_2 + c_A\alpha\|\ve_t\|_2^2.
\end{equation}
As long as refresh stays in the same local neighborhood, we do not need to derive a separate dynamical bound on $\ve_t^{\perp}$; instead, we treat the off-subspace contribution as part of a local perturbation term and absorb
\begin{equation}
\eta_{\mathrm{loc},t} := \xi_A\|\ve_t^{\perp}\|_2+c_A\|\ve_t\|_2^2
\end{equation}
into a local error sequence and write
\begin{equation}
\|\ve_{t+1}^S\|_2 \le (1-\alpha\mu_A)\|\ve_t^S\|_2 + \alpha\bigl(\varepsilon_A+\eta_{\mathrm{loc},t}\bigr).
\end{equation}
Unrolling the recursion gives
\begin{equation}
\|\ve_t^S\|_2 \le{} (1-\alpha\mu_A)^t\|\ve_0^S\|_2 + \sum_{k=0}^{t-1}(1-\alpha\mu_A)^{t-1-k} \Bigl( \alpha\varepsilon_A+ \alpha\eta_{\mathrm{loc},k} \Bigr).
\end{equation}
Since $\ve_0=\vdelta_B$, we have $\ve_0^S=\mP_S\vdelta_B$. If we define $\eta_{\mathrm{loc}}:=\sup_k \eta_{\mathrm{loc},k}$ over the local refresh trajectory, then
\begin{equation}
\sum_{k=0}^{t-1}(1-\alpha\mu_A)^{t-1-k} \alpha\bigl(\varepsilon_A+\eta_{\mathrm{loc},k}\bigr) \le \frac{\varepsilon_A+\eta_{\mathrm{loc}}}{\mu_A}.
\end{equation}
Therefore,
\begin{equation}
\|\mP_S(\vtheta_t-\vtheta_A^*)\|_2 \le (1-\alpha\mu_A)^t\|\mP_S\vdelta_B\|_2 + O\!\left(\frac{\varepsilon_A+\eta_{\mathrm{loc}}}{\mu_A}\right).
\end{equation}
Here $\eta_{\mathrm{loc}}$ summarizes bounded off-subspace and higher-order local errors along the refresh trajectory. In the ideal locally quadratic case with $\vg_A(\vtheta_A^*)=0$ and $\xi_A=0$, these error terms vanish, and the recursion reduces to
\begin{equation}
\ve_{t+1}^S = \left(\mI- \alpha \mP_S \mH_A(\vtheta_A^*)\mP_S\right)\ve_t^S,
\end{equation}
so that
\begin{equation}
\|\mP_S(\vtheta_t-\vtheta_A^*)\|_2 \le (1-\alpha\mu_A)^t\|\mP_S\vdelta_B\|_2.
\end{equation}
This proves the theorem.
\begin{table*}[t]
\centering
\scriptsize
\caption{Full benchmark-level results. Step denotes the selected checkpoint step for the corresponding run. AVG is the average over the four evaluation domains: Math, Code, QA, and CW.}
\label{tab:full_task_results}
\resizebox{\textwidth}{!}{
\begin{tabular}{lccccccccccccc}
\toprule
\multirow{2}{*}{Model} & \multirow{2}{*}{Step}
& \multicolumn{6}{c}{Math}
& \multicolumn{1}{c}{Code}
& \multicolumn{3}{c}{QA}
& \multicolumn{1}{c}{CW}
& \multirow{2}{*}{AVG} \\
\cmidrule(lr){3-8}\cmidrule(lr){9-9}\cmidrule(lr){10-12}\cmidrule(lr){13-13}
&
& AIME24 & AIME25 & AIME26 & OlympiadBench & HMMT & Math
& LiveCodeBench-v6
& SuperGPQA-test & MMLU-Pro & QA
& WritingBench & \\
\midrule
Base & --
& 48.75 & 42.50 & 40.63 & 61.68 & 22.40 & 43.19
& 29.57
& 47.32 & 73.96 & 60.64
& 82.44 & 53.96 \\
$\mathrm{Math}_s$ & 525
& 71.77 & 67.71 & 69.79 & 77.95 & 46.98 & 66.84
& 34.65
& 47.84 & 73.68 & 60.76
& 81.38 & 60.91 \\
$\mathrm{Code}_s$ & 600
& 61.46 & 61.46 & 62.71 & 71.38 & 41.15 & 59.63
& 52.67
& 47.84 & 73.93 & 60.89
& 82.40 & 63.90 \\
$\mathrm{QA}_s$ & 705
& 58.65 & 55.83 & 56.77 & 70.19 & 35.10 & 55.31
& 32.07
& 51.85 & 74.77 & 63.31
& 81.76 & 58.11 \\
$\mathrm{CW}_s$ & 120
& 44.27 & 38.85 & 35.00 & 59.74 & 21.04 & 39.78
& 28.15
& 47.16 & 73.84 & 60.50
& 86.24 & 53.67 \\
$\mathrm{Code}_o$ & 600
& 61.46 & 61.46 & 62.71 & 71.38 & 41.15 & 59.63
& 52.67
& 47.84 & 73.93 & 60.89
& 82.40 & 63.90 \\
$\mathrm{Math}_o$ & 180
& 73.33 & 66.56 & 70.83 & 77.57 & 44.17 & 66.49
& 50.69
& 47.53 & 73.51 & 60.52
& 81.44 & 64.79 \\
$\mathrm{QA}_o$ & 300
& 64.38 & 59.17 & 61.98 & 72.73 & 41.25 & 59.90
& 50.99
& 49.98 & 74.69 & 62.34
& 81.79 & 63.76 \\
$\mathrm{CW}_o$ & 135
& 62.29 & 56.04 & 58.65 & 70.89 & 40.42 & 57.66
& 50.47
& 49.91 & 74.77 & 62.34
& 86.52 & 64.25 \\
CGPO & 630
& 65.10 & 58.13 & 66.88 & 75.35 & 44.17 & 61.93
& 50.05
& 50.90 & 74.05 & 62.48
& 86.73 & 65.30 \\
JT & 1035
& 69.58 & 65.63 & 65.52 & 77.12 & 46.15 & 64.80
& 48.61
& 50.04 & 74.18 & 62.11
& 86.97 & 65.62 \\
Re-Math & 135
& 74.27 & 64.48 & 68.65 & 77.06 & 45.73 & 66.04
& 51.05
& 50.28 & 74.69 & 62.49
& 85.96 & 66.39 \\
QA $\rightarrow$ Math & 675
& 71.15 & 66.77 & 68.54 & 77.04 & 43.13 & 65.33
& 32.53
& 51.55 & 74.44 & 63.00
& 82.01 & 60.72 \\
Math $\rightarrow$ CW & 120
& 72.71 & 67.71 & 70.42 & 77.75 & 47.19 & 67.16
& 32.82
& 48.31 & 74.04 & 61.18
& 85.77 & 61.73 \\
\bottomrule
\end{tabular}
}
\end{table*}
\subsection{Stability of Other Domains under Near-Orthogonal Gradients}
\label{app:proof_other_domains}
A desirable property of refresh is that it should recover the target domain without substantially damaging other domains. We now show that this follows from global near-orthogonality and small refresh steps.
Consider one refresh step on domain $A$:
\begin{equation}
\vtheta_{t+1} = \vtheta_t-\alpha \vg_A(\vtheta_t).
\end{equation}
For another domain $C\neq A$, local smoothness gives
\begin{equation}
L_C(\vtheta_{t+1}) \le L_C(\vtheta_t) - \alpha \langle \vg_C(\vtheta_t),\vg_A(\vtheta_t)\rangle + \frac{\beta_C\alpha^2}{2}\|\vg_A(\vtheta_t)\|_2^2 .
\end{equation}
By Assumption 5,
\begin{equation}
|\langle \vg_C(\vtheta_t),\vg_A(\vtheta_t)\rangle| \le \kappa_{A,C} \|\vg_C(\vtheta_t)\|_2 \|\vg_A(\vtheta_t)\|_2.
\end{equation}
Therefore,
\begin{equation}
L_C(\vtheta_{t+1})-L_C(\vtheta_t) \le \alpha \kappa_{A,C} \|\vg_C(\vtheta_t)\|_2 \|\vg_A(\vtheta_t)\|_2 + \frac{\beta_C\alpha^2}{2}\|\vg_A(\vtheta_t)\|_2^2 .
\end{equation}
\paragraph{Corollary 1.}
\textit{Under Assumptions 1 and 5, the one-step increase of another domain objective $L_C$ during refresh on domain $A$ is bounded by}
\begin{equation}
L_C(\vtheta_{t+1})-L_C(\vtheta_t) \le \alpha \kappa_{A,C} \|\vg_C(\vtheta_t)\|_2 \|\vg_A(\vtheta_t)\|_2 + \frac{\beta_C\alpha^2}{2}\|\vg_A(\vtheta_t)\|_2^2 .
\end{equation}
\textit{Thus, when global gradients are nearly orthogonal, refresh steps are small, and the involved gradient norms are not too large, the one-step damage to other domains is limited.}
The same calculation also explains the instantaneous interference during sequential training. If one instead performs a single update on domain $B$, namely $\vtheta^+=\vtheta-\alpha \vg_B(\vtheta)$, then local smoothness of $L_A$ gives
\begin{equation}
L_A(\vtheta^+)-L_A(\vtheta) \le -\alpha\langle \vg_A(\vtheta),\vg_B(\vtheta)\rangle + \frac{\beta_A\alpha^2}{2}\|\vg_B(\vtheta)\|_2^2 .
\end{equation}
Under near-orthogonality, the first-order cross term is again small, so the immediate harm from a later-domain step is controlled mainly by higher-order curvature terms rather than by strong global gradient antagonism.
For a short refresh trajectory of $T$ steps, summing the one-step bound yields
\begin{equation}
L_C(\vtheta_T)-L_C(\vtheta_0) \le \sum_{t=0}^{T-1} \Big( \alpha \kappa_{A,C} \|\vg_C(\vtheta_t)\|_2 \|\vg_A(\vtheta_t)\|_2 + \frac{\beta_C\alpha^2}{2}\|\vg_A(\vtheta_t)\|_2^2 \Big) .
\end{equation}
Hence, when refresh is short and gradients remain nearly orthogonal along the trajectory, cumulative degradation to another domain stays limited.
An equivalent total-displacement view is obtained by writing $\vr_T:=\vtheta_T-\vtheta_0$. By local smoothness,
\begin{equation}
L_C(\vtheta_T)-L_C(\vtheta_0) \le \vg_C(\vtheta_0)^\top \vr_T + \frac{\beta_C}{2}\|\vr_T\|_2^2 .
\end{equation}
If the starting point is also approximately stationary for domain $C$, so that $\|\vg_C(\vtheta_0)\|_2\le \varepsilon_C$, then
\begin{equation}
L_C(\vtheta_T)-L_C(\vtheta_0) \le \varepsilon_C\|\vr_T\|_2 + \frac{\beta_C}{2}\|\vr_T\|_2^2 .
\end{equation}
Therefore, another domain remains stable whenever the total refresh displacement is small, even if we do not track every intermediate step in detail.
This corollary complements Theorem 1. Refresh can strongly reduce the target-domain conflict component because the target gradient is aligned with the local corrective direction in $S_{A,B}$. At the same time, its effect on other domains is bounded because full-domain gradients are approximately orthogonal at the global scale. This separation between local conflict and global near-orthogonality explains how Re-Math can recover Math while largely preserving Code, QA, and CW.
\begin{figure*}[t]
\centering
\includegraphics[width=0.95\textwidth]{acc_mean16_4tasks.pdf}
\caption{Validation dynamics during Re-Math refresh from $\mathrm{CW}_o$ across the four domains.}
\label{fig:refresh_dynamics}
\end{figure*}
\subsection{Extension: Alternating Refresh as Local Multi-Objective Optimization}
\label{app:proof_alt_refresh}
The same local view also explains why alternating refresh across domains can approach a local compromise among multiple objectives. The key point is that one refresh cycle is not best understood as ``forgetting $A$ and then forgetting $B$'' again; to first order, it behaves like one descent step on a single weighted local objective. Consider two domains $A$ and $B$. One cycle of alternating refresh first updates on $A$:
\begin{equation}
\vtheta' = \vtheta-\alpha_A \vg_A(\vtheta),
\end{equation}
and then updates on $B$:
\begin{equation}
\vtheta'' = \vtheta'-\alpha_B \vg_B(\vtheta').
\end{equation}
For small step sizes, local smoothness gives
\begin{equation}
\vg_B(\vtheta') = \vg_B(\vtheta)+O(\alpha_A\|\vg_A(\vtheta)\|_2).
\end{equation}
Thus, one refresh cycle satisfies
\begin{equation}
\vtheta'' = \vtheta - \alpha_A \vg_A(\vtheta) - \alpha_B \vg_B(\vtheta) + O(\alpha_A\alpha_B).
\end{equation}
To first order, alternating refresh therefore follows gradient descent on the weighted local objective
\begin{equation}
\Phi(\vtheta) = \alpha_A L_A(\vtheta) + \alpha_B L_B(\vtheta).
\end{equation}
More generally, alternating refresh over multiple domains approximates gradient descent on
\begin{equation}
\Phi(\vtheta) = \sum_{d\in\mathcal{D}} \alpha_d L_d(\vtheta).
\end{equation}
A stationary point of this local objective satisfies
\begin{equation}
\sum_{d\in\mathcal{D}} \alpha_d \vg_d(\vtheta^\dagger) \approx 0.
\end{equation}
For positive weights $\{\alpha_d\}$, this is the first-order stationarity condition of a weighted-sum scalarization, which is a standard local proxy for Pareto compromise under regularity assumptions. Therefore, while a single refresh targets recovery of one degraded domain, alternating refresh can be viewed as a local mechanism for approaching a Pareto-stationary compromise among domains under the weighted-sum scalarization view. In particular, it clarifies why repeated refresh can keep improving for several rounds without implying that all single-domain optima are simultaneously reachable.
% This perspective is intentionally local. Alternating refresh is not expected to return the model to each domain's single-domain optimum unless all objectives happen to share a common stationary point. In general, it instead moves the model toward a compromise point $\vtheta^\dagger$ that balances multiple domains in the neighborhood reached by training. Put differently, the method can contract the recoverable displacement component induced by sequential training, but it cannot in general remove intrinsic objective mismatch between domains.
% The same local approximation also suggests when repeated alternating refresh can keep improving. Around such a compromise point, write $\ve_m=\vtheta_m-\vtheta^\dagger$ and linearize the cycle map. To first order,
% \begin{equation}
% \end{equation}
% If the restriction of this operator to the shared conflict directions is contractive, then the recoverable conflict component decays geometrically across cycles. However, if domain optima are genuinely misaligned, the process converges only to a local compromise and leaves an irreducible conflict floor.
% This distinction is important for interpreting refresh experiments. Alternating refresh can remove the recoverable part of interference---the local displacement component that lies along contractive shared conflict directions---but it does not in general eliminate intrinsic objective mismatch between domains. Consequently, even when repeated refresh continues to improve a degraded domain for several rounds, the process should be understood as approaching a better local trade-off rather than recovering every domain's standalone optimum simultaneously.
% This extension provides a theoretical motivation for the additional refresh experiments in Section~\ref{sec:exp}, such as Re-QA, Re-Code, and Re-CW. These experiments test whether the local recoverability mechanism extends beyond a single Re-Math step and whether repeated refresh can move the policy toward a more balanced multi-domain trade-off.
\section{Additional Validation Results}
\subsection{Full Task-Level Results}
\label{app:full_results}
Table~\ref{tab:full_task_results} reports the full benchmark-level results for all model checkpoints used in the main task-level validation. For sequential and refresh runs, the step number in parentheses denotes the selected checkpoint step within the last training stage.
\subsection{Refresh Dynamics}
\label{app:refresh_dynamics}
\begin{wrapfigure}{r}{0.43\textwidth}
\vspace{-3.0em}
\centering
\includegraphics[width=0.42\textwidth]{dual_axis_chart.pdf}
\caption{Math and Code performance changes during Re-Code on $\mathrm{Math}_o$.}
\label{fig:recode_dual_axis}
\vspace{-2.0em}
\end{wrapfigure}
Figure~\ref{fig:refresh_dynamics} further confirms that Re-Math behaves as a short local correction rather than full retraining. Along the refresh trajectory from $\mathrm{CW}_o$, Math performance increases rapidly in the early steps and then approaches saturation, while Code, QA, and CW remain largely stable with only small fluctuations. This dynamic pattern supports the contraction view in Theorem~1: the refresh update mainly removes the Math-sensitive harmful displacement induced by later-domain training, instead of globally overwriting the policy or trading off other domains for Math recovery.
Figure~\ref{fig:recode_dual_axis} provides a complementary view through the Code $\rightarrow$ Math $\rightarrow$ Re-Code trajectory, further clarifying our local recoverability result. In the early refresh stage, Code improves while Math changes only mildly, consistent with Theorem~1: a short refresh can contract the harmful displacement on the shared conflict directions and recover the target domain within a local neighborhood. However, as Re-Code training continues, the update is no longer well described as a small local correction, and the Math decline becomes more visible. This behavior is exactly what Proposition~1 predicts: once the accumulated displacement grows, second-order damage to the previously optimized domain can increase even without strong global gradient opposition. Figure~\ref{fig:recode_dual_axis} should therefore be interpreted as evidence for a local small-update regime rather than unrestricted continued specialization: refresh is effective when it quickly removes the target-domain conflict component, but prolonged single-domain training can again perturb directions to which the other domain is locally sensitive.
\subsection{Directional Asymmetry of Interference}
\label{app:asym_directionality}
We additionally compare QA $\rightarrow$ Math with the reverse Math $\rightarrow$ QA ordering. As shown in Figure~\ref{fig:qa_math_dynamics}, in QA $\rightarrow$ Math, Math improves steadily during the Math stage while QA remains largely stable, unlike the stronger Math degradation observed when QA is trained after Math.
\begin{wrapfigure}{r}{0.50\textwidth}
\vspace{-0.0em}
\centering
\includegraphics[width=0.48\textwidth]{wandb_math_qa.pdf}
\caption{Validation dynamics for the reverse QA $\rightarrow$ Math ordering.}
\label{fig:qa_math_dynamics}
\vspace{-1.0em}
\end{wrapfigure}
This provides further evidence that cross-domain interference is directional rather than symmetric. This asymmetry is also consistent with our local perturbation analysis and Proposition~1: the damage to an earlier domain depends on whether the later update enters its curvature-sensitive shared directions. Here, the Math update appears to perturb QA-sensitive directions only weakly, whereas the QA update more strongly perturbs Math-sensitive directions in the reverse ordering.
\subsection{Direct Rollback on a Coordinate Proxy for the Conflict Subspace}
\label{app:proxy_conflict_appendix}
To more directly probe the localization claim behind Proposition~2, we intervene on the checkpoint pair $\mathrm{Code}_o \rightarrow \mathrm{Math}_o \rightarrow \mathrm{QA}_o$.
Treating checkpoints as their parameter vectors, define the QA-induced displacement as
\begin{equation}
\vdelta_{Q\mid M}=\mathrm{QA}_o-\mathrm{Math}_o.
\end{equation}
We restrict the intervention to MLP neurons, using the same neuron definition as in Appendix~\ref{app:neuron_definition}. For neuron $i$ in layer $\ell$, let $\Delta \vv_{M,i}^{(\ell)}$ and $\Delta \vv_{Q,i}^{(\ell)}$ denote the Math and QA task vectors in the concatenated gate/up/down parameterization from Appendix~\ref{app:directional_alignment_metric}.
We build a neuron-level proxy score
\begin{equation}
S_i^{(\ell)} = A_i^{(\ell)} M_i^{(\ell)} C_i^{(\ell)},
\end{equation}
where
\begin{equation}
A_i^{(\ell)}= \min\!\Big( \operatorname{Pct}(a_{M,i}^{(\ell)}), \operatorname{Pct}(a_{Q,i}^{(\ell)}) \Big),
\end{equation}
\begin{equation}
M_i^{(\ell)}= \operatorname{Pct}\!\left(\|\Delta \vv_{Q,i}^{(\ell)}\|_2\right),
\end{equation}
and
\begin{equation}
C_i^{(\ell)}= \max\!\left( 0,\, -\cos\!\left(\Delta \vv_{M,i}^{(\ell)},\Delta \vv_{Q,i}^{(\ell)}\right) \right).
\end{equation}
Here $a_{d,i}^{(\ell)}$ is the dataset-level activation score defined in Appendix~\ref{app:active_route_definition}, and $\operatorname{Pct}(\cdot)$ denotes percentile rank within a layer.
For activation collection, we use 512 samples and first generate up to 4096 tokens, followed by masked teacher-forcing forward passes to record MLP intermediate activations. Unless otherwise stated, we use a total intervention budget
\begin{equation}
B=\beta \times L \times d_{\mathrm{int}},
\end{equation}
with $\beta=2\%$, $L=36$, and $d_{\mathrm{int}}=9728$, giving $B=7004$ out of $350{,}208$ total MLP neurons.
To distribute the global budget across layers, we first compute a layer score by averaging the top-$\rho$ neuron scores within each layer, with $\rho=10\%$. The final layer budget is then allocated proportionally to
\begin{equation}
\exp\!\left((\bar S^{(\ell)})^\alpha\right),
\end{equation}
with $\alpha=2.0$, followed by rounding so that $\sum_{\ell} B_\ell=B$.
Given the selected neuron set $\hat S$, we revert only the QA increment on its gate/up/down parameters:
\begin{equation}
\vtheta_{\mathrm{rev}} = \mathrm{QA}_o-\mP_{\hat S}\vdelta_{Q\mid M},
\end{equation}
where $\mP_{\hat S}$ is the coordinate projection induced by $\hat S$. This intervention requires no additional optimization and provides a direct proxy-level test of whether a sparse subset of the QA displacement carries measurable Math damage.
For single-factor selectors (A, M, or C), we report two variants. In the fixed-budget variant, we keep the layer allocation from the full $A\times M\times C$ selector and only change the within-layer ranking. In the recomputed-budget variant, we recompute the layer allocation from the corresponding selector itself.
\begin{table*}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{3.2pt}
\caption{Additional ablations for the proxy conflict-subspace intervention at $\beta=2\%$. Fixed-budget variants keep the layer allocation of the full selector and change only within-layer ranking; recomputed-budget variants recompute both ranking and layer allocation.}
\label{tab:proxy_conflict_ablation_full}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lccccc|ccc|cccc}
\toprule
Selector & AIME24 & AIME25 & AIME26 & OlyBench & HMMT & Math Avg & $\Delta$ vs $\mathrm{QA}_o$ & Recovery & SuperGPQA & MMLU-Pro & QA Avg & $\Delta$ QA Avg \\
\midrule
$\mathrm{Math}_o$ & 73.33 & 66.56 & 70.83 & 77.57 & 44.17 & 66.49 & +6.59 & 100.0\% & 47.53 & 73.51 & 60.52 & $-1.81$ \\
$\mathrm{QA}_o$ & 64.38 & 59.17 & 61.98 & 72.73 & 41.25 & 59.90 & 0.00 & 0.0\% & 49.98 & 74.69 & 62.33 & 0.00 \\
Random & 63.54 & 59.90 & 60.00 & 73.05 & 40.94 & 59.49 & $-0.42$ & $-6.3\%$ & 49.97 & 74.71 & 62.34 & +0.01 \\
\midrule
A$\times$M$\times$C (main) & 66.25 & 60.21 & 63.65 & 73.42 & 42.71 & 61.25 & +1.35 & \textbf{20.4\%} & 50.11 & 74.43 & 62.27 & $-0.06$ \\
M$\times$C & 66.67 & 60.00 & 61.77 & 73.36 & 43.75 & 61.11 & +1.21 & 18.3\% & 50.16 & 74.58 & 62.37 & +0.04 \\
A$\times$C & 65.00 & 59.48 & 62.81 & 73.50 & 41.98 & 60.55 & +0.65 & 9.9\% & 49.92 & 74.65 & 62.29 & $-0.05$ \\
A$\times$M & 65.73 & 61.46 & 61.77 & 72.97 & 40.73 & 60.53 & +0.63 & 9.6\% & 50.25 & 74.49 & 62.37 & +0.04 \\
M (fixed budget) & 64.69 & 60.10 & 63.12 & 73.50 & 42.40 & 60.76 & +0.86 & 13.1\% & 49.93 & 74.56 & 62.25 & $-0.09$ \\
A (fixed budget) & 65.52 & 59.90 & 63.75 & 73.23 & 43.85 & 61.25 & +1.35 & \textbf{20.4\%} & 49.93 & 74.55 & 62.24 & $-0.09$ \\
C (fixed budget) & 65.83 & 60.21 & 61.77 & 73.30 & 42.60 & 60.74 & +0.84 & 12.7\% & 49.94 & 74.59 & 62.27 & $-0.07$ \\
M (recomputed) & 65.21 & 59.38 & 63.33 & 73.47 & 42.71 & 60.82 & +0.92 & 13.9\% & 50.14 & 74.69 & 62.41 & +0.08 \\
A (recomputed) & 66.15 & 60.42 & 61.88 & 73.41 & 43.12 & 61.00 & +1.09 & 16.6\% & 50.18 & 74.56 & 62.37 & +0.04 \\
C (recomputed) & 65.52 & 61.35 & 62.71 & 73.23 & 42.08 & 60.98 & +1.08 & 16.3\% & 50.15 & 74.57 & 62.36 & +0.03 \\
\bottomrule
\end{tabular}
}
\end{table*}
Table~\ref{tab:proxy_conflict_ablation_full} shows that the full selector recovers $20.4\%$ of the QA-induced Math loss, while the two-factor selector $M\times C$ remains close at $18.3\%$. By contrast, removing either update magnitude or directional conflict reduces recovery to about half of the full result. Interestingly, A-only under the full selector's layer budget matches the full selector, whereas A-only with its own recomputed budget drops to $16.6\%$. This indicates that shared activation alone is not sufficient to explain the best-performing intervention; part of its apparent strength comes from inheriting the conflict-aware layer allocation.
Figure~\ref{fig:proxy_conflict_budgets} shows that the depth profile of the intervention is not uniform. Once the conflict term is included, the layer budget becomes strongly U-shaped, concentrating more neurons in shallow and late layers; without the conflict term, the budget becomes much flatter. The selector $M\times C$ is the closest two-factor approximation to the full selector, whereas A-only can exhibit much smaller support overlap despite competitive recovery in the fixed-budget setting. This suggests that the proxy support is partly redundant: different sparse coordinate subsets can play similar functional roles as long as they target the harmful QA displacement.
Figure~\ref{fig:proxy_conflict_beta} shows a clear dose-response pattern. Increasing the revert budget from $1\%$ to $4\%$ raises recovery from $2.0\%$ to $29.4\%$, after which performance fluctuates around a saturation regime rather than improving monotonically. This suggests that a relatively small fraction of MLP neurons carries a substantial part of the harmful QA displacement.
\begin{figure}[t]
\centering
\includegraphics[width=0.7\textwidth]{combined_layer_analysis.pdf}
\caption{Layer-wise neuron selection analysis. Top: normalized layer scores; middle: budget allocation; bottom: Jaccard overlap with the $A \times M \times C$ set across 36 layers under different scoring criteria. The conflict factor $C$ dominates the non-uniform layer budget distribution, and two-factor combinations containing $C$ recover the highest overlap with the full composite set.}
\label{fig:proxy_conflict_budgets}
\end{figure}
The MLP-only results above show that a sparse rollback on proxy conflict coordinates can already recover a substantial part of the QA-induced Math loss, but the recovery saturates early and remains incomplete. To test whether the recoverable harmful displacement also extends beyond MLP coordinates, we extend the intervention to attention layers.
For layer $\ell$, query head $h$, and row index $r$, we define a fine-grained attention unit
\begin{equation}
\resizebox{0.93\linewidth}{!}{$
\mathrm{AttnUnit}(\ell,h,r)=\{ \Delta W_Q^{(\ell)}[h,r,:],\; \Delta W_O^{(\ell)}[:,h,r], \frac{1}{\sqrt{g}}\Delta W_K^{(\ell)}[\kappa(h),r,:], \frac{1}{\sqrt{g}}\Delta W_V^{(\ell)}[\kappa(h),r,:] \}.
$}
\end{equation}
where $\kappa(h)=\lfloor h/g \rfloor$ maps a query head to its shared KV head under grouped-query attention, and $g=4$ is the group size. Since each layer has $32$ query heads and head dimension $128$, this gives $32\times 128=4096$ attention units per layer, or $147{,}456$ attention units across all $36$ layers.
Unlike the MLP case, collecting per-unit activation statistics for attention would require decomposing attention outputs at the head-row level. We therefore use a conservative two-factor attention score
\begin{equation}
S_{\mathrm{attn}}^{(\ell,h,r)}= M_{\mathrm{attn}}^{(\ell,h,r)} \,C_{\mathrm{attn}}^{(\ell,h,r)},
\end{equation}
where $M_{\mathrm{attn}}$ measures the QA update magnitude of the unit and $C_{\mathrm{attn}}$ measures directional conflict between the Math and QA task vectors. MLP neurons keep the original three-factor score
\begin{equation}
S_{\mathrm{mlp}}^{(\ell,i)}= A_i^{(\ell)}M_i^{(\ell)}C_i^{(\ell)}.
\end{equation}
% \begin{figure}[t]
\begin{wrapfigure}{r}{0.45\textwidth}
\centering
% \includegraphics[width=0.45\textwidth]{beta_sweep_recovery.pdf}
\includegraphics[width=\linewidth]{beta_sweep_recovery.pdf}
\caption{Math recovery under different intervention budgets.}
\label{fig:proxy_conflict_beta}
\end{wrapfigure}
% \end{figure}
To keep the comparison with the MLP-only experiments fair, we define the total budget in the same units as before:
\begin{equation}
B=\beta \times N_{\mathrm{mlp}},
\end{equation}
where $N_{\mathrm{mlp}}=36\times 9728=350{,}208$. We first compute a joint layer score by combining the top-$\rho$ scores from MLP and attention units:
\begin{equation}
s_\ell= \frac{ \bar S_{\mathrm{mlp}}^{\rho}(\ell)\, d_{\mathrm{int}} + \bar S_{\mathrm{attn}}^{\rho}(\ell)\, n_{\mathrm{attn}} }{ d_{\mathrm{int}}+n_{\mathrm{attn}} },
\end{equation}
with $\rho=0.1$, $d_{\mathrm{int}}=9728$, and $n_{\mathrm{attn}}=4096$. The layer budget is then allocated by
\begin{equation}
b_\ell = B\cdot \frac{(s_\ell)^\alpha}{\sum_{\ell'} (s_{\ell'})^\alpha}, \quad \alpha=2.0.
\end{equation}
Within each layer, we merge all MLP-neuron and attention-unit scores into a single ranked list and select the top-$b_\ell$ units regardless of type.
For selected MLP neurons, the revert operation is identical to the MLP-only setting. For a selected attention unit $\mathrm{AttnUnit}(\ell,h,r)$, we fully revert the corresponding $W_Q$ and $W_O$ slices. For shared KV weights, if $k$ out of the $g$ query heads attached to the same KV head select row $r$, we revert the corresponding $W_K$ and $W_V$ rows by fraction $k/g$.
Figure~\ref{fig:proxy_conflict_beta} shows the joint MLP+Attn sweep. The joint selector is not uniformly better at the smallest budget: at $\beta=2\%$ it recovers $12.7\%$ of the QA-induced Math loss, below the MLP-only result of $20.4\%$. However, from $\beta=4\%$ onward it consistently surpasses the MLP-only sweep, reaching $36.1\%$ recovery at $4\%$ and $73.6\%$ at $32\%$. This pattern indicates that the MLP-only proxy already captures the highest-yield recoverable component under extremely sparse rollback, while the joint intervention reveals additional recoverable damage outside MLP coordinates. Recovery still remains incomplete even at larger budgets, suggesting residual interference outside the reverted MLP+Attn coordinates or non-coordinate-aligned effects. Although the best recovery is obtained with a relatively large $32\%$ budget, this is consistent with the redundancy observed above: multiple sparse coordinate subsets can have similar functional effects, and the current proxy selector is not expected to isolate the conflict subspace optimally. Improving the identification score to more precisely localize the shared conflict subspace is therefore a natural direction for future work.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \clearpage
\end{document}