From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging
Extracted source text · arXiv 2606.00357
\documentclass{article}
\usepackage[preprint]{neurips_2026}
\usepackage[utf8]{inputenc} %
\usepackage[T1]{fontenc} %
\usepackage{hyperref} %
\usepackage{url} %
\usepackage{booktabs} %
\usepackage{amsfonts} %
\usepackage{nicefrac} %
\usepackage{microtype} %
\usepackage{xcolor} %
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{amsmath}
\usepackage{xspace}
\usepackage{svg}
\usepackage{wrapfig}
\usepackage{amssymb}
\usepackage{cleveref}
\usepackage{pifont}
\usepackage{fontawesome5}
\setlength{\abovecaptionskip}{10pt}
\setlength{\belowcaptionskip}{5pt}
\setlength{\textfloatsep}{14pt}
\usepackage{amsthm}
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem{corollary}{Corollary}
\usepackage{enumitem}
\newcommand{\siyue}[1]{\textcolor{blue}{\bf\small [Siyue: #1]}}
\newcommand{\qi}[1]{\textcolor{olive}{\bf\small [Qi: #1]}}
\newcommand{\zoe}[1]{\textcolor{red}{\bf\small [Zoe: #1]}}
\newcommand{\chen}[1]{\textcolor{orange}{\bf\small [Chen: #1]}}
\newcommand{\yulin}[1]{\textcolor{brown}{\bf\small [Yulin: #1]}}
\newcommand{\eg}{\hbox{\emph{e.g.,}}\xspace}
\newcommand{\ie}{\hbox{\emph{i.e.,}}\xspace}
\newcommand{\ours}{Preference Delta Aggregation\xspace}
\newcommand{\shortours}{PDA\xspace}
\newcommand{\ourmerging}{Geometric Alignment Merging\xspace}
\newcommand{\shortmerging}{GAM\xspace}
\newcommand{\tulu}{Tülu3\xspace}
\title{From ``Weak'' Signals to Strong Models: Preference Delta Aggregation with LoRA Merging}
\author{%
Qi Sun$^{1}$\thanks{Equal contribution.} \quad
Siyue Zhang$^{2}$\footnotemark[1] \quad
Yulin Chen$^3$ \quad
Yuxiang Xue$^1$ \quad
Ru Peng$^4$ \quad
Chen Zhao$^1$ \\
\\
$^1$NYU Shanghai \quad $^2$NTU \quad $^3$NYU \quad $^4$ZJU \\
\\
\texttt{\{qs2196, yc7320, yx3044, cz1285\}@nyu.edu} \\
\texttt{siyue001@e.ntu.edu.sg} \quad \texttt{rupeng@zju.edu.cn} \\
\\
\vspace{-2mm}
\href{https://github.com/AlbertQiSun/Preference-Delta-Aggregation}{\faGithub\ \texttt{AlbertQiSun/Preference-Delta-Aggregation}}
}
\begin{document}
\maketitle
\input{sections/0_abstract}
\input{sections/1_introduction}
\input{sections/2_related_work}
\input{sections/3_new}
\input{sections/4_intial_experiments}
\input{sections/5_method}
\input{sections/6_merging}
\input{sections/7_experiment}
\input{sections/8_analysis}
\input{sections/9_conclusion}
\bibliographystyle{plainnat}
\bibliography{reference}
\clearpage
\appendix
\input{sections/10_appendix}
\clearpage
\input{checklist}
\end{document}
\begin{abstract}
Training strong large language models (LLMs) requires high-quality supervision, which is often scarce.
Recent work shows that paired preference data from weak–weaker model pairs (\eg Qwen3 4B over 1.7B), despite the limited quality of individual responses, can provide an effective supervision signal through relative quality deltas, which we term a ``weak'' signal.
This motivates a key research question: can multiple ``weak'' signals be constructively aggregated for improving strong models (\eg Qwen3 8B)?
To this end, we propose \ours\ (\shortours), the first framework that derives a preference delta from each weak-weaker model pair, instantiates it as a LoRA adapter learned through preference optimization, and aggregates the resulting deltas via LoRA merging.
To further mitigate directional interference during LoRA merging, we introduce \ourmerging (\shortmerging), a geometry-aware merging method that aligns adapter subspaces before aggregation, enabling more robust composition of diverse deltas.
Evaluations on knowledge reasoning and agentic search benchmarks show that aggregating multiple ``weak'' signals pushes performance beyond any single signal, with further gains as additional signals are incorporated.
Correspondingly, \shortours\ with \shortmerging improves the strong model by 6.8 and 7.3 points on average for knowledge reasoning and agentic search, respectively. It outperforms all single-delta and multi-delta baselines, exceeding the best single-delta baseline by 2.1 and 4.3 points.
Further analysis attributes these gains to the effective composition of complementary capabilities encoded across distinct preference deltas.\footnote{Code and data will be released after the review period.}
\end{abstract}
\section{Introduction} \label{sec:introduction}
High-quality data is widely recognized as a key ingredient in building strong large language models (LLMs). Accordingly, substantial prior work has focused on curating training data throughout the pipeline, spanning pretraining \citep{datacomp,fineweb,olmo} and post-training \citep{nemotron,toolmind,mdr}. However, many desirable tasks remain difficult to supervise effectively, either due to prohibitive annotation costs or because they exceed human expertise. This challenge has motivated growing interest in leveraging low-quality data to expand the frontier of LLM capabilities \citep{weaktostrong,varying}. Building on this direction, recent work introduces the Delta Learning Hypothesis \citep{geng2025deltalearninghypothesispreference}, which posits that relative quality differences between weak responses (\eg Qwen3 4B over\ 1.7B) can serve as effective ``weak'' supervision signals for improving strong models (\eg Qwen3-8B) through preference tuning.
The success of the Delta Learning Hypothesis naturally raises a key question: can multiple such ``weak'' signals be constructively aggregated to yield further gains? To investigate this, we first examine training-based aggregation strategies, including sequential preference optimization over multiple datasets and joint optimization on their mixture. However, sequential training suffers from severe catastrophic forgetting \citep{catas,mapping}, while joint training fails to outperform the best individual dataset due to gradient conflicts \citep{conflicts}. These results suggest that naive training-based aggregation does not effectively combine multiple ``weak'' signals.
Therefore, we propose the \ours\ framework (\shortours), which independently preference-tunes the strong student model on datasets generated by different weak-weaker model pairs using parameter-efficient training \citep{hu2021loralowrankadaptationlarge}, as illustrated in \Cref{fig:figure1}. Each resulting LoRA adapter captures an improvement direction induced by the quality deltas in its preference data, which we term a \emph{preference delta}. We then aggregate multiple preference deltas through LoRA merging techniques \citep{yadav2023tiesmergingresolvinginterferencemerging,stoica2024modelmergingsvdtie,lorahub}. However, conventional weight averaging ignores the geometry of low-rank update subspaces, often leading to directional interference among misaligned preference deltas. To address this issue, we introduce a novel LoRA merging method, \ourmerging\ (\shortmerging), which decomposes adapters into structured low-rank components and aligns their subspaces before aggregation, enabling more robust composition of diverse preference deltas.
Following prior work \citep{geng2025deltalearninghypothesispreference}, we evaluate the proposed \shortours and \shortmerging on knowledge reasoning tasks, and further extend the evaluation to more challenging agentic search settings requiring both reasoning and retrieval \citep{diffusion,mrmr}. We construct preference datasets from GSM8K \citep{cobbe2021trainingverifierssolvemath} and MuSiQue \citep{trivedi2022musiquemultihopquestionssinglehop} using multiple weak-weaker model pairs drawn from diverse model families, including Llama-3.2 \citep{grattafiori2024llama3herdmodels}, Qwen3 \citep{yang2025qwen3technicalreport}, DeepSeek-R1 \citep{deepseek}, AceSearcher \citep{xu2025acesearcherbootstrappingreasoningsearch}, and Search-R1 \citep{jin2025searchr1trainingllmsreason}. We evaluate strong student models, including Qwen3-8B and \tulu-8B \citep{lambert2025tulu3pushingfrontiers}, across diverse reasoning benchmarks as well as both single-hop and multi-hop search QA benchmarks, and compare against four categories of baselines: the original student models without fine-tuning, single-delta preference tuning, training-based multi-delta aggregation methods (\eg sequential and joint training), and existing LoRA merging approaches such as naive averaging \citep{ilharco2023editingmodelstaskarithmetic}, TIES-Merging \citep{yadav2023tiesmergingresolvinginterferencemerging}, and KnOTS \citep{stoica2024modelmergingsvdtie}.
\begin{figure}
\centering
\includegraphics[width=1\linewidth]{figures/Frame_1.png}
\caption{\textbf{(a)} Preference Delta Aggregation (PDA) independently preference-tunes a strong student model on preference datasets from different weak-weaker model pairs using LoRA fine-tuning, then merges the resulting adapters. \textbf{(b)} In parameter space, each adapter induces a distinct update direction, and PDA aggregates these deltas to compose complementary improvements, moving the student toward a better solution. \textbf{(c)} By aggregating multiple preference deltas, PDA achieves larger gains than baselines trained on preference data from any single weak-weaker model pair.}
\label{fig:figure1}
\end{figure}
Results show that \shortours-\shortmerging improves Qwen3-8B by an average of 6.8 and 7.3 points on knowledge reasoning and agentic search benchmarks, respectively, using only low-quality preference data generated from weaker models. \shortours-\shortmerging outperforms all single-delta and multi-delta baselines, including gains of 2.1 and 4.3 points over the strongest single-delta baseline on average. Similar improvements are consistently observed on \tulu-8B, demonstrating strong generalization across different student models. Notably, the gains continue to increase as additional preference deltas are incorporated, highlighting the benefit of aggregating diverse ``weak'' signals. Our analysis further suggests that these improvements stem from composing complementary capabilities encoded across distinct preference deltas. Our contributions can be summarized as follows:
\begin{enumerate}[
label=\textbf{(\arabic*)},
leftmargin=0pt,
itemindent=2.2em,
labelsep=0em,
align=left,
itemsep=4pt,
topsep=2pt,
parsep=0pt
]
\item \textbf{A framework for improving strong models by aggregating ``weak'' preference signals.}
We propose \shortours, the first framework that improves strong models without relying on high-quality data by leveraging paired preference data from multiple weak-weaker model pairs and aggregating the resulting signals through LoRA merging.
\item \textbf{A LoRA merging method that mitigates directional interference among preference deltas.}
We introduce \shortmerging, a geometry-aware LoRA merging method that decomposes adapters via SVD, aligns their low-rank subspaces on the Grassmannian manifold, averages the aligned singular components, and reconstructs the merged adapter.
\item \textbf{Strong empirical results on reasoning and agentic search tasks.}
Extensive experiments on knowledge reasoning and agentic search benchmarks show that \shortours-\shortmerging constructively aggregates multiple ``weak'' signals, consistently outperforming all single-delta and multi-delta baselines.
\end{enumerate}
\section{Related Work} \label{sec:related-work}
\paragraph{Weak-to-strong Generalization.}
Weak-to-strong generalization studies whether supervision from a weaker model can effectively improve a stronger one without collapsing to the weak supervisor’s performance ceiling \citep{burns2023weaktostronggeneralizationelicitingstrong,ye2026weaktostrong}. Prior studies largely center on two directions: uncovering capabilities already present in pretrained base models \citep{unreasonable}, and using models to recursively refine the data used for subsequent training \citep{meta}. In both settings, supervision is commonly delivered as pointwise judgments from a single weak teacher. However, recent work on AI feedback suggests that such absolute supervision can propagate the teacher’s systematic errors and hallucination biases \citep{zheng2023judgingllmasajudgemtbenchchatbot,li2024hrlaifimprovementshelpfulnessharmlessness}. This motivates relative supervision signals based on quality differences rather than absolute judgments \citep{weaktostrong,varying,geng2025deltalearninghypothesispreference}
\paragraph{Delta Learning.}
Existing alignment approaches rely heavily on human annotations or supervision from frontier models through frameworks such as RLHF or DPO \citep{ouyang2022traininglanguagemodelsfollow, bai2022traininghelpfulharmlessassistant, rafailov2024directpreferenceoptimizationlanguage}.
Delta learning relaxes this costly supervision requirement by constructing preference pairs from a weak model and its weaker variant, using the quality gap between their responses as an optimization signal for a stronger student \citep{geng2025deltalearninghypothesispreference}.
While this paradigm shares conceptual similarities with synthetic preference generation \citep{yuan2025selfrewardinglanguagemodels} and has shown promise on knowledge reasoning tasks, its applicability to complex multi-turn environments such as agentic search—where models must iteratively interact with external tools \citep{yao2023reactsynergizingreasoningacting}—remains underexplored. In this work, we first validate the Delta Learning Hypothesis on more challenging agentic search tasks, and then investigate whether aggregating multiple deltas can yield greater gains than any single delta, which we address with \ours.
\paragraph{LoRA Tuning and Merging.}
Low-Rank Adaptation (LoRA) \citep{hu2021loralowrankadaptationlarge} enables parameter-efficient adaptation of large language models by introducing lightweight low-rank modules, avoiding full-model fine-tuning for each downstream task. Its modular structure naturally supports post-hoc composition, leading to increasing interest in combining specialized capabilities learned from different tasks or domains \citep{jin2025datalessknowledgefusionmerging}.
A common line of work treats adapters as parameter updates that can be algebraically combined. Task Arithmetic \citep{ilharco2023editingmodelstaskarithmetic} models each update as a task vector and composes them through weighted addition. Follow-up methods such as TIES-merging \citep{yadav2023tiesmergingresolvinginterferencemerging} and DARE \citep{yu2024languagemodelssupermario} attempt to reduce interference by sparsifying updates and resolving sign conflicts. KnOTS \citep{stoica2024modelmergingsvdtie} further introduces singular value decomposition (SVD) to align adapter subspaces prior to merging. However, the linear averaging can substantially weaken useful signals when aggregating diverse preference directions within the same task. Thus, we propose \ourmerging, which separates magnitude from direction by decomposing adapters via SVD, aligning subspaces with orthogonal Procrustes, and independently composing directional bases and intrinsic magnitudes, enabling robust aggregation of diverse and conflicting signals.
\section{\ours}
To aggregate multiple ``weak'' supervision signals, we propose \textbf{\ours\ (\shortours)}, a two-stage framework that first derives a preference delta from each weak-weaker model pair and then aggregates the resulting LoRA-instantiated updates through parameter-space merging.
\paragraph{Deriving Preference Deltas.}
Training on paired responses enables models to learn from relative quality differences \citep{geng2025deltalearninghypothesispreference}. Following this principle, we construct \(n\) preference datasets from \(n\) model families. For the \(i\)-th family, we select a weak model \(\theta_{w_i}\) and a weaker (\ie smaller) model \(\theta_{w'_i}\). Given task queries, the two models generate paired responses, where the response from the stronger weak model \(\theta_{w_i}\) is automatically treated as the chosen sample. This yields a preference dataset \(\mathcal{D}_i\) consisting of labeled response pairs.
Each preference dataset \(\mathcal{D}_i\) can be used to improve the strong student model \(\theta_s\) through preference optimization, using objectives such as DPO \citep{rafailov2024directpreferenceoptimizationlanguage} or ORPO \citep{hong2024orpomonolithicpreferenceoptimization}. We refer to the update learned from \(\mathcal{D}_i\) as a \textit{preference delta}, denoted by \(\Delta_i\). Intuitively, each \(\Delta_i\) captures a transferable improvement direction induced by the relative quality difference encoded in preference data generated by the corresponding weak-weaker model pair.
\paragraph{Formulating Delta Aggregation.}
Given preference deltas \(\{\Delta_i\}_{i=1}^{n}\), our goal is to effectively aggregate these diverse signals to improve the strong student model $\theta_s$:
\[
\mathcal{A}(\{\Delta_i\}_{i=1}^{n};\theta_s)=\theta_s',
\]
where \(\mathcal{A}\) denotes an aggregation operator and \(\theta_s'\) is the resulting model. A straightforward way to aggregate these signals is through training-based strategies. For example, one may jointly train on the union of all preference datasets \(\bigcup_{i=1}^{n}\mathcal{D}_i\), or sequentially optimize over them \((\mathcal{D}_1 \rightarrow \mathcal{D}_2 \rightarrow \dots)\). However, such strategies can suffer from gradient conflicts~\citep{conflicts} and catastrophic forgetting~\citep{catas} during training.
\paragraph{Aggregating Deltas via LoRA Merging.}
\shortours instantiates each preference delta as an independently trained LoRA adapter.
For each preference dataset \(\mathcal{D}_i\), we fine-tune the same base student model \(\theta_s\) with LoRA, obtaining an adapter \(\Delta W_i\) that realizes the abstract preference delta \(\Delta_i\) in parameter space. Loading this adapter into the base model gives
\[
\theta_s^{(i)} = \theta_s \oplus \Delta W_i,
\]
where \(\oplus\) denotes applying the LoRA adapter to the target modules of \(\theta_s\). The learned adapters \(\{\Delta W_i\}_{i=1}^{n}\) are then merged to produce the final model \(\theta_s'\).
For a given adapted weight matrix, each LoRA adapter represents a low-rank update
\[
\Delta W_i = B_iA_i \in \mathbb{R}^{d \times k},
\]
where \(k\) and \(d\) denote the input and output dimensions,
\(B_i \in \mathbb{R}^{d \times r}\), \(A_i \in \mathbb{R}^{r \times k}\), and \(r \ll \min(d,k)\).
A standard post-hoc merging baseline averages the expanded updates:
\[
\Delta W_{\mathrm{AVG}}=\frac{1}{n}\sum_{i=1}^{n} B_iA_i,
\qquad
\theta_s' = \theta_s \oplus \Delta W_{\mathrm{AVG}}.
\]
However, naive averaging in Euclidean parameter space can suffer from destructive interference during model merging, including redundant updates and sign conflicts across adapters \citep{yadav2023tiesmergingresolvinginterferencemerging}. Moreover, independently trained LoRA adapters may exhibit rotationally misaligned low-rank subspaces, making direct averaging geometrically inconsistent. This motivates our geometry-aware LoRA merging method, which explicitly aligns low-rank subspaces before aggregation.
\section{\ourmerging} \label{sec:gam}
To address directional conflicts in LoRA merging, we further propose \textbf{\ourmerging\ (\shortmerging)}, which decomposes adapters via SVD, aligns their subspaces on the Grassmannian manifold, independently averages directional bases and magnitudes, and reconstruct the merged adapter.
\paragraph{Step 1: Decomposition.}
Given the low-rank factors $B_i$ and $A_i$ of each trained LoRA adapter, we first instantiate the full weight update matrix $\Delta W_i \in \mathbb{R}^{d \times k}$. We then compute its exact rank-$r$ thin singular value decomposition (SVD):
\begin{equation}
\Delta W_i = U_i S_i V_i^\top
\end{equation}
where $U_i \in \mathbb{R}^{d \times r}$ and $V_i \in \mathbb{R}^{k \times r}$ have strictly orthonormal columns, and $S_i \in \mathbb{R}^{r \times r}$ is a diagonal matrix containing the $r$ singular values.
\paragraph{Step 2: Subspace Alignment on the Grassmannian Manifold.}
The orthonormal bases $U_i$ and $V_i$ define $r$-dimensional low-rank subspaces. Importantly, the same subspace can be represented by many different orthonormal bases related through rotation. As a result, independently trained adapters may learn similar underlying subspaces while adopting rotationally misaligned internal coordinate systems. Directly averaging the basis vectors $U_i$ and $V_i$ can therefore lead to basis mismatch and destructive geometric interference. Geometrically, such subspaces can be viewed as points on the Grassmannian manifold \citep{edelman1998geometryalgorithmsorthogonalityconstraints}.
To resolve this issue, we align all adapters to a reference adapter $\Delta W_1$\footnote{We align all adapters to a reference adapter chosen as the one with the highest validation performance. Ablations in Appendix~\ref{app:reference_sensitivity} show that performance is insensitive to this choice.} using orthogonal Procrustes alignment \citep{hurleyorthonal}:
\begin{equation}
R^U_i = \arg\min_{R \in O(r)} \|U_i R - U_1\|_F, \qquad
R^V_i = \arg\min_{R \in O(r)} \|V_i R - V_1\|_F.
\end{equation}
The resulting rotations synchronize the basis orientations across adapters while preserving the underlying subspaces.\footnote{Appendix~\ref{app:alignment} analyzes the learned rotation matrices $R$, showing that independently trained adapters often learn similar subspaces with highly misaligned bases.} We apply unweighted alignment on the truncated bases so that secondary preference directions are not dominated by the largest principal components. The rotations are then absorbed into the singular components to preserve the original adapter output:
\begin{equation} \label{eq:aligned_factors}
\tilde{U}_i = U_i R^U_i, \qquad
\tilde{S}_i = (R^U_i)^\top S_i R^V_i, \qquad
\tilde{V}_i = V_i R^V_i.
\end{equation}
\paragraph{Step 3: Averaging and Reconstruction.}
Once aligned, the components are averaged independently:
\begin{equation}
\bar{U} = \frac{1}{n} \sum_{i=1}^n \tilde{U}_i, \quad \bar{S} = \frac{1}{n} \sum_{i=1}^n \tilde{S}_i, \quad \text{and} \quad \bar{V} = \frac{1}{n} \sum_{i=1}^n \tilde{V}_i.
\end{equation}
Finally, the averaged components are multiplied to reconstruct the final merged weight update:
\begin{equation}
\Delta W_{\text{GAM}} = \bar{U} \, \bar{S} \, \bar{V}^\top
\end{equation}
This unified dense update $\Delta W_{\text{GAM}} \in \mathbb{R}^{d \times k}$ captures the aggregated preference delta and can be directly added to the pre-trained weights for downstream deployment and evaluation. A theoretical analysis comparing \shortmerging with naive averaging is provided in Appendix~\ref{app:gam_math}.
\section{Experiments} \label{sec:experiments}
In this section, we evaluate the proposed \ours framework on two key tasks: knowledge reasoning and agentic search. We first outline the experimental setups for both tasks (\Cref{sec:reasoning_exp,sec:agentic_exp}), then detail the baselines used for evaluation (\Cref{sec:baselines}), and finally present the main empirical results and corresponding analysis (\Cref{sec:main_results}).
\subsection{Knowledge Reasoning Task Setup} \label{sec:reasoning_exp}
\paragraph{Tasks and Data.} We evaluate \ours on the knowledge reasoning task using four widely used benchmarks: MATH \citep{lightman2023letsverifystepstep}, GPQA \citep{rein2023gpqagraduatelevelgoogleproofqa}, MMLU \citep{hendrycks2021measuringmassivemultitasklanguage}, and GSM8K \citep{cobbe2021trainingverifierssolvemath}. To construct the preference data, we use the full GSM8K training set, consisting of 7,473 queries. For each query, we generate one pair of response trajectories for every weak–weaker model pair. Preference labels are automatically assigned based on model size, with responses from larger (weak) models treated as preferred over those from smaller (weaker) models ($y_{\text{weak}} \succ y_{\text{weaker}}$).
\paragraph{Model Pairs and Preference Deltas.} We use Qwen3-8B \citep{yang2025qwen3technicalreport} as the primary strong student model, and Tülu3-8B \citep{lambert2025tulu3pushingfrontiers} to evaluate cross-model generalization.
The preference deltas are derived from three pairs of weak and weaker models: Llama-3.2 3B over 1B \citep{grattafiori2024llama3herdmodels}, Qwen3 4B over 1.7B, and DeepSeek-R1 7B over 1.5B \citep{deepseek}. As detailed in Appendix \ref{app:abs_perf}, all selected weak and weaker LLMs consistently underperform the student model, thereby ensuring a setting with limited high-quality supervision.
\paragraph{Training and Merging Details.} For preference tuning, we employ Direct Preference Optimization (DPO) \citep{rafailov2024directpreferenceoptimizationlanguage} with LoRA adapters \citep{hu2021loralowrankadaptationlarge} for each preference pair. For our proposed \shortmerging, we extract the singular subspaces of the learned adapters via SVD and align them using the Procrustes method (detailed in \Cref{sec:gam}). Comprehensive hyperparameters for DPO training, as well as the parameter-sweeping configurations for the TIES-Merging baseline (e.g., the density retention fraction $p$), are provided in Appendix~\ref{app:training_details}.
\subsection{Agentic Search Task Setup} \label{sec:agentic_exp}
\paragraph{Tasks and Data.} We evaluate \ours on agentic search using three single-hop QA benchmarks (NQ \citep{kwiatkowski-etal-2019-natural}, TriviaQA \citep{joshi2017triviaqalargescaledistantly}, PopQA \citep{mallen2023trustlanguagemodelsinvestigating}) and four multi-hop QA benchmarks (HotpotQA \citep{yang2018hotpotqadatasetdiverseexplainable}, 2WikiMultiHopQA \citep{ho2020constructingmultihopqadataset}, Bamboogle \citep{press2023measuringnarrowingcompositionalitygap}, MuSiQue \citep{trivedi2022musiquemultihopquestionssinglehop}). Following the same process described in \Cref{sec:reasoning_exp}, the preference data is constructed using weak-weaker model pairs and 8,000 queries randomly sampled from the MuSiQue training set. Unlike the reasoning setup, agentic search trajectories require strict structural formatting. We apply a formatting filter only to the weak model’s (chosen) responses, while retaining formatting errors in the weaker model’s (rejected) responses to preserve the negative signal.
\paragraph{Model Pairs and Preference Deltas.} We use the same student models as in \Cref{sec:reasoning_exp}, namely Qwen3-8B and Tülu3-8B. We construct preference deltas from four LLM families: Llama-3.2 3B over 1B, AceSearcher 7B over 3B \citep{xu2025acesearcherbootstrappingreasoningsearch}, Qwen3 4B over 1.7B, and Search-R1 7B over 3B \citep{jin2025searchr1trainingllmsreason}. Llama 3.2 and Qwen3 are prompted using ReAct \citep{yao2023reactsynergizingreasoningacting} and one example to operate as search agents, whereas AceSearcher and Search-R1 are fine-tuned for this role. Similar to the model selection in \Cref{sec:reasoning_exp}, all selected weak and weaker LLMs underperform the student model as shown in Appendix~\ref{app:abs_perf}.
\paragraph{Training and Merging Details.} Unlike knowledge reasoning, agentic search requires strict structural formatting, where reasoning and retrieval steps are interleaved. In preliminary experiments, although DPO captures the preference deltas, it severely degrades the student’s format correctness rate (from 0.97 to 0.83). To address this issue, we adopt ORPO \citep{hong2024orpomonolithicpreferenceoptimization} for preference tuning, which incorporates an implicit SFT regularization to anchor formatting (restoring the correctness rate to 0.97) while optimizing preferences.
To ensure that the observed gains arise from preference alignment rather than format imitation, we introduce a direct SFT baseline trained solely on the chosen trajectories from Qwen3-4B, which underperforms the student baseline (see Appendix~\ref{app:sft}). Apart from the loss function, all other configurations remain consistent with the knowledge reasoning setup. Additional training and evaluation details are provided in Appendix~\ref{app:training_details}.
\subsection{Baselines} \label{sec:baselines}
We comprehensively evaluate the proposed \ours framework and the \ourmerging method against three categories of well-established baselines:
\textbf{(1) Student Baseline}: We evaluate the strong student LLMs, \ie Qwen3-8B and Tülu3-8B, without any fine-tuning.
\textbf{(2) Single Preference Delta Baselines}: We assess the same student model individually fine-tuned on different quality Deltas.
\textbf{(3) Multiple Preference Delta Baselines}: We compare methods that leverage signals from multiple quality and preference deltas, covering both training-based and LoRA-based approaches. For training-based aggregation methods, we consider \textit{Sequential Training}, where the student is fine-tuned consecutively across different deltas, and \textit{Joint Training}, which optimizes the student model at once with a mixture of data from multiple deltas. For LoRA-based aggregation methods within our \ours framework, we compare our proposed \shortmerging with existing LoRA merging methods, including \textit{Naive Averaging} \citep{ilharco2023editingmodelstaskarithmetic}, \textit{TIES-Merging} \citep{yadav2023tiesmergingresolvinginterferencemerging}, and \textit{KnOTS} \citep{stoica2024modelmergingsvdtie}.
\subsection{Main Results} \label{sec:main_results}
\input{figures/general_tasks}
\paragraph{``Weak'' signals can effectively improve strong LLMs across both knowledge reasoning and agentic search tasks.} On knowledge reasoning benchmarks (Table~\ref{tab:general_results}), we validate that relative quality deltas from weak–weaker model pairs provide an effective preference-learning signal for improving strong models, corroborating the Delta Learning Hypothesis of \citet{geng2025deltalearninghypothesispreference}. For example, fine-tuning the strong student model Qwen3-8B on preference data constructed from DeepSeek-R1 (7B over 1.5B) improves the average score from 72.5 to 77.2. To assess whether this effect transfers beyond reasoning tasks, we extend the Delta Learning Hypothesis to agentic search and provide its first empirical validation. As shown in Table~\ref{tab:main_results}, the strongest single delta (\ie Search-R1 7B over 3B) improves the student model’s average F1 score from 50.7 to 53.7. We further verify that these data are individually weak, as standard SFT on the same data degrades the student model (Appendix~\ref{app:sft}). Finally, different weak–weaker model pairs yield substantially different levels of improvement, revealing clear variation in effectiveness across model pairs.
\paragraph{LoRA merging methods can constructively aggregate ``weak'' signals, surpassing any single signal, with additional signals yielding further gains.} A natural approach for aggregating multiple signals is sequential preference tuning over multiple preference datasets. However, as shown in \Cref{tab:general_results,tab:main_results}, Sequential DPO ($2\Delta$) and Sequential ORPO ($2\Delta$) perform worse than all single preference-delta baselines, indicating severe catastrophic forgetting during sequential training. We further evaluate joint preference tuning on the combined dataset, which still performs worse than the best single-delta baseline. In contrast to above training-based approaches, LoRA-based aggregation methods demonstrate strong results. For example, \shortours-TIES ($3\Delta$) attains an average EM score of 77.7, surpassing the best single-delta baseline of 77.2 for Qwen3-8B on knowledge reasoning. Similarly, PDA-TIES ($4\Delta$) achieves an average F1 score of 57.4, exceeding the best single-delta baseline of 53.7 for Qwen3-8B on agentic search. Notably, for LoRA-based aggregation, performance improves consistently as more preference deltas are incorporated. For example, \shortours-\shortmerging ($4\Delta$) achieves the best average F1 score of 58.0 on agentic search for Qwen3-8B, outperforming $2\Delta$ by 3.2 points and $3\Delta$ by 0.6 points. Moreover, stronger individual deltas tend to contribute larger gains when included in the aggregation. These results suggest that different ``weak'' signals encode complementary information that can be constructively aggregated.
\paragraph{\ourmerging more effectively aggregates diverse ``weak'' signals than existing LoRA merging methods, yielding larger performance gains.}
Our proposed \shortours-\shortmerging consistently outperforms established methods such as \shortours-Averaging, \shortours-KnOTS, and \shortours-TIES across both domains. On knowledge reasoning tasks, where optimization directions are relatively aligned, naive averaging provides a strong baseline. Nevertheless, \shortours-\shortmerging achieves the best average Exact Match score of 79.3 in the $3\Delta$ setting for Qwen3-8B.
On the more challenging agentic search tasks, the limitations of existing LoRA merging methods become more pronounced. Since diverse preference deltas occupy distinct, near-orthogonal parameter subspaces (Appendix~\ref{app:alignment}), \shortours-TIES can discard useful update directions during magnitude-based sparsification. As a result, \shortours-TIES plateaus at 37.8 F1 on the MuSiQue subset at $4\Delta$ for Qwen3-8B, while \shortours-\shortmerging continues to improve. Qualitative analysis in Appendix~\ref{app:alignment} further shows that \shortours-\shortmerging better aligns diverse LoRA adapters before merging, thereby reducing interference among adapters. Case studies in Appendix~\ref{app:reasoning-cases} and ~\ref{app:search-cases} further illustrate that \shortours-\shortmerging yields more robust and effective reasoning behaviors. Overall, these results highlight the advantage of geometry-aware merging for aggregating diverse preference signals. The observed improvements remain consistent across different student models, including Qwen3-8B and \tulu-8B.
\input{figures/main_results}
\begin{table*}[t]
\centering
\caption{Exact Match scores on knowledge reasoning benchmarks. For multi-delta experiments, preference deltas are aggregated sequentially in the following order: DeepSeek-R1 $\rightarrow$ Qwen3 $\rightarrow$ Llama-3.2. For example, 2$\Delta$ denotes aggregating DeepSeek-R1 and Qwen3. The best score is highlighted in \textbf{bold} for single-delta and multi-delta baselines.}
\small
\setlength{\tabcolsep}{4pt} %
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{} l cccc c cccc c @{}}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{5}{c}{\textbf{Qwen3-8B}} & \multicolumn{5}{c}{\textbf{Tülu3-8B}} \\
\cmidrule(lr){2-6} \cmidrule(lr){7-11}
& \textbf{MATH} & \textbf{GPQA} & \textbf{MMLU} & \textbf{GSM8K} & \textbf{Avg.}
& \textbf{MATH} & \textbf{GPQA} & \textbf{MMLU} & \textbf{GSM8K} & \textbf{Avg.}\\
\midrule
Student Baseline & 81.5 & 42.1 & 78.5 & 88.0 & 72.5 & 78.2 & 38.5 & 75.2 & 85.1 & 69.3 \\
\midrule
\multicolumn{11}{@{}l}{\textit{Single Preference Delta}} \\
\quad Llama-3.2 (3B over 1B) & 82.8 & 42.6 & 79.1 & 88.5 & 73.3 & 79.5 & 39.2 & 76.0 & 86.0 & 70.2 \\
\quad Qwen3 (4B over 1.7B) & 84.5 & 44.0 & 80.2 & 90.2 & 74.7 & 81.2 & 40.5 & 77.5 & 88.2 & 71.9 \\
\quad DeepSeek-R1 (7B over 1.5B) & \textbf{87.2} & \textbf{46.5} & \textbf{81.8} & \textbf{93.4} & \textbf{77.2} & \textbf{84.6} & \textbf{43.1} & \textbf{79.2} & \textbf{91.0} & \textbf{74.5} \\
\midrule
\multicolumn{11}{@{}l}{\textit{Multiple Preference Deltas}} \\
\quad Sequential DPO (2$\Delta$) & 79.2 & 39.8 & 76.9 & 88.2 & 71.0 & 76.3 & 34.6 & 72.3 & 82.2 & 66.4 \\
\quad Joint DPO (3$\Delta$) & 85.1 & 44.3 & 80.6 & 89.6 & 74.9 & 82.7 & 40.2 & 78.4 & 89.5 & 72.7 \\
\quad \shortours-Averaging (3$\Delta$) & 86.8 & 46.2 & 80.5 & 92.5 & 76.5 & 83.5 & 42.8 & 78.4 & 90.1 & 73.7 \\
\quad \shortours-KnOTS (3$\Delta$) & 87.8 & 47.1 & 81.4 & 93.7 & 77.5 & 85.5 & 44.2 & 79.7 & 91.4 & 75.2 \\
\quad \shortours-TIES (2$\Delta$) & 87.6 & 46.8 & 80.4 & 93.7 & 77.1 & 85.2 & 43.9 & 79.5 & 91.2 & 75.0 \\
\quad \shortours-TIES (3$\Delta$) & 88.1 & 47.3 & 81.5 & 93.8 & 77.7 & 85.8 & 44.5 & 79.8 & 91.5 & 75.4 \\
\quad \shortours-\shortmerging (2$\Delta$) & 88.5 & 47.9 & 82.4 & 94.2 & 78.3 & 86.2 & 44.8 & 80.5 & 92.2 & 75.9 \\
\quad \shortours-\shortmerging (3$\Delta$) & \textbf{89.6} & \textbf{48.8} & \textbf{83.1} & \textbf{95.5} & \textbf{79.3} & \textbf{87.4} & \textbf{46.0} & \textbf{81.2} & \textbf{93.5} & \textbf{77.0} \\
\bottomrule
\end{tabular}%
}
\label{tab:general_results}
\end{table*}
\begin{table*}[t]
\centering
\caption{Token-level F1 scores on agentic search benchmarks. For multi-delta experiments, preference deltas are aggregated sequentially in the following order: Search-R1 $\rightarrow$ Qwen3 $\rightarrow$ AceSearcher $\rightarrow$ Llama-3.2. For example, $2\Delta$ denotes aggregating Search-R1 and Qwen3. The best score is highlighted in \textbf{bold} for single-delta and multi-delta baselines.}
\setlength{\tabcolsep}{3.5pt} %
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{} l ccc cccc c ccc cccc c @{}}
\toprule
\multirow{3}{*}{\textbf{Method}} &
\multicolumn{8}{c}{\textbf{Qwen3-8B}} &
\multicolumn{8}{c}{\textbf{\tulu-8B}} \\
\cmidrule(lr){2-9} \cmidrule(l){10-17}
& \multicolumn{3}{c}{\textbf{Single-hop QA}} & \multicolumn{4}{c}{\textbf{Multi-hop QA}} & \multirow{2}{*}{\textbf{Avg.}}
& \multicolumn{3}{c}{\textbf{Single-hop QA}} & \multicolumn{4}{c}{\textbf{Multi-hop QA}} & \multirow{2}{*}{\textbf{Avg.}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-8} \cmidrule(lr){10-12} \cmidrule(lr){13-16}
& \textbf{NQ} & \textbf{TQA} & \textbf{Pop.}
& \textbf{Hot.} & \textbf{2Wiki} & \textbf{Bam.} & \textbf{MuSi.} &
& \textbf{NQ} & \textbf{TQA} & \textbf{Pop.}
& \textbf{Hot.} & \textbf{2Wiki} & \textbf{Bam.} & \textbf{MuSi.} & \\
\midrule
Student Baseline
& 46.2 & 61.4 & 44.6 & 53.8 & 58.1 & 61.6 & 28.9 & 50.7
& 45.8 & 60.5 & 43.0 & 52.4 & 55.1 & 58.8 & 25.6 & 48.7 \\
\midrule
\multicolumn{17}{@{}l}{\textit{Single Preference Delta}} \\
\quad Llama-3.2 (3B over 1B) & 46.5 & 61.6 & 44.9 & 53.9 & 58.5 & 62.0 & 29.1 & 50.9 & 46.0 & 60.8 & 43.5 & 52.6 & 55.8 & 59.5 & 26.2 & 49.2 \\
\quad AceSearcher (7B over 3B) & 47.8 & 62.1 & 46.2 & 54.1 & 60.1 & 62.8 & 30.5 & 51.9 & 46.5 & 61.5 & 45.2 & 53.0 & 57.4 & 61.0 & 27.8 & 50.3 \\
\quad Qwen3 (4B over 1.7B) & 48.5 & 62.8 & 47.0 & 54.2 & 61.5 & 63.5 & 32.8 & 52.9 & 46.8 & 62.0 & 46.8 & 53.5 & 59.2 & 62.1 & 28.5 & 51.3 \\
\quad Search-R1 (7B over 3B) & \textbf{49.1} & \textbf{63.2} & \textbf{47.9} & \textbf{54.4} & \textbf{62.6} & \textbf{64.0} & \textbf{34.7} & \textbf{53.7} & \textbf{47.2} & \textbf{62.8} & \textbf{48.7} & \textbf{53.8} & \textbf{61.3} & \textbf{62.9} & \textbf{29.5} & \textbf{52.3} \\
\midrule
\multicolumn{17}{@{}l}{\textit{Multiple Preference Deltas}} \\
\quad Sequential ORPO (2$\Delta$) & 44.6& 58.2& 41.7& 51.4& 52.2& 59.5& 27.6& 47.9 & 42.8& 56.9& 39.8& 51.6& 50.1& 53.7& 21.4& 45.2 \\
\quad Joint ORPO (4$\Delta$)& 48.7& 62.8& 46.8& 53.6& 62.0& 63.6& 32.0& 52.8 & 47.0& 62.3& 47.1& 53.4& 60.6& 62.5& 28.9& 51.7 \\
\quad \shortours-Averaging (4$\Delta$)
& 53.2 & 65.4 & 50.3 & 57.9 & 64.9 & 69.5 & 36.9 & 56.9
& 51.2 & 65.8 & 52.5 & 56.4 & 63.8 & 68.3 & 32.0 & 55.7 \\
\quad \shortours-KnOTS (4$\Delta$) & 53.0 & 65.2 & 50.1 & 57.8 & 64.7 & 69.8 & 37.0 & 56.8 & 50.9 & 65.5 & 52.3 & 56.2 & 63.6 & 68.1 & 31.9 & 55.5 \\
\quad \shortours-TIES (2$\Delta$) & 51.6 & 64.7 & 49.2 & 56.8 & 63.4 & 66.9 & 35.6 & 55.5 & 49.0 & 64.1 & 50.7 & 55.5 & 62.8 & 66.4 & 31.6 & 54.3 \\
\quad \shortours-TIES (3$\Delta$)
& 52.4 & 65.0 & 51.2 & 58.2 & 65.2 & 69.7 & 37.8 & 57.1
& 50.9 & 65.4 & 52.6 & 56.8 & 63.8 & 68.6 & 32.0 & 55.7 \\
\quad \shortours-TIES (4$\Delta$)
& 52.8 & 65.3 & 51.5 & 58.8 & 65.4 & 70.3 & 37.8 & 57.4
& 51.4 & 65.7 & 52.8 & 57.2 & 64.0 & 68.8 & 32.1 & 56.0 \\
\quad \shortours-\shortmerging (2$\Delta$)
& 50.4 & 64.1 & 48.7 & 55.3 & 63.5 & 65.7 & 36.1 & 54.8
& 49.6 & 64.0 & 50.4 & 54.4 & 62.4 & 64.7 & 30.3 & 53.7 \\
\quad \shortours-\shortmerging (3$\Delta$)
& 52.8 & 65.7 & 51.6 & 58.6 & 65.6 & 69.8 & 37.9 & 57.4
& 51.1 & 65.6 & 52.7 & 57.1 & 63.8 & 68.8 & 32.4 & 55.9 \\
\quad \shortours-\shortmerging (4$\Delta$)
& \textbf{53.2} & \textbf{65.9} & \textbf{52.4} & \textbf{59.3} & \textbf{65.9} & \textbf{71.2} & \textbf{38.1} & \textbf{58.0}
& \textbf{51.6} & \textbf{65.9} & \textbf{52.9} & \textbf{57.2} & \textbf{64.2} & \textbf{69.1} & \textbf{32.8} & \textbf{56.2} \\
\bottomrule
\end{tabular}%
}
\label{tab:main_results}
\end{table*}
\section{Analysis} \label{sec:analysis}
\subsection{Are preference deltas always effective signals for learning stronger models?} \label{sec:self_play}
The Delta Learning Hypothesis \citep{geng2025deltalearninghypothesispreference} posits that a stronger student can effectively learn from a weak-weaker model pair, provided that the chosen response consistently outperforms the rejected one along informative axes. If the weak and weaker models trade wins randomly or exhibit noisy quality gaps, the preference signal degrades significantly. The recommended source pairs therefore typically share the same model family but differ in scale (\eg Qwen3 4B over 1.7B), which helps ensure a stable and coherent preference margin \citep{geng2025deltalearninghypothesispreference}. In our main experiments, we follow this guideline by constructing preference deltas exclusively from intra-family model pairs.
Beyond the setting studied in the Delta Learning Hypothesis, we identify an additional failure case: the preference delta derived from the ZeroSearch family (\ie 7B over 3B) \citep{sun2025zerosearchincentivizesearchcapability} fails to improve a stronger student model. In our experiments, training on this preference data caused a 1.3 F1 score drop on MuSiQue relative to the student baseline (\ie Qwen3-8B). ZeroSearch trains LLM search agents without real search engines by relying on a simulation LLM that generates progressively harder retrieval environments. We hypothesize that the degradation stems from mismatched task difficulties induced by different simulation LLMs during training, causing the two models to adapt along different informative axes despite sharing the same family. A similar failure was observed when experimenting with another self-play framework, SSRL \citep{ssrl}. These results suggest that preference deltas are not universally beneficial and should be selected carefully before aggregation.
\subsection{What capabilities are composed when aggregating preference deltas?}\label{sec:behavioral}
\input{figures/behavior}
Having established that aggregating preference deltas yields consistent quantitative gains, we next examine their qualitative effects. We conduct a controlled two-delta ablation on the agentic search task, where \shortours-\shortmerging aggregates two preference deltas. Prior work identifies four reasoning behaviors critical to effective search agents: Information Verification, Authority Evaluation, Adaptive Search, and Error Recovery \citep{jin2026beneficialreasoningbehaviorsagentic}. We therefore analyze these behaviors for models trained on each individual delta and compare them with the model trained on aggregated deltas. Table~\ref{tab:behavioral} shows that single-delta training induces strong specialization: the Search-R1 delta achieves high Adaptive Search (88.9\%) but weak Information Verification (5.3\%, down from 57.2\% in the vanilla model), whereas the Qwen3 delta attains strong Information Verification (75.9\%) but limited Adaptive Search (14.7\%).
Aggregating these two deltas composes their complementary strengths. \shortours-\shortmerging ($2\Delta$) achieves 56.7\% Adaptive Search and 61.2\% Information Verification simultaneously, avoiding the severe deficiencies each single-delta baseline exhibits on its non-dominant behavior. Both scores exceed the averages of the two specialists (51.8\% and 40.6\%, respectively), suggesting constructive composition rather than simple interpolation. Case studies in Appendix~\ref{app:search-cases} further show that aggregating multiple preference deltas enables the same model to exhibit complementary reasoning behaviors. Since agentic search requires coordinated competence across these behaviors, this more balanced behavioral profile plausibly explains the gains from preference delta aggregation.
\begin{table*}[t!]
\centering
\caption{Model behavior analysis on 500 MuSiQue dev-set samples, annotated by Qwen3-32B and verified by the authors. Single-delta baselines exhibit distinct behavioral specializations (\eg Search-R1 excels at adaptive search but struggles with verification), whereas \shortours-\shortmerging balances these complementary behavioral capabilities.}
\small
\resizebox{\textwidth}{!}{
\begin{tabular}{lccccc}
\toprule
Method & Steps & Verification (\%) & Adaptive Search (\%) & Authority (\%) & Recovery (\%)\\
\midrule
Student Qwen3-8B & 4.28 & 57.2 & 23.6 & 0.6 & 9.3\\
\midrule
\multicolumn{4}{l}{\textit{Single Preference Delta}} \\
\quad Qwen3 (4B over 1.7B) & 4.64 & 75.9 & 14.7 & 2.6 & 10.7\\
\quad Search-R1 (7B over 3B) & 4.09 & 5.3 & 88.9 & 0.8 & 24.3\\
\midrule
\shortours-\shortmerging (2$\Delta$) & 4.43 & 61.2 & 56.7 & 1.2 & 17.4\\
\bottomrule
\end{tabular}
}
\label{tab:behavioral}
\end{table*}
\section{Conclusion} \label{sec:conclusion}
In this work, we demonstrate that preference signals derived from relative quality gaps between weak model pairs can improve stronger student models across both knowledge reasoning and agentic search tasks. To achieve greater gains beyond a single preference delta, we introduce \shortours to constructively combine diverse signals. We show that standard training-based aggregation approaches, including joint and sequential training, struggle with gradient cancellation and catastrophic forgetting. \shortours addresses this issue by separately performing preference optimization via LoRA fine-tuning and merging the resulting adapters post hoc. To mitigate geometric interference during this merging phase, we propose \shortmerging that aligns the singular bases of different adapters on the Grassmannian manifold before averaging and merging. Empirical evaluations show that \shortours with \shortmerging improves the strong model by an average of 6.8 and 7.3 points on knowledge reasoning and agentic search, respectively, outperforming all single-delta and multi-delta baselines.
\section{Limitations} \label{sec:limitation}
While \shortours\ enables effective aggregation of multiple preference deltas, three limitations remain. First, complementary gains rely on directionally diverse signals; aggregating behaviorally homogeneous pairs yields only marginal improvements, highlighting the need for systematic diversity metrics for pair selection. Second, our experiments consider a limited set of weak-weaker model pairs, and the scaling behavior with broader or larger collections remains to be studied. Finally, although we evaluate both training-based and LoRA-based aggregation methods, other aggregation strategies may exist that were not explored in this work.
\section{Performance of Student and Supervisor Models}
\label{app:abs_perf}
To provide a comprehensive view of our experimental setup, we report the absolute zero-shot and few-shot performance of all models (both supervisors and students) used in this study. This serves two purposes: first, to confirm that our supervisor models exhibit a measurable performance gap (\ie preference delta) between the weak and weaker variants; second, to verify that the student models (Qwen3-8B and \tulu-8B) consistently outperform their respective supervisors, ensuring a valid weak-to-strong supervision setting.
As shown in \Cref{tab:absolute_performance}, the intended performance hierarchy is consistently observed. For general-purpose model families like Qwen3 and Llama-3.2, we report performance across all eight benchmarks. For domain-specialized families such as DeepSeek-R1 (reasoning-focused) and the agentic-search models Search-R1 and AceSearcher, we report performance only on their respective target domains. This unified absolute performance table provides the necessary context for the relative improvements discussed in the main results (\Cref{tab:main_results}).
\begin{table*}[h]
\centering
\caption{Performance of Student and Supervisor Models. We report zero-shot/few-shot metrics across Knowledge Reasoning and Agentic Search benchmarks. The capability hierarchy (Weaker $<$ Weak $<$ Student) is maintained across all model families, validating the data setup for preference extraction and aggregation.}
\setlength{\tabcolsep}{3.5pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{} l ccccc ccccc @{}}
\toprule
\multirow{2}{*}{\textbf{Model}} & \multicolumn{5}{c}{\textbf{Knowledge Reasoning}} & \multicolumn{5}{c}{\textbf{Agentic Search}} \\
\cmidrule(lr){2-6} \cmidrule(l){7-11}
& \textbf{MAT.} & \textbf{GPQ.} & \textbf{MML.} & \textbf{GSM.} & \textbf{Avg.} & \textbf{Hot.} & \textbf{2Wiki} & \textbf{Bam.} & \textbf{Mus.} & \textbf{Avg.} \\
\midrule
Qwen3-8B & 81.5 & 42.1 & 78.5 & 88.0 & 72.5 & 53.8 & 58.1 & 61.6 & 28.9 & 50.6 \\
\tulu-8B & 78.2 & 38.5 & 75.2 & 85.1 & 69.3 & 52.4 & 55.1 & 58.8 & 25.6 & 48.0 \\
\midrule
DS-R1-Distill-1.5B (Weaker) & 58.4 & 32.1 & 60.5 & 70.2 & 55.3 & -- & -- & -- & -- & -- \\
DS-R1-Distill-7B (Weak) & 72.5 & 39.4 & 73.1 & 83.6 & 67.2 & -- & -- & -- & -- & -- \\
\addlinespace[2pt]
Qwen3-1.7B (Weaker) & 56.2 & 30.5 & 58.3 & 68.4 & 53.4 & 41.2 & 43.5 & 46.8 & 18.2 & 37.4 \\
Qwen3-4B (Weak) & 68.1 & 36.8 & 67.2 & 78.5 & 62.7 & 48.5 & 51.2 & 53.4 & 23.6 & 44.2 \\
\addlinespace[2pt]
Llama-3.2-1B (Weaker) & 48.5 & 26.4 & 52.1 & 62.3 & 47.3 & 36.4 & 39.8 & 42.1 & 15.6 & 33.5 \\
Llama-3.2-3B (Weak) & 62.3 & 33.5 & 64.8 & 74.1 & 58.7 & 46.2 & 49.5 & 51.8 & 22.3 & 42.5 \\
Search-R1-3B (Weaker) & -- & -- & -- & -- & -- & 42.5 & 46.1 & 49.3 & 19.5 & 39.4 \\
Search-R1-7B (Weak) & -- & -- & -- & -- & -- & 50.1 & 54.3 & 57.2 & 26.4 & 47.0 \\
\addlinespace[2pt]
AceSearcher-3B (Weaker) & -- & -- & -- & -- & -- & 41.8 & 44.2 & 47.5 & 18.8 & 38.1 \\
AceSearcher-7B (Weak) & -- & -- & -- & -- & -- & 49.5 & 52.8 & 55.6 & 25.1 & 45.8 \\
\bottomrule
\end{tabular}%
}
\label{tab:absolute_performance}
\end{table*}
\section{Analysis of the Geometric Relationship of Weight Updates} \label{app:alignment}
To understand the mechanism behind the improved performance of \ourmerging, it is essential to examine the geometric relationship of the weight updates prior to aggregation. Parameter-efficient modules (e.g., LoRA) fine-tuned on distinct downstream tasks naturally reside in different low-rank subspaces. Standard averaging assumes these weights share a uniform directional basis, which often leads to parameter interference and capability degradation. To visualize this, we evaluate the directional consistency---measured by the subspace cosine similarity---across all attention heads and layers before the final merging step (see Figure~\ref{fig:head_level_alignment}).
\begin{figure}[htbp]
\centering
\label{fig:head_level_alignment}
\includegraphics[width=1\linewidth]{figures/graph_head.pdf}
\caption{Head-level Directional Consistency prior to aggregation. Heatmaps display the subspace cosine similarity across layers and attention heads for different merging strategies. Averaging and TIES-Merging show inherent geometric misalignment (blue regions). KnOTS exhibits partial consistency (light orange) due to the asymmetric alignment of only the left singular bases. \ours aligns both the left and right singular bases via orthogonal Procrustes, achieving uniform geometric consensus (dark red) across all dimensions.}
\end{figure}
\textbf{The insufficiency of element-wise sign consensus.} As shown in the top row of Figure~\ref{fig:head_level_alignment}, Naive Averaging exhibits notable geometric misalignment, indicated by the predominantly blue regions. TIES-Merging attempts to resolve this interference by enforcing element-wise sign consensus. However, its directional consistency remains low. This discrepancy highlights a fundamental limitation: resolving sign conflicts at the scalar level does not inherently align the underlying singular vectors. Consequently, the actual feature manifolds remain misaligned, indicating that sign consensus alone is insufficient to prevent interference during weight aggregation.
\textbf{The residual mismatch in asymmetric alignment.} KnOTS (bottom-left) partially mitigates this issue by projecting the adapters onto a shared left singular basis ($U$). While this improves consistency compared to na\"ive averaging, the alignment is structurally asymmetric. Because the right singular basis ($V$) remains unaligned prior to merging, a substantial degree of residual misalignment persists across the attention heads.
\textbf{Comprehensive alignment via \shortmerging.} In contrast, \shortmerging explicitly addresses this geometric mismatch on both sides. By solving the orthogonal Procrustes problem on the Grassmannian manifold, \shortmerging aligns both the left and right singular bases ($U$ and $V$) of the task-specific weights to a shared reference coordinate system. As demonstrated in the bottom-right of Figure~\ref{fig:head_level_alignment}, this dual alignment results in uniformly high directional consistency (dark red) across all layers and attention heads. By systematically aligning the geometric manifolds before merging, \shortmerging effectively minimizes cross-task interference and structurally preserves the specialized capabilities of each adapter. Comparison among TIES, KnOTS and \shortmerging shows that rotation $R$ is a key factor for our performance improvements.
\section{A Theoretical Perspective on \shortmerging vs. Naive Averaging}
\label{app:gam_math}
To understand why \ourmerging (\shortmerging) outperforms naive averaging, we analyze the merging operators under an idealized Signal-plus-Noise (SPN) setting. We provide heuristic scaling arguments and sketches to build intuition, rather than formal theoretical proofs.
Let the observed adapter weights be perturbations of a shared underlying ``true'' preference delta $\Delta W^* \in \mathbb{R}^{d \times k}$ of rank $r$, with its $r$-th singular value denoted as $\sigma_r^\star := \sigma_r(\Delta W^\star)$:
\begin{equation}\label{eq:spn}
\Delta W_i = \Delta W^* + \mathcal{E}_i, \quad i = 1, \dots, n,
\end{equation}
where $\mathcal{E}_i$ are independent, zero-mean sub-Gaussian optimization noise matrices. For tractability and to build intuition in this idealized setting, we initially model this as isotropic noise with $\mathbb{E}[\mathcal{E}_i] = 0$ and $\mathbb{E}[\|\mathcal{E}_i\|_F^2] = d k \sigma^2$ (i.e., $\sigma^2$ is the per-entry variance). While real preference tuning noise is highly anisotropic, we show in Section~\ref{sec:gam_variance} that our core variance-reduction conclusions continue to hold under generalized anisotropic noise structures.
\subsection{Suboptimality of Naive Averaging: The Dimensionality Curse}
For standard Euclidean averaging, the estimator is $\Delta \hat{W}_{avg} = \frac{1}{n} \sum_{i=1}^n \Delta W_i$. Because the expectation is linear, the estimator is unbiased ($\mathbb{E}[\Delta \hat{W}_{avg}] = \Delta W^*$). However, its MSE is strictly bottlenecked by the ambient dimensionality:
\begin{equation} \label{eq:mse_avg}
\text{MSE}(\text{Averaging}) = \mathbb{E} \left[ \left\| \frac{1}{n}\sum_{i=1}^n \mathcal{E}_i \right\|_F^2 \right] = \frac{dk\sigma^2}{n}.
\end{equation}
In the context of Large Language Models, $d$ and $k$ are extremely large (e.g., $\sim 4096$). Averaging absorbs noise from the entire parameter space, failing to exploit the intrinsic low-rank structure of the preference signal.
\subsection{Variance Reduction of \ourmerging (\shortmerging)}
\label{sec:gam_variance}
Our proposed \shortmerging explicitly resolves this curse of dimensionality by independently aligning both $U_i$ and $V_i$ via Orthogonal Procrustes mapping, completely decoupling magnitude ($\Sigma$) from direction. We now make the variance-reduction claim precise: under the idealized SPN model of Eq.~\ref{eq:spn}, \shortmerging behaves to leading order as a projection of the noisy average $\overline{\Delta W} := \tfrac{1}{n}\sum_i \Delta W_i$ onto the rank-$r$ manifold, and its Frobenius MSE scales with the intrinsic dimension of that manifold rather than the ambient dimension $dk$.
\paragraph{Idealization.}
We work under the following idealization, which is standard in the low-rank denoising literature and which we state explicitly here:
\begin{itemize}
\item[\textbf{(A1)}] In the small-noise regime, Procrustes alignment recovers the relative rotations between the $U_i$ and $V_i$ exactly, so that after alignment, averaging, and re-orthonormalization, \shortmerging coincides with the truncated rank-$r$ SVD of the noisy average: $\widehat{\Delta W}_{\text{\shortmerging}} = \mathcal{P}_r\!\bigl(\overline{\Delta W}\bigr)$.
\end{itemize}
Assumption (A1) is justified conceptually: in the noiseless limit, adapters sharing a common subspace differ only by orthogonal rotations. After exact Procrustes alignment, all $\Delta W_i$ collapse to a common rank-$r$ matrix. Averaging then trivially acts as the identity, and since $\mathcal{P}_r$ leaves rank-$r$ inputs invariant, the process perfectly matches the truncated SVD of the average. Under small perturbations, this equivalence holds up to higher-order curvature terms that are absorbed into the remainder.
\paragraph{The rank-$r$ manifold and its tangent space.}
The set of rank-$r$ matrices $\mathcal{M}_r \subset \mathbb{R}^{d \times k}$ is a smooth manifold of dimension
\begin{equation}
\dim \mathcal{M}_r = r(d+k-r),
\end{equation}
obtained by counting the parameters of a thin SVD: $dr - \binom{r+1}{2}$ for an orthonormal $U$ (representing the $dr$ entries minus the orthonormality constraints of the Stiefel manifold), $kr - \binom{r+1}{2}$ for an orthonormal $V$, and $r$ singular values. Writing $U^\star, V^\star$ for the orthonormal factors of $\Delta W^\star$ and $U^\star_\perp, V^\star_\perp$ for their orthogonal complements, the tangent space at $\Delta W^\star$ is
\begin{equation}
T_{\Delta W^\star}\mathcal{M}_r =
\bigl\{\, U^\star A V^{\star\top} + U^\star_\perp B V^{\star\top}
+ U^\star C V^{\star\top}_\perp \;:\; A, B, C \text{ free} \,\bigr\},
\end{equation}
with $A \in \mathbb{R}^{r\times r}$, $B \in \mathbb{R}^{(d-r)\times r}$, $C \in \mathbb{R}^{r\times(k-r)}$. Its Frobenius-orthogonal projector is
\begin{equation}
\mathcal{P}_T(X) = U^\star U^{\star\top} X + X V^\star V^{\star\top} - U^\star U^{\star\top} X V^\star V^{\star\top},
\label{eq:tangent_proj}
\end{equation}
and direct counting gives $\operatorname{rank}(\mathcal{P}_T) = r(d+k-r)$.
\paragraph{Projection lemma.}
The key fact we exploit is that an isotropic noise matrix, when projected onto an $m$-dimensional subspace of $\mathbb{R}^{d\times k}$, retains exactly $m$ units of expected squared Frobenius energy:
\begin{lemma}
\label{lem:projection_dof}
Let $N \in \mathbb{R}^{d\times k}$ have i.i.d.\ zero-mean entries with variance $\tau^2$, and let $\mathcal{P}_S$ be the orthogonal projector onto an $m$-dimensional subspace $S$ of $\mathbb{R}^{d\times k}$ (with respect to the Frobenius inner product). Then $\mathbb{E}\|\mathcal{P}_S(N)\|_F^2 = m\,\tau^2$.
\end{lemma}
\begin{proof}
Pick a Frobenius-orthonormal basis $\{E_1, \ldots, E_m\}$ of $S$. Then $\mathcal{P}_S(N) = \sum_{j=1}^m \langle N, E_j\rangle E_j$. Since the coefficients $\langle N, E_j\rangle$ have variance $\tau^2$, Pythagoras' theorem directly yields $\mathbb{E}\|\mathcal{P}_S(N)\|_F^2 = \sum_{j=1}^m \mathbb{E}|\langle N, E_j\rangle|^2 = m\,\tau^2$.
\end{proof}
\paragraph{First-order expansion of $\mathcal{P}_r$.}
Under the SPN model, $\overline{\Delta W} = \Delta W^\star + \overline{\mathcal{E}}$, where $\overline{\mathcal{E}} := \tfrac{1}{n}\sum_i \mathcal{E}_i$ has independent sub-Gaussian entries of variance $\sigma^2/n$. Assuming the minimum-signal condition $\sigma_r^\star \gg \sigma\sqrt{r(d+k)^3/n}$ holds (i.e., the signal sits well above the noise floor), standard perturbation theory for low-rank approximation provides the following expansion on a high-probability event:
\begin{equation}
\mathcal{P}_r(\Delta W^\star + \overline{\mathcal{E}})
= \Delta W^\star + \mathcal{P}_T(\overline{\mathcal{E}})
+ R_n,
\label{eq:first_order}
\end{equation}
where the remainder is bounded by $\|R_n\|_F = \mathcal{O}_p\!\Bigl(\tfrac{\|\overline{\mathcal{E}}\|_{\text{op}}^2}{\sigma_r^\star}\Bigr) = \mathcal{O}_p\!\bigl(\tfrac{\sigma^2(d+k)}{n\sigma_r^\star}\bigr)$.
\paragraph{Putting it together.}
To compute the expected MSE, we bound the cross-terms and the remainder's energy. The strict sub-Gaussian assumption combined with our minimum-signal condition ensures that $\mathbb{E}\|R_n\|_F^2 = \mathcal{O}\bigl(\frac{\sigma^4(d+k)^2}{n^2(\sigma_r^\star)^2}\bigr) = o(1/n)$. Furthermore, the cross-term is bounded via the Cauchy-Schwarz inequality:
\begin{equation}
|\mathbb{E}\langle \mathcal{P}_T(\overline{\mathcal{E}}), R_n\rangle| \leq \sqrt{\mathbb{E}\|\mathcal{P}_T(\overline{\mathcal{E}})\|_F^2} \sqrt{\mathbb{E}\|R_n\|_F^2}. \nonumber
\end{equation}
Given the rates above, this cross-term is also strictly $o(1/n)$. Combining (A1), Eq.~\ref{eq:first_order}, and Lemma~\ref{lem:projection_dof} with $\tau^2 = \sigma^2/n$ and $m = r(d+k-r)$, we obtain:
\begin{align}
\text{MSE}(\text{\shortmerging})
&= \mathbb{E}\bigl\|\widehat{\Delta W}_{\text{\shortmerging}} - \Delta W^\star\bigr\|_F^2 \nonumber \\
&= \mathbb{E}\bigl\|\mathcal{P}_T(\overline{\mathcal{E}})\bigr\|_F^2 + 2\mathbb{E}\langle \mathcal{P}_T(\overline{\mathcal{E}}), R_n\rangle + \mathbb{E}\|R_n\|_F^2 \nonumber \\
&\leq \frac{r(d+k-r)\,\sigma^2}{n} + o(1/n).
\label{eq:gam_mse}
\end{align}
Comparing Eq.~\ref{eq:gam_mse} to Eq.~\ref{eq:mse_avg}, the error is reduced by a factor of $\tfrac{r(d+k-r)}{dk} = \tfrac{r}{k} + \tfrac{r}{d} - \tfrac{r^2}{dk} \approx \mathcal{O}(r/d)$. Since $r \ll \min(d, k)$, this indicates that \shortmerging substantially denoises the aggregated preference signal compared to naive averaging by effectively constraining the optimization noise to the low-rank manifold.
\paragraph{Remarks.}
The argument above relies on the isotropy of the noise to apply Lemma~\ref{lem:projection_dof}. Under anisotropic noise with covariance matrix $\Sigma_{\overline{\mathcal{E}}} \in \mathbb{R}^{dk \times dk}$, the same projection-based denoising mechanism applies, yielding $\text{MSE}(\text{\shortmerging}) \approx \operatorname{tr}\!\bigl(\mathbf{P}_T\, \Sigma_{\overline{\mathcal{E}}}\bigr)$, where $\mathbf{P}_T \in \mathbb{R}^{dk \times dk}$ is the matrix representation of $\mathcal{P}_T$ (viewing $\overline{\mathcal{E}}$ as a vector in $\mathbb{R}^{dk}$). However, the exact $\mathcal{O}(r/d)$ reduction ratio is not strictly guaranteed; the quantitative benefit depends fundamentally on how the noise covariance interacts with the tangent space $T$ versus its orthogonal complement $T^\perp$.
\section{Experimental and Training Details}
\label{app:training_details}
In this section, we outline the hyperparameters, environmental setups, and data processing configurations used to extract preference deltas and evaluate the aggregated models. All training procedures are conducted on 4 NVIDIA H20 (96GB) GPUs.
\subsection{Optimization and Hyperparameters}
To parameterize the preference deltas efficiently, we apply Low-Rank Adaptation (LoRA) across all base models. The LoRA modules are systematically attached to the attention projection matrices (\texttt{q\_proj}, \texttt{k\_proj}, \texttt{v\_proj}, \texttt{o\_proj}) and MLP layers (\texttt{gate\_proj}, \texttt{up\_proj}, \texttt{down\_proj}) across all transformer blocks.
To improve clarity and reproducibility, the detailed optimization settings and hyperparameter configurations shared across both task domains are summarized in Table~\ref{tab:hyperparameters}. Gradient checkpointing is enabled in all runs to optimize memory consumption.
\begin{table}[htbp]
\centering
\small
\caption{Hyperparameters for Preference Delta Extraction. These configurations are consistently applied across both Knowledge Reasoning and Agentic Search domains unless otherwise specified.}
\begin{tabular}{@{} l c @{}}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
LoRA Rank ($r$) & 64 \\
LoRA Alpha ($\alpha$) & 128 \\
LoRA Dropout & 0.05 \\
Target Modules & \texttt{q, k, v, o, gate, up, down\_proj} \\
Optimizer & AdamW \\
Peak Learning Rate & $1.0 \times 10^{-5}$ \\
Learning Rate Scheduler & Cosine \\
Warmup Ratio & 10\% \\
Global Batch Size & 128 \\
Training Epochs & 1 \\
Max Sequence Length & 8192 \\
Preference Objective (Reasoning) & DPO ($\beta = 0.1$) \\
Preference Objective (Agentic) & ORPO ($\beta = 0.1$) \\
\bottomrule
\end{tabular}
\label{tab:hyperparameters}
\end{table}
\subsection{Knowledge Reasoning Details (DPO Setup)}
For the knowledge reasoning domain, the base models are fine-tuned via Direct Preference Optimization (DPO). The preference datasets are sampled from the training splits of the respective mathematical and reasoning benchmarks. During DPO tuning, the prompts are formatted strictly according to the default chat templates of the respective base models (e.g., Qwen and Llama). To encourage explicit logical derivation, a standard system prompt is utilized to instruct the model to provide step-by-step reasoning (e.g., ``Let's think step by step'') before formulating the final answer. This setup maintains a consistent structural format for reasoning trajectories across both the supervisor and student models.
\subsection{Agentic Search and Retrieval Details (ORPO Setup)}
The preference data is constructed using weak-weaker model pairs and 8,000 queries randomly sampled from the MuSiQue training set. Unlike the reasoning setup, agentic search trajectories require strict structural formatting. We apply a formatting filter only to the weak model’s (chosen) responses, while retaining formatting errors in the weaker model’s (rejected) responses to preserve the negative signal. As a result, the number of valid preference pairs varies: 7,997 for Search-R1, 7,852 for AceSearcher, 7,638 for Qwen3, and 7,121 for Llama-3.2.
Training and evaluating models for agentic search involves multi-turn interactions where the model iteratively issues search queries and processes external context to answer complex questions (e.g., HotpotQA, MuSiQue). For this domain, we utilize Odds Ratio Preference Optimization (ORPO) to construct the preference deltas.
\paragraph{Agentic Data Formulation.}
The agentic preference pairs are formatted as ReAct trajectories \citep{yao2023reactsynergizingreasoningacting}, comprising interleaved \texttt{Thought}, \texttt{Action}, and \texttt{Observation} steps. Models are explicitly optimized to output targeted action commands (e.g., \texttt{<search>query</search>}) to trigger the external retrieval tool during complex reasoning paths. The sequence length is set to 8192 tokens to accommodate the extensive context windows required by multi-hop retrieval tasks.
\paragraph{Retrieval Environment.}
To ensure reproducible and standardized evaluations, the external search environment is grounded in a fixed Wikipedia corpus (using the \texttt{enwiki-20200601} English Wikipedia dump). When a model generates a \texttt{Search} action, the environment queries the corpus using a dense retrieval model (\texttt{e5-large-v2}), which returns the top $k = 5$ most relevant passages based on embedding similarity. To manage context density and mitigate noise, each retrieved snippet is truncated to a maximum of 500 tokens before being appended to the model's context as an \texttt{Observation}. The model subsequently processes this new information to generate the next \texttt{Thought} or conclude with a final \texttt{<answer>answer</answer>}. This retrieval protocol is maintained uniformly across both the preference data generation and evaluation phases.
\section{Prompt Templates}
\label{app:prompts}
In this section, we provide the exact prompt templates utilized during the preference data extraction and evaluation phases. To standardize the generation distribution and ensure strict adherence to task-specific formats, we employ a one-shot prompting strategy across both domains. The one-shot examples explicitly demonstrate the expected reasoning traces and tool-use trajectories.
\subsection{Knowledge Reasoning Prompts}
For mathematical and logical reasoning tasks, the one-shot prompt instructs the model to articulate its derivation process before formulating the final answer. This structural constraint facilitates the generation of detailed step-by-step preference trajectories for DPO tuning.
\vspace{0.5em}
\noindent\textbf{System Prompt:}
\begin{quote}
\small\ttfamily
You are an expert mathematical and logical reasoning assistant. Please think step by step to solve the problem and carefully explain your derivation before providing the final answer.
\end{quote}
\noindent\textbf{User Input Template (One-shot):}
\begin{quote}
\small\ttfamily
Here is an example of how to answer a question:
Question: A train travels at a constant speed of 60 miles per hour. How far will it travel in 2.5 hours?
Answer: Let's think step by step. We know the formula for distance is Distance = Speed * Time. The given speed is 60 miles per hour, and the given time is 2.5 hours. Multiplying these values: 60 * 2.5 = 150. Therefore, the train will travel 150 miles. The final answer is 150.
Now, answer the following question:
Question: \{question\}
Answer:
\end{quote}
\subsection{Agentic Search Prompts}
For the agentic search domain, models are prompted to interact with the external dense retrieval environment (\texttt{e5-large-v2}) following the ReAct framework \citep{yao2023reactsynergizingreasoningacting}. The one-shot example explicitly dictates the strict interleaving format of thoughts, search actions, and observations.
\vspace{0.5em}
\noindent\textbf{System Prompt (ReAct Instructions):}
\begin{quote}
\small\ttfamily
Answer the following question from the user with the help of a Wikipedia search engine. Please reason step by step. You should think about what you need to know in order to answer the question, and then search for that information using the search engine. To perform a search operation, write a web search question and enclose it with <search> and </search>. You will immediately observe a piece of search results within the <information> and </information> tags. You can then use this retrieved information to continue your reasoning. You can repeat the search process many times. Once you think you have all the information you need, you can end the thinking process and provide the final answer. You MUST enclose your final answer with <answer> and </answer>.
\end{quote}
\noindent\textbf{User Input Template (One-shot):}
\begin{quote}
\small\ttfamily
Here is an example of the interaction format:
<search> Who were the people that captured Malakoff? </search>
<information> The French army under General MacMahon successfully captured the Malakoff redoubt on 8th. </information>
Okay, so the French people captured Malakoff. Now, the next step would be to figure out in what region Pilipsburg is located. I will write a web search to look that up.
<search> Where is Philipsburg located at? </search>
<information> Philipsburg is is the main town and capital of Sint Maarten, a constituent country of the Kingdom of the Netherlands. </information>
...[more thoughts shortened]...
Your final response:
<answer> November 12, 1625 </answer>
\end{quote}
\section{Case Studies: Knowledge Reasoning}
\label{app:reasoning-cases}
We present four GPQA reasoning traces in which the model trained on the strongest individual preference pair (\textbf{Single Preference Delta}) fails, whereas our proposed \textbf{\shortours-\shortmerging} successfully derives the correct answer.
The failures of the single-delta model typically fall into two categories: \emph{generation non-termination}, where the model enters a degenerate repetitive loop (Table~\ref{tab:rcase-loop}); and \emph{logical propagation errors}, where it produces an incorrect final answer due to intermediate arithmetic or domain knowledge mistakes (Tables~\ref{tab:rcase-arith}--\ref{tab:rcase-chem}).
Across a random sample of 60 GPQA error cases from the single-delta baseline, 35 cases exhibit format/termination failures and 25 exhibit logical errors; \shortours-\shortmerging effectively recovers all 60 cases.
Key reasoning steps are highlighted in \colorbox{gray!12}{\strut gray}, with critical errors in \colorbox{red!12}{\strut red} and the recovered correct derivations in \colorbox{green!12}{\strut green}.
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Termination Recovery:} The Single Preference Delta model derives the correct epistasis hierarchy but enters a degenerate repetition loop—repeating the same hesitation 59 times—and fails to emit a valid \texttt{ANSWER:} tag. \shortours-\shortmerging reaches the same conclusion and terminates cleanly.}
\label{tab:rcase-loop}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} You perform a high-throughput experiment on white lupine to find genes contributing to resistance to anthracnose. You receive candidate genes G1, G2, G3 and create knockouts... Which statement correctly describes the epistatic relationships? \\
\textbf{Gold:} A \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} None / Format Failure) \\[3pt]
\colorbox{gray!12}{Identifies hierarchy: G2 is epistatic to G1; G2 is epistatic to G3; G1 is epistatic to G3.} \\[2pt]
\colorbox{gray!12}{``So, gene 2 is the most upstream, then gene 1, then gene 3.''} \quad (\textit{Correct conclusion}) \\[2pt]
\colorbox{red!12}{``Wait, perhaps the correct answer is option A... But the data doesn't support that.''} \\[2pt]
$\hookrightarrow$ \textit{Repeated 59$\times$ until token limit; no answer extracted.} \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} A) \\[3pt]
\colorbox{gray!12}{Derives same hierarchy: G2 $\rightarrow$ G1 $\rightarrow$ G3.} \\[2pt]
\colorbox{green!12}{Identifies G2 as the transcription factor and G1/G3 as redundant. Terminates with \texttt{ANSWER: A}.} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Arithmetic Recovery:} The Single Preference Delta model formulates the correct decay equation but makes a magnitude error during division, obtaining $10^{-25}$ instead of $10^{-24}$. \shortours-\shortmerging executes the arithmetic correctly.}
\label{tab:rcase-arith}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} X is a meson resonance. What is the mean decay distance? Given: $E_X = 8$\,GeV, $m_X = 1.2$\,GeV, $\Gamma_X = 320$\,MeV. \\
\textbf{Gold:} A \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} C) \\[3pt]
\colorbox{gray!12}{Step 1: $\gamma = E_X / m_X \approx 6.667$} \\[2pt]
\colorbox{red!12}{Step 2: $\tau = \hbar / \Gamma_X = 6.582 \times 10^{-25} / 0.32 \approx 2.057 \times 10^{-25}$\,s} \quad (\textit{Should be $10^{-24}$}) \\[2pt]
\colorbox{gray!12}{Step 3: $L = \gamma c \tau \approx 4.11 \times 10^{-16}$\,m} $\rightarrow$ \textit{Selects C, off by exactly $10\times$.} \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} A) \\[3pt]
\colorbox{green!12}{$\tau = \hbar / \Gamma_X = 6.582 \times 10^{-25} / 0.32 = 2.057 \times 10^{-24}$\,s} \quad (\textit{Correct exponent}) \\[2pt]
\colorbox{gray!12}{$d = (p / m_X) \cdot c \cdot \tau \approx 4.07 \times 10^{-15}$\,m} $\rightarrow$ \textit{Selects A.} \\
\bottomrule
\end{tabular}
\end{table}
\clearpage %
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Algebraic Recovery:} The Single Preference Delta model drops a logarithmic factor when propagating stellar abundance ratios through solar reference values. \shortours-\shortmerging successfully tracks the transformation.}
\label{tab:rcase-astro}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} Two stars with $[\text{Si/Fe}]_1 = 0.3$, $[\text{Mg/Si}]_2 = 0.3$... Calculate the ratio $n_{\text{Si},1} / n_{\text{Si},2}$. \\
\textbf{Gold:} C \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} D) \\[3pt]
\colorbox{gray!12}{Star\_1: $[\text{Si/Fe}]_1 = 0.3 \Rightarrow n_{\text{Si},1} = 10^{0.3} \cdot n_{\text{Si},\odot}$} \\[2pt]
\colorbox{red!12}{Star\_2: Conflates intermediate terms, dropping a factor from $\log_{10}(n_{\text{Si},\odot}/n_\text{H})$.} \\[2pt]
$\Rightarrow$ Yields ratio $\approx 4.0$ (Selects D). \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} C) \\[3pt]
\colorbox{green!12}{Correctly tracks each bracket transformation through solar reference values.} \\[2pt]
$\Rightarrow$ Obtains ratio $\approx 12.6$ (Selects C). \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Domain Knowledge Recovery:} The Single Preference Delta model misidentifies the extent of reduction by red P + HI, treating it as a partial transformation. \shortours-\shortmerging applies the correct exhaustive reduction mechanism.}
\label{tab:rcase-chem}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} What is the index of hydrogen deficiency (IHD) of the product obtained when 2-formyl-5-vinylcyclohex-3-enecarboxylic acid is reacted with red phosphorus and excess HI? \\
\textbf{Gold:} B \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} D) \\[3pt]
\colorbox{gray!12}{Identifies starting IHD $= 5$ (1 ring + 2 C=C + 2 C=O).} \\[2pt]
\colorbox{red!12}{``Red P + HI converts -COOH to -COI. Double bonds remain unchanged.''} \\[2pt]
$\Rightarrow$ Selects D (IHD $= 3$). \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} B) \\[3pt]
\colorbox{green!12}{``Red P + HI is an exhaustive reduction: reduces all C=C and C=O groups.''} \\[2pt]
\colorbox{gray!12}{Product: fully saturated cyclohexane ring with alkyl substituents. IHD $= 1$.} \\
\bottomrule
\end{tabular}
\end{table}
\clearpage
\section{Case Studies: Agentic Search}
\label{app:search-cases}
To illustrate the behavioral complementation achieved by multiple preference deltas, we present four representative agentic search trajectories. Each case highlights a distinct failure mode overcome by our framework. Search queries are in \colorbox{green!15}{\strut green} and retrieved info in \colorbox{blue!10}{\strut blue}.
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Entity Disambiguation:} The Single Preference Delta model misidentifies the lead actor as a child actor and issues repetitive queries before generating an unsupported entity. \shortours-\shortmerging formulates a holistic query to resolve the entity accurately.}
\label{tab:case-entity}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} Who did the leading actor in \textit{Sammy Going South} play in \textit{The Ten Commandments}? \\
\textbf{Gold:} Dathan \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} Mered, 6 turns) \\[3pt]
\colorbox{green!15}{\texttt{<search>} Who is the leading actor in Sammy Going South? \texttt{</search>}} \\
\colorbox{blue!10}{\texttt{<information>} Fergus McClelland is the lead actor... \texttt{</information>}} \\[2pt]
\colorbox{red!12}{Repeatedly queries Fergus McClelland's role in The Ten Commandments (which does not exist).} \\
\textcolor{red}{\texttt{<answer>} Mered \texttt{</answer>}} \quad (\textit{Unsupported entity generation}) \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} Dathan, 2 turns) \\[3pt]
\colorbox{green!15}{\texttt{<search>} Who did the leading actor in Sammy Going South play in the ten commandments? \texttt{</search>}} \\
\colorbox{blue!10}{\texttt{<information>} Edward G. Robinson starred as Dathan in The Ten Commandments... \texttt{</information>}} \\[2pt]
\textcolor{teal}{\texttt{<answer>} Dathan \texttt{</answer>}} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Anchoring Bias Recovery:} The Single Preference Delta model exhibits anchoring bias, fixating on a 2004 film adaptation. \shortours-\shortmerging redirects its search to the 2017 series adaptation to resolve the query.}
\label{tab:case-recovery}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} The actor who plays Count Olaf in ``A Series of Unfortunate Events'' also plays which character in \textit{Batman Under the Red Hood}? \\
\textbf{Gold:} Nightwing / Dick Grayson \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} The Joker, 6 turns) \\[3pt]
\colorbox{blue!10}{\texttt{<information>} Jim Carrey played Count Olaf in the 2004 film... \texttt{</information>}} \\[2pt]
\colorbox{red!12}{Model repeatedly searches for Jim Carrey's role in Batman, failing to consider other adaptations.} \\
\textcolor{red}{\texttt{<answer>} The Joker \texttt{</answer>}} \quad (\textit{Hallucination due to anchoring}) \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} Dick Grayson, 4 turns) \\[3pt]
\colorbox{green!15}{\texttt{<search>} Which character does Neil Patrick Harris play in Batman Under the Red Hood? \texttt{</search>}} \\
\colorbox{blue!10}{\texttt{<information>} Neil Patrick Harris voiced Dick Grayson/Nightwing... \texttt{</information>}} \\[2pt]
\textcolor{teal}{\texttt{<answer>} Dick Grayson/Nightwing \texttt{</answer>}} \\
\bottomrule
\end{tabular}
\end{table}
\clearpage
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Multi-hop Chain Completion:} The Single Preference Delta model prematurely terminates the multi-hop chain at Indonesia. \shortours-\shortmerging correctly parses the full relational chain targeting Timor-Leste.}
\label{tab:case-multihop}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} Who is the president of the newly declared independent country of the country of the birthplace of Mulham Arufin? \\
\textbf{Gold:} Francisco Guterres \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} Joko Widodo, 3 turns) \\[3pt]
\colorbox{blue!10}{\texttt{<information>} Mulham Arufin was born in Indonesia... \texttt{</information>}} \\[2pt]
\colorbox{red!12}{Queries for the president of Indonesia, failing to resolve the ``newly declared independent'' clause.} \\
\textcolor{red}{\texttt{<answer>} Joko Widodo \texttt{</answer>}} \quad (\textit{Premature termination}) \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} Francisco Guterres, 5 turns) \\[3pt]
\colorbox{green!15}{\texttt{<search>} Who is the president of the newly declared independent country of Timor-Leste? \texttt{</search>}} \\
\colorbox{blue!10}{\texttt{<information>} Francisco Guterres is the president of Timor-Leste... \texttt{</information>}} \\[2pt]
\textcolor{teal}{\texttt{<answer>} Francisco Guterres \texttt{</answer>}} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htbp]
\centering
\renewcommand{\arraystretch}{1.2}
\caption{\textbf{Resolution Matching:} The Single Preference Delta model answers at an incorrect granularity (State instead of County). \shortours-\shortmerging identifies that the retrieved city requires an additional bridging query to map to the target county.}
\label{tab:case-granularity}
\vspace{0.3em}
\footnotesize
\begin{tabular}{p{0.95\textwidth}}
\toprule
\textbf{Question:} In which county was Larkin I. Smith born? \\
\textbf{Gold:} Pearl River County \\
\midrule
\textbf{Single Preference Delta} \quad (\ding{55} \textit{Output:} Mississippi, 3 turns) \\[3pt]
\colorbox{blue!10}{\texttt{<information>} Larkin I. Smith was a U.S. Representative from Mississippi... \texttt{</information>}} \\[2pt]
\colorbox{red!12}{Fails to resolve the specific county from general state-level information.} \\
\textcolor{red}{\texttt{<answer>} Mississippi \texttt{</answer>}} \quad (\textit{Resolution mismatch}) \\
\midrule
\textbf{\shortours-\shortmerging} \quad (\ding{51} \textit{Output:} Pearl River County, 3 turns) \\[3pt]
\colorbox{blue!10}{\texttt{<information>} Smith was born in Poplarville, Mississippi... \texttt{</information>}} \\[2pt]
\colorbox{green!15}{\texttt{<search>} In which county is Poplarville, Mississippi? \texttt{</search>}} \quad (\textit{Bridging query}) \\
\colorbox{blue!10}{\texttt{<information>} Poplarville is located in Pearl River County... \texttt{</information>}} \\[2pt]
\textcolor{teal}{\texttt{<answer>} Pearl River County \texttt{</answer>}} \\
\bottomrule
\end{tabular}
\end{table}
\section{Sensitivity Analysis of Reference Choice in \shortmerging}
\label{app:reference_sensitivity}
In Section 4, we introduce \ourmerging (\shortmerging), which utilizes orthogonal Procrustes alignment to project multiple LoRA adapters onto a shared reference coordinate system ($\phi_1$). While the Procrustes formulation is defined relative to a chosen base, we hypothesized that the specific choice of the reference adapter does not significantly impact the final aggregated performance. To empirically verify this, we conduct a sensitivity analysis for both the Knowledge Reasoning and Agentic Search domains by systematically rotating the reference role among all available domain-specific adapters.
\paragraph{Knowledge Reasoning.}
For the reasoning domain, we aggregate three distinct preference deltas. We evaluate the aggregated performance of the student model (e.g., Qwen3-8B) by iteratively setting each of the three adapters ($\Delta_{\text{DS-R1}}$, $\Delta_{\text{Qwen3}}$, and $\Delta_{\text{Llama}}$) as the reference base $\phi_1$. As shown in Table~\ref{tab:sensitivity_kr}, the maximum fluctuation in the average reasoning score is minimal (within 0.1 points).
\begin{table}[htbp]
\centering
\small
\caption{Sensitivity to Reference Choice on Knowledge Reasoning. Performance of the aggregated model when different preference deltas are selected as the reference base ($\phi_1$).}
\begin{tabular}{@{} l c @{}}
\toprule
\textbf{Reference Choice ($\phi_1$)} & \textbf{Knowledge Reasoning Avg.} \\
\midrule
$\Delta_{\text{DS-R1}}$ (DeepSeek-R1 preference delta) & 79.3 \\
$\Delta_{\text{Qwen3}}$ (Qwen3 preference delta) & 79.2 \\
$\Delta_{\text{Llama}}$ (Llama-3.2 preference delta) & 79.3 \\
\midrule
\textbf{Max Fluctuation} & \textbf{$\le$ 0.1} \\
\bottomrule
\end{tabular}
\label{tab:sensitivity_kr}
\end{table}
\paragraph{Agentic Search.}
Similarly, for the agentic search domain, we aggregate four distinct preference deltas. We repeat the sensitivity analysis by rotating the reference role among $\Delta_{\text{Search-R1}}$, $\Delta_{\text{AceSearcher}}$, $\Delta_{\text{Qwen3}}$, and $\Delta_{\text{Llama}}$. Table~\ref{tab:sensitivity_as} demonstrates that the final average performance remains robust regardless of the geometric anchor chosen.
\begin{table}[htbp]
\centering
\small
\caption{Sensitivity to Reference Choice on Agentic Search. Performance of the aggregated model when rotating the reference base among the four agentic preference deltas.}
\begin{tabular}{@{} l c @{}}
\toprule
\textbf{Reference Choice ($\phi_1$)} & \textbf{Agentic Search Avg. Score} \\
\midrule
$\Delta_{\text{Search-R1}}$ (Search-R1 preference delta) & 58.0 \\
$\Delta_{\text{AceSearcher}}$ (AceSearcher preference delta) & 58.0 \\
$\Delta_{\text{Qwen3}}$ (Qwen3 preference delta) & 57.9 \\
$\Delta_{\text{Llama}}$ (Llama-3.2 preference delta) & 58.0 \\
\midrule
\textbf{Max Fluctuation} & \textbf{$\le$ 0.1} \\
\bottomrule
\end{tabular}
\label{tab:sensitivity_as}
\end{table}
Across both diverse task domains and varying numbers of adapters, the geometric consistency of GAM maintains the intrinsic relative orientations between the preference deltas. This indicates that GAM is structurally robust to the initial selection of the reference coordinate system.
\section{Degradation from Standard Supervised Fine-Tuning}
\label{app:sft}
\begin{table*}[h]
\centering
\setlength{\tabcolsep}{3.5pt}
\caption{Performance comparison between Standard SFT and preference tuning using identical source data. SFT on the chosen responses forces the strong student model to mimic a suboptimal policy, degrading its capabilities. Conversely, preference tuning effectively extracts the relative capability delta, improving overall performance.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{} l ccc cccc c ccc cccc c @{}}
\toprule
\multirow{3}{*}{\textbf{Method}} &
\multicolumn{8}{c}{\textbf{Qwen3-8B}} &
\multicolumn{8}{c}{\textbf{Tülu3-8B}} \\
\cmidrule(lr){2-9} \cmidrule(l){10-17}
& \multicolumn{3}{c}{\textbf{Single-hop QA}} & \multicolumn{4}{c}{\textbf{Multi-hop QA}} & \multirow{2}{*}{\textbf{Avg.}}
& \multicolumn{3}{c}{\textbf{Single-hop QA}} & \multicolumn{4}{c}{\textbf{Multi-hop QA}} & \multirow{2}{*}{\textbf{Avg.}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-8} \cmidrule(lr){10-12} \cmidrule(lr){13-16}
& \textbf{NQ} & \textbf{TQA} & \textbf{Pop.}
& \textbf{Hot.} & \textbf{2Wiki} & \textbf{Bam.} & \textbf{MuSi.} &
& \textbf{NQ} & \textbf{TQA} & \textbf{Pop.}
& \textbf{Hot.} & \textbf{2Wiki} & \textbf{Bam.} & \textbf{MuSi.} & \\
\midrule
Student Baseline
& 46.2 & 61.4 & 44.6 & 53.8 & 58.1 & 61.6 & 28.9 & 50.7
& 45.8 & 60.5 & 43.0 & 52.4 & 55.1 & 58.8 & 25.6 & 48.7 \\
\midrule
\multicolumn{17}{@{}l}{\textit{Training on identical Qwen3 (4B over 1.7B) source data}} \\
\quad Standard SFT (Chosen only)
& 44.6 & 60.8 & 43.1 & 51.9 & 54.8 & 58.8 & 25.8 & 48.5
& 41.7 & 56.2 & 37.4 & 48.6 & 51.2 & 51.6 & 19.7 & 43.8 \\
\quad Single Preference Delta
& \textbf{48.5} & \textbf{62.8} & \textbf{47.0} & \textbf{54.2} & \textbf{61.5} & \textbf{63.5} & \textbf{32.8} & \textbf{52.9}
& \textbf{46.8} & \textbf{62.0} & \textbf{46.8} & \textbf{53.5} & \textbf{59.2} & \textbf{62.1} & \textbf{28.5} & \textbf{51.3} \\
\bottomrule
\end{tabular}%
}
\label{tab:appendix_sft}
\end{table*}
In our main experiments, we claim that the performance gains achieved by our framework reflect genuine capability elicitation from preference deltas, rather than simply benefiting from exposure to the task format or identical data. To empirically validate this, we conduct a control experiment using standard Supervised Fine-Tuning (SFT) under the same parameter-efficient setup.
Specifically, we extract the ``chosen'' responses from the Qwen3 (4B over 1.7B) preference pair dataset. Instead of applying preference optimization, we use standard SFT to train the strong student models (Qwen3-8B and Tülu3-8B) to directly mimic these chosen responses via behavioral cloning.
\paragraph{Implementation Details for SFT Baseline.}
To ensure a rigorous and fair comparison, we maintain identical hyperparameter settings for both the SFT baseline and our preference tuning experiments. Specifically, we use the LoRA \citep{hu2021loralowrankadaptationlarge} framework with a rank of $r=64$ and $\alpha=128$, targeting all linear layers. The models are trained using the AdamW optimizer with a peak learning rate of $1.0 \times 10^{-5}$ and a cosine learning rate scheduler. We use a total batch size of 128 and train for 1 epoch, which we found to be sufficient for the model to converge on the formatting of the source data without excessive overfitting. All experiments are conducted on 4$\times$NVIDIA H20 (96GB) GPUs using the LlamaFactory \citep{zheng2024llamafactoryunifiedefficientfinetuning} framework.
The results are presented in \Cref{tab:appendix_sft}. As expected, SFT leads to a noticeable degradation in performance across almost all tasks. For Qwen3-8B, the average F1 score drops from the baseline of 50.7 to 48.5. The degradation is even more pronounced for Tülu3-8B, which drops from 48.7 to 43.8.
This phenomenon occurs because the absolute response quality of the weaker model (4B) is inherently lower than the intrinsic capabilities of the strong student models (8B). SFT encourages the strong students to mimic the suboptimal reasoning trajectories and stylistic idiosyncrasies of the weaker model, leading to capability regression. In contrast, preference optimization on the identical data focuses on the \textit{relative directional delta}. This allows the strong student model to extract the underlying capability improvements without being bottlenecked by the weak model's absolute generation quality, successfully raising the average F1 to 52.9 and 51.3 for Qwen3-8B and Tülu3-8B, respectively.
\section*{NeurIPS Paper Checklist}
\begin{enumerate}
\item {\bf Claims}
\item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\item[] Answer: \answerYes{} %
\item[] Justification: We clearly state our contributions, including the \ours framework and the \ourmerging method, in the Abstract and \Cref{sec:introduction}, and support them with comprehensive empirical results in \Cref{sec:experiments}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the abstract and introduction do not include the claims made in the paper.
\item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A \answerNo{} or \answerNA{} answer to this question will not be perceived well by the reviewers.
\item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
\item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
\item {\bf Limitations}
\item[] Question: Does the paper discuss the limitations of the work performed by the authors?
\item[] Answer: \answerYes{} %
\item[] Justification: We discuss the limitations in \Cref{sec:limitation},including (i) the reliance on directionally diverse preference signals — aggregating behaviorally homogeneous pairs yields only marginal improvements; (ii) the limited set of weak-weaker model pairs evaluated, leaving open the scaling behavior with larger collections; and (iii) the existence of aggregation strategies beyond the training-based and LoRA-based methods explored here.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper has no limitation while the answer \answerNo{} means that the paper has limitations, but those are not discussed in the paper.
\item The authors are encouraged to create a separate ``Limitations'' section in their paper.
\item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
\item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
\item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
\item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
\item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
\item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
\item {\bf Theory assumptions and proofs}
\item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
\item[] Answer: \answerNA{} %
\item[] Justification: Our work proposes a novel algorithmic framework and geometric merging formulation based on established linear algebra (SVD, orthogonal Procrustes) rather than introducing formal theoretical bounds or mathematical proofs.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include theoretical results.
\item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
\item All assumptions should be clearly stated or referenced in the statement of any theorems.
\item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
\item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
\item Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
\item {\bf Experimental result reproducibility}
\item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
\item[] Answer: \answerYes{} %
\item[] Justification: We provide detailed descriptions of our experimental setup, prompt templates, baseline implementations, and hyperparameter choices in \Cref{sec:experiments} and Appendix \ref{app:training_details}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item If the paper includes experiments, a \answerNo{} answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
\item If the contribution is a dataset and\slash or model, the authors should describe the steps taken to make their results reproducible or verifiable.
\item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
\item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
\item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
\item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
\item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
\item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
\item {\bf Open access to data and code}
\item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
\item[] Answer: \answerNo{} %
\item[] Justification: To preserve anonymity during review, we do not release the code and data at submission time. We provide detailed implementation and experimental settings in the supplementary material, and will publicly release code, scripts, and processed data after the reviewing period.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that paper does not include experiments requiring code.
\item Please see the NeurIPS code and data submission guidelines (\url{https://neurips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item While we encourage the release of code and data, we understand that this might not be possible, so \answerNo{} is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
\item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://neurips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
\item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
\item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
\item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
\item {\bf Experimental setting/details}
\item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
\item[] Answer: \answerYes{} %
\item[] Justification: All necessary training details, including learning rates, batch sizes, LoRA configurations, and optimizer setups, are comprehensively documented in Appendix \ref{app:training_details}, Appendix \ref{app:prompts} and \Cref{sec:experiments}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
\item The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
\item {\bf Experiment statistical significance}
\item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
\item[] Answer: \answerNo{} %
\item[] Justification: We do not report error bars due to the excessively high computational cost of running multiple full fine-tuning and evaluation cycles for large language models (e.g., 8B parameters) across numerous diverse benchmarks. This is standard practice in LLM research.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item The authors should answer \answerYes{} if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
\item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
\item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
\item The assumptions made should be given (e.g., Normally distributed errors).
\item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
\item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
\item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
\item If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
\item {\bf Experiments compute resources}
\item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
\item[] Answer: \answerYes{} %
\item[] Justification: We detail the hardware infrastructure, specifically the type and number of GPUs used, as well as the approximate training and evaluation times in Appendix \ref{app:training_details}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
\item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
\item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
\item {\bf Code of ethics}
\item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
\item[] Answer: \answerYes{} %
\item[] Justification: Our research complies strictly with the NeurIPS Code of Ethics.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the authors have not reviewed the NeurIPS Code of Ethics.
\item If the authors answer \answerNo, they should explain the special circumstances that require a deviation from the Code of Ethics.
\item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
\item {\bf Broader impacts}
\item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
\item[] Answer: \answerNA{} %
\item[] Justification: Our work presents a general training and model-merging method for improving LLM reasoning and search performance, without introducing a specific deployment domain or application involving direct societal impact. It does not involve sensitive personal data, safety-critical decision making, or targeted high-risk use cases. Therefore, we consider broader societal impacts to be limited and not uniquely attributable to this work beyond those generally associated with large language models.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that there is no societal impact of the work performed.
\item If the authors answer \answerNA{} or \answerNo, they should explain why their work has no societal impact or why the paper does not address societal impact.
\item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
\item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
\item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
\item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
\item {\bf Safeguards}
\item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
\item[] Answer: \answerNA{} %
\item[] Justification: Our work focuses on a parameter merging methodology and utilizes existing open-source datasets and models. It does not introduce or release new high-risk foundational models or scraped datasets.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper poses no such risks.
\item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
\item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
\item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
\item {\bf Licenses for existing assets}
\item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
\item[] Answer: \answerYes{} %
\item[] Justification: All pre-trained models (e.g., Qwen, Llama) and datasets used in this work are publicly available. We properly cite their original papers and strictly adhere to their respective usage licenses.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not use existing assets.
\item The authors should cite the original paper that produced the code package or dataset.
\item The authors should state which version of the asset is used and, if possible, include a URL.
\item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
\item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
\item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
\item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
\item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
\item {\bf New assets}
\item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
\item[] Answer: \answerYes{} %
\item[] Justification: The primary new assets introduced are our codebase and merging scripts, which are thoroughly documented and provided once upon acceptance.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not release new assets.
\item Researchers should communicate the details of the dataset\slash code\slash model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
\item The paper should discuss whether and how consent was obtained from people whose asset is used.
\item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
\item {\bf Crowdsourcing and research with human subjects}
\item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
\item[] Answer: \answerNA{} %
\item[] Justification: Our research relies purely on automated model outputs and existing datasets; it does not involve any crowdsourcing or human subjects.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not involve crowdsourcing nor research with human subjects.
\item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
\item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
\item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
\item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
\item[] Answer: \answerNA{} %
\item[] Justification: Not applicable as this research does not involve human subjects.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not involve crowdsourcing nor research with human subjects.
\item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
\item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
\item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
\item {\bf Declaration of LLM usage}
\item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does \emph{not} impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
\item[] Answer: \answerYes{} %
\item[] Justification: The generation of weak-to-strong preference trajectories using existing LLMs (e.g., Qwen3-4B over 1.7B) is explicitly detailed in Section 3, as it forms the basis of our Preference Delta Aggregation framework.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
\item Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
\end{itemize}
\end{enumerate}