Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
Extracted source text · arXiv 2512.11470
\documentclass[11pt, letterpaper]{article}
\PassOptionsToPackage{svgnames,table,xcdraw}{xcolor}
\usepackage{times, latexsym}
\usepackage{url}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amsfonts} %
\usepackage{amssymb} %
\usepackage{amsthm} %
\usepackage{bm} %
\usepackage{natbib}
\usepackage{tcolorbox}
\definecolor{salmon}{RGB}{250, 128, 114}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{subcaption}
\usepackage{float}
\usepackage{pifont}
\usepackage{hyperref}
\hypersetup{
colorlinks=true,
linkcolor=blue,
citecolor=blue,
urlcolor=blue,
bookmarksnumbered=true,
pdfborder={0 0 0}
}
\usepackage{graphicx} %
\setlength{\parskip}{3pt plus 2pt minus 2pt}
\usepackage{configuration/iclr2026_conference}
\iclrfinalcopy %
\theoremstyle{definition}
\newtheorem{definition}{Definition}
\title{Rethinking Expert Trajectory Utilization in LLM Post-training}
\input{content/authors}
\date{}
\begin{document}
\maketitle
\input{content/0_abs}
\input{content/1_intro}
\input{content/2_related_works}
\input{content/3_preliminary}
\input{content/_analytical_framework}
\input{content/4_exp_settings}
\input{content/5_exp_results}
\input{content/6_conclude}
\clearpage
\bibliography{references}
\bibliographystyle{configuration/iclr2026_conference}
\clearpage
\appendix
\input{content/appx}
\end{document}
\author{
Bowen Ding\textsuperscript{\rm 1,2}~,
Yuhan Chen\textsuperscript{\rm 2~\thanks{\quad Work done at Westlake University as an intern.}}~,
Jiayang Lv\textsuperscript{\rm 2~\footnotemark[1]}~,
Jiyao Yuan\textsuperscript{\rm 4}~,
Qi Zhu\textsuperscript{\rm 4},
Shuangshuang Tian\textsuperscript{\rm 2~\footnotemark[1]},
\\
\bf Dantong Zhu\textsuperscript{\rm 2~\footnotemark[1]}~,
Futing Wang\textsuperscript{\rm 1,2}~,
Heyuan Deng\textsuperscript{\rm 4}~,
Fei Mi\textsuperscript{\rm 4}\footnotemark[2]~~,
Lifeng Shang\textsuperscript{\rm 4}~,
Tao Lin\textsuperscript{\rm 2,3~\thanks{\quad Corresponding authors.}},
\\ [1mm]
\textsuperscript{1} Zhejiang University~
\textsuperscript{2} School of Engineering, Westlake University \\
\textsuperscript{3} Institute of Advanced Technology, Westlake Institute for Advanced Study \\
\textsuperscript{4} Huawei Noah’s Ark Lab \\
$^{2}$\texttt{\{dingbowen, wangfuting, lintao\}@westlake.edu.cn} \\
$^{4}$\texttt{\{yuanjiyao1, zhuqi41, dengheyuan, mifei2, Shang.Lifeng\}@huawei.com} \\
}
\begin{abstract}
While effective post-training integrates Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), the optimal mechanism for utilizing expert trajectories remains
unresolved. We propose the \textbf{Plasticity-Ceiling Framework} to theoretically ground this landscape, decomposing performance into foundational SFT performance and
the subsequent RL plasticity. Through extensive benchmarking, we establish the \textbf{Sequential SFT-then-RL} pipeline as the superior standard, overcoming the
stability deficits of synchronized approaches. Furthermore, we derive precise scaling guidelines: (1) Transitioning to RL at the \textbf{SFT Stable or Mild Overfitting Sub-phase}
maximizes the final ceiling by securing foundational SFT performance without compromising RL plasticity; (2) Refuting ``Less is More'' in the context of {SFT-then-RL} scaling,
we demonstrate that \textbf{Data Scale} determines the primary post-training potential, while \textbf{Trajectory Difficulty} acts as a performance multiplier; and (3) Identifying that
the Minimum SFT Validation Loss serves as a robust indicator for selecting the expert trajectories that maximize the final performance ceiling.
Our findings provide actionable guidelines for maximizing the value extracted from expert trajectories. Code: ~\url{https://github.com/LINs-lab/RETU}.
\end{abstract}
\vspace{-1em}
\section{Introduction}
\vspace{-0.5em}
The transformation of pre-trained Large Language Models (LLMs) into powerful Large Reasoning Models (LRMs) hinges on effective post-training, which typically interleaves Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL)~\citep{deepseek-r1, glm45}.
SFT leverages expert trajectories (prompt-solution pairs) to instill reasoning priors via imitation, while RL methods such as GRPO~\citep{deepseek-math} allow models to leverage prompt-answer pairs to self-explore reasoning paths through reward incentives, without the expert trajectory utilization.
Despite the consensus on the necessity of both, a critical question remains unresolved:
\textit{What is the optimal mechanism to utilize expert trajectories (i.e., SFT data) to maximize the post-training performance ceiling?}
\vspace{-0.3em}
The methodology for effective expert trajectory utilization currently faces an unresolved paradigm dilemma. Recent works propose {Synchronized SFT-RL} (Syn-SFT-RL) algorithms, such as UPT \citep{lv2025unifiedviewlargelanguage}, SRFT \citep{Fu2025SRFT}, and LUFFY \citep{yan2025learningreasonoffpolicyguidance}, which integrate imitation loss directly into the RL optimization loop. While these methods often claim superior efficiency over sequential approaches, this advantage is critically constrained by their reliance on limited SFT data (only about 46K). This raises a fundamental question: whether {Syn-SFT-RL} can maintain its claimed superiority and robustness when provided with the substantially large-scale data necessary for achieving state-of-the-art ceilings.
Conversely, some LLM practitioners~\citep{qwen3, glm45, deepseek-r1} typically favor the straightforward sequential {SFT-then-RL} pipeline. However, the principles governing this successful approach remain largely empirical and lack systematic definition in two critical areas. First, concerning the \textbf{Optimal Timing} for switching from SFT to RL, the criteria lack a systematic definition. Second, regarding \textbf{Data Properties}, although the ``Less is More''~\citep{limo,s1} approach achieves comparable SFT accuracy with minimal data, it is unclear whether this compromises the subsequent RL scaling potential or leads to premature convergence. Similarly, while harder data push SFT boundaries~\citep{tong2024dartmath, d3}, its precise influence on the overall post-training ceiling remains unclarified. Consequently, these tensions highlight the urgent need for a unified framework to understand how the characteristics of SFT data dictate the entire post-training performance.
\input{fig_outlines/intro}
To rigorously address these systemic gaps, we propose a {Plasticity-Ceiling} analytical framework in \S~\ref{sec:framework}. This framework provides a unified view of all paradigms and enables the quantitative decomposition of the theoretical performance ceiling ($A_{\text{post}}$) into two measurable components: the\textbf{ SFT Performance} ($P_{\text{sft}}$) achieved under SFT compute $x_{\text{sft}}$, and the remaining \textbf{RL Plasticity} ($PL_{\text{rl}}$), which represents the maximum potential for subsequent RL improvement.
By conducting extensive experiments with the large-scale (i.e., 889K samples) SFT data on Qwen2.5-7B~\citep{qwen2025qwen25technicalreport} and validating on Llama3.2-3B~\citep{llama32} across six mathematical benchmarks, we demystify expert trajectory utilization and establish a rigorous standard for post-training scaling:
\ding{202} \textbf{Sequential Paradigm Dominance} (\S~\ref{sec:paradigm_compare}). We empirically establish the superiority of the {Sequential SFT-then-RL} pipeline over the unstable, sensitive {Synchronized} approach, as well as pure SFT and RL. A robust SFT phase is necessary to establish the foundational SFT performance ($P_{\text{sft}}$) and unlock the maximum plasticity ($PL_{\text{rl}}$) of subsequent RL.
\ding{203} \textbf{Switch RL until SFT Saturation} (\S~\ref{sec:compute_impact}). We identify the \textit{Stable or Mild-Overfitting Sub-phase} of validation loss saturation as the optimal SFT-to-RL transition window, where the $P_{\text{sft}}$ is maximized and $PL_{\text{rl}}$ is uncompromising.
\ding{204} \textbf{Scale and Difficulty Extend Ceiling} (\S~\ref{sec:data_impact}). We refute the ``Less is More'' hypothesis in the context of {SFT-then-RL} scaling. While minimal data yields SFT efficiency, the SFT data scale remains the primary determinant of the final ceiling, while the trajectory difficulty acts as a multiplier. Furthermore, the minimum SFT validation loss serves as a robust predictor of the final post-training ceiling.
Our contributions are summarized as follows: \ding{202} We propose the \textbf{{Plasticity-Ceiling Framework}}, a theoretical mechanism that decomposes post-training performance into realized SFT performance and the subsequent RL plasticity to guide paradigm selection. \ding{203} We systematically benchmark diverse training strategies, identifying the \textbf{Sequential SFT-then-RL} pipeline as the rigorous standard for stability and performance over synchronized approaches. \ding{204} We formulate precise \textbf{operational guidelines for scaling}, linking data properties and training dynamics to the final reasoning ceiling to enable predictable post-training development.
\vspace{-1em}
\begin{figure}[t]
\centering
\vspace{-0.5cm}
\includegraphics[width=1\linewidth]{
figures/intro_10.jpeg
}
\vspace{-0.5cm}
\caption{
\textbf{The conceptual overview of LLM post-training.}
Sequential SFT-then-RL (blue$\rightarrow$orange) achieves the highest performance ceiling $A_{\text{post}}$, outperforming Pure RL (orange) and Synchronized SFT-RL (striped blue--orange) paths.
Insets highlight that larger, harder data increases plasticity, and RL should start during the Stable SFT.}
\vspace{-0.7cm}
\label{fig:intro}
\end{figure}
\section{Related Works}
\noindent \textbf{Post-Training Paradigms.}
Post-training primarily relies on Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). While theoretical works attempt to unify them~\citep{roadsleadlikelihoodvalue, sft_dpo_unify}, they exhibit distinct empirical behaviors regarding generalization and distribution shifts~\citep{understand_trans, rl_razor}.
The sequential SFT-then-RL strategy is the industrial standard~\citep{practicaltwostagerecipemathematical, Vattikonda2025WebAgent}, though optimizing the transition is non-trivial; \citet{quagmires_sft_rl} caution that high SFT scores can be misleading, as over-fitted models may fail to improve during RL.
Alternatively, {Synchronized SFT-RL} methods like LUFFY~\citep{yan2025learningreasonoffpolicyguidance}, UPT~\citep{lv2025unifiedviewlargelanguage} and SRFT~\citep{Fu2025SRFT} integrate imitation directly into RL to boost efficiency. Our work systematically compares these paradigms to identify the optimal mechanism for maximizing the performance ceiling.
\noindent \textbf{Expert Trajectories Utilization.}
The properties of SFT data critically influence post-training.
Regarding \textbf{scale}, a ``Less is More'' philosophy suggests that minimal, high-quality data suffices for SFT~\citep{limo, s1}. However, others argue that scale remains essential for complex reasoning~\citep{climb_ladder}.
Regarding \textbf{difficulty}, methods like MetaMath~\citep{yu2023metamath} and D3~\citep{d3} demonstrate that harder, difficulty-aware data selection improves SFT outcomes.
Crucially, prior works often evaluate SFT in isolation. We extend this inquiry to the RL phase, investigating how SFT data scale and difficulty dictate the model's \textit{plasticity} (its headroom for subsequent RL scaling) rather than just immediate imitation accuracy.
\section{Preliminary}
\label{sec:preliminary} We summarize the algorithmic foundations used in our
study: (1) standard supervised fine-tuning (SFT), (2) reinforcement learning (GRPO
and DAPO), and (3) synchronized SFT–RL (Syn-SFT-RL) fusion methods used as single-stage
baselines.
\subsection{SFT}
SFT tunes the policy $\pi_{\theta}$ via imitation learning using the answer and expert
trajectory pair $(\mathbf{q},\bm{\tau})$ in the SFT dataset $\mathcal{D}_{\text{SFT}}$:
\begin{small}
\begin{equation}
\label{eq:sft_obj}\textstyle \mathcal{J}_{\text{SFT}}(\theta) \!=\! -\mathbb{E}
_{(\mathbf{q},\! \bm{\tau}) \sim \mathcal{D}_{\text{SFT}}}\!\left[\sum_{t=1}
^{|\bm{\tau}|}\!\log \pi_{\theta}(\bm{\tau}_{t}\!\mid\!\mathbf{q},\!\bm{\tau}_{<t})
\right ]
\end{equation}
\end{small}%
Such paradigm reliably imparts instruction-following and basic reasoning priors~\citep{LmrlGym},
but its performance is bounded by the training distribution~\citep{InstructGPT} and
lacks exploratory capability.
\subsection{RL}
RL extends the model beyond imitation by optimizing reward-guided
exploration. GRPO~\citep{deepseek-math} and DAPO~\citep{Yu2025DAPO} are two widely-used
RL algorithms.
\paragraph{GRPO} replaces a critic with a group-normalized advantage ($\mathbf{A}_{i,t}$). For
each query-answer pair $(\textbf{q},\textbf{a})$ in dataset
$\mathcal{D_{\text{RL}}}$, GRPO samples $G$ response trajectories $\{\bm{\tau}
_{i}\}_{i=1}^{G}$ based on the old policy $\pi_{\theta_{\text{old}}}$. Each trajectory receives a rule-derived reward score ${R}_{i}$. The group-normalized advantage is computed as:
\begin{small}
\begin{equation}
\label{eq:group_level_adv}A_{i,t}= \frac{R_{i}- \text{mean}(\{R_{j}\}_{j=1}^{G})}{\text{std}(\{R_{j}\}_{j=1}^{G})}
.
\end{equation}
\end{small}%
With the advantage, GRPO aims to maximize the expected advantage while regularizing the policy towards a reference policy $\pi_{\text{ref}}$ via the KL divergence term $\beta \cdot \mathbb{D}_{\text{KL}}\left[ \pi_{\theta}||\pi_{\text{ref}}\right]$. The policy loss $\mathcal{J}_{\text{GRPO}}{(\theta)}$ is:
\begin{small}
\begin{align}
\label{eq:grpo_obj} & \mathcal{J}_{\text{GRPO}}(\theta)\ = \mathbb{E}_{(\textbf{q}, \textbf{a})\sim \mathcal{D}_{\text{RL}}, \ \{\bm{\tau}_i\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}}\notag \\
& \left[ \frac{1}{\sum_{i=1}^{G}|\bm{\tau}_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|\bm{\tau}_{i}|}\min \left( r_{i}^{t}A_{i,t}, C_{i}^{t}A_{i,t}\right) \right] \\
& \quad - \beta \cdot \mathbb{D}_{\text{KL}}\left[ \pi_{\theta}||\pi_{\text{ref}}\right] \,, \notag
\end{align}
\end{small}%
where $r_{i}^{t}=\frac{\pi_{\theta}( \bm{\tau}_{i,t}\mid \textbf{q}_{},
\bm{\tau}_{i,<t})}{\pi_{\theta_{old}}( \bm{\tau}_{i,t}\mid \textbf{q}_{},
\bm{\tau}_{i,<t})}$ represents the importance ratio between the new and old policies
for token $\bm{\tau}_{i,t}$. Its clipped counterpart, $C_{i}^{t}=\text{clip}( r_{i}^{t}
,\ 1-\epsilon ,\ 1+\epsilon )$, confines the policy update within a trust
region, preventing excessively large and destabilizing policy updates.
\paragraph{DAPO}~\citep{Yu2025DAPO} further stabilizes training via asymmetric clipping $(\epsilon
_{\text{low}},\epsilon_{\text{high}})$ and dynamically filter the prompts with all correct or wrong on-policy generations. We adopt DAPO as our primary RL algorithm due to its robustness on
mathematical reasoning tasks.
\subsection{Syn-SFT-RL}~\label{sec:Syn-SFT-RL-preliminary}
The Syn-SFT-RL paradigm merges SFT and RL by injecting expert trajectories into the optimization
loop. We introduce three typical algorithms: LUFFY~\citep{yan2025learningreasonoffpolicyguidance}, SRFT~\citep{Fu2025SRFT}, and UPT~\citep{lv2025unifiedviewlargelanguage}.
\paragraph{LUFFY} modifies the $\mathcal{J}_{\text{GRPO}}(\theta)$ in Eq.~\ref{eq:grpo_obj} by jointly optimizing on-policy trajectories and off-policy ones.
It removes both the KL regularization and importance-ratio clipping, and aggregates token-level advantages over a mixture of SFT and RL data. The mixture dataset $\mathcal{D}_{\text{MIX}}$ contains triplets $(\bm{q}, \{\bm{\tau}_{j}\}_{j=1}^{N},\bm{a})$ with the prompt $\bm{q}$, $N$ expert trajectories $\{\bm{\tau}_{j}\}_{j=1}^{N}$ ($N$=1 as the official setup), and answer $\bm{a}$. Hence, LUFFY's loss is formalized as:
\begin{small}
\begin{equation}
\label{eq:luffy_obj}
\begin{aligned}
\mathcal{J}_{\text{LUFFY}}(\theta)
&=
\mathbb{E}_{\substack{
(\bm{q},\{\bm{\tau}_j\}_{j=1}^{N},\bm{a}) \sim \mathcal{D}_{\mathrm{MIX}}\\
\{\bm{\tau}_i\}_{i=1}^{G} \sim \pi_{\theta_{\mathrm{old}}}
}}
\Biggl[
\frac{1}{Z}
\sum_{j=1}^{N}
\sum_{t=1}^{|\bm{\tau}_j|}
\hat{r}_{j}^{t}\,\hat{A}_{j,t}
\\
&\qquad\qquad\qquad
+
\frac{1}{Z}
\sum_{i=1}^{G}
\sum_{t=1}^{|\bm{\tau}_i|}
r_{i}^{t}\,\hat{A}_{i,t}
\Biggr],
\end{aligned}
\end{equation}
\end{small}
where $Z = \sum_{j=1}^{N}|\bm{\tau}_j| + \sum_{i=1}^{G}|\bm{\tau}_i|$ normalizes over all tokens, and the mixed advantages are computed without normalization:
\begin{equation}
\small
\label{eq:luffy_adv}
\hat{A}_{i,t}
=
R_{i}
-
\text{mean}\!\left(
\{R_{j}\}_{j=1}^{N}
\cup
\{R_{i}\}_{i=1}^{G}
\right),
\end{equation}
To avoid entropy collapse on off-policy data, LUFFY further applies regularized importance shaping, which transforms the importance ratio $r_{j}^{t}$ to $\hat{r}_{j}^{t} = r_{j}^{t} / (r_{j}^{t} + \gamma)$ with a small constant $\gamma = 0.1$.
\paragraph{SRFT} combines four components: (i) the standard SFT loss $\mathcal{J}_{\text{SFT}}$ in Eq.~\ref{eq:sft_obj},
(ii) the off-policy loss $\mathcal{J}_{\text{off}}$ from LUFFY (the first term in Eq.~\ref{eq:luffy_obj}),
and (iii) on-policy objectives for positive and negative trajectories in Eq.~\ref{eq:pos_neg_obj}.
For $M$ on-policy positive rollouts $\{\bm{\tau}_i^+\}_{i=1}^{M}$ and $G-M$ on-policy negative ones $\{\bm{\tau}_j^-\}_{j=1}^{G-M}$, SRFT maximizes the likelihood of positive trajectories while suppressing that of negative ones:
\begin{equation}
\small
\label{eq:pos_neg_obj}
\begin{aligned}
\mathcal{J}_{\text{pos}}(\theta)
&=
-\mathbb{E}\Bigl[
\sum_{t=1}^{|\tau_i^+|}
\log \pi_{\theta}\bigl(\tau_{i,t}^{+}\mid \mathbf{q}_i,\tau_{i,<t}^{+}\bigr)
\Bigr],\\
\mathcal{J}_{\text{neg}}(\theta)
&=
\phantom{-}\mathbb{E}\Bigl[
\sum_{t=1}^{|\tau_j^-|}
\log \pi_{\theta}\bigl(\tau_{j,t}^{-}\mid \mathbf{q}_j,\tau_{j,<t}^{-}\bigr)
\Bigr].
\end{aligned}
\end{equation}
The final SRFT objective uses entropy-guided dynamic weights:
\begin{equation}
\small
\label{eq:srft_obj}
\begin{aligned}
\mathcal{J}_{\text{SRFT}}
&=
w_{1}\,\mathcal{J}_{\text{SFT}}
+
\mathcal{J}_{\text{off}}
+
w_{2}\,\mathcal{J}_{\text{pos}}
+
\mathcal{J}_{\text{neg}},
\\
w_{1}
&=
0.5\cdot\text{stop\_grad}\bigl(
e^{-\mathcal{H}(\pi_{\theta})}
\bigr),
\\
w_{2}
&=
0.1\cdot\text{stop\_grad}\bigl(
e^{\mathcal{H}(\pi_{\theta})}
\bigr).
\end{aligned}
\end{equation}
where $\mathcal{H}(\pi_{\theta})$ denotes the policy entropy and \texttt{stop\_grad} prevents gradients from flowing through the weights.
\paragraph{UPT} employs a hard gating mechanism to mix SFT and RL.
Let $p$ denote the average reward over the trajectories sampled for the current prompt $\textbf{q}$, and $\gamma$ be a threshold.
UPT defines a mixed loss
\begin{equation}
\small
\label{eq:upt_mixed}
\mathcal{J}_{\text{UPT}}
=
f_{p}\,\mathcal{J}_{\text{SFT}}
+
g_{p}\,\mathcal{J}_{\text{GRPO}},
\end{equation}
where $(f_p,g_p)$ are determined by $p$:
\begin{equation}
\small
\label{eq:upt_gate}
(f_p,g_p)
=
\begin{cases}
(1,0), & p \le \gamma,\\
(0,1), & p > \gamma .
\end{cases}
\end{equation}
When the model performs poorly on a prompt ($p \le \gamma$), the gate prioritizes SFT-style imitation.
Once the reward exceeds the threshold ($p > \gamma$), the gate switches to pure GRPO optimization to focus on exploration.
\section{The Plasticity-Ceiling Framework}
\label{sec:framework} To systematically evaluate the trade-offs between
different post-training paradigms, we propose the \textit{Plasticity-Ceiling}
analytical framework. Unlike prior works that study SFT or RL scaling in isolation~\citep{ScaleLaw_downstream, rlscale},
our framework treats the SFT-then-RL pipeline as a unified continuum. This
allows us to quantify the respective contributions of the SFT and RL phases to
the overall post-training performance ceiling ($A_{\text{post}}$), whose functional
form is defined in ~\ref{def:ceiling}.
\subsection{Decompose the Post-training Performance}
\label{sec:decomposition} Formally, we decompose the post-training performance
$P_{\text{post}}$ of the typical {SFT-then-RL} pipeline into three distinct
components based on the training stages:
\begin{equation}
\label{eq:Apost_decompose}\small
\begin{gathered}
P_{\text{post}}(x_{\text{sft}},x_{\text{rl}})=P_{0}+\underbrace{(P_{\text{sft}}(x_{\text{sft}})-P_{0})}_{\text{SFT gain} ,\ \Delta P_{\text{sft}}(x_{\text{sft}})}\\
\qquad \qquad \qquad \qquad \qquad +\underbrace{\
(P_{\text{rl}}(x_{\text{rl}})-P_{\text{sft}}\left(
x_{\text{sft}} \right) )}_{\text{RL gain} ,\ \Delta P_{\text{rl}}(x_{\text{rl}})}\\
P_{\text{sft}}\left( x_{\text{sft}}=0 \right) =P_{0},\ P_{\text{rl}}(x_{\text{rl}}=0)=P_{\text{sft}}\left(
x_{\text{sft}} \right) \,,
\end{gathered}
\end{equation}
where $P_{0}$ denotes the base model's initial performance, and $x_{\text{sft}}$,
$x_{\text{rl}}$ denote the compute cost (in FLOPs measured in Appx.~\ref{appx:flops_est})
during the SFT and RL phases, respectively. This decomposition explicitly isolates
the performance contributors: $\Delta P_{\text{sft}}$ represents the gain
realized from SFT given cost $x_{\text{sft}}$, while $\Delta P_{\text{rl}}$ represents
the gain from RL given cost $x_{\text{rl}}$.
Note that Eq.~\ref{eq:Apost_decompose} reduces to Pure-SFT when $x_{\text{rl}}= 0$,
and to Pure-RL (including {Syn-SFT-RL} variants) when $x_{\text{sft}}= 0$.
\subsection{Ceiling and Plasticity}
\label{sec:dynamics}
To estimate the asymptotic ceiling, we model the SFT or RL performance $P(x)$ as a
function of compute $x$ using sigmoidal power laws~\citep{rlscale}\footnote[1]{The sigmoidal power law enables the characterization of the scaling of most SFT or RL runs, except the unstable training instances, such as SRFT in Figure~\ref{fig:paradigm_compare} (Left).}:
\begin{equation}
\label{eq:sigmoid_power_law}\small P(x) = P_{\text{start}}+ \frac{A - P_{\text{start}}}{1
+ (x/C_{\text{mid}})^{-B}}\,,
\end{equation}
where $B$ and $C_{\text{mid}}$ dictate convergence dynamics. Based on this formulation, we define two fundamental properties that characterize the scaling process:
\begin{definition}[Asymptotic Ceiling]
\label{def:ceiling} The ceiling, denoted by \boldsymbol{$A$}, represents the
maximum performance achievable as computation goes to infinity.
\end{definition}
\begin{definition}[Plasticity]
\label{def:plasticity} The plasticity, denoted by \boldsymbol{$PL = A - P_{\text{start}}$},
measures the effective headroom available for improvement from the starting
performance $P_{\text{start}}$.
\end{definition}
\noindent
Then, we can extend these concepts into the {SFT-then-RL} pipeline.
Firstly, the SFT phase costs $x_{\text{sft}}$ compute to achieve a foundation performance $P_{\text{sft}}(x_{\text{sft}})$. After that, the RL phase initiates from the realized SFT outcome, the RL performance extends towards the ultimate
Post-training Ceiling ($A_{\text{post}}$) as the following scaling formulation:
\begin{equation}
\small \label{eq:rl_scaling}P_{\text{post}}(x_{\text{sft}}, x_{\text{rl}}) =
P_{\text{sft}}(x_{\text{sft}}) + \frac{A_{\text{post}}- P_{\text{sft}}(x_{\text{sft}})}{1
+ (x_{\text{rl}}/C_{\text{mid}_{\text{rl}}})^{-B_{\text{rl}}}}\,.
\end{equation}
Consequently, the RL plasticity becomes $PL_{\text{rl}}= A_{\text{post}}- P_{\text{sft}}
(x_{\text{sft}})$. Crucially, unlike $PL_{\text{sft}}$ which is fixed for a given
dataset, $PL_{\text{rl}}$ is dynamic and depends on the quality of the SFT
starting point.
\noindent
\textbf{Theoretical Implication.} The framework reveals a fundamental insight:
maximizing SFT efficiency depends solely on $P_{\text{sft}}$. However, if the
SFT data is suboptimal (e.g., limited in scale), it may shrink the $PL_{\text{rl}}$
and thereby constrain $A_{\text{post}}$.
In this work, we focus on the upper bound
$A_{\text{post}}$ resulting from different expert utilization training configurations.
This bound is derived by fitting a sigmoidal power law (Eq.~\ref{eq:sigmoid_power_law})
to the training points (Compute, Performance). We adopt a robust fitting
strategy detailed in Appx.~\ref{appx:robust_curve_fit} to estimate these curves,
which yields all $A_{\text{post}}$ values presented in~\S~\ref{sec:paradigm_compare}
and~\S~\ref{sec:sft_then_rl_paradigm}. The detailed fitting results are presented in Table~\ref{tab:fit_res_tab}.
\section{Experimental Setup}\label{sec:exp_setup}
To determine the optimal mechanism for utilizing expert trajectories, we organize experiments progressively to address three core research questions:
\noindent \textbf{RQ1: Paradigm Selection.}
Among {Pure-RL}, {Pure-SFT}, {Synchronized SFT-RL}, and {Sequential SFT-then-RL}, which paradigm establishes the most effective post-training baseline, and what are their characterizations?
\noindent \textbf{RQ2: Optimal SFT-to-RL Transition.} Building upon the optimal paradigm identified in \textbf{RQ1}, what is the optimal time to transit to RL from SFT for a maximum final ceiling?
\noindent \textbf{RQ3: Data Properties (Scale \& Difficulty).}
With the paradigm (\textbf{RQ1}) and optimal timing strategy (\textbf{RQ2}) established, what roles do data scale and difficulty play in maximizing the performance ceiling, and do they support or refute the ``Less is More'' hypothesis?
\subsection{Models and Data}
\paragraph{Models.}
We primarily use Qwen2.5-7B~\citep{qwen2025qwen25technicalreport} in §§~\ref{sec:paradigm_compare} and \ref{sec:sft_then_rl_paradigm}, and Llama3.2-3B~\citep{llama32} in §~\ref{sec:validate_llama} for cross-validation. In §~\ref{sec:paradigm_compare}, we apply {Syn-SFT-RL} algorithms to Qwen2.5-Math-7B~\citep{qwen2_5_math} to further examine the influence of model priors.
\input{fig_outlines/baselines_compare}
\paragraph{Training Data.}
We construct SFT datasets of varying scales and difficulties by curating mathematical trajectories from distilled DeepSeek outputs~\citep{amthink1_4m, amthink40m}.
The resulting datasets include the large-scale {SFT889K} with around 889K unique samples, three medium-scale variants controlled for difficulty ({Uniform/Easy/Hard102K}, refer to Table~\ref{tab:data_character} for the difficulty classification), and a held-out validation set {Val-199} with 199 prompt and trajectory pairs. To test data efficiency, we also include {S1K-1.1}~\citep{s1} (S1K for short), containing 1K high-quality R1-style trajectories.
For RL in the SFT-then-RL pipeline, we use {RL62K}, a filtered prompt set from Skywork-OR1-RL~\citep{skywork-or1}.
For {Syn-SFT-RL} methods, we augment RL62K with expert trajectories in SFT889K to create {MIX37K}, which is the subset of SFT889K. Refer to Appx.~\ref{appx:sft_data_curate} for details.
\paragraph{Benchmarks.}
To prevent data leakage, we filter out the benchmark prompts with over $0.8$ cosine similarity against our training set using Qwen3-8B-Embedding~\citep{qwen3embedding}.
We evaluate on the resulting 2,157 unique problems from the following cleaned benchmarks (counts denote original to cleaned):
{GSM8K}~\citep{cobbe2021gsm8k} (1319 to 1317),
{OlympiadBench}~\citep{olympiadbench} (675 to 291),
{Minerva}~\citep{minerva} (272 to 262),
{MATH}~\citep{lightman2023letsverifystepstep} (500 to 237),
and {AIME24/25}~\citep{numina_math_datasets} (30 to 25).
We report the average performance on these unique problems unless otherwise specified.
\subsection{Training and Evaluation}
\noindent \textbf{Training.} Our experiments include two primary paradigms:
(1) {Syn-SFT-RL}: we implement LUFFY~\citep{yan2025learningreasonoffpolicyguidance}, SRFT~\citep{Fu2025SRFT}, and UPT~\citep{lv2025unifiedviewlargelanguage} using the official codebase and recommended configurations.
(2) {Sequential SFT-then-RL}: we first fine-tune the base model on SFT data, then apply RL on the fine-tuned checkpoints. For comparison in §~\ref{sec:paradigm_compare}, we adopt GRPO and DAPO$_{d}$ (GRPO with dynamic difficulty sampling~\citep{Yu2025DAPO}) as the {Pure-RL} baseline. In the {SFT-then-RL} pipeline, we use the enhanced DAPO$_{dc}$ method, which further incorporates asymmetric ratio clipping into DAPO$_{d}$. See Appx.~\ref{appx:train_config} for full implementation details.
\noindent \textbf{Evaluation.}
We report {pass@1} accuracy sampled with a temperature of 0.7 and top-p 1.0 to ensure generation diversity. For the smaller AIME24/25 datasets, we use {Avg@16} for robust estimation. All responses are generated with a maximum length of 8,192 tokens.
\begin{figure*}[t]
\centering
\vspace{-0.5cm}
\includegraphics[width=\linewidth]{figures/combined_algorithm_comparison_final_revised_2.pdf}
\vspace{-0.75cm}
\caption{Compute–performance scaling of post-training paradigms under different initialization conditions.
(\textbf{Left}) Initializing from Qwen2.5-7B. \textbf{Early RL-like runs converge quickly (except unstable instances), while early SFT shows a mild performance disruption due to policy shift~\citep{chord-d}.}
(\textbf{Middle}) Initializing from a saturated SFT checkpoint (10,800 steps on Qwen2.5-7B). \textbf{SFT-then-DAPO$_{d}$ outperforms other paradigms.} DAPO$_{d}$ (74.3) and LUFFY (72.7) yield the highest ceilings among pure RL and Syn-SFT-RL paradigms, respectively.
(\textbf{Right}) Initializing from Qwen2.5-Math-7B. \textbf{UPT and LUFFY demonstrate notable efficiency advantages in this setting.}}
\vspace{-0.5cm}
\label{fig:paradigm_compare}
\end{figure*}
\vspace{-0.2cm}
\section{Experimental Results}
\label{sec:experimental_results}
\subsection{Paradigms Comparison}
\label{sec:paradigm_compare}
To determine the optimal paradigm (\textbf{RQ1}), we systematically benchmark four approaches: {Pure-SFT}, {Pure-RL} (GRPO, DAPO$_{d}$), {Syn-SFT-RL} (LUFFY, SRFT, UPT), and the {SFT-then-RL} pipeline. To ensure fairness, all RL (or Syn-SFT-RL) runs utilize MIX37K, a distribution-consistent subset of SFT889K. Figure~\ref{fig:paradigm_compare} demonstrates that MIX37K suffices to capture performance limits, as RL (or Syn-SFT-RL) methods typically saturate or destabilize within a single epoch.
\paragraph{Limitations of Syn-SFT-RL.} Contrasting prior claims~\citep{yan2025learningreasonoffpolicyguidance, lv2025unifiedviewlargelanguage, Fu2025SRFT},
our experiments reveal severe practical limitations in {Syn-SFT-RL}
methods, which exhibit training instability. For instance, SRFT shows performance
fluctuations with a standard deviation {2.6$\times$ higher} than the stable DAPO$_{d}$
baseline in Figure~\ref{fig:paradigm_compare} (Left) and fails to converge stably from a saturated SFT checkpoint (Figure~\ref{fig:paradigm_compare} Middle). Furthermore, they are highly sensitive to model priors. UPT's superior
efficiency is limited to Qwen2.5-Math-7B (Figure~\ref{fig:paradigm_compare} Right) and vanishes
on general-purpose models, quickly plateauing below GRPO and DAPO$_{d}$ (Figure~\ref{fig:paradigm_compare} Left).
\paragraph{RL Variants Trade Ceiling for Efficiency.} {Pure-RL}, and stable {Syn-SFT-RL} methods show a common trade-off: superior initial efficiency but a restricted ceiling. While
GRPO, DAPO$_d$, and LUFFY surge to around 71.5 points within 25 exaFLOPs (outperforming
{Pure-SFT}'s 69.8), they plateau prematurely, yielding negligible subsequent
gains (Figure~\ref{fig:paradigm_compare} Left). This suggests that without a dedicated
supervised phase for internalizing reasoning patterns, improvement headroom is
structurally limited.
\paragraph{SFT Foundation and Sequential RL Maximization.} In contrast, {Pure-SFT} demonstrates ``Slow but High'' scaling, achieving continuous improvement
through extensive imitation to reach a peak of 76.9 points, significantly
surpassing {Pure-RL} and {Syn-SFT-RL} ceilings, which are 74.3 and 72.7, respectively. Crucially, transitioning
to RL after SFT saturation successfully unlocks further gains (Figure ~\ref{fig:paradigm_compare}
Middle). The {SFT-then-RL} pipeline (SFT$\rightarrow$ DAPO$_\text{d}$) achieves the best performance with 78.1 points among all baselines, extending the post-training performance frontier by optimally synergizing
the SFT performance with further RL improvement.
\paragraph{Answer to RQ1: Sequential SFT-then-RL is the superior paradigm.} Large-scale
SFT provides the necessary robust foundation, which sequential RL then leverages
to maximize the final performance frontier.
\subsection{SFT-then-RL Pipeline}
\input{fig_outlines/vary_datasets}
\label{sec:sft_then_rl_paradigm} Building on the superiority of the {SFT-then-RL}
paradigm (\S~\ref{sec:paradigm_compare}), we now examine the two key factors governing
its final ceiling ($A_{\text{post}}$): SFT compute allocation (\textbf{RQ2}) and
data properties (\textbf{RQ3}). We first establish a robust timing strategy,
followed by an analysis of data scale and difficulty impacts.
\subsubsection{The Impact of SFT Compute Allocation}
\label{sec:compute_impact} Balancing realized SFT performance ($P_{\text{sft}}$)
against preserving RL plasticity ($PL_{\text{rl}}$) is crucial for determining
the optimal SFT-to-RL transition.
\paragraph{SFT Sub-phases.}
To rigorously identify the optimal transition point, we temporally partition the SFT process based on the SFT validation loss $L(x_{\text{sft}})$. The entire trajectory is segmented based on the following mathematically defined regions:
\begin{equation}
\small
\begin{aligned}
\mathcal{T}_{\text{stable}} &= \{\,x_{\text{sft}} \mid L(x_{\text{sft}}) \le (1+\delta)L_{\min}\,\}, \\
\mathcal{T}_{\text{mild}} &= \{x_{\text{sft}} \mid (1+\delta)L_{\min} < L(x_{\text{sft}}) \\
&\qquad < (1+\delta_2)\cdot L_{\min}\}, \\
\mathcal{T}_{\text{severe}} &= \{\,x_{\text{sft}} \mid L(x_{\text{sft}}) \ge (1+\delta_2) \cdot L_{\min}\,\}.
\end{aligned}
\end{equation}
where $L_{\min}$ is the global minimum validation loss observed during training, $(\delta, \delta_2)$ are tolerance thresholds being set as (0.02, 0.1) empirically. Therefore, we have
\begin{itemize}[leftmargin=12pt, nosep]
\item \textbf{Adaptive Sub-phase} ($\mathcal{P}_{\text{adapt}}$), where SFT is underfitting in the region.
\begin{small}
\begin{equation}
\mathcal{P}_{\text{adapt}} = \{x_{\text{sft}} \mid x_{\text{sft}} < \min \mathcal{T}_{\text{stable}}\}
\end{equation}
\end{small}
\item \textbf{Stable Sub-phase} ($\mathcal{P}_{\text{stable}}$), where the validation loss saturates within a small tolerance threshold of 2\% (i.e., $\delta=0.02)$.
\begin{small}
\begin{equation}
\mathcal{P}_{\text{stable}} = \mathcal{T}_{\text{stable}}
\end{equation}
\end{small}
\item \textbf{Mild Overfitting Sub-phase} ($\mathcal{P}_{\text{mild}}$), where the region where loss rises slightly but remains below the 10\% tolerance, representing the ``risky sweet spot.''
\begin{small}
\begin{equation}
\mathcal{P}_{\text{mild}} = \{x_{\text{sft}} \mid x_{\text{sft}} > \max \mathcal{T}_{\text{stable}} \text{ and } x_{\text{sft}} \in \mathcal{T}_{\text{mild}}\}
\end{equation}
\end{small}
\item \textbf{Severe Overfitting Sub-phase} ($\mathcal{P}_{\text{severe}}$), where loss significantly diverges ($\ge 10\%$ rise when $\delta_2=0.1$), leading to rapid plasticity collapse (see Easy102K in Figure~\ref{fig:vary_datasets}).
\begin{small}
\begin{equation}
\mathcal{P}_{\text{severe}} = \{x_{\text{sft}} \mid x_{\text{sft}} > \max \mathcal{T}_{\text{stable}} \text{ and } x_{\text{sft}} \in \mathcal{T}_{\text{severe}}\}
\end{equation}
\end{small}
\end{itemize}
\paragraph{The Dynamics of Post-training Ceiling.} The blue solid line in Figure~\ref{fig:vary_datasets}
illustrates how $A_{\text{post}}$ evolves across these phases.
We observe that initiating RL prematurely during the {Adaptive Sub-phase} is
consistently suboptimal because the model lacks foundational competence that
subsequent RL cannot fully recover. For instance, on SFT889K, switching early at
69.8 exaFLOPs yields a ceiling of only 81.1 points, whereas extending training
to the {Stable Sub-phase} (1047.6 exaFLOPs) boosts the ceiling to its peak of 85.7
points. Ideally, for high-quality data (e.g., SFT889K, Hard102K), the
{Stable Sub-phase} aligns perfectly with peak performance.
However, on limited or simple datasets (e.g., S1K, Easy102K), the peak ceiling
often shifts into the {Mild Overfitting Sub-phase}, indicating that a slightly delayed
transition is acceptable and can even be beneficial due to the improvement of $P_{\text{sft}}$.
Conversely, aggressively continuing SFT into the {Severe Overfitting Sub-phase} is
detrimental. As demonstrated on Easy102K in Figure~\ref{fig:vary_datasets}, training
SFT to 335.9 exaFLOPs leads to a rapid decline in the final ceiling due to a collapse
in RL plasticity.
\paragraph{Answer to RQ2: Train SFT to Saturation.} The optimal strategy is to surpass the {Adaptive Phase} and target the \textbf{Stable Sub-phase}, strictly avoiding {Severe Overfitting} to preserve the RL plasticity. While {Mild Overfitting} is permissible for small or easy datasets, the Stable Sub-phase remains the robust standard for scalable data to maximize the total ceiling.
\vspace{-0.5em}
\subsubsection{The Impact of SFT Data Properties}
\label{sec:data_impact} Data scale and difficulty are critical determinants of
the quality of the SFT prior. In this section, we focus on investigating how
these two fundamental data properties influence the asymptotic post-training
performance ceiling.
\paragraph{Larger Scale Begets Higher Ceiling.} Comparing datasets of varying scales
(S1K, Uniform102K, and SFT889K in Figure~\ref{fig:vary_datasets}) reveals that
while minimal data can achieve rapid initial SFT gains, extensive data scale is indispensable
for reaching a higher post-training ceiling. Initially, small-scale data exhibits
deceptive efficiency: S1K achieves an SFT performance of approximately 73.8
points using only 2.3 exaFLOPs, matching the performance level that requires 69.3
exaFLOPs on Uniform102K and 174.6 exaFLOPs on SFT889K.
However, this efficiency proves
to be unsustainable. The realized SFT performance $P_{\text{sft}}$ of S1K
saturates prematurely at this level. In contrast, Uniform102K and SFT889K
continue to improve with additional compute, reaching peak SFT performances of 74.8
and 76.3, respectively, thereby establishing a superior foundation for the subsequent
RL phase. Crucially, large-scale SFT also preserves greater RL plasticity.
SFT889K maintains an average $PL_{rl}$ of 9.4, exceeding both S1K and Uniform102K
by 5.7 points. Consequently, by leveraging both higher realized SFT performance $P
_{\text{sft}}$ and enhanced RL plasticity $PL_{rl}$, large-scale SFT unlocks a
significantly higher post-training ceiling.
\paragraph{Harder Data Elevates the Ceiling.} Controlling for scale (102K samples),
we examine the impact of trajectory difficulty using Easy102K, Uniform102K, and
Hard102K. Figure~\ref{fig:vary_datasets} (Bottom Row) reveals that training on harder
trajectories yields superior returns. Hard102K achieves a higher average SFT performance ($P_{\text{sft}}$) of 74.6 points, outperforming Easy102K
and Uniform102K by 1.5 and 0.8 percentage points, respectively.
More importantly,
data difficulty positively correlates with subsequent RL potential. Hard102K maintains
the highest average $PL_{\text{rl}}$ of 5.4, surpassing Easy102K and Uniform102K
by 1.2 and 1.7 points, respectively. Consequently, the synergistic combination of
higher SFT performance and enhanced RL plasticity makes harder data the superior
choice for maximizing the post-training ceiling.
\paragraph{Minimum Validation Loss as a Predictive Indicator.} A compelling finding
across diverse SFT data configurations is the strong negative correlation (Pearson
$r = -0.90$) between the minimum SFT validation loss and the maximal subsequent
post-training ceiling ($A_{\text{post}}$), as shown in Figure~\ref{fig:correlation}. This establishes minimum validation
loss as a valuable a priori indicator requiring no expensive RL training: a lower
minimum loss reliably signals greater overall post-training capacity within the {SFT-then-RL}
pipeline.
\paragraph{Answer to RQ3: Scale Dominates, Difficulty Optimizes.}
Refuting ``Less is More'', we establish Data \textbf{Scale} as the primary factor to improve the post-training ceiling, while \textbf{Difficulty} acts as a multiplier. Harder trajectories are helpful when the data scale is limited. Thus, scaling must prioritize volume before difficulty, with the final potential reliably predicted by the minimum SFT validation loss.
\input{fig_outlines/llama_sft_rl}
\input{tables/llama_validate}
\vspace{-0.3cm}
\subsection{The Validation on Llama3.2-3B}
~\label{sec:validate_llama}
We validate our findings on Llama3.2-3B ~\citep{llama32}
to ensure generalization across model architectures and sizes. To prioritize
practical relevance, we report the maximum achieved post-training performance ($\max
P_{\text{post}}$) instead of the theoretical ceiling ($A_{\text{post}}$), with
all RL training capped at 200 steps.
For \textbf{RQ1}, Table~\ref{tab:llama_validate} confirms that the SFT-then-RL
pipeline is vastly superior, boosting the Llama3.2-3B baseline to 68.7 points with a 29$\times$
gain. In contrast, single-stage methods struggle significantly: Pure-RL (DAPO$_{d}$) and
Syn-SFT-RL (LUFFY) yield only minimal improvements, and even UPT trails the
sequential approach by over 56 points.
For \textbf{RQ2}, consistent with our earlier findings, Llama3.2‑3B also
achieves peak performance during the SFT Stable Sub‑phase at 532.5 exaFLOPs, as shown in Figure~\ref{fig:llama_sft_then_rl}.
We observe that with Llama3.2‑3B, a light SFT (Adaptive Sub‑phase) fails to unlock
the potential of RL and can even lead to performance degradation (at 17.8 exaFLOPs). In contrast, as
SFT is intensified, $\max P_{\text{post}}$ rises steadily. Thus, for smaller
models, training SFT to saturation becomes even more critical to approaching the
model’s maximum post‑training potential.
For \textbf{RQ3}, to ensure fair comparison near the performance ceiling, we
select the SFT checkpoint exhibiting the minimum validation loss from each data
configuration for the subsequent RL phase. The impact of SFT data properties shows
the same pattern as Qwen2.5-7B as follows:
\begin{itemize}[leftmargin=12pt, nosep]
\item \textbf{Data Scale Dominance:} As
shown in Table~\ref{tab:llama_validate}, the {SFT-then-RL} instance
trained on the largest dataset (SFT889K) achieves the highest overall
performance of 70.1, significantly exceeding the models trained on the 102K-scale
(Uniform102K) and 1K-scale (S1K) datasets.
\item \textbf{Difficulty
Optimization:} While increasing difficulty (Hard102K) yields consistent gains
over easier subsets(Uniform102K, Easy102K), it cannot compensate for the performance
gap caused by insufficient scale.
\item \textbf{Predictive Power of
Validation Loss:} the strong negative correlation between minimum SFT validation
loss and the final performance ceiling persists (Pearson $r = -0.98$), reinforcing
validation loss as a robust indicator of post-training potential, consistent
with our observations in \S~\ref{sec:compute_impact}.
\end{itemize}
\begin{figure*}[t]
\centering
\vspace{-0.5cm}
\includegraphics[width=1\linewidth]{figures/vary_datasets_12.pdf}
\vspace{-0.5cm}
\caption{SFT Compute Scaling Dynamics of the SFT-then-RL Pipeline across Diverse Data Properties.
The charts illustrate the evolution of the post-training ceiling ($A_{\text{post}}$) against increasing SFT compute ($x_{\text{sft}}$). $A_{\text{post}}$ is decomposed into the SFT Performance ($P_{\text{sft}}$) and RL Plasticity ($PL_{rl}$). Background colors highlight the SFT sub-phases (Adaptive, Stable, Mild, and Severe Overfitting) defined by validation loss in \S~\ref{sec:compute_impact}. More details refer to~\ref{tab:fit_res_tab}.}
\vspace{-0.5cm}
\label{fig:vary_datasets}
\end{figure*}
\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{
figures/llama_3b_sft_narrow_compact_2.jpg
}
\vspace{-0.5cm}
\caption{
{The analysis of the max post-training performance $\max P_{\text{post}}$ when performing SFT-then-RL on Llama3.2-3B.} SFT is performed on SFT889K. \textbf{Stable sub-phase begets higher $\max P_{\text{post}}$ results.}}
\vspace{-0.5cm}
\label{fig:llama_sft_then_rl}
\end{figure}
\begin{table}[!htbp]
\vspace{-1em}
\caption{\textbf{Llama3.2-3B validation results.} We report the maximum post-training performance ($\max P_{\text{post}}$) and minimum SFT validation loss (Min. Val Loss). The strong negative correlation ({Pearson $r = -0.98$}) between SFT loss and peak post-training performance confirms that the SFT validation loss is a reliable predictor of the performance ceiling. DAPO$_d$ and DAPO$_{dc}$ are DAPO variants for the fair comparison, whose difference is detailed in~\ref{appx:rl_implement}. The highest performance and lowest loss are bolded.}
\label{tab:llama_validate}
\centering
\small
\resizebox{0.9\columnwidth}{!}{
\begin{tabular}{@{}cccc@{}}
\toprule
{Methods} & {Paradigm} & {$\max P_{\text{post}}$} & {Min. Val Loss} \\ \midrule
Llama3.2-3B & - & 2.3 & - \\ \midrule
DAPO$_d$ & Pure RL & 2.2 & - \\
UPT & \multirow{2}{*}{Syn-SFT-RL} & 12.2 & - \\
LUFFY & & 8.5 & - \\
S1K & \multirow{5}{*}{SFT} & 24.0 & 0.7 \\
Easy102K & & 52.0 & 0.59 \\
Uniform102K & & 53.2 & 0.54 \\
Hard102K & & 55.3 & 0.50 \\
SFT889K & & 67.1 & \textbf{0.40} \\
SFT889K$\rightarrow$ DAPO$_d$ & SFT-then-RL & \textbf{68.7} & - \\ \midrule
S1K $\rightarrow$ DAPO$_{dc}$ & \multirow{5}{*}{SFT-then-RL} & 24.9 & - \\
Easy102K $\rightarrow$ DAPO$_{dc}$ & & 53.7 & - \\
Uniform102K $\rightarrow$ DAPO$_{dc}$ & & 55.1 & - \\
Hard102K $\rightarrow$ DAPO$_{dc}$ & & 56.3 & - \\
SFT889K $\rightarrow$ DAPO$_{dc}$ & & \textbf{70.1} & - \\ \bottomrule
\end{tabular}
}
\vspace{-1.5em}
\end{table}
\section{Conclusions}~\label{sec:conclusion}
This work presents the {\textbf{Plasticity-Ceiling Framework}} for optimizing expert trajectory utilization,
formalizing the trade-off between supervised fine-tuning performance (\(P_{\text{sft}}\)) and reinforcement learning
plasticity (\(PL_{\text{RL}}\)). We derive three core principles for effective scaling: (1) The sequential
\textbf{SFT-then-RL pipeline} outperforms alternative paradigms in approaching the post-training performance ceiling.
(2) Within this pipeline, RL should be initiated at \textbf{SFT saturation}, a point reliably predicted by validation
loss minimization. (3) \textbf{SFT data scale} primarily determines the performance ceiling, and trajectory difficulty
further optimizes the ceiling when data is limited. Together, these findings transform expert trajectory optimization from
empirical guesswork into a systematic and predictable process, establishing a rigorous standard for maximizing reasoning model
performance.
\section{Experimental Platforms}
\label{appx:training_platform}
All SFT experiments in ~\S~\ref{sec:paradigm_compare} run on 16 GPUs; RL and Syn-SFT-RL experiments in ~\S~\ref{sec:paradigm_compare} are implemented on 8 GPUs, and RL experiments in ~\S~\ref{sec:sft_then_rl_paradigm} are conducted on 128 Ascend 910B NPUs.
\section{Training Configuration}
\label{appx:train_config}
\subsection{SFT}
We train SFT889K and all 102K variants with batch size of 512 and learning rate 1e-5 for 8 and 9 epochs, respectively.
To study severe overfitting, we continue training Easy102K up to 6{,}120 steps (335.9 exaFLOPs).
Checkpoints are saved every 360 steps (0.2 epochs for SFT889K, 1.8 epochs for 102K variants).
For S1K, we follow the official setup: batch size 16, learning rate 1e-5, weight decay 1e-4, and train for 5 epochs, with checkpoints saved every 62 steps (1 epoch).
\subsection{RL Practice}
For all RL and Syn-SFT-RL runs, we employ a binary correctness reward, where a correct trajectory receives a reward of 1 and an incorrect trajectory receives 0.
This correctness is verified using a script powered by Math-Verify~\citep{mathverify2024}.
Furthermore, token-level loss aggregation is uniformly applied across all runs.
\subsubsection{The RL in Paradigms Comparison}~\label{appx:rl_implement}
We summarize the RL configuration for the training in \S~\ref{sec:paradigm_compare}, including the Pure-RL (GRPO and $\text{DAPO}_{d}$), and Syn-SFT-RL (LUFFY, SRFT, UPT). The shared training hyperparameters for Pure-RL and Syn-SFT-RL methods are summarized in Table~\ref{tab:rl_config}. Unless specified, all algorithms use this default configuration.
\paragraph{Pure-RL.}
GRPO and $\text{DAPO}_{d}$ serve as Pure-RL baselines. $\text{DAPO}_{d}$ adds the dynamic difficulty sampling strategy~\citep{Yu2025DAPO} on GRPO. For $\text{DAPO}_{d}$, the dynamic difficulty sampling strategy employs a batch size of 128 responses per inference round, and the asymmetric clipping ratio strategy is not applied.
\input{tables/rl_config}
\noindent\textbf{Syn-SFT-RL.}
We adopt training configurations from the Unify-Post-Training codebase~\citep{lv2025unifiedviewlargelanguage}. UPT uses a smaller learning rate of $5\mathrm{e}{-6}$ instead of $1\mathrm{e}{-6}$.
For rollout generation, UPT adaptively allocates up to 8 trajectories between on-policy and off-policy samples, whereas LUFFY~\citep{yan2025learningreasonoffpolicyguidance} and SRFT~\citep{Fu2025SRFT} maintain a fixed 7:1 ratio of on-policy to expert trajectories per prompt. The maximum trajectory length for all Syn-SFT-RL algorithms is set to 8192, as suggested by \citet{yan2025learningreasonoffpolicyguidance}, \citet{Fu2025SRFT}, and \citet{lv2025unifiedviewlargelanguage}.
\subsubsection{The RL in SFT-then-RL pipeline}
Recognizing the superiority of $\text{DAPO}_{d}$ when starting from an SFT checkpoint in \S~\ref{sec:paradigm_compare}, we further improve $\text{DAPO}_{d}$ to $\text{DAPO}_{dc}$ as the RL method in the following {SFT-then-RL} experiments in \S~\ref{sec:sft_then_rl_paradigm} and \S~\ref{sec:validate_llama}. $\text{DAPO}_{dc}$ adds the asymmetric clipping ratio strategy to $\text{DAPO}_{d}$, setting $(\epsilon_{\text{high}}, \epsilon_{\text{low}})=(0.28, 0.2)$. In $\text{DAPO}_{dc}$, each inference round uses 128 responses for dynamic difficulty sampling. The batch size and update batch size are 64, the learning rate is 1e-6, and the rollout number is 8. Maximum lengths for prompt and response are 1024 and 8192, respectively. Furthermore, the entropy and KL term coefficients are 0, and the group advantage normalization is enabled.
\input{tables/data_character}
\vspace{-0.2cm}
\section{Compute Estimation}
\label{appx:flops_est}
We adopt \textbf{FL}oating-point \textbf{OP}erations (FLOPs) as our computational metric because it is hardware-agnostic and parallelization-agnostic, depending only on model architecture and sequence lengths during training.
We employ the FlopsCounter code~\citep{verl_flops_counter} of the Verl framework~\citep{verl} for estimation.
For SFT, FLOPs are estimated based on the sequence length of the SFT dataset; For RL and Syn-SFT-RL, we dynamically compute FLOPs using real-time prompt and response lengths recorded in TensorBoard~\citep{tensorboard} logs. During training, both forward and backward cost the computation.
\paragraph{Forward FLOPs Per-Token Estimation.} The the theoretical forward FLOPs per token is denoted as $\mathcal{F}_{\text{forward\_token}}$, based on the model configuration and average sequence length $S$. Let $L$ be the number of layers, $H$ the hidden size, $H_{ff}$ the intermediate size of the feed-forward network, and $V$ the vocabulary size. For the attention mechanism, we define $D_{KV}$ as the total dimension of the Key and Value heads, accounting for Grouped Query Attention (GQA)~\citep{gqa}.
First, we define the parameter counts for the constituent dense components. The MLP block, which utilizes a SwiGLU activation function with three linear projections (gate, up, and down), has a parameter count $P_{\text{MLP}}$. The linear projections in the attention layer (comprising $W_Q, W_K, W_V, W_O$) contribute $P_{\text{attn\_linear}}$. The embedding layer and the language model head share the vocabulary-dim parameters, denoted as $P_{\text{vocab}}$. These are formulated as:
\begin{small}
\begin{equation}
\begin{aligned}
P_{\text{MLP}} &= 3 H H_{ff} \\
P_{\text{attn\_linear}} &= H(H + 2D_{KV} + H) = 2H(H + D_{KV}) \\
P_{\text{vocab}} &= 2 V H
\end{aligned}
\end{equation}
\end{small}
The total FLOPs consists of the dense computation part ($\mathcal{F}_{\text{dense}}$) and the attention score computation part ($\mathcal{F}_{\text{attn\_core}}$). The dense part aggregates the parameters from all $L$ layers and the vocabulary projections, multiplied by a factor of 2 (for multiply-accumulate operations). The attention core part depends linearly on the sequence length $S$. The final estimation is given by:
\begin{small}
\begin{equation}
\begin{aligned}
\mathcal{F}_{\text{dense}} = & 2 \cdot \left[ L \cdot (P_{\text{MLP}} + P_{\text{attn\_linear}}) + P_{\text{vocab}} \right] \\
\mathcal{F}_{\text{attn\_core}} &= 4 \cdot S \cdot L \cdot H \\
\mathcal{F}_{\text{forward\_token}} &= \mathcal{F}_{\text{dense}} + \mathcal{F}_{\text{attn\_core}}
\end{aligned}
\end{equation}
\end{small}
\paragraph{Backward FLOPs Per-Token Estimation.}
According to~\citet{magatron_train} and~\citet{kaplan2020scalinglawsneurallanguage}, the theoretical backward FLOPs per token is approximately two times that of forward. Let $\mathcal{F}_{\text{forward\_token}}$ be the theoretical backward FLOPs per token:
\begin{equation}
\begin{aligned}
\mathcal{F}_{\text{backward\_token}} &= 2\cdot \mathcal{F}_{\text{forward\_token}}
\end{aligned}
\end{equation}
\subsection{SFT Per-step Estimation}
Per-step SFT accounts for one forward and one backward pass per step. Let $B$ denote the batch size (number of sequences), $S$ the average sequence length used for fine-tuning.
The total number of tokens processed during SFT per step is given by $T_{\text{total}} = B \cdot S$. Since the backward pass requires approximately twice the FLOPs of the forward pass, the total FLOPs per token during training is $3 \cdot \mathcal{F}_{\text{forward\_token}}$.
Therefore, the total computational cost for SFT, denoted as $\mathcal{F}_{\text{SFT}}$, is calculated as:
\begin{small}
\begin{equation}
\begin{aligned}
\mathcal{F}_{\text{train\_token}} &= \mathcal{F}_{\text{forward\_token}} + \mathcal{F}_{\text{backward\_token}} \\&= 3 \cdot \mathcal{F}_{\text{forward\_token}} \\
\mathcal{F}_{\text{SFT}} &= B \cdot S \cdot \mathcal{F}_{\text{train\_token}} \\
&= 3 \cdot B \cdot S \cdot \mathcal{F}_{\text{forward\_token}}
\end{aligned}
\end{equation}
\end{small}
\subsection{RL Per-step Estimation}
\paragraph{DAPO.} For DAPO, the computational cost per step is divided into a {Generation Phase} (dynamic sampling) and a {Training Phase} (actor update).
Let $B_{\text{gen}}$ denote the generation batch size, $K$ the number of dynamic sampling iterations, and $G$ the number of responses per prompt (i.e., group size). In the generation phase, the model explores a large solution space by generating $K \cdot B_{\text{gen}} \cdot G$ sequences. Since this phase involves only inference, the cost is purely forward FLOPs.
In the training phase, a subset of data (removing all correct and wrong trajectories) is selected, denoted by the training batch size $B_{\text{train}}$ (where $B_{\text{train}} < K \cdot B_{\text{gen}}$). The update step involves one forward pass to compute new log-probs and one backward pass. Following standard estimation, the combined update cost (forward + backward) is approximately $3$ times the forward cost per token~\citep{kaplan2020scalinglawsneurallanguage}.
Given the total sequence length $S = S_{\text{prompt}} + S_{\text{response}}$, the FLOPs for one DAPO step are estimated as:
\begin{small}
\begin{equation}
\begin{aligned}
\mathcal{F}_{\text{gen}} &= (K \cdot B_{\text{gen}} \cdot G) \cdot S \cdot \mathcal{F}_{\text{forward\_token}} \\
\mathcal{F}_{\text{train}} &= (B_{\text{train}} \cdot G) \cdot S \cdot 3 \cdot \mathcal{F}_{\text{forward\_token}} \\
\mathcal{F}_{\text{DAPO}} &= \mathcal{F}_{\text{gen}} + \mathcal{F}_{\text{train}} \\
= & (K\cdot B_{\text{gen}} + 3\cdot B_{\text{train}})\cdot G \cdot S \cdot \mathcal{F}_{\text{forward\_token}}
\end{aligned}
\end{equation}
\end{small}
\paragraph{GRPO.}
The algorithm serves as the baseline where no dynamic difficulty sampling is performed. In this setting, the generation batch size equals the training batch size ($B_{\text{gen}} = B_{\text{train}} = B$) and sampling is performed once ($K=1$). The model generates responses for all prompts in the batch and updates on all of them. Thus, the FLOPs estimation simplifies to:
\begin{small}
\begin{equation}
\begin{aligned}
\mathcal{F}_{\text{gen}} &= (1 \cdot B \cdot G) \cdot S \cdot \mathcal{F}_{\text{forward\_token}} \\
\mathcal{F}_{\text{train}} &= (B\cdot G) \cdot S \cdot 3 \cdot \mathcal{F}_{\text{forward\_token}} \\
\mathcal{F}_{\text{GRPO}} &= \mathcal{F}_{\text{gen}} + \mathcal{F}_{\text{train}} \\
&= 4 \cdot B \cdot G \cdot S \cdot \mathcal{F}_{\text{forward\_token}}
\end{aligned}
\end{equation}
\end{small}
\subsection{Syn-SFT-RL Per-step Estimation}
\paragraph{LUFFY and SRFT.}
Both LUFFY and SRFT integrate expert demonstrations into the RL optimization loop. Let $G$ denote the number of on-policy sampled trajectories (group size) and $N$ denote the number of expert trajectories per prompt.
In the {Generation Phase}, the model generates $G$ responses for each prompt in the batch $B$.
In the {Training Phase}, the model updates parameters using both the on-policy generated data and the off-policy expert data. Thus, the effective training batch size per prompt becomes $G + N$ ($G=7, N=1$ in~\S~\ref{appx:rl_implement}).
Given the real-time recorded average on/off-policy sequence length $S_{\text{on}}$ and $S_{\text{off}}$, the FLOPs for LUFFY and SRFT are calculated as the sum of inference cost on $G$ samples and update cost on $G+N$ samples:
\begin{small}
\begin{equation}
\begin{aligned}
\mathcal{F}_{\text{gen}} =& [(B \cdot G) \cdot S_{\text{on}} + (B \cdot N) \cdot S_{\text{off}} ]\cdot \mathcal{F}_{\text{forward\_token}} \\
\mathcal{F}_{\text{train}} =& 3[(B \cdot G) \cdot S_{\text{on}} + (B \cdot N) \cdot S_{\text{off}} ] \cdot \mathcal{F}_{\text{forward\_token}} \\
\mathcal{F}_{\text{Hybrid}} &= \mathcal{F}_{\text{gen}} + \mathcal{F}_{\text{train}} \\
=& 4[(B \cdot G) \cdot S_{\text{on}} + (B \cdot N) \cdot S_{\text{off}} ] \cdot \mathcal{F}_{\text{forward\_token}}
\end{aligned}
\end{equation}
\end{small}
Note that for SRFT, although it computes multiple loss terms (Eq.~\ref{eq:srft_obj}), the dominant computational overhead remains the forward and backward passes through the transformer backbone on the combined data tokens ($G+N$), making this estimation applicable to both algorithms.
\paragraph{UPT.}
The per-step FLOPs of UPT are estimated dynamically based on the actual composition of the training batch, which consists of $N_{\text{on}}$ on-policy samples processed via GRPO and $N_{\text{off}}$ expert samples processed via SFT.
Let $S_{\text{on}}$ be the average on-policy sequence length. The algorithm processes $G\cdot S_{\text{on}}$ tokens during the Generation Phase. In this phase, the computational cost is given by:
\begin{equation}
\small
\mathcal{F}_{\text{gen}} = G \cdot S_{\text{on}} \cdot \mathcal{F}_{\text{forward\_token}}
\end{equation}
Subsequently, the algorithm filters samples based on difficulty, retaining $N{\text{on}}$ on-policy samples and $N_{\text{off}}$ off-policy samples per batch. Consequently, the FLOPs consumption during the Training Phase is:
\begin{equation}
\small
\mathcal{F}_{\text{train}} = 3 \cdot \left( N_{\text{on}}\cdot S_{\text{on}} + N_{\text{off}}\cdot S_{\text{off}} \right) \cdot \mathcal{F}_{\text{forward\_token}}
\end{equation}
Therefore, the total computational cost for a single UPT step is formulated as:
\begin{small}
\begin{equation}
\begin{gathered}\mathcal{F}_{\text{UPT}} =\mathcal{F}_{\text{gen}} +\mathcal{F}_{\text{train}}\\ =\ \left[ G\cdot S_{\text{on}}+3\cdot \left( N_{\text{on}}\cdot S_{\text{on}}+N_{\text{off}}\cdot S_{\text{off}} \right) \right] \cdot \mathcal{F}_{\text{forward\_token}}\end{gathered}
\end{equation}
\end{small}
\section{Robust Curve Fitting}~\label{appx:robust_curve_fit}
In \S~\ref{sec:framework}, we model the SFT and RL scaling progress using sigmoidal curves~\citep{ruan2024observationalscalinglawspredictability, rlscale}.
To accurately model the relationship between computational investment (FLOPs) and model performance, particularly in the presence of training noise and potential anomalies, we employ a robust curve-fitting pipeline. This pipeline integrates an iterative outlier detection mechanism based on Modified Z-scores with a Least Trimmed Squares (LTS) regression optimization.
\subsection{Data Formulation}
Let $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N}$ denote the dataset, where $x_i$ represents cumulative FLOPs and $y_i$ represents the evaluation metric. The data is partitioned into a training set $\mathcal{D}_{\text{train}}$ (first $N_{\text{fit}}$ points) and a held-out validation set $\mathcal{D}_{\text{val}}$. Due to variations in training convergence across runs, the train-validation split may differ slightly. For most runs, approximately 85\% of the data is used for training, with the remaining 15\% reserved for validation.
\subsection{Robust Estimation Algorithm}~\label{appx:robust_estimate}
Standard least-squares estimation is highly sensitive to anomalies. To derive a scaling law that reflects the consistent signal rather than transient noise, we employ a hierarchical robust optimization framework that integrates iterative statistical filtering (Modified Z-score) with subset-based optimization (Least Trimmed Squares) to isolate the true performance signal, ensuring that the derived scaling laws are predictive and generalizable across different compute regimes.
\paragraph{Stage-1: Coarse Outlier Rejection (Modified Z-Score).}
First, we filter gross statistical anomalies. In each iteration, we compute residuals $r_i = y_i - f(x_i; \theta)$ and the median residual $\tilde{r} = \text{median}(\mathbf{r})$. To quantify deviation robustly, we calculate the Median Absolute Deviation (MAD)~\citep{huber2011robust, leys2013detecting}:
\begin{small}
\begin{equation}~\label{sec:mad}
\text{MAD} = \text{median}(|r_i - \tilde{r}|)
\end{equation}
\end{small}
Subsequently, the Modified Z-score $M_i$ is computed as~\citep{iglewicz1993detect}:
\begin{small}
\begin{equation}~\label{eq:outlier_detection}
M_i = \frac{0.6745 \cdot (r_i - \text{median}(r))}{\text{MAD}}
\end{equation}
\end{small}
Points where $|M_i| > \tau$ are removed from the active training set. The factor $0.6745$ normalizes the score such that it is consistent with the standard deviation under a normal distribution, while the use of MAD ensures resilience against extreme values that would skew a standard variance calculation.
\paragraph{Stage-2: Least Trimmed Squares Regression.}
To further refine the model against subsets of data that may distort the global trend, we employ Least Trimmed Squares (LTS)~\citep{rousseeuw1984least, rousseeuw1987robust} regression. Instead of minimizing the sum of all residuals, LTS regression minimizes only the smallest $h$ squared residuals:
\begin{small}
\begin{equation}~\label{eq:lts_obj}
\hat{\theta}_{\text{LTS}} = \underset{\theta}{\arg\min} \sum_{j=1}^{h} (r^2)_{(j)}(\theta)
\end{equation}
\end{small}
\noindent where $(r^2)_{(1)} \le \dots \le (r^2)_{(N_{\text{fit}})}$ are the ordered squared residuals over the training set, and $h = \lfloor N_{\text{fit}} \cdot \alpha \rfloor$ is determined by the parameter $\alpha$. We define ${H^{(k+1)}}$ as the set of indices corresponding to the $h$ smallest squared residuals, i.e., $H^{(k+1)} = \{i \mid r_i^2 \le (r^2)_{(h)}\}$.
We optimize this objective using the Concentration Step (C-step) algorithm \citep{rousseeuw2006computing}, which proceeds iteratively as follows:
\begin{itemize}[leftmargin=12pt, nosep]
\item \textbf{Estimation}: Compute squared residuals $r_i^2 = (y_i - f(x_i; \theta^{(k)}))^2$ for all $N_{\text{fit}}$ training points using the current parameters $\theta^{(k)}$.
\item \textbf{Selection}: Identify the index set $H^{(k+1)}$ corresponding to the $h$ smallest squared residuals.
\item \textbf{Update}: Update parameters to $\theta^{(k+1)}$ by fitting the model strictly to the data points indexed by $H^{(k+1)}$.
\item \textbf{Convergence}: Repeat the process until the parameter estimate $\theta$ stabilizes.
\end{itemize}
\paragraph{Fitting Results.}
Across all fitting instances, the average Root Mean Square Error (RMSE) on the validation split is 0.5, and the average fitting goodness $R^{2}$ on the training split is 0.88, indicating robust fits. We present the fitting results including $A - P_{\text{start}}$, $C_{\text{mid}}$ and $B$ of Eq.~\ref{eq:sigmoid_power_law} in Table~\ref{tab:fit_res_tab}, and visualize the SFT-then-RL curves in Figure~\ref{fig:fitting}.
\input{tables/fit_res_tab}
\input{fig_outlines/fit}
\section{Dataset Curation}
\label{dataset_summary}
We summarize the key characteristics of our SFT data in Table~\ref{tab:data_character}.
\subsection{Expert Trajectory Collection}
\label{appx:sft_data_curate}
\vspace{-0.1cm}
We curate high-quality reasoning trajectories from two large-scale datasets: AM-DeepSeek-R1-Distilled-1.4M~\citep{amthink1_4m} and AM-DeepSeek-Distilled-40M~\citep{amthink40m}.
To ensure data distribution consistency and quality, we retain only mathematics-domain data, select trajectories distilled from DeepSeek-R1-671B~\citep{deepseek-r1} to unify trajectory style, and perform deduplication based on prompt matching.
The resulting filtered datasets are denoted as amthink-1.4m and amthink-40m.
From these sources, we construct multiple datasets for our experiments: SFT889K, Uniform102K, Easy102K, and Hard102K for SFT training, and Val-199 for SFT validation.
\noindent\textbf{Difficulty Classification.}
To understand the influence of SFT data difficulty on post-training outcomes, we extract data of varying difficulty levels from amthink-40m.
We use the Win Rate (WR) as a proxy for problem difficulty, defined as the ratio of successful attempts $S$ to the total number of attempts $N$, i.e., $\mathrm{WR} = S/N$.
This metric quantifies a problem’s success probability: a higher WR indicates easier problems, and a lower WR indicates harder ones.
We derive WRs using the DeepSeek-Distilled-40M model with $N=4$ attempts across three models: DeepSeek-R1-Distill-1.5B, 7B, and DeepSeek-R1-671B~\citep{deepseek-r1}.
Based on WRs from the 1.5B model, problems are classified as Easy (WR $= 1.0$) or Hard (WR $= 0$ or $0.25$).
For comparison, we construct two datasets: Easy102K and Hard102K, each containing 102.4K samples from the respective difficulty pools.
We also uniformly sample 102.4K data points from amthink-40m to generate Uniform102K as a medium-scale, neutral-difficulty baseline.
\subsection{RL Data}
We curate RL62K (62.3K prompts) from Skywork-OR1-RL by filtering out extreme difficulty levels and prompts containing Chinese characters.
For {Syn-SFT-RL}, we construct MIX37K (36.7K samples) by augmenting these prompts with matched expert trajectories from SFT889K, excluding sequences exceeding 8{,}192 tokens as suggested by~\citep{yan2025learningreasonoffpolicyguidance} to ensure the complete trajectory utilization in each update step.
Crucially, this data scale is sufficient, as our experiments show that RL variants typically reach saturation or instability before exhausting a single epoch over MIX37K.
\begin{table}[htbp]
\centering
\small
\begin{tabular}{ll}
\hline
Hyperparameter & Default value \\
\hline
Batch size & 128 \\
Update batch size & 64 \\
Rollout number & 8 \\
Max prompt length & 1024 \\
Max response length & 8192 \\
Learning rate & $1\mathrm{e}{-6}$ \\
Entropy coefficient & 0.001 \\
KL loss term & removed \\
Std.\ in group advantage norm & removed \\
\hline
\end{tabular}
\caption{Shared training hyperparameters for GRPO, $\text{DAPO}_{d}$, LUFFY and SRFT in \S~\ref{sec:paradigm_compare}.}
\label{tab:rl_config}
\end{table}
\begin{table*}[t]
\centering
\caption{Statistical summary of the constructed SFT datasets. The table lists average prompt and response lengths, as well as Win Rate (WR) across different DeepSeek model sizes. These metrics confirm the intended difficulty stratification, distinguishing the complexity levels of Easy, Uniform, and Hard subsets.}
\label{tab:data_character}
\resizebox{0.8\textwidth}{!}{%
\begin{tabular}{@{}cccccc@{}}
\toprule
Dataset & Avg. prompt length & Avg. response length & WR (1.5B) & WR (7B) & WR (671B) \\ \midrule
EASY102K & 64 & 2253 & 1 & 0.98 & 0.99 \\
UNIFORM102K & 74 & 3673 & 0.79 & 0.84 & 0.87 \\
HARD102K & 101 & 8532 & 0.09 & 0.30 & 0.41 \\
SFT889K & 78 & 3693 & - & - & - \\
S1K & 127 & 9884 & - & - & - \\ \bottomrule
\end{tabular}%
}
\end{table*}
\begin{table*}[t]
\centering
\caption{\textbf{The fitting results of different SFT-then-RL configurations, including the Pure-RL (1st row).} Use-LTS denotes whether the Least Trimmed Squares (LTS) regression technique (Stage-2 in ~\S\text{\ref{appx:robust_estimate}}) was applied during curve fitting. For RL plasticity ($PL_{\text{rl}}$), SFT performance ($P_{\text{sft}}$), post-training ceiling ($A_{\text{post}}$), and the steepness ($B$), higher values are better; the maximum within each SFT configuration is bolded. Conversely, $C_{\text{mid}}$ (RL compute cost for $0.5 \times PL_{\text{rl}}$, indicating efficiency) should be lower, and its minimum is also bolded. Across all SFT data configurations, increasing SFT compute generally diminishes RL training efficiency (indicated by $C_{\text{mid}}$) but improves $A_{\text{post}}$. Notably, neither $C_{\text{mid}}$ nor $A_{\text{post}}$ is strictly monotonic with respect to SFT compute $x_{\text{sft}}$.}
\label{tab:fit_res_tab}
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}ccccccccc@{}}
\toprule
SFT data & SFT Step & SFT Compute $x_{\text{sft}}$ (exaFLOPs) & Use-LTS & $PL_{\text{rl}}=A_\text{post}-P_\text{sft}$ & $C_{\text{mid}}$ & \textit{B} & $P_{\text{sft}}$ & $A_\text{post}=PL_{rl}+P_{\text{sft}}$ \\ \midrule
\multirow{1}{*}{-} & 0 & 0 & FALSE & 25.2 & 1 & 1.3 & 46.1 & 71.3 \\ \midrule
\multirow{5}{*}{S1K} & 62 & 0.6 & FALSE & \textbf{4.5} & \textbf{11} & 2.1 & 68.2 & 72.6 \\
& 124 & 1.1 & FALSE & 4.1 & {16} & 1.6 & 70.6 & 74.6 \\
& 186 & 1.7 & FALSE & 3.3 & 46 & 2.5 & 72.7 & 76.0 \\
& 248 & 2.3 & FALSE & 3.7 & 79 & \textbf{3.1 }& \textbf{73.8} & \textbf{77.5} \\
& 310 & 2.9 & FALSE & {3.1} & 5 & 0.6 & 72.8 & 75.9 \\ \midrule
\multirow{5}{*}{Easy102K} & 360 & 19.8 & True ($\alpha=$0.85) & \textbf{4.7} & \textbf{16} & \textbf{1.4} & 70.9 & 75.5 \\
& 720 & 39.5 & True($\alpha=$0.85) & 4.2 & 50 & 0.8 & 73.4 & 77.5 \\
& 1080 & 59.3 & True ($\alpha=$0.85) & 4.6 & 27 & 0.5 & 73.5 & \textbf{78.1} \\
& 1440 & 79.0 & FALSE & 3.6 & 40 & 0.9 & \textbf{73.9} & 77.5 \\
& 1800 & 98.8 & FALSE & 4.1 & 40 & 0.4 & 73.7 & 77.8 \\ \midrule
\multirow{5}{*}{Uniform102K} & 360 & 34.7 & FALSE & \textbf{5.7} & 45 & 1.2 & 72.2 & 77.9 \\
& 720 & 69.3 & True ($\alpha=$0.85) & {3.1} & \textbf{13} & \textbf{2.1} & 73.7 & 76.8 \\
& 1080 & 104.0 & FALSE & 3.6 & 34 & 1.1 & 74.0 & 77.6 \\
& 1440 & 138.7 & FALSE & 3.6 & 43 & 1.1 & \textbf{74.8} & \textbf{78.3} \\
& 1800 & 173.4 & FALSE & 2.5 & 17 & 1.5 & 74.6 & 77.1 \\ \midrule
\multirow{5}{*}{Hard102K} & 360 & 89.3 & FALSE & 5.1 & \textbf{25} & 0.7 & 72.5 & 77.6 \\
& 720 & 178.5 & FALSE & 4.4 & 30 & 1.1 & 73.8 & 78.1 \\
& 1080 & 267.8 & FALSE & \textbf{9.8} & 76 & 2.1 & 75.4 & \textbf{85.2} \\
& 1440 & 357.0 & FALSE & 4.4 & 48 & \textbf{3.6} & \textbf{76.2} & 80.6 \\
& 1800 & 446.3 & FALSE & 3.5 & 48 & 0.6 & 75.3 & 78.7 \\ \midrule
\multirow{12}{*}{SFT889K}
& 360 & 34.9 & FALSE & \textbf{14.7} & 10 & 0.9 & 70.1 & 84.8 \\
& 720 & 69.8 & FALSE & 10.1 & \textbf{6} & 1.8 & 71.0 & 81.1 \\
& 1080 & 104.8 & FALSE & 8.8 & 8 & 2.1 & 72.3 & 81.1 \\
& 1440 & 139.7 & FALSE & 9.1 & 9 & 1.6 & 72.8 & 82.0 \\
& 1800 & 174.6 & FALSE & 8.3 & 8 & 1.9 & 73.7 & 82.0 \\
& 3600 & 349.2 & FALSE & 9.3 & 13 & 1.6 & 75.3 & 84.6 \\
& 5400 & 523.8 & FALSE & 8.8 & 12 & 1.6 & 76.0 & 84.8 \\
& 7200 & 698.4 & FALSE & 9.0 & 15 & 1.6 & 76.4 & 85.4 \\
& 9000 & 873.0 & FALSE & 8.2 & 15 & \textbf{2.5} & 76.5 & 84.8 \\
& 10800 & 1047.6 & FALSE & 9.4 & 13 & 1.5 & 76.3 & \textbf{85.7} \\
& 12600 & 1222.2 & FALSE & 9.3 & 13 & 1.7 & 76.0 & 85.2 \\
& 14080 & 1365.8 & FALSE & 8.3 & 17 & 1.8 & \textbf{76.8} & 85.1 \\ \bottomrule
\end{tabular}}
\end{table*}
\begin{figure*}[t!]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/fit/Correlation_Analysis.pdf}
\caption{Correlation Analysis}
\label{fig:correlation}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/fit/general_dapo_checkout_final.png}
\caption{SFT889K}
\label{fig:general}
\end{subfigure}
\vspace{10pt}
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/fit/s1k_dapo_checkout_final.png}
\caption{S1K}
\label{fig:s1k}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/fit/easy102K_dapo_checkout_final.png}
\caption{Easy102K}
\label{fig:easy}
\end{subfigure}
\vspace{10pt}
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/fit/uniform102K_dapo_checkout_final.png}
\caption{Uniform102K}
\label{fig:uniform}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/fit/hard102K_dapo_checkout_final.png}
\caption{Hard102K}
\label{fig:hard}
\end{subfigure}
\caption{\textbf{Visualization of SFT-then-RL fitting across different SFT data configurations.} (a) Correlation analysis between $A_{\text{post}}$ and Minimum Validation Loss. (b)-(f) The SFT-then-RL scaling dynamics under various data configurations. The SFT trajectory is depicted by a black dashed line. RL scaling curves initiated from different SFT steps are distinguished by a color gradient, where lighter shades indicate a higher number of SFT steps. The specific starting SFT step count for each RL curve is annotated in bold black text. Data points from the training split used for fitting the RL scaling curves are marked with solid circles, while those from the validation split used for assessing curve goodness-of-fit are marked with crosses. Magnified views are provided for the low-compute regions of SFT889K and S1K.}
\label{fig:fitting}
\end{figure*}
\begin{thebibliography}{52}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abadi et~al.(2016)Abadi, Agarwal, Barham, Brevdo, Chen, Citro, Corrado, Davis, Dean, Devin, Ghemawat, Goodfellow, Harp, Irving, Isard, Jia, Jozefowicz, Kaiser, Kudlur, Levenberg, Mane, Monga, Moore, Murray, Olah, Schuster, Shlens, Steiner, Sutskever, Talwar, Tucker, Vanhoucke, Vasudevan, Viegas, Vinyals, Warden, Wattenberg, Wicke, Yu, and Zheng]{tensorboard}
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg~S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.
\newblock Tensorflow: Large-scale machine learning on heterogeneous distributed systems, 2016.
\newblock URL \url{https://arxiv.org/abs/1603.04467}.
\bibitem[Abdulhai et~al.(2023)Abdulhai, White, Snell, Sun, Hong, Zhai, Xu, and Levine]{LmrlGym}
Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine.
\newblock Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023.
\newblock URL \url{https://arxiv.org/abs/2311.18232}.
\bibitem[Ainslie et~al.(2023)Ainslie, Lee-Thorp, de~Jong, Zemlyanskiy, Lebrón, and Sanghai]{gqa}
Joshua Ainslie, James Lee-Thorp, Michiel de~Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai.
\newblock Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023.
\newblock URL \url{https://arxiv.org/abs/2305.13245}.
\bibitem[Chen et~al.(2025)Chen, Huang, Gao, Wang, Yang, and Ji]{ScaleLaw_downstream}
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji.
\newblock Scaling laws for predicting downstream performance in llms, 2025.
\newblock URL \url{https://arxiv.org/abs/2410.08527}.
\bibitem[Cobbe et~al.(2021)Cobbe, Kosaraju, Bavarian, Chen, Jun, Kaiser, Plappert, Tworek, Hilton, Nakano, Hesse, and Schulman]{cobbe2021gsm8k}
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
\newblock Training verifiers to solve math word problems.
\newblock \emph{arXiv preprint arXiv:2110.14168}, 2021.
\bibitem[DeepSeek-AI(2025)]{deepseek-r1}
DeepSeek-AI.
\newblock Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
\newblock URL \url{https://arxiv.org/abs/2501.12948}.
\bibitem[Engine(2023)]{verl_flops_counter}
Volcano Engine.
\newblock {VERL} utils: {FLOPs} counter (line 149).
\newblock \url{https://github.com/volcengine/verl/blob/59049a66/verl/utils/flops_counter.py\#L149}, 2023.
\newblock version 59049a6; Accessed: 2024-12-01.
\bibitem[Fu et~al.(2025)Fu, Chen, Chai, Wang, Tu, Yin, Lin, Zhang, Zhu, and Zhao]{Fu2025SRFT}
Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao.
\newblock {SRFT}: A single-stage method with supervised and reinforcement fine-tuning for reasoning.
\newblock \emph{arXiv preprint arXiv:2506.19767}, 2025.
\newblock \doi{10.48550/arXiv.2506.19767}.
\newblock URL \url{https://arxiv.org/abs/2506.19767}.
\bibitem[GLM et~al.(2025)GLM, Zeng, Lv, Zheng, Hou, Chen, Xie, Wang, Yin, Zeng, Zhang, Wang, Zhong, Liu, Lu, Cao, Zhang, Huang, Wei, Cheng, An, Niu, Wen, Bai, Du, Wang, Zhu, Zhang, Wen, Wu, Xu, Huang, Zhao, Cai, Yu, Li, Ge, Huang, Zhang, Xu, Zhu, Li, Yin, Lin, Yang, Jiang, Ai, Zhu, Wang, Pan, Wang, Sun, Li, Li, Hu, Zhang, Peng, Tai, Zhang, Wang, Yang, Liu, Zhao, Liu, Yan, Liu, Chen, Li, Zhao, Ren, Jiao, Zhao, Yan, Wang, Gui, Zhao, Liu, Li, Li, Lu, Wang, Yuan, Li, Du, Du, Liu, Zhi, Gao, Wang, Yang, Xu, Fan, Wu, Ding, Wang, Zhang, Li, Xu, Zhao, Zhai, Du, Dong, Lei, Tu, Yang, Lu, Li, Li, Shuang-Li, Yang, Yi, Yu, Tian, Wang, Yu, Tam, Liang, Liu, Wang, Jia, Gu, Ling, Wang, Fan, Pan, Zhang, Zhang, Fu, Zhang, Xu, Wu, Lu, Wang, Zhou, Pan, Zhang, Wang, Li, Su, Geng, Zhu, Yang, Li, Wu, Li, Liu, Wang, Li, Zhang, Liu, Yang, Zhou, Qiao, Feng, Liu, Zhang, Wang, Yao, Wang, Liu, Chai, Li, Zhao, Chen, Zhai, Xu, Huang, Wang, Li, Dong, and Tang]{glm45}
Team GLM, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da~Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He~Liu, He~Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji~Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao, Ke~Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu~Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng~Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zihan Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang.
\newblock Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025.
\newblock URL \url{https://arxiv.org/abs/2508.06471}.
\bibitem[He et~al.(2024)He, Luo, Bai, Hu, Thai, Shen, Hu, Han, Huang, Zhang, Liu, Qi, Liu, and Sun]{olympiadbench}
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen~Leng Thai, Junhao Shen, Jinyi Hu, Xu~Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun.
\newblock Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
\bibitem[He et~al.(2025)He, Liu, Liu, Yan, Wang, Cheng, Zhang, Zhang, Xu, Shen, Li, Zeng, Wei, Cheng, An, Liu, and Zhou]{skywork-or1}
Jujie He, Jiacai Liu, Chris~Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo~An, Yang Liu, and Yahui Zhou.
\newblock Skywork open reasoner 1 technical report.
\newblock \emph{arXiv preprint arXiv:2505.22312}, 2025.
\bibitem[Huan et~al.(2025)Huan, Li, Zheng, Xu, Kim, Du, Poovendran, Neubig, and Yue]{understand_trans}
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue.
\newblock Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning, 2025.
\newblock URL \url{https://arxiv.org/abs/2507.00432}.
\bibitem[Huber \& Ronchetti(2011)Huber and Ronchetti]{huber2011robust}
P.J. Huber and E.M. Ronchetti.
\newblock \emph{Robust Statistics}.
\newblock Wiley Series in Probability and Statistics. Wiley, 2011.
\newblock ISBN 9781118210338.
\newblock URL \url{https://books.google.com.hk/books?id=j1OhquR_j88C}.
\bibitem[{Hugging Face}(2024)]{mathverify2024}
{Hugging Face}.
\newblock Math-verify.
\newblock \url{https://github.com/huggingface/Math-Verify}, 2024.
\bibitem[Iglewicz \& Hoaglin(1993)Iglewicz and Hoaglin]{iglewicz1993detect}
Boris Iglewicz and David~C Hoaglin.
\newblock \emph{How to detect and handle outliers}, volume~16.
\newblock Asqc Quality Press Milwaukee, WI, 1993.
\bibitem[Kang et~al.(2025)Kang, Kuchnik, Padthe, Vlastelica, Jia, Wu, and Ardalani]{quagmires_sft_rl}
Feiyang Kang, Michael Kuchnik, Karthik Padthe, Marin Vlastelica, Ruoxi Jia, Carole-Jean Wu, and Newsha Ardalani.
\newblock Quagmires in sft-rl post-training: When high sft scores mislead and what to use instead, 2025.
\newblock URL \url{https://arxiv.org/abs/2510.01624}.
\bibitem[Kaplan et~al.(2020)Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, and Amodei]{kaplan2020scalinglawsneurallanguage}
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom~B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
\newblock Scaling laws for neural language models, 2020.
\newblock URL \url{https://arxiv.org/abs/2001.08361}.
\bibitem[Khatri et~al.(2025)Khatri, Madaan, Tiwari, Bansal, Duvvuri, Zaheer, Dhillon, Brandfonbrener, and Agarwal]{rlscale}
Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai~Surya Duvvuri, Manzil Zaheer, Inderjit~S. Dhillon, David Brandfonbrener, and Rishabh Agarwal.
\newblock The art of scaling reinforcement learning compute for llms, 2025.
\newblock URL \url{https://arxiv.org/abs/2510.13786}.
\bibitem[Lewkowycz et~al.(2022)Lewkowycz, Andreassen, Dohan, Dyer, Michalewski, Ramasesh, Slone, Anil, Schlag, Gutman-Solo, Wu, Neyshabur, Gur-Ari, and Misra]{minerva}
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra.
\newblock Solving quantitative reasoning problems with language models, 2022.
\newblock URL \url{https://arxiv.org/abs/2206.14858}.
\bibitem[Leys et~al.(2013)Leys, Ley, Klein, Bernard, and Licata]{leys2013detecting}
Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata.
\newblock Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median.
\newblock \emph{Journal of Experimental Social Psychology}, 49\penalty0 (4):\penalty0 764--766, 2013.
\newblock ISSN 0022-1031.
\newblock \doi{https://doi.org/10.1016/j.jesp.2013.03.013}.
\newblock URL \url{https://www.sciencedirect.com/science/article/pii/S0022103113000668}.
\bibitem[LI et~al.(2024)LI, Beeching, Tunstall, Lipkin, Soletskyi, Huang, Rasul, Yu, Jiang, Shen, Qin, Dong, Zhou, Fleureau, Lample, and Polu]{numina_math_datasets}
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi~Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li~Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu.
\newblock Numinamath.
\newblock \url{[https://github.com/project-numina/aimo-progress-prize](https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf)}, 2024.
\bibitem[Lightman et~al.(2023)Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, and Cobbe]{lightman2023letsverifystepstep}
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
\newblock Let's verify step by step, 2023.
\newblock URL \url{https://arxiv.org/abs/2305.20050}.
\bibitem[Lv et~al.(2025)Lv, Zuo, Sun, Liu, Wei, Chen, He, Zhu, Zhang, Wang, Ding, and Zhou]{lv2025unifiedviewlargelanguage}
Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, Ning Ding, and Bowen Zhou.
\newblock Towards a unified view of large language model post-training, 2025.
\newblock URL \url{https://arxiv.org/abs/2509.04419}.
\bibitem[{Meta AI}(2024)]{llama32}
{Meta AI}.
\newblock Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.
\newblock \url{https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/}, September 2024.
\newblock Meta AI blog; accessed 2025-04-13; 15 minute read.
\bibitem[Muennighoff et~al.(2025)Muennighoff, Yang, Shi, Li, Fei-Fei, Hajishirzi, Zettlemoyer, Liang, Candès, and Hashimoto]{s1}
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang~Lisa Li, Li~Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
\newblock s1: Simple test-time scaling, 2025.
\newblock URL \url{https://arxiv.org/abs/2501.19393}.
\bibitem[Narayanan et~al.(2021)Narayanan, Shoeybi, Casper, LeGresley, Patwary, Korthikanti, Vainbrand, Kashinkunti, Bernauer, Catanzaro, Phanishayee, and Zaharia]{magatron_train}
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay~Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia.
\newblock Efficient large-scale language model training on gpu clusters using megatron-lm, 2021.
\newblock URL \url{https://arxiv.org/abs/2104.04473}.
\bibitem[Ouyang et~al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell, Welinder, Christiano, Leike, and Lowe]{InstructGPT}
Long Ouyang, Jeff Wu, Xu~Jiang, Diogo Almeida, Carroll~L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
\newblock Training language models to follow instructions with human feedback, 2022.
\newblock URL \url{https://arxiv.org/abs/2203.02155}.
\bibitem[Qwen et~al.(2025)Qwen, :, Yang, Yang, Zhang, Hui, Zheng, Yu, Li, Liu, Huang, Wei, Lin, Yang, Tu, Zhang, Yang, Yang, Zhou, Lin, Dang, Lu, Bao, Yang, Yu, Li, Xue, Zhang, Zhu, Men, Lin, Li, Tang, Xia, Ren, Ren, Fan, Su, Zhang, Wan, Liu, Cui, Zhang, and Qiu]{qwen2025qwen25technicalreport}
Qwen, :, An~Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le~Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu~Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
\newblock Qwen2.5 technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2412.15115}.
\bibitem[Rousseeuw(1984)]{rousseeuw1984least}
Peter~J Rousseeuw.
\newblock Least median of squares regression.
\newblock \emph{Journal of the American statistical association}, 79\penalty0 (388):\penalty0 871--880, 1984.
\bibitem[Rousseeuw \& Driessen(2006)Rousseeuw and Driessen]{rousseeuw2006computing}
Peter~J. Rousseeuw and Katrien Driessen.
\newblock Computing lts regression for large data sets.
\newblock \emph{Data Min. Knowl. Discov.}, 12\penalty0 (1):\penalty0 29–45, January 2006.
\newblock ISSN 1384-5810.
\newblock \doi{10.1007/s10618-005-0024-4}.
\newblock URL \url{https://doi.org/10.1007/s10618-005-0024-4}.
\bibitem[Rousseeuw \& Leroy(1987)Rousseeuw and Leroy]{rousseeuw1987robust}
Peter~J Rousseeuw and Annick~M Leroy.
\newblock \emph{Robust regression and outlier detection}.
\newblock John Wiley \& Sons, 1987.
\bibitem[Ruan et~al.(2024)Ruan, Maddison, and Hashimoto]{ruan2024observationalscalinglawspredictability}
Yangjun Ruan, Chris~J. Maddison, and Tatsunori Hashimoto.
\newblock Observational scaling laws and the predictability of language model performance, 2024.
\newblock URL \url{https://arxiv.org/abs/2405.10938}.
\bibitem[Shao et~al.(2024)Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, and Guo]{deepseek-math}
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.~K. Li, Y.~Wu, and Daya Guo.
\newblock Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
\newblock URL \url{https://arxiv.org/abs/2402.03300}.
\bibitem[Shenfeld et~al.(2025)Shenfeld, Pari, and Agrawal]{rl_razor}
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal.
\newblock Rl's razor: Why online reinforcement learning forgets less, 2025.
\newblock URL \url{https://arxiv.org/abs/2509.04259}.
\bibitem[Sheng et~al.(2024)Sheng, Zhang, Ye, Wu, Zhang, Zhang, Peng, Lin, and Wu]{verl}
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru~Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
\newblock Hybridflow: A flexible and efficient rlhf framework.
\newblock \emph{arXiv preprint arXiv: 2409.19256}, 2024.
\bibitem[Sun et~al.(2025)Sun, Zhou, Wang, Li, Dziri, and Song]{climb_ladder}
Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song.
\newblock Climbing the ladder of reasoning: What llms can-and still can't-solve after sft?, 2025.
\newblock URL \url{https://arxiv.org/abs/2504.11741}.
\bibitem[Swamy et~al.(2025)Swamy, Choudhury, Sun, Wu, and Bagnell]{roadsleadlikelihoodvalue}
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei~Steven Wu, and J.~Andrew Bagnell.
\newblock All roads lead to likelihood: The value of reinforcement learning in fine-tuning, 2025.
\newblock URL \url{https://arxiv.org/abs/2503.01067}.
\bibitem[Tian et~al.(2025)Tian, Zhao, Wang, Chen, Peng, Ji, Zhao, and Li]{amthink40m}
Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Yunjie Ji, Han Zhao, and Xiangang Li.
\newblock Deepdistill: Enhancing llm reasoning capabilities via large-scale difficulty-graded data training, 2025.
\newblock URL \url{https://arxiv.org/abs/2504.17565}.
\bibitem[Tong et~al.(2024)Tong, Zhang, Wang, Wu, and He]{tong2024dartmath}
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He.
\newblock Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving.
\newblock \url{https://arxiv.org/abs/2407.13690}, 2024.
\newblock URL \url{https://arxiv.org/abs/2407.13690}.
\newblock arXiv:2407.13690, cs.CL.
\bibitem[Vattikonda et~al.(2025)Vattikonda, Ravichandran, Pe{\~{n}}aloza, Nekoei, Thakkar, de~Chezelles, Gontier, Mu{\~{n}}oz-M{\'{a}}rmol, Shayegan, Raimondo, Liu, Drouin, Charlin, Pich{\'{e}}, Lacoste, and Caccia]{Vattikonda2025WebAgent}
Dheeraj Vattikonda, Santhoshi Ravichandran, Emiliano Pe{\~{n}}aloza, Hadi Nekoei, Megh Thakkar, Thibault Le~Sellier de~Chezelles, Nicolas Gontier, Miguel Mu{\~{n}}oz-M{\'{a}}rmol, Sahar~Omidi Shayegan, Stefania Raimondo, Xue Liu, Alexandre Drouin, Laurent Charlin, Alexandre Pich{\'{e}}, Alexandre Lacoste, and Massimo Caccia.
\newblock How to train your {LLM} web agent: A statistical diagnosis.
\newblock \emph{arXiv preprint arXiv:2507.04103}, 2025.
\newblock \doi{10.48550/arXiv.2507.04103}.
\newblock URL \url{https://arxiv.org/abs/2507.04103}.
\bibitem[Wang et~al.(2025)Wang, Cheng, Peng, Bao, Li, Guo, Li, Zeng, Zhou, and Qiu]{sft_dpo_unify}
Bo~Wang, Qinyuan Cheng, Runyu Peng, Rong Bao, Peiji Li, Qipeng Guo, Linyang Li, Zhiyuan Zeng, Yunhua Zhou, and Xipeng Qiu.
\newblock Implicit reward as the bridge: A unified view of sft and dpo connections, 2025.
\newblock URL \url{https://arxiv.org/abs/2507.00018}.
\bibitem[Yan et~al.(2025)Yan, Li, Hu, Wang, Cui, Qu, Cheng, and Zhang]{yan2025learningreasonoffpolicyguidance}
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu~Cheng, and Yue Zhang.
\newblock Learning to reason under off-policy guidance, 2025.
\newblock URL \url{https://arxiv.org/abs/2504.14945}.
\bibitem[Yang et~al.(2024)Yang, Zhang, Hui, Gao, Yu, Li, Liu, Tu, Zhou, Lin, Lu, Xue, Lin, Liu, Ren, and Zhang]{qwen2_5_math}
An~Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang.
\newblock Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.
\newblock \emph{arXiv preprint arXiv:2409.12122}, 2024.
\bibitem[Yang et~al.(2025)Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang, Lv, Zheng, Liu, Zhou, Huang, Hu, Ge, Wei, Lin, Tang, Yang, Tu, Zhang, Yang, Yang, Zhou, Zhou, Lin, Dang, Bao, Yang, Yu, Deng, Li, Xue, Li, Zhang, Wang, Zhu, Men, Gao, Liu, Luo, Li, Tang, Yin, Ren, Wang, Zhang, Ren, Fan, Su, Zhang, Zhang, Wan, Liu, Wang, Cui, Zhang, Zhou, and Qiu]{qwen3}
An~Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le~Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu~Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
\newblock Qwen3 technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.09388}.
\bibitem[Ye et~al.(2025)Ye, Huang, Xiao, Chern, Xia, and Liu]{limo}
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu.
\newblock Limo: Less is more for reasoning, 2025.
\newblock URL \url{https://arxiv.org/abs/2502.03387}.
\bibitem[Yoshihara et~al.(2025)Yoshihara, Yamaguchi, and Inoue]{practicaltwostagerecipemathematical}
Hiroshi Yoshihara, Taiki Yamaguchi, and Yuichi Inoue.
\newblock A practical two-stage recipe for mathematical llms: Maximizing accuracy with sft and efficiency with reinforcement learning, 2025.
\newblock URL \url{https://arxiv.org/abs/2507.08267}.
\bibitem[Yu et~al.(2023)Yu, Jiang, Shi, Yu, Liu, Zhang, Kwok, Li, Weller, and Liu]{yu2023metamath}
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu~Zhang, James~T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
\newblock Metamath: Bootstrap your own mathematical questions for large language models.
\newblock \emph{arXiv preprint arXiv:2309.12284}, 2023.
\bibitem[Yu et~al.(2025)Yu, Zhang, Zhu, Yuan, Zuo, Yue, Dai, Fan, Liu, Liu, Liu, Lin, Lin, Ma, Sheng, Tong, Zhang, Zhang, Zhang, Zhu, Zhu, Chen, Chen, Wang, Yu, Song, Wei, Zhou, Liu, Ma, Zhang, Yan, Qiao, Wu, and Wang]{Yu2025DAPO}
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu~Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei{-}Ying Ma, Ya{-}Qin Zhang, Lin Yan, Mu~Qiao, Yonghui Wu, and Mingxuan Wang.
\newblock {DAPO}: An open-source llm reinforcement learning system at scale.
\newblock \emph{arXiv preprint arXiv:2503.14476}, 2025.
\newblock \doi{10.48550/arXiv.2503.14476}.
\newblock URL \url{https://arxiv.org/abs/2503.14476}.
\bibitem[Zhang et~al.(2025{\natexlab{a}})Zhang, Zhang, Liu, Jin, Yang, Zheng, Liu, and Guo]{d3}
Jia Zhang, Chen-Xi Zhang, Yao Liu, Yi-Xuan Jin, Xiao-Wen Yang, Bo~Zheng, Yi~Liu, and Lan-Zhe Guo.
\newblock D3: Diversity, difficulty, and dependability-aware data selection for sample-efficient llm instruction tuning, 2025{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2503.11441}.
\bibitem[Zhang et~al.(2025{\natexlab{b}})Zhang, Xie, Sun, Chen, Wang, Li, Ding, and Zhou]{chord-d}
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou.
\newblock On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting, 2025{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2508.11408}.
\bibitem[Zhang et~al.(2025{\natexlab{c}})Zhang, Li, Long, Zhang, Lin, Yang, Xie, Yang, Liu, Lin, Huang, and Zhou]{qwen3embedding}
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An~Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou.
\newblock Qwen3 embedding: Advancing text embedding and reranking through foundation models.
\newblock \emph{arXiv preprint arXiv:2506.05176}, 2025{\natexlab{c}}.
\bibitem[Zhao et~al.(2025)Zhao, Wang, Peng, Zhao, Tian, Chen, Ji, and Li]{amthink1_4m}
Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li.
\newblock 1.4 million open-source distilled reasoning dataset to empower large language model training, 2025.
\newblock URL \url{https://arxiv.org/abs/2503.19633}.
\end{thebibliography}