Minimax Rates and Spectral Distillation for Tree Ensembles
Extracted source text · arXiv 2605.11841
\documentclass{article}
\PassOptionsToPackage{numbers, compress}{natbib}
\usepackage[preprint]{neurips_2026}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% Also
\usepackage{rotating}
\usepackage{multirow}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{hyperref}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{xfrac}
\usepackage{mathtools}
\usepackage{wrapfig}
\usepackage{tikz}
\usepackage{algorithm}
\usepackage{algorithmic}
\def\checkmark{\tikz\fill[scale=0.4](0,.35) -- (.25,0) -- (1,.7) -- (.25,.15) -- cycle;}
\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\DeclarePairedDelimiterX{\infdivx}[2]{(}{)}{%
#1\parallel #2%
}
\newcommand{\infdiv}{D\infdivx}
\newcommand{\kldiv}{D_\mathit{KL}\infdivx}
\usepackage{cancel}
\usepackage{adjustbox}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{conjecture}[theorem]{Conjecture}
\usepackage[american]{babel}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{url}
\usepackage{bm}
\usepackage{cancel}
\usepackage{comment}
\usepackage{natbib}
\usepackage{enumitem}
\usepackage{etoolbox}
\usepackage{subcaption}
\BeforeBeginEnvironment{wrapfigure}{\setlength{\intextsep}{0pt}}
\newcommand{\notedw}[1]{{\color{purple}[#1]}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\indep}{\perp\!\!\!\perp}
\DeclareMathOperator{\dep}{\not\! \perp\!\!\!\perp}
\setlength{\belowcaptionskip}{-10pt}
\title{Minimax Rates and\\Spectral Distillation for Tree Ensembles}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Binh Duc Vu\\
King's College London\\
\texttt{binh.vu@kcl.ac.uk}
\And
David S. Watson\\
King's College London\\
\texttt{david.watson@kcl.ac.uk}
}
\begin{document}
\maketitle
\input{main}
\bibliographystyle{plainnat}
\small{
\bibliography{references}
}
\input{appx}
%\input{checklist.tex}
\end{document}
\begin{abstract}
Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive minimax-optimal convergence for RF regression, showing that, under mild regularity conditions on tree growth, the eigenvalue decay of the induced kernel operator governs the statistical rate. Second, we exploit this spectral viewpoint to develop compression schemes for tree ensembles. For RFs, leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, leading singular vectors of the smoother matrix play an analogous role. Learning nonlinear maps for these spectral representations yields distilled models that are orders of magnitude smaller than the originals while maintaining competitive predictive performance.
Our methods compare favorably to state of the art algorithms for forest pruning and rule extraction, with applications to resource constrained computing.
\end{abstract}
\section{Introduction}\label{sec:intro}
% While supervised machine learning is rapidly evolving---largely driven by advances in deep learning---classical tree ensembles like random forests (RFs) and gradient boosting machines (GBMs) remain highly popular and versatile. This comes as a result of their ease-of-use, relative speed, and transparency, all while maintaining competitive results in tabular data processing tasks \citep{grinsztajn2022why, shwartz_tabular}, and are widely deployed in diverse fields including econometrics \citep{Athey2019}, bioinformatics \citep{li2022}, and engineering \citep{shoar2022}.
Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most popular and versatile supervised learning algorithms. Fast and easy to use, trees regularly attain state of the art results in tabular data tasks \citep{grinsztajn2022why, shwartz_tabular}, often beating out deep learning models that have proven more effective for image and language processing.
Trees are especially good at adapting to sparse signals, making them a common choice in high-dimensional domains where parametric methods may struggle, such as bioinformatics \citep{li2022}, econometrics \citep{Athey2019}, and engineering \citep{shoar2022}.
Despite their prevalence, tree ensembles remain poorly understood at a theoretical level. For RFs, most authors focus on idealized variants such as purely random forests \citep{breiman2000, genuer2012, biau_analysis, klusowski2021}, which make analysis more tractable but sacrifice the adaptive splitting that drives empirical performance.
% \citep{denil2014, Biau2016, scornet_hooker_2025}
Finite sample generalization guarantees remain rare, and existing results do little to explain the effectiveness of standard greedy splits. For GBMs, the situation is arguably worse: while boosting theory is well developed in the classification setting \citep{schapire2012, Buhlmann01062003, bartlett2004}, the specific inductive bias of tree-based gradient boosting---and in particular, how ensemble size interacts with tree complexity---lacks a comparably sharp characterization.
We adopt a spectral perspective that addresses both algorithms. For RFs, we work in the kernel tradition \citep{breiman2000, davies_rf_kernel, scornet_kernel}, showing that the eigenvalue decay of the integral operator induced by the RF kernel governs the statistical rate. Our main theoretical result is minimax optimality for an algorithm that closely approximates the original method of \citet{Breiman2001}, affirmatively resolving a long open question in the literature. The proof sheds light on how greedy splits inherently adapt to sparse signals, connecting algorithmic hyperparameters to the spectral geometry of the induced kernel.
Taking this spectral viewpoint, we observe that the principal eigenvectors of the RF kernel operator capture the most informative dimensions of the model.
GBMs, by contrast, do not admit a kernel representation, but an analogous compression scheme is achievable via the singular value decomposition of its implicit smoother matrix \citep{curth2023_double_descent}. In both cases, models can be distilled by learning the nonlinear mapping from the input space to an appropriate spectral representation. Since a few leading modes capture most of the problem's variance, the resulting models maintain competitive performance while being significantly smaller and faster than the originals, thus enabling deployment on resource-constrained devices where full ensembles are infeasible. Crucially, while previous methods require users to set an accuracy tolerance---leaving the final model size unpredictable---our method allows the size to be set explicitly. This provides a practical solution for deployment with strict memory budgets while remaining competitive on the empirical rate-distortion Pareto frontier.
%This spectral viewpoint extends naturally to a practical distillation scheme for tree ensembles. For RFs, the leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, the analogous role is played by leading singular vectors of the smoother matrix. In both cases, learning a nonlinear map from the input space to these spectral representations yields highly compressed models that maintain competitive predictive performance. The resulting distilled ensembles are orders of magnitude smaller than their parents, enabling deployment on resource-constrained devices where full ensembles are infeasible.
The remainder of this paper is structured as follows. We review related work in Sect. \ref{sec:related}. Following a brief summary of our setup and assumptions in Sect. \ref{sec:setup}, we state our main theoretical result in Sect. \ref{sec:main}.
This motivates a distillation scheme for tree ensembles, which we describe in Sect. \ref{sec:compression}.
Empirical results are presented in Sect. \ref{sec:exp}, including extensive benchmarks against state of the art methods. A brief discussion follows in Sect. \ref{sec:discussion}. Sect. \ref{sec:conclusion} concludes.
\section{Related Work}\label{sec:related}
\paragraph{RF Theory}
For good overviews of RF theory, see \citep{Biau2016, scornet_hooker_2025}. Early work in this area tended to prioritize asymptotic consistency results \citep{breiman2004, Meinshausen2006, biau_consistency, biau2010, genuer2012, Scornet2015}, often under simplifying assumptions.
% More recently, the focus has shifted to statistical inference \citep{mentch_hooker, Wager2018, Athey2019, Peng2022}.
Convergence rates are generally harder to come by, though some authors have established PAC-Bayes bounds for RFs \citep{lorenzen_PACbayes} and finite sample results for classification \citep{gao2020, gao2022}.
Minimax convergence, a common optimality target in nonparametric regression, has so far proven elusive.
Suboptimal rates have been derived for non-adaptive \citep{klusowski2021} and adaptive RFs \citep{Klusowski2024}.
An online variant of the RF algorithm known as Mondrian forests \citep{mondrian_forest} are known to be minimax optimal \citep{mourtada2017, mourtada2020}, as are so-called tessellation forests \citep{oreilly2024}, which use oblique splits.
However, the split mechanism in both cases is non-adaptive, and so these results do little to explain the performance of standard RFs, which clearly benefit from their greedy optimization objective.
Our theoretical approach differs markedly from previous work in this area. Classical minimax results assume that the sampling distribution is dominated by some reference measure over the input space $\mathbb R^d$, giving rates that scale poorly with data dimensionality \citep{tsybakov2008}.
However, in high-dimensional settings, data are often concentrated on a low-dimensional manifold \citep{bengio2013}. Algorithms that can adapt to such sparse structures often converge far faster than theory would suggest.
Building on this insight, several authors have taken a spectral approach to kernel regression, deriving novel minimax rates under a source condition and assumptions on eigenvalue decay \citep{caponetto2007, blanchard2020, rastogi2023, zhang2025}. We adapt these results to the RF context, establishing the minimax optimality of an adaptive RF algorithm.
\paragraph{Ensemble compression}
% Post-training tree ensemble compression can be split into two paradigms: pruning-based structural optimization and model distillation.
% \textbf{Pruning} compresses ensembles by directly reducing the original forest's architecture. Classical methods rely on coarse, tree-level removal (LRL1) or post-hoc global leaf refinement. Modern techniques optimize internal tree structures, but frequently impose rigid constraints, to ensure minimal loss of accuracy.
Pruning methods for tree ensembles selectively remove splits or entire trees from the forest.
This tradition goes back to the original CART algorithm, where truncating depth was proposed as a form of tree regularization \citep{breiman1984}.
Numerous ensemble variants have been developed since \citep{Tsoumakas2009}, notably including Leaf Refinement with $L_1$ pruning (LRL1) \citep{busch2023}, which aims to drop redundant trees; and
ForestPrune (FP) \citep{liu2023}, which optimizes tree depth on a per-tree basis.
Though FP generally compresses better than LRL1, it produces symmetric trees by default, which may be suboptimal in some settings.
Building on these methods, \citet{devos2025} propose Level-wise Optimization and Pruning (LOP), which traverses the ensemble by tree depth, optimizing a global $L_1$-regularized objective to simultaneously prune subtrees and update leaf predictions. These methods generally fix an error tolerance upfront and minimize model size subject to this budget.
% While LOP represents the current state-of-the-art in ensemble pruning, these methods generally rely on bounding accuracy, which makes explicit model size control impossible; the output model size depends entirely on the problem's inherent compressibility.
Distillation approaches, by contrast, typically fix a target size upfront and minimize error subject to a budget on the compression rate.
% On the other hand, \textbf{Distillation} approaches compression by abandoning or heavily deconstructing the original forest to train a smaller model.
% A parallel line of research focuses on distillation, which abandons or heavily deconstructs the original forest's structure to train a smaller model.
The method originated in deep learning, where knowledge distillation involves training a ``student'' network to mimic the soft labels of a larger ``teacher'' network \citep{hinton_distilling, hinton_distilling2}.
The approach has been adapted to tree ensembles.
\citet{zhou2016sat} use the parent ensemble as an oracle to generate pseudo-samples and stabilize the decision boundaries of a single surrogate tree, while \citet{vidal2020born} develop a dynamic program to construct a minimal decision tree that perfectly reproduces the original ensemble's predictions. Such exact distillation of a large forest into a single tree is NP-hard, and quickly becomes infeasible as dimensionality increases. % and therefore infeasible for inherently bottlenecks the model's representational capacity and usually incurs a prohibitive computational training costs.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/pipeline.pdf}
\caption{Summary of our pipeline. (a) Start with tabular training data for supervised learning (regression or classification). (b) Train a RF or GBM. (c) A kernel matrix $\mathbf{K}$ or smoother matrix $\mathbf{S}$ is extracted from the ensemble. (d) Compute the spectrum of the operator. (e) Under fast decay, RFs converge at the minimax-optimal rate. (f) The base ensemble is distilled by SCATE, which approximates model predictions by learning the top spectral directions.}
\label{fig:pipeline}
\end{figure*}
A related but distinct line of work aims to produce smaller, more interpretable models via rule extraction.
Prominent early examples include RuleFit \citep{friedman2008} and Node Harvest \citep{meinshausen2010}, which select the most informative splits via lasso penalties and constrained quadratic programming, respectively.
More recent methods, such as SIRUS \citep{benard2021, benard2} and FIRE \citep{liu2023fire, liu2025extracting}, generally produce stabler results.
% Building on this, \citet{liu2025extracting} introduce a novel framework that surpasses these prior approaches by jointly controlling both the number of extracted rules and their individual interaction depths, outperforming the methods listed above.
The SSF algorithm extracts nodes into isolated decision stumps \citep{alkhoury2024splitting}, while MicroForest \citep{yoo2025microforest} and MILP-based alternatives \citep{bonasera2026} reassemble high information-gain nodes from multiple source forests into a new architecture.
Although these methods enable fast, memory-efficient prediction on constrained devices, they are generally optimized for interpretability over performance, as we confirm in our experiments.
% their node-extraction techniques are inherently structure-destructive, discarding conditional parent-node hierarchies.
\section{Background}\label{sec:setup}
Consider the standard regression setting, with data $\mathcal D_N = \{\bm x_i, y_i\}_{i=1}^N$ sampled i.i.d. from the joint probability measure $\mu$ over the feature space $\mathcal X \subseteq \mathbb R^d$ and label space $\mathcal Y \subseteq \mathbb R$. The goal is to estimate the conditional expectation $f^*(\bm x) = \mathbb E[Y \mid \bm x]$.
% We do not necessarily assume additive errors or constant conditional variance.
The CART algorithm \citep{breiman1984} approximates $f^*$ by learning a recursive partition of $\mathcal X$. Axis-aligned splits are selected on a greedy basis to minimize a loss function over the training set $\mathcal D_N$, typically mean squared error for continuous $Y$. Splits continue until either samples are exhausted or some stopping criterion is met, e.g. target depth. Terminal nodes are labeled by taking leaf-wise averages of $Y$. This procedure is universally consistent under common regularity assumptions \citep{devroye1996, gyorfi2002}.
Despite their attractive asymptotic properties, trees can be high-variance predictors in practice.
RFs overcome this by taking an ensemble of independently randomized CART basis functions. GBMs, on the other hand, learn in a sequential manner, with each tree improving on the performance of the last.
Both models admit natural interpretations as \textit{smoothers} \citep{gams1990}, i.e. supervised learners of the form
\begin{align}\label{eq:smooth}
f(\bm x_0) = \sum_{i=1}^N ~w_i(\bm x_0) ~y_i,
\end{align}
where $\bm x_0 \in \mathcal X$ is an arbitrary point and $i \in [N]$ indexes the training data. Smoothers make predictions by taking linear combinations of training labels, with weights given by a potentially nonlinear smoothing function $w: \mathcal X \mapsto \mathbb R^N$. The differences between RFs and GBMs can be fully explained by their respective smoothing functions.
\paragraph{Random Forests}
In the case of RFs, the smoother is a kernel function \citep{breiman2000, davies_rf_kernel, scornet_kernel}.
Each tree induces a partition of $\mathcal X$ into hyperrectangular cells $\{\mathcal X_\ell\}_{\ell=1}^M$, one for each leaf.
Let $A(\cdot; b): \mathcal X \mapsto \{\mathcal X_\ell\}_{\ell=1}^M$ be a map from samples to cells for a given tree $b$.
The partition defines an indicator kernel that reports whether a pair of feature vectors are routed to the same leaf: $\hat k^{(b)}(\bm x, \bm x') = \bm 1 \big\{ A(\bm x; b) = A(\bm x'; b) \big\}$.
%\begin{align*}
% \hat k^{(b)}(\bm x, \bm x') = \bm 1 \big\{ A(\bm x; b) = A(\bm x'; b) \big\}.
%\end{align*}
Normalizing over training data and averaging across trees, we derive the empirical RF kernel:
\begin{align*} %\label{eq:rf_kernel}
\hat k_N^B(\bm x, \bm x') = \frac{1}{B} \sum_{b=1}^B \Big( \frac{\hat k^{(b)}(\bm x, \bm x')}{\sum_{i=1}^N \hat k^{(b)}(\bm x, \bm x_i)} \Big).
\end{align*}
This kernel provides the weights of Eq. \ref{eq:smooth} for RFs.
The resulting kernel matrix $\hat{\bm K} \in [0,1]^{N \times N}$ is symmetric, PSD, and doubly stochastic.
Because RF splits are adaptively selected to minimize a loss function, the kernel is not independent of outcomes $Y$. Still, the resulting RKHS is asymptotically universal and characteristic \citep{vu_RFAE}.
The population RF kernel has corresponding integral operator $\bm K: L^2(\mu_X) \mapsto L^2(\mu_X)$, defined by:
\begin{align*}
(\bm Kh)(\cdot) = \int_\mathcal X k(\cdot, \bm x) ~h(\bm x) ~d \mu_X(\bm x).
\end{align*}
% Because $\bm K$ is self-adjoint, compact, and positive on $L^2(p_X)$, the spectral theorem applies.
We denote the operator's eigendecomposition as $(\psi_i, \lambda_i)_{i \geq 1}$.
% Of course, the integral operator and eigenpairs inherit the kernel's dependence on $Y$.
We use subscripts to denote the number of leaves used to construct the trees over which the operator averages. For example, $\bm K_2$ denotes the operator that would result from an infinite ensemble of decision stumps.
\paragraph{Gradient Boosting}
GBMs learn sequentially, with each basis function fit to the residuals of the previous ensemble. Consider gradient boosting with learning rate $\eta \in (0, 1]$ and $L_2$ loss. The algorithm initializes $f_0(\bm x) = 0$ and iterates, at each round $b \in [B]$, the following steps:
\begin{enumerate}[noitemsep]
\item Fit a regression tree $\tilde{f}_b$ to the current residuals $\{(\bm x_i, y_i - f_{b-1}(\bm x_i))\}_{i=1}^N$.
\item Update $f_b(\bm x) \gets f_{b-1}(\bm x) + \eta \tilde{f}_b(\bm x)$.
\end{enumerate}
The final prediction is $\hat{f}_{\mathrm{GBM}}(\bm x_0) = f_B(\bm x_0)$. Smoothing weights must be constructed recursively to account for the sequential residual fitting.
In the first round, $\tilde{f}_1$ is a standard regression tree with indicator weights $\hat{s}_{\text{tree},1}(\bm x_0) = \hat k^{(1)}(\bm x_0, \cdot)$, giving $\hat{s}_{\text{boost},1}(\bm x_0) = \eta \, \hat{s}_{\text{tree},1}(\bm x_0)$. For subsequent rounds, the residual tree $\tilde{f}_b$ predicts leaf-wise averages of $y_i - f_{b-1}(\bm x_i)$, which can be decomposed into a standard tree prediction minus a correction for the accumulated ensemble predictions. Concretely, let $\{\mathcal X_\ell\}_{\ell=1}^{M}$ denote the leaves of the $b$th tree, and write $\bm{e}_{b}(\bm x_0) = \big(\bm 1\{\bm x_0 \in \mathcal X_1\}, \ldots, \bm 1\{\bm x_0 \in \mathcal X_{M}\}\big)$ for the leaf indicator vector. Then the prediction of the $b$th residual tree at $\bm x_0$ can be written as:
\[
\tilde{f}_b(\bm x_0) = \bigl(\hat{s}_{\mathrm{tree},b}(\bm x_0) - \hat{s}_{\mathrm{corr},b}(\bm x_0)\bigr) \bm{y},
\]
where $\hat{s}_{\mathrm{tree},b}(\bm x_0)$ are the standard tree weights for the $b$th tree and $\hat{s}_{\mathrm{corr},b}(\bm x_0) = \bm{e}_{b}(x_0) \hat{R}_b$ is a correction term, with $\hat{R}_b \in \mathbb{R}^{M_b \times N}$ the leaf-residual correction matrix whose $\ell$th row is the average of $\hat{s}_{\mathrm{boost},b-1}(\bm x_i)$ over training points $\bm x_i$ falling in leaf $\ell$. The GBM smoothing weights therefore satisfy the recursion:
\[
\hat{s}_{\mathrm{boost},b}(x_0) = \hat{s}_{\mathrm{boost},b-1}(\bm x_0) + \eta \bigl(\hat{s}_{\mathrm{tree},b}(\bm x_0) - \hat{s}_{\mathrm{corr},b}(\bm x_0)\bigr),
\]
with $\hat{s}_{\mathrm{boost},0}(\bm x_0) = \bm{0}$.
The function $\hat{s}_{\mathrm{boost},B}$ provides the smoothing weights of Eq. \ref{eq:smooth} for GBMs. Stacking over all training points yields the empirical smoother matrix $\hat{\bm S} \in \mathbb R^{N \times N}$.
Unlike the RF kernel operator $\bm K$, the GBM smoother $\bm S$ is generally neither symmetric nor PSD. This precludes a direct eigendecomposition of the kind used for RFs. However, the singular value decomposition $\bm S = \bm U \bm \Sigma \bm V^\top$ provides a natural spectral representation: the leading right singular vectors of $\bm S$ identify the directions in label space that the ensemble amplifies most, while the corresponding left singular vectors describe the associated structure in prediction space.
% This decomposition motivates the GBM compression scheme described in Sect.\ \ref{sec:compression}.
\section{Minimax Convergence}\label{sec:main}
% With fixed $\mu, \nu$, we say that one operator is a \textit{refinement} of another if it contains more leaves, since the trees of $\bm K_3$ contain an extra recursive partition of the trees of $\bm K_2$.
% One way of stating the universal consistency property of RFs is to say that, under the right tree-growing mechanisms $\nu$, for any $\mu$ we have $\lim_{M \rightarrow \infty} \bm K_M = \bm I$.
In this section, we present a novel minimax convergence result for RF regression. To the best of our knowledge, this constitutes the first such result for any adaptive variant of the algorithm.
We make the following regularity assumptions on tree growth:
\begin{itemize}[noitemsep]
\item[(A1)] \textit{Honesty}. Training data is partitioned into two disjoint folds: $\mathcal D_{\text{split}}$ for learning splits, and $\mathcal D_{\text{label}}$ for labelling leaves.
\item[(A2)] \textit{Subsampling}. Tree sample size satisfies $N^{(b)} \leq N$, with $N^{(b)} \rightarrow \infty$ as $N \rightarrow \infty$.
\item[(A3)] \textit{Inclusion}. The probability of splitting on any feature at any node is at least $\rho > 0$.
\item[(A4)] \textit{Balance}. Each split places a fraction $\gamma > 0$ of the available observations in each child node.
% \item[(A5)] \textit{Depth}. Each tree's number of leaves $M^{(b)}$ satisfies $M^{(b)} \rightarrow \infty, M^{(b)} / N \rightarrow 0$ as $N \rightarrow \infty$.
\end{itemize}
Assumptions (A1)-(A4) are common in the RF literature \citep{Meinshausen2006, scornet2016_asymptotics, Biau2016, Athey2019}. They represent a slight idealization of \citet{Breiman2001}'s classic algorithm, which does not rely on honest splitting and trains trees on bootstraps rather than subsamples.
Crucially, (A1)-(A4) do not sacrifice the adaptive nature of RFs, which is essential to their success.
We make the following assumption on the data generating process (DGP):
\begin{itemize}
\item[(B1)] \textit{Bounds}. The feature space is compact on the unit hypercube, $\mathcal X = [0,1]^d$, with marginal measure $\mu_X$ bounded away from $0$ and $\infty$, and outcomes on the unit interval $\mathcal Y = [0,1]$.
\end{itemize}
The bounds of (B1) are primarily for convenience; restrictions on $\mu_X$ avoid unwanted degeneracies.
Our remaining assumptions characterize the source class.
All $L^2$ norms and inner products from here on out are taken w.r.t. $\mu_X$ unless otherwise specified.
We write $\bar{\bm K}_M := \mathbb E_\nu[\hat{\bm K}_M \mid \mathcal D_{\text{split}}]$ to denote the expected kernel with $M$ leaves given the split data, with randomness over the tree generating mechanism $\nu$.
\begin{itemize}[noitemsep]
\item[(C1)] \textit{Source condition}. There exists some exponent $\alpha \in (0, 1]$, leaf count $q \in \mathbb N$, and function $g \in L^2$ with $\lVert g \rVert_{L^2} < \infty$, such that $f^* = \bm K_q^\alpha g$.
\item[(C2)] \textit{Polynomial decay}. The operator $\bm K_q$ has eigenvalues $\lambda_i \asymp i^{-\beta}$, for some $\beta > 1$.
\item[(C3)] \textit{Approximation}. For all $M \geq q$: $\|(\bm{I} - \bm{K}_M)\bm{K}_q^\alpha\|_{op} \lesssim M^{-\alpha\beta}$.
%\begin{align*}
% \sum_{i=1}^M \lVert (\bm I - \bm K_M) \psi_i \rVert_{L^2}^2 \lesssim M^{-2 \alpha \beta}.
%\end{align*}
\end{itemize}
(C1) is a standard source condition from inverse problems \citep{rastogi2023}, expressing smoothness of $f^*$ relative to the RF kernel operator.
% Note that the integral operator $\bm K_m$ depends on both the randomness of the DGP $\mu$ and the algorithm $\nu$, a point we revisit below.
(C2) imposes a compressibility assumption on the RF kernel, which posits that leading eigenfunctions are much more informative than the rest.
Some authors regard polynomial eigenvalue decay as a strong assumption to make in general \citep{blanchard2020, zhang2025}.
% , we find that it often understates the true rate in real world data.
% In fact, as our experiments below demonstrate, we occasionally observe \textit{exponential} spectral decay, implying strictly faster convergence rates.
% We focus on the polynomial case, as it yields classic minimax behavior.
However, (C2) can be empirically evaluated for any given dataset; our analysis on C2 in Appx. \ref{appx:c2}, demonstrate that it is a sufficient but not necessary condition for strong RF performance.
(C3) says that RF kernels approximate the identity at a bounded rate on the source class as depth increases. This is a substantive assumption, but arguably natural given (C1) and (C2): if $f^*$ is smooth relative to the kernel and the kernel's eigenvalues decay polynomially, it is reasonable to expect that deeper trees recover $f^*$ at a commensurate rate.\footnote{In some works of RF theory, $\mu_X$ is presumed uniform (e.g., \citep{Wager2018}). In this case, (C3) would follow from (A1)-(A4) and (B1). We conjecture that a similar result extends to bounded densities, but defer a formal proof to future work.}
%For more details on these assumptions and an empirical evaluation of (C2), see Appx. \ref{appx:c2}.
\begin{comment}
Finally, we make the following assumptions on the interaction between trees and our DGP:
\begin{itemize}
\item[(C1)] \textit{Curvature}. For every feature at each split, the risk reduction functional has second derivative at its maximizer bounded from above by some $\kappa < 0$.
\end{itemize}
This is a technical condition that ensures optimal split thresholds vary smoothly under perturbations.
\end{comment}
With these elements in place, we state our main result. See Appx. \ref{appx:proofs} for all proofs.
\begin{theorem}[Minimax convergence]\label{thm:minimax}
Assume (A1)-(A4), (B1), and (C1)-(C3).
Suppose that $\alpha \beta > 1/2$.
Then with $M \asymp N^{1 / (2 \alpha \beta + 1)}$ leaves per tree and $B=\infty$ trees, we have:
\begin{align*}
\mathbb E \Big[\|f^* - \hat f\|_{L^2}^2 \Big] \lesssim N^{- 2\alpha \beta / (2\alpha \beta + 1)}.
\end{align*}
This rate is minimax optimal over the source class defined by (C1)-(C3).
\end{theorem}
% For finite $B$, the bound acquires an additional $\mathcal O(1/B)$ term, which is negligible with just a few hundred trees. Whereas classical nonparametric theory tends to parameterize convergence in terms of smoothness and data dimensionality \citep{stone1982, tsybakov2008}, more recently authors have sought to derive rates in terms of the spectral regularity of a source class \citep{blanchard2020, rastogi2023, zhang2025}. This approach is well suited to RFs, which are known to naturally adapt to sparse signals. When the true regression function lies on a low-dimensional manifold, as (C2) suggests, RFs perform remarkably well even with very high ambient dimension, e.g. in genomics \citep{goldstein2011random, Chen2012}. While others have studied the performance of RFs in sparse regression settings \citep{Scornet2015}, we relax the additivity assumptions of previous works and derive minimax rates for adaptive kernels under a testable compressibility assumption. The cost for this generality is an extra eigenbasis stability assumption (C3), which we argue is well-motivated by the commutativity of tree-level integral operators.
\begin{comment}
We can potentially use these insights to learn a compressed distillation of the model $\hat f^B_N$. Let $\phi: \mathcal X \mapsto \hat{\mathcal H}_k$ be the canonical feature map, truncated to reflect finite $N, B$.
Let $\psi: \mathcal X \mapsto \mathbb R^N$ be the projection of inputs onto empirical kernel eigenvectors. Then we have several equivalent ways of representing the kernel matrix over training inputs:
\begin{align*}
\bm K &= \phi \phi^{\top}\\
&= \phi(\bm X) \phi(\bm X)^{\top}\\
&= \bm V \bm \Lambda \bm V^{\top}\\
&= \psi(\bm X) \bm \Lambda \psi(\bm X)^{\top}.
\end{align*}
Recall that model predictions can be written as:
\begin{align*}
\hat f^B_N(\bm X) = \bm K \bm y,
\end{align*}
where $\bm y$ denotes the length-$N$ vector of training labels. For test point $\bm x_0$, the model's prediction is:
\begin{align*}
\hat f^B_N(\bm x_0) = \bm k_0^{\top} \bm y,
\end{align*}
where
\begin{align*}
\bm k_0 &= \big[k_N^B(\bm x_0, \bm x_1), \dots, k_N^B(\bm x_0, \bm x_N) \big]\\
&= \phi \phi(\bm x_0)^{\top}\\
&= \bm V \bm \Lambda \psi(\bm x_0)^{\top}.
\end{align*}
Putting all the pieces together, we have:
\begin{align*}
\hat f^B_N(\bm x_0) &= \bm k_0^{\top} \bm y\\
&= \big( \bm V \bm \Lambda \psi(\bm x_0) \big)^{\top} \bm y\\
&= \psi(\bm x_0)^{\top} \bm \Lambda \bm V^{\top} \bm y.
\end{align*}
So: if we store the eigendecomposition of $\bm K$ up to rank $m$ and learn the function $\psi_m: \mathcal X \mapsto \mathbb R^m$, we will have a compressed version of the RF.
A half-baked idea: can we borrow the strategy of info theory/PAC-Bayes and get results for some variational prior (e.g., the Laplace kernel) at the cost of a KL-divergence term...?
%%% DISCUSSION %%%
The story that emerges from this analysis goes something like this: fast spectral decay implies fast convergence. Or, to put it another way: the more compressible the model, the better it generalizes.
The results above highlight a conceptual shift in how minimax rates for random forests should be understood. In classical nonparametric theory, convergence rates are usually expressed in terms of the ambient feature dimension $d$ and the smoothness $\alpha$ of the regression function. For example, if $f^* \in \mathcal H^{\alpha}([0,1]^d)$, the minimax rate is $n^{-2\alpha/(2\alpha+d)}$. In sparse settings, one often replaces $d$ by the number of active coordinates $s < d$, but the analysis still ties performance to the explicit feature dimension.
In contrast, the spectral viewpoint treats the random forest kernel as the primary object of analysis. The effective complexity of the regression estimator is then determined not by the raw input dimension but by the eigenvalue decay rate of the kernel operator. Polynomial eigen-decay $\lambda_i \asymp i^{-p}$ yields rates $N^{-2\alpha/(2\alpha+p)}$ for regression functions of smoothness $\alpha$, which align with the minimax rates for Sobolev function classes. Crucially, $p$ can implicitly capture sparsity, anisotropy, or low-dimensional structure without requiring explicit assumptions on the number of active coordinates.
This perspective emphasizes that the true dimensionality of random forest regression lies not in the input space $\mathcal X$ but in the adaptive operator that governs predictions. The spectral decay parameter $p$ quantifies how efficiently the forest partitions approximate $f^*$ by concentrating resolution along the relevant directions. This yields a more informative characterization of convergence, since the kernel itself naturally adapts to the data distribution and structure of the regression function.
The only unavoidable ingredient is smoothness. Without some control on the regularity of $f^*$, no kernel method can guarantee fast convergence. Within this constraint, however, the spectral approach reveals that RFs are capable of achieving minimax-optimal rates adaptively, bypassing explicit dependence on the ambient feature dimension.
\end{comment}
% minor name change?
% work in progress
% note to self to add: this particular way to compress is good because svd and
% eigenfunction decomposition is the 'optimal' LoRA for a matrix.
\section{Spectral Compression of Adaptive Tree Ensembles (SCATE)}\label{sec:compression}
% SCATE resolves the gap between pruning (lacking size control) and distillation (discarding information about the hierarchy). To our knowledge, SCATE is the first method to distill a discrete tree ensemble directly into a continuous neural network. This continuous representation allows practitioners to explicitly dictate the exact size and memory footprint of the final model via the network's dimensions, overcoming the limitations of bound-reliant methods like LOP. Additionally, translating the ensemble into a neural network replaces tree traversals with dense matrix multiplications, improving inference time on modern hardware.
This theoretical result motivates a spectral model distillation scheme, which we call \textbf{SCATE} (\textbf{S}pectral \textbf{C}ompression of \textbf{A}daptive \textbf{T}ree \textbf{E}nsembles).
SCATE uses a (small) neural network to learn the leading spectral directions of the ensemble's implicit linear operator---the kernel matrix $\hat{\bm{K}}$ for RFs, or the smoother matrix $\hat{\bm{S}}$ for GBMs.
% While this approach applies broadly to GBM architectures, this paper focuses on compressing XGBoost, given its ubiquity and strong performance.
For an RF $\hat{f}_N^B$ trained on $N$ samples, let $\bm{K} \in [0, 1]^{N \times N}$ be the training kernel matrix, with eigenvectors $\bm{\Psi}$ and eigenvalues $\bm{\Lambda} = \text{diag}(\bm{\lambda})$.
Let $\psi: \mathcal{X} \mapsto \mathbb{R}^N$ be a map from input features to $\bm{K}$'s bundled eigenfunctions, treated as a column vector.
Model predictions over the training data can be written as $\hat{f}_N^B(\bm{X}) = \bm{K}\bm{y}$, where $\bm{y}$ is a vector of training labels. For a test point $\bm{x}_0$, predictions rely on localized kernel weights $\bm{k}_0 = \bm{\Psi\Lambda} \psi(\bm{x}_0)$, yielding:
\begin{equation*}
\hat{f}_N^B(\bm{x}_0) = \bm{k}_0^\top \bm{y} = (\bm{\Psi\Lambda}\psi(\bm{x}_0))^\top \bm{y} = \psi(\bm{x}_0)^\top \bm{\Lambda\Psi}^\top \bm{y}.
\end{equation*}
This derivation reveals our compression strategy: we store the eigendecomposition of $\bm{K}$, truncated to rank $P \ll N$, and train a neural network to approximate the top $P$ eigenfunctions $\{\psi_i\}_{i=1}^P$. By defining a fixed coefficient vector $\bm{c} := \bm{\Lambda}_P \bm{\Psi}_P^\top \bm{y} \in \mathbb{R}^P$, test time predictions amount to evaluating:
\begin{equation*}
\hat{f}_N^B(\bm{x}_0) \approx \sum_{j=1}^P c_j \hat{\psi}_j(\bm{x}_0).
\end{equation*}
\begin{comment}
Consider again an RF $\hat{f}_N^B$ consisting of $B$ trees trained on $N$ samples. Let $\phi : \mathcal{X} \mapsto \hat{\mathcal{H}}_N^B$ be the canonical feature map, where $D_\phi$ representing the total number of leaves in the forest. Let $\Psi$ and $\Lambda = \text{diag}(\lambda)$ denote the eigenvectors and eigenvalues of the training kernel matrix $\bm{K} \in [0, 1]^{N \times N}$, respectively. By defining $\psi : \mathcal{X} \mapsto \mathbb{R}^N$ as the map from input features to the bundled eigenfunctions of $\bm{K}$, we can represent the kernel matrix in several equivalent ways:
\begin{align*}
\bm{K} &= \Phi\Phi^\top \quad \text{(Kernel definition)} \\
&= \phi(\bm{X})\phi(\bm{X})^\top \quad \text{(Using feature maps)} \\
&= \Psi\Lambda\Psi^\top \quad \text{(Eigendecomposition)} \\
&= \psi(\bm{X})\Lambda\psi(\bm{X})^\top \quad \text{(Using eigenfunctions)}
\end{align*}
Model predictions over the training data can be written concisely as $\hat{f}_N^B(\bm{X}) = \bm{K}\bm{y}$, where $\bm{y}$ is the length-$N$ vector of training labels. For a new test point $x_0$, the model's prediction relies on the localized kernel weights $\bm{k}_0 = [\bm{k}(x_0, x_1), \dots, \bm{k}(x_0, x_N)]$, allowing us to formulate the prediction as:
\begin{equation}
\hat{f}_N^B(x_0) = \bm{k}_0^\top \bm{y}
\end{equation}
Because $\bm{k}_0 = \Phi\phi(x_0)^\top = \Psi\Lambda\psi(x_0)^\top$, we can substitute this back into the prediction equation:
\begin{equation}
\hat{f}_N^B(x_0) = (\Psi\Lambda\psi(x_0)^\top)^\top \bm{y} = \psi(x_0)\Lambda\Psi^\top \bm{y}
\end{equation}
This derivation reveals our compression strategy: if we store the eigendecomposition of $\bm{K}$ up to a truncated rank $M$ (where $M \ll N$) and train a neural network to approximate just the top $M$ eigenfunctions $\{\psi_i\}_{i=1}^M$, we achieve a heavily compressed distillation of the RF. By defining a fixed coefficient vector $\bm{c} := \Lambda_M \Psi_M^\top \bm{y} \in \mathbb{R}^M$, the test prediction simply amounts to evaluating:
\begin{equation}
\hat{f}_N^B(x) = \sum_{j=1}^M c_j \hat{\psi}_j(x)
\end{equation}
\end{comment}
By the Eckart-Young-Mirsky theorem \citep{eckart1936approximation}, this rank-$P$ truncation provides the optimal low-rank matrix approximation under the Frobenius norm, ensuring we retain maximal information from the ensemble's smoother matrix.
An identical strategy works for GBMs, substituting $\bm S = \bm{U \Sigma V}^\top$ for $\bm K = \bm{\Psi \Lambda \Psi}^\top$ and learning the left singular vector function $u: \mathcal X \mapsto \mathbb R^P$. Note that, because we can compress any ensemble that generates a valid smoother matrix in accordance with Eq. \ref{eq:smooth}, SCATE can be applied to a model already pruned by a different algorithm, provided that leaf values are not altered by the process. (Which is the case for FP, for instance, but not for LOP.)
\paragraph{Learning spectral directions with SCATE}
Extracting eigenfunction mappings from a given matrix is a well-studied problem. Existing neural approaches, such as SpIN \citep{pfau2018spectral} and NeuralEF \citep{deng2022neuralef}, rely heavily on constrained optimization, explicitly enforcing eigenvector orthonormality within the loss function to guide the network. However, constrained optimization methods are computationally heavy, making them difficult to tune and design in low-size settings. Conversely, methods like NeuralSVD \citep{ryu2024operator} utilizes unconstrained optimization by assigning greater weight to eigenfunctions with stronger signals. However, in practice, we find that these methods struggle to learn efficiently in constrained settings due to their complex objectives --- see Appx. \ref{appx:neural_exp} for a small comparison.
To overcome these limitations, SCATE focuses on a simple architecture with a targeted loss formulation. The network is parameterized as a simple Multi-Layer Perceptron (MLP) with SiLU activations \citep{ELFWING20183}. SCATE is trained via a multi-task objective \citep{caruana97}, which minimizes a spectrum-weighted MSE, alongside an orthogonality penalty $\mathcal{L}_\text{ortho}$, which represents the sum of off-diagonal terms of the empirical Gram matrix:
\begin{equation*}
\min_g \sum_{i=1}^N \sum_{j=1}^P \lambda_j^2 \big( g_j(\bm x_i) - \psi_j(\bm x_i) \big)^2 + \gamma \mathcal{L}_{\text{ortho}}.
\end{equation*}
We explicitly weigh the MSE by the corresponding squared eigenvalue $\lambda_j^2$---or singular value $\sigma_j^2$ for SVD---to encode greater interest in the most informative, high-variance spectral directions, and set $\gamma = 0.001$, which we found balanced mode diversity and reconstruction error well.
\begin{comment}
\subsection{SCATE for XGBoost}
While standard RFs yield symmetric kernels, GBMs iteratively update residuals, resulting in an asymmetric smoother matrix $\hat{\bm{S}}$. SCATE adapts to this by replacing the eigendecomposition with SVD.
The training smoother matrix is decomposed into $\hat{\bm{S}} \approx \bm{U}_M \Sigma_M \bm{V}_M^\top$. The right singular vectors $\bm{V}_M$ and singular values $\Sigma_M$ are memorized to construct the coefficient vector, while SCATE-XGB is tasked with learning the left singular vectors $\bm{U}_M$, which act as the out-of-sample feature embeddings. The objective function is mirrored correspondingly, weighting the loss by the singular values $\sigma_j$ instead of the eigenvalues.
\end{comment}
\paragraph{Oracle Benchmark}
To isolate our network's approximation error from the inherent loss of a low-rank bottleneck, we benchmark against an eigenfunction oracle. Given direct access to the train-test cross-kernel matrix $\bm{K}_{test}$, the oracle computes the optimal least-squares mapping $\bm{W}$ from the scaled training eigenvectors $\bm{C} = \bm{\Psi\Lambda}$, solving $\bm{W}\bm{C}^\top = \bm{K}_{test}$. Truncating to the top $P$ dimensions, the reconstructed test kernel is $\tilde{\bm{K}} = \bm{W}_P \bm{C}_P^\top$. The oracle then computes predictions via $\tilde{\bm{K}}\bm{y}$, and establishes a strict lower bound on the Frobenius reconstruction error $\|\bm{K}_{test} - \tilde{\bm{K}}\|_F$ achievable by any rank-$P$ distillation.
Interestingly, this does not guarantee that $\tilde{\bm K}$ attains optimal performance in terms of MSE on the associated prediction task, although it is rarely far off in practice.
\section{Experiments}\label{sec:exp}
In this section, we evaluate SCATE's performance for both RFs and GBMs. Comprehensive details describing the experimental setup can be found in Appx. \ref{appx:exp}; code can be found in the supplement.
We use consistent hyperparameters throughout, with \texttt{num\_trees=250}, \texttt{max\_depth=15} for RFs and \texttt{num\_trees=100}, \texttt{max\_depth=6} for GBMs.
\paragraph{Spectra and Generalization} Our strategy is motivated by the idea that the strong performance of RFs (and, by analogy, GBMs) is driven by their capacity to learn rich, informative embeddings encoded in the principal directions of their respective smoothing functions.
Spectral decay has implications not just for the convergence rate of an ensemble, as in Thm. \ref{thm:minimax}, but for distillation as well. To demonstrate this, Fig. \ref{fig:visualisation} illustrates the predictive performance of SCATE and the spectral oracle as a function of the number of eigenfunctions ($p$) for the RF case. We highlight the \texttt{student} and \texttt{concrete} datasets as representative examples of ``easy'' (rapid eigendecay) and ``hard'' (slower eigendecay) scenarios, respectively. Results for additional datasets are provided in the Appx \ref{appx:exp}. We evaluate two different architectures for comparison---a small MLP of \texttt{width=16,depth=2} and a larger MLP of \texttt{width=128,depth=4}. While the larger network’s superior performance across both datasets confirms that increased expressivity is useful, the results also demonstrate a connection between compressibility and difficulty, where the model performance converges quicker for the dataset with faster decay.
%In \texttt{boston}, SCATE performs well, occasionally even surpassing the spectral oracle. By contrast, SCATE lags behind significantly in \texttt{auto\_mpg}. This suggests that while compressibility is fundamentally tied to the kernel's eigendecay signature, it is specifically sensitive to the information density of the leading eigenfunctions.
Because SCATE's resource-bound architecture is limited to learning top components, its success depends on how much of the model’s total predictive power is concentrated within these leading eigenfunctions.
\begin{figure}[t]
\centering
\includegraphics[width=0.7\linewidth]{figures/vis.pdf}
\caption{Spectral decay, kernel approximation error, and predictive performance across two datasets.}
\label{fig:visualisation}
\end{figure}
\begin{comment}
\begin{itemize}
\item 1 plot scate -> fast eigendecay -> good compression
\item 1 plot scate-xgb -> fast eigendecay -> good compression
\item 1 plot scate -> slow eigendecay -> bad compression/slower approach to pareto frontier.
\item 1 plot scatexgb -> slow eigendecay -> bad compression
\item details to note: small MLPs are only good for approximating up to a certain number of components - if it is likely to take more than this to approximate the rf, then the model will not do well.
\end{itemize}
\end{comment}
\paragraph{Na\"{i}ve Benchmark} To further validate our approach, we benchmark SCATE against three na\"{i}ve tree compression baselines: (1) a vanilla MLP mirroring SCATE's architecture, but trained to directly learn mimic RF predictions (Na\"{i}ve Distillation); (2) an RF artificially constrained to various sizes via hyperparemeter tuning (Na\"{i}ve RF); and (3) the base performance of the original RF. Whereas in Fig. \ref{fig:visualisation}, we fixed model size and varied the number of eigenfunctions $p$, here we do the inverse. With fixed $p = 50$---a value we found to be generally robust,
striking a reasonable balance between complexity (of the base ensemble) and learnability (for the compressed network)---we vary the model size to create a Pareto frontier of rate-distortion curves for various methods and datasets. The base RF provides a reference point, indicating an optimal performance threshold (albeit at high model size).
We visualize results for three datasets in Fig. \ref{fig:naive} (additional results provided in Appx. \ref{appx:exp}).
SCATE consistently dominates in this experiment, effectively matching the predictive performance of the base RF while using orders of magnitude less memory.
Na\"{i}ve RF approaches the same performance levels but at a far slower rate.
SCATE also significantly outperforms the Na\"{i}ve Distillation scheme, which struggles to approximate RF predictions using the comparatively impoverished signal provided by direct RF outputs. This aligns with a notable finding of \citet{hinton_distilling}, who initiated research into neural network distillation. They showed that small student networks can quickly attain strong performance in classification tasks when trained on softmax weights from a larger teacher network because these so-called ``soft labels'' provide far more information than the hard labels used to train the original model.
We find that eigenfunctions perform a similar role in forest distillation for regression, accelerating distillation with rich spectral signals.
% Furthermore, while the Na\"{i}ve RF also eventually approaches base performance, it does so at a significantly slower rate than SCATE. These results validate SCATE as a functional framework for RF distillation.
\paragraph{Compression Benchmark}
For this experiment, we evaluate the size-performance trade-off of SCATE against a suite of established compression techniques, including pruning (FP and LOP), rule extraction (TreeExtract and FIRE), and approximate distillation (SSF). We also introduce a hybrid approach, FP-SCATE, which applies ForestPrune to a base model before further refinement with SCATE. We excluded other techniques like SIRUS, RuleFit, and LRL1, as prior studies suggest they are dominated by the methods we use here.
We were unable to find an implementation of MicroForest.
For all SCATE variants, we maintain $p=50$ eigenfunctions, consistent with our previous analysis.
% scate is good
\begin{figure}[t]
\centering
\includegraphics[width=0.9\linewidth]{figures/naive.pdf}
\caption{Rate-performance Pareto curves across three benchmark datasets. Shaded regions indicate standard errors across $10$ trials.}
\label{fig:naive}
\end{figure}
We evaluate performance across several regression and classification datasets using $R^2$ and accuracy, respectively; further dataset details are provided in Appx. \ref{appx:exp}.
Because the benchmarked algorithms utilize different hyperparameter mechanisms to control compression rates,
% ranging from validation error tolerances (FP, LOP) to regularization penalties, to compare them,
we adopt a size-constrained evaluation strategy. For each method and run, we sweep across a broad hyperparameter range to generate a performance-vs-size profile.
We then report the best performance achieved at two fixed model sizes: 10KB and 100KB. These thresholds reflect the limited flash memory typical of microcontrollers, which generally operate with a budget under 1MB \citep{branco2019micro}. Original model sizes vary, but at 10KB, compression factors are on the order of $10^3$ for RFs and $10^2$ for GBMs --- see Appx. \ref{appx:big_table}. This experimental design is applied to both RF and GBM models, with the exception of SSF, which is restricted to RFs.
Notably, while the RF kernel $\bm{K}$ generalizes naturally to both tasks, the GBM smoother $\bm{S}$ is defined specifically for MSE loss. To extend SCATE to GBM classification, we train the base model using MSE loss—effectively treating classification as a regression problem—which limits our evaluation to the binary case.
As shown in Tables \ref{tab:rf} and \ref{tab:gbm}, SCATE yields highly competitive results for both RF and GBM compression, securing the highest average rank across all evaluated model sizes, with FP-SCATE coming in second for the RF benchmark.
In contrast, this hybrid method declines markedly when applied to GBMs. We attribute this decline to the weaker baseline performance of FP as a preprocessing step, compounded by the fact that FP's pruning mechanism alters the ensemble structure in a way that violates our recursive formula for computing $\bm S$. For a more comprehensive results of algorithm performance, see Appx. \ref{appx:extra_results}.
\begin{comment}
\begin{itemize}
\item Pruning methods: FP, LOP
\item Distillation methods: MF and SSF?
\item SCATE
\item Bonus combo: FP + SCATE
\item and do it all again for XGB, but the smoother matrix only works for the regression case [possible to do binary classification with MSE loss, perhaps just for the appendix tho?]
\end{itemize}
Present all this as tables (one for RFs, one for XGB) with rows for datasets, columns for methods. Reported numbers represent performance ($R^2$ or accuracy) at model size $10^{-2}$ MB.
-Runtime experiments (training and inference?)
-For the appendix: evaluating (C2) and (C3)
-For the appendix: why we don't use NeuralEF or NeuralSVD
\end{comment}
\begin{table}[t] %[htbp]
\centering
\caption{RF compression benchmark, averaged over $10$ trials. Best result per row in bold. NA indicates that the model could not reach the target size; TO indicates time out after 24 hours. For an expanded table with standard errors, see Appx. \ref{appx:big_table}}
\label{tab:results}
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\resizebox{\textwidth}{!}{
\begin{tabular}{lc|ccccccc|ccccccc}
\toprule
& & \multicolumn{7}{c|}{\textbf{Size = 10KB}} & \multicolumn{7}{c}{\textbf{Size = 100KB}} \\
\cmidrule{3-16}
\textbf{Dataset} & \textbf{RF} & \textbf{FP} & \textbf{LOP} & \textbf{FIRE} & \textbf{TE} & \textbf{SSF} & \textbf{SCATE} & \textbf{FP-SCATE} & \textbf{FP} & \textbf{LOP} & \textbf{FIRE} & \textbf{TE} & \textbf{SSF} & \textbf{SCATE} & \textbf{FP-SCATE} \\
\midrule
\texttt{abalone} & 0.55 & 0.35 & 0.50 & 0.28 & NA & NA & \textbf{0.56} & 0.54 & 0.53 & 0.51 & 0.28 & 0.55 & NA & \textbf{0.57} & 0.55 \\
\texttt{adult} & 0.85 & 0.79 & 0.83 & TO & NA & NA & \textbf{0.84} & 0.83 & 0.84 & 0.83 & TO & 0.84 & \textbf{0.85} & 0.85 & 0.85 \\
\texttt{auto\_mpg} & 0.69 & 0.49 & \textbf{0.63} & 0.62 & NA & 0.60 & 0.45 & 0.46 & 0.58 & \textbf{0.63} & 0.62 & 0.38 & 0.60 & 0.50 & 0.51 \\
\texttt{banknote} & 0.99 & 0.87 & 0.99 & 0.99 & NA & 1.00 & \textbf{1.00} & \textbf{1.00} & 0.92 & 0.99 & 0.99 & 0.89 & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} \\
\texttt{bc} & 0.96 & 0.95 & 0.92 & 0.95 & NA & 0.98 & \textbf{0.98} & 0.98 & 0.95 & 0.92 & 0.95 & 0.92 & 0.98 & \textbf{0.99} & 0.99 \\
\texttt{boston} & 0.85 & 0.60 & \textbf{0.80} & 0.68 & NA & NA & 0.78 & 0.79 & 0.80 & 0.82 & 0.69 & 0.81 & 0.72 & 0.81 & \textbf{0.83} \\
\texttt{california} & 0.80 & 0.45 & 0.68 & TO & TO & NA & 0.70 & \textbf{0.71} & 0.68 & 0.74 & TO & TO & NA & \textbf{0.75} & 0.74 \\
\texttt{churn} & 0.86 & 0.82 & 0.84 & 0.85 & NA & NA & \textbf{0.86} & 0.85 & 0.85 & 0.85 & 0.85 & 0.85 & 0.83 & \textbf{0.86} & 0.86 \\
\texttt{concrete} & 0.91 & 0.50 & 0.74 & 0.73 & NA & NA & 0.80 & \textbf{0.82} & 0.76 & 0.86 & 0.75 & 0.89 & \textbf{0.90} & 0.86 & 0.86 \\
\texttt{diabetes} & 0.76 & 0.76 & 0.74 & 0.77 & NA & NA & 0.79 & \textbf{0.79} & 0.76 & 0.74 & 0.77 & 0.66 & 0.73 & 0.79 & \textbf{0.80} \\
\texttt{friedman\_1} & 0.85 & 0.47 & 0.76 & 0.48 & NA & NA & \textbf{0.82} & 0.81 & 0.74 & 0.82 & 0.49 & 0.79 & NA & \textbf{0.86} & 0.86 \\
\texttt{spambase} & 0.95 & 0.87 & 0.92 & \textbf{0.94} & NA & NA & 0.93 & 0.93 & 0.91 & 0.93 & 0.94 & 0.89 & \textbf{0.94} & 0.94 & 0.94 \\
\texttt{student} & 0.85 & 0.64 & 0.80 & 0.71 & NA & 0.82 & \textbf{0.85} & 0.84 & 0.84 & 0.81 & 0.72 & 0.79 & 0.82 & \textbf{0.86} & 0.84 \\
\texttt{telco} & 0.79 & 0.75 & 0.77 & 0.79 & NA & NA & \textbf{0.80} & 0.80 & 0.80 & 0.77 & 0.79 & 0.79 & 0.78 & \textbf{0.81} & 0.80 \\
\texttt{wq} & 0.50 & 0.24 & 0.31 & 0.31 & NA & NA & \textbf{0.34} & 0.33 & 0.35 & 0.36 & 0.34 & 0.28 & 0.24 & \textbf{0.36} & 0.36 \\
\midrule
\textbf{Average Rank} & & 4.8 & 3.47 & 3.87 & 6.57 & 5.50 & \textbf{1.77} & 2.03 & 4.40 & 4.33 & 4.85 & 5.00 & 4.19 & \textbf{1.93} & 2.47 \\
\bottomrule
\end{tabular}
}
\label{tab:rf}
\vspace{-3mm}
\end{table}
\begin{table}[htbp]
\centering
\caption{GBM compression benchmark, averaged over $10$ trials. Best result per row in bold. NA indicates that the model could not reach the target size; TO indicates time out after 24 hours. For an expanded table with standard errors, see Appx. \ref{appx:big_table}}
\label{tab:gbm}
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\resizebox{\textwidth}{!}{
\begin{tabular}{lc|cccccc|cccccc}
\toprule
& & \multicolumn{6}{c|}{\textbf{Size = 10KB}} & \multicolumn{6}{c}{\textbf{Size = 100KB}} \\
\cmidrule{3-14}
\textbf{Dataset} & \textbf{GBM} & \textbf{FP} & \textbf{LOP} & \textbf{TE} & \textbf{FIRE} & \textbf{SCATE} & \textbf{FP-SCATE} & \textbf{FP} & \textbf{LOP} & \textbf{TE} & \textbf{FIRE} & \textbf{SCATE} & \textbf{FP-SCATE} \\
\midrule
\texttt{abalone} & 0.53 & 0.44 & 0.50 & 0.54 & 0.47 & \textbf{0.56} & 0.55 & 0.53 & 0.52 & 0.55 & 0.47 & \textbf{0.57} & 0.55 \\
\texttt{adult} & 0.85 & 0.82 & 0.84 & \textbf{0.85} & 0.81 & 0.85 & 0.85 & 0.85 & 0.85 & \textbf{0.86} & 0.81 & 0.85 & 0.85 \\
\texttt{auto\_mpg} & 0.66 & 0.60 & 0.58 & 0.38 & \textbf{0.62} & 0.44 & 0.13 & \textbf{0.69} & 0.58 & 0.38 & 0.62 & 0.48 & 0.21 \\
\texttt{banknote} & 0.98 & 0.95 & 0.98 & 0.99 & 0.98 & \textbf{1.00} & 1.00 & 0.99 & 0.98 & 1.00 & 0.98 & \textbf{1.00} & 1.00 \\
\texttt{bc} & 0.94 & 0.94 & 0.94 & 0.94 & 0.95 & \textbf{0.98} & 0.95 & 0.95 & 0.94 & 0.95 & 0.95 & \textbf{0.99} & 0.95 \\
\texttt{boston} & 0.86 & 0.78 & \textbf{0.81} & 0.79 & 0.78 & 0.79 & 0.71 & \textbf{0.87} & 0.83 & 0.82 & 0.78 & 0.84 & 0.76 \\
\texttt{california} & 0.83 & 0.58 & 0.76 & \textbf{0.79} & 0.75 & 0.71 & 0.70 & 0.80 & \textbf{0.82} & 0.79 & 0.75 & 0.79 & 0.76 \\
\texttt{churn} & 0.86 & 0.85 & 0.86 & \textbf{0.86} & 0.85 & 0.86 & 0.85 & 0.86 & 0.86 & \textbf{0.87} & 0.85 & 0.86 & 0.86 \\
\texttt{concrete} & 0.92 & 0.71 & 0.83 & \textbf{0.87} & 0.84 & 0.79 & 0.81 & 0.90 & 0.91 & \textbf{0.92} & 0.84 & 0.88 & 0.87 \\
\texttt{diabetes} & 0.74 & 0.77 & 0.76 & 0.78 & 0.74 & \textbf{0.79} & 0.73 & 0.78 & 0.76 & 0.78 & 0.74 & \textbf{0.80} & 0.73 \\
\texttt{friedman\_1} & 0.91 & 0.68 & 0.87 & \textbf{0.92} & 0.79 & 0.89 & 0.87 & 0.90 & 0.90 & 0.92 & 0.79 & \textbf{0.97} & 0.93 \\
\texttt{spambase} & 0.95 & 0.90 & 0.92 & \textbf{0.94} & 0.92 & 0.93 & 0.93 & 0.95 & 0.93 & \textbf{0.95} & 0.92 & 0.94 & 0.94 \\
\texttt{student} & 0.83 & 0.81 & 0.79 & 0.79 & 0.75 & \textbf{0.84} & 0.70 & \textbf{0.86} & 0.79 & 0.79 & 0.75 & 0.84 & 0.70 \\
\texttt{telco} & 0.79 & 0.79 & 0.79 & 0.80 & 0.79 & \textbf{0.81} & 0.79 & 0.80 & 0.79 & 0.81 & 0.79 & \textbf{0.81} & 0.80 \\
\texttt{wq} & 0.45 & 0.29 & 0.32 & 0.30 & \textbf{0.35} & 0.33 & 0.33 & \textbf{0.39} & 0.38 & 0.30 & 0.35 & 0.36 & 0.36 \\
\midrule
\textbf{Average Rank} & & 5.13 & 3.27 & 2.47 & 3.87 & \textbf{2.13} & 4.13 & 2.57 & 3.8 & 2.80 & 5.33 & \textbf{2.13} & 4.37 \\
\bottomrule
\end{tabular}
}
\label{tab:gbm}
\vspace{-3mm}
\end{table}
\section{Discussion}\label{sec:discussion}
Our theoretical results suggest that generalization in tree ensembles is strongly linked to the compressibility of the model. The more information is encoded in the leading eigenfunctions of the RF or singular vectors of the GBM, the better the model learns, and the more easily it can be distilled. This accords with the minimum description length principle \citep{rissanen1989, hutter2004universal, grunwald2007} and related information theoretic insights suggesting a profound connection between the dual tasks of prediction and compression.
Notably, our minimax optimality theorem is limited to RF regression. A promising direction for future work would be to pursue similar results for GBMs. This poses substantial technical challenges, as kernel methods do not apply. The smoother matrix is neither symmetric nor PSD, and boosting induces nontrivial dependencies between basis functions. Still, recent work suggests some theoretical overlap in the mechanics of tree ensembles that may be exploited for future results \citep{zhou}.
Our compression method differs fundamentally from traditional techniques in avoiding direct learning from the base model. Instead, SCATE learns to reproduce the leading spectral directions of a linear operator.
% constructs a kernel (or smoother) matrix that represents the underlying base model and approximates it by learning its top eigenfunctions.
This approach offers distinct advantages over classical pruning methods, which typically optimize the tree structure itself, making their computational cost heavily dependent on the base model's size. By contrast, SCATE relies solely on the kernel, which can be computed with a single forward pass of the training data.
Scaling up the complexity of the base model therefore has minimal impact on distillation time.
% Consequently, while we kept our model setups moderate to ensure a fair comparison with traditional baselines, SCATE can leverage more complex forests to enhance kernel quality, and subsequently the student MLP's performance, with minimal time overhead (albeit at some cost in space due to $\bm K$'s quadratic dependence on sample size).
Furthermore, whereas traditional pruning approaches usually output tree-like structures, which rely on inherently sequential branching logic, SCATE compiles into a network where inference amounts to computing fast, parallelizable matrix operations.
SCATE is not without its limitations. First, training imposes nontrivial computational demands: storing operators requires $\mathcal{O}(N^2)$ space, and truncated decomposition takes $\mathcal{O}(N^2P)$ time. However, alongside standard mitigations like randomized SVD \citep{Halko2011Finding} and Nystr{"o}m approximations \citep{WilliamsSeeger2001}, we find empirically that subsampling the training set to 10,000 observations produces no noticeable loss in distillation accuracy, effectively removing this bottleneck for large datasets. Second, deployment constraints restrict SCATE to using just the top $20$ to $50$ eigenfunctions. This may be insufficient for complex problems, leading SCATE's approximation quality to suffer—although our strong empirical results indicate this is not often the case in practice.
While our framework allows us to stipulate the final size of the model, choosing the optimal network architecture is nontrivial. In our experiments, a simple MLP proved effective in the resource-constrained setting, but more sophisticated alternatives may provide improvements.
Finally, our distilled model sacrifices the purported interpretability benefits of trees, which naturally lend themselves to rule extraction and formal verification \citep{izza2021, ignatiev2022using}.
\section{Conclusion} \label{sec:conclusion}
We have shown that tree ensembles can be compressed into significantly smaller models of comparable performance using spectral distillation. These results are motivated by a novel theoretical insight: that RFs are minimax-optimal when the eigenvalues of the kernel matrix decay at a polynomial rate. We conjecture that a similar result holds for GBMs, when singular values of the smoother matrix decay quickly. Extensive benchmarks show that our methods are competitive with state of the art pruning and rule extraction techniques for ensemble compression, providing new methods for resource constrained computing.
\newpage
\appendix
\normalsize
\section{Proofs}\label{appx:proofs}
We restate Thm. \ref{thm:minimax} for completeness.
\textbf{Theorem} [Minimax convergence].
Assume (A1)-(A4), (B1), and (C1)-(C3).
Suppose that $\alpha \beta > 1/2$.
Then with $M \asymp N^{1 / (2 \alpha \beta + 1)}$ leaves per tree and $B=\infty$ trees, we have:
\begin{align*}
\mathbb E \Big[\|f^* - \hat f\|_{L^2}^2 \Big] \lesssim N^{- 2\alpha \beta / (2\alpha \beta + 1)}.
\end{align*}
This rate is minimax optimal over the source class defined by (C1)-(C3).
\begin{proof}
Taking the expectation w.r.t. both the data generating process $\mu$ and the tree randomization mechanism $\nu$, we derive the limiting kernel:
\begin{align*}
k_\mu^\nu(\bm x, \bm x')
:= \lim_{N,B \rightarrow \infty} \hat k_N^B(\bm x, \bm x').
\end{align*}
Fix some $\mu, \nu$.
We distinguish between the limiting operator $\bm K$ and the finite sample operator $\hat{\bm K}$ (presumed to share the same depth/leaf count unless explicitly stated otherwise.)
That is, while $\hat{\bm K}$ is an empirical kernel matrix defined by $\hat k^B_N$ for finite $N$ and $B$, $\bm K$ is the operator defined by the limiting kernel $k^\nu_\mu$.
Model residuals can be decomposed as follows:
\[
f^* - \hat f = (f^* - \bar{\bm K}_M f^*) + (\bar{\bm K}_M f^* - \hat{\bm K}_M f^*) + (\hat{\bm K}_M f^* - \hat f),
\]
where $M$ denotes the leaf count given in Thm. \ref{thm:minimax}, and $\bar{\bm K}_M := \mathbb{E}_\nu[\bm K_M \mid \mathcal{D}_{\mathrm{split}}]$ denotes the data-dependent forest-level population operator — the average of per-tree population operators over partition randomness, conditional on the structure fold.
Taking $L^2$ norms and expectations, we have:
\begin{align*}
\mathbb{E} \Big[ \|f^* - \hat f\|_{L^2}^2 \Big]
&\lesssim
\underbrace{\|f^* - \bar{\bm K}_M f^*\|_{L^2}^2}_{\text{bias}^2}
+
\underbrace{
\mathbb{E} \Big[ \|\bar{\bm K}_M f^* - \hat{\bm K}_M f^*\|_{L^2}^2 \Big]
}_{\text{operator error}}
+
\underbrace{
\mathbb{E} \Big[ \|\hat{\bm K}_M f^* - \hat f\|_{L^2}^2 \Big]
}_{\text{variance}}.
\end{align*}
We derive upper bounds for each term, followed by matching lower bounds.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{lemma}[Bias]
\label{lem:bias}
Assume (C1), (C2), and (C3). Then for all $M \geq q$:
\[
\|f^* - \bar{\bm K}_M f^*\|_{L^2} \lesssim M^{-\alpha\beta}.
\]
\end{lemma}
\begin{proof}
By (C1), $f^* = \bm K_q^\alpha g$ where $\bm K_q$ has eigenfunctions $\{\psi_i\}_{i \geq 1}$
(ordered by decreasing eigenvalue) and eigenvalues $\lambda_i \asymp i^{-\beta}$ under (C2).
Let $\bm P_M$ denote the orthogonal projection onto
$\mathrm{span}\{\psi_1, \ldots, \psi_M\}$. Decompose:
\[
\|(\bm I - \bar{\bm K}_M)f^*\|_{L^2}
\leq \underbrace{\|(\bm I - \bar{\bm K}_M) \bm P_M f^*\|_{L^2}}_{\text{low-frequency}}
\;+\; \underbrace{\|(\bm I - \bar{\bm K}_M)(\bm I - \bm P_M) f^*\|_{L^2}}_{\text{high-frequency}}.
\]
\textbf{High-frequency tail.}
Since $\bar{\bm K}_M$ is a contraction (the rows integrate to unity), we have $\|\bar{\bm K}_M\|_{op} \leq 1$, and therefore $\|\bm I - \bar{\bm K}_M\|_{op} \leq 2$. Thus:
\[
\|(\bm I - \bar{\bm K}_M)(\bm I - \bm P_M) f^*\|_{L^2} \leq 2\, \|(\bm I - \bm P_M) f^*\|_{L^2}.
\]
Expanding in the eigenbasis of $\bm K_q$ and using the source condition (C1):
\[
\|(\bm I - \bm P_M) f^*\|_{L^2}^2
= \sum_{i > M} \lambda_i^{2\alpha} \langle g, \psi_i \rangle^2
\leq \lambda_{M+1}^{2\alpha} \, \|g\|_{L^2}^2
\asymp M^{-2\alpha\beta} \, \|g\|_{L^2}^2,
\]
where the inequality uses monotonicity of the eigenvalues and the final step uses (C2).
Therefore:
\[
\|(\bm I - \bar{\bm K}_M)(\bm I - \bm P_M)f^*\|_{L^2} \lesssim M^{-\alpha\beta} \, \|g\|_{L^2}.
\]
\textbf{Low-frequency component.}
Since $f^* = \bm{K}_q^\alpha g$ by (C1), we have $\bm{P}_M f^* = \bm{P}_M \bm{K}_q^\alpha g$.
Applying the operator norm bound from (C3) directly:
\[
\|(\bm I - \bar{\bm K}_M) \bm P_M f^*\|_{L^2}
\leq \|(\bm I - \bar{\bm K}_M) \bm K_q^\alpha\|_{op} \, \|g\|_{L^2}
\lesssim M^{-\alpha\beta} \, \|g\|_{L^2}.
\]
\medskip
\noindent\textbf{Combining.}
Both the high-frequency tail and the low-frequency component are bounded by
$M^{-\alpha\beta} \, \|g\|_{L^2}$. The latter factor is a finite constant by (C1), giving the final bias rate:
\[
\|f^* - \bar{\bm K}_M f^*\|_{L^2} \lesssim M^{-\alpha\beta},
\]
for all $M \geq q$. This completes the proof.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{lemma}[Operator error]
\label{lem:operator_error}
Suppose (A1), (A2), (A4), and (B1) hold. Then
\[
\mathbb E\bigl[\|(\hat{\bm K}_M - \bar{\bm K}_M)f^*\|^2_{L_2}\bigr]
\lesssim \frac{M}{N^{(b)}}.
\]
\end{lemma}
\begin{proof}
By honesty (A1), the training data is split into a \emph{structure fold} $\mathcal{D}_{\mathrm{split}}$,
used to learn the partition, and a \emph{labelling fold} $\mathcal{D}_{\mathrm{label}}$, used to
estimate leaf means. Conditional on $\mathcal{D}_{\mathrm{split}}$ and the tree randomness
$\{\omega_b\}_{b=1}^B$, the partition is fixed. The operator error then measures how well
the leaf-wise sample averages of $f^*$, computed from i.i.d.\ observations in
$\mathcal{D}_{\mathrm{label}}$ drawn independently of $\mathcal D_{\mathrm{split}}$, approximate the population cell means.
Fix a tree $b$ with $M$ leaves $\{\mathcal X_\ell\}_{\ell=1}^M$. Write $p_\ell := \mu_X(\mathcal X_\ell)$ for the population coverage of leaf $\ell$ and $N_\ell$ for the number of labelling-fold observations falling in leaf $\ell$. Balance (A4) ensures $p_\ell > 0$ for all leaves.
Conditional on $\mathcal D_{\mathrm{split}}$, the sample mean of $Y$ within leaf $\ell$ is an unbiased estimate of the population cell mean $\mathbb E[Y \mid \bm x \in \mathcal X_\ell]$, with pointwise variance bounded by $1/N_\ell$ (using $Y \in [0,1]$ under (B1)). The integrated estimation error for tree $b$ is:
\[
\int_{\mathcal X}
\mathrm{Var}\bigl(\bar Y \mid \bm x \in \mathcal X_\ell; \mathcal D_{\mathrm{split}}\bigr)
\, d\mu_X(\bm x)
= \sum_{\ell=1}^{M} \frac{p_\ell}{N_\ell}.
\]
We bound this quantity by taking expectations over the labelling fold conditional on $\mathcal{D}_{\mathrm{split}}$. By honesty (A1), $\mathcal{D}_{\mathrm{label}}$ is drawn independently of $\mathcal{D}_{\mathrm{split}}$, so the leaf counts $N_\ell$ follow a $\mathrm{Binomial}(N^{(b)}, p_\ell)$ distribution conditionally on the fixed partition. Applying the standard inverse binomial moment bound --- for $Z \sim \mathrm{Binomial}(n, p)$ with $np \geq 1$, $\mathbb{E}[1/Z] \leq 2/(np)$ --- gives:
\[
\mathbb{E}\biggl[\frac{p_\ell}{N_\ell} \,\bigg|\, \mathcal{D}_{\mathrm{split}}\biggr]
= p_\ell \cdot \mathbb{E}\biggl[\frac{1}{N_\ell} \,\bigg|\, \mathcal{D}_{\mathrm{split}}\biggr]
\leq p_\ell \cdot \frac{2}{N^{(b)} p_\ell}
= \frac{2}{N^{(b)}}.
\]
The condition $N^{(b)} p_\ell \geq 1$ holds for all leaves and all sufficiently large $N$, since $p_\ell > 0$ for all $\ell$ by balance (A4) and $N^{(b)} \to \infty$. Summing over $M$ leaves and applying the tower property:
\[
\mathbb{E}\biggl[\sum_{\ell=1}^{M} \frac{p_\ell}{N_\ell}\biggr] \leq \frac{2M}{N^{(b)}}.
\]
For a single tree $b$ with fixed partition, $(\hat{\bm K}^{(b)}_M - \bm K^{(b)}_M)f^*(\bm x)$ is the
difference between the sample and population cell mean at the leaf containing $\bm x$.
The conditional squared error is therefore bounded by the per-leaf estimation
variance:
\[
\mathbb E\bigl[(\hat{\bm K}_M^{(b)} f^*(\bm x) - \bm K^{(b)}_M f^*(\bm x))^2
\mid \mathcal{D}_{\mathrm{split}}\bigr]
\lesssim \frac{M}{N^{(b)}}.
\]
This bound holds for any partition that satisfies (A4), regardless of how that partition was selected.
It remains to pass from a single tree to the forest average. Write the forest-level operator error as:
$$\hat{K}_M f^* - \bar{K}_M f^* = \underbrace{\frac{1}{B}\sum_{b=1}^B \bigl(\hat{K}_M^{(b)} f^* - \bar{K}_M f^*\bigr)}_{\text{tree-level errors}}.$$
Each summand can be further decomposed:
$$\hat{K}_M^{(b)} f^* - \bar{K}_M f^* = \underbrace{\bigl(\hat{K}_M^{(b)} f^* - K_M^{(b)} f^*\bigr)}_{\text{estimation error (data noise)}} + \underbrace{\bigl(K_M^{(b)} f^* - \bar{K}_M f^*\bigr)}_{\text{Monte Carlo error (partition randomness)}},$$
where $K_M^{(b)}$ denotes the population-level operator for tree $b$ (i.e., using population cell means for the partition selected by tree $b$).
\textit{Monte Carlo error.} Conditional on $\mathcal{D}_{\mathrm{split}}$, the partition randomness $\{\omega_b\}_{b=1}^B$ is i.i.d.\ across trees. The summands $K_M^{(b)} f^* - \bar{K}_M f^*$ are therefore i.i.d.\ mean-zero (over $\omega_b$) with $\|K_M^{(b)} f^*\|_\infty \leq 1$ under (B1). The variance of the average scales as $1/B$:
$$\mathbb{E}\Bigl[\Bigl\|\frac{1}{B}\sum_{b=1}^B \bigl(K_M^{(b)} f^* - \bar{K}_M f^*\bigr)\Bigr\|_{L^2}^2 \,\Big|\, \mathcal{D}_{\mathrm{split}}\Bigr] \lesssim \frac{1}{B}.$$
This term vanishes as $B \to \infty$ and is negligible for a few hundred trees.
\textit{Estimation error.} Conditional on $\mathcal{D}_{\mathrm{split}}$ and $\{\omega_b\}_{b=1}^B$, each tree's partition is fixed — the structure fold and randomization seeds together fully determine the partition, leaving the labelling fold random. Since all trees subsample from the same finite dataset, the estimation errors $\hat{K}_M^{(b)} f^* - K_M^{(b)} f^*$ are correlated across trees — they are driven by the same underlying observations $\{y_i\}_{i=1}^N$. By Jensen's inequality applied to the convex forest average:
\begin{align*}
\mathbb{E}\Bigl[\Bigl\|\frac{1}{B}\sum_{b=1}^B \bigl(\hat{K}_M^{(b)} f^* - K_M^{(b)} f^*\bigr)\Bigr\|_{L^2}^2 \,\Big|\, \mathcal{D}_{\mathrm{split}}, \{\omega_b\}\Bigr] &\leq \frac{1}{B}\sum_{b=1}^B \mathbb{E}\bigl[\|\hat{K}_M^{(b)} f^* - K_M^{(b)} f^*\|_{L^2}^2 \,\big|\, \mathcal{D}_{\mathrm{split}}, \omega_b\bigr] \\
&\lesssim \frac{M}{N^{(b)}},
\end{align*}
where the last step uses the single-tree bound established above. Crucially, this bound does not improve with $B$: the shared dataset prevents the estimation errors from cancelling across trees.
\textit{Combining.} Taking squared $L^2$ norms, using the Cauchy-Schwarz inequality to handle the cross-term between estimation and Monte Carlo errors, and removing the conditioning by the tower property:
$$\mathbb{E}\bigl[\|(\hat{K}_M - \bar{K}_M) f^*\|_{L^2}^2\bigr] \lesssim \frac{M}{N^{(b)}} + \frac{1}{B}.$$
In the $B = \infty$ regime of Thm. \ref{thm:minimax}, the Monte Carlo term vanishes and the operator error reduces to $M/N^{(b)}$, completing the proof.
\end{proof}
\begin{lemma}[Variance]
\label{lem:variance}
Suppose (A1)-(A2), (A4), and (B1) hold. Then
\[
\mathbb E\bigl[\|\hat{\bm K}_M f^* - \hat{f}\|^2_{L_2}\bigr]
\lesssim \frac{M}{N^{(b)}}.
\]
\end{lemma}
\begin{proof}
Write $\varepsilon_i = y_i - f^*(\bm x_i)$ for the label noise. By the regression model,
$\mathbb E[\varepsilon_i \mid \bm x_i] = 0$ and $\text{Var}(\varepsilon_i \mid \bm x_i) \leq 1$, the latter following from $Y \in [0,1]$ under (B1).
We begin by noting that, at any point $\bm x \in \mathcal{X}$, the difference between the empirical prediction and the
kernel-smoothed truth is driven entirely by noise:
\[
\hat{f}(\bm x) - \hat{\bm K}_M f^*(\bm x)
= \sum_{i=1}^N \hat{k}_N^B(\bm x, \bm x_i)\, y_i - \sum_{i=1}^N \hat{k}_N^B(\bm x, \bm x_i)\, f^*(\bm x_i)
= \sum_{i=1}^N \hat{k}_N^B(\bm x, \bm x_i)\, \varepsilon_i.
\]
By honesty (A1), the residuals in the labelling fold are independent of the partition learned from the structure fold. Conditional on $\mathcal{D}_{\mathrm{split}}$, the
kernel weights are deterministic. Since the $\varepsilon_i$ are conditionally independent with mean
zero, the cross terms vanish and the pointwise variance is controlled by the squared kernel weights:
\[
\mathbb E\Bigl[\Bigl(\sum_{i=1}^N \hat{k}_N^B(\bm x, \bm x_i)\, \varepsilon_i\Bigr)^2
\;\Big|\; \mathcal{D}_{\mathrm{split}}\Bigr]
= \sum_{i=1}^N \hat{k}_N^B(\bm x, \bm x_i)^2\, \sigma^2(\bm x_i)
\leq \sum_{i=1}^N \hat{k}_N^B(\bm x, \bm x_i)^2.
\]
It remains to bound $\sum_i \hat{k}_N^B(\bm x, \bm x_i)^2$.
Recall that the RF algorithm is effectively an adaptive nearest neighbors procedure \citep{lin2006}, with tree-wise weight function
\begin{align*}
w_i^{(b)}(\bm x) = \frac{\hat{k}^{(b)}(\bm x, \bm x_i)}{\sum_j \hat{k}^{(b)}(\bm x, \bm x_j)},
\end{align*}
where the denominator equals $N_\ell^{(b)}(\bm x)$, the number of training points in the leaf containing $\bm x$ in tree $b$.
Within each leaf, $w_i^{(b)}(\bm x) = 1/N_\ell^{(b)}$ for neighbors and zero otherwise, so
\[
\sum_{i=1}^N w_i^{(b)}(\bm x)^2 = \frac{1}{N_\ell^{(b)}(\bm x)}.
\]
The empirical forest kernel averages these weights: $\hat{k}_N^B(\bm x, \bm x_i) = \frac{1}{B}\sum_b w_i^{(b)}(\bm x)$.
Applying Jensen's inequality to the square:
\[
\sum_{i=1}^N \hat{k}_N^B(\bm x, \bm x_i)^2
= \sum_{i=1}^N \Bigl(\frac{1}{B}\sum_{b=1}^B w_i^{(b)}(\bm x)\Bigr)^2
\leq \frac{1}{B} \sum_{b=1}^B \sum_{i=1}^N w_i^{(b)}(\bm x)^2
= \frac{1}{B} \sum_{b=1}^B \frac{1}{N_\ell^{(b)}(\bm x)}.
\]
Integrating both sides over $\mu_X$:
\[
\int_\mathcal X \sum_{i=1}^N \hat k^B_N(\bm x, \bm x_i)^2 ~d \mu_X(\bm x) \leq \frac{1}{B} \sum_{b=1}^B \sum_{\ell=1}^M \frac{p_\ell^{(b)}}{N_\ell^{(b)}},
\]
where $p_\ell^{(b)} = \mu_X(\mathcal X_\ell^{(b)})$ is the population coverage of leaf $\ell$ in tree $b$. By honesty (A1), the leaf counts $N_\ell^{(b)}$ follow a $\mathrm{Binomial}(N^{(b)}, p_\ell^{(b)})$ distribution conditionally on the fixed partition. Applying the inverse binomial moment bound as in Lemma~\ref{lem:operator_error}:
\[
\mathbb{E}\biggl[\frac{p_\ell^{(b)}}{N_\ell^{(b)}} \,\bigg|\, \mathcal{D}_{\mathrm{split}}\biggr] \leq \frac{2}{N^{(b)}}.
\]
Summing over $M$ leaves and averaging over $B$ trees:
\[
\int_\mathcal X \sum_{i=1}^N \hat k^B_N(\bm x, \bm x_i)^2 ~d \mu_X(\bm x) \lesssim \frac{M}{N^{(b)}}.
\]
Therefore:
\[
\mathbb E\bigl[\|\hat{\bm K}_M f^* - \hat{f}\|^2_{L_2}\bigr]
\lesssim \frac{M}{N^{(b)}}.
\]
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Upper bound}
Combining Lemmas \ref{lem:bias}, \ref{lem:operator_error}, and \ref{lem:variance} with the original error decomposition, we have:
\[
\mathbb E\bigl[\|f^* - \hat{f}\|^2_{L_2}\bigr]
\lesssim ~\underbrace{M^{-2\alpha\beta}}_{\text{bias}^2}
\;+\; \underbrace{\frac{M}{N^{(b)}}}_{\text{operator error}}
\;+\; \underbrace{\frac{M}{N^{(b)}}}_{\text{variance}}.
\]
Since the operator error and variance coincide in rate, the effective trade-off is:
\[
\mathbb E\bigl[\|f^* - \hat{f}\|^2_{L_2}\bigr]
\lesssim M^{-2\alpha\beta} + \frac{M}{N^{(b)}}.
\]
\paragraph{Balancing.}
Under (A1)–(A2), we grow trees on subsamples of size $N^{(b)} = cN$ for some $c \in (0,1)$, with a fraction reserved for labelling. Thus the effective estimation sample size is proportional to $N$, giving asymptotic equivalence of the summands:
\[
M^{-2\alpha\beta} \asymp \frac{M}{N},
\]
which gives
\[
M^{2\alpha\beta + 1} \asymp N,
\qquad\text{i.e.,}\qquad
M \asymp N^{1/(2\alpha\beta+1)}.
\]
Substituting back:
\begin{equation}\label{eq:upper}
\mathbb E\bigl[\|f^* - \hat{f}\|^2_{L_2}\bigr]
\lesssim N^{-2\alpha\beta/(2\alpha\beta+1)}.
\end{equation}
In practice, subsampling with $N^{(b)} < N$ may still be desirable for computational reasons or statistical inference \citep{Wager2018}, but the convergence rate does not require it.
\subsection*{Lower bound}
We show that no estimator can improve on the exponent $2\alpha\beta/(2\alpha\beta+1)$ over
the source class defined by (C1)-(C3).
\paragraph{The function class.}
Assumptions (C1) and (C2) define a spectral ellipsoid. The source condition
$f^* = \bm K_q^\alpha g$ with $\|g\|_{L_2} \leq R < \infty$ means the Fourier coefficients of $f^*$ in the
eigenbasis $\{\psi_i\}$ of $\bm K_q$ satisfy
\[
f^* = \sum_{i=1}^\infty \lambda_i^\alpha \langle g, \psi_i \rangle\, \psi_i,
\qquad \sum_{i=1}^\infty \langle g, \psi_i \rangle^2 \leq R^2.
\]
Writing $\theta_i = \langle f^*, \psi_i \rangle = \lambda_i^\alpha \langle g, \psi_i \rangle$,
the constraint becomes
\[
\sum_{i=1}^\infty \frac{\theta_i^2}{\lambda_i^{2\alpha}} \leq R^2,
\]
which defines an ellipsoid $\mathcal{E}(\alpha, \beta, R)$ in $L_2(\mu_X)$ with semi-axes
$r_i = R\,\lambda_i^\alpha \asymp R\, i^{-\alpha\beta}$ under (C2).
\paragraph{Reduction to a finite-dimensional subproblem.}
We apply a block-Fano construction (see, e.g., \citet{tsybakov2008}, Ch.~2). Let $a$
be a dimension parameter to be chosen. Rather than perturbing the first $a$ eigendirections,
we restrict perturbations to the high-frequency block $\mathcal{B}_a = \{a+1, \ldots, 2a\}$,
so that all terms scale identically. Consider functions of the form
\[
f_\omega = \delta \sum_{i=a+1}^{2a} \omega_i\, i^{-\alpha\beta}\, \psi_i,
\qquad \omega \in \{-1, +1\}^a,
\]
where $\delta > 0$ is to be chosen. Since all indices $i \in \mathcal{B}_a$ satisfy
$i \asymp a$, we have $i^{-2\alpha\beta} \asymp a^{-2\alpha\beta}$ uniformly over the block.
\textbf{Ellipsoid constraint.}
Substituting $f_\omega$ into the ellipsoid definition $\sum_i \theta_i^2 / \lambda_i^{2\alpha} \leq R^2$
with $\theta_i = \delta\,\omega_i\,i^{-\alpha\beta}$ and $\lambda_i^{2\alpha} \asymp i^{-2\alpha\beta}$:
\[
\sum_{i=a+1}^{2a} \frac{(\delta\,\omega_i\,i^{-\alpha\beta})^2}{i^{-2\alpha\beta}}
= \delta^2 \sum_{i=a+1}^{2a} 1 = a\delta^2.
\]
Thus $f_\omega \in \mathcal{E}(\alpha, \beta, R)$ requires $\delta \leq R / \sqrt{a}$,
which we enforce by setting $\delta = R/\sqrt{a}$.
\paragraph{Separation.}
For $i \in \mathcal{B}_a$, $i^{-2\alpha\beta} \asymp a^{-2\alpha\beta}$. For distinct
$\omega, \omega' \in \{-1,+1\}^a$ differing in at least one coordinate:
\[
\|f_\omega - f_{\omega'}\|^2_{L_2}
= \delta^2 \sum_{i:\,\omega_i \neq \omega'_i} 4\,i^{-2\alpha\beta}
\gtrsim \delta^2\, a^{-2\alpha\beta},
\]
since at least one term contributes and all terms are $\asymp a^{-2\alpha\beta}$.
By the Gilbert--Varshamov lemma \citep[Ch.~4]{guruswami2022essential}, there exists $\Omega \subseteq \{-1,+1\}^a$ with
$|\Omega| \geq 2^{a/8}$ such that any two distinct elements differ in at least $a/8$
coordinates. For $\omega, \omega' \in \Omega$:
\[
\|f_\omega - f_{\omega'}\|^2_{L_2}
\gtrsim \delta^2 \cdot \frac{a}{8} \cdot a^{-2\alpha\beta}
= \frac{\delta^2}{8}\, a^{1-2\alpha\beta}.
\]
\paragraph{KL control.}
Under (B1), $Y \in [0,1]$, so the conditional distributions $P(Y\mid\bm X=\bm x)$ have
bounded support. For the Fano construction, we may choose conditional distributions
supported on the unit interval with prescribed means $f_\omega(\bm x)$, for which the
per-observation KL divergence satisfies
\begin{align*}
\mathrm{KL}(P_{Y|\bm x}^{f_\omega} ~\|~ P_{Y|\bm x}^{f_{\omega'}}) \leq C(f_\omega(\bm x) - f_{\omega'}(\bm x))^2
\end{align*}
(see, e.g., \citep[§2.4]{tsybakov2008}). Integrating over $\mathcal{X}$ and taking the
product over $N$ i.i.d.\ observations:
\[
\mathrm{KL}(P_{f_\omega}^N ~\|~ P_{f_{\omega'}}^N)
\lesssim N\|f_\omega - f_{\omega'}\|^2_{L_2}
\lesssim N\delta^2\, a^{1-2\alpha\beta}.
\]
Note that, because all indices lie in $\mathcal{B}_a$, the sum $\sum_{i=a+1}^{2a} i^{-2\alpha\beta}
\asymp a \cdot a^{-2\alpha\beta} = a^{1-2\alpha\beta}$ exactly, with no reliance on
$p$-series convergence arguments. In particular, this calculation is valid for all $\alpha\beta > 0$,
and in particular for $\alpha\beta > 1/2$ as required by the theorem.
\paragraph{Applying Fano's inequality.}
By the generalized Fano lemma, the minimax risk over $\Omega$ is bounded from below:
\[
\inf_{\tilde{f}} \sup_{f_\omega \in \Omega}
\mathbb E\bigl[\|f_\omega - \tilde{f}\|^2_{L_2}\bigr]
\geq \frac{\delta^2\, a^{1-2\alpha\beta}}{8}
\Bigl(1 - \frac{\overline{\mathrm{KL}} + \log 2}{\log |\Omega|}\Bigr),
\]
where $\overline{\mathrm{KL}}$ denotes the average pairwise KL divergence. Using
$\log|\Omega| \geq a/8$ and requiring $\overline{\mathrm{KL}} \leq a/16$ to keep the
Fano term bounded away from zero, we need
\[
N\delta^2\, a^{1-2\alpha\beta} \lesssim a,
\qquad\text{i.e.,}\qquad
\delta^2 \lesssim \frac{a^{2\alpha\beta}}{N}.
\]
Substituting the constraint $\delta^2 = R^2/a$ (from the ellipsoid requirement), this
becomes $R^2/a \lesssim a^{2\alpha\beta}/N$, i.e.\ $a^{1+2\alpha\beta} \gtrsim NR^2$,
which is satisfied by the choice of $a$ below.
\paragraph{Optimizing.}
The lower bound on the minimax risk is of order $\delta^2 a^{1-2\alpha\beta}$.
Substituting $\delta^2 = R^2/a$:
\[
\inf_{\tilde{f}} \sup_{f^* \in \mathcal{E}}
\mathbb E\bigl[\|f^* - \tilde{f}\|^2_{L_2}\bigr]
\gtrsim \frac{R^2}{a} \cdot a^{1-2\alpha\beta}
= R^2\, a^{-2\alpha\beta}.
\]
To balance the KL constraint with this risk, we set $\delta^2 \asymp a^{2\alpha\beta}/N$,
which together with $\delta^2 = R^2/a$ gives $a \asymp (NR^2)^{1/(2\alpha\beta+1)}$.
For fixed $R$, choosing $a \asymp N^{1/(2\alpha\beta+1)}$ yields the lower bound:
\begin{equation}\label{eq:lower}
\inf_{\tilde{f}} \sup_{f^* \in \mathcal{E}(\alpha,\beta,R)}
\mathbb E\bigl[\|f^* - \tilde{f}\|^2_{L_2}\bigr]
\gtrsim N^{-2\alpha\beta/(2\alpha\beta+1)}.
\end{equation}
\subsection*{Combining upper and lower bounds}
Note that assumption (C3) constrains the RF operator family, not the regression function $f^*$. The source class over which the minimax risk is computed is therefore $\mathcal E(\alpha, \beta, R)$, as defined by (C1)–(C2). The upper bound holds for any RF estimator whose kernel family additionally satisfies (C3).
The upper bound (Eq. \ref{eq:upper}) and lower bound (Eq. \ref{eq:lower}) match exactly in rate:
\[
\mathbb E\bigl[\|f^* - \hat{f}_{\mathrm{RF}}\|^2_{L_2}\bigr]
\lesssim N^{-2\alpha\beta/(2\alpha\beta+1)}
\asymp \inf_{\tilde{f}} \sup_{f^* \in \mathcal{E}}
\mathbb E\bigl[\|f^* - \tilde{f}\|^2_{L_2}\bigr].
\]
The RF estimator $\hat{f}_{\mathrm{RF}}$ therefore achieves the minimax optimal rate over
the source class $\mathcal{E}(\alpha, \beta, R)$. \qed
\subsection*{Sobolev specialization}
The spectral formulation of Thm.~\ref{thm:minimax} recovers classical nonparametric rates as a special
case. When the limiting RF kernel admits a Fourier-type eigenbasis, the source class
$\mathcal{E}(\alpha, \beta, R)$ reduces to a Sobolev ellipsoid, and our rate matches the
well-known result of \citet{stone1982}.
\begin{corollary}[Sobolev rate]
\label{cor:sobolev}
Under the conditions of Thm.~\ref{thm:minimax}, suppose the limiting operator $\bm K_q$ has eigenvalues $\lambda_i \asymp i^{-2s/d}$ for some $s > d/2$, and that $f^* \in W^{s,2}(\mathcal X)$. Then the RF estimator achieves
\[
\mathbb E\bigl[\|f^* - \hat{f}_{\mathrm{RF}}\|^2_{L_2}\bigr]
\lesssim N^{-2s/(2s+d)},
\]
which is minimax optimal over $W^{s,2}(\mathcal X)$.
\end{corollary}
\textbf{Proof.}
The Sobolev space $W^{s,2}(\mathcal X)$ is characterized by Fourier coefficients
$\theta_i = \langle f^*, \psi_i \rangle$ satisfying
$\sum_i i^{2s/d} \theta_i^2 < \infty$. Under the eigenvalue scaling
$\lambda_i \asymp i^{-2s/d}$ (i.e., $\beta = 2s/d$ in (C2)), the source condition (C1) with
exponent $\alpha$ gives semi-axes $a_i \asymp i^{-\alpha \cdot 2s/d}$. Matching the Sobolev
ellipsoid requires $\alpha \cdot 2s/d = s/d$, i.e., $\alpha = 1/2$.
The condition $s > d/2$ is necessary: Thm.~\ref{thm:minimax} requires $\alpha\beta > 1/2$, and substituting $\alpha = 1/2$, $\beta = 2s/d$ gives $\alpha\beta = s/d$, so the theorem's hypothesis is equivalent to $s/d > 1/2$, i.e.\ $s > d/2$.
Substituting $\alpha = 1/2$ and $\beta = 2s/d$ into the rate of Thm. \ref{thm:minimax}:
\[
N^{-2\alpha\beta/(2\alpha\beta + 1)}
= N^{-2 \cdot \frac{1}{2} \cdot \frac{2s}{d} \,/\,
(2 \cdot \frac{1}{2} \cdot \frac{2s}{d} + 1)}
= N^{-\frac{2s/d}{2s/d + 1}}
= N^{-2s/(2s + d)}.
\]
This is the classical minimax rate over $W^{s,2}(\mathcal X)$ \citep{stone1982}. Note that
$\alpha = 1/2 \leq 1$, so the source condition is satisfied. The optimal leaf count is
$M \asymp N^{d/(2s+d)}$, which grows with $N$, as expected.
\paragraph{Remark 1: Eigenbasis alignment.}
This corollary identifies the source class $\mathcal{E}(\alpha, \beta, R)$ with the Sobolev ellipsoid $W^{s,2}(\mathcal{X})$ by matching the eigenvalue decay $\lambda_i \asymp i^{-2s/d}$ of the RF kernel operator $\bm K_q$ with the spectral asymptotics of the Laplacian on $\mathcal X$. This identification implicitly assumes that the eigenfunctions $\{\psi_i\}$ of $\bm K_q$ align with the Fourier basis---or more precisely, that they span the same ordered sequence of subspaces. This alignment is most natural when the marginal $\mu_X$ is uniform on $\mathcal X$. In this case, the symmetry of the domain and the measure together with the axis-aligned geometry of RF splits suggest that the leading eigenfunctions of $\bm K_q$ should approximate trigonometric functions, with low-frequency directions corresponding to the dominant split coordinates.
For non-uniform $\mu_X$, the eigenbasis of $\bm K_q$ will generally differ from the Fourier basis, reflecting the geometry of $\mu_X$ rather than that of the ambient domain.
The corollary is best interpreted as showing that \textit{when the RF kernel happens to see Sobolev-type geometry}, the resulting rate recovers \citet{stone1982}'s classical result.
\paragraph{Remark 2: Parameterization.}
The Sobolev specialization makes the role of each parameter transparent: $s$ controls
function smoothness, $d$ controls ambient dimension, and $\beta = 2s/d$ encodes the
effective complexity of the kernel. The fact that $\alpha = 1/2$ places $f^*$ in the RKHS
$\mathcal{H}_k$ is natural---it says the regression function is ``as smooth as the kernel
can represent,'' which is the typical regime for kernel methods. More generally,
Thm. \ref{thm:minimax} applies beyond Sobolev classes to any source class with polynomial spectral
decay, including settings where the kernel adapts to low-dimensional structure and the
ambient dimension $d$ does not enter the rate.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experimental Setup}\label{appx:exp}
RFs and GBMs were trained using the \texttt{scikit-learn} package. Exact matrix decompositions were computed with \texttt{numpy}, while approximate decompositions utilized \texttt{scipy}. The SCATE model was implemented within the \texttt{torch} framework. Finally, standard Python libraries were used to evaluate metrics such as model size and inference time.
While the complexity of our proposed method is independent of the base model's size, the majority of the pruning techniques we benchmark against extract information directly from the ensemble's internal rules and structure. Consequently, employing an overly large model would unfairly increase the compression difficulty for baseline methods, while offering us an advantage at no cost. To ensure a fair comparison, we standardized the hyperparameters across all RFs and GBMs, aligning them closely with the configurations used in \citep{devos2025}. Specifically, RFs were configured with \texttt{n\_estimators=250}, \texttt{max\_depth=15}, while GBMs were trained with \texttt{n\_estimators=100}, \texttt{max\_depth=6}. For GBMs, we additionally set \texttt{base\_score=0} to be correct to the definition of the smoother matrix $\bm{S}$.
For our experiments, we assembled small-to-medium datasets from three sources: the UCI Machine Learning Repository \citep{Dua:2019}, OpenML \citep{OpenML2013}, and Kaggle. To equitably demonstrate the performance of all methods, these datasets were chosen to provide a balanced mix of classification and regression tasks. During preprocessing, we remove all missing values and exclude temporal features, which tree-based ensembles cannot handle natively. Detailed dataset characteristics---including the number of samples, feature types, and task categories---are summarized in Table \ref{tab:datasets}.
\begin{table}[htbp] \small
\centering
\caption{Summary of datasets used. The task column indicates whether the dataset is used for Classification or Regression.}
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{l l r r r r l}
\toprule
\textbf{Dataset} & \textbf{Code} & \textbf{\#Samples} & \textbf{\#Numerical} & \textbf{\#Categorical} & \textbf{\# Total} & \textbf{Task} \\
\midrule
\href{https://www.cs.toronto.edu/\%7Edelve/data/boston/bostonDetail.html}{Boston Housing} & \texttt{boston} & 506 & 13 & 0 & 13 & Regression \\
\href{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html}{California Housing} & \texttt{california} & 20640 & 8 & 0 & 8 & Regression \\
\href{https://archive.ics.uci.edu/dataset/165/concrete+compressive+strength}{Concrete} & \texttt{concrete} & 1030 & 8 & 0 & 8 & Regression \\
\href{https://scikit-learn.org/stable/api/sklearn.datasets.html}{Friedman} & \texttt{friedman\_1} & 1000 & 10 & 0 & 10 & Regression \\
\href{https://archive.ics.uci.edu/dataset/186/wine+quality}{Wine Quality} & \texttt{wq} & 4898 & 12 & 0 & 12 & Regression \\
\href{https://archive.ics.uci.edu/dataset/320/student+performance}{Student Performance} & \texttt{student} & 649 & 16 & 17 & 33 & Regression \\
\href{https://archive.ics.uci.edu/dataset/1/abalone}{Abalone} & \texttt{abalone} & 4177 & 8 & 1 & 9 & Regression \\
\href{https://archive.ics.uci.edu/dataset/9/auto+mpg}{Auto MPG} & \texttt{auto\_mpg} & 398 & 7 & 0 & 7 & Regression \\
\href{https://www.openml.org/search?type=data\&sort=runs\&id=1462\&status=active}{Banknote Auth.} & \texttt{banknote} & 1372 & 4 & 1 & 5 & Classification \\
\href{https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic}{Breast Cancer} & \texttt{bc} & 570 & 30 & 1 & 32 & Classification \\
\href{https://archive.ics.uci.edu/dataset/34/diabetes}{Diabetes} & \texttt{diabetes} & 768 & 8 & 1 & 9 & Classification \\
\href{https://archive.ics.uci.edu/dataset/94/spambase}{Spambase} & \texttt{spambase} & 4601 & 58 & 1 & 59 & Classification \\
\href{https://archive.ics.uci.edu/dataset/2/adult}{Adult} & \texttt{adult} & 45222 & 5 & 9 & 14 & Classification \\
\href{https://www.kaggle.com/datasets/blastchar/telco-customer-churn}{Telco Churn} & \texttt{telco} & 7032 & 3 & 17 & 20 & Classification \\
\href{https://www.openml.org/search?type=data\&status=active\&id=46911}{Bank Customer Churn} & \texttt{churn} & 10000 & 6 & 5 & 11 & Classification \\
\bottomrule
\end{tabular}
\end{adjustbox}
\label{tab:datasets}
\end{table}
We describe the setup used for the experiments in Sect. \ref{sec:exp}. All experiments are run with 10 repetitions each (for every method/dataset combination, with different data splitting and model seeds).
\subsection{Spectra and Generalization} To evaluate the general performance of SCATE, we use all datasets except \texttt{adult} and \texttt{california}. Because our theoretical analysis is limited to RFs, this experiment focuses exclusively on them. For each dataset, we performed the following steps:
\begin{itemize}[noitemsep]
\item After splitting the data (train, validation, test) and fitting an RF to the training set, we constructed the RF kernel $\bm{K}$ and computed its exact eigendecomposition. The resulting eigenvalues were sorted to generate a decay curve. We also computed the cross-kernel matrices $\bm{K}_{train, test}$ and $\bm{K}_{train, validation}$.
\item We evaluate varying numbers of learned eigenfunctions, $P$, using the sequence \texttt{[2, 5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 150, 175, 200]}. We capped $P$ at 200, as subsequent eigenfunctions correspond to very small eigenvalues and contain minimal information.
\item \textbf{Oracle Baseline:} For each value of $P$, we calculate the Oracle baseline (Sect.~\ref{sec:compression}) by computing its approximation of $\bm{K}_{train, test}$ and its resulting predictions on the test data.
\item For each $P$, we train two separate SCATE models (Sect.~\ref{sec:compression}) for 200 epochs to learn the eigenfunction mappings: a small model (\texttt{width=16}, \texttt{depth=2}) and a large model (\texttt{width=128}, \texttt{depth=4}).
\item The test data is passed through each model to generate output eigenfunctions. These are used to compute the approximated kernel $\hat{\bm{K}}_{train\_test}$, which was then multiplied by the training labels $y$ to produce predictions. Performance is measured using accuracy for classification and $R^2$ for regression.
\end{itemize}
The resulting visualizations from this process are presented in Fig.~\ref{fig:visualisation}.
\subsection{Na\"ive Benchmark}In this experiment, we evaluate SCATE against two simple distillation baselines for RFs:
\begin{itemize}[noitemsep]
\item \textbf{Na\"ive MLP:} This approach uses the identical network architecture as SCATE. However, instead of learning eigenfunctions, it is trained to directly predict the output of each tree in the base model. This direct distillation scheme is inspired by \citep{hinton_distilling2}.
\item \textbf{Na\"ive RF:} A naturally smaller RF created by restricting the \texttt{max\_depth} and \texttt{n\_estimators} hyperparameters. Because default \texttt{scikit-learn} objects store metadata beyond what is needed for inference, we extract only the essential matrices into a custom \texttt{minimal\_RF} object to ensure an accurate model size comparison.
\end{itemize}
We then benchmark SCATE against the Na\"ive MLP, the Na\"ive RF, and the Base RF, on all datasets excluding \texttt{adult} and \texttt{california}, using the following pipeline:
\begin{itemize}[noitemsep]
\item We split the data, train the baseline RF, and construct the kernel exactly as described in the visualization experiment. We additionally record the performance of the baseline RF, as a measure of the performance we expect to converge to for all models.
\item Based on earlier visualization results (Fig.~\ref{fig:visualisation}) showing that SCATE learns most effectively up to 50 eigenfunctions, we fix $P=50$. Instead of varying $P$, we sweep the network architectures for SCATE and the Na\"ive MLP using combinations of \texttt{widths=[4, 8, 16, 32, 64, 128]} and \texttt{depths=[1, 2, 3, 4, 5]}.
\item Concurrently, we evaluate the Na\"ive RF baselines across combinations of \texttt{n\_estimators=[10, 50, 100, 200, 500]} and \texttt{max\_depth=[3, 4, 5, 6, 7, 8, 9, None]}.
\item We train the models on their respective targets (eigenfunctions for SCATE, tree outputs for the MLP, and raw data for the Na\"ive RF). We then record both the predictive performance and the final model size for every configuration.
\item For each dataset and method, we isolate the best-performing configurations at different model sizes to construct a convex Pareto frontier. Because model sizes scale differently across methods, some curves may contain very few data points if a majority of the configurations fall below the optimal frontier.
\end{itemize}
The resulting Pareto curves are plotted in Fig.~\ref{fig:naive}.
\subsection{Compression Benchmark}
For the compression benchmark, we test how well different methods perform compression. We focus on SCATE and FP-SCATE (which is SCATE applied to a model already shrunk by ForestPrune) under two strict memory limits: 10KB and 100KB. Unlike the earlier experiments, we also include GBMs here to compare performance.
Given all datasets in the pipeline, our benchmark proceeds as follows:
\begin{itemize}[noitemsep]
\item Similar to the earlier tests, we start by splitting the data, training a standard Random Forest, and creating the kernel matrix $\bm{K}$ for each dataset.
\item We test a wide range of settings (hyperparameters) for each method to see how they balance model size against accuracy. For every combination of settings and dataset, we record the compressed model's size and performance. Since different algorithms output different types of models, we measure size fairly by only counting size of the bare minimum objects needed to make predictions.
\item For each method and dataset, we find the best-performing model that fits under the 10KB and 100KB limits. The score of this best model becomes our final result for that budget, and we summarize these results in Table~\ref{tab:rf}.
\item Finally, we repeat this exact same process for the GBM models, constructing a smoother matrix $\bm{S}$ instead of an RF kernel. The results for the GBM tests are shown in Table~\ref{tab:gbm}.
\end{itemize}
Each method ends up being run around $50$ times for each dataset and seed, and the runs are doubled, since we evaluate on both RFs and GBMs, resulting in $50 \times 15 \times 10 \times 2 = 15000$ runs for each method, and we benchmark $7$ methods in total, resulting in $\sim 10^5$ runs in total. To address these computational demands, we run the compression benchmark from a high performance computing partition. Other experiments were run from a Macbook Pro, with an M5 Pro chip with 18-core CPU and 20-core GPU, and 24GB RAM. In order to accurately compare training and inference time between methods, we conduct a small runtime experiment on this machine, which can be found in Appx. \ref{appx:runtime}
This section details the implementation of each method included in our compression benchmark. For all baseline methods, we outline any structural modifications required to adapt them to our experimental framework, alongside the hyperparameter grids searched to identify the optimal model for each dataset.
\begin{itemize}
\item \textbf{SCATE:} Following the setup used in our na\"ive benchmark, we fix $P=50$ and evaluate configurations across \texttt{widths} $\in \{4, 8, 16, 32, 64, 128\}$ and \texttt{depths} $\in \{1, 2, 3, 4, 5\}$. For GBM compression, because our smoother matrix $\textbf{S}$ is strictly defined for MSE loss, we train a regressor for binary classification tasks and interpret the continuous outputs as linear probabilities.
\item \textbf{FP:} The ForestPrune algorithm \citep{liu2023} lacks native support for classification tasks. To address this, we apply the same workaround used in our own models by fitting a regressor for binary classification. We identify the optimal model by sweeping a hyperparameter grid generated via \texttt{np.flip(np.logspace(-2, 1.5, 50))}, mirroring the configuration in the algorithm's official implementation.
\item \textbf{FP-SCATE:} This pipeline sequentially applies FP followed by SCATE. We first isolate the best-performing FP model using the hyperparameter sweep described above. We then compress that specific model using SCATE, evaluating it over the same hyperparameter grid as base SCATE.
\item \textbf{LOP:} For the LOP method \citep{devos2025}, we search for the best model by sweeping tolerance values continuously from 0.01 to 1. To manage computational overhead, we impose a strict 10-minute timeout per run. In practice, this limit was only triggered a handful of times on the most computationally demanding datasets.
\item \textbf{FIRE and TE:} Similar to FP, neither the FIRE \citep{liu2023fire} nor TreeExtract \citep{liu2025extracting} algorithms natively handle compression for classification. Consequently, we employ the same regressor workaround for binary classification tasks. For FIRE, we sweep the $\alpha$ hyperparameter across values from 10 to 100. For TE, we evaluate models across a $\lambda$ range defined by \texttt{np.flip(np.logspace(1, 4, 50))}.
\item \textbf{SSF:} Because the SSF algorithm \citep{alkhoury2024splitting} is designed exclusively for compressing Random Forests, we exclude it from our GBM benchmarks. We determine the optimal SSF models by sweeping threshold values between 0 and 0.5, which covers the entire hyperparameter range specified in the original paper.
\end{itemize}
\section{Additional Results}\label{appx:extra_results}
We present additional results from the visualization experiment and the na\"ive benchmark, experiments comparing different methods of learning eigenfunctions, runtime experiments, and evaluations of assumptions made for our proof.
\subsection{Evaluating the C2 Assumption}\label{appx:c2}
Recall that Assumption C2 requires the operator $\bm{K}_q$ to have eigenvalues $\lambda_i \asymp i^{-\beta}$ for some $\beta > 1$. To evaluate this, we train a Random Forest (RF) on each benchmark dataset (Table \ref{tab:datasets}) and derive its empirical RF kernel. We then compute the eigendecomposition of these kernels and sort the eigenvalues in descending order. We then fit a regression curve to the eigenvalues as a function of their rank index. Specifically, we fit a linear regression of $\log \lambda$ on $\log i$, such that $\beta$ is the negative coefficient. We extract this decay parameter to check if it satisfies the $\beta > 1$ condition. The empirical decay and the estimated coefficients are plotted for visual analysis in Fig. \ref{fig:c2}. Note that we restrict our analysis to the top 100 eigencomponents; both since the empirical kernel merely approximates the true operator $\bm{K}_q$, causing the later components become increasingly noisy; and also to due to some datasets being relatively small in sample size.
\begin{figure}[htbp]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/regression_eigenvalue_decay.pdf}
\caption{Regression Datasets}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/classification_eigenvalue_decay.pdf}
\caption{Classification Datasets}
\end{subfigure}
\vspace{1em}
\caption{Empirical evaluation of the polynomial decay assumption (C2), across both regression (a)
and classification (b) datasets.}
\label{fig:c2}
\end{figure}
Surprisingly, our visual analysis reveals that Assumption C2 does not generally hold for the datasets in the benchmark, though the \texttt{bc} and \texttt{banknote} datasets appear to approach polynomial decay. Despite this widespread violation of the assumption, SCATE maintains competitive performance even in scenarios where $\beta < 1$. This suggests that while C2 constitutes a sufficient condition for strong model performance, it is not necessary condition. Ultimately, these results suggest that SCATE is highly robust to deviations from the polynomial decay assumption.
\subsection{Evaluating Different Methods for Learning Eigenfunctions}\label{appx:neural_exp}
We evaluate and compare the performance of two existing neural eigenfunction estimation approaches—NeuralEF \citep{deng2022neuralef} and NeuralSVD \citep{ryu2024operator}—against our proposed method, SCATE. Our evaluation follows two experimental setups: first, a capacity-constrained setting similar to the na\"ive benchmark, where we fix the model size and vary the number of target eigenfunctions $P$; second, a budget-constrained setting similar to the compression benchmark, where we fix $P$ and vary model sizes to identify the optimal configuration. Due to the relatively high computational demands of NeuralEF and NeuralSVD, we restrict this comparative analysis to five small datasets: \texttt{auto\_mpg}, \texttt{student}, \texttt{boston}, \texttt{bc}, and \texttt{diabetes}, serving as an illustrative side-by-side comparison. For SCATE, we maintain the experimental settings detailed in Sect. \ref{sec:compression} and Appx. \ref{appx:exp}.
Because the primary contributions of NeuralEF and NeuralSVD are novel learning objectives rather than specific model architectures, their objectives can theoretically be adapted to any neural network. However, to ensure a fair comparison, we utilize the network designs specified in each method's official implementation. We briefly summarize their methodologies below:
\begin{itemize}
\item \textbf{NeuralEF:} NeuralEF formulates kernel eigendecomposition as a series of asymmetric maximization problems. Because this formulation relies on the deflation of previously computed modes, the optimization exhibits a strict hierarchical dependency where latter eigenfunctions depend on the convergence of earlier ones. While their original objective is a constrained optimization problem, in practice, NeuralEF relaxes orthogonality into explicit penalty terms within the loss function and enforces unit-norm requirements via a $L_2$ batch normalization layer. This structure requires training a neural network for each of the $P$ eigenfunctions it aims to learn. Furthermore, because its objective is fundamentally rooted in maximizing the Rayleigh quotient, NeuralEF is strictly confined to self-adjoint operators. As a result, it cannot compute the singular value decomposition (SVD) of kernels, limiting its applicability in our context exclusively to Random Forest (RF) compression.
\item \textbf{NeuralSVD:} In contrast, NeuralSVD frames spectral decomposition as a fully unconstrained optimization task based on Schmidt’s low-rank approximation theorem. By introducing a "NestedLoRA" objective, it recovers the ordered singular functions by breaking the symmetry of the low-rank approximation. This approach provides several improvements over NeuralEF. First, NeuralSVD eliminates the need for explicit orthogonality penalties and normalization layers; instead, it learns unnormalized singular functions that naturally absorb the scale of their corresponding singular values. Second, rather than using a sequentially trained network, NeuralSVD employs a joint nesting technique that unifies the learning of all $P$ modes into a single global objective function. This allows the model to simultaneously and optimize all functions within a single, shared neural network, similar to SCATE. Finally, because its objective natively supports SVD, NeuralSVD is capable of compression of both RFs and Gradient Boosting Machines (GBMs).
\end{itemize}
\begin{table}[htbp]
\centering
\caption{RF compression benchmark for the three eigenfunction learning methods, averaged over $10$ trials. Best result per row in bold. NA indicates that the model could not reach the target size.}
\label{tab:neural_rf}
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\resizebox{\textwidth}{!}{
\begin{tabular}{lc|ccc|ccc}
\toprule
& & \multicolumn{3}{c|}{\textbf{Size = 10KB}} & \multicolumn{3}{c}{\textbf{Size = 100KB}} \\
\cmidrule{3-8}
\textbf{Dataset} & \textbf{RF} & \textbf{NeuralEF} & \textbf{NeuralSVD} & \textbf{SCATE} & \textbf{NeuralEF} & \textbf{NeuralSVD} & \textbf{SCATE} \\
\midrule
\texttt{auto\_mpg} & 0.69 & 0.086 $\pm$ 0.011 & 0.237 $\pm$ 0.020 & \textbf{0.453 $\pm$ 0.012} & 0.192 $\pm$ 0.005 & 0.320 $\pm$ 0.009 & \textbf{0.497 $\pm$ 0.009} \\
\texttt{bc} & 0.96 & 0.732 $\pm$ 0.018 & 0.804 $\pm$ 0.028 & \textbf{0.984 $\pm$ 0.003} & 0.881 $\pm$ 0.016 & 0.949 $\pm$ 0.008 & \textbf{0.987 $\pm$ 0.003} \\
\texttt{boston} & 0.85 & 0.149 $\pm$ 0.011 & 0.220 $\pm$ 0.015 & \textbf{0.775 $\pm$ 0.013} & 0.149 $\pm$ 0.011 & 0.252 $\pm$ 0.013 & \textbf{0.814 $\pm$ 0.012} \\
\texttt{diabetes} & 0.76 & 0.654 $\pm$ 0.011 & 0.658 $\pm$ 0.007 & \textbf{0.786 $\pm$ 0.010} & 0.654 $\pm$ 0.011 & 0.660 $\pm$ 0.007 & \textbf{0.794 $\pm$ 0.010} \\
\texttt{student} & 0.85 & NA & 0.116 $\pm$ 0.013 & \textbf{0.854 $\pm$ 0.014} & 0.134 $\pm$ 0.007 & 0.337 $\pm$ 0.018 & \textbf{0.857 $\pm$ 0.013} \\
\midrule
\textbf{Average Rank} & & 3.00& 2.00& \textbf{1.00} & 3.00 & 2.00 & \textbf{1.00} \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\linewidth]{figures/naive_neural.pdf}
\caption{Rate-performance Pareto curves across five benchmarkd datasets for three eigenfunction learning methods. Shaded regions indicate standard errors.}
\label{fig:naive_neural}
\end{figure}
From these results, we can see SCATE dominating compression performance. There are several reasons that may have caused subpar performance from the two competing methods, despite their theoretical complexity.
First, there is a fundamental difference in their optimization objectives: while NeuralEF and NeuralSVD are designed for mathematically exact spectral decomposition, SCATE is designed for kernel reconstruction. This can make a big difference in compression settings, where a network's learning capacity is heavily limited. In particular, both NeuralEF and NeuralSVD dedicate a substantial proportion of their capacity towards learning eigenvectors that are strictly orthogonal. This objective may be too complex for the limited learning capacity of small networks and is better suited for more expressive models.
In contrast, SCATE treats orthogonality as a soft penalty. Since top eigencomponents have strong signals, we do not necessarily need the orthogonality property to learn the top eigenfunctions effectively. Ultimately, this performance gap does not imply that NeuralEF or NeuralSVD are inferior methods; rather, their objectives and motivations are more appropriate for different settings than our own. As a consequence, we select SCATE as the primary method for learning the eigenfunctions in our RF compression experiments.
\subsection{Additional Visualization Results}\label{appx:extra_vis}
Complementing the two-dataset visualization in Fig. \ref{fig:visualisation}, Fig. \ref{fig:vis_full_reg} (regression) and Fig. \ref{fig:vis_rull_cl} (classification) show the eigenvalue decay, kernel approximation error, and predictive performance for the thirteen remaining datasets in the benchmark.
\begin{figure}[htbp]
\centering
\includegraphics[width=\linewidth]{figures/vis_appendix.pdf}
\caption{Spectral decay, kernel approximation error, and predictive performance across the remaining six regression datasets in the benchmark. Shaded regions indicate standard errors.}
\label{fig:vis_full_reg}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=\linewidth]{figures/vis_appendix_cl.pdf}
\caption{Spectral decay, kernel approximation error, and predictive performance across the remaining seven classification datasets in the benchmark. Shaded regions indicate standard errors.}
\label{fig:vis_rull_cl}
\end{figure}
These findings further illustrate that substantial improvements in compression performance are closely associated with a steep decay in the kernel's eigenvalues. Moreover, SCATE consistently matches Oracle performance, demonstrating robustness even when the kernel approximation error stops improving due to noise in the trailing eigencomponents.
\subsection{Additional Na\"ive Benchmark Results}\label{appx:extra_naive}
In Figure \ref{fig:naive_full}, we present the Pareto frontiers for all remaining datasets evaluated in the na\"ive benchmark, supplementing the initial results shown in Figure \ref{fig:naive}. These plots demonstrate that SCATE consistently serves as the dominant method for RF compression compared to na\"ive alternatives.
While there are isolated instances where baseline methods achieve higher scores, these occurrences typically coincide with anomalous scenarios where the compressed models outperform the uncompressed Base RF. While these performance gains are possible, they are neither predictable nor broadly replicable, and thus cannot be relied on in practice. In contrast, SCATE maintains stable performance across diverse scenarios, validating its robustness as a highly effective methodology for RF compression.
\begin{figure}[htbp]
\centering
% Figure 1
\includegraphics[width=\linewidth]{figures/naive_grid_12_datasets.pdf}
\caption{Rate-performance Pareto curves across the remaining twelve benchmark datasets. Shaded regions indicate standard errors.}
\label{fig:naive_full}
\end{figure}
\newpage
\subsection{Comprehensive Compression Benchmark Results}\label{appx:big_table}
As a supplement to Tables \ref{tab:rf} and \ref{tab:gbm}, we present the complete compression benchmark results in Table \ref{tab:full}. This expanded table reports the standard errors for all evaluated methods across each dataset. To identify the most competitive approaches, methods achieving performance within one standard error of the top-performing model are jointly highlighted in bold, though they remain ordered by their absolute mean performance. Notably, while the mean performance of several baseline models may initially appear comparable to our own, a more granular analysis incorporating these standard errors reveals that our method is superior, in the vast majority of scenarios, and validates our empirical rankings. We additionally report the size of the base RF and GBM models, which for 10KB show a compression factor around $10^3$ for RFs, and $10^2$ for GBMs, which is relatively inline with the values often tested for within the literature \citep{devos2025, liu2023}. To be fair with previous works, which reported leaf count as a measure of size, we strip down each base model to simply the matrices needed to perform inference (features, thresholds, left, right, and values) before computing its size.
\clearpage
\begin{sidewaystable}[htbp]
\centering
\caption{RF compression benchmark. Performance results at 10KB and 100KB memory budgets across different datasets and methods. Best result per row in bold, as well as results within a standard error of the best value}
\label{tab:rf_results}
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcc|ccccccc|ccccccc}
\toprule
~ & & & \multicolumn{7}{c|}{\textbf{Size = 10KB}} & \multicolumn{7}{c}{\textbf{Size = 100KB}} \\
\cmidrule{4-17}
\textbf{Dataset} & \textbf{RF} & \textbf{Base Size} & \textbf{ForestPrune} & \textbf{LOP} & \textbf{FIRE} & \textbf{TreeExtract} & \textbf{SSF} & \textbf{SCATE} & \textbf{FP-SCATE} & \textbf{ForestPrune} & \textbf{LOP} & \textbf{FIRE} & \textbf{TreeExtract} & \textbf{SSF} & \textbf{SCATE} & \textbf{FP-SCATE} \\
\midrule
\texttt{abalone} & 0.549 & 31.18 $\pm$ 0.110 & 0.347 $\pm$ 0.010 & 0.504 $\pm$ 0.009 & 0.282 $\pm$ 0.016 & NA & NA & \textbf{0.561 $\pm$ 0.004} & 0.543 $\pm$ 0.010 & 0.533 $\pm$ 0.004 & 0.511 $\pm$ 0.010 & 0.283 $\pm$ 0.016 & 0.548 $\pm$ 0.004 & NA & \textbf{0.568 $\pm$ 0.003} & 0.545 $\pm$ 0.010 \\
\texttt{adult} & 0.851 & 39.59 $\pm$ 0.200 & 0.792 $\pm$ 0.003 & 0.827 $\pm$ 0.003 & TO & TO & NA & \textbf{0.839 $\pm$ 0.001} & 0.834 $\pm$ 0.001 & 0.837 $\pm$ 0.001 & 0.835 $\pm$ 0.001 & TO & TO & \textbf{0.848 $\pm$ 0.001} & \textbf{0.847 $\pm$ 0.001} & 0.846 $\pm$ 0.001 \\
\texttt{auto\_mpg} & 0.694 & 1.09 $\pm$ 0.021 & 0.489 $\pm$ 0.016 & \textbf{0.632 $\pm$ 0.024} & \textbf{0.619 $\pm$ 0.027} & NA & 0.600 $\pm$ 0.025 & 0.453 $\pm$ 0.012 & 0.462 $\pm$ 0.010 & 0.578 $\pm$ 0.019 & \textbf{0.632 $\pm$ 0.024} & \textbf{0.619 $\pm$ 0.027} & 0.383 $\pm$ 0.007 & 0.601 $\pm$ 0.026 & 0.497 $\pm$ 0.009 & 0.511 $\pm$ 0.012 \\
\texttt{banknote} & 0.993 & 0.964 $\pm$ 0.010 & 0.874 $\pm$ 0.010 & 0.986 $\pm$ 0.004 & 0.990 $\pm$ 0.003 & NA & 0.999 $\pm$ 0.001 & \textbf{1.000 $\pm$ 0.000} & \textbf{1.000 $\pm$ 0.000} & 0.919 $\pm$ 0.011 & 0.986 $\pm$ 0.004 & 0.990 $\pm$ 0.003 & 0.892 $\pm$ 0.003 & \textbf{1.000 $\pm$ 0.000} & \textbf{1.000 $\pm$ 0.000} & \textbf{1.000 $\pm$ 0.000} \\
\texttt{bc} & 0.962 & 0.678 $\pm$ 0.014 & 0.950 $\pm$ 0.007 & 0.922 $\pm$ 0.027 & 0.955 $\pm$ 0.005 & NA & 0.976 $\pm$ 0.004 & \textbf{0.984 $\pm$ 0.003} & 0.980 $\pm$ 0.003 & 0.951 $\pm$ 0.007 & 0.922 $\pm$ 0.027 & 0.955 $\pm$ 0.005 & 0.920 $\pm$ 0.007 & 0.980 $\pm$ 0.003 & \textbf{0.987 $\pm$ 0.003} & \textbf{0.985 $\pm$ 0.003} \\
\texttt{boston} & 0.855 & 6.87 $\pm$ 0.020 & 0.600 $\pm$ 0.016 & \textbf{0.805 $\pm$ 0.023} & 0.684 $\pm$ 0.016 & NA & NA & 0.775 $\pm$ 0.013 & \textbf{0.790 $\pm$ 0.012} & 0.800 $\pm$ 0.015 & 0.818 $\pm$ 0.025 & 0.690 $\pm$ 0.016 & 0.810 $\pm$ 0.014 & 0.717 $\pm$ 0.013 & 0.814 $\pm$ 0.012 & \textbf{0.833 $\pm$ 0.011} \\
\texttt{california} & 129.4 & 0.309 $\pm$ 0.000 & 0.449 $\pm$ 0.027 & 0.684 $\pm$ 0.006 & TO & TO & NA & 0.700 $\pm$ 0.003 & \textbf{0.707 $\pm$ 0.006} & 0.676 $\pm$ 0.010 & \textbf{0.744 $\pm$ 0.005} & TO & TO & NA & \textbf{0.746 $\pm$ 0.004} & \textbf{0.743 $\pm$ 0.004} \\
\texttt{churn} & 0.862 & 31.67 $\pm$ 0.177 & 0.824 $\pm$ 0.004 & 0.844 $\pm$ 0.006 & 0.850 $\pm$ 0.002 & NA & NA & \textbf{0.859 $\pm$ 0.002} & 0.852 $\pm$ 0.007 & 0.853 $\pm$ 0.002 & 0.850 $\pm$ 0.004 & 0.850 $\pm$ 0.002 & 0.851 $\pm$ 0.002 & 0.828 $\pm$ 0.004 & \textbf{0.863 $\pm$ 0.002} & 0.855 $\pm$ 0.008 \\
\texttt{concrete} & 0.907 & 14.42 $\pm$ 0.025 & 0.504 $\pm$ 0.015 & 0.741 $\pm$ 0.021 & 0.731 $\pm$ 0.014 & NA & NA & 0.801 $\pm$ 0.008 & \textbf{0.820 $\pm$ 0.005} & 0.758 $\pm$ 0.014 & 0.855 $\pm$ 0.005 & 0.746 $\pm$ 0.014 & 0.894 $\pm$ 0.003 & \textbf{0.901 $\pm$ 0.004} & 0.862 $\pm$ 0.003 & 0.861 $\pm$ 0.003 \\
\texttt{diabetes} & 0.759 & 3.49 $\pm$ 0.040 & 0.757 $\pm$ 0.008 & 0.741 $\pm$ 0.014 & 0.765 $\pm$ 0.009 & NA & NA & 0.787 $\pm$ 0.010 & \textbf{0.794 $\pm$ 0.009} & 0.764 $\pm$ 0.009 & 0.741 $\pm$ 0.014 & 0.765 $\pm$ 0.009 & 0.661 $\pm$ 0.010 & 0.733 $\pm$ 0.007 & 0.794 $\pm$ 0.010 & \textbf{0.801 $\pm$ 0.009} \\
\texttt{friedman\_1} & 0.851 & 14.33 $\pm$ 0.016 & 0.474 $\pm$ 0.023 & 0.759 $\pm$ 0.012 & 0.482 $\pm$ 0.025 & NA & NA & \textbf{0.822 $\pm$ 0.006} & 0.811 $\pm$ 0.009 & 0.736 $\pm$ 0.014 & 0.825 $\pm$ 0.013 & 0.494 $\pm$ 0.025 & 0.794 $\pm$ 0.007 & NA & \textbf{0.863 $\pm$ 0.004} & 0.855 $\pm$ 0.006 \\
\texttt{spambase} & 0.949 & 7.22 $\pm$ 0.037 & 0.868 $\pm$ 0.005 & 0.916 $\pm$ 0.003 & \textbf{0.937 $\pm$ 0.002} & NA & NA & 0.929 $\pm$ 0.001 & 0.926 $\pm$ 0.002 & 0.913 $\pm$ 0.002 & 0.933 $\pm$ 0.002 & \textbf{0.944 $\pm$ 0.002} & 0.894 $\pm$ 0.003 & \textbf{0.944 $\pm$ 0.002} & 0.939 $\pm$ 0.001 & 0.936 $\pm$ 0.001 \\
\texttt{student} & 0.851 & 4.077 $\pm$ 0.014 & 0.644 $\pm$ 0.030 & 0.804 $\pm$ 0.015 & 0.710 $\pm$ 0.013 & NA & 0.818 $\pm$ 0.015 & \textbf{0.854 $\pm$ 0.014} & 0.836 $\pm$ 0.019 & 0.838 $\pm$ 0.014 & 0.805 $\pm$ 0.015 & 0.716 $\pm$ 0.013 & 0.791 $\pm$ 0.011 & 0.818 $\pm$ 0.015 & \textbf{0.857 $\pm$ 0.013} & 0.840 $\pm$ 0.019 \\
\texttt{telco} & 0.789 & 33.40 $\pm$ 0.142 & 0.747 $\pm$ 0.007 & 0.773 $\pm$ 0.008 & 0.789 $\pm$ 0.003 & NA & NA & \textbf{0.804 $\pm$ 0.003} & 0.800 $\pm$ 0.002 & 0.797 $\pm$ 0.002 & 0.775 $\pm$ 0.008 & 0.789 $\pm$ 0.003 & 0.789 $\pm$ 0.002 & 0.779 $\pm$ 0.004 & \textbf{0.808 $\pm$ 0.003} & 0.804 $\pm$ 0.002 \\
\texttt{wq} & 0.496 & 18.39 $\pm$ 0.071 & 0.236 $\pm$ 0.012 & 0.312 $\pm$ 0.006 & 0.307 $\pm$ 0.010 & NA & NA & \textbf{0.336 $\pm$ 0.008} & 0.332 $\pm$ 0.006 & 0.350 $\pm$ 0.010 & \textbf{0.358 $\pm$ 0.008} & 0.340 $\pm$ 0.008 & 0.281 $\pm$ 0.006 & 0.243 $\pm$ 0.010 & \textbf{0.362 $\pm$ 0.007} & 0.360 $\pm$ 0.007 \\
\midrule
\textbf{Average Rank} & & & 4.80 & 3.47 & 3.87 & 6.57 & 5.50 & \textbf{1.77} & 2.03 & 4.40 & 4.33 & 5.03 & 5.30 & 4.47 & \textbf{1.93} & 2.53 \\
\bottomrule
\end{tabular}
}
\vspace{1cm}
\caption{GBM compression benchmark. Performance results at 0.01 and 0.10 memory budgets across different datasets and methods. Best result per row in bold, as well as results within a standard error of the best value.}
\label{tab:gbm_results}
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcc|cccccc|cccccc}
\toprule
~ & & & \multicolumn{6}{c|}{\textbf{Size = 10KB}} & \multicolumn{6}{c}{\textbf{Size = 100KB}} \\
\cmidrule{4-15}
\textbf{Dataset} & \textbf{GBM} & \textbf{Base Size} & \textbf{FIRE} & \textbf{ForestPrune} & \textbf{FP-SCATE} & \textbf{LOP} & \textbf{SCATE} & \textbf{TreeExtract} & \textbf{FIRE} & \textbf{ForestPrune} & \textbf{FP-SCATE} & \textbf{LOP} & \textbf{SCATE} & \textbf{TreeExtract} \\
\midrule
\texttt{abalone} & 0.529 & 0.690 $\pm$ 0.004 & 0.465 $\pm$ 0.009 & 0.445 $\pm$ 0.004 & 0.547 $\pm$ 0.007 & 0.505 $\pm$ 0.006 & \textbf{0.562 $\pm$ 0.002} & 0.540 $\pm$ 0.003 & 0.465 $\pm$ 0.009 & 0.528 $\pm$ 0.004 & 0.552 $\pm$ 0.008 & 0.520 $\pm$ 0.007 & \textbf{0.571 $\pm$ 0.002} & 0.547 $\pm$ 0.004 \\
\texttt{adult} & 0.853 & 0.705 $\pm$ 0.003 & 0.814 $\pm$ 0.002 & 0.819 $\pm$ 0.002 & 0.846 $\pm$ 0.000 & 0.844 $\pm$ 0.002 & 0.846 $\pm$ 0.001 & \textbf{0.853 $\pm$ 0.001} & 0.814 $\pm$ 0.002 & 0.853 $\pm$ 0.001 & 0.853 $\pm$ 0.001 & 0.852 $\pm$ 0.001 & 0.853 $\pm$ 0.001 & \textbf{0.856 $\pm$ 0.001} \\
\texttt{auto\_mpg} & 0.658 & 0.500 $\pm$ 0.006 & \textbf{0.618 $\pm$ 0.032} & \textbf{0.602 $\pm$ 0.026} & 0.132 $\pm$ 0.333 & 0.580 $\pm$ 0.026 & 0.439 $\pm$ 0.016 & 0.379 $\pm$ 0.008 & 0.618 $\pm$ 0.032 & \textbf{0.691 $\pm$ 0.019} & 0.211 $\pm$ 0.342 & 0.580 $\pm$ 0.026 & 0.478 $\pm$ 0.010 & 0.379 $\pm$ 0.008 \\
\texttt{banknote} & 0.985 & 0.471 $\pm$ 0.023 & 0.982 $\pm$ 0.002 & 0.947 $\pm$ 0.008 & 0.998 $\pm$ 0.002 & 0.977 $\pm$ 0.005 & \textbf{1.000 $\pm$ 0.000} & 0.991 $\pm$ 0.002 & 0.982 $\pm$ 0.002 & 0.986 $\pm$ 0.002 & 0.998 $\pm$ 0.002 & 0.977 $\pm$ 0.005 & \textbf{1.000 $\pm$ 0.000} & 0.995 $\pm$ 0.001 \\
\texttt{bc} & 0.938 & 0.310 $\pm$ 0.034 & 0.946 $\pm$ 0.004 & 0.936 $\pm$ 0.008 & 0.947 $\pm$ 0.036 & 0.937 $\pm$ 0.007 & \textbf{0.984 $\pm$ 0.003} & 0.945 $\pm$ 0.006 & 0.946 $\pm$ 0.004 & 0.949 $\pm$ 0.005 & 0.949 $\pm$ 0.036 & 0.937 $\pm$ 0.007 & \textbf{0.990 $\pm$ 0.002} & 0.952 $\pm$ 0.005 \\
\texttt{boston} & 0.862 & 0.581 $\pm$ 0.005 & 0.777 $\pm$ 0.028 & 0.775 $\pm$ 0.026 & 0.710 $\pm$ 0.067 & \textbf{0.807 $\pm$ 0.026} & \textbf{0.790 $\pm$ 0.012} & \textbf{0.786 $\pm$ 0.015} & 0.777 $\pm$ 0.028 & \textbf{0.866 $\pm$ 0.014} & 0.759 $\pm$ 0.069 & 0.826 $\pm$ 0.026 & 0.842 $\pm$ 0.009 & 0.817 $\pm$ 0.013 \\
\texttt{california} & 0.829 & 0.803 $\pm$ 0.004 & 0.751 $\pm$ 0.004 & 0.583 $\pm$ 0.004 & 0.705 $\pm$ 0.003 & 0.758 $\pm$ 0.006 & 0.708 $\pm$ 0.004 & \textbf{0.792 $\pm$ 0.003} & 0.751 $\pm$ 0.004 & 0.801 $\pm$ 0.003 & 0.763 $\pm$ 0.006 & \textbf{0.823 $\pm$ 0.002} & 0.792 $\pm$ 0.002 & 0.792 $\pm$ 0.003 \\
\texttt{churn} & 0.861 & 0.711 $\pm$ 0.004 & 0.853 $\pm$ 0.002 & 0.847 $\pm$ 0.002 & 0.853 $\pm$ 0.006 & 0.862 $\pm$ 0.002 & 0.860 $\pm$ 0.002 & \textbf{0.864 $\pm$ 0.002} & 0.853 $\pm$ 0.002 & \textbf{0.865 $\pm$ 0.002} & 0.855 $\pm$ 0.006 & 0.862 $\pm$ 0.002 & \textbf{0.865 $\pm$ 0.002} & \textbf{0.866 $\pm$ 0.002} \\
\texttt{concrete} & 0.921 & 0.687 $\pm$ 0.005 & 0.845 $\pm$ 0.007 & 0.711 $\pm$ 0.008 & 0.814 $\pm$ 0.006 & 0.834 $\pm$ 0.007 & 0.790 $\pm$ 0.013 & \textbf{0.870 $\pm$ 0.004} & 0.845 $\pm$ 0.007 & 0.901 $\pm$ 0.004 & 0.868 $\pm$ 0.005 & 0.908 $\pm$ 0.004 & 0.879 $\pm$ 0.005 & \textbf{0.915 $\pm$ 0.003} \\
\texttt{diabetes} & 0.744 & 0.592 $\pm$ 0.004 & 0.744 $\pm$ 0.011 & 0.768 $\pm$ 0.009 & 0.728 $\pm$ 0.027 & 0.755 $\pm$ 0.013 & \textbf{0.791 $\pm$ 0.011} & 0.777 $\pm$ 0.008 & 0.744 $\pm$ 0.011 & 0.779 $\pm$ 0.009 & 0.730 $\pm$ 0.027 & 0.755 $\pm$ 0.013 & \textbf{0.797 $\pm$ 0.010} & 0.783 $\pm$ 0.009 \\
\texttt{friedman\_1} & 0.910 & 0.717 $\pm$ 0.005 & 0.790 $\pm$ 0.010 & 0.680 $\pm$ 0.015 & 0.867 $\pm$ 0.016 & 0.874 $\pm$ 0.017 & 0.893 $\pm$ 0.007 & \textbf{0.917 $\pm$ 0.003} & 0.790 $\pm$ 0.010 & 0.899 $\pm$ 0.005 & 0.927 $\pm$ 0.015 & 0.905 $\pm$ 0.004 & \textbf{0.972 $\pm$ 0.001} & 0.917 $\pm$ 0.003 \\
\texttt{spambase} & 0.945 & 0.507 $\pm$ 0.003 & 0.916 $\pm$ 0.002 & 0.904 $\pm$ 0.002 & 0.926 $\pm$ 0.002 & 0.922 $\pm$ 0.003 & 0.928 $\pm$ 0.002 & \textbf{0.944 $\pm$ 0.001} & 0.916 $\pm$ 0.002 & 0.946 $\pm$ 0.001 & 0.935 $\pm$ 0.001 & 0.934 $\pm$ 0.003 & 0.938 $\pm$ 0.001 & \textbf{0.951 $\pm$ 0.002} \\
\texttt{student} & 0.832 & 0.689 $\pm$ 0.006 & 0.752 $\pm$ 0.031 & 0.811 $\pm$ 0.014 & 0.699 $\pm$ 0.041 & 0.789 $\pm$ 0.033 & \textbf{0.837 $\pm$ 0.012} & 0.785 $\pm$ 0.014 & 0.752 $\pm$ 0.031 & \textbf{0.857 $\pm$ 0.009} & 0.701 $\pm$ 0.041 & 0.789 $\pm$ 0.033 & 0.838 $\pm$ 0.012 & 0.785 $\pm$ 0.014 \\
\texttt{telco} & 0.794 & 0.691 $\pm$ 0.007 & 0.793 $\pm$ 0.003 & 0.791 $\pm$ 0.003 & 0.793 $\pm$ 0.009 & 0.794 $\pm$ 0.004 & \textbf{0.806 $\pm$ 0.003} & \textbf{0.804 $\pm$ 0.003} & 0.793 $\pm$ 0.003 & 0.803 $\pm$ 0.003 & 0.798 $\pm$ 0.008 & 0.794 $\pm$ 0.004 & \textbf{0.809 $\pm$ 0.003} & 0.805 $\pm$ 0.003 \\
\texttt{wq} & 0.455 & 0.683 $\pm$ 0.005 & \textbf{0.346 $\pm$ 0.007} & 0.287 $\pm$ 0.005 & 0.331 $\pm$ 0.007 & 0.317 $\pm$ 0.007 & 0.333 $\pm$ 0.007 & 0.300 $\pm$ 0.006 & 0.346 $\pm$ 0.007 & \textbf{0.385 $\pm$ 0.007} & 0.356 $\pm$ 0.007 & \textbf{0.381 $\pm$ 0.006} & 0.358 $\pm$ 0.008 & 0.300 $\pm$ 0.006 \\
\midrule
\textbf{Average Rank} & & & 5.13 & 3.27 & 2.47 & 3.87 & \textbf{2.13} & 4.13 & 2.57 & 3.8 & 2.80 & 5.33 & \textbf{2.13} & 4.37 \\
\bottomrule
\end{tabular}
}
\label{tab:full}
\end{sidewaystable}
\clearpage
\subsection{Visualizing Compression Frontiers}\label{appx:extra_compression}
While Table \ref{tab:full} provides a granular analysis of performance at discrete memory constraints, we further evaluate SCATE across a continuous capacity spectrum ranging from 1KB to 100KB. Plotting traditional Pareto frontiers for all individual methods often results in severe visual clutter, as disparate hyperparameter configurations across techniques lead to irregular and non-uniform model sizes. Instead, to ensure a clear comparison, we isntead distill these results into two curves each dataset. The first curve tracks the peak performance of SCATE evaluated at 1KB intervals. The second curve is a composition of the state-of-the-art pruning methods: at each 1KB threshold, it adopts the maximum score achieved by \textit{any} competing baseline method (excluding the strictly weaker FP-SCATE variant). This baseline then represents a best-in-class curve against which we can evaluate SCATE.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\linewidth]{figures/rf_pareto.pdf}
\caption{Visualization curves for the RF compression benchmark. The solid line represents SCATE, and the dashed line represents the composite method.}
\label{fig:rf_pareto}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\linewidth]{figures/gbm_pareto.pdf}
\caption{Visualization curves for the GBM compression benchmark. The solid line represents SCATE, and the dashed line represents the composite method.}
\label{fig:gbm_pareto}
\end{figure}
The resulting performance curves are illustrated in Figure \ref{fig:rf_pareto} for RFs and Figure \ref{fig:gbm_pareto} for GBMs. An examination of these figures reveals SCATE’s robustness, especially around the 10KB point. In Fig. \ref{fig:rf_pareto}, SCATE frequently beats the composite curve or performs highly competitively, with the exception of the \texttt{auto\_mpg} dataset. In the GBM domain, the results are more tempered; SCATE slightly underperforms, though the differences are generally narrow. As discussed in Sect. \ref{sec:discussion}, this dynamic occurs because SCATE's compression footprint is bounded by the complexity of the kernel rather than the size of the original model. Consequently, we can attribute these results to competing methods inherently struggling more to compress the larger parameter counts of RFs.
\subsection{Runtime Experiments}\label{appx:runtime}
To assess the computational efficiency of the evaluated compression algorithms, we benchmark both training and inference times across three representative small-to-medium datasets: \texttt{student}, \texttt{wq}, and \texttt{telco}. We deliberately exclude the two largest datasets (\texttt{adult} and \texttt{california}) from this specific analysis; the prohibitive computational scaling of baseline rule extractors, specifically FIRE \citep{liu2023fire} and TreeExtract \citep{liu2025extracting}, resulted in frequent timeouts on these datasets, preventing a complete comparison.
Because these compression methods are primarily motivated by strict deployment constraints, runtime is most meaningfully analyzed as a function of the final model's memory footprint - as a result, we evaluate runtime across a continuous capacity spectrum. We discretize the target memory footprint into 5KB intervals, between 0KB to 100KB. For each interval, we compute the average training and inference times across all hyperparameter configurations that yield a compressed model falling within that specific 1KB memory bucket. This procedure is conducted independently for both RF and GBM base models, generating independent training and inference time profiles for each dataset (resulting in 12 comparative plots in total). The resulting plots are produced in
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\linewidth]{figures/rf_runtime_benchmark.pdf}
\caption{Training and inference time against model size plots for RFs, over three datasets}
\label{fig:rf_runtime}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\linewidth]{figures/gbm_runtime_benchmark.pdf}
\caption{Training and inference time against model size plots for GBMs, over three datasets}
\label{fig:gbm_runtime}
\end{figure}
Interestingly, our evaluations indicate that runtime remains largely invariant across different model complexities for the majority of the evaluated algorithms. Therefore, an additional comprehensive summary of these computational costs, reported as means and standard errors, is provided in Tab. \ref{tab:runtime_table}.
\begin{table}[htbp]
\centering
\caption{Average Training and Inference Runtimes over the entire evaluated hyperparameter set. All values are reported in seconds as Mean $\pm$ Standard Error. The fastest methods per column (including those within 1 standard error of the best) are bolded.} SSF results in GBM are blank, as SSF is an RF-exclusive method.
\label{tab:runtime_table}
\renewcommand{\arraystretch}{1.2}
\resizebox{\textwidth}{!}{
\begin{tabular}{l cc cc}
\toprule
& \multicolumn{2}{c}{\textbf{Random Forest}} & \multicolumn{2}{c}{\textbf{GBM}} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
\textbf{Method} & \textbf{Avg Training (s)} & \textbf{Avg Inference (s)} & \textbf{Avg Training (s)} & \textbf{Avg Inference (s)} \\
\midrule
\textbf{SCATE} & $3.0402 \pm 0.3668$ & $\mathbf{2.32 \times 10^{-4} \pm 3.04 \times 10^{-5}}$ & $2.8506 \pm 0.3340$ & $\mathbf{2.37 \times 10^{-4} \pm 2.96 \times 10^{-5}}$ \\
\textbf{FP-SCATE} & $33.7156 \pm 2.6515$ & $3.06 \times 10^{-4} \pm 4.26 \times 10^{-5}$ & $45.5539 \pm 5.5014$ & $\mathbf{2.41 \times 10^{-4} \pm 2.99 \times 10^{-5}}$ \\
\textbf{LOP} & $108.8504 \pm 18.2936$ & $2.69 \times 10^{-4} \pm 3.49 \times 10^{-5}$ & $36.3631 \pm 7.8968$ & $4.01 \times 10^{-4} \pm 5.67 \times 10^{-5}$ \\
\textbf{FP} & $\mathbf{0.4319 \pm 0.0347}$ & $2.46 \times 10^{-3} \pm 2.34 \times 10^{-4}$ & $\mathbf{0.1445 \pm 0.0422}$ & $1.97 \times 10^{-3} \pm 1.61 \times 10^{-4}$ \\
\textbf{TE} & $393.9479 \pm 146.0017$& $4.23 \times 10^{-3} \pm 1.66 \times 10^{-3}$ & $191.7248 \pm 19.0575$ & $1.18 \times 10^{-3} \pm 7.56 \times 10^{-5}$ \\
\textbf{SSF} & $6.3441 \pm 1.6434$ & $5.41 \times 10^{-3} \pm 1.52 \times 10^{-3}$ & -- & -- \\
\textbf{FIRE} & $2.7730 \pm 0.2128$ & $8.02 \times 10^{-2} \pm 3.61 \times 10^{-3}$ & $0.3109 \pm 0.0519$ & $1.65 \times 10^{-1} \pm 1.78 \times 10^{-2}$ \\
\bottomrule
\end{tabular}
}
\end{table}
These results highlight a distinct advantage of neural architectures regarding inference efficiency. Although SCATE and FP-SCATE do not achieve the absolute lowest training times—despite remaining highly competitive—they consistently deliver the fastest inference speeds. This is especially relevant for constrained deployment scenarios, ensuring that the models can operate effectively under limitations in both memory capacity and computational power. Furthermore, the relatively flat inference curves for SCATE (illustrated in Fig. \ref{fig:rf_runtime} and Fig. \ref{fig:gbm_runtime}) demonstrate that its computational overhead is highly robust to variations in model size, further solidifying its suitability for edge-computing environments.
\clearpage