TierBPF: Page Migration Admission Control for Tiered Memory via eBPF
Extracted source text · arXiv 2604.12300
%% The first command in your LaTeX source must be the \documentclass
%% command.
%%
%% For submission and review of your manuscript please change the
%% command to \documentclass[manuscript, screen, review]{acmart}.
%%
%% When submitting camera ready or to TAPS, please change the command
%% to \documentclass[sigconf]{acmart} or whichever template is required
%% for your publication.
%%
%%
\documentclass[sigplan,10pt]{template/acmart}
\usepackage{xspace}
\usepackage{pifont}% for \ding{51} (checkmark) and \ding{55} (cross)
\usepackage{booktabs}% for \toprule, \midrule, \bottomrule
\usepackage{multirow}% for \multirow in tables
\usepackage{url}
\usepackage{soul}
\usepackage{tcolorbox}% for observation boxes
\tcbuselibrary{skins}
\newcommand{\asaf}[1]{\textcolor{blue}{\{Asaf: #1\}}}
\newcommand{\tal}[1]{\textcolor{red}{\{Tal: #1\}}}
\usepackage{enumitem}
\setitemize{noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt}
\newenvironment{denseitemize}{
\begin{itemize}[topsep=2pt, partopsep=0pt, leftmargin=1.5em]
\setlength{\itemsep}{2pt}
\setlength{\parskip}{0pt}
\setlength{\parsep}{0pt}
}{\end{itemize}}
\definecolor{sherry}{RGB}{255,140,0}
\definecolor{dong}{RGB}{0,0,200}
\definecolor{checked}{RGB}{0,0,0}
% Define observation box style
\newtcolorbox{observationbox}[1][]{%
enhanced,
colback=gray!10,
colframe=gray!60,
boxrule=0.8pt,
arc=2pt,
left=6pt, right=6pt, top=4pt, bottom=4pt,
fonttitle=\bfseries,
#1
}
\newcounter{observation}
\newcommand{\observation}[1]{%
\refstepcounter{observation}%
\begin{observationbox}%
\textbf{Observation \theobservation:} #1%
\end{observationbox}%
}
% \begin{comment}
% Remove ACM Reference Format and Copyright information
\settopmatter{printacmref=false}
\renewcommand\footnotetextcopyrightpermission[1]{}
% \end{comment}
%%
%% \BibTeX command to typeset BibTeX logo in the docs
\AtBeginDocument{%
\providecommand\BibTeX{{%
Bib\TeX}}}
\begin{comment}
%% Rights management information. This information is sent to you
%% when you complete the rights form. These commands have SAMPLE
%% values in them; it is your responsibility as an author to replace
%% the commands and values with those provided to you when you
%% complete the rights form.
\setcopyright{acmlicensed}
\copyrightyear{2018}
\acmYear{2018}
\acmDOI{XXXXXXX.XXXXXXX}
%% These commands are for a PROCEEDINGS abstract or paper.
\acmConference[Conference acronym 'XX]{Make sure to enter the correct
conference title from your rights confirmation email}{June 03--05,
2018}{Woodstock, NY}
%%
%% Uncomment \acmBooktitle if the title of the proceedings is different
%% from ``Proceedings of ...''!
%%
%%\acmBooktitle{Woodstock '18: ACM Symposium on Neural Gaze Detection,
%% June 03--05, 2018, Woodstock, NY}
\acmISBN{978-1-4503-XXXX-X/2018/06}
\end{comment}
%%
%% Submission ID.
%% Use this when submitting an article to a sponsored event. You'll
%% receive a unique submission ID from the organizers
%% of the event, and this ID should be used as the parameter to this command.
% \acmSubmissionID{611}
%%
%% For managing citations, it is recommended to use bibliography
%% files in BibTeX format.
%%
%% You can then either use BibTeX with the ACM-Reference-Format style,
%% or BibLaTeX with the acmnumeric or acmauthoryear sytles, that include
%% support for advanced citation of software artefact from the
%% biblatex-software package, also separately available on CTAN.
%%
%% Look at the sample-*-biblatex.tex files for templates showcasing
%% the biblatex styles.
%%
%%
%% The majority of ACM publications use numbered citations and
%% references. The command \citestyle{authoryear} switches to the
%% "author year" style.
%%
%% If you are preparing content for an event
%% sponsored by ACM SIGGRAPH, you must use the "author year" style of
%% citations and references.
%% Uncommenting
%% the next command will enable that style.
%%\citestyle{acmauthoryear}
%%
%% end of the preamble, start of the body of the document source.
\begin{document}
%%
%% The "title" command has an optional parameter,
%% allowing the author to define a "short title" to be used in page headers.
%% \newcommand{\name}{FlexTier\xspace}
%% \title{\name: Enabling Flexible Page-Aware Memory Tiering With eBPF}
\newcommand{\name}{TierBPF\xspace}
\title{\name: Page Migration Admission Control for Tiered Memory via eBPF}
%\name: Using eBPF to Enable Flexible Memory Tiering System with Transparent Huge Pages
%\name: Deployment-Aware Memory Tiering
%%
%% The "author" command and its associated commands are used to define
%% the authors and their affiliations.
%% Of note is the shared affiliation of the first two authors, and the
%% "authornote" and "authornotemark" commands
%% used to denote shared contribution to the research.
\author{Xi Wang}
\affiliation{%
\institution{University of California, Merced}
\country{USA}
}
\author{Tal Zussman}
\affiliation{%
\institution{Columbia University}
\country{USA}
}
\author{Yuang Xu}
\affiliation{%
\institution{University of California, Merced}
\country{USA}
}
\author{Bin Ma}
\affiliation{%
\institution{University of California, Merced}
\country{USA}
}
\author{Asaf Cidon}
\affiliation{%
\institution{Columbia University}
\country{USA}
}
\author{Dong Li}
\affiliation{%
\institution{University of California, Merced}
\country{USA}
}
%%
%% By default, the full list of authors will be used in the page
%% headers. Often, this list is too long, and will overlap
%% other information printed in the page headers. This command allows
%% the author to define a more concise list
%% of authors' names for this purpose.
\renewcommand{\shortauthors}{Wang et al.}
%%
%% The abstract is a short summary of the work to be presented in the
%% article.
\begin{abstract}
Existing software-based memory tiering systems decide which pages to place on the slower or faster tier. However, they do not take into account two important factors that greatly influence application performance: the size of the migrated pages, and the underlying hardware device and tiering topology. We introduce \name, a software mechanism that can be plugged into existing memory tiering systems to take
these factors into account,
%\tal{"to take these factors into account"? otherwise it just repeats the previous sentence} into account page size and the tiering topology,
by making simple binary page admission decisions. %in the memory tiering system to refine page migration.
%We build such a layer based on a flexible eBPF mechanism and demonstrate its generality across three memory tiering systems (AutoNUMA~\cite{autonuma}, TPP~\cite{maruf2023tpp}, and Colloid~\cite{vuppalapati2024colloid}).
\name is implemented as a set of eBPF hooks, which allow users to define their own custom policies.
In order to make its decisions, \name utilizes a lightweight tracking mechanism for page profiling which is not dependent on the application's working set size. \name, integrated into three memory tiering systems and evaluated with 17 workloads, achieves geomean throughput gains of up to 17.7\% with improvements of up to 75\% for individual workloads.
%\name, integrated into three memory tiering systems, improves all 17 workloads on AutoNUMA and 16 of 17 on TPP, achieving geomean throughput gains of up to 17.7\% and individual workload improvements of up to 75\%.
%\textcolor{sherry}{We demonstrate that \name, integrated into three existing memory tiering systems, improves geometric-mean throughput on a CXL platform by 15.8\% for TPP, 8.9\% for AutoNUMA, and 5.1\% for Colloid, across 19 memory-intensive workloads.
%Memory tiering architectures are rapidly diversifying. \asaf{There is a wide diversity in memory tiering deployments, including device type, tier topology, page sizes and the ratio between slow and fast tier.} Yet existing memory tiering \asaf{migration} systems hard‑code static, architecture agnostic migration policies, implicitly assuming uniform bandwidth behavior and fixed page granularity. This rigidity \asaf{leads different policies to perform better than others, depending on the specific deployment configuration.} leads to growing inefficiencies, particularly in the interaction between Transparent Huge Pages (THP) and page migration. \asaf{Maybe don't mention huge pages yet} migrating a 2 MB THP wastes bandwidth and fast‑tier capacity, while splitting THPs into base pages sacrifices TLB coverage. Although Linux recently introduced multi‑size THP (mTHP), current systems do not exploit it during migration. We present \name, an eBPF‑based memory tiering system that brings required flexibility \asaf{that enables operators to customize the tiered memory migration decisions to the specific deployment configuration} to tiered memory management. \name decouples migration policy from kernel mechanisms using programmable eBPF hooks, enabling runtime adaptation without kernel recompilation. \asaf{As a case study, we focus on making tiered memory migration decisions that are aware of page sizes... Maybe also briefly explain why huge pages are important} It dynamically selects mTHP migration granularity based on contention, applies architecture‑aware migration admission tailored to platform bandwidth characteristics, and performs lightweight in‑kernel profiling using compact subpage histograms. Evaluated on both CXL‑ and PMEM‑based memory architectures \asaf{Maybe highlight here that to show how the system can be customized to different devices, we design custom policies for both CXL and non-volatile memory} and \textcolor{red}{xxxx memory tiering systems}, \name reduces migration traffic by up to \textcolor{red}{[XX\%]} and improves application performance by up to \textcolor{red}{[X.Y×]}, demonstrating that flexible, programmable migration policies are essential for efficient memory tiering across modern architectures.
\end{abstract}
\begin{comment}
%% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
%% Please copy and paste the code instead of the example below.
%%
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>00000000.0000000.0000000</concept_id>
<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>00000000.00000000.00000000</concept_id>
<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
<concept_significance>300</concept_significance>
</concept>
<concept>
<concept_id>00000000.00000000.00000000</concept_id>
<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
<concept_significance>100</concept_significance>
</concept>
<concept>
<concept_id>00000000.00000000.00000000</concept_id>
<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
<concept_significance>100</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[500]{Do Not Use This Code~Generate the Correct Terms for Your Paper}
\ccsdesc[300]{Do Not Use This Code~Generate the Correct Terms for Your Paper}
\ccsdesc{Do Not Use This Code~Generate the Correct Terms for Your Paper}
\ccsdesc[100]{Do Not Use This Code~Generate the Correct Terms for Your Paper}
%%
%% Keywords. The author(s) should pick words that accurately describe
%% the work being presented. Separate the keywords with commas.
\keywords{Do, Not, Use, This, Code, Put, the, Correct, Terms, for,
Your, Paper}
%% A "teaser" image appears between the author and affiliation
%% information and the body of the document, and typically spans the
%% page.
\begin{teaserfigure}
\includegraphics[width=\textwidth]{sampleteaser}
\caption{Seattle Mariners at Spring Training, 2010.}
\Description{Enjoying the baseball game from the third-base
seats. Ichiro Suzuki preparing to bat.}
\label{fig:teaser}
\end{teaserfigure}
\received{20 February 2007}
\received[revised]{12 March 2009}
\received[accepted]{5 June 2009}
\end{comment}
%%
%% This command processes the author and affiliation and title
%% information and builds the first part of the formatted document.
\maketitle
\pagestyle{plain}
%-------------------------------------------------------------------------------
\input{text/intro}
\input{text/background}
\input{text/motivation}
% \input{text/motivation.bak}
\input{text/overview}
\input{text/design}
\input{text/implementation}
\input{text/evaluation}
\input{text/related_work}
\input{text/conclusion}
\balance
\bibliographystyle{template/ACM-Reference-Format}
\bibliography{bib/sherry,bib/li}
%-------------------------------------------------------------------------------
\end{document}
\section{Introduction}
\label{sec:intro}
%Modern datacenter applications demand ever-larger memory capacities. To meet this demand without proportional cost increases, operators deploy \emph{tiered memory} systems that combine fast but expensive local DRAM with slower but cheaper memory technologies. The operating system manages these tiers transparently through \emph{page migration}: promoting hot pages to fast-tier memory and demoting cold pages to slow-tier memory~\cite{maruf2023tpp, vuppalapati2024colloid, xiang2024nomad}.
As the number of cores available in modern processors grows faster than local DRAM server capacity~\cite{maruf2023tpp,10.5555/3691938.3691941}, %\asaf{add citation, I think we have some data in our Memstrata paper}
datacenter operators are deploying tiered-memory architectures that pair fast, capacity-limited local DRAM with a slower, higher-capacity memory tier. \emph{Memory tiering systems} are responsible for migrating pages across these tiers.
Memory tiering systems are typically designed around a high-level abstraction: %of heterogeneous memory architecture: \textcolor{sherry}{memory is either \emph{fast} or \emph{slow}}.
a heterogeneous memory architecture composed of slow memory and fast memory. This abstraction simplifies the page migration decision based on page access frequency and recency~\cite{maruf2023tpp, vuppalapati2024colloid, xiang2024nomad,10.1145/3676642.3736119,eurosys24:mtm,atc24_hm}.
%to a question of page hotness, that is,
That is, migrating hot (frequently or recently accessed) pages to the fast tier, and keeping cold (infrequently used) pages in the slow tier. Albeit simple to reason about, this high-level abstraction causes two problems.
% Leveraging this abstraction~\cite{maruf2023tpp, vuppalapati2024colloid, xiang2024nomad,10.1145/3676642.3736119,eurosys24:mtm,atc24_hm}, existing memory tiering systems focus on how to timely and efficiently promote frequently (or recently) accessed pages to fast-tier memory and demote infrequently used pages to slower tiers.
%Today's slow-tier memory is no longer a single technology. CXL-attached DRAM uses a PCIe-based interconnect， Intel Optane PMEM connects through the DDR bus, and emerging technologies such as CXL-attached flash~\cite{} and disaggregated memory pools~\cite{} continue to expand the design space. These technologies differ not only in latency and bandwidth, but also in fundamental properties such as duplex mode, persistence, and cache coherence. As a result, the same migration policy can yield very different performance outcomes depending on the underlying hardware. A promotion strategy that improves throughput on a CXL platform may be counterproductive on Optane, or vice versa (\S\ref{subsec:mot-arch}). Yet the tiering systems deployed today are tightly coupled to specific assumptions about the slow tier. They hard-code a single migration policy into the kernel, leaving operators no way to adapt when the hardware changes. As memory architectures continue to diversify, this lack of flexibility becomes a growing problem.
%Slow-tier memory is no longer defined by a single technology. CXL-attached memory expansion connects over a PCIe link, Intel Optane PMEM is integrated via the DDR bus, and emerging substrates—such as CXL-attached flash~\cite{} and disaggregated memory pools~\cite{}—further broaden the design space. These platforms differ not only in latency and bandwidth, but also in architectural properties such as duplex mode, persistence, and cache coherence. As a result, migration policies interact differently with each memory substrate: a policy that improves performance on a CXL-based memory expansion may have little benefit—or even incur overhead—on persistent memory-based platforms (\S\ref{subsec:mot-arch}). However, existing tiered memory systems are typically designed around a fixed set of assumptions about the slow tier, embedding a single migration policy into the kernel. As memory technologies continue to diversify, this tight coupling limits the effectiveness of page migration across platforms.
\textbf{Problem 1: The abstraction ignores the interaction between page size and migration.} Many production systems enable Transparent Huge Pages (THP), allocating memory in 2\,MB pages to reduce the TLB miss overhead. However, page migration in memory tiering systems is intertwined with the page size. When the kernel migrates a 2\,MB THP, it must copy the entire page, even if only a small subset of its constituent base pages are frequently accessed. This behavior wastes migration bandwidth and may squander
%consume \tal{squander may be a better word, has a negative wasteful connotation}
scarce fast-tier capacity with cold data. Splitting the THP into 512 base pages and migrating only the hot ones avoids this waste, but at the price of increased TLB overhead.
The problem of granular hot memory regions in THP %\textcolor{checked}{leading to memory bloat/waste}
is not specific to migration. Indeed, Linux v6.8 attempted to address this by introducing multi-size THP (mTHP)~\cite{linux_mthp}, which supports intermediate page sizes from 16\,KB to 1\,MB, but the page size is determined only at \emph{allocation} time. As such, no existing tiering system exploits splitting pages into intermediate %\emph{mTHP} \tal{can remove mTHP and emphasize "intermediate sizes" instead}
sizes during page migration. In existing tiered memory systems, a page originally allocated as a 1\,MB page can only be migrated in one big chunk.
%\asaf{Apart from mTHP, are there existing systems that even look at migrating/splitting 4KB vs. 2MB page sizes during migration? If not, we may want to say so}
%Consequently, a promising middle ground remains unexplored: dynamically splitting a 2\,MB THP into an intermediate mTHP size (e.g., 16\,KB, 64\,KB or 256\,KB) that captures hot regions while retaining much of the TLB benefit of huge pages.
% Second, the tiered memory abstraction can miss memory architecture details, leading to suboptimal performance on tiered memory.
\textbf{Problem 2: The fast-slow tier abstraction hides deployment specific properties.}
Tiering deployments are heterogeneous, both in terms of the types of devices and in terms of the deployment topology. CXL-attached memory expansion connects over a PCIe link, Intel Optane PMEM is integrated via the DDR bus, while emerging substrates, such as CXL-attached flash~\cite{288786,10.1145/3669940.3707250} and disaggregated memory pools~\cite{10.1145/3575693.3578835,10.1145/3722212.3724460,10.1145/3712285.3759816}, further broaden the design space. These deployment configurations differ not only in latency and bandwidth, but also in architectural properties such as memory duplex mode (i.e., read-write interference), persistence, and cache coherence. As a result, there is no one-size-fits-all policy that works for any deployment configuration. For example, a policy that improves performance on a CXL-based memory expansion may have little benefit---or even incur overhead---on persistent memory-based platforms (\S\ref{subsec:mot-arch}). %However, the existing memory tiering systems, typically designed around the high-level memory abstract, can miss such architecture differences.
%existing tiered memory systems are typically designed around a fixed set of assumptions about the slow tier, embedding a single migration policy into the kernel.
%As memory technologies continue to diversify, building the memory tiering system upon the simplified slow vs. fast tiered memory abstraction limits the effectiveness of page migration across platforms.
%Beyond the choice of migration policy, the \emph{granularity} of migration also matters. Most production systems enable Transparent Huge Pages (THP), allocating memory in 2\,MB pages to reduce TLB miss overhead. However, page migration and THP interact poorly. When the kernel migrates a 2\,MB THP, it must copy the entire page, even if only a small fraction of its subpages are frequently accessed. This wastes migration bandwidth and occupies scarce fast-tier capacity with cold data. The alternative, splitting the THP into 512 base pages and migrating only the hot ones, destroys the TLB coverage that huge pages provide. Linux 6.8 introduced multi-size THP (mTHP), which supports intermediate page sizes from 16\,KB to 1\,MB, but only for \emph{allocation}. No existing system uses mTHP sizes when \emph{migrating} pages between memory tiers. This leaves a middle ground entirely unexplored: splitting a 2\,MB THP into an intermediate mTHP size (e.g., 64\,KB or 256\,KB) that captures the hot region without sacrificing all TLB benefits.
%%Beyond the choice of migration policy, the granularity of migration is equally critical. Most production systems enable Transparent Huge Pages (THP), allocating memory in 2\,MB pages to reduce TLB miss overhead. However, page migration interacts poorly with THP. When the kernel migrates a 2\,MB THP, it must copy the entire page, even if only a small subset of its constituent base pages are frequently accessed. This behavior wastes migration bandwidth and fills scarce fast-tier capacity with cold data. Splitting the THP into 512 base pages and migrating only the hot ones avoids this waste, but at the cost of destroying the TLB coverage that huge pages provide. Linux 6.8 partially addresses this tension by introducing multi-size THP (mTHP), which supports intermediate page sizes from 16\,KB to 1\,MB, but only at \emph{allocation} time. No existing system exploits mTHP sizes during \emph{migration} across memory tiers. Consequently, a promising middle ground remains unexplored: dynamically splitting a 2\,MB THP into an intermediate mTHP size (e.g., 64\,KB or 256\,KB) that captures hot regions while retaining much of the TLB benefit of huge pages.
These two problems share a common root cause: there is no existing mechanism for tiered memory systems to take page size or the tiered memory's deployment configuration into account. %In this paper, we present \name, a system makes tiered memory migration that introduces \textit{page-migration admission control} to fill this missing piece.
% To address the above problems, we design a new software mechanism, \textit{page-migration admission control}, which we integrate into the memory tiering system.
% This layer takes effect when the page migration decision has been made by the memory tiering system based upon the high-level memory abstraction. \asaf{Last sentence is vague} This layer works as a filter to refine the page migration decision by considering the impact of mTHP size and architecture features. \asaf{This last sentence is also a bit vague}
To this end, we introduce \name, a system that interfaces with existing tiered memory systems, making them adaptable to factors like page size granularity and tiering architecture. \name sits between the memory tiering system's migration decision-making and the page migration mechanism. Our key insight is \name makes
%\tal{Change to "\name makes", get rid of "Our key insight"?}
existing tiered memory systems aware of their broader context by applying \emph{page migration admission control}. When the tiering system decides that a page should be migrated, \name applies two \emph{filters} before the migration proceeds: a \emph{granularity filter} that determines the right page size for migration, and an \emph{architecture filter} that determines whether the migration actually benefits the deployed hardware. These two filters transform a coarse migrate-or-not decision into a fine-grained one that accounts for subpage access patterns, runtime conditions (i.e., memory bandwidth contention), and hardware characteristics (i.e., memory duplex mode).
\begin{comment}
Exploiting this middle ground is not straightforward.
Two challenges stand in the way.
First, the right mTHP size is not static.
Under low bandwidth contention, keeping pages at 2\,MB is optimal because TLB coverage dominates.
Under high bandwidth contention, which is common in multi-tenant cloud environments, smaller mTHP sizes are preferable because they reduce bandwidth waste and fast-tier capacity consumption per migration (\S\ref{subsec:mot-contention}).
A system must therefore adapt its page size selection dynamically based on runtime conditions.
\end{comment}
% In this paper, we develop this \textcolor{sherry}{Page-migration admission control} and explore how it can determine mTHP size as well as different deployment configurations, such as duplex mode, to improve the effectiveness of page migration.
% To determine mTHP size, we must consider runtime memory contention and be light-weight.... \textcolor{red}{(pending...)}
%\textcolor{sherry}{\textbf{Granularity filter: contention-aware mTHP splitting.}}
The granularity filter addresses Problem~1. Instead of migrating the full 2\,MB THP or splitting it all the way to 4\,KB, this filter selects an intermediate mTHP size that is large enough to preserve TLB benefits yet small enough to avoid migrating cold data. The choice is \emph{contention-aware}: under low slow-tier bandwidth contention, the filter keeps pages at 2\,MB for maximum TLB coverage; under high contention---common in multi-tenant cloud environments---it selects a smaller mTHP size (e.g., 64\,KB or 128\,KB), so each migration moves less data, wastes less bandwidth, and occupies less fast-tier capacity (\S\ref{subsec:mot-contention}).
%\textcolor{sherry}{\textbf{Architecture filter: architecture-aware migration admission.}}
The architecture filter addresses Problem~2. The specific device hardware affects the effectiveness of page migration. For example, CXL memory uses full-duplex links where reads and writes proceed on independent channels~\cite{yang2025cxlaimpodcxlmemoryneed}, while Intel Optane PMEM uses half-duplex DDR where reads and writes share the same bus. Hence, on a CXL system with read-dominant memory traffic, selectively holding back write-heavy pages from page promotion can improve bandwidth utilization by keeping the full-duplex channels balanced. The same policy on PMEM would merely prevent hot pages from reaching fast memory, with no bandwidth benefit. The page migration admission control allows memory tiering to customize migration decisions based on the duplex mode.
%Exploiting this middle ground, however, is fundamentally challenging. Two obstacles stand in the way. First, the optimal mTHP size is inherently dynamic. When migration bandwidth is unconstrained, preserving 2\,MB pages is preferable, as TLB coverage dominates performance. In contrast, under high migration bandwidth contention—common in multi-tenant cloud environments—smaller mTHP sizes are more effective, as they sharply reduce both wasted migration traffic and fast-tier capacity consumption per promotion (\S\ref{subsec:mot-contention}). A practical system must therefore adapt page-size decisions to evolving runtime conditions rather than rely on a static granularity choice.
\begin{comment}
Second, different memory architectures demand different migration strategies.
CXL memory uses full-duplex links where reads and writes proceed on independent channels, while Optane PMEM uses half-duplex DDR where reads and writes share the same bus.
This architectural difference means that a migration policy beneficial on one platform can hurt performance on another (\S\ref{subsec:mot-arch}).
For example, on a CXL system where the memory traffic is read-dominant, selectively holding back write-heavy pages from promotion can improve bandwidth utilization by keeping the full-duplex channels balanced.
The same policy on Optane would merely prevent hot pages from reaching fast memory, with no bandwidth benefit. Yet existing tiering systems (e.g., AutoNUMA~\cite{autonuma}, TPP~\cite{maruf2023tpp}, MEMTIS~\cite{lee2023memtis}, Colloid~\cite{vuppalapati2024colloid}, Nomad~\cite{xiang2024nomad} and HybridTier~\cite{10.1145/3676642.3736119}) all apply a single, architecture-agnostic migration policy.
\end{comment}
%Second, migration policies must be architecture-aware. Emerging memory platforms differ in how migration traffic interacts with hardware resources. CXL memory employs full-duplex links, allowing reads and writes to proceed on independent channels, whereas Intel PMEM relies on half-duplex DDR interfaces where reads and writes contend for the same bus. As a result, a migration strategy that improves performance on one architecture can degrade it on another (\S\ref{subsec:mot-arch}). For example, on a read-dominant CXL system, deliberately suppressing the promotion of write-heavy pages balances bidirectional bandwidth utilization and improves effective throughput. Applying the same policy on PMEM yields no such benefit; it merely delays the placement of hot pages into fast memory. Despite these fundamental differences, existing tiering systems—including AutoNUMA~\cite{autonuma}, TPP~\cite{maruf2023tpp}, MEMTIS~\cite{lee2023memtis}, Colloid~\cite{vuppalapati2024colloid}, Nomad~\cite{xiang2024nomad}, and HybridTier~\cite{10.1145/3676642.3736119}—all hard-code a single, architecture-agnostic migration policy.
% TODO: Consider adding a figure here that provides a high-level illustration
% of the FlexTier approach (e.g., a before/after comparison of 2MB migration
% vs. mTHP-aware migration), if space permits.
%These challenges call for flexibility in both migration granularity and migration strategy. Such flexibility that cannot be achieved by hard-coding a single policy into the kernel. We observe that eBPF, which allows safe, verified programs to run inside the kernel without recompilation, is well-suited for this task. By placing programmable hooks on the kernel's migration path, we can decouple \emph{what} to migrate and \emph{at what granularity} from the kernel's migration mechanism itself, adapting policy to the workload, the runtime environment, and the hardware.
%These challenges demand flexibility in both migration granularity and migration strategy—flexibility that cannot be achieved by hard-coding a single policy into the kernel. We observe that eBPF, which enables safe, verified programs to execute inside the kernel without recompilation, provides a natural substrate for such adaptability. By inserting programmable hooks along the kernel’s migration path, we cleanly decouple \emph{what} to migrate and \emph{at what granularity} from the migration mechanism itself. This separation allows migration policy to evolve dynamically with the workload, runtime conditions, and underlying hardware, without sacrificing safety or deployability.
\begin{comment}
In this paper, we present \name, an eBPF-based memory tiering system that makes THP migration both granularity-aware and architecture-aware.
\name introduces three key techniques.
(1)~\emph{Contention-aware mTHP splitting}: when slow-tier bandwidth is under pressure, \name splits 2\,MB THPs into the largest mTHP size that still captures the hot subpage region, preserving TLB coverage while avoiding bandwidth waste.
When bandwidth contention is low, pages remain at 2\,MB.
(2)~\emph{Architecture-aware migration admission}: on CXL platforms, \name evaluates each page's read/write ratio against real-time CXL traffic before allowing page migration to determine whether a page should be migrated or not. This hook is a no-op on half-duplex architectures.
(3)~\emph{Lightweight eBPF-based profiling}: \name processes hardware memory access samples (AMD IBS / Intel PEBS) entirely in-kernel via eBPF programs, avoiding the context switches and data copies of user-space profiling.
A compact global subpage histogram (4\,KB per process) replaces per-page tracking structures that can grow to hundreds of megabytes, enabling low-overhead profiling regardless of working set size.
\end{comment}
% To make the page migration admission control applicable to multiple memory tiering systems, we xxxxx.
The critical obstacle in supporting these filters is accurate and low-overhead memory profiling. Building and maintaining per-page profiling
%or per-page-size profiling
(as in MEMTIS~\cite{lee2023memtis}) is not scalable in terms of runtime and memory overhead. %\textcolor{sherry}{Both filters require accurate, low-overhead knowledge of per-page access behavior.}
Instead, \name introduces lightweight eBPF-based profiling. In particular, \name processes hardware memory-access samples entirely in-kernel using eBPF, avoiding the context switches and data copying suffered by the user-space profiling. In addition, \name uses a lightweight, compact global subpage histogram (using only 4\,KB per process) instead of per-page tracking structures, %that can otherwise grow to hundreds of megabytes,
enabling lightweight profiling with overhead independent of the workload's working set size.
To make page-migration admission control deployable across different tiering systems and adaptable to evolving memory architectures, we implement all policy logic as user-customizable eBPF programs. %eBPF allows safe, verified programs to execute inside the kernel without recompilation, making it a natural substrate for policy logic that must run on the migration hot path yet remain easy to update.
We expose three hooks along the kernel's NUMA migration path: one for split-size selection, one for subpage hot/cold classification, and one for migration admission. %When no eBPF program is loaded, the kernel falls back to its default behavior (no splitting, admit all migrations).
This design cleanly separates policy (i.e., which page to migrate and at what granularity) from mechanism (i.e., how to migrate), allowing the admission control to easily plug into existing memory tiering systems.
\begin{comment}
\textcolor{red}{(refine the following paragraph)}
In this paper, we present \name, an eBPF-based memory tiering system that makes THP migration both granularity-aware and architecture-aware. \name introduces three core techniques.
(1) Contention-aware mTHP splitting: when slow-tier bandwidth becomes constrained, \name selectively splits a 2\,MB THP into the largest mTHP size that still captures the hot subpage region, preserving most TLB coverage while eliminating wasted migration traffic. %When contention is low, pages remain intact at 2\,MB.
(2) Architecture-aware migration admission: on CXL platforms, \name evaluates each page’s read/write ratio against real-time CXL traffic and admits migration only when it improves bidirectional bandwidth utilization; on half-duplex architectures, this check becomes a no-op.
(3) Lightweight eBPF-based profiling: \name processes hardware memory access samples entirely in-kernel using eBPF, avoiding the context switches and data copies of user-space profiling. A lightweight, compact global subpage histogram (4\,KB per process) replaces per-page tracking structures that can otherwise grow to hundreds of megabytes, enabling consistently low-overhead profiling independent of working set size.
\end{comment}
%We evaluate \name on two diverse memory platforms (one with CXL memory expansion and the other with Intel PMEM) across representative memory-intensive workloads.
Our paper's major contributions are:
%% TODO: Fill in final numbers after evaluation is complete.
%% Example: \name improves application performance by up to X\% compared to
%% the best prior system, while reducing migration bandwidth by Y\%.
%\tal{switched to denseitemize environment to make the list take less space}
\begin{denseitemize}
% \item We identify a fundamental lack of flexibility in existing memory tiering systems in the face of growing architectural diversity.
\item We identify two problems in existing memory tiering systems caused by the fast-vs-slow memory abstraction: the lack of both migration-time page-granularity awareness and architecture-aware page migration filtering.
%We show that
%the interaction between THP and page migration, as well as
%the page migration admission control has become a first-order bottleneck that cannot be addressed by traditional static, kernel-hardcoded policies.
%With the emerging diversity across memory tiering architectures, we identify the necessity of bringing flexibility into memory tiering software system, especially in the interaction between THP and page migration, and page migration admission control.
%We identify the interaction between THP and page migration as a key inefficiency in tiered memory systems, and show that intermediate mTHP sizes can resolve the tension between migration precision and TLB coverage (\S\ref{sec:motivation}).
\item We present \name, the first system to support dynamic, contention-aware mTHP-size selection and memory duplex mode-aware page migration. \name implements all migration policies as eBPF programs, allowing operators to adapt migration policies to hardware topologies.
%We present \name, the first system to perform mTHP-granularity page migration with dynamic, contention-aware size selection and architecture-aware migration admission. All policies are implemented as eBPF programs, requiring minimal kernel changes and no kernel recompilation for policy updates (\S\ref{sec:key-ideas}--\S\ref{sec:design}).
%\item We introduce a lightweight profiling approach that uses eBPF to process hardware samples in-kernel with a compact global subpage histogram, reducing profiling memory overhead from hundreds of megabytes to 4\,KB per process (\S\ref{subsec:profiling}).
\item We evaluate \name on two memory platforms: (a) CXL memory expansion, and (b) Intel PMEM. We integrate \name with three memory tiering systems: AutoNUMA~\cite{autonuma}, TPP~\cite{maruf2023tpp}, and Colloid~\cite{vuppalapati2024colloid}. Our results demonstrate that \name significantly improves both adaptability and performance across architectures, consistently outperforming prior approaches under diverse workloads: \name achieves geomean throughput gains of 7.4-17.7\%, with improvements of up to 75\% for individual workloads, and outperforms MEMTIS by up to 26\%.
%\textcolor{sherry}{on the CXL platform, \name improves all 17 workloads on AutoNUMA and 16 of 17 on TPP, achieving 7.4--17.7\% geometric-mean throughput gains across all three systems and up to 75\% on individual workloads; on the Intel PMEM platform, \name outperforms MEMTIS by up to 26\%.} %\tal{This paragraph should be right before the major contributions and then just one or two sentences here describing the eval.}
\end{denseitemize}
%\name improves geometric-mean throughput by 15.8\% for TPP, 8.9\% for AutoNUMA, and 5.1\% for Colloid
\section{Background}
\label{sec:background}
%this section largely overlaps with the intro; delete it to save space.
\begin{comment}
\subsection{Memory Tiering}
\label{subsec:bg-tiering}
Modern datacenter servers increasingly utilize heterogeneous memory architectures that combine multiple memory technologies with different performance and cost characteristics. This heterogeneous memory architecture, called tiered memory, provides a cost-effective solution to increase memory capacity, utilization, and even bandwidth.
%A typical tiered-memory system consists of fast tier memory (e.g., local DRAM) offering higher bandwidth and shorter latency but limited memory capacity, and slow tier memory (e.g., CXL memory expansion) offering larger capacity at lower cost but with lower bandwidth and higher latency.
To manage tiered memory, system software (called memory tiering system in this paper) can enable a unified address space across memory tiers and model each memory tier as a NUMA node~\cite{}. %The goal of a memory tiering system is often straightforward: placing frequently (or recently) accessed data (\emph{hot data}) in the fast tier and infrequently accessed data (\emph{cold data}) in the slow tier, thereby maximizing application performance while keeping hardware costs manageable. The OS achieves this through \emph{page migration}: \emph{promotion} moves hot pages from slow tier to fast tier, while \emph{demotion} moves cold pages in the opposite direction to free up fast-tier capacity.
Several systems have been proposed to manage page migration in tiered memory~\cite{}. For example, AutoNUMA~\cite{autonuma}, the Linux kernel's built-in NUMA balancing mechanism, uses NUMA hinting faults to detect page access patterns and migrate pages accordingly. TPP~\cite{maruf2023tpp} introduces application-transparent page placement for CXL-enabled tiered memory. HeMem~\cite{raybuck2021hemem} manages tiered memory at the user level for NVM-based systems.
Colloid~\cite{vuppalapati2024colloid} takes a different angle, showing that balancing loaded access latency across tiers outperforms naive hotness-based placement. More recently, Nomad~\cite{xiang2024nomad} addresses migration-induced thrashing through non-exclusive page shadowing.
While all these systems make page migration decisions on a page-by-page basis, none of them consider the \emph{granularity} at which pages should be migrated as a major design concern. As we show in \S\ref{sec:motivation}, the page migration granularity has a significant impact on both migration efficiency and application performance.
\end{comment}
%\subsection{Multi-Size Transparent Huge Pages}
%\label{subsec:bg-thp}
\begin{comment}
Virtual-to-physical address translation relies on the Translation Lookaside Buffer (TLB). %a fixed-size hardware cache.
%When applications use large memory footprints with 4\,KB base pages,
Using 4\,KB as the page size, TLB misses can become a major source of overhead~\cite{}. Transparent Huge Pages (THP) mitigates this by \textcolor{red}{mapping memory in 2\,MB pages}, reducing address translation overhead and increasing TLB hits.
\end{comment}
\textbf{Multi-size transparent huge pages.} Virtual-to-physical address translation depends heavily on the TLB. %With 4\,KB pages, TLB misses can introduce significant overhead.
\textcolor{checked}{Using 4\,KB pages significantly limits TLB reach (e.g., a few hundred kilobytes per TLB), increasing TLB miss rates for modern memory-intensive workloads. Since each TLB miss can invoke a multi-level page-table walk that may cost tens to hundreds of cycles depending on cache residency, address-translation overheads alone have been shown to reduce application performance by up to 30\%~\cite{weisberg2008obsolete,mccalpin2012tlb,lustig2013tlb}. }
%\asaf{any way to quantify this more precisely? This is a bit too general}
THPs alleviate this by using 2\,MB pages, reducing translation frequency and improving TLB hit rates. %\asaf{Any way to add some numbers to this paragraph}
%However, 2\,MB THP introduces its own problems.
%Allocating physically contiguous 2\,MB regions is expensive under memory fragmentation. Therefore, Linux 6.8 introduced \emph{multi-size THP} (mTHP)~\cite{linux_mthp}, which supports anonymous page allocation at intermediate sizes: 16\,KB, 32\,KB, 64\,KB, 128\,KB, 256\,KB, 512\,KB, and 1\,MB, in addition to the traditional 4\,KB and 2\,MB. Each size can be individually enabled or disabled via sysfs on a system-wide or per-VMA basis. mTHP offers a middle ground: smaller THP sizes are easier to allocate, waste less memory, and still reduce TLB misses compared to 4\,KB pages.
However, 2\,MB THPs introduce their own challenges. Allocating physically contiguous 2\,MB regions becomes costly when memory is fragmented. To address this, Linux 6.8 introduced mTHP, which supports anonymous page allocation at intermediate sizes (16\,KB, 32\,KB, 64\,KB, 128\,KB, 256\,KB, 512\,KB, and 1\,MB) alongside the traditional 4\,KB and 2\,MB options. Each size can be enabled or disabled independently via sysfs. %\textcolor{sherry}{system-wide.}
mTHP provides a practical middle ground: smaller huge-page sizes are easier to allocate, reduce memory waste, and still improve TLB efficiency compared to 4\,KB pages. \textcolor{checked}{Internally, the Linux kernel represents each (m)THP as a \emph{folio}; we use folio and page interchangeably in this paper.}
\begin{comment}
Yet mTHP support today is limited to \emph{allocation}.
The kernel does not consider mTHP sizes when \emph{migrating} pages between memory tiers.
When a 2\,MB THP is targeted for migration, the kernel faces a binary choice: migrate the full 2\,MB page, or split it entirely into 4\,KB base pages.
Migrating an entire 2\,MB page is costly: it consumes migration bandwidth and occupies scarce fast-tier capacity---even when only a small fraction of the page is actually hot. Splitting a huge page into 512 4\,KB base pages diminishes the benefit of increased TLB hits.
No existing system explores the middle ground of splitting a THP into intermediate mTHP sizes before migration---a gap that \name addresses.
\end{comment}
\textcolor{checked}{However, current mTHP support is limited to \emph{allocation}. The kernel does not consider %intermediate mTHP
page sizes when \emph{migrating} pages across memory tiers.
%\asaf{I asked this question earlier -- do existing memory tiering consider splitting THP into 4KB pages when doing migration? Whether the answer is yes or no, worth saying so explicitly, if the answer is yes we should cite the work that does this early in the paper, if no, they we should we say we are only the system that takes into page size, not just takes into account intermediate page sizes..}
MEMTIS~\cite{lee2023memtis} improves on this by making a binary choice:
%When a 2\,MB THP is selected for migration, the kernel only takes a binary choice:
migrating the entire 2\,MB page or splitting it into 512 individual 4\,KB base pages for selective migration, with no consideration of other mTHP sizes. However, %migrating the full page is costly—it consumes migration bandwidth and occupies scarce fast-tier capacity, even when only a small portion of the page is actually hot;
splitting a 2\,MB page all the way down to 4\,KB pages loses the TLB performance benefits entirely, even when a 64\,KB or 128\,KB split would have been sufficient.} %Splitting the page into base pages sacrifices the TLB benefits of large mappings.
%No existing system explores the middle ground of splitting a THP into intermediate mTHP sizes prior to migration—a gap that \name fills. \asaf{I don't think you need to introduce \name yet, this is a background section}
%\textcolor{sherry}{MEMTIS~\cite{lee2023memtis} recognized this problem of skewed access within THPs. It dynamically ``splinters'' huge pages (2\,MB) when base-page (4\,KB) utilization is low. But MEMTIS only considers \emph{two sizes}---2\,MB and 4\,KB---and does not explore intermediate mTHP granularities. Splitting a 2\,MB page all the way down to 4\,KB pages discards the TLB performance benefits entirely, even when a 64\,KB or 128\,KB split would have been sufficient. Furthermore, MEMTIS's design \emph{cannot} scale to support intermediate mTHP sizes. The root cause is twofold. First, MEMTIS relies on \emph{per-page metadata tracking}: it maintains per-subpage access metadata for every 4\,KB subpage within each huge page, consuming up to 0.195\% of the total memory footprint~\cite{lee2023memtis} and growing linearly with the working set size (e.g., approximately 2\,GB on a 1\,TB server). Second, MEMTIS decides whether to split a huge page by comparing the actual fast-tier hit ratio against an estimated base-page hit ratio. That means that supporting $N$ candidate mTHP sizes in MEMTIS would require $N$ separate emulated histograms and hit-ratio estimations, multiplying both memory footprint and splitting decision computation. In short, MEMTIS's profiling design couples overhead to both the working set size and the number of candidate granularities, making it fundamentally unsuitable for multi-size THP splitting. \name fills this gap: it selects from the full spectrum of mTHP sizes using a fixed-size global subpage histogram (4\,KB per application), decoupling profiling cost from both the working set and the number of candidate granularities.}
\textcolor{checked}{Furthermore, MEMTIS's design cannot scale to support intermediate mTHP sizes, for three reasons. First, MEMTIS relies on per-page metadata tracking, which is not scalable: MEMTIS maintains per-subpage access metadata for every 4\,KB subpage within each huge page,
%consuming up to 0.195\% of the total memory footprint~\cite{lee2023memtis} and
which grows linearly with the working set size of the workload (e.g., tens of GB on a TB-scale server). The metadata is typically placed in fast memory because of its high access frequency, taking most of the fast memory capacity (tens of GBs~\cite{10.1145/3575693.3578835}) in a data-center server. This leaves little space in the fast tier for workload pages that could improve performance.}
%(e.g., approximately 2\,GB on a 1\,TB server).
\textcolor{checked}{Second, MEMTIS can lead to large runtime overhead. MEMTIS decides whether to split a huge page by comparing the measured fast-tier hit ratio against an estimated base-page hit ratio. That means that supporting $N$ candidate mTHP sizes in MEMTIS would need to build and maintain $N$ separate emulated histograms and hit-ratio estimates, multiplying both memory footprint and splitting-decision computation. Evaluating with memory-intensive benchmarks (LU.D and SP.D shown in Table~\ref{tab:benchmarks}), we observe that MEMTIS leads to up to 15\% runtime overhead because of the above computation and metadata management.}
\textcolor{checked}{Third, MEMTIS is architecture-dependent. In order to calculate fast-tier hit ratio to decide page size, MEMTIS relies on specific hardware counters to determine from which tier a memory access comes. This limits the generality of MEMTIS on diverse memory architectures.}
This motivates supporting multi-size THP during page migration in a scalable and architecture-independent manner.
%we call for a system design that support multi-size THP
%In short, MEMTIS's profiling design couples overhead to both the working set size and the number of candidate granularities, making it fundamentally unsuitable for multi-size THP splitting.
%\name fills this gap: it selects from the full spectrum of mTHP sizes using a fixed-size global subpage histogram (4\,KB per application), decoupling profiling cost from both the working set and the number of candidate granularity.
\begin{comment}
\subsection{Memory Architectures}
\label{subsec:bg-arch}
Not all slow-tier memory is created equal. We use two memory architectures as an example in this paper. %Two widely deployed technologies illustrate this.
\textbf{CXL-based memory expansion.} Compute Express Link (CXL) provides a cache-coherent interconnect for attaching external memory. CXL operates in full-duplex mode: reads and writes use separate channels and can proceed concurrently. This means CXL memory can sustain higher effective bandwidth under mixed traffic compared to a memory architecture with half-duplex DDR interface~\cite{wang2025cxl_performance}.
\asaf{I'd recommend describing CXL before Optane, since Optane was discontinued}
\asaf{Even for CXL, it can be deployed in many different ways (as a single host main memory extension, using MHD across hosts, over a switch etc. It can have different ratios of CXL vs local DRAM. I'd consider focusing on the heterogeneity within CXL deployments and deemphasizing Optane, since Optane itself has been discontinued and has never really been deployed at scale in production}
\textbf{Intel Optane Persistent Memory (PMEM).}
Optane PMEM connects via the DDR interface, which is \emph{half-duplex}: reads and writes share the same channel and cannot proceed simultaneously. Under mixed read/write traffic, effective bandwidth degrades because reads and writes must be serialized on the bus~\cite{wang2025cxl_performance}.
\asaf{There are other problems with Optane, like small writes can incur write amplification if I recall correctly}
This architectural difference has direct implications for page migration. On a full-duplex CXL link, migrating a read- or write-heavy page can generate balanced read/write traffic, further increasing bandwidth utilization for co-located applications. On a half-duplex Optane link, the same migration has far less effect. An effective memory tiering system must account for these differences---yet existing systems use a single, architecture-agnostic migration policy regardless of the underlying hardware.
\end{comment}
%\subsection{eBPF for Kernel Customization}
%\label{subsec:bg-ebpf}
%Extended Berkeley Packet Filter (eBPF) is a Linux kernel technology that allows user-defined programs to run in the kernel in a sandboxed environment, without modifying kernel source code or loading kernel modules. The eBPF verifier guarantees safety: programs cannot crash the kernel, access arbitrary memory, or enter infinite loops.
\textbf{eBPF for kernel customization.} Extended Berkeley Packet Filter (eBPF) is a Linux kernel technology that enables user-defined programs to run safely inside the kernel in a sandboxed environment, without requiring kernel source modifications or custom kernel modules. The eBPF verifier enforces strict safety guarantees: programs cannot crash the kernel, access arbitrary memory, or execute unbounded loops.
Originally designed for network packet filtering, eBPF has evolved into a general-purpose mechanism for extending kernel functionality. Recent work highlights its versatility in customizing core kernel subsystems~\cite{sched_ext,zussman2025cache_ext,mores2024ebpf_mm}.
%For example, sched\_ext~\cite{sched_ext} enables user-defined CPU scheduling policies through BPF, allowing rapid experimentation without recompiling the kernel. cache\_ext~\cite{zussman2025cache_ext} applies eBPF to the Linux page cache, supporting application-specific caching strategies. eBPF-mm~\cite{mores2024ebpf_mm} introduces a hook in the page-fault path to select page sizes at allocation time.
Collectively, these systems illustrate a broader trend: eBPF is making kernel memory management programmable.
%\name builds on this trend by using eBPF to make page-migration policies programmable—both in terms of migration granularity (mTHP size selection) and migration strategy (architecture-aware decisions). \asaf{I'd remove these last two sentences}
%Originally designed for network packet filtering, eBPF has evolved into a general-purpose kernel extension mechanism. Recent work has demonstrated its power for customizing core kernel subsystems. sched\_ext~\cite{sched_ext} allows user-defined CPU scheduling policies via BPF struct\_ops, enabling rapid iteration on scheduling algorithms without kernel recompilation. cache\_ext~\cite{zussman2025cache_ext} uses eBPF to customize the Linux page cache, allowing application-specific caching strategies. eBPF-mm~\cite{mores2024ebpf_mm} introduces an eBPF hook in the page fault path for page-size selection at \emph{allocation} time. These systems validate a broader trend: eBPF makes kernel memory management \emph{programmable}. \name builds on this trend, using eBPF to make page migration policies programmable---both in terms of migration granularity (mTHP size selection) and migration strategy (architecture-aware decisions).
\section{Motivation}
\label{sec:motivation}
%\textcolor{red}{(Compare the performance on Optane and CXL. Maybe using Memtis?)} \textcolor{red}{(memory contention on cloud servers with THP.)}
Despite recent advances in tiered memory architecture, THP support, and eBPF programmability, today's memory tiering stack suffers from three fundamental limitations. %%which we review below. % demonstrated in \S\ref{subsec:mot-dilemma}, \S\ref{subsec:mot-contention}, and \S\ref{subsec:mot-arch}.
\subsection{THP Migration Dilemma}
\label{subsec:mot-dilemma}
When THP is enabled, the kernel manages memory in 2\,MB pages by default. %This is good for TLB performance but
Consider a 2\,MB THP where only a small cluster of subpages is frequently accessed.
Existing memory tiering systems face two suboptimal choices.
\textbf{Option 1 (Linux default):} migrate the full 2\,MB THP. This wastes migration bandwidth---the system copies 2\,MB of data even if only 64\,KB is hot---and consumes scarce fast-tier capacity by allocating 2\,MB of DRAM for cold data.
\textcolor{checked}{Moreover, migrating a 2\,MB page requires a contiguous 2\,MB free region on the destination node, which is often unavailable under memory fragmentation, causing migrations to fail (\S\ref{sec:eval_deep}).}
%It also wastes fast-tier capacity, occupying 2\,MB of scarce DRAM for data that are mostly cold.
\textbf{Option 2 (e.g., MEMTIS~\cite{lee2023memtis}):} split the THP into 4\,KB base pages and migrate the hot ones. While this preserves bandwidth and fast-tier space, it destroys TLB coverage. After splitting, the application experiences more TLB misses, which can offset the performance gains of page migration. %placing hot data in fast memory.
\begin{figure}[!t]
\centering
% TODO: Insert histogram showing distribution of hot subpages within THPs
% across different workloads. X-axis: fraction of hot 4KB subpages within
% each 2MB THP. Y-axis: number of THPs.
\includegraphics[width=1\linewidth]{./figure/motivation_subpage_access_histogram.pdf}
\caption{Hot subpage distribution within 2\,MB THPs. We use workloads summarized in Table~\ref{tab:benchmarks}.}
\label{fig:hot-subpage-dist}
\end{figure}
%%%For most workloads, accesses are concentrated to only a small fraction of each THP, making full 2\,MB migration wasteful and full 4\,KB splitting unnecessary.
%\asaf{We should say what these workloads are somewhere}
%Figure~\ref{fig:hot-subpage-dist} shows the distribution of hot 4\,KB subpages within 2\,MB THPs across representative workloads. Access is rarely uniform: most THPs have a concentrated hot region that spans tens to hundreds of kilobytes---neither a single 4\,KB page nor the full 2\,MB. This suggests that intermediate mTHP sizes (e.g., 64\,KB or 256\,KB) could capture the hot region precisely, migrating only what is needed while preserving TLB coverage for the migrated region.
Figure~\ref{fig:hot-subpage-dist} shows the distribution of hot 4\,KB subpages within 2\,MB THPs across representative workloads. Access patterns are rarely uniform: most THPs contain a concentrated hot region spanning tens to hundreds of kilobytes---far larger than a single 4\,KB page but far smaller than the full 2\,MB. This suggests that intermediate mTHP sizes (e.g., 64\,KB or 256\,KB) could more precisely capture the hot region, migrating only what is necessary while preserving TLB coverage for the migrated portion.
%\textcolor{red}{(observation 1 can be simplified if we don't have space, because MEMTIS paper has mentioned the similar observation before.)}
\observation{Only a small subset of subpages within each THP is hot. Migrating at 2\,MB wastes bandwidth and fast-tier capacity; splitting to 4\,KB discards TLB benefits. Intermediate mTHP sizes can capture the hot region without either penalty.}
\label{obs:subpage-skew}
\subsection{There is No One Ideal Page Size}
\label{subsec:mot-contention}
%One Size Does Not Fit All Conditions
%Even if we could choose an mTHP size, which size is the right one? The answer depends on the runtime environment.
Deciding the mTHP size is challenging. With different levels of memory contention, the same application prefers different page sizes to improve performance.
\begin{figure}[t]
\centering
% TODO: Two sub-figures side by side.
% (a) Application performance (normalized) with different mTHP sizes,
% WITHOUT memory contention (solo execution).
% (b) Application performance (normalized) with different mTHP sizes,
% WITH memory contention (co-located workloads competing for fast tier).
\includegraphics[width=1\linewidth]{./figure/motivation_throughput_comparison_arrow.pdf}
\caption{The optimal mTHP size shifts under memory contention, as per \S\ref{sec:eval_method}. \textcolor{checked}{Stars indicate the best performance.}}
\label{fig:contention-size}
\end{figure}
%Without contention, larger pages (e.g., 2\,MB) perform best due to high TLB coverage. Under contention, smaller mTHP sizes (e.g., 64\,KB) are preferable because they consume less fast-tier capacity.
%When an application runs alone without memory pressure on the slow tier, larger THP sizes are preferred. The slow tier maintains short memory access latency, which is comparable to that of the fast tier. In this case, the latency of huge-page migration is small, and the bandwidth consumed by huge-page migration does not impact the performance of the slow tier. Therefore, the TLB benefit of 2\,MB pages dominates. But in cloud environments where multiple workloads share the same machine and memory tiers, the calculus changes. Under contention on the slow tier, migrating 2\,MB pages incurs higher overhead. On the one hand, migrating mistakenly detected hot pages wastes memory bandwidth. On the other hand, hot pages residing in slow memory suffer a greater performance penalty because of the increased memory access latency of the slow tier under contention. Therefore, smaller mTHP sizes waste less slow-tier bandwidth and less fast-tier capacity per migration, allowing more distinct hot regions to reside in fast memory simultaneously.
When an application runs alone and the slow tier is not under bandwidth pressure, larger THP sizes are generally preferable. In this setting, the slow tier can offer relatively low memory-access latency~\cite{vuppalapati2024colloid},
%—\st{close to that of the fast tier}-
so the cost of migrating huge pages is small, and the bandwidth consumed by such migrations does not meaningfully affect performance. As a result, the TLB benefits of 2\,MB pages dominate. However, in cloud environments where multiple workloads share the same machine and memory tiers, the trade-offs shift. Under contention in the slow tier, migrating 2\,MB pages becomes more expensive: misidentified hot pages waste migration bandwidth, and truly hot pages left in the slow tier suffer greater performance penalties due to increased access latency. In these scenarios, smaller mTHP sizes reduce both slow-tier bandwidth consumption and fast-tier capacity usage per migration, allowing more distinct hot regions to reside in fast memory simultaneously.
%Figure~\ref{fig:contention-size} illustrates that the optimal mTHP size shifts from 2\,MB to a smaller mTHP size under memory contention. This means that any \emph{static} page size policy---whether system-wide sysfs configuration or per-application profiles---will be suboptimal in dynamic, multi-tenant environments. The system needs to adapt its mTHP size selection at runtime based on the current memory architecture and memory pressure.
Figure~\ref{fig:contention-size} shows the optimal mTHP size under memory contention. The results show that there is no one-size-fits-all size: the optimal page size is different when varying the workloads and memory contention level. This implies that any static page-size policy---whether configured system-wide via sysfs or defined per application---will be suboptimal in a dynamic, multi-tenant environment. The system must instead adapt its mTHP size selection at runtime, guided by the memory architecture and prevailing memory pressure.
Linux mTHP~\cite{linux_mthp} allows administrators to enable or disable specific mTHP sizes via sysfs, but this is a static, manual configuration. It does not adapt to runtime conditions. eBPF-mm~\cite{mores2024ebpf_mm} uses eBPF to select page sizes at allocation time, but it relies on pre-defined application profiles and does not consider page migration. %in tiered memory.
Neither system provides \emph{dynamic}, contention-aware mTHP size selection for page migration.
\observation{The optimal mTHP size depends on workload and runtime memory contention.
Under low contention, larger pages are preferred for TLB coverage; under high contention, smaller mTHP sizes cause less overhead, waste less fast-tier capacity, and yield better overall performance.
A static size policy cannot adapt to these changes.}
\label{obs:contention}
\subsection{Architecture-Agnostic Migration}
\label{subsec:mot-arch}
%wastes Bandwidth
%, leaving substantial performance potential untapped.
%This is a missed opportunity, because different architectures have fundamentally different traffic characteristics.
\begin{figure}[!t]
\centering
% TODO: Two sub-figures showing bandwidth utilization.
% (a) Intel Optane: bandwidth under mixed read/write traffic showing
% half-duplex degradation.
% (b) CXL memory: bandwidth under mixed read/write traffic showing
% full-duplex behavior with independent read/write channels.
\includegraphics[width=1\linewidth]{./figure/motivation_bw_latency.pdf}
\caption{Memory bandwidth differs on different memory tiering architectures.}
\label{fig:arch-bandwidth}
\end{figure}
%Optane's half-duplex DDR link suffers under mixed read/write traffic, while CXL's full-duplex link sustains bandwidth regardless of traffic mix.
Existing tiered memory systems apply the same migration policy without considering the underlying memory architecture.
To explain why this is problematic, we use a case study of CXL memory and Optane persistent memory.
CXL memory provides full-duplex links where reads and writes use independent channels~\cite{yang2025cxlaimpodcxlmemoryneed}.
%On a CXL system with its full-duplex link, reads and writes use independent channels.
The migration policy can take advantage of this property by actively rebalancing the read/write traffic across the full-duplex channels. %Specifically, when CXL traffic is read-dominated, selectively promoting read-heavy pages while keeping write-heavy pages reduces CXL read pressure; when CXL traffic is write-dominated, promoting write-heavy pages while keeping read-heavy pages relieves CXL write pressure.
%On an Optane system with its half-duplex DDR interface, the picture is fundamentally different. Since read and write traffic compete for the same link, rebalancing read/write traffic does not improve bandwidth utilization but merely shifts contention from one direction to the other. Worse, selectively holding back read- or write-heavy pages from promotion prevents genuinely hot pages from reaching the fast tier, degrading overall performance. As a result, the same traffic-rebalancing policy that benefits CXL platform can hurt performance on Optane platform.
In contrast, an PMEM system has half-duplex DDR interfaces where read and write traffics compete for the same link. As a result, rebalancing read/write traffic does not improve bandwidth utilization but merely shifts contention from one direction to the other. Worse, selectively holding back read- or write-heavy pages from promotion prevents genuinely hot pages from reaching the fast tier, degrading overall performance. %Hence, a traffic-rebalancing policy that benefits the CXL platform can lose performance on the Optane platform. We must customize memory tiering system to the underlying memory architecture.
% As a result, a traffic‑rebalancing policy that performs well on a CXL platform can actually hurt performance on an Optane platform. Memory‑tiering systems must therefore be tailored to the underlying memory architecture. Figure~\ref{fig:arch-bandwidth} demonstrates this behavior by highlighting the bandwidth difference when using the same migration decision on the two memory architecture. \asaf{Maybe say something more about the figures} %This bandwidth difference shown in the figure implies that the same migration decision, whether to promote a page with a given read/write ratio, should yield different answers depending on the architecture.
As a result, a traffic‑rebalancing policy that performs well on a CXL platform can lose effectiveness on a PMEM platform. Memory‑tiering systems must therefore be tailored to the memory architecture. Figure~\ref{fig:arch-bandwidth} illustrates this discrepancy: CXL \textcolor{checked}{reaches 149\% and 125\% of the read-only peak bandwidth on PCIe4 and PCIe5, respectively,} with the mixed read/write pattern by leveraging its full‑duplex link, whereas PMEM, constrained by its half‑duplex DDR interface, experiences substantial bandwidth degradation (\textcolor{checked}{only 17\% of the read-only peak bandwidth}) under the same traffic pattern.
%These architectural differences imply that identical migration decisions can lead to markedly different performance outcomes depending on the memory architecture.
%As a result, a traffic‑rebalancing policy that performs well on a CXL platform can lose performance on an Optane platform. Memory‑tiering systems must therefore be tailored to the underlying memory architecture. Figure~\ref{fig:arch-bandwidth} supports the above conclusion: CXL sustains near-peak aggregate bandwidth across all read/write ratios thanks to its full-duplex link, while PMEM's half-duplex DDR interface suffers significant bandwidth degradation under mixed read/write traffic. Such architecture-level differences indicate that the same migration decisions can yield different performance, depending on the memory architecture.}
Despite this, most memory tiering systems (e.g., AutoNUMA~\cite{autonuma}, TPP~\cite{maruf2023tpp}, Nimble~\cite{yan2019nimble}, MEMTIS~\cite{lee2023memtis}, Colloid~\cite{vuppalapati2024colloid}, and Nomad~\cite{xiang2024nomad})
%all use architecture-agnostic migration policies.
do not fully consider architectural differences and cannot flexibly customize the policy for the hardware deployment.
%Different memory architectures exhibit fundamental different traffic characteristics (e.g., half-duplex vs.\ full-duplex).
\observation{Migration policies produce different outcomes on different hardware configurations. We must customize the policy to specific tiered-memory hardware setups to maximize performance.}
\label{obs:arch}
%A software-based solution that adapts migration policy to the memory architecture---without requiring hardware modifications---remains an open problem.
\begin{comment}
\subsection{A Case for Programmable Migration}
\label{subsec:mot-ebpf}
\textcolor{red}{(this section may be put into the introduction.)}
The above observations call for flexibility in the memory tiering system, accounting for workload characteristics, runtime performance feedback, and memory architecture.
%point to a common theme: the right migration decision depends on factors that vary across workloads, runtime conditions, and hardware configurations. No single hard-coded policy can be optimal across all these dimensions.
%This is the kind of problem that eBPF is designed to solve.
eBPF can be used to address this problem. Rather than modifying the kernel to design a page migration policy for each combination of workload, runtime performance, and memory architecture, we can expose well-defined hook points in the kernel's migration path and let eBPF programs implement the policy. Because eBPF programs are verified for safety before execution and run in a lightweight in-kernel sandbox, this approach introduces negligible overhead while allowing policies to be updated, tuned, and customized without kernel recompilation.
\textcolor{red}{(put this comparison to the related work section.)}
Table~\ref{tab:comparison} compares \name with the existing efforts.
%summarizes how \name compares to prior work across the key dimensions we have identified.
\end{comment}
\begin{comment}
\begin{table}[t]
\centering
\small
\caption{Comparison of memory tiering systems. }
\label{tab:comparison}
\begin{tabular}{lccccc}
\toprule
\textbf{System} & \textbf{Granularity} & \textbf{Dynamic} & \textbf{Arch-} & \textbf{eBPF} \\
& & \textbf{Size} & \textbf{Aware} & \textbf{Based} \\
\midrule
AutoNUMA & 4KB/2MB & \ding{55} & \ding{55} & \ding{55} \\
TPP & 4KB/2MB & \ding{55} & \ding{55} & \ding{55} \\
Nimble & 2MB & \ding{55} & \ding{55} & \ding{55} \\
MEMTIS & 4KB/2MB & Two sizes & \ding{55} & \ding{55} \\
Colloid & 4KB/2MB & \ding{55} & \ding{55} & \ding{55} \\
Nomad & 4KB/2MB & \ding{55} & \ding{55} & \ding{55} \\
\textbf{\name} & \textbf{mTHP range} & \ding{51} & \ding{51} & \ding{51} \\
\bottomrule
\end{tabular}
\end{table}
\end{comment}
\section{Key Ideas in \name}
\label{sec:key-ideas}
\name addresses the three gaps identified in \S\ref{sec:motivation}. First, it splits 2\,MB THPs into optimal mTHP sizes based on hot subpage distribution, avoiding both bandwidth waste and unnecessary TLB misses (\S\ref{subsec:mot-dilemma}). %TLB coverage loss %\asaf{rephrase: "TLB coverage loss", unclear}
Second, it activates the THP splitting policy based on runtime memory contention (\S\ref{subsec:mot-contention}). %\asaf{"dynamically activates" is unclear}
Third, it customizes migration decisions based on memory architecture characteristics (\S\ref{subsec:mot-arch}). The three policies are implemented as eBPF programs, making them safe, low-overhead, and deployable without further kernel modifications.
%The previous section identified three gaps in existing memory tiering systems: migrating at 2\,MB or 4\,KB granularity is a lose-lose choice (\S\ref{subsec:mot-dilemma}), no system adapts migration granularity to runtime contention (\S\ref{subsec:mot-contention}), and architecture-agnostic migration policies leave bandwidth on the table (\S\ref{subsec:mot-arch}). This section outlines how \name addresses each gap.
%\subsection{Contention-Aware mTHP Splitting}
%\label{subsec:idea-splitting}
\begin{comment}
\textcolor{red}{(this paragraph may be deleted.)}
The core problem is granularity. A 2\,MB THP contains 512 subpages, but typically only a cluster of them is hot (Observation~\ref{obs:subpage-skew}). Migrating all 512 subpages wastes bandwidth; splitting to individual 4\,KB pages discards TLB coverage.
% The key insight is that the hot region within a THP often aligns naturally with an intermediate mTHP boundary, for example 64\,KB, 256\,KB, or 1024\,KB, that is large enough to preserve TLB benefits yet small enough to avoid migrating cold data.
The key insight is that the hot region within a THP often aligns naturally with an intermediate mTHP boundary (e.g., 64\,KB, 256\,KB, or 1\,MB) that is large enough to preserve TLB benefits yet small enough to avoid migrating cold data.
\end{comment}
%\asaf{Shouldn't the first key idea be how you are able to estimate sub page access patterns? That seems like the most basic challenge you solved}
\textbf{Lightweight memory profiling.} \textcolor{checked}{Each policy decision in \name---which mTHP size to split into, which subpages are hot, and whether to admit a migration---relies on knowing page access patterns. \name maintains a \emph{global subpage histogram}---a fixed 512-entry array (4\,KB) per application---that \textcolor{checked}{maps} hardware-sampled accesses from all THPs \textcolor{checked}{to} their subpage offset. This constant-size structure does not grow with the working set of the workload. The profiling logic runs as an eBPF program attached to hardware performance events, aggregating samples directly in interrupt context and avoiding context switches and data copies between kernel and user space (\S\ref{subsec:profiling}). This lightweight profiling builds a foundation to guide three policy decisions, described below.}
\textbf{Contention-aware mTHP splitting.} \name dynamically selects the \emph{largest} mTHP size that captures most of the hot subpages within a dense memory region. The mTHP size is not a static configuration. Because the optimal size changes as memory contention varies (Observation~\ref{obs:contention}), \name activates THP splitting only when the slow-tier bandwidth is under pressure. %and keeps pages at 2\,MB otherwise.
\textcolor{checked}{Beyond reducing bandwidth waste, splitting also reduces migration failures: a smaller subfolio requires far less contiguous free space than a 2\,MB THP, so migrations succeed more often under memory fragmentation.}
%The decision is driven by a lightweight sub-page access profile built from hardware sampling, described in \S\ref{subsec:profiling}.
%\subsection{Architecture-Aware Migration Policy}
%\label{subsec:idea-admission}
%Even when a page is hot enough to promote, migrating it may not always help. On a CXL system with full-duplex channels, reads and writes flow independently. If CXL read bandwidth is the bottleneck, promoting read-heavy pages relieves pressure---but promoting write-heavy pages achieves little, because the CXL write channel is not congested (Observation~\ref{obs:arch}). On a half-duplex Optane system, the same selective policy would hurt, since reads and writes share one channel and all hot pages benefit equally from promotion.
\textbf{Architecture-aware migration policy.} Even when a page is hot enough to merit promotion, migrating it does not always help. On a CXL system with full‑duplex channels, reads and writes proceed independently. If CXL read bandwidth is the bottleneck, promoting read‑heavy pages alleviates pressure, but promoting write‑heavy pages offers little benefit, because the write channel is typically uncongested (Observation~\ref{obs:arch}). On a half‑duplex PMEM system, however, the same selective policy would be counterproductive: reads and writes share a single channel, so all hot pages benefit equally from promotion.
%\name introduces an architecture-aware \emph{migration admission} hook: before allowing a page to be promoted, an eBPF program checks the page's read/write ratio against real-time CXL traffic. Write-heavy pages are held back unless the CXL write channel is truly saturated. This hook is activated only on CXL platforms and is a no-op on Optane or single-tier systems.
\name adds an architecture‑aware migration‑admission hook. Before a page is promoted, an eBPF program evaluates its read/write ratio in the context of real‑time CXL traffic. Write‑heavy pages are deferred unless the CXL write channel is genuinely saturated. This hook is enabled only on CXL platforms and becomes a no‑op on PMEM (or other half-duplex memory architectures), or single‑tier systems.
%\subsection{eBPF as the Policy Layer}
%\label{subsec:idea-ebpf}
%The above two ideas require policy logic \emph{inside} the kernel's migration path, yet the right policy depends on the workload, the runtime performance, and memory architecture. Hard-coding any single policy into the kernel would be brittle.
%\name uses eBPF to decouple the migration policy from the migration mechanism. Three hooks in the kernel's NUMA migration path---one for split-size selection, one for subpage hot/cold classification, and one for migration admission---are exposed as eBPF attachment points. When no eBPF program is loaded, the kernel falls back to safe defaults (no splitting, allow all migrations). When a program is loaded, it overrides the default with a policy tailored to the current environment. This design achieves three properties simultaneously: \emph{safety} (the eBPF verifier guarantees termination and memory safety), \emph{low overhead} (policies execute in-kernel at native speed), and \emph{deployability} (policies can be updated without kernel recompilation).
\textbf{eBPF as the policy layer.} \name uses eBPF to decouple migration policy from migration mechanism. It exposes three hooks along the kernel’s NUMA migration path as eBPF attachment points: one for split‑size selection, one for subpage hot/cold classification, and one for migration admission. When no eBPF program is loaded, the kernel falls back to safe defaults (no splitting and unconditional migration).
% When a program is present, it overrides these defaults with a policy tailored to the current environment. \tal{this sentence can be deleted}
%This design simultaneously achieves three goals: safety (the eBPF verifier enforces termination and memory safety), low overhead (policies run in‑kernel at native speed), and deployability (policies can be updated without recompiling the kernel). \asaf{I think you can remove this last sentence, it's repetitive at this point in the paper}
%Figure~\ref{fig:overview} shows \name's overall architecture: user-space daemons generates policy decisions from profiling data, which are shared with in-kernel eBPF programs through pinned BPF maps. The eBPF programs then enforce these policies on the kernel's NUMA migration path.
% Figure~\ref{fig:overview} illustrates the architecture of \name. User‑space daemons derive policy decisions from profiling data and publish policy decisions to pinned BPF maps. In‑kernel eBPF programs then enforce these policies on the kernel's NUMA migration path. \asaf{I'd recommend describing the architecture in more detail.}
\textcolor{checked}{Figure~\ref{fig:overview} illustrates the architecture of \name, which spans three layers. In the kernel, an eBPF program attached to hardware performance events profiles memory accesses and aggregates them into BPF maps. User‑space daemons derive policy decisions from the profiling data and publish them to the pinned BPF maps. On the kernel's NUMA migration path, three eBPF hooks serving as page migration admission control enforce these decisions at migration time: (1)~\emph{mTHP Splitting Size} selects the target mTHP granularity, (2)~\emph{Hot/Cold Subfolio Classify} determines which subfolios to promote or not, and (3)~\emph{Migration Admission} accepts or defers a migration based on the current CXL traffic pattern.}
\begin{figure}[!t]
\centering
% TODO: System overview diagram showing:
% Left: User-space daemons (pid_daemon, subpage_analyzer, cxl_traffic_daemon)
% Center: Pinned BPF maps (shared state)
% Right: Kernel eBPF programs at three hook points
% Bottom: NUMA fault → migration path
\includegraphics[width=1\columnwidth]{./figure/overview.pdf}
\caption{\name architecture overview.}
\label{fig:overview}
\end{figure}
%%%Hardware sampling feeds access profiles into eBPF maps; user-space analyzers compute splitting decisions; in-kernel eBPF programs materialize policies at the three hook points on the NUMA migration path.
\section{\name Design}
\label{sec:design}
\begin{table}[!t]
\centering
\small
\caption{Three eBPF hooks in \name.}
\label{tab:hooks}
\begin{tabular}{p{2.8cm}p{2.0cm}p{2.2cm}}
\toprule
\textbf{Hook} & \textbf{Location} & \textbf{Default} \\
\midrule
\texttt{bpf\_thp\_pick\_\newline split\_order} & \texttt{mm/migrate.c} & 2MB (no split) \\
\addlinespace
\texttt{bpf\_subpage\_\newline is\_cold} & \texttt{mm/migrate.c} & 0 (all hot) \\
\addlinespace
\texttt{bpf\_thp\_numa\_\newline migrate\_admission} & \texttt{kernel/sched/\newline fair.c} & 0 (allow) \\
\bottomrule
\end{tabular}
\end{table}
\textcolor{checked}{We describe the three core mechanisms in \name. First, we describe the lightweight memory profiling subsystem (\S\ref{subsec:profiling}), which combines a fixed-size global subpage histogram, a dual blocked counting Bloom filter (bCBF), and eBPF-based in-kernel sampling to collect access patterns with minimal overhead. Second, we present the mTHP splitting algorithm (\S\ref{subsec:splitting}), which uses normalized Shannon entropy to detect skewed access distributions and selects the largest mTHP size whose heat density is sufficient to capture hot regions, ensuring that only hot subfolios are migrated. Third, we describe the duplex-aware migration admission control (\S\ref{subsec:admission}), which classifies each page's read/write ratio and admits or defers migrations based on real-time memory traffic.}
\textcolor{checked}{\name realizes its policies through three eBPF hooks inserted into the kernel's NUMA-migration path (Table~\ref{tab:hooks}). All three follow the same design pattern: a \texttt{noinline} kernel function annotated with \texttt{ALLOW\_ERROR\_INJECTION} that returns a safe default when no eBPF program is attached, but whose return value can be overridden by an \texttt{fmod\_ret} eBPF program. These hooks enable the three core mechanisms. We discuss the design in detail in this section.}
\subsection{Lightweight Memory Profiling}
\label{subsec:profiling}
Every policy decision in \name---which mTHP size to split to, which subfolios to promote, whether to admit a migration---relies on access profiles collected from hardware event sampling. The profiling must be both effective and cheap.
\subsubsection{Global Subpage Histogram in mTHP Splitting}
\label{subsubsec:histogram}
%\textcolor{sherry}{Even within the kernel, the choice of profiling data structure matters. Prior memory tiering systems such as MEMTIS~\cite{lee2023memtis} and HeMem~\cite{raybuck2021hemem} maintain per-page access metadata using page table entries or hash tables~\cite{10.1145/3676642.3736119}. These structures grow linearly with the working set: MEMTIS, for example, consumes up to 0.195\% of the total memory footprint (e.g., approximately 2\,GB on a 1\,TB server)~\cite{lee2023memtis}. Moreover, each hardware event sample requires a lookup into these per-page structures, incurring hash computations or multi-level page table traversals that can cause cache misses when the metadata exceeds the LLC capacity~\cite{10.1145/3676642.3736119}.}
%\asaf{I'd recommend flipping the order of 5.1.1 and 5.1.2. The hot subpage sampling is a more core academic contribution, than just processing profiling logic in eBPF, which I think a lot of eBPF-based profilers already do, and is more of just an implementation detail}
For mTHP splitting, what matters is not the exact access count of each individual page, but the \emph{access distribution pattern within a 2\,MB THP region} (i.e., which subpages are hot and which are cold). Instead of creating a subpage histogram of 512 entries per huge page, \name captures hot subpage distribution with a global subpage histogram: an array of 512 entries (representing 512 subpage offsets) per application, where each hardware event sample is mapped to a slot by: \texttt{page\_addr \% 511}. This single \textcolor{checked}{modulo} operation \textcolor{checked}{maps} accesses from all 2\,MB THPs of an application into one 512-slot array, requiring only $512 \times 8\,\text{B} = 4\,\text{KB}$ per application regardless of working set size.
\textbf{Why can a global histogram work?} Folding all THPs into one histogram loses access distribution patterns for individual THPs.
%individual per-THP patterns.
However, the key insight is that %\name uses sampling-based profiling,
with sampling-based profiling, the hardware sampler naturally focuses on frequently accessed pages. The global histogram is therefore dominated by the subpage distribution of \emph{hot} THPs---precisely the ones that are candidates for migration. Cold THPs contribute few samples and have little influence on the histogram. Since cold THPs should remain in slow memory and are not to be migrated, their subpage distribution is irrelevant to splitting decisions.
In other words, the global histogram is not a low-quality approximation to access distribution patterns. Rather, it is a direct representation of the THPs that matter for migration.
\begin{comment}
\paragraph{Overhead comparison with per-page tracking.}
The alternative---maintaining a per-page histogram as in MEMTIS~\cite{lee2023memtis}---scales with the working set.
MEMTIS adds 16\,B of metadata per 4\,KB page; on a 1\,TB server this translates to 3.9\,GB of metadata.
Table~\ref{tab:profiling-overhead} summarizes the key differences.
The global histogram uses a BPF \emph{array} lookup (a single pointer-arithmetic operation) rather than a hash map lookup (hash computation, bucket traversal, possible lock contention).
The 4\,KB array fits entirely in L1 cache, avoiding the scattered cache misses of a large hash table.
\begin{table}[t]
\centering
\small
\caption{Profiling data structure comparison. The global subpage histogram
achieves constant memory and constant-time updates regardless of working set size.}
\label{tab:profiling-overhead}
\begin{tabular}{lcc}
\toprule
& \textbf{Per-page tracking} & \textbf{Global histogram} \\
& (MEMTIS-style) & (\name) \\
\midrule
Data structure & Hash map & Fixed array \\
Memory per PID & $O(\text{working set})$ & $O(1)$: 4\,KB \\
Lookup type & Hash + probe & Array index \\
Cache behavior & Scattered misses & Fits in L1 \\
Scales with & Working set size & Nothing \\
\bottomrule
\end{tabular}
\end{table}
\end{comment}
\subsubsection{eBPF-Based Kernel Profiling}
\label{subsubsec:ebpf-profiling}
Prior tiering systems such as MEMTIS~\cite{lee2023memtis} and HeMem~\cite{raybuck2021hemem} process hardware event samples in user-space. As shown in Figure~\ref{fig:ebpf-vs-userspace}, the user-space approach follows a multi-step data path: the kernel serializes each sample into a perf ring buffer, wakes up a user-space thread, and the thread reads and parses each sample before updating its own metadata. This path incurs substantial overhead from multiple sources.
First, each raw sample gets copied multiple times: the kernel serializes a \texttt{perf\_record\_sample} into the perf ring buffer, the user-space thread reads it from the ring buffer, deserializes each field, and writes the extracted result into its own hash table.
Second, delivering these samples requires context switches: the kernel must wake up the user-space thread, and the thread returns to sleep after processing each batch. Third, processing $N$ individual samples in user-space also forces the CPU to alternate between ring buffer pages and hash table pages, causing CPU cache pollution as profiling metadata competes with application data for cache capacity. At high sampling rates (e.g., $10^5$--$10^6$ samples/sec), these costs compound and become significant.
\name avoids all these overheads by running the profiling logic inside the kernel as an eBPF program attached to perf events (as shown in Figure~\ref{fig:ebpf-vs-userspace}). When a hardware event sampling interrupt fires, the eBPF program executes directly in the interrupt context.
It reads the sampled address, computes the update, and writes to a BPF map---all without entering user-space.
The ring buffer is bypassed entirely: the raw sample never leaves the kernel, requiring no serialization, no parsing, and no cross-boundary data transfer.
User-space only reads the BPF maps when it needs to make a policy decision, and at that point it reads aggregated results (hundreds of entries), not raw samples (potentially millions).
\begin{figure}[!t]
\centering
% TODO: Side-by-side diagram showing:
% Left: User-space path: PMU → kernel perf → ring buffer → (context switch) →
% user thread → deserialize → hash table update → (context switch back)
% Overhead labels: multiple data copies, context switches, cache pollution,
% TLB pressure, per-sample parsing
% Right: eBPF path: PMU → kernel perf → eBPF program (NMI context) → BPF map
% (no context switch, no ring buffer, single atomic update)
% Bottom: comparison box: N raw samples crossing boundary (user-space) vs.
% M aggregated entries read on demand (eBPF), N >> M.
\begin{comment}
\fbox{\parbox{0.9\columnwidth}{\centering\vspace{3cm}
\textbf{Figure placeholder:} Data paths for memory profiling.
\emph{Left}: User-space approach---every raw sample traverses the ring buffer and crosses the kernel--user boundary, incurring context switches, multiple data copies, cache pollution, and per-sample parsing overhead.
\emph{Right}: \name's eBPF approach---samples are aggregated in-kernel via a single atomic map update; user space reads only the aggregated results on demand.
\vspace{3cm}}}
\end{comment}
\includegraphics[width=1\columnwidth]{./figure/profiling_path.pdf}
\caption{Comparison of profiling data paths.}
\label{fig:ebpf-vs-userspace}
\end{figure}
%%%user-space and eBPF-based
%The user-space path fetches $N$ raw samples across the kernel--user boundary with per-sample parsing and data copying; the eBPF path aggregates $N$ samples into $M$ map entries in-kernel ($M \ll N$), and the user space reads only $M$ entries when making policy decisions.
% Figure~\ref{fig:ebpf-vs-userspace} contrasts the two data paths. The key difference is the \emph{amount of data that crosses the kernel--user boundary}. The user-space path forces all $N$ raw samples to traverse the ring buffer, be parsed one by one, and update a user-space hash table---incurring context switches, data copies, cache pollution, and TLB pressure along the way. The eBPF path aggregates $N$ samples into $M$ map entries entirely within the kernel, and only the $M$ aggregated entries are read by user space when needed ($M \ll N$). At a sampling rate of $10^5$ samples/sec with $10^3$ distinct pages, the data reduction ratio $N/M$ reaches two orders of magnitude.
\subsubsection{R/W Classification in Migration Admission}
\label{subsubsec:rw-class}
\textcolor{checked}{For duplex-aware migration admission (\S\ref{subsec:admission}), \name must classify each page's read/write ratio. This is inherently a \emph{per-page} query that the global histogram cannot serve. Explicitly maintaining per-page counters is not scalable (\S\ref{sec:background}). %would scale with the working set.
\textcolor{checked}{To address this problem, \name introduces a \emph{blocked counting Bloom filter} (bCBF) inspired by HybridTier~\cite{10.1145/3676642.3736119}. A counting Bloom filter (CBF) provides space-efficient approximate count estimation; a blocked CBF further improves cache efficiency by partitioning counters into cache-line-sized blocks, so each lookup touches exactly one cache line.} Concretely, a bCBF maps each page address via hashing to $k$ counters within a single block. A \textsc{Get} returns the minimum of the $k$ counters; an \textsc{Update} increments only the counters at the current minimum~\cite{10.1145/3676642.3736119}.}
\textcolor{checked}{\name extends this mechanism with a \emph{dual}-bCBF design, one for reads and one for writes. The write ratio of a page is computed as $w_{\min} / (w_{\min} + r_{\min})$, where $w_{\min}$ and $r_{\min}$ are the \textsc{Get} results from the write and read bCBFs respectively.}
\name allocates the dual bCBF only when CXL memory is present and duplex-aware admission is active.
On PMEM or single-tier systems, \name does not need this counting mechanism, \textcolor{checked}{because the half-duplex interface does not benefit from read/write traffic rebalancing (\S\ref{subsec:mot-arch}).}
\subsection{mTHP Splitting}
\label{subsec:splitting}
Given the subpage histogram from \S\ref{subsec:profiling}, \name must decide what mTHP size to split into, which subfolios are hot, and which subfolios to migrate. \textcolor{checked}{We do not change the Linux kernel's inherent page merging mechanism.}
%Splitting decision algorithm
\subsubsection{THP Splitting Decision}
\label{subsubsec:split-size}
When slow-tier bandwidth is contended and \textcolor{checked}{THP splitting is activated}, a background user-space analyzer reads the global subpage histogram for each application and computes the target mTHP size. Let $c_i$ denote the sampled access count of the $i$-th 4\,KB subpage within a 2\,MB THP region ($i = 0, 1, \ldots, 511$), and let $C_{\text{total}} = \sum_{i=0}^{511} c_i$.
The algorithm to split THP has four stages.
\textbf{Stage 1: hot threshold determination.}
This stage separates hot subpages from cold ones. \name sorts subpages by their access counts %in descending order
($\tilde{c}_0 \geq \tilde{c}_1 \geq \cdots \geq \tilde{c}_{511}$) and finds the smallest $K^*$ so that the cumulative sum of top-$K^*$ subpages reaches a target coverage ratio $P$ (default 80\%):
\begin{equation}
\sum_{j=0}^{K^*} \tilde{c}_j \;\geq\; P \cdot C_{\text{total}}
\label{eq:hot-threshold}
\end{equation}
The hot threshold is then $T = \tilde{c}_{K^*}$.
%This coverage-based threshold adapts automatically to different degrees of access skew.
This coverage-based threshold adapts automatically to workload skew and varying access distributions, ensuring that the most frequently accessed subpages are consistently classified as hot.
\textbf{Stage 2: hot subfolio classification.}
Each subpage is classified using the threshold $T$:
\begin{equation}
h_i =
\begin{cases}
1 \;(\text{hot}), & \text{if } c_i \geq T \\
0 \;(\text{cold}), & \text{if } c_i < T
\end{cases}
\label{eq:hot-cold}
\end{equation}
producing a binary hot/cold map for 512 subpages in a THP.
\textbf{Stage 3: mTHP size selection via heat density.}
The goal of this stage is to select the largest mTHP size that still tightly captures a hot region. For each candidate size (2\,MB, 1\,MB, 512\,KB, 256\,KB, 128\,KB, 64\,KB), the THP is partitioned into aligned subfolios of $L$ contiguous subpages, where $L$ equals the candidate size divided by 4\,KB (e.g., $L = 128$ for a 512\,KB subfolio). The heat density of a subfolio is defined as the fraction of its subpages classified as hot:
\begin{equation}
\text{heat\_density}([i,\, i{+}L)) = \sum_{j=i}^{i+L-1} h_j \;/\; L
\end{equation}
\name selects the largest mTHP size for which at least one aligned subfolio achieves a heat density of at least $\tau_h$. \textcolor{checked}{A lower $\tau_h$ would admit subfolios with excessive cold data, wasting fast-tier capacity; a higher $\tau_h$ would force unnecessarily small splits, sacrificing TLB coverage. By default $\tau_h = 75\%$, keeping selected subfolios predominantly hot while still favoring larger mTHP sizes for less TLB pressure.} %%Because the splitting logic runs as an eBPF program, $\tau_h$ can be tuned per workload without kernel changes. \asaf{I don't know if I'd necessarily tout this -- there are many configurable knobs in the kernel as well that don't require you to make kernel changes}}
\begin{comment}
The goal is to find the largest mTHP size that can tightly capture a hot region.
For each candidate size (2\,MB, 1\,MB, 512\,KB, 256\,KB, 128\,KB, 64\,KB), the THP is divided into aligned subfolios of $L$ contiguous subpages, where $L$ equals the candidate size divided by 4\,KB (e.g., $L = 128$ for 512\,KB).
The \emph{heat density} of each subfolio is defined as the fraction of hot subpages within it:
\begin{equation}
\text{heat\_density}([i,\, i{+}L)) = \sum_{j=i}^{i+L-1} h_j \;/\; L
\label{eq:heat-density}
\end{equation}
\name picks the \textbf{largest size for which any aligned subfolio has heat density $\geq$ 75\%}.
This maximizes TLB coverage, because a larger mTHP requires fewer TLB entries for the migrated region, while ensuring that migrated subfolios are genuinely hot.
\end{comment}
\textbf{Stage 4: migration target selection.}
%After splitting, not all subfolios are migrated. A subfolio is added to the migration list if it satisfies either of two conditions: (1)~it is classified as \emph{hot} by heat density, or (2)~it contains the \emph{NUMA-faulted address} that triggered the migration. The NUMA fault indicates a recent locality mismatch that the hardware sampler may not yet have captured. In this way, \name selects subfolios to be migrated based on both \emph{frequency-based signals} (from the subpage histogram) and \emph{recency-based signals} (from the NUMA fault). Cold subfolios that do not contain the faulting address remain in slow memory.
After splitting, \name does not migrate all subfolios. A subfolio is added to the migration list if it satisfies either of two conditions: (1) it is classified as hot based on heat density, or
(2) it contains the NUMA‑faulted address that triggered the migration. The NUMA fault reflects a recent locality mismatch that the hardware event sampler may not yet have observed. By combining these two signals, \name selects subfolios using both frequency-based information (from the subpage histogram) and recency-based information (from the NUMA fault). Subfolios that are cold and do not contain the faulting address remain in slow memory.
\textbf{Flat distribution detection.}
%Before executing the stages above, \name checks if the access distribution is nearly uniform by computing the normalized Shannon entropy ~\cite{}
Before executing the stages above, \name first checks whether the access distribution is nearly uniform using normalized Shannon entropy ($H_{\text{norm}}$) ~\cite{pielou1966measurement}, \textcolor{checked}{an information-theoretic measure that quantifies the spread of a probability distribution as a single scalar in $[0, 1]$. Applied here, it captures how evenly accesses are distributed across the 512 subpages. 0 means all accesses hit one subpage, 1 means perfectly uniform}:
\begin{equation}
H_{\text{norm}} = \bigl({-\sum_{i} p_i \log_2 p_i}\bigr) \;/\; {\log_2 512}, \quad p_i = c_i / C_{\text{total}}
\label{eq:entropy}
\end{equation}
If $H_{\text{norm}} \geq 0.95$, the distribution is essentially uniform---there is no hot region to isolate, so splitting provides no benefit.
In this case, \name falls back to the default NUMA fault behavior: the entire THP is migrated without splitting, following the recency-based signal from the fault.
%\asaf{Someone may ask about merging? We only discussing splitting..}
\begin{table}[!t]
\centering
\small
\caption{Memory duplex mode-aware migration admission policy on the CXL memory. R=read and W=write.}
\label{tab:admission-policy}
\begin{tabular}{lccc}
\toprule
& \textbf{R-dominant} & \textbf{Balanced} & \textbf{W-dominant} \\
& \textbf{CXL traffic} & \textbf{CXL traffic} & \textbf{CXL traffic} \\
\midrule
R-heavy page & Allow & Allow & Prevent \\
W-heavy page & Prevent & Allow & Allow \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Kernel-Side Enforcement}
\label{subsubsec:split-migrate}
When a NUMA fault triggers migration of a THP and splitting is enabled, the kernel calls the first eBPF hook (\texttt{bpf\_thp\_pick\_split\_order}) to obtain the target mTHP splitting size.
It then splits the THP into subfolios of that order and calls the second hook (\texttt{bpf\_subpage\_is\_cold}) for each subfolio to detect hot subfolios.
Hot subfolios and the faulting subfolio are added to the migration list; cold subfolios are left in place.
\subsection{Duplex-Aware Migration Admission Control}
\label{subsec:admission}
The splitting pipeline above determines how to migrate (sub)pages. %The admission control
We also determine \emph{whether} to migrate or not.
A user-space daemon monitors the CXL read/write traffic by hardware event counters and stores a sliding window of measurements in a BPF map. On each NUMA fault, the admission eBPF hook (\texttt{bpf\_thp\_numa\_ migrate\_admission}) classifies the faulting page as read-heavy or write-heavy using the dual bCBF (\S\ref{subsubsec:rw-class}), and checks the current CXL traffic pattern. The admission control policy is summarized in Table~\ref{tab:admission-policy}: \name migrates pages whose type matches the dominant traffic, and holds back the rest. The admission control runs before the splitting stage: the eBPF hook decides whether to admit the page for migration and only admitted pages proceed to the splitting pipeline.
%The migration decision depends on both the page's read/write classification and overall CXL traffic pattern.
%\section{Implementation}
%\asaf{Do you want to say how many LOCs you had to change etc? In general this section doesn't provide much value, you could consider just integrating it into the previous section and/or removing some of the details}
%%%%\textbf{eBPF hooks.} \name adds three hooks to the kernel’s NUMA‑migration path (summarized in Table \ref{tab:hooks}). All three follow the same design pattern: a \texttt{noinline} kernel function annotated with \texttt{ALLOW\_ERROR\_INJECTION} that returns a safe default when no eBPF program is attached, but whose return value can be overridden by a \texttt{fmod\_ret} BPF program.
%Each hook returns a safe default when no eBPF program is loaded, preserving standard kernel behavior.
\begin{comment}
\textbf{Time decay.}
To prevent stale access data from dominating decisions, each histogram entry stores both a count and an epoch number. When the entry is next accessed, its count is right-shifted by the number of elapsed epochs (each epoch $T$ = 10\,s):
\begin{equation}
\texttt{count} \mathrel{>\!\!>=} \min \!\bigl((\texttt{now} / T) - \texttt{epoch},\; 10\bigr)
\label{eq:decay}
\end{equation}
We apply the same time‑decay mechanism to the dual bCPF on a periodic basis. This exponential decay ensures that memory‑access profiling reflects recent behavior, allowing \name to adapt quickly to phase changes in the workload. %behavior.
\end{comment}
%\textbf{Hardware event sampling backend.} \name collects hardware event samples using AMD Instruction-Based Sampling (IBS) or Intel Processor Event-Based Sampling (PEBS), depending on the platform. The eBPF profiling program attaches to perf events and processes each sample in an interrupt context, updating the subpage histogram and the bCBFs (if enabled). No sample data leaves the kernel during profiling; user-space daemons only read the aggregated BPF maps periodically to compute the policy decisions. The sampling rate is configurable; we use 4{,}000 samples/sec per CPU in our evaluation (\S\ref{sec:eva}).
\section{Evaluation}
\label{sec:eva}
\textcolor{checked}{\name comprises $\sim$6,800 lines of user-space code (eBPF programs, loaders, daemons, and analyzers) and 540 lines of kernel modifications to Linux v6.12 for eBPF hooks and page splitting. Since neither TPP nor Colloid support mTHP, we re-implemented both on Linux v6.12, requiring $\sim$270 and 1,160 lines of kernel changes, respectively.}
\subsection{Evaluation Methodology}
\label{sec:eval_method}
\textbf{Evaluation platforms.}
To evaluate \name, we use two servers with different memory architectures.
\begin{denseitemize}%[leftmargin=*,noitemsep,topsep=0pt]
\item A server with CXL memory expansion with dual-socket AMD EPYC 9555s (64 cores per socket). Each socket has 8$\times$32\,GB DDR5 DRAM as local memory. Socket~0 connects to a CXL Type-3 Memory Expansion device (128\,GB DDR4) over PCIe~5.0, providing a full-duplex link.
\item A server with Intel Optane PMEM with dual-socket Intel Xeon Gold 6252 (24 cores per socket). Each socket has 6$\times$16\,GB DDR4 DRAM as the fast tier and 6$\times$128\,GB Intel Optane PMEM as the slow tier. %The PMEM is attached via DDR channels that are half-duplex.
\end{denseitemize}
By default, we use the CXL server. To isolate the impact of page migration from cross-socket NUMA effects, we use Socket~0 only, following prior efforts in~\cite{lee2023memtis, maruf2023tpp, atc24_hm}.
\textcolor{checked}{For hardware event sampling, the sampling rate is 4{,}000 samples/sec per CPU.}
\textbf{System integration.} To evaluate the portability of \name, we integrate \name into three kernel-level memory tiering systems: AutoNUMA, TPP, and Colloid. \textcolor{checked}{For AutoNUMA, we use the vanilla Linux kernel v6.12. For TPP, we change AutoNUMA's hotness detection from NUMA hint fault latency to an LRU active-list check and add TPP's asynchronous background demotion mechanism~\cite{maruf2023tpp}. For Colloid, we use its TPP-based implementation~\cite{vuppalapati2024colloid} and replace Intel CHA-based memory tier latency monitoring with AMD Data Fabric performance counters.}
For each system, we set
\path{/proc/sys/kernel/numa_balan} \path{cing}=2 to enable promotion and
\path{/sys/kernel/mm/numa/demotion_enabled}=1 to enable demotion.
For THP, under \path{/sys/kernel/mm}, we set
(i)~\path{/transparent_hugepage/ena} \path{bled}=always,
(ii)~\path{/transparent_hugepage/defrag}=always, and
(iii)~\path{/transparent_hugepage/hugepages-2048kB/ena} \path{bled}=inherit.
\textbf{Baselines.}
On the CXL server, we integrate \name into AutoNUMA, TPP, and Colloid, and compare against each system's unmodified baseline.
On the PMEM-enabled server, we additionally compare \name-enhanced AutoNUMA against MEMTIS~\cite{lee2023memtis}, the state-of-the-art system for huge-page-aware page migration. We cannot evaluate MEMTIS on the CXL platform because MEMTIS relies on Intel PEBS events to distinguish memory access sources (i.e., local DRAM or remote DRAM) at the hardware level. AMD IBS, used on our CXL server, does not provide equivalent hardware performance counters.
%{hardware-level differentiation between local DRAM and CXL-attached memory accesses.}
\input table/benchmarks
\textbf{Workloads.} We evaluate \name on 17 memory-intensive benchmarks, summarized in Table~\ref{tab:benchmarks}.
The benchmarks span three categories:
(1)~12 graph-processing workloads from the GAP Benchmark Suite (GAPBS) ~\cite{gap}, consisting of four algorithms (betweenness centrality, breadth-first search, connected components, and PageRank) run on three input graphs (Twitter, uniform random, and Web);
(2)~an in-memory database engine (Silo-YCSB ~\cite{silo}); and
(3)~four high-performance computing benchmarks (LU, SP, CG, and MG) from the NAS Parallel Benchmarks (NPB)~\cite{nas}. These workloads have been used extensively in related work~\cite{10.1145/3712285.3759816,lee2023memtis,10.1145/3477132.3483550,maruf2023tpp,eurosys24:mtm,osdi25_aol,vuppalapati2024colloid}. All GAPBS and NPB benchmarks run with 12 threads; Silo-YCSB runs with 8 threads since \textcolor{checked}{it crashes with 12 threads}. All experiments use cgroup-based resource isolation; each benchmark runs within a dedicated cgroup with controlled CPU and memory limits to ensure reproducibility.
\textbf{Memory contention.} To emulate the bandwidth pressure commonly observed in virtualized environments, we run a memory stressor on the slow tier alongside benchmarks. The stressor spawns antagonist threads that issue sequential, non‑temporal memory accesses using \texttt{movntdqa}/\texttt{movntdq} instructions over a buffer allocated and pinned in the slow tier. By bypassing the CPU cache hierarchy, these accesses maximize memory bus utilization. The antagonist threads are pinned to dedicated CPU cores separate from the benchmark to avoid direct CPU contention. We vary the number of antagonist threads to generate sustained background traffic of approximately 0, 5, 10, and 20\,GB/s.
%\textcolor{sherry}{To emulate bandwidth pressure often encountered in virtualized environments, we run a memory stress tool on the slow memory alongside the benchmark. The tool spawns antagonist threads that perform sequential, non-temporal memory accesses (using SSE4.1 \texttt{movntdqa}/\texttt{movntdq} instructions) through a buffer allocated and pinned on the slow memory, bypassing the CPU cache hierarchy to maximize memory bus utilization. The antagonist threads are pinned to dedicated CPU cores separate from the benchmark. We vary the number of antagonist threads to produce sustained background traffic of approximately 0, 5, 10, and 20\,GB/s.}
\begin{comment}
\textit{Intel Optane Persistent Memory (PMEM).} Optane PMEM connects via the DDR interface, which is \emph{half-duplex}: reads and writes share the same channel and cannot proceed simultaneously.
Under mixed read/write traffic, effective bandwidth degrades because reads and writes must be serialized on the bus~\cite{wang2025cxl_performance}.
\textit{CXL-attached Memory.} Compute Express Link (CXL) provides a cache-coherent interconnect for attaching external memory. CXL operates in \emph{full-duplex} mode: reads and writes use separate channels and can proceed concurrently. This means CXL memory can sustain higher effective bandwidth under mixed traffic compared to Optane's half-duplex DDR interface~\cite{wang2025cxl_performance}.
This architectural difference has direct implications for page migration. On a full-duplex CXL link, migrating a read- or write-heavy page can generate balanced read/write traffic, further increasing bandwidth utilization for co-located applications. On a half-duplex Optane link, the same migration has far less effect. An effective memory tiering system must account for these differences---yet existing systems use a single, architecture-agnostic migration policy regardless of the underlying hardware.
\end{comment}
\subsection{Evaluation of THP Splitting}
\label{sec:eval_cxl_splitting}
\begin{figure}[!t]
\centering
% \includegraphics[width=1\columnwidth]{./figure/cxl_thp_splitting_comparison.pdf}
\includegraphics[width=1\columnwidth]{./figure/cxl_thp_splitting_comparison_0401-v6-arrow.pdf}
\caption{Performance of \name (mTHP splitting) integrated into three tiering systems on the CXL platform, normalized to the performance of vanilla tiering systems.}
\label{fig:cxl_splitting}
\end{figure}
Figure~\ref{fig:cxl_splitting} shows the end-to-end performance of \name integrated into AutoNUMA, TPP, and Colloid on the CXL-based server. All results are normalized to each system's unmodified baseline (i.e., 2\,MB THP without splitting). Across all 17 workloads, \name improves the geomean throughput of AutoNUMA by 12.3\%, TPP by 17.7\%, and Colloid by 7.4\%.
\subsubsection{Analysis by Workload}
Graph workloads benefit the most. For example, on AutoNUMA, bc-twitter improves by 30\%, cc-web by 24\%, cc-twitter by 23\%, and bfs-web by 21\%. These workloads exhibit highly skewed intra-THP access: only a fraction of subpages within each 2\,MB THP are frequently accessed. By splitting THPs and migrating only the hot subfolios, \name avoids wasting scarce fast-tier capacity on cold data. The effect is strongest on Twitter and Web graphs, whose power-law degree distributions concentrate accesses on a small set of high-degree vertices. The uniform random graphs (bc-urand, bfs-urand, etc.) show smaller but consistent gains, demonstrating that even they develop access hotspots when using iterative algorithms.
Silo consistently improves across all three systems (13\% on AutoNUMA, 19\% on TPP, and 20\% on Colloid). This improvement stems from YCSB's Zipfian access distribution, which induces well-defined hot and cold regions within THPs. With this highly concurrent transaction workload, frequent NUMA faults arise and trigger THP splitting.
%\textcolor{dong}{This is because the YCSB's Zipfian access distribution in Silo} creates clear hot and cold regions within THPs, and the database's concurrent transactions generate frequent NUMA faults that trigger splitting.
\textcolor{checked}{NPB workloads show consistent improvements on AutoNUMA (up to 9\%),
%(5--9\% across all four benchmarks),
and larger gains on TPP for CG (29\%) and SP (18\%). These array-based scientific kernels exhibit intra-THP access skew due to indirect indexing (CG), strided multi-dimensional array traversals (SP), and non-uniform access frequency across data grids (MG).}
\subsubsection{Analysis by Memory Tiering Software}
%TPP benefits the most among the three memory tiering systems (15.8\% geomean). Unlike AutoNUMA using hint fault latency (i.e., the elapsed time between unmapping a page and the subsequent access fault) to determine page hotness, TPP uses an LRU-based check: a page is considered hot if it resides on the kernel's active LRU list~\cite{maruf2023tpp}. For a 2\,MB THP containing 512 subpages, any single subpage access to a folio causes the entire folio as referenced, and puts it to the active list. As a result, TPP's LRU check is effectively disabled for 2\,MB folios---nearly every THP passes regardless of how much of it is actually hot. This causes TPP to over-promote cold data into the fast tier, filling it with folios that are only partially utilized. Splitting restores the effectiveness of the LRU filter: a 64\,KB subfolio has only 16 subpages, so cold subfolios are far more likely to age into the inactive list and be correctly identified as cold. This explains why TPP sees the largest improvement from \name.
TPP benefits the most among the three memory tiering systems, with a 17.7\% geometric-mean improvement. Unlike AutoNUMA, which infers page hotness from hint fault latency (i.e., the time between unmapping a page and the subsequent access fault), TPP relies on an LRU-based heuristic: a page is deemed hot if it resides on the kernel’s active LRU list \cite{maruf2023tpp}. However, for a 2\,MB THP comprising 512 subpages, a single access to any subpage causes the entire folio to be referenced and promoted to the active list. Consequently, TPP’s LRU check is rendered ineffective at THP granularity—nearly all THPs are set to be promoted regardless of how little of the folio is actually accessed. This behavior leads TPP to over-promote cold data into the fast tier, populating it with folios that are only sparsely utilized. Splitting restores the discriminative power of the LRU filter. A 64\,KB subfolio, for example, contains only 16 subpages, allowing cold subfolios to age into the inactive list and be correctly classified as cold. As a result, TPP experiences the largest performance gains. %from \name.
%Colloid shows the smallest improvement (5.1\% geomean) because its access-latency-balanced promotion policy already selects pages more judiciously. Nonetheless, \name still provides meaningful gains: silo\_ycsb improves by 20\%, bc-web by 20\%, and cc-web by 18\%. Even a well-tuned page selection policy cannot avoid wasting bandwidth when the migration granularity forces entire 2\,MB pages to move.
Colloid shows the smallest improvement (7.4\% geometric mean) because its access-latency–balanced promotion policy already selects pages more judiciously. Nevertheless, \name still yields meaningful gains: silo\_ycsb improves by 20\%, bc-web by 20\%, and cc-web by 18\%. These results highlight that even a well-tuned page selection policy cannot fully avoid bandwidth waste when migration is constrained to coarse-grained 2\,MB pages—moving an entire page incurs unnecessary data movement when only a fraction is hot.
\input{table/table_split_distribution_selected}
\subsubsection{How Does \name Split Pages?}
%Table~\ref{tab:split_dist_selected} reports the split-size distribution and post-split hot subfolio fraction for the four workloads with the largest improvement on AutoNUMA. Two observations stand out. First, \emph{no single mTHP size fits all workloads}. For example, bc-twitter spreads its splits across four mTHP sizes (64\,KB--512\,KB), because different THPs contain hot regions of different extents. This confirms that a static, one-size-fits-all splitting granularity would leave performance on the table; \name's per-THP size selection is essential. Second, \emph{hot subfolios are the minority}: across all four workloads the hot subfolio fraction ranges from 9.1\% (cc-twitter) to 53.2\% (bfs-web). Without splitting, the kernel would promote the entire 2\,MB THP, wasting up to 91\% of migration bandwidth and fast-tier capacity on cold data.
Table~\ref{tab:split_dist_selected} reports the split-size distribution and the fraction of hot subfolios after splitting for four workloads that benefit most from AutoNUMA. Two key observations emerge.
First, no single mTHP size fits all workloads. For instance, bc-twitter distributes its splits across four mTHP sizes (64\,KB -- 512\,KB), reflecting that the THP's hot region extent varies across execution phases. This underscores that a static, one-size-fits-all splitting granularity would leave substantial performance untapped; the dynamic size selection is essential.
%in \name is therefore essential.
Second, hot subfolios are typically a minority. Across the workloads, the fraction of hot subfolios ranges from just 9.1\% (cc-twitter) to 53.2\% (bfs-web). Without splitting, the kernel would promote entire 2\,MB THPs, wasting up to 91\% of migration bandwidth and fast-tier capacity on cold data.
\subsection{Why does Splitting Help? Further Analysis from the Perspective of Page Migration}
\label{sec:eval_deep}
\input{table/table_split_vs_nosplit}
%To understand \emph{why} mTHP splitting improves performance, we examine the page migration statistics collected via kernel vmstat counters on AutoNUMA. Table~\ref{tab:split_vs_nosplit} compares migration behavior with and without splitting for the 12 GAP workloads on AutoNUMA.
To understand why mTHP splitting improves performance, we analyze page migration statistics collected from kernel vmstat counters on AutoNUMA. Table~\ref{tab:split_vs_nosplit} compares migration with and without splitting across the 12 GAP workloads.
%The most striking change is in \textbf{migration failures}. Without splitting, the kernel attempts to migrate entire 2\,MB THPs. When the destination node lacks a contiguous 2\,MB free region, the migration fails. For bc-twitter, 18 million migrations fail without splitting; with splitting, only 80 thousand do---a 224$\times$ reduction. Similar reductions appear across all workloads: bc-urand drops from 8.5\,M to 11\,K (777$\times$), bfs-web from 2.7\,M to 83\,K (32$\times$), and cc-twitter from 7.7\,M to 694\,K (11$\times$).
The most pronounced change is the reduction in migration failures. Without splitting, the kernel attempts to migrate entire 2\,MB THPs; when the destination node cannot provide a contiguous 2\,MB free region, the migration fails. For bc-twitter, 18 million migration attempts fail without splitting, whereas with splitting only 80 thousand fail—a 224$\times$ reduction. Similar trends hold across all workloads: bc-urand drops from 8.5\,M to 11\,K failures (777$\times$), bfs-web from 2.7\,M to 83\,K (32$\times$), and cc-twitter from 7.7\,M to 694\,K (11$\times$).
%The mechanism is straightforward. After splitting a 2\,MB THP into, say, 64\,KB mTHP subfolios, each migration attempt requires only 64\,KB of contiguous free space on the destination node---a dramatically easier allocation to satisfy.
The underlying reason is clear. Splitting a 2\,MB THP into, for example, 64\,KB mTHP subfolios reduces migration’s allocation requirement to just 64\,KB of contiguous free space on the destination node—an order of magnitude easier constraint to satisfy. As a result, far fewer migrations fail, allowing AutoNUMA to effectively relocate hot data.
%This reduction in failures translates directly into \textbf{more successful migrations}. For bc-twitter, successful migrations nearly double from 1.26\,M to 2.44\,M (1.9$\times$). For cc-web, they grow from 1.94\,M to 4.28\,M (2.2$\times$), and for cc-twitter from 1.24\,M to 3.80\,M (3.1$\times$). As a result, more hot data is placed in fast memory, while cold subpages remain on the slow tier. Across all 12 workloads, splitting increases successful migrations by 1.0--3.1$\times$ while reducing failures by up to 777$\times$.
This sharp reduction in failures translates directly into more successful migrations. For bc-twitter, the number of successful migrations nearly doubles, increasing from 1.26\,M to 2.44\,M (1.9$\times$). For cc-web, successful migrations grow from 1.94\,M to 4.28\,M (2.2$\times$), and for cc-twitter from 1.24\,M to 3.80\,M (3.1$\times$). As a result, a larger fraction of hot data is successfully placed in the fast tier, while cold subpages remain in the slow tier. Across all 12 workloads, splitting increases the number of successful migrations by 1.0$\times$–3.1$\times$ while reducing migration failures by up to 777$\times$.
%In summary, splitting works through two reinforcing effects: it \emph{reduces the bar for migration success} (smaller allocations succeed more often) and it \emph{increases the precision of what migrates} (only hot subfolios move).
In summary, splitting improves performance through two reinforcing effects: it lowers the bar for migration success by reducing allocation size requirements, and it improves migration precision by ensuring that only hot subfolios are moved. Together, these effects enable AutoNUMA to migrate hot data more reliably and more selectively.
\subsection{Evaluation with Memory Bandwidth Contention}
\label{sec:eval_contention}
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{./figure/cxl_mem_contention_comparison_arrow.pdf}
\caption{Effects of CXL memory bandwidth contention on splitting benefit (AutoNUMA). The performance is normalized to that of the baselines without THP splitting.}
\label{fig:contention}
\end{figure}
%Each panel increases the background memory traffic from 0 to 20\,GB/s. Splitting benefit grows with contention.
%\name activates mTHP splitting only when slow-tier bandwidth is contended (\S\ref{subsec:splitting}). To validate this design and justify the contention level used in our main evaluation (\S\ref{sec:eval_cxl_splitting}), we run the 12 GAP workloads on AutoNUMA under four levels of CXL memory bandwidth contention: 0, 5, 10, and 20\,GB/s of sustained background traffic generated by a memory-streaming process on the CXL device.
\name activates THP splitting only when slow-tier bandwidth becomes contended (\S\ref{subsec:splitting}). To validate this design choice and motivate the contention level used in our main evaluation (\S\ref{sec:eval_cxl_splitting}), we run the 12 GAP workloads on AutoNUMA under four levels of CXL memory bandwidth pressure: 0, 5, 10, and 20\,GB/s of sustained background traffic, generated by a memory-streaming process on the CXL device.
%Figure~\ref{fig:contention} shows the results. Under \textbf{no contention} (0\,GB/s), splitting provides little benefit. The geomean improvement is only 3.4\%, and only 7 of 12 workloads improve. When bandwidth is plentiful, migrating cold subpages alongside hot ones within a 2\,MB THP has negligible cost---the extra bytes do not displace useful traffic. The TLB penalty of splitting often offsets the migration savings.
Figure~\ref{fig:contention} presents the results. Under no contention (\textbf{0\,GB/s}), splitting provides little benefit: the geometric-mean improvement is only 3.4\%, and just 7 of the 12 workloads see speedups. When bandwidth is abundant, migrating cold subpages alongside hot ones within a 2\,MB THP incurs negligible cost—the additional data does not displace useful traffic. In this regime, the increased TLB pressure introduced by splitting often offsets any savings in migration bandwidth.
%At \textbf{5\,GB/s}, splitting still does not help: the geomean is near zero ($-$0.1\%), and only 6 of 12 workloads improve. At this contention level, the bandwidth savings from splitting are not yet large enough to consistently offset the TLB overhead.
At \textbf{5\,GB/s} \textcolor{checked}{of memory contention}, splitting still fails to consistently pay off. The geometric-mean improvement is effectively zero (-0.1\%), and only 6 of the 12 workloads improve. At this moderate contention level, the bandwidth savings from less cold-data migration are not yet sufficient to reliably outweigh TLB overhead introduced by smaller page sizes.
%At \textbf{10\,GB/s}, splitting starts to show benefit, but the gains are uneven. The geomean rises to 7.0\%, yet only 9 of 12 workloads improve, and most of them see single-digit gains. The improvement is concentrated on workloads with highly skewed access patterns (e.g., bc-twitter at 18\%, bc-web at 20\%, cc-web at 31\%), while workloads with more uniform access remain flat or regress. At this contention level, splitting is not yet broadly effective.
At \textbf{10\,GB/s} \textcolor{checked}{of memory contention}, splitting begins to provide measurable benefits, but the gains remain uneven. The geometric-mean improvement rises to 7.0\%; however, only 9 of the 12 workloads improve, and most see only single‑digit speedups. The benefits are concentrated among workloads with highly skewed access patterns—for example, bc-twitter (18\%), bc-web (20\%), and cc-web (31\%)—while workloads with more uniform access patterns remain flat or even regress. At this level of contention, the bandwidth savings enabled by splitting are still insufficient to make it broadly effective.
%At \textbf{20\,GB/s}---the highest contention level---splitting delivers clear, consistent improvement. The geomean reaches 12.8\%, and 11 of 12 workloads improve, with bc-twitter at 30\%, cc-web at 24\%, cc-twitter at 24\%, and bfs-web at 21\%. Migrating a 2\,MB page that is only 20\% hot wastes 1.6\,MB of scarce bandwidth; splitting eliminates this waste, and the bandwidth savings decisively outweigh the TLB penalty. This is also the contention level used in our end-to-end evaluation (\S\ref{sec:eval_cxl_splitting}), because it represents a realistic scenario in which multiple memory-intensive workloads share a CXL device and bandwidth becomes the primary performance limiter.
At \textbf{20\,GB/s} of memory contention---the highest contention level---splitting delivers clear and consistent improvements. The geometric mean increases to 14.1\%, and all 12 workloads improve, including bc-twitter (30\%), cc-web (24\%), cc-twitter (23\%), and bfs-web (21\%). Under this level of pressure, migrating a 2\,MB page that is only 20\% hot wastes 1.6\,MB of scarce bandwidth; splitting eliminates this waste, and the resulting bandwidth savings decisively outweigh the TLB penalty. We therefore use this contention level in our end‑to‑end evaluation (\S\ref{sec:eval_cxl_splitting}), as it reflects a realistic scenario in which multiple memory‑intensive workloads share a CXL device and bandwidth becomes the primary performance bottleneck.
%These results validate \name's contention-aware activation: under low contention, \name keeps THPs at 2\,MB to preserve TLB coverage; as contention rises and bandwidth savings grow to outweigh TLB cost, \name activates splitting.
Together, these results validate \name’s contention‑aware activation policy. Under low contention, \name preserves 2\,MB THPs to maximize TLB coverage. As contention increases and bandwidth efficiency becomes more critical, \name selectively enables splitting, activating it only when its bandwidth savings outweigh the associated TLB costs.
\begin{figure}[!t]
\centering
% \includegraphics[width=\columnwidth]{./figure/intel_thp_splitting_comparison.pdf}
\includegraphics[width=\columnwidth]{./figure/intel_thp_splitting_comparison_0401-v4d_arrow.pdf}
\caption{Performance of \name \textcolor{checked}{and MEMTIS} on the PMEM platform, normalized to \textcolor{checked}{that of AutoNUMA.}}
\label{fig:intel_splitting}
\end{figure}
\input{table/table_tlb_miss}
%Generalization to Intel Optane PMEM
\subsection{Evaluation on PMEM}
\label{sec:eval_optane}
%Results compare AutoNUMA baseline, \name-enhanced AutoNUMA, and Memtis.
To demonstrate that \name's design generalizes beyond CXL, we evaluate mTHP splitting on the PMEM platform. \textcolor{checked}{We only enable the THP splitting policy, because maintaining mixed read and write traffic does not benefit performance given PMEM's half-duplex DDR interface.}
%Figure~\ref{fig:intel_splitting} shows the results. \name-enhanced AutoNUMA improves over the AutoNUMA baseline by 7.1\% in geomean across 17 workloads. The largest gains appear on graph workloads with skewed access: bc-twitter improves by 36\%, pr-twitter by 26\%, cc-twitter by 14\%, bfs-twitter by 11\%, and cc-web by 10\%. The benefit pattern on Optane mirrors the CXL results, confirming that THP splitting addresses a fundamental property of THP-based migration, that is the mismatch between a 2\,MB migration granularity and sub-page access skew.
Figure~\ref{fig:intel_splitting} presents the results. FlexTier-enhanced AutoNUMA outperforms the AutoNUMA baseline by 7.1\% in geometric mean across 15 workloads. The largest gains occur for the graph workloads with highly skewed access patterns: bc-twitter improves by 36\%, pr-twitter by 26\%, cc-twitter by 14\%, bfs-twitter by 11\%, and cc-web by 10\%. The performance trends on PMEM closely mirror those observed with CXL, confirming that THP splitting addresses a fundamental limitation of THP-based migration---namely, the mismatch between the coarse 2\,MB migration granularity and fine-grained, sub‑page access skew.
\textbf{Comparison with MEMTIS.}
%We also compare against MEMTIS~\cite{lee2023memtis}, the state-of-the-art system for huge-page-aware tiered memory. MEMTIS uses a background thread to split THPs, but it supports only two page sizes: the original 2\,MB THP or 4\,KB base pages. It cannot produce intermediate mTHP sizes (e.g., 64\,KB or 256\,KB). This all-or-nothing granularity leads to excessive TLB coverage loss when splitting is applied.
Figure~\ref{fig:intel_splitting} also compares against MEMTIS, a state‑of‑the‑art system for huge page‑aware tiered memory. MEMTIS relies on a background thread to split THPs, but supports only two page sizes: the original 2\,MB THP or 4\,KB base pages. It cannot generate intermediate mTHP sizes (e.g., 64\,KB or 256\,KB). This all‑or‑nothing splitting granularity incurs excessive loss of TLB coverage once splitting is applied.
%The impact is most visible on NPB workloads, which traverse large arrays and are sensitive to TLB coverage. Compared to the AutoNUMA baseline, MEMTIS degrades NPB-FT.C by 7\%, and NPB-MG.D by 7\%. In contrast, \name improves NPB-FT.C by 9\%, and NPB-MG.D by 11\%, because mTHP splitting preserves enough TLB coverage to avoid the penalty that 4\,KB splitting incurs.
\textcolor{checked}{The impact is most evident on NPB workloads, which traverse large arrays and are particularly sensitive to TLB coverage. Relative to the AutoNUMA baseline, MEMTIS degrades MG.D by 7\% and LU.D by 8\%. In contrast, \name improves MG.D by 11\% while maintaining LU.D’s performance, as mTHP splitting preserves sufficient TLB coverage to avoid the penalties incurred by aggressive 4\,KB splitting.}
\textbf{TLB misses.}
\textcolor{checked}{Table~\ref{tab:tlb_miss} confirms the above observations with the measurement of L2 DTLB misses. MEMTIS consistently causes the highest TLB miss rate across the workloads. For LU.D, MEMTIS’s L2 DTLB misses reach 3.26 per 1K instructions---12\% higher than \name (2.90) and 47\% higher than AutoNUMA (2.22). For MG.D, the same trend holds: MEMTIS’s L2 DTLB misses (0.21) exceed \name’s (0.18) by 17\%. For workloads with large working sets, this TLB penalty outweighs the migration-precision gains.}
%L2 DTLB misses are particularly costly because each one triggers a full page table walk.
Overall, \name outperforms MEMTIS on 11 of 15 workloads. %achieving a 4.5% geometric‑mean advantage.
These results underscore that splitting granularity matters: while splitting to 4\,KB base pages---as MEMTIS does---recovers migration precision, it does so at the cost of TLB efficiency. In contrast, the mTHP splitting in \name strikes a better balance, preserving TLB coverage while still enabling fine‑grained migration.
\subsection{Evaluation with Traffic Balancing}
\label{sec:rw}
%This section evaluates the architecture-aware migration admission policy (\S\ref{subsec:admission}), which leverages CXL's full-duplex channels by selectively admitting or deferring page migrations based on the page's read/write classification and the current CXL traffic pattern (Table~\ref{tab:admission-policy}). We refer to this policy as \emph{BalanceTraffic}. We evaluate it under two contention levels: 20\,GB/s (high) and 10\,GB/s (moderate) memory contention traffic.
This section evaluates the duplex-aware migration admission policy (\S\ref{subsec:admission}), which exploits CXL’s full-duplex channels by selectively admitting or deferring page migrations based on each page’s read/write classification and the instantaneous CXL traffic pattern (Table~\ref{tab:admission-policy}). We refer to this policy as \emph{BalanceTraffic}. We evaluate BalanceTraffic under two memory-contention regimes: high contention at 20\,GB/s and moderate contention at 10\,GB/s.
\begin{figure}[!t]
\centering
\includegraphics[width=1\columnwidth]{./figure/cxl_rebalance_traffic_t4_arrow.pdf}
\caption{\textcolor{checked}{Performance under 20\,GB/s CXL contention, normalized to (a) AutoNUMA and (b) TPP baselines.}}
\label{fig:rebalance_t4}
\end{figure}
\textbf{High contention (20\,GB/s).}
%Figure~\ref{fig:rebalance_t4} compares four configurations under 20\,GB/s contention. The key finding is that BalanceTraffic \emph{does not complement splitting under high contention}. On AutoNUMA, Split alone achieves a geomean improvement of 9.5\%, but adding BalanceTraffic on top (Split+BalanceTraffic) reduces this to 5.1\%. On TPP, the effect is more dramatic: Split alone improves geomean throughput by 15.8\%, but Split+BalanceTraffic drops to 1.3\%. At 20\,GB/s contention, bandwidth is already the primary bottleneck, and the dominant performance factor is whether hot pages reside in the fast tier or not. In this regime, rebalancing read/write traffic across CXL's full-duplex channels yields little additional benefit---the bottleneck is total bandwidth, not directional imbalance. Worse, BalanceTraffic's selective admission actively harms performance by holding back hot pages whose read/write type does not match the dominant traffic direction (Table~\ref{tab:admission-policy}), preventing them from reaching the fast tier where they are most needed.
As shown in Figure~\ref{fig:rebalance_t4},
%compares four configurations.
%The key result is that
BalanceTraffic does not complement splitting under high contention. On AutoNUMA, enabling Split alone improves geomean performance by 12.3\%, while Split+BalanceTraffic reduces the gain to 6.6\%. The effect is more pronounced on TPP: Split alone achieves 17.7\% geomean throughput improvement, whereas Split+BalanceTraffic yields only 1.8\%.
%BalanceTraffic does not complement splitting under high contention
This is due to the following: Under 20\,GB/s contention, aggregate bandwidth is already the dominant bottleneck, and performance is primarily determined by whether hot pages are placed in the fast tier. In this regime, rebalancing read and write traffic across CXL’s full‑duplex channels provides little additional benefit---the limiting factor is total bandwidth, not directional imbalance. Moreover, BalanceTraffic’s selective admission policy can be counterproductive: by deferring migrations of hot pages whose R/W classification does not match the dominant traffic direction (Table~\ref{tab:admission-policy}), it prevents them from reaching the fast tier, where they are most needed.
%BalanceTraffic applied alone (without Split) shows a modest geomean gain on AutoNUMA (+5.3\%) but is inconsistent---12 of 19 workloads improve while 6 degrade. On TPP the picture is worse: only 5 workloads improve, 8 degrade, and the geomean is +2.3\%.
% When applied alone (without Split), BalanceTraffic yields a modest geomean improvement on AutoNUMA (+5.3\%), but its effects are inconsistent: 12 of 19 workloads improve, while 6 regress. The results on TPP are weaker still—only 5 workloads see improvements, 8 degrade, and the overall geomean gain is just 2.3%.
\textcolor{checked}{When applied alone (without Split), BalanceTraffic yields a modest improvement. On TPP in particular, only 7 workloads see improvements while 10 workloads degrade, and the overall geomean gain is only 3.2\%.}
\textbf{Moderate contention (10\,GB/s).}
%See Figure~\ref{fig:rebalance_t2}. %shows BalanceTraffic,
%with less bandwidth pressure.
The results improve: BalanceTraffic achieves a geomean improvement of 5.2\% on AutoNUMA and 4.7\% on TPP, with 13 of 17 workloads improved (Figure~\ref{fig:rebalance_t2}). %on both systems.
With more bandwidth headroom, the full-duplex rebalancing has room to take effect without starving hot pages of fast-tier placement. As such, selectively deferring promotions that would add traffic to the non-bottleneck CXL channel shows benefits.
\begin{figure}[!t]
\centering
\includegraphics[width=1\columnwidth]{./figure/cxl_rebalance_traffic_t2_arrow.pdf}
\caption{\textcolor{checked}{Performance under 10\,GB/s CXL contention, normalized to (a) AutoNUMA and (b) TPP baselines.}}
\label{fig:rebalance_t2}
\end{figure}
\begin{table}[t]
\caption{Benchmarks used for evaluation.}
\label{tab:benchmarks}
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{lllr}
\toprule
\textbf{Suite} & \textbf{Benchmark} & \textbf{Input / Configuration} & \textbf{Footprint} \\
\midrule
\multirow{4}{*}{GAP~\cite{gap}}
& BC & \multirow{4}{*}{Twitter~/ Uniform Random~/ Web} & \multirow{4}{*}{12--24\,GB} \\
& BFS & & \\
& CC & & \\
& PR & & \\
\midrule
\multirow{6}{*}{NPB~\cite{nas}}
& LU.D & Lower-Upper Gauss-Seidel solver & 58\,GB \\
& SP.D & Scalar Penta-diagonal solver & 76\,GB \\
& CG.D & Conjugate Gradient & 16\,GB \\
& MG.D & Multigrid & 28\,GB \\
\midrule
Silo~\cite{silo}
& YCSB & YCSB, scale factor 200K & 29\,GB \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[t]
\centering
\caption{Page splitting for four workloads with the largest improvement on AutoNUMA.
\emph{Split target} columns show the number of split events for each mTHP size;
\emph{Hot Subfolios} is the fraction of resulting subfolios classified as hot.}
\label{tab:split_dist_selected}
\resizebox{\columnwidth}{!}{
\footnotesize
\begin{tabular}{l|rrrrr|r}
\toprule
& \multicolumn{5}{c|}{Split target (count) } & \\
Workload & 64KB & 128KB & 256KB & 512KB & 1MB & Hot Subfolios (\%) \\
\midrule
cc-twitter & 4,101 & 21 & 0 & 0 & 0 & 9.1\% \\
cc-web & 0 & 6,013 & 147 & 1,470 & 0 & 21.9\% \\
bc-twitter & 1,998 & 186 & 1,907 & 1,103 & 0 & 22.5\% \\
bfs-web & 0 & 0 & 422 & 1,170 & 881 & 53.2\% \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[t]
\centering
\caption{Page migration statistics: NoSplit vs. THP Split.}
\label{tab:split_vs_nosplit}
\resizebox{\columnwidth}{!}{
\footnotesize
\begin{tabular}{l|rr|rr|rr}
\toprule
& \multicolumn{2}{c|}{Page Demotion} & \multicolumn{2}{c|}{Page Promotion Fail} & \multicolumn{2}{c}{Page Promotion Success} \\
Workload & NoSplit & THP Split & NoSplit & THP Split & NoSplit & THP Split \\
\midrule
bc-twitter & 1.10\,M & 1.34\,M & 18.01\,M & 0.08\,M & 1.26\,M & 2.44\,M \\
bc-urand & 1.61\,M & 1.96\,M & 8.46\,M & 0.01\,M & 2.32\,M & 3.34\,M \\
bc-web & 1.44\,M & 2.20\,M & 6.58\,M & 1.18\,M & 1.94\,M & 3.79\,M \\
bfs-twitter & 1.48\,M & 1.07\,M & 4.53\,M & 2.18\,M & 1.63\,M & 1.71\,M \\
bfs-urand & 1.81\,M & 2.02\,M & 7.03\,M & 1.60\,M & 2.19\,M & 3.25\,M \\
bfs-web & 1.54\,M & 1.82\,M & 2.66\,M & 0.08\,M & 2.09\,M & 2.77\,M \\
cc-twitter & 1.07\,M & 2.20\,M & 7.66\,M & 0.69\,M & 1.24\,M & 3.80\,M \\
cc-urand & 1.81\,M & 2.19\,M & 1.28\,M & 0.004\,M & 2.31\,M & 2.91\,M \\
cc-web & 1.56\,M & 2.54\,M & 6.00\,M & 0.44\,M & 1.94\,M & 4.28\,M \\
pr-twitter & 1.07\,M & 1.12\,M & 5.69\,M & 2.95\,M & 1.37\,M & 1.80\,M \\
pr-urand & 1.71\,M & 1.71\,M & 0.53\,M & 0.19\,M & 2.08\,M & 2.17\,M \\
pr-web & 1.49\,M & 1.49\,M & 1.58\,M & 1.23\,M & 2.02\,M & 1.92\,M \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[t]
\centering
\small
\caption{L2 DTLB misses per 1K instructions on the PMEM platform. MEMTIS's 4\,KB splitting incurs the highest TLB miss rates; \name's mTHP splitting reduces TLB overhead.}
\label{tab:tlb_miss}
\begin{tabular}{lcccc}
\toprule
\textbf{Workload} & \textbf{AutoNUMA} & \textbf{\name} & \textbf{MEMTIS} \\
\midrule
MG.D & 0.12 & 0.18 & 0.21 \\
LU.D & 2.22 & 2.90 & 3.26 \\
\bottomrule
\end{tabular}
\end{table}
\section{Related Work}
\label{sec:related}
\textbf{Memory tiering systems.}
AutoNUMA~\cite{autonuma} is the default system in Linux to manage tiered memory based on access recency. \textcolor{checked}{TPP~\cite{maruf2023tpp} extends AutoNUMA with proactive page demotion and LRU active-list-based hot page promotion.} HeMem~\cite{10.1145/3477132.3483550}, MEMTIS~\cite{lee2023memtis}, MTM~\cite{eurosys24:mtm}, and FlexMem~\cite{atc24_hm} migrate pages based on access frequency. %\textcolor{red}{MEMTIS splits a huge page for selective subpage-migration, but does not control page sizes and xxxx, compared with \name.}
Memstrata~\cite{10.5555/3691938.3691941} utilizes Intel Flat Memory mode to build cache-level data migration. %and aims to be used as a multi-VM memory allocator.
NOMAD~\cite{xiang2024nomad} builds non-exclusive memory tiering, aiming to mitigate memory thrashing when fast memory is under pressure. Colloid~\cite{vuppalapati2024colloid} decides page migration by balancing access latencies across memory tiers. HybridTier~\cite{10.1145/3676642.3736119} tracks long-term access frequency and short-term access momentum simultaneously to adapt to shifting hotness distributions. AOL~\cite{osdi25_aol} considers the impact of memory-level parallelism to avoid unnecessary page migration. \name is different from those efforts in terms of page size selection and memory traffic distribution. %according to duplex mode.
There are application semantics-guided tiering systems, including Unimem~\cite{unimem:sc17}, Xmem~\cite{dulloor2016data}, memkind~\cite{memkind}, HM-ANN~\cite{neurips20:hm-ann}, and WarpX-HM~\cite{ics21:warpx}. %Those efforts leverage application semantics to guide page migration.
Some efforts~\cite{nvmw21:dram-cache,meswani2015heterogeneous,Ramos:ics11} enable memory tiering using specialized hardware. In contrast, \name is application-transparent and requires no hardware changes.
\textbf{CXL memory} can increase memory scalability and simplify software stacks~\cite{sharma2023introduction}. From the perspective of CXL applications, Wang et al.~\cite{10.1145/3712285.3759816} and Xu et al.~\cite{xu2026ccclnodespanninggpucollectives} leverage CXL memory sharing for point-to-point and collective communications; Wang et al.~\cite{ipdps25:cxl}, Liu et al.~\cite{10.1145/3676641.3715987}, Tang et al.~\cite{eurosys24_cxl}, Mao et al.~\cite{mao2024cxlinterferenceanalysischaracterizationmodern} and Ji et al.~\cite{micro23_cxl} characterize the performance of real CXL hardware; CXL memory is also used for tensor offloading for AI workloads~\cite{11408577,ipdps25:cxl,yang2025beluga} and database~\cite{10.1145/3722212.3724460}.
From the perspective of CXL system software, TPP~\cite{maruf2023tpp} refines system software for page migration to accommodate CXL. Pond~\cite{10.1145/3575693.3578835} introduces a CXL memory pooling system to hold cold data. Apta~\cite{patil2023apta} is designed for function-as-a-service over CXL. HydraRPC~\cite{ma2024hydrarpc} utilizes CXL HDM for data transmission. ReScure~\cite{11408527} focuses on reliable and secure CXL memory. There are also recent efforts to support CXL memory pooling/sharing across hosts through innovation in memory allocation~\cite{10.1145/3779212.3790149}, formal method-based checking~\cite{10.1145/3779212.3790150}, programming model~\cite{10.1145/3779212.3790121}, and CXL bridges~\cite{10.1145/3779212.3790245}. Lupin~\cite{zhu2024lupin} is designed to tolerate partial failures for distributed applications using a shared CXL pod for replication. \name works on CXL memory and provides new optimizations considering page size and memory duplex mode.
\textbf{eBPF for memory optimization.}
eBPF is used to customize memory management policies, such as huge page placement, page fault handling, and page table designs~\cite{mores2024ebpf,zussman2024custom}. P2Cache~\cite{lee2026p2cache}, cache\_ext~\cite{zussman2025cache_ext}, PageFlex~\cite{yelam2025pageflex}, and FetchBPF \cite{cao2024fetchbpf} use eBPF to customize page cache, prefetching, and swapping policies. \name leverages eBPF to provide policy flexibility for new scenarios of memory management.
%is a recent concurrent proposal for a framework that uses eBPF to customize page cache policies [52]. While P2Cache is conceptually similar to cache\_ext, it only allows LRU or MRU ordering with a single queue, limiting the types of policies that can be implemented. Similarly, PageFlex [76] customizes prefetching and swapping policies by offloading policies to userspace using eBPF for event notifications and madvise()to communicate policy decisions. PageFlex focuses on memory mapping-based page access rather than file-based access. PageFlex only delegates some “non-performance-critical” decisions to the custom policy and enforces these decisions through madvise() hints. In contrast, by implementing policies in-kernel, cache_ext can delegate all policy decisions to custom policies and enforce them directly. FetchBPF allows customizing Linux’s memory prefetching policy, and could easily be integrated into cache\_ext as an additional hook [13]
%Apta [60] is a design for function-as-aservice (FaaS) over CXL, it does not target shared memory and does not consider multiple architectures. %Li et al. [52]designed Pond, a memory pooling system for CXL.
%Maruf et al.~\cite{TPP} propose TPP for OS-level page placement on top of CXL.
\section{Conclusion}
Memory tiering architectures are rapidly diversifying, which calls for consideration of page-migration interactions with other system components and memory architecture details.
%when designing the memory tiering system.
This paper discusses how to improve the memory tiering system to accommodate emerging changes in the memory architecture, especially from the perspectives of page size and memory access distribution. \name enables sufficient flexibility to develop novel policies in diverse environments, such as more than two memory tiers or CXL-SSD, which could be implemented in the future. %which will behave very different if you hit the SSD-attached memory or miss it and go to the SSD layer%We expect a more flexible design in the future memory tiering system.
%\asaf{Maybe you can say that \name in the future can incorporate policies that take into other types of configurations, e.g., taking into account more than one tier of tiered memory, or CXL-SSD which will behave very different if you hit the SSD-attached memory or miss it and go to the SSD layer}