SpecMD: A Comprehensive Study On Speculative Expert Prefetching
Extracted source text · arXiv 2602.03921
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{caption} % for \captionof
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{icml2025}
% For theorems and such
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{bbm} % for \mathbbm
% For algorithms
\usepackage{algorithm}
\usepackage{algorithmic}
% For colored boxes
\usepackage{xcolor}
\usepackage{colortbl}
% if you use cleveref..
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \theoremstyle{plain}
% \newtheorem{theorem}{Theorem}[section]
% \newtheorem{proposition}[theorem]{Proposition}
% \newtheorem{lemma}[theorem]{Lemma}
% \newtheorem{corollary}[theorem]{Corollary}
% \theoremstyle{definition}
% \newtheorem{definition}[theorem]{Definition}
% \newtheorem{assumption}[theorem]{Assumption}
% \theoremstyle{remark}
% \newtheorem{remark}[theorem]{Remark}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{enumitem}
\usepackage{hyperref}
\usepackage{url}
\usepackage[capitalize,noabbrev]{cleveref}
\usepackage{graphicx}
\usepackage{subcaption} % For side-by-side figures with (a), (b) labels
\usepackage[most]{tcolorbox} % 'most' loads breakable and other libraries
\usepackage{booktabs} % For professional-looking tables (\toprule, \midrule, \bottomrule)
\usepackage{multirow} % To span table cells across multiple rows
\usepackage{amssymb} % For checkmark (\checkmark) and cross (\times) symbols
\usepackage{pifont} % For the cross symbol (\ding{55})
% Define custom comment box command
\newcommand{\mohammad}[1]{
\begin{tcolorbox}[colback=yellow!10,
colframe=red!50!black,
title=Comment,
sharp corners,
breakable]
#1
\end{tcolorbox}
}
\newcommand{\edit}[1]{\textcolor{red}{#1}}
% \title{Two Experts are Better Than One: \\ Trading off Inference Compute with Performance}
% \title{RoE: Roster of Experts \\ for Dynamic MoE Inference}
% \title{RoE: Hyper-Parallel Scaling of MoE Models \\}
\title{SpecMD: A Comprehensive Study On Speculative Expert Prefetching}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
% \author{
% Duc Hoang\footnotemark[1] \and \textbf{Ajay Jaiswal} \and \textbf{Mohammad Samragh} \and \textbf{Minsik Cho} \\ \and \textbf{Apple}
% }
\author{
\qquad\quad Duc Hoang\footnotemark[1] \qquad
\textbf{Ajay Jaiswal} \qquad
\textbf{Mohammad Samragh} \qquad
\textbf{Minsik Cho} \\
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\textbf{Apple}
}
% \icmlauthor{Duc Hoang}{Apple}
% \icmlauthor{Ajay Jaiswal}{Apple}
% \icmlauthor{Mohammad Samragh}{Apple}
% \icmlauthor{Minsik Cho}{Apple}
% \author{
% \begin{tabular}{c}
% Anonymous Author 1\textsuperscript{1,2},\;
% Anonymous Author 2\textsuperscript{2},\;
% Anonymous Author 3\textsuperscript{2},\;
% Anonymous Author 4\textsuperscript{2} \\
% \textsuperscript{1}Affiliation 1 (University)\quad
% \textsuperscript{2}Affiliation 2 (Company)
% \end{tabular}
% }
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\newcommand{\olmoe}{OLMoE-1B-7B-0125-Instruct}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
% --- DEFINE THE FOOTNOTE TEXT HERE ---
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
\footnotetext[1]{Corresponding author: \texttt{dhoang2@apple.com}}
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
% ------------------------------------
\maketitle
\begin{abstract}
Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model’s parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required.
Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop \textbf{SpecMD}, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose \textbf{Least-Stale}, a novel eviction policy that exploits MoE's predictable expert access patterns to reduce collision misses by up to $85\times$ over LRU. With such gains, we achieve over $88\%$ hit rates with up to $34.7\%$ Time-to-first-token (TTFT) reduction on OLMoE at only $5\%$ or $0.6GB$ of VRAM cache capacity.
\end{abstract}
\input{1_intro}
\input{2_related_work}
\input{3_methodology}
\input{4_evaluation}
\input{5_conclusion}
\bibliography{main}
\bibliographystyle{iclr2025_conference}
\input{6_appendix}
\newpage
% \input{appendix}
% \appendix
% \section{Appendix}
% You may include other additional sections here.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\section{Introduction}
Mixture-of-Experts (MoE) architectures scale language models to unprecedented sizes through sparse computation. By activating only $k$ of $N$ experts per token, MoE models like Mixtral-8x7B \cite{jiang2024mixtralexperts}, DeepSeek-V2 \cite{deepseekai2024deepseekv2strongeconomicalefficient}, and OLMoE \cite{muennighoff2025olmoeopenmixtureofexpertslanguage} achieve sub-linear computational cost while maintaining large model capacity. However, this efficiency comes at a cost: a naive deployment of MoE models requires all $N$ experts to remain memory-resident, making deployment prohibitively expensive.
For instance, assuming 16 bits per weight, OLMoE-1B-7B activates 2.39GB yet needs 12.0GB ready in memory. Mixtral-8x7B requires 88GB storage yet only uses 24GB. This linear memory scaling limits the MoE deployment on memory-constrained devices.
Instead of keeping the full MoE model in active memory, an alternative is to store only the subset of patronized experts in a fast but capacity-limited cache, making memory management central to performance which can be influenced by the four aspects: $\bullet$\textbf{routing}: how to select experts to be cached, $\bullet$\textbf{prefetching} how to predict and cache the experts needed in future, $\bullet$\textbf{eviction} how to free space in the cache for demanded experts, and $\bullet$\textbf{miss handling} how to respond when required experts are absent from the cache. The importance of the above four aspects varies depending on hardware specifications. For instance, routing is capacity-dependent, prefetching is constrained by bandwidth, eviction matters in low-capacity settings, and miss handling is critical in balancing user-experience and task-performance.
Existing approaches tackle the cache management challenges through routing modification \cite{skliar2025mixturecacheconditionalexpertsefficient}, LRU caching \cite{Eliseev2023FastIO}, and dynamic precision \cite{Tang2024HOBBITAM,Zhu2025EnablingMO}, but their policies are heavily hardware-centric. Hence, they overfit to very specific set of circumstances, leaving general insights into policy interactions underexplored. This raises a fundamental research question: \emph{when, where, and which combinations of these policies would offer the most benefits across different deployment scenarios?}
\begin{figure}[ht]
\centering
\includegraphics[width=0.5\columnwidth]{figures/teaser_olmoe_ttft_improvement.png}
\caption{Least-Stale eviction as drop-in improvement across baseline approaches. Our novel eviction policy achieves 10.7--34.7\% TTFT reduction on OLMoE-1B-7B at 5\% cache capacity (0.6 GB).}
% \vspace{-20pt}
\label{fig:teaser}
\end{figure}
It is for this end that we develop \textbf{SpecMD} (Speculative MoE Decoding), a standardized benchmarking framework designed to enable reproducible comparison of MoE caching strategies. Through this framework, we gain three capabilities: (1) controlled comparison of prior approaches under identical conditions, (2) exploration of policy interactions and compositions, and (3) characterization of performance across the full hardware constraint space.
Using SpecMD, we conducted an exhaustive benchmarking study of MoE caching policies. We tested LRU caching~\cite{Eliseev2023FastIO}, fetch cascade~\cite{Tang2024HOBBITAM}, and expert substitution~\cite{Zhu2025EnablingMO} under controlled and comparable conditions. We evaluate four MoE models (Mixtral-8x7B, OLMoE-1B-7B, Phi-3.5-MoE, Qwen1.5-MoE) and three different hardware configurations. From our studies, we made the following observations:
\begin{itemize}
\vspace{-0.12 in}
\item \textbf{MoE's expert access can cause wrongful eviction.} In MoE, experts are accessed following a deterministic sequential layer pattern, not recency-based reuse, rendering LRU and LFU policies perform quite poorly.
\item \textbf{Dynamic prefetching is better for cache.} We observe dynamic, score-based prefetching outperforms top-k approaches; despite lower overall prediction accuracy, score-based prefetching yields higher hit-rates and makes use of available bandwidth more effectively.
\item \textbf{Policy rankings shift across hardware regimes.} As expected, the best-performing strategy at 1\% capacity differs from that at 5\% or 25\%; bandwidth-limited and capacity-limited scenarios favor different approaches. No single set of policy dominates, but there are trends (see Figure~\ref{fig:q3b_per_layer_all} in Appendix).
\vspace{-0.1 in}
\end{itemize}
From these findings, we propose \textbf{Least-Stale}, an eviction policy that balances temporal and spatial awareness to minimize cache collision misses. Our contributions are:
\begin{itemize}
\vspace{-0.12 in}
\item \textbf{A Framework for Policy Research.} We introduce an open-source policy-first framework for MoE Cache to accelerate policy prototyping and interaction.
% We present an open-source framework that enables reproducible comparison of various policies across diverse hardware configurations. SpecMD enables us to replicate prior work under controlled conditions for testing and evaluation.
\item \textbf{Policies studies and Benchmarking.} Using SpecMD, we study policies interaction on different models and constraints and reveal key-insights for policy-makers.
\item \textbf{Least-Stale Eviction Policy.} Building on our findings, we propose Least-Stale, a flexible eviction policy that greatly reduces collision-misses by up to $85\times$ versus LRU at 1\% capacity, achieving 88-92\% hit rates. We show in Figure~\ref{fig:teaser} Least-Stale provides consistent gains across diverse baseline strategies.
\end{itemize}
\section{Related Work}
MoE memory constraints are addressed through several complementary approaches: routing modification, expert caching with prefetching and quantization. We focus on expert caching approaches, as they directly target cache management, i.e., how to decide which experts to load or evict and how to handle cache misses. Existing work evaluates 1-2 policy combinations in isolation, leaving policy interactions and performance across hardware constraints largely uncharacterized.
% Table~\ref{tab:related_policies} summarizes these policy dimensions.
\noindent{\bf Prefetching.} \citet{Eliseev2023FastIO} introduce speculative prefetching for memory-constrained MoE inference. Their approach predicts next-layer expert requirements using current-layer hidden states, enabling single-layer-ahead speculation. HOBBIT \cite{Tang2024HOBBITAM} extends this by leveraging gating input similarity across adjacent layers to prefetch multiple layers ahead. Both approaches predict based on architectural proximity but do not account for expert importance when bandwidth is limited. \citet{Zhu2025EnablingMO} addresses this through score-based prefetching, which filters experts by gate score percentile rather than fixed top-$k$ counts. This prioritizes high-importance experts when bandwidth cannot accommodate all predictions.
\noindent{\bf Eviction.} Eviction policies determine which cached experts to remove when capacity is exceeded. \citet{Eliseev2023FastIO} employ LRU caching, keeping a fixed number of recently-used experts across all layers. \citet{Tang2024HOBBITAM} proposes LHU (Least High-precision Used), a variant of LFU that prioritizes retaining high-precision experts. \citet{Zhu2025EnablingMO} use score-based eviction, removing experts with the lowest accumulated activation scores.
\noindent{\bf Miss handling.} For cache misses, \citet{Eliseev2023FastIO} trigger blocking loads from CPU memory. HOBBIT introduces precision cascading: dynamically loading lower-precision versions (int4 for float16, int2 for int8) based on gating score thresholds \cite{Tang2024HOBBITAM}. \citet{Zhu2025EnablingMO} employ expert substitution, replacing cache-miss experts with functionally similar cached experts whose scores fall within tolerance thresholds.
\noindent{\bf Routing.} MixtureCache \cite{skliar2025mixturecacheconditionalexpertsefficient} takes a fundamentally different approach by modifying routing rather than optimizing cache policies. They bias expert selection toward already-cached experts through learnable parameters applied to router logits. Their system employs LRU eviction but no prefetching mechanism, instead treating cache misses as inevitable and modifying routing to reduce misses.
\begin{figure*}[ht]
% \vspace{-20pt}
\centering
\includegraphics[width=\textwidth]{figures/MoE-Cache2.pdf}
% \vspace{-20pt}
\caption{Main(red) and Prefetch (yellow) loops operate synchronously and are directly invoked by the model's after gate. Watchdog (blue) operates asynchronously, constatnly monitoring the prefetch queue. }
% \vspace{-10pt}
\label{fig:moe_cache_overview}
\end{figure*}
\section{SpecMD: Framework and Policy Space}
% too Aggresive
%Prior work on MoE caching offers neither public repositories \cite{Eliseev2023FastIO,Zhu2025EnablingMO} nor hardware-agnostic solutions \cite{Tang2024HOBBITAM}, limiting reproducibility and systematic exploration.
We present a framework that prioritizes ease of use (straightforward integration with HuggingFace models), accessibility (no specialized hardware required), and policy modularity. We aim to enable straightforward policy comparison across models and hardwares on consumer-grade GPUs\footnote{open-sourcing SpecMD is underway.}.
\subsection{Policy Design Space}
Cache policies interact across four dimensions, shown in Figure~\ref{fig:moe_cache_overview}, each affecting \textit{quality}, \textit{speed}, or both. Understanding these interactions is the central challenge.
% —prior work evaluates a specific set of policies missing complex cross-policy interactions. For this reason,
We systematically explore this design space to characterize policy behaviors and identify optimal configurations under different quality-memory constraints.
\subsubsection{Routing Policies}
When an expert is requested by the gate module in an MoE block (top left of Figure~\ref{fig:moe_cache_overview}), it may not exist in the cache. In this case, the routing policy is the first module that we may visit.
Routing policies (shown as a green policy box in the Main Loop of Figure~\ref{fig:moe_cache_overview}) determine whether to influence expert selection based on cache state, impacting both \textit{performance} and \textit{speed}. This represents a trade-off between cache efficiency and routing integrity.
\textbf{Standard routing} preserves original model behavior using unmodified router logits, maintaining 100\% routing fidelity. All other policy dimensions operate around the model's original expert selections.
\textbf{Cache-aware routing} biases selection toward cached experts by boosting GPU-resident expert logits \cite{skliar2025mixturecacheconditionalexpertsefficient}. Let $z = G(x)$ represent original router logits, $C$ denote cached expert indices, and $\mathbbm{1}_C \in \{0,1\}^{|E|}$ indicates cached experts. Modified logits are
\begin{equation}
z' = z + \lambda \cdot \Delta_{avg} \cdot \mathbbm{1}_C,
\end{equation}
where $\lambda$ is hyperparameter used for tuning and $\Delta_{avg}$ is the historical average logit value. Modified logits $z'$ pass through softmax and top-$k$ selection as usual.
\subsubsection{Cache Miss Policies}
After the routing policy is applied, the expert may still not be available in cache. Cache miss policies, as the name suggested, handle missing experts from cache (this is showed in the Main Loop of Figure~\ref{fig:moe_cache_overview}). When cache misses occur, these policies can force immediate eviction to make space and may stall the model until the needed experts are fetched, substituted, or dropped. Unlike eviction/prefetching (primarily affecting speed), miss handling impacts both \textbf{quality} and \textbf{speed}. We implement three cache miss policies:
\textbf{Fetch policies} always fetch missing experts from CPU memory, preserving routing integrity at the cost of synchronous transfer latency. Basic Fetch guarantees 100\% fidelity but degrades speed when misses are frequent. We introduce two precision-aware variants: \textit{Fetch-lowest-bit} (e.g., int8:4) fetches the lowest available precision to minimize latency, while \textit{Fetch-priority} (e.g., int8:4:2) always fetches highest precision with fallback levels (int8$\rightarrow$int4$\rightarrow$int2) for graceful degradation under capacity constraints.
\textbf{Drop policies} selectively skip cache-miss experts. Skliar et al. \cite{skliar2025mixturecacheconditionalexpertsefficient} show MoE models are robust to expert variations: dropping top-ranked experts compromises performance, but models are resilient beyond rank 2, especially granular MoEs with many active experts. They demonstrate minimal perplexity degradation when swapping rank 3+ experts, which we leverage similarily by dropping only low-rank experts and always fetching high-rank ones.
\textbf{Substitution policies} replace cache-miss experts with functionally similar cached experts based on score proximity \cite{Zhu2025EnablingMO}, but quality degrades significantly in practice (Section 5.3).
\subsubsection{Eviction Policies}\label{sec:eviction}
After the cache-miss policy is applied, an expert may need to be loaded into the cache.
Eviction policies determine which experts to remove when cache capacity is exceeded, and play a fundamental role in both the Main and Watchdog loop as seen Figure~\ref{fig:moe_cache_overview}. In isolation, eviction policies influence \textbf{speed} aspect of a caching system. We support a couple broad categories of eviction policies (full details is in Appendix~\ref{app:baselines}):
\textbf{$\bullet$ Temporal policies.} LRU (Least Freqently Used) and LFU (Least Frequently Used) are classic examples that track and select the victim by access-time or access-frequency.
\textbf{$\bullet$ Spatial policies.} SB (Score-Based) and FLD (Furthest Layer Distance) select the victim based on its physical characteristic such as gate's logits or distance from current layer.
\subsubsection{Prefetch Strategies}
Prefetch strategies (illustrated in the Prefetch Loop of Figure~\ref{fig:moe_cache_overview}) load experts before they are requested, primarily influencing \textbf{speed}. The idea is to utilize the gate from an upcoming layer together with the activations of the current layer to predict future required experts. In our experiments we resort to prefetching only the very next layer experts.
Prefetch operates asynchronously via the Watchdog Loop (shown in blue in Figure~\ref{fig:moe_cache_overview}) that monitors capacity and enqueues prefetch requests to the GPU when space is available.
%We prefetch the top-$k$ experts per layer, where $k$ is model-specific ($k=8$ for OLMoE, $k=2$ for Mixtral, etc).
We support two configuration dimensions in SpecMD:
\textbf{$\bullet$ Top-k prefetch with overfetch.} The standard approach prefetches the top-k experts per layer based on router weight magnitude, where k is model-specific ($k=8$ for OLMoE, $k=2$ for Mixtral). We introduce an overfetch factor that multiplies the prefetch count to increase bandwidth utilization: with overfetch factor 1.5, OLMoE prefetches 12 experts instead of 8. This trades prefetch precision for cached experts recall at the expense of bandwidth.
\textbf{$\bullet$ Score-based approach.} Following \citet{Zhu2025EnablingMO}, this strategy filters experts by softmax score rather than fixed top-k count and prefetch all experts above a configurable percentile threshold (the default is 80th percentile). Unlike top-k approaches, this method adapts the number of prefetched experts based on score distribution.
\begin{algorithm}[t]
\caption{Gate Hook Processing}
\label{alg:gate_hook}
\begin{algorithmic}[1]
\STATE \textbf{Input:} Gate outputs, layer $l$
\STATE $(experts, weights, logits, router\_weights) \gets$ Gate outputs
\STATE // \textit{Phase 1: Routing modification}
\STATE $logits' \gets$ RoutingPolicy.apply$(l, logits)$
\STATE $experts' \gets \text{top\_k}(\text{softmax}(logits'))$
\STATE $weights' \gets router\_weights[experts']$
\STATE // \textit{Phase 2: Cache management (during expert execution)}
\FOR{each expert $e$ in $experts'$}
\IF{$e \notin$ GPU\_Cache}
\STATE MissPolicy.handle$(e)$
\ENDIF
\ENDFOR
\STATE // \textit{Phase 3: Prefetch submission}
\IF{$l$ has prefetch targets $\mathcal{T}$}
\FOR{each target layer $t \in \mathcal{T}$}
\STATE $predicted \gets$ PrefetchPolicy.predict$(t)$
\STATE PrefetchQueue.submit$(predicted, t)$
\ENDFOR
\ENDIF
\STATE \textbf{Return:} $(experts', weights', logits', router\_weights)$
\end{algorithmic}
% \vspace{-10pt}
\end{algorithm}
\subsection{Implementation Overview}
To avoid modifying a model's internal implementation, we utilized Pytorch forward hooks to intercepts and modify MoE gate outputs. We also standardize the gate-output protocal interface across architectures (Mixtral, OLMoE, Qwen) for more effective code-reuse. In Algorithm~\ref{alg:gate_hook} we show the core hook processing across three phases: routing modification, cache management, and prefetch submission. Complete implementation details appear in Appendix~\ref{app:implementation}.
\section{Least-Stale Eviction Policy}
In this section, we present our novel policy contribution Least-Stale, which addresses some of the gaps presented in traditional eviction policies for performance improvement.
\begin{figure}[ht]
\centering
\includegraphics[width=0.5\columnwidth,trim=0 20pt 0 10pt,clip]{figures/LS-diagram.pdf}
\vspace{-10pt}
\caption{\textbf{Top:} LRU eviction as a purely temporal policy. Blocks with higher intensity indicate higher eviction priority; consequently, experts from the immediately following layer are most likely to be evicted. \textbf{Bottom:} Illustration of the Least-Stale (LS) policy, which combines temporal and positional signals. Cached experts are marked as current (green, accessed in the current forward pass) or stale (red, accessed in a previous pass), while layer index encodes position. Higher color intensity corresponds to higher eviction probability. Stale experts have a higher eviction priority than current ones.}
\label{fig:ls_diagram}
% \vspace{-20pt}
\vspace{-0.5cm}
\end{figure}
\subsection{Motivation}
MoE expert access follows a deterministic front-to-back layer sequence (layer 0$\rightarrow$1$\rightarrow$2$\rightarrow$$\cdots$$\rightarrow$15) within each forward pass. Once we move on from a layer, that layer's experts will not be needed until the next forward pass. Temporal-only policies such as LRU and LFU utilize access-time and access-frequency as indicators for expert importance, and thus treat a recently accessed expert as important even though it is now irrelevant to the current forward pass. On the other hand, spatial-only policies like FLD and Score-Based use layer distance or router scores to indicate importance with no understanding of temporal relevancy. Regardless of approach, the outcome is the same: experts from future layers get evicted prematurely. We show this in Figure~\ref{fig:ls_diagram}-top where the immediate next layer experts are most likely to be evicted. This observation motivates our Least-Stale approach, which exploits the spatial-temporal structure of expert access to prevent premature evictions.
\subsection{The Least-Stale Policy}
Eviction policies determine which cached experts to evict when capacity is exceeded.
In many instances, eviction policies can cause \textbf{collision misses}, a type of cache miss where experts are evicted and then immediately needed again in the same forward pass. To address collision misses, Least-Stale (LS) combines both temporal factors (access time) and spatial awareness (layer positioning) to minimize collision misses.
% We call these two factors staleness and position, denoted by isPast and isStale binary values explained later in this section. In Figure~\ref{fig:ls_diagram}, we show the temporal factor using red or green, while the positional factor is shown illustrate this in .
\textit{Staleness and position.}
An expert is considered \emph{stale} if it was accessed in a previous forward cycle; otherwise, it is \emph{current}, meaning it is used in the current pass or selected by prefetching. We describe an expert as \emph{left} if it appears earlier than the current execution point in layer–expert order.
% , formally defined as:
% \begin{equation}
% \text{isLeft}(l, e) = \begin{cases}
% \text{true} & \text{if } l < l_c \text{ or } (l = l_c \text{ and } e < e_c) \\
% \text{false} & \text{otherwise}
% \end{cases}
% \end{equation}
Figure~\ref{fig:ls_diagram}-bottom illustrates how staleness and position jointly determine eviction priority. The cache is partitioned into a \emph{stale queue} containing experts from previous forward passes and a \emph{current queue} containing experts from the ongoing pass.
Eviction prioritizes stale experts, as recently used experts are more likely to be reused by upcoming tokens or refreshed by prefetching. Experts in the current queue are evicted only to prevent out-of-memory errors. Within each queue, experts follow FIFO ordering based on layer position, favoring spatial and temporal locality.
In practice, both queues are implemented as priority heaps, requiring $\mathcal{O}(N)$ space, $\mathcal{O}(\log N)$ insertion, and $\mathcal{O}(1)$ eviction, where $N$ is the cache capacity.
\section{Evaluation and Results}
Having motivated and described Least-Stale, we now present comprehensive evaluation results that validate our design choices and characterize performance across the full policy space.
\subsection{Experimental Setup}
% Using SpecMD's systematic evaluation capabilities, we conduct the first comprehensive comparative study of MoE caching policies across the full design space: eviction, prefetching, miss handling, and routing—each with multiple options across capacity constraints (1-25\% of model size), bandwidth limits, and precision configurations (int8, int4, int2).
To effectively traverse the huge policy decision-space across various models and capacity constraints, we adopt a staged evaluation strategy: we first isolate optimal configuration from performance-neutral policies, i.e eviction policies and prefetch policies, then evaluate quality-speed trade-offs using optimal configurations from early stages. This approach reduces the search space while ensuring fair comparison.
\textbf{$\bullet$ Models.} We evaluate across four MoE architectures: OLMoE-1B-7B \cite{muennighoff2025olmoeopenmixtureofexpertslanguage}, Mixtral-8x7B \cite{jiang2024mixtralexperts}, Qwen1.5-MoE-A2.7B\cite{qwen_moe}, and Phi-3.5-MoE \cite{abdin2024phi3technicalreporthighly}. Our selection covers both wide MoE with many small experts and deep MoE with a few yet large experts, enabling us to characterize how policy effectiveness varies with a model structure.
\textbf{$\bullet$ Tasks.} To measure realistic caching behaviour (with both prefill and decoding stages) we test tasks using autoregressive generation. These tasks are: GSM8K \cite{cobbe2021gsm8k} (mathematical reasoning), TruthfulQA \cite{lin2021truthfulqa} (factuality), and NaturalQuestions\cite{47761} (question answering). Performance is measured via exact match for GSM8K and GPT-5 as judge for QA tasks. During policy benchmarking, we sampled only 100 examples for each task.
\textbf{$\bullet$ Hardware emulation.} We evaluate all models on a single A100 GPU with 80GB VRAM. We software-limit our GPU cache to 1\%, 5\%, or 25\% of full model size, which amounts for example, to at most 21GB and as few as 0.8GB for Mixtral. We opted to use device native bandwidth, which on our device measures on average 5 GB/s. We quantize
experts into int8 and int4 using bitsandbytes \cite{dettmers2022llmint8} and optionally int2 using HQQ \cite{badri2023hqq}to evaluate cache miss policy comprehensively.
For detailed model specifications please see in Appendix~\ref{app:model_specs}.
\subsection{Least-Stale Effectiveness}
\label{sec:eviction_analysis}
We first validate the Least-Stale policy by analyzing its impact on collision misses and inference speed across different models and capacity levels. We can see how baseline eviction policies at various capacities scored higher collision rates in Figure~\ref{fig:q3b_all} which presents a 5\% cache capacity. More extensive results can be found in Figure~~\ref{fig:q3b_all_full} of the Appendinx.
For 5\% cache capacity in particular, Least-Stale achieves 1.6-1.9\% collision rates, while LRU reaches 4.5-12.6\% and SB suffers catastrophic 42.6-60.9\% rates. SB performs worst because historical gate scores capture neither spatial position nor temporal relevance; a highly-scored expert from layer 2 gets retained over a lower-scored expert from layer 10, even though layer 10 is spatially next. FLD, unlike LRU, tracks only expert distance from current layer and performed only slightly better than LRU.
\begin{figure}[t]
\centering
\includegraphics[width=0.9\columnwidth]{figures/q3b_collision_analysis_cap5pct.png}
\caption{Collision miss across four eviction policies (FLD, SB, LRU, LS) at 5\% capacity threshold showing total misses (gray) and collision misses (red). Refer to Section~\ref{sec:eviction} for abbreviations.}
\label{fig:q3b_all}
\vspace{-20pt}
\end{figure}
\textbf{$\bullet$ Spatial–Temporal awareness.} Least-Stale determines eviction priority by forward-pass cycles. Experts accessed in the \textit{current} pass are protected to prevent premature eviction of prefetched future experts and to increase the odds of reuse in the next forward cycle for left experts. Experts from \textit{previous} passes are marked as stale and prioritized for eviction, unless they are needed again via prefetching.
%—they won't be needed until we start processing a new token, giving prefetch time to reload them if necessary.
In Figure~\ref{fig:q3b_olmoe}, we illustrate the per-layer collision misses. Baseline policies tend to accumulate throughout the forward pass as they evict soon-needed experts. At 1\% capacity, LRU and Score-Based reach 8-12\% collision rates in deeper layers. Least-Stale maintains near-zero collisions across all layers by only evicting stale left-layer experts.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{figures/q3b_collision_analysis_per_layer_olmoe.png}
\vspace{-20pt}
\caption{Per-layer collision miss rates for OLMoE across 16 layers. Least-Stale (red) maintains near-zero collisions across layers.}
\label{fig:q3b_olmoe}
\vspace{-20pt}
\end{figure}
\textbf{$\bullet$ Speed Impact.} The collision reduction translates directly to TTFT gains. At 5\% capacity, Least-Stale reduces collision misses by 2.6-8.6× versus LRU and 58-91× versus Score-Based. When used as a drop-in replacement for existing approaches' eviction policies (keeping their prefetch and routing unchanged), Least-Stale achieves 10.7-34.7\% TTFT improvements on OLMoE (Figure~\ref{fig:teaser}), with consistent gains of 14.8-44.1\% across diverse baselines systems (Table~\ref{tab:eviction_ablation}).
\subsection{Prefetch Strategy Effectiveness}
\begin{tcolorbox}[colback=gray!10, colframe=black!50, title=\textbf{Key Insight: Prediction accuracy $\neq$ cache performance}]
Score-based prefetching outperforms top-k despite lower overall prediction accuracy. Giving each layer the ability to adapt the number of experts to prefetch based on gating scores proves more effective at utilizing available bandwidth that a fix threshold.
\end{tcolorbox}
\textbf{$\bullet$ Understanding the mismatch.} Using Least-Stale as our baseline eviction policy, we compare two prefetch strategies: top-k prefetching with different multipler rates and score-based percentile filtering. What we discover is that higher prediction accuracy does not guarantee better cache performance in term of speed.
In Figure~\ref{fig:q4_5pct} we show that top-k with a 1.0 factor achieves high precision and recall across all models. Yet the right column shows that score-based prefetching typically delivers higher hit rates with lower synchronous overhead (time spent waiting on cache misses) for the majority of the model we tested, with the exception of Mixtral.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{figures/q4_5pct.png}
\caption{Prefetch strategy comparison at 5\% capacity. Left: Precision vs recall shows top-k 1.0× achieves highest prediction accuracy. Right: Hit rate vs synchronous overhead reveals score-based 80th percentile achieves better cache performance for most models through dynamic adaptation.}
\label{fig:q4_5pct}
% \vspace{-20pt}
\end{figure}
\textbf{$\bullet$ Effective Bandwidth Utilization.} We find that the score-based approach more effectively adapts the number of experts to prefetch on a per-layer basis. Simply scaling up a fixed number of experts does not match the same performance. We conclude that adaptive prefetching more effectively allocates existing bandwidth to high-reward layers while reducing the number of experts prefetched for high-risk layers, thereby outperforming fixed top-k strategies in most cases.
\textbf{$\bullet$ The Exception.} We observe Mixtral as the exception. It is a deep MoE with a small total number of experts, but each expert is large ($\sim$336MB). For these types of MoEs, the simpler top-k strategy performs competitively. This occurs because the size of the experts is too large for the available bandwidth, which limits the benefit of a dynamic prefetch system. This suggests that deep MoEs with large expert sizes may be better suited to simpler top-k prefetching.
%We find that overfetch factors (1.5×, 2.0×) consistently underperform across all models, creating eviction pressure without improving hit rates.
\subsection{Quality-Speed Trade-offs}
\begin{figure}[h!]
\centering
\includegraphics[width=0.7\columnwidth]{figures/q5_cap5pct.png}
% \vspace{-20pt}
\caption{Quality-memory trade-offs for six miss handling policies at 5\% capacity. Performance and speed shown relative to Fetch policy baseline. We report the average accuracy among the tasks under study as performance. Points above the zero horizontal line indicate performance improvements compared to the baseline; points to the right of the zero vertical line indicate speed gains. Color intensity indicates cache-aware routing strength ($\lambda$). Substitution policies fall outside the chart range.}
\label{fig:q5_cap5pct}
% \vspace{-10pt}
\end{figure}
\begin{table*}[ht]
\centering
\caption{Baseline comparison across all models at 5\% cache capacity (4bit quantization, 5 GB/s bandwidth, with QKV Cache). TTFT in milliseconds. SpecMD is capable of exploring multiple configurations of the state-of-the-art polices, and one of them is our chosen configuration (Config 5) from Figure~\ref{fig:q5_cap5pct}, balancing between performance and overall speed.}
\small
\setlength{\tabcolsep}{2.5pt}
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}l|l|ccc|cc||ccc|cc@{}}
\toprule
\multicolumn{2}{c|}{ } & \multicolumn{5}{c||}{\textbf{OLMoE-1B-7B}} & \multicolumn{5}{c}{\textbf{Mixtral-8x7B}} \\
\multicolumn{2}{c|}{\textbf{Approach}} & \textbf{GSM8K} & \textbf{Truth.} & \textbf{NQ} & \textbf{TTFT} & \textbf{Token/s} & \textbf{GSM8K} & \textbf{Truth.} & \textbf{NQ} & \textbf{TTFT} & \textbf{Token/s} \\
\midrule
\multicolumn{2}{c|}{Baseline (8bit)} & 0.704 & 0.540 & 0.361 & 3166 & 1.5 & 0.569 & 0.345 & 0.510 & 10296 & 0.4 \\
\midrule
Config\#1& Spec$^+$, LRU $^\circ$, Fetch$^\square$, Stnd$^\mathsection$ & 0.677 & 0.555 & \textbf{0.343} & 3077 & 1.91 & \textbf{0.566} & 0.343 & \textbf{0.518} & 4238 & 0.98 \\
Config\#2& Score$^\spadesuit$, ScoreE$^\clubsuit$, Subs$^\diamondsuit$, Stnd & 0.073 & 0.493 & 0.159 & 2257 & 1.22 & 0.037 & 0.251 & 0.273 & \textbf{1620} & 0.75 \\
Config\#3& Spec, LHU$^\nabla$, Cascade$^\Delta$, Stnd & 0.606 & 0.548 & 0.291 & \textbf{1590} & 1.34 & 0.310 & 0.326 & 0.347 & 2322 & 0.71 \\
Config\#4 & No Pref, LRU, Fetch, Mod$^\psi$ & 0.664 & 0.558 & 0.332 & 2665 & 2.33 & \textbf{0.566} & 0.326 & 0.517 & 4333 & 1.07 \\
\rowcolor{blue!10} Config\#5 & Score, LS, Fetch, Stnd & \textbf{0.691} & \textbf{0.559} & 0.331 & 2220 & 1.69 & 0.528 & \textbf{0.356} & 0.471 & 3044 & 1.42 \\
\bottomrule
\end{tabular}
}
\vspace{1em}
\setlength{\tabcolsep}{2.5pt}
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}l|l|ccc|cc||ccc|cc@{}}
\toprule
\multicolumn{2}{c|}{ } & \multicolumn{5}{c||}{\textbf{Phi-3.5-MoE}} & \multicolumn{5}{c}{\textbf{Qwen1.5-MoE-A2.7B}} \\
\multicolumn{2}{c|}{\textbf{Approach}} & \textbf{GSM8K} & \textbf{Truth.} & \textbf{NQ} & \textbf{TTFT} & \textbf{Token/s} & \textbf{GSM8K} & \textbf{Truth.} & \textbf{NQ} & \textbf{TTFT} & \textbf{Token/s} \\
\midrule
\multicolumn{2}{c|}{Baseline (8bit)} & 0.697 & 0.774 & 0.594 & 7902 & 0.8 & 0.471 & 0.595 & 0.460 & 5728 & 2.3 \\
\midrule
Config\#1 & Spec$^+$, LRU $^\circ$, Fetch$^\Box$, Stnd $^\mathsection$ & \textbf{0.705} & 0.775 & 0.589 & 4452 & 1.69 & \textbf{0.506} & \textbf{0.614} & 0.473 & 3724 & 2.89 \\
Config\#2 & Score$^\spadesuit$, ScoreE$^\clubsuit$, Subs$^\diamondsuit$, Stnd$^\mathsection$ & 0.027 & 0.514 & 0.269 & \textbf{1906} & 0.86 & 0.008 & 0.480 & 0.408 & 2739 & 1.11 \\
Config\#3 & Spec, LHU$^\nabla$, Cascade$^\Delta$, Stnd & 0.604 & 0.743 & 0.563 & 3459 & 1.05 & 0.217 & 0.608 & 0.462 & 2492 & 2.19 \\
Config\#4 & No Pref, LRU$^\circ$, Fetch$^\Box$, Mod$^\psi$ & \textbf{0.705} & 0.777 & \textbf{0.591} & 5025 & 1.72 & 0.296 & 0.611 & 0.465 & 3489 & 3.50 \\
\rowcolor{blue!10} Config\#5 & Score, LS, Fetch, Stnd& 0.684 & \textbf{0.780} & 0.587 & 3863 & 1.11 & 0.497 & 0.607 & \textbf{0.474} & 2893 & 2.11 \\
\bottomrule
\end{tabular}
}
%%%maybe we can explain in the captions
% \begin{tablenotes}
$^\circ$ Speculative Prefetch; $^\square$ LRU Eviction~\cite{Eliseev2023FastIO}; $^\mathsection$ Fetch immediately for miss; $^\mathsection$Standard routing; $^\spadesuit$ Score-based Prefetch~\cite{Zhu2025EnablingMO}; $^\clubsuit$ Score-based Eviction~\cite{Zhu2025EnablingMO}; $^\diamondsuit$ Substitution with cached experts for miss~\cite{Zhu2025EnablingMO}; $^\nabla$ Least High-precision Eviction~\cite{Tang2024HOBBITAM}; $^\Delta$ Fetch with precision cascade ; $^\psi$ Routing Modification \cite{skliar2025mixturecacheconditionalexpertsefficient}
% \end{tablenotes}
\label{tab:baseline_comparison}
\vspace{-10pt}
\end{table*}
\textbf{$\bullet$ Trading Quality for Speed.} With optimal eviction and prefetch policies established, we now examine how different miss handling and routing strategies perform. By modifying these two policies, we can achieve a quality/speed tradeoff, shown in Figure~\ref{fig:q5_cap5pct}.
when studying miss-handling policies, we find that Fetch Priority (blue squares) consistently occupies the Pareto frontier across models. This approach stores multiple precision levels and selectively degrades low-importance experts (bottom 60th percentile) during cache misses.
For Mixtral at 8-bit, we observe 10\%-12\% performance improvements with 50-60\% speed gains.
For OLMoE at 4-bit, Fetch Priority maintains near-baseline performance (0-5\% loss) while achieving 20\%-30\% speedup.
\textbf{$\bullet$ The Trade-offs.} Drop Priority, as expected, provides the highest speed improvements at the cost of performance degradation. Qwen at 4-bit suffers -25\% to -30\% performance loss, while OLMoE, a wide MoE, shows only -5\% to -15\% degradation. This effectively demonstrates that more active experts can dilute the impact of dropping any single expert. We find that Fetch Lowest Bit and Drop Priority Cascade show inconsistent results across models, offering no clear advantage over simpler strategies. On the other hand, Substitution Score fails across all configurations, suggesting that replacing missing experts with functionally similar alternatives is unreliable, regardless of capacity level.
\textbf{$\bullet$ Cache-Aware Routing Impact.} We find that cache-aware routing is credited for 10\%-20\% of the achieved speed improvements in some cases (compare different intensities of the blue squares on each figure). The effectiveness depends critically on model architecture: OLMoE can tolerate higher routing bias ($\lambda$) than Mixtral, since wider expert distributions can redistribute the impact of routing perturbations more effectively. Our experiments show that $\lambda$ values between 0.1-0.5 work best, balancing speed gains with quality preservation; too conservative values provide minimal benefit, too aggressive values degrade quality unacceptably.
% —too conservative provides minimal benefit, too aggressive degrades quality unacceptably.
\subsection{SpecMD Mix-Policy Evaluation}
SpecMD's enables an easy way to mix and run ad-hoc policies across all four dimensions (eviction, prefetch, miss handling, routing) on the same testing platform. Through it, we can identify policies with favorable characteristic via policy evaluation. In previous sub-sections, we discussed at length policy interactions and the various tradeoffs; we now selected five set of mix policies to compare running full task evaluation. This we summarize in Table~\ref{tab:baseline_comparison} using 5\% capacity cache capacity and 4bit quantization.
\textbf{$\bullet$ Configuration selection methodology.} To ensure fair comparison and avoid cherry-picking, we first select a four configurations (\#1-\#4) obtained from our previous evaluation (Section 5.3) that characterized the pareto curve with enough policy diversity. We then select \textit{a configuration} from the same study that scores high on both performance and speed for most models.
% Specifically, we choose configurations that maximize the performance-to-speed (P/S) ratio across all tasks within each model. While this translation from 100-sample evaluation to full-scale inference is imperfect, it provides a good estimate and maintains consistency across all baselines.
\textbf{$\bullet$ Results interpretation.} Configurations selected using SpecMD perform well in most cases, but this is not the point. The point of SpecMD is the experimental platform which enables flexible configuration and unbiased study of the policies and their interactions. Using SpecMD, researchers can select optimal policy combinations based on their specific hardware constraints and quality requirements and share their findings more effectively.
\section{Conclusion}
This work presented SpecMD, a unified framework for evaluating MoE expert caching policies. Through GPU-based hardware emulation and minimal PyTorch integration, we enabled controlled comparison of caching strategies without specialized hardware requirements.
Our study challenges traditional caching assumptions for MoE architectures. We demonstrated that expert access follows deterministic sequential layer patterns rather than temporal locality, rendering conventional LRU/LFU approaches ineffective. The proposed Least-Stale eviction policy exploits this structure, achieving 3-9× collision miss reduction at practical capacity levels, translating as an average ~17\% improvement in TTFT, across different models and methods. Counterintuitively, we revealed that prediction accuracy does not guarantee cache performance—score-based prefetching delivers superior hit rates through dynamic bandwidth adaptation despite lower precision metrics.
\section*{Impact Statement}
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
\newpage
\appendix
\section{Implementation Details}
\label{app:implementation}
\subsection{Gate Hook Architecture}
Our framework intercepts MoE gate layer outputs through PyTorch forward hooks. When a gate layer executes, it outputs a 4-tuple: selected expert indices, routing weights for those experts, raw router logits, and normalized router weights across all experts. We wrap gate outputs to ensure consistent formatting across different MoE architectures (Mixtral, OLMoE, Qwen), providing a unified interface for downstream components. This standardization eliminates model-specific handling in policy implementations.
The hook manager registers forward hooks on each gate layer to intercept these outputs. Algorithm~\ref{alg:gate_hook} (main text) outlines the core hook logic. Upon receiving gate outputs, the routing policy can modify logits to prefer GPU-resident experts. When routing modifies expert selections, we apply dual-logit semantics: modified logits determine which experts to select, but original logits determine their contribution weights. This preserves model output integrity—even if cache-aware routing changes which experts execute, their contributions are weighted according to the original router decisions.
\subsection{Cache Manager Details}
For each selected expert during model execution, the cache manager checks GPU cache availability. Cache hits proceed immediately. Cache misses trigger the miss handling policy: fetch blocks until the expert loads from CPU, drop skips low-rank experts, or substitution replaces missing experts with cached alternatives. The cache manager maintains experts across CPU (pinned memory) and GPU (active cache) storage, supporting mixed-precision configurations where experts exist in multiple quantization levels (fp16, int8, int4, int2).
\subsection{Prefetch Manager and Watchdog Thread}
Concurrently, the prefetch manager anticipates future expert needs. The hook manager uses either current gate outputs or future gate functions on current gate logits to predict which experts subsequent layers will require, submitting prefetch requests to an asynchronous queue. A watchdog thread monitors this queue and GPU capacity, issuing non-blocking CUDA transfers when space allows. Prefetched experts populate the GPU cache before they are requested, reducing miss frequency. When capacity pressure exceeds limits, the eviction policy identifies victims—our Least-Stale policy prioritizes removing experts from already-processed layers while protecting experts needed for upcoming layers.
\subsection{Hardware Emulation}
Hardware constraints are emulated through software limits rather than requiring physical low-bandwidth devices. Bandwidth throttling delays expert transfers between CPU and GPU to simulate slower interconnects. Capacity limits restrict the available GPU cache below physical VRAM, forcing the system to make eviction and prefetch trade-offs. This emulation enables systematic exploration: evaluating a policy under 5 GB/s bandwidth and 25\% capacity requires only configuration changes, not specialized hardware.
\section{Baseline Eviction Policies}
\label{app:baselines}
We implement five baseline eviction strategies for comparison:
\textbf{LRU (Least Recently Used)} evicts the least recently accessed expert. However, MoE expert access follows deterministic sequential layer patterns rather than temporal patterns, making LRU ineffective for expert caching.
\textbf{LFU (Least Frequently Used)} evicts the expert with the lowest access count over the entire inference history. This policy prioritizes retaining frequently-accessed experts across all layers. However, it fails to account for layer-wise access patterns where experts from completed layers will not be needed until the next forward pass.
\textbf{LHU (Least High-precision Used)} \cite{Tang2024HOBBITAM} tracks high-precision expert usage frequency, prioritizing retention of frequently-accessed high-precision experts since cache misses for these incur significantly higher latency. This policy must be combined with fetch cascade precision policies (e.g., int8$\rightarrow$int4$\rightarrow$int2) that allow graceful degradation to lower precision on cache misses.
\textbf{FLD (Farthest Layer Distance)} \cite{Tang2024HOBBITAM} exploits layer-wise structure by retaining experts from nearer layers, which are more likely to be needed soon. This policy computes the distance between each cached expert's layer and the current inference layer, evicting experts from the farthest layers first.
\textbf{Score-based} \cite{Zhu2025EnablingMO} evicts experts with the lowest historical gate scores, using router activation values as importance signals. This policy maintains a running history of gate scores for each expert and evicts those with minimal contribution to model outputs. However, historical scores do not capture position in the forward pass, causing premature eviction of soon-needed experts.
\section{Complete Eviction Policy Results}
\label{app:eviction_results}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/q3b_collision_analysis_all.png}
\caption{Complete collision miss analysis across four eviction policies (FLD, SB, LRU, LS) at three capacity levels (1\%, 5\%, 25\%). Least-Stale (LS) achieves dramatically lower collision miss rates (red bars) compared to baseline policies across all models and capacity constraints. The pattern is consistent: LS achieves 10-34$\times$ lower collision rates than baseline policies.}
\label{fig:q3b_all_full}
\end{figure*}
Figure~\ref{fig:q3b_all_full} presents complete collision miss rate results referenced in Section 5.1, showing consistent performance across all capacity levels.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/q3b_collision_analysis_per_layer_all.png}
\caption{Per-layer collision miss analysis across all four models at three capacity levels (1\%, 5\%, 25\%). Least-Stale (red) maintains near-zero collisions by protecting experts in the current forward pass, while baseline policies show accumulating collisions throughout inference. The mechanism reveals why Least-Stale achieves dramatic improvements: it only evicts stale left-layer experts while baseline policies prematurely evict soon-needed experts.}
\label{fig:q3b_per_layer_all}
\end{figure*}
Figure~\ref{fig:q3b_per_layer_all} shows per-layer collision patterns for all models, revealing the mechanism behind Least-Stale's effectiveness across different architectures.
\section{Least-Stale Eviction as Drop-in Improvement}
\label{app:eviction_ablation}
Beyond our full system, we evaluate the Least-Stale eviction policy as a standalone contribution that can improve existing approaches. As visualized in \cref{fig:teaser} for OLMoE-1B-7B, Least-Stale provides consistent speedups across diverse baseline systems. Table~\ref{tab:eviction_ablation} shows comprehensive TTFT improvements across all four models when we replace each baseline's original eviction policy with Least-Stale while keeping their other components (prefetching, routing) unchanged.
\begin{table*}[h]
\centering
\small
\begin{tabular}{@{}llcc|c@{}}
\toprule
\textbf{Model} & \textbf{Approach} & \textbf{Original TTFT} & \textbf{w/ Least-Stale} & \textbf{Improvement} \\
& & \textbf{(ms)} & \textbf{(ms)} & \textbf{(\%)} \\
\midrule
\multicolumn{5}{l}{\textit{OLMoE-1B-7B}} \\
& Eliseev '23 & 3076.8 & 2748.1 & 10.7\% \\
& Zhu et al. '25 & 2257.2 & 1473.1 & 34.7\% \\
& HOBBIT '24 & 1590.1 & 1309.2 & 17.7\% \\
& MixtureCache '25 & 2664.6 & 2297.5 & 13.8\% \\
\midrule
\multicolumn{5}{l}{\textit{Mixtral-8x7B}} \\
& Eliseev '23 & 4238.0 & 3485.2 & 17.8\% \\
& Zhu et al. '25 & 1620.3 & 1460.8 & 9.8\% \\
& HOBBIT '24 & 2321.7 & 1725.2 & 25.7\% \\
& MixtureCache '25 & 4332.6 & 3573.2 & 17.5\% \\
\midrule
\multicolumn{5}{l}{\textit{Phi-3.5-MoE}} \\
& Eliseev '23 & 4451.7 & 3791.1 & 14.8\% \\
& Zhu et al. '25 & 1905.6 & 1987.6 & -4.3\% \\
& HOBBIT '24 & 3458.6 & 1933.6 & 44.1\% \\
& MixtureCache '25 & 5025.3 & 3791.1 & 24.6\% \\
\midrule
\multicolumn{5}{l}{\textit{Qwen1.5-MoE-A2.7B}} \\
& Eliseev '23 & 3724.0 & 3433.4 & 7.8\% \\
& Zhu et al. '25 & 2739.2 & 2620.9 & 4.3\% \\
& HOBBIT '24 & 2492.0 & 2033.7 & 18.4\% \\
& MixtureCache '25 & 3488.6 & 2685.4 & 23.0\% \\
\bottomrule
\end{tabular}
\caption{Least-Stale eviction policy as drop-in improvement for existing approaches. We replace each baseline's original eviction policy with Least-Stale while keeping their prefetching and routing strategies unchanged. Positive percentages indicate TTFT reduction (lower is better). This demonstrates Least-Stale's generalizability across different system architectures.}
\label{tab:eviction_ablation}
\vspace{-10pt}
\end{table*}
Least-Stale eviction provides consistent TTFT improvements across most configurations (10-44\% reduction), demonstrating its value as a portable component. However, SpecMD's advantage stems from the synergistic combination of eviction, prefetching, and cache-aware routing policies.
\section{Model Specifications}
\label{app:model_specs}
Table~\ref{tab:model_specs} provides detailed specifications for all models evaluated in this work.
\begin{table*}[h]
\centering
\caption{Detailed specifications for evaluated MoE architectures. All sizes in GB unless specified. Activated size represents memory required per forward pass.}
\label{tab:model_specs}
\resizebox{\textwidth}{!}{
\begin{tabular}{lcccccccc}
\toprule
\textbf{Model} & \textbf{Layers} & \textbf{Experts} & \textbf{Top-k} & \textbf{Expert Size} & \textbf{Total Size} & \textbf{Expert Params} & \textbf{Non-Expert} & \textbf{Activated} \\
& & \textbf{per Layer} & & & & & & \\
\midrule
OLMoE-1B-7B & 16 & 64 & 8 & 12 MB & 12.89 GB & 12.0 GB & 0.89 GB & 2.39 GB \\
Mixtral-8x7B & 32 & 8 & 2 & 336 MB & 86.99 GB & 84.0 GB & 2.99 GB & 23.99 GB \\
Qwen1.5-MoE-A2.7B & 24 & 60 & 4 & 16.5 MB & 26.67 GB & 23.2 GB & 3.46 GB & 5.01 GB \\
Phi-3.5-MoE & 32 & 16 & 2 & 150 MB & 77.99 GB & 75.0 GB & 2.99 GB & 12.37 GB \\
\bottomrule
\end{tabular}
}
\end{table*}
These specifications highlight the diversity of architectural choices: OLMoE uses many small experts with high top-k (fine-grained sparsity), while Mixtral uses few large experts with low top-k (coarse-grained sparsity). This diversity enables us to characterize policy effectiveness across different granularity regimes.
% \begin{table*}[t]
% \centering
% \caption{Policy space comparison for MoE expert caching. Prior work explores 1-2 policy combinations in isolation. \textsuperscript{\dag}Modifies routing decisions to prefer cached experts.}
% \small
% \begin{tabular}{@{}lcccc@{}}
% \toprule
% \textbf{Paper} & \textbf{Prefetch} & \textbf{Eviction} & \textbf{Miss Handling} & \textbf{Routing}\\
% \midrule
% \citet{Eliseev2023FastIO} & Speculative & LRU & Fetch & Standard\\
% \citet{Tang2024HOBBITAM} & Speculative & LHU & Fetch & Standard\\
% \citet{Zhu2025EnablingMO} & Score-based & Score-based & Substitution & Standard\\
% \citet{skliar2025mixturecacheconditionalexpertsefficient} & --- & LRU & Fetch & Modified\textsuperscript{\dag}\\
% \bottomrule
% \end{tabular}
% \label{tab:related_policies}
% \vspace{-10pt}
% \end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%