Online Linear Programming for Multi-Objective Routing in LLM Serving
Extracted source text · arXiv 2607.03948
%%%%%%%% ICML 2026 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2026} with \usepackage[nohyperref]{icml2026} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\newcommand{\pos}[1]{\bigl(#1\bigr)_+}
% Use the following line for the initial blind version submitted for review:
%\usepackage{icml2026}
% For preprint, use
% \usepackage[preprint]{icml2026}
% If accepted, instead use the following line for the camera-ready submission:
\usepackage[accepted]{icml2026}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\newcommand{\ind}{\mathbf{1}}
\newcommand{\xhdr}[1]{\vspace{1mm} \noindent{\bf #1}}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Online Linear Programming for Multi-Objective Routing in LLM Serving}
\begin{document}
\twocolumn[
\icmltitle{Online Linear Programming for Multi-Objective Routing in LLM Serving}
% It is OKAY to include author information, even for blind submissions: the
% style file will automatically remove it for you unless you've provided
% the [accepted] option to the icml2026 package.
% List of affiliations: The first argument should be a (short) identifier you
% will use later to specify author affiliations Academic affiliations
% should list Department, University, City, Region, Country Industry
% affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order. Ideally, you
% should not use this facility. Affiliations will be numbered in order of
% appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Zixi Chen}{comp}
\icmlauthor{Yinyu Ye}{sch}
\icmlauthor{Zijie Zhou}{sch}
\end{icmlauthorlist}
\icmlaffiliation{comp}{School of Mathematical Sciences, Peking University}
\icmlaffiliation{sch}{Department of Industrial Engineering and Decision Analytics, HKUST}
\icmlcorrespondingauthor{Zijie Zhou}{jerryzhou@ust.hk}
% You may provide any keywords that you find helpful for describing your
% paper; these are used to populate the "keywords" metadata in the PDF but
% will not be shown in the document
\icmlkeywords{LLM Serving, Online Linear Programming}
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column listing the
% affiliations and the copyright notice. The command takes one argument, which
% is text to display at the start of the footnote. The \icmlEqualContribution
% command is standard text for equal contribution. Remove it (just {}) if you
% do not need this facility.
% Use ONE of the following lines. DO NOT remove the command.
% If you have no special notice, KEEP empty braces:
\printAffiliationsAndNotice{} % no special notice (required even if empty)
% Or, if applicable, use the standard equal contribution text:
% \printAffiliationsAndNotice{\icmlEqualContribution}
\begin{abstract}
We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-level objectives (SLOs) and offer limited control over latency–throughput trade-offs, we introduce a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards. We apply an efficient bid-price control policy based on the online linear programming that admits requests when their SLO-weighted benefit exceeds their shadow prices. To meet millisecond decision requirements, we develop a warm-started, projected first-order updates that track the evolving dual shadow prices online with predictable runtime. We integrate our router into the Vidur simulator and demonstrate substantial improvements over standard baselines across multiple SLO regimes, including end-to-end latency, time-to-first-token, throughput, and tail performance. A big picture from our result: a science-based approach outperforms others based on heuristics.
\end{abstract}
\input{Intro}
%\input{Related}
\input{Model}
\input{Alg}
\input{Exp}
\input{Discussion}
\input{Limitation}
%\newpage
%\input{Theory}
\iffalse
\section*{Accessibility}
Authors are kindly asked to make their submissions as accessible as possible
for everyone including people with disabilities and sensory or neurological
differences. Tips of how to achieve this and what to pay attention to will be
provided on the conference website \url{http://icml.cc/}.
\section*{Software and Data}
If a paper is accepted, we strongly encourage the publication of software and
data with the camera-ready version of the paper whenever appropriate. This can
be done by including a URL in the camera-ready copy. However, \textbf{do not}
include URLs that reveal your institution or identity in your submission for
review. Instead, provide an anonymous URL or upload the material as
``Supplementary Material'' into the OpenReview reviewing system. Note that
reviewers are not required to look at this material when writing their review.
% Acknowledgements should only appear in the accepted version.
\section*{Acknowledgements}
\textbf{Do not} include acknowledgements in the initial version of the paper
submitted for blind review.
If a paper is accepted, the final camera-ready version can (and usually should)
include acknowledgements. Such acknowledgements should be placed at the end of
the section, in an unnumbered section that does not count towards the paper
page limit. Typically, this will include thanks to reviewers who gave useful
comments, to colleagues who contributed to the ideas, and to funding agencies
and corporate sponsors that provided financial support.
\fi
\section*{Impact Statement}
From a methodological perspective, our work establishes the first multi-objective optimization framework for LLM request routing, formulating the problem as online linear programming with time-coupled capacity constraints. This provides a principled bridge between the operations research community---where online resource allocation and bid-price control are well-studied---and the LLM serving systems community, where routing decisions have largely relied on heuristics designed for classical settings. Beyond any specific algorithmic improvements, the framework offers interpretable tools for systematic policy design: objective weights that transparently control SLO trade-offs, and shadow prices that diagnose capacity bottlenecks. We hope this work demonstrates the value of bringing optimization methodology to emerging LLM systems challenges and provides a foundation for future research at this intersection.
\iffalse
Large language model (LLM) serving has rapidly become a major component of modern computing infrastructure, operating at huge request volumes and substantial GPU cost. By explicitly optimizing routing decisions under realistic decode-side constraints, our approach improves system efficiency and service quality in an explainable way that can translate directly into economic benefits at scale. Even modest reductions in latency or improvements in throughput can meaningfully decrease required GPU capacity, lower operational expenditure for large deployments, while simultaneously improving user experience and retention through faster and more predictable response times.
From a methodological perspective, our work introduces the \textbf{first} optimization view of multi-objective LLM serving. Our work is the first attempt of framing multi-objective LLM routing as an online linear programming problem. The underline LP problems in our setting represents an instance of online resource allocation with time-coupled capacity constraints. We provide a principled online LP framework that directly controls and trades off throughput, TTFT, end-to-end latency, and tail objectives. This establishes a concrete bridge between online resource allocation theory community and practical LLM serving system community, enabling systematic policy design, transparent objective weighting, and a foundation for future work on efficient multi-metric control in LLM inference.
\fi
%To the best of our knowledge, this is the first attempt of using online LP theory and algorithm for LLM inference. The underline LP problems in our setting also represent more complicated and realistic resource allocations compared to those studied in classical online LP literature.
% In the unusual situation where you want a paper to appear in the
% references without citing it in the main text, use \nocite
\nocite{langley00}
\bibliography{example_paper}
\bibliographystyle{icml2026}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\onecolumn
\input{AppendixModel}
\input{AppendAlg}
\input{ExpAppend}
\end{document}
% This document was modified from the file originally made available by
% Pat Langley and Andrea Danyluk for ICML-2K. This version was created
% by Iain Murray in 2018, and modified by Alexandre Bouchard in
% 2019 and 2021 and by Csaba Szepesvari, Gang Niu and Sivan Sabato in 2022.
% Modified again in 2023 and 2024 by Sivan Sabato and Jonathan Scarlett.
% Previous contributors include Dan Roy, Lise Getoor and Tobias
% Scheffer, which was slightly modified from the 2010 version by
% Thorsten Joachims & Johannes Fuernkranz, slightly modified from the
% 2009 version by Kiri Wagstaff and Sam Roweis's 2008 version, which is
% slightly modified from Prasad Tadepalli's 2007 version which is a
% lightly changed version of the previous year's version by Andrew
% Moore, which was in turn edited from those of Kristian Kersting and
% Codrina Lauth. Alex Smola contributed to the algorithmic style files.
\section{Introduction} \label{sec:intro}
The demand for large language model (LLM) services has surged dramatically in recent years, with billions of requests being processed daily \cite{chatgptdemand}. As this demand continues to grow, optimizing the efficiency of LLM serving systems has become an increasingly critical challenge \cite{kwon2023efficient, pope2023efficiently, sheng2023flexgen, agrawal2024taming}. LLM serving consists of two key stages for each request: (i) the prefill stage, where the key-value (KV) pairs of the input are generated, and (ii) the decode stage, where the model autoregressively generates output tokens. Due to the massive volume of requests and limited computational resources, it is impossible to process all requests simultaneously. In this paper, we focus on optimizing the \emph{router}, which dispatches incoming requests to parallel computational workers \cite{jain2024intelligent,wu2025efficient,mei2025omnirouter}, and we cast this control problem as a general online resource allocation task with time-coupled constraints, specialized to decode-side LLM serving.
Optimizing the router requires specifying \emph{what} performance the system should optimize, yet real LLM services operate under heterogeneous service-level objectives (SLOs). For instance, real-time voice translation and interactive agents prioritize responsiveness, often measured by time-to-first-token, whereas chatbots emphasize end-to-end latency because users care about the time to receive a complete response; meanwhile, system operators also track throughput as a proxy for capacity and cost efficiency. These objectives are intrinsically in tension: increasing throughput typically relies on larger or more aggressive batching and higher device utilization, which raises queueing delays and degrades latencies, while prioritizing low latency often forces smaller batches or reserved capacity, reducing overall throughput and utilization. Because different products require different metric combinations and trade-offs, serving stacks frequently resort to retuning or even redesigning routing heuristics per deployment, motivating a general multi-objective control framework that can flexibly adapt routing behavior by adjusting objective weights rather than changing the router’s structure.
Many production serving stacks (e.g., vLLM \cite{kwon2023efficient}) and simulators (e.g., Vidur \cite{agrawal2024vidur}) adopt simple routing heuristics that fall into two broad families: \emph{greedy load-based} rules and \emph{randomized} rules. Representative greedy policies include join-shortest-queue and power-of-$d$ choices. While effective in classical parallel-server settings, these rules were designed for environments where job sizes are either known or homogeneous. In LLM serving, however, the relevant workload is driven by heterogeneous and uncertain decode lengths: two queues with the same number of requests can correspond to vastly different remaining work. Recent work has explored decode-length prediction \cite{fu2024efficient, zheng2023response, qiu2024efficient, shahout2024don}, workload-aware scheduling \cite{jain2024intelligent}, and multi-LLM routing \cite{wu2025efficient, mei2025omnirouter}, yet most deployed systems still rely on heuristics that provide limited control over SLO trade-offs. Randomized baselines such as round-robin and uniform routing avoid relying on imperfect workload estimates, but they ignore the instantaneous system state and cannot adapt to congestion. More fundamentally, neither family is derived from an explicit optimization objective, providing limited and non-transparent control over trade-offs among competing SLOs---precisely the flexibility required for multi-objective serving.
The challenge of dispatching requests to capacity-constrained resources under uncertainty is a classical problem in operations research, studied extensively under the framework of online linear programming (LP) \cite{agrawal2014dynamic, li2020simple, li2022online} and bid-price control in revenue management \cite{talluri1998analysis, akan2009bid}. A key insight from this literature is that good online decisions should balance the \emph{immediate benefit} of serving a request against the \emph{opportunity cost} of consuming scarce capacity that may be needed for future arrivals. LP \cite{luenberger1984linear} duality provides a principled way to quantify these opportunity costs through \emph{shadow prices}. In LLM serving, the scarce resources are batch slots and KV-cache memory on each device across future time steps---resources that are \emph{time-coupled} because a request admitted now occupies capacity throughout its decode duration. Beyond constraints, the \emph{objectives} themselves require careful design: aggregate SLOs such as tail latency are statistical properties that do not directly decompose into per-request rewards. A key modeling contribution of our work is a reward structure that converts these aggregate metrics into per-request incentives while ensuring that scheduling is usually beneficial. This combination of time-coupled constraints and SLO-decomposable objectives motivates our online LP framework.
% === Paragraph 5: Gap Statement + Contributions ===
To our knowledge, no prior work provides a multi-objective optimization framework for LLM routing that (i) explicitly models time-coupled KV-cache and batch-size constraints inherent to autoregressive decoding, (ii) decomposes aggregate SLOs---including tail latency---into per-request rewards suitable for online optimization, and (iii) derives computationally efficient policies grounded in LP duality. Our work bridges this gap by bringing operations research methodology to LLM serving. Specifically, we make the following contributions:
\begin{enumerate}
\item \textbf{Multi-objective optimization framework.} We formulate LLM routing as an online linear program with time-coupled batch and KV-cache constraints. The objective function explicitly incorporates multiple SLOs---throughput, end-to-end latency, time-to-first-token, and tail latency---through interpretable weights that operators can adjust to match product requirements.
\item \textbf{Efficient bid-price control algorithm.} We derive a bid-price policy where each routing decision compares the request's SLO-weighted benefit against its shadow-price cost. To meet millisecond decision latency requirements, we develop warm-started projected gradient updates that track evolving dual prices online without solving an LP at each step.
\item \textbf{Interpretable trade-off control.} Through the LP formulation, operators gain transparent control over SLO trade-offs: adjusting objective weights systematically shifts performance along the Pareto frontier, and shadow prices reveal which resources (batch slots vs.\ KV-cache) are bottlenecks under different workloads.
\end{enumerate}
We validate our framework through extensive simulation experiments using the Vidur simulator \cite{agrawal2024vidur}, demonstrating substantial improvements over standard routing heuristics across multiple SLO regimes. While full integration into production serving systems remains future work, our results establish the potential of optimization-based routing and provide a principled foundation for the systems community. We also discuss the structural insights that emerge from viewing LLM routing through an optimization lens---insights that may guide future system design even beyond our specific algorithm.
\iffalse
Our key contribution is to formulate routing as an instance of online resource allocation with time-coupled constraints and to derive a fast, interpretable policy from linear programming (LP). Linear programming is a scientific methodology for allocating scarce resources under capacity constraints \cite{luenberger1984linear,dantzig2002linear}, and in our setting the scarce resources are the per-device batch slots and KV-cache memory across future time steps. The dual variables of this LP admit a natural systems interpretation as \emph{shadow prices}, which quantify the marginal value of one additional unit of batch capacity (resp., KV memory) on each device, given current and anticipated congestion. Using these shadow prices, each candidate routing action is scored by a simple bid-price control rule that is cheaper than solving the primal LP and is naturally suited to online, sequential arrivals \cite{agrawal2014dynamic,li2020simple,li2022online}. Because decisions must be made within milliseconds, we do \textbf{not} solve an LP at any step; instead, we exploit the fact that consecutive per-step LPs differ only incrementally and update prices via warm-started first-order methods, producing predictable runtime while tracking near-optimal dual solutions. Finally, the LP formulation makes multi-objective control explicit and interpretable: by adjusting objective weights associated with different SLOs, the same routing mechanism can be reconfigured across products without redesigning the underlying policy, providing a general framework for multi-objective LLM serving.
\fi
\section{Mathematical Model}
\label{sec:model}
In this section, we present a mathematical model for multi-worker LLM serving under \emph{Prefill/Decode (PD) disaggregation} \cite{zhong2024distserve} and \emph{continuous batching}. A discussion of how the model and algorithms are adapted under PD mixing is deferred to Appendix~\ref{append:model}.
\paragraph{Time, devices, and requests.}
We consider a finite horizon of discrete periods $k=1,\dots,T$ and a set of decode devices indexed by $g\in[G]$. One period corresponds to one decode iteration of a batched decode kernel on a device.
An instance of requests $\mathcal J$ arrives sequentially over the horizon. Each request $j\in\mathcal J$ completes its prefill stage and arrives at time $t_j\in\{1,\dots,T\}$ with:
(i) a known prefill footprint $s_j$ (e.g., the number of KV-cache token slots already materialized by prefill), and
(ii) an unknown decode length $o_j$ (the number of decode tokens required to complete the response).
In practice, $o_j$ can be predicted using established techniques (e.g., \cite{zheng2023response,qiu2024efficient,fu2024efficient,chen2024kvdirect, shahout2024don}); we denote the predicted decode length by $\hat o_j$, which may be inaccurate.
\paragraph{Router-side buffering.}
After completing prefill, each request $j$ enters a \emph{central router queue} at time $t_j$.
%The router operates in discrete periods $k=1,\dots,T$, where each period coincides with one decode iteration on each device.
At the beginning of each period $k$, the router observes the current state of each decode device, and may \emph{release} a subset of queued requests to devices. This design is supported by recent versions of the vLLM engine, which implement router-side buffering \cite{vllmDataParallelDeploymentDocs_internalLB,vllmProductionStackRoadmap2025Q2_issue300}.
In this paper, we focus on a \emph{release-when-startable} design: the router dispatches a request to a device only if the request can be admitted into that device's active decode batch in the same period.
In implementation, one may still maintain short per-device FIFO queues, but under our policy these queues remain small.
\paragraph{Routing and admission decisions.}
For each request $j\in\mathcal J$, device $g\in[G]$, and period $\tau\in\{t_j,\dots,T\}$, we define a \textbf{routing/admission decision variable}
\[
x_{j,g,\tau}\in[0,1],
\]
interpreted as: ``request $j$ is released from the router queue and admitted to device $g$ at the beginning of period $\tau$.''
In an integral schedule, $x_{j,g,\tau}\in\{0,1\}$ and at most one pair $(g,\tau)$ is selected for each request: $\sum_{g\in[G]}\sum_{\tau=t_j}^{T} x_{j,g,\tau}\ \le\ 1, \forall j\in\mathcal J$.
\paragraph{Continuous batching within a decode device.}
Within each decode device, we model the classical continuous batching mechanism used by vLLM \cite{kwon2023efficient}.
At each period, device $g$ executes one batched decode iteration that processes \emph{one token} for every \emph{active} request currently in its batch.
A request $j$ requires $o_j$ such iterations to complete.
As soon as an active request finishes, the device can admit new requests into the batch in subsequent iterations, thereby maintaining a continuously updated batch (see Figure~\ref{fig:1}).
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{Figure1.jpg}
\caption{Illustration of Prefill/Decode (P/D) disaggregation with continuous batching. Each small box denotes a token, and colors indicate distinct requests. Within each decode worker, active requests are processed token-by-token in a continuously updated batch as completed requests are replaced by newly admitted ones.}
\label{fig:1}
\end{figure}
\paragraph{KV-cache memory accounting.}
During decoding, the KV-cache footprint of request $j$ increases by (approximately) one token slot per decoded token. We represent the KV-cache usage of request $j$ after $\theta$ decoded tokens by the kernel $m^{\text{mem}}_j(\theta)$:
\begin{equation} \label{eq:mem}
m^{\text{mem}}_j(\theta)=
\begin{cases}
s_j+\theta+1, & 0\le \theta<o_j,\\
0, & \text{otherwise},
\end{cases}
\end{equation}
where $\theta$ is the number of decode tokens already generated for request $j$. (Equivalently, while $j$ is active, its memory footprint equals its prefill footprint plus its current decoded length.)
\paragraph{Device feasibility constraints.}
A device cannot batch arbitrarily many requests due to two common constraints:
\begin{enumerate}
\item[\textbf{(i)}] \textit{Batch-size constraint:} at any period, each decode device can host at most $B$ active requests in the batch.
\item[\textbf{(ii)}] \textit{KV-cache capacity:} at any period, the total KV-cache usage of all active requests on a decode device cannot exceed a budget $M$; otherwise, the system must swap requests out and stall compute. In our model, we enforce routing/scheduling decisions that respect this capacity.
\end{enumerate}
Here, treating one decode iteration as one time period is reasonable since under PD disaggregation, decode workers process only decode iterations, eliminating the primary source of iteration-time variability identified in non-disaggregated systems. In a decode-only regime, each iteration is memory-bandwidth-bound: the dominant cost is loading model weights, which is constant regardless of batch composition. The attention component grows linearly with the total cached KV tokens, but profiling reported by \citet{jain2024intelligent} shows this slope is roughly $10\times$ smaller than the corresponding prefill slope, indicating that decode iteration time varies slowly with batch state.
\xhdr{Objective Metrics.}
LLM serving is naturally multi-objective. We consider the following standard performance metrics.
\paragraph{Service-level metrics.}
\begin{enumerate}
\item[\textbf{(1)}] \textit{Average/Tail end-to-end latency (EEL):}
Let $c_j$ denote the period when request $j$ completes (i.e., its final token is processed).
The end-to-end latency of request $j$ is $c_j-t_j$.
The average EEL is
\[
\frac{1}{|\mathcal J|}\sum_{j\in\mathcal J}(c_j-t_j).
\]
For a tail level $x\%$, the $x\%$ tail EEL is the $x\%$ quantile of the multiset
$\{\,c_j-t_j : j\in\mathcal J\,\}$.
\item[\textbf{(2)}] \textit{Average/Tail time-to-first-token (TTFT):}
Let $\tau_j$ denote the period when request $j$ starts decoding, i.e., when its \emph{first} output token is generated.
The TTFT of request $j$ is $\tau_j-t_j$.
The average TTFT is
\[
\frac{1}{|\mathcal J|}\sum_{j\in\mathcal J}(\tau_j-t_j),
\]
and the $x\%$ tail TTFT is the $x\%$ quantile of
$\{\,\tau_j-t_j : j\in\mathcal J\,\}$.
\end{enumerate}
\begin{remark}
%In production deployments, both EEL and TTFT include time spent in \emph{prefill} and \emph{decode}. In our PD-disaggregated setting, requests enter the router \emph{after} prefill has completed; hence the prefill component is exogenous to our routing decisions and is omitted from the latency definitions above.
We do not explicitly report \emph{time-per-output-token} (TPOT) since TPOT can be recovered as end-to-end latency minus TTFT and normalized by $o_j$.
\end{remark}
\paragraph{System-level metrics.}
\begin{enumerate}
\item[\textbf{(3)}] \textit{Throughput:}
Let $N^{\text{tok}}(T)$ be the total number of decode tokens processed over periods $1,\dots,T$ across all devices.
The (average) token throughput over the horizon is
\[
N^{\text{tok}}(T)/T.
\]
\item[\textbf{(4)}] \textit{Queries-per-second (QPS):}
Let $N^{\text{req}}(T)$ be the number of requests that complete by time $T$.
The (average) QPS over the horizon is
\[
N^{\text{req}}(T)/T.
\]
\end{enumerate}
\section{Methodology and Algorithm Design}
\label{sec:alg}
In this section, we present an online linear programming-based algorithm designed to optimize multiple objectives in the LLM serving process.
\subsection{Online Linear Programming Formulation}
\label{subsec:lp}
At each decoding step $k \in \{1, 2, \dots, T\}$, we define %the set of requests whose decode tokens are actively being processed as $\mathcal{J}^{(cont)}_k$, and
the set of requests that have not yet started decoding (i.e., they are waiting in the queue) as $\mathcal{J}^{(wait)}_k$. The activity indicator for decoding steps is defined as $a_j(\theta) = \ind\{0 \le \theta < \hat o_j\}$,
where $\theta$ represents the number of decoding steps processed by request $j$ since its start time. If a request starts at time $\tau$, then $a_j(k - \tau)$ indicates whether the request is currently being processed at time $k$ (taking values 1 or 0 depending on whether it is active and within the batch size limit). The memory usage of the request follows the kernel $m^{\text{mem}}_j(\theta)$ defined in \eqref{eq:mem}.
Given that each worker is subject to both a batch size constraint and a memory usage constraint, we define a constraint-type index $u \in \{\textsf{bat}, \textsf{mem}\}$, where:
\begin{itemize}
\item $\textsf{bat}$ represents the per-period batch size constraint;
\item $\textsf{mem}$ represents the per-period memory usage constraint.
\end{itemize}
We define the set of device-time-constraint coordinates as
\[
\mathcal{I} := [G] \times [T] \times \{\textsf{bat}, \textsf{mem}\}, \quad i = (g, k, u) \in \mathcal{I}.
\]
For each device-time-constraint coordinate $i = (g, k, u) \in \mathcal{I}$ and request $j$, the scheduling action column coefficient is defined as
\[
a_{i, (j, g', \tau)} =
\begin{cases}
a_j(k - \tau), & \text{if } u = \textsf{bat}, ~g' = g, \\
m^{\text{mem}}_j(k - \tau), & \text{if } u = \textsf{mem}, ~g' = g, \\
0, & \text{otherwise}.
\end{cases}
\]
For each device-time-constraint coordinate $i = (g, k, u)$, we set
\[
b_{(g, k, \textsf{bat})} = B_{g, k}, \quad b_{(g, k, \textsf{mem})} = M_{g, k},
\]
where $B_{g, k}$ represents the remaining batch size capacity (i.e., the difference between the total batch size limit $B$ and the number of active requests) of device $g$ at time $k$, and $M_{g, k}$ represents the remaining KV memory usage capacity of device $g$ at time $k$. At decision step $k \in [T]$, only requests that have not yet started (i.e., those in $\mathcal{J}^{(wait)}_k$) can be scheduled. We define the action (column) set as
\[
\mathcal{A}_k := \{(j, g, k) : j \in \mathcal{J}^{(wait)}_k, g \in [G]\}.
\]
The system-level constraints for each device $g$ at time $k$ can be written as
\[
\sum_{(j, g, k) \in \mathcal{A}_k} a_{i, (j, g, k)} x_{j, g, k} \le b_i, \quad \forall i = (g, k, u) \in \mathcal{I}.
\]
At step $k$, we make decisions only for the waiting requests $j \in \mathcal{J}^{(wait)}_k$ about which device and time to start processing. Although $x_{j, g, k}$ represents a one-time start decision, it induces resource usage across multiple future steps $t \ge k$ through the age-dependent kernels: $a_j(t - k)$ for batch occupancy and $m^{\text{mem}}_j(t - k)$ for KV memory usage. A request automatically stops contributing to the resource constraints once it finishes decoding, as $a_j(\theta) = 0$ and $m^{\text{mem}}_j(\theta) = 0$ when $\theta \ge \hat o_j$. The right-hand side $b_i$ represents the remaining capacity after subtracting the load of previously started requests, ensuring that newly admitted requests do not violate batch size or memory constraints on any device at any future decoding step.
\paragraph{Linear Multiple Objective and Decision Reward}
To formulate this problem as a linear program, we assign a reward to each decision $(j, g, k)$, corresponding to assigning request $j$ to device $g$ at time $k$. The reward function is given by
\begin{align*}
r_{j, g, k} \;&= \; \underbrace{\alpha \cdot \texttt{Thr}_{j, g, k}}_{\text{throughput}} - \underbrace{\beta \cdot \texttt{EEL}_{j, g, k}}_{\text{average end-to-end latency}} \\
&\quad - \underbrace{\gamma \cdot \texttt{TTFT}_{j, g, k}}_{\text{average TTFT}} + \underbrace{\sigma \cdot \texttt{QPS}_{j, g, k}}_{\text{queries per second}} \\
&\quad - \underbrace{\zeta_1 \cdot \texttt{Tail EEL}_{j, g, k}}_{\text{tail end-to-end latency}} - \underbrace{\zeta_2 \cdot \texttt{Tail TTFT}_{j, g, k}}_{\text{tail TTFT}},
\end{align*}
where $\alpha, \beta, \gamma, \sigma, \zeta_1, \zeta_2 \ge 0$ are parameters representing the relative importance of each objective. These parameters should be selected according to the specific product and its purpose, as different products may prioritize different metrics.
Next, we decompose each term in the reward to better understand how the metrics contribute to the overall decision.
For system-level metrics, the throughput contribution of an action is given by the number of tokens processed during the time horizon $[0, T]$. If a request is assigned, its true contribution to throughput is the minimum of its predicted decode length and the remaining time steps: $\texttt{Thr}_{j, g, k} = \min\{\hat o_j, T - k + 1\}$.
For QPS, the contribution of an action is simply 1, as each assigned request contributes one processed query:
$\texttt{QPS}_{j, g, k} = 1$.
For service-level metrics, consider the contribution of each action to the average latency. Over the time horizon $[0, T]$, we have:
\[
\texttt{EEL}_{j, g, k} = \min\{k + \hat o_j, T + 1\} - t_j, \quad \texttt{TTFT}_{j, g, k} = k - t_j.
\]
However, although we seek to minimize latency, we \textbf{cannot allow it to become negative}. If we provide negative penalties for latency, the router may choose not to assign requests in order to avoid incurring a penalty, especially as the time horizon $T$ approaches its end. To avoid this issue, we observe that both end-to-end latency and TTFT can be written as:
\begin{equation}\label{eq:el1}
\sum_{s=t_j}^{T} \ind\{\text{not finished by }s\} = (T - t_j + 1) - \pos{T - (k + \hat o_j - 1)},
\end{equation}
and
\begin{equation} \label{eq:ttft1}
\sum_{s=t_j}^{T} \ind\{\text{not started by }s\} = (T - t_j + 1) - (T - k + 1),
\end{equation}
respectively. In both \eqref{eq:el1} and \eqref{eq:ttft1}, the term $(T - t_j + 1)$ is independent of the decision process, and we can eliminate this term. The only penalty related to the decision is $-\pos{T - (k + \hat o_j - 1)}$ and $-(T - k + 1)$, which makes the penalties negative and the reward positive.
We also introduce an interesting and simple method to decompose statistical tail latency into an instant reward for the action. Taking tail TTFT as an example, suppose the service-level constraint is that the tail TTFT must be bounded by $t'$. We define an indicator reward as
\[
\texttt{Tail TTFT}_{j, g, k} = \ind\{k - t_j < t'\}.
\]
This guarantees that if the TTFT of a request is within the desired bound $t'$, a reward of 1 unit is added. To enforce a stricter tail TTFT requirement (e.g., the 99th percentile), we set a larger weight for $\zeta_2$, which increases the weight on tail TTFT in the reward and drives decisions to prioritize it. Thus, the weights associated with the tail latency metrics, $\zeta_1$ and $\zeta_2$, not only determine the relative importance of tail latency in comparison to other objectives, but also directly map the target quantiles for the tail latency. A detailed guideline on how to select these parameters will be provided in Section \ref{sec:exp}.
Finally, the overall positive reward for action $(j, g, k)$ is written as:
\begin{align*}
r_{j, g, k} \;&= \; \underbrace{\alpha \cdot \min\{\hat o_j, T - k + 1\}}_{\text{throughput}} + \underbrace{\beta \cdot \pos{T - (k + \hat o_j - 1)}}_{\text{end-to-end latency "saved steps"}} \\
&\quad + \underbrace{\gamma \cdot (T - k + 1)}_{\text{first-token "saved steps"}} + \underbrace{\sigma}_{\text{queries per second}} \\
&\quad + \underbrace{\zeta_1 \cdot \ind\{k + \hat o_j - t_j < t_1'\}}_{\text{indicator reward for tail EEL}} + \underbrace{\zeta_2 \cdot \ind\{k - t_j < t_2'\}}_{\text{indicator reward for tail TTFT}}.
\end{align*}
The corresponding online linear programming formulation at time $k$ is:
\begin{align}
\max_{x} \quad
& \sum_{(j, g, k) \in \mathcal{A}_k} r_{j, g, k} x_{j, g, k} \nonumber \\
\text{s.t.} \quad
& \sum_{(j, g, k) \in \mathcal{A}_k} a_{i, (j, g, k)} x_{j, g, k} \le b_i, \quad \forall i \in \mathcal{I}, \label{eq:pack} \\
& \sum_{g \in [G]} x_{j, g, k} \le 1, \quad \forall j \in \mathcal{J}^{(wait)}_k. \label{eq:one}
\end{align}
\subsection{Duality, Bid-Price Control, and Stochastic Program}
\label{subsec:dual}
The per-step LP provides a clean abstraction of the routing decision at time $k$.
However, in an LLM serving system the decision loop must operate at millisecond scale, so repeatedly solving an LP to optimality at every step is impractical.
We therefore leverage LP duality to derive a \emph{bid-price control} rule \cite{talluri1998analysis,akan2009bid,agrawal2014dynamic,li2022online}: instead of re-optimizing from scratch, we update the dual prices (resource shadow costs) and make decisions by comparing each request's reward against its implied resource cost.
%Because rewards and resource profiles are constructed from predictions (e.g., $\hat o_j$) and resources are reusable over time, we additionally include a feasibility-preserving selection step that greedily admits high bid-price actions without violating residual constraints.
\paragraph{Dual formulation and bid prices.}
Introduce dual variables $p_i\ge 0$ for the packing constraints~\eqref{eq:pack} and $y_j\ge 0$ for the per-request constraint~\eqref{eq:one}.
For the per-step LP at time $k$, the standard dual is
\begin{align*}
\min_{p\ge 0,\,y\ge 0}\quad
& \sum_{i\in\mathcal{I}} b_i\,p_i \;+\; \sum_{j\in\mathcal{J}^{(wait)}_k} y_j
\\
\text{s.t.}\quad
& \sum_{i\in\mathcal I} a_{i,(j,g,k)}\,p_i \;+\; y_j \;\ge\; r_{j,g,k},
\ \forall (j,g,k)\in\mathcal{A}_k.
\end{align*}
The dual variables admit concrete operational meaning. For each resource-time coordinate $i = (g, k, u) \in \mathcal{I}$, the dual variable $p_i$ represents the \emph{shadow price} of resource type $u$ on device $g$ at time $k$: it quantifies the marginal value of one additional unit of batch capacity (when $u = \mathsf{bat}$) or KV-cache memory (when $u = \mathsf{mem}$). High shadow prices indicate anticipated congestion on that resource, while low prices indicate spare capacity. For each request $j$, the dual variable $y_j$ represents the \emph{option value} of request $j$---the maximum net benefit (SLO reward minus imputed resource cost) achievable by routing $j$ to its best device.
For a fixed price vector $p$, the optimal slack variable is
\[
y_j^*(p)=\Bigg[\ \max_{g\in[G]}\ \Big\{\, r_{j,g,k} - \sum_{i\in\mathcal I} a_{i,(j,g,k)}\,p_i \,\Big\}\ \Bigg]_+.
\]
Substituting $y^*(p)$ yields the reduced dual objective
\[
\min_{p\ge 0}\quad
\sum_{i\in\mathcal{I}} b_i p_i
\;+\;
\sum_{j\in\mathcal{J}^{(wait)}_k}\Bigg[\ \max_{g\in[G]}\ \Big\{\, r_{j,g,k} - a_{(j,g,k)}^\top p \,\Big\}\ \Bigg]_+,
\]
where $a_{(j,g,k)}^\top p := \sum_{i\in\mathcal I} a_{i,(j,g,k)}p_i$ and $a_{(j,g,k)} := (a_{i,(j,g,k)})_{i\in\mathcal I}\in\mathbb{R}^{|\mathcal I|}_+$.
The quantity $a_{(j,g,k)}^\top p$ is the \emph{imputed resource cost} of admitting $(j,g,k)$ under shadow prices $p$.
Accordingly, the \emph{bid-price margin}
\[
\Delta_j(g) \;:=\; r_{j,g,k} - a_{(j,g,k)}^\top p
\]
measures the net benefit of assigning request $j$ to device $g$ at time $k$ after accounting for resource opportunity costs.
\paragraph{Sample-Average Approximation (SAA)-style price learning.}
The reduced dual objective depends on the random stream of arrivals and on predicted decode length profiles.
We therefore view $p$ as the solution to an underlying stochastic program and estimate it from historical observations.
Let $\mathsf{Hist}$ denote a history set of previously observed pairs $(r^\star,a^\star)$, where each $(r^\star,a^\star)$ corresponds to a candidate action generated in the past.
At decision step $k$, we compute prices by minimizing the SAA:
\begin{equation} \label{eq:dual}
f_k(p)
\;=\;
d_k^\top p \;+\; \frac{1}{\max\{1,|\mathsf{Hist}|\}}\sum_{(r^\star,a^\star)\in \mathsf{Hist}} \bigl(r^\star-(a^\star)^\top p\bigr)_+,
\end{equation}
where $
d_k:=\frac{b^{(k-1)}}{\max\{1,\widehat n_{\mathrm{remain}}(k)\}}.$
Here, to simplify the notation, we denote $b^{(k)} \;:=\; \bigl(b^{(k)}_i\bigr)_{i\in\mathcal I}\in\mathbb{R}^{|\mathcal I|}_+,$ and $b^{(k-1)}$ is the residual-capacity vector before making decisions at step $k$, and $\widehat n_{\mathrm{remain}}(k)$ is an forecast of the expected number of remaining admission opportunities.
%The normalization in $d_k$ converts remaining reusable resources into a per-opportunity budget, stabilizing the price scale across time.
\paragraph{Bid-price control online decision.}
Given the price vector $p_k$ learned from history, a natural bid-price rule is to accept a request $j$ if its best margin, $\Delta_j(g):= r_{j,g,k}-a_{(j,g,k)}^\top p_k$, is positive.
However, at a single step $k$ there may be many waiting requests, and admitting \emph{all} actions with positive margins can violate residual constraints.
Moreover, rewards and resource profiles are based on predictions (e.g., $\hat o_j$), which introduces estimation error.
To robustly enforce feasibility, we (i) compute bid price margins for all candidates, (ii) rank candidates by margin, and (iii) greedily admit the highest-margin actions subject to the residual constraints and the one-assignment-per-request rule. The pseudocode can be found in Algorithm \ref{alg:1} in Appendix \ref{append:code}.
\iffalse
\begin{algorithm}[t]
\caption{Action-History-Dependent (AHD) Bid Price Control Router}
\label{alg:1}
\begin{algorithmic}[1]
\STATE \textbf{Input:} initial resource capacities $b^{(0)}\in\mathbb{R}^{|\mathcal I|}_+$ indexed by $i=(g,s,u)\in\mathcal I$; horizon $T$; reward/column generator for $(r_{j,g,k},a_{(j,g,k)})$; weights $(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2)$.
\STATE \textbf{Output:} decisions $\{x_{j,g,k}\}_{k\in[T],\,j,\,g}$.
\STATE Initialize history $\mathsf{Hist}\leftarrow\emptyset$ and waiting queue $\mathcal J^{(wait)}_1\leftarrow\emptyset$.
\FOR{$k=1,2,\dots,T$}
\STATE Update waiting set $\mathcal J^{(wait)}_k$ by adding new arrivals at time $k$ and removing any requests started at time $k{-}1$.
\STATE Define action set $\mathcal A_k\leftarrow\{(j,g,k): j\in\mathcal J^{(wait)}_k,\ g\in[G]\}$ and initialize $x_{j,g,k}\leftarrow 0$ for all $(j,g,k)\in\mathcal A_k$.
%\STATE Set normalized capacities $d_k \leftarrow b^{(k-1)}/\max\{1,\widehat n_{\mathrm{remain}}(k)\}$.
\STATE \textbf{Dual price update (SAA over history):}
\begin{equation} \label{eq:dual2}
p_k \in \arg\min_{p\ge 0}\ f_k(p),
\end{equation}
where $f_k(p)$ is defined in Equation \eqref{eq:dual}.
%\STATE Generate candidates $\{(r_{j,g,k},a_{(j,g,k)})\}_{(j,g,k)\in\mathcal A_k}$.
\STATE \textbf{Bid-price margins:} For all $(j,g,k)\in\mathcal A_k$, compute $\Delta_j(g)\leftarrow r_{j,g,k}-a_{(j,g,k)}^\top p_k$.
\STATE \textbf{Update record for next-step learning:} if $\mathcal A_k\neq\emptyset$, pick $(j^\star,g^\star)\in\arg\max_{(j,g,k)\in\mathcal A_k}\Delta_j(g)$ and set $\mathsf{Hist}\leftarrow \mathsf{Hist}\cup\{(r_{j^\star,g^\star,k},a_{(j^\star,g^\star,k)})\}$.
\STATE \textbf{Greedy admission (bid-price control):} initialize working residuals $b^{(k)}\leftarrow b^{(k-1)}$.
\STATE Let $\mathcal C_k \leftarrow \{(j,g,k)\in\mathcal A_k:\ \Delta_j(g)>0\}$ sorted by decreasing $\Delta_j(g)$.
\FOR{each $(j,g,k)\in\mathcal C_k$ (in sorted order)}
\STATE \textbf{if} $a_{i,(j,g,k)}\le b^{(k)}_i$ for all $i\in\mathcal I$ \textbf{then}
\STATE \hspace{1.2em}set $x_{j,g,k}\leftarrow 1$ and update $b^{(k)} \leftarrow b^{(k)} - a_{(\cdot,(j,g,k))}$.
\STATE \textbf{end if}
\ENDFOR
\ENDFOR
\end{algorithmic}
\end{algorithm}
\fi
\subsection{Projected Online Dual Gradient Descent}
\label{subsec:ogd}
Algorithm~\ref{alg:1} computes bid prices by solving the convex dual subproblem:
\begin{equation} \label{eq:dual2}
p_k \in \arg\min_{p\ge 0}\ f_k(p),
\end{equation}
where $f_k(p)$ is defined in Equation \eqref{eq:dual}. While \eqref{eq:dual2} is substantially smaller than the primal LP, solving it to optimality at every decision step can still be nontrivial at millisecond-scale latency.
We therefore suggest a method based on online gradient descent \cite{zinkevich2003online,sun2020near} that updates $p_k$ from $p_{k-1}$.
This is motivated by the fact that the dual objective evolves smoothly over time: between steps $k-1$ and $k$, the history set and resource capacities change incrementally.
\iffalse
\paragraph{Objective and subgradient.}
Let $\mathsf{Hist}_k$ denote the history available at the beginning of step $k$ and set $N_k:=|\mathsf{Hist}_k|$.
Recall the convex, piecewise-linear objective \eqref{eq:dual}
\[
f_k(p)
\;:=\;
d_k^\top p
\;+\;
\frac{1}{\max\{1,N_k\}}
\sum_{(r,a)\in \mathsf{Hist}_k} (r-a^\top p)_+.
\]
A valid subgradient of $f_k$ at $p$ is
\begin{align*}
g_k(p) = d_k
\;-\;
\frac{1}{\max\{1,N_k\}}
\sum_{(r,a)\in\mathsf{Hist}_k}
\ind\{r-a^\top p>0\}\, a.
\end{align*}
\fi
\paragraph{Warm-started projected mini-batch updates.}
At step $k$, we initialize $p^{(0)}\leftarrow p_{k-1}$ and perform a small fixed number $K$ of projected subgradient steps using a mini-batch
$\mathcal{B}_k\subseteq\mathsf{Hist}_k$ sampled uniformly without replacement:
\begin{equation}\label{eq:ogd-update}
p^{(s+1)}
\leftarrow
\Pi_{\mathbb{R}^{|\mathcal I|}_+}\!\Big(p^{(s)} - \eta_k\,\widehat g_k(p^{(s)})\Big),
\end{equation}
where
\[
\widehat g_k(p)
:=
d_k - \frac{1}{|\mathcal{B}_k|}\sum_{(r,a)\in\mathcal{B}_k}\ind\{r-a^\top p>0\}\,a,
\]
for $s=0,1,\dots,K{-}1$, and output $p_k:=p^{(K)}$.
Here $\Pi_{\mathbb{R}^{|\mathcal I|}_+}$ denotes projection onto the nonnegative orthant, implemented entrywise as $(\cdot)_+$.
This first-order update has short and predictable runtime (controlled by $K$ and $|\mathcal{B}_k|$) and avoids repeatedly solving \eqref{eq:dual2}.
\paragraph{Implementation details.}
In our LLM serving instantiation, the constraint index $i=(g,k,u)$ induces a structured price vector with two tensors (batch-occupancy and memory) over $(g,k)$, and we compute $a^\top p$ efficiently using only the active time slots of each candidate column.
We also apply mild normalization to keep the hinge-loss gradients numerically stable.
%These details, together with a complete pseudocode routine for the projected mini-batch price update and the rule for recording samples into $\mathsf{Hist}_k$,
The details are provided in Appendix~\ref{append:alg}.
\paragraph{Computational overhead and scaling.}
The online routing path does not require solving a full primal LP at any step. At each period, the dual variable has dimension $2GT$, but updates are highly structured. Each routing step performs only $K$ warm-started projected gradient updates over a mini-batch sampled from the action history, and the dot product $\mathbf{a}^\top \mathbf{p}$ for any candidate column touches only the \emph{active} future slots of that request, so the per-step cost scales with the number of active candidates times their average decode length rather than the full $GT$ horizon. The state itself is summarized by $O(GT)$ memory, and the per-step time complexity is $O(KGH)$ where $H$ is the effective look-ahead horizon. Both quantities are linear in cluster size, so the method scales gracefully to larger deployments---each worker's price tensor can also be updated in parallel since they are decoupled in the gradient step. In our current implementation, on an Intel i9-13980HX CPU, the routing computation takes approximately $1$--$2$ms per period, comfortably within the $10$--$30$ms wall-clock time of a single decode iteration on an A100 GPU. This confirms that the method meets the millisecond-scale decision requirements of online LLM serving and is unlikely to be the bottleneck even at $64$+ workers.
\section{Numerical Experiments} \label{sec:exp}
In this section, we present the results of our numerical experiments. Due to lack of GPU resources, we conducted our experiments using the Vidur simulator \cite{agrawal2024vidur}, which models
realistic LLM serving behavior including continuous batching and
KV-cache management. While simulation cannot capture all aspects of
real deployments, it allows controlled study of routing policies
across diverse scenarios. Our routing algorithm has been integrated into the Vidur simulator, and the code is available at \url{https://github.com/qqwetidx/Online-Linear-Programming-for-Vidur}.
\subsection{Experimental Setup}
We simulate a scenario with four parallel NVIDIA A100 GPUs. To evaluate the performance of our proposed routing algorithm, we consider two types of request distributions:
\begin{itemize}
\item \textbf{Real Data:} We test on a conversational dataset provided by \cite{zheng2023lmsys}, which is publicly available at \url{https://huggingface.co/datasets/lmsys/lmsys-chat-1m}. The real-world results are summarized in the main text, with additional experiments available in Appendix \ref{append:real}.
\item \textbf{Synthetic Data:} We also simulate requests with varying prefill-to-decode (P/D) ratios, specifically: (i) P/D ratio 1:4 (more prefill than decode); (ii) P/D ratio 4:1 (more decode than prefill).
These settings simulate different workload patterns, ranging from long prefill phases to long decoding phases. The results for this synthetic dataset are presented in Appendix \ref{append:syn}.
\end{itemize}
For the real dataset, the request arrival process is modeled as a stationary Poisson process, with an arrival rate $\lambda \in \{0.4, 0.5\}$. We observe that a rate of $\lambda = 0.4$ ensures system stability, while $\lambda = 0.5$ leads to a slight overload. Each arriving request is randomly sampled from the dataset.
\subsection{Baseline Algorithms and Parameter Settings}
We compare the performance of our algorithm against all routing heuristics which are default in the Vidur simulator: (i) \textbf{Round Robin:} Requests are cyclically assigned to the GPUs in a fixed order, regardless of the current system state. (ii) \textbf{Least Outstanding Request (LOR):} This policy assigns requests to the GPU with the shortest queue. (iii) \textbf{Random:} Requests are randomly assigned to any of the four GPUs with equal probability $1/4$. (iv) \textbf{Power-of-2 Choices:} For each incoming request, two GPUs
are sampled uniformly at random and the request is assigned to the one
with fewer outstanding requests. This is a widely used randomized
load-balancing heuristic known to improve over both Random and pure
shortest-queue rules in classical parallel-server settings.
For our routing algorithm, we conduct experiments with various configurations to examine the effects of prediction accuracy and different objective weightings:
\begin{itemize}
\item \textbf{Prediction Accuracy:} We explore two scenarios for the predicted decode length $\hat{o}_j$: (i) Perfect Prediction: In some experiments, we assume perfect prediction, where $\hat{o}_j = o_j$.
(ii) Imperfect Prediction: In other experiments, we model a more realistic scenario with 20\% prediction error, where $\hat{o}_j \sim \text{Unif}(0.8 \cdot o_j, 1.2 \cdot o_j)$.
\item \textbf{Objective Weights:} We vary the weights $\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2$ for the different service-level objectives (SLOs) to observe the algorithm's performance under different priorities.
\end{itemize}
\subsection{Results}
We report sample results on the real-data setting described in Section~\ref{sec:exp}. Additional experiments across parameter combinations are deferred to Appendix~\ref{append:real}.
\begin{table}[t]
\centering
\small
\caption{Relative improvement (\%) over Round-Robin with noisy decode-length prediction
$\hat{o}_j \sim \text{Unif}(0.8 o_j, 1.2 o_j)$.}
\label{tab:p2c-stationary}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{lccccc}
\toprule
Method & Avg EEL$\downarrow$ & P95 EEL$\downarrow$ & P99 EEL$\downarrow$ & Thr.$\uparrow$ & SLO$\downarrow$ \\
\midrule
LOR & 0.67 & 4.01 & 6.19 & 0.53 & 0.98 \\
Power-of-2 & 1.30 & 3.84 & 6.06 & 0.73 & 1.84 \\
\textbf{Ours} & \textbf{45.75} & \textbf{42.49} & \textbf{25.85} & \textbf{0.90} & \textbf{11.10} \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{Real-data performance under representative multi-objective settings.}
Figure~\ref{fig:metrics-lam1} presents two representative configurations: one with perfect decode-length prediction ($\hat o_j=o_j$) and one with noisy prediction. In both cases, we choose a relatively large QPS weight $\sigma$ to discourage idling and sustain throughput under load, and we set $(\zeta_1,\zeta_2)$ large to emphasize tail-latency control.
Across both prediction regimes, our router achieves substantially smaller TTFT and tail TTFT than all baselines.
Importantly, from Table \ref{tab:p2c-stationary}, the advantage persists under noisy $\hat o_j$, indicating that the policy is robust to moderate prediction error.
\begin{figure}[t]
\centering
\begin{subfigure}{0.95\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wlat_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}\par\vspace{0.5em}
\begin{subfigure}{0.95\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wthr_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{27},\frac{1}{2048},\frac{1}{2048},5,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8o_j,1.2o_j)$}
\end{subfigure}
\caption{Real-data comparison between routing policies under two representative multi-objective settings.}
\label{fig:metrics-lam1}
\end{figure}
\paragraph{Controlling trade-offs by adjusting objective weights.}
A central advantage of our LP-based formulation is that it exposes an explicit, interpretable knob for trading off competing SLOs.
Figure~\ref{fig:metrics-lam2} illustrates this effect by considering two extreme weight choices.
In the first setting, we emphasize average latency by setting $(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2)=(0,1,1,0,0,0)$, which substantially reduces average end-to-end latency and average TTFT, but can worsen tail metrics because the policy is not incentivized to protect the slowest requests.
In the second setting, we emphasize tail control via $(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2)=(0,0,0,0,1,1)$, which markedly improves tail end-to-end latency and tail TTFT.
\begin{figure}[t]
\begin{subfigure}{0.95\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p5_wlat_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=(0,1,1,0,0,0)$;\ $\hat{o}_j=o_j$}
\end{subfigure}\par\vspace{0.5em}
\begin{subfigure}{0.95\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wtail_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=(0,0,0,0,1,1)$;\ $\hat{o}_j\sim \text{Unif}(0.8o_j,1.2o_j)$}
\end{subfigure}\par\vspace{0.5em}
\caption{Real-data comparison illustrating how objective weights shift performance trade-offs.}
\label{fig:metrics-lam2}
\end{figure}
\paragraph{Robustness to non-stationary workloads.}
A natural concern with any router that learns from historical
arrivals is whether it can adapt when the workload distribution
shifts. To stress-test our policy, we run a rate-shift experiment
in which the Poisson arrival rate switches from $\lambda=0.4$ to
$\lambda=0.6$ at the simulation midpoint, simulating a sudden load
spike. The router is \emph{not} informed of either the shift point
or the new rate. To handle this online, we equip the SAA-style price
update with a simple rolling-window mechanism: we monitor the
empirical arrival rate over a sliding window and, when a
statistically significant increase is detected, we discard stale
history samples and re-estimate using only recent observations.
This integrates naturally with the warm-started projected gradient
descent, since the dual prices
are already updated incrementally rather than recomputed from scratch.
Table~\ref{tab:nonstationary} reports the relative improvement of
each method over Round Robin under this non-stationary protocol with noisy
decode-length prediction.
\begin{table}[t]
\centering
\small
\caption{Relative improvement (\%) over Round Robin under a non-stationary rate-shift workload (Poisson rate switches from $\lambda=0.4$ to $\lambda=0.6$ at the midpoint; router uninformed of the shift). Decode lengths are noisy: $\hat{o}_j \sim \text{Unif}(0.8 o_j, 1.2 o_j)$.}
\label{tab:nonstationary}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{lccccc}
\toprule
Method & Avg & P95 & P99 & Thr. & SLO \\
& EEL$\downarrow$ & EEL$\downarrow$ & EEL$\downarrow$ & $\uparrow$ & Viol.$\downarrow$ \\
\midrule
LOR & 1.47 & 6.72 & 9.67 & 1.48 & 3.92 \\
Power-of-2 & 1.80 & 6.35 & 8.28 & 1.39 & 3.28 \\
\textbf{Ours} & \textbf{44.81} & \textbf{45.69} & \textbf{31.15} & \textbf{1.93} & \textbf{14.57} \\
\bottomrule
\end{tabular}
\end{table}
\section{Discussion} \label{sec:discussion}
Beyond algorithmic improvements, our LP framework provides structural insights about LLM routing and suggests directions for future system design.
\paragraph{What the framework reveals.}
First, shadow prices serve as \emph{congestion signals}: they reveal which resources---batch slots versus KV-cache memory---are bottlenecks under different workloads. In decode-heavy scenarios with long output sequences, memory prices dominate; in high-arrival-rate scenarios, batch prices dominate. This diagnostic information is unavailable from heuristic routers. Second, the framework exposes the \emph{trade-off structure} between competing SLOs: varying objective weights traces the Pareto frontier between throughput and latency, enabling systematic capacity planning rather than ad-hoc tuning. Third, our experiments demonstrate \emph{robustness to prediction error}: the bid-price mechanism degrades gracefully under inaccurate decode-length predictions because prices adapt online, whereas fixed heuristics cannot adjust to prediction errors.
\paragraph{Implications for system design.}
Our framework suggests several directions for practitioners. The shadow-price mechanism could be integrated with real-time telemetry (queue depths, memory utilization) for adaptive routing that responds to transient congestion. During overload, the framework naturally supports SLO-aware admission control: requests whose SLO contribution falls below their resource cost can be delayed or rejected, prioritizing high-value traffic. The formulation also extends naturally to heterogeneous GPU fleets by specifying device-specific capacity constraints $(B_g, M_g)$, enabling unified routing across mixed hardware.
\paragraph{When does optimization help most?}
Our experiments suggest the framework provides the largest benefits when (i) decode lengths are long and variable, so time-coupled constraints bind; (ii) multiple SLOs are genuinely in tension; and (iii) the system operates near full capacity. For light loads or very short decodes, simple heuristics often suffice---a regime our framework identifies through near-zero shadow prices. This characterization helps practitioners decide when optimization-based routing is worth the added complexity.
\section{Limitations and Future Directions} \label{sec:conclusion}
Our validation relies on simulation using the Vidur framework; real system implementation would face additional challenges including integration with serving engines (e.g., vLLM), handling of preemption and KV-cache swapping, and coordination overhead in distributed deployments. Addressing these engineering challenges and validating performance on production workloads is an important direction for future work.
The current framework also assumes homogeneous request priorities and does not explicitly model fairness across users or request classes. Extending the objective to incorporate priority tiers, fairness constraints, and load-balancing considerations---while preserving computational efficiency---is a natural next step. More broadly, we hope this work demonstrates the value of bringing principled optimization methodology to LLM serving and provides a foundation for future research at the intersection of operations research and machine learning systems.
\section{Supplementary Materials for Section \ref{sec:model}} \label{append:model}
\paragraph{Overview.}
Section~\ref{sec:model} considers PD disaggregation: prefill is completed upstream and each decode device only performs decoding. We now sketch an extension to \emph{PD mixing}, where each device may process both prefill and decode and continuous batching interleaves them within the same iteration.
\paragraph{Requests and routing.}
Requests arrive at times $t_j$ with known prompt length $s_j$ and unknown decode length $o_j$ (with prediction $\hat o_j$). The router selects a device $g$ and a dispatch period $\tau\ge t_j$ using the same decision variable $x_{j,g,\tau}$.
\paragraph{Mixed service discipline.}
After dispatch, request $j$ joins the FIFO queue of device $g$. When admitted into the active batch, it first undergoes a prefill step (materializing KV states for the prompt) and then executes $o_j$ decode steps. In a PD-mixing engine, a batch at a given period may contain requests in their prefill step and requests in decoding; we treat one such engine iteration as one time period in the model. %{\color{red} See Figure 2}
\iffalse
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{plots/Figure2.png}
\caption{Figure 2}
\label{fig:2}
\end{figure}
\fi
\paragraph{KV-cache memory accounting.}
Let $\theta$ denote the service step index of request $j$ since it becomes active on a device, where $\theta=0$ corresponds to its prefill step and $\theta=1,\dots,o_j$ correspond to its decode steps (generating the $\theta$th output token). We approximate that the prefill step allocates the full prompt KV-cache footprint at once. The resulting memory-usage kernel becomes
\[
m^{\text{mem}}_j(\theta)=
\begin{cases}
s_j, & \theta=0,\\
s_j+\theta, & 1\le \theta\le o_j,\\
0, & \text{otherwise}.
\end{cases}
\]
Compared with Section~\ref{sec:model}, the only change is the additional prefill step $\theta=0$, whose memory cost is $s_j$. (A finer-grained \emph{chunked prefill} model can be obtained by splitting prefill into multiple steps, but we use the above abstraction for clarity.)
\paragraph{Feasibility constraints.}
The same constraints apply to mixed batches: at any period, each device hosts at most $B$ active requests (counting both prefill and decode), and the total KV-cache usage of active requests must not exceed $M$.
\paragraph{Token accounting.}
In Section~\ref{sec:model}, all processed tokens are decode tokens. Under PD mixing, devices also process prompt tokens during prefill. When reporting throughput, one may either count only generated (decode) tokens, or count both prompt and generated tokens; both conventions appear in practice. Our formulations and algorithms apply under either convention once the corresponding token count is used consistently.
\paragraph{Latency metrics under mixing.}
Under PD mixing, TTFT and TEL naturally include both prefill and decode waiting/processing time, because the first output token cannot be produced until the request completes its prefill step and begins decoding. The definitions in Section~\ref{sec:model} remain valid if $t_j$ is interpreted as the original request arrival time before prefill and $\tau_j$ as the period when the first output token is generated.
\paragraph{Algorithmic adjustments.}
All changes to the online LP algorithm are conceptual substitutions of the decode-only model components by their mixed counterparts. Under the above abstraction, each request consumes $(1+\hat o_j)$ service steps on its assigned device (one prefill step plus $\hat o_j$ predicted decode steps), and all per-device capacity constraints are enforced using the modified memory kernel $m^{\text{mem}}_j(\cdot)$.
\paragraph{Batch-time variability and implications.}
A key difference from PD disaggregation is that the wall-clock duration of a mixed batch iteration is input-dependent: it varies with the composition of prefill versus decode work and with prompt lengths. Consequently, it is difficult to map the period-indexed model to real time with a single, accurate conversion factor. Our online LP algorithm can still be applied by treating each iteration as one abstract period and using workload predictions (e.g., $\hat o_j$ and $s_j$) for planning; however, because the per-iteration runtime is harder to quantify under PD mixing, the realized performance may be less stable.
\section{Supplementary Materials for Section~\ref{sec:alg}}
\label{append:alg}
\subsection{Vectorization: batch-occupancy and memory prices}
\label{append:code}
In this part, we provide the pseudocode of our algorithm.
\begin{algorithm}[h]
\caption{Action-History-Dependent (AHD) Bid Price Control Router}
\label{alg:1}
\begin{algorithmic}[1]
\STATE \textbf{Input:} initial resource capacities $b^{(0)}\in\mathbb{R}^{|\mathcal I|}_+$ indexed by $i=(g,s,u)\in\mathcal I$; horizon $T$; reward/column generator for $(r_{j,g,k},a_{(j,g,k)})$; weights $(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2)$.
\STATE \textbf{Output:} decisions $\{x_{j,g,k}\}_{k\in[T],\,j,\,g}$.
\STATE Initialize history $\mathsf{Hist}\leftarrow\emptyset$ and waiting queue $\mathcal J^{(wait)}_1\leftarrow\emptyset$.
\FOR{$k=1,2,\dots,T$}
\STATE Update waiting set $\mathcal J^{(wait)}_k$ by adding new arrivals at time $k$ and removing any requests started at time $k{-}1$.
\STATE Define action set $\mathcal A_k\leftarrow\{(j,g,k): j\in\mathcal J^{(wait)}_k,\ g\in[G]\}$ and initialize $x_{j,g,k}\leftarrow 0$ for all $(j,g,k)\in\mathcal A_k$.
%\STATE Set normalized capacities $d_k \leftarrow b^{(k-1)}/\max\{1,\widehat n_{\mathrm{remain}}(k)\}$.
\STATE \textbf{Dual price update (SAA over history):}
\begin{equation}
p_k \in \arg\min_{p\ge 0}\ f_k(p),
\end{equation}
where $f_k(p)$ is defined in Equation \eqref{eq:dual}.
%\STATE Generate candidates $\{(r_{j,g,k},a_{(j,g,k)})\}_{(j,g,k)\in\mathcal A_k}$.
\STATE \textbf{Bid-price margins:} For all $(j,g,k)\in\mathcal A_k$, compute $\Delta_j(g)\leftarrow r_{j,g,k}-a_{(j,g,k)}^\top p_k$.
\STATE \textbf{Update record for next-step learning:} if $\mathcal A_k\neq\emptyset$, pick $(j^\star,g^\star)\in\arg\max_{(j,g,k)\in\mathcal A_k}\Delta_j(g)$ and set $\mathsf{Hist}\leftarrow \mathsf{Hist}\cup\{(r_{j^\star,g^\star,k},a_{(j^\star,g^\star,k)})\}$.
\STATE \textbf{Greedy admission (bid-price control):} initialize working residuals $b^{(k)}\leftarrow b^{(k-1)}$.
\STATE Let $\mathcal C_k \leftarrow \{(j,g,k)\in\mathcal A_k:\ \Delta_j(g)>0\}$ sorted by decreasing $\Delta_j(g)$.
\FOR{each $(j,g,k)\in\mathcal C_k$ (in sorted order)}
\STATE \textbf{if} $a_{i,(j,g,k)}\le b^{(k)}_i$ for all $i\in\mathcal I$ \textbf{then}
\STATE \hspace{1.2em}set $x_{j,g,k}\leftarrow 1$ and update $b^{(k)} \leftarrow b^{(k)} - a_{(\cdot,(j,g,k))}$.
\STATE \textbf{end if}
\ENDFOR
\ENDFOR
\end{algorithmic}
\end{algorithm}
\subsection{Vectorization: batch-occupancy and memory prices}
\label{append:llm-vectorization}
In the fixed-horizon LLM instantiation, constraints are indexed by device $g\in[G]$, time slot $k\in[T]$, and type
$u\in\{\textsf{bat},\textsf{mem}\}$.
Accordingly, we maintain two nonnegative price tensors
\[
p^{\textsf{bat}}_{g,k}\ge 0,
\qquad
p^{\textsf{mem}}_{g,k}\ge 0,
\]
and flatten them into a single vector
\[
p \equiv \mathrm{vec}\!\big(p^{\textsf{bat}},p^{\textsf{mem}}\big)\in\mathbb{R}^{2GT}_+ \equiv \mathbb{R}^{|\mathcal I|}_+ .
\]
\paragraph{Efficient evaluation of $a^\top p$ for a candidate column.}
Consider a candidate action $(j,g,k)$ (start request $j$ on device $g$ at decision step $k$).
Its column vector $a_{(j,g,k)}$ has nonzeros only on indices with the same device $g$ and future time slots $k' \ge k$ where the request is active.
Let
\[
\mathcal K_{j,k}:=\{\, k'\in[T] : a_j(k'-k)=1 \,\}
\]
denote the active slots of request $j$ if started at $k$.
Then the dot product decomposes as
\begin{equation}\label{eq:inner-llm-append}
a_{(j,g,k)}^\top p
\;=\;
\sum_{k' \in\mathcal K_{j,k}} p^{\textsf{bat}}_{g,k'}
\;+\;
\sum_{k' \in\mathcal K_{j,k}} p^{\textsf{mem}}_{g,k'}\cdot \widetilde m_{j}(k'-k),
\end{equation}
where $\widetilde m_j(\cdot)$ is a per-slot memory multiplier derived from $m^{\text{mem}}_j(\cdot)$.
In code, we store the active index set $\mathcal K_{j,k}$ and the corresponding memory multipliers to avoid materializing the full $2GT$-dimensional column.
\paragraph{Reward and memory scaling.}
To stabilize hinge gradients, the implementation may apply (i) reward normalization and (ii) memory scaling:
\[
r^{\textsf{norm}} \;:=\; \min\Big\{\frac{r}{\texttt{reward\_normalizer}},\,1\Big\},
\qquad
\widetilde m_j(\cdot) \;:=\; \texttt{mem\_scale}\cdot m^{\text{mem}}_j(\cdot),
\]
where \texttt{mem\_scale} is a fixed constant (e.g., proportional to $1/\max_{g,s} M_{g,s}$) to keep the two resource types commensurate.
\paragraph{Constructing the scarcity vector $d_k$.}
At decision step $k$, we form remaining capacities per device and slot:
\[
\mathrm{remain}^{\textsf{bat}} := \big(B-\mathrm{occ}^{\textsf{bat}}\big)_+,\qquad
\mathrm{remain}^{\textsf{mem}} := \big(M-\mathrm{occ}^{\textsf{mem}}\big)_+,
\]
divide by an estimate of remaining arrivals $\widehat n_{\mathrm{remain}}(k)$, and flatten:
\begin{equation}\label{eq:dvec-flat-append}
d_k
\;=\;
\frac{1}{\max\{1,\widehat n_{\mathrm{remain}}(k)\}}\,
\mathrm{vec}\!\Big(\mathrm{remain}^{\textsf{bat}},\ \texttt{mem\_scale}\cdot \mathrm{remain}^{\textsf{mem}}\Big),
\end{equation}
(optionally with the same \texttt{reward\_normalizer} scaling applied as above).
\subsection{Projected mini-batch subgradient routine for the price update}
\label{append:resolve-prices}
\begin{algorithm}[H]
\caption{\textsc{ResolveDualPrices}$(d_k,\mathsf{Hist}_k,p_{k-1})$: projected mini-batch subgradient for \eqref{eq:dual}}
\label{alg:resolve-prices}
\begin{algorithmic}[1]
\STATE \textbf{Input:} scarcity vector $d_k\in\mathbb{R}^{|\mathcal I|}_+$; history $\mathsf{Hist}_k=\{(r_\ell,a_\ell)\}_{\ell=1}^{N_k}$; warm start $p_{k-1}\in\mathbb{R}^{|\mathcal I|}_+$; step size $\eta_k>0$; steps $K$; batch size $B$.
\STATE \textbf{Output:} updated dual prices $p_k\in\mathbb{R}^{|\mathcal I|}_+$.
\IF{$N_k=0$}
\STATE $p_k\leftarrow 0$.
\STATE \textbf{return}.
\ENDIF
\STATE $B \leftarrow \min\{B,N_k\}$.
\STATE $p \leftarrow p_{k-1}$.
\FOR{$s=1,2,\dots,K$}
\STATE Sample $\mathcal{B}\subseteq \{1,\dots,N_k\}$ uniformly without replacement, $|\mathcal{B}|=B$.
\STATE $\widehat g \leftarrow d_k$.
\FOR{each $\ell\in\mathcal{B}$}
\STATE Compute margin $\Delta_\ell \leftarrow r_\ell - a_\ell^\top p$ (via \eqref{eq:inner-llm-append}).
\IF{$\Delta_\ell>0$}
\STATE $\widehat g \leftarrow \widehat g - \frac{1}{B}\,a_\ell$.
\ENDIF
\ENDFOR
\STATE $p \leftarrow (p - \eta_k\,\widehat g)_+$ \COMMENT{projection onto $\mathbb{R}^{|\mathcal I|}_+$}
\ENDFOR
\STATE $p_k \leftarrow p$.
\STATE \textbf{return}.
\end{algorithmic}
\end{algorithm}
\subsection{How samples enter $\mathsf{Hist}_k$ in the online loop}
\label{append:hist-recording}
To keep the history compact, we record a small number of informative columns per decision step.
A simple choice (used in our simulator) is to record one column corresponding to the largest instantaneous margin under current prices:
\[
(j^\star,g^\star)\in\arg\max_{(j,g,k)\in\mathcal A_k}\ \Big\{\, r_{j,g,k} - a_{(j,g,k)}^\top p_k \Big\},
\qquad
\mathsf{Hist}_{k+1}\leftarrow \mathsf{Hist}_k\cup\{(r_{j^\star,g^\star,k},a_{(j^\star,g^\star,k)})\}.
\]
This design focuses the dual updates on the most competitive placement options under the current price vector while keeping the per-step learning overhead small.
\section{Supplementary Materials for Section \ref{sec:exp}} \label{append:exp}
\subsection{Experiments on Real Dataset} \label{append:real}
\paragraph{Results under perfect decode-length prediction.}
Figures~\ref{fig:app1}, \ref{fig:app2}, \ref{fig:app3} report comparisons between our router and the baseline policies under a stable arrival rate ($\lambda=0.4$), with objective weights chosen to emphasize (respectively) average latencies, tail latencies, and throughput.
Figures~\ref{fig:app1h}, \ref{fig:app2h}, \ref{fig:app3h} report the analogous experiments under a mildly overloaded regime ($\lambda=0.5$).
Across these settings, we make three consistent observations.
\begin{enumerate}
\item \textbf{Targeted improvements:} when the objective weights emphasize a particular metric family (average latency, tail latency, or throughput), our policy achieves the best performance on that targeted metric compared to all baselines.
\item \textbf{Trade-offs are explicit and predictable:} when we set the weights of non-target metrics to zero, the targeted metric typically improves further, but non-target metrics may degrade because they are no longer optimized.
\item \textbf{Overload amplifies tail effects:} under $\lambda=0.5$, tail latencies can deteriorate if tail objectives are assigned insufficient weight. This reinforces the practical importance of explicitly controlling tail-latency terms in overloaded regimes.
\end{enumerate}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wlat_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wlat_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Real-data comparison between routing policies under stable arrival process $\lambda=0.4$, with a focus on average latency.}
\label{fig:app1}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wtail_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wtail_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Real-data comparison between routing policies under stable arrival process $\lambda=0.4$, with a focus on tail latency.}
\label{fig:app2}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wthr_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,\frac{1}{400},\frac{1}{400},1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wthr_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,0,0,1,0,0\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Real-data comparison between routing policies under stable arrival process $\lambda=0.4$, with a focus on throughput.}
\label{fig:app3}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p5_wlat_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p5_wlat_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Real-data comparison between routing policies under overloaded arrival process $\lambda=0.5$, with a focus on average latency.}
\label{fig:app1h}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p5_wtail_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p4_wthr_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Real-data comparison between routing policies under overloaded arrival process $\lambda=0.5$, with a focus on tail latency.}
\label{fig:app2h}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p5_wthr_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,\frac{1}{400},\frac{1}{400},1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodeexact_lam0p5_wtail_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,0,0,1,0,0\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Real-data comparison between routing policies under overloaded arrival process $\lambda=0.5$, with a focus on throughput.}
\label{fig:app3h}
\end{figure}
\paragraph{Results under noisy decode-length prediction.}
We next evaluate robustness to prediction error by perturbing decode-length estimates via
$\hat o_j \sim \text{Unif}(0.8\,o_j,\,1.2\,o_j)$.
Figures~\ref{fig:app1n}, \ref{fig:app2n}, \ref{fig:app3n} present results for the stable regime ($\lambda=0.4$), and Figures~\ref{fig:app1hn}, \ref{fig:app2hn}, \ref{fig:app3hn} present results for the overloaded regime ($\lambda=0.5$), again with objective weights emphasizing average latencies, tail latencies, and throughput.
The qualitative conclusions mirror the perfect-prediction case: the policy continues to improve the metrics it is asked to optimize, exhibits clear trade-offs when objectives are reweighted, and benefits from explicitly weighting tail terms under overload.
Overall, these experiments indicate that the proposed router is robust to moderate decode-length prediction noise.
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wlat_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wlat_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Real-data comparison between routing policies with noisy decode length prediction under stable arrival process $\lambda=0.4$, with a focus on average latency.}
\label{fig:app1n}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wtail_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wtail_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Real-data comparison between routing policies with noisy decode length prediction under stable arrival process $\lambda=0.4$, with a focus on tail latency.}
\label{fig:app2n}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wthr_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,\frac{1}{400},\frac{1}{400},1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wthr_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,0,0,1,0,0\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Real-data comparison between routing policies with noisy decode length prediction under stable arrival process $\lambda=0.4$, with a focus on throughput.}
\label{fig:app3n}
\end{figure}
\paragraph{Tail weights and the achieved tail quantile.}
As discussed in Section~\ref{subsec:lp}, the tail weights $(\zeta_1,\zeta_2)$ do more than rescale objectives: they directly influence how aggressively the router prioritizes meeting a tail objective.
To make this explicit, Figure~\ref{fig:zeta2} fixes all parameters except $\zeta_2$, fixes a TTFT threshold $t_2'=49$, and reports the empirical \emph{satisfaction rate} $q(\zeta_2)\;:=\;\frac{1}{|\mathcal I|}\sum_{j\in\mathcal I}\ind\{\mathrm{TTFT}_j \le t_2'\}$,
which can be interpreted as the quantile level at which the TTFT threshold $t_2'$ is attained.
As $\zeta_2$ increases, $q(\zeta_2)$ increases on average (up to simulation noise), confirming that larger tail weights systematically push the policy toward satisfying a stricter tail requirement for a larger fraction of requests.
\begin{figure}[t]
\centering
\includegraphics[width=0.75\linewidth]{plots/tail_ttft_ratio_vs_x.pdf}
\caption{TTFT satisfaction rate as a function of the tail weight $\zeta_2$.}
\label{fig:zeta2}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p5_wlat_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p5_wlat_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Real-data comparison between routing policies with noisy decode length prediction under overloaded arrival process $\lambda=0.5$, with a focus on average latency.}
\label{fig:app1hn}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p5_wtail_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p4_wthr_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Real-data comparison between routing policies with noisy decode length prediction under overloaded arrival process $\lambda=0.5$, with a focus on tail latency.}
\label{fig:app2hn}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p5_wthr_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,\frac{1}{400},\frac{1}{400},1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srccsv_prefillcsv_decodecsv_omodenoisy_lam0p5_wtail_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(1,0,0,1,0,0\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Real-data comparison between routing policies with noisy decode length prediction under overloaded arrival process $\lambda=0.5$, with a focus on throughput.}
\label{fig:app3hn}
\end{figure}
\subsection{Experiments on Synthetic Dataset} \label{append:syn}
We complement the real-data experiments with synthetic workloads that control the prefill-to-decode (P/D) ratio, allowing us to stress-test the router under qualitatively different compute/memory regimes. In all cases, we compare against the same baseline policies as in the main text and report results under both perfect and noisy decode-length prediction.
\paragraph{P/D ratio $1{:}4$ (decode-heavy).}
Figures~\ref{fig:app114} and~\ref{fig:app214} report comparisons under a stable arrival rate for decode-heavy requests (P/D $=1{:}4$), with objective weights chosen to emphasize (respectively) average latencies and tail latencies. Figures~\ref{fig:app114n} and~\ref{fig:app214n} report the corresponding experiments under $20\%$ decode-length prediction noise.
The qualitative conclusions align with the real-data setting: (i) when the objective weights emphasize a particular metric family, our policy consistently improves that targeted metric relative to the baselines; (ii) reweighting objectives produces predictable trade-offs between average and tail performance; and (iii) the performance advantages persist under moderate prediction noise, indicating robustness of the bid-price control mechanism.
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodeexact_lam0p4_wlat_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodeexact_lam0p4_wlat_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $1:4$) comparison between routing policies under stable arrival process, with a focus on average latency.}
\label{fig:app114}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodeexact_lam0p4_wtail_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodeexact_lam0p5_wtail_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $1:4$) comparison between routing policies under stable arrival process, with a focus on tail latency.}
\label{fig:app214}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodenoisy_lam0p4_wlat_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodenoisy_lam0p4_wlat_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $1:4$) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on average latency.}
\label{fig:app114n}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodenoisy_lam0p4_wtail_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_100_400_prefill100+50_decode400+50_omodenoisy_lam0p5_wtail_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $1:4$) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on tail latency.}
\label{fig:app214n}
\end{figure}
\paragraph{P/D ratio $4{:}1$ (prefill-heavy / decode-light).}
Figures~\ref{fig:app141} and~\ref{fig:app241} report the analogous comparisons for prefill-heavy requests (P/D $=4{:}1$), again emphasizing (respectively) average and tail latency objectives, with Figures~\ref{fig:app141n} and~\ref{fig:app241n} showing the corresponding noisy-prediction results.
In this regime, decode workloads are relatively short, so decode-side batch and KV constraints are less likely to become binding; as a result, small changes in objective weights often lead to smaller changes in observed performance.
This highlights an important practical point: the proposed router is most impactful when decode-side congestion is substantial (e.g., long or highly variable decode lengths), where time-coupled batch/KV constraints meaningfully shape the feasible set and where objective-aware admission decisions provide the largest benefit.
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodeexact_lam0p4_wlat_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodeexact_lam0p4_wlat_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $4:1$) comparison between routing policies under stable arrival process, with a focus on average latency.}
\label{fig:app141}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodeexact_lam0p4_wtail_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodeexact_lam0p5_wtail_large_zero.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j=o_j$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $4:1$) comparison between routing policies under stable arrival process, with a focus on tail latency.}
\label{fig:app241}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodenoisy_lam0p4_wlat_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},1,1,1,\frac{1}{400},\frac{1}{400}\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodenoisy_lam0p4_wlat_large_zero.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,1,1,0,0,0\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $4:1$) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on average latency.}
\label{fig:app141n}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodenoisy_lam0p4_wtail_large_small.pdf}
\caption{$\lambda=0.4$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(\frac{1}{135},\frac{1}{400},\frac{1}{400},1,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\begin{subfigure}{0.45\linewidth}
\centering
\includegraphics[width=0.9\linewidth]{other_plots/metrics_srcnormal_400_100_prefill400+50_decode100+50_omodenoisy_lam0p5_wtail_large_small.pdf}
\caption{$\lambda=0.5$;\ $\left(\alpha,\beta,\gamma,\sigma,\zeta_1,\zeta_2\right)=\left(0,0,0,0,1,1\right)$;\ $\hat{o}_j\sim \text{Unif}(0.8\cdot o_j,1.2\cdot o_j)$}
\end{subfigure}
\caption{Synthetic data (P/D ratio $4:1$) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on tail latency.}
\label{fig:app241n}
\end{figure}