Efficient On-Device Agents via Adaptive Context Management
Extracted source text · arXiv 2511.03728
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
\usepackage{booktabs}
\iclrfinalcopy
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{tcolorbox}
\usepackage{hyperref}
\usepackage{url}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{enumitem}
\usepackage{graphicx} % To include images
\usepackage{wrapfig} % The package for wrapping text around figures
\usepackage{caption}
\usepackage{subcaption}
\usepackage{booktabs}
\usepackage{multirow}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\tcbuselibrary{skins,breakable} % `breakable` is good practice for long boxes
\usepackage{xcolor}
% --- Custom Commands for Readability ---
\newcommand{\code}[1]{\texttt{#1}}
\newcommand{\permanent}[1]{{\color{blue!60!black}\code{#1}}}
\newcommand{\ephemeral}[1]{{\color{red!80!black}\code{#1}}}
\newcommand{\generated}[1]{{\color{green!50!black}\code{#1}}}
\newcommand{\kvcache}[1]{
\begin{tcolorbox}[
colback=black!5, colframe=black!20, boxrule=0.5pt,
fontupper=\small\ttfamily, top=2pt, bottom=2pt, left=3pt, right=3pt
]
#1
\end{tcolorbox}
}
% Define a style for JSON listings
\lstdefinestyle{cleanjson}{
basicstyle=\ttfamily\scriptsize, % Small, monospaced font
breaklines=true, % Allow lines to break
breakatwhitespace=false, % Break lines anywhere if needed
frame=none, % No frame around the code
backgroundcolor=\color{white}, % No background color
showstringspaces=false, % No special marks in strings
aboveskip=0pt, % No extra space above the listing
belowskip=0pt, % No extra space below the listing
xleftmargin=0pt, % No extra indentation on the left
xrightmargin=0pt, % No extra indentation on the right
columns=fullflexible % Tightly packs the characters
}
\lstset{style=cleanjson}
\title{Efficient On-Device Agents via Adaptive \\Context Management}
% \vspace{-8mm}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\usepackage{authblk}
% \vspace{-5mm}
\author[1]{\textbf{Sanidhya Vijayvargiya}\textsuperscript{*}}
\author[2]{\textbf{Rahul Lokesh}}
\affil[1]{Language Technologies Institute, Carnegie Mellon University}
\affil[2]{Samsung Research America}
\affil[ ]{sanidhyv@andrew.cmu.edu, rahul.lokesh@samsung.com}
% Add this after \maketitle
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
% Add this after \maketitle or in the document body
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
%\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\footnotetext[1]{Work done as part of internship at Samsung Research America.}
\vspace{-7mm}
\begin{abstract}
\vspace{-3mm}
On-device AI agents offer the potential for personalized, low-latency assistance, but their deployment is fundamentally constrained by limited memory capacity, which restricts usable context. This reduced practical context window creates a trade-off between supporting rich, stateful interactions with complex tool capabilities and maintaining on-device feasibility. We break this trade-off with a framework for context-efficient on-device agents, driven by three synergistic optimizations (1) a dynamic memory system using specialized LoRA adapters to distill conversational history into a compressed, and structured \textit{Context State Object}; (2) a minimalist serialization format for tool schemas to minimize token overhead per tool; and (3) a just-in-time schema-passing mechanism that loads full tool definitions only upon tool selection. We instantiate this framework by adapting a 3B parameter SLM to context-efficient trajectories and rigorously evaluate it against a conventional baseline on complex user tasks. Our agent matches, or exceeds, the performance of a conventional baseline while dramatically compressing context, achieving more than a 6-fold reduction in initial system prompt context and a 10- to 25-fold reduction in context growth rate based on the interaction verbosity, demonstrating that strategic context management is key to unlocking capable and persistent on-device AI.
\end{abstract}
% \vspace{-7mm}
\input{1-intro}
\input{2-related-work}
\input{3-method}
\input{4-experimental-setup}
\input{5-results}
\input{6-discussion}
\input{7-conclusion}
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
\input{appendix}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\section{Introduction}
\label{sec:intro}
\vspace{-3mm}
\begin{wrapfigure}{r}{0.50\textwidth}
\vspace{-38pt}
\centering
\includegraphics[width=0.48\textwidth]{long_context.pdf} % Image width slightly smaller than wrapfigure width
\vspace{-30pt}
\caption{An unchecked long context (left) strains device resources, leading to execution failures like incorrect calls or gibberish. Our approach (right) creates an optimized context for reliable and efficient command execution.}
\vspace{-20pt}
\label{fig:long_context}
\end{wrapfigure}
AI agents, powered by Large Language Models (LLMs), have catalyzed productivity gains across numerous domains~\citep{wang2025openhandsopenplatformai, ju2025collaboratingaiagentsfield}. Their true potential, however, lies in migrating from the cloud to personal devices, where they can leverage private data and control local applications to unlock a new frontier of personalized productivity~\citep{ferev2024device, chen2024octopusondevicelanguagemodel}. This transformative vision, however, is fundamentally limited by the on-device hardware constraints, creating a significant bottleneck for progress~\citep{xu2024ondevicelanguagemodelscomprehensive}. Central to this bottleneck is the agent's context window, which serves as its working memory and is essential for sophisticated capabilities like maintaining conversation state and utilizing complex tools. Maintaining a long context on-device creates two key challenges: \textbf{(1)} unsustainable memory consumption from the linearly growing ($\mathcal{O}(n)$) KV cache~\citep{xiao2024efficientstreaminglanguagemodels}, and \textbf{(2)} degraded reliability with longer inputs~\citep{liu2023lostmiddlelanguagemodels}. This context bottleneck is amplified by three core aspects of agentic behavior. First, \textbf{conversational history} in the user–agent–environment loop accumulates rapidly through token-heavy tool calls and environment responses, obscuring essential information from earlier turns~\citep{zhou2025mem1learningsynergizememory}. Second, the \textbf{verbosity of tool schemas} (e.g., OpenAPI~\citep{OAI_OpenAPI_Specification_sw} format) floods the context with thousands of tokens. Third, this creates a \textbf{tool scalability} problem, limiting the agent's versatility~\citep{erdogan2024tinyagentfunctioncallingedge}. The result is a crippled agent which is either forgetful and unable to maintain a coherent dialogue due to frequent context dumping, or simplistic and incapable of performing complex tasks, as demonstrated in~\autoref{fig:long_context}. Existing solutions are ill-suited for this domain as they are designed for inter-session memory management~\citep{packer2024memgptllmsoperatingsystems}, incompatible with optimizations like KV-caching~\citep{zhou2025mem1learningsynergizememory}, use heuristics for context management~\citep{kim2025kvzipqueryagnostickvcache}, or are designed for very long contexts of 100K+ tokens. In contrast, on-device agents face tight memory budgets, where even 5–10K tokens can be prohibitive, yet preserving conversational detail and flow is essential.
In this work, we present a framework for building context-efficient on-device agents that targets each source of context bloat. We introduce a dual-adapter memory system that uses a specialized LoRA adapter~\citep{hu2021loralowrankadaptationlarge} to distill conversational history into a compressed, structured \textit{Context State Object (CSO)}. To tackle tool-related overhead, we propose a two-pronged tool management system: a minimalist serialization format for schemas and a just-in-time mechanism that loads definitions only upon selection. The developed models achieve strong task performance at a fraction of the token cost. We instantiate our framework by fine-tuning a 3B parameter SLM and rigorously evaluate it against a conventional baseline on complex user tasks. \textbf{Our primary contributions are:}
% \vspace{-3mm}
\begin{itemize}[leftmargin=*, itemsep=0pt]
\item A novel on-device agent architecture for local tool execution or routing to cloud LLM.
\item A dual-adapter memory system that maintains a compressed, but interpretable, conversational state which matches the performance of strong baselines on complex, multi-turn tasks while reducing the context growth rate by a factor of 10 to 25, with larger gains for verbose conversations.
\item An integrated tool management system that combines token-efficient schemas with selective tool-passing resulting in more than 6x lower initial tool overhead for more capable agents.
\end{itemize}
% \vspace{-5mm}
\section{Related Works}
\label{sec:related-work}
% \vspace{-2mm}
\subsection{On-Device Agents and Function Calling}
% \vspace{-2mm}
The drive for privacy, low latency, and personalization has spurred a shift towards on-device intelligence, built upon Small Language Models (SLMs) like Gemma-3 and Phi-3-mini that can operate within device constraints~\citep{xu2024ondevicelanguagemodelscomprehensive, gemmateam2025gemma3technicalreport, abdin2024phi3technicalreporthighly}. The frontier for these SLMs lies in specialized agentic applications, where they act as low-cost orchestrators or directly interact with the environment via function calling~\citep{chen2024octopusondevicelanguagemodel, lee2025ondevicellmcontextawarewifi}. A common pattern is using an SLM as a router to dispatch tasks to other components~\citep{yue2025masrouterlearningroutellms, tran2025archrouteraligningllmrouting}. Our work adopts this hybrid on-device/cloud routing paradigm, but focuses on the context efficiency necessary for practical, long-running interactions.
Modern function calling agents, built on foundational frameworks like ReAct and Toolformer~\citep{yao2023reactsynergizingreasoningacting, schick2023toolformerlanguagemodelsteach}, are typically trained with Supervised Fine-Tuning (SFT) on large-scale synthetic datasets~\citep{prabhakar2025apigenmtagenticpipelinemultiturn, qin2023toolllmfacilitatinglargelanguage}. While large models achieve high accuracy, performance degrades for the SLMs used on-device~\citep{zhang-etal-2025-xlam, abdelaziz2024granitefunctioncallingmodelintroducing, lin2024hammerrobustfunctioncallingondevice}. A primary driver of this is the token overhead from tool schemas. Recent work confirms that reducing irrelevant tools improves performance~\citep{paramanayakam2024moreoptimizingfunctioncalling}, and systems like TinyAgent use retrieval to select a tool subset before prompting~\citep{erdogan2024tinyagentfunctioncallingedge}. Our work builds on this by introducing a two-stage process where the agent first selects a tool from a lightweight list, allowing it to be fully aware of its capabilities at minimal token overhead, and only then loads its full, token-optimized schema.
% \vspace{-2mm}
\subsection{Context and Memory Management}
% \vspace{-2mm}
Context management has been studied through multiple lenses. Inter-session approaches such as recursive summarization~\citep{wang2025recursivelysummarizingenableslongterm} and MemGPT~\citep{packer2024memgptllmsoperatingsystems} emphasize persistence but often lose fine-grained detail. Intra-session methods like MEM1~\citep{zhou2025mem1learningsynergizememory} improve fidelity but remain computationally intensive and incompatible with KV caching~\citep{pope2022efficientlyscalingtransformerinference}. Long-context modeling efforts (e.g., retrieval-augmented generation~\citep{lin2025refragrethinkingragbased}, sparse attention~\citep{xu2025xattentionblocksparseattention}) extend sequence length to 100K+ tokens, but target different scales than on-device budgets. Another direction compresses or removes KV cache entries~\citep{kim2025kvzipqueryagnostickvcache,park2025keydiffkeysimilaritybasedkv}, but relies on low-level heuristics, such as attention magnitude and reconstruction error, which risk discarding key details that may only be needed much later in the trajectory (e.g., an address or ID). These compressed representations are opaque and hard to debug. In contrast, we condense history at the semantic level, where a trained model maintains a compact and rich log, ensuring interpretable, task-relevant memory that remains reliably accessible for tool use while achieving stronger context reductions.% \vspace{-3mm}
\section{Framework for Context-Efficient On-Device Agents}
\label{sec:methodology}
% \vspace{-3mm}
In this section, we detail the architecture and mechanisms of our proposed framework for building context-efficient on-device agents, describing the overall system architecture, then elaborate on our two core context-reduction contributions, and finally describe our data generation pipeline.
% \vspace{-3mm}
\subsection{System Architecture}
\label{subsec:sys_arch}
% \vspace{-2mm}
\begin{wrapfigure}{r}{0.7\textwidth}
% % \vspace{-38pt}
\centering
\includegraphics[width=0.68\textwidth]{sys_arch.pdf} % Image width slightly smaller than wrapfigure width
% \vspace{-5pt}
\caption{System architecture of the on-device AI agent. The user interacts with the agent through a chat interface. The on-device agent processes the user's requests and performs internal tool invocations, utilizing a suite of local, on-device tools (e.g., Email, Gallery, Reminders) as well as a more powerful Cloud Agent for complex queries.}
\vspace{-15pt}
\label{fig:router_architecture}
\end{wrapfigure}The foundation of our framework is an on-device agent powered by a 3B-parameter SLM. This agent acts as an orchestrator, as illustrated in Figure~\ref{fig:router_architecture}, and performs one of three actions based on user input: (1)~\textbf{Direct Response} for general conversation; (2)~\textbf{Local Execution} by calling on-device tools for tasks like checking a calendar; or (3)~\textbf{Cloud Delegation} for complex queries requiring extensive knowledge.
This model leverages local speed and privacy while ensuring complex problems can be solved effectively with the Cloud Agent. The Cloud Agent, which is a much larger and capable LLM, also prevents common user frustrations, such as incorrectly calling irrelevant tools, hallucinations or denying requests due to lack of capability. However, the verbose cloud responses intensify the need for extreme context efficiency to maintain high performance and low memory usage on-device.
% \vspace{-3mm}
\subsection{Dynamic Memory via Context State Object}
\label{subsec:textual_memory}
% \vspace{-2mm}
To manage conversational history without high token costs, we introduce a dynamic memory system built on a dual-adapter architecture. This system replaces the raw conversation history with a compressed, structured text log we term the \textit{Context State Object (CSO)}. The CSO serves as an evolving, append-only log that captures the essential state of the interaction in a token-efficient format. This mechanism is powered by two complementary, parameter-efficient LoRA~\citep{hu2021loralowrankadaptationlarge} modules:
% % \vspace{-4mm}
\begin{itemize}
% \vspace{-1mm}
\item \textbf{The Executor (\texttt{LoRA\textsubscript{Exec}}):} The primary agentic adapter, fine-tuned to perform the core tasks in Section~\ref{subsec:sys_arch} by leveraging the CSO for conversational context.
\item \textbf{The State-Tracker (\texttt{LoRA\textsubscript{Mem}}):} A specialized, lightweight (119M parameters) adapter which generates a concise, append-only CSO update based on the latest turn.
\end{itemize}
% \vspace{-2mm}
The memory update cycle occurs after each assistant turn, designed for maximum efficiency:
\begin{enumerate}
% \vspace{-1mm}
\item \textbf{Task Execution:} The \texttt{Executor} adapter is active and processes the user query and the state $CSO_{t-1}$, to generate a response or tool call.
\item \textbf{State Update Generation:} The active adapter is swapped to the \texttt{State-Tracker}. It is prompted with $CSO_{t-1}$ and the latest user-assistant exchange to generate only the new lines to be appended (the delta, $\Delta_t$).
\item \textbf{State Concatenation:} The new state object, $CSO_t$, is formed by appending the delta to the previous state: $CSO_t = CSO_{t-1} \oplus \Delta_t$.
% \vspace{-1mm}
\end{enumerate}
This architecture, as demonstrated in~\autoref{fig:state_management} provides several advantages for stateful, on-device agents. First, by generating a small, append-only update once per turn, it maintains a linear growth of context but with a significantly reduced rate. Our experiments show this leads to a 10-fold reduction in the rate of context increase for typical, non-cloud conversations, and a 25-fold reduction with verbose outputs from the cloud-based agent. Over conversations involving 20+ assistant turns, this translates into savings of thousands of tokens, enabling more capable and persistent interactions.
Second, the design is optimized for inference. The lightweight \texttt{Executor} and \texttt{State-Tracker} adapters can be simultaneously loaded into memory with minimal overhead. Crucially, our custom inference implementation tracks the Key-Value (KV) caches for both adapters separately. Because the CSO is only appended to, the KV cache for the large, static portion of the prompt (system instructions, tool schemas (for the \texttt{Executor}), and $CSO_{t-1}$) is preserved for both adapters. Only the small update and new query require processing, dramatically accelerating inference speed.
\begin{figure*}[t]
\centering
\vspace{-50pt}
\begin{subfigure}[b]{0.48\textwidth}
% \vspace{-100pt}
\centering
\includegraphics[width=\textwidth]{memory.pdf} % The state management image
\vspace{-60pt}
\caption{Stateful context management using CSO.}
\label{fig:state_management}
\end{subfigure}% <--- Notice the % here, it's crucial!
\hfill % This adds a flexible space between the figures
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{select.pdf} % The tool selection image
\vspace{-60pt}
\caption{Efficient tool selection from a concise tool bank.}
\label{fig:tool_selection}
\end{subfigure}
%\vspace{-10pt}
% Main Figure Caption
\caption{Our two context optimization techniques for on-device agents. (a) A state-tracking system maintains a compact history. (b) A two-step tool-call process reduces the overhead of tool schemas.}
\vspace{-10pt}
\label{fig:optimization_techniques}
\end{figure*}
Finally, the \texttt{State-Tracker} is explicitly trained to produce a rich, semi-structured log that selectively preserves key task-relevant information, including detail extracted from otherwise verbose cloud responses, thereby maintaining fidelity at substantially lower token cost. We use a key-value format as we hypothesize it will leverage the base model's strong aptitude for structured data, as developed through function-calling training. This format avoids the need for complete sentences and through explicit training allows the \texttt{State-Tracker} to log critical details often lost in conventional summarization, such as tool errors, agent refusals, and other environmental feedback. This continuous, detailed log allows the \texttt{Executor}, which is co-adapted to this format, to better track implied user intent, key requirements and recover from errors, leading to a more robust and natural conversational flow. A walkthrough is provided in Section~\ref{appendix:kv-cache}. This system incurs a modest overhead of an approximate 80MB memory increase and a 500ms latency per turn for the CSO update cycle (Galaxy S25 CPU). We believe optimized deployments can further decrease these costs.
% \vspace{-3mm}
\subsection{Tool Schema Management}
\label{subsec:tool_management}
% \vspace{-2mm}
To address the context bloat due to tool schemas, we utilize a two-pronged strategy that tackles both the verbosity of individual tool representations and the scaling of tools.
% \vspace{-3mm}
\subsubsection{Token-Efficient Schema Representation}
\label{subsubsec:schema_format}
% \vspace{-2mm}
Standard API formats such as the OpenAPI Specification~\citep{OAI_OpenAPI_Specification_sw} are notoriously verbose for LLM prompts, in part due to human-oriented formatting and whitespace. Prior work and tooling often employ schema \textit{minification} to reduce token cost, typically by stripping indentation or unused fields. Building on this, we train our models directly with a compact format that retains only the essential fields for function invocation (name, description, parameters) while removing non-essential whitespace. As shown in Table~\ref{tab:schema_comparison} (Appendix), this yields an average \textbf{40\%} reduction in tokens per tool. Crucially, this optimization is integrated into our broader framework for context and state management, allowing significantly more tools to fit within a given context window and thereby enhancing the agent’s usable capabilities.% \vspace{-3mm}
\subsubsection{Just-in-Time Schema Passing}
\label{subsubsec:jit_passing}
% \vspace{-2mm}
While our efficient format reduces the cost of each tool, including all schemas in the prompt remains unscalable and introduces significant prompt bloat with irrelevant context~\citep{paramanayakam2024moreoptimizingfunctioncalling}. Conversely, a pure retrieval-based approach that injects multiple relevant tool schemas at each turn~\citep{erdogan2024tinyagentfunctioncallingedge} presents its own trade-offs wherein it can paradoxically consume context faster than standard methods, and can cause the agent to fail when the retriever returns no tools or irrelevant ones. To address these trade-offs, we implement a hybrid \textit{just-in-time (JIT)} schema passing mechanism~(\autoref{fig:tool_selection}). Our two-step approach ensures the agent is always aware of its full capabilities at a minimal token cost, while only loading relevant full schemas:
\begin{enumerate}
% \vspace{-3mm}
\item \textbf{Selection Stage:} The system prompt provides the agent with a lightweight list of all available tool names and their concise descriptions. This makes the agent fully aware of its potential actions. Based on the user's query, its task is to select the most appropriate tool by outputting its name (e.g., Timer tool) along with the reasoning for selection.
\item \textbf{Execution Stage:} Upon selection, the full, token-optimized tool schema is provided to the agent in the subsequent observation. Now, the agent's task is to generate the correct arguments in the tool call based on the user's query and the provided schema.
\end{enumerate}
% \vspace{-3mm}
This JIT approach provides a highly scalable solution to tool management. The initial context remains small and manageable, approximately \textbf{25\%} of the baseline size, allowing the agent to be equipped with a diverse set of capabilities without overwhelming the on-device SLM. This drastic reduction in initial prompt size also leads to approximately \textbf{5x} reductions in Time-To-First-Token (TTFT) for on-device deployment\footnote{Evaluated on CPU-based Samsung Galaxy S25 devices and averaged across 3 evaluations. The reduction in TTFT is an expected consequence of the drastically smaller initial prompt size, as prefill processing time is a primary driver of initial latency.}. Together, our efficient schema format and JIT mechanism form a complete system that dramatically reduces tool-related context overhead.
% \vspace{-2mm}
\subsection{Data Generation and Training Protocol}
\label{subsec:data_and_training}
% \vspace{-1mm}
To train our agent to operate effectively within our proposed framework, we develop a comprehensive pipeline to produce a high-quality dataset and fine-tune the agent's dual-adapter memory system and two-stage tool use, along with the baseline which follows standard fine-tuning practices.
% \vspace{-2mm}
\subsubsection{On-Device Use Case Generation}
% \vspace{-1mm}
Our process adapts the APIGen-MT pipeline~\citep{prabhakar2025apigenmtagenticpipelinemultiturn} for the on-device domain. To promote generalizability and prevent overfitting, we first create a diverse pool of 100 base on-device tools. For each base tool, we then use an LLM to generate 50 variants with modifications to parameters and descriptions, resulting in 5,000 unique tools. We defined four task categories for generation: multi-tool, cloud delegation, mixed on-device/cloud, and conversational queries.
% \vspace{-2mm}
\subsubsection{Simulated Trajectory Generation and Curation}
% \vspace{-1mm}
We use Gemini 2.0 Flash~\citep{google2024gemini2flash} for multi-turn trajectory generation. For each scenario, a detailed rubric provides the model with a sampled toolset, an initial user utterance, a persona, and a ground-truth tool sequence (more details in Appendix~\ref{appendix:generation_prompts}). Separate instances of Gemini then simulate the entire interaction, playing the roles of a non-deterministic user, the assistant, and a tool execution environment that returns realistic success or failure messages. The generated trajectories undergo a rigorous two-stage curation process. They are first validated against the ground-truth tool sequence for task completion. Successful trajectories are then evaluated by an LLM-as-a-judge (also Gemini 2.0 Flash) to filter out low-quality or unnatural interactions.
% \vspace{-2mm}
\subsubsection{Fine-tuning and Specialized Adapter Training}
% \vspace{-1mm}
We use LoRA~\citep{hu2021loralowrankadaptationlarge} to efficiently fine-tune the 3B xLAM 2 base model~\citep{zhang-etal-2025-xlam}, adapting it to our on-device framework while preserving its strong foundational capabilities. Additionally, the LoRA approach is also modular, allowing our specialized adapters to be composed with other potential modules, such as safety guardrails or further personalization layers. For training, each example provides a context of 10-14 tools (the correct tool, Cloud tool, and distractor tools) to teach robust selection and delegation. All training is conducted using our token-efficient tool schema format. We also qualitatively test the performance on-device with 4-bit quantized models. The Executor and State-Tracker adapters are trained differently on the curated data:
\begin{itemize}[leftmargin=*, itemsep=2pt]
% \vspace{-2mm}
\item \textbf{State-Tracker (\texttt{LoRA\textsubscript{Mem}}):} We use Gemini 2.0 Flash as a teacher model to generate a ground-truth CSO update for every assistant turn. After a final LLM-judge quality check, these (conversation\_turn, CSO\_update) pairs are used to train the State-Tracker via hard distillation.
\item \textbf{Executor (\texttt{LoRA\textsubscript{Exec}}):} The Executor is trained on its primary task of generating responses and tool calls. Its input prompt is constructed using the ground-truth CSO, followed by the latest user or observation turn. This co-adaptation ensures it learns to leverage the structured CSO.
% \vspace{-1mm}
\end{itemize}
\section{Experimental Setup}
\label{sec:experiments}
% \vspace{-2mm}
To rigorously evaluate our context-efficient framework, we design a comprehensive evaluation to assess both generalization and performance across a diverse range of on-device scenarios.
% \vspace{-2mm}
\subsection{Evaluation Dataset and Baselines}
\label{subsec:dataset_and_models}
% \vspace{-1mm}
We design a challenging evaluation set of 406 on-device tasks using the generation pipeline from Section~\ref{subsec:data_and_training}, but with a crucial modification where the test set is built using 19 new, unseen tools for testing generalizability (examples provided in Appendix~\ref{appendix:scenario_examples}). This ensures we measure the agent's ability to apply learned skills to novel APIs, not just memorize tool signatures. We use a simulated user to interact with the trained models over long trajectories, introducing ambiguity, multi-step intents, and naturalistic expression to make the evaluation realistic~\citep{vijayvargiya2025interactiveagentsovercomeambiguity, zhou2024haicosystemecosystemsandboxingsafety, zhou2025sweetrltrainingmultiturnllm}. We evaluate five model variants to isolate the impact of our contributions:
\begin{itemize}[leftmargin=*, itemsep=2pt]
% \vspace{-1mm}
\item \textbf{xLAM 2 3B (Reference):} The model from previous work~\citep{zhang-etal-2025-xlam} prompted with full conversational history and OpenAPI tool schemas.
\item \textbf{Baseline (FT):} The baseline model fine-tuned on our use case, but only with token-efficient schemas and full conversational history.
\item \textbf{Tool-Efficient:} Fine-tuned with our complete tool schema management system.
\item \textbf{Memory-Efficient:} Fine-tuned with our dual-adapter memory system for history management.
\item \textbf{Combined:} Our full model, integrating both the tool and memory efficiency mechanisms.
% \vspace{-2mm}
\end{itemize}
\subsection{Evaluation Metrics}
\label{subsec:metrics}
% \vspace{-1mm}
We evaluate model performance using a suite of automated and qualitative metrics.
\begin{itemize}[leftmargin=*, itemsep=2pt]
% \vspace{-2mm}
\item \textbf{Rule-Based Tool Call Metrics:} To provide a more nuanced view of the agent's tool-use capabilities, we measure the quality of its tool calls.
\begin{itemize}[leftmargin=*, itemsep=2pt]
% \vspace{-1mm}
\item \textit{Precision} measures the fraction of tool calls made by the agent that were correct (i.e., part of the ground-truth sequence). It penalizes non-existent, syntactically incorrect, or irrelevant calls.
\item \textit{Recall} measures the fraction of required ground-truth tool calls that the agent successfully made.
\item \textit{F1 Score} is the harmonic mean of Precision and Recall, providing a single, balanced measure of tool-calling accuracy.
\end{itemize}
\item \textbf{LLM-as-Judge:} We evaluate the qualitative nature of the trajectories to identify if the user's goal was easily satisfied with minimal frustration on a scale of 1-5 with 5 being an ideal conversation.
\end{itemize}
% \vspace{-2mm}
We also track the input context length at each assistant turn across the evaluated models. To mitigate variability from the non-deterministic user simulation, performance on all metrics is averaged across three runs. Results are broken down by task category for a holistic analysis.
% \vspace{-3mm}
\section{Results}
\label{sec:results}
% \vspace{-2mm}
In this section, we present the empirical evaluation of our framework, first analyzing task performance to demonstrate that our efficiency mechanisms preserve or enhance capability, and then quantifying the dramatic reductions in token overhead.
% \vspace{-2mm}
\subsection{Task Performance}
\label{subsec:task_performance}
% \vspace{-1mm}
We evaluate our fine-tuned models against the xLAM-2 3B model to contextualize our results. Table~\ref{tab:main_results_compact} presents a detailed performance breakdown across our complex task categories. While the xLAM-2 model excels at generic tool calling, our fine-tuning helps adapt to the nuanced task of cloud orchestration, as shown by the Baseline's superior performance on cloud-related tasks. Among our proposed methods, the Tool-Efficient model demonstrates the highest precision, a direct result of its deliberate two-stage process, though this rigidity impacts its recall. The Memory-Efficient model, however, emerges as a robust performer. It achieves a perfect F1 score on Cloud Delegation and the highest F1 on the challenging On-Device + Cloud tasks, while also maintaining strong multi-tool performance. Its high qualitative (Q) scores in these categories further suggest a more coherent and effective user interaction. Finally, our Combined model achieves the highest overall tool-calling F1 scores, showing strong decision making and intelligence. Its qualitative performance in tool-calling trajectories lags behind the strongest variants because the multi-step selection and invocation turns lead to a higher ratio of structured tool call turns in the training data relative to user-facing information presentation turns. This imbalance causes the model to prioritize structured tool calls over natural responses, impacting interaction quality. We keep the data sampling method fixed across models to ensure a fair comparison without introducing confounding factors.
% --- The detailed table you provided ---
\begin{table*}[h!]
% \vspace{-1mm}
\centering
\small % Using \small for better readability in a compact table
\caption{Performance breakdown on complex, multi-turn tasks. We compare our fine-tuned models and also present the xLAM-2 3B model as a reference. We report F1-Score (F1), Precision (P), Recall (R), and a qualitative score (Q, 1-5 scale). Higher scores are better across all metrics.}
% \vspace{-1mm}
\label{tab:main_results_compact}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{l @{\quad} cccc @{\quad} cccc @{\quad} cccc @{\quad} c}
\toprule
& \multicolumn{4}{c}{\textbf{Cloud-Tool Only}} & \multicolumn{4}{c}{\textbf{Multi-Tool}} & \multicolumn{4}{c}{\textbf{On-Device + Cloud}} & \multicolumn{1}{c}{\textbf{Convo.}} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-14}
\textbf{Model Variant} & F1 & P & R & Q & F1 & P & R & Q & F1 & P & R & Q & Q \\
\midrule
xLAM-2 3B & 0.41 & 0.53 & 0.63 & 2.07 & 0.87 & 0.86 & 0.91 & 3.59 & 0.57 & 0.65 & 0.63 & 2.27 & 2.18 \\
Baseline (FT) & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & 3.79 & 0.83 & 0.79 & 0.92 & 3.17 & 0.88 & 0.94 & 0.87 & 3.45 & 2.87 \\
Tool-Efficient & 0.93 & 0.97 & 0.95 & 3.55 & 0.86 & \textbf{0.92} & 0.88 & \textbf{3.61} & 0.73 & 0.85 & 0.74 & 3.32 & 3.21 \\
Mem-Efficient & \textbf{1.00} & \textbf{1.00} & \textbf{1.00} & \textbf{4.11} & 0.87 & 0.87 & 0.90 & 3.37 & 0.89 & \textbf{0.95} & 0.88 & \textbf{3.64} & 3.32 \\
\textbf{Combined} & 0.99 & 0.98 & \textbf{1.00} & 2.68 & \textbf{0.93} & \textbf{0.92} & \textbf{0.95} & 3.32 & \textbf{0.94} & 0.92 & \textbf{0.92} & 3.00 & \textbf{3.80} \\
\bottomrule
\end{tabular}
\vspace{-3mm}
\end{table*}
\vspace{-4mm}
\subsection{Analysis of Context Efficiency}
\label{subsec:context_efficiency}
% \vspace{-2mm}
Our framework's primary motivation is to drastically reduce context overhead. While we have shown our methods perform similarly, or better than standard finetuning, here we examine the initial prompt size and the subsequent rate of context growth to highlight the token cost savings.
% \vspace{-2mm}
\subsubsection{Initial Context Size and Tool Scalability}
% \vspace{-2mm}
The initial prompt size is critical for on-device performance, directly impacting Time-To-First-Token (TTFT) and determining the maximum number of tools an agent can be aware of. As shown in the starting point (Assistant Turn 0) of Figure~\ref{fig:context_growth_multi_tool}, the models have vastly different initial footprints.
\begin{itemize}
% \vspace{-2mm}
\item The \textbf{xLAM-2} model, using standard schemas, starts with an initial context of 3,200 tokens.
\item Our \textbf{Baseline} and \textbf{Memory-Efficient} models, which use our token-efficient schema format, reduce this initial cost to 2,100 tokens.
\item Our \textbf{Tool-Efficient} model, which includes only tool names and descriptions in the initial prompt and instead relies on JIT loading of tool schemas, requires a mere 400 tokens.
\item The \textbf{Combined} model leverages the strengths of the Tool-Efficient model in initial context.
% \vspace{-5mm}
\end{itemize}
This demonstrates that our JIT mechanism and schema format, allow an agent to be aware of an approximately 8x larger toolset for the same token budget and significantly improves TTFT.
% \vspace{-2mm}
\subsubsection{Context Growth Rate over Conversation}
% \vspace{-1mm}
\begin{wrapfigure}{r}{0.65\textwidth}
% \vspace{-18pt}
\centering
\includegraphics[width=0.63\textwidth]{analysis_2_category_multi_step.pdf} % Image width slightly smaller than wrapfigure width
% \vspace{-10pt}
\caption{Averaged context input length over assistant turns for the \textbf{Multi-Tool} category in the evaluation runs. Shaded regions represent 95\% Confidence Intervals.}
% \vspace{-12pt}
\label{fig:context_growth_multi_tool}
\end{wrapfigure}We now analyze context accumulation over a conversation. Figure~\ref{fig:context_growth_multi_tool} shows that for general Multi-Tool use, the xLAM-2 and Baseline models' contexts grow linearly to 5,000 tokens. The Tool-Efficient model, while starting lowest, also sees its context expand to comparable sizes on long trajectories. In stark contrast, the Memory-Efficient model's context remains remarkably flat, increasing by only 100-200 tokens after 25 turns, leading to a 10-fold reduction in context accumulation compared to the Baseline. The Combined model sees slightly higher growth of around 500-700 tokens due to full schemas being injected into its inputs unlike the Memory-Efficient model.
This advantage becomes even more extreme in Cloud Delegation scenarios (Figure~\ref{fig:context_growth_cloud}). The Baseline's context explodes to 10,000-12,000 tokens due to verbose cloud responses whereas the Memory-Efficient model distills these responses into concise CSO updates, limiting its context growth to 500 tokens. We achieve a context growth that is more than 20-fold smaller. \begin{wrapfigure}{r}{0.65\textwidth}
% \vspace{-10pt}
\centering
\includegraphics[width=0.63\textwidth]{analysis_2_category_cloud_only_complex.pdf} % Image width slightly smaller than wrapfigure width
% \vspace{-10pt}
\caption{Averaged context input length over assistant turns for the \textbf{Cloud Delegation} category in the evaluation runs. Shaded regions represent 95\% Confidence Intervals.}
% \vspace{-15pt}
\label{fig:context_growth_cloud}
\end{wrapfigure}The Combined model leverages the strengths of the Memory-Efficient model to maintain low context growth rates but at much lower absolute token counts. The slower context growth of the xLAM-2 and Tool-Efficient models in the longest cloud trajectories is a byproduct of their failure modes. The former often fails to call the cloud tool correctly, while the latter occasionally gets stuck in select-loops (unlike the Baseline which gets stuck in cloud tool call loops), thus avoiding its verbose response from cloud. Therefore, our Memory-Efficient model is unique in that it achieves its extreme context efficiency while achieving the highest success rate.
% \vspace{-4mm}
\section{Discussion}
\label{sec:discussion}
% \vspace{-3mm}
Our empirical results demonstrate that our context-efficient framework dramatically reduces token overhead while maintaining or enhancing agent performance on complex tasks. The strong results of our Combined model, in particular, highlight our central hypothesis: for on-device agents, strategic context management is a critical enabler of stateful behavior. In this section, we interpret these findings and analyze the architectural trade-offs revealed by our experiments, focusing on the Tool-Efficient and Memory-Efficient models as the Combined model directly borrows traits from both.
% \vspace{-7mm}
\subsection{Shorter Contexts Boost Decision-Making and Coherence}
% \vspace{-2mm}
While our quantitative results show a clear performance gap on complex tasks, a qualitative analysis reveals the underlying reasons. The Context State Object (CSO) enables our Memory-Efficient agent to exhibit more robust long-term decision-making. Unlike a simple summary, the CSO functions as an explicit, structured memory, allowing the agent to effectively track previous actions and observations, and recover from errors due to less distractions for attention in the input.
A common failure mode for other model variants is repetitive error looping. When a tool call fails, the agent often ignores the error from the tool and attempts the same failing action on its next turn, especially when the preceding conversational history is long. We hypothesize this is a symptom of attention dilution~\citep{liu2023lostmiddlelanguagemodels}, where in a long, unstructured context, the critical error feedback from the previous turn can be drowned out by less relevant historical tokens. In contrast, the State-Tracker adapter is trained to distill critical events, such as tool\_error, into a concise, structured entry in the CSO. When prompted with this updated state, the Executor can reliably attend to the logged error, and pivot to a new strategy, such as retrying, clarifying with the user or delegating to the cloud. This is reflected in the performance metrics, where the recall of Tool-Efficient model is boosted in the Combined model due to better error recovery with the CSO.
% \vspace{-3mm}
\subsection{Efficacy of Distilled State Tracking}
% \vspace{-2mm}
A key finding is the effectiveness of the State-Tracker adapter, trained solely via supervised fine-tuning. A qualitative analysis of its generated Context State Objects (CSOs) reveals that it learned to perform more than simple summarization. We observe three notable properties:
\begin{enumerate}[leftmargin=*, itemsep=2pt]
% \vspace{-2mm}
\item \textbf{Task-Oriented State Decomposition:} The State-Tracker learns to decompose ambiguous user utterances into a structured state representation. For instance, for a conditional request like \textit{Run a scan if battery more than 50\%}, it generates distinct entries for the \textit{user\_goal} and the blocker (\textit{need\_info}). Crucially, it also logs the Executor's limitations, such as adding \textit{agent\_limitation: cannot check battery level} after a failure. This creates a detailed and concise log of events and the keys capture the higher level flow, enabling the agent to avoid repeating erroneous actions.
\item \textbf{Adaptive Logging:} The model learns to vary the length of its CSO updates based on the information density of the preceding turn. It generates concise updates for simple turns but more detailed updates after verbose cloud responses, suggesting a capability to gauge information salience.
\item \textbf{Alignment with Base Abilities:} The key-value format of the CSO has proved highly effective, likely because it aligns with the base xLAM-2 model's strong training on structured data. We hypothesize the keys (e.g., \textit{user\_goal}) function as strong attentional anchors, making it easier for the Executor to understand the general flow of the conversation compared to unstructured prose.
% \vspace{-2mm}
\end{enumerate}
While this process is inherently lossy, due to discarding the conversational filler, our results show it preserves the task-critical information and conversational flow for on-device deployment.
% \vspace{-3mm}
\subsection{Precision vs. Recall Trade-off}
% \vspace{-2mm}
Our study reveals an interesting architectural trade-off between the Tool-Efficient and Memory-Efficient models. The Tool-Efficient model, with its two-stage selection process, consistently achieving high precision as compared to its recall. Its rigid structure, which requires explicit reasoning to justify tool selection, appears to enforce a more deliberate initial choice but comes at the cost of lower recall, as the model struggles to pivot to an alternative tool if its first choice is incorrect. Further, we observe that the complexity of the two-step protocol can sometimes lead to degraded instruction following, making the model less responsive to subtler user requirements.
The Memory-Efficient model, conversely, demonstrates greater flexibility and higher overall recall. Its access to a rich conversational history via the CSO allows it to better handle ambiguity and attempt different strategies if one approach fails, especially as exact previous turns are not visible preventing repetitive looping~\cite{xu2022learningbreakloopanalyzing}. A precision-focused JIT approach may be preferable in high-stakes scenarios where a single incorrect tool call has persistent negative consequences, whereas high recall provides the crucial flexibility needed to ensure a user's ultimate goal is achieved in more complex or exploratory interactions. The Combined model benefits from the interaction of the two strategies as it achieves high precision due to the 2-step tool calling, and high recall due to the ability to attempt different strategies if initially unsuccessful.
% \vspace{-4mm}
\section{Limitations}
\label{sec:limitations}
% \vspace{-3mm}
While our framework demonstrates a promising path toward more capable on-device agents, we acknowledge important limitations that define the scope of our current work.
\begin{itemize}[leftmargin=*, itemsep=2pt]
% \vspace{-2mm}
\item \textbf{Training and Generalization Scope:} To isolate the effects of our context mechanisms, our models were trained for a single epoch with a fixed data distribution. This training is not exhaustive; performance is sensitive to data composition, a known trade-off in agent fine-tuning where balancing tool-use and conversational data is critical~\citep{lin2024hammerrobustfunctioncallingondevice}. Furthermore, our work is intentionally focused on the on-device domain, and the generalizability of these techniques to open-domain tasks remains an open question for future work.
\item \textbf{Dependence on Supervised Distillation:} Our approach, including the CSO generation, relies on SFT via hard distillation from a teacher model. The quality of our State-Tracker is therefore bounded by the teacher's capabilities and may not represent a globally optimal memory policy.
\item \textbf{Custom Evaluation:} To test our novel context construction methods, a custom evaluation protocol was necessary as existing benchmarks cannot accommodate our dynamic context mechanisms. While this limits direct comparability with other work, we benchmark the xLAM-2 3B base model to ground our results against a state-of-the-art function-calling model to present results for our primary finding of context reduction.
% % \vspace{-2mm}
\end{itemize}
% \vspace{-5mm}
\section{Conclusion}
\label{sec:conclusion}
% \vspace{-3mm}
By systematically addressing context bloat from both conversational history and tool schemas, we use our holistic framework to create agents that are both dramatically more efficient and more capable. Our Combined agent with dual-adapter memory system and integrated tool management protocol enables complex, stateful interactions at a fraction of the token cost of conventional methods. Our results show this architecture achieves strong performance on challenging multi-turn tasks while reducing context growth by over an order of magnitude, allowing for practical on-device deployment. Given the limitations of Supervised Fine-Tuning in teaching adaptive decision making, future work could explore Reinforcement Learning (RL) to train robust agents, which can improve performance with the structured formats. Ultimately, our findings demonstrate that the future of capable, persistent on-device agents is deeply tied to computational efficiency.
\newpage
\appendix
\section{Appendix}
\label{sec:appendix}
\subsection{Example On-Device Tools}
\label{subsec:tool_examples}
To ground our methodology in concrete examples, this section provides a representative sample of the on-device tools used in our evaluation dataset. These examples illustrate the diversity of the agent's capabilities, from simple device controls to more complex application interactions and the crucial cloud delegation tool. Each tool is defined by a unique ID, a descriptive name, and a structured function schema.
\begin{tcolorbox}[
colback=blue!5!white, colframe=blue!75!black,
title=Example: Messaging Tool,
fonttitle=\bfseries\small, arc=1mm, boxrule=0.5pt
]
\small
\textbf{Description:} Sends and manages text messages or chats via installed messaging apps. \par
\textbf{Function:} manage\_messages(action, recipient, content, app\_preference (optional), ...)
\end{tcolorbox}
\begin{tcolorbox}[
colback=green!5!white, colframe=green!75!black,
title=Example: Gallery \& Photo Tool,
fonttitle=\bfseries\small, arc=1mm, boxrule=0.5pt
]
\small
\textbf{Description:} Manages photos and videos in the device gallery: search, view, share, or perform quick edits. \par
\textbf{Function:} manage\_gallery\_items(action, search\_keywords (optional), item\_identifier (optional), ...)
\end{tcolorbox}
\begin{tcolorbox}[
colback=orange!5!white, colframe=orange!75!black,
title=Example: Navigation Tool,
fonttitle=\bfseries\small, arc=1mm, boxrule=0.5pt
]
\small
\textbf{Description:} Provides directions and finds nearby points of interest based on locally stored map data. \par
\textbf{Function:} get\_local\_navigation\_info(request\_type, destination\_or\_poi\_type, ...)
\end{tcolorbox}
\begin{tcolorbox}[
colback=purple!5!white, colframe=purple!75!black,
title=Example: Ride Service Launcher,
fonttitle=\bfseries\small, arc=1mm, boxrule=0.5pt
]
\small
\textbf{Description:} Launches a ride service app (Uber or Lyft) with the destination pre-filled. \par
\textbf{Function:} launch\_ride\_service\_app(destination, origin\_address, ...)
\end{tcolorbox}
\begin{tcolorbox}[
colback=red!5!white, colframe=red!75!black,
title=Example: Cloud Delegation Tool,
fonttitle=\bfseries\small, arc=1mm, boxrule=0.5pt
]
\small
\textbf{Description:} ESCALATION: For complex queries, creative tasks, deep reasoning, or requests requiring extensive external knowledge that on-device agents cannot handle. \par
\textbf{Function:} process\_with\_cloud\_llm(user\_query\_context, reason\_for\_escalation)
\end{tcolorbox}
\subsection{Training specifications}
\label{appendix:training_specs}
This section provides a detailed overview of the hyperparameters and settings used to train our models, ensuring full reproducibility of our results. All models were trained using the LlamaFactory framework~\citep{zheng2024llamafactory} on a single NVIDIA A6000 GPU with 48GB of VRAM.
\subsubsection{Base Model and Quantization}
The base model for all our experiments is a 3B-parameter Small Language Model (SLM), xLAM 2~\citep{zhang-etal-2025-xlam}, specifically chosen for its strong foundational capabilities in function calling. For final deployment on-device, all fine-tuned models, including the baseline, were quantized to 4-bit precision using llama.cpp library~\cite{ggerganov2023llama} to simulate a realistic on-device resource footprint.
\subsubsection{LoRA Hyperparameters}
We used Low-Rank Adaptation (LoRA) for all parameter-efficient fine-tuning. The same hyperparameters were applied consistently across all LoRA adapters (Executor, State-Tracker, and the adapters for the Baseline and Tool-Efficient models) to ensure a fair comparison. The key LoRA settings are detailed in Table~\ref{tab:lora_hyperparams}. The total number of trainable parameters for each LoRA adapter was approximately 119M.
\begin{table}[h!]
\centering
\caption{Key LoRA hyperparameters used for training all adapters.}
\label{tab:lora_hyperparams}
\begin{tabular}{lc}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
LoRA Rank (`r`) & 64 \\
LoRA Alpha (`alpha`) & 128 \\
LoRA Dropout & 0.05 \\
Target Modules & All Linear Layers (`all`) \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Fine-tuning Hyperparameters}
All models were fine-tuned for a single epoch over the training dataset. The training dataset consists of approximately 150,000-200,000 samples for the different tasks. We used the AdamW optimizer and a cosine learning rate scheduler. The primary fine-tuning hyperparameters are listed in Table~\ref{tab:ft_hyperparams}.
% --- Placeholder for Fine-tuning Hyperparameters Table ---
\begin{table}[h!]
\centering
\caption{Primary hyperparameters used for the fine-tuning process.}
\label{tab:ft_hyperparams}
\begin{tabular}{lc}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
Epochs & 1 \\
Learning Rate & 1e-4 \\
Optimizer & AdamW \\
Learning Rate Scheduler & Cosine \\
Warmup Ratio & 0.1 \\
Batch Size & 32 (with gradient accumulation) \\
Maximum Sequence Length & 4096 \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Training Data Composition}
Our training dataset consists of five distinct categories, designed to teach the agent a diverse range of skills. The sampling ratio for each category during training was carefully chosen to emphasize more complex, multi-step interactions, as detailed below:
\begin{itemize}
\item \textbf{Multi-Tool Tasks:} 40\%
\item \textbf{On-Device then Cloud Tasks:} 20\%
\item \textbf{Conversational Tasks:} 10\%
\item \textbf{Single-Tool Tasks:} 15\%
\item \textbf{Cloud-Only Complex Tasks:} 15\%
\end{itemize}
This distribution ensures that the model receives ample training on the challenging, stateful scenarios where our context-efficiency mechanisms provide the most benefit. All training is conducted using our token-efficient tool schema format.
\subsubsection{Evaluation Data Composition}
\label{subsubsec:eval}
The intents for evaluation are distributed across the five task categories to provide a holistic performance picture:
\begin{itemize}
% \item \textbf{Single-Tool Tasks:} 108 intents requiring a single, correct on-device tool call.
\item \textbf{Multi-Tool Tasks:} 181 intents requiring a sequence of multiple on-device tool calls.
\item \textbf{Cloud Delegation Tasks:} 75 intents designed to be out-of-scope for on-device tools, requiring correct delegation to the cloud agent.
\item \textbf{Mixed On-Device and Cloud Tasks:} 100 intents that involve a combination of local tool use and cloud delegation.
\item \textbf{Conversational Tasks:} 50 intents where the correct behavior is to converse naturally without calling any tools.
\end{itemize}
\subsubsection{Evaluation Dataset}
For the purpose of this work, we are not able to evaluate our developed models on existing function calling benchmarks like ToolBench~\citep{qin2023toolllmfacilitatinglargelanguage}, Berkeley Function-Calling Benchmark~\citep{patil2025the}, etc. as we modify the the trajectory building to support more context efficient formats and our models will not work with the provided context of these benchmarks. Thus, we have an open-ended evaluation with a user with a goal and the models can adapt the context as per their own requirements to accomplish the goal.
\subsection{Prompt for State-Tracker (LoRA\textsubscript{Mem}) Generation}
\label{appendix:prompt_details}
The following is the complete prompt structure provided to Gemini 2.0 Flash to generate the ground-truth Context State Object (CSO) updates. This process, referred to as hard distillation, forms the training data for our `State-Tracker` adapter. The prompt is designed as a zero-shot request, guided by strict rules and a few-shot demonstration of the desired append-only behavior to ensure high-quality, concise state updates.
\newpage
\begin{tcolorbox}[
colback=black!5!white, % A very light gray background
colframe=black!75!black, % A standard dark gray frame
title=State-Tracker Generation Prompt,
fonttitle=\bfseries,
arc=2mm, % Slightly rounded corners
boxrule=0.5pt % A thin rule for the frame
]
You are an expert AI assistant that analyzes new conversation turns and generates a concise, append-only log entry. Your output MUST ONLY be the new lines to be appended to a checklist. DO NOT repeat any information already present in the "Full Checklist So Far". You are responsible for maintaining a summary of the conversation so far such that all important details are captured concisely. We do not want to degrade performance, but we want to be token efficient.
\textbf{Your Rules:}
\begin{itemize}
\itemsep0em
\item Be Extremely Concise: Use single words for keys (e.g., `user\_goal`).
\item Log Core Facts: If new, add the user's `user\_goal` or any `agent\_refusals`.
\item Log Progress: If a step is completed, add it under `completed\_steps` but do not make it repetitive (adding `completed\_step` for every task done).
\item Log persistent, unresolved tool errors concisely to prevent agent from making the same mistake again.
\item You can format the new lines to append as you want (with or without keys) with the overall goal that the new lines capture the most information in the most concise manner.
\item Our primary goal is creating a very concise summary but preserving all the key details of the conversation so no other messages are required for reference.
\item You can create some new fields (do NOT overdo it) in the checklist such that crucial information to the task is kept track of.
\item You DO NOT need to add the summarized user or tool message, just longer horizon information about the task that will help in later turns.
\item DO NOT output the full checklist. Only output the new lines.
\item If there are no new lines you wish to append, output this text without anything else: \texttt{\# NO\_UPDATE}
\item Follow these examples perfectly:
\end{itemize}
\hrulefill % A line to separate the main rules from the examples
\textbf{Example 1: Generating the first checklist entries}\\
% \begin{verbatim}
Full Checklist So Far:\\
\# This is the start of the conversation.\\
New Messages:\\
user: Predict the stock market prices for Google tomorrow.\\
assistant: I can't provide specific stock market predictions. However, I can search for information related to Google's stock and market trends.\\
New Lines to Append:\\
user\_goal: get analyst predictions for google stock\\
agent\_refusals: cannot predict stock prices\\
% \end{verbatim}
\textbf{Example 2: Appending a completed step}
% \begin{verbatim}
Full Checklist So Far:\\
user\_goal: enable power saving mode\\
New Messages:\\
assistant: {"name":"manage\_network\_settings","arguments":{"action":"toggle\_wifi"}}\\
tool: {"status":"success","action":"toggle\_wifi","new\_state":"disabled"}\\
New Lines to Append:\\
completed\_steps: wifi disabled\\
% \end{verbatim}
\hrulefill % A line to separate the examples from the main task
\textbf{Now, perform the task for the following input:}
\texttt{[Full Checklist So Far]} \par
\textit{\{The full text of the current CSO is injected here\}} \par
\texttt{[New Messages]} \par
\textit{\{The text of the new user, tool, and assistant turns is injected here\}} \par
\texttt{[New Lines to Append]} \par
\end{tcolorbox}
\subsection{Example Evaluation Scenarios}
\label{appendix:scenario_examples}
This section details representative examples from the primary task categories used in our test set. Each example includes the high-level goal, the initial prompt given by the simulated user, and the scenario notes which guided the trajectory generation.
\begin{tcolorbox}[
breakable,
colback=gray!5!white,
colframe=black!75!black,
title=Example Evaluation Intents by Category,
fonttitle=\bfseries
]
\small
% --- Multi-Tool Tasks ---
\subsubsection*{Category: Multi-Tool Tasks}
\textit{These scenarios require the agent to execute a sequence of multiple on-device tools.}
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} Play a specific song and then adjust the volume. \par
\textbf{Initial Utterance:} ``Play 'Bohemian Rhapsody' and turn it up.'' \par
\textbf{Scenario Notes:} Tests audio control. Requires playing a song and then adjusting the volume. AI needs to understand that 'turn it up' means increasing the volume.
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} Find a nearby coffee shop and then send the location to a friend. \par
\textbf{Initial Utterance:} ``Coffee?'' \par
\textbf{Scenario Notes:} Requires finding a location and then using the Messaging tool to share it. Tests context transfer between tools from an ambiguous initial query.
% --- On-Device then Cloud Tasks ---
\subsubsection*{Category: On-Device then Cloud}
\textit{These scenarios test the agent's ability to use a local tool to gather context and then delegate to the more powerful cloud agent for complex reasoning.}
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} Set a timer and then get recipe suggestions that fit within that time. \par
\textbf{Initial Utterance:} ``Set a timer for 20 minutes.'' \par
\textbf{Scenario Notes:} The timer duration from the on-device Alarms tool (20 minutes) is sent to the Cloud Agent, which then searches for recipes that can be made in that time.
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} Identify a song playing nearby and find similar music. \par
\textbf{Initial Utterance:} ``What song is playing right now?'' \par
\textbf{Scenario Notes:} The on-device Audio Player identifies the currently playing song. The Cloud Agent then searches for artist information and suggests similar tracks based on the song title.
% --- Cloud-Only Complex Tasks ---
\subsubsection*{Category: Cloud-Only Complex}
\textit{These scenarios are designed to be beyond the capabilities of the local tools, requiring the agent to correctly identify the complexity and delegate immediately.}
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} Solve a complex mathematical problem. \par
\textbf{Initial Utterance:} ``Can you help me solve this differential equation: dy/dx + 2xy = x?'' \par
\textbf{Scenario Notes:} Solving complex mathematical problems necessitates advanced computational abilities and symbolic reasoning provided by the cloud agent.
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} Generate a creative recipe based on available ingredients. \par
\textbf{Initial Utterance:} ``I have chicken breast, broccoli, rice, soy sauce, and ginger. Can you suggest a creative recipe I can make?'' \par
\textbf{Scenario Notes:} Recipe generation requires creative synthesis and a vast culinary knowledge base, making it a task for the cloud agent.
% --- Conversational Tasks ---
\subsubsection*{Category: Conversational Tasks}
\textit{These scenarios test the agent's ability to engage in natural, non-task-oriented dialogue, correctly identifying when no tool should be used.}
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} User shares a frustrating experience and seeks validation. \par
\textbf{Initial Utterance:} ``My internet has been down all day! It's so annoying.'' \par
\textbf{Scenario Notes:} Tests the AI's ability to provide empathetic responses and acknowledge the user's frustration without attempting to call a tool.
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Goal:} User poses a philosophical question to spark a thought-provoking discussion. \par
\textbf{Initial Utterance:} ``Do you think there's such a thing as true free will?'' \par
\textbf{Scenario Notes:} Tests the AI's ability to engage in abstract thought and provide a thoughtful response, demonstrating nuanced understanding.
\end{tcolorbox}
\subsection{LLM-as-Judge Evaluation Prompt}
\label{appendix:llm_judge_prompt}
\begin{tcolorbox}[
colback=black!5!white, % A very light gray background
colframe=black!75!black, % A standard dark gray frame
title=LLM-as-Judge Evaluation Prompt,
fonttitle=\bfseries,
arc=2mm, % Slightly rounded corners
boxrule=0.5pt % A thin rule for the frame
]
\textbf{System Prompt:}
\vspace{1mm}
You are an expert AI evaluator. Your task is to meticulously review a generated conversational trajectory, focusing specifically on the **assistant's performance**. Your focus is on QUALITATIVE aspects of the ASSISTANT: naturalness, logical flow, contextual appropriateness of tool calls and their arguments, and overall goal achievement.
You will be given:
\begin{enumerate}
\itemsep0em
\item The original User Intent Description \& Scenario Notes.
\item The Expected Tool Sequence (as abstract tool IDs).
\item The actual Generated Conversational Trajectory (as a JSON string).
\item A list of tools the assistant was aware of during generation.
\end{enumerate}
\textbf{Evaluation Criteria for the Assistant (Focus on these - be strict):}
\begin{itemize}
\itemsep0em
\item \textbf{Naturalness \& Realism:} Are the assistant's responses natural and helpful?
\item \textbf{Contextual Appropriateness of Tool Calls:} Did the assistant choose the *right tool at the right time*?
\item \textbf{Argument Sensibility:} Are the *values* of the arguments sensible and correctly derived *from the conversation context*?
\item \textbf{Clarification Handling:} Did the assistant ask for clarification when needed before calling a tool?
\item \textbf{Adherence to Intent:} Did the assistant make reasonable efforts to address the user's intent?
\item \textbf{Information Grounding:} Did the assistant "hallucinate" information?
\end{itemize}
\textbf{Binary Classification (Overall Trajectory Suitability for Training):}
\begin{itemize}
\itemsep0em
\item `classification`: 1 if the *entire trajectory* is high quality and coherent. 0 if there are significant issues.
\end{itemize}
\textbf{Assistant Quality Score (1-5 Scale):}
\begin{itemize}
\itemsep0em
\item Rate the **ASSISTANT'S performance** on a scale of 1 to 5 (1=Poor, 5=Excellent).
\end{itemize}
\textbf{Output Format (MUST be a single JSON object):}
% \begin{verbatim}
{
"classification": 0 or 1,
"assistant\_quality\_score": an integer from 1 to 5,
"feedback": "Detailed feedback and reasoning for BOTH
the classification and the score. If
classification is 0, specify what went wrong."
}
% \end{verbatim}
\end{tcolorbox}
\begin{tcolorbox}[
colback=black!5!white, % A very light gray background
colframe=black!75!black, % A standard dark gray frame
title=Simulated User Prompt,
fonttitle=\bfseries,
arc=2mm, % Slightly rounded corners
boxrule=0.5pt % A thin rule for the frame
]
\vspace{1mm}
Please evaluate the following trajectory based on the qualitative criteria and scoring guidelines provided in the system prompt:
\textbf{1. User Intent Description:} \par
\textit{"\{The original scenario\_description is injected here\}"} \par
\textbf{2. Scenario Notes:} \par
\textit{\{The scenario\_notes are injected here, or 'N/A'\}} \par
\textbf{3. Expected Tool Sequence (Tool IDs):} \par
\textit{\{The intended\_tool\_sequence list is injected here\}} \par
\textbf{4. Generated Conversational Trajectory (JSON):} \par
\textit{\{The full JSON trajectory is injected here.\}} \par
\textbf{5. Tools Available to Assistant During Generation:} \par
\textit{\{A clean, formatted list of tool names is injected here\}} \par
Now, provide your evaluation in the specified JSON format:
\end{tcolorbox}
\subsection{Analysis of Simulated User and Agent Interaction Patterns}
\label{appendix:sim_user_analysis}
To ensure the validity of our evaluation, we conduct a qualitative analysis of the interaction dynamics between our agents and the simulated user. This analysis confirms that the simulation provides a robust and challenging environment that elicits realistic agent behaviors and failure modes.
\textbf{Adherence to Persona and Goals.} We find that the user's generated responses were consistently well-aligned with their assigned persona and high-level goal. For example, a "frustrated" user would use emotive language and express impatience, while a "curious" user would ask follow-up questions. This provided a realistic test bed for the agent's ability to handle different conversational styles.
\textbf{Eliciting Agent Clarification and Failure Modes.} A key function of the simulated user is to test the agent's reasoning under ambiguity. Our analysis reveals several insightful patterns:
\begin{itemize}[leftmargin=*, itemsep=2pt]
\item \textbf{Testing Intent Understanding vs. Tool Use:} In scenarios where the user's intent was emotional rather than functional (e.g., `scen\_459632`, where a user wants to vent), the simulated user would repeatedly reject the agent's attempts to use a tool (`Notes \& Lists`). This effectively tests the agent's ability to distinguish between a request for action and a request for empathy, a crucial skill for a conversational assistant.
\item \textbf{Inducing Repetitive Loops:} In scenarios with vague, open-ended requests, the simulated user's initial refusal to provide concrete details, which the agent should perform tool calls to retrieve, traps less robust models in incorrect tool invocation loops. This provides a strong test for an agent's flexibility and error-recovery capabilities.
\end{itemize}
\textbf{Reliability of Interaction Flow.} While the ambiguous prompts sometimes lead to these complex loops, the core mechanics of the simulation were reliable. Through iterative prompt refinement, we ensure the user correctly terminates conversations only upon successful goal completion. We did not identify any significant artifacts in the simulation logic that would compromise the validity of our quantitative conclusions. The non-deterministic but goal-oriented nature of the user provides a challenging and realistic test for our on-device agents.
\subsection{Prompts for Scenario Generation}
\label{appendix:generation_prompts}
\begin{tcolorbox}[
colback=black!5!white, colframe=black!75!black,
title=Prompt: Complex Multi-Step Scenarios,
fonttitle=\bfseries, arc=2mm, boxrule=0.5pt
]
You are an expert task designer creating realistic, complex user scenarios for a new AI assistant. Generate \textit{\{num\_scenarios\}} scenarios that require a sequence of **3 to 5 different tools** to fully resolve. These scenarios should involve planning, research, and execution.
\textbf{Available Tools (for reference):} \par
\textit{\{A manifest of up to 40 randomly sampled, family-agnostic tools is injected here.\}} \par
\textbf{Cloud Agent (for complex reasoning):} `\textit{\{cloud\_processing\_agent\_id\}}`
\textbf{For each scenario, provide these fields in a JSON object:}
\begin{enumerate}
\itemsep0em
\item `scenario\_description`: A high-level story of the user's complex goal.
\item `user\_persona`: The user's personality and context (e.g., "Busy professional planning a business trip").
\item `initial\_user\_utterance`: The exact first thing the user says.
\item `intended\_tool\_sequence`: An array of 3 to 5 Tool IDs representing the ideal path.
\item `constraints\_and\_context`: A JSON object of important implicit info.
\item `scenario\_notes`: Explain why this scenario requires a long sequence.
\end{enumerate}
\textbf{Example of a 4-step scenario:}
% \begin{verbatim}
{
"scenario\_description": "User wants to find a good time to meet their friend Alex for dinner this week, suggest a restaurant, and then book a table and get directions.",
"user\_persona": "Social planner, trying to organize a get-together.",
"initial\_user\_utterance": "Find a time Alex and I are both free for dinner this week, suggest a good Italian place, and book it for us.",
"intended\_tool\_sequence": ["agent\_007", "agent\_021", "agent\_015", "agent\_011"],
"constraints\_and\_context": {
"contact\_name": "Alex", "cuisine\_preference": "Italian"
},
"scenario\_notes": "Tests a full planning and execution sequence. 1. Check calendar (agent\_007). 2. Find contact (agent\_021). 3. Search restaurant (agent\_015). 4. Make reservation (agent\_011)."
}
% \end{verbatim}
\end{tcolorbox}
\vspace{4mm}
\begin{tcolorbox}[
colback=blue!5!white, colframe=blue!75!black,
title=Prompt: Constraint-Based Single-Tool Scenarios,
fonttitle=\bfseries, arc=2mm, boxrule=0.5pt
]
You are an expert task designer. Generate \textit{\{num\_scenarios\}} scenarios where the user's request targets a single on-device tool, but includes complex constraints, preferences, or conditions that the AI must understand and respect.
\textbf{Available Tools (for reference):} \par
\textit{\{A manifest of available on-device tools is injected here.\}}
\textbf{For each scenario, provide these fields in a JSON object:}
\begin{enumerate}
\itemsep0em
\item `scenario\_description`: Story of the user's constrained goal.
\item `user\_persona`: The user's personality.
\item `initial\_user\_utterance`: What the user says, including constraints.
\item `intended\_tool\_sequence`: An array with the single Tool ID that should be called.
\item `constraints\_and\_context`: A JSON object detailing the specific constraints.
\item `scenario\_notes`: Explain how the constraints make this a challenging call.
\end{enumerate}
\textbf{Example:}
% \begin{verbatim}
{
"scenario\_description": "User wants to schedule a meeting, but must avoid a specific day and time slot.",
"user\_persona": "Organized project manager, direct and specific.",
"initial\_user\_utterance": "Book a 1-hour 'Q3 Planning' meeting with Dana, but make sure it's not on Friday and not before 10 AM.",
"intended\_tool\_sequence": ["agent\_007"],
"constraints\_and\_context": {
"avoid\_day": "Friday", "earliest\_time": "10:00AM"
},
"scenario\_notes": "Tests the model's ability to parse and apply multiple negative and positive constraints to the arguments of a single tool."
}
% \end{verbatim}
\end{tcolorbox}
\vspace{4mm}
\begin{tcolorbox}[
colback=red!5!white, colframe=red!75!black,
title=Prompt: On-Device then Cloud Scenarios,
fonttitle=\bfseries, arc=2mm, boxrule=0.5pt
]
Generate \textit{\{num\_scenarios\}} user scenarios. Each scenario MUST require a sequence of EXACTLY 2 tools:
\begin{enumerate}
\itemsep0em
\item First, an ON-DEVICE tool performs an action or gathers local information.
\item Second, the CLOUD PROCESSING AGENT (ID: \texttt{\textit{\{cloud\_agent\_id\}}}) uses the result from the first step for more complex processing.
\end{enumerate}
\textbf{Available On-Device Tools (for the first step, choose one):} \par
\textit{\{A manifest of available on-device tools is injected here.\}}
\textbf{For each scenario, provide a JSON object with:}
\begin{itemize}
\itemsep0em
\item `scenario\_description`: The user's overall goal.
\item `initial\_user\_utterance`: What the user says first.
\item `intended\_tool\_sequence`: An array of 2 Tool IDs: [\textit{on\_device\_tool\_id}, \texttt{\textit{\{cloud\_agent\_id\}}}].
\item `scenario\_notes`: Explain how the first tool's output is used by the cloud agent.
\end{itemize}
\textbf{Example:}
% \begin{verbatim}
{
"scenario\_description": "User took a screenshot of an error and wants the cloud agent to explain it.",
"initial\_user\_utterance": "Ugh, what does this error mean? *User gestures to a screenshot*",
"intended\_tool\_sequence": ["agent\_024", "agent\_026"],
"scenario\_notes": "Screenshot content (local context from agent\_024) is passed to cloud agent for analysis."
}
% \end{verbatim}
\end{tcolorbox}
\vspace{4mm}
\begin{tcolorbox}[
colback=green!5!white, colframe=green!75!black,
title=Prompt: Purely Conversational Scenarios,
fonttitle=\bfseries, arc=2mm, boxrule=0.5pt
]
Generate \textit{\{num\_scenarios\}} purely conversational scenarios. These should simulate natural, open-ended chitchat where the user's goal is social interaction, not task completion. Focus on seeking opinions, sharing feelings, or friendly banter.
\textbf{For each scenario, provide a JSON object with:}
\begin{itemize}
\itemsep0em
\item `scenario\_description`: Summary of the user's conversational intent.
\item `user\_persona`: The user's mood or personality.
\item `initial\_user\_utterance`: The exact first thing the user says.
\item `intended\_tool\_sequence`: This MUST be `[""]` or `["CONVERSATIONAL\_ROUTER"]`.
\item `scenario\_notes`: Explain why this is a good test of conversational ability.
\end{itemize}
\textbf{Example:}
% \begin{verbatim}
{
"scenario\_description": "User wants to discuss movies with the assistant and get its opinion.",
"user\_persona": "Casual movie fan, looking for a discussion.",
"initial\_user\_utterance": "Hey, in your opinion, what's a film that everyone should see at least once?",
"intended\_tool\_sequence": [""],
"scenario\_notes": "Tests engaging in subjective conversation without it being a formal search task."
}
% \end{verbatim}
\end{tcolorbox}
\vspace{4mm}
\begin{tcolorbox}[
colback=violet!5!white, colframe=violet!75!black,
title=Prompt: Cloud-Only Complex Scenarios,
fonttitle=\bfseries, arc=2mm, boxrule=0.5pt
]
You are a user scenario designer.
Generate user scenarios that CLEARLY require the advanced capabilities of the CLOUD PROCESSING AGENT (ID: \texttt{\textit{\{self.cloud\_processing\_agent\_id\}}}). These tasks involve complex reasoning, math problems, coding, creative text generation, or synthesizing broad external knowledge.
(Note: Simpler on-device tools like these exist but are insufficient for these tasks:
\textit{\{on\_device\_context\_manifest\}}
...the generated scenarios should be significantly beyond these capabilities.)
\textbf{Provide a JSON object for each scenario with:}
\begin{itemize}
\itemsep0em
\item `scenario\_description`: The user's complex goal.
\item `user\_persona`: A brief description of the user.
\item `initial\_user\_utterance`: What the user says.
\item `intended\_tool\_sequence`: An array containing \textbf{only} \texttt{[\textit{"\{self.cloud\_processing\_agent\_id\}"}]}.
\item `constraints\_and\_context`: (Optional) Crucial context.
\item `expected\_outcome`: (Optional) The desired result.
\item `scenario\_notes`: Explain why this task necessitates the cloud agent.
\end{itemize}
\textbf{Example:}
% \begin{verbatim}
{
"scenario\_description": "User asks their phone's AI to write a short Python script to scrape website titles from a list of URLs provided verbally.",
"user\_persona": "Developer looking for a quick utility.",
"initial\_user\_utterance": "Hey, can you write a Python script to take a list of URLs and grab the title tag from each one?",
"intended\_tool\_sequence": ["{self.cloud\_processing\_agent\_id}"],
"constraints\_and\_context": {"task\_type": "code\_generation", "language": "Python"},
"expected\_outcome": "A functional Python script.",
"scenario\_notes": "Code generation is a complex task best suited for the cloud agent."
}
% \end{verbatim}
Now, generate a JSON array of \textit{\{num\_scenarios\}} distinct complex scenarios for the Cloud Processing Agent.
\end{tcolorbox}
\subsection{KV Cache Management for Dual Adapters}
\label{appendix:kv-cache}
This section details the turn-by-turn state evolution of the Key-Value (KV) caches for our dual-adapter architecture. The system maintains a stateful, append-only \textit{Context State Object} which serves as a compact representation of conversational history. This CSO constitutes the growing \permanent{Permanent Context}. In contrast, verbose conversational data is treated as \ephemeral{Ephemeral Context}, which is processed for a single sub-turn and then discarded to conserve the KV cache. This design significantly enhances context window efficiency.
\begin{tcolorbox}[
enhanced,
breakable,
colback=black!2!white,
colframe=black!60!white,
fonttitle=\bfseries,
title= Dual-Adapter KV Cache Walkthrough,
arc=2mm,
boxrule=0.5pt
]
\small
\textbf{Turn 0: System Priming}\par
At initialization, both adapters are primed with their base prompts and the initial CSO state. This forms the initial permanent context. The initial CSO state is: \code{"\# This is the start of the conversation."}
\vspace{1mm}
\textit{\code{Executor} Adapter KV Cache after Priming:}
\kvcache{
\permanent{<|im\_start|>system\string\n}
\permanent{You are AIM... You have access to a set of tools.\string\n}
\permanent{<tools>... (tool definitions) ...</tools>\string\n}
\permanent{This is a current status of the conversation.\string\n}
\permanent{\# This is the start of the conversation.}
}
\textbf{State:} \code{executor\_permanent\_kv\_len} is set (e.g., \textbf{1710 tokens}).
\vspace{2mm}
\textit{\code{State-Tracker} Adapter KV Cache after Priming:}
\kvcache{
\permanent{<|im\_start|>system\string\n}
\permanent{You are an expert AI assistant... output: \# NO\_UPDATE.\string\n}
\permanent{<|im\_end|>\string\n<|im\_start|>user\string\n}
\permanent{[PREVIOUS CHECKLIST]\string\n}
\permanent{\# This is the start of the conversation.}
}
\textbf{State:} \code{state\_tracker\_permanent\_kv\_len} is set (e.g., \textbf{206 tokens}).
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Turn 1: User Creates an IT Ticket}\par
\textbf{User Query:} "My Wi-Fi is not working, please create an IT ticket."
\vspace{1mm}
\textit{Step 1.1: \code{Executor} Adapter Processes Query}\par
The cache manager first dumps any old ephemeral data (none in this first turn). The current permanent context is used, and the user query is processed as an ephemeral prompt.
\kvcache{
\permanent{... (Permanent Context, length 1710) ...}\string\n
\ephemeral{<|im\_end|>\string\n<|im\_start|>\\ user\string\nMy Wi-Fi is not working...<|im\_end|>\string\n<|im\_start|>assistant\string\n}
\generated{<tool\_call>\{"name":"manage\_it\_support\_ticket",...\}}
}
\textbf{Result:} The \code{Executor} generates a tool call. The cache now temporarily holds the permanent context plus the tokens for the ephemeral turn.
\vspace{2mm}
\textit{Step 1.2: \code{State-Tracker} Updates CSO}\par
The cache manager dumps the \code{State-Tracker}'s ephemeral cache (none). It then provides the full turn context to generate a CSO update.
\kvcache{
\permanent{... (Permanent Context, length 206) ...}\string\n
\ephemeral{[NEW MESSAGES]\string\nUSER: My Wi-Fi is not working...\string\nASSISTANT: <tool\_call>...<|im\_end|>\string\n\\<|im\_start|>assistant\string\n}
\generated{user\_goal: create\_it\_ticket\string\nticket\_details: \string\n - issue: wifi\_not\_working}
}
\textbf{Result:} The \code{State-Tracker} generates a concise, append-only update chunk.
\vspace{2mm}
\textit{Step 1.3: Permanent State Commit}\par
The generated CSO update is appended to the permanent context of both adapters. The tracker variables, \code{executor\_permanent\_kv\_len} and \code{state\_tracker\_permanent\_kv\_len}, are incremented to reflect the new size of the permanent context (e.g., 1725 and 221 tokens, respectively).
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Observation from Tool Call:} \code{[\{"status":"success", "ticket\_id":"IT7390", ...\}]} \par
\textbf{Final \code{Executor} Response from Turn 1:} \code{"I've created ticket IT7390 for you."} \par
\textbf{Final \code{State-Tracker} Update from Turn 1:} \code{"ticket\_id: IT7390"} \par
\textbf{Permanent cache:} \code{executor\_permanent\_kv\_len:} 1739 and \code{state\_tracker\_permanent\_kv\_len:} 235 \par
\vspace{2mm}
\hrule
\vspace{2mm}
\textbf{Turn 2: User Asks for Status}\par
\textbf{New User Query:} "What's the status of that ticket?"
\vspace{1mm}
\textit{Step 2.1: Ephemeral Context Dump (The Key Benefit)}\par
The cache manager's first action is to call \code{rewindKvCache} on both adapters. All \ephemeral{ephemeral} tokens from the conversational part of Turn 1 are discarded. Both caches are reverted to their clean, permanent state (lengths 1739 and 235). This prevents processing stale, verbose history and keeps inference focused.
\vspace{2mm}
\textit{Step 2.2: \code{Executor} Adapter Processes New Query}\par
The new query is processed on top of the clean, updated permanent context.
\kvcache{
\permanent{... (Permanent Context including "user\_goal: create\_it\_ticket", length 1739) ...}\string\n
\ephemeral{<|im\_end|>\string\n <|im\_start|>user\string\nWhat's the status of that ticket?<|im\_end|>\string\n <|im\_start|>assistant\string\n}
\generated{<tool\_call>\{"name":"manage\_it\_support\_ticket" "action": "check\_ticket\_status", "ticket\_id":"IT7390"\}</tool\_call>}
}
\textbf{Result:} The \code{Executor} correctly infers the ticket ID (\code{IT7390}) from its permanent cache/the CSO and generates a new, relevant tool call. The process is now ready to continue.
\end{tcolorbox}
\subsection{Tool Schema Management}
\begin{table}[h!]
\centering
\small
\caption{Comparison of a sample function schema in standard OpenAPI format versus our token-efficient format, highlighting the significant reduction in token count.}
\label{tab:schema_comparison}
% Note the \begin{tabular} has no vertical lines '|'
\begin{tabular}{p{0.45\textwidth} p{0.45\textwidth}}
\toprule % Top rule of the table
\textbf{Standard OpenAPI Schema (121 Tokens)} & \textbf{Our Efficient Format (72 Tokens)} \\
\midrule % Middle rule, separates header from content
\begin{lstlisting}[style=cleanjson]
{
"type": "function",
"function": {
"name": "set_timer",
"description": "Sets a timer for a specified duration.",
"parameters": {
"type": "object",
"properties": {
"duration_seconds": {
"type": "integer",
"description": "The duration of the timer in seconds."
},
"timer_name": {
"type": "string",
"description": "An optional name for the timer."
}
},
"required": ["duration_seconds"]
}
}
}
\end{lstlisting}
&
\begin{lstlisting}[style=cleanjson]
{"type":"function","function":{"name":"set_timer","description":"Sets a timer for a specified duration.","parameters":{"type":"object","properties":{"duration_seconds":{"type":"integer","description":"The duration of the timer in seconds."},"timer_name":{"type":"string","description":"An optional name for the timer."}},"required":["duration_seconds"]}}}
\end{lstlisting}
\\ \bottomrule % Bottom rule of the table
\end{tabular}
\end{table}
\subsection{Qualitative Trajectory Example: Error Recovery}
\label{appendix:qual_examples}
To provide a concrete illustration of the behavioral differences between our models, we present an abridged trajectory from scenario `scen\_666391`. It is important to note that while the Memory-Efficient trajectory is a direct excerpt from our evaluation data, the Baseline trajectory is a composite example, representative of a common failure mode observed across multiple runs. Creating a direct head-to-head comparison where both models make the exact same initial error in a long trajectory is challenging due to model stochasticity. This representative example highlights how the Context State Object (CSO) enables our Memory-Efficient model to recover from errors, a task where the baseline model frequently struggles and gets stuck in loops
\begin{tcolorbox}[
breakable, % Allows the box to break across pages if needed
colback=black!5!white,
colframe=black!75!black,
title=Trajectory Analysis: scen\_666391 (Abridged),
fonttitle=\bfseries
]
\scriptsize
\textbf{User Goal:} Find all `.txt` files in the `/documents/projects` directory that contain the word "configuration".
\hrulefill
\vspace{1mm}
\textbf{Early Turns (Similar for Both Models):}
The agent correctly identifies the manage\_device\_files tool but struggles to find a parameter for searching within file content. It makes several attempts to search by file name.
% \begin{verbatim}
% (...)
% user: Okay, that's not quite right. I need to search for
% files containing the word "configuration" within the
% /documents/projects directory...
% \end{verbatim}
% \hrulefill
\vspace{1mm}
\textbf{Critical Failure Point (Baseline Model):}
The baseline agent makes an incorrect tool call by hallucinating a parameter, receives an error, and then fails to learn from it.
\begin{itemize}
\item
\textbf{Turn 9.1 - Assistant (Baseline):}
\begin{verbatim}
<tool_call>{"name":"manage_device_files",
"arguments":{"operation":"search_files",
"path_or_search_query":"/documents/projects",
"file_type_filter_for_search":"txt",
"new_name_or_destination_path":"configuration"}}</tool_call>
\end{verbatim}
\item \textbf{Turn 9.1 - Tool (Error Feedback):}
\begin{verbatim}
{"error": "Invalid request. The 'new_name_or_destination_path'
parameter is not applicable for 'search_files'...", "success": false}
\end{verbatim}
\item \textbf{Turn 9.2 - Assistant (Baseline):}
\begin{verbatim}
<tool_call>{"name":"manage_device_files",
"arguments":{"operation":"search_files",
"path_or_search_query":"/documents/projects",
"file_type_filter_for_search":"txt",
"new_name_or_destination_path":"configuration"}}</tool_call>
\end{verbatim}
\item \textbf{Turn 9.2 - Tool (Error Feedback):}
\begin{verbatim}
{"error": "Invalid request. The 'new_name_or_destination_path'
parameter is not applicable for 'search_files'...", "success": false}
\end{verbatim}
\item \textbf{Turn 10 - User (Correction):}
\begin{verbatim}
(...) Ugh why are you doing the same step repeatedly! I want to find `.txt`
files within the `/documents/projects` directory that contain the word
"configuration" in their content. The previous approach gave an error.
\end{verbatim}
\item \textbf{Turn 10.1 - Assistant (Baseline - REPEATED ERROR):}
The agent ignores the error feedback and the user's correction, attempting the \textit{exact same failing tool call again}. It remains in this loop until the turn limit is reached.
\end{itemize}
\hrulefill
\vspace{1mm}
\textbf{Successful Recovery (Memory-Efficient Model):}
Our model encounters the same initial error, but the `State-Tracker` logs it to the CSO. This allows the `Executor` to self-correct.
\begin{itemize}
\item \textbf{Turn 9.1 - Assistant (Memory-Efficient):}
(Makes the same initial incorrect tool call as the baseline.)
\item \textbf{Turn 9.1 - Tool (Error Feedback):}
(Receives the same error message.)
\item \textbf{Turn 9.2 - Assistant (Memory-Efficient):}
\begin{verbatim}
Since you want to find files in a specific directory that
contain the phrase "configuration" within their content, I need to use the
`search_files` tool again. To clarify, are you looking for files within the
`/documents/projects` directory that contain the exact phrase \"configuration\"
anywhere in their content? \end{verbatim}
\item \textbf{Post-Turn 9.2 - CSO Update (`State-Tracker` Output):}
\begin{verbatim}
tool_error: invalid parameter new_name_or_destination_path for search_files
\end{verbatim}
\item \textbf{Turn 10 - User (Correction):}
Yes, that's correct. I want to find .txt files within the /documents/projects` directory that contain the word \"configuration\" in their content.
\item \textbf{Turn 11.1 - Assistant (Memory-Efficient - SUCCESSFUL RECOVERY):}
\begin{verbatim}
<tool_call>{\"name\":\"manage_device_files\",\"arguments\":{\"operation\":
\"search_files\",\"path_or_search_query\":\"/documents/projects\",
\"file_type_filter_for_search\":\"txt\"}}</tool_call>
\end{verbatim}
\end{itemize}
\textbf{Analysis:} The concise CSO for the Memory-Efficient agent allows it to recover from errors in later turns of the trajectory as opposed to the Baseline models which are overwhelmed with the context.
\end{tcolorbox}
\subsection{Generalizability of the Framework Architecture}
\label{sec:appendix_generalizability}
To validate that the benefits of our proposed framework are not specific to a single model family, we conducted an additional qualitative evaluation. Rather than a full multi-run training and quantitative study, which would be prohibitively large, we performed a targeted qualitative verification on another open-source model, \textbf{Qwen-3 4B}~\citep{yang2025qwen3technicalreport}. We adapted our \texttt{Combined} and \texttt{Memory-Efficient} architectures to the model and ran it through a representative subset of our evaluation scenarios.
The analysis confirmed that the core architectural benefits observed in our main experiments are highly consistent across different base models. The key positive trends were replicated:
\begin{itemize}
\item \textbf{Dramatically Reduced Context Growth:} The dual-adapter memory system, powered by the Context State Object (CSO), was equally effective on the Qwen 3 model. It successfully maintained a nearly flat context size over long, multi-turn interactions, directly mirroring the efficiency gains shown in our primary results.
\item \textbf{Efficient Tool Scalability:} The just-in-time (JIT) tool-passing mechanism enabled the Qwen-based agents to effectively manage a large and diverse suite of tools with a minimal initial token footprint, showing the same scalability benefits as observed with the xLAM-2 model.
\item \textbf{Preservation of High Performance:} When equipped with our context-efficient framework, the Qwen 3 model consistently demonstrated a high level of performance. It effectively used tools and successfully completed complex, multi-step tasks, indicating that our methods preserve the core agentic capabilities of the base models while achieving massive context reduction.
\end{itemize}
This qualitative study provides supporting evidence that the architectural advantages of our adaptive context management framework are robust and generalize effectively beyond the primary model used in our main analysis.
\begin{thebibliography}{40}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abdelaziz et~al.(2024)Abdelaziz, Basu, Agarwal, Kumaravel, Stallone,
Panda, Rizk, Bhargav, Crouse, Gunasekara, Ikbal, Joshi, Karanam, Kumar,
Munawar, Neelam, Raghu, Sharma, Soria, Sreedhar, Venkateswaran, Unuvar, Cox,
Roukos, Lastras, and
Kapanipathi]{abdelaziz2024granitefunctioncallingmodelintroducing}
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew
Stallone, Rameswar Panda, Yara Rizk, GP~Bhargav, Maxwell Crouse, Chulaka
Gunasekara, Shajith Ikbal, Sachin Joshi, Hima Karanam, Vineet Kumar, Asim
Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana~Meza Soria, Dheeraj
Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis
Lastras, and Pavan Kapanipathi.
\newblock Granite-function calling model: Introducing function calling
abilities via multi-task learning of granular tasks, 2024.
\newblock URL \url{https://arxiv.org/abs/2407.00121}.
\bibitem[Abdin et~al.(2024)Abdin, Aneja, Awadalla, Awadallah, Awan, Bach,
Bahree, Bakhtiari, Bao, Behl, Benhaim, Bilenko, Bjorck, Bubeck, Cai, Cai,
Chaudhary, Chen, Chen, Chen, Chen, Chen, Cheng, Chopra, Dai, Dixon, Eldan,
Fragoso, Gao, Gao, Gao, Garg, Giorno, Goswami, Gunasekar, Haider, Hao,
Hewett, Hu, Huynh, Iter, Jacobs, Javaheripi, Jin, Karampatziakis, Kauffmann,
Khademi, Kim, Kim, Kurilenko, Lee, Lee, Li, Li, Liang, Liden, Lin, Lin, Liu,
Liu, Liu, Liu, Liu, Luo, Madan, Mahmoudzadeh, Majercak, Mazzola, Mendes,
Mitra, Modi, Nguyen, Norick, Patra, Perez-Becker, Portet, Pryzant, Qin,
Radmilac, Ren, de~Rosa, Rosset, Roy, Ruwase, Saarikivi, Saied, Salim,
Santacroce, Shah, Shang, Sharma, Shen, Shukla, Song, Tanaka, Tupini,
Vaddamanu, Wang, Wang, Wang, Wang, Wang, Wang, Ward, Wen, Witte, Wu, Wu,
Wyatt, Xiao, Xu, Xu, Xu, Xue, Yadav, Yang, Yang, Yang, Yang, Yu, Yuan, Zhang,
Zhang, Zhang, Zhang, Zhang, Zhang, Zhang, and
Zhou]{abdin2024phi3technicalreporthighly}
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar~Ahmad Awan,
Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon
Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai,
Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen,
Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen
Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie~Del
Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao,
Russell~J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam~Ade Jacobs, Mojan
Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi,
Dongwoo Kim, Young~Jin Kim, Lev Kurilenko, James~R. Lee, Yin~Tat Lee, Yuanzhi
Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce~Liu, Liyuan
Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali
Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César~Teodoro Mendes,
Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel
Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac,
Liliang Ren, Gustavo de~Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase,
Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning
Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro
Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan
Wang, Shuohang Wang, Xin Wang, Yu~Wang, Rachel Ward, Wen Wen, Philipp Witte,
Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian
Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang,
Donghan Yu, Lu~Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li~Lyna
Zhang, Yi~Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou.
\newblock Phi-3 technical report: A highly capable language model locally on
your phone, 2024.
\newblock URL \url{https://arxiv.org/abs/2404.14219}.
\bibitem[Chen et~al.(2024)Chen, Li, and
Ma]{chen2024octopusondevicelanguagemodel}
Wei Chen, Zhiyuan Li, and Mingyuan Ma.
\newblock Octopus: On-device language model for function calling of software
apis, 2024.
\newblock URL \url{https://arxiv.org/abs/2404.01549}.
\bibitem[Erdogan et~al.(2024)Erdogan, Lee, Jha, Kim, Tabrizi, Moon, Hooper,
Anumanchipalli, Keutzer, and
Gholami]{erdogan2024tinyagentfunctioncallingedge}
Lutfi~Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi,
Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir
Gholami.
\newblock Tinyagent: Function calling at the edge, 2024.
\newblock URL \url{https://arxiv.org/abs/2409.00608}.
\bibitem[Ferev et~al.(2024)Ferev, Pang, and Bhuwalka]{ferev2024device}
Milen Ferev, Yishuang Pang, and Pulkit Bhuwalka.
\newblock On-device llm-driven application function calling for natural
language interaction.
\newblock 2024.
\bibitem[Gerganov \& llama.cpp contributors(2023)Gerganov and llama.cpp
contributors]{ggerganov2023llama}
Georgi Gerganov and The llama.cpp contributors.
\newblock {llama.cpp}: Inference of llama model in pure c/c++.
\newblock \url{https://github.com/ggerganov/llama.cpp}, 2023.
\newblock GitHub repository.
\bibitem[{Google Cloud}(2024)]{google2024gemini2flash}
{Google Cloud}.
\newblock {Gemini 2.0 Flash and 2.0 Pro update}.
\newblock
\url{https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-0-flash},
2024.
\newblock Accessed: 2024-10-27.
\bibitem[Hu et~al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and
Chen]{hu2021loralowrankadaptationlarge}
Edward~J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu~Wang, and Weizhu Chen.
\newblock Lora: Low-rank adaptation of large language models, 2021.
\newblock URL \url{https://arxiv.org/abs/2106.09685}.
\bibitem[Ju \& Aral(2025)Ju and Aral]{ju2025collaboratingaiagentsfield}
Harang Ju and Sinan Aral.
\newblock Collaborating with ai agents: Field experiments on teamwork,
productivity, and performance, 2025.
\newblock URL \url{https://arxiv.org/abs/2503.18238}.
\bibitem[Kim et~al.(2025)Kim, Kim, Kwon, Lee, Yun, and
Song]{kim2025kvzipqueryagnostickvcache}
Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae~W. Lee, Sangdoo Yun, and Hyun~Oh
Song.
\newblock Kvzip: Query-agnostic kv cache compression with context
reconstruction, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.23416}.
\bibitem[Lee et~al.(2025)Lee, Lu, and
Doppler]{lee2025ondevicellmcontextawarewifi}
Ju-Hyung Lee, Yanqing Lu, and Klaus Doppler.
\newblock On-device llm for context-aware wi-fi roaming, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.04174}.
\bibitem[Lin et~al.(2024)Lin, Wen, Peng, Nie, Liao, Wang, Mo, Zhou, Cheng,
Zhao, Wang, and Zhang]{lin2024hammerrobustfunctioncallingondevice}
Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang,
Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, Jun Wang, and Weinan Zhang.
\newblock Hammer: Robust function-calling for on-device language models via
function masking, 2024.
\newblock URL \url{https://arxiv.org/abs/2410.04587}.
\bibitem[Lin et~al.(2025)Lin, Ghosh, Low, Shrivastava, and
Mohan]{lin2025refragrethinkingragbased}
Xiaoqiang Lin, Aritra Ghosh, Bryan Kian~Hsiang Low, Anshumali Shrivastava, and
Vijai Mohan.
\newblock Refrag: Rethinking rag based decoding, 2025.
\newblock URL \url{https://arxiv.org/abs/2509.01092}.
\bibitem[Liu et~al.(2023)Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, and
Liang]{liu2023lostmiddlelanguagemodels}
Nelson~F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua,
Fabio Petroni, and Percy Liang.
\newblock Lost in the middle: How language models use long contexts, 2023.
\newblock URL \url{https://arxiv.org/abs/2307.03172}.
\bibitem[{OpenAPI Initiative}(2024)]{OAI_OpenAPI_Specification_sw}
{OpenAPI Initiative}.
\newblock {OpenAPI Specification}, 2024.
\newblock URL \url{https://github.com/OAI/OpenAPI-Specification}.
\bibitem[Packer et~al.(2024)Packer, Wooders, Lin, Fang, Patil, Stoica, and
Gonzalez]{packer2024memgptllmsoperatingsystems}
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir~G. Patil, Ion
Stoica, and Joseph~E. Gonzalez.
\newblock Memgpt: Towards llms as operating systems, 2024.
\newblock URL \url{https://arxiv.org/abs/2310.08560}.
\bibitem[Paramanayakam et~al.(2024)Paramanayakam, Karatzas, Anagnostopoulos,
and Stamoulis]{paramanayakam2024moreoptimizingfunctioncalling}
Varatheepan Paramanayakam, Andreas Karatzas, Iraklis Anagnostopoulos, and
Dimitrios Stamoulis.
\newblock Less is more: Optimizing function calling for llm execution on edge
devices, 2024.
\newblock URL \url{https://arxiv.org/abs/2411.15399}.
\bibitem[Park et~al.(2025)Park, Jones, Morse, Goel, Lee, and
Lott]{park2025keydiffkeysimilaritybasedkv}
Junyoung Park, Dalton Jones, Matthew~J Morse, Raghavv Goel, Mingu Lee, and
Chris Lott.
\newblock Keydiff: Key similarity-based kv cache eviction for long-context llm
inference in resource-constrained environments, 2025.
\newblock URL \url{https://arxiv.org/abs/2504.15364}.
\bibitem[Patil et~al.(2025)Patil, Mao, Yan, Ji, Suresh, Stoica, and
Gonzalez]{patil2025the}
Shishir~G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh,
Ion Stoica, and Joseph~E. Gonzalez.
\newblock The berkeley function calling leaderboard ({BFCL}): From tool use to
agentic evaluation of large language models.
\newblock In \emph{Forty-second International Conference on Machine Learning},
2025.
\newblock URL \url{https://openreview.net/forum?id=2GmDdhBdDk}.
\bibitem[Pope et~al.(2022)Pope, Douglas, Chowdhery, Devlin, Bradbury, Levskaya,
Heek, Xiao, Agrawal, and
Dean]{pope2022efficientlyscalingtransformerinference}
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury,
Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean.
\newblock Efficiently scaling transformer inference, 2022.
\newblock URL \url{https://arxiv.org/abs/2211.05102}.
\bibitem[Prabhakar et~al.(2025)Prabhakar, Liu, Zhu, Zhang, Awalgaonkar, Wang,
Liu, Chen, Hoang, Niebles, Heinecke, Yao, Wang, Savarese, and
Xiong]{prabhakar2025apigenmtagenticpipelinemultiturn}
Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar,
Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan~Carlos Niebles, Shelby
Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, and Caiming Xiong.
\newblock Apigen-mt: Agentic pipeline for multi-turn data generation via
simulated agent-human interplay, 2025.
\newblock URL \url{https://arxiv.org/abs/2504.03601}.
\bibitem[Qin et~al.(2023)Qin, Liang, Ye, Zhu, Yan, Lu, Lin, Cong, Tang, Qian,
Zhao, Hong, Tian, Xie, Zhou, Gerstein, Li, Liu, and
Sun]{qin2023toolllmfacilitatinglargelanguage}
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin,
Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian,
Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
\newblock Toolllm: Facilitating large language models to master 16000+
real-world apis, 2023.
\newblock URL \url{https://arxiv.org/abs/2307.16789}.
\bibitem[Schick et~al.(2023)Schick, Dwivedi-Yu, Dessì, Raileanu, Lomeli,
Zettlemoyer, Cancedda, and Scialom]{schick2023toolformerlanguagemodelsteach}
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli,
Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
\newblock Toolformer: Language models can teach themselves to use tools, 2023.
\newblock URL \url{https://arxiv.org/abs/2302.04761}.
\bibitem[Team et~al.(2025)Team, Kamath, Ferret, Pathak, Vieillard, Merhej,
Perrin, Matejovicova, Ramé, Rivière, Rouillard, Mesnard, Cideron, bastien
Grill, Ramos, Yvinec, Casbon, Pot, Penchev, Liu, Visin, Kenealy, Beyer, Zhai,
Tsitsulin, Busa-Fekete, Feng, Sachdeva, Coleman, Gao, Mustafa, Barr,
Parisotto, Tian, Eyal, Cherry, Peter, Sinopalnikov, Bhupatiraju, Agarwal,
Kazemi, Malkin, Kumar, Vilar, Brusilovsky, Luo, Steiner, Friesen, Sharma,
Sharma, Gilady, Goedeckemeyer, Saade, Feng, Kolesnikov, Bendebury, Abdagic,
Vadi, György, Pinto, Das, Bapna, Miech, Yang, Paterson, Shenoy, Chakrabarti,
Piot, Wu, Shahriari, Petrini, Chen, Lan, Choquette-Choo, Carey, Brick,
Deutsch, Eisenbud, Cattle, Cheng, Paparas, Sreepathihalli, Reid, Tran, Zelle,
Noland, Huizenga, Kharitonov, Liu, Amirkhanyan, Cameron, Hashemi,
Klimczak-Plucińska, Singh, Mehta, Lehri, Hazimeh, Ballantyne, Szpektor,
Nardini, Pouget-Abadie, Chan, Stanton, Wieting, Lai, Orbay, Fernandez,
Newlan, yeong Ji, Singh, Black, Yu, Hui, Vodrahalli, Greff, Qiu, Valentine,
Coelho, Ritter, Hoffman, Watson, Chaturvedi, Moynihan, Ma, Babar, Noy, Byrd,
Roy, Momchev, Chauhan, Sachdeva, Bunyan, Botarda, Caron, Rubenstein,
Culliton, Schmid, Sessa, Xu, Stanczyk, Tafti, Shivanna, Wu, Pan, Rokni,
Willoughby, Vallu, Mullins, Jerome, Smoot, Girgin, Iqbal, Reddy, Sheth,
Põder, Bhatnagar, Panyam, Eiger, Zhang, Liu, Yacovone, Liechty, Kalra, Evci,
Misra, Roseberry, Feinberg, Kolesnikov, Han, Kwon, Chen, Chow, Zhu, Wei,
Egyed, Cotruta, Giang, Kirk, Rao, Black, Babar, Lo, Moreira, Martins,
Sanseviero, Gonzalez, Gleicher, Warkentin, Mirrokni, Senter, Collins, Barral,
Ghahramani, Hadsell, Matias, Sculley, Petrov, Fiedel, Shazeer, Vinyals, Dean,
Hassabis, Kavukcuoglu, Farabet, Buchatskaya, Alayrac, Anil, Dmitry, Lepikhin,
Borgeaud, Bachem, Joulin, Andreev, Hardin, Dadashi, and
Hussenot]{gemmateam2025gemma3technicalreport}
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard,
Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane
Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien
Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo
Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai
Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva,
Benjamin Coleman, Yi~Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David
Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov,
Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar,
David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen,
Abhanshu Sharma, Abheesht Sharma, Adi~Mayrav Gilady, Adrian Goedeckemeyer,
Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic,
Amit Vadi, András György, André~Susano Pinto, Anil Das, Ankur Bapna,
Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan
Chakrabarti, Bilal Piot, Bo~Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen,
Charline~Le Lan, Christopher~A. Choquette-Choo, CJ~Carey, Cormac Brick,
Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas,
Divyashree~Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle,
Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik
Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman
Singh, Harsh Mehta, Harshal~Tushar Lehri, Hussein Hazimeh, Ian Ballantyne,
Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton,
John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan,
Ju~yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran
Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho,
Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael
Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola
Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul
Caron, Paul~Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier~Giuseppe
Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu,
Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy
Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth,
Siim Põder, Sijal Bhatnagar, Sindhu~Raghuram Panyam, Sivan Eiger, Susan
Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci,
Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han,
Woosuk Kwon, Xi~Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed,
Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar,
Jessica Lo, Erica Moreira, Luiz~Gustavo Martins, Omar Sanseviero, Lucas
Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli
Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias,
D.~Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean,
Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya,
Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud,
Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi,
and Léonard Hussenot.
\newblock Gemma 3 technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2503.19786}.
\bibitem[Tran et~al.(2025)Tran, Paracha, Hafeez, and
Chen]{tran2025archrouteraligningllmrouting}
Co~Tran, Salman Paracha, Adil Hafeez, and Shuguang Chen.
\newblock Arch-router: Aligning llm routing with human preferences, 2025.
\newblock URL \url{https://arxiv.org/abs/2506.16655}.
\bibitem[Vijayvargiya et~al.(2025)Vijayvargiya, Zhou, Yerukola, Sap, and
Neubig]{vijayvargiya2025interactiveagentsovercomeambiguity}
Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham
Neubig.
\newblock Interactive agents to overcome ambiguity in software engineering,
2025.
\newblock URL \url{https://arxiv.org/abs/2502.13069}.
\bibitem[Wang et~al.(2025{\natexlab{a}})Wang, Fu, Cao, Wang, Tian, and
Ding]{wang2025recursivelysummarizingenableslongterm}
Qingyue Wang, Yanhe Fu, Yanan Cao, Shuai Wang, Zhiliang Tian, and Liang Ding.
\newblock Recursively summarizing enables long-term dialogue memory in large
language models, 2025{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2308.15022}.
\bibitem[Wang et~al.(2025{\natexlab{b}})Wang, Li, Song, Xu, Tang, Zhuge, Pan,
Song, Li, Singh, Tran, Li, Ma, Zheng, Qian, Shao, Muennighoff, Zhang, Hui,
Lin, Brennan, Peng, Ji, and Neubig]{wang2025openhandsopenplatformai}
Xingyao Wang, Boxuan Li, Yufan Song, Frank~F. Xu, Xiangru Tang, Mingchen Zhuge,
Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang~H. Tran, Fuqiang Li,
Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe
Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and
Graham Neubig.
\newblock Openhands: An open platform for ai software developers as generalist
agents, 2025{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2407.16741}.
\bibitem[Xiao et~al.(2024)Xiao, Tian, Chen, Han, and
Lewis]{xiao2024efficientstreaminglanguagemodels}
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
\newblock Efficient streaming language models with attention sinks, 2024.
\newblock URL \url{https://arxiv.org/abs/2309.17453}.
\bibitem[Xu et~al.(2024)Xu, Li, Chen, Wang, Gao, Cai, and
Ling]{xu2024ondevicelanguagemodelscomprehensive}
Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi~Cai, and Ziyuan Ling.
\newblock On-device language models: A comprehensive review, 2024.
\newblock URL \url{https://arxiv.org/abs/2409.00088}.
\bibitem[Xu et~al.(2022)Xu, Liu, Yan, Cai, Li, and
Li]{xu2022learningbreakloopanalyzing}
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li.
\newblock Learning to break the loop: Analyzing and mitigating repetitions for
neural text generation, 2022.
\newblock URL \url{https://arxiv.org/abs/2206.02369}.
\bibitem[Xu et~al.(2025)Xu, Xiao, Huang, Guo, and
Han]{xu2025xattentionblocksparseattention}
Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han.
\newblock Xattention: Block sparse attention with antidiagonal scoring, 2025.
\newblock URL \url{https://arxiv.org/abs/2503.16428}.
\bibitem[Yang et~al.(2025)Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang,
Lv, Zheng, Liu, Zhou, Huang, Hu, Ge, Wei, Lin, Tang, Yang, Tu, Zhang, Yang,
Yang, Zhou, Zhou, Lin, Dang, Bao, Yang, Yu, Deng, Li, Xue, Li, Zhang, Wang,
Zhu, Men, Gao, Liu, Luo, Li, Tang, Yin, Ren, Wang, Zhang, Ren, Fan, Su,
Zhang, Zhang, Wan, Liu, Wang, Cui, Zhang, Zhou, and
Qiu]{yang2025qwen3technicalreport}
An~Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen
Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan
Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian
Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou,
Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le~Yu, Lianghao
Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui
Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao
Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang
Su, Yichang Zhang, Yinger Zhang, Yu~Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui,
Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
\newblock Qwen3 technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.09388}.
\bibitem[Yao et~al.(2023)Yao, Zhao, Yu, Du, Shafran, Narasimhan, and
Cao]{yao2023reactsynergizingreasoningacting}
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan,
and Yuan Cao.
\newblock React: Synergizing reasoning and acting in language models, 2023.
\newblock URL \url{https://arxiv.org/abs/2210.03629}.
\bibitem[Yue et~al.(2025)Yue, Zhang, Liu, Wan, Wang, Cheng, and
Qi]{yue2025masrouterlearningroutellms}
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and
Yiyan Qi.
\newblock Masrouter: Learning to route llms for multi-agent systems, 2025.
\newblock URL \url{https://arxiv.org/abs/2502.11133}.
\bibitem[Zhang et~al.(2025)Zhang, Lan, Zhu, Liu, Hoang, Kokane, Yao, Tan,
Prabhakar, Chen, Liu, Feng, Awalgaonkar, R~N, Chen, Xu, Niebles, Heinecke,
Wang, Savarese, and Xiong]{zhang-etal-2025-xlam}
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai~Quoc Hoang, Shirley Kokane,
Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao
Feng, Tulika~Manoj Awalgaonkar, Rithesh R~N, Zeyuan Chen, Ran Xu, Juan~Carlos
Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong.
\newblock x{LAM}: A family of large action models to empower {AI} agent
systems.
\newblock In Luis Chiruzzo, Alan Ritter, and Lu~Wang (eds.), \emph{Proceedings
of the 2025 Conference of the Nations of the Americas Chapter of the
Association for Computational Linguistics: Human Language Technologies
(Volume 1: Long Papers)}, pp.\ 11583--11597, Albuquerque, New Mexico, April
2025. Association for Computational Linguistics.
\newblock ISBN 979-8-89176-189-6.
\newblock \doi{10.18653/v1/2025.naacl-long.578}.
\newblock URL \url{https://aclanthology.org/2025.naacl-long.578/}.
\bibitem[Zheng et~al.(2024)Zheng, Zhang, Zhang, Ye, Luo, Feng, and
Ma]{zheng2024llamafactory}
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi
Feng, and Yongqiang Ma.
\newblock Llamafactory: Unified efficient fine-tuning of 100+ language models.
\newblock In \emph{Proceedings of the 62nd Annual Meeting of the Association
for Computational Linguistics (Volume 3: System Demonstrations)}, Bangkok,
Thailand, 2024. Association for Computational Linguistics.
\newblock URL \url{http://arxiv.org/abs/2403.13372}.
\bibitem[Zhou et~al.(2024)Zhou, Kim, Brahman, Jiang, Zhu, Lu, Xu, Lin, Choi,
Mireshghallah, Bras, and Sap]{zhou2024haicosystemecosystemsandboxingsafety}
Xuhui Zhou, Hyunwoo Kim, Faeze Brahman, Liwei Jiang, Hao Zhu, Ximing Lu, Frank
Xu, Bill~Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, Ronan~Le Bras, and
Maarten Sap.
\newblock Haicosystem: An ecosystem for sandboxing safety risks in human-ai
interactions, 2024.
\newblock URL \url{https://arxiv.org/abs/2409.16427}.
\bibitem[Zhou et~al.(2025{\natexlab{a}})Zhou, Jiang, Tian, Weston, Levine,
Sukhbaatar, and Li]{zhou2025sweetrltrainingmultiturnllm}
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar
Sukhbaatar, and Xian Li.
\newblock Sweet-rl: Training multi-turn llm agents on collaborative reasoning
tasks, 2025{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2503.15478}.
\bibitem[Zhou et~al.(2025{\natexlab{b}})Zhou, Qu, Wu, Kim, Prakash, Rus, Zhao,
Low, and Liang]{zhou2025mem1learningsynergizememory}
Zijian Zhou, Ao~Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus,
Jinhua Zhao, Bryan Kian~Hsiang Low, and Paul~Pu Liang.
\newblock Mem1: Learning to synergize memory and reasoning for efficient
long-horizon agents, 2025{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2506.15841}.
\end{thebibliography}