GraphRouter: A Graph-based Router for LLM Selections
Extracted source text · arXiv 2410.03834
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{booktabs} % for better rules in the table
% \usepackage{hyperref}
\usepackage[colorlinks,linkcolor=red,citecolor=blue,urlcolor=blue]{hyperref}
\usepackage{url}
\usepackage{subfigure}
\usepackage{wrapfig}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{xspace}
\usepackage{xcolor}
\usepackage{listings}
\usepackage{tabu}
\usepackage{tabularx}
\usepackage{longtable}
% \usepackage{algpseudocode}
\usepackage{microtype}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\newcommand{\para}[1]{{\vspace{3pt} \bf \noindent #1 \hspace{0pt}}}
\newcommand{\ours}{\textsc{GraphRouter}\xspace}
\usepackage{xspace}
\usepackage{url}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{colortbl}
\usepackage{rotating}
\usepackage{hyperref}
\usepackage{tcolorbox}
\usepackage{colortbl}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{makecell}
\usepackage{bbding}
\usepackage{listings}
\usepackage{wrapfig}
\usepackage{tabu}
\usepackage{tabularx}
\definecolor{royalblue(web)}{rgb}{0.25, 0.41, 0.88}
\definecolor{blue-violet}{rgb}{0.54, 0.17, 0.89}
\definecolor{brightmaroon}{rgb}{0.76, 0.13, 0.28}
\definecolor{darkmagenta}{rgb}{0.55, 0.0, 0.55}
\definecolor{bleudefrance}{rgb}{0.19, 0.55, 0.91}
\definecolor{palatinateblue}{rgb}{0.15, 0.23, 0.89}
\definecolor{royalblue(web)}{rgb}{0.25, 0.41, 0.88}
\definecolor{whitesmoke}{rgb}{0.96, 0.96, 0.96}
\definecolor{thulianpink}{rgb}{0.87, 0.44, 0.63}
\definecolor{amber(sae/ece)}{rgb}{1.0, 0.49, 0.0}
\definecolor{darkblue}{rgb}{0.0, 0.0, 0.55}
\definecolor{alizarin}{rgb}{0.82, 0.1, 0.26}
\definecolor{asparagus}{rgb}{0.53, 0.66, 0.42}
\definecolor{darkspringgreen}{rgb}{0.09, 0.45, 0.27}
\definecolor{columbiablue}{rgb}{0.61, 0.87, 1.0}
\definecolor{wildblueyonder}{rgb}{0.64, 0.68, 0.82}
\definecolor{trolleygrey}{rgb}{0.5, 0.5, 0.5}
\definecolor{paleaqua}{rgb}{0.74, 0.83, 0.9}
\definecolor{bubblegum}{rgb}{0.99, 0.76, 0.8}
\definecolor{coralred}{rgb}{1.0, 0.25, 0.25}
\definecolor{green(ryb)}{rgb}{0.4, 0.69, 0.2}
\definecolor{flame}{rgb}{0.89, 0.35, 0.13}
\definecolor{bittersweet}{rgb}{1.0, 0.44, 0.37}
\definecolor{darksalmon}{rgb}{0.91, 0.59, 0.48}
\definecolor{emerald}{rgb}{0.31, 0.78, 0.47}
\definecolor{green(pigment)}{rgb}{0.0, 0.65, 0.31}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.96,0.96,0.94}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.96,0.96,0.94}
\definecolor{forestgreen}{RGB}{34, 139, 34}
\usepackage{pgfplots}
\usepackage{pifont}
\pgfplotsset{compat=1.17}
\definecolor{custompurple}{HTML}{800080}
\usepackage{subcaption}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage{xspace}
\usepackage{caption}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{tabularx}
\usepackage{url} %
\usepackage{booktabs} %
\usepackage{amsfonts} %
\usepackage{nicefrac} %
\usepackage{microtype} %
\usepackage{xcolor} %
\usepackage{graphicx}
\usepackage{float}
\usepackage{scalerel,amssymb}
\newcommand{\mb}{\mathbf}
\title{\method: A Graph-based Router for LLM Selections}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\iclrfinalcopy
\author{Tao Feng, Yanzhen Shen, Jiaxuan You \\
Department of Computer Science \\
University of Illinois Urbana Champaign
Urbana, IL, USA \\
\texttt{\{taofeng2, yanzhen4, jiaxuan\}@illinois.edu}
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\newcommand{\method}{\texttt{GraphRouter}\xspace}
\newcommand{\jiaxuan}[1]{{{\textcolor{orange}{[Jiaxuan: #1]}}}}
%\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
The rapidly growing number and variety of Large Language Models (LLMs) present significant challenges in efficiently selecting the appropriate LLM for a given query, especially considering the trade-offs between performance and computational cost. Current LLM selection methods often struggle to generalize across new LLMs and different tasks because of their limited ability to leverage contextual interactions among tasks, queries, and LLMs, as well as their dependence on a transductive learning framework. To address these shortcomings, we introduce a novel inductive graph framework, named as \method, which fully utilizes the contextual information among tasks, queries, and LLMs to enhance the LLM selection process. \method constructs a heterogeneous graph comprising task, query, and LLM nodes, with interactions represented as edges, which efficiently captures the contextual information between the query's requirements and the LLM's capabilities. Through an innovative edge prediction mechanism, \method is able to predict attributes (the effect and cost of LLM response) of potential edges, allowing for optimized recommendations that adapt to both existing and newly introduced LLMs without requiring retraining. Comprehensive experiments across three distinct effect-cost weight scenarios have shown that \method substantially surpasses existing routers, delivering a minimum performance improvement of 12.3\%. In addition, it achieves enhanced generalization across new LLMs settings and supports diverse tasks with at least a 9.5\% boost in effect and a significant reduction in computational demands. This work endeavors to apply a graph-based approach for the contextual and adaptive selection of LLMs, offering insights for real-world applications. Our codes for \method will soon be released at \url{https://github.com/ulab-uiuc/GraphRouter}.
\end{abstract}
\input{Sec_1_Introduction}
\input{Sec_3_Methods}
\input{Sec_4_Experimental_Setup}
\input{Sec_5_Experiments}
\input{Sec_2_Relatedworks}
\input{Sec_6_Discussion}
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
\input{Sec_7_Appendix}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\section{Introduction}
The field of Large Language Models (LLMs) is advancing quickly, offering an increasingly diverse range of models that vary in size, functionality, and computational demands \citep{bang2023gptcache,liu2023tcra}. Although larger models tend to deliver better performance, their high computational costs make them inefficient for many less complex tasks \citep{snell2024scaling,chen2024role}. Additionally, LLMs demonstrate varied performance across different types of queries and tasks \citep{ahmed2024studying,zhang2024pybench}, especially with the development of domain-specific LLMs \citep{singhal2022large, luo2022biogpt}. These challenges make it difficult to recommend the optimal LLM services to users that strike the balance between performance and cost for their specific needs. Therefore, our paper aims to raise attention to this pressing research question: \textit{Given the vast and continuously evolving landscape of LLMs, how to recommend appropriate LLMs for various user queries with different implied tasks?}
% are continuously being updated and refined,
Existing researchers have proposed to develop a router to assign a specific LLM to each user query. Hybrid LLM \citep{ding2024hybrid} trains a binary score router function to determine whether to select a small LLM or a large LLM for a specific query. Although it balances cost and performance, it is limited to just two LLMs, which falls short of addressing the real-world demand for a wide range of LLMs. Some other studies \citep{dai2024cost,chen2023frugalgpt} have introduced more advanced router models to address the challenge of selecting among a limited set of LLMs (typically 3 to 5). More specifically, FrugalGPT \citep{chen2023frugalgpt} proposes a router model based on BERT \citep{devlin2018bert} to determine whether to switch to a larger LLM or not, and C2MAB-V \citep{dai2024cost} constructs a bandit-based router to balance between the exploration and exploitation when choosing LLM for the user. However, as shown in Table \ref{Tab:GraphRouter_intro}, they are still limited in the following aspects: 1) Relying solely on basic BERT-based embeddings to distinguish queries, and on names or indices to distinguish LLMs, they fail to fully leverage the contextual information from the interaction between the task, query, and LLM. This makes it challenging to achieve a router with strong generalization capabilities. 2) These methods rely on a \textit{transductive} learning framework \citep{arnold2007comparative,joachims2003transductive}, which makes them ill-suited for real-world applications where new LLMs are frequently introduced.
% LLMs are frequently updated or replaced.
When new LLMs are presented, these approaches require retraining with few-shot interaction data before they can be used -- a process that is impractical for recommending LLMs to a large number of users in real-time; 3) They train a dedicated router for each specific task, greatly increases the computational overhead and complexity in real-world applications when multiple tasks are present.
To address these challenges, we introduce \method, a graph-based router for LLM selection. \method utilizes an inductive graph framework to effectively leverage contextual information, allowing it to generalize to new LLMs and adapt to diverse tasks. Specifically, to fully utilize the contextual information for different queries and tasks, \method constructs a heterogeneous graph that contains three types of nodes: task node, query node and LLM node. The interaction information between them is represented as edges in a graph. For instance, the reward (including performance and cost) of an LLM responding to a query is modeled as an edge between the query node and the LLM node. Then, we are able to transform the task of predicting the cost and performance of an LLM-query pair to an edge prediction task. After forecasting the properties of the edges, we recommend the most suitable LLM to the user based on their preferences for performance and cost. In addition, in real-world scenarios, new LLMs are frequently developed, so an effective framework should also have the ability to accommodate these evolving models. In order to make \method generalizable to new LLMs, we make efforts in two key aspects. For the input, we utilize a generative LLM such as GPT-4o to generate a descriptive text for each LLM, outlining key details such as its strengths, token pricing, and context length. Based on this, we derive an initial embedding for each LLM using a moderate-size pre-trained language model (e,g, BERT \citep{devlin2018bert}). This approach offers an advantage over directly using one-hot encoding, as it enables us to generate inductive and more informative initial embeddings for new LLMs.
% , promoting better flexibility and adaptability.
For the \method model, we further develop a heterogeneous GNN that aggregates information from neighboring nodes of different types; given few-shot data, we verified that a trained \method can generalize to new LLM nodes without retraining.
In summary, our main contributions are as follows:
\begin{itemize}
\item To the best of our knowledge, we are the first work to build router for LLM selections from the graph perspective, which gives new insight to graph-enhanced LLM research.
\item We propose an inductive graph framework that fully leverages contextual information among tasks, queries, and LLMs, enabling it to generalize to new LLMs and adapt to a variety of tasks without retraining.
\item In three experiment settings with different performance and cost tradeoffs, \method outperformed the baseline models by at least 12.3\%. Furthermore, in scenarios where new LLMs are introduced in the testing data, our method not only saves significant training time but also improves performance by at least 9.5\% compared to the baselines.
\end{itemize}
\begin{table}[t]
\caption{\textbf{Comparison of \method with existing methods from three perspectives: contextual info, generalization to new LLMs, and multi-task support.} Compared to other approaches, \method introduces an inductive graph framework that fully leverages contextual information, enabling it to generalize to new LLMs and adapt to a variety of tasks.}
\label{Tab:GraphRouter_intro}
\centering
\resizebox{\textwidth}{!}{\begin{tabular}{lccc}
\toprule
\textbf{Method} & \textbf{Contextual Info} & \textbf{Generalization to New LLMs} & \textbf{Multi-task Support} \\
\midrule
Hybrid LLM \citep{ding2024hybrid} & Indices & \textcolor{red}{\textbf{\ding{55}}} & \textcolor{red}{\textbf{\ding{55}}} \\
FrugalGPT \citep{chen2023frugalgpt} & LLM name & \textcolor{red}{\textbf{\ding{55}}} & \textcolor{red}{\textbf{\ding{55}}} \\
C2MAB-V \citep{dai2024cost} & One-hot embedding & \textcolor{red}{\textbf{\ding{55}}} & \textcolor{red}{\textbf{\ding{55}}} \\
\midrule
\rowcolor{cyan!10} \method & Graph-based contexts & \textcolor{forestgreen}{\textbf{\ding{51}}} & \textcolor{forestgreen}{\textbf{\ding{51}}} \\
\bottomrule
\end{tabular}}
\end{table}
\section{GraphRouter: Graph-based Router for LLM Selection}
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{figure/intro_1.pdf}
\caption{\textbf{Overview of \method's LLM selection process.} As depicted in the left section, the LLM selection process begins with the user inputting a query that belongs to a certain task. When the router receives the query and the task, it will analyze the input and choose the most appropriate $LLM_n$ for generation. Then, $LLM_n$ is used to generate the response. In the end, this response, along with the measured effectiveness and cost, is returned to the user. The right side of the figure illustrates example interaction records, which contain contextual information like task, user query, selected LLM, response, performance, and cost, in a table. These contextualized data are then utilized to train the router.}
\label{fig:2.1_}
\end{figure*}
\subsection{Preliminaries} \label{sec:Preliminaries_}
We introduce the LLM selection problem in this section. As shown in the left part of Figure \ref{fig:2.1_}, the process involves multiple steps, with the router being the most critical component. The router first receives the user query containing task information. Its goal is to choose an appropriate LLM based on the incoming information in the user query to ensure optimal performance and minimal cost (LLM API cost). After calculation, the router chooses a suitable $LLM_n$ to answer the user query. Finally, the response is returned to the user with its performance and cost. Such an interaction process generates rich contextual data, which contains information on tasks, queries, selected LLM, response, performance, and cost. We organize the data in a table, as shown on the right.
\subsection{Motivating Examples}
Traditional LLM selection methods \citep{ding2024hybrid, chen2023frugalgpt, dai2024cost} often only use ID or name information to model LLM information, which does not effectively utilize the contextual information (introduced in Sec \ref{sec:Preliminaries_}) generated from the interaction between LLM and query. Here, we use some examples to illustrate the importance of contextual information. (1) We first argue that contextual information is important because it captures the variance in the ability of different LLMs to respond to diverse queries. We can first observe from Figure \ref{fig:Alpaca distribution_} that the performance of different LLMs in response to queries can differ significantly. Therefore, understanding the performance patterns of how LLMs handle queries is crucial for LLM selection, and these patterns are embedded in the contextual information between the queries and the LLMs. In addition, from Figure \ref{fig: probability distribution_}, we can also observe that smaller LLMs sometimes outperform larger LLMs on certain queries. Even if we have unlimited budgets and can blindly rely on the largest LLM at a high cost, we still may not achieve optimal performance. This also emphasizes the importance of capturing the varying capabilities of LLMs in handling queries based on contextual information. (2) We also claim that the importance of contextual information lies in its ability to capture the differences in how a single LLM responds to queries across different tasks. Through Figures \ref{fig:Alpaca distribution_} and \ref{fig:SQUAD distribution_}, we can observe that certain LLMs exhibit significant differences in their performance across two different tasks, such as Mixtral-8x7B \citep{jiang2024mixtral}. These two examples indicate that the performance may vary greatly on different LLMs and tasks. This suggests that in addition to understanding the capabilities of LLMs, the router must also understand the differences and similarities of each task. However, as these critical attributes of LLMs and tasks could not be sufficiently represented by their names or IDs, an effective use of the contextual information that encompasses the interaction between task, query, and LLM is needed.
\begin{figure}[t]
\centering
\begin{minipage}{0.32\linewidth}
\includegraphics[width=\linewidth]{figure/pdf_for_task.pdf}
\caption{\textbf{The probability distribution of a small LLM (LLaMA-3 (7b)) having a better performance value than a large LLM (LLaMA-3 (70b)) by $t$ on the Alpaca dataset, where $t \in [-1, 1]$.}}
\label{fig: probability distribution_}
\end{minipage}
\hfill % 用于在两个minipage之间添加一些空白，可以调整或删除以适应你的布局需求
\begin{minipage}{0.32\linewidth}
\includegraphics[width=\linewidth]{figure/violin_alpaca.pdf}
\caption{\textbf{Distribution of the performance of different LLMs in response to queries on the Alpaca task}. Specifically, we plot the performance of ten LLMs of varying sizes.}
\label{fig:Alpaca distribution_}
\end{minipage}
\hfill
\begin{minipage}{0.32\linewidth}
\includegraphics[width=\linewidth]{figure/violin_SQUAD.pdf}
\caption{\textbf{Distribution of the performance of different LLMs responding to queries on the SQUAD task}. In particular, we illustrate the performance of ten LLMs of different sizes.}
\label{fig:SQUAD distribution_}
\end{minipage}
\end{figure}
\subsection{GraphRouter Framework}
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{figure/intro_2.pdf}
\caption{\textbf{Overview of \method methodology}. \method first converts the interaction data among tasks, queries, and LLMs into a graph. Specifically, as illustrated on the right side, tasks, queries, and LLMs from the left table are represented as task nodes, query nodes, and LLM nodes, respectively. Moreover, their relationships derived from the interaction data are modeled as edge features. With this structure, we leverage a GNN to embed both node and edge features, ultimately producing the probability distribution of the selected LLM.}
\label{fig:overview}
\end{figure*}
\para{Method Overview.} As shown in Figure \ref{fig:overview}, \method first transforms the interaction data among tasks, queries, and LLMs into a graph. Specifically, as shown in the right side of Figure \ref{fig:overview}, we model tasks, queries, and LLMs in the left table as task nodes, query nodes, and LLM nodes, while the relationships derived from the interaction data are represented by edge features. We apply GNN to embed the node and edge features and use them for training and testing.
\para{Initialize node/edge features.} As shown in Figure \ref{fig:overview}, we have three types of nodes (task node $h^{(l)}_t$, query node $h^{(l)}
_q$, and LLM node $h^{(l)}_{m}$) and two types of edges (task-query edge $w_{tq}$ and LLM-query edge $w_{mq}$).
Given the inherent differences of task, query, and LLMs, during the initialization of their nodes, we adopt different strategies. For the initialization of task nodes, we utilize an additional LLM, such as GPT-4o, to generate descriptions of the tasks, and then encode the description to obtain its embedding $e_t$. More specifically, we take the average output token embedding after feeding the description into the moderate-size pre-trained language model (PLM), such as BERT \citep{devlin2018bert}. The initialization of query nodes is also obtained by embedding the query $e_q$ through the same PLM. As for the initialization of LLM nodes, the traditional approach often initializes directly using the name or ID of the LLM, which not only limits its ability to generalize to new LLMs but also omits important background information. Here, we still adopt a prompt-based approach. We design prompts for an LLM to describe the capabilities of each LLM. In addition, we also add information about the cost of each LLM after the description. Then, similar to how we obtain the task's embedding, we use the same PLM to compute the initial embeddings $e_l$ of the different LLMs. All the descriptions we generate for different tasks and LLMs can be found in Appendix \ref{sec:prompt}.
As for the task-query edges, we assign a value of $\textbf{1}$ for their initialization. For the initialization of LLM-query edges, we jointly consider the performance and cost information in the interaction data, and assign the concatenation of performance and cost as their initial features.
\para{Predict via a heterogeneous GNN.} We implement the predictive model $f_{\phi}$ over task nodes, query nodes, and LLM nodes using a heterogeneous GNN, as shown in Figure \ref{fig:overview}. For aggregating different types of nodes and edges, we use heterogeneous aggregation with different learnable weights. The objective of the GNN is to learn expressive node embeddings $\mb{h}$ through an iterative weighted aggregation of the local network neighborhoods. The $l$-th iteration of the $\text{GraphConv}( \cdot )$, or the node embeddings update of the $l$-th layer, is represented as:
\begin{equation}
\label{eq:sage}
\mb{h}_t^{(l)} = \mb{U}^{(l)}_t\textsc{Concat}\Big(\textsc{Mean}\Big(\{\textsc{ReLU}(\mb{W}^{(l)}_t\mb{h}_q^{(l-1)}),q \in \mathcal{N}(t)\}\Big), \mb{h}_t^{(l-1)}\Big),
\end{equation}
\begin{equation}
\mb{h}_q^{(l)} = \mb{U}^{(l)}_q\textsc{Concat}\Big(\textsc{Mean}\Big(\{\textsc{ReLU}(\mb{w}_{\mathbf{1}[t\in V_t, m\in V_t]}\mb{W}_{\mathbf{1}[t\in V_d, m\in V_t]}^{(l)}\mb{h}_u^{(l-1)},u \in \mathcal{N}(v)\}\Big), \mb{h}_q^{(l-1)}\Big),
\end{equation}
\begin{equation}
\label{eq:sage}
\mb{h}_m^{(l)} = \mb{U}^{(l)}_m\textsc{Concat}\Big(\textsc{Mean}\Big(\{\textsc{ReLU}(\mb{w}_{mq}^{T}\mb{W}^{(l)}_m\mb{h}_q^{(l-1)}),q \in \mathcal{N}(m)\}\Big), \mb{h}_m^{(l-1)}\Big),
\end{equation}
where $\mb{h}^{(l)}$ is the node embedding after $l$ iterations, $\mb{h}_t^{(l)},\mb{h}_q^{(l)},\mb{h}_m^{(l)}$ have been initialized as $\mb{h}_t^{(0)},\mb{h}_q^{(0)},\mb{h}_m^{(0)}=e_t, e_q, e_m$ respectively
as explained above, and $\mathcal{N}(v)$ denotes the direct neighbors of $v$. $\mathbf{1}[v\in V_d, u\in V_t]$ indicates the message type (whether from task to query, or LLM to query), and $\mb{U}^{(l)},\mb{W}^{(l)}$ are learnable parameters. In addition, $\mb{w}_{\mathbf{1}[t\in V_t, m\in V_t]}$ indicates that different edge types correspond to different edge features; specifically, if it is from task to query, it is represented as $w_{tq}$, and from LLM to query, it is represented as $w_{mq}$.
Following the update of the task, query, and LLM node embeddings, we obtain the query-task combined embedding as $\mb{h}_{qt}^{(l)}=\textsc{MLP}(\textsc{Concat}(\mb{h}_t^{(l)},\mb{h}_q^{(l)}))$. We model the LLM selection problem as an edge prediction problem and generate training data in the following way. We first determine the best LLM for each query in the training set based on the performance (best reward described in Sec \ref{metric}) achieved by different LLMs and then set the edge labels of the query to other LLMs to 0 and the edge label of the query to the best LLM to 1. As such, LLM prediction can be made through $\text{EdgePred}(\cdot)$ in the form of $\hat{y}_{logits} = \textsc{MEAN} \Big( \textsc{Dot} (\mb{h}_{qt}^{(l)},\mb{h}_m^{(l)})\Big)$. We have summarized the detailed training process of \method in Algorithm \ref{alg:router}, whose details are shown as follows. In addition, in the testing of \method, we identify the LLM node that has maximum edge logits with the query node as the best LLM, which can be computed as \begin{equation} \hat{y} = \arg\max_{m} \Big(\text{EdgePred}(h_{qt}, h_{m})\Big). \end{equation}
\begin{algorithm}[h]
\caption{Training of \method}
\label{alg:router}
\begin{algorithmic}[1]
\Require Dataset $\mathcal{D}_\text{train} = \{(\mb{x},y)\}$. A parameterized heterogeneous GNN $f_\phi$. Task-query edge weights $\mb{w}_{tq}$ and LLM-query edge weights $\mb{w}_{mq}$. Number of GNN layers $L$.
\State Initialize the embeddings of task nodes, query nodes, and LLM nodes, $h^{(0)}_t, h^{(0)}_q, h^{(0)}_m$, using PLM.
\For {each iteration $i$}
\State $M \leftarrow \text{SampleMiniEdgeBatch}(\mathcal{D}_\text{train}) $
\State Mask the edges in $\mathcal{D}_\text{train}$ that are in $M$ and obtain the labels of the edges in $T^{(i)}_{\text{m}}$
\For {$l=1$ to $L$ }
\State $ \mb{h}_t^{(l)},\mb{h}_q^{(l)},\mb{h}_m^{(l)} \leftarrow \text{GraphConv} (\mb{h}_t^{(l-1)},\mb{h}_q^{(l-1)},\mb{h}_m^{(l-1)},\mb{w}_{tq},\mb{w}_{mq})$ with $f_\phi$
\EndFor
\State $\hat{y}_{logits} \leftarrow \text{EdgePred}( \mb{h}_t^{(l)},\mb{h}_q^{(l)}),\mb{h}_m^{(l)}$ with $f_\phi$
\State $\text{Backward}\left( \text{Criterion}(\hat{y}_{logits}, \{ \{ y_j \}_{j \in T^{(i)}_{\text{m}}} \} \in M) \right)$
\EndFor
\end{algorithmic}
\end{algorithm}
\para{\method for new LLMs setting.} Traditional routers cannot generalize to new LLMs directly under few-shots settings, as they require retraining through interactions with the query for each new LLM. This is inadequate for keeping up with the rapidly evolving changes in LLMs in the real world. To test our framework and baselines under this real-world setting, following \citep{cao2023relational,fey2023relational}, we construct an auxiliary dataset with the interaction data of the new LLMs on queries sampled uniformly from the training set. This auxiliary dataset is not involved in the training process but serves as a few-shot examples during the testing phase.
\section{Experimental Setup}
\subsection{Datasets and LLM Descriptions} \label{dataset_des}
We select data from four different types of tasks, whose statistics are summarized in Table \ref{Data_Table}.
\begin{itemize}[leftmargin=1em, itemsep=0pt, topsep=0pt]
\item \textbf{Alpaca} \citep{taori2023stanford} is a hybrid question-answer (QA) dataset containing 52k samples used for fine-tuning the \textbf{Alpaca} model. The dataset is automatically generated by the \textbf{self-instruct} \citep{wang2022self} framework, which iteratively prompts the language model to generate new training instances given a few manually written instructions.
\item \textbf{GSM8K} \citep{cobbe2021training} evaluates the model's ability for multi-step mathematical reasoning with 8.5k linguistically diverse grade school math word problems.
\item \textbf{SQUAD} \citep{rajpurkar2016squad} is a crowdsourced reading comprehension dataset based on Wiki articles. It contains over 100k QA pairs connected to over 500 articles.
\item \textbf{Multi-News} \citep{fabbri2019multi} is a benchmark on multi-document summarization. It consists of 56k news articles and summary pairs where the news articles are extracted from newser.com and the summary is written by professional editors.
\end{itemize}
Furthermore, we introduced 10 LLMs of varying sizes into our problem, with their statistics shown in Table \ref{llm_dis}. All of the LLMs and their token costs here are accessed through the Together API \footnote{https://docs.together.ai/docs/inference-models}.
\subsection{Data Preprocessing and Splitting} \label{data_pre}
\begin{wraptable}{r}{6cm} % Adjusted width to better fit the content
\vspace{-4em}
\caption{Overview of Datasets.}\label{Data_Table}
\centering
\setlength{\tabcolsep}{5pt}
\renewcommand\arraystretch{1.2} % Adjusted for better spacing
\resizebox{0.4\textwidth}{!}{% Adjusted resize box to better fit the table
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Dataset} &\textbf{Task Type} &\textbf{Metric} & \textbf{Cases} \\ \midrule
\hline
Alpaca & Hybrid QA & F1 & 600 \\
GSM8K & Reasoning & Accuracy & 600 \\
SQUAD & Reading Comprehension & F1 & 600 \\
Multi-News & Summary & F1 & 600\\
\bottomrule
\end{tabular}%
}
\vspace{-1em}
\end{wraptable}
Given the above dataset and LLMs, we construct a multi-task interaction dataset described in Sec \ref{sec:Preliminaries_}. Specifically, we combine all the datasets of four tasks together first. For each query, we utilize ten LLMs in Sec \ref{dataset_des} to answer it and obtain the corresponding response. Then the response is compared with its ground truth to get its performance using the metric of each task described in Table \ref{Data_Table}. Furthermore, the cost is calculated with the number of input tokens and output tokens and the cost of different LLMs in Table \ref{llm_dis}. Here we utilize GPT-2 as in \citep{chen2023frugalgpt} to calculate the number of tokens.
After obtaining the multi-task interaction dataset, we split the dataset according to different experimental settings. We mainly have two major settings. The first is the standard setting, where all LLMs are visible in both the training and test sets, with some new queries appearing in the test set. The data is divided into training, validation, and test sets in a ratio of 70\% : 10\%: 20\%, based on different queries. In the case of new LLM setting, we assume that the first six LLMs in Table \ref{llm_dis} are observable, while the remaining four are new LLMs. Therefore, based on the standard setting, we first remove data related to the latter four LLMs from the training and validation sets, while keeping the test set as it is in the standard setting. Furthermore, following \citep{cao2023relational,fey2023relational}, we construct an auxiliary dataset with the interaction data of the four new LLMs on 80 queries sampled uniformly from the training set. This auxiliary dataset is not involved in the training process but serves as a few-shots during the testing phase.
\subsection{Baseline Methods}
We compare our \ours model with the following baselines. We first compare \method with two rule-based baselines:
\begin{itemize}[leftmargin=1em, itemsep=0pt, topsep=0pt]
\item \textbf{Largest LLM} always selects the largest LLM available.
\item \textbf{Smallest LLM} always selects the smallest LLM available.
\end{itemize}
Then we compare \method with a prompt-based baseline:
\begin{itemize}[leftmargin=1em, itemsep=0pt, topsep=0pt]
\item \textbf{Prompt LLM} incorporates the query, candidate models, and objectives (e.g., prioritizing effectiveness) directly into the prompt, and feeds it into an external LLM (e.g., GPT-4) to select the most suitable LLM from a pool of candidates.
\end{itemize}
Further, \method is compared with three representative routers for LLM selection:
\begin{itemize}[leftmargin=1em, itemsep=0pt, topsep=0pt]
\item \textbf{Hybrid LLM }\citep{ding2024hybrid}, when given a small LLM and a large LLM, trains a pre-trained language model to assign queries to the small or large model. We use LLaMA-2 (7b) and Llama-3.1-Turbo (70b) as our small and large LLM respectively, as they are the smallest and the largest LLM available. We also replace DeBERTa \citep{he2020deberta} with RoBERTa \citep{liu2019roberta} as the pre-trained language model and observe better performance.
\item \textbf{FrugalGPT} \citep{chen2023frugalgpt} utilizes a pre-trained language model to predict the score of the generation result of all LLMs given a query, and then selects the LLM with the highest score within a given cost. We also use RoBERTa \citep{liu2019roberta} as the router's backbone model.
\item \textbf{C2MAB-V} \citep{dai2024cost} uses a bandit-based model for LLM selection, which regards each LLM as an arm and implements an exploration mechanism to search for a better solution.
\end{itemize}
Finally, we set up a gold baseline as the optimal solution for LLM selection. The purpose of setting up the baseline is to see how far \method is from the optimal solution.
\begin{itemize}[leftmargin=1em, itemsep=0pt, topsep=0pt]
\item \textbf{Oracle} defines the theoretical upper bound of the reward, where each query has been routed to the optimal LLM via oracle information.
\end{itemize}
\begin{wraptable}{r}{6cm} % Adjusted width to better fit the content
\vspace{-3em}
\caption{Statistics of different LLMs and their costs on Together API.}\label{llm_dis}
\centering
\setlength{\tabcolsep}{5pt}
\renewcommand\arraystretch{1.2} % Adjusted for better spacing
\resizebox{0.4\textwidth}{!}{% Adjusted resize box to better fit the table
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{LLM} &\textbf{Size} & \textbf{Cost per 1M tokens} \\ \midrule
LLaMA-3 (7b) & 7b & 0.2 \\
Mixtral-8x7B & 56b & 0.6 \\
NousResearch & 34b & 0.8 \\
LLaMA-2 (7b) & 7b & 0.2\\
Mistral-7b & 7b & 0.2 \\
LLaMA-3 (70b) & 70b & 0.9 \\
LLaMA-3-Turbo (8b) & 8b & 0.2 \\
LLaMA-3-Turbo (70b) & 70b & 0.9\\
Llama-3.1-Turbo (70b) & 70b & 0.9 \\
Qwen-1.5 (72b) & 72b & 0.9\\
\bottomrule
\end{tabular}%
}
\vspace{-1em}
\end{wraptable}
\subsection{Metrics} \label{metric}
We utilize three metrics to evaluate the performance of \method and baselines.
\begin{itemize}[leftmargin=1em, itemsep=0pt, topsep=0pt]
\item \textbf{Performance} is to evaluate the average quality of the responses across different queries given by each method, which is introduced in Sec \ref{data_pre} and Table \ref{Data_Table}.
\item \textbf{Cost} evaluates the average LLM inference cost when responding to the queries, which is described in Sec \ref{data_pre}.
\item \textbf{Reward} is used to measure how well a method balances performance and cost. Different users may have varying levels of emphasis on performance and cost. Therefore, we define three scenarios: \textbf{Performance First, Balance, and Cost First}, to correspond to situations where users prioritize high performance, value both high performance and low cost equally, or prioritize low cost, respectively. Specifically, to eliminate the influence of scale, we first normalize both performance and cost. Then, we define the score as $Reward = \alpha \cdot Performance - \beta \cdot Cost$. For the three scenarios, we set the values of $\alpha$ and $\beta$ to (1, 0), (0.5, 0.5), and (0.2, 0.8), respectively.
\end{itemize}
\subsection{Implementation Details} In the training stage, we set the graph neural network as a two-layer graph attention network, with a 32-dim hidden dimension. The batch size is 32, and the max training epoch is set to 1000. We use Adam optimizer \citep{diederik2014adam} for model training and gradually decay the learning rate from 1e-3 to 0 with LambdaLR scheduler. We implement our proposed method using PyTorch\footnote{\space https://pytorch.org/} and PyG\footnote{\space https://pytorch-geometric.readthedocs.io/en/latest/}, and all the experiments are conducted on a single NVIDIA A100 Tensor Core GPU. As for LLMs, we rely on API calling from Together AI\footnote{\space https://www.together.ai/} to obtain responses.
\section{Experimental Results}
\subsection{Comparison with existing baselines.}
As shown in Table \ref{tab:comparison_results}, we compare \method with seven baselines in three scenarios. We can observe that \method consistently and substantially surpasses existing routers, delivering a minimum effect improvement of 12.28\% on metric Reward compared to the strongest baselines. Additionally, we observe that \method achieves at least 88.89\% of the optimal solution (Table \ref{tab:comparison_results}, row Oracle), further demonstrating the superiority of our framework. On the other hand, compared with the two rule-based LLM, \method achieves a better trade-off between Performance and Cost, therefore achieving a higher effect on Reward. Analyzing the effect of Prompt LLM, Hybrid LLM \citep{ding2024hybrid}, and FrugalGPT \citep{chen2023frugalgpt}, we demonstrate that without sufficient contextualized information, even LLM and trained moderate-size LM struggle to understand the query and candidates LLM effectively, even if we ignore their high inference costs. These comparisons and results validate our claim that effective usage of contextual information is crucial for selecting the optimal LLM.
\begin{table*}[t]
\centering
\caption{\textbf{Comparison of Various Methods on Multi-task Interaction Dataset across Three Distinct Performance-Cost Weight Scenarios}. Bold and underline denote the best and second-best results. All datasets are evaluated on Performance, Cost, and Reward. Each number is the average of multiple rounds.}
\setlength\tabcolsep{3pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{cccccccccc}
\toprule
\textbf{Scenario}
& \multicolumn{3}{c}{\textbf{Performance First}}
& \multicolumn{3}{c}{\textbf{Balance}}
& \multicolumn{3}{c}{\textbf{Cost First}} \\
\cmidrule(lr){2-4}\cmidrule(lr){5-7}\cmidrule(lr){8-10}
& \textbf{Performance} & \textbf{Cost} & \textbf{Reward}
& \textbf{Performance} & \textbf{Cost} & \textbf{Reward}
& \textbf{Performance} & \textbf{Cost} & \textbf{Reward} \\
\midrule
Largest LLM & 0.431 &0.871 & 0.431 & 0.431 &0.871 & -0.220 & 0.431 &0.871 & -0.701 \\
Smallest LLM & 0.279 & 0.031 & 0.279 & 0.279 & 0.031 & 0.124 & 0.279 & 0.031 & -0.009\\
\midrule
Prompt LLM & 0.474 & 0.812 & 0.474 & 0.285 & 0.0551 & 0.115 & 0.283 & 0.108 & -0.03 \\
\midrule
Hybrid LLM & 0.510 & 0.871 & 0.510 & 0.470 & 0.451 & 0.009 & 0.276 & 0.151 & -0.066 \\
FrugalGPT & 0.517 & 0.671 & \underline{0.517} & 0.400 & 0.072 & 0.164 & 0.411 & 0.031 & \underline{0.057}\\
C2MAB-V &0.479 &0.871 &0.479 &0.423 &0.031 &\underline{0.196} &0.279 &0.031 &0.031 \\
\midrule
\textbf{GraphRouter} &0.539 &0.725 &\textbf{0.539} &0.448 &0.031 & \textbf{0.209} &0.446 &0.031 &\textbf{0.064} \\
\midrule
Oracle &0.588 &0.586 &0.588 &0.504 &0.040 &0.231 &0.483 &0.031 &0.072 \\
\bottomrule
\end{tabular}
}
\label{tab:comparison_results}
\end{table*}
\subsection{Generalization ability to new LLMs}
To validate the generalization ability of \method when facing new LLMs, we conduct experiments as described in Sec \ref{data_pre} in scenario \textbf{Balance}. To compare with other baselines, we add the auxiliary dataset into their training dataset. Specifically, we compare \method (few-shots) with HybridLLM, FrugalGPT, C2MAB-V, and \method (trained) on Reward and time cost (training time + inference time). We report our results in Table \ref{tab:few_shots}. We can observe that in comparison to the most costly \textbf{C2MAB-V \citep{dai2024cost}}, \method (few-shots) not only achieves substantial performance improvements in Reward by almost 10\% but also greatly reduces Time Cost by over 99\%. The amount of Reward Improvement and Time Cost Reduction has significantly surpassed those of other baselines. Additionally, compared to \method (trained), our approach significantly reduces time cost with only a slight performance loss. These observations demonstrate that \method is both effective and efficient in generalizing to new LLMs.
\begin{table*}[t]
\centering
\caption{\textbf{Comparison of methods in the few-shot setting on Reward, Time Cost, and the corresponding percentage Reward improvements and Time Cost reduction rate, relative to the most costly method (C2MAB-V \citep{dai2024cost})}.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{ccccc}
\toprule
\textbf{Method}
& \textbf{Reward}
& \textbf{Reward Improvement(\%)}
& \textbf{Time Cost}
& \textbf{Time Cost Reduction(\%)} \\
\midrule
HybridLLM & 0.01 & -94.71 & 273.45 & 49.57 \\
FrugalGPT & 0.171 & -9.52 & 63.15 & 88.35 \\
C2MAB-V & 0.189 & 0.00 & 542.25 & 0.00 \\
GraphRouter (few-shots) & 0.207 & 9.52 & 3.00 & 99.45 \\
GraphRouter (Trained) & 0.219 & 15.87 & 30.00 & 94.47 \\
\bottomrule
\end{tabular}
}
\label{tab:few_shots}
\end{table*}
\begin{figure}[t]
\centering
\begin{minipage}{0.45\linewidth}
\includegraphics[width=\linewidth]{figure/ablation_size.pdf}
\caption{\textbf{The Reward of \method varies with the input size of the GNN.} Here, we set our in-channels equal to out-channels. As observed in the figure, the Reward of \method initially increases and then decreases, achieving the highest Reward when the size is 32.}
\label{fig:ablation_size}
\end{minipage}
\hfill % 用于在两个minipage之间添加一些空白，可以调整或删除以适应你的布局需求
\begin{minipage}{0.48\linewidth}
\includegraphics[width=\linewidth]{figure/ablation_layer.pdf}
\caption{\textbf{The Reward of \method varies with the number of GNN layers.} As shown in the figure, the Reward of \method initially increases and then decreases, reaching its highest Reward when the number of layers is 2.}
\label{fig:ablation_layer}
\end{minipage}
\end{figure}
\subsection{Ablation Studies}
\textbf{How does \method perform with varying sizes of GNN?} The size of a GNN is an important factor to consider when designing GNN algorithms. It not only affects the performance of the GNN but also introduces additional computational overhead if the size is too large. To find an optimal GNN size for \method, we explored sizes ranging from 16 to 80, as shown in Figure \ref{fig:ablation_size}. As depicted in the figure, the Reward of \method initially improves as the size increases, reaching its peak at a size of 32, after which it starts to decline.
\textbf{What is the impact of different numbers of GNN layers on \method's effectiveness?} The number of GNN layers has a significant impact on the expressiveness of the GNN. A shallow GNN struggles to learn deep contextual information, whereas an overly deep GNN can lead to issues such as over smoothing and overfitting. Moreover, increasing the number of layers also raises the computational cost. To determine the optimal number of GNN layers for \method, we conduct an exploration with the number of layers ranging from 0 to 5, as shown in Figure \ref{fig:ablation_layer}. As depicted in the figure, the Reward of \method initially improves with more layers but then declines, achieving its highest Reward when the number of layers is 2.
\section{Additional Related Works}
\para{LLM selection.} With the scaling of the number of parameters in LLMs, their inference cost is also rapidly increasing. To optimize inference cost, several works have proposed using model switching to mitigate this issue.
When given a small and a large LLM, \cite{zhu2023optimal} fine-tune a pre-trained language model as the model router to predict whether using a small LLM is sufficient. HybridLLM \citep{ding2024hybrid} improves on this by transforming training data to compensate for the influence of imbalanced labels. Other approaches move beyond choosing between two LLMs and introduce settings where multiple LLMs are present. \cite{chen2023frugalgpt} train the router to predict the reliability score given a query and an LLM index. \cite{vsakota2024fly} generalize the router's training objective to accommodate the scenario where additional cost or performance constraints exist. \cite{stripelis2024polyrouter} examine the effectiveness of lighter routers built upon the k-nearest neighbors algorithm or Multilayer Perceptrons. C2MAB-V \citep{dai2024cost} employs a bandit-based model for LLM selection, treating each LLM as an arm and incorporating an exploration mechanism to find an improved solution.
Different from previous approaches, where the router only learns from query-model interaction, our \ours fully utilizes the information in the training data by jointly modeling the query-model, query-query, and model-model relationship. This allows us to learn effective representations for tasks, queries, and models, enabling better generalizability.
\para{Graph for modeling relationships.} Graphs have demonstrated great potential in modeling complex relationships \citep{fey2023relational,cao2023relational,gao2020ci,chen2022gndan,wu2022graph,yang2021consisrec}. Solving relational data with graphs often involves extracting nodes and edges from the data, and then modeling their relationships with embeddings. Traditional graph algorithms, such as label propagation \citep{xie2022graphhop}, directly utilize edge relationships to propagate known labels to target nodes. With the advancement of deep learning, graph neural networks (GNNs) have become the more popular approach for researchers to model relationships within data. They are also found to have widespread application in fields such as recommendation systems \citep{Erxue2022recommender} and social networks \citep{wu2020social}. GNNs can be broadly classified into Message Passing Neural Networks (MPNNs), which include models like GCN \citep{Thomas2017GCN}, GraphSAGE \citep{hamilton2017graphsage}, and GAT \citep{velivckovic2017gat}, as well as non-MPNN architectures \citep{wu2022nodeformer, graphormer}. Additionally, Heterogeneous Graph Neural Networks (HeterGNNs) \citep{hgcn, hgt} have also been proposed to handle more complex graph data. In recent years, researchers have begun exploring the zero-shot or few-shot capabilities of GNNs \citep{fey2023relational,cao2023relational,gao2020ci,chen2022gndan}, aiming to address more complex real-world challenges, such as the cold start problem in recommendation systems. Building on these studies, we incorporate GNNs' powerful ability to represent contextual heterogeneous relationships and their zero-shot capabilities into the LLM selection problem.
\section{Conclusion and Discussion}
\paragraph{Conclusion.} We present \method, a graph inductive framework for LLM routing during inference with multiple LLMs. This work is the first to address the LLM routing problem by reframing it as an edge prediction task between query nodes and LLM nodes. Using graph structure, we fully capture contextual information from prior interaction data to learn effective task, query, and graph representations. Through our experiments on the combined dataset from four open-domain QA datasets, and with three different application scenarios, we demonstrated the superiority of \method over competitive LLM selection baselines and showed that our framework is on par with the ideal "God's-eye view" solution. Beyond traditional settings, we also tested our framework in a more challenging setting where new LLMs were introduced during test time, and we demonstrated \method's strong generalization ability compared to previous baselines. We hope that \method, along with our approach of incorporating interaction data through graphs, will facilitate future research on LLM routing.
\paragraph{Limitations}
This work primarily serves as exploratory work to validate the idea of how modeling past interaction data in a graph could enhance the process of LLM selection. We acknowledge that leveraging more complex graph signals, such as paths, or the taxonomy of LLMs (e.g., family trees like LLaMA2 → LLaMA3 → LLaMA3.1 \citep{touvron2023llama}), could further improve \method's performance, but that is beyond the scope of this paper, and we leave it for future work.
\paragraph{Future Work}
Some other interesting questions to explore in the future include: 1) When answering complex queries, prompting methods like \textbf{Chain-of-Thought} \citep{wei2022chain}, \textbf{Tree-of-Thought} \citep{yao2024tree} are also widely used to enhance the reasoning ability of LLMs. Given the vast number of these methods, predicting the generation result and selecting the best one remains a great challenge. On the other hand, selecting the prompting method is similar to selecting the best LLM for inference, as we are both aiming to predict the generation result based on past interaction data. As a result, it is interesting to explore whether \method could also be adapted to this task. 2) In a multi-agent system, it is critical to choose the appropriate LLM for each module on a specific query and task. It would be valuable to conduct experiments on whether an inductive graph framework like \method could also excel on this challenging task, where multiple LLMs are being selected simultaneously.
\clearpage
\appendix
\section{Description for tasks and LLMs}\label{sec:prompt}
Using descriptions generated by LLM and embeddings derived from BERT as initial node embeddings for GNNs enhances their expressiveness and generalization capabilities of. Here, we have listed descriptions of different tasks and various LLMs obtained using GPT-4o. Specifically, GPT-4o provides insights into the unique characteristics and challenges of different tasks, as well as the size, cost, and particular strengths of different LLMs. The detailed descriptions are shown in the table below.
\begin{table}[h]
\centering
\caption{\textbf{Description of Alpaca task}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
The Alpaca dataset is designed for instruction-following tasks, where the model is required to generate coherent and contextually appropriate responses to given instructions or prompts. It focuses on understanding diverse user requests and providing informative and accurate outputs based on those instructions.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of GSM8K task}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
The GSM8K dataset is tailored for mathematical problem-solving tasks. It consists of natural language math problems that require the model to comprehend the problem statement, apply the correct mathematical operations, and provide the solution. The primary challenge lies in both parsing complex language and performing accurate calculations.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of SQUAD task}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
The SQuAD dataset is focused on question-answering tasks, where the model is given a passage of text and needs to extract or generate a precise answer to a question based on the content of the passage. The dataset emphasizes comprehension, retrieval of relevant information, and concise answer generation.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of Multi-News task}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
The Multi-News dataset is aimed at text summarization tasks. It contains multiple news articles on the same topic, and the model's objective is to generate a concise and comprehensive summary that integrates information from all the articles. The challenge is to distill key points while maintaining coherence and avoiding redundancy. \\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of LLaMA-3 (7b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
This is a relatively small-sized model (7 billion parameters) designed for general-purpose language tasks. Its low cost per million tokens (0.2) makes it an affordable option for many applications requiring quick responses with moderate accuracy.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of Mixtral-8x7B}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
With a combined size of 56 billion parameters, this model aims to provide stronger language modeling capabilities. Its cost per million tokens is 0.6, reflecting its balance between performance and affordability for more complex tasks.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of NousResearch (34b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
A mid-sized model with 34 billion parameters, suitable for handling moderately complex language tasks. Its cost is higher at 0.8 per million tokens, indicating a greater computational demand, likely due to its enhanced capabilities over smaller models.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of LLaMA-2 (7b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
Compact model at 7 billion parameters, it offers similar capabilities and pricing to LLaMA-3 (7b) at a cost of 0.2 per million tokens. It's an efficient choice for tasks requiring decent performance without high computational costs. \\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of Mistral-7b}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
With 7 billion parameters, Mistral-7b is optimized for lightweight tasks, balancing speed and efficiency. Its cost per million tokens is 0.2, making it cost-effective for standard use cases without the need for complex computations.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of LLaMA-3 (70b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
A larger variant of LLaMA-3, this model has 70 billion parameters, providing advanced capabilities for complex tasks. Its cost per million tokens is 0.9, indicating its higher computational demand and enhanced performance.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of LLaMA-3-Turbo (8b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
A variant optimized for speed and efficiency with 8 billion parameters. Its cost per million tokens is only 0.2, suggesting that it is designed to handle tasks quickly while being highly cost-effective.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of LLaMA-3-Turbo (70b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
This model, at 70 billion parameters, is tailored for high performance with an emphasis on efficiency. The cost is 0.9 per million tokens, reflecting its advanced capabilities for a broad range of tasks requiring more computation. \\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of Llama-3.1-Turbo (70b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
Large model with 70 billion parameters, likely to offer strong capabilities for various language tasks. Its cost is also 0.9 per million tokens, suggesting similar performance and computational needs as other 70b models.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{table}[h]
\centering
\caption{\textbf{Description of Qwen-1.5 (72b)}.}
\begin{tabular}{p{13cm}}
\toprule[1.1pt]
With 72 billion parameters, Qwen-1.5 is among the largest models in the list, designed for high-complexity tasks. Its cost per million tokens is 0.9, making it comparable to other high-performance models in terms of both capability and expense.\\
\bottomrule[1.1pt]
\label{prompt-template:idea-discussion}
\end{tabular}
\end{table}
\begin{thebibliography}{45}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ahmed et~al.(2024)Ahmed, Bird, Devanbu, and Chakraborty]{ahmed2024studying}
Toufique Ahmed, Christian Bird, Premkumar Devanbu, and Saikat Chakraborty.
\newblock Studying llm performance on closed-and open-source data.
\newblock \emph{arXiv preprint arXiv:2402.15100}, 2024.
\bibitem[Arnold et~al.(2007)Arnold, Nallapati, and Cohen]{arnold2007comparative}
Andrew Arnold, Ramesh Nallapati, and William~W Cohen.
\newblock A comparative study of methods for transductive transfer learning.
\newblock In \emph{Seventh IEEE international conference on data mining workshops (ICDMW 2007)}, pp.\ 77--82. IEEE, 2007.
\bibitem[Bang(2023)]{bang2023gptcache}
Fu~Bang.
\newblock Gptcache: An open-source semantic cache for llm applications enabling faster answers and cost savings.
\newblock In \emph{Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)}, pp.\ 212--218, 2023.
\bibitem[Cao et~al.(2023)Cao, You, and Leskovec]{cao2023relational}
Kaidi Cao, Jiaxuan You, and Jure Leskovec.
\newblock Relational multi-task learning: Modeling relations between data and tasks.
\newblock \emph{arXiv preprint arXiv:2303.07666}, 2023.
\bibitem[Chen \& Varoquaux(2024)Chen and Varoquaux]{chen2024role}
Lihu Chen and Ga{\"e}l Varoquaux.
\newblock What is the role of small models in the llm era: A survey.
\newblock \emph{arXiv preprint arXiv:2409.06857}, 2024.
\bibitem[Chen et~al.(2023)Chen, Zaharia, and Zou]{chen2023frugalgpt}
Lingjiao Chen, Matei Zaharia, and James Zou.
\newblock Frugalgpt: How to use large language models while reducing cost and improving performance.
\newblock \emph{arXiv preprint arXiv:2305.05176}, 2023.
\bibitem[Chen et~al.(2022)Chen, Hong, Xie, Peng, You, Ding, and Shao]{chen2022gndan}
Shiming Chen, Ziming Hong, Guosen Xie, Qinmu Peng, Xinge You, Weiping Ding, and Ling Shao.
\newblock Gndan: Graph navigated dual attention network for zero-shot learning.
\newblock \emph{IEEE transactions on neural networks and learning systems}, 35\penalty0 (4):\penalty0 4516--4529, 2022.
\bibitem[Cobbe et~al.(2021)Cobbe, Kosaraju, Bavarian, Chen, Jun, Kaiser, Plappert, Tworek, Hilton, Nakano, et~al.]{cobbe2021training}
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et~al.
\newblock Training verifiers to solve math word problems.
\newblock \emph{arXiv preprint arXiv:2110.14168}, 2021.
\bibitem[Dai et~al.(2024)Dai, Li, Liu, Yu, and Lui]{dai2024cost}
Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John Lui.
\newblock Cost-effective online multi-llm selection with versatile reward models.
\newblock \emph{arXiv preprint arXiv:2405.16587}, 2024.
\bibitem[Devlin(2018)]{devlin2018bert}
Jacob Devlin.
\newblock Bert: Pre-training of deep bidirectional transformers for language understanding.
\newblock \emph{arXiv preprint arXiv:1810.04805}, 2018.
\bibitem[Diederik(2014)]{diederik2014adam}
P~Kingma Diederik.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{(No Title)}, 2014.
\bibitem[Ding et~al.(2024)Ding, Mallick, Wang, Sim, Mukherjee, Ruhle, Lakshmanan, and Awadallah]{ding2024hybrid}
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks~VS Lakshmanan, and Ahmed~Hassan Awadallah.
\newblock Hybrid llm: Cost-efficient and quality-aware query routing.
\newblock \emph{arXiv preprint arXiv:2404.14618}, 2024.
\bibitem[Fabbri et~al.(2019)Fabbri, Li, She, Li, and Radev]{fabbri2019multi}
Alexander~R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir~R Radev.
\newblock Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.
\newblock \emph{arXiv preprint arXiv:1906.01749}, 2019.
\bibitem[Fey et~al.(2023)Fey, Hu, Huang, Lenssen, Ranjan, Robinson, Ying, You, and Leskovec]{fey2023relational}
Matthias Fey, Weihua Hu, Kexin Huang, Jan~Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec.
\newblock Relational deep learning: Graph representation learning on relational databases.
\newblock \emph{arXiv preprint arXiv:2312.04615}, 2023.
\bibitem[Gao \& Xu(2020)Gao and Xu]{gao2020ci}
Junyu Gao and Changsheng Xu.
\newblock Ci-gnn: Building a category-instance graph for zero-shot video classification.
\newblock \emph{IEEE Transactions on Multimedia}, 22\penalty0 (12):\penalty0 3088--3100, 2020.
\bibitem[Hamilton et~al.(2017)Hamilton, Ying, and Leskovec]{hamilton2017graphsage}
Will Hamilton, Zhitao Ying, and Jure Leskovec.
\newblock Inductive representation learning on large graphs.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[He et~al.(2020)He, Liu, Gao, and Chen]{he2020deberta}
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.
\newblock Deberta: Decoding-enhanced bert with disentangled attention.
\newblock \emph{arXiv preprint arXiv:2006.03654}, 2020.
\bibitem[Hu et~al.(2020)Hu, Dong, Wang, and Sun]{hgt}
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun.
\newblock Heterogeneous graph transformer.
\newblock In \emph{Proceedings of the web conference 2020}, pp.\ 2704--2710, 2020.
\bibitem[Jiang et~al.(2024)Jiang, Sablayrolles, Roux, Mensch, Savary, Bamford, Chaplot, Casas, Hanna, Bressand, et~al.]{jiang2024mixtral}
Albert~Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra~Singh Chaplot, Diego de~las Casas, Emma~Bou Hanna, Florian Bressand, et~al.
\newblock Mixtral of experts.
\newblock \emph{arXiv preprint arXiv:2401.04088}, 2024.
\bibitem[Joachims(2003)]{joachims2003transductive}
Thorsten Joachims.
\newblock Transductive learning via spectral graph partitioning.
\newblock In \emph{Proceedings of the 20th international conference on machine learning (ICML-03)}, pp.\ 290--297, 2003.
\bibitem[Kipf \& Welling(2017)Kipf and Welling]{Thomas2017GCN}
Thomas~N. Kipf and Max Welling.
\newblock Semi-supervised classification with graph convolutional networks.
\newblock In \emph{{ICLR} (Poster)}. OpenReview.net, 2017.
\bibitem[Liu et~al.(2023)Liu, Li, Xiang, Wang, and Qian]{liu2023tcra}
Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and Yiming Qian.
\newblock Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction.
\newblock \emph{arXiv preprint arXiv:2310.15556}, 2023.
\bibitem[Liu(2019)]{liu2019roberta}
Yinhan Liu.
\newblock Roberta: A robustly optimized bert pretraining approach.
\newblock \emph{arXiv preprint arXiv:1907.11692}, 2019.
\bibitem[Luo et~al.(2022)Luo, Sun, Xia, Qin, Zhang, Poon, and Liu]{luo2022biogpt}
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu.
\newblock Biogpt: generative pre-trained transformer for biomedical text generation and mining.
\newblock \emph{Briefings in bioinformatics}, 23\penalty0 (6):\penalty0 bbac409, 2022.
\bibitem[Min et~al.(2022)Min, Rong, Xu, Bian, Zhao, Huang, Luo, Lin, and Ananiadou]{Erxue2022recommender}
Erxue Min, Yu~Rong, Tingyang Xu, Yatao Bian, Peilin Zhao, Junzhou Huang, Da~Luo, Kangyi Lin, and Sophia Ananiadou.
\newblock Masked transformer for neighhourhood-aware click-through rate prediction.
\newblock \emph{CoRR}, abs/2201.13311, 2022.
\bibitem[Peng et~al.(2019)Peng, Li, Gong, Song, Ning, Lai, and Yu]{hgcn}
Hao Peng, Jianxin Li, Qiran Gong, Yangqiu Song, Yuanxing Ning, Kunfeng Lai, and Philip~S Yu.
\newblock Fine-grained event categorization with heterogeneous graph convolutional networks.
\newblock \emph{arXiv preprint arXiv:1906.04580}, 2019.
\bibitem[Rajpurkar(2016)]{rajpurkar2016squad}
P~Rajpurkar.
\newblock Squad: 100,000+ questions for machine comprehension of text.
\newblock \emph{arXiv preprint arXiv:1606.05250}, 2016.
\bibitem[{\v{S}}akota et~al.(2024){\v{S}}akota, Peyrard, and West]{vsakota2024fly}
Marija {\v{S}}akota, Maxime Peyrard, and Robert West.
\newblock Fly-swat or cannon? cost-effective language model choice via meta-modeling.
\newblock In \emph{Proceedings of the 17th ACM International Conference on Web Search and Data Mining}, pp.\ 606--615, 2024.
\bibitem[Singhal et~al.(2022)Singhal, Azizi, Tu, Mahdavi, Wei, Chung, Scales, Tanwani, Cole-Lewis, Pfohl, et~al.]{singhal2022large}
Karan Singhal, Shekoofeh Azizi, Tao Tu, S~Sara Mahdavi, Jason Wei, Hyung~Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et~al.
\newblock Large language models encode clinical knowledge.
\newblock \emph{arXiv preprint arXiv:2212.13138}, 2022.
\bibitem[Snell et~al.(2024)Snell, Lee, Xu, and Kumar]{snell2024scaling}
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
\newblock Scaling llm test-time compute optimally can be more effective than scaling model parameters.
\newblock \emph{arXiv preprint arXiv:2408.03314}, 2024.
\bibitem[Stripelis et~al.(2024)Stripelis, Hu, Zhang, Xu, Shah, Jin, Yao, Avestimehr, and He]{stripelis2024polyrouter}
Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He.
\newblock Polyrouter: A multi-llm querying system.
\newblock \emph{arXiv preprint arXiv:2408.12320}, 2024.
\bibitem[Taori et~al.(2023)Taori, Gulrajani, Zhang, Dubois, Li, Guestrin, Liang, and Hashimoto]{taori2023stanford}
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori~B Hashimoto.
\newblock Stanford alpaca: An instruction-following llama model, 2023.
\bibitem[Touvron et~al.(2023)Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozi{\`e}re, Goyal, Hambro, Azhar, et~al.]{touvron2023llama}
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth{\'e}e Lacroix, Baptiste Rozi{\`e}re, Naman Goyal, Eric Hambro, Faisal Azhar, et~al.
\newblock Llama: Open and efficient foundation language models.
\newblock \emph{arXiv preprint arXiv:2302.13971}, 2023.
\bibitem[Veli{\v{c}}kovi{\'c} et~al.(2017)Veli{\v{c}}kovi{\'c}, Cucurull, Casanova, Romero, Lio, and Bengio]{velivckovic2017gat}
Petar Veli{\v{c}}kovi{\'c}, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.
\newblock Graph attention networks.
\newblock \emph{arXiv preprint arXiv:1710.10903}, 2017.
\bibitem[Wang et~al.(2022)Wang, Kordi, Mishra, Liu, Smith, Khashabi, and Hajishirzi]{wang2022self}
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah~A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.
\newblock Self-instruct: Aligning language models with self-generated instructions.
\newblock \emph{arXiv preprint arXiv:2212.10560}, 2022.
\bibitem[Wei et~al.(2022)Wei, Wang, Schuurmans, Bosma, Xia, Chi, Le, Zhou, et~al.]{wei2022chain}
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed~Chi, Quoc~V Le, Denny Zhou, et~al.
\newblock Chain-of-thought prompting elicits reasoning in large language models.
\newblock \emph{Advances in neural information processing systems}, 35:\penalty0 24824--24837, 2022.
\bibitem[Wu et~al.(2022{\natexlab{a}})Wu, Zhao, Li, Wipf, and Yan]{wu2022nodeformer}
Qitian Wu, Wentao Zhao, Zenan Li, David~P Wipf, and Junchi Yan.
\newblock Nodeformer: A scalable graph structure learning transformer for node classification.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 27387--27401, 2022{\natexlab{a}}.
\bibitem[Wu et~al.(2022{\natexlab{b}})Wu, Sun, Zhang, Xie, and Cui]{wu2022graph}
Shiwen Wu, Fei Sun, Wentao Zhang, Xu~Xie, and Bin Cui.
\newblock Graph neural networks in recommender systems: a survey.
\newblock \emph{ACM Computing Surveys}, 55\penalty0 (5):\penalty0 1--37, 2022{\natexlab{b}}.
\bibitem[Wu et~al.(2020)Wu, Lian, Xu, Wu, and Chen]{wu2020social}
Yongji Wu, Defu Lian, Yiheng Xu, Le~Wu, and Enhong Chen.
\newblock Graph convolutional networks with markov random field reasoning for social spammer detection.
\newblock In \emph{Proceedings of the AAAI conference on artificial intelligence}, volume~34, pp.\ 1054--1061, 2020.
\bibitem[Xie et~al.(2022)Xie, Wang, and Kuo]{xie2022graphhop}
Tian Xie, Bin Wang, and C-C~Jay Kuo.
\newblock Graphhop: An enhanced label propagation method for node classification.
\newblock \emph{IEEE Transactions on Neural Networks and Learning Systems}, 34\penalty0 (11):\penalty0 9287--9301, 2022.
\bibitem[Yang et~al.(2021)Yang, Liu, Dou, Ma, and Yu]{yang2021consisrec}
Liangwei Yang, Zhiwei Liu, Yingtong Dou, Jing Ma, and Philip~S Yu.
\newblock Consisrec: Enhancing gnn for social recommendation via consistent neighbor aggregation.
\newblock In \emph{Proceedings of the 44th international ACM SIGIR conference on Research and development in information retrieval}, pp.\ 2141--2145, 2021.
\bibitem[Yao et~al.(2024)Yao, Yu, Zhao, Shafran, Griffiths, Cao, and Narasimhan]{yao2024tree}
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.
\newblock Tree of thoughts: Deliberate problem solving with large language models.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Ying et~al.(2021)Ying, Cai, Luo, Zheng, Ke, He, Shen, and Liu]{graphormer}
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di~He, Yanming Shen, and Tie-Yan Liu.
\newblock Do transformers really perform badly for graph representation?
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 28877--28888, 2021.
\bibitem[Zhang et~al.(2024)Zhang, Pan, Wang, Cai, Zheng, Zeng, and Liu]{zhang2024pybench}
Yaolun Zhang, Yinxu Pan, Yudong Wang, Jie Cai, Zhi Zheng, Guoyang Zeng, and Zhiyuan Liu.
\newblock Pybench: Evaluating llm agent on various real-world coding tasks.
\newblock \emph{arXiv preprint arXiv:2407.16732}, 2024.
\bibitem[Zhu et~al.(2023)Zhu, Sheng, Zheng, Barrett, Jordan, and Jiao]{zhu2023optimal}
Banghua Zhu, Ying Sheng, Lianmin Zheng, Clark Barrett, Michael~I Jordan, and Jiantao Jiao.
\newblock On optimal caching and model multiplexing for large model inference.
\newblock \emph{arXiv preprint arXiv:2306.02003}, 2023.
\end{thebibliography}