MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction
Extracted source text · arXiv 2512.07846
%%
%% This is file `sample-sigconf-authordraft.tex',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% samples.dtx (with options: `all,proceedings,bibtex,authordraft')
%%
%% IMPORTANT NOTICE:
%%
%% For the copyright see the source file.
%%
%% Any modified versions of this file must be renamed
%% with new filenames distinct from sample-sigconf-authordraft.tex.
%%
%% For distribution of the original source see the terms
%% for copying and modification in the file samples.dtx.
%%
%% This generated file may be distributed as long as the
%% original source files, as listed above, are part of the
%% same distribution. (The sources need not necessarily be
%% in the same archive or directory.)
%%
%%
%% Commands for TeXCount
%TC:macro \cite [option:text,text]
%TC:macro \citep [option:text,text]
%TC:macro \citet [option:text,text]
%TC:envir table 0 1
%TC:envir table* 0 1
%TC:envir tabular [ignore] word
%TC:envir displaymath 0 word
%TC:envir math 0 word
%TC:envir comment 0 0
%%
%% The first command in your LaTeX source must be the \documentclass
%% command.
%%
%% For submission and review of your manuscript please change the
%% command to \documentclass[manuscript, screen, review]{acmart}.
%%
%% When submitting camera ready or to TAPS, please change the command
%% to \documentclass[sigconf]{acmart} or whichever template is required
%% for your publication.
%%
%%
\documentclass[sigconf]{acmart} % authordraft
%%
\usepackage{csquotes}
\usepackage{amsfonts,amsmath,amsthm}
\newtheorem{remark}{Remark}
\usepackage{float} % in the preamble
\usepackage{multirow} % for \multirow in tables
%% \BibTeX command to typeset BibTeX logo in the docs
\usepackage{makecell}
\AtBeginDocument{%
\providecommand\BibTeX{{%
Bib\TeX}}}
%% Rights management information. This information is sent to you
%% when you complete the rights form. These commands have SAMPLE
%% values in them; it is your responsibility as an author to replace
%% the commands and values with those provided to you when you
%% complete the rights form.
\setcopyright{acmlicensed}
\copyrightyear{2025}
\acmYear{2025}
% \acmDOI{XXXXXXX.XXXXXXX}
%% These commands are for a PROCEEDINGS abstract or paper.
% \acmConference[Conference acronym 'XX]{Make sure to enter the correct
% conference title from your rights confirmation email}{June 03--05,
% 2018}{Woodstock, NY}
\acmConference[Preprint]{Preprint}{2025}{Arxiv}
%
% Uncomment \acmBooktitle if the title of the proceedings is different
% from ``Proceedings of ...''!
%
%\acmBooktitle{Woodstock '18: ACM Symposium on Neural Gaze Detection,
% June 03--05, 2018, Woodstock, NY}
% \acmISBN{978-1-4503-XXXX-X/2018/06}
%%
%% Submission ID.
%% Use this when submitting an article to a sponsored event. You'll
%% receive a unique submission ID from the organizers
%% of the event, and this ID should be used as the parameter to this command.
%%\acmSubmissionID{123-A56-BU3}
%%
%% For managing citations, it is recommended to use bibliography
%% files in BibTeX format.
%%
%% You can then either use BibTeX with the ACM-Reference-Format style,
%% or BibLaTeX with the acmnumeric or acmauthoryear sytles, that include
%% support for advanced citation of software artefact from the
%% biblatex-software package, also separately available on CTAN.
%%
%% Look at the sample-*-biblatex.tex files for templates showcasing
%% the biblatex styles.
%%
%%
%% The majority of ACM publications use numbered citations and
%% references. The command \citestyle{authoryear} switches to the
%% "author year" style.
%%
%% If you are preparing content for an event
%% sponsored by ACM SIGGRAPH, you must use the "author year" style of
%% citations and references.
%% Uncommenting
%% the next command will enable that style.
%%\citestyle{acmauthoryear}
\newcommand{\Shusen}[1]{\textcolor{blue}{Shusen: #1}}
%%
%% end of the preamble, start of the body of the document source.
\begin{document}
%%
%% The "title" command has an optional parameter,
%% allowing the author to define a "short title" to be used in page headers.
\title{MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction}
%%
%% The "author" command and its associated commands are used to define
%% the authors and their affiliations.
%% Of note is the shared affiliation of the first two authors, and the
%% "authornote" and "authornotemark" commands
%% used to denote shared contribution to the research.
\author{
Guoyao Li\quad Ran He\textsuperscript{*}\quad Shusen Jing\textsuperscript{*}\quad Kayhan Behdin\textsuperscript{*}\quad Yubo Wang\textsuperscript{*}\quad \\
Sundara Raman Ramachandran\textsuperscript{*}\quad Chanh Nguyen\textsuperscript{*}\quad Jian Sheng\textsuperscript{*}\quad Xiaojing Ma\textsuperscript{*}\quad Chuanrui Zhu\textsuperscript{*}\quad Sriram Vasudevan\quad Muchen Wu\quad Sayan Ghosh\quad Lin Su\quad Qingquan Song\textsuperscript{\dag}\quad Xiaoqing Wang\quad Zhipeng Wang \quad Qing Lan \quad Yanning Chen\quad Jingwei Wu\quad Luke Simon\textsuperscript{\dag}\quad Wenjing Zhang\quad \\
Qi Guo\textsuperscript{\ddag}\quad Fedor Borisyuk\textsuperscript{\ddag}\quad \\
\texttt{\{guoyli, rahe, sjing, kbehdin, yubwang, suramachandran, cnguyen, jsheng, xjma, chuzhu, svasudevan, muwu, sayaghosh, lsu, qsong, xiaoqwang, zhipwang, qlan, yannchen, jingwu, lsimon, wzhang, qguo, fborisyuk\}@linkedin.com} \\
LinkedIn, Mountain View, CA, USA
}
%%
%% By default, the full list of authors will be used in the page
%% headers. Often, this list is too long, and will overlap
%% other information printed in the page headers. This command allows
%% the author to define a more concise list
%% of authors' names for this purpose.
\renewcommand{\shortauthors}{Guoyao Li et al.}
%%
%% The abstract is a short summary of the work to be presented in the
%% article.
\begin{abstract}
Large language models (LLMs) excel at capturing semantic nuances and therefore show impressive relevance ranking performance in modern recommendation and search systems.
However, they suffer from high computational overhead under industrial latency and throughput requirements. In particular, cross-encoder ranking systems often create long context prefill-heavy workloads, as the model has to be presented with the user, query and item information.
To this end, we propose \textbf{MixLM}, a novel LLM-based ranking framework, which significantly improves the system throughput via reducing the input context length, while preserving the semantic strength of cross-encoder rankers.
In contrast to a standard ranking system where the context is presented to the model as pure text, we propose to use \textbf{mix-interaction}, a mixture of text and embedding tokens to represent the input.
Specifically, MixLM encodes all items in the catalog into a few embedding tokens and stores in a nearline cache. The encoded item descriptions are used during online inference, effectively reducing the item length from a few thousand text tokens to a few embedding tokens.
We share insights from deploying our MixLM framework to a real-world search application at LinkedIn, including a detailed discussion of our training pipelines, as well as a thorough analysis of our online serving infrastructure optimization. Comparing with strong baselines, MixLM increased throughput by 10.0x under the same latency budget, while maintaining relevance metrics. The efficiency gains delivered by MixLM enabled full-traffic deployment of LLM-powered search, which resulted in a significant 0.47\% increase in Daily Active Users (DAU) in online A/B tests.
% We share concrete lessons from co-training our online ranker and near-line embedding flows for our semantic search application, presenting insights regarding custom loss functions and scaling laws for the mixed-input system.
% In addition, we discuss several lessons from optimizing the online and near-line serving infrastructure for a mixed-input system, exploring ideas such as prompt prefix caching and amortized prefill computation.
% Large language models (LLMs) excel at capturing semantic nuances and therefore yield strong relevance gains in modern ranking and search.
% However, they suffer from high computational overhead under industrial latency and throughput requirements. In particular, ranking and search applications often create long context prefill-heavy workloads, as the model has to be presented with the user, query and item information---
% LLMs suffer from high computational complexity under such workloads.
% To tackle this, we propose \textbf{MixLM}, a novel LLM-based ranking framework, which significantly improves the system throughput via reducing the input context length, while preserving the semantic strength of full-text LLM rankers.
% In contrast to a standard ranking system where the context is presented to the model as pure text, we propose to use a mixture of text and embedding tokens to represent the input context.
% Specifically, in order to reduce the input context length, MixLM introduces a new \emph{mix-interaction} mechanism, by substituting the long text of each candidate item with a few embedding tokens that are cached near-line. Then, the ranker LLM is fed by a mixed input consisting of these compact embeddings together with the query and the auxiliary signals in natural language. We share concrete lessons from co-training our online ranker and near-line embedding flows for a semantic search application, presenting insights regarding custom loss functions and scaling laws for the mixed-input system. In addition, we discuss several lessons from optimizing the online and near-line serving infrastructure for a mixed-input system, exploring ideas such as prompt prefix caching and amortized prefill computation.
% We deploy our MixLM framework to a semantic search system in a large professional social network platform, where MixLM increased throughput 150\% and latency by 30\%, increase DAU by 0.02\%, while maintaining relevance metrics.
% Large language models (LLMs) have shown great potential in recommender systems due to their strong ability to process text-based features. Despite their effectiveness, deploying LLMs for online serving is hindered by their considerable computational demands, which conflict with the tight latency and throughput constraints of industrial applications. To tackle this, we propose \textbf{MixLM}, a novel LLM ranking framework, which substantially reduces the GPU costs while maintain high-throughput and ranking quality. Our system consists of an encoder LLM that encodes item information into embeddings and a ranker LLM that predicts relevance scores for query–item pairs. The encoder LLM precomputes and stores item embeddings in a near-line cache. During online serving, the item text is replaced with the corresponding embeddings retrieved from the near-line cache and combined with other textual inputs to be processed by the ranker LLM. In this way, the token length of item information is greatly reduced, leading to significantly lower computational costs and latency during online serving, considering the quadratic relationship between computational complexity and token length. In addition, we introduce several training strategies to enhance generalization, including a multi-stage training schedule, auxiliary tasks, and various regularization techniques. Moreover, we demonstrate our efficient online and near-line serving infrastructure, adopting prompt prefix caching and amortized prefill computation. Finally, we deploy the proposed MixLM in a semantic search system on a large professional social networking platform, achieving a 150\% increase in throughput, a 0.02\% improvement in daily active users (DAU), and a 30\% reduction in latency, while maintaining comparable performance across other online and offline metrics.
% This enables us to increase the online ranking throughput by xx via relegating some of the computations to a near-line embedding generation flow, while maintaining the ranking quality.
% MixLM's hierarchical architecture, trained end-to-end\textcolor{blue}{co-train, interesting}, enables the model to capture rich interactions among queries, features, and compressed items that dual-encoder retrieval cannot model.
% The drastically smaller per-item input reduces computation substantially. Furthermore, it unlocks aggressive prompt-prefix caching and amortizes pre-fill computation, yielding even more significant throughput and latency gains.
% First deployed at LinkedIn’s AI Job Search,
% \textcolor{blue}{what are the technical contributions? we show ablations, scaling laws, multi-task nature, new loss functions, etc}
% \textcolor{blue}{meta's paper}
% increases ranking absolute QPS 10000,
% The explosive growth of large-scale corpora is rapidly outpacing the inference budget of contemporary language models, challenging the deployment of high-quality semantic search in production environments. We introduce “Scaling Semantic Search with Embedding Compression for Large Language Models,” an end-to-end framework that (i) aggressively compresses candidate documents into dense vectors, (ii) pre-computes and stores these vectors in a near-line cache, and (iii) injects them directly into a LLM-based ranker as token-level embeddings, completely bypassing the conventional tokenizer and embedding layers. Inspired by recent multimodal language models, our architecture feeds the compressed embeddings into the model’s attention stack, enabling richer cross-interaction than approaches that concatenate embeddings only at the MLP head.
% To ensure the language model interprets the injected vectors faithfully, we propose a novel alignment recipe that jointly tunes the compression encoder and LLM-ranker. The resulting system delivers state-of-the-art ranking quality, achieving 0.94 NDCG on a large-scale job–query benchmark, the highest reported in an industry setting. In production, the near-line pre-computation of item embeddings reduces online GPU usage by 80 \%, and our shared-prefix optimization further cuts decoding latency for multi-query batches.
% To the best of our knowledge, this is the first industrial deployment of a large-language-model-centric semantic search engine that scales to hundreds of millions of items while simultaneously advancing quality, efficiency, and cost.
\end{abstract}
%%
%% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
%% Please copy and paste the code instead of the example below.
%%
% \begin{CCSXML}
% <ccs2012>
% <concept>
% <concept_id>10010147.10010257.10010293.10010294</concept_id>
% <concept_desc>Computing methodologies~Neural networks</concept_desc>
% <concept_significance>500</concept_significance>
% </concept>
% <concept>
% <concept_id>10002951.10003317.10003347.10003350</concept_id>
% <concept_desc>Information systems~Recommender systems</concept_desc>
% <concept_significance>500</concept_significance>
% </concept>
% <concept>
% <concept_id>10002951.10003317.10003338.10003343</concept_id>
% <concept_desc>Information systems~Learning to rank</concept_desc>
% <concept_significance>500</concept_significance>
% </concept>
% </ccs2012>
% \end{CCSXML}
% \ccsdesc[500]{Computing methodologies~Neural networks}
% \ccsdesc[500]{Information systems~Recommender systems}
% \ccsdesc[500]{Information systems~Learning to rank}
%%
%% Keywords. The author(s) should pick words that accurately describe
%% the work being presented. Separate the keywords with commas.
% \keywords{Natural language processing, Large language models, recommender systems}
%% A "teaser" image appears between the author and affiliation
%% information and the body of the document, and typically spans the
%% page.
% \received{20 February 2007}
% \received[revised]{12 March 2009}
% \received[accepted]{5 June 2009}
%%
%% This command processes the author and affiliation and title
%% information and builds the first part of the formatted document.
\maketitle
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
\footnotetext[1]{Equal key contributors.}
\footnotetext[2]{Work done while at LinkedIn.}
\footnotetext[3]{Correspondence authors.}
\section{Introduction}
Large Language Models (LLMs) have been widely adopted across industrial applications due to their strong capabilities in both generative and predictive tasks. In particular, search and recommender systems have emerged as major application areas where LLMs deliver substantial quality improvements over existing approaches~\citep{relevance2,relevance3,relevance4,llmrs}.
% (LLM) reranking has become a cornerstone of modern information-retrieval stacks, powering web search, product discovery, and professional-network recommendations.
For example, when used as a relevance judge, an LLM can interpret subtle linguistic cues and capture conceptual relationships, enabling it to retrieve and rank semantically relevant results that traditional lexical methods may overlook.
Despite their impressive text-understanding capabilities, deploying large language models (LLMs) in search and recommender systems remains challenging due to their substantial computational overhead, particularly when processing long text inputs, making it difficult to meet the stringent industrial requirements for high throughput and low latency.
As an example, a standard (cross-encoder) approach to using LLMs for ranking and search applications is to form an input text prompt, including user's query and candidate item(s) description, that instructs the LLM to determine if the query and the item are relevant~\citep{relevance2,relevance3,relevance-biencoder,rankvicuna}. This requires the LLM to process long input prompts. Such workloads with heavy \emph{prefill} often lead to large latency, making them unacceptable for real-world industrial systems, as a result of the quadratic complexity of the self-attention mechanism, a key component of LLM architectures, with regards to the input context length. Often, in real-world implementation of LLM-based semantic search and recommender systems, the LLM has to be replaced with a smaller model, or the features that lead to long input prompts have to be dropped from the online serving stack~\citep{related-ebay,related-pinterest,related-walmart}.
Thus, reducing the input context length for LLM-based ranking and search applications remains an important research problem.
% More specifically in search and recommender systems, a single user request retrieves hundreds to thousands distinct candidate items, making the context length of each candidate dominate the cost, whereas the shared query and user history can be efficiently optimized through LLM prefix caching or other shared-prefix techniques.
% Despite the impressive text-understanding capability, deploying LLMs for search and recommender system applications remains challenging, as serving LLMs in high-throughput low-latency applications can be infeasible or prohibitively expensive.
% However, bringing an LLM to production scale is still constrained by these interlocking factors:
% 1. Context length. In the classic “query ⟨SEP⟩ document” prompt, the document dominates the token budget; hundreds of tokens per item push self-attention’s O(L²) cost through the roof.
% 2. Throughput \& latency. High-traffic search sites must serve tens of thousands of queries per second while keeping p95 latency well below 300 ms. Long prompts multiplied by hundreds of candidates saturate GPUs, throttling QPS and blowing past latency SLAs.
% 3. Computational Cost. The resulting compute load makes “Retrieve-then-LLM” reranking several orders of magnitude more expensive than classical vector search; GPU inference becomes the single largest operating expense.
In this work, we introduce \textbf{MixLM}, an LLM-based ranking framework designed to substantially reduce input context length for semantic search applications while preserving the semantic richness of full-text. MixLM operates by compressing each candidate document, often containing thousands of tokens, into a compact set of learned embedding tokens using an LLM encoder. These embeddings are stored in a nearline cache and, during online inference, are mixed with the user's natural language query (and auxiliary textual signals) and processed by a ranker LLM. By reducing the document representation from thousands of tokens to a handful of embedding tokens, this mixed-input design achieves \emph{orders-of-magnitude lower computational complexity}, while retaining the rich query–document interactions characteristic of full-text LLM rankers.
Beyond the core modeling approach, we present insights from developing and deploying MixLM in LinkedIn's production semantic job search system. We describe how to jointly train the encoder and ranker models in an end-to-end manner, and we introduce new distillation losses that effectively transfer knowledge from an existing full-text LLM ranker into the mixed-interaction architecture, ensuring alignment between embedding-based and text-based ranking behaviors. We further study scaling properties of MixLM with respect to training data size and the number of embedding tokens per item.
From a systems perspective, we provide a comprehensive analysis of MixLM's impact on serving infrastructure. By reducing the context length through embedding compression, MixLM yields substantially higher ranking throughput. We detail optimization techniques tailored for mixed-input LLMs, including nearline caching and amortized prefill computation, and show how these techniques compound the efficiency gains.
Finally, we demonstrate MixLM’s effectiveness at industrial scale through its deployment in LinkedIn’s AI-driven Job Search. MixLM:
\begin{itemize}\setlength\itemsep{0pt}
\item improves serving throughput by \textbf{10.0$\times$} under the same latency budget,
\item preserves the relevance quality of a strong full-text LLM baseline.
\end{itemize}
Combined with its efficiency advantages, MixLM enabled full LLM-ranking-based search deployment and delivered a significant \textbf{0.47\%} lift in Daily Active Users (DAU) in a large-scale online A/B experiment.
\paragraph{Summary of Contributions:}
\begin{itemize}\setlength\itemsep{0pt}
\item We present \textbf{MixLM}, an industrial-scale ranking framework that aims to reduce the prompt length for ranker LLMs by substituting candidate items' textual description with a few tokens of embeddings.
\item We introduce an end-to-end training pipeline that jointly optimizes the encoder LLM and the ranker LLM, that allow for capturing rich query–item–feature interactions beyond the capacity of bi-encoder architectures.
\item We describe a production-ready serving stack that incorporates several system optimizations pertaining to our mixed input ranking system, achieving an order-of-magnitude improvement in latency and GPU throughput.
\item We share lessons and insights from developing and deploying our MixLM framework online to LinkedIn's semantic job search, the primary job search experience on the platform.
\end{itemize}
Taken together, these advances make LLM-based ranking practically deployable and economically sustainable on the industrial scale, enabling search and recommender systems that demand both deep language understanding and real-time responsiveness.
\section{Semantic Search}
\paragraph{Semantic Search} refers to retrieving and ranking items based on their \emph{semantic similarity} to a given query rather than exact lexical overlap. A typical pipeline includes (1) \emph{representation} of queries and items into dense embeddings using neural encoders, (2) \emph{retrieval} of top-$k$ candidates via exhaustive vector search—for example, GPU-accelerated brute-force search, (3) \emph{reranking} of the retrieved candidates using a higher-capacity model that jointly considers query–item interactions, and (4) \emph{application of business logic} such as personalization and diversity.
In this work, we focus on the core ranking problem within this semantic search framework. Specifically, we adopt a pointwise ranking structure that takes as input the textual information of the query and item (and additional user profile information, if appropriate), and predicts either the relevance between the query and item or the likelihood of user engagement. A product policy specifies how to grade the relevance of each (query, job) pair. The ranking system is then trained to optimize Normalized Discounted Cumulative Gain at the tenth position (NDCG@10) using policy-provided labels.
More concretely, given a query $q$ and a candidate item $j$, we aim to predict how relevant the pair $(q,j)$ are.
In a basic cross-encoder ranking framework, one can be create a textual prompt, taking the form:
\begin{equation}\label{eq:promt}
\text{prompt}(q,j)=\text{system prefix},q,j
\end{equation}
where the system prefix include chat template tags and instructions to the LLM to determine if the job posting $j$ is a good match to the query $q$.
Given that the relevance problem studied here is a binary classification problem\footnote{In this work, we do not study setwise or listwise ranking frameworks.}, this textual prompt can be passed through an LLM with an appropriate classification output head, to estimate the relevance score. Particularly, the LLM (with a binary classification head) can estimate the probability distribution $(p_{\text{yes}},p_{\text{no}})$ where a higher value of $p_{\text{yes}}$ indicates a higher likelihood of relevance match:
\begin{equation}
p_{\text{yes}}(q,j)=\mathbb{P}(j~\text{is relevant to}~q)\in[0,1].
\end{equation}
% During the training, we like to minimize the distance between the estimated values $(p_{\text{yes}},p_{\text{no}})$ from the ranking model, and the ground truth labels $(p^*_{\text{yes}},p^*_{\text{no}})$ provided by the product policy (we discuss how the ground truth is generated in Section~\ref{sec:illuminatorprompt}). In particular, we might to choose to minimize the Kullback–Leibler (KL) divergence between these two probability distributions,
% \begin{equation}\label{eq:kl_loss}
% \text{loss}=KL\left((p^*_{\text{yes}},p^*_{\text{no}})\Vert(p_{\text{yes}},p_{\text{no}})\right)
% \end{equation}
% with $\mathrm{KL}(p\Vert q)=\sum_i p_i\log\!\left(p_i/q_i\right)$.
\paragraph{LinkedIn Semantic Job Search}In this work, we focus on LinkedIn’s Semantic Job Search, which redefines job discovery through an AI-driven focus on intent rather than keywords. By interpreting natural language inputs, it captures users’ underlying goals and preferences, minimizing dependence on exact word matches between queries and job descriptions. This semantic understanding allows for more accurate and flexible retrieval, overcoming vocabulary gaps and delivering search results that align more closely with how members naturally express their career interests.
In practice, the prompt in~\eqref{eq:promt} can be lengthy. In fact, in our semantic job search system, most of the prompt will be dominated with item description $j$, which has a median length of 900 tokens, and can take up to 2100 tokens. The lengthy prompts lead to unacceptable latency and throughput overhead, which makes deploying such a text-based system to production costly.
To handle job search traffic spanning various product surfaces, our ranking system must process and score about 3.15 million items per second, making ranking efficiency absolutely critical.
Even though through previous summarization-solution\citep{behdin2025scaling} we were able to reduce the item description length and rollout LLM ranking in limited-scale traffic, the prompt context length remains the major computational bottleneck and hard blocker for LinkedIn's Semantic Job Search's full deployment.
% The ranking model is trained as a supervised classification task using cross-entropy loss over labeled data. Formally, given training samples $(X, Y)$ drawn from distribution $\mathcal{D}$ and a parameterized ranker $F(\cdot;\Theta)$ (e.g., an LLM), the objective is:
% \begin{equation}
% \min_{\Theta} \mathcal{L}_{\text{CE}}(\Theta)
% = \mathbb{E}_{(X, Y) \sim \mathcal{D}} \, l_{\text{CE}}(F(X;\Theta), Y),
% \end{equation}
% where $l_{\text{ce}}$ denotes the sample-level cross-entropy loss.
% While LLMs excel at encoding rich semantic relations, their large model size and high inference cost limit online serving efficiency. Thus, the central challenge of this work is to \textbf{design an efficient ranker architecture that preserves semantic understanding while significantly reducing inference complexity for real-time applications.}
\section{Architecture of MixLM}\label{sec:arch}
\begin{figure}
\includegraphics[width=0.4\textwidth]{Figures/emb_paper_serving.png}
\caption{Architecture of MixLM. In the offline stage, item documents are processed by an encoder LLM to obtain compact embedding representations, which are stored in a near-line cache. At serving time, these embeddings are fetched and concatenated with the user’s query and optional auxiliary text features, and the resulting mixed input is passed to the ranker LLM to generate relevance scores.
}
\label{fig:teaser}
\end{figure}
As the computational complexity of LLM inference scales quadratically with input length, reducing the number of tokens can substantially lower inference costs. In this work, we propose a text–embedding mix-interaction architecture that compresses item text description into an embedding vector using an encoder LLM. The resulting embeddings are then combined with other textual inputs and fed into a ranker LLM for output prediction. Because the item embeddings are precomputed and stored in a near-line cache, the number of tokens that must be processed during online inference is greatly reduced, leading to significant efficiency gains. %The detailed design of the ranker architecture and the corresponding training strategies are presented in the following sections.
\paragraph{Detailed Structure}
Our ranking architecture consists of two components: an encoder LLM, which computes embeddings that summarize item information, and a ranker LLM, which predicts the final task scores. See Figure~\ref{fig:teaser} for an overview.
Before diving deeper into our model architecture, let us
recall that an LLM can be decomposed into three main components: The input embeddings layer, the transformer blocks and the optional output head. For an LLM, the input embeddings layer creates the input's embedding representation. The transformer layers then form the output embedding representations, given the input embeddings. Finally, the LLM output is generated via an optional output head. For example, for a relevance classification model (in contrast to generative LLMs), we might assume the output head is a binary classification head which generates the estimated probability that the query and item are relevant.
For a sequence of length $T$ and a fixed set of tokenized vocabulary $\mathbb{V}$, we
let $g_R(\cdot; \omega_R): \mathbb{V}^{T} \rightarrow \mathbb{R}^{T \times H}$
to denote the input embedding layer of the Ranker LLM, parameterized by $\omega_R$, with a hidden dimension of $H$. We also let $f_R(\cdot; \theta_R): \mathbb{R}^{T \times H} \rightarrow [0,1]$ denote the Ranker LLM decoder and output head, parameterized by $\theta_R$, where the output quantifies the probability of query-item match (i.e., $p_{\text{yes}}$).
The Ranker LLM can thus be expressed as
\begin{gather}
F_R(\cdot; \Theta_R) = (f_R \circ g_R)(\cdot; \omega_R, \theta_R): \mathbb{V}^{T} \rightarrow [0,1],
\end{gather}
where $\Theta_R = [\omega_R, \theta_R]$ represents all ranker parameters.
Similarly, let $\Theta_E = [\omega_E, \theta_E]$ denote the parameters of the Encoder LLM. The Encoder LLM for a sequence of length $T$ can be parameterized as
\begin{gather}
F_E(\cdot; \Theta_E) = (f_E \circ g_E)(\cdot; \omega_E, \theta_E): \mathbb{V}^{T} \rightarrow \mathbb{R}^{T \times H},
\end{gather}
where $g_E(\cdot; \omega_E): \mathbb{V}^{T} \rightarrow \mathbb{R}^{T \times H}$ is the embedding layer and $f_E(\cdot; \theta_E): \mathbb{R}^{T \times H} \rightarrow \mathbb{R}^{T \times H}$ is the transformer that maps token embeddings to hidden representations (note that we do not have an output head for the encoder model).
In our mix-interaction system, we decompose the prompt in~\eqref{eq:promt} into two parts, corresponding to ranker and encoder parts, and tokenize each part:
\begin{equation}
\begin{aligned}
X_R &= \text{tokenize}([\text{system prefix},q]) \in\mathbb{V}^{T_R} \\
X_E &= \text{tokenize}([\text{prefix},j])\in\mathbb{V}^{T_E}
\end{aligned}
\end{equation}
where $T_R$ ($T_E$) is the length of $X_R$ ($X_E$).
Note that $X_E$ does not depend on the query $q$ but only the item description $j$, and therefore, we can encode and store $X_E$ in an offline cache. We then let $h_E = F_E(X_E) \in \mathbb{R}^{T_E \times H}$ to be the encoded item. To compress this item representation, we sample $h_E$ to a shorter sequence of length $T_S$ (specifically in MixLM, we use last N sampling), yielding $h_S = \mathrm{Samp}(F_E(X_e)) \in \mathbb{R}^{T_S \times H}$. In general, one can consider several sampling techniques. In our work, for a give $T_S\geq 1$, we sample the $T_S$ embeddings in $h_E$ corresponding to the last $T_S$ input tokens. In particular, if $T_S=1$, we only take the embedding representation of the last input token.
% Given an input sequence of token IDs $x = [X_p; X_e]$, where $X_p$ denotes the prefix text and $X_e \in \mathbb{R}^{T_E}$ represents the encoder text, we first feed $X_e$ into the encoder LLM to obtain hidden representations .
Next, the input token embeddings of the prefix $X_R$ are computed via the ranker embedding layer, i.e., $h_R = g_R(X_R)$. The two representations $h_R,h_S$ are concatenated to form the combined embedding sequence $[h_P; h_S] \in \mathbb{R}^{(T_R+T_S) \times H}$. The final ``yes'' probability is predicted by the ranker transformer:
\begin{equation}\label{eq:concat}
p_{\text{yes}}(q,j;\Theta) = f_R\left([h_P; h_S]\right)\in [0,1],
\end{equation}
where $\Theta = [\Theta_R, \Theta_E]$ denotes the full set of trainable parameters.
Since $h_S = \mathrm{Samp}(F_E(X_e))$ can be precomputed and stored in a near-line cache, the effective sequence length during online inference is reduced from $T_R + T_E$ to $T_R + T_S$. In practice, $T_S\ll T_E$, where we might take $T_S=1$ and $T_E$ can be up to 2100, representing orders of magnitudes of reduction in the prompt length. Furthermore, when ranking multiple items, $X_R$ is shared across all prompts that correspond to the same user and query. As the result, the Key–Value (KV) cache of $X_R$ of be used leading to further reduction of computational complexity.
% For the single logit relevance or engagement signal prediction, we minimize the following objective during the training.
% \begin{gather}
% \min_{\Theta} \mathcal{L}_{ce}(\Theta) \triangleq \mathbb{E}_{(X_p, X_e, Y) \sim \mathcal{D}}l_{ce}(F([X_p, X_e];\Theta),Y)
% \end{gather}
% where $Y$ is the label, and $l_{ce}$ is the sample cross-entropy loss.
% \subsection{Considerations on Encoder and Ranker LLM}
% \subsubsection{Ranker LLM}
% The ranker LLM is responsible for scoring and ordering items according to their relevance to user queries and member profiles. To meet the stringent throughput and latency requirements of online serving, the input length, model size, and output dimension are kept as compact as possible. Our ranker LLM takes as input the query text, user profile text, user engagement history, and item representations in embedding format, and produces a single scalar logit as output.
% During online serving, the query and member text are processed once, and the resulting key–value (KV) cache is stored and shared across different item candidates. This allows inference to begin directly from the item embeddings, significantly reducing redundant computation. The final output logit corresponds to the log-likelihood ratio between two special tokens, \texttt{<Yes>} and \texttt{<No>}, computed as the difference between their logits from the language model head. By employing a compact pre-trained 0.6B model, the end-to-end latency remains well within a few hundred milliseconds, even under production-scale workloads.
% \subsubsection{Encoder LLM}
% For each item, the encoder LLM converts the item text into one or more dense vector representations that encapsulate the semantic information necessary for the ranker LLM to evaluate relevance with respect to user queries and profiles.
% Leveraging language models as embedding encoders has a long and rich history, evolving from early distributional methods such as \textit{word2vec} to modern contextual encoders like BERT. Unlike static embeddings, LLM-based encoders capture deep contextual dependencies, compositional semantics, and task-relevant relationships among entities and actions—properties that are critical for ranking tasks, where subtle linguistic nuances often determine relevance.
% Our encoder is initialized from an auto-regressive language model pre-trained on a large-scale text corpus. While previous work typically adapts such encoders via contrastive fine-tuning, we adopt a task-specific joint training paradigm. Specifically, the encoder LLM and ranker LLM are co-trained end-to-end under a combined ranking and alignment objective. This collaborative optimization leads to encoder representations that are explicitly tuned to maximize downstream ranking performance.
\section{Training}\label{sec:arch}
In this section, we describe the models, datasets and training recipes for the \textbf{MixLM} framework deployed to LinkedIn's semantic job search system.
\begin{table*}[h]
\small
\caption{MixLM Training Stages: In the stage I, we use the chain-of-thought reasoning traces from our in-house relevance judge to pretrain a 0.6B model on the domain-specific data. In stage II, we apply Supervised Fine-Tuning (SFT) on the model from stage I with the text-only ranking dataset, which serves as a teacher model in the next stage. In stage III, we co-train the encoder and ranker LLMs using a ranking dataset. We include several different loss functions in this stage, including the ranking SFT loss, a
distillation loss and a self-alignment regularization.}
\label{tab:training}
\begin{tabular}{lccc}
\toprule
Stages & \makecell{Stage I: \\ Domain-specific Fine-Tuning} & \makecell{Stage II: \\Ranking Teacher Training} & \makecell{Stage III: \\Joint Encoder-Ranking Training} \\
\midrule
Data & Reasoning Dataset &Ranking Dataset &Ranking Dataset \\
Number of Samples & 180k & 10.9M & 10.9M\\
Input Prompt Type & Text (chain-of-thoughts) & Text & Text + Embedding\\
Maximum Sequence Length & 2\,048 & 2\,048 & 2\,176 \\
Trainable Module(s) & Pretrained LLM & Ranker LLM & Encoder LLM \& Ranker LLM \\
Training Objective & Distillation from relevance judge & SFT & SFT, Distillation from ranking teacher, self-alignment\\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Training Overview}
Our MixLM framework requires us to train both the ranker and encoder LLMs. In particular, we require the encoded item embeddings available from the encoder model to align with the ranker model's input embeddings, so the ranker model can make use of the encoded information. In general, as long as the ranker LLM's hidden size is the same as encoder's final output size (which allows us to concatenate embeddings in~\eqref{eq:concat}), they can differ in architecture. However, to streamline the training process, we use models that have the same architecture for ranker and encoder, each with 0.6B parameters. We follow a three-stage training recipe for MixLM, summarized in Table~\ref{tab:training}.
In stage I, we perform supervised fine-tuning on a 0.6B pretrained model using a domain-specific dataset (details to follow). The model resulting from this stage acts as the base model for our ranker model training. In stage II, using a purely text prompt, we train a ranker model using the labeled ground truth, which serves as a \emph{teacher} model for our third training stage. This is motivated by our experience that presenting the model with complete textual item descriptions results in high ranking quality. In stage III, we co-train the ranker and the encoder models using custom loss functions that allow us to distill the text-based model from the second stage into a mixed input model. We use the model from the first training stage as the initial model checkpoint for the second stage and the ranker in the third stage, and a 0.6B General Text Embedding (GTE) model trained using contrastive training as the initial encoder checkpoint in the third stage.
% \section{Training of MixLM}
\begin{figure}
\includegraphics[width=0.5\textwidth]{Figures/emb_paper_training.png}
\caption{Detailed training pipeline of training state III.}
\label{fig:loss figure}
\end{figure}
\subsection{Datasets}
\subsubsection{Label Generation}\label{sec:illuminatorprompt}
We sampled query--item pairs from real user logs, and train a 7B LLM as our relevance judge to generate 5-point graded scores along several dimensions, together with rationales, such that it aligns with the product policy.
The integer labels were subsequently normalized to continuous values in the range $[0, 1]$ to create the ground truth labels $p^*_{\text{yes}}(q,j)$ for query-job pairs.
This 7B in-house LLM operates using the following template:
\begin{verbatim}
[CRITERIA] : <matching guidelines>
[EXAMPLES] : <reasoning and output format>
[QUERY] : <query text>
[CANDIDATE] : <item text>
Analyze the query--candidate pair and determine whether
they are a good match. Return a single matching score
(0/1/2/3/4) with detailed reasoning.
\end{verbatim}
We gather model's chain-of-thought traces~\citep{wei2022chain}, as well as the final matching score which we use in different stages of training.
% label each pair by prompting full-size LLM, e.g. GPT or Claude,
% These dimension-wise grades are aggregated into a final label for the pair.
\subsubsection{Teacher Training Prompts in Stage II}\label{sec:teacher-prompt}
For the textual teacher training, we applied the following chat template to the ranker LLM:
\begin{verbatim}
[SYSTEM] : Output only 'Yes' or 'No' based on how well
the job matches the query; no extra text.
[QUERY] : <query text>
[CANDIDATE] : <item text>
yes or no ?
\end{verbatim}
\subsubsection{Mixed Input Ranking Training Prompts in Stage III}\label{sec:mixedprompt}
For the joint ranking training (co-train the encoder LLM and ranker LLM), we applied the following chat template to the ranker LLM:
\begin{verbatim}
[SYSTEM] : Output only 'Yes' or 'No' based on how well
the job matches the query; no extra text.
[QUERY] : <query text>
[CANDIDATE] : <item embeddings>
yes or no ?
\end{verbatim}
where the \texttt{<item embeddings>} is the output from encoder LLM. The encoder LLM use the following prompt ``\texttt{Item information: <item text>}''. Note that the ranking prompt used here is significantly shorter compared to the teacher one in Section~\ref{sec:teacher-prompt} which uses full item text.
\subsection{Three-stage Training}
Next, we provide the details of our training pipeline.
\subsubsection{Stage I: Domain Reasoning Fine-Tuning}
In the first stage, we train a general pretrained LLM using a domain-specific reasoning dataset to strengthen its capability for logical inference and contextual understanding over query–item pairs. This intermediate training phase enables the model to grasp the rationale behind assigning relevance scores, thereby enhancing its generalization performance in downstream ranking tasks.
Concretely, we distill~\citep{hinton2014distilling} our 7B relevance judge into a 0.6B pretrained model, using KL divergence between the logits of the two models. As for the data, we randomly select $180$K samples following the prompt given in Section~\ref{sec:illuminatorprompt}, together with the chain-of-thoughts responses of the relevance judge model.
% For our first stage (pretraining), and record their chain-of-thoughts in the response to the prompt above. As for the second and third stages (teacher ranker, and joint ranker and encoder training), which we use for training.
% parameter LLM on approximately $180$K reasoning examples (see the prompt presented in Section~\ref{sec:illuminatorprompt}) to instill domain knowledge and chain-of-thought reasoning skills tailored to professional-network scenarios. To further accelerate convergence and stabilize optimization, we incorporate a distillation objective that minimizes the
\subsubsection{Stage II: Ranking Teacher Training}
In the next step, we train a purely textual ranking model. Although this intermediate model cannot be served online due to the long input prompts, it will serve as a teacher model when training the mixed input MixLM model. As we demonstrate, using such a ranking teacher model results in better convergence and training quality.
Specifically, starting from the checkpoint available from our first training stage, we perform Supervised Fine-Tuning (SFT) using the input prompt discussed in Section~\ref{sec:teacher-prompt}. We generate $10.9$M examples based on the prompt provided above, and normalize the integer matching scores, available from the 7B judge, into ground truth probabilities $(p^*_{\text{yes}},p^*_{\text{no}})$. The SFT is then performed using a KL divergence loss as in~\eqref{eq:kl_loss}. We use $(\hat{p}_{\text{yes}},\hat{p}_{\text{no}})$ to denote the output probability distribution from the final checkpoint for this training stage.
\subsubsection{Stage III: Joint Encoder-Ranking Training}\label{sec:stageiii}
In the final training stage, we train both ranker and encoder models to estimate relevance matching scores, using the mixed input prompt from Section~\ref{sec:mixedprompt}. Specifically, the relevance scores are calculated using~\eqref{eq:concat}. The trainable parameters are $\Theta = [\Theta_R, \Theta_E]$ which encompasses all parameters of ranker and encoder models.
To this end, we initialize the ranker module in Figure~\ref{fig:teaser} via the final checkpoint from our first stage training. For the encoder module, we use a pretrained 0.6B GTE model. We use the same 10.9M sample dataset from the previous stage for training.
Given the specific structure of the ranking stack, we have observed using custom loss functions can improve the training quality and the final models' ranking quality. Below we discuss the details of the loss functions we use.
\paragraph{SFT Loss} Our first loss function is a KL divergence SFT loss, that ensures the predicted probabilities from MixLM are close to the ground truth. This loss function is given as
$$\mathcal{L}_{\text{SFT}}(\Theta)=\text{KL}\left((p^*_{\text{yes}},p^*_{\text{no}})||({p}_{\text{yes}}(\Theta),{p}_{\text{no}}(\Theta))\right)$$
where $({p}_{\text{yes}},{p}_{\text{no}})$ are the predicted probability of relevance match (see~\eqref{eq:concat}), and $(p^*_{\text{yes}},p^*_{\text{no}})$ are ground truth probabilities.
\paragraph{Ranking Distillation Loss} In addition to comparing to the ground truth, we compare the output distribution from MixLM to the one from the ranking teacher model from Stage II. More specifically, we define
$$\mathcal{L}_{\text{distill}}(\Theta)=\text{KL}\left((\hat{p}_{\text{yes}},\hat{p}_{\text{no}})||({p}_{\text{yes}}(\Theta),{p}_{\text{no}}(\Theta))\right)$$
where we recall that, $(\hat{p}_{\text{yes}},\hat{p}_{\text{no}})$ is the output distribution of ranking teacher with full text prompt in Section~\ref{sec:teacher-prompt}.
% This loss function is motivated by knowledge distillation~\citep{hinton} approaches, where knowledge from a (usually bigger) teacher model is transferred to another (usually smaller) one. However, we note an important difference between our approach here and a general knowledge distillation framework. Here, the MixLM model, which acts as the student model, is not necessarily a smaller model compared to the teacher. In fact, MixLM, which includes ranker and encoder modules, has more parameters compared to the ranking teacher. The difference is that the teacher model operates on a pure text prompt, from which it can be easier to extract information compared to the compact embeddings utilized by MixLM. As we discuss in Section~\ref{sec:experiments}, including guidance from a pure text teacher is beneficial to MixLM's ranking quality.
This loss function is inspired by knowledge distillation~\citep{hinton2014distilling}, where knowledge from a (typically larger) teacher model is transferred to a (usually smaller) student model. In contrast, our student model, MixLM, which consists of an encoder and a ranker, can in fact be larger than the teacher model. As discussed in Section~\ref{sec:experiments}, the teacher operating on pure text is easier to train to provide high-quality predictions that are cleaner than the original dataset labels. When training MixLM, the teacher label helps reducing gradient variance and facilitates better convergence.
\paragraph{Self-Alignment Loss} Finally, we use two self-alignment loss functions, that help to ensure the encoder model's output embeddings are aligned with the input embeddings of the ranker model. From our experience, including these loss functions has a regularizaiton effect that can prevent overfitting. To this end, we pass the full text prompt (from Section~\ref{sec:teacher-prompt}) through the mixed input model's ranker module (the model parameterized by $\Theta_R$ from section~\ref{sec:arch}), and collect the output probability distribution (which we denote by $(\bar{p}_{\text{yes}},\bar{p}_{\text{no}})$). Additionally, we collect the last layer's (before the output head) hidden states corresponding to the last input token, which we denote by $\bar{h}_{\text{last}}\in\mathbb{R}^{H}$. Note that these quantities are functions of only $\Theta_R$. For the same sample, we also collect the output probability distribution (denoted as $(p_{\text{yes}},p_{\text{no}})$) and the last layer's hidden states for the last input token (denoted as $h_{\text{last}}\in\mathbb{R}^H$) when we use the mixed input prompt from Section~\ref{sec:mixedprompt}. These quantities, as expected, are functions of all trainable parameters $\Theta$. Then, we use loss functions that encourage the resulting quantities from these two setups to be similar. First, we use a loss function $\mathcal{L}_{\text{hidden-align}}$ to encourage the last hidden states from the mixed input and textual prompts to be similar:
$$\mathcal{L}_{\text{hidden-align}}(\Theta) = 1 - \mathrm{cossim} \big(h_{\text{last}}(\Theta), \bar{h}_{\text{last}}(\Theta_R)\big) $$
where $\mathrm{cossim}$ denotes the cosine similarity of two vectors.
In addition, we use another alignment loss to ensure the predicted probabilities remain similar:
$$\mathcal{L}_{\text{pred-align}}(\Theta) = \text{KL}\left((\bar{p}_{\text{yes}}(\Theta_R),\bar{p}_{\text{no}}(\Theta_R))|| (p_{\text{yes}}(\Theta),p_{\text{no}}(\Theta))\right). $$
The overall alignment loss is a linear combination of hidden state and prediction alignment losses:
$$\mathcal{L}_{\text{align}}(\Theta) = \alpha \cdot \mathcal{L}_{\text{pred-align}}(\Theta) \;+\; \beta \cdot \mathcal{L}_{\text{hidden-align}}(\Theta)$$
where $\alpha,\beta\geq 0$ are given hyper-parameters.
Finally, the full training objective integrates all components discussed above. Specifically, we use
$$\mathcal{L}_{\text{total}} (\Theta) = \mathcal{L}_{\text{SFT}}(\Theta)
+ \lambda_{\text{distill}} \mathcal{L}_{\text{distill}}(\Theta)
+ \lambda_{\text{align}} \mathcal{L}_{\text{align}}(\Theta),
$$
where $\lambda_{\text{distill}}\geq0$ and $\lambda_{\text{align}}\geq 0$ are hyper-parameters that are given a priori. Figure~\ref{fig:loss figure} summarizes the training pipeline for this stage.
% \begin{remark}
% We note that ranking distillation loss and self-alignment loss functions follow different objectives. The distillation loss provides training guidance from a purely text-based ranking stack, that was trained without using any mixed input prompt. On the other hand, the self-alignment ensure MixLM's ranker module remains aligned, regardless if it is given a mixed input prompt or a text prompt. In particular, the self-alignment loss does not depend at all on the ranking teacher model from the second stage.
% \end{remark}
Like multi-modal language models, such as LLaVA~\cite{liu2023visual} and BLIP-2~\cite{li2023blip}, the goal of Stage~III is to align the input space of the ranker LLM and the embedding space of the encoder LLM. This is challenging because 1) unlike the discrete vocabulary of text, item embeddings have an infinitely large continuous space, resulting in higher sample complexity; 2) the embeddings encode much more complex and entangled information than text tokens, requiring greater training effort to preserve and decode such information; and 3) item embeddings have never been encountered during foundational pretraining.
To train MixLM more effectively, we apply a phased curriculum learning strategy, decomposing the joint optimization into two distinct phases with different loss weights, i.e., ($\lambda_{\text{distill}}$ and $\lambda_{\text{align}}$): In \textbf{Phase 1}, we prioritizes the space-alignment by increasing $\lambda_{\text{align}}$ and decreasing $\lambda_{\text{distill}}$, so that we can train the encoder to produce embeddings that integrate seamlessly with the ranker's internal representations. In \textbf{Phase 2}, decrease $\lambda_{\text{align}}$ and increase $\lambda_{\text{distill}}$, focusing on adjusting ranker LLM to improve the relevance score prediction.
This approach mirrors the multi-stage training strategies successfully employed in multi-modal LMs, where phased training with different optimization objectives in each phase improves both alignment quality and downstream task performance. During the training, we employ a standard AdamW optimizer, combined with a cosine annealing scheduler with linear warmup.
% During this stage, we employ a standard AdamW optimizer, combined with a cosine annealing scheduler with linear warmup during training.
% The final training stage is trying to solve a "multi-modal-like" alignment problem, where we are aligning ranker LLM and encoder LLM. Inspired by recent advances in multi-modal alignment, particularly the approaches used in vision-language models like LLaVa~\cite{liu2023visual} and BLIP-2~\cite{li2023blip}, we adopt a phased co-training strategy. This phased training paradigm, which can be characterized as a form of curriculum learning, decomposes the joint optimization into two distinct phases with different loss weightings ($\lambda_{\text{distill}}$ and $\lambda_{\text{align}}$):
% \Shusen{$\bullet$ \textbf{Phase 1 (Alignment-focused)}: We increase $\lambda_{\text{align}}$ and decrease $\lambda_{\text{distill}}$ to train the encoder to produce embeddings that integrate seamlessly with the ranker's internal representations. \textbf{Phase 2} }
% We prioritize alignment between the encoder's output embeddings and the ranker's embedding space by assigning a high weight to the self-alignment loss (a larger $\lambda_{\text{align}}$) while maintaining a lower weight for the KL divergence loss to ground-truth labels. This phase ensures the encoder learns to produce embeddings that integrate seamlessly with the ranker's internal representations.
% $\bullet$ \textbf{Phase 2 (Task Performance-focused)}: We increase $\lambda_{\text{distill}}$ and decrease $\lambda_{\text{align}}$ to better match ground-truth relevance judgments after a good alignment from phase 1.
% We shift the focus toward final task performance by increasing the weight on the KL divergence loss while reducing the self-alignment regularization weight (a smaller $\lambda_{\text{align}}$). This allows the model to better match ground-truth relevance judgments after a good alignment from phase 1.
% In the second stage, the encoder LLM and ranker LLM are trained jointly in an end-to-end manner on a large-scale ranking dataset. The training objective minimizes the cross-entropy loss:
% \begin{equation}
% \min_{\Theta} \mathcal{L}_{\mathrm{CE}}(\Theta)
% \triangleq
% \mathbb{E}_{(X_p, X_e, Y) \sim \mathcal{D}}
% \big[l_{\mathrm{CE}}(F([X_p, X_e]; \Theta), Y)\big],
% \end{equation}
% where $F(\cdot; \Theta)$ denotes the overall MixLM model parameterized by $\Theta$, and $Y$ between $0$ and $1$ represents the ground-truth relevance label.
% Through this optimization, the model learns to predict a single scalar value that reflects the relevance between a query–item pair, corresponding to a (0–1) ranking score generated by the final output token of the ranker LLM. Specifically, we train the MixLM on a dataset of $10.9$M ranking examples. The query and re provided to the ranker LLM as plain text. Candidate-item text is processed by the encoder LLM; we sample the embeddings of its final N tokens and pass them to the ranker LLM. The ranking loss is computed on the ranker LLM output and gradients are back-propagated through both the ranker LLM and encoder LLM parameters. Specifically, we train MixLM on a large-scale dataset containing $10.9$M ranking examples. The query and other static features are provided to the ranker LLM in plain-text form, while the candidate item text is processed by the encoder LLM. We extract the embeddings of the final $N$ tokens from the encoder LLM and feed them into the ranker LLM, i.e.,
% $$\mathrm{Samp}: \mathbb{R}^{B\times T_E \times H}\rightarrow \mathbb{R}^{B\times N \times H}, h_E \mapsto h_E[:, -N:, :]$$.
\subsection{Training Efficiency}
During the ranking training phase, we co-train the encoder LLM and the ranker LLM end-to-end. We deploy PyTorch’s Fully Sharded Data Parallel~\citep{zhao2023fsdp} to shard the decoder parameters, gradients, and optimizer states across 8 nodes (each with 8 NVIDIA H100 GPUs). Within each node, GPUs communicate via NVLink, and inter-node communication is via InfiniBand.
To reduce communication overhead, we enable layer-wise prefetching: before computing the next forward layer, we prefetch its parameters; during the backward pass, we prefetch gradients of upcoming layers. To further reduce memory footprint, we incorporate Liger Kernel~\citep{hsu2024liger}, a set of optimized Triton kernels (e.g. fused RMSNorm, RoPE, SwiGLU, fused CrossEntropy) that deliver higher throughput and lower memory usage.
In our setup, the total compute cost to train on 70B tokens under the ranking objective is approximately 700 H100 GPU-hours.
% To further enhance ranking performance and training stability, several auxiliary objectives are incorporated, including internal alignment losses between hidden states and external alignment losses with contrastive or distillation-based supervision. These auxiliary tasks guide the encoder–ranker pair toward producing semantically consistent and discriminative representations.
% In this section, we describe the training recipe for MixLM. To make the model generalize better, we propose a two-stages training. In the first stage, we do a reasoning training: Ranker LLM's is trained on a reasoning dataset, to conduct reasoning and analysis for queries and items in domain specific data. At the second stage, we do the ranking ranking training: Encoder LLM + Ranker LLM end-to-end training, through minizing the CE loss below.
% $$\min_{\Theta} \mathcal{L}_{ce}(\Theta) \triangleq \mathbb{E}_{(X_p, X_e, Y) \sim \mathcal{D}}l_{ce}(F([X_p, X_e];\Theta),Y)$$
% the Encoder LLM and Ranker LLM are trained end to end, on a ranking dataset. The major goal is for the model to output a 0-1 ranking score, as the single output token of MixLM. To enhance the model performance, we also introduced several auxiliary losses in the training.
% Each pair, together with task instructions, was fed to a large teacher model (either our proprietary LLM or a commercial API such as GPT or Claude). In this work we rely exclusively on our in-house model. The prompting template is:
% Executing this procedure produced 10.9 M records.
% • Reasoning subset: We randomly selected 180 K examples and retained the full prompt plus the teacher’s chain-of-thought; these data are used for domain-specific reasoning (§X.X).
% • Ranking subset: For the remaining 10.9 M examples, we kept only (query, item, score), yielding the dataset used in ranking training.
% \subsubsection{Domain-Specific Reasoning Training}
% We first fine-tune the ranker LLM on the 180 K reasoning examples to instill domain knowledge and chain-of-thought skills for professional-network tasks. To accelerate convergence, we incorporate a distillation loss that minimizes the KL divergence between the logits of the ranker LLM and those of the teacher model.
% \subsubsection{Ranking Training}
% Next, we train the full MixLM end-to-end on the 10.9 M ranking examples. The query and member‐history texts are provided to the ranker LLM as plain text. Candidate-item text is processed by the encoder LLM; we sample the embeddings of its final N tokens and pass them to the ranker LLM. The ranking loss is computed on the ranker LLM output and gradients are back-propagated through both the ranker LLM and encoder LLM parameters.
% \textbf{Auxiliary Losses in Ranking Training}
% One of the major challenges during training is enabling the ranker LLM to understand embeddings produced by the encoder LLM, which it has not encountered during foundational pretraining. To address this issue, we introduce internal and external alignment techniques to enhance the training process.
% \subsection{Internal and External Alignment Regularization}
% \subsubsection{Internal Alignment}
% For internal alignment, we enforce both prediction-level and hidden-state-level consistency between the proposed architecture and the ranker LLM when processing the full-text input. To be specific, let $P_{Y|X_p, X_e}^\Theta$ and $P^{\Theta_R}_{Y|X_p, X_e}$ denote the prediction distribution of the \texttt{<Yes>} and \texttt{<No>} from the proposed encoder-ranker LLM architecture $F([X_p,X_e];\Theta)$ and the ranker LLM with full-text $F_R([X_p,X_e];\Theta_R)$, respectively, then the prediction-level internal alignment regularization can be expressed as
% $$\mathcal{L}_{\text{pred-align}}(\Theta) = \mathrm{KL}\!\left(P_{Y|X_p, X_e}^{\Theta_R} \;\|\; P_{Y|X_p, X_e}^\Theta \right). $$
% The hidden-state-level consistency is measured by cosine similarity, which results in the following regularizer
% $$\mathcal{L}_{\text{hidden-align}}(\Theta) = 1 - \mathbb{E}_{X_p,X_e}\mathrm{cossim} \big(h^{\Theta}([X_p,X_e]), h^{\Theta_R}([X_p,X_e])\big) $$
% where $h^{\Theta}([X_p,X_e])$ and $h^{\Theta_R}([X_p,X_e])$ are the last hidden-states representations of the ranker LLM from the proposed architecture $F([X_p,X_e];\Theta)$ and the ranker LLM with full-text $F_R([X_p,X_e];\Theta_R)$, respectively. The \textbf{internal-alignment} regularizer can be represent as a weighted combination of the above regularizers
% where $\alpha$, $\beta$ are hyperparameters controlling relative importance.
% A central difficulty in embedding compression is that the compressed representation (e.g., the compressed tokens) is injected into the ranker LLM at a different point than the original full-text job description. Without explicit constraints, this compressed path can diverge from the behavior of the base ranker, leading to semantic drift. To mitigate this, we introduce \textbf{self-alignment losses} and \textbf{guided alignment (distillation) loss} that enforce consistency between the compressed and full-text pathways.
% Let $p_{\theta}(y \mid q, \hat{d})$ denote the ranker LLM’s prediction distribution when using compressed job tokens, and $p_{\theta}(y \mid q, d)$ denote the ranker’s distribution when using full text.
% We design two complementary self-alignment objectives:
% \begin{itemize}
% \item \textbf{Prediction-level alignment.}
% We minimize the KL divergence between the compressed and full-text output distributions over the relevance labels $y\in \{\text{Yes}, \text{No}\}$:
% $$\mathcal{L}_{\text{pred-align}} = \mathrm{KL}\!\left( p_{\theta}(y \mid q, d) \;\|\; p_{\theta}(y \mid q, \hat{d}) \right)$$
% This encourages the compressed pathway to reproduce the same decision boundary as the full model.
% \item \textbf{Hidden-state alignment.}
% Let $h_{\text{full}}$ and $h_{\text{comp}}$ be the final hidden representations of the last token in the full and compressed pathways, respectively. We minimize their cosine distance:
% $$\mathcal{L}_{\text{hidden-align}} = 1 - \cos\!\big(h_{\text{comp}}, h_{\text{full}}\big)$$
% This ensures the compressed embedding token induces latent representations that remain geometrically aligned with those from full-text pathway.
% \end{itemize}
% The combined self-alignment loss is a weighted mixture:
% $$\mathcal{L}_{\text{align}} = \alpha \cdot \mathcal{L}_{\text{pred-align}} \;+\; \beta \cdot \mathcal{L}_{\text{hidden-align}}$$
% where $\alpha$, $\beta$ are hyperparameters controlling relative importance.
% \subsubsection{External Alignment}
% The external alignment enforces prediction to be consistent with the output of a well-trained teacher ranker LLM with full text, which is a essentially knowledge-distillation process expressed as
% $$\mathcal{L}_{\text{ext-align}}(\Theta) = \mathrm{KL}\!\left(Q_{Y|X_p, X_e} \;\|\; P_{Y|X_p, X_e}^\Theta \right)$$
% where $Q_{Y|X_p, X_e}$ is teacher model with full-text input.
% In addition to self-alignment, our framework leverages knowledge distillation as a form of guided alignment between compressed and full-text pathways. Specifically, the teacher ranker LLM takes full-text input and provides soft targets that guide the student model not only in embedding space but also in the student ranker’s predictions. This allows the student to produce outputs equivalent to reading the full input, effectively transferring both representational and task-level knowledge from teacher to student.
% \begin{itemize}
% \item \textbf{Guided alignment/Distillation.} Let $\mathbf{y}_s$ and $\mathbf{y}_t$ be student and teacher logits, where student model takes compressed embedding and teacher model takes full-text pathway.
% $$\mathcal{L}_{\text{distill}} = \text{KL}\Big(
% \text{softmax}(\mathbf{y}_s / T) \;\|\; \text{softmax}(\mathbf{y}_t / T) \Big) \cdot T^2$$
% \end{itemize}
% This multi-task objective ensures that compressed embeddings not only match the teacher’s predictions but also remain aligned with the full-text latent representations, leading to more stable and effective training.
% \subsubsection{Auxiliary tasks for improved alignment}
% In our framework, we expect the encoder LLM to compress the useful information, which is then decoded and leveraged by the ranker LLM for the prediction task. Achieving this alignment is often challenging, particularly when training data is limited. To address this issue, we design auxiliary tasks to enhance the alignment and model generalization.
% The auxiliary tasks are introduced prior to the ranking tasks, where the ranker models are trained to answer questions related to the information encoded in the embeddings. In our framework, we employ two auxiliary SFT tasks: ``repeat" and ``reasoning". In the ``repeat" task, the ranker model is trained to reproduce the exact tokens encoded by the encoder LLM. In the ``reasoning" task, the ranker model is trained to generate explanations for its ranking predictions based on the information contained in both the embeddings and the accompanying text. The training of auxiliary tasks can be further represented as
% \begin{gather}
% \min_{\Theta} \mathcal{L}_{aux}(\Theta) \triangleq \frac{1}{|\mathcal{D}|}\sum_{(x,y)\in \mathcal{D}}l_{ce}(F(x;\Theta)[-L_y:],y)
% \end{gather}
% where the indexing $-L_y:$ refers to the last $L_y$ logits of the ranker LLM output, and $L_y$ is the length of target $y$, i.e., token IDs of the embedding information or reasoning.
\section{System Implementation}
\subsection{Online System}
The online inference architecture adopts a two-layer design consisting of a feature generation layer and a serving layer (see Figure~\ref{fig:serving} for an overview). The feature generation layer functions as a hybrid system that combines offline refresh with near-line ingestion and generation to maintain comprehensive and up-to-date feature coverage. The offline pipeline, orchestrated through Flyte, periodically performs large-scale GPU-based inference to regenerate the full corpus of text and embedding features. In parallel, the near-line component continuously ingests real-time updates reflecting changes in the population of scoring candidates and generates corresponding features through GPU inference. Because near-line processing is not latency-sensitive, GPU resources can be utilized with high throughput and efficiency. All generated features are persistent in a high-throughput distributed storage system, ensuring consistency, low-latency availability for downstream serving.
The serving layer performs real-time inference by retrieving precomputed features and assembling mixed text–embedding prompts that integrate user queries with candidate scoring items. Prior to dispatching requests to the SGLang inference engine~\citep{sglang} for GPU-based scoring, several system-level optimizations—including distributed caching, latency reduction mechanisms, and scalable serving infrastructure—are applied to minimize feature retrieval overhead and improve serving efficiency. This layered architecture enables effective reuse of precomputed representations while preserving responsiveness across both near-line and real-time applications. Empirical evaluations demonstrate that these design choices significantly enhance inference efficiency, GPU utilization, and system scalability, supporting production-scale deployment of large language model–based ranking systems.
\begin{figure}
\centering
\includegraphics[width=1\linewidth]{Figures/midtier.png}
\caption{Online Serving Architecture for LinkedIn's Semantic Job Search}
\label{fig:serving}
\end{figure}
\subsection{Inference Engine}
\subsubsection{Text-Embedding Mixed Input}
Common open-source LLM inference engines focus on pure-text inputs, restricting requests to tokenized sequences.
To enable mix-interaction in MixLM, we extended the input interface to additionally accept precomputed embedding tensors.
These embeddings are transmitted using a structured \textit{feature payload} abstraction added to the gRPC request schema.
Each feature payload encapsulates a binary buffer containing tensor values, along with metadata specifying the tensor's shape,
element type, and optional serialization format (e.g., NumPy~\citep{numpy}-compatible).
This design allows clients to include dense embeddings alongside text inputs without modifying the text-based interface
or introducing format-specific logic in the serving stack.
On the server side, the preprocessing module decodes and validates these feature payloads into native tensor objects, applying metadata-driven reshaping and casting as necessary.
The resulting tensors are then attached to the model input context for downstream inference.
This approach enables direct reuse of the embeddings computed by upstream retrieval or feature pipelines,
avoiding redundant re-computation and reducing latency.
The extended schema remains fully backward-compatible with existing text-only requests
while introducing minimal overhead in serialization and deserialization.
\begin{table*}[t]
\small
\caption{Attention and linear-layer compute under different conditions, in terms of $(T_q, T_i, N_i)$, where $T_i$ denotes the full item-text length and $K$ is the MixLM compression factor (item length becomes $T_i/K$ in the prompt). The symbol $\propto$ indicates proportionality up to architecture-dependent constants (layers, heads, hidden size, etc.). The bottom blocks instantiate the expressions for $N_i = 250$ and $K = 450$.}
\label{tab:complex1}
\begin{tabular}{lll}
\toprule
\textbf{Conditions}
& \textbf{Total attention cost}
& \textbf{Total linear-layer cost} \\
\midrule
Naive (full-text items)
& $\propto N_i (T_q + T_i)^2$
& $\propto N_i (T_q + T_i)$ \\
\hline
Amortized prefill (full-text items)
& $\propto T_q^2 + N_i (2 T_i T_q + T_i^2)$
& $\propto T_q + N_i T_i$ \\
\hline
\makecell[l]{Amortized prefill \\ $+$ MixLM (item length $T_i/K$)}
& $\propto T_q^2 + N_i \bigl( 2 (T_i/K) T_q + (T_i/K)^2 \bigr)$
& $\propto T_q + N_i T_i / K$ \\
\midrule
\makecell[l]{Naive (full-text items) \\ with $N_i = 250$}
& $\propto 250 (T_q + T_i)^2$
& $\propto 250 (T_q + T_i)$ \\
\hline
\makecell[l]{Amortized prefill (full-text items) \\ with $N_i = 250$}
& \makecell[l]{
$\propto T_q^2 + 250 (2 T_i T_q + T_i^2)$ \\
$= T_q^2 + 500 T_i T_q + 250 T_i^2$
}
& $\propto T_q + 250 T_i$ \\
\hline
\makecell[l]{Amortized prefill $+$ MixLM \\ with $N_i = 250$, $K = 450$}
& \makecell[l]{
$\propto T_q^2 + 250 \bigl( 2 (T_i/450) T_q + (T_i/450)^2 \bigr)$ \\
$= T_q^2 + \frac{10}{9} T_i T_q + \frac{1}{810} T_i^2$
}
& \makecell[l]{
$\propto T_q + 250 T_i / 450$ \\
$= T_q + \frac{5}{9} T_i$
} \\
\bottomrule
\end{tabular}
\end{table*}
\subsubsection{Shared-Prefix Prefill Optimization}
In large-scale search and recommendation platforms, a single ranking request typically contains hundreds to thousands of candidate items, all associated with the same query and user-history features. After we compress each item's description into a small, fixed set of embedding tokens, the computational burden of inference shifts predominantly to the repeating query/user-history side, because these tokens now constitute the majority of the prompt. For example, in our production system a candidate item was previously represented by roughly 900 tokens as median; with MixLM this representation is reduced to only two tokens—one special token and one embedding token.
To improve our GPU utilization during online inference, we use a batched inference mechanism. That is, for each query, we batch several prompts, each corresponding to a different candidate item, and run inference on the batch. Given that all prompts in a batch share the same prefix, this allows for reusing the KV cache across different prompts. This is particularly important when combined with the mixed-input system: once the item representation is reduced to just a few tokens, the shared query-prefix computation dominates the overall inference cost.
Let $T_q$ and $T_i$ denote the number of shared query tokens and the number of item tokens in the \emph{full-text} representation, respectively, and let $N_i$ denote the ranking depth. Each transformer block consists of a multi-head self-attention module and a set of linear layers (query/key/value projections and a feed-forward network). For a sequence of length $L$, we write the attention and linear costs using proportionality (ignoring architecture-dependent constants such as depth, number of heads, and hidden/FFN dimensions) as
\[
\text{Self-attention cost} \;\propto\; L^2,
\qquad
\text{Linear/FFN cost} \;\propto\; L.
\]
Thus, attention is quadratic in the sequence length $L$, while the linear layers are linear in $L$ but may have a larger constant factor.
\paragraph{Naïve prefill cost (full-text items).}
Without amortization, each candidate item forms an independent sequence of length $(T_q + T_i)$. Across $N_i$ items, the total attention and linear prefill costs scale as
\[
F_{\text{att}}^{\text{naive}}
\;\propto\;
N_i (T_q + T_i)^2,
\qquad
F_{\text{lin}}^{\text{naive}}
\;\propto\;
N_i (T_q + T_i).
\]
When both $T_q$ and $T_i$ are large, $F_{\text{att}}^{\text{naive}}$ contains dominant contributions from $N_i T_q^2$, $N_i T_i^2$, and $N_i T_q T_i$, while $F_{\text{lin}}^{\text{naive}}$ grows linearly in $(T_q + T_i)$.
\paragraph{Amortized prefill (full-text items).}
With \textit{amortized prefill}, only the first sequence in a batch performs the full prefill over the query prefix; the remaining $N_i - 1$ sequences reuse its KV cache and do not recompute query self-attention. The attention cost decomposes into
\[
F_{\text{att}}^{\text{amort, full}}
\;\propto\;
\underbrace{T_q^2}_{\text{query prefix once}}
\;+\;
\underbrace{N_i \bigl( 2 T_i T_q + T_i^2 \bigr)}_{\text{items attending to prefix and themselves}},
\]
while the linear layers are applied once to the query prefix and once to each item token:
\[
F_{\text{lin}}^{\text{amort, full}}
\;\propto\;
T_q + N_i T_i.
\]
\paragraph{Amortized prefill with MixLM compression.}
In MixLM, we use $T_i$ to denote the length of the original full item text, and introduce a compression factor $K > 1$ such that the item is represented by $T_i / K$ tokens in the mixed-input prompt. In our online A/B test, we obtained a compression factor of $K \approx 450$, i.e., each item is compressed from hundreds of tokens down to two tokens in the prompt.
Under amortized prefill with MixLM, the query prefix of length $T_q$ is still processed once, but each item now contributes only $T_i / K$ tokens. The attention cost becomes
\[
F_{\text{att}}^{\text{amort, MixLM}}
\;\propto\;
T_q^2
\;+\;
N_i \Bigl( 2 (T_i/K) T_q + (T_i/K)^2 \Bigr),
\]
and the linear-layer cost is
\[
F_{\text{lin}}^{\text{amort, MixLM}}
\;\propto\;
T_q + N_i T_i / K.
\]
Compared to the full-text setting, both the attention and linear components of the item-dependent cost are reduced by approximately a factor of $K$ on the item side. In particular, the dominant cross-term in attention changes from $N_i T_i T_q$ to $N_i (T_i/K) T_q$, and the item-side linear cost changes from $N_i T_i$ to $N_i (T_i/K)$.
% \paragraph{Which component dominates in LLM ranking?}
% In modern decoder-only LLMs, the FFN and projection (linear) layers often account for a substantial fraction of the total FLOPs via their cost $\propto L$, while the attention layers are responsible for the quadratic dependence on sequence length via $\propto L^2$. In our ranking setup, $T_q$ is on the order of hundreds of tokens, the full-text item length $T_i$ is also large (hundreds of tokens), and the ranking depth $N_i$ is in the hundreds. In the naïve full-text setting, both $F_{\text{att}}^{\text{naive}}$ and $F_{\text{lin}}^{\text{naive}}$ are substantial. Amortized prefill eliminates repeated quadratic computation over the query prefix, and MixLM further reduces the item-side length from $T_i$ to $T_i/K$ with $K \approx 450$, so that the marginal cost per additional item scales with $(T_i/K)$ instead of $T_i$ in both attention and linear layers. As a result, the ranking cost per item is dominated by cross-attention and linear computation that are effectively \emph{linear} in the original item length up to the compression factor, and the empirical end-to-end speedup closely follows these proportionality relations. In Table~\ref{tab:complex1}, we summarize the dependence on $(T_q, T_i, N_i)$ for both attention and linear computation, and also show concrete instantiations for our production setting with $N_i = 250$ and $K = 450$.
To realize this amortization, we explored two approaches:
\begin{itemize}
\item \textbf{In-batch prefix caching}
We reuse the key--value (KV) states from the first prompt so that all items in the batch share the same query-prefix computation in a single forward pass. The attention computation merges two operations:
\begin{enumerate}
\item suffix tokens attend to all prefix tokens from the first prompt via dense paged attention;
\item suffix tokens attend to themselves via regular causal attention.
\end{enumerate}
\item \textbf{Multi-item scoring}
We score multiple items against a single query within one forward pass by concatenating the query and all item texts into a single sequence \cite{sglang_pr_10979}, separated by a special delimiter token. FlashInfer’s item-aware masking ensures that each item attends only to the query and not to any other item. During prefill, metadata such as prefix length, item boundaries, and maximum item length is passed directly to the FlashInfer kernel to build the correct attention mask and prevent cross-item dependencies.
\end{itemize}
Because $T_i$ is extremely small in our mixed-input approach, amortized prefill dramatically reduces redundant prefix computation and yields substantial inference-throughput improvements in production.
% In our initial experiments, using amortized prefill led to only a 20\% increase in throughput, due to other bottlenecks present in the system. As we discuss below, we have optimized other aspects of the serving system, in order to unlock the full potential of MixLM system in terms of throughput.
% \begin{figure}[h]
% \centering
% \includegraphics[width=\linewidth]{Figures/amortized.png}
% \caption{Saved GPU Compute}
% \label{fig:saved_compute}
% \end{figure}
%\subsubsection{Multi-Process server architecture}
% After optimizing both the model and the \texttt{sglang} engine, we identified preprocessing in the Python gRPC service as the new performance bottleneck. Due to Python’s Global Interpreter Lock (GIL), tasks such as gRPC handling, data serialization, and request preprocessing are constrained to a single CPU core per process. To address this limitation, we introduced a multiprocessing server architecture that decouples the gRPC server from the \texttt{sglang} engine. This separation allows us to fully exploit the parallelism of the \texttt{sglang} engine, resulting in a further 40\% increase in end-to-end throughput.
\subsubsection{CPU Overhead Reduction}
\paragraph{Multi-Process gRPC}
After optimizing both the model and the SGLang engine, we observed that request preprocessing in the Python gRPC service became the new performance bottleneck.
Although the gRPC server is implemented with asynchronous Python, tasks such as request deserialization, feature decoding, and context construction remain constrained by Python's Global Interpreter Lock (GIL) and the single-threaded nature of the asyncio event loop.
High request volume can exacerbate these bottlenecks due to contention on the event loop and per-request overhead, limiting throughput despite asynchronous I/O.
To overcome this constraint, we implemented a multiprocessing server architecture that decouples the gRPC server from the SGLang engine.
This design enables preprocessing and request handling to run across multiple CPU cores in parallel, while the SGLang engine executes independently.
As a result, this architecture yields a 40\% improvement in end-to-end throughput.
\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth]{Figures/grpc_service.png}
\caption{Multiprocessing server for gRPC}
\label{fig:multi_server}
\end{figure}
\paragraph{Batch Send.}
To better leverage amortized prefill, the SGLang scheduler must group prompts that share the same query. To address this, we introduced an explicit \textit{batch send} pathway in which the entire set of item prompts for a query is serialized and sent as a single ZMQ message. This guarantees that the scheduler receives and processes the full batch together, allowing it to admit the batch as a unit and apply in-batch prefix caching efficiently.
% To support this, we introduced an explicit \textit{batch send} code path that transmits the entire request-side batch in a single ZMQ message, ensuring that the scheduler admits and schedules the batch as a whole.
% This change relies on small, explicit batch structures and is controlled by the existing \texttt{tokenizer-batch} flag.
\paragraph{Multi-Process Pallellization}
Given our workload characteristics, the prefill phase completes rapidly on GPU, but request scheduling on a single CPU core became the bottleneck. We addressed this by deploying multiple serving engine's scheduler processes (e.g., 5 workers) sharing a single GPU instance, with each process allocated a dedicated memory partition (e.g., 20\% GPU memory per worker). This parallelizes CPU scheduling overhead across multiple cores while maintaining full GPU utilization, effectively pipelining batch preparation with GPU execution.
\subsubsection{Eliminating Redundant KV-Cache Operations}
SGLang's default behavior caches computed key-value pairs during prefill to enable efficient autoregressive decoding. However, our scoring use case is prefill-only with no decode phase. We modified the cache management to immediately free KV cache memory upon request completion rather than persisting it for non-existent decode operations. This eliminates unnecessary memory allocation, cache eviction logic, and pool management overhead on the critical path. More importantly, it increases effective GPU memory capacity, allowing higher batch sizes and concurrent request throughput.
\section{Experimental Results}\label{sec:experiments}
% \textbf{Datasets and Evaluation}
We perform offline and online experiments to understand the ranking quality and throughput improvements of our proposed framework. We use a held-out test set, that has been labeled using our internal LLM judge.
We compare our mixed input system with a few baselines. The first baseline is a bi-encoder embedding-based retrieval model, which has been distilled from our LLM relevance judge. This is the retrieval layer of our semantic search stack. The next baseline is a pure-text system, that is, the 0.6B model available from our second training stage. This represents the ideal ranking model, though it is too inefficient to be deployed. In addition, we compare with LinkedIn's limited-scale production baseline~\citep{behdin2025scaling} that uses a near-line text-summarization model to summarize item descriptions and apply aggressive pruning to the ranker model.
In terms of metrics, we report the NDCG@10 on the labeled held-out set. Additionally, we report the maximum throughput we can achieve for each baseline in our production setup. In particular, we report the maximum number of items we can score per GPU per second, while ensuring the P99 end-to-end latency remains below 500 milliseconds.
\subsection{Comparisons with Baselines}
\begin{table}[t]
\caption{Rankin quality comparison of several ranking approaches: We report NDCG@10 with respect to the labels provided by our relevance judge. See Section~\ref{sec:experiments} for a description of baselines.}
\small
\label{tab:sota1}
\begin{tabular}{lc}
\toprule
Model & NDCG@10\\
\midrule
Full Text & 0.9432\\
Summarized, Pruned & 0.9218\\
\textbf{MixLM} & 0.9239\\
\midrule
Embedding Retrieval & 0.8380\\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{GPU efficiency comparison of ranking approaches. For each ranker, we report the observed maximum throughput (items scored per H100 GPU per second) under a fixed 500 ms latency budget. For reference, we also include the throughput of the retrieval layer (embedding-based exhaustive GPU retrieval, deployed on A100 GPUs). See Section~\ref{sec:experiments} for details on the baseline methods.}
\small
\label{tab:sota2}
\begin{tabular}{lrrr}
\toprule
Model & QPS(Items/Sec/GPU) &Latency (ms) &GPU\\
\midrule
Full Text &290 &<500 &H100\\
Summarized, Pruned &2\,200 &<500 &H100\\
\textbf{MixLM} &22\,000 &<500 &H100\\
\midrule
Embedding Retrieval &$>1.6 \times 10^9$ & <100 &A100\\
\bottomrule
\end{tabular}
\end{table}
Table \ref{tab:sota1} summarizes the ranking quality and Table \ref{tab:sota2} summarizes system efficiency of the baselines we discussed. Embedding-based retrieval, while able to process well over $1.6 \times 10^9$ documents per seconds on a single weaker A100 GPU, delivers the weakest ranking quality and is therefore usually employed only as a coarse first-stage filter. Feeding the full documents to a large language model
(pure-text LLM) pushes NDCG to 0.9432, but its heavy token footprint limits throughput to merely 290 items/sec/GPU. Compressing each document into a short summary before prompting the LLM (summarized text LLM) and pruning ranker LLM's parameters by ~37\% \cite{behdin2025scaling} alleviates the latency bottleneck somewhat—throughput rises to ~2,200 items/sec/GPU—yet ranking quality NDCG@10 is still an acceptable value as 0.9218. Our proposed MixLM achieves a similar NDCG@10 of 0.9239. However, we observe that MixLM allows us to scale up the throughput to 22,000 items/sec/GPU, i.e., a 10.0x speed-up over summarized-text LLM and a 75.9x speed-up over full-text LLM, at the cost of only a 1.8-point NDCG gap with respect to the latter. In other words, MixLM hits a sweet spot in the quality–efficiency trade-off: it is orders of magnitude faster than existing LLM-ranking baselines while retaining virtually the same retrieval effectiveness, making it an attractive drop-in replacement for latency-critical production systems.
\subsection{Online A/B Test}
Table \ref{tab:abtest} summarizes the findings. After rolling out MixLM in LinkedIn’s AI Job Search and running extensive A/B tests, we observed: MixLM is has on-par metrics comparing with the existing limited-scale production baseline(summarized text LLM) ~\citep{behdin2025scaling}. With the 10.0x gain in serving throughput under the same latency budget, MixLM empowered the full-traffic rollout of LLM-ranking-based Semantic Job Search and for the first time, consequently producing a significant 0.47\% increase in Daily Active Users, compared with the traditional job search.
\begin{table}[t]
\caption{Online A/B Test}
\label{tab:abtest}
\begin{tabular}{lc}
\toprule
Experiment Group & DAU \\
\midrule
% raw-text &-0.20\% & -0.02\% \\
Classic Job Search & -- \\
\textbf{Semantic Job Search (powered by MixLM)} & \textbf{+0.47\%} \\
% \midrule
% Embedding-Retrieval-Only & -12\% &-0.16\%\\
\bottomrule
\end{tabular}
\end{table}
\subsection{Ablation Analysis}
We conducted extensive ablation studies on a smaller subset that mirrors the distribution of the full training corpus, which is common practice in LLM training because full-scale experiments would have required high GPU hours. Within this setting, we systematically analyzed every major component of MixLM training—(1) dataset size, (2) number of token embeddings per item, (3) domain-reasoning fine-tuning, (4) auxiliary loss functions, and (5) curriculum learning strategies. Also, we conducted ablations on different inference engine optimization methods aimed at improving the computation of shared prefix prompt (query and the user-history).
\subsubsection{Scaling Up}
Expanding the amount of training data had a clear positive effect on ranking quality. In the ablation study, we progressively enlarged the corpus from 160 K to 1.08 M instances, observing a consistent increase in NDCG@10 at each step. Motivated by this trend, the model deployed in the online A/B experiment was trained on the full dataset of 10.9 M samples.
A similar monotonic gain was observed when we varied the representational granularity of each item. Raising the number of embedding tokens per item from 1 to 50 produced steady improvements in NDCG@10. Nonetheless, to meet strict latency requirements, the production configuration retains a single embedding token per item (followed by 1 special end-of-sentence text token).
% \begin{table}[t]
% \caption{Ablations on Dataset Size}
% \label{tab:ablationsample}
% \begin{tabular}{cc}
% \toprule
% Num Training Sample & NDCG@10 \\
% \midrule
% 160k & 0.880 \\
% 400k & 0.9050 \\
% 800k & 0.9080 \\
% 1.08M & 0.9134 \\
% \bottomrule
% \end{tabular}
% \end{table}
\begin{table}[t]
\caption{Ablations on Dataset Size, on small sub-dataset}
\label{tab:ablationsample}
\begin{tabular}{lc}
\toprule
Num Training Sample & NDCG@10 \\
\midrule
160k & -- \\
400k & +0.0250 \\
800k & +0.0280 \\
1.08M & +0.0334 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{Embeddings Token Number per Item, on small sub-dataset}
\label{tab:ablationtoken}
\begin{tabular}{lc}
\toprule
Tokens/item & NDCG@10 \\
\midrule
1 & -- \\
5 & +0.0017 \\
10 & +0.0044 \\
20 & +0.0111 \\
30 & +0.0158 \\
40 & +0.0172 \\
50 & +0.0198 \\
\bottomrule
\end{tabular}
\end{table}
% \begin{table}[t]
% \caption{Token Number per Item, on small dataset}
% \label{tab:ablationtoken}
% \begin{tabular}{ccc}
% \toprule
% Tokens/item & NDCG@10 \\
% \midrule
% 1 & 0.8536 \\
% 5 & 0.8553 \\
% 10 & 0.8580 \\
% 20 & 0.8647 \\
% 30 & 0.8694 \\
% 40 & 0.8708 \\
% 50 & 0.8734 \\
% \bottomrule
% \end{tabular}
% \end{table}
\subsubsection{Domain Reasoning Fine-Tuning}
The ranker LLM serves as the backbone of our model architecture: both the query tokens and the compressed embedding tokens pass through this ranker to produce the final binary (yes/no) prediction. Consequently, the quality of the ranker LLM directly determines the overall ranking performance.
We found that employing a stronger base model for the ranker substantially improves the final results. Specifically, we compared two configurations: (1) using a open-source pretrained 0.6B model as the base, and (2) using a domain reasoning fine-tuned version of the same model. The results, summarized in Table~\ref{tab:ranker-quality}, clearly show that the domain reasoning tuned base model yields superior bi-encoder performance.
\begin{table}[h]
\caption{Effect of Domain Reasoning Fine-tuning on ranker LLM's Ranking performance, on small sub-dataset}
\label{tab:ranker-quality}
\centering
\begin{tabular}{l c}
\hline
\textbf{Ranker Base Model} & \textbf{NDCG@10} \\
\hline
vanilla open-source LLM & -- \\
Domain-Reasoning Tuned LLM & \textbf{+0.0185} \\
\hline
\end{tabular}
\end{table}
% \begin{table}[h]
% \centering
% \begin{tabular}{l c}
% \hline
% \textbf{Ranker Base Model} & \textbf{NDCG@10} \\
% \hline
% open-source LLM & 0.8252 \\
% task-tuned LLM & 0.8437 \\
% \hline
% \end{tabular}
% \caption{Effect of ranker LLM quality on retrieval performance.}
% \label{tab:ranker-quality}
% \end{table}
Based on these findings, we have adopted Domain Reasoning Fine-Tuning as the first stage of our training recipe. In the first stage, we obtain a more powerful base model for the ranker LLM via task-specific COT data, guided by our internal 7B relevance judge LLM as the teacher model. After that, we use this tuned model as the Ranker LLM's starting base model, to initialize the training of stage III: Joint Ranking Training
\subsubsection{Self-Alignment and Auxiliary Losses}
To better understand the contribution of each auxiliary loss, we conducted an ablation study by selectively disabling different components. In all cases, we used the same setup (small sub-dataset, 50 embedding tokens per item) and varied the presence of distillation loss and self-alignment loss.
% \begin{table}[h]
% \caption{Ablation study on auxiliary losses: Last 50 tokens}
% \label{tab:ablationloss}
% \begin{tabular}{l c}
% \hline
% \textbf{Setup} & \textbf{NDCG@10} \\
% \hline
% no auxiliary loss & 0.8642 \\
% +self-alignment loss & 0.8656 \\
% +distillation loss & 0.8733 \\
% +self-alignment +distillation losses & 0.8750 \\
% \hline
% \end{tabular}
% \end{table}
\begin{table}[h]
\caption{Ablation study on auxiliary losses: 50 embedding tokens per item, on small sub-dataset}
\label{tab:ablationloss}
\begin{tabular}{l c}
\hline
\textbf{Setup} & \textbf{NDCG@10} \\
\hline
no auxiliary loss & -- \\
+self-alignment loss & +0.0014 \\
+distillation loss & +0.0091 \\
+self-alignment +distillation losses & \textbf{+0.0108} \\
\hline
\end{tabular}
\end{table}
We observe that:
\begin{itemize}
\item Distillation loss contributes a significant improvement (+0.009 relative to KL-only).
\item Self-alignment alone provides only marginal gains over the KL-only baseline.
\item Combining distillation and self-alignment achieves the best performance (NDCG@10 +0.0108), confirming that the two auxiliary losses are complementary.
\end{itemize}
These results highlight the importance of including distillation in addition to KL and self-alignment losses to maximize ranking quality.
% \begin{table*}
% \caption{Ablations of Loss, Small Dataset}
% \label{tab:ablationloss}
% \begin{tabular}{cccc}
% \toprule
% KL Loss & Distill Loss & Alignment Loss & NDCG@10 \\
% \midrule
% Yes & No & No & 0.863788 \\
% No & Yes & No & 0.876095 \\
% No & No & Yes & 0.459739 \\
% Yes & Yes & No & 0.874731 \\
% Yes & No & Yes & 0.863091 \\
% No & Yes & Yes & 0.876887 \\
% Yes & Yes & Yes & 0.874159 \\
% \bottomrule
% \end{tabular}
% \end{table*}
\subsubsection{Curriculum Learning Strategy}
In addition to the auxiliary loss components, we investigated the effect of different phased training strategies that align with curriculum learning principles for Stage III in Section~\ref{sec:stageiii}. Curriculum learning suggests that training with carefully sequenced objectives or loss weightings can lead to better convergence and final model performance. In the context of text-embedding alignment problem, we explored the following curriculum strategies:
\begin{itemize}
\item \textbf{No Curriculum Learning (Baseline)}: focus more on the KL divergence loss but less on distillation loss and self-alignment regularization.
\item \textbf{Two-Phase Curriculum Strategy}: On top of the baseline, add an alignment phase at the beginning that has high self-alignment weights with low KL divergence weights.
\item \textbf{Three-Phase Curriculum Strategy}: Add an intermediate phase to two-phase curriculum, that has equal weighting across all three loss objectives (distillation, SFT, and self-alignment).
\end{itemize}
As seen in Table~\ref{tab:curriculum-learning}, curriculum learning improves ranking performance, indicating the alignment is necessary. The two-phase curriculum outperforms the three-phase approach, suggesting that a direct transition between alignment-focused and task-performance-focused training is more effective than including an intermediate balanced phase. This finding indicates that once the ranker-encoder alignment is sufficiently established, the model can more directly optimize for the downstream ranking task without the gradual intermediate phase.
\begin{table}[h]
\caption{Ablation study on curriculum learning strategies, on small sub-dataset}
\label{tab:curriculum-learning}
\begin{tabular}{l c}
\hline
\textbf{Curriculum Strategy} & \textbf{NDCG@10} \\
\hline
no curriculum learning (Task) & -- \\
two-phase (Alignment $\to$ Task) & \textbf{+0.0020} \\
three-phase (Alignment $\to$ Balanced $\to$ Task) & +0.0015 \\
\hline
\end{tabular}
\end{table}
\subsubsection{Inference Optimizations}
To evaluate the effectiveness of our inference optimizations for the MixLM framework, we conducted an ablation study measuring maximum throughput (QPS, items per second per GPU), within the same p99 latency budget of <=500 ms. The experiments were performed on an NVIDIA H100 80GB HBM3 GPU with CUDA 12.8. The workload consists of a shared prefix of 60 text tokens (search query and user history) and a batch size of 50 candidate items. In different experiment settings, candidate items have different corresponding representations: full item description texts (\enquote{raw-text}, 766 tokens/item, 38300 tokens/batch), summarized item description texts (\enquote{summarized-text}, 145 tokens/job, 7250 tokens/batch), and (\enquote{Mix-LM}, 1 embedding + 1 special token = 2 tokens/job, 100 tokens/batch). Each representation is evaluated without optimizations, with In-Batch Prefix Caching and with Multi-Job Scoring.
\textbf{Table~\ref{tab:ablation_inference}} presents the maximum items throughput per second per GPU, for p99 latency under 500ms. For each configuration, it demonstrates the impact of input compression and inference optimizations. MixLM with In-batch prefix caching achieves the highest throughput (22,000 QPS), contributing to the increase in 10.0x throughput. These results highlight the efficiency gains from compressing job descriptions and optimizing query processing, validated on an H100 GPU.
\begin{table}[t]
\caption{Ablation Study on Inference Optimizations (\textless{}500 ms latency)}
\label{tab:ablation_inference}
\centering
\begin{tabular}{l l r}
\toprule
Configuration & Prefix Optimization & QPS (Items/s/GPU) \\
\midrule
\multirow{3}{*}{Raw-text}
& None & 270 \\
& Multi-Job Scoring & 275 \\
& In-Batch Prefix Cache & 290 \\
\midrule
\multirow{3}{*}{Summarized, Pruned}
& None & 1\,650 \\
& Multi-Job Scoring & 2\,100 \\
& In-Batch Prefix Cache & 2\,200 \\
\midrule
\multirow{3}{*}{MixLM}
& None & 3\,000 \\
& Multi-Job Scoring & 20\,000 \\
& In-Batch Prefix Cache & \textbf{22\,000} \\
\bottomrule
\end{tabular}
\end{table}
% \begin{table}[t]
% \caption{Ablation Study on Inference Optimizations <500 ms latency}
% \label{tab:ablation_inference}
% \centering
% \begin{tabular}{lcc}
% \toprule
% Configuration & QPS(Items/Sec/GPU) \\
% \midrule
% Raw-text \& no optimizations & 150 \\
% Summarized,Pruned \& no optimizations & 2000 \\
% MixLM no optimizations & 3000 \\
% \midrule
% Raw-text + In-Batch Prefix Cache & 300 \\
% Summarized,Pruned + In-Batch Prefix Cache & 2400 \\
% MixLM + In-Batch Prefix Cache & \textbf{22,000} \\
% \midrule
% Raw-text + Multi-Job Scoring & 270 \\
% Summarized,Pruned + Multi-Job Scoring & 2190 \\
% MixLM + Multi-Job Scoring & 20,000 \\
% \bottomrule
% \end{tabular}
% \Description{A table showing maximum Jobs per second for p99 latency under 500ms for raw-text (300 tokens/job), summarized-text (180 tokens/job), and Mix-LM (2 tokens/job) configurations, with no optimizations, In-Batch Prefix Caching, or Multi-Job Scoring, on an H100 GPU with a query of 60 tokens and batch size of 50 jobs.}
% \end{table}
\section{Related Works}
\subsection{LLMs for Recommender Systems}
Recent years have seen a surge of interest in utilizing LLMs for search and recommender system applications. For example, several papers have reported that LLMs' understanding of natural language makes them powerful relevance judges. Moreover, LLMs are strong ranker which can extract users' preferences based on their past interactions~\citep{relevance4}. This has prompted a long line of research, investigating how LLMs can be integrated to existing recommender system, or even used as standalone recommenders~\citep{llmrs}. For example,~\citep{related1} show that LLMs can be successful in solving the cold-start problem in recommender systems, where not much historical data is available for the user. The cold-start problem was further studied by~\citep{related5} through the lens of instruction tuning. A similar approach was also used by~\citep{related4}.~\citep{related3} discuss using LLMs to predict user actions and preferences when the model is presented with the past data as a text prompt.
\subsection{LLMs for Semantic Search}
More related to our work, there is a long line of research on how LLMs can benefit semantic search applications. A common approach to enhance relevance ranking using LLMs is the bi-encoder approach~\citep{relevance-biencoder,relevance-biencoder2}, where the query and the item are passed through two separate encoders (or \emph{towers}) to obtain embedding representations, and the resulting representations are compared to judge the relevance. This approach allows for a computationally efficient implementation, and has been extensively utilized in practice~\citep{facebook-ebr}. However, this representation-based approach might not be able to capture nuanced semantic clues, due to limited interaction between queries and documents. In response, other work has studied late interaction representation-based models~\citep{relevance-colbert}.
An alternative approach to using LLMs for semantic search is the cross-encoder approach, where a single model scores the relevance of a query and an item (or multiple items under a list-wise ranking scheme) in a single prompt. Cross-encoder systems with LLM backbones have proved to be strong scorers for ranking systems~\citep{sachan2022improving,zhuang2024setwise,zhang2024two,rankvicuna,relevance3}. Unfortunately, serving LLM-based cross-encoders as online relevance scorers at an industrial scale can be prohibitively expensive in practice, as scoring numerous pairs of query-item using an LLM would require access to significant computational resources~\citep{related-pinterest,relevance-distill,relevance-distill2,rel-distill-reason}.
To overcome the computational requirements of LLM-based cross-encoders, other papers have proposed alternative approaches to use LLMs in relevance scoring applications. For example, \emph{LLM-as-judge} systems make use of LLMs as policy teachers by assigning graded relevance labels and rationales to query–item pairs. This allows for creating training datasets that can be used to distill the knowledge from LLMs to more efficient architectures~\citep{related-pinterest,relevance-tencent,relevance-distill3,relevance-distill4,rd-bench,rel-distill-reason}. Another approach is \emph{LLM-as-augmenter}, where LLMs generate paraphrased queries, summarize long documents, and synthesize hard negatives to enrich training data~\citep[and references therein]{behdin2025scaling,relevance-neg}.
\subsection{Industrial Application of LLMs for RecSys}
In addition to understanding the methodological aspects of MixLM, we also focus on a real-world semantic search that is deployed at an industrial scale. Related to this,~\citep{related-pinterest} discuss how an LLM can be used as a relevance judge. However, the LLM-based system is eventually distilled into a non-LLM architecture for online serving. Similar challenges have been reported by~\citep{related-ebay} and~\citep{related-walmart}, where either the model or some of the text features need to be altered for online serving. This further motivates our work to study methods that improve the efficiency of LLM-based recommender systems. In a recent work,~\citep{relevance-tencent} study the distillation an LLM-based system into a BERT-type~\citep{bert} model. Notably, for most these real-world applications discussed here, to meet efficiency goal, either a non-LLM model has to be deployed, or some features have to be removed. In contrast, in our real-world application, we deploy the LLM online, and all features are consumed by our encoder model to create item embedding representations.
\subsection{Mixed Input LLMs for RecSys}
Mixed-input LLMs for recommender systems has also been studied by some prior work.~\citep{related2} propose to substitute parts of the input text prompt with embeddings available from traditional (non-LLM) embedding systems. In another work,~\citep{chen2024hllm} propose to use embeddings to present user's past interactions, instead of a text prompt.
We note that however, our work differs from the aforementioned papers in a few key areas. First, we study a different prediction problem compared to~\citep{chen2024hllm,related2} for a relevance scoring system, where nuanced semantic understanding is of high importance. In addition, we present concrete lessons for training our MixLM system, including custom loss and regularization functions that differentiate our work. For example, we show that how an existing pure-text system can be distilled into the mixed-input framework using the custom loss functions we introduce.
Finally,~\citep{related2}'s analysis is limited to offline benchmarks, while we deploy our mixed input framework online for an A/B test. Moreover, even though~\citet{chen2024hllm} discuss online A/B testing, their hierarchical framework is used offline to generate user and item features, which are then fed to an online recommender system, and therefore, the LLM is not deployed for online inference. This is different from our work where the LLM is used for online ranking.
Hence, we present lessons from optimizing the serving infrastructure for such a mixed input system, as well as providing online quality metrics.
Concurrent to our work,~\citep{lin2025refrag} studies a mixed input system for Retrieval-Augmented Generation (RAG) applications, which is a different problem from ours. Additionally, we study a real-world deployment of our system, while the analysis of~\citep{lin2025refrag} is mostly contained to an offline study. In particular, we present several lessons and insights from serving infrastructure optimizations.
\subsection{Multimodal Large Language Model Training}
A common way to build multimodal large language models (MLLMs) is to couple a pretrained text-only LLM with modality-specific encoders via lightweight connectors. Early systems such as Flamingo~\citep{alayrac2022flamingo} and Kosmos-1~\citep{huang2023language} showed that treating visual features as additional “tokens’’ for a transformer backbone enables strong performance on captioning, VQA, and open-ended dialogue, typically by inserting cross-attention layers or training a unified multimodal transformer on large image–text corpora.
Subsequent work has emphasized reusing vision encoders and LLMs, sometime adding a small bridge between them. BLIP-2~\citep{li2023blip} uses a Querying Transformer to align vision features with the LLM embedding space, and LLaVA-style models~\citep{liu2023visual} learn a simple projection that maps CLIP features directly into the token embedding space of an LLM, followed by visual instruction tuning. In all these approaches, the key idea is to bypass explicit tokenization of non-text inputs and inject continuous modality-specific embeddings into the LLM, preserving its architecture and linguistic knowledge while adding multimodal capability. Our MixLM follows this paradigm, directly feeding learned embeddings into the language model to replace the tokenizer and embedding layer.
\section{Conclusion}
In this work, we introduced MixLM, a novel framework that bridges the gap between the semantic richness of large language model–based rankers and the stringent efficiency requirements of industrial search and recommender systems. By representing candidate items as compact embedding tokens and introducing the mix-interaction mechanism, MixLM effectively reduces the input context length while maintaining the expressiveness of full-text LLMs. Our co-training strategy for online rankers and near-line embedding flows further enables tight coupling between semantic understanding and serving efficiency.
Empirical results from deployment in a large-scale professional social network demonstrate that MixLM delivers substantial system gains—10.0x times improvement in throughput, under same latency budget. MixLM enabled LLM search system's deployment on all LinkedIn users' traffic, thereby producing a measurable 0.47\% increase in daily active users (DAU). These findings highlight the potential of mixed-input architectures to reshape the design space of LLM-based ranking systems, offering a practical path toward scalable, low-latency, and semantically powerful search in real-world environments.
Future work will explore adapting co-trained embeddings for modeling member history and exploring MixLM for retrieval tasks.
\begin{acks}
To all who collaborated with us to create Mix-LM.
\end{acks}
%%
%% The next two lines define the bibliography style to be used, and
%% the bibliography file.
\bibliographystyle{ACM-Reference-Format}
\bibliography{sample-base}
\end{document}
\endinput
%%
%% End of file `sample-sigconf-authordraft.tex'.