Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems
Extracted source text · arXiv 2506.06252
\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath, amssymb, amsfonts}
\usepackage{algorithm}
\usepackage{subfig}
\usepackage{makecell}
\usepackage{algorithmic}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}
\title{Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems}
% {\footnotesize \textsuperscript{*}Note: Sub-titles are not captured in Xplore and
% should not be used}
% \thanks{Identify applicable funding agency here. If none, delete this.}
% }
% \author{Anonymous Authors\\Paper under double-blind review}
\author{\IEEEauthorblockN{ Bo Ren, Yu Shi, Jinyu Li} \IEEEauthorblockA{\textit{Microsoft}}}
\maketitle
\begin{abstract}
End-to-End Automatic Speech Recognition (ASR) has advanced significantly
yet still struggles with rare and domain-specific entities. This paper
introduces a simple yet efficient prompt-based biasing technique for
contextualized ASR, enhancing recognition accuracy by leverage a unified
multi-task learning framework. The approach comprises two key components:
a prompt biasing model which is trained to determine when to focus on entities
in prompt, and a entity filtering mechanism which efficiently filters
out irrelevant entities. Our method significantly enhances ASR accuracy on
entities, achieving a relative 30.7\% and 18.0\% reduction in Entity
Word Error Rate compared to the baseline model with shallow fusion on in-house
domain dataset with small and large entity lists, respectively. The
primary advantage of this method lies in its efficiency and simplicity
without any structure change, making it lightweight and highly efficient.
\end{abstract}
\begin{IEEEkeywords}
speech recognition, contextual biasing, human-computer interaction
\end{IEEEkeywords}
\section{Introduction}
In recent years, End-to-End (E2E) automatic speech recognition (ASR) systems
have made significant progress thanks to advances in deep learning models
\cite{li2022recent, prabhavalkar2023end, graves2014towards,chan2015listen,chiu2018state,prabhavalkar2017comparison,watanabe2017hybrid,
sainath2020streaming, Li2020Developing, li2020comparison}. However, these
systems still face challenges in accurately recognizing rare words and
domain-specific terms. To enhance ASR performance, researchers have proposed
various methods
\cite{williams2018contextual,zhao2019shallow,pundak2018deep,jain2020contextual,he2019streaming}
to improve the performance on specific contexts. Contextual biasing techniques
leverage external information to boost recognition accuracy, particularly
when handling rare or specialized entities. Typically, the external
information is provided to the ASR system during decoding in the form of an
words/entities list, which is often referred as biasing list.
Extensive research has been conducted on contextual biasing techniques for E2E
ASR systems, which can be broadly classified into two major categories. Shallow
fusion methods \cite{williams2018contextual,zhao2019shallow} integrate external
language models (LMs) with ASR systems during decoding by weighting LM scores,
allowing the system to prioritize contextually relevant terms. Further advancements
in shallow fusion include sub-word regularization, pre-training, grapheme-to-grapheme
pronunciation learning, and deep integration with neural network language
models \cite{le2021deep,le2021contextualized}. However, a fundamental
limitation of shallow fusion is its reliance on post-contextual boosting, which
requires the model to generate the correct expected prefix terms in the candidate
pool without access to external contextual information.
To address these limitations, deep biasing methods directly incorporate
contextual information into E2E ASR models, enabling joint optimization. A
notable example is the Contextual Listen, Attend and Spell (CLAS) system,
which integrates ASR components with contextual embeddings
\cite{pundak2018deep}. The CLAS system has been further enhanced to include phonetic
information, leveraging pronunciation knowledge for improved recognition of
rare words \cite{bruguier2019phoebe,chen2019joint}. Other approaches have explored
intermediate biasing loss and attention mechanisms to improve contextual modeling
in CLAS systems \cite{shakeel2024contextualized}. However, these methods often
rely on additional encoders to embed contextual information, introducing
computational overhead and complexity. Moreover, attention mechanisms may
struggle to scale effectively with large biasing lists, a common scenario in
real-world applications \cite{pundak2018deep}.
Transformer-based architectures have recently become the backbone of state-of-the-art
E2E ASR systems \cite{waswani2017attention}. These models consistently
outperform traditional Recurrent Neural Network (RNN) approaches and are now
widely used in both research and industry
\cite{li2022recent,chang2021context,dong2018speech,gulati2020conformer,radford2023robust,prabhavalkar2023end,zhang2020transformer}.
However, integrating contextual biasing effectively into Transformer models
remains a challenging problem, with direct impact on user experience and commercial
deployment.
There is also increasing interest in utilizing large language models (LLMs) to
enhance contextual biasing in ASR \cite{lakomkin2024end,yang2024ctc}.
Although these methods have demonstrated promising results, they often require
significant computational resources and add architectural complexity. As LLM-based
solutions are still emerging and not yet widely adopted in production ASR
systems, our work focuses on practical Transformer-based E2E ASR models,
leaving LLM integration for future research.
In this work, we propose a straightforward and effective method for contextual
biasing in Transformer-based ASR systems using a multi-task learning framework.
Our approach, called \textbf{Prompt Biasing}, leverages the Transformer's cross-attention
mechanism to introduce contextual information as a prompt to the decoder. By
employing a unified multi-task learning setup with dedicated task tokens, the
model can efficiently handle both biasing and non-biasing scenarios without requiring
any architectural changes. Additionally, we introduce an efficient entity filtering
strategy that rely on the same model during decoding, enabling robust
performance even with large biasing lists. Experimental results demonstrate that
our lightweight Prompt Biasing model consistently outperforms the baseline with
shallow fusion on domain-specific datasets, achieving a relative reduction
in Entity Word Error Rate (EWER) of 30.7\% and 18.0\% for small and large
biasing lists, respectively.
\begin{figure*}
\centering
\subfloat[\footnotesize{Standard Transformer ASR training}]{\includegraphics[width=0.5\textwidth,trim=280 160 280 130, clip]{baseline.pdf}}
\subfloat[\footnotesize{Unified multi-task training for contextual biasing}]{\includegraphics[width=0.5\textwidth,trim=280 160 280 130, clip]{train.pdf}}
\quad \subfloat[\footnotesize{Multi-task token format for Prompt Biasing training}]{\includegraphics[width=\textwidth, trim=0 210 0 210, clip]{format.pdf}}
\caption{\textbf{Overview of the proposed Prompt Biasing method.} (a) Standard
Transformer ASR training, where the model predicts the next token from
previous tokens. (b) Unified multi-task training for contextual biasing,
where the model predicts the next token using both previous tokens and
contextual information provided as a prompt. (c) Multi-task token format
for Prompt Biasing training: biasing and non-biasing tasks are indicated
by special task tokens (\texttt{<hit>}/\texttt{<miss>}), and the biasing
list is included as a prompt starting with the \texttt{<sop>} token.}
\label{fig:multi-task}
\end{figure*}
\section{Related Work}
This section provides an overview of recent advancements in Transformer-based
ASR systems and contextual biasing techniques, with a particular focus on state-of-the-art
developments in Transformer architectures.
\subsection{Transformers based E2E ASR}
Recent advancements in Transformer-based architectures have significantly
advanced the field of end-to-end ASR. The Speech-Transformer \cite{dong2018speech}
first demonstrated the effectiveness of self-attention and cross-attention mechanisms
for modeling long-range dependencies in speech, resulting in improved
recognition accuracy and efficiency. Building on this, the Transformer-Transducer
framework incorporates a Transformer encoder within a transducer model, enabling
efficient streaming recognition with competitive performance \cite{zhang2020transformer,
chen2021developing}. The Conformer architecture further enhances Transformer
models by integrating convolutional layers, which capture local acoustic features
and improve robustness in challenging acoustic conditions \cite{gulati2020conformer}.
Collectively, these innovations have established Transformer-based models as
the state-of-the-art for ASR, offering superior accuracy, scalability, and
computational efficiency compared to traditional RNN-based systems.
More recently, Whisper \cite{radford2023robust} has emerged as a versatile end-to-end
Transformer-based speech processing system, trained on large-scale
multilingual data. Whisper supports not only speech recognition but also translation
and language identification within a unified framework, further
demonstrating the flexibility and effectiveness of Transformer architectures
in speech applications.
\subsection{Contextual biasing}
While shallow fusion techniques can be easily applied to Transformer-based ASR
systems, their performance improvements remain limited due their post-training
adjustment, and also be concerned to either under- or over- biasing issue \cite{williams2018contextual}. %TODO need a reference
Deep biasing techniques, such as the CLAS system \cite{pundak2018deep}, have
historically relied on RNN architectures.
Some studies have attempted to enhance contextual biasing capabilities in Transformer-based
models. For instance, certain works introduce a Tree-Constrained Pointer Generator
(TCPGen) module \cite{Sun_2021}, which dynamically adjusts transcription by
interpolating between the original model and TCPGen distributions
\cite{sun23e_interspeech}. Another approach, PromptASR, integrates an additional
pre-trained text encoder and injects contextual information by adding cross-attention
modules after the acoustic self-attention modules\cite{PromptASR}. Authors in
\cite{contextual_adapters_2022} proposed to introduce contextual adapters for
personalization in neural transducer based ASR models.These methods either introduce
auxiliary models or require significant architectural modifications to improve
contextual biasing in Transformer ASR models. In contrast, our approach is designed
to avoid any architectural modifications. We leverage the native multi-task
learning capabilities of the Transformer model, enabling effective contextual
biasing without introducing extra computational overhead or complexity.
\section{Proposed Approach}
In this section, we outline the core principles of integrating biasing tasks
into ASR systems via a multi-task learning framework, and we further detail the
entity filtering technique employed during the decoding process.
\subsection{Multi-task Learning for Contextual Biasing}
Figure \ref{fig:multi-task}a illustrates a standard Transformer-based ASR system
focused solely on speech recognition. In contrast, Whisper \cite{radford2023robust}
introduces a unified multi-task framework using conditional prefix tokens (e.g.,
transcription or translation) to differentiate tasks during training and decoding.
However, Whisper does not natively support contextual biasing functionality.
Inspired by Whisper \cite{radford2023robust}, we propose to formulate the
contextual biasing task as a multi-task learning problem, where the model is
trained to handle both biasing and non-biasing tasks within a unified framework.
The proposed multi-task learning framework is illustrated in Figure
\ref{fig:multi-task}b. In this framework, the model is trained to predict the
next token based on both the previous tokens and the contextual information provided
as a prompt. The contextual information is typically a list of entities or
phrases that the model should focus on during recognition. This prompt is integrated
into the Transformer decoder, allowing the model to condition its
predictions on the provided contextual information.
As illustrated in Figure~\ref{fig:multi-task}c, we adopt a unified training
format for both biasing and non-biasing tasks. Each training sample is structured
similarly, with special tokens indicating the presence or absence of
entities. The \texttt{<hit>} token marks entities that appear in the audio, while
the \texttt{<miss>} token marks those that do not. To clearly separate the prompt
(i.e. contextual information) from the transcription, we use a special \texttt{<sop>}
token at the start of the prompt and the standard \texttt{<sot>} token at the
beginning of the transcription. When no contextual information is provided (i.e.,
for regular recognition), the \texttt{<miss>} token is used, treating it as a
non-biasing task.
The core innovation of our approach lies in employing multi-task learning with
specialized task tokens to distinguish between biasing and non-biasing tasks below:
\begin{enumerate}
\item \textbf{Biasing Task:}
When certain entities in the prompt appears in the audio, the model
is given the \texttt{<hit>} token. This task token directs the model
to concentrate on the prompt content, thereby effectively leveraging
the external contextual information to improve the recognition accuracy
on these hit entities. Not all entities in the prompt are required
to be present in the audio, and the model is trained to learn where to
focus from data.
\item \textbf{Non-biasing Task:}
When none of entities in the prompt are present in the audio, the
model receives the \texttt{<miss>} token. This task token instructs
the model to disregard the prompt, enhancing its robustness by reducing
the influence of irrelevant information.
\end{enumerate}
The introduction of specialized task tokens enables our multi-task framework
to seamlessly handle both biasing and non-biasing tasks within a unified
architecture. During inference, these tokens allow the model to accurately
distinguish when contextual prompt information should be utilized and when
it should be disregarded, ensuring robust and context-appropriate recognition
performance.
\begin{table}
\centering
\caption{Examples from models trained \textbf{\textit{With/Without}} task
tokens. The biasing list includes coffee, milk, chocolate.}
\begin{tabular}{l|c|c}
\toprule \textbf{System} & \textbf{Task Tokens} & \textbf{Output} \\
\midrule Reference & N/A & \makecell{So I've added four toffee almond \textbf{milk} \\ hot cocoas what else?} \\
\midrule Recognition & Without & \makecell{coffee coffee \textbf{milk} \\ hot cocoa no no no no no} \\
\midrule Recognition & With & \makecell{So I've added 4 toffee almond \textbf{milk} \\ hot cocos what else?} \\
\bottomrule
\end{tabular}
\label{tab:example}
\end{table}
Table \ref{tab:example} presents outputs from models trained with and without
task tokens. The absence of task tokens leads the model to overfit to the
prompt, resulting in repeated or irrelevant words (e.g., \textbf{coffee})
and reduced transcription accuracy. In contrast, incorporating task tokens
enables the model to correctly identify and utilize relevant entities from
the prompt while preserving accurate recognition of the remaining audio
content. This example highlights that merely supplying prompt information is
insufficient; without a multi-task framework and explicit task tokens, the model
is prone to hallucinations and transcription errors. The proposed multi-task
approach with task tokens allows the model to effectively differentiate
between biasing and non-biasing scenarios, leveraging contextual information
only when appropriate. This design enhances transcription accuracy and
facilitates the integration of contextual biasing without introducing
architectural complexity.
\begin{algorithm}
[!b]
\caption{Entity Filtering}
\label{alg:filtering}
\begin{algorithmic}
[1] % The [1] adds line numbers
\REQUIRE Audio input, Entities in biasing list \ENSURE Filtered list
of entities \STATE \textbf{$hi$}: index of the \texttt{<hit>} token
\STATE $H \gets \text{Encoder}(\text{audio})$ \COMMENT{Compute encoder embedding}
\FOR{each entity $E$ in the biasing list} \STATE $L \gets \text{Decoder}
(E, H)$ \COMMENT{Compute logits for task tokens} \STATE
$P_{h}\gets \text{softmax}(L)[hi]$ \COMMENT{Probability on \texttt{<hit>} token}
\STATE $P_{\text{hit}}\gets \frac{1}{N}\sum_{i=1}^{N}P_{h}[i]$ \COMMENT{Average over sub-word units}
\ENDFOR \STATE Filter out entities with $P_{\text{hit}}< 0.5$
\RETURN Filtered list of entities
\end{algorithmic}
\end{algorithm}
\begin{figure*}
\centering
\includegraphics[width=\textwidth, trim=10 120 40 120, clip]{
inference.pdf
}
\caption{\textbf{Decoding process for the Prompt Biasing model}. The
model is trained to predict \texttt{<hit>} or \texttt{<miss>} for each
sub-word in the prompt, enabling effective filtering of irrelevant entities.
During decoding, entity filtering (see Algorithm~\ref{alg:filtering}) greatly
reduces the biasing list size. }
\label{fig:decoding}
\end{figure*}
\subsection{Entity Filtering for Biasing}
Previous studies on CLAS have shown that attention-based neural biasing
methods encounter difficulties when processing large biasing lists
\cite{pundak2018deep}. Our own experiments, as reflected in rows C1–C3 of
Table~\ref{tab:biasing}, corroborate this finding: large biasing lists
introduce substantial noise, which diminishes the model’s ability to accurately
focus on relevant entities. Our analysis reveals a clear correlation between
the precision of the biasing list and the effectiveness of contextual
biasing—the more targeted and relevant the biasing list, the greater the improvement
in biasing performance. This observation motivates the introduction of an
entity filtering mechanism to systematically eliminate irrelevant candidates
from large biasing lists.
In our approach, we utilize the multi-task trained model to efficiently filter
out irrelevant entities from large biasing lists. Unlike typical Transformer
training schemes, where losses on conditional input tokens are ignored and these
tokens serve solely as contextual cues, our method treats prompt tokens as
both conditional and predictive inputs. Specifically, during training, each sub-word
unit of the entities in the prompt is labeled with either a \texttt{<hit>}
or \texttt{<miss>} token, depending on its presence in the audio content (see
Figure~\ref{fig:multi-task}b). This enables the model to learn to identify
which entities in the prompt are actually present in the audio. Consequently,
the same model can be directly leveraged for entity filtering during
inference, eliminating the need for a separate filtering model.
As depicted in Figure~\ref{fig:decoding}, we adopt a progressive entity
filtering strategy during decoding, utilizing the multi-task model to identify
and exclude irrelevant entities from the biasing list. This process
substantially reduces the size of the candidate list; for example, initial
lists containing up to 2,000 entities can be efficiently narrowed to 10–20
highly relevant candidates. By eliminating distractors, the filtering
mechanism enhances the model’s ability to focus on pertinent entities, thereby
improving recognition accuracy. The detailed steps of this entity filtering
approach are provided in Algorithm~\ref{alg:filtering}. Although the method requires
evaluating each entity in the biasing list, it is highly amenable to
parallelization and can be executed with minimal latency on modern hardware
accelerators such as GPUs.
\begin{table*}
[t]
\centering
\caption{Performance on in-house domain dataset with various biasing
lists.}
\begin{tabular}{ll|ccc}
\toprule Row & Model & Biasing List & EWER(\%) & WER(\%) \\
\midrule A1 & Baseline & N/A & 7.76 & 4.75 \\
\midrule B1 & Baseline + Shallow Fusion & \textbf{Small} & 5.83 & 4.67 \\
B2 & Baseline + Shallow Fusion & \textbf{Large} & 6.04 & 4.74 \\
\midrule C1 & Prompt Biasing & \textbf{Exact} & 1.80 & 4.60 \\
C2 & Prompt Biasing & \textbf{Small} & 4.16 & 4.54 \\
C3 & Prompt Biasing & \textbf{Large} & 5.61 & 4.80 \\
\midrule D1 & Prompt Biasing + Entity Filtering & \textbf{Small} & 4.04 & 4.46 \\
D2 & Prompt Biasing + Entity Filtering & \textbf{Large} & 4.95 & 4.71 \\
\bottomrule
\end{tabular}
\label{tab:biasing}
\end{table*}
\section{Experiments}
Our model utilizes a Transformer encoder-decoder architecture, with the encoder
enhanced by Conformer layers \cite{gulati2020conformer} to improve speech recognition
accuracy. To increase computational efficiency, a convolutional down-sampling
module is applied prior to the Conformer layers, reducing the input frame
rate by a factor of 8. The encoder consists of 18 layers, while the decoder comprises
6 layers. The model is initially pre-trained on a comprehensive in-house
dataset encompassing diverse audio conditions and interaction scenarios, following
the standard speech recognition configuration shown in Figure~\ref{fig:multi-task}a.
This pre-trained model serves as the baseline for all subsequent Prompt Biasing
experiments, enabling a direct assessment of the proposed biasing approach.
\subsection{Training Details}
We fine-tune our model from a pre-trained Transformer backbone, eliminating the
need for training from scratch. The vocabulary is augmented with special
tokens (\texttt{<sop>}, \texttt{<hit>}, and \texttt{<miss>}) to support the proposed
multi-task learning framework. The training dataset comprises both biasing
and non-biasing samples, with biasing samples constituting 65\% of the total.
For each sample, entities in the biasing list are randomly selected from
either the reference transcription or a pool of negative phrases drawn from
diverse text corpora, with each entity limited to a maximum of five words.
Task tokens are assigned based on the presence of these entities in the
audio. To manage memory consumption, each training sample contains fewer
than 20 entities in the biasing list. The model is trained on approximately 5,400
hours of anonymized paired audio-text data using the AdamW optimizer \cite{loshchilov2017decoupled}
and a linear decay learning rate schedule, with a peak learning rate of
$2.24 \times 10^{-4}$ and a brief warmup phase.
\subsection{Evaluation Settings}
We conducted a comprehensive evaluation of the proposed Prompt Biasing
approach using an in-house domain-specific dataset covering 9 popular
domains and comprising approximately 570,000 words. Each utterance was annotated
by human experts to identify entities of interest. To measure recognition
performance on these entities, we utilized the Entity Word Error Rate (EWER),
which specifically quantifies errors within labeled entities.
To systematically investigate the effect of biasing list size, we constructed
three types of biasing lists for each utterance:
\begin{enumerate}
\item \textbf{Exact:} Consists solely of the entities present in the
reference transcription for the given utterance.
\item \textbf{Small:} Augments the \textbf{Exact} list with approximately
50 randomly selected distractor entities.
\item \textbf{Large:} Augments the \textbf{Exact} list with approximately
1800 randomly selected distractor entities.
\end{enumerate}
To assess the robustness of the model to noisy contextual information, we
conducted experiments on a large in-house dataset comprising 7.6 million
words. For each utterance, approximately 100 entities were randomly sampled
from external text sources and added as distractors to the prompt, simulating
the presence of irrelevant entities. The overall Word Error Rate (WER) was
then evaluated on this dataset to determine the model's resilience to such
noise.
For a fair comparison with conventional contextual biasing methods in E2E
ASR systems, we employ the shallow fusion technique with the baseline model.
During decoding, the provided biasing list is dynamically converted into a weighted
finite-state transducer (WFST) graph, following the approach described in
\cite{williams2018contextual}. This WFST is then integrated into the baseline
model's beam search process. The shallow fusion weight is set to 1.6 to
achieve a balance between contextual biasing effectiveness and overall
transcription accuracy.
\subsection{Experimental Results}
\subsubsection{Contextual Biasing Performance}
Rows C1–C3 in Table~\ref{tab:biasing} present the performance of the Prompt Biasing
model evaluated with three different types of biasing lists on our in-house domain
dataset. First of all, when an ideal ground truth list (i.e. \textbf{Exact}
list) is provided, the Prompt Biasing model achieves an impressive EWER of
1.8\%, significantly outperforming its baseline model (i.e. A1 in Table
\ref{tab:biasing}) by 76.8\% relatively. This demonstrates the effectiveness
of our approach in leveraging contextual information to enhance the entity recognition
accuracy. In addition to the strong EWER performance, the Prompt Biasing model
achieves a competitive WER of 4.60\%, slightly outperforming the baseline.
This indicated that the model effectively leverages contextual information
to improve entity recognition while maintaining overall transcription
quality. In addition to evaluating the Prompt Biasing model with the ideal \textbf{Exact}
biasing list, we also assess its performance using two more realistic, noisy
biasing lists: \textbf{Small} and \textbf{Large}. These lists better reflect
practical scenarios where the biasing list may not be entirely accurate or
precise. The \textbf{Small} list contains approximately 50 entities, while the
\textbf{Large} list includes around 1800 entities. As shown in Table~\ref{tab:biasing}
(rows C2 and C3), the Prompt Biasing model achieves strong EWERs of 4.16\% and
5.61\% with the \textbf{Small} and \textbf{Large} lists, respectively, along
with competitive WERs of 4.54\% and 4.80\%. These results demonstrate that the
model maintains effective contextual biasing and robust general transcription
accuracy even when provided with imperfect biasing lists.
Beyond the strong EWER performance of the Prompt Biasing model across various
biasing lists, we observe a clear trend of increasing EWER as the size of
the biasing list grows. Specifically, EWER rises from 1.80\% with the \textbf{Exact}
list to 4.16\% with the \textbf{Small} list, and further to 5.61\% with the
\textbf{Large} list. This pattern suggests that larger and noisier biasing
lists introduce additional challenges, reducing the model's ability to accurately
focus on relevant entities. These results underscore the importance of
providing precise and relevant contextual information to maximize the
effectiveness of contextual biasing.
\subsubsection{Impact of Entity Filtering}
Given the observed benefits of providing a precise biasing list, we further evaluate
the impact of our entity filtering strategy detailed in Algorithm \ref{alg:filtering}.
As shown in rows D1 and D2 of Table~\ref{tab:biasing}, applying entity filtering
leads to a notable improvement in EWER for both the \textbf{Small} and
\textbf{Large} noisy biasing lists, compared to the standard prompt biasing
model (rows C2 and C3). Specifically, EWER decreases from 4.16\% to 4.04\%
for the \textbf{Small} list and from 5.61\% to 4.95\% for the \textbf{Large}
list. These results demonstrate that entity filtering effectively enhances biasing
performance, particularly when handling large and noisy biasing lists. Additionally,
general transcription performance, as measured by WER, is also slightly
improved—from 4.54\% to 4.46\% for the \textbf{Small} list and from 4.80\%
to 4.71\% for the \textbf{Large} list—indicating that the entity filtering
strategy not only improves entity recognition but also helps maintain
overall transcription quality.
\subsubsection{Comparison to Shallow Fusion}
Shallow fusion is a widely adopted technique for enhancing contextual
biasing performance in ASR systems. As shown in Table~\ref{tab:biasing}, we
compare our proposed Prompt Biasing model with entity filtering (rows D1 and
D2) against the baseline model with shallow fusion (rows B1 and B2) across
different biasing list sizes. Our approach consistently outperforms shallow fusion,
achieving a relative reduction in EWER of 30.7\% for the \textbf{Small}
biasing list (D1 vs. B1) and 18.0\% for the \textbf{Large} biasing list (D2
vs. B2). These results highlight the effectiveness of Prompt Biasing with entity
filtering in leveraging contextual information to improve entity recognition
accuracy. Additionally, our method maintains competitive overall transcription
quality, with WERs of 4.46\% and 4.71\% for the \textbf{Small} and \textbf{Large}
lists, respectively, compared to 4.67\% and 4.74\% for shallow fusion. This demonstrates
that our approach not only enhances entity recognition but also preserves general
ASR performance, even with large and noisy biasing lists.
\subsubsection{Robustness Against Noise}
Table \ref{tab:robustness} presents the performance of the models on a 7.6M-word
in-house dataset under both standard (no biasing list) and noisy biasing list
conditions. The standard setting corresponds to conventional ASR without
contextual information. Compared to the baseline, the Prompt Biasing model
exhibits only a marginal increase in WER of 0.04\% in the absence of contextual
information, demonstrating minimal impact on general ASR performance. Furthermore,
when evaluated with a noisy biasing list, the Prompt Biasing model shows a negligible
WER degradation of 0.06\%, indicating strong robustness to irrelevant or inaccurate
contextual information.
\begin{table}[t]
\centering
\caption{Performance on in-house dataset for robustness benchmark.}
\begin{tabular}{lcc}
\hline
Model & Biasing List & WER(\%) \\
\hline
Baseline & N/A & 6.91 \\
\hline
Prompt Biasing & N/A & 6.95 \\
Prompt Biasing & Noisy & 6.97 \\
\hline
\end{tabular}
\label{tab:robustness}
\end{table}
\section{Conclusion and Future Work}
In this paper, we presented a lightweight and effective prompt-based contextual
biasing approach for Transformer-based ASR systems. By casting contextual
biasing as a multi-task learning problem, our method enables seamless integration
of contextual information without requiring any architectural modifications.
We also introduced an efficient entity filtering mechanism that
significantly enhances biasing performance, especially when dealing with large
and noisy biasing lists. Experimental results show that our approach
consistently outperforms shallow fusion techniques in both entity
recognition and overall transcription accuracy. In future work, we plan to
investigate the combination of Prompt Biasing with shallow fusion and
explore the integration of large audio language models to further advance
contextual ASR capabilities.
% \bibliographystyle{IEEEtran}
%\bibliographystyle{plain}
%\bibliography{mybib}
\input{main.bbl}
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{li2022recent}
J.~Li \emph{et~al.}, ``Recent advances in end-to-end automatic speech recognition,'' \emph{APSIPA Transactions on Signal and Information Processing}, vol.~11, no.~1, 2022.
\bibitem{prabhavalkar2023end}
R.~Prabhavalkar, T.~Hori, T.~N. Sainath, R.~Schl{\"u}ter, and S.~Watanabe, ``End-to-end speech recognition: A survey,'' \emph{IEEE/ACM Transactions on Audio, Speech, and Language Processing}, 2023.
\bibitem{graves2014towards}
A.~Graves and N.~Jaitly, ``Towards end-to-end speech recognition with recurrent neural networks,'' in \emph{International conference on machine learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2014, pp. 1764--1772.
\bibitem{chan2015listen}
W.~Chan, N.~Jaitly, Q.~Le, and O.~Vinyals, ``Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,'' in \emph{2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 2016, pp. 4960--4964.
\bibitem{chiu2018state}
C.-C. Chiu, T.~N. Sainath, Y.~Wu, R.~Prabhavalkar, P.~Nguyen, Z.~Chen, A.~Kannan, R.~J. Weiss, K.~Rao, E.~Gonina \emph{et~al.}, ``State-of-the-art speech recognition with sequence-to-sequence models,'' in \emph{2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2018, pp. 4774--4778.
\bibitem{prabhavalkar2017comparison}
R.~Prabhavalkar, K.~Rao, T.~N. Sainath, B.~Li, L.~Johnson, and N.~Jaitly, ``A comparison of sequence-to-sequence models for speech recognition,'' in \emph{Interspeech 2017}, 2017, pp. 939--943.
\bibitem{watanabe2017hybrid}
S.~Watanabe, T.~Hori, S.~Kim, J.~R. Hershey, and T.~Hayashi, ``Hybrid ctc/attention architecture for end-to-end speech recognition,'' \emph{IEEE Journal of Selected Topics in Signal Processing}, vol.~11, no.~8, pp. 1240--1253, 2017.
\bibitem{sainath2020streaming}
T.~N. Sainath, Y.~He, B.~Li, A.~Narayanan, R.~Pang, A.~Bruguier, S.-y. Chang, W.~Li, R.~Alvarez, Z.~Chen \emph{et~al.}, ``A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,'' in \emph{Proc. ICASSP}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 6059--6063.
\bibitem{Li2020Developing}
J.~Li, R.~Zhao, Z.~Meng, Y.~Liu, W.~Wei, S.~Parthasarathy, V.~Mazalov, Z.~Wang, L.~He, S.~Zhao, and Y.~Gong, ``Developing rnn-t models surpassing high-performance hybrid models with customization capability,'' in \emph{Interspeech 2020}, 2020, pp. 3590--3594.
\bibitem{li2020comparison}
J.~Li, Y.~Wu, Y.~Gaur, C.~Wang, R.~Zhao, and S.~Liu, ``On the comparison of popular end-to-end models for large scale speech recognition,'' in \emph{Proc. Interspeech}, 2020, pp. 1--5.
\bibitem{williams2018contextual}
I.~Williams, A.~Kannan, P.~S. Aleksic, D.~Rybach, and T.~N. Sainath, ``Contextual speech recognition in end-to-end neural network systems using beam search.'' in \emph{Interspeech}, 2018, pp. 2227--2231.
\bibitem{zhao2019shallow}
D.~Zhao, T.~N. Sainath, D.~Rybach, P.~Rondon, D.~Bhatia, B.~Li, and R.~Pang, ``Shallow-fusion end-to-end contextual biasing.'' in \emph{Interspeech}, 2019, pp. 1418--1422.
\bibitem{pundak2018deep}
G.~Pundak, T.~N. Sainath, R.~Prabhavalkar, A.~Kannan, and D.~Zhao, ``Deep context: end-to-end contextual speech recognition,'' in \emph{2018 IEEE spoken language technology workshop (SLT)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2018, pp. 418--425.
\bibitem{jain2020contextual}
M.~Jain, G.~Keren, J.~Mahadeokar, G.~Zweig, F.~Metze, and Y.~Saraf, ``Contextual rnn-t for open domain asr,'' in \emph{Interspeech 2020}, 2020, pp. 11--15.
\bibitem{he2019streaming}
Y.~He, T.~N. Sainath, R.~Prabhavalkar, I.~McGraw, R.~Alvarez, D.~Zhao, D.~Rybach, A.~Kannan, Y.~Wu, R.~Pang \emph{et~al.}, ``Streaming end-to-end speech recognition for mobile devices,'' in \emph{ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019, pp. 6381--6385.
\bibitem{le2021deep}
D.~Le, G.~Keren, J.~Chan, J.~Mahadeokar, C.~Fuegen, and M.~L. Seltzer, ``Deep shallow fusion for rnn-t personalization,'' in \emph{2021 IEEE Spoken Language Technology Workshop (SLT)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 251--257.
\bibitem{le2021contextualized}
D.~Le, M.~Jain, G.~Keren, S.~Kim, Y.~Shi, J.~Mahadeokar, J.~Chan, Y.~Shangguan, C.~Fuegen, O.~Kalinli, Y.~Saraf, and M.~L. Seltzer, ``Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,'' in \emph{Interspeech 2021}, 2021, pp. 1772--1776.
\bibitem{bruguier2019phoebe}
A.~Bruguier, R.~Prabhavalkar, G.~Pundak, and T.~N. Sainath, ``Phoebe: Pronunciation-aware contextualization for end-to-end speech recognition,'' in \emph{ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019, pp. 6171--6175.
\bibitem{chen2019joint}
Z.~Chen, M.~Jain, Y.~Wang, M.~L. Seltzer, and C.~Fuegen, ``Joint grapheme and phoneme embeddings for contextual end-to-end asr,'' in \emph{Interspeech 2019}, 2019, pp. 3490--3494.
\bibitem{shakeel2024contextualized}
M.~Shakeel, Y.~Sudo, Y.~Peng, and S.~Watanabe, ``Contextualized end-to-end automatic speech recognition with intermediate biasing loss,'' in \emph{Interspeech 2024}, 2024, pp. 3909--3913.
\bibitem{waswani2017attention}
A.~Waswani, N.~Shazeer, N.~Parmar, J.~Uszkoreit, L.~Jones, A.~Gomez, L.~Kaiser, and I.~Polosukhin, ``Attention is all you need,'' in \emph{NIPS}, 2017.
\bibitem{chang2021context}
F.-J. Chang, J.~Liu, M.~Radfar, A.~Mouchtaris, M.~Omologo, A.~Rastrow, and S.~Kunzmann, ``Context-aware transformer transducer for speech recognition,'' in \emph{2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 503--510.
\bibitem{dong2018speech}
L.~Dong, S.~Xu, and B.~Xu, ``Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,'' in \emph{2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2018, pp. 5884--5888.
\bibitem{gulati2020conformer}
A.~Gulati, J.~Qin, C.-C. Chiu, N.~Parmar, Y.~Zhang, J.~Yu, W.~Han, S.~Wang, Z.~Zhang, Y.~Wu, and R.~Pang, ``Conformer: Convolution-augmented transformer for speech recognition,'' in \emph{Interspeech 2020}, 2020, pp. 5036--5040.
\bibitem{radford2023robust}
A.~Radford, J.~W. Kim, T.~Xu, G.~Brockman, C.~McLeavey, and I.~Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in \emph{Proceedings of the 40th International Conference on Machine Learning}, ser. ICML'23.\hskip 1em plus 0.5em minus 0.4em\relax JMLR.org, 2023.
\bibitem{zhang2020transformer}
Q.~Zhang, H.~Lu, H.~Sak, A.~Tripathi, E.~McDermott, S.~Koo, and S.~Kumar, ``Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,'' in \emph{2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 2020, pp. 7829--7833.
\bibitem{lakomkin2024end}
E.~Lakomkin, C.~Wu, Y.~Fathullah, O.~Kalinli, M.~L. Seltzer, and C.~Fuegen, ``End-to-end speech recognition contextualization with large language models,'' in \emph{ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2024, pp. 12\,406--12\,410.
\bibitem{yang2024ctc}
G.~Yang, Z.~Ma, Z.~Gao, S.~Zhang, and X.~Chen, ``Ctc-assisted llm-based contextual asr,'' in \emph{2024 IEEE Spoken Language Technology Workshop (SLT)}, 2024, pp. 126--131.
\bibitem{chen2021developing}
X.~Chen, Y.~Wu, Z.~Wang, S.~Liu, and J.~Li, ``Developing real-time streaming transformer transducer for speech recognition on large-scale dataset,'' in \emph{Proc. ICASSP}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 5904--5908.
\bibitem{Sun_2021}
G.~Sun, C.~Zhang, and P.~C. Woodland, ``Tree-constrained pointer generator for end-to-end contextual speech recognition,'' in \emph{2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)}, 2021, pp. 780--787.
\bibitem{sun23e_interspeech}
G.~Sun, X.~Zheng, C.~Zhang, and P.~C. Woodland, ``Can contextual biasing remain effective with whisper and gpt-2?'' in \emph{Interspeech 2023}, 2023, pp. 1289--1293.
\bibitem{PromptASR}
X.~Yang, W.~Kang, Z.~Yao, Y.~Yang, L.~Guo, F.~Kuang, L.~Lin, and D.~Povey, ``Promptasr for contextualized asr with controllable style,'' in \emph{ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 2024, pp. 10\,536--10\,540.
\bibitem{contextual_adapters_2022}
K.~M. Sathyendra, T.~Muniyappa, F.-J. Chang, J.~Liu, J.~Su, G.~P. Strimel, A.~Mouchtaris, and S.~Kunzmann, ``Contextual adapters for personalized speech recognition in neural transducers,'' in \emph{ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 2022, pp. 8537--8541.
\bibitem{loshchilov2017decoupled}
I.~Loshchilov and F.~Hutter, ``Decoupled weight decay regularization,'' in \emph{International Conference on Learning Representations}, 2017.
\end{thebibliography}