Activation Steering for Accent Adaptation in Large Audio Language Models
Extracted source text · arXiv 2603.05813
% *==================================================================================*
% * Review vs. Camera-Ready settings *
% *==================================================================================*
%
% REVIEW: Use the following command for submitting the paper (double-blind,
% for review):
% \documentclass{Interspeech}
%
% CAMERA-READY: Use the following command for the camera-ready version, one
% affiliation per line:
\documentclass[cameraready]{Interspeech}
% *==================================================================================*
% **************************************
% * *
% * STOP ! DO NOT DELETE ! *
% * READ THIS FIRST *
% * *
% * This template also includes *
% * important INSTRUCTIONS that you *
% * must follow when preparing your *
% * paper. Read it BEFORE replacing *
% * the content with your own work. *
% **************************************
%==================================================================================
% Title
% Must exactly match the title entered into the paper submission system
\title{Activation Steering for Accent Adaptation in Speech Foundation Models
}
%==================================================================================
% Authors
\author[affiliation={1},equalcontribution]{Jinuo}{Sun}
\author[affiliation={1},equalcontribution]{Yang}{Xiao}
\author[affiliation={1}]{Sung Kyun}{Chung}
\author[affiliation={1}]{Qiuchi}{Hu}
\author[affiliation={2}]{Gongping}{Huang}
\author[affiliation={1}]{Eun-Jung}{Holden}
\author[affiliation={1}]{Ting}{Dang}
%==================================================================================
% Affiliations
\address{$^1$
School of Computer Science, The University of Melbourne, Australia \\
$^2$
Wuhan University, China
}
%==================================================================================
% Emails
\email{jinuosun@gmail.com, yxiao9550@student.unimelb.edu.au, ting.dang@unimelb.edu.au}
%==================================================================================
% Keywords
\keywords{speech recognition, human-computer interaction, computational paralinguistics}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\newcommand{\red}[1]{\textcolor{red}{#1}}
\usepackage{comment}
%==================================================================================
% Content
\usepackage{caption} % Recommended for customizing captions
\usepackage{subcaption} % Required for the subfigure environment
\usepackage{cite}
\captionsetup{font=small}
\captionsetup[subfigure]{font=small,skip=1pt}
\newcommand{\fixedsubcap}[1]{%
\caption{\parbox[t][2.2\baselineskip][t]{\linewidth}{\centering #1}}%
}
\usepackage{soul, xcolor, graphicx}
\newcommand{\td}[1]{\sethlcolor{pink}\hl{[Ting: #1]}}
\newcommand{\rev}[1]{\textcolor{red}{#1}}
\begin{document}
\maketitle
% the abstract here must exactly match the abstract entered into the paper submission system
\begin{abstract}
% 1000 characters. ASCII characters only. No citations.
Accent variability remains a major errors in automatic speech recognition, yet most adaptation methods rely on parameter fine-tuning without understanding where accent information is encoded. We treat accent variation as an interpretable subspace in hidden representations and investigate whether it can be identified and controlled directly in activation space. We extract layer-wise encoder activations and estimate mean-shift directions capturing accent-induced representation shifts. By injecting these directions into individual layers and measuring how they align accented and standard embeddings, we derive a layer-wise accent sensitivity profile, revealing that accent information concentrates in a narrow band of middle encoder layers. Leveraging this structure, we further introduce parameter-free accent steering that modifies representations during inference without updating model weights. Experiments across eight accents show consistent word error rate reductions.
\end{abstract}
\input{01-intro}
% \section{Introduction}
% \td{overall it is good; the content is already there; some feedback for your reference.}
% % 第一段应该就是，SFM，LSLM他们很好用，把asr提的很高，but还是面临方言的问题
% % Automatic Speech Recognition (ASR) aims to transcribe spoken language into text, serving as the critical interface between human communication and machine understanding. In recent years, the paradigm of ASR has fundamentally shifted from traditional hybrid systems to large-scale speech foundation models. Architectures trained on hundreds of thousands of hours of weakly supervised or self-supervised data have unlocked unprecedented transcription accuracy and zero-shot generalization across diverse acoustic environments. However, despite these sweeping advancements, performance suffers a precipitous drop when these models encounter accented speech . Because foundation models inevitably over-index on the standard, high-resource pronunciation distributions prevalent in their training corpora, accented speech manifests as an out-of-distribution anomaly. This mismatch introduces systematic acoustic and phonetic deviations that disrupt the model's internal representations, leading to a noticeable degradation in Word Error Rate (WER) and creating a persistent barrier to equitable and robust speech recognition.
% Automatic Speech Recognition (ASR)~\cite{malik2021asr_survey} aims to transcribe spoken language into text. It serves as a vital bridge between human communication and machine understanding. Recently, ASR has shifted from traditional hybrid systems to large speech foundation models~\cite{yang2024speech_foundation_models}. These models are trained on massive amounts of data, achieving excellent accuracy and generalization across various environments. However, their performance drops significantly when processing accented speech~\cite{mas-lora, ASR_in_diverse_accents}. This issue occurs because foundation models focus too much on standard, high-resource pronunciation found in their training data. As a result, accented speech becomes an out-of-distribution anomaly. This mismatch causes acoustic and phonetic differences that disturb the model's internal representations. Consequently, the Word Error Rate (WER) increases, which creates a barrier to fair and robust speech recognition.\td{here you already introduced the foundation model; then next paragraph you mention the adaptation method before llms; kind of mixed; also it would be good start with the key questions of accent directly; no need to discuss much on ASR as this community is familiar with ASR. }
% % The challenges posed by accented speech are multifaceted and typically fall into two distinct categories: native regional accents and non-native (L2) accents. Native accents involve complex, deeply rooted dialectal variations, such as vowel shifts, unique consonant assimilations, and distinct prosodic rhythms that deviate significantly from broadcast standard speech. Conversely, non-native accented speech is heavily influenced by the phonological and articulatory constraints of a speaker’s mother tongue (L1) transferring to their second language. This L1 interference results in predictable but highly disruptive phonetic substitutions, deletions, and non-standard stress patterns. Both forms of variation warp the expected acoustic feature alignments, forcing the ASR system to navigate a highly variable and sparse acoustic space, which drastically increases the likelihood of token misprediction.
% Accented speech presents challenges that generally fall into two categories: native regional accents and non-native accents. Native accents feature deeply rooted regional differences. These include vowel shifts and unique rhythms that differ greatly from standard speech~\cite{vctk}. On the other hand, non-native accents are heavily influenced by the speaker's first language~\cite{l2}. This influence causes predictable but disruptive phonetic changes, such as sound substitutions and unusual stress patterns. Both types of variation change the expected acoustic features. Therefore, the ASR system must process a highly variable acoustic space, which significantly increases the chance of prediction errors.\td{good to give the details; In our cases i am not sure if we discuss these results in specific details or if our methods specifically target the different types of accents such as proposing different analysis or method for different methods; if not no need to elaborate it too much. }
% % Prior to the proliferation of large speech language models, addressing accent variability relied primarily on rigid acoustic model adaptation and extensive data augmentation. Initial approaches attempted to bridge the phonetic gap by incorporating dialect-specific lexicons or multi-pronunciation dictionaries. Subsequently, statistical adaptation techniques, such as Maximum a Posteriori (MAP) adaptation and Maximum Likelihood Linear Regression (MLLR), alongside domain adversarial training, were employed to mathematically align accented acoustic features with standard distributions. While these methods provided modest improvements, they were fundamentally constrained. They required substantial amounts of annotated target-accent data, necessitated computationally expensive retraining pipelines, and struggled with scalability. Furthermore, these frameworks were generally designed to adapt to a single accent at a time, failing to generalize gracefully across a multitude of diverse accent profiles.
% Before the rise of large speech models, solving the accent problem relied on fixed acoustic model adaptation and extensive data augmentation. Early methods used specific dictionaries to cover different pronunciations~\cite{huang2000pda}. Later, statistical techniques like Maximum a Posteriori (MAP) adaptation and Maximum Likelihood Linear Regression (MLLR) were used. These methods tried to align accented acoustic features with standard distributions mathematically~\cite{nallasamy2012accent}. \td{these provides some details; good; but not in a systematic way; so what are the methods generally applied for speech; we can think of fine-tuning, or data augmentation right, here MAP or MLLR may just be one adaptation/or fine tuning methods. }Although these approaches offered some improvements, they had major limits. They required large amounts of labeled accent data and expensive retraining processes. Furthermore, these methods usually adapted to only one accent at a time. Thus, they failed to generalize well across many different accents~\cite{poor_generalization, unseen_gap_1, unseen_gap_2}.
% % With the advent of massive speech foundation models, adaptation strategies have increasingly pivoted toward Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA) and bottleneck adapters. These approaches offer a compelling advantage: they introduce only a minimal number of trainable parameters to adapt a frozen base model to specific accents, mitigating catastrophic forgetting and significantly reducing computational overhead compared to full fine-tuning. However, PEFT strategies present notable gaps. Existing methods typically optimize parameters globally or semi-globally across the transformer blocks without explicitly localizing where accent-related acoustic knowledge resides within the network. This blind adaptation leads to inefficient parameter updates and risks entangling phonetic adjustments with deeper, language-modeling semantic representations, potentially degrading the model's core linguistic reasoning capabilities.
% With the development of large speech foundation models, researchers have turned to Parameter-Efficient Fine-Tuning (PEFT) methods~\cite{houlsby2019adapters}. Common examples include Low-Rank Adaptation (LoRA) and bottleneck adapters~\cite{mas-lora, hu2021lora, learn_and_dont_forget, parameter-efficient_model_reprogramming}. These methods offer a clear advantage by adding only a few trainable parameters to adapt a frozen base model. This approach reduces computational costs and prevents catastrophic forgetting~\cite{kirkpatrick2017ewc} compared to full fine-tuning. However, PEFT strategies still have notable drawbacks. Current methods usually optimize parameters globally across the network~\cite{song2024lora_whisper, prasad2024lowrank_asr}. They do not locate exactly where accent-related acoustic knowledge is stored. This undirected adaptation causes inefficient updates. It also risks mixing phonetic changes with deeper semantic representations, which could harm the model's basic language abilities.
% % To circumvent the limitations of weight-updating methods, activation steering—or vector steering—has emerged as a highly effective inference-time intervention . Originating in text-based LLMs to control generation style or mitigate hallucinations, a steering vector represents a targeted representational shift, calculated as the mean difference in hidden state activations between two contrasting data distributions. By injecting these residual vectors into the forward pass, we can dynamically shift the model's internal representations toward a desired state without altering a single weight. For accented speech, steering vectors offer a lightweight, zero-training mechanism to pull out-of-distribution acoustic representations back onto the standard pronunciation manifold. The primary challenge of this approach, however, lies in precise localization. Applying the shift at the wrong depth can either fail to capture early low-level acoustic variations or inadvertently distort high-level semantic tokens later in the network.
% To avoid the limitations of weight‑updating methods, activation steering has emerged as an effective inference‑time alternative, also known as vector steering. Originally developed for text‑based large language models, a steering vector induces a targeted shift in internal representations by taking the average difference in hidden states between two data distributions or behaviors \cite{Category-wise_safety_steering, Affine_concept_editing, dynamic_steering}. By adding such vectors at selected layers during the forward pass, one can change the model’s internal states without modifying any weights. For accented speech, steering vectors provide a lightweight method with no extra training. They pull unusual acoustic representations back to the standard pronunciation space. However, the main challenge of this approach is finding the exact location to apply the shift~\cite{pasad2021layerwise, pasad2023comparative}. Applying the vector at the wrong network layer can cause problems. It might miss early acoustic variations or damage high-level semantic meanings in deeper layers.\td{good discussion, also inlcuded in the revised version. }
% % In this work, we propose a novel framework that leverages activation steering for inference-time accent adaptation in speech foundation models. To the best of our knowledge, we are the first to systematically analyze and adopt vector steering for both native and non-native accented speech. Our contributions are three-fold: First, we utilize activation patching with paired standard and accented utterances to precisely localize accent-sensitive layers within the speech model, mapping where acoustic divergence occurs. Second, we extract mean-shift vectors between accents and introduce an inference-time steering mechanism to dynamically correct acoustic deviations. Finally, through rigorous cross-validated metrics and leave-out experiments, we demonstrate that mean-shift steering yields significant WER improvements across diverse accents while strictly preserving the model's underlying semantic integrity.
% %
% In this work, we propose a new framework that uses activation steering for inference-time accent adaptation in speech foundation models. To the best of our knowledge, we are the first to analyze and apply vector steering systematically for both native and non-native accented speech. Our contributions are divided into three main parts. First, we use activation patching with paired standard and accented audio to locate accent-sensitive layers accurately. This step maps exactly where acoustic differences occur. Second, we extract mean-shift vectors between different accents. We then introduce a steering mechanism during inference to correct acoustic errors dynamically. Finally, we evaluate our method through rigorous experiments. We demonstrate that mean-shift steering significantly improves WER across various accents while maintaining the model's capacity for non-accented speech. \td{Good; it would be better to frame this question at a higher level: if the accent variations correspond to structured and controllable subspaces? This is what we want to understand and analyze. So here is the scientific question we want to answer and the assumption that we rely on for steering vector; the details presented are the methods to answer this question.}
% Templates are provided on the conference website for Microsoft Word\textregistered, and \LaTeX. We strongly recommend \LaTeX\xspace
% which can be used conveniently in a web browser on \url{https://overleaf.com} where this template is available in the Template Gallery.
% \input{02-locate}
\input{02-merge-analysis-and-steering}
% \input{03-steering}
\input{04-experiment}
\input{05-results}
% \subsection{General guidelines}
% Authors are encouraged to describe how their work relates to prior work by themselves and by others, and to make clear statements about the novelty of their work. This may require careful wording to avoid unmasking the identity of the authors in the version submitted for review (guidance in Section \ref{section:doubleblind}). All submissions must be compliant with the \textbf{ISCA Code of Ethics for Authors}, the \textbf{Pre-prints Policy}, and the \textbf{Conference Policy}. These can be found on the conference website under the ``For authors'' section.
% \subsubsection{Originality}
% All papers submitted to Interspeech 2026 must be original contributions that are not currently submitted to any other conference, workshop, or journal, nor will be submitted to any other conference, workshop, or journal during the review process of Interspeech 2026. Cross-checks with submissions from other conferences will be carried out to enforce this rule.
% \subsubsection{Generative AI tools}
% All (co-)authors must be responsible and accountable for the work and content of the paper, and they must consent to its submission. Generative AI tools cannot be a co-author of the paper. They can be used for editing and polishing manuscripts, but should not be used for producing a significant part of the manuscript.
% \blue{The use of generative AI tools should be acknowledged in a subsection titled Generative AI Use Disclosure between the Acknowledgments of the References.}
% \subsubsection{\blue{List of authors}}
% {\color{blue}
% When preparing the author list, please observe the following guidelines:
% \begin{itemize}
% \item Clearly list all authors who have made a significant contribution to the work.
% \item The maximum number of authors in the author list is 20. If there are more than 20 contributing authors, the additional authors should be acknowledged in a footnote or in the acknowledgments section.
% \item The use of ORCID identifiers for authors is optional but strongly encouraged. It will become mandatory in the future.
% \item Submissions must be anonymous: author names and affiliations must not appear in the manuscript during review. However, the full list of authors (and affiliations) must be entered in the submission platform when submitting the paper for review.
% \item During the development phase of the paper, authors may keep the full (non-anonymized) list of authors and affiliations in their working copy to verify layout and space constraints. Anonymization should be applied at the final stage before generating the version for submission.
% \end{itemize}
% }
% \subsubsection{Conference theme}
% \blue{The theme for Interspeech 2026, "Speaking Together," reflects our dedication to the celebration of all voices. This conference aims to showcase groundbreaking work in under-resourced languages, atypical speech and initiatives to improve access to speech technology for all speakers.}
% \subsubsection{Participation in reviewing}
% Interspeech uses peer review to evaluate submitted manuscripts. At least one co-author of each paper should be volunteering to review for the Interspeech conference. If you are eligible, but not yet a reviewer for ISCA conferences, you should sign up at
% \begin{center}
% \url{https://isca-speech.org/Reviewing}
% \end{center}
% using the same email address as in your CMT account.
% The eligibility criteria can be located through the above link. Papers for which no author qualifies as ISCA reviewer do not have to follow this guideline. During submission, authors should indicate the email of any co-authors who should be added to the reviewer database.
% \subsubsection{Accessibility recommendations}
% \blue{Authors are encouraged to have their papers fulfill specific requirements relating to \textbf{figures} (see Section \ref{sec:figures}) and \textbf{tables} (see Section \ref{sec:tables}).}
% \subsubsection{Scientific reporting check list}
% The following checklist will be part of the submission form and includes a list of guidelines we expect Interspeech 2026 papers to follow. A compact version of the checklist is available under ``For authors'' section. Not every point in the check list will be applicable to every paper. Ideally, all guidelines that do apply to a certain paper should be satisfied. Nevertheless, not satisfying a guideline that applies to your paper is not necessarily grounds for rejection.
% \begin{enumerate}
% \item Claims and limitations - for all papers
% \begin{itemize}
% \item The paper clearly states what claims are being investigated, and why.
% \item The novelties are clearly explained.
% \item The main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope.
% \item The limitations of your work are described.
% \item The relevant assumptions made in your work are stated in the paper.
% \end{itemize}
% \item If models are used, the paper includes information about the following:
% \begin{itemize}
% \item Justification for each selected model, algorithm, or architecture.
% \item Sufficient details of the model (architecture), including reference(s) to prior literature.
% \end{itemize}
% \item If data sets are used, the paper includes information about the following:
% \begin{itemize}
% \item Relevant details such as languages, audio duration distribution, number of examples, and label distributions, or a reference where that information can be found.
% \item Details of train/validation (development)/test splits. Ideally, the lists should be released with the supplementary material if not already publicly available.
% \item Explanation of all pre-processing steps, including any criteria used to exclude samples, if applicable.
% \item Reference(s) to all data set(s) drawn from the existing literature.
% \item For newly collected data, a complete description of the data collection process, such as subjects, instructions to annotators and methods for quality control and a statement on whether ethics approval was necessary.
% \end{itemize}
% \item If using non-public data sets or pre-trained models:
% \begin{itemize}
% \item The paper includes a discussion on the reason/s for using non-public data sets.
% \item Full details of the dataset are included in the paper to enable comparison to similar data sets and tasks.
% \item For papers that use publicly available, pre-trained (e.g. foundational or self-supervised) models, link to the model is provided. For papers that use non-public pre-trained models, description of the data (including quantity) used for training that model is provided.
% \end{itemize}
% \item If reporting experimental results, the paper includes:
% \begin{itemize}
% \item An explanation of evaluation metrics used.
% \item An explanation of how models are initialized, if applicable.
% \item Some measure of statistical significance of the reported gains or confidence intervals.
% \item A description of the computing infrastructure used and the average runtime for each model or algorithm (e.g. training, inference etc).
% \item The number of parameters in each model.
% \end{itemize}
% \item If hyperparameter search (including choice of architecture or features and any other development decision) was done, the paper includes:
% \begin{itemize}
% \item Final results on a held-out evaluation set not used for hyperparameter tuning.
% \item Hyperparameter configurations for best-performing models.
% \item The method for choosing hyperparameter values to explore, the range of the hyperparameter values, and the criterion used to select among them.
% \end{itemize}
% \item If source code is used:
% \begin{itemize}
% \item The code is or will be made publicly available and/or sufficient details are included in the paper for the work to be reproduced.
% \item For publicly available software, the corresponding version numbers and links and/or references to the software.
% \end{itemize}
% \end{enumerate}
% For any paper that uses machine learning, we encourage authors to provide statistical significance tests and follow well-established processes for evaluating any statistical or machine learning model. \blue{McNemar’s test \cite{mcnemar1947} is one of the most popular tests used in the literature to report statistical significance (e.g., \cite{fiscus1997post, chollet1995evaluation, williams2008exploiting, busch2025multilingual, lin2025voice, lee2025performance}). Other examples include ANOVA \cite{allbert2025evaluating}, adding error bars~\cite{miller2024addingerrorbarsevals}, and bootstrapping approaches~\cite{ferrer2024goodpracticesevaluationmachine, url2025}.}
% \subsection{Double-blind review}
% \label{section:doubleblind}
% Interspeech~2026 uses double-blind review to support the integrity of the review process.
% \subsubsection{Version submitted for review}
% The manuscript submitted for review must not include any information that might reveal the authors' identities or affiliations. This also applies to the metadata in the submitted PDF file (guidance in Section \ref{section:pdf_sanitise}), uploaded multimedia and online material (guidance in Section \ref{section:multimedia}).
% Take particular care to cite your own work in a way that does not reveal that you are also the author of that work. For example, do not use constructions like ``In previous work [23], we showed that \ldots''. Instead use something like ``Jones et al. [23] showed that \ldots''.
% Acknowledgements should not be included in the version submitted for review.
% {\color{blue}
% For \LaTeX{} users, make sure to activate the review mode of the template by uncommenting the line \verb|\documentclass{Interspeech}| and commenting the line \verb|\documentclass[cameraready]{Interspeech}| at the beginning of the template.}
% Papers that reveal the identity of the authors will be rejected without review.
% Note that the full list of authors must still be provided in the online submission system, since this is necessary for detecting conflicts of interest.
% \subsubsection{Camera-ready version}
% Authors should include names and affiliations in the final version of the manuscript, for publication.
% {\color{blue}
% For \LaTeX{} users, make sure to activate the camera-ready mode of the template by commenting the line \verb|\documentclass{Interspeech}| and uncommenting the line \verb|\documentclass[cameraready]{Interspeech}| at the beginning of the template.
% }
% Include the country as part of each affiliation. Do not use company logos anywhere in the manuscript, including in affiliations and Acknowledgements. After acceptance, authors may of course reveal their identity in other ways, including: adjusting the wording around self-citations; adding further acknowledgements; updating multimedia and online material. Acknowledgements, if any, should be added in the camera-ready version.
% \subsubsection{Pre-prints}
% \label{section:preprints}
% \blue{Please visit the submission policy under the \textit{For Authors} section of the Interspeech 2026 website (\url{https://interspeech2026.org/}) for a detailed explanation of the policy on pre-prints for this Interspeech. Interspeech no longer enforces an anonymity period for submissions published in archive repositories. While uploading a version online is permitted, your official submission to Interspeech must not contain any author-identifying information.}
% \section{Format}
% \subsection{Layout}
% Authors should observe the following specification for page layout by using the provided template. Do not modify the template layout! Do not reduce the line spacing!
% \subsubsection{Page layout}
% \begin{itemize}
% \item Paper size must be DIN A4.
% \item Two columns are used except for the title section and for large figures or tables that may need a full page width.
% \item Left and right margin are \SI{20}{\milli\metre} each.
% \item Column width is \SI{80}{\milli\metre}.
% \item Spacing between columns is \SI{10}{\milli\metre}.
% \item Top margin is \SI{25}{\milli\metre} (except for the first page which is \SI{30}{\milli\metre} to the title top).
% \item Bottom margin is \SI{35}{\milli\metre}.
% \item Text height (without headers and footers) is maximum \SI{235}{\milli\metre}.
% \item Page headers and footers must be left empty.
% \item No page numbers.
% \item Check indentations and spacing by comparing to the example PDF file.
% \end{itemize}
% \subsubsection{Section headings}
% Section headings are centred in boldface with the first word capitalised and the rest of the heading in lower case. Sub-headings appear like major headings, except they start at the left margin in the column. Sub-sub-headings appear like sub-headings, except they are in italics and not boldface. See the examples in this file. No more than 3 levels of headings should be used.
% \subsubsection{Fonts}
% Times or Times Roman font is used for the main text. Font size in the main text must be 9 points, and in the References section 8 points. Other font types may be used if needed for special purposes. \LaTeX\xspace users should use Adobe Type 1 fonts such as Times or Times Roman, which is done automatically by the provided \LaTeX\xspace class. Do not use Type 3 (bitmap) fonts. Phonemic transcriptions should be placed between forward slashes and phonetic transcriptions between square brackets, for example \textipa{/lO: \ae nd O:d3/} vs. \textipa{[lO:r@nO:d@]}, and authors are encouraged to use the terms `phoneme' and `phone' correctly \cite{moore19_interspeech}.
% \subsubsection{Hyperlinks}
% For technical reasons, the proceedings editor may strip all active links from the papers during processing. URLs can be included in your paper if written in full, e.g., \url{https://www.interspeech2026.org/}. The text must be all black. Please make sure that they are legible when printed on paper.
% \subsection{Figures}\label{sec:figures}
% Figures must be centred in the column or page. Figures which span 2 columns must be placed at the top or bottom of a page.
% Captions should follow each figure and have the format used in Figure~\ref{fig:speech_production}. Diagrams should preferably be vector graphics. Figures must be legible when printed in monochrome on DIN A4 paper; a minimum font size of 8 points for all text within figures is recommended. Diagrams must not use stipple fill patterns because they will not reproduce properly in Adobe PDF. See \textit{APA Guidelines regarding the use of color in Figures} (\url{https://apastyle.apa.org/style-grammar-guidelines/tables-figures/colors}).
% From an accessibility standpoint, please ensure that your paper remains coherent for a reader who cannot see your figures. Please make sure that:
% \begin{itemize}
% \item all figures are referenced in the text.
% \item figures are fully explained either in text, in their caption, or in a combination of the two.
% \item instead of visualizing differences between areas in your figures only by color, those differences can also be illustrated by using different textures or patterns.
% \item all text, including axis labels, is at least as large as the content text surrounding the figures.
% \item \blue{We encourage to use color-blind–friendly color palettes (see \url{https://www.color-blindness.com/coblis-color-blindness-simulator/})}
% \end{itemize}
% \subsection{Tables}\label{sec:tables}
% An example of a table is shown in Table~\ref{tab:example}. The caption text must be above the table. Tables must be legible when printed in monochrome on DIN A4 paper; a minimum font size of 8 points is recommended.
% From an accessibility standpoint, please make sure that:
% \begin{itemize}
% \item tables are not images.
% \item all table headings are clearly marked as such, and are distinct from table cells.
% \item all tables are referenced in the text.
% \end{itemize}
% \begin{table}[th]
% \caption{This is an example of a table}
% \label{tab:example}
% \centering
% \begin{tabular}{ r@{}l r }
% \toprule
% \multicolumn{2}{c}{\textbf{Ratio}} &
% \multicolumn{1}{c}{\textbf{Decibels}} \\
% \midrule
% $1$ & $/10$ & $-20$~~~ \\
% $1$ & $/1$ & $0$~~~ \\
% $2$ & $/1$ & $\approx 6$~~~ \\
% $3.16$ & $/1$ & $10$~~~ \\
% $10$ & $/1$ & $20$~~~ \\
% \bottomrule
% \end{tabular}
% \end{table}
% \subsection{Equations}
% Equations should be placed on separate lines and numbered. We define
% %
% \begin{align}
% x(t) &= s(t') \nonumber \\
% &= s(f_\omega(t))
% \end{align}
% %
% where \(f_\omega(t)\) is a special warping function. Equation \ref{equation:eq2} is a little more complicated.
% %
% \begin{align}
% f_\omega(t) &= \frac{1}{2 \pi j} \oint_C
% \frac{\nu^{-1k} \mathrm{d} \nu}
% {(1-\beta\nu^{-1})(\nu^{-1}-\beta)}
% \label{equation:eq2}
% \end{align}
% %
% \begin{figure}[t]
% \centering
% \includegraphics[width=\linewidth]{figure.pdf}
% \caption{Schematic diagram of speech production.}
% \label{fig:speech_production}
% \end{figure}
% \subsection{Style}
% Manuscripts must be written in English. Either US or UK spelling is acceptable (but do not mix them).
% \subsubsection{International System of Units (SI)}
% Use SI units, correctly formatted with a non-breaking space between the quantity and the unit. In \LaTeX\xspace this is best achieved using the \texttt{siunitx} package (which is already included by the provided \LaTeX\xspace class). This will produce
% \SI{25}{\milli\second}, \SI{44.1}{\kilo\hertz} and so on.
% \begin{table}[b!]
% \caption{Main predefined styles in Word}
% \label{tab:word_styles}
% \centering
% \begin{tabular}{ll}
% \toprule
% \textbf{Style Name} & \textbf{Entities in a Paper} \\
% \midrule
% Title & Title \\
% Author & Author name \\
% Affiliation & Author affiliation \\
% Email & Email address \\
% AbstractHeading & Abstract section heading \\
% Body Text & First paragraph in abstract \\
% Body Text Next & Following paragraphs in abstract \\
% Index & Index terms \\
% 1. Heading 1 & 1\textsuperscript{st} level section heading \\
% 1.1 Heading 2 & 2\textsuperscript{nd} level section heading \\
% 1.1.1 Heading 3 & 3\textsuperscript{rd} level section heading \\
% Body Text & First paragraph in section \\
% Body Text Next & Following paragraphs in section \\
% Figure Caption & Figure caption \\
% Table Caption & Table caption \\
% Equation & Equations \\
% \textbullet\ List Bullet & Bulleted lists \\\relax
% [1] Reference & References \\
% \bottomrule
% \end{tabular}
% \end{table}
% \section{Specific information for Microsoft Word}
% For ease of formatting, please use the styles listed in Table \ref{tab:word_styles}. The styles are defined in the Word version of this template and are shown in the order in which they would be used when writing a paper. When the heading styles in Table \ref{tab:word_styles} are used, section numbers are no longer required to be typed in because they will be automatically numbered by Word. Similarly, reference items will be automatically numbered by Word when the ``Reference'' style is used.
% If your Word document contains equations, you must not save your Word document from ``.docx'' to ``.doc'' because this will convert all equations to images of unacceptably low resolution.
% \section{Submissions}
% Information on how and when to submit your paper is provided on the conference website.
% \subsection{Manuscript}
% Authors are required to submit a single PDF file of each manuscript. The PDF file should comply with the following requirements: (a) no password protection; (b) all fonts must be embedded; (c) text searchable (do ctrl-F and try to find a common word such as ``the''), and (d) the document properties do not reveal the identity or username. The conference organisers may contact authors of non-complying files to obtain a replacement. Papers for which an acceptable replacement is not provided in a timely manner will be withdrawn.
% \subsubsection{Embed all fonts}
% It is \textit{very important} that the PDF file embeds all fonts! PDF files created using \LaTeX, including on \url{https://overleaf.com}, will generally embed all fonts from the body text. However, it is possible that included figures (especially those in PDF or PS format) may use additional fonts that are not embedded, depending how they were created.
% On Windows, the bullzip printer can convert any PDF to have embedded and subsetted fonts. On Linux \& MacOS, converting to and from Postscript will embed all fonts:
% \\
% \noindent\textsf{pdf2ps file.pdf}\\
% \noindent\textsf{ps2pdf -dPDFSETTINGS=/prepress file.ps file.pdf}
% \subsubsection{Sanitise PDF metadata}
% \label{section:pdf_sanitise}
% Check that author identity is not revealed in the PDF metadata. The provided \LaTeX\xspace class ensures this. Metadata can be inspected using a PDF viewer.
% \subsection{Optional supplementary material or links to online material}
% \label{section:multimedia}
% \subsubsection{Supplementary material}
% \label{section:supplementary}
% Interspeech offers the option of submitting supplementary material.
% These files are meant for code, data or audio-visual illustrations that cannot be conveyed in text, tables and graphs. Just as with figures used in your manuscript, make sure that you have sufficient author rights to all other materials that you submit. This material will be available to reviewers and committee members for evaluating the paper, but it will not be published with the paper or accessible in the archives. To make material available for future readers, please use online resources (see section \ref{section:repos}). Supplementary material is intended to support the reviewers in assessing the paper; however, the paper should be understandable on its own. Reviewers may choose to examine the supplementary material but are not required to do so.
% In line with the double-blind review policy, make sure that the supplementary material is anonymised. Unless strictly necessary, avoid to provide code at this step, which is usually harder to anonymise.
% Your supplementary material must be submitted in a single ZIP file limited to 100 MB for each separate paper. Within the ZIP file you can use folders to organise the files. In the ZIP file you should include a \texttt{README.txt} or \texttt{index.html} file to describe the content. In the manuscript, refer to a supplementary file by filename. Use short file names with no spaces.
% \subsubsection{Online resources such as web sites, blog posts, code, and data}
% \label{section:repos}
% It is common for manuscripts to provide links to web sites (e.g., as an alternative to including a ZIP file as supplementary material), code repositories, data sets, or other online resources. Provision of such materials is generally encouraged; however, they should not be used to circumvent the limit on manuscript length. In particular, reviewers will not be required to read any content other than the main manuscript.
% Links included in the version submitted for review should not reveal the identity or affiliation of the authors. If this is not possible, then the links should not be provided in the version submitted for review, but they can be added in the final version of the paper, if accepted. A placeholder can be included in the paper with a text like: ``[to ensure author anonymity, the link to the resource will be added after the review process]''. Links and repositories provided for review should be anonymous. In particular, names in repositories are forbidden.
% Authors are encouraged to group any supplementary material in a single repository. If the paper is accepted, authors will have the option to include a single link to this repository in the paper's metadata on ISCA Archives. For that purpose, the link must be a DOI link. Free DOI links can for instance be obtained through Zenodo, with additional guidance available in GitHub Docs (\url{https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content})
% \section{Discussion}
% Authors must proofread their PDF file prior to submission, to ensure it is correct. Do not rely on proofreading the \LaTeX\xspace source or Word document. \textbf{Please proofread the PDF file before it is submitted.}
% \section{Acknowledgments}
% {\color{blue}Acknowledgments should be included only in the camera-ready version, not in the version submitted for review. For regular papers, pages 5 and 6, and for long papers, pages 9 and 10, are reserved exclusively for acknowledgments, disclosures of the use of generative AI tools, and references. No other content may appear on these pages. Any appendices must be contained within the first four pages for regular papers and within the first eight pages for long papers.
% Acknowledgments and references may begin on an earlier page if space permits.}
% \ifcameraready
% The Interspeech 2026 organizers
% \else
% The authors
% \fi
% would like to thank ISCA and the organizing committees of past Interspeech conferences for their help and for kindly providing the previous version of this template.
% {\color{blue}
% \section{Generative AI Use Disclosure}
% The extent of Generative AI use must be disclosed. This section may be in the 5th or 6th pages of regular papers, or the 9th or 10th pages of long papers. ISCA policy says: \textit{All (co-)authors must be responsible and accountable for the work and content of the paper, and they must consent to its submission. Any generative AI tools cannot be a co-author of the paper. They can be used for editing and polishing manuscripts, but should not be used for producing a significant part of the manuscript}.}
% \section{Reference Format}
% {\color{blue}It is ISCA policy that papers submitted to Interspeech should refer to peer-reviewed publications. References to non-peer-reviewed publications (including public repositories such as arXiv, Preprints, and HAL, software, and personal communications) should only be made if there is no peer-reviewed publication available, and should be kept to a minimum.
% References should be in standard IEEE format (follows the IEEEtran format), numbered in order of appearance, for example \cite{Davis80-COP} is cited before \cite{Rabiner89-ATO}. For longer works such as books, provide a single entry for the complete work in the References, then cite specific pages \cite[pp.\ 417--422]{Hastie09-TEO} or a chapter \cite[Chapter 2]{Hastie09-TEO}. Multiple references may be cited in a list \cite{Smith22-XXX, Jones22-XXX}.}
\clearpage
\section{Generative AI Use Disclosure}
We use generative AI tools for polishing the manuscript, e.g., correcting the grammar.
\bibliographystyle{IEEEtran}
\bibliography{mybib}
\end{document}
%%% Local Variables:
%%% mode: LaTeX
%%% TeX-master: t
%%% End:
\section{Introduction}
Accent variability remains a persistent challenge for automatic speech recognition (ASR) systems~\cite{ASR_in_diverse_accents}. Systematic differences in phoneme realization, prosody, and phonotactic patterns across regional and non-native accents often lead to recognition errors that disproportionately affect certain speaker populations~\cite{vctk, l2}. In real-world deployments, including voice assistants, call centers, and educational technologies, such performance disparities degrade user experience and raise important concerns regarding fairness and accessibility~\cite{koenecke2020racial}. Despite advances in large-scale pretraining and robust acoustic modeling~\cite{xiao2025dg}, recognition gaps across accents continue to be observed.
Conventional accent adaptation techniques typically rely on supervised fine-tuning, accent-specific modeling, or data augmentation~\cite{ghorbani2022domain,qian2022layer,prabhu2023accented}. While effective in traditional ASR settings, these approaches become increasingly costly and operationally restrictive in the era of foundation-scale speech models. Full-parameter adaptation is computationally expensive and may compromise generalization across diverse accents and tasks~\cite{poor_generalization, unseen_gap_1, unseen_gap_2}. As modern speech systems are frequently deployed as shared, large-scale backbones, there is growing need for lightweight and scalable adaptation mechanisms.
Recent large audio language models (LALMs), such as Whisper~\cite{radford2023robust}, combine powerful acoustic modeling with large language modeling to achieve strong zero-shot generalization. While Parameter-Efficient Fine-Tuning (PEFT) methods and bottleneck adapters~\cite{houlsby2019adapters,hu2021lora,learn_and_dont_forget, parameter-efficient_model_reprogramming} offer a more lightweight adaptation approach by adding only a small number of trainable parameters on top of a frozen base model, these approaches still optimize the added parameters heuristically~\cite{song2024lora_whisper, prasad2024lowrank_asr,mas-lora, xu2024towards}. Without explicitly localizing or constraining updates to layers and subspaces that are most sensitive to accent-related variation, these methods may perform unnecessary adaptation in accent-insensitive regions of the model and risk entangling accent compensation with higher-level semantic representations, limiting both efficiency and control.
Understanding how accent variation is organized in representation space is critical for designing scalable and controllable adaptation strategies. If accent corresponds to identifiable subspaces in hidden activations, then targeted representation-level interventions may offer an alternative to parameter-intensive fine-tuning~\cite{xiao2026adapting}.
Recent studies in large language models show that high-level attributes, such as sentiment, style, or safety~\cite{Category-wise_safety_steering, Affine_concept_editing, dynamic_steering,turner2024activation,zou2023representation}, can align with approximately linear directions in activation space. These directions, often referred to as steering vectors, are typically constructed as mean activation shifts between contrasting concepts~\cite{rimsky2024steering,ilharcoediting}. When added to hidden states at selected layers, they nudge representations toward desired regions of the space, enabling controllable behavioral modulation at inference time without modifying model parameters. This perspective motivates the central question of this work: \emph{Does accent variation correspond to a structured and controllable subspace in LALMs?}
% In this paper, we first conduct a layer-wise analysis of hidden activations to characterize the geometric structure of accent-related variation. This analysis serves two purposes. From an \emph{interpretability} standpoint, it reveals how accent information is distributed across layers and whether it exhibits consistent structural patterns. From a \emph{controllability} standpoint, it determines whether accent variation is sufficiently organized to permit targeted intervention.
% Building on this analysis, we construct steering vectors to modulate accent-related representations during inference and evaluate their effectiveness in reducing accent-induced recognition disparities. Rather than updating model parameters, our approach injects learned accent-specific directions into hidden states, enabling lightweight and localized adaptation.
% To the best of our knowledge, we are the first to analyze and apply vector steering systematically for both native and non-native accented speech.
% \td{The results on xx demonstrate that mean-shift steering significantly improves WER across xx accents while maintaining the model’s capacity for non-accented speech.}
% By bridging accent robustness and representation-level controllability, this work advances understanding of how LALMs encode accent variation and explores a scalable pathway toward accent-adaptive ASR.\td{we may need to slightly tailor the intro a bit once we have the results; now we mention the controllability, scalability, and interoperability, and the key point is not fully clarified among them. }
In this paper, we first conduct a layer-wise analysis of hidden activations to understand the geometric structure of accent variations. This investigation serves two primary purposes. From an \emph{interpretability} perspective, it reveals how the model distributes accent information across layers. Furthermore, from a \emph{controllability} perspective, it determines whether these accent features are organized enough to allow direct adjustments. Based on this foundation, we construct steering vectors to adjust accent representations during model inference. Our method inserts learned accent directions into specific hidden states. We subsequently evaluate their effectiveness in automatic speech recognition (ASR) tasks across diverse pronunciations. %This technique achieves extremely light and targeted adaptation.
To the best of our knowledge, this is the first study to systematically analyze and apply vector steering for accent-robust automatic speech recognition. Our extensive experiments on the VCTK and L2-ARCTIC datasets demonstrate the middle layers and late layers are the most sensitive to accent-induced representation shifts. Moreover, mean shift steering significantly improves the word error rate across eight distinct accents. We observe that interventions in the middle layers provide the most effective control. This study offers a principled and scalable way to reduce accent-induced recognition disparities in speech foundation models, advancing both robustness and fairness in real-world ASR deployment.%More importantly, in the VCTK, our method achieves better performance gains than using conventional PEFT methods.
% Ultimately,
% Consequently, it establishes a highly scalable and data-efficient pathway for building adaptive recognition systems.
\section{Accent Subspace Analysis and Steering}
\subsection{Layer-wise Accent Subspace Analysis}
We analyze how accent-related variation is organized across encoder layers and quantify how strongly each layer responds to accent-specific representation shifts. This yields a layer-wise sensitivity profile that highlights which layers are most suitable for subsequent accent steering.
\subsubsection{Accent Representation Perturbation}
We construct text-matched utterance pairs to isolate accent-related acoustic variation from linguistic content. For each target accent, we create two types of pairs: (i) \textit{cross-standard-accent} pairs $(x_s, x_a)$, where $x_a$ is an accented utterance and $x_s$ is a standard-English utterance with an identical transcript, and (ii) \textit{within-single-accent} pairs $(x_a, x_a)$ formed by different speakers from the same accent group. The cross-standard-accent pairs capture systematic accent-induced differences, while the within-single-accent pairs serve as a control to account for speaker-specific factors such as timbre or prosody, helping us separate accent effects from general inter-speaker variation. Given accented speech samples ${x_a}$ and standard speech samples ${x_s}$, our layer-wise analysis then quantifies, at each encoder layer, how the latent representations of accented speech differ from those of standard English, and how effectively a layer supports aligning these two clusters. Concretely, for each layer $l$, we first extract the token-level hidden activations $\mathbf{H}^{(l)}$. For each speech sample $i$, we apply mean pooling over time to obtain an utterance-level representation $\bar{\mathbf{h}}_{i}^{(l)}$. Based on these pooled representations, we compute an accent mean-shift direction from a source accent group $s$ to a target accent group $a$:
\begin{equation}
\mathbf{d}_{s \rightarrow a}^{(l)}
=
\frac{1}{|G_s|}\sum_{j \in G_s}\bar{\mathbf{h}}_{j}^{(l)}
-
\frac{1}{|G_a|}\sum_{i \in G_a}\bar{\mathbf{h}}_{i}^{(l)}
\label{eq:mean_shift}
\end{equation}
where $G_s$ and $G_a$ denote the sample sets corresponding to the standard and accent speech, respectively. This vector $\mathbf{d}_{s \rightarrow a}^{(l)}$ characterizes the accent shift between the two representation clusters at layer $l$.
To probe the sensitivity of layer $l$ to this accent direction, we apply a controlled perturbation to its hidden activations:
\begin{equation}
\tilde{\mathbf{H}}^{(l)} = \mathbf{H}^{(l)} + \alpha \cdot \mathbf{d}_{s \rightarrow a}^{(l)},
\end{equation}
where $\alpha$ is optimized to $1.0$, corresponding to a one-unit accent mean-shift perturbation, and $\mathbf{d}_{s \rightarrow a}^{(l)}$ is broadcast across all time steps of $\mathbf{H}^{(l)}$. The resulting perturbed activations $\tilde{\mathbf{H}}^{(l)}$ are then propagated through the remaining encoder and projector layers. %In Section~2.1.2, we measure how this intervention changes the alignment between accented and standard representations to derive a layer-wise Accent Alignment Score, which in turn defines the sensitivity profile used to identify accent-sensitive layers.
% We construct text-matched utterance pairs to isolate accent-related acoustic variation from linguistic content. For each target accent, we form (i) \textit{cross-standard-accent} pairs $(x_s, x_a)$, where $x_a$ is an accented utterance and $x_s$ is a standard-English utterance with identical transcript, and (ii) \textit{within-single-accent} pairs $(x_a, x_a)$, different speakers from same accent group as a control condition. The within-single-accent pairs help account for speaker-specific acoustic variation unrelated to accent (e.g., timbre or prosody), allowing us to further distinguish accent effects from general inter-speaker differences.
% Our layer-wise analysis aims to quantify, at each encoder layer, how the latent representations of accented speech differ from those of standard English, and how strongly a layer supports aligning these two representation clusters. Concretely, for a dataset of accented speech $\{x_a\}$ and standard speech $\{x_s\}$, we extract their hidden representations in the latent space and characterize the accent shift by an accent-specific direction (i.e., the directional subspace between the two clusters). We then probe each layer by applying a controlled displacement along this direction—moving the accented cluster toward the standard cluster—and measure the resulting change in alignment to obtain a layer-wise sensitivity profile.
% Specifically, for each layer $l$, we extract the token-level hidden activations $\mathbf{H}^{(l)}$. Next, for each speech sample $i$, we apply mean pooling. This operation creates a sample level representation $\bar{\mathbf{h}}_{i}^{(l)}$. Based on these pooled results, we calculate the mean shift vector. We define this vector $\mathbf{d}_{s \rightarrow a}^{(l)}$ for a direction from source $a$ to target $b$ as follows:
% \begin{equation}
% \mathbf{d}_{s \rightarrow a}^{(l)}
% =
% \frac{1}{|G_b|}\sum_{j \in G_b}\bar{\mathbf{h}}_{j}^{(l)}
% -
% \frac{1}{|G_a|}\sum_{i \in G_a}\bar{\mathbf{h}}_{i}^{(l)}
% \label{eq:mean_shift}
% \end{equation}
% In this equation, $G_a$ and $G_b$ represent the sample sets of the source accent and the target accent, respectively.
% % We then perform the intervention as: -- tone已改为如下：
% To probe layer $l$, we apply a controlled perturbation to the hidden activations:
% \begin{equation}
% \tilde{\mathbf{H}}^{(l)} = \mathbf{H}^{(l)} + \alpha \cdot \mathbf{d}_{s \rightarrow a}^{(l)},
% \end{equation}
% where $\alpha$ is set to $1.0$, representing a one-unit accent mean-shift perturbation, and $\mathbf{d}_{s \rightarrow a}^{(l)}$ is broadcast across all time steps of $\mathbf{H}^{(l)}$, yielding the perturbed hidden activations $\tilde{\mathbf{H}}^{(l)}$.
\subsubsection{Quantifying Layer Sensitivity via Accent Alignment Score (AAS)}
Given the layer-wise perturbation defined above,
% we next require a metric to quantify how strongly each layer controls accent-related alignment in downstream representations.
we quantify how much each layer contributes to reducing the representation gap between accented and standard speech.
Our analysis is conducted on the audio encoder of Qwen2-Audio-7B~\cite{qwen2audio}, which consists of 32 Whisper-style encoder layers followed by a multi-modal projector. Since accent variation is primarily expressed through acoustic patterns, we focus on hidden-state transformations within the audio encoder and measure their propagated effects in the projector space, which provides the final speech representation passed to the downstream language model.
For each source-target speech pair, we apply the mean-shift displacement $\mathbf{d}_{s \rightarrow a}^{(l)}$ at audio encoder layer $l$ and measure its propagated effect in the multi-modal projector space.
% We use the projector space because it is the final speech representation passed to the downstream language model, providing a unified space to assess how a layer-wise perturbation changes the resulting speech embedding.
Because the projector output remains time-dependent, we further apply mean pooling over the projector time steps to obtain a single utterance-level representation for each sample, on which cosine similarity is computed.
We define the Accent Alignment Score (AAS) as the change in cosine similarity between the perturbed source representation and the target representation:
\begin{equation}
\mathrm{AAS}_{s \rightarrow a}^{(l)} = \cos\!(\tilde{\mathbf{z}}_{\mathrm{proj}}^{\,a, (l)}, \mathbf{z}_{\mathrm{proj}}^{\,b}) - \cos\!(\mathbf{z}_{\mathrm{proj}}^{\,a}, \mathbf{z}_{\mathrm{proj}}^{\,b})
\end{equation}
% In this formula, $\tilde{\mathbf{z}}_{\mathrm{proj}}^{\,a,(l)}$ denotes the projector output of the source speech samples after intervention at layer $l$. Meanwhile, $\mathbf{z}_{\mathrm{proj}}^{\,a}$ and $\mathbf{z}_{\mathrm{proj}}^{\,b}$ represent the baseline projector outputs for the source and target speech samples, respectively. A positive $\mathrm{AAS}$ indicates that the intervention moves the source representation closer to the target accent representation.
Here, $\tilde{\mathbf{z}}_{\mathrm{proj}}^{\,a,(l)}$ denotes the projector output of the source speech sample after applying the layer-$l$ perturbation. Meanwhile, $\mathbf{z}_{\mathrm{proj}}^{\,a}$ and $\mathbf{z}_{\mathrm{proj}}^{\,b}$ represent the baseline projector outputs for the source and target speech samples, respectively. A positive $\mathrm{AAS}$ indicates that the layer-wise perturbation moves the source representation closer to the target accent representation.
We apply the same procedure to both cross-standard-accent and within-single-accent pairs, and compute their AAS values at each layer. Specifically for within-single-accent pairs, we calculated a within accent mean-shift vector $\mathbf{d}_{a \rightarrow a}^{(l)}$ between two speakers of the same accent, following the same AAS calculation procedure.
To isolate the additional alignment gain observed in cross pairs beyond general speaker variation, we compute the specificity score, which measures the amount of the patching effect that truly comes from the accent:
\begin{equation}
\mathrm{Spec}^{(l)}
=
\begin{aligned}
\overline{\mathrm{AAS}}_{\mathrm{cross}}^{(l)}
-
\overline{\mathrm{AAS}}_{\mathrm{within}}^{(l)}
\end{aligned}
\end{equation}
where $\mathrm{Spec}^{(l)}$ denotes \textit{Specificity} score at layer $l$, and $\overline{\mathrm{AAS}}$ denotes the average score across all evaluated speech pairs in their respective sets. The subscript $\mathrm{cross}$ denotes cross-standard--accent pairs, while $\mathrm{within}$ denotes within-accent pairs. A positive $\mathrm{Spec}^{(l)}$ value proves that the patching effect at this specific layer is indeed caused by accent differences. Consequently, it is not caused by other speaker-level variations. Based on this logic, the layer sensitivity score is defined as follows:
\vspace{-3mm}
\begin{equation}
\mathrm{Sensitivity}^{(l)} = \max(0, \mathrm{Spec}^{(l)})
\end{equation}
\vspace{-6mm}
which is used for ranking across layers. In addition, our layer-wise analysis adopts a bidirectional design (\textit{standard} $\rightarrow$ \textit{target} and \textit{target} $\rightarrow$ \textit{standard}) to reduce direction-specific bias and provide a more robust estimate of layer sensitivity. Each direction is computed independently and then averaged.
\subsection{Inference-Time Accent Steering}
The layer-wise sensitivity analysis identifies candidate layers for controllable accent intervention. Building on the analysis, we next construct steering vectors to modulate accent-related representation during inference. We then test whether steering at these layers can produce measurable shifts that improve the downstream ASR performance of accented speech toward that of standard speech. To extract the generalized steering vector and avoid the effect of speakers and sentences, we design an \textit{extraction set} and an \textit{evaluation set}. They are isolated from the speakers and texts. This design ensures that the steering vector is estimated from data that shares neither speaker identity nor text content with the evaluation data.
For steering, we reuse the mean-shift direction defined in \eqref{eq:mean_shift}, but compute it only from the extraction set. Before injection, we normalize the direction $\mathbf{d}_{s \rightarrow a}^{(l)}$ to unit norm:
\begin{equation}
\hat{\mathbf{d}}_{s \rightarrow a}^{(l)}
=
\frac{\mathbf{d}_{s \rightarrow a}^{(l)}}{\left\lVert \mathbf{d}_{s \rightarrow a}^{(l)} \right\rVert}
\end{equation}
This separates direction from magnitude and makes the steering strength parameter $\alpha$ comparable across layers.
During inference, we inject the normalized steering vector into the hidden states at the selected layer $l$:
\begin{equation}
\tilde{\mathbf{H}}^{(l)}
=
\mathbf{H}^{(l)} + \alpha \hat{\mathbf{d}}_{s \rightarrow a}^{(l)}
\end{equation}
where $\alpha$ controls the steering strength. The vector $\hat{\mathbf{d}}_{s \rightarrow a}^{(l)}$ is broadcast across all time steps of $\mathbf{H}^{(l)}$. We implement this intervention using a forward hook, so no model parameters are modified. We propose that the vector of an accent direction estimated from a subset of speech data can generalize effectively to unseen speakers and unseen utterances. The evaluation is in the Results section.
\section{Experiment Settings}
\subsection{Datasets and Accents}
\noindent \textbf{Native accents:} We adopt the VCTK dataset~\cite{vctk}, a high-fidelity corpus featuring speakers with diverse regional accents, to study native English variations. Specifically, we select Scottish, South African, Canadian, Irish, and Northern Irish accents for our experiments.
% , applying a four-fold cross-validation method for each.
As VCTK includes a standard English group, we utilize this subset directly as our reference for comparison.
\noindent \textbf{Non-native accents:} We utilize the L2-ARCTIC corpus~\cite{l2}, which provides manual phonetic-level annotations of speech from diverse linguistic backgrounds. In this study, we focus on Hindi, Arabic, and Spanish accents.
% , evaluating each via four-fold cross-validation.
Since L2-ARCTIC lacks an internal native reference, we draw native speakers from its source, the CMU-ARCTIC dataset~\cite{cmu}, which consists of phonetically balanced sentences designed for speech synthesis. Because both datasets utilize identical reading scripts, the CMU-ARCTIC data serve as an ideal, matched reference group.
\vspace{-5pt}
\subsection{Pair Construction and Data Splits}
\noindent \textbf{Accent Subspace Analysis:} For each of the above accents, we construct 1000 cross-standard-accent pairs and 500 within-single-accent pairs. No data split is required.
\noindent \textbf{Inference-Time Accent Steering:} For each of the above target accents, we adopt a strict data splitting protocol: $80\%$ of the speakers are assigned to the extraction set, while the remaining $20\%$ are reserved for the evaluation set.
We further enforce no transcript overlap between extraction pairs and evaluation utterances to avoid text leakage.
For each accent group, we randomly sample 1,000 speech pairs from the extraction set to estimate the mean-shift vector.
\vspace{-5pt}
\subsection{Evaluation Protocol}
We evaluate steering in a single-layer sweep across 32 encoder layers, where each layer and each steering strength $\alpha$ are tested independently. We choose $\alpha$ = [0.5, 1, 2, 5] to cover different scale of steering strength.
% 需要说alpha sweep么？不同取值的alpha实验（results中也有related comment）
Steering effectiveness is evaluated by the resulting change in Word Error Rate (WER).
To reduce bias from overly easy or overly difficult samples, the evaluation set is constructed by balanced sampling: half of the utterances are drawn from samples with $\mathrm{WER}=0$, and the other half from samples with $\mathrm{WER}>0$. In a typical setting, we sample 100 utterances from each group, yielding 200 evaluation samples in total.
% For every accent utterance in the evaluation set, we perform both baseline generation (without steering) and steered generation, and then compare their WER values.
% The $\mathrm{WER}>0$ subset is further stratified by WER interval to avoid over-representation of edge cases. Our primary metric is the WER change after steering:
% \begin{equation}
% \Delta \mathrm{WER}
% =
% \mathrm{WER}_{\mathrm{steered}} - \mathrm{WER}_{\mathrm{baseline}},
% \end{equation}
% where a negative value indicates improved recognition under steering.
% \subsection{LoRA PEFT Baseline}
% We use LoRA as a conventional PEFT baseline for comparison. The LoRA adapters are configured with rank $r=16$, $\alpha=16$, and dropout $=0.05$, and are applied to the \texttt{q\_proj} and \texttt{v\_proj} modules across all 32 layers of the audio encoder. Training is conducted for 3 epochs with an effective batch size of 16 and a learning rate of $5\times10^{-5}$. To ensure a fair comparison, we strictly control the amount of training data exposed to the LoRA adapters, so that the number of unique utterances used for fine-tuning exactly matches the number used to compute the mean-shift vectors.
% For any accent group evaluated with $n$-fold cross-validation, we use the same $n$-fold training splits for LoRA and report the final results after averaging across folds.
\section{Results and analysis}
% 排版乱的 咋排都排不对 GPT也de不出来 TAT
% 主要发现
% 1. 15-20层真的是适合steering的窗口，跨native和non-native accents的共性
% 2. 21-30层不那么适合steering，pattern across accents vary. 尤其在non-native accents上非常容易造成over-steering而表征崩溃。即使是在native accents上也没有显著超过mid layers的WER改善，并且各accents趋势分化，pattern不泛化，可控性较差。原因引用2.4 late layer analysis。
% 3. 早期层0-14确实不适合steering（应证了2.4分析）。过早的干涉的结果是delta WER在0上下徘徊，无显著pattern。
% 4. Steering的强势场景在于limited accent sample pairs的高效改善。见表可知，finetune lora的主要优势存在于大量training samples，在少量samples上finetune性能非常糟糕；相对的，steering却能在极少量的samples上实现性能的大幅提升。即便同时拥有大量数据，也能获得接近finetune的性能提升。
% 此外，steering不涉及任何参数更新的nature，和按需灵活计算与挂载的特性，让该方法不存在对于原始/target口音的任何损伤。
% 这将充分解决当前LALM面临繁多accent变种时难以快速实现普遍性能提升的挑战，通过易收集的少量样本，即可在inference阶段实时带来显著的性能提升。完全满足轻量化、高性能、易使用的需求。
% 5. 第31层无一例外显著的WER大退化，应证了之前拒绝在31层分析和注入的决策，tail layer不适合进行Steering。
% 是否需要介绍不同alpha sweep结果？还是说就提一嘴alpha=5时效果最好，pattern清晰
% =========================
% Figure
% =========================
\begin{figure*}[t!]
\centering
\noindent
% -------- Col 1 (a over b) --------
\begin{minipage}[t]{0.32\textwidth}
\centering
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{figures/sensitivity_vctk_5accent_normalized_smooth_window7_scaled.pdf}
\fixedsubcap{Layer-wise accent sensitivity analysis on VCTK dataset}
\label{fig:sens_vctk}
\end{subfigure}
\vspace{3pt}
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{figures/sensitivity_l2arctic_3accent_normalized_smooth_window5_scaled.pdf}
\fixedsubcap{Layer-wise accent sensitivity analysis on L2ARCTIC dataset}
\label{fig:sens_l2}
\end{subfigure}
\end{minipage}\hfill% 关键：不要空行，行尾加 %
% -------- Col 2 (c over d) --------
\begin{minipage}[t]{0.32\textwidth}
\centering
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{figures/steering_vctk_5accent_alpha5_scaled.pdf}
\fixedsubcap{Steering $\Delta$WER on VCTK dataset}
\label{fig:steering_vctk}
\end{subfigure}
\vspace{3pt}
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{figures/steering_l2arctic_3accent_alpha5_scaled.pdf}
\fixedsubcap{Steering $\Delta$WER on L2-ARCTIC dataset}
\label{fig:steering_l2arctic}
\end{subfigure}
\end{minipage}\hfill%
% -------- Col 3 (e over f) --------
\begin{minipage}[t]{0.32\textwidth}
\centering
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{figures/alpha_sensitivity_vctk_native_scaled.pdf}
\fixedsubcap{Alpha sensitivity on average $\Delta$WER on VCTK dataset}
\label{fig:alpha_vctk}
\end{subfigure}
\vspace{3pt}
\begin{subfigure}[t]{\linewidth}
\centering
\includegraphics[width=0.8\linewidth]{figures/alpha_sensitivity_l2arctic_non_native_scaled.pdf}
\fixedsubcap{Alpha sensitivity on average $\Delta$WER on L2-ARCTIC dataset}
\label{fig:alpha_l2arctic}
\end{subfigure}
\end{minipage}
\vspace{-5mm}
% -------- Big caption (Figure 1) --------
\caption{Layer-wise accent subspace analysis and steering results with alpha sweep.}
\label{fig:accent_subspace_steering}
\vspace{-5mm} % 可选：整体收紧
\end{figure*}
% Layer analysis results - 1 large figure?
% \begin{figure*}[t!]
% \centering
% \begin{subfigure}[t]{0.32\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figures/sensitivity_vctk_5accent_normalized_smooth_window7_scaled.pdf}
% \vspace{-5mm}
% \caption{Layer-wise accent sensitivity analysis on VCTK dataset}
% \end{subfigure}
% \hfill
% % \vspace{-6mm} % adjust (or use a negative value to tighten)
% \begin{subfigure}[t]{0.32\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figures/sensitivity_l2arctic_3accent_normalized_smooth_window5_scaled.pdf}
% \vspace{-5mm}
% \caption{Layer-wise accent sensitivity analysis on L2-ARCTIC dataset}
% \end{subfigure}
% \hfill
% % \vspace{-6mm}
% \begin{subfigure}[t]{0.32\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figures/steering_vctk_5accent_alpha5_scaled.pdf}
% \vspace{-4mm}
% \caption{Steering $\Delta$WER on VCTK dataset}
% \label{fig:steering_vctk}
% % \vspace{-4mm}
% \end{subfigure}
% \hfill
% % Steering results - 2 figures
% \begin{subfigure}[t]{0.32\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figures/steering_l2arctic_3accent_alpha5_scaled.pdf}
% \vspace{-4mm}
% \caption{Steering $\Delta$WER on L2-ARCTIC dataset}
% \label{fig:steering_l2arctic}
% % \vspace{-4mm}
% \end{subfigure}
% \hfill
% % \caption{Layer-wise accent sensitivity analysis across VCTK and L2-ARCTIC datasets.}
% % \label{fig:accent_sensitivity}
% \begin{subfigure}[t]{0.32\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figures/alpha_sensitivity_vctk_native_scaled.pdf}
% \vspace{-4mm}
% \caption{Alpha sensitivity on average $\Delta$WER on VCTK dataset}
% \label{fig:alpha_vctk}
% \vspace{-6mm}
% \end{subfigure}
% \hfill
% \begin{subfigure}[t]{0.32\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figures/alpha_sensitivity_l2arctic_non_native_scaled.pdf}
% \vspace{-5mm}
% \caption{Alpha sensitivity on average $\Delta$WER on L2-ARCTIC dataset}
% \vspace{-6mm}
% \label{fig:alpha_l2arctic}
% \end{subfigure}
% \end{figure*}
% \begin{figure}[t]
% \centering
% \includegraphics[width=\linewidth]{figures/steering_l2arctic_3accent_alpha5_scaled.pdf}
% \vspace{-4mm}
% \caption{Steering $\Delta$WER on L2-ARCTIC dataset}
% \vspace{-4mm}
% \label{fig:steering_l2arctic}
% \end{figure}
% Alpha sweep analysis - 2 figures
% \begin{figure}[t]
% \centering
% \includegraphics[width=\linewidth]{figures/alpha_sensitivity_vctk_native_scaled.pdf}
% \vspace{-4mm}
% \caption{Alpha sensitivity on average $\Delta$WER on VCTK dataset}
% \label{fig:alpha_vctk}
% \vspace{-4mm}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[width=\linewidth]{figures/alpha_sensitivity_l2arctic_non-native_scaled.pdf}
% \vspace{-4mm}
% \caption{Alpha sensitivity on average $\Delta$WER on L2-ARCTIC dataset}
% \vspace{-4mm}
% \label{fig:alpha_l2arctic}
% \end{figure}
\subsection{Accent Subsapce Analysis}
\vspace{-5pt}
% To reduce scale differences, we normalize each accent group independently. Layer 31 is also excluded, since intervention at this layer is followed only by the linear multi-modal projector and therefore does not reflect further representation transformation in the model. As Figure~\ref{}, the sensitivity curves on the VCTK dataset show a shared cross-accent pattern: in the early layers (0-14), sensitivity is generally low and relatively flat; from around layer 15, a local mid-layer peak begins to emerge and gradually decays; from layer 21 onward, sensitivity rises sharply and reaches a global maximum around the last layers.
% To reduce scale differences, we normalize each accent group independently. We exclude layer 31 because this layer is followed only by the linear multi-modal projector. As in Figure~\ref{fig:sens_vctk}, the sensitivity curves on the VCTK dataset show a shared cross-accent pattern: in the early layers (0-14), sensitivity is generally low; from around layer 15, a peak begins to emerge; from layer 21 onward, sensitivity rises sharply and reaches a global maximum around the last layers.
% The sensitivity curves on the L2-ARCTIC dataset exhibit a similar pattern to that shown in Figure~\ref{fig:sens_l2}. However, compared with VCTK, L2-ARCTIC shows stronger global fluctuations. This pattern suggests that non-native accents induce more distributed and less uniform representational changes across the layers in the encoder.
To reduce scale differences, each accent group is normalized independently. Layer 31 is excluded as it directly precedes the linear multi-modal projector. Sensitivity analysis on VCTK shows a shared pattern: low sensitivity in early layers (0–14), an emerging peak near layer 15, and a sharp rise to global maximums from layer 21 onward. L2-ARCTIC exhibits similar trends but with stronger global fluctuations, suggesting non-native accents induce more distributed representational changes.
% Based on the sensitivity results, we further divide the potential steering window into three stages: early layers (0-14), middle layers (15-19), and late layers (20-30). The early layers mainly process raw acoustic information and remain at a relatively low level of abstraction. For native accents, sensitivity in the early layers is generally weak. For non-native accents, in contrast, multiple relatively high peaks appear in the early layers, indicating that the acoustic deviations introduced by non-native speech can already trigger clear responses. This suggests that non-native accents often involve stronger and more direct low-level acoustic deviations. From a steering perspective, making early intervention less likely to produce consistent and controllable effects. Taken together, the analysis suggests that the middle layers and late layers are potentially suitable places for steering vectors.
The steering window is divided into early (0–14), middle (15–19), and late (20–30) stages. Early layers process low-abstraction acoustic information~\cite{pasad2021layer,pasad2023comparativ}; native accents show weak sensitivity here, while non-native accents trigger higher peaks due to stronger low-level acoustic deviations. This makes early intervention less controllable. Middle and late layers are identified as the most suitable locations for steering vectors.
\subsection{Analysis of Layer-Wise Steering Dynamics}
\vspace{-5pt}
\subsubsection{Steering Effect on Native Accents}
\vspace{-5pt}
We first analyze the layer-wise steering effect on the VCTK dataset to understand native accent representations. As shown in Figure~\ref{fig:steering_vctk}, we observe a consistent trend across the Scottish, South African, Canadian, Irish, and Northern Irish groups. In the early layers, the intervention yields minimal impact on the mean word error rate. However, the middle layers exhibit a significant reduction in recognition errors
% \td{is the analysis part measured by sensitivity instead of erro rate? why we discuss error rate here?}.
This reduction peaks in the mid sections, reaching a maximum drop of nearly 0.3 for certain regions. Conversely, applying vectors to the late layers degrades performance sharply. For example, layer 31 experiences massive error increases across all groups. This geometric pattern suggests that native accent characteristics are primarily encoded in intermediate representations. Therefore, targeted interventions in these middle sections provide optimal control without disrupting higher level semantic understanding.
% \td{all the analysis compltely ignores the alpha values while you have alpha in the figure which may confuse the readers. If you decide to only show alpha of 5, mention it in the experimental setup and justify why. Readers might be intersted in the alpha values and range, you may want to add the discussion somewhere.}
\vspace{-5pt}
\subsubsection{Steering Effect on Non-native Accents}
\vspace{-5pt}
We extend our analysis to non-native speech using the L2-ARCTIC corpus. We evaluate the mean shift steering effect on Hindi, Arabic, and Spanish variations. Similar to the native setup, the performance changes reveal a distinct spatial structure as shown in Figure~\ref{fig:steering_l2arctic}. The early layers show relatively stable performance with minimal metric fluctuations. Subsequently, the middle layers again demonstrate a clear improvement in recognition accuracy. Although the error reduction is smaller than that of native speech, reaching around negative 0.05, the localized trend remains highly consistent. Furthermore, steering in the final layers causes severe degradation. Specifically, layer 31 alone adds approximately 4.0 to the error rate across all three linguistic backgrounds. This structure confirms that non-native accent variations also occupy a controllable subspace. Consequently, applying targeted steering to specific middle layers offers a scalable pathway for adaptive systems.
\subsubsection{Cross-Accent Steering Pattern and Insights}
% Based on the empirical results, we identify distinct layer-wise patterns that remain consistent across both native and non-native accents. First, the early layers, specifically layers 0 to 14, are largely unresponsive to steering. This intervention results in minimal metric changes, which aligns with our previous representation analysis. Next, the middle layers, from 15 to 20, emerge as the optimal window for targeted intervention. Within this region, we observe a consistent and significant reduction in recognition errors for all evaluated linguistic backgrounds. Therefore, this shared pattern confirms that accent characteristics are effectively isolated in these intermediate representations. Conversely, applying steering vectors to the late layers yields highly unstable results. The performance patterns across different accents diverge significantly in this region. For non-native accents, late-layer steering frequently causes severe representation collapse and over-steering. Even for native accents, the error reduction does not exceed that of the middle layers. This lack of a generalized pattern indicates poor controllability. Finally, intervention at the terminal layer, layer 31, consistently results in massive performance degradation across all test groups. This uniform failure confirms that the tail layer is unsuitable for vector injection.
% Based on the empirical results, we identify distinct layer-wise patterns that remain consistent across both native and non-native accents. First, the early layers, specifically layers 0 to 14, are largely unresponsive to steering. This intervention results in minimal metric changes, which aligns with our previous representation analysis. Next, the middle layers, from 15 to 19, emerge as the optimal window for targeted intervention. Within this region, we observe a consistent and significant reduction in recognition errors for all evaluated linguistic backgrounds. This suggests that representations in the middle layers are not yet fully fixed, while accent-related perturbations have already begun to propagate effectively and stably through the model. Therefore, this shared pattern confirms that accent characteristics are effectively isolated in these intermediate representations, making them a more suitable intervention window for controllable steering. Conversely, applying steering vectors to the late layers yields highly unstable results. The performance patterns across different accents diverge significantly in this region. For non-native accents, late-layer steering frequently causes severe representation collapse and over-steering. Even for native accents, the error reduction does not exceed that of the middle layers. Combined with the layer-wise sensitivity analysis, we attribute this to increasingly fixed representations in the late layers, where too few remaining encoder layers are left to absorb and reorganize the injected accent direction. This lack of a generalized pattern indicates poor controllability. Finally, intervention at the terminal layer, layer 31, consistently results in massive performance degradation across all test groups. This uniform failure confirms that the tail layer is unsuitable for vector injection.
Empirical results reveal consistent layer-wise patterns for both native and non-native accents. Layers 0–14 are largely unresponsive to steering, yielding minimal metric changes. Conversely, the middle layers (15–19) emerge as the optimal window, showing consistent, significant error reductions across all linguistic backgrounds. This suggests middle-layer representations are not yet fixed, allowing accent perturbations to propagate stably and confirming these intermediate states effectively isolate accent characteristics for controllable steering.Late-layer steering produces highly unstable, divergent results. For non-native accents, it frequently causes representation collapse; even for native accents, gains do not exceed those of the middle layers. This poor controllability likely stems from increasingly fixed representations, leaving too few layers to reorganize injected directions. Finally, terminal layer 31 consistently causes massive performance degradation, confirming the tail layer is unsuitable for injection.
\vspace{-5pt}
\subsubsection{Steering Strength Analysis}
\vspace{-5pt}
To compare how different $\alpha$ values affect performance, we compute, for both native and non-native accents, the mean $\Delta$WER steering effect under each $\alpha$ injection strength. Figures~\ref{fig:alpha_vctk} and~\ref{fig:alpha_l2arctic} illustrate the resulting $\alpha$-parameter sensitivity trends. We observe that as $\alpha$ increases, the layer-wise steering effect exhibits larger fluctuations and higher improvement peaks. Notably, once the steering strength exceeds a critical threshold, both native and non-native accents show signs of an accelerated collapse starting from around layer 27.
Specifically, for native accents, $\alpha=2$ yields strong steering effects in the late layers with a generally depth-progressive trend, but performs poorly in the mid layers. In contrast, with $\alpha=5$, the mid-layer performance improves substantially, surpassing the late layers and reaching higher peaks, while the late layers display stronger oscillations and clearer collapse behavior. For non-native accents, the overall trend is more consistent: the mid layers constitute the globally optimal intervention window, whereas the collapse in the late layers becomes increasingly pronounced as $\alpha$ grows, with the collapse onset shifting toward earlier layers. Overall, the $\alpha$ sweep further corroborates that the mid layers (15-19) form a shared and most suitable steering window across both native and non-native accents. We therefore suggest that, when performing single-layer accent-direction steering, one should initially try a relatively large $\alpha$ injection strength and, once the improvement magnitude reaches the desired level at a particular $\alpha$, conduct small local adjustments around that value to better trade off peak gains against the risk of window collapse.
% Table~\ref{tab:steering_finetune_compare} presents the comparative results of this fine-tuning approach and our proposed steering method.
% Our comparative analysis reveals a critical relationship between the available training sample size and the chosen adaptation method. The fine-tuning approach demonstrates its primary advantage when a large volume of training data is accessible. For instance, the baseline method achieves superior error reduction on the Arabic, Hindi, and Spanish sets, which each contain approximately 800 training pairs. Conversely, this parameter updating technique struggles severely in data-scarce scenarios. When applied to the South African, Canadian, Northern Irish, and Irish accents, which offer fewer than 100 samples, the fine-tuning performance is remarkably poor. In stark contrast, the steering method excels in these limited sample conditions. Our approach realizes massive performance improvements using very few sample pairs. Furthermore, because this steering mechanism requires no parameter updates, it strictly prevents any damage to the original model capabilities. This characteristic provides a highly efficient and flexible solution for real-time accent adaptation during inference.
\begin{table}[t]
\centering
\scriptsize
\caption{Comparison of steering and fine-tuning across accents. Lower WER is better. The Train column reports the exact number of available training speech pairs for PEFT. The Base metric represents the initial error rate of the unadapted model. Subsequently, the Steer and PEFT columns denote the recognition error rates after applying our proposed steering method and the parameter fine-tuning approach.}
\vspace{-2mm}
\label{tab:steering_finetune_compare}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccc}
\toprule
\toprule
Accent & Train & Base & Steer & PEFT & St.\,$\Delta$ & PEFT\,$\Delta$ \\
\midrule
Scottish & 197 & 26.72\% & 6.80\% & 9.25\% & \textbf{-19.92\%} & -17.47\% \\
S. Afr. & 44 & 29.86\% & 4.35\% & 27.10\% & \textbf{-25.51\%} & -2.76\% \\
Canadian & 51 & 37.27\% & 3.47\% & 32.60\% & \textbf{-33.80\%} & -4.67\% \\
N. Irish & 49 & 36.27\% & 6.64\% & 31.57\% & \textbf{-29.63\%} & -4.70\% \\
Irish & 87 & 31.91\% & 6.41\% & 30.28\% & \textbf{-25.50\%} & -1.63\% \\
\midrule
Arabic & 802 & 18.13\% & 10.07\% & 7.20\% & \textbf{-8.06\%} & -10.93\% \\
Hindi & 790 & 14.26\% & 10.22\% & 7.82\% & \textbf{-4.04\%} & -6.44\% \\
Spanish & 796 & 15.31\% & 9.39\% & 8.61\% & \textbf{-5.92\%} & -6.70\% \\
\bottomrule
\bottomrule
\end{tabular}}
\vspace{-6mm}
\end{table}
\vspace{-5pt}
\subsection{Comparable study with PEFT Baseline}
% \subsection{LoRA Fine-Tuning Baseline}
% We use LoRA as a conventional PEFT baseline for comparison. The LoRA adapters are configured with rank $r=16$, $\alpha=16$, and dropout $=0.05$, and are applied to the \texttt{q\_proj} and \texttt{v\_proj} modules across all 32 layers of the audio encoder. Training is conducted for 3 epochs with an effective batch size of 16 and a learning rate of $5\times10^{-5}$. \td{this should be in the experimental setup.}
% , using a linear decay schedule with 5\% warmup and bf16 precision.
% To ensure a fair comparison, we strictly control the amount of training data exposed to the LoRA adapters, so that the number of unique utterances used for fine-tuning exactly matches the number used to compute the mean-shift vectors. For any accent group evaluated with $n$-fold cross-validation, we use the same $n$-fold training splits for LoRA and report the final results after averaging across folds.
Table 1 compares the fine-tuning approach with our proposed steering method. Analysis reveals a critical link between training sample size and adaptation efficacy. Fine-tuning excels with large datasets, such as the Arabic, Hindi, and Spanish sets containing approximately 800 training pairs. Conversely, parameter-updating techniques struggle in data-scarce scenarios. For accents with fewer than 100 samples—including South African, Canadian, Northern Irish, and Irish—fine-tuning performance is remarkably poor. In contrast, steering excels under these constraints, achieving 4.04 to 33.80 percentage-point WER reductions (28.3\%–90.7\% relative) across eight accents using very few samples. Furthermore, as steering requires no parameter updates, it preserves original model capabilities, providing an efficient, flexible solution for inference adaptation.
% Our approach realizes massive performance improvements using very few sample pairs.
% \td{give the quantitive improvmeent numbers. e,g., xx\% improvement.}
% Furthermore, because this steering mechanism requires no parameter updates, it strictly prevents any damage to the original model capabilities. This characteristic provides a highly efficient and flexible solution for real-time accent adaptation during inference.
\section{Conclusion}
In this paper, we propose a lightweight method to mitigate accent variability in automatic speech recognition. Initially, our analysis identified the middle layers(15-19) in the audio encoder of Large Audio LMs as the optimal region for targeted intervention
% \td{this is not very scientific finding. you want to highlight that these layers are more sensitive to the accent vacations in the latent space. }.
Based on this geometric insight, we introduced a parameter-free adaptation technique using mean shift steering vectors. Consequently, this approach significantly reduces recognition errors across diverse accents. Furthermore, it vastly outperforms conventional fine-tuning in data-scarce environments without requiring weight updates. Ultimately, this research provides a highly scalable pathway for developing inclusive speech technologies.