First-Token Broadcasters: Mechanistic Origins of Language Identity and Distributed Robustness in Transformers
Extracted source text · arXiv 2606.22361
\documentclass[11pt]{article}
\usepackage[preprint]{acl}
\usepackage{times}
\usepackage{latexsym}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{microtype}
\usepackage{inconsolata}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{CJKutf8}
\usepackage[most]{tcolorbox}
\usepackage{enumitem}
\usepackage{float}
\title{First-Token Broadcasters: Mechanistic Origins of Language\\
Identity and Distributed Robustness in Transformers}
\author{
\textbf{Arjun Pillai}$^1$, \textbf{Christian Hoang}$^2$, \textbf{Anjelo Jann Laroza}$^3$ \\
$^1$Irvington High School, \quad $^2$GenAI4E, \quad $^3$Mapua University \\
}
\begin{document}
\maketitle
\begin{abstract}
Why do multilingual language models sometimes generate in the wrong
language, and why is this so hard to fix? We introduce \textbf{Language
Identity Head Ablation (LIHA)}, a causal intervention that zeros each
attention head individually and measures the resulting language switch
rate across a parallel dataset of 2,700 prompt-language pairs spanning
seven languages. Applied to GPT-2, LIHA identifies a small set of
\emph{first-token broadcaster} heads---led by L6H1 (switch rate 0.32,
3.23$\sigma$ above the population mean)---that attend persistently to
the first prompt token, propagating its language signal throughout
generation. Compensatory redistribution when heads are ablated is
statistically significant ($p < 10^{-5}$) and follows a directional,
hierarchical pattern: compensation always recruits heads in layers above
the ablated head, suggesting a feedforward cascade rather than global
diffusion. To probe how training regime shapes these circuits, we apply
LIHA to a controlled pair---Qwen2.5-1.5B-Base and
Qwen2.5-1.5B-Instruct---identical in architecture and size, differing
only in training. The base model is nearly flat (max SR$=$0.016,
200/336 heads at SR$=$0.0); the instruct model concentrates causal
influence sharply at layer~0, led by L0H5 (SR$=$0.224,
8.93$\sigma$ above mean), with all other layers near zero. This
controlled comparison provides direct causal evidence that instruction
tuning reorganizes language identity circuits toward early-layer
localization. Extended experiments with Chinese and Russian confirm
that first-token broadcasting is script-specific in GPT-2, with
non-Latin languages handled at layer~0---the same locus as the
instruction-tuned model.
Code and data will be released upon publication.
\end{abstract}
\section{Introduction}
% Remove the findings, I think its duplicated with the contribution part
% Remove a redundant paragraph
Language identity confusion---where a model given a French prompt
continues in English, or drifts mid-generation into the wrong
language---has been documented across many models
\citep{wendler2024, zhang2023, marchisio-etal-2024-understanding},
but its internal mechanism is not well understood. Prior work has found
neurons and heads whose activations \emph{correlate} with language
\citep{tang2024, kojima2024}, but correlation does not imply causation.
We take a different approach: for each attention head in a model, we
zero its output during generation and directly measure how often the
output language changes. This causal lens reveals not just which heads
matter, but \emph{why} language confusion is so hard to fix---and how
dramatically the answer changes across training regimes.
Our main contributions for this paper are:
\begin{enumerate}[noitemsep]
\item A causal, head-level analysis of language identity across three model families (LIHA, Eq.~\ref{eq:switch_rate}), complementing prior correlational work \citep{tang2024, kojima2024};
\item Discovery of \emph{first-token broadcasting} as a concrete interpretable mechanism;
\item Statistically validated \emph{hierarchical compensatory redistribution}: $p < 10^{-5}$, always downstream, never upstream;
\item The first controlled base-vs.-instruct evidence that instruction tuning reorganizes language identity circuits toward early-layer localization, using models identical in architecture and size.
\end{enumerate}
% Our main findings are:
% (1)~\textbf{Partial localization in GPT-2}---L6H1 leads at
% SR$=$0.32 (3.23$\sigma$ above mean), with a second tier at 0.28
% (\S\ref{sec:ablation}).
% (2)~\textbf{First-token broadcasting}---top heads attend to the first
% prompt token with mean weight 0.85--0.94 at every step
% (\S\ref{sec:attention}).
% (3)~\textbf{Hierarchical compensatory redistribution}---statistically
% significant ($p < 10^{-5}$) and directional: compensation always
% recruits heads in layers \emph{above} the ablated head
% (\S\ref{sec:multi}).
% (4)~\textbf{Convergent probing}---peak layer decodability aligns with
% top causal head layers: 96\% (GPT-2), 100\% (BLOOM)
% (\S\ref{sec:probing}).
% (5)~\textbf{Instruction tuning reorganizes the circuit}---in a
% controlled base-vs.-instruct comparison (Qwen2.5-1.5B), the instruct
% model shows 14$\times$ higher max SR and sharp layer-0 concentration
% (L0H5, 8.93$\sigma$) versus a near-flat base model (\S\ref{sec:qwen}).
% (6)~\textbf{Language-family specificity}---the GPT-2 broadcasting
% mechanism does not extend to Chinese or Russian, which use layer-0
% heads---the same locus as the instruction-tuned model (\S\ref{sec:extended}).
\section{Related Work}
\paragraph{Mechanistic interpretability.}
\citet{elhage2021} established the residual stream framework for component-level causal analysis of transformers. \citet{wang2023} traced indirect object identification in GPT-2 to a sparse 26-head circuit; \citet{conmy2023} introduced automated circuit discovery;
\citet{meng2022} localized factual associations to MLP modules.
\paragraph{Multilingual representations and language confusion.}
\citet{conneau2020} showed multilingual BERT acquires cross-lingual structure without explicit supervision; \citet{wendler2024} found LLMs process inputs
in English internally---consistent with our finding that English is immune to ablation. \citet{marchisio-etal-2024-understanding} demonstrate that state-of-the-art models frequently switch to unintended languages; we
approach this mechanistically via causal head ablation.
\paragraph{Language-specific components, probing, and steering.}
\citet{tang2024} identified language-specific neurons; \citet{kojima2024} found heads preferentially processing specific language families. Activation analysis identifies language-\emph{selective} components;
ablation identifies language-\emph{causal} ones---sets that are not identical.
Linear probing \citep{alain2016, tenney2019} provides our second line of
evidence. \citet{zou2023} and \citet{li2023} demonstrated inference-time
steering; our negative amplification results distinguish language identity
from more localized steerable properties.
% Create a new section to emphasize the contribution
\section{Proposed Method}
Let $\mathcal{M}$ be a transformer with $L$ layers and $H$ heads per
layer. For head $h_{l,i}$ at layer $l$, head index $i$, define the
ablated model $\mathcal{M}^{-h_{l,i}}$ as $\mathcal{M}$ with that
head's residual stream contribution set to zero. The
\textbf{switch rate} of head $h_{l,i}$ over prompt set
$\mathcal{X} = \{x_1, \ldots, x_n\}$ is:
\begin{equation}
S(h_{l,i}) = \frac{1}{n} \sum_{j=1}^{n}
\mathbf{1}\!\left[\mathcal{L}(\mathcal{M}^{-h_{l,i}}(x_j)) \neq
\mathcal{L}(\mathcal{M}(x_j))\right]
\label{eq:switch_rate}
\end{equation}
A head with high $S(h_{l,i})$ is causally responsible for language
identity: removing it reliably changes output language.
\section{Experimental Setup}
\label{sec:setup}
\paragraph{Models and implementation.}
% Add the BLOOM model information, rewrite the word
Our experiments are conducted across diverse language model architectures to ensure cross-model generalizability. We evaluate \mbox{GPT-2 small} \citep{radford2019}, which features 12 layers and 12 attention heads per layer ($d_{\text{model}} = 768$), totaling 144 heads. To investigate the effects of scale and training regimes, we analyze Qwen2.5-1.5B \citep{qwen2025}, comprising 28 layers with a Grouped-Query Attention (GQA) mechanism configured with 12 query heads and 2 key-value heads, totaling 336 heads; we compare its base and instruct variants, which share an identical architecture and differ solely in their training optimization. Finally, to capture insights from a natively multilingual model family, we incorporate BLOOM-560m \citep{scao2022bloom}, featuring 24 layers with 16 attention heads per layer ($d_{\text{model}} = 1024$), totaling 384 heads.
Operating within the residual stream framework \citep{elhage2021}, where each attention head contributes additively to the hidden states, we isolate individual head dynamics by zeroing out their post-projection outputs. Mechanistically, this ablation is implemented via PyTorch forward hooks registered on the output projection matrix (\texttt{o\_proj}). For evaluation, text generation is performed using greedy decoding with a fixed budget of 40 tokens per prompt, and downstream language classification is determined via \texttt{langdetect} \citep{shuyo2010}.
\paragraph{Dataset.}
We construct a parallel multilingual dataset drawn from Flores-200
\citep{flores200}, spanning five European languages (English, French,
German, Spanish, Italian; 500 sentences each for GPT-2; 25 each for
Qwen), Chinese (100 sentences), and Russian (100 sentences), yielding
2,700 prompt-language pairs in total. The first 5 prompts per language
are hand-written sentence starters; the remainder are from the Flores-200
devtest split, filtered for quality (20--300 characters, $>$50\%
alphabetic) and verified via 3-way majority vote (\texttt{langdetect},
\texttt{langid}, fastText). All European languages achieved
$\geq$99.6\% verified prompts.
\paragraph{Protocols.}
GPT-2 switch rates are computed on the full 500-prompt-per-language
dataset (2,500 European prompts); multi-head ablation accuracy on the
same set (1-percentage-point granularity). Switch rate measures
\emph{change from baseline output}; accuracy measures \emph{absolute
correctness}---not circularly defined. We sweep all 144 GPT-2 heads
and all 336 Qwen2.5-1.5B heads (both variants). For BLOOM we sweep
every third layer (128 heads). To test steerability, we scale top head
outputs by $2\times$, $3\times$, and $5\times$ \citep{li2023}. All
switch rates report 95\% bootstrap CIs (10,000 samples). Compensatory
redistribution significance uses 100,000-sample permutation tests on
first-token attention weight deltas. Zero-ablation is the standard
intervention in mechanistic interpretability \citep{elhage2021,
wang2023}; we discuss mean ablation as an alternative and its
limitations for English-dominant models in \S\ref{sec:limitations}.
% Group all results into one section
\section{Single-Model Results}
\label{sec:single-model}
\subsection{Single-Head Ablation (GPT-2)}
Figure~\ref{fig:heatmap} shows the language switch rate across all
144 GPT-2 heads. The map is sparse, with elevated rates in a cluster
spanning layers 0--10.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{fig1_ablation_heatmap.pdf}
\caption{\textbf{Language switch rate across all 144 GPT-2 attention
heads.} A small cluster spanning layers 0--10 shows disproportionate
causal influence. L6H1 leads at 0.32 (99.3rd percentile,
3.23$\sigma$ above mean); a second tier (L0H4, L3H1, L9H9) sits
at 0.28.}
\label{fig:heatmap}
\end{figure}
The mean switch rate is 0.163 ($\sigma{=}0.049$). L6H1 leads at
SR$=$0.32 (99.3rd percentile, 3.23$\sigma$ above mean, 1.97$\times$
effect size), the highest-ranked head by both switch rate and effect
size. A second tier (L0H4, L3H1, L9H9) sits at 0.28; given the sample
size at this prompt count, the 95\% bootstrap CIs for these heads
overlap with L6H1's (Table~\ref{tab:top_heads}), so we treat L6H1 as
the leading candidate head rather than as statistically separated from
this tier. Table~\ref{tab:top_heads} reports the top five GPT-2 heads.
\begin{table}[t]
\caption{\textbf{Top five GPT-2 heads by switch rate} with 95\%
bootstrap CIs (500 prompts per language).}
\label{tab:top_heads}
\centering\small
\begin{tabular}{lcc}
\toprule
\textbf{Head} & \textbf{Switch Rate} & \textbf{95\% CI} \\
\midrule
L6H1 & 0.32 & {[0.16, 0.52]} \\
L0H4 & 0.28 & {[0.12, 0.44]} \\
L3H1 & 0.28 & {[0.12, 0.48]} \\
L9H9 & 0.28 & {[0.12, 0.48]} \\
L7H3 & 0.24 & {[0.08, 0.40]} \\
\midrule
Population mean & 0.163 & -- \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{Per-language breakdown.}
English is entirely immune: baseline accuracy is 99.2\% and remains
99.2\% under L6H1 ablation. Italian is most vulnerable (switch rate
0.6 under L6H1 ablation; see Appendix~\ref{app:perlang}), consistent
with GPT-2's English-dominant training.
\subsection{Redundancy and Amplification}
Figure~\ref{fig:accuracy_curve} shows language accuracy under
progressive multi-head ablation on the full 2,500-prompt European set.
Baseline accuracy is 43.5\%~[0.416, 0.454]---lower than the 25-prompt
pilot (76\%), correctly reflecting GPT-2's difficulty with French (6.6\%)
and Italian (20.4\%) at scale; switch rate results are unaffected.
Ablating L6H1 alone ($k{=}1$) reduces accuracy to 39.2\%~[0.373,
0.411], a 4.3-point directional decline; the overlapping CIs are
expected under distributed encoding, where no single head accounts for
a large share of the signal. Ablating the top ten heads reduces
accuracy to 32.4\%~[0.306, 0.343], an 11.1-point total reduction.
Critically, accuracy \textbf{never falls below chance (20\%)} and the
degradation is monotonic---the signature of redundant distributed
encoding \citep{wang2023}.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{fig3_accuracy_curve.pdf}
\caption{\textbf{Language accuracy under progressive multi-head
ablation} (2,500-prompt evaluation, 95\% bootstrap CIs). Monotonic
gradual degradation---never reaching chance---is the signature of
redundant distributed encoding. Dotted line: chance (0.20).}
\label{fig:accuracy_curve}
\end{figure}
\paragraph{Compensatory redistribution.}
We test whether compensation is significant via 100,000-sample
permutation tests on first-token attention weight deltas. The top-5
mean delta exceeds the permutation null for all three ablated heads
($p < 10^{-5}$). Individual compensators are significant: ablating
L6H1, L9H8 shows $z{=}3.45$ ($p{=}0.0006$); ablating L0H4, L2H3
shows $z{=}3.97$ ($p{<}0.0001$). Table~\ref{tab:compensatory} reports
the L6H1 compensators.
\begin{table}[t]
\caption{\textbf{Top compensating heads when L6H1 is ablated.} All
individually significant ($p{<}0.05$). Compensators cluster in
layers 7--9, above the ablated layer~6 ($p{=}0.002$).}
\label{tab:compensatory}
\centering\small
\begin{tabular}{lrrrr}
\toprule
\textbf{Head} & \textbf{Base} & \textbf{Ablated} & $\boldsymbol{\Delta}$ & $\boldsymbol{z}$ \\
\midrule
L9H8 & 0.267 & 0.275 & $+$0.007 & $3.45^{**}$ \\
L7H3 & 0.745 & 0.751 & $+$0.006 & $3.01^{**}$ \\
L7H10 & 0.873 & 0.879 & $+$0.006 & $2.70^{**}$ \\
L7H1 & 0.833 & 0.838 & $+$0.005 & $2.65^{**}$ \\
L7H6 & 0.702 & 0.705 & $+$0.003 & $2.10^{*}$ \\
\bottomrule
\end{tabular}
\smallskip
{\footnotesize $^{**}p{<}0.01$;\enspace$^{*}p{<}0.05$;\enspace permutation test, $n{=}100{,}000$.}
\end{table}
This redistribution follows a \textbf{directional, hierarchical}
pattern: in all three ablation conditions, compensators concentrate in
layers \emph{above} the ablated head (layer-clustering $p < 0.006$).
Ablating L0H4 (layer~0) recruits layer~2; ablating L6H1 (layer~6)
recruits layers~7--9; ablating L9H9 (layer~9) recruits layers~10--11.
This consistent forward directionality suggests the language signal
propagates through the residual stream as a feedforward cascade.
\paragraph{Amplification.}
Amplifying top heads by $2\times$, $3\times$, or $5\times$
\citep{li2023} produces no accuracy improvement at any scale,
confirming that the redundancy is not overcome by increasing a single
head's contribution.
\subsection{First-Token Broadcasting}
Figure~\ref{fig:attn_comparison} shows L6H1's attention on correct and
confused prompts: every query position attends to the first token with
weights 0.62--1.00. Table~\ref{tab:taxonomy} introduces our functional
taxonomy.
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{fig_attention_comparison.pdf}
\caption{\textbf{L6H1 attention on a correct French prompt (upper)
and confused Italian prompt (lower).} First-token focus is consistent
in both cases; confusion arises downstream.}
\label{fig:attn_comparison}
\end{figure}
\begin{table}[t]
\caption{\textbf{Functional taxonomy of top language-critical heads.}}
\label{tab:taxonomy}
\centering\small
\begin{tabular}{llcc}
\toprule
\textbf{Head} & \textbf{Type} & \textbf{First-Tok Attn} & \textbf{Entropy} \\
\midrule
L6H1 & Broadcaster & 0.876 & 0.517 \\
L10H4 & Broadcaster & 0.848 & 0.677 \\
L7H3 & Broadcaster & 0.892 & 0.538 \\
L1H10 & Distributed & 0.112 & 2.013 \\
\bottomrule
\end{tabular}
\end{table}
Broadcaster heads attend to the first prompt token with mean weight
0.85--0.89 at every generation step. Entropy analysis confirms this
is stable across all 40 steps (mean entropy 1.11 vs.\ 1.34 for random
heads; Appendix~\ref{app:entropy}). Confused prompts show \emph{higher}
first-token attention in L6H1 (0.923 vs.\ 0.847 on correct prompts):
confusion does not arise from a broadcasting failure but from downstream
representational weakness. We note that first-token broadcasting
describes a correlate of language identity maintenance rather than a
complete mechanistic account. Path patching \citep{wang2023}---to
distinguish heads promoting the target language from those suppressing
alternatives---remains an important next step.
\subsection{Convergent Evidence from Probing}
In GPT-2, probing accuracy rises from 36\% at the embedding layer to
96\% at layers~10--11. In BLOOM, it reaches 100\% at layer~18 through
layer~24. Layers containing top causal heads show substantially higher
probing accuracy in both models (GPT-2: 85\% vs.\ 58\%, 27-point gap;
BLOOM: top-switch layers all above 92\%), providing convergent causal
and representational evidence for the same layers.
\section{Cross-Model Comparisons}
\label{sec:cross-model}
To directly test whether training regime reorganizes language identity
circuits, we apply LIHA to Qwen2.5-1.5B-Base and
Qwen2.5-1.5B-Instruct---models with identical architecture, size, and
GQA configuration (2 KV heads, 12 Q heads, 28 layers, 336 total heads),
differing only in whether instruction tuning was applied. Both are swept
fully across all 336 heads with 125 prompts (25 per language).
\paragraph{Base model: near-uniform flatness.}
The base model shows nearly uniform low causal influence across all
layers and heads: mean SR$=$0.0036 ($\sigma{=}0.0046$), maximum
SR$=$0.016 (L0H0, $z{=}2.70$), and 200 of 336 heads at SR$=$0.0.
No head exceeds SR$=$0.1. This near-flat distribution indicates that
language identity is not strongly localized in any individual head of
the pretrained model, consistent with distributed encoding.
\paragraph{Instruct model: sharp layer-0 concentration.}
The instruct model shows a qualitatively different organization. L0H5
leads at SR$=$0.224 [0.152, 0.296], 8.93$\sigma$ above the population
mean of 0.026 ($\sigma{=}0.022$)---a 14$\times$ increase in max SR
over the base model. All 12 layer-0 heads are elevated above the
population mean (layer-0 mean SR$=$0.093 vs.\ 0.023 for layers~1--27,
a 4.1$\times$ difference). Table~\ref{tab:qwen_top} reports the top
five instruct heads with 95\% bootstrap CIs.
\begin{table}[t]
\caption{\textbf{Top five Qwen2.5-1.5B-Instruct heads} by switch rate
(125 prompts, 95\% bootstrap CIs). English is immune across all heads.}
\label{tab:qwen_top}
\centering\small
\begin{tabular}{lccc}
\toprule
\textbf{Head} & \textbf{SR} & \textbf{95\% CI} & \textbf{$\sigma$ above} \\
& & & \textbf{mean} \\
\midrule
L0H5 & 0.224 & {[0.152, 0.296]} & 8.93 \\
L0H11 & 0.144 & {[0.088, 0.208]} & 5.33 \\
L1H9 & 0.136 & {[0.080, 0.200]} & 4.97 \\
L0H7 & 0.120 & {[0.064, 0.176]} & 4.25 \\
L1H7 & 0.112 & {[0.064, 0.168]} & 3.89 \\
\midrule
Population mean & 0.026 & -- & -- \\
\bottomrule
\end{tabular}
\end{table}
The instruct model is not fully localized to a single head: six heads
exceed SR$=$0.1, and secondary peaks appear at layers~17 (L17H4,
SR$=$0.080) and~21 (L21H3, SR$=$0.096). However, layer~0 is
structurally dominant and the contrast with the flat base model is
unambiguous. English is immune in both variants (\texttt{switch\_en}$=$0.0
for all top heads), and German is most vulnerable (\texttt{switch\_de}
up to 0.44), replicating the language-specificity pattern observed
in GPT-2.
\paragraph{Cross-model comparison.}
Table~\ref{tab:crossmodel} summarizes all models. The progression from
GPT-2 (distributed, middle layers) to Qwen-Base (near-uniform) to
Qwen-Instruct (layer-0 concentrated) reveals that instruction tuning
is the causally active ingredient in circuit reorganization, since
architecture and scale are held constant in the Qwen comparison. BLOOM
is included as a reference point illustrating maximal redundancy under
balanced multilingual pretraining.
\begin{table*}[t]
\caption{\textbf{Cross-model comparison of language identity
organization.} The Qwen base/instruct pair isolates the effect of
instruction tuning. BLOOM is shown as a redundancy reference.}
\label{tab:crossmodel}
\centering\small
\begin{tabular}{lcccc}
\toprule
\textbf{Property} & \textbf{GPT-2} & \textbf{BLOOM-560m} & \textbf{Qwen-1.5B Base} & \textbf{Qwen-1.5B Instruct} \\
\midrule
Heads swept & 144 & 128$^{\dagger}$ & 336 & 336 \\
Max SR & 0.32 & 0.16 & 0.016 & 0.224 \\
Top head $\sigma$ & 3.23 & 2.60 & 2.70 & 8.93 \\
\# heads SR$>$0.1 & 15 & 4 & 0 & 6 \\
\# heads SR$=$0 & -- & -- & 200 & 19 \\
Critical layers & 0--10 & 0--21 & none & 0 \\
English immune & YES & YES & YES & YES \\
\bottomrule
\end{tabular}
\smallskip
{\footnotesize $^{\dagger}$Every 3rd layer sampled; 128 of 384 heads.}
\end{table*}
\subsection{Generalization to Typologically Distant Languages}
All GPT-2 experiments used Indo-European languages in Latin script.
We extend to Chinese (zh) and Russian (ru). Table~\ref{tab:extended}
shows that L6H1, L0H4, L3H1, and L9H9---with European switch rates
of 0.28--0.32---produce SR$=$0.0 on both Chinese and Russian. The
first-token broadcasting mechanism is entirely inert for
typologically distant languages. Different heads, concentrated in
layers~0--4, govern non-European languages---the same layer-0 locus
as the instruction-tuned Qwen model. Notably, L1H10---classified as a
\emph{distributed} (non-broadcaster) head in Table~\ref{tab:taxonomy}---retains
a non-trivial switch rate on both Chinese (0.20) and Russian (0.20),
where the European broadcasters (L6H1, L0H4, L3H1, L9H9) drop to zero
(Table~\ref{tab:extended}). This suggests that distributed heads may
carry more cross-lingually general language information than the
script-specific first-token broadcasters.
\begin{table}[t]
\caption{\textbf{Language-family specificity.} Left: top European heads
show SR$=$0 on Chinese/Russian. Right: different heads govern
non-European languages, concentrated in layers 0--4.}
\label{tab:extended}
\centering\tiny
\begin{tabular}{lccc|lccc}
\toprule
\textbf{Head} & \textbf{EU} & \textbf{ZH} & \textbf{RU}
& \textbf{Head} & \textbf{All} & \textbf{ZH} & \textbf{RU} \\
\midrule
L6H1 & 0.32 & 0.00 & 0.00 & L0H0 & 0.40 & 0.20 & 0.60 \\
L0H4 & 0.28 & 0.00 & 0.00 & L4H5 & 0.30 & 0.40 & 0.20 \\
L3H1 & 0.28 & 0.00 & 0.00 & L4H2 & 0.30 & 0.40 & 0.20 \\
L9H9 & 0.28 & 0.00 & 0.00 & L0H1 & 0.20 & 0.00 & 0.40 \\
L1H10 & 0.24 & 0.20 & 0.20 & L0H7 & 0.20 & 0.20 & 0.20 \\
\bottomrule
\end{tabular}
\end{table}
\section{Discussion}
\label{sec:discussion}
\paragraph{A training-regime theory of language identity circuits.}
Our experiments reveal a clear progression: language identity
organization changes systematically with training regime, and the
Qwen base-vs.-instruct comparison isolates instruction tuning as
the causal factor. \emph{Pretrained-only models} (GPT-2) distribute
the language signal across 10+ heads in middle layers---redundant and
resistant to intervention. \emph{Deliberately multilingual pretrained
models} (BLOOM) show even greater redundancy, making single-head
intervention essentially ineffective (0\% accuracy drop from top-10
ablation). \emph{Instruction-tuned models} (Qwen-Instruct) reorganize
toward layer-0 concentration, with the base model of identical
architecture remaining flat---directly implicating the instruction
tuning process rather than architecture or scale.
\paragraph{The layer-0 convergence.}
A striking pattern runs through three independent findings: Chinese and
Russian in GPT-2 use layer-0 heads; Qwen-Instruct concentrates at
layer~0; and the Qwen base model, while flat, has its single strongest
head also at layer~0 (L0H0, SR$=$0.016). Layer~0 is closest to the
tokenizer: token IDs in Cyrillic, Chinese characters, or
instruction-formatted text carry strong script-level signals
immediately. Models that resolve language identity early---either
because the tokens are unambiguous (non-Latin scripts) or because
training has made language-following a primary objective (instruction
tuning)---converge on layer~0 as the locus of causal influence.
\paragraph{Why inference-time interventions fail (in GPT-2).}
Ablating ten heads causes only 11.1 points of monotonic accuracy decline,
never reaching chance---compensation is statistically significant
($p{<}10^{-5}$) and hierarchically absorbed downstream.
Qwen-Instruct's layer-0 concentration suggests instruction-tuned models
may be more amenable to targeted intervention \citep{zou2023}, since
the signal is less redundantly distributed.
\paragraph{Future directions.}
Path patching \citep{wang2023} would distinguish heads that promote
the target language from those that suppress alternatives. The
directional redistribution pattern (Table~\ref{tab:compensatory})
suggests a feedforward cascade structure worth characterizing precisely.
Whether the layer-0 concentration in Qwen-Instruct extends to larger
instruction-tuned models and whether it enables more reliable language
control are important open questions. While this work maps the language identity circuits within text-based transformers, an important direction for future research is validating these mechanistic dynamics in speech-driven, end-to-end multimodal architectures \citep{pham2025silvar}.
\section{Conclusion}
\label{sec:conclusion}
In this paper, we introduced Language Identity Head Ablation (LIHA) to causally investigate how multilingual models maintain language identity during generation. Applied to GPT-2, LIHA uncovers a distributed network of first-token broadcaster heads that exhibit robust, hierarchical downstream compensation when ablated. Crucially, through a controlled comparison of Qwen2.5-1.5B variants, we provide the first causal evidence that instruction tuning fundamentally reorganizes these circuits, shifting language identity localization from distributed middle layers to the earliest layer. These findings offer a mechanistic explanation for language identity maintenance and highlight how training regimes explicitly shape internal model circuitry, opening new avenues for targeted inference-time interventions and multilingual robustness.
\section{Limitations}
\label{sec:limitations}
\paragraph{Dataset and statistical power.}
GPT-2 switch rates use 500 prompts per European language; CIs remain
moderately wide (L6H1: [0.16, 0.52]). Qwen switch rates use 125
prompts; the top head CI [0.152, 0.296] is sufficient for the
base-vs.-instruct contrast but would benefit from expansion. At
$n{=}125$, the minimum resolvable nonzero switch rate is
$1/125 \approx 0.008$, so the large number of Qwen-Base heads at
SR$=$0.0 (200/336) partly reflects measurement resolution rather than
a guarantee of exactly zero causal effect; a larger-scale sweep of the
base model would strengthen the flatness claim in \S\ref{sec:cross-model}.
\paragraph{Ablation methodology.}
Zero-ablation cannot distinguish heads promoting the correct language
from those suppressing alternatives. Compensatory redistribution tests
use summary statistics (means over prompts) rather than per-prompt
observations, making the test conservative but appropriate.
We additionally attempted mean ablation---replacing each head's output
with its mean activation across the dataset---as a methodological
check. The resulting switch rates were uncorrelated with zero-ablation
rankings (Spearman $\rho{=}{-}0.24$), a pattern we attribute to
GPT-2's English-dominant training: the mean activation of
language-critical heads encodes an English prior rather than a
language-neutral baseline \citep{wendler2024}, so mean ablation
effectively substitutes one language signal (English) for another
(the prompt language) without removing language information altogether.
This confirms that zero-ablation is the appropriate intervention for
detecting language-causal heads in English-dominant models, and is
consistent with its standard use throughout the mechanistic
interpretability literature \citep{elhage2021, wang2023, conmy2023}.
\paragraph{GQA caveat.}
Qwen2.5-1.5B uses Grouped Query Attention. Zeroing a Q-head slice in
GQA has different internal semantics than in standard MHA, since
multiple Q-heads share the same KV projections. We hook \texttt{o\_proj}
output for methodological consistency across architectures, but
head-level ablation in GQA requires further validation.
\paragraph{Model scope.}
GPT-2 small and BLOOM-560m are established mechanistic interpretability
testbeds \citep{elhage2021, wang2023, conmy2023}. The Qwen comparison
controls for architecture and scale but a single model family; further
replication across instruction-tuned model families is needed to
generalize the training-regime theory.
\section*{Ethical Considerations}
Understanding language identity mechanisms could inform more reliable
multilingual systems. The redundancy we document acts as an incidental
robustness property against adversarial language-switching attacks.
All experiments use publicly available pretrained models and constructed
prompts; no personal data was used.
\section*{Acknowledgments}
Omitted for anonymous review.
\bibliography{references}
\clearpage
\appendix
\section{Entropy Over Generation Time}
\label{app:entropy}
\begin{figure}[H]
\centering
\includegraphics[width=\columnwidth]{fig_entropy_over_time.pdf}
\caption{\textbf{Attention entropy over generation steps.}
Broadcaster heads (solid) maintain lower entropy (mean 1.11) than
random heads (dashed, mean 1.34) across all 40 steps.}
\label{fig:entropy}
\end{figure}
\section{Bootstrap and Permutation Test Details}
\label{app:bootstrap}
Switch rate CIs use 10,000 bootstrap samples (resampling with
replacement, 2.5th/97.5th percentiles). Compensatory redistribution
uses 100,000-sample permutation tests: the observed top-5 mean delta
is compared against the distribution of top-5 means from 100,000
random draws of size 5 from the full 143-head delta population.
Layer-clustering tests compare observed layer entropy of top-5
compensators against the entropy from 100,000 random 5-head draws.
\section{Full GPT-2 Ablation Sweep}
\label{app:full_sweep}
\begin{table}[H]
\caption{All GPT-2 heads with switch rate $> 0.15$.}
\label{tab:full_sweep}
\centering\small
\begin{tabular}{lcc}
\toprule
\textbf{Head} & \textbf{Switch Rate} & \textbf{Delta Acc.} \\
\midrule
L6H1 & 0.32 & $+$0.04 \\
L0H4 & 0.28 & $+$0.04 \\
L9H9 & 0.28 & $-$0.12 \\
L3H1 & 0.28 & $+$0.00 \\
L10H4 & 0.24 & $-$0.08 \\
L9H11 & 0.24 & $-$0.04 \\
L8H6 & 0.24 & $+$0.00 \\
L9H4 & 0.24 & $+$0.00 \\
L7H3 & 0.24 & $-$0.04 \\
L1H0 & 0.24 & $+$0.08 \\
L1H10 & 0.24 & $+$0.04 \\
L5H1 & 0.24 & $+$0.00 \\
L6H8 & 0.24 & $-$0.04 \\
L3H4 & 0.20 & $-$0.04 \\
L4H11 & 0.20 & $+$0.00 \\
\bottomrule
\end{tabular}
\end{table}
\section{Per-Language Switch Rates (GPT-2)}
\label{app:perlang}
\begin{table}[H]
\caption{Per-language switch rates for the top five GPT-2 heads from
Table~\ref{tab:top_heads}, computed on the 5 hand-written
sentence-starter prompts per language (\S\ref{sec:setup}).}
\label{tab:per_language_app}
\centering\small
\begin{tabular}{lccccc}
\toprule
\textbf{Head} & \textbf{EN} & \textbf{FR} & \textbf{DE} & \textbf{ES} & \textbf{IT} \\
\midrule
L6H1 & 0.0 & 0.2 & 0.2 & 0.4 & 0.6 \\
L0H4 & 0.0 & 0.2 & 0.2 & 0.4 & 0.4 \\
L3H1 & 0.0 & 0.2 & 0.2 & 0.4 & 0.4 \\
L9H9 & 0.0 & 0.2 & 0.0 & 0.4 & 0.6 \\
L7H3 & 0.0 & 0.2 & 0.2 & 0.2 & 0.4 \\
\bottomrule
\end{tabular}
\smallskip
{\footnotesize Note: this qualitative breakdown uses the 5 hand-written
prompts per language and therefore differs slightly from the aggregate
switch rates over the full 500-prompt set reported in
Table~\ref{tab:top_heads}.}
\end{table}
\section{Extended Language Prompts}
\label{app:prompts}
\begin{table}[H]
\caption{Chinese and Russian prompts used in the extended language
experiments. Russian is shown in transliteration; Cyrillic originals
are in \texttt{data/prompts\_ru.csv}.}
\label{tab:extended_prompts}
\centering\small
\begin{tabular}{lp{5.5cm}}
\toprule
\textbf{Lang} & \textbf{Prompt} \\
\midrule
zh & \begin{CJK}{UTF8}{gbsn}今天的天气非常\end{CJK} \\
zh & \begin{CJK}{UTF8}{gbsn}我想告诉你关于\end{CJK} \\
zh & \begin{CJK}{UTF8}{gbsn}科学家们发现了\end{CJK} \\
zh & \begin{CJK}{UTF8}{gbsn}生活中最重要的事是\end{CJK} \\
zh & \begin{CJK}{UTF8}{gbsn}从前有一个\end{CJK} \\
\midrule
ru & \textit{Segodnya pogoda ochen\textquotesingle} \\
ru & \textit{Ya khotel by rasskazat\textquotesingle{} vam o} \\
ru & \textit{Uchyonye obnaruzhili, chto} \\
ru & \textit{Samoye vazhnoye v zhizni~--- eto} \\
ru & \textit{Odnazhdy zhil-byl} \\
\bottomrule
\end{tabular}
\end{table}
\end{document}