RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain
Extracted source text · arXiv 2607.05171
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2026
% The authors should use one of these tracks.
% Before accepting by the NeurIPS conference, select one of the options below.
% 0. "default" for submission
% \usepackage{neurips_2026}
% the "default" option is equal to the "main" option, which is used for the Main Track with double-blind reviewing.
% 1. "main" option is used for the Main Track
% \usepackage[main]{neurips_2026}
% 2. "position" option is used for the Position Paper Track
% \usepackage[position]{neurips_2026}
% 3. "eandd" option is used for the Evaluations & Datasets Track
% \usepackage[eandd]{neurips_2026}
% 4. "creativeai" option is used for the Creative AI Track
% \usepackage[creativeai]{neurips_2026}
% 5. "sglblindworkshop" option is used for the Workshop with single-blind reviewing
% \usepackage[sglblindworkshop]{neurips_2026}
% 6. "dblblindworkshop" option is used for the Workshop with double-blind reviewing
% \usepackage[dblblindworkshop]{neurips_2026}
% After being accepted, the authors should add "final" behind the track to compile a camera-ready version.
% 1. Main Track
% \usepackage[main, final]{neurips_2026}
% 2. Position Paper Track
% \usepackage[position, final]{neurips_2026}
% 3. Evaluations & Datasets Track
% \usepackage[eandd, final]{neurips_2026}
% 4. Creative AI Track
% \usepackage[creativeai, final]{neurips_2026}
% 5. Workshop with single-blind reviewing
% \usepackage[sglblindworkshop, final]{neurips_2026}
% 6. Workshop with double-blind reviewing
% \usepackage[dblblindworkshop, final]{neurips_2026}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
% For workshops (5., 6.), the authors should add the name of the workshop, "\workshoptitle" command is used to set the workshop title.
% \workshoptitle{WORKSHOP TITLE}
% "preprint" option is used for arXiv or other preprint submissions
\usepackage[preprint]{neurips_2026}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2026}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{amsmath}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
\title{RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Omer Moussa \\
Max Planck Institute for Software Systems\\
Saarbrücken, Germany\\
\texttt{omoussa@mpi-sws.org} \\
% examples of more authors
\And
Mariya Toneva \\
Max Planck Institute for Software Systems \\
Saarbrücken, Germany \\
\texttt{mtoneva@mpi-sws.org} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
Language understanding in the brain is context-dependent, varying across experimental stimuli and individuals, which makes it difficult to build computational models that generalize across both. This calls for a foundation model of language-evoked brain activity that can capture shared structure while adapting efficiently to new participants and inputs. We introduce RABBiT (Rapidly Adaptive BOLD foundation model via BraIn-Tuning), a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction. A comprehensive evaluation on 324 participants across multiple unseen fMRI datasets shows that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages. With as little as 10 minutes of participant-specific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. RABBiT's performance is driven by two key innovations: (1) learned region-specific attention, and (2) a decomposition of brain responses into shared and subject-specific components, combined with a brain-tuned speech backbone. In addition to supporting strong predictive accuracy, the structured, region-specific representations that RABBiT learns enable interpretability. By eliminating the need for extensive per-participant data and model fitting, RABBiT enables scalable population-level analyses of language in the human brain. We make the code available at \textcolor{blue}{https://github.com/bridge-ai-neuro/rabbit}.
\end{abstract}
\input{1-intro}
\input{2-related_work}
\input{3-methods}
\input{4-results}
\input{5-discussion_conclusion}
\newpage
% \section*{References}
\bibliographystyle{plain}
\bibliography{main}
% \bibliographystyle{unsrtnat}
% \bibliographystyle{plainnat}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \newpage
% \input{checklist.tex}
\appendix
\include{6-appendix}
\end{document}
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.40\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/teaser.pdf}
\caption{RABBiT transfers after training.}
\label{fig:teaser}
\end{subfigure}\hfill
\begin{subfigure}[b]{0.58\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/transformer.pdf}
\caption{Cross-attention module in brain transformer.}
\label{fig:trans}
\end{subfigure}\\[6pt]
\begin{subfigure}[b]{\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/training.pdf}
\caption{Training: predict fMRI per ROI, compare to measured.}
\label{fig:training}
\end{subfigure}
\caption{\textbf{Overview of RABBiT.}
\textbf{(a)}~A single encoder that supports two
transfer regimes for new subjects. \emph{Zero-shot}: predicting group-level responses; tested on 324 unseen participants across 16 held-out studies with naturalistic listening; RABBiT's predictions on auditory ROIs already saturate near the inter-subject consistency. \emph{Few-shot}: adapting to a new subject with small data; RABBiT rapidly and effectively adapts to unseen subjects with only $\sim$10 minutes of paired audio, improving idiosyncratic language ROIs with 3 orders of magnitude fewer parameters than voxel-wise models. \textbf{(b)}~The cross-attention readout: learnable ROI query tokens attend to audio output tokens of the speech backbone, yielding a per-ROI latent $z_i$ that feeds to prediction heads. \textbf{(c)}~Training pipeline. Audio is encoded by a speech model into a temporal token sequence; the cross-attention transformer routes those tokens to ROI latents; the per-ROI latent passes through our SID decomposition (learnable shared + per-subject idiosyncratic components) to form the predicted fMRI response; the loss is applied between predicted and measured fMRI. The whole pipeline has $\approx$12.9M trainable parameters.}
\label{fig:overview}
\end{figure}
\section{Introduction}
\label{sec:intro}
Natural language comprehension evokes brain responses that are both reliable across listeners and strongly shaped by individual variation. Early auditory regions exhibit relatively consistent responses across participants, whereas higher-order language regions vary substantially across brains \citep{nastase2021narratives}. This creates a central challenge for language fMRI modeling: a useful encoder must capture the shared structure of language processing while remaining adaptable to individual listeners.
Most existing brain encoding approaches address only one side of this problem. Standard voxel-wise encoding models fit a separate readout for each participant \citep{huth2016natural,antonello2024scaling}. Although effective with sufficient data, these methods require collecting new fMRI recordings and fitting a high-dimensional model for every participant. More recent brain-tuning approaches improve the underlying speech representations using fMRI supervision \citep{moussa2025improving,moussa2025multibrain,vattikonda2025brainwavlm}, but still rely on participant-specific voxel-wise fitting at inference time. In parallel, recent foundation-style brain encoders improve zero-shot prediction across participants \citep{dAscoli2026TribeV2,beliy2025brainlm}, yet they are not designed for efficient adaptation to new individuals, particularly at high cortical resolution. These limitations motivate a foundation model for language-evoked brain activity: a single encoder that predicts the shared response structure of unseen listeners while supporting rapid adaptation to individual brains from limited calibration data.
We introduce \textbf{RABBiT} (\textbf{R}apidly \textbf{A}daptive \textbf{B}OLD foundation model via \textbf{B}ra\textbf{I}n-\textbf{T}uning; Fig.\ref{fig:overview}), a compact audio-to-fMRI encoder designed for accurate zero-shot predictions and efficient few-shot adaptation to new participants. RABBiT first brain-tunes a self-supervised speech backbone using LoRA adapters, adapting speech representations directly for cortical prediction. A novel \textbf{Temporal Brain Transformer (TBT)} then constructs explicit ROI-level representations by learning region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented voxel-wise processing. Finally, a novel \textbf{Shared--Idiosyncratic Decomposition (SID)} maps each ROI representation to cortical activity through a low-rank shared component and a low-rank participant-specific deviation. The shared component enables accurate zero-shot population prediction, while the deviation pathway supports efficient few-shot adaptation for a new participant with only a small number of trainable parameters.
%This design makes RABBiT a direct test of whether individual variation in language cortex is structured enough to be modeled, rather than relearned from scratch for every participant. If the shared pathway captures reliable speech-evoked structure, zero-shot predictions should approach the inter-subject reliability limit. If SID isolates meaningful subject-specific variation, few-shot calibration should improve the regions where the group response is least sufficient. %If the temporal brain transformer learns regional routing, its ROI queries interpretable organization in the language network.
In summary, our main contributions are: \begin{itemize} \item[(i)] We introduce \textbf{RABBiT}, a compact brain-tuned audio-to-fMRI encoder that supports both zero-shot population prediction and parameter-efficient few-shot adaptation to new subjects. \item[(ii)] We propose \textbf{Temporal Brain Transformer (TBT)} as one key component of RABBiT, which learns region-specific attention over speech representations, enabling explicit ROI-level temporal routing instead of fixed temporal pooling or independent voxel-wise processing. In addition to improving the predictive performance, TBT learns interpretable ROI-query representations that we show recover a coarse cortical hierarchy from auditory to higher-order language regions.
\item[(iii)] We propose \textbf{Shared--Idiosyncratic Decomposition (SID)} as a second key component of RABBiT, which learns low-rank factorizations of the neural response into shared population structure and subject-specific deviations. SID preserves zero-shot accuracy while enabling efficient adaptation to new participants from minutes of fMRI data. \item[(iv)] We evaluate on two held-out naturalistic listening cohorts totaling 324 unseen participants, and show that RABBiT's zero-shot performance exceeds that of much larger baselines, including TRIBEv2. It can also reproduce canonical language network localizers without any re-training, indicating its utility as a model organism for in-silico neuroscience of language and speech.
\item[(v)] We show that few-shot SID adaptation significantly beats voxel-wise ridge encoders trained on the same calibration data. The largest gains occur in higher-order language ROIs that vary most across participants, confirming the model's central intuition. \end{itemize}
\section{Related Work}
\label{sec:relatedwork}
\textbf{Pretrained brain encoding models.} Pretrained language and speech models are widely used to predict brain responses to naturalistic language \citep{wehbe2014simultaneously,jain2018incorporating,toneva2019interpreting,schrimpf2021neural,huth2016natural,antonello2024scaling,millet2022toward,vaidya2022self,oota2024speech}. However, the dominant approach---participant-specific voxel-wise ridge regression---does not support either zero-shot prediction for unseen participants or efficient few-shot adaptation.
\textbf{Brain-tuned speech encoders.} Brain-tuning methods improve speech representations using fMRI supervision \citep{moussa2025improving,moussa2025braininter,moussa2025multibrain,vattikonda2025brainwavlm}. These approaches enhance features later used by participant-specific readouts, whereas RABBiT integrates brain-tuning as part of a full encoder designed for both zero-shot and few-shot prediction.
\textbf{Zero-shot brain foundation models.}
Recent large-scale brain encoders move toward zero-shot prediction across participants. TRIBEv2 \citep{dAscoli2026TribeV2}, for example, combines large multimodal backbones with a large transformer and subject-conditioned output heads to predict group-level responses. In contrast, RABBiT is a compact, parameter-efficient audio-to-fMRI foundation model specialized for auditory and language regions. It supports accurate zero-shot prediction while producing higher-resolution outputs (4x the resolution of TRIBEv2) and enabling efficient adaptation to new participants.
\textbf{Few-shot transfer to new participants.} Voxel-wise ridge regression remains a strong participant-specific baseline when sufficient fMRI data are available \citep{huth2016natural,antonello2024scaling}, but performs poorly in low-data regimes due to its large number of parameters. Although TRIBEv2 also supports participant adaptation, it requires fine-tuning large model components and constructing new output heads \citep{dAscoli2026TribeV2}. RABBiT instead freezes the shared encoder and adapts only a low-rank subject-specific pathway. The closest related approach, Universal Brain Encoder \citep{beliy2024wisdom}, performs few-shot adaptation for image-to-fMRI prediction, whereas RABBiT supports both speech-based zero-shot prediction and efficient few-shot adaptation within a unified framework.
\textbf{Region-level transformer routing.} RABBiT's temporal brain transformer is inspired by transformer-based brain encoders for vision \citep{adeli2025transformer}, which use ROI queries to route visual information to cortical regions. RABBiT extends this idea to temporal speech representations, multi-subject training, and shared--idiosyncratic decoding through SID, yielding a reusable region-level temporal routing mechanism rather than a participant-specific readout.
Overall, prior work lacks a compact speech-to-fMRI encoder that simultaneously supports accurate zero-shot prediction and efficient few-shot adaptation. RABBiT addresses this gap by combining brain-tuning, ROI-level temporal routing, and a low-rank shared--idiosyncratic decomposition.
\section{Methods}
\label{sec:methods}
We motivate here the design choices for RABBiT and provide details about each model component, the training process, and the zero-shot and few-shot evaluation settings.
\subsection{Low-rank Decomposition of Brain Responses}
\label{sec:methods-shareddev}
\subsubsection{Intuition}
\label{sec:methods-shareddev-int}
A transferable encoder for language-evoked brain activity must balance two competing objectives: capturing subject-specific structure while remaining scalable across participants. A single population-level readout is scalable but obscures reliable individual variability, whereas participant-specific readouts scale poorly with cohort size and cortical resolution \citep{dAscoli2026TribeV2}. This tradeoff is particularly acute in fMRI, where outputs are high-dimensional and subject-level data are limited.
To characterize which response components are shared across participants, we measure, for each auditory and language ROI, how well an individual's response correlates with the group average when that individual is included versus excluded from the average. Higher-order language regions show substantial degradation under leave-one-out averaging (App.\ref{app:sid_pre}), indicating stronger idiosyncratic structure, whereas early auditory regions remain comparatively stable across participants, consistent with prior inter-subject correlation analyses \citep{nastase2021narratives}.
Motivated by this structure, we introduce a novel \textbf{Shared--Idiosyncratic Decomposition (SID)} in which each ROI response is modeled as the sum of a low-rank shared component and a low-rank subject-specific deviation. The shared component captures response structure that transfers across listeners, while the deviation component models individual variability without requiring a full participant-specific voxel-wise readout. This decomposition naturally supports both transfer settings used throughout the paper. In zero-shot prediction, RABBiT combines the shared component with the average learned deviation to estimate population responses (Sec.\ref{sec:methods-zs}). In few-shot calibration, only the low-rank deviation parameters are updated, enabling efficient subject adaptation without relearning the full cortical mapping (Sec.\ref{sec:methods-fs}).
%A transferable encoder for language regions in the brain has to avoid two bad extremes. A single population readout is scalable, but it washes out reliable subject-specific structure. A full readout per participant can fit that structure, but its parameters grow with the cohort and with cortical resolution \citep{dAscoli2026TribeV2}. This tradeoff is especially costly for fMRI, where the output space is large and subject-level data are limited.
%We therefore first ask which parts of the response are actually shared. For each auditory and language ROI, we compare how well a subject's response correlates with the group average when that subject is included versus when the subject is left out. ROIs whose correlation drops sharply under leave-one-out averaging are not well explained by the population alone; in our data, this occurs most strongly in higher-order language regions (App.\ref{app:sid_pre}; consistent with inter-subject correlation analyses, \citealp{nastase2021narratives}). Other regions (mostly early-to-mid auditory ones) remain more stable across subjects and should be predictable from a population model.
%This motivates our novel \textbf{Shared--Idiosyncratic Decomposition (SID)}: each brain ROI response is represented as a low-rank population component plus a smaller low-rank subject deviation. The shared term captures the response geometry that transfers across listeners. The deviation term gives the model a bounded way to represent subject-specific structure without allocating a full voxel-wise head to every participant. The same decomposition gives the two transfer modes used throughout the paper. In zero-shot prediction, RABBiT uses the shared component together with the average learned deviation to estimate the population response (Sec.\ref{sec:methods-zs}). In few-shot calibration, only the low-rank deviation coefficients are updated, so the calibration data are spent on subject correction rather than on relearning a full cortical readout (Sec.\ref{sec:methods-fs}).
% Recent multi-subject brain encoders predict all voxels jointly with per-subject heads \citep{dAscoli2026TribeV2}. This can work with very large datasets, but fMRI data are scarce (largest datasets have tens of hours per subject) and the cortical output space is high-dimensional. The problem is sharper in multi-subject training: a single mean readout averages out useful subject-specific signal, while one head per participant grows linearly with the cohort and limits scaling at high cortical resolution as well as opens more room for overfitting.
% Our approach is motivated by a simple \textbf{shared--idiosyncratic response diagnostic}: correlating each subject's response with the group average versus the leave-one-out group average shows that some regions (mainly high-level language ones) drop sharply when the subject is excluded, while others remain more stable across subjects (App.\ref{}; consistent with the inter-subject correlation literature, \citealp{nastase2021narratives}). Thus, some regions are naturally more \textbf{shared}, while others are more \textbf{idiosyncratic} and require per-subject calibration since even the best group-level prediction can't predict them well. We therefore introduce the \textbf{Shared--Idiosyncratic decomposition (SID)}, where a brain response is modeled as a low-rank \textbf{shared component} plus a meaningful small low-rank \textbf{per-subject deviation from it}.
% Using this structure for training from multiple participants (Fig.\ref{fig:training}) resolves \textbf{cohort scaling} problems because new subjects add no full prediction head, just a small learnable deviation. It also enables \textbf{zero-shot prediction} through the shared component and deviation averaging (which represents meaningful population-level Idiosyncrasies; more details in Sec.\ref{sec:methods-zs}), as well as \textbf{cheap few-shot calibration} by updating only the low-rank deviation coefficients rather than a full voxels-wise model; more details in Sec.\ref{sec:methods-fs}.
\subsubsection{Formulation}
\label{sec:methods-shareddev-form}
Shared-Idiosyncratic Decomposition (SID) decomposes each brain region-of-interest (ROI's) response into a population-shared component and a small per-subject part. Concretely, let $z_i \in \mathbb{R}^{d_o}$ be the representation used to predict the brain response (e.g., the output of temporal brain transformer; Sec.\ref{sec:methods-trans}). Then for ROI $i$ with $V_i$ voxels, SID predicts the ROI response for subject $s$ as \[ \hat{y}^{(s)}_i = \pi_i(z_i)\, \Phi_i + \rho_i(z_i)\, \Delta_{i,s}, \] where %\[ \Phi_i \in \mathbb{R}^{M \times V_i}, \qquad \Delta_{i,s} \in \mathbb{R}^{R \times V_i}, \qquad R < M. \]
$\Phi_i \in \mathbb{R}^{M \times V_i}$ is the shared basis that defines the population response subspace for ROI $i$, $\Delta_{i,s} \in \mathbb{R}^{R \times V_i}$ is the deviation basis that defines the subject-specific correction subspace, $R < M$, and the coefficient maps $\pi_i:\mathbb{R}^{d_o}\to\mathbb{R}^{M}$ and $\rho_i:\mathbb{R}^{d_o}\to\mathbb{R}^{R}$ make both terms stimulus-dependent.
SID can be viewed as a per-ROI mixed-effects readout consisting of a shared population component and a low-rank subject-specific component. The rank constraint serves as the key regularizer, enabling subject-specific flexibility only where supported by data while preventing an unconstrained voxel-wise per-subject model. Unlike independent subject-specific heads \citep{dAscoli2026TribeV2}, the deviation components form a compact basis of systematic individual differences from the population response.
We initialize the bases from the same empirical split that motivated the model: the shared bases from PCA of the mean training response, and the deviation bases from SVD of each subject's residual from that mean. This starts training from cross-subject structure in the shared term and residual subject structure in the deviation term; initialization details are given in App.\ref{app:sid-init}.
% Here, we introduce the mathematical formulation of our Shared-Idiosyncratic (SID) approach, motivated earlier. We decompose each brain region-of-interest (ROI's) response into a population-shared component and a small per-subject part. Concretely, for ROI $i$ with $V_i$ cortical vertices and subject $s$, the per-vertex prediction $\hat{y}^{(s)}_i \in \mathbb{R}^{V_i}$ is
% \[
% \hat{y}^{(s)}_i \;=\; \alpha_i(z_i)\, B^{\mathrm{sh}}_i \;+\; \beta_i(z_i)\, B^{\mathrm{dev}}_{i,s}.
% \]
% $\alpha_i:\mathbb{R}^{d_o}\!\to\!\mathbb{R}^{K}$ and $\beta_i:\mathbb{R}^{d_o}\!\to\!\mathbb{R}^{R}$ are small linear maps with $R<K$: $K$ is the dimensionality of the population-shared response space, and $R$ is the budget for what the population alone cannot capture (we find $R < K$ to suffice in practice; Sec.\ref{sec:methods-training}). $B^{\mathrm{sh}}_i \in \mathbb{R}^{K\times V_i}$ is the population-shared basis, and $B^{\mathrm{dev}}_{i,s} \in \mathbb{R}^{R\times V_i}$ is a per-subject deviation basis (one per training subject).
% SID is a per-ROI mixed-effects parameterization: $B^{\mathrm{sh}}_i$ is the population-level shared information, $B^{\mathrm{dev}}_{i,s}$ the subject-level disctinct information. To ensure a strong prior and stable training, the bases are initialized from per-ROI PCA of the mean-of-subjects fMRI ($B^{\mathrm{sh}}_i$) and per-subject SVD of the deviations from that mean ($B^{\mathrm{dev}}_{i,s}$). This biases the shared bases to capture cross-subject structure and the deviation bases to capture only what the shared cannot. Full details about the bases initializations is found in App.\ref{}.
% This formulation relates other parallels in the broader literature. Each of the SID terms is similar to Low-rank adaptation (LoRA) trainable $A$ and $B$ matrices \citep{hu2022lora}, but with strong prior initialization and stimulus-dependent coefficients. In graphics, \textbf{parametric shape models} --- e.g.\ FLAME for 3D heads \citep{FLAME} --- use a learned shared template plus low-dimensional per-instance offsets (knobs) to avoid free-form prediction over a high-dimensional output space; SID plays the same role here, with the per-ROI shared bases as templates, the deviation coefficients $\beta_i(z_i)$ as knobs, and the bounded budget $R$ keeping idiosyncratic structure efficient.
\subsection{Temporal Brain Transformer}
\label{sec:methods-trans}
SID defines the mapping from ROI-level latents $z_i$ to voxel-level brain responses, while the temporal brain transformer determines how each region constructs its latent representation from the speech stream. In contrast, voxel-wise ridge regression lacks an explicit region-level representation: it applies fixed temporal pooling (e.g., HRF convolution or lag windows \citep{huth2016natural,antonello2024scaling}) followed by independent regression for each cortical vertex. RABBiT replaces both components with learned ROI-level routing, where each region adaptively integrates temporal context and shares a common latent representation across its voxels via SID.
%SID defines how an ROI latent $z_i$ becomes a voxel-level brain response. The temporal brain transformer defines how each region obtains this latent. A voxel-wise ridge readout has no such region-level object: it consumes speech features pooled by a fixed temporal aggregator, typically an HRF kernel or pre-specified lag window \citep{huth2016natural,antonello2024scaling}, and then fits every cortical vertex as an independent subject-specific regression target. RABBiT replaces both choices with learned ROI-level routing: each region learns what temporal context to use, and the resulting latent is reused across the vertices of that region through SID.
\textbf{Cross-attention module.} Concretely, each cross-attention block (Fig.\ref{fig:trans}) lets a learnable ROI query token $q_i$ attend over a window of speech model output tokens. The attention profile $\alpha_{i,\cdot}$ is the model's learned temporal pooling rule for ROI $i$: it specifies which parts of the speech context are used to form that region's state. Because the query is shared across subjects, this temporal profile can be inspected directly from the trained model, without fitting a separate per-subject temporal readout; full mathematical details including the choice of positional embedding in App.\ref{app:trans}.
% $\{x_t\}_{t=1}^{T}$. For a single attention head, \[
% \alpha_{i,t} = \mathrm{softmax}_t\!\big(\langle W_Q q_i,\; k_t \rangle\big),
% \]
% The attention profile $\alpha_{i,\cdot}$ is the model's learned temporal pooling rule for ROI $i$: it specifies which parts of the speech context are used to form that region's state. Because the query is shared across subjects, this temporal profile can be inspected directly from the trained model, without fitting a separate per-subject temporal readout; full mathematical details including the choice of positional embedding in App.\ref{}.
\textbf{Temporal brain transformer.} The full temporal brain transformer stacks $L$ such cross-attention blocks over the ROI queries, following the design of \citep{adeli2025transformer}. The first layer reads backbone features into the ROI queries; later layers re-attend with updated region states, refining what each region selects from the speech stream. Multi-head attention lets a region combine complementary temporal patterns in parallel, such as faster acoustic structure and slower contextual structure, while the FFN in each block transforms the resulting ROI state. The final state is the per-ROI latent $z_i$ passed to SID.
This architecture is powerful without becoming voxel-wise. The transformer learns temporal routing once per ROI, rather than learning separate temporal filters for every vertex and subject. Vertex-level and subject-specific structure are handled downstream by SID's low-rank shared and deviation bases. As a result, the routing module remains compact: our main model uses two cross-attention layers with eight heads per layer and hidden dimension $256$, for $\sim$2.4M parameters reused across participants; full architecture details are given in Sec.\ref{sec:methods-training}.
% A voxel-wise ridge readout treats every cortical vertex as an independent regression target with its own per-subject weights. The speech features it consumes are pooled with a fixed temporal aggregator (typically a fixed HRF kernel \citep{huth2016natural,antonello2024scaling}), leaving no room for specialized per-ROI temporal integration. RABBiT's temporal brain transformer replaces with learnable anatomical ROI queries: each cortical region learns its own aggregation over the whole temporal context over multiple brain cross-attention layers.
% \textbf{Cross-attention module.} Concretely, each cross-attention module (Fig.\ref{fig:trans}) works as follows, a learnable ROI query token $q_i$ over the input window of backbone tokens $\{x_t\}$, region $i$'s attention is
% \[
% a_{i,t} = \mathrm{softmax}_t\!\big(\langle W_Q q_i,\; W_K x_t \rangle\big),
% \] and the per-ROI latent $z_i = \sum_t
% \alpha_{i,t}\, W_V x_t \in \mathbb{R}^{d_o}$ feeds the SID readout (Sec.\ref{sec:methods-shareddev}). The attention distribution $\alpha_{i,\cdot}$ is itself the temporal receptive field of region $i$: every query produces its own, directly usable from the trained model with no per-subject tuning (Sec.\ref{sec:methods-ablation}).
% \textbf{Whole brain transformer model.} The temporal brain transformer follows traditional transformers recipes (including \citep{adeli2025transformer}): it stacks $N$ such cross-attention layers over the ROI queries, with multi-head attention and a feed-forward (FFN) block at each layer; full architecture details are found in Sec.\ref{sec:methods-training}). The first layer reads backbone features into the queries; subsequent layers re-attend with updated queries, sharpening what each region picks out. Multi-head attention expands the capacity to attend across distinct temporal patterns in parallel (e.g., one head can pick up fast phonetic structure, another the slower envelope). Despite these multiple powerful components, the whole model is paramter-efficient (2.4M paramters for 2 layers and 8 heads each) because of inherent query attention structure that doesn't scale linearly with the number of subjects or number of voxels as in linear encoding models.
\subsection{Brain-tuning of the Speech Backbone}
\label{sec:methods-btune}
The temporal brain transformer learns which parts of the speech stream each ROI should use, but this routing is only as useful as the tokens it receives. A pretrained speech model provides strong acoustic and linguistic features, but its representations can be improved to predict cortical responses better with brain-tuning. Following recent work \citep{moussa2025improving,moussa2025multibrain}, we therefore brain-tune the speech backbone during RABBiT training using LoRA adapters \citep{hu2022lora}.
Brain-tuning is integrated directly into the encoder: gradients propagate from the SID readout through the temporal brain transformer into the LoRA adapters, adapting speech representations for both shared population prediction and subject-specific deviations. The adapter constraint enables parameter-efficient adaptation while preserving a shared backbone across participants, avoiding both full model retraining and reliance on fixed pretrained features.
%This makes brain-tuning part of the encoder rather than a separate representation improvement step. Gradients pass from the SID readout, through the temporal brain transformer, and into the LoRA adapters, so the speech tokens are pushed to be useful for population prediction and subject deviations. The adapter constraint keeps this shift parameter-efficient and shared across participants, avoiding both full backbone retraining and purely relying on off-the-shelf speech representations.
We follow the multi-subject brain-tuning setup of \cite{moussa2025multibrain}; dataset construction and optimization details are provided in App.\ref{app:btune}.
% Following recent work (\cite{moussa2025improving, moussa2025multibrain}, we brain-tune a speech backbone via parameter-efficient fine-tuning. The body is frozen and LoRA adapters \citep{hu2022lora} are inserted only into the audio encoder. We think of brain-tuning the backbone as an essential step that does some of the heavy lifting in absorbing information from the training data while not needing to train a big model from scratch or purely rely on the pretrained speech-specific representations of the model. We follow the same details from \citet{moussa2025multibrain} for brain-tuning including how to create paired data and how to train on multiple subjects; Full details in App.\ref{}.
\subsection{Dataset Details}
\label{sec:methods-datasets}
\textbf{Surface projection.} All fMRI data are projected to the FreeSurfer \texttt{fsaverage6} cortical surface (with 42K vertices per hemisphere) \citep{fischl1999high} and then parcellated to 30 Brain ROIs, totaling $\sim$41K cortical vertices. The ROIs span auditory and language cortex as well as visual, motor, and theory-of-mind control regions (see App.\ref{app:roi_deets} for the full list of ROIs and the rationale for choosing them)
%All fMRI data are projected to the FreeSurfer \citep{fischl2012freesurfer} \texttt{fsaverage6} cortical surface ($\sim$40{,}962 vertices per hemisphere, then parcellated 30 bilateral HCP-MMP1 ROIs of Sec.\ref{sec:methods-trans} \citep{glasser2016multi} (totaling $\sim$41K voxels). These ROIs span auditory and language regions as well as vision, motor, and theory of mind regions (full details of projection and ROIs list in App.\ref{app:roi_deets}). Our choice to expand the ROIs beyond vision is to stress-test our model ability to generate high resolution wide-brain prediction without impacting performance or blowing up the trainable parameters% (which we confirmed by comparison to more restricted ROIs sets; see App.\ref{})
\textbf{Training data.} We train on the \textbf{Friends} subset of CourtoisNeuroMod \citep{boyle2025friends, st2026cneuromod} --- 6 participants watching naturalistic audio-visual episodes ($\sim$565K TRs total). Additional details on participant hours, data splits, and TR duration are provided in App.\ref{app:train_data}. We also evaluated training on additional datasets, including Moth Radio \citep{lebel2023natural} ($\sim$113K TRs), but did not observe consistent performance improvements across settings (App.\ref{app:mixed_data}). This suggests that integrating heterogeneous naturalistic datasets may require more careful treatment, which we leave to future work.
\textbf{Evaluation data.} Zero-shot evaluation uses two held-out listening cohorts with no participant overlap with training: seven stories from \textbf{Narratives} \citep{nastase2021narratives} (mean length 15\,min; $\sim$30 participants per story) and the English \textbf{Le Petit Prince} audiobook \citep{Li2022LePetitPrince} (49 participants, 9 sections, $\sim$90\,min). Together, these cohorts provide 324 unseen participants for zero-shot group-response evaluation. Per-story durations, participant counts, and inclusion criteria are given in App.\ref{app:eval_data:dets}.
%We evaluate on two unseen participant cohorts. From the \textbf{Narratives} dataset \citep{nastase2021narratives} we use 7 held-out stories (mean length 15\,min; on average 30 participants per story); we add \textbf{Le Petit Prince} \citep{Li2022LePetitPrince} as another independent corpus from a different recording pipeline (49 participants; the English audiobook in 9 scanning sections totalling $\sim$90\,min), probing generalisation beyond Narratives. The two cohorts comprise of 324 unseen participants. Per-story durations, participant counts, and inclusion criteria can be found in App.\ref{app:eval_data:dets}.
% \textbf{Surface projection.} All fMRI data were projected to the FreeSurfer `fsaverage6` (fs6) cortical surface; then, 30 bilateral HCP-MMP1 ROIs \citep{glasser2016multi} of Sec.\ref{sec:methods-trans} are parcellated. Audio is HRF-aligned to TR resolution. Per-dataset preprocessing and the audio-to-TR alignment recipe are listed in App.\ref{}.
% \textbf{Training data.} We train our model on the \textbf{Friends} subset of CourtoisNeuroMod \citep{boyle2025friends} ($\sim$565K TRs across 6 participants). Per-subject hours, splits and TR length in App.\ref{}.
% \textbf{Evaluation data.} We evaluate on \textbf{7 held-out stories from the Narratives corpus} \citep{nastase2021narratives} (\textbf{30} participants per story on average; mean length \textbf{15}\,min) plus \textbf{Le Petit Prince} \citep{Li2022LePetitPrince} (\textbf{49} participants; \textbf{9 stories}; \textbf{90}\,min); \textbf{324 unseen participants} in total, none of whom appear in any training set. The Narratives stories were selected for their naturalistic listening setup. Le Petit Prince is included as an*independent more difficult corpus from a different study, probing generalisation beyond the Narratives recording pipeline. Per-story durations, participant counts and full inclusion criteria in App.\ref{}.
\subsection{Training Details}
\label{sec:methods-training}
The full training proceeds as follows. Input audio is encoded into temporal speech representations by the speech backbone (Sec.\ref{sec:methods-btune}), which are mapped to ROI-level latents by the temporal brain transformer (Sec.\ref{sec:methods-trans}). These latents are then mapped onto fMRI responses through the SID readout (Sec.\ref{sec:methods-shareddev}). Training minimizes the difference between predicted and true fMRI by jointly optimizing the SID parameters, temporal brain transformer, and LoRA-based speech adapters (Fig.\ref{fig:training}).
%Putting it all together, the whole training pipeline works as follows. Input audio is encoded by a speech model into a temporal token sequence (Sec.\ref{sec:methods-btune}); the temporal brain transformer routes those tokens to ROI latents (Sec.\ref{sec:methods-trans}); the per ROI latent passes through our low-rank decomposition(learnable shared + per-subject Idiosyncratic components) to form the predicted fMRI response (Sec.\ref{sec:methods-shareddev}); the loss is then applied between predicted and measured fMRI. For the training phase, we backpropgate the loss through the SID part: training the bases and coefficients, the brain transformer layers and the speech backbone (Fig\ref{fig:training})
\textbf{Loss.} We train jointly on $S{=}6$ participants from Friends, in fsaverage6 space (Sec.\ref{sec:methods-datasets}). The loss combines a per-ROI z-scored MSE and a Pearson-correlation term. We train until convergence on a validation set. Full details for the loss and training hyperparameters can be found in App.\ref{app:loss_train}.
\textbf{Model components details.} We use Wav2Vec2.0-base (90M params) \citep{baevski2020wav2vec} as our main speech backbone. Using a bigger pretrained model did not lead to significant gains (see Sec.\ref{res:ablation}). For brain-tuning, we use LoRA rank 8 after ablation (App.\ref{app:btune}). For the brain transformer, we use 2 layers, with 8 heads and a hidden dimension of 256 (full architecture ablations are in App.\ref{app:trans}). For the SID matrices, we use $M=100$ and $R=15$ after ablation (see App.\ref{app:sid-dims} for more details).
% \subsection{Evaluation Details}
% \label{sec:methods-eval}
% For evaluation, predictions are generated per TR and concatenated
% within each held-out story; we report vertex-wise Pearson $r$ between
% the concatenated predictions and the measured BOLD. Comparisons in
% Sec.\ref{sec:results}.
\subsection{Zero-shot Evaluation}
\label{sec:methods-zs}
% \subsubsection{Generalization across novel stimuli and participants}
% \label{sec:methods-zs-gen}
For a new stimulus, RABBiT performs zero-shot inference by estimating the population-level response using the training subjects. The shared component is used directly, while the subject-specific deviation basis is replaced by its average across the $S$ training participants (see App.\ref{app:sid_pred}).
%\subsubsection{Baselines}
\label{sec:methods-zs_baselines}
\textbf{Linear baseline.} As a lower bound on performance, we train a linear baseline over the mean pooled outputs of the speech model (no brain transformer, no brain-tuning, and no SID).
\textbf{Per-subject Full-rank baseline.} To validate our low-rank SID approach, we also report a subject-specific heads baseline (i.e., one full readout head per-subject) while keeping everything else the same. This adds around $6$x more trainable parameters. If our model matches or beats that at zero-shot settings, this confirms our SID decomposition is sound and not an efficiency-performance trade-off.
\textbf{TRIBEv2.} As a strong baseline, we compare with the multimodal TRIBEv2 \citep{dAscoli2026TribeV2}. We use the official code to infer the model's output for the same evaluation naturalistic listening stories (which zeros-out the visual component of TRIBEv2). To represent it fairly, we compute the whole brain correlation on a subset of narratives and compare it with the reported range of the original paper (see App. \ref{app:tribe-baseline}).
To report performance over different fMRI studies (Sec.\ref{sec:methods-datasets}), we use group-level Pearson correlation ($r_\mathrm{group}$) between the predictions and group-level mean of the study, following \citep{dAscoli2026TribeV2}. For each of these studies, we obtain and validate the audio-onset (a standard practice in fMRI studies to account for the mismatch between stimulus and machine recording times); full details can be found in App.\ref{app:eval_data:audio_onset}.
\subsection{In-silico Analyses and Ablations}
\label{sec:methods-ablation}
Beyond held-out encoding accuracy, we detail here the use of RABBiT as a model organism to localize the language network, the interpretability analyses of our temporal brain transformer, as well as the ablations used to validate the role of each component of our RABBiT model.
\textbf{In-silico language localization.} We ask whether our model reproduces \emph{known functional selectivity} when probed with controlled stimuli, treating the trained encoder as an in-silico subject that can be submitted to the same functional localizers used in human fMRI. We use an auditory adaptation of the functional language-localizer approach \citep{fedorenko2010new}: $18$\,s blocks of intact speech contrasted against acoustically matched, spectrally degraded (unintelligible) speech, a standard intelligibility manipulation that isolates linguistic processing above low-level acoustics \citep{scott2000identification, davisjohnsrude2003}. Each of the $32$ source-matched intact/degraded materials is presented to the model as a continuous stimulus; we slide the model's $1.49$\,s TR window with its trained hemodynamic delay basis over the block and read out the sustained (plateau) response. The per-vertex \textbf{Intact$-$Degraded contrast map} is evaluated with a paired $t$-test across the $32$ source-matched pairs, leaving voxels with $q<0.0001$ (FDR-corrected) on the \texttt{fsaverage6} surface.
\textbf{Cortical hierarchy from query embeddings.} In addition to improving accuracy and parameter-efficiency, the brain transformer enables analyses that linear voxel-wise readouts do not. The ROI query tokens are themselves data-driven learned representations of brain regions, and the relationships among them can be tested and related to literature about the brain's organization of language processing. We test this with a simple greedy walk: starting from primary auditory cortex (A1), at each step we move to the nearest unvisited ROI in query space using cosine similarity of query embeddings. If the queries have learned the right structure, the walk should retrace the cortical hierarchy of speech-to-language processing --- primary auditory $\to$ belt $\to$ STG/STS $\to$ higher-order temporal and frontal regions --- a structure that was never given to the model as supervision.
\textbf{Ablations.} Lastly, we ablate each design choice in turn to assess its contribution. Freezing wav2vec2.0 isolates the contribution of brain-tuning. Replacing the brain transformer with a direct linear readout isolates the importance of our brain transformer. Replacing SID with independent per-subject heads tests whether parameter sharing costs accuracy. As a robustness check, we also swap in WavLM-large \citep{chen2022wavlm} --- a larger (317M parameters) and more capable speech model --- to test whether the choice of model architecture or size is a bottleneck.
\subsection{Few-shot Evaluation}
\label{sec:methods-fs}
Given a short calibration window of paired audio-fMRI from a new participant, we perform few-shot adaptation by updating only the per-ROI deviation coefficients (the $\rho_i$ terms in Sec.\ref{sec:methods-shareddev-form}), while keeping other components frozen. This results in only $\sim$115K coefficients to update (as opposed to $\sim$200M ridge parameters). This efficiency is enabled by the SID formulation, which isolates subject-specific variability into a compact set of parameters that target idiosyncratic cortical responses not captured by the population-level component (Sec.\ref{sec:methods-shareddev}); see App.\ref{app:fs_math} for full details.
%\subsubsection{Baselines}
\label{sec:methods-fs_baselines}
\textbf{No SID RABBiT baseline.} To assess the role of SID in few-shot transfer, we construct a variant in which SID is replaced by linear readouts applied to the transformer outputs. We then apply the same few-shot adaptation, updating only the linear readout. This ablation tests whether structured low-rank decomposition is necessary for effective adaptation from limited calibration data (more in App.\ref{app:fs_direct}).
\textbf{Ridge baselines.} We compare RABBiT few-shot calibration against two voxel-wise ridge encoders trained on the same calibration data (details in App.\ref{app:fs_ridge}). The first uses pretrained speech features, following the canonical subject-specific encoding setup \citep{antonello2024scaling}. The second uses multi-brain-tuned features, which are expected to be more data-efficient than pretrained features \citep{moussa2025multibrain}. This second baseline tests whether RABBiT's few-shot gains come merely from inherited brain-tuned representations, or from SID's low-rank subject-adaptation pathway.
% \textbf{Pretrained ridge baseline.} Linear ridge encoding over pretrained features is the canonical subject-specific method; thus, comparing our few-shot gains to it is essential. We follow the same approach as \citep{moussa2025multibrain, antonello2024scaling} to train this baseline (full details in App.\ref{app:fs_ridge})
% \textbf{Brain-tuned ridge baseline.} Representations of Multi-brain-tuned models should be more data efficient than the pretrained ones \citep{moussa2025multibrain}. Therefore, we use it a baseline against our few-shot as well to isolate whether RABBiT's few-shot gain comes from SID or inherited brain-tuned features. We follow the same approach as the authors in training it (full details in App.\ref{app:fs_ridge}).
% For our model and the baselines, we report the mean over the per-subject correlation across different fMRI calibration sizes.
\section{Results}
\label{sec:results}
%\subsection{Evaluation Setup}
\label{res:setup}
Our evaluation tests transfer without the need for full participant-specific retraining. In zero-shot prediction, RABBiT uses audio input to predict group-average responses of unseen participants to held-out stories, evaluating how well the model captures shared population structure. In few-shot adaptation, a short paired audio--fMRI segment is provided for target participants and only the SID deviation pathway is updated, testing the efficiency of adaptation to subject-specific responses. We additionally perform ablations to identify the model components driving performance, and query-embedding analyses to characterize the learned cortical structure.
%The results test transfer after training, not another round of full subject-specific fitting. In zero-shot, RABBiT receives only the audio from held-out stories and predicts the group-average response of unseen subjects; this tests whether the decomposition captures reliable population structure. In few-shot, a short paired audio--fMRI segment is available for target subjects, and only the SID deviation pathway is adapted; this tests whether idiosyncratic signal can be recovered without refitting a voxel-wise model. We then use ablations and query-embedding analyses to identify which components carry the gain and what cortical structure the model learns.
\begin{figure}[t]
\centering
% \includegraphics[width=0.9\textwidth]{figs/legend_dots_lang_paper_with_params.pdf}\\[2pt]
\begin{subfigure}[t]{0.55\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/dots_lang_paper_horiz.pdf}
\caption{Zero-shot comparison across cohorts}
\label{fig:zs_cohorts}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.40\linewidth}
\centering
% \includegraphics[width=0.4\textwidth]{figs/legend_delta_vs_tribe.pdf}\\[2pt]
\includegraphics[width=\linewidth]{figs/brain_aggregate_delta_fs6_focus_with_legend.pdf}
\caption{Zero-shot difference with TRIBEv2}
\label{fig:brain-zs}
\end{subfigure}
\begin{subfigure}[t]{0.65\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/vert_story_twosided_bracket_sensorimotorgrouped_with_legend.pdf}
\caption{Per-ROI comparison}
\label{fig:perroi-zs}
\end{subfigure}\\[6pt]
% \begin{subfigure}[t]{0.8\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figs/brain_21styear_compact.png}
% \caption{Zero-shot performance mean brain plot}
% \label{fig:brain-zs}
% \end{subfigure}
\caption{\textbf{Zero-shot prediction in unseen participants.} (\subref{fig:zs_cohorts}) Group-level prediction over language ROIs for two cohorts. RABBiT exceeds both the inter-subject consistency estimates and the performance of the much larger baselines TRIBEv2 and linear encoders. (b) Voxel-wise correlation gain of RABBiT over upsampled TRIBEv2 (matched cortical resolution; Sec.\ref{sec:relatedwork}). (c) ROI group-level correlation averaged over held-out stories and segments. RABBiT yields the largest gains in auditory and temporal regions, with comparable performance to TRIBEv2 in higher-order language regions.
}
\end{figure}
\subsection{Zero-shot Evaluation}
\label{res:zs}
We evaluate RABBiT's zero-shot predictions on Narratives and {Le Petit Prince}, totaling 324 unseen participants over two cohorts (Sec.\ref{sec:methods-datasets}), and report $r_{\mathrm{group}}$ over language ROIs. Because the target is a group-average response, a relevant reference point is the leave-one-out inter-subject consistency (ISC). RABBiT surpasses this ISC estimate on both held-out cohorts (Fig.\ref{fig:zs_cohorts}). It substantially outperforms the linear baseline and the much larger TRIBEv2 model, while matching the Per-Subject head variant that uses roughly five times more parameters. The important point is therefore not only that RABBiT is accurate, but that SID preserves zero-shot accuracy while replacing independent per-subject heads with a compact shared--deviation readout.
The per-region comparison makes the pattern more diagnostic with statistical significance (two-sided t-test) measured across the different studies (Fig.\ref{fig:perroi-zs}; Fig.\ref{fig:brain-zs}). RABBiT improves most over TRIBEv2 in auditory and temporal language regions. In other language regions, such as the IFG and MFG, the two perform comparably and near the inter-subject consistency estimate. The only exception is precuneus/PCC, where TRIBEv2 performs better. Outside the speech-language network, predictions in the visual and motor regions expectedly remain near chance, due to the nature of the test stimulus.
%regions where an audio-driven encoder should be most informative (e.g., auditory cortex and STsG/STS). In higher-order language regions such as MFG, posterior temporal cortex, and angular gyrus, the two models are mostly statistically tied, with both near the inter-subject ceiling. The main exception is precuneus/PCC, where TRIBEv2 performs better. Outside the speech-language target network, visual and motor regions remain near chance, expected due to the nature of the stimulus. %RABBiT but surprising for TRIBEv2; we attribute it to TRIBE's multimodality not being fully utilized during audio inference (see Sec.\ref{sec:methods-fs}).
% \begin{figure}[t]
% \centering
% \includegraphics[width=0.9\textwidth]{figs/legend_dots_lang_paper.pdf}
% \begin{subfigure}[t]{0.49\linewidth}
% \includegraphics[width=\linewidth]{figs/dots_lang_paper_no_legend.pdf}
% \caption{Zero-shot comparison across cohorts}
% \label{fig:zs_cohorts}
% \end{subfigure}\hfill
% \begin{subfigure}[t]{0.49\linewidth}
% \includegraphics[width=\linewidth]{figs/horiz_story_twosided_bracket_lang.pdf}
% \caption{Per-ROI comparison}
% \label{fig:perroi-zs}
% \end{subfigure}
% \begin{subfigure}[t]{0.8\linewidth}
% \includegraphics[width=\linewidth]{figs/brain_21styear_compact.png}
% \caption{ Zero-shot Performance mean brain plot }
% \label{fig:brain-zs}
% \end{subfigure}
% \caption{\textbf{Zero-shot prediction in unseen participants.}(\subref{fig:zs_cohorts}) Group-level prediction over bilateral language ROIs. Gray points show single-subject peer-ISC (using one subject to predict leave-one-out group response), giving the scale inter-subject reliability; the dashed line shows the leave-one-out ISC estimate. RABBiT reaches this reliability estimate on both held-out cohorts, while TRIBEv2 and the linear baseline remain below it. (\subref{fig:perroi-zs}) Per-ROI $r_{\mathrm{group}}$ averaged over held-out stories/sections. RABBiT improves most over TRIBEv2 in auditory and temporal speech regions; the models are mostly tied across higher-order language ROIs where both approach the inter-subject ceiling.}
% \end{figure}
\subsection{In-silico Analyses and Ablations Results}
\label{res:ablation}
\textbf{Language Localization.} RABBiT's Intact$-$Degraded contrast map reproduces a lot of the canonical, strongly left-lateralized language response \citep{scott2000identification, hickok2007cortical}. Fig.\ref{fig:fed-ln} shows the contrast map of the voxels with $q<0.0001$; intact speech is preferred in left anterior temporal cortex ($t=+5.7$), angular gyrus ($+6.2$), middle frontal gyrus ($+5.8$) and superior temporal gyrus/sulcus ($+2.2$). Despite the pretrained backbone being acoustic in nature, after naturalistic brain-tuning, RABBiT recovers the spatial signature of the language network without ever being trained on a controlled localizer or any non-naturalistic datasets.
%We next ask whether the gains come from the intended structure of the model: brain-tuned speech tokens, ROI-level temporal routing, and a compact shared--deviation readout.
\textbf{Learned Query Embeddings.} The learned ROI queries provide a direct view into the temporal brain transformer. Because each query specifies the speech context an ROI attends to, their embedding geometry reflects the model's cortical organization. Starting from A1 and greedily stepping to the nearest unvisited ROI in query space (cosine similarity), we recover a coherent trajectory without any hierarchy or spatial supervision. The resulting path progresses from primary and belt auditory cortex through STG/STS and temporal association areas toward parietal and frontal language regions (Fig.\ref{fig:trans_emb}), revealing an interpretable cortical structure absent in voxel-wise ridge models.
%The learned ROI queries give a direct probe of the temporal brain transformer. Each query determines what speech context an ROI reads, so its embedding geometry should reflect how the model organizes cortex. We test this with a minimal probe: starting at A1, we greedily step to the closest unvisited ROI, based on cosine similarity of the ROI queries. Without hierarchy or adjacency supervision, the resulting walk traces a coarse speech-to-language progression, moving from primary and belt auditory regions through STG/STS and temporal association cortex toward parietal and frontal language regions (Fig.\ref{fig:trans_emb}). Thus, the same query tokens used for prediction also expose an interpretable cortical organization, a region-level object absent from voxel-wise ridge models.
\textbf{Component Ablations.} We further perform ablations to quantify component contributions. Removing brain-tuning yields the largest drop in zero-shot performance ($\sim60\%$; Fig.\ref{fig:ablation_comp}, $p<0.001$), and removing the temporal brain transformer also significantly degrades performance ($p<0.05$). In contrast, replacing SID with larger per-subject heads provides no significant improvement despite $\sim$5$\times$ more parameters, and swapping Wav2Vec2.0 for WavLM-large has negligible effect. Together, these results indicate that brain-tuning and ROI-level routing are essential for zero-shot prediction, while SID provides compact subject modeling without sacrificing performance.
%We further perform ablations to identify the load-bearing components. Removing brain-tuning causes the largest drop in zero-shot prediction, by around $60\%$ (Fig.\ref{fig:ablation_comp}; paired two-sided Wilcoxon, $p<0.001$), and removing the temporal brain transformer also significantly hurts performance ($p<0.05$). In contrast, per-subject heads lead to no significant increase despite using $\sim$5$\times$ more parameters, and replacing Wav2Vec2.0 with WavLM-large has no significant impact. Together, these ablations confirm that brain-tuning and the temporal brain transformer are essential for RABBiT's zero-shot performance, while SID keeps the readout compact without any measurable zero-shot cost.
\begin{figure}[t]
\centering
\begin{subfigure}[t]{0.7\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/tribe_style_intact_vs_degraded.pdf}
\caption{Language Network: strong left fronto-temporal lateralization}
\label{fig:fed-ln}
\end{subfigure}\\[6pt]
\begin{subfigure}[t]{0.49\linewidth}
\includegraphics[width=\linewidth]{figs/hierarchy_path_2d_2.pdf}
\caption{Learned hierarchy from query embeddings}
\label{fig:trans_emb}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.42\linewidth}
\includegraphics[width=\linewidth]{figs/appendix_variants_lang_sig.pdf}
\caption{Ablations of model components}
\label{fig:ablation_comp}
\end{subfigure}
\caption{\textbf{Model analysis and ablations.} (\subref{fig:fed-ln}) Language network localizes reproduce the canonical left-lateralized language areas. (\subref{fig:trans_emb}) Interpretation of ROI-query embeddings learned by TBT, revealing a learned coarse speech-to-language progression. (\subref{fig:ablation_comp}) Ablation of model components shows the importance of TBT and brain-tuning for RABBiT's strong zero-shot performance. }
\end{figure}
\subsection{Few-shot Evaluation}
\label{res:fs}
We further test whether RABBiT can improve over zero-shot prediction for a new participant using limited fMRI calibration data. For each participant, we freeze the speech backbone, temporal brain transformer, and shared SID pathway, and update only the low-rank per-ROI deviation parameters ($\sim$115K parameters; Sec.\ref{sec:methods-fs}). We compare against voxel-wise ridge regression trained on the same calibration data using either pretrained or brain-tuned speech features \citep{huth2016natural,antonello2024scaling,moussa2025multibrain}. This is a strong baseline: ridge fits a full voxel-wise subject-specific readout, whereas RABBiT only learns a compact correction to its zero-shot prediction. We evaluate primarily on long narrative segments ($\sim$50 min) to allow for increasing amounts of calibration data (see App.\ref{app:fs_calib}).
RABBiT few-shot improves over zero-shot with only 5 minutes of calibration and outperforms both ridge baselines at every tested duration from 5 to 40 minutes (Fig.\ref{fig:fewshot}). Similar trends hold on an additional dataset (15 min; App.\ref{app:fs_add_results}). The strongest ridge variant (brain-tuned features) improves with more data but starts below zero-shot and requires substantially more calibration to match performance. In contrast, RABBiT begins from a strong population model and uses calibration data exclusively to refine subject-specific deviations, yielding roughly three orders of magnitude fewer trainable parameters than ridge while achieving better prediction.
Improvements are concentrated in specific regions. At $10$ minutes, the largest gains occur in IFG, supramarginal gyrus, mPFC, angular gyrus, MFG/DLPFC, and precuneus/PCC (Fig.\ref{fig:perroi-fewshot}), reaching $30$--$90\%$ relative improvement. These are the same higher-order language regions identified as most idiosyncratic in our analysis in Sec.\ref{sec:methods-shareddev}, indicating that few-shot adaptation selectively recovers structured individual variability. Ridge baselines do not show comparable targeted recovery at low data regimes, though they approach similar trends with substantially more data (App.\ref{app:fs_add_results}).
\begin{figure}[t]
\centering
\includegraphics[width=0.9\textwidth]{figs/legend.pdf}
\begin{subfigure}[t]{0.47\linewidth}
\includegraphics[width=\linewidth]{figs/w2v_saturation_paper_4bar_sorted_no_legend.pdf}
\caption{Few-shot Calibration}
\label{fig:fewshot}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.49\linewidth}
\includegraphics[width=\linewidth]{figs/per_roi_top_C400_with_direct_no_legend.pdf}
\caption{ROI $\%$ change after 10\,min of calibration}
\label{fig:perroi-fewshot}
\end{subfigure}
\begin{subfigure}[t]{0.6\linewidth}
\includegraphics[width=\linewidth]{figs/brain_delta_21styear_C400_fewshot_focus_with_legend.png}
\caption{few-shot difference with zero-shot for 10 mins of calibration}
\label{fig:brain-fs}
\end{subfigure}
\caption{\textbf{Few-shot calibration for new participants.} (\subref{fig:fewshot}) RABBiT few-shot performance improves over zero-shot with only 5 minutes of calibration data and outperforms both ridge baselines. SID is key to this ability. (\subref{fig:perroi-fewshot}) Per-ROI percent change in performance relative to zero-shot with 10 minutes of calibration (other calibration sizes reported in App.\ref{app:fs_add_results}). RABBiT's largest gains occur in higher-order ROIs that were most idiosyncratic in the shared--idiosyncratic diagnostic (Sec.\ref{sec:methods-shareddev-int}). (\subref{fig:brain-fs}) Difference between few-shot and zero-shot predictions after 10mins of calibration; results for more calibration sizes and other baselines can be found in App.\ref{app:fs_add_results}. }
\end{figure}
\section{Discussion and Conclusion} \label{sec:discussion}
The central insight of RABBiT is that accurate personalization in predicting language-evoked fMRI does not require fitting a new high-dimensional model for every participant. Instead, RABBiT separates prediction into a shared speech-to-brain mapping and a compact subject-specific correction pathway. This structure naturally supports both transfer regimes explored in this work: zero-shot prediction through the shared component, and few-shot adaptation through low-rank subject deviations.
RABBiT bridges a gap between prior brain-tuning approaches and recent foundation-style brain encoders. Brain-tuning improves speech representations for fMRI prediction, but still relies on participant-specific voxel-wise readouts \citep{moussa2025improving,moussa2025multibrain,vattikonda2025brainwavlm}. Foundation-style models move toward population-level prediction, but often depend on large backbones, subject-specific heads, or coarse spatial outputs \citep{dAscoli2026TribeV2,beliy2025brainlm}. In contrast, RABBiT shows that a compact audio-only encoder can achieve highly accurate group-level prediction while supporting efficient subject adaptation from only minutes of calibration data. Importantly, calibration updates only the compact SID deviation pathway rather than relearning the full speech-to-brain mapping.
The regional structure of these gains is particularly informative. Higher-order language regions both exhibit the largest inter-subject variability and benefit most from few-shot adaptation, indicating that individual differences are structured rather than noise-like. SID captures this structure in a compact representation that can be efficiently adapted with limited data.
The temporal brain transformer provides a complementary level of structure by learning ROI-specific temporal queries over the speech stream. Beyond improving prediction, these learned queries recover a coarse progression from auditory to higher-order language regions without explicit supervision, suggesting a promising framework for probing cortical language hierarchies.
Together, these results position RABBiT as a foundation model of language-evoked brain activity. The model predicts responses for unseen listeners without additional fMRI, adapts efficiently to new individuals, and exposes interpretable regional structure. More broadly, our findings suggest that effective brain foundation models should explicitly separate shared and individual variability and model how different brain regions integrate information over time.
%The main lesson from RABBiT is that personalization in predicting language-evoked fMRI does not have to mean fitting a new high-dimensional model for every subject. Standard voxel-wise encoders spend subject-specific data learning an entire cortical readout from scratch \citep{huth2016natural,antonello2024scaling}. RABBiT instead separates the problem into a shared speech-to-brain encoder and a small subject-correction pathway. This division explains the two transfer regimes: zero-shot prediction works when the shared pathway captures the reliable population response, and few-shot calibration works when the target subject only has to adjust the low-rank deviation from that response.
%This addresses a gap left by both brain-tuning and recent foundation-style brain encoders. Prior brain-tuning work showed that fMRI can improve speech representations, but prediction in a new participant still required fitting a participant-specific voxel-wise readout \citep{moussa2025improving,moussa2025multibrain,vattikonda2025brainwavlm}. Large multimodal encoders move closer to zero-shot population prediction, but typically rely on large backbones, subject-specific heads, or coarse output structure \citep{dAscoli2026TribeV2,beliy2025brainlm}. RABBiT shows that a compact audio-only encoder can reach the ceiling estimate for group-level language responses while also supporting efficient subject-specific adaptation. The few-shot result is especially important: with only minutes of calibration, updating the SID deviation pathway outperforms voxel-wise ridge trained on the same data. The calibration data are not used to relearn the whole map from speech to cortex; they are used to correct the parts of the population prediction that are most subject-specific.
%The regional pattern of few-shot gains highlight this most. The ROIs that benefit most from calibration are the same higher-order language regions that showed the largest leave-one-out drops in the shared--idiosyncratic diagnostic. This confirms our intuition: individual differences are not evenly distributed across cortex, and they are not simply noise around a group average. They have enough structure to be isolated, represented compactly, and recovered with little data. In this sense, SID is not just a compression mechanism; with enough data, it provides a practical coordinate system for studying how individual brains depart from the population response.
%The temporal brain transformer adds a complementary important structure. Rather than assigning every vertex an independent temporal readout, RABBiT learns ROI-level queries that decide what speech context each region should use. These queries are useful for prediction, but they are also analyzable model objects. The greedy walk through query-embedding space reconstructs a coarse speech-to-language trajectory from auditory cortex toward temporal, parietal, and frontal language regions, with no hierarchy supervision. This in itself could be a useful tool to investigate hypotheses about the language cortical hierarchy for future work.
%%This should not be read as a definitive anatomical map. It is stronger as a model-based analysis: the same parameters that route speech information for prediction also recover a recognizable organization of the language network, something voxel-wise ridge readouts do not expose directly.
%Together, the results support RABBiT as a step toward scalable, foundational encoding models of language-evoked brain activity. The model predicts the shared response of new listeners without new fMRI, adapts to individuals from short calibration scans, and exposes interpretable regional structure along the way. More broadly, it suggests that useful brain foundation models should not only get larger; they should make the right structure explicit: what is shared across people, what varies across individuals, and how different brain regions route information over time. By making our code and trained models publicly available, we aim to foster reproducibility and encourage further research on brain-tuning. We hope that our work will not only advance the integration of language models into cognitive neuroscience but also inspire new approaches to modeling human language comprehension.
\paragraph{Limitations and Future Work.} RABBiT is evaluated primarily on English naturalistic listening and auditory/language regions. Extending the evaluation to multilingual, conversational, reading, and multimodal settings will assess the generality of its shared and subject-specific representations. Second, the training cohort remains relatively small, and preliminary experiments combining datasets did not yield proportional gains, suggesting that scaling training across datasets may require improved handling of inter-dataset variability. Finally, we only partially explored scaling to larger speech backbones; a more systematic investigation of model scaling remains a direction for future work.
\section{Datasets and preprocessing}
\label{app:data}
All experiments use a common cortical space so that the trained encoder can be evaluated easily across datasets and participants without refitting. This section details the parcellation of ROIs, the additional details of training and evaluation data, and the exploratory mixed-data run that motivated training on Friends alone (App.\ref{app:mixed_data}).
\subsection{Brain ROIs Details}
\label{app:roi_deets}
% RABBiT predicts high-resolution responses on a shared cortical surface rather than dataset-specific voxel grids. We project all responses to \texttt{fsaverage6} and restrict evaluation to the 30 bilateral HCP-MMP1 ROIs used throughout the paper.
All fMRI is projected to FreeSurfer's \texttt{fsaverage6} surface ($40{,}962$ vertices per hemisphere; mean inter-vertex spacing $\sim$2\,mm) \citep{fischl1999high, fischl2012freesurfer} and restricted to $30$ bilateral HCP-MMP1 ROIs \citep{glasser2016multi}, totaling $\sim$$41{,}394$ vertices ($\sim$half of the full \texttt{fsaverage6} cortex). The $30$ ROIs cover the canonical auditory and language network plus sensorimotor, default-mode/theory-of-mind, and visual regions. We expand beyond language regions to test our model ability and efficiency to learn around 50\% of the cortical surface while being parameter-efficient. It stress tests the brain transformer's ability to scale the output size with substantial increase in the trainable parameters as with the full linear ridge models.
The $30$ ROIs are organized into $15$ functional groups (each with LH and RH instances), with HCP-MMP1 parcels enumerated for the auditory and language groups:
\begin{itemize}
\setlength\itemsep{0pt}
\item \textbf{Auditory.} \emph{aud\_primary} (A1); \emph{aud\_belt} (A4, A5, LBelt, MBelt, PBelt).
\item \textbf{Superior temporal and temporal pole.} \emph{stg\_sts} (STGa, STSda, STSdp, STSva, STSvp, STV); \emph{temporal\_pole} (TE1a, TE1m, TE1p, TE2a, TE2p, TGd, TGv); \emph{posterior\_temp} (PHT, TPOJ1, TPOJ2, TPOJ3).
\item \textbf{Parietal language.} \emph{angular\_gyrus} (PGi, PGs, PGp, PFm); \emph{supramarginal} (PFop, PFt, PFcm, PSL).
\item \textbf{Frontal language.} \emph{ifg} (44, 45, 47l); \emph{mfg\_dlpfc} (55b, IFJa, IFSp, SFL, 8Av, 8BL).
\item \textbf{Default-mode and theory-of-mind controls.} \emph{mpfc\_tom}; \emph{precuneus\_pcc}.
\item \textbf{Sensorimotor and insular controls.} \emph{motor}; \emph{insula\_fop}.
\item \textbf{Visual controls.} \emph{visual\_early} (V1--V4); \emph{visual\_higher} (V6, V6A, V7, V8, LO1, LO2, MT, MST, FST parcels).
\end{itemize}
\subsection{Training data}
\label{app:train_data}
We train on the Friends subset of CourtoisNeuroMod \citep{boyle2025friends, st2026cneuromod}: $S{=}6$ participants watching naturalistic audio-visual TV episodes at TR\,=\,$1.49$\,s, totaling $\sim$$565{,}000$ TRs ($\sim$$234$ hours, $\sim$$39$ hours per participant). RABBiT is audio-only, so only the audio track is used as input; we use the audio of an audio-visual corpus to keep input modality and recording pipeline comparable to our held-out audio-only evaluation cohorts.
The fMRI is provided in volumetric MNI space at 2\,mm resolution and projected onto \texttt{fsaverage6} surface vertices, with the $\sim$$41{,}394$ vertices belonging to the $30$ ROIs retained per clip. The released CourtoisNeuroMod data are already z-scored per voxel per run, so no further temporal centering is applied --- consistent with the SID shared-basis initialization (App.\ref{app:sid-init}). Within each subject's clips we hold out a fixed 4 clips of validation set and a fixed 4 clips for test set (held-out evaluations on training participants); the split is identical across subjects so validation losses are directly comparable.
\subsection{Evaluation data}
\label{app:eval_data}
The two held-out cohorts together include $324$ unseen participant--story instances --- $275$ across the $7$ Narratives stories \citep{nastase2021narratives} (the column total of Table\ref{tab:narratives_per_story}, with each story drawn from an independent participant pool) plus $49$ unique Le Petit Prince participants \citep{Li2022LePetitPrince} who heard all $9$ audiobook sections. None overlap with the Friends training cohort.
\subsubsection{Per-cohort details}
\label{app:eval_data:dets}
\textbf{Narratives \citep{nastase2021narratives}.} We use $7$ stories from the Narratives corpus (OpenNeuro \texttt{ds002345}), each with an independent participant pool: four longer stories and three shorter, large-cohort stories (Table\ref{tab:narratives_per_story}).
\begin{table}[h]
\centering
\small
\begin{tabular}{lrrrl}
\toprule
Story & TRs & Duration (min) & Participants & Calibrated audio onset (s) \\
\midrule
\textit{21styear} & 2249 & 56.2 & 20 & 0.0 \\
\textit{slumlordreach} & 1210 & 30.0 & 18 & 3.0 \\
\textit{tunnel} & 1037 & 25.5 & 23 & 0.0 \\
\textit{sherlock} & 725 & 18.0 & 36 & 3.0 \\
\textit{pieman} & 300 & 7.5 & 82 & 0.0 \\
\textit{notthefall} & 400 & 9.7 & 56 & 1.5 \\
\textit{prettymouth} & 475 & 11.9 & 40 & 1.5 \\
\midrule
\textbf{total / mean} & \textbf{6396} & \textbf{$\sim$159} & \textbf{275} & --- \\
\bottomrule
\end{tabular}
\caption{\textbf{Narratives held-out stories used in zero-shot evaluation.} TRs at the corpus's native TR\,=\,$1.5$\,s. Participant counts are independent across stories; the column total is the participant--story instance count. Calibrated audio onsets are validated as in App.\ref{app:eval_data:audio_onset}.}
\label{tab:narratives_per_story}
\end{table}
\textbf{Le Petit Prince \citep{Li2022LePetitPrince}.} The English audiobook portion of OpenNeuro \texttt{ds003643}: $49$ native English speakers listening to $9$ scanning sections totaling $\sim$$93$ minutes at TR\,=\,$2.0$\,s on a 3T GE Discovery MR750 (a different scanner from CourtoisNeuroMod or Narratives), providing an out-of-distribution test of RABBiT's transfer beyond the Narratives recording conventions. The released preprocessed runs already strip the leading silence at audio onset, so the audio plays at scanner-time zero and no per-section onset calibration is required (the A1-LH shift sweep peaks at $0$ or $+1$ TR for $6$ of the $9$ sections; the $3$ noise-dominated sections are evaluated at the same $0$-TR convention). We use an audio onset of $0$ for the LPP dataset.
\subsubsection{Audio--fMRI alignment and onset validation}
\label{app:eval_data:audio_onset}
The released stimulus audio for each Narratives story typically begins with a brief leading silence whose duration is neither zero nor the BIDS-listed music-start time. We therefore calibrate the audio-onset value using by estimating it first from metadata and then try multiple values for the shift on a small subset and then measure the A1-belt response activation start. This is used as a correction for the onset time estimated at the beginning; this onset is used consistently across models. Table\ref{tab:narratives_per_story} shows the estimated onset values that are used identically by every model (for $4$ of the $7$ stories the calibrated onset disagrees with the BIDS music-start time).
It's important to note that the audio-onset offset is a property of how the stimulus file was exported, not of the model or the brain --- the same number for every participant on a given story. Fixing it once per story removes a confounding free parameter from the evaluation, so residual differences across models reflect model quality rather than alignment search.
\subsection{Training with mixed data}
\label{app:mixed_data}
We added the Moth Radio Hour subset of \citep{lebel2023natural} (we chose $6$ participants, $\sim$$113{,}000$ TRs at TR\,=\,$2.0$\,s, $\sim$$63$ hours) to the Friends mix, yielding a $12$-participant combined cohort with the same SID architecture (the shared bases $\Phi$ was obtained from both dataset mixtures while the Idiosyncratic part remains per subject.). During training, we forced a $1{:}2$ Moth-to-Friends batch ratio per training block to account for the size mismatch between the datasets. Across the four narrative story zero-shot evaluation, the combined model achieves a story-mean vertex-wise $r$ of $0.108$ versus $0.117$ for Friends-only (an $\sim$$8\%$ relative drop). The combined model matches Friends-only on Friends-side validation loss, ruling out undertraining. We have also tried per-subject Full-rank alternatives (see Sec.\ref{sec:methods-zs}) and got similar behavior. This suggests that mixing datasets during training is not straightforward and scaling law doesn't apply linearly regardless of the training data sources, complementary to the findings in \citep{moussa2025multibrain}; thus, we defer this to future work.
\section{Shared–Idiosyncratic diagnostic and SID details}
\label{app:sid_deets}
% The SID readout is motivated by a simple empirical observation: some regions are well predicted by the group response, while others lose reliability when the target listener is excluded. This section gives the diagnostic, the resulting ROI pattern, and the initialization used to encode that split in the model.
% \subsection{Empirical Shared-Idiosyncratic Study}
% \label{app:sid_pre}
% We quantify how population-like each ROI is by comparing a subject's response to the group average with and without that subject included.
% \begin{figure}[h]
% \centering
% \begin{subfigure}[b]{0.9\textwidth}
% \includegraphics[width=1\textwidth]{figs/consistency_lang_avg_tunnel_21styear_slumlord.pdf}
% % \caption{Normalized alignment for primary auditory}
% % \label{fig:algin_scores_aud}
% \end{subfigure}
% \caption{Shared-vs-Idiosyncratic regions analyses for language responses.}
% \end{figure}
% \subsection{SID Bases and Initialization}
% \label{app:sid-init}
% SID is initialized from the same shared/residual split used in the diagnostic: the shared basis is estimated from the mean response, and the deviation bases from subject residuals around that mean.
% \subsection{Coefficient Choice and Explained Variance}
% \label{app:sid-dims}
% The SID ranks determine the tradeoff between compression and subject-specific capacity. We choose ranks that capture most ROI variance while keeping the few-shot update small.
The SID readout is motivated by a simple empirical observation: some cortical regions are well predicted by the group-mean response across listeners, while others lose reliability sharply when the target listener is excluded from that mean.
\subsection{The shared--idiosyncratic diagnostic}
\label{app:sid_pre}
For each ROI $i$ and subject $s$ with stimulus-aligned response $y^{(s)}_i \in \mathbb{R}^{T \times V_i}$, we compute the vertex-averaged Pearson correlation between $y^{(s)}_i$ and the cohort-mean response $\bar{y}_i$ (\emph{including} subject $s$), then repeat with the leave-one-out cohort-mean $\bar{y}^{(-s)}_i$. The drop between the two measures how much of subject $s$'s response is captured only when they are part of the mean: a large drop means the LOO mean predicts subject $s$ poorly, so the response in ROI $i$ is more idiosyncratic than shared.
Fig.\ref{fig:sid_diagnostic} plots this contrast across the auditory and language ROIs, averaged over three Narratives stories. Early auditory regions (A1, auditory belt) and superior temporal cortex remain stable under leave-one-out averaging --- their responses are largely shared, consistent with high inter-subject correlations in naturalistic listening \citep{nastase2021narratives}. Higher-order language and theory-of-mind regions (IFG, MFG/DLPFC, supramarginal, angular gyrus, mPFC) show substantial drops, indicating subject-specific structure that a population mean alone cannot capture. SID's per-subject deviation pathway gives these idiosyncratic regions a bounded budget of subject-specific structure to draw on; the shared component handles the regions on the shared end.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.9\textwidth}
\includegraphics[width=1\textwidth]{figs/consistency_lang_avg_tunnel_21styear_slumlord.pdf}
\end{subfigure}
\caption{\textbf{Shared--idiosyncratic diagnostic across language and auditory ROIs.} Vertex-averaged correlation of each subject's response with the cohort-mean fMRI when the subject is included versus excluded, averaged across three Narratives stories (\textit{21styear}, \textit{tunnel}, \textit{slumlordreach}) and across participants. Auditory and STG/STS regions remain stable; higher-order language regions show substantial drops.}
\label{fig:sid_diagnostic}
\end{figure}
\subsection{SID Bases construction and initialization}
\label{app:sid-init}
The SID bases are warm-started from an empirical decomposition of the training fMRI. For each ROI $i$ and training subject $s$, we run a per-ROI PCA on $20{,}000$ subsampled TRs from that subject's clips, taking the top $M=100$ principal components as a per-subject basis $\Phi^{(s)}_i$. The shared basis $\Phi_i$ is the cross-subject average re-orthogonalized via SVD: $\Phi_i = \mathrm{SVD}\big(S^{-1}\sum_s \Phi^{(s)}_i\big)$. The idiosyncratic (deviation) basis $\Delta_{i,s}$ is the SVD of $\Phi^{(s)}_i - \Phi_i$ truncated to the top $R=15$ right singular vectors --- the dominant directions in which subject $s$ departs from the shared basis. During joint training the bases are anchored to their initialization with $\mathcal{L}_{\mathrm{anchor}} = \lambda_{\Phi} \sum_i \|\Phi_i - \Phi_i^{(0)}\|_F^2 + \lambda_{\Delta} \sum_{i,s} \|\Delta_{i,s} - \Delta_{i,s}^{(0)}\|_F^2$ with $\lambda_{\Phi}{=}\lambda_{\Delta}{=}10^{-3}$. This value was chosen empirically based on grid log-scale hypereparamter search over a subset of the training data. We find very strong values to prevent generalization and low values to cause basis drift easily, both of which hurt performance. %plus a soft orthogonality term $\mathcal{L}_{\mathrm{ortho}} = \mu \sum_i \|\Phi_i^\top \Phi_i - I_M\|_F^2$ with $\mu{=}10^{-4}$ that prevents column collapse.
% The trained deviation bases retain meaningful per-subject structure rather than collapsing. Across the $30$ ROIs and all $\binom{6}{2}{=}15$ subject pairs, the mean per-rank cosine similarity between matched deviation bases is $+0.026$ (essentially uncorrelated); the aggregate row-subspace overlap is $0.82$ --- well above the random baseline $\sqrt{R/V_i} \approx 0.14$ but below $1.0$, so bases share a coarse subspace per ROI while pointing in distinct subject-specific directions. Linguistic regions share the most cross-subject structure (overlap $0.88$--$0.89$ in IFG, supramarginal, posterior temporal), visual / motor regions the least ($0.65$--$0.74$). This is what makes the per-subject deviation pathway --- and the few-shot adaptation that updates only the deviation coefficients --- structurally non-trivial.
\textbf{ More details on the Idiosyncratic (Deviation) bases $\Delta_{i,s}$.} The per-subject deviation basis is built from each subject's residual in basis-coefficient space: we form $\Phi^{(s)}_i - \Phi_i$, take its SVD, and keep the top $R=15$ right singular vectors as $\Delta_{i,s} \in \mathbb{R}^{R \times V_i}$. By construction, $\Delta_{i,s}$ spans the dominant directions in which subject $s$'s within-ROI response geometry departs from the cohort-shared basis.
% \textbf{Anchor regularization.} During joint training the bases drift from their PCA/SVD initialization as the backbone and the brain transformer change. We penalize that drift with an $L_2$ anchor,
% \[
% \mathcal{L}_{\mathrm{anchor}} \;=\; \lambda_{\Phi} \sum_{i} \big\|\Phi_i - \Phi_i^{(0)}\big\|_F^2 \;+\; \lambda_{\Delta} \sum_{i,s} \big\|\Delta_{i,s} - \Delta_{i,s}^{(0)}\big\|_F^2,
% \]
% where $\Phi_i^{(0)}$ and $\Delta_{i,s}^{(0)}$ are the (frozen) initial bases. With anchor weights too small the bases drift uncontrollably and zero-shot accuracy collapses, because the prediction subspace can no longer represent the cross-subject structure it was initialized to capture; with anchor weights too large the bases are effectively frozen, and few-shot adaptation has nothing to leverage. We use $\lambda_{\Phi} = \lambda_{\Delta} = 10^{-3}$ in all reported runs.
\subsection{Choice of shared and deviation ranks}
\label{app:sid-dims}
The shared rank $M$ is chosen as a coverage threshold: at $M=100$ the cumulative explained variance of the per-subject PCA reaches $0.993$ averaged across $6$ subjects $\times 30$ ROIs (Table\ref{tab:sid_var_K}); only bilateral motor cortex ($V \approx 5{,}000$) is below $95\%$. Auditory cortex saturates by $M{=}20$, language ROIs by $M{=}50$. $M{=}100$ is conservative; $M{=}50$ would lose only $\sim$$2\%$ mean coverage.
\begin{table}[h]
\centering
\small
\begin{tabular}{rrrrrr}
\toprule
$M$ & mean cum.\ var. & std & min ROI & median ROI & max ROI \\
\midrule
5 & 0.737 & 0.114 & 0.463 & 0.755 & 0.978 \\
10 & 0.836 & 0.095 & 0.587 & 0.863 & 0.997 \\
20 & 0.914 & 0.071 & 0.697 & 0.938 & 1.000 \\
30 & 0.947 & 0.055 & 0.763 & 0.970 & 1.000 \\
50 & 0.975 & 0.036 & 0.843 & 0.990 & 1.000 \\
75 & 0.988 & 0.022 & 0.900 & 0.997 & 1.000 \\
\textbf{100} & \textbf{0.993} & \textbf{0.015} & \textbf{0.933} & \textbf{0.999} & \textbf{1.000} \\
\bottomrule
\end{tabular}
\caption{\textbf{Cumulative explained variance of the per-subject shared bases as a function of $M$.} $n{=}180$ cells per row ($6$ subjects $\times 30$ ROIs). The default $M{=}100$ retains $99.3\%$ of within-ROI variance on average.}
\label{tab:sid_var_K}
\end{table}
The deviation rank $R=15$ is a regularization budget, not a coverage threshold: at $R{=}15$ the deviation basis retains only $29\%$ of the per-subject residual variation. To reach $80$--$90\%$ residual energy would require $R \approx 50$--$75$, granting the readout disproportionate per-subject flexibility. A joint $M{=}200$, $R{=}30$ ablation at the same backbone and transformer mostly probes the $R$ axis ($M{>}100$ adds at most $0.7\%$ coverage, while $R{:}15{\to}30$ nearly doubles deviation flexibility; in our experiments, this scaling does not improve held-out performance.
% \begin{table}[h]
% \centering
% \small
% \begin{tabular}{rrrrrr}
% \toprule
% $R$ & mean residual varience & std & min ROI & median ROI & max ROI \\
% \midrule
% 5 & 0.100 & 0.006 & 0.095 & 0.099 & 0.133 \\
% 10 & 0.197 & 0.011 & 0.187 & 0.196 & 0.263 \\
% \textbf{15} & \textbf{0.292} & \textbf{0.017} & \textbf{0.277} & \textbf{0.289} & \textbf{0.387} \\
% 20 & 0.383 & 0.021 & 0.363 & 0.380 & 0.504 \\
% 30 & 0.550 & 0.027 & 0.522 & 0.546 & 0.705 \\
% 50 & 0.805 & 0.026 & 0.777 & 0.802 & 0.943 \\
% 75 & 0.966 & 0.008 & 0.951 & 0.966 & 1.000 \\
% \bottomrule
% \end{tabular}
% \caption{\textbf{Varience explained by the per-subject deviation bases as a function of $R$.} The default $R{=}15$ retains $29\%$ of residual energy --- a regularization budget, not a coverage threshold.}
% \label{tab:sid_var_R}
% \end{table}
\section{RABBiT Architecture and Training Details }
\label{app:training}
This section gives the full architecture, loss, and optimization details that reproduce the headline configuration.
\subsection{Speech backbone and brain-tuning}
\label{app:btune}
The speech backbone supplies RABBiT with a temporal stream of frame-level audio tokens that the brain transformer routes into ROI latents. We use Wav2Vec2.0-base \citep{baevski2020wav2vec} as the backbone in all reported runs --- a $90$M-parameter self-supervised model trained on $960$ hours of LibriSpeech --- because it is the standard acoustic SSL backbone in the brain-tuning lineage \citep{moussa2025improving, moussa2025multibrain, vattikonda2025brainwavlm} and produces a feature sequence at $50$\,Hz ($20$\,ms per frame), giving the brain transformer a fine-grained acoustic stream to pool over.
\textbf{LoRA adapters.} We brain-tune the backbone with parameter-efficient LoRA \citep{hu2022lora} of rank $r=8$ (after ablation; Fig.\ref{fig:lora_rank}), inserted only in the linear projections of the audio-encoder transformer blocks. The pretrained weights of every module are kept frozen; only the LoRA matrices are updated in the backbone over the course of training. This keeps the backbone shared across all training subjects this adds approximately $0.6$M trainable backbone parameters compared to the $90$M frozen body.
\textbf{Learning rate and Freeze warmup.} We use a learning rate of $1e-4$ with a linear warmup period of $10\%$ and then a linear decay over epochs. For the first $3$ epochs the backbone is held entirely frozen and only the brain transformer and SID readout are updated; LoRA training begins thereafter, governed by the same early-stopping criterion (patience $5$).
\textbf{Training hardware} We use AdamW optimizer to update the trainable paramters of the model; we train with a batch size of 256 for 30 epochs with early stopping available with a 5 epoch patience. The training takes around 10 hours for 30 epochs on a single H200 GPU, with early stopping usually stopping the training at 10-13 epochs.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.6\textwidth}
\includegraphics[width=1\textwidth]{figs/app/lpp_rank_curve_all.pdf}
\end{subfigure}
\caption{\textbf{LoRA rank ablation on Le Petit Prince zero-shot.} Group-level $r_{\mathrm{group}}$ on language ROIs as a function of LoRA rank. Performance peaks at rank $8$ (our default) and is robust across nearby ranks; very small ranks under-parameterize the backbone adaptation and very large ranks begin to overfit the limited fMRI training signal.}
\label{fig:lora_rank}
\end{figure}
\subsection{Temporal brain transformer architecture}
\label{app:trans}
The temporal brain transformer is a stack of transformer cross-attention blocks (Fig.\ref{fig:trans} in main text). We chose $L=2$ layers for it and a hidden dimension $d_o=256$, $H=8$ heads, and FFN dimension $4 d_o = 1024$ with GELU non-linearity. Backbone features at $50$\,Hz are projected to $d_o$ via a learned linear layer with sinusoidal positional encoding added to keys at each cross-attention. The $30$ ROI queries ($q_i \in \mathbb{R}^{d_o}$, one per bilateral ROI; $\sim$$7.7$K params) are initialized to zero and combined with a learned per-ROI positional embedding. Each block applies self-attention over the queries, then cross-attention from queries to backbone tokens (values gathered without positional info), then FFN. The second-layer self-attention is the load-bearing one for cross-ROI coordination: its inputs carry stimulus-specific information from the first cross-attention, which lets each query condition its second-pass extraction on the global brain state.
Two ablations on Le Petit Prince zero-shot support these choices: depth $L=2$ is the empirical peak (Fig.\ref{fig:trans_depth}), with deeper variants ($L \in \{4, 8, 10\}$) regressing on the all-vertex aggregate by $0.008$--$0.010$ $r$; hidden dimension $d_o = 256$ is also the peak (Fig.\ref{fig:trans_hidden}), with both narrower ($d_o \in \{64, 128\}$) and wider ($d_o = 512$) configurations regressing on all three scopes.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.6\textwidth}
\includegraphics[width=1\textwidth]{figs/app/lpp_depth_curve_full_all.pdf}
\end{subfigure}
\caption{\textbf{Brain-transformer depth ablation on Le Petit Prince zero-shot.} Group-level $r_{\mathrm{group}}$ on language ROIs as a function of decoder depth $L$. Performance peaks at $L=2$ (our default); deeper variants do not improve held-out language-ROI prediction.}
\label{fig:trans_depth}
\end{figure}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.6\textwidth}
\includegraphics[width=1\textwidth]{figs/app/lpp_hidden_curve_full_all.pdf}
\end{subfigure}
\caption{\textbf{Brain-transformer hidden dim ablation on Le Petit Prince zero-shot.} Group-level $r_{\mathrm{group}}$ on language ROIs as a function of decoder hidden dimension $H$. Performance peaks at $256$ (our default); deeper variants do not improve held-out language-ROI prediction.}
\label{fig:trans_hidden}
\end{figure}
\subsection{SID prediction and zero-shot inference}
\label{app:sid_pred}
After the brain transformer produces a per-ROI latent $z_i \in \mathbb{R}^{d_o}$, the SID readout maps $z_i$ to vertex-level fMRI in two parallel pathways: shared coefficients $\pi_i(z_i) \in \mathbb{R}^M$ combine with the shared basis $\Phi_i$, and deviation coefficients $\rho_i(z_i) \in \mathbb{R}^R$ combine with the per-subject deviation basis $\Delta_{i,s}$ (both $\pi_i, \rho_i$ are linear). For ROI $i$ and subject $s$,
\[
\hat{y}^{(s)}_i \;=\; \pi_i(z_i)\, \Phi_i \;+\; \rho_i(z_i)\, \Delta_{i,s}.
\]
For a held-out subject the idiosyncratic (deviation) basis $\Delta_{i,s}$ is unknown; we replace it with the cohort average $\bar{\Delta}_i = \frac{1}{S}\sum_s \Delta_{i,s}$, giving $\hat{y}_i = \pi_i(z_i)\,\Phi_i + \rho_i(z_i)\,\bar{\Delta}_i$. This single substitution is the only change between training-time and zero-shot inference. Replacing the cohort-averaged deviation pathway with zero (predicting only the shared component) recovers approximately half the signal of the averaged-deviation prediction across the auditory and language ROIs of the held-out cohorts: even when no individual training-subject's deviation basis matches the new participant, the cohort average captures population-level structure the new participant inherits as a population draw.
\subsection{Loss and training details}
\label{app:loss_train}
Training optimizes the full encoder end-to-end against measured fMRI responses across the six Friends training participants (App.\ref{app:train_data}), using the loss
\[
\mathcal{L} \;=\; \sum_i \big(\,s_i\,\mathcal{L}^{\mathrm{L2}}_i \;+\;\lambda_{\mathrm{corr}}\,\mathcal{L}^{\mathrm{corr}}_i\,\big) \;+\; \mathcal{L}_{\mathrm{anchor}}
\]
where $\mathcal{L}^{\mathrm{L2}}_i$ is the per-ROI vertex-wise squared error between predicted and measured z-scored fMRI, $\mathcal{L}^{\mathrm{corr}}_i$ is the per-ROI correlation loss ($1$ minus the per-ROI vertex-mean Pearson correlation), and $s_i$ is a per-ROI learned scalar that lets each ROI calibrate its own squared-error weight independently. The anchor term ($\mathcal{L}_{\mathrm{anchor}}$) is defined in App.\ref{app:sid-init}.
\textbf{Why the per-ROI scaled-correlation form.} The squared-error term alone produces predictions well-calibrated in magnitude but only weakly correlated with the target on the higher-order ROIs whose response geometry is harder to recover; the correlation term alone produces predictions that are correlated but unscaled, and trains poorly because correlation gradients are scale-invariant. The scaled-correlation form combines the two, so each ROI gets gradient pressure on both correlation and magnitude, with a per-ROI learned scale that absorbs cross-ROI variation in noise floor (auditory ROIs have a much higher signal-to-noise ratio than e.g. mPFC and need less L2 weight to be well-fit). We use $\lambda_{\mathrm{corr}} = 1$ throughout. Our choice for this value was also done empirically, similar to the anchor search (see App.\ref{app:sid-init})
\textbf{Optimization.} Parameters are optimized with AdamW. The brain transformer, coefficient maps, and LoRA adapters share a base learning rate; the SID bases use a smaller learning rate that keeps them close to their PCA/SVD initialization while still allowing fine-grained refinement under gradients from the brain-tuned backbone.
% \textbf{Trainable parameter accounting.} The trainable footprint is approximately $12.9$M parameters: $\sim$$1.2$M from the LoRA backbone adapters (rank $16$ on the $90$M-frozen body); $\sim$$2.4$M from the temporal brain transformer (input projection $\sim$$197$K, query embeddings $\sim$$7.7$K, two decoder blocks $\sim$$2.1$M); $\sim$$0.9$M from the SID coefficient maps ($\pi_i$ at $\sim$$770$K and $\rho_i$ at $\sim$$115$K across the $30$ ROIs); and $\sim$$7.9$M from the SID bases themselves (shared $\Phi_i$ at $\sim$$4.1$M for $M{=}100$ across all $V \approx 41{,}394$ vertices, plus per-subject $\Delta_{i,s}$ at $\sim$$3.7$M for $R{=}15$ across $S{=}6$ training subjects).
% \subsection{Parameter count}?
\section{Zero-shot evaluation and Baselines}
\label{app:zs_baselines}
Zero-shot evaluation tests whether the trained encoder predicts the reliable group response of unseen listeners without using any fMRI from those listeners.
\subsection{Accuracy metrics and the inter-subject consistency}
\label{app:zs_acc}
We report two metrics per (model, ROI, story). The group-level correlation $r_{\mathrm{group}}^{(i)} = \mathrm{vertex\_mean}_{v \in i}\big[\mathrm{corr}_t(\hat{y}[t,v], \bar{y}^{(\text{cohort})}[t,v])\big]$ targets the cohort-mean fMRI; the per-subject consistency $r^{(i,s)}$ targets each individual subject's fMRI and is averaged across the $N$ subjects in the cohort. Both inputs are z-scored per vertex over time before correlation. The leave-one-out inter-subject correlation (LOO ISC), computed by holding out each subject in turn and correlating the mean fMRI of the other $N{-}1$ with the held-out subject, is the cleanest stimulus-only predictor the cohort itself provides and serves as the natural consistency (ceiling) for $r^{(i,s)}$. It is not a strict ceiling for $r_{\mathrm{group}}$: a model whose prediction is identical for every subject can exceed LOO ISC in $r_{\mathrm{group}}$ when the cohort is small or the ROI is highly stereotyped, because the cohort-mean target is itself noisy at small $N$ but the model's prediction is noiseless across subjects (RABBiT's A1 reaches $r_{\mathrm{group}} \approx 0.55$ vs LOO ISC $\approx 0.27$ on the headline Narratives stories).
An important note is that for RABBiT, we use the same audio onset as the other models and add the hemodynamic delay of $3$TRs as well, a standard practice since audio at time $t$ correspond a later fMRI, typically 4-6 seconds after onset \citep{antonello2024scaling}; TRIBEv2 does the same but it bakes the delay into its prediction and training from the start.
% Each held-out story has a per-cohort residual hemodynamic offset combining the audio-file leading silence (calibrated once per story; App.\ref{app:eval_data:audio_onset}) and any residual difference between the encoder's learned hemodynamic compensation and the cohort's preprocessing pipeline. We report each model at its own per-story optimum $\Delta t^{\!*}$ (TRs), selected via a fine sweep on left-hemisphere A1 $r_{\mathrm{group}}$ at $0.1$-TR resolution over $[-3, +6]$ TRs. A1 LH is sharply peaked and stimulus-locked, which avoids noise-driven shift selection from weaker models and contains the per-ROI report's double-dipping to a single ROI (the other $17$ language ROIs are reported at a shift selected by an independent ROI). The selected $\Delta t^{\!*}$ values agree across encoder variants within $\sim$$0.3$ TR (per-story Pearson $r = 0.9985$ across $15$ stories between RABBiT and the WavLM-large variant, and again between RABBiT and the Per-Subject Full-Rank baseline), confirming that per-story shift is a stim-driven property of the audio--scanner alignment, not an encoder-specific artifact.
\subsection{Linear and Per-Subject Full-Rank baselines}
\label{app:indp_baselines}
\textbf{Linear baseline.} The Linear baseline removes everything that distinguishes RABBiT from a frozen-feature linear probe: a frozen Wav2Vec2.0-base, mean-pooling over time within each TR-aligned window to a single $768$-dim vector, and a per-ROI linear layer mapping that vector to all $V_i$ vertices in ROI $i$. There is no temporal brain transformer and no SID; the only trainable parameters are the per-ROI linear readouts. Any gap between RABBiT and the Linear baseline must come from brain-tuning, the temporal brain transformer, or SID, not from the backbone or the vertex parcellation.
\textbf{Per-Subject Full-Rank baseline.} The Per-Subject Full-Rank baseline preserves the brain-tuned backbone and the temporal brain transformer but replaces the SID readout with one independent full-rank linear layer per ROI per training subject (no shared--deviation factorization). This adds $\sim$$5\times$ more total parameters than RABBiT (per-subject heads grow linearly with the cohort) but otherwise matches RABBiT exactly. If RABBiT matches or beats this variant in zero-shot, the SID compression has no measurable accuracy cost --- the bounded shared-plus-deviation parametrization is at least as expressive as independent per-subject heads at this cohort size.
\subsection{TRIBEv2 baseline}
\label{app:tribe-baseline}
TRIBEv2 \citep{dAscoli2026TribeV2} is a tri-modal encoder over video, audio, and text trained on multiple large public fMRI corpora, with frozen LLaMA / Wav2Vec-BERT / V-JEPA backbones, learned MLP projectors per modality, an $8$-layer cross-modal transformer, and subject-conditioned linear output heads predicting onto $\sim$$20{,}000$ fsaverage5 vertices at $1$\,Hz with a $5$\,s hemodynamic offset baked into its training contract. \emph{Vision modality is zeroed at our inference and must be disclosed:} our held-out cohorts are pure naturalistic listening, so TRIBEv2's vision input is replaced with a zero tensor and the visual contribution to the cross-modal combiner reduces to a constant baseline. (TRIBEv2 was trained with $30\%$ modality dropout, so missing-modality input is in-distribution, but the model still optimizes for the joint case.) RABBiT outperforming TRIBEv2 on visual ROIs is therefore an audio-only-vs-audio-only comparison.
We run TRIBEv2 on each held-out story with the released model weights, chunking the audio at $62.4375$\,s per chunk and stitching per-chunk predictions; the stream is aligned to scanner time using the calibrated audio-onset values of App.\ref{app:eval_data:audio_onset} with the $5$\,s hemodynamic offset unmodified. Whole-brain $r_{\mathrm{group}}$ on a subset of Narratives stories falls within the range reported in the original TRIBEv2 paper (Fig.\ref{fig:tribe_validation}), confirming that our inference and alignment pipeline represents the baseline fairly.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includegraphics[width=1\textwidth]{figs/tribe_rgroup_WholeBrain.png}
\end{subfigure}
\caption{\textbf{Whole-brain $r_{\mathrm{group}}$ for TRIBEv2 across several Narratives stories.} TRIBEv2 predictions correlated against cohort-mean fMRI on the full cortical surface, with the inter-subject ceiling overlaid per story. Numbers fall within the range reported in the original TRIBEv2 paper.}
\label{fig:tribe_validation}
\end{figure}
% \subsection{Additional zero-shot results}
% \label{app:zs_detailed_res}
% The main text reports the language-ROI aggregate and the per-ROI comparison; here we provide the supporting per-story tables, the brain-tuning ablation that asks where each model's gain comes from, and the ROI-subset ablation that asks whether predicting only language ROIs would be sufficient.
% \textbf{Per-story $r_{\mathrm{group}}$ on Narratives and Le Petit Prince.} Table\ref{tab:zs_per_story} reports the language-aggregate $r_{\mathrm{group}}$ per story, with the LOO inter-subject ceiling alongside. Across the $7$ Narratives stories, RABBiT exceeds the LOO ceiling on the four small-cohort stories ($N \leq 36$) and approaches but does not exceed it on the three larger-cohort stories ($N \in \{40, 56, 82\}$), where the ceiling itself rises with cohort size; TRIBEv2 follows the same qualitative pattern at lower absolute values; the Linear baseline crosses the ceiling only on \textit{tunnel}. Across the $9$ Le Petit Prince sections, RABBiT exceeds the LOO ceiling and TRIBEv2 by $\sim$$+0.02$ and $\sim$$+0.01$ respectively on the language aggregate at matched temporal shift.
% \begin{table}[h]
% \centering
% \small
% \begin{tabular}{lrrrrr}
% \toprule
% Story & $N$ & LOO ISC & RABBiT & TRIBEv2 & Linear \\
% \midrule
% \textit{21styear} & 20 & 0.145 & \textbf{0.281} & 0.107 & \textbf{0.177} \\
% \textit{slumlordreach} & 18 & 0.136 & \textbf{0.195} & \textbf{0.151} & 0.107 \\
% \textit{tunnel} & 23 & 0.158 & \textbf{0.261} & \textbf{0.188} & \textbf{0.213} \\
% \textit{sherlock} & 36 & 0.096 & \textbf{0.151} & \textbf{0.125} & \textbf{0.116} \\
% \textit{pieman} & 82 & 0.219 & 0.172 & 0.110 & 0.072 \\
% \textit{notthefall} & 56 & 0.295 & 0.175 & 0.147 & $-0.007$ \\
% \textit{prettymouth} & 40 & 0.343 & 0.157 & 0.022 & 0.062 \\
% \midrule
% \textbf{Narratives mean (7 stories)} & --- & 0.199 & \textbf{0.199} & 0.121 & 0.106 \\
% \midrule
% \textbf{LPP mean (9 sections, 49 subj)} & 49 & 0.163 & \textbf{0.184} & \textbf{0.172} & 0.106 \\
% \bottomrule
% \end{tabular}
% \caption{\textbf{Per-story group-level correlation $r_{\mathrm{group}}$ on the language ROIs (LH+RH).} Bold entries beat the LOO inter-subject ceiling for that story. RABBiT exceeds the ceiling on every story whose cohort size is at most $36$; on the larger-cohort stories the ceiling itself rises and no model reaches it. The LPP row is the mean across $9$ audiobook sections, all using the same $49$ participants.}
% \label{tab:zs_per_story}
% \end{table}
% \textbf{Brain-tuning ablation.} The headline gap between RABBiT and TRIBEv2 closes substantially when RABBiT's backbone is held frozen (no LoRA adapters; readout still trains). Table\ref{tab:btune_ablation} reports the language-aggregate $r_{\mathrm{group}}$ for the four-way comparison: RABBiT (tuned), RABBiT with no brain-tuning, TRIBEv2, and the Linear baseline, with the LOO inter-subject ceiling alongside. Two findings, in opposite directions:
% \begin{itemize}
% \setlength\itemsep{0pt}
% \item On Narratives, RABBiT-no-brain-tuning lands at $r_{\mathrm{group}} \approx 0.15$ on the language aggregate, statistically indistinguishable from TRIBEv2 ($\approx 0.14$); the $+0.06$ lift to RABBiT-tuned is what places RABBiT above both.
% \item On Le Petit Prince, RABBiT-no-brain-tuning \emph{under}-performs TRIBEv2 by $\sim$$0.05$; brain-tuning does not just lift RABBiT past TRIBEv2 but rescues it from a starting point below TRIBEv2.
% \end{itemize}
% The interpretation is that brain-tuning is the lever, but it serves two different purposes: in-distribution it pushes a TRIBE-equivalent starting point above TRIBEv2; out-of-distribution it rescues an under-performing starting point past TRIBEv2.
% \begin{table}[h]
% \centering
% \small
% \begin{tabular}{lrrrrr}
% \toprule
% Cohort / scope & RABBiT (tuned) & RABBiT (no brain-tuning) & TRIBEv2 & Linear & LOO ISC \\
% \midrule
% Narratives, lang (LH+RH) & \textbf{0.201} & 0.146 & 0.143 & 0.101 & 0.192 \\
% Narratives, lang LH & \textbf{0.218} & 0.150 & 0.150 & 0.077 & 0.185 \\
% Narratives, lang RH & \textbf{0.186} & 0.142 & 0.135 & 0.125 & 0.199 \\
% LPP, lang (LH+RH) & \textbf{0.195} & 0.103 & 0.153 & 0.062 & 0.163 \\
% LPP, lang LH & \textbf{0.200} & 0.109 & 0.151 & 0.076 & 0.161 \\
% LPP, lang RH & \textbf{0.190} & 0.096 & 0.154 & 0.048 & 0.165 \\
% \bottomrule
% \end{tabular}
% \caption{\textbf{Brain-tuning ablation across two evaluation cohorts.} Group-level $r_{\mathrm{group}}$ on language ROIs, comparing RABBiT (with brain-tuning), RABBiT with the backbone frozen (no LoRA), TRIBEv2, and the Linear baseline; the LOO inter-subject ceiling is shown alongside. RABBiT-no-brain-tuning lands close to TRIBEv2 on Narratives but under-performs TRIBEv2 on Le Petit Prince; brain-tuning lifts RABBiT past both baselines on both cohorts.}
% \label{tab:btune_ablation}
% \end{table}
% \textbf{ROI-subset ablation.} A natural question is whether RABBiT's gains require predicting all $30$ ROIs, or whether a model trained on a smaller language-only ROI set would do equally well. We test this with two reduced-ROI variants of RABBiT, retaining the same backbone, transformer, and SID architecture but trained against fewer ROIs: a $16$-ROI \emph{language-only} subset (early auditory, STG/STS, temporal pole, posterior temporal, angular gyrus, supramarginal, IFG, MFG/DLPFC, bilaterally) and a $10$-ROI \emph{early-language} subset (the auditory and superior-temporal subset of the previous). On Le Petit Prince ($n{=}9$ sections per ROI), both subset variants improve over the full $30$-ROI model on the same language ROIs by an aggregate $+0.012$--$+0.014$ $r_{\mathrm{group}}$ (paired Wilcoxon, $p = 0.002$ for language-only and $p = 0.044$ for early-language), with the largest per-ROI gains in angular gyrus, temporal pole, and MFG. On Narratives ($n{=}4$ headline stories per ROI) the same comparison is under-powered --- the minimum achievable Wilcoxon $p$-value at $n{=}4$ is $0.125$ --- and shows no significant aggregate effect. The LPP result indicates that wider ROI coverage is not free at small training-cohort sizes: trading off coverage for ROI-specific capacity is a defensible alternative configuration for a language-centric application, while the full $30$-ROI model is the right default for an encoder evaluated as a foundation model.
\section{Few-shot Calibration and Baselines}
\label{app:fs_deets}
Few-shot evaluation tests whether a short calibration scan can recover listener-specific signal without refitting a full voxel-wise model.
\subsection{Calibration setup}
\label{app:fs_calib}
We use the longest single-stimulus Narratives cohort, \textit{21styear} (56 minutes per participant, $20$ participants from App.\ref{app:eval_data:dets}), as the headline benchmark; one participant (sub-$244$) is excluded for quality reasons, leaving $19$. A second, smaller-stimulus replication on \textit{slumlordreach} ($\sim$$30$ minutes, $18$ participants) is in App.\ref{app:fs_add_results}.
For each participant we hold out a fixed test segment (the last $400$ TRs of the story, $\sim$$10$ minutes) and use a contiguous prefix as the calibration window. We sweep five calibration ($C$ for short) durations: $200$, $400$, $800$, $1200$, $1600$ TRs ($\sim$$5$, $10$, $20$, $30$, $40$ minutes at TR\,=\,$1.49$\,s); the test segment is identical across calibration sizes so any difference in test-set $r$ reflects calibration size only. We report per-subject vertex-mean $r$ on the test segment, averaged across the $19$ participants with $\pm$standard-error bands. Cross-validation across the full story would let calibration size co-vary with test position, mixing the calibration-size effect with temporal-coherence effects; the fixed-test protocol avoids this confound.
\subsection{RABBiT few-shot adaptation}
\label{app:fs_math}
RABBiT personalization freezes the backbone, the brain transformer, the SID shared bases $\Phi_i$, the shared coefficient maps $\pi_i$, and the per-subject deviation bases (initialized at the cohort average $\bar{\Delta}_i$ used at zero-shot inference). The only parameters updated are the per-ROI deviation coefficient maps $\rho_i: \mathbb{R}^{d_o} \to \mathbb{R}^{R}$, totaling $\sim$$115{,}000$ parameters across the $30$ ROIs. We minimize a per-ROI vertex-wise correlation loss with an anchor regularizer that penalizes drift of $\rho_i$ from its pretrained value:
\[
\mathcal{L}^{\mathrm{fs}} \;=\; \sum_i \mathcal{L}^{\mathrm{corr}}_i(\rho_i)\;+\; \lambda_{\rho} \sum_i \big\|\rho_i - \rho_i^{(0)}\big\|_F^2.
\]
The anchor weight $\lambda_{\rho}$ is the only material hyperparameter at calibration time. At calibration sizes $C \geq 200$ TRs, $\lambda_{\rho} = 5{\times}10^{-5}$ is at or near the held-out optimum across a $4$-subject probe sweep (per-subject optimum varies $10^{-5}$ to $10^{-3}$, but population-mean held-out $r$ is flat across this range to within $\sim$$0.005$). At smaller calibrations ($C \leq 100$) it is too weak: occasional subjects collapse below their own zero-shot performance, and a stronger $\lambda_{\rho} = 10^{-3}$ stabilizes them at the cost of $\sim$$0.005$--$0.01$ relative loss at $C \geq 800$. We use $\lambda_{\rho} = 5{\times}10^{-5}$ in all reported headline saturation curves. Optimization is AdamW for several hundred steps at a fixed learning rate, held constant across calibration sizes.
\subsection{Ridge baselines}
\label{app:fs_ridge}
We compare RABBiT few-shot calibration against two voxel-wise ridge encoders trained on the same per-subject calibration window. Both follow the canonical pipeline of \citep{huth2016natural} and \citep{antonello2024scaling}: extract per-frame audio features from a frozen backbone, build a delayed stimulus matrix using FIR taps spanning the hemodynamic window, z-score per fold, fit ridge regression per vertex via cross-validated alpha selection on a GPU implementation of bootstrap ridge.
The pretrained-features ridge uses features from a frozen Wav2Vec2.0-base (no brain-tuning) --- the canonical subject-specific encoding setup of the field. The brain-tuned-features ridge uses features from a multi-brain-tuned Wav2Vec2.0-base \citep{moussa2025multibrain} --- a backbone that has already absorbed cross-subject signal during multi-subject brain-tuning training but has no learned readout structure. The performance gap between the two ridge variants and RABBiT few-shot directly attributes RABBiT's gain to the SID readout pathway versus the backbone. A vertex-wise ridge readout on the headline $30$-ROI vertex set ($V_{\mathrm{tot}} \approx 41{,}394$) and a multi-tap FIR delay basis has $\sim$$222$M trainable parameters with the $768$-dim wav2vec backbone and $\sim$$297$M with the $1024$-dim WavLM backbone --- approximately three orders of magnitude more than the $\sim$$115{,}000$ parameters that RABBiT few-shot updates per participant.
\subsection{No-SID RABBiT (direct fine-tune) baseline}
\label{app:fs_direct}
The few-shot variant of the Per-Subject Full-Rank baseline (App.\ref{app:indp_baselines}) tests whether the gains in App.\ref{app:fs_math} come from the SID structure itself or merely from freezing most of the model and updating a small head. Architecture is identical to RABBiT except the SID readout is replaced by one per-ROI linear layer mapping $z_i$ directly to all $V_i$ vertices (no shared--deviation factorization). The full readout is $\sim$$10.6$M parameters per participant ($\sim$$92\times$ more than RABBiT few-shot's $\sim$$115{,}000$, but $\sim$$22\times$ fewer than brain-tuned ridge). At calibration time we adapt this per-ROI linear readout with the same correlation-plus-anchor objective and training schedule.
Because this baseline lacks RABBiT's strong shared-bases prior, few-shot adaptation initially \emph{loses} held-out signal at small calibrations before recovering. At $C = 200$ TRs ($\sim$$5$\,min), the No-SID readout falls to language-aggregate $r = 0.118$ versus its own zero-shot value of $r = 0.136$ --- a $\sim$$13\%$ drop. RABBiT few-shot at the same calibration size starts above its zero-shot ($r = 0.139$) and gains a further $\sim$$+0.01$ from adaptation, never going below zero-shot. The No-SID baseline only recovers parity by $C \approx 800$ TRs ($\sim$$20$\,min); this is the structural advantage SID's bounded subject-adaptation pathway provides at small calibrations.
\subsection{Additional few-shot results}
\label{app:fs_add_results}
Table\ref{tab:fewshot_stats} reports paired two-sided $t$-tests across the $19$-subject panel between consecutive sorted conditions of the saturation curve. RABBiT few-shot significantly outperforms the No-SID direct fine-tune at every calibration size from $5$ to $40$ minutes ($p < 0.001$ for $5$--$30$\,min, $p \approx 0.006$ at $40$\,min). The No-SID direct fine-tune does not significantly outperform the brain-tuned ridge baseline from $10$ minutes onward (numerical $\Delta r \approx +0.003$--$+0.005$, $p > 0.10$).
\begin{table}[h]
\centering
\small
\begin{tabular}{lrrr}
\toprule
Calibration & ridge $\to$ direct & direct $\to$ RABBiT few-shot & ridge $\to$ RABBiT few-shot \\
\midrule
5\,min ($C{=}200$) & $\Delta\,{+}0.009$, $p{=}0.029$\,*\, & $\Delta\,{+}0.030$, $p{<}0.001$\,***\, & *** \\
10\,min ($C{=}400$) & $\Delta\,{+}0.005$, $p{=}0.108$\,n.s.\, & $\Delta\,{+}0.021$, $p{<}0.001$\,***\, & *** \\
20\,min ($C{=}800$) & $\Delta\,{+}0.003$, $p{=}0.201$\,n.s.\, & $\Delta\,{+}0.014$, $p{<}0.001$\,***\, & *** \\
30\,min ($C{=}1200$) & $\Delta\,{+}0.004$, $p{=}0.112$\,n.s.\, & $\Delta\,{+}0.009$, $p{=}0.0001$\,***\, & *** \\
40\,min ($C{=}1600$) & $\Delta\,{+}0.003$, $p{=}0.172$\,n.s.\, & $\Delta\,{+}0.005$, $p{=}0.006$\,**\, & ** \\
\bottomrule
\end{tabular}
\caption{\textbf{Pairwise paired two-sided $t$-tests on the language-aggregate saturation curve.} $n{=}19$ participants on \textit{21styear}; $\Delta r$ between consecutive sorted conditions.}
\label{tab:fewshot_stats}
\end{table}
The headline per-ROI figure (main text Fig.\ref{fig:perroi-fewshot}) shows the per-ROI percent change at $C{=}400$. Fig.\ref{fig:perroi_21st_calibsizes} gives the per-ROI breakdown at the four other calibration sizes ($C \in \{200, 800, 1200, 1600\}$) on \textit{21styear}: the qualitative ROI ordering is preserved, the magnitude grows monotonically with calibration size, and the largest improvements remain concentrated in higher-order language and DMN ROIs (IFG, supramarginal, mPFC, angular gyrus, MFG/DLPFC, precuneus/PCC). Fig.\ref{fig:slumlord_saturation} and Fig.\ref{fig:perroi_slum_calibsizes} give the corresponding saturation curve and per-ROI breakdown on \textit{slumlordreach} ($18$ participants, calibration sizes $\{100, 200, 400, 600\}$ TRs); at $C{=}400$ on the wav2vec backbone, RABBiT few-shot reaches $r = 0.114$ on the LH-language aggregate vs $r = 0.098$ for No-SID direct and $r = 0.094$ for brain-tuned ridge. Two methodological notes: \textit{slumlordreach} has a different optimal residual hemodynamic shift than \textit{21styear} ($\Delta t \approx 2$ vs $3$ TRs), and we use per-story-calibrated comparisons throughout; the headline anchor $\lambda_{\rho} = 5{\times}10^{-5}$ used in all reported saturation curves is appropriate for $C \geq 200$ TRs (a stronger $\lambda_{\rho} = 10^{-3}$ is recommended for very-small-calibration regimes $C \leq 100$).
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figs/legend.pdf}
\begin{subfigure}[b]{0.8\textwidth}
\includegraphics[width=\linewidth]{figs/app/w2v_saturation_paper_slumlord_4bar_sorted_no_legend.pdf}
\end{subfigure}
\caption{\textbf{Few-shot saturation curve on \textit{slumlordreach}.} Per-subject mean correlation $r$ on the held-out test segment versus calibration duration on $18$ participants}
\label{fig:slumlord_saturation}
\end{figure}
\begin{figure}[h]
\centering
% \includegraphics[width=0.5\linewidth]{figs/legend.pdf}
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/fig2_app/brain_delta_21styear_C400_ridge_pre_focus_with_legend.pdf}
\caption{Pretrained ridge --- zero-shot for $C = 400$ TRs ($\sim$$10$\,min)}
\label{fig:brain_ridge_fs}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/fig2_app/brain_delta_21styear_C400_ridge_bt_focus_with_legend.png}
\caption{Brain-tuned ridge --- zero-shot for $C = 400$ TRs ($\sim$$10$\,min)}
\label{fig:brain_ridge_bt_fs}
\end{subfigure}
\vspace{4pt}
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/fig2_app/brain_delta_21styear_C800_fewshot_focus_with_legend.pdf}
\caption{Few-shot --- zero-shot for $C = 800$ TRs ($\sim$$20$\,min)}
\label{fig:brain_fs_800}
\end{subfigure}\hfill
\caption{\textbf{Additional brain plots for the delta in $r$ relative to zero-shot on \textit{21styear}.} }
\label{fig:brain_21st_calibsizes}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.5\linewidth]{figs/legend.pdf}
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/perroi-final/per_roi_top_C200_with_direct_no_legend.pdf}
\caption{$C = 200$ TRs ($\sim$$5$\,min)}
\label{fig:perroi_21st_C200}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/perroi-final/per_roi_top_C800_with_direct_no_legend.pdf}
\caption{$C = 800$ TRs ($\sim$$20$\,min)}
\label{fig:perroi_21st_C800}
\end{subfigure}
\vspace{4pt}
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/perroi-final/per_roi_top_C1200_with_direct_no_legend.pdf}
\caption{$C = 1200$ TRs ($\sim$$30$\,min)}
\label{fig:perroi_21st_C1200}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/perroi-final/per_roi_top_C1600_with_direct_no_legend.pdf}
\caption{$C = 1600$ TRs ($\sim$$40$\,min)}
\label{fig:perroi_21st_C1600}
\end{subfigure}
\caption{\textbf{Per-ROI percent change in $r$ relative to zero-shot on \textit{21styear}, across four additional calibration sizes.} Each panel reports per-ROI deltas for RABBiT few-shot, brain-tuned ridge, and pretrained-features ridge. $C{=}400$ panel is in the main text.}
\label{fig:perroi_21st_calibsizes}
\end{figure}
%NEWFIG
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth]{figs/legend.pdf}
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/fig2_app/per_roi_paper_slumlordv2/per_roi_top_C100_slumlord_with_direct_no_legend.pdf}
\caption{$C = 100$ TRs ($\sim$$2.5$\,min)}
\label{fig:perroi_slum_C100}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/fig2_app/per_roi_paper_slumlordv2/per_roi_top_C200_slumlord_with_direct_no_legend.pdf}
\caption{$C = 200$ TRs ($\sim$$5$\,min)}
\label{fig:perroi_slum_C200}
\end{subfigure}
\vspace{4pt}
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/fig2_app/per_roi_paper_slumlordv2/per_roi_top_C400_slumlord_with_direct_no_legend.pdf}
\caption{$C = 400$ TRs ($\sim$$10$\,min)}
\label{fig:perroi_slum_C400}
\end{subfigure}\hfill
\begin{subfigure}[t]{0.69\textwidth}
\includegraphics[width=\linewidth]{figs/fig2_app/per_roi_paper_slumlordv2/per_roi_top_C600_slumlord_with_direct_no_legend.pdf}
\caption{$C = 600$ TRs ($\sim$$15$\,min)}
\label{fig:perroi_slum_C600}
\end{subfigure}
\caption{\textbf{Per-ROI percent change in $r$ relative to zero-shot on \textit{slumlordreach}, across four calibration sizes.} Each panel includes the No-SID direct fine-tune as a comparator.}
\label{fig:perroi_slum_calibsizes}
\end{figure}
%NEWFIG
\begin{thebibliography}{10}
\bibitem{adeli2025transformer}
Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte.
\newblock Transformer brain encoders explain human high-level visual responses.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)}, 2025.
\newblock Spotlight.
\bibitem{antonello2024scaling}
Richard Antonello, Aditya Vaidya, and Alexander Huth.
\newblock Scaling laws for language encoding models in fmri.
\newblock {\em Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem{baevski2020wav2vec}
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.
\newblock wav2vec 2.0: A framework for self-supervised learning of speech representations.
\newblock {\em Advances in Neural Information Processing Systems}, 33:12449--12460, 2020.
\bibitem{beliy2024wisdom}
Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani.
\newblock The wisdom of a crowd of brains: A universal brain encoder.
\newblock {\em arXiv preprint arXiv:2406.12179}, 2024.
\bibitem{beliy2025brainlm}
Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani.
\newblock The wisdom of a crowd of brains: A universal brain encoder, 2025.
\bibitem{chen2022wavlm}
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu~Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et~al.
\newblock {WavLM}: Large-scale self-supervised pre-training for full stack speech processing.
\newblock {\em IEEE Journal of Selected Topics in Signal Processing}, 16(6):1505--1518, 2022.
\bibitem{dAscoli2026TribeV2}
St{\'e}phane d'Ascoli, J{\'e}r{\'e}my Rapin, Yohann Benchetrit, Teon Brookes, Katelyn Begany, Jos{\'e}phine Raugel, Hubert Banville, and Jean-R{\'e}mi King.
\newblock A foundation model of vision, audition, and language for in-silico neuroscience.
\newblock 2026.
\bibitem{davisjohnsrude2003}
Matthew~H Davis and Ingrid~S Johnsrude.
\newblock Hierarchical processing in spoken language comprehension.
\newblock {\em Journal of Neuroscience}, 23(8):3423--3431, 2003.
\bibitem{fedorenko2010new}
Evelina Fedorenko, Po-Jang Hsieh, Alfonso Nieto-Casta{\~n}{\'o}n, Susan Whitfield-Gabrieli, and Nancy Kanwisher.
\newblock New method for fmri investigations of language: defining rois functionally in individual subjects.
\newblock {\em Journal of Neurophysiology}, 104(2):1177--1194, 2010.
\bibitem{fischl2012freesurfer}
Bruce Fischl.
\newblock Freesurfer.
\newblock {\em NeuroImage}, 62(2):774--781, aug 2012.
\bibitem{fischl1999high}
Bruce Fischl, Martin~I. Sereno, Roger B.~H. Tootell, and Anders~M. Dale.
\newblock High-resolution intersubject averaging and a coordinate system for the cortical surface.
\newblock {\em Human Brain Mapping}, 8(4):272--284, 1999.
\bibitem{boyle2025friends}
Alessandro~T Gifford, Domenic Bersch, Marie St-Laurent, Basile Pinsard, Julie Boyle, Lune Bellec, Aude Oliva, Gemma Roig, and Radoslaw~M Cichy.
\newblock The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies.
\newblock {\em arXiv preprint arXiv:2501.00504}, 2024.
\bibitem{glasser2016multi}
Matthew~F Glasser, Timothy~S Coalson, Emma~C Robinson, Carl~D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian~F Beckmann, Mark Jenkinson, et~al.
\newblock A multi-modal parcellation of human cerebral cortex.
\newblock {\em Nature}, 536(7615):171--178, 2016.
\bibitem{hickok2007cortical}
Gregory Hickok and David Poeppel.
\newblock The cortical organization of speech processing.
\newblock {\em Nat. Rev. Neurosci.}, 8(5):393--402, May 2007.
\bibitem{hu2022lora}
Edward~J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu~Wang, and Weizhu Chen.
\newblock Lo{RA}: Low-rank adaptation of large language models.
\newblock In {\em International Conference on Learning Representations}, 2022.
\bibitem{huth2016natural}
Alexander~G Huth, Wendy~A de~Heer, Thomas~L Griffiths, Fr{\'e}d{\'e}ric~E Theunissen, and Jack~L Gallant.
\newblock Natural speech reveals the semantic maps that tile human cerebral cortex.
\newblock {\em Nature}, 532(7600):453--458, 2016.
\bibitem{jain2018incorporating}
Shailee Jain and Alexander Huth.
\newblock Incorporating context into language encoding models for fmri.
\newblock {\em Advances in Neural Information Processing Systems}, 31, 2018.
\bibitem{lebel2023natural}
Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander~G Huth.
\newblock A natural language fmri dataset for voxelwise encoding models.
\newblock {\em Scientific Data}, 10(1):555, 2023.
\bibitem{Li2022LePetitPrince}
Jixing Li, Shohini Bhattasali, Shulin Zhang, Berta Franzluebbers, Wen-Ming Luh, R.~Nathan Spreng, Jonathan~R. Brennan, Yiming Yang, Christophe Pallier, and John Hale.
\newblock Le petit prince multilingual naturalistic fmri corpus.
\newblock {\em Scientific Data}, 9(1):530, 2022.
\bibitem{millet2022toward}
Juliette Millet, Charlotte Caucheteux, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, et~al.
\newblock Toward a realistic model of speech processing in the brain with self-supervised learning.
\newblock {\em Advances in Neural Information Processing Systems}, 35:33428--33443, 2022.
\bibitem{moussa2025improving}
Omer Moussa, Dietrich Klakow, and Mariya Toneva.
\newblock Improving semantic understanding in speech language models via brain-tuning.
\newblock In {\em The Thirteenth International Conference on Learning Representations (ICLR)}, 2025.
\bibitem{moussa2025braininter}
Omer Moussa and Mariya Toneva.
\newblock Brain-tuned speech models better reflect speech processing stages in the brain.
\newblock {\em arXiv preprint arXiv:2506.03832}, 2025.
\bibitem{moussa2025multibrain}
Omer Moussa and Mariya Toneva.
\newblock Brain-tuning improves generalizability and efficiency of brain alignment in speech models.
\newblock In {\em Advances in Neural Information Processing Systems (NeurIPS)}, 2025.
\bibitem{nastase2021narratives}
Samuel~A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher~J Honey, Yaara Yeshurun, Mor Regev, et~al.
\newblock The ``narratives'' fmri dataset for evaluating models of naturalistic language comprehension.
\newblock {\em Scientific data}, 8(1):250, 2021.
\bibitem{oota2024speech}
Subba~Reddy Oota, Emin {\c{C}}elik, Fatma Deniz, and Mariya Toneva.
\newblock Speech language models lack important brain-relevant semantics.
\newblock {\em Annual Meeting of the Association for Computational Linguistics (ACL)}, 2024.
\bibitem{schrimpf2021neural}
Martin Schrimpf, Idan~Asher Blank, Greta Tuckute, Carina Kauf, Eghbal~A Hosseini, Nancy Kanwisher, Joshua~B Tenenbaum, and Evelina Fedorenko.
\newblock The neural architecture of language: Integrative modeling converges on predictive processing.
\newblock {\em Proceedings of the National Academy of Sciences}, 2021.
\bibitem{scott2000identification}
Sophie~K Scott, C~Catrin Blank, Stuart Rosen, and Richard~JS Wise.
\newblock Identification of a pathway for intelligible speech in the left temporal lobe.
\newblock {\em Brain}, 123(12):2400--2406, 2000.
\bibitem{st2026cneuromod}
Marie St-Laurent, Basile Pinsard, Oliver Contier, Elizabeth DuPre, Katja Seeliger, Valentina Borghesani, Julie~A Boyle, Lune Bellec, and Martin~N Hebart.
\newblock Cneuromod-things, a densely-sampled fmri dataset for visual neuroscience.
\newblock {\em Scientific Data}, 2026.
\bibitem{toneva2019interpreting}
Mariya Toneva and Leila Wehbe.
\newblock Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).
\newblock {\em Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem{vaidya2022self}
Aditya~R Vaidya, Shailee Jain, and Alexander Huth.
\newblock Self-supervised models of audio effectively explain human cortical responses to speech.
\newblock In {\em International Conference on Machine Learning}, pages 21927--21944, 2022.
\bibitem{vattikonda2025brainwavlm}
Nishitha Vattikonda, Aditya~R. Vaidya, Richard~J. Antonello, and Alexander~G. Huth.
\newblock {BrainWavLM}: Fine-tuning speech representations with brain responses to language, 2025.
\bibitem{wehbe2014simultaneously}
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell.
\newblock Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses.
\newblock {\em PloS One}, (11), 2014.
\end{thebibliography}