Audio LLMs Know When They Can’t Hear You

Training the reliability predictor requires labels that reflect the transcription behavior of the target Audio LLM. We therefore construct a transcription reliability dataset through model-grounded reliability labeling, in which reliability boundaries are estimated directly from the model’s transcription performance under controlled acoustic corruption.

For a speech utterance $x$, reference transcript $y$, and target Audio LLM $f(\cdot)$, we measure the word error rate (WER) as $E(f(x),y)$, where $E(\cdot,\cdot)$ denotes the transcription error. Given an additive interference signal $n$, the signal-to-noise ratio (SNR) is defined as

where $\lVert\cdot\rVert_{2}$ denotes the $\ell_{2}$ norm. For a desired SNR $s$, we scale the interference signal and define the resulting corrupted waveform as

Since transcription error generally increases as SNR decreases, we define the critical SNR for model $f$ and WER threshold $\tau$ as

Thus, $s_{\tau,f}(x,n)$ is the lowest SNR at which model $f$ achieves a WER no greater than $\tau$ for the speech–corruption pair $(x,n)$.

Figure 2 illustrates the distribution of critical SNR corresponding to a WER of $10\%$ across numerous speech–corruption pairs evaluated using Qwen2-Audio-7B-Instruct. Although the average critical SNR differs across corruption types, each category spans a wide range. Thus, even within the same corruption category, a single SNR threshold cannot reliably determine the transcription reliability of an input. This motivates estimating critical SNRs separately for each speech–corruption pair.

In our experiments, we define four reliability classes: (i) reliable (WER $=0$), (ii) minor degradation ($0<\mathrm{WER}\leq 10\%$), (iii) moderate degradation ($10\%<\mathrm{WER}\leq 30\%$), and (iv) severe degradation ($\mathrm{WER}>30\%$). These classes are defined by the WER thresholds $\tau\in\{0.0,0.1,0.3\}$. For each speech–corruption pair, we estimate the corresponding critical SNRs and store them in a reliability label bank, referred to hereafter as the label bank.

Figure 3 summarizes the label-bank construction procedure. For each speech–corruption pair, we keep the speech utterance and corruption source fixed and vary only the SNR. Each resulting waveform is transcribed by the target Audio LLM and compared with the reference transcript to compute WER. Assuming an approximately monotonic relationship between SNR and transcription error, we use binary search to estimate the critical SNR associated with each WER threshold. Because these boundaries are estimated independently for every speech–corruption pair, the resulting label bank captures the transcription behavior of the target Audio LLM rather than relying on nominal corruption strength alone.

We consider both additive interference, including environmental noise and music, and reverberant conditions in which room reverberation is applied before additive interference. For each speech–corruption pair, the underlying speech, corruption source, and room impulse response, when applicable, are kept fixed while only the SNR is varied. To ensure high-quality reliability labels, we apply quality-control filtering before admitting a speech–corruption pair to the label bank. In particular, we exclude pairs for which the clean utterance is already transcribed poorly, reverberation alone causes substantial transcription error, the estimated critical SNRs are insufficiently separated or violate their expected ordering, or valid SNR intervals for the reliability classes cannot be identified.

After filtering, training examples are generated by sampling SNRs within the retained class-specific SNR intervals. This allows the acoustic conditions to vary across training while preserving the reliability class assigned by the target Audio LLM. Validation and test examples are fixed so that all methods are evaluated on identical waveforms. Additional details on preprocessing, partitioning, critical-SNR estimation, quality-control criteria, and sample generation are provided in Appendix C.

Given an input audio clip and a target Audio LLM, our goal is to predict the reliability of the model’s transcription without access to the reference transcript. Figure 4 illustrates the proposed architecture. We use the pretrained audio encoder of the target Audio LLM as the feature extractor and keep all of its parameters frozen. For an input waveform $x$, the encoder produces a sequence of frame-level representations, $\{h_{1},\ldots,h_{T}\}$. Freezing the encoder preserves the representations learned during large-scale pretraining and ensures that training the reliability predictor does not modify the underlying Audio LLM.

The frame-level representations produced by the audio encoder are not necessarily equally informative for transcription reliability. We therefore use a lightweight learned temporal pooling module that assigns a scalar weight $\beta_{t}$ to each frame representation $h_{t}\in\mathbb{R}^{d}$. For a sequence of $T$ vectors, $\beta_{t}$ is computed using a learned linear projection followed by a Softmax operation:

where $w\in\mathbb{R}^{d}$ and $b\in\mathbb{R}$ are learned parameters. The aggregated seqeunce is summarized as

This produces a single utterance embedding $h_{\mathrm{utt}}$ while allowing the predictor to place different emphasis on different portions of the audio. The resulting embedding is then passed to a lightweight two-layer MLP that outputs logits for the four reliability classes.

The proposed predictor introduces negligible overhead compared with the standard Audio LLM inference pipeline. Since the audio encoder is already executed to process the input audio, the additional cost is limited to the lightweight pooling module and classification head. For example, in Qwen2-Audio-7B-Instruct, the predictor contains only 0.33M trainable parameters compared with the 636.97M frozen audio-encoder parameters (0.052%), and adds approximately 2.12M FLOPs compared with the 1.89T FLOPs required by the audio encoder (0.00011%). Detailed overhead comparisons across all evaluated Audio LLMs are provided in Appendix F.

Although our exact problem definition is not formulated in contemporary literature, we perform comparison with most related baselines as follows:

Nominal SNR: we treat the nominal mixing SNR as an oracle corruption-strength baseline; its exact value is available in our synthetically generated mixtures, but would generally not be accessible for real-world recordings where the clean speech and interference signals are not separately observed.

No-reference speech quality and intelligibility predictors: these methods assess audio quality by assigning a numeric value to a recording. We study Audiobox-Aesthetics (Tjandra et al., 2025), DNSMOS (Reddy et al., 2021), NISQA (Mittag et al., 2021), and TorchAudio-SQUIM (Kumar et al., 2023). For these methods, we use Production Quality (PQ), overall quality (OVRL), mean opinion score (MOS), and short-time objective intelligibility (STOI), respectively.

Audio LLM Generation Uncertainty: we additionally evaluate two Audio LLM generation uncertainty measures from the target Audio LLM: mean token probability, obtained by averaging the probabilities assigned to the generated tokens, and mean token entropy, obtained by averaging predictive entropy across generation steps.

For the three scalar baseline families described above, we fit three ordered decision thresholds using only the in-domain training data to maximize four-class classification accuracy. These thresholds are then fixed for all evaluation sets.

Transcript-conditioned WER estimation: We study the method proposed in Fe-WER (Park et al., 2025), which estimates WER after decoding using an auxiliary model. The model is trained on our in-domain training data using the original architecture, with HuBERT-large (Hsu et al., 2021) for acoustic representations and XLM-R-large (Conneau et al., 2020) for representations of the generated transcript. At evaluation time, the predicted WER is mapped directly to the same WER intervals used to define our four reliability classes. For the reliable class, which corresponds to zero WER, requiring a continuous regressor to predict exactly zero is overly restrictive. Since an utterance with $L$ reference words has a minimum nonzero WER of $\frac{1}{L}$, we assign predictions with $\widehat{\mathrm{WER}}<\frac{1}{L}$ to the reliable class and use the remaining fixed WER thresholds for the other classes. For trainable models, we use the in-domain validation set to select the best-performing checkpoint.

Table 1 reports the main results using Qwen2-Audio-7B-Instruct (Chu et al., 2024) as the target Audio LLM. Our reliability predictor achieves the strongest overall performance in both evaluation settings. On the in-domain test set, it achieves $82.60\%$ accuracy, $81.10\%$ macro-F1, and $0.18$ MAE. The strongest competing method, Fe-WER, reaches $75.05\%$ accuracy, $70.77\%$ macro-F1, and $0.27$ MAE. Our method therefore improves accuracy by $7.55$ percentage points and macro-F1 by $10.33$ points while reducing MAE by $0.09$.

Under cross-domain evaluation, our predictor retains $79.55\%$ accuracy and $78.09\%$ macro-F1 with an MAE of $0.22$. Among the baselines, Qwen mean token probability achieves the highest macro-F1 at $66.16\%$, while Fe-WER achieves the highest accuracy at $69.69\%$ and the lowest MAE at $0.33$. Relative to the strongest baseline for each metric, our method improves accuracy by $9.86$ points and macro-F1 by $11.93$ points while reducing MAE by $0.11$.

Importantly, the proposed predictor also exhibits a relatively small generalization gap: moving from the in-domain to the cross-domain test set reduces accuracy by only $3.05$ points and macro-F1 by $3.01$ points. In contrast, Fe-WER drops by $5.36$ accuracy points and $6.10$ macro-F1 points. This suggests that reliability information captured by the frozen Audio LLM encoder transfers more effectively across unseen speech, environmental interference, and room acoustics than transcript-conditioned WER estimation trained on the same in-domain data.

The acoustic corruption proxy and no-reference speech quality and intelligibility predictors are substantially weaker overall. Even nominal SNR, despite oracle access to the mixing SNR, reaches only $65.14\%$ in-domain accuracy and $60.85\%$ cross-domain accuracy. This supports the motivation in Section 3.1: SNR alone does not fully determine the transcription behavior of an Audio LLM.

Finally, Audio LLM generation uncertainty provides a considerably stronger signal than the no-reference speech quality and intelligibility predictors, but remains substantially below our reliability predictor. This indicates that uncertainty in the generated token distribution is related to transcription failure but does not fully capture the reliability information already present in the model’s audio-encoder representations. Our predictor also avoids language-model decoding: for Qwen2-Audio-7B-Instruct, the prediction head requires only $2.12$M FLOPs compared with $4.21$T FLOPs for the decoder, making the decoder roughly $2.0$ million times more computationally expensive than the added prediction head. Detailed comparisons are provided in Appendix F.

Additional studies: We study two more Audio LLMs, namely Phi-4-Multimodal-Instruct (Abouelenin et al., 2025) and MOSS-Audio-8B (Yang et al., 2026), in Appendix D, where we follow the same baseline families and evaluation protocol. Appendix G.1 analyzes the temporal aggregation strategy used to summarize frame-level audio-encoder representations, while Appendix G.2 studies the encoder depth at which those representations are probed. We also provide confusion-matrix analyses in Appendix E to examine error patterns across reliability classes.

Constructing a reliability label bank requires repeatedly evaluating the target Audio LLM to estimate critical SNRs across speech–corruption pairs and WER thresholds. We therefore investigate whether a label bank constructed for one Audio LLM can be reused to train a reliability predictor for another, thereby reducing the cost of constructing a complete model-specific label bank. We study this effect among three Audio LLMs: Qwen2-Audio-7B-Instruct (Qwen2), Phi-4-Multimodal-Instruct (Phi-4), and MOSS-Audio-8B (MOSS).

The left panel of Figure 5 shows the performance of the reliability classifier for the Audio LLMs under study. The diagonal entries show the performance when the target model itself is used to construct the training label bank. We refer to the off-diagonal entries as cross-model transfer accuracy, as they show the performance when the label bank constructed using one model is used to train a reliability predictor for another.

We make two observations. First, transfer between Qwen2 and Phi-4 is effective, largely preserving accuracy. In contrast, transfer between MOSS and either of the other two models results in a substantially larger degradation in accuracy. This raises the question: why does transfer work better between Qwen2 and Phi-4 than between MOSS and the other models? We hypothesize that this variation in transfer performance is related to the mismatch between the critical-SNR boundaries of the source and target models. Using the critical-SNR definition in Eq. 3, we define the critical-SNR mismatch between a source model $f_{s}$ and a target model $f_{t}$ as

where $\mathcal{P}$ denotes the shared speech–corruption pairs and $\mathcal{T}$ denotes the WER thresholds defining the reliability classes. The middle panel of Figure 5 compares the critical-SNR mismatch with the corresponding accuracy drop under cross-model transfer. Each blue circle represents a transfer from source model $f_{s}$ to target model $f_{t}$, denoted by $f_{t}\leftarrow f_{s}$. We observe a positive correlation ($r=0.78$), indicating that larger differences between the models’ critical-SNR boundaries are associated with greater transfer degradation.

The right panel of Figure 5 examines whether the critical-SNR mismatch can be estimated without constructing the complete target label bank. For each fraction of shared speech–corruption pairs, we compute the mismatch using the corresponding critical SNRs from the source and target models. We repeat this random subsampling procedure 20 times and report the mean and standard deviation. Even with a small fraction of the shared pairs, the estimated mismatch is close to the value from the full label bank, while its variance decreases as more pairs are used. This suggests that the critical-SNR mismatch can be estimated from a relatively small number of measurements.

Motivated by this observation, we estimate the shift between the source and target critical-SNR boundaries using only a sampled fraction of the shared speech–corruption pairs. Let $\mathcal{P}_{\rho}\subset\mathcal{P}$ denote a subset containing a fraction $\rho$ of the shared pairs. For transfer from $f_{s}$ to $f_{t}$, we estimate a separate shift for each WER threshold $\tau\in\mathcal{T}$ as

Here, $\delta_{s\rightarrow t,\tau}^{(\rho)}$ estimates the systematic shift between the source and target models for reliability boundary $\tau$. We use these estimated shifts to adjust the critical-SNR values of the source label bank to approximate those of the target model. Specifically, we align each source critical-SNR as

Using these shifted critical SNRs improves transfer accuracy. We show two examples as the yellow triangles in the middle panel of Figure 5, where the Phi-4 and Qwen2 label banks are adjusted for transfer to MOSS. The alignment uses only $5\%$ of the shared speech–corruption pairs to estimate the boundary-specific shifts between the source and target critical-SNR boundaries. For MOSS$\leftarrow$Phi-4, the accuracy drop decreases from $12.9$ to $2.1$ points, while for MOSS$\leftarrow$Qwen2, it decreases from $9.7$ to $1.7$ points. By using only $5\%$ of the shared pairs, we can estimate the boundary-specific shifts that largely close the performance gap introduced by direct cross-model transfer, substantially reducing the need for target-specific label construction.

Additional study: We provide an ablation of the critical-SNR alignment strategy in Appendix G.3, comparing global and per-boundary shift estimation using mean and median aggregation.