ALAS: An Automatic Latent Alignment Score for Audio Language Models
Extracted source text · arXiv 2505.19937
\documentclass{Interspeech}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{xcolor}
\usepackage{tcolorbox}
\usepackage{float}
\definecolor{usercolor}{RGB}{0, 102, 204} % Blue
\definecolor{speechcolor}{RGB}{204, 0, 0} % Red
\definecolor{transcolor}{RGB}{0, 153, 51} % Green
\definecolor{taskcolor}{RGB}{255, 140, 0} % Orange
\definecolor{responsecolor}{RGB}{128, 0, 128} % Purple
% 2023-10-21 modified by Simon King (Simon.King@ed.ac.uk)
% 2024-01 modified by TPC Chairs of Interspeech 2024
% 2024-10 modified by Antoine Serrurier for Interspeech 2025
% 2024-12 modified by TPC Chairs of Interspeech 2025
% **************************************
% * DOUBLE-BLIND REVIEW SETTINGS *
% **************************************
% Comment out \interspeechcameraready when submitting the
% paper for review.
% If your paper is accepted, uncomment this to produce the
% 'camera ready' version to submit for publication.
\interspeechcameraready
% **************************************
% * *
% * STOP ! DO NOT DELETE ! *
% * READ THIS FIRST *
% * *
% * This template also includes *
% * important INSTRUCTIONS that you *
% * must follow when preparing your *
% * paper. Read it BEFORE replacing *
% * the content with your own work. *
% **************************************
% title here must exactly match the title entered into the paper submission system
\title {ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs
}
% the order of authors here must exactly match the order entered into the paper submission system
% note that the COMPLETE list of authors MUST be entered into the paper submission system at the outset, including when submitting your manuscript for double-blind review
\author[affiliation={1,2}]{Pooneh}{Mousavi}
\author[affiliation={3}]{Yingzhi}{Wang}
\author[affiliation={1,2}]{Mirco}{Ravanelli}
\author[affiliation={4,2}]{Cem}{Subakan}
%The maximum number of authors in the author list is 20. If the number of contributing authors is more than this, they should be listed in a footnote or the acknowledgement section.
% if you have too many addresses to fit within the available space, try removing the "\\" newlines
\affiliation{Concordia University, }{}{}
\affiliation{Mila-Quebec AI Institute, }{}{}
\affiliation{Centrale Supélec, }{}{}
\affiliation{Laval University}{}{}
\email{}
\keywords{Large Language Models, Multimodal LLMs, Cross-modal alignment}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\newcommand{\cem}[1]{\textcolor{red}{cem: #1}}
\usepackage{comment}
\begin{document}
\maketitle
% the abstract here must exactly match the abstract entered into the paper submission system
\begin{abstract}
% 1000 characters. ASCII characters only. No citations.
%Large Language Models (LLMs) are widely used in Spoken Language Understanding (SLU), traditionally relying on Automatic Speech Recognition (ASR) to convert speech into text {for} processing {and prediction}. However, recent SLU models process audio directly by adapting speech input into LLMs for better multimodal learning.
%A key component of these models is the aligner, which fuses speech and text representations. While various methods exist for fusing these modalities, there is no standard metric to evaluate alignment quality.
%cems version
%A key consideration for these models is the cross-modal alignment between text and audio modalities, which is a telltale sign as to whether or not LLM is able to associate semantic meaning to audio segments. While various methods exist for fusing these modalities, there is no standard metric to evaluate alignment quality in LLMs.
%In this work, we propose a new metric, \textbf{ALAS} (\textbf{A}utomatic \textbf{L}atent \textbf{A}lignment \textbf{S}core), to measure how well a multi-modal LLM for SLU aligns speech and text. We also analyze alignment patterns across different LLM layers to better understand where alignment begins and how it evolves. Our study examines the correlation between audio and text representations across transformer layers, for two different tasks (Spoken Question Answering and Emotion Recognition). We showcase that our metric behaves as expected across different layers and different tasks.
%providing insights into cross-modal learning and the interpretability of Speech LLMs.
Large Language Models (LLMs) are widely used in Spoken Language Understanding (SLU). Recent SLU models process audio directly by adapting speech input into LLMs for better multimodal learning.
A key consideration for these models is the cross-modal alignment between text and audio modalities, which is a telltale sign as to whether or not LLM is able to associate semantic meaning to audio segments. While various methods exist for fusing these modalities, there is no standard metric to evaluate alignment quality in LLMs.
In this work, we propose a new metric, {ALAS} ({A}utomatic {L}atent {A}lignment {S}core). Our study examines the correlation between audio and text representations across transformer layers, for two different tasks (Spoken Question Answering and Emotion Recognition). We showcase that our metric behaves as expected across different layers and different tasks.
\end{abstract}
\section{Introduction}
% Large Language Models (LLMs) \cite{touvron2023llama, chowdhery2023palm, brown2020language, du2021glm, achiam2023gpt} have achieved strong performance in text-based tasks, often achieving human capabilities in natural language processing (NLP). A key factor behind this success is instruction tuning \cite{peng2023instruction,zhang2023instruction, taori2023alpaca}, where models are trained on instruction-response pairs to improve their ability to follow open-ended prompts.
% Expanding LLMs beyond text has become an important research direction, with increasing interest in multimodal learning \cite{tang2023salmonn,chen2023x, chu2024qwen2, wu2023decoder}. Recent studies explore ways to connect LLMs with encoders for other data types, such as images, speech, audio, and music. These models typically use a connection module to align different input modalities, trained through cross-modal pretraining and instruction tuning.
% The performance of these models heavily depends on how well the alignment module integrates different modalities. Various methods have been proposed to improve alignment, including multi-stage training, pretraining, and instruction tuning, which help bridge the gap between pretrained components (LLMs and encoders) and newly initialized modules (connection layers and adaptors).
% Despite these efforts, there is no standard metric to evaluate how well speech and text are aligned in SLU models. \cem{Measuring the quality of the cross-modal alignment is an important problem since it gives an indication as to whether or not LMM is able to learn semantic information from the audio modality. -- we can expand more on this maybe-} In this work, we introduce a new metric, \textbf{ALAS} (\textbf{A}utomatic \textbf{L}atent \textbf{A}lignment \textbf{S}core), to measure alignment quality. We also analyze how alignment patterns evolve across different layers of an LLM, identifying where alignment begins and how it develops over the network.
Large Language Models (LLMs) \cite{touvron2023llama, chowdhery2023palm, brown2020language, du2021glm, achiam2023gpt} have recently demonstrated very strong performance in text-based tasks, often reaching human-level capabilities in natural language processing (NLP). A key factor contributing to this success is instruction tuning \cite{peng2023instruction,zhang2023instruction, taori2023alpaca,2023arXiv230605425L, zhang-etal-2023-video}, where models are trained on instruction-response pairs to improve their ability to follow open-ended prompts.
There has been increasing interest in multimodal learning, where LLMs are extended to process diverse inputs such as images, speech, and audio \cite{tang2023salmonn,chen2023x, chu2024qwen2, wu2023decoder, li2023blip,dai2023instructblip, rubenstein2023audiopalm,zhan2024anygpt,xie2024mini}. These models typically integrate a pretrained modality-specific encoder with a connection (adapter) module that fuses different input representations. The alignment process is crucial because it ensures that different modalities are mapped into a shared representation space, allowing the model to extract meaningful cross-modal relationships. Training such models often involves cross-modal pretraining and instruction tuning to improve their ability to process multimodal data effectively.
Alignment plays a fundamental role in the success of these multimodal models, particularly in speech-language understanding (SLU). Tan et al. \cite{tan2024ssr} explored various alignment strategies for optimizing the connection module and showed that stronger alignment leads to improved model performance. Effective alignment ensures that spoken and textual representations convey the same semantic information, reducing ambiguity and improving model performance. Several approaches have been developed to address the speech-text alignment problem. The Montreal Forced Aligner (MFA) \cite{mcauliffe2017montreal}, built on the Kaldi framework \cite{Povey_ASRU2011_2011}, provides phoneme-level alignment using forced alignment techniques. Connectionist Temporal Classification (CTC) \cite{graves2006connectionist} is widely used for speech-text alignment, as it enables alignment without explicit labels \cite{sainath2020emitting,huang2024less}. More recently, models such as UnitY2 \cite{barrault2023seamless} and ZMM-TTS \cite{gong2024zmm} have introduced unit-based alignment techniques using pretrained encoders, demonstrating improved performance across multiple languages.
Another important strategy for improving alignment in multimodal LLMs involves training with large-scale aligned speech-to-text datasets. Many models adopt a multi-stage training approach \cite{chu2024qwen2,tang2023salmonn,hu2024wavllm,gong2023listen}, where the first stage is dedicated to learning high-quality alignment between auditory and textual representations, mostly using large-scale automatic speech recognition or audio captioning datasets. This is followed by fine-tuning for specific downstream tasks.
However, there is no standardized metric to evaluate how well an SLU model learns speech-text alignment. Measuring alignment quality is critical for understanding whether an LLM effectively captures semantic relationships between speech and text. A well-defined alignment metric would provide deeper insights into cross-modal learning and help guide improvements in model design.
In this work, our contributions are as follows,
\begin{itemize}
\item We propose ALAS (Automatic Latent Alignment Score), a new metric for assessing alignment quality in speech-text multimodal LLMs.
\item We qualitatively and quantitatively analyze how alignment patterns evolve across different layers of an LLM, identifying at which stage alignment begins and how it develops throughout the network.
\item We apply ALAS on two different tasks (Spoken Question Answering and Emotion Recognition) and analyze the level of alignment between the text and audio modalities.
\end{itemize}
\section{Methodology}
The ALAS pipeline, illustrated in Figure \ref{fig:alas_pipeline}, consists of four steps:
\begin{enumerate*}[label={\arabic*)}]\item Speech Input Processing,
\item Text Input Processing,
\item Obtaining Reference Alignment, and
\item Measuring the Alignment Score.
\end{enumerate*}
Our pipeline is fully automatic and does not require any training or manual alignment. It extracts a reference alignment and compares the Speech LLM's alignment against it. Additionally, it is easily extendable to any Speech LLM model.
\begin{figure*}[t]
\centering
\scalebox{0.9}{ % Adjust the scale factor as needed
\includegraphics[width=\textwidth]{figures/alas-pipeline.pdf}
}
\caption{The Automatic Latent Alignment Score (ALAS) pipeline: \textbf{(Step 1)} Obtaining latent representations of the audio input,\textbf{(Step 2)} Obtaining latent representations of the transcription input, \textbf{(Step 3)} Extracting reference alignment from Whisper, and \textbf{(Step 4)} Calculating the ALAS score using cross-modal similarities and the alignment path between the two modalities.
}
\label{fig:alas_pipeline}
\end{figure*}
\subsection{Automatic Pipeline for Latent Alignment Score}
In order to measure the alignment score, we need an aligned test dataset, meaning that for each audio sample, we must have the corresponding transcription. We begin by selecting an appropriate dataset and defining the assigned task. We use speech question answering (LibriSQA dataset \cite{zhao2023librisqa}), and emotion recognition (IEMOCAP dataset \cite{busso2008iemocap}) in this paper. \\
\noindent \textbf{Step 1: Speech Input Processing.}
The speech input is passed through the Speech LLM along with an instruction, such as asking the model to answer a question based on the given audio. We extract layer-wise representations for the input sequence. Since alignment focuses on modality-specific input representations, we trim the input sequence to exclude any appended instructions at the beginning or end. This results in a tensor of shape \textit{(num\_layers, audio\_seq\_len, latent\_dimension)}. \\
\noindent \textbf{Step 2: Text Input Processing.}
The corresponding transcription is passed through the same model to obtain layer-wise representations for the text input. In this step, we do not input any audio to the model. Instead, we instruct the model to perform the task only based on the provided transcription. Since alignment focuses on modality-specific input, we apply the same trimming process to remove any appended instructions at the beginning or end of the transcription. To ensure that both audio and text inputs contain similar task-relevant information, we filter out input pairs where the similarity between their generated responses is below a predefined threshold. \\
\noindent \textbf{Step 3: Obtaining Reference Alignment.}
After completing the first two steps, which entails obtaining the audio latent representations, text latent representations, we now obtain reference alignment from Whisper. We use Whisper for getting a reference alignment since Whisper-v3 is able to give time stamps for each word given an audio file that contains speech.
%These outputs are then passed to our scoring module, which computes the final alignment score (step 4).
To be able to compare the text-audio alignment of the LLM with the Whisper's reference time stamps, we convert the tokens into word-level tokens if needed.
%To ensure compatibility, the tokenizer used by the Speech LLM must align with Whisper’s word-based alignment.
Note that if the LLM's tokenizer is already word-based (e.g. this is the case for QWEN), no additional processing is needed. However, if the tokenizer operates at the BPE (Byte-Pair Encoding) or character level, we must align and reconstruct the word boundaries to match the reference alignment. This ensures that both the text and audio sequences have the same length for accurate alignment.
Note that we map the time stamps (in seconds) that are given by Whisper to the token index to compare the Whisper alignments with the alignments obtained through the ALAS pipeline. \\
%\cem{we need to talk about how we convert the time stamps into token index.} \\
\noindent \textbf{Step 4: Measuring the Alignment Score.}
Given three inputs—text latent representations, audio latent representations, and the reference alignment from Whisper—we compute the final alignment score using our scoring module. Our scoring module consists of two sub-modules. As we discuss next in more detail, we first measure the cross-modal cosine similarities, and then apply monotonic-alignment-search algorithm \cite{kim2020glowttsgenerativeflowtexttospeech} on the cross-modal cosine similarity matrices.
\begin{figure*}[ht]
\includegraphics[width=0.99\textwidth, trim=0.4cm 0cm 0.4cm 0cm, clip]{qwen_heatmaps.pdf}
\caption{
%Qualitative comparison of the alignments obtained for QWEN (\textbf{left}) and SALMONN (\textbf{right}). The x-axis are the time steps for the audio representations, and the y-axis are the time steps for the text representations. The red trace demonstrates the alignment found by ALAS, and the white trace shows the Whisper alignment.
Qualitative comparison of the alignments obtained for Qwen2-Audio-Instruct-7B (left) and Salmonn-7B (right). In each sub-figure, the horizontal axis represents the audio time series, with each frame corresponding to 20 ms for Qwen2-Audio-Instruct-7B and 330 ms for Salmonn-7B. The vertical axis represents the sequence of text tokens, where Qwen2-Audio-Instruct-7B utilizes word-level tokens while Salmonn-7B employs BPE tokens. In every sub-figure, the white trajectory denotes the reference alignment derived from Whisper, whereas the red trajectory indicates the current alignment generated by applying the MAS algorithm on top of the similarity pattern shown with the heatmaps.
}
\label{fig:heatmaps}
\end{figure*}
\subsection{Measuring the Cross-Modal Similarities}
To measure the cross-modal-alignment between the text and audio, we calculate the cosine similarities between the latent representations of text and audio. For the $l$'th layer of the LLM, this similarity is calculated as follows,
\begin{align}
S^l_{i,j} = \frac{\langle a_i^l , t_j^l \rangle }{ ||a_i^l||\cdot ||t_j^l|| },
\end{align}
where $S^l_{i,j} \in [-1, 1]$ denotes the cosine similarity between the audio representation at time-step $i$, and text representation at time-step $j$, for layer $l$. Note that $a_i^l, t_j^l$ are $K=4096$ dimensional vectors that are respectively the latent states of the transformer at layer $l$.
\subsection{Automatic Latent Alignment Score (ALAS)}
After obtaining the similarity score matrix, $S^l \in [0,1]^{T \times A}$, we obtain the alignment path between the two modalities using the monotonic-alignment-search algorithm \cite{kim2020glowttsgenerativeflowtexttospeech}, where $T$ and $A$ respectively denote the total number of time steps respectively in the text and audio modalities.
In order then to assess the quality of the alignments that we estimate through similarities and monotonic-alignment-search, we obtain word-level alignments from Whisper\footnote{\url{https://huggingface.co/openai/whisper-large-v3}} \cite{radford2022robustspeechrecognitionlargescale}. The final score is defined such that,
\begin{align}
\text{ALAS} := \frac{1}{A}\sum_{i=1}^A | \widetilde S_i - G_i |,
\end{align}
where $\widetilde S_i $ denotes the $i$'th step of the path such that $\widetilde S=\{ \widetilde S_1, \widetilde S_2, \dots, \widetilde S_A\}$, and $G_i$ denotes the $i$'th step of the ground truth path such that $G = \{ G_1, G_2, \dots, G_A\}$.
\section{Experiments}
We evaluate ALAS with two popular multimodal LLMs: SALMONN-7B\cite{tang2023salmonn}\footnote{\url{https://huggingface.co/tsinghua-ee/SALMONN-7B/tree/main}} and Qwen2-Audio-Instruct-7B\cite{chu2024qwen2}\footnote{\url{https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct}} (We refer to these models as SALMONN and QWEN for brevity). These models have achieved state-of-the-art (SOTA) results on various audio question-answering tasks. Both models leverage multitask learning and multistage training to improve alignment between modalities.
\begin{figure*}[h!]
\centering
\includegraphics[width=\textwidth]{quantitative_figure.pdf}
\caption{Quantitative comparison of the alignments through \emph{Automatic Latent Alignment Score (ALAS)} for QWENN and SALMONN for LibriSQA and IEMOCAP datasets. The horizontal axes represent the layer-id, and the vertical axes represent the corresponding ALAS score. }
\label{fig:quantitative}
\end{figure*}
%For aligned datasets,
We experiment with two speech datasets that contain paired audio and text. The first dataset is LibriSQA\cite{zhao2023librisqa}, which is designed for free-form and open-ended Speech Question Answering (SQA). Given an audio clip or its corresponding transcription, we prompt the model with a question in text format and evaluate its response. The second dataset is IEMOCAP\cite{busso2008iemocap}, which is used for emotion recognition. The task involves classifying the input into one of four emotion categories: happy, sad, angry, and neutral. The choice of these datasets allows us to analyze the alignment patterns of multimodal speech LLMs in two different types of tasks. LibriSQA requires the model to extract semantic meaning from speech, making it a useful benchmark for evaluating how well the model aligns spoken and textual representations in tasks that rely on linguistic understanding. In contrast, emotion recognition focuses more on acoustic features rather than linguistic content, providing insight into how these models process non-semantic auditory cues. By evaluating the alignment score on both datasets, we can examine how well the alignment mechanism generalizes across tasks that pertain to different aspects of speech.
In Figure \ref{fig:prompt} in the Appendix, we illustrate the prompts used for both QWEN and SALMONN models.
For the Spoken Question-Answering Task, to measure response similarity, we use Sentence-BERT\cite{reimers-2019-sentence-bert}\footnote{\url{https://huggingface.co/sentence-transformers}}, where we pass each generated response through the sentence transformer to compute a similarity score. We discard the pairs with a similarity score lower than 0.7 to ensure that both text and audio contain comparable task-relevant information. We observe that QWEN and SALMONN respectively obtains scores of 0.71 and 0.76 respectively on the SQA task. For the Emotion Recognition task, the models respectively obtain scores of 21.56\%, and 49.30\% \cite{wang2024audiobenchuniversalbenchmarkaudio} on the IEMOCAP dataset.
% \begin{itemize}
% % SALMONN SQA
% \item \textbf{[SALMONN\textbar SQA \textbar Speech]} {\color{usercolor} USER:} {\color{speechcolor} \verb|<Speech><SpeechHere></Speech>|} {\color{taskcolor} \verb|<Question>? Answer in 10-15 words.|}
% \item \textbf{[SALMONN \textbar SQA \textbar Transcription]} {\color{usercolor} USER:} {\color{transcolor} \verb|<Text><Transcription></Text>|} {\color{taskcolor} \verb|<Question>? Answer in 10-15 words.|}
% % SALMONN ER
% \item \textbf{[SALMONN \textbar ER \textbar Speech]} {\color{usercolor} USER:} {\color{speechcolor} \verb|<Speech><SpeechHere></Speech>|} {\color{taskcolor} Describe the emotion in one word (anger, happiness, sadness, neutrality).}
% \item \textbf{[SALMONN \textbar ER \textbar Transcription]} {\color{usercolor} USER:} {\color{transcolor} \verb|<Text><Transcription></Text>|} {\color{taskcolor} Describe the emotion in one word (anger, happiness, sadness, neutrality).}
% % QWEN SQA
% \item \textbf{[QWEN \textbar SQA \textbar Speech]} {\color{usercolor} SYSTEM:} You are a helpful assistant.{\color{usercolor} USER:} Audio: {\color{speechcolor} \verb|<|audio\_bos\verb|>| \verb|<|SpeechHere\verb|>| \verb|<|audio\_eos\verb|>|}Audio Transcription {\color{taskcolor} \verb|<Question>? Answer in 10-15 words.|}
% \item \textbf{[QWEN \textbar SQA \textbar Transcription]} {\color{usercolor} USER:} Audio 1: {\color{transcolor} \verb|<|audio\_bos\verb|>| \verb|<|audio\_eos\verb|>|} Audio Transcription: {\color{transcolor} \verb|<Transcription>|} {\color{taskcolor} \verb|<Question>? Answer in 10-15 words.|}
% % QWEN ER
% \item \textbf{[QWEN \textbar ER \textbar System]} {\color{usercolor} SYSTEM:} You are a helpful assistant.
% \item \textbf{[QWEN \textbar ER \textbar Speech]} {\color{usercolor} USER:} Audio 1: {\color{speechcolor} \verb|<|audio\_bos\verb|>| \verb|<|SpeechHere\verb|>| \verb|<|audio\_eos\verb|>|} {\color{taskcolor} Classify the speaker's emotion in one word (anger, happiness, sadness, neutrality).}
% \item \textbf{[QWEN \textbar ER \textbar Transcription]} {\color{usercolor} USER:} Audio 1: {\color{transcolor} \verb|<|audio\_bos\verb|>| \verb|<|audio\_eos\verb|>|} Audio Transcription: {\color{transcolor} \verb|<Transcription>|} {\color{taskcolor} Classify the speaker's emotion in one word (anger, happiness, sadness, neutrality).}
% % Assistant response for all
% \item \textbf{[ALL \textbar Assistant Response]} {\color{responsecolor} ASSISTANT:} \verb|"|
% \end{itemize}
\subsection{Qualitative Analysis}
\label{sec:qualitative analysis}
In Figure \ref{fig:heatmaps}, we qualitatively compare the cross-modal similarity patterns obtained with Qwen2-Audio-Instruct-7B (left) and Salmonn-7B (right), for the LibriSQA dataset (SQA task). We showcase the similarity $S^l$ for layers 0, 5, 10, 32, where layer 0 represents the output after applying a linear transformation, just before the transformer layers of the LLMs. We observe that at layer 0, the similarity patterns exhibit much more vague diagonal-maximal-paths compared to the crisp diagonal-similarity patterns obtained for later layers, which suggests that alignment between modalities become more distinguishable on the later transformer layers. That is the say, we observe a much clear diagonal-like path for later layers (e.g layer 32), compared to the earlier layers (e.g. layer 0, layer 5). Note that layer 0 denotes the representations right after the feature extractor, and layer 1 is the output of the first transformer layer of the LLM.
In Figure \ref{fig:heatmaps}, we also showcase the monotonic alignment search algorithm applied on the similarity patterns. We show the Whisper alignment path with a white trace (estimated by using the word boundaries estimated by Whisper), and we use a red trace for the alignment path found after applying the monotic search algorithm on the similarity matrix $S^l$ for layer $l$ (this is the path that is used to compute ALAS, which we call \emph{the ALAS-path}). We observe that for QWEN, earlier layers indicate more significant deviation from the ground truth path (approximated by Whisper). Especially on layer 5, we observe that the deviation is very visible. However with later layers, we observe that the ALAS-paths get distinctly more similar the Whisper-path. We quantitatively verify these observations in the next section.
%As shown in Figure 2, both Qwen2-Audio-Instruct-7B and Salmonn-7B exhibit distinct alignment patterns across all displayed layers, including the final one. This observation indicates that effective speech-text alignment is crucial for spoken language understanding tasks in multimodal LLMs.
%For Salmonn-7B, the alignment remains consistently strong across the four illustrated layers, whereas Qwen2-Audio-Instruct-7B displays a more dispersed alignment at the fifth layer. Consequently, we performed a quantitative comparison to further investigate how alignment evolves among layers.
%In Figure \ref{}, we qualitatively compare the representations obtained with QWEN and Salmonn.
\subsection{Quantitative Analysis}
In Figure \ref{fig:quantitative}, we show the average-ALAS score with respect to different layers, for QWEN and SALMONN on LibriSQA and IEMOCAP. We observe that for the LibriSQA task, the later layers generally tend to exhibit better ALAS score, as noted for the qualitative comparisons as well, in Section \ref{sec:qualitative analysis}.
We note that at layer 0 (before the transformer layers) the ALAS score is a bit better compared to the immediately subsequent transformer layers of the LLM. This is not very surprising since Qwen2-Audio-Instruct-7B, and SALMONN-7B both uses Whisper-v3 \cite{radford2022robustspeechrecognitionlargescale} as the feature encoder, which outputs semantically informed representations. The application of subsequent transformer layers initially projects these representations that decreases the similarity as shown by the plots, however, the semantic alignment quickly catches up with more layers, and later even decreases further, for instance for QWEN on the LibriSQA dataset. For SALMONN on the LibriSQA dataset, we observe that the alignments closely follow the reference Whisper alignments in absolute sense. This is reasonable since the temporal resolution of SALMONN on the audio domain is lower compared to QWEN.
Another interesting observation that arises from this quantitative analysis is that, unlike the question-answering task on the LibriSQA dataset, for the Emotion Recognition task the level of alignment actually decreases for the last few layers. This is in fact reasonable since Emotion Recognition is not necessarily a semantic task. For certain recordings the speakers express their emotions through speech patterns and prosody, and not necessarily through the semantic content of the audio. This observation therefore suggests that ALAS is able to capture the reduction in semantic alignment between the audio and text modalities for different tasks.
\section{Conclusions}
Semantic alignment between different modalities is an important property for a multimodal LLM, which indicates that the LLM is indeed able to learn semantic relationships between text and the audio modality. Measuring the semantic alignment is therefore an important problem to assess multi-modal LLMs.
In this paper, we propose a new metric (ALAS) for measuring the semantic alignment between the audio and text modalities of multimodal speech LLMs. We have tested this metric for two different tasks within the domain of spoken language understanding (SLU). We have analyzed the ALAS scores qualitatively and quantitatively for two popular multimodal LLMs, for different layers and we observe that ALAS is able to pick-up on expected patterns. These observed patterns include, increasing alignment with respect to layers on a more semantic task like question-answering, and decreasing alignment on a less semantic task such as emotion recognition. We will release our code upon acceptance of this paper to allow researchers to easily evaluate their multi-modal LLMs.
\bibliographystyle{IEEEtran}
\bibliography{mybib}
\newpage
\appendix
\onecolumn
\section{Prompt Structure}
\begin{figure*}[ht]
\centering
\includegraphics[width=0.95\textwidth]{prompt_v2.pdf}
\caption{The prompt structure different models (SALMONN and QWEN) for the Spoken-Question-Answering (SQA), and Emotion Recognition tasks. The tags $<$Question$>$, $<$Transcription$>$, $<$Task$>$, $<$SpeechHere$>$ are replaced with the associated entries for each prompt instance. }
\label{fig:prompt}
\end{figure*}
We describe the prompt structure for our models in Figure \ref{fig:prompt}.
\end{document}
% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{touvron2023llama}
H.~Touvron, T.~Lavril, G.~Izacard, X.~Martinet, M.-A. Lachaux, T.~Lacroix, B.~Rozi{\`e}re, N.~Goyal, E.~Hambro, F.~Azhar \emph{et~al.}, ``Llama: Open and efficient foundation language models,'' \emph{arXiv preprint arXiv:2302.13971}, 2023.
\bibitem{chowdhery2023palm}
A.~Chowdhery, S.~Narang, J.~Devlin, M.~Bosma, G.~Mishra, A.~Roberts, P.~Barham, H.~W. Chung, C.~Sutton, S.~Gehrmann \emph{et~al.}, ``Palm: Scaling language modeling with pathways,'' \emph{Journal of Machine Learning Research}, vol.~24, no. 240, pp. 1--113, 2023.
\bibitem{brown2020language}
T.~Brown, B.~Mann, N.~Ryder, M.~Subbiah, J.~D. Kaplan, P.~Dhariwal, A.~Neelakantan, P.~Shyam, G.~Sastry, A.~Askell \emph{et~al.}, ``Language models are few-shot learners,'' \emph{Advances in neural information processing systems}, vol.~33, pp. 1877--1901, 2020.
\bibitem{du2021glm}
Z.~Du, Y.~Qian, X.~Liu, M.~Ding, J.~Qiu, Z.~Yang, and J.~Tang, ``Glm: General language model pretraining with autoregressive blank infilling,'' \emph{arXiv preprint arXiv:2103.10360}, 2021.
\bibitem{achiam2023gpt}
J.~Achiam, S.~Adler, S.~Agarwal, L.~Ahmad, I.~Akkaya, F.~L. Aleman, D.~Almeida, J.~Altenschmidt, S.~Altman, S.~Anadkat \emph{et~al.}, ``Gpt-4 technical report,'' \emph{arXiv preprint arXiv:2303.08774}, 2023.
\bibitem{peng2023instruction}
B.~Peng, C.~Li, P.~He, M.~Galley, and J.~Gao, ``Instruction tuning with gpt-4,'' \emph{arXiv preprint arXiv:2304.03277}, 2023.
\bibitem{zhang2023instruction}
S.~Zhang, L.~Dong, X.~Li, S.~Zhang, X.~Sun, S.~Wang, J.~Li, R.~Hu, T.~Zhang, F.~Wu \emph{et~al.}, ``Instruction tuning for large language models: A survey,'' \emph{arXiv preprint arXiv:2308.10792}, 2023.
\bibitem{taori2023alpaca}
R.~Taori, I.~Gulrajani, T.~Zhang, Y.~Dubois, X.~Li, C.~Guestrin, P.~Liang, and T.~B. Hashimoto, ``Alpaca: a strong, replicable instruction-following model; 2023,'' \emph{URL https://crfm. stanford. edu/2023/03/13/alpaca. html}.
\bibitem{2023arXiv230605425L}
B.~{Li}, Y.~{Zhang}, L.~{Chen}, J.~{Wang}, F.~{Pu}, J.~{Yang}, C.~{Li}, and Z.~{Liu}, ``{MIMIC-IT: Multi-Modal In-Context Instruction Tuning},'' \emph{arXiv e-prints}, p. arXiv:2306.05425, Jun. 2023.
\bibitem{zhang-etal-2023-video}
\BIBentryALTinterwordspacing
H.~Zhang, X.~Li, and L.~Bing, ``Video-{LL}a{MA}: An instruction-tuned audio-visual language model for video understanding,'' in \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, Y.~Feng and E.~Lefever, Eds.\hskip 1em plus 0.5em minus 0.4em\relax Singapore: Association for Computational Linguistics, Dec. 2023, pp. 543--553. [Online]. Available: \url{https://aclanthology.org/2023.emnlp-demo.49/}
\BIBentrySTDinterwordspacing
\bibitem{tang2023salmonn}
C.~Tang, W.~Yu, G.~Sun, X.~Chen, T.~Tan, W.~Li, L.~Lu, Z.~Ma, and C.~Zhang, ``Salmonn: Towards generic hearing abilities for large language models,'' \emph{arXiv preprint arXiv:2310.13289}, 2023.
\bibitem{chen2023x}
F.~Chen, M.~Han, H.~Zhao, Q.~Zhang, J.~Shi, S.~Xu, and B.~Xu, ``X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages,'' \emph{arXiv preprint arXiv:2305.04160}, 2023.
\bibitem{chu2024qwen2}
Y.~Chu, J.~Xu, Q.~Yang, H.~Wei, X.~Wei, Z.~Guo, Y.~Leng, Y.~Lv, J.~He, J.~Lin \emph{et~al.}, ``Qwen2-audio technical report,'' \emph{arXiv preprint arXiv:2407.10759}, 2024.
\bibitem{wu2023decoder}
J.~Wu, Y.~Gaur, Z.~Chen, L.~Zhou, Y.~Zhu, T.~Wang, J.~Li, S.~Liu, B.~Ren, L.~Liu \emph{et~al.}, ``On decoder-only architecture for speech-to-text and large language model integration,'' in \emph{2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2023, pp. 1--8.
\bibitem{li2023blip}
J.~Li, D.~Li, S.~Savarese, and S.~Hoi, ``Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,'' in \emph{International conference on machine learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2023, pp. 19\,730--19\,742.
\bibitem{dai2023instructblip}
W.~Dai, J.~Li, D.~Li, A.~M.~H. Tiong, J.~Zhao, W.~Wang, B.~Li, P.~Fung, and S.~Hoi, ``Instructblip: Towards general-purpose vision-language models with instruction tuning,'' \emph{arXiv preprint arXiv:2305.06500}, 2023.
\bibitem{rubenstein2023audiopalm}
P.~K. Rubenstein \emph{et~al.}, ``Audiopalm: A large language model that can speak and listen,'' \emph{arXiv preprint arXiv:2306.12925}, 2023.
\bibitem{zhan2024anygpt}
J.~Zhan, J.~Dai, J.~Ye, Y.~Zhou, D.~Zhang, Z.~Liu, X.~Zhang, R.~Yuan, G.~Zhang, L.~Li \emph{et~al.}, ``Anygpt: Unified multimodal llm with discrete sequence modeling,'' \emph{arXiv preprint arXiv:2402.12226}, 2024.
\bibitem{xie2024mini}
Z.~Xie and C.~Wu, ``Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities,'' \emph{arXiv preprint arXiv:2410.11190}, 2024.
\bibitem{tan2024ssr}
W.~Tan, H.~Inaguma, N.~Dong, P.~Tomasello, and X.~Ma, ``Ssr: Alignment-aware modality connector for speech language models,'' \emph{arXiv preprint arXiv:2410.00168}, 2024.
\bibitem{mcauliffe2017montreal}
M.~McAuliffe, M.~Socolof, S.~Mihuc, M.~Wagner, and M.~Sonderegger, ``Montreal forced aligner: Trainable text-speech alignment using kaldi.'' in \emph{Interspeech}, vol. 2017, 2017, pp. 498--502.
\bibitem{Povey_ASRU2011_2011}
D.~Povey, A.~Ghoshal, G.~Boulianne, L.~Burget, O.~Glembek, N.~Goel, M.~Hannemann, P.~Motlicek, Y.~Qian, P.~Schwarz, J.~Silovsky, G.~Stemmer, and K.~Vesely, ``The kaldi speech recognition toolkit,'' in \emph{IEEE 2011 Workshop on Automatic Speech Recognition and Understanding}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE Signal Processing Society, Dec. 2011, iEEE Catalog No.: CFP11SRW-USB.
\bibitem{graves2006connectionist}
A.~Graves, S.~Fern{\'a}ndez, F.~Gomez, and J.~Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in \emph{Proceedings of the 23rd international conference on Machine learning}, 2006, pp. 369--376.
\bibitem{sainath2020emitting}
T.~N. Sainath, R.~Pang, D.~Rybach, B.~Garc{\'\i}a, and T.~Strohman, ``Emitting word timings with end-to-end models.'' in \emph{INTERSPEECH}, 2020, pp. 3615--3619.
\bibitem{huang2024less}
R.~Huang, X.~Zhang, Z.~Ni, L.~Sun, M.~Hira, J.~Hwang, V.~Manohar, V.~Pratap, M.~Wiesner, S.~Watanabe \emph{et~al.}, ``Less peaky and more accurate ctc forced alignment by label priors,'' in \emph{ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2024, pp. 11\,831--11\,835.
\bibitem{barrault2023seamless}
L.~Barrault, Y.-A. Chung, M.~C. Meglioli, D.~Dale, N.~Dong, M.~Duppenthaler, P.-A. Duquenne, B.~Ellis, H.~Elsahar, J.~Haaheim \emph{et~al.}, ``Seamless: Multilingual expressive and streaming speech translation,'' \emph{arXiv preprint arXiv:2312.05187}, 2023.
\bibitem{gong2024zmm}
C.~Gong, X.~Wang, E.~Cooper, D.~Wells, L.~Wang, J.~Dang, K.~Richmond, and J.~Yamagishi, ``Zmm-tts: Zero-shot multilingual and multispeaker speech synthesis conditioned on self-supervised discrete speech representations,'' \emph{IEEE/ACM Transactions on Audio, Speech, and Language Processing}, 2024.
\bibitem{hu2024wavllm}
S.~Hu, L.~Zhou, S.~Liu, S.~Chen, L.~Meng, H.~Hao, J.~Pan, X.~Liu, J.~Li, S.~Sivasankaran \emph{et~al.}, ``Wavllm: Towards robust and adaptive speech large language model,'' \emph{arXiv preprint arXiv:2404.00656}, 2024.
\bibitem{gong2023listen}
Y.~Gong, H.~Luo, A.~H. Liu, L.~Karlinsky, and J.~Glass, ``Listen, think, and understand,'' \emph{arXiv preprint arXiv:2305.10790}, 2023.
\bibitem{zhao2023librisqa}
Z.~Zhao, Y.~Jiang, H.~Liu, Y.~Wang, and Y.~Wang, ``Librisqa: Pioneering free-form and open-ended spoken question answering with a novel dataset and framework,'' \emph{arXiv preprint arXiv:2308.10390}, 2023.
\bibitem{busso2008iemocap}
C.~Busso, M.~Bulut, C.-C. Lee, A.~Kazemzadeh, E.~Mower, S.~Kim, J.~N. Chang, S.~Lee, and S.~S. Narayanan, ``{IEMOCAP: Interactive emotional dyadic motion capture database},'' \emph{Language resources and evaluation}, vol.~42, pp. 335--359, 2008.
\bibitem{kim2020glowttsgenerativeflowtexttospeech}
\BIBentryALTinterwordspacing
J.~Kim, S.~Kim, J.~Kong, and S.~Yoon, ``Glow-tts: A generative flow for text-to-speech via monotonic alignment search,'' 2020. [Online]. Available: \url{https://arxiv.org/abs/2005.11129}
\BIBentrySTDinterwordspacing
\bibitem{radford2022robustspeechrecognitionlargescale}
\BIBentryALTinterwordspacing
A.~Radford, J.~W. Kim, T.~Xu, G.~Brockman, C.~McLeavey, and I.~Sutskever, ``Robust speech recognition via large-scale weak supervision,'' 2022. [Online]. Available: \url{https://arxiv.org/abs/2212.04356}
\BIBentrySTDinterwordspacing
\bibitem{reimers-2019-sentence-bert}
\BIBentryALTinterwordspacing
N.~Reimers and I.~Gurevych, ``Sentence-bert: Sentence embeddings using siamese bert-networks,'' in \emph{Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing}.\hskip 1em plus 0.5em minus 0.4em\relax Association for Computational Linguistics, 11 2019. [Online]. Available: \url{https://arxiv.org/abs/1908.10084}
\BIBentrySTDinterwordspacing
\bibitem{wang2024audiobenchuniversalbenchmarkaudio}
\BIBentryALTinterwordspacing
B.~Wang, X.~Zou, G.~Lin, S.~Sun, Z.~Liu, W.~Zhang, Z.~Liu, A.~Aw, and N.~F. Chen, ``Audiobench: A universal benchmark for audio large language models,'' 2024. [Online]. Available: \url{https://arxiv.org/abs/2406.16020}
\BIBentrySTDinterwordspacing
\end{thebibliography}