Can Quantized Audio Language Models Perform Zero-Shot Spoofing Detection?
Extracted source text · arXiv 2506.06756
\documentclass{Interspeech}
% 2023-10-21 modified by Simon King (Simon.King@ed.ac.uk)
% 2024-01 modified by TPC Chairs of Interspeech 2024
% 2024-10 modified by Antoine Serrurier for Interspeech 2025
% 2024-12 modified by TPC Chairs of Interspeech 2025
% **************************************
% * DOUBLE-BLIND REVIEW SETTINGS *
% **************************************
% Comment out \interspeechcameraready when submitting the
% paper for review.
% If your paper is accepted, uncomment this to produce the
% 'camera ready' version to submit for publication.
\interspeechcameraready
% **************************************
% * *
% * STOP ! DO NOT DELETE ! *
% * READ THIS FIRST *
% * *
% * This template also includes *
% * important INSTRUCTIONS that you *
% * must follow when preparing your *
% * paper. Read it BEFORE replacing *
% * the content with your own work. *
% **************************************
% title here must exactly match the title entered into the paper submission system
% \title{Zero-Shot Audio Spoofing Detection with Quantized Language Models}
% \title{Can Quantized Audio Language Models Enable Zero-Shot Spoofing Detection?}
\title{Can Quantized Audio Language Models Perform Zero-Shot Spoofing Detection?}
% the order of authors here must exactly match the order entered into the paper submission system
% note that the COMPLETE list of authors MUST be entered into the paper submission system at the outset, including when submitting your manuscript for double-blind review
\author[]{Bikash}{Dutta*}
\author[]{Rishabh}{Ranjan*}
\author[]{Shyam}{Sathvik}
\author[]{Mayank}{Vatsa}
\author[]{Richa}{Singh}
%The maximum number of authors in the author list is 20. If the number of contributing authors is more than this, they should be listed in a footnote or the acknowledgement section.
% if you have too many addresses to fit within the available space, try removing the "\\" newlines
\affiliation{}{Indian Institute of Technology Jodhpur}{India}
\email{\{d22cs051, ranjan.4, b22ee036, mvatsa, richa\}@iitj.ac.in}
\keywords{large audio language models, quantization, audio spoofing.}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\usepackage{comment}
\usepackage{multirow}
\begin{document}
\maketitle
\begin{abstract}
% Recent advances in Large Audio Language Models (LALMs) have demonstrated remarkable capabilities in audio processing tasks. However, deploying these sophisticated models in real-world applications presents significant challenges, particularly regarding computational resources and model efficiency. This study investigates the intersection of two critical challenges: the zero-shot capabilities of LALMs in detecting novel audio spoofing attacks and the impact of model quantization on their performance. We conduct a comprehensive analysis of five state-of-the-art LALMs—GAMA, LTU-AS, Meralion, Qwen-Audio, and SALMON—under different quantization scales. Our research evaluates how these models maintain their effectiveness in identifying sophisticated audio spoofing attacks while operating under reduced precision constraints. The study provides empirical insights into the trade-offs between model efficiency and detection accuracy, offering practical guidelines for deploying LALMs in resource-constrained environments without compromising their ability to combat emerging audio security threats.
%CMT ABSTRACT
Quantization is essential for deploying large audio language models (LALMs) efficiently in resource-constrained environments. However, its impact on complex tasks, such as zero-shot audio spoofing detection, remains underexplored. This study evaluates the zero-shot capabilities of five LALMs, GAMA, LTU-AS, MERaLiON, Qwen-Audio, and SALMONN, across three distinct datasets: ASVspoof2019, In-the-Wild, and WaveFake, and investigates their robustness to quantization (FP32, FP16, INT8). Despite high initial spoof detection accuracy, our analysis demonstrates severe predictive biases toward spoof classification across all models, rendering their practical performance equivalent to random classification. Interestingly, quantization to FP16 precision resulted in negligible performance degradation compared to FP32, effectively halving memory and computational requirements without materially impacting accuracy. However, INT8 quantization intensified model biases, significantly degrading balanced accuracy. These findings highlight critical architectural limitations and emphasize FP16 quantization as an optimal trade-off, providing guidelines for practical deployment and future model refinement.
%Recent advances in Large Audio Language Models have demonstrated remarkable capabilities in audio processing tasks. However, deploying these sophisticated models in real-world applications presents significant challenges, particularly regarding computational resources and model efficiency. This study investigates the intersection of two critical challenges: the zero-shot capabilities of LALMs in detecting novel audio spoofing attacks and the impact of model quantization on their performance. We comprehensively evaluate five state-of-the-art LALMs, GAMA, LTU-AS, MERaLiON, Qwen-Audio, and SALMON, under different quantization scales (FP32, FP16, INT8) across three diverse datasets with varying class distributions. Our findings showcase critical limitations in current LALMs for spoofing detection: while models achieve high spoof detection rates, they exhibit severe bias toward spoof classification, resulting in near-zero bonafide detection accuracy. Despite these performance limitations, our analysis provides valuable insights into quantization effects, showing that FP16 precision maintains performance while significantly reducing computational requirements, whereas INT8 quantization exacerbates classification biases. The study provides practical deployment guidelines and identifies critical architectural improvements needed for real-world spoofing detection.
%Recent advances in Large Audio Language Models (LALMs) have demonstrated remarkable capabilities in audio processing tasks. However, deploying these sophisticated models in real-world applications presents significant challenges, particularly regarding computational resources and model efficiency. This study investigates the intersection of two critical challenges: the zero-shot capabilities of LALMs in detecting novel audio spoofing attacks and the impact of model quantization on their performance. We comprehensively analyze five state-of-the-art LALMs—GAMA, LTU-AS, Meralion, Qwen-Audio, and SALMON under different quantization scales. Our research showcases their effectiveness in identifying audio spoofing attacks while operating under reduced precision. The study provides empirical insights into the trade-offs between model efficiency and performance, offering practical guidelines for deploying LALMs in resource-constrained environments without compromising their performance.
\end{abstract}
\def\thefootnote{*}\footnotetext{These authors contributed equally to this work}\def\thefootnote{\arabic{footnote}}
\section{Introduction}
Large Audio Language Models (LALMs) \cite{peng2024survey} have significantly advanced audio processing tasks such as speech recognition, sound event detection, and audio captioning \cite{NEURIPS2023_3a2e5889, gong2023listen}. These models, trained on extensive multimodal datasets, frequently achieve near-human proficiency, demonstrating impressive capabilities in complex audio understanding and generation. A key feature of LALMs is their strong zero-shot learning ability \cite{wang2023neural, ji2024mobilespeech, rag_audio}, enabling them to effectively generalize to new and unseen tasks without explicit task-specific training. This capability is particularly valuable in domains with limited or rapidly evolving labeled data, such as audio spoofing detection.
\begin{figure}[h!]
\centering
\includegraphics[width=\columnwidth]{Images/abstract.pdf}
\caption{Overview of our evaluation framework addressing three research questions (RQs). RQ1 evaluates the zero-shot spoof detection capabilities of prominent LALMs. RQ2 investigates the impact of quantization precision (FP32, FP16, INT8) on model performance and computational efficiency. RQ3 offers practical guidelines for optimizing LALMs for real-world deployment, categorizing model performance as either ‘GOOD’ or ‘BAD’ based on robustness and efficiency.}
\label{fig:zero_shot_performance}
\vspace{-10pt}
\end{figure}
Despite their powerful capabilities, the practical deployment of LALMs is challenging due to substantial computational resource requirements, especially on devices with limited memory and processing power. To address these challenges, quantization has emerged as the essential optimization strategy. Quantization involves reducing the numerical precision of model parameters from higher precision formats, such as 32-bit floating-point, to lower precision formats, such as 16-bit floating-point (FP16) or 8-bit integers (INT8) \cite{quant2021}. This precision reduction significantly decreases memory usage and boosts inference speeds, making it feasible to deploy sophisticated audio-language models in real-time and resource-constrained environments. However, quantization is not without trade-offs, and its impact on the performance of audio tasks such as zero-shot spoof detection, remains largely unexplored.
Audio spoofing detection is an important domain where the potential of LALMs and the challenges of their practical deployment intersect. Rapid advancements in audio synthesis technologies, such as text-to-speech (TTS) and voice conversion (VC), have enabled the creation of highly realistic synthetic audio. These spoofed audio samples pose substantial threats to speaker verification systems, voice-controlled devices, and forensic audio analysis \cite{DBLP:conf/icb/RanjanVS23, ranjan_faking_fluent, ranjan_context_encoded}. Traditional spoof detection methods primarily rely on labeled datasets that rapidly become obsolete due to the continuous evolution of spoofing techniques. Consequently, these conventional approaches struggle to maintain effectiveness against emerging and sophisticated attacks.
In this context, zero-shot learning offers a promising alternative by leveraging pretrained audio-language models to detect spoofed audio samples without requiring labeled examples of new or emerging spoofing techniques. However, the effectiveness and robustness of zero-shot spoof detection capabilities under different quantization conditions remain inadequately studied. Understanding the interplay between zero-shot learning and quantization is crucial for deploying robust audio models in resource-constrained real-world scenarios.
%Zero-shot learning offers a promising solution by leveraging pretrained models to detect spoofed audio without requiring additional labeled examples of new spoofing techniques. However, the effectiveness of zero-shot spoof detection under different quantization constraints remains inadequately studied. Understanding the interplay between zero-shot learning capabilities and model quantization is crucial for effectively deploying robust audio models in real-world scenarios where computational resources are limited.
This study addresses this critical research gap by evaluating the zero-shot spoof detection performance of five state-of-the-art LALMs, GAMA, LTU-AS, MERaLiON, Qwen-Audio, and SALMONN, across different quantization precisions (16-bit and 8-bit). Through detailed experiments and analysis, we explore how these models perform in zero-shot detection scenarios and explicitly quantify the effects of quantization on their accuracy and reliability. Our key contributions are:
\begin{itemize}
\item Conduct the first comprehensive analysis of zero-shot spoof detection capabilities of quantized LALMs, providing foundational insights into inherent strengths and limitations.
\item Deliver empirical insights into trade-offs among quantization precision, model accuracy, and computational efficiency, enabling informed decision-making for practical deployments.
\item Present practical guidelines and recommendations for optimizing LALMs, ensuring robust spoof detection performance within realistic computational constraints.
\end{itemize}
%Large Audio Language Models (LALMs) \cite{peng2024survey} have revolutionized the field of audio processing, demonstrating remarkable capabilities in tasks ranging from speech recognition to sound event detection. These models \cite{NEURIPS2023_3a2e5889, gong2023listen}, trained on vast multimodal datasets, excel in understanding and generating audio content, often approaching human-level performance. A key feature of LALMs is their ability to perform zero-shot learning \cite{wang2023neural, ji2024mobilespeech, rag_audio}, generalizing to unseen tasks without additional training. This capability opens new avenues for solving complex audio processing problems, particularly in scenarios with limited labelled data.
%As LALMs grow in size and complexity, their deployment in real-world applications faces significant challenges, especially on resource-constrained devices. Quantization \cite{quant2021} has emerged as a crucial technique for addressing these computational challenges. By reducing the precision of model parameters from high-precision formats (e.g., 32-bit floating point) to lower-precision formats (e.g., 8-bit or 16-bit integers), quantization can significantly decrease memory usage and improve inference speed. This optimization is particularly important for deploying sophisticated audio-language models in resource-constrained environments or for real-time applications. However, the impact of quantization on model performance, especially in complex tasks like zero-shot audio spoofing detection, remains an open question that requires careful investigation.
%One domain where the potential of LALMs and the challenges of their deployment intersect is audio spoofing detection. The rapid advancement of audio synthesis technologies, such as text-to-speech (TTS) and voice conversion (VC), has led to a proliferation of highly realistic fake audio. These synthetic audio samples pose significant risks to speaker verification systems, voice-controlled devices, and forensic analysis \cite{DBLP:conf/icb/RanjanVS23, ranjan_faking_fluent, ranjan_context_encoded}. Traditional approaches to audio spoofing detection often struggle to keep pace with evolving attack methods, as they rely on labeled datasets that quickly become outdated. Zero-shot learning offers a promising solution to this challenge, potentially enabling LALMs to detect novel spoofing techniques without prior exposure. However, the effectiveness of zero-shot learning in identifying sophisticated audio spoofing attacks, especially when models are subject to quantization, remains unexplored.
%This study aims to bridge this knowledge gap by systematically analyzing the zero-shot capabilities of state-of-the-art LALMs in audio spoofing detection while evaluating the impact of quantization on their performance. We investigate five prominent LALMs: GAMA, LTU-AS, Meralion, Qwen-Audio, and SALMON under two quantization scales: 8-bit and 16-bit precision. By understanding how LALMs perform in zero-shot scenarios and how quantization impacts their capabilities, we can better address the dual challenges of detecting emerging audio spoofing attacks and deploying large-scale models in resource-constrained environments. Our research contributes to the field by:
\begin{figure*}[]
\centering
\includegraphics[width=0.95\linewidth]{Images/accuracy_plot.pdf}
\vspace{-7pt}
\caption{Illustrate \textbf{Accuracy (part a)} and \textbf{Balanced Accuracy (part b)} for five Large Audio Language Models (GAMA, LTU-AS, MERaLiON, Qwen-Audio, and SALMONN) across three datasets: ASVspoof2019 (yellow), In-the-Wild (pink), and WaveFake (purple). The results are presented for three quantization levels: FP32 (left), FP16 (center), and Int8 (right). Each plot highlights the models' ability to detect audio spoofing attacks under varying precision settings, showcasing differences in robustness and generalization capabilities across datasets and quantization strategies.}
\label{fig:accuracy_plot}
\vspace{-5pt}
\end{figure*}
%\begin{itemize}
% \item To the best of our knowledge, this study is the first to investigate the zero-shot spoof detection capabilities of LALMs and the effects of quantization on their performance, thereby providing foundational insights into these critical aspects.
% \item Providing a comprehensive evaluation of the zero-shot capabilities of LALMs in detecting novel audio spoofing attacks.
% \item Empirically analyzing the effects of quantization on model performance in terms of both accuracy and efficiency.
% \item Offering practical guidelines for optimizing LALMs for deployment without compromising their robustness against emerging threats.
%\end{itemize}
% The emergence of Large Audio-Language Models (LLALMs) has revolutionized the field of audio processing, enabling significant advancements in tasks such as speech recognition, sound event detection, and audio-text alignment. These models, trained on vast multimodal datasets, have demonstrated remarkable capabilities in understanding and generating audio content, often rivalling human-level performance in specific domains. A particularly notable feature of these models is their ability to perform \textbf{zero-shot learning}, where they generalize to unseen tasks or domains without requiring additional task-specific training. This capability has opened new avenues for solving complex problems in audio processing, especially in scenarios where labelled data is scarce or impractical to obtain. However, as these models grow larger and more computationally intensive, their deployment in real-world applications, especially on resource-constrained devices, has become increasingly challenging. To address this issue, \textbf{quantization} has emerged as a critical optimization technique, enabling the compression of model parameters to reduce memory usage and computational overhead.
% While LLALMs have demonstrated impressive generalization across a variety of tasks, their application to security-critical domains such as \textbf{audio spoofing detection} remains underexplored. Audio spoofing, which involves generating synthetic or manipulated audio to deceive systems or individuals, has become a growing threat due to advancements in generative technologies like text-to-speech (TTS) and voice conversion (VC). These technologies enable attackers to create highly realistic fake audio with minimal input data, posing significant risks to applications such as speaker verification systems, voice-controlled devices, and forensic analysis. In this context, zero-shot learning offers a promising solution by equipping LLALMs with the ability to detect novel spoofing techniques without requiring prior exposure during training. However, the practical effectiveness of zero-shot learning in detecting sophisticated spoofing attacks remains an open question—one that is further complicated by the potential performance degradation introduced by quantization.
% This study seeks to address these challenges by systematically analyzing the zero-shot capabilities of state-of-the-art LLALMs for audio spoofing detection and evaluating the impact of quantization on their performance. Specifically, we investigate five prominent LLALMs—GAMA, LTU-AS, Meralion, Qwen-Audio, and SALMON—under two quantization scales: 8-bit and 16-bit precision. By examining how these models perform across a suite of zero-shot tasks designed for detecting previously unseen spoofing techniques, we aim to uncover critical insights into the trade-offs between computational efficiency and detection accuracy.
% This paper focuses on evaluating the zero-shot learning capabilities of large audio-language models in the context of audio spoofing detection while simultaneously analyzing how quantization affects their performance. The primary objective is twofold: first, to assess how well these models can detect novel spoofing techniques without prior training on specific attack types; and second, to determine whether quantization-induced constraints compromise their ability to generalize effectively.
% The study involves a rigorous experimental framework that evaluates five leading LLALMs across diverse zero-shot tasks. These tasks simulate real-world scenarios where new spoofing methods must be detected without relying on labeled training data for those specific attacks. Additionally, each model is subjected to two levels of quantization—8-bit and 16-bit precision—to explore the trade-offs between computational efficiency and task performance.
% The importance of this study lies at the intersection of two pressing challenges: ensuring robust detection of emerging audio spoofing attacks and enabling efficient deployment of large-scale models in resource-constrained environments.
% Zero-shot learning is particularly critical for audio spoofing detection because traditional supervised approaches rely heavily on labelled datasets that may not adequately represent future attack scenarios. Generative technologies such as zero-shot TTS and neural codec-based synthesis are evolving rapidly, enabling attackers to produce highly realistic fake audio with minimal input data. For instance, recent advancements allow for cloning a speaker's voice using just a few seconds of speech or generating high-fidelity deepfake speech through neural codecs. These developments make it impractical to anticipate all possible attack vectors during model training.
% Zero-shot learning addresses this limitation by leveraging pre-trained representations learned from large-scale datasets to infer patterns that distinguish genuine from manipulated audio—even when encountering novel attack types for the first time. However, while promising in theory, the practical effectiveness of zero-shot learning in detecting sophisticated spoofing attacks remains underexplored.
% At the same time, deploying large-scale LLALMs for real-time or edge-based applications introduces significant computational challenges. Models like GAMA and Qwen-Audio are resource-intensive due to their size and complexity. Quantization provides a potential solution by reducing model precision from higher formats (e.g., 32-bit floating point) to lower ones (e.g., 8-bit integers). This optimization can significantly reduce memory usage and inference latency while enabling deployment on resource-constrained platforms such as mobile devices or embedded systems.
% However, quantization can introduce numerical errors that degrade model performance—particularly in complex tasks like zero-shot audio spoofing detection that require high sensitivity to subtle acoustic features. Understanding how different levels of quantization affect model robustness is essential for balancing efficiency with effectiveness.
% By investigating these intertwined challenges—zero-shot generalization and quantization—this study aims to provide actionable insights into designing scalable yet robust systems for audio spoofing detection.
% To address these questions, we adopt a systematic approach that combines rigorous experimental design with comprehensive evaluation metrics. Five state-of-the-art LALMs—GAMA, LTU-AS, Meralion, Qwen-Audio, and SALMON—are evaluated across a suite of zero-shot tasks specifically designed for detecting unseen audio spoofing techniques. These tasks include scenarios involving novel forms of voice synthesis and manipulation that were not represented in the training data.
% Each model is subjected to two levels of quantization: 8-bit (low precision) and 16-bit (higher precision). These configurations allow us to explore how computational efficiency impacts task performance under varying levels of precision. Performance is assessed using standard metrics such as equal error rate (EER), classification accuracy, and signal-to-quantization noise ratio (SQNR). Additionally, computational metrics like inference speed and memory footprint are analyzed to evaluate the practical implications of quantization.
% The results are then analyzed to identify patterns in model performance across different quantization levels and zero-shot tasks. This includes examining whether certain architectures are more resilient to quantization-induced degradation or better suited for generalizing across diverse attack types.
% This study makes several key contributions: it provides a detailed evaluation of the zero-shot capabilities of leading LALMs in detecting novel audio spoofing attacks; it empirically analyzes how different quantization scales affect model performance in both accuracy and efficiency; and it offers practical guidelines for optimizing LALMs for deployment in resource-constrained environments without compromising robustness against emerging threats.
% By addressing these challenges holistically, this paper advances the state-of-the-art in scalable audio-language modeling while ensuring robust protection against evolving forms of audio manipulation.
% \section{Experiment Design} To evaluate the zero-shot capabilities of the Large Audio Language Models(LALMs), we use the pre-trained models to test the audio spoofing datasets.
% \begin{figure}[t!]
% \centering
% \includegraphics[width=1\linewidth]{Images/data_dist.pdf}
% \caption{The figure illustrates the distribution of spoofed (pink) and bonafide (green) audio samples in percentages across three datasets: ASVspoof2019, In-the-Wild, and WaveFake. This shows all the possibilities of over- and under-sampling with respect to spoofed and bonafide samples.}
% \label{fig:enter-label}
% \end{figure}
%\vspace{-12pt}
\section{Formulation and Experimental Setup}
Consider an audio sample represented as a sequence of acoustic frames \( X = \{x_1, x_2, \dots, x_T\} \), where each frame \( x_i \in \mathbb{R}^d \), \( T \) is the total number of frames, and \( d \) is the dimensionality of each frame. Let \( f_\theta \) denote a pretrained LALM parameterized by weights \( \theta \). Given an audio sample ${X}$ and a prompt $p$, the LALM outputs a probability distribution over the vocabulary $\mathcal{V}$: $\mathbf{P} = f_\theta({X}, p) \in \Delta^{|\mathcal{V}|-1}$. The binary classification decision is based on comparing the probabilities of the target classification tokens, assigning the class $y = \text{spoof} \text{ if } P(t_{\text{spoof}} \mid X, p) > P(t_{\text{bonafide}} \mid X, p), \text{ else bonafide}$, where \( t_{\text{spoof}}, t_{\text{bonafide}} \in \mathcal{V} \) are the vocabulary tokens corresponding to the class labels.
%\( y = \text{spoof} \) if \( P(t_{\text{spoof}}) > P(t_{\text{bonafide}}) \), and \( y = \text{bonafide} \) otherwise,
In the zero-shot spoof detection scenario, \( f_\theta \) leverages general audio representations learned during pretraining, without explicit fine-tuning on spoof-specific data. Model quantization reduces the precision of parameters, defined by $\theta_{\text{quant}} = \mathcal{Q}_p(\theta)$, where $p \in \{32, 16, 8\}$ represents precision levels (FP32, FP16, INT8 respectively), and $\mathcal{Q}_p$ is the quantization function converting high-precision parameters into lower-precision formats.
This study systematically explores three key research questions: \textbf{(i) RQ1 - Zero-shot spoof detection}, which investigates the intrinsic capability of pretrained LALMs to identify spoofed audio without explicit fine-tuning; \textbf{(ii) RQ2 - Impact of quantization}, examining how varying quantization precision (FP32, FP16, INT8) affects the performance of LALMs; and \textbf{(iii) RQ3 - Deployment optimization}, aimed at providing practical guidelines for selecting optimal quantization precision that balances detection performance with computational efficiency.
\noindent \textbf{Experimental Framework and Baseline Models:}
To rigorously assess the zero-shot spoof detection capabilities, robustness to quantization, and practical deployment considerations of LALMs, we developed a comprehensive experimental framework. We evaluated five representative state-of-the-art pretrained LALMs selected for their architectural diversity and proven effectiveness in audio-related tasks. Specifically, \textbf{GAMA \cite{ghosh-etal-2024-gama}} integrates an Audio Q-Former, which generalizes audio semantics, with multi-layer Audio Spectrogram Transformer (AST) modules capturing detailed audio characteristics, leveraging the pretrained Llama model \cite{touvron2023llama} for advanced audio understanding. \textbf{LTU-AS \cite{gong_ltuas}} combines Whisper's robust encoder-decoder architecture \cite{radford2022whisper} with a specialized time- and layer-wise transformer (TLTR) to jointly process speech transcription and audio event detection, further refining these representations through integration with Llama. \textbf{MERaLiON \cite{he2024meralionaudiollmtechnicalreport}}, specifically tailored for multilingual environments, combines a localized version of Whisper-large-v2 with SEA-LION V3 \cite{sea_lion_2024}, and incorporates an adaptor module explicitly designed to enhance audio-text alignment and multilingual capability. \textbf{Qwen-Audio \cite{Qwen-Audio}}, developed by Alibaba, fuses Whisper-base’s audio processing capabilities with a transformer decoder, enabling versatile audio-text alignment applicable across various audio forms including speech, music, and environmental sounds. Finally, \textbf{SALMONN \cite{tang2024salmonn}} integrates Whisper and BEATs audio encoders through a window-level Q-Former combined with a Vicuna-based \cite{vicuna2023} LLM, providing unified and sophisticated multimodal processing across speech, audio events, and music.
\begin{table}[]
\caption{Distribution of spoofed and bonafide samples across ASVspoof2019, In-the-Wild, and WaveFake datasets.}
\vspace{-5pt}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{|l|c|c|c|}
\hline
Dataset & \multicolumn{1}{l|}{Spoof} & \multicolumn{1}{l|}{Bonafide} & \multicolumn{1}{l|}{Total Samples} \\ \hline
ASVspoof2019 & 63882 & 7355 & 71237 \\ \hline
In-the-Wild & 11816 & 19963 & 31779 \\ \hline
WaveFake & 13100 & 13100 & 26200 \\ \hline
\end{tabular}
}
\label{table:datastats}
\vspace{-15pt}
\end{table}
Our experiments utilize multiple diverse datasets containing bonafide and synthetically spoofed audio samples that were created using various synthesis methods. We specifically selected datasets that reflect distinct class distributions, thereby allowing comprehensive evaluation under realistic scenarios: ASVspoof2019 \cite{nautsch2021asvspoof} is significantly skewed toward spoof samples, making it challenging for models to correctly identify genuine audio; In-the-Wild \cite{muller2022does} predominantly comprises bonafide samples, testing the models’ sensitivity and specificity in a practical context; and WaveFake \cite{frank2021wavefakedatasetfacilitate} presents a balanced dataset equally composed of spoofed and bonafide samples, providing a controlled setting for evaluating model generalization and robustness. Table \ref{table:datastats} summarizes key dataset characteristics. This carefully chosen suite of datasets enables insightful analysis and enhances the practical relevance of our findings for effectively deploying robust LALMs in real-world applications.
\noindent \textbf{Evaluation Metrics and Implementation Details:}
To thoroughly evaluate model performance, we employ a suite of complementary metrics: F1-Score, Accuracy, Balanced Accuracy, Matthews Correlation Coefficient (MCC), and classwise accuracy. The F1-Score provides a balanced measure of precision and recall, particularly valuable in handling class imbalance. Accuracy captures the overall prediction correctness, while Balanced Accuracy averages recall across classes, making it highly effective for imbalanced scenarios. MCC offers a comprehensive assessment of classification quality by considering all confusion matrix categories. Classwise accuracy delivers detailed insights into model behavior across individual classes. All experiments were executed using an NVIDIA A100 GPU, leveraging mixed-precision inferencing to enhance computational efficiency and accelerate matrix computations.
%%%% old one %%%%%
%\section{Experimental Setup}
%To demonstrate the effectiveness of this work, we design an experimental framework that systematically evaluates the capabilities of Audio Language Models (LALMs) across multiple dimensions. Our methodology carefully considers three interconnected aspects of LALM performance that are crucial for real-world deployment. First, we examine the zero-shot capabilities of these models in detecting novel audio spoofing attacks, which is essential because attackers constantly develop new techniques that may not have been seen during training. We carefully select a diverse range of pre-trained LALMs, including GAMA, LTU-AS, Meralion, Qwen-Audio, and SALMON, to ensure our findings are representative of the current state of the art.
%The foundation of our evaluation rests on the multiple diverse datasets, where we use the evaluation subsets of the dataset containing bonafide audio samples and attacked synthesized spoofed audio samples curated through various manipulation techniques. This allows us to evaluate model performance across a spectrum of spoofing methods, ranging from basic to sophisticated audio generation approaches. To gain a deeper understanding of the models' behavior, we deliberately selected three datasets with distinct class distributions: one that is highly imbalanced towards the spoof class (ASVspoof2019 \cite{nautsch2021asvspoof}), one that is imbalanced towards the bonafide class (In-the-Wild \cite{muller2022does}), and one that is perfectly balanced between both classes (WaveFake \cite{frank2021wavefakedatasetfacilitate}) that can be observed in Table \ref{table:datastats}. This selection ensures a comprehensive evaluation of the models' robustness and generalization capabilities under varying data distributions. By examining performance across these datasets, we aim to uncover how class imbalance affects the LALMs' ability to detect spoofed and bonafide audio samples, providing critical insights for their real-world application.
\begin{table*}[h!]
\caption{Comparison of five Large Audio Language Models evaluated on the different datasets. Each row represents a specific metric: F1-Score, Accuracy, Balanced Accuracy, Matthews Correlation Coefficient (MCC), (Spoof or Bonafide) class-wise Accuracy. The columns list the performance of each model under three quantization levels (FP32, FP16, INT8), with values separated by slashes.}
\vspace{-7pt}
% \caption{The table provides a detailed comparison of five Large Audio-Language Models (GAMA, LTU-AS, MERaLiON, Qwen, and SALMONN) evaluated on the datasets ${D_{i\in{ASVspoof2019,In-the-wild,WaveFake}}}$. Each row represents a specific metric: F1-Score, Accuracy, Balanced Accuracy, Matthews Correlation Coefficient (MCC), (Spoof or Bonafide) class-wise Accuracy. The columns list the performance of each model under three quantization levels (FP32, FP16, INT8), with values separated by slashes. This structure allows for a direct comparison of model robustness and efficiency across different precision settings.}
\resizebox{1\linewidth}{!}{
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% \begin{tabular}{|lrrrrrrrrr|}
% \hline
% \multicolumn{10}{|c|}{ASVspoof2019} \\ \hline
% \multicolumn{1}{|c|}{\multirow{2}{*}{Model}} & \multicolumn{3}{c|}{FP32} & \multicolumn{3}{c|}{FP16} & \multicolumn{3}{c|}{INT8} \\ \cline{2-10}
% \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} \\ \hline
% \multicolumn{1}{|l|}{GAMA} & \multicolumn{1}{r|}{0.415} & \multicolumn{1}{r|}{0.570} & \multicolumn{1}{r|}{-0.104} & \multicolumn{1}{r|}{0.422} & \multicolumn{1}{r|}{0.580} & \multicolumn{1}{r|}{-0.096} & \multicolumn{1}{r|}{0.164} & \multicolumn{1}{r|}{0.454} & -0.419 \\ \hline
% \multicolumn{1}{|l|}{LTU-AS} & \multicolumn{1}{r|}{0.488} & \multicolumn{1}{r|}{0.925} & \multicolumn{1}{r|}{-0.039} & \multicolumn{1}{r|}{0.489} & \multicolumn{1}{r|}{0.925} & \multicolumn{1}{r|}{-0.035} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.946} & 0.014 \\ \hline
% \multicolumn{1}{|l|}{MERaLiON} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.946} & \multicolumn{1}{r|}{0.000} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.946} & \multicolumn{1}{r|}{0.000} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.946} & 0.000 \\ \hline
% \multicolumn{1}{|l|}{Qwen-Audio} & \multicolumn{1}{r|}{0.323} & \multicolumn{1}{r|}{0.734} & \multicolumn{1}{r|}{-0.231} & \multicolumn{1}{r|}{0.323} & \multicolumn{1}{r|}{0.733} & \multicolumn{1}{r|}{-0.232} & \multicolumn{1}{r|}{0.321} & \multicolumn{1}{r|}{0.731} & -0.233 \\ \hline
% \multicolumn{1}{|l|}{SALMON} & \multicolumn{1}{r|}{0.458} & \multicolumn{1}{r|}{0.297} & \multicolumn{1}{r|}{-0.065} & \multicolumn{1}{r|}{0.496} & \multicolumn{1}{r|}{0.265} & \multicolumn{1}{r|}{-0.007} & \multicolumn{1}{r|}{0.467} & \multicolumn{1}{r|}{0.293} & -0.053 \\ \hline
% \end{tabular}
\begin{tabular}{|lccccc|}
\hline
\multicolumn{6}{|c|}{\textbf{ASVspoof2019 (FP32 / FP16 / INT8)}} \\ \hline
\multicolumn{1}{|l|}{\textbf{Metrics / Model}} & \multicolumn{1}{c|}{\textbf{GAMA}} & \multicolumn{1}{c|}{\textbf{LTU-AS}} & \multicolumn{1}{c|}{\textbf{Meralion}} & \multicolumn{1}{c|}{\textbf{Qwen}} & \textbf{Salmonn} \\ \hline
\multicolumn{1}{|l|}{\textbf{F1-Score}} & \multicolumn{1}{c|}{0.570 / 0.580 / 0.454} & \multicolumn{1}{c|}{0.925 / 0.925 / 0.946} & \multicolumn{1}{c|}{0.946 / 0.946 / 0.946} & \multicolumn{1}{c|}{0.734 / 0.733 / 0.731} & 0.297 / 0.293 / 0.265 \\ \hline
\multicolumn{1}{|l|}{\textbf{Accuracy}} & \multicolumn{1}{c|}{0.424 / 0.433 / 0.293} & \multicolumn{1}{c|}{0.861 / 0.861 / 0.897} & \multicolumn{1}{c|}{0.897 / 0.897 / 0.897} & \multicolumn{1}{c|}{0.579 / 0.578 / 0.576} & 0.237 / 0.237 / 0.226 \\ \hline
\multicolumn{1}{|l|}{\textbf{Balanced Accuracy}} & \multicolumn{1}{c|}{0.415 / 0.422 / 0.164} & \multicolumn{1}{c|}{0.488 / 0.489 / 0.499} & \multicolumn{1}{c|}{0.500 / 0.500 / 0.500} & \multicolumn{1}{c|}{0.323 / 0.323 / 0.321} & 0.458 / 0.467 / 0.496 \\ \hline
\multicolumn{1}{|l|}{\textbf{MCC}} & \multicolumn{1}{c|}{-0.104 / -0.096 / -0.419} & \multicolumn{1}{c|}{-0.039 / -0.035 / 0.014} & \multicolumn{1}{c|}{0.000 / 0.000 / 0.000} & \multicolumn{1}{c|}{-0.231 / -0.232 / -0.233} & -0.065 / -0.053 / -0.007 \\ \hline
\multicolumn{1}{|l|}{\textbf{Spoof Accuracy}} & \multicolumn{1}{c|}{0.426 / 0.436 / 0.327} & \multicolumn{1}{c|}{0.958 / 0.957 / 1.000} & \multicolumn{1}{c|}{1.000 / 1.000 / 1.000} & \multicolumn{1}{c|}{0.646 / 0.645 / 0.643} & 0.179 / 0.177 / 0.156 \\ \hline
\multicolumn{1}{|l|}{\textbf{Bonafide Accuracy}} & \multicolumn{1}{c|}{0.405 / 0.407 / 0.000} & \multicolumn{1}{c|}{0.017 / 0.020 / 0.001} & \multicolumn{1}{c|}{0.000 / 0.000 / 0.000} & \multicolumn{1}{c|}{0.000 / 0.000 / 0.000} & 0.737 / 0.756 / 0.836 \\ \hline
\end{tabular}
}
\label{table:performance}
\end{table*}
\begin{table*}[h!]
% \caption{The table provides a detailed comparison of five Large Audio-Language Models (GAMA, LTU-AS, MERaLiON, Qwen, and SALMONN) evaluated on the In-the-Wild dataset. Each row represents a specific metric: F1-Score, Accuracy, Balanced Accuracy, Matthews Correlation Coefficient (MCC), (Spoof or Bonafide) class-wise Accuracy. The columns list the performance of each model under three quantization levels (FP32, FP16, INT8), with values separated by slashes. This structure allows for a direct comparison of model robustness and efficiency across different precision settings.}
\resizebox{1\linewidth}{!}{
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% \begin{tabular}{|lrrrrrrrrr|}
% \hline
% \multicolumn{10}{|c|}{In-the-Wild} \\ \hline
% \multicolumn{1}{|c|}{\multirow{2}{*}{Model}} & \multicolumn{3}{c|}{FP32} & \multicolumn{3}{c|}{FP16} & \multicolumn{3}{c|}{INT8} \\ \cline{2-10}
% \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} \\ \hline
% \multicolumn{1}{|l|}{GAMA} & \multicolumn{1}{r|}{0.429} & \multicolumn{1}{r|}{0.014} & \multicolumn{1}{r|}{-0.231} & \multicolumn{1}{r|}{0.433} & \multicolumn{1}{r|}{0.014} & \multicolumn{1}{r|}{-0.223} & \multicolumn{1}{r|}{0.201} & \multicolumn{1}{r|}{0.177} & -0.591 \\ \hline
% \multicolumn{1}{|l|}{LTU-AS} & \multicolumn{1}{r|}{0.438} & \multicolumn{1}{r|}{0.489} & \multicolumn{1}{r|}{-0.254} & \multicolumn{1}{r|}{0.439} & \multicolumn{1}{r|}{0.490} & \multicolumn{1}{r|}{-0.250} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.542} & 0.000 \\ \hline
% \multicolumn{1}{|l|}{MERaLiON} & \multicolumn{1}{r|}{0.563} & \multicolumn{1}{r|}{0.247} & \multicolumn{1}{r|}{0.246} & \multicolumn{1}{r|}{0.565} & \multicolumn{1}{r|}{0.254} & \multicolumn{1}{r|}{0.250} & \multicolumn{1}{r|}{0.560} & \multicolumn{1}{r|}{0.238} & 0.238 \\ \hline
% \multicolumn{1}{|l|}{Qwen-Audio} & \multicolumn{1}{r|}{0.344} & \multicolumn{1}{r|}{0.408} & \multicolumn{1}{r|}{-0.471} & \multicolumn{1}{r|}{0.363} & \multicolumn{1}{r|}{0.425} & \multicolumn{1}{r|}{-0.438} & \multicolumn{1}{r|}{0.355} & \multicolumn{1}{r|}{0.418} & -0.452 \\ \hline
% \multicolumn{1}{|l|}{SALMON} & \multicolumn{1}{r|}{0.616} & \multicolumn{1}{r|}{0.510} & \multicolumn{1}{r|}{0.236} & \multicolumn{1}{r|}{0.620} & \multicolumn{1}{r|}{0.507} & \multicolumn{1}{r|}{0.247} & \multicolumn{1}{r|}{0.661} & \multicolumn{1}{r|}{0.592} & 0.313 \\ \hline
% \end{tabular}
\begin{tabular}{|lccccc|}
\hline
\multicolumn{6}{|c|}{\textbf{In-the-Wild (FP32 / FP16 / INT8)}} \\ \hline
\multicolumn{1}{|l|}{\textbf{Metrics / Model}} & \multicolumn{1}{c|}{\textbf{GAMA}} & \multicolumn{1}{c|}{\textbf{LTU-AS}} & \multicolumn{1}{c|}{\textbf{Meralion}} & \multicolumn{1}{c|}{\textbf{Qwen}} & \textbf{Salmonn} \\ \hline
\multicolumn{1}{|l|}{\textbf{F1-Score}} & \multicolumn{1}{c|}{0.014 / 0.014 / 0.177} & \multicolumn{1}{c|}{0.489 / 0.490 / 0.542} & \multicolumn{1}{c|}{0.247 / 0.254 / 0.238} & \multicolumn{1}{c|}{0.408 / 0.425 / 0.418} & 0.510 / 0.507 / 0.592 \\ \hline
\multicolumn{1}{|l|}{\textbf{Accuracy}} & \multicolumn{1}{c|}{0.536 / 0.542 / 0.193} & \multicolumn{1}{c|}{0.329 / 0.330 / 0.372} & \multicolumn{1}{c|}{0.670 / 0.672 / 0.668} & \multicolumn{1}{c|}{0.256 / 0.270 / 0.264} & 0.648 / 0.657 / 0.662 \\ \hline
\multicolumn{1}{|l|}{\textbf{Balanced Accuracy}} & \multicolumn{1}{c|}{0.429 / 0.433 / 0.201} & \multicolumn{1}{c|}{0.438 / 0.490 / 0.500} & \multicolumn{1}{c|}{0.563 / 0.565 / 0.560} & \multicolumn{1}{c|}{0.344 / 0.363 / 0.355} & 0.616 / 0.620 / 0.661 \\ \hline
\multicolumn{1}{|l|}{\textbf{MCC}} & \multicolumn{1}{c|}{-0.231 / -0.223 / -0.591} & \multicolumn{1}{c|}{-0.254 / -0.250 / 0.000} & \multicolumn{1}{c|}{0.246 / 0.250 / 0.238} & \multicolumn{1}{c|}{-0.254 / -0.250 / 0.000} & 0.236 / 0.313 / 0.247 \\ \hline
\multicolumn{1}{|l|}{\textbf{Spoof Accuracy}} & \multicolumn{1}{c|}{0.009 / 0.009 / 0.233} & \multicolumn{1}{c|}{0.864 / 0.866 / 1.000} & \multicolumn{1}{c|}{0.146 / 0.150 / 0.140} & \multicolumn{1}{c|}{0.864 / 0.866 / 1.000} & 0.493 / 0.475 / 0.659 \\ \hline
\multicolumn{1}{|l|}{\textbf{Bonafide Accuracy}} & \multicolumn{1}{c|}{0.849 / 0.170 / 0.858} & \multicolumn{1}{c|}{0.012 / 0.013 / 0.000} & \multicolumn{1}{c|}{0.981 / 0.980 / 0.981} & \multicolumn{1}{c|}{0.012 / 0.013 / 0.000} & 0.740 / 0.764 / 0.664 \\ \hline
\end{tabular}
}
\end{table*}
%\vspace{-3pt}
%\subsection{Baseline Models}
%We evaluate our approach using five state-of-the-art Large Audio Language Models (LALMs), each chosen for their demonstrated capabilities in audio processing and audio-related tasks. These models represent diverse architectural approaches and have shown promising results in other audio analysis tasks. We provide a detailed examination of each baseline model below:
%\noindent \textbf{GAMA\cite{ghosh-etal-2024-gama}:} is an advanced audio-language model that integrates multiple audio representations with a pre-trained language model to enhance audio understanding and complex reasoning. It employs an Audio Q-Former for semantic generalization and a multi-layer Audio Spectrogram Transformer (AST) with aggregation modules to capture diverse surface-level and high-level audio features, which are then passed to Llama \cite{touvron2023llama}.
%\noindent \textbf{LTU-AS\cite{gong_ltuas}:} is a joint audio-speech understanding model that integrates Whisper's \cite{radford2022whisper} and LLaMA's capabilities to process spoken text, and audio events. The architecture combines Whisper's encoder-decoder system for speech transcription with a time- and layer-wise transformer (TLTR) to extract rich audio embeddings from Whisper's multi-layer representations, which are then passed into a fine-tuned Llama \cite{touvron2023llama}.
%\noindent \textbf{MERaLiON\cite{he2024meralionaudiollmtechnicalreport}:} is a multimodal speech-text large language model specifically designed for Singapore's multilingual environment, combining a localized Whisper-large-v2 \cite{radford2022whisper} and SEA-LION V3 \cite{sea_lion_2024} to handle linguistic diversity, local accents, and code-switching. This model also employs an adaptor module to align audio and text decoders.
%\noindent \textbf{Qwen-Audio\cite{Qwen-Audio}:} is Alibaba's advanced LALM designed for universal audio understanding, processing diverse inputs like speech, music, environmental sounds, and songs alongside text. The architecture integrates Whisper-base with a transformer decoder to align audio with text embeddings.
%\noindent \textbf{SALMON\cite{tang2024salmonn}:} (Speech Audio Language Music Open Neural Network) is a multimodal large language model that integrates speech, audio events, and music processing through a unified architecture. It combines Whisper's and BEATs audio encoder via a window-level Q-Former, which fuses audio inputs into tokens with a Vicuna-based \cite{vicuna2023} LLM.
%These baseline models were selected based on their strong performance on standard audio-related benchmark tasks and their diverse architectural approaches to audio processing.
% \begin{table*}[t!]
% \resizebox{1\linewidth}{!}{%
% \begin{tabular}{llrrrrrrrrrrrrrrr}
% \hline
% \multicolumn{1}{c}{\multirow{2}{*}{\textbf{Quantization}}} & \multicolumn{1}{c}{\multirow{2}{*}{\textbf{Dataset}}} & \multicolumn{3}{c}{\textbf{GAMA}} & \multicolumn{3}{c}{\textbf{LTU-AS}} & \multicolumn{3}{c}{\textbf{Meralion}} & \multicolumn{3}{c}{\textbf{Qwen}} & \multicolumn{3}{c}{\textbf{Salmon}} \\ \cline{3-17}
% \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{l}{F1-Score} & \multicolumn{1}{l}{ROC-AUC} & \multicolumn{1}{l}{MCC} & \multicolumn{1}{l}{F1-Score} & \multicolumn{1}{l}{ROC AUC} & \multicolumn{1}{l}{MCC} & \multicolumn{1}{l}{F1-Score} & \multicolumn{1}{l}{ROC-AUC} & \multicolumn{1}{l}{MCC} & \multicolumn{1}{l}{F1 Score} & \multicolumn{1}{l}{ROC AUC} & \multicolumn{1}{l}{MCC} & \multicolumn{1}{l}{F1 Score} & \multicolumn{1}{l}{ROC AUC} & \multicolumn{1}{l}{MCC} \\ \hline
% \multirow{3}{*}{\textbf{FP32}} & ASVspoof2019 & 0.57 & 0.42 & -0.1 & 0.93 & 0.49 & -0.04 & 0.95 & 0.5 & 0 & 0.73 & 0.32 & -0.23 & 0.3 & 0.46 & -0.07 \\
% & In-the-Wild & 0.01 & 0.43 & -0.23 & 0.49 & 0.44 & -0.25 & 0.25 & 0.56 & 0.25 & 0.41 & 0.34 & -0.47 & 0.51 & 0.62 & 0.24 \\
% & WaveFake & 0.51 & 0.45 & -0.11 & & & & 0.67 & 0.5 & 0 & 0.3 & 0.18 & -0.69 & 0.46 & 0.47 & -0.06 \\ \hline
% \multirow{3}{*}{\textbf{FP16}} & ASVspoof2019 & 0.45 & 0.16 & -0.42 & 0.92 & 0.49 & -0.03 & 0.95 & 0.5 & 0 & 0.73 & 0.32 & -0.23 & 0.29 & 0.47 & -0.05 \\
% & In-the-Wild & 0.18 & 0.2 & -0.59 & 0.49 & 0.44 & -0.25 & 0.25 & 0.57 & 0.25 & 0.42 & 0.36 & -0.45 & 0.51 & 0.62 & 0.25 \\
% & WaveFake & 0.28 & 0.16 & -0.71 & 0.66 & 0.49 & -0.05 & 0.67 & 0.5 & 0 & 0.3 & 0.18 & -0.69 & 0.46 & 0.47 & -0.06 \\ \hline
% \multirow{3}{*}{\textbf{Int8}} & ASVspoof2019 & 0.58 & 0.42 & -0.1 & 0.95 & 0.5 & 0.01 & 0.95 & 0.5 & 0 & 0.73 & 0.32 & -0.23 & 0.26 & 0.5 & -0.01 \\
% & In-the-Wild & 0.01 & 0.43 & -0.22 & 0.54 & 0.5 & 0 & 0.24 & 0.56 & 0.24 & 0.43 & 0.36 & -0.44 & 0.59 & 0.66 & 0.31 \\
% & WaveFake & 0.51 & 0.45 & -0.11 & 0.67 & 0.5 & 0.01 & 0.67 & 0.5 & 0 & 0.3 & 0.18 & -0.69 & 0.44 & 0.47 & -0.06 \\ \hline
% \end{tabular}
% }
% \end{table*}
%\vspace{-6pt}
%\subsection{Evaluation Metrics \& Implementation Details}
%To ensure a comprehensive assessment of our models' performance, we employ multiple complementary evaluation metrics that capture different aspects of detection capability. The F1-Score, computed as the harmonic mean of precision and recall, provides a balanced measure of the model's ability to correctly identify both bonafide and spoofed audio samples while accounting for class imbalance. We also utilize Accuracy, the ratio of correctly predicted observations to total observations, and Balanced Accuracy, which is particularly useful for imbalanced datasets. The Matthews Correlation Coefficient (MCC), provides a reliable metric for classification tasks as it accounts for all four confusion matrix categories. Classwise Accuracy offers insights into the model's performance across different categories. All experiments were conducted on a single NVIDIA A100 GPU, leveraging accelerated matrix computations and utilizing mixed precision inferencing.
% to ensure efficient model evaluation.
%($\text{F1} = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$)
%, defined as $\text{MCC} = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}$
% \begin{figure*}[t!]
% \centering
% \includegraphics[width=1\linewidth]{Images/roc-plots.pdf}
% \caption{Enter Caption}
% \label{fig:enter-label}
% \end{figure*}
%\vspace{-8pt}
\section{Results and Analysis}
This section presents an in-depth analysis addressing our three primary research questions using five state-of-the-art LALMs across multiple datasets and precision levels.
%The following section addresses three key research questions (RQ) concerning the robustness, efficiency, and practical deployment of Large Audio-Language Models (LALMs) for detecting novel audio spoofing attacks. The analysis evaluates zero-shot detection capabilities (RQ1), the impact of quantization on model performance (RQ2), and practical optimization strategies for deployment without compromising robustness (RQ3). Five state-of-the-art LALMs—GAMA, LTU-AS, MERaLiON, Qwen-Audio, and SALMONN—are evaluated across diverse datasets and experimental conditions.
\begin{table*}[h!]
% \caption{Comparison of five Large Audio Language Models. Following the same structure as Table \ref{table:performance}.}
% \vspace{-7pt}
\resizebox{1\linewidth}{!}{
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% \begin{tabular}{|lrrrrrrrrr|}
% \hline
% \multicolumn{10}{|c|}{WaveFake} \\ \hline
% \multicolumn{1}{|c|}{\multirow{2}{*}{Model}} & \multicolumn{3}{c|}{FP32} & \multicolumn{3}{c|}{FP16} & \multicolumn{3}{c|}{INT8} \\ \cline{2-10}
% \multicolumn{1}{|c|}{} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} & \multicolumn{1}{c|}{Balanced Acc} & \multicolumn{1}{c|}{F1 Score} & \multicolumn{1}{c|}{MCC} \\ \hline
% \multicolumn{1}{|l|}{GAMA} & \multicolumn{1}{r|}{0.446} & \multicolumn{1}{r|}{0.511} & \multicolumn{1}{r|}{-0.113} & \multicolumn{1}{r|}{0.445} & \multicolumn{1}{r|}{0.512} & \multicolumn{1}{r|}{-0.114} & \multicolumn{1}{r|}{0.163} & \multicolumn{1}{r|}{0.280} & -0.713 \\ \hline
% \multicolumn{1}{|l|}{LTU-AS} & \multicolumn{1}{r|}{0.495} & \multicolumn{1}{r|}{0.661} & \multicolumn{1}{r|}{-0.039} & \multicolumn{1}{r|}{0.494} & \multicolumn{1}{r|}{0.66} & \multicolumn{1}{r|}{-0.048} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.667} & 0.010 \\ \hline
% \multicolumn{1}{|l|}{MERaLiON} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.667} & \multicolumn{1}{r|}{0.000} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.667} & \multicolumn{1}{r|}{0.000} & \multicolumn{1}{r|}{0.500} & \multicolumn{1}{r|}{0.667} & 0.000 \\ \hline
% \multicolumn{1}{|l|}{Qwen-Audio} & \multicolumn{1}{r|}{0.177} & \multicolumn{1}{r|}{0.300} & \multicolumn{1}{r|}{-0.691} & \multicolumn{1}{r|}{0.177} & \multicolumn{1}{r|}{0.301} & \multicolumn{1}{r|}{-0.691} & \multicolumn{1}{r|}{0.176} & \multicolumn{1}{r|}{0.300} & -0.692 \\ \hline
% \multicolumn{1}{|l|}{SALMON} & \multicolumn{1}{r|}{0.469} & \multicolumn{1}{r|}{0.456} & \multicolumn{1}{r|}{-0.061} & \multicolumn{1}{r|}{0.470} & \multicolumn{1}{r|}{0.456} & \multicolumn{1}{r|}{-0.061} & \multicolumn{1}{r|}{0.470} & \multicolumn{1}{r|}{0.438} & -0.060 \\ \hline
% \end{tabular}
\begin{tabular}{|lccccc|}
\hline
\multicolumn{6}{|c|}{\textbf{WaveFake (FP32 / FP16 / INT8)}} \\ \hline
\multicolumn{1}{|l|}{\textbf{Metrics / Model}} & \multicolumn{1}{c|}{\textbf{GAMA}} & \multicolumn{1}{c|}{\textbf{LTU-AS}} & \multicolumn{1}{c|}{\textbf{Meralion}} & \multicolumn{1}{c|}{\textbf{Qwen}} & \textbf{Salmonn} \\ \hline
\multicolumn{1}{|l|}{\textbf{F1-Score}} & \multicolumn{1}{c|}{0.511 / 0.512 / 0.280} & \multicolumn{1}{c|}{0.661 / 0.660 / 0.667} & \multicolumn{1}{c|}{0.667 / 0.667 / 0.667} & \multicolumn{1}{c|}{0.300 / 0.301 / 0.300} & 0.456 / 0.456 / 0.438 \\ \hline
\multicolumn{1}{|l|}{\textbf{Accuracy}} & \multicolumn{1}{c|}{0.446 / 0.445 / 0.163} & \multicolumn{1}{c|}{0.495 / 0.494 / 0.500} & \multicolumn{1}{c|}{0.500 / 0.500 / 0.500} & \multicolumn{1}{c|}{0.177 / 0.177 / 0.176} & 0.469 / 0.470 / 0.470 \\ \hline
\multicolumn{1}{|l|}{\textbf{Balanced Accuracy}} & \multicolumn{1}{c|}{0.446 / 0.445 / 0.163} & \multicolumn{1}{c|}{0.495 / 0.494 / 0.500} & \multicolumn{1}{c|}{0.500 / 0.500 / 0.500} & \multicolumn{1}{c|}{0.177 / 0.177 / 0.176} & 0.469 / 0.470 / 0.470 \\ \hline
\multicolumn{1}{|l|}{\textbf{MCC}} & \multicolumn{1}{c|}{-0.113 / -0.114 / -0.713} & \multicolumn{1}{c|}{-0.039 / -0.048 / 0.010} & \multicolumn{1}{c|}{0.000 / 0.000 / 0.000} & \multicolumn{1}{c|}{-0.691 / -0.691 / -0.692} & -0.061 / -0.061 / -0.060 \\ \hline
\multicolumn{1}{|l|}{\textbf{Spoof Accuracy}} & \multicolumn{1}{c|}{0.580 / 0.582 / 0.325} & \multicolumn{1}{c|}{0.982 / 0.981 / 1.000} & \multicolumn{1}{c|}{1.000 / 1.000 / 1.000} & \multicolumn{1}{c|}{0.354 / 0.354 / 0.353} & 0.445 / 0.445 / 0.414 \\ \hline
\multicolumn{1}{|l|}{\textbf{Bonafide Accuracy}} & \multicolumn{1}{c|}{0.311 / 0.309 / 0.000} & \multicolumn{1}{c|}{0.009 / 0.008 / 0.000} & \multicolumn{1}{c|}{0.000 / 0.000 / 0.000} & \multicolumn{1}{c|}{0.000 / 0.000 / 0.000} & 0.494 / 0.494 / 0.527 \\ \hline
\end{tabular}
}
\vspace{-8pt}
\end{table*}
\noindent \textbf{RQ1 - Zero-Shot Detection Capabilities:}
We evaluated the zero-shot detection capabilities of GAMA, LTU-AS, MERaLiON, Qwen-Audio, and SALMONN across three distinct datasets: ASVspoof2019, In-the-Wild, and WaveFake. Initial results on the ASVspoof2019 indicated promising spoof detection performance, with MERaLiON and LTU-AS achieving high F1-scores of 0.946 and 0.925, respectively, at FP32 precision. However, deeper examination of balanced accuracy and class-wise accuracy metrics highlights critical issues. We observed a severe bias toward classifying most or all inputs as spoofed. This bias artificially inflated spoof detection accuracy while drastically reducing the ability to detect bonafide samples. As a result, the overall accuracy metrics were misleading and resembled random predictions.
Further dataset-wise analysis provides deeper insights into the distinct behaviors of each model. The spoof-heavy ASVspoof2019 dataset exacerbated model biases, causing robust models such as MERaLiON and LTU-AS to frequently misclassify genuine audio samples as spoofed. The balanced WaveFake dataset offered a clearer assessment of generalization capabilities, but here too, models exhibited substantial biases. The predominantly bonafide In-the-Wild dataset highlighted LALMs' difficulties in accurately detecting genuine audio, emphasizing fundamental limitations in these architectures.
Quantitative evaluations further emphasized these concerns. MERaLiON demonstrated exceptional stability across quantization levels, consistently achieving an identical F1-score (0.946) and overall accuracy (0.897). However, its perfect spoof detection (accuracy of 1.000) came at the complete expense of bonafide detection (accuracy of 0.000). LTU-AS displayed similarly robust spoof detection, with an anomalous improvement in spoof accuracy under INT8 precision (from ~0.958 to 1.000). This anomaly was not indicative of improved performance but rather a reinforcement of the spoof-classification bias. GAMA showed heightened sensitivity to quantization, with its balanced accuracy deteriorating sharply from 0.415 at FP32 precision to 0.164 under INT8 precision, indicating potential vulnerabilities.
Qwen-Audio maintained moderate yet consistent spoof detection performance (F1-scores around 0.732) across precision levels but entirely failed to detect bonafide samples. SALMONN uniquely presented relatively strong bonafide detection capabilities (0.737–0.836 accuracy), despite generally poor overall metrics (F1-score ranging from 0.265 to 0.297). Across all models, MCC values remained consistently low or negative, emphasizing severe prediction bias and inadequate balanced performance. These findings highlight a troubling trend: current LALMs are strongly biased towards spoof detection at the expense of accurate bonafide identification, reflecting critical limitations that must be addressed through substantial architectural refinements to achieve reliable, balanced spoof detection suitable for real-world applications.
%\vspace{20pt}
%\textbf{old - To evaluate the zero-shot performance of LALMs for audio spoofing detection, we tested the models on three datasets: ASVspoof2019, In-the-Wild Spoofing Dataset, and WaveFake. Our analysis reveals significant limitations in the current state of these models for spoof classification. While initial results suggested that MERaLiON and LTU-AS achieved superior performance with F1-scores of 0.925 and 0.946 respectively on ASVspoof2019 in FP32 precision, a deeper examination of balanced accuracy and class-wise accuracy metrics exposed critical shortcomings. Many models, including those with seemingly high accuracy, were found to be classifying all samples into a single class. This behavior results in misleadingly high overall accuracy but extremely poor balanced and class-wise accuracy, effectively rendering their performance equivalent to random prediction. The performance variation illustrated in Figure \ref{fig:accuracy_plot} and Table \ref{table:performance} further highlights these inconsistencies. While models trained with diverse audio-text alignment strategies (e.g., MERaLiON's adaptor module) showed potential for better generalization, the overall results indicate that current LALMs are unsuitable for reliable spoof classification in their present state.}
%\textbf{These findings not only emphasize the importance and need for significant improvements but the performance comparison across all LALMs on the ASVspoof2019 dataset reveals significant variations in detection capabilities and quantization resilience. MERaLiON demonstrates exceptional performance stability, maintaining a consistent F1-score of 0.946 across all quantization levels. This consistency extends to its accuracy metrics, with a steady 0.897 performance regardless of precision reduction. Notably, MERaLiON achieves perfect spoof detection accuracy (1.000) but fails to identify any bonafide samples, suggesting a strong bias toward classifying inputs as spoofed. LTU-AS shows similarly robust performance, with F1-scores ranging from 0.925 to 0.946 across quantization levels. Interestingly, its performance slightly improves under INT8 quantization, with the F1-score increasing from 0.925 to 0.946. This counter-intuitive improvement might suggest that the reduced precision helps mitigate overfitting or provides some regularization effect, but in actuality, the model exhibits extremely high spoof detection accuracy (reaching 1.000 under INT8) but struggles with bonafide detection, achieving only 0.001-0.020 accuracy. GAMA displays moderate performance with notable quantization sensitivity. Its F1-score varies from 0.570 to 0.454 across quantization levels, with significant degradation under INT8 precision. The balanced accuracy shows particularly severe deterioration, dropping from 0.415 to 0.164 under INT8 quantization. This suggests that GAMA's architecture may be more sensitive to precision reduction. Qwen maintains relatively stable but moderate performance across quantization levels, with F1-scores ranging from 0.734 to 0.731. However, like several other models, it shows complete failure in bonafide detection (0.000 accuracy) while maintaining moderate spoof detection capability (around 0.645). SALMON demonstrates the lowest overall F1-scores (0.265-0.297) but shows an interesting pattern of being the only model with relatively strong bonafide detection capabilities (0.737-0.836 accuracy). This unique characteristic suggests a fundamentally different approach to the classification task compared to other models. The Matthews Correlation Coefficient (MCC) values are consistently low or negative across all models, indicating suboptimal overall classification performance despite high individual metric scores in some cases. This suggests that while models may excel at detecting spoofed samples, they struggle to maintain balanced performance across both classes. These results highlight a concerning trend across most models: a strong bias toward spoof detection at the expense of bonafide recognition. This imbalance suggests that current LALM architectures may need significant refinement to achieve more balanced detection capabilities for real-world applications.}
% To evaluate the zero-shot performance of Language-Audio Large Models (LALMs) in detecting novel audio spoofing attacks, we tested the models on three datasets: ASVspoof2019, In-the-Wild Spoofing Dataset, and WaveFake. Our analysis reveals significant limitations in the current state of these models for spoof classification. While initial results suggested that \textbf{MERaLiON} and \textbf{LTU-AS} achieved superior performance with F1-scores of 0.925 and 0.946 respectively on ASVspoof2019 in FP32 precision, a deeper examination of balanced accuracy and class-wise accuracy metrics exposed critical shortcomings. Many models, including those with seemingly high accuracy, were found to be classifying all samples into a single class. This behavior results in misleadingly high overall accuracy but extremely poor balanced and class-wise accuracy, effectively rendering their performance equivalent to random prediction. The performance variation illustrated in Figure \ref{fig:accuracy_plot} and Table \ref{table:performance} further underscores these inconsistencies. While models trained with diverse audio-text alignment strategies (e.g., MERaLiON's adaptor module) showed potential for better generalization, the overall results indicate that current LALMs are unsuitable for reliable spoof classification in their present state. These findings not only emphasize the importance and need for significant improvements.
% \begin{figure}[t!]
% \centering
% \includegraphics[width=\columnwidth]{images/zero_shot_performance.pdf}
% \caption{}
% \label{fig:zero_shot_performance}
% \end{figure}
\noindent\textbf{RQ2 - Effects of Quantization on Model Performance:} Evaluating quantization at FP32, FP16, and INT8 precision levels showed performance trade-offs, highlighting the varying sensitivity of LALMs to reduced numerical precision. Generally, lowering precision negatively impacted model accuracy and reliability. However, we observed instances where INT8 quantization seemingly improved certain metrics, notably spoof detection accuracy. Upon deeper inspection, we identified these apparent improvements as statistical artifacts resulting from increased classification bias toward the spoof class. Specifically, INT8-quantized models excessively favored spoof predictions, artificially inflating overall accuracy while severely degrading balanced accuracy and genuine audio detection capability
%Upon deeper inspection, we identified these instances as artifacts resulting from increased bias toward spoof classification, where models at INT8 precision excessively predicted the spoof class, inflating overall accuracy while severely degrading balanced accuracy and genuine audio detection.
A practically significant finding was the minimal performance difference between FP32 and FP16 across most models. Models such as MERaLiON and LTU-AS demonstrated almost identical performance at both FP32 and FP16 precision levels, suggesting FP16 quantization as an optimal practical choice. Specifically, FP16 precision can substantially reduce memory usage, approximately halving the required computational resources, without negatively impacting genuine detection performance. This finding positions FP16 quantization as an efficient strategy for deploying LALMs in real-world, resource-constrained scenarios.
GAMA, on the other hand, exhibited pronounced sensitivity to quantization, with a dramatic decline in balanced accuracy from FP32 to INT8 precision, indicating that certain architectural elements may significantly influence model resilience to precision reduction. In contrast, MERaLiON maintained consistency across all precision levels, potentially attributable to its adaptive audio-text alignment modules and multilingual training strategy. These modules likely enhance the robustness of internal representations, helping mitigate performance degradation under reduced precision conditions. Similarly, LTU-AS’s performance stability at FP16 precision suggests that integrating multiple transformer layers and multimodal embedding strategies can provide quantization resilience.
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[]
\caption{Comparison of memory usage (GB) and inference time (seconds per 100 samples) at different quantization precisions.}
\vspace{-5pt}
\label{table:mem-time}
\resizebox{1\linewidth}{!}{
\begin{tabular}{|l|cc|cc|cc|}
\hline
\multirow{2}{*}{\textbf{Model / Factor}} & \multicolumn{2}{c|}{\textbf{FP32}} & \multicolumn{2}{c|}{\textbf{FP16}} & \multicolumn{2}{c|}{\textbf{INT8}} \\ \cline{2-7}
& \multicolumn{1}{c|}{\textbf{Memory}} & \textbf{Time} & \multicolumn{1}{c|}{\textbf{Memory}} & \textbf{Time} & \multicolumn{1}{c|}{\textbf{Memory}} & \textbf{Time} \\ \hline
\textbf{GAMA} & \multicolumn{1}{c|}{26.19} & 129.06 & \multicolumn{1}{c|}{13.12} & 126.20 & \multicolumn{1}{c|}{6.94} & 177.76 \\ \hline
\textbf{LTU-AS} & \multicolumn{1}{c|}{25.35} & 98.67 & \multicolumn{1}{c|}{12.71} & 91.43 & \multicolumn{1}{c|}{6.74} & 150.49 \\ \hline
\textbf{Meralion} & \multicolumn{1}{c|}{37.05} & 90.77 & \multicolumn{1}{c|}{18.49} & 57.55 & \multicolumn{1}{c|}{10.24} & 107.31 \\ \hline
\textbf{Qwen} & \multicolumn{1}{c|}{31.35} & 52.51 & \multicolumn{1}{c|}{15.64} & 50.06 & \multicolumn{1}{c|}{9.12} & 112.87 \\ \hline
\textbf{Salmonn} & \multicolumn{1}{c|}{51.47} & 1020.38 & \multicolumn{1}{c|}{27.34} & 960.38 & \multicolumn{1}{c|}{16.08} & 1140.80 \\ \hline
\end{tabular}
}
\vspace{-10pt}
\end{table}
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
% TIME TIME TIME TIME
As precision decreases from FP32 to INT8, memory consumption consistently drops across all models. Moving from FP32 to FP16 reduces both memory consumption and inference time for all models as shown in Table \ref{table:mem-time}, demonstrating clear efficiency gains. However, while INT8 quantization achieves the greatest memory reduction, it paradoxically increases inference time compared to FP16. This suggests that INT8 quantization, while highly effective for memory optimization, introduces computational overhead due to data type conversions and associated processing, negating its execution-speed benefits.
%\vspace{50pt}
%\textbf{Old - The impact of quantization on model accuracy and efficiency is thoroughly analyzed by evaluating LALMs under FP32, FP16, and Int8 precision settings, revealing a complex landscape of performance trade-offs. As anticipated, model performance generally degraded with reduced precision; however, our investigation uncovered some intriguing anomalies. In certain instances, quantized models appeared to outperform their higher-precision counterparts that can be observed in Table \ref{table:performance}, a phenomenon that initially puzzled us. Upon deeper assessment we found that these were because of model predicting only single class that can also be observed in class-wise accuracy in Table \ref{table:performance} effectively reducing models to a random predictor, emphasizing the critical importance of thorough, multi-faceted evaluation in quantization studies. Notably, our analysis revealed only minute performance differences between FP32 and FP16 precision across most models, a finding with significant implications for real-world deployments. This negligible performance gap suggests that FP16 models could be preferentially employed in resource-constrained environments, offering substantial memory and computational savings (theoretically halves/ practically approx. one-third of original memory requirements) without meaningful sacrifices in performance.}
% The impact of quantization on model accuracy and efficiency was analyzed by evaluating LALMs under FP32, FP16, and Int8 precision settings. While FP32 precision delivered the highest accuracy across all models, Int8 quantization provided a balanced trade-off between computational efficiency and performance for robust models like \textbf{MERaLiON} and \textbf{LTU-AS}. For instance, MERaLiON's F1-score on ASVspoof2019 dropped marginally from 0.95 (FP32) to 0.92 (Int8), while its inference latency reduced by 40\%. However, \textbf{Qwen-Audio} exhibited significant degradation under Int8 quantization, with its ROC-AUC dropping from 0.62 (FP32) to 0.48 (Int8), highlighting its sensitivity to reduced precision. Figure~\ref{fig:quantization_tradeoff} visualizes the trade-offs between accuracy and efficiency across different quantization settings.
% \begin{figure}[t!]
% \centering
% \includegraphics[width=\columnwidth]{images/quantization_tradeoff.pdf}
% \caption{}
% \label{fig:quantization_tradeoff}
% \end{figure}
\noindent\textbf{RQ3 - Practical Guidelines for Deployment:}
In our experiments, none of the tested LALMs demonstrated sufficient balanced performance or reliability suitable for immediate real-world deployment in spoof detection scenarios. The prevalent issue of pronounced predictive bias toward spoof classification significantly undermines their practical applicability, highlighting an urgent need for architectural and methodological refinements. Nevertheless, our findings offer clear and actionable insights for practical deployment strategies. FP16 quantization consistently emerged as the most favorable precision level across evaluated models, delivering a near-optimal balance between computational efficiency and detection accuracy. The minimal performance difference between FP32 and FP16 precision levels observed for models such as MERaLiON and LTU-AS reinforces the practicality of deploying FP16 quantized LALMs. Specifically, FP16 precision substantially reduces computational and memory requirements, effectively halving resource usage, making it particularly suitable for edge computing and mobile application scenarios.
INT8 quantization presents a more complex and nuanced trade-off. While theoretically ideal for extremely resource-constrained environments, our analysis shows significant risks, as INT8 quantization notably exacerbates predictive biases, severely compromising balanced classification performance. Therefore, INT8 deployment thus requires cautious consideration and rigorous pre-deployment validation processes to ensure reliability and performance integrity in practical scenarios.
%Our study highlights the necessity of a comprehensive evaluation framework that includes not only standard metrics like accuracy but also critical balanced metrics such as balanced accuracy, class-wise accuracy, and MCC. Such a holistic evaluation approach is essential to accurately assess model biases and genuine detection capabilities, thereby ensuring effective and reliable real-world deployment.
Finally, our results suggest clear directions for future research and practical deployment enhancements. Architectural features such as adaptive alignment modules, robust multimodal embeddings, and multilingual training appear critical for enhancing quantization resilience. Incorporating these elements could mitigate quantization-induced biases, substantially improving the reliability and performance of LALMs in resource-constrained environments. These insights provide a foundational roadmap for future efforts aimed at refining LALMs into truly effective, balanced, and robust models suitable for real-world audio spoof detection.
%\vspace{20pt}
%\textbf{old - Our comprehensive analysis of LALMs for audio spoofing detection has yielded critical insights that significantly impact deployment strategies and future research directions. Foremost among our findings is that, in their current state, none of the evaluated models demonstrate sufficient reliability or robustness to be considered deployment-ready for real-world spoof detection applications. This highlights the urgent need for substantial improvements. Despite this limitation, our study has uncovered valuable insights that can guide future developments and optimizations. Notably, The negligible performance gap suggests that FP16 models could be preferentially employed in memory-constrained environments, offering substantial computational savings without significant accuracy trade-offs. However, our investigation into Int8 quantization revealed a more complex landscape. While Int8 quantization can dramatically reduce computational requirements, making models feasible for deployment on low-resource machines, it demands complete in-depth pre-deployment evaluation. Our research uncovered counterintuitive scenarios where quantized models appeared to outperform their higher-precision counterparts, a phenomenon that, upon deeper scrutiny, proved to be a statistical artifact rather than a genuine improvement. This finding empirically emphasizes the critical importance of comprehensive, multi-faceted evaluation protocols before any deployment decision.}
% Based on empirical findings, we propose the following guidelines for optimizing LALMs for deployment without compromising their robustness against emerging spoofing threats:
% \begin{itemize}
% \item \textbf{Model Selection:} Robust models like MERaLiON and LTU-AS are recommended for deployment due to their consistent performance across datasets and quantization levels.
% \item \textbf{Quantization Strategy:} Int8 quantization is suitable for computationally constrained environments but should be applied cautiously to sensitive models like Qwen-Audio.
% \item \textbf{Dataset Diversity:} Incorporating diverse training data that includes non-speech sounds (e.g., environmental noises) and varied spoofing techniques can enhance zero-shot capabilities.
% \item \textbf{Instruction Tuning:} Models trained with synthetic instruction datasets demonstrate improved reasoning abilities and should be prioritized for tasks requiring complex decision-making.
% \end{itemize}
% In conclusion, this study highlights the strengths and limitations of state-of-the-art LALMs in detecting novel audio spoofing attacks under zero-shot settings. While MERaLiON and LTU-AS lead in robustness and efficiency, further research into adaptive quantization techniques is essential to address vulnerabilities in less resilient models like Qwen-Audio. These insights provide a roadmap for deploying LALMs in real-world applications while ensuring resilience against emerging threats.
% In light of these insights, we propose a nuanced approach to LALM optimization and deployment: (1) Prioritize robust models like MERaLiON and LTU-AS that demonstrate consistent performance across datasets and quantization levels; (2) Leverage FP16 precision as a default for balancing performance and efficiency; (3) Approach Int8 quantization with caution, especially for sensitive models, and always conduct thorough pre-deployment evaluations. These guidelines, derived from our empirical findings, offer deep insights for model deployment in real world scenarios.
\section{Conclusion}
This study presents the first comprehensive evaluation of zero-shot spoof detection capabilities of quantized LALMs, uncovering significant biases toward spoof classification that undermine practical applicability. Despite promising initial spoof detection results, deeper analysis of balanced metrics revealed fundamental limitations across all evaluated models, emphasizing the urgent need for substantial architectural refinements. Our analysis elucidates clear performance-precision trade-offs, highlighting FP16 quantization as an optimal choice that effectively halves memory and computational requirements with negligible accuracy degradation compared to FP32. Conversely, INT8 quantization, despite theoretical benefits in extreme resource-constrained scenarios, significantly intensifies predictive biases, requiring cautious consideration supported by rigorous pre-deployment validation. Additionally, our results showcased an interesting trade-off: while INT8 quantization significantly reduces memory consumption, it unexpectedly increases inference time compared to FP16. This highlights the necessity of carefully evaluating quantization strategies not only in terms of memory but also computational speed. Future architectures should incorporate adaptive alignment modules, robust multimodal embeddings, and multilingual training for improved quantization resilience. These strategies provide foundational guidance for deploying reliable, efficient LALMs in spoof detection.
%This study presents the first comprehensive evaluation of zero-shot spoof detection capabilities of quantized large audio-language models (LALMs), uncovering significant biases toward spoof classification that undermine practical applicability. Despite promising initial spoof detection results, deeper analysis of balanced metrics revealed fundamental limitations across all evaluated models, emphasizing the urgent need for substantial architectural refinements. Our analysis also elucidates clear performance-precision trade-offs. Specifically, FP16 quantization emerges as an optimal choice, effectively halving memory and computational requirements with negligible accuracy degradation compared to FP32. In contrast, INT8 quantization, despite its theoretical benefits for extreme resource constraints, significantly intensifies predictive biases and thus requires cautious consideration supported by rigorous pre-deployment validation processes. Future architectural improvements should explicitly incorporate adaptive alignment modules, robust multimodal embeddings, and multilingual training strategies to enhance quantization resilience and balanced classification performance. These practical insights provide foundational guidance toward deploying reliable, efficient, and robust LALMs for real-world audio spoof detection applications.
%This study presents the first comprehensive analysis of zero-shot spoofing detection capabilities in quantized LALMs. Through systematic evaluation of five state-of-the-art models across multiple datasets and quantization levels, we showcase critical limitations in current LALMs: severe classification bias toward spoof detection with near-zero bonafide accuracy, rendering them unsuitable for practical deployment despite high overall accuracy metrics. Our quantization analysis also demonstrates that FP16 precision offers an optimal balance, maintaining performance while halving computational requirements, whereas INT8 quantization exacerbates existing biases. We provide practical deployment guidelines: (a) prioritize robust models like MERaLiON and LTU-AS, (b) leverage FP16 as the default precision for efficiency without performance loss, and (c) approach INT8 quantization cautiously with thorough pre-deployment validation. These findings highlight fundamental architectural limitations requiring urgent attention before LALMs can achieve reliable, balanced spoofing detection for real-world applications. Our work establishes a foundation for future research directions toward more robust and practically deployable audio language models.
%This study presents an analysis of the zero-shot capabilities of state-of-the-art LALMs in the critical domain of audio spoofing detection, while simultaneously evaluating the nuanced impact of quantization on their performance. By meticulously investigating SOTA LALMs on varying quantization scales, we explain the intricate interplay between model complexity and computational efficiency for audio spoof detection. Our research unveils substantial variability in detection performance across different LALMs and quantization levels, driven by inherent challenges of zero-shot learning. In light of these insights, we propose a nuanced approach to LALM optimization and deployment: (a) prioritizing robust models like MERaLiON and LTU-AS that demonstrate consistent performance across datasets and quantization levels, (b) leveraging FP16 precision as a default for balancing performance and efficiency, and approaching Int8 quantization with caution, especially for sensitive models, while conducting thorough pre-deployment evaluations. (c) The study's multifaceted contributions, encompassing a comprehensive evaluation of zero-shot capabilities, an empirical analysis of quantization effects, and the formulation of practical optimization guidelines, serve as a cookbook for future advancements in the field. This research not only bridges critical knowledge gaps but also opens up new pathways for the next iterations of LALMs, fostering more robust, efficient, and adaptable models.
\vspace{-8pt}
\section{Acknowledgment}
This research is supported by a grant from the NSM, MeitY. The authors also gratefully acknowledge the support of IndiaAI and Meta through Srijan: Centre of Excellence for Generative AI.
% 3 table 3 dataset columns -> models,
\bibliographystyle{IEEEtran}
\bibliography{refs, wav}
\end{document}
% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{peng2024survey}
J.~Peng, Y.~Wang, Y.~Xi, X.~Li, X.~Zhang, and K.~Yu, ``A survey on speech large language models,'' \emph{in arXiv-2410.18908, CoRR}, 2025.
\bibitem{NEURIPS2023_3a2e5889}
S.~Deshmukh, B.~Elizalde, R.~Singh, and H.~Wang, ``Pengi: An audio language model for audio tasks,'' in \emph{Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems}, 2023.
\bibitem{gong2023listen}
Y.~Gong, H.~Luo, A.~H. Liu, L.~Karlinsky, and J.~R. Glass, ``Listen, think, and understand,'' in \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem{wang2023neural}
C.~Wang, S.~Chen, Y.~Wu, Z.~Zhang, L.~Zhou, S.~Liu, Z.~Chen, Y.~Liu, H.~Wang, J.~Li, L.~He, and et. al., ``Neural codec language models are zero-shot text to speech synthesizers,'' \emph{IEEE Transactions on Audio, Speech and Language Processing}, vol.~33, pp. 705--718, 2025.
\bibitem{ji2024mobilespeech}
S.~Ji, Z.~Jiang, H.~Wang, J.~Zuo, and Z.~Zhao, ``Mobilespeech: {A} fast and high-fidelity framework for mobile zero-shot text-to-speech,'' in \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, 2024, pp. 13\,588--13\,600.
\bibitem{rag_audio}
B.~Dutta, R.~Ranjan, A.~Jain, R.~Singh, and M.~Vatsa, ``Can rag-driven enhancements amplify audio llms for low-resource languages?'' in \emph{IEEE International Conference on Acoustics, Speech and Signal Processing}, 2025, pp. 1--5.
\bibitem{quant2021}
M.~Nagel, M.~Fournarakis, R.~A. Amjad, Y.~Bondarenko, M.~van Baalen, and T.~Blankevoort, ``A white paper on neural network quantization,'' \emph{in arXiv-2106.08295, CoRR}, 2021.
\bibitem{DBLP:conf/icb/RanjanVS23}
R.~Ranjan, M.~Vatsa, and R.~Singh, ``Sv-deit: Speaker verification with deitcap spoofing detection,'' in \emph{{IEEE} International Joint Conference on Biometrics}, 2023, pp. 1--10.
\bibitem{ranjan_faking_fluent}
R.~Ranjan, B.~Dutta, M.~Vatsa, and R.~Singh, ``Faking fluent: Unveiling the achilles' heel of multilingual deepfake detection,'' in \emph{{IEEE} International Joint Conference on Biometrics}, 2024, pp. 1--10.
\bibitem{ranjan_context_encoded}
R.~Ranjan, M.~Vatsa, and R.~Singh, ``Context encoded multi-modal attention network for detecting audio spoofing,'' in \emph{{IEEE} International Joint Conference on Biometrics}, 2024, pp. 1--11.
\bibitem{ghosh-etal-2024-gama}
S.~Ghosh, S.~Kumar, A.~Seth, C.~K.~R. Evuru, U.~Tyagi, S.~Sakshi, O.~Nieto, R.~Duraiswami, and D.~Manocha, ``{GAMA:} {A} large audio-language model with advanced audio understanding and complex reasoning abilities,'' in \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, 2024, pp. 6288--6313.
\bibitem{touvron2023llama}
H.~Touvron, T.~Lavril, G.~Izacard, X.~Martinet, M.~Lachaux, T.~Lacroix, B.~Rozi{\`{e}}re, N.~Goyal, E.~Hambro, F.~Azhar, A.~Rodriguez, A.~Joulin, E.~Grave, and G.~Lample, ``Llama: Open and efficient foundation language models,'' \emph{in arXiv-2302.13971, CoRR}, 2023.
\bibitem{gong_ltuas}
Y.~Gong, A.~H. Liu, H.~Luo, L.~Karlinsky, and J.~R. Glass, ``Joint audio and speech understanding,'' in \emph{{IEEE} Automatic Speech Recognition and Understanding Workshop}, 2023, pp. 1--8.
\bibitem{radford2022whisper}
A.~Radford, J.~W. Kim, T.~Xu, G.~Brockman, C.~McLeavey, and I.~Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in \emph{Proceedings of the 40th International Conference on Machine Learning}, 2023, pp. 28\,492--28\,518.
\bibitem{he2024meralionaudiollmtechnicalreport}
Y.~He, Z.~Liu, S.~Sun, B.~Wang, W.~Zhang, X.~Zou, N.~F. Chen, and A.~T. Aw, ``Meralion-audiollm: Bridging audio and language with large language models,'' \emph{in arXiv-2412.09818, CoRR}, 2024.
\bibitem{sea_lion_2024}
AI-Singapore, ``Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia,'' in \url{https://github.com/aisingapore/sealion}, 2024.
\bibitem{Qwen-Audio}
Y.~Chu, J.~Xu, X.~Zhou, Q.~Yang, S.~Zhang, Z.~Yan, C.~Zhou, and J.~Zhou, ``Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,'' \emph{in arXiv-2311.07919, CoRR}, 2023.
\bibitem{tang2024salmonn}
C.~Tang, W.~Yu, G.~Sun, X.~Chen, T.~Tan, W.~Li, L.~Lu, Z.~Ma, and C.~Zhang, ``{SALMONN:} towards generic hearing abilities for large language models,'' in \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem{vicuna2023}
W.-L. Chiang, Z.~Li, Z.~Lin, Y.~Sheng, Z.~Wu, H.~Zhang, L.~Zheng, S.~Zhuang, Y.~Zhuang, J.~E. Gonzalez, I.~Stoica, and E.~P. Xing, ``Vicuna: An open-source chatbot impressing gpt-4 with 90\%* chatgpt quality,'' in \url{https://lmsys.org/blog/2023-03-30-vicuna}, 2023.
\bibitem{nautsch2021asvspoof}
A.~Nautsch, X.~Wang, N.~W.~D. Evans, T.~H. Kinnunen, V.~Vestman, M.~Todisco, H.~Delgado, M.~Sahidullah, J.~Yamagishi, and K.~A. Lee, ``Asvspoof 2019: Spoofing countermeasures for the detection of synthesized, converted and replayed speech,'' \emph{{IEEE} Transactions on Biometrics, Behavior, and Identity Science}, vol.~3, no.~2, pp. 252--265, 2021.
\bibitem{muller2022does}
N.~M. M{\"{u}}ller, P.~Czempin, F.~Dieckmann, A.~Froghyar, and K.~B{\"{o}}ttinger, ``Does audio deepfake detection generalize?'' in \emph{International Speech Communication Association}, 2022, pp. 2783--2787.
\bibitem{frank2021wavefakedatasetfacilitate}
J.~Frank and L.~Sch{\"{o}}nherr, ``Wavefake: {A} data set to facilitate audio deepfake detection,'' in \emph{Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track}, 2021.
\end{thebibliography}