RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection

EEG-based auditory attention decoding has progressed from handcrafted pipelines to end-to-end models that jointly capture spectral, spatial, and temporal neural patterns. MBSSFCC (Jiang et al. 2022) combines multi-band differential-entropy topographies with ConvLSTM for spatio-temporal attention representation. BSAnet (Cai et al. 2023) adopts a biologically inspired spiking attention network to model attention-related EEG activity. DBPNet (Ni et al. 2024) employs parallel temporal-attentive and frequency-residual branches to capture dynamic and multi-band spectral-spatial information. DARNet (Yan et al. 2024) integrates spatio-temporal construction with dual-attention refinement for spatial distribution and long-range dependency modeling. DHGCN (Zhou et al. 2025) constructs temporal and spatial hypergraphs to learn higher-order dependencies across time points and channels. FAConformer (Wang et al. 2026a) combines band-specific CNN-Transformer encoders, frequency-aware attention, and band-wise supervision to learn within-band representations and adaptive cross-band interactions.

Despite these advances, EEG remains susceptible to low signal-to-noise ratios and environmental artifacts, which hinder stable representation learning under varying recording conditions. Moreover, EEG-only decoding may not fully characterize auditory attention in realistic audio-visual scenes. These limitations motivate multimodal AAD models to exploit complementary information while preserving robust neural representations.

EOG signals provide behavioral evidence complementary to neural activity and have gained increasing attention in multimodal AAD. Gehmacher et al. (Gehmacher et al. 2024) combined eye tracking with magnetoencephalography and showed that gaze followed attended natural speech, and was associated with attention-related neural responses. Wilroth et al. (Wilroth et al. 2025b) inferred attended-speech labels from eye-tracking features to supervise portable EEG-based speech-reconstruction models when ground-truth labels were unavailable. Kosmyna et al. (Kosmyna et al. 2022) evaluated low-channel EEG/EOG glasses and reported target-speaker detection performance comparable to a research-grade EEG headset. Xu et al. (Xu et al. 2026) further demonstrated that EOG-derived eyeblink sequences contain attention-related information and improve short-window AAD when combined with EEG.

However, existing work remains constrained by indirect eye-movement supervision and simplified EOG modeling. Consequently, the complementary relationship between EEG and EOG signals remains insufficiently modeled. These limitations motivate a unified multimodal AAD framework that explicitly captures modality-specific information and effective cross-modal interactions.

As illustrated in Figure 1, RAMamba-Net comprises a backbone network and the RAM optimization module. The backbone network contains four components: an EEG encoder, an EOG encoder, a cross-modal fusion module, and classifiers. The two encoders first transform EEG and EOG signals into modality-specific feature representations. The fusion module promotes bidirectional interaction between the two modalities. The final fused feature vectors from each modality are concatenated and fed into the multimodal, unimodal, and band classifiers to identify the attended speaker. This backbone provides the basis for the reliability-aware learning strategy.

The backbone enables bidirectional EEG-EOG interaction but does not explicitly account for sample-wise variations in modality quality. RAM therefore infers the relative reliability of EEG and EOG from modality-specific representations that preserve their distinct signal characteristics, and adaptively regulates their contributions prior to classification. For each modality, the token sequence is mean-pooled into a sample-level descriptor for reliability estimation:

Mean pooling summarizes the modality-specific token sequence into a fixed-dimensional representation, while stop-gradient decouples reliability estimation from encoder optimization, preventing the RAM objective from directly reshaping the modality representations.

RAMamba-Net is optimized with a composite objective consisting of a multimodal classification loss, a unimodal supervision loss, and a band-wise auxiliary loss. The multimodal classification loss supervises the final prediction obtained after cross-modal interaction and reliability-aware fusion:

where $\mathbf{y}$ denotes the ground-truth label, and $\mathrm{CE}(\cdot,\cdot)$ denotes the cross-entropy loss. To preserve the discriminative capability of each modality-specific encoder, the unimodal loss directly supervises the auxiliary predictions from the EEG and EOG branches:

The band-wise auxiliary loss further supervises each EEG frequency band through the independent lightweight classifier:

where $\mathcal{F}_{k}(\cdot)$ denotes the classifier for the $k$-th band. The overall training objective is formulated as:

where $\lambda_{uni}$ controls the contribution of unimodal supervision. More specific details about the model structure of RAMamba-Net are provided in the supplementary material.

We evaluate the proposed RAMamba-Net on two commonly used public AAD datasets.

To keep each continuous recording segment within an independent subset, each subject’s signals were chronologically partitioned before windowing. The first 90% formed the training pool, while the remaining 10% were reserved for testing. A validation set equal in size to the test set was then sampled from the training pool, and the remaining data were used for training. After data partition, each subset was independently segmented into 2-second decision windows. This resulted in an approximately 8:1:1 split across training, validation, and testing sets.

Common spatial pattern (Ang et al. 2008) was applied separately to EEG and EOG data to extract modality-specific spatial patterns. The filters were estimated from the training pool and applied to the validation and test sets. Given the Nyquist frequency after downsampling, AVGC and DTU were divided into 7 and 8 frequency bands, respectively. To comprehensively evaluate decoding performance, we report classification accuracy, balanced accuracy (BCA), Macro-F1, and Cohen’s Kappa, covering overall discrimination, class-wise balance, and corrected agreement.

All models were implemented in PyTorch and trained on a single NVIDIA GeForce RTX 3090 GPU. Each experiment was repeated with the seed list $\{41,42,43,44,45\}$. We used Adam with an initial learning rate of $5\times 10^{-4}$ and a weight decay of $3\times 10^{-4}$. The batch size was set to 8 for AVGC and 16 for DTU. Each model was trained for up to 200 epochs, using early stopping with a patience of 10. The trade-off parameter $\mathcal{L}_{uni}$ was set to 0.8 for both datasets. The reliability module coefficients $\beta$ and $\omega$ were set to 0.1 and 0.5, respectively, and the number of Transformer layers and attention heads $H_{c}$ was set to 2.

Before studying EEG-EOG fusion, we first evaluated each modality separately to explore their unimodal discriminative ability. We compared three categories of representative deep AAD models. CNN-based models include EEGNet (Lawhern et al. 2018), SCNN (Schirrmeister et al. 2017), and IFNet (Wang et al. 2023). CNN-Transformer models include CTNet (Zhao et al. 2024), TMSA-Net (Zhao and Zhu 2025), EEGConformer (Song et al. 2022), MSCFormer (Zhao et al. 2025), MSVTNet (Liu et al. 2024), and DBConformer (Wang et al. 2026b). AAD-specific models include DBPNet (Ni et al. 2024), DARNet (Yan et al. 2024), DHGCN (Zhou et al. 2025), and FAConformer (Wang et al. 2026a). In particular, DBPNet is not included in the EOG-only comparison, as it is specifically designed for 64-channel EEG with frequency band branches.

As reported in Table 1, EEG consistently outperformed EOG across datasets and models, confirming its dominant role in AAD, while EOG remained clearly above chance level, indicating complementary attention-related information. FAConformer and DBConformer achieved the best EEG and EOG decoding performance, respectively, suggesting that frequency-aware temporal-spectral modeling better suits EEG, whereas temporal-spatial dual-branch modeling is more effective for EOG.

Overall, EEG provides stronger neural evidence, while EOG captures complementary ocular dynamics. Their distinct characteristics motivate modality-specific rather than shared encoders.

Table 2 compares RAMamba-Net with four unimodal and multimodal baselines, using DBConformer and FAConformer as the EOG-only and EEG-only backbones, respectively. Concat. directly combines the two modality-specific features, while RAMamba-Net w/o RAM retains cross-modal attention but removes the reliability-aware module.

The results showed that EOG provides useful attention-related information, and EEG serves as a strong unimodal reference. Direct concatenation yielded only limited gains, indicating that simple feature fusion cannot fully exploit cross-modal complementary information. In contrast, explicit cross-modal interaction produced more consistent improvements by facilitating information exchange between EEG and EOG.

RAMamba-Net achieved the best results across all metrics and both datasets, with larger gains on AVGC. These results showed that cross-modal interaction and reliability-aware modulation jointly improve the accuracy, balance, and robustness of AAD by adaptively regulating modality contributions. Paired $t$-tests with adjusted $p$-values are provided in the supplementary material to verify statistical significance.

To further examine how multimodal fusion affects representation learning, we visualized the features before classification using $t$-SNE (Van der Maaten and Hinton 2008). We compared EEG-only, Concat., and RAMamba-Net to cover three representative stages: unimodal decoding, direct multimodal fusion, and reliability-aware multimodal fusion. This comparison can assess whether the proposed interaction and reliability mechanisms yield more discriminative feature distributions. The visualization results on AVGC and DTU are shown in Figure 2.

The EEG-only features exhibited limited separability, with attention samples still mixed in the embedding space. Direct concatenation brings EOG information into the representation, but the feature distributions remained insufficiently organized, indicating weak cross-modal alignment.

RAMamba-Net yielded the clearest class structure on both datasets, with learned features forming more compact intra-class clusters and larger inter-class margins. The improvement was more evident on DTU, where the two attention classes are separated into well-structured regions. This suggests that RAMamba-Net produces more discriminative and robust feature representations than unimodal decoding and direct concatenation, effectively enhancing multimodal representation learning.

To evaluate robustness to modality degradation, we compared Concat. and RAMamba-Net under different EEG and EOG noise levels. The noise level of each modality was varied independently from 0 to 1. A value of 0 denotes a clean signal, while 1 indicates severe corruption with little clean information retained. The first two heatmaps reported classification accuracy under each noise combination. The last heatmap showed the accuracy gain. The results on AVGC and DTU are presented in Figure 3.

With direct feature concatenation, accuracy declined mainly as EEG noise increased. The influence of EOG noise was less pronounced on both datasets. This suggests that the fusion process remains strongly dominated by EEG and is less robust when the primary modality is degraded. In contrast, RAMamba-Net maintained higher accuracy across nearly all noise conditions. The gain maps remained largely positive, even when both modalities were corrupted. This demonstrates that the proposed framework improves not only clean-signal performance but also robustness to changing signal quality. Bidirectional interaction preserves complementary cross-modal information, while RAM suppresses unreliable inputs and adaptively rebalances contributions from each modality. Together, these mechanisms enable more stable decoding under both modality-specific and joint noise perturbations.