Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Menghua Jiang Affiliation: School of Computer Science and Engineering, Sun Yat-sen University Email: jiangmh26@mail2.sysu.edu.cn Haokai Gao Affiliation: School of Computer Science, South China Normal University Xiangui Kang ^†^†thanks: Corresponding authors. Affiliation: School of Computer Science and Engineering, Sun Yat-sen University Haifeng Hu Affiliation: School of Electronics and Information Technology, Sun Yat-sen University Sijie Mai¹¹footnotemark: 1 Affiliation: School of Computer Science, South China Normal University

Abstract

Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.

1 Introduction

Multimodal Sentiment Analysis (MSA) integrates text, audio, and vision to better capture complex human emotional states Zadeh et al. (2016). With advances in multimodal learning Fang et al. (2025); Li et al. (2024b); He et al. (2026), MSA models leverage complementary cross-modal information, narrowing the gap between human expression and machine understanding, and showing promise in applications such as mental health analysis Ye et al. (2025) and human–robot interaction Liu et al. (2017). However, practical deployment remains challenging: multimodal data are often incomplete, noisy, or uncertain due to imperfect collection Zhang et al. (2024), sensor noise Zhuang et al. (2026), and privacy constraints Jaiswal and Provost (2020), which significantly degrades model reliability and predictive performance.

To achieve optimal multimodal performance, existing studies are often conducted under idealized or isolated settings, either implicitly assuming that all modalities available during training remain fully accessible at inference time, or treating modality missingness and noise contamination as separate problems. Under the full-modality setting, some approaches focus on modeling cross-modal discrepancies. For example, MODS Yang et al. (2026) captures modality importance via dynamic dominant-modality selection, while DecAlign Qian et al. (2026) achieves cross-modal alignment through prototype-guided optimal transport and distribution matching. To handle noisy inputs, several methods adopt an information-theoretic perspective, where MIB Mai et al. (2023b) introduces an information bottleneck Tishby et al. (2000) to filter redundant information, while OMIB Wu et al. (2025a) imposes theoretical constraints on the regularization weights to enable more stable bottleneck representation learning. To address modality missingness, existing approaches typically rely on reconstruction or cross-sample information augmentation. For instance, CyIN Lin et al. (2025) reconstructs missing information via cross-modal cycle translation, while HME Zhuang et al. (2025b) enhances representations by retrieving semantically relevant information from other samples.

However, in real-world scenarios, modality missingness and noise are often unknown and dynamically evolving, posing significant challenges to methods developed under idealized or isolated settings Zhuang et al. (2026). Furthermore, many existing approaches improve robustness to incomplete inputs at the expense of performance in full-modality settings, making it difficult to achieve consistent performance across diverse conditions within a unified framework Lin et al. (2025). Finally, noisy data typically introduce aleatoric uncertainty Hu et al. (2025), yet most existing denoising strategies overlook this factor and fail to explicitly model modality reliability during the denoising process.

To address the above challenges, we propose a unified framework called Reliability-aware Cross-sample Enhancement (RCE), which learns robust multimodal representations under complex scenarios where noise corruption and modality missingness coexist. RCE consists of three key components: ❶ Adaptive Variational Information Bottleneck, which maps each modality into a von Mises–Fisher (vMF) distribution parameterized by directional and concentration variables, explicitly capturing modality-wise uncertainty and enabling quality-aware information compression; ❷ Reliability-aware Modality Enhancement, which mitigates information deficiency caused by missing or degraded modalities by retrieving semantically consistent and high-confidence neighbors from a large candidate pool to enrich and calibrate modality representations; and ❸ Hyper-Modality Generation and Multilevel Fusion, which models cross-modal interactions via hyper-modality representation learning, integrates enhanced modality features with global contextual information, and employs a multilevel reliability-aware fusion mechanism to adaptively assign weights based on modality confidence, enabling unified modeling from intra-modality to cross-modality interactions. Through these designs, RCE achieves consistent and robust performance across diverse scenarios, including full-modality, noisy, and missing-modality settings.

Our main contributions are summarized as follows:

•

We propose a unified framework, termed RCE, to address multimodal sentiment analysis under realistic scenarios encompassing full-modality, noisy, and missing-modality conditions, without requiring separate treatment of each condition.
•

We develop a reliability-aware paradigm that incorporates explicit uncertainty modeling, facilitating adaptive information compression, cross-sample information augmentation, and multi-level fusion for learning robust multimodal representations.
•

Extensive experiments on four benchmark datasets and three evaluation settings demonstrate that RCE consistently outperforms existing methods with stronger robustness.

2 Related Works

2.1 Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) aims to integrate text, audio, and visual signals to predict human emotional states Zadeh et al. (2016). Most existing methods assume that all modalities are available at inference time and focus on designing sophisticated fusion strategies to learn discriminative multimodal representations Zadeh et al. (2017); Zhang et al. (2023a); Wu et al. (2025b); Mai et al. (2025); Fang et al. (2025); Wang et al. (2025); Jiang et al. (2025b); Jiang et al. (2025a); Mai and Han (2026). Some studies focus on cross-modal alignment, such as DecAlign Qian et al. (2026), which decouples shared and modality-specific features and leverages prototype-guided optimal transport. Other methods model modality importance, such as MODS Yang et al. (2026), which enhances cross-modal interactions through dynamic primary modality selection. In addition, some approaches improve representation capacity by incorporating additional semantic information. For example, PSA-MF Xie et al. (2026) introduces personality-aware embeddings for fine-grained fusion. Although these methods perform well under full-modality settings, their performance degrades significantly in real-world scenarios with missing modalities, thereby limiting their practical applicability.

2.2 Multimodal Sentiment Analysis with Missing Modalities

To address missing modalities, existing MSA methods attempt to compensate for incomplete data from multiple perspectives. One line of work focuses on reconstructing missing modalities from observed ones Pham et al. (2019); Zhao et al. (2021); Wang et al. (2023a); Lian et al. (2023); Zhang et al. (2024). For example, IMDer Wang et al. (2023b) employs a diffusion-based model for reconstruction, while CyIN Lin et al. (2025) restores missing information in the latent space via cross-modal cyclic translation. Another line of work adopts a teacher–student framework Zhuang et al. (2025a). In this paradigm, methods such as HRLF Li et al. (2024a) leverage a teacher model trained on fully observed multimodal data to guide student models in handling missing modalities through knowledge distillation. In addition, some approaches enhance representations using cross-sample information. CorrKD Li et al. (2024b) proposes a sample-level contrastive distillation mechanism to capture global cross-sample correlations, whereas HME Zhuang et al. (2025b) enriches observed modalities by retrieving semantically relevant clues from other samples, thereby avoiding explicit reconstruction.

Despite their effectiveness, these methods typically assume noise-free data, which limits their applicability in real-world scenarios with substantial noise. Moreover, HME Zhuang et al. (2025b) and CorrKD Li et al. (2024b) rely on samples within the current mini-batch, leading to unstable and limited neighborhood information. Semantically similar samples with opposite sentiment polarity may further introduce “negative enhancement,” degrading model performance. In contrast, RCE retrieves semantically consistent and high-confidence neighbor representations from a large-scale candidate pool, resulting in more robust multimodal representations. We also provide a review of related work on multimodal sentiment analysis with noisy modalities. Please refer to Appendix B.1.

Refer to caption — Figure 1: Overview of the RCE framework. For clarity, we provide detailed descriptions and definitions of all notations used in the figure in Appendix A.

3 Methodology

The overall architecture of the proposed RCE framework is shown in Figure 1. It comprises four modules. The Pre-Processing module handles multimodal inputs and extracts modality-specific representations (see Section 3.2). The Adaptive Variational Information Bottleneck (AVIB) module performs reliability-aware adaptive compression (see Section 3.3). The Reliability-aware Modality Enhancement (RME) module leverages reliable samples to enhance representations (see Section 3.4). Finally, the Hyper-Modality Generation and Multilevel Fusion module generates hyper-modality representations and fuses them with original representations via reliability-aware multilevel fusion (see Section 3.5).

3.1 Task Formulation

MSA aims to infer the speaker’s emotional state from video segments comprising textual ( $X_{T}$ ), acoustic ( $X_{A}$ ), and visual ( $X_{V}$ ) modalities. Each modality is represented as a sequence of features, i.e., $X_{m}\in\mathbb{R}^{L_{m}\times d_{m}}$ , where $m\in\{T,A,V\}$ indexes the modality. Here, $L_{m}$ and $d_{m}$ denote the sequence length and feature dimension of modality $m$ , respectively. Given a training dataset $\mathcal{D}=\{(X_{T}^{i},X_{A}^{i},X_{V}^{i},y^{i})\}_{i=1}^{N}$ , the objective is to learn a model $\mathcal{M}_{\phi}$ that predicts $\hat{y}^{i}=\mathcal{M}_{\phi}(X_{T}^{i},X_{A}^{i},X_{V}^{i})$ , where $\phi$ denotes the learnable parameters. Depending on the task setting, the prediction $\hat{y}^{i}$ can be either a continuous sentiment score (regression) or a discrete emotion category (classification).

3.2 Pre-Processing

To map heterogeneous multimodal input features into a unified representation space, following prior work Tsai et al. (2019); Li et al. (2024b); Zhuang et al. (2025b), we employ one-dimensional convolutions to project features from each modality, yielding $\hat{X}_{m}^{i}=W_{1D}^{m}(X_{m}^{i})$ , where $W_{1D}^{m}$ denotes the modality-specific one-dimensional convolutional projection layer and $\hat{X}_{m}^{i}\in\mathbb{R}^{L_{m}\times d}$ , where $d$ is the shared embedding dimension across modalities. Subsequently, the projected feature sequences are fed into a Transformer Vaswani et al. (2017) encoder to model intra-modal temporal dependencies. Positional encoding is added to preserve sequential order information, yielding modality-specific representations $S_{m}^{i}=F_{\phi_{m}}(\hat{X}_{m}^{i}+PE(L_{m},d))$ , where $F_{\phi_{m}}$ denotes the modality-specific Transformer encoder with parameters $\phi_{m}$ , and $PE(L_{m},d)$ denotes positional encoding of length $L_{m}$ and dimension $d$ . Finally, global max pooling is applied to obtain the initial global feature representation of each modality, i.e., $H_{m}^{i}=GMP(S_{m}^{i})$ , where $H_{m}^{i}\in\mathbb{R}^{d}$ and $GMP(\cdot)$ denotes the global max pooling operation.

3.3 Adaptive Variational Information Bottleneck

To achieve adaptive compression of information conditioned on modality quality, we propose the AVIB module, consisting of two key components: vMF-based uncertainty estimation and quality-aware adaptive compression. We describe each component in detail below.

vMF-based Uncertainty Estimation.

To explicitly model sample-level reliability across modalities, we map each modality’s feature representation to the von Mises–Fisher (vMF) distribution on the unit hypersphere. For the $i$ -th sample and modality $m$ , $H_{m}^{i}$ parameterizes a posterior distribution over $z$ , i.e., $z_{m}^{i}\sim\mathrm{vMF}(\mu_{m}^{i},\kappa_{m}^{i})$ , where $\mu_{m}^{i}$ and $\kappa_{m}^{i}$ denote the mean direction and concentration parameter, respectively. Specifically, we define:

\mu_{m}^{i}=\frac{f_{m}(H_{m}^{i})}{\|f_{m}(H_{m}^{i})\|_{2}}\in\mathbb{S}^{d-1},\quad\kappa_{m}^{i}=\mathrm{softplus}(g_{m}(H_{m}^{i}))+\epsilon\in\mathbb{R}^{+},

(1)

where $f_{m}(\cdot)$ and $g_{m}(\cdot)$ are two separate subnetworks used to estimate the directional and concentration parameters, respectively, and $\epsilon$ is a small constant for numerical stability. Under this formulation, $\mu_{m}^{i}$ represents the principal direction of the latent representation on the unit hypersphere, while $\kappa_{m}^{i}$ characterizes the degree of concentration of the distribution around this direction. A larger $\kappa_{m}^{i}$ corresponds to a more concentrated posterior distribution. Following prior work Ming et al. (2023); Du et al. (2024); Hu et al. (2025), we interpret it as a proxy for sample-level reliability of the corresponding modality, reflecting how trustworthy the modality is for prediction. Additional details are provided in Appendix C.1.

Quality-aware Adaptive Compression.

Building on the vMF representation, we model directional information and sample-wise posterior uncertainty within a unified hyperspherical space, enabling modality-quality-aware adaptive compression.

We define the variational posterior as $q(z_{m}^{i}\mid H_{m}^{i})=\mathrm{vMF}(\mu_{m}^{i},\kappa_{m}^{i})$ , where $\mu_{m}^{i}$ and $\kappa_{m}^{i}$ are estimated from modality-specific features $H_{m}^{i}$ . Our goal is to learn a compressed latent representation $z_{m}^{i}$ that preserves task-relevant information while discarding redundant and noisy components in the original representation. The corresponding information bottleneck objective is:

\mathcal{L}_{\text{VIB}}^{m}=\beta I(H_{m}^{i};z_{m}^{i})-I(z_{m}^{i};Y),

(2)

where $\beta$ is a hyperparameter controlling the trade-off between compression and predictive power.

Since the mutual information $I(H_{m}^{i};z_{m}^{i})$ is intractable, we use variational inference to approximate it. In the absence of additional prior knowledge and to preserve directional isotropy, we adopt a uniform distribution on the unit hypersphere as the prior: $p(z_{m}^{i})=\mathrm{Uniform}(\mathbb{S}^{d-1})$ . Accordingly, the original objective can be reformulated as the following tractable variational objective:

\mathcal{L}_{\text{AVIB}}^{m}=\beta\cdot\mathrm{KL}\Big(q(z_{m}^{i}\mid H_{m}^{i})\;\|\;p(z_{m}^{i})\Big)-\mathbb{E}_{z_{m}^{i}\sim q}\big[\log p(Y\mid z_{m}^{i})\big],

(3)

where the expectation term $\mathbb{E}_{z_{m}^{i}\sim q}\big[\log p(Y\mid z_{m}^{i})\big]$ can be implemented via the task loss $\mathcal{L}_{\text{task}}(\hat{y}_{m},y)$ .

To enable differentiable sampling on the unit hypersphere, we adopt an approximate sampling strategy based on direction–orthogonal decomposition. Given vMF parameters $(\mu,\kappa)$ , the latent variable is constructed as:

z=w\mu+\sqrt{1-w^{2}}\,v,

(4)

where $w\in[-1,1]$ is a scalar random variable governed by $\kappa$ , and $v\sim\mathrm{Uniform}(\mathbb{S}^{d-2})$ is a unit vector sampled from the subspace orthogonal to $\mu$ . This construction ensures that $z$ lies on the unit hypersphere $\mathbb{S}^{d-1}$ . Under this formulation, the KL divergence admits a closed-form expression:

\mathrm{KL}(\mathrm{vMF}(\mu,\kappa)\;\|\;\mathrm{Uniform})=\kappa\frac{I_{d/2}(\kappa)}{I_{d/2-1}(\kappa)}+\log C_{d}(\kappa)-\log C_{d}(0),

(5)

when $\kappa=0$ , the vMF distribution reduces to the uniform distribution on the unit hypersphere, with normalization constant $C_{d}(0)=\frac{\Gamma(d/2)}{2\pi^{d/2}}$ .

During training, we introduce auxiliary prediction branches based on modality-specific latent samples and jointly optimize the task loss with the KL regularization. This mechanism assigns a higher information cost to highly concentrated posteriors, encouraging the model to retain them only when the modality provides discriminative information. Conversely, low-quality or noisy modalities are driven toward more dispersed posteriors approaching the uniform prior, reducing their impact on downstream predictions. In this way, AVIB achieves adaptive information compression conditioned on modality quality.

3.4 Reliability-aware Modality Enhancement

Inspired by HME Zhuang et al. (2025b), we enrich each modality by retrieving semantically relevant cues from other samples. However, HME is prone to “negative enhancement” and relies solely on mini-batch samples, leading to insufficient and unstable neighborhood information. To address these limitations, we propose the RME module. Unlike methods that rely only on mini-batch samples, RME maintains a modality-specific memory bank of historical samples, enabling the retrieval of more informative and semantically similar neighbors from a larger candidate pool.

Specifically, for modality $m$ , let $z_{m}^{i}\in\mathbb{R}^{d}$ denote the latent representation of the $i$ -th sample in the current mini-batch obtained from the AVIB module, with corresponding vMF mean direction $\mu_{m}^{i}$ and concentration parameter $\kappa_{m}^{i}$ . The batch-level representations are denoted as $Z_{m}^{(b)}=\{z_{m}^{1},\dots,z_{m}^{B}\}$ , $M_{m}^{(b)}=\{\mu_{m}^{1},\dots,\mu_{m}^{B}\}$ , and $K_{m}^{(b)}=\{\kappa_{m}^{1},\dots,\kappa_{m}^{B}\}$ , where $B$ is the batch size. We maintain a modality-specific memory bank with capacity $N_{b}$ : $\mathcal{B}_{m}=\{(z_{m}^{r},\mu_{m}^{r},\kappa_{m}^{r},y^{r})\}_{r=1}^{N_{m}}$ with $N_{m}\leq N_{b}$ , where $y^{r}$ is the label. For a given sample $i$ , its candidate neighbor set is defined as:

\mathcal{C}_{i}^{m}=\begin{cases}\mathcal{B}_{m},&\text{if }N_{m}=N_{b},\\ \{(z_{m}^{j},\mu_{m}^{j},\kappa_{m}^{j},y^{j})\mid j\neq i\},&otherwise.\end{cases}

(6)

In practice, the memory bank is rebuilt before each training epoch by a forward pass over the training set without gradient updates. The model retrieves neighbors from the memory bank only when it is fully populated, providing a broader and more stable candidate space; otherwise, it falls back to intra-batch retrieval. This design avoids unreliable neighbors in early training while enabling more effective cross-sample complementarity once sufficient historical samples are accumulated.

Notably, to avoid recency bias and ensure balanced retention under limited capacity, we use reservoir sampling Vitter (1985) for memory updates, such that after observing $t$ samples, each sample is retained with probability $\min(1,N_{b}/t)$ . More details are provided in Appendix C.2.

To improve the stability of neighbor retrieval, we do not compute similarity directly on the sampled representations $z_{m}^{i}$ , but instead perform matching in a more stable reference direction space. Specifically, for a candidate sample $j$ in $\mathcal{C}_{i}^{m}$ , the similarity is defined as $S_{m}(i,j)={\mu_{m}^{i}}^{\top}\mu_{m}^{j}$ , where $\mu_{m}^{i},\mu_{m}^{j}$ are unit vectors on the hypersphere. Thus, this is equivalent to cosine similarity Li et al. (2024b). This design effectively reduces fluctuations caused by sampling noise, making the similarity measure more robust in reflecting semantic consistency between samples. Based on this, we select only the top- $k$ most similar neighbors to form the neighborhood set $\mathcal{N}_{i}^{m}$ , i.e.,

\mathcal{N}_{i}^{m}=\text{Top-}k(\mathcal{C}_{i}^{m}),

(7)

where $k=\lfloor\rho|\mathcal{C}_{i}^{m}|\rfloor$ , and $\rho\in(0,1]$ is a proportion hyperparameter. This design helps filter out weakly related candidates and reduces the introduction of irrelevant noise.

To improve the reliability of neighbor aggregation, we incorporate both label distance and confidence constraints into the similarity weighting. For any neighbor $j\in\mathcal{N}_{i}^{m}$ , its weight is defined as

e_{ij}^{(m)}=S_{m}(i,j)-\alpha|y_{i}-y_{j}|+\log(1+\kappa_{m}^{j}),

(8)

where $y_{i}$ and $y_{j}$ are the labels of the current sample and its neighbor, respectively, and $\alpha$ controls the penalty for label discrepancy. The similarity term captures semantic proximity, the label distance penalizes mismatched labels, and $\log(1+\kappa_{m}^{j})$ favors high-confidence neighbors. The weights are normalized to obtain aggregation coefficients, and the neighbor-enhanced representation is given by

\tilde{z}_{m}^{i}=\sum_{j\in\mathcal{N}_{i}^{m}}w_{ij}^{(m)}z_{m}^{j},\quad w_{ij}^{(m)}=\frac{\exp(e_{ij}^{(m)})}{\sum_{k\in\mathcal{N}_{i}^{m}}\exp(e_{ik}^{(m)})}.

(9)

If no valid neighbors are available, we set $\tilde{z}_{m}^{i}=z_{m}^{i}$ . This mechanism promotes aggregation from semantically similar, label-consistent, and high-confidence samples, leading to more robust modality enhancement.

Notably, the RME module is used only during training and is not required at inference time. It can also be applied to incomplete data by retrieving semantically relevant cues from available samples, avoiding explicit modality reconstruction and remaining effective under missing modalities.

3.5 Hyper-Modality Generation and Multilevel Fusion

Hyper-Modality Representation Generation.

Next, following HME Zhuang et al. (2025b), we employ a Perceiver-style latent Transformer Zhang et al. (2023a); Zhang et al. (2024) to encode the modality-enhanced representations from the RME module, aiming to obtain compact and context-aware modality-level representations. Specifically, for each modality $m$ , we introduce a set of learnable latent prompts $E_{m}^{\prime}\in\mathbb{R}^{l_{p}\times d}$ , where $l_{p}$ is the prompt length and $d$ is the feature dimension. Taking $E_{m}^{\prime}$ as queries and the enhanced representation $\tilde{z}_{m}^{i}$ as keys and values, we apply cross-attention:

\bar{E}_{m}^{i}=ATTN_{\psi_{m}}(E_{m}^{\prime},\tilde{z}_{m}^{i})=\text{Softmax}\left(\frac{E_{m}^{\prime}W_{m}^{Q}(\tilde{z}_{m}^{i}W_{m}^{K})^{\top}}{\sqrt{d}}\right)\tilde{z}_{m}^{i}W_{m}^{V},

(10)

where $ATTN_{\psi_{m}}(\cdot)$ denotes the cross-attention function parameterized by $\psi_{m}$ , and $W_{m}^{Q},W_{m}^{K},W_{m}^{V}\in\mathbb{R}^{d\times d}$ are learnable projection matrices. After several standard Transformer blocks, we apply global average pooling over the latent prompt dimension to obtain the modality-level representation: $E_{m}^{i}=GAP(\bar{E}_{m}^{i})$ , where $E_{m}^{i}\in\mathbb{R}^{d}$ and $GAP(\cdot)$ denotes the global average pooling operation.

Furthermore, to model shared multimodal context, we introduce a shared Perceiver-style Latent Transformer. Unlike the modality-specific branch, this branch takes the original unimodal representations of the current sample as input. Given $H_{T}^{i},H_{A}^{i},H_{V}^{i}$ , we construct $H_{S}^{i}=[H_{T}^{i};H_{A}^{i};H_{V}^{i}]\in\mathbb{R}^{3\times d}$ and apply the same latent cross-attention mechanism to extract global contextual information, yielding the shared representation: $E_{S}^{i}=\text{LatentTransformer}_{S}(H_{S}^{i})$ , $E_{S}^{i}\in\mathbb{R}^{d}$ .

Subsequently, we leverage cross-modal complementarity to construct hyper-modality representations. For any $m,m_{1},m_{2}\in\{T,A,V\}$ with $m\neq m_{1}$ , $m\neq m_{2}$ , and $m_{1}\neq m_{2}$ , we first model inter-modal interactions via cross-attention:

C_{m\to m_{1}}^{i}=ATTN_{\psi_{m,m_{1}}}(E_{m}^{i},E_{m_{1}}^{i}),\quad C_{m\to m_{2}}^{i}=ATTN_{\psi_{m,m_{2}}}(E_{m}^{i},E_{m_{2}}^{i}).

(11)

Then, using the shared representation $E_{S}^{i}$ as the query and the interaction features from other modalities as the keys and values, we derive the hyper-modality representation:

R_{m}^{i}=ATTN_{\psi_{S,m}}(E_{S}^{i},[C_{m\to m_{1}}^{i};C_{m\to m_{2}}^{i}]),

(12)

where $R_{m}^{i}\in\mathbb{R}^{d}$ . This allows each modality to absorb complementary information from others while incorporating global shared context, leading to more robust cross-modal representations.

Reliability-aware Multilevel Fusion.

To integrate multi-granularity information across stages, we design a hierarchical aggregation mechanism from intra-modality fusion to cross-modality weighting.

For each modality $m\in\{T,A,V\}$ , we first fuse the original unimodal representation $H_{m}^{i}$ with the hyper-modality representation $R_{m}^{i}$ under shared context. Specifically, we concatenate them and pass them through a lightweight attention network to obtain fusion logits: $a_{m}^{i}=F_{m}([H_{m}^{i}\|R_{m}^{i}])$ , where $F_{m}(\cdot)$ denotes a lightweight mapping function composed of a linear layer, Dropout, and ReLU. Since the concentration parameter $\kappa_{m}^{i}$ in the vMF distribution reflects representation uncertainty, we use it to adaptively modulate the fusion weights, yielding the intra-modality fused representation:

u_{m}^{i}=\beta_{m,1}^{i}H_{m}^{i}+\beta_{m,2}^{i}R_{m}^{i},\quad\beta_{m}^{i}=\text{Softmax}\left(\frac{a_{m}^{i}\odot[1,\kappa_{m}^{i}]}{\sqrt{d}}\right),

(13)

where $\odot$ denotes element-wise multiplication and $d$ is the feature dimension. This allows adaptive balancing between unimodal and enhanced representations based on modality reliability.

To obtain a global representation, we further perform reliability-aware fusion over $R_{T}^{i},R_{A}^{i},R_{V}^{i}$ . We compute fusion logits via concatenation: $a_{H}^{i}=F_{H}([R_{T}^{i}\|R_{A}^{i}\|R_{V}^{i}])\in\mathbb{R}^{3}$ . The modality reliability weights are defined as $\gamma^{i}=\text{Softmax}([\kappa_{T}^{i},\kappa_{A}^{i},\kappa_{V}^{i}])$ . The logits are then modulated and normalized: $\eta^{i}=\text{Softmax}\left(\frac{a_{H}^{i}\odot\gamma^{i}}{\sqrt{d}}\right)$ . The resulting global representation is

u_{H}^{i}=\eta_{T}^{i}R_{T}^{i}+\eta_{A}^{i}R_{A}^{i}+\eta_{V}^{i}R_{V}^{i}.

(14)

Finally, we form four tokens by combining the three intra-modality representations and the global representation: $U^{i}=[u_{T}^{i};u_{A}^{i};u_{V}^{i};u_{H}^{i}]\in\mathbb{R}^{4\times d}$ , which are fed into a two-layer Transformer encoder to model cross-level interactions: $\hat{U}^{i}=\text{TransformerEncoder}(U^{i})$ . The output tokens are concatenated to form the final joint representation: $r^{i}=[\hat{u}_{T}^{i}\|\hat{u}_{A}^{i}\|\hat{u}_{V}^{i}\|\hat{u}_{H}^{i}]\in\mathbb{R}^{4d}$ . The final representation $r^{i}$ is then passed to a two-layer MLP to produce the prediction: $\hat{y}^{i}=\text{MLP}(r^{i})$ .

Overall Objective.

To jointly optimize task performance and latent representation compression, we define the overall objective as

\mathcal{L}_{all}=\mathcal{L}_{task}(\hat{y},y)+\lambda\cdot\mathcal{L}_{\text{AVIB}},

(15)

where $\mathcal{L}_{\text{AVIB}}=\frac{1}{3}\sum_{m\in\{T,A,V\}}\mathcal{L}_{\text{AVIB}}^{m}$ , and $\lambda$ is a hyperparameter that balances the task loss and the information bottleneck regularization. For classification and regression tasks, $\mathcal{L}_{task}$ is defined as

\mathcal{L}_{task}=\begin{cases}CrossEntropy(\hat{y},y),&classification\\ |\hat{y}-y|,&regression\end{cases}.

(16)

4 Experiments

4.1 Experimental Setup

We evaluate RCE on MSA under full, noisy, and missing-modality settings, where both noisy and missing modalities are evaluated under two protocols. To further assess generalization, we also evaluate it on multimodal humor detection (MHD) and multimodal sarcasm detection (MSD) tasks.

Datasets. For the MSA task, we conduct experiments on two widely used datasets, CMU-MOSI Zadeh et al. (2016) and CMU-MOSEI Zadeh et al. (2018b). For the MHD and MSD tasks, we adopt the UR-FUNNY Hasan et al. (2019) and MUStARD Castro et al. (2019) datasets, respectively. Details are provided in Appendix D.1.

Baselines. We compare RCE with several state-of-the-art methods, including recent full-modality approaches such as DecAlign Qian et al. (2026) [ICLR’26], MODS Yang et al. (2026) [AAAI’26], and PSA-MF Xie et al. (2026) [AAAI’26]. In addition, we focus on two representative categories of methods that are most relevant to RCE. The first category consists of information bottleneck-based approaches, including C-MIB Mai et al. (2023b) [TMM’23], KAN-MCP Luo et al. (2025) [ACM MM’25], OMIB Wu et al. (2025a) [ICML’25], and CyIN Lin et al. (2025) [NeurIPS’25]. The second category includes uncertainty-aware fusion methods such as QMF Zhang et al. (2023b) [ICML’23], PML Hu et al. (2025) [IJCAI’25], and HME Zhuang et al. (2025b) [NeurIPS’25]. Detailed descriptions of these methods are provided in Appendix D.2.

Due to space limitations, details on feature extraction, implementation, and evaluation metrics are provided in Appendix D.3, Appendix D.4, and Appendix D.5, respectively. In addition, extensive additional experimental results are provided in Appendix E for a more comprehensive analysis.

4.2 Quantitative Results

Table 1: Comparison on CMU-MOSI and CMU-MOSEI. Best and Second results are highlighted.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
MODS Yang et al. (2026)	49.27	85.83	85.96	0.688	0.798	54.32	85.88	86.14	0.527	0.772
PSA-MF Xie et al. (2026)	46.50	86.43	86.19	0.686	0.807	55.00	86.30	86.28	0.521	0.774
DecAlign Qian et al. (2026)	45.07	85.75	85.82	0.735	0.811	55.02	86.48	86.07	0.543	0.768
C-MIB Mai et al. (2023b)	47.01	87.33	87.28	0.650	0.836	52.47	86.08	86.05	0.551	0.773
ITHP Xiao et al. (2024)	46.57	88.40	88.40	0.650	0.848	52.44	86.80	86.83	0.529	0.789
KAN-MCP Luo et al. (2025)	46.86	88.85	88.83	0.631	0.853	53.67	87.18	87.20	0.523	0.787
OMIB Wu et al. (2025a)	47.59	88.85	88.79	0.636	0.844	53.54	87.29	87.27	0.527	0.786
QMF Zhang et al. (2023b)	47.59	88.40	88.35	0.634	0.846	52.34	87.35	87.21	0.530	0.789
PML Hu et al. (2025)	47.01	88.09	88.03	0.658	0.837	53.91	87.07	87.04	0.527	0.786
HME Zhuang et al. (2025b)	48.32	88.09	88.11	0.620	0.856	54.25	86.82	86.89	0.513	0.792
RCE (Ours)	50.22	89.31	89.26	0.589	0.861	55.44	87.29	87.33	0.503	0.799

Full-modality Setting. The results under the fully multimodal setting are reported in Table 1. This setting is adopted by most existing MSA methods. It can be observed that, compared with state-of-the-art approaches such as DecAlign Qian et al. (2026) and MODS Yang et al. (2026), RCE achieves the best performance on nearly all metrics. In particular, RCE significantly outperforms existing methods on Acc7 and MAE, highlighting its capability for fine-grained sentiment prediction. This advantage stems from its reliability-aware compression, cross-sample enhancement, and multilevel fusion, which collectively facilitate the learning of robust and discriminative multimodal representations.

Missing-modality Setting. We compare RCE with state-of-the-art methods specifically designed for missing modality scenarios, including HME Zhuang et al. (2025b) and CyIN Lin et al. (2025), under the random missing protocol. The results (see Figure 2) show that RCE consistently outperforms these methods across most missing rates. Notably, even without modality-specific tuning, RCE demonstrates strong generalization ability. We further provide comparisons under fixed missing protocols against a variety of advanced methods, with results reported in Appendix E.1. Overall, RCE significantly enhances robustness to diverse missing patterns by leveraging informative cues from other reliable and available samples.

Figure 2: Comparison on CMU-MOSEI under the random missing protocol.

MHD and MSD Tasks. To further evaluate the generalization capability of RCE on other multimodal tasks, we conduct experiments on the MHD and MSD tasks using the UR-FUNNY Hasan et al. (2019) and MUStARD Castro et al. (2019) datasets. As shown in Figure 4, RCE outperforms the best-performing baselines, AtCAF Huang et al. (2025) and MOAC Mai et al. (2025), by over 3.2% and 1.5% in accuracy, respectively, achieving state-of-the-art performance. These results further demonstrate the strong generalization ability of RCE across different multimodal tasks.

4.3 Ablation Study

To better understand the contribution of each component in RCE, we conduct an ablation study on several key modules of the proposed framework: (i) ‘w/o AVIB’ removes the AVIB module and directly uses the modality representations without adaptive compression; (ii) ‘w/o RME’ removes the RME module; (iii) ‘w/o MB’ disables the memory bank and only uses mini-batch samples for neighbor retrieval; (iv) ‘w/o CP’ removes the confidence promotion term in neighbor weighting; (v) ‘w/o HMG’ removes the hyper-modality generation component; and (vi) ‘w/o RF’ removes reliability-aware fusion and replaces it with standard attention weighting. The results are reported in Table 2. Full results and additional variant analyses are provided in Appendix E.3.

Effects of the RCE Components. Overall, removing most components degrades performance on both CMU-MOSI and CMU-MOSEI, demonstrating the effectiveness of the proposed modules. Among them, ‘w/o HMG’ causes the largest drop on CMU-MOSI, with Acc7 decreasing from 50.22 to 46.72 and MAE increasing from 0.589 to 0.637, highlighting the importance of hyper-modality integration. Removing AVIB also consistently harms performance, indicating that vMF-based uncertainty estimation and adaptive compression effectively suppress noisy modality information. In addition, ‘w/o MB’ yields lower Acc7 on both datasets, suggesting that retrieving reliable neighbors from the memory bank provides more stable enhancement than using only mini-batch samples. Some components exhibit dataset- and metric-dependent effects. Specifically, ‘w/o CP’ causes only minor changes on CMU-MOSEI, while ‘w/o RF’ slightly improves Acc7 but worsens MAE, suggesting that reliability-aware fusion contributes more to prediction calibration than classification accuracy. Although ‘w/o RME’ slightly improves Acc7 on CMU-MOSI, it increases MAE and degrades performance on CMU-MOSEI, indicating that RME generally improves robustness despite minor fluctuations on smaller datasets. Overall, AVIB, MB, and HMG are the most influential components.

Figure 3: Results on UR-FUNNY and MUStARD.

Figure 4: Feature visualization.

Table 2: Ablation results of RCE components on CMU-MOSI and CMU-MOSEI. We report Acc7 and MAE. Full results are provided in Appendix E.3.

Dataset	RCE	w/o AVIB	w/o RME	w/o MB	w/o CP	w/o HMG	w/o RF
CMU-MOSI	50.22 / 0.589	48.03 / 0.646	50.36 / 0.610	48.03 / 0.616	47.30 / 0.610	46.72 / 0.637	48.47 / 0.626
CMU-MOSEI	55.44 / 0.503	54.60 / 0.518	54.10 / 0.519	53.50 / 0.511	55.14 / 0.504	54.77 / 0.506	55.52 / 0.507

4.4 Further Analysis

We provide additional analyses in the appendix, including parameter sensitivity analysis (Appendix E.5), testing stability analysis (Appendix E.6), model complexity analysis (Appendix E.7), and theoretical complexity analysis (Appendix E.8).

Feature Visualization. To investigate the differences between the original modality representations $H_{m}$ and the generated hyper-modality representations $R_{m}$ , we visualize them using t-SNE Van der Maaten and Hinton (2008). As shown in Figure 4, the hyper-modality representations form distinct clusters, indicating that RCE captures complementary sentiment-relevant information beyond unimodal features.

Table 3: Training negative enhancement rates on CMU-MOSI and CMU-MOSEI.

Dataset	Method	Text	Audio	Vision
CMU-MOSI	HME	46.29%	46.90%	46.90%
CMU-MOSI	RCE	2.60%	5.90%	6.48%
CMU-MOSEI	HME	38.40%	38.87%	38.87%
CMU-MOSEI	RCE	10.57%	31.25%	26.08%

Negative Enhancement Analysis. As shown in Table 3, we report the training-stage negative enhancement rates of HME Zhuang et al. (2025b) and RCE across text, audio, and vision modalities. Compared with HME, RCE consistently reduces negative enhancement cases across all modalities, showing that the proposed reliability-aware and label-aware enhancement strategy effectively suppresses sentiment-conflicting information from neighboring samples. On CMU-MOSI, the rates drop from around 46% to below 7%. On CMU-MOSEI, RCE also achieves clear reductions, especially in text, while the smaller improvement in audio may be due to the higher ambiguity of acoustic sentiment cues. Detailed results and analysis are provided in Appendix E.4.

5 Conclusion

In this paper, we propose Reliability-aware Cross-sample Enhancement (RCE), a unified framework for robust multimodal sentiment analysis under full, noisy, and missing-modality settings. RCE learns robust multimodal representations by estimating modality reliability with adaptive variational information bottleneck, retrieving reliable cross-sample cues for modality enhancement, and performing reliability-aware multilevel fusion for prediction.

Limitations. RCE introduces additional hyperparameters and a larger number of parameters. Although it is generally stable within moderate parameter ranges, dataset-specific tuning may still be needed. Future work will explore adaptive and lightweight enhancement strategies.

References

Castro et al. (2019) S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria Towards multimodal sarcasm detection (an _obviously_ perfect paper). In ACL, pp. 4619–4629. Cited by: §D.1, §4.1, §4.2.
Degottex et al. (2014) G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer COVAREP—a collaborative voice analysis repository for speech technologies. In ICASSP, pp. 960–964. Cited by: §D.3.
Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova Bert: pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171–4186. Cited by: §D.3, §D.3.
Du et al. (2024) C. Du, Y. Wang, S. Song, and G. Huang Probabilistic contrastive learning for long-tailed visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (9), pp. 5890–5904. Cited by: §C.1, §3.3.
Fang et al. (2025) Y. Fang, W. Huang, G. Wan, K. Su, and M. Ye EMOE: modality-specific enhanced dynamic emotion experts. In CVPR, pp. 14314–14324. Cited by: §1, §2.1.
Gao et al. (2024) Z. Gao, X. Jiang, X. Xu, F. Shen, Y. Li, and H. T. Shen Embracing unimodal aleatoric uncertainty for robust multimodal fusion. In CVPR, pp. 26876–26885. Cited by: §E.2.
Gong et al. (2024) P. Gong, J. Liu, X. Zhang, X. Li, L. Wei, and H. He Adaptive multimodal graph integration network for multimodal sentiment analysis. IEEE Transactions on Audio, Speech and Language Processing 33, pp. 23–36. Cited by: §B.1.
Han et al. (2021) W. Han, H. Chen, A. Gelbukh, A. Zadeh, L. Morency, and S. Poria Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis. In ICMI, pp. 6–15. Cited by: §D.2, §D.4.
Hasan et al. (2021) M. K. Hasan, S. Lee, W. Rahman, A. Zadeh, R. Mihalcea, L. Morency, and E. Hoque Humor knowledge enriched transformer for understanding multimodal humor. In AAAI, pp. 12972–12980. Cited by: §D.1, §D.3.
Hasan et al. (2019) M. K. Hasan, W. Rahman, A. B. Zadeh, J. Zhong, M. I. Tanveer, L. Morency, and M. E. Hoque UR-funny: a multimodal language dataset for understanding humor. In EMNLP-IJCNLP, pp. 2046–2056. Cited by: §D.1, §4.1, §4.2.
Hazarika et al. (2022) D. Hazarika, Y. Li, B. Cheng, S. Zhao, R. Zimmermann, and S. Poria Analyzing modality robustness in multimodal sentiment analysis. In ACL, pp. 685–696. Cited by: §B.1.
He et al. (2026) K. He, B. Chen, Y. Ding, F. Li, C. Teng, and D. Ji Pase: prototype-aligned calibration and shapley-based equilibrium for multimodal sentiment analysis. In AAAI, Vol. 40, pp. 30960–30968. Cited by: §1.
He et al. (2020) P. He, X. Liu, J. Gao, and W. Chen Deberta: decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654. Cited by: §D.3.
Hu et al. (2025) P. Hu, Y. Qin, Y. Gou, Y. Li, M. Yang, and X. Peng Probabilistic multimodal learning with von mises-fisher distributions. In IJCAI, pp. 5390–5398. Cited by: §C.1, §D.2, §D.4, §E.1, Table 12, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, §1, §3.3, §4.1, Table 1.
Huang et al. (2025) C. Huang, J. Chen, Q. Huang, S. Wang, Y. Tu, and X. Huang AtCAF: attention-based causality-aware fusion network for multimodal sentiment analysis. Information Fusion 114, pp. 102725. Cited by: §D.1, §D.2, §D.5, §4.2.
Jaiswal and Provost (2020) M. Jaiswal and E. M. Provost Privacy enhanced multimodal neural representations for emotion recognition. In AAAI, Vol. 34, pp. 7985–7993. Cited by: §1.
Jiang et al. (2025a) M. Jiang, Y. Jiang, H. Hu, and S. Mai Towards minimal causal representations for human multimodal language understanding. arXiv preprint arXiv:2509.21805. Cited by: §2.1.
Jiang et al. (2025b) M. Jiang, Y. Lin, B. Chen, H. Hu, Y. Jiang, and S. Mai Disentangling bias by modeling intra-and inter-modal causal attention for multimodal sentiment analysis. arXiv preprint arXiv:2508.04999. Cited by: §2.1.
Lan et al. (2020) Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut Albert: a lite bert for self-supervised learning of language representations. In ICLR, Cited by: §D.3.
Li et al. (2023a) H. Li, X. Li, P. Hu, Y. Lei, C. Li, and Y. Zhou Boosting multi-modal model performance with adaptive gradient modulation. In ICCV, pp. 22214–22224. Cited by: §D.2.
Li et al. (2024a) M. Li, D. Yang, Y. Liu, S. Wang, J. Chen, S. Wang, J. Wei, Y. Jiang, Q. Xu, X. Hou, et al. Toward robust incomplete multimodal sentiment analysis via hierarchical representation learning. In NeurIPS, Vol. 37, pp. 28515–28536. Cited by: §2.2.
Li et al. (2024b) M. Li, D. Yang, X. Zhao, S. Wang, Y. Wang, K. Yang, M. Sun, D. Kou, Z. Qian, and L. Zhang Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities. In CVPR, pp. 12458–12468. Cited by: §1, §2.2, §2.2, §3.2, §3.4.
Li et al. (2023b) Y. Li, Y. Wang, and Z. Cui Decoupled multimodal distilling for emotion recognition. In CVPR, pp. 6631–6640. Cited by: §D.2, §D.4.
Li et al. (2026) Y. Li, Y. Wang, Y. Ding, S. Zhang, K. Lu, and C. Guan Decoupled hierarchical distillation for multimodal emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §D.2, §D.5.
Lian et al. (2023) Z. Lian, L. Chen, L. Sun, B. Liu, and J. Tao Gcnet: graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on pattern analysis and machine intelligence 45 (7), pp. 8419–8432. Cited by: §2.2.
Liang et al. (2019) P. P. Liang, Z. Liu, Y. H. Tsai, Q. Zhao, R. Salakhutdinov, and L. Morency Learning representations from imperfect time series data via tensor rank regularization. In ACL, pp. 1569–1576. Cited by: §B.1.
Lin et al. (2025) R. Lin, Q. He, S. Mai, Y. Zeng, A. Xiong, L. Huang, Y. Tan, and H. Hu CyIN: cyclic informative latent space for bridging complete and incomplete multimodal learning. In NeurIPS, Cited by: §D.2, §D.3, §1, §1, §2.2, §4.1, §4.2.
Liu et al. (2017) Z. Liu, M. Wu, W. Cao, L. Chen, J. Xu, R. Zhang, M. Zhou, and J. Mao A facial expression emotion recognition based human-robot interaction system.. IEEE CAA J. Autom. Sinica 4 (4), pp. 668–676. Cited by: §1.
Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §D.4.
Luo et al. (2025) M. Luo, Y. Jiang, and S. Mai Towards explainable fusion and balanced learning in multimodal sentiment analysis. In ACM MM, pp. 1997–2006. Cited by: §B.1, §D.2, §D.4, §D.5, §E.1, §E.2, §E.7, Table 12, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, §4.1, Table 1.
Mai and Han (2026) S. Mai and S. Han CaReFlow: cyclic adaptive rectified flow for multimodal fusion. arXiv preprint arXiv:2602.19140. Cited by: §2.1.
Mai et al. (2023a) S. Mai, Y. Sun, A. Xiong, Y. Zeng, and H. Hu Multimodal boosting: addressing noisy modalities and identifying modality contribution. IEEE Transactions on Multimedia 26, pp. 3018–3033. Cited by: §B.1.
Mai et al. (2023b) S. Mai, Y. Zeng, and H. Hu Multimodal information bottleneck: learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multimedia 25, pp. 4121–4134. Cited by: §B.1, §D.2, §D.4, §E.1, Table 12, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, §1, §4.1, Table 1.
Mai et al. (2025) S. Mai, Y. Zeng, and H. Hu Learning by comparing: boosting multimodal affective computing through ordinal learning. In WWW, pp. 2120–2134. Cited by: §D.2, §D.5, §2.1, §4.2.
Mao et al. (2023) H. Mao, B. Zhang, H. Xu, Z. Yuan, and Y. Liu Robust-msa: understanding the impact of modality noise on multimodal sentiment analysis. In AAAI, Vol. 37, pp. 16458–16460. Cited by: §B.1.
Ming et al. (2023) Y. Ming, Y. Sun, O. Dia, and Y. Li How to exploit hyperspherical embeddings for out-of-distribution detection?. In ICLR, Cited by: §C.1, §3.3.
Pham et al. (2019) H. Pham, P. P. Liang, T. Manzini, L. Morency, and B. Póczos Found in translation: learning robust joint representations by cyclic translations between modalities. In AAAI, Vol. 33, pp. 6892–6899. Cited by: §2.2.
Pramanick et al. (2022) S. Pramanick, A. Roy, and V. M. Patel Multimodal learning using optimal transport for sarcasm and humor detection. In WACV, pp. 3930–3940. Cited by: §D.2, §D.4.
Qian et al. (2026) C. Qian, S. Xing, S. Li, Y. Zhao, and Z. Tu Decalign: hierarchical cross-modal alignment for decoupled multimodal representation learning. In ICLR, Cited by: §D.2, §1, §2.1, §4.1, §4.2, Table 1.
Rahman et al. (2020) W. Rahman, M. K. Hasan, S. Lee, A. Zadeh, C. Mao, L. Morency, and E. Hoque Integrating multimodal information in large pretrained transformers. In ACL, pp. 2359. Cited by: §D.2, §D.4.
Tishby et al. (2000) N. Tishby, F. C. Pereira, and W. Bialek The information bottleneck method. arXiv preprint physics/0004057. Cited by: §B.1, §1.
Tsai et al. (2019) Y. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L. Morency, and R. Salakhutdinov Multimodal transformer for unaligned multimodal language sequences. In ACL, pp. 6558. Cited by: §3.2.
Van der Maaten and Hinton (2008) L. Van der Maaten and G. Hinton Visualizing data using t-sne.. Journal of machine learning research 9 (11). Cited by: §4.4.
Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In NeurIPS, Vol. 30. Cited by: §3.2.
Vitter (1985) J. S. Vitter Random sampling with a reservoir. ACM Transactions on Mathematical Software 11 (1), pp. 37–57. Cited by: §C.2, §3.4.
Wang et al. (2025) P. Wang, Q. Zhou, Y. Wu, T. Chen, and J. Hu DLF: disentangled-language-focused multimodal sentiment analysis. In AAAI, pp. 21180–21188. Cited by: §2.1.
Wang et al. (2023a) Y. Wang, Z. Cui, and Y. Li Distribution-consistent modal recovering for incomplete multimodal learning. In CVPR, pp. 22025–22034. Cited by: §2.2.
Wang et al. (2023b) Y. Wang, Y. Li, and Z. Cui Incomplete multimodality-diffused emotion recognition. In NeurIPS, Vol. 36, pp. 17117–17128. Cited by: §2.2.
Wu et al. (2025a) Q. Wu, Y. Shao, J. Wang, and X. Sun Learning optimal multimodal information bottleneck representations. In ICML, Cited by: §B.1, §D.2, §D.4, §E.1, Table 12, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, §1, §4.1, Table 1.
Wu et al. (2025b) S. Wu, D. He, X. Wang, L. Wang, and J. Dang Enriching multimodal sentiment analysis through textual emotional descriptions of visual-audio content. In AAAI, pp. 1601–1609. Cited by: §2.1.
Xiao et al. (2024) X. Xiao, G. Liu, G. Gupta, D. Cao, S. Li, Y. Li, T. Fang, M. Cheng, and P. Bogdan Neuro-inspired information-theoretic hierarchical perception for multimodal learning. In ICLR, Cited by: §B.1, §D.2, §D.4, §D.5, §E.1, Table 12, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, Table 1.
Xie et al. (2026) H. Xie, K. Zhu, Z. Wen, J. Tao, X. Liu, R. Fu, and C. Li PSA-mf: personality-sentiment aligned multi-level fusion for multimodal sentiment analysis. In AAAI, Cited by: §D.2, §2.1, §4.1, Table 1.
Xu et al. (2025) Z. Xu, D. Yang, M. Li, Y. Wang, Z. Chen, J. Chen, J. Wei, and L. Zhang Debiased multimodal understanding for human language sequences. In AAAI, pp. 14450–14458. Cited by: §D.1, §D.2, §D.4, §D.5.
Xue and Marculescu (2023) Z. Xue and R. Marculescu Dynamic multimodal fusion. In CVPR, pp. 2575–2584. Cited by: §B.1.
Yang et al. (2022) D. Yang, S. Huang, H. Kuang, Y. Du, and L. Zhang Disentangled representation learning for multimodal emotion recognition. In ACM MM, pp. 1642–1651. Cited by: §D.2, §D.4.
Yang et al. (2026) D. Yang, M. Li, X. Wu, Z. Chen, K. Jiang, K. Liu, P. Zhai, and L. Zhang Improving multimodal sentiment analysis via modality optimization and dynamic primary modality selection. In AAAI, Cited by: §D.2, §1, §2.1, §4.1, §4.2, Table 1.
Ye et al. (2025) J. Ye, J. Zhang, and H. Shan Depmamba: progressive fusion mamba for multimodal depression detection. In ICASSP, pp. 1–5. Cited by: §1.
Yu et al. (2021) W. Yu, H. Xu, Z. Yuan, and J. Wu Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In AAAI, pp. 10790–10797. Cited by: §D.2, §D.4.
Zadeh et al. (2017) A. Zadeh, M. Chen, S. Poria, E. Cambria, and L. Morency Tensor fusion network for multimodal sentiment analysis. In EMNLP, pp. 1103–1114. Cited by: §2.1.
Zadeh et al. (2018a) A. Zadeh, Y. C. Lim, and L. Morency Openface 2.0: facial behavior analysis toolkit tadas baltrušaitis. In FG, pp. 59–66. Cited by: §D.3.
Zadeh et al. (2016) A. Zadeh, R. Zellers, E. Pincus, and L. Morency Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intelligent Systems 31, pp. 82–88. Cited by: §D.1, §1, §2.1, §4.1.
Zadeh et al. (2018b) A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L. Morency Multimodal language analysis in the wild: cmu-mosei dataset and interpretable dynamic fusion graph. In ACL, pp. 2236–2246. Cited by: §D.1, §4.1.
Zhang et al. (2024) H. Zhang, W. Wang, and T. Yu Towards robust multimodal sentiment analysis with incomplete data. In NeurIPS, Vol. 37, pp. 55943–55974. Cited by: §1, §2.2, §3.5.
Zhang et al. (2023a) H. Zhang, Y. Wang, G. Yin, K. Liu, Y. Liu, and T. Yu Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In EMNLP, pp. 756–767. Cited by: §2.1, §3.5.
Zhang et al. (2023b) Q. Zhang, H. Wu, C. Zhang, Q. Hu, H. Fu, J. T. Zhou, and X. Peng Provable dynamic fusion for low-quality multimodal data. In ICML, pp. 41753–41769. Cited by: §D.2, §D.4, §E.1, §E.2, §E.2, Table 12, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, §4.1, Table 1.
Zhang et al. (2023c) Y. Zhang, Y. Yu, D. Zhao, Z. Li, B. Wang, Y. Hou, P. Tiwari, and J. Qin Learning multitask commonness and uniqueness for multimodal sarcasm detection and sentiment analysis in conversation. IEEE Transactions on Artificial Intelligence 5, pp. 1349–1361. Cited by: §D.2.
Zhao et al. (2021) J. Zhao, R. Li, and Q. Jin Missing modality imagination network for emotion recognition with uncertain missing modalities. In ACL, pp. 2608–2618. Cited by: §2.2.
Zhuang et al. (2025a) Y. Zhuang, M. Liu, W. Bai, Y. Zhang, X. Zhang, J. Deng, and F. Ren CMAD: correlation-aware and modalities-aware distillation for multimodal sentiment analysis with missing modalities. In ICCV, pp. 4626–4636. Cited by: §2.2.
Zhuang et al. (2026) Y. Zhuang, M. Liu, Y. Zhang, J. Deng, and F. Ren TMDC: a two-stage modality denoising and complementation framework for multimodal sentiment analysis with missing and noisy modalities. In AAAI, Vol. 40, pp. 2281–2289. Cited by: §1, §1.
Zhuang et al. (2025b) Y. Zhuang, L. Minhao, W. Bai, Y. Zhang, W. Li, J. Deng, and F. Ren Hyper-modality enhancement for multimodal sentiment analysis with missing modalities. In NeurIPS, Cited by: §D.2, §D.3, §D.4, §E.1, §E.2, §E.2, §E.7, Table 12, Table 6, Table 6, Table 6, Table 7, Table 7, Table 7, §1, §2.2, §2.2, §3.2, §3.4, §3.5, §4.1, §4.2, §4.4, Table 1.

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis
(Appendix)

The Use of Large Language Models (LLMs)

In this work, large language models (LLMs) were used solely for language polishing to improve clarity and fluency. All scientific content, including the experimental design, results, and conclusions, is the responsibility of the authors. No LLM was involved in the ideation, analysis, or interpretation of the research, and all LLM-assisted content was carefully reviewed and verified by the authors.

Appendix A Notations

We provide a comprehensive overview of the commonly used notations and their definitions in Table 4.

Table 4: Notation and Definitions

Notation	Definition
$X_{m}^{i}$	Input feature sequence of modality $m$ for sample $i$ .
$L_{m}$	Sequence length of modality $m$ .
$d_{m}$	Original feature dimension of modality $m$ .
$d$	Shared embedding dimension across all modalities.
$\hat{X}_{m}^{i}$	Projected modality feature after 1D convolution.
$S_{m}^{i}$	Sequence representation after modality-specific Transformer encoder.
$H_{m}^{i}$	Modality-specific representation obtained after pre-processing.
$z_{m}^{i}$	Latent representation sampled from AVIB.
$\mu_{m}^{i}$	Mean direction of the vMF distribution.
$\kappa_{m}^{i}$	Concentration parameter of the vMF distribution reflecting modality reliability.
$q(z_{m}^{i}\mid H_{m}^{i})$	Variational posterior of modality $m$ .
$p(z_{m}^{i})$	Uniform prior distribution on the unit hypersphere.
$\mathcal{B}_{m}$	Memory bank for modality $m$ .
$\mathcal{C}_{i}^{m}$	Candidate neighbor set for sample $i$ .
$\mathcal{N}_{i}^{m}$	Selected top- $k$ neighbors.
$S_{m}(i,j)$	Directional similarity between sample $i$ and $j$ .
$w_{ij}^{(m)}$	Aggregation weight for neighbor $j$ .
$\tilde{z}_{m}^{i}$	Enhanced modality representation from RME.
$\alpha$	Label discrepancy penalty coefficient.
$\rho$	Proportion for selecting top- $k$ neighbors.
$l_{p}$	Length of latent prompts.
$E_{m}^{i}$	Modality-level latent representation.
$E_{S}^{i}$	Shared cross-modal representation.
$R_{m}^{i}$	Hyper-modality representation.
$a_{m}^{i}$	Fusion logits for intra-modality aggregation.
$\beta_{m}^{i}$	Adaptive fusion weights for intra-modality fusion.
$u_{m}^{i}$	Intra-modality fused representation.
$\gamma^{i}$	Reliability weights derived from modality concentration.
$\eta^{i}$	Reliability-aware modality fusion weights.
$u_{H}^{i}$	Global multimodal representation.
$U^{i}$	Token set combining intra- and inter-modality representations.
$r^{i}$	Final joint representation.

Appendix B More Related Work

B.1 Multimodal Sentiment Analysis with Noisy Modalities

Existing approaches for handling modality noise can be broadly categorized into three groups. The first group, noise augmentation methods, simulates real-world noise by injecting perturbations into features or raw inputs and developing corresponding defenses. For example, Robust-MSA [35] constructs an interactive analysis platform to visualize the impact of modality-specific noise on model performance, and incorporates simple defense strategies to improve interpretability. Similarly, MSA-Robustness [11] introduces a suite of diagnostic tools to assess the sensitivity of multimodal models to noise in individual modalities, and systematically explores robust training strategies to enhance stability under noisy perturbations. However, designing dedicated defenses for different noise types is complex and often lacks generalization.

The second group focuses on noise identification and filtering mechanisms to suppress noisy signals and improve the robustness of multimodal representations [7]. For instance, DynMM [54] introduces a dynamic gating mechanism that adaptively adjusts modality contributions or fusion pathways based on input features, thereby mitigating the influence of noisy modalities. However, such implicit weighting strategies lack explicit reliability modeling and thus remain susceptible to noise. To further improve modality contribution estimation, Multimodal Boosting [32] leverages unimodal prediction losses as supervisory signals to identify noisy modalities. Nevertheless, this approach relies on the construction of absolute labels, which are difficult to define precisely and may introduce additional noise during training.

The third group adopts representation regularization strategies, which have emerged as a promising direction. These methods typically leverage techniques such as tensor rank minimization [26] or information-theoretic principles [41] to extract discriminative representations from noisy features. For example, MIB [33], based on the information bottleneck principle [41], suppresses redundancy and noise in both unimodal and multimodal representations by learning minimal sufficient task-relevant features. ITHP [51] designates a primary modality while treating the remaining modalities as auxiliary detectors to extract salient information. OMIB [49] further extends the multimodal information bottleneck framework by constraining regularization weights to ensure an optimal bottleneck representation. Similarly, KAN-MCP [30] integrates cross-modal modeling with Dimensionality Reduction and Denoising Modal Information Bottleneck (DRD-MIB)-based feature compression to learn more robust representations.

Despite the success of representation regularization methods, particularly those based on the information bottleneck principle, such approaches typically compress all modalities equally and lack the ability to distinguish modality reliability, which makes them less effective in the presence of noise or heterogeneous modality quality. In contrast, RCE introduces an Adaptive Variational Information Bottleneck (AVIB) in the von Mises–Fisher (vMF) representation space. Specifically, AVIB jointly models directional information and sample-wise posterior uncertainty within a unified hyperspherical space, enabling reliability-aware adaptive compression that preserves informative signals from reliable modalities while suppressing noise from less reliable ones.

Appendix C Methodological Details

C.1 von Mises–Fisher Distribution

In this section, we provide a detailed description of the von Mises–Fisher (vMF) distribution and its role in modeling sample-level uncertainty in our RCE framework.

vMF-based Uncertainty Estimation.

To explicitly model sample-level reliability differences across modalities, we map the feature representation of each modality onto a probability distribution defined on the unit hypersphere. Specifically, for the $i$ -th sample and modality $m$ , the representation $H_{m}^{i}$ is used to parameterize a von Mises–Fisher (vMF) posterior distribution:

z_{m}^{i}\sim\mathcal{V}_{d}(\mu_{m}^{i},\kappa_{m}^{i}),

(17)

where $\mu_{m}^{i}\in\mathbb{S}^{d-1}$ denotes the mean direction and $\kappa_{m}^{i}\in\mathbb{R}^{+}$ is the concentration parameter.

The parameters $\mu_{m}^{i}$ and $\kappa_{m}^{i}$ are computed as:

\mu_{m}^{i}=\frac{f_{m}(H_{m}^{i})}{\|f_{m}(H_{m}^{i})\|_{2}}\in\mathbb{S}^{d-1},\qquad\kappa_{m}^{i}=\mathrm{softplus}(g_{m}(H_{m}^{i}))+\epsilon\in\mathbb{R}^{+},

(18)

where $f_{m}(\cdot)$ and $g_{m}(\cdot)$ are two separate subnetworks used to estimate the directional and concentration parameters, respectively. The normalization ensures that $\mu_{m}^{i}$ lies on the unit hypersphere, while the softplus function guarantees that $\kappa_{m}^{i}$ remains positive. The constant $\epsilon$ is a small value introduced to ensure numerical stability.

Under this formulation, $\mu_{m}^{i}$ represents the principal direction of the latent representation on the hypersphere, while $\kappa_{m}^{i}$ controls the dispersion of the distribution around this direction. Intuitively, a larger $\kappa_{m}^{i}$ results in a more concentrated distribution, indicating higher certainty, whereas a smaller $\kappa_{m}^{i}$ corresponds to a more dispersed distribution, reflecting greater uncertainty. Following prior work [36, 4, 14], we interpret the concentration parameter $\kappa_{m}^{i}$ as a proxy for sample-level reliability of the corresponding modality. This interpretation allows us to quantify the confidence of each modality and later leverage it for reliability-aware fusion.

Probability Density Function.

The vMF distribution is defined on the unit hypersphere $\mathbb{S}^{d-1}\subset\mathbb{R}^{d}$ . The probability density function of a random variable $z\in\mathbb{S}^{d-1}$ is given by:

p(z\mid\mu_{m}^{i},\kappa_{m}^{i})=C_{d}(\kappa_{m}^{i})\exp\left(\kappa_{m}^{i}{\mu_{m}^{i}}^{\top}z\right),

(19)

where $C_{d}(\kappa_{m}^{i})$ is the normalization constant defined as:

C_{d}(\kappa_{m}^{i})=\frac{(\kappa_{m}^{i})^{\frac{d}{2}-1}}{(2\pi)^{\frac{d}{2}}I_{\frac{d}{2}-1}(\kappa_{m}^{i})}.

(20)

Here, $I_{\nu}(\cdot)$ denotes the modified Bessel function of the first kind of order $\nu$ , which is defined as:

I_{\nu}(t)=\sum_{j=0}^{\infty}\frac{\left(\frac{t}{2}\right)^{2j+\nu}}{j!\,\Gamma(j+\nu+1)},

(21)

where $\Gamma(\cdot)$ is the Gamma function.

Discussion.

The vMF distribution provides a natural way to model directional data on the hypersphere. In our setting, it enables a unified representation that jointly captures both the semantic direction (via $\mu_{m}^{i}$ ) and the uncertainty (via $\kappa_{m}^{i}$ ). This property is important for multimodal learning, where different modalities may exhibit varying levels of reliability across samples. By leveraging the concentration parameter as an uncertainty measure, our framework achieves adaptive, reliability-aware representation learning.

C.2 Reservoir Sampling Strategy

We use reservoir sampling [45] to maintain a fixed-size modality-specific memory bank while avoiding bias toward recently observed samples. For each modality $m$ , the memory bank is denoted as

\mathcal{B}_{m}=\{(z_{m}^{r},\mu_{m}^{r},\kappa_{m}^{r},y^{r})\}_{r=1}^{N_{m}},

(22)

where $N_{m}\leq N_{b}$ and $N_{b}$ is the memory bank capacity. During memory construction, samples are sequentially processed by a forward pass without gradient updates. Let $t$ denote the number of samples that have been observed so far for updating the memory bank of modality $m$ . For the $t$ -th observed sample, we obtain its latent representation $z_{m}^{t}$ , vMF mean direction $\mu_{m}^{t}$ , concentration parameter $\kappa_{m}^{t}$ , and label $y^{t}$ .

If the memory bank is not full, i.e., $t\leq N_{b}$ , the sample is directly inserted into $\mathcal{B}_{m}$ . Once the memory bank reaches its capacity, i.e., $t>N_{b}$ , the new sample is retained with probability $N_{b}/t$ . Specifically, we draw a random integer

q\sim\mathrm{Uniform}\{1,2,\dots,t\}.

(23)

If $q\leq N_{b}$ , we replace the $q$ -th element in $\mathcal{B}_{m}$ with the current sample; otherwise, the current sample is discarded. The update rule can be written as:

\mathcal{B}_{m}[q]\leftarrow(z_{m}^{t},\mu_{m}^{t},\kappa_{m}^{t},y^{t}),\quad\text{if }q\leq N_{b}.

(24)

Otherwise, $\mathcal{B}_{m}$ remains unchanged.

This strategy guarantees that after observing $t$ samples, each sample has equal probability of being retained in the memory bank:

\Pr\left((z_{m}^{s},\mu_{m}^{s},\kappa_{m}^{s},y^{s})\in\mathcal{B}_{m}\right)=\min\left(1,\frac{N_{b}}{t}\right),\quad\forall s\in\{1,\dots,t\}.

(25)

When $t\leq N_{b}$ , all observed samples are stored. When $t>N_{b}$ , each observed sample is retained with probability $N_{b}/t$ , regardless of its arrival order.

For completeness, we briefly justify this property. For a sample observed at step $s\leq N_{b}$ , it is initially inserted into the memory bank. At any later step $\tau>N_{b}$ , it is replaced only if the new sample is accepted and its stored position is selected. This happens with probability

\frac{N_{b}}{\tau}\cdot\frac{1}{N_{b}}=\frac{1}{\tau}.

(26)

Therefore, the probability that this sample survives from step $N_{b}+1$ to step $t$ is

\prod_{\tau=N_{b}+1}^{t}\left(1-\frac{1}{\tau}\right)=\prod_{\tau=N_{b}+1}^{t}\frac{\tau-1}{\tau}=\frac{N_{b}}{t}.

(27)

For a sample observed at step $s>N_{b}$ , it is inserted with probability $N_{b}/s$ . If inserted, it survives each subsequent step $\tau=s+1,\dots,t$ with probability $1-1/\tau$ . Hence its final retention probability is

\frac{N_{b}}{s}\prod_{\tau=s+1}^{t}\left(1-\frac{1}{\tau}\right)=\frac{N_{b}}{s}\prod_{\tau=s+1}^{t}\frac{\tau-1}{\tau}=\frac{N_{b}}{s}\cdot\frac{s}{t}=\frac{N_{b}}{t}.

(28)

Thus, reservoir sampling maintains a uniformly sampled subset of historical samples under the fixed capacity constraint.

In our implementation, the above procedure is applied independently for each modality-specific memory bank. At the beginning of each training epoch, we rebuild the memory bank by forwarding training samples through the AVIB module without gradient updates and updating $\mathcal{B}_{m}$ using the reservoir sampling rule. This yields a balanced and order-agnostic memory bank for neighbor retrieval. When the memory bank is fully populated, RME retrieves candidate neighbors from $\mathcal{B}_{m}$ ; otherwise, it uses intra-batch candidates as described in Section 3.4.

Appendix D Experimental Details

D.1 Datasets Information

We evaluate the proposed RCE on four benchmark datasets covering three tasks: multimodal sentiment analysis (MSA), multimodal humor detection (MHD), and multimodal sarcasm detection (MSD). We next provide a brief description of each dataset.

CMU-MOSI [61]: A widely adopted benchmark for MSA, consisting of over 2,000 video segments collected from online sources. Each segment is assigned a sentiment intensity score on a seven-point Likert scale ranging from $-3$ (most negative) to $3$ (most positive).

CMU-MOSEI [62]: One of the largest and most diverse MSA datasets, comprising more than 22,000 video segments from over 1,000 YouTube speakers across approximately 250 topics. Each segment is annotated with both categorical emotions (six classes) and sentiment scores on the same $-3$ to $3$ scale as CMU-MOSI. In our experiments, we focus on the sentiment annotations for consistency.

UR-FUNNY [10]: A benchmark dataset for MHD, derived from TED talk videos featuring 1,741 speakers. Each target segment, referred to as a punchline, is annotated across language, acoustic, and visual modalities, while preceding segments serve as contextual input. Punchlines are identified using the laughter tag in transcripts, which indicates audience laughter; negative samples are constructed when no such cue is present. The dataset is split into 7,614 training, 980 validation, and 994 testing samples. Following prior works [9, 15, 53], we adopt version 2 of UR-FUNNY.

MUStARD [1]: A dataset for MSD, collected from popular television series such as Friends, The Big Bang Theory, The Golden Girls, and Sarcasmaholics. It contains 690 video segments labeled as sarcastic or non-sarcastic. Each instance includes the target punchline along with preceding dialogue to provide context.

D.2 Baselines

This section offers a thorough overview of the baseline methods utilized in our study.

MODS [56] [AAAI’26]: Modality Optimization and Dynamic Primary Modality Selection (MODS) tackles modality imbalance by adaptively identifying the most informative modality for each sample. It incorporates a graph-based dynamic sequence compression module to minimize redundancy in non-linguistic modalities, and leverages a primary-modality-focused cross-attention mechanism to strengthen cross-modal integration.

PSA-MF [52] [AAAI’26]: Personality-Sentiment Aligned Multi-level Fusion (PSA-MF) integrates personality traits into the feature extraction process to learn personalized sentiment representations, and employs a hierarchical fusion strategy to progressively combine multimodal information, thereby enhancing sentiment recognition performance;

DecAlign [39] [ICLR’26]: Decoupled Cross-modal Alignment (DecAlign) tackles multimodal heterogeneity by decomposing representations into modality-specific and shared components. It employs a prototype-guided optimal transport strategy to align modality-specific features and leverages MMD-based distribution matching to enforce cross-modal consistency, further enhanced by a multimodal transformer for high-level fusion.

MIB [33] [TMM’23]: Multimodal Information Bottleneck (MIB) applies the information bottleneck principle to learn compact, task-relevant representations by reducing redundancy and noise in both unimodal and multimodal features. It provides three variants—E-MIB, L-MIB, and C-MIB—that impose information constraints at different stages of multimodal fusion.

ITHP [51] [ICLR’24]: Information-Theoretic Hierarchical Perception (ITHP) leverages the information bottleneck principle by assigning a primary modality and utilizing auxiliary modalities as detectors to extract salient information.

KAN-MCP [30] [ACM MM’25]: Kolmogorov–Arnold Network with Multimodal Clean Pareto (KAN-MCP) combines interpretable cross-modal modeling with DRD-MIB-based feature compression and denoising, enabling discriminative representation learning while mitigating modality imbalance.

OMIB [49] [ICML’25]: Optimal Multimodal Information Bottleneck (OMIB) extends the MIB framework by theoretically constraining regularization weights to ensure an optimal bottleneck representation, while dynamically adjusting modality-specific regularization to alleviate modality imbalance.

QMF [65] [ICML’23]: Quality-aware Multimodal Fusion (QMF) provides a theoretically grounded framework for robust multimodal learning by analyzing multimodal fusion from a generalization perspective. It leverages uncertainty estimation to dynamically assess modality quality, enabling adaptive cross-modal interaction and mitigating the impact of low-quality data, thereby promoting more reliable and quality-aware feature integration.

PML [14] [IJCAI’25]: Probabilistic Multimodal Learning (PML) models representations with von Mises–Fisher (vMF) distributions to capture intrinsic uncertainty and learn reliable directional features. It employs a vMF-based prototypical contrastive learning paradigm to enhance class discrimination, and introduces a reliability-aware fusion mechanism to dynamically regulate modality contributions and resolve cross-modal conflicts.

HME [70] [NeurIPS’25]: Hyper-Modality Enhancement (HME) tackles missing modality scenarios by enriching observed modalities with semantically relevant cues from other samples, avoiding explicit reconstruction. It further adopts an uncertainty-aware fusion strategy to adaptively balance original and enhanced representations, improving robustness to incomplete and noisy inputs.

CyIN [27] [NeurIPS’25]: Cyclic INformative Learning (CyIN) addresses dynamic missing modality scenarios by constructing an informative latent space via token- and label-level Information Bottleneck (IB) applied cyclically across modalities. It further introduces cross-modal cyclic translation to reconstruct missing modalities through forward and reverse processes, enabling unified optimization for both complete and incomplete multimodal learning while enhancing cross-modal interaction and fusion.

Self-MM [58] [AAAI’21]: Self-Supervised Multi-task Multimodal (Self-MM) sentiment analysis framework generates pseudo labels for each modality based on annotated global sentiment labels, enabling the learning of more discriminative unimodal representations.

FDMER [55] [ACM MM’22]: Feature-Disentangled Multimodal Emotion Recognition (FDMER) alleviates distribution gaps and redundancy across modalities by decomposing features into modality-invariant and modality-specific subspaces. It employs common and private encoders with adversarial training to enforce consistency and diversity, and utilizes a cross-modal attention fusion module to learn adaptive multimodal representations.

DMD [23] [CVPR’23]: Decoupled Multimodal Distillation (DMD) improves emotion recognition by separating each modality into modality-relevant and modality-exclusive components, and conducting adaptive cross-modal knowledge distillation through a dynamic graph structure.

AGM [20] [ICCV’23]: Adaptive Gradient Modulation (AGM) addresses modality competition in multimodal learning by adaptively adjusting gradients during training to balance modality contributions across various fusion strategies. It further introduces a novel metric based on the mono-modal concept to quantify competition strength, providing insights into modality dominance and improving overall model performance.

AtCAF [15] [Information Fusion’25]: Attention-based Causality-Aware Fusion (AtCAF) learns causality-aware multimodal representations through a text debiasing module and counterfactual cross-modal attention for sentiment analysis.

SuCI [53] [AAAI’25]: Subject Causal Intervention (SuCI) mitigates subject-related spurious correlations by modeling subjects as confounders and applying causal intervention, leading to more robust multimodal language understanding and improved cross-subject generalization.

DHMD [24] [TPAMI’26]: Decoupled Hierarchical Multimodal Distillation (DHMD) separates multimodal features into modality-invariant and modality-specific components via a self-regression mechanism, and adopts a hierarchical distillation framework—combining graph-based coarse-grained and dictionary-based fine-grained distillation—to improve cross-modal alignment and emotion recognition.

MAG [40] [ACL’20]: Multimodal Adaptation Gate (MAG) introduces an adaptation gate that enables large pre-trained transformers to incorporate multimodal signals during the fine-tuning stage.

BBFN [8] [ICMI’21]: Bi-Bimodal Fusion Network (BBFN) conducts both fusion and separation over pairwise modality representations, and utilizes a gated Transformer to alleviate modality imbalance.

MuLOT [38] [WACV’22]: Multimodal Learning using Optimal Transport (MuLOT) addresses resource-constrained sarcasm and humor detection by leveraging self-attention to model intra-modal relationships and optimal transport to capture cross-modal alignment, followed by multimodal attention fusion to integrate complementary information across modalities.

MIL [66] [TAI’23]: Multimodal Interaction Learning (MIL) is a multitask framework for joint sarcasm and sentiment detection that models both shared and task-specific features. It introduces a cross-modal target attention mechanism to enhance multimodal fusion and facilitate interaction between related tasks.

MOAC [34] [WWW’25]: Multimodal Ordinal Affective Computing (MOAC) improves affective modeling by incorporating ordinal learning at both the label level (coarse-grained) and feature level (fine-grained) within multimodal representations.

D.3 Feature Extraction

Textual Modality: For the CMU-MOSI and CMU-MOSEI datasets, we adopt DeBERTa [13] to extract high-level textual representations, following prior work. For the UR-FUNNY dataset, ALBERT [19] is employed as the text encoder. Specifically, the context and punchline token sequences are concatenated to form the input sequence: $U_{l}=C_{l}\oplus\texttt{[SEP]}\oplus P_{l}$ , where [SEP] is used to separate the context tokens $C_{l}$ and punchline tokens $P_{l}$ . For the MUStARD dataset, contextual word representations are extracted using a pretrained BERT [3] model.

For the experiments reported in Figure 2 of the main paper, in order to ensure a fair comparison with CyIN [27], both RCE and HME [70] adopt BERT [3] to extract textual modality features.

Acoustic Modality: Acoustic features for CMU-MOSI, CMU-MOSEI, UR-FUNNY, and MUStARD are extracted using COVAREP [2], including 12-dimensional MFCCs, pitch, speech polarity, glottal closure instants, and spectral envelope descriptors. These features are computed over the full audio segment of each utterance, forming temporal sequences that capture dynamic vocal variations.

Visual Modality: For CMU-MOSI and CMU-MOSEI, visual features are obtained using Facet (iMotions 2017, https://imotions.com/), including facial action units, landmarks, head pose, and other expression-related cues, organized as temporal sequences to capture facial dynamics. For UR-FUNNY and MUStARD, OpenFace [60] is used to extract facial action units along with both rigid and non-rigid facial shape parameters.

For the CMU-MOSI dataset, the feature dimensions of the textual, acoustic, and visual modalities are 768, 74, and 47, respectively. For CMU-MOSEI, the corresponding dimensions are 768, 74, and 35. For both UR-FUNNY and MUStARD, the dimensions of the textual, acoustic, and visual modalities are 768, 60, and 36, respectively. In addition, UR-FUNNY includes an extra humor centric feature (HCF) modality with a dimension of 4. Details of the HCF feature extraction process can be found in [9].

D.4 Implementation Details

We implement RCE using the PyTorch framework and conduct all experiments on a single NVIDIA GeForce RTX 4090 GPU, using CUDA 12.8 and PyTorch 2.8.0. The model is optimized using AdamW [29] with linear warm-up. Detailed hyper-parameter settings for different datasets are summarized in Table 5.

Several hyper-parameters are shared across all datasets. Specifically, warm-up is applied in all experiments, and AdamW is consistently adopted as the optimizer. The modality enhancement process uses a modality-specific memory bank with capacity $N_{b}$ , where neighbor retrieval is performed based on the top- $k$ ratio $\rho$ . The prompt length is denoted by $l_{p}$ , the label penalty weight by $\alpha$ , the AVIB trade-off coefficient by $\beta$ , and the overall loss balancing weight by $\lambda$ .

For the MSA datasets CMU-MOSI and CMU-MOSEI, we use relatively larger feature dimensions and memory bank capacities to capture richer multimodal contextual information. Specifically, the feature dimensions are set to $d=256$ and $d=192$ , respectively, while the memory bank capacities are set to $1024$ and $2048$ . The batch sizes are set to $48$ for CMU-MOSI and $128$ for CMU-MOSEI, with initial learning rates of $2\times 10^{-5}$ and $3\times 10^{-5}$ , respectively. Both datasets use a dropout rate of $0.4$ and AVIB trade-off coefficient $\beta=1\mathrm{e}{-3}$ . For the MHD and MSD datasets UR-FUNNY and MUStARD, we adopt comparatively smaller dropout rates to preserve discriminative multimodal cues. The feature dimensions are set to $d=192$ and $d=144$ , respectively. The prompt lengths differ across datasets, with $l_{p}=3$ for UR-FUNNY and $l_{p}=8$ for MUStARD. Both datasets use a memory bank capacity of $1024$ , while the top- $k$ ratios are set to $0.3$ and $0.5$ , respectively. The AVIB trade-off coefficients are set to $1\mathrm{e}{-4}$ for UR-FUNNY and $1\mathrm{e}{-2}$ for MUStARD.

To determine the optimal configuration, we perform a grid search with thirty random trials on the validation set. The batch size is selected from ${32,48,64,128,256}$ , the initial learning rate is searched over ${1\mathrm{e}{-6},3\mathrm{e}{-6},8\mathrm{e}{-6},1\mathrm{e}{-5},2\mathrm{e}{-5},3\mathrm{e}{-5}}$ , and the feature dimension $d$ is selected from ${96,144,192,256}$ . The dropout rate is tuned within ${0.1,0.2,0.3,0.4,0.5}$ , the prompt length $l_{p}$ is searched over ${2,3,4,6,8}$ , and the memory bank capacity $N_{b}$ is selected from ${256,512,1024,2048}$ . In addition, the top- $k$ ratio $\rho$ is tuned within ${0.1,0.2,\dots,0.7}$ , while the label penalty weight $\alpha$ and loss balancing weight $\lambda$ are tuned within ${0.1,0.2,\dots,1.0}$ . The AVIB trade-off coefficient $\beta$ is searched over ${1\mathrm{e}{-2},1\mathrm{e}{-3},1\mathrm{e}{-4},1\mathrm{e}{-5}}$ . The final model is selected based on the configuration that achieves the lowest MAE on the validation set.

Notably, for a fair comparison, we re-implement C-MIB [33], ITHP [51], KAN-MCP [30], OMIB [49], QMF [65], PML [14], and HME [70] using the same feature extraction strategy as RCE. The implementations are based on their publicly available official codebases, and the hyper-parameter search is conducted according to the parameter ranges reported in their original papers using thirty random search trials on the validation set to determine the optimal configuration. For the remaining baselines, unless otherwise specified, the reported results are directly taken from the corresponding original papers.

For the MHD task, the results of Self-MM [58], FDMER [55], and DMD [23] are taken from the SuCI [53] paper. For the MSD task, the results of MAG-XLNet [40] and BBFN [8] are taken from the MuLOT [38] paper. Results for all other baselines are directly taken from their corresponding original papers.

Table 5: Hyper-parameter settings of RCE across different datasets.

Hyper-parameter	CMU-MOSI	CMU-MOSEI	UR-FUNNY	MUStARD
Batch Size	48	128	256	32
Epochs	50	10	10	20
Warm-up	✓	✓	✓	✓
Initial Learning Rate	$2\times 10^{-5}$	$3\times 10^{-5}$	$1\times 10^{-5}$	$3\times 10^{-6}$
Optimizer	AdamW	AdamW	AdamW	AdamW
Dropout Rate	0.4	0.4	0.1	0.1
Feature Dimension $d$	256	192	192	144
Prompt Length $l_{p}$	4	6	3	8
Memory Bank Capacity $N_{b}$	1024	2048	1024	1024
Top- $k$ Ratio $\rho$	0.6	0.4	0.3	0.5
Label Penalty Weight $\alpha$	0.4	0.1	0.3	0.3
AVIB Trade-off Coefficient $\beta$	1e-3	1e-3	1e-4	1e-2
Loss Balancing Weight $\lambda$	0.7	1.0	0.6	0.4

D.5 Evaluation Metrics

We evaluate the model’s performance on the MSA task using a set of well-established metrics, reported for both CMU-MOSI and CMU-MOSEI datasets. For interpretability, classification results are presented as percentages. These metrics are calculated as follows:

Seven-category Classification Accuracy (Acc7): Measures the model’s ability to predict fine-grained sentiment categories by dividing the sentiment score range ( $-3$ to $3$ ) into seven equal intervals. The metric is defined as:

\mathrm{Acc7}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\big(\hat{c}_{i}=c_{i}\big),

(29)

where $c_{i}$ and $\hat{c}_{i}$ denote the ground-truth and predicted categories of sample $i$ , respectively, and $\mathbf{1}(\cdot)$ is the indicator function. Higher values indicate better fine-grained sentiment classification.

Binary Classification Accuracy (Acc2): Following prior work [51, 30], we report binary sentiment classification results by distinguishing negative ( $<0$ ) and positive ( $>0$ ) samples, excluding neutral cases. The metric is formulated as:

\mathrm{Acc2}=\frac{TP+TN}{TP+TN+FP+FN},

(30)

where $TP$ , $TN$ , $FP$ , and $FN$ denote true positives, true negatives, false positives, and false negatives, respectively.

Weighted F1-score (F1): Computes the harmonic mean of precision and recall while considering class-specific weights to mitigate imbalance. It is formulated as:

\mathrm{F1}=2\cdot\frac{Precision\cdot Recall}{Precision+Recall},

(31)

where $\mathrm{Precision}=\tfrac{TP}{TP+FP}$ and $\mathrm{Recall}=\tfrac{TP}{TP+FN}$ .

Mean Absolute Error (MAE): Represents the average magnitude of prediction errors with respect to the ground-truth sentiment scores. It directly corresponds to the original sentiment scale, making it both intuitive and informative:

\mathrm{MAE}(\hat{y},y)=\frac{1}{n}\sum_{i=1}^{n}|\hat{y}_{i}-y_{i}|,

(32)

where $y_{i}$ is the true label, $\hat{y}_{i}$ is the predicted value, and $n$ is the total number of predictions.

Pearson Correlation Coefficient (Corr): Quantifies the strength and direction of the linear relationship between predicted and true sentiment scores:

\mathrm{Corr}(x,y)=\frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}\sqrt{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}},

(33)

where $x_{i}$ and $y_{i}$ denote predicted and ground-truth values, respectively, and $\bar{x},\bar{y}$ are their means.

For the MHD and MSD tasks, following prior work [53, 15, 34, 24], we report binary accuracy only, which measures the model’s capability to differentiate between humorous and non-humorous, as well as sarcastic and non-sarcastic, instances.

Appendix E More Experimental Results

Table 6: Performance comparison of RCE and baseline methods under fixed missing protocol.

Testing Condition: {t}
Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
C-MIB [33]	45.69	87.33	87.28	0.644	0.842	51.76	85.66	85.62	0.550	0.771
ITHP [51]	43.50	86.72	86.71	0.678	0.838	52.40	86.96	86.94	0.541	0.791
KAN-MCP [30]	45.69	88.24	88.21	0.635	0.851	53.87	86.80	86.80	0.531	0.774
OMIB [49]	49.49	87.63	87.51	0.632	0.844	53.37	87.18	87.14	0.529	0.786
QMF [65]	42.92	87.63	87.65	0.691	0.846	51.32	85.75	85.31	0.547	0.789
PML [14]	45.99	87.94	87.86	0.657	0.837	53.84	86.99	86.94	0.526	0.787
HME [70]	41.17	88.55	88.49	0.710	0.856	54.08	87.15	87.15	0.518	0.787
RCE (Ours)	46.72	89.01	88.94	0.615	0.860	55.18	86.96	86.93	0.505	0.795
Testing Condition: {t, a}
C-MIB [33]	43.94	84.73	84.74	0.725	0.804	51.24	85.50	85.48	0.560	0.760
ITHP [51]	44.53	86.87	86.87	0.689	0.838	53.54	86.60	86.71	0.541	0.787
KAN-MCP [30]	47.15	86.87	86.84	0.623	0.853	53.46	86.85	86.90	0.526	0.788
OMIB [49]	48.32	88.40	88.32	0.633	0.844	53.37	87.15	87.11	0.529	0.786
QMF [65]	45.69	88.09	88.04	0.653	0.846	52.10	86.93	86.72	0.532	0.790
PML [14]	46.28	87.94	87.86	0.656	0.837	53.50	86.99	86.92	0.530	0.787
HME [70]	45.99	88.70	88.70	0.620	0.856	54.08	87.13	86.95	0.519	0.793
RCE (Ours)	48.61	89.16	89.11	0.596	0.860	55.27	87.40	87.44	0.502	0.800
Testing Condition: {t, v}
C-MIB [33]	47.30	87.63	87.62	0.651	0.840	51.30	86.33	86.31	0.561	0.768
ITHP [51]	44.09	84.89	84.90	0.722	0.816	52.25	86.35	86.46	0.550	0.792
KAN-MCP [30]	45.55	87.63	87.60	0.668	0.834	53.03	86.57	86.66	0.528	0.784
OMIB [49]	48.61	88.09	88.00	0.633	0.844	53.52	87.24	87.21	0.527	0.786
QMF [65]	45.11	87.79	87.80	0.665	0.847	52.94	86.99	86.77	0.529	0.790
PML [14]	46.42	87.94	87.86	0.656	0.837	53.65	87.15	87.15	0.528	0.786
HME [70]	44.09	89.16	89.14	0.653	0.856	54.06	85.47	85.64	0.519	0.792
RCE (Ours)	48.47	89.01	88.93	0.592	0.861	54.92	87.49	87.39	0.503	0.800

E.1 Results under Fixed Missing Protocols

Under the fixed missing protocol, we further evaluate RCE against a wide range of competitive missing-modality methods, including C-MIB [33], ITHP [51], KAN-MCP [30], OMIB [49], QMF [65], PML [14], and HME [70]. As reported in Table 6, RCE achieves the best overall performance across most fixed modality-availability settings. Specifically, when only the textual modality is available, RCE obtains the best results on four out of five metrics on CMU-MOSI and three out of five metrics on CMU-MOSEI, showing that the proposed method can still produce reliable predictions under highly sparse modality conditions. When both text and audio are available, RCE consistently outperforms all baselines on both datasets across all evaluation metrics, demonstrating its strong ability to exploit complementary acoustic cues in addition to the dominant textual modality.

For the setting where text and vision are available, RCE also exhibits highly competitive and stable performance. On CMU-MOSEI, it achieves the best results on all five metrics, while on CMU-MOSI it obtains the lowest $MAE$ and highest correlation, with $Acc2$ and $F1$ remaining very close to the best-performing baseline. These results suggest that RCE is not limited to a specific modality combination, but can adaptively benefit from different observed modalities. Overall, under the fixed missing protocol, RCE achieves the best performance on 24 out of 30 metric comparisons, validating its robustness to diverse deterministic missing patterns. The improvements can be attributed to the reliability-aware modality enhancement mechanism, which retrieves informative cross-sample cues from the memory bank, and the multilevel fusion strategy, which adaptively integrates unimodal, enhanced, and cross-modal representations according to modality reliability.

E.2 Results under Noisy Modalities

In addition to missing-modality evaluation, we conduct noise robustness experiments to further examine whether RCE can handle corrupted but still observable modalities.

Table 7: Comparison under different Gaussian noise intensities on CMU-MOSI and CMU-MOSEI.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
Gaussian Noise Intensity: 1.0
C-MIB [33]	42.34	85.65	85.68	0.750	0.782	51.07	84.50	84.65	0.560	0.766
ITHP [51]	45.55	86.11	86.10	0.697	0.822	52.10	84.03	84.27	0.564	0.781
KAN-MCP [30]	45.99	85.80	85.76	0.710	0.805	52.04	86.88	86.84	0.543	0.775
OMIB [49]	42.34	83.51	83.51	0.751	0.785	50.36	84.42	84.63	0.562	0.770
QMF [65]	44.96	86.41	86.36	0.691	0.809	50.81	85.86	85.78	0.554	0.762
PML [14]	44.38	86.41	86.42	0.732	0.795	52.62	85.94	85.85	0.547	0.766
HME [70]	45.69	86.56	86.55	0.677	0.813	54.08	85.88	85.94	0.527	0.778
RCE (Ours)	46.42	87.02	86.88	0.676	0.815	54.12	86.80	86.76	0.518	0.783
Gaussian Noise Intensity: 5.0
C-MIB [33]	28.47	76.18	76.27	1.063	0.578	48.29	82.96	82.75	0.616	0.695
ITHP [51]	37.08	79.24	79.34	0.908	0.683	49.06	82.60	82.55	0.602	0.717
KAN-MCP [30]	39.56	77.71	77.84	0.923	0.677	50.23	82.54	82.44	0.596	0.685
OMIB [49]	39.42	79.08	79.20	0.859	0.716	50.08	82.68	82.20	0.588	0.706
QMF [65]	33.43	77.56	77.58	0.948	0.678	46.56	83.45	82.97	0.615	0.716
PML [14]	36.93	79.24	79.32	0.880	0.713	51.30	83.20	83.18	0.581	0.715
HME [70]	42.77	80.31	80.37	0.801	0.719	51.54	80.19	80.50	0.585	0.721
RCE (Ours)	38.83	80.76	80.74	0.886	0.689	51.00	83.51	83.36	0.577	0.715
Gaussian Noise Intensity: 10.0
C-MIB [33]	15.62	41.98	25.28	1.466	-0.010	45.55	73.54	72.01	0.707	0.546
ITHP [51]	28.91	69.31	69.14	1.081	0.562	46.00	74.89	73.27	0.702	0.567
KAN-MCP [30]	14.74	42.29	25.14	1.493	-0.004	47.02	75.11	73.85	0.683	0.567
OMIB [49]	15.47	43.05	27.04	1.462	0.062	45.75	73.62	71.95	0.705	0.535
QMF [65]	26.28	68.09	68.23	1.107	0.538	46.87	74.34	71.80	0.688	0.561
PML [14]	15.33	44.12	29.96	1.441	0.158	47.36	75.39	73.77	0.687	0.555
HME [70]	32.26	67.48	66.45	1.039	0.581	47.79	74.72	73.26	0.678	0.559
RCE (Ours)	32.85	72.82	72.61	0.983	0.625	48.09	75.41	72.50	0.678	0.560

Noise setting.

Following the noise evaluation protocols used in EAU [6] and QMF [65], we further examine the robustness of RCE under corrupted multimodal inputs. Since the original noise operations are mainly designed for image-like data, we adapt them to multimodal sentiment analysis with text tokens and continuous acoustic/visual feature sequences. For the textual modality, noise is directly applied to the encoded input_ids by replacing selected token ids, which minimally changes the preprocessing pipeline. Textual noise can be applied to the train, validation, and test splits. For acoustic and visual modalities, we adapt Gaussian and Salt-pepper noise to continuous feature sequences. Specifically, Gaussian noise is not generated using image-style amplitude scaling; instead, the perturbation scale is determined by the standard deviation of valid features in each sample multiplied by $\texttt{noise}/10$ . For Salt-pepper noise, rather than replacing pixels with 0 or 255, we replace selected valid feature elements with the minimum or maximum value of the corresponding sample. Acoustic and visual noise is applied only to the test split to evaluate robustness under corrupted inference-time observations. The noise intensity parameter noise ranges from 0 to 10, where larger values indicate stronger corruption. For text, after a sample-level trigger with probability 0.5, each token is replaced with probability $\texttt{noise}/10$ . For acoustic and visual Gaussian noise, a sample-level trigger with probability 0.5 determines whether noise is added, and the perturbation magnitude is proportional to $\texttt{noise}/10$ . For acoustic and visual Salt-pepper noise, the corruption process is first triggered with probability 0.5, and the corruption strength is further controlled by $\texttt{noise}/10$ .

Table 8: Comparison under different Salt-pepper noise intensities on CMU-MOSI and CMU-MOSEI.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
Salt-pepper Noise Intensity: 1.0
C-MIB	42.04	85.80	85.80	0.751	0.786	51.93	85.52	85.41	0.555	0.765
ITHP	15.47	42.29	25.14	1.471	-0.030	52.49	85.75	85.71	0.552	0.777
KAN-MCP	48.47	87.02	87.02	0.657	0.835	52.98	86.85	86.90	0.531	0.777
OMIB	37.52	78.32	78.43	0.903	0.691	53.24	86.16	86.24	0.533	0.776
QMF	43.65	84.58	84.60	0.735	0.789	48.24	83.65	83.56	0.602	0.706
PML	43.50	85.95	85.89	0.721	0.802	52.23	85.14	85.03	0.545	0.763
HME	45.99	86.87	86.88	0.657	0.830	53.46	86.63	86.55	0.524	0.776
RCE (Ours)	48.47	87.79	87.84	0.634	0.829	54.04	86.99	86.99	0.527	0.779
Salt-pepper Noise Intensity: 5.0
C-MIB	39.42	78.63	78.75	0.927	0.661	48.07	83.40	83.34	0.610	0.715
ITHP	14.31	46.26	36.20	1.472	0.118	49.97	83.07	83.21	0.601	0.722
KAN-MCP	31.68	76.64	76.73	1.002	0.646	51.60	84.25	84.14	0.567	0.728
OMIB	41.61	81.07	81.08	0.834	0.711	51.04	83.95	83.89	0.579	0.727
QMF	38.54	80.15	80.16	0.876	0.695	48.52	82.65	82.41	0.606	0.707
PML	35.18	77.71	77.84	0.950	0.667	51.41	83.18	82.96	0.581	0.715
HME	26.86	66.56	66.33	1.266	0.464	51.93	83.43	83.45	0.574	0.714
RCE (Ours)	40.29	81.07	80.62	0.820	0.723	50.89	82.27	82.14	0.584	0.705
Salt-pepper Noise Intensity: 10.0
C-MIB	31.09	67.94	67.99	1.059	0.579	46.59	73.34	71.85	0.694	0.558
ITHP	26.57	61.83	60.35	1.157	0.528	45.68	75.17	74.36	0.702	0.572
KAN-MCP	31.97	67.18	66.17	1.084	0.603	48.24	75.08	72.80	0.675	0.568
OMIB	28.61	65.34	65.10	1.096	0.538	46.87	75.11	73.48	0.688	0.546
QMF	32.85	65.04	64.01	1.055	0.583	46.22	74.64	71.87	0.698	0.530
PML	29.34	68.40	67.71	1.048	0.582	47.38	74.39	72.68	0.678	0.569
HME	31.97	67.48	67.63	1.045	0.575	46.20	74.56	72.58	0.690	0.546
RCE (Ours)	33.28	73.44	72.35	1.019	0.586	48.26	75.33	73.27	0.668	0.576

Results under Gaussian noise.

Table 7 reports the comparison under different Gaussian noise intensities. Overall, RCE demonstrates strong robustness across both datasets, especially under moderate and severe Gaussian perturbations. When the noise intensity is 1.0, RCE achieves the best results on most metrics. On CMU-MOSI, RCE obtains the best $Acc7$ , $Acc2$ , $F1$ , and $MAE$ , while its $Corr$ remains very close to the best result. On CMU-MOSEI, RCE achieves the best $Acc7$ , $MAE$ , and $Corr$ , and remains highly competitive on $Acc2$ and $F1$ . These results indicate that when the perturbation is mild, RCE can effectively preserve useful sentiment cues while suppressing unreliable corrupted information.

As the Gaussian noise intensity increases to 5.0, all methods experience performance degradation, but RCE still maintains competitive performance. On CMU-MOSI, RCE achieves the best $Acc2$ and $F1$ , showing that it preserves binary sentiment discrimination under stronger feature perturbation, although HME [70] obtains better $Acc7$ , $MAE$ , and $Corr$ in this setting. On CMU-MOSEI, RCE achieves the best $Acc2$ , $F1$ , and $MAE$ , demonstrating stable performance on the larger-scale dataset. Under the most severe Gaussian noise intensity of 10.0, RCE shows clearer advantages. On CMU-MOSI, it achieves the best performance across all five metrics, improving $Acc2$ and $F1$ notably compared with the strongest baselines. On CMU-MOSEI, RCE also obtains the best $Acc7$ , $Acc2$ , and $MAE$ , while remaining close to the best $Corr$ . These results suggest that the reliability-aware enhancement and multilevel fusion mechanisms help RCE maintain robust predictions when acoustic and visual observations are heavily corrupted.

Results under Salt-pepper noise.

Table 8 presents the results under Salt-pepper noise with different corruption intensities. Compared with Gaussian noise, Salt-pepper noise introduces more abrupt feature-level corruption by replacing valid acoustic and visual elements with extreme values from the same sample, making it a challenging test of robustness to outlier-like perturbations [65]. When the noise intensity is 1.0, RCE achieves the best $Acc7$ , $Acc2$ , $F1$ , and $MAE$ on CMU-MOSI, and the best $Acc7$ , $Acc2$ , $F1$ , and $Corr$ on CMU-MOSEI. This shows that RCE is effective under mild sparse corruption and can still extract reliable complementary information from noisy continuous features.

When the Salt-pepper noise intensity increases to 5.0, the performance of all methods decreases more noticeably. On CMU-MOSI, RCE achieves the best $Acc2$ , $F1$ , $MAE$ , and $Corr$ , and remains close to the best $Acc7$ . This indicates that RCE is particularly effective at preserving overall sentiment discrimination and regression quality under medium-level outlier corruption. On CMU-MOSEI, however, KAN-MCP [30] and HME [70] obtain better results on several metrics, while RCE becomes less competitive. This suggests that medium Salt-pepper corruption may introduce feature-level outliers that affect RCE’s retrieved or fused representations on the larger dataset. Nevertheless, under the strongest Salt-pepper noise intensity of 10.0, RCE again shows robust overall performance. It achieves the best $Acc7$ , $Acc2$ , $F1$ , and $MAE$ on CMU-MOSI, and the best $Acc7$ , $Acc2$ , $MAE$ , and $Corr$ on CMU-MOSEI. These results demonstrate that RCE is generally resilient to severe outlier-style corruption. The advantage is especially clear under high noise intensity, where reliability-aware fusion can reduce the influence of unreliable modality information and better exploit stable cues from the remaining useful representations.

E.3 Detailed Ablation Study

Table 9: Ablation details.

Method	CMU-MOSI					CMU-MOSEI
Method	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7% $\uparrow$	Acc2% $\uparrow$	F1% $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
w/o AVIB	48.03	87.79	87.77	0.646	0.842	54.60	86.35	86.47	0.518	0.785
w/o RME	50.36	88.09	88.02	0.610	0.857	54.10	87.04	87.04	0.519	0.794
w/o MB	48.03	88.24	88.20	0.616	0.853	53.50	87.18	87.15	0.511	0.794
w/o LP	49.34	88.70	88.62	0.603	0.859	56.06	86.60	86.67	0.505	0.795
w/o CP	47.30	87.94	87.81	0.610	0.856	55.14	87.18	87.17	0.504	0.798
w/o HMG	46.72	87.63	87.56	0.637	0.841	54.77	86.82	86.76	0.506	0.794
w/o ES	45.69	87.94	87.82	0.654	0.842	55.33	87.51	87.42	0.502	0.797
w/o RF	48.47	87.33	87.20	0.626	0.851	55.52	87.51	87.47	0.507	0.796
RCE	50.22	89.31	89.26	0.589	0.861	55.44	87.29	87.33	0.503	0.799

To further examine the contribution of each component in RCE, we report detailed ablation results in Table 9. Specifically, ‘w/o AVIB’ removes the Adaptive Variational Information Bottleneck, where modality representations are directly used without vMF-based uncertainty estimation, latent sampling, and KL regularization. ‘w/o RME’ removes the Reliability-aware Modality Enhancement module by directly setting the enhanced representation as the original modality representation. ‘w/o MB’ keeps the enhancement mechanism but disables the memory bank, using only mini-batch samples for neighbor retrieval. ‘w/o LP’ removes the label-distance penalty in neighbor weighting, while ‘w/o CP’ removes the confidence promotion term based on $\kappa$ . ‘w/o HMG’ removes the hyper-modality generation module and directly uses modality-level representations for fusion. ‘w/o ES’ removes the shared sample-level representation, and ‘w/o RF’ disables reliability-aware fusion by replacing $\kappa$ -modulated fusion with vanilla attention weights.

Overall, the complete RCE achieves the best or competitive results on most metrics, especially on CMU-MOSI, where it obtains the best $Acc2$ , $F1$ , $MAE$ , and $Corr$ . This indicates that the proposed components are complementary and jointly improve both classification accuracy and regression quality. On CMU-MOSI, removing ES leads to the largest degradation in $Acc7$ and $MAE$ , with $Acc7$ dropping from 50.22 to 45.69 and $MAE$ increasing from 0.589 to 0.654, showing that the shared sample-level representation is important for capturing global multimodal context. Removing HMG also causes a clear performance drop, especially in $Acc7$ and $MAE$ , indicating that hyper-modality generation contributes to robust cross-modal representation learning. In addition, removing AVIB consistently weakens all metrics on CMU-MOSI, demonstrating that vMF-based uncertainty estimation and adaptive compression help reduce noisy and redundant modality information. The memory bank also plays an important role: compared with RCE, ‘w/o MB’ obtains lower $Acc7$ , $MAE$ , and $Corr$ , suggesting that historical reliable samples provide more stable neighborhood information than mini-batch-only retrieval.

For the internal design of RME, the effects of label penalty and confidence promotion show different tendencies. On CMU-MOSI, ‘w/o LP’ achieves relatively strong classification results but is still inferior to RCE in $Acc2$ , $F1$ , $MAE$ , and $Corr$ , suggesting that label consistency is beneficial for stable prediction. In contrast, removing CP causes a larger decline in $Acc7$ , indicating that confidence-aware neighbor weighting helps suppress unreliable enhancement samples. On CMU-MOSEI, several ablated variants obtain slightly better results on individual metrics, such as ‘w/o LP’ on $Acc7$ and ‘w/o ES’ or ‘w/o RF’ on $Acc2$ / $F1$ . This may be because CMU-MOSEI contains more training samples and provides more stable multimodal statistics, making some auxiliary constraints less critical for coarse-grained classification. Nevertheless, the complete RCE still achieves the best $Corr$ and nearly the best $MAE$ , showing better overall regression consistency and prediction calibration. In particular, although ‘w/o RF’ slightly improves $Acc2$ and $F1$ on CMU-MOSEI, it performs worse than RCE in $MAE$ and $Corr$ , indicating that reliability-aware fusion is more helpful for fine-grained sentiment intensity estimation. Overall, these results show that AVIB, RME with memory-based retrieval, HMG, ES, and RF contribute from different aspects, and their combination yields the most balanced performance across datasets and evaluation metrics.

E.4 Negative Enhancement Analysis

Table 10: Detailed training-stage negative enhancement statistics on CMU-MOSI and CMU-MOSEI.

Dataset	Method	Modality	Samples	Negative Enhancement Samples	Negative Enhancement Rate
CMU-MOSI	HME	Text	61,400	28,420	46.29%
		Audio	61,400	28,797	46.90%
		Vision	61,400	28,797	46.90%
	RCE	Text	61,400	1,595	2.60%
		Audio	61,400	3,621	5.90%
		Vision	61,400	3,976	6.48%
CMU-MOSEI	HME	Text	127,350	48,904	38.40%
		Audio	127,350	49,500	38.87%
		Vision	127,350	49,500	38.87%
	RCE	Text	127,350	13,458	10.57%
		Audio	127,350	39,798	31.25%
		Vision	127,350	33,218	26.08%

We further analyze the negative enhancement phenomenon during training. For each sample, we compare the sentiment direction of its ground-truth label with the sentiment direction of the samples used for enhancement. Specifically, for HME, the retrieved samples whose similarities exceed the threshold are averaged to obtain the enhancement source. For RCE, we use the final weighted neighbors after top- $k$ retrieval, label-distance penalty, and reliability-aware weighting. If the weighted average label of the enhancement sources has the opposite sentiment polarity to the current sample, this case is counted as negative enhancement.

The reported rate is computed as the number of negative enhancement cases divided by the number of valid enhanced samples in the training stage. In our logs, CMU-MOSI contains 61,400 valid training enhancement cases over all epochs, corresponding to 1,228 valid training samples per epoch, while CMU-MOSEI contains 127,350 valid training enhancement cases, corresponding to 12,735 valid training samples per epoch. Neutral samples are not counted since their sentiment polarity is ambiguous.

As shown in Table 3, HME suffers from severe negative enhancement. On CMU-MOSI, nearly 46% of training samples are negatively enhanced across all modalities. In contrast, RCE reduces the negative enhancement rates to 2.60%, 5.90%, and 6.48% for text, audio, and vision, respectively. This demonstrates that the proposed label-aware and reliability-aware enhancement mechanism effectively suppresses sentiment-conflicting information from neighboring samples.

On CMU-MOSEI, RCE also consistently reduces negative enhancement compared with HME. The reduction is most significant in the text modality, from 38.40% to 10.57%. However, the decrease for the audio modality is relatively smaller. This may be because acoustic cues in CMU-MOSEI are more ambiguous and diverse: samples with similar acoustic patterns may still express different sentiment polarities. As a result, even after reliability-aware weighting, some sentiment-conflicting audio neighbors can still receive non-negligible weights, leading to a higher remaining negative enhancement rate.

(a) AVIB Trade-off Coefficient

\beta

(b) Memory Bank Capacity

N_{b}

Figure 5: Parameter sensitivity analysis of

\beta

and

N_{b}

on the CMU-MOSI dataset.

(a) Label Penalty Weight

\alpha

(b) Loss Balancing Weight

\lambda

k

Ratio

\rho

Figure 6: Parameter sensitivity analysis of

\alpha

\lambda

, and

\rho

on the CMU-MOSI dataset.

E.5 Parameter Sensitivity Analysis

We analyze the sensitivity of RCE to five important hyperparameters, including the AVIB trade-off coefficient $\beta$ , the memory bank capacity $N_{b}$ , the label penalty weight $\alpha$ , the loss balancing weight $\lambda$ , and the top- $k$ ratio $\rho$ . The results on CMU-MOSI and CMU-MOSEI are shown in Fig. 5, Fig. 6, Fig. 7, and Fig. 8.

(1) AVIB trade-off coefficient $\beta$ . The coefficient $\beta$ controls the strength of the information bottleneck regularization in AVIB. On CMU-MOSI, RCE obtains competitive performance under a wide range of $\beta$ values, with the best $Acc7$ achieved around $\beta=1e$ - $1$ and the best $MAE$ obtained at $\beta=1e$ - $3$ . On CMU-MOSEI, the model is also relatively stable, and $\beta=1e$ - $3$ achieves both strong $Acc7$ and the lowest $MAE$ . These results indicate that a moderate bottleneck constraint helps remove noisy or redundant information, while overly large or overly small values may weaken either task-relevant information preservation or regularization effectiveness. Therefore, we adopt $\beta=1e$ - $3$ as the default setting.

(a) AVIB Trade-off Coefficient

\beta

(b) Memory Bank Capacity

N_{b}

Figure 7: Parameter sensitivity analysis of

\beta

and

N_{b}

on the CMU-MOSEI dataset.

(a) Label Penalty Weight

\alpha

(b) Loss Balancing Weight

\lambda

k

Ratio

\rho

Figure 8: Parameter sensitivity analysis of

\alpha

\lambda

, and

\rho

on the CMU-MOSEI dataset.

(2) Memory bank capacity $N_{b}$ . The memory bank capacity determines the number of historical samples available for reliability-aware modality enhancement. On both datasets, very small memory banks provide limited neighborhood information and lead to inferior performance. As $N_{b}$ increases, the performance generally improves, showing that a larger candidate pool can provide more stable and informative enhancement samples. On CMU-MOSI, $N_{b}=512$ achieves the best $Acc7$ , while $N_{b}=1024$ obtains the lowest $MAE$ . On CMU-MOSEI, $N_{b}=512$ also achieves the best $Acc7$ , and $N_{b}=1024$ or $2048$ gives the best $MAE$ . This suggests that a moderate-to-large memory bank is beneficial, but continuously increasing the capacity does not always bring further gains.

(3) Label penalty weight $\alpha$ . The label penalty weight $\alpha$ controls how strongly label discrepancy is penalized during neighbor weighting. On CMU-MOSI, the model performs best around $\alpha=0.4$ , where both $Acc7$ and $MAE$ achieve the best results. When $\alpha$ becomes too large, performance drops clearly, suggesting that overly strong label constraints may suppress useful complementary samples. On CMU-MOSEI, the performance is relatively stable for most $\alpha$ values, although $\alpha=0.1$ gives the best result. This may be because CMU-MOSEI contains more samples and more diverse sentiment expressions, making a mild label penalty sufficient for filtering conflicting neighbors.

(4) Loss balancing weight $\lambda$ . The weight $\lambda$ balances the main prediction loss and the AVIB-related auxiliary objective. On CMU-MOSI, RCE achieves the best performance when $\lambda=0.7$ , indicating that a proper contribution from AVIB is important for robust sentiment prediction. On CMU-MOSEI, the model remains stable across different $\lambda$ values, with the best $Acc7$ around $\lambda=0.4$ and the lowest $MAE$ around $\lambda=1.0$ . Overall, moderate values of $\lambda$ provide a good balance between prediction accuracy and representation regularization.

(5) Top- $k$ ratio $\rho$ . The top- $k$ ratio $\rho$ controls the proportion of retrieved neighbors used for modality enhancement. If $\rho$ is too small, the model may not obtain sufficient complementary information. If $\rho$ is too large, less relevant or sentiment-conflicting samples may be introduced. On CMU-MOSI, the best $Acc7$ is obtained around $\rho=0.5$ , while the lowest $MAE$ is achieved at $\rho=0.6$ . On CMU-MOSEI, $\rho=0.4$ achieves the best overall performance. These results show that selecting a moderate number of reliable neighbors is more effective than using either too few or too many candidates.

Overall, RCE is robust to these hyperparameters on both datasets. Although different datasets prefer slightly different values, the performance remains stable within moderate ranges. This indicates that the proposed reliability-aware enhancement and multilevel fusion mechanisms are not overly sensitive to hyperparameter choices.

E.6 Testing Stability Analysis

Table 11 reports the testing stability of RCE on the CMU-MOSI and CMU-MOSEI datasets. For each dataset, we run RCE with five different random seeds and compute the mean performance together with the 95% confidence interval for each metric. Specifically, the confidence interval is calculated as $\bar{x}\pm t_{0.975,4}\frac{s}{\sqrt{5}}$ , where $\bar{x}$ and $s$ denote the mean and standard deviation over the five runs, respectively. The narrow intervals on both CMU-MOSI and CMU-MOSEI show that RCE has small performance variation across different random initializations, indicating stable and reproducible results.

Table 11: Testing stability with five random seeds under the complete-modality setting (95% confidence intervals).

Dataset	$Acc7\%$	$Acc2\%$	$F1\%$	$MAE$	$Corr$
CMU-MOSI	$49.81\pm 0.56$	$88.98\pm 0.97$	$88.93\pm 0.97$	$0.598\pm 0.009$	$0.859\pm 0.004$
CMU-MOSEI	$55.14\pm 0.37$	$87.25\pm 0.35$	$87.26\pm 0.31$	$0.510\pm 0.006$	$0.792\pm 0.006$

E.7 Model Complexity Analysis

We compare the computational complexity of RCE with state-of-the-art methods in terms of FLOPs, memory usage, parameter count, and training time. As shown in Table 12, RCE introduces moderate additional cost due to the adaptive variational information bottleneck, memory bank-based reliability-aware modality enhancement, and multilevel fusion modules. Specifically, RCE requires 12.660G FLOPs and 9.49GB memory, which are comparable to most baselines and only slightly higher than HME [70]. Although RCE has the largest parameter count, its training time is 8.54s per epoch, lower than both HME [70] and KAN-MCP [30]. This indicates that the proposed enhancement and fusion mechanisms do not incur prohibitive overhead. Overall, RCE improves robustness and prediction performance while maintaining acceptable computational complexity.

Table 12: Comparison of model complexity on the CMU-MOSI dataset with batch size 48.

Method	FLOPs	Memory	Parameters	Per-Epoch Training Time
C-MIB [33]	11.773 G	8.48 GB	190.93 M	2.93 s
ITHP [51]	11.523 G	8.36 GB	184.88 M	3.43 s
KAN-MCP [30]	11.501 G	13.36 GB	184.76 M	10.62 s
OMIB [49]	11.615 G	8.10 GB	189.05 M	3.70 s
QMF [65]	12.529 G	8.41 GB	196.47 M	4.09 s
PML [14]	12.144 G	8.42 GB	188.27 M	3.99 s
HME [70]	12.407 G	9.02 GB	228.53 M	10.30 s
RCE (Ours)	12.660 G	9.49 GB	251.77 M	8.54 s

E.8 Theoretical Complexity Analysis

We further analyze the computational complexity of the key components in RCE. Let $B$ denote the batch size, $d$ the hidden dimension, $M$ the number of modalities, $N_{b}$ the memory bank capacity, and $\rho$ the top- $k$ retrieval ratio. The AVIB module mainly consists of vMF parameter estimation, latent sampling, and auxiliary prediction, with a complexity of $O(MBd^{2})$ . The RME module retrieves reliable neighbors from the modality-specific memory bank. Its similarity computation requires $O(MBN_{b}d)$ , and weighted aggregation over selected neighbors requires $O(MB\rho N_{b}d)$ . The hyper-modality generation and multilevel fusion modules are based on lightweight latent attention and fusion operations, with an approximate complexity of $O(MBd^{2})$ since the number of latent prompts and fusion tokens is small and fixed. Therefore, the overall additional complexity of RCE is mainly dominated by memory-bank retrieval, i.e., $O(MBN_{b}d)$ , while other components scale linearly with respect to the batch size and number of modalities. Since $M$ and $N_{b}$ are fixed hyperparameters in practice, RCE remains computationally tractable. This is also consistent with Table 12, where RCE achieves stronger robustness with only moderate additional cost compared to HME and other missing-modality baselines.