M$^3$amba: CLIP-driven Mamba Model for Multi-modal Remote Sensing Classification
Extracted source text · arXiv 2503.06446
\documentclass[lettersize,journal]{IEEEtran}
\usepackage{amsmath,amsfonts}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{array}
\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage{textcomp}
\usepackage{stfloats}
\usepackage{url}
\usepackage{verbatim}
\usepackage{graphicx}
\usepackage{cite}
\usepackage{multirow} % 一般用以设计表格，将所行合并
\usepackage{multicol} % 合并多列
\usepackage{multirow} % 合并多行
\usepackage{float} % 图片浮动
\usepackage{makecell} % 三线表-竖线
\usepackage{booktabs} % 三线表-短细横线
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{xcolor}
\hyphenation{op-tical net-works semi-conduc-tor IEEE-Xplore}
% updated with editorial comments 8/9/2021
\begin{document}
\title{M$^3$amba: CLIP-driven Mamba Model for Multi-modal Remote Sensing Classification}
\author{Mingxiang Cao,
Weiying Xie,~\IEEEmembership{Senior Member,~IEEE},
Xin Zhang,
Jiaqing Zhang,
Kai Jiang,
Jie Lei,~\IEEEmembership{Member,~IEEE},
% Zhenman Fang,~\IEEEmembership{Member,~IEEE},
Yunsong Li,~\IEEEmembership{Member,~IEEE}
% <-this % stops a space
\thanks{This work was supported in part by the National Natural Science Foundation of China under Grant 62121001, Grant 62322117 and Grant 62371365, 62071360, in part by the National Key R\&D Program of China (No. 2023YFE0208100), and supported by the Innovation Fund of Xidian University under Grant YJSJ25007. (Corresponding~author: Weiying Xie.)
Mingxiang Cao, Weiying Xie, Xin Zhang, Jiaqing Zhang, Kai Jiang, and Yunsong Li are with the State Key
Laboratory of Integrated Services Networks, Xidian University, Xi'an 710071,
China (e-mail: mingxiangcao@stu.xidian.edu.cn; wyxie@xidian.edu.cn; xinzhang\_xd@163.com; jqzhang\_2@stu.xidian.edu.cn; xdjiangkai@foxmail.com; ysli@mail.xidian.edu.cn).
Jie Lei is at the School of Electrical and Data Engineering at the University of Technology Sydney (e-mail: jie.lei@uts.edu.au).}}
%\author{IEEE Publication Technology,~\IEEEmembership{Staff,~IEEE,}
% % <-this % stops a space
%\thanks{This paper was produced by the IEEE Publication Technology Group. They are in Piscataway, NJ.}% <-this % stops a space
%\thanks{Manuscript received April 19, 2021; revised August 16, 2021.}}
% The paper headers
\markboth{IEEE Transactions on Circuits and Systems for Video Technology}%
{Shell \MakeLowercase{\textit{et al.}}: A Sample Article Using IEEEtran.cls for IEEE Journals}
%\IEEEpubid{0000--0000/00\$00.00~\copyright~2021 IEEE}
% Remember, if you use this you must call \IEEEpubidadjcol in the second
% column for its text to clear the IEEEpubid mark.
\IEEEpubid{\begin{minipage}{\textwidth}\ \centering
Copyright \copyright 2025 IEEE. Personal use of this material is permitted. \\
However, permission to use this material for any other purposes must be obtained
from the IEEE by sending an email to pubs-permissions@ieee.org.
\end{minipage}}
\maketitle
\begin{abstract}
Multi-modal fusion holds great promise for integrating information from different modalities. However, due to a lack of consideration for modal consistency, existing multi-modal fusion methods in the field of remote sensing still face challenges of incomplete semantic information and low computational efficiency in their fusion designs. Inspired by the observation that the visual language pre-training model CLIP can effectively extract strong semantic information from visual features, we propose M$^3$amba, a novel end-to-end CLIP-driven Mamba model for multi-modal fusion to address these challenges. Specifically, we introduce CLIP-driven modality-specific adapters in the fusion architecture to avoid the bias of understanding specific domains caused by direct inference, making the original CLIP encoder modality-specific perception. This unified framework enables minimal training to achieve a comprehensive semantic understanding of different modalities, thereby guiding cross-modal feature fusion. To further enhance the consistent association between modality mappings, a multi-modal Mamba fusion architecture with linear complexity and a cross-attention module Cross-SS2D are designed, which fully considers effective and efficient information interaction to achieve complete fusion. Extensive experiments have shown that M$^3$amba has an average performance improvement of at least 5.98\% compared with the state-of-the-art methods in multi-modal hyperspectral image classification tasks in the remote sensing field, while also demonstrating excellent training efficiency, achieving a double improvement in accuracy and efficiency. The code is released at https://github.com/kaka-Cao/M3amba.
\end{abstract}
\begin{IEEEkeywords}
Deep learning, Remote Sensing, Multi-modal, Feature Fusion, CLIP model, Mamba.
\end{IEEEkeywords}
\section{Introduction}
\begin{figure}[htbp]
\centering
\includegraphics[width=1\linewidth]{intro_qikan.pdf}
\caption{\textbf{Left:} Proposed CLIP-driven modality-specific adapters guide the fusion process to produce complete features through semantic information interactions. \textbf{Right:} Common methods perform pairwise fusion by training encoders and fusion networks, which lacks consideration of semantic consistency and leads to incomplete representation. \(I_1\) and \(I_2\) represent inputs from different modalities.}
\label{intro_fig}
\vspace{-10pt}
\end{figure}
Multi-modal fusion tasks involve integrating features from various data modalities, such as visible light, infrared, and LiDAR, to achieve more complete feature representations \cite{zhao2023tufusion, li2024feddiff, ma2023mbsi}. This is crucial for applications like remote sensing scene understanding \cite{li2024mdfl}. Unfortunately, due to hardware limitations, a single sensor cannot capture all the complex details of an image \cite{xie2024fusionmamba}, but the semantic information in multi-modal images is always consistent, so it is particularly important to effectively leverage comprehensive semantic information across different modalities to guide fusion \cite{zhang2024simfusion}. Existing methods focus on training different models for semantic feature extraction, but these models have limitations. CNN-based approaches cannot capture comprehensive semantic information due to their limited receptive fields, while Transformer-based methods have global modeling capabilities but incur high training costs and lack transferability. The advent of Contrastive Language-Image Pre-training (CLIP) \cite{radford2021learning} offers a solution to these issues. CLIP achieves significant success in various visual tasks by pre-training with a large corpus of image-text pairs obtained from the web. Given its robustness and transferability, CLIP is often used to directly infer effective image features. For example, Lin \textit{et al.} \cite{lin2022frozen} used frozen CLIP output features to capture semantic information between adjacent frames. Yang \textit{et al.} \cite{yang2024inf} used CLIP image embedding for global understanding, allowing the model to learn from high-level semantic information. This confirms CLIP's powerful ability to learn rich semantic visual representations and its ability to transfer to other visual tasks. However, direct inference often results in weak perception capabilities of downstream tasks \cite{gao2024clip}, and the potential of this approach in the field of multi-modal images remains underexplored.
\IEEEpubidadjcol
% For example, Lin \textit{et al.} \cite{lin2022frozen} introduced a video learning framework that uses frozen CLIP output features to train video recognition models, thereby reducing computational and memory resource consumption while enhancing video network performance. This confirms CLIP's powerful capability to learn rich semantic visual representations and its ability to transfer to other visual tasks.
% In recent years, significant progress has been made in adapting large-scale foundational pre-trained models to specific downstream tasks in the multi-modal domain, primarily driven by the rise of large-scale visual language pre-training models such as Contrastive Language-Image Pretraining (CLIP) \cite{radford2021learning}. However, in the multi-modal image domain, the potential remains underexplored. This domain involves fusing features from various data modalities to obtain a more complete feature representation (e.g., visible light, infrared, LiDAR), which is critical for applications like remote sensing, medical imaging, and natural scene understanding. Our goal is to reduce the dependency of deep models on nonlinear computations while maintaining robust transferability. Despite the advent of self-attention mechanisms facilitating the interaction of information flow between modality tokens, they introduce scalability challenges with quadratic computational demands and lack the rich representation capacity for multi-modal fusion features.
%\begin{figure}[t]
% \vspace{5pt}
% \centering
% \includegraphics[width=0.9\linewidth]{bubble1.pdf}
% \vspace{-5pt}
% \caption{In a comparative study of efficiency on the MFNet dataset against ViT, VMamba, and other CNN-based methods, our M$^3$amba achieves dual improvements in both accuracy and computational efficiency. The size of the bubble represents the number of parameters.}
% \label{intro_fig}
% \vspace{-10pt}
%\end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[width=1\linewidth]{bubble.pdf}
% \caption{\textbf{Left:} Proposed CLIP-driven modality-specific adapters guide the fusion process to produce complete features through semantic information interactions. \textbf{Right:} Common methods use encoders pre-trained with only images for pairwise fusion, resulting in fusion redundancy. \(I_1\) and \(I_2\) represent inputs from different modalities.}
% \label{intro_fig}
% \end{figure}
It is noteworthy that multi-modal image data captured and processed by different sensors often exhibit characteristics of consistency and complementarity \cite{zhang2020deep}. These traits can significantly enhance the capabilities of deep learning models, facilitating more detailed and nuanced task interpretation. Thus, obtaining complete multi-modal fusion features is crucial for optimizing tasks. Compared to traditional linear concatenation methods in CNN architectures, approaches based on attention mechanisms have shown satisfactory results. Ma \textit{et al.} \cite{ma2022swinfusion} introduced a novel universal image fusion framework based on cross-domain distance learning and Swin Transformer, achieving comprehensive integration and global interaction of multi-modal complementary information. Although the emergence of self-attention mechanisms has facilitated the interaction of information flow between modal features, they bring challenges of quadratic computational requirements and lack the rich semantic representation capabilities of multi-modal fusion features.
% Multi-modal feature fusion, due to the non-uniqueness of its fused features, represents a typical ill-posed problem. Thus, obtaining complete and unique multi-modal fused features is crucial for task optimization. Compared to the direct linear concatenation methods of traditional CNN architectures, attention-based methods have shown satisfactory results. Ma \textit{et al.} \cite{ma2022swinfusion} proposed a novel universal image fusion framework based on cross-domain long-distance learning and the Swin Transformer, achieving comprehensive integration and global interaction of multi-modal complementary information. Recently, the CLIP framework has achieved remarkable success in various visual tasks, including semantic segmentation, object detection, and image classification. Wang \textit{et al.} \cite{wang2024m2} introduced M$^2$-CLIP, a novel multi-modal architecture to enhance the temporal representation capabilities of visual encoders, demonstrating outstanding performance in video action recognition and confirming CLIP's strong generalization capability in single-source visual tasks.
% The distinctions between M$^3$amba and other multi-modal image feature fusion methods are illustrated in Figure 1. Common methods directly combine intermediate features from two modalities into a single fusion encoder network \cite{peng2024fusionmamba, li2024mambadfuse}, often resulting in incomplete alignment within the fusion space but are efficient. Cross-feature fusion enhances multi-modal data fusion performance by promoting dynamic feature interactions \cite{xie2024fusionmamba, dong2024fusion}. Although these methods have significantly advanced the field of multi-modal feature fusion, they either relied on the high computational costs of complex models or lacked generalization capabilities in terms of the completeness of the fused features.
% Although these methods have significantly advanced the field of multi-modal feature fusion, they still suffer from two major drawbacks: 1) Emphasizing the direct linear combination or concatenation of bimodal features leads to fusion redundancy, neglecting the critical role of multi-modal feature interactions; 2) When interacting features, the domain gap between multi-modal features remains, which does not ensure efficient fusion.
%
Thus, a natural question arises: Can we obtain comprehensive visual semantic information from different modalities with minimal computational overhead to guide a fusion network for complete feature fusion? To overcome the shortcomings of existing methods, we propose the first CLIP-driven \textbf{M}amba model for \textbf{M}ulti-\textbf{M}odal fusion, named \textbf{M$^3$amba}. This unified framework benefits from the linear time complexity provided by Mamba \cite{gu2023mamba} and demonstrates improved transferability through CLIP image encoder and modality-specific adapters, as shown in the application transfer from the natural image domain to the remote sensing domain. Specifically, as shown in Fig. \ref{intro_fig}, we implant CLIP-driven modality-specific adapters into the encoder branch to introduce modality-specific semantic understanding with minimal training, thereby learning comprehensive visual semantic representations. This semantically guided cross-modal fusion enhances the understanding of global and local associative sequences, ensuring more robust and coherent feature representations. To achieve complete fusion, we introduce a three-branch state space model (SSM) that effectively interacts with CLIP semantic features through the proposed Cross-SS2D module to extract consistent and complementary information. This fusion process models the feature space of sequential data without the need for quadratic computational complexity, thus capturing inter-modal dependencies at minimal computational cost. While common methods perform pairwise fusion by training the encoder and fusion network from scratch, incomprehensive semantic information leads to incomplete fusion features and often redundant representations. Extensive experiments show that M$^3$amba outperforms current mainstream CNN, Transformer, and Mamba architectures in both performance and training efficiency. This reflects the great potential of the CLIP and Mamba dual-driven unified framework in multi-modal fusion tasks. To summarize, our contributions are three-fold:
\begin{itemize}
\item[$\bullet$] We propose M$^3$amba, a novel end-to-end CLIP-driven Mamba model for effective and efficient multi-modal fusion. To our best knowledge, M$^3$amba is the first model that synergistically optimizes the powerful visual capabilities of CLIP and the efficient computational performance of Mamba.
% We propose the first CLIP-driven Mamba model for multi-modal fusion, which employs modality-specific adapters to generate comprehensive semantic information, guiding efficient multi-modal fusion.
\item[$\bullet$] We design a multi-modal Mamba fusion architecture embedded with the proposed Cross-SS2D module, aiming to capture the comprehensive representation with linear complexity by enhancing the interactivity between modality mappings.
% We design a three-branch fusion architecture embedded with the Cross-SS2D module, accurately capturing the consistency and complementarity between different modalities while maintaining linear complexity, achieving complete feature fusion.
\item[$\bullet$] Extensive experiments in the field of multi-modal remote sensing show that M$^3$amba surpasses SOTA methods in terms of effectiveness and training efficiency, demonstrating the potential of the CLIP and Mamba dual-driven multi-modal fusion framework.
\end{itemize}
\section{Related Works}
\subsection{Large-scale Image Representation Learning}
With the emergence of weakly labeled data at the scale of entire websites, we have witnessed a surge in new models for general visual representation learning. Concurrently, the scale of image models built through regular supervised learning has also been rapidly increasing \cite{zhai2022scaling, riquelme2021scaling}. To further enhance visual representation capabilities, researchers have begun focusing on contrastive learning and self-supervised learning with large datasets and large models. The success of BERT \cite{devlin2018bert} has sparked an emerging direction of constructing large-scale visual models using masked visual modeling \cite{he2022masked, xie2022simmim}.
Vision-language models (VLMs), such as CLIP \cite{radford2021learning} and ALIGN \cite{jia2021scaling}, are trained on billion-scale, often noisy image-text datasets. These models consist of modality-specific encoders (image and text) that generate embeddings for each modality, therefore CLIP has been widely used as the vision backbone for its semantic representative feature and promising scalability even in the age of Multi-modal Large Language Models (MLLMs).
Additionally, several methods apply the CLIP image encoder to 2D image variations to achieve better quality or controllability \cite{ye2023ip, ramesh2022hierarchical}. Some methods use the CLIP image encoder to extract features for rendering images \cite{mohammad2022clip, sanghi2022clip, xu2023dream3d}, while others adopt CLIP to extract semantically rich video representations \cite{luo2022clip4clip, luo2023towards, luo2024zero}. Approaches like LiT \cite{zhai2022lit} and BLIP-2 \cite{li2023blip} reduce the training costs of CLIP-like models by deploying pre-trained unimodal models.
% Similar to these methods, our approach leverages a pre-trained CLIP image encoder. However, we focus on enhancing the homogeneity of feature space representations across both modalities using the strong semantic representation capabilities of CLIP. Moreover, our strategy of not involving the pre-trained CLIP model in further training brings only a slight training overhead while providing robust aligning semantic features.
% With the advent of large-scale labeled data, we have witnessed a surge in new models for general visual representation learning. Researchers have increasingly focused on using large datasets and substantial models for contrastive learning and self-supervised learning.
%Visual language models (VLMs), such as CLIP \cite{radford2021learning}, are trained on billion-scale, noisy image-text datasets.
%These models consist of encoders for specific modalities (image and text).
%To date, CLIP has been widely used as a visual backbone due to its strong semantic features and good scalability. For example, several methods apply the CLIP image encoder to 2D image transformations to achieve better quality or controllability \cite{ye2023ip, ramesh2022hierarchical}. Some methods use the CLIP image encoder to extract features for rendering images \cite{mohammad2022clip, sanghi2022clip, xu2023dream3d}, while others use CLIP to extract semantically rich video representations \cite{luo2022clip4clip, luo2023towards, luo2024zero}.
Inspired by these developments, we utilize a pre-trained CLIP image encoder to extract semantic information from multi-modal images. However, its perception of a specific modality often suffers from understanding deviations due to domain gaps. Therefore, we introduce modality-specific adapters to further enhance intra-modal semantic understanding with minimal training overhead.
It is worth noting that in our implementation, we choose CLIP instead of ALIGN mainly based on the following considerations: 1) CLIP has stronger migration ability and more stable performance; 2) CLIP provides a more complete pre-trained model and interface; 3) Research in the field of remote sensing shows that CLIP is more suitable for processing the semantic features of remote sensing images \cite{dong2024changeclip}.
\subsection{Multi-modal Fusion in Remote Sensing}
%Facing the increasingly complex and diverse demands of scene representation applications \cite{shvetsova2022everything, prakash2021multi}, deep learning has made significant contributions to overcoming the technical bottlenecks in multi-modal feature fusion, particularly through CNNs and Transformers \cite{deng2020deep, cao2023multimodal, xie2023yolo, xie2023sparsefusion}. While these methods integrate complementary information between different modalities, they are still limited in terms of computational efficiency and fusion completeness.
%Recently, Mamba \cite{gu2023mamba} has emerged as a promising candidate for the next-generation foundational model backbone, as it demonstrates better scaling laws than Transformers while maintaining linear time complexity. FusionMamba \cite{peng2024fusionmamba} incorporates VSS blocks into two U-shaped networks to extract spatial and spectral features and adapts SSM to handle dual inputs. SurvMamba \cite{chen2024survmamba} introduces Hierarchical Interaction Mamba (HIM) to facilitate efficient intra-modal interactions at different granularities and aggregate cascaded multi-modal features. Additionally, \cite{dong2024fusion} proposes the Fusion-Mamba Block (FMB) for feature fusion, connecting two backbone networks to enhance feature representation.
Faced with the increasingly complex and diverse needs of scene representation applications, deep learning has made significant contributions to overcoming the technical bottleneck of multi-modal feature fusion \cite{mei2023lightweight,mei2023rotation}, and researchers have been advancing the study of multi-modal classification tasks in remote sensing, especially through CNN and Transformers \cite{roy2023multimodal, lu2023coupled, ding2024uncertainty, zhang2024multimodal, wang2023mutually, dong2023joint, wang2025s3f2net,mei2021hyperspectral,mei2022learning}. Although these methods integrate complementary information between different modalities, they are still limited in terms of computational efficiency and fusion completeness.
Recently, Mamba has emerged as a promising candidate for the next generation of base model backbones as it exhibits better scaling than Transformers while maintaining linear time complexity and has been developed in the field of multi-modal remote sensing. SpectralMamba \cite{yao2024spectralmamba} achieved improved performance by dynamically simplifying but fully modeling hyperspectral data in both spatial-spectral space and hidden state space. S$^2$Mamba \cite{wang2024s} mined spatial-spectral context features to achieve more efficient and accurate land cover analysis. MiM \cite{zhou2024mamba} introduced an innovative deployment architecture for hyperspectral image classification, improving the model's efficiency through a novel centralized Mamba Cross Scan (MCS) mechanism and T-Mamba encoder design.
% In contrast to these methods, we leverage the aligning semantic feature representations generated by CLIP to thoroughly extract independent features from each modality to supplement the fused features, mapping unimodal features and fused features into a hidden state space for interaction. This approach reconstructs the hidden states of each modality, enhancing the representational consistency of the fused features.
% Facing the increasingly complex and diverse demands of scene representation applications \cite{shvetsova2022everything},
%Deep learning has made significant contributions to overcoming the technical bottlenecks in multi-modal fusion, particularly through CNNs and Transformers \cite{deng2020deep, cao2023multimodal, xie2023yolo, xie2023sparsefusion}. While these methods integrate complementary information between different modalities, they are still limited in terms of computational efficiency and fusion completeness. Recently, the emergence of Mamba \cite{gu2023mamba} provides a new solution for efficient multi-modal fusion.
% Mamba \cite{gu2023mamba} has emerged as a promising candidate for the next-generation foundational model backbone, as it demonstrates better scalability than Transformers while maintaining linear time complexity.
% FusionMamba \cite{peng2024fusionmamba} incorporated VSS blocks into two U-shaped networks to extract spatial and spectral features.
%SurvMamba \cite{chen2024survmamba} introduced Hierarchical Interaction Mamba (HIM), to facilitate efficient intra-modal interactions at different granularities and aggregate cascaded multi-modal features.
%Dong \textit{et al.} \cite{dong2024fusion} proposed the Fusion-Mamba Block (FMB) for feature fusion, connecting two backbone networks to enhance feature representation.
Compared to these methods, M$^3$amba uses comprehensive semantic features adapted by CLIP to guide the fusion of multi-modal features with linear complexity, while Cross-SS2D ensures feature consistency and uses complementary information as gain.
% This method reconstructs the hidden states of each modality, enhancing the completeness of the fused features.
\section{Methodology}
\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{framework_qikan.pdf}
\vspace{-15pt}
\caption{Overview of the M$^{3}$amba framework. For clarity, we split the end-to-end process of training into three stages: Feature Adaptation, Mamba Fusion with Cross-SS2D, and Training Objective. By utilizing the M$^{3}$amba framework to perform a feature-level fusion of the two modalities, we can apply the complete fusion features to different downstream tasks. MLP Head consists of several convolutional and linear layers, and CE stands for Cross Entropy.}
\label{framework_fig}
\vspace{-11pt}
\end{figure*}
\subsection{Problem Formulation}
In recent years, multi-modal data fusion has emerged as a crucial technique for enhancing the performance of various tasks, particularly in the field of remote sensing. In this paper, we focus on the application of multi-modal fusion to the pixel-level classification problem. This can be defined as accurately assigning each pixel in images from two modalities to the corresponding class. Given an image $X$ consisting of $q$ pixels, we aim to perform a pixel-level classification task using data from different modalities $X_1, X_2\in\mathbb{R}^{h\times w\times c}$. Both modalities learn features of the same scene with label information $L\in\mathbb{R}^{h\times w\times m}$, and the label for the $p$-th pixel can be represented as a one-hot vector $L^{q} = \{0^{m-1}, 1\}$, where $m$ denotes the number of classes. The objective of multi-modal pixel-level classification is to develop and train a model \( \psi(X_1, X_2) \) that effectively maps images from different modalities to the classification distribution \( C_{max}(X_1, X_2; L) \), indicating the probability that each pixel is associated with different classes under the guidance of the labels. A binary prediction map can be obtained through hard classification by a threshold \( \tau \) on the maximum probability for different classes.
\begin{equation}
\psi\left(X_1,X_2\right)=\left\{\begin{matrix}0,&\text{if } C_{\max}\left(X_1,X_2; L\right)<\tau,\\1,&\text{otherwise.}\end{matrix}\right.
\end{equation}
% Based on the above objectives and baseline models, we propose the M$^{3}$amba network architecture.
Based on the above objectives, the logits can be expressed as \( C_{max}(X_1, X_2; L) \), specifically formulated as \( C_{max}(X_1, X_2; L) = \vartheta(X_1, X_2 \mid \beta_1, \beta_2) \). Here, we map the image feature space to the classification space using a nonlinear target model \( \vartheta(\cdot) \), where \( \beta_1 \) and \( \beta_2 \) represent the parameters of the two modality branches.
\subsection{Preliminaries}
\noindent\textbf{State Space Models}
State Space Models (SSMs) represent a class of sequence-to-sequence modeling systems characterized by constant dynamics over time, often used to represent linear time-invariant systems.
Due to their linear complexity, SSMs can efficiently capture the inherent dynamics of a system through implicit mapping to latent states.
Mathematically, an SSM is typically represented as a linear ordinary differential equation (ODE):
\begin{equation}
\frac{dh(t)}{dt} = Ah(t) + Bx(t), y(t) = Ch(t) + Dx(t),
\end{equation}
where \( x(t) \in \mathbb{R} \), \( h(t) \in \mathbb{R}^n \), and \( y(t) \in \mathbb{R} \) represent the input, hidden state, and output, respectively, with \( n \) being the state size. The remaining parameters include the state transition matrix \( A \in \mathbb{R}^{n \times n} \), projection parameters \( B \in \mathbb{R}^{n \times 1} \), \( C \in \mathbb{R}^{1 \times n} \), and skip connection \( D \in \mathbb{R} \).
To handle discrete sequences such as images and text, the ODE needs to be converted into a discrete function by introducing a predefined time scale parameter \( \Delta \in \mathbb{R}^D\). The discretization process of the above equations is as follows:
\begin{equation}
\overline{A} =\exp(\Delta A), \\
\overline{B} =(\Delta A)^{-1}(\exp(\Delta A)-I)\cdot\Delta B,
\end{equation}
\begin{equation}
\label{eq:4}
h^t=\overline{A}h^{t-1}+\overline{B}x^t, \\
y^{t}=Ch^{t} + Dx^{t},
\end{equation}
all matrices maintain the same dimensions across iterations of the operation. Furthermore, following Mamba, the matrix $\overline{B}$ can be approximated by a first-order Taylor series:
\begin{equation}
\overline{B}=(\exp(A)-I)A^{-1}B\approx(\Delta A)(\Delta A)^{-1}\Delta B=\Delta B.
\end{equation}
%\begin{figure*}[htbp]
% \centering
% \includegraphics[width=1\linewidth]{SEG1.pdf}
% \vspace{-20pt}
% \caption{Qualitative comparison results on MFNet and PST900 datasets. The first three rows and the last two rows correspond to MFNet and PST900, respectively.}
% \label{SEG_fig}
% \vspace{-11pt}
%\end{figure*}
\noindent\textbf{2D-Selective-Scan Mechanism}
Due to the one-dimensional and causal properties of the Mamba selection mechanism, directly applying Mamba to visual tasks is unsuitable.
For instance, while two-dimensional spatial information plays a crucial role in vision-related tasks, it is secondary in one-dimensional sequence modeling. Additionally, causal processing of input data prevents Mamba from absorbing information from parts of the data not yet scanned.
These differences result in a limited receptive field, unable to capture potential correlations with unexplored patches.
VMamba \cite{liu2024vmamba} introduces a Two-Dimensional Selective Scanning (SS2D) mechanism to solve this problem.
SS2D transforms the input image into patch sequences along the horizontal and vertical axes, scanning in four directions: top-left to bottom-right, bottom-right to top-left, top-right to bottom-left, and bottom-left to top-right, generating independent sequences. This four-way scanning ensures that each element in the feature map contains information from all other positions in various directions. Consequently, it establishes a comprehensive global receptive field without linearly increasing computational complexity. The generated four sequences are then individually processed using selective SSM. Finally, all feature sequences are transformed back to their original 2D layout and merged to reconstruct the 2D feature map. SS2D is a core component of the Visual State Space (VSS) module, as shown in Figure \ref{framework_fig}, on which we build the hidden state space for cross-modal feature fusion.
\begin{algorithm}[tb]
\caption{Algorithm of Cross-SS2D.}
\label{alg:algorithm}
\textbf{Input}: Different modalities of data $X_{1}$, $X_{2}$. \\
\textbf{Parameter}: Weight matrices $weight_{1}$, $bias_{1}$, $weight_{2}$, $bias_{2}$, $A_{1}$, $B_{1}$, $A_{2}$, $B_{2}$. \\
\textbf{Output}: Fusion feature $H_{f}$. \\
\begin{algorithmic} [1]%[1] enables line numbers
\STATE \textbf{Initial} $A_{1}$, $B_{1}$, $A_{2}$, $B_{2}$. \\
\FOR{each block N = 1, 2, ..., 9}
\STATE \textbf{1. Compute System Matrices}
% \STATE \textbf{1. Compute Crossed Matrix Parameter A, B}
\STATE \quad $As_{1}$ $\leftarrow$ $-exp(A_{1})$ \quad $As_{2}$ $\leftarrow$ $-exp(A_{2})$
\STATE \quad $Bs_{1}$ $\leftarrow$ $Split(einsum(X_{1}, weight_{1}) + bias_{1})$
\STATE \quad $Bs_{2}$ $\leftarrow$ $Split(einsum(X_{2}, weight_{2}) + bias_{2})$
\STATE \textbf{2. Cross Assignment}
\STATE \quad $As_{f}$ $\leftarrow$ $(As_{1} + As_{2}) / 2$
\STATE \quad $Bs_{1}$ $\leftrightarrow$ $Bs_{2}$
\STATE \textbf{3. Extract Fusion Features}
\STATE \quad $H_{f}$ $\leftarrow$ $selective\_scan(As_{f}, Bs_{1}, Bs_{2})$
% \STATE \quad $H_{2}$ $\leftarrow$ $selective\_scan(As_{f}, Bs_{1})$
\ENDFOR
% \WHILE{condition}
% \STATE Do some action.
% \IF {conditional}
% \STATE Perform task A.
% \ELSE
% \STATE Perform task B.
% \ENDIF
% \ENDWHILE
\STATE \textbf{return} $H_{f}$ to the model.
\end{algorithmic}
\end{algorithm}
\subsection{Network Architecture}
\noindent\textbf{Overview}
The overall structure of M$^{3}$amba, illustrated in Figure \ref{framework_fig}, is a multi-modal Mamba model built on top of a fixed CLIP image encoder, with images from different modalities used as inputs.
%The fusion of features from both modalities significantly impacts the performance of downstream tasks.
Features corresponding to the same class from different modalities contain both consistent information and complementary insights. For instance, LiDAR provides enhanced spatial resolution, while hyperspectral data offers superior spectral resolution; infrared sensors excel at capturing thermal radiation, and visible light sensors are proficient at detailed texture data. However, a single sensor cannot capture all the complex details of an image, but the semantic information of images under the same scene is always consistent. Therefore, our goal is to \textbf{maximize the description of consistent information while leveraging complementary insights as benefits under comprehensive semantic feature guidance}.
Inspired by this, our proposed method aims to balance the informational discrepancies between the two modalities while utilizing their consistency and complementarity.
For clarity, our network structure is conceptually divided into two parts, but it operates as a single end-to-end training process. Initially, modality-specific adapters are embedded in the pre-trained CLIP image encoder to capture comprehensive semantic information from different modalities. The output features, along with the input images, are then processed by Mamba's three-branch feed-forward network: \( \vartheta_1 \), \( \vartheta_2 \), and \( \vartheta_f \).
Guided by an unsupervised consistency loss, the supervised fusion network with linear complexity is optimized to fully capture the features across modalities.
Specifically, the fusion network consists of three main branches: two consistency branches \( \vartheta_1 \) and \( \vartheta_2 \), \( \vartheta_1 \) processes the feature sequence of the first modality (HSI), and \( \vartheta_2 \) processes the feature sequence of the second modality (LiDAR). They share the same network architecture, but use independent parameters to maintain the uniqueness of each modality feature, and map the unimodal and comprehensive semantic features to the hidden state space for interaction, so that complementary features and their consistency information can be thoroughly extracted. Additionally, the fusion branch \( \vartheta_f \) employs a selective scanning mechanism for global modeling, further integrating the complete fusion features. The entire fusion network undergoes \( N \) iterative updates, and the classification outputs of the three branches are finally used for loss function computation.
% \noindent\textbf{CLIP Fusion Representation Generation}
\noindent\textbf{Feature Adaptation}
By learning from a large number of image-text pairs, CLIP
%CLIP has an inherent advantage in extracting deep semantic visual features.
can match images with their corresponding natural language descriptions. During training, the encoders are optimized by extracting features from input samples and aligning them in the embedding space using contrastive loss.
This enables zero-shot classification and highlights CLIP's inherent advantage in extracting deep semantic visual feature representations.
Leveraging this, we use the CLIP image encoder \(E_v\) to infer multi-modal input images \(X_1\) and \(X_2\), mapping them to the same feature space to reduce the domain gap between modalities and learn visual representations with rich semantics. In addition, in order to endow the original CLIP encoder with modality-specific perception capabilities for guiding more comprehensive feature fusion, we introduce modality-specific adapters. The resulting multi-modal features \(Y_1\) and \(Y_2\) have comprehensive modal scene semantic understanding and ensure more robust and coherent feature representations. This can be expressed as the equation:
\begin{equation}
Y_1, Y_2 = E_v^{Adapter_1}(X_1), E_v^{Adapter_2}(X_2),
\end{equation}
where \( E_v^{Adapter_1} \) and \( E_v^{Adapter_2} \) represent the CLIP image encoder with modality-specific adapters added. The learnable adapters consist of two linear layers with residual connections, inserted after each attention block. We discard the last layer of the encoder because it is dedicated to classification. The resulting output features \( Y_1\) and \( Y_2\) are concatenated into \( Z \) after feature size transformation to provide guidance for the fusion network.
\noindent\textbf{Mamba Fusion}
As analyzed in the related work, previous multi-modal fusion methods excessively focused on the differing relationships between modalities, often neglecting the consistency across the data. Our fusion objective is to obtain complete fusion features between modalities, involving capturing local differences and overall invariants.
% Driven by this, we derive the overall invariants between modalities to achieve a comprehensive representation of fused modal features:
% \begin{equation}
% H=\Omega(X_1|\beta_1)=\Omega(X_2|\beta_2)=\Omega(X_1,X_2|\beta_1,\beta_2).
% \end{equation}
% Here, \( H \) represents the consistent features, \( \Omega(\cdot) \) denotes the feature extraction operation. This equation is subject to the constraints of $C_{max}(H)=L, J_\Omega\neq0$, \( L \) is the corresponding one-hot label vector, and \( J_\Omega \) represents the Jacobian matrix of \( \vartheta(\cdot) \).
By leveraging the obtained CLIP semantic feature \( Z \), we facilitate the fusion process to learn complete feature representations. Therefore, the entire encoding process can be expressed as:
\begin{gather}
Out_1 =\vartheta_1(X_1,Z|\beta_1),\\
Out_2 =\vartheta_2(X_2,Z|\beta_2), \\
Out_f =\vartheta_f(Out_1+Out_2|\beta_f),
\end{gather}
where \(Out_1\), \(Out_2\), and \(Out_f\) are the outputs of the three branches, and \(\beta_1\), \(\beta_2\), and \(\beta_f\) are their corresponding parameters. The fusion of features from both modalities significantly impacts the performance of downstream tasks. To that effect, designing a cross-modal complete attention mechanism becomes crucial.
% To achieve this, we propose a cross-modal consistency attention mechanism that effectively aligns and integrates features across different modalities. This mechanism enhances the model's ability to maintain and utilize the inherent relationships between modalities, ensuring a robust and coherent feature representation. By incorporating the consistency features \(H\) into the attention mechanism, we enhance the interaction and fusion of features from different modalities, improving classification and segmentation performance.
\noindent\textbf{Cross-SS2D}
As shown in Algorithm \ref{alg:algorithm}, the Cross-SS2D takes two features as input and generates a fused output while preserving the original shape of the features. The two inputs come from the comprehensive semantic features of CLIP and the unimodal image features, fully extracting independent features from each modality to complement the fused features and leveraging complementary insights to encompass comprehensive modal information.
To achieve the model's context-aware capability, linear projection layers are employed to produce the system matrices \(B\), \(C\), and \(\Delta\) from the inputs. As specified in Equation \ref{eq:4}, matrices \(\overline{A}\) and \(\overline{B}\) encode the previous hidden state \(h^{t-1}\) and the input \(x^t\) to compute the current state \(h^t\).
Inspired by the cross-attention mechanisms \cite{chen2021crossvit} widely used in multi-modal tasks, we aim to facilitate the interaction of information between multiple modalities within the 2D selective scanning module. To achieve this goal, we use the \(\overline{B}\) matrix generated by the complementary modality in the selective scanning operation, enabling the SSM to provide complementary information for the current modality guided by the features from another modality. Additionally, the \(\overline{A}\) matrices corresponding to the two modalities are averaged to provide consistent information for the multi-modal input, ultimately resulting in the fusion feature \(H_f\). Specifically, this process can be expressed as:
\begin{gather}
\overline{A}_{1}=\exp(\Delta_{1}A_{1}),\overline{A}_{2}=\exp(\Delta_{2}A_{2}),\\
\overline{B}_{1}=\Delta_{1}B_{1},\overline{B}_{2}=\Delta_{2}B_{2},\\
% h_{1}^{t}=\frac{\overline{A}_{1} + \overline{A}_{2}}{2}h_{1}^{t-1}+\overline{B}_{2}x_{1}^{t},\\
h_{1}^{t}=(\overline{A}_{1} + \overline{A}_{2})h_{1}^{t-1}/2+\overline{B}_{2}x_{1}^{t},\\
% h_{2}^{t}=\frac{\overline{A}_{1} + \overline{A}_{2}}{2}h_{2}^{t-1}+\overline{B}_{1}x_{2}^{t},\\
H_f=y_{1}^t=C_{1}h_{1}^t+D_{1}x_{1}^t,
% y_{2}^t=C_{2}h_{2}^t+D_{2}x_{2}^t,
\end{gather}
where $\Delta$ is a predefined time scale parameter, \( x^t \) represents the input at time \( t \), and \( y^t \) denotes the selective scan output. Subscripts 1 and 2 represent unimodal features and CLIP features, respectively. To prevent redundant fusion, we choose $y_1^t$ as the fusion feature. \( \overline{A}_{1} \) and \( \overline{A}_{2} \) are used as the averaged state transition matrices providing multi-modal consistency, \( \overline{B}_{1} \) and \( \overline{B}_{2} \) are used as the cross-modal projection matrices providing complementary information. By implementing this Cross-SS2D fusion mechanism, we achieve a balanced and comprehensive representation of multi-modal features, leveraging both the complementary and consistent aspects of the input data. In addition, the A matrix enhances the feature consistency between modalities, making the fusion process smoother. The B matrix guides feature selection and avoids the transmission of redundant information. Through the interaction of the A matrix and the B matrix, Cross-SS2D can establish an efficient information transmission mechanism between modalities, improving the efficiency of feature selection and fusion between different modalities.
\begin{table*}[htbp]
\centering
\renewcommand{\arraystretch}{1.3}
\setlength{\tabcolsep}{4mm}
\caption{Ablation Studies on the Houston2013, the Augsburg, and the MUUFL Dataset. The Best Result is \textbf{Highlighted}}
\begin{tabular}{c|ccc|ccc|ccc}
\hline\hline
& \multicolumn{3}{c|}{\textbf{Houston2013}} & \multicolumn{3}{c|}{\textbf{Augsburg}} & \multicolumn{3}{c}{\textbf{MUUFL}} \\
\hline\hline
\textbf{Scheme} & \textbf{OA}(\%) & \textbf{AA}(\%) & \textbf{$\kappa$}(×100) & \textbf{OA}(\%) & \textbf{AA}(\%) & $\kappa$(×100) & \textbf{OA}(\%) & \textbf{AA}(\%) & $\kappa$(×100) \\
\hline
(A) & 95.02 & 95.06 & 94.66 & 95.65 & 84.52 & 93.72 & 96.25 & 91.03 & 95.00 \\
(B) & 95.84 & 95.84 & 95.55 & 96.27 & 86.68 & 94.66 & 97.16 & 94.70 & 96.22 \\
(C) & 96.85 & 96.86 & 96.62 & 97.47 & 88.88 & 96.36 & 97.40 & 95.02 & 96.54 \\
\textbf{(D)} & \textbf{97.31} & \textbf{97.32} & \textbf{97.11} & \textbf{98.19} & \textbf{93.32} & \textbf{97.39} & \textbf{97.84} & \textbf{96.10} & \textbf{97.13} \\
\hline\hline
\end{tabular}
\label{ablation_tab}
% \vspace{-10pt}
\end{table*}
\begin{table*}[htbp]
\centering
\renewcommand{\arraystretch}{1.3}
\setlength{\tabcolsep}{3mm}
\caption{Comparison Results on the Houston2013, the Augsburg, and the MUUFL Dataset. The Best Result is \textbf{Highlighted}}
\begin{tabular}{c|c|ccc|ccc|ccc}
\hline\hline
\multirow{2}{*}{\textbf{Method}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c|}{\textbf{Houston2013}} & \multicolumn{3}{c|}{\textbf{Augsburg}} & \multicolumn{3}{c}{\textbf{MUUFL}} \\
& &\textbf{OA}(\%) & \textbf{AA}(\%) & \textbf{$\kappa$}(×100) & \textbf{OA}(\%) & \textbf{AA}(\%) & $\kappa$(×100) & \textbf{OA}(\%) & \textbf{AA}(\%) & $\kappa$(×100) \\
\hline\hline
CALC$_{23}$ &CNN &88.97 &90.78 &88.06 &91.73 &62.62 &87.98 &93.94 &74.09 &92.00 \\
HCT$_{23}$ &CNN\&ViT &91.15 &92.28 &90.40 &90.90 &65.24 &86.83 &92.95 &80.50 &90.69 \\
UACL$_{24}$ &CNN &95.37 &95.99 &95.00 &89.19 &74.30 &84.80 &88.29 &89.11 &84.78 \\
Semi-ViT$_{22}$ & ViT &85.46 &86.71 &86.71 &86.64 &64.67 &84.05 &92.46 &79.63 &89.49 \\
MFT$_{23}$ & ViT &89.80 &91.51 &88.89 &90.49 &60.36 &86.26 &94.34 &81.48 &92.51 \\
ExViT$_{23}$ & ViT &91.40 &92.60 &90.66 &87.91 &63.20 &82.82 &94.37 &83.37 &92.54 \\
MiM$_{24}$ &Mamba &92.89 &94.21 &92.28 &88.63 &67.57 &86.56 &92.65 &82.22 &91.59 \\
S$^{2}$Mamba$_{24}$ &Mamba &93.36 &94.09 &92.79 &89.34 &70.41 &89.45 &94.19 &83.72 &91.98 \\
\textbf{M$^3$amba(Ours)} & Mamba & \textbf{97.31} & \textbf{97.32} & \textbf{97.11} & \textbf{98.19} & \textbf{93.32} & \textbf{97.39} & \textbf{97.84} & \textbf{96.10} & \textbf{97.13} \\
\hline\hline
\end{tabular}
\label{hsi_tab}
\vspace{-10pt}
\end{table*}
% \noindent\textbf{Training Objective}
\subsection{Training Objective}
Given two modality inputs \( X_1^i \) and \( X_2^i \), our training strategy involves enhancing the supervised fusion network with linear complexity under the direction of an unsupervised loss to fully capture consistent and complementary features between modalities, where \( X_1^i \) represents the \( i \)-th sample of the first modality. For the supervised loss, we create an object mask map based on the ground truth, aiming to promote the consistency of the output probability distribution with the label \( L \) under the strong supervision constraint of the mask. The supervised loss calculation is as follows:
\begin{figure}[htbp]
\centering
\includegraphics[width=1\linewidth]{box_plot.pdf}
\caption{Comparison of box plots with other methods on three datasets.}
\label{boxplot_fig}
\end{figure}
\begin{multline}
\mathcal{L}_{Sup}=\sum_{i=1}^{X^{i}}\mathcal{L}_{CE}[(L,f_{1}\left(Out_1\right));
(L,f_{2}\left(Out_2\right));\\
(L,f_{f}\left(Out_f\right)) \ | \ Mask],
\end{multline}
% \begin{tiny}
% \begin{equation}
% \mathcal{L}_{Sup}=\sum_{i=1}^{X^{i}}\mathcal{L}_{CE}[((L,f_{1}\left(Out_1\right))+
% (L,f_{2}\left(Out_2\right))+(L,f_{f}\left(Out_f\right))) | Mask],
% \end{equation}
% \end{tiny}
% where \( f_1(\cdot) \), \( f_2(\cdot) \), and \( f_f(\cdot) \) are neural networks that obtain the vectorized output for each sample.
where \( f_1(\cdot) \), \( f_2(\cdot) \), and \( f_f(\cdot) \) represent the MLP Head.
% To learn the nonlinear embeddings between different modalities,
To minimize the differences between the unique and complete feature predictions,
%, promoting consistent representations,
we calculate the sum of cross-modal cosine similarity losses of three different features for unsupervised learning. The proposed unsupervised consistency loss is calculated as follows:
\begin{multline}
\mathcal{L}_{Unsup}=\sum_{i=1}^{X^{i}}[2-\cos(f_{1}\left(Out_1\right),f_{f}\left(Out_f\right))-\\
\cos(f_{2}\left(Out_2\right),f_{f}\left(Out_f\right))].
\end{multline}
% \begin{tiny}
% \begin{equation}
% \mathcal{L}_{Unsup}=\sum_{i=1}^{X^{i}}[2-\cos(f_{1}\left(Out_1\right),f_{f}\left(Out_f\right)-\cos(f_{2}\left(Out_2\right),f_{f}\left(Out_f\right))].
% \end{equation}
% \end{tiny}
The complete loss function consists of supervised loss and unsupervised consistency loss. However, there is a consensus that strong supervised learning can provide more accurate guidance for model optimization. Therefore, we utilize supervised loss as the primary update and the unsupervised consistency loss as auxiliary optimization, as shown in the following equation: $\mathcal{L}_{Total} = \lambda_1\mathcal{L}_{Sup} + \lambda_2\mathcal{L}_{Unsup}$. In particular, $\lambda_1$ is usually greater than $\lambda_2$.
% \begin{equation}
% \mathcal{L}_{Total} = Self\text{-}Adapted(\lambda_1\mathcal{L}_{Sup} + \lambda_2\mathcal{L}_{Unsup}),
% \end{equation}
\begin{figure}[htbp]
\centering
% \vspace{-5pt}
\includegraphics[width=0.9\linewidth]{ABLATION_QIKAN.pdf}
\caption{t-SNE for ablation on three datasets. The results from top to bottom correspond to Houston2013, Augsburg
, and MUUFL datasets respectively.}
\label{ablation_fig}
\end{figure}
\section{Experiments and Analysis}
\subsection{Experimental Settings}
\subsubsection{Datasets}
To validate the effectiveness of the proposed method in analyzing multi-modal remote sensing data, we utilize three hyperspectral multi-modal datasets: the Houston2013 dataset \cite{debes2014hyperspectral}, the Augsburg dataset \cite{hong2021multimodal}, and the MUUFL dataset \cite{gader2013muufl} for remote sensing classification tasks.
\begin{table}[htbp]
\small
%\begin{table}[H]
% \vspace{-2.4mm}
\renewcommand{\arraystretch}{1.3}
\setlength{\tabcolsep}{1.5mm}
% \setlength{\extrarowheight}{-1pt}
% \setlength{\arrayrulewidth}{0.8pt} % Set the vertical line width
\caption{A List of the Number of Training and Testing Samples for Each Class in Houston2013 Dataset}
\label{tab-houstondataset}
\centering
\begin{tabular}{ccc|ccc}
\hline\hline
\textbf{Class No.} & \textbf{Training} & \textbf{Testing}
&\textbf{Class No.} & \textbf{Training} & \textbf{Testing} \\
\hline\hline
1 & 198 & 1053 & 9 & 193 & 1059 \\
2 & 190 & 1064 & 10 & 191 & 1036 \\
3 & 192 & 505 & 11 & 181 & 1054 \\
4 & 188 & 1056 & 12 & 192 & 1041 \\
5 & 186 & 1056 & 13 & 184 & 285 \\
6 & 182 & 143 & 14 & 181 & 247 \\
7 & 196 & 1072 & 15 & 187 & 473 \\
8 & 191 & 1053 &&& \\
\hline\hline
\end{tabular}
\end{table}
\begin{table}[htbp]
\small
\renewcommand{\arraystretch}{1.3}
\setlength{\tabcolsep}{1.5mm}
% \setlength{\extrarowheight}{-1pt}
% \setlength{\arrayrulewidth}{0.8pt} % Set the vertical line width
\caption{A List of the Number of Training and Testing Samples for Each Class in Augsburg Dataset}
\label{Augsburg}
\centering
\begin{tabular}{ccc|ccc}
\hline\hline
\textbf{Class No.} & \textbf{Training} & \textbf{Testing}
&\textbf{Class No.} & \textbf{Training} & \textbf{Testing} \\
\hline \hline
1 & 146 & 13361 & 5 & 248 & 26609 \\
2 & 7 & 1638 & 6 & 52 & 523 \\
3 & 264 & 30065 & 7 & 23 & 1507 \\
4 & 21 & 3830 &&&\\
\hline\hline
\end{tabular}
\end{table}
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.3}
\setlength{\tabcolsep}{1.5mm}
% \setlength{\extrarowheight}{-1pt}
% \setlength{\arrayrulewidth}{0.8pt} % Set the vertical line width
\caption{A List of the Number of Training and Testing Samples for Each Class in MUUFL Dataset}
\label{MUUFL}
\centering
\begin{tabular}{ccc|ccc}
\hline\hline
\textbf{Class No.} & \textbf{Training} & \textbf{Testing}
&\textbf{Class No.} & \textbf{Training} & \textbf{Testing} \\
\hline \hline
1 & 1162 & 22084 & 7 & 112 & 2121 \\
2 & 214 & 4056 & 8 & 312 & 5928 \\
3 & 344 & 6538 & 9 & 69 & 1316 \\
4 & 91 & 1735 & 10 & 9 & 174 \\
5 & 334 & 6353 & 11 & 13 & 256 \\
6 & 23 & 443 &&& \\
\hline\hline
\end{tabular}
\end{table}
\noindent\textbf{The Houston2013 Dataset}
The Houston2013 HSI-LiDAR Dataset is a widely used resource for remote sensing research, specifically designed for land cover classification tasks. It features hyperspectral imagery (HSI) captured by the ITRES CASI-1500 sensor over the University of Houston and nearby rural areas in Texas, USA. This dataset was initially collected in June 2012 and later made available for the IEEE GRSS Data Fusion Contest in 2013. The HSI comprises 144 spectral bands, spanning from 380 nm to 1050 nm, with a spatial resolution of 10 nm, while the LiDAR data provide a single-channel elevation map. Both data modalities share a spatial resolution of 2.5 meters, and the entire dataset consists of $349 \times 1905$ pixels. Predefined training and testing sets are available for classification tasks, with details in Table \ref{tab-houstondataset}.
\noindent\textbf{The Augsburg Dataset}
The Augsburg HS-SAR-LiDAR Dataset originates from a multi-modal data collection effort in Augsburg, Germany, and includes hyperspectral, synthetic aperture radar (SAR), and digital surface model (DSM) images. The hyperspectral imagery was collected using the HySpex sensor, delivering data across 180 spectral bands between 400 nm and 2500 nm. The selected study area includes 332,485 pixels with a ground sampling distance (GSD) of 30 meters. Additionally, the dataset provides four SAR bands and DSM elevation data, with the ground truth derived from manual labeling. This dataset is ideal for urban area classification, covering various land cover types. The details are outlined in the Table \ref{Augsburg}.
\noindent\textbf{The MUUFL Dataset}
The MUUFL Gulfport Scene Dataset was gathered in November 2010 at the Gulf Park Campus of the University of Southern Mississippi in Long Beach, Mississippi. The dataset consists of hyperspectral images, captured by the CASI-1500 sensor, with 64 available spectral bands ranging from 375 nm to 1050 nm. The spatial resolution is approximately 0.54 by 1.0 meters. The dataset also includes LiDAR elevation data from two grates, providing further information for terrain analysis. The scene covers 11 distinct land cover categories, with labeled pixels available for training and testing. The details are outlined in the Table \ref{MUUFL}.
%Additionally, to better understand the generalization capability of M$^3$amba to other multi-modal scene understanding tasks, we conduct experiments on two multi-modal (RGB-Thermal) semantic segmentation datasets, MFNet \cite{ha2017mfnet} and PST900 \cite{shivakumar2020pst900}, demonstrating that M$^3$amba performs well across various scenarios.
\begin{table}[htbp]
\centering
\small
\setlength{\tabcolsep}{0.8mm}
\renewcommand{\arraystretch}{1.3}
% \setlength{\arrayrulewidth}{1.8pt} % Set the vertical line width
% \vspace{-5pt}
\caption{Average Training Time Analysis on Three Datasets. The Best Result is \textbf{Highlighted}}
\begin{tabular}{c|ccccccc}
\hline\hline
& Ours & MiM &ExViT & MFT & Semi-ViT & HCT & CALC \\
\hline\hline
Train. (min) &\textbf{17.67} &18.71 &44.03 &42.62 &50.92 &24.28 &29.15 \\
AA (\%) &\textbf{96.10} &82.22 &83.37 &81.48 &79.63 &80.50 &74.09 \\
\hline\hline
\end{tabular}
\label{computation_cost_cls_tab}
% \vspace{-10pt}
\end{table}
\begin{table}[!h]
\centering
\small
\setlength{\tabcolsep}{3mm}
\renewcommand{\arraystretch}{1.3}
% \setlength{\arrayrulewidth}{1.8pt} % Set the vertical line width
% \vspace{-5pt}
\caption{Ablation Study of Feature Adaptation with Average Results on Three Datasets}
\begin{tabular}{c|cccc}
\hline\hline
& OA(\%) & AA(\%) & $\kappa$($\times 100$) & Train. (min) \\
\hline\hline
% (A) 95.64 90.20 94.46
% (D) 97.78 95.58 97.21
Infer & 95.23 & 90.19 & 94.17 & 15.82 \\
Fine-tune & 98.10 & 95.36 & 97.40 & 46.08 \\
Ours & 97.78 & 95.58 & 97.21 & 17.67 \\
\hline\hline
\end{tabular}
\label{ablation_feature_adaptation}
% \vspace{-10pt}
\end{table}
\subsubsection{Evaluation metrics}
To evaluate the performance in classification, we employ three metrics: Overall Accuracy (OA), Average Accuracy (AA), and the kappa ($\kappa$). OA measures the ratio of correctly classified samples to the total number of samples. AA represents the average accuracy across all classes. The $\kappa$ coefficient is a statistical
metric assessing the agreement between the classification map generated by the model and the ground truth.
\subsubsection{Implement details}
All experiments are conducted on a server equipped with an NVIDIA A100 Tensor Core GPU. For hyperspectral remote sensing images, due to their dense objects and high resolution, the data samples are usually cropped to a size of 32 × 32 and use patch size 1. For all datasets, we use ViT-B/16 as the backbone of CLIP,
and the training process employs the AdamW optimizer with an initial learning rate set to 1e-4, a weight decay of 1e-3, and a batch size of 8, spanning 200 epochs.
% In addition, a step scheduler with a step size of 50 and a gamma value of 0.9 is utilized.
% the patch sizes are 2 and 16 respectively, and the latent dimension is 384.
% Additionally, a step scheduler with a step size of 50 and a gamma value of 0.9 was utilized.
% \begin{table}[htbp]
% \centering
% \small
% \setlength{\tabcolsep}{0.2mm}
% \caption{Comparison results on the test set of MFNet dataset. The best result is \textbf{highlighted}.}
% \begin{tabular}{c|cccccccc|c}
% \hline
% Method & Car & Person & Bike & Curve & Car Stop & Guar. & Cone & Bump & mIoU \\
% \hline
% MFNet$_{17}$ &65.9 &58.9 &42.9 &29.9 &9.9 &0.0 &25.2 &27.7 &39.7 \\
% RTFNet$_{19}$ &87.4 &70.3 &62.7 &45.3 &29.8 &0.0 &29.1 &55.7 &53.2 \\
% FEANet$_{21}$ &87.8 &71.1 &61.1 &46.5 &22.1 &6.6 &55.3 &48.9 &55.3 \\
% % CMX$_{22}$ &98.3 &90.1 &75.2 &64.5 &50.2 &35.3 &8.5 &54.2 &60.6 &59.7 \\
% EAEFNet$_{23}$ &87.6 &72.6 &63.8 &48.6 &35.0 &14.2 &52.4 &58.3 &58.9 \\
% SegMiF$_{23}$ &87.8 &71.4 &63.2 &47.5 &31.1 &6.6 &48.9 &50.3 &56.1 \\
% LASNet$_{23}$ &84.2 &67.1 &56.9 &41.1 &\textbf{39.6} &\textbf{18.9} &48.8 &40.1 &54.9 \\
% CAINet$_{24}$ &88.5 &66.3 &\textbf{68.7} &\textbf{55.4} &31.5 &9.0 &48.9 &\textbf{60.7} &58.6 \\
% \textbf{Ours} & \textbf{90.6} & \textbf{75.2} &66.2 &48.8 & 37.4 & 5.3 & \textbf{56.1} &59.5 & \textbf{59.7} \\
% \hline
% \end{tabular}
% \label{mfnet_tab}
% \end{table}
% \begin{table}[htbp]
% \centering
% \setlength{\tabcolsep}{0.8mm}
% \caption{Computational efficiency analysis on MFNet dataset. The best result is \textbf{highlighted}.}
% \begin{tabular}{c|cccccc}
% \hline
% & Ours & CAINet & LASNet & EAEFNet & EGFNet & SegMiF \\
% \hline
% Params (M) &21.40 &\textbf{12.16} &93.58 &147.31 & 62.82 & 45.60 \\
% FLOPs (G) &\textbf{100.44} &123.62 &233.81 &200.40 &201.29 & 332.03 \\
% mIoU (\%) &\textbf{59.7} &58.6 &54.9 &58.9 &55.3 & 56.1 \\
% \hline
% \end{tabular}
% \label{computation_cost_seg_tab}
% \end{table}
%\begin{table}[htbp]
% \centering
% \setlength{\tabcolsep}{0.7mm}
% % \vspace{-5pt}
%
% \begin{tabular}{c|ccccc}
% \hline
% & Ours & CAINet & LASNet & EAEFNet & EGFNet \\
% \hline
% Params (M) &21.40 &\textbf{12.16} &93.58 &147.31 & 62.82 \\
% FLOPs (G) &\textbf{100.44} &123.62 &233.81 &200.40 &201.29 \\
% mIoU (\%) &\textbf{59.70} &58.60 &54.90 &58.90 &55.30 \\
% \hline
% \end{tabular}
% \caption{Computational efficiency analysis on MFNet dataset. The best result is \textbf{highlighted}.}
% \label{computation_cost_seg_tab}
% \vspace{-10pt}
%\end{table}
\begin{table*}[htbp]
% \vspace{-0.3in}
\small
\renewcommand{\arraystretch}{1.3}
\centering
\setlength{\tabcolsep}{1.1mm}{
\caption{OA, AA and Kappa Coefficient on the Houston2013 Dataset by Considering HSI and LiDAR Data. The Best Result is \textbf{Highlighted}. H and L Respectively Indicate that Our Method is Trained Using Only HSI or LiDAR Data.}
\label{tab-vsh}
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c}
\hline
\hline
~ & \textbf{Class Name} & \textbf{CALC$_{23}$} & \textbf{HCT$_{23}$} & \textbf{UACL$_{24}$} & \textbf{Semi-ViT$_{22}$} & \textbf{MFT$_{23}$} & \textbf{ExViT$_{23}$} & \textbf{MiM$_{24}$}&\textbf{S$^{2}$Mamba$_{24}$} & \textbf{M$^3$amba} & \textbf{H} & \textbf{L}\\
% ~ & Pulisher & TGRS(20) & TGRS(22) & IGARSS(21) & TGRS(23) & IF(23) & IF(24) & TGRS(23) & \\
\hline
\hline
1 & Healthy grass & 78.63 & 97.34 & 93.68 & 82.59 & 82.34 & 82.91 & 94.89 & 83.10 & \textbf{100.00} & 95.19 & 89.73 \\
2 & Stressed grass & 83.83 & 96.62 & 97.10 & 82.33 & 88.78 & 98.68 & 98.27 & \textbf{100.00} & \textbf{100.00} & 98.13 & 84.66 \\
3 & Synthetic grass & 93.86 & 84.75 & 99.84 & 97.43 & 98.15 & 99.60 & \textbf{100.00} & 99.60 & \textbf{100.00} & 92.12 & 91.67 \\
4 & Tree & 86.55 & 96.78 & 96.82 & 92.93 & 94.35 & 99.15 & 96.87 & 98.20 & \textbf{100.00} & 79.58 & 81.32 \\
5 & Soil & 99.72 &\textbf{100.00} & 99.55 & 99.84 & 99.12 & 99.91 & 99.86 & \textbf{100.00} & 99.90 & 97.13 & 88.29 \\
6 & Water & 97.90 & 96.50 & 96.82 & 84.15 & 99.30 & 99.30 & \textbf{99.65} & 95.80 & 92.65 & 80.45 & 73.59 \\
7 & Residential & 91.42 & 82.09 & 93.29 & 87.84 & 88.56 & 96.08 & 91.42 & 89.37 & \textbf{99.81} & 81.94 & 83.57 \\
8 & Commercial & 92.88 & 95.54 & 87.31 & 79.93 & 86.89 & 90.03 & 87.17 & 88.60 & \textbf{97.93} & 88.39 & 87.48 \\
9 & Road & 87.54 & 90.84 & 93.66 & 82.94 & 87.91 & 86.12 & 84.31 & 92.45 & \textbf{98.87} & 81.47 & 75.84 \\
10 & Highway & 68.53 & 58.88 & 95.26 & 52.93 & 64.70 & 72.97 & 92.38 & 92.57 & \textbf{99.48}& 89.39 & 86.53 \\
11 & Railway & 93.36 & 97.53 & 96.39 & 80.99 & 98.64 & 88.99 & 90.03 & 91.56 & \textbf{99.20} & 87.47 & 83.07 \\
12 & Park lot 1 & \textbf{95.10} & 90.11 & 93.52 & 91.07 & 94.24 & 90.39 & 89.58 & 90.97 & 73.50 & 70.18 & 52.95 \\
13 & Park lot 2 & 92.98 & 97.19 & 97.13 & 87.84 & 90.29 & 90.18 & 93.08 & 89.12 & \textbf{99.25} & 86.63 & 67.32 \\
14 & Tennis court & \textbf{100.00} & \textbf{100.00} & \textbf{100.00} & \textbf{100.00} & 99.73 & 99.60 & 97.44 & \textbf{100.00} & 99.69 & 91.19 & 84.25 \\
15 & Running track & 99.37 & \textbf{100.00} & 99.56 & 99.65 & 99.58 & 95.14 & 98.33 & \textbf{100.00} & 99.57 & 96.25 & 84.61 \\
\hline
~ & OA(\%) & 88.97 & 91.15 & 95.37 & 85.46 & 89.80 & 91.40 & 92.89 & 93.36 & \textbf{97.31} & 86.59 & 79.97 \\
~ & AA(\%) & 90.78 & 92.28 & 95.99 & 86.71 & 91.51 & 92.60 & 94.21 & 94.09 & \textbf{97.32} & 87.70 & 80.99 \\
~ & $\kappa$($\times 100$) & 88.06 & 90.40 & 95.00 & 86.71 & 88.89 & 90.66 & 92.28 & 92.79 & \textbf{97.11} & 86.33 & 80.71 \\
\hline
\hline
\end{tabular}}
\end{table*}
\begin{figure*}[htbp]
\centering
\includegraphics[width=1\linewidth]{zuotu/houston2013_1.0_new.pdf}
\centering
% \vspace{-0.3in}
\caption{Visualization of false-color HSI and LiDAR images using different comparison methods based on the Houston2013 dataset. H and L respectively indicate that our method is trained using only HSI or LiDAR data.}
\vspace{-0.1in}
\label{houston2013}
\end{figure*}
\subsection{Ablation Study}
\subsubsection{Ablation analysis of different components}
To evaluate the individual contributions of different components in our proposed method, we conduct ablation studies on three remote sensing datasets. Specifically, we design four schemes: (A) directly concatenating multi-modal images to replace the output of the feature adaptation stage; (B) not using our Cross-SS2D module and using the original SS2D module to process both inputs and concatenate the output to the fusion branch; (C) removing unsupervised consistency loss; (D) the complete M$^3$amba. The results of the ablation study are shown in Table \ref{ablation_tab}. We observe that M$^3$amba achieves optimal performance across the three datasets. In scheme (A), directly using the original image information to guide the fusion leads to an average OA reduction of 2.14\% on the three datasets, because the modality adapter reduces the inter-domain differences between different modalities, making each modality more accurate in specific tasks. When this module is removed, the fusion process of the modalities is affected, resulting in insufficient information interaction, which affects the overall classification effect. The effectiveness of our cross-attention fusion module Cross-SS2D is demonstrated in (B). Without the Cross-SS2D fusion module, OA, AA, and $\kappa$ decreased by 1.36\%, 3.17\%, and 1.73\% on average, respectively. This is because Cross-SS2D improves the synchronization and complementarity between modal features through the interaction of selective scanning and global information. After removing this module, the model cannot effectively fuse the information of the two modalities, resulting in a decrease in classification accuracy. The removal of the unsupervised consistency loss in scheme (C) also led to a significant decrease in performance. This is because the unsupervised consistency loss makes the features between the two modalities more matched by promoting the consistency of cross-modal features. After removing this loss function, the complementary information between the modalities cannot be effectively synchronized, resulting in inconsistency in information between the modalities, which affects the classification accuracy after fusion. In summary, our proposed CLIP feature adaptation and Mamba fusion method can perform joint optimization and effectively improve the performance of multi-modal learning. Additionally, to more intuitively demonstrate the impact of different components on performance,
% we extracted feature maps and used t-SNE to visualize the final classification results, as shown in Figure \ref{ablation_fig} and Figure \ref{featuremap_fig}, respectively.
%we visualize the classification map and use t-SNE to visualize the final classification results, as shown in Figure \ref{ablation_fig} and \ref{ablation_classmap}.
we use t-SNE to visualize the final classification results, as shown in Figure \ref{ablation_fig}.
The visualization results of different schemes have the same trend as the results in the table. In general, our M$^3$amba achieves complete feature fusion under the guidance of rich semantics. Comprehensive ablation experiments confirm that M$^3$amba achieves optimal results in both accuracy and visualization.
\begin{table*}[htbp]
% \vspace{-0.2in}
% \renewcommand{\arraybackslash}{1.2}
\small
\renewcommand{\arraystretch}{1.3}
\centering
% \vspace{0.2in}
\setlength{\tabcolsep}{1.1mm}{
\caption{OA, AA and Kappa Coefficient on the Augsburg Dataset by Considering HSI and LiDAR Data. The Best Result is \textbf{Highlighted}. H and L Respectively Indicate that Our Method is Trained Using Only HSI or LiDAR Data.}
\label{tag-Augsburg}
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c}
\hline\hline
~ & \textbf{Class Name} & \textbf{CALC$_{23}$} & \textbf{HCT$_{23}$} & \textbf{UACL$_{24}$} & \textbf{Semi-ViT$_{22}$} & \textbf{MFT$_{23}$} & \textbf{ExViT$_{23}$} & \textbf{MiM$_{24}$}&\textbf{S$^{2}$Mamba$_{24}$} & \textbf{M$^3$amba} & \textbf{H} & \textbf{L} \\
\hline\hline
1 & Forest & 94.34 & 94.23 & 93.66 & 90.41 & 94.65 & 92.89 & 93.91 & 93.90 & \textbf{99.63} & 96.13 & 96.88 \\
2 & Commercial Area & 98.24 & 98.54 & 90.70 & 92.64 & 96.90 & 97.28 & 88.61 & 90.55 &\textbf{99.24} & 91.59 & 64.10 \\
3 & Residential Area & 78.07 & 43.79 & 68.91 & 60.41 & 69.80 & 64.44 & 60.93 & 69.09 & \textbf{85.19} & 78.32 & 82.71 \\
4 & Industrial Area & 94.57 & 95.33 & 92.38 & 83.40 & 93.98 & 86.63 & 86.55 & 92.70 & \textbf{99.60} & 97.63 & 55.17 \\
5 & Low Plants & 28.68 & 67.88 & 56.58 & 59.41 & 32.70 & 57.74 & 68.29 & 70.30 & \textbf{95.39} & 79.59 & 61.35 \\
6 & Allotment & 2.20 & 4.82 & 47.63 & 18.44 & 10.52 & 17.28 & 24.46 & 25.84 & \textbf{81.29} & 70.23 & 29.65 \\
7 & Water & 42.27 & 52.09 & 70.26 & 27.51 & 23.98 & 26.14 & 50.24 & 50.49 & \textbf{92.89} & 85.48 & 63.76 \\
\hline
~ & OA(\%) & 91.73 & 90.90 & 89.19 & 86.64 & 90.49 & 87.91 & 88.63 & 89.34 & \textbf{98.19} & 89.36 & 77.11 \\
~ & AA(\%) & 62.62 & 65.24 & 74.30 & 64.67 & 60.36 & 63.20 & 67.57 & 70.41 & \textbf{93.32} &85.57 & 64.80 \\
~ & $\kappa$($\times 100$) & 87.98 & 86.83 & 84.80 & 84.05 & 86.26 & 82.82 & 86.56 & 89.45 & \textbf{97.39} & 88.10 & 78.79 \\
\hline\hline
\end{tabular}}
\end{table*}
\begin{figure*}[htbp]
% \vspace{-0.1in}
\centering
\includegraphics[width=1\linewidth]{zuotu/Augsburg_1.0_new.pdf}
\centering
% \vspace{-0.2in}
\caption{Visualization of false-color HSI and LiDAR images using different comparison methods based on the Augsburg dataset. H and L respectively indicate that our method is trained using only HSI or LiDAR data.}
% \vspace{-0.1in}
\label{Augsburg_2.0}
\end{figure*}
%\begin{figure}[htbp]
% \centering
% \includegraphics[width=1\linewidth]{ablation_classmap.pdf}
% \caption{False-color HSI and LiDAR image visualization for ablation studies on three datasets.}
% \label{ablation_classmap}
%\end{figure}
\subsubsection{Ablation analysis of feature adaptation}
To further explore the effectiveness of semantically guided fusion through feature adaptation, we set up three schemes, as shown in Table \ref{ablation_feature_adaptation}. We use ViT-B/16 as the backbone of CLIP. Infer refers to directly using the pre-trained CLIP image encoder to infer multi-modal images, and Fine-tune refers to fine-tuning the CLIP image encoder. We introduce modality-specific adapters to train only a very small number of parameters. It can be observed that direct inference can bring a small improvement in training efficiency, but the lack of modality-specific scene understanding leads to a significant decrease in accuracy. This problem can be alleviated by fine-tuning, but the time cost is high. Our method aims to balance training efficiency and the performance of downstream tasks, achieving the best balance.
%\begin{table*}[htbp]
% \centering
% \setlength{\tabcolsep}{1.8mm}
% \vspace{-5pt}
%
% \begin{tabular}{c|ccccccccc|c}
% \hline
% \textbf{Method} & \textbf{Unlabeld} & \textbf{Car} & \textbf{Person} & \textbf{Bike} & \textbf{Curve} & \textbf{Car Stop} & \textbf{Guardrail} & \textbf{Color Cone} & \textbf{Bump} & \textbf{mIoU}(\%) \\
% \hline
% % MFNet$_{17}$ &96.9 &65.9 &58.9 &42.9 &29.9 &9.9 &0.0 &25.2 &27.7 &39.7 \\
% % RTFNet$_{19}$ &\textbf{98.5} &87.4 &70.3 &62.7 &45.3 &29.8 &0.0 &29.1 &55.7 &53.2 \\
% % FEANet$_{21}$ &98.3 &87.8 &71.1 &61.1 &46.5 &22.1 &6.6 &55.3 &48.9 &55.3 \\
% % CMX$_{22}$ &98.3 &90.1 &75.2 &64.5 &50.2 &35.3 &8.5 &54.2 &60.6 &59.7 \\
% FDCNet$_{22}$ &98.2 &87.5 &72.4 &61.7 &43.8 &27.2 &7.3 &52.0 &56.6 &56.3 \\
% EAEFNet$_{23}$ &97.6 &87.6 &72.6 &63.8 &48.6 &35.0 &14.2 &52.4 &58.3 &58.9 \\
% SegMiF$_{23}$ &98.1 &87.8 &71.4 &63.2 &47.5 &31.1 &6.6 &48.9 &50.3 &56.1 \\
% LASNet$_{23}$ &97.4 &84.2 &67.1 &56.9 &41.1 &\textbf{39.6} &\textbf{18.9} &48.8 &40.1 &54.9 \\
% EGFNet$_{23}$ &98.0 &87.6 &69.8 &58.8 &42.8 &33.8 &7.0 &48.3 &47.1 &54.8 \\
% ECFNet$_{24}$ &98.0 &85.7 &73.5 &59.7 &45.7 &36.7 &4.0 &47.4 &55.1 &56.2 \\
% CAINet$_{24}$ &\textbf{98.4} &88.5 &66.3 &\textbf{68.7} &\textbf{55.4} &31.5 &9.0 &48.9 &\textbf{60.7} &58.6 \\
% % MRFS$_{24}$ &\textbf{98.6} &89.4 &\textbf{75.4} &65.0 &49.0 &37.2 &5.4 &53.1 &58.8 &59.1 \\
% \textbf{M$^3$amba(Ours)} &\textbf{98.4} & \textbf{90.6} & \textbf{75.2} &66.2 &48.8 & 37.4 & 5.3 & \textbf{56.1} &59.5 & \textbf{59.7} \\
% \hline
% \end{tabular}
% \caption{Comparison results on the test set of MFNet dataset. The best result is \textbf{highlighted}.}
% \label{mfnet_tab}
% \vspace{-10pt}
%\end{table*}
\begin{table*}[htbp]
% \vspace{0.3in}
% \renewcommand{\arraybackslash}{1.2}
\small
\renewcommand{\arraystretch}{1.3}
\centering
\setlength{\tabcolsep}{0.9mm}{
\caption{OA, AA and Kappa Coefficient on the MUUFL Dataset by Considering HSI and LiDAR Data. The Best Result is \textbf{Highlighted}. H and L Respectively Indicate that Our Method is Trained Using Only HSI or LiDAR Data.}
\label{tab-M}
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c}
\hline\hline
~ & \textbf{Class Name} & \textbf{CALC$_{23}$} & \textbf{HCT$_{23}$} & \textbf{UACL$_{24}$} & \textbf{Semi-ViT$_{22}$} & \textbf{MFT$_{23}$} & \textbf{ExViT$_{23}$} & \textbf{MiM$_{24}$}&\textbf{S$^{2}$Mamba$_{24}$} & \textbf{M$^3$amba} & \textbf{H} & \textbf{L}\\
\hline\hline
1 & Trees & 97.31 & 97.03 & 91.29 & 97.89 & 97.90 & 98.58 & 98.64 & 98.93 & \textbf{99.57} & 91.13 & 96.79 \\
2 & Grass-Pure & \textbf{93.00} & 90.29 & 82.06 & 79.71 & 92.11 & 87.70 & 92.85 & 88.05 & 92.50 & 87.04 & 86.07 \\
3 & Grass-Groundsurface & 91.57 & 90.07 & 77.79 & 84.67 & 91.80 & 90.96 & 92.54 & 91.31 & \textbf{96.18} & 90.60 & 89.35 \\
4 & Dirt-And-Sand & 95.10 & 94.18 & 90.13 & 89.40 & 91.59 & 90.61 & 92.33 & 90.96 & \textbf{95.95} & 93.79 & 81.41 \\
5 & Road-Materials & 95.91 & 93.86 & 88.16 & 93.81 & 95.60 & 94.73 & 96.34 & 95.08 & \textbf{98.88} & 94.78 & 85.97 \\
6 & Water &99.32 & 95.71 & 98.50 & 81.04 & 88.19 & 93.68 & 88.93 & 94.03 & \textbf{99.43} & 92.75 & 80.19 \\
7 & Buildings’Shadow & \textbf{92.69} & 87.09 & 91.45 & 83.92 & 90.27 & 90.05 & 91.01 & 90.40 & 92.32 & 80.10 & 72.91 \\
8 & Buildings & 98.45 & 96.61 & 92.91 & 98.11 & 97.26 & 97.76 & 98.00 & 98.11 & \textbf{99.44} & 97.40 & 99.32 \\
9 & Sidewalk &51.60 & 46.35 & 75.71 & 59.12 & 61.35 & 68.54 & 62.09 & 68.89 & \textbf{92.92}& 85.37 & 51.32 \\
10 & Yellow-Curb & 0.00 & 18.97 & \textbf{96.07} & 14.37 & 17.43 & 23.56 & 18.17 & 23.91 & 94.30 & 84.65 & 11.49 \\
11 & ClothPanels & 0.00 & 75.39 & \textbf{96.18} & 80.86 & 72.79 & 80.86 & 73.53 & 81.21 & 95.61 & 90.19 & 78.93 \\
\hline
~ & OA(\%) & 93.94 & 92.95 & 88.29 & 92.46 & 94.34 & 94.37 & 92.65 & 94.19 & \textbf{97.84}& 88.91 & 84.16 \\
~ & AA(\%) & 74.09 & 80.50 & 89.11 & 79.63 & 81.48 & 83.37 & 82.22 & 83.72 & \textbf{96.10} &89.80 & 75.80 \\
~ & $\kappa$($\times 100$) & 92.00 & 90.69 & 84.78 & 89.49 & 92.51 & 92.54 & 91.59 & 91.98 & \textbf{97.13} & 85.75 & 83.78 \\
\hline\hline
\end{tabular}}
\end{table*}
\begin{figure*}[htbp]
% \vspace{0.1in}
\centering
\includegraphics[width=1\linewidth]{zuotu/muufl_2.0_new.pdf}
\centering
% \vspace{-0.2in}
\caption{Visualization of false-color HSI and LiDAR images using different comparison methods based on the MUUFL Gulfport scene dataset. H and L respectively indicate that our method is trained using only HSI or LiDAR data.}
% \vspace{-0.1in}
\label{muufl_2.0}
\end{figure*}
\subsection{Comparisons with Previous Methods}
To better demonstrate the superiority of our method, we conduct a comprehensive performance comparison with recent advanced multi-modal fusion methods, including CNN-based structures CALC \cite{lu2023coupled}, HCT \cite{zhao2022joint}, UACL \cite{ding2024uncertainty}; Transformer-based structures Semi-ViT \cite{cai2022semi}, MFT \cite{roy2023multimodal}, ExViT \cite{yao2023extended}; and Mamba-based structures MiM \cite{zhou2024mamba}, S$^2$Mamba \cite{wang2024s}. The subscript of each method in the comparative experiment table represents the publication time of this method, for example, the subscript 24 means it was published in 2024. The average results are summarized in Table \ref{hsi_tab}, with our model achieving significant improvements in overall accuracy (OA), average accuracy (AA), and the kappa ($\kappa$) coefficient across all datasets.
%In the Houston2013 dataset, as shown in Table \ref{houston2013}, M$^3$amba achieves the highest overall performance, which is a significant improvement over the closest competitor, UACL. The superior performance of M$^3$amba can be attributed to its CLIP-driven modality-specific adapters, which allow for better cross-modal semantic understanding compared to CNN-based methods. Notably, M$^3$amba demonstrates a robust performance across nearly all land cover classes, achieving perfect classification accuracy for similar categories like Stressed grass and Synthetic grass. This can be explained by the cross-modal attention mechanism (Cross-SS2D), which effectively captures complementary information between HSI and LiDAR data, ensuring more consistent and complete feature representations.
In the Houston2013 dataset, as shown in Table \ref{tab-vsh}, M$^3$amba achieves the highest overall performance, which is a significant improvement over the closest competitor, UACL. The superior performance of M$^3$amba can be attributed to its CLIP-driven modality-specific adapters, which allow for better cross-modal semantic understanding compared to CNN-based methods. This is especially important in this dataset, where high spectral resolution and spatial detail are critical for accurate classification. For example, M$^3$amba achieves perfect classification accuracy for similar categories like Stressed grass and Synthetic grass, which often pose a challenge in terms of distinguishing subtle spectral differences. The CLIP-driven modality-specific adapters ensure that the model can better differentiate these two categories by refining the feature representations of both HSI and LiDAR modalities. This highlights how M$^3$amba’s ability to capture complementary information from different modalities—thanks to its Cross-SS2D module—improves classification accuracy for fine-grained classes. Additionally, M$^3$amba demonstrates robust performance across nearly all land cover classes, but its significant improvement is particularly noticeable in categories like Highway and Road, where the fusion of spectral and spatial information provided by the model enhances feature extraction and reduces misclassification.
%The Augsburg dataset presents a more challenging scenario, but M$^3$amba again surpasses all competing methods with an OA of 98.19\%, an AA of 93.32\%, and a kappa coefficient of 97.39\%, as shown in Table \ref{tag-Augsburg}.
The Augsburg dataset presents a more challenging scenario, but M$^3$amba again surpasses all competing methods with an OA of 98.19\%, an AA of 93.32\%, and a kappa coefficient of 97.39\%, as shown in Table \ref{tag-Augsburg}. One key reason for this outstanding performance is the ability of the Cross-SS2D module to maintain feature integrity across different data modalities. For example, in the Forest category, M$^3$amba outperforms competitors by leveraging both the spectral details from HSI and the structural information from LiDAR, which is crucial for distinguishing different types of vegetation. Similarly, in Industrial areas, where spatial complexity and high variability in land cover types are prevalent, M$^3$amba shows consistent improvements by modeling the interactions between spectral and spatial features more effectively than the CALC model.
The MUUFL dataset contains a mix of urban and natural land cover categories, as shown in Table \ref{tab-M}, M$^3$amba outperforms all other methods, achieving an OA of 97.84\%, an AA of 96.10\%, and a kappa coefficient of 97.13\%. Compared to the MiM model, which attains an OA of 92.65\%, M$^3$amba offers a 5.19\% improvement. In particular, M$^3$amba performs well in the sidewalk class, achieving an accuracy of 92.92\%, far exceeding other comparison methods. These results highlight the effectiveness of the Cross-SS2D module in maintaining the integrity of fine-grained features across different data modalities and its ability to model long sequences, which is a key advantage over other Mamba-based methods such as MiM and S$^2$Mamba. In the Buildings category, the model’s performance is also significantly enhanced. The fusion of HSI and LiDAR data provides a richer representation of building structure and surface features, which is often difficult to achieve with single-modal approaches. The model’s ability to effectively leverage both spatial and spectral features allows it to better capture complex building shapes and variations in urban environments.
%The average comparison results on the Houston2013, Augsburg, and MUUFL datasets are shown in Table \ref{hsi_tab}. Our evaluation indicates that the proposed method achieves the highest OA, AA, and $\kappa$ scores across the three multi-modal classification tasks, surpassing other methods. Specifically, the OA metric on the three datasets showed performance improvements of at least 1.94\%, 6.46\%, and 3.47\%, respectively.
%In summary, M$^3$amba achieves comprehensive performance improvements across the three multi-modal datasets, making it an effective solution for multi-modal classification.
To investigate the specific contribution of the complementarity of different modal data to the fusion performance, we train M$^3$amba using only HSI or LiDAR data and compare it with the results of multi-modal data training. The results in Tables \ref{tab-vsh}, \ref{tag-Augsburg}, \ref{tab-M} show that when combining the two data modalities, the spectral advantages of HSI and the spatial structure advantages of LiDAR can be simultaneously exploited to improve classification accuracy in complex scenes. For example, although the spectral features of objects such as buildings may be similar in HSI, the elevation information of LiDAR helps to accurately identify these objects. Objects with obvious spectral differences, such as water and soil, have significantly improved accuracy in the fused classification results. We can also find that the complementarity of the two modalities allows some categories to reach 100\% accuracy after fusion, which does not happen when using only single modal data.
Furthermore, as shown in Figure \ref{boxplot_fig}, the box plot of our method is taller and more compact than the competing methods, indicating that our method achieves robust detection performance for each class. This effectively mitigates the issue of low detection accuracy for individual classes observed in the competing methods.
%\begin{table}[htbp]
% \centering
% \setlength{\tabcolsep}{1.0mm}
% % \vspace{-5pt}
%
% \begin{tabular}{c|ccccc|c}
% \hline
% \textbf{Method} & \textbf{B.G.} & \textbf{H.D.} & \textbf{B.P.} & \textbf{Ex.} & \textbf{Sur.} & \textbf{mIoU}(\%) \\
% \hline
% % MFNet$_{17}$ &98.6 &41.1 &64.3 &60.4 &20.7 &57.0 \\
% % GMNet$_{21}$ &99.4 &85.2 &83.8 &73.8 &78.4 &84.1 \\
% % DSGBINet$_{22}$ &99.4 &75.0 &85.1 &79.3 &75.6 &82.9 \\
% FDCNet$_{22}$ &99.2 &70.4 &72.2 &71.5 &72.4 &77.1 \\
% EAEFNet$_{23}$ &\textbf{99.7} &80.4 &87.7 &\textbf{84.0} &76.2 &85.6 \\
% SegMiF$_{23}$ &99.3 &66.0 &81.4 &76.3 &75.5 &79.7 \\
% LASNet$_{23}$ &99.5 &82.8 &86.5 &77.8 &75.5 &84.4 \\
% EGFNet$_{23}$ &99.6 &76.1 &\textbf{90.6} &80.0 &80.9 &85.4 \\
% ECFNet$_{24}$ &99.1 &73.1 &84.1 &71.5 &73.7 &80.3 \\
% CAINet$_{24}$ &99.5 &80.3 &88.0 &77.2 &78.7 &84.7 \\
% \textbf{M$^3$amba(Ours)} &99.6 & \textbf{87.6} & \textbf{90.6} & 80.1 & \textbf{83.4} & \textbf{88.3} \\
% \hline
% \end{tabular}
% \caption{Comparison results on the test set of PST900 dataset. The best result is \textbf{highlighted}.}
% \label{pst900_tab}
% \vspace{-5pt}
%\end{table}
%\noindent\textbf{Segmentation}
%M$^3$amba shows potential in the task of natural image segmentation. To further demonstrate the generalizability of our method, we select multi-modal semantic segmentation datasets in the domains of urban scenes. We compare our approach against 7 state-of-the-art multi-modal semantic segmentation methods on the MFNet and PST900 datasets. These methods include ECFNet \cite{shen2024ecfnet}, EAEFNet \cite{liang2023explicit}, SegMiF \cite{liu2023multi}, LASNet \cite{li2022rgb}, CAINet \cite{lv2024context}, FDCNet \cite{zhao2022feature}, and EGFNet \cite{dong2023egfnet}. Table \ref{mfnet_tab} and Table \ref{pst900_tab} show the semantic segmentation results for each class on the MFNet and PST900 datasets, respectively. Our method outperforms all other methods on both datasets, achieving performance improvements of at least 0.8\% and 2.7\%, respectively, demonstrating the strong generalizability and superiority of our approach. The complete class names in Table \ref{pst900_tab} are Background, Hand-Drill, Backpack, Fire-Extinguisher, and Survivor respectively.
In addition to the accuracy metrics, our method also demonstrates a notable reduction in training time compared to other models, as shown in Table \ref{computation_cost_cls_tab}. Our method achieves the lowest average training time across the three datasets, outperforming CNN-based, Transformer-based, and Mamba-based architectures and far exceeds all other methods in terms of average accuracy. This is attributed to the design of our efficient fusion architecture, in stark contrast to the quadratic complexity in traditional Transformer models. The combination of efficient feature fusion and comprehensive semantic guidance makes M$^3$amba both an accurate and computationally efficient model for multi-modal remote sensing tasks.
%To analyze the training efficiency, a comparison of training time for three different architectures including CNN-based, Transformer-based, and Mamba-based methods is shown in Table \ref{computation_cost_cls_tab}. Our method has lower training time than all other models and far exceeds all other methods in terms of average accuracy.
%In addition, to evaluate the computational efficiency of the compared methods, we conduct a detailed statistical analysis of the MFNet dataset, as shown in Table \ref{computation_cost_seg_tab}. The metrics include the number of parameters involved in training and the floating-point operations (FLOPs). The summarized results in the table indicate that our method achieves optimal performance with the smallest FLOPs, although it has slightly more parameters than CAINet. Overall, our method offers an acceptable level of complexity and strikes a good balance between efficiency and performance.
% \begin{figure}[htbp]
% \centering
% \includegraphics[width=0.9\linewidth]{houston.pdf}
% \caption{Visualization of false-color HSI and LiDAR images using M$^3$amba based on the Houston2013 dataset.}
% \label{houston_fig}
% \end{figure}
% \begin{figure*}[t]
% \centering
% \includegraphics[width=1\linewidth]{feature.pdf}
% \caption{Feature maps for ablation of the MUUFL dataset.}
% \label{featuremap_fig}
% \end{figure*}
% \begin{figure}[t]
% \centering
% \includegraphics[width=1.0\linewidth]{computation_mfnet.pdf}
% \caption{.}
% \label{computation_mfnet_fig}
% \end{figure}
\subsection{Analysis of Failure Cases}
Although M$^3$amba achieves the highest accuracy on multiple categories in multiple datasets, there are still cases of poor classification, especially in complex scenes, where some categories perform poorly, partly due to the ambiguity of features or spectral overlap. Through the analysis of failed cases, we find the following two points. Texture fuzzy categories: such as Water (in Table \ref{tab-vsh}), because its texture features are similar to the surrounding environment in hyperspectral images, lead to classification errors. Especially in LiDAR data, these objects lack significant elevation differences, which affects the judgment of the model. Complex scene problems: In densely built-up areas in cities, objects such as parking lots (in Table \ref{tab-vsh}) lack clear spatial structure and have similar spectral features to the surrounding environment (such as roads and buildings), resulting in reduced classification accuracy.
\subsection{Result Visualization}
We visualize the results by assigning a unique color to each class. Figure \ref{houston2013}, Figure \ref{Augsburg_2.0}, and Figure \ref{muufl_2.0} demonstrate the excellent performance of our method on the full remote sensing image dataset, and our visualization is still better than most of the comparison methods even when trained with only single-modal data. M$^3$amba extracts fundamental properties of multi-modal images by establishing comprehensive fused features and invariant representations. It utilizes CLIP to generate comprehensive semantic features to guide the efficient fusion of multi-modal features. This enhances the model's generalization ability and preserves complete information between modalities, resulting in richer and more detailed representations in the classification maps. Overall, M$^3$amba is highly suitable for generating more robust and intricate detailed multi-modal classification maps.
% \begin{figure}[htbp]
% \centering
% \includegraphics[width=1.0\linewidth]{muufl.pdf}
% \caption{Visualization of false-color HSI and LiDAR images using M$^3$amba based on the MUUFL dataset.}
% \label{muufl_fig}
% \end{figure}
% \begin{figure}[htbp]
% \centering
% \includegraphics[width=1.0\linewidth]{augsburg.pdf}
% \caption{Visualization of false-color HSI and LiDAR images using M$^3$amba based on the Augsburg dataset.}
% \label{augursburg_fig}
% \end{figure}
\section{Conclusion}
In this paper, we introduce M$^3$amba, a novel end-to-end CLIP-driven Mamba unified framework for multi-modal fusion, combining CLIP's powerful multi-modal visual semantic representation capabilities with the efficiency of SSM. By introducing CLIP-driven modality-specific adapters and the Mamba fusion architecture embedded with the cross-attention module Cross-SS2D, M$^3$amba addresses the primary challenges of existing methods, including high computational complexity, limited generalization, and incomplete feature fusion, achieving optimal performance in both accuracy and efficiency. Extensive evaluations on multiple remote sensing multi-modal datasets demonstrate the generalizability and interpretability of M$^3$amba. This work showcases the tremendous potential for future research in efficient and effective multi-modal learning, highlighting the potential of combining pre-trained multi-modal models with state space modeling techniques to advance the field of multi-modal fusion. In addition, due to the efficient training and excellent performance of M$^3$amba, in future work, we will focus on verifying the effectiveness of M$^3$amba in specific application scenarios, especially in multiple practical scenarios such as agricultural monitoring, urban planning, and ecological protection, as well as exploring new directions such as dynamic target detection and real-time multi-modal data processing, to further demonstrate its wide applicability and strong potential in practical remote sensing tasks.
%\ifCLASSOPTIONcaptionsoff
%\newpage
%\fi
%{
\ifCLASSOPTIONcaptionsoff
\newpage
\fi
{
\bibliographystyle{IEEEtran}
\bibliography{refs.bib}
}
%}
%\vspace{-30pt}
%\vspace{10pt}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{Mingxiang_Cao}}]{Mingxiang Cao} received the B.E. degree in Telecommunications Engineering from Xidian University, Xi’an, China in 2023. He is currently pursuing the M.S. degree with the Image Coding and Processing Center at State Key Laboratory of Integrated Service Network, Xidian University, Xi'an, China. His research interests include multimodal image processing, remote sensing classification, and object detection.
\end{IEEEbiography}
%\vspace{-0.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{Weiying_Xie}}]{Weiying Xie}(Senior Member, IEEE) received the Ph.D. degree in communication and information systems from Xidian University, Xi’an, China, in 2017. Currently, she is a Professor with the State Key Laboratory of Integrated Services Networks, Xidian University. She has published over 50 articles in refereed journals and proceedings, including IEEE Transactions on Image Processing, IEEE Transactions on Geoscience and Remote Sensing, IEEE Transaction on Neural Networks and Learning Systems, IEEE Transactions on Cybernetics, and Conference on IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) and Association for the Advancement of Artificial Intelligence (AAAI). Her research interests include neural networks, machine learning, hyperspectral image processing, and high-performance computing.
\end{IEEEbiography}
%\vspace{-1.5cm}
%\vspace{13pt}
%\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{Daixun_Li}}]{Daixun Li}
%received the B.E. degree in telecommunications engineering from Xidian University, Xi’an, China, in 2022, where he is currently pursuing the M.S. degree with the Image Coding and Processing Center, State Key Laboratory of Integrated Service Network.,His research interests include multimodal image processing, remote sensing classification, and federated learning.
%\end{IEEEbiography}
%\vspace{13pt}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{zhangxin}}]{Xin Zhang}
received the B.E. degree in Telecommunications Engineering from Xidian University, Xi'an, China in 2019. She is currently pursuing the Ph.D. degree with the Image Coding and Processing Center at State Key Laboratory of Integrated Services Networks, Xidian University, Xi'an, China. Her research interests span efficient deep learning, machine learning, and computer vision. Specifically, she is interested in model compression for computer vision models (CNN, ViT), knowledge distillation to both models and datasets, and general CV tasks (foundational model training and downstream applications).
\end{IEEEbiography}
%\vspace{13pt}
%\vspace{-6.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{zhangjiaqing}}]{Jiaqing Zhang}
received the B.E. degree in Telecommunications Engineering from Ningbo University, Zhejiang, China in 2019. She is currently pursuing the Ph.D. degree with the Image Coding and Processing Center at State Key Laboratory of Integrated Service Network, Xidian University, Xi'an, China.
Her research interests include multimodal image processing, remote sensing object detection, and network compression.
\end{IEEEbiography}
%\vspace{13pt}
%\vspace{-5.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{jiangkai}}]{Kai Jiang}
received the B.E. degree in information engineering and the Ph.D. degree in communication and information systems from Xidian University, Xi’an, China, in 2019 and 2024, respectively. His research interests include image processing and deep learning.
\end{IEEEbiography}
\vspace{-10.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{leijie}}]{Jie Lei}
received his M.S. degree in Telecommunications and Information Systems and his Ph.D. degree in Signal and Information Processing from Xidian University, China, in 2006 and 2010, respectively. He was a Visiting Scholar at the Department of Computer Science at the University of California, Los Angeles, USA, from 2014 to 2015. He served as a Professor at the School of Telecommunications Engineering, Xidian University, until 2023. Currently, he is a Research Fellow at the School of Electrical and Data Engineering at the University of Technology Sydney.
His research interests include wireless communication, remote sensing image processing, machine learning, and customized computing for big data applications.
\end{IEEEbiography}
\vspace{-10.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=2in,clip,keepaspectratio]{Yunsong_Li}}]{Yunsong Li}
(Member, IEEE) received the M.S. degree in telecommunication and information systems and the Ph.D. degree in signal and information processing from Xidian University, China, in 1999 and 2002, respectively. He joined the School of Telecommunications Engineering, Xidian University in 1999 where he is currently a Professor. Prof. Li is the director of the image coding and processing center at the State Key Laboratory of Integrated Service Networks. His research interests focus on image and video processing and high-performance computing.
\end{IEEEbiography}
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{zhao2023tufusion}
Y.~Zhao, Q.~Zheng, P.~Zhu, X.~Zhang, and W.~Ma, ``Tufusion: A transformer-based
universal fusion algorithm for multimodal images,'' \emph{IEEE Transactions
on Circuits and Systems for Video Technology (TCSVT)}, vol.~34, pp.
1712--1725, 2023.
\bibitem{li2024feddiff}
D.~Li, W.~Xie, Z.~Wang, Y.~Lu, Y.~Li, and L.~Fang, ``Feddiff: Diffusion model
driven federated learning for multi-modal and multi-clients,'' \emph{IEEE
Transactions on Circuits and Systems for Video Technology (TCSVT)}, 2024.
\bibitem{ma2023mbsi}
M.~Ma, W.~Ma, L.~Jiao, X.~Liu, F.~Liu, L.~Li, and S.~Yang, ``Mbsi-net:
Multimodal balanced self-learning interaction network for image
classification,'' \emph{IEEE Transactions on Circuits and Systems for Video
Technology (TCSVT)}, vol.~34, pp. 3819--3833, 2023.
\bibitem{li2024mdfl}
D.~Li, W.~Xie, J.~Zhang, and Y.~Li, ``Mdfl: Multi-domain diffusion-driven
feature learning,'' in \emph{Proceedings of the AAAI Conference on Artificial
Intelligence (AAAI)}, vol.~38, no.~8, 2024, pp. 8653--8660.
\bibitem{xie2024fusionmamba}
X.~Xie, Y.~Cui, C.-I. Ieong, T.~Tan, X.~Zhang, X.~Zheng, and Z.~Yu,
``Fusionmamba: Dynamic feature enhancement for multimodal image fusion with
mamba,'' \emph{arXiv preprint arXiv:2404.09498}, 2024.
\bibitem{zhang2024simfusion}
X.~Zhang, A.~Liu, G.~Yang, Y.~Liu, and X.~Chen, ``Simfusion: A semantic
information-guided modality-specific fusion network for mr images,''
\emph{Information Fusion (IF)}, p. 102560, 2024.
\bibitem{radford2021learning}
A.~Radford, J.~W. Kim, C.~Hallacy, A.~Ramesh, G.~Goh, S.~Agarwal, G.~Sastry,
A.~Askell, P.~Mishkin, J.~Clark \emph{et~al.}, ``Learning transferable visual
models from natural language supervision,'' in \emph{Proceedings of the
IEEE/CVF International Conference on Machine Learning (ICML)}.\hskip 1em plus
0.5em minus 0.4em\relax PMLR, 2021, pp. 8748--8763.
\bibitem{lin2022frozen}
Z.~Lin, S.~Geng, R.~Zhang, P.~Gao, G.~De~Melo, X.~Wang, J.~Dai, Y.~Qiao, and
H.~Li, ``Frozen clip models are efficient video learners,'' in
\emph{Proceedings of the IEEE/CVF European Conference on Computer Vision
(ECCV)}.\hskip 1em plus 0.5em minus 0.4em\relax Springer, 2022, pp. 388--404.
\bibitem{yang2024inf}
Z.~Yang, H.~Jiang, W.~Hong, J.~Teng, W.~Zheng, Y.~Dong, M.~Ding, and J.~Tang,
``Inf-dit: Upsampling any-resolution image with memory-efficient diffusion
transformer,'' \emph{arXiv preprint arXiv:2405.04312}, 2024.
\bibitem{gao2024clip}
P.~Gao, S.~Geng, R.~Zhang, T.~Ma, R.~Fang, Y.~Zhang, H.~Li, and Y.~Qiao,
``Clip-adapter: Better vision-language models with feature adapters,''
\emph{International Journal of Computer Vision (IJCV)}, vol. 132, no.~2, pp.
581--595, 2024.
\bibitem{zhang2020deep}
C.~Zhang, Y.~Cui, Z.~Han, J.~T. Zhou, H.~Fu, and Q.~Hu, ``Deep partial
multi-view learning,'' \emph{IEEE Transactions on Pattern Analysis and
Machine Intelligence (TPAMI)}, vol.~44, no.~5, pp. 2402--2415, 2020.
\bibitem{ma2022swinfusion}
J.~Ma, L.~Tang, F.~Fan, J.~Huang, X.~Mei, and Y.~Ma, ``Swinfusion: Cross-domain
long-range learning for general image fusion via swin transformer,''
\emph{IEEE/CAA Journal of Automatica Sinica (IJAS)}, vol.~9, no.~7, pp.
1200--1217, 2022.
\bibitem{gu2023mamba}
A.~Gu and T.~Dao, ``Mamba: Linear-time sequence modeling with selective state
spaces,'' \emph{arXiv preprint arXiv:2312.00752}, 2023.
\bibitem{zhai2022scaling}
X.~Zhai, A.~Kolesnikov, N.~Houlsby, and L.~Beyer, ``Scaling vision
transformers,'' in \emph{Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR)}, 2022, pp. 12\,104--12\,113.
\bibitem{riquelme2021scaling}
C.~Riquelme, J.~Puigcerver, B.~Mustafa, M.~Neumann, R.~Jenatton,
A.~Susano~Pinto, D.~Keysers, and N.~Houlsby, ``Scaling vision with sparse
mixture of experts,'' \emph{Advances in Neural Information Processing Systems
(NeurIPS)}, vol.~34, pp. 8583--8595, 2021.
\bibitem{devlin2018bert}
J.~Devlin, M.-W. Chang, K.~Lee, and K.~Toutanova, ``Bert: Pre-training of deep
bidirectional transformers for language understanding,'' \emph{arXiv preprint
arXiv:1810.04805}, 2018.
\bibitem{he2022masked}
K.~He, X.~Chen, S.~Xie, Y.~Li, P.~Doll{\'a}r, and R.~Girshick, ``Masked
autoencoders are scalable vision learners,'' in \emph{Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 2022,
pp. 16\,000--16\,009.
\bibitem{xie2022simmim}
Z.~Xie, Z.~Zhang, Y.~Cao, Y.~Lin, J.~Bao, Z.~Yao, Q.~Dai, and H.~Hu, ``Simmim:
A simple framework for masked image modeling,'' in \emph{Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 2022,
pp. 9653--9663.
\bibitem{jia2021scaling}
C.~Jia, Y.~Yang, Y.~Xia, Y.-T. Chen, Z.~Parekh, H.~Pham, Q.~Le, Y.-H. Sung,
Z.~Li, and T.~Duerig, ``Scaling up visual and vision-language representation
learning with noisy text supervision,'' in \emph{Proceedings of the IEEE/CVF
International Conference on Machine Learning (ICML)}.\hskip 1em plus 0.5em
minus 0.4em\relax PMLR, 2021, pp. 4904--4916.
\bibitem{ye2023ip}
H.~Ye, J.~Zhang, S.~Liu, X.~Han, and W.~Yang, ``Ip-adapter: Text compatible
image prompt adapter for text-to-image diffusion models,'' \emph{arXiv
preprint arXiv:2308.06721}, 2023.
\bibitem{ramesh2022hierarchical}
A.~Ramesh, P.~Dhariwal, A.~Nichol, C.~Chu, and M.~Chen, ``Hierarchical
text-conditional image generation with clip latents,'' \emph{arXiv preprint
arXiv:2204.06125}, vol.~1, no.~2, p.~3, 2022.
\bibitem{mohammad2022clip}
N.~Mohammad~Khalid, T.~Xie, E.~Belilovsky, and T.~Popa, ``Clip-mesh: Generating
textured meshes from text using pretrained image-text models,'' in
\emph{SIGGRAPH Asia 2022 Conference Papers}, 2022, pp. 1--8.
\bibitem{sanghi2022clip}
A.~Sanghi, H.~Chu, J.~G. Lambourne, Y.~Wang, C.-Y. Cheng, M.~Fumero, and K.~R.
Malekshan, ``Clip-forge: Towards zero-shot text-to-shape generation,'' in
\emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2022, pp. 18\,603--18\,613.
\bibitem{xu2023dream3d}
J.~Xu, X.~Wang, W.~Cheng, Y.-P. Cao, Y.~Shan, X.~Qie, and S.~Gao, ``Dream3d:
Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image
diffusion models,'' in \emph{Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR)}, 2023, pp. 20\,908--20\,918.
\bibitem{luo2022clip4clip}
H.~Luo, L.~Ji, M.~Zhong, Y.~Chen, W.~Lei, N.~Duan, and T.~Li, ``Clip4clip: An
empirical study of clip for end to end video clip retrieval and captioning,''
\emph{Neurocomputing}, vol. 508, pp. 293--304, 2022.
\bibitem{luo2023towards}
D.~Luo, J.~Huang, S.~Gong, H.~Jin, and Y.~Liu, ``Towards generalisable video
moment retrieval: Visual-dynamic injection to image-text pre-training,'' in
\emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2023, pp. 23\,045--23\,055.
\bibitem{luo2024zero}
D.~\vspace{0mm}Luo, J.~Huang, S.~Gong, H.~Jin, and Y.~Liu, ``Zero-shot video
moment retrieval from frozen vision-language models,'' in \emph{Proceedings
of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
2024, pp. 5464--5473.
\bibitem{zhai2022lit}
X.~Zhai, X.~Wang, B.~Mustafa, A.~Steiner, D.~Keysers, A.~Kolesnikov, and
L.~Beyer, ``Lit: Zero-shot transfer with locked-image text tuning,'' in
\emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2022, pp. 18\,123--18\,133.
\bibitem{li2023blip}
J.~Li, D.~Li, S.~Savarese, and S.~Hoi, ``Blip-2: Bootstrapping language-image
pre-training with frozen image encoders and large language models,'' in
\emph{Proceedings of the IEEE/CVF International Conference on Machine
Learning (ICML)}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2023, pp.
19\,730--19\,742.
\bibitem{dong2024changeclip}
S.~Dong, L.~Wang, B.~Du, and X.~Meng, ``Changeclip: Remote sensing change
detection with multimodal vision-language representation learning,''
\emph{ISPRS Journal of Photogrammetry and Remote Sensing (ISPRS)}, vol. 208,
pp. 53--69, 2024.
\bibitem{mei2023lightweight}
S.~Mei, G.~Zhang, N.~Wang, B.~Wu, M.~Ma, Y.~Zhang, and Y.~Feng, ``Lightweight
multiresolution feature fusion network for spectral super-resolution,''
\emph{IEEE Transactions on Geoscience and Remote Sensing (TGRS)}, vol.~61,
pp. 1--14, 2023.
\bibitem{mei2023rotation}
S.~Mei, R.~Jiang, M.~Ma, and C.~Song, ``Rotation-invariant feature learning via
convolutional neural network with cyclic polar coordinates convolutional
layer,'' \emph{IEEE Transactions on Geoscience and Remote Sensing (TGRS)},
vol.~61, pp. 1--13, 2023.
\bibitem{roy2023multimodal}
S.~K. Roy, A.~Deria, D.~Hong, B.~Rasti, A.~Plaza, and J.~Chanussot,
``Multimodal fusion transformer for remote sensing image classification,''
\emph{IEEE Transactions on Geoscience and Remote Sensing (TGRS)}, vol.~61,
pp. 1--20, 2023.
\bibitem{lu2023coupled}
T.~Lu, K.~Ding, W.~Fu, S.~Li, and A.~Guo, ``Coupled adversarial learning for
fusion classification of hyperspectral and lidar data,'' \emph{Information
Fusion (IF)}, vol.~93, pp. 118--131, 2023.
\bibitem{ding2024uncertainty}
K.~Ding, T.~Lu, and S.~Li, ``Uncertainty-aware contrastive learning for
semi-supervised classification of multimodal remote sensing images,''
\emph{IEEE Transactions on Geoscience and Remote Sensing (TGRS)}, 2024.
\bibitem{zhang2024multimodal}
J.~Zhang, J.~Lei, W.~Xie, G.~Yang, D.~Li, and Y.~Li, ``Multimodal informative
vit: Information aggregation and distribution for hyperspectral and lidar
classification,'' \emph{IEEE Transactions on Circuits and Systems for Video
Technology (TCSVT)}, 2024.
\bibitem{wang2023mutually}
J.~Wang and X.~Tan, ``Mutually beneficial transformer for multimodal data
fusion,'' \emph{IEEE Transactions on Circuits and Systems for Video
Technology (TCSVT)}, vol.~33, no.~12, pp. 7466--7479, 2023.
\bibitem{dong2023joint}
W.~Dong, T.~Yang, J.~Qu, T.~Zhang, S.~Xiao, and Y.~Li, ``Joint contextual
representation model-informed interpretable network with dictionary aligning
for hyperspectral and lidar classification,'' \emph{IEEE Transactions on
Circuits and Systems for Video Technology (TCSVT)}, vol.~33, no.~11, pp.
6804--6818, 2023.
\bibitem{wang2025s3f2net}
X.~Wang, L.~Song, Y.~Feng, and J.~Zhu, ``S3f2net: Spatial-spectral-structural
feature fusion network for hyperspectral image and lidar data
classification,'' \emph{IEEE Transactions on Circuits and Systems for Video
Technology (TCSVT)}, 2025.
\bibitem{mei2021hyperspectral}
S.~Mei, X.~Li, X.~Liu, H.~Cai, and Q.~Du, ``Hyperspectral image classification
using attention-based bidirectional long short-term memory network,''
\emph{IEEE Transactions on Geoscience and Remote Sensing (TGRS)}, vol.~60,
pp. 1--12, 2021.
\bibitem{mei2022learning}
S.~Mei, Y.~Geng, J.~Hou, and Q.~Du, ``Learning hyperspectral images from rgb
images via a coarse-to-fine cnn,'' \emph{Science China Information Sciences
(SCIS)}, vol.~65, pp. 1--14, 2022.
\bibitem{yao2024spectralmamba}
J.~Yao, D.~Hong, C.~Li, and J.~Chanussot, ``Spectralmamba: Efficient mamba for
hyperspectral image classification,'' \emph{arXiv preprint arXiv:2404.08489},
2024.
\bibitem{wang2024s}
G.~Wang, X.~Zhang, Z.~Peng, T.~Zhang, X.~Jia, and L.~Jiao,
``S\textsuperscript{2} mamba: A spatial-spectral state space model for
hyperspectral image classification,'' \emph{arXiv preprint arXiv:2404.18213},
2024.
\bibitem{zhou2024mamba}
W.~Zhou, S.-I. Kamata, H.~Wang, M.-S. Wong \emph{et~al.}, ``Mamba-in-mamba:
Centralized mamba-cross-scan in tokenized mamba model for hyperspectral image
classification,'' \emph{arXiv preprint arXiv:2405.12003}, 2024.
\bibitem{liu2024vmamba}
Y.~Liu, Y.~Tian, Y.~Zhao, H.~Yu, L.~Xie, Y.~Wang, Q.~Ye, and Y.~Liu, ``Vmamba:
Visual state space model,'' \emph{arXiv preprint arXiv:2401.10166}, 2024.
\bibitem{chen2021crossvit}
C.-F.~R. Chen, Q.~Fan, and R.~Panda, ``Crossvit: Cross-attention multi-scale
vision transformer for image classification,'' in \emph{Proceedings of the
IEEE/CVF International Conference on Computer Vision (ICCV)}, 2021, pp.
357--366.
\bibitem{debes2014hyperspectral}
C.~Debes, A.~Merentitis, R.~Heremans, J.~Hahn, N.~Frangiadakis, T.~van
Kasteren, W.~Liao, R.~Bellens, A.~Pi{\v{z}}urica, S.~Gautama \emph{et~al.},
``Hyperspectral and lidar data fusion: Outcome of the 2013 grss data fusion
contest,'' \emph{IEEE Journal of Selected Topics in Applied Earth
Observations and Remote Sensing (JSTARS)}, vol.~7, no.~6, pp. 2405--2418,
2014.
\bibitem{hong2021multimodal}
D.~Hong, J.~Hu, J.~Yao, J.~Chanussot, and X.~X. Zhu, ``Multimodal remote
sensing benchmark datasets for land cover classification with a shared and
specific feature learning model,'' \emph{ISPRS Journal of Photogrammetry and
Remote Sensing (ISPRS)}, vol. 178, pp. 68--80, 2021.
\bibitem{gader2013muufl}
P.~Gader, A.~Zare, R.~Close, J.~Aitken, and G.~Tuell, ``Muufl gulfport
hyperspectral and lidar airborne data set,'' \emph{Univ. Florida,
Gainesville, FL, USA, Tech. Rep. REP-2013-570}, 2013.
\bibitem{zhao2022joint}
G.~Zhao, Q.~Ye, L.~Sun, Z.~Wu, C.~Pan, and B.~Jeon, ``Joint classification of
hyperspectral and lidar data using a hierarchical cnn and transformer,''
\emph{IEEE Transactions on Geoscience and Remote Sensing (TGRS)}, vol.~61,
pp. 1--16, 2022.
\bibitem{cai2022semi}
Z.~Cai, A.~Ravichandran, P.~Favaro, M.~Wang, D.~Modolo, R.~Bhotika, Z.~Tu, and
S.~Soatto, ``Semi-supervised vision transformers at scale,'' \emph{Advances
in Neural Information Processing Systems (NeurIPS)}, vol.~35, pp.
25\,697--25\,710, 2022.
\bibitem{yao2023extended}
J.~Yao, B.~Zhang, C.~Li, D.~Hong, and J.~Chanussot, ``Extended vision
transformer (exvit) for land use and land cover classification: A multimodal
deep learning framework,'' \emph{IEEE Transactions on Geoscience and Remote
Sensing (TGRS)}, vol.~61, pp. 1--15, 2023.
\end{thebibliography}