Isolated Channel Vision Transformers: From Single-Channel Pretraining to Multi-Channel Finetuning
Extracted source text · arXiv 2503.09826
\documentclass{bmvc2k}
\usepackage{booktabs}
% \usepackage{caption}
% \usepackage{subcaption}
\usepackage{graphicx}
\usepackage{amssymb}
%% Enter your paper number here for the review copy
%\bmvcreviewcopy{1132}
\title{Isolated Channel Vision Transformers:\\ From Single-Channel Pretraining to Multi-Channel Finetuning}
% Enter the paper's authors in order
% \addauthor{Name}{email/homepage}{INSTITUTION_CODE}
\addauthor{Wenyi Lian}{wenyi.lian@it.uu.se}{1}
\addauthor{Patrick Micke}{patrick.micke@igp.uu.se}{2}
\addauthor{Joakim Lindblad}{joakim.lindblad@it.uu.se}{1}
\addauthor{Nata\v{s}a Sladoje}{natasa.sladoje@it.uu.se}{1}
% Enter the institutions
% \addinstitution{Name\\Address}
\addinstitution{
Department of Information Technology\\
Uppsala University\\
Uppsala, Sweden
}
\addinstitution{
Department of Immunology, Genetics and Pathology\\
Uppsala University\\
Uppsala, Sweden
}
\runninghead{Lian et al.}{Isolated Channel Vision Transformers (IC-ViT)}
% Any macro definitions you would like to include
% These are not defined in the style file, because they don't begin
% with \bmva, so they might conflict with the user's own macros.
% The \bmvaOneDot macro adds a full stop unless there is one in the
% text already.
\def\eg{\emph{e.g}\bmvaOneDot}
\def\Eg{\emph{E.g}\bmvaOneDot}
\def\etal{\emph{et al}\bmvaOneDot}
%-------------------------------------------------------------------------
% Document starts here
\begin{document}
\maketitle
\input{sec/0_abstract}
\input{sec/1_intro}
\input{sec/2_related_work}
\input{sec/3_method}
\input{sec/4_experiments}
\input{sec/5_analysis}
\input{sec/6_conclusion}
%------------------------------------------------------------------------
\bibliography{main}
\end{document}
\begin{abstract}
Vision Transformers (ViTs) have achieved remarkable success in standard RGB image analysis. However, applying ViTs to multi-channel imaging (MCI) data, \eg, for medical and remote sensing applications, remains a challenge. In particular, MCI data often consist of layers acquired from different modalities. Directly training ViTs on such data can obscure modality/channel-specific information and impair performance.
In this paper, we propose \textit{Isolated Channel ViT (IC-ViT)}, a simple yet effective training framework for large-scale MCI data. By randomly sampling one channel per image per iteration, IC-ViT learns channel-specific representations without requiring multi-channel fusion during pretraining. These representations are later integrated during finetuning to capture cross-channel dependencies in downstream tasks.
Experiments on various benchmarks, including JUMP-CP and CHAMMI for cell microscopy, and So2Sat-LCZ42 for satellite imaging, show the proposed IC-ViT outperforms existing channel-adaptive approaches by 4–14\%. Moreover, its efficient training makes it a suitable candidate for large-scale pretraining of foundation models on heterogeneous data. Our code is available at \url{https://github.com/shermanlian/IC-ViT}.
\end{abstract}
\section{Introduction}
\label{sec:intro}
Self-supervised learning (SSL) of Vision Transformers (ViTs), as demonstrated by methods like DINO~\cite{caron2021emerging} and DINOv2~\cite{oquab2023dinov2}, has shown clear advantages~\citep{oquab2023dinov2,xu2024whole,awais2025foundation} in learning generalizable representations. The application of an SSL model typically consists of two stages: (1) pretraining on large-scale unlabeled data to learn general representations; and (2) fine-tuning on downstream datasets to address specific, task-oriented problems~\cite{ericsson2022self,zhou2024comprehensive}. However, a key limitation of ViT-based SSL models is that, in both pretraining and fine-tuning stages, they are typically designed for standard RGB images and cannot naturally handle inputs with more than three channels or heterogeneous channel configurations, as commonly encountered in multi-channel imaging (MCI)~\cite{cai2020review}.
% Multi-channel imaging (MCI) is increasingly important in domains such as medical imaging and remote sensing, as it enables the capture of richer and more detailed information compared to traditional grayscale or RGB imaging~\cite{dimitrovski2023current,cai2020review,wu2016deep}. However, the diversity in channel types and modalities across MCI datasets presents significant challenges in developing effective methods to extract and integrate information from heterogeneous channels~\cite{misra2020multi,lee2019mcsip,liu2017deep,nie2019multi}.
A common strategy to address this challenge is to finetune large-scale pretrained ViTs on MCI data, typically by modifying the patch embedding layer to accommodate the varying number of input channels. However, directly transferring ViTs pretrained on RGB data to multi-modal MCI datasets often leads to mismatches in input representation~\cite{de2021efficient}, resulting in suboptimal exploitation of the available multi-channel information~\cite{zhou2024comprehensive}.
Recent works have thus shifted focus toward modifying the ViT architecture to support multi-channel inputs~\citep{savarese2019learning,bao2024channel,bourriez2024chada,pham2024enhancing,chen2023chammi}. These approaches, commonly referred to as \textit{channel-adaptive ViTs}, extract embeddings independently from each input channel and concatenate them into a single sequence to preserve channel-specific information. However, this expansion in sequence length substantially raises computational costs and significantly prolongs training time, posing a critical challenge in large-scale self-supervised learning~\cite{bao2024channel}.
\begin{figure}[t]
%\setlength{\abovecaptionskip}{-0.1in}
\setlength{\belowcaptionskip}{-0.1in}
\begin{center}
\includegraphics[width=1.\linewidth]{figs/teaser_3.pdf}
% \vspace{0.5em}
\caption{
\textbf{Left:} Visualization of the attention maps generated from the standard Vision Transformer (ViT)~\cite{dosovitskiy2021an}, ChannelViT (C-ViT)~\cite{bao2024channel}, and the proposed \textbf{I}solated \textbf{C}hannel \textbf{ViT} (IC-ViT). Note that ViT compromises all channels into one patch token while ChannelViT and IC-ViT generate patch tokens for each channel individually. Our IC-ViT tends to extract information from both fluorescence and brightfield channels, while the latter is often ignored by ChannelViT.
\textbf{Right:} Performance comparison on the JUMP-CP validation dataset~\cite{chandrasekaran2024three}.
}
\label{fig:teaser}
\end{center}
\end{figure}
In this paper, we present \textbf{Isolated Channel ViT (IC-ViT)}, a simple yet effective self-supervised learning method tailored for multi-channel imaging data. Specifically, IC-ViT is pretrained by learning representations from only a single channel per image at each iteration, rather than relying on all channels simultaneously. This isolated-channel strategy significantly reduces the computational cost of pretraining while enabling effective adaptation to multi-channel downstream tasks. While pretraining focuses on single-channel representations, fine-tuning on downstream tasks allows IC-ViT to integrate information across channels, leading to significantly better task performance.
As illustrated in \autoref{fig:teaser} (left), the attention maps generated by IC-ViT for an eight-channel JUMP-CP microscopy image indicate the model’s ability to capture informative features from both modalities. In addition, the comparison of training curves in the right panel of \autoref{fig:teaser} further confirms its effectiveness for downstream classification. Moreover, the scalability of IC-ViT to large-scale MCI datasets with diverse channel and modality compositions underscores its potential as a foundation model for MCI applications.
% In summary, our key contributions are as follows: 1) We propose a simple yet effective self-supervised learning framework IC-ViT for multi-channel images; 2) Our results show that ViTs pretrained on individual channels can still learn robust representations, supporting channel-wise patchifying as an effective design choice for multi-channel inputs; 3) The proposed IC-ViT effectively handles inputs with varying channel configurations, showing its applicability to diverse multi-channel imaging scenarios; 4) Extensive experiments and analysis on different datasets and tasks illustrate the effectiveness of the proposed method.
% \begin{itemize}
% \item We propose a simple yet effective self-supervised learning framework IC-ViT for multi-channel images.
% \item Our results show that ViTs pretrained on individual channels can still learn robust representations, supporting channel-wise patchifying as an effective design choice for multi-channel inputs.
% \item The proposed IC-ViT effectively handles inputs with varying channel configurations, showing its applicability to diverse multi-channel imaging scenarios.
% \item Extensive experiments and analysis on different datasets and tasks illustrate the effectiveness of the proposed method.
% \end{itemize}
\section{Related work}
\label{sec:related_work}
% \paragraph{Self-supervised Learning}
% Self-supervised learning (SSL) has become increasingly popular due to its effectiveness in learning high-quality representations without requiring costly annotations of large-scale datasets~\cite{wang2022self,krishnan2022self,hendrycks2019using,he2019moco,chen2020mocov2}. A pioneering SSL method is SimCLR~\cite{chen2020simple,chen2020big}, which effectively learns meaningful representations from unlabeled data by maximizing similarity between different augmented views of the same sample through contrastive learning. Following the success of contrastive learning on Convolutional Neural Networks (CNNs)~\cite{madhusudana2022image}, researchers have explored alternative SSL approaches that eliminate the need for explicit negative samples and adapt better to other architectures, such as ViTs~\cite{atito2021sit,he2019moco,bao2022beit}. DINO~\cite{caron2021emerging} is the first method to utilize self-distillation for SSL of ViTs, effectively learning representations without requiring explicit negative samples. It adopts a teacher-student approach in which the student network is trained by minimizing differences with a teacher network. DINOv2 \cite{oquab2023dinov2} further improves upon DINO by incorporating masked image modeling techniques into training, enhancing architecture, and leveraging larger-scale data. Masked Autoencoder (MAE)~\cite{MaskedAutoencoders2021} is another SSL method for ViTs that does not require negative samples and learns by reconstructing heavily masked image patches.
\begin{figure}[t]
% \setlength{\abovecaptionskip}{-0.1in}
\setlength{\belowcaptionskip}{-0.1in}
\centering
\includegraphics[width=.62\linewidth]{figs/correlation_map.pdf}
% \vspace{0.5em}
\caption{Inter-channel correlation analysis on the JUMP-CP dataset~\cite{chandrasekaran2024three}.
\textbf{Top:} Pairwise correlations of channel tokens and features across eight microscopy channels. \textbf{Bottom:} Example of a multi-channel microscopy image and its channel-wise attention maps, obtained from a ViT pretrained with DINO on single-channel inputs.}
\label{fig:corr_map}
\end{figure}
%\citet{scheibenreif2022self} demonstrated that applying self-supervised learning (SSL) to Vision Transformers (ViTs) for land-cover segmentation and classification leads to significant performance improvements, with fine-tuned models achieving up to 30\% higher accuracy than their supervised counterparts.
%Recently, DINO has found common use as an encoder for pretraining foundation models~\cite{awais2025foundation, xu2024whole}, benefiting from the label-free nature of self-supervised learning and global feature extraction capabilities of ViT. While this approach is highly effective on large-scale datasets, it faces challenges when applied to images with varying numbers of channels, missing modalities, or incomplete information. Some kind of channel-adaptive ViT is needed for such heterogeneous input settings.
%-------------------------------------------------------------------------
% \paragraph{Channel-adaptive Vision Transformers}
While ViTs~\cite{dosovitskiy2021an} have been highly successful, they have mainly been applied to RGB images~\cite{han2022survey,thisanke2023semantic}. However, multi-channel imaging is finding increased use in many fields, since it captures richer information beyond traditional RGB~\cite{misra2020multi,lee2019mcsip,liu2017deep,nie2019multi}.
This has led to the development of channel-adaptive models for multi-channel data~\cite{savarese2019learning, ha2016hypernetworks, chen2023chammi}. Recently, ClimaX~\cite{nguyen2023climax} is presented as a variant of the ViT that employs variable tokenization and aggregation methods to process heterogeneous weather and climate data. DepthwiseViT~\cite{chen2023chammi} proposes to process each input channel separately using a depthwise convolution layer. The extracted features are then aggregated by averaging, forming a new representation that is then passed into the ViT backbone. ChannelViT~\cite{bao2024channel} and ChAda-ViT\cite{bourriez2024chada} both generate patch tokens independently for each channel and incorporate learnable channel embeddings to explicitly differentiate between channels. Additionally, ChannelViT introduces Hierarchical Channel Sampling (HCS) during training to improve training efficiency. Building upon ChannelViT, DiChaViT~\cite{pham2024enhancing} further enhances feature diversity by adopting a Diverse Channel Sampling (DCS) strategy. DCS selectively samples less similar channels, reducing redundancy and mitigating over-similarity in channel representations.
However, existing channel-adaptive methods suffer from high computational costs due to channel-to-sequence expansion.
%\newpage
\section{Method}
\label{sec:method}
% Pointing out the problems in multi-channel imaging: 1) why it's important to process each channel individually; 2) why pretraining is important or how self-supervised pretraining improves the patch tokens; 3) what changes in multi-channel ViT pretraining.
Multi-channel images often consist of layers acquired through different imaging protocols or modalities. A random example of an eight-channel image from the JUMP-CP dataset, including both fluorescence and brightfield microscopy channels, is shown at the bottom of \autoref{fig:corr_map}. These layers typically contain distinct yet complementary information that should be jointly leveraged for downstream analysis. In this section, we start from the ViT architecture and study \textit{why} channel-wise patchifying matters for multi-channel images and explore \textit{how} to efficiently extract informative features across channels.
\begin{figure}[t]
% \setlength{\belowcaptionskip}{-0.15in}
\centering
\includegraphics[width=0.65\linewidth]{figs/vit_patchify.pdf}
% \vspace{1em}
\caption{Illustration of channel-wise patchifying in ViT. Each channel is patchified individually, and the resulting patches are embedded into a sequence of vectors, with position and `\!\texttt{[CLS]}\!' embeddings added, to form the input to the transformer. $C_i$ denotes the embeddings of the $i$th channel.}
\label{fig:vit_patchify}
\end{figure}
\subsection{Channel-wise Patchifying ViT}
\label{subsec:channel_patchify}
\paragraph{Preliminary}
Image patchifying is an important step in all vision transformers which splits the 2D image $\textbf{x} \in \mathbb{R}^{H \times W \times C}$ into $N$ patches $\textbf{x}_n \in \mathbb{R}^{P^2 \times C}$, where $P$ is the patch size and $N=HW/P^2$. ViT then maps each image patch into an embedding using a learnable linear projection $\mathbf{W}\in\mathbb{R}^{P^2C \times D}$ and concatenates all patches as a vector sequence $\mathbf{z}\in\mathbb{R}^{N \times D}$, where $D$ is the embedding dimension. In addition, a position embedding is added to retain positional information and a learnable classification token (`\!\texttt{[CLS]}\!') embedding is inserted at the start, to serve for final prediction.
\paragraph{Correlation between channels}
Before introducing the channel-wise patchifying ViT, we examine inter-channel correlations in MCI data. Specifically, we pretrain a single ViT with DINO~\cite{caron2021emerging} on single-channel inputs from the JUMP-CP microscopy dataset~\cite{chandrasekaran2024three} for self-supervised representation learning. The results are illustrated at the top of \autoref{fig:corr_map}, showing pairwise correlations of patch tokens and learned channel features across the eight channels in JUMP-CP.
We observe that tokens from fluorescence (FL) and brightfield (BF) channels exhibit lower correlations than tokens within the same modality, indicating that the two modalities provide complementary information. The high correlation observed among brightfield channels is largely due to their nearly identical visual appearance, which naïve token fusion cannot effectively differentiate. By isolating channels during pretraining, it is possible to extract subtle yet discriminative features from visually redundant channels, leading to more informative representations for downstream tasks. In contrast, projecting all channels into a single embedding is akin to ``early fusion'' in multimodal networks~\cite{ramachandram2017deep}, an approach that has been shown to be ineffective in many multimodal learning tasks~\citep{joze2020mmtm,he2023co,lian2025let}. Compared to patch tokens, the learned channel features exhibit stronger cross-modality correlations, suggesting that the model effectively learns shared representations bridging FL and BF channels. The attention maps in \autoref{fig:corr_map} (bottom) further confirm that the model attends to both FL and BF channels. These observations show the effectiveness of isolated-channel training and motivate channel-wise patchifying as a natural solution for multi-channel imaging~\citep{bao2024channel,pham2024enhancing}.
\paragraph{Channel-wise patchifying}
The main idea of channel-wise patchifying is to split the image into multiple single-channel images and patchify these images individually. The same projection layer is used for all channels. After projection, all channel patch embeddings are concatenated as the input sequence to the transformer. An overview of this process is shown in \autoref{fig:vit_patchify}. The resulting sequence $\tilde{\mathbf{z}}\in\mathbb{R}^{NC \times D}$ captures information from all channels and is $C$ times longer than the sequence in the original ViT. Moreover, a learnable channel embedding is often added to indicate the spatial index of each channel, similar to the position embedding in ViT~\cite{bao2024channel}. The transformer encoder then leverages self-attention to capture dependencies across patches and channels, yielding more robust feature representations.
\begin{figure}
% \setlength{\abovecaptionskip}{-0.1in}
% \setlength{\belowcaptionskip}{-0.2in}
\centering
\includegraphics[width=0.59\linewidth]{figs/s2m_pretraining.pdf}
% \vspace{1em}
\caption{(a) Single-channel pretraining combined. (b) Multi-channel finetuning framework. IC-ViT samples single channel images for pretraining and uses all channels for prediction.}
\label{fig:s2m_scheme}
\end{figure}
\subsection{Efficient Multi-Channel Pretraining}\label{subsec:MCI_pretraining}
We have demonstrated the importance of channel-wise patchifying in Sec.~\ref{subsec:channel_patchify} and illustrated the effectiveness of feature extraction with DINO pretraining in \autoref{fig:corr_map}. However, as noted earlier, channel-wise patchifying extends the sequence length by $C$ times, thereby leading to a quadratic growth in computational cost with respect to the number of channels due to the self-attention mechanism. This significantly limits the scalability of pretraining, particularly for self-supervised methods such as DINO~\cite{caron2021emerging} and DINOv2~\cite{oquab2023dinov2}, which process multiple augmented views per iteration and require dual networks (teacher and student) throughout training.
\paragraph{Single-channel pretraining}
Training ViTs on individual channels, as described in Sec.~\ref{subsec:channel_patchify}, has empirically shown the ability to extract informative channel representations.
Since channel-wise patchifying introduces no channel-specific parameters, the number of channels can be adapted without modifying the network architecture (ViT naturally supports variable sequence lengths).
Then, the ViT pretrained on the individual channels can be finetuned on multi-channel images (MCI) for a downstream task. We refer to this approach as \textbf{I}solated \textbf{C}hannel \textbf{ViT} (IC-ViT). It is highly efficient for MCI pretraining and facilitates transfer to downstream tasks. An overview of the framework is presented in \autoref{fig:s2m_scheme}. In the pretraining stage, one channel is randomly sampled from each image and patchified as in a standard ViT for single-channel (grayscale) inputs. DINO~\cite{caron2021emerging} is employed to learn self-supervised representations from individual channels.
During fine-tuning, channel-wise patchifying is applied to all channels simultaneously to leverage global cross-channel information.
\autoref{fig:attn-map-method} illustrates the attention maps of IC-ViT under different training settings. Across all cases, the model attends more strongly to fluorescence channels, which typically contain clearer and more discriminative information than brightfield channels. Pretraining enables the model to also capture structural information from brightfield channels, which becomes better integrated with fluorescence features after finetuning. In contrast, models trained directly from scratch fail to adequately attend to brightfield channels, resulting in inferior performance compared to the finetuned model. Further details are provided in Sec.~\ref{sec:experiments}.
\paragraph{MCI foundation models}
With its efficient pretraining and finetuning scheme, the proposed IC-ViT is scalable to large MCI datasets with diverse channels and modalities.
A key challenge in training models on multimodal multi-channel datasets is the difficulty of unifying heterogeneous image channels within the same training batch, which is required by most deep learning frameworks. Existing solutions typically adopt either (1) padding all images along the channel dimension, analogous to sentence padding in language models~\cite{sutskever2014sequence}, or (2) performing separate forward passes for different channel batches followed by gradient aggregation for parameter updates~\cite{pham2024enhancing}.
However, both approaches are rigid and constrained by variability in channel numbers.
In contrast, IC-ViT addresses this challenge by sampling a single channel from each image at a time, thereby ensuring channel uniformity within the batch while maintaining training efficiency.
\section{Experiments}
\label{sec:experiments}
In this section, we evaluate the proposed IC-ViT method on various tasks and MCI datasets.
\begin{figure}
\setlength{\belowcaptionskip}{-0.2in}
\centering
\includegraphics[width=0.6\linewidth]{figs/attention_map_method.pdf}
% \vspace{0.5em}
\caption{Attention maps on an eight-channel microscopy image from JUMP-CP dataset~\cite{chandrasekaran2024three}, generated by IC-ViT under different training settings: pretraining only (top), after supervised fine-tuning (middle), and trained directly with labels without pretraining (bottom).}
\label{fig:attn-map-method}
\end{figure}
%In this section, we evaluate the proposed IC-ViT across multiple tasks and diverse multi-channel imaging datasets, and compare its performance with state-of-the-art channel-adaptive approaches.
%-------------------------------------------------------------------------
\subsection{Datasets and Implementation}
\label{subsec:dataset}
% \paragraph{Datasets}
We evaluate IC-ViT on three publicly available multi-channel benchmarks from the medical and remote sensing domains:
1) \textit{JUMP-CP}~\cite{chandrasekaran2024three}, a widely used benchmark for cell microscopy imaging, with each image captured across eight channels (five fluorescence and three brightfield). Following previous works~\cite{bao2024channel,pham2024enhancing}, we use the compound perturbation plate `\texttt{BR00116991}', which comprises 127K training images, 45K validation images, and 45K test images.
2) \textit{CHAMMI}~\cite{chen2023chammi}, a microscopy benchmark for channel-invariant models which combines three multi-channel imaging datasets with varying channel counts: WTC-11 hiPSC (WTC-11, 3 channels, 65K images)~\cite{viana2023integrated}, Human Protein Atlas (HPA, 4 channels, 67K images)~\cite{thul2017subcellular}, and Cell Painting (CP, 5 channels, 88K images)~\cite{way2022morphology}.
3) \textit{So2Sat-LCZ42}~\cite{Zhu2020So2Sat}, a remote sensing dataset consisting of Sentinel-1 (8 radar channels) and Sentinel-2 (10 multispectral channels) imagery, with 352K training images, 24K validation images, and 24K test images.
% \begin{itemize}
% \item
% \textit{JUMP-CP}~\cite{chandrasekaran2024three}, a widely used benchmark for cell microscopy imaging. This dataset has 160 perturbation in total, with each image captured across eight channels: five fluorescence and three brightfield. It includes two main tasks: perturbation detection and perturbation matching. Following previous works~\citep{bao2024channel,pham2024enhancing}, we focus on perturbation detection and use the compound perturbation plate `\texttt{BR00116991}', which comprises 127K training images, 45K validation images, and 45K test images.
% \vspace{.5em}
% \item \textit{CHAMMI}~\cite{chen2023chammi}, a microscopy benchmark for channel-invariant models which combines three multi-channel imaging datasets with varying channel counts: WTC-11 hiPSC (WTC-11, 3 channels, 65K images)~\cite{viana2023integrated}, Human Protein Atlas (HPA, 4 channels, 67K images)~\cite{thul2017subcellular}, and Cell Painting (CP, 5 channels, 88K images)~\cite{way2022morphology}. This evaluation framework includes 9 biologically relevant tasks. We use the official evaluation code provided by the authors of CHAMMI for performance measurement~\cite{pham2024enhancing,chen2023chammi}.
% \vspace{.5em}
% \item \textit{So2Sat-LCZ42}~\cite{Zhu2020So2Sat}, a remote sensing dataset comprising high-resolution multi-spectral images from 42 cities worldwide, annotated with 17 labels reflecting various local climate zones (LCZ). This dataset offers a rich blend of synthetic aperture radar (SAR) and optical information with 8 Sentinel-1 channels and 10 Sentinel-2 channels. In our experiments of LCZ prediction, we use the second version of this dataset (referred to as the `city split' in previous works~\citep{bao2024channel,pham2024enhancing}), which consists of 352K training images, 24K validation images, and 24K test images.
% \end{itemize}
% \paragraph{Training details}
% The training of IC-ViT consists of two stages: pretraining and fine-tuning. We employ DINO~\cite{caron2021emerging} for self-supervised pretraining on unlabeled training sets. The small version of Vision Transformer (ViT-S/16) is used as the backbone. The patch size is set to 16 for all tasks except So2Sat, in which we lower the patch size to 8, to better capture high-resolution satellite details and align with previous works~\cite{bao2024channel,pham2024enhancing}.
We apply the proposed single-channel sampling strategy for pretraining in all tasks. During both pretraining and fine-tuning, only the training data are used. For fine-tuning, we follow the same experimental setup as ChannelViT~\cite{bao2024channel} and DiChaViT~\cite{pham2024enhancing} to ensure fair comparison. For example, we use cross-entropy loss for JUMP-CP and So2Sat, and proxy-based loss~\cite{teh2020proxynca} for CHAMMI. For all downstream tasks, we employ the AdamW optimizer with a learning rate schedule that starts with a warm-up phase for the first 10 epochs.
% \paragraph{Evaluation}
For both JUMP-CP and So2Sat, the epoch achieving the highest validation accuracy is chosen, and the test set performance of that checkpoint is reported as the final top-1 accuracy. For CHAMMI, we follow the official CHAMMI evaluation protocol~\cite{chen2023chammi}, using the final checkpoint and reporting macro-F1 scores. Please refer to the Supplementary Materials for additional details on datasets, training settings, and results.
% \paragraph{Comparison approaches}
% We compare the proposed IC-ViT to the following state-of-the-art channel-adaptive approaches: DepthwiseViT~\cite{chen2023chammi}, TemplateMixingViT~\cite{savarese2019learning}, HyperNetViT~\cite{ha2016hypernetworks}, ChAda-ViT~\cite{bourriez2024chada}, ChannelViT~\cite{bao2024channel}, and DiChaViT~\cite{pham2024enhancing}. To ensure a fair comparison, all models are fine-tuned under the same downstream supervision using identical loss functions and training splits. IC-ViT and all baseline methods are trained with the same task-specific objectives.
%Considering that ChannelViT uses exactly the same architecture as our IC-ViT, we regard it as the main reference method. %Following~\cite{pham2024enhancing}, the HCS~\cite{bao2024channel} is applied to all models to ensure a fair comparison.
%-------------------------------------------------------------------------
\subsection{Results}
\begin{table}[t]
\setlength{\belowcaptionskip}{-0.1in}
\begin{center}
\resizebox{.99\linewidth}{!}{
\begin{tabular}{l @{\qquad} c c c @{\qquad} c c @{\qquad} c c}
\toprule
& \multicolumn{3}{c}{CHAMMI~\cite{chen2023chammi}\quad\null}
& \multicolumn{2}{c}{JUMP-CP~\cite{chandrasekaran2024three}\qquad\null}
& \multicolumn{2}{c}{\!So2Sat~\cite{Zhu2020So2Sat}} \\
\cmidrule(r{2em}){2-4} \cmidrule(r{2em}){5-6} \cmidrule(r){7-8}
\textbf{Model}
& {WTC} & {HPA}& {Avg F1-score}
& {Full} & {Partial}
& {Full} & {Partial} \\
\midrule
DepthwiseViT~\cite{chen2023chammi}& 50.35 &\textbf{71.52}&60.94&49.86&44.98&60.41&43.41\\
TemplateMixingViT~\cite{savarese2019learning}& 49.51& 64.52 &57.02&52.48&43.85&55.86 &37.28\\
HyperNetViT~\cite{ha2016hypernetworks}& 45.17& 63.90&54.54 &47.07 &42.43& 60.73& 41.88\\
ChAda-ViT~\cite{bourriez2024chada} & 67.08 &60.67&63.88 &65.03& 42.15& 56.98& 12.38\\
ChannelViT~\cite{bao2024channel} &67.66 &62.14 &64.90 &67.51 &56.49 &61.03& 46.16 \\
DiChaViT~\cite{pham2024enhancing}& 73.36 & 63.35 & 68.36& 69.19 & 57.98 & 63.36 & 47.76 \\
IC-ViT (Ours) & \textbf{74.45}&70.76&\textbf{72.54}&\textbf{83.43}&\textbf{66.01}&\textbf{67.10}&\textbf{48.21}\\
\bottomrule
\end{tabular}}
\end{center}
\caption{Comparison of method performance on test sets of three multi-channel image benchmarks: CHAMMI, JUMP-CP, and So2Sat. Best scores are shown in \textbf{bold}.}
\label{tab:main_ret}
\end{table}
%\label{subsec:results}
%\begin{figure}[t]
% \begin{center}
% \includegraphics[width=.8\linewidth]{figs/attention_map_experiment.pdf}
% \end{center}
% \caption{Comparison of channel-wise attention maps generated by ChannelViT (C-ViT) and our IC-ViT on JUMP-CP dataset. IC-ViT better exploits information available in the channels.}
% \label{fig:attmap-experiment}
%\end{figure}
We compare the proposed IC-ViT to the following state-of-the-art channel-adaptive approaches: DepthwiseViT~\cite{chen2023chammi}, TemplateMixingViT~\cite{savarese2019learning}, HyperNetViT~\cite{ha2016hypernetworks}, ChAda-ViT~\cite{bourriez2024chada}, ChannelViT~\cite{bao2024channel}, and DiChaViT~\cite{pham2024enhancing}. For fairness, IC-ViT and all baseline methods are trained under the same experimental protocol, with identical task objectives, loss functions, and training splits. Importantly, IC-ViT employs the same number of parameters as ChannelViT, ensuring that performance differences are attributable to training strategies rather than model capacity.
The quantitative results are reported in \autoref{tab:main_ret}. For the JUMP-CP and So2Sat datasets, our evaluations cover two practical scenarios: (1) training and testing with all channels (denoted as ``Full''); and (2) training with all, but testing only with a partial set of channels (denoted as ``Partial''), simulating situations where only one modality is available at inference (the fluorescence channels for JUMP-CP and the Sentinel-1 channels for So2Sat).
It is observed that the proposed IC-ViT consistently outperforms other approaches across most tasks and datasets. On the JUMP-CP dataset, IC-ViT achieves an improvement of 14–16\% over ChannelViT and DiChaViT in full-channel settings, and 8–10\% in partial-channel settings, demonstrating both its robustness to missing channels and its overall effectiveness. %An attention map comparison between ChannelViT and IC-ViT is provided in ~\autoref{fig:attmap-experiment}.% It indicates the efficient usage of available information by the IC-ViT model.
Similarly, on the CHAMMI benchmark, our method surpasses DiChaViT, the second-best method, by 4.18\% in overall F1-score, further confirming its strong capability for robust representation learning and generalization.
\section{Analysis and Discussion}
\label{sec:analysis}
%In this section we present several ablation experiments to analyze the key properties of the proposed framework.
%-------------------------------------------------------------------------
% \begin{figure}[t]
% \centering
% \begin{minipage}[t]{0.28\linewidth}
% \centering
% \includegraphics[width=1.\linewidth]{figs/if-pre.pdf}
% \end{minipage}
% \begin{minipage}[t]{0.28\linewidth}
% \centering
% \includegraphics[width=1.\linewidth]{figs/dino-channelvit.pdf}
% % \label{fig:dino-hcsvit}
% \end{minipage}
% \begin{minipage}[t]{0.35\linewidth}
% \vspace{-.9in}
% \begin{center}
% \resizebox{1.\linewidth}{!}{
% \begin{tabular}{@{\;}lcc@{\;}}
% \toprule
% Model (S/16) & \#Channels & Time (hh:mm) \\
% \midrule
% ViT & 8 & 21:56 \\ %0 days, 21 hours, 56 minutes
% IC-ViT w/ HCS& $-$ & 86:28 \\ % 3 days, 14 hours, 28 minutes
% IC-ViT & 8 & \!\!126:53 \\ %5 days, 6 hours, 53 minutes
% IC-ViT & 5 & 56:03 \\ %2 days, 8 hours, 3 minutes
% IC-ViT & 3 & 28:53\\ %1 day, 4 hours, 53 minutes
% IC-ViT (proposed) & 1 & 09:16\\ %0 days, 9 hours, 16 minutes
% \bottomrule
% \end{tabular}}
% \end{center}
% % \label{tab:pretraining-time}
% \end{minipage}
% \caption{(Left:) Validation accuracy for IC-ViT vs. IC-ViT without DINO pretraining. (Middle) Training curves of IC-ViT and its variant using HCS strategy~\cite{bao2024channel}. (Right:) Pretraining time (in hours:minutes) across different models and channel configurations on the JUMP-CP training set. Batch sizes are adjusted based on the available memory. The number of channels in ``IC-ViT w/ HCS'' is dynamically selected (denoted with ``$-$'').}
% \label{fig:if-pre}
% \end{figure}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.28\linewidth}
\centering
\includegraphics[width=1.\linewidth]{figs/if-pre.pdf}
\end{minipage}
\begin{minipage}[t]{0.28\linewidth}
\centering
\includegraphics[width=1.\linewidth]{figs/dino-channelvit.pdf}
% \label{fig:dino-hcsvit}
\end{minipage}
\begin{minipage}[t]{0.35\linewidth}
\vspace{-.9in}
\begin{center}
\resizebox{1.\linewidth}{!}{
\begin{tabular}{@{\;}lcc@{\;}}
\toprule
Model (S/16) & \#Channels & Time (hh:mm) \\
\midrule
ViT & 8 & 21:56 \\ %0 days, 21 hours, 56 minutes
IC-ViT w/ HCS& $-$ & 86:28 \\ % 3 days, 14 hours, 28 minutes
IC-ViT & 8 & \!\!126:53 \\ %5 days, 6 hours, 53 minutes
IC-ViT & 5 & 56:03 \\ %2 days, 8 hours, 3 minutes
IC-ViT & 3 & 28:53\\ %1 day, 4 hours, 53 minutes
IC-ViT (proposed) & 1 & 09:16\\ %0 days, 9 hours, 16 minutes
\bottomrule
\end{tabular}}
\end{center}
% \label{tab:pretraining-time}
\end{minipage}
% \vspace{.1in}
\caption{(Left:) Validation accuracy for IC-ViT vs. IC-ViT without DINO pretraining. (Middle) Training curves of IC-ViT and its variant using HCS strategy~\cite{bao2024channel}. (Right:) Pretraining time (in hours:minutes) across different models and channel configurations on the JUMP-CP training set. Batch sizes are adjusted based on the available memory. The number of channels in ``IC-ViT w/ HCS'' is dynamically selected (denoted with ``$-$'').}
\label{fig:if-pre}
\end{figure}
\paragraph{Effectiveness of self-supervised learning}
% \label{subsec:ssl}
We evaluate the impact of self-supervised pretraining in our method by comparing models trained with and without DINO on the JUMP-CP training set. As shown in \autoref{fig:if-pre} (left), the model pretrained with DINO significantly outperforms its supervised-only counterpart in downstream validation accuracy, highlighting the benefits of self-supervised representations.
%-------------------------------------------------------------------------
% \begin{table}
% \begin{center}
% \resizebox{.6\linewidth}{!}{
% \begin{tabular}{@{\;}lccc@{\;}}
% \toprule
% Model (S/16) & \#Channels &Batch size& Time (hh:mm) \\
% \midrule
% ViT & 8 & 128 & 21:56 \\ %0 days, 21 hours, 56 minutes
% IC-ViT w/ HCS& $-$ & 16 & 86:28 \\ % 3 days, 14 hours, 28 minutes
% IC-ViT & 8 & 16 & \!\!126:53 \\ %5 days, 6 hours, 53 minutes
% IC-ViT & 5 & 32 & 56:03 \\ %2 days, 8 hours, 3 minutes
% IC-ViT & 3 & 64 & 28:53\\ %1 day, 4 hours, 53 minutes
% IC-ViT (proposed) & 1 & 128 & 09:16\\ %0 days, 9 hours, 16 minutes
% \bottomrule
% \end{tabular}}
% \end{center}
% \caption{Pretraining time (in hours:minutes) across different models and channel configurations on the JUMP-CP training set. Batch sizes are adjusted based on the available memory. The number of channels in `IC-ViT w/ HCS' is dynamically selected (denoted with `$-$').}
% \label{tab:pretraining-time}
% \end{table}
% \begin{table}
% \begin{center}
% \resizebox{.4\linewidth}{!}{
% \begin{tabular}{@{\;}cccc@{\;}}
% \toprule
% \#Channels & 1 & 3&5\\
% \midrule
% Accuracy&83.25&79.44 & 78.09 \\
% \bottomrule
% \end{tabular}
% }
% \end{center}
% \caption{Validation accuracy (\%) for different counts of channels randomly selected during the pretraining.}
% \label{tab:channel-count}
% \end{table}
\paragraph{Impact of channel sampling strategies on pretraining efficiency}
A major limitation of channel-adaptive approaches is the increased computational cost caused by expanding the channel dimension into the sequence length. We compare the pretraining times of ViT and IC-ViT variants employing different channel sampling strategies on the JUMP-CP training set over 100 epochs. All experiments are conducted on four 80GB A100 GPUs with consistent training configurations, except that the batch size varies with memory usage across channel settings.
As shown in \autoref{fig:if-pre} (right), pretraining IC-ViT on all channels (8 channels) without any sampling strategy takes approximately 127 hours (5 days and 7 hours). IC-ViT with Hierarchical Channel Sampling (HCS) instead samples a fixed number of channels at each iteration, thereby exposing the model to more channel information than our single-channel strategy. While incorporating HCS during DINO pretraining reduces the duration to 86 hours (3 days and 14 hours), Bao et al.~\cite{bao2024channel} reported that it can introduce instability into the self-distillation objective, which we also observe. More importantly, as shown in \autoref{fig:if-pre} (middle), although this variant achieves higher accuracy in the early stages of training, its advantage diminishes over time, resulting in comparable final performance to our method, which achieves similar results with substantially less training time (requiring only 9 hours).
% In the proposed IC-ViT, only one channel is sampled per image at each iteration, reducing the pretraining time to 9 hours and indicating the method’s potential for scaling to large datasets. We further investigate the effect of using multiple randomly selected channels during pretraining. As shown in \autoref{fig:if-pre} (right), pretraining with 3 channels takes roughly 3 times longer than with a single channel, while using 5 channels increases the time by nearly sixfold. The pretrained models are then fine-tuned on the JUMP-CP training set for task-specific evaluation.
% \autoref{tab:channel-count} reports the validation accuracy corresponding to different numbers of channels used during pretraining. Interestingly, increasing the number of channels during pretraining does not lead to better performance compared to the single-channel setup, highlighting the efficiency of the single-channel strategy without compromising downstream performance.
%-------------------------------------------------------------------------
\paragraph{Effectiveness of channel-specific representation learning}
The standard ViT~\cite{dosovitskiy2021an} treats all patches uniformly without differentiating among input channels, thereby neglecting the potentially independent semantic content carried by each channel/modality~\cite{bao2024channel}.
We pretrain both ViT and IC-ViT using DINO on the JUMP-CP training set, and subsequently fine-tune them on the same dataset for the downstream classification task. The corresponding training curves are shown in \autoref{fig:others} (left). While the standard ViT initially achieves higher accuracy, its performance stagnates after about 10 epochs. In contrast, IC-ViT continues to improve steadily, demonstrating the advantage of channel-specific representation learning.
%----------------------Robustness Evaluation on Random Channel Subsets---------------------------------------------------
\begin{table}[t]
\begin{center}
\resizebox{.63\linewidth}{!}{
\begin{tabular}{@{}c@{\,}cccc@{}}
\toprule
\#channels \
& \#comb.\
& ViT \
& ChannelViT \
& IC-ViT (Ours) \
\\
\midrule
8 &1& 56.87 &68.09 &83.41 \\
7 &8& 49.35{\color{darkgray} \scriptsize$\pm$09.38} &61.02{\color{darkgray} \scriptsize $\pm$09.78}& 79.72{\color{darkgray} \scriptsize $\pm$06.87} \\
6 &28& 42.38{\color{darkgray} \scriptsize $\pm$10.64} & 53.45{\color{darkgray} \scriptsize $\pm$12.40}& 72.42{\color{darkgray} \scriptsize$\pm$10.11}\\
5 &56& 35.78{\color{darkgray} \scriptsize$\pm$10.18} &45.50{\color{darkgray} \scriptsize$\pm$13.23} & 63.40{\color{darkgray} \scriptsize$\pm$12.31}\\
4 &70& 29.84{\color{darkgray} \scriptsize$\pm$08.32} &37.37{\color{darkgray} \scriptsize$\pm$12.25}& 52.56{\color{darkgray} \scriptsize$\pm$13.10} \\
3 &56& 24.94{\color{darkgray} \scriptsize$\pm$05.43} &29.68{\color{darkgray} \scriptsize$\pm$09.22}& 40.33{\color{darkgray} \scriptsize$\pm$11.71} \\
2 &28& 21.54{\color{darkgray} \scriptsize$\pm$02.37} &23.77{\color{darkgray} \scriptsize$\pm$04.89}& 28.80{\color{darkgray} \scriptsize$\pm$07.45} \\
1 &8& 19.92{\color{darkgray} \scriptsize$\pm$00.51}& 20.85{\color{darkgray} \scriptsize$\pm$01.64}& 21.84{\color{darkgray} \scriptsize$\pm$02.54} \\
\bottomrule
\end{tabular}
}
\end{center}
\caption{Test accuracy on JUMP-CP~\cite{chandrasekaran2024three}. `\#comb.' denotes the number of $\binom{8}{k}$ combinations of channels. The results of ViT and ChannelViT are reported from their original paper~\cite{bao2024channel}.}
\label{tab:channel-missing}
\end{table}
% \begin{figure}[t]
% \centering
% \begin{minipage}[t]{0.45\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figs/dino-vit.pdf}
% \caption{Performance curves of IC-ViT and ViT, both pretrained and finetuned on the JUMP-CP dataset.}
% \label{fig:dino-vit}
% \end{minipage}
% \hspace{0.1in}
% \begin{minipage}[t]{0.45\linewidth}
% \centering
% \includegraphics[width=\linewidth]{figs/channel-missing.pdf}
% \caption{Average test accuracy (\%) across all combinations of channels on the JUMP-CP dataset.}
% \label{fig:plot-channel-missing}
% \end{minipage}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[width=0.6\linewidth]{figs/dino-vit.pdf}
% \caption{Performance curves of IC-ViT and ViT, both pretrained and finetuned on the JUMP-CP dataset.}
% \label{fig:dino-vit}
% \end{figure}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.28\linewidth}
\centering
\includegraphics[width=1.\linewidth]{figs/dino-vit.pdf}
\end{minipage}
\begin{minipage}[t]{0.28\linewidth}
\centering
\includegraphics[width=1.\linewidth]{figs/addition_dataset.pdf}
% \label{fig:dino-hcsvit}
\end{minipage}
\begin{minipage}[t]{0.35\linewidth}
\vspace{-.9in}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{@{} l cc@{}}
\toprule
& Validation acc. & Test acc. \\
\midrule
{JUMP-CP} & 83.25 & 83.41 \\
{JUMP-CP+CHAMMI} & \textbf{85.40} & \textbf{85.69} \\
\bottomrule
\end{tabular}
}
% \label{tab:pretraining-time}
\end{minipage}
% \vspace{.1in}
\caption{(Left:) Performance curves of IC-ViT and ViT, both pretrained and finetuned on the JUMP-CP dataset. (Middle:) Multi-dataset pretraining evaluation. Validation accuracy curves for the baseline model (pretrained solely on JUMP-CP) and the extended model (pretrained on both JUMP-CP and CHAMMI). The extended pretraining generally results in higher accuracy throughout training. (Right:) Test accuracy at the epoch of highest validation accuracy. Best results in \textbf{bold}.}
\label{fig:others}
\end{figure}
\paragraph{Robustness evaluation on channel subsets}
\autoref{tab:main_ret} shows that IC-ViT outperforms baseline methods even when an entire modality is absent at inference, indicating its potential for handling inputs with missing channels. We further extend this evaluation to scenarios with randomly missing channels by analyzing test accuracy on the JUMP-CP dataset when different subsets of channels are removed during inference. All possible channel combinations are considered, and \autoref{tab:channel-missing} reports the mean and standard deviation across these combinations for different methods.
As illustrated in \autoref{tab:channel-missing}, IC-ViT consistently achieves higher average accuracy across different channel subsets, demonstrating strong robustness to partial or missing inputs. Moreover, the steady improvement in accuracy with more available channels indicates that our single-channel sampling strategy effectively leverages independent semantic information from each channel, enhancing the overall feature diversity and informativeness.
%------------------potential as encoder for foudation models-------------------------------------------------------
\paragraph{Pretraining with extended datasets}
As discussed in Sec.~\ref{subsec:MCI_pretraining}, IC-ViT has the potential to serve as a foundation model for MCI tasks by pretraining on extended datasets with a larger variety of multi-channel inputs. To provide a preliminary evaluation of this capability, we compare a baseline IC-ViT model pretrained solely on the JUMP-CP dataset with an extended version pretrained on both JUMP-CP and CHAMMI. Both models are subsequently fine-tuned on the JUMP-CP training set under the same experimental settings. As shown in \autoref{fig:others} (middle), the model pretrained on the combined dataset generally leads to higher validation accuracy across most training epochs. As summarized in \autoref{fig:others} (right), pretraining with additional data results in over a 2\% improvement in both validation and test accuracy, further highlighting the model’s effectiveness for pretraining on heterogeneous multi-channel data.
\paragraph{Limitations and future work}
While IC-ViT demonstrates strong empirical performance across diverse MCI benchmarks, several limitations remain. First, the single-channel pretraining strategy inevitably neglects inter-channel correlations during representation learning, which may lead to partial information loss, especially in domains with low channel redundancy or high cross-channel dependencies. Second, our current uniform channel sampling treats all channels equally, which could be suboptimal in the presence of noisy or less informative channels. Future work could incorporate adaptive or learnable channel weighting mechanisms to better exploit heterogeneous data. Third, our experiments are limited to DINO-based self-supervised pretraining; extending the framework to other SSL paradigms such as MAE~\cite{he2022masked}, BYOL~\cite{grill2020bootstrap}, or SimMIM~\cite{xie2022simmim} would further validate its generality. Finally, the evaluation is restricted to microscopy and remote sensing datasets. Broader investigations on other multi-channel modalities, such as RGB-D imaging or medical MRI, will be essential to assess the scalability and universality of IC-ViT.
\section{Conclusion}
\label{sec:conclusion}
In this study, we tackle the challenge of learning from multi-channel imaging (MCI) data, where input channels often correspond to distinct modalities or measurements. We propose \textit{Isolated Channel Vision Transformer (IC-ViT)}, a simple yet effective framework that performs pretraining on single-channel inputs to learn channel-specific representations, while naturally supporting datasets with varying channel configurations. Upon fine-tuning, IC-ViT integrates complementary information across channels, leading to improved performance on downstream MCI tasks. Through extensive experiments on diverse datasets and tasks, we demonstrate that IC-ViT outperforms existing channel-adaptive approaches. Its efficiency and scalability make it a strong candidate for large-scale pretraining in MCI imaging.
\paragraph{Acknowledgement}
The computations and data handling were enabled by resources provided by Chalmers e-Commons at Chalmers.
\begin{thebibliography}{36}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Awais et~al.(2025)Awais, Naseer, Khan, Anwer, Cholakkal, Shah, Yang, and Khan]{awais2025foundation}
Muhammad Awais, Muzammal Naseer, Salman Khan, Rao~Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad~Shahbaz Khan.
\newblock Foundation models defining a new era in vision: a survey and outlook.
\newblock \emph{IEEE TPAMI}, 2025.
\bibitem[Bao et~al.(2024)Bao, Sivanandan, and Karaletsos]{bao2024channel}
Yujia Bao, Srinivasan Sivanandan, and Theofanis Karaletsos.
\newblock Channel vision transformers: An image is worth 1 x 16 x 16 words.
\newblock In \emph{ICLR}, 2024.
\newblock URL \url{https://openreview.net/forum?id=CK5Hfb5hBG}.
\bibitem[Bourriez et~al.(2024)Bourriez, Bendidi, Cohen, Watkinson, Sanchez, Bollot, and Genovesio]{bourriez2024chada}
Nicolas Bourriez, Ihab Bendidi, Ethan Cohen, Gabriel Watkinson, Maxime Sanchez, Guillaume Bollot, and Auguste Genovesio.
\newblock {ChAda-ViT}: Channel adaptive attention for joint representation learning of heterogeneous microscopy images.
\newblock In \emph{CVPR}, pages 11556--11565, 2024.
\bibitem[Cai et~al.(2020)Cai, Gao, and Zhao]{cai2020review}
Lei Cai, Jingyang Gao, and Di~Zhao.
\newblock A review of the application of deep learning in medical image classification and segmentation.
\newblock \emph{Annals of Translational Medicine}, 8\penalty0 (11):\penalty0 713, 2020.
\bibitem[Caron et~al.(2021)Caron, Touvron, Misra, J{\'e}gou, Mairal, Bojanowski, and Joulin]{caron2021emerging}
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv{\'e} J{\'e}gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.
\newblock Emerging properties in self-supervised vision transformers.
\newblock In \emph{ICCV}, pages 9650--9660, 2021.
\bibitem[Chandrasekaran et~al.(2024)Chandrasekaran, Cimini, Goodale, Miller, Kost-Alimova, Jamali, Doench, Fritchman, Skepner, Melanson, et~al.]{chandrasekaran2024three}
Srinivas~Niranj Chandrasekaran, Beth~A Cimini, Amy Goodale, Lisa Miller, Maria Kost-Alimova, Nasim Jamali, John~G Doench, Briana Fritchman, Adam Skepner, Michelle Melanson, et~al.
\newblock Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations.
\newblock \emph{Nature Methods}, 21\penalty0 (6):\penalty0 1114--1121, 2024.
\bibitem[Chen et~al.(2023)Chen, Pham, Wang, Doron, Moshkov, Plummer, and Caicedo]{chen2023chammi}
Zitong~Sam Chen, Chau Pham, Siqi Wang, Michael Doron, Nikita Moshkov, Bryan Plummer, and Juan~C Caicedo.
\newblock {CHAMMI}: A benchmark for channel-adaptive models in microscopy imaging.
\newblock \emph{NeurIPS}, 36:\penalty0 19700--19713, 2023.
\bibitem[de~La~Comble and Prepin(2021)]{de2021efficient}
Alo{\"\i}s de~La~Comble and Ken Prepin.
\newblock Efficient transfer learning for multi-channel convolutional neural networks.
\newblock In \emph{2021 17th International Conference on Machine Vision and Applications (MVA)}, pages 1--6. IEEE, 2021.
\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby]{dosovitskiy2021an}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at scale.
\newblock In \emph{ICLR}, 2021.
\newblock URL \url{https://openreview.net/forum?id=YicbFdNTTy}.
\bibitem[Ericsson et~al.(2022)Ericsson, Gouk, Loy, and Hospedales]{ericsson2022self}
Linus Ericsson, Henry Gouk, Chen~Change Loy, and Timothy~M Hospedales.
\newblock Self-supervised representation learning: Introduction, advances, and challenges.
\newblock \emph{IEEE Signal Processing Magazine}, 39\penalty0 (3):\penalty0 42--62, 2022.
\bibitem[Grill et~al.(2020)Grill, Strub, Altch{\'e}, Tallec, Richemond, Buchatskaya, Doersch, Avila~Pires, Guo, Gheshlaghi~Azar, et~al.]{grill2020bootstrap}
Jean-Bastien Grill, Florian Strub, Florent Altch{\'e}, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila~Pires, Zhaohan Guo, Mohammad Gheshlaghi~Azar, et~al.
\newblock Bootstrap your own latent-a new approach to self-supervised learning.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 21271--21284, 2020.
\bibitem[Ha et~al.(2017)Ha, Dai, and Le]{ha2016hypernetworks}
David Ha, Andrew~M. Dai, and Quoc~V. Le.
\newblock Hypernetworks.
\newblock In \emph{ICLR}, 2017.
\newblock URL \url{https://openreview.net/forum?id=rkpACe1lx}.
\bibitem[Han et~al.(2022)Han, Wang, Chen, Chen, Guo, Liu, Tang, Xiao, Xu, Xu, et~al.]{han2022survey}
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An~Xiao, Chunjing Xu, Yixing Xu, et~al.
\newblock A survey on vision transformer.
\newblock \emph{IEEE TPAMI}, 45\penalty0 (1):\penalty0 87--110, 2022.
\bibitem[He et~al.(2022)He, Chen, Xie, Li, Doll{\'a}r, and Girshick]{he2022masked}
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll{\'a}r, and Ross Girshick.
\newblock Masked autoencoders are scalable vision learners.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 16000--16009, 2022.
\bibitem[He et~al.(2023)He, Wang, Zhao, and Chen]{he2023co}
Xiaoyu He, Yong Wang, Shuang Zhao, and Xiang Chen.
\newblock Co-attention fusion network for multimodal skin cancer diagnosis.
\newblock \emph{Pattern Recognition}, 133:\penalty0 108990, 2023.
\bibitem[Joze et~al.(2020)Joze, Shaban, Iuzzolino, and Koishida]{joze2020mmtm}
Hamid Reza~Vaezi Joze, Amirreza Shaban, Michael~L Iuzzolino, and Kazuhito Koishida.
\newblock Mmtm: Multimodal transfer module for cnn fusion.
\newblock In \emph{CVPR}, pages 13289--13299, 2020.
\bibitem[Lee et~al.(2019)Lee, Lee, Kim, Song, Kim, and Ro]{lee2019mcsip}
Jae-Hyeok Lee, Sangmin~S Lee, Hak~Gu Kim, Sa-Kwang Song, Seongchan Kim, and Yong~Man Ro.
\newblock Mcsip net: Multichannel satellite image prediction via deep neural network.
\newblock \emph{IEEE Transactions on Geoscience and Remote Sensing}, 58\penalty0 (3):\penalty0 2212--2224, 2019.
\bibitem[Lian et~al.(2025)Lian, Lindblad, Stark, Hirsch, and Sladoje]{lian2025let}
Wenyi Lian, Joakim Lindblad, Christina~Runow Stark, Jan-Micha{\'e}l Hirsch, and Nata{\v{s}}a Sladoje.
\newblock Let it shine: Autofluorescence of papanicolaou-stain improves ai-based cytological oral cancer detection.
\newblock \emph{Computers in Biology and Medicine}, 185:\penalty0 109498, 2025.
\bibitem[Liu et~al.(2017)Liu, Zhang, Adeli, and Shen]{liu2017deep}
Mingxia Liu, Jun Zhang, Ehsan Adeli, and Dinggang Shen.
\newblock Deep multi-task multi-channel learning for joint classification and regression of brain status.
\newblock In \emph{International Conference on Medical Image Computing and Computer-Assisted Intervention}, pages 3--11. Springer, 2017.
\bibitem[Misra et~al.(2020)Misra, Jeon, Lee, Managuli, Jang, and Kim]{misra2020multi}
Sampa Misra, Seungwan Jeon, Seiyon Lee, Ravi Managuli, In-Su Jang, and Chulhong Kim.
\newblock Multi-channel transfer learning of chest x-ray images for screening of covid-19.
\newblock \emph{Electronics}, 9\penalty0 (9):\penalty0 1388, 2020.
\bibitem[Nguyen et~al.(2023)Nguyen, Brandstetter, Kapoor, Gupta, and Grover]{nguyen2023climax}
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh~K Gupta, and Aditya Grover.
\newblock {C}lima{X}: A foundation model for weather and climate.
\newblock In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, \emph{Proc. of the 40th International Conference on Machine Learning}, volume 202 of \emph{Proceedings of Machine Learning Research}, pages 25904--25938. PMLR, 23--29 Jul 2023.
\bibitem[Nie et~al.(2019)Nie, Lu, Zhang, Adeli, Wang, Yu, Liu, Wang, Wu, and Shen]{nie2019multi}
Dong Nie, Junfeng Lu, Han Zhang, Ehsan Adeli, Jun Wang, Zhengda Yu, LuYan Liu, Qian Wang, Jinsong Wu, and Dinggang Shen.
\newblock Multi-channel 3d deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages.
\newblock \emph{Scientific Reports}, 9\penalty0 (1):\penalty0 1103, 2019.
\bibitem[Oquab et~al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, HAZIZA, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski]{oquab2023dinov2}
Maxime Oquab, Timoth{\'e}e Darcet, Th{\'e}o Moutakanni, Huy~V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu~Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.
\newblock {DINO}v2: Learning robust visual features without supervision.
\newblock \emph{Transactions on Machine Learning Research}, 2024.
\newblock ISSN 2835-8856.
\newblock URL \url{https://openreview.net/forum?id=a68SUt6zFt}.
\newblock Featured Certification.
\bibitem[Pham and Plummer(2024)]{pham2024enhancing}
Chau Pham and Bryan Plummer.
\newblock Enhancing feature diversity boosts channel-adaptive vision transformers.
\newblock In A.~Globerson, L.~Mackey, D.~Belgrave, A.~Fan, U.~Paquet, J.~Tomczak, and C.~Zhang, editors, \emph{NeurIPS}, volume~37, pages 89782--89805. Curran Associates, Inc., 2024.
\newblock URL \url{https://proceedings.neurips.cc/paper_files/paper/2024/file/a376fb45a0396c6fbf2f914dce8f7316-Paper-Conference.pdf}.
\bibitem[Ramachandram and Taylor(2017)]{ramachandram2017deep}
Dhanesh Ramachandram and Graham~W Taylor.
\newblock Deep multimodal learning: A survey on recent advances and trends.
\newblock \emph{IEEE Signal Processing Magazine}, 34\penalty0 (6):\penalty0 96--108, 2017.
\bibitem[Savarese and Maire(2019)]{savarese2019learning}
Pedro Savarese and Michael Maire.
\newblock Learning implicitly recurrent {CNN}s through parameter sharing.
\newblock In \emph{ICLR}, 2019.
\newblock URL \url{https://openreview.net/forum?id=rJgYxn09Fm}.
\bibitem[Sutskever et~al.(2014)Sutskever, Vinyals, and Le]{sutskever2014sequence}
Ilya Sutskever, Oriol Vinyals, and Quoc~V Le.
\newblock Sequence to sequence learning with neural networks.
\newblock \emph{Advances in neural information processing systems}, 27:\penalty0 3104--3112, 2014.
\bibitem[Teh et~al.(2020)Teh, DeVries, and Taylor]{teh2020proxynca}
Eu~Wern Teh, Terrance DeVries, and Graham~W Taylor.
\newblock Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis.
\newblock In \emph{ECCV}, pages 448--464. Springer, 2020.
\bibitem[Thisanke et~al.(2023)Thisanke, Deshan, Chamith, Seneviratne, Vidanaarachchi, and Herath]{thisanke2023semantic}
Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vidanaarachchi, and Damayanthi Herath.
\newblock Semantic segmentation using vision transformers: A survey.
\newblock \emph{Engineering Applications of Artificial Intelligence}, 126:\penalty0 106669, 2023.
\bibitem[Thul et~al.(2017)Thul, {\AA}kesson, Wiking, Mahdessian, Geladaki, Ait~Blal, Alm, Asplund, Bj{\"o}rk, Breckels, et~al.]{thul2017subcellular}
Peter~J Thul, Lovisa {\AA}kesson, Mikaela Wiking, Diana Mahdessian, Aikaterini Geladaki, Hammou Ait~Blal, Tove Alm, Anna Asplund, Lars Bj{\"o}rk, Lisa~M Breckels, et~al.
\newblock A subcellular map of the human proteome.
\newblock \emph{Science}, 356\penalty0 (6340):\penalty0 eaal3321, 2017.
\bibitem[Viana et~al.(2023)Viana, Chen, Knijnenburg, Vasan, Yan, Arakaki, Bailey, Berry, Borensztejn, Brown, et~al.]{viana2023integrated}
Matheus~P Viana, Jianxu Chen, Theo~A Knijnenburg, Ritvik Vasan, Calysta Yan, Joy~E Arakaki, Matte Bailey, Ben Berry, Antoine Borensztejn, Eva~M Brown, et~al.
\newblock Integrated intracellular organization and its variations in human ips cells.
\newblock \emph{Nature}, 613\penalty0 (7943):\penalty0 345--354, 2023.
\bibitem[Way et~al.(2022)Way, Natoli, Adeboye, Litichevskiy, Yang, Lu, Caicedo, Cimini, Karhohs, Logan, et~al.]{way2022morphology}
Gregory~P Way, Ted Natoli, Adeniyi Adeboye, Lev Litichevskiy, Andrew Yang, Xiaodong Lu, Juan~C Caicedo, Beth~A Cimini, Kyle Karhohs, David~J Logan, et~al.
\newblock Morphology and gene expression profiling provide complementary information for mapping cell state.
\newblock \emph{Cell Systems}, 13\penalty0 (11):\penalty0 911--923, 2022.
\bibitem[Xie et~al.(2022)Xie, Zhang, Cao, Lin, Bao, Yao, Dai, and Hu]{xie2022simmim}
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi~Dai, and Han Hu.
\newblock Simmim: A simple framework for masked image modeling.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 9653--9663, 2022.
\bibitem[Xu et~al.(2024)Xu, Usuyama, Bagga, Zhang, Rao, Naumann, Wong, Gero, Gonz{\'a}lez, Gu, et~al.]{xu2024whole}
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier Gonz{\'a}lez, Yu~Gu, et~al.
\newblock A whole-slide foundation model for digital pathology from real-world data.
\newblock \emph{Nature}, 630\penalty0 (8015):\penalty0 181--188, 2024.
\bibitem[Zhou et~al.(2024)Zhou, Li, Li, Yu, Liu, Wang, Zhang, Ji, Yan, He, et~al.]{zhou2024comprehensive}
Ce~Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et~al.
\newblock A comprehensive survey on pretrained foundation models: A history from bert to chatgpt.
\newblock \emph{International Journal of Machine Learning and Cybernetics}, pages 1--65, 2024.
\bibitem[Zhu et~al.(2020)Zhu, Hu, Qiu, Shi, Kang, Mou, Bagheri, Haberle, Hua, Huang, Hughes, Li, Sun, Zhang, Han, Schmitt, and Wang]{Zhu2020So2Sat}
XiaoXiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Jian Kang, Lichao Mou, Hossein Bagheri, Matthias Haberle, Yuansheng Hua, Rong Huang, Lloyd Hughes, Hao Li, Yao Sun, Guichen Zhang, Shiyao Han, Michael Schmitt, and Yuanyuan Wang.
\newblock So2sat lcz42: A benchmark data set for the classification of global local climate zones [software and data sets].
\newblock \emph{IEEE Geoscience and Remote Sensing Magazine}, 8\penalty0 (3):\penalty0 76--89, 2020.
\newblock \doi{10.1109/MGRS.2020.2964708}.
\end{thebibliography}