Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation
Extracted source text · arXiv 2305.00603
\documentclass{article} % For LaTeX2e
\usepackage{iclr2023_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs} % professional-quality tables
\usepackage{amsmath}
\usepackage{xspace}
\makeatletter
\DeclareRobustCommand\onedot{\futurelet\@let@token\@onedot}
\def\@onedot{\ifx\@let@token.\else.\null\fi\xspace}
\def\eg{\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot}
\def\ie{\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot}
\def\cf{\emph{c.f}\onedot} \def\Cf{\emph{C.f}\onedot}
\def\etc{\emph{etc}\onedot} \def\vs{\emph{vs}\onedot}
\def\wrt{w.r.t\onedot} \def\dof{d.o.f\onedot}
\def\etal{\emph{et al}\onedot}
\makeatother
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\crefname{section}{Section}{Sections}
\crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}
\usepackage{svg}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage{makecell}
\usepackage{utfsym}
\setlength{\abovecaptionskip}{0.01cm}
\title{Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\makeatletter
\newcommand{\myfnsymbol}[1]{%
\expandafter\@myfnsymbol\csname c@#1\endcsname
}
% Mapping of how the \thanks symbols will be interpreted sequentially
\newcommand{\@myfnsymbol}[1]{%
\ifcase #1
% 0
\or 1% 1
\or 2% 2
\or \TextOrMath{\textasteriskcentered}{*}% 3
\or \TextOrMath{\textdagger}{\dagger}% 4
\fi
}
% Just to make things explicit in the code what it means
\newcommand{\affiliationA}{\@myfnsymbol{1}}
\newcommand{\affiliationB}{\@myfnsymbol{2}}
\newcommand{\equalcontributor}{\@myfnsymbol{3}}
\newcommand{\correspondingA}{\@myfnsymbol{4}}
\makeatother
%\setlength{\textheight}{15\baselineskip}% Just for this example
% Rest of your document here
\author{Tianxiang Hao$^{1,3}$, Hui Chen$^{2,3}$\textsuperscript{\correspondingA}, Yuchen Guo$^{3}$, Guiguang Ding$^{1,3}$\textsuperscript{\correspondingA} \\
$^{1}$School of Software, Tsinghua University \, $^{2}$Department of Automation, Tsinghua University\\
$^{3}$Beijing National Research Center for Information Science and Technology (BNRist)\\
\texttt{\{beyondhtx,jichenhui2012,yuchen.w.guo\}@gmail.com, dinggg@tsinghua.edu.cn} \\
}
%\renewcommand{\thefootnote}{\fnsymbol{footnote}}
%\footnotetext[1]{These authors contributed equally}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
% Thanks notes for title uses \myfnsymbol
\renewcommand{\thefootnote}{\myfnsymbol{footnote}}
\maketitle
% Layout the \thanks notes in the order you want
%\footnotetext[1]{Address of Institute A}%
%\footnotetext[2]{Address of Institute B}%
%\footnotetext[3]{These authors contributed equally}%
\footnotetext[4]{Corresponding authors.}%
\setcounter{footnote}{0}% Restart footnote counter
% Footnotes for rest of document uses \fnsymbol (or whatever you choose)
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
\maketitle
\begin{abstract}
Recently, transformers have shown strong ability as visual feature extractors, surpassing traditional convolution-based models in various scenarios. However, the success of vision transformers largely owes to their capacity to accommodate numerous parameters. As a result, new challenges for adapting a well-trained transformer to downstream tasks arise. On the one hand, classic fine-tuning tunes all parameters in a huge model for every downstream task and thus easily falls into an overfitting situation, leading to inferior performance. On the other hand, on resource-limited devices, fine-tuning stores a full copy of all parameters and thus is usually impracticable for the shortage of storage space. However, few works have focused on how to efficiently and effectively transfer knowledge in a vision transformer. Existing methods did not dive into the properties of visual features, leading to inferior performance. Moreover, some of them bring heavy inference cost though benefiting storage. To tackle these problems, we propose consolidator to achieve efficient transfer learning for large vision models. Our consolidator modifies the pre-trained model with the addition of a small set of tunable parameters to temporarily store the task-specific knowledge while freezing the backbone model during adaptation. Motivated by the success of group-wise convolution, we adopt grouped connections across the features extracted by fully connected layers to construct tunable parts in a consolidator. To further enhance the model's capacity to transfer knowledge under a constrained storage budget and keep inference efficient, we consolidate the parameters in two stages: 1. between adaptation and storage, and 2. between loading and inference. On a series of downstream visual tasks, our consolidator can reach up to 7.56 better accuracy than full fine-tuning with merely 0.35\% parameters, and outperform state-of-the-art parameter-efficient tuning methods by a clear margin. Code is available at \href{https://github.com/beyondhtx/Consolidator}{github}.
\end{abstract}
\section{Introduction}
\label{sec:introduction}
Recently, transformer architectures originated from natural language processing (NLP)~\citep{vaswani2017attention} demonstrate considerable capacity in computer vision~\citep{dosovitskiy2020image,touvron2021training,liu2021swin}. Vision transformers, along with traditional convolutional neural networks (CNNs)~\citep{krizhevsky2012imagenet,he2016deep,simonyan2014very}, are widely used as feature extractors to generate strong and general visual representations via deriving knowledge from massive images. Thanks to the abundant information in such representations, we can adapt the pre-trained models to downstream tasks by a simple fine-tuning strategy.
However, fine-tuning is not a good solution for adaptation. As is well known, the scale of vision models grows faster and faster in recent years. On the one hand, fine-tuning which tunes all parameters in such a huge model easily falls into an overfitting situation, leading to inferior performance. On the other hand, fine-tuning inflicts heavy storage burdens. Since fine-tuning intensively tunes all parameters, it maintains a full copy of the model's parameters for each task. Therefore, fine-tuning can cause a huge storage burden when there are many tasks to be adapted, resulting in impracticality in real-world scenarios, especially in resource-constrained situations, \eg, embedded systems.
Efforts have been made to improve the performance as well as reduce the storage overhead of fine-tuning. For example, adapter~\citep{houlsby2019parameter,karimi2021compacter}, prompt tuning~\citep{li2021prefix,lester2021power,zhou2021learning} and LoRA~\citep{hu2021lora} inject tunable parameters and freeze the backbone during adaptation. In the vision field, VPT~\citep{jia2022vpt} directly leverage learnable prompts, AdaptFormer~\citep{chen2022adaptformer} adopts parallel adapter, NOAH~\citep{zhang2022noah} searches for the optimal combinations of the three representative modules, \ie, adapter, LoRA, and VPT, and SSF~\citep{lian2022ssf} use additional scaling and shifting parameters for adaptation. Despite their acceptable performance, existing methods suffer from two common conflicts: 1. trade-off between the inference efficiency and the adaptation performance, and 2. trade-off between the adaptation performance and the number of stored parameters. Previous works~\citep{houlsby2019parameter} show that introducing more tunable parameters can achieve more fruitful results. However, extra parameters can bring significantly larger computation and storage cost, resulting in low inference efficiency and more storage space. Therefore, one essential question is raised: \textit{can we design a module that can share the same inference cost as an ordinary model while enjoying superior capacity against existing methods}?
In this paper, we propose a generic module, dubbed consolidator, to tackle the aforementioned issues. The proposed consolidator is designed as a mergeable adapter that accompanies the fully connected (FC) layer in the vision models. Specifically, to enrich the model capacity under a limited parameter budget, we take inspiration from the success of group-wise convolution~\citep{howard2017mobilenets,ma2018shufflenet,liu2022convnet} and build our consolidator as grouped connected (GC) layers. To enhance the flexibility, we further reorder channels for each group connection, followed by a droppath regularizer. Benefiting from the inference-time linearity of GC, channel reorder, and droppath operations, the proposed consolidator can be perfectly consolidated into the original FC layer of a vision model, leading to no extra inference cost.
Our consolidator can be easily expanded as a multi-branch topology without breaking the linearity. Practically, we can simultaneously equip several GC layers with channel reordering for communications between different groups of feature channels. After adaptation, we can first consolidate the multi-branch GC layers into one single sparse parameter matrix and store the sparse matrix for each task. Such property can enhance the model's transferability and achieve a considerable storage reduction when the number of tasks scales up. During inference, such a sparse parameter matrix can be merged into the backbone model as well, resulting in no inference cost. Thanks to the twice consolidation, the proposed consolidator can greatly promote efficient and effective visual adaptation.
To verify the superiority of consolidator, we conduct extensive experiments and analysis on a series of downstream recognition tasks. Experimental results show that our consolidator can surpass full fine-tuning by 7.56 top-1 accuracy with merely 0.35\% parameters per task. Compared to state-of-the-art methods, such as NOAH, AdaptFormer and SSF our method can consistently reach better performance while enjoying no inference cost. On other fundamental visual tasks, \ie, object detection and semantic segmentation, our consolidator shows great power as well.
Overall, we summarize our contributions as follows.
(i) We propose a basic module, dubbed consolidator, for effective and efficient visual transfer learning. To enhance the transferability under limited tunable parameters, our consolidator is designed as a mergeable grouped connected (GC) layer with a channel reorder layer and a droppath regularizer. We extend the single branch to a multi-branch topology for better flexibility and transferability.
(ii) We design a two-stage consolidation scheme by merging corresponding parameters in the training-storage phase and loading-inference phase. In this way, we can maximally dig the adaptation capacity of the model under a constrained storage budget, with no extra inference cost.
(iii) We conduct extensive experiments and analysis on various downstream tasks. Results show that the proposed consolidator method can consistently outperform state-of-the-art methods with fewer stored parameters but superior performance.
\section{Related Works}
\label{related_works}
\textbf{Parameter-efficient transfer learning.} In the language field, works~\citep{houlsby2019parameter,pfeiffer2021adapterfusion,li2021prefix,lester2021power,zaken2021bitfit,hu2021lora,karimi2021compacter,liu2021p,ding2022delta} have been done to efficiently transfer the knowledge of pre-trained transformers to downstream language tasks. In the field of visual adaptation, several explorations have also been made to adapt vision transformers efficiently. \citet{jia2022vpt} and \citet{bahng2022visual} directly apply prompt-tuning. \citet{jie2022convolutional} integrates additional tunable convolution layers. NOAH~\citep{zhang2022noah} first trains a large supernet with three modules, VPT, LoRA, and adapter, and then searches for the optimal configurations of each module for every transformer block using evolution algorithm~\citep{chen2021AutoFormer}. AdaptFormer~\citep{chen2022adaptformer} adds parallel adapters instead of serial ones. SSF~\citep{lian2022ssf} tunes additional scaling and shifting parameters for adaptation. It is also shown that classic methods such as LoRA~\citep{hu2021lora} and adapter~\citep{houlsby2019parameter} can lead to good performance for vision transformers. However, existing methods suffer from the two trade-offs as we discussed in \cref{sec:introduction}, resulting in difficulties in fully digging the adaptation capacity of vision models efficiently. To solve the problems, we present a mergeable adapter, named consolidator, and introduce a two-stage consolidation design to perfectly balance the trade-offs, leading to efficient and effective visual adaptation.
\begin{figure}
\begin{subfigure}{\linewidth}
\centering
\includegraphics[width=0.75\linewidth]{imgs/Consolidator_6x6.pdf}
\end{subfigure}
\setlength{\belowcaptionskip}{-0.55cm}
\caption{Consolidator tuning versus full fine-tuning. Consolidator adds tunable multi-branch grouped connected layers to the original fully connected layers. The tunable parameters are merged via addition into one single sparse matrix before storage to reduce the needed storage space. Between loading and inference, the parameters in the sparse matrix will be merged back into the original fully connected layer. Consolidator greatly enlarges the model's adaptation capacity under a constrained storage budget with no extra inference cost. \textbf{Best viewed in color.}}
\label{fig:Consolidator}
\end{figure}
\textbf{Inference-efficient structures.} Many works~\citep{ding2019acnet,guo2020expandnets,ding2021diverse,ding2021repvgg,ding2021resrep,ding2022scaling31} strive to design a generic convolution architecture to realize superior capacity while enjoying no inference cost. For example, RepVGG~\citep{ding2021repvgg} integrates an extra 1$\times$1 convolution to strengthen the main 3$\times$3 convolution. However, existing methods are typically designed for CNNs. As for the popular vision transformer architectures, rare works investigate how to effectively strengthen their capacity while introducing no extra inference cost. LoRA~\citep{hu2021lora} and SSF~\citep{lian2022ssf} offer possible solutions, but they do not explore the consolidation process between training and storage, leading to inferior performance under a given storage budget. In this paper, we adopt parallel GC layers to replace the functionality of the original FC layers in vision models, which shows strong abilities for visual adaptation. Furthermore, we expand the existing one-stage training-inference consolidation to a two-stage process: 1. training-storage consolidation, and 2. loading-inference consolidation. Such a two-stage design can maximally dig the adaptation capacity of the pre-trained model under a constrained storage budget, with no extra inference cost. Extensive experiments show that our consolidator can outperform state-of-the-art methods in both the number of tunable parameters and the adaptation performance.
\section{Methodology}
\label{methodology}
\subsection{Preliminaries}
In this paper, we mainly focus on the adaptation for vision transformers~\citep{dosovitskiy2020image,liu2021swin}. A typical vision transformer~\citep{dosovitskiy2020image} consists of $L$ serial blocks. In each encoder, there are a multi-head self-attention module (MHSA) and a multi-layer perceptron (MLP). Formally, a batch of input images $\textbf{x}_{input}\in \mathbb{R}^{B\times 3\times H\times W}$ will be first reshaped into a sequence of flattened 2D patches $\textbf{x}_{p}\in \mathbb{R}^{B\times N\times(P^{2}\cdot C)}$, where $C$ is the number of channels and $(P,P)$ is the resolution of each patch, and $N=NW/P^{2}$ is the number of patches. Then the patches are mapped to $D$ channel dimensions with a linear projection. Next, a classification token is appended and we can get $\textbf{x}_{1}\in \mathbb{R}^{B\times (N+1)\times D}$. Here we use $\textbf{x}_{l}\in \mathbb{R}^{B\times (N+1)\times D}$ to denote the input of $l$-th ($1\leq l\leq L$) block. Its output $\textbf{x}_{l+1}=\textbf{x}_{l}'+\mbox{MLP}(\mbox{LayerNorm}(\textbf{x}_{l}'))$ where $\textbf{x}_{l}'=\textbf{x}_{l}+\mbox{MHSA}(\mbox{LayerNorm}(\textbf{x}_{l}))$. For MHSA, the input features are first processed by three FC layers to generate matrices $Q,K,V$, and the output is calculated by $\mbox{Softmax}(\frac{QK^T}{\sqrt{d}})V$ and then projected by another FC layer. Therefore, the parametric components of MHSA are four FC layers. The parametric components of MLP are two FC layers as well. Therefore, we formulate our consolidator for the FC layers (see \cref{fig:Consolidator}), covering all the parametric components in each MHSA and MLP. We will show that such a design can realize both efficiency and effectiveness in \cref{sec:experiments}. Notably, our method is also applicable for MLP~\citep{lian2022mlp} and CNN~\citep{liu2022convnet} and can reach good results as in \cref{tab:generalization_experiment}.
\subsection{Consolidator}
For efficient transfer learning, we merely tune and store the parameters in consolidators while freezing other parameters in the pre-trained model. In this subsection, we will introduce our design of consolidator, an efficient and effective module for adapting vision transformers, in detail.
\textbf{Grouped connections.} Inspired by the success of group convolution in extracting visual features, we hence assume that the cross-channel information exchange is redundant in visual adaptation, and aim to design consolidator by reducing the cross-channel connections between sequential features to minimize the number of stored parameters for downstream tasks while keeping maximum capacity. Therefore, for each FC layer, we add a concurrent module consisting of a grouped connected layer. Formally, for an input $\textbf{x}\in \mathbb{R}^{D}$, the output $\textbf{x}'\in \mathbb{R}^{E}$ of a GC layer with group $g$, weight $\textbf{W}\in \mathbb{R}^{g\times \frac{E}{g}\times \frac{D}{g}}$ and bias $\textbf{b}\in \mathbb{R}^{E}$ is formulated by $\textbf{x}'=\mbox{GC}(\textbf{x})=\sum_{j=1}^{g}{\mbox{Pad}(\textbf{W}_{j}\textbf{x}_{\frac{(j-1)D}{g}:\frac{jD}{g}}, j)}+\textbf{b}$.
Here $\mbox{Pad}(\textbf{z}, j)$ prepends $\frac{(j-1)D}{g}$ zeros and appends $\frac{(g-j)D}{g}$ zeros for $\textbf{z}\in \mathbb{R}^{\frac{E}{g}}$ according to another input $j$. In this way, the output channels in the $j$-th group only interact with the input channels in the $j$-th group, and thus we reduce the cross-channel connections as expected.
To flexibly reach different ratios of stored parameters, we adopt a multi-branch topology in our consolidator. There is a GC layer with weight $\textbf{W}^{(i)}\in \mathbb{R}^{g^{(i)}\times \frac{E}{g^{(i)}}\times \frac{D}{g^{(i)}}}$ and bias $\textbf{b}_{i}\in \mathbb{R}^{E}$ for $i$-th branch with group $g^{(i)}$. During adaptation, consolidator and the original FC layer take the same input and their outputs are summed up to produce the new output $\textbf{y}$. Formally, for each FC layer with weight $\textbf{W}\in \mathbb{R}^{E\times D}$ and bias $\textbf{b}\in \mathbb{R}^{E}$, the output of the whole layer modified by $m$ GC branches is $\textbf{y}=\textbf{W}\textbf{x}+\textbf{b}+\sum_{i=1}^{m}(\sum_{j=1}^{g^{(i)}}{\mbox{Pad}(\textbf{W}_{j}^{(i)}\textbf{x}_{\frac{(j-1)D}{g^{(i)}}:\frac{jD}{g^{(i)}}}, j)}+\textbf{b}^{(i)})$.
\textbf{Channel reorder.}
To flexibly tune the total number of parameters and enrich the exchange of information flow, we prepend a ``ChannelReorder'' operation to every branch in our consolidator by manually adjusting the permutation of input features along the channel dimension. In general, we adopt shuffle operation~\citep{zhang2018shufflenet,ma2018shufflenet} to accomplish such a purpose.
Formally, given input $\textbf{x}\in \mathbb{R}^{\ast\times D}$ where ``$\ast $'' means any number of dimensions including none, we shuffle it into $g$ groups and perform recombination across the last dimension. Formally, we first reshape $\textbf{x}$ into $\textbf{x}'\in \mathbb{R}^{\ast\times g\times \frac{D}{g}}$, and then transpose the last two dimension and get $\textbf{x}''\in \mathbb{R}^{\ast\times \frac{D}{g} \times g}$, and then reshape $\textbf{x}''$ into $\textbf{x}'''\in \mathbb{R}^{\ast\times D}$, which is the final output. A pythonic style formulation is $\mbox{ChannelReorder}(g, \textbf{x})=(\textbf{x}.\mbox{reshape}(\ast, g,\frac{D}{g})).\mbox{transpose}(-2, -1).\mbox{reshape}(\ast, D)$
We set shuffle groups $g=g_{i}$ in the $i$-th branch, where $g_{i}$ is the group of the corresponding GC layer. In this way, there are few overlaps between the weight matrices of distinct branches, and the model capacity is greatly expanded each time a new branch is contained.
\textbf{Stochastic depth of pre-trained weight.}
To further enlarge the model's adaptation capacity, we append a droppath~\citep{huang2016droppath} layer to each branch. For small downstream datasets, dropping the consolidator path with a higher ratio $p$ can help reduce overfitting and catastrophic forgetting, which may be beneficial to the performance. Empirically shown in \cref{subsec:ablation}, droppath is more effective than standard dropout~\citep{srivastava2014dropout} in the current situation probably for the following reasons. The parameters of the whole layer degrade into the pre-trained weight parameters with probability $p$. The frozen pre-trained parameters contain domain-generic knowledge, which may help the adaptation. Overall, a model modified by consolidator will have a stochastic depth of pre-trained state of parameters during each forward pass, and different consolidators will be activated for different training samples.
\textbf{Two-stage consolidation.}
Now we have constructed all the elements of a consolidator. Formally, the output of the whole layer after being modified by consolidator is
$\textbf{y}=\textbf{W}\textbf{x}+\textbf{b}+\mbox{Droppath}(p, \sum_{i=1}^{m}(\sum_{j=1}^{g^{(i)}}{\mbox{Pad}(\textbf{W}_{j}^{(i)}\mbox{ChannelReorder}(g^{(i)},\textbf{x})_{\frac{(j-1)D}{g^{(i)}}:\frac{jD}{g^{(i)}}}, j)}+\textbf{b}^{(i)}))$.
Since all the operations in a consolidator are inference-time linear, we can easily consolidate the domain-specific knowledge into the domain-agnostic knowledge in the pre-trained backbone model, in both the training-storage phase and the loading-inference phase.
1. Training-storage consolidation. All we need to store in a consolidator are $\textbf{W}^{(i)}$ and $\textbf{b}^{(i)}$. However, there are some parameters corresponding to the same entry, and thus they can be merged into a single one. As shown in \cref{fig:Consolidator}, we also tune the bias of the original FC layer in addition to the parameters in consolidator. It is easy to find that the duplicate biases in all branches and the original bias can be merged into a single one. And there are some duplicate entries in the weight matrix as well, so we can merge all weight matrices into one single sparse matrix. Consolidating such duplicate entries can largely benefit storage. Formally, we use $\widetilde{\textbf{W}}$ and $\widetilde{\textbf{b}}$ to denote the matrix that we need to store on the disk. Since channel reorder is a linear operation, we can apply a reverse operation to $\textbf{W}^{(i)}$ to simulate the effect of the reorder applied to $\textbf{x}$. And we have $\widetilde{\textbf{W}}=\sum_{i=1}^{m}{\mbox{ChannelReorder}^{-1}(g^{(i)},\mbox{Compact}(\textbf{W}^{(i)}))}$. Here $\mbox{Compact}$ reshapes the input matrix for the preparation of reordering its channels. It is easy to verify that $\mbox{ChannelReorder}^{-1}(g^{(i)},\bullet)=\mbox{ChannelReorder}(\frac{D}{g^{(i)}},\bullet)$.
2. Loading-inference consolidation. After loading the merged sparse weight matrix and merged bias matrix to memory, we can directly add them back to the weight matrix and bias matrix of the original FC layer. Formally, we use $\hat{\textbf{W}}$ and $\hat{\textbf{b}}$ to denote the final weight and bias of the FC layer for inference. Then $\hat{\textbf{W}}=\textbf{W}+\widetilde{\textbf{W}}$, $\hat{\textbf{b}}=\widetilde{\textbf{b}}$. In this way, no additional inference cost is brought.
Overall, our consolidator reduces the storage space by using grouped connected layers and consolidating some duplicate entries. The training time non-linearity, \eg, droppath, which turns out to be linear in inference time, effectively enriches model capacity under a constrained storage budget. Finally, we can consolidate the task-specific knowledge into the backbone model by merging the inference time linear components to enjoy free, efficient, and effective transfer learning.
\section{Experiments}
\label{sec:experiments}
\subsection{Experimental settings}
\textbf{Baselines.} We select several state-of-the-art parameter-efficient methods as our baselines, including Full, Head, Bias~\citep{zaken2021bitfit}, Adapter~\citep{houlsby2019parameter}, VPT~\citep{jia2022vpt}, LoRA~\citep{hu2021lora}, AdaptFormer~\citep{chen2022adaptformer}, NOAH~\citep{zhang2022noah} and SSF~\citep{lian2022ssf}. Note that Adapter, AdaptFormer, VPT, and NOAH will bring heavy inference cost to the pre-trained model which may cause troubles in resource-limited devices, while LoRA, SSF and our consolidator are more friendly for deployment with no extra inference cost brought.
\textbf{VTAB-1k.} We first run experiments on VTAB-1k~\citep{zhai2019vtab} benchmark, which covers a wide range of visual domains in 19 datasets. Each dataset contains 1000 images picked from the original dataset for training, and the size of test set stays unchanged, varying from 711 to 73728.
\textbf{Full data setting.} VTAB-1k merely has a small number of training images, and thus the capacity of a huge transformer is redundant to some extent. Therefore, to check the performance of parameter-efficient methods in a data-sufficient situation, we select 10 widely-used datasets for visual recognition in various domains and use the original training, validation, and test split for experiments. On average, a dataset in this setting contains a lot more images for training, leaving huge space for the model to adapt. The chosen datasets include natural pictures (Caltech101~\citep{fei2004caltech101}, Cifar10~\citep{krizhevsky2009cifar}, Cifar100~\citep{krizhevsky2009cifar}), fine-grained classification (CUB200~\citep{wah2011cub}, OxfordFlowers~\citep{nilsback2008flowers}, OxfordPets~\citep{parkhi2012pets}, StanfordDogs~\citep{khosla2011stanforddogs}), textures (DTD~\citep{cimpoi2014dtd}), scene classification (SUN397~\citep{xiao2010sun}) and satellite images (EuroSAT~\citep{helber2019eurosat}).
\begin{table}[]
\fontsize{7.2}{8.7}
\selectfont
\setlength{\tabcolsep}{0.7mm}
\begin{center}
\setlength{\belowcaptionskip}{-0.3cm}
\caption{Full results on the VTAB-1k~\citep{zhai2019vtab} benchmark. The bold font denotes the best accuracy and the underline font denotes the second best accuracy in each column. Consolidator gives the strongest results, surpasses full fine-tuning by 7.56 accuracy on average, and outperforms the state-of-the-art methods with low storage overhead and no inference cost.}
\label{tab:in21k_vit_base_vtab}
\begin{tabular}{lc|ccccccc|cccc|cccccccc|c}
\toprule
& & \multicolumn{7}{c}{\textbf{Natural}} \vline & \multicolumn{4}{c}{\textbf{Specialized}} \vline & \multicolumn{8}{c}{\textbf{Structured}} \vline & \\
& \rotatebox{90}{\# params} & \rotatebox{90}{Cifar100} & \rotatebox{90}{Caltech101} & \rotatebox{90}{DTD} & \rotatebox{90}{Flowers102} & \rotatebox{90}{Pets} & \rotatebox{90}{SVHN} & \rotatebox{90}{Sun397} & \rotatebox{90}{Camelyon} & \rotatebox{90}{EuroSAT} & \rotatebox{90}{Resisc45} & \rotatebox{90}{Retinopathy} & \rotatebox{90}{Clevr-Count} & \rotatebox{90}{Clevr-Dist} & \rotatebox{90}{DMLAB} & \rotatebox{90}{KITTI-Dist} & \rotatebox{90}{dSpr-Loc} & \rotatebox{90}{dSpr-Ori} & \rotatebox{90}{sNORB-Azim} & \rotatebox{90}{sNORB-Ele} & \rotatebox{90}{Average} \\
\midrule
Full & 100\% & 68.9 & 87.7 & 64.3 & 97.2 & 86.9 & 87.4 & 38.8 & 79.7 & 95.7 & 84.2 & 73.9 & 56.3 & 58.6 & 41.7 & 65.5 & 57.5 & 46.7 & 25.7 & 29.1 & 68.97 \\
Head & 0.04\% & 63.4 & 85.0 & 63.2 & 97.0 & 86.3 & 36.6 & 51.0 & 78.5 & 87.5 & 68.6 & 74.0 & 34.3 & 30.6 & 33.2 & 55.4 & 12.5 & 20.0 & 9.6 & 19.2 & 57.64 \\
Bias & 0.10\% & 72.8 & 87.0 & 59.2 & 97.5 & 85.3 & 59.9 & 51.4 & 78.7 & 91.6 & 72.9 & 69.8 & 61.5 & 55.6 & 32.4 & 55.9 & 66.6 & 40.0 & 15.7 & 25.1 & 65.22 \\
\midrule
VPT & 0.75\% & \textbf{78.8} & 90.8 & 65.8 & 98.0 & 88.3 & 78.1 & 49.6 & 81.8 & \textbf{96.1} & 83.4 & 68.4 & 68.5 & 60.0 & 46.5 & 72.8 & 73.6 & 47.9 & \textbf{32.9} & 37.8 & 71.97 \\
Adapter & 0.36\% & 69.8 & 91.2 & 68.8 & 99.0 & 89.9 & 85.7 & 53.8 & 82.3 & 95.5 & 83.7 & \textbf{76.1} & 82.4 & 65.0 & 48.2 & 80.5 & 74.5 & 49.7 & 29.8 & 39.2 & 74.28 \\
LoRA & 0.34\% & 68.1 & 90.1 & 69.8 & 98.9 & 90.8 & 84.8 & 54.0 & 83.2 & 95.6 & 84.2 & 73.9 & 82.4 & \underline{68.7} & 49.4 & 80.0 & \underline{81.7} & 46.1 & 31.4 & \underline{41.8} & 74.64 \\
AdaptFormer & 0.36\% & 71.0 & 91.1 & 69.9 & 99.3 & 90.5 & 87.4 & \underline{54.8} & 84.1 & 95.9 & 85.9 & 75.8 & \textbf{83.1} & 63.8 & 49.6 & 79.6 & 76.5 & 45.1 & 30.9 & 39.2 & 74.82 \\
NOAH & 0.52\% & 69.6 & \textbf{92.7} & 70.2 & 99.1 & 90.4 & 86.1 & 53.7 & 84.4 & 95.4 & 83.9 & 75.8 & \underline{82.8} & \textbf{68.9} & 49.9 & \underline{81.7} & \textbf{81.8} & 48.3 & \underline{32.8} & \textbf{44.2} & 75.48 \\
SSF & 0.29\% & 69.0 & \underline{92.6} & \textbf{75.1} & \underline{99.4} & \textbf{91.8} & \underline{90.2} & 52.9 & \textbf{87.4} & \underline{95.9} & \textbf{87.4} & 75.5 & 75.9 & 62.3 & \textbf{53.3} & 80.6 & 77.3 & \textbf{54.9} & 29.5 & 37.9 & \underline{75.69} \\
Ours & 0.35\% & \underline{74.2} & 90.9 & \underline{73.9} & \textbf{99.4} & \underline{91.6} & \textbf{91.5} & \textbf{55.5} & \underline{86.9} & 95.7 & \textbf{86.6} & \underline{75.9} & 81.2 & 68.2 & \underline{51.6} & \textbf{83.5} & 79.8 & \underline{52.3} & 31.9 & 38.5 & \textbf{76.53} \\
\bottomrule
\end{tabular}
\end{center}
\end{table}
\subsection{Main results}
We first choose a ViT-B~\citep{dosovitskiy2020image} with 86M parameters as a base model.
\textbf{VTAB-1k} \cref{tab:in21k_vit_base_vtab} presents the full results on VTAB-1k benchmark. Overall, our consolidator is the best parameter-efficient method. On 12 of 19 datasets, consolidator achieves the best or second best top-1 accuracy. Notably, consolidator surpasses the state-of-the-art methods NOAH and SSF by a clear margin, with low storage overhead and no inference cost.
\textbf{Full data setting} \cref{tab:in21k_vit_base_full_data} presents the full results on full data setting. Overall, our consolidator still performs best. An interesting observation is that the rank of full fine-tuning rises as the training data increase. None of the parameter-efficient methods can reach comparable performance with full tine-tuning other than our consolidator within 0.5\% parameter storage overhead. In contrast, the parameter-efficient methods can reach at least 5\% higher accuracy on VTAB-1k than full fine-tuning under comparable or even lower storage budget (around 0.5\%), as shown in \cref{tab:in21k_vit_base_vtab}.
\begin{table}
\begin{minipage}[t]{0.62\linewidth}
\fontsize{7.2}{8.7}
\selectfont
\setlength{\tabcolsep}{0.8mm}
\centering
%\setlength{\belowcaptionskip}{-0.3cm}
\caption{Full results on data-sufficient scenarios. It is more challenging to take full advantage of a large amount of data within a fairly small number of stored parameters. Consolidator earns the best or second best in all 10 datasets.}
%In this setting, full fine-tuning is a strong competitor and only consolidator can reach a comparable or better performance than full fine-tuning with merely around 0.5\% parameters stored.}
\label{tab:in21k_vit_base_full_data}
\begin{tabular}{lc|cccccccccc|c}
\toprule
& & \multicolumn{10}{c}{Full data} \vline\\
& \rotatebox{90}{\# params} & \rotatebox{90}{Caltech101} & \rotatebox{90}{Cifar10} & \rotatebox{90}{Cifar100} & \rotatebox{90}{CUB} & \rotatebox{90}{DTD} & \rotatebox{90}{Flowers102} & \rotatebox{90}{Pets} & \rotatebox{90}{Dogs} & \rotatebox{90}{Sun397} & \rotatebox{90}{EuroSAT} & \rotatebox{90}{Average}\\
\midrule
Full & 100\% & 90.9 & \underline{99.1} & \textbf{92.6} & \textbf{87.7} & \textbf{75.6} & 98.9 & \textbf{93.9} & 84.8 & 70.9 & \textbf{98.6} & \underline{89.30}\\
Head & 0.10\% & 90.0 & 88.8 & 71.6 & 83.5 & 71.3 & 98.2 & 90.5 & 78.4 & 67.0 & 95.6 & 83.49\\
Bias & 0.22\% & 90.2 & 99.0 & 91.9 & 86.4 & 73.5 & 98.5 & 92.6 & 85.4 & 71.0 & 97.7 & 88.62\\
\midrule
Adapter & 0.55\% & \underline{91.0} & 99.0 & 92.2 & 86.3 & 72.6 & 98.7 & 92.5 & \underline{85.9} & \underline{71.9} & 97.4 & 88.75\\
LoRA & 0.53\% & 90.1 & 99.0 & 91.8 & 85.4 & 71.9 & 98.1 & 92.1 & 85.6 & 71.3 & 97.2 & 88.25\\
AdaptFormer & 0.55\% & 90.9 & 99.0 & 92.3 & 86.5 & 73.4 & \underline{98.9} & 92.9 & 85.2 & 71.4 & 97.1 & 88.76\\
SSF & 0.30\% & 90.9 & 99.0 & 92.1 & 85.7 & 72.6 & 98.3 & 92.3 & 85.7 & 71.6 & 97.8 & 88.60\\
Ours & 0.50\% & \textbf{91.4} & \textbf{99.1} & \underline{92.5} & \underline{87.0} & \underline{74.5} & \textbf{99.0} & \underline{93.2} & \textbf{86.4} & \textbf{72.1} & \underline{97.9} & \textbf{89.31}\\
\bottomrule
\end{tabular}
\end{minipage}\qquad
\begin{minipage}[t]{0.32\linewidth}
\fontsize{7.}{8.7}
\selectfont
\setlength{\tabcolsep}{0.8mm}
\centering
%\setlength{\belowcaptionskip}{-0.3cm}
\makeatletter\def\@captype{table}\makeatother\caption{Adaptation results for a self-supervised model, MoCo v3 ViT-B. Consolidator earns the best average accuracy}.
\label{tab:mocov3_vit_base_both}
\begin{tabular}{l|cc|cc}
\toprule
& \multicolumn{2}{c}{VTAB-1k} \vline & \multicolumn{2}{c}{Full data} \\
& \rotatebox{90}{\# params} & \rotatebox{90}{Average} & \rotatebox{90}{\# params} & \rotatebox{90}{Average}\\
\midrule
Full & 100\% & 69.55 & 100\% & 86.24\\
Head & 0.04\% & 59.62 & 0.10\% & 75.42\\
Bias & 0.10\% & 69.15 & 0.22\% & 80.96\\
\midrule
Adapter & 0.36\% & 73.22 & 0.55\% & 86.15\\
LoRA & 0.34\% & 72.73 & 0.53\% & 79.70\\
AdaptFormer & 0.36\% & \underline{74.03} & 0.55\% & \underline{86.26} \\
NOAH & 0.42\% & 73.55 & —— & —— \\
SSF & 0.29\% & 51.41 & 0.30\% & 80.73 \\
Ours & 0.35\% & \textbf{74.71} & 0.50\% & \textbf{86.41}\\
\bottomrule
\end{tabular}
\end{minipage}
\end{table}
\subsection{Results for self-supervised vision transformer}
Then we use a self-supervised trained transformer by MoCov3~\citep{he2020momentum} as our target model.
Seen from \cref{tab:mocov3_vit_base_both}, on both VTAB-1k and full data setting, consolidator consistently reaches the highest average top-1 accuracy. AdaptFormer gives the second highest accuracy. SSF significantly falls behind others (51.41 on VTAB and 80.73 on full data setting) when dealing with MoCo v3 ViT-B, showing limited generalization ability for self-supervised visual models.
\begin{table}
\setlength\tabcolsep{1pt}
\fontsize{7.5}{9}
\selectfont
\begin{center}
\setlength{\belowcaptionskip}{-0.4cm}
\caption{Adaptation performance for more models in full data setting. Consolidator consistently reaches a better result than full fine-tuning within a very small set of parameters which is needed to be stored. Generally, it is more difficult to reach a better result than full fine-tuning when the model capacity is insufficient, \ie, the model does not accommodate enough parameters in total.}
\label{tab:generalization_experiment}
\begin{tabular}{ll|p{1.4cm}<{\centering}p{1.4cm}<{\centering}p{1.4cm}<{\centering}p{1.4cm}<{\centering}p{1.4cm}<{\centering}|p{1.4cm}<{\centering}}
%\begin{tabular}{ll|ccccc|c}
\toprule
\multicolumn{2}{l|}{} & \multicolumn{5}{c|}{Supervised Learning} & MAE \\
\multicolumn{2}{l}{Architecture} \vline & ViT-S & ViT-L & Swin-B & AS-MLP-B & ConvNeXt-B & ViT-B\\
\midrule
\multicolumn{2}{l}{Total Parameters} \vline & 22M & 303M & 87M & 87M & 88M & 86M \\
\midrule
\multirow{2}*{Full} & Average & \underline{89.10} & \underline{90.07} & 91.00 & \textbf{87.75} & \textbf{91.92} & 82.77 \\
& \# params & 100\% & 100\% & 100\% & 100\% & 100\% & 100\%\\
\midrule
\multirow{2}*{Head} & Average & 81.87 & 87.92 & 90.45 & 83.49 & 90.62 & 58.10 \\
& \# params & 0.20\% & 0.04\% & 0.13\% & 0.13\% & 0.13\% & 0.10\%\\
\midrule
\multirow{2}*{Bias} & Average & 87.86 & 89.81 & \underline{91.00} & 86.26 & 91.61 & 79.80 \\
& \# params & 0.44\% & 0.13\% & 0.36\% & 0.32\% & 0.28\% & 0.22\%\\
\midrule
\multirow{2}*{LoRA} & Average & 87.55 & 89.86 & 90.95 & \multirow{2}*{--} & \multirow{2}*{--} & 82.22 \\
& \# params & 5.12\% & 0.33\% & 0.80\% & & & 1.82\%\\
\midrule
\multirow{2}*{Adapter} & Average & 88.28 & 89.91 & 90.98 & \multirow{2}*{--} & \multirow{2}*{--} & \underline{82.89} \\
& \# params & 5.09\% & 0.35\% & 0.78\% & & & 1.80\%\\
\midrule
\multirow{2}*{Ours} & Average & \textbf{89.12} & \textbf{90.52} & \textbf{91.28} & \underline{86.71} & \underline{91.79} & \textbf{83.32} \\
& \# params & 5.06\% & 0.33\% & 0.77\% & 1.13\% & 1.04\% & 1.78\% \\
\bottomrule
\end{tabular}
\end{center}
\end{table}
\subsection{Results for more pre-trained models}
To further verify the generalization ability of consolidator, we conduct extensive experiments in \cref{tab:generalization_experiment} based on full data setting. First, we apply consolidator to supervised learned models with larger (ViT-L) or smaller (ViT-S) size than standard ViT-B. Compared with full fine-tuning, we achieve a comparable performance for ViT-S with 5.06\% parameters. For ViT-L, merely 0.33\% parameters can lead to 0.45 higher than full fine-tuning. Then we experiment on Swin-B, a hierarchical architecture using shifted windows to introduce locality for better recognition. We observe a 0.28 improvement while storing only 0.77\% parameters. Next, we further verify the effectiveness of consolidator on other vision architectures other than transformers, \eg AS-MLP~\citep{lian2022mlp} and ConvNeXt~\citep{liu2022convnet}.
Finally, we experiment on ViT-B pre-trained by a generative SSL method, MAE, as a comparison with the contrastive SSL method MoCo v3. Our method stores 1.78\% parameters onto the disk while enjoying 0.55 higher accuracy.
Generally, it is more difficult to perfom better than full
fine-tuning when the model does not have enough parameters to fully leverage the information from massive training data.
\subsection{Adaptation across varying scales of stored parameters}
\begin{figure}
\centering
\begin{subfigure}{0.8\linewidth}
\centering
\includegraphics[width=\linewidth]{imgs/Param_and_Data_scale.pdf}
\end{subfigure}
\setlength{\belowcaptionskip}{-0.55cm}
\caption{Left: the Adaptation results corresponding to varying ratios of stored parameters. Clearly, consolidator consistently outperforms LoRA, adapter, and full fine-tuning by a significant margin across a wide range of parameter scales from 0.5\% to 10\%. And each of the three methods will reach higher accuracy if we increase its storage budget and tune more parameters. \quad Right: adaptation results corresponding to varying sampling rates of data. Consolidator performs best across all sampling rates. As the sampling rate decreases, full fine-tuning slightly falls off its advantage over adapter and LoRA, which is consistent with our previous observations.}
\label{fig:Param_scale}
\end{figure}
Next, we seek to find the principle between downstream accuracy and the number of stored parameters for all parameter-efficient methods with flexible parameter scales, based on full data setting. Results are shown in \cref{fig:Param_scale} left. LoRA, adapter, and consolidator reach better performance as the number of parameters increases. In various parameter scales (from 0.5\% to 10\%), our consolidator consistently outperforms all competitors by a clear margin.
\subsection{Adaptation across varying data sampling ratios}
\cref{fig:Param_scale} right shows adaptation results corresponding to varying sampling rates of datasets, based on full data setting. For all sampling rates, consolidator keeps the best adaptation ability. In addition, as the sampling rate decreases, full fine-tuning slightly falls off its advantage over Adapter and LoRA, which is consistent with previous observations on VTAB-1k and full data setting.
\subsection{Results on object detection and semantic segmentation.}
We further verify our method on downstream object detection and semantic segmentation tasks. We adopt Swin-Base as the backbone which can provide hierarchical features. Experiments are done on PASCAL VOC 07+12~\citep{Everingham10voc} for detection and PASCAL VOC 12~\citep{Everingham10voc} for segmentation. We adopt Faster R-CNN~\citep{ren2015faster} and UperNet~\citep{xiao2018unified} separately for detection and segmentation framework. Seen from \cref{tab:det_seg}, our consolidator significantly outperforms full training and detection/segmentation head training with a small number of parameters stored, showing great potential for broader usage.
\subsection{Ablation studies}
\label{subsec:ablation}
We do controlled experiments to identify the effect of individual components in our module design. We report tuned parameters, stored parameters, and accuracy in \cref{tab:ablation}. We experiment on 5 datasets with different domains: Caltech101, DTD, OxfordFlowers, StanfordDogs, and EuroSAT.
\begin{table}
\fontsize{8}{10}
\selectfont
\begin{center}
\setlength{\belowcaptionskip}{-0.3cm}
\caption{Performance on downstream object detection and semantic segmentation tasks. Compared with tuning detection/segmentation head only, consolidator reaches much better mAP/mIoU with negligible parameters increase and surpasses the performance of tuning all parameters in both backbone and task head as well. }
\label{tab:det_seg}
\begin{tabular}{l|c|ccc|ccc}
\toprule
\multirow{2}*{Method} & \multirow{2}*{Backbone} & \multicolumn{3}{c|}{Object Detection} & \multicolumn{3}{c}{Semantic Segmentation} \\
& & Framework & \# params & mAP & Framework & \# params & mIoU \\
\midrule
Full & \multirow{3}*{Swin-B} & \multirow{3}*{Faster-RCNN} & 100\% & 85.85 & \multirow{3}*{UperNet} & 100\% & 83.24\\
Head & & & 16.74\% & 84.89 & & 28.48\% & 82.31\\
\textbf{Consolidator} & & & \textbf{17.22\%} & \textbf{86.64} & & \textbf{28.89\%} & \textbf{83.96}\\
\bottomrule
\end{tabular}
\end{center}
\end{table}
\textbf{Droppath \textit{v.s.} Dropout.} We first investigate our choice of Droppath. Droppath with a consolidator of ($g^{(1)}=96$, $g^{(2)}=192$) as the base model. As shown in \cref{tab:ablation}, compared with dropout and no drop-type layer, droppath can obtain 0.44 and 0.48 performance improvement, respectively, well demonstrating the effectiveness of encouraging stochastic depth for consolidator.
\textbf{ChannelReorder.} The effect of ChannelReorder operation mainly lies in separating the entries into different branches to reduce the repetitive ones. In a multi-branch case like ($g^{(1)}=96,g^{(2)}=192$), ChannelReorder brings 0.38 accuracy improvement. Furthermore, it is helpful even if there is only one branch like ($g^{(1)}=384$), where ChannelReorder still slightly raises the accuracy by 0.1.
\textbf{Duplication of bias and weight.} Based on the consolidator with ($g^{(1)}=96,g^{(2)}=192$), we can see duplicating bias is relatively effective. Compared with tuning the original bias and only tuning two extra biases, tuning all three biases can lead to 0.44 and 0.1 performance improvement, respectively, with the same storage cost. Additionally, we also compare tuning original bias \textit{v.s.} not tuning bias, and the former only has a slight 0.1 accuracy advantage, which further verifies the effectiveness owes mostly to the delicate consolidation design instead of simple bias tuning. Besides bias, we test on a consolidator with ($g^{(1)}=96$) to investigate the effect of integrating duplicate weights. However, this kind of consolidation does not bring notable improvement.
\textbf{Structured \textit{v.s.} Unstructured.} One potential limitation of consolidator is that $g^{(i)}$ can not be selected arbitrarily for it must be a factor of the channels. This may cause trouble when fairly few parameters other than head parameters, \eg 0.0001\%, are required to be tuned. A solution is to adopt unstructured sparsity instead of structured block-wise sparsity in consolidator branches to flexibly control the parameter number. We simulate the situation with ($g^{(1)}=384$) for comparison with a unstructured implementation. Seen from \cref{tab:ablation}, the unstructured branch whose tunable parameter number equals that of a branch with $g=384$ faces a slight accuracy drop, 0.14. In summary, the unstructured consolidator can be a sub-optimal choice when needed, with a slight performance drop and more training cost due to the unstructured sparse matrix being unfriendly to the hardware.
\begin{table}
\fontsize{7}{8.4}
\selectfont
\setlength{\tabcolsep}{0.5mm}
\begin{center}
\setlength{\belowcaptionskip}{-0.5cm}
\caption{Effect of individual designs for a consolidator module.}
\label{tab:ablation}
\begin{tabular}{c|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|c|c}
%\toprule
\hline
\makecell[c]{Branches} & \makecell[c]{Original\\Bias} & \makecell[c]{Extra\\Bias} & \makecell[c]{Channel\\Reorder} & \makecell[c]{Dropout} & \makecell[c]{Droppath} & \makecell[c]{Tuned\\Params} & \makecell[c]{stored\\Params} & Accuracy & $\Delta$Accuracy\\
\hline
%\midrule
96, 192 & \usym{2713} & \usym{2713} & \usym{2713} & \usym{2717} & \usym{2713} & 1.95\% & 1.75\% & 87.36 & -0.00\\
96, 192 & \usym{2713} & \usym{2713} & \usym{2713} & \usym{2713} & \usym{2717} & 1.95\% & 1.75\% & 86.92 & -0.44\\
96, 192 & \usym{2713} & \usym{2713} & \usym{2713} & \usym{2717} & \usym{2717} & 1.95\% & 1.75\% & 86.88 & -0.48\\
96, 192 & \usym{2713} & \usym{2713} & \usym{2717} & \usym{2717} & \usym{2713} & 1.95\% & 1.24\% & 86.98 & -0.38\\
96, 192 & \usym{2717} & \usym{2713} & \usym{2713} & \usym{2717} & \usym{2713} & 1.85\% & 1.75\% & 87.26 & -0.10\\
96, 192 & \usym{2713} & \usym{2717} & \usym{2713} & \usym{2717} & \usym{2713} & 1.76\% & 1.75\% & 86.92 & -0.44\\
96, 192 & \usym{2717} & \usym{2717} & \usym{2713} & \usym{2717} & \usym{2713} & 1.66\% & 1.65\% & 86.82 & -0.54\\
\hline
384 & \usym{2713} & \usym{2713} & \usym{2713} & \usym{2717} & \usym{2713} & 0.56\% & 0.47\% & 87.08 & -0.00\\
384 & \usym{2713} & \usym{2713} & \usym{2717} & \usym{2717} & \usym{2713} & 0.56\% & 0.47\% & 86.98 & -0.10\\
384, 384 & \usym{2713} & \usym{2713} & \usym{2713} & \usym{2717} & \usym{2713} & 0.92\% & 0.47\% & 86.96 & -0.12\\
\hline
384 & \usym{2717} & \usym{2717} & \usym{2713} & \usym{2717} & \usym{2713} & 0.37\% & 0.37\% & 86.84 & -0.00\\
384 (unst)& \usym{2717} & \usym{2717} & \textbf{N/A} & \usym{2717} & \usym{2713} & 0.37\% & 0.37\% & 86.70 & -0.14\\
\hline
%\bottomrule
\end{tabular}
\end{center}
\end{table}
\section{Conclusions}
We propose consolidator, a novel method to achieve both parameter- and inference-efficient visual adaptation. Consolidator adds a few tunable, mergeable modules along each fully connected layer in the pre-trained model and keeps most of the original parameters frozen during adaptation. We design a two-stage consolidation to dramatically boost performance under a given storage budget. The duplicate entries in a consolidator will be merged into a single matrix and stored on disk. Finally, we consolidate the task-specific parameters in consolidator into the tasks-agnostic parameters in the pre-trained model, bringing no extra inference cost. On various tasks, consolidator outperforms all state-of-the-art competitors significantly, showing strong scalability and generalization ability.
\subsubsection*{Acknowledgments}
This work was supported by National Key R\&D Program of China (No. 2021ZD0114703), National
Natural Science Foundation of China (Nos. 61925107, 62271281, U1936202, 61571269), Beijing Natural Science Foundation (No. L223023), China Postdoctoral Science Foundation (BX2021161) and Zhejiang Lab Open Research Project (NO. K2022KI0AB01).
\bibliography{iclr2023_conference}
\bibliographystyle{iclr2023_conference}
\appendix
\section{Experiment Settings}
\subsection{Image recognition}
\subsubsection{Datasets}
\textbf{VTAB-1k.} VTAB-1k~\citep{zhai2019vtab} consists of 19 visual datasets in three groups: Natural, Specialized, and Structured, containing images collected from a wide range of visual domains. Each dataset is divided into the training set (800 images), validation set (200 images), and test set (the original set). The final adaptation accuracy is reported on the test set. All models are fine-tuned on the full 1000 labeled images, \ie, train+val set. The validation set is used for tuning some hyperparameters.
\textbf{Full data.} We select 10 widely-used visual datasets with sufficient data to further verify the effectiveness of the parameter-efficient methods. The datasets are shown in the following part:
\textit{Caltech101}~\citep{fei2004caltech101} contains pictures of natural objects belonging to 101 classes, with 3060 images for training and 6084 images for testing.
\textit{CIFAR-10/100}~\citep{krizhevsky2009cifar} contains pictures of natural objects belonging to 10/100 classes, with 50000 images for training and 10000 images for testing.
\textit{Caltech-UCSD Birds-200-2011 (CUB)}~\citep{wah2011cub} contains pictures of birds belonging to 200 fine-grained bird classes, with 5994 images for training and 5794 images for testing.
\textit{Describable Textures Dataset (DTD)}~\citep{cimpoi2014dtd} contains texture images in the wild belonging to 47 human-centric classes, with 5640 images in total which are equally split into the training, validation, and test set.
\textit{OxfordFlowers102}~\citep{nilsback2008flowers} contains pictures of flowers belonging to 102 fine-grained flower classes, with 1020 images for training, 1020 images for validating, and 6149 images for testing.
\textit{Oxford-IIITPets}~\citep{parkhi2012pets} contains pictures of pets belonging to 37 fine-grained pet classes, with 3680 images for training and 3669 images for testing.
\textit{StanfordDogs}~\citep{khosla2011stanforddogs} contains pictures of dogs belonging to 120 fine-grained dog classes, with 12000 images for training and 8580 images for testing.
\textit{Scene UNderstanding (SUN397)}~\citep{xiao2010sun} contains pictures of scenes belonging to 397 classes, with 76128 images for training, 10875 images for validating, and 21750 images for testing.
\textit{EuroSAT}~\citep{helber2019eurosat} contains pictures based on Sentinel-2 satellite images belonging to 10 classes, with 27000 images in total.
Some of the datasets do not contain a validation split, and we will manually select \~10\% random images from the training set as the validation set.
To better utilize the massive data in various domains, we follow the practice in~\citep{jiang2020dalib,jiang2022transferability} to make preparation and divide data splits. For data augmentation, we adopt a standard pipeline. In training, we do a random resize crop to 224$\times$224, random horizontal flip, and normalization for each input image. In test, we do a resize to 256$\times$256, center crop to 224$\times$224, and normalization for each input image.
\subsubsection{Training hyperparameters.}
On \textbf{VTAB-1k}, we follow the hyperparameters in VPT~\citep{jia2022vpt} for full fine-tuning, Head, Bias, and VPT, and mainly follow the hyperparameters in NOAH~\citep{zhang2022noah} and SSF~\citep{lian2022ssf} for adapter, LoRA, NOAH, and consolidator. The detailed hyperparameters for each tuning method can be found in \cref{tab:hyperparam_method}.
On \textbf{full data setting}, we do a quick grid search to choose a proper set of training hyperparameters based on the performance of full fine-tuning for every well-trained visual representation. All training hyperparameters are shown in \cref{tab:hyperparam_training,tab:hyperparam_method}. Following the practice in~\citep{jiang2022transferability,jiang2020dalib}, we reduce the learning rate of backbone parameters to 0.1x that of head parameters. We then adopt the same hyperparameters for all parameter-efficient variants as well as our consolidator after the hyperparameters have been chosen according to full fine-tuning results.
\subsubsection{Methods.}
\textit{Full}: Ordinary fine-tuning. It tunes all parameters and stores all parameters to disk for every downstream task, leading to heavy storage cost.
\textit{Head}: Also known as linear probing. It only tunes the classification head and freezes other parameters.
\textit{Bias}~\citep{zaken2021bitfit}: Bias only tunes the biases and freezes all the weights in a pre-trained model.
\textit{Adapter}~\citep{houlsby2019parameter}: Adapter inserts a sequence of tunable parameters, including a down projection, a non-linearity (here we use GELU~\citep{hendrycks2016gelu}) and an up projection layer, into each encoder block. The adapters are serially connected with the backbone layers.
\textit{LoRA}~\citep{hu2021lora}: LoRA adds a concurrent branch containing low-rank weight matrices for efficient parameter update. A LoRA module contains serial down projection and up projection layers without non-linearity, and thus it can be merged into the backbone parameters before inference.
\textit{VPT}~\citep{li2021prefix,jia2022vpt}: VPT appends tunable virtual tokens to the inputs of transformer blocks, which will participate in the calculation of the subsequent blocks along with the actual tokens.
\textit{NOAH}~\citep{zhang2022noah}: NOAH first trains a large supernet with all three modules, VPT, LoRA, and adapter, and then searches for the optimal configurations of each module for each layer using the NAS algorithm introduced by AutoFormer~\citep{chen2021AutoFormer}. In the end, NOAH retrains the best subnet candidates to produce the final result.
\textit{AdaptFormer}~\citep{chen2022adaptformer}: AdaptFormer adopts parallel adapters~\citep{houlsby2019parameter} and a scale operation for each encoder block.
\textit{SSF}~\citep{lian2022ssf}: SSF adds tunable scale and shift parameters for each operation of the backbone model. The added parameters can be merged into the original model and thus SSF brings no inference cost.
\textbf{Implementation details.}
On \textbf{VTAB-1k}, we follow the same implementations for LoRA and adapter with NOAH~\citep{zhang2022noah}.
On \textbf{Full data setting}, the detailed implementations are shown as follows. For LoRA~\citep{hu2021lora}, we follow its original implementation to do low-rank re-parameterization and tune merely two weight matrices $\textbf{W}_{q}$ and $\textbf{W}_{v}$, which generate the attention matrices $\textbf{Q}$ and $\textbf{V}$, in every transformer encoder block. For bias, we tune all the biases (including the parameters outside encoder blocks) in the model. For adapter, we follow its original implementation to additionally tune the parameters of LayerNorm~\citep{ba2016layernorm} and add the tunable adapter modules to the end of each MHSA and MLP before calculating the residual connections. For consolidator, we follow the practice of adapter to tune the LayerNorm as well and only add consolidator to the linear layers in MHSA and MLP of each transformer encoder block. We sweep the drop ratio in \{0.0, 0.2, 0.5, 0.8\}. We choose training hyperparameters according to the performance of full-finetuning for each visual representation. The hyperparameters are finally configured as in \cref{tab:hyperparam_method}. When applying consolidator to Swin-B, we skip the first stage of transformer encoder blocks that contain a relatively small part of parameters (0.6\% of total parameters) compared with others and merely insert consolidator for linear layers in MHSA and MLP of blocks of last 3 stages.
\subsection{Object detection}
On the downstream object detection task, we adopt Faster R-CNN~\citep{ren2015faster} framework with FPN~\citep{lin2017fpn} to verify the effectiveness of consolidator on Pascal VOC 07+12 dataset~\citep{Everingham10voc}. We use a Swin-B pre-trained on IN-21k as the backbone model. The consolidator setting is the same as the setting of Swin-B in \cref{tab:hyperparam_method}. For hyperparameters, we adopt AdamW as the optimizer with a learning rate of 1e-4 and weight decay of 5e-2, and train for 8 epochs in total. The learning rate is decayed by a factor of 10 after 6 epochs. The first 300 iterations are trained with a warmup ratio of 1e-3 for the learning rate. The data augmentation is the same as the default strategy in mmdetection~\citep{mmdetection}.
\subsection{Semantic segmentation}
On the downstream semantic segmentation task, we adopt UperNet~\citep{xiao2018unified} to verify the effectiveness of consolidator on Pascal VOC 12 dataset~\citep{Everingham10voc}. We use a Swin-B pre-trained on IN-21k as the backbone model. The consolidator setting is the same as the setting of Swin-B in \cref{tab:hyperparam_method}. For hyperparameters, we adopt AdamW as the optimizer with a learning rate of 6e-5 and weight decay of 1e-2, and train for 20000 iterations in total. The learning rate is decayed by a polynomial scheduler with a power of 1.0. The first 200 iterations are trained with a warmup ratio of 1e-6 for the learning rate. The data augmentation is the same as the default strategy in mmsegmentation~\citep{mmseg2020}.
\begin{table}
\fontsize{7.5}{12}
\selectfont
\setlength{\tabcolsep}{0.6mm}
\begin{center}
\setlength{\belowcaptionskip}{-0.3cm}
\caption{The method-related hyperparameters.}
\label{tab:hyperparam_method}
\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
& model & Setting & LoRA & Adapter & Consolidator\\
\hline
\multirow{2}*{Supervised, MoCo v3} & \multirow{2}*{ViT-B} & VTAB-1k & rank=8 & hidden=16 & ($g^{(1)}$=384)\\
\cline{3-6}
& & Full data & rank=10 & hidden=9 & ($g^{(1)}$=384)\\
\hline
Supervised & ViT-S & Full data & rank=58 & hidden=56 & ($g^{(1)}$=48, $g^{(2)}$=64, $g^{(3)}$=96)\\
\hline
Supervised & ViT-L & Full data & rank=9 & hidden=8 & ($g^{(1)}$=512)\\
\hline
Supervised & Swin-B & Full data & rank=12 & hidden=10 & ($g^{(1)}$=256)\\
\hline
Supervised & AS-MLP-B & Full data & —— & —— & ($g^{(1)}$=128)\\
\hline
Supervised & ConvNeXt-B & Full data & —— & —— & ($g^{(1)}$=128)\\
\hline
MAE & ViT-B & Full data & rank=40 & hidden=38 & ($g^{(1)}$=96, $g^{(2)}$=192)\\
\hline
\end{tabular}
\end{center}
\end{table}
%Consolidator Setting & ($g_{1}$=48, $g_{2}$=64, $g_{3}$=96) & ($g_{1}$=512) & ($g_{1}$=256) & ($g_{1}$=96, $g_{2}$=192)\\
\iffalse
\begin{table}
\fontsize{7.5}{12}
\selectfont
\setlength{\tabcolsep}{0.6mm}
\begin{center}
\setlength{\belowcaptionskip}{-0.3cm}
\caption{The training hyperparameters.}
\label{tab:hyperparam_training}
\begin{tabular}{|l|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}|}
\hline
& \multicolumn{2}{c}{VTAB-1k} \vline & \multicolumn{5}{c}{Full data} \vline\\
\hline
& Supervised & MoCo v3 & \multicolumn{3}{c|}{Supervised} & MAE & MoCo v3\\
\hline
model & ViT-B & ViT-B & ViT-S & ViT-L & Swin-B & ViT-B & ViT-B\\
\hline
pretraining dataset & IN-21k & IN-1k & IN-21k & IN-21k & IN-21k & IN-1k & IN-1k\\
\hline
optimizer & adamw & adamw & sgd & sgd & sgd & sgd & sgd\\
\hline
warmup epochs & 10 & 10 & 5 & 5 & 5 & 5 & 5\\
\hline
epochs & 100 & 100 & 40 & 40 & 40 & 40 & 40\\
\hline
batch size & 64 & 64 & 128 & 32 & 64 & 64 & 64\\
\hline
lr & 1e-2, 1e-3 & 1e-2, 1e-3 & 1e-3 & 1e-3 & 1e-3 & 1e-2 & 1e-2\\
\hline
wd & 1e-4 & 1e-4 & 1e-4 & 1e-4 & 1e-4 & 1e-4 & 1e-4\\
\hline
\end{tabular}
\end{center}
\end{table}
\fi
\begin{table}
\fontsize{7.5}{12}
\selectfont
\setlength{\tabcolsep}{0.6mm}
\begin{center}
\caption{The training hyperparameters on full data setting.}
\label{tab:hyperparam_training}
\begin{tabular}{|l|p{1.4cm}<{\centering}|p{1.4cm}<{\centering}|p{1.4cm}<{\centering}|p{1.4cm}<{\centering}|p{1.4cm}<{\centering}|p{1.4cm}<{\centering}|p{1.4cm}<{\centering}|}
\hline
& \multicolumn{5}{c|}{Supervised} & MAE & MoCo v3\\
\hline
model & ViT-S & ViT-L & Swin-B & AS-MLP-B & ConvNeXt-B & ViT-B & ViT-B\\
\hline
pretraining dataset & IN-21k & IN-21k & IN-21k & IN-1k & IN-21k & IN-1k & IN-1k\\
\hline
optimizer & sgd & sgd & sgd & sgd & sgd & sgd & sgd\\
\hline
warmup epochs & 5 & 5 & 5 & 5 & 5 & 5 & 5\\
\hline
epochs & 40 & 40 & 40 & 40 & 40 & 40 & 40\\
\hline
batch size & 128 & 32 & 64 & 48 & 64 & 64 & 64\\
\hline
lr & 1e-3 & 1e-3 & 1e-3 & 1e-3 & 1e-3 & 1e-2 & 1e-2\\
\hline
wd & 1e-4 & 1e-4 & 1e-4 & 1e-4 & 1e-4 & 1e-4 & 1e-4\\
\hline
\end{tabular}
\end{center}
\end{table}
\begin{figure}
\centering
\begin{subfigure}{\linewidth}
\centering
\includegraphics[width=\linewidth]{imgs/Consolidator_speed.pdf}
\end{subfigure}
\caption{Left: comparison of the inference speed (images/second). Right: comparison of the training speed (images/second). We can conclude that consolidator tuning maintains good throughput during training across various storage budgets, and bring no extra cost compared with normal fine-tuning in inference period.}
\label{fig:Speed}
\end{figure}
\section{Training and Inference Cost}
Many the classic parameter-efficient tuning methods, \eg adapter~\citep{houlsby2019parameter} and Adaptformer~\citep{chen2022adaptformer} introduce non-negligible extra cost in inference period and thus slow down the processing speed. In contrast, consolidator tuning shares identical structure with the original model and bring no extra inference cost. We quantitatively show the training cost and inference cost across different parameter scales in \cref{fig:Speed}. Here we do not show the cost of SSF for it can not be adapted to different scales of parameter budget. In addition, VPT and NOAH both search for an optimal structure from a large searching space and thus it is hard to fairly measure their cost as well.
\begin{table}
\fontsize{9}{12}
\selectfont
\begin{center}
\setlength{\belowcaptionskip}{-0.3cm}
\caption{Sensitivity test on consolidator's hyperparameters. Given a particular target storage budget, we may have several choices in selecting the branches and groups. As seen below, such choices make little influence on the final results. The performance of consolidator is relatively stable under a given storage budget.}
\label{tab:sensitivity}
\begin{tabular}{l|cccc}
\toprule
\# params & 0.75\% & 1.01\% & 4.88\% & 6.42\%\\
\midrule
branches$_{more}$ & (256,768) & (256,384,768) & (48,64,96) & (24,48)\\
acc$_{more}$ & 87.02 & 87.04 & 87.18 & 87.28\\
\midrule
branches$_{less}$ & (192) & (128) & (32,64) & (16)\\
acc$_{less}$ & 86.98 & 87.00 & 87.14 & 87.36\\
\midrule
$\Delta$acc & 0.04 & 0.04 & 0.04 & -0.08\\
\bottomrule
\end{tabular}
\end{center}
\end{table}
\section{Sensitivity of groups and branches in consolidator}
Given a particular target storage budget, we may have several choices in selecting the branches and groups. As seen in \cref{tab:sensitivity}, such choices make little influence on the final results. The performance of consolidator is relatively stable under a given storage budget. When the budget increases, the performance of consolidator increases as well. Such monotonical property is helpful for real-world applications, making it easy to tune hyperparameters and rapidly find an optimal candidate under a given storage budget.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{58}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ba et~al.(2016)Ba, Kiros, and Hinton]{ba2016layernorm}
Jimmy~Lei Ba, Jamie~Ryan Kiros, and Geoffrey~E Hinton.
\newblock Layer normalization.
\newblock \emph{arXiv preprint arXiv:1607.06450}, 2016.
\bibitem[Bahng et~al.(2022)Bahng, Jahanian, Sankaranarayanan, and
Isola]{bahng2022visual}
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.
\newblock Exploring visual prompts for adapting large-scale models.
\newblock \emph{arXiv preprint arXiv:2203.17274}, 2022.
\bibitem[Chen et~al.(2019)Chen, Wang, Pang, Cao, Xiong, Li, Sun, Feng, Liu, Xu,
Zhang, Cheng, Zhu, Cheng, Zhao, Li, Lu, Zhu, Wu, Dai, Wang, Shi, Ouyang, Loy,
and Lin]{mmdetection}
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu~Xiong, Xiaoxiao Li,
Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng,
Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu,
Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen~Change Loy, and
Dahua Lin.
\newblock {MMDetection}: Open mmlab detection toolbox and benchmark.
\newblock \emph{arXiv preprint arXiv:1906.07155}, 2019.
\bibitem[Chen et~al.(2021)Chen, Peng, Fu, and Ling]{chen2021AutoFormer}
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling.
\newblock Autoformer: Searching transformers for visual recognition.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV)}, pp.\ 12270--12280, October 2021.
\bibitem[Chen et~al.(2022)Chen, Ge, Tong, Wang, Song, Wang, and
Luo]{chen2022adaptformer}
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and
Ping Luo.
\newblock Adaptformer: Adapting vision transformers for scalable visual
recognition.
\newblock \emph{Advances in Neural Information Processing Systems}, 2022.
\bibitem[Cimpoi et~al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and
Vedaldi]{cimpoi2014dtd}
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea
Vedaldi.
\newblock Describing textures in the wild.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 3606--3613, 2014.
\bibitem[Contributors(2020)]{mmseg2020}
MMSegmentation Contributors.
\newblock {MMSegmentation}: Openmmlab semantic segmentation toolbox and
benchmark.
\newblock \url{https://github.com/open-mmlab/mmsegmentation}, 2020.
\bibitem[Ding et~al.(2022{\natexlab{a}})Ding, Qin, Yang, Wei, Yang, Su, Hu,
Chen, Chan, Chen, et~al.]{ding2022delta}
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su,
Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et~al.
\newblock Delta tuning: A comprehensive study of parameter efficient methods
for pre-trained language models.
\newblock \emph{arXiv preprint arXiv:2203.06904}, 2022{\natexlab{a}}.
\bibitem[Ding et~al.(2019)Ding, Guo, Ding, and Han]{ding2019acnet}
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han.
\newblock Acnet: Strengthening the kernel skeletons for powerful cnn via
asymmetric convolution blocks.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on
computer vision}, pp.\ 1911--1920, 2019.
\bibitem[Ding et~al.(2021{\natexlab{a}})Ding, Hao, Tan, Liu, Han, Guo, and
Ding]{ding2021resrep}
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji~Liu, Jungong Han, Yuchen Guo, and
Guiguang Ding.
\newblock Resrep: Lossless cnn pruning via decoupling remembering and
forgetting.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pp.\ 4510--4520, 2021{\natexlab{a}}.
\bibitem[Ding et~al.(2021{\natexlab{b}})Ding, Zhang, Han, and
Ding]{ding2021diverse}
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding.
\newblock Diverse branch block: Building a convolution as an inception-like
unit.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pp.\ 10886--10895, 2021{\natexlab{b}}.
\bibitem[Ding et~al.(2021{\natexlab{c}})Ding, Zhang, Ma, Han, Ding, and
Sun]{ding2021repvgg}
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian
Sun.
\newblock Repvgg: Making vgg-style convnets great again.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pp.\ 13733--13742, 2021{\natexlab{c}}.
\bibitem[Ding et~al.(2022{\natexlab{b}})Ding, Zhang, Han, and
Ding]{ding2022scaling31}
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding.
\newblock Scaling up your kernels to 31x31: Revisiting large kernel design in
cnns.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pp.\ 11963--11975, 2022{\natexlab{b}}.
\bibitem[Dosovitskiy et~al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn,
Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly,
et~al.]{dosovitskiy2020image}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et~al.
\newblock An image is worth 16x16 words: Transformers for image recognition at
scale.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Everingham et~al.(2010)Everingham, Van~Gool, Williams, Winn, and
Zisserman]{Everingham10voc}
M.~Everingham, L.~Van~Gool, C.~K.~I. Williams, J.~Winn, and A.~Zisserman.
\newblock The pascal visual object classes (voc) challenge.
\newblock \emph{International Journal of Computer Vision}, 88\penalty0
(2):\penalty0 303--338, June 2010.
\bibitem[Fei-Fei et~al.(2004)Fei-Fei, Fergus, and Perona]{fei2004caltech101}
Li~Fei-Fei, Rob Fergus, and Pietro Perona.
\newblock Learning generative visual models from few training examples: An
incremental bayesian approach tested on 101 object categories.
\newblock In \emph{2004 conference on computer vision and pattern recognition
workshop}, pp.\ 178--178. IEEE, 2004.
\bibitem[Guo et~al.(2020)Guo, Alvarez, and Salzmann]{guo2020expandnets}
Shuxuan Guo, Jose~M Alvarez, and Mathieu Salzmann.
\newblock Expandnets: Linear over-parameterization to train compact
convolutional networks.
\newblock \emph{Advances in Neural Information Processing Systems}, 33, 2020.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 770--778, 2016.
\bibitem[He et~al.(2020)He, Fan, Wu, Xie, and Girshick]{he2020momentum}
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
\newblock Momentum contrast for unsupervised visual representation learning.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition}, pp.\ 9729--9738, 2020.
\bibitem[Helber et~al.(2019)Helber, Bischke, Dengel, and
Borth]{helber2019eurosat}
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.
\newblock Eurosat: A novel dataset and deep learning benchmark for land use and
land cover classification.
\newblock \emph{IEEE Journal of Selected Topics in Applied Earth Observations
and Remote Sensing}, 12\penalty0 (7):\penalty0 2217--2226, 2019.
\bibitem[Hendrycks \& Gimpel(2016)Hendrycks and Gimpel]{hendrycks2016gelu}
Dan Hendrycks and Kevin Gimpel.
\newblock Gaussian error linear units (gelus).
\newblock \emph{arXiv preprint arXiv:1606.08415}, 2016.
\bibitem[Houlsby et~al.(2019)Houlsby, Giurgiu, Jastrzebski, Morrone,
De~Laroussilhe, Gesmundo, Attariyan, and Gelly]{houlsby2019parameter}
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin
De~Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
\newblock Parameter-efficient transfer learning for nlp.
\newblock In \emph{International Conference on Machine Learning}, pp.\
2790--2799. PMLR, 2019.
\bibitem[Howard et~al.(2017)Howard, Zhu, Chen, Kalenichenko, Wang, Weyand,
Andreetto, and Adam]{howard2017mobilenets}
Andrew~G Howard, Menglong Zhu, Bo~Chen, Dmitry Kalenichenko, Weijun Wang,
Tobias Weyand, Marco Andreetto, and Hartwig Adam.
\newblock Mobilenets: Efficient convolutional neural networks for mobile vision
applications.
\newblock \emph{arXiv preprint arXiv:1704.04861}, 2017.
\bibitem[Hu et~al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and
Chen]{hu2021lora}
Edward~J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu~Wang, and Weizhu Chen.
\newblock Lora: Low-rank adaptation of large language models.
\newblock \emph{arXiv preprint arXiv:2106.09685}, 2021.
\bibitem[Huang et~al.(2016)Huang, Sun, Liu, Sedra, and
Weinberger]{huang2016droppath}
Gao Huang, Yu~Sun, Zhuang Liu, Daniel Sedra, and Kilian~Q Weinberger.
\newblock Deep networks with stochastic depth.
\newblock In \emph{European conference on computer vision}, pp.\ 646--661.
Springer, 2016.
\bibitem[Jia et~al.(2022)Jia, Tang, Chen, Cardie, Belongie, Hariharan, and
Lim]{jia2022vpt}
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath
Hariharan, and Ser-Nam Lim.
\newblock Visual prompt tuning.
\newblock \emph{arXiv preprint arXiv:2203.12119}, 2022.
\bibitem[Jiang et~al.(2022)Jiang, Shu, Wang, and
Long]{jiang2022transferability}
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long.
\newblock Transferability in deep learning: A survey.
\newblock \emph{arXiv preprint arXiv:2201.05867}, 2022.
\bibitem[Jie \& Deng(2022)Jie and Deng]{jie2022convolutional}
Shibo Jie and Zhi-Hong Deng.
\newblock Convolutional bypasses are better vision transformer adapters.
\newblock \emph{arXiv preprint arXiv:2207.07039}, 2022.
\bibitem[Junguang~Jiang \& Long(2020)Junguang~Jiang and Long]{jiang2020dalib}
Bo~Fu Junguang~Jiang, Baixu~Chen and Mingsheng Long.
\newblock Transfer-learning-library.
\newblock \url{https://github.com/thuml/Transfer-Learning-Library}, 2020.
\bibitem[Karimi~Mahabadi et~al.(2021)Karimi~Mahabadi, Henderson, and
Ruder]{karimi2021compacter}
Rabeeh Karimi~Mahabadi, James Henderson, and Sebastian Ruder.
\newblock Compacter: Efficient low-rank hypercomplex adapter layers.
\newblock \emph{Advances in Neural Information Processing Systems}, 34, 2021.
\bibitem[Khosla et~al.(2011)Khosla, Jayadevaprakash, Yao, and
Li]{khosla2011stanforddogs}
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.
\newblock Novel dataset for fine-grained image categorization: Stanford dogs.
\newblock In \emph{First Workshop on Fine-Grained Visual Categorization, IEEE
Conference on Computer Vision and Pattern Recognition}. Citeseer, June 2011.
\bibitem[Krizhevsky et~al.(2009)Krizhevsky, Hinton,
et~al.]{krizhevsky2009cifar}
Alex Krizhevsky, Geoffrey Hinton, et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock 2009.
\bibitem[Krizhevsky et~al.(2012)Krizhevsky, Sutskever, and
Hinton]{krizhevsky2012imagenet}
Alex Krizhevsky, Ilya Sutskever, and Geoffrey~E Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock \emph{Advances in neural information processing systems},
25:\penalty0 1097--1105, 2012.
\bibitem[Lester et~al.(2021)Lester, Al-Rfou, and Constant]{lester2021power}
Brian Lester, Rami Al-Rfou, and Noah Constant.
\newblock The power of scale for parameter-efficient prompt tuning.
\newblock \emph{arXiv preprint arXiv:2104.08691}, 2021.
\bibitem[Li \& Liang(2021)Li and Liang]{li2021prefix}
Xiang~Lisa Li and Percy Liang.
\newblock Prefix-tuning: Optimizing continuous prompts for generation.
\newblock \emph{arXiv preprint arXiv:2101.00190}, 2021.
\bibitem[Lian et~al.(2022{\natexlab{a}})Lian, Yu, Sun, and Gao]{lian2022mlp}
Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao.
\newblock As-mlp: An axial shifted mlp architecture for vision.
\newblock In \emph{International Conference on Learning Representations},
2022{\natexlab{a}}.
\bibitem[Lian et~al.(2022{\natexlab{b}})Lian, Zhou, Feng, and
Wang]{lian2022ssf}
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.
\newblock Scaling \& shifting your features: A new baseline for efficient model
tuning.
\newblock \emph{Advances in Neural Information Processing Systems},
2022{\natexlab{b}}.
\bibitem[Lin et~al.(2017)Lin, Doll{\'a}r, Girshick, He, Hariharan, and
Belongie]{lin2017fpn}
Tsung-Yi Lin, Piotr Doll{\'a}r, Ross Girshick, Kaiming He, Bharath Hariharan,
and Serge Belongie.
\newblock Feature pyramid networks for object detection.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 2117--2125, 2017.
\bibitem[Liu et~al.(2021{\natexlab{a}})Liu, Ji, Fu, Du, Yang, and
Tang]{liu2021p}
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang.
\newblock P-tuning v2: Prompt tuning can be comparable to fine-tuning
universally across scales and tasks.
\newblock \emph{arXiv preprint arXiv:2110.07602}, 2021{\natexlab{a}}.
\bibitem[Liu et~al.(2021{\natexlab{b}})Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and
Guo]{liu2021swin}
Ze~Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
\newblock Swin transformer: Hierarchical vision transformer using shifted
windows.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pp.\ 10012--10022, 2021{\natexlab{b}}.
\bibitem[Liu et~al.(2022)Liu, Mao, Wu, Feichtenhofer, Darrell, and
Xie]{liu2022convnet}
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell,
and Saining Xie.
\newblock A convnet for the 2020s.
\newblock \emph{arXiv preprint arXiv:2201.03545}, 2022.
\bibitem[Ma et~al.(2018)Ma, Zhang, Zheng, and Sun]{ma2018shufflenet}
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.
\newblock Shufflenet v2: Practical guidelines for efficient cnn architecture
design.
\newblock In \emph{Proceedings of the European conference on computer vision
(ECCV)}, pp.\ 116--131, 2018.
\bibitem[Nilsback \& Zisserman(2008)Nilsback and
Zisserman]{nilsback2008flowers}
Maria-Elena Nilsback and Andrew Zisserman.
\newblock Automated flower classification over a large number of classes.
\newblock In \emph{2008 Sixth Indian Conference on Computer Vision, Graphics \&
Image Processing}, pp.\ 722--729. IEEE, 2008.
\bibitem[Parkhi et~al.(2012)Parkhi, Vedaldi, Zisserman, and
Jawahar]{parkhi2012pets}
Omkar~M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV~Jawahar.
\newblock Cats and dogs.
\newblock In \emph{2012 IEEE conference on computer vision and pattern
recognition}, pp.\ 3498--3505. IEEE, 2012.
\bibitem[Pfeiffer et~al.(2021)Pfeiffer, Kamath, R{\"u}ckl{\'e}, Cho, and
Gurevych]{pfeiffer2021adapterfusion}
Jonas Pfeiffer, Aishwarya Kamath, Andreas R{\"u}ckl{\'e}, Kyunghyun Cho, and
Iryna Gurevych.
\newblock Adapterfusion: Non-destructive task composition for transfer
learning.
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume}, pp.\
487--503, 2021.
\bibitem[Ren et~al.(2015)Ren, He, Girshick, and Sun]{ren2015faster}
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
\newblock Faster r-cnn: Towards real-time object detection with region proposal
networks.
\newblock \emph{Advances in neural information processing systems}, 28, 2015.
\bibitem[Simonyan \& Zisserman(2014)Simonyan and Zisserman]{simonyan2014very}
Karen Simonyan and Andrew Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock \emph{arXiv preprint arXiv:1409.1556}, 2014.
\bibitem[Srivastava et~al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever, and
Salakhutdinov]{srivastava2014dropout}
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Salakhutdinov.
\newblock Dropout: a simple way to prevent neural networks from overfitting.
\newblock \emph{The journal of machine learning research}, 15\penalty0
(1):\penalty0 1929--1958, 2014.
\bibitem[Touvron et~al.(2021)Touvron, Cord, Douze, Massa, Sablayrolles, and
J{\'e}gou]{touvron2021training}
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, and Herv{\'e} J{\'e}gou.
\newblock Training data-efficient image transformers \& distillation through
attention.
\newblock In \emph{International Conference on Machine Learning}, pp.\
10347--10357. PMLR, 2021.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
Kaiser, and Polosukhin]{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Wah et~al.(2011)Wah, Branson, Welinder, Perona, and
Belongie]{wah2011cub}
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge
Belongie.
\newblock The caltech-ucsd birds-200-2011 dataset.
\newblock 2011.
\bibitem[Xiao et~al.(2010)Xiao, Hays, Ehinger, Oliva, and
Torralba]{xiao2010sun}
Jianxiong Xiao, James Hays, Krista~A Ehinger, Aude Oliva, and Antonio Torralba.
\newblock Sun database: Large-scale scene recognition from abbey to zoo.
\newblock In \emph{2010 IEEE computer society conference on computer vision and
pattern recognition}, pp.\ 3485--3492. IEEE, 2010.
\bibitem[Xiao et~al.(2018)Xiao, Liu, Zhou, Jiang, and Sun]{xiao2018unified}
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.
\newblock Unified perceptual parsing for scene understanding.
\newblock In \emph{Proceedings of the European conference on computer vision
(ECCV)}, pp.\ 418--434, 2018.
\bibitem[Zaken et~al.(2021)Zaken, Ravfogel, and Goldberg]{zaken2021bitfit}
Elad~Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.
\newblock Bitfit: Simple parameter-efficient fine-tuning for transformer-based
masked language-models.
\newblock \emph{arXiv preprint arXiv:2106.10199}, 2021.
\bibitem[Zhai et~al.(2019)Zhai, Puigcerver, Kolesnikov, Ruyssen, Riquelme,
Lucic, Djolonga, Pinto, Neumann, Dosovitskiy, et~al.]{zhai2019vtab}
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos
Riquelme, Mario Lucic, Josip Djolonga, Andre~Susano Pinto, Maxim Neumann,
Alexey Dosovitskiy, et~al.
\newblock A large-scale study of representation learning with the visual task
adaptation benchmark.
\newblock \emph{arXiv preprint arXiv:1910.04867}, 2019.
\bibitem[Zhang et~al.(2018)Zhang, Zhou, Lin, and Sun]{zhang2018shufflenet}
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.
\newblock Shufflenet: An extremely efficient convolutional neural network for
mobile devices.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 6848--6856, 2018.
\bibitem[Zhang et~al.(2022)Zhang, Zhou, and Liu]{zhang2022noah}
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.
\newblock Neural prompt search.
\newblock \emph{arXiv preprint arXiv:2206.04673}, 2022.
\bibitem[Zhou et~al.(2021)Zhou, Yang, Loy, and Liu]{zhou2021learning}
Kaiyang Zhou, Jingkang Yang, Chen~Change Loy, and Ziwei Liu.
\newblock Learning to prompt for vision-language models.
\newblock \emph{arXiv preprint arXiv:2109.01134}, 2021.
\end{thebibliography}