Learning without Forgetting for Vision-Language Models
Extracted source text · arXiv 2305.19270
\documentclass{article}
\usepackage[nonatbib,preprint]{neurips_2023}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{caption} % DO NOT CHANGE THIS AND DO NOT ADD ANY OPTIONS TO IT
\usepackage{hyperref}
\usepackage{wrapfig}
\usepackage[american]{babel}
\usepackage{subfigure}
\usepackage{csquotes}
%\usepackage{amssymb}
\usepackage{bm}
%\usepackage{paralist,algorithmic,algorithm}
\usepackage{paralist,algorithm}
\usepackage{subfigure}
\usepackage{multirow}
\newcommand{\cs}[1]{\texttt{\char`\\#1}}
\newcommand{\argmax}{\mathop{\arg\max}}
\newcommand{\argmin}{\mathop{\arg\min}}
\newcommand{\med}{\mathop{\textrm{med}}}
\newcommand{\x}{{\bf x}}
\newcommand{\eg}{{\em e.g.}}
\newcommand{\ie}{{\em i.e.}}
\newcommand{\aaa}{{\bf a}}
\newcommand{\bb}{{\bf b}}
\newcommand{\s}{{\bf s}}
\newcommand{\1}{{\bf 1}}
\newcommand{\p}{{\bf p}}
\newcommand{\bml}{{\bm l}}
\newcommand{\w}{{\bf w}}
\newcommand{\cc}{{\bf c}}
\newcommand{\vv}{{\bf v}}
\newcommand{\ttt}{{\bf t}}
\newcommand{\e}{{\bf e}}
\newcommand{\z}{{\bf z}}
\newcommand{\h}{{\bf h}}
\newcommand{\oo}{{\bf o}}
\newcommand{\m}{{\bf m}}
\newcommand{\y}{{\bf y}}
\newcommand{\ud}{\mathrm{d}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Y}{\mathcal{Y}}
\newcommand{\expect}{\mathbb{E}}
\newcommand{\sign}{\mathtt{sign}}
\newcommand{\mathI}{\mathcal{I}}
\newcommand{\ta}{$\mathcal{T}_{\rm Ref}$}
\newcommand{\defeq}{\stackrel{\text{def}}{=}}
\newcommand{\cev}[1]{\reflectbox{\ensuremath{\vec{\reflectbox{\ensuremath{#1}}}}}}
\newcommand{\uu}{{\bf u}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\Sup}{\mathcal{S}}
\newcommand{\Q}{\mathcal{Q}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\tr}{\mathop{\mathrm{tr}}}
\newcommand{\name}{{\sc Proof }}
\newcommand{\mame}{{\sc Proof}}
%PROjectiOn Fusion
\newcommand{\bfname}[1]{{\bf #1}}
\newtheorem{remark}{Remark}
\usepackage{accents}
\newlength{\dhatheight}
\newcommand{\doublehat}[1]{%
\settoheight{\dhatheight}{\ensuremath{\hat{#1}}}%
\addtolength{\dhatheight}{-0.35ex}%
\hat{\vphantom{\rule{1pt}{\dhatheight}}%
\smash{\hat{#1}}}}
\newcommand{\experimentsection}[1]{\subsection{#1}}
\newcommand{\etal}{{\em et al.}}
\usepackage[noend]{algpseudocode}
\usepackage{xcolor,colortbl}
\definecolor{Gray}{gray}{0.85}
\definecolor{Redo}{rgb}{0.95,0.69,0.51}
\definecolor{LightCyan}{rgb}{0.88,1,1}
\usepackage{afterpage}
\usepackage{pifont}% http://ctan.org/pkg/pifont
\title{Learning without Forgetting for\\ Vision-Language Models}
\author{%
Da-Wei Zhou$^1$, Yuanhan Zhang$^2$, Jingyi Ning$^1$, Han-Jia Ye$^1$, De-Chuan Zhan$^1$, Ziwei Liu$^2$\thanks{Han-Jia Ye and Ziwei Liu are corresponding authors.} \\
$^1$
State Key Laboratory for Novel Software Technology, Nanjing University\\
$^2$ S-Lab, Nanyang Technological University\\
\texttt{\{zhoudw, yehj, zhandc\}@lamda.nju.edu.cn, ningjy@smail.nju.edu.cn}, \\
\texttt{\{yuanhan002,ziwei.liu\}@ntu.edu.sg}\\
}
\begin{document}
\maketitle
\input{abstract}
\input{intro.tex}
\input{related.tex}
\input{prelim.tex}
\input{method.tex}
\input{experiments.tex}
\input{conclusion.tex}
\input{supp_exp}
{
\small
\bibliographystyle{ieee_fullname}
\bibliography{paper}
}
\end{document}
\begin{abstract}
Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world, which requires a learning system to adapt to new tasks without forgetting former ones. While traditional CIL methods focus on {\em visual} information to grasp core features, recent advances in Vision-Language Models (VLM) have shown promising capabilities in learning generalizable representations with the aid of {\em textual} information. However, when continually trained with new classes, VLMs often suffer from catastrophic forgetting of former knowledge. Applying VLMs to CIL poses two major challenges: \textbf{1)} how to adapt the model without forgetting; and \textbf{2)} how to make full use of the multi-modal information. To this end, we propose PROjectiOn Fusion (\textbf{\textsc{Proof}}) that enables VLMs to learn without forgetting. To handle the first challenge, we propose training task-specific projections based on the frozen image/text encoders. When facing new tasks, new projections are expanded and former projections are fixed, alleviating the forgetting of old concepts. For the second challenge, we propose the fusion module to better utilize the cross-modality information. By jointly adjusting visual and textual features, the model can capture semantic information with a stronger representation ability. Extensive experiments on nine benchmark datasets validate \mame\ achieves state-of-the-art performance.
\end{abstract}
\section{Introduction}
In our ever-changing world, training data often comes in a stream format with new classes, requiring a learning system to absorb them continually~\cite{gomes2017survey,geng2020recent}. To address the challenge of learning emerging new classes, Class-Incremental Learning (CIL) has been proposed~\cite{rebuffi2017icarl}. However, in CIL, the absence of former classes triggers catastrophic forgetting~\cite{french1999catastrophic}, where learning new concepts overwrites the knowledge of old ones and results in decline in performance~\cite{li2016learning}. Numerous efforts have been made~\cite{liu2021rmm,douillard2022dytox,zhao2021mgsvf,sunexploring,wang2022sparcl,zhangsimple} to combat catastrophic forgetting in the machine learning field.
With the rapid development of pre-training techniques~\cite{han2021pre}, recent years have witnessed the transition of CIL research from training from scratch~\cite{wu2019large,hou2018lifelong,zhao2020maintaining} to {\em utilizing pre-trained models} (PTM)~\cite{wang2022dualprompt,wang2022learning,seale2022coda}. With the help of PTM, \eg, Vision Transformers~\cite{dosovitskiy2020image}, incremental models are born with strong transferability to grasp the {\em visual} features.
Facing the domain gap introduced by the incremental classes, they only need to learn a limited number of additional parameters~\cite{jia2022visual,chenadaptformer,lianscaling} as the {\em patches} to bridge the gap, which significantly simplifies the challenge of incremental learning.
While pre-trained ViT-based CIL methods focus on learning the {\em visual} features to recognize new concepts, recent advances in Vision-Language Models (VLM) have demonstrated the potential of {\em textual} information in building generalized feature representations. A typical work, \ie, contrastive language-image pre-training~\cite{radford2021learning} (CLIP), maps the visual and textual information in the shared embedding space, enabling robust learning and recognition of concepts from diverse sources.
This integration of visual and textual modalities presents a promising avenue for developing continual learning models that can effectively adapt to real-world scenarios.
Extending VLMs to CIL faces two significant challenges.
First, sequentially tuning the VLM overwrites the innate generalizability and former concepts, leading to forgetting and poor performance on future tasks. Second, relying solely on textual information for classification neglects the valuable cross-modal features present in the multi-modal inputs. To fully utilize this information, it is necessary to explore methods for cross-modal fusion beyond textual features.
Correspondingly, we aim to turn a VLM into a continual learner that is both {\em retentive} and {\em comprehensive}.
Retentive refers to the model's ability to maintain its pre-trained capabilities, thereby preserving generalizability and enabling it to perform well on future tasks without forgetting.
Comprehensive refers to the model's capacity to integrate and adjust information from multiple modalities.
By leveraging these characteristics, we can mitigate catastrophic forgetting and use cross-modal features to build more robust classifiers as data evolves.
In this paper, we propose PROjectiOn Fusion (\textbf{\textsc{Proof}}) to address catastrophic forgetting in VLM. To make the model retentive, we freeze the pre-trained image/text backbones and append liner projections on top of them.
The task-specific information is encoded in the corresponding projection layer by mapping the projected features.
When facing new tasks, new projections are extended while old ones are frozen, preserving former knowledge.
Besides, we aim to fuse the information from different modalities via cross-model fusion, which allows for the query embedding to be adjusted with context information. Consequently, \name efficiently incorporates new classes and meanwhile resists forgetting old ones, achieving state-of-the-art performance on nine benchmark datasets. We also investigate the zero-shot performance of VLM with new evaluation protocols and metrics, and find that \name maintains its zero-shot performance with a simple modification.
\section{Related Work}
\noindent\bfname{Vision-Language Model (VLM) Tuning:} Recent years have witnessed the prosperity of research in VLMs, \eg, CLIP~\cite{radford2021learning}, ALIGN~\cite{jia2021scaling}, CoCa~\cite{yu2022coca}, Florence~\cite{yuan2021florence}, BLIP~\cite{li2023blip}, CLIPPO~\cite{tschannen2022image}, and Flamingo~\cite{alayrac2022flamingo}.
These models are pre-trained on vast amounts of images and texts, achieving a unified embedding space across modalities.
With great generalizability, they can be applied for downstream tasks in a zero-shot manner. However, a domain gap still exists between the pre-trained and downstream datasets, requiring further tuning for better performance.
CoOp and CoCoOp~\cite{zhou2022learning,zhou2022conditional} apply prompt learning~\cite{li2021prefix} into VLM tuning with learnable prompt tokens. Subsequent works explore VLM tuning via adapter tuning~\cite{gao2021clip}, prompt distribution learning~\cite{lu2022prompt}, task residual learning~\cite{yu2022task}, similarity learning~\cite{zhang2022tip}, descriptor learning~\cite{mao2022doubly}, and optimal transport mapping~\cite{chen2023plot}. However, they only focus on adapting VLM to downstream tasks while overlooking the {\em forgetting} of former ones.\\
\noindent\bfname{Class-Incremental Learning (CIL):} aims to learn from evolutive data and absorb new knowledge without forgetting~\cite{zhou2023class}. Replay-based methods~\cite{luo2023class,aljundi2019gradient,chaudhry2018riemannian,liu2020mnemonics,chaudhry2018efficient} save and replay former instances to recover old knowledge when learning new ones. Knowledge distillation-based methods~\cite{rebuffi2017icarl,li2016learning,douillard2020podnet} build the mapping between models as regularization.
Parameter regularization-based methods~\cite{kirkpatrick2017overcoming,aljundi2018memory,zenke2017continual,aljundi2019task} weigh the importance of different parameters as regularization. Model rectification-based methods~\cite{shi2022mimicking,zhao2020maintaining,wu2019large,yu2020semantic} rectify the inductive bias for unbiased predictions. Dynamic networks~\cite{yan2021dynamically,wang2023beef,zhou2022model,wang2022foster} show strong performance by expanding the network structure as data evolves.\\
\noindent\bfname{CIL with VLM:} Aforementioned CIL methods aim to train an incremental model from scratch, while it would be easier to start with a pre-trained model~\cite{lee2023pre}. The integration of pre-trained Vision Transformer~\cite{dosovitskiy2020image} into CIL has attracted the attention of the community, and most methods~\cite{wang2022dualprompt,wang2022learning,seale2022coda} employ parameter-efficient tuning techniques to learn without forgetting. S-Prompt~\cite{wang2022s} explores CLIP in {\em domain}-incremental learning, but the application of VLM in CIL remains relatively unexplored. WiSE-FT~\cite{wortsman2022robust} utilizes weight ensemble for robust finetuning, while it cannot be extended to multiple tasks.
This paper aims to address this research gap by presenting a comprehensive solution for tuning vision-language models without suffering from forgetting.
\section{From Old Classes to New Classes}
In this section, we introduce the background information about class-incremental learning and vision language models. We also discuss the na\"ive solutions for tuning VLM in CIL.
\subsection{Class-Incremental Learning}
Given a data stream with emerging new classes, class-incremental learning aims to continually incorporate the knowledge and build a unified classifier~\cite{zhou2023class}.
We denote the sequence of $B$ training sets without overlapping classes as $\left\{\D^{1}, \D^{2}, \cdots, \D^{B}\right\}$, where $\D^{b}=\left\{\left(\x_{i}, y_{i}\right)\right\}_{i=1}^{n_b}$ is the $b$-th training set with $n_b$ instances.
A training instance $\x_i \in \R^D$ belongs to class $y_i \in Y_b$. $Y_b$ is the label space of task $b$, and $Y_b \cap Y_{b^\prime} = \varnothing$ for $b\neq b^\prime$. Following the typical CIL setting~\cite{rebuffi2017icarl,hou2019learning,wu2019large}, a fixed number of {\em exemplars} from the former classes are selected as the exemplar set $\mathcal{E}$.
During the $b$-th incremental stage, we can only access data from $\D^b$ and $\mathcal{E}$ for model training.
The target is to build a unified classifier for all seen classes $\mathcal{Y}_b=Y_1 \cup \cdots Y_b$ continually. In other words, we hope to find a model $f(\x): X\rightarrow\mathcal{Y}_b$ that minimizes the expected risk:
\begin{equation} \label{eq:totalrisk}
f^*=\underset{f \in \mathcal{H}}{\operatorname{argmin}} \; \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}_{t}^1\cup\cdots\mathcal{D}_{t}^b} \mathbb{I}\left(y \neq f(\mathbf{x})\right) \,,
\end{equation}
where $\mathcal{H}$ denotes the hypothesis space and $\mathbb{I}(\cdot)$ is the indicator function. $\mathcal{D}_{t}^b$ denotes the data distribution of task $b$. Following \cite{wang2022dualprompt,wang2022learning,wang2022s}, we assume that a pre-trained vision-language model is available as the initialization for $f(\x)$, which will be introduced in Section~\ref{sec:vlm}.
\subsection{Vision-Language Model} \label{sec:vlm}
This paper focuses on contrastive
language-image pre-training (CLIP)~\cite{radford2021learning} as the VLM. During pre-training, CLIP jointly learns an image encoder $g_i(\cdot): \R^D\rightarrow\R^d$ and a text encoder $g_t(\cdot): \R^{Dt}\rightarrow\R^d$ in a contrastive manner, where $D/Dt$ are input dimensions of image/text, and $d$ is the embedding dimension.
CLIP projects a batch of image-text pairs into a shared embedding space.
It maximizes the cosine similarity of paired inputs and minimizes it for unmatched ones. Benefiting from the massive training data, CLIP can synthesize a {\em zero-shot classifier} that generalizes to unseen classes. The output of CLIP is formulated as:
\begin{equation} \label{eq:clip_pred}
p(y_i \mid \x)=\frac{\exp \left(\cos \left(\z, \w_i \right) / \tau\right)}{\sum_{j=1}^{|\mathcal{Y}_b|} \exp \left(\cos \left(\z, \w_j \right) / \tau\right)} \,,
\end{equation}
where $\cos(\cdot,\cdot)$ denotes cosine similarity, $\tau$ is learnable temperature parameter, $\z=g_i(\x)$ is the image embedding. Correspondingly, $\w_i$ is the text embedding of class $y_i$ obtained by feeding templated texts, \eg, ``a photo of a [CLASS]'' into the text encoder.
We denote the templated text of class $i$ as $\mathbf{t}_i$. Eq.~\ref{eq:clip_pred} aims to find the most similar text $\mathbf{t}_i$ that maximizes the cosine similarity to the query image.
\subsection{Overcome Forgetting in Class-Incremental Learning} \label{sec:baseline}
CIL, as a long-standing problem, has garnered significant attention from the research community.
In this section, we introduce two typical solutions for adapting pre-trained models with new classes.
\noindent\textbf{Vision-Based Learning:}
Traditional CIL methods primarily rely on the {\em image encoder} to capture the patterns of new classes.
One such method, L2P~\cite{wang2022learning}, leverages visual prompt tuning~\cite{jia2022visual}
to enable incremental updates of a pre-trained Vision Transformer~\cite{dosovitskiy2020image}.
By keeping the image encoder frozen, L2P trains a learnable prompt pool $\mathbf{Pool}$ and combines it with patch embeddings to obtain instance-specific embeddings. The optimization target can be formulated as:
\begin{equation} \label{eq:l2p}
\mathcal{L}= \ell \left(h
\left( \bar{g_i}\left(\x_{i}, \mathbf{Pool} \right) \right) , {y}_{i}\right)+ \mathcal{L}_{reg} \,,
\end{equation}
where $h(\cdot)$ is the classification head, $\bar{g_i}$ is the frozen image encoder, $\mathcal{L}_{reg}$ is the regularization loss for prompt selection. By freezing the encoder, Eq.~\ref{eq:l2p} grasps the new pattern with humble forgetting.
\noindent\textbf{CLIP Tuning:}
The issue of tuning VLM without forgetting in CIL remains unaddressed, as previous works have solely focused on transferring CLIP to downstream tasks without considering the performance of former tasks.
For instance, CoOp~\cite{zhou2022learning} converts text inputs into a learnable prompt, \ie, $\mathbf{t}_i=$ [V]$_1$[V]$_2\cdots$[V]$_M$[CLASS]$_i$.
The posterior probability in Eq.~\ref{eq:clip_pred} is transformed into:
\begin{equation} \label{eq:coop}
p(y_i \mid \x)=\frac{\exp \left(\cos \left(\z, g_t(\ttt_i) \right) / \tau\right)}{\sum_{j=1}^{|\mathcal{Y}_b|} \exp \left(\cos \left(\z, g_t(\ttt_j) \right) / \tau\right)} \,.
\end{equation}
With the help of the learned prompt, Eq.~\ref{eq:coop} enables the model to be transferred to the downstream task. However, since the prompt template is shared for all tasks, sequentially tuning CoOp will suffer catastrophic forgetting of former concepts.
\noindent\textbf{Discussions:}
Current methods focus on different aspects of CIL. Vision-based methods (\eg, Eq.~\ref{eq:l2p}) address the issue of forgetting but neglect the valuable semantic information conveyed in texts.
Conversely, CLIP's pre-trained text encoder captures class-wise relationships that can enhance model learning.
Meanwhile, transfer learning methods (\eg, Eq.~\ref{eq:coop}) effectively leverage the cross-modal information, while sequentially tuning them suffers the catastrophic forgetting of former concepts.
Is it possible to combine the cross-modal information and meanwhile resist catastrophic forgetting?
\section{{\scshape{Proof}}: Projection Fusion for VLM}
Observing the limitations of typical vision-based methods in utilizing textual information and forgetting in CLIP tuning, we aim to leverage cross-modality knowledge in CLIP while effectively mitigating forgetting.
To this end, we must make the model {\em retentive} and {\em comprehensive}.
Retentive represents the ability to adapt to downstream tasks without forgetting, and we propose projections to map the pre-trained features in the projected feature space.
Our unique training strategy ensures the preservation of former knowledge by freezing old projections and expanding new ones for new tasks.
The comprehensive aspect involves co-adapting and utilizing cross-modal information to enhance unified predictions.
The query instance's embedding is influenced by both visual and textual information, allowing for instance-specific adaptation and enabling comprehensive predictions.
In the following sections, we introduce the learning paradigm and the co-adaptation process. Lastly, we provide detailed guidelines for training and inference.
\subsection{Expandable Feature Projection}
CLIP is known for its strong zero-shot performance~\cite{radford2021learning}, \ie, Eq.~\ref{eq:clip_pred} obtains competitive results even without explicit training on the specific tasks. However, given the domain gap between pre-trained and downstream tasks, an {\em adaptation} process is still necessary to capture the characteristics of the latter. Specifically, we introduce a {\em linear layer} (denoted as ``{\bf projection}'') which is appended after the frozen image and text embeddings to facilitate the matching of pair-wise projected features. Denoting the projection of image/text as $P_i(\cdot):\R^d\rightarrow\R^d$ and $ P_t(\cdot):\R^d\rightarrow\R^d$, Eq.~\ref{eq:clip_pred} is transformed into:
\begin{equation} \label{eq:proj_pred}
p(y_i \mid \x)=
\underbrace{
\frac{\exp \left(\cos \left( P_i \left(\z\right), P_t \left(\w_i\right) \right) / \tau\right)}{\sum_{j=1}^{|\mathcal{Y}_b|} \exp \left(\cos \left( P_i \left(\z\right), P_t \left(\w_j\right) \right) / \tau\right)}
}_\text{Projected Matching}
\,.
\end{equation}
We denote the classification based on Eq.~\ref{eq:proj_pred} as $f_\textbf{PM}(\x)$.
By freezing the image and text encoders,
it aligns the downstream features in the projected space, allowing the model to encode the relevant downstream information into projection layers.
Since the pre-trained model outputs generalizable features, the projection layer learns to {\em recombine} features in a data-driven manner. For instance, in a task involving `birds,' the projection would assign a higher weight to features like `beaks' and `wings.'
This adaptation enables the projected features to better discern and recognize downstream tasks.
\noindent\textbf{Expandable Projections:} However, sequentially training a {\em single} projection layer still leads to forgetting of former tasks, resulting in confusion when combining old and new concepts. To this end, we expand {\em task-specific} projections for each new task. Specifically, we append a newly initialized projection layer $P_i^b, P_t^b$ when a new task $\mathcal{D}^b$ arrives. This results in a set of projections: $\{P_i^1, P_i^2, \cdots P_i^b,\}$, $\{P_t^1, P_t^2, \cdots P_t^b,\}$, and we adopt the {\em aggregation} as the output, \ie,
\begin{equation} \label{eq:proj_sum} \textstyle
P_i(\z)=\sum_{m=1}^b P_i^m \left(\z\right), \quad P_t(\w)=\sum_{n=1}^b P_t^n \left(\w\right) \,.
\end{equation}
In Eq.~\ref{eq:proj_sum}, projected features from different stages are mapped and aggregated to capture the different emphases of former and latter tasks.
For example, former tasks might emphasize `beak' features for bird recognition, while later tasks may focus on `beards' features to differentiate cats.
The aggregation of different projections produces a comprehensive representation of the query instance.
By substituting Eq.~\ref{eq:proj_sum} into Eq.~\ref{eq:proj_pred}, the model aligns the unified features in the joint space.
\noindent\textbf{How to resist forgetting of former projections?} To overcome forgetting old concepts, we freeze the projections of former tasks when learning new ones, \ie, $\{\bar{P_i^1}, \bar{P_i^2}, \cdots P_i^b,\}$ (same for $P_t$). It allows the newly initialized projection to learn the {\em residual} information of new tasks, incorporating new concepts while preserving the knowledge of former ones.
During the learning process of task $b$, we optimize the cross-entropy loss to encode the task-specific information into the current projections.
\noindent\textbf{Effect of projections}: The illustration of projections are shown in Figure~\ref{figure:teaser} (left). \name learns projections based on the pre-trained encoders, which fits new patterns and maintains the generalizability of pre-trained model.
The parameter number of each projection layer is $d\times d$, which is neglectable for the pre-trained model.
Furthermore, the model learns new projections for new tasks, and task-specific projections fit new concepts easily. Since we only optimize the current projections and freeze old ones, the former knowledge is preserved, and forgetting is alleviated.
\begin{figure}[t]
\vspace{-8mm}
\begin{center}
{\includegraphics[width=.98\columnwidth]{./pics/teaser}}
\end{center}
\vspace{-5mm}
\caption{\small Illustration of \mame.
The model learns expandable projections and aggregates them to get the aggregated features. The query instance, prototype features, textual features, and context prompts are fed into the cross-modal fusion.
The fusion process utilizes self-attention to co-adapt the input set, which outputs the adapted features.
The adapted query embedding is separately matched among the visual prototypes and textual features to get the final prediction. {\color{Redo}Red} parts are trainable while {\color{gray}gray} ones are frozen.
}
\vspace{-7mm}
\label{figure:teaser}
\end{figure}
\subsection{Contextualizing Projections with Projection Fusion}
In Eq.~\ref{eq:proj_pred}, the projected visual and textual features are directly matched in the joint space.
However, it would be beneficial to further {\em refine} these features to capture the {\em contextual relationship} between images and texts.
For instance, when the query instance is a `panda,' it is desirable to adjust the visual and textual features in a {\em coherent} manner to highlight discriminative attributes such as black eyes and ears.
Similarly, when the query instance is a `cat,' features like beards and tails should be emphasized.
This adjustment process involves jointly adapting the query embedding and the context (\eg, textual information) to obtain a {\em contextualized} embedding. Correspondingly, we propose a {\em set-to-set} function that contextualizes and fuses the query embeddings and contextual information.
Specifically, we denote the adaptation function as $\mathcal{T}(\cdot)$. It receives the query instance and context information as bags, \ie, $[P_i(\z), \textbf{Context}]$, and outputs the set of adjusted embeddings while being permutation-invariant: $\mathcal{T}([P_i(\z), \textbf{Context}])=[\tilde{P_i(\z)}, \tilde{\textbf{Context}}]$.
$\mathcal{T}(\cdot)$ encodes the set information and performs adaptation on each component.
In the following, we describe the construction of the context information $\textbf{Context}$ and provide details on the implementation of the set-to-set function.
\noindent\textbf{How to define the context?} In Eq.~\ref{eq:proj_pred}, the mapping is established between the query instance and the textual information (\ie, classifiers).
The classifiers represent the typical textual description of the corresponding class, \ie, the common feature.
Hence, a na\"ive idea is to utilize textual features as the context, \ie, $\textbf{Context}=\mathbf{W}$, $\mathbf{W}=[P_t(\w_1), P_t(\w_2), \cdots, P_t(\w_{|\mathcal{Y}_b|})]\in\R^{|\mathcal{Y}_b|\times d}$ is the concatenation of all textual classifiers. However, recent works find an inherent domain gap~\cite{liang2022mind} between the visual and textual embeddings in VLM.
The gap leads to visual and textual embeddings residing in two separate clusters in the embedding space, which hinders effective pair-wise mapping.
Correspondingly, we leverage visual prototype features~\cite{snell2017prototypical} as a useful tool for capturing the common characteristics of each class.
Define the {\em visual prototype} of class $k$ as:
$ \p_k=\frac{1}{N}{\sum_{j=1}^{|\mathcal{{D}}^b|}\mathbb{I}(y_j=k)g_i(\x_j)}
$, where $N={\sum_{j=1}^{|\mathcal{{D}}^b|}\mathbb{I}(y_j=k)}$.
They are calculated via forward pass at the beginning of each incremental stage and stay fixed in subsequent tasks.
Visual prototypes are {\em representative} features of the corresponding class, which can serve as the {\em visual context} to adjust the embeddings.
Hence, we augment the context with projected visual information, \ie, $\textbf{Context}=[\mathbf{P}, \mathbf{W}]$, where $\mathbf{P}=[P_i(\p_1), P_i(\p_2), \cdots, P_i(\p_{|\mathcal{Y}_b|})]\in \R^{|\mathcal{Y}_b|\times d}$ is the concatenation of all visual prototypes. Combining prototypes from multiple modalities help the model adapt and fuse information in a cross-modal manner, which goes beyond simple visual-textual matching.
\noindent\textbf{Implementing $\mathcal{T}$ with Self-Attention:} In our implementation, we use the self-attention (SA) mechanism~\cite{vaswani2017attention,lin2017structured} as the cross-modal fusion function $\mathcal{T}$. Being permutation invariant, SA is good at outputting adapted embeddings even with long dependencies, which naturally suits the characteristics of the adaptation function.
Specifically, SA keeps the triplets (query $\mathcal{Q}$, key, $\mathcal{K}$, and value $\mathcal{V}$). The inputs are projected into the same space, \ie, $K=W_{K}^{\top}\left[\,\mathbf{k}_{k}; \, \forall \mathbf{k}_{k} \in \mathcal{K} \,\right] \in \mathbb{R}^{d \times|\mathcal{K}|}$. Similar projections are made for $\mathcal{Q}$ and $\mathcal{V}$. The query $\x_q\in\mathcal{Q}$ is matched against a list of keys $K$ where each key has a value $V$. The output is the sum of all the values weighted by the proximity of the key to the query point:
\begin{align}\label{eq:transformer} \textstyle
\tilde{P}_i(\z)={P}_i(\z)+\sum_{k} \alpha_{q k} V_{:, k} \,,
\end{align}
where $
\alpha_{q k} \propto \exp \left(\frac{{P}_i(\z)^{\top} W_{Q} \cdot K}{\sqrt{d}}\right)$, $V_{:, k}$ is the $k$-th column of $V$. The adaptation process is the same for other components in $\textbf{Context}$. Specifically, we have $\mathcal{Q}=\mathcal{K}=\mathcal{V}=[P_i(\z), \textbf{Context}]$.
\noindent\textbf{Effect of Cross-Modal Fusion}:
The illustration of the projection fusion is shown in Figure~\ref{figure:teaser} (right). We utilize the visual and textual information of seen classes as context information to help adjust the {\em instance-specific} embeddings. The fusion model is trained incrementally to adjust embeddings to reflect the context information as data evolves. With the contextualized embeddings, we can conduct the {\em visual mapping} and {\em textual matching}:
\begin{equation} \label{eq:fuse}
p(y_i \mid \x)=
\underbrace{
\frac{\exp \left(\cos \left(\tilde{P}_i(\z), \tilde{P}_i(\p_i) \right) / \tau\right)}{\sum_{j=1}^{|\mathcal{Y}_b|} \exp \left(\cos \left(\tilde{P}_i(\z), \tilde{P}_i(\p_j) \right) / \tau\right)}
}_{\text{Visual Matching}}
+
\underbrace{
\frac{\exp \left(\cos \left(\tilde{P}_i(\z), \tilde{P}_t(\w_i) \right) / \tau\right)}{\sum_{j=1}^{|\mathcal{Y}_b|} \exp \left(\cos \left(\tilde{P}_i(\z), \tilde{P}_t(\w_j) \right) / \tau\right)}
}_{\text{Textual Matching}}
\,.
\end{equation}
In Eq.~\ref{eq:fuse}, the model assigns logits to the query instance by the similarity to the {\em adapted} visual and textual prototypes.
The incorporation of cross-modal matching enhances the prediction performance.
\noindent\textbf{Learning Context Prompts}: In addition to visual prototypes and textual classifiers, we also introduce a set of learnable {\em context prompts} $\{\mathbf{c}^1,\cdots,\mathbf{c}^b\}, \mathbf{c}^i\in \R^{c\times d}$ to be optimized as data evolves. $c$ denotes the length of each prompt.
Similar to projection layers, we make the context prompts {\em expandable} to catch the new characteristics of new tasks. We initialize a new context prompt while learning a new task and freeze others $\{\bar{\mathbf{c}}^1, \bar{\mathbf{c}}^2,\cdots,{\mathbf{c}^b}\}$.
The context prompts serve as {\em adaptable} context information, enhancing the co-adaption.
Hence, the context information is formulated as $\textbf{Context}=[\mathbf{P}, \mathbf{W}, \mathbf{C}]$, where $\mathbf{C}$ is the aggregation of all context prompts. Note that $\mathbf{C}$ only encodes the task-specific information into the self-attention process, which does not serve as the matching target in Eq.~\ref{eq:fuse}.
\subsection{Summary of \name}
In \mame, we first enable learning new concepts via projected mapping. Then, to accommodate new concepts without interference from previous ones, we initialize new projections for each new task and freeze the former ones. Besides, we utilize self-attention to adjust the embeddings of the query instance and the context information to promote cross-modal fusion.
Figure~\ref{figure:teaser} illustrates three predictions, \ie, projected matching (Eq.~\ref{eq:proj_pred}), visual/textual matching (Eq.~\ref{eq:fuse}).
We denote these models as $f_\text{PM}(\x), f_\text{VM}(\x), f_\text{TM}(\x)$, respectively.
During training, we optimize the cross-entropy loss:
\begin{equation} \label{eq:loss} \textstyle
\min_{\{P_i^b,P_t^b,\mathcal{T},\mathbf{c}^b \}} \ell(f_\text{PM}(\x),y) + \ell(f_\text{VM}(\x),y) +\ell(f_\text{TM}(\x),y) \,.
\end{equation}
In Eq.~\ref{eq:loss}, all pre-trained weights are frozen, and we only optimize these {\em additional} parameters. For inference, we aggregate the three logits, \ie, $f(\x)=f_\text{PM}(\x)+f_\text{VM}(\x)+f_\text{TM}(\x)$.
We give the pseudo-code of \name in the supplementary.
\afterpage{
\vspace{-7mm}
\begin{table}[t]
\caption{\small Average and last performance of different methods.
The first and second columns represent the methods with and without exemplars.
The performance of L2P and DualPrompt are reproduced with the source code with exemplars. The best performance is shown in bold. Full results are reported in supplementary.
}\label{tab:benchmark}
% \vspace{-3mm}
\centering
\resizebox{1.0\textwidth}{!}{%
\begin{tabular}{@{}lccccccccccccccc}
\toprule
\multicolumn{1}{c}{\multirow{3}{*}{Method}}
&
\multicolumn{1}{c}{\multirow{3}{*}{Exemplar}}
&
\multicolumn{4}{c}{ImageNet-R } &
\multicolumn{4}{c}{CUB } & \multicolumn{4}{c}{UCF }
\\
& &
\multicolumn{2}{c}{B0 Inc20} &
\multicolumn{2}{c}{B100 Inc20} & \multicolumn{2}{c}{B0 Inc20} &
\multicolumn{2}{c}{B100 Inc20} &
\multicolumn{2}{c}{B0 Inc10} &
\multicolumn{2}{c}{B50 Inc10} &
\\
& &
{$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
\\
\midrule
Finetune & \ding{55} & 1.37 & 0.43& 1.01 & 0.88& 2.06 & 0.64& 0.56 & 0.47& 4.51 & 1.59& 1.21 & 0.80\\
Finetune LiT~\cite{zhai2022lit}& \ding{55} & 64.88 & 30.42& 57.75 & 29.77& 58.15 & 35.28& 51.95 & 35.96& 79.25 & 64.84& 81.79 & 65.4\\
Finetune CoOp~\cite{zhou2022learning}& \ding{55} &60.73 & 37.52& 54.20 & 39.77& 27.61 & 8.57& 24.03 & 10.14& 47.85 & 33.46& 42.02 & 24.74\\
SimpleCIL~\cite{zhou2023revisiting}& \ding{55} & 81.06 & 74.48& 76.84 & 74.48& 83.81 & 77.52& 79.75 & 77.52& 90.44 & 85.68& 88.12 & 85.68\\
ZS-CLIP~\cite{radford2021learning}& \ding{55} &83.37 & 77.17& 79.57 & 77.17 & 74.38 & 63.06& 67.96 & 63.06& 75.50 & 67.64& 71.44 & 67.64\\
\midrule
CoOp~\cite{zhou2022learning}& \ding{51} &82.40 & 76.20& 79.76 & 77.13& 77.34 & 68.70& 74.09 & 67.47& 90.13 & 86.24& 88.36 & 85.71\\
iCaRL~\cite{rebuffi2017icarl}& \ding{51} & 72.22 & 54.38& 68.67 & 60.15& 82.04 & 74.74& 78.57 & 75.07 & 89.47 & 84.34& 88.51 & 84.11\\
MEMO~\cite{zhou2022model}& \ding{51} & 80.00 & 74.07 & 76.72 & 73.95& 77.32 & 65.69 & 72.88 & 66.41& 84.02 & 74.08& 82.58 & 75.48\\
L2P~\cite{wang2022learning} & \ding{51} & 75.73 & 67.22& 74.15 & 71.20& 79.23 & 68.54& 75.85 & 71.12& 88.71 & 83.93& 86.51 & 83.22\\
DualPrompt~\cite{wang2022dualprompt} & \ding{51} &78.47 & 70.82& 72.98 & 69.18 & 83.21 & 74.94& 78.06 & 74.27& 89.48 & 85.41& 86.96 & 84.65\\
\midrule
\rowcolor{LightCyan}\name & \ding{51} & \bf85.34 &\bf 80.10& \bf82.32 & \bf80.30 & \bf84.93 &\bf 79.43&\bf 81.67 &\bf 79.18& \bf92.34 &\bf 89.92&\bf 91.70 & \bf89.16\\
\bottomrule
\end{tabular}
}
\end{table}
\begin{figure*}[t]
\vspace{-3mm}
\begin{center}
\subfigure[\small Aircraft Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/aircraft}}
\subfigure[\small CIFAR100 Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/cifar}}
\subfigure[\small Cars Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/cars}}\\
\vspace{-4mm}
\subfigure[\small SUN Base150 Inc30]
{\includegraphics[width=.325\columnwidth]{./pics/sunb150}}
\subfigure[\small Food Base50 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/foodb50}}
\subfigure[\small ObjectNet Base100 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/objectnetb100}}\\
\end{center}
\vspace{-5mm}
\caption{
\small
Incremental performance of different methods. We report the performance gap after the last incremental stage of \name and the runner-up method at the end of the line. Finetune-based methods in Table~\ref{tab:benchmark} are not plotted due to their inferior performance.
\vspace{-3mm}
}
\label{figure:benchmark}
\end{figure*}
}
\section{Experiment}
In this section, we compare \name in comparison to state-of-the-art methods on benchmark datasets to investigate the capability of overcoming forgetting. We also conduct ablations to analyze the effect of each component in the model.
Furthermore, we address a fundamental issue in VLM training known as zero-shot degradation. Finally, we extend \name to other VLMs to verify the universality of proposed method. Further details and experimental results can be found in the supplementary.
\subsection{Experimental Setup}\label{sec:setup}
\noindent {\bf Dataset}: Following the benchmark CIL settings~\cite{rebuffi2017icarl,wang2022learning,wang2022dualprompt,yu2020semantic,zhou2023revisiting}, we evaluate the performance on \bfname{CIFAR100}~\cite{krizhevsky2009learning}, \bfname{CUB200}~\cite{WahCUB2002011}, \bfname{ObjectNet}~\cite{barbu2019objectnet}, and \bfname{ImageNet-R}~\cite{deng2009imagenet}. We also follow the setting in VLM tuning~\cite{zhou2022learning}, and formulate \bfname{FGVCAircraft}~\cite{maji2013fine}, \bfname{StanfordCars}~\cite{krause20133d}, \bfname{Food101}~\cite{bossard2014food}, \bfname{SUN397}~\cite{xiao2010sun} and \bfname{UCF101}~\cite{soomro2012ucf101} into CIL setting.
Specifically, we sample (a subset of) 100 classes from CIFAR100, Aircraft, Cars, Food, UCF, 200 classes from CUB200, ObjectNet, ImageNet-R, and 300 classes from SUN to ease the data split.
Following~\cite{rebuffi2017icarl}, the class order of training classes is shuffled with random seed 1993. The dataset splits are denoted as \bfname{Base-$x$, Inc-$y$}, where $x$ represents the number of classes in the first stage, and $y$ represents the number of new classes in each subsequent task.
$x=0$ means each task contains $y$ classes. More details are reported in the supplementary.
\noindent {\bf Comparison methods:} We first compare to SOTA CIL methods iCaRL~\cite{rebuffi2017icarl}, MEMO~\cite{zhou2022model}, SimpleCIL~\cite{zhou2023revisiting} L2P~\cite{wang2022learning}, DualPrompt~\cite{wang2022dualprompt}. Denote the baseline of sequential finetuning as Finetune; we combine it with different tuning techniques, \eg, LiT~\cite{zhai2022lit} and CoOp~\cite{zhou2022learning}. We also report the zero-shot performance of CLIP as ZS-CLIP by matching the query instance to the template (Eq.~\ref{eq:clip_pred}).
\noindent {\bf Implementation details:}
We deploy all methods with PyTorch~\cite{paszke2019pytorch} and PyCIL~\cite{zhou2023pycil} on Tesla V100.
We use the \emph{same} network backbone ViT-B/16 for all compared methods for {\em fair comparison}.
We experiment with two commonly used pre-trained CLIP weights, \ie, OpenAI~\cite{radford2021learning} and OpenCLIP LAION-400M~\cite{ilharco_gabriel_2021_5143773}.
The model is trained with a batch size of 64 for 5 epochs, and we use SGD with momentum for optimization. The learning rate starts from $0.001$ and decays with cosine annealing.
Following~\cite{rebuffi2017icarl}, we use the herding~\cite{welling2009herding} algorithm to select 20 exemplars per class for rehearsal.
The context prompt length is set to 3, and the head of self-attention is set to 1. The template for classification is the same as~\cite{mu2022slip}.
The source code will be made publicly available upon acceptance.
\noindent\textbf{Performance Measure:} Denote the Top-1 accuracy after the $b$-th stage as $\mathcal{A}_b$, we follow~\cite{rebuffi2017icarl} to use $\mathcal{A}_B$ (last stage performance) and $\bar{\mathcal{A}}=\frac{1}{B}\sum_{b=1}^{B}\mathcal{A}_b$ (average performance) for evaluation.
\subsection{Benchmark Comparison} \label{sec:benchmark}
We report the results on nine benchmark datasets using ViT-B/16 (OpenCLIP LAION-400M) in Table~\ref{tab:benchmark} and Figure~\ref{figure:benchmark}.
These splits include the scenarios with large and small base classes.
Notably, \name consistently achieves the best performance among all the methods compared.
Sequential finetuning of the model with contrastive loss leads to significant forgetting, irrespective of the tuning techniques employed (\eg, LiT and CoOp).
Since SimpleCIL and ZS-CLIP do not finetune the model parameters, they achieve competitive results by transferring the knowledge from the pre-training stage into the downstream tasks.
However, most methods achieve better performance than ZS-CLIP, indicating the importance of incremental learning on downstream tasks.
Specifically, we can draw three key conclusions from these results.
1) The first stage performance of \name surpasses that of the typical prompt learning method, CoOp, thus validating the effectiveness of learning projections for downstream tasks.
2) The performance curve of \name consistently ranks at the top across all methods, demonstrating its capability to resist forgetting.
3) Compared to vision-only methods (\ie, L2P and DualPrompt), \name exhibits substantial improvement, indicating textual and visual information can be co-adapted to facilitate incremental learning.
\begin{figure*}[t]
\vspace{-7mm}
\begin{center}
\subfigure[\small OpenAI weight]
{\includegraphics[width=.323\columnwidth]{./pics/radar_} \label{figure:ablationa}}
\subfigure[\small Compositional components]
{\includegraphics[width=.323\columnwidth]{./pics/ablation}
\label{figure:ablationb}}
\subfigure[\small Context prompt length]
{\includegraphics[width=.323\columnwidth]{./pics/ablations}
\label{figure:ablationc}}
\end{center}
\vspace{-5mm}
\caption{\small Ablation study. {\bf Left:} experiments on 9 benchmarks with OpenAI weights. {\bf Middle:} ablation study on compositional components in \mame. Every part improves the performance of CIL. {\bf Right:} $\mathcal{A}_B$ and $\bar{\mathcal{A}}$ with change of context prompts. The performance is robust to the change of context prompt length.
} \label{figure:ablation}
\vspace{-5mm}
\end{figure*}
\subsection{Ablation Study} \label{sec:ablation}
\noindent\textbf{Different backbone weights:} The comparison in Section~\ref{sec:benchmark} is based on LAION-400M pre-trained CLIP. As another popular pre-trained weight, we also explore the performance of the weights provided by OpenAI. We report the last accuracy $\mathcal{A}_B$ of four competitive methods on nine benchmarks in Figure~\ref{figure:ablationa}. We report the full results of the incremental performance in the supplementary. As depicted in the figure, \name still performs the best on all datasets among all compared methods.
\noindent\textbf{Compositional components:} We experiment on CIFAR100 B0 Inc10 to investigate the importance of each part in \mame. Specifically, we compare the performance of \name and its sub-modules, \ie, projections and cross-modal fusion.
The results, shown in Figure~\ref{figure:ablationb}, indicate that training expandable projections or the fusion module individually can both enhance the performance of vanilla CLIP.
This suggests that the expandable task representation and cross-modal information can help the learning process. Furthermore, when combining them together, we find `Projection \& Fusion' further shows better performance than any of them, verifying that they can work together by fusing the expandable representations. Lastly, when incorporating the context prompts, the model shows the best performance among all variations, verifying the effectiveness of expandable task-specific prompts in incremental learning. Ablations verify the importance of each component in \mame.
\noindent\textbf{Number of context prompts:} Figure~\ref{figure:ablationb} verifies the strong performance of context prompts, and we explore the appropriate length $c$ of the context prompt on CIFAR100 B0 Inc10. By varying the number of $c$ among $\{1,2,3,4,5,6,7,8,9,10,30,50,100\}$, we report the average performance and last performance of \name in Figure~\ref{figure:ablationc}. As shown in the figure, the performance of \name is robust with the change of the prompt length, and we set $c=3$ as the default length.
\begin{figure*}[t]
\vspace{-7mm}
\begin{center}
\subfigure[\small Unseen class accuracy]
{\includegraphics[width=.323\columnwidth]{./pics/unseen_acc}
\label{figure:zs-a}}
\subfigure[\small LAION score]
{\includegraphics[width=.323\columnwidth]{./pics/laion_score}
\label{figure:zs-b}}
\subfigure[\small $\mathcal{A}_\text{S} \,,\mathcal{A}_\text{U} \,, \mathcal{A}_\text{HM}$ ]
{\includegraphics[width=.323\columnwidth]{./pics/diff_accs_last}
\label{figure:zs-c}}\\
\end{center}
\vspace{-5mm}
\caption{\small Experiment on zero-shot performance. {\bf Left:} accuracy on unseen classes during incremental learning. {\bf Middle:} LAION score during incremental learning. {\bf Right:} accuracy of seen, unseen, and harmonic mean (HM) at the last incremental stage. \mame$^\dagger$ strikes a balance between adaptivity and the ZS performance.
} \label{figure:zs}
\vspace{-5mm}
\end{figure*}
\subsection{Exploring Zero-Shot Performance}\label{sec:zero_shot}
CLIP is known to have the zero-shot (ZS) ability, \ie, even if the model has not been trained for recognizing the image, it can still predict the possibility of an image $\x$ belonging to the class $y$ by matching the cosine similarity via Eq.~\ref{eq:clip_pred}.
The strong generalizability of CLIP makes it a popular model in computer vision.
However, in CIL, the model is {\em continuously} updated with the downstream task, which weakens the generalizability and harms the ZS performance~\cite{wortsman2022robust} on subsequent tasks. In this section, we explore the ZS performance degradation of CLIP and propose a variation of \name to maintain the ZS performance.
\noindent\textbf{Evaluation protocol for ZS performance:} Current CIL methods focus on evaluating `seen' classes, \ie, evaluating $\mathcal{Y}_b=Y_1 \cup \cdots Y_b$ after learning task $b$. However, since CLIP exhibits ZS performance, we can also assess the performance on `unseen' classes $\mathcal{Y}_u=Y_{b+1} \cup \cdots Y_B$ to investigate the ZS performance. Correspondingly, we can obtain the performance metrics $\mathcal{A}_\text{S}$ (seen classes), $\mathcal{A}_\text{U}$ (unseen classes), and $\mathcal{A}_\text{HM}$ (harmonic mean of $\mathcal{A}_{S}$ and $\mathcal{A}_{U}$) after each task.
Additionally, based on the LAION400M~\cite{schuhmann2021laion} pre-trained CLIP, we also utilize a subset of 10,000 image-text pairs from LAION400M, and calculate the matching score of them, \ie, cosine similarity of image-text embeddings.
We denote the average matching score as {\em LAION score}, which indicates the matching degree of the adapted model on the {\em upstream} tasks.
Given the relationship between generalizability and the upstream task, the LAION score serves as an effective measure of ZS performance.
\noindent\textbf{Results:} We compare the aforementioned measures on CIFAR100 B0 Inc10. Apart from compared methods in Section~\ref{sec:benchmark}, we also report a variation of \mame, namely \mame$^\dagger$.
The only difference lies in the design of the projection, where \mame$^\dagger$ uses a {\em residual} format $P_i(\z)=\sum_{m=1}^b \left(P_i^m \left(\z\right) + \z\right)$ as the output (same for $P_t$).
To investigate the ZS performance as model updates, we show the accuracy on unseen classes $\mathcal{A}_\text{U}$ along incremental stages in Figure~\ref{figure:zs-a}, where ZS-CLIP shows the best performance.
Due to the incorporation of pre-trained information into the projected features, \mame$^\dagger$ maintains competitive ZS performance.
Conversely, other methods experience a decline in ZS performance as their focus shifts to downstream tasks.
We observe a similar trend in Figure~\ref{figure:zs-b}, where \mame$^\dagger$ achieves a LAION score similar to that of ZS-CLIP.
Lastly, we report $\mathcal{A}_\text{S},\mathcal{A}_\text{U}, \mathcal{A}_\text{HM}$ in the last incremental stage in Figure~\ref{figure:zs-c}.
We can infer a {\em trade-off} between the adaptivity on downstream tasks and the generalizability of ZS performance. Compared to \mame, \mame$^\dagger$ sacrifices the adaptivity to maintain ZS performance, strikings a balance between seen and unseen classes. Therefore, when ZS performance is essential, using \mame$^\dagger$ is the preferred choice.
\begin{figure*}[t]
\vspace{-5mm}
\begin{center}
{\includegraphics[width=.98\columnwidth]{./pics/suppfig/retrieval}}
\end{center}
\caption{\small The training protocol of five incremental stages in Flickr30K. We split training instances into five tasks, \ie, walk, stand, run, ride, and play. The training/testing sets do not include images that do not fall into these tasks. We use the pre-trained BEiT-3 as the initialization and sequentially learn cross-modal retrieval tasks. At the end of each task, the model is evaluated on all previously learned concepts.
} \label{figure:supp_retrieval_protocol}
\end{figure*}
\begin{table}[t]
\caption{\small Average and last performance of different methods. The best is in bold. The first row stands for the text retrieval task, and the second is the image retrieval task.
}\label{tab:supp_retrieval}
\centering
% \resizebox{1.0\textwidth}{!}{%
\begin{tabular}{@{}lccccccccccccccc}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{Method}}
&
\multicolumn{6}{c}{Image $\rightarrow$ Text }
\\
&
{$R_B@1$} & {$\bar{R}@1$}& {$R_B@5$} & {$\bar{R}@5$}
& {$R_B@10$} & {$\bar{R}@10$}
\\
\midrule
Finetune & 48.79 & 62.89 & 76.38 & 85.04 & 85.68 & 91.84\\
DER~\cite{yan2021dynamically} & 78.37 & 84.48 & 96.34 & 98.23 & 99.06 & 99.59 \\
MEMO~\cite{zhou2022model} & 83.18 & 87.79 & 96.57 & 98.27 & 99.16 & 99.66 \\
\midrule
\rowcolor{LightCyan} \name & \bf85.68 & \bf89.43 & \bf97.07 &\bf 98.68 & \bf99.79 &\bf 99.86 \\
% \bottomrule
\end{tabular}
\begin{tabular}{@{}lccccccccccccccc}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{Method}}
&
\multicolumn{6}{c}{Text $\rightarrow$ Image }
\\
&
{$R_B@1$} & {$\bar{R}@1$}& {$R_B@5$} & {$\bar{R}@5$}
& {$R_B@10$} & {$\bar{R}@10$}
\\
\midrule
Finetune & 37.35 & 51.33 & 67.38 & 77.77 & 77.95 & 85.55\\
DER~\cite{yan2021dynamically} & 66.71 & 74.18 & 89.63 & 93.00 & 94.84 & 96.69 \\
MEMO~\cite{zhou2022model} & 69.53 & 76.35 & 91.89 & 94.44 & 96.09 & 97.32 \\
\midrule
\rowcolor{LightCyan} \name & \bf 72.10 & \bf78.01 & \bf93.10 &\bf 95.27 &\bf 96.92 &\bf 97.90 \\
\bottomrule
\end{tabular}
% }
\end{table}
\begin{figure*}[t]
\vspace{-5mm}
\begin{center}
\subfigure[IR@1]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/retrieval_ir1}}
\subfigure[IR@5]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/retrieval_ir5}}
\subfigure[IR@10]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/retrieval_ir10}}\\
\subfigure[TR@1]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/retrieval_tr1}}
\subfigure[TR@5]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/retrieval_tr5}}
\subfigure[TR@10]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/retrieval_tr10}}\\
\end{center}
\vspace{-4mm}
\caption{\small
Incremental performance of each method. IR means the recall of image retrieval, and TR denotes the recall of text retrieval.
{\bf \name consistently outperforms other compared methods with a substantial margin on the incremental cross-modal retrieval task.}
}
\label{figure:retrieval}
\end{figure*}
\subsection{Extension to Other Vision Language Models} \label{sec:other_vlms}
In the main paper, we use CLIP as an exemplar VLM due to its popularity and representativeness. However, the field of vision-language models is rapidly advancing, and various models are available. Therefore, in this section, we extend our \name framework to another widely used vision-language model, namely BEiT-3~\cite{wang2022image}, focusing on the cross-modal retrieval task.
BEiT-3 is a popular VLM that demonstrates promising performance across multiple vision-language tasks. When fine-tuning BEiT-3 for cross-modal retrieval, it functions as a {\em dual encoder}, similar to CLIP, featuring a dual-branch structure. As the retrieval task differs from classification, we adopt a degradation of \name by solely employing the projection expansion strategy without implementing cross-modal fusion. We refer the readers to the BEiT-3 paper~\cite{wang2022image} for more details about the backbone model.
For evaluation, we employ the Flickr30K dataset~\cite{plummer2015flickr30k} to assess the performance of incremental cross-modal retrieval. Flickr30K comprises 31,783 images collected from the Flickr image-sharing platform, encompassing diverse themes such as daily life, travel, people, food, and scenes. Each image in the dataset is accompanied by {\em five} manually annotated textual descriptions, which provide descriptive information capturing the main content and context of the images. To formulate an incremental data stream, we utilize keyword matching to identify images containing different actions (\eg, walk, stand, run, ride, play). Then, we split the training instances into five subsets based on these specific actions.
Figure~\ref{figure:supp_retrieval_protocol} illustrates the formulation of the stream, while images not associated with these actions are excluded from training.
To create a balanced testing set, we maintain a 5:1 training-to-testing ratio for splitting the training and testing pairs.
Following the instructions provided by BEiT\footnote{\url{https://github.com/microsoft/unilm/blob/master/beit3/README.md}},
we use `beit3\_base\_itc\_patch16\_224\footnote{\url{https://conversationhub.blob.core.windows.net/beit-share-public/beit3/pretraining/beit3\_base\_itc\_patch16\_224.pth}}' as the VLM's initialization.
For evaluation, we employ standard cross-modal retrieval measures, namely $R@1$, $R@5$, and $R@10$. The retrieval is conducted in two directions: image $\rightarrow$ text and text $\rightarrow$ image. Similarly to the CIL evaluation, we also report the last recall $R_B@1$ and the average recall $\bar{R}@1$ across incremental stages.
To provide a comparative analysis, we compare \name against typical fine-tuning as the baseline and modify MEMO~\cite{zhou2022model} and DER~\cite{yan2021dynamically} for comparison. These methods represent state-of-the-art CIL approaches that can be adapted with minor modifications to the current task. However, methods such as L2P and DualPrompt are unsuitable for cross-modal retrieval tasks as they do not focus on cross-modal matching.
The experimental results are presented in Table~\ref{tab:supp_retrieval}, and the incremental performance of each measure is depicted in Figure~\ref{figure:retrieval}. As evident from these figures, fine-tuning the model with new concepts leads to catastrophic forgetting in cross-modal retrieval tasks. However, equipping the model with incremental learning abilities alleviates forgetting. Among all the compared methods, \name consistently achieves the best performance across different retrieval tasks and metrics, thereby verifying its effectiveness in mitigating forgetting in VLMs.
Experiments conducted on different VLMs and tasks establish \name as a {\em unified and general} framework. Future work involves extending \name to other VLMs and applications, such as image captioning~\cite{vinyals2015show} and VQA~\cite{antol2015vqa}.
\section{Conclusion} \label{sec:conclusion}
Real-world learning systems necessitate the ability to continually acquire new knowledge. In this paper, we aim to equip the popular VLM with the CIL ability. Specifically, we learn the expandable projections so that visual and textual information can be aligned incrementally.
This expansion technique allows for the integration of new concepts without compromising previous ones.
Additionally, we enforce cross-modality fusion with self-attention mechanism, where visual and textual information are jointly adapted to produce instance-specific embeddings.
Extensive experiments validate the effectiveness of our proposed \mame. Furthermore, we demonstrate that a simple variation of \name preserves the model's zero-shot capability during updating.\\
\noindent\textbf{Limitations:} Possible limitations include the usage of exemplars, where storage constraints and privacy issues may happen. Future works include extending the model to exemplar-free scenarios.
\newpage
\setcounter{section}{0}
\renewcommand{\thesection}{\Alph{section}}
\begin{center}
\textbf{\large Supplementary Material}
\end{center}
In the main paper, we present a method to prevent forgetting in vision-language models through projection expansion and fusion. The supplementary material provides additional details on the experimental results mentioned in the main paper, along with extra empirical evaluations and discussions. The organization of the supplementary material is as follows:
\begin{itemize}
\item Section~\ref{sec:pseudo_code} presents the pseudo code of \mame, explaining the training and testing pipeline.
\item Section~\ref{sec:supp_exp} reports comprehensive experimental results from the main paper, including the full results of nine benchmark datasets with two data splits, as well as the results obtained using OpenAI weights. Furthermore, this section includes additional ablations such as variations of projection types, results from multiple runs, and an analysis of the number of parameters.
\item Sections~\ref{sec:supp_exp_detail} and \ref{sec:broader_impact} provide detailed information on the experiments, including dataset and exemplar selection details, an introduction to the compared methods, and a discussion of the broader impacts.
\end{itemize}
\section{Pseudo Code} \label{sec:pseudo_code}
In this section, we provide a detailed explanation of \name by presenting the pseudo-code in Alg~\ref{alg1}. In each incremental stage, we are provided with the training dataset $\D^{b}$ and the exemplar set $\mathcal{E}$, with the objective of updating the current model $f(\cdot)$. Prior to training, we initially extract visual prototypes for the new classes (Line~\ref{line:1}). These prototypes are calculated using the frozen visual embedding $g_i(\cdot)$, ensuring their stability throughout model updates. Subsequently, we freeze the former projections and context prompts, while initializing new projections and context prompts specifically for the new incremental task (Line~\ref{line:2} to Line~\ref{line:4}). These steps represent the model expansion process, which is followed by the subsequent learning process.
During the learning process, we concatenate the training instances from the current dataset and the exemplar set, initiating a for-loop. For each instance-label pair, we calculate the projected visual and textual embeddings (Line~\ref{line:6} to Line~\ref{line:9}). Subsequently, we compute the projected matching loss (Line~\ref{line:10}) to encode task-specific information into the current projection layers. Based on the projected features, we derive context information and perform cross-modal fusion (Line~\ref{line:11} to Line~\ref{line:13}). Consequently, we obtain three logits for model updating and utilize the cross-entropy loss to update these modules (Line~\ref{line:14}). The updated model is then returned as the output of the training process.
\textbf{Discussions:}
Besides the simple addition operation, there exist alternative methods for aggregating information from multiple projections. However, due to the requirement of fixed input dimensionality for cross-modal fusion, we refrain from using concatenation as the aggregation function. Furthermore, it is worth noting that MEMO~\cite{zhou2022model} can be viewed as a specific case where concatenation is employed for aggregation. Nonetheless, its inferior performance (as shown in Table~\ref{tab:supp_benchmark}) suggests that summation is a more favorable choice.
\begin{algorithm}[t]
\caption{Training \name for CIL }
\label{alg1}
\raggedright
{\bf Input}: Training dataset: $\D^{b}$; Exemplar set: $\mathcal{E}$; Current model: $f(\cdot)$;\\
{\bf Output}: Updated model;
\begin{algorithmic}[1]
\State Extract prototypes $\mathbf{p}$ for each new class in $\D^{b}$; \label{line:1}
\State Freeze current projections and context prompts; \label{line:2}
\State Initialize new projections for the visual and textual branches, $P_i^b, P_t^b$; {\Comment{\color{cyan}{Expand projections}}}
\State Initialize new context prompt $\mathbf{c}^b$; \label{line:4}
\For{$(\x,y) \in \D^b\cup\mathcal{E}$} {\Comment{\color{cyan}{Incremental learning}}}
\State Calculate the visual embedding $\z=g_i(\x)$; \label{line:6}
\State Calculate the projected visual feature $P_i(\z)$;
\State Calculate the textual embedding $\w$ of all seen classes;
\State Calculate the projected textual embeddings of all seen classes $P_t(\w)$;
\label{line:9}
\State Calculate the logits for projected matching $f_\textbf{PM}(\x)$ via Eq.~5;{\Comment{\color{cyan}{Projected matching}}} \label{line:10}
\State Calculate the projected visual features for all visual prototypes $\p$; \label{line:11}
\State Conduct cross-modal fusion via Eq.~7; {\Comment{\color{cyan}{Cross-modal fusion}}}
\State Calculate the logits for visual and textual matching via Eq.~8; {\Comment{\color{cyan}{Visual \& textual matching}}}\label{line:13}
\State Calculate the loss via Eq.~9; update the model; \label{line:14}
\EndFor
\Return the updated model;
\end{algorithmic}
\end{algorithm}
\section{Additional Experimental Results}\label{sec:supp_exp}
This section presents further experimental results of \mame, including comparisons with multiple runs, analysis of parameter numbers, and ablations on projection types. Additionally, we report the results of using OpenAI pre-trained CLIP and provide the full results mentioned in the main paper.
\begin{figure*}[t]
\begin{center}
\subfigure[CIFAR100 B50 Inc10]
{\includegraphics[width=.483\columnwidth]{./pics/suppfig/cifarb50_errorbar}
\label{figure:tsnea}}
\subfigure[ImageNet-R B100 Inc20]
{\includegraphics[width=.483\columnwidth]{./pics/suppfig/inrb100_errorbar}
\label{figure:tsneb}}
\end{center}
\caption{ Results of multiple runs for CIFAR100 and ImageNet-R. The solid line represents the mean performance, while the shaded area indicates the standard deviation. {\bf \name consistently and robustly outperforms other methods by a substantial margin.}
} \label{figure:multiple_run}
\end{figure*}
\subsection{Multiple Runs}
Following~\cite{rebuffi2017icarl}, we conduct typical CIL comparisons by randomly splitting the classes with a fixed seed of 1993, and these results are reported in the main paper. In this supplementary section, we perform multiple runs by varying the random seed among \{$1993,1994,1995,1996,1997$\}. We repeat the comparison on CIFAR100 Base50 Inc10 and ImageNet-R Base100 Inc20 five times and present the results in Figure~\ref{figure:multiple_run}. The solid line represents the mean performance, while the shaded area indicates the standard deviation. From these figures, it is evident that \name consistently outperforms other methods by a significant margin across different dataset splits. These results validate the robustness of \mame.
\begin{figure*}[t]
\begin{center}
{\includegraphics[width=.6\columnwidth]{./pics/suppfig/params}}
\end{center}
\caption{
Number of parameters in different methods. The shaded area represents the parameters used during training but dropped during inference. \textbf{\name achieves state-of-the-art performance with a comparable number of parameters to other methods.}
} \label{figure:supp_param}
\end{figure*}
\subsection{Parameter Analysis}
As mentioned in the main paper, the additional parameters in \name come from two sources: the projections and the fusion module. The projection layers are implemented with a single linear layer, each containing $d\times d$ parameters, where $d=512$ is the embedding dimension. Similarly, the cross-modal fusion is implemented with a single-head self-attention mechanism, and the number of parameters is determined by the weight matrices $W_Q$, $W_K$, and $W_V$, each containing $d\times d$ parameters. These extra parameters are negligible compared to the large backbone of the pre-trained CLIP model, which has approximately 150 million parameters.
\begin{figure*}[t]
\begin{center}
{\includegraphics[width=.6\columnwidth]{./pics/suppfig/supp_proj_type}}
\end{center}
\caption{ Variations of projection layers. {\bf The choice of using a single linear layer as the projection layer achieves the best performance.}
} \label{figure:supp_projection}
\end{figure*}
To provide a clear comparison of the parameter numbers for each method, we present the details in Figure~\ref{figure:supp_param} using CIFAR100 B0 Inc10 as an example. The figure illustrates that \name has a similar parameter scale to other finetune-based methods, while achieving significantly stronger performance. SimpleCIL, which only utilizes the vision branch, requires fewer parameters for the textual branch but lacks the zero-shot capability. L2P and DualPrompt also only require the vision branch but need an additional encoder to identify the appropriate prompt, resulting in a higher parameter count compared to \mame.
\subsection{Variation of Projection Types}
Apart from simple linear layers, there are other methods to implement the projection layers, such as layer-wise rescale (SSF)~\cite{lianscaling} and Adapter~\cite{houlsby2019parameter}. SSF learns a $d$-dimensional rescale parameter to project the features, while Adapter learns both the down-projection and up-projection for feature mapping. In this section, we explore the performance of these projection methods on CIFAR100 B0 Inc10 and present the results in Figure~\ref{figure:supp_projection}. The figure clearly demonstrates that using a single linear layer as the projection layer achieves the best performance among all methods, indicating its superiority.
Furthermore, this result suggests that a simple linear mapping can effectively bridge the gap between visual and textual domains.
\subsection{Variation of Context Information}
\begin{figure*}[t]
\begin{center}
{\includegraphics[width=.6\columnwidth]{./pics/suppfig/supp_context}}
\end{center}
\caption{ Variations of context information. {\bf The choice of using visual prototypes, textual prototypes, and context prompts as the context information achieves the best performance.}
} \label{figure:supp_context}
\end{figure*}
In the main paper, we discuss the composition of the context information $\mathbf{Context}$, which should include information from visual prototypes, textual classifiers, and context prompts. In this section, we conduct ablations to demonstrate the effectiveness of constructing $\mathbf{Context}$ with $[\mathbf{P}, \mathbf{W}, \mathbf{C}]$. Specifically, we perform experiments on CIFAR100 B0 Inc10 and change the context construction to $\mathbf{Context}=\mathbf{P}$ (visual prototypes only), $\mathbf{Context}=\mathbf{W}$ (textual prototypes only), $\mathbf{Context}=[\mathbf{P}, \mathbf{W}]$ (visual and textual prototypes), and $\mathbf{Context}=[\mathbf{P}, \mathbf{W}, \mathbf{C}]$ (current choice). We keep the same classification rule for these ablations, \ie, classification via Eq.~9. When visual/textual prototypes are not included in the context, we use the projected features without adaptation as the matching target in Eq.~8. The results are presented in Figure~\ref{figure:supp_context}.
From the results, we observe that using visual prototypes or textual prototypes alone yields similar performance, and the impact of adjustment is marginal. However, when both visual and textual prototypes are jointly utilized as context information, the model can learn from cross-modality and achieve better performance. Lastly, the introduction of context prompts into the context further enhances the performance of \mame, resulting in the best performance among all variations.
\begin{figure*}[t]
\begin{center}
\subfigure[ Aircraft Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/aircraft_openai}}
\subfigure[CIFAR100 Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/cifar_openai}}
\subfigure[Cars Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/cars_openai}}\\
\subfigure[ImageNet-R Base0 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/inr_openai}}
\subfigure[CUB Base0 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/cub_openai}}
\subfigure[UCF Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/ucf_openai}}\\
\subfigure[SUN Base0 Inc30]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/sun_openai}}
\subfigure[Food Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/food_openai}}
\subfigure[ObjectNet Base0 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/suppfig/objectnet_openai}}\\
\end{center}
\caption{
Incremental performance of different methods when using {\bf OpenAI weights}. We report the performance gap after the last incremental stage of \name and the runner-up method at the end of the line. {\bf \name consistently achieves the best performance regardless of the pre-trained weights used.}
}
\label{figure:benchmark_openai}
\end{figure*}
\subsection{Different Pre-trained Weights}
In the main paper, we discussed two popular weights for pre-trained CLIP: OpenAI~\cite{radford2021learning}\footnote{\url{https://github.com/openai/CLIP}} and OpenCLIP~\cite{ilharco_gabriel_2021_5143773}\footnote{\url{https://github.com/mlfoundations/open_clip}}. We primarily presented the results of the OpenCLIP pre-trained model in the main paper, while providing the results of the OpenAI weights using a radar chart. In this section, we present the full results of the OpenAI pre-trained CLIP on nine benchmark datasets in Figure~\ref{figure:benchmark_openai}. The results demonstrate that \name consistently achieves the best performance among all methods, regardless of the pre-trained weights used. This highlights the robustness of \name in the learning process.
\subsection{Full Results}
We provide the complete results of the benchmark comparison in the main paper, which are presented in Table~\ref{tab:supp_benchmark} and Figures~\ref{figure:benchmark_full} and \ref{figure:benchmark_large_base}. These results are obtained using OpenCLIP pre-trained weights on LAION-400M~\cite{ilharco_gabriel_2021_5143773}. Table~\ref{tab:supp_benchmark} displays the average and last accuracy for the nine benchmark datasets. Figures~\ref{figure:benchmark_full} and \ref{figure:benchmark_large_base} illustrate the incremental performance with varying numbers of base classes. Across all these evaluations, \name consistently outperforms the compared methods, demonstrating its superior performance.
\begin{table}[t]
\caption{Average and last performance comparison of different methods.
The first and second columns represent the methods with and without exemplars.
The performance of L2P and DualPrompt are reproduced with the source code with exemplars. The best performance is shown in bold.
}\label{tab:supp_benchmark}
\centering
\resizebox{1.0\textwidth}{!}{%
\begin{tabular}{@{}lccccccccccccccc}
\toprule
\multicolumn{1}{c}{\multirow{3}{*}{Method}}
&
\multicolumn{1}{c}{\multirow{3}{*}{Exemplar}}
&
\multicolumn{4}{c}{Aircraft } &
\multicolumn{4}{c}{CIFAR100 } & \multicolumn{4}{c}{Cars }
\\
& &
\multicolumn{2}{c}{B0 Inc10} &
\multicolumn{2}{c}{B50 Inc10} & \multicolumn{2}{c}{B0 Inc10} &
\multicolumn{2}{c}{B50 Inc10} &
\multicolumn{2}{c}{B0 Inc10} &
\multicolumn{2}{c}{B50 Inc10} &
\\
& &
{$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
\\
\midrule
Finetune & \ding{55} & 3.16 & 0.96 & 1.72 & 1.05 & 7.84 & 4.44& 5.30 & 2.46& 3.14 & 1.10 & 1.54 & 1.13\\
Finetune LiT~\cite{zhai2022lit}& \ding{55} & 27.74 & 14.28 & 25.10 & 13.77 & 44.66 & 14.69& 27.69 & 7.67& 84.12 & 72.37& 83.08 & 78.23\\
Finetune CoOp~\cite{zhou2022learning}& \ding{55} & 14.54 & 7.14 & 13.05 & 7.77 & 47.00 & 24.24 & 41.23 & 24.12& 36.46 & 21.65& 37.40 & 20.87\\
SimpleCIL~\cite{zhou2023revisiting}& \ding{55} &59.24 & 48.09 & 53.05 & 48.09 & 84.15 & 76.63& 80.20 & 76.63& 92.04 & 86.85 & 88.96 & 86.85\\
ZS-CLIP~\cite{radford2021learning}& \ding{55} &26.66 & 17.22 & 21.70 & 17.22& 81.81 & 71.38& 76.49 & 71.38& 82.60 & 76.37& 78.32 & 76.37\\
\midrule
CoOp~\cite{zhou2022learning}& \ding{51} &44.26 & 39.87 & 41.81 & 39.18& 83.37 & 73.36& 78.34 & 73.04& 89.73 & 84.91& 87.98 & 86.60\\
iCaRL~\cite{rebuffi2017icarl}& \ding{51} & 53.60 & 43.98 & 50.40 & 45.33& 79.91 & 63.94& 71.94 & 63.00& 89.38 & 84.95& 86.71 & 84.19\\
MEMO~\cite{zhou2022model}& \ding{51} & 42.24 & 25.41 & 38.16 & 27.75& 84.67 & 74.98 & 80.75 & 75.34 & 88.23 & 81.31& 84.90 & 81.83\\
L2P~\cite{wang2022learning} & \ding{51} & 55.06 & 44.88 & 47.78 & 43.37 & 76.42 & 66.21 & 72.67 & 67.88& 83.81 & 72.44& 79.76 & 73.47\\
DualPrompt~\cite{wang2022dualprompt} & \ding{51} &55.95 & 46.53 & 50.93 & 46.50& 79.07 & 70.06& 74.81 & 70.75 & 85.30 & 74.35 & 81.32 & 75.85\\
\rowcolor{LightCyan}\name & \ding{51} & \bf 61.00 & \bf 53.59 & \bf 59.99 & \bf 58.90 & \bf86.70 &\bf 79.05 & \bf82.92 & \bf78.87& \bf93.26 &\bf 89.84 & \bf90.53 & \bf89.54\\
% \bottomrule
\end{tabular}
}
\resizebox{1.0\textwidth}{!}{%
\begin{tabular}{@{}lccccccccccccccc}
\toprule
\multicolumn{1}{c}{\multirow{3}{*}{Method}}
&
\multicolumn{1}{c}{\multirow{3}{*}{Exemplar}}
&
\multicolumn{4}{c}{ImageNet-R } &
\multicolumn{4}{c}{CUB } & \multicolumn{4}{c}{UCF }
\\
& &
\multicolumn{2}{c}{B0 Inc20} &
\multicolumn{2}{c}{B100 Inc20} & \multicolumn{2}{c}{B0 Inc20} &
\multicolumn{2}{c}{B100 Inc20} &
\multicolumn{2}{c}{B0 Inc10} &
\multicolumn{2}{c}{B50 Inc10} &
\\
& &
{$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
\\
\midrule
Finetune & \ding{55} & 1.37 & 0.43& 1.01 & 0.88& 2.06 & 0.64& 0.56 & 0.47& 4.51 & 1.59& 1.21 & 0.80\\
Finetune LiT~\cite{zhai2022lit}& \ding{55} & 64.88 & 30.42& 57.75 & 29.77& 58.15 & 35.28& 51.95 & 35.96& 79.25 & 64.84& 81.79 & 65.40\\
Finetune CoOp~\cite{zhou2022learning}& \ding{55} &60.73 & 37.52& 54.20 & 39.77& 27.61 & 8.57& 24.03 & 10.14& 47.85 & 33.46& 42.02 & 24.74\\
SimpleCIL~\cite{zhou2023revisiting}& \ding{55} & 81.06 & 74.48& 76.84 & 74.48& 83.81 & 77.52& 79.75 & 77.52& 90.44 & 85.68& 88.12 & 85.68\\
ZS-CLIP~\cite{radford2021learning}& \ding{55} &83.37 & 77.17& 79.57 & 77.17 & 74.38 & 63.06& 67.96 & 63.06& 75.50 & 67.64& 71.44 & 67.64\\
\midrule
CoOp~\cite{zhou2022learning}& \ding{51} &82.40 & 76.20& 79.76 & 77.13& 77.34 & 68.70& 74.09 & 67.47& 90.13 & 86.24& 88.36 & 85.71\\
iCaRL~\cite{rebuffi2017icarl}& \ding{51} & 72.22 & 54.38& 68.67 & 60.15& 82.04 & 74.74& 78.57 & 75.07 & 89.47 & 84.34& 88.51 & 84.11\\
MEMO~\cite{zhou2022model}& \ding{51} & 80.00 & 74.07 & 76.72 & 73.95& 77.32 & 65.69 & 72.88 & 66.41& 84.02 & 74.08& 82.58 & 75.48\\
L2P~\cite{wang2022learning} & \ding{51} & 75.73 & 67.22& 74.15 & 71.20& 79.23 & 68.54& 75.85 & 71.12& 88.71 & 83.93& 86.51 & 83.22\\
DualPrompt~\cite{wang2022dualprompt} & \ding{51} &78.47 & 70.82& 72.98 & 69.18 & 83.21 & 74.94& 78.06 & 74.27& 89.48 & 85.41& 86.96 & 84.65\\
\rowcolor{LightCyan}\name & \ding{51} & \bf85.34 &\bf 80.10& \bf82.32 & \bf80.30 & \bf84.93 &\bf 79.43&\bf 81.67 &\bf 79.18& \bf92.34 &\bf 89.92&\bf 91.70 & \bf89.16\\
% \bottomrule
\end{tabular}
}
\resizebox{1.0\textwidth}{!}{%
\begin{tabular}{@{}lccccccccccccccc}
\toprule
\multicolumn{1}{c}{\multirow{3}{*}{Method}}
&
\multicolumn{1}{c}{\multirow{3}{*}{Exemplar}}
&
\multicolumn{4}{c}{SUN } &
\multicolumn{4}{c}{Food } & \multicolumn{4}{c}{ObjectNet }
\\
& &
\multicolumn{2}{c}{B0 Inc30} &
\multicolumn{2}{c}{B150 Inc30} & \multicolumn{2}{c}{B0 Inc10} &
\multicolumn{2}{c}{B50 Inc10} &
\multicolumn{2}{c}{B0 Inc20} &
\multicolumn{2}{c}{B100 Inc20} &
\\
& &
{$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
& {$\bar{\mathcal{A}}$} & ${\mathcal{A}_B}$
\\
\midrule
Finetune & \ding{55} &4.51 & 1.59& 0.78 & 0.72& 3.49 & 1.71& 2.14 & 1.52& 1.34 & 0.47 & 0.69 & 0.54\\
Finetune LiT~\cite{zhai2022lit}& \ding{55} & 79.25 & 64.84& 38.23 & 20.00& 40.62 & 12.96& 29.74 & 12.05& 43.27 & 17.46& 32.85 & 17.17\\
Finetune CoOp~\cite{zhou2022learning}& \ding{55} &45.93 & 23.11 & 39.33 & 24.89& 36.01 & 14.18& 33.13 & 18.67& 21.24 & 6.29& 16.21 & 6.82\\
SimpleCIL~\cite{zhou2023revisiting}& \ding{55} & 82.13 & 75.58& 78.62 & 75.58& 87.89 & 81.65& 84.73 & 81.65& 52.06 & 40.13& 45.11 & 40.13\\
ZS-CLIP~\cite{radford2021learning}& \ding{55} &79.42 & 72.11& 74.95 & 72.11& 87.86 & 81.92& 84.75 & 81.92& 38.43 & 26.43 & 31.12 & 26.43\\
\midrule
CoOp~\cite{zhou2022learning}& \ding{51} &80.46 & 73.44& 77.68 & 73.06& 85.38 & 76.15& 81.74 & 76.35& 46.16 & 33.81& 40.40 & 34.47\\
iCaRL~\cite{rebuffi2017icarl}& \ding{51} &78.56 & 67.30 & 74.74 & 69.07 & 84.12 & 71.68& 78.86 & 70.64& 45.28 & 26.97 & 37.22 & 26.15\\
MEMO~\cite{zhou2022model}& \ding{51}&81.48 & 73.45 & 78.00 & 73.87 & 89.18 & 82.85& 86.50 & 83.08& 46.98 & 33.37& 41.62 & 34.67\\
L2P~\cite{wang2022learning} & \ding{51} & 79.83 & 72.14 & 76.16 & 72.32& 84.48 & 75.22& 85.04 & 80.56& 46.18 & 34.00& 43.90 & 39.57\\
DualPrompt~\cite{wang2022dualprompt} & \ding{51} &80.14 & 73.06& 77.25 & 73.82& 87.12 & 81.27 & 85.37 & 82.36 & 53.13 & 40.59& 45.84 & 40.37\\
\rowcolor{LightCyan}\name & \ding{51} & \bf83.57 & \bf77.28& \bf80.70 & \bf77.49& \bf90.04 & \bf84.73& \bf87.52 &\bf 84.74&\bf 55.28 &\bf 44.36& \bf49.64 &\bf 43.65 \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{figure*}[t]
\begin{center}
\subfigure[\small Aircraft Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/aircraft}}
\subfigure[CIFAR100 Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/cifar}}
\subfigure[Cars Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/cars}}\\
\subfigure[ImageNet-R Base0 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/inr}}
\subfigure[CUB Base0 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/cub}}
\subfigure[UCF Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/ucf}}\\
\subfigure[SUN Base0 Inc30]
{\includegraphics[width=.325\columnwidth]{./pics/sun}}
\subfigure[Food Base0 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/food}}
\subfigure[ObjectNet Base0 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/objectnet}}\\
\end{center}
\caption{
Incremental performance of different methods. We report the performance gap after the last incremental stage of \name and the runner-up method at the end of the line.
}
\label{figure:benchmark_full}
\end{figure*}
\begin{figure*}[t]
\begin{center}
\subfigure[\small Aircraft Base50 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/aircraftb50}}
\subfigure[CIFAR100 Base50 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/cifarb50}}
\subfigure[Cars Base50 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/carsb50}}\\
\subfigure[ImageNet-R Base100 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/inrb100}}
\subfigure[CUB Base100 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/cubb100}}
\subfigure[UCF Base50 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/ucfb50}}\\
\subfigure[SUN Base150 Inc30]
{\includegraphics[width=.325\columnwidth]{./pics/sunb150}}
\subfigure[Food Base50 Inc10]
{\includegraphics[width=.325\columnwidth]{./pics/foodb50}}
\subfigure[ObjectNet Base100 Inc20]
{\includegraphics[width=.325\columnwidth]{./pics/objectnetb100}}\\
\end{center}
\caption{
Incremental performance of different methods with large base classes. We report the performance gap after the last incremental stage of \name and the runner-up method at the end of the line.
}
\label{figure:benchmark_large_base}
\end{figure*}
\section{Experimental Details} \label{sec:supp_exp_detail}
This section provides detailed information about the experiments conducted, including the introduction of datasets, exemplar selection, and the methods compared in the paper.
\subsection{Dataset Introduction}
In our evaluation, we utilize nine datasets, which are introduced in Table~\ref{tab:supp_dataset} in the main paper. It is worth noting that some of these datasets have a larger number of classes, but we select a subset of classes for ease of data split and evaluation.
\noindent\textbf{Exemplar Selection:} As mentioned in the main paper, we follow the exemplar selection approach in \cite{rebuffi2017icarl, wu2019large, hou2019learning} to utilize herding algorithm~\cite{welling2009herding}. In addition, there are two typical methods \cite{zhou2023class} to store these exemplars in memory.
\begin{enumerate}
\item \textit{Fixed Memory Budget:} In this approach, a fixed memory budget of $K$ instances is allocated. Given the number of seen classes denoted as $|\mathcal{Y}_b|$, the model selects $\frac{K}{|\mathcal{Y}_b|}$ exemplars per class after each incremental stage.
\item \textit{Expandable Exemplar Set:} In this method, an expandable exemplar set is maintained as the data evolves. With the number of exemplars per class denoted as $k$, the model stores $|\mathcal{Y}_b|\times k$ exemplars in total after each incremental stage.
\end{enumerate}
We evaluate both protocols using these benchmark datasets in our experiments. Specifically, we employ the first policy for CIFAR100 and Food, keeping a total of 2,000 exemplars. Since these datasets consist of 100 classes, the average number of exemplars per class after the last incremental stage is 20.
We adopt the second policy for the other datasets and store 20 exemplars per class.
\begin{table*}
\caption{Introduction about benchmark datasets.
}
\label{tab:supp_dataset}
\centering
\begin{tabular}{lcccccc}
\toprule
Dataset & \# training instances & \# testing instances & \# Classes& Link \\ \midrule
CIFAR100 & 50,000 & 10,000 & 100 & \href{https://www.cs.toronto.edu/~kriz/cifar.html}{\nolinkurl{Link}}\\
CUB200 & 9,430 & 2,358 & 200 & \href{https://www.vision.caltech.edu/datasets/cub_200_2011/}{\nolinkurl{Link}}\\
ImageNet-R & 24,000 & 6,000 & 200 & \href{https://github.com/hendrycks/imagenet-r}{\nolinkurl{Link}}\\
ObjectNet & 26,509 & 6,628 & 200 & \href{https://objectnet.dev/}{\nolinkurl{Link}}\\
\midrule
Aircraft & 6,667 & 3,333 & 100 & \href{https://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft/archives/fgvc-aircraft-2013b.tar.gz}{\nolinkurl{Link}}\\
Cars & 4,135 & 4,083 & 100 & \href{http://ai.stanford.edu/~jkrause/car196/cars_train.tgz}{\nolinkurl{Link}}\\
UCF & 10,053 & 2,639 & 100 & \href{https://drive.google.com/file/d/10Jqome3vtUA2keJkNanAiFpgbyC9Hc2O/view?usp=sharing}{\nolinkurl{Link}}\\
SUN & 72,870 & 18,179 & 300 & \href{http://vision.princeton.edu/projects/2010/SUN/SUN397.tar.gz}{\nolinkurl{Link}}\\
Food & 79,998 & 20,012 & 100 & \href{https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/}{\nolinkurl{Link}}\\
\bottomrule
\end{tabular}
% }
\end{table*}
\subsection{Compared methods introduction}
This section provides an overview of the compared methods discussed in the main paper. These methods, listed in the order presented in Table~\ref{tab:supp_benchmark}, include:
\begin{itemize}
\item \textbf{Finetune:} This baseline method involves finetuning the pre-trained CLIP model using contrastive loss. No regularization terms are set, and no part of the model is frozen, allowing us to observe the forgetting phenomenon in sequential learning.
\item \textbf{Finetune LiT~\cite{zhai2022lit}:} Following LiT, which freezes the image encoder and only finetunes the text encoder, we design Finetune LiT with CIL. Similar to finetune, we sequentially tune the model with contrastive loss while the image encoder is frozen during optimization.
\item \textbf{Finetune CoOp~\cite{zhou2022learning}:} Following the CoOp method, this approach freezes both the image encoder and text encoder. It optimizes a learnable prompt tensor $\mathbf{t}$ (as in Eq.4) using contrastive loss without utilizing any historical data for rehearsal.
\item \textbf{SimpleCIL~\cite{zhou2023revisiting}:} This method relies on the pre-trained image encoder and does not involve the text encoder. The frozen image encoder extracts class centers (prototypes) for each new class, and a cosine classifier is utilized for classification. Since the model is not updated via backpropagation, it showcases the generalizability of the pre-trained vision encoder on downstream tasks.
\item \textbf{ZS-CLIP~\cite{radford2021learning}:} This baseline freezes the pre-trained CLIP model and predicts the logits of each incoming class using cosine similarity (Eq.~2). It serves as a reference for the performance of pre-trained CLIP on downstream tasks.
\item \textbf{CoOp (with exemplars):} This method combines the CoOp approach with exemplar rehearsal. During the learning of new classes, the model utilizes a combination of the current dataset and exemplar set to optimize the learnable prompt.
\item \textbf{iCaRL~\cite{rebuffi2017icarl}:} iCaRL is a typical class-incremental learning algorithm that employs knowledge distillation and exemplar replay to mitigate forgetting. It combines contrastive loss with distillation loss to learn new classes while retaining knowledge of old classes.
\item \textbf{MEMO~\cite{zhou2022model}:} As a state-of-the-art class-incremental learning algorithm based on network expansion, MEMO is modified to be compatible with the CLIP structure. The image and text encoders are expanded for new tasks, and the concatenated features are used for prediction based on cosine similarity.
\item \textbf{L2P~\cite{wang2022learning}:} L2P is a state-of-the-art class-incremental learning algorithm utilizing pre-trained vision transformers. In this case, the text encoder of CLIP is dropped, and a prompt pool (as in Eq.~3) is learned to adapt to evolving data. Another pre-trained image encoder is required to select the appropriate prompt during inference.
\item \textbf{DualPrompt~\cite{wang2022dualprompt}:} DualPrompt is an extension of L2P that incorporates two types of prompts: general prompts and expert prompts. It also relies on another pre-trained image encoder for prompt retrieval.
\end{itemize}
It is important to note that these methods are compared fairly, meaning they are initialized with the same pre-trained weights for incremental learning.
Since some compared methods are not designed with the CLIP encoder, we modify their backbone into pre-trained CLIP for a fair comparison.
We use the same number of exemplars for a fair comparison of the methods with exemplars.
\section{Broader Impacts} \label{sec:broader_impact}
In this work, we address the class-incremental learning problem with vision-language models, which is a fundamental challenge in machine learning. Our focus is on tackling the forgetting problem that arises when sequentially finetuning a vision-language model. We propose solutions to project and integrate features from multiple modalities for unified classification. Our research provides valuable insights for applications that struggle with managing the forgetting issue in large pre-trained vision-language models. However, there are still ample opportunities for further exploration in this field. Therefore, we aspire to stimulate discussions on class-incremental learning in real-world scenarios and encourage more research to develop practical models for this purpose.
We also acknowledge the ethical considerations associated with this technology. It is crucial to recognize that individuals expect learning systems to refrain from storing any personal information for future rehearsal. While there are risks involved in AI research of this nature, we believe that developing and demonstrating such techniques are vital for comprehending both the beneficial and potentially concerning applications of this technology. Our aim is to foster discussions regarding best practices and controls surrounding these methods, promoting responsible and ethical utilization of technology.
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{alayrac2022flamingo}
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr,
Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds,
et~al.
\newblock Flamingo: a visual language model for few-shot learning.
\newblock {\em NeurIPS}, 35:23716--23736, 2022.
\bibitem{aljundi2018memory}
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and
Tinne Tuytelaars.
\newblock Memory aware synapses: Learning what (not) to forget.
\newblock In {\em ECCV}, pages 139--154, 2018.
\bibitem{aljundi2019task}
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.
\newblock Task-free continual learning.
\newblock In {\em CVPR}, pages 11254--11263, 2019.
\bibitem{aljundi2019gradient}
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.
\newblock Gradient based sample selection for online continual learning.
\newblock In {\em NeurIPS}, pages 11816--11825, 2019.
\bibitem{antol2015vqa}
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C~Lawrence Zitnick, and Devi Parikh.
\newblock Vqa: Visual question answering.
\newblock In {\em ICCV}, pages 2425--2433, 2015.
\bibitem{barbu2019objectnet}
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan
Gutfreund, Josh Tenenbaum, and Boris Katz.
\newblock Objectnet: A large-scale bias-controlled dataset for pushing the
limits of object recognition models.
\newblock {\em NeurIPS}, 32, 2019.
\bibitem{bossard2014food}
Lukas Bossard, Matthieu Guillaumin, and Luc Van~Gool.
\newblock Food-101--mining discriminative components with random forests.
\newblock In {\em ECCV}, pages 446--461. Springer, 2014.
\bibitem{chaudhry2018riemannian}
Arslan Chaudhry, Puneet~K Dokania, Thalaiyasingam Ajanthan, and Philip~HS Torr.
\newblock Riemannian walk for incremental learning: Understanding forgetting
and intransigence.
\newblock In {\em ECCV}, pages 532--547, 2018.
\bibitem{chaudhry2018efficient}
Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed
Elhoseiny.
\newblock Efficient lifelong learning with a-gem.
\newblock In {\em ICLR}, 2018.
\bibitem{chen2023plot}
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun
Zhang.
\newblock Plot: Prompt learning with optimal transport for vision-language
models.
\newblock In {\em ICLR}, 2023.
\bibitem{chenadaptformer}
Shoufa Chen, GE Chongjian, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and
Ping Luo.
\newblock Adaptformer: Adapting vision transformers for scalable visual
recognition.
\newblock In {\em NeurIPS}, 2022.
\bibitem{deng2009imagenet}
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
\newblock Imagenet: A large-scale hierarchical image database.
\newblock In {\em CVPR}, pages 248--255, 2009.
\bibitem{dosovitskiy2020image}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et~al.
\newblock An image is worth 16x16 words: Transformers for image recognition at
scale.
\newblock In {\em ICLR}, 2020.
\bibitem{douillard2020podnet}
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo
Valle.
\newblock Podnet: Pooled outputs distillation for small-tasks incremental
learning.
\newblock In {\em ECCV}, pages 86--102, 2020.
\bibitem{douillard2022dytox}
Arthur Douillard, Alexandre Ram{\'e}, Guillaume Couairon, and Matthieu Cord.
\newblock Dytox: Transformers for continual learning with dynamic token
expansion.
\newblock In {\em CVPR}, pages 9285--9295, 2022.
\bibitem{french1999catastrophic}
Robert~M French.
\newblock Catastrophic forgetting in connectionist networks.
\newblock {\em Trends in cognitive sciences}, 3(4):128--135, 1999.
\bibitem{gao2021clip}
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang,
Hongsheng Li, and Yu Qiao.
\newblock Clip-adapter: Better vision-language models with feature adapters.
\newblock {\em arXiv preprint arXiv:2110.04544}, 2021.
\bibitem{geng2020recent}
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen.
\newblock Recent advances in open set recognition: A survey.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
43(10):3614--3631, 2020.
\bibitem{gomes2017survey}
Heitor~Murilo Gomes, Jean~Paul Barddal, Fabr{\'\i}cio Enembreck, and Albert
Bifet.
\newblock A survey on ensemble learning for data stream classification.
\newblock {\em CSUR}, 50(2):1--36, 2017.
\bibitem{han2021pre}
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu,
Yuan Yao, Ao Zhang, Liang Zhang, et~al.
\newblock Pre-trained models: Past, present and future.
\newblock {\em AI Open}, 2:225--250, 2021.
\bibitem{hou2018lifelong}
Saihui Hou, Xinyu Pan, Chen~Change Loy, Zilei Wang, and Dahua Lin.
\newblock Lifelong learning via progressive distillation and retrospection.
\newblock In {\em ECCV}, pages 437--452, 2018.
\bibitem{hou2019learning}
Saihui Hou, Xinyu Pan, Chen~Change Loy, Zilei Wang, and Dahua Lin.
\newblock Learning a unified classifier incrementally via rebalancing.
\newblock In {\em CVPR}, pages 831--839, 2019.
\bibitem{houlsby2019parameter}
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin
De~Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
\newblock Parameter-efficient transfer learning for nlp.
\newblock In {\em ICML}, pages 2790--2799, 2019.
\bibitem{ilharco_gabriel_2021_5143773}
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas
Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John
Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.
\newblock Openclip, July 2021.
\bibitem{jia2021scaling}
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le,
Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
\newblock Scaling up visual and vision-language representation learning with
noisy text supervision.
\newblock In {\em ICML}, pages 4904--4916. PMLR, 2021.
\bibitem{jia2022visual}
Menglin Jia, Luming Tang, Bor{-}Chun Chen, Claire Cardie, Serge~J. Belongie,
Bharath Hariharan, and Ser{-}Nam Lim.
\newblock Visual prompt tuning.
\newblock In {\em ECCV}, pages 709--727. Springer, 2022.
\bibitem{kirkpatrick2017overcoming}
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume
Desjardins, Andrei~A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
Grabska-Barwinska, et~al.
\newblock Overcoming catastrophic forgetting in neural networks.
\newblock {\em PNAS}, 114(13):3521--3526, 2017.
\bibitem{krause20133d}
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
\newblock 3d object representations for fine-grained categorization.
\newblock In {\em ICCV Workshop}, pages 554--561, 2013.
\bibitem{krizhevsky2009learning}
Alex Krizhevsky, Geoffrey Hinton, et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock Technical report, 2009.
\bibitem{lee2023pre}
Kuan-Ying Lee, Yuanyi Zhong, and Yu-Xiong Wang.
\newblock Do pre-trained models benefit equally in continual learning?
\newblock In {\em WACV}, pages 6485--6493, 2023.
\bibitem{li2023blip}
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
\newblock Blip-2: Bootstrapping language-image pre-training with frozen image
encoders and large language models.
\newblock {\em arXiv preprint arXiv:2301.12597}, 2023.
\bibitem{li2021prefix}
Xiang~Lisa Li and Percy Liang.
\newblock Prefix-tuning: Optimizing continuous prompts for generation.
\newblock In {\em ACL}, pages 4582--4597, 2021.
\bibitem{li2016learning}
Zhizhong Li and Derek Hoiem.
\newblock Learning without forgetting.
\newblock In {\em ECCV}, pages 614--629. Springer, 2016.
\bibitem{lianscaling}
Dongze Lian, Zhou Daquan, Jiashi Feng, and Xinchao Wang.
\newblock Scaling \& shifting your features: A new baseline for efficient model
tuning.
\newblock In {\em NeurIPS}, 2022.
\bibitem{liang2022mind}
Victor~Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James~Y Zou.
\newblock Mind the gap: Understanding the modality gap in multi-modal
contrastive representation learning.
\newblock {\em NeurIPS}, 35:17612--17625, 2022.
\bibitem{lin2017structured}
Zhouhan Lin, Minwei Feng, Cicero Nogueira~dos Santos, Mo Yu, Bing Xiang, Bowen
Zhou, and Yoshua Bengio.
\newblock A structured self-attentive sentence embedding.
\newblock {\em arXiv preprint arXiv:1703.03130}, 2017.
\bibitem{liu2021rmm}
Yaoyao Liu, Bernt Schiele, and Qianru Sun.
\newblock Rmm: Reinforced memory management for class-incremental learning.
\newblock {\em NeurIPS}, 34, 2021.
\bibitem{liu2020mnemonics}
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun.
\newblock Mnemonics training: Multi-class incremental learning without
forgetting.
\newblock In {\em CVPR}, pages 12245--12254, 2020.
\bibitem{lu2022prompt}
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian.
\newblock Prompt distribution learning.
\newblock In {\em CVPR}, pages 5206--5215, 2022.
\bibitem{luo2023class}
Zilin Luo, Yaoyao Liu, Bernt Schiele, and Qianru Sun.
\newblock Class-incremental exemplar compression for class-incremental
learning.
\newblock {\em arXiv preprint arXiv:2303.14042}, 2023.
\bibitem{maji2013fine}
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.
\newblock Fine-grained visual classification of aircraft.
\newblock {\em arXiv preprint arXiv:1306.5151}, 2013.
\bibitem{mao2022doubly}
Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin
Wang, and Carl Vondrick.
\newblock Doubly right object recognition: A why prompt for visual rationales.
\newblock {\em arXiv preprint arXiv:2212.06202}, 2022.
\bibitem{mu2022slip}
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.
\newblock Slip: Self-supervision meets language-image pre-training.
\newblock In {\em ECCV}, pages 529--544. Springer, 2022.
\bibitem{paszke2019pytorch}
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et~al.
\newblock Pytorch: An imperative style, high-performance deep learning library.
\newblock In {\em NeurIPS}, pages 8026--8037, 2019.
\bibitem{plummer2015flickr30k}
Bryan~A Plummer, Liwei Wang, Chris~M Cervantes, Juan~C Caicedo, Julia
Hockenmaier, and Svetlana Lazebnik.
\newblock Flickr30k entities: Collecting region-to-phrase correspondences for
richer image-to-sentence models.
\newblock In {\em ICCV}, pages 2641--2649, 2015.
\bibitem{radford2021learning}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et~al.
\newblock Learning transferable visual models from natural language
supervision.
\newblock In {\em ICML}, pages 8748--8763. PMLR, 2021.
\bibitem{rebuffi2017icarl}
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph~H
Lampert.
\newblock icarl: Incremental classifier and representation learning.
\newblock In {\em CVPR}, pages 2001--2010, 2017.
\bibitem{schuhmann2021laion}
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk,
Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran
Komatsuzaki.
\newblock Laion-400m: Open dataset of clip-filtered 400 million image-text
pairs.
\newblock {\em arXiv preprint arXiv:2111.02114}, 2021.
\bibitem{seale2022coda}
James Seale~Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla,
Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira.
\newblock Coda-prompt: Continual decomposed attention-based prompting for
rehearsal-free continual learning.
\newblock {\em arXiv e-prints}, pages arXiv--2211, 2022.
\bibitem{shi2022mimicking}
Yujun Shi, Kuangqi Zhou, Jian Liang, Zihang Jiang, Jiashi Feng, Philip~HS Torr,
Song Bai, and Vincent~YF Tan.
\newblock Mimicking the oracle: An initial phase decorrelation approach for
class incremental learning.
\newblock In {\em CVPR}, pages 16722--16731, 2022.
\bibitem{snell2017prototypical}
Jake Snell, Kevin Swersky, and Richard Zemel.
\newblock Prototypical networks for few-shot learning.
\newblock In {\em NIPS}, pages 4080--4090, 2017.
\bibitem{soomro2012ucf101}
Khurram Soomro, Amir~Roshan Zamir, and Mubarak Shah.
\newblock Ucf101: A dataset of 101 human actions classes from videos in the
wild.
\newblock {\em arXiv preprint arXiv:1212.0402}, 2012.
\bibitem{sunexploring}
Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan.
\newblock Exploring example influence in continual learning.
\newblock In {\em NeurIPS}, 2022.
\bibitem{tschannen2022image}
Michael Tschannen, Basil Mustafa, and Neil Houlsby.
\newblock Image-and-language understanding from pixels only.
\newblock {\em arXiv preprint arXiv:2212.08045}, 2022.
\bibitem{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock In {\em NIPS}, pages 5998--6008, 2017.
\bibitem{vinyals2015show}
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.
\newblock Show and tell: A neural image caption generator.
\newblock In {\em CVPR}, pages 3156--3164, 2015.
\bibitem{WahCUB2002011}
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.
\newblock {The Caltech-UCSD Birds-200-2011 Dataset}.
\newblock Technical Report CNS-TR-2011-001, California Institute of Technology,
2011.
\bibitem{wang2023beef}
Fu-Yun Wang, Da-Wei Zhou, Liu Liu, Han-Jia Ye, Yatao Bian, De-Chuan Zhan, and
Peilin Zhao.
\newblock {BEEF}: Bi-compatible class-incremental learning via energy-based
expansion and fusion.
\newblock In {\em ICLR}, 2023.
\bibitem{wang2022foster}
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan.
\newblock Foster: Feature boosting and compression for class-incremental
learning.
\newblock In {\em ECCV}, pages 398--414, 2022.
\bibitem{wang2022image}
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti
Aggarwal, Owais~Khan Mohammed, Saksham Singhal, Subhojit Som, et~al.
\newblock Image as a foreign language: Beit pretraining for all vision and
vision-language tasks.
\newblock {\em arXiv preprint arXiv:2208.10442}, 2022.
\bibitem{wang2022s}
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong.
\newblock S-prompts learning with pre-trained transformers: An occam’s razor
for domain incremental learning.
\newblock In {\em NeurIPS}, 2022.
\bibitem{wang2022sparcl}
Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren,
Stratis Ioannidis, Yanzhi Wang, and Jennifer Dy.
\newblock Sparcl: Sparse continual learning on the edge.
\newblock In {\em NeurIPS}, 2022.
\bibitem{wang2022dualprompt}
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee,
Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et~al.
\newblock Dualprompt: Complementary prompting for rehearsal-free continual
learning.
\newblock In {\em ECCV}, pages 631--648, 2022.
\bibitem{wang2022learning}
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren,
Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.
\newblock Learning to prompt for continual learning.
\newblock In {\em CVPR}, pages 139--149, 2022.
\bibitem{welling2009herding}
Max Welling.
\newblock Herding dynamical weights to learn.
\newblock In {\em ICML}, pages 1121--1128, 2009.
\bibitem{wortsman2022robust}
Mitchell Wortsman, Gabriel Ilharco, Jong~Wook Kim, Mike Li, Simon Kornblith,
Rebecca Roelofs, Raphael~Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi,
Hongseok Namkoong, et~al.
\newblock Robust fine-tuning of zero-shot models.
\newblock In {\em CVPR}, pages 7959--7971, 2022.
\bibitem{wu2019large}
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and
Yun Fu.
\newblock Large scale incremental learning.
\newblock In {\em CVPR}, pages 374--382, 2019.
\bibitem{xiao2010sun}
Jianxiong Xiao, James Hays, Krista~A Ehinger, Aude Oliva, and Antonio Torralba.
\newblock Sun database: Large-scale scene recognition from abbey to zoo.
\newblock In {\em CVPR}, pages 3485--3492. IEEE, 2010.
\bibitem{yan2021dynamically}
Shipeng Yan, Jiangwei Xie, and Xuming He.
\newblock Der: Dynamically expandable representation for class incremental
learning.
\newblock In {\em CVPR}, pages 3014--3023, 2021.
\bibitem{yu2022coca}
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and
Yonghui Wu.
\newblock Coca: Contrastive captioners are image-text foundation models.
\newblock {\em arXiv preprint arXiv:2205.01917}, 2022.
\bibitem{yu2020semantic}
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei
Cheng, Shangling Jui, and Joost van~de Weijer.
\newblock Semantic drift compensation for class-incremental learning.
\newblock In {\em CVPR}, pages 6982--6991, 2020.
\bibitem{yu2022task}
Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang.
\newblock Task residual for tuning vision-language models.
\newblock {\em arXiv preprint arXiv:2211.10277}, 2022.
\bibitem{yuan2021florence}
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao,
Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et~al.
\newblock Florence: A new foundation model for computer vision.
\newblock {\em arXiv preprint arXiv:2111.11432}, 2021.
\bibitem{zenke2017continual}
Friedemann Zenke, Ben Poole, and Surya Ganguli.
\newblock Continual learning through synaptic intelligence.
\newblock In {\em ICML}, pages 3987--3995, 2017.
\bibitem{zhai2022lit}
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers,
Alexander Kolesnikov, and Lucas Beyer.
\newblock Lit: Zero-shot transfer with locked-image text tuning.
\newblock In {\em CVPR}, pages 18123--18133, 2022.
\bibitem{zhang2022tip}
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu
Qiao, and Hongsheng Li.
\newblock Tip-adapter: Training-free adaption of clip for few-shot
classification.
\newblock In {\em ECCV}, pages 493--510. Springer, 2022.
\bibitem{zhangsimple}
Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin~Sean Lim,
and Alvin Jia.
\newblock A simple but strong baseline for online continual learning: Repeated
augmented rehearsal.
\newblock In {\em NeurIPS}, 2022.
\bibitem{zhao2020maintaining}
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia.
\newblock Maintaining discrimination and fairness in class incremental
learning.
\newblock In {\em CVPR}, pages 13208--13217, 2020.
\bibitem{zhao2021mgsvf}
Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, and Xi Li.
\newblock Mgsvf: Multi-grained slow vs. fast framework for few-shot
class-incremental learning.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
2021.
\bibitem{zhou2023pycil}
Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, and De-Chuan Zhan.
\newblock Pycil: a python toolbox for class-incremental learning.
\newblock {\em SCIENCE CHINA Information Sciences}, 66(9):197101--, 2023.
\bibitem{zhou2023class}
Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei
Liu.
\newblock Deep class-incremental learning: A survey.
\newblock {\em arXiv preprint arXiv:2302.03648}, 2023.
\bibitem{zhou2022model}
Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan.
\newblock A model or 603 exemplars: Towards memory-efficient class-incremental
learning.
\newblock In {\em ICLR}, 2023.
\bibitem{zhou2023revisiting}
Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu.
\newblock Revisiting class-incremental learning with pre-trained models:
Generalizability and adaptivity are all you need.
\newblock {\em arXiv preprint arXiv:2303.07338}, 2023.
\bibitem{zhou2022conditional}
Kaiyang Zhou, Jingkang Yang, Chen~Change Loy, and Ziwei Liu.
\newblock Conditional prompt learning for vision-language models.
\newblock In {\em CVPR}, pages 16816--16825, 2022.
\bibitem{zhou2022learning}
Kaiyang Zhou, Jingkang Yang, Chen~Change Loy, and Ziwei Liu.
\newblock Learning to prompt for vision-language models.
\newblock {\em International Journal of Computer Vision}, 130(9):2337--2348,
2022.
\end{thebibliography}