Rethinking Continual Learning with Progressive Neural Collapse
Extracted source text · arXiv 2505.24254
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
% \usepackage{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{ragged2e}
\usepackage{tabularray}
\usepackage{placeins}
\usepackage{amsmath,amssymb,amsthm,bm}
\renewcommand{\theequation}{\arabic{equation}}
\newtheorem{definition}{Definition}
\newtheorem{proposition}{Proposition}
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\usepackage{hyperref}
\usepackage{cleveref}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{placeins}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{caption}
\usepackage{geometry}
\usepackage{wrapfig}
\expandafter\def\expandafter\normalsize\expandafter{%
\normalsize
\setlength\abovedisplayskip{1.5pt}
\setlength\belowdisplayskip{1.5pt}
\setlength\abovedisplayshortskip{1.5pt}
\setlength\belowdisplayshortskip{1.5pt}
}
\newcommand{\sen}[1]{{\color{red}(Sen: #1)}}
\title{Rethinking Continual Learning with Progressive Neural Collapse}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{
Zheng Wang$^1$ \quad Wanhao Yu$^2$ \quad Li Yang$^2$ \quad Sen Lin$^1$ \\
$^1$Department of Computer Science at University of Houston\\
$^2$Department of Computer Science at University of North Carolina at Charlotte\\
\texttt{\{zwang214, slin50\}@cougarnet.uh.edu}, \texttt{\{wyu6, lyang50\}@charlotte.edu}
}
\begin{document}
\maketitle
\begin{abstract}
Continual Learning (CL) seeks to build an agent that can continuously learn a sequence of tasks, where a key challenge, namely Catastrophic Forgetting, persists due to the potential knowledge interference among different tasks. On the other hand, deep neural networks (DNNs) are shown to converge to a terminal state termed Neural Collapse during training, where all class prototypes geometrically form a static simplex equiangular tight frame (ETF). These maximally and equally separated class prototypes make the ETF an ideal target for model learning in CL to mitigate knowledge interference. Thus inspired, several studies have emerged very recently to leverage a fixed global ETF in CL, which however suffers from key drawbacks, such as \emph{impracticability} and \emph{limited performance}.
%1) \emph{Impracticability:} The predefined ETF requires the knowledge of total class numbers for the entire CL; 2) \emph{Limited performance:} The small distance between different class targets in the predefined ETF makes class discrimination extremely difficult, significantly impeding the learning performance.
To address these challenges and fully unlock the potential of ETF in CL, we propose \textbf{Progressive Neural Collapse (ProNC)}, a novel framework that completely removes the need of a fixed global ETF in CL. Specifically, ProNC progressively expands the ETF target in a principled way by adding new class prototypes as vertices for new tasks, ensuring maximal separability across all encountered classes with minimal shifts from the previous ETF. We next develop a new CL framework by
plugging ProNC into commonly used CL algorithm designs, where distillation is further leveraged to balance between target shifting for old classes and target aligning for new classes. Extensive experiments show that our approach significantly outperforms related baselines while maintaining superior flexibility, simplicity, and efficiency.
%Continual Learning (CL) seeks to emulate human lifelong learning capabilities by enabling models to adapt to new tasks while retaining previously acquired knowledge. A persistent challenge in CL is Catastrophic Forgetting, where deep neural networks (DNNs) degrade performance on prior tasks when incrementally learning new classes. While recent methods mitigate forgetting by enforcing a static simplex equiangular tight frame (ETF) classifier---inspired by the geometric optimality of Neural Collapse (NC)---they rely on impractical assumptions, such as predefined knowledge of total class counts and restrictive warm-start initialization. These approaches further compromise intratask discrimination, as new classes disrupt the fixed ETF geometric symmetry. To address these limitations, we propose \textbf{Progressive Neural Collapse (ProNC)}, a novel framework that dynamically adapts the ETF structure during CL while synchronizing with exemplar rehearsal to preserve historical knowledge. Unlike static ETF-based methods, ProNC incrementally expands the ETF geometry by adding new class prototypes as vertices, ensuring maximal separability and angular uniformity across all encountered classes. This eliminates the need for prior class-count knowledge or warm-start constraints, enabling robust learning under arbitrary task sequences. Crucially, ProNC leverages exemplars to stabilize feature distributions and guide the geometric alignment of new prototypes with the evolving ETF, ensuring backward compatibility with historical tasks. By unifying dynamic geometric adaptation with exemplar-based rehearsal, ProNC preserves the core principles of NC, such as within-class variability collapse and classifier-prototype equivalence, throughout incremental training. Our work bridges the gap between NC's theoretical optimality and real-world CL's practical demands, offering a scalable, theoretically grounded solution to catastrophic forgetting. Experimental results demonstrate that ProNC achieves superior performance across all evaluations on the CIFAR-10, CIFAR-100, and Tiny ImageNet benchmark datasets, outperforming existing methods in comparative analyses.
\end{abstract}
\section{Introduction}
Continual Learning (CL) has gained significant attention in recent years, aiming to build an agent to mimic the extraordinary human abilities to learn different tasks in a lifelong manner.
A central challenge here is \textit{\textbf{Catastrophic Forgetting}}, i.e., deep neural networks (DNNs) exhibit a pronounced tendency to lose previously acquired knowledge when trained on new tasks \cite{mccloskey1989catastrophic}. Within the spectrum of CL scenarios, class-incremental learning (CIL) presents the most formidable setting \cite{masana2022class},
%requiring the construction of a unified classifier capable of distinguishing all observed classes cumulatively \cite{masana2022class}.
where the model must not only address the current task by differentiating intra-task classes but also retain the knowledge of prior tasks by distinguishing historical classes from newly introduced ones. Nevertheless, achieving this dual objective remains particularly challenging, as evidenced by
the suboptimal performance of existing CL methodologies.
%The human brain exhibits extraordinary lifelong learning capabilities, enabling humans to acquire new knowledge while retaining previously learned information without experiencing forgetting or interference. In contrast, artificial intelligence (AI) systems face significant challenges in achieving comparable adaptability. While deep learning models have achieved remarkable success in static, closed-label environments \cite{lecun2015deep}, they struggle to incrementally learn new classes without degrading performance on previously learned ones \cite{polikar2001learn++, cauwenberghs2000incremental}. A central challenge in \textbf{\textit{Continual Learning(CL)}} is \textit{\textbf{Catastrophic Forgetting}}, a phenomenon where in deep neural networks (DNNs) exhibit a pronounced tendency to lose previously acquired knowledge when trained on new tasks \cite{mccloskey1989catastrophic}. Within the spectrum of CL scenarios, class-incremental learning (CIL) presents the most formidable setting, requiring the construction of a unified classifier capable of distinguishing all observed classes cumulatively \cite{masana2022class}. The model must not only address the current task by differentiating intra-task classes but also retain the knowledge of prior tasks by distinguishing historical classes from newly introduced ones. Nevertheless, achieving this dual objective remains particularly challenging, as evidenced by the suboptimal performance of existing CL methodologies. A fundamental limitation lies in the inherent design of conventional linear classifiers in DNNs, which lack a principled mechanism to ensure discriminative separation across all classes in evolving feature spaces. Furthermore, their susceptibility to feature distribution shifts during incremental training raises critical questions about their theoretical viability for robust CIL.
Recent studies (e.g., \cite{papyan2020prevalence}) have identified a compelling empirical phenomenon in DNN training termed \textbf{\textit{Neural Collapse (NC)}}. During the terminal phase of training—when the training error asymptotically approaches zero—the last-layer features of samples within the same class converge to their class-specific mean, while the means of all classes align with their corresponding classifier prototypes. These prototypes further collapse geometrically to form the vertices of a \textbf{\textit{Simplex Equiangular Tight Frame (ETF)}}, a maximally symmetric and separable configuration. This phenomenon results in four critical properties: (1) \textit{Feature Collapse}: Features from samples within the same class converge to their class-specific mean, effectively eliminating within-class variability. (2) \textit{ETF Geometric Alignment}: The class-specific means for all classes align with the vertices of an ETF. (3) \textit{Classifier-Prototype Equivalence}: These class means further align with the weights of the linear classifier.
%forming a shared structure that mirrors the same simplex ETF geometry.
(4) \textit{Decision Simplification}: %Collectively, these properties render the traditional linear classifier redundant. Instead,
Predictions reduce to a nearest-class-mean rule, where test samples are assigned to the class whose feature mean is the closest.
%A key idea emerging in recent research is that the elegant Neural Collapse (NC) properties of deep neural network (DNN) training naturally characterize an ideal model for continual learning (CL).
%A promising direction in continual learning (CL) research involves leveraging the Neural Collapse (NC) phenomenon—a set of geometric properties describing the terminal phase of deep neural network (DNN) training—to design robust CL algorithms.
A key idea emerging here is that the elegant NC properties of DNN training naturally characterize an ideal
model for CL: All classes seen so far, including the old classes for old tasks and the new classes for the current task,
will have nearly-zero within-class variability, and their corresponding class-means are equally and perfectly separated.
Several studies \cite{NCT1, dang2024memory, yang2023neural} have recently emerged to leverage NC and predefine a fixed global ETF as the model training target in CL, which however suffer
from significant limitations:
1) Setting the number of vertices in the predefined ETF requires the knowledge of total class number encountered during CL before learning the first task, which is clearly not practical. 2) When the total class number is very large, the distance between any two vertices in the predefined ETF will be very small. Pushing class means towards these closely located vertices will unnecessarily hinder class discrimination, especially in early stages of CL when the number of seen classes that the model has to discriminate among is much smaller and the distance between these class means is larger. 3) NC posits that ETFs emerge naturally from feature convergence during training. Predefining the ETF contradicts this emergent behavior, as random initialization of the ETF risks geometric misalignment between learned features and the imposed topology.
%This mismatch undermines NC’s foundational premise of organic prototype separation.
%1) Current methods enforce a static, predefined ETF (Fig.~\ref{fig:structure}b) where the number of vertices equals the cumulative class count encountered during incremental learning. This assumes a priori knowledge of the total class cardinality—an impractical requirement for real-world open-world CL scenarios. 2) The static ETF initialization imposes a fixed geometric configuration that requires all incremental tasks to inherit the predefined final structure. This framework fundamentally conflicts with the natural evolution of class prototypes during early learning phases. Specifically, initial tasks should intrinsically develop an ETF geometry with fewer vertices corresponding to the current class cardinality and wider angular separation between prototypes compared to the ultimate target configuration. The discrepancy between the imposed terminal geometry and the transient optimal structure during incremental training creates a fundamental misalignment. Consequently, the model is compelled to optimize toward an artificial target state (the final ETF) rather than the naturally emerging geometric configuration appropriate for each learning phase. This artificial optimization landscape hinders the model's ability to attain the authentic terminal phase characteristics predicted by Neural Collapse, as the training trajectory is constrained by the predetermined geometric constraints rather than emerging organically from feature-space evolution.3) NC posits that ETFs emerge naturally from class-mean convergence during training. Predefining the ETF contradicts this emergent behavior, as random initialization of the ETF structure risks geometric misalignment between learned features and the imposed topology. This mismatch undermines NC’s foundational premise of organic prototype separation.
To handle these limitations, a key insight is that the number of vertices in the target ETF for model training should match the total number of classes seen so far to achieve maximal across-class separation anytime in CL. Thus inspired, instead of relying on a predefined ETF with critical design flaws, we seek to develop a novel approach that can appropriately adapt the ETF target for CL in order to fully unleash the potential of NC. To this end, our contributions can be summarized as follows:
\emph{1) A principled approach for ETF expansion.} By rethinking the objective of CL in classification as facilitating progressive NC with a growing ETF after learning each new task, we propose a novel approach, namely ProNC, to dynamically adjust the target ETF during CL. Specifically, ProNC first extracts the initial ETF target that emerges from first task training, and then expands the ETF target by adding new class prototypes as vertices prior to new task learning, to ensure maximal separability across all encountered classes without causing dramatic shifts from the previous ETF. In principle, ProNC can be broadly applied in CL frameworks as a new type of feature regularization.
\emph{2) A simple and flexible framework for CL based on ProNC.} We next develop a new CL framework by plugging ProNC into commonly used CL algorithm designs. In particular, building upon the standard cross-entropy loss for new task learning, we introduce two additional losses, i.e., the alignment loss and the distillation loss. The former seeks to push the learned class features towards the corresponding target ETF provided by ProNC, whereas the later follows a standard idea of knowledge distillation to mitigate feature shifts for old classes. Guided by the target ETF, a nearest-class-mean classifier will be used to replace the standard linear classifier. Such a CL framework is very simple, without introducing any new complex designs, and flexible, where different components such as distillation can be replaced by other advanced designs.
\emph{3) Comprehensive experiments for performance evaluation.} We perform comprehensive experiments on multiple standard benchmarks for both class incremental learning and task incremental learning, to evaluate the effectiveness of our CL approach compared with related baseline approaches. It can be shown that our approach significantly outperforms the baselines especially on larger datasets and also enjoys much less forgetting, without introducing more computation costs. In particular, extensive ablation studies are conducted to justify the benefits of ProNC in terms of maximizing feature separation among different classes and minimizing feature shifts across CL.
% Our approach outperforms the
% best baseline approach by 114.40\% with large-scale dataset and limited memory size.\sen{may need to add more to emphasize the performance.}
%Specifically, rather than relying on a fixed predefined ETF, we propose a progressive method, \textbf{\textit{Progressive Neural Collapse}}, that dynamically adapts during CL and synergizes with incremental task learning. As shown in Figure \ref{fig:structure}a, the ETF is not predefined but trained incrementally, starting from the base task. As training data grows, the ETF expands by adding new vertices to accommodate novel classes while preserving geometric optimality. Our method achieves state-of-the-art (SoTA) performance in both class-incremental and task-incremental learning scenarios.
\section{Related Work}
%Continual learning methods seek to enable neural networks to learn new tasks sequentially without catastrophic forgetting \cite{kirkpatrick2018reply}. In the class-incremental learning (CIL) setting, the model must accommodate new classes over time while maintaining a unified classifier for all encountered classes, a challenging scenario due to the absence of old class data during new class training \cite{rebuffi2017icarl}.
%\sen{this section needs to be rewritten. you can refer to our previous papers or some other recent studies. Some studies are in the wrong category. NC-based approaches also have nothing to do with architectures. This is basic knowledge you need to have.}
\textbf{Continual Learning.} In general, existing CL approaches can be divided into several categories.
\emph{1) Regularization-based approaches} seek to regularize the change on model parameters that are important to previous tasks \cite{zenke2017continual,chaudhry2018riemannian}. For instance, EWC \cite{kirkpatrick2017overcoming} penalized updating important weights characterized based on Fisher Information matrix. MAS \cite{aljundi2018memory} characterized the weight importance based on the sensitivity of model updates if this weight is changed. \cite{liu2022continual} proposed an approach that recursively modified the gradient update to minimize forgetting. The Bayesian framework has also been substantially investigated to implicitly penalize the parameter changes \cite{lee2017overcoming,nguyen2017variational,zeno2021task}.
\emph{2) Memory-based approaches} store information for previous tasks, which have shown very strong performance and
can be further divided into two categories, i,e, rehearsal-based and orthogonal-projection based approaches. Rehearsal-based approaches \cite{riemer2018learning,chaudhry2018efficient,rolnick2019experience} store a subset of data for previous tasks and replay them together with current data for new task learning.
Some studies focused on how to select and manage the replay data to achieve better performance and efficiency, such as the use of reservoir sampling \cite{chrysakis2020online}, %information-theory based memory selection \cite{sun2022information},
coreset-based memory selection \cite{borsos2020coresets,tiwari2022gcr}, and data compression \cite{wang2022memory}. Some other studies investigated how to utilize the replay data, such as imposing constraints on gradient update \cite{chaudhry2018efficient}, combining with knowledge distillation \cite{rebuffi2017icarl,buzzega2020dark,hou2019learning,gao2022r},
%leveraging the idea of meta-learning \cite{riemer2018learning,javed2019meta},
contrastive learning based approaches \cite{cha2021co2l,wenprovable}. The use of generative data has also been explored in CL for rehearsal-based approaches \cite{shin2017continual}.
Instead of storing data samples,
orthogonal-projection based approaches \cite{farajtabar2020orthogonal,wang2021training,saha2021gradient,lin2022trgp,lin2022beyond} store gradient or basis information to reconstruct the input subspaces of old tasks, so as to modify the model parameters only along the direction orthogonal to these subspaces.
\emph{3) Architecture-based approaches} freeze the important parameters for old tasks, train the remaining parameters to learn new tasks and expand the model if needed. Notably, PNN \cite{rusu2016progressive} preserved the weights for previous tasks and progressively expanded the network architecture to learn new tasks.
LwF \cite{li2017learning} split the model parameters into two parts, where task-shared parameters are used to extract common knowledge and task-specific parameters are expanded for new tasks. Some studies \cite{yoon2017lifelong,hung2019compacting,yang2021grown} combined the strategies of weight selection, model pruning and expansion.
\emph{4) Pretrained-model based approaches} leverage the powerful pretrained models and seek to efficiently adapt the models to continuously learn new tasks. To enable lightweight finetuning of the pretrained models in CL, prompt-based approaches have gained much attention by constructing a pool of prompts, selecting and optimizing the appropriate prompts for each task \cite{wang2022learning,wang2022dualprompt,smith2023coda}. Some studies focused on how to leverage the strong representation capability of pretrained models. For example, \cite{zhou2025revisiting} proposed a simple CL baseline that directly leverages the prototype features of the pretrained models without any training on new tasks. \cite{mcdonnell2023ranpac} proposed a frozen random projection layer to capture the interactions among features.
\textbf{Neural Collapse.}
The NC phenomenon during the terminal state of DNN training was first discovered in \cite{papyan2020prevalence}, which has further motivated a lot of studies on understanding NC. For example,
NC has been investigated under different settings, e.g., imbalanced learning \cite{yang2022inducing, xie2023neural}, and also been applied in different domains, e.g., semantic segmentation \cite{shen2025in2nect, xie2025hard} and language models \cite{wu2024linguistic, zhu2024neural}. Very recently, several studies have emerged to leverage NC to facilitate better CL algorithm designs. \cite{yang2023neural} first proposed to use a fixed global ETF target for feature-classifier alignment in few-shot CL with an ETF classifier, whereas \cite{NCT1} applied the same idea to more general CL setups. \cite{dang2024memory} further integrated this idea with contrastive learning based CL. However, as mentioned earlier, the reliance on a fixed global ETF suffers from critical drawbacks, which we aim to address in this work.
%try to regularize the model with former data and control the optimization direction. Adam-NSCL \cite{wang2021training} proposes to sequentially optimize network parameters by projecting the candidate parameter update into the approximate null space of all previous tasks. OWM \cite{zeng2019continual} only allows weight modification in the direction orthogonal to the subspace spanned by all previously learned inputs. Knowledge distillation, which transfers knowledge from old tasks to new ones, is also a kind of regularization. LwF \cite{li2017learning} aligns logit distributions, while UCIR \cite{hou2019learning} and relational distillation \cite{gao2022r} preserve feature relationships. These methods balance stability and plasticity but face challenges in logit bias and feature drift.
%\textbf{Rehearsal-based methods} By storing a subset of past examples (often called exemplars) and mixing them with new data during training to maintain knowledge of past tasks. iCaRL \cite{rebuffi2017icarl} combined exemplar replay with knowledge distillation to incrementally build a unified classifier, establishing a strong baseline for CIL. Subsequent work refined how exemplars are selected and used, for example, \cite{chaudhry2018riemannian} suggests selecting the samples with high prediction entropy and near the decision boundary. The model will perform better by training with these outliers. Generative replay shows the potential to model the distribution and generate instances \cite{goodfellow2014generative}. GR \cite{shin2017continual} firstly proposes to utilize the generative adversarial network (GAN) in CIL. In each updating process, it utilizes the GAN to generate the instances from former classes and then updates the GAN and classification model with both old and new classes.
%\textbf{Architecture-based methods} expand model capacity incrementally. DEN \cite{yoon2017lifelong} and DER \cite{yan2021dynamically} dynamically add neurons or entire backbones per task, while prompt-based methods(L2P \cite{wang2022learning}, coda-prompt \cite{smith2023coda}) adapt pre-trained Vision Transformers via task-specific tokens. Though achieving state-of-the-art performance, these methods incur growing memory costs, necessitating compression techniques like FOSTER \cite{wang2022foster}.
%\textbf{Neural Collapse} was first proposed by \cite{papyan2020prevalence}, which is a phenomenon that happens in the terminal phase(training error becomes 0). Recently it attracts researchers' attention, and some recent approaches apply it to continual learning. \cite{NCT1} first leverages this phenomenon to continue learning, which pre-defined an ETF for all the classes, and aligned the feature class means to the ETF. Several studies combine Neural Collapse with contrastive continuous learning to reduce the reliance on exemplars by using Neural Collapse as the contrastive anchors. \cite{dang2024memory, montmaur2024neural}
\section{Preliminaries}
\textbf{Problem Setup.} %\label{sec:problem_set}
We consider a general CL setup where a sequence of tasks \(\mathbb{T}=\{t\}_{t=1}^{T}\) arrives sequentially. Each task \(t\) is associated with a dataset \(\mathbb{D}_{t}=\{(\mathbf{x}_{t,i},y_{t,i})\}_{i=1}^{N_{t}}\) containing \(N_{t}\) input-label pairs. %Here, \(\mathbf{x}_{t,i}\) denotes the input vector and \(y_{t,i}\) represents the label vector for the i-th sample in task \(t\).
A fixed capacity model parameterized by $\theta$ will be trained to learn one task at a time. This work focuses on two widely studied settings: class-incremental learning (Class-IL) and task-incremental learning (Task-IL). In both settings, there is no overlap in class labels across tasks, ensuring \(\mathbb{D}_{t} \cap \mathbb{D}_{t^{\prime} } = \emptyset\) for any two distinct tasks \(t \neq t^{\prime}\). In Class-IL, the model does not have task specific information, requiring all data to be classified through a unified global classifier. In Task-IL, task-specific identifier is provided, enabling classification via dedicated task-level classifiers.
\textbf{Neural Collapse.} To formally characterize the NC phenomenon \cite{papyan2020prevalence} emerged during terminal training phases of DNNs, it is necessary to first define the simplex ETF geometry.
%—characterized by the emergence of a simplex equiangular tight frame (ETF) geometry in classifier weights during terminal training phases—fundamentally requires prior formalization of the ETF structure to establish its theoretical foundations.
\begin{definition}[Simplex Equiangular Tight Frame]\label{def:etf}
A simplex equiangular tight frame (ETF) is a set of vectors \(\{\mathbf{e}_k\}_{k=1}^K \in \mathbb{R}^d\) (\(d \geq K-1\)) with the following properties. 1) \textbf{Equal Norm:} All vectors have identical \(\ell_2\)-norm, i.e., $\|\mathbf{e}_k\|_2 = 1, \forall k \in \{1, \dots, K\}$.
2) \textbf{Equiangularity:} The inner product between any two distinct vectors is minimal and constant, i.e., $\mathbf{e}_{k_1}^\top \mathbf{e}_{k_2} = -\frac{1}{K-1}, \forall k_1 \neq k_2$.
Then, a simplex ETF can be constructed from an orthogonal basis \(\mathbf{U} \in \mathbb{R}^{d \times K}\) (where \(\mathbf{U}^\top \mathbf{U} = \mathbf{I}_K\)) via:
\begin{equation}\label{eq:etf_construction}
\textstyle
\mathbf{E} = \sqrt{\frac{K}{K-1}} \mathbf{U} \left(\mathbf{I}_K - \frac{1}{K} \mathbf{1}_K \mathbf{1}_K^\top \right),
\end{equation}
where \(\mathbf{E} = [\mathbf{e}_1, \dots, \mathbf{e}_K]\) is the ETF matrix, \(\mathbf{I}_K\) is the identity matrix, and \(\mathbf{1}_K\) is the all-ones vector.
\end{definition}
The NC phenomenon can be then characterized by the following four properties \cite{papyan2020prevalence}:
\textbf{(NC1)}: The last-layer features of samples within the same class collapse to their within-class mean, resulting in vanishing intra-class variability: the covariance \(\bm{\Sigma}_{W}^{(k)} \to \mathbf{0}\), where \(\bm{\Sigma}_{W}^{(k)} = \mathrm{Avg}_i \left\{ (\bm{\mu}_{k,i} - \bm{\mu}_{k})(\bm{\mu}_{k,i} - \bm{\mu}_{k})^\top \right\}\), \(\bm{\mu}_{k,i}\) is the feature of sample \(i\) in class \(k\), and \(\bm{\mu}_{k}\) is the within-class feature mean of class \(k\);
\textbf{(NC2)}: The centered class means \(\{\hat{\boldsymbol{\mu}}_k\}\) align with a simplex ETF, where \(\hat{\bm{\mu}}_{k} = (\bm{\mu}_{k} - \bm{\mu}_{G}) / \|\bm{\mu}_{k} - \bm{\mu}_{G}\|\) and the global mean \(\boldsymbol{\mu}_G = \frac{1}{K}\sum_{k=1}^K \boldsymbol{\mu}_k\);
\textbf{(NC3)}: The centered class means align with their corresponding classifier prototypes, i.e., \(\hat{\bm{\mu}}_{k} = \mathbf{w}_{k}/\|\mathbf{w}_{k}\|\), \(1 \leq k \leq K\), where \(\mathbf{w}_{k}\) is the class prototype of class \(k\);
\textbf{(NC4)}: Under NC1--NC3, predictions reduce to a nearest-class-center rule, i.e., \(\arg\max_k \langle \bm{\mu}, \mathbf{w}_k \rangle = \arg\min_k \|\bm{\mu} - \bm{\mu}_k\|\), where \(\bm{\mu}\) is the last-layer feature of a sample for prediction.
\vspace{-0.1cm}
\section{Continual Learning with Progressive Neural Collapse}
\vspace{-0.1cm}
\subsection{Progressive Neural Collapse}
To completely remove the need of predefining a global fixed ETF as the feature learning target for CL, we next seek to answer the following two important questions: 1) \emph{How should the base ETF target be initialized?} 2) \emph{How should the ETF target be adapted during CL?}
\textbf{\textit{1) ETF initialization after first task learning.}} Previous studies \cite{NCT1} randomly initialize the ETF target because a global ETF that matches the total class number in CL needs to be defined, which could lead to potential misalignment between the predefined ETF and learned features during task learning.
Note that
during the training of Task $1$ in CL, models trained through standard supervised learning theoretically develop last-layer class feature means $\{\bm{\mu}_c\}_{c=1}^{K_1}$ that converge to a simplex ETF $\mathbf{E}^{d \times K_1}$, where $\bm{\mu}_c\in\mathbb{R}^d$ and $K_1$ is the number of classes in Task $1$.
Thus motivated, the initial ETF should be extracted from the first task training to address the misalignment, which leads to an ETF target that matches the number of classes in the first task.
%During the initial task phase of continual learning (Task $t=1$), models trained through standard supervised learning protocols theoretically develop last-layer class means $\{\bm{\mu}_c\}_{c=1}^{K_1}$ that converge to a simplex equiangular tight frame (ETF) $\mathbf{E}^{d \times K_1}$.
However, in practice it is difficult to fully reach the asymptotic convergence regime of model training with zero training loss, such that the learned class feature means $\tilde{\bm{M}}_{K_1} = \{\tilde{\bm{\mu}}_c\}_{c=1}^{K_1}$ for Task $1$ will not strictly satisfy the ETF properties, as also corroborated in our empirical observations.
Here $\tilde{\bm{\mu}}_c$ is the empirical feature mean over samples within the class $c$.
To handle this, a key step is to find the right ETF target that is closest to $\tilde{\bm{M}}_{K_1}$ after the training for Task $1$ converges, i.e.,
\begin{equation}\label{eq:min_ETF}
\textstyle
\mathbf{E}^* = \operatorname*{argmin}_{\mathbf{E}} \| \tilde{\bm{M}}_{K_1} - \mathbf{E}\|_F^2.
\end{equation}
Towards this end, based on \Cref{def:etf}, we can have the following theorem that characterizes the nearest ETF after learning the first task:
\begin{theorem}\label{thm:equi}
Let $\mathbf{U'}=\sqrt{\frac{K_1-1}{K_1}}\tilde{\bm{M}}_{K_1}\left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right)$ and the SVD of $\mathbf{U'}$ is $\mathbf{W\Sigma V}^\top$. Then the ETF matrix $\mathbf{E}^*$ in \Cref{eq:min_ETF} can be obtained as follows:
\begin{align}\label{eq:con}
\textstyle
\mathbf{E}^*=\sqrt{\frac{K_1}{K_1-1}} \mathbf{WV}^\top \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right).
\end{align}
\end{theorem}
Given the learned class feature means $\tilde{\bm{M}}_{K_1}$, \Cref{thm:equi} immediately indicates a three-step procedure to construct the initial ETF target that aligns well with the task learning: 1) Construct $\mathbf{U'}$ from $\tilde{\bm{M}}_{K_1}$, 2) Conduct SVD on $\mathbf{U'}$, 3) Construct $\mathbf{E}^*$ based on \Cref{eq:con}.
% note that $\mathbf{E}^*$ can be expressed as the follows based on \Cref{eq:etf_construction} :
% % empirical observations from practical implementations demonstrate that class feature means $\tilde{\bm{M}_{k_1}} = \{\tilde{\bm{\mu}}_{c}\}$ approximate ETF geometry rather than strictly satisfying it. Although the model does not fully reach the asymptotic convergence regime of training, the learned class mean $\{\tilde{\bm{\mu}}_{k_1}\}$ empirically approximates an ETF $\mathbf{E}^*$, which satisfies:
% % \begin{equation}\label{min_ETF}
% % \mathbf{E}^* = \operatorname*{argmin}_{\mathbf{E}} \left\| \tilde{\bm{M}} - \mathbf{E}\right\|_F^2
% % \end{equation}
% Although the class mean matrix $\tilde{\bm{M}_{k_1}}$ does not strictly satisfy the equiangular tight frame (ETF) properties, it admits a structured decomposition that approximates ETF-like geometric regularity. Specifically, $\tilde{\bm{M}_{k_1}}$ can be expressed as:
% \begin{equation}\label{eq:M}
% \tilde{\bm{M}} = \sqrt{\frac{K_1}{K_1-1}} \mathbf{U'} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right),
% \end{equation}
% where $\mathbf{U'} \in \mathbb{R}^{d \times K_1}$ is not an orthogonal matrix, but approximates, and given by Eq.~\ref{eq:etf_construction}:
% \begin{equation}\label{opti_ETF}
% \mathbf{E}^* = \sqrt{\frac{K_1}{K_1-1}} \mathbf{U^*} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right),
% \end{equation}
% where $\mathbf{U^*}\in \mathbb{R}^{d \times K_1}$ denotes the corresponding orthogonal matrix. This yields:
% \begin{equation}
% \begin{split}
% \tilde{\bm{M}} - \mathbf{E}^* &= {\sqrt{\frac{K_1}{K_1-1}} \mathbf{U'} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right)} - {\sqrt{\frac{K_1}{K_1-1}} \mathbf{U^*} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right)} \\
% &= \sqrt{\frac{K_1}{K_1-1}}\left(\mathbf{U'} - \mathbf{U^*} \right)
% \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right).
% \end{split}
% \end{equation}
% Given that $\sqrt{\frac{K_1}{K_1-1}}$ is a constant scalar and $\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top$ constitutes a fixed projection matrix, minimizing $\left\| \tilde{\bm{M}} - \mathbf{E}^*\right\|_F^2$ becomes equivalent to minimizing $\left\| \mathbf{U'} - \mathbf{U^*}\right\|_F^2$.
% \begin{proposition}[Nearest Orthogonal Matrix via SVD]\label{NOrth}
% \label{thm:nearest-orthogonal}
% Let \( A \in \mathbb{R}^{m \times n} \) be a real matrix with Singular Value Decomposition (SVD) \( A = W \Sigma V^\top \), where \( W \in \mathbb{R}^{m \times s} \) and \( V \in \mathbb{R}^{s \times n} \) are orthogonal matrices, and \( \Sigma \in \mathbb{R}^{s \times s} \) is a diagonal matrix of singular values with \( s = \min(m, n) \). The nearest orthogonal matrix \( Q \in \mathbb{R}^{m \times n} \) to \( A \) under Frobenius norm is:
% \[
% Q = WV^\top.
% \]
% \end{proposition}
% As established in Proposition~\ref{NOrth} \cite{zou2006sparse}, the optimal orthogonal matrix is derived via singular value decomposition:
% \begin{equation}\label{opti_Orth}
% \mathbf{U^*} = \mathbf{WV}^\top \quad \text{(from SVD on $\mathbf{U'} = \mathbf{W\Sigma V}^\top$)}.
% \end{equation}
% Input $\mathbf{U^*}$ to Eq.~\ref{eq:etf_construction}, we can formalize the $\mathbf{E}^*$.
\textbf{\textit{2) ETF expansion prior to new task learning.}}
Given the initial ETF matrix $\mathbf{E}_1=\mathbf{E}^*$ where the number of vertices matches the number of classes in Task $1$, the next step is to progressively expand the ETF target as new classes come in with new tasks, in order to achieve two objectives: 1) the number of newly expanded vertices in the new ETF target will match the number of new classes in the new task; 2) the vertices in the new ETF target that match old classes will not significantly shift from their original positions in the old ETF target, so as to reduce catastrophic forgetting.
A key insight here is that the ETF matrix $\mathbf{E}$ is indeed determined by the corresponding orthogonal basis $\mathbf{U}$ as shown in \Cref{eq:etf_construction}, and keeping the orthogonal basis unchanged when expanding the ETF will in principle reduce the shift from the old ETF. This motivates a novel ETF expansion strategy by expanding the constructing orthogonal basis, which includes two steps:
\emph{(Step a)} Let $K_{t}$ represent the total number of classes until any task $t$. When a new task $t\geq 2$ arrives with $K_t-K_{t-1}$ classes, the ETF target expansion will be triggered before learning task $t$, which seeks to obtain a new target $\mathbf{E}_t$ with $K_t$ vertices from the previous ETF target $\mathbf{E}_{t-1}$ with $K_{t-1}$ vertices. In particular, the original orthogonal basis \(\mathbf{U}_{t-1} \in \mathbb{R}^{d \times K_{t-1}}\) of $\mathbf{E}_{t-1}$ will be expanded to \(\mathbf{U}_{t} \in \mathbb{R}^{d \times K_{t}}\) by appending \(K_{t} - K_{t-1}\) new orthogonal vectors. These new vectors are generated via Gram-Schmidt orthogonalization against the existing $K_{t-1}$ basis vectors in \(\mathbf{U}_{t-1}\), ensuring that \(\mathbf{U}_{t}\) retains orthonormality across all \(K_{t}\) vectors.
\emph{(Step b)} Substituting \(\mathbf{U}_{t}\) and \(K_{t}\) into \Cref{eq:etf_construction} will lead to an expanded ETF target with \(K_{t}\) vertices. This extended ETF serves as the predefined geometric configuration for feature learning in task \(t\), maintaining uniform angular separation and maximal equiangularity among all seen classes.
%By decoupling the ETF expansion from class-specific data (e.g., mean vectors), the framework ensures theoretical consistency in inter-class separability while preserving the core ETF properties required for stable sequential learning.
%In incremental tasks \(\mathbb{T} = \{t_i\}_{i=1}^T\), where the number of classes increases from \(K^t\) to \(K^{t+1}\) between tasks, the Equiangular Tight Frame (ETF) structure must be extended \textit{proactively} at the beginning of task \(t_{k+1}\). To achieve this, the original orthogonal matrix \(\mathbf{U} \in \mathbb{R}^{d \times K^t}\) is expanded to \(\mathbf{U}^{t+1} \in \mathbb{R}^{d \times K^{t+1}}\) by appending \(K^{t+1} - K^t\) new orthogonal vectors. These vectors are generated via Gram-Schmidt orthogonalization against the existing columns of \(\mathbf{U}\), ensuring that \(\mathbf{U}^{t+1}\) retains orthonormality across all \(K^{t+1}\) classes.
\vspace{-0.5cm}
\subsection{A Continual Learning Framework based on Progressive Neural Collapse}
In what follows, we seek to incorporate the idea of progressive neural collapse (ProNC) into commonly used CL algorithm designs, where the model will be trained to push the learned class features towards the progressively expanded ETF for each task during CL.
More specifically, we focus on the model training for tasks $t\geq 2$, whereas the first task learning follows a standard supervised learning procedure with widely used loss functions, e.g., cross-entropy loss $\mathcal{L}_{\text{ce}}$. For tasks $t\geq 2$, we first apply ProNC to generate a newly expanded ETF target $\mathbf{E}_t$ before learning task $t$. The loss function design for model training will include three different loss terms, i.e., a \textbf{supervised term}, an \textbf{alignment term}, and a \textbf{distillation term}. The first supervised term follows standard loss designs in CL and leverages the cross-entropy loss to facilitate intra-task classification, while we introduce the other two loss terms below in detail.
\emph{\textbf{1) Alignment with the ETF target.}} The alignment loss pushs the learned class features towards the ETF target $\mathbf{E}_t=[\mathbf{e}_{1,t},...,\mathbf{e}_{K_t,t}]$, which characterizes the cosine similarity between the feature $\bm{\mu}_{k,i}$ for sample $i$ in class $k$ and the corresponding vertex $\mathbf{e}_{k,t}$ in the ETF $\mathbf{E}_t$ for class $k$ \cite{yang2022inducing,NCT1}:
\begin{equation}
\textstyle
\mathcal{L}_{\mathrm{align}}( \bm{\mu}^t_{k,i}, \mathbf{e}_{k,t})
= \frac{1}{2} ( \mathbf{e}_{k,t}^\top \bm{\mu}^t_{k,i} - 1 )^2.
\label{eq:dr_loss}
\end{equation}
Here $\bm{\mu}^t_{k,i}$ corresponds to the normalized feature extracted from the last layer of the current model when learning task $t$. A small $\mathcal{L}_{\mathrm{align}}$ implies that the learned normalized feature for each sample aligns well the corresponding ETF vertex for the class that the sample belongs to, minimizing the intra-class variability while forcing different class feature means equally separated. This indeed provides a new type of feature regularization based on ETF, which can be widely adopted and is also very powerful as shown later in our experimental results.
\emph{\textbf{2) Distillation to further reduce forgetting.}} While ProNC expands the ETF target without dramatically shifting from the old ETF, the vertices that map to old classes will inevitability change after ETF expansion due to geometric properties of ETF. To handle this and further reduce catastrophic forgetting, we next borrow the idea of knowledge distillation, which is a widely used technique in CL to reduce catastrophic forgetting \cite{rebuffi2017icarl,hou2019learning,buzzega2020dark,NCT1}. In particular, we consider a typical distillation loss which characterizes the cosine similarity between the normalized features, for the same data sample, from the current model and that from the model obtained after learning the previous task, to maintain the simplicity and flexibility of our framework:
\begin{equation}
\textstyle
\mathcal{L}_{\text{distill}}(\bm{\mu}_{k,i}^{(t-1)}, \bm{\mu}_{k,i}^{(t)}) = \frac{1}{2} ( (\bm{\mu}_{k,i}^{(t-1)})^\top \bm{\mu}_{k,i}^{(t)} - 1 )^2.
\label{eq:distill}
\end{equation}
Here $\bm{\mu}_{k,i}^{(t-1)}$ is the normalized last layer feature for the sample $i$ in class $k$ extracted from the learned model after task $t-1$.
% Let \( f \) denote the backbone network, which processes an input \(\bm{x}\) through parameters \(\theta_f\) to generate an output feature vector \(\bm{\mu} \in \mathbb{R}^d\). The primary training objective is to optimize the backbone features such that they converge toward their respective class prototypes in the NC ETF. At optimal convergence, all class features align with the ETF, thereby satisfying the neural collapse condition and resolving feature-classifier misalignment.
% The loss function for each session comprises three components: \textit{(i)} an \textbf{alignment term} enforcing feature-ETF correspondence, \textit{(ii)} a \textbf{distillation term} preserving feature consistency across sessions, and\textit{(iii)} a \textbf{supervised term} to directly optimize decision boundaries.
% The previous study \cite{NCT1} proposes a misalignment loss to align the last layer features with their corresponding prototypes, which can be formulated as:
% \begin{equation}
% \mathcal{L}_{\mathrm{align}}\left( \hat{\bm{\mu}}_i, \hat{\mathbf{w}}_{y_i} \right)
% = \frac{1}{2} \left( \hat{\mathbf{w}}_{y_i}^\top \hat{\bm{\mu}}_i - 1 \right)^2
% \label{eq:dr_loss}
% \end{equation}
% Following the knowledge of distillation technique from~\cite{hou2019learning, NCT1}, we preserve the backbone network trained in the previous session. The distillation loss is formulated as:
% \begin{equation}
% \mathcal{L}_{\text{distill}}\left(\hat{\bm{\mu}}_{i}^{(t-1)}, \hat{\bm{\mu}}_{i}^{(t)}\right) = \frac{1}{2} \left( \left(\hat{\bm{\mu}}_{i}^{(t-1)}\right)^\top \hat{\bm{\mu}}_{i}^{(t)} - 1 \right)^2,
% \label{eq:distill}
% \end{equation}
% where \(\hat{\bm{\mu}}_{i}^{(t-1)}\) denotes features generated by the previous session's backbone network.
% The cross-entropy loss explicitly optimizes inter-class decision boundaries through posterior probability maximization, while inherently suppressing intra-task classification errors via margin enhancement in feature space.
To best unlock the potential of ProNC, we also leverage the standard data replay in the CL framework: the replay data from previous tasks will be mixed together with the current data before the current task learning, such that each minibatch of data during model training will include both replay data and current data. Combining all three loss terms will lead to the final instance-loss function:
\begin{equation}
\textstyle
\mathcal{L} =\mathcal{L}_{\text{ce}} + \lambda_1 \cdot \mathcal{L}_{\text{align}} + \lambda_2 \cdot \mathcal{L}_{\text{distill}},
\label{eq:total_loss}
\end{equation}
which is averaged over all samples in a minibatch for model update. Here $\lambda_1$ and $\lambda_2$ are coefficients to balance between pushing the learned features towards the target ETF and reducing the vertex shift from the previous ETF for minimizing forgetting.
The overall workflow is shown in \Cref{fig:flowchart}.
\begin{figure}
\vspace{-0.5cm}
\centering
\includegraphics[width=0.65\linewidth]{Flowchart.pdf}
\caption{Overflow of our proposed CL framework for new task learning based on the mixture of current task data and replay data. The new model $f_t$ is trained towards the expanded ETF target, with forgetting further reduced based on feature distillation. }
\label{fig:flowchart}
\vspace{-0.5cm}
\end{figure}
\textbf{Inference.} During inference for model performance evaluation, instead of using the linear classifier for classification, we leverage the widely used cosine similarity \cite{peng2022few,gidaris2018dynamic,wang2018cosface,hou2019learning} between the extracted sample feature from the learned model and the vertices in the target ETF as the classification criteria. Specifically, for any testing data sample $i$, let $\bm{\mu}_j$ denote the normalized last layer feature extracted from the tested model. Then the class prediction result will be \(\arg\max_{k} \bm{\mu}_j^\top \bm{e}_k\), where $\bm{e}_k$ is the $k$-th column vector in the target ETF.
% given a backbone-generated feature \(\bm{\mu} \in \mathbb{R}^d\), we employ cosine similarity with the target \(\mathbf{\hat{W}}_{\mathrm{ETF}}\) as the classification criterion: \(\arg\max_{k} \cos\theta(\bm{\mu}, \hat{\bm{w}}_{k})\), where \(\hat{\bm{w}}_{k}\) represents the \(k\)-th column vector in \(\mathbf{\hat{W}}_{\mathrm{ETF}}\). This cosine similarity criterion, widely adopted in CIL~\cite{hou2019learning} and computer vision tasks~\cite{peng2022few,gidaris2018dynamic,wang2018cosface}, effectively mitigates bias toward novel classes induced by learnable parameters or feature modulus.
\section{Experimental Results}
\subsection{Experimental Setups}
\textbf{\textit{Datasets.}} We evaluate the performance on three standard CL benchmarks, i.e., \textbf{Seq-CIFAR-10}~\cite{krizhevsky2009learning} that partitions 10 classes into 5 sequential tasks (2 classes per task), \textbf{Seq-CIFAR-100}~\cite{krizhevsky2009learning} that comprises 10 tasks with 10 classes per task, and \textbf{Seq-TinyImageNet} \cite{le2015tiny} divides 200 classes into 10 sequential tasks (20 classes per task), and consider both Class-IL and Task-IL scenarios.
% Following established continual learning benchmarks, we evaluate our method using three standardized datasets: \textbf{Split CIFAR-10}, \textbf{Split CIFAR-100}, and \textbf{Split TinyImageNet} which . Our experiments adhere to both class-incremental (Class-IL) and task-incremental (Task-IL) evaluation protocols, as detailed in Section~\ref{sec:problem_set}.
\textbf{\textit{Baseline approaches.}} We evaluate our CL approach against state-of-the-art (SOTA) approaches, including various replay-based approaches, i.e., ER~\cite{riemer2019learning}, iCaRL~\cite{rebuffi2017icarl}, GEM~\cite{lopez2017gradient}, GSS~\cite{aljundi2019gradient}, DER~\cite{buzzega2020dark}, DER++~\cite{buzzega2020dark},
and contrastive learning based approaches, i.e., Co\textsuperscript{2}L~\cite{cha2021co2l}, CILA~\cite{wenprovable}, and MNC\textsuperscript{3}L~\cite{dang2024memory}. We also compare our approach with NCT \cite{NCT1} that predefines a fixed global ETF target.
\noindent\textbf{\textit{Implementation details.}} To ensure a fair comparison, we train a ResNet-18 backbone~\cite{he2016deep} using the same number of epochs for all approaches, i.e., 50 epochs for every task in Seq-CIFAR-10 and Seq-CIFAR-100, where 100 epochs for every task in Seq-TinyImageNet to accommodate its higher complexity. For contrastive learning approaches Co\textsuperscript{2}L, CILA, and MNC\textsuperscript{3}L, we follow their original implementations by removing the final classification layer of ResNet-18, and append a two-layer projection head with ReLU activation to map backbone features into a $d$-dimensional embedding space ($d=128$ for Seq-CIFAR-10/100, $d=256$ for Seq-TinyImageNet). A separate linear classifier is subsequently trained on the frozen embeddings for these contrastive learning methods.
\noindent\textbf{\textit{Evaluation metrics.}} We consider two standard evaluation metrics in CL \cite{chaudhry2018riemannian, buzzega2020dark, rebuffi2017icarl, dang2024memory}, i.e., final average accuracy (FAA) and average forgetting (FF).
Let $T$ denote the total number of tasks (start with task 0 and end with task $T-1$) in CL and $a^{t}_{i}$ be the model accuracy on
the $i$-th task after learning the task $t\in [0, T-1]$.
The FAA and FF are defined as:
\begin{align*}
\textstyle
\text{FAA} = \frac{1}{T} \sum_{i=0}^{T-1} a^{T-1}_{i}, ~\text{FF} = \frac{1}{T-1} \sum_{i=0}^{T-2} \max_{t\in\{0,...,T-2\}} a_i^t - a_i^{T-1}.
\end{align*}
\begin{table}
\vspace{-0.3cm}
\caption{Performance comparison under various setups.
%FAA is Final Average Accuracy and FF is Final Forgeting.
All results are averaged over multiple runs. The final version with error bars is in the appendix.}
\label{tab:main}
\centering
\resizebox{0.8\textwidth}{!}{
\begin{tabular}{@{}lccccccc@{}}
\toprule
\multirow{2}{*}{Buffer} & \multirow{2}{*}{Method} & \multicolumn{2}{c}{Seq-CIFAR-10} & \multicolumn{2}{c}{Seq-CIFAR-100} & \multicolumn{2}{c}{Seq-TinyImageNet} \\
\cmidrule(lr){3-4} \cmidrule(lr){5-6} \cmidrule(lr){7-8}
& & Class-IL & Task-IL & Class-IL & Task-IL & Class-IL & Task-IL \\
\cmidrule(lr){3-4} \cmidrule(lr){5-6} \cmidrule(lr){7-8}
& & FAA (FF) & FAA (FF) & FAA (FF) & FAA (FF) & FAA (FF) & FAA (FF) \\
\midrule
\multirow{11}{*}{200} & ER ~\cite{riemer2019learning} & 44.79 (59.30) & 91.19 (6.07) & 21.78 (75.06) & 60.19 (27.38) & 8.49 (76.53) & 38.17 (40.47) \\
& iCaRL ~\cite{rebuffi2017icarl} & 49.02 (23.52) & 88.99 (25.34) & 28 (47.20) & 51.43 (36.20) & 7.53 (31.06) & 28.19 (42.47) \\
& GEM ~\cite{lopez2017gradient} & 25.54 (80.36) & 90.44 (9.57) & 20.75 (77.40) & 58.84 (29.59) & -- & -- \\
& GSS ~\cite{aljundi2019gradient} & 39.07 (72.48) & 88.8 (8.49) & 19.42 (77.62) & 55.38 (32.81) & -- & -- \\
& DER ~\cite{buzzega2020dark} & 61.93 (35.79) & 91.4 (6.08) & 31.23 (62.72) & 63.09 (25.98) & 11.87 (64.83) & 40.22 (40.43) \\
& DER++ ~\cite{buzzega2020dark} & 64.88 (32.59) & 91.92 (5.16) & 28.13 (60.99) & 66.80 (23.91) & 11.34 (73.47) & 43.06 (39.02) \\
& Co$^2$L ~\cite{cha2021co2l} & 51.27 (30.17) & 84.69 (2.91) & 18.09 (64.14) & 49.19 (27.83) & 12.95 (62.04) & 38.40 (40.75) \\
& CILA ~\cite{wenprovable} & 59.68 (37.52) & 91.36 (5.89) & 19.49 (64.01) & 53.93 (33.07) & 12.98 (63.11) & 37.32 (41.40) \\
& MNC$^3$L~\cite{dang2024memory} & 51.09 (33.74) & 85.07 (4.90) & 15.81 (62.51) & 43.91 (39.79) & 10.57 (59.68) & 32.78 (45.10) \\
& NCT ~\cite{NCT1} & 51.59 (22.48) & 80.63 (1.41) & 26.38 (27.40) & 75.75 (4.79) & 10.95 (49.33) & 52.71 (15.88) \\
& \textbf{Ours} & \textbf{65.58 (32.75)} & \textbf{96.86 (0.65)} & \textbf{42.99 (36.07)} & \textbf{85.63 (4.40)} & \textbf{27.44 (42.81)} & \textbf{69.09 (9.42)} \\
\midrule
\multirow{11}{*}{500} & ER ~\cite{riemer2019learning} & 57.74 (43.22) & 93.61 (3.50) & 27.66 (67.96) & 66.23 (17.37) & 9.99 (75.21) & 48.64 (30.73) \\
& iCaRL ~\cite{rebuffi2017icarl} & 47.55 (28.20) & 88.22 (22.61) & 33.25 (40.99) & 58.16 (27.90) & 9.38 (37.30) & 31.55 (39.44) \\
& GEM ~\cite{lopez2017gradient} & 26.2 (78.93) & 92.16 (5.60) & 25.54 (71.34) & 66.31 (20.44) & -- & -- \\
& GSS ~\cite{aljundi2019gradient} & 49.73 (59.18) & 91.02 (6.37) & 21.92 (74.12) & 60.28 (26.57) & -- & -- \\
& DER ~\cite{buzzega2020dark} & 70.51 (24.02) & 93.40 (3.72) & 41.36 (49.07) & 71.73 (25.98) & 17.75 (59.95) & 51.78 (28.21) \\
& DER++ ~\cite{buzzega2020dark} & 72.70 (22.38) & 93.88 (4.66) & 38.20 (49.18) & 74.77 (15.75) & 19.38 (58.75) & 51.91 (25.47) \\
& Co$^2$L ~\cite{cha2021co2l} & 61.78 (17.79) & 89.51 (2.65) & 26.64 (48.60) & 62.32 (23.47) & 18.71 (49.64) & 50.74 (13.80) \\
& CILA ~\cite{wenprovable} & 67.82 (18.22) & 93.29 (0.65) & 31.27 (45.67) & 68.29 (16.98) & 18.09 (64.14) & 49.19 (27.83) \\
& MNC$^3$L~\cite{dang2024memory} & 52.20 (27.88) & 85.94 (3.15) & 22.29 (46.09) & 56.43 (24.29) & 11.52 (44.96) & 36.32 (39.00) \\
& NCT~\cite{NCT1} & 60.93 (13.82) & 81.27 (3.84) & 33.77 (27.85) & 75.87 (5.06) & 18.24 (50.90) & 62.30 (6.84) \\
& \textbf{Ours} & \textbf{73.95 (26.75)} & \textbf{96.95 (0.24)} & \textbf{48.94 (24.32)} & \textbf{86.38 (4.35)} & \textbf{29.06 (38.58)} & \textbf{69.77 (9.52)} \\
\bottomrule
\end{tabular}
}
\vspace{-0.5cm}
\end{table}
\vspace{-0.1cm}
\subsection{Main Results} \label{sec:results}
%Tab.~\ref{tab:performance} presents experimental results for buffer sizes 200 and 500. For baseline methods including Co\textsuperscript{2}L~\cite{cha2021co2l}, CIIA~\cite{wenprovable}, and MNC\textsuperscript{3}L~\cite{dang2024memory}, their original implementations utilized 500 training epochs for the initial task followed by 100 epochs for subsequent tasks. To ensure fair comparison with other methods, we standardized the training protocol by reducing both initial and incremental task epochs to 50 for CIFAR-10/100 benchmarks, while maintaining 100 epochs for Tiny-ImageNet to accommodate its higher complexity.
\Cref{tab:main} shows the performance comparison for both Class-IL and Task-IL under different memory budgets, i.e., buffer size 200 and 500.
It is clear that our approach significantly and consistently outperforms all the baseline approaches across all considered CL settings, datasets, and buffer sizes.
In particular, the performance improvement in our approach becomes more substantial on larger datasets and under a smaller buffer size. For example, consider a buffer size of 200. On Seq-CIFAR-100, our approach outperforms the best baseline approaches, i.e., DER for Class-IL and NCT for Task-IL, by 37.65\% and 13.04\%, respectively. On Seq-TinyImageNet, our approach outperforms the best baseline approaches, i.e., CILA for Class-IL and NCT for Task-IL, by 111.40\% and 31.08\%, respectively. In particular, the performance of our approach is outstanding when the buffer size is 200, especially for Task-IL when we focus on differentiating classes within a specific task. This implies that our approach is more robust to different buffer sizes and the principle of ProNC can even work well with a small amount of replay data. Moreover,
by leveraging the NCT in a more principled manner, our approach dominates the previous approach NCT where a predefined global ETF target hinders class discrimination by unnecessarily forcing class means towards closely located vertices.
Besides, it can be seen from \Cref{tab:main} that among most scenarios our approach and NCT achieve much less forgetting compared to other baseline approaches, while our approach shows even better forgetting than NCT in 8 out of 12 settings. The reason is because the vertices in the ETF target provide an fixed anchor and goal for old task features during new task learning, which the model training can follow much easily in the feature space. In other words, instead of simply constraining the shifts from old features or important weights as in previous studies, the ETF target offers an additional fixed goal from which the model should not shift the new features too far way.
These results indicate the huge potential of leveraging NC and ETF in guiding the feature learning for CL.
More interestingly, in contrast to previous replay-based approaches \cite{riemer2019learning,rebuffi2017icarl,lopez2017gradient,aljundi2019gradient,buzzega2020dark} which require a replay buffer in principle, \emph{our approach can even work without replay}. Specifically, with an \emph{empty} replay buffer, our approach can achieve a FAA of $30.87\%$ for Class-IL and $84.33\%$ for Task-IL on Seq-CIFAR-100, and $24.43\%$ for Class-IL and $68.08\%$ for Task-IL on Seq-TinyImageNet, which almost outperforms all baselines in \Cref{tab:main} under a buffer size of 200 and even of 500. \emph{This phenomenal performance implies that our approach indeed offers a new and powerful feature regularization based on ProNC, which can be widely applied in various CL scenarios.}
%As shown in Tab.~\ref{tab:performance}, our method significantly outperforms the other rehearsal-based methods across various CL settings, datasets, and buffer sizes. For Seq-Cifar-10, our method exceeds DER++ by 5.6\% in Class-IL and 5.9\% in Task-IL with buffer size 200, while with buffer size 500, our method exceeds DER++ by 3\% in Class-IL and 3.24\% in Task-IL. For Seq-Cifar-100, our method outperforms MNC\textsuperscript{3}L by 21.54\% in Class-IL and DER++ by 28.32\% in Task-IL at buffer size 200; with buffer size 500, our method exceeds MNC\textsuperscript{3}L by 20.62\% in Class-IL and outperforms DER++ by 12.29\% in Task-IL. On Seq-Tiny-ImageNet, our method outperforms MNC\textsuperscript{3}L by 68.49\% in Class-IL and 51.93\% in Task-IL with buffer size 200, and exceeds CILA by 34.11\% in Class-IL and 26.02\% in Task-IL with buffer size 500. These results demonstrate our method's superior performance in large sequence data and limited buffer size scenarios.
%Our method demonstrates consistent superiority over existing approaches across all evaluated scenarios, particularly in sequential learning benchmarks such as Seq-CIFAR-100 and Seq-Tiny-ImageNet, even under constrained memory buffer sizes (200 and 500). This advantage arises from the properties of the Equiangular Tight Frame (ETF) structure, which becomes more pronounced in large-scale sequential datasets with a high initial base class count. Specifically, as the number of base classes increases, the initial ETF spans a higher-dimensional geometric space, and its subsequent incremental extension induces minimal displacement to each class vertex. This property mitigates interference between old and new classes, preserving the geometric uniformity critical for stability in continual learning.
\vspace{-0.1cm}
\subsection{Ablation Study}
\vspace{-0.1cm}
In what follows, we will conduct various ablation studies to build a comprehensive understanding of our approach, where most studies are for Class-IL on Seq-CIFAR-100 with a buffer size 500.
\begin{figure}[htbp]{}
\centering
\vspace{-0.5cm}
% First row of figures
\begin{subfigure}[b]{0.28\textwidth}
\includegraphics[width=\linewidth]{cosin_sim+1_k-1.pdf}
\vspace{-0.5cm}
\caption{Cosine similarity of feature means between any two different classes, averaged over all seen classes.}
\label{fig:max_angle}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.28\textwidth}
\includegraphics[width=\linewidth]{svd_withinclass.pdf}
\vspace{-0.5cm}
\caption{Std of cosine similarity of feature means between any two different classes in the same task, averaged over tasks.}
\label{fig:separate_within_class}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.28\textwidth}
\includegraphics[width=\linewidth]{Cosine_similarity_between_first_model_feature_and_final_model_feature.pdf}
\vspace{-0.5cm}
\caption{Cosine similarity between best previous features and final features for the same class, averaged over all seen classes.}
\label{fig:forgetting}
\end{subfigure}
% Second row of figures
\begin{subfigure}[b]{0.28\textwidth}
\includegraphics[width=\linewidth]{Cos_Simi_all_class_means_proto.pdf}
\vspace{-0.5cm}
\caption{Cosine similarity between one class feature mean and the classifier prototype of any seen different class.}
\label{fig:all_mean_classifier}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.28\textwidth}
\includegraphics[width=\linewidth]{Cos_Simi_within_task_class_means_proto.pdf}
\vspace{-0.5cm}
\caption{Cosine similarity between one class feature mean and the classifier prototype of a different class in the same task.}
\label{fig:within_mean_classifier}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.28\textwidth}
\includegraphics[width=\linewidth]{Robust_of_loss_weights2.pdf}
\vspace{-0.5cm}
\caption{Performance robustness of our approach under different values of the hyperparameters $(\lambda_1, \lambda_2)$.}
\label{fig:para}
\end{subfigure}
\caption{In (a)-(e), X-axis is the task ID during CL
and Y-axis is the (average or std) value of the corresponding cosine similarity. In (f), X-axis is the accuracy FAA and Y-axis is the value of $(\lambda_1, \lambda_2)$.
% Model Analyze. (a) $\text{Avg}_{k\neq k'}(cos\angle(\frac{\mu_k-\mu_G}{||\mu_k-\mu_G||}, \frac{\mu_{k'}-\mu_G}{||\mu_{k'}-\mu_G||})+1/(K-1))$ of all classes, (b) $\text{Avg}(\text{std}(cos\angle(\frac{\mu_{k,t}-\mu_G}{||\mu_{k,t}-\mu_G||}, \frac{\mu_{k',t}-\mu_G}{||\mu_{k',t}-\mu_G||})))$ of all classes within the same task, (c) $\text{Avg}(cos\angle(f_\theta^{best}(x), f_\theta^{cur}(x)))$ of all exemplars, (d) $\text{Avg}_{k\neq k'}(cos\angle(\frac{\mu_k-\mu_G}{||\mu_k-\mu_G||}, \frac{w_{k'}}{||w_{k'}||}))$ of all classes, (e) $\text{Avg}_{k\neq k'}(cos\angle(\frac{\mu_{k,t}-\mu_G}{||\mu_{k,t}-\mu_G||}, \frac{w_{k',t}}{||w_{k',t}||}))$ of all classes within the same task (f) Classify accuracy with different sets of coefficients of weights.
}
\vspace{-0.6cm}
\label{fig:model_analyze}
\end{figure}
\emph{Feature learning behaviors.}
To understand the superior performance of our approach in contrast to important baseline approaches, we first delve into the feature learning behaviors during CL by characterizing different types of feature correlations. In particular, based on our analysis of the ETF target and \Cref{tab:main}, compared to NCT \cite{NCT1} with a predefined global ETF, our approach should enjoy the following benefits:
1) A lower cosine similarity between different class feature means which is also closer to theoretically maximum separation $-\frac{1}{K_t-1}$ for $K_t$ seen classes until task $t$ according to \Cref{def:etf}. This can be confirmed in \Cref{fig:max_angle}, where we calculate $\text{Avg}_{k\neq k'}(\langle\frac{\mu_k-\mu_G}{||\mu_k-\mu_G||}, \frac{\mu_{k'}-\mu_G}{||\mu_{k'}-\mu_G||}\rangle+\frac{1}{K_t-1})$ of the features after learning every task $t$, averaged for any two seen classes $k$ and $k'$. Thanks to ProNC, our approach achieves a smaller value, which is also closer to $0$, than NCT.
2) All class feature means should be almost equally separated within the same task. To show this, we evaluate the standard deviation (std) of across-class cosine similarity for the same task, averaged over all seen tasks in CL, i.e., $\text{Avg}(\text{std}(\langle\frac{\mu_{k}-\mu_G}{||\mu_{k}-\mu_G||}, \frac{\mu_{k'}-\mu_G}{||\mu_{k'}-\mu_G||}\rangle))$, where classes $k$ and $k'$ are in the same task. \Cref{fig:separate_within_class} shows that our approach achieves a smaller average std (also closer to $0$) than NCT, implying the different class means are closer to equal separation. Besides, we also follow the metrics in NCT \cite{NCT1} to evaluate 1) the cosine similarity between the feature mean of class $k$ and the classifier prototype $w_{k'}$ of a different class $k'$, averaged over all seen classes until task $t$, and 2) the same cosine similarity but for classes within the same task which is averaged over all seen tasks during CL. As shown in \Cref{fig:all_mean_classifier} and \ref{fig:within_mean_classifier}, our approach achieves a lower cosine similarity compared to NCT and DER++, which implies easier class discrimination.
Moreover, \Cref{tab:main} demonstrates the exceptional performance of our approach in addressing forgetting. To understand this, we characterize the cosine similarity between 1) the learned features from the best performing previous model and 2) that from the final model, for the same class, averaged over all seen classes so far. This is consistent to the definition of forgetting. As shown in \Cref{fig:forgetting}, our approach achieves the higher similarity than both NCT and DER++, indicating its superior capability in handling feature shifts during CL and then minimizing forgetting based on ProNC.
%To evaluate the efficacy of our proposed Progressive ETF framework in establishing maximal separation among classes both globally and equal separation within tasks, we analyze the cosine similarity between feature means, defined as $\text{Avg}_{k\neq k'}(cos\angle(\frac{\mu_k-\mu_G}{||\mu_k-\mu_G||}, \frac{\mu_{k'}-\mu_G}{||\mu_{k'}-\mu_G||})$. As illustrated in \Cref{fig:max_angle}, we measure the deviation from optimality by computing the difference between observed cosine similarities and the theoretical maximum separation in an ETF, $\frac{-1}{K-1}$. Our ProNC method consistently demonstrates lower values compared to NCT, indicating superior angular separation between class means. For within-task class separation, an ideal configuration would exhibit uniform angular distribution. \Cref{fig:separate_within_class} presents the standard deviation of cosine similarities between different classes within the same task, formulated as $std(cos\angle(\mu_{k,t}, \mu_{k', t}))$, where values approaching zero indicate more uniform separation. Our approach yields consistently smaller standard deviations than NCT, confirming more balanced within-task separation.
%Furthermore, we conduct a comparative analysis of feature-classifier status, i.e., $\text{Avg}_{k\neq k'}(cos\angle(\frac{\mu_k-\mu_G}{||\mu_k-\mu_G||},\frac{w_{k'}}{||w_{k'}||})$ across our Progressive Neural Collapse, DER++~\cite{buzzega2020dark}, and NCT~\cite{NCT1}. \Cref{fig:all_mean_classifier} demonstrates that our method achieves significantly lower average cosine similarities between features and classifier prototypes of different classes. Similarly, \Cref{fig:within_mean_classifier} confirms that our approach maintains the lowest cosine similarities between features and classifier prototypes for different classes within the same task. This proved that the angles between different class means in our approach are larger than the the angles in the other 2 approaches.
%To quantitatively assess catastrophic forgetting, we examine the representational stability by measuring cosine similarity between features at peak task performance and those in the final model. As evidenced in \Cref{fig:forgetting}, ProNC exhibits substantially higher cosine similarity values between these feature representations compared to alternative methods, thereby demonstrating enhanced resistance to catastrophic forgetting. These results collectively validate the superior representational properties and stability of our Progressive Neural Collapse framework.
\begin{wraptable}{r}{0.32\textwidth}
\vspace{-0.45cm}
%\begin{table}[t]
\footnotesize
\centering
\caption{Impact of different components.}
\vspace{-0.2cm}
\label{tab:comp}
\resizebox{0.32\textwidth}{!}{
\small
\begin{tabular}{@{}lc@{}}
\toprule
\textbf{Variant} & \textbf{Performance} \\
\midrule
Ours & 48.94 \\
(a) w/o $\mathcal{L}_{\text{ce}}$ & 44.97 \\
(b) w/o $\mathcal{L}_{\text{align}}$ & 23.22 \\
(c) w/o $\mathcal{L}_{\text{distill}}$ & 19.96 \\
(d) w/ predefined base ETF & 44.99 \\
(e) w/ predefined global ETF & 33.51 \\
(f) w/ linear classifier & 44.49 \\
\bottomrule
\end{tabular}
}
\vspace{-0.3cm}
\end{wraptable}
\emph{Impacts of different components.}
To understand the impact of different design components on our approach,
%we conduct the ablation study for Class-IL on Seq-CIFAR-100 with a buffer size 500 under
we investigate six different variants, as shown in \Cref{tab:comp}. Here in (a)-(c) we remove one of the three loss terms in \Cref{eq:total_loss}, respectively. Clearly, removing either $\mathcal{L}_{\text{align}}$ or $\mathcal{L}_{\text{distill}}$ will significantly degrade the performance, highlighting the importance of the designed ETF target and the right balance between learning stability and plasticity. On the other hand, while removing $\mathcal{L}_{\text{ce}}$ will slightly decrease the performance, this variant still outperforms the baseline approaches as shown in \Cref{tab:main}, indicating the benefit of our approach mainly from $\mathcal{L}_{\text{align}}$ and $\mathcal{L}_{\text{distill}}$. In (d), we predefine a base ETF target and expand it for new tasks based on ProNC, and the degraded performance shows the benefit of naturally aligning the base ETF with feature learning in the first task. In (e), we replace the entire ProNC with a predefined fixed global ETF as in \cite{NCT1} and the performance drops dramatically, corroborating the importance of ProNC for setting an appropriate ETF target in new task learning. In (f), we replace our cosine similarity based classifier by using the standard linear classifier, and the performance drop further highlights the usefulness of ETF in classification by providing equally separated feature representation targets for different classes.
% To verify the effectiveness of the three terms of the loss function and ProNC, we perform an ablation experiment with the class-IL setup on the Seq-CIFAR-100 dataset with five variants. $(a)$ without $\mathcal{L}_{\text{ce}}$, we train the model without training the linear classifier. $(b)$ without $\mathcal{L}_{\text{align}}$$\mathcal{L}_{\text{align}}$, and use linear classifier. $(c)$ Training with a randomly defined initial ETF and extending it in incremental tasks. $(d)$ Training with a fixed pre-defined ETF. $(e)$ Using a linear classifier instead of the ETF. For all the ablation experiments, the buffer size is 500.
% As summarized in Table~\ref{tab:ablation}, our ablation study reveals critical insights into the contributions of each component in the proposed method: The cross-entropy loss $\mathcal{L}_{\text{ce}}$ drives a significant performance gain of $\mathbf{+8.03\%}$, underscoring its necessity for task-specific discriminative learning.
% Our full method achieves a $\mathbf{+109.22\%}$ improvement over a naive combination of experience replay and feature distillation, highlighting the importance of synergistic optimization between stability and plasticity.
% Extending from an ETF that was obtained by training yields $\mathbf{+7.97\%}$ gain compared to extending from a fixed predefined ETF, demonstrating the benefits of learned geometric configurations. Our method outperforms training with a fixed ETF by $\mathbf{+44.97\%}$, emphasizing the need for adaptive feature alignment.
% Replacing the standard linear classifier with an ETF structure further improves accuracy by $\mathbf{+9.19\%}$, validating its role in enforcing maximally separable representations.
% These results collectively affirm that the proposed method’s performance stems from its \textbf{cohesive integration} of task-adaptive geometric regularization, dynamic prototype refinement, and stability-preserving distillation.
% \begin{table}[t]
% \centering
% \setlength{\tabcolsep}{4pt}
% \small
% \begin{minipage}[t]{0.45\textwidth}
% \centering
% \caption{Ablation Study on Seq-Cifar-100 with memory size 500}
% \label{tab:ablation}
% \begin{tabular}{@{}ll@{}}
% \toprule
% \textbf{Variant} & \textbf{Performance} \\
% \midrule
% Ours & 48.58 \\
% w/o $\mathcal{L}_{\text{ce}}$ & 44.97 \\
% w/o $\mathcal{L}_{\text{align}}$ & 23.22 \\
% w/o $\mathcal{L}_{\text{distill}}$ & 19.96 \\
% w/ pre-defined base ETF & 44.99 \\
% w/ fixed ETF & 33.51 \\
% w/ linear classifier & 44.49 \\
% \bottomrule
% \end{tabular}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.45\textwidth}
% \centering
% \caption{Average Time Cost per Epoch and Task (seconds)}
% \label{tab:time_cost}
% \begin{tabular}{@{}lll@{}}
% \toprule
% \textbf{Method} & \textbf{Epoch} & \textbf{Task} \\
% \midrule
% ER~\cite{riemer2019learning} & 3.03 & 151.48 \\
% iCaRL~\cite{rebuffi2017icarl} & 1.60 & 84.77 \\
% GEM~\cite{lopez2017gradient} & 3.26 & 162.64 \\
% GSS~\cite{aljundi2019gradient} & 28.86 & 1425.26 \\
% DER~\cite{buzzega2020dark} & 3.07 & 153.21 \\
% DER++~\cite{buzzega2020dark} & 5.03 & 253.34 \\
% Co\textsuperscript{2}L~\cite{cha2021co2l} & 2.52 & 126.19 \\
% CIIA~\cite{wenprovable} & 2.77 & 138.59 \\
% MNC\textsuperscript{3}L~\cite{dang2024memory} & 2.82 & 141.01 \\
% Ours & 2.91 & 146.71 \\
% \bottomrule
% \end{tabular}
% \end{minipage}
% \end{table}
\begin{wraptable}{r}{0.22\textwidth}
\vspace{-0.4cm}
%\begin{table}[t]
\footnotesize
\centering
\caption{Average training time per epoch and per task (seconds).}
\vspace{-0.2cm}
\scalebox{0.70}{
%\small
\label{tab:time_cost}
\begin{tabular}{@{}lll@{}}
\toprule
\textbf{Method} & \textbf{Epoch} & \textbf{Task} \\
\midrule
ER~\cite{riemer2019learning} & 3.03 & 151.48 \\
iCaRL~\cite{rebuffi2017icarl} & 1.60 & 84.77 \\
GEM~\cite{lopez2017gradient} & 3.26 & 162.64 \\
GSS~\cite{aljundi2019gradient} & 28.86 & 1425.26 \\
DER~\cite{buzzega2020dark} & 3.07 & 153.21 \\
DER++~\cite{buzzega2020dark} & 5.03 & 253.34 \\
Co\textsuperscript{2}L~\cite{cha2021co2l} & 2.52 & 126.19 \\
CIIA~\cite{wenprovable} & 2.77 & 138.59 \\
MNC\textsuperscript{3}L~\cite{dang2024memory} & 2.82 & 141.01 \\
NCT~\cite{NCT1} & 3.41 & 172.02\\
Ours & 2.91 & 146.71 \\
\bottomrule
\end{tabular}
}
\vspace{-0.4cm}
\end{wraptable}
\emph{Training cost.} To evaluate the computation efficiency of our approach, we summarize in
\Cref{tab:time_cost} the average training time per epoch and per task for all approaches. It can be seen that our approach is more efficient than all considered replay-based approaches except iCaRL.
The pronounced latency of GSS arises from its quadratic time complexity in gradient correlation computations ($O(N^2)$ under a buffer size $N$) and memory-intensive operations for maintaining sample importance matrices. In contrast, iCaRL achieves higher efficiency by employing a binary cross-entropy loss, which enjoy a constant time complexity per class ($O(1)$), compared to the linear class-dependent complexity ($O(C)$, $C$ being the total number of classes) of conventional cross-entropy loss.
%This design choice reduces computational overhead significantly while preserving competitive accuracy. Compared with the time cost of recent approaches, our method is faster than the methods.
Meanwhile, the per-epoch training time of our approach is comparable with the recent contrastive-learning based approaches. In summary, by leveraging NC in a principled way, our approach not only significantly outperforms baseline approaches by setting new SOTA performance, but also maintains a high efficiency due to the simplicity of our framework and robustness to buffer sizes. This highlights the great potential of our approach in practical resource-limited CL scenarios.
%Our method achieves competitive computational efficiency, with an average epoch time of 2.91 seconds and task time of 146.71 seconds, outperforming several replay-based (e.g., ER, DER++) and regularization-based (e.g., GEM) approaches while remaining comparable to methods with contrastive losses like MNC\textsuperscript{3}L (2.82s/epoch). This efficiency stems from a streamlined architecture that likely integrates parameter isolation or dynamic sparsity, minimizing redundant computations without significant compromises in stability. The results suggest a favorable tradeoff between speed and plasticity, positioning our method as a practical candidate for resource-constrained continual learning scenarios where computational overhead and catastrophic forgetting must be addressed.
% \begin{table}
% \centering
% \caption{Average Time Cost per Epoch and Task (seconds)}
% \label{tab:time_cost}
% \begin{tabular}{lll}
% \toprule
% \textbf{Method} & \textbf{Epoch (s)} & \textbf{Task (s)} \\
% \midrule
% ER~\cite{riemer2019learning} & 3.03 & 151.48 \\
% iCaRL~\cite{rebuffi2017icarl} & 1.60 & 84.77 \\
% GEM~\cite{lopez2017gradient} & 3.26 & 162.64 \\
% GSS~\cite{aljundi2019gradient} & 28.86 & 1425.26 \\
% DER~\cite{buzzega2020dark} & 3.07 & 153.21 \\
% DER++~\cite{buzzega2020dark} & 5.03 & 253.34 \\
% Co\textsuperscript{2}L~\cite{cha2021co2l} & 2.52 & 126.19 \\
% CIIA~\cite{wenprovable} & 2.77 & 138.59 \\
% MNC\textsuperscript{3}L~\cite{dang2024memory} & 2.82 & 141.01 \\
% Ours & 2.91 & 146.71 \\
% \bottomrule
% \end{tabular}
% \end{table}
\begin{wraptable}{r}{0.63\textwidth}
\vspace{-0.4cm}
%\begin{table}[t]
%\footnotesize
\centering
% \caption{Average training time cost per epoch and per task (seconds).}
% \vspace{-0.2cm}
% \begin{table}
\caption{FAA, FF and total training time comparison with contrastive learning based approaches under their training setups.}
\label{tab:contra_setting}
\resizebox{0.63\textwidth}{!}{
\small
\begin{tabular}{@{}lccccccc@{}}
\toprule
\multirow{2}{*}{Buffer} & \multirow{2}{*}{Method} & \multicolumn{3}{c}{Seq-CIFAR-100} & \multicolumn{3}{c}{Seq-TinyImageNet} \\
\cmidrule(lr){3-5} \cmidrule(lr){6-8}
& & Class-IL & Task-IL & Time(S) & Class-IL & Task-IL & Time(S) \\
\midrule
\multirow{3}{*}{200}
& Co$^2$L ~\cite{cha2021co2l} & 27.38 (67.82) & 42.37 (38.22) & 4362 & 13.88 (73.25) & 42.37 (47.11) & 12494 \\
& MNC$^3$L~\cite{dang2024memory} & 34.04 (52.40) & 59.46(33.66) & 3904 & 15.52 (52.07) & 44.59 (33.76) & 10922 \\
& \textbf{Ours} & \textbf{42.99 (36.07)} & \textbf{85.63 (4.40)} & 1482 & \textbf{27.44 (42.81)} & \textbf{69.09 (9.42)} & 12137 \\
\midrule
\multirow{3}{*}{500}
& Co$^2$L ~\cite{cha2021co2l} & 37.02 (51.23) & 62.44 (26.31) & 4380 & 20.12 (65.15) & 53.04 (39.22) & 12100 \\
& MNC$^3$L~\cite{dang2024memory} & 40.25 (46.09) & 65.85 (24.29) & 3979 & 20.31 (46.08) & 53.46 (26.45) & 12669 \\
& \textbf{Ours} & \textbf{48.94 (24.32)} & \textbf{86.38 (4.35)} & 1588 & \textbf{29.06 (38.58)} & \textbf{69.77 (9.52)} & 12350 \\
\bottomrule
\end{tabular}
}
\vspace{-0.5cm}
\end{wraptable}
\emph{More comparison with contrastive learning based approaches.}
Contrastive learning based approaches usually suffer from high computation costs due to the nature of contrastive learning with data augmentation. In the original implementation of these approaches, e.g., Co\textsuperscript{2}L~\cite{cha2021co2l} and MNC\textsuperscript{3}L~\cite{dang2024memory},
500 training epochs is used for the initial task, and followed by different epochs for each subsequent task, i.e., 100 epochs for Seq-CIFAR-100 and 50 epochs for Seq-TinyImageNet. To further demonstrate the superior performance of our approach, we evaluate our approach under the standard setup in this paper, i.e., 50 epochs for Seq-CIFAR-100 and 100 epochs for Seq-TinyImageNet, against these approaches under their original implementation in \Cref{tab:contra_setting}. Clearly, while the performance of the contrastive learning based approaches improves with more training epochs, our approach still achieves significantly better performance with an even shorter training time.
\emph{Robustness with respect to hyperparameters.} We conduct experiments to demonstrate the impact of $\lambda_1$ and $\lambda_2$ in \Cref{eq:total_loss}. As shown in \Cref{fig:para}, the performance of our approach is very stable under a wide range of hyperparameter selections, indicating the robustness of our approach.
\vspace{-0.1cm}
\section{Conclusion}
\vspace{-0.1cm}
Neural collapse in DNN training characterizes an ideal feature learning target for CL with maximally and equally separated class prototypes through a simplex ETF, whereas recent studies leverage this by predefining a fixed global ETF target, suffering from impracticability and limited performance. To address this and unlock the potential of NC in CL, we propose a novel approach namely progressive neural collapse (ProNC), by initializing the ETF to align with first task leaning and progressively expanding the ETF for each new task without significantly shifting from the previous ETF. This will ensure maximal and equal separability across all encountered classes anytime during CL, without the global knowledge of total class numbers in CL. Building upon ProNC, we introduce a simple and flexible CL framework with minimal changes on standard CL frameworks, where the model is trained to push the learned sample features towards the corresponding ETF target and distillation with data replay is leveraged to further reduce forgetting. Extensive experiments have demonstrated the dominating performance of our approach against state-of-the-art baseline approaches, while maintaining superior efficiency and flexibility.
One limitation here is that we assume clear task boundaries and it is interesting to see how our approach can be extended to more general setups.
We hope this work will serve as initial steps for showcasing the great potentials of NC in facilitating better CL algorithm design and inspire further research in CL community along this interesting direction.
%This paper introduces Progressive Neural Collapse (ProNC), a novel framework that incrementally extends the ETF structure during continual learning, eliminating impractical assumptions about total class counts required by static ETF methods. By progressively expanding the ETF geometry by adding new vertices as new classes are encountered, ProNC maintains maximal angular separation between classes while integrating with exemplar rehearsal to preserve historical knowledge. Extensive evaluations on CIFAR-10, CIFAR-100, and Tiny ImageNet demonstrate ProNC's superior performance over state-of-the-art methods in both class-incremental and task-incremental scenarios. Ablation studies confirm the critical contributions of each component, highlighting the importance of dynamic geometric adaptation and feature-classifier alignment. The computational efficiency analysis further positions ProNC as a practical solution for real-world applications. ProNC bridges the theoretical elegance of Neural Collapse with the practical demands of continual learning, offering a more flexible and robust approach to mitigating catastrophic forgetting.
\clearpage
\bibliography{references.bib}
\bibliographystyle{plain}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \newpage
% \section*{NeurIPS Paper Checklist}
% % %%% BEGIN INSTRUCTIONS %%%
% % The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: {\bf The papers not including the checklist will be desk rejected.} The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page
% % limit.
% % Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
% % \begin{itemize}
% % \item You should answer \answerYes{}, \answerNo{}, or \answerNA{}.
% % \item \answerNA{} means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
% % \item Please provide a short (1–2 sentence) justification right after your answer (even for NA).
% % % \item {\bf The papers not including the checklist will be desk rejected.}
% % \end{itemize}
% % {\bf The checklist answers are an integral part of your paper submission.} They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
% % The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "\answerYes{}" is generally preferable to "\answerNo{}", it is perfectly acceptable to answer "\answerNo{}" provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "\answerNo{}" or "\answerNA{}" is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer \answerYes{} to a question, in the justification please point to the section(s) where related material for the question can be found.
% % IMPORTANT, please:
% % \begin{itemize}
% % \item {\bf Delete this instruction block, but keep the section heading ``NeurIPS Paper Checklist"},
% % \item {\bf Keep the checklist subsection headings, questions/answers and guidelines below.}
% % \item {\bf Do not modify the questions and only use the provided macros for your answers}.
% % \end{itemize}
% %%% END INSTRUCTIONS %%%
% \begin{enumerate}
% \item {\bf Claims}
% \item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We clearly state the motivation behind our design, summarize our key contributions, and present a concise overview of the experimental results in both the abstract and the introduction. These claims accurately reflect the paper's scope and are consistent with the results presented throughout the paper.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the abstract and introduction do not include the claims made in the paper.
% \item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
% \item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
% \item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
% \end{itemize}
% \item {\bf Limitations}
% \item[] Question: Does the paper discuss the limitations of the work performed by the authors?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We have discussed the limitations of our work in the paper.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
% \item The authors are encouraged to create a separate "Limitations" section in their paper.
% \item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
% \item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
% \item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
% \item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
% \item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
% \item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
% \end{itemize}
% \item {\bf Theory assumptions and proofs}
% \item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The theoretical results in the paper are supported by complete and rigorous proofs provided in the appendix.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include theoretical results.
% \item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
% \item All assumptions should be clearly stated or referenced in the statement of any theorems.
% \item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
% \item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
% \item Theorems and Lemmas that the proof relies upon should be properly referenced.
% \end{itemize}
% \item {\bf Experimental result reproducibility}
% \item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We ensure reproducibility by including all necessary details to replicate our main experimental results. This includes releasing our code in the supplementary material along with configuration details.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
% \item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
% \item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
% \item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
% \begin{enumerate}
% \item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
% \item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
% \item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
% \item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
% \end{enumerate}
% \end{itemize}
% \item {\bf Open access to data and code}
% \item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We provide open access to our code in the supplementary material.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that paper does not include experiments requiring code.
% \item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
% \item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
% \item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
% \item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
% \item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
% \end{itemize}
% \item {\bf Experimental setting/details}
% \item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We provide all experimental settings in both the main text and the appendix. The choices of hyper-parameters are justified through empirical tuning.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
% \item The full details can be provided either with the code, in appendix, or as supplemental material.
% \end{itemize}
% \item {\bf Experiment statistical significance}
% \item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We report error bars for all main experimental results in the appendix. The error bars reflect variability due to random seeds in the training process.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
% \item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
% \item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
% \item The assumptions made should be given (e.g., Normally distributed errors).
% \item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
% \item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
% \item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
% \item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
% \end{itemize}
% \item {\bf Experiments compute resources}
% \item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We provide details on the computational environment used for our experiments, e.g., the type of GPUs, in the appendix.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
% \item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
% \item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
% \end{itemize}
% \item {\bf Code of ethics}
% \item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The research presented in this paper fully conforms to the NeurIPS Code of Ethics. We ensure transparency, reproducibility, and responsible research conduct.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
% \item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
% \item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
% \end{itemize}
% \item {\bf Broader impacts}
% \item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We discuss the broader implications and potential benefits of our work in the paper.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that there is no societal impact of the work performed.
% \item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
% \item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
% \item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
% \item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
% \item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
% \end{itemize}
% \item {\bf Safeguards}
% \item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: Our work does not include data or models that have a high risk for misuse.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper poses no such risks.
% \item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
% \item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
% \item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
% \end{itemize}
% \item {\bf Licenses for existing assets}
% \item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: All external assets used in our work, including datasets and code, are properly credited in the paper.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not use existing assets.
% \item The authors should cite the original paper that produced the code package or dataset.
% \item The authors should state which version of the asset is used and, if possible, include a URL.
% \item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
% \item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
% \item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
% \item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
% \item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
% \end{itemize}
% \item {\bf New assets}
% \item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We introduce new assets in the form of code, all of which are thoroughly documented.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not release new assets.
% \item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
% \item The paper should discuss whether and how consent was obtained from people whose asset is used.
% \item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
% \end{itemize}
% \item {\bf Crowdsourcing and research with human subjects}
% \item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: Our paper does not involve crowdsourcing nor research with human subjects.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
% \item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
% \end{itemize}
% \item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
% \item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: This paper does not involve crowdsourcing nor research with human subjects.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
% \item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
% \item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
% \end{itemize}
% \item {\bf Declaration of LLM usage}
% \item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
% %this research?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
% \item Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
% \end{itemize}
% \end{enumerate}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\section{More Experimental Results}
\subsection{Experiment Details}
\subsubsection{Datasets}
\textbf{Seq CIFAR-10}: Based on CIFAR-10 dataset~\cite{krizhevsky2009learning}, this benchmark partitions 10 classes into 5 sequential tasks (2 classes per task), and each class has 5000 and 1000 32 $\times$ 32 images each for training and testing, respectively;
\textbf{Seq CIFAR-100}: Constructed from CIFAR-100~\cite{krizhevsky2009learning}, it comprises 10 tasks with 10 classes per task, and each class has 500 and 100 32 $\times$ 32 images each for training and testing, respectively;
\textbf{Seq TinyImageNet}: Adapted from the TinyImageNet dataset \cite{le2015tiny}, this benchmark divides 200 classes into 10 sequential tasks (20 classes per task), and each class has 500 and 50 64 $\times$ 64 images each for training and testing, respectively.
\subsubsection{Implementation Details}
All experiments are conducted on a single RTX 4090 GPU. For all datasets, we employ a modified ResNet18 network architecture~\cite{he2016deep}, where the kernel size of the first convolutional layer is modified from 7$\times$7 to 3$\times$3, and the stride is changed from 2 to 1. The batch size is set to 32 across all experiments. The number of training epochs is set to 50 for Sequential CIFAR-10 and Sequential CIFAR-100, and 100 for Sequential TinyImageNet.
For ProNC, we consider the following hyperparameters: learning rate ($\eta$), weight of the alignment loss ($\lambda_1$), weight of the distillation loss ($\lambda_2$), momentum ($mom$), and weight decay ($wd$). The hyperparameters are selected through grid search. The chosen hyperparameters are presented in \Cref{tab:Hyper}, and their corresponding search spaces are provided in \Cref{tab:Space}.
\begin{table}[ht]
\caption{Hyperparameters of ProNC}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{|c|c|c|c|}
\hline
Method & Buffer size & Dataset & Hyperparameter \\ \hline
\multirow{6}{*}{Ours} & \multirow{3}{*}{200} & Seq-CIFAR-10 & $\eta$: 0.01, $\lambda_1$: 13, $\lambda_2$: 90, $mom$: 0, $wd$: 0\\ \cline{3-4}
& & Seq-CIFAR-100 & $\eta$: 0.03, $\lambda_1$: 18, $\lambda_2$: 170, $mom$: 0, $wd$: 0\\ \cline{3-4}
& & Seq-Tiny-ImageNet & $\eta$: 0.03, $\lambda_1$:20 , $\lambda_2$: 165, $mom$: 0, $wd$: 0 \\ \cline{2-4}
& \multirow{3}{*}{500} & Seq-CIFAR-10 & $\eta$: 0.01, $\lambda_1$:12 , $\lambda_2$: 80 , $mom$: 0, $wd$: 0\\ \cline{3-4}
& & Seq-CIFAR-100 & $\eta$: 0.03, $\lambda_1$:18 , $\lambda_2$: 170 , $mom$: 0, $wd$: 0\\ \cline{3-4}
& & Seq-Tiny-ImageNet & $\eta$: 0.03, $\lambda_1$:20 , $\lambda_2$: 200, $mom$: 0, $wd$: 0 \\ \hline
\end{tabular}
}
\label{tab:Hyper}
\end{table}
\begin{table}[ht]
\caption{Search Spaces for Hyperparameters}
\centering
\begin{tabular}{lc}
\hline
Hyperparameter & Values \\
\hline
$\eta$ & $\{0.01, 0.03, 0.05. 0.1, 0.3\}$ \\
$\lambda_1$ & $\{5,10,12,13,18,20\}$ \\
$\lambda_2$ & $\{50,75, 80, 90, 165, 170, 200\}$ \\
$mom$ & $\{0, 0.9\}$ \\
$wd$ & $\{0, 10^{-5}, 5\times10^{-5}\}$\\
\hline
\end{tabular}
\label{tab:Space}
\end{table}
Our code is implemented based on the Continual Learning platform Mammoth~\cite{boschini2022class}, which is also provided in the supplementary materials.
\subsection{More Results}
\subsubsection{Final Average Accuracy with Error Bars}
% Accuracy table
\begin{table}[ht]
\caption{Final average accuracies comparison under various setups.
All results are averaged over multiple runs.}
\label{tab:performance}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}lccccccc@{}}
\toprule
\multirow{2}{*}{Buffer} & \multirow{2}{*}{Method} & \multicolumn{2}{c}{Seq-Cifar-10} & \multicolumn{2}{c}{Seq-Cifar-100} & \multicolumn{2}{c}{Seq-Tiny-ImageNet} \\
\cmidrule(lr){3-4} \cmidrule(lr){5-6} \cmidrule(lr){7-8}
& & Class-IL & Task-IL & Class-IL & Task-IL & Class-IL & Task-IL \\
\midrule
\multirow{10}{*}{200} & ER ~\cite{riemer2019learning} & $44.79 \pm 1.86$ & $91.19 \pm 0.94$ & $21.78 \pm 0.48$ & $60.19 \pm 1.01$ & $8.49 \pm 0.16$ & $38.17 \pm 2.00$ \\
& iCaRL ~\cite{rebuffi2017icarl} & $49.02 \pm 3.20$ & $88.99 \pm 2.13$ & $28 \pm 0.91$ & $51.43 \pm 1.47$ & $7.53 \pm 0.79$ & $28.19 \pm 1.47$ \\
& GEM ~\cite{lopez2017gradient} & $25.54 \pm 0.76$ & $90.44 \pm 0.94$ & $20.75 \pm 0.66$ & $58.84 \pm 1.00$ & -- & -- \\
& GSS ~\cite{aljundi2019gradient} & $39.07 \pm 5.59$ & $88.8 \pm 2.89$ & $19.42 \pm 0.29$ & $55.38 \pm 1.34$ & -- & -- \\
& DER ~\cite{buzzega2020dark} & $61.93 \pm 1.79$ & $91.4 \pm 0.92$ & $31.23 \pm 1.38$ & $63.09 \pm 1.09$ & $11.87 \pm 0.78$ & $40.22 \pm 0.67$ \\
& DER++ ~\cite{buzzega2020dark} & $64.88 \pm 1.17$ & $91.92 \pm 0.60$ & $28.13 \pm 0.51$ & $66.80 \pm 0.41$ & $11.34 \pm 1.17$ & $43.06 \pm 1.16$ \\
& Co$^2$L ~\cite{cha2021co2l} & $51.27 \pm 1.86$ & $84.69 \pm 1.52$ & $18.09\pm0.49$ & $49.19\pm0.91$ & $12.95 \pm 0.06$ & $37.07 \pm 1.62$ \\
& CILA [47] & $59.68 \pm 0.65$ & $91.36 \pm 0.08$ & $19.49 \pm 0.53$ & $53.93 \pm 1.02$ & $12.98 \pm 0.10$ & $37.32 \pm 1.87$ \\
& MNC\textsuperscript{3}L~\cite{dang2024memory} & $52.20\pm1.56$ & $85.94\pm0.22$ & $15.81 \pm 0.48$ & $43.91 \pm 0.76$ & $10.57\pm1.66$ & $32.78\pm3.54$ \\
& NCT ~\cite{NCT1} & $51.59 \pm 0.41$ & $80.63 \pm 0.46$ & $26.38 \pm 0.57$ & $75.75 \pm 0.17$ & $10.95 \pm 1.45$ & $52.71 \pm 4.12$ \\
& \textbf{Ours} & $\mathbf{65.58 \pm 0.15}$ & $\mathbf{96.86 \pm 0.10}$ & $\mathbf{42.99 \pm 0.85}$ & $\mathbf{85.63 \pm 0.73}$ & $\mathbf{27.44 \pm 1.00}$ & $\mathbf{69.09 \pm 0.65}$ \\
\midrule
\multirow{10}{*}{500} & ER ~\cite{riemer2019learning} & $57.74 \pm 2.48$ & $93.61 \pm 0.27$ & $27.66 \pm 0.61$ & $66.23 \pm 1.52$ & $9.99 \pm 0.29$ & $48.64\pm0.46$ \\
& iCaRL ~\cite{rebuffi2017icarl} & $47.55 \pm 3.95$ & $88.22 \pm 2.62$ & $33.25 \pm 1.25$ & $58.16 \pm 1.76$ & $9.38 \pm 1.53$ & $ 31.55\pm3.27$ \\
& GEM ~\cite{lopez2017gradient} & $26.2 \pm 1.26$ & $92.16 \pm 0.64$ & $25.54 \pm 0.65$ & $66.31 \pm 0.86$ & -- & -- \\
& GSS ~\cite{aljundi2019gradient} & $49.73 \pm 4.78$ & $91.02 \pm 1.57$ & $21.92 \pm 0.34$ & $60.28 \pm 1.18$ & -- & -- \\
& DER ~\cite{buzzega2020dark} & $70.51 \pm 1.67$ & $93.40 \pm 0.21$ & $41.36 \pm 1.76$ & $71.73 \pm 0.74$ & $17.75 \pm 1.14$ & $51.78\pm0.88$ \\
& DER++ ~\cite{buzzega2020dark} & $72.70 \pm 1.36$ & $93.88 \pm 0.50$ & $38.20 \pm 1.00$ & $74.77 \pm 0.31$ & $19.38 \pm 1.41$ & $51.91 \pm 0.68$ \\
& Co$^2$L ~\cite{cha2021co2l} & $61.78 \pm 4.22$ & $89.51 \pm 2.45$ & $26.64 \pm 1.42$ & $62.32 \pm 0.19$ & $18.71\pm0.84$ & $50.74\pm1.24$ \\
& CILA ~\cite{wenprovable} & $67.82 \pm 0.33$ & $93.29 \pm 0.24 $ & $31.27 \pm 0.17$ & $68.29 \pm 0.46$ & $18.09\pm0.49$ & $49.19\pm0.91$ \\
& MNC\textsuperscript{3}L~\cite{dang2024memory} & $52.20\pm1.56$ & $85.94\pm0.22$ & $22.29 \pm 0.18$ & $56.43 \pm 0.29$ & $11.52\pm0.01$ & $36.32\pm0.05$ \\
& NCT~\cite{NCT1} & $60.93 \pm 0.94$ & $81.27 \pm 0.24$ & $33.84 \pm 0.38$ & $76.06 \pm 0.52$ & $18.24 \pm 0.62$ & $62.30 \pm 0.41$ \\
& \textbf{Ours} & $\mathbf{73.95 \pm 0.68}$ & $\mathbf{96.95 \pm 0.14}$ & $\mathbf{48.94 \pm 0.44}$ & $\mathbf{86.38 \pm 0.43}$ & $\mathbf{29.06 \pm 0.32}$ & $\mathbf{69.77 \pm 0.89}$ \\
\bottomrule
\end{tabular}
}
\end{table}
% Forgetting table
\newpage
\subsection{Final Forgetting with Error Bars}
\begin{table}[ht]
\caption{Final forgetting comparison under various setups.
All results are averaged over multiple runs.}
\label{tab:forgetting}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}lccccccc@{}}
\toprule
\multirow{2}{*}{Buffer} & \multirow{2}{*}{Method} & \multicolumn{2}{c}{Seq-Cifar-10} & \multicolumn{2}{c}{Seq-Cifar-100} & \multicolumn{2}{c}{Seq-Tiny-ImageNet} \\
\cmidrule(lr){3-4} \cmidrule(lr){5-6} \cmidrule(lr){7-8}
& & Class-IL & Task-IL & Class-IL & Task-IL & Class-IL & Task-IL \\
\midrule
\multirow{10}{*}{200} & ER ~\cite{riemer2019learning} & $59.30\pm2.48$ & $6.07\pm1.09$ & $75.06\pm0.63$ & $27.38\pm1.46$ & $76.53\pm0.51$ & $40.47\pm1.54$ \\
& iCaRL ~\cite{rebuffi2017icarl} & $23.52\pm1.27$ & $25.34\pm1.64$ & $47.20\pm1.23$ & $36.20\pm1.85$ & $31.06\pm1.91$ & $42.47\pm2.47$ \\
& GEM ~\cite{lopez2017gradient} & $80.36\pm5.25$ & $9.57\pm2.05$ & $77.40\pm1.09$ & $29.59\pm1.66$ & $-$ & $-$ \\
& GSS ~\cite{aljundi2019gradient} & $72.48\pm4.45$ & $8.49\pm2.05$ & $77.62\pm0.76$ & $32.81\pm1.75$ & $-$ & $-$ \\
& DER ~\cite{buzzega2020dark} & $35.79\pm2.59$ & $6.08\pm0.70$ & $62.72\pm2.69$ & $25.98\pm1.55$ & $64.83\pm1.48$ & $40.43\pm1.05$ \\
& DER++ ~\cite{buzzega2020dark} & $32.59\pm2.32$ & $5.16\pm0.21$ & $60.99\pm1.52$ & $23.91\pm0.55$ & $73.47\pm1.23$ & $39.02\pm0.43$ \\
& Co$^2$L ~\cite{cha2021co2l} & $30.17\pm8.57$ & $2.91\pm5.25$ & $64.14\pm0.69$ & $36.81\pm0.63$ & $62.04\pm0.28$ & $40.75\pm3.55$ \\
& CILA ~\cite{wenprovable} & $37.52\pm5.84$ & $5.49\pm0.74$ & $64.01\pm0.18$ & $33.07\pm0.96$ & $63.11\pm0.61$ & $41.40\pm0.51$ \\
& MNC$^3$L~\cite{dang2024memory} & $33.74\pm1.65$ & $4.90\pm0.15$ & $62.51 \pm 0.40$ & $39.79\pm 0.98$ & $59.68\pm2.02$ & $45.10\pm1.57$ \\
& NCT ~\cite{NCT1} & $\mathbf{22.48\pm19.50}$ & $1.41\pm1.09$ & $\mathbf{27.40\pm1.65}$ & $4.79\pm0.07$ & $49.33\pm4.47$ & $15.88\pm0.95$ \\
& \textbf{Ours} & $32.75\pm4.71$ & $\mathbf{0.65\pm0.08}$ & $36.07\pm0.51$ & $\mathbf{4.40\pm0.82}$ & $\mathbf{42.81\pm0.56}$ & $\mathbf{9.42\pm0.58}$ \\
\midrule
\multirow{10}{*}{500} & ER ~\cite{riemer2019learning} & $43.22\pm2.10$ & $3.50\pm0.53$ & $67.96\pm0.78$ & $17.37\pm1.06$ & $75.21\pm0.54$ & $30.73\pm0.62$ \\
& iCaRL ~\cite{rebuffi2017icarl} & $28.20\pm2.41$ & $22.61\pm3.97$ & $40.99\pm1.02$ & $27.90\pm1.37$ & $37.30\pm1.42$ & $39.44\pm0.84$ \\
& GEM ~\cite{lopez2017gradient} & $78.93\pm6.53$ & $5.60\pm0.96$ & $71.34\pm0.78$ & $20.44\pm1.13$ & $-$ & $-$ \\
& GSS ~\cite{aljundi2019gradient} & $59.18\pm4.00$ & $6.37\pm1.55$ & $74.12\pm0.42$ & $26.57\pm1.34$ & $-$ & $-$ \\
& DER ~\cite{buzzega2020dark} & $24.02\pm1.63$ & $3.72\pm0.55$ & $49.07\pm2.54$ & $25.98\pm1.55$ & $59.95\pm2.31$ & $28.21\pm0.97$ \\
& DER++ ~\cite{buzzega2020dark} & $22.38\pm4.41$ & $4.66\pm1.15$ & $49.18\pm2.19$ & $15.75\pm0.48$ & $58.75\pm1.93$ & $25.47\pm1.03$ \\
& Co$^2$L ~\cite{cha2021co2l} & $17.79\pm3.36$ & $2.65\pm1.00$ & $48.60\pm1.34$ & $23.47\pm0.27$ & $49.64\pm1.14$ & $13.80\pm2.10$ \\
& CILA ~\cite{wenprovable} & $18.22 \pm 1.32$ & $0.65 \pm 0.13$ & $45.67\pm1.02$ & $16.98\pm0.56$ & $64.14\pm0.69$ & $27.83\pm1.32$ \\
& MNC$^3$L~\cite{dang2024memory} & $27.88\pm2.75$ & $3.15 \pm 0.42$ & $46.09\pm0.58$ & $24.29 \pm 0.50$ & $44.96\pm0.09$ & $39.00\pm0.17$ \\
& NCT~\cite{NCT1} & $\mathbf{13.82\pm3.64}$ & $3.84\pm0.35$ & $24.37\pm5.42$ & $\mathbf{3.97\pm0.75}$ & $50.90\pm1.16$ & $\mathbf{6.84\pm0.87}$ \\
& $\textbf{Ours}$ & $26.75\pm1.29$ & $\mathbf{0.24\pm0.14}$ & $\mathbf{24.32\pm0.24}$ & $4.35\pm0.17$ & $\mathbf{38.58\pm0.39}$ & $9.52\pm0.96$ \\
\bottomrule
\end{tabular}
}
\end{table}
\subsection{Generality and Flexibility of Our Designs}
As discussed in Section \ref{sec:results}, our approach introduces a novel and effective feature regularization method based on ProNC, which can be generally incorporated into different CL frameworks. To further verify this, we conduct more experiments by plugging ProNC into established replay-based methods (iCaRL \cite{rebuffi2017icarl} and LUCIR \cite{hou2019learning}) with the alignment loss as an additional regularization term in their objective functions. In particular, we keep all the components the same as the original designs in these studies (including the classifier design) and only incorporate ProNC with the alignment loss.
Table \ref{tab:flex} shows the performance comparison between the original implementations of iCaRL and LUCIR and the versions enhanced with ProNC-based regularization. The experiments are conducted under the Class-IL setting on Seq-CIFAR-100 with a memory buffer size of 500. It can be seen that incorporating the ProNC regularization yields substantial performance improvements, i.e., 16.90\% over iCaRL and 8.28\% over LUCIR, which clearly demonstrates the potential and generality of ProNC as a feature regularization for CL.
% Two tables side by side using minipage
\begin{table}[htbp]
\centering
\begin{minipage}{0.48\textwidth}
\centering
\caption{Impact of ProNC regularization term}
\label{tab:flex}
\begin{tabular}{@{}lc@{}}
\toprule
\textbf{Methods} & \textbf{FAA} \\
\midrule
iCaRL~\cite{rebuffi2017icarl} & 33.25 \\
iCaRL with ProNC & 38.87 \\
LUCIR~\cite{hou2019learning} & 37.68 \\
LUCIR with ProNC & 40.80 \\
\bottomrule
\end{tabular}
\end{minipage}%
\hfill
\begin{minipage}{0.48\textwidth}
\centering
\caption{Impact of different loss design}
\label{tab:loss}
\begin{tabular}{@{}lc@{}}
\toprule
\textbf{Loss Combination} & \textbf{FAA} \\
\midrule
$\mathcal{L}_{\text{align}}$ + $\mathcal{L}_{\text{distill}}$ & 48.94 \\
$\mathcal{L}_{\text{align}}$ + $l_{2}$-Norm loss & 49.42 \\
$l_{2}$-Norm loss + $\mathcal{L}_{\text{distill}}$ & 48.61 \\
$l_{2}$-Norm loss + $l_{2}$-Norm loss & 48.66 \\
\bottomrule
\end{tabular}
\end{minipage}
\end{table}
Moreover, our approach is also very flexible in the sense that the loss terms in the final objective function can be replaced by other designs. To show this, we replace the cosine-similarity in the loss functions, i.e., $\mathcal{L}_{\text{align}}$ and $\mathcal{L}_{\text{distill}}$, by using a standard $l_{2}$-norm, and conduct experiments under the Class-IL setting on Seq-CIFAR-100 with a memory buffer size of 500.
As shown in Table \ref{tab:loss},
the performance of our approach is stable under different design combinations when we replace the cosine similarity in any of the two loss terms, which further corroborate the flexibility of our approach. In principle, our approach can be generally applied with a loss function that seeks to minimize the distance between the learned features and the ETF target/old features.
%The results demonstrate that our approach's effectiveness is not dependent on a specific loss function design. Rather, any loss function, which is capable of reducing the distance between two normalized features, can be utilized within our framework while achieving comparable performance.
\section{Proof of Theorem 1}
Based on the definition of ETF, we know that $\mathbf{E}^*$ can be expressed as
\begin{equation}\label{opti_ETF}
\mathbf{E}^* = \sqrt{\frac{K_1}{K_1-1}} \mathbf{U^*} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right),
\end{equation}
where $\mathbf{U^*}\in \mathbb{R}^{d \times K_1}$ denotes the corresponding orthogonal matrix.
Because $\mathbf{U'}=\sqrt{\frac{K_1-1}{K_1}}\tilde{\bm{M}}_{K_1}\left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right)$, we can have
\begin{equation}\label{eq:M}
\tilde{\bm{M}} = \sqrt{\frac{K_1}{K_1-1}} \mathbf{U'} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right).
\end{equation}
This is true since $\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top$ is an orthogonal projection matrix with rank $K_1-1$ and also equal to its pseudoinverse.
It is clear that
\begin{equation*}
\begin{split}
\tilde{\bm{M}}_{K_1} - \mathbf{E}^* &= {\sqrt{\frac{K_1}{K_1-1}} \mathbf{U'} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right)} - {\sqrt{\frac{K_1}{K_1-1}} \mathbf{U^*} \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right)} \\
&= \sqrt{\frac{K_1}{K_1-1}}\left(\mathbf{U'} - \mathbf{U^*} \right)
\left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right).
\end{split}
\end{equation*}
Since $\sqrt{\frac{K_1}{K_1-1}}$ is a scalar and $\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top$ is a fixed projection matrix, finding $\mathbf{E}^*$ to minimize $\| \tilde{\bm{M}}_{K_1} - \mathbf{E}^*\|_F^2$ is equivalent to finding an orthogonal matrix $\mathbf{U^*}$ that minimizes $\left\| \mathbf{U'} - \mathbf{U^*}\right\|_F^2$.
To this end, the following lemma \cite{zou2006sparse} characterizes the nearest orthogonal matrix to any real matrix.
\begin{lemma}[Nearest Orthogonal Matrix via SVD]\label{NOrth}
\label{thm:nearest-orthogonal}
Let \( \mathbf{A} \in \mathbb{R}^{m \times n} \) be a real matrix with Singular Value Decomposition (SVD) \( \mathbf{A} = \mathbf{W} \mathbf{\Sigma} \mathbf{V}^\top \), where \( \mathbf{W} \in \mathbb{R}^{m \times s} \) and \( \mathbf{V} \in \mathbb{R}^{s \times n} \) are orthogonal matrices, and \( \mathbf{\Sigma} \in \mathbb{R}^{s \times s} \) is a diagonal matrix of singular values with \( s = \min(m, n) \). The nearest orthogonal matrix \( \mathbf{Q} \in \mathbb{R}^{m \times n} \) to \( \mathbf{A} \) under Frobenius norm is:
\[
\mathbf{Q} = \mathbf{W}\mathbf{V}^\top.
\]
\end{lemma}
Therefore, given the SVD of $\mathbf{U'}$ as $\mathbf{W\Sigma V}^\top$, the orthogonal matrix closest to $\mathbf{U'}$ can be represented as $\mathbf{U^*}=\mathbf{W}\mathbf{V}^\top$, which will lead to an ETF matrix
\begin{align*}
\mathbf{E}^*=\sqrt{\frac{K_1}{K_1-1}} \mathbf{W}\mathbf{V}^\top \left(\mathbf{I}_{K_1} - \frac{1}{K_1} \mathbf{1}_{K_1} \mathbf{1}_{K_1}^\top \right).
\end{align*}
\end{document}
\begin{thebibliography}{10}
\bibitem{aljundi2018memory}
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.
\newblock Memory aware synapses: Learning what (not) to forget.
\newblock In {\em Proceedings of the European conference on computer vision (ECCV)}, pages 139--154, 2018.
\bibitem{aljundi2019gradient}
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.
\newblock Gradient based sample selection for online continual learning.
\newblock {\em Advances in neural information processing systems}, 32, 2019.
\bibitem{borsos2020coresets}
Zal{\'a}n Borsos, Mojmir Mutny, and Andreas Krause.
\newblock Coresets via bilevel optimization for continual learning and streaming.
\newblock {\em Advances in neural information processing systems}, 33:14879--14890, 2020.
\bibitem{boschini2022class}
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara.
\newblock Class-incremental continual learning into the extended der-verse.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence}, 2022.
\bibitem{buzzega2020dark}
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.
\newblock Dark experience for general continual learning: a strong, simple baseline.
\newblock {\em Advances in neural information processing systems}, 33:15920--15930, 2020.
\bibitem{cha2021co2l}
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin.
\newblock Co$^2$l: Contrastive continual learning.
\newblock In {\em Proceedings of the IEEE/CVF International conference on computer vision}, pages 9516--9525, 2021.
\bibitem{chaudhry2018riemannian}
Arslan Chaudhry, Puneet~K Dokania, Thalaiyasingam Ajanthan, and Philip~HS Torr.
\newblock Riemannian walk for incremental learning: Understanding forgetting and intransigence.
\newblock In {\em Proceedings of the European conference on computer vision (ECCV)}, pages 532--547, 2018.
\bibitem{chaudhry2018efficient}
Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.
\newblock Efficient lifelong learning with a-gem.
\newblock {\em arXiv preprint arXiv:1812.00420}, 2018.
\bibitem{chrysakis2020online}
Aristotelis Chrysakis and Marie-Francine Moens.
\newblock Online continual learning from imbalanced data.
\newblock In {\em International Conference on Machine Learning}, pages 1952--1961. PMLR, 2020.
\bibitem{dang2024memory}
Trung-Anh Dang, Vincent Nguyen, Ngoc-Son Vu, and Christel Vrain.
\newblock Memory-efficient continual learning with neural collapse contrastive.
\newblock {\em arXiv preprint arXiv:2412.02865}, 2024.
\bibitem{farajtabar2020orthogonal}
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li.
\newblock Orthogonal gradient descent for continual learning.
\newblock In {\em International conference on artificial intelligence and statistics}, pages 3762--3773. PMLR, 2020.
\bibitem{gao2022r}
Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang.
\newblock R-dfcil: Relation-guided representation learning for data-free class incremental learning.
\newblock In {\em European Conference on Computer Vision}, pages 423--439. Springer, 2022.
\bibitem{gidaris2018dynamic}
Spyros Gidaris and Nikos Komodakis.
\newblock Dynamic few-shot visual learning without forgetting.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 4367--4375, 2018.
\bibitem{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 770--778, 2016.
\bibitem{hou2019learning}
Saihui Hou, Xinyu Pan, Chen~Change Loy, Zilei Wang, and Dahua Lin.
\newblock Learning a unified classifier incrementally via rebalancing.
\newblock In {\em 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pages 831--839. IEEE, 2019.
\bibitem{hung2019compacting}
Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen.
\newblock Compacting, picking and growing for unforgetting continual learning.
\newblock {\em Advances in neural information processing systems}, 32, 2019.
\bibitem{kirkpatrick2017overcoming}
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei~A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et~al.
\newblock Overcoming catastrophic forgetting in neural networks.
\newblock {\em Proceedings of the national academy of sciences}, 114(13):3521--3526, 2017.
\bibitem{krizhevsky2009learning}
Alex Krizhevsky, Geoffrey Hinton, et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock 2009.
\bibitem{le2015tiny}
Yann Le, and Xuan Yang.
\newblock Tiny imagenet visual recognition challenge.
\newblock {\em CS 231N}, 7(7):3, 2015.
\bibitem{lee2017overcoming}
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang.
\newblock Overcoming catastrophic forgetting by incremental moment matching.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{li2017learning}
Zhizhong Li and Derek Hoiem.
\newblock Learning without forgetting.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence}, 40(12):2935--2947, 2017.
\bibitem{lin2022beyond}
Sen Lin, Li~Yang, Deliang Fan, and Junshan Zhang.
\newblock Beyond not-forgetting: Continual learning with backward knowledge transfer.
\newblock {\em Advances in Neural Information Processing Systems}, 35:16165--16177, 2022.
\bibitem{lin2022trgp}
Sen Lin, Li~Yang, Deliang Fan, and Junshan Zhang.
\newblock Trgp: Trust region gradient projection for continual learning.
\newblock {\em arXiv preprint arXiv:2202.02931}, 2022.
\bibitem{liu2022continual}
Hao Liu and Huaping Liu.
\newblock Continual learning with recursive gradient optimization.
\newblock {\em arXiv preprint arXiv:2201.12522}, 2022.
\bibitem{lopez2017gradient}
David Lopez-Paz and Marc'Aurelio Ranzato.
\newblock Gradient episodic memory for continual learning.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{masana2022class}
Marc Masana, Xialei Liu, Bart{\l}omiej Twardowski, Mikel Menta, Andrew~D Bagdanov, and Joost Van De~Weijer.
\newblock Class-incremental learning: survey and performance evaluation on image classification.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence}, 45(5):5513--5533, 2022.
\bibitem{mccloskey1989catastrophic}
Michael McCloskey and Neal~J Cohen.
\newblock Catastrophic interference in connectionist networks: The sequential learning problem.
\newblock In {\em Psychology of learning and motivation}, volume~24, pages 109--165. Elsevier, 1989.
\bibitem{mcdonnell2023ranpac}
Mark~D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton Van~den Hengel.
\newblock Ranpac: Random projections and pre-trained models for continual learning.
\newblock {\em Advances in Neural Information Processing Systems}, 36:12022--12053, 2023.
\bibitem{nguyen2017variational}
Cuong~V Nguyen, Yingzhen Li, Thang~D Bui, and Richard~E Turner.
\newblock Variational continual learning.
\newblock {\em arXiv preprint arXiv:1710.10628}, 2017.
\bibitem{papyan2020prevalence}
Vardan Papyan, XY~Han, and David~L Donoho.
\newblock Prevalence of neural collapse during the terminal phase of deep learning training.
\newblock {\em Proceedings of the National Academy of Sciences}, 117(40):24652--24663, 2020.
\bibitem{peng2022few}
Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian~C Lovell.
\newblock Few-shot class-incremental learning from an open-set perspective.
\newblock In {\em European Conference on Computer Vision}, pages 382--397. Springer, 2022.
\bibitem{rebuffi2017icarl}
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph~H Lampert.
\newblock icarl: Incremental classifier and representation learning.
\newblock In {\em Proceedings of the IEEE conference on Computer Vision and Pattern Recognition}, pages 2001--2010, 2017.
\bibitem{riemer2018learning}
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.
\newblock Learning to learn without forgetting by maximizing transfer and minimizing interference.
\newblock {\em arXiv preprint arXiv:1810.11910}, 2018.
\bibitem{riemer2019learning}
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.
\newblock Learning to learn without forgetting by maximizing transfer and minimizing interference.
\newblock In {\em International Conference on Learning Representations}. International Conference on Learning Representations, ICLR, 2019.
\bibitem{rolnick2019experience}
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.
\newblock Experience replay for continual learning.
\newblock {\em Advances in neural information processing systems}, 32, 2019.
\bibitem{rusu2016progressive}
Andrei~A Rusu, Neil~C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.
\newblock Progressive neural networks.
\newblock {\em arXiv preprint arXiv:1606.04671}, 2016.
\bibitem{saha2021gradient}
Gobinda Saha, Isha Garg, and Kaushik Roy.
\newblock Gradient projection memory for continual learning.
\newblock {\em arXiv preprint arXiv:2103.09762}, 2021.
\bibitem{shen2025in2nect}
Junao Shen, Qiyun Hu, Tian Feng, Xinyu Wang, Hui Cui, Sensen Wu, and Wei Zhang.
\newblock In2nect: Inter-class and intra-class neural collapse tuning for semantic segmentation of imbalanced remote sensing images.
\newblock In {\em Proceedings of the AAAI Conference on Artificial Intelligence}, volume~39, pages 6814--6822, 2025.
\bibitem{shin2017continual}
Hanul Shin, Jung~Kwon Lee, Jaehong Kim, and Jiwon Kim.
\newblock Continual learning with deep generative replay.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{smith2023coda}
James~Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira.
\newblock Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 11909--11919, 2023.
\bibitem{tiwari2022gcr}
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy.
\newblock Gcr: Gradient coreset based replay buffer selection for continual learning.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 99--108, 2022.
\bibitem{wang2018cosface}
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.
\newblock Cosface: Large margin cosine loss for deep face recognition.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 5265--5274, 2018.
\bibitem{wang2022memory}
Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhenguo Li, Yi~Zhong, and Jun Zhu.
\newblock Memory replay with data compression for continual learning.
\newblock {\em arXiv preprint arXiv:2202.06592}, 2022.
\bibitem{wang2021training}
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu.
\newblock Training networks in null space of feature covariance for continual learning.
\newblock In {\em Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition}, pages 184--193, 2021.
\bibitem{wang2022dualprompt}
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et~al.
\newblock Dualprompt: Complementary prompting for rehearsal-free continual learning.
\newblock In {\em European conference on computer vision}, pages 631--648. Springer, 2022.
\bibitem{wang2022learning}
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.
\newblock Learning to prompt for continual learning.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 139--149, 2022.
\bibitem{wenprovable}
Yichen Wen, Zhiquan Tan, Kaipeng Zheng, Chuanlong Xie, and Weiran Huang.
\newblock Provable contrastive continual learning.
\newblock In {\em Forty-first International Conference on Machine Learning}, 2019.
\bibitem{wu2024linguistic}
Robert Wu and Vardan Papyan.
\newblock Linguistic collapse: Neural collapse in (large) language models.
\newblock {\em Advances in Neural Information Processing Systems}, 37:137432--137473, 2024.
\bibitem{xie2023neural}
Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He.
\newblock Neural collapse inspired attraction--repulsion-balanced loss for imbalanced learning.
\newblock {\em Neurocomputing}, 527:60--70, 2023.
\bibitem{xie2025hard}
Lu~Xie, Weigang Li, and Yuntao Zhao.
\newblock Hard example learning based on neural collapse for class-imbalanced semantic segmentation.
\newblock {\em Applied Soft Computing}, page 112755, 2025.
\bibitem{yang2021grown}
Li~Yang, Sen Lin, Junshan Zhang, and Deliang Fan.
\newblock Grown: Grow only when necessary for continual learning.
\newblock {\em arXiv preprint arXiv:2110.00908}, 2021.
\bibitem{yang2022inducing}
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao.
\newblock Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network?
\newblock {\em Advances in neural information processing systems}, 35:37991--38002, 2022.
\bibitem{yang2023neural}
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao.
\newblock Neural collapse inspired feature-classifier alignment for few-shot class incremental learning.
\newblock {\em arXiv preprint arXiv:2302.03004}, 2023.
\bibitem{NCT1}
Yibo Yang, Haobo Yuan, Xiangtai Li, Jianlong Wu, Lefei Zhang, Zhouchen Lin, Philip Torr, Dacheng Tao, and Bernard Ghanem.
\newblock Neural collapse terminus: A unified solution for class incremental learning and its variants.
\newblock {\em arXiv preprint arXiv:2308.01746}, 2023.
\bibitem{yoon2017lifelong}
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung~Ju Hwang.
\newblock Lifelong learning with dynamically expandable networks.
\newblock {\em arXiv preprint arXiv:1708.01547}, 2017.
\bibitem{zenke2017continual}
Friedemann Zenke, Ben Poole, and Surya Ganguli.
\newblock Continual learning through synaptic intelligence.
\newblock In {\em International conference on machine learning}, pages 3987--3995. PMLR, 2017.
\bibitem{zeno2021task}
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry.
\newblock Task-agnostic continual learning using online variational bayes with fixed-point updates.
\newblock {\em Neural Computation}, 33(11):3139--3177, 2021.
\bibitem{zhou2025revisiting}
Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu.
\newblock Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need.
\newblock {\em International Journal of Computer Vision}, 133(3):1012--1032, 2025.
\bibitem{zhu2024neural}
Didi Zhu, Zexi Li, Min Zhang, Junkun Yuan, Jiashuo Liu, Kun Kuang, and Chao Wu.
\newblock Neural collapse anchored prompt tuning for generalizable vision-language models.
\newblock In {\em Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}, pages 4631--4640, 2024.
\bibitem{zou2006sparse}
Hui Zou, Trevor Hastie, and Robert Tibshirani.
\newblock Sparse principal component analysis.
\newblock {\em Journal of computational and graphical statistics}, 15(2):265--286, 2006.
\end{thebibliography}