A mini-batch training strategy for deep subspace clustering networks
Extracted source text · arXiv 2507.19917
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% The authors should use one of these tracks.
% Before accepting by the NeurIPS conference, select one of the options below.
% 0. "default" for submission
% \usepackage{neurips_2025}
% the "default" option is equal to the "main" option, which is used for the Main Track with double-blind reviewing.
% 1. "main" option is used for the Main Track
% \usepackage[main]{neurips_2025}
% 2. "position" option is used for the Position Paper Track
% \usepackage[position]{neurips_2025}
% 3. "dandb" option is used for the Datasets & Benchmarks Track
% \usepackage[dandb]{neurips_2025}
% 4. "creativeai" option is used for the Creative AI Track
% \usepackage[creativeai]{neurips_2025}
% 5. "sglblindworkshop" option is used for the Workshop with single-blind reviewing
% \usepackage[sglblindworkshop]{neurips_2025}
% 6. "dblblindworkshop" option is used for the Workshop with double-blind reviewing
% \usepackage[dblblindworkshop]{neurips_2025}
% After being accepted, the authors should add "final" behind the track to compile a camera-ready version.
% 1. Main Track
% \usepackage[main, final]{neurips_2025}
% 2. Position Paper Track
% \usepackage[position, final]{neurips_2025}
% 3. Datasets & Benchmarks Track
% \usepackage[dandb, final]{neurips_2025}
% 4. Creative AI Track
% \usepackage[creativeai, final]{neurips_2025}
% 5. Workshop with single-blind reviewing
% \usepackage[sglblindworkshop, final]{neurips_2025}
% 6. Workshop with double-blind reviewing
% \usepackage[dblblindworkshop, final]{neurips_2025}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
% For workshops (5., 6.), the authors should add the name of the workshop, "\workshoptitle" command is used to set the workshop title.
% \workshoptitle{WORKSHOP TITLE}
% "preprint" option is used for arXiv or other preprint submissions
\usepackage[preprint]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% \usepackage{graphicx}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage{bbm}
\usepackage{verbatim}
\usepackage[pdftex]{graphicx}
\title{A mini-batch training strategy for deep subspace clustering networks}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Yuxuan Jiang $^1$\thanks{E-mail: jiangyuxuan372@gmail.com} \quad Chenwei Yu $^1$ \quad Zhi Lin $^1$ \quad Xiaolan Liu $^2$\\
% Department of Computer Science\\
% The Hong Kong University of Science and Technology\\
% Pittsburgh, PA 15213 \\
% \texttt{jiangyuxuan372@gmail.com} \\
% examples of more authors
% \And
% Affiliation \\
% Address \\
% \texttt{email} \\
% \AND
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
$^1$The Hong Kong University of Science and Technology \\
$^2$South China University of Technology \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
% Splitting training data into mini-batches is a fundamental strategy for most machine learning tasks, which reduces computational cost and makes training deeper architectures feasible. However, for deep subspace clustering networks (DSC) which usually consist an autoencoder and a self-expressive module, existing works typically put all data into one batch, restricting its application to low-resolution images and shallow networks. This problem is caused by the self-expressive module which is used for learning a self-representation coefficient matrix for spectral clustering. In this paper, (1) we propose a mini-batch training strategy for DSC by maintaining a memory bank which stores all features.
% Our approach makes it possible to train deeper architectures with high-resolution images in subspace clustering tasks. (2) To efficiently fine-tuning large-scale pre-trained encoders in subspace clustering, we devise a decoder-free framework and leverage contrastive learning instead of autoencoding for representation learning. Experiments demonstrate that our approach not only achieves comparable performance with full-batch methods, but outperforms other state-of-the-art subspace clustering methods on COIL100 and ORL datasets by fine-tuning deep networks.
Mini-batch training is a cornerstone of modern deep learning, offering computational efficiency and scalability for training complex architectures. However, existing deep subspace clustering (DSC) methods, which typically combine an autoencoder with a self-expressive layer, rely on full-batch processing. The bottleneck arises from the self-expressive module, which requires representations of the entire dataset to construct a self-representation coefficient matrix. In this work, we introduce a mini-batch training strategy for DSC by integrating a memory bank that preserves global feature representations. Our approach enables scalable training of deep architectures for subspace clustering with high-resolution images, overcoming previous limitations. Additionally, to efficiently fine-tune large-scale pre-trained encoders for subspace clustering, we propose a decoder-free framework that leverages contrastive learning instead of autoencoding for representation learning. This design not only eliminates the computational overhead of decoder training but also provides competitive performance. Extensive experiments demonstrate that our approach not only achieves performance comparable to full-batch methods, but outperforms other state-of-the-art subspace clustering methods on the COIL100 and ORL datasets by fine-tuning deep networks.
\end{abstract}
\section{Introduction}
\input{sec/1Intro}
\section{Related work}
\input{sec/2Related}
\section{Method}
\input{sec/3Method}
\section{Experiment}
\input{sec/4Experiment}
\section{Conclusion}
In this paper, we propose a mini-batch training strategy for deep subspace clustering networks. Compared with traditional full-batch methods, our memory bank-based approach achieves competitive performance with lower memory cost. Based on it, we design a decoder-free framework which replaces autoencoding with contrastive learning, containing less parameters while achieving comparable results. Extensive experiments demonstrate the feasibility and effectiveness of fine-tuning large-scale pre-trained models for subspace clustering. For now, our method cannot deal with large scale datasets, mainly due to the limitation of self-expressive subspace clustering. In future work, we aim to address this problem to achieve broader applicability.
\section{Broader Impact}
Our mini-batch DSC framework enables efficient subspace clustering of high-dimensional visual data like medical images and satellite imagery. By eliminating the need for labeled data, the method could benefit domains where annotation is expensive or impractical. Potential considerations include: privacy implications when processing sensitive unlabeled datasets (e.g., facial images), and biases in cluster structures that may emerge from pre-trained encoders. Besides our decoder-free design reduces energy consumption compared to traditional autoencoder approaches, which is more environmentally friendly.
% \clearpage
\bibliographystyle{abbrvnat}
\bibliography{sec/reference}
\clearpage
\appendix
\input{sec/Appendix}
% \newpage
% \section*{NeurIPS Paper Checklist}
\begin{comment}
%%% BEGIN INSTRUCTIONS %%%
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: {\bf The papers not including the checklist will be desk rejected.} The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page
limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
\begin{itemize}
\item You should answer \answerYes{}, \answerNo{}, or \answerNA{}.
\item \answerNA{} means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
\item Please provide a short (1–2 sentence) justification right after your answer (even for NA).
% \item {\bf The papers not including the checklist will be desk rejected.}
\end{itemize}
{\bf The checklist answers are an integral part of your paper submission.} They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "\answerYes{}" is generally preferable to "\answerNo{}", it is perfectly acceptable to answer "\answerNo{}" provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "\answerNo{}" or "\answerNA{}" is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer \answerYes{} to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
\begin{itemize}
\item {\bf Delete this instruction block, but keep the section heading ``NeurIPS Paper Checklist"},
\item {\bf Keep the checklist subsection headings, questions/answers and guidelines below.}
\item {\bf Do not modify the questions and only use the provided macros for your answers}.
\end{itemize}
%%% END INSTRUCTIONS %%%
\end{comment}
\begin{comment}
\begin{enumerate}
\item {\bf Claims}
\item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our main contribution is a mini-batch training strategy for deep subspace clustering, which is clearly mentioned in both abstract and introduction.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the abstract and introduction do not include the claims made in the paper.
\item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
\item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
\item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
\item {\bf Limitations}
\item[] Question: Does the paper discuss the limitations of the work performed by the authors?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We discuss the limitations in the section of conclusion, mainly about the scale of datasets.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
\item The authors are encouraged to create a separate "Limitations" section in their paper.
\item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
\item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
\item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
\item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
\item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
\item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
\item {\bf Theory assumptions and proofs}
\item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: This paper does not include theoretical results.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include theoretical results.
\item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
\item All assumptions should be clearly stated or referenced in the statement of any theorems.
\item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
\item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
\item Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
\item {\bf Experimental result reproducibility}
\item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We provide the experimental details (including training strategy, hyper-parameters) to ensure the reproducibility.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
\item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
\item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
\item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
\item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
\item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
\item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
\item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
\item {\bf Open access to data and code}
\item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
\item[] Answer: \answerNo{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The code and data have not sorted out yet, but will be released soon.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that paper does not include experiments requiring code.
\item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
\item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
\item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
\item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
\item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
\item {\bf Experimental setting/details}
\item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: All the experimental details are included in section \ref{sec_experiment}.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
\item The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
\item {\bf Experiment statistical significance}
\item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
\item[] Answer: \answerNo{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our experiments exhibit minimal randomness. And Prior works in deep subspace clustering typically do not report error bars, as performance variance across runs is negligible under controlled settings.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
\item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
\item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
\item The assumptions made should be given (e.g., Normally distributed errors).
\item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
\item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
\item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
\item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
\item {\bf Experiments compute resources}
\item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: All experiments can run within 16GB of GPU memory (the specific GPU used is already mentioned in the paper), and most of them complete within an hour.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
\item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
\item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
\item {\bf Code of ethics}
\item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We strictly follow the NeurIPS Code of Ethics.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
\item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
\item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
\item {\bf Broader impacts}
\item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our work may have privacy implications on sensitive unlabeled datasets and the decoder-free framework is more envirmentally friendly.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that there is no societal impact of the work performed.
\item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
\item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
\item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
\item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
\item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
\item {\bf Safeguards}
\item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our work does not have such risk for misuse.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper poses no such risks.
\item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
\item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
\item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
\item {\bf Licenses for existing assets}
\item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The code and dataset of DSC \cite{DSC} are used and mentioned correctly.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not use existing assets.
\item The authors should cite the original paper that produced the code package or dataset.
\item The authors should state which version of the asset is used and, if possible, include a URL.
\item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
\item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
\item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
\item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
\item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
\item {\bf New assets}
\item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: This paper does not release new assets.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not release new assets.
\item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
\item The paper should discuss whether and how consent was obtained from people whose asset is used.
\item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
\item {\bf Crowdsourcing and research with human subjects}
\item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Not related to our work.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
\item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
\item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
\item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our work does not involve crowdsourcing.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
\item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
\item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
\item {\bf Declaration of LLM usage}
\item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
%this research?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The core methodology does not involve LLMs.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
\item Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
\end{itemize}
\end{enumerate}
\end{comment}
\end{document}
% Subspace clustering is a technique in machine learning and data analysis that seeks to identify distinct subspaces within high-dimensional data, where each subspace corresponds to a meaningful cluster \cite{Subspace_clustering}. Unlike traditional clustering methods that group data points based on direct similarities, subspace clustering recognizes that data often lie in multiple lower-dimensional subspaces rather than a single, uniform distribution. Subspace clustering is widely applied in computer vision, image segmentation, and pattern recognition.
% There are many influential spectral clustering-based methods such as sparse subspace clustering (SSC) \cite{SSC} and low-rank representation (LRR) \cite{LRR}, which serve as the foundation for many subsequent studies. This type of clustering method focuses on building a good affinity matrix for spectral clustering. Compared with traditional clustering algorithms such as k-means, they achieve better clustering performance.
% However, one of their limitations is that SSC and LRR are based on the idea of writing a data point as a linear or affine combination of neighboring data points. In practical problems, data often involve non-linear relationships. A straightforward approach is adopting autoencoders for nonlinear embedding learning, and then conducting SSC or other methods in the hidden space. Compared with SSC, although this approach achieves significant improvement, it decouples clustering objectives from feature learning.
% Later, \cite{DSC} proposed to add a self-expressive layer between the encoder and the decoder, the coefficient of which is used for constructing an affinity matrix. Thus DSC jointly optimizes the autoencoder and the clustering objective. Despite the simplicity of self-expressive module, it requires features of all data in the forward process. DSC and following works typically downsample input images and load them all in a single batch, which prevents its application to deeper networks.
% Some works have tried for solving this problem \cite{efficientDSC} \cite{deepSC} by replacing the self-expressive module with some new mechanisms, making the implementation more difficult. In this paper, we directly generalize DSC to a mini-batch training strategy by maintaining a memory bank. Hence our method makes it possible to training deeper networks in subspace clustering.
% Conventional deep subspace clustering methods use shallow networks for each datasets, and train them from scratch. But nowadays there are many powerful pre-trained deep networks such as ResNets \cite{resnet} and ViTs \cite{vit}. The full-batch training strategy obstacles fine-tuning them for subspace clustering tasks. Although we can fine-tune these encoders by simply combining them with a decoder, and conduct mini-batch gradient descent with MSE loss, we find such methods bring trivial improvement as the learned features are not well-adapted to subspace clustering. With the help of our mini-batch approach, we can take the advantage of introducing clustering objective into feature learning. Moreover, considering the absence of pre-trained decoders, we introduce a novel decoder-free framework that replaces autoencoder with contrastive learning for representation learning. Experimental results demonstrate that our approach achieves state-of-the-art performance on two challenging datasets among subspace clustering methods.
% To conclude, the main contributions of this paper are as follows:
% \begin{enumerate}
% \item We generalize traditional full-batch based deep subspace clustering methods to a mini-batch training framework, showing comparable performance. Moreover, for more efficiently fine-tuning pre-trained encoders, we propose a contrastive learning based framework.
% \item We propose a balancing strategy between batch size and learning rate for stabilizing the training process.
% \item We demonstrate the feasibility for fine-tuning large-scale pre-trained deep networks in subspace clustering tasks, showing its superiority over the state-of-the-art subspace clustering methods.
% \end{enumerate}
In recent years, advances in technology have led to the generation of massive high-dimensional data for computer vision and machine learning. Such data pose challenges, including difficulty in analysis and high computational costs. Fortunately, these high-dimensional data often lie in a union of low-dimensional subspaces. For example, face images captured under varying lighting conditions reside in a nine-dimensional subspace \cite{subspace_lambertian}, and the motion trajectories of rigidly moving objects in a video belong to different subspaces of at most three dimensions \cite{subspace_motion}. To effectively uncover these underlying structures, subspace clustering has emerged as a powerful technique for grouping data points based on their intrinsic subspace membership. Over the past several decades, it has had wide applications in computer vision \citep{sc_cv1}\cite{sc_cv2}\cite{li2015temporal}\cite{multitvc}, image processing \cite{sc_ip1} and systems theory \cite{sc_st1}.
% Clustering is one branch of unsupervised learning, aiming to group data into a given number of clusters. A popular algorithm is subspace clustering which has wide applications in computer vision \citep{sc_cv1}\cite{sc_cv2}\cite{sc_cv3}\cite{multitvc}, image processing \cite{sc_ip1} and systems theory \cite{sc_st1}. It aims to cluster the data into multiple subspaces and find a low-dimensional subspace fitting each group of points. Several subspace clustering approaches have been proposed, including algebraic methods \cite{sc_algebra1}\cite{sc_algebra2}, iterative methods \cite{sc_itera1}, statistical methods \cite{sc_stat1}, and spectral clustering-based methods \cite{sc_spetral1}\cite{sc_spetral2}.
Some classical subspace clustering methods, such as Sparse Subspace Clustering (SSC) \cite{SSC} and Low-Rank Representation (LRR) \cite{LRR}, cluster data drawn from multiple low-dimensional linear or affine subspaces embedded in a high-dimensional space. However, real-world datasets usually contain nonlinear relationships. Some works have tried to nonlinearly map input data into latent space for clustering using kernel tricks \cite{kernel_subspace1} \cite{kernel_subspace2} or autoencoders, but these two-step approaches separate feature learning from clustering objectives. Consequently, the learned representations are not well-adapted to clustering tasks, limiting their effectiveness.
To bridge this gap, \cite{DSC} introduce the DSC framework where a self-expressive layer resides between the encoder and the decoder, enabling joint optimization of feature learning and clustering. Despite its conceptual simplicity, this method requires processing the entire dataset in a single forward pass, which necessitates downsampling and full-batch training. These constraints limit its applicability to deeper networks and high-resolution data.
Subsequent work \cite{efficientDSC} \cite{deepSC} attempts to circumvent this issue by replacing the self-expressive module with alternative mechanisms, though at the cost of increased implementation complexity. In contrast, our work generalizes DSC to mini-batch training via a memory bank that preserves global feature correlations, enabling scalable training with deeper architectures.
Conventional deep subspace clustering methods adopt shallow networks trained from scratch, while modern pre-trained models like ResNets \cite{resnet} and Vision Transformers (ViTs) \cite{vit} offer richer representations, yet adapting them to subspace clustering is impractical under full-batch training. Although we can build decoders and fine-tune these encoders via mini-batch gradient descent, it yields only marginal improvements, as the features remain unaligned with clustering objectives. Leveraging our mini-batch training strategy, we prove the effectiveness of fine-tuning large-scale pretrained models in subspace clustering. Besides, since the training data for subspace clustering are usually insufficient for training large decoders, we propose a decoder-free architecture that substitutes autoencoding with contrastive learning \cite{simclr}, making it more efficient for fine-tuning deep networks. Experiments on benchmark datasets demonstrate that our method achieves state-of-the-art performance among subspace clustering approaches.
% \subsection{Sparse Subspace Clustering (SSC)}
% Subspace clustering is an important problem with numerous applications in image processing, e.g. image representation and compression, and computer vision,e.g. image/motion/video segmentation. Traditional methods including iterative, statistical, factorization-based, spectral clustering, algebraic and information-theoretic approaches\cite{Subspace_clustering}. \cite{SSC} proposed a method based on the fact that each point in a union of subspaces has a SR with respect to a dictionary formed by all other data points and they obtained the SR 'exactly' by using $l_1$ optimization.
% In conclusion, SSC showed that, under mild conditions, the NP hard problem of writing a point as a sparse combination of other points can be solved efficiently using $l_1$ minimization.It also showed how the segmentation of the data can be easily obtained from this sparse representation. Additionally, SSC can be used with auto-encoder(AE) together, where AE is a feature extractor \cite{DSC}.
\subsection{Mini-batch training}
The adoption of mini-batch optimization in training deep neural networks has been widely studied due to its computational and statistical advantages. Unlike full-batch gradient descent, mini-batch methods partition the dataset into smaller subsets, enabling efficient memory usage and parallelizable computations, which is critical for large-scale vision tasks \cite{efficient_backprop}. Empirical evidence suggests that mini-batch training introduces stochasticity, acting as a regularizer to mitigate overfitting while maintaining convergence stability compared to pure batch gradient descent(BGD). For instance, ResNet \cite{resnet} and other vision architectures leverage mini-batch SGD with momentum to achieve robust feature learning across diverse datasets. Further analysis by Smith et al. \cite{smith2017bayesian} highlights the trade-off between batch size and gradient variance, demonstrating that smaller batches enhance generalization by escaping sharp minima—a key factor in vision model robustness.
\subsection{Self-expressiveness}
Self-expressiveness \cite{SSC} denotes the property that a given data point can be represented as a linear combination of points from the same subspace. Suppose we have data $\mathbf{X} \in \mathcal{R}^{N\times d}$, self-expressiveness can be represented as $\mathbf{X}=\mathbf{C}\mathbf{X}$, where $\mathbf{C}$ is the self-representation coefficient matrix. By applying $l_1$ or $l_2$ regularization on $\mathbf{C}$, $\mathbf{C}$ has a block diagonal structure (after permutation), and each block corresponds to one particular subspace. To avoid each data point being represented by itself, there is another restriction: $diag(\mathbf{C})=\mathbf{0}$. Once $\mathbf{C}$ is obtained, an affinity matrix can be constructed by $|\mathbf{C}|+|\mathbf{C}^T|$ or other heuristics, and used for spectral clustering for final results.
Since self-expressiveness only holds for linear subspaces, \cite{DSC} propose a deep subspace clustering network (DSC) which consists of an autoencoder and a self-expressive layer. The autoencoder helps to learn a latent space through nonlinear mapping, and the self-expressive layer is used to mimic the self-expressiveness property. Under this framework, we can learn representations that are well-suited for subspace clustering and a self-representation coefficient matrix that can be used for spectral clustering.
% \subsection{Deep Subspace Clustering(DSC)}
% Deep neural network architecture for unsupervised subspace clustering is proposed by \cite{DSC} with development of deep learning, which can be seen as a improvement of AE+SSC. The key idea of \cite{DSC} is to introduce a novel self-expressive layer between the encoder and the decoder to mimic the “self-expressiveness” property that has proven effective in traditional subspace clustering. This deep subspace clustering network allows us to directly learn the affinities between all data points through one neural network.
\subsection{Contrastive learning with a Memory Bank}
In unsupervised learning, contrastive learning often employs a pretext task known as instance discrimination \cite{intsdisc}. For a given image, two data augmentations are applied to generate two variants: $\mathbf{X}$, which serves as the anchor, and $\mathbf{X}^+$, which forms a positive pair with $\mathbf{X}$. Images from different instances, denoted as $\mathbf{X}^-$, are treated as negative samples. The objective of this framework is to minimize the distance between the positive pair representations while maximizing the distance between the anchor and all negative pairs.
The performance of contrastive learning benefits from the number of negative samples. To this end, either a large batch size \cite{simclr} or a memory bank is typically used \cite{intsdisc} \cite{moco2020}. However, using a large batch size often comes with the challenge of excessive memory usage, which is typically unaffordable on most devices. As an alternative, a memory bank that stores the representations of a large number of samples from the dataset offers a more memory-efficient solution. Similarly, traditional DSC methods can be seen as large-batch-based approaches, while our method leverages a memory bank to achieve the same objective more efficiently.
% However, the representations in memory bank are usually come from encoders at multiple steps, thus they are less \textit{consistent}, which could lead to poor performance for some tasks. To this end, \cite{moco2020} propose to leverage a momentum encoder for maintaining it.
% For learning a self-representation coefficient matrix, it is essential to ensure the consistency. But in our work, we do not use the momentum encoders, instead we alleviate this problem by striking a balance between batch size and learning rate.
% Given a set of data points $ \mathbf{X}=[x_{1},x_{2},...,x_{N} ]\in R^{d\times N} $, where $N$ is the number of samples and $d$ is the dimension of samples, SC supposes the set of data points lies in a union of $k$ low-dimensional subspaces $ \left \{ S_{i} \right \} _{i=1}^{k} $, each corresponding to a cluster. Many methods have been proposed such as factorization methods and self-expressiveness based methods. The former factorizes the data matrix, while the latter, which has become popular in recent years, attempts to represent data points as linear combinations of other samples from the same subspace. Pioneering self-expressiveness based methods includes SSC and LRR, both of which add $l_1$-norm or nuclear norm to the self-expressive coefficient matrix, making it sparse or low-rank. SSC and LRR solve the following optimization problems respectively:
% \begin{equation}
% \underset{\mathbf{C}}{min} \space \left \| \mathbf{C} \right \| _1 \quad s.t.\mathbf{X}=\mathbf{X}\mathbf{C}, \quad diag(\mathbf{C})=0
% \end{equation}
% \begin{equation}
% \underset{\mathbf{C}}{min} \space \left \| \mathbf{C} \right \| _* \quad s.t.\mathbf{X}=\mathbf{X}\mathbf{C}
% \end{equation}
% where $c_{i,j}$ in $\mathbf{C}$ denotes the correlation between the $i$th data point and the $j$th data point. $diag(\mathbf{C})=0$ requires each data point not be represented by itself, but rather by other data points from the same subspace. $\left \| \mathbf{C} \right \| _1$ denotes $l_1$-norm and $\left \| \mathbf{C} \right \| _*$ indicates nuclear norm. Affinity matrix $\mathbf{A}$ is usually denoted by $\mathbf{A}=\left | \mathbf{C} \right | +\left | \mathbf{C}^T \right | $, which is passed to spectral clustering to get the ultimate clustering results.
\begin{figure}[t]
\centering
\subfloat[]{
\label{se_comparison}
\includegraphics[width=0.3\textwidth]{imgs/DSC_BDSC.pdf}
}
\subfloat[]{
\label{overview_clbdsc}
\includegraphics[width=0.7\textwidth]{imgs/CLBDSC.pdf}
}
\caption{Overview of (a) a comparison between the self-expressive module for DSC (top) and BDSC (bottom). (b) the framework of CLBDSC. Data in the current batch and memory bank are highlighted in yellow and blue, respectively. }
\end{figure}
\subsection{Preliminary}
The deep subspace clustering network (DSC) is composed of an encoder $\mathcal{E}$, a decoder $\mathcal{D}$ and a self-expressive layer parameterized by the matrix $\mathbf{C}$ between them. Given a dataset $\mathbf{X}\in \mathcal{R}^{N \times d}$ containing $N$ samples of $d$ dimensions, in the forward pass, $\mathbf{X}$ is processed by the encoder to acquire its representation $\mathbf{Z}\in \mathcal R^{N\times h}$. The self-expressive layer then reconstructs $\mathbf{Z}$ as $\hat {\mathbf Z} = \mathbf C \mathbf Z$, where $\mathbf {C}\in \mathcal R^{N\times N}, diag(\mathbf C)=\mathbf{0}$ is a matrix capturing linear correlations between samples. Finally, the decoder $\mathcal{D}$ maps $\mathbf{\hat{Z}}$ back to the original image space, generating reconstructed inputs $\mathbf{\hat{X}}$. The DSC framework is optimized to jointly minimize the reconstruction error between $\mathbf{X}$ and $\mathbf{\hat{X}}$, as well as the discrepancy between $\mathbf{Z}$ and $\mathbf{\hat{Z}}$.
A key challenge arises during mini-batch training, where the entire dataset is split into $k$ mini-batches: $\mathbf{X}=[\mathbf{X}_1; \dots; \mathbf{X}_i;\dots;\mathbf{X}_k]$. For a batch $\mathbf{X}_i \in \mathcal{R}^{n \times d}$ where $n$ denotes the batch size, the latent representation $\mathbf{Z}_i\in \mathcal R^{n\times h}$ has reduced dimensionality ($n<N$), making the self-expressive layer inapplicable: its parameter $\mathbf{C}$ inherently requires pairwise relationships across the \textit{entire} dataset rather than individual batches. To address this issue, we introduce a memory bank that persistently maintains latent representations of all training samples. This repository dynamically updates during training, enabling consistent access to global subspace relationships while retaining computational efficiency through batch-wise processing. A comparison of the self-expressive module for DSC and our mini-batch training method (BDSC) is shown in Figure \ref{se_comparison}.
% strategy for DSC, we leverage a memory bank for storing features of the whole datasets. This is necessary as the forward process of self-expressive layer requires it as an input. But how to dynamically update it is a main problem. Besides, we need to ensure features are come from similar encoders, otherwise it is unresonable to representation new features by outdated ones.
% Compared with traditional DSC networks, BDSC contains an additional module \textit{feature memory bank}. As the self-expressive module requires all features of the whole dataset, we store features in it and dynamically update features for each batch in the training stage.
\subsection{Generalized DSC framework}
Our proposed framework extends the architecture of DSC by incorporating an additional memory bank. The training process consists of two stages: pre-training and fine-tuning, where the goal is to first learn meaningful representations and subsequently adapt them for subspace clustering.
\textbf{Pre-training}: In this stage, we disable the self-expressive layer, allowing the autoencoder to independently learn data representations. During the forward process, the encoder $\mathcal{E}$ projects a batch of input samples $\mathbf{X}_i$ into a low-dimensional latent space, producing their latent representation $\mathbf{Z}_i = \mathcal{E}(\mathbf{X}_i)$. The decoder $\mathcal{D}$ then reconstructs this latent variable back to the original space, producing the output $\mathbf{\hat{X}}_i = \mathcal{D}(\mathbf{Z}_i)$. The network is optimized by minimizing the reconstruction loss, specifically the Mean Squared Error (MSE):
\begin{equation}
\label{loss_recon}
\mathcal{L}_{recon} = \|\mathbf{X}_i - \mathbf{\hat{X}}_i\|_F^2
\end{equation}
After the pre-training stage, we initialize the feature memory bank by iterating over the dataset and storing the corresponding latent representations $\mathbf{Z} = [\mathbf{Z}^{(0)}_1; \dots; \mathbf{Z}^{(0)}_i;\dots;\mathbf{Z}^{(0)}_k]$.
\textbf{Fine-tuning}: In the fine-tuning stage, the self-expressive layer is activated to refine the latent representations by enforcing structural constraints. For the \(i\)-th batch in epoch \(t\), the encoder generates latent features \(\mathbf{Z}_i^{(t)} = \mathcal{E}(\mathbf{X}_i^{(t)})\) which is used to update the memory bank as \(\mathbf{Z} = [\mathbf{Z}_1^{(t)}; \ldots; \mathbf{Z}_i^{(t)}; \ldots; \mathbf{Z}_k^{(t-1)}]\). The self-expressive layer reconstructs the current representation as \(\mathbf{\hat{Z}}_i^{(t)} = (\mathbf{C}\mathbf{Z})_i\), using both current and historical representations. Finally, the decoder recovers samples from these latent representations as \(\mathbf{\hat{X}}_i = \mathcal{D}(\mathbf{\hat{Z}}_i^{(t)})\), enabling joint optimization of both reconstruction accuracy and subspace clustering consistency via gradient backpropagation. The loss function for optimizing the self-expressive layer is:
\begin{equation}
\mathcal{L}_{se} = \|\mathbf{Z}_i - \mathbf{\hat{Z}}_i\|_F^2 \quad \textit{s.t. }diag(\mathbf{C})=\mathbf{0}
\end{equation}
For regularization, we adopt $l_2$ norm on $\mathbf{C}$, i.e., $\mathcal{L}_{reg}=\|\mathbf{C}\|_2$ . The overall loss function is a weighted sum of the reconstruction loss, the self-expressive loss, and the regularization term:
\begin{equation}
\label{loss_ae}
\mathcal{L} = \mathcal{L}_{recon} +\alpha\mathcal{L}_{se}+\beta\mathcal{L}_{reg}
\end{equation}
where $\alpha, \beta$ are balancing coefficients. After back propagation, $\mathbf{Z}$ is detached from the computational graph. Notably, when the batch size is set to $n=N$, BDSC reduces to DSC, as the entire memory bank is updated at each iteration.
\subsection{Consistency for memory bank}
For learning a good self-expressive coefficient matrix, we need to ensure that features come from similar encoders. This issue does not arise in full-batch methods, but in mini-batch training, features in the memory bank are about the encoders at multiple different steps over the past epoch, and are less consistent.
When the number of splits $k$ is large, features from the first batch and the last batch are likely to be encoded by significantly different encoders. This can significantly disrupt the learning of self-expressive coefficients. In such cases, the features from the first batch may become "outdated" and unsuitable for representing newer features. Our goal is to maintain consistency between features from different batches. Since the size of subspace clustering datasets is not large (usually < 10k), for simplicity, we do not adopt a momentum encoder \cite{moco2020} and instead mitigate the issue by reducing the encoder’s learning rate, thereby slowing down the updates to its parameters. In practice, the relationship between the learning rate $lr$ and the number of splits $k$ is given by $lr \propto \frac{1}{k}$. In other words, as the number of splits increases, we reduce the learning rate to ensure consistent feature encoding.
\subsection{Decoder-free framework}
Many large-scale pre-trained models are encoder-only. However, subspace clustering datasets are typically small, making them inadequate for training large decoders. This raises the question whether DSC can be generalized to a decoder-free framework. In DSC, the decoder plays a critical role in learning the latent space, but recent self-supervised methods, such as contrastive learning, offer an alternative approach. In this section, we propose a contrastive learning-based DSC network that can also be optimized using mini-batches. Within this framework, our memory bank can also serve as a source for sampling negative examples.
Unlike previous contrastive learning works \cite{simclr} \cite{byol}, we retain the original images $\mathbf{X}_i$ and apply a single augmentation to obtain ${\mathbf{X}}_i^+$. In the forward pass, we encode both $\mathbf{X}_i$ and ${\mathbf{X}}_i^+$ to generate their respective representations $\mathbf{Z}_i$ and $\mathbf{Z}_i^+$. The representation $\mathbf{Z}_i$ is then passed through the self-expressive module. To compute the contrastive loss, we treat $\mathbf{\hat{Z}}_i$ as the anchor, $\mathbf{Z}_i^+$ as the positive pair, and $\mathbf{Z}^-_i$ from the memory bank as negative pairs. The InfoNCE loss \cite{infonce} is adopted:
\begin{equation}
\mathcal{L}_{nce} = - \log \frac{\exp(\hat{\mathbf{z}}\cdot \mathbf{z}^+/\tau)}
{\sum_{k=0}^{N-n}\exp(\mathbf{\hat{z}}\cdot \mathbf{z}_k^-/\tau)}
\end{equation}
where $\tau$ is the temperature, and $\mathbf{\hat{z}}$, $\mathbf{z}^+$ , $\mathbf{z}_k^-$ denote representations from $\mathbf{\hat{Z}}_i$, $\mathbf{Z}_i^+$, $\mathbf{Z}^-_i$ (all normalized to unit length). As $N$ is not excessively large, all samples from the memory bank, except the current batch, are treated as negative samples. We take $\mathbf{\hat{z}}$ instead of $\mathbf{z}$ as the anchor to prevent the decoupling of representation learning from subspace clustering. To ensure that each representation in the memory bank corresponds to a deterministic input, we update the memory bank with $\mathbf{z}$ rather than $\mathbf{z}^+$.
The final objective function shares the structure of equation (\ref{loss_ae}), but $\mathcal{L}_{recon}$ is replaced with $\mathcal{L}_{nce}$. We denote this contrastive learning-based approach as CLBDSC. Its framework is presented in Figure \ref{overview_clbdsc}.
\label{sec_experiment}
\subsection{Experimental setups}
\textbf{Datasets}: Since classical subspace clustering methods and deep subspace clustering methods have quadratic training time and space complexities in terms of the number of samples, we focus on using deep architectures and high-resolution for small or medium-scale datasets (sample < 10k). Following previous works, four subspace clustering datasets are used for evaluating our method, including ORL, COIL-20, COIL-100 and Extended Yale-B.
ORL \cite{ORL} has 400 face images of 40 people in $98 \times112$ grayscale with different expression details such as wearing glasses or not, smiling or not, eyes open or closed. For each subject, the images were taken under varying lighting conditions.
COIL-100 and COIL-20 \cite{COIL20} datasets consist of images of 100 and 20 objects, respectively, placed on a motorized turntable. For each object, 72 images are taken at poses intervals of 5 degrees that cover a 360-degree range. The COIL-100 dataset contains RGB images, whereas the COIL-20 dataset consists of grayscale images.
The Extended Yale-B dataset \cite{yaleb} contains 2432 facial images of 38 individuals from 9 poses and under 64 illumination settings.
\textbf{Evaluating metrics}: We choose two popular metrics, i.e. clustering accuracy rate (ACC), normalized mutual information (NMI), to evaluate the performance of our models. Both metrics take values in [0,1], with higher values indicating better performance.
ACC is defined as:
\begin{equation}
ACC=\frac{\sum_{i=1}^{N}\mathbbm{1}(p_i,r_i)}{N}
\end{equation}
where $p_i$ is the predicted label for the $i_{th}$ sample, and $r_i$ is the real label. $\mathbbm{1}(\cdot,\cdot)$ is the indicator function that satisfies $\mathbbm{1}(p_i,r_i)=1$ if $p_i=r_i$ and $\mathbbm{1}(p_i,r_i)=0$ otherwise.
NMI is defined as follows:
\begin{equation}
NMI=\frac{2\mathbf{I}(p,r)}{\mathbf{H}(p)+\mathbf{H}(r)}
\end{equation}
where $\mathbf{I}(\cdot,\cdot)$ is the mutual information. $\mathbf{H}$ represents the entropy.
Our proposed method is implemented using PyTorch 2.0.1. We adopt Adam \cite{adam} as the optimizer with default momentum parameters. For CLBDSC, we pre-train the self-expressive layer with frozen encoders before full fine-tuning. All the experiments are conducted on one NVIDIA RTX 3080Ti.
% \subsection{Architecture}
% \begin{figure}[h]
% \centering
% % First
% \begin{subfigure}[b]{0.45\textwidth}
% \centering
% \includegraphics[width=\textwidth]{imgs/AE32.pdf}
% \caption{}
% \end{subfigure}
% \hfill
% % Second
% \begin{subfigure}[b]{0.45\textwidth}
% \centering
% \includegraphics[width=\textwidth]{imgs/DeRN18.pdf}
% \caption{}
% \end{subfigure}
% \caption{}
% \end{figure}
\subsection{Approximating Full-Batch Training with mini-batch Strategies}
\label{sec_approx}
To assess whether our mini-batch training strategy achieves performance comparable to full-batch methods, we conduct experiments using architectures and hyper-parameters identical to those of DSC \cite{DSC} and convert all datasets to grayscale. Specifically, images from ORL, COIL20/100 are resized to \(32\times32\), and Yale-B to \(42\times42\). The batch size is set to 32 for ORL, COIL20, 128 for Yale-B, and 256 for COIL100 to ensure computational efficiency. For improving the consistency in the feature memory bank, we set a small learning rate of $0.0001$.
As shown in Table~\ref{approx}, our method (BDSC) achieves performance comparable to that of full-batch DSC, sometimes surpassing it, demonstrating that our mini-batch training is a good alternative to traditional full-batch methods. Results for ORL, COIL20, and Yale-B are based on both the publicly available PyTorch implementation\footnote{https://github.com/XifengGuo/DSC-Net.git} of DSC and our reimplementation. While for COIL100, we report results based on the original TensorFlow implementation.
\begin{table}[h]
\centering
\caption{Comparison with full-batch strategy.}
\begin{tabular}{ccccc}
\toprule
&Metric& DSC(reported) & DSC(our implement) & BDSC \\ \midrule
\multirow{2}{*}{ORL}&ACC& 0.855&0.863& 0.888\\
&NMI& 0.920&0.918& 0.950\\ \midrule
\multirow{2}{*}{COIL20}&ACC& 0.910&0.916& 0.919\\
&NMI& 0.959&0.961& 0.972\\ \midrule
\multirow{2}{*}{COIL100}&ACC& 0.690&0.632& 0.666\\
&NMI& -&0.887& 0.896\\ \midrule
\multirow{2}{*}{Extended Yale-B}&ACC& 0.973&0.973& 0.959\\
&NMI& 0.963&0.963& 0.946\\
% Yale-b & 0.835 & & \\
\bottomrule
\end{tabular}
\label{approx}
\end{table}
% \begin{table}[h]
% \centering
% \caption{Caption}
% \begin{tabular}{ccccc}
% \toprule
% &Metric& AE+SSC & DSC & BDSC \\ \midrule
% \multirow{2}{*}{ORL}&ACC& 0.835 & 0.895& 0.935\\
% &NMI& 0.905 & 0.944& 0.970\\ \midrule
% \multirow{2}{*}{COIL20}&ACC& 0.873 & 0.899& 0.915\\
% &NMI& 0.942 & 0.937& 0.955\\ \midrule
% \multirow{2}{*}{COIL100}&ACC& 0.575 & 0.614& 0.678\\
% &NMI& 0.829 & 0.859& 0.894\\
% % Yale-b & 0.835 & & \\
% \bottomrule
% \end{tabular}
% \label{approx}
% \end{table}
\subsection{Experiments with high-resolution images and deeper networks}
\label{sec_deeper}
Our mini-batch training strategy significantly reduces the memory cost and thus facilitates the deployment of deeper architectures and higher-resolution inputs. For evaluating whether subspace clustering performance benefits from deeper networks, in this section, we conduct experiments on ORL and COIL100 datasets. The architectures used in this section are detailed in the Appendix.
\begin{table}[h]
\caption{Augmentations applied in CLBDSC for each dataset}
\label{aug}
\centering
\begin{tabular}{c|cc}\toprule
Dataset & COIL100& ORL\\ \midrule
\multirow{4}{*}{Augment}
& Crop and Resize & Crop and Resize\\
&Color Jitter & Gray Scale\\
&Random Grayscale &Horizontal Flip\\
&Horizontal Flip \\ \bottomrule
\end{tabular}
\end{table}
\paragraph{ORL} For BDSC, we employ a 3-layer encoder with batch normalization \citet{BN} applied after each convolutional layer, followed by GELU \citet{GELU} activation. The decoder mirrors the encoder but omits batch normalization. We flatten the features extracted by the encoder before the self-expressive module. Input images of size \(32 \times 32\) are encoded into 256-dimensional latent features. We set the learning rate to $1\times10^{-3}$, $\alpha$ to 50 and $\beta$ to 1. The autoencoder is first pre-trained without the self-expressive module for 100 epochs, and then fine-tuned for 1000 epochs.
For CLBDSC, we utilize \(128 \times 128\) resolution inputs and apply the augmentations detailed in Table~\ref{aug} to generate positive pairs, including random cropping (with resizing), color distortion, horizontal flipping and random grayscale \citet{color_resize} \citet{other_aug} \citet{color_resize}. We adopt 6-layer encoder and set the learning rate to $5\times10^{-5}$, keeping other settings the same with BDSC.
\paragraph{COIL100} For BDSC, we design an autoencoder containing an ImageNet-1k pre-trained ResNet-18 as encoder and a lightweight decoder containing only 6M parameters, which is smaller than the encoder (11M). This design prioritizes efficient representation learning over reconstruction quality, as better reconstruction does not necessarily indicate better representation learning. Batch normalization and ReLU activation follow each deconvolutional layer.
For CLBDSC, we adopt the same ResNet-18 encoder with \(128 \times 128\) inputs and augmentations from Table~\ref{aug}. For both methods, given that the encoder is pre-trained, we directly fine-tune the model for 80 epochs. A learning of $5\times10^{-5}$ is used, with $\alpha$, $\beta$ set to 1.
As shown in Figure \ref{Com_SOTA_orl_coil100}, our methods outperform previous state-of-the-art subspace clustering approaches, including DSCNSS \cite{DSCNSS_2023}, DSSC \cite{DSSC_2021}, DCFSC \cite{DCFSC_2019}, DPSC \cite{DPSC_2019}, S\(^2\)ConvSCN \cite{cnnSC}, and MLRDSC-DA \cite{augSC}. These results underscore the effectiveness of our mini-batch training strategy and decoder-free contrastive learning framework in deeper networks.
\begin{figure}[h] % 创建一个图形环境
\centering
\begin{subfigure}[b]{0.45\textwidth} % 第一列子图，占45%宽度
\centering
\includegraphics[width=1\textwidth]{imgs/orl_table.pdf} % 第一张图片
\caption{ORL} % 子图1标题
\end{subfigure}
\hfill % 在两列之间加入间隔
\begin{subfigure}[b]{0.45\textwidth} % 第二列子图，占45%宽度
\centering
\includegraphics[width=1\textwidth]{imgs/coil-100_table.pdf} % 第二张图片
\caption{COIL-100} % 子图2标题
\end{subfigure}
\caption{Comparison with SOTA subspace clustering methods on COIL100 and ORL}
\label{Com_SOTA_orl_coil100}
\end{figure}
% \begin{table}[h]
% \centering
% \caption{Comparison with SOTA subspace clustering methods on COIL100 and ORL}
% \begin{tabular}{cccccccccc} \toprule
% Dataset& Method&DSCNSS & DSSC & DCFSC & DPSC& S$^2$ConvSCN&MLRDSC-DA &BDSC &CLBDSC\\ \midrule
% \multirow{2}{*}{COIL100}
% &ACC&0.714&0.725&0.727&0.754&0.733&0.793&0.806&0.829 \\
% &NMI& - & 0.729&- &0.759&-&-&0.963&0.964 \\ \midrule
% \multirow{2}{*}{ORL}
% &ACC&0.892&0.885&0.852&0.754&0.895&0.898&0.935&0.923 \\
% &NMI& - & 0.935&- &0.759&&-&0.970&0.955 \\
% \bottomrule
% \end{tabular}
% \label{rn18}
% \end{table}
% \begin{table}[h]
% \centering
% \caption{Settings}
% \begin{tabular}{cccc}
% \toprule
% & resolution & channel & params \\ \midrule
% baseline & $32\times32$ & 1 & 3177\\
% BDSC & $128\times128$& 3 & 18M\\ \bottomrule
% \end{tabular}
% \label{settings}
% \end{table}
% \begin{table}[h]
% \centering
% \caption{COIL100}
% \begin{tabular}{ccccc} \toprule
% Method &resolution & channel& ACC & NMI \\ \midrule
% AE+SSC &\multirow{3}{*}{$32\times32$} & \multirow{3}{*}{1} & 0.575 &0.829 \\
% DSC & & & 0.614 &0.859 \\
% BDSC & & & 0.678 &0.894 \\ \midrule
% RN18(pre-trained)+SSC&\multirow{4}{*}{$128\times128$}& \multirow{4}{*}{3} & 0.661 &0.906 \\
% RN18(fine-tuned)+SSC&& & 0.698 &0.936 \\
% BDSC && & 0.806 &0.963 \\
% CLBDSC && & 0.812 &0.961 \\
% \bottomrule
% \end{tabular}
% \label{rn18}
% \end{table}
\paragraph{Visualization of Embedded Representation.} Figure \ref{tsne} presents the t-SNE visualization of the learned embedded representation on COIL100. We compare DSC \cite{DSC} with our two models based on ResNet18. Representations learned by our deeper models are apparently distributed in tighter clusters, especially for CLBDSC, which demonstrates the ability to learn more discriminative representations.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=1\textwidth]{imgs/DSC100.jpg}
\caption{DSC}
% \label{bs}
\end{subfigure}
\hfill
% \begin{subfigure}[b]{0.45\textwidth}
% \centering
% \includegraphics[width=1\textwidth]{imgs/pretrained100.jpg}
% \caption{pretrained}
% \label{lr}
% \end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=1\textwidth]{imgs/BDSC100.jpg}
\caption{BDSC}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=1\textwidth]{imgs/CLBDSC100.jpg}
\caption{CLBDSC}
% \label{bs}
\end{subfigure}
\caption{Visualization of the embedded representations with t-SNE on COIL100. Samples from different classes are marked in different colors.}
\label{tsne}
\end{figure}
\subsection{Ablation study}
\paragraph{Influence of learning rate and batch size.} To validate our analysis on memory bank consistency, we vary the batch size \(\{32, 64, 128, 256\}\) and learning rate \(\{3 \times 10^{-5}, 6 \times 10^{-5}, 10^{-4}, 3 \times 10^{-4}, 6 \times 10^{-4}, 10^{-3}\}\) on COIL20, while keeping other settings identical to Section \ref{sec_approx}. Since we have 1,440 samples in total, the maximum number of splits per epoch reaches 45 when using the smallest batch size (32). Figure \ref{lr_bs} shows the results, revealing two key trends: (1) smaller batch sizes require lower learning rates to improve memory bank consistency, and (2) results with larger batch sizes exhibit less sensitivity to the choice of learning rate due to fewer dataset splits.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=1\textwidth]{imgs/acc_lr_bs.pdf}
\caption{Clustering Accuracy}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=1\textwidth]{imgs/nmi_lr_bs.pdf}
\caption{Normalized Mutual Information}
\end{subfigure}
\caption{Influence of batch size and learning rate.}
\label{lr_bs}
\end{figure}
% \begin{table}
% \begin{minipage}{0.45\textwidth}
% \centering
% \caption{ACC}
% \begin{tabular}{c|cccc}\toprule
% lr \textbackslash bs & 32 & 64 & 128 & 256\\ \midrule
% 3e-5 &0.893&0.890&0.881&0.871\\
% 6e-5 &0.875&0.885&0.892&0.857 \\
% 1e-4 &0.919&0.880&0.894&0.860\\
% 3e-4 &0.847&0.872&0.903&0.916\\
% 6e-4 &0.736&0.840&0.848&0.876\\
% 1e-3 &0.640&0.750&0.861&0.863\\
% \bottomrule
% \end{tabular}
% \end{minipage}
% \begin{minipage}{0.45\textwidth}
% \centering
% \caption{MNI}
% \begin{tabular}{c|cccc}\toprule
% lr \textbackslash bs & 32 & 64 & 128 & 256\\ \midrule
% 3e-5 &0.952&0.957&0.937&0.926\\
% 6e-5 &0.956&0.961&0.960&0.955\\
% 1e-4 &0.972&0.947&0.958&0.956\\
% 3e-4 &0.935&0.956&0.962&0.961\\
% 6e-4 &0.886&0.941&0.954&0.933\\
% 1e-3 &0.822&0.897&0.942&0.951\\
% \bottomrule
% \end{tabular}
% \end{minipage}
% \end{table}
\paragraph{Influence of balancing coefficients.} We analyze the impact of loss-balancing coefficients \(\alpha\) and \(\beta\) by testing \(\alpha \in \{1, 10, 50, 100, 200\}\) and \(\beta \in \{1, 5, 10\}\) on ORL for computational efficiency. As shown in Figure \ref{coefficients}, a small \(\beta\) and a moderate \(\alpha\) (around 50) enhance the performance. However, these parameters have a relatively minor effect on final results compared to the learning rate.
% These findings suggest a trade-off between explicit loss weighting and implicit regularization from batch size, guiding practical parameter selection—for instance, prioritizing \(\beta < 5\) and dynamically adjusting \(\alpha\) based on batch size to balance optimization stability and task performance.
% We set the range of $\alpha$ to $\{1,10,50,100,200\}$ and that of $\beta$ to $\{1,5,10\}$. Larger $\beta$ makes convergences more unstable. When setting a large batch size, $\alpha$ should be decreased.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=1\textwidth]{imgs/Accuracy.pdf}
\caption{Clustering Accuracy}
\label{lr}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=1\textwidth]{imgs/nmi.pdf}
\caption{Normalized Mutual Information}
\label{bs}
\end{subfigure}
\caption{Influence of balancing coefficients}
\label{coefficients}
\end{figure}
\paragraph{Influence of multi-task training} Our framework jointly optimizes representation learning and clustering, making it a form of multi-task training. To assess its impact, we compare our methods (BDSC, CLBDSC) with two-step training approaches: (1) clustering (typically via SSC) on features extracted from pre-trained or fine-tuned autoencoders (AE+SSC), and (2) clustering on features learned by contrastive learning (CL+SSC).
As shown in Table~\ref{ablation}, joint training significantly outperforms two-stage approaches. On COIL100, fine-tuning an autoencoder yields only a 3.7\% accuracy improvement over pre-trained features, whereas contrastive learning (CL+SSC) achieves a 9\% increase. However, integrating clustering objectives directly into training, as in BDSC and CLBDSC, results in accuracies exceeding 80\%. A similar trend is observed on ORL, where multi-task optimization consistently outperforms two-step training, highlighting the importance of unifying representation learning and subspace clustering.
% Our objective can be regarded as multi-task training: representation learning together with clustering, which help to learn representation adapted to subspace clustering tasks. We compare our two methods with their two-step counterparts, i.e., feature learning followed by clustering.
% For BDSC, we compare it with (1) SSC on representations learned by pre-trained models (if used) (2) SSC on representations learned by fine-tuned autoencoders (AE+SSC). And for CLBDSC, we compare it with SSC on representations learned by contrastive learning (CL+SSC). Table \ref{ablation} presents the results. For COIL100, Fine-tuning the network by autoencoding brings trivial improvement (+3.7\%), while contrastive learning is more effective (+9\%). By adapting learned representation to subspace clustering, both methods achieve accuracy higher than 0.8. The same phenomenon can also be observed on ORL.
\begin{table}[h]
\centering
\caption{Comparison with two-step training.}
\begin{tabular}{cccc} \toprule
Dataset &Method & ACC& NMI \\ \midrule
\multirow{5}{*}{COIL100}
& ResNet18 (pre-trained)+SSC & 0.661 &0.906\\
& ResNet18 (fine-tuned with AE)+SSC & 0.698 &0.936 \\
&BDSC & 0.813 &0.963 \\
&CL+SSC & 0.752 &0.948 \\
&CLBDSC & 0.829 &0.964 \\\midrule
\multirow{4}{*}{ORL}
&AE+SSC & 0.853 &0.909 \\
&BDSC & 0.935 &0.970 \\
&CL+SSC & 0.905 &0.948 \\
&CLBDSC & 0.923 &0.955 \\
\bottomrule
\end{tabular}
\label{ablation}
\end{table}
\section{Architectures}
Figure \ref{orl_coil100} presents the architectures used in Section \ref{sec_deeper}. All parts are randomly initialized except for the ResNet18 backbone. Since ORL is a grayscale dataset, the input channel of its networks is 1 instead of 3.
As the self-expressive layer takes inputs of shape $(N, c*d)$, we flatten the outputs of encoders, then reshape the outputs of the self-expressive layer to $(N, c, \sqrt{d}, \sqrt{d})$ before feeding them into decoders. However, for BDSC on COIL100, since features are processed by average pooling and then flattened within the ResNet18 backbone, we employ a fully connected layer in the decoder to adjust the dimension to $(512, 4, 4)$.
\begin{figure}[h]
\centering
% First
\begin{subfigure}[b]{0.3\textwidth}
\centering
\includegraphics[width=\textwidth]{imgs/AE32.pdf}
\caption{}
\end{subfigure}
\hfill
% Second
\begin{subfigure}[b]{0.3\textwidth}
\centering
\includegraphics[width=\textwidth]{imgs/DeRN18.pdf}
\caption{}
\end{subfigure}
\hfill
% Third
\begin{subfigure}[b]{0.3\textwidth}
\centering
\includegraphics[width=\textwidth]{imgs/CLBDSC_ORL.pdf}
\caption{}
\end{subfigure}
\caption{Architectures for (a) BDSC on ORL, (b) BDSC on COIL100, and (c) CLBDSC on ORL.}
\label{orl_coil100}
\end{figure}
\begin{thebibliography}{38}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abavisani et~al.(2020)Abavisani, Naghizadeh, Metaxas, and Patel]{augSC}
M.~Abavisani, A.~Naghizadeh, D.~Metaxas, and V.~Patel.
\newblock Deep subspace clustering with data augmentation.
\newblock \emph{Advances in Neural Information Processing Systems}, 33:\penalty0 10360--10370, 2020.
\bibitem[Baek et~al.(2021)Baek, Yoon, Song, and Yoon]{DSSC_2021}
S.~Baek, G.~Yoon, J.~Song, and S.~M. Yoon.
\newblock Deep self-representative subspace clustering network.
\newblock \emph{Pattern Recognition}, 118:\penalty0 108041, 2021.
\bibitem[Basri and Jacobs(2003)]{subspace_lambertian}
R.~Basri and D.~W. Jacobs.
\newblock Lambertian reflectance and linear subspaces.
\newblock \emph{IEEE transactions on pattern analysis and machine intelligence}, 25\penalty0 (2):\penalty0 218--233, 2003.
\bibitem[Cai et~al.(2022)Cai, Fan, Guo, Wang, Zhang, and Zhang]{efficientDSC}
J.~Cai, J.~Fan, W.~Guo, S.~Wang, Y.~Zhang, and Z.~Zhang.
\newblock Efficient deep embedded subspace clustering.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 1--10, 2022.
\bibitem[Chen et~al.(2023)Chen, Lu, Wei, and Geng]{DSCNSS_2023}
C.~Chen, H.~Lu, H.~Wei, and X.~Geng.
\newblock Deep subspace image clustering network with self-expression and self-supervision.
\newblock \emph{Applied Intelligence}, 53\penalty0 (4):\penalty0 4859--4873, 2023.
\bibitem[Chen et~al.(2020)Chen, Kornblith, Norouzi, and Hinton]{simclr}
T.~Chen, S.~Kornblith, M.~Norouzi, and G.~Hinton.
\newblock A simple framework for contrastive learning of visual representations.
\newblock In \emph{International conference on machine learning}, pages 1597--1607. PmLR, 2020.
\bibitem[Dosovitskiy et~al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et~al.]{vit}
A.~Dosovitskiy, L.~Beyer, A.~Kolesnikov, D.~Weissenborn, X.~Zhai, T.~Unterthiner, M.~Dehghani, M.~Minderer, G.~Heigold, S.~Gelly, et~al.
\newblock An image is worth 16x16 words: Transformers for image recognition at scale.
\newblock \emph{arXiv preprint arXiv:2010.11929}, 2020.
\bibitem[Grill et~al.(2020)Grill, Strub, Altch{\'e}, Tallec, Richemond, Buchatskaya, Doersch, Avila~Pires, Guo, Gheshlaghi~Azar, et~al.]{byol}
J.-B. Grill, F.~Strub, F.~Altch{\'e}, C.~Tallec, P.~Richemond, E.~Buchatskaya, C.~Doersch, B.~Avila~Pires, Z.~Guo, M.~Gheshlaghi~Azar, et~al.
\newblock Bootstrap your own latent-a new approach to self-supervised learning.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 21271--21284, 2020.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{resnet}
K.~He, X.~Zhang, S.~Ren, and J.~Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 770--778, 2016.
\bibitem[He et~al.(2020)He, Fan, Wu, Xie, and Girshick]{moco2020}
K.~He, H.~Fan, Y.~Wu, S.~Xie, and R.~Girshick.
\newblock Momentum contrast for unsupervised visual representation learning.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 9729--9738, 2020.
\bibitem[Hendrycks and Gimpel(2016)]{GELU}
D.~Hendrycks and K.~Gimpel.
\newblock Gaussian error linear units (gelus).
\newblock \emph{arXiv preprint arXiv:1606.08415}, 2016.
\bibitem[Hong et~al.(2006)Hong, Wright, Huang, and Ma]{sc_ip1}
W.~Hong, J.~Wright, K.~Huang, and Y.~Ma.
\newblock Multiscale hybrid linear models for lossy image representation.
\newblock \emph{IEEE Transactions on Image Processing}, 15\penalty0 (12):\penalty0 3655--3671, 2006.
\bibitem[Ioffe and Szegedy(2015)]{BN}
S.~Ioffe and C.~Szegedy.
\newblock Batch normalization: Accelerating deep network training by reducing internal covariate shift.
\newblock In \emph{International conference on machine learning}, pages 448--456. pmlr, 2015.
\bibitem[Ji et~al.(2017)Ji, Zhang, Li, Salzmann, and Reid]{DSC}
P.~Ji, T.~Zhang, H.~Li, M.~Salzmann, and I.~Reid.
\newblock Deep subspace clustering networks.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus, S.~Vishwanathan, and R.~Garnett, editors, \emph{Advances in Neural Information Processing Systems}, volume~30. Curran Associates, Inc., 2017.
\bibitem[Kingma and Ba(2014)]{adam}
D.~P. Kingma and J.~Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{arXiv preprint arXiv:1412.6980}, 2014.
\bibitem[Krizhevsky et~al.(2012)Krizhevsky, Sutskever, and Hinton]{color_resize}
A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock \emph{Advances in neural information processing systems}, 25, 2012.
\bibitem[LeCun et~al.(2002)LeCun, Bottou, Orr, and M{\"u}ller]{efficient_backprop}
Y.~LeCun, L.~Bottou, G.~B. Orr, and K.-R. M{\"u}ller.
\newblock Efficient backprop.
\newblock In \emph{Neural networks: Tricks of the trade}, pages 9--50. Springer, 2002.
\bibitem[Lee et~al.(2005)Lee, Ho, and Kriegman]{yaleb}
K.-C. Lee, J.~Ho, and D.~J. Kriegman.
\newblock Acquiring linear subspaces for face recognition under variable lighting.
\newblock \emph{IEEE Transactions on pattern analysis and machine intelligence}, 27\penalty0 (5):\penalty0 684--698, 2005.
\bibitem[Li et~al.(2015)Li, Li, and Fu]{li2015temporal}
S.~Li, K.~Li, and Y.~Fu.
\newblock Temporal subspace clustering for human motion segmentation.
\newblock In \emph{Proceedings of the IEEE international conference on computer vision}, pages 4453--4461, 2015.
\bibitem[Li et~al.(2024)Li, Wang, Li, Yuan, and Wang]{deepSC}
Y.~Li, S.~Wang, C.~Li, Y.~Yuan, and G.~Wang.
\newblock Towards very deep representation learning for subspace clustering.
\newblock \emph{IEEE Transactions on Knowledge and Data Engineering}, 36\penalty0 (7):\penalty0 3568--3579, 2024.
\bibitem[Liu et~al.(2010)Liu, Lin, and Yu]{LRR}
G.~Liu, Z.~Lin, and Y.~Yu.
\newblock Robust subspace segmentation by low-rank representation.
\newblock In \emph{Proceedings of the 27th international conference on machine learning (ICML-10)}, pages 663--670, 2010.
\bibitem[Nene et~al.(1996)Nene, Nayar, Murase, et~al.]{COIL20}
S.~A. Nene, S.~K. Nayar, H.~Murase, et~al.
\newblock Columbia object image library (coil-20).
\newblock 1996.
\bibitem[Oord et~al.(2018)Oord, Li, and Vinyals]{infonce}
A.~v.~d. Oord, Y.~Li, and O.~Vinyals.
\newblock Representation learning with contrastive predictive coding.
\newblock \emph{arXiv preprint arXiv:1807.03748}, 2018.
\bibitem[Patel and Vidal(2014)]{kernel_subspace1}
V.~M. Patel and R.~Vidal.
\newblock Kernel sparse subspace clustering.
\newblock In \emph{2014 ieee international conference on image processing (icip)}, pages 2849--2853. IEEE, 2014.
\bibitem[Patel et~al.(2015)Patel, Van~Nguyen, and Vidal]{kernel_subspace2}
V.~M. Patel, H.~Van~Nguyen, and R.~Vidal.
\newblock Latent space sparse and low-rank subspace clustering.
\newblock \emph{IEEE Journal of Selected Topics in Signal Processing}, 9\penalty0 (4):\penalty0 691--701, 2015.
\bibitem[Samaria and Harter(1994)]{ORL}
F.~S. Samaria and A.~C. Harter.
\newblock Parameterisation of a stochastic model for human face identification.
\newblock In \emph{Proceedings of 1994 IEEE workshop on applications of computer vision}, pages 138--142. IEEE, 1994.
\bibitem[Seo et~al.(2019)Seo, Koo, and Jeon]{DCFSC_2019}
J.~Seo, J.~Koo, and T.~Jeon.
\newblock Deep closed-form subspace clustering.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops}, pages 0--0, 2019.
\bibitem[Smith and Le(2017)]{smith2017bayesian}
S.~L. Smith and Q.~V. Le.
\newblock A bayesian perspective on generalization and stochastic gradient descent.
\newblock \emph{arXiv preprint arXiv:1710.06451}, 2017.
\bibitem[Szegedy et~al.(2015)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and Rabinovich]{other_aug}
C.~Szegedy, W.~Liu, Y.~Jia, P.~Sermanet, S.~Reed, D.~Anguelov, D.~Erhan, V.~Vanhoucke, and A.~Rabinovich.
\newblock Going deeper with convolutions.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 1--9, 2015.
\bibitem[Tomasi and Kanade(1992)]{subspace_motion}
C.~Tomasi and T.~Kanade.
\newblock Shape and motion from image streams under orthography: a factorization method.
\newblock \emph{International journal of computer vision}, 9:\penalty0 137--154, 1992.
\bibitem[Vidal(2009)]{SSC}
E.~E.~R. Vidal.
\newblock Sparse subspace clustering.
\newblock In \emph{2009 IEEE conference on computer vision and pattern recognition (CVPR)}, volume~6, pages 2790--2797, 2009.
\bibitem[Vidal et~al.(2003)Vidal, Soatto, Ma, and Sastry]{sc_st1}
R.~Vidal, S.~Soatto, Y.~Ma, and S.~Sastry.
\newblock An algebraic geometric approach to the identification of a class of linear hybrid systems.
\newblock In \emph{42nd IEEE international conference on decision and control (IEEE Cat. No. 03CH37475)}, volume~1, pages 167--172. IEEE, 2003.
\bibitem[Vidal et~al.(2008)Vidal, Tron, and Hartley]{sc_cv2}
R.~Vidal, R.~Tron, and R.~Hartley.
\newblock Multiframe motion segmentation with missing data using powerfactorization and gpca.
\newblock \emph{International Journal of Computer Vision}, 79:\penalty0 85--105, 2008.
\bibitem[Wang et~al.(2023)Wang, Sun, Yuan, Gao, and Lu]{multitvc}
L.~Wang, D.~Sun, Z.~Yuan, Q.~Gao, and Y.~Lu.
\newblock Multi-view clustering based on graph learning and view diversity learning.
\newblock \emph{The Visual Computer}, 39\penalty0 (12):\penalty0 6133--6149, 2023.
\bibitem[Wu et~al.(2018)Wu, Xiong, Yu, and Lin]{intsdisc}
Z.~Wu, Y.~Xiong, S.~X. Yu, and D.~Lin.
\newblock Unsupervised feature learning via non-parametric instance discrimination.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 3733--3742, 2018.
\bibitem[Yang et~al.(2008)Yang, Wright, Ma, and Sastry]{sc_cv1}
A.~Y. Yang, J.~Wright, Y.~Ma, and S.~S. Sastry.
\newblock Unsupervised segmentation of natural images via lossy data compression.
\newblock \emph{Computer Vision and Image Understanding}, 110\penalty0 (2):\penalty0 212--225, 2008.
\bibitem[Zhang et~al.(2019)Zhang, Li, You, Qi, Zhang, Guo, and Lin]{cnnSC}
J.~Zhang, C.-G. Li, C.~You, X.~Qi, H.~Zhang, J.~Guo, and Z.~Lin.
\newblock Self-supervised convolutional subspace clustering network.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 5473--5482, 2019.
\bibitem[Zhou et~al.(2019)Zhou, Xiao, Liu, Zhou, Hancock, et~al.]{DPSC_2019}
L.~Zhou, B.~Xiao, X.~Liu, J.~Zhou, E.~R. Hancock, et~al.
\newblock Latent distribution preserving deep subspace clustering.
\newblock In \emph{28th International joint conference on artificial intelligence}, 2019.
\end{thebibliography}