Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch
Extracted source text · arXiv 2504.12801
\documentclass{article}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{multirow}
\usepackage{wrapfig}
\usepackage{algorithm}
\usepackage{algorithmic}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
\usepackage{hyperref}
\usepackage{amssymb}% http://ctan.org/pkg/amssymb
\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
% For code block
\usepackage{listings}
\usepackage{xcolor}
\lstset{
language=Python,
basicstyle=\ttfamily\scriptsize, % Smaller font size
keywordstyle=\color{blue}\bfseries,
commentstyle=\color{green!50!black},
stringstyle=\color{red},
showstringspaces=false,
frame=single,
breaklines=true,
captionpos=b,
tabsize=4,
numbers=none, % No line numbers
}
% Attempt to make hyperref and algorithmic work together better:
% \newcommand{\theHalgorithm}{\arabic{algorithm}}
\usepackage{xspace}
\newcommand{\signin}{\emph{Sign-In}\xspace}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
% \usepackage{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
\usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
% \theoremstyle{setting}
\newtheorem{setting}[theorem]{Setting}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\title{Sign-In to the Lottery:\\ Reparameterized Sparse Training}
% \title{Sign-In: Reparameterizing Sparse Training}
\author{
Advait Gadhikar\thanks{Equal contribution} \quad
Tom Jacobs\footnotemark[1] \quad
Chao Zhou \quad
Rebekka Burkholz \\
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany \\
\texttt{\{advait.gadhikar, tom.jacobs, chao.zhou, burkholz\}@cispa.de}
}
\begin{document}
\maketitle
\begin{abstract}
The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training, presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization, given a sparse mask. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.
\end{abstract}
\input{intro}
\input{RelatedWork}
\input{Motivation}
\input{Algorithm}
\input{theory}
\input{experiments}
\input{Conclusion}
\section{Acknowledgements}
We gratefully
acknowledge the Gauss Centre for Supercomputing e.V. for
funding this project by providing computing time on the
GCS Supercomputer JUWELS at Jülich Supercomputing
Centre (JSC) \citet{juelich2021juwels}. We are also grateful for funding from
the European Research Council (ERC) under the Horizon
Europe Framework Programme (HORIZON) for proposal
number 101116395 SPARSE-ML.
\bibliography{iclr2024_conference}
\bibliographystyle{icml2025}
% \newpage
% \section*{NeurIPS Paper Checklist}
% \begin{enumerate}
% \item {\bf Claims}
% \item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The claims in the introduction and abstract match the experiments.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the abstract and introduction do not include the claims made in the paper.
% \item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
% \item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
% \item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
% \end{itemize}
% \item {\bf Limitations}
% \item[] Question: Does the paper discuss the limitations of the work performed by the authors?
% \item[] Answer: \answerYes{}{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \cref{thm : impossible}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
% \item The authors are encouraged to create a separate "Limitations" section in their paper.
% \item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
% \item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
% \item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
% \item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
% \item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
% \item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
% \end{itemize}
% \item {\bf Theory assumptions and proofs}
% \item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \cref{theorem : sign flip main}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include theoretical results.
% \item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
% \item All assumptions should be clearly stated or referenced in the statement of any theorems.
% \item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
% \item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
% \item Theorems and Lemmas that the proof relies upon should be properly referenced.
% \end{itemize}
% \item {\bf Experimental result reproducibility}
% \item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \cref{expt-setup-para}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
% \item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
% \item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
% \item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
% \begin{enumerate}
% \item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
% \item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
% \item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
% \item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
% \end{enumerate}
% \end{itemize}
% \item {\bf Open access to data and code}
% \item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We have submitted code.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that paper does not include experiments requiring code.
% \item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
% \item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
% \item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
% \item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
% \item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
% \end{itemize}
% \item {\bf Experimental setting/details}
% \item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \cref{expt-setup-para}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
% \item The full details can be provided either with the code, in appendix, or as supplemental material.
% \end{itemize}
% \item {\bf Experiment statistical significance}
% \item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: Each run was repeated thrice.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
% \item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
% \item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
% \item The assumptions made should be given (e.g., Normally distributed errors).
% \item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
% \item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
% \item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
% \item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
% \end{itemize}
% \item {\bf Experiments compute resources}
% \item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \cref{expt-setup-para}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
% \item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
% \item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
% \end{itemize}
% \item {\bf Code of ethics}
% \item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We follow the code of ethics.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
% \item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
% \item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
% \end{itemize}
% \item {\bf Broader impacts}
% \item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification:
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that there is no societal impact of the work performed.
% \item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
% \item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
% \item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
% \item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
% \item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
% \end{itemize}
% \item {\bf Safeguards}
% \item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification:
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper poses no such risks.
% \item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
% \item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
% \item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
% \end{itemize}
% \item {\bf Licenses for existing assets}
% \item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We have submitted our code.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not use existing assets.
% \item The authors should cite the original paper that produced the code package or dataset.
% \item The authors should state which version of the asset is used and, if possible, include a URL.
% \item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
% \item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
% \item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
% \item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
% \item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
% \end{itemize}
% \item {\bf New assets}
% \item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We have submitted code.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not release new assets.
% \item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
% \item The paper should discuss whether and how consent was obtained from people whose asset is used.
% \item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
% \end{itemize}
% \item {\bf Crowdsourcing and research with human subjects}
% \item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification:
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
% \item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
% \end{itemize}
% \item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
% \item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification:
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
% \item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
% \item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
% \end{itemize}
% \item {\bf Declaration of LLM usage}
% \item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
% %this research?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification:
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
% \item Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
% \end{itemize}
% \end{enumerate}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\input{Appendix}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
\section{Introduction}
As neural network sizes scale to billions of parameters, building efficient training pipelines is of paramount importance.
Sparsification plays a pivotal role in enabling such efficient pipelines \citep{frantar2023sparsegpt}.
State-of-the-art sparsification methods rely on eliminating parameters during training \citep{kuznedelev2024cap}.
% or by exploring multiple sparse masks dynamically \citep{evci-rigl, struct-dst, chen2021chasing} to find high-performing sparse networks.
The success of these methods can be attributed to finding an optimal combination of parameters and sparse masks \citep{paul2023unmasking}.
However, identifying them requires training a dense network before it can be sparsified such that information from the dense network can be inherited by the subsequent sparse network \citep{gadhikar2024masks}.
It is then no surprise that the most successful sparsification methods rely on dense training phases.
Dense-to-sparse methods like gradual pruning \citep{han-efficient-nns} or continuous sparsification \citep{kusupati20} explicitly start from a dense network.
% Dynamic sparse training methods like RiGL \citep{evci-rigl} utilize dense gradient information. %to achieve state-of-the-art performance.
However, this comes at the cost of high computational and memory demands during the dense training phase.
Moreover, the benefits of sparsification become mostly available towards the end of training or during inference.
Is there a way to train sparse networks from the start, given a sparse mask?
The Lottery Ticket (LT) Hypothesis \citep{frankle2019lottery} posits that there exist sparse networks at random initialization which enjoy optimal parameter-mask coupling and can achieve the same performance as their dense counterpart.
Finding such sparse networks at initialization would unlock the benefits of sparse training from scratch.
A plethora of pruning at initialization (PaI) methods have attempted to find such sparse networks at initialization \citep{snip, synflow, grasp}, however, they struggle to match the generalization performance of dense-to-sparse training methods.
Understanding and overcoming the limitations of PaI remains an open research problem and is the focus of our work.
The main challenge for PaI methods is to find highly performant parameter-mask combinations by leveraging task-specific information efficiently \citep{kumarno, frankle2021review}.
The problem is thus comprised of two parts: a) learning a mask and b) optimizing the weights corresponding to the mask, which is hard if the mask is extremely sparse.
% Here, we are less concerned with the mask itself but focus on the hard problem of training the parameters given a mask.
Here we are less concerned about learning a mask, but instead focus on the hard problem of optimizing parameter-mask combinations, given a mask at initialization.
At the center of our investigation is the question:
% \textit{Can a given sparse networks be trained from scratch such that they inherit task-specific information, effectively bridging the gap between dense-to-sparse training and PaI? And what are its limitations?}
\textit{What hinders training a sparse mask, and can the gap between PaI and dense-to-sparse be closed?}
The key to addressing this question lies in promoting parameter \emph{sign flips}.
Echoing the analysis of learning rate rewinding (LRR) by \citet{gadhikar2024masks}, through a broader empirical investigation of dense-to-sparse training methods, we identify a critical phenomenon responsible for finding optimal parameter-mask combinations: the ability to flip parameter signs during training.
We term this phenomenon sign alignment, as the signs encode all relevant information additionally to the mask (and data) to enable effective training from scratch. %alignment and are sufficient to identify a trainable sparse network.
Moreover, as we show, sign alignment occurs early in the dense-to-sparse training process, promoting convergence toward flatter minima.
% We demonstrate that initial dense training is crucial for enabling sign alignment, which explains why PaI methods face challenges when trained from scratch.
In contrast, PaI faces challenges in stabilizing sign flips.
In line with this observation, it was shown in a theoretically tractable setting of one-hidden-neuron that training a sparse network from scratch fails to flip signs from a bad initialization \citep{gadhikar2024masks}.
%As we prove in a theoretical
%\cite{signs}
%Motivated by this insight, we study the behaviour of parameter signs under gradient flow in a theoretically tractable student-teacher setup with a two-layer network consisting of one neuron.
Motivated by this insight, we propose a way to flip and learn correct signs during sparse training from scratch in a higher number of cases (see \cref{fig:quadrant-flips}).
We achieve this via \signin, a reparameterization $m \odot w$ of the non-masked parameters, providing them with an inner degree of freedom (where $m$ and $w$ are continuous parameters).
Note that this reparameterization, depending on the inner scaling, has a bias towards sparsity.
This has been used by the continuous sparsification methods Spred \citep{ziyin2023spredsolvingl1penalty} and PILoT \citep{jacobs2024maskmirrorimplicitsparsification}.
In contrast, we do not use the reparameterization for sparsification but for improving the optimization geometry to enable sign alignment and thus seek to counteract its sparsity bias.
We propose an inner scaling that promotes sign flips by rebalancing network layers and, correspondingly, the speed in which they learn.
This provably recovers a different sign case than dense training and could thus be seen as complementary (see \cref{fig:quadrant-flips}).
Moreover, in combination with a moderate degree of over-parameterization, which tends to be common in PaI, \signin fully solves the one-hidden-neuron problem.
%acts orthogonally to dense training,
%This inner degree creates a new balance between layers which is responsible for the sign flip. -> you just repeat the previous sentence
%Importantly, this mechanism acts orthogonally to dense training, offering an orthogonal approach. -> double orthogonal...
% This poses the question: Could there exist another unknown parameterization that could replace the sign flipping mechanism of dense training?
% Unfortunately, our answer is negative and we derive an impossibility statement.
%Furthermore, we show that it is impossible to replicate the sign flipping mechanism achieved by initial dense training in dense-to-sparse methods for sparse training with any reparameterization of the network.
%Instead, we propose an alternate mechanism to enable sign flips such that the correct signs can still be recovered with sparse training (see \cref{fig:quadrant-flips}).
% Our proposed mechanism is complementary to the sign flips achieved by dense training.
%\signin guarantees that the correct signs are learned with sparse training in our two-layer network example in contrast to standard training which fails.
%We verify this
%demonstrate in large-scale experiments that \signin also demonstrates its effectiveness by
%Moreover, since \signin complements the sign flips enabled by dense training,
In line with this insight, we verify in experiments that existing dense-to-sparse methods can be improved by utilizing \signin.
Most importantly, we demonstrate across multiple experiments
that \signin significantly enhances the performance of sparse training from scratch, including those using random masks.
We attribute this improvement to better sign alignment and the ability to discover flatter minima.
However, as we show, \signin cannot replicate the same sign flipping mechanism as that of initial dense training, and is still not competitive with the accuracy of dense-to-sparse methods.
%Hence, training sparse networks from scratch is inherently limited in comparison with dense-to-sparse methods.
This poses the question: Could there exist another unknown parameterization that could replace the sign flipping mechanism of dense training?
Unfortunately, the answer is negative, as our proof of an impossibility statement reveals.
%our answer is negative and we derive an impossibility statement.
While we can report progress in the right direction,
%This is substantiated by the fact that there exists no reparameterization that can replace dense training.
it remains an open problem to completely close the gap between PaI and dense-to-sparse methods.
% With our theoretical analysis, we aim to outline promising paths towards a solution.
%Overall our analysis shows that enabling appropriate sign flips is crucial for improving sparse training which can be achieved via our proposed method \signin.
%By doing so we can significantly improve the performance of any sparse training method including PaI and random masks.
%However, we also show that sparse training from scratch is limited because of its inability to flip the same signs as a dense-to-sparse training method.
%Although, we reduce the gap between sparse training and dense-to-sparse method.
\textbf{Contributions}
%\begin{itemize}
1) We empirically show that dense-to-sparse training methods identify trainable parameter-mask combinations via early sign alignment allowing them to find flatter minima and achieve state-of-the-art performance.
% 2) Parameter signs combined with their masks via sign alignment are sufficient for sparse training with random magnitudes and this sign alignment is absent in sparse networks trained from scratch.
2) We provide evidence for the following hypothesis: Across dense-to-sparse training masks, the learned signs define an initialization that enables competitive sparse training from scratch. A fully random initialization (that is decoupled from the mask) performs on par or worse than PaI (including random pruning).
%This sign alignment is absent in sparse networks trained from scratch.
3) To learn signs more effectively, we propose \signin, a weight reparameterization, which can provably \textemdash using Riemannian gradient flow and dynamical systems analysis \textemdash recover correct signs in a case that is complementary to overparameterization.
%in contrast to sparse training from scratch in a two-layer network.
%\signin is complementary to the sign flips enabled by dense-to-sparse methods.
4) Extensive empirical evaluation shows that \signin can significantly improve the performance of sparse training from scratch by improving sign alignment and finding flatter minima.
%\end{itemize}
\section{Related Work}
\textbf{Lottery Ticket Hypothesis (LTH)}
Training sparse neural networks from scratch is a hard problem \citep{kumarno}, but \citet{frankle2019lottery} posit that there exist initializations that make sparse networks trainable, i.e., Lottery Tickets.
% They occur as subnetworks of random source networks.
%and introduce Iterative Magnitude Pruning (IMP) to retroactively find these subnetworks.
However, their approach to find such initializations by iterative magnitude pruning (IMP) is computationally expensive and its performance does not transfer to larger models, even though lottery tickets should exist in theory \cite{cnnexist,convexist,depthexist,uniExist,BNtheoryCNN}.
\citet{rewindVsFinetune} improved its accuracy by Weight Rewinding (WR) and Learning Rate Rewinding (LRR).
Both WR and LRR are similarly expensive, but inherit the initialization from denser networks.
% \citet{paul2023unmasking, gadhikar2024masks} further validate that dense training enables sparse networks to find the right loss basin by flipping parameter signs appropriately.
\textbf{Dense-to-sparse methods}
%Dense-to-sparse training is known to achieve state-of-the-art performance in neural network sparsification.
%Iterative pruning methods like Learning Rate Rewinding are conceptually similar but require longer training \citep{rewindVsFinetune, han-efficient-nns}.
While dense training has been conjectured to be a critical factor to enter a good loss basin \citet{paul2023unmasking} or quadrant \cite{gadhikar2024masks}, other methods aim to reduce its duration.
Gradual pruning methods like AC/DC \citep{peste2021acdc}, CAP \citep{kuznedelev2024cap}, and WoodFisher \citep{singh2020woodfisher} prune based on a parameter importance score, interspersed with training.
Continuous sparsification methods such as STR \citep{kusupati20}, spred \citep{ziyin2023spredsolvingl1penalty}, and PILoT \citep{jacobs2024maskmirrorimplicitsparsification} reparameterize a dense neural network to induce sparsity.
Although such dense training phases seem to be critical to learn good parameter signs \cite{gadhikar2024masks}, we propose a complementary method, which also works for pruning at initialization.
An orthogonal class of algorithms, dynamic sparse training, includes RiGL \citep{evci-rigl, struct-dst, chen2021chasing}, SET \citep{mocanu-adaptive-sparse-train}, MEST \citep{yuan2021mest}, CHTs \citep{chts} which dynamically modify a sparse mask during training.
However, our work focuses on training a fixed mask from scratch.
\textbf{Pruning at initialization (PaI)}
A variety of methods have been proposed to sparsify networks at initialization based on different scores that assess parameter importance like saliency in SNIP \citep{snip} and GraSP \citep{grasp}, signal flow in Synflow \citep{synflow} and maximal paths \citep{pham-paths}.
Other methods include, DPaI \citep{xiangdpai}, Phew\citep{patil2021phew}, ProsPr \citep{alizadehprospect}.
However, randomly pruned sparse network can perform at par with these methods \citep{random-pruning-vita,er-paper,ma2021sanity,su2020sanity}, calling our ability to identify problem specific masks into question \citep{frankle2021review, jain2024winning}.
%highlighting the need for dense training to improve performance
\citet{kumarno} characterize the challenge from an information-theoretic lens, showing that initial dense training, in contrast to PaI, increases effective model capacity.
Our goal is not to address the mask identification problem but to improve sparse training given a mask, by a reparameterization.
% The above literature suggests that sparsely initialized networks suffer a huge disadvantage
\textbf{Reparameterization}
Reparameterizations have been studied from the perspective of implicit bias and gradient flow \citep{pmlr-v125-chizat20a, Li2022ImplicitBO, woodworth2020kernel, gunasekar2020characterizing, gunasekar2017implicit, chou2024induce, vaškevičius2019implicitregularizationoptimalsparse, li2023implicitregularizationgroupsparsity, Zhao_2022, li2021implicitsparseregularizationimpact, domin2024from, jacobs2024maskmirrorimplicitsparsification, jacobs2025mirrormirrorflowdoes, Marcotte2023AbideBT, marcotte2024momentumconservationlawseuclidean, liang2025implicit}.
It explains how overparameterization can improve generalization and how this is affected by various training factors such as the initialization, learning rate, noise, and explicit regularization.
An important assumption in these works is that the activation function is linear.
This simplification enables the reparameterization to be expressed using mirror flow or Riemannian gradient flow frameworks \citep{Li2022ImplicitBO}.
In contrast, we employ a reparameterization specifically to induce sign flips in the presence of a non-linear activation function.
%We can use a Riemannian gradient flow description despite the non-linear activation function as the reparameterization is applied to each weight separately.
Our proposed reparameterization has been applied to induce continuous sparsification \citep{ziyin2023spredsolvingl1penalty,jacobs2024maskmirrorimplicitsparsification, kolb2025deep} with the help of explicit regularization.
To combat this trend, we rescale the parameters dynamically exploiting a rescale invariance.
%Although explicit regularization is needed for good performance, it also prevents sign flips in this case.
%Therefore we rescale with the inner degree of freedom to reduce the side effect of explicit regularization and improve sign alignment.
%\section{Masks With Correct Signs Are Enough For Sparse Training}
\section{Correct Signs Facilitate Sparse Training}
\begin{figure}[ht!]
\centering
\includegraphics[width=0.8\linewidth]{Figures/flips_masks.png}
\caption{\textbf{Sign learning} a $90\%$ sparse ResNet50 on ImageNet. The majority of signs are flipped early for dense-to-sparse methods, upto warmup. Moreover, signs stabilize after warmp-up. In contrast, signs do not stabilize after warmup for sparse training from scratch for different masks.}
\label{fig:sign-motivation}
\label{fig:flips}
\label{fig:flip_masks}
\end{figure}
% \begin{figure}[ht!]
% \centering
% % Second subfigure
% \subfigure[Dense-to-sparse]{
% \includegraphics[width=0.2\textwidth]{Figures/flips.png}
% \label{fig:flips}
% }
% \subfigure[Sparse (PaI)]{
% \includegraphics[width=0.24\textwidth]{Figures/flips_masks.png}
% \label{fig:flip_masks}
% }
% % % Third subfigure
% % \subfigure[Sharpness]{
% % \includegraphics[width=0.29\textwidth]{Figures/eigs.png}
% % \label{fig:eigs}
% % }
% \caption{\textbf{Sign learning} in dense-to-sparse methods compared with training from scratch for a $90\%$ sparse ResNet50 on ImageNet. (a) The majority of signs are flipped early, upto warmup. Moreover, signs stabilize after warmp-up. (b) In contrast, signs do not stabilize after warmup for sparse training from scratch with different masks.}
% \label{fig:sign-motivation}
% \end{figure}
%\citet{paul2023unmasking, gadhikar2024masks} find that iterative pruning methods like Learning Rate Rewinding (LRR) and Weight Rewinding (WR), inherit information about the loss landscape from initial dense training by flipping parameter signs.
% This information is communicated via sign flipping of parameters \citep{gadhikar2024masks}.
%Inspired by \citet{paul2023unmasking, gadhikar2024masks}
\citet{paul2023unmasking} have established that iterative pruning methods like Learning Rate Rewinding (LRR) and Weight Rewinding (WR) rely on the effective exploration of the loss landscape during dense overparameterized training.
Dense-to-sparse training inherits its performance by training overparameterized in the beginning, identifying a good loss basin.
\citet{gadhikar2024masks} have shown that this loss basin can be characterized by the weight signs and thus the quadrant of the solution.
%Inspired by this finding, we investigate how signs are learned by dense-to-sparse methods and why they are more important than magnitudes for sparse training.
Our first objective is to show that this finding is not unique to LRR.
% which can be seen as variants of lottery tickets and are therefore expected to naturally facilitate training from scratch.
In fact, it is a common phenomenon that points to a critical limitation of PaI to learn signs.
%failure case of training from scratch to flip signs meaningfully
%We conjecture that
%Across dense-to-sparse training masks, the learned signs define an initialization that enables competitive sparse training from scratch. A fully random initialization (that is decoupled from the mask) performs on par or worse than PaI (including random pruning).
Our experiments reveal two key findings: (i) Initializing a network with an aligned sign and mask is sufficient for successful sparse training.
(ii) Sign alignment happens early during training for dense-to-sparse methods and guides the training trajectory towards flatter minima.
Specifically, we study three well-established sparsification methods to draw insights on sign alignment: magnitude based pruning with AC/DC \citep{peste2021acdc}, continuous sparsification with STR \citep{kusupati20}, and dynamic sparse training with RiGL \citep{evci-rigl}.
The first two methods leverage information from dense training by alternating dense and sparse training and gradual sparsification by soft thresholding respectively.
Although, RiGL does not involve a dense training phase, it retains dense gradient information, which we observe serves a similar purpose.
For comparison with sparse training, we train a randomly pruned sparse mask from scratch, which has been shown to be competitive with (and even better than) other PaI methods \citep{random-pruning-vita} and serves as a strong baseline.
Noteworthy is the fact that the random mask contains no information about the task.
Even though other masks should therefore have an advantage, as we will see later, they cannot shine through because of limited trainability.
%but it cannot show with random initializations.
%In contrast, we find
%As we will see later
%their tend to perform
%Yet, we show that the training success
% This information is employed to align signs with the mask during training.
\begin{wraptable}{r}{0.58\textwidth}
% \begin{table}[ht!]
\centering
\caption{\textbf{Sign alignment} for a ResNet50 trained on ImageNet with $90\%$ sparsity. Baseline performance is recovered when the learned mask is initialized with the learned parameter signs, but not with the learned parameter magnitude and random signs.}
\label{tab:sign-importance}
\begin{tabular}{lcc | c}
\toprule
Method & AC/DC & STR & RiGL
\\
\midrule
Baseline & $\underline{74.68}$ & $\underline{73.65}$ & $\underline{73.75}$ \\ \midrule
Epoch $10$ sign\textsubscript{ + learned mask} & $73.97$ & $71.7$ & $73.32$ \\
Epoch $30$ sign \textsubscript{ + learned mask} & $\mathbf{74.89}$ & $\mathbf{73.91}$ & $73.7$\\
Final sign\textsubscript{ + learned mask} & $74.88$ & $73.77$ & $\mathbf{73.74}$ \\ \midrule
Epoch $10$ mag\textsubscript{ + learned mask} & $70.93$ & $68.01$ & $72.30$ \\
Final mag\textsubscript{ + learned mask} & $70.94$ & $68.35$ & $72.40$ \\
\midrule
Random init \textsubscript{ + learned mask} & $70.6$ & $68.38$ & $71.89$\\
\bottomrule
\end{tabular}
% \end{table}
\end{wraptable}
\textbf{Signs are learned early}
To study the alignment between the sparse mask and parameter signs, we track them during dense-to-sparse training of a ResNet50 on ImageNet (see \cref{expt-setup-para} for details).
As shown in \cref{fig:flips}, the initial signs and final learned signs are significantly different since approximately $50\%$ of the signs are flipped for all methods (blue bar), i.e., the signs at initialization have a close to random overlap with the final learned signs.
As training progresses, upto the warmup phase ($10$-th epoch), a significant number of signs flip, as shown by the red bar in \cref{fig:flips}.
After the warmup phase, fewer signs flip in addition (green bar in \cref{fig:flips}).
% Most importantly, these initial sign flips are already close to the finally learned ones as there is a large overlap between the signs at warmup and the final learned ones (green bar in \cref{fig:overlap}).
%We find a large overlap between the signs at warmup and the final learned ones (green bar in \cref{fig:overlap}).
In comparison, training the same masks from scratch starting from a random re-initialization (which is decoupled from the dense-to-sparse training procedure)
does not identify stable signs early.
The performance of these masks is at par with a random mask trained from scratch (see \cref{tab:mw-differentt-masks} in the appendix) in the absence of sign alignment.
Moreover, the sign flip statistics show a similar pattern as training a random mask from scratch.
A significant number of sign flips occur after warmup, and the overlap between initialization, warmup and the final signs is closer to random (see \cref{fig:flip_masks}).
Across all three dense-to-sparse baselines, we observe that signs are learned early during training and remain largely stable after warmup.
In contrast, sparse training does not find stable signs to enable sign alignment, reflecting in its worse performance.
% The task-specific information that should also be encoded in a mask does not seem to affect the training from scratch performance.
% We identify the parameter signs coupled with sparse masks.
\textbf{Signs are more informative than magnitude}
%After observing characteristic differences in sign statistics between dense-to-sparse training and training from scratch, we can wonder: How much information do
As we verify next, the critical information about the task that is missing from the mask are the parameter signs.
%that the signs carry the critical information to facilitate sparse training.
%Having identified stable sign learning in dense-to-sparse methods, we now verify if these signs can be used for sparse training.
To do so, we use signs learned at different stages during training, the learned mask and random magnitudes for each method to initialize a sparse network.
As reported in \cref{tab:sign-importance}, final signs combined with the mask are sufficient for sparse training and achieve performance comparable to the baseline.
Moreover, since the signs are identified as early as the $10$-th epoch during training, these early signs (combined with the mask) recover baseline performance with an error of $\approx 1 \%$.
Full baseline performance can be recovered by using signs from the $30$-th epoch.
In contrast, if we initialize the sparse network with magnitudes learned at intermediate epochs, learned mask and random signs, we barely improve over training a randomly intialized sparse mask from scratch \cref{tab:sign-importance}.
Our results not only suggest that sign alignment is sufficient for sparse training, but it also happens early during dense-to-sparse training.
Sparse training from scratch with falls short in comparison.
%However, this is not the case in sparse training from scratch as the network does not enjoy the benefit of initial dense training to enable alignment.
% Early learning of signs in training can play a similar role as the LRR algorithm \citep{paul2023unmasking}, in identifying the loss basin, which is verified by the flatter minima found by dense-to-sparse methods in \cref{fig:eigs}.
% On the other hand, sparse training is unable to find a stable sign alignment (see \cref{fig:overlap}), which results in poor performance and a sharper minima \cref{fig:eigs}.
\textbf{Flatter loss landscapes with dense-to-sparse methods}
To dive deeper into the benefits of sign alignment for trainability, we
track the maximum eigenvalue of the Hessian of the loss as a proxy of sharpness \citep{jiangfantastic}.
It can also be interpreted as a proxy measure for the tractability (or conditioning) of the optimization problem.
As shown in \cref{fig:mw-eigs}, the dense-to-sparse baselines AC/DC, STR and RiGL have smaller sharpness indicating a flatter loss landscape once signs align during training.
% RiGL also converges to a similarly flat minimum, however it has a slightly sharper trajectory likely due to dynamic mask shuffling.
In comparison, the random sparse network converges to a sharper minimum.
Our observations lead to the conclusion that dense-to-sparse methods are effective because they inherit critical information from dense training in the form of \emph{parameter signs} during the early training phase. These signs play a crucial role in dictating the training trajectory of the sparse network.
In contrast, this mechanism is absent in training a fixed mask from scratch, resulting in poor performance and a sharper loss landscape during training, which indicates convergence to an inferior loss basin.
A natural question arises: Can we achieve a similar sign alignment in sparse training from scratch?
While it seems impossible to replicate the benefits of initial dense training while starting sparse, we demonstrate that sign alignment can be improved using an orthogonal sign flipping mechanism, which we induce by a network reparameterization.
As we show, this reparameterization helps to learn correct ground truth signs in a theoretically tractable case of a single neuron two-layer network.
%reparameterizing the network as we
% Following these observations, we ask the question, can we enable sign-flips in a random sparse mask such that stable sign configurations are identified which translate to improved performance.
% A crucial roadblock in performing this exercise is the lack of initial overparametrization which is crucial in enabling these sign-flips.
% In the next section, we introduce a reparametrization of the parameters, which enables appropriate sign-flips and is able to provably recover the correct solution on a toy single hidden neuron example.
% We also validate our proposed reparametrization on large scale experiments.
% We characterize the role of signs and magnitudes during sparse training, by studying the alignment between the sparse mask identified by a dense-to-sparse method with the signs.
% We start by initializing a network with the sparse mask identified by AC/DC, STR or RiGL, and isolating the parameter signs and magnitudes.
% Results of these ablations on ImageNet are reported in \cref{tab:sign-importance}.
% A sparse network initialized with the final sparse mask and parameter signs learned by a dense-to-sparse method, with randomly initialized magnitudes and trained with SGD (without any dense training) is able to match the performance of the original baseline.
% However, repeating the same experiment with learned magnitudes and randomly initialized signs with the final sparse mask leads to significantly worse performance.
% These results verify our first claim that signs aligned with the mask contain most of the information for sparse training.
% This highlights the inability of sparse neural networks to correctly identify the signs for different mask architectures.
% Drawing from the statistics, that a large number of signs already flip and stabilize by the end of the warmup phase, we verify if these signs are already sufficient for sparse training.
% We perform a similar ablation as before, initializing a network with the learned sparse mask, and the signs learned at warmup by each of the baselines.
% This initialization too is able to reach within $1\%$ of the baseline for AC/DC and RiGL (see \cref{tab:sign-importance}, indicating that signs align with the mask early in training.
% In case of STR, we observe that alignment happens slightly later at the $30$-th epoch.
% Therefore, the most important sign flips happen in the first part of the training.
% Hence, if we want to improve sparse training we need to allow for sign flips in the first part of training.
% \begin{itemize}
% \item Random masks have a low overlap between signs early in training and final signs. Signs do not stabilize early.
% \item Dense-to-sparse schemes affect a majority of the sign-flips early in training and find stable sign configurations as early as the $10$-th epoch.
% \item These sign configurations, also correlate with flatness in the loss landscape as we observe dense-to-sparse schemes have a lower maximum eigenvalue of the Hessian throughout training in comparison to a random mask.
% \end{itemize}
% \begin{itemize}
% \item Add PiLoT here and LRR for motivation.
% \item ACDC with mw to show that mw learns differently from dense training and is complementary to it.
% \item Take the mask learned from each of these methods, then do MW with it and see if we can bridge the gap correct signs + mask and sign flipping with mw.
% \item Also do the random initialization with the learned mask for these methods to show that this is not enough.
% \end{itemize}
% \begin{itemize}
% \item ResNet50 ImageNet experiments on all the baselines which highlight that the sign is sufficient to achieve good performance.
% \item The signs learned at warmup are also good enough in most cases. For STR, the signs are learned slightly later at the time of sparsification.
% \item Once learned they have a high overlap with the final signs.
% \item This phenomenon is absent in the case of sparse training from scratch with a random mask.
% \item We also see that the sign flips enable better conditioning for these dense-to-sparse training methods as they follow loss trajectories with lower sharpness (flatter) and flatness tends to correlate with better generalization.
% \item On the contrary, having only the knowledge of magnitude is not sufficient to retain performance of the mask.
% \end{itemize}
%\section{Algorithm: Enabling Sign Flips}
\section{Sign-In: Learning Sign Flips}\label{sec: alg sign in}
To improve the learning of task relevant sign flips, we introduce a reparameterization.
%that induces a Riemannian gradient flow \citep{Li2022ImplicitBO}. This holds for general continuously differentiable objective function $L : \mathbb{R}^m \rightarrow \mathbb{R}$.
Every weight parameter $\theta$ is replaced by the product of two parameters $m \odot w$, where $\odot$ denotes a pointwise multiplication (Hadamard product).
%Particularly, both $a$ and $w$ are reparameterized with $g(a_1, a_2) = a_1 a_2$ and $g(w_1, w_2) = w_1 w_2$.
This introduces an additional degree of freedom $\beta$, which is determined by the initialization and captured by $m^2 - w^2 = \beta \mathbf{1}$,
where $\beta > 0$ and $\mathbf{1} = (1, ..., 1)$.
%We exploit the degrees of freedom to improve sign alignment, as we show in Section \ref{sec:theory}.
According to Lemma 4.8 \citep{Li2022ImplicitBO}, the reparameterization induces a Riemannian gradient flow in the original parameter $\theta$:
\begin{equation}\label{alg: general riemannian gf}
d \theta_t = - \sqrt{\theta_t^2 + \beta} \odot \nabla L(\theta) dt, \qquad \theta_0 = \theta_{init}
\end{equation}
This holds for general continuously differentiable objective function $L : \mathbb{R}^m \rightarrow \mathbb{R}$.
%Eq.~(\ref{alg: general riemannian gf}) provides critical insights into leveraging the degrees of freedom to recover the sign.
%We have observed that using the same inner scaling $\beta_1 = \beta_2 = \beta>0$ still has benefits when using a finite learning rate and having more than one input weight.
To induce helpful sign flips, we have to deal with two factors: stochastic noise \citep{pesme2021implicit} and weight decay \citep{jacobs2024maskmirrorimplicitsparsification}.
Both factors have the same effect on the training dynamics: They induce sparsity by shrinking $\beta$.
Yet, if $\beta = 0$, we cannot sign flip due to the $\sqrt{\theta_t^2}$ in the Riemannian gradient flow Eq.~(\ref{alg: general riemannian gf}).
We address this issue with Algorithm \ref{app: alg sign in} in \cref{app: alg} by resetting the inner scaling $\beta$ without changing the weights of the network every $p$ epochs up to epoch $T_2$.
In all our experiments, we choose the same scaling of $\beta=1$ for all parameters, as this leads to stable results.
% Furthermore, we apply a (random) mask $S$.
%The parameters are initialized such that $m_0^2 +w_0^2 =1$, scheme as in \cite{jacobs2024maskmirrorimplicitsparsification}, this is to make sure the learning rate is
%Next, we reset the inner scaling $\beta$
We employ an analytic solution to apply the rescaling on both $m$ and $w$, as given in Appendix \ref{appendix : init}.
This gives the weights more opportunities to align their signs correctly, as proven for a simple example in Section~\ref{sec:theory}.
%We exploit the degrees of freedom to improve sign alignment, as we show in Section \ref{sec:theory}.
%In all our experiments, we choose a scaling of $\beta=1$ for all parameters.
\textbf{Computational and memory cost}
Sign-In doubles the amount of parameters relative to sparse training.
This is also the case for continuous sparsification methods that utilize the implicit bias of $m \odot w$ towards sparsity, like spred \citep{ziyin2023spredsolvingl1penalty} and PILoT \citep{jacobs2024maskmirrorimplicitsparsification}.
Nevertheless, according to \cite{ziyin2023spredsolvingl1penalty}, the training time of a ResNet50 with $m \odot w$ parameterization on ImageNet increases roughly by 5\% only and the memory cost is negligible if the batch size is larger than $50$. Furthermore, at inference, we would return to the representation $ \theta$ by merging $\theta = m \odot w$.
Therefore, \signin is able to improve sparse training with minimal overhead as also highlighted by a FLOP comparison in \cref{tab:flops-signin}.
%In addition, training with a sparse initialization can lead to a speed up on specialized hardware such as that of Cerebras \citep{10123162}.
%This is possible for the reparameterization as we have the following equivalence:
%\begin{equation*}
% S \odot m \odot w = S\odot m \odot S \odot w.
%\end{equation*}
% \begin{algorithm}[ht!]
% \caption{Sign-In}
% \label{alg: sign in}
% \begin{algorithmic}
% \STATE {\bfseries Input:} objective $L$, scaling $\beta$, frequency $p$, epochs $T$, mask $S$, stop rescaling epoch $T_2$
% \STATE Initialize $L(S \odot m_0 \odot w_0)$ such that $m_0 \odot w_0 = \theta_0$ and $m_0^2 - w_0^2 = \beta \mathbf{I}$.
% \FOR{$i=1$ {\bfseries to} $T$}
% \IF{$i$ mod $p == 0$ and $i < T_2$}
% \STATE Rescale $m_i \odot w_i = \theta_i$ such that $m_i^2 - w_i^2 = \beta \mathbf{I}$
% \ENDIF
% \STATE $m_{i+1}, w_{i+1} = \text{Optimizer}\left(L\left(S \odot m_i \odot w_i\right)\right)$
% \ENDFOR
% \STATE {\bfseries Return} Model parameters $\theta_T = S\odot m_T \odot w_T$
% \end{algorithmic}
% \end{algorithm}
%\section{Sign Learning in Student-Teacher Setups}
\section{Sign Learning in Theory}
\label{sec:theory}
%\begin{figure}[ht]
% \centering
% Second subfigure
%\subfigure[Dense.]{
% \includegraphics[width=0.23\textwidth]{Figures/ToyExamples/dense_final.png}
% \label{fig:dense}
%}
%\subfigure[Good signs.]{
% \includegraphics[width=0.23\textwidth]{Figures/ToyExamples/goodsigns_init.png}
% \label{fig:good sparse}
%}
%\subfigure[Bad signs.]{
% \includegraphics[width=0.45\linewidth]{Figures/ToyExamples/badsigns_final.png}
% \label{fig:bad sparse}
%}
%\subfigure[Reparameterization.]{
% \includegraphics[width=0.45\linewidth]{Figures/ToyExamples/badsigns_finalmw.png}
% \label{fig:mw}
%}
%\caption{Representation of a two layer student-teacher neural network inspired by \citet{chizat2020lazy}. Signs are crucial in learning sparse representations. A dense neural network can easily learn the representation in Fig.~\ref{fig:dense}. Sparse networks can learn when signs are correct Fig.~\ref{fig:good sparse} and fail with bad signs Fig.~\ref{fig:bad sparse}. The reparameterization enables learning the representation with bad signs in Fig.~\ref{fig:mw}.}
%\label{fig:sign-representation}
%\end{figure}
%We consider a two-layer neural network with a balanced initialization in a student-teacher setup. This setup captures the essential dynamics between two sparse layers in a neural network.
%We begin by introducing the setup and restating the result from \citet{gadhikar2024masks}, which demonstrates that recovering the correct signs of the weights is not always possible in this context. Due to the balanced initialization \citep{dugradient} the layers stay coupled, preventing recovering the correct signs of the weights in both layers.
\citet{gadhikar2024masks} proposed a simple testbed to compare two iterative pruning algorithms, IMP and LRR.
LRR's superior performance is explained by its ability to flip the sign of a critical parameter during a dense training phase.
As we show, the same mechanism distinguishes sparse training from scratch with random initialization and dense training.
The next section is dedicated to the question how \signin changes this picture.
%Our main goal is to reason in this context how a \signin parameterization aids sign learning in case of unfortunate initializations.
%Remarkably, \signin resolves an orthogonal case and
%which is the mechanism that we are looking for to
%to reason about their ability to flip parameter signs and how it can be supported by \signin.
Remarkably, it provably resolves another case than dense (overparameterized) training.
In combination with dense training, it can flip even all relevant signs and solve the learning problem (see Figure~\ref{fig:quadrant-flips}).
\begin{figure}[ht!]
\centering
\begin{tikzpicture}[scale=0.7]
% Define colors for quadrants
\definecolor{green}{rgb}{0.3, 0.9, 0.3} % Bright Green
\definecolor{red}{rgb}{0.9, 0.2, 0.2} % Bright Red
% First square
\begin{scope}[shift={(-3,0)}] % Shift left for the first square
% Draw quadrants
\fill[green] (0,0) rectangle (1,1); % Top-right
\fill[red] (-1,0) rectangle (0,1); % Top-left
\fill[red] (-1,-1) rectangle (0,0); % Bottom-left
\fill[red] (0, -1) rectangle (1,0); % Bottom-right
% Draw axes
\draw[thick, <->] (-1.1,0) -- (1.1,0) node[below right] {}; % Horizontal axis
\draw[thick, <->] (0,-1.1) -- (0,1.1) node[above left] {}; % Vertical axis
\node at (0.5, 0.5) { \checkmark};
\node at (-0.5,0.5) { \xmark};
\node at (-0.5,-0.5) { \xmark};
\node at (0.5,-0.5) { \xmark};
% Add label below the square
% \node[anchor=north] at (0,-1.4) {(a)};
\node at (0, 1.9) {Sparse};
\node[left=-5pt] at (-0.15, 1.15) {$w$};
\node at (1.2, -0.15) {$a$};
\end{scope}
% Second square
\begin{scope}[shift={(0,0)}] % Shift right for the second square
% Draw quadrants
\fill[green] (0,0) rectangle (1,1); % Top-right
\fill[red] (-1,0) rectangle (0,1); % Top-left
\fill[red] (-1,-1) rectangle (0,0); % Bottom-left
\fill[green] (0,-1) rectangle (1,0); % Bottom-right
% Draw axes
\draw[thick, <->] (-1.1,0) -- (1.1,0) node[below right] {}; % Horizontal axis
\draw[thick, <->] (0,-1.1) -- (0,1.1) node[above left] {}; % Vertical axis
\node at (0.5, 0.5) { \checkmark};
\node at (-0.5,0.5) { \xmark};
\node at (-0.5,-0.5) { \xmark};
\node at (0.5,-0.5) { \checkmark};
% \node[anchor=north] at (0,-1.4) {(b)};
\node at (0, 1.9) {Overparam};
\node[left=-5pt] at (-0.15, 1.15) {$w$};
\node at (1.2, -0.15) {$a$};
\end{scope}
% Second square
\begin{scope}[shift={(3,0)}] % Shift right for the second square
% Draw quadrants
\fill[green] (0,0) rectangle (1,1); % Top-right
\fill[green] (-1,0) rectangle (0,1); % Top-left
\fill[red] (-1,-1) rectangle (0,0); % Bottom-left
\fill[red] (0,-1) rectangle (1,0); % Bottom-right
% Draw axes
\draw[thick, <->] (-1.1,0) -- (1.1,0) node[below right] {}; % Horizontal axis
\draw[thick, <->] (0,-1.1) -- (0,1.1) node[above left] {}; % Vertical axis
\node at (0.5, 0.5) { \checkmark};
\node at (-0.5,0.5) { \checkmark};
\node at (-0.5,-0.5) { \xmark};
\node at (0.5,-0.5) { \xmark};
% \node[anchor=north] at (0,-1.4) {(c)};
% \node at (0, 1.4) {Sparse + \signin};
\node[align=center] at (0, 2) {Sparse\\ + \signin};
\node[left=-5pt] at (-0.15, 1.15) {$w$};
\node at (1.2, -0.15) {$a$};
\end{scope}
% Second square
\begin{scope}[shift={(6,0)}] % Shift right for the second square
% Draw quadrants
\fill[green] (0,0) rectangle (1,1); % Top-right
\fill[green] (-1,0) rectangle (0,1); % Top-left
\fill[green] (-1,-1) rectangle (0,0); % Bottom-left
\fill[green] (0,-1) rectangle (1,0); % Bottom-right
% Draw axes
\draw[thick, <->] (-1.1,0) -- (1.1,0) node[below right] {}; % Horizontal axis
\draw[thick, <->] (0,-1.1) -- (0,1.1) node[above left] {}; % Vertical axis
\node at (0.5, 0.5) { \checkmark};
\node at (-0.5,0.5) { \checkmark};
\node at (-0.5,-0.5) { \checkmark};
\node at (0.5,-0.5) { \checkmark};
% \node[anchor=north] at (0,-1.4) {(c)};
% \node at (0, 1.4) {Overparam + \signin};
\node[align=center] at (0, 2) {Overparam\\+ \signin};
\node[left=-5pt] at (-0.15, 1.15) {$w$};
\node at (1.2, -0.15) {$a$};
\end{scope}
\end{tikzpicture}
\caption{\textbf{\signin recovers the solution in a different case than overpameterization} for a two-layer network by sign flipping. Combining the two solves all cases (empirically).}
\label{fig:quadrant-flips}
\end{figure}
%Following this,
%we use the reparameterization to recover the sign of the outer layer, given that the inner layer has the correct sign.
%This is the main mechanism used by our proposed Algorithm \ref{alg: sign in} (\signin).
%Next, we show empirically for the multi-input (overparameterized) case that all input layers can be recovered.
%Furthermore, we show that no reparameterization can recover the sign of the inner layer for the single-input (sparse) case.
%We first analyze the case with a single input and a single neuron and then generalize to multiple inputs and multiple neurons.
\textbf{Single neuron setting}
%\begin{setting}[Single neuron]
Let $\{\mathbf{z}_i\}_{i =1}^n$ be a data set of i.i.d $\mathcal{N}(\mathbf{0},\mathbf{I})$ random variables.
Consider a two-layer neural network consisting of a single hidden neuron $f : \mathbb{R}^d
\rightarrow \mathbb{R}$ such that
% \begin{equation*}
% f_0(a,w,z) := a \sigma \left( w z \right),
% \end{equation*}
$f(\mathbf{z} \mid a, \mathbf{w}) := a \sigma \left( \mathbf{w}^T \mathbf{z} \right)$, where $\sigma( \cdot ) := \max\{\cdot, 0\}$ denotes the Rectified Linear Unit (ReLU).
%Without loss of generality, we choose the teacher to be a two-layer neural network $\Tilde{f}_0$ such that $\Tilde{a} > 0$, $\Tilde{w} > 0$ and $\Tilde{a} \Tilde{w} = 1$.
The labels $\{y_i\}_{i=1}^n$ are generated from a sparse teacher $y_i = \Tilde{f} \left(\mathbf{z}_i\mid \Tilde{a}, \Tilde{\mathbf{w}}\right)$ with $\Tilde{a} > 0$ and $\Tilde{\mathbf{w}} = (1/\Tilde{a},0,0,..,0)$.
On this data, we train a student two-layer neural network with gradient flow and randomly balanced parameter initialization, i.e., $a^2_{\text{in}} = \norm{\mathbf{w_{\text{in}}}}^2_2$.
The objective is to minimize the mean squared error (MSE), which is given by $L(a,\mathbf{w}) : =
\frac{1}{2n} \sum_{i = 1}^n \left(f\left(\mathbf{z}_i \mid a, \mathbf{w}\right) - y_i\right)^2$.
%\end{setting}
% \begin{equation*}
% L(a,w) : =
% \frac{1}{2n} \sum_{i = 1}^n \left(a \sigma\left(w z_i\right) - y_i\right)^2.
% \end{equation*}
The training dynamics with loss function $L$ is described by the gradient flow:
\begin{equation}
\begin{cases}\label{theory : gf}
d a_t = - \partial_a L(a_t, \mathbf{w}_{t})dt,\qquad a_0 = a_{\text{in}}\\
d \mathbf{w}_{t} = -\nabla_{\mathbf{w}} L(a_t, \mathbf{w}_{t})dt, \qquad \mathbf{w}_0 = \mathbf{w}_{\text{in}}.
\end{cases}
\end{equation}
In this setting, we are interested in comparing two cases:
a) We call the multi-dimensional case with $d>1$ the overparameterized case, as we could also set the weight parameters $w_j$ with index $j>1$ to zero.
We also denote the student as $f_1$.
Dense-to-sparse training would typically first train all parameters, then prune all weights with $j>1$, and continue training only $w_1$ and $a$ after pruning.
b) The one-dimensional case $d=1$ can be seen as special case where we mask the components $j>1$ from the start. It is equivalent to considering a 1-dimensional problem with $d=1$.
We also write $f_0$ for the student, which faces generally a harder problem, as we establish next.
\textbf{Sparse training}
If the student is one-dimensional ($d=1$), it can only recover the ground truth if it is initialized with the correct signs, i.e.
%The training dynamics Eq.~(\ref{theory : gf}) can only recover the ground truth teacher network $\Tilde{f}_0$
when $a_{in} > 0$ and $w_{in} > 0$.
%This is encapsulated in the following theorem by \citet{gadhikar2024masks}.
\begin{theorem}(Theorem 2.1 \citep{gadhikar2024masks}) \label{thm : 2.1 Advait}
In the one-dimensional single neuron setting, Eq.~(\ref{theory : gf}) with $d=1$, the student can recover the ground truth given sufficiently many samples, if $a_{in} > 0$ and $w_{1,in} > 0$. In all other cases ($a_{in} > 0$, $w_{1,in} < 0$), ($a_{in} < 0$, $w_{in} > 0$) and ($a_{in} < 0$, $w_{1,in} < 0$), it will not attain the correct solution.% to a correct solution.
\end{theorem}
% \begin{figure}[ht!]
% \centering
% \includegraphics[width=0.8\linewidth]{Figures/mw_success.pdf}
% \caption{\signin recovers the solution in a different case from dense-to-sparse methods for a two-layer network by sign flipping.}
% \label{fig:quadrant-flips}
% \end{figure}
% \paragraph{Reparameterization of the network}
\textbf{\signin reparameterization}
The \signin reparameterization, which we have introduced in Section \ref{sec: alg sign in}, can resolve the additional case $a_{in} < 0, w_{1,in} >0$ when $d=1$.
%To improve the learning, we introduce a parameterization that induces Riemannian gradient flow \citep{Li2022ImplicitBO}.
Note that both $a$ and $w$ are reparameterized with different inner scalings denoted by $\beta_1$ and $\beta_2$. %with $g(a_1, a_2) = a_1 a_2$ and $g(w_1, w_2) = w_1 w_2$.
%This introduces two additional degrees of freedom, which are determined by the initialization.This is captured by the following set of equations:
%\begin{equation*}
% \begin{cases}
% a_1^2 - a_2^2 = \beta_1 \\
% w_1^2 - w_2^2 = \beta_2.
% \end{cases}
%\end{equation*}
%We exploit the degrees of freedom to recover the case $a_0 < 0, w_0 >0$.
%According to Lemma 4.8 \citep{Li2022ImplicitBO} the reparameterization induces a
The corresponding Riemannian gradient flow in the original parameters $a$ and $w_1$ is:
\begin{equation}
\begin{cases}
d a_t = - \sqrt{a_t^2 + \beta_1} \partial_a L(a_t, w_{1,t})dt,\quad a_0 = a_{\text{in}}\\
d w_{1,t} = -\sqrt{w_{1,t}^2 + \beta_2}\partial_{w_1} L(a_t, w_{1,t})dt, \quad w_{1,0} = w_{1,\text{in}}.
\end{cases}\label{theory : riemann gf}
\end{equation}
%The Riemannian gradient flow in Eq.~(\ref{theory : riemann gf}) provides critical insights into leveraging the degrees of freedom to recover the sign.
If $w_{1,t} > 0$ for all $t > 0$, $L$ is continuously differentiable.
%We will refer to Eq.(\ref{theory : riemann gf}) and other reparameterized network trainings as the \signin gradient flow.
\begin{theorem}[Sign-In improves PaI]\label{theorem : sign flip main}
Let $a$ and $w_1$ follow the \signin gradient flow dynamics in Eq.~(\ref{theory : riemann gf})
%with a random balanced parameter initialization
and let $\beta_1 > \beta_2>0$. If $w_{1,in} > 0$, then the student $f_0$ can learn the correct target with probability $1- \left(\frac{1}{2}\right)^n$.
If $w_{1,in} \leq 0$, it still fails to recover the ground truth.
\end{theorem}
Proof. The proof is given after Theorem \ref{theorem : sign flip appendix} in the appendix. The main steps are: a) Characterization of the gradient field at balanced initialization, breaking the coupling that causes it to approach zero.
b) Determining the direction of the stable and unstable manifold at the origin (saddle point).
c) Demonstrating that the dynamical system must converge towards a stationary point.
\begin{remark}
The proof combines tools from dynamical systems theory and mirror flow theory to show convergence, which might be of independent interest.
\end{remark}
To illustrate Theorem \ref{theorem : sign flip main}, we simulate the toy setup for both the standard gradient flow and the reparameterized gradient flow.
Figure \ref{fig:1d riemann vs normal} shows that the standard gradient flow converges only when the signs are correctly initialized, whereas the reparameterized model also succeeds if $a_{in} < 0, w_{1, in} > 0$. Other initializations lead the student to converge to the origin and thus a function that is constant at $0$, i.e. training fails.
% \begin{figure}[ht]
% \centering
% \includegraphics[width=0.35\linewidth]{Figures/signin_gf.png}
% % \includegraphics[width=1\linewidth]{Figures/ToyExamples/TrajOverparamMasksignResult.pdf}
% \caption{\textbf{Student-teacher setup training one neuron} For a single input $d=1$, when $w_1 \leq 0$ both methods fail to reach the ground truth. If $w_1 > 0$, the \signin gradient flow succeeds, whereas gradient flow fails in this case when additionally $a < 0$.}
% \label{fig:1d riemann vs normal}
% \end{figure}
Note that the condition $\beta_1 > \beta_2>0$ in Theorem~\ref{theorem : sign flip main} is necessary.
In practice, this suggests that each layer would require different scaling, which can be hard to tune.
% This practically is hard to implement as this suggest that all layers need different scaling.
We show next that this is not needed and the same scaling can be used for each layer in practice.
\textbf{Multi-dimensional input}
For multi-dimensional input ($d>1$), the outer layer parameter $a$ cannot sign flip like in case of $d=1$ (Lemma 2.2 \citep{gadhikar2024masks}).
However, we have shown that when the rescaling of $a$ is larger then that of $w_1$, i.e., $\beta_1 > \beta_2>0$, the sign of $a$ is recovered, if $w_1$ initialized with the correct sign.
This is impractical for larger neural networks with more layers, since we have to choose a different scaling $\beta$ for all layers.
Nonetheless, for a multi-dimensional input layer, different rescaling is not necessary, as we verify empirically next.
% We empirically investigate the multi-dimensional network $f_1: \mathbb{R} \times \mathbb{R}^d \times \mathbb{R}^d
% \rightarrow \mathbb{R}$ described by $ f_1(\mathbf{z} \mid a,\mathbf{w}) : = a \sigma( \mathbf{w}^T\mathbf{z})
% $.
% We generate a data set $\{\mathbf{z}_i\}_{i=1}^n$ consisting of i.i.d. random variables drawn from $N(0, \mathbf{I}_d)$.
% The labels are generated from a teacher network $\Tilde{f}_1$ such that $\tilde{a} = 1$ and $\mathbf{\tilde{w}} = \left[1, 0, ..., 0\right]$.
% % $\tilde{a} = 1$ and $\mathbf{\tilde{w}}$ is a random vector.
% In the experiment, we set $n = 500$ and $d = 5$.
% We reparameterize the student network $(a, \textbf{w})$ with $(a_1 a_2, \mathbf{w}_1 \odot \mathbf{w}_2)$, where $\odot$ is the pointwise product (Hadamard product).
% The inner scaling for all reparameterizations is set to $\beta =1$.
% The initialization of the training is balanced.
%We set $a_{in} = -1$ and $\mathbf{w}_{in} \sim N(0,\mathbf{I}_d)$ followed by normalization. We also set $w_{init, 0} > 0$, which is the first element of the vector $\mathbf{w}_{in}$.
We train the student network with $d=5$ for time $T=200$ with gradient descent and a learning rate $\eta > 0$ such that total steps are $T / \eta$.
% The simulation time is set to $T = 100$, which means we run gradient descent with learning rate $\eta > 0$ for $\frac{T}{\eta}$ steps.
The experiment is repeated for $100$ different initializations.
The success percentages training runs are presented in Table \ref{tab:multi input}.
We observe that \signin consistently recovers the sign of $a$, with same scaling $\beta=1$.
% Note that if all signs of $\mathbf{w}_0$ need to be correct, as in the $1-$dimensional case, the success probability would be $(\frac{1}{2})^d \approx 0.1 \%$, which is significantly smaller than the observed $18\%$.
\begin{wraptable}{r}{0.5\textwidth}
% \begin{table}[ht!]
\centering
\caption{Percentage of sign recoveries if $a_0<0$ for standard and \signin gradient descent using different learning rates $\eta$. }
%The \signin gradient flow recovers the sign of $a$ in the multi-dimensional case, while the gradient flow fails even for larger learning rate.}
\label{tab:multi input}
\begin{tabular}{lcc}
\toprule
Method & GF & \signin GF
\\
\midrule
$\eta = 0.001$ & $0 \%$ & $100\%$ \\
$\eta = 0.01$ & $0\%$ & $100\%$ \\
\bottomrule
\end{tabular}
% \end{table}
\end{wraptable}
A potential explanation is that a finite learning rate $\eta > 0$ induces overshooting, leading to sign flips.
However, gradient flow with the standard parameterization does not recover the correct sign of $a$ due to this overshooting.
Accordingly, the success of \signin cannot be attributed entirely to this factor.
The key factor is that the balance between $a$ and $||\mathbf{w}||_{L_2}$ is broken.
Intuitively, for $a_t$ to sign flip, it needs to move faster than $||\mathbf{w}_t||_{L_2}$ as in the one-dimensional case (see Figure \ref{fig:1d riemann vs normal}).
We observe that $|a_{in}|^2 = ||\mathbf{w}_{in}||^2_{L_2} \geq w_{k,in}^2 $ for all $k \in [d]$.
Therefore, the reparameterization induces a relative speed up for $a$ compared to $||\mathbf{w}||_{L_2}$, i.e.,
\begin{equation}
\label{eq:rescaling}
\sqrt{a^2_{in} + \beta} = \sqrt{||\mathbf{w}_{in}||^2_{L_2} + \beta} \geq \frac{1}{n} \sum_{k =1}^n \sqrt{w_{k,in}^2 + \beta}
\end{equation}
This indicates $a$ can potentially sign flip without different rescaling in-between layers.
\textbf{No replacement for overparameterization}
While it is remarkable that the combination of \signin and overparameterized training can solve our theoretical toy problem, ideally, we would like to facilitate training from scratch ($d=1$) without utilizing any dense training.
%The question remains whether this could be
\signin is not able to achieve this without dense training but does there exist another reparameterization that could?
Unfortunately, the answer is negative, as we establish next.
Consider the one-dimensional problem again, with a continuous differentiable reparameterization $g_1 \in C^1(M_1, \mathbb{R})$ and $g_2 \in C^1(M_2, \mathbb{R})$ where $M_1$ and $M_2$ are smooth manifolds. Then we reparametrize both $a$ and $w$ such that $a = g_1(\mathbf{a})$ and $w_1 = g_2(\mathbf{w})$.
%that induces a Riemannian gradient flow.
%In this case, the sign of $w$ is impossible to recover.
%We show that for these reparameterizations it is impossible to recover the sign of $w$.
%Consider a Riemannian gradient flow with a metric $g$ such that
%\begin{equation}
%\begin{cases}\label{theory : riemann gf general}
%d a_t = - g_a^{-1}(a_t, w_t) \partial_a L(a_t, w_t)dt,\qquad a_0 = a_{\text{in}}\\
% d w_t = -g_w^{-1}(a_t, w_t)\partial_w L(a_t, w_t)dt, \qquad w_0 = w_{\text{in}}.
%\end{cases}
%\end{equation}
%It follows from the Nash embedding theorem that there must exist a reparameterization of $(a,w)$ such that the training dynamics is given by Eq.~(\ref{theory : riemann gf general}), see Theorem 5.1 \citep{Li2022ImplicitBO}.
\begin{theorem}[No reparameterization replaces overparameterization]\label{thm : impossible}
If $w_{1, in} < 0$, then there exists no continuous parameterization $(g_1, g_2)$ for which the resulting gradient flow reaches the global optimum.
\end{theorem}
Proof.
The flow has to move through $w_1 = 0$.
We can assume there exists an $\mathbf{w}^* \in M_2$ such that $g(\mathbf{w}^*) = 0$, otherwise it is impossible to move through $w_1 = 0$.
This implies that $\partial_a L = \partial_{w_1} L = 0$.
Thus, no continuously differentiable reparameterization can recover the sign of $w_1$. $\square$
According to Theorem \ref{thm : impossible}, from an optimization perspective, we can not replace overameterization with a reparameterization to recover the second case in Figure \ref{fig:quadrant-flips}.
This impossibility further emphasizes the difficulty of optimizing sparse networks without a dense training phase.
%Therefore, different approaches have to be employed to gain further improvement beyond reparameterization.
%A potential way is to use different activation functions that are continuously differentiable.
%Nevertheless, this might come at the cost of other trainability issues.
\textbf{Multiple neurons}
Do our insights still hold if we increase the number of neurons?
%We also explore a synthetic case with multiple neurons.
To verify the broader validity of our claims, we consider a neural network $f_2 : \mathbb{R}^d \rightarrow \mathbb{R}$ given by $f_2(\mathbf{z} \mid \mathbf{a}, \mathbf{W}) := \sum_{i = 1}^k a_i \sigma \left( \mathbf{w}_i^T z\right)$.
The data is generated from a similarly structured teacher network $\Tilde{f}_2$ with $k =3$ neurons and input dimension $d = 2$.
The weights are initialized and trained as outlined in \cref{appendix : neuron picture}.
% Furthermore, $\tilde{a}_i$ are i.i.d. Rademacher random variables, i.e. $\mathbf{\tilde{a}} \sim Uni(\{-1,1\})$, and $\tilde{w}_{i,j} \sim N(0,1)$ i.i.d. for all $i,j \in [k,d]$ and normalized over the input dimension.
% We initialize the dense network with $k = 20$ and the so-called COB initialization based on the rich regime in \citep{chizat2020lazy}.
% All weights are initialized with $N(0, 1/d)$ and we ensure that $a_i = -a_{i+10}$ for $i \in [10]$.
% In the sparse case, we learn $k =3$ neurons.
A difficult sign initialization is chosen as $a_i > 0$ for all $i \in [3]$.
% The learning rate used is $\eta = 1$ for the sparse setup and $\eta = 2.15$ for the dense setup.
% The scaling for \signin is $\beta =2$.
Figure \ref{fig:sign-representation main} illustrates each neuron by a vector $|a_i| \mathbf{w}_i$.
The teacher neurons are vectors touching the unit circle and student neurons are denoted with markers.
%We illustrate the benefit the reparameterization (\signin) in case of bad alignment.
% Sparse initialization with good alignment and the dense case are also discussed in Appendix \ref{appendix : neuron picture}.
% Also in these cases, \signin usually resolves the failure cases of gradient flow.
\begin{figure}[ht]
\centering
\subfigure[Single neuron.]{\includegraphics[width=0.3\linewidth]{Figures/signin_gf.png}
\label{fig:1d riemann vs normal}
}
\subfigure[Gradient flow.]{
\includegraphics[width=0.25\linewidth]{Figures/ToyExamples/badsigns_final_main.pdf}
\label{fig:bad sparse main}
}
\subfigure[\signin.]{
\includegraphics[width=0.25\linewidth]{Figures/ToyExamples/badsigns_finalmw.pdf}
\label{fig:mw main}
}
\caption{\textbf{Student-teacher setup} (a) Training \textbf{one neuron} for a single input $d=1$, when $w_1 \leq 0$ both methods fail to reach the ground truth. If $w_1 > 0$, the \signin gradient flow succeeds, whereas gradient flow fails in this case when additionally $a < 0$.
(b) Representation of a two layer student-teacher neural network with \textbf{multiple neurons} inspired by \citet{chizat2020lazy} for sparse training from scratch fails with bad signs. (c) \signin enables learning the representation with bad signs.}
\label{fig:sign-representation main}
\end{figure}
%The gradient field $F$ of $\left(a, ||\mathbf{w}||_{L_2}\right)$ at initialization is given by:
%\begin{equation*}
%\begin{cases}
% F_1 := - \sqrt{a_0^2 + \beta^2} \partial_a \mathbf{L} \\
% F_2 :=-\frac{1}{||\mathbf{w}_0||_{L_2}} \left( \sqrt{\mathbf{w_0}^2 + \beta^2} \odot \mathbf{w}_0 \right) ^T\nabla_{\mathbf{w}_0} \mathbf{L}
%\end{cases}
%\end{equation*}
%In order to break the balance we need to have $F_1 < -F_2$, given that $a < 0$.
%Moreover, from the balanced gradient flow %we have that
%\begin{equation*}
% \partial_a \mathbf{L} = -\frac{1}{||\mathbf{w}_0||_{L_2}} \mathbf{w}_0^T\nabla_{\mathbf{w}_0} \mathbf{L} \text{ and } a_0 = -||\mathbf{w}_0||_{L_2}.
%\end{equation*}
%If we have alignment between the initialization $\mathbf{w}_0$ and $\mathbf{w}^*$ we have that $\mathbf{w_0}^T\nabla_{\mathbf{w}_0} \mathbf{L} < 0 $.
% \begin{table}[ht!]
% \centering
% \caption{Percentage of sign recoveries of $a>0$ for both the gradient and Riemannian gradient flow. The reparameterized gradient flow recovers the sign of $a$ for a significant amount as well in the multi-dimensional case, while the gradient flow fails even for larger learning rate.}
% \label{tab:multi input}
% \begin{tabular}{lcc}
% \toprule
% Method & Gf & Reparameterization gf
% \\
% \midrule
% $\eta = 0.001$ & $0 \%$ & $18\%$ \\
% $\eta = 0.1$ & $0\%$ & $84 \%$ \\
% \bottomrule
% \end{tabular}
% \end{table}
%\section{Dynamic Rescaling To Induce Sign Flips}
%\textcolor{blue}{
%In \citep{gadhikar2024masks} it is shown both theoretically and experimentally that a sparse network has difficulty with sign flipping.
%This results in deprecating performance.
%We show that the paramterization $m \odot w$ as proposed by \citep{vaškevičius2019implicitregularizationoptimalsparse} and used in \citep{ziyin2023spredsolvingl1penalty,jacobs2024maskmirrorimplicitsparsification, pesme2021implicit, Even2023SGDOD, woodworth2020kernel} can be used to induce sign flips.
%This can be done by using the degree of freedom we get from the reparameterization.
%Therefore, we can exploit this to induce a rescaling.
%The rescaling allows us to control the relative size of the gradient.
%}
%\textcolor{blue}{
%The initialization we consider is the constant scaling initialization.
%The initialization is given by:
%\begin{equation}\label{theory : init}
% m \odot w = x \qquad \text{ and } \qquad m^2 - w^2 = \alpha
%\end{equation}
%We show that $\alpha$ characterizes the ability to sign flip.
%The characterization is shown in a two layer neural network.
%}
%\textcolor{blue}{
%Let $f : \mathbb{R}^d \times \mathbb{R} \rightarrow \mathbb{R}$ be a two layer neural network with data $Z$ and labels $y$:
%\begin{equation*}
% f(w, a) : = a \sigma(x Z),
%\end{equation*}
%where $\sigma : \mathbb{R} \rightarrow \mathbb{R}_+$ is the Rectified Linear Unit (ReLU).
%The network is trained with the MSE loss function,
%\begin{equation*}
% L\left(x, a\right) := \frac{1}{2n} \sum_{i = 1}^n \left(a \sigma\left(x^T z_i\right) - y_i\right)^2.
%\end{equation*}
%We now re-parameterize both $a$ and $x$ with $m \odot w$. This results in a Riemannian gradient flow for $x$ and $a$.
%\begin{equation}
% \begin{cases}\label{theory : riemannian gf}
% d x_t = - \sqrt{x_t^2 + \alpha^2}\odot \nabla_x L(x_t, a_t)\qquad x_0 = x_{\text{in}}\\
% d a_t = - \sqrt{a_t^2 + \alpha^2}\odot \nabla_a L(x_t, a_t) \qquad a_0 = a_{\text{in}}.
% \end{cases}
%\end{equation}
%}
%Furthermore, $m_t$ and $w_t$ follow the gradient flow:
%\begin{equation*}
% \begin{cases}
% d m_t = - a_t\frac{1}{n}\sum_{i = 1}^n w_t \odot z_i\mathbb{I}_{x_t^T z_i \geq 0} \left(a_t \sigma\left(x_t^T z_i\right) - y_i\right) \qquad m_0 = m_{\text{in}} \\
% d w_t = - a_t\frac{1}{n}\sum_{i = 1}^n m_t\odot z_i\mathbb{I}_{x_t^T z_i \geq 0} \left(a_t \sigma\left(x_t^T z_i\right) - y_i\right) \qquad w_0 = w_{\text{in}}
% \end{cases}
%\end{equation*}
%\textcolor{blue}{
%The result is stated informally in Theorem \ref{Thm : sign flip a informal}.
%\begin{theorem}\label{Thm : sign flip a informal}
% Consider the dynamics in Eq.~(\ref{theory : riemannian gf}) with balanced initialization $|a| = ||x||$. Then the sign of $a$ is recovered.
%\end{theorem}
%Proof sketch.
%The proof uses tools from dynamical systems.
%We show that the gradient field of $(a, ||x||)$ stays in a particular quadrant and can only reach the zero loss in this way.
%The alteration makes it possible for $a$ to move "faster" than $||x||$ leading to a sign flip.
%}
%\textcolor{blue}{
%This result shows a non-trivial benefit of overparameterization.
%Furthermore, the characterization of $\alpha$ leads to a meaningful way of choosing $\alpha$ in practice.
%It might be of independent interest that the occurring sign flip happens when we are between the $L2$ or $L1$ regimes i.e. the kernel and rich regimes. This highlights that perforance gains also can be due to the transition between the kernel and rich regimes.
%}
%\textcolor{blue}{
%The implication of Theorem \ref{Thm : sign flip a informal} is that IMP can recover more signs in later cycles and thus can catch up to LRR.
%By definition of IMP the parameters are reset to their original value thus resetting $\alpha$, allowing for sign flips under an appropriate choice of $\alpha$.
%In contrast, LRR only enjoys this benefit in the first cycle as the scaling $\alpha$ decays in each cycle due to stochastics and regularization.
%This makes it more difficult to change signs later on, nevertheless it is not necessary for LRR as it has found already the "correct" signs.
%To summarize, IMP with the parameterization $m \odot w$ can recover more signs due to re initializing $\alpha$.
%Thus giving a way for IMP to improve performance at higher sparsity levels.
\section{Experiments}
We show that \signin provably improves PaI in a small scale example (\cref{theorem : sign flip main}), now we have to verify that its merits also translate to larger scale settings.
While our main focus is on PaI and thus training sparse networks from scratch, our theory implies that \signin could also boost dense-to-sparse training performance, as their combination can resolve cases where either would fail.
However, we do not expect training from scratch with \signin to outperform training starting from well aligned signs, as \signin acts orthogonally to dense-to-sparse training in our theoretical setting.
\signin is expected to facilitate different sign flips.
%of them orthogonal to
%discuss interpretation with respect to dense training
%Our experiments verify that the improved sign flipping with \signin as explained by \cref{theorem : sign flip main} translates to largescale experiments and can significantly improve the trainability of sparse networks.
%We also show that \signin works orthogonal to dense training by combining it with dense-to-sparse training methods.
\textbf{Experimental Setup}
\label{expt-setup-para}
We perform experiments on standard vision benchmarks including CIFAR10, CIFAR100 \citep{cifar}, and ImageNet \citep{imagenet} training a ResNet20, ResNet18, and ResNet50 model, respectively.
For architectural variation, we also train a vision transformer DeiT Small \citep{touvron2021training} on ImageNet.
All training details are provided in \cref{tab:training-details} in Appendix \ref{appendix : training details}.
\begin{table*}[ht!]
\centering
\caption{Sparse training with \signin.}
\label{tab:sign-recovery}
\begin{tabular}{lcccc}
\toprule
Dataset & Method & $s = 0.8$ & $s = 0.9$ & $s = 0.95$ \\
\midrule
\multirow{2}{*}{CIFAR10 \textsubscript{ + ResNet20}} & Random & $88.25 (\pm 0.35)$ & $86.25(\pm 0.3)$ & $83.56 (\pm 0.23)$ \\
& Random + \signin & $\mathbf{89.37 (\pm 0.14)}$ & $\mathbf{87.83 (\pm 0.11)}$ & $\mathbf{84.74 (\pm 0.06)}$ \\
\midrule
\multirow{2}{*}{CIFAR100\textsubscript{ + ResNet18}} & Random & $73.95(\pm{0.1})$ & $72.96(\pm{0.24})$ & $71.36(\pm{0.16})$ \\
& Random + \signin & $\mathbf{75.32 (\pm 0.25)}$ & $\mathbf{73.94 (\pm 0.13)}$ & $\mathbf{72.51 (\pm 0.14)}$ \\
\midrule
\multirow{2}{*}{ImageNet\textsubscript{ + ResNet50}} & Random & $73.87(\pm 0.06)$ & $71.56(\pm 0.03)$ & $68.72(\pm 0.05)$ \\
& Random + \signin & $\mathbf{74.12 (\pm 0.09)}$ & $\mathbf{72.19 (\pm 0.18)}$ & $\mathbf{69.38 (\pm 0.1)}$ \\
% 0.1040216644 0.08801833142 0.08001666493
% 74.145 72.285 69.44
\bottomrule
\end{tabular}
\end{table*}
\textbf{Scaling reset}
An integral part of \signin is that we dynamically reset the internal scaling to combat the implicit bias towards sparsity of the parameterization.
%The reparameterization $m \odot w$ introduces a degree of freedom which can better enable sign flips.
%However, these parameters still need to be rescaled such that one changes faster than the other in order to make it flip its sign, as illustrated in \cref{eq:rescaling}.
To ablate its contribution, we report results for \signin with and without resetting the scaling for a random mask in \cref{tab:rescaling-appendix} in the appendix.
Indeed, without the scaling reset, the performance gain is less significant.
This confirms that resetting mitigates the effect of stochastic noise and weight decay, enabling better sign flips.
%the resetting significantly improves validation accuracy.
%This shows that the resetting is of importance to improve performance.
\textbf{\signin improves sparse training}
Table \ref{tab:sign-recovery} reports results for sparse training with a random mask.
The layerwise sparsity ratios are chosen as balanced such that each layer has the same number of nonzero parameters \citep{er-paper}.
Sparse training with \signin as per \cref{app: alg sign in} consistently outperforms standard training, substantiating our theoretical results.
A similar improvement is observed for other PaI methods in \cref{tab:sign-recovery-appendix} in Appendix \ref{appendix ablation}.
We apply \signin also to different sparse masks identified with PaI or dense-to-sparse methods as reported in \cref{tab:mw-differentt-masks} in \cref{app:difft-masks}.
While layerwise sparsity ratios determine the overall trainability of a sparse network, \signin improves performance in better trainable settings.
%when layers have similar sparsities.
%Interestingly, the
\textbf{Vision transformer with \signin}
\signin is also able to improve sparse training of a randomly masked vision transformer.
A DeiT Small architecture (trained for $300$ epochs from scratch) gains more than $1\%$ in performance with \signin, as reported in \cref{tab:deit-results}, \cref{app:vit-signin}.
% \begin{table}[ht!]
% \centering
% \caption{\textbf{DeiT Small} on ImageNet, trained from scratch.}
% \label{tab:deit-results}
% \begin{tabular}{lcc}
% \toprule
% Method & $s=0.5$
% \\
% \midrule
% Random & $69.31 (\pm 0.7)$ \\
% Random + \signin (ours) & $\mathbf{70.62 (\pm 0.4)}$ \\
% \bottomrule
% \end{tabular}
% \end{table}
% In case of sparse training with PaI or dense-to-sparse masks as reported in \cref{tab:mw-differentt-masks} in the appendix, we find that layerwise sparsity ratios influence trainability of sparse networks, and extremely sparse layers can be detrimental to performance, however this is not a problem in dense-to-sparse training.
% Empirically we find balanced \citep{er-paper} sparsity ratios to be the best performing as no layer is too sparse.
\textbf{Improved sign alignment}
Figure \ref{fig:mw-flips} validates the \signin mechanism by tracking the percentage of sign flips during training for a $90\%$ sparse ResNet50 on ImageNet.
Across the full duration of training, \signin enables strictly more sign flips than sparse training.
This improves sign alignment, as reflected in the improved performance and the flatter minima (see \cref{fig:mw-eigs}).
Since \signin employs an alternate mechanism for sign flipping, it cannot learn stable signs early like dense-to-sparse methods.
Instead, it enables more sign flips throughout training as compared to standard sparse training, to aid alignment by better exploration of the parameter space.
% \begin{wrapfigure}{l}{0.4\textwidth}
% \centering
% \vspace{-1.5em} % Adjust vertical space above
% \begin{subfigure}[b]
% \includegraphics[width=\linewidth]{Figures/mw_flip_compare.png}
% \caption{Sign Flips}
% \label{fig:mw-flips}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]
% \includegraphics[width=\linewidth]{Figures/eigs_mw.png}
% \caption{Sharpness}
% \label{fig:mw-eigs}
% \end{subfigure}
% \caption{\textbf{Sign flips with \signin.} Training a random, $90\%$ sparse ResNet50 on ImageNet with \signin induces (a) more sign flips during training and finds (b) a flatter minimum.}
% \label{fig:sign-alignment-mw}
% \vspace{-1em} % Adjust vertical space below
% \end{wrapfigure}
\textbf{\signin improves other sparse training methods}
%\begin{itemize}
% \item flipping w is more important than a, hence better performance with dense-to-sparse than with \signin, connect this to \cref{fig:quadrant-flips} that not all quadrants are equal.
% \item Although \signin improves sparse training, it cannot close the gap bw dense to sparse and sparse training because sign flipping is orthogonal to dense-to-sparse highlighting that all quandrants are not equal.
%\end{itemize}
%\signin
%works orthogonally to dense training for sign alignment.
Only a combination of \signin and overparameterization is able to solve all sign flip cases in our theoretical setting, as also confirmed by numerical experiments in \cref{tab:toy-expts}, \cref{app:toy-expts}.
Therefore, we expect the combination of dense-to-sparse or dynamic sparse training methods with \signin to achieve further gains in performance.
%Accordingly We parameterize all the weights of the network with $m \odot w$ and perform AC/DC.
%This combination can improve the AC/DC baseline as reported in
\cref{app:acdc-signin} shows that both AC/DC and RiGL are improved with \signin.
\begin{wrapfigure}{r}{0.55\textwidth}
% \begin{figure}[ht!]
\centering
% Second subfigure
\subfigure[Sign Flips]{
%\includegraphics[width=0.19\textwidth]{Figures/mw_flip_compare.png}
\includegraphics[width=0.21\textwidth]{Figures/mw_flip_compare.png}
%\includegraphics[width=0.311\textwidth]{Figures/mw_flip_compare.png}
\label{fig:mw-flips}
}
\subfigure[Sharpness]{
%\includegraphics[width=0.21\textwidth]{Figures/eigs_mw.png}
\includegraphics[width=0.24\textwidth]{Figures/eigs_mw.png}
%\includegraphics[width=0.34\textwidth]{Figures/eigs_mw.png}
\label{fig:mw-eigs}
}
\caption{\textbf{Sign flips with \signin} Training a random, $90\%$ sparse ResNet50 on ImageNet with \signin induces (a) more sign flips during training and finds (b) a flatter minimum.}
\label{fig:sign-alignment-mw}
% \end{figure}
\end{wrapfigure}
\textbf{Implications of theory}
%Therefore, highlighting that the quadrants in Figure \ref{fig:quadrant-flips} are not of equal importance.
%Therefore, highlighting that the quadrants in Figure \ref{fig:quadrant-flips} are not of equal importance.
The surprising success of \signin combined with overparameterization in our theoretical setting relies on the last layer $a$ learning at a higher speed than the first layer $w_1$.
This enables $a$ to flip its sign if necessary, which is impossible in the same setting without \signin. $w_1$ can follow once the sign of $a$ is correctly learned.
We conjecture that weights in layers close to the input would have a harder time flipping in deep neural networks consisting of more layers and neurons, as other layers on top slow its learning down.
In line with this narrative, \signin promotes more sign flips in layers closer to the output, as shown in \cref{fig:acdc-mw-flips} in the appendix.
%We conjecture that $a$ would have a harder time flipping in deep neural networks consisting of more layers and neurons, as other layers on top slow its learning down.
%For that reason, the case that $w_1$ flips is likely the more dominant case, as also suggested by \cref{fig:acdc-mw-flips} in the appendix, which reports layerwise sign flip statistics and confirms that \signin increases more the number of sign flips in layers close to the output.
To understand how these sign flips relate to generalization, we must analyze more complex scenarios in future and see whether more effective sign learning could further boost dense-to-sparse training.
The primary focus of this work has been to improve sparse training from scratch and \signin overcomes one of its major limitations regarding learning parameter signs.
%has outlined a significant step to close the gap to dense-to-sparse training.
%provides a significant step in the right direction.
%to understand how to further facilitate sign learning, nevertheless, \signin provides a significant step in the right direction.
%\signin provides a significant step in the right direction.
%improves the overall performance and sign flips in comparison.
%We conjecture that that the quadrants in Figure \ref{fig:quadrant-flips}
%Flipping $w$ correctly, which seems to carry higher importance, is thus an important open challenge, to further close the gap between PaI and dense-to-spare methods.
% \begin{table}[ht!]
% \centering
% \caption{\textbf{\signin improves AC/DC} for a ResNet50 trained on ImageNet.}
% \label{tab:acdc-mw}
% \setlength\tabcolsep{2.2pt}
% \begin{tabular}{lccc}
% \toprule
% Method & $s=0.8$ & $s=0.9$ & $s=0.95$
% \\
% \midrule
% AC/DC & $75.83(\pm0.02)$ & $74.75(\pm0.02)$ & $72.59(\pm0.11)$\\
% \midrule
% AC/DC & \multirow{2}{*}{$\mathbf{75.9(\pm0.14})$} &\multirow{2}{*}{$74.74(\pm0.12)$}& \multirow{2}{*}{$\mathbf{72.88(\pm0.13)}$}\\
% + \signin & & &\\
% \bottomrule
% % 0.02355626833 0.02278251181 0.1133490969
% % 75.82333333 74.75866667 72.59333333
% \end{tabular}
% \end{table}
% \paragraph{Reset the scaling}
% We verify that the resetting of the scaling is important to enhance PaI.
% %The reparameterization $m \odot w$ introduces a degree of freedom which can better enable sign flips.
% %However, these parameters still need to be rescaled such that one changes faster than the other in order to make it flip its sign, as illustrated in \cref{eq:rescaling}.
% For a random mask we report the results for \signin with and without reset in \cref{tab:rescaling-appendix} in the appendix.
% Indeed, without resetting the scaling, the performance gain is less significant.
% Thus confirming that resetting is meaningful as it can mitigate the effects of stochastic noise and weight decay and enable sign flips.
% %the resetting significantly improves validation accuracy.
% %This shows that the resetting is of importance to improve performance.
% \paragraph{Vision transformer with \signin}
% \signin is also able to improve the sparse training of a random mask for a vision transformer.
% We train a DeiT Small architecture from scratch for $300$ epochs and find that \signin improves sparse training by more than $1\%$ as reported in \cref{tab:deit-results}.
% \begin{table}[ht!]
% \centering
% \caption{DeiT Small on ImageNet, trained from scratch.}
% \label{tab:deit-results}
% \begin{tabular}{lcc}
% \toprule
% Method & $s=0.5$
% \\
% \midrule
% Random & $69.31 (\pm 0.7)$ \\
% Random + \signin (ours) & $\mathbf{70.62 (\pm 0.4)}$ \\
% \bottomrule
% \end{tabular}
% \end{table}
%\section{Discussion}
\section{Conclusion}
Parameter signs that are aligned with a given mask are sufficient to train a sparse network from scratch.
We have empirically verified this conjecture for masks that result from dense-to-sparse training algorithms.
% This is enabled by access to well aligned signs, which are learned early during dense-to-sparse training.
Regardless of the sparse mask,
% being task specific or not
learning highly performant parameter signs is a great challenge for PaI.
%We have therefore attributed the limitation of training from scratch
%Expanding on this insight, we improve sparse training by employing the reparameterization $m \odot w$ to learn signs more effectively.
%, which improves sign alignment.
%The improvement holds across multiple setups with sparse initialization.
%This is in contrast to continuous sparsifcation methods.
%They used the reparameterization for sparsifcation.
To aid parameter sign flipping, we have proposed to reparameterize a neural network by \signin.
% and have
% demonstrated its effectiveness in a diverse range of experiments as well as a theoretically tractable setting.
%Furthermore, the improvement works orthogonal to overparameterization.
For a single neuron network, we have proven with theory from Riemannian gradient flows and dynamical systems that \signin %can resolve an initialization scenario, which is unsolvable for overparameterization.
resolves a case that is complementary to overparameterized training which translates to largescale performance.
% Their combination, overparameterization with \signin, even recovers the ground truth in all possible scenarios.
%Moreover, when \signin is combined with more overparameterization we observe empirically that we can recover all cases.
% This is supported by enhancing AC/DC with \signin.
Ideally, we would like to make dense training phases obsolete to save computational resources.
Although, as we have shown, a reparameterization alone will not be able to achieve this.
Closing the gap between training sparse, randomly initialized networks from scratch and dense-to-sparse training remains a hard open problem.
Our work brings us closer to this goal with two key insights: a) sign alignment is enough for training a sparse initialized network and b) reparameterization can improve sign alignment.
\section{Proof of Theorem \ref{theorem : sign flip main}}
In this section we show the main theoretical result of the paper. We drop the dependence on the index for $w_1$ as we are only concerned with the one dimensional case here.
\begin{theorem}\label{theorem : sign flip appendix}
Let $\Tilde{f}$ be the teacher and $f$ be the student network such that $a$ and $w$ follow the gradient flow dynamics in Eq.~(\ref{theory : riemann gf}) with a random balanced parameter initialization. Moreover, let $\beta_1 > \beta_2>0$. If $w_{in} > 0$, then $f$ can learn the correct target with probability $1 - \left(\frac{1}{2}\right)^n$.
In the other case ($w_{in} \leq 0$) learning fails.
\end{theorem}
Proof.
The proof idea is to show that for a balanced initialization with $w_{in} > 0$ the flow for $t> 0$ always enters the open set
\begin{equation*}
\Gamma_{0}:= \{ (a,w) \in \mathbb{R}^2 : a < -w, w > 0 \}.
\end{equation*}
Furthermore, we show that the flow stays in the open set $\Gamma_{0}$.
The system is a Riemannian gradient flow which implies that the flow converges towards a stationary point in $\Gamma_0$.
It remains to be shown that the stationary point at the origin is a saddle and the stable manifold of the origin is not in $\Gamma_0$.
Thus, the remaining stationary points are the global optimizers.
First we show that for balanced initializations $|a_{in}| = w_{in} > 0$ enter the region $\Gamma_{0}$.
In case $a_{in} = w_{in} > 0$, we have $(a_{in}, w_{in}) \in \Gamma_{0}$.
In case $-a_{in} = w_{in} > 0 $, we have that $(a_{in},w_{in}) \in \Bar{\Gamma}_{0} \setminus \Gamma_{0}$ i.e. the boundary of $\Gamma_{0}$.
Therefore, we need to show the gradient field at $(a_{in}, w_{in})$ points into $\Gamma_{0}$.
The balanced initialization implies that
\begin{equation*}
\partial_a L(a_{in}, w_{in}) = - \partial_w L(a_{in}, w_{in}).
\end{equation*}
Moreover, since $\Tilde{a} > 0$, $\partial_a L(a_{in}, w_{in}) > 0$.
Using that $\beta_1 > \beta_2 > 0$ we have that the gradient field satisfies:
\begin{align*}
d a_{in} &= -\sqrt{a_{in}^2 + \beta_1} \partial_a L(a_{in}, w_{in}) dt \\ &= \sqrt{w_{in}^2 + \beta_1} \partial_w L(a_{in}, w_{in}) dt\\ &< \sqrt{w_{in}^2 + \beta_2} \partial_w L(a_{in}, w_{in}) dt = -dw_{in}
\end{align*}
Therefore there is a $t_0 > 0$ such that $a_{t_0} < -w_{t_0} < 0$. Thus there is a $t_0 > 0$ such that $(a_{t_0}, w_{t_0}) \in \Gamma_{0}$.
We have entered the set $\Gamma_{0}$, we have to show that we cannot leave the set $\Gamma_0$.
This can be shown by computing the gradient field at the boundaries.
The boundary can be split up into three cases:
\begin{itemize}
\item $B_1 := \{(a,w) \in \mathbb{R}^2 : -a = w > 0\}$
\item $B_2 := \{(a,w) \in \mathbb{R}^2 : w = 0, a > 0 \}$
\item The origin $\{(0,0)\}$
\end{itemize}
The first case of $B_1$ is covered by the balanced initialization.
For the second case of $B_2$ we can compute the gradient field again. We now only need that $d w_t > 0$. We linearize $d w_t$:
\begin{equation*}
dw_t = C\sqrt{ \beta_2}a_t dt > 0,
\end{equation*}
where $C = \frac{1}{n}\sum_{i = 1}^n \max \{0,z_i\}^2> 0$ with probability $1- \left(\frac{1}{2}\right)^n$.
The last case is the saddle point at the origin which we show is not possible to be reached from the open set $\Gamma_0$.
Thus for all $(a_{in}, w_{in}) \in \Gamma_{0}$ we have that for all $t \geq 0$, $(a_t, w_t) \in \Gamma_0$ or $\lim_{t \rightarrow \infty} (a_t, w_t) = (0,0)$.
In the case that $w > 0$ the flow can be written as a dynamical system on a Riemannian manifold.
This allows us to guarantee convergence to a stationary point.
The flow is given by
\begin{equation*}
\begin{cases}
d a_t = -C\sqrt{a_t^2 + \beta_1} \left( a_tw_t^2 -w_t\right)dt \qquad a_0 = a_{\text{in}} \\
d w_t = -C\sqrt{w_t^2 + \beta_2} \left(a_t^2 w_t -a_t\right) dt \qquad w_0 = w_{\text{in}},
\end{cases}
\end{equation*}
where $C = \frac{1}{n}\sum_{i = 1}^n \max \{0,z_i\}^2> 0$ with probability $1- \left(\frac{1}{2}\right)^n$.
This dynamical system has stationary points at the origin and the set $aw = 1$.
The dynamical system is a Riemannian gradient flow system therefore the flow converges to a stationary point.
The stationary point at the origin is a saddle point.
Therefore, the only way of getting stuck at the origin is when we initialize on the associated stable manifold.
We show that this not possible for the balanced initialization.
We calculate the linearization of the stable manifold and use that the balanced initialization stays in $\Gamma_0$.
The linearization at the origin $(0,0)$ is given by
\begin{equation*}
\begin{cases}
d a_t = C\sqrt{ \beta_1} w_t dt \\
d w_t = C\sqrt{ \beta_2}a_t dt.
\end{cases}
\end{equation*}
By a direct calculation of the eigen vectors the linearization of the stable manifold is given by the vector $\left( -\sqrt{\frac{|\beta_1|}{|\beta_2|}}, 1\right)$.
It follows from $\beta_1 > \beta_2 >0$ that the linearization of the stable manifold is outside of $\Gamma_{0}$.
Suppose that from $\Gamma_{0}$ the stable manifold is reachable.
Then there is a continuous differentiable curve $\gamma_t$ with initialization $\gamma_0 = (a_{in}, w_{in}) \in \Gamma_{0}$ such that $\lim_{t \rightarrow \infty} \gamma_t = (0,0)$.
This is not possible as it violates the gradient field at the boundaries of $\Gamma_0$.
Thus, the flow does not converge to the stationary point at the origin.
This concludes the first part, since the only set of stationary points are the set of global optima.
The other two remaining cases fail as the boundary at $w = 0$ is not differentiable and the gradient flow stops there. $\square$
\newpage
\section{\signin algorithm}
\label{app: alg}
We provide pseudocode for \signin for in \cref{sec: alg sign in} with rescaling as explained in \cref{sec: alg sign in}.
\begin{algorithm}[ht!]
\caption{Sign-In}
\label{app: alg sign in}
\begin{algorithmic}
\STATE {\bfseries Input:} objective $L$, scaling $\beta$, frequency $p$, epochs $T$, mask $S$, stop rescaling epoch $T_2$
\STATE Initialize $L(S \odot m_0 \odot w_0)$ such that $m_0 \odot w_0 = \theta_0$ and $m_0^2 - w_0^2 = \beta \mathbf{I}$.
\FOR{$i=1$ {\bfseries to} $T$}
\IF{$i$ mod $p == 0$ and $i < T_2$}
\STATE Rescale $m_i \odot w_i = \theta_i$ such that $m_i^2 - w_i^2 = \beta \mathbf{I}$
\ENDIF
\STATE $m_{i+1}, w_{i+1} = \text{Optimizer}\left(L\left(S \odot m_i \odot w_i\right)\right)$
\ENDFOR
\STATE {\bfseries Return} Model parameters $\theta_T = S\odot m_T \odot w_T$
\end{algorithmic}
\end{algorithm}
\section{Algorithm Initialization}\label{appendix : init}
We derive here an exact analytic formula for the initialization of the reparameterization.
Since the reparameterization is pointwise we can restrict ourself to the one dimensional case.
For this, we need to solve the system of equations for arbitrary $x \in \mathbb{R}$ and $\beta > 0$:
\begin{equation*}
\begin{cases}
m_0 w_0 = x \\
m_0^2 - w_0^2 = \beta
\end{cases}
\end{equation*}
We find that a solution is given by:
\begin{equation*}
m_0 = u \text{ and } w_0 = \frac{x}{u}
\end{equation*}
where $u = \sqrt{\alpha + \sqrt{x^2 + \alpha^2}}$, with $\alpha = \frac{\beta}{2}$.
Note that the solution can also be written as
\begin{equation*}
m_0 = \frac{v + \frac{\alpha}{v}}{\sqrt{2}} \text{ and } w_0 = \frac{v - \frac{\alpha}{v}}{\sqrt{2}}
\end{equation*}
where $v = \sqrt{x + \sqrt{x^2 + \alpha^2}}$ with $\alpha = \frac{\beta}{2}$.
\newpage
\section{Multiple Neurons Visualization}\label{appendix : neuron picture}
We also explore the multi neuron case.
The neural network $f_2 : \mathbb{R}^k \times \mathbb{R}^{kd} \times \mathbb{R}^d \rightarrow \mathbb{R}$ is given by $f_2(\mathbf{a}, \mathbf{W}, \mathbf{z}) := \sum_{i = 1}^k a_i \sigma \left( \mathbf{w}_i^T z\right)$.
The data is generated from a similar structured teacher network $\Tilde{f}_2$ with $k =3$ neurons and input dimension $d = 2$.
Furthermore, $\mathbf{\tilde{a}} \sim Uni(\{-1,1\})$ i.e. i.i.d. Rademacher random variables and $\mathbf{\tilde{w}}_{i,j} \sim N(0,1)$ for all $i,j \in [k,d]$ and normalized over the input dimension.
We initialize the dense network with $k = 20$ and the so-called COB initialization based on the rich regime in \citep{chizat2020lazy}.
All weights are initialized with $\mathcal{N}(0, 1/d)$ and we ensure that $a_i = -a_{i+10}$ for $i \in [10]$.
In the sparse cases we initialize $k =3$ neurons.
In the good signs case we initialize such that the signs of the weights align with the teacher network.
In contrast for the bad signs we choose $a_i > 0$ for all $i \in [3]$.
The learning rate used is $\eta = 1$ for the sparse setup and $\eta = 2.15$ for the dense setup.
Moreover, the scaling for the reparameterization is $\beta =2$.
This results in Figure \ref{fig:sign-representation}, where we represent each neuron as $|a_i| \mathbf{w}_i$.
The teacher neurons are vectors touching the unit circle and student neurons are denoted with markers.
We illustrate both the benefit of correct sign alignment (Fig.~\ref{fig:good sparse}) and the reparameterization in case of bad alignment (Fig.~\ref{fig:mw}).
This is contrasted with dense training (Fig.~\ref{fig:dense}) and bad alignment (Fig.~\ref{fig:bad sparse}).
\begin{figure}[ht]
\centering
%Second subfigure
\subfigure[Dense.]{
\includegraphics[width=0.23\textwidth]{Figures/ToyExamples/dense_final.pdf}
\label{fig:dense}
}
\subfigure[Good signs.]{
\includegraphics[width=0.23\textwidth]{Figures/ToyExamples/goodsigns_init.pdf}
\label{fig:good sparse}
}
\subfigure[Bad signs.]{
\includegraphics[width=0.23\linewidth]{Figures/ToyExamples/badsigns_final.pdf}
\label{fig:bad sparse}
}
\subfigure[\signin.]{
\includegraphics[width=0.23\linewidth]{Figures/ToyExamples/badsigns_finalmw.pdf}
\label{fig:mw}
}
\caption{Representation of a two layer student-teacher neural network inspired by \citet{chizat2020lazy}. Signs are crucial in learning sparse representations. A dense neural network can easily learn the representation in Fig.~\ref{fig:dense}. Sparse networks can learn when signs are correct Fig.~\ref{fig:good sparse} and fail with bad signs Fig.~\ref{fig:bad sparse}. The reparameterization enables learning the representation with bad signs in Fig.~\ref{fig:mw}.}
\label{fig:sign-representation}
\end{figure}
%The second part of Theorem \ref{theorem : sign flip appendix}) can be generalized to an arbitrary Riemannian gradient flow.
%A Riemannian gradient flow is given by a metric $g$ such that
%\begin{equation}
% \begin{cases}\label{theory : riemann gf appendix}
% d a_t = - g_a^{-1}(a_t, w_t) \partial_a L(a_t, w_t)dt,\qquad a_0 = a_{\text{init}}\\
% d w_t = -g_w^{-1}(a_t, w_t)\partial_w L(a_t, w_t)dt, \qquad w_0 = w_{\text{init}}.
% \end{cases}
%\end{equation}
%\begin{theorem}\label{non existence rgf : appendix}
% If $w_0 < 0$ then there exists no metric $g$ such that the Riemannian gradient flow in Eq.~(\ref{theory : riemann gf appendix}) can converge toward the global optimum.
%\end{theorem}
%Proof.
%The Riemannian gradient flow in Eq.~(\ref{theory : riemann gf appendix} has to move trough $w_t = 0$ to reach the ground truth. This implies that $\partial_a L(a_t, w_t) = \partial_w L(a_t, w_t) = 0$. Therefore, a Riemannian gradient flow can not reach the ground truth. $\square$
%\section{Proof of Multidimensional case}
%We have seen in the one-dimensional case that $w_0 >0$ is necessary to recover the sign of $a$.
%This means that $w_0$ already aligns with $w^*$
%We generalize this alignment property to the multidimensional case, where $\mathbf{w}_0 \in \mathbb{R}^d$ aligns with the ground truth $\textbf{w}^* \in \mathbb{R}^d$ for $d \in \mathbb{N}^+$.
%\begin{definition}
% The initialization of $\textbf{w}_0$ is said to align with $\textbf{w}^*$ with respect to $\{ \mathbf{z}_i\}_{i = 1}^n$ if and only if $\text{sign}\left(\textbf{w}_{0}^T \textbf{z}_i\right) = \text{sign}\left(\left(\textbf{w}^*\right)^T \textbf{z}_i\right)$ for all $i \in [n]$.
%\end{definition}
%\begin{equation*}
% \begin{cases}
% d a_t = -\sqrt{a_t^2 + \beta} \sum_{i\in J}\left(\textbf{w}_t^T \textbf{z}_i\right)\left( a_t\textbf{w}_t^T \textbf{z}_i -a^* (\textbf{w}^*)^T \textbf{z}_i \right)dt \qquad a_0 = a_{\text{init}} \\
% d \textbf{w}_t = -\sqrt{\textbf{w}_t^2 + \beta} \odot \sum_{i\in J} a_t \textbf{z}_i\left(a_t \textbf{w}_t^T \textbf{z}_i -a^* (\textbf{w}^*)^T \textbf{z}_i\right) dt \qquad \textbf{w}_0 = \textbf{w}_{\text{init}},
% \end{cases}
%\end{equation*}
\newpage
\section{Training Details}\label{appendix : training details}
In this section we provide the details of the additional hyperparameters used for experiments.
Table \ref{tab:training-details} we provide the standard hyperparameters such as learning rate and weight decay.
We note that the loss used for both CIFAR10 and CIFAR100 is the cross-entropy loss and for ImageNet the label smoothing loss with confidence $0.9$.
For the $m \odot w$ parametrization we also apply Frobenius decay i.e. $(m \odot w)^2$ with the same strength as weight decay for CIFAR10 and CIFAR100.
We provide these extra details in Table \ref{tab:training-details sign in}. The code used is based on TurboPrune as in \citep{Nelaturu_TurboPrune_High-Speed_Distributed}.
\begin{table}[ht!]
\centering
\caption{Training Details for all experiments presented in the paper.}
\label{tab:training-details}
\begin{tabular}{lccccccc}
\toprule
\textbf{Dataset} & \textbf{Model} & \textbf{LR} & \textbf{WD} & \textbf{Epochs} & \textbf{Batch Size} & \textbf{Optim} & \textbf{Schedule} \\
\midrule
CIFAR10 & ResNet20 & $0.2$ & $1e-4$ & $150$ & $512$ & SGD & Triangular\\
\midrule
CIFAR100 & ResNet18 & $0.2$ & $1e-4$ & $150$ & $512$ & SGD & Triangular\\
\midrule
\multirow{2}{*}{ImageNet} & ResNet50 & $0.25$ & $5e-5$ & $100$ & $1024$ & SGD& Triangular\\
& DeIT Small & $0.001$ & $1e-1$ & $300$ & $1024$ & AdamW & Triangular\\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[ht!]
\centering
\caption{Training Details for the additional \signin parameters. The weight decay in this table overides the weight decay in Table \ref{tab:training-details}.}
\label{tab:training-details sign in}
\begin{tabular}{lccccc}
\toprule
\textbf{Dataset} & \textbf{Model} & $\mathbf{\beta}$ & $T_2$ & \textbf{Frobenius Decay} & \textbf{Weight decay}\\
\midrule
CIFAR10 & ResNet20 & $1$ & $75$ & $1e-4$ & $0$\\
\midrule
CIFAR100 & ResNet18 & $1$ & $75$ & $1e-4$ & $0$\\
\midrule
\multirow{2}{*}{ImageNet} & ResNet50 & $1$ & $75$ & $0$ & $5e-5$ \\ & DeIT Small & $1$ & $225$ & $0$ & $1e-1$\\
\bottomrule
\end{tabular}
\end{table}
\newpage
\section{Experiments on two-layer networks with \signin}
\label{app:toy-expts}
We empirically verify the ability to recover correct signs for dense-to-sparse methods and \signin on the two-layer network with a student-teacher setup described in \cref{sec:theory}.
Further, we also empirically evaluate the effect of combining dense-to-sparse methods with \signin on the two-layer network.
We repeat $100$ runs for each case, with data generated from a teacher $\tilde{a}=1, \tilde{w}=\left[1, 0, .. , 0\right]$, such that the labels are $y = \tilde{f_0}(\tilde{a}, \tilde{w}, z)$.
For the multidimensional case we use $d=5$.
Each setting is trained with Gradient Descent for $2e-5$ iterations with a learning rate $=0.01$.
Results reported in \cref{tab:toy-expts} are the success fraction over $100$ runs.
We empirically observe that it is possible to successfully recover all cases when combining overparameterized training with \signin.
Even the most difficult case that is unreachable for both on their own i.e. $a < 0$ and $w < 0$.
Now it is possible due to \signin being able to flip $a$, whereas overparameterized training can flip $w$ after that.
This is also observed with improved performance of AC/DC with \signin (see \cref{tab:acdc-mw}).
% However, in spite of combining both these methods, we are unable to recover the case $a<0$ and $w<0$.
\begin{table}[ht!]
\centering
\caption{Experiments on two-layer networks in the student-teacher setup presented in \cref{theorem : sign flip main}.}
\label{tab:toy-expts}
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{4}{c}{Initial Signs} \\
\cmidrule{2-5}
& $a > 0, w > 0$ & $a < 0, w > 0$ & $a > 0, w < 0$ & $a < 0, w < 0$ \\
\midrule
Sparse & $1$ & $0$ & $0$ & $0$ \\
% \midrule
Overparam & $1$ & $0$ & $1$ & $0$ \\
% \midrule
\signin & $1$ & $1$ & $0$ & $0$ \\
% \midrule
Overparam + \signin & $1$ & $1$ & $1$ & $1$ \\
\bottomrule
\end{tabular}
\end{table}
\begin{figure}[ht]
\centering
\includegraphics[width=0.4\linewidth]{Figures/signin_gf_multidim.png}
\caption{Student-teacher setup: training one neuron with multidimensional inputs $d=5$. Combining the overparameterized case with \signin can recover all cases .}
\label{fig:multidim riemann vs normal}
\end{figure}
\paragraph{Failure case for Overparam + \signin}
We find that Overparam + \signin does not always succeed in recovering all the cases, and requires sufficient overparameterization to correctly flip $a<0$.
We choose the teacher network $ \tilde{f_0}(\tilde{a}, \tilde{w}, z)$ with $\tilde{a} = 1$ and $\tilde{\mathbf{w_{ij}}} \sim N(0, 1)$, and train the student with $d=2$ for the case with $a<0$.
We find that Overparam + \signin is not able resolve this case every time and requires additional overparameterization to improve success rates.
\newpage
\section{\signin flips signs in later layers}
Here we validate that the sign flips enabled by \signin are different from those enabled with dense-to-sparse training.
For the case $s=0.9$ reported in \cref{tab:acdc-mw} for AC/DC and AC/DC + \signin, we plot the layerwise sign flips for three stages, initialization to warmup ($10$-th epoch), warmup to final and initialization to final.
Additionally, we also compare the layerwise sign flips of sparse training a randomly reinitialized AC/DC mask.
While AC/DC identifies the correct sign flips upto warmup, \signin enables more sign flips during subsequent training for later layers in the network (see orange line in \cref{fig:warmup}).
\begin{figure}[ht!]
\centering
% Second subfigure
\subfigure[Init to Warmup]{
\includegraphics[width=0.3\textwidth]{Figures/init_warmup_acdc_mw.png}
\label{fig:init}
}
\subfigure[Warmup to Final]{
\includegraphics[width=0.3\textwidth]{Figures/warmup_final_acdc_mw.png}
\label{fig:warmup}
}
% % Third subfigure
\subfigure[Init to Final]{
\includegraphics[width=0.3\textwidth]{Figures/init_final_acdc_mw.png}
\label{fig:final}
}
\caption{Layerwise sign flips for \signin for AC/DC, AC/DC + \signin and sparse training with an AC/DC mask on a ResNet50 with $90\%$ trained on ImageNet.}
\label{fig:acdc-mw-flips}
\end{figure}
\newpage
\section{\signin with other PaI methods}\label{appendix ablation}
We report sparse training performance with \signin across other PaI methods including SNIP \citep{snip} and Synflow \citep{synflow} in \cref{tab:sign-recovery-appendix}.
We use the same setup as described in Appendix \ref{appendix : training details}.
We observe in Table \ref{tab:sign-recovery-appendix} that \signin improves all PaI methods.
Moreover, in most cases the random mask performs best over all.
% However, in most cases the random mask is a better baseline for comparison.
% NPB C10 Res20 87.30 ± 0.39 85.98 ± 0.32 82.06 ± 0.9
% NPB+Sign-In C10 Res20 88.87 ± 0.016 86.88 ± 0.51 83.37 ± 0.65
% NPB C100 Res18 74.92 ± 0.63 72.79 ± 0.56 71.49 ± 0.24
% NPB+Sign-In C100 Res18 75.59 ± 0.15 74.72 ± 0.18 73.28 ± 0.40
\begin{table*}[ht]
\centering
\caption{\signin with PaI methods on CIFAR10 and CIFAR100.
($*$ denotes a single run, as the other seeds crashed).}
\label{tab:sign-recovery-appendix}
\begin{tabular}{lcccc}
\toprule
Dataset & Method & $s = 0.8$ & $s = 0.9$ & $s = 0.95$ \\
\midrule
\multirow{6}{*}{CIFAR10 \textsubscript{ + ResNet20}} & Random & $88.41 (\pm 0.23)$ & $86.41(\pm 0.1)$ & $83.62 (\pm 0.28)$ \\
& Random + \signin & $89.37 (\pm 0.14)$ & $\mathbf{87.83 (\pm 0.11)}$ & $\mathbf{84.74 (\pm 0.06)}$ \\
% & Random + $m \odot w$ & $89.11 (\pm 0.08)$ & $87.13 (\pm 0.23)$ & $83.88 (\pm 0.02)$ \\
& SNIP & $ 88.12(\pm 0.5)$ & $ 85.82(\pm 0.2)$ & $ 81.98(\pm 0.32)$ \\
& SNIP + \signin & $\mathbf{89.88 (\pm 0.09)}$ & $87.27 (\pm 0.13)$ & $83.37 (\pm 0.06)$ \\
& Synflow & $ 88.15(\pm0.17) $ & $ 85.32(\pm0.24)$ & $73.22^*$ \\
& Synflow + \signin & $89.26 (\pm 0.05)$ & $86.47 (\pm 0.06)$ & $80.13^*$ \\
& NPB & $ 87.30(\pm0.39) $ & $ 85.98(\pm0.32)$ & $82.06(\pm0.9)$ \\
& NPB + \signin & $88.87 (\pm 0.02)$ & $86.88 (\pm 0.51)$ & $83.37(\pm0.65)$ \\
\midrule
\multirow{6}{*}{CIFAR100\textsubscript{ + ResNet18}} & Random & $74.01(\pm{0.1})$ & $73.07(\pm{0.19})$ & $71.41(\pm{0.18})$ \\
& Random + \signin & $75.32 (\pm 0.25)$ & $73.94 (\pm 0.13)$ & $72.51 (\pm 0.14)$ \\
% & Random + $m \odot w$ & $74.62 (\pm 0.24)$ & $73.18 (\pm 0.01)$ & $72.08 (\pm 0.25)$ \\
& SNIP & $ 74.08(\pm0.28) $ & $ 72.73(\pm 0.32)$ & $ 71.21(\pm 0.15)$ \\
& SNIP + \signin & $75.16 (\pm 0.15)$ & $73.64 (\pm 0.32)$ & $72.03 (\pm 0.50)$ \\
& Synflow & $ 73.41(\pm0.28) $ & $ 71.37(\pm 0.42)$ & $ 68.56(\pm0.17) $ \\
& Synflow + \signin & $ 75.16(\pm0.20) $ & $ 73.64(\pm0.33) $ & $ 72.03(\pm0.70) $ \\
& NPB & $ 74.92(\pm0.63) $ & $ 72.79(\pm0.56)$ & $71.49(\pm0.24)$ \\
& NPB + \signin & $\mathbf{75.59 (\pm 0.15)}$ & $\mathbf{74.72 (\pm 0.18)}$ & $\mathbf{73.28(\pm0.4)}$ \\
% \midrule
% \multirow{4}{*}{ImageNet\textsubscript{ + ResNet50}} & Random & $73.87(\pm 0.1)$ & $71.55(\pm 0.04)$ & $68.69(\pm 0.03)$ \\
% & Random + \signin & $\mathbf{74.2}$ & $\mathbf{72.14}$ & $\mathbf{69.82}$ \\
% & SNIP $m \odot w$ (ours) & & & \\
% & Synflow $m \odot w$ (ours) & & & \\
\bottomrule
\end{tabular}
\end{table*}
\paragraph{Different masks}
\label{app:difft-masks}
We perform sparse training with a random reinitialization of masks identified by dense-to-sparse and compare them with PaI methods.
Once reinitialized, dense-to-sparse masks loose alignment with parameters and are at par with PaI methods \citep{jain2024winning, frankle2021review}.
These experts reveal two key insights:
\begin{itemize}
\item Layerwise sparsity ratios determine trainiability of sparse networks in the absence of sign alignment.
\item \signin benefits from low sparsity in initial layers.
\end{itemize}
First, we observe that in case of masks identified by STR and PaI with Snip and Synflow achieve poor performance for both standard training and \signin.
The layerwise sparsity ratios of these masks reveal sparse initial layers ($>90\%$) or highly sparse ($>99\%$) later layers, which potentially affect the overall trainability of the network.
In contrast, the RiGL mask, which is a random mask with balanced layerwise sparsity ratio \citep{er-paper}, performs the best over all other masks with \signin.
Here initial layers are dense while later layers always have sparsity $<99\%$.
Second, we observe that when initial layers have low sparsity, \signin can outperform standard training (RiGL, AC/DC, Snip) but struggles if this is not the case (STR, Synflow).
A potential cause would be the inability of initial layers to flip $w$ (see \cref{theorem : sign flip main}).
% This is in line with \signin's inability to flip $w$ in the two-layer example, as it finds initial layers harder to flip with sparser layers.
% For Snip, we follow the implementation of \citet{frankle2021review}, however, the masks identified at $s=0.9$.
\begin{table}[ht!]
\centering
\caption{\signin with different masks for a ResNet50 trained on ImageNet with $90\%$ sparsity and random initialization. ($*$ denotes a single run, as the runs for other seeds crashed.}
\label{tab:mw-differentt-masks}
\begin{tabular}{lcc}
\toprule
\multirow{2}{*}{Mask} & \multicolumn{2}{c}{Init}\\
\cmidrule{2-3}
& Random & Random + \signin
\\
\midrule
AC/DC & $70.66_{\pm0.12}$ & $70.96_{\pm0.09}$ \\
RiGL & $72.02_{\pm0.23}$ & $\mathbf{72.48_{\pm0.19}}$ \\
STR & $68.36_{\pm0.17}$ & $67.81_{\pm0.34}$ \\
Snip & $52.9^*$ & $54.27^*$ \\
Synflow & $60.66_{\pm0.2}$ & $60.59_{\pm0.07}$ \\
Random & $71.56 _{\pm0.03}$& $72.19 _{\pm0.18}$ \\
\bottomrule
% 0.09756347222 70.96666667
% 0.1903656832 72.48
% 0.3479268378 67.81333333
% #DIV/0! 54.27
% 0.07420440239 60.59
\end{tabular}
\end{table}
\begin{figure}[ht!]
\centering
\includegraphics[width=0.5\linewidth]{Figures/layerwise_ratios.png}
\caption{Layerwise sparsity ratios at $90\%$ sparsity for a ResNet50.}
\label{fig:layerwise-ratios}
\end{figure}
\newpage
\section{Sign Importance}
In order to identify the amount of correct signs required for training with the sparse mask to match the respective dense-to-sparse baseline (via sign-alignment), we perturb an increasing fraction of signs as reported in Table \ref{tab:sign_perturb_acdc}.
Perturbing more than $10\%$ of the signs starts degrading performance rapidly.
\begin{table}[ht!]
\centering
\caption{Ablation study on the importance of weight signs.}
\label{tab:sign_perturb_acdc}
\begin{tabular}{lr}
\toprule
\textbf{Method} & $s=0.9$ \\
\midrule
AC/DC baseline (from Table 1) & $74.68$ \\
AC/DC Mask + $100\%$ correct signs (from Table 1) & $74.88$ \\
AC/DC Mask + $95\%$ correct signs & $74.44$ \\
AC/DC Mask + $90\%$ correct signs & $73.54$ \\
AC/DC Mask + $80\%$ correct signs & $71.81$ \\
AC/DC Mask + random signs (from Table 1) & $70.88$ \\
\bottomrule
\end{tabular}
\end{table}
\section{Vision transformer with \signin}
\label{app:vit-signin}
\signin is also able to improve sparse training of a randomly masked vision transformer.
A DeiT Small architecture (trained for $300$ epochs from scratch) gains more than $1\%$ in performance with \signin, as reported in \cref{tab:deit-results}.
\begin{table}[ht!]
\centering
\caption{\textbf{DeiT Small} on ImageNet, trained from scratch.}
\label{tab:deit-results}
\begin{tabular}{lcc}
\toprule
Method & $s=0.5$
\\
\midrule
Random & $69.31 (\pm 0.7)$ \\
Random + \signin (ours) & $\mathbf{70.62 (\pm 0.4)}$ \\
\bottomrule
\end{tabular}
\end{table}
\section{\signin improves other sparse training methods}
\label{app:acdc-signin}
Only a combination of \signin and overparameterization is able to solve all sign flip challenges in our theoretical setting, as also confirmed by numerical experiments, which are provided in \cref{tab:toy-expts} in \cref{app:toy-expts}.
Therefore, we expect the combination of dense-to-sparse methods with \signin to achieve further gains in performance.
\cref{tab:acdc-mw} confirms that AC/DC improves with \signin.
However, the improvement is less significant compared to PaI. %What is going on?
\begin{table}[ht!]
\centering
\caption{\textbf{\signin improves AC/DC} for a ResNet50 trained on ImageNet.}
\label{tab:acdc-mw}
\setlength\tabcolsep{2.2pt}
\begin{tabular}{lccc}
\toprule
Method & $s=0.8$ & $s=0.9$ & $s=0.95$
\\
\midrule
AC/DC & $75.83(\pm0.02)$ & $74.75(\pm0.02)$ & $72.59(\pm0.11)$\\
\midrule
AC/DC & \multirow{2}{*}{$\mathbf{75.9(\pm0.14})$} &\multirow{2}{*}{$74.74(\pm0.12)$}& \multirow{2}{*}{$\mathbf{72.88(\pm0.13)}$}\\
+ \signin & & &\\
\bottomrule
% 0.02355626833 0.02278251181 0.1133490969
% 75.82333333 74.75866667 72.59333333
\end{tabular}
\end{table}
\signin also improves dynamic sparse training with RiGL and MEST by enabling better sign flips.
\begin{table}[ht!]
\centering
\caption{\textbf{\signin improves RiGL} for a ResNet50 trained on ImageNet.}
\label{tab:rigl-mw}
\setlength\tabcolsep{2.2pt}
\begin{tabular}{lccc}
\toprule
% RiGL 75.02 ± 0.1 73.70 ± 0.2 71.89 ± 0.07
% RiGL + Sign-In 75.02 ± 0.1 74.27 ± 0.08 73.07 ± 0.17
Method & $s=0.8$ & $s=0.9$ & $s=0.95$
\\
\midrule
RiGL & $75.02(\pm0.1)$ & $73.7(\pm0.2)$ & $71.89(\pm0.07)$\\
\midrule
RiGL & \multirow{2}{*}{$75.02(\pm0.1)$} &\multirow{2}{*}{$\mathbf{74.27(\pm0.08)}$}& \multirow{2}{*}{$\mathbf{73.07(\pm0.17)}$}\\
+ \signin & & &\\
\bottomrule
% 0.02355626833 0.02278251181 0.1133490969
% 75.82333333 74.75866667 72.59333333
\end{tabular}
\end{table}
\begin{table}[ht!]
\centering
\caption{\textbf{\signin improves MEST} for a ResNet50 trained on ImageNet.}
\label{tab:mest-mw}
\setlength\tabcolsep{2.2pt}
\begin{tabular}{lccc}
\toprule
Method & $s=0.8$ & $s=0.9$ & $s=0.95$
\\
\midrule
MEST & $74.67$ & $73.1$ & $70.84$\\
\midrule
MEST & \multirow{2}{*}{$\mathbf{74.74}$} &\multirow{2}{*}{$\mathbf{73.32}$}& \multirow{2}{*}{$\mathbf{71.41}$}\\
+ \signin & & &\\
\bottomrule
\end{tabular}
\end{table}
\section{Rescaling in \signin}
While our proposed pointwise reparameterization with $m\odot w$ allows better sign flips for sparse training, we find that the rescaling step outlined in \cref{app: alg} is essential to maximize the benefits of the reparameterization as show with improved performance of \signin over $m\odot w$ in \cref{tab:rescaling-appendix}.
\begin{table*}[ht!]
\centering
\caption{Effect of Rescaling on \signin.}
\label{tab:rescaling-appendix}
\begin{tabular}{lcccc}
\toprule
Dataset & Method & $s = 0.8$ & $s = 0.9$ & $s = 0.95$ \\
\midrule
\multirow{3}{*}{CIFAR10 \textsubscript{ + ResNet20}} & Random & $88.41 (\pm 0.23)$ & $86.41(\pm 0.1)$ & $83.62 (\pm 0.28)$ \\
& Random + \signin & $\mathbf{89.37 (\pm 0.14)}$ & $\mathbf{87.83 (\pm 0.11)}$ & $\mathbf{84.74 (\pm 0.06)}$ \\
& Random + $m \odot w$ & $89.11 (\pm 0.08)$ & $87.13 (\pm 0.23)$ & $83.88 (\pm 0.02)$ \\
\midrule
\multirow{3}{*}{CIFAR100\textsubscript{ + ResNet18}} & Random & $74.01(\pm{0.1})$ & $73.07(\pm{0.19})$ & $71.41(\pm{0.18})$ \\
& Random + \signin & $\mathbf{75.32 (\pm 0.25)}$ & $\mathbf{73.94 (\pm 0.13)}$ & $\mathbf{72.51 (\pm 0.14)}$ \\
& Random + $m \odot w$ & $74.62 (\pm 0.24)$ & $73.18 (\pm 0.01)$ & $72.08 (\pm 0.25)$ \\
\midrule
\multirow{2}{*}{ImageNet\textsubscript{ + ResNet50}} & Random & $73.87(\pm 0.1)$ & $71.55(\pm 0.04)$ & $68.69(\pm 0.03)$ \\
& Random + \signin & $\mathbf{74.12 (\pm 0.09)}$ & $\mathbf{72.19 (\pm 0.18)}$ & $\mathbf{69.38 (\pm 0.1)}$ \\
& Random + $m \odot w$ & $73.96 (\pm 0.04)$ & $71.83 (\pm 0.17)$ & $69.15 (\pm 0.09)$\\
% 0.03895970796 0.1778459258
% 73.96466667 71.83
\bottomrule
\end{tabular}
\end{table*}
\newpage
\section{Code for \signin in PyTorch}
We provide an example of integrating \signin in any training setup in PyTorch by simply replacing the Conv (or Linear) layer with \ref{code:conv_mask_mw}.
\begin{lstlisting}[caption={Conv2d layer with \signin parametrization of \(m \odot w\).}, label={code:conv_mask_mw}]
import torch
import torch.nn as nn
class ConvMaskMW(nn.Conv2d):
"""
Conv2d layer with a parametrization of m*w for the weights.
Additionally a mask is applied during the forward pass to the weights of the layer.
Args:
**kwargs: Keyword arguments for nn.Conv2d.
"""
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.register_buffer("mask", torch.ones_like(self.weight))
self.alpha = alpha = 0.5
u = torch.sqrt(self.weight.to(self.weight.device) + torch.sqrt(self.weight.to(self.weight.device)*self.weight.to(self.weight.device) + alpha*alpha))
self.m = nn.Parameter((u + alpha/u)/np.sqrt(2), requires_grad=True)
self.weight.data = (u - alpha/u)/np.sqrt(2)
def merge(self):
# This method combines m and w to give an effective weight, for inference or during pruning.
self.weight.data = self.m.to(self.weight.device) * self.weight
def rescale_mw(self):
x = self.m.to(self.weight.device) * self.weight.to(self.weight.device)
u = torch.sqrt(self.alpha + torch.sqrt(x*x + self.alpha*self.alpha))
self.m.data = u
self.weight.data = x/u
def forward(self, x):
"""Forward pass with masked weights."""
sparseWeight = self.mask.to(self.weight.device) * self.m.to(self.weight.device) * self.weight.to(self.weight.device)
return F.conv2d(
x,
sparseWeight,
self.bias,
self.stride,
self.padding,
self.dilation,
self.groups,
)
\end{lstlisting}
\section{FLOPS required for \signin}
We perform a FLOP comparison for Sign-In vs. other PaI methods.
The FLOP overhead due to the reparameterization in Sign-In is negligible and only required during training, as $m \odot w$ can be merged after training, during inference.
For example, the FLOPs for a convolutional layer are:
\[
(2 \times H_{\text{out}} \times W_{\text{out}} \times C_{\text{out}} \times K \times K \times C_{\text{in}}) + (C_{\text{out}} \times C_{\text{in}} \times K \times K)
\]
An additional term $(C_{\text{out}} \times C_{\text{in}} \times K \times K)$ is introduced by the elementwise product $m \odot w$, but it is negligible with respect to the other term.
We report the FLOPs per forward pass for a ResNet50.
\begin{table}[ht!]
\centering
\setlength{\tabcolsep}{8pt}
\caption{\textbf{FLOP Comparison} of Sign-In vs. other PaI methods on ResNet50. Sign-In introduces minimal overhead while improving accuracy.}
\begin{tabular}{lccc}
\toprule
\textbf{Method} & \textbf{Training FLOPs (G)} & \textbf{Inference FLOPs (G)} & \textbf{Acc} \\
\midrule
Dense & 8.2 & 8.2 & 76.89 \\
\midrule
Sparse ($s=0.8$) & 4.99 & 4.99 & 73.87 \\
Sign-In ($s=0.8$) & 5 & 4.99 & 74.12 \\
\midrule
Sparse ($s=0.9$) & 3.89 & 3.89 & 71.56 \\
Sign-In ($s=0.9$) & 3.9 & 3.89 & 72.19 \\
\bottomrule
\end{tabular}
\label{tab:flops-signin}
\end{table}
\section{High sparsity regime ablation}
In this section we provide results for the high sparsity regime.
\begin{table*}[ht]
\centering
\caption{\signin with PaI methods on CIFAR10 and CIFAR100 in the high sparsity regime.}
\label{tab:cifar-high-sparsity}
\begin{tabular}{lccc}
\toprule
Dataset & Method & $s = 0.98$ & $s = 0.99$ \\
\midrule
\multirow{6}{*}{CIFAR10 \textsubscript{ + ResNet20}} & Random & $88.41 (\pm 0.23)$ & $86.41(\pm 0.1)$ \\
& Random + \signin & $89.37 (\pm 0.14)$ & $\mathbf{87.83 (\pm 0.11)}$ \\
% & Random + $m \odot w$ & $89.11 (\pm 0.08)$ & $87.13 (\pm 0.23)$ & $83.88 (\pm 0.02)$ \\
& SNIP & $ 88.12(\pm 0.5)$ & $ 85.82(\pm 0.2)$ \\
& SNIP + \signin & $\mathbf{89.88 (\pm 0.09)}$ & $87.27 (\pm 0.13)$ \\
& Synflow & $ 88.15(\pm0.17) $ & $ 85.32(\pm0.24)$ \\
& Synflow + \signin & $89.26 (\pm 0.05)$ & $86.47 (\pm 0.06)$ \\
\midrule
\multirow{6}{*}{CIFAR100\textsubscript{ + ResNet18}} & Random & $74.01(\pm{0.1})$ & $73.07(\pm{0.19})$ \\
& Random + \signin & $\mathbf{75.32 (\pm 0.25)}$ & $\mathbf{73.94 (\pm 0.13)}$ \\
% & Random + $m \odot w$ & $74.62 (\pm 0.24)$ & $73.18 (\pm 0.01)$ & $72.08 (\pm 0.25)$ \\
& SNIP & $ 74.08(\pm0.28) $ & $ 72.73(\pm 0.32)$ \\
& SNIP + \signin & $75.16 (\pm 0.15)$ & $73.64 (\pm 0.32)$ \\
& Synflow & $ 73.41(\pm0.28) $ & $ 71.37(\pm 0.42)$ \\
& Synflow + \signin & $ 75.16(\pm0.20) $ & $ 73.64(\pm0.33) $ \\
% \midrule
% \multirow{4}{*}{ImageNet\textsubscript{ + ResNet50}} & Random & $73.87(\pm 0.1)$ & $71.55(\pm 0.04)$ & $68.69(\pm 0.03)$ \\
% & Random + \signin & $\mathbf{74.2}$ & $\mathbf{72.14}$ & $\mathbf{69.82}$ \\
% & SNIP $m \odot w$ (ours) & & & \\
% & Synflow $m \odot w$ (ours) & & & \\
\bottomrule
\end{tabular}
\end{table*}
% \subsection{FLOP comparison}
% \begin{table*}[ht!]
% \centering
% \caption{FLOP comparison, per forward pass during training for ImageNet.}
% \label{tab:flop-compare}
% \begin{tabular}{lcccc }
% \toprule
% Model & Method & $s = 0.8$ & $s = 0.9$ & $s = 0.95$ \\
% \midrule
% \multirow{3}{*}{ResNet50} & Dense Baseline & \multicolumn{3}{c}{8.2G} \\
% \cmidrule{2-5}
% & Random & $5$G & $3.9$G & $2.7$G \\
% & Random + \signin & $7.4$G & $5.7$G & $4.1$G \\
% \bottomrule
% \end{tabular}
% \end{table*}
% 500400 iterations for imagenet
% 8.2 per fw pass for acdc
% ACDC total = 8.2 * 2.5 + 3.9 * 2.5 = 30.25
% mw total = 5.7 * 5 = 28.5
% random total = 3.9 * 5 = 19.5
% STR = 30.5
\begin{thebibliography}{61}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Alizadeh et~al.(2022)Alizadeh, Tailor, Zintgraf, van Amersfoort, Farquhar, Lane, and Gal]{alizadehprospect}
Alizadeh, M., Tailor, S.~A., Zintgraf, L.~M., van Amersfoort, J., Farquhar, S., Lane, N.~D., and Gal, Y.
\newblock Prospect pruning: Finding trainable weights at initialization using meta-gradients.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\bibitem[Burkholz(2022{\natexlab{a}})]{convexist}
Burkholz, R.
\newblock Convolutional and residual networks provably contain lottery tickets.
\newblock In \emph{International Conference on Machine Learning}, 2022{\natexlab{a}}.
\bibitem[Burkholz(2022{\natexlab{b}})]{depthexist}
Burkholz, R.
\newblock Most activation functions can win the lottery without excessive depth.
\newblock In \emph{Advances in Neural Information Processing Systems}, 2022{\natexlab{b}}.
\bibitem[Burkholz(2024)]{BNtheoryCNN}
Burkholz, R.
\newblock Batch normalization is sufficient for universal function approximation in {CNN}s.
\newblock In \emph{International Conference on Learning Representations}, 2024.
\bibitem[Burkholz et~al.(2022)Burkholz, Laha, Mukherjee, and Gotovos]{uniExist}
Burkholz, R., Laha, N., Mukherjee, R., and Gotovos, A.
\newblock On the existence of universal lottery tickets.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\bibitem[Chen et~al.(2021)Chen, Cheng, Gan, Yuan, Zhang, and Wang]{chen2021chasing}
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., and Wang, Z.
\newblock Chasing sparsity in vision transformers: An end-to-end exploration.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 19974--19988, 2021.
\bibitem[Chizat \& Bach(2020)Chizat and Bach]{pmlr-v125-chizat20a}
Chizat, L. and Bach, F.
\newblock Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss.
\newblock In Abernethy, J. and Agarwal, S. (eds.), \emph{Proceedings of Thirty Third Conference on Learning Theory}, volume 125 of \emph{Proceedings of Machine Learning Research}, pp.\ 1305--1338. PMLR, 09--12 Jul 2020.
\newblock URL \url{https://proceedings.mlr.press/v125/chizat20a.html}.
\bibitem[Chizat et~al.(2020)Chizat, Oyallon, and Bach]{chizat2020lazy}
Chizat, L., Oyallon, E., and Bach, F.
\newblock On lazy training in differentiable programming, 2020.
\bibitem[Chou et~al.(2024)Chou, Maly, and Stöger]{chou2024induce}
Chou, H.-H., Maly, J., and Stöger, D.
\newblock How to induce regularization in linear models: A guide to reparametrizing gradient flow, 2024.
\bibitem[da~Cunha et~al.(2022)da~Cunha, Natale, and Viennot]{cnnexist}
da~Cunha, A., Natale, E., and Viennot, L.
\newblock Proving the lottery ticket hypothesis for convolutional neural networks.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\bibitem[Deng et~al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei]{imagenet}
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.
\newblock {ImageNet: A Large-Scale Hierarchical Image Database}.
\newblock In \emph{CVPR09}, 2009.
\bibitem[Domin{\'e} et~al.(2024)Domin{\'e}, Anguita, Proca, Braun, Kunin, Mediano, and Saxe]{domin2024from}
Domin{\'e}, C. C.~J., Anguita, N., Proca, A.~M., Braun, L., Kunin, D., Mediano, P. A.~M., and Saxe, A.~M.
\newblock From lazy to rich: Exact learning dynamics in deep linear networks.
\newblock In \emph{UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models}, 2024.
\newblock URL \url{https://openreview.net/forum?id=yAE3LOjgA4}.
\bibitem[Evci et~al.(2020)Evci, Gale, Menick, Castro, and Elsen]{evci-rigl}
Evci, U., Gale, T., Menick, J., Castro, P.~S., and Elsen, E.
\newblock Rigging the lottery: Making all tickets winners.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 2943--2952. PMLR, 2020.
\bibitem[Frankle \& Carbin(2019)Frankle and Carbin]{frankle2019lottery}
Frankle, J. and Carbin, M.
\newblock The lottery ticket hypothesis: Finding sparse, trainable neural networks.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\bibitem[Frankle et~al.(2021)Frankle, Dziugaite, Roy, and Carbin]{frankle2021review}
Frankle, J., Dziugaite, G.~K., Roy, D., and Carbin, M.
\newblock Pruning neural networks at initialization: Why are we missing the mark?
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Frantar \& Alistarh(2023)Frantar and Alistarh]{frantar2023sparsegpt}
Frantar, E. and Alistarh, D.
\newblock Sparsegpt: Massive language models can be accurately pruned in one-shot.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 10323--10337. PMLR, 2023.
\bibitem[Gadhikar \& Burkholz(2024)Gadhikar and Burkholz]{gadhikar2024masks}
Gadhikar, A. and Burkholz, R.
\newblock Masks, signs, and learning rate rewinding.
\newblock In \emph{Twelfth International Conference on Learning Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=qODvxQ8TXW}.
\bibitem[Gadhikar et~al.(2023)Gadhikar, Mukherjee, and Burkholz]{er-paper}
Gadhikar, A.~H., Mukherjee, S., and Burkholz, R.
\newblock Why random pruning is all we need to start sparse.
\newblock In \emph{International Conference on Machine Learning}, 2023.
\bibitem[Gunasekar et~al.(2017)Gunasekar, Woodworth, Bhojanapalli, Neyshabur, and Srebro]{gunasekar2017implicit}
Gunasekar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., and Srebro, N.
\newblock Implicit regularization in matrix factorization, 2017.
\bibitem[Gunasekar et~al.(2020)Gunasekar, Lee, Soudry, and Srebro]{gunasekar2020characterizing}
Gunasekar, S., Lee, J., Soudry, D., and Srebro, N.
\newblock Characterizing implicit bias in terms of optimization geometry, 2020.
\bibitem[Han et~al.(2015)Han, Pool, Tran, and Dally]{han-efficient-nns}
Han, S., Pool, J., Tran, J., and Dally, W.
\newblock Learning both weights and connections for efficient neural network.
\newblock \emph{Advances in neural information processing systems}, 28, 2015.
\bibitem[Jacobs \& Burkholz(2025)Jacobs and Burkholz]{jacobs2024maskmirrorimplicitsparsification}
Jacobs, T. and Burkholz, R.
\newblock Mask in the mirror: Implicit sparsification.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\newblock URL \url{https://openreview.net/forum?id=U47ymTS3ut}.
\bibitem[Jacobs et~al.(2025)Jacobs, Zhou, and Burkholz]{jacobs2025mirrormirrorflowdoes}
Jacobs, T., Zhou, C., and Burkholz, R.
\newblock Mirror, mirror of the flow: How does regularization shape implicit bias?
\newblock In \emph{Forty-second International Conference on Machine Learning}, 2025.
\newblock URL \url{https://openreview.net/forum?id=MLiR9LS5PW}.
\bibitem[Jain et~al.(2024)Jain, Adnan, Sharma, and Ioannou]{jain2024winning}
Jain, R., Adnan, M., Sharma, E., and Ioannou, Y.
\newblock Winning tickets from random initialization: Aligning masks for sparse training.
\newblock In \emph{UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models}, 2024.
\newblock URL \url{https://openreview.net/forum?id=xLiQjuoUyl}.
\bibitem[Jiang et~al.(2019)Jiang, Neyshabur, Mobahi, Krishnan, and Bengio]{jiangfantastic}
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S.
\newblock Fantastic generalization measures and where to find them.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\bibitem[{Jülich Supercomputing Centre}(2021)]{juelich2021juwels}
{Jülich Supercomputing Centre}.
\newblock {JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre}.
\newblock \emph{Journal of large-scale research facilities}, 7:\penalty0 A183, 2021.
\newblock \doi{10.17815/jlsrf-7-183}.
\bibitem[Kolb et~al.(2025)Kolb, Weber, Bischl, and R{\"u}gamer]{kolb2025deep}
Kolb, C., Weber, T., Bischl, B., and R{\"u}gamer, D.
\newblock Deep weight factorization: Sparse learning through the lens of artificial symmetries.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\newblock URL \url{https://openreview.net/forum?id=vNdOHr7mn5}.
\bibitem[Krizhevsky et~al.(2009)Krizhevsky, Hinton, et~al.]{cifar}
Krizhevsky, A., Hinton, G., et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock 2009.
\bibitem[Kumar et~al.(2024)Kumar, Luo, and Sellke]{kumarno}
Kumar, T., Luo, K., and Sellke, M.
\newblock No free prune: Information-theoretic barriers to pruning at initialization.
\newblock In \emph{Forty-first International Conference on Machine Learning}, 2024.
\bibitem[Kusupati et~al.(2020)Kusupati, Ramanujan, Somani, Wortsman, Jain, Kakade, and Farhadi]{kusupati20}
Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S., and Farhadi, A.
\newblock Soft threshold weight reparameterization for learnable sparsity.
\newblock In \emph{Proceedings of the International Conference on Machine Learning}, July 2020.
\bibitem[Kuznedelev et~al.(2024)Kuznedelev, Kurti{\'c}, Frantar, and Alistarh]{kuznedelev2024cap}
Kuznedelev, D., Kurti{\'c}, E., Frantar, E., and Alistarh, D.
\newblock Cap: Correlation-aware pruning for highly-accurate sparse vision models.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Lasby et~al.(2023)Lasby, Golubeva, Evci, Nica, and Ioannou]{struct-dst}
Lasby, M., Golubeva, A., Evci, U., Nica, M., and Ioannou, Y.
\newblock Dynamic sparse training with structured sparsity.
\newblock \emph{arXiv preprint arXiv:2305.02299}, 2023.
\bibitem[Lee et~al.(2019)Lee, Ajanthan, and Torr]{snip}
Lee, N., Ajanthan, T., and Torr, P. H.~S.
\newblock Snip: single-shot network pruning based on connection sensitivity.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\bibitem[Li et~al.(2021)Li, Nguyen, Hegde, and Wong]{li2021implicitsparseregularizationimpact}
Li, J., Nguyen, T.~V., Hegde, C., and Wong, R. K.~W.
\newblock Implicit sparse regularization: The impact of depth and early stopping, 2021.
\newblock URL \url{https://arxiv.org/abs/2108.05574}.
\bibitem[Li et~al.(2023)Li, Nguyen, Hegde, and Wong]{li2023implicitregularizationgroupsparsity}
Li, J., Nguyen, T.~V., Hegde, C., and Wong, R. K.~W.
\newblock Implicit regularization for group sparsity, 2023.
\newblock URL \url{https://arxiv.org/abs/2301.12540}.
\bibitem[Li et~al.(2022)Li, Wang, Lee, and Arora]{Li2022ImplicitBO}
Li, Z., Wang, T., Lee, J.~D., and Arora, S.
\newblock Implicit bias of gradient descent on reparametrized models: On equivalence to mirror descent.
\newblock \emph{ArXiv}, abs/2207.04036, 2022.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:250407876}.
\bibitem[Liang \& Montufar(2025)Liang and Montufar]{liang2025implicit}
Liang, S. and Montufar, G.
\newblock Implicit bias of mirror flow for shallow neural networks in univariate regression.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\newblock URL \url{https://openreview.net/forum?id=IF0Q9KY3p2}.
\bibitem[Liu et~al.(2021)Liu, Chen, Chen, Shen, Mocanu, Wang, and Pechenizkiy]{random-pruning-vita}
Liu, S., Chen, T., Chen, X., Shen, L., Mocanu, D.~C., Wang, Z., and Pechenizkiy, M.
\newblock The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Ma et~al.(2021)Ma, Yuan, Shen, Chen, Chen, Chen, Liu, Qin, Liu, Wang, et~al.]{ma2021sanity}
Ma, X., Yuan, G., Shen, X., Chen, T., Chen, X., Chen, X., Liu, N., Qin, M., Liu, S., Wang, Z., et~al.
\newblock Sanity checks for lottery tickets: Does your winning ticket really win the jackpot?
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 12749--12760, 2021.
\bibitem[Marcotte et~al.(2023)Marcotte, Gribonval, and Peyr'e]{Marcotte2023AbideBT}
Marcotte, S., Gribonval, R., and Peyr'e, G.
\newblock Abide by the law and follow the flow: Conservation laws for gradient flows.
\newblock \emph{ArXiv}, abs/2307.00144, 2023.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:259316537}.
\bibitem[Marcotte et~al.(2024)Marcotte, Gribonval, and Peyré]{marcotte2024momentumconservationlawseuclidean}
Marcotte, S., Gribonval, R., and Peyré, G.
\newblock Keep the momentum: Conservation laws beyond euclidean gradient flows, 2024.
\newblock URL \url{https://arxiv.org/abs/2405.12888}.
\bibitem[Mocanu et~al.(2018)Mocanu, Mocanu, Stone, Nguyen, Gibescu, and Liotta]{mocanu-adaptive-sparse-train}
Mocanu, D.~C., Mocanu, E., Stone, P., Nguyen, P.~H., Gibescu, M., and Liotta, A.
\newblock Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.
\newblock \emph{Nature Communications}, 9\penalty0 (1), jun 2018.
\newblock \doi{10.1038/s41467-018-04316-3}.
\newblock URL \url{https://doi.org/10.1038%2Fs41467-018-04316-3}.
\bibitem[Nelaturu et~al.(2024)Nelaturu, Gadhikar, and Burkholz]{Nelaturu_TurboPrune_High-Speed_Distributed}
Nelaturu, S.~H., Gadhikar, A., and Burkholz, R.
\newblock {TurboPrune: High-Speed Distributed Lottery Ticket Training}, 2024.
\newblock URL \url{https://github.com/nelaturuharsha/TurboPrune}.
\bibitem[Patil \& Dovrolis(2021)Patil and Dovrolis]{patil2021phew}
Patil, S.~M. and Dovrolis, C.
\newblock Phew: Constructing sparse networks that learn fast and generalize well without training data.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 8432--8442. PMLR, 2021.
\bibitem[Paul et~al.(2023)Paul, Chen, Larsen, Frankle, Ganguli, and Dziugaite]{paul2023unmasking}
Paul, M., Chen, F., Larsen, B.~W., Frankle, J., Ganguli, S., and Dziugaite, G.~K.
\newblock Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask?
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\newblock URL \url{https://openreview.net/forum?id=xSsW2Am-ukZ}.
\bibitem[Pesme et~al.(2021)Pesme, Pillaud-Vivien, and Flammarion]{pesme2021implicit}
Pesme, S., Pillaud-Vivien, L., and Flammarion, N.
\newblock Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity, 2021.
\bibitem[Peste et~al.(2021)Peste, Iofinova, Vladu, and Alistarh]{peste2021acdc}
Peste, A., Iofinova, E., Vladu, A., and Alistarh, D.
\newblock Ac/dc: Alternating compressed/decompressed training of deep neural networks, 2021.
\bibitem[Pham et~al.(2023)Pham, Liu, Xiang, Le, Wen, Tran-Thanh, et~al.]{pham-paths}
Pham, H., Liu, S., Xiang, L., Le, D.~D., Wen, H., Tran-Thanh, L., et~al.
\newblock Towards data-agnostic pruning at initialization: What makes a good sparse mask?
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\bibitem[Renda et~al.(2020)Renda, Frankle, and Carbin]{rewindVsFinetune}
Renda, A., Frankle, J., and Carbin, M.
\newblock Comparing rewinding and fine-tuning in neural network pruning.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Singh \& Alistarh(2020)Singh and Alistarh]{singh2020woodfisher}
Singh, S.~P. and Alistarh, D.
\newblock Woodfisher: Efficient second-order approximation for neural network compression.
\newblock \emph{Advances in Neural Information Processing Systems}, 33:\penalty0 18098--18109, 2020.
\bibitem[Su et~al.(2020)Su, Chen, Cai, Wu, Gao, Wang, and Lee]{su2020sanity}
Su, J., Chen, Y., Cai, T., Wu, T., Gao, R., Wang, L., and Lee, J.~D.
\newblock Sanity-checking pruning methods: Random tickets can win the jackpot.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 20390--20401, 2020.
\bibitem[Tanaka et~al.(2020)Tanaka, Kunin, Yamins, and Ganguli]{synflow}
Tanaka, H., Kunin, D., Yamins, D.~L., and Ganguli, S.
\newblock Pruning neural networks without any data by iteratively conserving synaptic flow.
\newblock In \emph{Advances in Neural Information Processing Systems}, 2020.
\bibitem[Touvron et~al.(2021)Touvron, Cord, Douze, Massa, Sablayrolles, and J{\'e}gou]{touvron2021training}
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J{\'e}gou, H.
\newblock Training data-efficient image transformers \& distillation through attention.
\newblock In \emph{International conference on machine learning}, pp.\ 10347--10357. PMLR, 2021.
\bibitem[Vaškevičius et~al.(2019)Vaškevičius, Kanade, and Rebeschini]{vaškevičius2019implicitregularizationoptimalsparse}
Vaškevičius, T., Kanade, V., and Rebeschini, P.
\newblock Implicit regularization for optimal sparse recovery, 2019.
\newblock URL \url{https://arxiv.org/abs/1909.05122}.
\bibitem[Wang et~al.(2020)Wang, Zhang, and Grosse]{grasp}
Wang, C., Zhang, G., and Grosse, R.~B.
\newblock Picking winning tickets before training by preserving gradient flow.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Woodworth et~al.(2020)Woodworth, Gunasekar, Lee, Moroshko, Savarese, Golan, Soudry, and Srebro]{woodworth2020kernel}
Woodworth, B., Gunasekar, S., Lee, J.~D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N.
\newblock Kernel and rich regimes in overparametrized models, 2020.
\bibitem[Xiang et~al.(2025)Xiang, Nguyen-Tri, Nguyen, Pham, Than, Tran-Thanh, and Wen]{xiangdpai}
Xiang, L., Nguyen-Tri, Q., Nguyen, L.-C., Pham, H., Than, K., Tran-Thanh, L., and Wen, H.
\newblock Dpai: Differentiable pruning at initialization with node-path balance principle.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\bibitem[Yuan et~al.(2021)Yuan, Ma, Niu, Li, Kong, Liu, Gong, Zhan, He, Jin, et~al.]{yuan2021mest}
Yuan, G., Ma, X., Niu, W., Li, Z., Kong, Z., Liu, N., Gong, Y., Zhan, Z., He, C., Jin, Q., et~al.
\newblock Mest: Accurate and fast memory-economic sparse training framework on the edge.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 20838--20850, 2021.
\bibitem[Zhang et~al.(2025)Zhang, Zhao, Wu, Liao, Michieli, and Cannistraci]{chts}
Zhang, Y., Zhao, J., Wu, W., Liao, Z., Michieli, U., and Cannistraci, C.~V.
\newblock Brain-inspired sparse training enables transformers and llms to perform as fully connected.
\newblock \emph{arXiv preprint arXiv:2501.19107}, 2025.
\bibitem[Zhao et~al.(2022)Zhao, Yang, and He]{Zhao_2022}
Zhao, P., Yang, Y., and He, Q.-C.
\newblock High-dimensional linear regression via implicit regularization.
\newblock \emph{Biometrika}, 109\penalty0 (4):\penalty0 1033–1046, February 2022.
\newblock ISSN 1464-3510.
\newblock \doi{10.1093/biomet/asac010}.
\newblock URL \url{http://dx.doi.org/10.1093/biomet/asac010}.
\bibitem[Ziyin \& Wang(2023)Ziyin and Wang]{ziyin2023spredsolvingl1penalty}
Ziyin, L. and Wang, Z.
\newblock spred: Solving $l_1$ penalty with sgd, 2023.
\newblock URL \url{https://arxiv.org/abs/2210.01212}.
\end{thebibliography}