From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Extracted source text · arXiv 2405.06038
\documentclass[lettersize,journal]{IEEEtran}
\usepackage{amsmath,amsfonts}
% \usepackage{algorithmic}
% \usepackage{algorithm}
\usepackage{array}
\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage{textcomp}
\usepackage{stfloats}
\usepackage{url}
\usepackage{verbatim}
\usepackage{graphicx}
\usepackage{cite}
\hyphenation{op-tical net-works semi-conduc-tor IEEE-Xplore}
% updated with editorial comments 8/9/2021
% own packages and def
\usepackage{graphicx}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\usepackage{xspace}
\usepackage{subfig}
\usepackage{tikz} %
\usetikzlibrary{svg.path}
\usetikzlibrary{petri,positioning,backgrounds}
\usepackage{threeparttable}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{makecell}
\usepackage{algorithm2e}
% \SetKwComment{Comment}{/* }{ */}
\SetKwComment{Comment}{// }{}%
\RestyleAlgo{ruled}
\makeatletter
\DeclareRobustCommand\onedot{\futurelet\@let@token\@onedot}
\def\@onedot{\ifx\@let@token.\else.\null\fi\xspace}
\def\eg{\emph{e.g}\onedot}
\def\ie{\emph{i.e}\onedot}
\def\aka{\emph{a.k.a}\onedot}
\def\etc{\emph{etc}\onedot}
\def\vs{\emph{vs}\onedot}
\def\etal{\emph{et al}\onedot}
\makeatother
\begin{document}
\title{From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks}
% \author{IEEE Publication Technology,~\IEEEmembership{Staff,~IEEE,}
% % <-this % stops a space
% \thanks{This paper was produced by the IEEE Publication Technology Group. They are in Piscataway, NJ.}% <-this % stops a space
% \thanks{Manuscript received April 19, 2021; revised August 16, 2021.}}
\author{Xue Geng, Zhe Wang, Chunyun Chen, Qing Xu, Kaixin Xu, Chao Jin, Manas Gupta, \\ Xulei Yang, Zhenghua Chen, Mohamed M. Sabry Aly, Jie Lin, Min Wu, Xiaoli Li~\IEEEmembership{Fellow, IEEE}
\thanks{Xue Geng, Zhe Wang, Qing Xu, Kaixin Xu, Chao Jin, Manas Gupta, Xulei Yang, Zhenghua Chen, Min Wu and Xiaoli Li are with the Institute for Infocomm Research, A*STAR, Singapore. (email: geng\_xue, wang\_zhe, xu\_qing, Xu\_Kaixin, jin\_chao, manas\_gupta, yang\_xulei, chen\_zhenghua, wumin, xlli@i2r.a-star.edu.sg).}
\thanks{Chunyun Chen and Mohamed M. Sabry Aly are with the Nanyang Technological University (email: chunyun001@e.ntu.edu.sg, msabry@ntu.edu.sg).}
\thanks{Dr. Jie Lin, Dr. Min Wu and Dr. Xiaoli Li are the corresponding authors.}}
% The paper headers
\markboth{Journal of \LaTeX\ Class Files,~Vol.~14, No.~8, August~2021}%
{Shell \MakeLowercase{\textit{et al.}}: A Sample Article Using IEEEtran.cls for IEEE Journals}
\IEEEpubid{0000--0000/00\$00.00~\copyright~2021 IEEE}
% Remember, if you use this you must call \IEEEpubidadjcol in the second
% column for its text to clear the IEEEpubid mark.
\maketitle
\begin{abstract}
Deep neural networks (DNNs) have been widely used in many artificial intelligence (AI) tasks. However, deploying them %DNNs
brings significant challenges due to %when considering
the huge cost of memory, energy, and computation. %To save the cost, model compression techniques have been widely explored and used for memory saving and compute acceleration.
To address these challenges, researchers have developed various model compression techniques such as model quantization and model pruning. % to save memory and speed up computation.
Recently, there has been a surge in research of
% In the past few years, a wide spread of
compression methods %have been proposed
to achieve model efficiency while retaining the performance.
Furthermore, more and more works focus on customizing the DNN hardware accelerators to better leverage the model compression techniques. %In addition to efficient computation, privacy is also a critical issue when deploying DNNs for preserving integrity, reliability, and availability. However, the amount of related work is incredibly vast and diverse.
In addition to efficiency, preserving security and privacy is critical for deploying DNNs.
%with integrity, reliability, and availability.
However, the vast and diverse body of related works can be overwhelming.
This inspires us to conduct a comprehensive survey on recent research toward the goal of high-performance, cost-efficient, and safe deployment of DNNs. %In this article, for the model compression methods, we first review the mainstream approaches, including model quantization, model pruning, knowledge distillation, and the optimization of non-linear operations. Then we introduce the recent advances in designing the hardware accelerators to adapt to the efficient model compression approaches. Thirdly, we explain how to secure the DNNs deployment using homomorphic encryption. Ultimately, we discuss several issues, such as hardware evaluation, generalization, and integration of various compression approaches. Finally, this article attempts to construct a big picture of efficient DNNs from the algorithm, hardware accelerators, and security perspective.
Our survey first covers the mainstream model compression techniques such as model quantization, model pruning, knowledge distillation, and optimizations of non-linear operations. We then introduce recent advances in designing hardware accelerators that can adapt to efficient model compression approaches. Additionally, we discuss how homomorphic encryption can be integrated to secure DNN deployment. Finally, we discuss several issues, such as hardware evaluation, generalization, and integration of various compression approaches. Overall, we aim to provide a big picture of efficient DNNs, from algorithm to hardware accelerators and security perspectives.
\end{abstract}
\begin{IEEEkeywords}
Network compression, network quantization, network pruning, knowledge distillation, homomorphic encryption, network acceleration.
\end{IEEEkeywords}
\section{Introduction}
\label{sec:introduction}
\IEEEPARstart{D}{eep} neural networks (DNNs) are currently the foundation of many modern artificial intelligence (AI) applications with superior performance. However, with millions or even billions of parameters \cite{chung2022scaling}, %DNNs bring fundamental challenges in deploying them to devices when considering the cost (e.g., energy, and latency).
deploying DNNs to devices presents fundamental challenges due to high costs such as computational and energy costs. %such as energy and latency.
In particular, in addition to the performance accuracy~\cite{sze2017efficient}, it is also essential to consider %we might consider
the latency or throughput, which determines if the DNN can run fast in real-time. Besides, we need to %might
take the following hardware-related factors into account: 1) energy and power, which determines the actual running cost; 2) area, which determines the actual design cost; and 3) storage, which determines if the DNN can be stored, or run with intermediate data. Finally, when deploying DNNs, privacy is critical to ensure secure deployment. % secure the deployment.
To deploy DNNs successfully, various computing architectures have been proposed to meet the above criteria. %given the above criteria.
Different computing architectures might need to address various challenges when deploying DNNs. For example, cloud computing aims to centralize the computing resources and lets the cloud not directly interfere with the end user. However, it faces the latency issue and the tremendous cost of maintaining large-scale cloud centers. Meanwhile, edge computing helps reduce the latency by running DNNs at the edge (IoT devices) with direct interference from the user. It has been widely used in many areas, including autonomous driving, smart agriculture, surveillance, etc. \cite{varghese2016challenges}. However, edge devices provide a limited resource for computation. Model compression techniques have been widely explored to meet the above criteria among various computing architectures.
\begin{table*}[tbp]
\centering
\scriptsize
\caption{summary of this article}
\vspace{-5pt}
\label{tab:summary}
\begin{tabular}{l|cc|l}
\hline
Preliminaries & \multicolumn{2}{c|}{Brief Descriptions of Efficient and Safe DNNs} & \multicolumn{1}{l}{Sections} \\ \hline
\multirow{16}{*}{Model compression} & \multicolumn{1}{c|}{\multirow{5}{*}{Quantization and Entropy Encoding}}
& Uniform and Non-uniform Quantization & Section \ref{sec:uniform_nonuniform} \\ \cline{3-4}
& \multicolumn{1}{c|}{}
& Mixed-precision Quantization & Section \ref{sec:mixed_precision} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Transform Quantization & Section \ref{sec:transform_quantization} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Quantization for Transformers & Section \ref{sec:quant_vi} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Entropy Encoding & Section \ref{sec:entropy_encoding} \\ \cline{2-4}
& \multicolumn{1}{c|}{\multirow{2}{*}{Network Pruning}}
& Pruning Strategies - Before, During, and After Training & Section \ref{subsec:pruning_schemes} \\ \cline{3-4}
& \multicolumn{1}{c|}{} &
Pruning Paradigms - Unstructured, Structured, and Semi-Structured Pruning & Section \ref{subsec:pruning_paradigms} \\ \cline{2-4}
% & \multicolumn{1}{c|}{} & Structured Pruning & Section \ref{sec:structured_pruning} \\ \cline{3-4}
% & \multicolumn{1}{c|}{} & Semi-structured Pruning & Section \ref{subsub:semi-pruning} \\ \cline{2-4}
& \multicolumn{1}{c|}{\multirow{2}{*}{Knowledge Distillation}}
& Structured Knowledge & Section \ref{sec:structured_knowledge} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Distillation Schemes & Section \ref{subsec:distillation_schemes} \\ \cline{2-4}
& \multicolumn{1}{c|}{\multirow{3}{*}{Non-linear Operations}} & Non-maximum Suppression & Section \ref{sec:nms} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Softmax & Section \ref{sec:softmax} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Activation functions & Section \ref{sec:activation_functions} \\ \cline{2-4}
& \multicolumn{1}{c|}{\multirow{2}{*}{NAS \& TinyML}}
& NAS & Section~\ref{subsec:nas}\\ \cline{3-4}
& \multicolumn{1}{c|}{} & TinyML & Section~\ref{subsec:tinyml}
\\ \hline
\multirow{7}{*}{Hardware Optimization} & \multicolumn{1}{c|}{\multirow{4}{*}{Accelerators on Linear Operations}} & Hardware Accelerating on CNNs and Transformers & Section \ref{sec:cnns_transformers} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & \begin{tabular}[c]{@{}c@{}}Hardware Accelerating after Quantization: \\ Mixed-Precision and Entropy Encoding\end{tabular} & Section \ref{sec:accelerator_quantization} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Hardware Accelerating after Pruning: Sparse Architecture & Section \ref{sec:accelerator_pruning} \\ \cline{2-4}
& \multicolumn{1}{c|}{\multirow{2}{*}{Accelerators on Non-linear Operations}} & Non-maximum Suppression & Section \ref{sec:accelerator_nms} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Softmax & Section \ref{sec:hardware_softmax} \\ \cline{2-4}
& \multicolumn{1}{c|}{Stochastic Computing Architecture }
& Stochastic Computing Architecture & Section \ref{subsec:stochastic_architecture} \\ \cline{1-4}
\multirow{3}{*}{Homomorphic Encryption} & \multicolumn{1}{c|}{\multirow{3}{*}{Homomorphic Neural Networks}} & Homomorphic Neural Networks & Section \ref{sec:ahe_hnn} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Compression on Homomorphic Neural Networks & Section \ref{sec:ahe_compression} \\ \cline{3-4}
& \multicolumn{1}{c|}{} & Hardware on Homomorphic Neural Networks & Section \ref{sec:ahe_accelerator} \\ \hline
% Challenges \& Future Trends & & & Section \\ \hline
% \multicolumn{3}{c|}{Challenges \& Future Trends} & Section \\ \hline
\end{tabular}
\vspace{-10pt}
\end{table*}
Model compression aims at the compact model to reduce the hardware cost. A smaller model requires fewer arithmetic operations and thus improves the inference speed, requires less memory bandwidth to fetch data and thus saves energy consumption, and requires less on-chip memory to store and, therefore, is less expensive. In particular, we review the mainstream approaches to compressing the DNNs, including model quantization, network pruning, and knowledge distillation, and the optimization of non-linear operations such as softmax and non-maximum suppression (NMS) as they play an essential role in deploying DNNs.
%Moreover, deploying a compressed model efficiently is challenging as existing hardware accelerators are optimized for uncompressed DNNs, resulting in a massive waste of computation cycles and memory bandwidth compared with running on compressed DNN models~\cite{deng2020model}.
In addition, effectively deploying a compressed model poses a challenge as current hardware accelerators are optimized for uncompressed DNNs, leading to significant wastage of computation cycles and memory bandwidth when running on compressed DNN models~\cite{deng2020model}.
Therefore, we need to co-design the algorithm and the hardware. This paper also presents an overview of the custom hardware accelerators, \eg, Field Programmable Gate Arrays (FPGAs) and Application-Specific Integrated Circuits (ASICs), which are designed for the compressed DNNs in terms of various network operations.
\IEEEpubidadjcol
%Besides efficient computation, privacy is critical for preserving integrity, reliability, and availability in third-party computing, especially cloud computing. Traditionally, encryption of confidential information was the standard solution before using the cloud model. It may protect user data privacy from a non-trustworthy third party but cannot support effective ciphertext computing. Conventional cryptosystems successfully protect stored data and data in transit but do not protect the data while decrypted to be processed.
In addition to efficient computation, privacy is crucial in maintaining the integrity, reliability, and availability of third-party computing, particularly in cloud computing. In the past, encryption of confidential information was the conventional approach before using the cloud model. While it may safeguard user data privacy from an untrustworthy third party, it cannot facilitate effective ciphertext computing. Conventional cryptosystems effectively protect stored data and data in transit, but they fail to secure the data while it is decrypted for processing.
Therefore, many researchers have been exploring Fully Homomorphic Encryption (FHE). FHE is a valuable capability in distributed computation and heterogeneous networking. In this survey, we present a comprehensive review of the latest FHE techniques for DNNs and how model compression helps in these scenarios.
Few surveys exist on deep neural compression\cite{deng2020model, mishra2023transforming}. However, these works mainly focus on existing models' compression and acceleration algorithms. For example, \cite{cheng2018model, mishra2020survey, neill2020overview, alqahtani2021literature} mainly focused on compression algorithms of existing models, including quantization and knowledge distillation. In addition to the compression algorithms, \cite{berthelier2021deep} also discussed neural architecture optimization. \cite{deng2020model} also discussed the hardware accelerators, especially on mixed-precision data and sparse architectures. In addition to these topics, our survey also discusses non-linear operation accelerators. \cite{mittal2021survey} mainly focused on hardware accelerators for Recurrent Neural Networks (RNNs). \cite{nan2019deep} explored the performances of various model compression techniques on the mobile platform. \cite{xu2022survey} surveyed many works in model compression and acceleration for pre-trained language models. \cite{gupta2022compression} discussed the compression algorithms for NLP tasks. \cite{mishra2023transforming} talked about the model compression techniques for IoT applications. Besides, some surveys exist on specific network compression techniques such as quantization \cite{guo2018survey, gholami2021survey} or network pruning \cite{xu2020convolutional, liu2020pruning, reed1993pruning}, architecture search \cite{elsken2019neural, ren2021comprehensive, wistuba2019survey}, knowledge distillation \cite{gou2021knowledge, wang2021knowledge}.
In summary, while some surveys focus on specific aspects of deep neural compression, there is no survey as our article on model compression from the perspective of algorithm-efficient, hardware-accelerating, and deployment-securing DNNs. These three aspects are critical for many real-world applications that require small model sizes, hardware acceleration, and secure environments to save memory, energy, and computation costs. Our contributions are summarised as follows:%Therefore, our article aims to address this gap in the literature and provides a comprehensive review of recent advances in these areas, which are critical for both advancing research and enabling economic impact. By synthesizing and analyzing the latest findings and approaches, our survey aims to inform researchers and practitioners about the current state of the field and to guide future efforts toward developing more efficient, faster, and more secure deep neural networks.
% \textcolor{red}{The survey covers three perspectives, namely algorithm-efficient, hardware-accelerating, and deployment-securing DNNs. The contributions of this article could be stated more clearly, e.g., through making a list to respectively explain the importance of these three aspects with detailed descriptions.}
\begin{itemize}
\item We present a comprehensive overview on neural network compression techniques. Efficient compression algorithms are essential for reducing memory and storage footprint of deep learning models.
%, making them suitable for resource-constrained devices.
They enable faster, more energy-efficient inference, reduce bandwidth requirements, and offer adaptability to different deployment scenarios by facilitating efficient fine-tuning, thus optimizing model deployment.
\item We also compile the literature on accelerating DNNs considering the hardware constraints. Hardware-aware accelerators play a crucial role in optimizing the use of specialized hardware for model inference, ensuring efficient hardware utilization, scalability, energy efficiency, and low latency.
% They offer the flexibility to tailor hardware solutions for specific performance needs.%, making them valuable for a wide range of applications, from edge computing to data centers.
\item To secure computations on encrypted data, we review Homomorphic Encryption (HE) for applications like Homomorphic Neural Networks (HNNs) in privacy-sensitive domains. In particular, we focus on model compression and hardware accelerators, specifically for HE.
% However, challenges arise, such as limited support for floating-point arithmetic and increased computational intensity. Therefore, we reviews literatures on model compression and hardware accelerators, specifically for HE.
% in the context of securing neural network deployments. Deployment-security algorithms, discussed in the literature, emphasize safeguarding models by preserving data privacy, defending against adversarial attacks, and ensuring regulatory compliance. These algorithms also bolster model integrity, fortifying against tampering and enhancing resilience in the deployment environment. Their significance lies in upholding the trustworthiness and security of deployed machine learning models.
\item We summarize the challenges for compressing, accelerating and securing DNNs and discuss several future directions in the promising field.
\end{itemize}
% Model Compression and Acceleration for Deep Neural Networks
The organization of this article is shown in Table \ref{tab:summary} and is summarized as follows. In Section \ref{sec:compression}, we describe the model compression techniques, including quantization, pruning, knowledge distillation, and optimization on non-linear operations. Section \ref{sec:compression_hardware} discusses how the hardware accelerates the compressed model. Section \ref{sec:compression_ahe} presents an overview of the privacy-preserving technique with Homomorphic Encryption (HE). Finally, Section \ref{sec:challenges} presents the challenges and opportunities of model compression and hardware-software co-design in real-world applications, before concluding the paper.
% 1. accuracy 2. efficiency 3. big models.
% including ResNet, DenseNet, MobileNet, EfficientNet and Transformer, BERT, GPT.
% Neural networks take their inspiration from the notion that a neuron's computation involves a weighted sum of the input values. These weighted sums correspond to the value scaling performed by the synapses and the combining of those values in the neuron. Furthermore, the neuron doesn't just output that weighted sum, since the computation associated with a cascade of neurons would then be a simple linear algebra operation. Instead, there is a functional operation within the neuron that is performed on the combined inputs. This operation appears to be a non-linear function that causes a neuron to generate an output only if the inputs cross some threshold. Thus by analogy, neural networks apply a non-linear function to the weighted sum of the input values. We look at what some of those non-linear functions are in Section~\ref{sssec:non-linearity}.
% Fig.~\ref{fig:DNN_101_ns} shows a diagrammatic picture of a computational neural network. The neurons in the input layer receive some values and propagate them to the neurons in the middle layer of the network, which is also frequently called a `hidden layer'. The weighted sums from one or more hidden layers are ultimately propagated to the output layer, which presents the final outputs of the network to the user. To align brain-inspired terminology with neural networks, the outputs of the neurons are often referred to as \emph{activations}, and the synapses are often referred to as \emph{weights} as shown in Fig.~\ref{fig:DNN_101_ns}. We will use the activation/weight nomenclature in this article.
% Fig.~\ref{fig:DNN_101_weights} shows an example of the computation at each layer:
% $y_{j}=f(\sum\limits_{i=1}^{3} W_{ij} \times x_{i} + b)$, where $W_{ij}$, $x_{i}$ and $y_{j}$ are the weights, input activations and output activations, respectively, and \emph{$f(\cdot)$} is a non-linear function described in Section~\ref{sssec:non-linearity}. The bias term $b$ is omitted from Fig.~\ref{fig:DNN_101_weights} for simplicity.
% We really don't describe all the nuances of this figure yet,
% and it uses the neuron synapse nomenclature.
% \begin{figure}
% \centering{
% \subfigure[Neurons and synapses]{
% \includegraphics[width=0.4\linewidth]{figures/DNN_101_ns.pdf}
% \label{fig:DNN_101_ns}
% }\hfill
% \subfigure[Compute weighted sum for each layer]{
% \includegraphics[width=0.52\linewidth]{figures/DNN_101_weights.pdf}
% \label{fig:DNN_101_weights}
% }\\
% }
% \caption{ Simple neural network example and terminology (Figure adopted from~\cite{cs231n}).
% }
% \label{fig:neuralnetwork}
% \end{figure}
% Within the domain of neural networks, there is an area called \emph{deep learning}, in which the neural networks have more than three layers, i.e., more than one hidden layer. Today, the typical number of network layers used in deep learning ranges from five to more than a thousand. In this article, we will generally use the terminology \emph{deep neural networks (DNNs)} to refer to the neural networks used in deep learning.
% DNNs are capable of learning high-level features with more complexity and abstraction than shallower neural networks. An example that demonstrates this point is using DNNs to process visual data. In these applications, pixels of an image are fed into the first layer of a DNN, and the outputs of that layer can be interpreted as representing the presence of different low-level features in the image, such as lines and edges. At subsequent layers, these features are then combined into a measure of the likely presence of higher-level features, e.g., lines are combined into shapes, which are further combined into sets of shapes. And finally, given all this information, the network provides a probability that these high-level features comprise a particular object or scene. This deep feature hierarchy enables DNNs to achieve superior performance in many tasks.
\section{Neural Network Compression}
\label{sec:compression}
\noindent We briefly introduce the general deep neural networks (DNNs) in Appendix A and then we introduce advanced neural network compression techniques. We classify existing approaches into four main categories:
\begin{itemize}
\item Quantization and Entropy Encoding: Quantization attempts to reduce the bit width of the data % through
in a DNN, aiming at reducing the model size for memory saving and simplifying the operations for accelerating computation.
\item Network Pruning: %The parameter pruning and sharing-based methods explore the redundancy in the model parameters and try to remove the redundant and non-critical ones.
This category of methods aims to explore redundancy in the model parameters, eliminating non-critical and redundant parameters. This results in a more efficient model.
\item Knowledge Distillation (KD): The KD methods learn a distilled model by training a more compact neural network to reproduce the output of a larger network.
\item Hardware-friendly Non-linear Operations: %The design of hardware-friendly non-linear operations aims to design hardware-friendly procedures (e.g., linear functions) as a substitute for non-linear operations to shorten the hardware design cycle and speed up the inference.
This category of methods aims to design hardware-friendly procedures (such as linear functions) to substitute non-linear operations, shortening the hardware design cycle and speeding up inference, which is better suited for deployment on hardware platforms.
\end{itemize}
%- Backbone optimization (architecture search)
%- Quantization
%- Pruning
%- KD
%- Low-rank, Sparse learning
\subsection{Quantization and Entropy Coding}
\label{subsec:quantization}
Benefiting from discretizing the parameters, quantized DNNs have % less
reduced memory cost and lower computation, as discretized parameters, after quantization, can be stored and calculated more efficiently.
Moreover, %utilizing
by taking advantage of the peaky distribution of discretized values, entropy coding can compress the size of quantized parameters, leading higher compression ratio~\cite{han16iclr}.
This subsection will review the main quantization and entropy coding approaches. In particular,
first, we discuss the quantization approaches built by uniform quantizer and non-uniform quantizer.
Second, we introduce specific quantization schemes developed for neural network compression, including mixed-precision and transform quantization.
Finally, we discuss the entropy coding techniques. %works.
\subsubsection{Uniform Quantizer and Non-Uniform Quantizer}
\label{sec:uniform_nonuniform}
\begin{figure}[tbp]
\centering
\includegraphics[width=0.45\textwidth]{figs_quantization/quantization.pdf}
\caption{(a) Quantization with uniform quantizer: the space is divided into equidistant quantization centroids. (b) Quantization with non-uniform quantizer: the space can be divided into polygonal cells based on data distribution. (c) Mixed-precision quantization: different dimensions (layers, channels, or kernels in the case of neural network quantization) are quantized with different bit widths. (d) Transform quantization with mixed precision: the space is first rotated and then quantized with mixed precision.}
\label{fig:quant}
\vspace{-10pt}
\end{figure}
Quantization approaches are ususally built based on uniform quantizers and non-uniform quantizers.
A uniform quantizer discretizes parameters onto equidistant quantization values as shown in Fig.~\ref{fig:quant}(a), while a non-uniform quantizer maps parameters onto irregular values decided by the distribution as shown in Fig.~\ref{fig:quant}(b).
A non-uniform quantizer can reduce quantization error by offering greater flexibility in selecting quantization values. Nevertheless, the need for look-up tables to store discretized quantization values introduces additional implementation overheads.
%Figure * illustrates an example of uniform quantization and non-uniform quantization.
Uniform quantizers are commonly employed in various studies due to their hardware-friendly nature. For instance, LQ-Nets \cite{Zhang18eccv} proposed the joint training of a quantized DNN and its associated quantizers, ensuring compatibility with bit operations. INQ \cite{inq} focused on converting pre-trained neural networks into low-precision versions, restricting weights to power of two or zero. DOREFA-NET \cite{dorefa} suggested training neural networks with low bit width weights, activations, and gradients. Additionally, numerous works utilize uniform quantizers to quantize DNNs to very low bit widths (1-bit or 2-bits).
% Some low-precision quantization approaches which quantizing neural networks to very low bit width (1-bit or 2-bits) adopt the uniform quantizers.
% %As such, most quantization works adopt uniform quantizers.
% %The critical element for the success of a uniform quantizer is the setting of step size and range.
% For instance, Hubara et al. \cite{bwn} proposed a training approach to train quantized weights and activations with binary format. Rastegari et al. \cite{xnor} presented XNOR-Nets which approximate convolutions using primarily binary operations.
For example, Li \textit{et al.} \cite{Li16nips} introduced Ternary Weight Networks (TWNs) with weights constrained to 1, 0, and -1 (2-bits). Additional relevant literature on uniform quantizers can be explored in Appendix B.
% Finally, Zhu \textit{et al.} \cite{zhu2016trained} proposed Trained Ternary Quantization (TTQ) which trains quantized networks with low precision (2-bits) from scratch.
% %Overall, the above methods are all low-precision quantization approaches quantizing neural networks to very low bit width (1-bit or 2-bits) to reduce memory consumption and associated computational costs.
% Many quantization approaches with uniform quantizers are proposed to quantize neural networks to any precision.
% For example, LQ-Nets \cite{Zhang18eccv} proposed jointly training a quantized, bit-operation-compatible DNN and its associated quantizers.
% INQ \cite{inq} targeted to convert pre-trained neural networks into low-precision versions whose weights are constrained to be either power of two or zero.
% DOREFA-NET \cite{dorefa} proposed to train neural networks with low bit-width weights, activations and gradients.
% %Banner \textit{et al.} \cite{post4bits} proposed a practical 4-bit post-training quantization approach, which does not involve training the quantized model.
% Nagel \textit{et al.} \cite{datafree} proposed data-free quantization through weight equalization and bias correction.
% Gong \textit{et al.} \cite{DSQ} proposed Differentiable Soft Quantization (DSQ) to bridge the gap between the full-precision and low-bit networks.
% Chen \textit{et al.} \cite{layerwise} presented a layer-wise quantization technique for deep neural networks, which requires very limited training data (only 1\% of the original dataset).
% Zhuang \textit{et al.} \cite{Zhuang2018cvpr} proposed two-stage optimization, namely progressive optimization and joint optimization, to train low-precision weights and activations.
% Jung el al. \cite{QIL} proposed learning to quantize activations and weights via a trainable quantizer that transforms and discretizes them.
% SAT \cite{SAT} directly scaled down weights in the last layer to alleviate over-fitting.
% Esser \textit{et al.} \cite{esser2019learned} involves estimating and scaling the task loss gradient at the quantizer step size of each weight and activation layer, enabling it to be learned simultaneously with other network parameters. %proposed to
% %estimate and scale the task loss gradient at each weight and activation layer’s quantizer step size, such that it can be learned in conjunction with other network parameters.
% Finally,
% Zhuang \textit{et al.} \cite{Auxi} proposed a method involves training a network with low-precision using an auxiliary module that operates at full-precision. %to train the low-precision network with a full-precision auxiliary module.
On the other hand, quantization with a non-uniform quantizer involves k-means or hashing tricks and may additionally involve the Discrete Cosine Transform (DCT) and residual quantization.
In particular, Gong \textit{et al.} \cite{pq} investigated a vector quantization method to compress the weights by applying k-means clustering and conducting product quantization.
HashNets \cite{hash} used a low-cost hash function to randomly group connection weights into hash buckets, and all links within the same hash bucket are quantized into a single parameter value.
Chen \textit{et al.} \cite{frequency} proposed to convert weights to the frequency domain with a DCT and used hash functions to group frequency parameters into hash buckets, where parameters assigned to the same hash bucket share a single quantization value.
Stock \textit{et al.} \cite{revisit} introduced a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights, with only a set of unlabelled data needed at quantization time.
APoT \cite{APoT} proposed an efficient nonuniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks.
SYQ \cite{Faraone2018arxiv} introduced a quantization method to reduce information loss by learning a symmetric codebook for particular weight subgroups.
Table I in Appendix B provides a comprehensive summary and comparative analysis of various quantization methods, including those utilizing both uniform and non-uniform quantizers.
\subsubsection{Mixed-Precision Quantization}
\label{sec:mixed_precision}
Mixed-precision quantization as shown in Fig.~\ref{fig:quant}(c) uses %different
distinct bit widths to quantize weights and activations of different layers, channels, or kernels.
Because weights and activation in various layers, channels, or kernels %react differently to quantization,
exhibit different levels of sensitivity to quantization, %instead of using equal bit widths to quantize all of them,
and thus allocating different bit widths %in different parts
%to different sections
is more reasonable and %brings higher quantization precision.
can deliver higher accuracy than uniform quantization.
Mixed-precision quantization can be categorized as layer-wise, channel-wise, and kernel-wise approaches based on the minimal elements %to decide bit width.
used to determine the bit width.
ReLeQ~\cite{releq} introduced an end-to-end deep reinforcement learning (RL) framework to automate the process of discovering quantization bit width.
Alternatively, HAQ~\cite{wang19cvpr} leveraged RL to determine quantization bit width but employed a hardware simulator to generate direct feedback signals to the RL agent.
DNAS~\cite{dnas} presented a differentiable neural architecture search framework to explore the hyper-parameter space of quantization bit width.
Differentiable Quantization (DQ)~\cite{dq} learned quantizer parameters, such as step size and range, through training with straight-through gradients and then inferred quantization bit width based on the learned step size and range.
Hessian AWare Quantization (HAWQ)~\cite{HAWQ} introduced a second-order quantization method to select the quantization bit width for each layer based on the Hessian spectrum.
%Adaptive Loss-aware Quantization (ALQ) \cite{ALQ} directly minimized network loss w.r.t. quantized weights and used network loss to decide quantization bit width.
All the above methods deploy a layer-wise bit allocation scheme.
\begin{table*}[!htbp]
\centering
\scriptsize
\begin{threeparttable}
\caption{A summary of mixed-precision quantization approaches for the results on ImageNet.
%Our approach optimizes the quantization bit width of channels and improves system-level performance on hardware platforms. We provide a fast linear-complexity method to solve the bit allocation problem.
}
\vspace{-5pt}
\begin{tabular}{l | c | c | c | c | c }
\hline
% \multirow{2}{*}{Method} & Bit Allocation & \multirow{2}{*}{Model} & \multirow{2}{*}{Weight} & \multirow{2}{*}{Activation} & \multirow{2}{*}{\textcolor{blue}{Accuracy (Original)}} \\
% & Scheme & & & & \\
Methods & Bit Allocation Scheme & Model & Weight & Activation & Accuracy (Original) \\
\hline
\multirow{2}{*}{ReLeQ \cite{releq}} & \multirow{2}{*}{Layer-wise} & AlexNet & 5 Bits MP & 5 Bits MP & --- (---) \tnote{a} \\ %-0.1\%\tnote{a}
& & MobileNet-V2 & 6 Bits MP & 6 Bits MP & --- (---) \tnote{a} \\ % -0.3\%
\hline
\multirow{2}{*}{HAQ \cite{wang19cvpr}} & \multirow{2}{*}{Layer-wise} & MobileNet-V2 & 4 Bits MP & 4 Bits MP & 67.0\% (71.8\%) \\
& & ResNet-50 & 2 Bits MP & 32 Bits & 70.6\% (76.2\%) \\
\hline
DNAS \cite{dnas} & Layer-wise & ResNet-18 & 1.5 Bits MP & 32 Bits & 69.6\% (71.0\%) \\
\hline
\multirow{2}{*}{DQ \cite{dq}} & \multirow{2}{*}{Layer-wise} & MobileNet-V2 & 4 Bits MP & 4 Bits MP & 70.6\% (70.2\%) \\
& & ResNet-18 & 4 Bits MP & 4 Bits MP & 70.7\% (70.3\%) \\
\hline
\multirow{3}{*}{HAWQ \cite{HAWQ}} & \multirow{3}{*}{Layer-wise} & GoogleNet & 2 Bits MP & 4 Bits MP & 75.5\% (77.5\%) \\
& & ResNet-50 & 2 Bits MP & 4 Bits MP & 75.5\% (77.4\%) \\
& & SqueezeNet & 3 Bits MP & 8 Bits MP & 68.0\% (69.4\%) \\
\hline
ALQ \cite{ALQ} & Kernel-wise & ResNet-18 & 1 Bit MP & 2 Bits MP & 63.2\% (69.8\%) \\
\hline
\multirow{3}{*}{Banner \textit{et al.} \cite{post4bits}} & \multirow{3}{*}{Channel-wise} & VGG & 4 Bits MP & 4 Bits MP & 70.5\% (71.6\%) \\
& & GoogleNet & 4 Bits MP & 4 Bits MP & 66.4\% (77.2\%) \\
& & ResNet-50 & 4 Bits MP & 4 Bits MP & 73.8\% (76.1\%) \\
\hline
\multirow{2}{*}{FracBits \cite{FracBits}} & \multirow{2}{*}{Kernel-wise} & MobileNet-V2 & 3 Bits MP & 3 Bits MP & 68.2\% (71.8\%) \\
& & ResNet-18 & 3 Bits MP & 3 Bits MP & 69.8\% (70.2\%) \\
\hline
DMBQ \cite{DMBQ} & Channel-wise & ResNet-18 & 1 Bit MP & 2 Bits MP & 63.5\% (70.3\%) \\
\hline\multirow{3}{*}{AutoQ \cite{autoq}} & \multirow{3}{*}{Kernel-wise} & ResNet-50 & 4 Bits MP & 4 Bits MP & 74.5\% (74.8\%) \\
& & SqueezeNet & 4 Bits MP & 4 Bits MP & 56.5\% (56.9\%) \\
& & MobileNet-V2 & 4 Bits MP & 4 Bits MP & 70.8\% (71.1\%) \\
\hline
\end{tabular}\label{table:mixed}
\begin{tablenotes}
\item[a] ReLeQ \cite{releq} did not report the detailed accuracy and only reported the accuracy gap between original and compressed models (\textit{i.e.}, -0.1\% and -0.5\%) on AlexNet and MobileNet-V2 respectively.
% \item[b] \textcolor{blue}{ReLeQ \cite{releq} only reports a -0.5\% accuracy gap between original and compressed models, without individual accuracy.}
\end{tablenotes}
\end{threeparttable}
\vspace{-10pt}
\end{table*}
\begin{table*}[!htbp]
\scriptsize
\centering
\caption{A summary of compression approaches with entropy coding for the results on ImageNet.
%Our approach optimizes the quantization bit width of channels and improves system-level performance on hardware platforms. In addition, we provide a fast linear-complexity method to solve the bit allocation problem.
}
\vspace{-5pt}
% \resizebox{\columnwidth}{!}{
\begin{tabular}{ l c | c | c | c }
\hline
% \multirow{2}{*}{Method} & \multirow{2}{*}{Entropy Coding} & \multirow{2}{*}{Model} & \multirow{2}{*}{Original} & Compression & \multirow{2}{*}{Accuracy} \\
Methods & Entropy Coding & Model & Compression Ratio & Accuracy (Original) \\
\hline
\multirow{2}{*}{Deep Compression~\cite{han16iclr}} & \multirow{2}{*}{Huffman Coding (FVL)} & AlexNet & $35 \times$ & 57.2\% (57.2\%) \\
& & VGG & $49 \times$ & 68.2\% (68.5\%) \\
\hline
\multirow{2}{*}{Coreset \cite{coresets}} & \multirow{2}{*}{Huffman Coding (FVL)} & AlexNet & $55 \times$ & 57.1\% (57.2\%) \\
& & VGG & $238 \times$ & 68.1\% (68.9\%) \\
\hline
\multirow{3}{*}{DeepCABAC \cite{DeepCABAC}} & \multirow{3}{*}{Arithmetic Coding (FVL)} & VGG & $64 \times$ & 69.4\% (69.4\%) \\
& & ResNet-50 & $17 \times$ & 74.1\% (76.1\%) \\
& & MobileNet-V1 & $8 \times$ & 66.2\% (70.7\%) \\
\hline
EPR \cite{EPR} & Arithmetic Coding (FVL) & ResNet-50 & $19 \times$ & 74.0\% (75.0\%) \\
\hline
\multirow{2}{*}{RDOC \cite{zhe2021rate}} & \multirow{2}{*}{Tunstall Coding (VFL)} & ResNet-50 & $20 \times$ & 75.0\% (75.5\%) \\
& & MobileNet-V2 & $10 \times$ & 70.2\% (71.0\%) \\
\hline
\end{tabular}
% }
\label{table:coding}
\vspace{-5mm}
\end{table*}
Compared with layer-wise approaches, channel-wise or kernel-wise approaches can achieve higher accuracy because of their fine-grained quantization.
To this end, Khoram \textit{et al.} \cite{khoram18iclr} proposed Adaptive Quantization (AQ) which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized.
Qu \textit{et al.} \cite{ALQ} proposed Adaptive Loss-aware Quantization (ALQ) which achieves an average bit width below one-bit without significant loss in inference accuracy.
Meanwhile, Banner \textit{et al.} \cite{post4bits} adopted channel-wise bit allocation to improve quantization precision and provided an analytic solution for quantization bit width, %assuming that parameters obey certain distributions.
assuming certain distributions of parameters.
FracBits \cite{FracBits} generalized quantization bit width to an arbitrary real number to make it differentiable and learned channel-wise bit allocation during training.
Distribution-aware Multi-Bit Quantization (DMBQ) \cite{DMBQ} proposed a loss-guided bit width allocation strategy to adjust the bit width of weights and activations channel-wisely.
%Note that PTQ, FracBits, and DMBQ did not consider performance's impact on hardware platforms when designing their approaches.
Lastly, AutoQ \cite{autoq} presented a hierarchical deep reinforcement learning approach to find the quantization bit width of channels %and optimize hardware metrics (e.g., latency and energy) simultaneously.
while simultaneously optimizing hardware metrics such as latency and energy.
Table \ref{table:mixed} summarizes the mixed-precision quantization approaches discussed above.
In Table~\ref{table:mixed}, the bit width of mixed-precision quantization approaches is the average bit width across all the layers. For example, if a two-layer model quantizes one layer to 4 bits and the other one to 8 bits, then the average bit width of this mixed-precision quantization is (4+8)/2 = 6 bits. The variance in reported accuracies for CNN architectures like MobileNet-V2 across literature~\cite{dq, autoq} complicates the evaluation of quantization's impact on model performance. This inconsistency could be due to the differences in dataset pre-processing, training protocols, or minor architectural modifications. To accurately evaluate quantization effects, it requires a standardized approach towards reporting the performance metrics of both original and quantized models. This would facilitate a more precise isolation of quantization's effects from other influencing factors. As a result, emphasizing the relative changes in model accuracy resulting from quantization—rather than focusing solely on absolute accuracy figures—offers a more robust measure for evaluating the effectiveness of quantization techniques.
Such an approach would help in better understanding the trade-offs between model efficiency and performance in the context of quantization, thereby contributing to more informed decisions in model optimization and deployment.
Therefore, we present the accuracies of the original and compressed models to demonstrate the relative change in model accuracy.
\subsubsection{Transform Quantization}
\label{sec:transform_quantization}
Transform quantization decorrelates and quantizes the weights, which first rotates the space and then quantizes the data after rotation with mixed precision as shown in Fig.~\ref{fig:quant}(d).
Because the redundancy between the dimension of weights can be removed after decorrelation, transform quantization could quantize weights to lower bit width in the transformed space.
% Recently, Young \textit{et al.} \cite{young2021transform} proposed a unified framework that unified quantization and dimensionality reduction
% (decorrelation) techniques to enable low bit-rate quantization of deep neural networks.
% In particular, a theory of rate and distortion for neural network quantization was introduced, which poses optimum quantization as a rate–distortion optimization problem.
% They then derived optimal End-to-end Learned Transform (ELT) to solve optimal bit-width allocation following decorrelation.
% Experiments demonstrated that transform quantization advances
% state of the art in neural network quantization.
A unified framework was proposed \cite{young2021transform} to enable low bit-rate quantization of deep neural networks by combining quantization and dimension reduction (decorrelation) techniques. The framework introduces a theory of rate and distortion for neural network quantization, which formulates optimum quantization as a rate-distortion optimization problem. Optimal End-to-end Learned Transform (ELT) was then derived to solve optimal bit width allocation following decorrelation. Experimental results demonstrated that transform quantization significantly improved the state-of-the-art in neural network quantization.
A similar idea was explored in other neural network compression works.
% Wang \textit{et al.} \cite{dct} proposed to compress neural networks first by spatially decorrelating their weights with the DCT, followed by vector quantization to the decorrelated weights.
% In parallel with the DCT techniques, other works sought to reduce the dimensionality of neural network layers via PCA and related techniques.
% Denton \textit{et al.} \cite{svd} applied the SVD across kernels in convolution layers or across the rows and columns of weight matrices for fully-connecting layers
% projecting input or output to low-rank subspaces.
% Zhang \textit{et al.} \cite{svd_p} extended this method using the generalized SVD, while
% Kim \textit{et al.} \cite{svd_p2} applyied SVD on all four axes of convolutional tensors.
% Furthermore, Li \textit{et al.} \cite{svd_p3} partitioned the kernels into subsets, and then used SVD on each subset of kernels separately.
Wang \textit{et al.}~\cite{dct} proposed a technique to compress neural networks that involved spatially decorrelating the weights using the DCT, followed by vector quantization on the decorrelated weights. Alongside the DCT approach, other methods aimed to reduce the dimension of neural network layers using principal component analysis (PCA) and related techniques. \cite{svd} employed singular value decomposition (SVD) to perform low-rank projection across kernels in convolution layers or across the rows and columns of weight matrices for fully-connected layers. Similarly, \cite{svd_p} extended this method using the generalized SVD. \cite{svd_p2} further applied SVD on all four axes of convolutional tensors. Additionally, Li \textit{et al.}~\cite{svd_p3} partitioned the kernels into subsets and performed SVD on each subset of kernels separately.
\subsubsection{Quantization for Transformers}
\label{sec:quant_vi}
Transformers~\cite{li2021bossnas,chen2021autoformer} now stand as the leading architecture for a range of computer vision tasks, encompassing image classification, object detection, and semantic segmentation.
However, achieving high-accuracy performance in transformers comes with the trade-off of substantial computational complexity, often involving tens of millions or even more parameters in a model.
The substantial number of parameters poses a significant challenge for deploying transformers on mobile devices and hinders their practical applications.
As a result, the quantization of transformers becomes more important and has received attentions.
\cite{bondarenko2021understanding} observed that activations of the transfer have high dynamic activation ranges and structured outliers and thus propose a per-embedding-group quantization scheme.
\cite{dettmers2022gpt3} introduced a mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication.
Liu \textit{et al.}~\cite{liu2021post} recently introduced post-training quantization for transformers, integrating ranking-aware loss, bias correction, and mixed-precision techniques.
In particular, the inclusion of a ranking-aware loss aimed to preserve the relative order of quantized attention maps within transformers. Additionally, a bias correction method was implemented to adjust the distribution of quantized weights and activations. The exploration of mixed-precision quantization involved assigning more bit widths to layers with higher sensitivity, thus ensuring optimal performance retention.
% Therefore it is better to jointly optimize the quantization of all the layers to reduce the overall output error caused by quantization.
% Different from [*], we propose a unified quantization framework to jointly optimize the quantization of the network.
Ding \textit{et al.} \cite{ding2022towards} went a step further in enhancing by refining the calibration process following linear layers.
\subsubsection{Entropy Coding}
\label{sec:entropy_encoding}
%Entropy coding can further compress quantized values in a lossless way.
Quantized values can be further compressed using entropy coding in a lossless manner.
There are two types of entropy coding methods
\cite{taubmanjpeg2000}: Fixed-to-Variable Length (FVL) entropy coding and Variable-to-Fixed Length (VFL) entropy coding.
%Fixed-to-Variable Length (FVL) entropy coding encodes a fixed number of symbols into variable-length codes. Typical FVL coding methods include arithmetic coding and Huffman coding.
FVL coding maps a fixed number of symbols to variable-length codes, %typically
using arithmetic or Huffman coding.
FVL entropy coding methods achieve high coding efficiency (high compression ratio), but their decoding process is inefficient as it decodes codes bit by bit, leading to slower processing.
%. Still, decoding the FVL coder is inefficient because it needs to decode the codes bit-by-bit, which could slow down the processing since decoding should be done on the fly.
%Unlike FVL entropy coding, Variable-to-Fixed Length (VFL) entropy coding encodes a variable number of symbols into the fixed length of codes.
%As such, VFL coders can decode byte-by-byte, and thus the decoding is much faster than FVL coders.
%Tunstall coding \cite{Tunstall1067phd} is a standard VFL encoding method.
It has comparable coding efficiency as Huffman coding, with $10\times$ to $15\times$ faster decoding speed.
On the other hand, VFL coding maps a variable number of symbols to fixed-length codes, enabling byte-by-byte decoding, resulting in much faster decoding compared to FVL coding. Tunstall coding \cite{Tunstall1067phd} is a popular VFL coding method that achieves comparable coding efficiency as Huffman coding but offers decoding speeds that are $10\times$ to $15\times$ faster.
Several studies in the literature %adopted entropy coding in their approaches
have utilized entropy coding to compress DNNs.
%Deep Compression \cite{han16iclr} proposed a compression framework
One such approach is the Deep Compression framework \cite{han16iclr}, which includes
pruning, quantization, and Huffman coding.
Similarly, Coreset-Based \cite{coresets} Compression coupled quantization with Huffman coding and exploited the weight redundancies to improve the compression ratio.
As mentioned above, the decoding of Huffman coding is inefficient, %which could slow down the processing at the inference stage.
potentially slowing down the inference stage.
To address this, DeepCABAC \cite{DeepCABAC} proposed a context-adaptive binary arithmetic coding to compress DNNs.
EPR \cite{EPR} represented the weights of neural networks in a latent space and used arithmetic coding to compress the representation. To address this,
DeepCABAC and EPR employed arithmetic coding to obtain a higher compression ratio, although arithmetic coding has the most increased computational complexity, which is very difficult to implement.
RDOC \cite{zhe2021rate} adopted Tunstall coding to obtain superior compression capability and fast decoding speed.
Table \ref{table:coding} summarizes the works that utilize entropy coding for neural network compression.
% \begin{table*}[ht]
% \centering
% \scriptsize
% \caption{Classification of different pruning algorithms by their type, stage of doing pruning, and whether they are structured (denoted by 'S') or unstructured (denoted by 'U'). More comprehensive summary of Pruning at initialization scheme can be found in~\cite{ijcai2022p786}}
% \vspace{-5pt}
% \begin{tabular}{p{2.1cm}|p{3.5cm}|p{6.3cm}|p{4.5cm}}
% %\toprule
% \hline
% Pruning methods & Pruning at initialization & Pruning with training & Pruning after training (one-shot) \\
% \hline
% %\midrule
% First or second order methods & & Optimal Brain Damage \cite{NIPS1989_250} [U], WoodFisher \cite{woodfisher} [U], MFAC \cite{mfac} [U], \textcolor{blue}{HALP~\cite{shen2022structural}, Taylor~\cite{molchanov2019importance}, RD~\cite{xu2023efficient}} & Optimal Brain Surgeon \cite{NIPS1992_647} [U], \textcolor{blue}{WoodFisher \cite{woodfisher} [U]} \\
% \hline
% %\midrule
% Gradient-based & SNIP \cite{lee2018snip} [U], GraSP \cite{Wang2020Picking} [U] & \textcolor{blue}{Learning-Compression~\cite{carreira2018learning}, ProbMask~\cite{zhou2021effective} [U], Hinge~\cite{li2020group} [S]} & \textcolor{blue}{OTO~\cite{chen2021only} [S]} \\
% %\midrule
% \hline
% Sensitivity or custom objectives based & FORCE \cite{jorge2021progressive} [U] & DPF \cite{Lin2020Dynamic} [S,U], DST \cite{LIU2020Dynamic} [U], Molchanov \textit{et al.} \cite{molchanov2016pruning} [U], \textcolor{blue}{SCOP~\cite{tang2020scop} [S]} & \\
% %\midrule
% \hline
% % Similarity-based & & & Srinivas \textit{et al.} \cite{Srinivas_2015} [S] \\
% % \hline
% %\midrule
% Regularization based & & STR \cite{pmlr-v119-kusupati20a} [S,U], Savarese \textit{et al.} \cite{ContinuousSparsification2020} [U], Wang \textit{et al.} \cite{wang2021neural} [S,U] & \\
% %\midrule
% \hline
% Magnitude based & & RigL \cite{pmlr-v119-evci20a} [S,U], LAP \cite{Park2020LookaheadAF} [S,U], GMP \cite{Zhu2018ToPO} [U], LAMP \cite{lee2021layeradaptive} [U], Global MP \cite{globalmp} [U], \textcolor{blue}{DSR~\cite{ICML-2019-MostafaW} [U], SM\cite{sparse_momentum} [U], Han \textit{et al.}\cite{han2015learning} [U]} & Global MP \cite{globalmp} [S,U] \\ %\textcolor{red}{\cite{?}}
% \hline
% %\midrule
% RL based & & \textcolor{blue}{PuRL \cite{gupta2020learning} [U], AMC \cite{he2018amc} [S,U], RNP \cite{NIPS2017_6813} [S]} & \\
% \hline
% %\bottomrule
% \end{tabular}
% \label{table:pruning_methods}
% \vspace{-5pt}
% \end{table*}
% \noindent
In summary, different quantization approaches have advantages and disadvantages regarding the computational complexity, quantization loss, hardware implementation, etc. In uniform quantization, the space is divided
into equidistant quantization centroids. Such quantization scheme is straight-forward for implementation without additional overhead involved, but the quantization loss is relatively high. Non-uniform quantization is able to achieve less quantization loss because the quantizer is more flexible and can divide the space into different clusters based on the distributions of data. The drawbacks of non-uniform quantization is that it requires multiple code-books to store the quantization centroids. Mixed-precision quantization uses different bit widths to quantize different parts of parameters. It is more reasonable than uniform quantization because different parts of parameters may react distinctively to quantization. However, specific hardware is needed to support the computation with mixed precision (\textit{e.g.}, 4-bit vs. 8-bit). Different with all the others, transform quantization firstly rotates the space before quantization to remove redundancy between the dimensions. As a result, lower bit widths are able to be achieved after the space transform. The problem is that a large rotation matrix is usually needed for the space transform.
\vspace{-2.5mm}
\subsection{Network Pruning}
\label{subsec:pruning}
% \subsubsection{Overview of Pruning}
%Pruning has become a popular compression technique owing to its simplicity and effective results.
Pruning is a widely used technique for compressing neural networks by removing neurons from networks.
This helps reducing the storage size required for the network parameters, as well as the memory access during inference to load the surviving weights (neurons) from off-chip to on-chip memory.
% As benchmarked in \cite{gao2018dynamic}, the GPU memory usage can be reduced by more than 2 times for typical CNN architectures. Less neurons also means less computation required for inference, thereby reducing the latency as well as the power consumption.
%, as it offers a simple yet effective solution.
%\textcolor{red}{can you re-organize this part?}
Over the years, numerous algorithms for pruning have been developed.
% Some methods are
% % first or second-order derivatives \cite{NIPS1989_250,NIPS1992_647},
% gradient-based pruning~\cite{NIPS1989_250, woodfisher, mfac, shen2022structural, molchanov2019importance, xu2023efficient},
% customized criteria based pruning~\cite{molchanov2016pruning, LIU2020Dynamic, Lin2020Dynamic, tang2020scop},
% % distance or similarity measure based pruning \cite{Srinivas_2015},
% regularization based methods~\cite{pmlr-v119-kusupati20a, ContinuousSparsification2020, wang2021neural}, and magnitude-based pruning \cite{Zhu2018ToPO, Park2020LookaheadAF, pmlr-v119-evci20a, lee2021layeradaptive}.
% \cite{han2015learning} reinvigorated the pruning field by discovering a way to iteratively prune and retrain a neural network, achieving high sparsity rates while preserving accuracy. \cite{gale2019state} performed large-scale experiments demonstrating that simple, magnitude-based pruning yields the best results. % when done gradually.
%Another exciting way to do pruning is Runtime Neural Pruning \cite{NIPS2017_6813}, which utilizes reinforcement learning (RL) to train an RL agent that dynamically picks smaller sub-networks during inference.
% Runtime neural pruning is an interesting topic that utilizes reinforcement learning (RL) to train an agent that can dynamically select smaller sub-networks during inference. \cite{NIPS2017_6813}.
% \cite{he2018amc} designed the first RL approach during the training stage. However, a drawback of RL-based methods is that they typically require an additional training budget to train the RL agent \cite{gupta2020learning, he2018amc}.
%Another paradigm for pruning is whether pruning is done
%\textcolor{red}{seems categorization is as below. but later comparisons is based on structured/unstructured pruning.}
\subsubsection{Pruning Strategies - Before, During, and After Training}
\label{subsec:pruning_schemes}
Pruning solutions can be categorized according to multiple schemes. One scheme is when pruning is done with respect to the training of the model. Three choices exist here - pruning before training, during-training (pruning with training), or post-training (pruning after training).
% Recently, an increasing number of methods have emerged, advocating for pruning before training to enhance the effective utilization of information from the network training procedure.
% However, recently, more and more methods have been proposed to conduct pruning prior to the training to more effectively use information from network training procedure.
% is completed to more effectively use information from network training procedure.
\textbf{Pruning before training} It aims to identify sparse structures in dense network at initialization and directly train the sparse network, with the merit of accelerating training and back-propagation.
SNIP~\cite{lee2018snip} set the pruning criterion as the normalized magnitudes of gradients while GraSP~\cite{Wang2020Picking} pruned those weights whose removal will result in least decrease in the gradient norm after pruning.
FORCE~\cite{jorge2021progressive} presented a modified saliency metric based on \cite{lee2018snip}.
% \cite{wang2019picking} engineered the gradient flow when training sparse networks from scratch.
\begin{table*}[ht]
\centering
\scriptsize
\caption{Classification of different pruning algorithms by their type, stage of doing pruning, and whether they are structured (denoted by `S') or unstructured (denoted by `U'). More comprehensive summary of Pruning at initialization scheme can be found in~\cite{ijcai2022p786}.}
\vspace{-5pt}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|l}
%\toprule
\hline
Pruning methods & Pruning with training \\
\hline
%\midrule
Gradient based & Optimal Brain Damage \cite{NIPS1989_250} [U], WoodFisher \cite{woodfisher} [U], MFAC \cite{mfac} [U], HALP~\cite{shen2022structural}, Taylor~\cite{molchanov2019importance}, RD~\cite{xu2023efficient} \\
\hline
%\midrule
% Gradient-based & \textcolor{blue}{Learning-Compression~\cite{carreira2018learning}, ProbMask~\cite{zhou2021effective} [U], Hinge~\cite{li2020group} [S]} \\
%\midrule
% \hline
% Sensitivity or custom objectives based
Customized criteria based
& DPF \cite{Lin2020Dynamic} [S,U], DST \cite{LIU2020Dynamic} [U], Molchanov \textit{et al.} \cite{molchanov2016pruning} [U], SCOP~\cite{tang2020scop} [S] \\
%\midrule
\hline
% Similarity-based & & & Srinivas \textit{et al.} \cite{Srinivas_2015} [S] \\
% \hline
%\midrule
Regularization based & STR \cite{pmlr-v119-kusupati20a} [S,U], Savarese \textit{et al.} \cite{ContinuousSparsification2020} [U], Wang \textit{et al.} \cite{wang2021neural} [S,U] \\
%\midrule
\hline
Magnitude based & RigL \cite{pmlr-v119-evci20a} [S,U], LAP \cite{Park2020LookaheadAF} [S,U], GMP \cite{Zhu2018ToPO} [U], LAMP \cite{lee2021layeradaptive} [U], Global MP \cite{globalmp} [U], DSR~\cite{ICML-2019-MostafaW} [U], SM\cite{sparse_momentum} [U]
%Han \textit{et al.}\cite{han2015learning} [U]
\\ %\textcolor{red}{\cite{?}}
\hline
%\midrule
Reinforcement learning based & PuRL \cite{gupta2020learning} [U], AMC \cite{he2018amc} [S,U], RNP \cite{NIPS2017_6813} [S] \\
\hline
%\bottomrule
\end{tabular}
}
\label{table:pruning_methods}
\vspace{-5mm}
\end{table*}
\textbf{Pruning with training} It starts with a dense network from scratch and gradually prunes out more connections from the networks during training. %~\cite{bellec2018deep}. Compared to iterative pruning in post-training scheme, here the model is trained from scratch.
% \cite{kusupati2020soft,chen2022state,chen2022unified} adopt soft threshold pruning to learn element-wise threshold for the neurons during training.
Researchers proposed different pruning criteria (\textit{e.g.}, estimation of inverse Hessian in WoodFisher~\cite{woodfisher} and MFAC~\cite{mfac}, Taylor-based criterion in Molchanov \textit{et al.}~\cite{molchanov2016pruning}, $L2$ regularization based pruning~\cite{wang2021neural}, magnitude-based criterion~\cite{pmlr-v119-evci20a} and reinforcement learning based methods~\cite{gupta2020learning}), to remove less important parameters.
Some approaches either finished the pruning in one training pass (\eg, DPF~\cite{Lin2020Dynamic}) or multiple training passes~\cite{frankle2018the}.
% Another way is gradual pruning, allowing for re-training between pruning steps.
% 1) iterative pruning. 2) prune and grow. 3) pruning criterions.
% WoodFisher~\cite{woodfisher} and MFAC~\cite{mfac} proposes different ways to estimate of the inverse Hessian applied in neural network pruning.
% ProbMask~\cite{zhou2021effective} leverages projected gradient descent with progressive pruning stretegy to directly train sparse networks.
% Molchanov \textit{et al.}~\cite{molchanov2016pruning} adopted taylor-based criterion as a importance score for connection.
% \cite{zhang2022optg} integrates supermask training with gradient-driven sparsity for training sparse networks.
Another important technique is \textbf{prune-and-regrow}~\cite{pmlr-v119-evci20a}. Specifically, instead of removing a neuron once and for all during a pruning cycle in the training, neurons are given chances to grow back later.
RigL~\cite{pmlr-v119-evci20a} droped parameters using parameter magnitudes and grow
the connections with the highest magnitude gradients.
% \cite{mocanu2018scalable} randomly regrow connections to the network at the end of each training epoch.
DSR~\cite{ICML-2019-MostafaW} randomly selected pruned neurons to grow back during training but keeps the active neurons in the whole network the same throughout.
% \cite{bellec2018deep} makes the regrowth decision based on posterior behavior of the possible sparse configurations.
SM\cite{sparse_momentum} pruned weights
with small magnitude and growed new weights to fill in missing connections with the highest momentum magnitude. %relaxes the formulation and adopts weight momentum for the decisions, reducing the optimization complexity.
%
A comprehensive summary of the above pruning methods during training can be found in Tab.~\ref{table:pruning_methods}.
In summary, pruning before or during training offers the advantage of reducing the computational requirements throughout training, specifically in terms of floating-point operations per second (FLOPs).
% In summary, the advantage of pruning before or during training is the reduced computation required during training, \textit{i.e.}, the floating point operations per second (FLOPs).
%Theoretically, training sparse networks is more efficient than training dense networks as FLOPs savings can also be accrued during the training process, as %less multiply and adds
% fewer multiply and add operations
% need to be done in each iteration. Furthermore, gradual pruning methods GMP \cite{Zhu2018ToPO} have proven to be an effective way of pruning neural networks during training, achieving high accuracies while %achieving FLOPs reduction
% reducing FLOPs during the training process as well.
\textbf{Post-training Pruning}
% Initially, pruning was suggested as a post-training step.
Different from pruning with training, post-training pruning typically conducts one-shot pruning after the model has been trained on a dataset~\cite{banner2019post, frantar2022optimal, kwon2022fast}.
Post-training pruning in the context of Transformers has drawn significant interest due to growing concerns about the substantial resources and time needed for re-training.
\cite{frantar2022optimal} proposed a post-training compression framework which covers both weight pruning and quantization in a unified setting based on an efficient realization of the
classical Optimal Brain Surgeon (OBS) framework~\cite{NIPS1989_250}.
\cite{kwon2022fast} introduced a fast post-training framework that automatically
prunes the Transformer model using structured sparsity methods.
% : train a dense model, prune it in one shot, and then finally either with or without fine-tune the resulting sparse network.
% \cite{NIPS1989_250} was the very early work in one-shot pruning to employ second-order optimization techniques, specifically the Hessian matrix, to determine the importance of each parameter. The pioneer work by Han \textit{et al.}~\cite{han2015learning} established the standard for the network pruning process.
% \cite{han2015learning} are pioneer works in CNN pruning adopt a magnitude-base iterative pruning scheme for LeNet and AlexNet.
% \cite{frankle2018the} proposed the lottery hypothesis deriving a weight-rewinding technique in iterative-pruning. %magnitude iterative-pruning
% \cite{morcos2019one,zhu2018prune} adopts magnitude-based importance scores to threshold low-scored connections globally.
% \cite{gale2019state,evci2020rigging} leverage architectural heuristics to determine layer-wise pruning rate.
% \cite{lamp} improved the magnitude-based scores like in \cite{morcos2019one} by considering inter-layer score ranking.
% Several efforts also prune CNNs data-dependently, considering the influence of pruning on the model output.
% Molchanov \textit{et al.}\cite{molchanov2016pruning} derived a first-order taylor-based pruning criterion.
% \cite{isik2022information} assumed Laplacian distribution of CNN weights to approximate output distortion to determine layer-wise pruning ratio.
% \cite{wang2022rdo} and RD~\cite{xu2023efficient} leverage rate-distortion theory to derive layer-wise pruning ratios that achieves optimal rate-distortion performance.
% }
% It can be observed that while a diversity of pruning methods are more adopted by pruning with/after training methods, gradient-based pruning methods are more seen in pruning at initialization scheme, in order to achieve correct gradient propagation on the sparse networks. }
\subsubsection{Pruning Paradigms - Unstructured, Structured, and Semi-Structured Pruning}
\label{subsec:pruning_paradigms}
Another essential consideration is whether pruning occurs at the weight or channel level. It can be categorized into three types: unstructured pruning, structured pruning, and semi-structured pruning.
%, which involves removing individual weights, while structured pruning involves removing entire neurons or channels. In the following, we delve into structured, unstructured pruning and semi-structured pruning, with recent research highlighting the effectiveness of magnitude-based pruning. %although approaches like Runtime Neural Pruning and RL-based methods, while promising, may demand extra training resources.
% Another important scheme is whether pruning is done at a weight or channel level. Weight level pruning, whereby individual weights are removed from the network, is called unstructured pruning. Channel or neuron-level pruning, whereby entire neurons or channels are removed from the network, is called structured pruning.
% Below, we discuss structured and unstructured pruning solutions in more detail.
% Recent research has shown the efficacy of magnitude-based pruning. Existing approaches like Runtime Neural Pruning and RL-based methods have demonstrated promising results but may require additional training resources.
\textbf{Unstructured Pruning}
% \label{sec:unstructured_pruning}
%\textcolor{red}{can u give a definition on unstructured pruning and structured pruning}
%We list the top-performing algorithms for unstructured pruning in Table \ref{table:resnet50_imagenet}.
Unstructured pruning refers to remove individual weights.
Table~\ref{table:resnet50_imagenet} summarizes the top-performing algorithms for unstructured pruning. Among them,
global magnitude pruning (Global MP)~\cite{globalmp} achieved the top performance in terms of sparsifying parameters. This method ranks the weights by their absolute magnitude and prunes the smallest ones, resulting in high parameter sparsity. %and STR achieves the top performance in terms of sparsifying FLOPs. Global magnitude pruning works by ranking the weights by their absolute magnitude and pruning the smallest ones. STR works by reparameterizing the weights using a soft threshold operator and learning the sparsity rates using gradient descent.
On the other hand, STR~\cite{pmlr-v119-kusupati20a} achieved the top performance in terms of sparsifying FLOPs. It reparameterizes the weights using a soft threshold operator and learns the sparsity rates using gradient descent.
GMP~\cite{Zhu2018ToPO} developed a simple gradual pruning approach that requires minimal tuning.
DNW~\cite{DNW} provided an effective mechanism for discovering sparse subnetworks of predefined architectures in a single training run.
% RigL~\cite{pmlr-v119-evci20a} updates the topology of the sparse network during training using parameter magnitudes and infrequent gradient calculation.
% DPF~\cite{Lin2020Dynamic} proposed a dynamic pruning scheme that incorporates an error feedback in a natural way and finds a trained sparse model in one training pass.
%ERK denotes Erdős-Rényi-Kernel which modifies the original Erdős-Rényi~\cite{Mocanu2018} formulation by including the kernel dimensions in the scaling factors.
% Other notable methods with good performance are GMP \cite{Zhu2018ToPO}, DSR \cite{ICML-2019-MostafaW}, DNW \cite{DNW}, SM \cite{sparse_momentum}, RigL \cite{pmlr-v119-evci20a}, WoodFisher \cite{woodfisher}, MFAC \cite{mfac} and DPF \cite{Lin2020Dynamic}.
% GMP~\cite{Zhu2018ToPO} we also develop a simple gradual pruning
% approach that requires minimal tuning and can be seamlessly incorporated within the training process.
% DSR \cite{ICML-2019-MostafaW} developed a
% novel dynamic reparameterization method that yielded the
% highest parameter efficiency in training sparse deep residual networks.
% DNW \cite{DNW} DNW provides an
% effective mechanism for discovering sparse subnetworks of predefined architectures
% in a single training run.
% SM \cite{sparse_momentum} networks that rival the performance
% of their dense counterparts with a single training run – no re-training is required. We start with
% random initializations and maintain sparse weights throughout training while also speeding up the
% overall training time. We achieve this by developing sparse momentum, an algorithm which uses the
% exponentially smoothed gradient of network weights (momentum) as a measure of persistent errors to
% identify which layers are most efficient at reducing the error and which missing connections between
% neurons would reduce the error the most.
% RigL \cite{pmlr-v119-evci20a}, Our method updates the
% topology of the sparse network during training by
% using parameter magnitudes and infrequent gradient calculation
% WoodFisher \cite{woodfisher}, o compute a
% faithful and efficient estimate of the inverse Hessian in neural network pruning.
% MFAC \cite{mfac} we investigate
% matrix-free, linear-time approaches for estimating Inverse-Hessian Vector Products
% (IHVPs) for the case when the Hessian can be approximated as a sum of rank-one
% matrices, as in the classic approximation of the Hessian by the empirical Fisher
% matrix
\begin{table}[!ht]
\vspace{-2.5mm}
\scriptsize
\begin{threeparttable}
\caption{Results of unstructured pruning on ResNet-50 on ImageNet.
\vspace{-2.5mm}
%Global MP outperforms SOTA pruning algorithms on weight sparsity, and STR achieves the highest FLOPs-accuracy trade-off.
}
% \begin{tabular}[t]{p{2.9cm}p{0.8cm}p{0.8cm}p{0.8cm}p{0.8cm}}
\begin{tabular}[t]{l|c|c|c|c}
\toprule
\multirow{1}{*}{Methods} & Top-1 & Params & Sparsity & FLOPs pruned\\
\midrule
ResNet-50 & 77.0\% & 25.6M\ & 0.00\% & 0.0\%\\
\midrule
GMP~\cite{Zhu2018ToPO} & 75.60\% & 5.12M\ & 80.00\% & 80.0\%\\
DSR\tnote{*\#} \ \cite{ICML-2019-MostafaW} & 71.60\% & 5.12M\ & 80.00\% & 69.9\%\\
DNW~\cite{DNW} & 76.00\% & 5.12M\ & 80.00\% & 80.0\%\\
SM~\cite{sparse_momentum} & 74.90\% & 5.12M\ & 80.00\% & -\\
SM(ERK)~\cite{pmlr-v119-evci20a} & 75.20\% & 5.12M\ & 80.00\% & 58.0\%\\
RigL\tnote{*} \cite{pmlr-v119-evci20a} & 74.60\% & 5.12M\ & 80.00\% & 77.5\%\\
RigL(ERK)~\cite{pmlr-v119-evci20a} & 75.10\% & 5.12M\ & 80.00\% & 58.0\%\\
DPF~\cite{Lin2020Dynamic} & 75.13\% & 5.12M\ & 80.00\% & 80.0\%\\
STR~\cite{pmlr-v119-kusupati20a} & 76.19\% & 5.22M\ & 79.55\% & 81.3\%\\
\textbf{Global MP (One-shot)}~\cite{globalmp} & \textbf{76.84\%} & 5.12M & 80.00\% & 72.4\%\\
\underline{Global MP (Gradual)}~\cite{globalmp} & \underline{76.12\%} & 5.12M & 80.00\% & 76.7\%\\
\midrule
% GMP & 73.91\% & 2.56M\ & 90.00\% & 90.0\%\\
% DNW & 74.00\% & 2.56M\ & 90.00\% & 90.0\%\\
% SM & 72.90\% & 2.56M\ & 90.00\% & 60.1\%\\
% SM + ERK & 72.90\% & 2.56M\ & 90.00\% & 76.5\%\\
% RigL* & 72.00\% & 2.56M\ & 90.00\% & 87.4\%\\
% RigL + ERK & 73.00\% & 2.56M\ & 90.00\% & 76.5\%\\
% DPF\# & 74.55\% & 4.45M\ & 82.60\% & 90.0\%\\
% STR & 74.73\% & 3.14M\ & 87.70\% & 90.2\%\\
% \textbf{Global MP (One-shot)} & \textbf{75.28\%} & 2.56M & 90.00\% & 82.8\%\\
% \underline{Global MP (Gradual)} & \underline{74.83\%} & 2.56M & 90.00\% & 87.8\%\\
% \bottomrule
\end{tabular}
%\begin{tabular}[t]{p{2.9cm}p{0.8cm}p{0.8cm}p{0.8cm}p{0.8cm}}
%\toprule
%\multirow{1}{*}{Method} & Top-1 Acc & Params & Sparsity & FLOPs pruned\\
%\midrule
%ResNet-50 & 77.0\% & 25.6M\ & 0.00\% & 0.0\%\\
%\midrule
%GMP & 70.59\% & 1.28M\ & 95.00\% & 95.0\%\\
%DNW & 68.30\% & 1.28M\ & 95.00\% & 95.0\%\\
%RigL* & 67.50\% & 1.28M\ & 95.00\% & 92.2\%\\
%RigL + ERK & 70.00\% & 1.28M\ & 95.00\% & 85.3\%\\
%WoodFisher & 72.12\% & 1.28M & 95.00\% & -\\
%MFAC & 72.32\% & 1.28M & 95.00\% & -\\
%STR & 70.97\% & 1.33M\ & 94.80\% & 95.6\%\\
%STR & 70.40\% & 1.27M\ & 95.03\% & 96.1\%\\
%STR & 70.23\% & 1.24M\ & 95.15\% & 96.0\%\\
%\underline{Global MP (One-shot)} & \underline{71.56\%} & 1.20M & 95.30\% & 89.3\%\\
%\textbf{Global MP (Gradual)} & \textbf{72.14\%} & 1.20M & 95.30\% & 93.1\%\\
%\midrule
%GMP & 57.90\% & 0.51M\ & 98.00\% & 98.0\%\\
%DNW & 58.20\% & 0.51M\ & 98.00\% & 98.0\%\\
%STR & 61.46\% & 0.50M & 98.05\% & 98.2\%\\
%\underline{Global MP (One-shot)} & \underline{61.80}\% & 0.50M & 98.05\% & 93.7\%\\
%\textbf{Global MP (Gradual)} & \textbf{66.57}\% & 0.50M & 98.05\% & 96.2\%\\
%\bottomrule
%\end{tabular}
\begin{tablenotes}
\item[*] The first layer of the network architecture is dense.
\item[\#] The last layer of the network architecture is dense.
\end{tablenotes}
\end{threeparttable}
\label{table:resnet50_imagenet}
% \vspace{-2.5mm}
\end{table}
\textbf{Structured Pruning}
% \label{sec:structured_pruning}
%We list the top-performing algorithms for structured pruning in Table \ref{table:resnet50_structured}.
Structured pruning targets the removal of organized neurons, such as channel-wise or block-wise pruning. Certain studies~\cite{8953212, lin2020hrank} established metrics, excluding $l1$-norm~\cite{Wang2020PruningFS}, to decide specific channel pruning. HRank~\cite{lin2020hrank} mathematically proved that filters with lower-rank feature maps contain less informative content, suggesting their suitability for removal. Another research direction explores automatic pruning without meticulous hyper-parameter adjustments, with DHP \cite{10.1007/978-3-030-58598-3_36} introduced a differentiable pruning technique using hypernetworks. This approach allows for automatic network pruning by using hypernetworks that take latent vectors as input and generate weight parameters for the backbone network. In parallel, a research line focuses on simultaneously pruning diverse redundant structures \cite{8953628, CHL00L21}. For instance, GAL~\cite{8953628} achieved simultaneous end-to-end pruning of diverse structures, such as channels, filters, and blocks, using label-free generative adversarial learning. GAL employs a sparse soft mask to scale specific structures' outputs to zero.
% \textcolor{blue}{Structured pruning
% ~\cite{8953628, 8953212, lin2020hrank, 10.1007/978-3-030-58598-3_36}
% intends to remove structured neurons such as channel-wise or block-wise pruning. Certain studies, such as those by Lin \textit{et al.}\cite{8953212, lin2020hrank}, concentrate on establishing metrics to determine the pruning of specific channels except $l1$-norm~\cite{Wang2020PruningFS}. HRank~\cite{lin2020hrank} demonstrates mathematically that filters with lower-rank feature maps contain less informative content, making them candidate for removal. An alternative research direction explores automatic pruning without meticulous hyper-parameter adjustments. DHP \cite{10.1007/978-3-030-58598-3_36} introduces a differentiable pruning technique using hypernetworks, where specifically designed hypernetworks take latent vectors as input and generate weight parameters for the backbone network, facilitating automatic network pruning. Another research line focuses on prune diverse redundant structures ~\cite{8953628, CHL00L21} simultaneously. For example, GAL~\cite{8953628} simultaneously pruned diverse redundant structures (\textit{e.g.}, channels, filters, and blocks) in an end-to-end fashion without iterative training, using label-free generative adversarial learning through a sparse soft mask, scaling specific structures' outputs to zero. }
3D~\cite{CHL00L21} pruned a model by determining the optimal value of a pruned network along three dimensions, \textit{i.e.}, layers, filters, and image resolution. It did this by conducting multiple pruning runs to build a database of pruning rates along each dimension and the associated accuracy. It then used the Lagrangian theorem to fit the data to an optimal pruning function.
% 1). Determine which to prune. 2) Differential pruning method. 3) Different scales in one universal mode.
Table \ref{table:resnet50_structured} presents the top-performing algorithms for structured pruning.
\vspace{-5mm}
% PScratch \cite{Wang2020PruningFS} prunes the model at initialization using layer-wise sparsity ratios and a global pruning threshold to ensure that the target FLOPs are met. 3D \cite{CHL00L21} prunes a model by determining the optimal value of a pruned network along three dimensions, \textit{i.e.}, layers, filters, and image resolution. It does this by conducting multiple pruning runs to build a database of pruning rates along each dimension and the associated accuracy. It then uses the Lagrangian theorem to fit the data to an optimal pruning function. %Other notable methods with good performance are GAL \cite{8953628}, FPGM \cite{8953212}, HRank \cite{lin2020hrank} and DHP \cite{10.1007/978-3-030-58598-3_36}.
% GAL~\cite{8953628} simultaneously pruned diverse redundant structures (\textit{e.g.}, channels, filters, and blocks) in an end-to-end fashion without iterative training, using label-free generative adversarial learning (GAL) through a sparse soft mask, scaling specific structures' outputs to zero.
% FPGM~\cite{8953212} prune the most replaceable filters containing redundant information using Geometric Median (GM)~\cite{fletcher2008robust}.
% HRank~\cite{lin2020hrank} mathematically prove that filters with lower-rank feature maps are less informative and thus less important to preserve accuracy, which can be removed first.
% DHP \cite{10.1007/978-3-030-58598-3_36} introduces a differentiable pruning method via hypernetworks for automatic network pruning. The specifically designed hypernetworks take latent vectors as input and generate the weight parameters of the backbone network.
% \noindent
\begin{table}[!ht]
\centering
\scriptsize
\caption{Structured pruning outcomes for ResNet-50 on ImageNet.}
\vspace{-5mm}
\resizebox{0.45\textwidth}{!}{
\begin{tabular}[t]{lcccc}
\toprule
\multirow{1}{*}{Methods} & Top-1 Acc & Params & Sparsity & FLOPs pruned\\
\midrule
ResNet-50 & 76.15\% & 25.6M\ & 0.00\% & 0.0\%\\
\midrule
GAL~\cite{8953628} & 71.95\% & 21.25M\ & 17.00\% & 43.0\% \\ %2.33G\\
FPGM~\cite{8953212} & 74.83\% & - & - & 54.0\% \\ %1.88G\\
HRank~\cite{lin2020hrank} & 74.98\% & 16.13M\ & 37.00\% & 44.0\% \\ %2.29G\\
DHP~\cite{10.1007/978-3-030-58598-3_36} & 75.45\% & 11.78M\ & 54.00\% & 49.9\% \\ %2.05G\\
\textbf{PScratch}\cite{Wang2020PruningFS} & 75.45\% & \textbf{9.22M}\ & \textbf{64.00\%} & \textbf{49.9\%} \\ %2.05G}\\
\textbf{3D}~\cite{CHL00L21} & \textbf{75.90\%} & 12.03M\ & 53.00\% & \textbf{49.9\%} \\ %2.05G}\\
\bottomrule
\end{tabular}
}
\label{table:resnet50_structured}
% \vspace{-5mm}
\end{table}
%\subsubsection{Practical insights for pruning and future work}
%Researchers and practitioners today have many options to choose suitable pruning techniques for their use case. However, it is to be noted that complex algorithms don't always mean that the performance will be superior. For example, it has been shown that simple pruning algorithms like Global magnitude pruning \cite{globalmp} and GMP \cite{Zhu2018ToPO} produce state-of-the-art results \cite{gale2019state}. On the other hand, complex pruning algorithms like RL-based or heuristics-based algorithms do not provide much improvement.
%
%Secondly, a well-known fact in pruning is that non-uniform sparsity is almost always better than uniform sparsity, i.e., varying the pruning ratio across layers is better than keeping a constant pruning ratio across layers \cite{gale2019state}. This holds even in pruning before training or pruning at initialization \cite{liu2022the}. Hence, finding optimal sparsity rates per layer remains an important and complex challenge. Also, sparsity rates vary from model to model; therefore, a good pruning solution is unique to a model. Therefore, a one-size-fits-all solution that is portable across models does not work as well as a unique solution.
%
%Lastly, hyper-tuning can be undertaken for different sparsity targets, i.e., the same set of hyper-parameters may not always work the best for various sparsity targets. Algorithms typically tune hyper-parameters per sparsity target which yields superior results \cite{pmlr-v119-kusupati20a}. Momentum, weight-decay, learning rate (and learning rate schedule), and label smoothing are some hyper-parameters that may need to be varied. Batch size typically does not need to be varied much. Optimizing these hyper-parameters will ensure the pruned models perform well at varying sparsities.
%
%Many directions exist for future work in pruning. Firstly, FLOPs can be added to the optimization function to optimize both parameters and FLOPs during pruning jointly. Most pruning algorithms today do not do joint optimization of these two metrics. Some techniques develop methodologies to look at FLOPs and parameters both but do so separately, optimizing each criterion on a stand-alone basis \cite{he2018amc}. In addition, some algorithms do not explicitly optimize for FLOPs but see advantages on FLOPs as well \cite{pmlr-v119-kusupati20a}. Therefore, the opportunity exists for more work to be done in jointly optimizing both FLOPs and parameters, ensuring that both parameters and FLOPs are sparsified well.
%
%Secondly, a growing area of interest is pruning before training or at initialization. However, so far, most algorithms perform only as well as random pruning \cite{liu2022the}, and no moral way has achieved higher performance than random pruning. This leaves the room open for better algorithms for pruning at initialization to be found. The main advantage of pruning at initialization is that FLOPs can also be reduced during training. This makes the training also sparse, along with the inference; for models like large language models that require long training durations, sparse training becomes even more critical. A generalizable pruning at initialization solution across a different family of models might see wide adoption in the pruning community.
%
%Thirdly, how to distribute sparsity across layers remains elusive and has not been theoretically answered with solid mathematical justification. Most techniques try a bottom-up approach to come out with pruning methods and justify them empirically. However, a top-down approach that mathematically proves why different layers deserve different sparsity ratios remains to be discovered.
%
%Lastly, deep learning hardware today is not optimized to run pruned models directly. More specifically, pruning has to be done in a structured manner, i.e., removing whole neurons and filters to enjoy actual on-device speed-ups. Unstructured pruning, which can do more fine-grained pruning and thus achieve higher pruning rates, does not enjoy actual speed-ups when deployed on the hardware. Hence, the hardware gap needs to be bridged where new hardware can support the direct application of unstructured pruning.
\textbf{Semi-structured Pruning}
Aside from the two major sparsity patterns, semi-structured pruning scheme is also adopted by few literature, which aligns the sparsity patterns with the GEMM (GEneric Matrix Multiplication) designs to achieve handy hardware acceleration while still benefits from relatively low accuracy drop. \cite{lagunas2021block} pruned feedforward weights in Transformer-based language models (\textit{e.g.}, BERT) in block-sparsity pattern and achieved a trade-off between accuracy retaining and acceleration.
Apart from the regular block shaped type of semi-structure, another N:M sparsity~\cite{zhou2021learning} arises recently available on NVIDIA's A-series GPUs. Several Vision Transformers pruning works~\cite{fang2022algorithm,yang2023global,yu2023boost} explored using such sparsity scheme to achieve fine-grained feed-forward and attention pruning.
By examining Table~\ref{table:resnet50_imagenet} and Table~\ref{table:resnet50_structured}, it becomes evident that structured pruning is conducive to hardware constraints. Nevertheless, it prunes fewer neurons and FLOPs, resulting in lower memory costs when compared to fine-grained unstructured pruning while maintaining similar levels of accuracy. More memory analysis of pruning can be found in~\cite{cheng2023survey}.
% \textbf{Discussions}
There are some interesting works on exploring network pruning to meet specific computational constraints. One important technique is \textbf{slimmable network}~\cite{yu2018slimmable,yu2019universally,li2021dynamic}. Instead of optimizing a fixed and unified sparse network for inference, slimmable network aims to dynamically determine the network widths (\eg, layers, channels, or blocks) at runtime, to predict the sample with minimal performance sacrifice.
\cite{yu2018slimmable} was one of the pioneer works in slimmable network, which proposed networks with variable width (number of channels) and an altered BatchNorm strategy to train the network with switchable widths by sharing network parameters.
% \cite{yu2019universally} proposed switchable batch normalization and in-place distillation in the training to achieve higher performance than vanilla slimmable network.
\cite{li2021dynamic} adopted learnable gating layers to decide the layer widths.
Another important research line is \textbf{Once-for-All} networks~\cite{cai2019once}. Instead of training multiple models for various resource constraints, Once-for-All networks learn a shared set of parameters that can be applied to different subnetworks. These subnetworks vary in depth, width, or other architectural aspects, allowing the model to adapt to diverse computational requirements.
% Since more complex design are proposed later but more or less similar to NAS and less fit into pruning framework, one can refer to this dynamic network survey~\cite{han2021dynamic} for a more comprehensive review.
% \subsubsection{Summary}
In summary, network pruning has a long history and its effectiveness has been verified by numerous research works. %Existing works have shown their pros and cons in terms of accuracy preservation, hardware friendliness, and optimization effectiveness.
Unstructured pruning is the de-facto scheme known for best performance preservation, but limited by the difficulties to deploy on real hardware to achieve speedups. The new N:M sparsity scheme on NVIDIA's Ampere platform offers a great fine-grained sparsity pattern similar unstructured pruning and with remarkable acceleration, but the restriction of row-wise sparsity constraint and the limited availability are still outstanding issues to address. Block-structured and Structured pruning look at larger structures in the networks as elements for pruning which is more hardware friendly. However, the dependency of neurons within a pruning group (\emph{e.g.} A 2d block of neurons for block-structured or all the neurons in a channel in structured pruning) poses challenge to evaluate the impact of removing the whole pruning group. The optimal pruning criteria for minimizing accuracy loss while removing redundancy remains an open question, especially considering the continuous evolution of novel neural network architectures. Classic criteria such as magnitude-based and derivative-based importance are originally proposed under post-train-pruning, where a prune-then-finetune procedure is required. These criteria are further borrowed into more advanced schemes such as prune-at-initialization and prune-during-training, where pruning are interleaved with the training.
\vspace{-5pt}
\subsection{Knowledge Distillation}
\label{subsec:kd}
%Knowledge distillation (KD) aims to distill the learned knowledge from a cumbersome model (\textit{Teacher}) to a lightweight model (\textit{Student}). The student model is generally designed in advance to satisfy the latency and footprint requirements on resource-limited devices. It is compressed in the sense that it has less trainable parameters and floating-point operations (FLOPs) but can achieve same or similar accuracy as the teacher model. Moreover, there is no critical constrains on student architecture, thus it makes designing a hardware-efficiency network architecture possible.
Knowledge Distillation (KD) compresses the learned knowledge from a large model (Teacher) into a smaller model (Student), which is designed to meet the latency and footprint requirements on resource-limited devices. This compression results in fewer trainable parameters and FLOPs while maintaining similar accuracy to the teacher model. The student architecture is flexible and can be optimized for hardware efficiency.
Due to its flexibility and efficiency in model compression, it has drawn a lot of interest and has also been verified in many research areas such as computer vision~\cite{wang2017model, lee2018teacher, zhu2021student, park2021learning}, neural language processing~\cite{kim2016sequence, gordon2019explaining,wang2021joint}, speech recognition~\cite{chebotar2016distilling, kwon2020adaptive,li2021mutual}, time series regression~\cite{xu2021kdnet, xu2022contrastive} and so on.
% Two problems are needed in all KD-related research: % what knowledge is and how to transfer it to the student effectively.
% defining what constitutes knowledge, and developing effective methods for transferring it to the student.
%In this subsection, we first summarize the structured knowledge in the literature and then discuss the popular distillation schemes.
Two key challenges in KD research are defining what constitutes knowledge and developing effective methods for transferring that knowledge to the student model.
In this subsection, we first provide an overview of the structured knowledge in the literature, before delving into a discussion of the popular distillation schemes.
\subsubsection{Structured Knowledge}
\label{sec:structured_knowledge}
Generally, knowledge can be broadly categorized into three types: knowledge from logits, features, and relations. %Furthermore, these three types of knowledge can be utilized separately or simultaneously.
\textbf{Knowledge from logits}
The terminology of ``knowledge distillation'' was first introduced by Hinton \textit{et al.} \cite{hinton2015distilling}. The logits from the softmax layer of a complex teacher model, softened by a temperature factor, were employed as the ``dark knowledge'' to guide the training process of a compact student model. Compared with ground truth labels, the softened logits serve as a regularization term to improve the generalization capability of the student model. Zhang \textit{et al.} \cite{zhang2018better} formalized the logits from multiple teachers as a graph and presented a logit graph distillation approach for video classification. More recently, Zhao \textit{et al.} \cite{Zhao_2022_CVPR} reformulated teacher's logits into two parts: TCKD and NCKD, which are a binary probability vector for target class and an independent probability vector for non-target classes, respectively. They empirically showed that the NCKD is critical to the success of distillation and therefore proposed to decouple them by introducing two independent hyper-parameters.
%Although many other KD-related works have also demonstrated its effectiveness \cite{chen2017learning, yuan2019obtain, hegde2020variational}, the logits-based knowledge is originally designed for classification tasks but not for regression tasks whose output is a scalar instead of a probability distribution vector. However, some recent works (e.g.,\cite{saputra2019distilling, xu2021kdnet,xu2022contrastive}) also demonstrate its effectiveness on time series regression tasks. Directly minimizing the distance of teacher and student's scalar predictions could also help to improve the generalization capability of compact students.
Many KD works have shown the effectiveness of using logits-based knowledge \cite{chen2017learning, yuan2019obtain, hegde2020variational}. However, this type of knowledge is originally designed for classification tasks but not for regression tasks where the output is a scalar. Despite this, recent works such as \cite{saputra2019distilling, xu2021kdnet, xu2022contrastive} have demonstrated the effectiveness of logits-based knowledge on time series regression tasks. By directly minimizing the distance between the teacher and student's scalar predictions, the generalization capability of the compact student can be improved.
%This limitation hinders its applications from regression tasks like object detection with bounding-box regression and time series regression.
\textbf{Knowledge from feature representations}
Utilizing the feature representations from the teacher's intermediate layers as the knowledge has received significant attention as they contain more meaningful information than logits. Directly aligning teacher and student's features via a specific distance metric can force a student to learn similar representations to its teacher. However, direct regression of teacher and student's feature maps can lead to suboptimal performance, resulting in less effective knowledge transfer.
% Apart from the logits, utilizing the feature representations from the teacher's intermediate layers as the knowledge has also %drawn much attention
% received significant attention as they contain more meaningful information than logits. Intuitively, directly aligning teacher and student's features via specific distance metric (\textit{e.g.} Euclidean distance) would force a student to learn similar representations to its teacher \cite{romero2014fitnets,saputra2019distilling, chen2022knowledge,zhu2021student}. Direct regression of teacher and student's feature maps %would result in a sub-optimal problem, decreasing the efficacy of knowledge transferring.
% can lead to suboptimal performance, resulting in less effective knowledge transfer.
Therefore, other well-structured knowledge derived from feature representations was further presented, such as attention maps~\cite{zagoruyko2016paying}, the flow of solution procedure~\cite{yim2017gift}, factor transferring~\cite{kim2018paraphrasing}, mutual information~\cite{ahn2019variational} and more. % so on.
Furthermore, Liu \textit{et al.}~\cite{liu2021exploring} %considered the spatial correlations between different channels of features as the knowledge and aligned the diversity and homology of feature space between teacher and student.
proposed using the spatial correlations between different channels of features as the knowledge to align the diversity and homology of feature space between the teacher and student.
%Lin \textit{et al.} \cite{lin2022knowledge} reconstructed student's feature maps with a novel target-aware transformer to avoid the sub-optimal issue of % directly
direct regression teacher and student's feature maps.
Lin \textit{et al.}~\cite{lin2022knowledge} addressed the sub-optimal issue of direct regression of teacher and student's feature maps by reconstructing the student's feature maps with a novel target-aware transformer.
Recently, Shang \textit{et al.}~\cite{shang2021lipschitz} formulated the knowledge as Lipschitz constants via a transmitting matrix derived from the input and output feature maps of each network block. Huang \textit{et al.}~\cite{huang2021revisiting} argued that we should encourage the student not only to inherit the knowledge from the teacher but also to explore more diverse feature representations via designing a novel dis-similarity loss among feature representations. Finally, Ji \textit{et al.}~\cite{ji2021show} explored the similarity relations between teacher and student's feature maps from different layers via an attention network to avoid manual layer pairing.
%In summary, feature representations-based knowledge is more informative than logits-based knowledge and can be applied to classification and regression tasks. However, it has some drawbacks. Firstly, the task-specified knowledge derived from feature representations may not generalize well on other tasks.
%In summary, feature representations provide more informative knowledge than logits for classification and regression tasks, but have drawbacks. One drawback is that task-specific knowledge from feature representations may not generalize to other tasks.
% In summary, feature representations offer more informative knowledge than logits for classification and regression tasks. However, they have a drawback that task-specific knowledge from feature representations may not be useful for other tasks.
% Moreover, When distilling knowledge between different network architectures, pre-defined feature-based knowledge can harm the performance of students \cite{xu2021kdnet}. Choosing the right intermediate layers for distillation is difficult, especially when the teacher and student models differ significantly in capacity.
%Moreover, when distilling knowledge between disparate network architectures, the pre-defined feature-based knowledge may negatively impact students' performance \cite{xu2021kdnet}. Lastly, %it is still inexplicable to properly select the knowledgeable intermediate layers for distillation, especially when there is a big gap between the teacher and student's model capacity.
%selecting the appropriate and knowledgeable intermediate layers for distillation remains a challenging problem, especially when there is a large gap in model capacity between the teacher and student models.
\textbf{Knowledge from instance relation}
Instead of simply mimicking teachers' responses via logits or feature representations on individual training instances, the relationships between different instances could also serve as knowledge. For example, Passalis and Tefas ~\cite{passalis2018learning} modeled teacher's knowledge as a probability distribution among a batch of samples and transferred it by minimizing the distribution divergence between teacher and student. In a similar vein, Park \textit{et al.}~\cite{park2019relational} proposed transferring the mutual relations instead of actual representations by introducing two novel distillation losses, allowing student to learn the structured relations among different samples. Liu \textit{et al.}~\cite{liu2019knowledge} formulated the instance relationship graph as the knowledge source, and Huang \textit{et al.}~\cite{huang2022knowledge} aligned the inter-class and intra-class relation simultaneously for instances from different classes and same class, allowing for knowledge transfer from stronger teachers or more robust training strategies. Similarly, Yun \textit{et al.}~\cite{yun2020regularizing} minimized the intra-class variance among different instances having the same labels. Moreover, contrastive learning has been introduced for student representation learning and sample relation optimizing~\cite{tian2019contrastive, zhu2021complementary,xu2022contrastive}. Specifically, the teacher and student-generated outputs from the same sample are generally considered positive pairs, while the outputs of teachers generated from different samples are regarded as negative pairs. Contrastive learning minimizes the distance of positive pairs and maximizes the distance of negative pairs.
%Although relation-based knowledge distillation works well, it requires more complex training, like extra memory for data storage and graph construction. %Contrastive learning aims to minimize the distance between positive pairs while simultaneously maximizing the distance between negative pairs. Although relation-based knowledge distillation approaches have demonstrated satisfying performance, they inevitably involve more complex training processes, such as additional memory requirements for historical data storage, graph construction, etc.
To sum up, although the effectiveness of the aforementioned structured knowledge has been verified by many existing works, they have their advantages and disadvantages. The logit-based knowledge is the most widely-used knowledge due to its superior simplicity. However, the limited information provided by the logits severely hinders the efficacy of knowledge transformation. Meanwhile, its performance on other tasks (\textit{e.g.}, regression task) still requires more exploration. On the contrary, feature representations can offer more informative knowledge than logits for classification and regression tasks. However, they have a drawback that task-specific knowledge from feature representations may not be useful for other tasks. When distilling knowledge between different network architectures, pre-defined feature-based knowledge can harm the performance of students \cite{xu2021kdnet}. Choosing the right intermediate layers for distillation is challenging, especially when the teacher and student models differ significantly in capacity. Combining logit-based and feature-based knowledge is a common strategy in many KD-related works. Lastly, although some works have demonstrated that relation-based knowledge distillation works well, it does involve more complex training processes \cite{wang2021knowledge,peng2019correlation}, like extra memory for data storage and graph construction. Generally, it is also very challenging to integrate with the other two types of structured knowledge.
\subsubsection{Distillation Schemes}
\label{subsec:distillation_schemes}
Distillation schemes pertain to efficiently transferring knowledge from the teacher model to the student model, categorized into three types briefly: a) metric-specified vs. metric-free distillation; b) teacher-specified vs. Teacher-free distillation; and c) data-available vs. data-free distillation. More details can be found in Appendix C.
%\subsubsection{Challenges and future trends}
%Although the effectiveness of KD in model compression has been validated by extensive applications, there are still some challenges. Firstly, a widely accepted understanding of why KD works is that it would improve students' generalization ability via the `dark' knowledge provided by teachers. Despite some prior works (\textit{e.g.} \cite{phuong2019towards,allen2020towards}) that intend to do theoretical explanation or empirical evaluation, their work is not comprehensive, and the theoretical analysis on KD remains insufficient. Secondly, the design of teacher-student architecture is not fully investigated. Existing works choose teacher-student architectures mainly based on prior experience. However, it has been shown that the capacity gap between teacher and student would hinder the efficacy of knowledge transferring \cite{cho2019efficacy, mirzadeh2020improved}. Therefore, designing a suitable student or choosing a decent teacher remains challenging in knowledge distillation. Lastly, despite various knowledge types being proposed in existing works, the quality of knowledge or generalization to other applications has not been comprehensively studied.
%
%As a flexible and effective method for model compression, KD can be further integrated with other compression techniques such as neural architecture search (NAS) \cite{xie2022performance} and weight pruning \cite{aghli2021combining}. Furthermore, in the aspect of knowledge transfer, there is a growing trend to combine KD with other deep learning schemes to enhance their performances, such as continual learning \cite{yao2019adversarial, cheraghian2021semantic}, reinforcement learning \cite{fan2021reinforced, wang2021multi,tsantekidis2021diversity}, self-supervised learning \cite{noroozi2018boosting, xu2020knowledge} and so on. Since they are beyond the scope of this survey, we will not discuss the details here.
% \vspace{-5pt}
\subsection{Non-linear operations}
\label{subsec:nonlinear}
%\textbf{Can you write high level idea/description of non-linear operations here?}
Most existing works focus on linear matrix operations in convolutional layers or fully connected layers which account for over 99\% of the total operations in modern DNNs \cite{geng2018hardware}.
The implementation of non-linear blocks, such as Softmax and Sigmoid, presents significant difficulties for hardware. This is due to two main reasons. Firstly, convolutional operations are much simpler and cheaper to perform using a multiplier and adder, whereas non-linear functions like exponential require a more complex hardware unit. Secondly, the compute ratio per unit area for non-linear blocks is currently an order of magnitude lower than that for convolutional layers in existing accelerators. However, there has been a recent increase in research efforts to optimize non-linear operations for hardware implementations.
%However, the non-linear blocks (\eg, softmax and sigmoid) pose great challenges for hardware implementation. First, a convolutional operation using a multiplier and an adder, much cheaper than non-linear functions like exponential, which usually requires a complex hardware unit. Second, in current accelerators, the compute ratio per unit area for nonlinear blocks is order of magnitude of lower than that for convolutional layers. Recently, more and more works explore to optimize the non-linear operations for hardware implementations.
In this section, we review the literature which aims cost-efficient inference for non-linear operations.
\subsubsection{NMS}
\label{sec:nms}
\IncMargin{0.66em}
%The classical greedy NMS algorithm~\cite{greedynms}, as demonstrated in Algorithm~\ref{alg:nms}, first sorts the bounding boxes based on their respective class confidence scores and then iteratively removes the box in the ranking list that has a significant overlap (IoU) with the selected one.
The greedy non-maximum suppression (NMS) algorithm~\cite{greedynms} (Algorithm in Appendix D) sorts bounding boxes by class confidence scores and removes boxes with significant overlap (IoU) iteratively.
Table \ref{tab:nms_comparing} provides an overview of various NMS algorithms. In particular, except greedy NMS, we summarize them into six types. The first three types intend to optimize the sub-module in the greedy algorithm, whereas the last three seek an entire algorithm replacement.
% to replace the entire algorithm.
The six types are: \textbf{1) greedy-sorting}:
In the greedy NMS, the boxes are sorted based on the categorical confidence but not the location confidence, and thus the ranking list might not be reliable. This type of methods~\cite{bodla2017soft, tychsen2018improving, jiang2018acquisition} focuses on producing a more reliable ranking list by %exploiting
incorporating extra %more
location-based information %to affect
into the ranking. For example, SoftNMS~\cite{bodla2017soft} decayed the confidence score of the remaining detection boxes using a function with the IoU as an argument, thereby affecting the sorting. % and thus affects the sorting;
\textbf{2) greedy-finalizing box}: it focuses on finalizing the box coordination~\cite{gahlert2020visibility, he2019bounding}. \textbf{3) greedy-duplicate check}: these methods focus on duplicate bounding box check criteria~\cite{liu2019adaptive, salscheider2021featurenms}. For example, \cite{liu2019adaptive} dynamically modified the IoU threshold in the duplicate check using object density. \textbf{4) end-to-end}: these methods replace NMS with a neural network sub-module and train in an end-to-end manner without manual intervention~\cite{hosang2016convnet,hosang2017learning, hu2018relation}. \textbf{5) clustering}: this kind of methods focuses on parallel clustering predicted bounding boxes to speed up the NMS~\cite{rothe2014non, gpunms2}. \textbf{6) pooling}: it focuses on reformulating NMS as a max pooling operation which is inherently
parallel and hardware-friendly%friendly to the hardware
~\cite{shapoolnms, maxpoolnms, psrrmaxpool}.
%as the sorting and duplicate check are a global sequential operation.
\begin{table*}[t]\centering
\caption{Comparing NMS algorithms in the literature}\label{tab:nms-alg-lit}
\vspace{-5pt}
\scriptsize
\begin{tabular}{lccccc}\toprule
Methods &Types & Network Replacement & Parallelable? & Two-stage detector? &Time Complexity \\\cmidrule{1-6}
GreedyNMS~\cite{greedynms} & greedy &\xmark &\xmark &\cmark &$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\\cmidrule{1-6}
SoftNMS~\cite{bodla2017soft} & 1) &\xmark &\xmark &\cmark &$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\\cmidrule{1-6}
AdaptiveNMS~\cite{liu2019adaptive} &3) &\xmark &\xmark &\cmark &$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\\cmidrule{1-6}
TNetNMS~\cite{hosang2016convnet} &4) &\cmark &\cmark &\cmark &Nework architecture related \\\cmidrule{1-6}
FeatureNMS~\cite{salscheider2021featurenms} &3) &\xmark &\xmark &\cmark &$\mathcal{O}(nm)$ \\\cmidrule{1-6}
vg-NMS~\cite{gahlert2020visibility} &2) &\xmark &\xmark &\cmark &$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\\cmidrule{1-6}
MaxpoolNMS~\cite{maxpoolnms} &6) &\xmark &\cmark &\xmark &$\mathcal{O}(n)$ \\\cmidrule{1-6}
PSRR-MaxpoolNMS~\cite{psrrmaxpool} &6) &\xmark &\cmark &\cmark &$\mathcal{O}(n)$ \\\cmidrule{1-6}
ClusteringNMS~\cite{rothe2014non} &5) &\xmark &\cmark &\xmark &$\mathcal{O}(nm)$ \\\cmidrule{1-6}
GPUNMS~\cite{gpunms2} &5)&\xmark &\cmark &\xmark &$\mathcal{O}(nm)$ \\\cmidrule{1-6}
FitnessNMS~\cite{tychsen2018improving} &1) &\xmark &\xmark &\cmark &$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\\cmidrule{1-6}
SofterNMS~\cite{he2019bounding} &2) &\xmark &\xmark &\cmark &$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\\cmidrule{1-6}
GnetNMS~\cite{hosang2017learning} &4) &\cmark &\cmark &\cmark &Network architecture related \\\cmidrule{1-6}
IoU-Net~\cite{jiang2018acquisition} &1) &\xmark &\cmark &\cmark &$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\\cmidrule{1-6}
RelationNetNMS~\cite{hu2018relation} &4) &\cmark &\cmark &\cmark & Nework architecture related \\\midrule
% \bottomrule
\end{tabular}
\label{tab:nms_comparing}
\vspace{-5mm}
\end{table*}
\begin{table}[!htp]\centering
% \small
\caption{Comparing Softmax algorithms in the literature}\label{tab:softmax-alg-lit }
\vspace{-5pt}
\scriptsize
\begin{tabular}{lcl}\toprule
Methods &Types & Stage \\\cmidrule{1-3}
GPUSoftmax~\cite{joulin2017efficient} & 1) & train \\\cmidrule{1-3} %&$d\sqrt{k}$ \\\cmidrule{1-4}
SvdSoftmax~\cite{shim2017svd} & 3) & inference \\\cmidrule{1-3}
AdaptiveSoftmax~\cite{blanc2018adaptive} & 2) & train \\\cmidrule{1-3}
SOFT~\cite{lu2021soft} & 5) & train \& inference \\\cmidrule{1-3}
ANNSoftmax~\cite{zhao2021ann} & 2) &train \\\cmidrule{1-3}
Sigsoftmax~\cite{chen2018learning} & 5) & train \& inference \\\cmidrule{1-3}
L2S~\cite{chen2018learning} & 1) &inference \\\cmidrule{1-3}
DS~\cite{liao2019doubly} & 1) &inference \\\cmidrule{1-3}
RF-Softmax~\cite{rawat2019sampled} & 2) & train \\\cmidrule{1-3}
RegularizerSoftmax~\cite{andreas2015accuracy} & 4) & inference\\\cmidrule{1-3}
% \bottomrule
\end{tabular}
\label{tab:softmax}
\vspace{-5mm}
\end{table}
\subsubsection{Softmax}
\label{sec:softmax}
Softmax is a general operation to predict the class probability in many tasks. Its equation is $\sigma(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$
% \begin{equation}
% \label{eq:softmax}
% \sigma(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}
% \end{equation}
where $z_i$ denotes the $i$-th element of the input vector $\mathbf{z}$. $K$ denotes the number of classes. Softmax involves expensive non-linear operations such as exponentials and divisions, and its computational cost can become prohibitively expensive when dealing with a large number of classes.
%Mathematically, Softmax involves non-linear operations such as exponentials and divisions. %: exponential operations and division operations.
%Besides, when the number of classes $K$ is large, the computational cost will be prohibitive as we compute the exponential operations across all classes.
% However, when $K$ is large, the computational cost becomes prohibitively expensive as we need to compute the exponential function across all classes.
%Many researchers from the NLP domain attempt to speed up the softmax computation without modifying the softmax equation, as the computation of the softmax frequently dominates the complexity of neural networks when the vocabulary is enormous.
%In the NLP domain, researchers have explored various methods to speed up the computation of the softmax function, which can dominate the complexity of neural networks, especially when dealing with enormous vocabularies.
Researchers have developed methods to speed up softmax computation in Natural Language Processing (NLP), as it can dominate the complexity of neural networks with large vocabularies.
Table~\ref{tab:softmax} summarizes %shows
various softmax approximation methods in the literature, which can be categorized into five types: %. In particular, we summarize them into five types:
1) \textbf{class-based hierarchical softmax}~\cite{joulin2017efficient}: . A tree structure is constructed to estimate the softmax value along the tree depth, avoiding the need to traverse the whole vocabulary. %instead of traversing the whole vocabulary. %For each word's computation, its complexity is $d*h$ where $d$ is the dimension of the input vector, and $h$ is the depth of the leaf word node.
\textbf{2) sampling-based softmax}: it chooses a small subset of possible outputs and trains only with those. \textbf{3) differentiated softmax}: this restricts the effective parameters, using the fraction of the total output matrix. The matrix allocates a higher dimensional representation to frequent words and only a lower dimensional vector to rare words. \textbf{4) self-normalization}: it employs an additional training loss term~\cite{andreas2015accuracy}, which leads to a normalization factor close to 1. \textbf{5) softmax replacement}: a new probability computation function is proposed for accuracy improvement~\cite{chen2018learning} or computational saving~\cite{lu2021soft}.
%\textcolor{red}{table from chunyun}
% \begin{table}[t]
% \centering
% \scriptsize
% \caption{Shared properties among different activation functions}
% \label{tab:act_properties}
% \begin{tabular}{c|c|c|c|c}
% \hline
% Methods & Bounded? & Continuous? & Zero-centered? & Differentiable? \\
% \hline
% Sigmoid & Yes & Yes & No & Yes \\
% Tanh & Yes & Yes & Yes & Yes \\
% ReLU & No & Yes & No & Yes (not at x=0) \\
% LReLU & No & Yes & Yes & Yes(not at x=0) \\
% PReLU & No & Yes & Yes & Yes(not at x=0) \\
% \hline
% \end{tabular}
% \end{table}
% \begin{table}
% \scriptsize
% \centering
% \caption{Problems faced by different activation functions }
% \label{tab:act_problems}
% \begin{tabular}{c|c|c}
% \hline
% Function & dead neuron? & Vanishing gradient?\\
% \hline
% Logistic sigmoid & No & Yes \\
% Tanh & No & Yes \\
% ReLU & Yes & No \\
% LReLU & No & Partially \\
% PReLU & No & Partially \\
% \hline
% \end{tabular}
% \end{table}
\subsubsection{Activation functions}
\label{sec:activation_functions}
%The activation function (e.g., ReLU, PReLU, Sigmoid) is expected to have below properties:
%1) nonlinear 2) differentiable; 3) continuous; 4) bounded; 5) zero-centered. %Table \ref{tab:act_properties} shows the shared properties among different activation functions.
%The activation functions only with linear operations are easy to implement. In contrast, the activation functions with exponential or complex non-linear operations require linear approximations to speed up inference.
% Activation functions such as ReLU, PReLU, and Sigmoid are crucial components of neural networks. To be effective, an activation function should possess certain properties, including being nonlinear, differentiable, continuous, bounded, and zero-centered.
% While activation functions with linear operations are easy to implement, those with exponential or complex nonlinear operations may require linear approximations to speed up inference. Therefore, choosing an appropriate activation function depends on the specific requirements of the neural network and the computational resources available.
Activation functions like ReLU~\cite{krizhevsky2012imagenet}, PReLU~\cite{he2015delving}, and Sigmoid are crucial in neural networks and should be nonlinear, differentiable, continuous, bounded, and zero-centered. The implementation of activation functions with linear operations is easy, while exponential or complex nonlinear operations may require linear approximations for faster inference. Choosing an appropriate activation function depends on the specific neural network requirements and the available computational resources.
% \subsubsection{Summary}
To enhance the efficiency and performance of neural networks, the optimization of non-linear blocks, including activation functions like softmax and sigmoid, is a crucial focus of research. The challenges arise from the computational complexity of these functions, particularly in hardware implementations where simpler operations, such as those in convolutional layers, are more cost-effective. Recent efforts have been directed towards devising hardware-friendly activation functions, employing quantization techniques, utilizing look-up tables, and exploring fixed-point arithmetic. %Custom hardware accelerators, parallelization strategies, and hybrid quantization techniques are also being investigated to tailor solutions for specific non-linear operations.
These optimization endeavors aim to strike a balance between computational efficiency and model accuracy, facilitating the deployment of neural networks in diverse hardware environments, including edge devices and real-time systems.
% Optimizing non-linear blocks in neural networks is a complex task fraught with challenges. The computational intricacy of functions like softmax and sigmoid, particularly in comparison to simpler convolutional operations, demands specialized hardware-friendly solutions. Balancing precision and computational efficiency introduces trade-offs that can impact model accuracy. Memory bandwidth constraints and the dynamic nature of neural networks pose additional hurdles, as does the need to develop custom hardware accelerators and ensure adaptability to resource-constrained edge devices. Achieving a delicate equilibrium between accuracy and efficiency, addressing parallelization complexities, and meeting real-time constraints are ongoing challenges. Researchers are actively exploring innovative strategies to overcome these obstacles, aiming to make the optimization of non-linear blocks more practical and applicable across diverse hardware architectures and real-world applications.
% and Table \ref{tab:act_problems} shows the problems incurred by various activation functions.
%- \textbf{Sigmoid}
%\begin{equation}
% f(x) = \frac{\mathrm{1} }{\mathrm{1} + e^{-x}}
%\end{equation}
%
%- \textbf{ReLU}
%\begin{equation}
% f(x) = max(0, x)
%\end{equation}
%
%- \textbf{Leaky ReLU}
%\begin{equation}
%f(x) = \left\{\begin{matrix}
%0.01x & x \leqslant 0 \\
% x & \text{otherwise}
%\end{matrix}\right.
%\end{equation}
%
%- \textbf{PReLU}
%
%\begin{equation}
%f(x) = \left\{\begin{matrix}
%ax & x \leqslant 0 \\
% x & \text{otherwise}
%\end{matrix}\right.
%\end{equation}
%where $a$ is a learnable parameter.
% \vspace{-3.5mm}
\subsection{NAS and TinyML}
\label{sec:nas_tinyml}
% \textcolor{red}{TODO?}
\vspace{-2pt}
\subsubsection{NAS}
\label{subsec:nas}
The research on Neural Architecture Search (NAS) for efficient model design has yielded significant advancements in automating the creation of compact yet high-performing neural networks. Notable works such as EfficientNet~\cite{tan2019efficientnet}, ProxylessNAS~\cite{cai2018proxylessnas}, MnasNet~\cite{tan2019mnasnet}, FBNet~\cite{wu2019fbnet}, DARTS~\cite{liu2018darts} and Once-for-All~\cite{cai2019once}, showcase various approaches to NAS that consider efficiency in terms of model size, computational resources, and hardware constraints. These methods leverage techniques like gradient-based optimization~\cite{liu2018darts}, platform-awareness~\cite{wu2019fbnet}, parameter sharing~\cite{pham2018efficient, chen2021autoformer}, various network operations~\cite{li2021bossnas}, and specialized network training~\cite{cai2019once} to find architecture configurations that strike a balance between efficiency and performance.
% The research on Tiny Machine Learning (TinyML) has brought a lot of attention to deploy the deep learning models to resource-constrained edge devices such as micro-controllers and IoT devices. There is a growing body of literature and resources on this topic. The methods leverage techniques like considering the hardware constraints~\cite{ancilotto2023xinet} such as energy, RAM, and MAC. ~\cite{lin2021mcunetv2} attempt to balance the imbalanced distributions of memory in CNNs by patch-by-patch scheduling. ~\cite{cai2020tinytl} proposes to reduce the memory of activations in training by freezing the weights in transfer learning.
\subsubsection{TinyML}
\label{subsec:tinyml}
The study of Tiny Machine Learning (TinyML) has garnered significant attention due to its focus on deploying deep learning models on edge devices with limited resources, such as IoT devices. There is a growing body of literature and available resources in this field. These methods make use of various techniques, taking into account hardware constraints, including factors like energy consumption, Random Access Memory (RAM), and multiply–accumulate (MAC) operations. For example, Ancilotto \textit{et al.} discussed considerations related to hardware constraints in their work~\cite{ancilotto2023xinet}. Additionally, Lin \textit{et al.} addressed the issue of imbalanced memory distributions in Convolutional Neural Networks (CNNs) by implementing patch-by-patch scheduling~\cite{lin2021mcunetv2}. Cai \textit{et al.} proposed a method to reduce memory usage during training by freezing weights in transfer learning~\cite{cai2020tinytl}.
\section{When NN compression meets Hardware}
\label{sec:compression_hardware}
\subsection{Overview}
% \textcolor{red}{
% Linear: computation complexity, \# parameters.
% Non-linear: hardware friendly
% }
Performance, power, and area (PPA) are the three most important metrics in evaluating hardware cost.
Table II in Appendix E summarizes the quantitative and relative energy and area consumption of various operations such as adding and multiplication in different data formats.
%We observe that if it is required to gain high energy efficiency, the data locality should be good for one hardware since data reading is relatively expensive.
We observe that energy efficiency can be achieved through good data addressing, as data reading is relatively expensive.
It costs 166.7$\times$ and 1,666.7$\times$ more energy to read 32-bit data from an 8KB static random-access memory (SRAM) and 1MB SRAM than perform an 8-bit integer addition, respectively. % adding, respectively.
%Second,
In addition, quantizing float-point data to integers, from high precision (\eg, 32-bit) to low precision (\eg, 8-bit), reduces energy consumption. For instance, a 32-bit float-point addition %adding costs
requires 30$\times$ more energy than an 8-bit integer addition. %adding.
Finally, the trend of the area costs of different operations is
similar to their energy costs.
For the linear operations in DNNs, quantization, and pruning~\etc can be applied to reduce the number of weights or even activations, hence optimizing the PPA of the hardware.
For the non-linear operations, however, hardware-friendly approximation algorithms are needed to save the hardware cost.
% These algorithms can provide a good balance between accuracy and computational efficiency, allowing for implementing non-linear operations in resource-constrained environments.
Hardware accelerators generally target either Field Programmable Gate Arrays (FPGAs) or Application-Specific Integrated Circuits (ASICs). One accelerator can either be mapped into the FPGA flexibly for quick validation and further design space explorations or be fabricated as a high-performance ASIC chip when its functionalities and architecture are frozen and it is well-validated.
% \subsubsection{Linear ops}
% -- Backbone
% -- Quantization
% -- Pruning
% \subsubsection{Algorithms on Non-linear Operations}
% -- NMS
% -- Others
\subsection{Hardware Accelerating on Linear Operations}
\subsubsection{Hardware Accelerating on CNNs and Transformers}
\label{sec:cnns_transformers}
With the rapid advancements of DNNs, the corresponding hardware accelerators have become increasingly important.
Most CNN accelerators use direct convolution as their primary computation method, which accumulates the product of the inputs and corresponding weights in a certain dataflow.
Members of DianNao family~\cite{chen2014diannao,chen2014dadiannao,liu2015pudiannao} computes convolutions directly.
DianNao~\cite{chen2014diannao} exploited the locality properties of large-scale layers, achieving impressive results with only 3 mm$^2$ at 65 nm.
However, the memory wall becomes the bottleneck when computing classifier and convolutional layers with private kernels~\cite{chen2014dadiannao}.
To address this, DaDianNao~\cite{chen2014dadiannao} processed convolution with the data from nearby SRAM buffers and embedded dynamic random access memory (eDRAM) banks.
NVDLA~\cite{nvdla}, an open-source hardware inference accelerator, provided direct convolution mode. A wide multiply-accumulate (MAC) pipeline was implemented to parallel process the convolutional operations with memory bandwidth optimization.
Another accelerator, Maeri~\cite{kwon2018maeri} supported arbitrary dataflows by utilizing fine-grained reconfigurable tree-based interconnection network typologies to shape different sizes and numbers of virtual neurons.
By transforming the regular convolution into Generalized Matrix-matrix Multiplication (GEMM) or Generalized Matrix-vector Multiplication (GEMV), some matrix multiplication optimization algorithms, such as Strassen algorithm~\cite{strassen1969gaussian}, Winograd algorithm~\cite{winograd1980arithmetic} and Fast Fourier Transform (FFT)~\cite{brigham1967fast} can be applied to speed up the computation. For example,
Cong~\etal\cite{cong2014minimizing} applied the Strassen algorithm to the convolution transformed in GEMM and reduced the number of multiplications from $\mathcal{O}(N^3)$ to $\mathcal{O}(N^{2.807})$.
Lavin~\cite{lavin2016fast} and NVDLA~\cite{nvdla} applied Winograd convolution to reduce the number of multiplications while increasing the adders for the transformation.
The FFT algorithm converts the data to the more computationally efficient Fourier domain.
Previous work~\cite{ben1997fast} explored the application of FFT to accelerate inference, while
more recent works have attempted to map FFT on GPU platforms~\cite{mathieu2013fast,vasilache2014fast}.
The architecture proposed by Liang~\etal\cite{liang2019evaluating} also supported the FFT algorithm.
Over the past few years, the Transformer model has replaced Recurrent Neural Networks (RNNs) and CNNs in the Natural Language Processing (NLP) and Computer Vision (CV) domains.
However, Transformer models have different computation patterns than traditional neural networks, with more weights and computations, necessitating careful hardware accelerator design.
%However, the computation patterns of Transformer models differ from those of traditional neural networks. %are different between the traditional neural networks (e.g., CNNs and RNNs) and Transformer models.
%Specifically, the transformer models have more weights and computations, so they require careful hardware accelerator design. %designing the hardware accelerators carefully.
The hardware accelerating solutions for Transformers are reported in~\cite{lu2020hardware,ham20203,hu2021vis,sun2022vaqf,yu2022nn,marchisio2023swifttron,nag2023vita}.
The accelerator for the multi-head attention (MHA) ResBlock and the position-wise feed-forward network (FFN) ResBlock in the NLP Transformer was first reported by Lu~\etal~\cite{lu2020hardware}.
In A$^3$~\cite{ham20203}, an approximate method was applied for efficient attention computation.
Vis-TOP~\cite{hu2021vis} focused on accelerating Visual Transformers (ViTs) and presented a customized hardware architecture for a three-layer, two-level basic Transformer. % three-layer and two-level basic transformer architecture with customized hardware.
VAQF~\cite{sun2022vaqf} supported quantized ViTs with binary weights and low-bit activations on FPGA for inference.
SwiftTron~\cite{marchisio2023swifttron} directly mapped the ViT on ASIC with an area of 273 mm$^2$ and power of 33.64 W, adopting approximated non-linear functions proposed by NN-LUT~\cite{yu2022nn}, where they were computed via Piece-wise Linear Functions.
The accelerator introduced by Nag~\etal~\cite{nag2023vita} targeted FPGAs with dedicated computation blocks for the GEMMs in the MHA.
%To summarize, hardware accelerators for neural networks can use direct convolution or employ optimization algorithms like Strassen or Winograd to speed up computation. Furthermore, specialized hardware accelerators have been developed for the Transformer models.
In summary, hardware accelerators for DNNs can use direct convolution or optimization algorithms like Strassen or Winograd to accelerate computation. Furthermore, specialized accelerators have been created for Transformer models.
\subsubsection{Accelerating Networks after Quantization: Mixed-Precision and Entropy Coding}
\label{sec:accelerator_quantization}
% \ref{sec:cnns_transformers}
The bit width of weights in different layers or channels can be different to balance model precision and size.
%achieve a trade-off between precision and model size.
Table~\ref {tab:hw-mix-pre} summarizes the popular hardware accelerators that support multiple precision data formats. In~\cite{yin20171}, 8-bit and 16-bit fixed-point NN layers were supported.
Zhang~\etal~\cite{zhang2019new} proposed a new flexible unit that supports five different precision in training and inference.
LNPU~\cite{lee20197} was a %very high energy efficiency
highly energy-efficient
DNN accelerator that supports training and inference, %whose energy efficiency can be up to 25.3 TFLOPS/W
achieving up to 25.3 TFLOPS/W energy efficiency
when processing a highly sparse weight layer in 8-bit float-point data format.
Zhou~\etal~\cite{zhou2020convolutional} proposed a CNN accelerator that supports mixed precision computations both within-layer and layer-wise.
By introducing a fine-grained mixed precision unit, %within-layer mixed-precision operations are enabled. Hence this hardware reduces nearly 50\% of its computation area and around 12.1\% dynamic power in AlexNet and VGG16 compared with the baseline.
the hardware enables within-layer mixed-precision operations, which reduces computation area by nearly 50\% and dynamic power by around 12.1\% in AlexNet and VGG16 compared to the baseline.
Some works exploit the low-bit precision such as 4-bit or sub-4-bit.
The accelerator proposed by Fleischer~\etal~\cite{fleischer2018scalable} supported very low-precision data formats such as binary and ternary for aggressive inference performance.
In addition, the 16-bit float-point data format is supported for high accuracy in training, hence achieving 24 Tera Operations Per Second (TOPS) inference performance in a $9~mm^2$ chip.
Wang~\etal~\cite{wang2018design} introduced a flexible design flow that supports mixed-precision computations of hybrid neural networks with quantized activation and quantized weights targeting FPGAs.
With parameterized computation engines, the introduced AccELB performs binary and ternary convolution without multiplications.
The flexible 4-core AI chip introduced by Agrawal~\etal~\cite{agrawal20219} supported both training and inference with multiple precisions, \ie, INT2, INT4, hybrid-FP8, FP16, and FP32 with a performance of up to 102.4 TOPS.
\begin{table*}[!t]
\centering
\scriptsize
\caption{A Summary of mixed-precision hardware and their compute density (TOPS/W or TOPS/mm$^2$).}
\vspace{-5pt}
\label{tab:hw-mix-pre}
\resizebox{\textwidth}{!}{%
\begin{threeparttable}
\begin{tabular}{l|c|c|c|c|c|c|c|c}
\hline
Methods &
\begin{tabular}[c]{@{}c@{}}Supporting\\ Format\end{tabular} &
Platform &
Area &
Technology &
\begin{tabular}[c]{@{}c@{}}Inference\\ Performance\end{tabular} &
Frequency &
\begin{tabular}[c]{@{}c@{}}Training or \\ Inference\end{tabular} &
\begin{tabular}[c]{@{}c@{}}Compute Density\\ (TOPS/W or TOPS/mm$^2$)\end{tabular}\\ \hline
Yin~\etal~\cite{yin20171} &
INT8, INT16 &
ASIC &
19.36 mm$^2$ &
65 nm lp &
368.4 GPOS &
200 MHz &
Inference &
1.28 TOPS/W \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}l@{}}Zhang~\etal\\ ~\cite{zhang2019new}~\tnote{a}\end{tabular}}&
INT4 &
\multirow{5}{*}{ASIC} &
\multirow{5}{*}{2943 $\mu\text{m}^2$} &
\multirow{5}{*}{28 nm} &
29.6 GOPS / unit &
\multirow{5}{*}{3.7 GHz} &
\multirow{5}{*}{Both} &
4.81 TOPS/W \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
INT8 &
&
&
&
14.8 GOPS / unit &
&
&
2.09 TOPS/W \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
INT16 &
&
&
&
7.4 GOPS / unit &
&
&
1.01 TOPS/W \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
FP8 &
&
&
&
14.8 GFLOPS / unit &
&
&
1.10 TFLOPS/W \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
FP16 &
&
&
&
7.4 GFLOPS / unit &
&
&
0.55 TFLOPS/W \\ \hline
\multirow{2}{*}{LNPU~\cite{lee20197}} &
FP8 &
\multirow{2}{*}{ASIC} &
\multirow{2}{*}{16 mm$^2$} &
\multirow{2}{*}{65 nm} &
600 GFLOPS &
\multirow{2}{*}{200 MHz} &
\multirow{2}{*}{Both} &
3.48 TFLOPS/W (FP8, 0\% sparsity) \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
FP16 &
&
&
&
300 GFOPS &
&
&
25.3 TFLOPS/W (FP8, 90\% sparsity) \\ \hline
Zhou~\etal~\cite{zhou2020convolutional} &
INT8, INT16 &
FPGA &
-- &
-- &
-- &
-- &
Inference &
-- \\ \hline
\multirow{3}{*}{\begin{tabular}[c]{@{}l@{}}Fleischer~\etal\\ ~\cite{fleischer2018scalable}~\tnote{b}\end{tabular}} &
Binary &
\multirow{3}{*}{ASIC} &
\multirow{3}{*}{9 mm$^2$} &
\multirow{3}{*}{14 nm} &
24 TOPS &
\multirow{3}{*}{1.5 GHz} &
\multirow{3}{*}{Both} &
2.67 TOPS/mm$^2$ \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
Ternary &
&
&
&
12 TOPS &
&
&
1.33 TOPS/mm$^2$ \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
FP16 &
&
&
&
1.5 TFLOPS &
&
&
0.17 TOPS/mm$^2$ \\ \hline
AccELB~\cite{wang2018design} &
\begin{tabular}[c]{@{}c@{}}Hybrid-INT1, \\ INT2, INT4, INT8\end{tabular} &
FPGA &
-- &
-- &
0.49-10.3 TOPS &
200 MHz &
Inference &
-- \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}l@{}}Agrawal~\etal\\~\cite{agrawal20219}\end{tabular}} &
INT2 &
\multirow{5}{*}{ASIC} &
\multirow{5}{*}{16 mm$^2$} &
\multirow{5}{*}{7 nm} &
-- &
\multirow{5}{*}{1.0-1.6 GHz} &
\multirow{5}{*}{Both} &
-- \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
INT4 &
&
&
&
64-102.4 TOPS &
&
&
8.9-16.5 TOPS/W \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
Hybrid-FP8 &
&
&
&
16-25.6 TFLOPS &
&
&
1.9-3.5 TFLOPS/W \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
FP16 &
&
&
&
8-12.8 TFLOPS &
&
&
0.98-1.8 TFLOPS/W \\ \cline{2-2} \cline{6-6} \cline{9-9}
&
FP32 &
&
&
&
-- &
&
&
-- \\ \hline
\end{tabular}%
\begin{tablenotes}
\scriptsize
\item[a] This work proposes one computation unit. Hence the frequency is computed by $1/\text{delay}$, where the delay is reported. The inference performance is computed by $2 \times \text{\#parallel op } \times \text{frequency}$, where the number of parallel operations in one unit is reported. The energy efficiency is computed by $1/\text{energy \#op}$, where the energy per operation is reported.
\item[b] The energy efficiency is not reported; area efficiency is reported instead.
\end{tablenotes}
\end{threeparttable}
}
\vspace{-5mm}
\end{table*}
Besides quantization, entropy coding can lower hardware costs while maintaining accuracy by encoding weights in a more condensed representation, resulting in fewer bits per variable~\cite{recanatesi2019dimensionality}. This is accomplished by taking advantage of the peaked distribution of quantized values.
%entropy coding can reduce hardware cost without compromising accuracy %any accuracy loss
%by encoding the weights in a more compact representation, achieving a lower number of bits per variable~\cite{recanatesi2019dimensionality} by utilizing the peaky distribution of the quantized values.
The coding schemes are classified into Fixed-to-Variable (F2V) and Variable-to-Fixed (V2F) entropy coding methods.
F2V coding methods, such as arithmetic coding~\cite{oktay2019scalable,wiedemann2019deepcabac} and Huffman coding~\cite{dubey2018coreset,han2015deep}, encode a fixed number of symbols into variable-length codewords. %a variable length of the codeword.
Hence, it is challenging for the F2V coding methods to decode parallelly.
F2V coding methods have very high computational complexity for decoding.
%Given the number of codewords (quantized values) $n$ and the reciprocal compression ratio $k$, the decoding complexity is $O(n \cdot k)$.
The decoding complexity of F2V coding methods is $O(n \cdot k)$, where $n$ is the number of codewords, and $k$ is the reciprocal compression ratio. Conversely, V2F coding methods, such as Tunstall coding~\cite{chen2021efficient}, encode multiple symbols to a fixed number of bits.
During the decoding stage, it is feasible to process multiple bits simultaneously and decode multiple symbols per clock cycle. Additionally, parallel decoding of the encoded string is possible by dividing it into bit chunks of fixed length according to the codeword length.
%In the decoding stage, it is possible to process multiple bits each time and decode multiple symbols per clock cycle. Furthermore, decoding the encoded string in parallel is also feasible, as the encoded string can be split into fixed-length bit chunks based on the length of codewords.
% What is Huffman coding
%In the Huffman coding algorithm, all symbols are first listed in decreasing order of their probabilities. A Huffman tree is then %Then a Huffman tree is
%constructed step by step, with each step adding two symbols with the smallest probabilities to the top of the partial Huffman tree and removing them from the list. % where each step adds two symbols with the smallest probabilities to the top of the partial Huffman tree and removes these two symbols from the list.
%Finally, an auxiliary symbol is added to the list to represent the two removed original symbols.
Huffman coding algorithm first lists symbols in decreasing order of their probabilities. Then, it constructs a Huffman tree by adding two symbols with the smallest probabilities at each step and removing them from the list. Finally, an auxiliary symbol is added to represent the two removed original symbols. Arithmetic coding assigns one code to the entire input instead of coding individual symbols like Huffman coding. It starts with a specified interval and symbolically reads the input while narrowing down the gap.
%Fig.~\ref{fig:huffman_tree} illustrates an example of the construction of the Huffman tree.
% In this example, we have three symbols \emph{A}, \emph{B}, and \emph{C} with probabilities 0.625, 0.25, and 0.125, respectively---these probabilities indicate the likelihood of a weight having this symbol.
% Firstly, \emph{A}, \emph{B} and \emph{C} are listed according to their probabilities (Fig.~\ref{fig:huffman_tree_a}). Then \emph{B} and \emph{C} are replaced by the combined symbol \emph{BC} with a probability of 0.375 (Fig.~\ref{fig:huffman_tree_b}). Finally, \emph{A} is combined with \emph{BC} and is added to the top of the partial tree (Fig.~\ref{fig:huffman_tree_c}).
% Table~\ref{tab:huffman_tree_codewords} lists the codebook of this example in Huffman coding.
% According to this codebook, sequence "AAABABAC" is encoded as a binary string "00010010011".
% \begin{figure}[t!]
% \centering
% \resizebox{0.48\columnwidth}{!}{
% \subfloat[]{
% \label{fig:huffman_tree_a}
% \centering
% \begin{tikzpicture}
% [
% level distance=10mm,
% sibling distance=15mm,
% selectednode/.style={rectangle, draw=red!60, fill=red!5, very thick, minimum size=7mm},
% unselectednode/.style={minimum size=7mm},
% ]
% \path[use as bounding box, draw=black!50, line width=0.1pt](-0.15\textwidth,-12ex) rectangle (0.15\textwidth,6ex);
% \node[selectednode] (node_b) {\emph{B}};
% \node[] (node_b_prob) [above=1mm of node_b] {0.25};
% \node[unselectednode] (node_a) [left=of node_b] {\emph{A}};
% \node[] (node_a_prob) [above=1mm of node_a] {0.625};
% \node[selectednode] (node_c) [right=of node_b] {\emph{C}};
% \node[] (node_c_prob) [above=1mm of node_c] {0.125};
% \end{tikzpicture}
% }
% }
% \resizebox{0.48\columnwidth}{!}{
% \subfloat[]{
% \label{fig:huffman_tree_b}
% \centering
% \begin{tikzpicture}
% [
% level distance=8mm,
% sibling distance=2cm,
% selectednode/.style={rectangle, draw=red!60, fill=red!5, very thick, minimum size=7mm},
% unselectednode/.style={minimum size=7mm},
% ]
% \path[use as bounding box, draw=black!50, line width=0.1pt](-0.2\textwidth,-12ex) rectangle (0.1\textwidth,6ex);
% \node[selectednode] (node_bc) {\emph{BC}}
% child {
% [fill] circle (2pt)
% node[below] {\emph{B}}
% edge from parent
% node[left,above] {0}
% }
% child {
% [fill] circle (2pt)
% node[below] {\emph{C}}
% edge from parent
% node[right,above] {1}
% }
% ;
% \node[] (node_bc_prob) [above=1mm of node_bc] {0.375};
% \node[selectednode] (node_a) [left=of node_bc] {\emph{A}};
% \node[] (node_a_prob) [above=1mm of node_a] {0.625};
% \end{tikzpicture}
% }
% }
% \hfill
% \resizebox{0.48\columnwidth}{!}{
% \subfloat[]{
% \label{fig:huffman_tree_c}
% \centering
% \begin{tikzpicture}
% [
% level distance=8mm,
% sibling distance=2cm,
% selectednode/.style={rectangle, draw=red!60, fill=red!5, very thick, minimum size=7mm},
% unselectednode/.style={minimum size=7mm},
% ]
% \path[use as bounding box, draw=black!50, line width=0.1pt](-0.12\textwidth,-15ex) rectangle (0.18\textwidth,6ex);
% \node[selectednode] (node_abc) {\emph{ABC}}
% child {
% [fill] circle (2pt)
% node[below] {\emph{A}}
% edge from parent
% node[left,above] {0}
% }
% child {
% [fill] circle (2pt)
% child {
% [fill] circle (2pt)
% node[below] {\emph{B}}
% edge from parent
% node[left,above] {0}
% }
% child {
% [fill] circle (2pt)
% node[below] {\emph{C}}
% edge from parent
% node[right,above] {1}
% }
% edge from parent
% node[right,above] {1}
% }
% ;
% \node[] (node_abc_prob) [above=1mm of node_abc] {1};
% \end{tikzpicture}
% }
% }
% \caption{An example of the Huffman tree}
% \label{fig:huffman_tree}
% \end{figure}
% \begin{table}[t!]
% \centering
% \scriptsize
% \caption{The codebook of the example Huffman tree}
% \label{tab:huffman_tree_codewords}
% \resizebox{0.6\columnwidth}{!}{%
% \begin{tabular}{|c|c|c|}
% \hline
% \textbf{Probability} & \textbf{Symbol} & \textbf{Codeword} \\ \hline
% 0.625 & \textit{A} & 0 \\ \hline
% 0.250 & \textit{B} & 10 \\ \hline
% 0.125 & \textit{C} & 11 \\ \hline
% \end{tabular}%
% }
% \end{table}
% What is Arithmetic coding
%As Huffman coding does, arithmetic coding assigns one code to the entire input instead of separating the inputs into individual symbols and replacing each with an integral number of bits.
%This coding method starts with a specified interval; then, the inputs are read symbolically while the gap is narrowed down.
%For instance, three symbols \emph{A}, \emph{B}, and \emph{C} with probabilities 0.625, 0.25, and 0.125, are assigned the subintervals $\left(0, \frac{5}{8}\right]$, $\left(\frac{5}{8}, \frac{7}{8}\right]$ and $\left(\frac{7}{8}, 1\right]$.
%As Fig.~\ref{fig:arithmetic_line} shows, to encode the string "AAABABAC", we start with the interval $\left(0, 1\right]$. The first symbol \emph{A} reduces the subinterval from its 0 point to 62.5\% point, whose result is $\left(0, \frac{5}{8}\right]$. The second \emph{A} reduces $\left(0, \frac{5}{8}\right]$ in the same way to $\left(0, \frac{5^2}{2^6}\right]$, then $\left(0, \frac{5^3}{2^9}\right]$. Step by step, the last \emph{C} narrows the subinterval from $\left(\frac{5^4 \times 37}{2^{17}}, \frac{5^5 \times 31}{2^{19}}\right]$ to $\left(\frac{5^4 \times 1233}{2^{22}}, \frac{5^5 \times 31}{2^{19}}\right]$. In this case, any number in the subinterval $\left(\frac{5^4 \times 1233}{2^{22}}, \frac{5^5 \times 31}{2^{19}}\right]$, such as $(0.0010111101)_2$ is able to represent the string "AAABABAC".
% \begin{figure}[t!]
% \centering
% \resizebox{\columnwidth}{!}{
% \begin{tikzpicture}
% \node[above] at (3.125, 0) {\emph{A}};
% \node[above] at (7.5, 0) {\emph{B}};
% \node[above] at (9.5, 0) {\emph{C}};
% \node[above] at (0, 0) {0};
% \node[above] at (6.25, 0) {$\frac{5}{8}$};
% \node[above] at (8.75, 0) {$\frac{7}{8}$};
% \node[above] at (10, 0) {1};
% \draw[color=yellow!66,line width=5] (0, 0) -- (6.25, 0);
% \draw[|-|] (0, 0) -- (6.25, 0);
% \draw[-|] (6.25, 0) -- (8.75, 0);
% \draw[-|] (8.75, 0) -- (10, 0);
% \fill (1.845, 0) circle (2pt);
% % To the next symbol
% \draw (0.01, -0.2) -- (0.01, -0.8);
% \draw (6.25, -0.2) -- (10, -0.8);
% \node[above] at (0, -1.5) {0};
% \node[above] at (6.25, -1.5) {$\frac{25}{64}$};
% \node[above] at (8.75, -1.5) {$\frac{35}{64}$};
% \node[above] at (10, -1.5) {$\frac{5}{8}$};
% \draw[color=yellow!66,line width=5] (0, -1.5) -- (6.25, -1.5);
% \draw[|-|] (0, -1.5) -- (6.25, -1.5);
% \draw[-|] (6.25, -1.5) -- (8.75, -1.5);
% \draw[-|] (8.75, -1.5) -- (10, -1.5);
% \fill (2.956, -1.5) circle (2pt);
% % To the next symbol
% \draw (0.01, -1.7) -- (0.01, -2.3);
% \draw (6.25, -1.7) -- (10, -2.3);
% \node[above] at (0, -3) {0};
% \node[above] at (6.25, -3) {$\frac{125}{512}$};
% \node[above] at (8.75, -3) {$\frac{175}{512}$};
% \node[above] at (10, -3) {$\frac{25}{64}$};
% \draw[color=yellow!66,line width=5] (0, -3) -- (6.25, -3);
% \draw[|-|] (0, -3) -- (6.25, -3);
% \draw[-|] (6.25, -3) -- (8.75, -3);
% \draw[-|] (8.75, -3) -- (10, -3);
% \fill (4.73, -3) circle (2pt);
% % To the next symbol
% \draw (0.01, -3.2) -- (0.01, -3.8);
% \draw (6.25, -3.2) -- (10, -3.8);
% \node[above] at (0, -4.5) {0};
% \node[above] at (6.25, -4.5) {$\frac{625}{4096}$};
% \node[above] at (8.75, -4.5) {$\frac{875}{4096}$};
% \node[above] at (10, -4.5) {$\frac{125}{512}$};
% \draw[color=yellow!66,line width=5] (6.25, -4.5) -- (8.75, -4.5);
% \draw[|-|] (0, -4.5) -- (6.25, -4.5);
% \draw[-|] (6.25, -4.5) -- (8.75, -4.5);
% \draw[-|] (8.75, -4.5) -- (10, -4.5);
% \fill (7.568, -4.5) circle (2pt);
% % To the next symbol
% \draw (6.25, -4.7) -- (0.01, -5.3);
% \draw (8.75, -4.7) -- (10, -5.3);
% \node[above] at (0, -6) {$\frac{625}{4096}$};
% \node[above] at (6.25, -6) {$\frac{6250}{32768}$};
% \node[above] at (8.75, -6) {$\frac{6750}{32768}$};
% \node[above] at (10, -6) {$\frac{875}{4096}$};
% \draw[color=yellow!66,line width=5] (0, -6) -- (6.25, -6);
% \draw[|-|] (0, -6) -- (6.25, -6);
% \draw[-|] (6.25, -6) -- (8.75, -6);
% \draw[-|] (8.75, -6) -- (10, -6);
% \fill (5.273, -6) circle (2pt);
% \end{tikzpicture}
% }
% \caption{The first five steps of the Arithmetic Coding example}
% \label{fig:arithmetic_line}
% \end{figure}
% Review of Huffman coding
In Deep Compression~\cite{han2015deep} and Coreset-base Compression~\cite{dubey2018coreset}, Huffman coding~\cite{cormen2009introduction} was used to compress the quantized weights.
% Review of Arithmetic coding
%In~\cite{oktay2019scalable} and~\cite{wiedemann2019deepcabac}, another Fixed-to-Variable (F2V)~\cite{rabbani2002jpeg2000} coding method---arithmetic coding~\cite{witten1987arithmetic}---was adopted.
However, in~\cite{oktay2019scalable} and~\cite{wiedemann2019deepcabac}, another Fixed-to-Variable (F2V)\cite{rabbani2002jpeg2000} coding method, arithmetic coding\cite{witten1987arithmetic}, was used.
% However
Nevertheless, %the decoding stage of the F2V methods is inefficient because the encoded string is processed on a bit-by-bit basis.
%Moreover, it is challenging to develop parallel implementations for the decoding; F2V codewords are of variable length and can not be indexed (or memory usage would be very inefficient).
it is challenging to develop parallel implementations for decoding, as F2V codewords are of variable length and cannot be indexed efficiently, making in-efficient indexing and preventing the decoding of multiple symbols per clock cycle. %. As a result, we cannot decode multiple symbols per single clock cycle.
In~\cite{chen2021efficient}, the Tunstall coding method was to compress ResNet and MobileNet after compression, where two Tunstall hardware decoders are designed and compared. Table~\ref{tab:hw-coding} summarizes these coding methods. Table III in Appendix E~\cite{chen2021efficient} reports the 256 entries Huffman hardware decoder consumes $3.14\times$ more hardware resources and costs $6.23\times$ more time to decode the same amount of weights.
%Fig.~\ref{fig:tunstall_tree} draws an example of the construction of the Tunstall tree. For instance, three symbols \emph{A}, \emph{B}, and \emph{C} with probabilities 0.625, 0.25, and 0.125, are assigned the subintervals $\left(0, \frac{5}{8}\right]$, $\left(\frac{5}{8}, \frac{7}{8}\right]$ and $\left(\frac{7}{8}, 1\right]$. %with the same symbols as the previous section.
%We construct a 3-bit Tunstall tree to encode the symbols.
% Specifically, the construction of the Tunstall tree is performed iteratively.
% We start with a single symbol at the root with probability 1.0.
% For each iteration, the leaf node with the highest probability will be selected and converted to a subtree with $K$ children, where $K$ is the total number of symbols and $K=3$ in this example.
% For instance, at the second iteration (Fig.~\ref{fig:tunstall_tree_b}), the leaf node \emph{A}, which has the most significant probability 0.625 (Fig.~\ref{fig:tunstall_tree_a}), is selected and converted into a subtree with children \emph{AA}, \emph{AB}, and \emph{AC}.
% The third iteration converts leaf node \emph{AA}, whose probability is 0.391, to the father node of a new subtree.
% The iteration continues until the number of leaf nodes reaches $2^{C}$, where $C$ is the bit size of the Tunstall codewords and $C=3$ here.
% The codebook of this example is listed in Table \ref{tab:tunstall_tree_codewords}.
% According to this codebook, the sequence "AAABABAC" is encoded as a binary string "000101011100".
% \begin{figure}[t!]
% \centering
% \resizebox{0.3\columnwidth}{!}{
% \subfloat[]{
% \label{fig:tunstall_tree_a}
% \centering
% \begin{tikzpicture}
% [level distance=7mm, sibling distance=12mm]
% \node[token]{}
% child {
% [fill] circle (2pt)
% node[below] {0.625}
% edge from parent
% node[left] {\emph{A}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.25}
% edge from parent
% node[left] {\emph{B}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.125}
% edge from parent
% node[right] {\emph{C}}
% }
% ;
% \end{tikzpicture}
% }
% }
% \resizebox{0.4\columnwidth}{!}{
% \subfloat[]{
% \label{fig:tunstall_tree_b}
% \centering
% \begin{tikzpicture}
% [level distance=7mm, sibling distance=12mm]
% \node[token]{}
% child {
% [fill] circle (2pt)
% child {
% [fill] circle (2pt)
% node[below] {0.391}
% edge from parent
% node[left] {\emph{A}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.156}
% edge from parent
% node[left] {\emph{B}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.078}
% edge from parent
% node[left] {\emph{C}}
% }
% edge from parent
% node[left] {\emph{A}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.25}
% edge from parent
% node[left] {\emph{B}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.125}
% edge from parent
% node[right] {\emph{C}}
% }
% ;
% \end{tikzpicture}
% }
% }
% \hfill
% \resizebox{0.5\columnwidth}{!}{
% \subfloat[]{
% \label{fig:tunstall_tree_c}
% \centering
% \begin{tikzpicture}
% [level distance=8mm, sibling distance=12mm]
% \node[token]{}
% child {
% [fill] circle (2pt)
% child {
% [fill] circle (2pt)
% child {
% [fill] circle (2pt)
% node[below] {0.244}
% edge from parent
% node[left] {\emph{A}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.098}
% edge from parent
% node[left] {\emph{B}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.049}
% edge from parent
% node[left] {\emph{C}}
% }
% edge from parent
% node[left] {\emph{A}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.156}
% edge from parent
% node[left] {\emph{B}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.078}
% edge from parent
% node[left] {\emph{C}}
% }
% edge from parent
% node[left] {\emph{A}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.25}
% edge from parent
% node[left] {\emph{B}}
% }
% child {
% [fill] circle (2pt)
% node[below] {0.125}
% edge from parent
% node[right] {\emph{C}}
% }
% ;
% \end{tikzpicture}
% }
% }
% \caption{An example of the Tunstall tree}
% \label{fig:tunstall_tree}
% \end{figure}
% \begin{table}[t!]
% \scriptsize
% \caption{The codebook of the example Tunstall tree}
% \label{tab:tunstall_tree_codewords}
% \centering
% \resizebox{0.8\columnwidth}{!}{%
% \begin{tabular}{|c|c|c|c|c|c|c|c|}
% \hline
% \textbf{Codeword} & \textbf{000} & \textbf{001} & \textbf{010} & \textbf{011} & \textbf{100} & \textbf{101} & \textbf{110} \\ \hline
% \textbf{Symbols} & \textit{AAA} & \textit{AAB} & \textit{AAC} & \textit{AB} & \textit{AC} & \textit{B} & \textit{C} \\ \hline
% \end{tabular}%
% }
% \end{table}
\begin{table}[t!]
\scriptsize
\centering
\caption{The hardware implementations of various coding methods and the size of the ResNet-50 after compression.}
\vspace{-5pt}
\label{tab:hw-coding}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{l|c|c|c|c|c}
\hline
Methods &
\begin{tabular}[c]{@{}c@{}}Coding \\ Method\end{tabular} &
\begin{tabular}[c]{@{}c@{}}Coding \\ Schemes\end{tabular} &
Parallelable &
\begin{tabular}[c]{@{}c@{}}Memory \\ Aligned\end{tabular} &
\begin{tabular}[c]{@{}c@{}}Size After\\ Compression (MB)\end{tabular} \\ \hline
\begin{tabular}[c]{@{}l@{}}Deep \\ Compression ~\cite{han2015deep}\end{tabular} & Huffman & F2V & \xmark & \xmark & 6.06 \\ \hline
\begin{tabular}[c]{@{}l@{}}Coreset-base \\ Compression~\cite{dubey2018coreset}\end{tabular} & Huffman & F2V & \xmark & \xmark & 6.46 \\ \hline
Oktay~\etal~\cite{oktay2019scalable} & Arithmetic & F2V & \xmark & \xmark & 5.49 \\ \hline
Deepcabac~\cite{wiedemann2019deepcabac} & Arithmetic & F2V & \xmark & \xmark & 6.06 \\ \hline
Chen~\etal~\cite{chen2021efficient} & Tunstall & V2F & \cmark & \cmark & 5.85 \\ \hline
\end{tabular}%
}
\vspace{-5mm}
\end{table}
% \vspace{-0.3cm}
%\begin{table*}[t!]
%\scriptsize
%\centering
%\caption{The coding methods are applied in hardware implementations.}
%\label{tab:hw-coding}
%\resizebox{\textwidth}{!}{%
%\begin{tabular}{|c|c|c|c|c|c|}
%\hline
%\textbf{Work} &
% \textbf{\begin{tabular}[c]{@{}c@{}}Coding \\ Method\end{tabular}} &
% \textbf{\begin{tabular}[c]{@{}c@{}}Coding \\ Schemes\end{tabular}} &
% \textbf{Parallelable?} &
% \textbf{Memory Aligned} &
% \textbf{\begin{tabular}[c]{@{}c@{}}Size After Compression\\ (ResNet-50) (MB)\end{tabular}} \\ \hline
%\textit{\textbf{Deep Compression~\cite{han2015deep}}} & Huffman & F2V & \xmark & \xmark & 6.06 \\ \hline
%\textit{\textbf{Coreset-base Compression~\cite{dubey2018coreset}}} & Huffman & F2V & \xmark & \xmark & 6.46 \\ \hline
%\textit{\textbf{Oktay~\etal~\cite{oktay2019scalable}}} & Arithmetic & F2V & \xmark & \xmark & 5.49 \\ \hline
%\textit{\textbf{Deepcabac~\cite{wiedemann2019deepcabac}}} & Arithmetic & F2V & \xmark & \xmark & 6.06 \\ \hline
%\textit{\textbf{Chen~\etal~\cite{chen2021efficient}}} & Tunstall & V2F & \cmark & \cmark & 5.85 \\ \hline
%\end{tabular}%
%}
%\end{table*}
% Table~\ref{tab:tun_huf_cmp}~\cite{chen2021efficient} summaries the detailed theoretical hardware performance of the 7-stage 24 memory banks Logic-Oriented Tunstall decoder and the 1024 entries Memory-Oriented Tunstall decoder with the 256 entries Huffman decoder. The Huffman decoder decodes one weight per clock cycle.
% All implementations are synthesized and mapped to FPGA.
% As summarised, the 256 entries Huffman hardware decoder consumes $3.14\times$ more hardware resources and costs $6.23\times$ more time to decode the same amount of weights.
In summary, mixed-precision and entropy coding are adopted by hardware accelerators after the quantization of the DNNs for higher compute density and performance. However, aggressive quantization would introduce significant accuracy loss.
Instead, entropy coding maintains accuracy while compressing the model.
However, F2V coding methods challenge the parallel decoding on the hardware, which introduces extra overhead to the throughput, while the decoding process of the V2F coding methods, such as the Tunstall coding, is hardware-friendly and preferred.
\subsubsection{Accelerating Networks after Pruning: Sparse Architecture}
\label{sec:accelerator_pruning}
To reduce the energy efficiency significantly and boost computation speed with fewer computation units, recently works~\cite{lee20197,chen2016eyeriss,han2016eie,parashar2017scnn,chen2019eyeriss} proposed accelerators that support sparse neural networks.
As summarized in~\cite{dave2021hardware}, there are five hardware accelerating methods for sparse neural networks:
\begin{itemize}
\item Storing the compressed data in off-chip memory to optimize the memory capacity requirement and improve energy efficiency;
\item Storing the compressed data in on-chip memory to optimize the memory capacity requirement and improve energy efficiency;
\item Skipping zero elements to improve energy efficiency;
\item Reducing ineffectual computation cycles to improve performance and energy efficiency;
\item Balancing the workloads of different processing elements to increase performance;
\end{itemize}
Lee~\etal~\cite{lee20197} proposed an architecture called LNPU that supports sparse DNNs with fine-grained mixed precision of FP8-FP16.
Sparsity is exploited with intra- and inter-channel accumulation with the input load balancer (ILB) to alleviate the imbalanced workload problem caused by irregular sparsity, improving PE utilization.
Only non-zero weights are kept in the internal buffers in Cambricon-X~\cite{zhang2016cambricon}, while SCNN~\cite{parashar2017scnn} compressed both non-zero weights and activations in both dynamic random-access memory (DRAM) and internal buffers.
Sparse encoding methods can reduce memory access, increase energy efficiency, and accelerate computation time for accelerators by adapting them to sparse tensors. This is because most sparsity encoding methods only store non-zero elements.
%Sparse encoding methods can be adapted to a sparse tensor to reduce memory access, increase energy efficiency, and reduce the computation time of the accelerators, as only non-zero elements are stored in most sparsity encoding methods.
These encoding methods are Coordinate (COO), COO-1D, Run-length Coding (RLC), Bitmap, Compressed Sparse Row (CSR), Compressed Sparse Column (CSC)~\cite{gustavson1972some} and Compressed Sparse Fiber (CSF)~\cite{smith2015splatt}.
Table~\ref{tab:sparse-encodings-overhead} summarizes the storage overhead and decoding taxonomy for standard encoding methods suitable for sparse data~\cite{dave2021hardware} while Table~\ref{tab:hw-sparsity-encoding} summarizes the area and energy efficiency of the hardware accelerators that use sparsity encoding methods.
\begin{table}[t!]
\centering
\scriptsize
\caption{Storage overhead and decoding taxonomy for common encoding methods. Vector $d$ stores $n$ dimensions of a tensor that contains $NNZ$ non-zero elements~\cite{dave2021hardware}.}
\vspace{-5pt}
\label{tab:sparse-encodings-overhead}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{l|c|c}
\hline
Format &
Storage Overhead (bits) &
Decoding Taxonomy
% \begin{tabular}[c]{@{}c@{}}Decoding\\ Taxonomy\end{tabular}
\\ \hline
COO & $NNZ \times \sum_1^n { \lceil \log_2{d_i} \rceil}$ & Direct \\ \hline
COO-1D & $NNZ \times \lceil \log_2{\prod_1^n d_i} \rceil$ & Direct \\ \hline
RLC & $NNZ \times B$ & Single-step \\ \hline
Bitmap & $\prod_1^n d_i$ & Single-step \\ \hline
CSR & $NNZ \times \lceil \log_2{d_1} \rceil$ + $(d_0 + 1) \times \lfloor \log_2{NNZ} + 1 \rfloor$ & Double-step \\ \hline
CSC & $NNZ \times \lceil \log_2{d_0} \rceil$ + $(d_1 + 1) \times \lfloor \log_2{NNZ} + 1 \rfloor$ & Double-step \\ \hline
\end{tabular}%
}
\vspace{-4.5mm}
\end{table}
The COO sparsity encoding method was adopted in~\cite{hegde2019extensor,yuan2018sticker}.
COO-1D was utilized in~\cite{zhang2019snap, kang2019accelerator, lu2019efficient}.
RLC was used in~\cite{chen2016eyeriss, zhang2019compact, parashar2017scnn, lee20197}.
Bitmap was used in~\cite{zhang2019compact, yuan2018sticker, zhou2018cambricon, nvdla}.
Mishra~\etal~\cite{mishra2017fine} introduced CSR into the accelerator.
Data in~\cite{han2016eie, chen2019eyeriss, mishra2017fine, han2017ese} was compressed in CSC format.
Extensor~\cite{hegde2019extensor} utilized CSF encoding format.
More specifically, Eyeriss v2~\cite{chen2019eyeriss} encoded the weights in CSC data format for both on- and off-chip accessing to reduce the bandwidth requirements and energy consumption. Hence $1.2\times$ and $1.3\times$ improvement in speed and energy consumption are obtained by introducing CSC into Eyeriss v2.
To best utilize the benefit of CSC data format, the PE in Eyeriss v2 is specially designed, where there are registers to store both address vectors and data vectors in the CSC compressed data, which is drawn in Fig. 1 within Appendix E. % Fig.~\ref{fig:eyeriss_v2_pe}.
Sticker~\cite{yuan2018sticker} compressed the sparse weight into CSC format offline and decodes weight on-chip. In addition, there are COO encoders and decoders in Sticker to adopt the COO format for activations online.
Zhang~\etal~\cite{zhang2019snap} designed an associative index matching (AIM) module in their hardware accelerator SNAP before ending the data into a multiplier array to find the non-zero partial sum position, \textit{i.e.}, both the input activation and the corresponding weight are non-zero.
A similar module was integrated in Cambricon-S~\cite{zhou2018cambricon} where the bitmap encoding method was deployed; hence the comparators in SNAP are replaced by AND gates, depicted in Fig. 2 within Appendix E. %~\ref{fig:hw-zhou2018cambricon}.
Envision~\cite{moons201714} exploited sparsity by representing the data in bitmap format, storing those flags in a GRD memory, hence guarding both memory fetches and MAC operations.
% A run-length encoding scheme was utilized in SCNN~\cite{parashar2017scnn} that encoded the number of zeros between elements into the index vector. Hence only non-zero weights and activations are processed.
SCNN~\cite{parashar2017scnn} employed a run-length encoding scheme, encoding the number of zeros between elements into the index vector, thus processing only non-zero weights and activations.
\begin{table*}[!t]
\centering
% \normalsize
\caption{The hardware accelerators that use sparsity encoding methods}
\vspace{-5pt}
\label{tab:hw-sparsity-encoding}
\resizebox{\textwidth}{!}{%
\begin{threeparttable}
\begin{tabular}{l|c|c|c|c|c|c|c}
\hline
Methods &
Technology &
Area &
Frequency &
Energy Efficiency (TOPS/W)
% \begin{tabular}[c]{@{}c@{}}Energy Efficiency\\ (TOPS/W)\end{tabular}
&
Zero Elements Ratio &
Encoding Format &
Data Format\\ \hline
\textit{Eyeriss v2~\cite{chen2019eyeriss}} &
65 nm &
% \begin{tabular}[c]{@{}c@{}}2695k gates\\ (NAND-2)\end{tabular}
2695k gates (NAND-2) &
200 MHz &
0.96 &
68.98\% &
CSC &
16b \\ \hline
\textit{Envision~\cite{moons201714}} & 28 nm & 1.87 mm$^2$ & 200 MHz & 1.3 & 5-82\% & Bitmap & 16b \\ \hline
\textit{Sticker~\cite{yuan2018sticker}} &
65 nm &
7.8 mm$^2$ &
200 MHz &
62.1 &
both 5\% &
\begin{tabular}[c]{@{}c@{}}COO for activation, CSC for weight\end{tabular} &
8b \\ \hline
\textit{SNAP~\cite{zhang2019snap}} & 16 nm & 2.4 mm$^2$ & 260 MHz & 21.55 & both 10\% & COO-1D & 16b \\ \hline
\end{tabular}%
% \begin{tablenotes}
% \scriptsize
% \item[a] The original is 5\%, we guess it should be 95\% instead.
% \end{tablenotes}
\end{threeparttable}
}
\vspace{-2.5mm}
\end{table*}
In summary, some works store the pruned weights in the memory to reduce the memory footprint and bandwidth requirements.
Sparse encoding methods are introduced in the hardware accelerators for better performance and higher efficiency while introducing additional computational complexity to the decoding process.
\vspace{-5pt}
\subsection{Hardware Accelerating on Non-linear Operations}
\subsubsection{NMS}
\label{sec:accelerator_nms}
Although there are lots of algorithms that optimize NMS, hardware-accelerated NMS methods are explored less.
The CPU-NMS~\cite{rothe2014non} and GPU-NMS~\cite{gpunms, gpunms2} techniques are designed to operate on CPU and GPU platforms. GPU-NMS V2~\cite{gpunms2} can process 1027 bounding boxes in just 0.324 ms on a GeForce GTX 1060 running at 1.70 GHz. Shi \textit{et al.}\cite{powernms} were motivated by GPU-NMS\cite{gpunms} to develop a power-efficient NMS accelerator that merges 1000 bounding boxes in 12.79 microseconds at 400 MHz. Despite this, the accelerated NMS still takes considerable time compared to the execution time of convolution operations. Moreover, these solutions lack support for high-resolution images.
% CPU-NMS~\cite{rothe2014non} and GPU-NMS~\cite{gpunms, gpunms2} target CPU and GPU platforms. GPU-NMS V2~\cite{gpunms2} could process 1027 bounding boxes in $0.324$ ms on GeForce GTX 1060 at 1.70 GHz. Inspired by GPU-NMS~\cite{gpunms}.
% Shi \textit{et al.} ~\cite {powernms} implemented a power-efficient NMS accelerator that merges 1000 bounding boxes in $12.79\ \mu$s at 400MHz.
% However, the accelerated NMS still occupies a significant time compared with the execution time of convolution operations. Additionally, these solutions do not support high image resolution.
MaxpoolNMS~\cite{maxpoolnms} and PSRR-MaxpoolNMS~\cite{psrrmaxpool} reformulate NMS as MaxPool operation, which is inherently parallel and friendly to the hardware.
Inspired by these two NMS algorithms, ShapoolNMS~\cite{shapoolnms} is proposed, which is a salable NMS hardware accelerator that supports high image resolution and both one- and two-stage object detectors. These works are summarized in Table~\ref{tab:alg-nms}.
As the number of boxes increases, ShapoolNMS~\cite{shapoolnms} surpasses GreedyNMS hardware, GPU-NMS V2~\cite{gpunms2}, and Shi~\etal's~\cite{powernms} solutions in terms of performance. This is because the time complexity of GPU-NMS and Shi~\etal's method is at least $\mathcal{O}(nm)$\cite{powernms}, whereas ShapoolNMS has a time complexity of $\mathcal{O}(n)$, as outlined at Table IV in Appendix E.%as outlined in Table \ref{tab:nms-hw-perf-cmp}.
%ShapoolNMS~\cite{shapoolnms} increasingly outperform GreedyNMS hardware, GPU-NMS V2~\cite{gpunms2}, and Shi~\etal~\cite{powernms} with an increased number of boxes.
% This is due to the time complexity of GPU-NMS, and Shi~\etal is at least $\mathcal{O}(nm)$~\cite{powernms}, while that of ShapoolNMS is $\mathcal{O}(n)$, as summarized in Table~\ref{tab:nms-hw-perf-cmp}.
\begin{table*}[!t]
\centering
\scriptsize
\caption{Comparing the NMS algorithms and hardware implementations}
\vspace{-5pt}
\label{tab:alg-nms}
\resizebox{0.93\textwidth}{!}{%
\begin{tabular}{l|c|c|c|c|c|c}
\hline
NMS Algorithm/HW &
Platform &
Hardware Friendly?
% \textbf{\begin{tabular}[c]{@{}c@{}}Hardware\\ Friendly?\end{tabular}}
&
High Resolution?
% \textbf{\begin{tabular}[c]{@{}c@{}}High Image \\ Resolution?\end{tabular}}
&
Parallelable? &
% \textbf{\begin{tabular}[c]{@{}c@{}}Supports \\ 2-stage detectors?\end{tabular}}
Two-stage detectors?
&
Time Complexity \\ \hline
GreedyNMS~\cite{greedynms} &
CPU &
\xmark &
\cmark &
\xmark &
\cmark &
$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\ \hline
SoftNMS~\cite{bodla2017soft} &
CPU &
\xmark &
\cmark &
\xmark &
\cmark &
$\mathcal{O}(n\log(n)) + \mathcal{O}(nm)$ \\ \hline
MaxpoolNMS~\cite{maxpoolnms} &
CPU &
\cmark &
\cmark &
\cmark &
\xmark &
$\mathcal{O}(n)$ \\ \hline
PSRR-MaxpoolNMS~\cite{psrrmaxpool}&
CPU &
\cmark &
\cmark &
\cmark &
\cmark &
$\mathcal{O}(n)$ \\ \hline
CPU-NMS~\cite{rothe2014non} &
CPU &
-- &
\xmark &
\cmark &
\xmark &
$\mathcal{O}(nm)$ \\ \hline
GPU-NMS~\cite{gpunms2}&
GPU &
-- &
\xmark &
\cmark &
\xmark &
$\mathcal{O}(nm)$ \\ \hline
Shi~\etal~\cite{powernms} &
ASIC &
-- &
\xmark &
\cmark &
\xmark &
$\mathcal{O}(nm)$ \\ \hline
ShapoolNMS~\cite{shapoolnms} &
ASIC &
-- &
\cmark &
\cmark &
\cmark &
$\mathcal{O}(n)$ \\ \hline
\end{tabular}%
}
\vspace{-2.5mm}
\end{table*}
% Fig.~\ref{fig:hw-shapoolnms-top} illustrates the architecture of ShapoolNMS~\cite{shapoolnms}, including Relationship Recovery Ways (RRWs), MaxPool Kernel Index Compute Units (MKICUs), XBar and Score Map Memories (SMMs). As reported in~\cite{shapoolnms}, ShapoolNMS achieves $8.54\times$ speedup than state-of-the-art and $42,713\times$ speedup versus GreedyNMS software implementations.
The structure of ShapoolNMS~\cite{shapoolnms} is depicted in Fig. 3 within Appendix E. %Fig.\ref{fig:hw-shapoolnms-top}.
According to\cite{shapoolnms}, ShapoolNMS outperformed existing state-of-the-arts by an $8.54\times$ speedup and a $42,713\times$ speedup compared to GreedyNMS.
\subsubsection{Softmax}
\label{sec:hardware_softmax}
Softmax
%(Eq.~\ref{eq:softmax})
involves expensive operations such as exponentiation and division, which is challenging for hardware platforms dealing with large input vectors. The exponential operations' wide range can also cause overflow problems with limited hardware resources~\cite{yuan2016efficient}. %, leading to implementation on software platforms.
%The original Softmax equation (as shown in Eq.~\ref{eq:softmax}) %contains some operations, such as exponentiation and division, that are relatively expensive for hardware platforms, especially with an enormous size $N$ of the input vector.
%involves expensive operations such as exponentiation and division, which can be challenging for hardware platforms, particularly when dealing with large input vector sizes.
%Additionally, as the range of the exponential operations is very wide, an overflow problem can be caused due to the limited hardware resources~\cite{yuan2016efficient}.
%Hence, the classification stage is usually implemented on software platforms.
%However, the softmax layer has been a bottleneck for real-time applications due to the hardware accelerating on other convolutional layers.
Unfortunately, the Softmax layer can be a bottleneck for real-time applications as other convolutional layers can be hardware accelerated.
% The Softmax layer can become a bottleneck for real-time applications since hardware acceleration is possible for other convolutional layers.
As summarized in Table~\ref{tab:hw-softmax}, the hardware acceleration of the softmax algorithm can be classified into three categories:
% \textcolor{red}{
\begin{itemize}
\item[A.] \textbf{Direct optimization}: optimize the hardware implementation of exponential and division units directly.
% Directly optimizing the exponential units and division units;
\item[B.] \textbf{Mathematical transformation}: Apply mathematical transforming (\eg, logarithmic transforming) to replace the exponential operations.
\item[C.] \textbf{Mathematical reformulation}: Reformulate the de-facto softmax equation into other hardware-friendly equations, such as replacing the exponent base $e$ to $2$, new softmax layer based on integral stochastic computing.
\end{itemize}
% }
\begin{table*}[t!]
\centering
\caption{The summary of softmax hardware implementations, optimizations, and average mean squared error (MSE) or accuracy loss.
\vspace{-2.5mm}
}
% \vspace{-2.5pt}
\label{tab:hw-softmax}
\resizebox{\textwidth}{!}{%
\begin{threeparttable}
\begin{tabular}{l|c|c|c|c|c}
\hline
\multicolumn{1}{l|}{Methods}&
Softmax Equation &
% \textbf{Exp Optimization Category\tnote{a}} &
\begin{tabular}[c]{@{}c@{}}Exponential Function \\ Optimization Category\tnote{a}\end{tabular} &
% \textbf{Division Optimization Category\tnote{a}} &
\begin{tabular}[c]{@{}c@{}}Division Optimization\\ Category\tnote{a}\end{tabular} &
Avg MSE &
\begin{tabular}[c]{@{}c@{}}Accuracy\\ Loss\end{tabular} \\ \hline
Yuan~\etal~\cite{yuan2016efficient} &
$\ln(\sigma(z_i)) = (z_i-(z)_{\max})-\ln(\sum\nolimits_{L=1}^{K}e^{z_j-(z)_{\max}})$ &
A (LUT-EXP) &
B (logarithmic) &
-- &
-- \\ \hline
Geng~\etal~\cite{geng2018hardware} &
% $Eq.~\ref{eq:softmax}$
$\sigma(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$
&
A (LUT-EXP, LUT-PLF) &
A (bit shift) &
-- &
-- \\ \hline
Li~\etal~\cite{li2018efficient} &
-- &
A (LUT-EXP) &
A (optimized divider) &
$1.5 \times 10^{-5}$ &
-- \\ \hline
Wang~\etal~\cite{wang2018high} &
$\sigma {(z)_j} = \exp ({x_i} - ln(\sum\limits_{j = 1}^N {{e^{{x_j}}}} ))$ that ${e^{{y_i}}} = {2^{{y_i} \cdot {{\log}_2}e}}$ &
C (base 2) &
B (exp-log) &
$8 \times 10^{-5}$ &
-- \\ \hline
Sun~\etal~\cite{sun2018high} &
$\sigma {(z)_j} = {e^{{z_j} - \max ( z )}}/\sum\limits_{k = 1}^K {{e^{{z_k} - \max ( z )}}}$ &
A (GLUT-EXP) &
A (bit shift) &
$10^{-8}$ &
-- \\ \hline
Hu~\etal~\cite{hu2018efficient} &
$\sigma {(z)_j} =\exp(-h_{j}')exp({-ln\sum _{j=1}^{m}e^{-h_{\acute {j}}}})$ &
C (Integral SC) &
B (logarithmic) &
0.03 &
-- \\ \hline
Du~\etal~\cite{du2019efficient} &
$\sigma {(z)_j} = \exp((z_i-(z)_{\max})-\ln(\sum\nolimits_{L=1}^{K}e^{z_j-(z)_{\max}}))$ &
A (GLUT-EXP) &
B (logarithmic) &
-- &
0.24\% \\ \hline
Kouretas~\etal~\cite{kouretas2020hardware} &
$\sigma {(z)_j} = e^{z_j-(z)_{\max}}$ &
A (LUT-EXP) &
B (logarithmic) &
-- &
0\% \\ \hline
Wang~\etal~\cite{wang2019customized} &
$\sigma {(z)_j} = \frac{N^{z_j}}{\sum\nolimits_{i = 1}^j N^{x_i}}$ &
C (dynamic learned) &
-- &
-- &
-- \\ \hline
Cardarilli~\etal~\cite{cardarilli2021pseudo} &
$\sigma {(z)_j} =\frac{2^{x_j}}{\sum _{k=1}^{N} 2^{x_k}}$ &
C (base 2) &
C (approximated) &
$10^{-4}$ &
-- \\ \hline
Spagnolo~\etal~\cite{spagnolo2021aggressive} &
NA &
A (accumulated) &
A (bit shift) &
-- &
0\% \\ \hline
Stevens~\etal~\cite{stevens2021softermax} &
NA &
C (base 2) &
-- &
-- &
0.5\% \\ \hline
\end{tabular}%
\begin{tablenotes}
\scriptsize
% \item[a] These are reported accuracy loss instead of Average MSE.
\item[a] The optimization categories are (A) Direct optimization, (B) Mathematical transformation, and (C) Mathematical reformulation. Detailed optimization methods are explained in brackets.
\end{tablenotes}
\end{threeparttable}
}
\vspace{-2.5mm}
\end{table*}
Yuan~\cite{yuan2016efficient}, for the first time, proposed an efficient hardware implementation of softmax that uses a logarithmic transformation to eliminate division operations. Low-complexity subtractors replace complex division units. To address overflow issues, down-scale parameters are introduced in exponential units, reducing data width. %an efficient hardware implementation of softmax that utilizes a logarithmic transformation technique to avoid division operations.
%In this case, the complex division units are replaced by low-complexity subtractors.
%Besides, to address the overflow problem, the down-scale parameters are introduced in exponential units, enabling the reduction in data width.
% Eq.~\ref{eq:yuan2016efficient} shows the softmax operation after optimization.
The $e^{x_i}$ operations are approximated by creating a lookup table (LUT) that stores the discrete results with the subset of $x_i$.
The logarithmic unit is also based on LUT implementation.
% \begin{equation}
% \ln(\sigma(z_i)) = (z_i-(z)_{\max})-\ln(\sum\nolimits_{L=1}^{K}e^{z_j-(z)_{\max}})
% \label{eq:yuan2016efficient}
% \end{equation}
As the final results in~\cite{yuan2016efficient} were based on a transformed version of the softmax equation, %is after transformation,
Du~\etal~\cite{du2019efficient} proposed a corrected version that the exponential function is applied to the final result. %, which is shown in Eq.~\ref{eq:du2019efficient}.
The exponential function is split into multiple basics based on the calculation rule. % in Eq.~\ref{eq:exp-rule}.
Hence a group of LUTs (GLUT) was introduced to store those basics.
Sun~\etal~\cite{sun2018high} also proposed a similar architecture that utilizes GLUT to optimize the exponential function.
% \begin{equation}
% \sigma(z_i) = \exp((z_i-(z)_{\max})-\ln(\sum\nolimits_{L=1}^{K}e^{z_j-(z)_{\max}}))
% \label{eq:du2019efficient}
% \end{equation}
% \begin{equation}
% e^i = e^{ax} \times e^{by} \times e^{cz}
% \label{eq:exp-rule}
% \end{equation}
%\begin{table*}
%\centering
%\caption{6 combinations for softmax operations defined in~\cite{geng2018hardware}.}
%\scriptsize
%\label{tab:softmax_modules}
% \begin{tabular}{|c|l|}
% \hline
% Methods & Definition \\ \hline
% Approx\_A & Exponential function $\leftarrow$ the look-up table (LUT). \\ \hline
% Approx\_B & Exponential function $\leftarrow$ the look-up table; Division operation $\leftarrow$ bit shifting. \\ \hline
% Approx\_C & Exponential function $\leftarrow$ piecewise linear function (PLF). \\ \hline
% Approx\_D & Exponential function $\leftarrow$ piecewise linear function, in which the coefficients are the power of two (PLF-B); \\ \hline
% Approx\_E & Exponential function $\leftarrow$ piecewise linear function, while Division operation $\leftarrow$ bit shifting ($2^n$). \\ \hline
% Approx\_F & Exponential function $\leftarrow$ piecewise linear function, in which the coefficient is in the power of two; Division operation $\leftarrow$ bit shifting. \\ \hline
% \end{tabular}
%\end{table*}
%\begin{table*}[t]
%\centering
%\caption{Object detection accuracy on VOC2007 and COCO2014 and instance segmentation accuracy on VOC 2012, with the 6 Softmax approximation variants that are reported in~\cite{geng2018hardware} }
%\small
% \begin{tabular}{| c | c | c | c | c | c | c | c |}
% \hline
% \multicolumn{2}{|c|}{Methods} & \multirow{4}{*}{\makecell{min. \\ table size}} & \multicolumn{2}{c|}{\makecell{VOC2007 \\ mAP@0.5} } & \multicolumn{2}{c|}{\makecell{COCO2014 \\ mAP@[.5-.95]}} & \makecell{VOC2012 \\ mAP@0.5} \\ \cline{1-2} \cline{4-8}
%$e^{x_i}$ & $\sum{e^x}$ & & \makecell{Faster \\ R-CNN} & \makecell{R-FCN \\ (ResNet50)} & \makecell{Faster \\ R-CNN} & \makecell{R-FCN \\ (ResNet101)} & \makecell{MNC \\ (VGG16)}\\ \hline
% \multirow{2}{*}{\makecell{LUT-PLF}}
% & 2$^n$ & 32 & 69.84 & 70.31 & 21.30 & 26.50 & 63.75\\
% & / & 32 & 72.30 & 71.69 & 24.10 & 28.10 & 64.65 \\ \hline
% \multirow{2}{*}{\makecell{LUT-PLF-B} }
% & 2$^n$ & 32 & 68.12 & 68.86 & 20.80 & 25.10 & 64.27 \\
% & / & 32 & 72.06 & 71.37 & 23.70 & 27.90 & 62.96 \\ \hline
% \multirow{2}{*}{\makecell{LUT-EXP} }
% & 2$^n$ & 128 & 69.95 & 69.82 & 20.20 & 25.50 & 62.52 \\
% & / & 128 & 72.68 & 71.64 & 24.10 & 27.40 & 64.40 \\ \hline
% $e^{x_i}$ & / & - & 72.52 & 71.76 & 24.20 & 28.20 & 63.50 \\ \hline
% \end{tabular}
% \label{tab:ops_performance}
%\end{table*}
In addition to approximating $e^{x_i}$ directly based on LUT (LUT-EXP), Geng~\etal~\cite{geng2018hardware} also proposed another approximation technique using a Piecewise Linear Function (LUT-PLF). Besides, Geng~\etal proposed six softmax operation combinations, including exponential functions, power-of-twos, and look-up tables.
% approximating of it by introducing the Piecewise Linear Function (LUT-PLF)
% as shown in Eq.~\ref{eq:geng2018hardware}.
%The LUTs store the values of $\alpha ^{s}$, $x^s_l$ and pre-calculated $y^s_l - \alpha ^{s} * x^s_l$, %Hence $e^{x_i}$ is approximated with one multiply and one addition.
%which are used to approximate $e^{x_i}$ with one multiply and one addition.
%Bit shifts in this work approximate multiplication and division.
% LUTs store $\alpha ^{s}$, $x^s_l$, and pre-calculated $y^s_l - \alpha ^{s} * x^s_l$, used to approximate $e^{x_i}$ with one multiply and one addition. Bit shifts approximate multiplication and division.
%Geng~\etal also proposed six softmax operation combinations involving exponential functions, power-of-twos, and look-up tables. The LUT-PLF approximation, which uses coefficients as powers of two to implement the exponential function, achieves the best accuracy, energy consumption, and chip area results. The paper's experimental results used various networks and benchmark datasets.
% The LUT-PLF approximation achieved the best accuracy, energy consumption, and chip area results by using coefficients as powers of two to implement the exponential function. The experiments included various networks and benchmark datasets.
%six combinations for softmax operations regarding exponential functions, power-of-twos, and look-up tables. %Table~\ref{tab:softmax_modules}.
%The approximation that implements the exponential function by LUT-PLF, in which the coefficients are the power of two, gains the optimal results considering the accuracy, energy consumption, and chip area.
%This work reported their experiment results on various networks among several benchmark datasets
%The experimental results reported in the paper were obtained using various networks and benchmark datasets.
%a large dataset, namely, Faster R-CNN (VGG16) on VOC2007, R-FCN (ResNet-101) on COCO2014, and MNC (VGG16) on VOC 2012.% in Table~\ref{tab:ops_performance}.
% \begin{equation}
% \begin{aligned}
% f^s(x)&= \alpha ^{s} * (x - x^s_l) + y^s_l
% = \alpha ^{s} * x + (y^s_l - \alpha ^{s} * x^s_l) \\
% &x \in [x^s_l, x^s_r], \quad y^s_l = e^{x^s_l}, \quad s \in [1,S]
% \label{eq:geng2018hardware}
% \end{aligned}
% \end{equation}
In~\cite{cardarilli2021pseudo}, \cite{wang2018high} and \cite{stevens2021softermax}, the exponential function was optimized by replacing base exponent with 2.
%The proposed softmax hardware architecture in~\cite{cardarilli2021pseudo} reformulated softmax as in Eq.~\ref{eq:cardarilli2021pseudo}.
% \begin{equation}
% \begin{aligned}
% \widetilde{p}_i &=\frac{2^{x_i}}{\sum _{k=1}^{N} 2^{x_k}} =2^{x_i-\text {exp}_{\text{sum}}} \cdot \frac{1}{ 1 \cdot \text {mant}_{\text{sum}}} \\
% % &=2^{x_i-\text {exp}_{\text{sum}}} \cdot \frac{1}{ 1 \cdot \text {mant}_{\text{sum}}}
% \end{aligned}
% \label{eq:cardarilli2021pseudo}
% \end{equation}
% \begin{equation}
% \begin{aligned}
% \widetilde{p}_i &=\frac{2^{x_i}}{\sum _{k=1}^{N} 2^{x_k}} \\
% &=2^{x_i-\text {exp}_{\text{sum}}} \cdot \frac{1}{ 1 \cdot \text {mant}_{\text{sum}}}
% \end{aligned}
% \label{eq:cardarilli2021pseudo}
% \end{equation}
Hu~\etal~\cite{hu2018efficient} reformulated the softmax layer using integral stochastic computing for the exponent operations and logarithmic transformation to avoid the division operations. Spagnolo~\cite{spagnolo2021aggressive} proposed approach optimizes the softmax through aggressive optimization, replacing complex processes with hardware-friendly operations. Specifically, right-shifting operations replace division, and addition operations replace exponential operations.
As Table~\ref{tab:hw-softmax-ppa} summarized, most Softmax hardware accelerators target ASICs, with a relatively small area and power overhead compared with the convolution-based or Transformer-based hardware accelerators. However, none of them optimize the Softmax with a long input sequence or a lot of classes, crucial for Transformers. This gap presents a key opportunity to innovate Softmax accelerators for Transformer models. %, potentially enhancing performance and efficiency in complex neural network processing.
\begin{table*}[t!]
\scriptsize
\centering
\caption{The summary of softmax hardware accelerators.}
\vspace{-5pt}
\label{tab:hw-softmax-ppa}
\resizebox{0.92\textwidth}{!}{%
\begin{threeparttable}
\begin{tabular}{l|c|c|c|c|c|c}
\hline
\multicolumn{1}{l|}{
Methods} &
Platform &
Technology &
Frequency &
Area (LUT) &
Power &
Performance \\ \hline
Yuan~\etal~\cite{yuan2016efficient}\tnote{a} &
ASIC &
90 nm &
250 MHz &
4.3576 $\times 10^4 \mu\text{m}^2$ &
3228.2 mW &
10 classes \\ \hline
Geng~\etal~\cite{geng2018hardware} &
ASIC &
65 nm &
500 MHz &
38 $\mu\text{m}^2$ &
0.086 mW &
R-FCN (ResNet-101) on COCO 2014 \\ \hline
Li~\etal~\cite{li2018efficient} &
ASIC &
45 nm &
3.3 GHz &
3.4348 $\times 10^4 \mu\text{m}^2$ &
-- &
16-bit input \\ \hline
Wang~\etal~\cite{wang2018high} &
ASIC &
28 nm &
2.8 GHz &
1.5 $\times 10^4 \mu\text{m}^2$ &
51.60 mW &
22.4 G/s (16-bit) \\ \hline
Sun~\etal~\cite{sun2018high} &
ASIC &
65 nm &
1 GHz &
0.44 mm $\mu\text{m}^2$ &
333 mW &
16-bit valid input \\ \hline
Hu~\etal~\cite{hu2018efficient} &
FPGA &
-- &
-- &
10746 LUTs &
603 mW &
-- \\ \hline
Du~\etal~\cite{du2019efficient} &
ASIC &
65 nm &
500 MHz &
0.64 mm$^2$ &
0.82 mW &
Target on binary and ternary NN \\ \hline
Kouretas~\etal~\cite{kouretas2020hardware} &
ASIC &
90 nm &
255 MHz &
2.5597 $\times 10^4 \mu\text{m}^2$ &
1576.3 mW &
5 classes \\ \hline
Cardarilli~\etal~\cite{cardarilli2021pseudo} &
ASIC &
90 nm &
310 MHz &
4.6816 $\times 10^4 \mu\text{m}^2$ &
2.769 mW &
8-bit integer, 10 classes \\ \hline
Spagnolo~\etal~\cite{spagnolo2021aggressive} &
ASIC &
28 nm &
2.5 GHz &
3595 $\mu\text{m}^2$ &
NA &
16-bit integer, 10 classes \\ \hline
\end{tabular}%
\begin{tablenotes}
% \scriptsize
\item[a] The data is reported in~\cite{kouretas2020hardware}.
\end{tablenotes}
\end{threeparttable}
}
\vspace{-5mm}
\end{table*}
% \subsubsection{Others}
% \vspace{-5pt}
% \subsection{Hardware Architectures}
\vspace{-5pt}
\subsection{Stochastic Computing Architecture}
\label{subsec:stochastic_architecture}
Power-efficient approximate computing techniques, Stochastic Computing (SC), is a random, non-weighted, bitstream-based unary computing technique. An SC circuit necessitates minimal hardware, low power consumption, and high fault tolerance to computational errors~\cite{gaines1969stochastic}.
It has found widespread application in various network architectures, including CNNs, RNNs, and MLPs~\cite{liu2020survey}. However, due to lengthy sequences and a high demand for stochastic number generators (SNGs), achieving competitive computational latency and energy consumption is challenging for SC neural networks.
% A notable reduction in hardware demands and power consumption can be achieved by making moderate trade-offs regarding inference accuracy and computational speed.
Recent advancements in stochastic computing techniques have greatly enhanced the performance of SC Neural Networks (SC NNs), bringing them to a level of competitiveness with conventional binary designs while using significantly fewer hardware resources.
Some designs focus on improve the mostly cost random number generators (RNGs), leading to better hardware and energy efficiency~\cite{salehi2020low}. Some research focus on designing efficient hardware architecture to implement SC NNs~\cite{frasser2022fully, moran2022digital}. \cite{moran2022digital} proposed a hardware implementation on stochastic computing for Radial Basis Function (RBF) neural networks.
\section{When NN compression meets Homomorphic Encryption}\label{sec:compression_ahe}
%The success of neural networks is primarily attributed to the abundance of large training datasets available today. However, data privacy could be a big concern, especially when computations are made on untrusted third-party servers. Fully Homomorphic Encryption (FHE) has emerged as a powerful cryptography technique that enables computations on encrypted data without requiring a decryption key, thereby offering a solution for protecting data privacy in an untrusted computation environment. This section will review the homomorphic neural networks and how NN compression helps in homomorphic neural networks.
\subsection{Motivation}
\label{sec:ahe_hnn}
Homomorphic Encryption (HE) stands as a distinctive encryption paradigm enabling computations on encrypted data (ciphertexts) without the requirement of the decryption key. The decrypted results of these computations are equivalent to outcomes derived from standard arithmetic operations performed on unencrypted data. Homomorphic Encryption can be categorized into Partial Homomorphic Encryption (PHE) and Fully Homomorphic Encryption (FHE) based on the supported homomorphic arithmetic operations. PHE supports either homomorphic addition, as introduced by Paillier~\cite{paillier1999public}, or homomorphic multiplication, as outlined by Rivest~\cite{rivest1978method}, between two ciphertexts. In contrast, FHE, a more comprehensive variant pioneered by Gentry~\cite{gentry2009fully}, accommodates both homomorphic addition and multiplication on ciphertexts. Additionally, FHE incorporates the \textit{bootstrapping} primitive, although its practical performance remains a challenge, it holds the potential to reduce noise in ciphertexts, rendering them amenable for arbitrary depth of computations.
Given the powerful features of FHE, it is a natural choice of technology for protecting data privacy during computation in untrusted environment. Notably, the concept of Homomorphic Neural Networks (HNNs) involves the utilization of FHE to encrypt both neural network models and input data. This application ensures the confidentiality of models and data throughout training or inference processes conducted in untrusted settings, such as cloud computing or multi-party collaborations. This becomes particularly crucial in domains where data privacy holds paramount importance, as evidenced in sectors like medical or financial applications~\cite{isakov2019survey}.
Fig.~\ref{fig:hcnn} illustrates the application of Homomorphic Neural Networks (HNN) in preserving the privacy of medical images for disease diagnosis within a cloud deployment, which is also referred to as the Machine Learning as a Service scenario. The process begins with the client generating HE keys to encrypt her private images, transforming them into ciphertexts denoted as $Enc(X)$. Subsequently, these ciphertexts are transmitted to the cloud server to request disease diagnosis on the encrypted images. The cloud server then performs evaluations using the HNN-based disease diagnosis model on the image ciphertexts, yielding the encrypted diagnosis result, denoted as $Enc(Y)$. This encrypted result is relayed back to the client, who decrypts it into plaintext using her secret decryption key. Notably, throughout this entire process, the cloud server remains incapable of accessing any input data from the client or intermediate data during HNN evaluation in plaintext, as it lacks the decryption key.
However, it is non-trivial task to apply FHE to NNs to construct Homomorphic NNs, and the process faces multiple challenges. First, many mainstream FHE schemes, such as BGV~\cite{yagisawa2015fully} and BFV~\cite{fan2012somewhat} schemes, do not natively support float point arithmetic which are often found in neural network computations. Second, applying FHE to neural networks usually incur large performance and resource overheads in terms of computation and memory demands. This is attributed to the fact that plain data expands into large polynomials during homomorphic computation, and simple operations, like multiplications, translate into complex polynomial operations homomorphically~\cite{aikata2023reed}. Third, deeper networks involve deeper computations, which requires larger FHE parameters or even the expensive bootstrapping operations to control the noise growth in ciphertexts, which further increases the computationally intensity. Fortunately, neural network compression techniques can help to alleviate these challenges. This section reviews how NN compression helps on HNNs.
\begin{figure}[!ht]
\centering
%\includegraphics[width=.85\textwidth, height=5cm]{hcnn_operation.pdf}
\includegraphics[width=.46\textwidth]{he-fig/fhe_hcnn.pdf}
\caption{Homomorphic neural network for secure cloud computing in Secure Machine Learning as a Service scenario.}
\label{fig:hcnn}
\vspace{-5mm}
\end{figure}
% \subsection{Fully Homomorphic Encryption}
%Since its first introduction by Rivest et al., ~\cite{rivest1978data},
% FHE is %has always been
% an intriguing technology due to its unique ability to compute encrypted data without the decryption key. %In 2009, Gentry proposed the first construction of an FHE scheme~\cite{gentry2009fully}. Since then, this field has seen significant advancements, and several new FHE schemes have been proposed~\cite{brakerski2014leveled,fan2012somewhat,cheon2017homomorphic}. Generally, the FHE plaintext and ciphertext spaces are polynomial rings.
%FHE is instantiated
% The basic principle of FHE is
% to preserve the algebraic structure between plain text and ciphertext, which allows for two primary computational operations: homomorphic addition and multiplication. %and provides the user with two primary computational operations: homomorphic addition and multiplication.
% These operations can manipulate ciphertexts and produce encrypted results equivalent to the corresponding plaintext results after decryption, as demonstrated in Figure \ref{fig: fhe:operations}.
% \begin{figure}[!ht]
% \centering
% \includegraphics[width=0.65\columnwidth]{he-fig/fhe_operations.pdf}
% \vspace{-0.3cm}
% \caption{Homomorphic operations}
% \label{fig:fhe:operations}
% \end{figure}
% \vspace{-0.8cm}
%Modern FHE schemes conceal plaintext messages with noise that can be identified and removed with the secret key~\cite{brakerski2011fully}. As we compute on encrypted data, the noise magnitude accumulated in a ciphertext increases at a specific rate (high rate for multiplication and low rate for addition). As long as the noise is below a certain threshold, which depends on the encryption parameters, decryption can filter out the noise and retrieve the plaintext message successfully. Although FHE schemes include a primitive (known as \textit{bootstrapping}) to refresh the noise~\cite{gentry2009fully} inside ciphertexts, it is extremely computationally expensive. Instead, one can use a \textit{levelled} FHE scheme~\cite{brakerski2014leveled} that allows evaluating circuits of multiplicative depth below a certain threshold, which the encryption parameters can control. In this way, one can avoid expensive bootstrapping operations by selecting the appropriate encryption parameters that can accommodate the computational needs of the applications.
%First, we introduce a representative leveled FHE scheme, the Brakerski-Fan-Vercauteren (BFV) scheme~\cite{fan2012somewhat}. The BFV scheme is governed by three key parameters: $t$, $q$, and $n$. First, the plaintext space is a finite field controlled by the plaintext coefficient modulus $t$. To prevent computational overflows, $t$ must be set to be large enough to accommodate any intermediate result of the homomorphic evaluations. Second, the scheme limits the number of homomorphic operations performed on the ciphertext before decryption fails. We refer to this limit as the computation noise budget, which can be controlled by the ciphertext coefficient modulus $q$. Third, the underlying ring dimension $n$ is set to guarantee the targeted security level $\lambda$. For typical security requirements in practical applications, $\lambda$ is set to a minimum of $80\, \rm{bits}$. We remark that the choice of $n$ and $q$ significantly impacts the performance of FHE schemes in terms of computational and memory requirements. They also affect the data expansion rate due to encryption. The ciphertext size can be estimated at least $2*n*\log_{2}{q}$ bits.
%Next, we describe another representative leveled FHE scheme, the CKKS scheme~\cite{cheon2017homomorphic}.
%The plaintext and ciphertext are ring elements of a polynomial ring $R_q=Z_q[X]/(X^N+1)$, where $X^N+1$ is the polynomial modulus with degree $N$ and $Z_q[x]$ is the polynomial with integer coefficients based on modulus $q$. In particular, $q$ is the product of a group of prime factors, where the number of primes is called the ``\textit{level}'' of the ciphertext. When the input data is first encrypted, its ciphertext is at the highest level, say level $L$. Then along with the computations, the ciphertext may gradually move down to lower levels by removing one prime factor from $q$ at a time. In General, $L$ determines the largest multiplication depth a single ciphertext can have.
%CKKS supports standard FHE primitives like \textit{encode} and \textit{decode}, \textit{encrypt} and \textit{decrypt}, \textit{addition} and \textit{multiplication} (with both ciphertexts and plaintexts). Besides, a unique feature of CKKS is that it supports fixed-point arithmetic for approximate computing on encrypted numbers. The real input numbers are first scaled with a large scaling factor and rounded to the nearest integer (quantization) to implement this. Then they are encoded into plaintexts and subsequently encrypted into ciphertexts. To maintain a constant scaling factor in the ciphertext after multiplication, CKKS offers an efficient \textit{rescaling} procedure which moves down the ciphertext to the next lower level by removing a prime factor from coefficient modulus $q$, at the same time, scales down the amplified scaling factor in the ciphertext by the prime that removed from $q$. As mentioned, one can drastically improve FHE performance via Single-Instruction Multiple-Data (SIMD) packing methods. In CKKS, a vector of up to $N/2$ complex numbers can be encoded in a single plaintext element. This allows one to perform parallelized SIMD homomorphic operations on packed ciphertexts efficiently. Packing can be viewed as the ciphertext having independent slots, each concealing one data item. To manipulate the slots within a ciphertext, CKKS offers the \textit{rotate} primitive that can circularly shift the data locations across the slots.
\subsection{NN Compression for HNNs}\label{sec:ahe_compression}
Various model compression techniques discussed in Section~\ref{sec:compression}, such as model quantization, pruning, and low-rank factorization, are commonly employed to enhance the inference speed of Homomorphic Neural Networks (HNNs). For instance, in~\cite{zhang2019encrypted}, Lloyd-max quantization was utilized to quantize weights based on their local density during training. In~\cite{lou2019she}, log-quantization was employed to transform real-valued data into power-of-two values, effectively converting matrix multiplication operations into faster shift operations to accelerate inference.
Some approaches concentrate on binarizing the network to expedite HNNs inference, as observed in works like~\cite{bourse2018fast} and~\cite{pmlr-v80-sanyal18a}. TAPAS~\cite{pmlr-v80-sanyal18a}, for instance, optimized FHE-based encrypted predictions by leveraging binary networks.
Furthermore, specific strategies adopt network pruning to diminish inference latency. For example, CryptoNets~\cite{chou2018faster} applied unstructured weight pruning to reduce the inference latency of Homomorphic neural networks, bypassing homomorphic operations related to pruned weights. Some techniques amalgamate quantization and pruning to expedite inference, as exemplified in~\cite{chou2018faster}, which proposes both pruning and quantization of models to minimize the required operations and enhance weight sparsity.
Another avenue explores low-rank factorization techniques. A recent approach presented in~\cite{lu2021ffconv} integrates low-rank factorization with FHE ciphertext packing, effectively mitigating rotation overhead and significantly accelerating the inference process.
%A typical scenario to apply FHE to neural networks is as a service in the cloud. As illustrated in Figure \ref{fig:hcnn}, %a pre-trained neural network model is available on the cloud, whose weights are learned through training on unencrypted data.
%taking image classification as an example, the referral source (also termed data owner or client) provides an input image with intensities stored as pixel values, termed as \emph{plaintext}.
% \textcolor{blue}{Homomorphic neural networks find applications in secure cloud computing, and confidential data analysis.}
%FHE can be applied to neural networks as a service in the cloud.
% Figure \ref{fig:hcnn} shows a general case of using HNNs in secure cloud computing. %a typical scenario for image classification, where the referral source (also called data owner or client) provides an input image with pixel values stored as plaintext.
% In an FHE scheme, plaintext pixel values are encrypted into ciphertext. The referral source sends the ciphertexts (\textcolor{blue}{$Enc(X)$} to the cloud server to initiate a request for a particular inference. The trained CNN is evaluated homomorphically on the ciphertexts by the cloud server to produce an encrypted prediction. The encrypted prediction \textcolor{blue}{($Enc(Y)$)} is then sent back to the referral source \textcolor{blue}{(Client)} and decrypted using a secret key.
% This approach separates the client and cloud servers, with only the client able to encrypt or decrypt the data, and all CNN computations are performed on encrypted data in the cloud. This ensures sensitive data remains secure and private even when processed on untrusted servers. %An FHE scheme maps the plaintext pixel values into an encrypted form, termed as \emph{ciphertext}. To initiate a request for a particular inference, the referral source communicates the ciphertexts to the cloud server. In turn, the cloud server calls the trained CNN and evaluates it homomorphically on the ciphertexts to produce an encrypted prediction. This encrypted prediction is then communicated back to the referral source, wherein a secret key is used to decrypt the predicted result. We highlight that this solution offers a clear separation between the client and cloud servers -- only the client can encrypt or decrypt the data, and all CNN computations are performed on encrypted data in the cloud. This ensures that sensitive data remains secure and private, even when processed on untrusted servers.
% \textcolor{blue}{%Implementing homomorphic neural networks can be computationally intensive and less efficient than processing data in the clear. It involves complex cryptographic operations that can slow down computation. A lot of ongoing research explores to make them more efficient and practical for a wider range of applications. Neural network compression is a hot topic to make HNNs more efficient.
% Using homomorphic neural networks can be slower and more computationally demanding compared to processing data without encryption. This is attributed to the fact that plain data expands into large polynomials during homomorphic computation, and simple operations, like multiplication, translate into complex polynomial operations homomorphically~\cite{aikata2023reed}. On one hand, researchers have proposed acceleration techniques on various platforms, e.g., GPUs, CPUs, FPGAs, and ASICs. On the other hand, advanced compression techniques have been explored to improve their efficiency and applicability.}
% A customized framework is required to use homomorphic neural networks because traditional deep learning frameworks like TensorFlow and PyTorch are only designed for plain data. The customized framework needs to be built exclusively on top of FHE primitives such as additions, multiplications, and other internal operations on ciphertexts.
%One cannot directly use the traditional deep learning frameworks (TensorFlow, PyTorch, etc.) for homomorphic neural networks since they are designed to run with plain data only. Instead, a customized framework built exclusively on top of FHE primitives (additions, multiplications, and other internal operations on ciphertexts) is needed.
%A typical Neural Network model comprises linear and non-linear computation layers. Taking Convolutional Neural Network (CNN) as an example, primary linear computation layers include convolution and fully connected layers, which compute weighted sums (and sometimes add a bias term) on some or all of the input data. In a homomorphic CNN, the weights can be kept as plaintexts or ciphertexts in the cloud inference servers, depending on whether there is a need to protect the privacy of the model itself and whether the input data to the network is encrypted as ciphertexts. Since the linear layers involve only additions and multiplications, they can be directly implemented on top of the FHE libraries. In contrast, the non-linear layers must be specially handled since they contain operations that FHE does not support. In particular, CryptoNets \cite{gilad2016cryptonets} uses the square function to approximate the ReLU activation and the scaled mean pooling to replace the max pooling. The scaled mean pooling layer can be implemented as a particular case of the convolution layer, where all the weights in the filters are 1.
% \subsection{Challenges with Homomorphic NN}
% \subsubsection{Numerical Representation and Precision}
% \label{sec:AHE_numerical}
% The first challenge to applying FHE to neural networks is the choice of plaintext space for Homomorphic NN computation. The weights and inputs of a neural network are usually decimals, represented in floating-point. Unfortunately, these cannot be directly encoded and processed in most FHE libraries with integers. Therefore, \textit{scalar encoding} needs to be adopted, approximating these decimals with integers.
% It is done by multiplying them with some scaling factor \(\Delta\) and rounding the result to the nearest integer.
% Then, numbers encoded with the same scaling factor can be computed using integer addition or multiplication. The scaling factor \(\Delta\) corresponds to the number of bits of precision of the approximation and the upper bound on the approximation.
% However, computing numbers with different scaling factors is not straightforward.
% For example, suppose we have two messages \(\Delta_1 m_1, \Delta_2 m_2\) with two different scaling factors, where \(\Delta_1 < \Delta_2\):
% \begin{equation*}
% \begin{aligned}
% \Delta_1 m_1 + \Delta_2 m_2 = \Delta_2 ( m_2 + \Delta_2/\Delta_1 m_1);\\
% \Delta_1 m_1 \times \Delta_2 m_2 = \Delta_1\Delta_2 (m_1m_1).
% \end{aligned}
% \end{equation*}
% Adding the two encoded numbers will not result in their standard sum, and their scaling factors must be adjusted to be the same before they can be added together. On the other hand, multiplication will change the scaling factor of the result to \(\Delta_1\Delta_2\).
% This means that as homomorphic multiplications are done on encoded data, the scaling factor in the outputs will increase.
% Therefore, the scaling factor in the output needs to be reduced to the original, or the plaintext modulus has to be large enough to accommodate the increased scaling factor that results from homomorphic computations.
% A neural network with a larger number of bits precision implies larger \(\Delta\), which means that the required plaintext modulus will also be larger, possibly several hundreds or thousands of bits.
% \subsubsection{Computational Complexity and Resource Footprint}
% Although FHE provides a promising privacy-preserving computation paradigm, a large performance gap still hinders its practical usage. FHE operations are typically 10 to $10^6$ times slower than plaintext operations. FHE ciphertexts are usually represented as very large degrees ($2^{10} - 2^{15}$) of polynomials with very large coefficients ($10^2 - 10^3$ bits modulus). FHE operations typically involve large polynomials' addition, multiplication, and modulus manipulation. Unfortunately, these big-number computations cannot fit well into the machine words of modern computer processors, leading to inefficient processing performance.
% So far, a lot of research and engineering efforts have been made to accelerate the computational speed of FHE by parallel computation architecture of multi-core CPUs, GPUs, and FPGAs. While these parallel processors can improve the FHE computation speed by around two orders of magnitude, they cannot narrow the FHE performance gap further. Most recent studies by researchers have realized that FHE performance is not only bounded by the computational speed, but the main memory bandwidth is another major cause of performance bottleneck. This is because stridden memory accesses limit the FHE algorithms, such as the FFT operations, and the bootstrapping operations have relatively low arithmetic intensity. Therefore, the main memory bandwidth heavily bounds them. Moreover, many ciphertexts could be created when applying FHE to neural network training or inference. Expanding ciphertext size could result in frequent data swapping between host memory and acceleration devices. Therefore, if we solely focus on computation acceleration for FHE without considering accelerating memory access, we will likely be left with underutilized computation devices bounded by memory access performance.
% \subsubsection{Noise Growth in Ciphertexts}
% FHE ciphertexts conceal plaintext messages with noise that can be identified and removed with the secret key~\cite{brakerski2011fully}. The noise magnitude can be accumulated inside a ciphertext along with its computation. As long as the noise is below a certain threshold controlled by the encryption parameters, decryption can filter out the noise and retrieve the plaintext message successfully; otherwise, the plaintext message could be corrupted, and decryption could fail. Although FHE schemes include a primitive (known as \textit{bootstrapping}) to reduce the noise inside ciphertexts~\cite{STOC:Gentry09}, it is extremely computationally intensive. Instead, a more practical way is to carefully select the encryption parameters to provide just enough noise budget for a ciphertext to accommodate a predefined maximum depth of computation under a specific evaluation circuit.
% \subsubsection{Non-linear Operations}
% \label{sec:ahe_nonlinear}
% FHE schemes' Computation is generally limited to addition and multiplication operations over ciphertexts.
% As a result, it is easy to compute polynomial functions with FHE schemes.
% As with all FHE schemes, encryption injects a bit of noise into the data, and each operation on ciphertexts increases the noise within it.
% Decryption is possible as long as the noise does not exceed some threshold. Otherwise, the decrypted results are essentially meaningless.
% For neural networks, a major stumbling block for translation to the FHE domain is the activation functions.
% These are usually not polynomials and, therefore, unsuitable for evaluation in the FHE domain.
% Therefore, it should be approximated by some polynomial function to try to retain as much accuracy as possible.
% The choice of approximating polynomials depends on the desired performance of the Homomorphic NN.
% It also determines the computational complexity and depth of the activation layers.
% Moreover, with scalar encoding, there is another effect to consider. Namely, the scaling factor on the output will be dependent on the depth of the approximation, i.e., if the scaling factor of the inputs to the activation layer is \(\Delta\), then the scaling factor of the outputs will be rough \(\Delta^{1 + \lceil \log d \rceil}\), where \(d\) is the degree of the polynomial.
% Similar to activations, the usual functions used in pooling layers, maximum~(\(\mathsf{max}(\mathbf{z}) = \max_{1 \leq i \leq n} z_i\)), \(\ell_2\)-norm and mean~(\(\mathsf{avg}(\mathbf{z}) = \frac{1}{n}\sum_{i=1}^{n} z_i\)) for inputs \(\mathbf{z} = (z_1, \ldots, z_n)\), are generally non-polynomial.
% Although \(\mathsf{avg}\) is a linear function, division in FHE schemes is more involved and requires different plaintext encoding methods~(see Dowlin \etal~\cite{IEEE:DGL+17}).
% In CryptoNets~\cite{gilad2016cryptonets}, a variant of the mean function, called scaled-mean~(\(\sum_{i=1}^{n}z_i\)), introduces an additional \(n\) factor over \(\mathsf{avg}\) without impacting its performance.
% \vspace{-15pt}
\subsection{Hardware for HNNs}
\label{sec:ahe_accelerator}
Researchers have proposed acceleration techniques for HNNs on various hardware platforms, \textit{e.g.}, GPUs, Multi-core CPUs, FPGAs, and ASICs. Typically, hardware aids HNNs by speeding up HE operations, as demonstrated in various works~\cite{khalil2010hardware, cousins2016designing, lee2020optimizing, khalil2010hardware}. In particular, \cite{cousins2016designing} introduced an FPGA-based computation accelerator within a Homomorphic Encryption Processing Unit (HEPU) co-processor, mitigating computational bottlenecks in lattice encryption primitives to improve the practicality of computing on encrypted data. Another approach involved optimizing hardware costs, including memory usage and energy consumption, as explored in works such as~\cite{aikata2023reed, kim2023sharp, kim2022bts}. For instance, \cite{aikata2023reed} proposed a cost-effective chiplet-based FHE implementation with a non-blocking inter-chiplet communication strategy to reduce data exchange overhead. Moreover, \cite{doroz2014accelerating} achieved the first full realization of FHE in hardware, while \cite{reagen2021cheetah} presented a custom hardware accelerator architecture combining algorithmic optimizations to accelerate server-side HE inference toward plaintext speeds.
% Various approaches have been explored to accelerate the Homomorphic Encryptions in GPUs, CPUs, FPGAs, and ASICs.
% While attempts have been made to bridge
% this gap with GPU-based solutions [4], [29], they have yet
% to match the performance of FPGA-based works [1], [40],
% [53], [70], which not only exhibit superior performance but
% also provide re-programmability and real-time verifiability.
% Nonetheless, there is still a need to narrow the runtime gap
% between plain and homomorphic computations in FPGA-based
% solutions. Currently, the most notable acceleration results have
% been achieved through ASIC works. However, it is important
% to note that in pursuit of maximizing acceleration, several
% works have deviated from the critical requirement of ensuring
% real-life deployability. This is mainly due to the monolithic
% implementation approach adopted by all ASIC-based works
% in the literature.
% \subsection{How NN Compression helps on Homomorphic NNs}
% \label{sec:ahe_compression}
%Various compression techniques have emerged in the literature to build light-weight neural networks for faster inference, different compression techniques have appeared in the literature~\cite{cheng2018model} such as quantization~\cite{dorefa} and pruning \cite{Rastegari_2016, han2015deep, ICLR2017} to build light-weight neural networks for faster inference.
% In recent years, several compression techniques have emerged in the literature to build light-weight neural networks~\cite{cheng2018model} for faster inference. Quantization~\cite{dorefa} and pruning \cite{Rastegari_2016, han2015deep, ICLR2017} are two popular techniques that have been extensively studied.
%In particular, quantization involves representing network variables, such as weights and activations, with low-precision numbers, leading to smaller model sizes and lower computational costs.
%Recent works~\cite{dorefa} have shown that uniform quantization can be used to reduce the bit-width to 2-bit without significant accuracy drop. The extreme case of quantization is binary neural networks~\cite{Rastegari_2016}, which have weights and activations that are either 1 or -1. Furthermore, quantization can also be integrated with FHE-based Neural Networks to speed up the inference process. For instance, TAPAS~\cite{pmlr-v80-sanyal18a} accelerated FHE-based encrypted prediction via optimizations on binary networks.
% Recent works~\cite{dorefa} have shown that uniform quantization can reduce bit-width to 2-bit without significant accuracy loss. Binary neural networks~\cite{Rastegari_2016} represent an extreme case of quantization, where weights and activations are either 1 or -1.
% \textcolor{blue}{Various model compression techniques in Section~\ref{sec:compression} including model quantization, pruning, and low-rank factorization have been widely used to speed up the inference of Homomorphic NNs. For example, ~\cite{zhang2019encrypted} used Lloyd-max quantization to quantize weights based on the local density in the training. ~\cite{lou2019she} used log-quantization to convert the real-valued data into power-of-two values, i.e., matrix multiplication operations to shift operations, to speed up the inference.
% Some works focus on binarizing the network to speed up the inference of Homomorphic NNs~\cite{bourse2018fast, pmlr-v80-sanyal18a}. TAPAS~\cite{pmlr-v80-sanyal18a} accelerated FHE-based encrypted prediction via optimizations on binary networks. Secondly, some works use network pruning to reduce the inference latency. For example, CryptoNets~\cite{chou2018faster} used unstructured weight pruning to reduce the inference latency of Homomorphic neural networks by skipping the homomorphic operations associated with pruned weights. Some works combine quantization and pruning to speed up the inference. For example,
% ~\cite{chou2018faster}
% proposes to prune and quantize the models to reduce the number of required operations and increase the sparsity in the weights' polynomial encoding. Another direction is to use low-rank factorization techniques. A recent approach proposed in \cite{ffconv} integrates low-rank factorization with FHE ciphertext packing to reduce rotation overhead and achieve a significant acceleration in inference.
% }
% \vspace{-8pt}
\subsection{Challenges in Compressing HNNs}
Performing compression on HNNs presents significant challenges due to the computational and operational complexity of homomorphic encryption, in which multiple factors needed to be taken into consideration, such as the homomorphic encryption schemes, and the ciphertext packing strategies. Notably, neural network quantization works well with bit-wise TFHE schemes~\cite{chillotti2020tfhe} or integer-based BFV schemes~\cite{fan2012somewhat}, but not with CKKS schemes~\cite{cheon2017homomorphic} which encodes float-point numbers. On the other hand, neuron or filter based pruning may not work well with packed weights or activations in ciphertexts. Compatibility issues with homomorphic encryption libraries and the absence of standardization create obstacles in achieving effective compression. Researchers are actively working on overcoming these challenges to make compression in HNNs more practical for real-world applications.
% \textcolor{red}{difficulties of compression on HNNs; advantages and disadvantages}
% Quantization can also speed up the inference process in FHE-based neural networks. For example,
%Pruning is another popular technique that involves removing the redundant weights from the over-parameterized neural network models, which can be categorized into unstructured weight pruning~\cite{han2015deep} and structured filter pruning~\cite{ICLR2017, lin2020hrank}. %The former involves pruning weights that satisfy specific criteria, such as the magnitude of weights smaller than a pre-defined threshold. The latter prunes filters are based on pruning criteria such as L1/L2 norm of filters.
% Pruning is another popular technique that involves removing redundant weights from over-parameterized neural network models. It can be categorized into unstructured pruning~\cite{han2015deep} and structured pruning~\cite{ICLR2017, lin2020hrank}.
% For example, CryptoNets~\cite{chou2018faster} used unstructured weight pruning to reduce the inference latency of Homomorphic neural networks by skipping the homomorphic operations associated with pruned weights. Structured filter pruning removes entire filters from the network. However, it often has a low pruning rate, which limits the inference speed-up. %Structured filter pruning, on the other hand, removes entire filters from the network models based on pruning criteria such as L1/L2 norms of filters. Nevertheless, structured filter pruning often has a low pruning rate, limiting the inference speed-up.
%Another line of research in neural network compress is low-rank factorization~\cite{Masana_2017, KossaifiTBPHP20, Idelbayev_2020_CVPR}, which decomposes a convolution layer into the product of two smaller convolutions with a lower rank. %Though low-rank factorization has been widely studied for accelerating neural network inference in the plaintext domain, %it is mainly unexplored for secure neural network inference with data encrypted by HE schemes.
%it has not been extensively explored for secure neural network inference with data encrypted by HE schemes.
% Low-rank factorization is another line of research in neural network compression \cite{Masana_2017, KossaifiTBPHP20, Idelbayev_2020_CVPR}, which decomposes a convolution layer into the product of two smaller convolutions with a lower rank.
% Low-rank factorization is a well-studied method for accelerating neural network inference in the plaintext domain, but it has not been thoroughly explored for secure neural network inference with encrypted data using HE schemes.
% A recent approach proposed in \cite{ffconv} integrates low-rank factorization with FHE ciphertext packing to reduce rotation overhead and achieve a significant acceleration in inference.
%Recently, a novel approach was proposed in \cite{ffconv}, which integrates low-rank factorization with FHE ciphertext packing to reduce the rotation overhead and achieve significant acceleration in inference. % encrypted data inference.
%In \cite{ffconv}, the authors proposed to integrate low-rank factorization with FHE ciphertext packing to reduce the rotation overhead and thus achieve considerable inference acceleration on encrypted data.
\section{Challenges, Future Trends and Conclusions}
\label{sec:challenges}
\noindent In this article, we have summarized recent works on compressing and accelerating DNNs. %. Here we talk about more details on potential challenges and future trends.
Next, we discuss some potential challenges and future trends.
\textbf{Generalization} While most state-of-the-art compression approaches are designed or verified on CNNs, their performance on other types of networks, especially larger and more complex models such as GPT-3 \cite{brown2020language} and LLaMA \cite{touvron2023llama} with billion parameters, and tiny models is not well established. For example, to reduce the cost of training complex models, post-quantization \cite{yao2022zeroquant, ding2022towards} has been explored to quantize the weights and activations after the model is trained. \cite{xu2022etinynet} further quantizes tiny models ($\le 1\text{MB}$) in aggressive low-bit while retaining the accuracy.
Besides, most latest approaches focus on 2D data, which limits their real applications to handle other data, such as 3D and time-series data. 3D models can be highly complex, with millions of polygons and texture maps. Compressing such models can be a challenging task that requires specialized algorithms and hardware.
Furthermore, due to the existence of domain discrepancy between the model development and deployment stage, compressed models trained on historical data may suffer from performance degradation at the deployment stage. Although some existing works\cite{yang2020mobileda,ryu2022knowledge} have already attempted to simultaneously address domain shift while compressing the model with knowledge distillation, cross-domain model compression is still not well explored.
% Furthermore, cross-domain model compression remains unexplored. Compressed models that are trained on one domain can be easily applied to other domains without the need for additional training. This can save time and resources and improve the efficiency of the model deployment.
% \textcolor{red}{tiny-ml}
%Current model compression techniques focus on applications for 2D data. Deploying them on various applications, including 3D and time-series data remains challenging.
\textbf{Evaluation} Although evaluation metrics (FLOPs, Model size, Latency) have been proposed for evaluating model compression, two % several
drawbacks still exist. First, no standard hardware exists to evaluate various compression techniques. Consequently, %; thus,
many studied %works
focus on theoretical analysis using FLOPs and model size. However, the cost of custom hardware is also affected by its memory and computation capability. Second, %some works use custom hardware to evaluate their approaches. However, the hardware of some works is outdated, so it is not easy to make a fair comparison when conducting experiments on different hardware.
some works use outdated or non-standard hardware for evaluation, making it difficult to perform a fair comparison when conducting experiments on different hardware.
%as it is challenging to run the model compression technique on custom hardware, some works focus on the theoretical analysis without considering the
\textbf{Hardware-software co-design} Hardware constraints in various platforms, such as mobile and IoT chips, %are still the bottleneck that pullbacks
pose a significant challenge for the development of DNNs. %It remains challenging to design specific compression techniques for various platforms.
Designing specific compression techniques for these platforms remains a critical bottleneck. Each platform would provide different operating conditions in energy per operation, memory-access latency, memory capacity, etc. Thus, selecting the compression mechanism should be \textit{energy aware} to maximize the gains of applying such techniques.
Besides, fully using the existing compression approaches is critical. For example, efficiently deploying the sparse architecture of an unstructured pruning network to custom hardware remains a challenge.
%challenging.
\textbf{Integration} Many compression approaches, like model pruning, quantization and knowledge distillation (KD), can be integrated together as they compress the DNNs from different perspectives. For example, there is a growing trend to combine KD with quantization and pruning.
\textbf{ Heterogeneous resource management} Considering the workload and network conditions, it is necessary to distribute DNN processing tasks across heterogeneous computational platforms, including CPU, GPU, the cloud, and smartphones. In this way, the system can remain responsive to near real-time needs.
However, how to connect and collaborate with different devices/platforms remains unexplored. Overall, efficient heterogeneous resource management is critical for the successful deployment of DNNs on a range of platforms. Continued research in this area can help optimize the performance of DNNs while minimizing resource usage.
%Xiaoli - I added this conclusion - you guys may further improve it.
Finally, our survey offers a comprehensive and unique perspective on neural network compression from three different angles: model compression, hardware accelerators, and homomorphic encryption. To the best of our knowledge, this is the first survey paper that comprehensively investigates this problem from software, hardware, and security perspectives. By exploring existing techniques and potential future improvements, we aim to encourage the broader adoption of these approaches in real-world applications. Such adoption can create significant economic and social benefits by making AI more sustainable and cost-efficient in secure environments that save memory, energy, and computation. We believe that our survey can provide valuable information for researchers looking to address critical research problems and further advance the field for the benefit of society.
%Hardware constraints in various small platforms (e.g., mobile, robotic, self-driving cars) are still a significant problem hindering the extension of deep CNNs. How to make full use of the limited computational source available and how to design unique compression methods for such platforms are still challenges that need to be addressed.
%Although there have been benchmarks proposed for evaluating model compression and acceleration, as introduced in Section 2.2, there are several drawbacks in the current evaluation. First, there is no generally recognized setting for model compression and acceleration evaluation. Different studies often yield models with different speed-up ratios, number of parameters, and accuracy. Thus, it is often difficult to directly compare them, not to mention differences in hardware.
%?? Networks: CNN, LSTM, GNN, Transformers
%
%?? Applications: 2D, 3D, Time-series
%
%?? Performance Evaluations
%
%Although irregular sparsity of DNNs can achieve an extremely high ratio, \, e.g., \textcolor{red}{95\%}, it is very difficult for hardware accelerators to utilize.
%An increasing number of software/hardware co-optimization research works are exploring the regular sparse hence the hardware could utilize the spare with less complexity.
%
%Another trend is the hardware acceleration for transformers, such as ViT accelerators and NLP transformer accelerators.
%However, as there are millions of categories in NLP transformers, the softmax operation has been the bottleneck of NLP transformer accelerators as current softmax hardware accelerators only support a relatively small number of categories.
%The hardware solution of accelerating softmax operations in NLP transformers should be proposed with software/hardware co-design and co-optimization.
% Besides, more ViT hardware solutions will be reported.
\section{Acknowledgements}
\label{sec:acknowledgements}
\noindent This research is supported by the Agency for Science, Technology, and Research (A*STAR) under its Funds (AME Programmatic Funds A1892b0026, A19E3b0099, MTC Programmatic Fund M23L7b0021, NRF AME Young Individual Research Grant A2084c1067 and Career Development Fund C210812035). However, any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reflect the views of the A*STAR.
\bibliographystyle{IEEEtran}
\bibliography{IEEEabrv, reference.bib}
% \begin{thebibliography}{1}
% \bibliographystyle{IEEEtran}
% \bibitem{ref1}
% {\it{Mathematics Into Type}}. American Mathematical Society. [Online]. Available: https://www.ams.org/arc/styleguide/mit-2.pdf
% \bibitem{ref2}
% T. W. Chaundy, P. R. Barrett and C. Batey, {\it{The Printing of Mathematics}}. London, U.K., Oxford Univ. Press, 1954.
% \bibitem{ref3}
% F. Mittelbach and M. Goossens, {\it{The \LaTeX Companion}}, 2nd ed. Boston, MA, USA: Pearson, 2004.
% \bibitem{ref4}
% G. Gr\"atzer, {\it{More Math Into LaTeX}}, New York, NY, USA: Springer, 2007.
% \bibitem{ref5}M. Letourneau and J. W. Sharp, {\it{AMS-StyleGuide-online.pdf,}} American Mathematical Society, Providence, RI, USA, [Online]. Available: http://www.ams.org/arc/styleguide/index.html
% \bibitem{ref6}
% H. Sira-Ramirez, ``On the sliding mode control of nonlinear systems,'' \textit{Syst. Control Lett.}, vol. 19, pp. 303--312, 1992.
% \bibitem{ref7}
% A. Levant, ``Exact differentiation of signals with unbounded higher derivatives,'' in \textit{Proc. 45th IEEE Conf. Decis.
% Control}, San Diego, CA, USA, 2006, pp. 5585--5590. DOI: 10.1109/CDC.2006.377165.
% \bibitem{ref8}
% M. Fliess, C. Join, and H. Sira-Ramirez, ``Non-linear estimation is easy,'' \textit{Int. J. Model., Ident. Control}, vol. 4, no. 1, pp. 12--27, 2008.
% \bibitem{ref9}
% R. Ortega, A. Astolfi, G. Bastin, and H. Rodriguez, ``Stabilization of food-chain systems using a port-controlled Hamiltonian description,'' in \textit{Proc. Amer. Control Conf.}, Chicago, IL, USA,
% 2000, pp. 2245--2249.
% \end{thebibliography}
% \newpage
% \section{Biography Section}
% If you have an EPS/PDF photo (graphicx package needed), extra braces are
% needed around the contents of the optional argument to biography to prevent
% the LaTeX parser from getting confused when it sees the complicated
% $\backslash${\tt{includegraphics}} command within an optional argument. (You can create
% your own custom macro containing the $\backslash${\tt{includegraphics}} command to make things
% simpler here.)
% \vspace{11pt}
% \bf{If you include a photo:}\vspace{-33pt}
% \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{fig1}}]{Michael Shell}
% Use $\backslash${\tt{begin\{IEEEbiography\}}} and then for the 1st argument use $\backslash${\tt{includegraphics}} to declare and link the author photo.
% Use the author name as the 3rd argument followed by the biography text.
% \end{IEEEbiography}
% \vspace{11pt}
% \bf{If you will not include a photo:}\vspace{-33pt}
% \begin{IEEEbiographynophoto}{John Doe}
% Use $\backslash${\tt{begin\{IEEEbiographynophoto\}}} and the author name as the argument followed by the biography text.
% \end{IEEEbiographynophoto}
\vfill
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{100}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{chung2022scaling}
H.~W. Chung, L.~Hou, S.~Longpre, B.~Zoph, Y.~Tay, W.~Fedus, E.~Li, X.~Wang, M.~Dehghani, S.~Brahma \emph{et~al.}, ``Scaling instruction-finetuned language models,'' \emph{arXiv preprint arXiv:2210.11416}, 2022.
\bibitem{sze2017efficient}
V.~Sze, Y.-H. Chen, T.-J. Yang, and J.~S. Emer, ``Efficient processing of deep neural networks: A tutorial and survey,'' \emph{Proceedings of the IEEE}, 2017.
\bibitem{varghese2016challenges}
B.~Varghese, N.~Wang, S.~Barbhuiya, P.~Kilpatrick, and D.~S. Nikolopoulos, ``Challenges and opportunities in edge computing,'' in \emph{IEEE SmartCloud}, 2016.
\bibitem{deng2020model}
L.~Deng, G.~Li, S.~Han, L.~Shi, and Y.~Xie, ``Model compression and hardware acceleration for neural networks: A comprehensive survey,'' \emph{Proceedings of the IEEE}, 2020.
\bibitem{mishra2023transforming}
R.~Mishra and H.~Gupta, ``Transforming large-size to lightweight deep neural networks for iot applications,'' \emph{ACM Computing Surveys}, 2023.
\bibitem{cheng2018model}
Y.~Cheng, D.~Wang, P.~Zhou, and T.~Zhang, ``Model compression and acceleration for deep neural networks: The principles, progress, and challenges,'' \emph{IEEE Signal Processing Magazine}, 2018.
\bibitem{mishra2020survey}
R.~Mishra, H.~P. Gupta, and T.~Dutta, ``A survey on deep neural network compression: Challenges, overview, and solutions,'' \emph{arXiv preprint arXiv:2010.03954}, 2020.
\bibitem{neill2020overview}
J.~O. Neill, ``An overview of neural network compression,'' \emph{arXiv preprint arXiv:2006.03669}, 2020.
\bibitem{alqahtani2021literature}
A.~Alqahtani, X.~Xie, and M.~W. Jones, ``Literature review of deep network compression,'' in \emph{Informatics}.\hskip 1em plus 0.5em minus 0.4em\relax Multidisciplinary Digital Publishing Institute, 2021.
\bibitem{berthelier2021deep}
A.~Berthelier, T.~Chateau, S.~Duffner, C.~Garcia, and C.~Blanc, ``Deep model compression and architecture optimization for embedded systems: A survey,'' \emph{Journal of Signal Processing Systems}, 2021.
\bibitem{mittal2021survey}
S.~Mittal and S.~Umesh, ``A survey on hardware accelerators and optimization techniques for rnns,'' \emph{Journal of Systems Architecture}, 2021.
\bibitem{nan2019deep}
K.~Nan, S.~Liu, J.~Du, and H.~Liu, ``Deep model compression for mobile platforms: A survey,'' \emph{Tsinghua Science and Technology}, 2019.
\bibitem{xu2022survey}
C.~Xu and J.~McAuley, ``A survey on model compression for natural language processing,'' \emph{arXiv preprint arXiv:2202.07105}, 2022.
\bibitem{gupta2022compression}
M.~Gupta and P.~Agrawal, ``Compression of deep learning models for text: A survey,'' \emph{TKDD}, 2022.
\bibitem{guo2018survey}
Y.~Guo, ``A survey on methods and theories of quantized neural networks,'' \emph{arXiv preprint arXiv:1808.04752}, 2018.
\bibitem{gholami2021survey}
A.~Gholami, S.~Kim, Z.~Dong, Z.~Yao, M.~W. Mahoney, and K.~Keutzer, ``A survey of quantization methods for efficient neural network inference,'' \emph{arXiv preprint arXiv:2103.13630}, 2021.
\bibitem{xu2020convolutional}
S.~Xu, A.~Huang, L.~Chen, and B.~Zhang, ``Convolutional neural network pruning: A survey,'' in \emph{CCC}, 2020.
\bibitem{liu2020pruning}
J.~Liu, S.~Tripathi, U.~Kurup, and M.~Shah, ``Pruning algorithms to accelerate convolutional neural networks for edge applications: A survey,'' \emph{arXiv preprint arXiv:2005.04275}, 2020.
\bibitem{reed1993pruning}
R.~Reed, ``Pruning algorithms-a survey,'' \emph{IEEE TNNLS}, 1993.
\bibitem{elsken2019neural}
T.~Elsken, J.~H. Metzen, and F.~Hutter, ``Neural architecture search: A survey,'' \emph{JMLR}, 2019.
\bibitem{ren2021comprehensive}
P.~Ren, Y.~Xiao, X.~Chang, P.-Y. Huang, Z.~Li, X.~Chen, and X.~Wang, ``A comprehensive survey of neural architecture search: Challenges and solutions,'' \emph{CSUR}, 2021.
\bibitem{wistuba2019survey}
M.~Wistuba, A.~Rawat, and T.~Pedapati, ``A survey on neural architecture search,'' \emph{arXiv preprint arXiv:1905.01392}, 2019.
\bibitem{gou2021knowledge}
J.~Gou, B.~Yu, S.~J. Maybank, and D.~Tao, ``Knowledge distillation: A survey,'' \emph{IJCV}, 2021.
\bibitem{wang2021knowledge}
L.~Wang and K.-J. Yoon, ``Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,'' \emph{T-PAMI}, 2021.
\bibitem{han16iclr}
S.~Han, H.~Mao, and W.~J. Dally, ``Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,'' in \emph{ICLR}, 2016.
\bibitem{Zhang18eccv}
D.~Zhang, J.~Yang, D.~Ye, and G.~Hua, ``Lq-nets: learned quantization for highly accurate and compact deep neural networks,'' in \emph{ECCV}, 2018.
\bibitem{inq}
A.~Zhou, A.~Yao, Y.~Guo, L.~Xu, and Y.~Chen, ``Incremental network quantization: {T}owards lossless {cnn}s with low-precision weights,'' in \emph{arXiv preprint arXiv:1702.03044}, 2017.
\bibitem{dorefa}
S.~Zhou, Y.~Wu, Z.~Ni, X.~Zhou, H.~Wen, and Y.~Zou, ``Dorefa-net: {T}raining low bitwidth convolutional neural networks with low bitwidth gradients,'' in \emph{arXiv preprint arXiv:1606.06160}, 2016.
\bibitem{Li16nips}
F.~Li, B.~Zhang, and B.~Liu, ``Ternary weight networks,'' in \emph{NIPS Workshop}, 2016.
\bibitem{pq}
Y.~Gong, L.~Liu, M.~Yang, and L.~Bourdev, ``Compressing deep convolutional networks using vector quantization,'' in \emph{ICLR}, 2015.
\bibitem{hash}
W.~Chen, J.~T. Wilson, S.~Tyree, K.~Q. Weinberger, and Y.~Chen, ``Compressing neural networks with the hashing trick,'' in \emph{ICML}, 2015.
\bibitem{frequency}
W.~Chen, J.~Wilson, S.~Tyree, K.~Q. Weinberger, and Y.~Chen, ``Compressing convolutional neural networks in the frequency domain,'' in \emph{SIGKDD}, 2016.
\bibitem{revisit}
P.~Stock, A.~Joulin, R.~Gribonval, B.~Graham, and H.~Jegou, ``And the bit goes down: revisiting the quantization of neural networks,'' in \emph{ICLR}, 2020.
\bibitem{APoT}
Y.~Li, X.~Dong, and W.~Wang, ``Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks,'' in \emph{ICLR}, 2020.
\bibitem{Faraone2018arxiv}
J.~Faraone, N.~Fraser, M.~Blott, and P.~H. Leong, ``Syq: Learning symmetric quantization for efficient deep neural networks,'' in \emph{arXiv}, 2018.
\bibitem{releq}
A.~T. Elthakeb, P.~Pilligundla, F.~Mireshghallah, A.~Yazdanbakhsh, and H.~Esmaeilzadeh, ``Releq: A reinforcement learning approach for deep quantization of neural networks,'' in \emph{NeurIPS Workshop on ML for Systems}, 2018.
\bibitem{wang19cvpr}
K.~Wang, Z.~Liu, Y.~Lin, J.~Lin, and S.~Han, ``H{A}{Q}: Hardware-aware automated quantization with mixed precision,'' in \emph{CVPR}, 2019.
\bibitem{dnas}
B.~Wu, Y.~Wang, P.~Zhang, Y.~Tian, P.~Vajda, and K.~Keutzer, ``Mixed precision quantization of convnets via differentiable neural architecture search,'' in \emph{ICLR}, 2019.
\bibitem{dq}
S.~Uhlich, L.~Mauch, F.~Cardinaux, K.~Yoshiyama, J.~A. Garcia, S.~Tiedemann, T.~Kemp, and A.~Nakamura, ``Mixed precision dnns: All you need is a good parametrization,'' \emph{arXiv preprint arXiv:1905.11452}, 2019.
\bibitem{HAWQ}
Z.~Dong, Z.~Yao, A.~Gholami, M.~W. Mahoney, and K.~Keutzer, ``Hawq: Hessian aware quantization of neural networks with mixed-precision,'' in \emph{arXiv}, 2019.
\bibitem{ALQ}
Z.~Qu, Z.~Zhou, Y.~Cheng, and L.~Thiele, ``Adaptive loss-aware quantization for multi-bit networks,'' in \emph{arXiv}, 2020.
\bibitem{post4bits}
R.~Banner, Y.~Nahshan, E.~Hoffer, and D.~Soudry, ``Post-training 4-bit quantization of convolution networks for rapid-deployment,'' \emph{arXiv preprint arXiv:1810.05723}, 2018.
\bibitem{FracBits}
L.~Yang and Q.~Jin, ``Fracbits: Mixed precision quantization via fractional bit-widths,'' \emph{arXiv preprint arXiv:2007.02017}, 2020.
\bibitem{DMBQ}
S.~Zhao, T.~Yue, and X.~Hu, ``Distribution-aware adaptive multi-bit quantization,'' in \emph{CVPR}, 2021.
\bibitem{autoq}
Q.~Lou, F.~Guo, M.~Kim, L.~Liu, and L.~Jiang, ``Autoq: Automated kernel-wise neural network quantizations,'' in \emph{ICLR}, 2020.
\bibitem{coresets}
A.~Dubey, M.~Chatterjee, and N.~Ahuja, ``Coreset-based neural network compression,'' in \emph{arXiv:1807.09810}, 2018.
\bibitem{DeepCABAC}
S.~W. et~al., ``Deepcabac: Context-adaptive binary arithmetic coding for deep neural network compression,'' in \emph{ICML Workshop}, 2019.
\bibitem{EPR}
D.~Oktay, J.~Ballé, S.~Singh, and A.~Shrivastava, ``Scalable model compression by entropy penalized reparameterization,'' in \emph{ICLR}, 2020.
\bibitem{zhe2021rate}
W.~Zhe, J.~Lin, M.~S. Aly, S.~Young, V.~Chandrasekhar, and B.~Girod, ``Rate-distortion optimized coding for efficient cnn compression,'' in \emph{DCC}, 2021.
\bibitem{khoram18iclr}
S.~Khoram and J.~Li, ``Adaptive quantization of neural networks,'' in \emph{ICLR}, 2018.
\bibitem{young2021transform}
S.~Young, Z.~Wang, D.~Taubman, and B.~Girod, ``Transform quantization for cnn compression,'' \emph{T-PAMI}, 2021.
\bibitem{dct}
Y.~Wang, C.~Xu, S.~You, D.~Tao, and C.~Xu, ``Cnnpack: packing convolutional neural networks in the frequency domain,'' in \emph{NIPS}, 2016.
\bibitem{svd}
E.~L. Denton, W.~Zaremba, J.~Bruna, Y.~LeCun, and R.~Fergus, ``Exploiting linear structure within convolutional networks for efficient evaluation,'' in \emph{NIPS}, 2014.
\bibitem{svd_p}
X.~Zhang, J.~Zou, X.~Ming, K.~He, and J.~Sun, ``Efficient and accurate approximations of nonlinear convolutional networks,'' \emph{CVPR}, 2015.
\bibitem{svd_p2}
Y.-D. Kim, E.~Park, S.~Yoo, T.~Choi, L.~Yang, and D.~Shin, ``Compression of deep convolutional neural networks for fast and low power mobile applications,'' in \emph{ICLR}, 2016.
\bibitem{svd_p3}
Y.~Li, S.~Gu, L.~V. Gool, and R.~Timofte, ``Learning filter basis for convolutional neural network compression,'' \emph{ICCV}, 2019.
\bibitem{li2021bossnas}
C.~Li, T.~Tang, G.~Wang, J.~Peng, B.~Wang, X.~Liang, and X.~Chang, ``Bossnas: Exploring hybrid cnn-transformers with block-wisely self-supervised neural architecture search,'' in \emph{ICCV}, 2021.
\bibitem{chen2021autoformer}
M.~Chen, H.~Peng, J.~Fu, and H.~Ling, ``Autoformer: Searching transformers for visual recognition,'' in \emph{ICCV}, 2021.
\bibitem{bondarenko2021understanding}
Y.~Bondarenko, M.~Nagel, and T.~Blankevoort, ``Understanding and overcoming the challenges of efficient transformer quantization,'' in \emph{EMNLP}, 2021.
\bibitem{dettmers2022gpt3}
T.~Dettmers, M.~Lewis, Y.~Belkada, and L.~Zettlemoyer, ``Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,'' \emph{BIPS}, 2022.
\bibitem{liu2021post}
Z.~Liu, Y.~Wang, K.~Han, W.~Zhang, S.~Ma, and W.~Gao, ``Post-training quantization for vision transformer,'' \emph{NIPS}, 2021.
\bibitem{ding2022towards}
Y.~Ding, H.~Qin, Q.~Yan, Z.~Chai, J.~Liu, X.~Wei, and X.~Liu, ``Towards accurate post-training quantization for vision transformer,'' in \emph{ACM MM}, 2022.
\bibitem{taubmanjpeg2000}
D.~S. Taubman and M.~W. Marcellin, ``Jpeg2000 image compression fundamentals, standards and practice,'' \emph{Journal of Electronic Imaging}, 2002.
\bibitem{Tunstall1067phd}
B.~P. Tunstall, ``Synthesis of noiseless compression codes,'' in \emph{PhD diss., Georgia Institute of Technology}, 1967.
\bibitem{lee2018snip}
N.~Lee, T.~Ajanthan, and P.~H. Torr, ``Snip: Single-shot network pruning based on connection sensitivity,'' in \emph{ICLR}, 2018.
\bibitem{Wang2020Picking}
C.~Wang, G.~Zhang, and R.~Grosse, ``Picking winning tickets before training by preserving gradient flow,'' in \emph{ICLR}, 2020.
\bibitem{jorge2021progressive}
P.~de~Jorge, A.~Sanyal, H.~Behl, P.~Torr, G.~Rogez, and P.~K. Dokania, ``Progressive skeletonization: Trimming more fat from a network at initialization,'' in \emph{ICLR}, 2021.
\bibitem{ijcai2022p786}
H.~Wang, C.~Qin, Y.~Bai, Y.~Zhang, and Y.~Fu, ``Recent advances on neural network pruning at initialization,'' in \emph{IJCAI}, 2022.
\bibitem{NIPS1989_250}
Y.~LeCun, J.~S. Denker, and S.~A. Solla, ``Optimal brain damage,'' in \emph{NIPS}, 1990.
\bibitem{woodfisher}
S.~P. Singh and D.~Alistarh, ``Woodfisher: Efficient second-order approximation for neural network compression,'' in \emph{NIPS}, 2020.
\bibitem{mfac}
E.~Frantar, E.~Kurtic, and D.~Alistarh, ``M-{FAC}: Efficient matrix-free approximations of second-order information,'' in \emph{NIPS}, 2021.
\bibitem{shen2022structural}
M.~Shen, H.~Yin, P.~Molchanov, L.~Mao, J.~Liu, and J.~M. Alvarez, ``Structural pruning via latency-saliency knapsack,'' \emph{NIPS}, 2022.
\bibitem{molchanov2019importance}
P.~Molchanov, A.~Mallya, S.~Tyree, I.~Frosio, and J.~Kautz, ``Importance estimation for neural network pruning,'' in \emph{CVPR}, 2019.
\bibitem{xu2023efficient}
K.~Xu, Z.~Wang, X.~Geng, M.~Wu, X.~Li, and W.~Lin, ``Efficient joint optimization of layer-adaptive weight pruning in deep neural networks,'' in \emph{ICCV}, 2023.
\bibitem{Lin2020Dynamic}
T.~Lin, S.~U. Stich, L.~Barba, D.~Dmitriev, and M.~Jaggi, ``Dynamic model pruning with feedback,'' in \emph{ICLR}, 2020.
\bibitem{LIU2020Dynamic}
J.~Liu, Z.~XU, R.~SHI, R.~C.~C. Cheung, and H.~K. So, ``Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers,'' in \emph{ICLR}, 2020.
\bibitem{molchanov2016pruning}
P.~Molchanov, S.~Tyree, T.~Karras, T.~Aila, and J.~Kautz, ``Pruning convolutional neural networks for resource efficient inference,'' in \emph{ICLR}, 2017.
\bibitem{tang2020scop}
Y.~Tang, Y.~Wang, Y.~Xu, D.~Tao, C.~Xu, C.~Xu, and C.~Xu, ``Scop: Scientific control for reliable neural network pruning,'' \emph{NIPS}, 2020.
\bibitem{pmlr-v119-kusupati20a}
A.~Kusupati, V.~Ramanujan, R.~Somani, M.~Wortsman, P.~Jain, S.~Kakade, and A.~Farhadi, ``Soft threshold weight reparameterization for learnable sparsity,'' in \emph{ICML}, 2020.
\bibitem{ContinuousSparsification2020}
P.~Savarese, H.~Silva, and M.~Maire, ``Winning the lottery with continuous sparsification,'' in \emph{NIPS}, 2020.
\bibitem{wang2021neural}
H.~Wang, C.~Qin, Y.~Zhang, and Y.~Fu, ``Neural pruning via growing regularization,'' in \emph{ICLR}, 2021.
\bibitem{pmlr-v119-evci20a}
U.~Evci, T.~Gale, J.~Menick, P.~S. Castro, and E.~Elsen, ``Rigging the lottery: Making all tickets winners,'' in \emph{ICML}, 2020.
\bibitem{Park2020LookaheadAF}
S.~Park, J.~Lee, S.~Mo, and J.~Shin, ``Lookahead: a far-sighted alternative of magnitude-based pruning,'' in \emph{ICLR}, 2020.
\bibitem{Zhu2018ToPO}
M.~Zhu and S.~Gupta, ``To prune, or not to prune: exploring the efficacy of pruning for model compression,'' in \emph{ICLR Workshop}, vol. abs/1710.01878, 2018.
\bibitem{lee2021layeradaptive}
J.~Lee, S.~Park, S.~Mo, S.~Ahn, and J.~Shin, ``Layer-adaptive sparsity for the magnitude-based pruning,'' in \emph{ICLR}, 2021.
\bibitem{globalmp}
M.~Gupta, E.~Camci, V.~R. Keneta, A.~Vaidyanathan, R.~Kanodia, C.-S. Foo, W.~Min, and L.~Jie, ``Is complexity required for neural network pruning? a case study on global magnitude pruning,'' \emph{arXiv preprint arXiv:2209.14624}, 2022.
\bibitem{ICML-2019-MostafaW}
H.~Mostafa and X.~Wang, ``{Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization},'' in \emph{{ICML}}, 2019.
\bibitem{sparse_momentum}
T.~Dettmers and L.~Zettlemoyer, ``Sparse networks from scratch: Faster training without losing performance,'' \emph{CoRR}, vol. abs/1907.04840, 2019.
\bibitem{gupta2020learning}
M.~Gupta, S.~Aravindan, A.~Kalisz, V.~Chandrasekhar, and L.~Jie, ``Learning to prune deep neural networks via reinforcement learning,'' \emph{ICML AutoML Workshop}, 2020.
\bibitem{he2018amc}
{He, Yihui}, J.~Lin, Z.~Liu, H.~Wang, L.-J. Li, and S.~Han, ``Amc: Automl for model compression and acceleration on mobile devices,'' in \emph{ECCV}, 2018.
\bibitem{NIPS2017_6813}
J.~Lin, Y.~Rao, J.~Lu, and J.~Zhou, ``Runtime neural pruning,'' in \emph{NIPS}, 2017.
\bibitem{frankle2018the}
J.~Frankle and M.~Carbin, ``The lottery ticket hypothesis: Finding sparse, trainable neural networks,'' in \emph{ICLR}, 2019.
\bibitem{banner2019post}
R.~Banner, Y.~Nahshan, and D.~Soudry, ``Post training 4-bit quantization of convolutional networks for rapid-deployment,'' \emph{NIPS}, 2019.
\bibitem{frantar2022optimal}
E.~Frantar and D.~Alistarh, ``Optimal brain compression: A framework for accurate post-training quantization and pruning,'' \emph{NIPS}, 2022.
\bibitem{kwon2022fast}
W.~Kwon, S.~Kim, M.~W. Mahoney, J.~Hassoun, K.~Keutzer, and A.~Gholami, ``A fast post-training pruning framework for transformers,'' \emph{NIPS}, 2022.
\bibitem{DNW}
M.~Wortsman, A.~Farhadi, and M.~Rastegari, ``Discovering neural wirings,'' in \emph{NIPS}, 2019.
\bibitem{8953212}
Y.~He, P.~Liu, Z.~Wang, Z.~Hu, and Y.~Yang, ``Filter pruning via geometric median for deep convolutional neural networks acceleration,'' in \emph{CVPR}, 2019.
\bibitem{lin2020hrank}
M.~Lin, R.~Ji, Y.~Wang, Y.~Zhang, B.~Zhang, Y.~Tian, and L.~Shao, ``Hrank: Filter pruning using high-rank feature map,'' in \emph{CVPR}, 2020.
\bibitem{Wang2020PruningFS}
Y.~Wang, X.~Zhang, L.~Xie, J.~Zhou, H.~Su, B.~Zhang, and X.~Hu, ``Pruning from scratch,'' in \emph{AAAI}, 2020.
\bibitem{10.1007/978-3-030-58598-3_36}
Y.~Li, S.~Gu, K.~Zhang, L.~Van~Gool, and R.~Timofte, ``Dhp: Differentiable meta pruning via hypernetworks,'' in \emph{ECCV}, 2020.
\bibitem{8953628}
S.~Lin, R.~Ji, C.~Yan, B.~Zhang, L.~Cao, Q.~Ye, F.~Huang, and D.~Doermann, ``Towards optimal structured cnn pruning via generative adversarial learning,'' in \emph{CVPR}, 2019.
\bibitem{CHL00L21}
W.~Wang, M.~Chen, S.~Zhao, L.~Chen, J.~Hu, H.~Liu, D.~Cai, X.~He, and W.~Liu, ``Accelerate cnns from three dimensions: A comprehensive pruning framework,'' in \emph{ICML}, 2021.
\bibitem{lagunas2021block}
F.~Lagunas, E.~Charlaix, V.~Sanh, and A.~M. Rush, ``Block pruning for faster transformers,'' in \emph{EMNLP}, 2021.
\bibitem{zhou2021learning}
A.~Zhou, Y.~Ma, J.~Zhu, J.~Liu, Z.~Zhang, K.~Yuan, W.~Sun, and H.~Li, ``Learning n: m fine-grained structured sparse neural networks from scratch,'' \emph{arXiv preprint arXiv:2102.04010}, 2021.
\bibitem{fang2022algorithm}
C.~Fang, A.~Zhou, and Z.~Wang, ``An algorithm--hardware co-optimized framework for accelerating n: M sparse transformers,'' \emph{IEEE T-VLSI}, 2022.
\bibitem{yang2023global}
H.~Yang, H.~Yin, M.~Shen, P.~Molchanov, H.~Li, and J.~Kautz, ``Global vision transformer pruning with hessian-aware saliency,'' in \emph{CVPR}, 2023.
\bibitem{yu2023boost}
C.~Yu, T.~Chen, Z.~Gan, and J.~Fan, ``Boost vision transformer with gpu-friendly sparsity and quantization,'' in \emph{CVPR}, 2023.
\bibitem{cheng2023survey}
H.~Cheng, M.~Zhang, and J.~Q. Shi, ``A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations,'' \emph{arXiv preprint arXiv:2308.06767}, 2023.
\bibitem{yu2018slimmable}
J.~Yu, L.~Yang, N.~Xu, J.~Yang, and T.~Huang, ``Slimmable neural networks,'' in \emph{ICLR}, 2018.
\bibitem{yu2019universally}
J.~Yu and T.~S. Huang, ``Universally slimmable networks and improved training techniques,'' in \emph{ICCV}, 2019.
\bibitem{li2021dynamic}
C.~Li, G.~Wang, B.~Wang, X.~Liang, Z.~Li, and X.~Chang, ``Dynamic slimmable network,'' in \emph{CVPR}, 2021.
\bibitem{cai2019once}
H.~Cai, C.~Gan, T.~Wang, Z.~Zhang, and S.~Han, ``Once-for-all: Train one network and specialize it for efficient deployment,'' \emph{ICLR}, 2020.
\bibitem{wang2017model}
C.~Wang, X.~Lan, and Y.~Zhang, ``Model distillation with knowledge transfer from face classification to alignment and verification,'' \emph{arXiv preprint arXiv:1709.02929}, 2017.
\bibitem{lee2018teacher}
H.~J. Lee, W.~J. Baddar, H.~G. Kim, S.~T. Kim, and Y.~M. Ro, ``Teacher and student joint learning for compact facial landmark detection network,'' in \emph{MMM}, 2018.
\bibitem{zhu2021student}
Y.~Zhu and Y.~Wang, ``Student customized knowledge distillation: Bridging the gap between student and teacher,'' in \emph{ICCV}, 2021.
\bibitem{park2021learning}
D.~Y. Park, M.-H. Cha, D.~Kim, B.~Han \emph{et~al.}, ``Learning student-friendly teacher networks for knowledge distillation,'' in \emph{NIPS}, 2021.
\bibitem{kim2016sequence}
Y.~Kim and A.~M. Rush, ``Sequence-level knowledge distillation,'' \emph{arXiv preprint arXiv:1606.07947}, 2016.
\bibitem{gordon2019explaining}
M.~A. Gordon and K.~Duh, ``Explaining sequence-level knowledge distillation as data-augmentation for neural machine translation,'' \emph{arXiv preprint arXiv:1912.03334}, 2019.
\bibitem{wang2021joint}
Z.-R. Wang and J.~Du, ``Joint architecture and knowledge distillation in cnn for chinese text recognition,'' \emph{Pattern Recognition}, 2021.
\bibitem{chebotar2016distilling}
Y.~Chebotar and A.~Waters, ``Distilling knowledge from ensembles of neural networks for speech recognition.'' in \emph{Interspeech}, 2016.
\bibitem{kwon2020adaptive}
K.~Kwon, H.~Na, H.~Lee, and N.~S. Kim, ``Adaptive knowledge distillation based on entropy,'' in \emph{ICASSP}, 2020.
\bibitem{li2021mutual}
Z.~Li, Y.~Ming, L.~Yang, and J.-H. Xue, ``Mutual-learning sequence-level knowledge distillation for automatic speech recognition,'' \emph{Neurocomputing}, 2021.
\bibitem{xu2021kdnet}
Q.~Xu, Z.~Chen, K.~Wu, C.~Wang, M.~Wu, and X.~Li, ``Kdnet-rul: A knowledge distillation framework to compress deep neural networks for machine remaining useful life prediction,'' \emph{IEEE Transactions on Industrial Electronics}, 2021.
\bibitem{xu2022contrastive}
Q.~Xu, Z.~Chen, M.~Ragab, C.~Wang, M.~Wu, and X.~Li, ``Contrastive adversarial knowledge distillation for deep model compression in time-series regression tasks,'' \emph{Neurocomputing}, 2022.
\bibitem{hinton2015distilling}
G.~Hinton, O.~Vinyals, J.~Dean \emph{et~al.}, ``Distilling the knowledge in a neural network,'' \emph{arXiv preprint arXiv:1503.02531}, 2015.
\bibitem{zhang2018better}
C.~Zhang and Y.~Peng, ``Better and faster: knowledge transfer from multiple self-supervised learning tasks via graph distillation for video classification,'' in \emph{IJCAI}, 2018.
\bibitem{Zhao_2022_CVPR}
B.~Zhao, Q.~Cui, R.~Song, Y.~Qiu, and J.~Liang, ``Decoupled knowledge distillation,'' in \emph{CVPR}, 2022.
\bibitem{chen2017learning}
G.~Chen, W.~Choi, X.~Yu, T.~Han, and M.~Chandraker, ``Learning efficient object detection models with knowledge distillation,'' in \emph{NIPS}, 2017.
\bibitem{yuan2019obtain}
C.~Yuan and R.~Pan, ``Obtain dark knowledge via extended knowledge distillation,'' in \emph{AIAM}, 2019.
\bibitem{hegde2020variational}
S.~Hegde, R.~Prasad, R.~Hebbalaguppe, and V.~Kumar, ``Variational student: Learning compact and sparser networks in knowledge distillation framework,'' in \emph{ICASSP}, 2020.
\bibitem{saputra2019distilling}
M.~R.~U. Saputra, P.~P. De~Gusmao, Y.~Almalioglu, A.~Markham, and N.~Trigoni, ``Distilling knowledge from a deep pose regressor network,'' in \emph{ICCV}, 2019.
\bibitem{zagoruyko2016paying}
S.~Zagoruyko and N.~Komodakis, ``Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,'' in \emph{ICLR}, 2017.
\bibitem{yim2017gift}
J.~Yim, D.~Joo, J.~Bae, and J.~Kim, ``A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,'' in \emph{CVPR}, 2017.
\bibitem{kim2018paraphrasing}
J.~Kim, S.~Park, and N.~Kwak, ``Paraphrasing complex network: Network compression via factor transfer,'' in \emph{NIPS}, 2018.
\bibitem{ahn2019variational}
S.~Ahn, S.~X. Hu, A.~Damianou, N.~D. Lawrence, and Z.~Dai, ``Variational information distillation for knowledge transfer,'' in \emph{CVPR}, 2019.
\bibitem{liu2021exploring}
L.~Liu, Q.~Huang, S.~Lin, H.~Xie, B.~Wang, X.~Chang, and X.~Liang, ``Exploring inter-channel correlation for diversity-preserved knowledge distillation,'' in \emph{ICCV}, 2021.
\bibitem{lin2022knowledge}
S.~Lin, H.~Xie, B.~Wang, K.~Yu, X.~Chang, X.~Liang, and G.~Wang, ``Knowledge distillation via the target-aware transformer,'' in \emph{CVPR}, 2022.
\bibitem{shang2021lipschitz}
Y.~Shang, B.~Duan, Z.~Zong, L.~Nie, and Y.~Yan, ``Lipschitz continuity guided knowledge distillation,'' in \emph{ICCV}, 2021.
\bibitem{huang2021revisiting}
Z.~Huang, X.~Shen, J.~Xing, T.~Liu, X.~Tian, H.~Li, B.~Deng, J.~Huang, and X.-S. Hua, ``Revisiting knowledge distillation: An inheritance and exploration framework,'' in \emph{CVPR}, 2021.
\bibitem{ji2021show}
M.~Ji, B.~Heo, and S.~Park, ``Show, attend and distill: Knowledge distillation via attention-based feature matching,'' in \emph{AAAI}, 2021.
\bibitem{passalis2018learning}
N.~Passalis and A.~Tefas, ``Learning deep representations with probabilistic knowledge transfer,'' in \emph{ECCV}, 2018.
\bibitem{park2019relational}
W.~Park, D.~Kim, Y.~Lu, and M.~Cho, ``Relational knowledge distillation,'' in \emph{CVPR}, 2019.
\bibitem{liu2019knowledge}
Y.~Liu, J.~Cao, B.~Li, C.~Yuan, W.~Hu, Y.~Li, and Y.~Duan, ``Knowledge distillation via instance relationship graph,'' in \emph{CVPR}, 2019.
\bibitem{huang2022knowledge}
T.~Huang, S.~You, F.~Wang, C.~Qian, and C.~Xu, ``Knowledge distillation from a stronger teacher,'' in \emph{NIPS}, 2022.
\bibitem{yun2020regularizing}
S.~Yun, J.~Park, K.~Lee, and J.~Shin, ``Regularizing class-wise predictions via self-knowledge distillation,'' in \emph{CVPR}, 2020.
\bibitem{tian2019contrastive}
Y.~Tian, D.~Krishnan, and P.~Isola, ``Contrastive representation distillation,'' in \emph{ICLR}, 2020.
\bibitem{zhu2021complementary}
J.~Zhu, S.~Tang, D.~Chen, S.~Yu, Y.~Liu, M.~Rong, A.~Yang, and X.~Wang, ``Complementary relation contrastive distillation,'' in \emph{CVPR}, 2021.
\bibitem{peng2019correlation}
B.~Peng, X.~Jin, J.~Liu, D.~Li, Y.~Wu, Y.~Liu, S.~Zhou, and Z.~Zhang, ``Correlation congruence for knowledge distillation,'' in \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, 2019, pp. 5007--5016.
\bibitem{geng2018hardware}
X.~Geng, J.~Lin, B.~Zhao, A.~Kong, M.~M.~S. Aly, and V.~Chandrasekhar, ``Hardware-aware softmax approximation for deep neural networks,'' in \emph{ACCV}, 2018.
\bibitem{greedynms}
N.~Dalal and B.~Triggs, ``Histograms of oriented gradients for human detection,'' in \emph{CVPR}, 2005.
\bibitem{bodla2017soft}
N.~Bodla, B.~Singh, R.~Chellappa, and L.~S. Davis, ``Soft-nms--improving object detection with one line of code,'' in \emph{ICCV}, 2017.
\bibitem{tychsen2018improving}
L.~Tychsen-Smith and L.~Petersson, ``Improving object localization with fitness nms and bounded iou loss,'' in \emph{CVPR}, 2018.
\bibitem{jiang2018acquisition}
B.~Jiang, R.~Luo, J.~Mao, T.~Xiao, and Y.~Jiang, ``Acquisition of localization confidence for accurate object detection,'' in \emph{ECCV}, 2018.
\bibitem{gahlert2020visibility}
N.~G{\"a}hlert, N.~Hanselmann, U.~Franke, and J.~Denzler, ``Visibility guided nms: Efficient boosting of amodal object detection in crowded traffic scenes,'' \emph{arXiv preprint arXiv:2006.08547}, 2020.
\bibitem{he2019bounding}
Y.~He, C.~Zhu, J.~Wang, M.~Savvides, and X.~Zhang, ``Bounding box regression with uncertainty for accurate object detection,'' in \emph{CVPR}, 2019.
\bibitem{liu2019adaptive}
S.~Liu, D.~Huang, and Y.~Wang, ``Adaptive nms: Refining pedestrian detection in a crowd,'' in \emph{CVPR}, 2019.
\bibitem{salscheider2021featurenms}
N.~O. Salscheider, ``Featurenms: Non-maximum suppression by learning feature embeddings,'' in \emph{ICPR}, 2021.
\bibitem{hosang2016convnet}
J.~Hosang, R.~Benenson, and B.~Schiele, ``A convnet for non-maximum suppression,'' in \emph{German Conference on Pattern Recognition}, 2016.
\bibitem{hosang2017learning}
{J. Hosang}, {R. Benenson}, and {B. Schiele}, ``Learning non-maximum suppression,'' in \emph{CVPR}, 2017.
\bibitem{hu2018relation}
H.~Hu, J.~Gu, Z.~Zhang, J.~Dai, and Y.~Wei, ``Relation networks for object detection,'' in \emph{CVPR}, 2018.
\bibitem{rothe2014non}
R.~Rothe, M.~Guillaumin, and L.~V. Gool, ``Non-maximum suppression for object detection by passing messages between windows,'' in \emph{ACCV}, 2014.
\bibitem{gpunms2}
D.~Oro \emph{et~al.}, ``{Work-Efficient Parallel Non-Maximum Suppression Kernels},'' \emph{The Computer Journal}, 08 2020.
\bibitem{shapoolnms}
C.~Chen, T.~Zhang, Z.~Yu, A.~Raghuraman, S.~Udayan, J.~Lin, and M.~M.~S. Aly, ``Scalable hardware acceleration of non-maximum suppression,'' in \emph{DATE}, 2022.
\bibitem{maxpoolnms}
L.~Cai \emph{et~al.}, ``Maxpoolnms: getting rid of nms bottlenecks in two-stage object detectors,'' in \emph{CVPR}, 2019.
\bibitem{psrrmaxpool}
T.~Zhang \emph{et~al.}, ``Psrr-maxpoolnms: Pyramid shifted maxpoolnms with relationship recovery,'' in \emph{CVPR}, 2021.
\bibitem{joulin2017efficient}
A.~Joulin, M.~Ciss{\'e}, D.~Grangier, H.~J{\'e}gou \emph{et~al.}, ``Efficient softmax approximation for gpus,'' in \emph{ICML}, 2017.
\bibitem{shim2017svd}
K.~Shim, M.~Lee, I.~Choi, Y.~Boo, and W.~Sung, ``Svd-softmax: Fast softmax approximation on large vocabulary neural networks,'' in \emph{NIPS}, 2017.
\bibitem{blanc2018adaptive}
G.~Blanc and S.~Rendle, ``Adaptive sampled softmax with kernel based sampling,'' in \emph{ICML}, 2018.
\bibitem{lu2021soft}
J.~Lu, J.~Yao, J.~Zhang, X.~Zhu, H.~Xu, W.~Gao, C.~Xu, T.~Xiang, and L.~Zhang, ``Soft: Softmax-free transformer with linear complexity,'' in \emph{NIPS}, 2021.
\bibitem{zhao2021ann}
K.~Zhao, L.~Song, Y.~Zhang, P.~Pan, Y.~Xu, and R.~Jin, ``Ann softmax: acceleration of extreme classification training,'' \emph{Proceedings of the VLDB Endowment}, 2021.
\bibitem{chen2018learning}
P.~H. Chen, S.~Si, S.~Kumar, Y.~Li, and C.-J. Hsieh, ``Learning to screen for fast softmax inference on large vocabulary neural networks,'' \emph{arXiv preprint arXiv:1810.12406}, 2018.
\bibitem{liao2019doubly}
S.~Liao, T.~Chen, T.~Lin, D.~Zhou, and C.~Wang, ``Doubly sparse: Sparse mixture of sparse experts for efficient softmax inference,'' \emph{arXiv preprint arXiv:1901.10668}, 2019.
\bibitem{rawat2019sampled}
A.~S. Rawat, J.~Chen, F.~X.~X. Yu, A.~T. Suresh, and S.~Kumar, ``Sampled softmax with random fourier features,'' in \emph{NIPS}, 2019.
\bibitem{andreas2015accuracy}
J.~Andreas, M.~Rabinovich, M.~I. Jordan, and D.~Klein, ``On the accuracy of self-normalized log-linear models,'' \emph{NIPS}, 2015.
\bibitem{krizhevsky2012imagenet}
A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton, ``Imagenet classification with deep convolutional neural networks,'' \emph{NIPS}, 2012.
\bibitem{he2015delving}
K.~He, X.~Zhang, S.~Ren, and J.~Sun, ``Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,'' in \emph{ICCV}, 2015.
\bibitem{tan2019efficientnet}
M.~Tan and Q.~Le, ``Efficientnet: Rethinking model scaling for convolutional neural networks,'' in \emph{ICML}, 2019.
\bibitem{cai2018proxylessnas}
H.~Cai, L.~Zhu, and S.~Han, ``Proxylessnas: Direct neural architecture search on target task and hardware,'' \emph{arXiv preprint arXiv:1812.00332}, 2018.
\bibitem{tan2019mnasnet}
M.~Tan, B.~Chen, R.~Pang, V.~Vasudevan, M.~Sandler, A.~Howard, and Q.~V. Le, ``Mnasnet: Platform-aware neural architecture search for mobile,'' in \emph{CVPR}, 2019.
\bibitem{wu2019fbnet}
B.~Wu, X.~Dai, P.~Zhang, Y.~Wang, F.~Sun, Y.~Wu, Y.~Tian, P.~Vajda, Y.~Jia, and K.~Keutzer, ``Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,'' in \emph{CVPR}, 2019.
\bibitem{liu2018darts}
H.~Liu, K.~Simonyan, and Y.~Yang, ``Darts: Differentiable architecture search,'' \emph{ICLR}, 2019.
\bibitem{pham2018efficient}
H.~Pham, M.~Guan, B.~Zoph, Q.~Le, and J.~Dean, ``Efficient neural architecture search via parameters sharing,'' in \emph{ICML}, 2018.
\bibitem{ancilotto2023xinet}
A.~Ancilotto, F.~Paissan, and E.~Farella, ``Xinet: Efficient neural networks for tinyml,'' in \emph{ICCV}, 2023.
\bibitem{lin2021mcunetv2}
J.~Lin, W.-M. Chen, H.~Cai, C.~Gan, and S.~Han, ``Mcunetv2: Memory-efficient patch-based inference for tiny deep learning,'' \emph{arXiv preprint arXiv:2110.15352}, 2021.
\bibitem{cai2020tinytl}
H.~Cai, C.~Gan, L.~Zhu, and S.~Han, ``Tinytl: Reduce memory, not parameters for efficient on-device learning,'' \emph{NIPS}, 2020.
\bibitem{chen2014diannao}
T.~Chen, Z.~Du, N.~Sun, J.~Wang, C.~Wu, Y.~Chen, and O.~Temam, ``Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning,'' \emph{ACM SIGARCH Computer Architecture News}, 2014.
\bibitem{chen2014dadiannao}
Y.~Chen, T.~Luo, S.~Liu, S.~Zhang, L.~He, J.~Wang, L.~Li, T.~Chen, Z.~Xu, N.~Sun \emph{et~al.}, ``Dadiannao: A machine-learning supercomputer,'' in \emph{MICRO}, 2014.
\bibitem{liu2015pudiannao}
D.~Liu, T.~Chen, S.~Liu, J.~Zhou, S.~Zhou, O.~Teman, X.~Feng, X.~Zhou, and Y.~Chen, ``Pudiannao: A polyvalent machine learning accelerator,'' \emph{ACM SIGARCH Computer Architecture News}, 2015.
\bibitem{nvdla}
\BIBentryALTinterwordspacing
{Nvidia}. (2018) {NVIDIA Deep Learning Accelerator}. [Online]. Available: \url{http://nvdla.org/primer.html}
\BIBentrySTDinterwordspacing
\bibitem{kwon2018maeri}
H.~Kwon, A.~Samajdar, and T.~Krishna, ``Maeri: Enabling flexible dataflow mapping over dnn accelerators via reconfigurable interconnects,'' \emph{ACM SIGPLAN Notices}, 2018.
\bibitem{strassen1969gaussian}
V.~Strassen, ``Gaussian elimination is not optimal,'' \emph{Numerische mathematik}, 1969.
\bibitem{winograd1980arithmetic}
S.~Winograd, \emph{Arithmetic complexity of computations}.\hskip 1em plus 0.5em minus 0.4em\relax Siam, 1980.
\bibitem{brigham1967fast}
E.~O. Brigham and R.~Morrow, ``The fast fourier transform,'' \emph{IEEE spectrum}, 1967.
\bibitem{cong2014minimizing}
J.~Cong and B.~Xiao, ``Minimizing computation in convolutional neural networks,'' in \emph{ICANN}, 2014.
\bibitem{lavin2016fast}
A.~Lavin and S.~Gray, ``Fast algorithms for convolutional neural networks,'' in \emph{CVPR}, 2016.
\bibitem{ben1997fast}
S.~Ben-Yacoub, ``Fast object detection using mlp and fft,'' IDIAP, Tech. Rep., 1997.
\bibitem{mathieu2013fast}
M.~Mathieu, M.~Henaff, and Y.~LeCun, ``Fast training of convolutional networks through ffts,'' \emph{arXiv preprint arXiv:1312.5851}, 2013.
\bibitem{vasilache2014fast}
N.~Vasilache, J.~Johnson, M.~Mathieu, S.~Chintala, S.~Piantino, and Y.~LeCun, ``Fast convolutional nets with fbfft: A gpu performance evaluation,'' \emph{arXiv preprint arXiv:1412.7580}, 2014.
\bibitem{liang2019evaluating}
Y.~Liang, L.~Lu, Q.~Xiao, and S.~Yan, ``Evaluating fast algorithms for convolutional neural networks on fpgas,'' \emph{IEEE TCAD}, 2019.
\bibitem{lu2020hardware}
S.~Lu, M.~Wang, S.~Liang, J.~Lin, and Z.~Wang, ``Hardware accelerator for multi-head attention and position-wise feed-forward in the transformer,'' in \emph{SOCC}, 2020.
\bibitem{ham20203}
T.~J. Ham, S.~J. Jung, S.~Kim, Y.~H. Oh, Y.~Park, Y.~Song, J.-H. Park, S.~Lee, K.~Park, J.~W. Lee \emph{et~al.}, ``A\^{} 3: Accelerating attention mechanisms in neural networks with approximation,'' in \emph{HPCA}, 2020.
\bibitem{hu2021vis}
W.~Hu, D.~Xu, Z.~Fan, F.~Liu, and Y.~He, ``Vis-top: Visual transformer overlay processor,'' \emph{arXiv preprint arXiv:2110.10957}, 2021.
\bibitem{sun2022vaqf}
M.~Sun, H.~Ma, G.~Kang, Y.~Jiang, T.~Chen, X.~Ma, Z.~Wang, and Y.~Wang, ``Vaqf: Fully automatic software-hardware co-design framework for low-bit vision transformer,'' \emph{arXiv preprint arXiv:2201.06618}, 2022.
\bibitem{yu2022nn}
J.~Yu, J.~Park, S.~Park, M.~Kim, S.~Lee, D.~H. Lee, and J.~Choi, ``Nn-lut: neural approximation of non-linear operations for efficient transformer inference,'' in \emph{DAC}, 2022.
\bibitem{marchisio2023swifttron}
A.~Marchisio, D.~Dura, M.~Capra, M.~Martina, G.~Masera, and M.~Shafique, ``Swifttron: An efficient hardware accelerator for quantized transformers,'' \emph{arXiv preprint arXiv:2304.03986}, 2023.
\bibitem{nag2023vita}
S.~Nag, G.~Datta, S.~Kundu, N.~Chandrachoodan, and P.~A. Beerel, ``Vita: A vision transformer inference accelerator for edge applications,'' \emph{arXiv preprint arXiv:2302.09108}, 2023.
\bibitem{yin20171}
S.~Yin, P.~Ouyang, S.~Tang, F.~Tu, X.~Li, L.~Liu, and S.~Wei, ``A 1.06-to-5.09 tops/w reconfigurable hybrid-neural-network processor for deep learning applications,'' in \emph{Symposium on VLSI Circuits}, 2017.
\bibitem{zhang2019new}
H.~Zhang, D.~Chen, and S.-B. Ko, ``New flexible multiple-precision multiply-accumulate unit for deep neural network training and inference,'' \emph{IEEE Transactions on Computers}, 2019.
\bibitem{lee20197}
J.~Lee, J.~Lee, D.~Han, J.~Lee, G.~Park, and H.-J. Yoo, ``7.7 lnpu: A 25.3 tflops/w sparse deep-neural-network learning processor with fine-grained mixed precision of fp8-fp16,'' in \emph{ISSCC}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019.
\bibitem{zhou2020convolutional}
X.~Zhou, L.~Zhang, C.~Guo, X.~Yin, and C.~Zhuo, ``A convolutional neural network accelerator architecture with fine-granular mixed precision configurability,'' in \emph{ISCAS}, 2020.
\bibitem{fleischer2018scalable}
B.~Fleischer, S.~Shukla, M.~Ziegler, J.~Silberman, J.~Oh, V.~Srinivasan, J.~Choi, S.~Mueller, A.~Agrawal, T.~Babinsky \emph{et~al.}, ``A scalable multi-teraops deep learning processor core for ai trainina and inference,'' in \emph{IEEE Symposium on VLSI Circuits}, 2018.
\bibitem{wang2018design}
J.~Wang, Q.~Lou, X.~Zhang, C.~Zhu, Y.~Lin, and D.~Chen, ``Design flow of accelerating hybrid extremely low bit-width neural network in embedded fpga,'' in \emph{FPL}, 2018.
\bibitem{agrawal20219}
A.~Agrawal, S.~K. Lee, J.~Silberman, M.~Ziegler, M.~Kang, S.~Venkataramani, N.~Cao, B.~Fleischer, M.~Guillorn, M.~Cohen \emph{et~al.}, ``9.1 a 7nm 4-core ai chip with 25.6 tflops hybrid fp8 training, 102.4 tops int4 inference and workload-aware throttling,'' in \emph{ISSCC}, 2021.
\bibitem{recanatesi2019dimensionality}
S.~Recanatesi \emph{et~al.}, ``Dimensionality compression and expansion in deep neural networks,'' \emph{arXiv preprint arXiv:1906.00443}, 2019.
\bibitem{oktay2019scalable}
D.~Oktay \emph{et~al.}, ``Scalable model compression by entropy penalized reparameterization,'' \emph{arXiv preprint arXiv:1906.06624}, 2019.
\bibitem{wiedemann2019deepcabac}
S.~Wiedemann \emph{et~al.}, ``Deepcabac: Context-adaptive binary arithmetic coding for deep neural network compression,'' \emph{arXiv preprint arXiv:1905.08318}, 2019.
\bibitem{dubey2018coreset}
A.~Dubey \emph{et~al.}, ``Coreset-based neural network compression,'' in \emph{ECCV}, 2018.
\bibitem{han2015deep}
S.~Han, H.~Mao, and W.~J. Dally, ``Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,'' in \emph{ICLR}, 2016.
\bibitem{chen2021efficient}
C.~Chen, Z.~Wang, X.~Chen, J.~Lin, and M.~M.~S. Aly, ``Efficient tunstall decoder for deep neural network compression,'' in \emph{DAC}, 2021.
\bibitem{cormen2009introduction}
T.~H. Cormen, C.~E. Leiserson, R.~L. Rivest, and C.~Stein, \emph{Introduction to algorithms}.\hskip 1em plus 0.5em minus 0.4em\relax MIT press, 2009.
\bibitem{rabbani2002jpeg2000}
M.~Rabbani, ``Jpeg2000: Image compression fundamentals, standards and practice,'' \emph{Journal of Electronic Imaging}, 2002.
\bibitem{witten1987arithmetic}
I.~H. Witten \emph{et~al.}, ``Arithmetic coding for data compression,'' \emph{Communications of the ACM}, 1987.
\bibitem{chen2016eyeriss}
Y.-H. Chen, T.~Krishna, J.~S. Emer, and V.~Sze, ``Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks,'' \emph{IEEE Journal of Solid-State Circuits}, 2016.
\bibitem{han2016eie}
S.~Han, X.~Liu, H.~Mao, J.~Pu, A.~Pedram, M.~A. Horowitz, and W.~J. Dally, ``Eie: efficient inference engine on compressed deep neural network,'' in \emph{ISCA}, 2016.
\bibitem{parashar2017scnn}
A.~Parashar, M.~Rhu, A.~Mukkara, A.~Puglielli, R.~Venkatesan, B.~Khailany, J.~Emer, S.~W. Keckler, and W.~J. Dally, ``Scnn: An accelerator for compressed-sparse convolutional neural networks,'' in \emph{ISCA}, 2017.
\bibitem{chen2019eyeriss}
Y.-H. Chen, T.-J. Yang, J.~Emer, and V.~Sze, ``Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices,'' \emph{IEEE Journal on Emerging and Selected Topics in Circuits and Systems}, 2019.
\bibitem{dave2021hardware}
S.~Dave, R.~Baghdadi, T.~Nowatzki, S.~Avancha, A.~Shrivastava, and B.~Li, ``Hardware acceleration of sparse and irregular tensor computations of ml models: A survey and insights,'' \emph{Proceedings of the IEEE}, 2021.
\bibitem{zhang2016cambricon}
S.~Zhang, Z.~Du, L.~Zhang, H.~Lan, S.~Liu, L.~Li, Q.~Guo, T.~Chen, and Y.~Chen, ``Cambricon-x: An accelerator for sparse neural networks,'' in \emph{MICRO}, 2016.
\bibitem{gustavson1972some}
F.~G. Gustavson, ``Some basic techniques for solving sparse systems of linear equations,'' in \emph{Sparse matrices and their applications}, 1972.
\bibitem{smith2015splatt}
S.~Smith, N.~Ravindran, N.~D. Sidiropoulos, and G.~Karypis, ``Splatt: Efficient and parallel sparse tensor-matrix multiplication,'' in \emph{IPDPS}, 2015.
\bibitem{hegde2019extensor}
K.~Hegde, H.~Asghari-Moghaddam, M.~Pellauer, N.~Crago, A.~Jaleel, E.~Solomonik, J.~Emer, and C.~W. Fletcher, ``Extensor: An accelerator for sparse tensor algebra,'' in \emph{MICRO}, 2019.
\bibitem{yuan2018sticker}
Z.~Yuan, J.~Yue, H.~Yang, Z.~Wang, J.~Li, Y.~Yang, Q.~Guo, X.~Li, M.-F. Chang, H.~Yang \emph{et~al.}, ``Sticker: A 0.41-62.1 tops/w 8bit neural network processor with multi-sparsity compatible convolution arrays and online tuning acceleration for fully connected layers,'' in \emph{IEEE symposium on VLSI circuits}, 2018.
\bibitem{zhang2019snap}
J.-F. Zhang, C.-E. Lee, C.~Liu, Y.~S. Shao, S.~W. Keckler, and Z.~Zhang, ``Snap: A 1.67—21.55 tops/w sparse neural acceleration processor for unstructured sparse deep neural network inference in 16nm cmos,'' in \emph{VLSIC}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019.
\bibitem{kang2019accelerator}
H.-J. Kang, ``Accelerator-aware pruning for convolutional neural networks,'' \emph{IEEE TCSVT}, 2019.
\bibitem{lu2019efficient}
L.~Lu, J.~Xie, R.~Huang, J.~Zhang, W.~Lin, and Y.~Liang, ``An efficient hardware accelerator for sparse convolutional neural networks on fpgas,'' in \emph{FCCM}, 2019.
\bibitem{zhang2019compact}
J.~J. Zhang, P.~Raj, S.~Zarar, A.~Ambardekar, and S.~Garg, ``Compact: On-chip compression of activations for low power systolic array based cnn acceleration,'' \emph{TECS}, 2019.
\bibitem{zhou2018cambricon}
X.~Zhou, Z.~Du, Q.~Guo, S.~Liu, C.~Liu, C.~Wang, X.~Zhou, L.~Li, T.~Chen, and Y.~Chen, ``Cambricon-s: Addressing irregularity in sparse neural networks through a cooperative software/hardware approach,'' in \emph{MICRO}, 2018.
\bibitem{mishra2017fine}
A.~K. Mishra, E.~Nurvitadhi, G.~Venkatesh, J.~Pearce, and D.~Marr, ``Fine-grained accelerators for sparse machine learning workloads,'' in \emph{ASP-DAC}, 2017.
\bibitem{han2017ese}
S.~Han, J.~Kang, H.~Mao, Y.~Hu, X.~Li, Y.~Li, D.~Xie, H.~Luo, S.~Yao, Y.~Wang \emph{et~al.}, ``Ese: Efficient speech recognition engine with sparse lstm on fpga,'' in \emph{ACM/SIGDA FPGA}, 2017.
\bibitem{moons201714}
B.~Moons, R.~Uytterhoeven, W.~Dehaene, and M.~Verhelst, ``14.5 envision: A 0.26-to-10tops/w subword-parallel dynamic-voltage-accuracy-frequency-scalable convolutional neural network processor in 28nm fdsoi,'' in \emph{ISSCC}, 2017.
\bibitem{gpunms}
D.~Oro \emph{et~al.}, ``Work-efficient parallel non-maximum suppression for embedded gpu architectures,'' in \emph{ICASSP}, 2016.
\bibitem{powernms}
M.~Shi \emph{et~al.}, ``A fast and power-efficient hardware architecture for non-maximum suppression,'' \emph{TCAS-II}, 2019.
\bibitem{yuan2016efficient}
B.~Yuan, ``Efficient hardware architecture of softmax layer in deep neural network,'' in \emph{SOCC}, 2016.
\bibitem{li2018efficient}
Z.~Li, H.~Li, X.~Jiang, B.~Chen, Y.~Zhang, and G.~Du, ``Efficient fpga implementation of softmax function for dnn applications,'' in \emph{ASID}, 2018.
\bibitem{wang2018high}
M.~Wang, S.~Lu, D.~Zhu, J.~Lin, and Z.~Wang, ``A high-speed and low-complexity architecture for softmax function in deep learning,'' in \emph{APCCAS}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2018.
\bibitem{sun2018high}
Q.~Sun, Z.~Di, Z.~Lv, F.~Song, Q.~Xiang, Q.~Feng, Y.~Fan, X.~Yu, and W.~Wang, ``A high speed softmax vlsi architecture based on basic-split,'' in \emph{ICSICT}, 2018.
\bibitem{hu2018efficient}
R.~Hu, B.~Tian, S.~Yin, and S.~Wei, ``Efficient hardware architecture of softmax layer in deep neural network,'' in \emph{ICDSP}, 2018.
\bibitem{du2019efficient}
G.~Du, C.~Tian, Z.~Li, D.~Zhang, Y.~Yin, and Y.~Ouyang, ``Efficient softmax hardware architecture for deep neural networks,'' in \emph{Proceedings of the 2019 on Great Lakes Symposium on VLSI}, 2019.
\bibitem{kouretas2020hardware}
I.~Kouretas and V.~Paliouras, ``Hardware implementation of a softmax-like function for deep learning,'' \emph{Technologies}, 2020.
\bibitem{wang2019customized}
K.-Y. Wang, Y.-D. Huang, Y.-L. Ho, and W.-C. Fang, ``A customized convolutional neural network design using improved softmax layer for real-time human emotion recognition,'' in \emph{AICAS}, 2019.
\bibitem{cardarilli2021pseudo}
G.~C. Cardarilli, L.~Di~Nunzio, R.~Fazzolari, D.~Giardino, A.~Nannarelli, M.~Re, and S.~Span{\`o}, ``A pseudo-softmax function for hardware-based high speed image classification,'' \emph{Scientific reports}, 2021.
\bibitem{spagnolo2021aggressive}
F.~Spagnolo, S.~Perri, and P.~Corsonello, ``Aggressive approximation of the softmax function for power-efficient hardware implementations,'' \emph{IEEE Transactions on Circuits and Systems II: Express Briefs}, 2021.
\bibitem{stevens2021softermax}
J.~R. Stevens, R.~Venkatesan, S.~Dai, B.~Khailany, and A.~Raghunathan, ``Softermax: Hardware/software co-design of an efficient softmax for transformers,'' in \emph{DAC}, 2021.
\bibitem{gaines1969stochastic}
B.~R. Gaines, ``Stochastic computing systems,'' \emph{Advances in Information Systems Science}, 1969.
\bibitem{liu2020survey}
Y.~Liu, S.~Liu, Y.~Wang, F.~Lombardi, and J.~Han, ``A survey of stochastic computing neural networks for machine learning applications,'' \emph{IEEE TNNLS}, 2020.
\bibitem{salehi2020low}
S.~A. Salehi, ``Low-cost stochastic number generators for stochastic computing,'' \emph{VLSI}, 2020.
\bibitem{frasser2022fully}
C.~F. Frasser, P.~Linares-Serrano, I.~D. de~Los~Rios, A.~Moran, E.~S. Skibinsky-Gitlin, J.~Font-Rossello, V.~Canals, M.~Roca, T.~Serrano-Gotarredona, and J.~L. Rossello, ``Fully parallel stochastic computing hardware implementation of convolutional neural networks for edge computing applications,'' \emph{IEEE TNNLS}, 2022.
\bibitem{moran2022digital}
A.~Mor{\'a}n, L.~Parrilla, M.~Roca, J.~Font-Rossello, E.~Isern, and V.~Canals, ``Digital implementation of radial basis function neural networks based on stochastic computing,'' \emph{IEEE Journal on Emerging and Selected Topics in Circuits and Systems}, 2022.
\bibitem{paillier1999public}
P.~Paillier, ``Public-key cryptosystems based on composite degree residuosity classes,'' in \emph{Eurocrypt}, 1999.
\bibitem{rivest1978method}
R.~L. Rivest, A.~Shamir, and L.~Adleman, ``A method for obtaining digital signatures and public-key cryptosystems,'' \emph{Communications of the ACM}, 1978.
\bibitem{gentry2009fully}
C.~Gentry, ``Fully homomorphic encryption using ideal lattices,'' in \emph{STOC}, 2009.
\bibitem{isakov2019survey}
M.~Isakov, V.~Gadepally, K.~M. Gettings, and M.~A. Kinsy, ``Survey of attacks and defenses on edge-deployed neural networks,'' in \emph{HPEC}, 2019.
\bibitem{yagisawa2015fully}
M.~Yagisawa, ``Fully homomorphic encryption without bootstrapping,'' \emph{Cryptology ePrint Archive}, 2015.
\bibitem{fan2012somewhat}
J.~Fan and F.~Vercauteren, ``Somewhat practical fully homomorphic encryption,'' \emph{Cryptology ePrint Archive}, 2012.
\bibitem{aikata2023reed}
A.~Aikata, A.~C. Mert, S.~Kwon, M.~Deryabin, and S.~S. Roy, ``Reed: Chiplet-based scalable hardware accelerator for fully homomorphic encryption,'' \emph{arXiv preprint arXiv:2308.02885}, 2023.
\bibitem{zhang2019encrypted}
S.-X. Zhang, Y.~Gong, and D.~Yu, ``Encrypted speech recognition using deep polynomial networks,'' in \emph{ICASSP}, 2019.
\bibitem{lou2019she}
Q.~Lou and L.~Jiang, ``She: A fast and accurate deep neural network for encrypted data,'' \emph{NIPS}, 2019.
\bibitem{bourse2018fast}
F.~Bourse, M.~Minelli, M.~Minihold, and P.~Paillier, ``Fast homomorphic evaluation of deep discretized neural networks,'' in \emph{Crypto}, 2018.
\bibitem{pmlr-v80-sanyal18a}
A.~Sanyal, M.~Kusner, A.~Gascon, and V.~Kanade, ``{TAPAS}: Tricks to accelerate (encrypted) prediction as a service,'' in \emph{ICML}, 2018.
\bibitem{chou2018faster}
E.~Chou, J.~Beal, D.~Levy, S.~Yeung, A.~Haque, and L.~Fei-Fei, ``Faster cryptonets: Leveraging sparsity for real-world encrypted inference,'' \emph{arXiv preprint arXiv:1811.09953}, 2018.
\bibitem{lu2021ffconv}
Y.~Lu, J.~Lin, C.~Jin, Z.~Wang, M.~Wu, K.~M.~M. Aung, and X.~Li, ``Ffconv: Fast factorized convolutional neural network inference on encrypted data,'' \emph{arXiv preprint arXiv:2102.03494}, 2021.
\bibitem{khalil2010hardware}
M.~Khalil-Hani, V.~P. Nambiar, and M.~Marsono, ``Hardware acceleration of openssl cryptographic functions for high-performance internet security,'' in \emph{International Conference on Intelligent Systems, Modelling and Simulation}, 2010.
\bibitem{cousins2016designing}
D.~B. Cousins, K.~Rohloff, and D.~Sumorok, ``Designing an fpga-accelerated homomorphic encryption co-processor,'' \emph{IEEE Transactions on Emerging Topics in Computing}, 2016.
\bibitem{lee2020optimizing}
D.~Lee, W.~Lee, H.~Oh, and K.~Yi, ``Optimizing homomorphic evaluation circuits by program synthesis and term rewriting,'' in \emph{PLDI}, 2020.
\bibitem{kim2023sharp}
J.~Kim, S.~Kim, J.~Choi, J.~Park, D.~Kim, and J.~H. Ahn, ``Sharp: A short-word hierarchical accelerator for robust and practical fully homomorphic encryption,'' in \emph{ISCA}, 2023.
\bibitem{kim2022bts}
S.~Kim, J.~Kim, M.~J. Kim, W.~Jung, J.~Kim, M.~Rhu, and J.~H. Ahn, ``Bts: An accelerator for bootstrappable fully homomorphic encryption,'' in \emph{ISCA}, 2022.
\bibitem{doroz2014accelerating}
Y.~Dor{\"o}z, E.~{\"O}zt{\"u}rk, and B.~Sunar, ``Accelerating fully homomorphic encryption in hardware,'' \emph{IEEE Transactions on Computers}, 2014.
\bibitem{reagen2021cheetah}
B.~Reagen, W.-S. Choi, Y.~Ko, V.~T. Lee, H.-H.~S. Lee, G.-Y. Wei, and D.~Brooks, ``Cheetah: Optimizing and accelerating homomorphic encryption for private inference,'' in \emph{HPCA}, 2021.
\bibitem{chillotti2020tfhe}
I.~Chillotti, N.~Gama, M.~Georgieva, and M.~Izabach{\`e}ne, ``Tfhe: fast fully homomorphic encryption over the torus,'' \emph{Journal of Cryptology}, 2020.
\bibitem{cheon2017homomorphic}
J.~H. Cheon, A.~Kim, M.~Kim, and Y.~Song, ``Homomorphic encryption for arithmetic of approximate numbers,'' in \emph{ASIACRYPT}, 2017.
\bibitem{brown2020language}
T.~Brown, B.~Mann, N.~Ryder, M.~Subbiah, J.~D. Kaplan, P.~Dhariwal, A.~Neelakantan, P.~Shyam, G.~Sastry, A.~Askell \emph{et~al.}, ``Language models are few-shot learners,'' \emph{NIPS}, 2020.
\bibitem{touvron2023llama}
H.~Touvron, T.~Lavril, G.~Izacard, X.~Martinet, M.-A. Lachaux, T.~Lacroix, B.~Rozi{\`e}re, N.~Goyal, E.~Hambro, F.~Azhar \emph{et~al.}, ``Llama: Open and efficient foundation language models,'' \emph{arXiv preprint arXiv:2302.13971}, 2023.
\bibitem{yao2022zeroquant}
Z.~Yao, R.~Yazdani~Aminabadi, M.~Zhang, X.~Wu, C.~Li, and Y.~He, ``Zeroquant: Efficient and affordable post-training quantization for large-scale transformers,'' \emph{NIPS}, 2022.
\bibitem{xu2022etinynet}
K.~Xu, Y.~Li, H.~Zhang, R.~Lai, and L.~Gu, ``Etinynet: Extremely tiny network for tinyml,'' in \emph{AAAI}, 2022.
\bibitem{yang2020mobileda}
J.~Yang, H.~Zou, S.~Cao, Z.~Chen, and L.~Xie, ``Mobileda: Toward edge-domain adaptation,'' \emph{IEEE Internet of Things Journal}, 2020.
\bibitem{ryu2022knowledge}
M.~Ryu, G.~Lee, and K.~Lee, ``Knowledge distillation for bert unsupervised domain adaptation,'' \emph{Knowledge and Information Systems}, 2022.
\end{thebibliography}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{liu2017survey}
W.~Liu, Z.~Wang, X.~Liu, N.~Zeng, Y.~Liu, and F.~E. Alsaadi, ``A survey of deep neural network architectures and their applications,'' \emph{Neurocomputing}, 2017.
\bibitem{thakkar2021comprehensive}
A.~Thakkar and K.~Chaudhari, ``A comprehensive survey on deep neural networks for stock market: The need, challenges, and future directions,'' \emph{Expert Systems with Applications}, 2021.
\bibitem{Krizhevsky2012nips}
A.~Krizhevsky, I.~Sutskever, and G.~Hinton, ``Image{N}et classification with deep convolutional neural networks,'' in \emph{NIPS}, 2012.
\bibitem{Szegedy2015CVPR}
C.~Szegedy, W.~Liu, Y.~Jia, P.~Sermanet, S.~Anguelov, D.~Erhan, V.~Vanhoucke, and A.~Rabinovich, ``Going deeper with convolutions,'' in \emph{CVPR}, 2015.
\bibitem{Simonyan2015ICLR}
K.~Simonyan and A.~Zisserman, ``Very deep convolutional networks for large-scale image recognition,'' in \emph{ICLR}, 2015.
\bibitem{he2016CVPR}
K.~He, X.~Zhang, S.~Ren, and J.~Sun, ``Deep residual learning for image recognition,'' in \emph{CVPR}, 2016.
\bibitem{i2016squeezenet}
F.~N. Iandola, S.~Han, M.~W. Moskewicz, K.~Ashraf, W.~J. Dally, and K.~Keutzer, ``Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size,'' \emph{arXiv preprint arXiv:1602.07360}, 2016.
\bibitem{huang2017densely}
G.~Huang, Z.~Liu, L.~Van Der~Maaten, and K.~Q. Weinberger, ``Densely connected convolutional networks,'' in \emph{CVPR}, 2017.
\bibitem{Sandler2018arXiv}
M.~Sandler, A.~Howard, M.~Zhu, A.~Zhmoginov, and L.-C. Chen, ``Mobilenetv2: Inverted residuals and linear bottlenecks,'' in \emph{arXiv}, 2018.
\bibitem{vaswani2017attention}
A.~Vaswani, N.~Shazeer, N.~Parmar, J.~Uszkoreit, L.~Jones, A.~N. Gomez, {\L}.~Kaiser, and I.~Polosukhin, ``Attention is all you need,'' \emph{NIPS}, 2017.
\bibitem{brown2020language}
T.~Brown, B.~Mann, N.~Ryder, M.~Subbiah, J.~D. Kaplan, P.~Dhariwal, A.~Neelakantan, P.~Shyam, G.~Sastry, A.~Askell \emph{et~al.}, ``Language models are few-shot learners,'' \emph{NIPS}, 2020.
\bibitem{bwn}
I.~Hubara, M.~Courbariaux, D.~Soudry, R.~El-Yaniv, and Y.~Bengio, ``Quantized neural networks: Training neural networks with low precision weights and activations,'' in \emph{arXiv:1609.07061}, 2016.
\bibitem{xnor}
M.~Rastegari, V.~Ordonez, J.~Redmon, and A.~Farhadi, ``{XNOR}-{NET}: {I}mage{n}et classification using binary convolutional neural networks,'' in \emph{arXiv preprint arXiv:1603.05279}, 2016.
\bibitem{Li16nips}
F.~Li, B.~Zhang, and B.~Liu, ``Ternary weight networks,'' in \emph{NIPS Workshop}, 2016.
\bibitem{zhu2016trained}
C.~Zhu, S.~Han, H.~Mao, and W.~J. Dally, ``Trained ternary quantization,'' \emph{arXiv preprint arXiv:1612.01064}, 2016.
\bibitem{Zhang18eccv}
D.~Zhang, J.~Yang, D.~Ye, and G.~Hua, ``Lq-nets: learned quantization for highly accurate and compact deep neural networks,'' in \emph{ECCV}, 2018.
\bibitem{inq}
A.~Zhou, A.~Yao, Y.~Guo, L.~Xu, and Y.~Chen, ``Incremental network quantization: {T}owards lossless {cnn}s with low-precision weights,'' in \emph{arXiv preprint arXiv:1702.03044}, 2017.
\bibitem{dorefa}
S.~Zhou, Y.~Wu, Z.~Ni, X.~Zhou, H.~Wen, and Y.~Zou, ``Dorefa-net: {T}raining low bitwidth convolutional neural networks with low bitwidth gradients,'' in \emph{arXiv preprint arXiv:1606.06160}, 2016.
\bibitem{datafree}
M.~Nagel, M.~Baalen, T.~Blankevoort, and M.~Welling, ``Datafree quantization through weight equalization and bias correction,'' \emph{CVPR}, 2019.
\bibitem{DSQ}
R.~Gong, X.~Liu, S.~Jiang, T.~Li, P.~Hu, J.~Lin, F.~Yu, and J.~Yan, ``Differentiable soft quantization: Bridging full-precision and low-bit neu- ral networks,'' in \emph{ICCV}, 2019.
\bibitem{layerwise}
S.~Chen, W.~Wang, and S.~J. Pan, ``Deep neural network quantization via layer-wise optimization using limited training data,'' in \emph{AAAI}, 2019.
\bibitem{Zhuang2018cvpr}
B.~Zhuang, C.~Shen, M.~Tan, L.~Liu, and I.~Reid, ``Towards effective low-bitwidth convolutional neural networks,'' in \emph{CVPR}, 2018.
\bibitem{QIL}
S.~Jung, C.~Son, S.~Lee, J.~Son, J.-J. Han, Y.~Kwak, S.~J. Hwang, and C.~Choi, ``Learning to quantize deep networks by optimizing quantization intervals with task loss,'' in \emph{CVPR}, 2019.
\bibitem{SAT}
Q.~Jin, L.~Yang, Z.~Liao, and X.~Qian, ``Neural network quantization with scale-adjusted training,'' \emph{BMVC}, 2020.
\bibitem{esser2019learned}
S.~K. Esser, J.~L. McKinstry, D.~Bablani, R.~Appuswamy, and D.~S. Modha, ``Learned step size quantization,'' \emph{arXiv preprint arXiv:1902.08153}, 2019.
\bibitem{Auxi}
B.~Zhuang, L.~Liu, M.~Tan, C.~Shen, and I.~Reid, ``Training quantized neural networks with a full-precision auxiliary module,'' in \emph{CVPR}, 2020.
\bibitem{pq}
Y.~Gong, L.~Liu, M.~Yang, and L.~Bourdev, ``Compressing deep convolutional networks using vector quantization,'' in \emph{ICLR}, 2015.
\bibitem{hash}
W.~Chen, J.~T. Wilson, S.~Tyree, K.~Q. Weinberger, and Y.~Chen, ``Compressing neural networks with the hashing trick,'' in \emph{ICML}, 2015.
\bibitem{frequency}
W.~Chen, J.~Wilson, S.~Tyree, K.~Q. Weinberger, and Y.~Chen, ``Compressing convolutional neural networks in the frequency domain,'' in \emph{SIGKDD}, 2016.
\bibitem{revisit}
P.~Stock, A.~Joulin, R.~Gribonval, B.~Graham, and H.~Jegou, ``And the bit goes down: revisiting the quantization of neural networks,'' in \emph{ICLR}, 2020.
\bibitem{APoT}
Y.~Li, X.~Dong, and W.~Wang, ``Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks,'' in \emph{ICLR}, 2020.
\bibitem{Faraone2018arxiv}
J.~Faraone, N.~Fraser, M.~Blott, and P.~H. Leong, ``Syq: Learning symmetric quantization for efficient deep neural networks,'' in \emph{arXiv}, 2018.
\bibitem{hinton2015distilling}
G.~Hinton, O.~Vinyals, J.~Dean \emph{et~al.}, ``Distilling the knowledge in a neural network,'' \emph{arXiv preprint arXiv:1503.02531}, 2015.
\bibitem{romero2014fitnets}
A.~Romero, N.~Ballas, S.~E. Kahou, A.~Chassang, C.~Gatta, and Y.~Bengio, ``Fitnets: Hints for thin deep nets,'' \emph{arXiv preprint arXiv:1412.6550}, 2014.
\bibitem{chen2022knowledge}
D.~Chen, J.-P. Mei, H.~Zhang, C.~Wang, Y.~Feng, and C.~Chen, ``Knowledge distillation with the reused teacher classifier,'' in \emph{CVPR}, 2022.
\bibitem{wang2017model}
C.~Wang, X.~Lan, and Y.~Zhang, ``Model distillation with knowledge transfer from face classification to alignment and verification,'' \emph{arXiv preprint arXiv:1709.02929}, 2017.
\bibitem{wang2019distilling}
T.~Wang, L.~Yuan, X.~Zhang, and J.~Feng, ``Distilling object detectors with fine-grained feature imitation,'' in \emph{CVPR}, 2019.
\bibitem{xu2021kdnet}
Q.~Xu, Z.~Chen, K.~Wu, C.~Wang, M.~Wu, and X.~Li, ``Kdnet-rul: A knowledge distillation framework to compress deep neural networks for machine remaining useful life prediction,'' \emph{IEEE Transactions on Industrial Electronics}, 2021.
\bibitem{xu2022contrastive}
Q.~Xu, Z.~Chen, M.~Ragab, C.~Wang, M.~Wu, and X.~Li, ``Contrastive adversarial knowledge distillation for deep model compression in time-series regression tasks,'' \emph{Neurocomputing}, 2022.
\bibitem{xu2017training}
Z.~Xu, Y.-C. Hsu, and J.~Huang, ``Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks,'' in \emph{ICLR Workshop}, 2018.
\bibitem{gao2019adversarial}
L.~Gao, H.~Mi, B.~Zhu, D.~Feng, Y.~Li, and Y.~Peng, ``An adversarial feature distillation method for audio classification,'' \emph{IEEE Access}, 2019.
\bibitem{chung2020feature}
I.~Chung, S.~Park, J.~Kim, and N.~Kwak, ``Feature-map-level online adversarial knowledge distillation,'' in \emph{ICML}, 2020.
\bibitem{park2020feature}
S.~Park and N.~Kwak, ``Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks,'' in \emph{ECAI 2020}, 2020.
\bibitem{liu2021semantics}
Y.~Liu, K.~Wang, G.~Li, and L.~Lin, ``Semantics-aware adaptive knowledge distillation for sensor-to-vision action recognition,'' \emph{IEEE TIP}, 2021.
\bibitem{chebotar2016distilling}
Y.~Chebotar and A.~Waters, ``Distilling knowledge from ensembles of neural networks for speech recognition.'' in \emph{Interspeech}, 2016.
\bibitem{wu2019multi}
M.-C. Wu, C.-T. Chiu, and K.-H. Wu, ``Multi-teacher knowledge distillation for compressed video action recognition on deep neural networks,'' in \emph{ICASSP}, 2019.
\bibitem{fukuda2017efficient}
T.~Fukuda, M.~Suzuki, G.~Kurata, S.~Thomas, J.~Cui, and B.~Ramabhadran, ``Efficient knowledge distillation from an ensemble of teachers.'' in \emph{Interspeech}, 2017.
\bibitem{yuan2021reinforced}
F.~Yuan, L.~Shou, J.~Pei, W.~Lin, M.~Gong, Y.~Fu, and D.~Jiang, ``Reinforced multi-teacher selection for knowledge distillation,'' in \emph{AAAI}, 2021.
\bibitem{furlanello2018born}
T.~Furlanello, Z.~Lipton, M.~Tschannen, L.~Itti, and A.~Anandkumar, ``Born again neural networks,'' in \emph{ICML}, 2018.
\bibitem{tian2020rethinking}
Y.~Tian, Y.~Wang, D.~Krishnan, J.~B. Tenenbaum, and P.~Isola, ``Rethinking few-shot image classification: a good embedding is all you need?'' in \emph{ECCV}, 2020.
\bibitem{zhang2018deep}
Y.~Zhang, T.~Xiang, T.~M. Hospedales, and H.~Lu, ``Deep mutual learning,'' in \emph{CVPR}, 2018.
\bibitem{ji2021refine}
M.~Ji, S.~Shin, S.~Hwang, G.~Park, and I.-C. Moon, ``Refine myself by teaching myself: Feature refinement via self-knowledge distillation,'' in \emph{CVPR}, 2021.
\bibitem{lopes2017data}
R.~G. Lopes, S.~Fenu, and T.~Starner, ``Data-free knowledge distillation for deep neural networks,'' \emph{NIPS LLD Workshop}, 2017.
\bibitem{yin2020dreaming}
H.~Yin, P.~Molchanov, J.~M. Alvarez, Z.~Li, A.~Mallya, D.~Hoiem, N.~K. Jha, and J.~Kautz, ``Dreaming to distill: Data-free knowledge transfer via deepinversion,'' in \emph{CVPR}, 2020.
\bibitem{chen2019data}
H.~Chen, Y.~Wang, C.~Xu, Z.~Yang, C.~Liu, B.~Shi, C.~Xu, C.~Xu, and Q.~Tian, ``Data-free learning of student networks,'' in \emph{ICCV}, 2019.
\bibitem{zhang2021data}
Y.~Zhang, H.~Chen, X.~Chen, Y.~Deng, C.~Xu, and Y.~Wang, ``Data-free knowledge distillation for image super-resolution,'' in \emph{CVPR}, 2021.
\bibitem{nayak2019zero}
G.~K. Nayak, K.~R. Mopuri, V.~Shaj, V.~B. Radhakrishnan, and A.~Chakraborty, ``Zero-shot knowledge distillation in deep networks,'' in \emph{ICML}, 2019.
\bibitem{ma2021undistillable}
H.~Ma and T.~Chen, ``Undistillable: Making a nasty teacher that cannot teach students,'' in \emph{ICLR}, 2021.
\bibitem{jandial2022distilling}
S.~Jandial, Y.~Khasbage, A.~Pal, V.~N. Balasubramanian, and B.~Krishnamurthy, ``Distilling the undistillable: Learning from a nasty teacher,'' in \emph{ECCV}, 2022.
\bibitem{horowitz20141}
M.~Horowitz, ``1.1 computing's energy problem (and what we can do about it),'' in \emph{ISSCC}, 2014.
\bibitem{vivienne2019efficient}
V.~Sze, ``Efficient processing of deep neural networks: from algorithms to hardware architectures,'' in \emph{NIPS}, 2019.
\bibitem{chen2021efficient}
C.~Chen, Z.~Wang, X.~Chen, J.~Lin, and M.~M.~S. Aly, ``Efficient tunstall decoder for deep neural network compression,'' in \emph{DAC}, 2021.
\bibitem{chen2019eyeriss}
Y.-H. Chen, T.-J. Yang, J.~Emer, and V.~Sze, ``Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices,'' \emph{IEEE Journal on Emerging and Selected Topics in Circuits and Systems}, 2019.
\bibitem{zhang2019snap}
J.-F. Zhang, C.-E. Lee, C.~Liu, Y.~S. Shao, S.~W. Keckler, and Z.~Zhang, ``Snap: A 1.67—21.55 tops/w sparse neural acceleration processor for unstructured sparse deep neural network inference in 16nm cmos,'' in \emph{VLSIC}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019.
\bibitem{zhou2018cambricon}
X.~Zhou, Z.~Du, Q.~Guo, S.~Liu, C.~Liu, C.~Wang, X.~Zhou, L.~Li, T.~Chen, and Y.~Chen, ``Cambricon-s: Addressing irregularity in sparse neural networks through a cooperative software/hardware approach,'' in \emph{MICRO}, 2018.
\bibitem{psrrmaxpool}
T.~Zhang \emph{et~al.}, ``Psrr-maxpoolnms: Pyramid shifted maxpoolnms with relationship recovery,'' in \emph{CVPR}, 2021.
\bibitem{gpunms2}
D.~Oro \emph{et~al.}, ``{Work-Efficient Parallel Non-Maximum Suppression Kernels},'' \emph{The Computer Journal}, 08 2020.
\bibitem{powernms}
M.~Shi \emph{et~al.}, ``A fast and power-efficient hardware architecture for non-maximum suppression,'' \emph{TCAS-II}, 2019.
\bibitem{shapoolnms}
C.~Chen, T.~Zhang, Z.~Yu, A.~Raghuraman, S.~Udayan, J.~Lin, and M.~M.~S. Aly, ``Scalable hardware acceleration of non-maximum suppression,'' in \emph{DATE}, 2022.
\end{thebibliography}