DeepShift: Towards Multiplication-Less Neural Networks
Extracted source text · arXiv 1905.13298
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage{cvpr}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
% Include other packages here, before hyperref.
\usepackage{subcaption}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{siunitx}
\usepackage{dsfont}
\newcommand{\blap}[1]{{\begin{tabular}[t]{@{}l@{}}#1\end{tabular}}}
\usepackage{authblk}
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}
\cvprfinalcopy % *** Uncomment this line for the final submission
\def\cvprPaperID{4524} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
% Pages are numbered in submission mode, and unnumbered in camera-ready
\ifcvprfinal\pagestyle{empty}\fi
\begin{document}
%%%%%%%%% TITLE
\title{DeepShift: Towards Multiplication-Less Neural Networks}
\author[1]{Mostafa Elhoushi}
\author[1]{Zihao Chen}
\author[1]{Farhan Shafiq}
\author[1]{Ye Henry Tian}
\author[1]{Joey Yiwei Li}
\affil[1]{Huawei Technologies \\
Toronto Hetrogeneous Compilers Lab \\
19 Allstate Parkway, Markham, ON Canada \authorcr
\tt m.elhoushi@ieee.org, \{\tt zihao.chen, farhan.shafiq, ye.tian2, joey.yiwei.li\}@huawei.com}
\maketitle
%\thispagestyle{empty}
%%%%%%%%% ABSTRACT
\begin{abstract}
Deployment of convolutional neural networks (CNNs) in mobile environments, their high computation and power budgets proves to be a major bottleneck. Convolution layers and fully connected layers, because of their intense use of multiplications, are the dominant contributer to this computation budget. This paper proposes to tackle this problem by introducing two new operations: convolutional shifts and fully-connected shifts, that replace multiplications all together with bitwise shift and sign flipping instead. For inference, both approaches may require only 6 bits to represent the weights. This family of neural network architectures (that use convolutional shifts and fully-connected shifts) are referred to as DeepShift models. We propose two methods to train DeepShift models: DeepShift-Q that trains regular weights constrained to powers of 2, and DeepShift-PS that trains the values of the shifts and sign flips directly. Training the DeepShift versions of ResNet18 architecture from scratch, we obtained accuracies of 92.33\% on CIFAR10 dataset, and Top-1/Top-5 accuracies of 65.63\%/86.33\% on Imagenet dataset. Training the DeepShift version of VGG16 on ImageNet from scratch, resulted in a drop of less than 0.3\% in Top-5 accuracy. Converting the pre-trained 32-bit floating point baseline model of GoogleNet to DeepShift and training it for 3 epochs, resulted in a Top-1/Top-5 accuracies of 69.87\%/89.62\% that are actually higher than that of the original model. Further testing is made on various well-known CNN architectures. Last but not least, we implemented the convolutional shifts and fully-connected shift GPU kernels and showed a reduction in latency time of 25\% when inferring ResNet18 compared to an unoptimized multiplication-based GPU kernels. The code is available online at \href{https://github.com/mostafaelhoushi/DeepShift}{https://github.com/mostafaelhoushi/DeepShift}.
\end{abstract}
%%%%%%%%% BODY TEXT
\section{Introduction}
Deep Neural Networks are increasingly being targeted for mobile and IoT applications. Devices at the edge have a lower power and price budget as well as constrained memory size. Moreover, the amount of communication between memory and compute also has a major role in the power requirements of a CNN. Moreover, if communication between device and cloud become necessary (e.g. in case of model updates etc), model size could affect the connectivity costs. Therefore, for mobile / IoT inference applications, model optimization, size reduction, faster inference and lower power consumption are key areas of research.
Several approaches are being considered to address this need and as such these efforts can be divided into a few categories. One approach is to build efficient models from the ground up resulting in novel network architectures, however that proves to be a task requiring a lot of training resource to try multiple variants of an architecture to find the best fit. Another approach is to start with a big model initially and prune it. Since among the many parameters in the network, some are redundant and don’t contribute a lot to the output, hence a ranking is assigned to each parameters based on contribution to the output. Low ranking parameters can be done away with (pruned), without effecting the accuracy too much. Another technique is to start with a big model and reduce the model size by applying quantization to smaller bit-width floating or fixed-point numbers. In some cases the quantized models are retrained to regain some of the accuracy. Key attractions of these technique are that they can be easily applied to various kinds of networks and they not only reduce model size but also require less complex compute units on the underlying hardware. This results in smaller model footprint, less working memory (and cache), faster computation on supporting platforms and lower power consumption. Moreover, some optimization techniques replace multiplication with cheaper operations such as binary XNOR operations or bitwise shifts. Binary XNOR techniques may have high accuracy on small datasets such as MNIST or CIFAR10, but suffer high degradation on complex datasets such as Imagenet. On the other hand, bit-wise shift techniques have proven to show minimal drop in accuracy on Imagenet.
This paper presents an approach to reduce computation and power budget of CNNs by replacing regular multiplcation-based convolution and linear operations (a.k.a fully-connected layer or matrix multiplication) with bitwise-shift-based convolution and linear operations respectively. Applying bitwise shift operation on an element is mathematically equivalent to multiplying it by a power of 2. We compare our work with 4 other similar approaches that replace multiplications with bitwise shift and sign flips: INQ \cite{Zhou2017}, LogNN \cite{Miyashita2016ConvolutionalNN}, and LogQuant \cite{Cai:2018:DLL:3291280.3291800}. We propose two approaches: DeepShift-Q and DeepShift-PS. Both our DeepShift approaches are based on training the shift values (i.e. the powers of 2) and sign flips either indirectly or directly, while INQ, LogNN, and LogQuant are based on rounding 32-bit floating-point weights to powers of 2. To the best of our knowledge, this paper is the first to propose training the shift values directly, which is the main novelty of the paper.
This paper makes the following contributions:
\begin{itemize}
\item We introduce 2 different methods to train DeepShift networks, neural networks with powers of 2 weights: each method trains the powers of 2 at run-time, and when computing the parameters gradients at train-time (see Section \ref{sec:deepshift-networks}).
\item We conduct experiments for each training method, which show that it is possible to train DeepShift networks on MNIST, CIFAR-10 and Imagenet, from scratch or from a pre-trained full-precision weights, and achieve nearly state-of-the-art results (see Section \ref{sec:benchmark-results}).
\item We developed a bitwise-shift-based matrix multiplication and convolution GPU kernel with which it is possible to run the DeepShift ResNet18 architecture with 25\% faster inference times than the baseline ResNet18 model with an unoptimized GPU kernel (see Section \ref{subsec:gpu-implementation} ).
\end{itemize}
The code for training and running our DeepShift networks using both methods is available online at \href{https://github.com/mostafaelhoushi/DeepShift}{https://github.com/mostafaelhoushi/DeepShift}.
\section{DeepShift Networks} \label{sec:deepshift-networks}
%TODO: re-do figure in Microsoft Visio
\begin{figure*}
\centering
\begin{subfigure}[b]{.3\textwidth}
\includegraphics[width=\textwidth]{mac2sac}
\caption{}\label{fig:shiftops:mac2sac}
\end{subfigure}\qquad
\begin{subfigure}[b]{.3\textwidth}
\includegraphics[width=\textwidth]{linearshift}
\caption{}\label{fig:shiftops:linear}
\end{subfigure}\qquad
\begin{subfigure}[b]{.3\textwidth}
\includegraphics[width=\textwidth]{convshift}
\caption{}\label{fig:shiftops:conv}
\end{subfigure}
\caption{(a) Multiply and Accumulate (MAC), which is the main operation in most neural networks, is replaced in DeepShift with bitwise shift and addition/subtraction. (b) Weights of original linear operator vs. proposed shift linear operator. (c) Weights of original convolution operator vs. proposed shift convolution operator}
\label{fig:shiftops}
\end{figure*}
As shown in Figure \ref{fig:shiftops}, the main concept of this paper is to replace multiplication with bitwise shift and sign flip. If the underlying binary representation of an input number, $ x $ is in integer or fixed-point format, a bit-wise shift of $ \tilde{p} $ bits, where $ \tilde{p} $ is an integer, to the left (or right) is mathematically equivalent to multiplying by a positive (or negative power) of 2:
\begin{equation}
\begin{split}
2^{\tilde{p}}x = \begin{cases}
x << \tilde{p} \text{\; if } \tilde{p} > 0 \\
x >> \tilde{p} \text{\; if } \tilde{p} < 0 \\
x \text{\; if } \tilde{p} = 0
\end{cases}
\end{split}
s.t. \space \tilde{p} \in \mathds{Z}
\end{equation}
For simplicity, we will use $ << \tilde{p} $ to implicitly denote a left shift if $ \tilde{p} $ is positive and right shift if $ \tilde{p} $ is negative.
Bitwise shift can only be equivalent to multiplying by a positive number, because $ 2^{\pm \tilde{p}} > 0 $ for any real value of $ \tilde{p} $. However, in neural networks, it is necessary for the training to have the equivalent of multiplying by negative numbers in its search space, especially in convolutional neural networks where filters with both positive and negative values contribute to detecting edges. Therefore, we also need to use the sign flip (a.k.a negation) operation. We use a ternary sign operator, i.e., an operator that can either leave its operand unchanged, replace it with 0, or changes the sign of its operand. The sign flip operation can be represented mathematically as:
\begin{equation}
\begin{split}
\tilde{s}x = \begin{cases}
-x \text{\; if } \tilde{s} = -1 \\
0 \text{\; if } \tilde{s} = 0 \\
x \text{\; if } \tilde{s} = +1
\end{cases}
\end{split}
s.t. \space \tilde{s} \in \{-1, 0, +1\}
\end{equation}
Similar to bitwise shift, sign flip is a computationally cheap operation too as it involves returning the 2's complement of a number.
We introduce novel operators, LinearShift and ConvShift, that, in the forward pass, replace multiplication with bitwise shift and sign flip. Hence, the weight matrix, $ W $, whether it is used for linear operation (i.e., $ Y = WX + b $) or convolution operation (i.e., $ Y = W \circledast X + b $), will be replaced by elementwise product of the shift matrix $ \tilde{P} $ and sign matrix $ \tilde{S} $.
\begin{equation} \label{eqn:deepshift-fwd-pass}
W \to \tilde{S} \cdot 2^{\tilde{P}}
\end{equation}
In the following sub-sections we shall present two different methods to train the LinearShift and ConvShift operators of our DeepShift models: DeepShift-Q and DeepShift-PS. Both approaches use Equation \ref{eqn:deepshift-fwd-pass} for the forward pass but differ in the backward pass and differ in how $ \tilde{P} $ and $ \tilde{S} $ are derived. The two approaches are summarized in Figure \ref{fig:training-method}.
\begin{figure*}
\centering
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{deepshift-q}
\caption{DeepShift-Q}\label{fig:training-method:deepshift-q}
\end{subfigure}\qquad
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{deepshift-ps}
\caption{DeepShift-PS}\label{fig:training-method:deepshift-ps}
\end{subfigure}
\caption{This paper presents 2 training algorithms to train the bitwise shift and negation parameters of a model: (a) DeepShift-Q, and (b) DeepShift-PS.}
\label{fig:training-method}
\end{figure*}
\subsection{DeepShift-Q}
The DeepShift-Q approach, training is similar to the regular training approach of linear or convolution operations with weight $ W $, but in the forward pass and backward pass the weight matrix is quantized to $ \tilde{W}_{q} $ by rounding it to the nearest power of 2:
\begin{equation} \label{eqn:sign-equation}
\begin{split}
S &= \text{sign}(W) \\
\tilde{S} &= S
\end{split}
\end{equation}
\begin{equation} \label{eqn:shift-equation}
\begin{split}
P &= \text{log}_2(\text{abs}(W)) \\
\tilde{P} &= \text{round}(P) \\
\end{split}
\end{equation}
\begin{equation}
\begin{split}
\tilde{W}_{q} = \tilde{S} \cdot 2^{\tilde{P}}
\end{split}
\end{equation}
Hence, the forward pass for our LinearShift operator will be $ Y = \tilde{W}_{q}X = (\tilde{S} \cdot 2^{\tilde{P}})X $ and for our ConvShift operator will be $ Y = \tilde{W}_{q} \circledast X + b = (\tilde{S} \cdot 2^{\tilde{P}}) \circledast X + b $.
The gradients of the backward pass can be expressed as:
\begin{equation} \label{eqn:deepshift-q-bwd-pass}
\begin{split}
\frac{\partial C}{\partial X} & = \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial X} = \frac{\partial C}{\partial Y}\tilde{W}_{q} \\
\frac{\partial C}{\partial W} & = \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial \tilde{W}_{q}}\frac{\partial \tilde{W}_{q}}{\partial W} \\
\frac{\partial C}{\partial b} & = \frac{\partial C}{\partial Y}
\end{split}
\end{equation}
where $ \frac{\partial C}{\partial Y} $ is the gradient input to the operator (derivative of the model loss (a.k.a cost function), $ C $, with respect to the operator output), $ \frac{\partial C}{\partial X} $ is the gradient output to the operator (derivative of the model loss with respect to the operator input)\footnote{We have simplified the expression of $ \frac{\partial C}{\partial X} $ in Equation \ref{eqn:deepshift-q-bwd-pass}. To be exact, it is equivalent to $ \frac{\partial C}{\partial y}\tilde{W}_{q}^T $ for LinearShift and $ X \circledast \frac{\partial C}{\partial y} $ for ConvShift.}, and $ \frac{\partial C}{\partial W} $ is the derivative of the model loss with respect to the operator weights \footnote{For LinearShift, $ \frac{\partial C}{\partial W} = x^T\frac{\partial C}{\partial y} $ and for ConvShift $ \frac{\partial C}{\partial W} = W_{rot180^{\circ}} \circledast \frac{\partial C}{\partial y} $}.
Since $ W_q $ is a rounded value of $ W $, the differentiation $ \frac{\partial \tilde{W}_{q}}{\partial W} $ is zero at all points except at the points where $ W $ is a power of 2. This will result in $ W $ not being updated during backpropagation, and hence no learning or updating of $ W $. We solve this issue by using a straight-through estimator (STE) \cite{yin2018understanding}:
\begin{equation}
\frac{\partial \tilde{W}_{q}}{\partial W} \approxeq 1
\end{equation}
Hence, $ \frac{\partial C}{\partial W} $ is set to:
\begin{equation}
\frac{\partial C}{\partial W} \approxeq \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial \tilde{W}_{q}}
\end{equation}
Figure \ref{fig:training-method:deepshift-q} summarizes the forward and backward passes of DeepShift-Q.
\subsection{DeepShift-PS}
DeepShift-PS on the other hand directly uses the shift, $ P $, and sign flip, $ S $, as the trainable parameters:
\begin{equation}
\begin{split}
\tilde{P} &= \text{round}(P) \\
\tilde{S} &= \text{sign}(\text{round}(S)) \\
\tilde{W}_{ps} &= \tilde{S} \cdot 2^{\tilde{P}}
\end{split}
\end{equation}
Note the definition of sign flip matrix, $ \tilde{S} $, implies that it is a ternary value rather than a binary parameter, as each element in the matrix can take one of 3 values:
\begin{equation}
\tilde{s} = \begin{cases}
-1 \text{\; if } s \leq -0.5 \\
0 \text{\; if } 0 < s < 0.5 \\
+1 \text{\; if } s \geq 0.5
\end{cases}
\end{equation}
where $ s $ is an element of matrix $ S $ and $ \tilde{s} $ is an element of $ \tilde{S} $.
The gradients of the backward pass can be expressed as:
\begin{equation}
\begin{split}
\frac{\partial C}{\partial X} &= \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial X} \\
&= \frac{\partial C}{\partial Y}W_{ps}^T \\
\frac{\partial C}{\partial P} &= \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial \tilde{W}_{ps}}\frac{\partial \tilde{W}_{ps}}{\partial \tilde{P}}\frac{\partial{\tilde P}}{\partial P} \\
& = \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial \tilde{W}_{ps}}\tilde{W_{ps}} \text{ln}2 \frac{\partial \tilde{P}}{\partial P} \\
\frac{\partial C}{\partial S} & = \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial \tilde{W}_{ps}}\frac{\partial \tilde{W}_{ps}}{\partial \tilde{S}}\frac{\partial \tilde{S}}{\partial S} \\
\frac{\partial L}{\partial b} & = \frac{\partial L}{\partial Y}
\end{split}
\end{equation}
We use the straight through estimator to set $ \frac{\partial \tilde{P}}{\partial P} \approxeq 1 $ and $ \frac{\partial \tilde{S}}{\partial S} \approxeq 1 $. To evaluate, $ \frac{\partial \tilde{W}_{ps}}{\partial \tilde{S}} $ we use the following derivation, treating the gradient of the sign operation as a straight-through estimator ($ \partial \text{sign}(x) = \partial x $) \footnote{We have tried an alternative method to express $ \frac{\partial \tilde{W}_{ps}}{\partial \tilde{S}} $ as $ \text{abs}(W_{ps}) $ but the training accuracy was very low. }:
\begin{equation}
\begin{split}
\tilde{S} &= \text{sign}(\tilde{W_{ps}}) \\
\partial \tilde{S} &= \partial \text{sign}(\tilde{W_{ps}}) \\
\partial \tilde{S} &\approxeq \partial \tilde{W_{ps}} \\
\frac{\partial \tilde{W}_{ps}}{\partial \tilde{S}} &\approxeq 1
\end{split}
\end{equation}
Hence, the gradients of the weights are set to:
\begin{equation}
\begin{split}
\frac{\partial C}{\partial P} &\approxeq \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial \tilde{W}_{ps}}\tilde{W_{ps}} \text{ln}2 \\
\frac{\partial C}{\partial S} &\approxeq \frac{\partial C}{\partial Y}\frac{\partial Y}{\partial \tilde{W}_{ps}}
\end{split}
\end{equation}
Figure \ref{fig:training-method:deepshift-ps} summarizes the forward and backward passes of DeepShift-PS.
%\subsection{LinearShift Operator}
%The linear operator (a.k.a fully-connected operator) is based on matrix multiplication. The forward pass can be expressed as:
%\begin{equation}
%y = Wx + b
%\end{equation}
%where $ x $ is the input that can be represented as a matrix of size $ B \times m_{in} $, $ y $ is the output that can be represented as a matrix of size $ B \times m_{out} $, $ W $ is the trainable weight matrix of size $ m_{in} \times m_{out} $, and $ b $ is the trainable bias vector of size $ m_{out} \times 1 $. $ B $ is the batch size while $ m_{in} $ is the input feature size and $ m_{out} $ is the output feature size.
%
%The backward pass of the linear operator can be expressed as:
%\begin{equation}
%\begin{split}
%\frac{\partial L}{\partial x} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial x} = \frac{\partial L}{\partial y}W^T \\
%\frac{\partial L}{\partial W} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial W} = x^T\frac{\partial L}{\partial y} \\
%\frac{\partial L}{\partial b} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial b} = \frac{\partial L}{\partial y}
%\end{split}
%\end{equation}
%where $ \frac{\partial L}{\partial y} $ is the gradient input to the operator (derivative of the model loss, $ L $, with respect to the operator output), $ \frac{\partial L}{\partial x} $ is the gradient output to the operator (derivative of the model loss with respect to the operator input), and $ \frac{\partial L}{\partial W} $ is the derivative of the model loss with respect to the operator weights.
%
%In this paper we introduce the shift linear operator, which in the forward pass replaces matrix multiplication with bit-wise shift and a negation. The forward pass is defined as:
%\begin{equation}
%y = ((-1)^{\text{round}(N)} \cdot (2)^{\text{round}(S)})x + b
%\label{linearshift_fwd_pass}
%\end{equation}
%where $ N $ is the matrix of negation, $ S $ is the matrix of shift values, and $ \cdot $ refers to elementwise multiplication of the 2 matrices. The size of both $ B $ and $ S $ is $ m_{in} \times m_{out} $. $ b $ is the bias vector which is similar to the original linear operator. $ S $, $ N $, and $ b $ are all trainable parameters.
%
%To help deduce the backward pass, we are going to use the term $ V = (-1)^{\text{round}(N)} \dot (2)^{\text{round}(S)} $:
%\begin{equation}
%\begin{split}
%\frac{\partial L}{\partial x} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial x} = \frac{\partial L}{\partial y}V^T \\
%\frac{\partial L}{\partial S} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial V}\frac{\partial V}{\partial S} = \left( x^T\frac{\partial L}{\partial y} \right) \cdot V\text{ln}2 \\
%\frac{\partial L}{\partial N} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial N}\frac{\partial N}{\partial S} = \left( x^T\frac{\partial L}{\partial y} \right) \cdot V\text{ln}(-1) \\
%\frac{\partial L}{\partial b} & = \frac{\partial L}{\partial y}
%\end{split}
%\end{equation}
%
%Note that the backward pass result in non-integer values for the powers of $ -1 $ and $ 2 $. However, in the forward pass they are rounded to enable implementing them as bitwise negation and shift respectively.
%
%\subsection{ConvShift Operator}
%The forward pass of the original convolution operator is expressed as:
%\begin{equation}
%y = W \circledast x + b
%\end{equation}
%where $ W $ has dimensions $ c_{out} \times c_{in} \times h \times w $, $ c_{in} $ is the input channel size, $ c_{out} $ is the output channel size, $ h $ and $ w $ are the height and width of the convolution filters. The backward pass of convolution is deduced in \cite{LeCun:1999:ORG:646469.691875} as:
%\begin{equation}
%\begin{split}
%\frac{\partial L}{\partial x} & = x \circledast \frac{\partial L}{\partial y} \\
%\frac{\partial L}{\partial W} & = W_{rot180^{\circ}} \circledast \frac{\partial L}{\partial y} \\
%\frac{\partial L}{\partial b} & = \frac{\partial L}{\partial y}
%\end{split}
%\end{equation}
%
%Likewise, our proposed convolutional shift (which we will refer to as ConvShift) operator has the forward pass defined as:
%\begin{equation}
%y = ((-1)^{\text{round}(N)} \cdot (2)^{\text{round}(S)}) \circledast x + b
%\label{convshift_fwd_pass}
%\end{equation}
%where $ N $ and $ S $ are the negation and shift matrices and have dimensions $ c_{out} \times c_{in} \times h \times w $.
%
%Similarly, to deduce the backward pass, we are going to use the term $ V = (-1)^{\text{round}(N)} \dot (2)^{\text{round}(S)} $:
%\begin{equation}
%\begin{split}
%\frac{\partial L}{\partial x} & = x \circledast \frac{\partial L}{\partial y} \\
%\frac{\partial L}{\partial S} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial V}\frac{\partial V}{\partial S} = \left( W_{rot180^{\circ}} \circledast \frac{\partial L}{\partial y}\right) \cdot V\text{ln}2 \\
%\frac{\partial L}{\partial N} & = \frac{\partial L}{\partial y}\frac{\partial y}{\partial V}\frac{\partial V}{\partial S} = \left( W_{rot180^{\circ}} \circledast \frac{\partial L}{\partial y}\right) \cdot V\text{ln}(-1) \\
%\frac{\partial L}{\partial b} & = \frac{\partial L}{\partial y}
%\end{split}
%\end{equation}
\section{Implementation}
To implement the forward and backward passes, we follow an approach similar to that of \cite{NIPS2016_6573}. We used PyTorch to define the forward pass and backward pass for the 2 custom ops: LinearShift and for ConvShift. In order to emulate the precision of an actual bitwise shift hardware implementation, the input data to the LinearShift and ConvShift operators and their bias parameters are rounded to fixed-point format precision before applying the forward pass.
To initialize the weights, we use Kamming initialization for $ W_q $ in DeepShift-Q, and uniform random distribution to initialize $ P $ and $ S $ in DeepShift-PS.
In DeepShift-PS, L2 normalization has been slightly modified to occur on $ W_{PS} $ rather than on $ P $ on $ S $, i.e. the regularization term added is $ \sum{W_{PS}^2} = \sum{{2^PS}^2} $ rather than $ \sum{P^2} + \sum{S^2} $.
%The code and some of the model binary files have been submitted in a compressed file along with the paper submission and will be made available online on GitHub upon acceptance.
\section{Benchmark Results} \label{sec:benchmark-results}
We have tested the training and inference results on 3 datasets: MNIST \cite{lecun-mnisthandwrittendigit-2010}, CIFAR10 \cite{cifar10_report}, and Imagenet \cite{imagenet_cvpr09}. For each dataset, we have tested a group of architectures. We have explored various evaluations for the models:
\begin{enumerate}
\item \textbf{Original Version}: evaluating the original architecture with standard convolution and linear operators, \label{enum:eval_original}
\item \textbf{DeepShift Versions}:
\begin{enumerate}
\item \textbf{Train from Scratch}: start with randomly initialized weights, convert all the convolution and linear operators to their shift counterparts, and train from scratch using either training method Deepshift-Q or DeepShift-PS, \label{enum:train_shift_scratch}
\item \textbf{Train from Pre-trained Baseline}: start with baseline model that is pretrained with 32-bits floating points weights, convert all the convolution and linear operators to their shift counterparts, convert the weights (using Equations \ref{eqn:sign-equation} and \ref{eqn:sign-equation}) and train using either training method Deepshift-Q or DeepShift-PS. % TODO: check equation references here
\end{enumerate}
\end{enumerate}
We have noticed - in terms of validation accuracy results - that the best optimizer for DeepShift-PS is Rectified Adam (RAdam) \cite{liu2019radam} while the best optimizer for DeepShift-Q and regular 32-bit floating point training is stochastic gradient descent (SGD).
% For each dataset: table results and possibly some per epoch training table.
% Showing best accuracy result
\subsection{MNIST Data Set}
Two simple models were trained and tested on the MNIST dataset:
\begin{itemize}
\item \textbf{Simple FC}: a simple fully-connected model consisting of 3 linear layers with feature output sizes 512, 512, and 10 respectively. Dropout layers with probability of 0.2 were inserted in between the layers. All intermediate layers had a ReLu activation following it.
\item \textbf{Simple CNN}: a model consisting of 2 convolutional layers and 2 linear layers. The 2 convolutional layers had output channels sizes 20 and 50 respectively, and both had kernel sizes of 5x5 and strides of 1. Max pooling layers of window size 2x2 followed by ReLu activation were inserted after each convolution layer. The linear layers had output feature sizes of 500 and 10 respectively.
\end{itemize}
A learning rate of 0.01, a momentum of 0.0, as well as batch size of 64 was used to train. The training plot is shown in Figure \ref{fig:mnist}. The accuracy on the validation set is shown in Table \ref{mnist-results}.
\begin{figure*}
\centering
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{mnist-linear}
\caption{Simple FC}\label{fig:mnist:linear}
\end{subfigure}\qquad
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{mnist-conv}
\caption{Simple CNN}\label{fig:mnist:conv}
\end{subfigure}
\caption{MNIST training from scratch}
\label{fig:mnist}
\end{figure*}
\begin{table}
\caption{DeepShift accuracy results on MNIST dataset}
\label{mnist-results}
\centering
\begin{tabularx}{\columnwidth}{l*{2}{>{\centering\arraybackslash}X}}
\toprule
Method & \shortstack[l]{Train \\ from Scratch} & \shortstack[l]{Train \\ from Pre-trained} \\
\cmidrule(r){1-3}
Simple FC & & \\
\midrule
Original & 96.92\% & - \\
DeepShift-Q & 97.03\% & 94.91\% \\
DeepShift-PS & 98.26\% & 98.26\% \\
\midrule
Simple CNN & & \\
\midrule
Original & 98.75\% & - \\
DeepShift-Q & 98.81\% & 99.15\% \\
DeepShift-PS & 99.12\% & 99.16\% \\
\bottomrule
\end{tabularx}
\end{table}
\subsection{CIFAR10 Data Set}
ResNet18 and MobileNetv2 models were trained on scratch on CIFAR10 usnig DeepShit-PS method. The models were trained with momentum of 0.9 and weight decay of \num{1e-4}. The loss criterion used was categorical cross entropy. The learning rate used to train was 0.1 that decays by a factor of 0.1 at epochs 80, 120, and 160, the number of epochs for training was 200, and the batch size was 128. For ResNet18, we have added an additional test of training the baseline (multiplication-based) 32-bit floating point weights using RAdam, to compare it with DeepShift-PS that uses RAdam and ensure the comparisons are fair. The best accuracy reached by the baseline ResNet18 model with SGD was 94.86\% and with RAdam was 94.06\%.
For training starting with a pre-trained model, the learning rate was $ 1 \times 10^-4 $, and the number of epochs was 15.
The training plots are shown in \ref{fig:cifar10} while the numerical results for evaluating on the validation set for various scenarios are shown in Table \ref{cifar10-results}.
\begin{table}
\caption{DeepShift accuracy results on CIFAR10 dataset}
\label{cifar10-results}
\centering
\begin{tabularx}{\columnwidth}{l*{2}{>{\centering\arraybackslash}X}}
\toprule
Method & \shortstack[l]{Train \\ from Scratch} & \shortstack[l]{Train \\ from Pre-trained} \\
\cmidrule(r){1-3}
ResNet18 & & \\
\midrule
Original & 94.86\% & - \\
DeepShift-Q & 94.03\% & 94.52\% \\
DeepShift-PS & 92.34\% & 94.52\% \\
\midrule
MobileNet & & \\
\midrule
Original & 92.42\% & - \\
DeepShift-Q & 93.53\% & 92.75\% \\
DeepShift-PS & 91.97\% & 92.48\% \\
\bottomrule
\end{tabularx}
\end{table}
\begin{figure*}
\centering
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{cifar10-resnet18}
\caption{ResNet18}\label{fig:cifar10:resnet18}
\end{subfigure}\qquad
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{cifar10-mobilenet}
\caption{MobileNet}\label{fig:cifar10:mobielenet}
\end{subfigure}
\caption{CIFAR10 training from scratch}
\label{fig:cifar10}
\end{figure*}
\subsection{Imagenet Data Set}
% TODO: Update plots for ResNet18, and add plot for VGG16 + Add analysis
For Imagenet, ResNet18 and VGG16 were trained from scratch with momentum of 0.9 and weight decay of \num{1e-4}. The loss criterion used was categorical cross entropy, the number of epochs was 90, the learning rate used to train from scratch was 0.1 that decays by a factor of 0.1 every 30 epochs. As can be seen from the training plot in Figure \ref{fig:imagenet-resnet18}, and the results in Table \ref{imagenet-results}, the best accuracy reached when training ResNet18 from scratch was less than the full-precision model by around 4\% in Top-5 accuracy and less than the 3\% in Top-1 accuracy, but for VGG16 the drop was less than 0.3\%. For ResNet18, DeepShift-PS was slighlty better than DeepShift-Q in Top-1 accuracy, but vice versa for DeepShift-PS. Training VGG16 using DeepShift-PS is under progress. We have shown comparison to other work that trained models from scratch multiplication replaced with XNOR operations.
For training from a pre-trained model, we show the results for ResNet18, ResNet50, AlexNet, VGG16, and GoogleNet. We see that for GoogleNet, the accuracy obtained was higher than the original model. We show results to compare with other work that converted 32-bit weights to powers of 2. %The number of epochs and learning rate used to train a pre-trained converted model is specified for each model in Table \ref{imagenet-results}. The learning rate for training each converted model was manually tuned by looking at the accuracy of the first few batches of training: if accuracy was decreasing to that below of the untrained converted DeepShift model, if the accuracy dropped as compared to the untrained model, then the weights have "unlearnt" and therefore the learning rate was too high and needed to decreases.
%Looking at the results of Table \ref{imagenet-results}, we can see that different models had varying results. The best result obtained was for ResNet152 which had a Top-1 accuracy of 75.56\% and Top-5 accuracy of 92.75\%. It is noteworthy that due to limited time, many of the models were trained for only 4 epochs. Training them for more epochs may result in better accuracies. More complex models tend to have better results when converted to DeepShift. A slime model like MobileNetv2 got around 6\% in reduction of accuracy when all the multiplications in it have been removed. This is considered a strong advantage when compared to other acceleration methods (e.g., XNOR networks, quantization, or pruning) that have negative results for optimizing MobileNets. Nevertheless, other slim networks such as SqueezeNets suffered drastic reduction in accuracy. It is yet to be analyzed why MobileNetv2 had almost 0\% accuracy when weights were converted without further training, while it had above 84\% Top-5 accuracy when trained for a few epochs.
\begin{figure*}
\centering
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{imagenet-resnet18-top1}
\caption{ResNet18 Top-1 Accuracy}\label{fig:imagenet:resnet18-top1}
\end{subfigure}\qquad
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{imagenet-resnet18-top5}
\caption{ResNet18 Top-5 Accuracy}\label{fig:imagenet:resnet18-top5}
\end{subfigure}
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{imagenet-vgg16-top1}
\caption{VGG16 Top-1 Accuracy}\label{fig:imagenet:vgg16-top1}
\end{subfigure}
\begin{subfigure}[b]{.4\textwidth}
\includegraphics[width=\textwidth]{imagenet-vgg16-top5}
\caption{VGG16 Top-5 Accuracy}\label{fig:imagenet:vgg16-top5}
\end{subfigure}
\caption{Imagenet training from scratch}
\label{fig:imagenet-resnet18}
\end{figure*}
\begin{table*}
\caption{DeepShift accuracy results on Imagenet dataset and comparison with other methods. Columns ``A'' and ``W'' show the number of bits used for each model to represent actiations and weights in each method. For fair comparison, we also reported a separate column for change from baseline that each paper had, as different papers reported different baselines.}
\label{imagenet-results}
\centering
\begin{tabularx}{\textwidth}{l*{6}{>{\centering\arraybackslash}X}}
\toprule
Method & A & W & \multicolumn{2}{c}{Top-1} & \multicolumn{2}{c}{Top-5} \\
\midrule % \cmidrule(r){1-4}
& Accuracy & & & Delta & Accuracy & Delta \\
\midrule
\bottomrule
ResNet18 & & & & & & \\
\midrule
Original & 32 & 32 & 69.76\% & - & 89.08\% & - \\
\midrule
\multicolumn{7}{l}{Train from Scratch} \\
\midrule
\shortstack[l]{DeepShift-Q from Scratch [Ours]} & 32 & 6 & 65.28\% & -4.48\% & 86.45\% & -2.63\% \\
\shortstack[l]{DeepShift-PS from Scratch [Ours]} & 32 & 6 & \textbf{65.63\%} & -4.13\% & 86.33\% & -2.75\% \\
BWN \cite{Rastegari2016} & 32 & 1 & 60.80\% & -9.30\% & 83.00\% & -6.20\% \\
TWN \cite{Li2016} & 32 & 2 & 61.80\% & \textbf{-3.60\%} & 84.20\% & -2.56\% \\
\midrule
\multicolumn{7}{l}{Train from Pre-Trained Model} \\
\midrule
\shortstack[l]{DeepShift-Q from Pre-trained [Ours]} & 32 & 6 & 68.67\% & -1.09\% & 88.61\% & -0.47\% \\
\shortstack[l]{DeepShift-PS from Pre-trained [Ours]} & 32 & 6 & 68.32\% & -1.44\% & 88.41\% & -0.67\% \\
INQ \cite{Zhou2017} & 32 & 5 & 68.98\% & +0.71\% & 89.10\% & +0.41\% \\
ABC \cite{Lin2017BCNN} & 32 & 5 & 68.3\% & -1.00\% & 87.9\% & -1.30\% \\
\midrule
\bottomrule
ResNet50 & & & & & & \\
\midrule
Original & 32 & 32 & 76.13\% & - & 92.86\% & - \\
\midrule
\multicolumn{7}{l}{Train from Pre-Trained Model} \\
\midrule
\shortstack[l]{DeepShift-Q from Pre-trained [Ours]} & 32 & 6 & 75.32\% & -0.81\% & 92.65\% & -0.21\% \\
\shortstack[l]{DeepShift-PS from Pre-trained [Ours]} & 32 & 6 & 75.29\% & -0.84\% & 92.55\% & -0.31\% \\
INQ \cite{Zhou2017} & 32 & 5 & 74.81\% & +1.59\% & 92.45\% & +1.21\% \\
\midrule
\bottomrule
GoogleNet & & & & & & \\
\midrule
Original & 32 & 32 & 69.78\% & - & 89.53\% & - \\
\midrule
\multicolumn{7}{l}{Train from Pre-Trained Model} \\
\midrule
\shortstack[l]{DeepShift-Q from Pre-trained [Ours]} & 32 & 6 & 69.51\% & -0.27\% & 89.24\% & -0.29\% \\
\shortstack[l]{DeepShift-PS from Pre-trained [Ours]} & 32 & 6 & 69.87\% & +0.09\% & 89.62\% & +0.09\% \\
INQ \cite{Zhou2017} & 32 & 5 & 69.02\% & +0.13\% & 89.28\% & +0.25\% \\
LogQuant \cite{Cai:2018:DLL:3291280.3291800} & 32 & 6 & 69.14\% & -0.36\% & 88.86\% & -0.28\% \\
\midrule
\bottomrule
VGG16 & & & & & & \\
\midrule
Original & 32 & 32 & 71.59\% & - & 90.38\% & - \\
\midrule
\multicolumn{7}{l}{Train from Scratch} \\
\shortstack[l]{DeepShift-Q from Scratch [Ours]} & 32 & 6 & 70.87\% & -0.71\% & 90.09\% & -0.29\% \\
\midrule
\multicolumn{7}{l}{Train from Pre-Trained Model} \\
\midrule
\shortstack[l]{DeepShift-Q from Pre-trained [Ours]} & 32 & 6 & 71.30\% & -0.29\% & 90.27\% & -0.09\% \\
\shortstack[l]{DeepShift-PS from Pre-trained [Ours]} & 32 & 6 & 71.12\% & -0.47\% & 90.08\% & -0.30\% \\
INQ \cite{Zhou2017} & 32 & 5 & 70.82\% & +2.28\% & 90.30\% & +1.65\% \\
LogNN \cite{Miyashita2016ConvolutionalNN} & 4 & 5 & N/A & N/A & 89.00\% & -0.50\% \\
LogQuant \cite{Cai:2018:DLL:3291280.3291800} & 32 & 6 & N/A & N/A & 89.62\% & -0.31\% \\
\midrule
\bottomrule
AlexNet & & & & & & \\
\midrule
Original & 32 & 32 & 56.52\% & - & 79.07\% & - \\
\midrule
\multicolumn{7}{l}{Train from Pre-Trained Model} \\
\midrule
\shortstack[l]{DeepShift-Q from Pre-trained [Ours]} & 32 & 6 & 54.97\% & -1.55\% & 78.26\% & -0.81\% \\
\shortstack[l]{DeepShift-PS from Pre-trained [Ours]} & 32 & 6 & 55.33\% & -1.19\% & 78.40\% & -0.67\% \\
INQ \cite{Zhou2017} & 32 & 6 & 57.39\% & +0.15\% & 80.46\% & +0.23\% \\
LogNN \cite{Miyashita2016ConvolutionalNN} & 4 & 5 & N/A & N/A & 75.1\% & -1.7\% \\
\midrule
\bottomrule
\end{tabularx}
\end{table*}
%\begin{table*}
% \caption{DeepShift Top-1 and Top-5 accuracy results on Imagenet dataset. E = Epochs, LR = Learning Rate, B = Batch Size}
% \label{imagenet-results}
% \centering
% \begin{tabularx}{\textwidth}{l|Xc|Xcll}%{l*{6}{>{\centering\arraybackslash}X}}
% \toprule
% Model & \multicolumn{2}{c}{Original Version} & \multicolumn{4}{c}{DeepShift Version} \\
% \cmidrule(r){2-7}
% & Size & Acc. & Size & \shortstack[l]{Convert Original Weights\\(No Additional Training)} & \multicolumn{2}{l}{\shortstack[l]{Convert Original Weights\\ + Training}} \\
% \midrule
% AlexNet & 233MB & 56.52\% / 79.07\% & 36.4MB & 42.99\% / 67.40\% & 50.73\% / 75.01\% & \blap{E: 90, LR: 0.0001\\ B: 256} \\
% VGG11 & 507MB & 69.02\% / 88.63\% & 79.2MB & 46.76\% / 71.29\% & 66.38\% / 87.30\% & \blap{E: 128, LR: 0.1\\ B: 32} \\
% VGG11-bn & 507MB & 70.37\% / 89.81\% & 79.3MB & 37.49\% / 61.94\% & 66.36\% / 87.50\% & \blap{E: 24, LR: 0.005\\ B:16} \\
% VGG13 & 508MB & 69.93\% / 89.25\% & 79.3MB & 60.34\% / 82.56\% & 68.09\% / 88.22\% & \blap{E: 10, LR: 0.1\\ B: 64} \\
% VGG13-bn & 508MB & 71.59\% / 90.37\% & 79.4MB & 45.92\% / 70.38\% & 57.97\% / 81.83\% & \blap{E: 4, LR: 0.1\\ B: 32} \\
% VGG16 & 528MB & 71.59\% / 90.38\% & 82.5MB & 65.25\% / 86.30\% & 70.28\% / 89.77\% & \blap{E: 9, LR: 0.01\\ B: 8 } \\
% VGG16-bn & 528MB & 73.36\% / 91.52\% & 82.6MB & 56.30\% / 79.77\% & 71.98\% / 90.81\% & \blap{E: 5, LR: 0.001\\ B: 16} \\
% VGG19 & 548MB & 72.38\% / 90.88\% & 85.7MB & 66.61\% / 87.21\% & 69.91\% / 89.46\% & \blap{E: 5, LR: 0.1\\ B: 16} \\
% VGG19-bn & 548MB & 74.22\% / 91.84\% & 85.8MB & 58.96\% / 82.02\% & 72.85\% / 91.15\% & \blap{E: 2, LR: 0.001\\ B: 16} \\
% DenseNet121 & 30.8MB & 74.43\% / 91.97\% & 5.35MB & 46.40\% / 71.95\% & 73.30\% / 91.52\% & \blap{E: 200, LR: 0.01\\ B:64} \\
% DenseNet169 & 54.6MB & 75.60\% / 92.81\% & 9.55MB & 39.24\% / 63.93\% & 72.84\% / 91.28\% & \blap{E: 5, LR: 0.01\\ B: 32} \\
% DenseNet201 & 77.2MB & 76.90\% / 93.37\% & 13.5MB & 51.84\% / 75.83\% & 73.83\% / 91.80\% & \blap{E: 5, LR: 0.01\\ B: 32} \\
% DenseNet161 & 110MB & 77.14\% / 93.56\% & 18.7MB & 61.97\% / 84.64\% & 73.92\% / 91.88\% & \blap{E: 13, LR: 0.01\\ B: 32} \\
% ResNet18 & 44.6MB & 69.76\% / 89.08\% & 7.04MB & 41.53\% / 67.29\% & 68.14\% / 88.40\% & \blap{E: 200, LR: 0.1\\ B: 256} \\
% ResNet34 & 83.2MB & 73.31\% / 91.42\% & 13.1MB & 56.26\% / 80.22\% & 72.06\% / 90.79\% & \blap{E: 115, LR=0.1\\ B:256} \\
% ResNet50 & 97.7MB & 76.13\% / 92.86\% & 15.6MB & 41.30\% / 65.10\% & 73.90\% / 91.76\% & \blap{E: 100, LR: 0.1\\ B: 64} \\
% ResNet101 & 170MB & 77.37\% / 93.55\% & 27.3MB & 52.59\% / 76.57\% & 69.21\% / 88.95\% & \blap{E: 4, LR: 0.1\\ B: 64} \\
% ResNet152 & 230MB & 78.31\% / 94.05\% & 37.0MB & 46.14\% / 69.15\% & 75.97\% / 92.85\% & \blap{E: 10, LR: 0.01\\ B: 32} \\
% MobileNetv1 & 16.2MB & 69.57\% / 89.07\% & 2.68MB & 0.14\% / 0.67\% & 61.88\% / 84.11\% & \blap{E: 50, LR: 0.01\\ B: 128} \\
% MobileNetv2 & 13.5MB & 71.81\% / 90.42\% & 2.33MB & 0.10\% / 0.48\% & 64.78\% / 86.27\% & \blap{E: 200, LR: 0.01\\ B: 128} \\
% SqueezeNet1-0 & 4.76MB & 58.09\% / 80.42\% & 0.76MB & 12.56\% / 29.92\% & 21.71\% / 44.74\% & \blap{E: 10, LR: 0.0001\\ B: 256} \\
% SqueezeNet1-1 & 4.71MB & 58.18\% / 80.62\% & 0.75MB & 4.01\% / 12.19\% & 15.50\% / 35.25\% & \blap{E: 10, LR: 0.0001\\ B: 256} \\
% \bottomrule
% \end{tabularx}
%\end{table*}
\section{Efficiency Analysis} \label{sec:efficiency-analysis}
Binary multiplication for integer or fixed-point format consists of multiple bit-wise shifts, ANDs, and additions. Floating point (FP) multiplication consist of more steps such as adding the exponents, and normalizing. In terms of CPU cycles, taking 32-bit Intel Atom instruction processor as an example, integer and FP multiplication instruction take 5 to 6 clock cycles, while a bit-wise shift instruction takes only 1 clock cycle (\cite{AgnerInstructionTables}). (\cite{Asati2009}) shows for 16-bit architectures, average power and area of multipliers are at least $9.7\times$, and $1.45\times$, respectively, of bitwise shifter: 22.05 nW to 54.83 nW vs 1.32 nW to 2.27 nW, and 8.7K to 29K transistor count vs 2K to 6K only. Hence, replacing multiplication with bitwise shift reduces energy. A common optimization in C++ compilers is to detect an integer multiplication with a (constant that is known at compilation time to be) a power of 2 and replace it with bitwise shift. Our contribution is enabling bitwise shift in training neural networks and achieving good accuracy.% We emphasize that we replace each multiplication with a \textit{single} bitwise shift. \\
DeepShift also achieves model storage compression by representing the weights with fewer bits. Since activations are represented us 32-bit fixed point activations, the range of shift values that we need to support are -32 to +32. Even if training results in shift values that are outside those ranges, they will be mathematically equivalent to multiplying by 0. Moreover, we noticed that shift values are rarely positive (i.e., most of the time $ \text{abs}(W_Q) < 1 $, hence $ P < 0 $ as $ P = \text{log}_2(\text{abs}(\text{round}(W_Q))) $ \footnote{This expression holds for both $ W_Q $ and $ W_PS $}). Therefore, we only need to support shift values of 0 to -32, and this requires 5 bits. Adding the bit required for the sign flip operation, we need 6 bits in total.
In addition to saving area and power for a potential custom hardware accelerator, our main advantages are to reduce the number of bits (thereby to reduce memory footprint of the model), and increase energy savings for transferring weights between different layers of memory of a GPU or CPU (\cite{Sze_2017}). That's needed in mobile and IoT applications, where memory footprint, even more so than FLOPs, are a primary constraint. Energy savings is crucial for applications like drones that fly on batteries, whether the computing platform is a low-end CPU, GPU, or FPGA. Also, bitwise shift operators are a good candidate for in-memory computing research \cite{Sze_2017}, as they're cheaper in size and energy to place in or near memory than multiplication.
\subsection{GPU Implementation} \label{subsec:gpu-implementation}
While, various custom hardware designs have been proposed by \cite{LogNet_7953288}, \cite{Vogel:2018:EHA:3240765.3240803} and \cite{ShiftCNN}, we have implemented a GPU kernel that inferred a bit-wise-shift based ResNet18 in 1490 seconds on Imagenet compared to baseline of 2010 seconds. To the best of our knowledge, this is the first paper to implement a CUDA kernel convolution using bitwise shifts.
It is noteworthy that the baseline that we compared against is a custom multiplication-based convolution kernel which is slower than NVIDIA's cuDNN optimized kernel. Future work can be done to optimize our bitwise-shift CUDA kernels to compare them with NVIDIA's cuDNN multiplication-based kernels, such as fusing convolution with ReLu, tuning the tiling factors, and performing JIT (just-in-time) compilation of the CUDA kernels at runtime.
Some of the challenges that we faced when implementing the bitwise-shift convolution kernels, were that:
- the GPU's instruction set architecture was not optimized for using smaller bit-width parameters that were needed to represent the shift and sign. Having to extract the 6-bit shift value and 1-bit sign value from 32-bit integers using masking and extra shifts incurred a lot of overhead, and
- if-else statements on the values of the sign bits to determine whether to add or subtract during accumulation, or if-else conditions on whether the shift values are positive or negative to determine whether to shift left or right. If-else conditions stall instruction execution and are a bottleneck to pipelining instructions in hardware
These were bottlenecks that we had to mitigate in our code, and limited the speedup of the kernel compared to multiplication-based kernels. The implementation of the kernel can be found in our GitHub repo.
%TODO: Explain that optimized difficult to beat because of tuned tiling parameters per dimension (add citation) + fusing with Relu + jit compilation
%TODO: Explain that we compressed shift representation using variable length scheme
%TODO: Add Zihao's figure
%\subsection{Further Analysis}
%We have analyzed the minimum and maximum values of shifts encountered in all the models and found that the maxium was 1 while the minimum was -70. This means that the shift values can be represented using 8-bits rather than 32-bits as regular weights of convolutions are represented. Also, the negation can be saved as a single bit too.
%Comparing the differences in weight values between DeepShift models with converted weights without training, and their weights after training, we found that the shift and negate values did not change at all for any of the layers for any of the models tested for Imagenet and CIFAR10, while the difference was only in the bias values.
% Histogram of shift values -> number of bits required to represent
% Compare how much shifts and signs change between conversion only and conversion + training
\section{Related Work}
%This section covers related works which fall in three categories. First, innovative new network architectures have been proposed that reduce the model sizes as well as computation requirements. \cite{Iandola2017} proposed SqueezeNet achieving 50x fewer parameters compared to AlexNet. \cite{Howard2017} proposed MobileNets, based on a streamlined architecture that uses depth-wise separable convolutions to build light weight DNNs, followed by MobileNetV2 \cite{Sandler2018} introducing inverted residual structure where the input and output of the residual blocks are thin bottleneck layers opposite to traditional residual models. \cite{Chen2019DropAO} Proposed reducing spatial redundancy in CNN by replacing usual convolution operations with Octave Convolution, a spatialy less-redundant variant.
%Secondly, model quantization techniques have been an active research area and various approaches have been developed. \cite{Courbariaux2015} proposed Binary Connect constraining weights to only two values (-1 or 1). \cite{Rastegari2016} improve on Binary Connect, introducing Binary Weight Networks (BWNs), employing channel-wise scaling factors, while \cite{Li2016} introduced Ternary Weight Networks (TWNs). \cite{Zhu2017} extended TWN with Trained Ternary Quantization (TTQ), with non-uniform and trainable scaling factors. Other works like Binarized Neural Networks \cite{Courbariaux2016}, XNOR-Net \cite{Rastegari2016}, Bi-Real Net \cite{Liu2018} and ABCNet \cite{Lin2017BCNN} quantize both weights and activations to either -1 or 1.
\cite{Zhou2017} proposed Incremental Network Quantization (INQ) to quantize pre-trained full-precision DNNs with weights constrained to zeros and powers of two. This is accomplished by iteratively partitioning the weights into two sets, one of which is quantized while the other is retrained to compensate for accuracy degradation.
LogNN \cite{Miyashita2016ConvolutionalNN} is another method that converts weights as well as activations to powers of 2. However, it requires manual empirical tuning rather than training. The authors of LogNN did present an algorithm to train weights and activations in powers of 2, but it was only implemented on pre-trained model on CIFAR10 dataset. Furthermore, the authors did not explain how sign flip of weights should be trained. The authors of LogNN also presented a hardware implementation in \cite{LogNet_7953288}. LogQuant \cite{Cai:2018:DLL:3291280.3291800} presented a method to convert the weights of a pre-trained model to powers of 2, such that no further training is needed and with minimal decrease in accuracy.
%TODO: Explain APoT ?
To the best of our knowledge, DeepShift is the first method to train a model from scratch using powers of 2.
%\cite{Cai2017} proposed low-precision activations using Half-Wave Gaussian Quantization (HWGQ) while weights are binarized using BWN. \cite{Faraone2018} introduced Symmetric Quantization (SYQ) by pixel-wise scaling factors and fixed-point activation quantization. DoReFa-Net \cite{Zhou2016}, PACT \cite{Choi2018} and PACT-SAWB \cite{ChoiChunag2018} allows weights and activation to be variable configurable. \cite{Baskin2018} proposed uniform noise injection for non-uniform quantization (UNIQ) of both weights and activations. \cite{Lin2017} use multi-bit quantization. \cite{Zhang2018} proposed an adaptively learnable quantizer (LQ-Nets).
%Thirdly pruning can be used to reduce model redundancy. Several pruning works have been proposed that employ various kinds of ranking mechanisms. \cite{Hanson1989} introduced hyperbolic and exponential biases for pruning in the late 80s. \cite{Hassibi1993} and \cite{LeCun1990} published some of the other earlier works on pruning. More recently, \cite{Han2016CoRR} proposed Deep Compression, a method that leverages pruning, weight sharing and Huffman Coding for model compression. \cite{Lin2017} proposed Runtime Neural Pruning, a framework which prunes the deep neural network dynamically at the runtime. Pruning introduces another complexity to the mix, since after pruning the computation units need to be able to handle sparse matrix arithmetic which adds some overhead.
%Finally, efficient kernel libraries \cite{GEMMLOWP} , \cite{IntelMKLDNN}, \cite{KeilCMSISNN}, \cite{QualcommSNPE}, \cite{NvidiaTensorRT} and custom hardware targeting reduced precesion have been developed to improve inference performance \cite{NVDLA}, \cite{Han2016ISCA}, \cite{Umuroglu2017}.
\section{Conclusion and Future Work}
We have introduced DeepShift neural networks, which replace multiplications in the forward pass with bitwise operations - bitwise shift and negation, that can potentially lead to dramatic reduction in computation time, power consumption, and memory requirements during inference and training. We have proved that the accuracies of DeepShift networks on Imagenet and other datasets are near state-of-the-art.
%The hardware realization of DeepShift neural networks is yet to be done, and is needed to evaluate the actual speedup in performance. Designing parallel architectures for bitwise shift and negation on vectors rather than on individual registers may face its own challenges but it is expected to have faster execution times than their multiplication counterparts. While training from pre-trained weights - rounded to their bitwise shift counterparts - result in much higher accuracies that are close to the state-of-the-art, training DeepShift networks from scratch results in low accuracies. Therefore, further research is required to find better random initialization methods for weights to enable training DeepShift networks from scratch. Also, the current design of DeepShift networks still need multiplications in the backward pass (as non-integer powers of 2 are used). Researching into using bitwise shifts in the backward pass may lead to dramatic seed-up in training as well.
While other methods that tackle neural network speed ups such as BNNs perform well on small datasets (e.g., MNIST and CIFAR10) but suffer significant degradation on big datasets such as Imagenet, DeepShift networks proved that they are suitable for Imagenet.
Also, a GPU kernel of performing convolution using bitwise shift has been implemented as a proof-of-concept. Developing GPU instruction set architectures to make them more optimized for smaller bitwidth data types, and shift operations (e.g., an assembly instruction that can deduce bitwise shift should be done in the opposite direction if the shift value is negative) will help in making bitwise-shift-based training and inference more common. Moreover, existence of vector CPU instructions that can perform bitwise shifts given a vector of integers, and a vector of shift values, can make DeepShift more optimal on CPUs.
For future work, we suggest to look into training models from scratch with both activations and weights represented as powers of 2, similar to \cite{Miyashita2016ConvolutionalNN} that did that for inference only. Also, we would research into creating the bitwise-shift convolution kernels for CPU, as well as apply the well-known optimizations (fusing with activation, tiling tuning, and JIT compilation) for the GPU kernel.
\subsubsection*{Acknowledgments}
We thank Ahmed Eltantawy for providing us valuable feedback to implement the CUDA kernels. We thank Sara Elkerdawy for providing valuable insights during our discussions with her. We thank also the developers of PyTorch for providing example scripts and pre-trained model binary files to reproduce the accuracies of various models on the Imagenet dataset.
{\small
\bibliographystyle{ieee_fullname}
\bibliography{paper}
}
\end{document}
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{Asati2009}
Abhijit~Rameshwar Asati.
\newblock {\em A Comparative Study of High Performance CMOS Multipliers, Barrel
Shifters and Modeling of NBTI Degradation in Nanometer Scale Digital VLSI
Circuits}.
\newblock PhD thesis, Birla Institute of Technology and Science, Pilani,
Rajasthan, India, 2009.
\bibitem{Cai:2018:DLL:3291280.3291800}
Jingyong Cai, Masashi Takemoto, and Hironori Nakajo.
\newblock A deep look into logarithmic quantization of model parameters in
neural networks.
\newblock In {\em Proceedings of the 10th International Conference on Advances
in Information Technology}, IAIT 2018, pages 6:1--6:8, New York, NY, USA,
2018. ACM.
\bibitem{imagenet_cvpr09}
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.
\newblock {ImageNet: A Large-Scale Hierarchical Image Database}.
\newblock In {\em CVPR09}, 2009.
\bibitem{AgnerInstructionTables}
Agner Fog.
\newblock Instruction tables: Lists of instruction latencies, throughputs and
micro-operation breakdowns for intel, amd and via cpus.
\newblock \url{https://www.agner.org/optimize/instruction_tables.pdf}, 2018.
\bibitem{ShiftCNN}
Denis Gudovskiy and Luca Rigazio.
\newblock Shift{CNN}: Generalized low-precision architecture for inference of
convolutional neural networks.
\newblock {\em arXiv preprint arXiv:1706.02393}, 2017.
\bibitem{NIPS2016_6573}
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua
Bengio.
\newblock Binarized neural networks.
\newblock In D.~D. Lee, M. Sugiyama, U.~V. Luxburg, I. Guyon, and R. Garnett,
editors, {\em Advances in Neural Information Processing Systems 29}, pages
4107--4115. Curran Associates, Inc., 2016.
\bibitem{cifar10_report}
A. Krizhevsky.
\newblock {Learning Multiple Layers of Features from Tiny Images}.
\newblock Technical report, University of Toronto, Department of Computer
Science, 2009.
\bibitem{lecun-mnisthandwrittendigit-2010}
Yann LeCun and Corinna Cortes.
\newblock {MNIST} handwritten digit database.
\newblock 2010.
\bibitem{LogNet_7953288}
E.~H. {Lee}, D. {Miyashita}, E. {Chai}, B. {Murmann}, and S.~S. {Wong}.
\newblock Lognet: Energy-efficient neural networks using logarithmic
computation.
\newblock In {\em 2017 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP)}, pages 5900--5904, March 2017.
\bibitem{Li2016}
Fengfu Li and Bin Liu.
\newblock Ternary weight networks.
\newblock {\em CoRR}, abs/1605.04711, 2016.
\bibitem{Lin2017BCNN}
Xiaofan Lin, Cong Zhao, and Wei Pan.
\newblock Towards accurate binary convolutional neural network.
\newblock In I. Guyon, U.~V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S.
Vishwanathan, and R. Garnett, editors, {\em Advances in Neural Information
Processing Systems 30}, pages 345--353. Curran Associates, Inc., 2017.
\bibitem{liu2019radam}
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng
Gao, and Jiawei Han.
\newblock On the variance of the adaptive learning rate and beyond.
\newblock {\em arXiv preprint arXiv:1908.03265}, 2019.
\bibitem{Miyashita2016ConvolutionalNN}
Daisuke Miyashita, Edward~H. Lee, and Boris Murmann.
\newblock Convolutional neural networks using logarithmic data representation.
\newblock {\em ArXiv}, abs/1603.01025, 2016.
\bibitem{Rastegari2016}
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.
\newblock Xnor-net: Imagenet classification using binary convolutional neural
networks.
\newblock {\em CoRR}, abs/1603.05279, 2016.
\bibitem{Sze_2017}
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel~S. Emer.
\newblock Efficient processing of deep neural networks: A tutorial and survey.
\newblock {\em Proceedings of the IEEE}, 105(12):2295–2329, Dec 2017.
\bibitem{Vogel:2018:EHA:3240765.3240803}
Sebastian Vogel, Mengyu Liang, Andre Guntoro, Walter Stechele, and Gerd
Ascheid.
\newblock Efficient hardware acceleration of cnns using logarithmic data
representation with arbitrary log-base.
\newblock In {\em Proceedings of the International Conference on Computer-Aided
Design}, ICCAD '18, pages 9:1--9:8, New York, NY, USA, 2018. ACM.
\bibitem{yin2018understanding}
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley~J. Osher, Yingyong Qi, and
Jack Xin.
\newblock Understanding straight-through estimator in training activation
quantized neural nets.
\newblock In {\em International Conference on Learning Representations}, 2019.
\bibitem{Zhou2017}
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen.
\newblock Incremental network quantization: Towards lossless cnns with
low-precision weights.
\newblock {\em CoRR}, abs/1702.03044, 2017.
\end{thebibliography}