TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
Extracted source text · arXiv 1705.07878
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading nips_2017
%
% to avoid loading the natbib package, add option nonatbib:
%\usepackage[]{nips_2017}
%\usepackage{nips_2017}
% to compile a camera-ready version, add the [final] option, e.g.:
\usepackage[nonatbib, final]{nips_2017}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{subfiles}
\usepackage[nottoc]{tocbibind}
\usepackage{textcomp}
\usepackage[square, numbers, sort]{natbib}
\usepackage{graphicx}
\usepackage{amssymb}
\bibliographystyle{unsrtnat}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{threeparttable}
\usepackage[marginal, flushmargin]{footmisc}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage[algo2e,ruled,linesnumbered]{algorithm2e}
\usepackage{multirow}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{makecell}
\DeclareMathOperator{\tr}{tr}
\DeclareMathOperator{\diag}{diag}
\usepackage{comment}
\usepackage{color}
\usepackage{etoolbox}
\newbool{inccomment}
\setbool{inccomment}{true}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\newtheorem{theorem}{Theorem}
\newtheorem{assumption}{Assumption}
\newtheorem{lemma}{Lemma}
\usepackage{wrapfig}
\usepackage{caption}
\usepackage[titletoc,title]{appendix}
\newcommand{\hl}[1]{\ifbool{inccomment}{{\color{magenta}#1}}{}}
\newcommand{\ww}[1]{\ifbool{inccomment}{{\color{blue} #1}}{}}
\newcommand{\yc}[1]{\ifbool{inccomment}{{\color{red} #1}}{}}
\newcommand{\terngrad}{\textit{TernGrad}}
\newcommand{\alexnet}{\textit{AlexNet}}
\newcommand{\lenet}{\textit{LeNet}}
\newcommand{\cifarnet}{\textit{CifarNet}}
\newcommand{\googlenet}{\textit{GoogLeNet}}
\newcommand{\vggneta}{\textit{VggNet-A}}
\newcommand{\eg}{\textit{e.g.}}
\newcommand{\ie}{\textit{i.e.}}
\newcommand{\etal}{\textit{et al.}}
\newcommand{\etc}{\textit{etc}}
%\title{TernGrad: Reducing Communication in Distributed Deep Learning by Ternary Gradients}
\title{TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning}
% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.
\author{
Wei Wen\textsuperscript{1}, Cong Xu\textsuperscript{2}, Feng Yan\textsuperscript{3}, Chunpeng Wu\textsuperscript{1}, Yandan Wang\textsuperscript{4}, Yiran Chen\textsuperscript{1}, Hai Li\textsuperscript{1} \\
\\
\textsuperscript{1}Duke University, \textsuperscript{2}Hewlett Packard Labs, \textsuperscript{3}University of Nevada -- Reno, \textsuperscript{4}University of Pittsburgh\\
\textsuperscript{1}\texttt{\{wei.wen, chunpeng.wu, yiran.chen, hai.li\}@duke.edu} \\
\textsuperscript{2}\texttt{cong.xu@hpe.com}, \textsuperscript{3}\texttt{fyan@unr.edu}, \textsuperscript{4}\texttt{yaw46@pitt.edu}\\
% Wei Wen \\
% Duke University\\
% \texttt{wei.wen@duke.edu} \\
% \And
% Cong Xu \\
% Hewlett Packard Labs\\
% \texttt{cong.xu@hpe.com} \\
% \And
% Feng Yan \\
% University of Nevada, Reno\\
% \texttt{fyan@unr.edu} \\
% \And
% Chunpeng Wu \\
% Duke University\\
% \texttt{chunpeng.wu@duke.edu} \\
% \And
% Yandan Wang \\
% University of Pittsburgh\\
% \texttt{yaw46@pitt.edu} \\
% \And
% Yiran Chen \\
% Duke University\\
% \texttt{yiran.chen@duke.edu} \\
% \And
% Hai Li \\
% Duke University\\
% \texttt{hai.li@duke.edu} \\
}
\begin{document}
% \nipsfinalcopy is no longer used
\maketitle
\begin{abstract}
\vspace{-6pt}
High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training.
% that arises for data parallelism.
In this work, we propose \terngrad{} that uses ternary gradients to accelerate distributed deep learning in data parallelism.
Our approach requires only three numerical levels $\{-1,0,1\}$, which can aggressively reduce the communication time.
%Moreover, \textit{parameter localization} and \textit{scaler sharing} schemes reduce the synchronization overheads of both gradients and parameters.
We mathematically prove the convergence of \terngrad{} under the assumption of a bound on gradients.
Guided by the bound, we propose \textit{layer-wise ternarizing} and \textit{gradient clipping} to improve its convergence.
Our experiments show that applying \terngrad{} on \alexnet{} doesn't incur any accuracy loss and can even improve accuracy.
%In our experiments of \terngrad{}, \alexnet{} doesn't incur any accuracy loss and even can improve accuracy.
The accuracy loss of \googlenet{} induced by \terngrad{} is less than $2\%$ on average.
Finally, a performance model is proposed to study the scalability of \terngrad{}.
Experiments show significant speed gains for various deep neural networks.
Our source code is available~\footnote{\url{https://github.com/wenwei202/terngrad}}.
%Communication for synchronization of gradients and parameters is often the bottleneck for distributed training in data parallelism.
%We propose \terngrad{}, a new approach using ternary gradients to accelerate distributed deep learning.
%Our ternary gradients have only three numerical levels of $\{-1,0,1\}$, and therefore aggressively reduces the communication time.
%With proposed \textit{scaler sharing} scheme, \terngrad{} can reduce synchronization time for both gradients and parameters.
%Based on proposed performance model, \terngrad{} successfully reduces communication time by \ww{$?\times$}.
%We also mathematically guarantee the convergence of \terngrad{} under the assumption of a bound on the gradients.
%Guided by the bound, we successfully train \alexnet{} on ImageNet by \terngrad{} without accuracy loss or improved accuracy in some cases, and \googlenet{} with small accuracy loss.
\end{abstract}
\input{introduction}
\input{related_work}
\input{methodology}
\input{experiments}
\input{discussion}
%\input{conclusion}
\subsubsection*{Acknowledgments}
This work was supported in part by NSF CCF-1744082 and DOE SC0017030.
Any opinions, findings, conclusions or recommendations expressed
in this material are those of the authors and do not necessarily reflect the views of NSF, DOE, or their contractors.
\small
\bibliography{nips2017}
\newpage
\begin{appendices}
\input{supp_proof}
\input{supp_perfmodel}
\end{appendices}
\end{document}
\section{Introduction}
\label{sec:intro}
\vspace{-6pt}
%@Frameworks: tensorflow, mxnet, adam， sparknet with caffe， petuum
The remarkable advances in deep learning is driven by data explosion and increase of model size.
The training of large-scale models with huge amounts of data are often carried on distributed systems~\cite{distbelief}\cite{abadi2016tensorflow}\cite{coates2013deep}\cite{recht2011hogwild}\cite{chilimbi2014project}\cite{xing2015petuum}\cite{moritz2015sparknet}\cite{chen2015mxnet}\cite{zhang2015deep}, where data parallelism is adopted to exploit the compute capability empowered by multiple workers~\cite{li2017scaling}.
\textit{Stochastic Gradient Descent} (SGD) is usually selected as the optimization method because of its high computation efficiency.
In realizing the data parallelism of SGD, model copies in computing workers are trained in parallel by applying different subsets of data.
%Afterward,
A centralized parameter server performs \textit{gradient synchronization} by collecting all gradients and averaging them to update parameters.
%Finally
The updated parameters will be sent back to workers, that is, \textit{parameter synchronization}.
Increasing the number of workers helps to reduce the computation time dramatically.
However, as the scale of distributed systems grows up, the extensive gradient and parameter synchronizations prolong the communication time and even amortize the savings of computation time~\cite{recht2011hogwild}\cite{li2014scaling}\cite{li2014communication}.
A common approach to overcome such a network bottleneck is \textit{asynchronous SGD}~\cite{distbelief}\cite{recht2011hogwild}\cite{moritz2015sparknet}\cite{li2014communication}\cite{ho2013more}\cite{zinkevich2010parallelized}, which continues computation by using stale values without waiting for the completeness of synchronization.
The inconsistency of parameters across computing workers, however, can degrade training accuracy and incur occasional divergence~\cite{pan2017revisiting}\cite{Zhang2016Staleness}.
Moreover, its workload dynamics make the training nondeterministic and hard to debug.
From the perspective of inference acceleration, sparse and quantized \textit{Deep Neural Networks} (DNNs) have been widely studied, such as~\cite{han2015deep}\cite{wen2016learning}\cite{park2017faster}\cite{hubara2016binarized}\cite{rastegari2016xnor}\cite{zhou2016dorefa}\cite{wen2017iclrlearning}\cite{ott2016recurrent}\cite{lin2015neural}.
However, these methods generally aggravate the training effort.
Researches such as sparse logistic regression and Lasso optimization problems~\cite{recht2011hogwild}\cite{li2014communication}\cite{bradley2011parallel} took advantage of the sparsity inherent in models and achieved remarkable speedup for distributed training.
A more generic and important topic is how to accelerate the distributed training of dense models by utilizing sparsity and quantization techniques.
For instance, Aji and Heafield~\cite{aji2017sparse} proposed to heuristically sparsify dense gradients by dropping off small values in order to reduce gradient communication.
For the same purpose, quantizing gradients to low-precision values with smaller bit width has also been extensively studied~\cite{zhou2016dorefa}\cite{seide20141}\cite{tomiokaqsgd}\cite{gupta2015deep}.
Our work belongs to the category of gradient quantization, which is an orthogonal approach to sparsity methods.
We propose \terngrad{} that quantizes gradients to ternary levels $\{-1,0,1\}$ to reduce the overhead of \textit{gradient synchronization}.
Furthermore, we propose \textit{scaler sharing} and \textit{parameter localization}, which can replace \textit{parameter synchronization} with a low-precision gradient pulling.
%we propose \textit{scaler sharing} across all workers and \textit{parameter caching} within each worker, which not only can reduce the overhead of \textit{gradient synchronization} but also eliminate the \textit{parameter synchronization} replacing which with a low-precision gradient pulling;
Comparing with previous works, our major contributions include:
(1)~we use ternary values for gradients to reduce communication;
(2)~we mathematically prove the convergence of \terngrad{} in general by proposing a statistical bound on gradients;
(3)~we propose \textit{layer-wise ternarizing} and \textit{gradient clipping} to move this bound closer toward the bound of standard SGD. These simple techniques successfully improve the convergence;
(4)~we build a performance model to evaluate the speed of training methods with compressed gradients, like \terngrad{}.
%
%\hl{Comparing with previous works, our experiments on \alexnet{} and \googlenet{} tested on ImageNet dataset achieve either better accuracy or less communication traffic.
%when the other factor is the same, }
\section{Related work}
\label{sec:related}
\vspace{-6pt}
%\textbf{Sparse gradients.}
\textbf{Gradient sparsification.}
Aji and Heafield~\cite{aji2017sparse} proposed a heuristic gradient sparsification method that truncated the smallest gradients %to zeros
and transmitted only the remaining large ones.
The method greatly reduced the gradient communication and achieved 22\% speed gain on $4$ GPUs for a neural machine translation, without impacting the translation quality.
An earlier study by Garg \etal{}~\cite{garg2009gradient} adopted the similar approach, but targeted at sparsity recovery instead of training acceleration.
Our proposed \terngrad{} is orthogonal to these sparsity-based methods.
%\textbf{Low-precision gradients.}
\textbf{Gradient quantization.}
\textit{DoReFa-Net}~\cite{zhou2016dorefa} derived from \alexnet{} reduced the bit widths of weights, activations and gradients to 1, 2 and 6, respectively.
%Derived from \alexnet{}, \textit{DoReFa-Net} by S.~Zhou~\etal~\cite{zhou2016dorefa} adopted 1-bit weights, 2-bit activations and 6-bit gradients.
However, \textit{DoReFa-Net} showed $9.8\%$ accuracy loss as it targeted at acceleration on single worker.
S.~Gupta~\etal~\cite{gupta2015deep} successfully trained neural networks on MNIST and CIFAR-10 datasets using 16-bit numerical precision for an energy-efficient hardware accelerator.
%\hl{[weight or gradient in [27]? \ww{All data representation -- weights, activations, gradients.}]}
Our work, instead, tends to speedup the distributed training by decreasing the communicated gradients to three numerical levels $\{-1,0,1\}$.
F.~Seide~\etal~\cite{seide20141} applied 1-bit SGD to accelerate distributed training and empirically verified its effectiveness in speech applications.
As the gradient quantization is conducted by columns, a floating-point scaler \textit{per column} is required.
So it cannot yield speed benefit on convolutional neural networks~\cite{tomiokaqsgd}.
%However, it quantizes gradients \textit{per column} and demands a floating scaler per column, because of which, it does not yield any communication benefits when applied to convolutional neural networks~\cite{tomiokaqsgd}.
Moreover, ``\textit{cold start}'' of the method~\cite{seide20141} requires floating-point gradients %is necessary %in the beginning stage, likely
to converge to a good initial point for the following 1-bit SGD.
More importantly, it is unknown what conditions can guarantee its convergence.
Comparably, our \terngrad{} can start the DNN training from scratch and we prove the conditions that promise the convergence of \terngrad{}. A.~T.~Suresh~\etal{}~\cite{suresh2016distributed} proposed stochastic rotated quantization of gradients, and reduced gradient precision to 4 bits for MNIST and CIFAR dataset. However, \terngrad{} achieves lower precision for larger dataset (\eg{} ImageNet), and has more efficient computation for quantization in each computing node.
Very recently, a preprint by D.~Alistarh~\etal{}~\cite{tomiokaqsgd} presented QSGD that explores the trade-off between accuracy and gradient precision.
The effectiveness of gradient quantization was justified and the convergence of QSGD was provably guaranteed.
Compared to QSGD developed simultaneously, our \terngrad{} shares the same concept but advances in the following three aspects:
(1) we prove the convergence from the perspective of statistic bound on gradients. The bound also explains why multiple quantization buckets are necessary in QSGD;
(2) the bound is used to guide practices and inspires techniques of \textit{layer-wise ternarizing} and \textit{gradient clipping};
(3) \terngrad{} using only $3$-level gradients achieves $0.92\%$ top-1 accuracy \textit{improvement} for \alexnet{}, while $1.73\%$ top-1 accuracy \textit{loss} is observed in QSGD with $4$ levels.
The accuracy loss in QSGD can be eliminated by paying the cost of increasing the precision to 4 bits (16 levels) and beyond.
%Convergence of QSGD is provably guaranteed.
%\ww{QSGD is simultaneously developed with \terngrad{} and further justifies the effectiveness of gradient quantization.}
%However, besides contributions in Section~\ref{sec:intro}, \terngrad{} differs from QSGD in many ways. First, we proves the convergence from the perspective of statistic bound on gradients and the bound explains why multiple quantization buckets are necessary in QSGD.
%Second, the bound can guide practices and inspire the technique of \textit{gradient clipping}.
%Finally, \terngrad{} achieves $0.92\%$ top-1 accuracy improvement for \alexnet{} using only $3$ levels, comparing with $1.73\%$ top-1 accuracy loss in QSGD using $4$ levels. The accuracy loss in QSGD can be eliminated, but by increasing precision to 4-bit (16-level).
\section{Problem Formulation and Our Approach}
\label{sec:method}
\vspace{-6pt}
\input{method_algorithm}
\input{method_convergence}
\input{method_practical}
\subsection{Problem Formulation and \terngrad{} }
\label{sec:method:algo}
\vspace{-6pt}
%\begin{figure}
% \centering
% \includegraphics[width=.5\columnwidth]{figures/terngrad-crop.pdf}
% \caption{\textit{A variant of distributed stochastic gradient descent under data parallelism.}}
% \label{fig:terngrad}
%\end{figure}
Figure~\ref{fig:terngrad} formulates the distributed training problem of synchronous SGD using data parallelism.
At iteration $t$, a mini-batch of training samples are split and fed into multiple workers ($i \in \{1,...,N\}$).
Worker $i$ computes the gradients $ \bm{g}^{(i)}_t$ of parameters \textit{w.r.t.} its input samples $\bm{z}_t^{(i)}$.
All gradients are first synchronized and averaged at \textit{parameter server}, and then sent back to update workers.
Note that parameter server in most implementations~\cite{distbelief}\cite{li2014communication} are used to preserve shared \textit{parameters}, while here we utilize it in a slightly different way of maintaining shared \textit{gradients}.
%In most implementations~\cite{distbelief}\cite{li2014communication}, parameter servers maintain shared \textit{parameters}; but in Figure~\ref{fig:terngrad}, parameter servers maintain shared \textit{gradients}.
In Figure~\ref{fig:terngrad}, each worker keeps a copy of parameters locally.
We name this technique as \textit{parameter localization}.
The parameter consistency among workers can be maintained by random initialization with an identical seed.
%By pulling gradients (instead of parameters) from parameter servers, we
\textit{Parameter localization} changes the communication of parameters in floating-point form to the transfer of quantized gradients that require much lighter traffic.
%The reason to pull gradients from parameter servers instead of pulling parameters is that it can eliminate communication of floating parameters replacing it with a smaller traffic of quantized gradients.
%Unlike some implementations~\cite{li2014communication}\cite{distbelief} which allocate parameters in the master servers (\ie{} \textit{Parameter Servers}), we maintain an identical copy of parameters in each worker. This can reduce more communications by pulling
%when the gradients are ternary. In detail, ternarizing gradients can reduce \textit{worker-to-master} communications, but pull $\overline{\bm{w}_t}$ from parameter servers cannot benefit from the ternary gradients.
Note that our proposed \terngrad{} can be integrated with many settings like \textit{Asynchronous SGD}~\cite{distbelief}\cite{recht2011hogwild}, even though the scope of this paper only focuses on the distributed SGD in Figure~\ref{fig:terngrad}.
%\begin{algorithm}%[H]
% \small
% \SetAlgoNoLine
% \SetKwInOut{Input}{Worker}
% \SetKwInOut{Output}{Server}
% \caption{\terngrad: distributed SGD training using ternary gradients}
% \label{algo:terngrad}
% \Input{$i=1,...,N$}
% \Indp
% Input $\bm{z}_t^{(i)}$, a portion of a mini-batch of training samples $\bm{z}_t$
%
% Compute gradients $ \bm{g}_t^{(i)}$ \textit{w.r.t.} $\bm{z}_t^{(i)}$
%
% Ternarize gradients to $ \tilde{\bm{g}}_t^{(i)} = ternarize( \bm{g}_t^{(i)})$
%
% Push ternary $ \tilde{\bm{g}}_t^{(i)}$ to the server
%
% Pull $ \overline{ \bm{g}_t } $ from the server
%
% Update parameters $\bm{w}_{t+1} \leftarrow \bm{w}_{t} - \eta \cdot \overline{ \bm{g}_t} $
%
% \Indm\Output{}
% \Indp
% Average ternary gradients $ \overline{ \bm{g}_t } = \sum_{i} \tilde{\bm{g}}_t^{(i)} / N $
%\end{algorithm}
\textbf{Algorithm~\ref{algo:terngrad}} formulates the $t$-th iteration of \terngrad{} algorithm according to Figure~\ref{fig:terngrad}.
Most steps of \terngrad{} remain the same as traditional distributed training, except that gradients shall be quantized into ternary precision before sending to parameter server.
More specific, $ternarize(\cdot)$ aims to reduce the communication volume of gradients.
It randomly quantizes gradient $\bm{g}_t$~\footnote{Here, the superscript of $\bm{g}_t$ is omitted for simplicity.} to a ternary vector with values $\in \{-1, 0,+1\}$.
Formally, with a random binary vector $\bm{b}_t$, $\bm{g}_t$ is ternarized as
\begin{equation}
\label{eq:ternarize}
\tilde{\bm{g}}_t = ternarize(\bm{g}_t) = s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t,
\end{equation}
where
\begin{equation}
\label{eq:s}
s_t \triangleq max\left( abs\left( \bm{g}_t \right) \right)
\end{equation}
is a \textit{scaler} that can shrink $\pm 1$ to a much smaller amplitude.
$\circ$ is the Hadamard product.
$sign(\cdot)$ and $abs(\cdot)$ respectively returns the sign and absolute value of each element.
Giving a $\bm{g}_t$, each element of $\bm{b}_t$ independently follows the Bernoulli distribution
\begin{equation}
\label{eq:ber_dist_1st}
\begin{cases}
P(b_{tk}=1~|~\bm{g}_t) = |g_{tk}|/s_t\\
P(b_{tk}=0~|~\bm{g}_t) = 1 - |g_{tk}|/s_t
\end{cases},
\end{equation}
where $b_{tk}$ and $g_{tk}$ is the $k$-th element of $\bm{b}_t$ and $\bm{g}_t$, respectively.
This \textit{stochastic rounding}, instead of deterministic one, is chosen by both our study and QSGD~\cite{tomiokaqsgd}, as stochastic rounding has an unbiased expectation and has been successfully studied for low-precision processing~\cite{hubara2016binarized}\cite{gupta2015deep}.
%We chose this \textit{stochastic rounding} in our study instead of using deterministic one, considering it has an unbiased expectation and was successfully studied for low-precision processing~\cite{hubara2016binarized}\cite{gupta2015deep}\cite{tomiokaqsgd} \ww{[QSGD uses exactly the same formulation of (2). Here I am just trying to say this sort of trick is popular and is neither novel in QSGD.]}.
Theoretically, ternary gradients can at least reduce the \textit{worker-to-server} traffic by a factor of ${32}/{log_2(3)}=20.18 \times$.
Even using 2 bits to encode a ternary gradient, the reduction factor is still $16 \times$.
In this work, we compare \terngrad{} with $32$-bit gradients, considering $32$-bit is the default precision in modern deep learning frameworks.
Although a lower-precision (\eg{} $16$-bit) may be enough in some scenarios, it will not undervalue \terngrad{}.
As aforementioned, \textit{parameter localization} reduces \textit{server-to-worker} traffic by pulling quantized gradients from servers.
However, summing up ternary values in $\sum_{i} \tilde{\bm{g}}_t^{(i)}$ will produce more possible levels and thereby the final averaged gradient $\overline{\bm{g}_t}$ is no longer ternary as shown in Figure~\ref{fig:histograms}(d).
It emerges as a critical issue when workers use different scalers $s^{(i)}_t$.
To minimize the number of levels, we propose a shared scaler % $s_t = max ( \{ s_t^{(i)} \} : i = 1...N)$
\begin{equation}
\label{eq:scalar_avg}
s_t = max ( \{ s_t^{(i)} \} : i = 1...N)
\end{equation}
across all the workers. We name this technique as \textit{scaler sharing}.
The sharing process has a small overhead of transferring $2N$ floating scalars.
By integrating \textit{parameter localization} and \textit{scaler sharing}, the maximum number of levels in $\overline{\bm{g}_t}$ decreases to $2N+1$.
As a result, the \textit{server-to-worker} communication reduces by a factor of $32/log_2(1+2N)$, unless $N\geq2^{30}$.
\begin{minipage}{.40\textwidth}
\centering
\vspace{9pt}
\includegraphics[width=1.0\columnwidth]{figures/terngrad-crop.pdf}
\captionof{figure}{Distributed SGD with data parallelism.}
\label{fig:terngrad}
\end{minipage}
\hfill
\begin{minipage}{.50\textwidth}
\begin{algorithm}[H]
\small
\SetAlgoNoLine
\SetKwInOut{Input}{Worker}
\SetKwInOut{Output}{Server}
\caption{\terngrad: distributed SGD training using ternary gradients.}
\label{algo:terngrad}
\Input{~$i=1,...,N$}
\Indp
Input $\bm{z}_t^{(i)}$, a part of a mini-batch of training samples $\bm{z}_t$
Compute gradients $ \bm{g}_t^{(i)}$ under $\bm{z}_t^{(i)}$
Ternarize gradients to $ \tilde{\bm{g}}_t^{(i)} = ternarize( \bm{g}_t^{(i)})$
Push ternary $ \tilde{\bm{g}}_t^{(i)}$ to the server
Pull averaged gradients $ \overline{\bm{g}_t} $ from the server
Update parameters $\bm{w}_{t+1} \leftarrow \bm{w}_{t} - \eta \cdot \overline{\bm{g}_t}$
\Indm\Output{}
\Indp
Average ternary gradients $ \overline{ \bm{g}_t } = \sum_{i} \tilde{\bm{g}}_t^{(i)} / N $
\end{algorithm}
\end{minipage}
\subsection{Convergence Analysis and Gradient Bound}
\label{sec:method:convergence}
\vspace{-6pt}
We analyze the convergence of \terngrad{} in the framework of online learning systems.
An online learning system adapts its parameter $\bm{w}$ to a sequence of observations to maximize performance.
Each observation $\bm{z}$ is drawn from an unknown distribution, and a loss function $Q(\bm{z},\bm{w})$ is used to measure the performance of current system with parameter $\bm{w}$ and input $\bm{z}$.
The minimization target then is the loss expectation
\begin{equation}
C(\bm{w}) \triangleq \mathbf{E}\left\lbrace Q(\bm{z},\bm{w}) \right\rbrace.
\label{eq:minimize}
\end{equation}
In \textit{General Online Gradient Algorithm}~(GOGA)~\cite{bottou1998online}, parameter is updated at learning rate $\gamma_{t}$ as
\begin{equation}
\bm{w}_{t+1} = \bm{w}_{t} - \gamma_{t} \bm{g}_t
= \bm{w}_{t} - \gamma_{t} \cdot \nabla_{\bm{w}} Q(\bm{z}_t, \bm{w}_t),
\end{equation}
where % $\bm{g} \triangleq \nabla_{\bm{w}} Q(\bm{z}, \bm{w})$
\begin{equation}
\bm{g} \triangleq \nabla_{\bm{w}} Q(\bm{z}, \bm{w})
\end{equation}
and the subscript $t$ denotes %the corresponding value at
observing step $t$.
In GOGA, $\mathbf{E}\left\lbrace \bm{g} \right\rbrace$ is the gradient of the minimization target in Eq.~(\ref{eq:minimize}).
%To reduce communication volume of gradients, \terngrad{} randomly quantize $\bm{g}_t$ to a ternary vector, each value of which belongs to $\{0,+1, -1\}$.
According to Eq.~(\ref{eq:ternarize}), the parameter in \terngrad{} is updated, such as
\begin{equation}
\bm{w}_{t+1} = \bm{w}_{t} - \gamma_{t} \left( s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right),
\end{equation}
where $s_t \triangleq max\left( abs\left( \bm{g}_t \right) \right) $ is a \textit{random variable} depending on $\bm{z}_t$ and $\bm{w}_t$.
As $\bm{g}_t$ is known for given $\bm{z}_t$ and $\bm{w}_t$, Eq.~(\ref{eq:ber_dist_1st}) is equivalent to
\begin{equation}
\label{eq:ber_dist}
\begin{cases}
P(b_{tk}=1~|~\bm{z}_t,\bm{w}_t) = |g_{tk}|/s_t\\
P(b_{tk}=0~|~\bm{z}_t,\bm{w}_t) = 1 - |g_{tk}|/s_t
\end{cases}.
\end{equation}
%where $b_{tk}$ and $g_{tk}$ is the $k$-th element of $\bm{b}_t$ and $\bm{g}_t$, respectively.
At any given $\bm{w}_t$, the expectation of ternary gradient satisfies
\begin{equation}
\label{eq:expected_grad}
\mathbf{E}\left\lbrace s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right\rbrace
= \mathbf{E}\left\lbrace s_t \cdot sign\left( \bm{g}_t \right) \circ \mathbf{E} \left\lbrace \bm{b}_t | \bm{z}_t \right\rbrace \right\rbrace
= \mathbf{E}\left\lbrace \bm{g}_t \right\rbrace
= \nabla_{\bm{w}} C(\bm{w}_t),
\end{equation}
which is an unbiased gradient of minimization target in Eq.~(\ref{eq:minimize}).
The convergence analysis of \terngrad{} is adapted from the convergence proof of GOGA presented in~\cite{bottou1998online}.
We adopt two assumptions, which were used in analysis of the convergence of standard GOGA in~\cite{bottou1998online}.
Without explicit mention, vectors indicate column vectors here.
\begin{assumption}
\label{assumption1}
$C(\bm{w})$ has a single minimum $\bm{w}^*$ and gradient $ - \nabla_{\bm{w} } C(\bm{w})$ always points to $\bm{w}^*$, \textit{i.e.},
\begin{equation}
\forall \epsilon > 0, \inf\limits_{ \left||\bm{w} - \bm{w}^* \right||^2 > \epsilon} \left(\bm{w} - \bm{w}^* \right)^T \nabla_{\bm{w} } C(\bm{w}) > 0.
\end{equation}
\end{assumption}
Convexity is a subset of Assumption~\ref{assumption1}, and we can easily find non-convex functions satisfying it.
\begin{assumption}
\label{assumption2}
Learning rate $\gamma_{t}$ is positive and constrained as
\begin{equation}
\begin{cases}
\sum_{t=0}^{+\infty} \gamma^2_{t} < + \infty \\
\sum_{t=0}^{+\infty} \gamma_{t} = + \infty
\end{cases},
\end{equation}
which ensures $\gamma_{t}$ decreases neither very fast nor very slow respectively.
\end{assumption}
We define the square of distance between current parameter $\bm{w}_t$ and the minimum $\bm{w}^*$ as % $h_t \triangleq \left|\left| \bm{w}_t - \bm{w}^* \right|\right|^2$,
\begin{equation}
h_t \triangleq \left|\left| \bm{w}_t - \bm{w}^* \right|\right|^2,
\end{equation}
where $||\cdot||$ is $\ell_2$ norm.
We also define the set of all random variables before step $t$ as % $\bm{X}_t \triangleq \left( \bm{z}_{1...t-1}, \bm{b}_{1...t-1} \right) $.
\begin{equation}
\bm{X}_t \triangleq \left( \bm{z}_{1...t-1}, \bm{b}_{1...t-1} \right).
\end{equation}
Under Assumption~\ref{assumption1} and Assumption~\ref{assumption2}, using Lyapunov process and Quasi-Martingales convergence theorem, L.~Bottou~\cite{bottou1998online} proved
\begin{lemma}
\label{lemma1}
If $\exists A, B >0$ s.t.
\begin{equation}
\mathbf{E}\left\lbrace \left( h_{t+1} - \left( 1+\gamma^2_{t} B \right) h_t \right) | \bm{X}_t \right\rbrace \leq -2 \gamma_{t} (\bm{w}_t - \bm{w}^*)^T \nabla_{\bm{w}} C(\bm{w}_t) + \gamma^2_{t} A,
\end{equation}
%$\mathbf{E}\left\lbrace \left( h_{t+1} - \left( 1+\gamma^2_{t} B \right) h_t \right) | \bm{X}_t \right\rbrace \leq -2 \gamma_{t} (\bm{w}_t - \bm{w}^*)^T \nabla_{\bm{w}} C(\bm{w}_t) + \gamma^2_{t} A$,
then $C(\bm{z}, \bm{w})$ converges \textbf{almost surely} toward minimum $\bm{w}^*$, \ie,
$P\left( \lim_{t \rightarrow +\infty} \bm{w}_t = \bm{w}^*\right) = 1$.
\end{lemma}
%\begin{proof}
% This lemma was proved based on Lyapunov process and Quasi-Martingales convergence theorem in~\cite{bottou1998online}.
%\end{proof}
We further make an assumption on the gradient as
\begin{assumption} [Gradient Bound]
\label{assumption3}
The gradient $\bm{g}$ is bounded as
\begin{equation}
\mathbf{E}\left\lbrace max(abs(\bm{g})) \cdot ||\bm{g}||_1 \right\rbrace \leq A + B \left||\bm{w} - \bm{w}^* |\right|^2 ,
\end{equation}
%$\mathbf{E}\left\lbrace max(abs(\bm{g})) \cdot ||\bm{g}||_1 \right\rbrace \leq A + B \left||\bm{w} - \bm{w}^* |\right|^2$,
where $A, B > 0 $ and $||\cdot||_1$ is $\ell_1$ norm.
\end{assumption}
With Assumption~\ref{assumption3} and Lemma~\ref{lemma1}, we prove Theorem~\ref{theorem1} ( in \textbf{Appendix~\ref{appendixa}}):
\begin{theorem}
\label{theorem1}
When online learning systems update as
\begin{equation}
\bm{w}_{t+1} = \bm{w}_{t} - \gamma_{t} \left( s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right)
\end{equation}
using stochastic ternary gradients, they converge \textbf{almost surely} toward minimum $\bm{w}^*$, \ie, $P\left( \lim_{t \rightarrow +\infty} \bm{w}_t = \bm{w}^*\right) = 1$.
\end{theorem}
Comparing with the gradient bound of standard GOGA~\cite{bottou1998online}
\begin{equation}
\mathbf{E}\left\lbrace||\bm{g}||^2 \right\rbrace \leq A + B \left||\bm{w} - \bm{w}^* |\right|^2 ,
\end{equation}
the bound in Assumption~\ref{assumption3} is stronger because
\begin{equation}
max(abs(\bm{g})) \cdot ||\bm{g}||_1 \geq ||\bm{g}||^2 .
\end{equation}
We propose \textit{layer-wise ternarizing} and \textit{gradient clipping} to make two bounds closer, which shall be explained in Section~\ref{sec:method:practical}.
A side benefit of our work is that, by following the similar proof procedure, we can prove the convergence of GOGA when Gaussian noise $\mathcal{N}(0,\sigma^2)$ is added to gradients~\cite{neelakantan2015adding}, under the gradient bound of
\begin{equation}
\mathbf{E}\left\lbrace||\bm{g}||^2 \right\rbrace \leq A + B \left||\bm{w} - \bm{w}^* |\right|^2 - \sigma^2 .
\end{equation}
Although the bound is also stronger, Gaussian noise encourages active exploration of parameter space and improves accuracy as was empirically studied in~\cite{neelakantan2015adding}.
Similarly, the randomness of ternary gradients also encourages space exploration and improves accuracy for some models, as shall be presented in Section~\ref{sec:exp}.
\subsection{Feasibility Considerations}
\label{sec:method:practical}
%\vspace{-6pt}
%Four figures:
%original gradients, clipped gradients, ternary gradients, final multi-level gradients.
\begin{figure}[b]
\centering
\includegraphics[width=.95\columnwidth]{figures/histograms-crop.pdf}
\caption{Histograms of (a) original floating gradients, (b) clipped gradients, (c) ternary gradients and (d) final averaged gradients. Visualization by TensorBoard. The DNN is \alexnet{} distributed on two workers, and vertical axis is the training iteration. As examples, top row visualizes the third convolutional layer and bottom one visualizes the first fully-connected layer.}
\label{fig:histograms}
%\vspace{-12pt}
\end{figure}
% In this section, some practical techniques are introduced to improve the convergence of \terngrad{}.
The gradient bound of \terngrad{} in Assumption~\ref{assumption3} is stronger than the bound in standard GOGA.
Pushing the two bounds closer can improve the convergence of \terngrad{}.
In Assumption~\ref{assumption3}, $max\left( abs\left( \bm{g} \right) \right)$ is the maximum absolute value of \textit{all} the gradients in the DNN.
So, in a large DNN, $max\left( abs\left( \bm{g} \right) \right)$ could be relatively much larger than most gradients, implying that the bound in \terngrad{} becomes much stronger.
Considering the situation, we propose \textit{layer-wise ternarizing} and \textit{gradient clipping} to reduce $max\left( abs\left( \bm{g} \right) \right)$ and therefore shrink the gap between these two bounds.
\textit{Layer-wise ternarizing} is proposed based on the observation that the range of gradients in each layer changes as gradients are back propagated.
Instead of adopting a large global maximum scaler, we independently ternarize gradients in each layer using the layer-wise scalers.
More specific, we separately ternarize the gradients of biases and weights by using Eq.~(\ref{eq:ternarize}), where $\bm{g}_t$ could be the gradients of biases or weights in each layer.
To approach the standard bound more closely, we can split gradients to more buckets and ternarize each bucket independently as D.~Alistarh~\etal{}~\cite{tomiokaqsgd} does.
However, this will introduce more floating scalers and increase communication. When the size of bucket is one, it degenerates to floating gradients.
Layer-wise ternarizing can shrink the bound gap resulted from the dynamic ranges of the gradients across layers.
However, the dynamic range within a layer still remains as a problem.
We propose \textit{gradient clipping}, which limits the magnitude of each gradient $g_i$ in $\bm{g}$ as
\begin{equation}
\label{eq:clipping}
f(g_i) =
\begin{cases}
g_i & |g_i| \leq c \sigma\\
sign(g_i) \cdot c \sigma & |g_i|>c \sigma
%-c \sigma & g_i<-c \sigma
\end{cases},
\end{equation}
where $\sigma$ is the standard derivation of gradients in $\bm{g}$. %and $c$ is a constant.
In distributed training, gradient clipping is applied to every worker before ternarizing.
$c$ is a hyper-parameter to select, but we cross validate it only once and use the constant in all our experiments.
Specifically, we used a CNN~\cite{Alex_NIPS2012_4824} trained on CIFAR-10 by momentum SGD with staircase learning rate and obtained the optimal $c=2.5$. %, which is a very reasonable value.
Suppose the distribution of gradients is close to Gaussian distribution as shown in Figure~\ref{fig:histograms}(a), very few gradients can drop out of $[-2.5\sigma,2.5\sigma]$.
Clipping these gradients in Figure~\ref{fig:histograms}(b) can significantly reduce the scaler but slightly changes the length and direction of original $\bm{g}$.
Numerical analysis shows that \textit{gradient clipping} with $c=2.5$ only changes the length of $\bm{g}$ by $1.0\%-1.5\%$ and its direction by $2^{\circ}-3^{\circ}$.
%\hl{for any reasonable $\sigma$ and the size of $\bm{g}$. [This sentence is unclear.]}
In our experiments, $c=2.5$ %is stable
remains valid across multiple databases (MNIST, CIFAR-10 and ImageNet), various network structures (\lenet, \cifarnet, \alexnet, \googlenet, \etc) and training schemes (momentum, vanilla SGD, adam, \etc ).
The effectiveness of \textit{layer-wise ternarizing} and \textit{gradient clipping} can also be explained as follows. %from another perspective.
When the scalar $s_t$ in Eq.~(\ref{eq:ternarize}) and Eq.~(\ref{eq:ber_dist_1st}) is very large, most gradients have a high possibility to be ternarized to zeros, leaving only a few gradients to large-magnitude values.
%most gradients have the probabilities of almost $1$ being ternarized to zeros, leaving few gradients ternarized to values with a large magnitude.
The scenario raises a severe parameter update pattern: most parameters keep unchanged while others likely overshoot.
This will introduce large training variance.
Our experiments on \alexnet{} show that by applying both \textit{layer-wise ternarizing} and \textit{gradient clipping} techniques, \terngrad{} can converge to the same accuracy as standard SGD.
Removing any of the two techniques can result in accuracy degradation, \eg{}, $3\%$ top-1 accuracy loss without applying \textit{gradient clipping} as we shall show in Table~\ref{tab:alexnet}.
%For example, \alexnet{} suffers $3\%$ accuracy loss without \textit{gradient clipping}.
\section{Experiments}
\label{sec:exp}
\vspace{-6pt}
We first investigate the convergence of \terngrad{} under various training schemes on relatively small databases and show the results in Section~\ref{sec:exp-small-dataset}.
Then the scalability of \terngrad{} to large-scale distributed deep learning is explored and discussed in Section~\ref{sec:imagenet}.
The experiments are performed by TensorFlow\cite{abadi2016tensorflow}.
We maintain the exponential moving average of parameters by employing an exponential decay of $0.9999$~\cite{pan2017revisiting}.
The accuracy is evaluated by the final averaged parameters.
This gives slightly better accuracy in our experiments.
For fair comparison, in each pair of comparative experiments using either floating or ternary gradients, all the other training hyper-parameters are the same unless differences are explicitly pointed out.
In experiments, when SGD with momentum is adopted, momentum value of $0.9$ is used.
When polynomial decay is applied to decay the \textit{learning rate} (LR), the power of $0.5$ is used to decay LR from the base LR to zero.
\input{exp-small-dataset}
\input{exp-imagenet}
\begin{figure}[b]
\centering
\includegraphics[width=1.0\columnwidth]{figures/lenet-crop.pdf}
\caption{Accuracy vs. worker number for baseline and \terngrad{}, trained with (a) momentum SGD or (b) vanilla SGD. In all experiments, total mini-batch size is $64$ and maximum iteration is $10K$.}
\label{fig:lenet}
\vspace{-6pt}
\end{figure}
\subsection{Integrating with Various Training Schemes}
\label{sec:exp-small-dataset}
\vspace{-6pt}
We study the convergence of \terngrad{} using \lenet{} on MNIST and a ConvNet~\cite{Alex_NIPS2012_4824} (named as \cifarnet{}) on CIFAR-10.
\lenet{} is trained without data augmentation.
While training \cifarnet{}, images are randomly cropped to $24 \times 24$ images and mirrored.
Brightness and contrast are also randomly adjusted.
During the testing of \cifarnet{}, only center crop is used.
Our experiments cover the scope of SGD optimizers over vanilla SGD, SGD with momentum~\cite{qian1999momentum} and Adam~\cite{kingma2014adam}.
%\input{tables/mnist}
Figure~\ref{fig:lenet} shows the results of \lenet{}.
All are trained using polynomial LR decay with weight decay of $0.0005$.
The base learning rates of momentum SGD and vanilla SGD are $0.01$ and $0.1$, respectively.
Given the total mini-batch size $M$ and the worker number $N$, the mini-batch size per worker is $M/N$.
Without explicit mention, mini-batch size refers to the total mini-batch size in this work.
Figure~\ref{fig:lenet} shows that \terngrad{} can converge to the similar accuracy within the same iterations, using momentum SGD or vanilla SGD.
The maximum accuracy gain is $0.15\%$ and the maximum accuracy loss is $0.22\%$.
Very importantly, the communication time per iteration can be reduced.
The figure also shows that \terngrad{} generalizes well to distributed training with large $N$.
No degradation is observed even for $N=64$, which indicates one training sample per iteration per worker.
%Notice that there is accuracy discrepancy by \terngrad{} when distributed over $2$ or $8$ workers. This comes from the discrepancy of gradient statistics $\sigma$ in Eq.~(\ref{eq:clipping}), when distributed across either $2$ or $8$ workers.
\input{tables/cifar10}
Table~\ref{tab:cifar10} summarizes the results of \cifarnet{}, where all trainings terminate after the same epochs.
%\hl{To diversify experimental settings, different training schemes are adopted.}
Adam SGD is used for training.
Instead of keeping total mini-batch size unchanged, we maintain the mini-batch size per worker.
Therefore, the total mini-batch size linearly increases as the number of workers grows.
Though the base learning rate of $0.0002$ seems small, it can achieve better accuracy than larger ones like $0.001$ for baseline.
In each pair of experiments, \terngrad{} can converge to the accuracy level with less than 1\% degradation.
The accuracy degrades under a large mini-batch size in both baseline and \terngrad{}.
This is because parameters are updated less frequently and large-batch training tends to converge to poorer sharp minima~\cite{keskar2016large}.
However, the noise inherent in \terngrad{} can help converge to better flat minimizers~\cite{keskar2016large}, which could explain the smaller accuracy gap between the baseline and \terngrad{} when the mini-batch size is $2048$.
In our experiments of \alexnet{} in Section~\ref{sec:imagenet}, \terngrad{} even improves the accuracy in the large-batch scenario.
This attribute is beneficial for distributed training as a large mini-batch size is usually required.
\begin{table}
\small
\caption{ Results of \terngrad{} on \cifarnet{}.}
\label{tab:cifar10}
\centering
\resizebox{.95\textwidth}{!}{
\begin{tabular}{ccccccl}
\toprule
SGD & base LR & total mini-batch size & iterations & gradients & workers & accuracy\\
\midrule
\multirow{2}{*}{Adam} & \multirow{2}{*}{0.0002} & \multirow{2}{*}{128} & \multirow{2}{*}{300K} & floating & 2 & 86.56\% \\
& & & & \terngrad{} & 2 & 85.64\% (-0.92\%) \\
\cmidrule{1-7}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\multirow{2}{*}{Adam} & \multirow{2}{*}{0.0002} & \multirow{2}{*}{2048} & \multirow{2}{*}{18.75K} & floating & 16 & 83.19\% \\
& & & & \terngrad{} & 16 & 82.80\% (-0.39\%) \\
\bottomrule
\end{tabular}
}
\vspace{-6pt}
\end{table}
\subsection{Scaling to Large-scale Deep Learning}
\label{sec:imagenet}
\vspace{-6pt}
We also evaluate \terngrad{} by \alexnet{} and \googlenet{} trained on ImageNet.
It is more challenging to apply \terngrad{} to large-scale DNNs.
It may result in some accuracy loss when simply replacing the floating gradients with ternary gradients while keeping other hyper-parameters unchanged.
However, we are able to train large-scale DNNs by \terngrad{} successfully after making some or all of the following changes:
(1)~decreasing dropout ratio to keep more neurons;
(2)~using smaller weight decay; and
(3)~disabling ternarizing in the last classification layer.
Dropout can regularize DNNs by adding randomness, while \terngrad{} also introduces randomness.
Thus, dropping fewer neurons helps avoid over-randomness.
Similarly, as the randomness of \terngrad{} introduces regularization, smaller weight decay may be adopted.
% No ternarizing is added to biases in each layer, because its size is small and cannot provide good statistics for Eq.~(\ref{eq:clipping}).
We suggest not to apply ternarizing to the last layer, considering that the one-hot encoding of labels generates a skew distribution of gradients and the symmetric ternary encoding $\{-1,0,1\}$ is not optimal for such a skew distribution.
%No ternarizing is added to the last layer, because one-hot encoding of labels generates a skew distribution of gradients, for which the symmetric ternary encoding $\{-1,0,1\}$ may not be optimal.
Though asymmetric ternary levels could be an option, we decide to stick to floating gradients in the last layer for simplicity.
The overhead of communicating these floating gradients is small, as the last layer occupies only a small percentage of total parameters, like 6.7\% in \alexnet{} and 3.99\% in \textit{ResNet-152}~\cite{Kaiming_ResNet_ICCV}.
%We could select asymmetric ternary levels to encode them, but floating gradients are used in the last layer to make it simple. The overhead of communicating these floating gradients is small, considering the percentage of parameters in the last layer is small. For example, the last layer of \alexnet{} and \textit{ResNet-152}~\cite{Kaiming_ResNet_ICCV} occupies only 6.7\% and 3.99\% parameters, respectively.
All DNNs are trained by momentum SGD with Batch Normalization~\cite{batchnorm_2015} on convolutional layers.
\alexnet{} is trained by the hyper-parameters and data augmentation depicted in Caffe.
\googlenet{} is trained by polynomial LR decay and data augmentation in~\cite{szegedy2016rethinking}.
Our implementation of \googlenet{} does not utilize any auxiliary classifiers, that is, the loss from the last softmax layer is the total loss.
%No auxiliary classifiers are used in our implementation of \googlenet{}, that is, the loss from the last softmax layer is the total loss.
More training hyper-parameters are reported in corresponding tables and published source code. Validation accuracy is evaluated using only the central crops of images.
The results of \alexnet{} are shown in Table~\ref{tab:alexnet}.
Mini-batch size per worker is fixed to $128$.
% so the total mini-batch size linearly increases with the number of workers $N$.
For fast development, all DNNs are trained through the same epochs of images.
In this setting, when there are more workers, the number of iterations becomes smaller and parameters are less frequently updated.
To overcome this problem, we increase the learning rate for large-batch scenario~\cite{li2017scaling}.
Using this scheme, SGD with floating gradients successfully trains \alexnet{} to similar accuracy, for mini-batch size of $256$ and $512$.
However, when mini-batch size is $1024$, the top-1 accuracy drops $0.71\%$ for the same reason as we point out in Section~\ref{sec:exp-small-dataset}.
\input{tables/alexnet}
\begin{figure}[b]
\centering
\vspace{-6pt}
\includegraphics[width=\columnwidth]{figures/alexnet-crop.pdf}
\caption{\alexnet{} trained on $4$ workers with mini-batch size $512$: (a) top-1 validation accuracy, (b) training data loss and (c) sparsity of gradients in first fully-connected layer (fc6) vs. iteration. }
\label{fig:alexnet}
\end{figure}
\terngrad{} converges to approximate accuracy levels regardless of mini-batch size.
Notably, it improves the top-1 accuracy by $0.92\%$ when mini-batch size is $1024$, because its inherent randomness encourages to escape from poorer sharp minima~\cite{neelakantan2015adding}\cite{keskar2016large}.
Figure~\ref{fig:alexnet} plots training details vs. iteration when mini-batch size is $512$.
Figure~\ref{fig:alexnet}(a) shows that the convergence curve of \terngrad{} matches well with the baseline's, demonstrating the effectiveness of \terngrad{}.
The training efficiency can be further improved by reducing communication time as shall be discussed in Section~\ref{sec:discussion}.
The training data loss in Figure~\ref{fig:alexnet}(b) shows that \terngrad{} converges to a slightly lower level, which further proves the capability of \terngrad{} to minimize the target function even with ternary gradients.
A smaller dropout ratio in \terngrad{} can be another reason of the lower loss.
%Though we can approach the loss curve of the baseline by tuning dropout ratio, the lower loss further proves the capability of \terngrad{} to minimize the target function even with ternary gradients.
Figure~\ref{fig:alexnet}(c) simply illustrate that on average $71.32\%$ gradients of a fully-connected layer (fc6) are ternarized to zeros.
%So sparse format like \textit{Compressed Sparse Row} (CSR) could be used to compress gradients.
%Each index of a nonzero in CSR will occupy $log_2 M$ bits, where $M$ is the number of gradients.
%When $M$ is large, CSR is sub-optimal comparing with the simple 2-bit encoder, unless an extremely high sparsity.
\input{tables/googlenet}
Finally, we summarize the results of \googlenet{} in Table~\ref{tab:googlenet}.
On average, the accuracy loss is less than $2\%$.
In \terngrad{}, we adopted all that hyper-parameters (except dropout ratio and weight decay) that are well tuned for the baseline~\cite{GoogleNet_2015}.
%By tuning those hyper-parameters, we could gain a higher accuracy.
Tuning these hyper-parameters specifically for \terngrad{} could further optimize \terngrad{} and obtain higher accuracy.
\begin{table}
\caption{ Accuracy comparison for \alexnet{}.}
\label{tab:alexnet}
\centering
\resizebox{1.\textwidth}{!}{
\begin{tabular}{ccccccccc}
\toprule
base LR & mini-batch size & workers & iterations & gradients & weight decay & DR\textsuperscript{$\dagger$} & top-1 & top-5 \\
\midrule
\multirow{3}{*}{0.01} & \multirow{3}{*}{256} & \multirow{3}{*}{2} & \multirow{3}{*}{370K} & floating & 0.0005 & 0.5 & 57.33\% & 80.56\%\\
& & & & \terngrad{} & 0.0005 & 0.2 & 57.61\% & 80.47\%\\
& & & & \terngrad{}-noclip \textsuperscript{$\ddagger$} & 0.0005 & 0.2 & 54.63\% & 78.16\%\\
\cmidrule{1-9}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\multirow{2}{*}{0.02} & \multirow{2}{*}{512} & \multirow{2}{*}{4} & \multirow{2}{*}{185K} & floating & 0.0005 & 0.5 & 57.32\% & 80.73\%\\
& & & & \terngrad{} & 0.0005 & 0.2 & 57.28\% & 80.23\%\\
\cmidrule{1-9}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\multirow{2}{*}{0.04} & \multirow{2}{*}{1024} & \multirow{2}{*}{8} & \multirow{2}{*}{92.5K} & floating & 0.0005 & 0.5 & \textbf{56.62\%} & 80.28\%\\
& & & & \terngrad{} & 0.0005 & 0.2 & \textbf{57.54\%} & 80.25\%\\
\bottomrule
\multicolumn{8}{l}{\textsuperscript{$\dagger$} DR: dropout ratio, the ratio of dropped neurons. \textsuperscript{$\ddagger$} \terngrad{} without gradient clipping. }
\end{tabular}
}
\vspace{-12pt}
\end{table}
\begin{table}
\caption{ Accuracy comparison for \googlenet{}.}
\label{tab:googlenet}
\centering
\resizebox{.95\textwidth}{!}{
\begin{tabular}{cccccccc}
\toprule
base LR & mini-batch size & workers & iterations & gradients & weight decay & DR & top-5 \\
\midrule
\multirow{2}{*}{0.04} & \multirow{2}{*}{128} & \multirow{2}{*}{2} & \multirow{2}{*}{600K} & floating & 4e-5 & 0.2 & 88.30\% \\
& & & & \terngrad{} & 1e-5 & 0.08 & 86.77\%\\
\cmidrule{1-8}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\multirow{2}{*}{0.08} & \multirow{2}{*}{256} & \multirow{2}{*}{4} & \multirow{2}{*}{300K} & floating & 4e-5 & 0.2 & 87.82\% \\
& & & & \terngrad{} & 1e-5 & 0.08 & 85.96\% \\
\cmidrule{1-8}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\multirow{2}{*}{0.10} & \multirow{2}{*}{512} & \multirow{2}{*}{8} & \multirow{2}{*}{300K} & floating & 4e-5 & 0.2 & 89.00\% \\
& & & & \terngrad{} & 2e-5 & 0.08 & 86.47\% \\
\bottomrule
\end{tabular}
}
%\vspace{-6pt}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\columnwidth]{figures/perf-model.pdf}
\caption{Training throughput on two different GPUs clusters:
(a) 128-node GPU cluster with 1Gbps Ethernet, each node has 4 NVIDIA GTX 1080 GPUs and one PCI switch;
(b) 128-node GPU cluster with 100 Gbps InfiniBand network connections, each node has 4 NVIDIA Tesla P100 GPUs connected via NVLink. Mini-batch size per GPU of \alexnet{}, \googlenet{} and \vggneta{} is $128$, $64$ and $32$, respectively}
\label{fig:perfmodel}
\end{figure}
\section{Performance Model and Discussion}
\label{sec:discussion}
\vspace{-6pt}
Our proposed \terngrad{} requires only three numerical levels $\{-1,0,1\}$, which can aggressively reduce the communication time.
Moreover, our experiments in Section~\ref{sec:exp} demonstrate that within \textit{the same iterations}, \terngrad{} can converge to \textit{approximately the same accuracy} as its corresponding baseline.
Consequently, a dramatical throughput improvement on the distributed DNN training is expected.
%Therefore, we expect a dramatical improvement on the training throughput on distributed GPUs/machines, which shall be evaluated in this section.
%Our experiment results demonstrate \terngrad{} can converge to almost the same accuracy within the \textit{same iterations} in comparison with its baseline. We expect the algorithm to speed up training throughput on distributed GPUs/machines significantly.
Due to the resource and time constraint, unfortunately, we aren't able to perform the training of more DNN models like \vggneta{}~\cite{Vggnet_2014} and distributed training beyond 8 workers.
We plan to continue the experiments in our future work.
We opt for using a performance model to conduct the scalability analysis of DNN models when utilizing up to 512 GPUs, with and without applying \terngrad{}.
Three neural network models---\alexnet{}, \googlenet{} and \vggneta{}---are investigated.
In discussions of performance model, \textit{performance} refers to training speed.
Here, we extend the performance model that was initially developed for CPU-based deep learning systems~\cite{fengKDD15} to estimate the performance of distributed GPUs/machines.
The key idea is combining the lightweight profiling on single machine with analytical modeling for accurate performance estimation.
In the interest of space, please refer to \textbf{Appendix~\ref{appendixb}} for details of the performance model.
%We study and compare the scalability of DNN models when utilizing up to 512 GPUs, with and without applying \terngrad{}.
%Three neural network models---\alexnet{}, \googlenet{} and \vggneta{}---are investigated.
Figure~\ref{fig:perfmodel} presents the training throughput on two different GPUs clusters.
%We assume the weak scaling in this scalability study.
%\hl{The batch size per GPU is consistent with our previous accuracy tests in Section~\ref{sec:exp}.}
%Our results show that \terngrad{} dramatically increases the training throughput for the three DNNs when scaling beyond 8 nodes (64 GPUs) because the amount of communication during the gradient averaging phase is significantly reduced with fewer number of bits per gradient.
Our results show that \terngrad{} effectively increases the training throughput for the three DNNs. The speedup depends on the communication-to-computation ratio of the DNN, the number of GPUs, and the communication bandwidth.
DNNs with larger communication-to-computation ratios (\eg{} \alexnet{} and \vggneta{}) can benefit more from \terngrad{} than those with smaller ratios (\eg{}, \googlenet{}).
%For \vggneta{} with a large communication-to-computation ratio, \terngrad{} can increase the training throughput by $7.5\times$ from $1,642$~images/second to $12,574$~images/second on 128 nodes (512 GPUs).
Even on a very high-end HPC system with InfiniBand and NVLink, \terngrad{} is still able to double the training speed of \vggneta{} on 128 nodes as shown in Figure~\ref{fig:perfmodel}(b).
Moreover, the \terngrad{} becomes more efficient when the bandwidth becomes smaller, such as 1Gbps Ethernet and PCI switch in Figure~\ref{fig:perfmodel}(a) where \terngrad{} can have $3.04 \times$ training speedup for \alexnet{} on $8$ GPUs.
\section{Convergence Analysis of \terngrad{}}
\label{appendixa}
%\begin{theorem}
% \label{theorem1}
% When online learning systems update as $\bm{w}_{t+1} = \bm{w}_{t} - \gamma_{t} \left( s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right)$ using stochastic ternary gradients, they converge \textbf{almost surely} toward minimum $\bm{w}^*$, \ie, $P\left( \lim_{t \rightarrow +\infty} \bm{w}_t = \bm{w}^*\right) = 1$.
%\end{theorem}
Proof of Theorem 1:
\begin{proof}
\begin{equation}
h_{t+1} - h_t = -2 \gamma_{t} (\bm{w}_t - \bm{w}^*)^T \left( s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right)
+ \gamma^2_{t} \left|\left| s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right|\right|^2.
\end{equation}
We have
\begin{equation}
\label{eq:expected_delta_d}
\begin{split}
\mathbf{E}\left\lbrace \left( h_{t+1} - h_t \right) | \bm{X}_t \right\rbrace
& = -2 \gamma_{t} (\bm{w}_t - \bm{w}^*)^T \mathbf{E} \left\lbrace \left( s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right) | \bm{X}_t \right\rbrace \\
& ~~~~~ + \gamma^2_{t} \mathbf{E} \left\lbrace \left. \left|\left| s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right|\right|^2 ~\right| \bm{X}_t \right\rbrace.
\end{split}
\end{equation}
Eq.~(\ref{eq:expected_delta_d}) satisfies based on the fact that $\gamma_{t}$ is deterministic, and $\bm{w}_t$ is also deterministic given $\bm{X}_t$.
According to $\mathbf{E}\left\lbrace s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right\rbrace = \nabla_{\bm{w}} C(\bm{w}_t)$,
\begin{equation}
\begin{split}
& \mathbf{E}\left\lbrace \left( h_{t+1} - h_t \right) | \bm{X}_t \right\rbrace
+ 2 \gamma_{t} \cdot (\bm{w}_t - \bm{w}^*)^T \cdot \nabla_{\bm{w}} C(\bm{w}_t) \\
& = \gamma^2_{t} \cdot \mathbf{E} \left\lbrace \left. \left|\left| s_t \cdot sign\left( \bm{g}_t \right) \circ \bm{b}_t \right|\right|^2 ~\right| \bm{X}_t \right\rbrace \\
& = \gamma^2_{t} \cdot \mathbf{E} \left\lbrace \left. s^2_t \left|\left| \bm{b}_t \right|\right|^2 ~\right| \bm{w}_t \right\rbrace
= \gamma^2_{t} \cdot \mathbf{E} \left\lbrace \left. s^2_t \cdot \mathbf{E} \left\lbrace \left. \left|\left| \bm{b}_t \right|\right|^2 \right| \bm{z}_t,\bm{w}_t \right\rbrace ~\right| \bm{w}_t \right\rbrace \\
& = \gamma^2_{t} \cdot \mathbf{E} \left\lbrace \left. s^2_t \cdot \sum_k \mathbf{E} \left\lbrace \left. b_{tk}^2 \right| \bm{z}_t,\bm{w}_t \right\rbrace ~\right| \bm{w}_t \right\rbrace \\
\end{split}
\end{equation}
Based on the Bernoulli distribution of $b_{tk}$ and Assumption~3, we further have
\begin{equation}
\begin{split}
& \mathbf{E}\left\lbrace \left( h_{t+1} - h_t \right) | \bm{X}_t \right\rbrace
+ 2 \gamma_{t} \cdot (\bm{w}_t - \bm{w}^*)^T \cdot \nabla_{\bm{w}} C(\bm{w}_t) \\
& = \gamma^2_{t} \cdot \mathbf{E} \left\lbrace s_t \left|\left| \bm{g}_t \right|\right|_1 \right\rbrace
= \gamma^2_{t} \cdot \mathbf{E} \left\lbrace max\left( abs\left( \bm{g}_t \right)\right) \cdot \left|\left| \bm{g}_t \right|\right|_1 \right\rbrace \\
& \leq A \gamma^2_{t} + B \gamma^2_{t} \left||\bm{w}_t - \bm{w}^* \right||^2
= A \gamma^2_{t} + B \gamma^2_{t} h_t .
\end{split}
\end{equation}
That is
\begin{equation}
\mathbf{E}\left\lbrace \left( h_{t+1} - \left( 1+\gamma^2_{t} B \right) h_t \right) | \bm{X}_t \right\rbrace \leq -2 \gamma_{t} (\bm{w}_t - \bm{w}^*)^T \nabla_{\bm{w}} C(\bm{w}_t) + \gamma^2_{t} A ,
\end{equation}
which satisfies the condition of Lemma~1 and proves Theorem~1.
The proof can be extended to mini-batch SGD by treating $\bm{z}$ as a mini-batch of observations instead of one observation.
\end{proof}
\section{Performance Model}
\label{appendixb}
%Our experiment results have demonstrated \terngrad{} can converge to similar accuracy within the \textit{same iterations} in comparsion with its baseline. We expect the algorithm to speed up traing throughput on distributed GPUs/machines signficantly. Thus we build a performance model to estimate how much faster we can train different NNs when applying the technique.
%Due to the resource constraint, we opt for using a performance model for conducting scalability analysis.
As mentioned in the main context of our paper, the performance model was developed based on the one initially proposed for CPU-based deep learning systems~\cite{fengKDD15}. We extended it to model GPU-based deep learning systems in this work.
Lightweight profiling is used in the model.
%More specifically, we first profile the training performance on a single worker (\eg, a single machine with a single GPU) and then use the proposed performance model to estimate the training speed when scaling to multiple workers (multiple GPUs and/or multiple machines).
We ran all performance tests with distributed TensorFlow on a cluster of 4 machines, each of which has 4 GTX 1080 GPUs and one Mellanox MT27520 InfiniBand network card.
Our performance model was successfully validated against the measured results by the server cluster we have.
%, \ww{Some data to show that our model is accurate?}.
There are two scaling schemes for distributed training with data parallelism: a) \textit{strong scaling} that spreads the same size problem across multiple workers, and b) \textit{weak scaling} that keeps the size per worker constant when the number of workers increases~\cite{snavely2002framework}. Our performance model supports both scaling models.
We start with strong scaling to illustrate our performance model. According to the definition of strong scaling, here the same size problem is corresponding to the same mini-batch size.
In other words, the more workers, the less training samples per worker. Intuitively, more workers bring more computing resources, meanwhile inducing higher communication overhead.
The goal is to estimate the throughput of a system that uses $j$ machines with $i$ GPUs per machine and mini-batch size of $K$\footnote{For ease of the discussion, we assume symmetric system architecture. The performance model can be easily extended to support heterogeneous system architecture.}. Note the total number of workers equals to the total number of GPUs on all machines, i.e., $N=i*j$. We need to distinguish workers within a machine and across machines due to their different communication patterns.
%we model the impacts in both computation and communication.
Next, we illustrate how to accurately model the impacts in communication and computation to capture both the benefits and overheads.
\noindent {\bf Communication.}
For GPUs within a machine, first, the gradient $\bm{g}$ computed at each GPU needs to be accumulated together.
Here we assume all-reduce communication model, that is, each GPU communicates with its neighbor until all gradient $\bm{g}$ is accumulated into a single GPU. The communication complexity for $i$ GPUs is $log_{2}i$.
The GPU with accumulated gradient then sends the accumulated gradient to CPU for further processing. Note for each communication (either GPU-to-GPU or GPU-to-CPU), the communication data size is the same, i.e., $|\bm{g}|$.
Assume that within a machine, the communication bandwidth between GPUs is $C_{gwd}$\footnote{For ease of the discussion, we assume GPU-to-GPU communication has \textit{Dedicated Bandwidth}.} and the communication bandwidth between CPU and GPU is $C_{cwd}$, then the communication overhead within a machine can be computed as $\frac{|\bm{g}|}{C_{gwd}} * log_{2}i + \frac{|\bm{g}|}{C_{cwd}}$. We successfully used NCCL benchmark to validate our model. For communication between machines, we also assume all-reduce communication model, so the communication time between machines are: $(C_{ncost} + \frac{|\bm{g}|}{C_{nwd}}) * log_{2}j$, where $C_{ncost}$ is the network latency and $C_{nwd}$ is the network bandwidth. So the total communication time is $T_{comm}(i,j,K,|\bm{g}|) = \frac{|\bm{g}|}{C_{gwd}} * log_{2}i + \frac{|\bm{g}|}{C_{cwd}} + (C_{ncost} + \frac{|\bm{g}|}{C_{nwd}}) * log_{2}j$. We successfully used OSU Allreduce benchmark to validate this model.
\noindent {\bf Computation.}
To estimate computation time, we rely on profiling the time for training a mini-batch of totally $K$ images on a machine with a single CPU and a single GPU. We define this profiled time as $T(1,1,K,|\bm{g}|)$. In strong scaling, each work only trains $\frac{K}{N}$ samples, so the total computation time is $T_{comp}(i,j,K,|\bm{g}|) = (T(1,1,K,|\bm{g}|) - \frac{|\bm{g}|}{C_{cwd}}) * \frac{1}{N}$, where $\frac{|\bm{g}|}{C_{cwd}}$ is the communication time (between GPU and CPU) included in when we profile $T(1,1,K,|\bm{g}|)$.
% (marked as $T(1,1,K,|\bm{g}|)$)
Therefore, the time to train a mini-batch of $K$ samples is:
\begin{equation}
\begin{split}
T_{strong}(i,j,K,|\bm{g}|) & = T_{comp}(i,j,K,|\bm{g}|) + T_{comm}(i,j,K,|\bm{g}|) \\
& = (T(1,1,K,|\bm{g}|) - \frac{|\bm{g}|}{C_{cwd}}) * \frac{1}{N} \\
&~~ + \frac{|\bm{g}|}{C_{gwd}} * log_{2}i + \frac{|\bm{g}|}{C_{cwd}} + (C_{ncost} + \frac{|\bm{g}|}{C_{nwd}}) * log_{2}j.
\end{split}
\end{equation}
The throughput of strong scaling is:
\begin{equation}
Tput_{strong}(i,j,K,|\bm{g}|) = \frac{K}{T_{strong}(i,j,K,|\bm{g}|)}.
\end{equation}
For weak scaling, the difference is that each worker always trains $K$ samples. So the mini-batch size becomes $N*K$.
In the interest of space, we do not present the detailed reasoning here.
Basically, it follows the same logic for developing the performance model of strong scaling. We can compute the time to train a mini-batch of $N*K$ samples as follows:
\begin{equation}
\begin{split}
T_{weak}(i,j,K,|\bm{g}|) & = T_{comp}(i,j,K,|\bm{g}|) + T_{comm}(i,j,K,|\bm{g}|) \\
& = T(1,1,K,|\bm{g}|) - \frac{|\bm{g}|}{C_{cwd}} + \frac{|\bm{g}|}{C_{gwd}} * log_{2}i + \frac{|\bm{g}|}{C_{cwd}} + (C_{ncost} + \frac{|\bm{g}|}{C_{nwd}}) * log_{2}j \\
& = T(1,1,K,|\bm{g}|) + \frac{|\bm{g}|}{C_{gwd}} * log_{2}i + (C_{ncost} + \frac{|\bm{g}|}{C_{nwd}}) * log_{2}j.
\end{split}
\end{equation}
% + \frac{|\bm{g}|}{C_{cwd}}
So the throughput of weak scaling is:
\begin{equation}
Tput_{weak}(i,j,K,|\bm{g}|) = \frac{N*K}{T_{weak}(i,j,K,|\bm{g}|)}.
\end{equation}
\begin{comment}
The input of the model:
\begin{itemize}
\item The profiled throughput of using a single GPU $Tput(1,1,K)$.
\item The communication bandwidth $C_cwd$ between CPU and GPU.
\item The communication bandwidth $C_gwd$ between GPUs for system with multiple GPUs on a single machine.
\item The network bandwidth $C_nwd$ for multiple machines system.
\end{itemize}
The output of the model is the time for training one mini batch $T(i,j,K)$.
\end{comment}
%In order to balance the accuracy and complexity of the performance model, we rely on both simple profiling and modeling.
%In order to make accurate estimation
%1. GPU parallel
%Initial experiments on Async???
%initial experiments on LSTM and Linear Regression???
\begin{thebibliography}{45}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Dean et~al.(2012)Dean, Corrado, Monga, Chen, Devin, Mao, aurelio
Ranzato, Senior, Tucker, Yang, Le, and Ng]{distbelief}
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao,
Marc\textquotesingle aurelio Ranzato, Andrew Senior, Paul Tucker, Ke~Yang,
Quoc~V. Le, and Andrew~Y. Ng.
\newblock Large scale distributed deep networks.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
1223--1231. 2012.
\bibitem[Abadi et~al.(2016)Abadi, Agarwal, Barham, Brevdo, Chen, Citro,
Corrado, Davis, Dean, Devin, et~al.]{abadi2016tensorflow}
Mart{\'\i}n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen,
Craig Citro, Greg~S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et~al.
\newblock Tensorflow: Large-scale machine learning on heterogeneous distributed
systems.
\newblock \emph{arXiv preprint:1603.04467}, 2016.
\bibitem[Coates et~al.(2013)Coates, Huval, Wang, Wu, Catanzaro, and
Andrew]{coates2013deep}
Adam Coates, Brody Huval, Tao Wang, David Wu, Bryan Catanzaro, and Ng~Andrew.
\newblock Deep learning with cots hpc systems.
\newblock In \emph{International Conference on Machine Learning}, pages
1337--1345, 2013.
\bibitem[Recht et~al.(2011)Recht, Re, Wright, and Niu]{recht2011hogwild}
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu.
\newblock Hogwild: A lock-free approach to parallelizing stochastic gradient
descent.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
693--701, 2011.
\bibitem[Chilimbi et~al.(2014)Chilimbi, Suzue, Apacible, and
Kalyanaraman]{chilimbi2014project}
Trishul~M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman.
\newblock Project adam: Building an efficient and scalable deep learning
training system.
\newblock In \emph{OSDI}, volume~14, pages 571--582, 2014.
\bibitem[Xing et~al.(2015)Xing, Ho, Dai, Kim, Wei, Lee, Zheng, Xie, Kumar, and
Yu]{xing2015petuum}
Eric~P Xing, Qirong Ho, Wei Dai, Jin~Kyu Kim, Jinliang Wei, Seunghak Lee, Xun
Zheng, Pengtao Xie, Abhimanu Kumar, and Yaoliang Yu.
\newblock Petuum: A new platform for distributed machine learning on big data.
\newblock \emph{IEEE Transactions on Big Data}, 1\penalty0 (2):\penalty0
49--67, 2015.
\bibitem[Moritz et~al.(2015)Moritz, Nishihara, Stoica, and
Jordan]{moritz2015sparknet}
Philipp Moritz, Robert Nishihara, Ion Stoica, and Michael~I Jordan.
\newblock Sparknet: Training deep networks in spark.
\newblock \emph{arXiv preprint:1511.06051}, 2015.
\bibitem[Chen et~al.(2015)Chen, Li, Li, Lin, Wang, Wang, Xiao, Xu, Zhang, and
Zhang]{chen2015mxnet}
Tianqi Chen, Mu~Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao,
Bing Xu, Chiyuan Zhang, and Zheng Zhang.
\newblock Mxnet: A flexible and efficient machine learning library for
heterogeneous distributed systems.
\newblock \emph{arXiv preprint:1512.01274}, 2015.
\bibitem[Zhang et~al.(2015)Zhang, Choromanska, and LeCun]{zhang2015deep}
Sixin Zhang, Anna~E Choromanska, and Yann LeCun.
\newblock Deep learning with elastic averaging sgd.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
685--693, 2015.
\bibitem[Li(2017)]{li2017scaling}
Mu~Li.
\newblock \emph{Scaling Distributed Machine Learning with System and Algorithm
Co-design}.
\newblock PhD thesis, Carnegie Mellon University, 2017.
\bibitem[Li et~al.(2014{\natexlab{a}})Li, Andersen, Park, Smola, Ahmed,
Josifovski, Long, Shekita, and Su]{li2014scaling}
Mu~Li, David~G Andersen, Jun~Woo Park, Alexander~J Smola, Amr Ahmed, Vanja
Josifovski, James Long, Eugene~J Shekita, and Bor-Yiing Su.
\newblock Scaling distributed machine learning with the parameter server.
\newblock In \emph{OSDI}, volume~14, pages 583--598, 2014{\natexlab{a}}.
\bibitem[Li et~al.(2014{\natexlab{b}})Li, Andersen, Smola, and
Yu]{li2014communication}
Mu~Li, David~G Andersen, Alexander~J Smola, and Kai Yu.
\newblock Communication efficient distributed machine learning with the
parameter server.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
19--27, 2014{\natexlab{b}}.
\bibitem[Ho et~al.(2013)Ho, Cipar, Cui, Lee, Kim, Gibbons, Gibson, Ganger, and
Xing]{ho2013more}
Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin~Kyu Kim, Phillip~B
Gibbons, Garth~A Gibson, Greg Ganger, and Eric~P Xing.
\newblock More effective distributed ml via a stale synchronous parallel
parameter server.
\newblock In \emph{Advances in neural information processing systems}, pages
1223--1231, 2013.
\bibitem[Zinkevich et~al.(2010)Zinkevich, Weimer, Li, and
Smola]{zinkevich2010parallelized}
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex~J Smola.
\newblock Parallelized stochastic gradient descent.
\newblock In \emph{Advances in neural information processing systems}, pages
2595--2603, 2010.
\bibitem[Pan et~al.(2017)Pan, Chen, Monga, Bengio, and
Jozefowicz]{pan2017revisiting}
Xinghao Pan, Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Jozefowicz.
\newblock Revisiting distributed synchronous sgd.
\newblock \emph{arXiv preprint:1702.05800}, 2017.
\bibitem[Zhang et~al.(2016)Zhang, Gupta, Lian, and Liu]{Zhang2016Staleness}
Wei Zhang, Suyog Gupta, Xiangru Lian, and Ji~Liu.
\newblock Staleness-aware async-sgd for distributed deep learning.
\newblock In \emph{Proceedings of the Twenty-Fifth International Joint
Conference on Artificial Intelligence}, IJCAI'16, pages 2350--2356. AAAI
Press, 2016.
\newblock ISBN 978-1-57735-770-4.
\newblock URL \url{http://dl.acm.org/citation.cfm?id=3060832.3060950}.
\bibitem[Han et~al.(2015)Han, Mao, and Dally]{han2015deep}
Song Han, Huizi Mao, and William~J Dally.
\newblock Deep compression: Compressing deep neural networks with pruning,
trained quantization and huffman coding.
\newblock \emph{arXiv preprint arXiv:1510.00149}, 2015.
\bibitem[Wen et~al.(2016)Wen, Wu, Wang, Chen, and Li]{wen2016learning}
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li.
\newblock Learning structured sparsity in deep neural networks.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
2074--2082, 2016.
\bibitem[Park et~al.(2017)Park, Li, Wen, Tang, Li, Chen, and
Dubey]{park2017faster}
J~Park, S~Li, W~Wen, PTP Tang, H~Li, Y~Chen, and P~Dubey.
\newblock Faster cnns with direct sparse convolutions and guided pruning.
\newblock In \emph{International Conference on Learning Representations
(ICLR)}, 2017.
\bibitem[Hubara et~al.(2016)Hubara, Courbariaux, Soudry, El-Yaniv, and
Bengio]{hubara2016binarized}
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua
Bengio.
\newblock Binarized neural networks.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
4107--4115, 2016.
\bibitem[Rastegari et~al.(2016)Rastegari, Ordonez, Redmon, and
Farhadi]{rastegari2016xnor}
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.
\newblock Xnor-net: Imagenet classification using binary convolutional neural
networks.
\newblock In \emph{European Conference on Computer Vision}, pages 525--542.
Springer, 2016.
\bibitem[Zhou et~al.(2016)Zhou, Wu, Ni, Zhou, Wen, and Zou]{zhou2016dorefa}
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He~Wen, and Yuheng Zou.
\newblock Dorefa-net: Training low bitwidth convolutional neural networks with
low bitwidth gradients.
\newblock \emph{arXiv preprint arXiv:1606.06160}, 2016.
\bibitem[Wen et~al.(2017)Wen, He, Rajbhandari, Wang, Liu, Hu, Chen, and
Li]{wen2017iclrlearning}
Wei Wen, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Fang Liu, Bin Hu, Yiran
Chen, and Hai Li.
\newblock Learning intrinsic sparse structures within long short-term memory.
\newblock \emph{arXiv:1709.05027}, 2017.
\bibitem[Ott et~al.(2016)Ott, Lin, Zhang, Liu, and Bengio]{ott2016recurrent}
Joachim Ott, Zhouhan Lin, Ying Zhang, Shih-Chii Liu, and Yoshua Bengio.
\newblock Recurrent neural networks with limited numerical precision.
\newblock \emph{arXiv:1608.06902}, 2016.
\bibitem[Lin et~al.(2015)Lin, Courbariaux, Memisevic, and
Bengio]{lin2015neural}
Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio.
\newblock Neural networks with few multiplications.
\newblock \emph{arXiv:1510.03009}, 2015.
\bibitem[Bradley et~al.(2011)Bradley, Kyrola, Bickson, and
Guestrin]{bradley2011parallel}
Joseph~K Bradley, Aapo Kyrola, Danny Bickson, and Carlos Guestrin.
\newblock Parallel coordinate descent for l1-regularized loss minimization.
\newblock \emph{arXiv preprint arXiv:1105.5379}, 2011.
\bibitem[Aji and Heafield(2017)]{aji2017sparse}
Alham~Fikri Aji and Kenneth Heafield.
\newblock Sparse communication for distributed gradient descent.
\newblock \emph{arXiv preprint:1704.05021}, 2017.
\bibitem[Seide et~al.(2014)Seide, Fu, Droppo, Li, and Yu]{seide20141}
Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu.
\newblock 1-bit stochastic gradient descent and its application to
data-parallel distributed training of speech dnns.
\newblock In \emph{Interspeech}, pages 1058--1062, 2014.
\bibitem[Alistarh et~al.(2017)Alistarh, Grubic, Li, Tomioka, and
Vojnovic]{tomiokaqsgd}
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic.
\newblock Qsgd: Communication-efficient sgd via gradient quantization and
encoding.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
1707--1718, 2017.
\bibitem[Gupta et~al.(2015)Gupta, Agrawal, Gopalakrishnan, and
Narayanan]{gupta2015deep}
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan.
\newblock Deep learning with limited numerical precision.
\newblock In \emph{ICML}, pages 1737--1746, 2015.
\bibitem[Garg and Khandekar(2009)]{garg2009gradient}
Rahul Garg and Rohit Khandekar.
\newblock Gradient descent with sparsification: an iterative algorithm for
sparse recovery with restricted isometry property.
\newblock In \emph{Proceedings of the 26th Annual International Conference on
Machine Learning}, pages 337--344. ACM, 2009.
\bibitem[Suresh et~al.(2016)Suresh, Yu, McMahan, and
Kumar]{suresh2016distributed}
Ananda~Theertha Suresh, Felix~X Yu, H~Brendan McMahan, and Sanjiv Kumar.
\newblock Distributed mean estimation with limited communication.
\newblock \emph{arXiv:1611.00429}, 2016.
\bibitem[Bottou(1998)]{bottou1998online}
L{\'e}on Bottou.
\newblock Online learning and stochastic approximations.
\newblock \emph{On-line learning in neural networks}, 17\penalty0 (9):\penalty0
142, 1998.
\bibitem[Neelakantan et~al.(2015)Neelakantan, Vilnis, Le, Sutskever, Kaiser,
Kurach, and Martens]{neelakantan2015adding}
Arvind Neelakantan, Luke Vilnis, Quoc~V Le, Ilya Sutskever, Lukasz Kaiser,
Karol Kurach, and James Martens.
\newblock Adding gradient noise improves learning for very deep networks.
\newblock \emph{arXiv preprint:1511.06807}, 2015.
\bibitem[Krizhevsky et~al.(2012)Krizhevsky, Sutskever, and
Hinton]{Alex_NIPS2012_4824}
Alex Krizhevsky, Ilya Sutskever, and Geoffrey~E. Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
1097--1105. 2012.
\bibitem[Qian(1999)]{qian1999momentum}
Ning Qian.
\newblock On the momentum term in gradient descent learning algorithms.
\newblock \emph{Neural networks}, 12\penalty0 (1):\penalty0 145--151, 1999.
\bibitem[Kingma and Ba(2014)]{kingma2014adam}
Diederik Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{arXiv preprint:1412.6980}, 2014.
\bibitem[Keskar et~al.(2017)Keskar, Mudigere, Nocedal, Smelyanskiy, and
Tang]{keskar2016large}
Nitish~Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy,
and Ping Tak~Peter Tang.
\newblock On large-batch training for deep learning: Generalization gap and
sharp minima.
\newblock In \emph{International Conference on Learning Representations}, 2017.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{Kaiming_ResNet_ICCV}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 770--778, 2016.
\bibitem[Ioffe and Szegedy(2015)]{batchnorm_2015}
Sergey Ioffe and Christian Szegedy.
\newblock Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
\newblock \emph{arXiv preprint:1502.03167}, 2015.
\bibitem[Szegedy et~al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and
Wojna]{szegedy2016rethinking}
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Wojna.
\newblock Rethinking the inception architecture for computer vision.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 2818--2826, 2016.
\bibitem[Szegedy et~al.(2015)Szegedy, Liu, Jia, Sermanet, Reed, Anguelov,
Erhan, Vanhoucke, and Rabinovich]{GoogleNet_2015}
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir
Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.
\newblock Going deeper with convolutions.
\newblock \emph{arXiv preprint:1409.4842}, 2015.
\bibitem[Simonyan and Zisserman(2014)]{Vggnet_2014}
Karen Simonyan and Andrew Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock \emph{arXiv preprint:1409.1556}, 2014.
\bibitem[Yan et~al.(2015)Yan, Ruwase, He, and Chilimbi]{fengKDD15}
Feng Yan, Olatunji Ruwase, Yuxiong He, and Trishul~M. Chilimbi.
\newblock Performance modeling and scalability optimization of distributed deep
learning systems.
\newblock In \emph{Proceedings of the 21th {ACM} {SIGKDD} International
Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia,
August 10-13, 2015}, pages 1355--1364, 2015.
\newblock \doi{10.1145/2783258.2783270}.
\newblock URL \url{http://doi.acm.org/10.1145/2783258.2783270}.
\bibitem[Snavely et~al.(2002)Snavely, Carrington, Wolter, Labarta, Badia, and
Purkayastha]{snavely2002framework}
Allan Snavely, Laura Carrington, Nicole Wolter, Jesus Labarta, Rosa Badia, and
Avi Purkayastha.
\newblock A framework for performance modeling and prediction.
\newblock In \emph{Supercomputing, ACM/IEEE 2002 Conference}, pages 21--21.
IEEE, 2002.
\end{thebibliography}