On Higher-order Moments in Adam
Extracted source text ยท arXiv 1910.06878
\pdfoutput=1
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2019
% ready for submission
% \usepackage{neurips_2019}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2019}
% to compile a camera-ready version, add the [final] option, e.g.:
\usepackage[final,nonatbib]{neurips_2019}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2019}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{amsmath}
\usepackage{graphicx,color}
\usepackage{epsfig}
\usepackage{amsmath,amsthm,amssymb}
\usepackage{multirow}
\newtheorem{thm}{Theorem}
\newtheorem{assumption}{Assumption}
\newtheorem{corollary}{Corollary}
\newtheorem{lem}{Lemma}
\newtheorem{proposition}{Proposition}
%\newtheorem{proof}{Proof}
%\theoremstyle{plain}
\newtheorem{defn}{Definition}
\newtheorem{example}{Example}
\theoremstyle{remark}
\newtheorem{rem}{Remark}
\newtheorem{notation}{Notation}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\x}{\mathbf{x}}
\newcommand{\f}{\mathbf{f}}
\newcommand{\F}{\mathbf{F}}
\newcommand{\w}{\mathbf{w}}
\newcommand{\E}{\mathbb{E}}
\graphicspath{{figures/}}
\newcommand{\chin}[1]{\textcolor{red}{\bf (#1)}}
\usepackage{subcaption}
\usepackage{algorithm}
\usepackage{algorithmic}
\title{On Higher-order Moments in Adam}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Zhanhong Jiang\\
Johnson Controls International\\
\texttt{zhanhong.jiang@jci.com}
\And
Aditya Balu\\
Iowa State University\\
\texttt{baditya@iastate.edu} \\
\And
Sin Yong Tan\\
Iowa State University\\
\texttt{tsyong98@iastate.edu}\\
\And
Young M Lee \\
Johnson Controls International\\
\texttt{young.m.lee@jci.com}\\
\And
Chinmay Hegde \\
New York University \\
\texttt{chinmay@iastate.edu}
\\
\And
Soumik Sarkar\\
Iowa State University\\
\texttt{soumiks@iastate.edu} \\
}
\begin{document}
\maketitle
\begin{abstract}
In this paper, we investigate the popular deep learning optimization routine, Adam, from the perspective of statistical moments. While Adam is an adaptive lower-order moment based (of the stochastic gradient) method, we propose an extension namely, HAdam, which uses higher order moments of the stochastic gradient. Our analysis and experiments reveal that certain higher-order moments of the stochastic gradient are able to achieve better performance compared to the vanilla Adam algorithm. We also provide some analysis of HAdam related to odd and even moments to explain some intriguing and seemingly non-intuitive empirical results. %We also design a new method by using the second central moment (instead of the second raw moment) so as to take into account the exponentially weighted moving variance for the adaptive learning rate in the updates. Both analytically and empirically, the new method performs similarly compared to Adam.
\end{abstract}
\section{Introduction}\label{intro}
Adaptive gradient descent methods have become very popular in the optimization methods for machine learning, especially deep learning. Adagrad~\cite{duchi2011adaptive}, Adadelta~\cite{zeiler2012adadelta}, RMSprop~\cite{tieleman2012lecture}, and Adam~\cite{kingma2014adam} are the most popular optimizers that perform quite well in practice and hence have received considerable attention. While moments of stochastic gradients play a critical role in these optimizers, so far, only the first and second raw moments have been used in practice. To the best of our knowledge, the use of higher-order moments have not been reported for use in the aforementioned adaptive gradient descent methods; these could be beneficial for speeding up training of deep neural networks.
This paper presents some interesting findings observed from using Adam with higher order moments. Although comprehensive theoretical analysis is not presented in this context, interesting findings arise. We hope these findings could provide some useful insights for the community to improve the understanding of these adaptive gradient descent methods and to develop more robust optimizers. Further, we show, some experimental evidences of better performance of the higher-order moments. This further reveals an interesting finding that odd-order moments lead to the divergence of Adam. It should also be noted that the higher-order moments can be applied to other adaptive gradient descent methods which have the similar form as Adam.
\section{Preliminaries}
\label{prelim}
Deep learning problems can be cast as optimization problems, which mostly are represented by an empirical risk minimization (ERM) as follows:
\begin{equation}
\textnormal{min}\frac{1}{n}\sum^n_{i=1}f^i(x)
\end{equation}
where $n$ is the cardinality of a dataset, $f$ is the mapping from $\mathcal{R}^d\to\mathcal{R}$, and $x\in\mathcal{R}^d$ signifies the weight to be learned from data. Due to the broad applicability of ERM, %$f$ has different properties for different applications.
for deep learning, $f$ is generically assumed to be non-convex and smooth, which is followed in this study.
Before we investigate the algorithm for minimizing $f$, we define formally several key concepts.
\begin{defn}\label{defn1}
The $k$-th moment of a real-valued continuous function $h(y)$ of a real variable $y$ about a value $c$ is defined as
\begin{equation}\label{k_moment}
p_k=\int_{-\infty}^{\infty}(y-c)^kh(y)dy
\end{equation}
\end{defn}
Assume that $Y$ is a random variable. If $h(y)$ is a probability density function and $c=0$, then the $k$-th moment is the expected value of $Y^k$ and called a \textit{raw} moment.
% Based on the above two definitions, it can be obtained that the moments about its mean $\mu$ are called central moment, which will be a key definition in the next section.
Thus, Eq.~\ref{k_moment} can be converted into
\begin{equation}\label{moment_zero}
p'_k=\mathbb{E}(Y^k)=\int_{-\infty}^{\infty}y^kdH(y)
\end{equation}
where $H(y)$ is the cumulative probability distribution function. The first raw moment is the mean, denoted by $\mu=\mathbb{E}(Y)$. We now introduce an approximation of any $k$-th raw moment that could be useful in a discrete manner.
\begin{defn}\label{defn3}
$\frac{1}{N}\sum^N_{j=1}Y^k_j$ is defined as the $k$-th raw sample moment, which can be used to estimate the $k$-th raw moment shown in Eq.~\ref{moment_zero} for all $k$, where $N$ is the number of samples drawn from a population.
\end{defn}
With the preliminary tools required, we now present the core update law for Adam:
\begin{align}
m_t &= \beta_1m_{t-1} + (1-\beta_1)g_t\\
V_t &= \beta_2V_{t-1} + (1-\beta_2)g^2_t\\
x_t &= x_{t-1} - \alpha\frac{\sqrt{1-\beta_2^t}}{1-\beta_1^t}\frac{m_t}{\sqrt{V_t}+\epsilon}
\label{eqn:adam}
\end{align}
where $\beta_1, \beta_2\in[0,1)$, $g_t$ is the stochastic gradient of $f$ at $x_t$, $\alpha$ is the step size, and $\epsilon$ is chosen as a small constant.
There has been a large body of work in the literature that investigates the convergence of Adam as well as how adaptive learning rate benefits the convergence. We refer the interested readers to~\cite{ruder2016overview,zhou2018convergence} for more details.
\section{High-order Moment in Adam}
\label{high}
Using the core update law of Adam as shown in Eq. \ref{eqn:adam}, it can be rewritten using induction as following (assuming that $m_0$ and $V_0$ are both initialized as 0 in this context for simplicity.):
\begin{align}
m_t &= (1-\beta_1)\sum^{t-1}_{i=0}\beta_1^ig_{t-i}\\
V_t &= (1-\beta_2)\sum^{t-1}_{i=0}\beta_2^ig_{t-i}^2\label{eqn:induction}
\end{align}
% We now connect the last two equations with their corresponding raw sample moments.
% Define
% $$\hat{m}_t=\frac{1}{t}\sum^{t-1}_{i=0}g_{t-i},\;\hat{V}_t=\frac{1}{t}\sum^{t-1}_{i=0}g^2_{t-i}$$ as the first and second raw sample moments of stochastic gradient $g$.
It can be observed that $m_t$ and $V_t$ are different from Definition~\ref{defn3} due to the usage of exponential moving average. In~\cite{kingma2014adam}, the authors called $m_t$ and $V_t$ biased raw sample moments and the third equation of Adam is able to correct the bias correspondingly. In this paper, we extend the second moment to high-order moment. The following algorithmic framework summarizes the High-order Adam (HAdam).
\begin{algorithm}[h]
\caption{HAdam}
\begin{algorithmic}[1]\label{hadam}
\STATE {\textbf{Input}: $m_0=0, V_0=0, x_0, \alpha, \epsilon, \beta_1, \beta_2, k=2d, d=\{1,2,...\}$}
\FOR{$t=1:M$}
\STATE {Calculate the stochastic gradient $g_t$}
\STATE {$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t$}
\STATE {$V_t = \beta_2 V_{t-1} + (1-\beta_2) g^k_t$}
\STATE{$x_t=x_{t-1} - \alpha\frac{\sqrt[k]{1-\beta_2^t}}{1-\beta_1^t}\frac{m_t}{\sqrt[k]{V_t}+\epsilon}$}
\ENDFOR
\end{algorithmic}
\end{algorithm}
For analysis, we assume that $\epsilon=0$. In HAdam, Line 6 signifies the update law of $x_t$ which includes the bias correction, as done in \cite{kingma2014adam}. Using Equation~\ref{eqn:induction} and updating them for higher-order moments, we get
\begin{equation}\label{biased_update}
\mathbb{E}[V_t] =\mathbb{E}[ (1-\beta_2)\sum^{t-1}_{i=0}\beta_2^ig^k_{t-i}]=\mathbb{E}[g^k_t](1-\beta_2)\sum^{t-1}_{i=0}\beta_2^i+\zeta=\mathbb{E}[g^k_t](1-\beta^t_2)+\zeta,
\end{equation}
where $\zeta$ is the non-stationarity caused by the true $k$-th raw moment $\mathbb{E}[g^k_t]$. Essentially, $\zeta=0$ if $\mathbb{E}[g^k_t]$ is stationary or $\zeta$ can be arbitrarily small when $\beta_1$ is manually chosen such that stochastic gradients in the past are assigned with small weights. By dividing by the term $1-\beta_2^t$ on both sides in Eq.~\ref{biased_update}, the initialization biases associated with $V_t$ can be corrected. The effective step taken at time instant $t$ is $\Delta x_t=\alpha \frac{\hat{m}_t}{\sqrt[k]{\hat{V}_t}}$, where $\hat{m}_t = \frac{m_t}{1-\beta_1^t}$ and $\hat{V}_t = \frac{V_t}{1-\beta_2^t}$. It has been illustrated in \cite{kingma2014adam} that the effective magnitude of steps for each time instant taken can be approximately bounded by the stepsize, i.e., $|\Delta x_t|\lessapprox \alpha$ when the order is equal to 2, which corresponds to Adam. This bound mainly follows from $|\mathbb{E}[g]/\sqrt{\mathbb{E}[g^2]}|\leq 1$.
We next investigate the third and fourth raw moments of the HAdam (i.e. $k = 3,4$) to obtain some insights. A detailed analysis of generic order moments will be deferred to an extended version of this work.
The approach to study the third and fourth raw moments is to check the metric as follows
\begin{equation}
\mathcal{M}_k = |\mathbb{E}[g]/\sqrt[k]{\mathbb{E}[g^k]}|
\end{equation}
In HAdam, we would also like to see if the bound $|\Delta x_t|\lessapprox \alpha$ holds true for higher orders.% such that the statistical analysis here may provide a rough answer to the question that why the odd-even orders behave significantly different in the aforementioned experiment results.
\textbf{Fourth raw moment:} The metric defined earlier can be represented as $\mathcal{M}_4 = |\mathbb{E}[g]/\sqrt[4]{\mathbb{E}[g^4]}|$. As
\begin{equation}
\mathbb{E}[g^4]=\mathbb{E}[(g^2)^2]=(\mathbb{E}[g^2])^2+Var(g^2),
\end{equation}
where $Var(\cdot)$ is the variance operator, then the following can be obtained
\begin{equation}
\mathbb{E}[g^4]=(Var(g))^2+2Var(g)(\mathbb{E}[g])^2+Var(g^2)+(\mathbb{E}[g])^4\geq (\mathbb{E}[g])^4
\end{equation}
The first equality follows from that $\mathbb{E}[g^2] = (\mathbb{E}[g])^2 + Var(g)$. Therefore, for the fourth raw moment, the metric $\mathcal{M}_k$ still holds true.
\textbf{Third raw moment:} A concept of \textit{skewness} is adopted in this context to help characterize the analysis. Denote by $\gamma$, the skewness of the stochastic gradients, which can be expressed as follows:
\begin{equation}
\gamma = \frac{\mathbb{E}[g^3]-3\mathbb{E}[g]Var(g)-(\mathbb{E}[g])^3}{(\sqrt{Var(g)})^3},
\end{equation}
which yields that $\mathbb{E}[g^3]=\gamma(\sqrt{Var(g)})^3+3\mathbb{E}[g]Var(g)+(\mathbb{E}[g])^3$. The skewness value of stochastic gradients can be negative, positive, or undefined \cite{doane2011measuring}, and therefore the condition $\mathcal{M}_3\leq 1$ is not rigorously guaranteed.
When $\mathcal{M}_3\gg 1$, the effective step taken could be significantly large, i.e., $|\Delta x_t|\gg \alpha$, resulting in the divergence. The upper bound for the effective step can also be understood as establishing a \textit{trust region} around the parameter value $x_t$ at time instant $t$ and the parameter space beyond this region cannot benefit from the current stochastic gradient. For HAdam with arbitrary even-orders, we can generalize the above analysis from the fourth raw moment such that $\mathcal{M}_k\leq 1, k=2d, d=\{1,2,...\}$. Thus, when $k\gg 2$, we have $|\mathbb{E}[g]/\sqrt[k]{\mathbb{E}[g^k]}\ll|\mathbb{E}[g]/\sqrt{\mathbb{E}[g^2]}|\leq 1$. %However, from the experimental results, we have observed that when the even order is much larger than 2, the convergence to a certain stationary point is extremely slow such that it performs as similar as one odd order.
% \section{Second Central Moment}
% \label{scm}
% Either for Adam or HAdam, they all adopt raw moments in their respective update laws. In this section, we propose using the second central moment but maintain a similar analytical form of Adam. Before discussing this variant, we present the algorithmic framework.
% \begin{algorithm}[htbp]
% \caption{Adam with second central moment Version 1 (this may be removed)}
% \begin{algorithmic}[1]\label{adam_cm}
% \STATE {\textbf{Input}: $m_0=0, V_0=0, Q_0=0, x_0, \alpha, \epsilon, \beta_1, \beta_2$}
% \FOR{$t=1:M$}
% \STATE {Calculate the stochastic gradient $g_t$}
% \STATE{$\delta_t=g_t - m_{t-1}$}
% \STATE {$m_t = m_{t-1} + \beta_1 \delta_t$}
% \STATE {$Q_t = (1-\beta_1)(Q_{t-1} + \beta_1 \delta^2_t)$}
% \STATE {$V_t = \beta_2V_{t-1}+(1-\beta_2)g^2_t$}
% \STATE{$x_t=x_{t-1} - \alpha\frac{\sqrt{1-\beta^t_2}}{1-\beta_1^t}\frac{m_t}{\sqrt{V_t}+\sqrt{Q_t(1-\beta_2^t)}+\epsilon}$}
% \ENDFOR
% \end{algorithmic}
% \end{algorithm}
% \begin{algorithm}[htbp]
% \caption{Adam with second central moment Version 2}
% \begin{algorithmic}[1]\label{adam_cm}
% \STATE {\textbf{Input}: $m_0=0, Q_0=0, x_0, \alpha, \epsilon, \beta_1$}
% \FOR{$t=1:M$}
% \STATE {Calculate the stochastic gradient $g_t$}
% \STATE{$\delta_t=g_t - m_{t-1}$}
% \STATE {$m_t = m_{t-1} + \beta_1 \delta_t$}
% \STATE {$Q_t = (1-\beta_1)(Q_{t-1} + \beta_1 \delta^2_t)$}
% %\STATE {$V_t = \beta_2V_{t-1}+(1-\beta_2)g^2_t$}
% \STATE{$x_t=x_{t-1} - \alpha\frac{1}{\sqrt{1-\beta_1^t}}\frac{m_t}{\sqrt{(1-\beta_1^t)m_t^2+Q_t}+\epsilon}$}
% \ENDFOR
% \end{algorithmic}
% \end{algorithm}
% Different from Adam, Algorithm~\ref{adam_cm} instead uses the second central moment for $Q_t$ as $\delta_t$ represents the difference between the current stochastic gradient $g_t$ and its previous exponential moving average $m_{t-1}$. Lines 4 and 5 in Algorithm~\ref{adam_cm} are equal to the biased first raw moment in Adam. The significant difference is Line 6 for $V_t$, which is updated by using the \textit{exponentially weighted moving variance}.
% To correct the bias in the first raw moment, $m_t$ is divided by $1-\beta_1^t$, while for $V_t$, it is divided by $1-\beta_2^t$. In this context, we do not do the same thing to the new term $Q_t$ as it denotes the exponentially weighted moving variance that can be used to obtain the exponentially weighted moving standard deviation, which is $\sqrt{Q_t}$. During training, this can be helpful as the information of a statistical significance of a deviation from the mean is provided to ``force" the stochastic gradient to vary less drastically.
% % Also, based on the fact that $\mathbb{E}[g^2] = \mathbb{E}[g]^2 + Var[g]$, an approximation for the second raw moment is conducted for the Algorithm~\ref{adam_cm} in the following $$\bigg(\frac{m_t}{1-\beta_1^t}\bigg)^2+\frac{V_t}{1-\beta^t_1}.$$
% We still assume that $\epsilon=0$. Therefore, the step taken at each time instant $$\Delta x_t = \alpha\frac{\frac{m_t}{1-\beta_1^t}}{\sqrt{\frac{V_t}{1-\beta_2^t}}+\sqrt{Q_t}}=\alpha\frac{\hat{m}_t}{\sqrt{\hat{V}_t}+\sqrt{Q_t(1-\beta_2^t)}}.$$ It can be suggested that with $\Delta x_t$ above, the analysis for Adam can still be applicable here as $|\mathbb{E}[g]/(\sqrt{\mathbb{E}[g^2]}+a)\leq 1$ when $a\geq 0$. The convergence analysis for this algorithm is left as a future work.
\section{Numerical Results}\label{exp}
In this section, we show some preliminary results of HAdam on the CIFAR-10 dataset. We use a simple network architecture: we use two convolutional layers followed by a max pooling layer, a dropout layer, a dense layer, another dropout layer, and the final dense layer. The total number of learnable parameters is roughly 2,100,000. The default step size of $0.001$ is used and the number of epochs is 50.
\begin{table}
\caption{HAdam performance with different orders on Cifar 10 with 5 epochs}
\label{hadam_cifar}
\centering
\resizebox{0.95\textwidth}{!}{
\begin{tabular}{llllllllll}
\toprule
% \multicolumn{2}{c}{Part} \\
% \cmidrule(r){1-2}
Order & 2&3&4&5&6&7&8&9 \\
\midrule
Training Loss& 1.0323&nan&0.9508 &nan&0.9684&nan&0.9201&nan \\
Training Accuracy& 0.6343& 0.1000&0.6632 &0.1000&0.6588&0.1000&0.6761&0.1000 \\
Test Loss& 1.0451&nan&0.9512&nan&0.9518&nan&0.9254&nan \\
Test Accuracy& 0.6306 &0.1000&0.6705&0.1000&0.6703&0.1000&\textbf{0.6812}&0.1000 \\
\bottomrule
\end{tabular}
}
\end{table}
Table~\ref{hadam_cifar} shows the performance of HAdam on the CIFAR-10 dataset with different orders of moment. The training and testing performance after 5 epochs is reported. We observe that using odd-order moments in HAdam results in divergence of the algorithm, and that even-order moments have better performance than regular Adam. Since, higher-order raw sample moments should have played similar roles either for the odd or even moments, this interesting finding is somewhat counter-intuitive. However, the results could be explained from the analysis done for HAdam(order=3). At the same time, the even higher orders performs better.
%the harder it is to estimate, as evidently shown in Table~\ref{hadam_mnist_1} in the supplementary materials. More results for higher order moments on MNIST and CIFAR 10 can be found in the supplementary materials.
Figure~\ref{cifar_exp} depicts the performance comparison between HAdam and Adam. Two interesting observations can be made from Figure~\ref{cifar_exp}. The first one is in terms of training: HAdam performs better than Adam, which may be attributed to a better adaptive learning rate when the order is higher than 2. The second interesting finding is that in HAdam, higher order moments can lead to more overfitting, but still with quite similar testing performance as Adam. To summarize, HAdam allows for faster training and achieves quite similar testing performance compared with Adam. Beyond existing works, validation on models with more complex architectures will be addressed in future work.
% \begin{table}
% \caption{HAdam performance with different orders on MNIST with 5 epochs}
% \label{hadam_mnist}
% \centering
% \begin{tabular}{llllllllllll}
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% Order & 2&3&4&5&6&7&8&9 \\
% \midrule
% Tra. Los.& 0.0454&nan&0.0463 &nan&0.0574&nan&0.0699&nan \\
% Tra. Acc.& 0.9857& 0.0987&0.9857 &0.0987&0.9822&0.0987&0.9788&0.0987 \\
% Tes. Los.& 0.0315&nan&0.0274&nan&0.0332&nan&0.0357&nan \\
% Tes. Acc.& 0.9896 &0.0980&\textbf{0.9911}&0.0980&0.9891&0.0980&0.9877&0.0980 \\
% \bottomrule
% \end{tabular}
% \end{table}
% \begin{figure}[h!]
% \includegraphics[width=0.9\textwidth]{Figures/MNIST.png}
% \centering
% \caption{Performance comparison between HAdam and Adam on MNIST}
% \label{mnist_exp}
% \end{figure}
\begin{figure}[h!]
\includegraphics[width=1.1\textwidth, trim={1.7in, 0.5in, 0.0in, 1.0in}, clip]{Figures/CIFAR_10.png}
\centering
\caption{Performance comparison between HAdam and Adam on Cifar 10}
\label{cifar_exp}
\end{figure}
% \begin{figure}[h!]
% \includegraphics[width=0.9\textwidth]{Figures/CIFAR_10_1.png}
% \centering
% \caption{Performance comparison of Adam with and without central moment on CIFAR 10}
% \label{cifar_exp_1}
% \end{figure}
\section{Conclusions}\label{conclusion}
This paper investigated the choice of moments in the Adam optimizer. Specifically, we found that odd-order moments could result in divergence of Adam, while even-order moment achieve better or similar performance as Adam. The convergence is faster and the training loss is better, but generalization could be worse than regular Adam. The results are also aligned with insights from our preliminary analysis which suggest why even-order moments could have convergence, and why odd-order moments could result in divergence.
\bibliographystyle{unsrt}
\bibliography{adam}
\clearpage
% \section*{Supplementary Materials}\label{supp}
% \subsection*{Additional Results}
% \begin{table}
% \caption{HAdam performance with different orders on MNIST with 5 epochs}
% \label{hadam_mnist_1}
% \centering
% \begin{tabular}{llllllllllll}
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% Order & 10&11&12&13&14&15&16&17 \\
% \midrule
% Tra. Los.& 14.2840&nan&14.5947 &nan&14.4542&nan&14.3005&nan \\
% Tra. Acc.& 0.1138& 0.0987&0.0945 &0.0987&0.1032&0.0987&0.11288&0.0987 \\
% Tes. Los.& 14.4273&nan&14.6804&nan&14.4547&nan&14.5595&nan \\
% Tes. Acc.& 0.1049 &0.0980&0.0892&0.0980&0.1032&0.0980&0.0967&0.0980 \\
% \bottomrule
% \end{tabular}
% \end{table}
% \begin{table}
% \caption{HAdam performance with different orders on CIFAR 10 with 5 epochs}
% \label{hadam_cifar}
% \centering
% \begin{tabular}{llllllllllll}
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% Order & 10&11&12&13&14&15&16&17 \\
% \midrule
% Tra. Los.& 14.4927&nan&14.5272 &nan&14.4673&nan&14.5343&nan \\
% Tra. Acc.& 0.0998& 0.1000&0.0987 &0.1000&0.1024&0.1000&0.0983&0.1000 \\
% Tes. Los.& 14.5063&nan&14.5063&nan&14.5063&nan&14.4047&nan \\
% Tes. Acc.& 0.1000 &0.1000&0.1000&0.1000&0.1000&0.1000&0.1063&0.1000 \\
% \bottomrule
% \end{tabular}
% \end{table}
% \begin{table}
% \caption{HAdam performance with different orders on CIFAR 10 with 100 epochs in ResNet\_v1}
% \label{hadam_cifar}
% \centering
% \begin{tabular}{ccccccccc}
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% Order & 2 & 4 & 6 & 8 & 10 & 12 & 14 & 16 \\
% \midrule
% Tra. Los.& 0.1253 & 0.1457 & 0.1364 & 0.1348 & 0.2788 & 1.1625 & 274.1689 & 3571.5247 \\
% Tra. Acc.& 100 & 100 & 100 & 100 & 100 & 79.03 & 21.19 & 16.05 \\
% Tes. Los.& 1.2462 & 1.3992 & 1.554 & 1.6725 & 2.0994 & 1.5161 & 243.1689 & 3568.4024 \\
% Tes. Acc.& 80.77 & 78.06 & 76.2 & 74.2 & 71.11 & 67.61 & 10 & 16.566 \\
% \bottomrule
% \end{tabular}
% \end{table}
% \begin{table}
% \caption{HAdam performance with different orders on CIFAR 10 with 100 epochs in ResNet\_v2}
% \label{hadam_cifar}
% \centering
% \begin{tabular}{ccccccccc}
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% Order & 2 & 4 & 6 & 8 & 10 & 12 & 14 & 16 \\
% \midrule
% Tra. Los.& 0.1549 & 0.2092 & 0.2312 & 0.2371 & 0.4113 & 5.5038 & 102.043 & 203.52 \\
% Tra. Acc.& 100 & 100 & 100 & 100 & 99.95 & 52.75 & 18.61 & 19.01 \\
% Tes. Los.& 1.1471 & 1.2604 & 1.469 & 1.5691 & 2.6718 & 6.7286 & 96.885 & 203.52 \\
% Tes. Acc.& 82.33 & 79.76 & 77.47 & 76.62 & 64.45 & 32.03 & 15.38 & 17.77 \\
% \bottomrule
% \end{tabular}
% \end{table}
\end{document}
\begin{thebibliography}{1}
\bibitem{duchi2011adaptive}
John Duchi, Elad Hazan, and Yoram Singer.
\newblock Adaptive subgradient methods for online learning and stochastic
optimization.
\newblock {\em Journal of Machine Learning Research}, 12(Jul):2121--2159, 2011.
\bibitem{zeiler2012adadelta}
Matthew~D Zeiler.
\newblock Adadelta: an adaptive learning rate method.
\newblock {\em arXiv preprint arXiv:1212.5701}, 2012.
\bibitem{tieleman2012lecture}
Tijmen Tieleman and Geoffrey Hinton.
\newblock Lecture 6.5-rmsprop: Divide the gradient by a running average of its
recent magnitude.
\newblock {\em COURSERA: Neural networks for machine learning}, 4(2):26--31,
2012.
\bibitem{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}, 2014.
\bibitem{ruder2016overview}
Sebastian Ruder.
\newblock An overview of gradient descent optimization algorithms.
\newblock {\em arXiv preprint arXiv:1609.04747}, 2016.
\bibitem{zhou2018convergence}
Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu.
\newblock On the convergence of adaptive gradient methods for nonconvex
optimization.
\newblock {\em arXiv preprint arXiv:1808.05671}, 2018.
\bibitem{doane2011measuring}
David~P Doane and Lori~E Seward.
\newblock Measuring skewness: a forgotten statistic?
\newblock {\em Journal of statistics education}, 19(2), 2011.
\end{thebibliography}
\begin{thebibliography}{1}
\bibitem{duchi2011adaptive}
John Duchi, Elad Hazan, and Yoram Singer.
\newblock Adaptive subgradient methods for online learning and stochastic
optimization.
\newblock {\em Journal of Machine Learning Research}, 12(Jul):2121--2159, 2011.
\bibitem{zeiler2012adadelta}
Matthew~D Zeiler.
\newblock Adadelta: an adaptive learning rate method.
\newblock {\em arXiv preprint arXiv:1212.5701}, 2012.
\bibitem{tieleman2012lecture}
Tijmen Tieleman and Geoffrey Hinton.
\newblock Lecture 6.5-rmsprop: Divide the gradient by a running average of its
recent magnitude.
\newblock {\em COURSERA: Neural networks for machine learning}, 4(2):26--31,
2012.
\bibitem{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}, 2014.
\bibitem{ruder2016overview}
Sebastian Ruder.
\newblock An overview of gradient descent optimization algorithms.
\newblock {\em arXiv preprint arXiv:1609.04747}, 2016.
\bibitem{zhou2018convergence}
Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu.
\newblock On the convergence of adaptive gradient methods for nonconvex
optimization.
\newblock {\em arXiv preprint arXiv:1808.05671}, 2018.
\bibitem{doane2011measuring}
David~P Doane and Lori~E Seward.
\newblock Measuring skewness: a forgotten statistic?
\newblock {\em Journal of statistics education}, 19(2), 2011.
\end{thebibliography}