Improving Resnet-9 Generalization Trained on Small Datasets
Extracted source text · arXiv 2309.03965
\documentclass{article} % For LaTeX2e
\usepackage{iclr2021_conference,times}
\usepackage{graphicx} % Required for including images
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\def\fp{\buildrel \rm{+}\over f}
\usepackage{tabularx}
\newcommand{\w}{\mathbf{w}}
\newcommand{\x}{\mathbf{x}}
%\DeclareMathOperator*{\argmax}{argmax~} % for subscripts and superscrits
%\DeclareMathOperator*{\argmin}{argmin~} % for subscripts and superscrits
%\DeclareMathOperator{\sign}{sign} % for functions
\usepackage{subfig}
\usepackage{hyperref}
\usepackage{url}
\title{Improving ResNet-9 Generalization Trained on Small Datasets}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
% \author{Omar Mohamed Awad,
% % \thanks{The job was done while working at Huawei TRC Ascend Team},
% Habib Hajimolahoseini\thanks{Same contribution as the first author in writing the paper}, Michael Lim, Gurpreet Gosal, Walid Ahmed, Yang Liu \\
% Huawei TRC Ascend Team
% % \texttt{\{hippo,brain,jen\}@cs.cranberry-lemon.edu} \\
% Gordon Deng \\
% % Huawei TRC Ascend Team \\
% % University of the Witwatersrand \\
% % Joburg, South Africa \\
% % \texttt{\{robot,net\}@wits.ac.za} \\
% % \AND
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email}
% }
\author{\textbf{Omar Mohamed Awad}\thanks{Corresponding author} , \textbf{Habib Hajimolahoseini}\thanks{Same contribution as the first author in writing the paper} , \textbf{Michael Lim}, \textbf{Gurpreet Gosal}, \\
\textbf{Walid Ahmed}, \textbf{Yang Liu}, \textbf{Gordon Deng} \\
Ascend Team, Toronto Research Center, Huawei Technologies\\
\texttt{omar.mohamedawad@huawei.com}\\
\texttt{habib.hajimolahoseini@huawei.com}
% \texttt{\{omar, gordon\}@cerebras.net}\\
% \texttt{\{habib.hajimolahoseini, michael.lim, gurpreet.gosal\}@huawei.com}\\
% \texttt{\{walid.ahmed1, yang.liu8\}@huawei.com}
}
%\author{Omar Mohamed Awad\thanks{The job was done while working at Huawei TRC Ascend Team}\\
% Huawei Technologies \\ Toronto Research Center \\ Ascend Team
% \And Habib Hajimolahoseini\thanks{Same contribution as the first author in writing the paper} \\
% Huawei TRC Ascend Team \And Michael Lim \\
% Huawei TRC Ascend Team \And Gurpreet Gosal \\
% Huawei TRC Ascend Team \And Walid Ahmed \\
% Huawei TRC Ascend Team \And Yang Liu \\
% Huawei TRC Ascend Team
% % \texttt{\{hippo,brain,jen\}@cs.cranberry-lemon.edu} \\
% \And
% Gordon Deng \\
% Huawei TRC Ascend Team }
% University of the Witwatersrand \\
% Joburg, South Africa \\
% \texttt{\{robot,net\}@wits.ac.za} \\
% \AND
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email}
% }
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
This paper presents our proposed approach that won the first prize at the ICLR competition "Hardware Aware Efficient Training".
The challenge is to achieve the highest possible accuracy in
an image classification task in less than 10 minutes.
The training is done on a small dataset of 5000 images picked randomly from CIFAR-10 dataset.
The evaluation is performed by the competition organizers on a secret dataset with 1000 images of the same size.
Our approach includes applying a series of technique for improving the generalization of ResNet-9 including: sharpness aware optimization, label smoothing, gradient centralization, input patch whitening as well as meta-learning based training.
Our experiments show that the ResNet-9 can achieve the accuracy of 88\% while trained only on a 10\% subset of CIFAR-10 dataset in less than 10 minuets.
\end{abstract}
\section{Introduction}
Deep learning models have become larger and larger over the past few years which leads to models with huge numbers of training parameters.
Training such a huge model requires a large amount of memory and computational power \cite{hajimolahoseinistrategies, hajimolahoseinicompressing,walid2022training,walid2022speeding} as well as big datasets.
However, for some applications, especially on the edge devices with online learning capabilities, the memory and training time could be limited and a large dataset may not be available too \cite{li2021short, hajimolahoseini2023methods, hajimolahoseini2019deep}.
In cases like this, the model is highly prone to the issue of overfitting and may also not be able to generalize well.
Hence, having a model which is able to learn fast on a small dataset and generalize well is highly beneficial \cite{hajimolahoseini2018ecg}.
There are some techniques proposed in the literature which try to improve the training efficiency using different approaches during training \cite{yong2020gradient, foret2020sharpness, zhang2018shufflenet}.
For a detailed review of related work the reader is referred to \cite{coleman2017dawnbench}.
In this competition, the goal is to reach to the highest possible accuracy in an image classification task in less than 10 minuets.
The training is performed on a 10\% subset of CIFAR10 dataset including 10 classes, each with 500 training samples and 100 test samples
of $32\times32$ pixels RGB images \cite{krizhevsky2009learning}.
The evaluation
is performed on mini-ImageNet dataset (hidden
at development time) of size similar to the development
dataset \cite{deng2009imagenet}.
No pre-trained weights are allowed so the models are trained from scratch.
More details about the proposed methodology is presented in the following sections.
\section{Methodology}
\subsection{Baseline model}
Inspired by most of the recent approaches in the similar competitions \cite{coleman2017dawnbench}, we adopt the well-known ResNet-9 architecture \cite{he2016deep} as our baseline.
Due to its relatively small size, it could be beneficial in preventing the over-fitting issue.
However, for training in such a short time on a small dataset, some strategies need to be applied in order to increase the accuracy of the model and its generalization ability.
% \vspace*{-1cm}
% \begin{figure}[!h]
% \begin{center}
% \includegraphics[angle=0,scale=0.5]{figs/resnet9.png}
% \end{center}
% \caption{ResNet-9 Architecture \textcolor{red}{Copied}}
% \label{fig:resnet9}
% \end{figure}
\subsection{Optimizer}
During the training of the baseline model, we observed that the generalization of first-order optimizers such as SGD was sub-par, since the test accuracy of ResNet-9 trained on the 10\% of CIFAR10 dataet could not reach higher than 76.65\%.
Therefore, a Sharpness Aware Minimization (SAM) technique is employed to improve the generalization of the model \cite{foret2020sharpness}.
For models with a very large capacity, the value of training loss does not necessarily guarantee the ability of the model to generalize well \cite{foret2020sharpness}.
In contrast, SAM optimizer minimizes both the value and sharpness of the loss function at the same time.
In other words, it looks for parameters in the surrounding area that have a uniformly low loss.
This method converts the minimization problem into a min-max optimization on which SGD could be performed more efficiently.
Instead of looking for parameter values that show a low training loss, SAM searches for those parameter values whose entire surrounding area has a uniformly low training loss.
It means that the neighborhoods that have both low loss and low curvature \cite{foret2020sharpness}.
% The SAM optimizer algorithms is described in Algorithm 1.
% \begin{figure}[h]
% \begin{center}
% \includegraphics[scale=0.5]{figs/sam.png}
% \end{center}
% \caption{SAM algo. \textcolor{red}{Copied}}
% \label{fig:sam}
% \end{figure}
\subsection{Gradient Centralization}
% As shown in Algorithm 2,
Another technique that can help improving the generalization of small models is Gradient Centralization (GC) \cite{yong2020gradient}.
It centralizes the gradients so that they have zero mean.
This is done by creating a projected gradient descent method with a constrained loss function.
By regularizing the solution space of model parameters, GC helps to reduce the possibility of overfitting on training data and improving generalization of trained models, especially for small datasets.
% \begin{figure}[h]
% \begin{center}
% \includegraphics[width=\textwidth]{figs/gc_alg.png}
% \end{center}
% \caption{GC algo. \textcolor{red}{Copied}}
% \label{fig:gc}
% \end{figure}
\subsection{Improved Preprocessing}
We use a set of techniques called Improved Preprocessing (IP). The methods are described in the following sections:
\subsubsection{Label Smoothing}
In order to improve our training strategy, we use label smoothing technique \cite{pereyra2017regularizing}.
It blends the one-hot target probabilities with a uniform distribution over class labels inside the cross entropy loss as follow:
\begin{equation}\label{eq:labelsmoothing}
y_{ls} = (1-\alpha) \times y_{hot} + \frac{\alpha}{K}
\end{equation}
where $y_{ls}$ and $y_{hot}$ represent the label smoothed and one-hot probabilities and $K$ and $\alpha$ are the number of classes ($K=10$ for our case) and smoothing factor, respectively.
In our experiments, we set the smoothing factor $\alpha=0.1$ and number of classes are $K=10$.
This helps to stabilize the output distribution and prevents the network from making overconfident predictions which might inhibit further training.
\subsubsection{Weight Decay}
Weight decay is another regularization method that we use which keeps the model weights as small as possible:
\begin{equation}\label{eq:weight_decay}
% \tag{3}
L_{new}(w) = L_{org}(w) + \lambda{w^{T}w}
\end{equation}
where $L_{new}(w)$ and $L_{org}(w)$ are the new and original loss functions and $\lambda$ is the decay factor which we set to $\lambda = 0.0005$..
The small values of weights guarantees that the network behaviour will not change much if we change a few random inputs which in turn makes it difficult for the regularized network to learn local noise in the data.
\subsubsection{Activation Function}
We replaced the RELU activation function of ResNet-9 with CELU \cite{barron2017continuously} which helps improving the generalization since smoothed functions lead to a less expressive function class:
\begin{equation}\label{eq:celu}
\begin{split}
\text{CELU}(x,\alpha) = \begin{Bmatrix} x & x \geq 0 \\
\alpha (\exp(\frac{x}{\alpha})-1) & \text{otherwise} \end{Bmatrix}\end{split}
\end{equation}
% Fig.\ref{fig:celu} and Fig.\ref{fig:relu} show the difference between RELU and CELU activation functions.
According to \ref{eq:celu}, CELU converges to ReLU as $\alpha \to 0$.
In our experiments we set $\alpha = 0.3$.
% \begin{figure}[h]
% \centering
% \subfloat[CELU]{\includegraphics[scale=0.8]{figs/celu.png}\label{fig:celu}}
% \hfill
% \subfloat[ReLU]{\includegraphics[scale=0.8]{figs/relu.png}\label{fig:relu}}
% \caption{CELU activation function compared to RELU.}
% \end{figure}
\subsubsection{Input Patch Whitening}
In order to reduce the covariance between channels and pixels we apply PCA whitening to $3{\times}3$ patches of inputs as an initial $3{\times}3$ convolution with fixed (non-trainable) weights followed by a trainable $1{\times}1$ convolution \cite{jegou2012negative}.
\subsection{Meta-learning based Training}
As a mechanism for generalizing the knowledge learned over many few-shot learning tasks, meta-learning is a promising training approach for few-shot learning problems.
A collection of different (but similar) few-shot learning tasks are learned in parallel to learn representations that are common to all tasks (e.g. Omniglot dataset).
This meta-learning approach has been reframed as a single-task algorithm for training on small dataset (10 classes of Mini-ImageNet) – named Meta-Learning based Training Procedure (MLTP).
MLTP was applied in hopes to get good generalization from a small dataset, and ultimately higher classification accuracy.
In this approach, the Mini-ImageNet dataset is broken up into 2 tasks (each containing 250 samples of each class), and applied as a 2-task MLTP problem. The base architecture used is ResNet9.
This process is depicted in Fig.\ref{fig:metalearning1}.
\begin{figure}[h]
\begin{center}
\includegraphics[width=\textwidth]{figs/metalearning1.png}
\end{center}
\caption{Meta-learning Algorithm}
\label{fig:metalearning1}
\end{figure}
% \begin{figure}[h]
% \begin{center}
% \includegraphics[scale=0.6]{figs/metalearning2.png}
% \end{center}
% \caption{Meta-learning Algorithm}
% \label{fig:metalearning2}
% \end{figure}
\section{Evaluation}
The hardware used in this competition is an Nvidia V100 GPU which has a memory of 32GB, running with an Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz processor, with a RAM of 12GB.
We randomly pick 5000 images from the CIFAR-10 dataset (500 images from each of the 10 classes) for training.
The experimental results are shown in Table \ref{tbl:table}. According to this table, although each of the SAM, GC and meta-learning techniques can improve the accuracy of the baseline model individually, however,
combining the SAM optimizer and the IP techniques (SAM+IP) can boost the model accuracy by 9.95\% and 7.7\% on Mini-CIFAR10 and Mini-ImageNet datasets, respectively when comparing to the baseline model.
\begin{table}[htb]
\centering\footnotesize
\caption{Accuracy \& Performance Results}
\label{tbl:table}
\begin{tabular}{|c|c|c|c|c|}
\hline
\textbf{Model} & \multicolumn{2}{|c|}{\textbf{Test Accuracy (\%)}} & \multicolumn{2}{|c|}{\textbf{Epochs (total runtime within 10 minutes)}} \\ \hline
& Mini-CIFAR10 & Mini-ImageNet & Mini-CIFAR10 & Mini-ImageNet \\ \hline
Baseline & 76.65 & 80.3 & 168 & 159 \\ \hline
SAM & 79.9 & 83.25 & 151 & 161 \\ \hline
\textbf{SAM+IP} & \textbf{86.6} & \textbf{88} & \textbf{ 200} & \textbf{200} \\ \hline
SAM+GC & 83.15 & 84.2 & 99 & 102 \\ \hline
Meta-learning & 82.87 & 85.04 & 132 & 136 \\ \hline
\end{tabular}
\end{table}
\section{Conclusion}
In this paper, we adopted some techniques for improving the generalization of ResNet-9 model when training on a small dataset in a very short time.
Experimental results reveal that thechniques such as SAM, GC, IP and meta learning can boost the performance of the baseline model especially when they are combined together.
It is also worth mentioning that the proposed methods are orthogonal to each other and therefore they can be combined together.
Due to time limitation of the competition, we did not experiment with all different combinations of the proposed methods. This is left for future work.
% \subsubsection*{Author Contributions}
% If you'd like to, you may include a section for author contributions as is done
% in many journals. This is optional and at the discretion of the authors.
% \subsubsection*{Acknowledgments}
% Use unnumbered third level headings for the acknowledgments. All
% acknowledgments, including those to funding agencies, go at the end of the paper.
\bibliography{iclr2021_conference}
\bibliographystyle{iclr2021_conference}
% \appendix
% \section{Appendix}
% You may include other additional sections here.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{18}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ahmed et~al.(2022)Ahmed, Hajimolahoseini, Wen, and Liu]{walid2022speeding}
Walid Ahmed, Habib Hajimolahoseini, Austin Wen, and Yang Liu.
\newblock Speeding up resnet architecture with layers targeted low rank decomposition.
\newblock In \emph{Edge Intelligence Workshop}, 2022.
\bibitem[Barron(2017)]{barron2017continuously}
Jonathan~T Barron.
\newblock Continuously differentiable exponential linear units.
\newblock \emph{arXiv preprint arXiv:1704.07483}, 2017.
\bibitem[Coleman et~al.(2017)Coleman, Narayanan, Kang, Zhao, Zhang, Nardi, Bailis, Olukotun, R{\'e}, and Zaharia]{coleman2017dawnbench}
Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris R{\'e}, and Matei Zaharia.
\newblock Dawnbench: An end-to-end deep learning benchmark and competition.
\newblock \emph{Training}, 100\penalty0 (101):\penalty0 102, 2017.
\bibitem[Deng et~al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei]{deng2009imagenet}
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li~Fei-Fei.
\newblock Imagenet: A large-scale hierarchical image database.
\newblock In \emph{2009 IEEE conference on computer vision and pattern recognition}, pp.\ 248--255. Ieee, 2009.
\bibitem[Foret et~al.(2020)Foret, Kleiner, Mobahi, and Neyshabur]{foret2020sharpness}
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.
\newblock Sharpness-aware minimization for efficiently improving generalization.
\newblock \emph{arXiv preprint arXiv:2010.01412}, 2020.
\bibitem[Hajimolahoseini et~al.()Hajimolahoseini, Rezagholizadeh, Partovinia, Tahaei, Awad, and Liu]{hajimolahoseinicompressing}
Habib Hajimolahoseini, Mehdi Rezagholizadeh, Vahid Partovinia, Marzieh Tahaei, Omar~Mohamed Awad, and Yang Liu.
\newblock Compressing pre-trained language models using progressive low rank decomposition.
\bibitem[Hajimolahoseini et~al.(2018)Hajimolahoseini, Hashemi, and Redfearn]{hajimolahoseini2018ecg}
Habib Hajimolahoseini, Javad Hashemi, and Damian Redfearn.
\newblock Ecg delineation for qt interval analysis using an unsupervised learning method.
\newblock In \emph{2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pp.\ 2541--2545. IEEE, 2018.
\bibitem[Hajimolahoseini et~al.(2019)Hajimolahoseini, Redfearn, and Krahn]{hajimolahoseini2019deep}
Habib Hajimolahoseini, Damian Redfearn, and Andrew Krahn.
\newblock A deep learning approach for diagnosing long qt syndrome without measuring qt interval.
\newblock In \emph{Advances in Artificial Intelligence: 32nd Canadian Conference on Artificial Intelligence, Canadian AI 2019, Kingston, ON, Canada, May 28--31, 2019, Proceedings 32}, pp.\ 440--445. Springer, 2019.
\bibitem[Hajimolahoseini et~al.(2022)Hajimolahoseini, Ahmed, Rezagholizadeh, Partovinia, and Liu]{hajimolahoseinistrategies}
Habib Hajimolahoseini, Walid Ahmed, Mehdi Rezagholizadeh, Vahid Partovinia, and Yang Liu.
\newblock Strategies for applying low rank decomposition to transformer-based models.
\newblock In \emph{36th Conference on Neural Information Processing Systems (NeurIPS2022)}, 2022.
\bibitem[Hajimolahoseini et~al.(2023)Hajimolahoseini, Kumar, and Gordon]{hajimolahoseini2023methods}
Habib Hajimolahoseini, Kaushal Kumar, and DENG Gordon.
\newblock Methods, systems, and media for computer vision using 2d convolution of 4d video data tensors, April~20 2023.
\newblock US Patent App. 17/502,588.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pp.\ 770--778, 2016.
\bibitem[J{\'e}gou \& Chum(2012)J{\'e}gou and Chum]{jegou2012negative}
Herv{\'e} J{\'e}gou and Ond{\v{r}}ej Chum.
\newblock Negative evidences and co-occurences in image retrieval: The benefit of pca and whitening.
\newblock In \emph{European conference on computer vision}, pp.\ 774--787. Springer, 2012.
\bibitem[Krizhevsky et~al.(2009)Krizhevsky, Hinton, et~al.]{krizhevsky2009learning}
Alex Krizhevsky, Geoffrey Hinton, et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock 2009.
\bibitem[Li et~al.(2021)Li, Mesbahi, Kobyzev, Rashid, Mahmud, Anchuri, Hajimolahoseini, Liu, and Rezagholizadeh]{li2021short}
Tianda Li, Yassir~El Mesbahi, Ivan Kobyzev, Ahmad Rashid, Atif Mahmud, Nithin Anchuri, Habib Hajimolahoseini, Yang Liu, and Mehdi Rezagholizadeh.
\newblock A short study on compressing decoder-based language models.
\newblock \emph{arXiv preprint arXiv:2110.08460}, 2021.
\bibitem[Pereyra et~al.(2017)Pereyra, Tucker, Chorowski, Kaiser, and Hinton]{pereyra2017regularizing}
Gabriel Pereyra, George Tucker, Jan Chorowski, {\L}ukasz Kaiser, and Geoffrey Hinton.
\newblock Regularizing neural networks by penalizing confident output distributions.
\newblock \emph{arXiv preprint arXiv:1701.06548}, 2017.
\bibitem[Walid~Ahmed(2022)]{walid2022training}
Yang~Liu. Walid~Ahmed, Habib~Hajimolahoseini.
\newblock Training acceleration of low-rank decomposed networks using sequential freezing and rank quantization.
\newblock In \emph{EIW 2022, Montr{\'e}al, Canada.}, 2022.
\bibitem[Yong et~al.(2020)Yong, Huang, Hua, and Zhang]{yong2020gradient}
Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang.
\newblock Gradient centralization: A new optimization technique for deep neural networks, 2020.
\bibitem[Zhang et~al.(2018)Zhang, Zhou, Lin, and Sun]{zhang2018shufflenet}
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.
\newblock Shufflenet: An extremely efficient convolutional neural network for mobile devices.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pp.\ 6848--6856, 2018.
\end{thebibliography}